From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Set
Authors: Mara Finkelstein
Daniel Deutsch
Parker Riley
Juraj Juraska
Geza Kovacs
Markus Freitag
Venue: ICML 2025 poster (2025)
Keywords: LLM-as-a-Judge
Autorater
Fine-grained evaluation
Machine Translation
In-context learning
Primary area: general_machine_learning->evaluation
TL;DR: We propose a method for developing an LLM-as-a-Judge metric which is specialized to a given test set, and show that this metric significantly outperforms non-specialized metrics.

Abstract: As LLMs continue to become more powerful and versatile, human evaluation has become intractable at scale and reliance on automatic metrics has become the norm. Recently, it has been shown that LLMs are themselves state-of-the-art evaluators for many tasks. These *Autoraters* are typically designed so that they generalize to new systems *and* test sets. In practice, however, evaluation is performed on a small set of fixed, canonical test sets, which are carefully curated to measure the capabilities of interest and are not changed frequently. In this work, we design a method which specializes a prompted Autorater to a given test set, by leveraging historical ratings on the test set to construct in-context learning (ICL) examples. We evaluate our *Specialist* method on the task of fine-grained machine translation evaluation, and show that it dramatically outperforms the state-of-the-art XCOMET metric by 54% and 119% on the WMT'23 and WMT'24 test sets, respectively. We perform extensive analyses to understand the representations learned by our Specialist metrics, and how variability in rater behavior affects their performance. We also verify the generalizability and robustness of our Specialist method across different numbers of ICL examples, LLM backbones, systems to evaluate, and evaluation tasks.

From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Set
Mara Finkelstein Daniel Deutsch Parker Riley
Juraj Juraska
Geza Kovaes Markus Freitag
Abstract
has come to the fore in the era of large language models (LLMs). Moreover, while human evaluation has historically been considered the gold standard for measuring model quality, it has become a key bottleneck during model devel- opment. In addition to being costly, slow, and difficult to scale, human evaluation is also limited by subjectivity (Kr- ishna al., 2023) and high variability in judgments across human raters (Karpinska et al., 2021; Riley et al., 2024; Zhang et al., 2024b). Increasingly, automatic metrics are replacing human evaluation for measuring the quality of generative models, and LLMs themselves have been shown to be state-of-the-an evaluators (also known as Autoraters) across a range of capabilities (Kim et al., 2023; 2024; Vu et al., 2024; Li et al., 2023).
As LLMs continue to become more powerful and versatile, human evaluation has become in- tractable at scale and reliance on automatic met- rics has become the norm. Recently, it has been shown that LLMs are themselves state-of-the-art evaluators for many tasks. These Autoraters are typically designed so that they generalize to new systems and test sets. In practice, however, evalua- tion performed on small set of fixed, canonical test sets, which are carefully curated measure the capabilities of interest and are not changed fre- quently. In this work, we design a method which specializes a prompted Autorater to given test set, by leveraging historical ratings on the test set to construct in-context learning (ICL) examples. We evaluate our Specialist method on the task of fine- inneggrined machine translation evaluation, and show that it dramatically outperforms the state- of-the-art XCOMET metric by 54% and 119% on the WM and 24 test sets, respec- tively. We also evaluate our method on the task of open-ended story generation evaluation, where we show that it outperforms the ope-specia baseline by 31% on the HANNA benchmark. We perform extensive analyses understand the rep- resentations learned by Specialist metrics, and how variability in rater behavior affects their per- formance. We also verify the generalizability and robustness of our Specialist method across differ- ent numbers of ICL examples, LLM backbones, systems to evaluate, and evaluation tasks.
The race to build ever-more- perm--rrrrorpanttttt LLMs has acceler- ated not only this shift from human to automatic evaluation, but has also brought demand for standard test sets on which LLM quality is measured and compared (Hendrycks et al., 2020; Liang et al., 2022; Zheng et al., 2023a;b). Evaluat- ing new systems on fixed of benchmarks, which are carefully curated to measure certain capabilities of interest and not changed frequently, allows for fair comparison against previous work and is the standard in the literature. Thus, while automatic metrics are typically designed so that they generalize to new systems and test sets, in practice, is very important that the evaluation metric being used work well across systems on the given test set, and less important that the metric generalize to other, unseen and unused test sets. In this work, propose simple and highly effective method to build LLM-based Autoraters which are special- ized to given test set, by leveraging historical ratings on the test set to construct ICI. examples.
Our contributions can be summarized as follows:
1. Introduction
We propose a novel method for constructing LLM- based Autoraters for NLG evaluation, which are spe- cialized to a given test set (see Figure la). This method only requires multi-shot prompting (no finetuning).
While evaluation of natural language generation (NLG) sys- tems has been long-standing challenge, its importance
'Google. Correspondence to: Mara Finkelstein
<mrrafin@ goa.inioooooll..mm.......
We show that this method can be used to create an automatic metric for fine-grained machine translation (MT) evaluation (called Specialist AutoMQM) which dramatically outperforms the existing state-of-the-art,
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to Test Set
Legand
Spectaltet
Flaud, diftrrent source
Wiatorical Model Ourpuns
Historical Ratinge
ICL
Auttorster Evalusten New System
[Ray(Ko)]
Paatrr
How one you?
(xu)
Chance loarred
good.
Rator
AMtiMQMM
Tranalone pertencce
[i pit
Ray ((ww)) por
N (4m]
Aig Rssatiber
Treckess cate modismo. Traduct cato frase. Trokk anocite freciszo
Tredkaso aste rod Traduce cate carte frome
Truckee satt frame
diecares
(a) Specialist Method
(b) AutoMQM Task
Figure 1. (a) Illustration of the Specialiss method, compared against the F Fixed, di tftrent source baseline. for prompting an LLM-based Autorater. Both methods construct unique set ICL examples for every test set example, consisting ratings of historical system outputs for some fixed source. The difference between these methods is that the Specialist ICL examples consist of ratings of outputs from the same source as the test example. (b) Illustration of the AutoMQM task for fine-grained MT evaluation.
achieving character-level F1 improvements of 54% and 119% on the 2MT' and 2M test sets, respec- tively, relative to XCOMET (Guerreiro et al., 2023).
made by LLMs, and their explanations are unreliable.
Modeling Rater Behavior For many (NLG) tasks on which LLMs are evaluated, there is high variability in judg- ments across human raters (Karpinska et al., 2021; Riley et al., 2024). For some (open- text generation) tasks, the evaluation criteria have some degree of subjectivity (Kr- ishna et al., 2023). Especially for expert-level evaluation tasks, differences in rater quality and conscientiousness can manifest inter- annotato disagreement (Karpinska et al., 2021). Raters can also have different stylistic preferences, and some grade more leniently or harshly than others (Ri- ley et al., 2024). Given these differences in rater behavior, several recent studies have sought to model behavior of mul- tiple raters when designing automatic metrics (Zhang et al., 2024b; Geva al., 2019; Chen al., 2024; Golazizian et al., 2024). In this work, we instead investigate whether our pro- posed method effectively specializes to a single rater. In practice, modeling a single, high-quality rater is often more desirable than modeling multiple, noisy raters.
We perform extensive ablations and analyses to ver- ify that the representations that Specialist AutoMQM learns from the ICL examples are non-trivial. We also show that the Specialist method is robust to the choice of LLM and to the systems being evaluated, and that this method generalizes to the different, but related, task of score prediction for machine translation.
We investigate how variability in judgments across dif- ferent human raters affects performance of Specialist AutoMQM, and conclude that this metric specializes not only to the test set, but also to the rater.
2. Related Work
LLM-as-a-Judge Autoraters Recently, it has been shown that, for many NLG tasks, LLMS are themselves state-of-the-ar evaluators (Kim al., 2023; 2024; Vu et al., 2024; Li et al., 2023). Some of these "LLLM-as-a-Judge Autoraters are finetuned on human judgements (Kim al., 2023; 2024; Vu et al., 2024; Li et al., 2023), while others are simply prompted (Kocmi& & Federmann, 2023a; Yuan et al., 2023). Prompting LLMs with in-context learning (ICL) examples is a common approach for eliciting their rea- soning and
Machine Translation Evaluation In this work, we focus the task of MT evaluation to investigate the effective- ness of our Specialist method. MT is a core NLG task, and automatic MT evaluation is one of the most well-studied evaluation problems in NLP (Callison-Burde et al., 2008; Freitag et al., 2023). In line with broader trends, research in automatic MT evaluation has recently shifted towards the LLM-as-a-Judge paradigm. Koemi & Federmann (2023b) showed that LLMs prompted to predict scalar scores state-of -ft evaluators of MT quality at the system level (though still lag behind finetuned MT evaluation metrics at the segment level). While these score-based metrics have high correlation with human judgments, the scores that they produce are difficult to interpret, and do not provide ac-
capabilities (Tanzer et al., 2023; Yan al., 2023). While traditional automatic metrics predict scalar quality scores, the transition towards gener- ative Autoraters for evaluation opens up the possibility to elicit feedback more flexibly. including fine-grained and interpretable feedback (Fernandes et al., 2023; Koemi & Federmann, 2023a). However, Kamoi et al. (2024) showed that GPT-4 and Claude-3 have low recall in detecting errors
2

---

From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Set
tionable insights into the limitations of the model being evaluated how to improve it (Xu al., 2024; Zhang et al., 2024a). Recent work in creating interpretable automatic MT metrics has built upon an existing, satte-of-the-ar frame- work for interpretable human evaluation of translation qual- ity: the Multidimensional Quality Metrics (MQM; Lommel et al., 2014 4; Freitag et al., 2021) framework, in which profes- sional annotators are asked identify and label individual error spans in MT outputs, along with the corresponding enror category (e.g..fleency accuracy, etc.) and severity (minor, major, or critical). Fernandes et al. (2023) and Koemi& & Federmann (2023a) showed that LLMs can be few-shot prompted to provide MQM error annotations of MT outputs. However, these prompted Autoraters still un- derperform (Freitag et al., 2023) XCOMET (Guerreiro et al., 2023), an encoder model finetuned on human-generated MQM data, which predicts both scalar quality scores (with regression head) and error spans nnon-generativel)) via token-level tagging.
of (human-generaaed) ratings Ras, (from different transla- tion systems Mj) on the same test set. The pseudo-SxS setting primarily considered in this work has the additional requirement that, for each test set example 2;, all ratings N [0]] be performed by a fixed rater. In this work, the different translations for each input example come from different translation systems, but they could in principle also be sampled from single model (e.g,, using a diversity- promoting sampling algorithm). The Specialist method constructs ICL examples to be used for prompting an LLM- as-a-Judge on a per-example basis, so that ICL examples are unique for every example in the test set. In particular, for a given input z; in the test set, the ICL examples are con- structed from all of the (historical) ratings of translations of this same input (line 9 in Algorithm 1). That is, given a new translation system M to evaluate on the test set, the ICL. examples used to evaluate the translation Y = M (x;) are given by { (RM, j=1' Once the ICL examples are con- structed, the LLM ís prompted with these demonstrations, as well as the corresponding source r; and model translation Y to evaluate (line 13 in Algorithm 1).
{ RM, [i]] 1=ز
[
3. Specialist Method
In this work, we propose the Specialist method for develop- ment of a prompted L.I. LLM--s-a-Judge metric, which special- izes the metric to given test set based the ICL examples provided. This method will be phrased in terms of the MT evaluation task, but its formulation generalizes to any NL.G evaluation task (i.c., any task which evaluates output from generative models).
Algorithm 1 Specialist Method for Automatic Evaluation
Given:
1: Test set x= {=)) K
2: Translation system M to evaluate
Require:
3: Off-the-shelf LLM E' to use as the prompted Autorater
4: Set R of ratings on X for N translation systems: R =
{RM, 1--1 where M, * M for all j E (1,...,NN N).
Prerequisites First, we establish some basic terminology. The objective is to evaluate the performance of an MT sys- tem (i.e., model) M on a fixed test set. A test set simply consists of a set of sources X, which are the inputs to the system(s) to be evaluated. (In this setting, we do not require access to gold reference translations of these sources.) The translations YM are the outputs of M the test set: that is, Yas +M((x) x E X}. Evaluation of system M on the test set (whether by MQM, AutoMQM, score prediction, etc.) produces a set of ratings RM= = (rating(y): : E Yur for the translations YM--
Pseudo-SxS Constraint: For each i, ratings [1]]
j-1 were performed by single rater.
Ensure: Ratings Ry- of system M on test set X
5: RM- + 0
6: # Iterate over examples in the test set
7: for i 4- 1 to K do
8: # Construct ICL examples from all historical ratings for the same input x;
9:
(ICL. examples), = (RM, [1))
10:
# Compute output of M on this test set example
11:
Yi--- (2x)))
Specialist Algorithm The Specialist method can be sum- marized as follows: Given access to test set X augmented with historical translation quality ratings from multiple sys- tems, and given the predictions of new translation system M which we want to evaluate on this test set, the Specialist metric evaluates the quality of M*(x) for every E X by prompting an LLM with ICL examples constructed from all ratings of historical translations of the same input xr. See Figure Ia for an illustration of the Specialist method.
12:
# Prompt E to evaluate the translation of the new
system M* of input x;, given the historical ratings
13: Rh, E (ICL examples); Iı, Yir-
14: Append Rig- to RM
15: end for
16: Return Ry-
Specialist Method in Practice The main constraint in de- velopment of a Specialist metric is the availability of ratings to use as ICL examples for the given test set. However, note that it is much more efficient to collect a set of ratings from a
More formally, Algorithm 1 outlines the details of this method, which requires access to multiple (historical) sets
3

---

From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Set
few translation systems for test set as one-off investment, than to repeatedly depend on human annotators for evalua- tion of new translation models (e.g., throughout the model development process). Performance of Specialist metrics as a function of the number of ratings will be explored in $5.3, where we show that ratings from only 3 translation systems are sufficient to exceed the state-of-the-art.
examples are shuffled across test examples.
Fixed, different source The same global set of ICL ex- amples (per test set) as the Specialist model is used, but these are permuted so that, for a given test example, its ICL examples come from a fixed source, which strictly differ- ent than that of the test example, but has the same rater. See Figure Ia for an illustration of this setup vs the Specialist.
4. Experimental Setup
For the "Shuffled sources" and "Fixed, different source" baselines, the following constraint is enforced: The ICL examples for a given test example cannot include any trans- lations, whether from the same or different source, produced by the same system as that which produced the test transla- tion. Moreover, both of these baselines the same number of ICL examples per test example the Specialist model.
4.1. Evaluation Task: AutoMQM
In this work, we develop Specialist metric for MT evalua- tion based on the Multidimensional Quality Metrics (MQM; Lommel et al., 2014; Freitag et al., 2021) protocol (see Section 2). We refer to any automatic evaluation metric which performs the task of MQM as AutoMQM. That is, an AutoMQM metric predicts error spans, and identifies corresponding error categories and severities, according to the MQM framework (Figure lb). In $5.6, we also evaluate our proposed Specialist method on the task of MT evalu- ation via direct assessment (scalar score prediction). The instructions used to prompt our AutoMQM and direct as- sessment metrics are shown in Figures 5 and 6 in Appendix A, respectively. The AutoMQ prompt was adapted from the GEMBA instructions (Koemi & Federmann, 2023a). As indicated in the prompt, the output (and ICL examples) are expected to be provided in JSON format, with each er- ror having span, severity and category fields. See Table 7 in Appendix A for an example AutoMQM output.
4.3. Test Sets
The Specialist method (described in $3) depends having access to a test set augmented with multiple ratings (of dif- ferent translations) for each input, Such ratings have already been collected as part of the Conference on Machine Trans- lation (WMT) Metrics Shared Tasks in 2023 (Freitag et al., 2023) and 2024 (Freitag et al., 2024). We will refer to these datasets as WM and WM respectively. We use the WMT' 23 MQM ratings for Inglish-German (en-*de) and Chinese-English (zh-ven), and the 2M MQM ratings for English (en-*de), English-Spanish (en (en->es), and Japanese- (ja-zhh. We exclude the human- generated references, so that our metrics are reference-free (i.e., QE). See Table 8 in Appendix B for the number of translation systems per language pair. Except for the en en->es W4 dataset, all ratings were collected in a pseudo-SxS fashion (Riley et al., 2024), which means that fixed rater was assigned to rate all translations of given input.
4.2. Models
We use the Gemini 1.5 Pro model (Gemini Team, 2024) as the prompted LLM Autorater for all experiments (unless otherwise indicated; see 55.1).
4.2.1. BASELINES
4.3. 1. ADDITIONAL ROUNDS OF RATINGS
We compare our proposed Specialist AutoMQM metric against the following baselines:
In order to better understand how inter-rater variability af- fects the performance of the Specialist metric, we take ad- vantage of additional rounds of MQM ratings (see $5.5).
XCOMET (Guerreiro et al., 2023) State-of-the-art auto- matic metric for span-based MT evaluation.
GEMBA-MQM (Kocmi & Federmann, 2023a) Closest precedent to our proposed metric, which also prompts an LLM (GPT-4) for the task of MQM prediction. GEMBA uses fixed set of 3 (English-German English-Czech, and Chinese-English) ICL examples.
WM Round2 and Round3 Two additional rounds of WMT' MQM ratings, rated by the same set of raters as in the first round, but with individual translations being assigned strictly different raters in each round. As with the first round, the second two rounds of ratings were also collected in pseudo-SxS fashion.
MetricX (Juraska et al., 2024) State-of-the-art automatic metric for MT evaluation and winner of the WM Met- rics Shared Task (Freitag et al., 2024). This model is used as baseline for the Specialist Scorer experiments in 85.6.
2M Multi-Rater Subset An extension to the (Round1) WMT' zh-ten MQM ratings, whereby 10% of the test set (18 source segments x 15 systems = 270 examples) was rated by all raters
Shuffled sources The same global set of ICL examples (per test set) as the Specialist model is used, but these ICI.
4

---

From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to Test Set
Character-level F1
Same source as test
Fixed source
Same rater
WMT
WMT
Baselines
Ia) XCOMET-XXI-QQ
33.50
16.23
lb) GEMBA-MQM-QE
X
X
X
31.99
1c) Shuffled sources
X
X
34.65
24.46
1d) Fixed, different source
X
27.06
19.96
2a) Specialist
/
51.59*
35.59*
Table 1. Character-level Fl on the 23 and 24 test sets, for Baseline and Specialist metrics (see $4.2 for metric descriptions). averaged over all language pairs per test set. See Table 11 in Appendix for F1, precision, and recall, broken out by language pair. For WMT"23, the "Shuffled sources" and "Fixed, different source" results computed as the average over 10 runs with different random seeds. See Table 12 in Appendix B for the variance across runs. Asterisks (4) indicate scores which are statistically significantly better than XCOMET (row la), according paired permutation test.
4.4. Creation of Specialist ICL examples
5. Results and Discussion
The Specialist metric (as described in (3) evaluates the qual- ity of a single translation system M given known ratings for set of N other translation systems [ Here, for each test set (WMT'23 and WMT we have access to -ound ratings for all system outputs. In order to meta-evaluate the Specialist AutoMQM metric, we first col- lect predictions from this metric for each system, via hold- one-out prompting; that is, for whichever system we are evaluating, we exclude that system's ratings from the ICL examples and prompt with the ratings from the remaining systems. Then, we gather the Specialist AutoMQM predic- tions across all systems to perform meta-evaluation of this metric over the entire test set of interest. (Note that we also include a performance breakdown by system in 55.2.) See Tables (WMT'23) and 10 (WMT'24) in Appendix for the average number of ICL examples and average number of total errors across ICL examples per test example.
The main results are shown in Table 1. First note that the "Shuffled sources" baseline (row Ic) already performs (at least) on par with the state-of-the-art AutoMQM models (XCOMET in row la and GEMBA in row lb). Also note that the "Fixed, different source" baseline (row Id) under- performs "Shuffled sources", which suggests that special- izing to a different input is worse than no specialization. In contrast, the "Specialist" metric (row 2a) dramatically outperforms all of the baselines, with a 54% improvement in Fl score relative to XCOMET on WMT'23, and a 119% rel- ative improvement on 24. Recall that the difference between the "Specialist" setting and the "Fixed, different source" setting is that, in the former, ratings from transla- tions of the same source as the test example are provided as demonstrations. Thus, same-source demonstrations are crucial to the success of our method, and its success cannot be attributed only to (i) providing demonstrations of ratings from different translations of some fixed source, or (ii) pro- viding demonstrations from the same rafer as the test rating ground truth.
4.5. Meta-evaluation
To meta-evaluate Specialist AutoMQM, we compute the character-level precision, recall, and Fl span tagging evalu- ation metrics (Blain et al., 2023). Given gold and predicted ratings, these metrics compute the precision, recall, and Fl of predicting whether a character in the hypothesis transla- tion is included in an error span or not. Partial credit of 0.5 is given if the predicted rating correctly marks a character as an error but predicts the incorrect severity. To meta-evaluate the Specialist Scorer (which is not span-based metric) in 85.6, we report segment-level pairwise accuracy with tie calibration (Acc23; Deutsch et al. (2023)), which was used to evaluate WMT'23 and W4 Metrics Shared Task submissions. Acc23 rewards metrics for correctly ranking translations and correctly predicting ties, in combination with a tie calibration procedure that introduces ties into metric scores so that the meta-evaluation is fairer.
5.1. Is Specialist Auto Robust to the Choice of LLM?
The results reported in Table I used the Gemini 1.5 Pro LLM. To investigate whether gains from this method generalize to other LLMs, we also performed the same comparison using GPT-4o (Achiam et al., 2023) and Claude 3.5 Sonnet (Bai et al., 2022). As shown in Table 2, Specialist AutoMQM also substantially outperforms Shuffled AutoMQM when prompting these other LLMs. Moreover, note that Spe- cialist AutoMQM with the GPT-4o backbone outperforms GEMBA (which is also a prompted GPT-4 model, albeit an earlier version) by an even larger margin. This supports the effectiveness of our approach over baselines using ex- ternal (different-ssurcee) ICL examples. In the remaining experiments, we continue to use the Gemini 1.5 Pro LLM.
5

---

From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to Test Set
Gestage
Avstage NON Score System (a) 2M en-de
Average NON SCOrE of System
(b) WMT 23 zh-en
Figure 2. Specialist AutoMQM performance 23 translation system. The Specialist and Shuffled baseline models from Table I (rows 2a and Ic, respectively) are compared. The average MQM score of each system, along with name, shown on the x-axis. The average character-level F1 of the AutoMQM model when evaluating this system only is shown on the y-axis.
Character- F1
en en-
zh->en
for certain quality tier. Note that both the Specialist and shuffled baseline AutoMQM models tend to perform worse on the highest-quality systems (e.g. GPT4-5shot), likely due to limitations the underlying translation capabilities of the backbone language model used for AutoMQM.
Bavelines
XCOMET-XXL-QE
32.71
34.29
GEMBA-MQM-QE
29.80
34.17
Gemini-AutoMQM
Shuffled sources
31,49
37.80
Specialist AutoMQM
45.71
57.47
5.3. How Does Specialist Auto Performance Scale as a Function of Number of ICL Examples?
GPI-Ao-AttoMOM
Shuffled sources
33.58
38.38
The most expensive and time-consuming step in the devel- opment of Specialist metric collecting ratings use as demonstrations. Thus, it is useful to understand the marginal improvements in performance that can be expected as re- sult of collecting additional ratings. In this ablation, we randomly select subsets of Specialist AutoMQM ICL. exam- plcs in the range [1, num systems 1]. While scaling ICL examples, we incrementally add single new example the existing (for each test example), so that every examples of given size is superset of the ICI. examples for all sizes less than n. Note that when the number of ICL examples reaches its maximum (of num systems 1), this corresponds to the results reported in row 2a of Table 1. As shown in Figure 3, increasing the number of ICL examples improves character-level Fl monotonically up to ICL examples for en dde, and up to 12 ICL examples for zh-ten. We see that only 3 ICL examples Specialist AutoMQM to outperform the XCOMET baseline (for both en end and zh>>en) and that Specialist AutoMQM outperforms the "Shuffled sources" setting (Table 1, row Ic) at every ICL example set size.
Specialist AutoMQM
48.32
56.78
Chade:3.-Sonnet-AuuMMMMM
Shufiled sources
35.02
40.86
Specialist AutoMQM
48.49
56.13
Table Comparison of Specialist AutoMQM the shuffled base- line (WMT'23 test set) for three different LLMs: Gemini 1.5 Pro, GPT-4o, and Claude-3.5-Somnet. For all three LLMs, Specialist AutoMQM substantially outperforms the shuffled bascline.
ICL
5.2. How Does Specialist AutoMQM Performance Vary Across Translation Systems?
In practice, the Specialist AutoMQM metric would likely be used to evaluate the quality of new translation system(s), given ratings from historical systems. The WMT'23 and WMT'24 datasets contain rated translations from at least a dozen systems per language pair (Table 8). These systems are of varying quality, and aggregate meta-evaluation of AutoMQM (Table could hide per-system differences in metric performance. Here, compare performance of Spe- cialist AutoMQM against the "Shuffled sources" baseline on per-system basis for WM As shown in Figure 2, Specialist AutoMQM outperforms the shuffled baseline for every zh-te system, and for every en en->de system ex- cept the lowest-quality one. Thus, Specialist AutoMQM outperformance consistent for translation systems across the quality spectrum, and cannot be explained by gains only
needed for
5.4. Is Specialist Auto Simply Copying Errors From ICL Examples?
5.4. LEARNING WHEN TO ABSTAIN
In view of the results from $5.3, i.e., that increasing the number of ICL. examples improves performance of Spe-
6

---

From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to Test Set
8.
B
18
Prccazn
XCOMETT Stuffid cendine
XCONETES
Stufted condinel
10
ILL coorples
of cumrpko
(a) WMT'23 en-de
(b) 23 zh-en
Figure 3. Specialist AutoMQM performance a function of number of ICL examples used. For comparison, XCOMET performance, as well as ICL example scaling for the "Shuffled sources" baseline (Table I, row Ic), are also shown.
cialist AutoMQM, this raises the question of whether the gains are simply due to the model copying errors that it is shown the ICL examples. As shown in Table 14 in Appendix (left two columns), when comparing Specialist AutoMQM prompted with 3 versus 11 ICL examples, the AutoMQM system prompted with 11 ICL examples predicts fewer errors which direct copies of spans from all 11 ICL examples, than the AutoMQM system prompted with ICL examples. The comparison is even more pronounced when comparing the "Shuffled Sources" baseline against Special- ist AutoMQM with 11 ICL. examples (right two columns of Table 14). After removing errors predicted by both sys- tems, the shuffled baseline predicts 6,374 errors which are direct copies of spans from Specialist ICL examples, while Specialist AutoMQM only predicts 494 such errors (even though Specialist AutoMQM predicts more total errors than the shuffled baseline). Thus, Specialist AutoMQM is ab- staining from predicting errors that is shown via ICL examples, while the shuffled baseline, which has not been shown these errors, is predicting them more liberally.
The gap in precision, on the other hand, quantifies the ex- tent to which Specialist AutoMQM correctly abstains from predicting errors present in ICL examples. See examples of such predictions in Table 17 (Appendix B).
Character-level F1
en-*de
zh-ten
Shuffled sources
31.49
37.80
Parrot
27.59
36.52
Specialist AutoMQM
45.71
57.47
Table 3. "Parrot" vs Specialist AutoMQM performance (3MT"23),
5.5. Is Specialist Auto.MQM Specialized Only to Test Sets, or Also to Raters?
By construction, Specialist AutoMQM is specialized to a test set. Since the WMT'23 and 24 test sets are con- structed in a pseudo-SxS fashion (with the exception of WM en-*es), Specialist AutoMQM is also prompted with ICL examples rated by the same rater as the test rating ground truth. Here, we seek to understand whether Spe- cialist AutoMQM also specializes to the rater. To answer this question, we take advantage of the additional rounds of WM MQM ratings, as described in $4.3.1.
5.4.2. PARROT MODEL BASELINE
To quantify how much of Specialist AutoMQM's perfor- mance can be attributed to eтrоr copying from ICL exam- ples, we construct an artificial baseline, which we call the "Parrot". This model has access to the same ICL examples as Specialist AutoMQM, and makes predictions as follows: For every error present in ICL examples for which there is matching span in the test translation, predict this as error. As shown in Table 3, there is large gap in character-level F1 between the Parrot model and Specialist AutoMQM (27.6 vs 45.7 for WM en-*de, and 36.5 to 57.5 for WM zh ze). Thus, the performance of Specialist AutoMQM cannot be solely explained by naive copying behavior. The large gap in recall between the Parrot model and Special- ist MQM (see Table 15 in Appendix B) quantifies the extent to which Specialist AutoMQM correctly identifies errors not present in ICL examples. Examples of such Au- toMQM predictions are shown in Table 16 (Appendix B).
Character- F1 en en>xd
zh-ten
Human agreement
la) Round2
34.91
38.68
1b) Round3
38.46
39.16
Specialist
2a) Round1 ICL
45.71
57.47
2b) Round2 ICL
30.80
38.83
Table 4. Specialist AutoMQM performance when prompting and evaluating using different raters (WMT'23 test set). Results are reported using the official Round1 test set.
Prompting with Different Raters In the first set of exper- iments, we use the Specialist AutoMQM set-up, but prompt with ICL examples from Round2 (rather than Roundl) rat- ings. We always evaluate using the official Round1 ratings.
7

---

From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to Test Set
reteri
0.49
0:33
0:45
0.36
0:37
0.41
0:35
0.35
#2ter2
0.3
0.37
0.51
0:54
01:99
0:37
0.35
0.32
0.6
0:47
0:53
0.37
0.47
0:48
rster
0.3
3
0:44
0.59
0:41
0.5
0.5
0.42
0.
0.5
03
rutard
0.32
0.53
0.45
0.49
0:38
0.47
0:45
0.4
raterm4
0.31
0.59
0.39
0:30
0:43
0.43
0
044
10cor>
0.39
0.44
0.55
0:41
0:31
8.42
0.35
0.35
020er
0.54
0:42
0.32
0.00
0.57
0.53
0.57
0.3
0.37
0:45
rater6
0.5
0.48
meter7
0.32
0:41
0.46
6:35
66
0.56
rster7
0.5
0.44
0.45
0.39
reterd
0:34
0:41
0:42
0.14
0.36
0.67
0:30
0.53
<sterd
0.33
0:42
0:37
0:43
0.37
0.47
0:30
0.0
0.0
rsterjd_2
(a) AutoMQM Character F1
(b)
Character F1
Figure 4. Cross-rater performance of AutoMQM and human annotators on the zh->en 2M Multi-Rater Subset (34.3.1). In Figure (a), Specialist AutoMQM is prompted with ICL examples from the icl reltrater.it .id rater (vertical axis), and evaluated using the ratings from the test_rater_id rater (horizontal axis). In Figure (b), the matrix of character-level F1 scores between all pairs of human annotators is shown.
As shown in Table 4 (row 2b), performance drops to that of the "Shuffled sources" " baseline (Table 1, row Ic). Thus, Specialist AutoMQM also specializes to the rater (on a per- example basis; recall that with the pseudo-SxS setup, raters can still vary across different inputs). This is not surprising, since there are large (and often competing) differences in behavior across raters: See rows la) and 1b) in Table 4 for the inter-annotator agreement across rounds. Note that the Round2 Specialist (row 2b) performs on par with the inter-annotator agreement for zh->en, and the Roundl Spe- cialist (row 2a) outperforms the inter-annotator agreement for both language pairs, likely because it is able to match specific rater behavior from the ICL examples. In follow- up experiment, we provide merged ratings from Round2 and Round3 as ICL. examples. As shown in Table 18 in Appendix B, merging ratings (row 2c) improves recall at the cost of lower precision. The drop in precision likely rep- resents not an actual quality drop, but under-annotation of errors (low recall) by the Roundl raters (used as the ground truth). It remains an open question how combine ratings from multiple raters to create better ground truth.
that when prompting and evaluating with different raters, on average Specialist AutoMQM agrees with the raters as much as the raters agree with each other.
2M
2M
Segment-level Acc23
en- zh thh en de on->es ja ja->zh
MetricX-24-QE
59.44
54.48
52.45
68.48
52.69
AutoMQM
Shuffled sources Specialist
53.55
49.62
50.39
68.21
51.57
58.13
57.79
60.38
68.91
55.01
Score Prediction
Shuffled sources
52.32
49.32
47.96
68.43
52.58
Specialist
56.77
55.93
56.78
68.58
56.56
Table 5. Comparison Specialist AutoMQM vs the Specialist Scorer, on the WMT' and WMT'24 test sets.
5.6. Does the Specialist Method Generalize to Other Automatic Evaluation Tasks?
ICL Rater x Test Set Rater Comparison The aggregate results reported in Table could mask individual cross- rater dynamics. While we do not have access to ratings from all raters for every system output across all test set examples, we do have access to the WM Multi-Rater Subset ratings ($4.3.1), for which all 8 raters rated 10% of the zh-ten dataset. We use this 2MT'23 Multi-Rater Subset to understand AutoMQM performance for all pairs of raters (where one rater is the ICL example annotator and the other rates the test set example). by computing the entire numeraters num_raters matrix of Fl scores for every (ICL rater, test set rater) pair. These results are shown in Figure 4(a). As expected, the Fl scores on the matrix diagonal are highest (though note that rater2 and rater4 also have high agreement). For comparison against AutoMQM, Figure 4(b) shows the inter- annotator agreement (character-level F1) over the same WM Multi-Rater Subset. Observe
Translation Direct Assessment We have seen that the Specialist method for prompting LLMs-as-Judges achieves sfate-of-the-at performance for the task of AutoMQM. Here, we consider the task of scoring translation quality (without providing error annotations). In this task, we prompt the LLM to generate a float quality score on scale from 0- 100. (See Figure 6 in Appendix A for the prompt used.) As shown in Table 5, the task of score prediction also benefits substantially from the Specialist method (relative to the shuffled baseline). Here, we report segment-level accuracy, and also compare against MetricX-24 (Juraska et al., 2024), the state-of tfeth--rtt automatic score prediction metric for machine translation. Also note that Specialist Au- toMQM has a quality advantage over the Specialist Scorer, while also offering the added benefit of interpretability (per- haps because LLMs are better at natural language text gen- eration versus generation of numbers). Finally, observe that
8

---

From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Set
Relevance (excl. HS)
Relevance (incl. HS)
Average (excl. HS)
Average (inel. HS)
Pearson
Spearman
Pearson
Spearman
Pearson
Spearman
Pearson
Spearman
Shuffled baseline Specialist
43.71
41.51
59.58
52.88
36.74
32.35
54.32
44.40
49.01
46.99
63.76
57.90
46.34
42.36
66.16
54.78
Table Story-level Pearson and Spearman correlations of the Shuffled baseline vs Specialist metric, evaluated on the HANNA benchmark. We report results for both the relevance criterion and the average over all criteria, both with human stories included (indicated in table by incl. HS) and without human stories (excl. HS).
both Specialist models outperform MetricX-24 across all 2M and 2M language pairs except 2M en
follow-up papers, such as Yuan et al. (2023), included them.
As shown in Table 6, the Specialist metric beats the Shuffled baseline according to both criteria (relevance and average score) and both meta-evaluation metrics (story-level Pearson and Spearman correlations). For average score prediction (excluding human stories), the Specialist metric outperforms the Shuffled baseline by 26.1% 1% according to Pearson corre- lation, and by 30.9% according to Spearman correlation.
Long-Form Story Generation Assessment Both tasks considered so far have evaluated translation quality (via either error annotation or direct assessment). To validate the generalization of the Specialist method, here we con- sider the task of story generation, which is an open-ended NLG task with fundamentally different characteristics from translation evaluation. To evaluate story generation, we use the HANNA benchmark (Chhun et al., 2022), which satisfies the requirements of the Specialist method since contains outputs from multiple systems for every input prompt, augmented with human annotations. While the an- notations were not collected in pseudo-SxS fashion ($4.3), each output was annotated by three human raters, and (fol- lowing Chhun et al., 2022) we average over these ratings during meta mnttaevvlluttion to smooth out inter-rater variability. The annotations evaluate each story according to six crite- ria: relevance, coherence, empathy, surprise, engagement, and complexity. We evaluate our baseline and Specialist metrics on the first criterion (relevance) and on the average score over all criteria (which represents an overall indication of the story's 's quality). and report story-level Pearson and Spearman correlations.
6. Conclusion
In this work, we have proposed the Specialist method for development of automatic evaluation metrics which are spe- cialized to given test set. We have shown that Special- ist AutoMQM dramatically outperforms all existing state- of-the-art span-based MT evaluation metrics, on both the WMT' and 2M test sets. Specialist evaluators are easy to implement, as they are simply multi-shot prompted LLMs. Moreover, the Specialist method is task-agnostic, and an immediate avenue for future work would be ap- ply this method development of metrics for other NLG tasks. These Specialist metrics could serve a power- ful alternative to human judges in evaluating LLM quality across a wide range of capabilities. Another avenue for future work is to better understand how to combine ratings from multiple raters, both for creation of ICL examples for Specialist metrics, and for creation of more trustworthy test sets (which capable of measuring super-human per- formance). Finally, the Specialist method as framed here requires
We follow the same experimental methodology as described in Section 4.4, and construct Specialist ICL examples for each test example via holdtone-syttemmouu prompting: that is, for whichever system output we are evaluating, we ex- clude that system's ratings from the ICL examples and prompt with the ratings from the remaining systems. This simulates the real-world use case of evaluating a new sys- tem. Moreover, as described in Section 4.2, we enforce the following constraint when constructing ICL examples for the Shuffled baseline: The ICL examples for given test example cannot include any outputs, whether from the same or different prompt, produced by the same system as that which produced the test output. We evaluate both with and without human stories included, since the HANNA paper (Chhun et al., 2022) excluded human stories in its meta- evaluation (as these were considered outliers), while some
ratings to be used as ICL exam- ples, but future work could explore whether L.MSs are also capable of generating these ratings.

---

From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to Test Set
Impact Statement
Freitag, M., Mathur, N., Lo, C.-k., Avramidis, E., Rei, R., Thompson, B., Koemi, T., Blain, F., Deutsch, D., Stewart, C., et al. Results of wmt23 metrics shared task: Met- rics might be guilty but references are not innocent. In Proceedings of the Eighth Conference on Machine Trans- lation, pp. 578-628, 2023.
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
References
Freitag, M., Mathur, N., Deutsch, D., Lo, C.-K., Avramidis, E., Rei, R., Thompson, B., Blain, F., Kocmi, T., Wang, J., et al. Are Ilms breaking mt metries? results of the wmt24 metrics shared task. In Proceedings of the Ninth Conference on Machine Translation, pp. 47-81, 2024.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L.,, Akkaya, I,, Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,
Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08874, 2023.
Gemini Team. Gemini: A family of highly capable mul- timodal models, 2024. abs / 221...12120
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKin- non, C., et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.0088 12.00033, 2022.
Geva, M., Goldberg, Y., and Berant, J. Are modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets. arXiv preprint 1rXiv:990.007890 2019.
Blain, F., Zerva, C., Rei, R., Guerreiro, N. M., Kanojia, D., de Souza, J. G., Silva, B., Vaz, T., Jingxuan, Y., Azadi, F., et al. Findings of the wmt 2023 shared task on quality estimation. In Proceedings of the Eighth Conference on Machine Translation, pp. 629 653,, 2023.
Golazizian, P., Ziabari, A. S., Omrani, A., and Dehghani, M. Cost-efficient subjective task annotation and modeling through few-shot annotator adaptation. arXiv preprint 1xXi::2422.44101,, 2024.
Callison-Burch. C., Koehn, P., Monz, C., Schroeder, J., and Fordyce, C. S. Proceedings of the third workshop statistical machine translation. In Proceedings of the Third Workshop on Statistical Machine Translation, 2008.
Guerreiro, N. M., Rei, R., van Stigt, D., Coheur, L., Colombo, P., and Martins, A. F. xcomet: Transparent machine translation evaluation through fine- fine-grained error detection. arXiv preprint arXiv:2310.10440 2023.
Chen, B., Wang, X., Peng, S., Litschko, R., Korho- nen, A., and Plank, B. seeing the big through the small": Can Iims approximate human judgment distri- butions on nli from few explanations? arXiv preprint arXiv:2406. 17600, 2024.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint 2XXx:2009.03300 2020.
Chhun, C., Colombo, P., Clavel, C., and Suchanek, F. M. human criteria and automatic metries: benchmark of the evaluation of story generation. arXiv preprint arXiv:2208. 11646, 2022.
Juraska, J., Deutsch, D., Finkelstein, M., and Freitag, M. Metricx-24: The google submission to the wmn 2024 metrics shared task. arXiv preprint arXiv:2410.03985 2024.
Deutsch, D., Foster, G., and Freitag, M. Ties matter: Meta- evaluating modern metrics with pairwise accuracy and tie calibration. arXiv preprint arXiv:2305. /4324, 2023.
Kamoi, R., Das, S. S., Lou, R., Ahn, J. J., Zhao, Y., Lu, X., Zhang, N., Zhang, Y.. Zhang, R. H., Vummanthala, S. R., et al. Evaluating lims detecting errors in Ilm responses. arXiv preprint mrXiv:2404.03602 2024.
Fernandes, P., Deutsch, D., Finkelstein, M., Riley, P., Mar- tins, A. F., Neubig, G., Garg, A., Clark, J. H., Freitag, M.. and Firat, O. The devil is in the errors: Leveraging large language models for fine-graine fine-grained machine transla- tion evaluation. arXiv preprint 23Xiv:2308.0722 2023.
Karpinska, M., Akoury, N., and lyyer, M. The perils of using mechanical turk to evaluate open-ended text generation. arXiv preprint 1tX.:::219000833. 2021.
Freitag, M., Foster, G., Grangier, D., Ratnakar, V., Tan, Q., and Macherey, W. Experts, errors, and context: A large-scale study of human evaluation for machine trans- lation. Transactions of the Association for Computational Linguistics, 9:1460-14744 2021.
Kim, S., Shin, J., Cho. Y., Jang, J., Longpre, S., Lee, H., Yun, S., Shin, S., Kim, S., Thorne, J., et al. Prometheus: Induc- ing fine-grained evaluation capability in language models. The Twelfth International Conference on Learning Representations, 2023.
10

---

From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to Test Set
Kim, S., Suk, J., Longpre, S., Lin, B. Y., Shin, J., Welleck, S., Neubig, G., Lee, M., Lee, K., and Seo, M. Prometheus 2: An open source language model special- ized in evaluating other language models. arXiv preprint arXiv:2405.01535 2024.
Yan, J., Xu, J., Song, C., Wu, C., Li, Y., and Zhang, Y. Understanding in-context learning from repetitions. arXiv preprint arXiv:2310.00297, 2023.
Yuan, P., Feng. S., Li, Y. Wang. X., Pan, B., Wang, H., and Li, K. Batcheval: Towards human-like text evaluation. arXiv preprint arXiv:2401.0044 2023.
Kocmi, T. and Federmann, C. Gemba-mam mqq:: Detecting translation quality error spans with gpt-4, arXiv preprint arXiv:2310.13988, 2023a.
Zhang. L. H., Dadkhahi, H., Finkelstein, M., Trabelsi, F,,
Luo, J., and Freitag, M. Learning from others' mistakes: Finetuning machine translation models with span-level er- ror annotations, 2024a. URL pnttps://arxiv..
Kocmi, Т. and Federmann, C. Large language models are state-of-the-art evaluators of translation quality, arXiv preprint arXiv:23002.1 14000 2023b.
abs
Krishna, K., Bransom, E., Kuehl, B., lyyer, M., Dasigi, P., Cohan, A., and Lo, K. Longeval: Guidelines for human evaluation of faithfulness in long-form summarization. arXiv preprint arXiv:2301. 132998 2023.
Zhang, M. J., Wang. Z.,, Hwang, J. D., Dong. Y., Delalleau, O., Choi, Y., Choi, E., Ren, X., and Pyatkin, Diverging preferences: When do annotators disagree and do models know? arXiv preprint arXi::2410.1444 14632, 2024b.
Li, J., Sun, S., Yuan, W., Fan, R.-Z., Zhao, H. and Liu, P. Generative judge for evaluating alignment. arXiv preprint arXiv:2310.05470. 2023.
Zheng, L., Chiang, W.-L., Sheng, Y., Li, T., Zhuang, S., Wu, Z., Zhuang, Y., Li. Z., Lin, Z.. Xing, P., al. Lmsys-chat-Imm A large-scale real-world 11m conversa- tion dataset. arXiv preprint arXiv:2309.1 11998, 2023a.
Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al. Holistic evaluation of language models. arXiv preprint (xXiv:22110091 2022.
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., al. Judging Ilm-as-a-judge with mt-bench and chatbot arena. Ad- vances in Neural Information Processing Systems, 36: 46595-46623, 2023b.
Lommel, A., Uszkoreit, H., and Burchardt, A. Multidimen- sional quality metrics (mqm): A framework for declaring and describing translation quality metrics. Tradumàtica, (12):0455 463, 2014.
Riley, P., Deutsch, D., Foster, G., Ratnakar, V., Dabír- moghaddam, A., and Freitag, M. Finding replicable human evaluations via stable ranking probability. arXiv preprint arXiv:2404.01474 2024.
Tanzer, G., Suzgun, M., Visser, E., Jurafsky. D., and Melas- Kyriazi, L. benchmark for learning to translate a new language from one grammar book. arXiv preprint arXiv:2309. 130575 2023.
Vu, T., Krishna, K., Alzubi, S., Tar, C., Faruqui, M., and Sung, Y.-H. Foundational autoraters: Taming large lan- guage models for better automatic evaluation. arXiv preprint arXiv:2407. 10817, 2024.
Xu, W., Deutsch, D., Finkelstein, M., Juraska, J., Zhang, B., Liu, Z., Wang, W. Y., Li, L., and Freitag, M. LLM- Refine: Pinpointing and refining large language mod- els via fine-grained actionable feedback. In Duh, K., Gomez, H., and Bethard, S. (eds.), Findings of the Asso- ciation for Computational Linguistics: NAACL 2024, pp. 1429-1445, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/w1/2224 findings- naacl.92 URL https:/ /aaaaaaannnooooyoo.o org/2024. Findi mediings-nacci 92.

---

From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to Test Set
A. Implementation Details
Figure shows the AutoMQM prompt template, Table shows an example AutoMQM output, and Figure shows the direct assessment scoring prompt template.
AutoMQM Prompt Template
You are an annotator for the quality of machine translation. Your task is to identify errors and assess the quality of the translation.
Based on the source segment and machine translation surrounded with triple backticks, identify error types in the translation and classify them. The categories of errors are: accuracy (addition, mistranslation, omission, untranslated text), fluency (character encoding, grammar, inconsistency. punctuation, register, spelling), style (awkward), terminology (inappropriate for context, inconsistent use), hon-translation, other, or no-error.
Rach error is classified as one of three severities: critical, major, and minor. Critical errors inhibit comprehension the flow, but what the text is trying to say is still understandable, Minor errors are technically errors, but do not disrupt the flow hinder comprehension.
the text. Major errors disrupt
Make sure your response is strict and valid json object that could be parsed with json. () in python.
ICL examples
(source_language) source: (ssource) ""
(target_language) translation:
""(transiation) '
(errors JSON format)
lsouuae_language) source: "'[source) (target_languager translation: ""(translation] "*. [errors in JSON format:
Test example
[source__anguagee source: " ((source)
translation:
(t(rranslation) '*
Figure 5. AutoMQM prompt, with placeholders for Starget_language (tranaag (translation) (again, for both ICL examples and the example), and ferrors JSON format) (for ICL examples only).
(scurce) (for both ICL examples and the test example),
12

---

From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to Test Set
Source
#5 18.. 22 t
Hypothesis
I'm sorry that we had to drive 200 kilometers from the country to pick up my goods!
Output
satm that", "sevarity": "nines", "catagozy": "stylo/comatoral
swwward"),
("epent;
"severicy"; "minor", "category"; "accoracy/nisternn
{"opants "the country", "ceverity"; "nago=", *category"; "acurracy/iistrrassatttoo
Table Example Specialist AutoMQM output (from the 2MT zh ten test set). As per the AutoMQM prompt (Figure 5 in Appendix A). the output is in JSON format, with fields for enorr epan, sever ity, and cat caorg. The highlighting added to the hypothesis for illustrative purposes, to indicate the locations of the predicted major (dark red) and minor (light red) erros..
Direct Assessment Scoring Prompt Template
You are judge for the quality of machine translation. Based the source segment and machine translation surrounded with triple backticks, your task is to assess the quality of the machine translation on a continuous scale from 0 to 10C. A score of means "No meaning preserved", then the scale goes through "Some meaning preserved", to "Most meaning preserved and few grammar mistakes", up to score of 100, which means "Perfect meaning and grammar".
ICI. examples
[source_language) source: ('(ourcee) [target_language) translation:
"((translation) '
Score: [[[score]]]
[source_language source: "'source) [target_language] translation: ("(trassaation) '*
Score: [[[score]]]
Test example
@rource_lanquagee source: '. (sourcce) (target_language) translation: ""(transllatoon iN
(source} (for both ICL examples and the test example), (target.language), (translation (again, for both ICL examples and the test example), and {score) (for ICL examples only).
Figure Direct Assessment prompt, with placeholders for
13

---

From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to Test Set
B. Supplemental Results
B.1. Additional Figures and Tables
B.I.1. EXPERIMENTAL SETUP
Number of systems
23
24
de
12
17
zh->en
15
N/A
ees
N/A
13
ja ja->zh
N/A
13
Table 8. Number of translation systems for each language pair (WMT'23 and WMT
Avg # ICL examples / Avg errors per test example
en->de
zh-ten
No filtering
11.0/34.8
14.0/30.7
Filtered
10.5/27.4
13.3/25.8
Table 9. Average number of ICI. examples and average number of total errors in ICI. examples, test example (WMT"23 test set). The filtered setting removes all translations from ICL examples which are exact matches to the test translation, and removes all individual errors from ICL examples which exactly match ground-truth error span in the test translation.
Avg # ICL examples / Avg # errors per test example
en-tde
en-tes
ja-
No filtering Filtered
17.0/22.8
13.0/5.8
13.1/13.3
16.0/19.4
12.4/5.6
12.8/12.8
Table 10. Average number of ICL examples and average number of total errors in ICL examples, per test example (WMT'24 test set). The filtered setting removes translations ICL examples which are exact matches the translation, removes individual emror from ICL examples which exactly match ground-truth error span in the test translation.
1.1.2. MAIN RESULTS
en en->de
zh zh-ten
F1
Precision
Recall
F1
Precision
Recall
Baselines Ia) XCOMET-XXL
32.71
28.66
38.10
34.29
39.70
30.18
1b) GEMBA-MQM-QI
29.80
32.04
27.85
34.17
39.87
29.89
Ic) Shuffled sources
31.49
28.34
35.45
37.80*
32.79*
44.62*
Id) Fixed, different source
22.85
26.08
20.35
31.27
34.54
28.58
Specialist
2a) Specialist
45.71
45.04*
46.40*
57.47*
54.05*
61.36*
2b) Specialist Filter
38.32*
39.05"
37.61*
50.72*
49.32*
52.21*
Table 1.. Specialist AutoMQM results on WMT'23 test set. See $4.2 for a description of of Baseline and Specialist systems, and see 1 for description of the filtered setting. Results for the "Shuffled sources" and "Fixed, different source" baselines reported as the average over 10 runs with different random seeds. See Table 12 for the variance across runs. Asterisks (4) indicate scores which are statistically significantly better than XCOMET (row la), according to paired permutation test.
14

---

From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to Test Set
en enddde
zh zh-ten
F1
Precision
Recall
Precision Recall
Shuffled sources
AVG
31.49
28.34
35,45
37.80
32.79
44.62
Shuffled sources
STDEV
0.71
0.62
1.01
0.33
0.33
0.50
Fixed, different source Fixed, different source
AVG
22.85
26.08
20.35
31.27
34.54
28.58
STDEV
0.76
0.93
0.97
0.48
0.57
0.67
Table 12. Average (AVG) character levell FI. precision, and recall over runs of the "Shuffled sources" and "Fixed, different source" baselines with different random seeds. Standard deviation (STDEV) over the 10 runs also reported.
en->de
en-ces
ja- Precision
F1
Precision
Recall
F1
Precision
Recall
F1
Recall
Eavelines
Ia) XCOMET-XXL-QE
24.28
19.63
31.83
10.11
6.02
31.42
14.30
11.80
18.16
1b) Shuffled sources
26.12
19.67
38.84
26.12"
19.67*
38.84
26.44*
32.46"
22.30"
Ic) Fixed, different source
18.23
19.82
16.87
14.89
11.09
22.65
26.77*
32.25*
22.89*
Specialist
2a) Specialist
43.04"
39.16"
47.76
26.58*
20.05"
39.43"
37.16*
38.06"
36.30"
2b) Specialist Filter
32.83*
31.07"
34.79"
25.58*
19.34
37.79*
35.73*
36.83"
34.69*
Table 13. Specialist AutoMQM results on the 24 test Note that en->es ratings were not collected in pseudo-SxS fashion (see $4.3), which explains the smaller performance delta between the Specialist method and the baselines for this language pair. See $4.2 for a description of all of the Baseline and Specialist systems, and see $B.2.1 for description of the filtered setting. Asterisks (+) indicate scores which are statistically significantly better than XCOMET (row la), according to paired permutation test. Note: The en->es MQM data not collected pseudo-SxS fashion, ratings from different raters presented ICL examples in the Specialist setup for this language pair.
B.1.3. ERROR COPYING ABLATIONS
3 ICL Examples
11 ICL Examples
Shuffled
11 ICL Examples
I) Total predicted error count
14,539
14,481
10,298
14,481
2) Disjoint error count
9,185
9,127
7,913
12,096
3) Disjoint error count with exact match example errors
4,142
2,327
6,374
494
Table Pairwise comparison of predicted errors copied from ICL examples, for different AutoMQM systems. The left two columns show comparison of the Specialist AutoMQM (Table row 2a) prompted with 3 11 ICI. examples, and the right two columns show comparison the latter against the shuffled baseline (Table 1, row Ic). Row shows total predicted error count over the full WMT' shows the number of errors predicted by the given system which were predicted by the other system being
en-tde test compared, and row 3 shows the subset of these errors which are exact matches to errors from all 1l (same-source) ICL examples.
en-*de
zh-ten
Fl
Precision
Recall
F1
Precision
Recall
Shuffled sources
31.49
28.34
35.45
37.80
32.79
44.62
Parrot Specialist AutoMQM
27.59
25.62
29.88
36.52
34.92
38.27
45.71
45.04
46.40
57.47
54.05
61.36
Table 15. "Parrot model" vs Specialist AutoMQM performance (WMT'23 test set). The Parrot has access to the same ICL examples as Specialist AutoMQM, and makes predictions as follows: For every error present ICL examples for which there is matching span in the test translation, predict this as an eтоo..
15

---

From Jack of All Trades to Master of One: Specializing LLM-bbsed Autoraters to Test Set
Source
34CMBJ
T
Test Example Hypothesis
34CM mattress not usually thick. so it not necessary to place the bed directly on the mattress.
ICL Examples
A 34cm mattress is not typically thick, you could even sleep directly on the mattress without a bed.
The 34CM mattress not generally thick, and you can sleep directly on the mattress without a bed.
The 34CM mattress is unusually thick, you can sleep directly on the mattress without bed.
34CM mattress is not generally thick, do not sleep directly on the mattress can be.
The 34cm mattress is not usually thick, so you can sleep directly on the mattress without the bed.
34 cm mattress is not as thick as usual. but the beds can be used directly.
The 34CM mattress not so thick, you can just sleep on the mattress without the bed.
Source
Test Example Hypothesis ICL Example Errors
I was scared that I deleted all the games from my favorites.
I was scared that I deleted all the music in my favorites. I was scared that I deleted all the fun in my favorites. I'm afraid deleted all the music from my collection. was in the middle of a conversation.
am so scared to remove all the items in my collection.
Scared me so much deleted all my favorites of its music
Table 16. Examples of where Specialist AutoMQM predicts errors not present in ICI, examples (WMT'23 zh-ten test set). Green highlighting in the Test Example Hypothesis shows where Specialist AutoMQM correctly predicted error span that was present in the ICL examples, while red highlighting indicates span (correctly) copied from ICL examples. Red highlighting in the ICL Examples indicates the error spans that were marked by human MQM annotators (and provided to Specialist AutoMQM as demonstrations).
Source
ICL Example Hypothesis
The title still a customer, but the text becomes a customer.
Test Example Hypothesis
The title still refers to the customer, but in the body of the text, it has changed to client.
Source
i.. Let an old man with 3 wounds get out of bed and open the door to receive the courier and change to pay!
ICL Example Hypothesis
Test Example Hypothesis
To get an old man with wounds on his body to get out of bed and open the door to receive the package, he still has to find change to pay!
Table 17. Examples of where Specialist AutoMQM correctly abstains from copying errors in ICL examples (WMT'23 zh-ten test set). Red highlighting indicates that the span was marked as error by the human MQM annotators, and green highlighting indicates that the span was not marked as an error.
16

---

From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to Test Set
B. 1.4. RATER ABLATIONS
en en-*de
zh-ten
F1
Precision
Recall
F1
Precision
Recall
Human agrecment
la) Round2
34.91
38.16
32. 17
38.68
39.00
38.36
1b) Round3
38.46
40.26
36.82
39.16
40.06
38.29
Specialist
2a) Round1 ICL
45.71
45.04
46.40
57.47
54.05
61.36
2b) Round2 ICI.
30.80
30.87
30.74
38.83
36.65
41.30
2c) Round2 Round3 ICL
30.83
26.65
36.56
38.48
29.91
53.93
Table Specialist MuM performance when prompting and evaluating using different raters (WMT'23 test set). "Round Round3" indicates that ratings from these rounds were merged. Results are reported using the official Round1 test set.
Round1 ICL Examples
Round2 ICL examples
test.set.rater
rcterate
Fl
icl rater-id
F1
num examples
raterl
raterl
0.39
rater8
0.30
672
rater2
rater2
0.50
rater6
0.35
540
rater3
rater3
0,45
rater!
0.33
564
rater4
rater4
0.46
raterl0
0.33
552
rater5
rater5
0.46
rater7
0.26
588
rater6
rater6
0.51
rater4
0.36
540
rater7
rater7
0.46
rater9
0.29
492
rater8
rater8
0.37
rater2
0.27
528
rater9
rater9
0,48
rater3
0.34
528
rater10
rater10
0.51
rater5
0.26
516
Table 19. Specialist AutoMQM performance WMT'23 en->de Roundl, when prompting using Roundl Round2 ICL examples, broken out by rater split. In each round WMT" ratings, there are a total of 10 en- raters. The examples in the test set are then approximately split evenly across all raters (such that all translations of the source segment allocated to the same rater). Note that the variance across raters when using different-rater (Round2) ICL examples is not very high, and using Roundl ICL examples outperforms Round2 ICL examples for every split
B.2. Additional Ablations
B.2.1. FILTERING ICL EXAMPLES TO REMOVE EXACT-MATCH ERRORS
Here, isolate the effect on performance of showing Specialist AutoMQM errors ICL examples which are an exact match ground-truth error in the test translation. In partieular, filter ICL examples to i) remove errors with the same span (but not necessarily the same category or severity) ground-truth errors present in the test translation, and ii) entirely exclude all translations (rather than just removing exact-match errors) in ICL examples which are exact matches to the test translation.
As expected, filtering the ICL examples by removing all error spans present in the ground truth ("Specialist + Filter" setting, row 2b Tables 11 and 13 for WMT'23 and WMT'24, respectively) does incur some degradation in performance relative the "Specialist", but still significantly outperforms all baselines, including the state-of-the -ech--rr-----ttttt XCOMET and "Shuffled source" models. Also note that filtering to remove individual enrors from ICL examples in some sense unfairly disadvantages the model, since this procedure excludes real errors from the demonstrations, and these are, in fact, exactly those errors which would be correct for the model to predict.
For this filtered Specialist AutoMQM, we computed exact match rates with respect to i) ground truth errors spans in the test translations and ii) error spans present in ICL examples. The results are shown in Table 20. Observe that, even though the model was not shown any of the ground truth errors in the provided demonstrations, 17.1 (for en->de) and 23.7% (for
17

---

From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to Test Set
Exact Match Error %
en en->de
zh-ten
1) Ground Truth
17.10
22.70
2) ICL Examples
27.52
26.10
3) ICL Examples
(incl. sub-span super-span)
65.25
68.79
Table 20, Exact match error rate Specialist AutoMQM predictions, as percentage of total predicted errors, with respect to the ground truth spans (row 1) and error spans present in ICL examples (row Row 3 shows the match rate when predicted error spans which are either sub-spans or super-spans of errois present in ICL examples are also counted as matches. Results are presented for "Specialist + Filter" 23 en->de Specialist AutoMQM (Table 13, row 2b), the model is not shown demonstrations of any emror with exact match to the ground truth errors in the test tramslation.
zh zh->en) of the errors that it predicts exact matches to the ground truth, while 26-27% of the errors that it predicts exact matches to ICI. example errors spans. If predicted error spans which are either sub-spans super-spans of errors present in ICL examples are also counted as matches, then the match rate more than doubles, to 65-68%. This suggests that Specialist AutoMQM also taking into account the semantics of the errors in the ICL examples, and is able to generalize its predictions to account for modified versions these errors present the test translations.
B.2.2. CAN WE Do BETTER? AUGMENTING SPECIALIST AUTOMQM WITH MORE ICL EXAMPLES
Specialist AutOMQM only the
ratings from the test set ICL examples, which limits the number of ICL examples to num systems Modern LLMs can handle much longer context than these examples occupy, in this ablation, we investigate whether augmenting the same-source ICL examples provided to Specialist AutoMQM with other ICL examples from the test set can further enhance performance. In particular, for each test set example, we first provide the ICL examples from the shuffled baseline, then concatenate the ICL examples from Specialist AutoMQM. As shown in Table 21, augmenting Specialist AutoMQM with additional examples results small drop in character-level F1. due lower recall (despite small improvement in precision). Recall that in Figure 3, we also saw that Specialized AutoMQM performance saturates at around 10 (same-source) ICL examples. This suggests that there is substantial headroom improve AutoMQM's performance by filling up the LLM's long context window, either with same-soure üifference-soure ICI. examples.
en en-*de
zh->en
F1
Precision
Recall
F1
Precision
Recall
Shuffled sources
31.12
27.93
35.13
37.62
32.45
44.74
Specialist AutoMQM
45.71
45.04
46.40
57.47
54.05
61.36
Shuffled Sources Specialist AutoMQM
44.75
46.17
43.42
57.36
55.79
59.02
Table 21. Comparison of prompting with i) only shuffled sources, ii) only same-source examples, iii) both. Adding additional ICL examples gives higher precision at the cost of lower recall.
18


Reviews:
[{"review_id": "71E3ftubPP", "reviewer": "Reviewer_inJi", "summary": "This paper introduces the \"Specialist\" method, a novel approach to specialize LLM-based Autoraters to specific test sets by leveraging historical ratings through in-context learning (ICL) examples. The method is applied to fine-grained machine translation (MT) evaluation (AutoMQM), achieving significant performance gains—54% and 119% relative F1 improvements on the WMT'23 and WMT'24 test sets respectively, outperforming the state-of-the-art XCOMET metric. The method is robust across different LLM backbones and evaluation tasks, demonstrates sensitivity to rater variability, and suggests potential for broader NLG evaluation applications, though further experimental validation is required.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, thank you very much for your thoughtful and detailed review of our paper. We will address your concerns below.\n>*Q1*: Could you clarify how their method relates to, differs from, or improves upon BatchEval: Towards Human-like Text Evaluation? (same as *Weakness 3*)\n\nBatchEval is only related to our work through our use of in-context learning (ICL), as both approaches involve exposing the model to predictions from either itself (BatchEval) or the ground truth (Specialist) on different examples. BatchEval exposes the model to its own predictions on a batch of unrelated other samples. In this work, we expose the model to ground-truth predictions on a batch of related (same-source) samples. The approaches are not in competition with each other: one could provide multiple same-source historical annotations as ICL examples (our work) while asking the evaluator model to evaluate a batch of different outputs from the same source in one or more stages (BatchEval). We will add this BatchEval reference to the Related Work section in the final version of our paper.\n> *Weakness 1*: The method's reliance on historical annotations might limit applicability in settings without existing annotation resources.\n\nIt is true that access to (an offline collection of) human annotations is inherent to our work, which proposes a novel technique for using human evaluation to improve automatic metrics (rather than using human evaluation as the de facto evaluation method itself) via test set specialization. This work contributes to research on how to use human annotations to enhance the quality of automatic metrics, which we believe is an important and under-explored field, and our hope is that the results in our paper showing the effectiveness of the Specialist method will provide motivation to the broader NLG community to acquire human annotations and build Specialist metrics for other tasks and datasets.\n\n> *Q2*: Could you validate the method across other widely-used NLG evaluation tasks such as Topical-Chat, FED, HANNA, or QAGS?\n(same as *Weakness 1*)\n\nThank you for pointing us to a suitable non-MT NLG evaluation dataset, which has enabled us to improve our paper by validating the generalization of our method. We have added a new experiment showing the effectiveness of the Specialist method on the HANNA benchmark (https://arxiv.org/pdf/2208.11646; one of your recommendations), which evaluates story generation. Some additional details of our experimental setup are below:\n- We evaluated both with and without human stories included. The HANNA paper excluded human stories, as these were considered outliers, while some follow-up papers included them.\n- The HANNA dataset includes 6 criteria: relevance, coherence, empathy, surprise, engagement, and complexity. Due to time constraints, we report results on the first criterion (Relevance) and on the average over all criteria (where the averaged score represents an overall indication of the story's quality).\n- We follow the same experimental methodology as described in Section 4.4 of our paper, and construct Specialist ICL examples per each test example via hold-one-system-out prompting. This simulates the real-world use case of evaluating a new system.\n\nAs shown in the table below (which reports story-level Pearson and Spearman correlations), our Specialist method demonstrated substantial gains on HANNA, outperforming the shuffled baseline across both tasks and all meta-evaluation metrics. On the Relevance task (REL), the Specialist achieved a Pearson of 49.0 (excl. human stories), compared to 43.7 for the shuffled baseline, which already outperformed the winning metric (BARTScore; Pearson=42.6) reported in the [HANNA paper](https://arxiv.org/pdf/2208.11646). The gap in performance between the Specialist and shuffled baseline is even wider when evaluating on the average score, with Specialist Pearson of 46.3 (66.2 incl. human stories), compared to 36.7 (54.3 incl. human stories) for the shuffled baseline.\n|Metric|REL (excl. human)||REL (incl. human)| |AVG (excl. human)| |AVG (incl. human)| |\n|:-----------|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|\n||r_p|r_s|r_p|r_s|r_p|r_s|r_p|r_s|\n|Shuffled baseline|43.71|41.51|59.58|52.88|36.74|32.35|54.32|44.40|\n|Specialist|49.01|46.99|63.76|57.90|46.34|42.36|66.16|54.78|\n\nWe believe this new experiment has strengthened our paper substantially by showing that the Specialist method generalizes to a long-form text generation evaluation task (with very different characteristics than machine translation evaluation).\n\nWe hope the additions we have made to our paper (especially the results on the HANNA benchmark and the comparison with the BatchEval method) allay your concerns and will be taken into account when assigning your final score. Please let us know if you have any further questions or if we can provide any additional clarifications to help finalize your assessment of our paper.", "claims_and_evidence": "The claims of superior performance and robustness are convincingly supported by extensive experimentation and thorough comparisons against strong baselines (XCOMET, GEMBA-MQM, MetricX-24). However, the claim regarding the broader applicability to other NLG evaluation tasks lacks empirical validation beyond MT evaluation.", "methods_and_evaluation_criteria": "The proposed method—constructing ICL examples from historical ratings—is highly suitable for the MT evaluation problem, leveraging existing human annotations effectively. The chosen evaluation criteria (character-level F1 for AutoMQM and Acc23 for scoring tasks) are appropriate and align well with the MT evaluation standards used in the literature. However, additional evaluation across a broader range of NLG tasks (e.g., Topical-Chat, FED, HANNA, QAGS) would strengthen the method's claimed generalization.", "theoretical_claims": "No explicit theoretical proofs or claims are provided or necessary for this empirical study. The focus is empirical validation rather than theoretical contributions.", "experimental_designs_or_analyses": "The experimental design is thorough and sound within the context of MT evaluation. The authors employ strong baselines, rigorous ablations, and analyses to validate the impact of the method, including investigating the effect of ICL example counts, error copying behaviors, robustness across translation systems, and inter-rater variability. Yet, the lack of experimentation on additional NLG tasks limits the overall generalizability claim.", "relation_to_broader_scientific_literature": "The paper builds effectively on the existing literature around automatic MT evaluation, specifically extending the LLM-as-a-Judge paradigm. It directly relates to and advances beyond GEMBA-MQM and XCOMET methods by introducing specialization via ICL examples from historical evaluations. However, it lacks discussion of the relation, similarities, and differences to the closely related BatchEval method (BatchEval: Towards Human-like Text Evaluation), which also employs a comparative evaluation framework.", "essential_references_not_discussed": "The essential reference \"BatchEval: Towards Human-like Text Evaluation,\" which presents a related comparative-based evaluation method, should be discussed to contextualize this paper's contributions further.", "other_strengths_and_weaknesses": "Strengths:\n\n1. The paper is well written.\n\n2. Innovative and simple approach with practical significance.\n\n3. Extensive empirical validation and robustness analyses.\n\n4. Clearly articulated results and experimental methodology.\n\nWeaknesses:\n\n1. The method's reliance on historical annotations might limit applicability in settings without existing annotation resources.\n\n2. Generalization to broader NLG evaluation tasks remains experimentally unverified.\n\n3. Absence of a detailed comparative discussion with the BatchEval method, which uses similar comparative-based ideas.", "other_comments_or_suggestions": "None.", "questions_for_authors": "Questions For Authors*\n\nQ1: Could you clarify how their method relates to, differs from, or improves upon BatchEval: Towards Human-like Text Evaluation?\n\nQ2: Could you validate the method across other widely-used NLG evaluation tasks such as Topical-Chat, FED, HANNA, or QAGS?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7Ii8V9NGp5", "reviewer": "Reviewer_tvUC", "summary": "This paper introduces an LLM-based automatic evaluation method, called \"Specialist\". At a high-level, Specialist closely imitates a human rater's behavior by making ICL examples from the ratings (1) of the same rater (2) on the same example (3) on different model outputs. In other words, the only place where extrapolation from past data points happen is from annotations on past model outputs to new model outputs. Results show that when measured by character-level F1 of the annotated error spans, Specialist outputs the state-of-the-art machine translation metric XCOMEt by 54% and 119% respectively on WMT 23 and WMT 24 test sets.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, thank you very much for your positive and thorough review of our paper. We will address your outstanding concerns below, starting with the question which you stated as most important. \n> Can the authors explain to me what they are trying to achieve in 5.4.1 and what the results mean?\n\nSections 5.4.1 and 5.4.2 both address the question of whether the Specialist model is simply copying errors in the ICL examples, but from two different angles, to provide further evidence for the claim that the success of the Specialist cannot be attributed to naive copying behavior. In Section 5.4.1, we compare copying behavior of prompted Autoraters using Specialist vs non-Specialist (and fewer Specialist) ICL examples, while in Section 5.4.2, we compare the Specialist prompted Autorater with a non-model-based baseline, which makes predictions by directly copying ICL example errors. While the gap in precision between the Specialist Autorater and the Parrot baseline in Section 5.4.2 shows that the Specialist metric is not copying all of the errors that it could from the ICL examples, this alone does not establish that the Specialist ICL examples are actually *teaching the model* which errors to *abstain from predicting*. This latter claim is substantiated in Section 5.4.1, where we contextualize the Specialist's copying behavior with respect to that of other prompted Autoraters, and show that providing *more* Specialist ICL examples results in a *decrease* in copying behavior with respect to these examples. We hope this has clarified the difference between Sections 5.4.1 and 5.4.2, but if anything remains unclear, please let us know.\n\n> Section 5.6: Why did you use an alternative prompt, instead of directly converting the annotated MQM errors into MQM scores?\n\nIn Table 5, we are actually directly comparing these two settings. That is, to compute Acc23 for the *AutoMQM Specialist*, we are indeed converting the predicted MQM errors into scores. As shown in the table, this outperforms the *Score Prediction Specialist*, which directly prompts the LLM to generate a float quality score on a scale from 0-100 (aligning with standard direct assessment prompting techniques traditionally used for LLM-as-a-Judge evaluation).\n\nAs for the meta-evaluation metrics used, and in particular your concern regarding leniency to error classification and severity, note that the character-level F1 does indeed penalize incorrect error severities. See Section 4.5 of the paper: \"Partial credit of 0.5 is given if the predicted rating correctly marks a character as an error but predicts the incorrect severity.\" To address your concern about leniency towards error classification, we report the \"binary error classification F1\" meta-evaluation metric below, which computes the F1 of the rating-level binary error vs no-error decision. This complements Table 1 in the paper, and we see that the Specialist metric outperforms all others according to this evaluation too.\n|Binary error class. F1|WMT'23|WMT'24|\n|:----------------------|:-----:|:-----:|\n|XCOMET|83.35|52.36|\n|GEMBA|73.43|-|\n|Shuffled sources|80.25|49.82|\n|Fixed, different source|70.36|49.92|\n|Specialist|89.74|65.23|\n\n> [...] this is only useful when one has existing human evaluation on a dataset and one is using \"Specialist\" to obtain accurate, explainable evaluations for future systems developed on this test set. \n\nIt is true that the Specialist method specializes to a fixed test set by design. By removing the constraint that the Autorater generalize to new test sets, we achieve large gains in the metric's ability to generalize to new systems, and this is often a tradeoff that is worth making. Many benchmarks (e.g., WMT for machine translation, XLSum for Summarization, SQuAD for question answering) are carefully curated and repeatedly used to allow for fair comparison against previous work (including during the model development process). Moreover, the Specialist method provides a paradigm for joint development of *automatic metrics* and *test sets* (which are typically developed independently, but don't need to be, and we get quality gains if they're developed jointly). Thus, when seeking to evaluate on \"an arbitrary new dataset where no prior human evaluation is available\", the metric development process which accompanies development of this new test set would involve collection of a small set of human annotations of system outputs on this test set.\n\n> [...] resemblance to the spirit of \"test-time training\" (https://arxiv.org/pdf/1909.13231)\n\nWe agree that Test-Time Training has some similarity in spirit with our work, in that we both seek to specialize a model to a particular test set example using data related to that example. While they use related examples from a simpler auxiliary task with automatic labels (image rotation), we use related examples from the same task with ground-truth labels. Additionally, our method does not involve parameter updates.\n\nThank you!", "claims_and_evidence": "All claims made in the submission is clearly and convincingly supported by their experimental results.", "methods_and_evaluation_criteria": "The method makes sense to me, but the authors could make it clearer how this is useful in real-world applications. From my perspective, this is only useful when one has existing human evaluation on a dataset and one is using \"Specialist\" to obtain accurate, explainable evaluations for future systems developed on this test set. It can't be (1) used on an arbitrary new dataset where no prior human evaluation is available (2) used for alternative QE metric use cases such as data filtering.\n\nThe evaluation criteria makes sense, but is constrained to finer granularities such as span-level and segment-level. I have two small doubts, mostly on Section 5.6:\n\n- Why did you use an alternative prompt, instead of directly converting the annotated MQM errors into MQM scores?\n- Current evaluation is only constrained to segment-level. I'd like to see how this stacks up to other WMT metrics on system-level on the acc23 dataset. (Specifically, where it stands in Table 1 in the [WMT 24 metrics shared task paper](https://aclanthology.org/2024.wmt-1.2.pdf).)", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "I carefully checked through all the details in Section 4 and haven't spot any issues.", "relation_to_broader_scientific_literature": "The proposed Specialist method is a nice addition to the existing LLM-based machine translation evaluation methods. While there are both methods that directly predict scores (GEMBA) as well as ones that predicts MQM annotations (AutoMQM), the contribution of this paper is orthogonal to existing work and concerns about the usage of ICL examples. It can serve as a nice addition to both of these methods in the intended applications.", "essential_references_not_discussed": "Not that I'm aware of.", "other_strengths_and_weaknesses": "In general, this is a nicely-written paper with good novelty and well applicability to machine translation development, with potentials to extend to evaluations of other generative tasks. The only reservation I have is I'm having trouble understanding Section 5.4.1. The logic in the section is hard to follow, and it looks a little duplicative to 5.4.2 in that both mentioned \"if the model is simple copying errors in the ICL examples\". Can the authors explain to me what they are trying to achieve in 5.4.1 and what the results mean?", "other_comments_or_suggestions": "- I think this paper shares some resemblance to the spirit of \"test-time training\" (https://arxiv.org/pdf/1909.13231). The authors may consider drawing an analogy to justify their usage of existing information in the test examples.\n- In general, the evaluation is somewhat skewed to error span annotation, while being rather lenient to error classification and severity -- both are actually critical when converting MQM annotations to scores. I wonder why this skew exists. (My comment in \"Methods And Evaluation Criteria\" is also along that line.)", "questions_for_authors": "The most crucial question is regarding question 5.4.1 (see \"Other Strengths And Weaknesses\"). Please make sure you address that one.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JXK92QwcKv", "reviewer": "Reviewer_iJij", "summary": "This paper presents a novel approach to enhancing automatic evaluation metrics based on LLMs, focusing on Machine Translation (MT) evaluation. The authors propose a Specialist method that leverages historical human-generated ratings on a fixed test set to construct ICL demonstrations. This specialization allows an LLM-based Autorater, termed \"Specialist AutoMQM,\" to outperform existing state-of-the-art metrics on fine-grained MT evaluation tasks.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, thank you very much for your feedback, which has helped us to improve our paper. In response to your primary concern, namely that the Specialist method's \"effectiveness on tasks with fundamentally different characteristics (e.g., open-ended generation, [...]) remains uncertain\", we have added a new experiment showing the effectiveness of the Specialist method on the HANNA benchmark (https://arxiv.org/pdf/2208.11646), which evaluates story generation. For additional details of our experimental setup and results, please see our response to Reviewer inJi. As shown in the table below (which reports story-level Pearson and Spearman correlations), our Specialist method demonstrated substantial gains on HANNA, outperforming the shuffled baseline across both tasks and all meta-evaluation metrics. \n|Metric|REL (excl. human)||REL (incl. human)| |AVG (excl. human)| |AVG (incl. human)| |\n|:-----------|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|\n||r_p|r_s|r_p|r_s|r_p|r_s|r_p|r_s|\n|Shuffled baseline|43.71|41.51|59.58|52.88|36.74|32.35|54.32|44.40|\n|Specialist|49.01|46.99|63.76|57.90|46.34|42.36|66.16|54.78|\n\nWe also appreciate that, despite your concerns, you appreciate the soundness of our experimental design and the rigorous ablations which \"ensure thorough validation of the effectiveness of same-source ICL specialization\". We would also like to address your concerns about our method's novelty and practicality:\n- Novelty: Your review states that \"adjusting ICL demonstrations to enhance or analyze the performance of specific tasks has already been extensively studied\". This is a broad field of study in its own right, and no supporting citations were provided. The \"Essential References Not Discussed\" section is also empty. If you believe essential references are missing, we would like to include them, but cannot act on this incomplete comment. Moreover, our contribution is not limited to showing that our proposed method for constructing ICL examples enhances performance of LLM-as-a-Judge metrics. More broadly, our work proposes a novel technique for using human evaluation to improve automatic metrics (rather than using human evaluation as the de facto evaluation method itself) via test set specialization, and includes an extensive study of how rater behavior affects performance of LLM-as-a-Judge metrics prompted with human ratings. Research on how to use human annotations to enhance the quality of automatic metrics is an important research area in its own right. To the best of our knowledge, our Specialist method has never been proposed, and yields enormous gains over the existing SOTA in MT evaluation (54% and 119%, respectively, on WMT'23 and WMT'24 test sets). While we acknowledge the simplicity of the method, its dramatic effectiveness is evident from the results presented in the paper.\n- Generalization to real-world use cases: We would like to address your concern that the WMT benchmark, despite being \"standard and widely used\", may not \"adequately reflect real-world translation scenarios.\" The submissions to WMT (submitted by researchers from institutions across both academia and industry) include a diverse collection of systems of varying quality, including both LLM-based and non-LLM-based (e.g., small encoder-decoder) systems. As shown in Figure 2 in our paper (and discussed in Section 5.2), the Specialist metric's outperformance is consistent for translation systems across the quality spectrum, and cannot be explained by gains only for a certain quality tier. These results imply that the Specialist metric has *better* generalization to new systems, when removing the constraint that it must also generalize to new test sets.\n- Cost: The review states that \"evaluation latency, cost implications, and practicality [...] have not been discussed.\" Please note that *these considerations are in fact discussed in Section 3 of the paper, in the paragraph entitled \"Specialist Method in Practice\"*. The review also states that \"large-scale deployment of prompted LLM-based metrics can be computationally costly\". Once deployed, our proposed Specialist metric does not have any additional computational overhead relative to standard LLM-based metrics. Moreover, large-scale deployment of LLM-based metrics has been widely adopted and, for many real-world applications, is more practical, efficient, and cost-effective than running human evaluations. In fact, \"rapid model iteration\" is virtually impossible without relying on automatic metrics.\n\nWe hope the additions we have made to our paper (especially the new results on the HANNA benchmark for story generation evaluation) allay your concerns about our method's generalizability to other NLG evaluation tasks, and that you will consider increasing our score. Please let us know if you have any further questions or if we can provide any additional clarifications to help finalize your assessment of our paper.", "claims_and_evidence": "The authors' main claim includes a framework for effectively conducting automatic evaluations of NLG tasks on a fixed set that remains largely unchanged. However, it seems this work includes several limitations.\n\nThe authors broadly claim the generalizability of the Specialist method across various NLG tasks. However, explicit experiments were limited to two closely related MT tasks (AutoMQM and Direct Assessment scoring). The method’s effectiveness on tasks with fundamentally different characteristics (e.g., open-ended generation, dialogue evaluation, summarization quality assessment) remains uncertain. Without explicit validation, these claims are somewhat overstated.", "methods_and_evaluation_criteria": "The proposed Specialist method relies heavily on the availability of high-quality historical ratings from human annotators for each test set. This requirement is a major practical limitation in contexts where obtaining such annotations might be prohibitively expensive, labor-intensive, or challenging (e.g., low-resource languages or less common test sets).\n\nIn particular, the proposed method lacks any significant technical novelty. The approach of carefully adjusting ICL demonstrations to enhance or analyze the performance of specific tasks has already been extensively studied.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental design is sound, involving rigorous ablations (e.g., varying ICL examples, \"Shuffled sources,\" and \"Fixed, different source\" baselines), ensuring thorough validation of the effectiveness of same-source ICL specialization.\n\nHowever, all experiments rely on WMT test sets, which, despite being standard and widely used, may not adequately reflect real-world translation scenarios or fully capture the variability found in practical applications. The performance improvements reported might therefore not fully generalize outside WMT benchmarks.", "relation_to_broader_scientific_literature": "I could not find the any broader impact for this paper.", "essential_references_not_discussed": "I think", "other_strengths_and_weaknesses": "The Specialist method depends on prompting large models like Gemini 1.5 Pro, GPT-4o, or Claude. In practice, large-scale deployment of prompted LLM-based metrics can be computationally costly. Moreover, evaluation latency, cost implications, and practicality for real-time evaluations during rapid model iterations have not been discussed.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hg93YoE5hZ", "reviewer": "Reviewer_19zf", "summary": "While traditional LLM evaluators (Autoraters) are generally trained for generalization to a broad set of tasks, this paper studies specializing existing models into autoraters for particular known test sets. In particular, the proposed new prompting strategy leverages in-context learning examples obtained from historical ratings of each individual test set example. Empirically, the authors focus on the machine translation task, and show substantial improvements meta-evaluating their metric against past considered autorater methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, thank you very much for your careful and comprehensive review of our paper. We appreciate that you recognize our sound experimental setup and clear presentation, as well as convincing results and parameter studies. We would like to address your concerns about the practicality of the Specialist method.\n> I find the practical value of the proposed method is hindered by its inherent requirements and costs of having full access to the evaluation set of examples.\n\nIt is true that the Specialist method specializes to a fixed test set by design. By removing the constraint that the metric generalize to new test sets, we achieve large gains in the metric's ability to generalize to new systems, and this is often a tradeoff that is worth making. It is common to hillclimb for many years on a fixed test set (e.g., WMT for machine translation, XLSum for Summarization, SQuAD for question answering). Eventually these benchmarks saturate and must be refreshed, but benchmarks for core NLG tasks are often carefully curated to measure certain capabilities of interest and are repeatedly used to allow for fair comparison against previous work (including during the model development process). Thus, the Specialist method provides a paradigm for joint development of Automatic Metrics and Test Sets (which are typically developed independently, but don't need to be, and we get quality gains if they're developed jointly).\n\n> I think this would make it inappropriate to rate specialized models and applications where successful translations can be hard to obtain or costly (e.g., underrepresented languages, long passages).\n\nWe interpret this concern to relate to the cost of collecting outputs from multiple models on the given test set, for the sake of then collecting human annotations to construct Specialist ICL examples, but please correct if this is a misinterpretation. If \"successful translations\" are hard to obtain, this would apply equally to the collection of outputs for the creation of ICL examples as it would at test time. Thus, this begs the question of whether this task should be used for evaluation in the first place, or would be a bottleneck at every evaluation.\n\nThe above point aside, it is true that for some NLG tasks, there are not that many different models/systems which can perform the given task (and from which to collect ICL examples for the Specialist Autorater). However, as we show in the ICL example scaling experiments in Section 5.3, only 3 ICL examples (i.e., ratings of outputs from 3 other systems) are needed to outperform the state-of-the-art. For many NLG tasks, including those tracked most closely on LLM leaderboards, we do indeed have easy access to outputs from multiple systems. \n\nMoreover, as discussed in the \"Specialist Algorithm\" paragraph in Section 3, while \"the different translations for each input example come from different translation systems [in this work], they could in principle also be sampled from a single model (e.g., using a diversity promoting sampling algorithm).\" This is a promising direction for future work, especially for tasks which are not well represented in available models.\n\n> The scalability is also a potential concern, as with more comprehensive and larger test sets, there will be linearly scaling costs.\n\nThis work explores how to use human annotations to enhance the quality of automatic metrics and, while the cost of collecting Specialist annotations for a test set does scale linearly with the size, evaluation using automatic metrics is much cheaper than directly depending on human annotations for evaluation, without sacrificing quality (as shown in the comparison against inter-annotator agreement in Section 5.5).\n\nMoreover, with larger test sets, the imperative to depend on automatic metrics (rather than human evaluation) increases too. One-off collection of human ratings on a large(r) test set to build an automatic metric is much more tractable than repeated human evaluations on this test set. Our hope is that the results in our paper showing the effectiveness of the Specialist method will provide motivation to the broader NLG community to acquire human annotations and build Specialist metrics for other tasks and datasets.\n\nWe hope our responses and clarifications have allayed your concerns about our method's practicality, and that you will consider increasing our score. Please let us know if you have any further questions or if we can provide any additional clarifications to help finalize your assessment of our paper.", "claims_and_evidence": "The claims are supported by solid empirical evidence (see Experiments Section below).", "methods_and_evaluation_criteria": "The proposed improvement is simple and very intuitive.\n\nI find the practical value of the proposed method is hindered by its inherent requirements and costs of having full access to the evaluation set of examples. I think this would make it inappropriate to rate specialized models and applications where successful translations can be hard to obtain or costly (e.g., underrepresented languages, long passages).\n\nThe scalability is also a potential concern, as with more comprehensive and larger test sets, there will be linearly scaling costs.", "theoretical_claims": "The paper does not make theoretical claims.", "experimental_designs_or_analyses": "The experimental breadth is sound. The authors apply their method prompting a powerful Gemini LLM as the autorater. They evaluate their method against relevant recent baselines across 5 different target/source language combinations.\n\nOn top of convincing results, the authors also provide interesting parameter studies to highlight the sensitivities of their method (e.g., performance as a function of the base LLM autorater/number of in-context samples) and evaluate potential failure modes (e.g, compare performance vs. 'Parrot' baseline also copying the in context example's errors).", "relation_to_broader_scientific_literature": "The work is a direct extension of prior prompt-based LLM autoraters systems for machine translation. The main difference with prior work such as [1, 2] is the use of in-context examples that are specific to each individual sample in the test set.\n\n[1] Fernandes, Patrick, et al. \"The devil is in the errors: Leveraging large language models for fine-grained machine translation evaluation.\" arXiv preprint arXiv:2308.07286 (2023)\n\n[2] Kocmi, Tom, and Christian Federmann. \"GEMBA-MQM: Detecting translation quality error spans with GPT-4.\" arXiv preprint arXiv:2310.13988 (2023).", "essential_references_not_discussed": "I believe the paper discusses most of the relevant prior work.", "other_strengths_and_weaknesses": "The paper is clear and well-written.\n\nThe novelty of the methodology is relatively limited due to the incremental nature of the work.", "other_comments_or_suggestions": "No additional comments.", "questions_for_authors": "No additional questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This is an empirical paper with a clear contribution to the growing literature on LLM-based evaluation metrics. While the technique is relatively simple, its demonstrated effectiveness on key benchmarks  justifies its contribution. 

There are some concerns raised around generalizability and practicality of the method.
The majority of reviewers, however, found the work valuable and sound, and recommend acceptance.
Decision: Accept (poster)