Do LLMs Understand Your Translations? Evaluating Paragraph-level MT with Question Answering
Authors: Patrick Fernandes
Sweta Agrawal
Emmanouil Zaranis
Andre Martins
Graham Neubig
Venue: COLM 2025 (2025)
Keywords: llm-based metric
machine translation
evaluation
question generation
question answering
TL;DR: An extrinsic approach to evaluate long translations, using LLMs to generate and answer reading comprehension questions.

Abstract: Despite the steady progress in machine translation evaluation, existing automatic metrics struggle to capture how well meaning is preserved beyond sentence boundaries. We posit that reliance on a single intrinsic quality score, trained to mimic human judgments, might be insufficient for evaluating translations of long, complex passages, and a more “pragmatic” approach that assesses how accurately key information is conveyed by a translation in context is needed. We introduce TREQA (Translation Evaluation via Question-Answering), a framework that extrinsically evaluates translation quality by assessing how accurately candidate translations answer reading comprehension questions that target key information in the original source or reference texts. In challenging domains that require long-range understanding, such as literary texts, we show that TREQA is competitive with and, in some cases, outperforms state-of-the-art neural and LLM-based metrics in ranking alternative paragraph-level translations, despite never being explicitly optimized to correlate with human judgments. Furthermore, the generated questions and answers offer interpretability: empirical analysis shows that they effectively target translation errors identified by experts in evaluated datasets.

Published as a conference paper at COLM 2025
Do LLMs Understand Your Translations? Evaluating Paragraph-level MT with Question Answering
Patrick Fernandes Sweta Agrawal
Emmanouil Zaranis
André F.T. Martins
Graham Neubig
V Carnegie Mellon University
Instituto de Telecomunicações
Instituto Superior Técnico, Universidade de Lisboa
UUnabell
pfernand@cs cww.ddddssccceeeecomomommmo cmu.edu
Abstract
Despite the steady progress in machine translation evaluation, existing automatic metrics struggle to capture how well meaning is preserved beyond sentence boundaries. We posit that reliance to mimic human judgments, might be insufficient for evaluating translations of long, complex passages, and a more "pragmatic" approach that assesses how accurately key information is conveyed by translation in context is needed. We introduce TREQA (Translation Evaluation via Question-Ana Answering), framework that extrinsically evaluates translation quality by assessing how accurately candidate translations answer reading comprehension questions that target key information in the original source reference texts. In challenging domains that require long- range understanding, such as literary texts, we show that TREQA is competitive with and, in some cases, outperforms state-of-the-at neural and LLM-based metrics in ranking alternative paragraph-leyy translations, despite never being explicitly optimized correlate with human judgments. Furthermore, the generated questions and answers offer interpretability: empirical analysis shows that they effectively target translation errors identified by experts in evaluated datasets. Our code is available at https//gituu.ccmdeeepspppfffi
single intrinsic quality score, trained
Introduction
Machine translation (MT) has progressed significantly recent years, with top systems achieving translation quality comparable to human-generated references at the sentence level for many high- resource languages (Koemi et al., 2024). A large part of this progress is driven by equally striking improvements in automatic evaluation metrics, where sentence-level neural metrics such as COMET (Rei et al., 2022a) and MetricX (Juraska et al., 2023) play a crucial role in assessing and improving MT models (Peter et al., 2023; Agrawal et al., 2024).
Despite these advances, progress in MT and automatic evaluation beyond the sentence level has been slow and limited. Challenges persist both in developing systems that translate full documents coherently and consistently and in designing evaluation metrics that can appropriately measure the accuracy of translated cross-sentence phenomena. While various factors contribute to this slow progress, including the field's difficulty in moving away from sentence-by-senteme & Junezys-Dowmunt, 2023), we posit that fundamental issue lies in the reliance on intrinsic quality scores that, while useful for assessing individual sentences, fail to capture the complex, multi-facete nature of document- that predict these scores are typically trained on large amount of human quality judgments at the sentence laceulaacumlltted over the years of WMT evaluations (Koehn Monz, 2006; Koemi al., 2024). However, such large-scale manual data collection becomes impractical at the document level, limiting the applicability of intrinsic- based measures for broader translation tasks.
translation (Post
translation. Furthermore, existing metrics like BLEURT and COMET
Equal contribution.

---

Published as a conference paper at COLM 2025
Reference
Questions
Question
Answer Correctness
Generation
Candidate(s)
Question
Answerting
Cancidate
Ground-truth
Answers
Answera
Source
Reference
Figure I: Approach overview: TREQA assesses translation quality through reading comprehension question-answeriig framework. First, a set of questions Q is generated either in candidate-aware (indicated by the green arrow) or a candidate-agnostic manner. Next, the QA system produces ground-truth and candidate answers. Finally, the Answer Correctness system evaluates and ranks candidate passages by comparing the candidate answers with the ground-truth answers obtained in the previous step. Access to reference passages is indicated by the yellow arrows.
Evaluating translations beyond individual sentences requires taking considers how well a translation serves its intended purpose- L.e., how well it enables readers to use the information effectively readers of the source text. For example, readers of translated literary texts should get the same core information as if they were native readers of the original text. This type of extrinsic evaluation- quality of the translation is tested through reading comprehension questions rather than merely predicting a single score-has been explored in prior works due its potential better measure the real- world utility of translations (Tomita al., 1993; Fuji et al., 2001; Berkaab et al., 2011; Scarton & Specia, 2016; Forcada et al., 2018; Krubiński et al., 2021). However, most of these approaches were dropped in favor of the simpler judgment-basee evaluation due to poor scalability since substantial human effort is required to generate questions and verify answers. Furthermore, questions were generated uninformed by the hypotheses being evaluated, making it necessary to exhaustively cover all key concepts to capture all errors.
pragmatic approach that
We argue that, in the age of LLMs, the time is ripe to reconsider the usefulness of extrinsic evaluation for MT. In fact, LLMs can automate much of the human-intensive effort in question generation and answer verification steps, which has hampered previous approaches, as they have been shown to generate good questions test comprehension in other natural language generation tasks (Eyal et al., 2019; Deutsch & Roth, 2021; Deutsch et al., 2021; Nan et al.. 2021; Fabbri et al., 2022). In automatic metric, TREQA (Translation Evaluation
particular, we propose question-answwrrngg via Question Answering), which uses L.MSs to generate questions about content in the source or reference texts targeting key concepts, phrases, and entities. To narrow down the space of possible questions and improve their effectiveness in localizing errors, we optionally condition the question generation on the set of hypotheses being evaluated. This allows the model generate more targeted questions based on key differences between the hypotheses and the source (or the reference). We then assess translation quality by measuring how well answers extracted from the candidate translations align with those from the original source texts or references. Importantly, and unlike most state- of-the-art metrics, TREQA fully unsupervised: does not require any training on human quality assessments dataset.
We evaluate TREQA on literary texts - an especially challenging and underexplored domain for MT evaluation as these texts demand deep semantic understanding. multi-sentence reasoning, and subtle discourse-enforcer meaning nuances that traditional metrics would fail to capture. We show that TREQA is competitive with ranking alternative paragraph-level translations literary texts translated into English from multiple languages, including Polish, Japanese, Russian, French, German, and Chinese, both in the reference-
outperforms state-of-the-an neural and LLM-based metrics in
2

---

Published as a conference paper at COLM 2025
based and reference- free (or quality-estinatiin:: QE) setups. TREQA performs particularly better when question generation is conditioned on evaluated candidates - a setting that benefits from using more capable LLMs. While literature is open to interpretation, many questions generated by TREQA focus on elements such as character relationships, event sequences, and causal links, which have stable, answerable properties across translations. Furthermore, we demonstrate that the questions generated by TREQA precisely target expert-marked error spans, and generating these specific errors (rather than paraphrasing them) is critical for accurately capturing translation quality differences.
Nevertheless, despite their promise, the use of LLMs still pose challenges: we find that LLM- evaluation can result in biased indicators of quality, such as position biases where generated answers favor candidates based on their order. We further find that variations in the generated question set can occur even with greedy decoding due to minor changes in low-level CUDA operations, causing evaluation instability. Interestingly, candidate- by specifically targeting error spans and thus constraining the generation space, making evaluations more robust even when sampling. Finally, we discuss failure cases of TREQA, opening avenues for future research directions for improving controlled cross- rross-lingual question generation and question answering (QA) and designing metrics for assessing answer correctness.
generation significantly reduces this variance
2 Translation Evaluation via Question Answering (TREQA)
Formalization We aim evaluate the quality of candidate translation ÿ. given a source text and, optionally, a reference translation y. Drawing inspiration from prior research on using automatic QA-based metrics for evaluating MT (and other text generation tasks; Li al., 2024), we hypothesize that, given relevant comprehension questions about the information conveyed in the source text x, high-quality translation should allow a QA model answer them perfectly." Any deviations or errors in y (such as omissions, mistranslations, or distortions of meaning) can result in lower QA scores, either because the necessary information is missing or because the mistranslation introduces a mismatch in semantic content between x and y.
To quantify the quality of ÿ. we compare the answers derived from á (via a QA model) with the reference answers directly obtained from x or a high-quality reference translation, y. Formally, given a distribution of possible questions TG, that can probe the semantic content a ground-truth text r (source text x and/or reference translation y). M(grr) is the quality score of 9 such that:
1
M (g.r) = Equa, Ip (QQA().9). ~
Σ ((AA((B.)).AAI QEQ
Q
where, is the set of questions sampled from Ty using an automatic question-answa generation (QAG) model; QA(ğ.9) 9) is then the QA model's answer to the question q using the candidate translation G: and p(ā, a) is a comparison function that measures the similarity between the QA model's answer, å (derived from ÿ) and the reference answer, a (derived from y or x). Together, these comparisons serve as a proxy for measuring how well the translation preserves the source text's s meaning.
However, we need to make several design decisions to implement this metric effectively: the choice, granularity, and coverage of questions generated by the QÁG model matter substantially, irrelevant or quor-quality questions may fail to probe critical aspects of the semantic fidelity between ÿ and x or y. Relatedly, the use of capable QA models that understand the content of the text and the questions, as well as the choice of comparison functions that can accurately measure the semantic similarity between the answers are crucial. We next discuss these components in detail.
2.1 Question-Answer Generation
The quality of the evaluation depends heavily on the questions used to probe semantic accuracy. We hypothesize that good questions should idcally be discriminativ--alee to distinguish between good and bad translations should focus concepts salient to the context, where meaningful divergences between translations are likely to occur, while disregarding superficial (easy) details.
TTii is assuming access to "perfect" QA which simplifies the theoretical framework, however, in practice, QA models are imperfect and pose many limitations (Kamoi al., 2023).
3

---

Published as a conference paper at COLM 2025
LLMs are particularly well-suited tools for generating such targeted questions. Their broad knowledge and reasoning capabilities allow them to identify important information and formulate probing questions that focus key semantic elements. And, prior work has demonstrated LLMs' effectiveness in generating discriminative questions for monolingual text generation tasks like simplification (Trienes et al., 2024) and summarization (Kim et al., 2024b).
When prompting LLMs to generate question-answa choice of text to condition generation on plays a crucial role in shaping the types of questions produced. Importantly, we always generate both questions and answers, as opposed to just questions, to ensure that the questions are anchored in the provided text(s) and target verifiable information (Ushio et al., 2023). We consider two different settings:
pairs, which are then parsed into a set Q, the
In this setup, the LLM generates question-ansee annswerss based on the source text x (making the system a quality estimation metric), the reference translation y, or both. The questions are generated in "isolation" " without any information on the candidates being evaluated. This approach allows questions that target salient and contextually important concepts without being influenced by any particular translation and its potential errors, but might suffer from high variance (as the set of questions needed to evaluate all possible translation errors can be intractably large).
I. Candidate-agnostic QAG:
II. Candidate-aware QAG:
We assume prior knowledge of the candidate translations to be evaluated. Using this information, generate questions targeting potential errors grounded in the candidate translations and reference texts, r (source and/or reference). To accomplish this, we prompt the LLM with the complete set of candidate translations.
Figure 2a shows the prompts used for each approach when both reference and source texts are provided inputs (QE prompts are shown in Appendix 7). Although research on multilingual or cross-lingual capability of LLMs for question generation (when the source or reference is not in English) has been limited, some studies have suggested that fine-tuned QAG systems can generalize to unseen languages (Kumar et al., 2019). In this work, however, we constrain the generation of question-answe pairs English. This choice is motivated by the fact that the capabilities of LLMs in English, nevertheless, are far superior to those in other languages, as evidenced by prior work (Etxaniz et al., 2023). Nevertheless, our setup still probes the gross-lingual understanding abilities of LLMs, given that the source or reference texts may be in other languages.
2.2 Question Answering
Given questions Q, we face the challenge of selecting a QA model capable of answering them. Similar to question generation, LLMS also well-suited for this task, with prior work demonstrating their effectiveness in QA-based evaluation (Fabbri et al., 2022; Xiao et al., 2023; Agrawal & Carpuat, 2024; Säuberli & Clematide, 2024; Trienes et al., 2024). Furthermore, while QG more open-ended and requires generating diverse and meaningful questions, QA is a more constrained task that involves extracting relevant information from the text. Hence, while larger, more capable models may be necessary for high-quality QG, smaller, more efficient models could be sufficient for QA, provided they have strong reading comprehension abilities.? We focus on extractive QA, as this ensures that only information explicitly present in the text is considered rather than allowing generative QA model to hallucinate plausible but incorrect answers (Rajpurkar et al., 2016; Mallick et al., 2023).
Note that the questions are always generated in English; however, the context could be English or non-English. To support both settings, we design two prompts: a) monolingual, ask the LLM to extract the answer from the context as-is and b) crosslingual, generate an answer given the cross-lingen lross-lingual context and English questions. Figure 2b shows our prompt(s) for answering questions using the candidate translation y.
Answer (Re) (Ree-Gennrationn While we prompt the LLM to generate both questions and answers, these answers are not used directly. Instead, they help scaffold high-quality, answerable questions (Ushio et al., 2023). These initial answers may differ stylistically or semantically from those produced by the QA model, even with access to the same text. Moreover, the QAG system can introduce bias by extracting answers from the candidate texts rather than the ground-truth reference (as confirmed by our analysis in Section 5). To address this, we re- re-generate answers using the QA model by providing
We present ablation analysis for using smaller models for QA in Appendix D.3.
4

---

Published as a conference paper at COLM 2025
Prompt for Question Generation (with optional candidates)
QA Prompts for Monolingual and Crosshingual Contexts
System: You are expert teacher tasked with designing reading comprehension questions.
System: You are helpful assistant skilled extractive question answering.
User: Generate question-answer pairs verify translation accuracy. Each answer should be key phrase, concept, entity from the original passage (source or reference) that is important in context and could help detect potential errors or mistranslations in the alternatives.
Monolingual setring
User: Given the following passage and question, extract
the exact answer from the passage, The answer should be short span found verbatim in the passage.
The questions answers must preserving meaning. Questions should be diverse and cover different aspects of the passage.
English while
Crosslingual sotting
User: Given the following passage and question, the answer in English using only the information from the passage. The answer should concise and faithful the original content.
Q: <question1>
A: <answerl>
Q: <question2>
A: <answer2>>
Passage: {passage)
Source Passage: (arc passage)
Reference Passage: (ref passage)
#
Question:
{question}
Alternative Passages: {alternatives]
Question-Answw Ansiier Pairs:
###
Answer:
Prompts used in the reference-bused QAG setting (left) and in QA (right). Text in gray denotes optional components for candidate-aware QAG and is omitted in candidate-agnostic setups. QE prompt for QAG is shown in Figure 7. In the monolingual setting, the context (source, reference, or candidates) is in English, while in the crosslingual setting, the context is in some other language.
Figure 2:
the question and reference (or source) text: QA(r, This aligns answer generation with the QA model's behavior and reduces bias, leading to more consistent and robust evaluation.
Answer Correctness We evaluate answer correctness using LERC (Fabbri et al., 2022), a learned metric that scores semantic similarity between a generated and reference answer, conditioned on the context and question, on a 1-5 scale. Trained on crowdsourced judgments of QA outputs, LERC has been widely adopted in QA-based summarization evaluation (Huang & Zhang, 2021; Deutsch & Roth, 2022) and outperforms surface-level metrics like exact match and word-level F1, which often fail to capture semantic equivalence."
3 Experimental Setup
Data: We evaluate our metric on the following datasets, focusing on the xx-en translation direction:" a) LITERARY Dataset, released by Karpinska & lyyer (2023) that includes MQM (Multidimensional Quality Metrics) error span annotation and expert preferences for automatically generated translations of paragraphs of literary texts. We use the XX-EN subset covering Polish, Japanese, French, German, Russian, and Chinese as source languages, retaining 180 instances out of 540. and b) PAR3HUMAN Dataset which includes expert preference between Google Translate output versus reference translation written by a human of public-domain foreign language novels for three language pairs and German-Englian English). We measure contrastive accuracy how well the metric predicts expert preferences between translation pairs given either the source (reference- free) or both the source and the reference (reference-based).
Ffrench-Englise
Mussian-Engllise
Baselines: We compare TREQA against several baseline metrics: (1) lexical metrics such CHRR (Popović, 2015); (2) sentence-level neural metrics like COMET (Rei et al., 2022a), COMETQE (Rei et al., 2022b), efference-based and reference-free versions of XCOMET (Guerreiro al., 2024) and MTEQA (Krubiński et al., 2021) a reference-based QA metric that uses keyphrases as extracted answers; and (3) L.Le- metrics grounded in GEMBAMQM (Koemi & Federmann, 2023a) using the same LLM backbone as TREQA. Further details on these metrics are provided in Appendix B.
In our preliminary experiments with prompting L.MMs for evaluating answer correctness, we found that they often fail to generate diverse correctness scores, with many answers scoring ties
*This choice is motivated by LLMs' superior performance in English compared to other languages (Etxaniz al., 2024), the availability of a robust answer comparator (LERC) in English, and existing paragraph-level translation judgments for this direction.
5

---

Published as a conference paper at COLM 2025
LITERARY (XX-FNN)
PAR3HUMAN
Metric
Model
Cands?
Ref
QF
DE-
FR-EN
RU-FN
CHRF
52.2 2
COMET
62.2
COMETQE
46. 4
37.3
33.3 2
51.3
XCOMET
61.70 2
55.6 3
53.3
41.3 2
54.7
MTEQA
3
GPT-40
40.6 4
44.0 3
33.3 2
$2.0
GEMBAMQM
SONNET-3.5
51.1 4
62.0
54.0
61.3
QWEN2.5-72B-IT
55.0
55.3
44.0 2
55.3
59.4 2
43.9 4
44.0 3
40.0 2
54.0
GPT-40
65.0
56.7 2
46.7
43.3 2
56.0
63.3
56.1
44.7
42.7 2
52.7
TREQA
SONNET-3.5
58.3 2
55.0 3
$2.0
47.3 2
58.0
X
64.4
57.8 1
44.7 3
40.0 2
57.3
64.4
46.7 4
45.4
46.7
58.7
Table Meta-Evalu with Contrastive Accuracy on LITERARY and PAR3HUMAN, with both reference-baaee based and reference-free (QE) metries on the former and QE metrics on the latter. Methods highlighted with blue background use LLMs.
LLM Backbones: For TREQA (as well as GEMBAMQM) we experiment with a diverse set of state-of-the-arr language models, including both the leading closed models and more reproducible open-weight options: for closed models, we explore both GPT-4o (Hurst et al., 2024) and SONNET- 3.5 (Anthropie, 2024)3, two of the leading closed-source models across many NLP tasks and that support the languages targeted. For open-source models, we experiment with the QWEN2.5- IEEN2.5-72B-1 and its smaller siblings QWEN2.5-32B generally competitive with closed-source models in multilingual settings (Yang al., 2024).
and QwEN2.5-7B-IT and this family of models are
Decoding As mentioned, for all these models, we use one of the prompts from Figure 2.6 To ensure that TREQA is reproducible and produces consistent results, comparing to other metrics, setting the temperature to 0.7 However, due to the open-ended nature of the question generation and since 2024), we also explore the impact of sampling on the performance of TREQA with the QWEN2.5 models, by decoding with the recommend sampling parameters."
greedy decoding when
tasks obtain better results sampling from LLMs (Song et al.,
Significance Testing We report clusters for metrics based on statistically significant performance gaps. Specifically, we conduct pairwise Wilcoxon signed-rank tests between all metric pairs within each group, where the dataset and the type of evaluation (Ref or QE) define groups. To account for multiple comparisons, we apply the Renjamini-Hocchber Based on the corrected p-values (thresholded at 0.05), we construct significance matrix indicating which metric differences are statistically significant. This matrix serves two purposes: identifying metrics that are significantly better than others for ranking, and acting as distance matrix for agglomerative clustering to group metrics with similar significance patterns, resulting in final ranking.
FDR correction to the resulting p p-values.
4 Results
Table 1 shows contrastive accuracy for TREQA and baseline metrics on all datasets.
TREQA outperforms is competitive with baselines in both reference-based and reference- free setups. On the LITERARY dataset, TREQA achieves higher contrastive accuracy than lexical
5 -gpt-40- -0-24-0--2--------- 006; claude-3- wtl-ssonnet-222 -2244102
Whill TREQA by default requires question conditioning on all candidates, we present an ablation on using single candidate to probe translation errors in Appendix D.2.
We found that, even with greedy decoding, I.I.ss could produce different generations depending the specific environment setup. We use vilm==0. 6.2 to perform decoding.
Spemperature-O, top_p=0.___ 0. top_k=20

---

Published as a conference paper at COLM 2025
Literary
Par3Human (rw-ea)
0.70
XCDMETTXOOME Without Cundiiatixx With
Grewdy Decoding
0.55
0.50
32B
Figure 3: Mean, minimum, and maximum accuracy when sampling QA pairs 5 times with default sampling settings, as well as with greedy decoding and baseline performance, on LITERARY (reference-based) and PAR3HUMAN (RU+EEN, reference-free) datasets.
and neural metrics, particularly when using strong closed-source LLMs like GPT-40. Traditional lexical reference-based metries, such as CHRF, perform near-random levels. COMET and XCOMET achieve slightly stronger results but struggle to exceed 60%, while their QE counterparts perform even worse. Directly prompting LLMs to evaluate translations using GEMBAMQM also does not yield better results than state-of-the-art QE models like XCOMET, even with the best closed-source models. Whereas, TREQA consistently benefits from strong LLMs and outperforms all others on LITERARY when candidates included, especially with SONNET-3 3.5 and QWEN2.5-72B IT. On the PAR3HUMAN dataset, the picture is more nuanced: in the DE EN direction, TREQA lags behind GEMRA MQM-based scores, and only marginally outperforms COMETQE. However, in the other two directions, TREQA proves a competitive evaluator, only being surpassed by GEMBAMQM with SONNET-3.5 (which is the best metric overall). Interestingly, in the FR direction, except GEMBAMQM with SONNET-3.5 5, none of the other metrics surpasses random performance, hinting the difficulty of current metries in complex, long-form domains.
There is no consensus on the best model for question generation, but candidate-aware generation generally outperforms candidate-agnostic. Analysis across both datasets reveals that the optimal LLM for question generation varies: GPT-40 performs best on LITERARY, while SONNET-3.5 leads on PAR3HUMAN. In general, TREQA benefits from candidate-aware setup, where the model sees the candidate translation when generating questions. However, there are exceptions-mos notably, SONNET-3.5 3.5 achieves its highest performance LITERARY in the aandidate-agnoste setting. This discrepancy may stem from the inherent stochasticity of the question-generation process, an aspect we explore more detail next.
Sampling from larger models improves TREQA'S 's accuracy over smaller models, and candidate conditioning improves mean and reduces variance. As discussed earlier, the eped-ended nature of question generation creates a vast space of possible QA pairs that can uncover different translation errors. Relying on a single deterministic generation (e.g., greedy decoding) may, therefore, underrepresent TREQA's 's full potential, as different questions expose different aspects of translation quality. To study this, Figure 3 shows the mean, minimum, and maximum contrastive accuracy of TREQA across five sampled runs-both with and without candidate conditioning, using various sizes of the Qwen family. Greedy decoding results and baseline performance from XCOMET also included for comparison. As expected, accuracy improves with model size, with QWEN2.5-72B-I1 consistently outperforming its smaller variants in both greedy and sampling settings. Comparing sampling to greedy decoding, the impact of candidate conditioning is mixed when looking at individual scores no consistent pattern emerges across model sizes or datasets. However, when analyzing the distribution of outcomes across sampling runs, clearer trends appear: including candidates degrades performance for the smaller model but improves mean performance for the two larger models. This suggests that generating questions highlighting differences in candidate translations requires more capable models. Additionally, conditioning on candidates can make TREQA more stable, shown by the low variance in accuracy of the QWEN2.5-72B-IP model on the LITERARY dataset. Finally, the ceiling of TREQA performance is notably high, with one run achieving nearly 70% the LITERARY dataset.
7

---

Published as a conference paper at COLM 2025
Reference based
Reference free (QE)
Accuracy (witbout candidates)
Recall twith candidates)
Accuracy candidates)
Overlap (Chrb) candidates)
Recall (without candidates)
Brcec Overlap (Chrf) (with cancklates)
0.4
0.2
OWEN2.6-728-17
7-78-75-728-17
Figure 4: Accuracy, exact error coverage via recall and partial coverage (using CHRFF) for the TREQA models presented in Table I: recall reliably reflects accuracy trends.
Answers
Regenerated Answers
XIOMBE-Qg
Negative Pirst
Randon: Order
Pesitive Pirtt
0.5
Figure 5: Impact on accuracy of presenting either the best candidate first, last, or in random order
(Figure 2), using the originally generated answers (left) and regenerated (right).
Ablation Analysis
In this section, we examine how key design choices in TREQA affect its ability to generate meaningful and error-targeting questions, as well the overall accuracy of candidate ranking.
Do the generated questions target translation errors? We hypothesize that candidate-aware generation enables the LLM to produce questions that specifically target translation errors. To test this, evaluate the overlap between model-generated answers and annotated error spans in the LITERARY dataset. We use two complementary metrics: (I) exact coverage via recall-whether the generated answer fully contains annotated span-and (II) partial coverage using CHRF (Popović, 2015), which quantifies the similarity between generated answers and error spans (see Appendix C for additional details). Exact recall offers poarse-grained signal and may favor longer responses, while CHRF provides a finer-grained similarity measure. Figure shows these metrics for the TREQA models from Table 1, comparing reference-basee and reference-fre setups with candidate- aware and candidate-agnose reliably reflects accuracy trends across models and settings. While candidate-aware generation consistently improves CHRF, exact recall better tracks overall performance. This is especially clear with QWEN2.5-722
question generation. Recall with error spans
in the QE setup: CHRF improves with candidates but recall drops, suggesting the answers include related fragments observation highlights that identifying the entire error span is more critical than merely achieving surface-level similarity through partial matches paraphrased content.
paraphrases without fully covering the enror span. This
How important it is to regenerate the answers with the QA system? To assess whether answer generation truly necessary, we compare accuracy using QAG-generated gold answers versus QA-generated ones. Figure shows TREQA results when the best worst candidate is presented either first, last, or in random order, and when using either the original or regenerated answers. We find that performance with QAG -eG-generteed answers increases sharply when the best candidate is presented first, and drops below random when the worst is first. This reveals a strong positional bias: when generating questions conditioned on candidates, LLMS often ignore instructions and instead
8

---

Published as a conference paper at COLM 2025
QAG Fails Don't target errors Subsumes crrors
QAG Failss Aren't discr minntiive
Reforement bogan iose his
thought gig gring the Pettenstuugg
nenor After zoch shot, ongg ha fiod for lang time. Hop Miler each situ, dogs would how longg lime. Hbp After coch shot, the dogs howled for long timo.
Hyp Ostap began blind. Then ne decded go to Pesersourg
Hyp Ostop had started to go bilnd, He nad cedded go to St. Fetensourg
Q. did Ostap decide do tre started
sight?
Qe What napperied alcer each guronot Ac After each shot, digs barked for long time.
began to signt, thought atouut going
Prtersbuug
H/g+ Ostap
po to Hyp A: Ostop decied St. Petersburg
Hagp A. Afler each shut, dugs woull howl for a long lime Hyp- & After each shot, the dogs howled for a long time.
QA Faills Inaccurate given context
Ambiguous Preference
Refenence: Girandma's sattr rousi Dac's was son,
Referencer Porhaps untororriiisttyy sty,, wwatter seop suffering from Ded's departure, juSt like was
Hyp My grandmether had younger aster .. cousin was
itp- grandmothe" youugg sister.. cousin here the oldest
Hyp +: Perhaas tu subconscluully hett that she should sutter becouse of my dadts degartur just like me.
How cousin cldess son
Grandmars sister?
Hyp Deep down, I fel. Jhat site should remtain pain and sonnow, just like me. oue to farhers departure.
Hypi Answer. the cousin eldest son of oreeddmee siter. Hyp Answer: The cousin the eldest son of andimasssss litle sister.
Figure We highlight several failure cases of TREQA: questions include errors (top-left) or target same error for both candidates (top-right), QA generates answers not supported by context (bottom-left) and scoring fails when expert preferences are ambiguous (bottom-right).
copy answers directly from the first espeiill in the QE setting (see also Wei et al., 2024). This behavior undermines instruction-folle following and suggests shallow reasoning. Regenerating answers mitigates the issue by forcing the model to ground its output in the reference rather than copying from candidates, thus better reflecting true comprehension.
Where does TREQA fail? To better understand where TREQA falls short, we also manually analyze failure cases using the QWEN2.5-722) ---x2- model. We find that emrors fall into three main categories: (1) generated questions fail to target errors or inherit candidate errors in the questions (Kamoi et al., 2023) or are discriminative between candidates; (2) incorrect answers (paraphrased or inferred by the QA model) given the context (for ground truth or candidate) and (3) cases where the expert preference ambiguous or underspecified, making reliable evaluation difficult. We provide examples of some of these categories in Figure 6. While TREQA is competitive with strong baselines and offers interpretable outputs, our analysis suggests several promising directions for future improvement. In particular, more controlled question generation targeting errors in a pragmatic way, improved QA systems, and better alignment answer matching could further improve the reliability and precision of our metric, especially in challenging domains."
Related Work
Automatic Evaluation Prior automatic evaluation methods apply sentence-level metrics directly to longer texts or explicitly adapt them for paragraph-Ievel evaluation (Deutsch et al., 2023; Raunak et al., 2023; Vernikos al., 2022), often assuming one-to-one sentence alignment. Such assumptions fail in realistic scenarios involving structural variations (sentence segmentation, reordering, restructuring). Discourse-level metries (Joty al., 2017; Jiang et al., 2022) address these by evaluating coherence through discourse categories. In contrast, TREQA uses L.M--dervved answers to capture deeper semantic and pragmatic nuances beyond span-level alignments.
QA-based Evaluation QA-based evaluation methods have gained popularity across NLP tasks such as summarization (Durmus et al., 2020; Wang et al., 2020: Scialom et al., 2021; Fabbri et al., 2022: Liu et al., 2024), simplification (Angrosh et al., 2014; Agrawal & Carpuat, 2024), knowledge- based dialogues Honovich et al. (2021), long-form generation (Tan et al., 2024), and machine translation (MT) (Berkaab et al., 10 Forcada et al., 2018; Krubiński et al., 2021: Han et al., 2022). Specifically, Krubiński et al. (2021) )) propose MTEQA, focusing on keyphrase-drive correlating strongly with human judgments at the system-level. Our method differs by employing
questions
We show the distribution of question types generated by our models in D.I and an ablation cross- -ingual QA performance, finding that, compared to its English QA performance, QWRN2.5-72B-IT exhibits substantial for improvement cross-lingual QA (see Appendix D.4).

---

Published as a conference paper at COLM 2025
LLMs directly generate comprehension questions targeting deeper tross-lingual semantic fidelity, directly conditioned on candidate and ground-trun truth texts.
LLM-as-a-judge Recent approaches utilizing LLMs evaluative judges have emerged across NLP tasks (Liu et al., 2023; Shen et al., 2023; Kim et al., 2024a; Hada et al., 2024; Vu et al., 2024). typically producing implicit quality assessments. For MT, Koemi & Federmann (2023b) use LLMs for holistic judgments, whereas Fernandes et al. (2023) employ detailed error assessments. Concurrently, Shafayat al. (2024) propose rubric-based questions generated by LLMs for holistic evaluation. Distinctively, our QA-driven LLM approach explicitly assesses cross-lingual comprehension, pinpointing individual inaccuracies and providing complementary, precise insights sensitive semantic accuracy.
7 Practical Recommendations
Our experiments offer several practical insights for those looking to use TREQA to evaluate paragraph-level MT, particularly in complex domains like literary texts as discussed below:
Conditioning the generation of questions on candidate translations to be evaluated along with the source / reference texts (the "with candidates" setting) generally yields the best results, as helps the LLM focus on discriminative differences and potential errors (Sections 4, 5, D.1).
Start with Candidate-Aware QAG:
Use Powerful LLMs: Leveraging the most capable LLMs available for both Question-Answer Generation and Question Answering is crucial. Although there is no single best model across all scenarios, models like GPT-4o, SONNET-3.5, and QWEN2.5-72B-IT consistently perform well (Table 1, Figure 3).
Always Regenerate Answers: Re-generating answers using the QA model, grounded in the source/reference text, is essential to mitigate strong positional biases and hallucinations and ensure that the evaluation truly reflects comprehension of the ground truth rather than copying from candidates (Figure 5).
Consider Sampling for QAG: While greedy decoding offers consistency, using sampling (e.g.. temperature > 0) during question generation can explore wider range of potential issues and often leads to higher mean accuracy over multiple runs, especially with larger models (Figure 3).
Non-English Target Texts: The current TREQA framework relies on English questions and shows reduced QA performance when the context is not in English (D.4). Hence for evaluating translations into languages other than English, we would recommend leveraging newer multilingual
LLMs with stronger cross-lingual QA capabilities with TREQA.
Iterate and Inspect: The questions generated by TREQA offer interpretability. Manually inspecting the questions and answers can provide insights into why a translation favored and can help refine prompts settings for specific cases.
Conclusion
Our work underscores the potential of pragmatic and extrinsic evaluation approaches such as TRBQA to assess the quality of translation of complex, long-range contexts found in literary domains. By leveraging LLM- M.M--gnnrated comprehension questions that explicitly target content (key concepts, phrases, and entities) in the source or reference texts and target errors in translations and key concepts required, TREQA not only matches but often surpasses existing state-of-the-art metrics, while also providing interpretability through explicit identification of translation errors. Although challenges remain due to the limited cross- cross-lingual capabilities of current models, our framework paves the way for future research towards more informative and practically useful translation evaluation methodologies.
Acknowledgments
We thank Jessy Li, António Farinhas, Eleftheria Briakou, Duarte Alves, and the SARDINE lab team for helpful discussions and feedback on earlier versions of the paper. This work was supported by EU's Horizon Europe Research and Innovation Actions (UTTER, contract 101070631), by the project DECOLLAGE (ERC-2022-CoG 101088763), by the Portuguese Recovery and Resilience
10

---

Published as a conference paper at COLM 2025
Plan through project C645008882 (Center for Responsible AI), and by FCT/MECI through national funds and when applicable co-funded EU funds under UID/50008: Instituto de Telecomunicççees.
References
Sweta Agrawal and Marine Carpuat. Do text simplification systems preserve meaning? a human evaluation via reading comprehension. Transactions of the Association for Computational Linguistics, 12:432-445 448, 2024. doi: 10.1162/tac a 00653. URL
org/2024. tacllaacll 1111--1.4/4..
Sweta Agrawal, José G. C. De Souza, Ricardo Rei, António Farinhas, Gonçalo Faria, Patrick Fernandes, Nuno M Guerreiro, and Andre Martins. Modeling user preferences with automatic metries: Creating high-quality preference dataset for machine translation. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 14503-14510 Miami, Florida, USA, November /t/5/////222...mmmmmmoooo
2024. Association for Computational Linguistics. doi: https: aa httpa/aaaaannttoa..aaaaaao www.//y.ooroooooooooo..o 2taz///22222 2024 emnlp ma in.. 803/.
URI.
Mandya Angrosh, Tadashi Nomoto, and Advaith Siddhartham. Lexico-syntacee text simplification and compression with typed dependencies. In Junichi Tsujii and Jan Hajic (eds.), Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pp. 1996-2006, Dublin, Ireland, August 2014. Dublin City University and Association for Computational Linguistics. URL https::///// hwwp:/hool....oo....oooooo
AI Anthropic. The claude 3 model family: Opus, sonnet, haiku. 2024. URL https://api. semanticschon .rg Sttg//orppssiin 268232499.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. On the cross-lingual transferability of monolingual representations. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4623-4637, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/ 2020.acl- main.421..... URL https: /atee///aaaaaaaaaiaa www.attonnnnnoooooocccaaaoooaata org/ 2020 mall-mman....m 421/.
Jan Berkaab, Martin Černaa, and Ondřej Bojarb. Quiz-based evaluation of machine translation. The Prague Bulletin of Mathematical Linguistics, (95):77-86, 2011.
Shuyang Cao and Lu Wang. Controllable open-ended question generation with new question type ontology. In Chengqing Zong, Fei Xia, Wenjic Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting the
the Association for Computational Linguistics and the I 1ith International
Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 6424 6439, Online, August 2021. Association for Computational Linguistics. doi: 10. (0.18653/v1/202 lacl--lon. 502. URL https: /aa/aalamm aacccannhologyy org/ 2 2021 acc- -ong. 502/.
Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. InfoXLM: An cntormmtioonthhoreeee framework for cross- lingual language model pre-training. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techmologies, pp. 3576-3588, Online, June 2021. Association for Computational Linguistics. doi: 18.18653/ 3 1/202 mwacl-main.n20 URL https: hictps://cclantholo0.
org//0211 naacl nwacl-malaaa.aa.aa in. 280/.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. Linguistics. doi: 10. 10.18653/v 1 1/2020.acll.ccc 2012000.llmmii..71 747. URL https: : / ac holonnooooogyy..ooooooooooooooom org/ 202 a00.cmmma 747/-
Online, July 2020. Association for Computational
11

---

Published as a conference paper at COLM 2025
Daniel Deutsch and Dan Roth. Understanding the extent to which content quality metries measure the information quality of summaries. In Arianna Bisazza and Omri Abend (eds.), Proceedings of the 25th Conference Computational Natural Language Learning. pp. 300-309 Online, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2022 wtt.////.1....oon/// .conll-1.24 URL https: /tt///laaaaaoooaaeeo orgg 2021
Daniel Deutsch and Dan Roth. Benchmarking answer verification methods for question answering- based summarization evaluation metrics. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Computationnll Linguistics: ACL 2022, pp. 3759-3765, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/ v1/2022. 21/2022.fndinns--a findinngs-alc.29900 29.. URL https: / www.aaaaaaoooooo..ooooooooooolaa www.ga...r..oooo org. http:/222....nnnnng.eee findings-act 296/.
Daniel Deutsch, Tania Bedrax-Weist and Dan Roth. Towards question-ansswering an automatic metric for evaluating the content quality of a summary. Transactions of the Association for Computational Linguistics, 9:774-789, 2021. doi: a URL https://
aclanthology. wttg////10.. wat.ttaclll.... 1.77/.
Daniel Deutsch, Juraj Juraska, Mara Finkelstein, and Markus Freitag. Training and meta-evaluating machine translation evaluation metrics at the paragraph level. In Philipp Koehn, Barry Haddow, Tom Koemi, and Christof Monz (eds.), Proceedings of the Eighth Conference on Machine Translation, pp. 996-1013, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 00.18653/11/2223.www 1.96. URL https: http::///acaaattholo 96/.
org/2025 wmt-1.
Esin Durmus, He He, and Mona Diab. FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5055-5070, Online, July 2020. Association for Computational Linguistics. doi: 10. 00.18653/11/0220.0 acl- 454..454. URL https: //taaantholclag.ayooooaa.. org/ 2020. acl.main..mm.... 454/.
Julen Etxaniz, Gorka Azkune, Aitor Soroa, Oier Lopez de Lacalle, and Mikel Artetxe. Do multilingual language models think better in english? arXiv preprint 2.ti::230000/2222, 2023.
Julen Etxaniz, Gorka Azkune, Aitor Soroa, Oier Lopez de Lacalle, and Mikel Artetxe. Do multilingual language models think better in English? In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers). pp. 550-564, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 18653vv1/ 2024.naacl-shor
URL https: / aclanthoooogyy org / 222/222242222 www.c-aaahhrrrrraaaa 46/.
Matan Eyal, Tal Baumel, and Michael Elhadad. Question answering metric for news article summarization. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 3938-3948, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10.18653/v 1/N59- URL https: hataaaaaaanttooooooooeoeaac org/ /et//N99--2 Nttp/NN99111////
automatic evaluation
Alexander Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. QAFactEval: Improved QA-based factual consistency evaluation for summarization. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2587-2601, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10. 00111865611//// htt333////222.nn// htt.:// naael22naalmaaan maaaaaaaaann..ammaam 187. URL https: / /t///alaamat www.cnnnoooooy.....ooooooooooole org/ 2022. www.aaaanna....... mamaain18.....
Patrick Fermandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, André Martins, Graham Neubig, Ankush Garg, Jonathan Clark, Markus Freitag, and Orhan Firat. The devil is in the errors: Leveraging large language models for fine- fine-grained machine translation evaluation. In Philipp Koehn, Barry Haddow, Tom Koemi, and Christof Monz (eds.), Proceedings of the Eighth Conference Machine Translation, pp. 1066-1083. Singapore, December 2023.
12

---

Published as a conference paper at COLM 2025
Association for Computational Linguistics. doi: 10.18653/11/2223.mmm w001: URL https: //aclanthology org//o///002 wmt wmt-1.100/.
Mikel L. Forcada, Carolina Scarton, Lucia Specia, Barry Haddow, and Alexandra Birch. Exploring gap filling as a cheaper alternative to reading comprehension questionnaires when evaluating machine translation for gisting. In Ondiej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Matt Post, Lucia Specia, Marco Turchi, and Karin Verspoor (eds.), Proceedings of the Third Conference on Machine Translation: Research Papers, pp. 192-203, Brussels, Belgium, October 2018. Association for Computational Linguistics, doi: 10. N0.18633/1/W118-3322 URL https: / ////acaaaa aclanthology org /118-18-- 6320/.
M. Fuji, N. Hatanaka, E. Ito, S. Kamei, H. Kumai, T. Sukehiro, T. Yoshimi, and H. Isahara. Evaluation method for determining groups of users who find MT "useful". In Bente Maegaard (ed.), Proceedings of Machine Translation Summit VIII, Santiago de Compostela, Spain, September 18-22 2001. URL https: ht//aalaaaaa http://aaaaanntta....ao.o.ooa.at
org/ org/2001. 2001. mt surmit-papers 200p
Nuno M. Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and André F. Martins. xcomet: Transparent machine translation evaluation through fine-grained error detection. Transactions of the Association for Computational Linguistics, 12:979-995, 2024. doi: 10.1162/
tacl a 00683. URL https:
hattp:://aclaattelogs /alaaaaanntthggge httpa//oooo...../222.e//// wwt.oorgg/222./ ta1224..acl11...
Rishav Hada, Varun Gumma, Adrian Wynter, Harshita Diddee, Mohamed Ahmed, Monojit Choudhury, Kalika Bali, and Sunayana Sitaram. Are large language model-based evaluators the solution to scaling up multilingual evaluation? In Yvette Graham and Matthew Purver (eds.). Findings of the Association for Computational Linguistics: EACL 2024, pp. 1051- 1070. St. Julian's, ss Malta, March 2024. Association for Computational Linguistics. URL https: / aclanthologyy.....o....o.ooollle orgg/02//442000... findings-eacl. 71/.
HyoJung Han, Marine Carpuat, and Jordan Boyd-Graber. SimQA: Detecting simultancous MT errors through word-by-word question answering. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. pp. 5-95-5599111 Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 13138663/vll 11263/////222....2///// mamp222emmlpmmm..... 2022. emnlp-maiii /7
URL https: : / aalaaccaattte www.annhhllgyyy org/
Or Honovich, Leshem Choshen, Roee Aharoni, Ella Neeman, Idan Szpektor, and Omri Abend. q²: Evaluating factual consistency question answering. In Marie-Francine Moens, Xuanjing Huang. Lucia Specia, and Scott Wen- tau Yih (eds.), Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pp. 7856-7870, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 18.18653/v 1/2021 wewpllmmmlppaaa main.619. URL https: //aclanthology. .rg/ 2021. emnlp-man malp-mmain... 619/.
gowwlede-groundeer dialogues via question generation and
Nannan Huang and Xiuzhen Zhang. Evaluation of review summaries via question-answering. In Afshin Rahimi, William Lane, and Guido Zuccon (eds.), Proceedings of the 19th Annual Workshop of the Australasian Language Technology Association, pp. 87-96, Online, December 2021. Australasian Language Technology Association. URL /al//claantt acw.athhoonntcy.cccc org/ 2021.alta- 9/.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-40 system card. arXiv preprint arXiv:2410.21272 2024.
Yuchen Jiang, Tianyu Liu, Shuming Ma, Dongdong Zhang, Jian Yang, Haoyang Huang, Rico Sennrich, Ryan Cotterell, Mrinmaya Sachan, and Ming Zhou. BlonDe: An automatic evaluation metric for document-level machine translation. In Marine Carpuat, Marie-Cathering de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1550-1565, Seattle, United States, July 2022. Association for Computational
13

---

Published as a conference paper at COLM 2025
Linguistics. doi: 10.18663//122022mmaac--ho URL /Rt..cttss///ccaaaatlloooo/o 2022. .nwacl-main.n11.
Shafiq Joty, Francisco Guzmán, Lluís Märquez, and Preslav Nakov, Discourse structure in machine translation evaluation. Computational Linguistics, (S(4))683-722, December 2017. doi: 10.1162/
COLI a 00099.. URL
Juraj Juraska, Mara Finkelstein, Daniel Deutsch, Aditya Siddhant, Mehdi Mirzazadeh, and Markus Freitag. MetrieX-23: The Google submission to the WMT 2023 metrics shared task. In Philipp Koehn, Barry Haddow, Tom Koemi, and Christof Monz (eds.), Proceedings of the Eighth Conference on Machine Translation, pp. 756-767, Singapore, December 2023. Association for Computational Linguistics. doi: 10.186533/1/2000 12203///12222.2 wmm- 1.63. URL https: /aa//aaaaaaanoooooo//aaa org/2020 wmt 63/.
Ryo Kamoi, Tanya Goyal, and Greg Durrett. Shortcomings of question answering based factuality frameworks for eтror localization. In Andreas Vlachos and Isabelle Augenstein (eds.), Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 132-146, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. doi: 11886633/1122233 h2tp://eee......mmamii.. macl-main.11. URL https /at////aalatttooo/ wwtp:.rrg//oo 2023g eacl-mailii 11/.
Marzena Karpinska and Mohit lyyer. Large language models effectively leverage document-level context for literary translation, but critical errors persist. In Philipp Koehn, Barry Haddow, Tom Koemi, and Christof Monz (eds.), Proceedings of the Eighth Conference Machine Translation, pp. 419-451, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 18653/v1/2023
URL httpttps: / ac anthology. - org/ 2023. wmt-1.411 141/.
Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Prometheus 2: An open source language model specialized in evaluating other language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 4334-4353, Miami, Florida, USA, November 2024a. Association for Computational Linguistics. doi: 10. 1008866//11222244mmmmmmm/ https: haapss//aallatthllgggoe -
URL
2024 emnlp-ma in 248/.
Yekyung Kim, Yapei Chang, Marzena Karpinska, Aparna Garimella, Varun Manjunatha, Kyle Lo, Tanya Goyal, and Mohit lyyer. FABLES: Evaluating faithfulness and content selection in book-length summarization. In First Conference on Language Modeling, 2024b. URL https:
/ /popenreview net htt.://www..e/otmmmtttteeae wiwammiiidfffoSaa...
Tom Koemi and Christian Federmann. GEMBA-MQM: Detecting translation quality error spans with GPT-4. In Philipp Koehn, Barry Haddow, Tom Kocmi, and Christof Monz (eds.), Proceedings of the Eighth Conference Machine Translation, pp. 768-775, Singapore, December 2023a. Association for Computational Linguistics. doi: 10. 10.18653/w1/20233www
URL https:
// //acllnthology org/ 2023. wmt-1.62 146/
Tom Koemi and Christian Federmann. Large language models are state-of-the-art evaluators of translation quality. In Mary Nurminen, Judith Brenner, Maarit Koponen, Sirkku Latomaa, Mikhail Mikhailov, Frederike Schierl, Tharindu Ranasinghe, Eva Vanmassenhove, Sergi Alvarez Vidal, Nora Aranberri. Mara Nunziatini, Carla Parra Escartín, Mikel Forcada, Maja Popovic, Carolina Scarton, and Helena Moniz (eds.), Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pp. 193-203 203, Tampere, Finland, June 2023b. European Association for Machine Translation. URL https: : httpps://ccaatthllooo// acl.athoooooooooo.cccooooooooola . wwt./org//2222.. 2223. eamt-1. 19/.
Tom Koemi, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Marzena Karpinska, Philipp Koehn, Benjamin Marie, Christof Monz, Kenton Murray, Masaaki Nagata, Martin Popel, Maja Popović, Mariya Shmatova, Steinthór Steingrimsson, and Vilém Zouhar. Findings of the WMT24 general machine translation shared task: The LLM era is here but MT is not solved yet. In Barry Haddow, Tom Kocmi, Philipp Koehn, and Christof Monz (eds.), Proceedings of the Ninth Conference on Machine Translation, pp. 1-46, Miami, Florida, USA,
14

---

Published as a conference paper at COLM 2025
Association for Computational Linguistics. doi: 10.18653/11/2224 wwt.-......mt1........... 1.1. URL https::///// / / httaaaaannntoooooooooooooo orq/ 2024. 11244..t11111
November 2024.
Philipp Koehn and Christof Monz (eds.). Proceedings on the Workshop on Statistical Machine Translation, New York City, June 2006. Association for Computational Linguistics. URLhttpa.
// //aclanthology org/ WD --00-3100000 31000/.
Mateusz Krubiński, Erfan Ghadery, Marie-Francine Moens, and Pavel Pecina. Just ask! evaluating machine translation by asking and answering questions. In Loic Barrault, Ondrej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussa, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Tom Koemi, Andre Martins, Makoto Morishita, and Christof Monz (eds.), Proceedings of the Sixth Conference on Machine Translation, pp. 495-506, Online, November 2021. Association for Computational Linguistics. https: //acl hnt 1 ogy. org/ 2 2021 1, wmt-1. 58//
Vishwajeet Kumar, Nitish Joshi, Arijit Mukherjee, Ganesh Ramakrishnan, and Preethi Jyothi. Cross- lingual training for automatic question generation. In Anna Korhonen, David Traum, and Lluís Märquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v 191111199111 1481. URL ht ps:::ps://s /atle///aaaaaaaa /aaaaaaaannnooooooee
Linguistics, pp.
Zhen Li, Xiaohan Xu, Tao Shen, Can Xu, Jia-Chen Gu, Yuxuan Lai, Chongyang Tao, and Shuai Ma. Leveraging large language models for NLG evaluation: Advances and challenges. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 16028- 1604,, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.
896. URL https: al aww.aanooantoooo.oocoooo org/ 2024 emnlp maa-pp-mainnn main. 896/
Junyuan Liu, Zhengyan Shi, and Aldo Lipani. SummEQuAA: Summarization evaluation via question answering using large language models. In Bhavana Dalvi Mishra, Greg Durrett, Peter Jansen, Ben Lipkin, Danilo Neves Ribeiro, Lionel Wong, Xi Ye, and Wenting Zhao (eds.). Proceedings of the 2nd Workshop on Natural Language Reasoning and Structured Explanations ( (@ACL. 2024), pp. 46-55, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL https: http///aaaaaahhohcoco/o
1/5.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: NLG evaluation using gpt-4 with better human alignment. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 2511-2522, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 1t111863/1122223.mmlll http://22...mmmmma.m......... URL https: h/wpa//hhccaclc...c/o//.o org/ 2023. www.eemainn.n.m.mememeemmmmmmmae 153/.
Prabir Mallick, Tapas Nayak, and Indrajit Bhattacharya. Adapting pre-trained generative models for extractive question answering. In Sebastian Gehrmann, Alex Wang, João Sedoc, Elizabeth Clark, Kaustubh Dhole, Khyathi Raghavi Chandu, Enrico Santus, and Hooman Sedghamiz (eds.), Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM), pp. 128-137, Singapore, December 2023. Association for Computational Linguistics. URL https: http://alaaatthooocooo
wtgg/0233300 g.m- 1././
Nitika Mathur, Timothy Baldwin, and Trevor Cohn. Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4984-4997, Online, July 2020. Association for Computational Linguistics. doi: 10. 1/2533/l/2020.ccll main.448.. URL https: / ht//aalaam/ anthhllohhooyy..ooo org/ 2020. awl-maamma.... 448/.
Feng Nan, Cicero Nogueira dos Santos, Henghui Zhu, Patrick Ng, Kathleen McKeown, Ramesh Nallapati, Dejiao Zhang, Zhiguo Wang, Andrew O. Arnold, and Bing Xiang. Improving factual consistency of abstractive summarization via question answering. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference
15

---

Published as a conference paper at COLM 2025
Natural Language Processing (Volume 1: Long Papers), pp. 6881-6894, Online, August 2021. Association for Computational Linguistics. doi: 10. 00.18633/1/2002 lacl-long..36. URL https: ptttp:://aclanthlloogg wrt.ooool....... acl- acc--oog. 536/.
Jan- aan-Thostten Peter, David Vilar, Daniel Deutsch, Mara Finkelstein, Juraj Juraska, and Markus Freitag. There's s data like better data: Using QE metrics for MT data filtering. In Philipp Koehn, Barry Haddow, Tom Koemi, and Christof Monz (eds.), Proceedings of the Eighth Conference on Machine Translation, pp. 561-577, Singapore, December 2023, Association for Computational Linguistics, doi: 10.18653/ 3/2622//1/2223333 wmt-1.50. URL hitps: ///aaaaaaaanttoooo..ooooo org. 20233. wmt-1.50/.
Maja Popović. chrF: character n-gram F-score for automatic MT evaluation. In Ondfej Bojar, Rajan Chatterjee, Christian Federmann, Barry Haddow, Chris Hokamp. Matthias Huck, Varvara Logacheva, and Pavel Pecina (eds.), Proceedings of the Tenth Workshop on Statistical Machine Translation, pp. 392-395, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10. 10.186653w1/W15530444 Stq/w15-3044
URL https: http:://////acln hatp:::://alantholooyy.
Matt Post and Marcin Junczys-Dowmunt. Escaping the sentence-level paradigm in machine 2023.
translation. arXiv preprint
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Jian Su, Kevin Duh, and Xavier Carreras (eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v1/ D16- 1264. URL https: / / aallnthhlooyyy .rggpoorgDDD. 1-166--244444 1264/.
Vikas Raunak, Tom Koemi, and Matt Post. Evaluating metrics for document-context evaluation in machine translation. In Philipp Koehn, Barry Haddow, Tom Koemi, and Christof Monz (eds.), Proceedings of the Eighth Conference on Machine Translation, pp. 812-81 114814, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.18633/11/0023 wmt- 1.68. URL https: /a////aaaannttoogooeeeeeeee
org/ 2023. wmt amtt1.68/
Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In Philipp Koehn, Loic Barrault, Ondfej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Tom Koemi, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Marco Turchi, and Marcos Zampieri (eds.), Proceedings of the Seventh Conference on Machine Translation (WMT), pp. 578-585, Abu Dhabi, United Arab Emirates (Hybrid), December 2022a. Association for Computational Linguistics. URL https:ttt//.c aac/ antho logy. org/ 2022. wnt 11art-11.55 1.52/.
Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André T. Martins. CometKiwi: IST-unbabel 2022 submission for the quality estimation shared task. In Philipp Koehn, Loic Barrault, Ondřej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Tom Kocmi, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Marco Turchi, and Marcos Zampieri (eds.), Proceedings of the Seventh Conference on Machine Translation (WMT), pp. 634-645, Abu Dhabi, United Arab Emirates (Hybrid), December 2022b. Association for Computational Linguistics. URL https::p///........ waaaaaaanhholoooo logy. org/o/////2222 2022 wmt - 1. 60/.
Andreas Säuberli and Simon Clematide. Automatic generation and evaluation of reading comprehension test items with large language models. In Rodrigo Wilkens, Rémi Cardon, Amalia Todirascu, and Núria Gala (eds.), Proceedings of the 3rd Workshop Tools and Resources for People with REAding Difficulties (READI) CREC-COLING 2024, pp. 22-37, Torino, Italia, May 2024. ELRA and ICCL. URL https: /acl aachannholll ogy. org/ 2024 readi -1 -..3/
16

---

Published as a conference paper at COLM 2025
Carolina Scarton and Lucia Specia. A reading comprehension corpus for machine translation evaluation. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelnik. Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Tenth International Conference on Language Resources and Evaluation (LRECC' 16). pp. 3652-3658, Portorož, Slovenia, May 2016. European Language Resources Association (ELRA). URL https: / hat/a/aaaaanntoooooeoeeeeam or9/91115599
Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. QuestEval: Summarization asks for fact-based evaluation. In Marie- Francine Moens, Xuanjing Huang. Lucia Specia, and Scott Wen-tau Yih (eds.). Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6594 6604, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational org/
Linguistics. doi: 10. 10.18663/11/2211 wwwwmmmmiinn.....mmmmmmmm URL https: hatpa/aaannnttoooo 2021. wwwne-nnnllaaeee maallp-main.... 529/.
Sheikh Shafayat, Dongkeun Yoon, Woori Jang, Jiwoo Choi, Alice Oh, and Seohyon Jung. A 2-step framework for automated literary translation evaluation: Its promises and pitfalls. arXiv preprint arXiv:2412.01344 2024.
Chenhui Shen, Liying Cheng, Xuan-Phi Nguyen, Yang You, and Lidong Bing. Large language models are not yct human-level evaluators for abstractive summarization. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 421554233 Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v 1/2023 findings-emmlp.227 findings emn pp..1p.........
URL https: http//aaaaaattttyyy.aooaa wttpthhhooogy..r/22222.2
Yifan Song, Guoyin Wang, Sujian Li, and Bill Yuchen Lin. The good, the bad, and the greedy: Evaluation of Ilms should not ignore non-determinism. 2024.
Haochen Tan, Zhijiang Guo, Zhan Shi, Lu Xu, Zhili Liu, Yunlong Feng, Xiaoguang Li, Yasheng Wang, Lifeng Shang, Qun Liu, and Linqi Song. ProxyQA: An alternative framework for evaluating long-form text generation with large language models. Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computaaional Linguistics (Volume 1: Long Papers), pp. 6806-6827, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18853/v www.02...12-12on.ooo URL https: hatpp//////llatholool/y/ org/ ac1- 368/.
Masaru Tomita, Masako Shirai, Junya Tsutsumi, Miki Matsumura, and Yuki. Evaluation of MT systems by TOEFL. In Proceedings of the Fifth Conference on Theoretical and Methodological Issues in Machine Translation of Natural Languages, Kyoto, Japan, July 14-16 1993. URL https: ptttps:/aaclatten /acaaaaaannncoooccacaaa org/ 1993. tmi-1.221
Jan Trienes, Sebastian Joseph. Jörg Schlötterer, Christin Seifert, Kyle Lo. Wei Xu, Byron Wallace, and Junyi Jessy Li. InfoLossQA: Characterizing and recovering information loss in text simplification. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4263- 4294, Bangkok, Thailand, August 2024, Association for Computational Linguistics. doi: 10.18653/ al/2024.aclloong225 URL https aac/aaaaantte wwwaacnnnnncolyyo. org/ 2024 acl-long 1esilloonn.23.1.. 234/.
Asahi Ushio, Fernando Alva-Manchego, and Jose Camacho-Collados. An empirical comparison of LM-based question and answer generation methods. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 19262-14272 Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 2tt.://////223...nnn----ee 2023. findingnnnnnnin acl. 889/.
URL https: / /aa/aalaattttt /aclanthhoooyyy.o..o org/
Giorgos Vernikos, Brian Thompson, Prashant Mathur, and Marcello Federico. Embarrassingly easy document-level MT metrics: How to convert any pretrained metric into document-level metric. In Philipp Koehn, Loic Barrault, Ondřej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Tom Koemi, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa,
17

---

Published as a conference paper at COLM 2025
Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Marco Turchi, and Marcos Zampieri (eds.), Proceedings of the Seventh Conference on Machine Translation (WMT), pp. 118-128, Abu Dhabi, United Arab Emirates (Hybrid), December 2022. Association for Computational Linguisties. URL https::////.......p.. / ac antho ogy- org/ 2.22 wmt -1 -1.6/ 6/.
Tu Vu, Kalpesh Krishna, Salaheddin Alzubi, Chris Tar, Manaal Faruqui, and Yun-Hsuan Sung. Foundational autoraters: Taming large language models for better automatic evaluation. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 17086-17105, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18655 18653/v 12224.emmmmmmmain 949. URL https: //a/ccaannttttt awt.aaaaanoooooyy.oooooooooooooa org / 2024 emnlp- ma iin.inn... 949/.
Alex Wang, Kyunghyun Cho, and Mike Lewis. Asking and answering questions to evaluate the factual consistency of summaries. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5008-5020, Online, July 2020. Association for Computational Linguistics. doi: 18.13v53/v// 2020.acl-main.455 URL https: / /tt///alanntttt aawaathhannotoyyyycocc org/ 22/g/2222 2020 acl -main...... 450/.
Sheng-Lun Wei, Cheng-Kurn Kuang Wu, Hen-Hsen Huang, and Hsin-Hsi Chen. Unveiling selection biases: Exploring order and token sensitivity large language models. In Lun-Wei Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Findings of the Association for Computational Linguistics: ACL 2024, pp. 5598-5621, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 1...1865//1/2222 0i120244.indinns-aa..33... URL https: aal/cclaannttt aawaaaaaaaantooooooooooooooooooa org/ acl. 333/.
Changrong Xiao, Sean Xin Xu, Kunpeng Zhang, Yufang Wang, and Lei Xia. Evaluating reading comprehension exercises generated by LLMs: A showcase of ChatGPT in education applications. In Ekaterina Kochmar, Jill Burstein, Andrea Horbach, Ronja Laarmann-Quante, Nitin Madnani, Anaïs Tack, Victoria Yaneva, Zheng Yuan, and Torsten Zesch (eds.). Proceedings of the 18th Workshop on Innovative Use of NLP Building Educational Applications (BEA 2023), pp. 610- 625, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. (0.18553/v1 2023.bea- wetthea--...... 1.52. URL https: /tt.t:p:///alaatttt /accaaaaccoooyyyoo.oo org / 2023 bea-1.52..
An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 15/ 2024.
18

---

Published as a conference paper at COLM 2025
A
Additional Prompts
Prompt for Question Generation (with optional candidates)
System: are an expert teacher tasked with designing reading comprehension questions.
User: Generate question-answer pairs to verify translation accuracy. Each answer should be key phrase, concept, or entity from the original passage that is important in context and could help detect potential errors mistranslations in the alternatives.
The questions and answers answer is preserved. The questions should be diverse and cover different aspects the passage. Answer the format:
be strictly in English while ensuring that the meaning of the
Q: <question1>
A: <answerl>
Q: <question2>
A: <answer2>
Original Passage: {src passage}
Alternative Passages: {alternatives)
Question-Answer Pairs:
Figure 7: Prompt used generating question-answer pairs the reference-free setting. Text gray optional.
B Metric Baselines
We compare against the following metrics:
CHRF (Popović, 2015): A character n-gram-based F-score metric that measures translation similarity. It has been demonstrated to be strong baseline, outperforming lexical metrics such as BLEU (Mathur et al., 2020).
COMET (Rei al., 2022a): reference-based neural metric based XL--Robeera (Conneau et al., 2020), trained on a large collection of direct assessments of translation quality collected over the past decade as part of the WMT evaluation campaigns.
COMET-QE (Rei et al., 2022b): A reference- version of COMET based on InfoXLM-R (Chi al., 2021) that predicts a quality score given source text and its translation. Similar to reference- based COMET, it is trained on direct assessments.
XCOMET (Guerreiro et al., 2024): An improved version of COMET, leveraging larger XLM- Roberta backbones and trained to predict both sentence-level direct assessments of translation quality and token-level MQM error span annotations. We compare against both the reference-based and reference-free free variants.
GEMBAMQM (Koemi & Federmann, 2023a): An LLM-based metric that uses fixed three-shot prompt to detect MQM еттог spans and compute a quality score in reference- free manner. We use the same LLM backbone TREQA for fair comparison.
MTEQA (Krubinski et al., 3021): A system-level QA-based evaluation metric that uses keyphrases extracted from the reference to generate questions and computes lexical score comparing the answers to those generated using MT candidates.
C Additional Details for Ablation Analysis
In this part, we provide additional details on the analysis conducted in Section 5, regarding coverage of translation errors. As mentioned the main text, we use two complementary metries: (I) exact
coverage via recall and (II) partial coverage using CHRF (Popović, 2015). Particularly, we compute:
(I) the exact coverage via recall, as the total number of error spans covered by at least one of the available answers, over the total number of error spans (in the entire dataset).
19

---

where A Published as a conference paper at COLM 2025
GPT-4o
SONNET-3.5
QWEN2.5-728-IT
14
Number questions
12
15
10
15
20
10
15
20
Number Errors
Number of Errors
Number of Errors
Figure 8: Number of questions vs. number of errors for all models: TREQA (no cands) achieves the highest correlation with number of errors.
(II) the partial coverage (PC) using CHRF, by measuring the maximum similarity of each error span with the available answers of the k-th passage, simş a chrF(a, et) and then averaging
E NN sim!
over the total number of annotated error spans in the entire dataset, PC
Σ N+
denotes the set of candidate-ansee answers of the k-th passage, et the i-th error span of the k-th passage, & the number of error spans in the k-th passage, and K the total number passages.
D Additional Ablations
D.1 Type and Number of Questions
We report the number of questions against the number of expert-annotated errors across all settings and models in Figure 8. Interestingly, the candidate-agnostic, reference- seference-based variant of TREQA exhibits the strongest correlation. Close behind are the candidate-awar QAG models, both the reference-based (TREQA) and the reference-free (TREQA-QE) variants. In contrast, the candidate- agnostic QE variant consistently lags behind behind-except in the case of SONNET-3.. where it performs comparatively better. These findings suggest that access to the reference provides strong and useful signal for identifying potential errors even for relatively weak models, while relying on the source text alone may not offer sufficient
for effective error detection.
In Figure 9, we present an analysis of the types of questions generated by all TREQA models, categorized according to the ontology proposed by prior work (Cao Wang, 2021). To ensure consistency and reliability in question type generation, we prompt GPT-40 using the instruction template provided and validated by Trienes al. (2024). Across all TREQA variants, observe consistent distribution in question types: questions assessing conceptual understanding and the consequences of actions or events dominate across models. This trend suggests that the models effectively follow the given instruction, successfully generating questions that probe key ideas and implications presented the passages.
A more fine-grained comparison reveals that candidate-awar variants of TREQA generate relatively higher proportion of concept- and consequence-basee questions, while producing fewer judgmental questions, those that require subjective or personal interpretation from the answerer. This shift indicates that incorporating candidate information steers the models toward generating questions that more grounded in the passage content, reducing reliance open- open-ended opinion-base prompts. Such behavior further supports the utility of candidate-and aware generation in focusing question types toward more diagnostic and error-relevant content.
20

---

Published as a conference paper at COLM 2025
TREQA twish cantitaless
TREQA (without
candiSated) TREGA DE (witbaa candidateo)
MIOLEDURAL
Question Types
Figure 9: Distribution of question types for all models using ontoglogy defined by Cao & Wang (2021).
LITERARY
PAR3HUMAN TREQA-QE
QA System
Cands? TREQA
DE-
FR FR-+EN
RU-+NN
QWEN2.5-72B-IT
MULTI
64.4
46.7
45.3
46.7
57.3
SINGLE
60.0
54.4
43.3
43.3
57.3
QWEN2.5-7B-IT
MULTI
61.1
55.0
49.3
42.7
52.7
SINGLE
62.8
53.3
43.3
43.3
60.0
Table Impact of using candidate-aware generation when using single vs multi-candidate question generation and different QA systems on Contrastive Accuracy across datasets.
D.2 Is TREQA useful in single-candidate setup?
While TREQA by default conditions the generation of questions simultaneously on all candidates available in the first step, we also explore an alternative approach which question generation is conditioned upon single candidate at time. This alternative enables closer examination of the specific errors of each candidate, and pooling the questions generated by multiple candidates can facilitate comparative analysis. However, performing such a comparative analysis requires making N separate calls to the question generation module one for each candidate being evaluated which makes this method less efficient. Table reports the contrastive accuracy obtained all evaluated datasets (LITERA ARY and PAR3HUMAN] when using SINGLE E vs MULTI candidate-aware question generation. While MULTI average outperforms SINGLE it leverages comparative context across candidates to produce more discriminative questions, the results show that the questions generated conditioned individual candidates can also provide weaker but useful signal when ranking alternative passages.
D.3 How does a weaker QA system paired with a strong QAG performs?
Table shows the contrastive accuracy on the evaluated dataset using relatively smaller QA system (QWEN2.5-7B-I7) As hypothesized earlier, QWEN2.5-7B-IT performs competitively with SWEN2.5-72B-1 and in most QE settings even better. We speculate that smaller models like QWEN2.5 5-T-IT may be better suited to handling straightforward factual questions, especially those that rely on surface-level cues lexical mismatches. In contrast, QWEN2.5-72B. shows clear gains when paired with candidate-aware question generation, where questions often demand deeper reasoning or nuanced error interpretation.
D.4 How well do current LLM-based QA systems perform?
We benchmark the performance of the QWEN2.5-72B-I7 QA system on human-authored English questions sourced from the XQUAD dataset (Artetxe et al., 2020), using prompts described in Section 2. To assess answer correctness comprehensively. both English (denoted as QUIP) and non-Emn non-English languages (denoted as QUIP (xx)). Additionally, we report several context-agnostic metrics commonly employed in QA evaluations, including CHRF, F1,
evaluate responses using contexts in
21

---

Published as a conference paper at COLM 2025
PROMPT
CONTEXT
QUIP (EN)
QUIP (XX)
CHRF
F1
LR
Extractive
EN
4.76
81.78
0.77
3.083
Crosslingual
EN
4.41
56.20
0.46
6.851
4.16
4.18
46.19
0.36
8.172
3.99
4.10
42.95
0.32
8.686
ZH
4.22
4.22
46.49
0.36
8.085
Table 3: Several answer correctness metrics for QA Evaluation using QWEN2.5-72B-IT.
Deratty
8.10
175
125
75-
8.04
25
Relrevrre-aace
Refevvrco-tree00
Referwarc Rooodd
Reforence- Free/OF)
Figure 10: Answer complexity measures for TREQA when using cands.
and Length Ratio (LR), detailed in Table 3. Our findings demonstrate that the EXTR ACTIVE prompt notably outperforms the CROSSLINGUAL prompt in English contexts. Specifically, using the latter results in significant performance degradation, evidenced by sharp decrease in CIRF and F1 scores by 25.58 and 0.31, respectively. This drop in performance is accompanied by increased Length Ratio (LR), reflecting higher complexity and reduced precision in generated responses. Furthermore, performance cross-lingual QA tasks consistently falls behind the all evaluated languages. These results highlight the difficulties in effectively achieving cross- poos-linggul QA capabilities. While improved prompt engineering could partially address this issue, we believe that improved trosslligqua/mmutilingu pretraining remains crucial to further progress.
baseline across
D.5 How complex are the answers obtained from the QAG/QA system?
We assess the linguistic complexity of answers by examining both the regenerated and original reference answers, well the answers obtained from the candidate passages. Specifically quantify answer complexity by measuring the total number of words and the density of name entities present in each sentence. Figure 10 presents the results. Crucially, the regenerated reference answers produced by the QA system, exhibit significant differences between the siterence-based and reference- free (QE) metrics in terms of both length and named entity density. Particularly, we observe that across reference-fie free metrics, the regenerated answers are characterized by excessive verbosity and minimal amount named entities, when compared to original reference answers those produced from the QAG system and those derived from the candidate passages. On the contrary, under reference-beet bassed metrics, the regenerated answers are considerably less verbase and more similar to candidate-ded derived answers, regarding both length and the inclusion of named entities. These findings, in conjunction with the observations discussed in the preceding paragraphs, underscore the challenges associated with achieving effective cross-lingual capabilities especially in scenarios where access to the reference passage is unavailable.
Mcasured by the spaCy toolkit: https://gi http on/spayy.
22


Reviews:
[{"review_id": "KjpaMxfjYn", "reviewer": "Reviewer_zwtQ", "summary": "The paper presents a method for MT quality evaluation and quality estimation based on automatic question answering. Questions (and answers) are automatically generated from source (and reference) and the method compares whether the same questions can be answered also based on the test translations. The results should competitive performance on two datasets (Literary dataset and PAR3HUMAN).", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The idea itself is not novel but the novel part is using LLMs to generate the answers. The main experiments are accompanied by several additional experiments, ablation studies, and qualitative analysis.", "reasons_to_reject": "Limited evaluation. The experiments are done on two specific datasets only. There area other resources that could have been used to provide a better and more complete picture about the performance of the method (WMT resources for evaluation and QE), including those based on direct assessment.  The evaluation is also limited in terms of languages, the target language is always EN. \nNo attention is paid to systems level evaluation -- but this is given the choice of datasets.\n\nThe questions are limited to those that are extractive. There are good reasons for that but doing a contrastive experiments not limited to extractive questions would be very interesting.", "questions_to_authors": "I miss technical details of the experiments, such as the number of generated questions per sentence/paragraph and distribution of QA results. And some detailed analysis of the generated questions (though I appreciate the ablation studies in Appendix D) -- e.g how detailed are the questions.", "extra_scores": {}}, {"review_id": "ofKYGg2gls", "reviewer": "Reviewer_qzD2", "summary": "This work proposes an alternative framework for evaluating machine translation (MT) systems through a question answering (QA) task rather than some form of direct string comparisons between the MT outputs and reference translations. The framework involves asking one set of LLMs to generate questions based on the source text (potentially with the reference translation and candidate translations as additional inputs), then asking another set of LLMs to answer those questions based on each candidate translation, and finally evaluating the accuracy of the candidate answers against ground-truth answers. \n\nAn empirical study was carried out to assess the proposed framework. The evaluation is thorough: multiple alternative MTE metrics were considered, including another LLM-based metric; multiple datasets were used, covering several source languages; several experimental conditions were tried (with and without reference, with and without candidates). The results show the proposed metric TREQA to be comparable to another LLM-based metric, GEMBAMQM for the evaluative task (accuracy of matching expert preferences between translation pairs). While the evaluative task makes it possible to directly compare with a wide range of existing metrics, it feels somewhat distant from the original goal of coming up with a more extrinsic oriented evaluation task.   \n\nAdditional analytical experiments were conducted to better understand the performance of TREQA, but the findings were either not too surprising (candidate-aware generation seems to help, sampling from larger models seem to help) or still vague (ways in which TREQA fail). \n\nAspects of the paper are not as well described. For example, the paper stated that answer correctness is measured using LERC, but we do not know how varying this choice might impact the overall success of the framework. The application of the main experimental metric (contrastive accuracy) is also not really described (a reader of the paper may not be able to recreate the procedure of the experiment).", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed framework is well-motivated. It would be nice to have a more robust and in-depth way to evaluate MT beyond sentence-level comparisons with reference translations. \n\nThe overall experimental methodology is thorough.", "reasons_to_reject": "The validation of the proposed framework ultimately still seems tied to some sort of human preference at the sentence-level. \n\nThe performance of the framework seems to depend on several factors (choice of QAG model, choice of QA system, choice of an answer correctness metric), and it is not sufficiently clear how someone might make these choices for some other use scenario (where the target language is not English, for example).", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2QHyJr2gPt", "reviewer": "Reviewer_unEd", "summary": "The paper presents a framework to evaluate MT using QA, with questions generated from elements in the source or reference translations. It addresses the problem that MT evaluation metrics are still mostly at the sentence-level. \n\nThe framework uses LLMs to generate questions that target the content of either the source or reference texts. These questions are conditioned on the set of translation hypotheses, so they focus on the differences between the hypotheses. The framework then evaluates how well answers generated from each translation align with those derived from the source or reference texts. \n\nThe evaluation always concerns the into English direction, but context can be non-English. \n\nThe results are mixed but generally positive and competitive.\n\nI really like that the paper is a serious attempt at an extrinsic evaluation metric for MT. However, there are definitely discussion points related to the method (see questions below). In addition, clarity can be improved. I had some trouble understanding some central points like translation directions for the QA and where and how the question-answer pairs are being generated.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- proposal to do extrinsic evaluation of MT that is ambitious and new \n\n- strong potential to evolve into a more robust and comprehensive framework with further development", "reasons_to_reject": "- Potential concerns regarding methodological choices (some could be a matter of clarification):\n\n\ti) the idea is to evaluate if people with access to the translations have the same understanding as people with access to the sources. So why choosing literature as test? Wouldn't a genre with more objectively measurable success/fail outcomes, for example instructions, be better suited? The literary genre seems open to interpretation and hence as confounding factor.\n\n\tii) document-level metrics for MT seem relevant but missing", "questions_to_authors": "- are hallucinations a concern in the QA generation step? \n\n- in section 2.2, lines 155-162, I found the explanation of the answer regeneration somewhat unclear. Is it the case that both an LLM and a separate QA system are used to generate answers? If yes, does this mean that the questions and answers are produced by different systems or is it that different models integrated within the QA process? This becomes somewhat clearer in section 5 (I think), but I'd recommend to clarify this part earlier in the text. Basically, I'd offer a very simplified explanation of the method parts in the introduction. On the same topic, about paragraph at line 262, could you elaborate on what do you mean \"true comprehension\" in this context?\n\n- what's the reasoning for not including a document-level metric among the baselines? \n\n- looking at table 1, it seems that GembaMQM is somewhat better, what is TREQA's added value? \n\n- In section 5, paragraph at line 246, are these answers that include an error and explicitly say there's an error in the translation? As a potential user, how do you know the translation has an error? \n\nSmall stuff:\n\n- Figure 6 is difficult to read when the paper is printed", "extra_scores": {}}, {"review_id": "a0xhSFkO4l", "reviewer": "Reviewer_oatu", "summary": "This work proposes a new metric to evaluate paragraph-level machine translation. Specifically, TREQA, is an extrinsic metric that evaluates the quality of translations using question-answering. The metric relies on using LLMs to generate questions, and also to check for answer consistency between the source and the translation. The proposed metric is meta-evaluated on two dataset for paragraph-level MT.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. One of the main advantages of the proposed metric is that it is completely unsupervised — it does not require any human judgements to train the metric and only relies on consistency of the answer.\n\n2. The paper considered multiple source languages (Polish, French, Japanese, Russian etc.) which is great. \n\n3. The qualitative error analysis to understand where and how the metric fails (line 273 and Figure 6) is really useful.\n\n4. The set of baselines used in the main experiment and the range of LLM-backbones considered seem substantially large and diverse.", "reasons_to_reject": "1. Conceptually unclear if using QA to evaluate translation makes as much sense as using QA to evaluate other things such as factuality (which prior work did). For translation, it is possible that the candidate contains errors (e.g. grammatical errors, only important entities are translated correctly etc) in a way that QA can still work while translation quality being poor. In general whether this framework makes sense would critically depend on the kind of questions generated — from what I understand, most questions are ‘local’ but what is really required for paragraph-level MT is questions requiring information across multiple sentences. Currently there is no analysis / evidence to support that questions have this property.\n\n    a. For example, if a translation of an English sentence contains ‘Paris’, one might still answer ‘where’ questions correctly from a poor translation. But more complex questions would be harder to answer.\n\n    b. For things like evaluating factuality of summaries (which prior work did) such ‘local’ questions make a lot of sense. \n\n    c. More broadly, I’m concerned that the metric the way it is implemented right now might be easy to ‘game’ — get a high score for poor translations.\n\n    d. Maybe combining intrinsic and extrinsic evals could be a good idea?\n\n2. Overall the meta-evaluation results in Table 1 also doesn’t always seem great. For Literary dataset since random chance is 50%, the performance doesn’t seem that much better especially since there are only 180 examples (line 174). Some statistical significance tests could be useful. The proposed metric also doesn’t perform that great on the Par3Human dataset.\n\n3. Some of the claims also seem to be generalized incorrectly – e.g. line 224 states including candidates reduces variance but that is only true for 1 out of 2 dataset — I don’t know if that’s a strong enough signal to make this conclusion.\n\n4. Only the xx-en translation direction is considered now. While the paper does mention a reason for this (can use existing paragraph-level translation judgements for this direction), I think the fact that LLMs are much better at English than other languages makes it unclear if this method will generalize to the other direction.\n\n5. (minor) The general idea of using QA to evaluate translations (or using QA to evaluate other things like factuality in summarization) has been around for a long time, so the main novelty could be somewhat limited — using LLMs to generate the questions seems to be the main difference.", "questions_to_authors": "1. Line 216 – something more actionable on this would be good. If one is using this metric today, how should one select the model for question-generation? Do we always need some expert preferences to do model selection?\n\n2. Footnote 8: ‘depending the’ → ‘depending on the’", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces TREQA (Translation Evaluation via Question-Answering), a framework for evaluating paragraph-level MT quality via question answering. Instead of relying solely on intrinsic metrics trained to mimic human judgments, TREQA employs LLMs to generate and answer comprehension questions about the source or reference texts, using the alignment between candidate and reference answers as a proxy for translation quality.

Reviewers note that revisiting extrinsic MT eval with LLM-powered QA is new and well-motivated and has strong potential. The empirical parts of the paper have been highlighted as thorough. One reviewer calls the paper a serious attempt at an extrinsic evaluation metric for MT. The paper acknowledges variance in LLM generations, issues with hallucinations, and positional bias, but provides thoughtful ablations and discussions.

Some reviewers criticize that the target language is only English. The improvement margins, especially on the small LITERARY dataset, are not always compelling, and statistical testing was absent (though promised for the final version). Some point out that the choice of literature translation is not ideal, and that the evaluation quality depends on the quality of the generated questions.

Overall, the paper introduces an interpretable approach to paragraph-level MT evaluation. Its core ideas are mostly novel, empirically grounded, and interesting to the community.
Decision: Accept