Are Large Reasoning Models Good Translation Evaluators? Analysis and Performance Boost
Authors: Runzhe Zhan
Zhihong Huang
Xinyi Yang
Lidia S. Chao
Min Yang
Derek F. Wong
Venue: NeurIPS 2025 poster ()
Keywords: large reasoning model
evaluation
machine translation
analysis
Primary area: evaluation
TL;DR: We provides the first systematic analysis of large reasoning model as translation evaluator and offer insights in improvements.

Abstract: Recent advancements in large reasoning models (LRMs) have introduced an intermediate "thinking" process prior to generating final answers, improving their reasoning capabilities on complex downstream tasks. However, the potential of LRMs as evaluators for machine translation (MT) quality remains underexplored. We provides the first systematic analysis of LRM-as-a-judge in MT evaluation. We identify key challenges, revealing LRMs require tailored evaluation materials, tend to "overthink" simpler instances and have issues with scoring mechanisms leading to overestimation. To address these, we propose to calibrate LRM thinking by training them on synthetic, human-like thinking trajectories. Our experiments on WMT24 Metrics benchmarks demonstrate that this approach largely reduces thinking budgets by ~35x while concurrently improving evaluation performance across different LRM scales from 7B to 32B (e.g., R1-Distill-Qwen-7B achieves a +8.7 correlation point improvement). These findings highlight the potential of efficiently calibrated LRMs to advance fine-grained automatic MT evaluation.

Are Large Reasoning Models Good Translation Evaluators? Analysis and Performance Boost
Runzhe Zhan'
Zhihong Huang' Xinyi Yang Min Yang" Derek F. Wong
Lidia S. Chao
'TLPCCT Lab, Department of Computer and Information Science, University of Macau
2hennzen Institute of Advanced Technology, Chinese Academy of Sciences
nlp2ct. {runzhe, zhihong, xinyi] Aggmail com
min. yangasia.. ac en, [derekfu, (idiascc edu.mo
Abstract
Recent advancements in large reasoning models (LRMs) have introduced an in- termediate "thinking" process prior to generating final answers, improving their reasoning capabilities on complex downstream tasks. However, the potential of LRMs as evaluators for machine translation (MT) quality remains underexplored. We provides the first systematic analysis of ..M--asa-judge in MT evaluation. We identify key challenges, revealing LRMs require tailored evaluation materials, tend to "overthink" simpler instances and have issues with scoring mechanisms leading to overestimation. To address these, we propose to calibrate LRM thinking by training them on synthetic, human-like thinking trajectories. Our experiments on WMT24 Metrics benchmarks demonstrate that this approach largely reduces thinking budgets by ~355x while concurrently improving evaluation performance across different LRM scales from 7B to 32B (e.g.. Diistill--wen---- achieves a +8.7 correlation point improvement). These findings highlight the potential efficiently calibrated LRMs advance fine-grained automatic MT evaluation.
Code: O ThinMQM
Models: ThinMQM Models
Introduction
The development of reliable automated metrics that accurately mimic human judgments of translation quality vital for advancing machine translation (MT) models [1, 3, 4. 5]. However, modeling the comprehensive and complex "evaluation process" inherent in human assessment is challenging. Previous research has explored various paradigms to capture this process. from deterministic rule- based metrics [6, 7] and embedding spacie similarities [8] to end-to-end neural networks [9, 10]. More recently, the rise of large language models (LLMs) as a judge [11, 12] has marked substantial leap forward. LLMSs provide a convenient mechanism for customizing the evaluation process through both natural language, marking a significant advancement in evaluation methodology [13, 14, 15).
Despite these advancements, assessing translation quality is rarely a simple "0-1" binary matching task. often requires deliberate. analytical cognitive effort, which is akin to "System 2" thinking [16], even for human annotators [17]. This suggests that emerging large reasoning models (LRMs) [18], which enhance reasoning capabilities by generating intermediate "thoughts" before producing final solutions, similar to human reflective thinking, may offer stronger foundation for modeling the complex process of MT evaluation. While LRMs have often demonstrated remarkable performance boosts in solving mathematical and scientific challenges [19, 20], their potential as judges, i.c., LRM-as-a-judge in the specific context of MT evaluation remains largely unexplored.
"Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

To provide a comprehensive understanding and practical insights into the application of LRM-LS judge for MT evaluation, this paper the first to systematically address the following key questions: 1) How do current LRMs perform in MT evaluation lasks when compared to human judgments? 2) What are the specific failure modes or inefficiencies encountered when applying LRMs to MT evaluation? 3) How can we develop an efficient and effective alignment strategy to tailor LRMs specifically for MT evaluation?
To this end, we employ LRMs in MT evaluation under the multidimensional quality metrics (MQM) framework [21, 22], following previous tfateoo-thhe-ar LLM -u---ss--juuge design principles [13, 14]. We then perform a meta-evaluation and analysis across wide range of model series and sizes, including DeepSeek- R1 671B [19], QwQ 32B [23], and RI-Distilled models [19]. Through careful examination of critical factors in designing LRM-as-ar jadgge our findings reveal several key insights. We observe a disagreement between L. and LRMS in their perception of evaluation materials, with strong LRMs benefiting more significantly from alignment with human-like evaluation protocols. Our results also suggest need to rethink the design of multi-stage scoring mechanisms, as there are pitfalls related to overestimation problems and ambiguous contributions from auxiliary scoring models. Moreover, concerning thinking behaviors, we reveal that LRMs are not always efficient in allocating their thinking budget and tend towards "overttinkiig" for easier evaluation instances.
Furthermore, based on these findings, we propose a simple yet effective method steer LRM perform Thinking-calibratel calibrated MQM (ThinMQM) scoring by training them on synthetic evaluation trajectories designed to mimic human-like scoring rubrics. Experimental results on the most recent WMT24 Metries benchmarks [24, 25] show that this method can largely reduce thinking budgets by approximately <55 while improving the evaluation performance of LRMs different model scales (notably, R1-Distill-Qwen
achieves a +8.7 correlation point improvement), as shown in Figure 1. Follow-up analysis reveals that such trajectory steering calibrates the scoring distribution and reduces the overestimation problem. These results align with our analysis, revealing substantial potential in developing LRM-as-a-judge for MT evaluation, yet highlight the necessity of controlling thinking budgets and performing careful calibration to fully realize this promise.
WMT24 METRICS AvG. CORK.
WMM24 METRICS AvC. CORR.
Gembo ESA
Gemini-2. Pro
ThinMON (Clun)
ThiMMQN (Oun)
70
D 871B
70
ThieMONS-25 (uun)
Da 6718 (5)
DeepBock RI 6713
DeegGeeb-R1
OwQ2-2211 (8)
65
Disil Llama Seep/Seeee0) 6718
65
RR Liina 40 (R)
OZ 18 MR
Ri-Dsalll- Llama (1
Disill Liana 58 Devll Qwen 7B
60
-i-eelll--ormmmm (R)
Ri-Daatll Liama (S)
Datill Qwess 79
60
Ri-Deell-Qwmmm77 ()
50
O
Qwen
55
RL-Daall-Quee--BB1
55
10
102
10a
10
102
10a
Model Size (# billion parameters, log scale)
AVG. Thinking Budget (# Tokens)
(a) Comparison of model sizes.
(b) Comparison of thinking budget of LRMs.
Figure 1: Performance comparison of various evaluation models on WMT24 metrics tasks. The ThinMQM models (Ours) achieve strong performance with competitive efficiency. "S, R. J" denote different evaluation inputs: Source-only, seference--asedd and Joint evaluation materials, respectively.
2 Preliminaries
2.1 Paradigms in Machine Translation Evaluation
Formally, traditional automatic evaluation metrics for machine translation can be abstractly defined as mapping m from a set of input materials X to an output score y. The metric m may take the form of a rule-based text matching algorithm [6, 7, 26, 27] or a parameterized model [8, 9, 28, 29, 10]. Typically, the input set X, i.c., the materials required for machine translation evaluation, comprises three elements: the machine translation hypothesis h, the reference translation r, and the source text in the original language. Evaluation is generally based on combinations of these elements, falling into two main categories: reference-based and reference-free. In the reference-based setting, the model hypothesis is compared directly with the reference translation (Ref.). i.c., XRet. (h, r). In

---

(a) Framework of L LRM-as-judge in machine translation evaluation.
MOM Scale
Okay, let's start by comparing the transiation the source and First, the opening line
(Critical) -25 pts
Few-shot
(Major) -5 pts (Minor)
pts
Source
Critical: no-erro
Translation
Major: acuracyliistranistton Minor: no-error
[Rubrics)
Scores
Reference
(0-100)
LLMS/LRMS
Inputs
LRMs
Thinking
Error Annotations
Scoring
X Unclear Inputs Contributions
X Overestimation & Overthinking
(b) Diagnose and analyze the key issues and calibrate thinking process.
ThinMQM: THINking calibrated MQM Scoring
Input Materials
thinking: Unify Annotation Scoring
Direct Answer
[Src.] (Ret. (Translation)
First, need to analyze translation and classify the errors.
Tass (Error Severity: (error spans)
Score: (v)
Basod on above
Zow Finally, can calculate the final score: (-n)---e ---------- (-n) (v2
Tailored Materials
Post-training Synthetic, Human-Aligned Evaluation Trajectory
Figure 2: Overview of our research framework. (a) We decompose the general LRMs-as-judee pipeline for MT evaluation and identify key issues. (b) Guided by analytical findings, we propose ThinMQM, which establishes a more effective reasoning process.
the reference-free setting, evaluation is performed by comparing the hypothesis with the source text (Src.), i.e., Xsre. = (h, s). Additionally, it is also possible to use all three components jointly (Jonit.), i.., XJoint (h, s, r). for scoring. In contrast, the emerging LLM-as-a-judge paradigm does not strictly conform to an end-to-end X + y paradigm, as it is not a parameterized regression model but rather a generative language model. Although such models can provide more detailed output information, such as explanations for the given scores, in most cases the desired score must be extracted post hoc from the model outputs using rule-basee extractor or auxiliary scoring models.
2.2 MQM Framework and Related Work
Even we can list all the possible combinations of evaluation materials. modeling detailed scoring process remains a challenge. Earlier automated metrics focused on providing a single score value I6, 30], but they lacked consideration or transparency in aligning with human evaluation process, especially for neural metrics. Previous practices in collecting human annotations centered around direct assessment scores [31, 32], neglecting establish fine-grained, unified framework for score annotation. MQM is a professional scoring framework for translation quality assessment, designed to guide multi-dimensional evaluation of translations, including aspects such as fluency, terminology, and style [17, 22]. Typically, human annotators perform error span annotations, assign severity levels (e.g., major or minor), and finally, the score is aggregated by certain severity weights. In most cases, a major error incurs penalty of -5 points, while minor eтror results in a deduction of -1 point (or -0.1 for fluency errors). Critical errors, such as non- con-translation, are generally penalized by -25 points. The total score for an instance is computed by aggregating the penalties of all identified errors.
To develop more fine-grained machine translation evaluation metrics, recent research has focused on constructing automated methods aligned with MQM scoring. This has emerged as the primary evaluation task in the recent WMT leaderboard [33, 25J. The end-to-end approach [34, 9, 35, 29] essentially constructed upon pre-trained language models by finetuning on MQM scoring data. The LLM-as-judge approach to perform MQM scoring largely mirrors human annotation procedures: first, the model is instructed with MQM guidelines, and then it either extracts error span annotations to compute a score, or directly outputs quality score. Currently, one of the most widely adopted and effective methods is the GEMBA series [13, 14] based on prompting GPT models. Its GEMBA-MQM variant further leverages in-context learning (ICL) [36. 37] by using three-shot demonstrations to assist in the evaluation process. We adapt it to LRM scope and conduct analysis in this paper.
3 Understanding LRM Behaviors in MT Evaluation
Figure illlsstrtes our research framework. In this section, we aim to address the first two research questions: how well LRM performs in MT evaluation and what failures occur in the practice.
3

---

3.1 Experimental Setup
Methodology As introduced earlier, following the successful SOTA practice LLMs, we replicate GEMBA-MQM methodology on LRM as the basis for our analysis. We first instruct the LRM to annotate error spans in the translation according to the MQM framework, categorizing them into critical, major, and minor errors. Based on these error spans, we then apply rule-based scoring mechanism to compute the final score for each translation. The penalty scheme for each type of error follows the approach described in Section 2.2.
Building on this, we conduct experiments with all possible input material combinations Xsre.vRef..JJing to investigate the influence of evaluation materials in Section 3.2. Meanwhile, the demonstrations used in ICL follow the same format those in the GEMBA-MQM. is worth noting that, since the MQM variant in the GEMBA series is reference-free for reference- based setups (i.e., Ref. and Joint.), we supplement the demonstrations with reference information and adjust the prompt templates accordingly. Detailed prompts are provided in the Appendix C.2. We report the main results based on rule-based scoring mechanism and will discuss alternative model-based scoring methods (the logic is the same as the ESA prompting variant of GEMBA) in Section 3.3.
Models Setups We employ several mainstream LRM models across different sizes and architectures, including Deepseek-Re R1 67 671B, QwQ 32B, as well as distilled variants of RI trained via knowledge distillation: RI Ri-Distill-LLaMA performance on complex reasoning tasks. Due to computational constraints, we are unable to deploy the open-source version of the R1 model locally and instead access via API for experiments. All other models are deployed using the vL.MM framework 2. The decoding parameters are set as follows: temperature is set to 0.6, top, and top: is set to 0.95, 20. Our selection of DeepSeek-RI is driven by its transparency of reasoning trajectories. In contrast, other frontier models, such as 03 and Gemini-2.5 Pro, do not expose their internal reasoning processes. This limitation makes them unsuitable for fair and fine- grained analysis. Nevertheless, report the evaluation performance of Gemini-2.5 Pro in Section 4.2, but exclude it from the analytical sections.
and RI-Distill-Qwen 7B. These models have demonstrated strong
Data We chose the WMT24 Metrics Shared Task [24] for our evaluation data in order to prevent potential issues with data contamination [38, 39, 40J. We confirmed that the release date for the WMT24 MQM data is after the knowledge cutoff for the models aforementioned. This task involves assessing the correlation between the evaluation models' scores and the human expert MQM scores, both the system and segment levels. The WMT24 Metrics task includes three language pairs: English-German (En-De), English-Spanist (En-Es), and Japanese-Chinese (Ja-Zh), with around 20 machine translation systems for each pair.
Meta-E Evaluation Metrics on key metrics such as system-level soft pairwise accuracy [41] (SPA) and tie-calibrated segment-level pairwise accuracy [42] (Accty). Specfically, for SPA, it can be formally expressed as:
We used the same meta-evaluation settings as WMT24 official, focusing
N - N-1 N-1
SPA
Σ
(1-
(1
p.j
2
j=it1
where N is the number of systems. pi is the p-value that system i is better than system j based
on human judgments, and ph is the same based on metric scores. (N) normalizes over all pairs. Higher values of these metrics indicates stronger agreement between human and metric rankings. We used the MTME' to maintain consistency with the official calculations. We also report permutation-based significance testing with 1,000 resampling times [42] in comparison experiments.
3.2 Impact of Evaluation Materials
Contribution Quantification In MT evaluation, since the translation hypothesis h is present in all evaluation scenarios, it is necessary to assess the impact of source and reference information on evaluation performance. While it is feasible to enumerate and experiment with all combinations of
hhttps:/ggithu www.coo///o/o.oc htt.eh/..oooooooolleeeeeeerrrata

---

evaluation materials, denoted as XSrc ..t...eeevvv... vJoint,, quantitatively isolating the contribution of each component remains challenging due to the overlapping presence of source and reference across multiple evaluation settings. To address this, we adopt the Shapley Value [43] as principled measure to quantify the individual contributions of source and reference information to evaluation outcomes across different models and evaluation settings.
Formally, the Shapley Value of the source information o, is defined as:
| - |s 1)! (o(s' U - v(s'))
E
Φ,
[N
s'CN
where N = (s, denotes the set of all materials that may affect evaluation (source and reference r). The translation hypothesis h is always present and thus not part of the combinatorial set. The function v(.) represents the evaluation performance under a specific evaluation setting, which we quantify using system-level and segment-level metries. The set s' refers to all subsets of N excluding 8, i... {0, r). In particular, the case corresponds to an evaluation setting with neither source nor reference (i.c., translation-only). vh) is invalid value as translation-only is not valid input, thus we only approximate it using an available configuration, namely u u(1,, r}), to estimate v(h). Therefore, taking into account the practical constraints of machine translation evaluation, we refer the approximated Shapley Value here as @TT in order to distinguish from the strict definition in Eq.2. The calculation of MT follows analogously.
x System-
Segment-L
All
X10
System-Lavel
Segment-l Lvell evel
All
1.0
0.8
Shapley Value
Shapley Value
0.6
-2
02
0.0
RID 70
RIDAL 88
QwQ2
RI 17
RID-Q
QwO 32n
an 6730
RID
RID-L 8B
QwQ
RI 871
RIDQ
RID- 50
QwQ 32B
RJ 671B
RID-, RDD-Q 78
Q+Q 328
RJ 1717
RID-Q.7 7B
RID-I
QwQ
RJ 8718
Positive Contribution
Negative Contribution
Positive Contribution
Negative Contribution
(a) Source contribution.
(b) Reference contribution.
Figure 3: Shapley value analysis of input contributions of evaluation materials at different evaluation granularities. "RID-L./Q." refers to the -1-Distill-Llamee Qwen models.
Results and Discussion The results presented Table along with the significance tests, intuitively suggest that using either source or reference information as evaluation materials represents the most effective choice for LRM-based evaluation. However, this observation contingent on model scale. The illustrative results shown in Figure 3, derived from Shapley Value &MT analysis, reveal pattern: for smaller-seat LRMs (7/8B), source information is detrimental to evaluation quality, whereas reference information contributes positively. This trend is reversed in larger LRMs, such as QwQ 32B and R1 671B, where source information becomes beneficial and reference information less so.
Previous work [44] on the LLM-as-a-judge observed that LLMs tend to become "lost in source" during MT evaluation, regardless model size. However, our findings suggest that this phenomenon may not generalize to the LRM-as-a-jun judge setting. This distinction is plausible, given that LRMs typically have stronger reasoning capabilities on complex tasks [45, 19, 20] compared to general- purpose LLMs. Earlier LLMs may have lacked the capacity to effectively model the cross-lingual relationships between source and translation.
As for the observed adverse impact of reference information on LRM-based evaluation, we hypoth- esize two possible factors. First, MQM human annotations are inherently reference-free. focusing solely on the source and translation without relying on references. Second, the quality of the refer- ence itself significantly affects the correlation of automatic metries with human judgment, as prior work [46, 33] has pointed out, "BLEU (or Metrics) might be guilty, but reference not innocent". These findings highlight the need for scale-aware design evaluation setups: the choice of evaluation materials should be informed by the capabilities and limitations of the model size in question.
5

---

Table 1: Comparison of different evaluation material setups. The highest value in each model is bolded, and indicates results that are significantly better (p < 0.05) based on permutation tests.
En-De
En-Es
Ja-Zh
Avg.
Materials
SPA (%)
Acceq
SPA (%)
Acceq
SPA (%)
Accep
SPA
Accez
All
Deepseck-RR
671B
Src.
82.1
47.4
77.8
68.0
90.4
46.8'
83.4
54.1
68.8
Ref. Joint.
80.4
43.0
67.3
68.0
88.6
43.6
78.8
51.5
65.2
82.4
46.6
75.6
68.0
91.1
44.0
83.0
52.9
68.0
QwQ 32B
Src.
79.8
46.8'
76.1
68.0
91.9
46,91
82.6
53.9
68.3
Ref.
84.2
42.9
68.7
68.0
94.3
43.5
82.4
51.5
66.9
Joint.
81.7
44.2
72.2
68.0
92.5
44.3
82.1
52.2
67.2
R RI-Distill-LLama 8B
Src.
72.3
42.9
68.0
74.2
43.5
70.8
51.5
61.1
Ref.
71.8
42.9
78.5
68.0
84.7
43.5
78.3
51.5
64.9*
Joint.
72.9
42.9
65.1
68.0
78.7
43.5
72.2
51.5
61.9
R1 R1-Distill- Qwlln 7B
Src.
52.6
42.9
53.9
68.0
68.3
43.5
58.3
51.5
54.9
Ref. 67.3*
42.9
61.0
68.0
83.8
43.5
70.7
51.5
61.1
Joint.
58.4
42.9
57.0
68.0
86.1
43.5
67.2
51.5
59.3
3.3 Pitfalls of Scoring Mechanisms
Table 2: Effect of changing the rule-based scoring weights on average correlation Avg. A
Signed:
Sign
R1:6778
0.87
0.09
1.00
0.08
0.16
1.00
0.38
0.22
Model
0.89
0.5
peoaue
(Aceeg SPA) metrics.
0.05
0.09
0.76
0.89
1.00
0.025
Model
Src.
Joint.
Ref. A
RID
Q.7.Q
0.00
0.74
0.48
0.04
0.69
0.56
0.06
1.00
0.69
671B +0.60
+0.60
0050
40.23
Sec.
Ref.
[ontt
Buc.
Join
Sec.
But.
Joint
QwQ 328
+0,20
+0.50
40,30
40.33
Qwen-32B Scoeing
RID-L.8B
-1.20
-0.20
-0.00
-0.73
Figure 4: Significance testing of the contribution of the auxiliary
RID-Q.7B
-0.10
-0.30
-1.10
-0.50
scoring model (Qwen-2.5 32B). Test results (p 0.05) are high- lighted in distinct colors and scales.
Arg A 0.13 +0.15 0.53
Dilemma in Post-Scoring Since LRM-based evaluators generate descriptive outputs in autore- gressive manner, an additional critical factor influencing scoring is how error spans are extracted and scored. In this work, consider two commonly adopted paradigms: re-scoring. The rule-based scoring introduced before aggregate scores under MQM standard, offer- ing transparency. In contrast, the model-based scoring paradigm allows for additional stage of reflection but lacks transparency. Naturally, for a rule-based scorer, the robustness to rule changes worth noticing. Furthermore, when auxiliary models used for scoring, it becomes difficult to disentangle whether observed improvements stem from the LRM or auxiliary scorer.
and model- based
We first investigate using the LRM itself to score enror spans annotated via the GEMBA-ESA protocol. The results show no improvement over the rule-based scorer; in fact, we observe a slight performance drop (e.g., QwQ 32B yields mean performance 68.3 + 68. 1), along with significantly higher inference costs. Next, we employ an auxiliary model (Qwen-2.5 32B) to perform the same scoring procedure. In this setting, find that the meta-evaluation results of LRM closely align with the that of the auxiliary model itself. This raises a critical question: are the observed gains attributable to the LRM outputs, simply the auxiliary model? To address this, conduct statistical significance testing and in-depth comparison of score distributions.
Results and Discussions To answer above question, we perform significance testing between scores produced by the "L_MM + auxilary" setup and those from the auxiliary model alone. The results in Figure 4 demonstrate that the re-scoring process using an external model fails to provide clear attribution regarding the source of evaluation performance. This finding highlights the need to either

---

enhance the scoring capabilities of the original LRM or adopt transparent, rule-based extraction and scoring mechanisms. Figure further compares the distributions of two scoring paradigms.
The results reveal a persistent overestimation issue in model-based evaluation. 'he instances that hu- man annotators consider error-free are still judged as erroneous by the LRM. This indirectly confirms that the improvements brought by auxiliary mod- els do not fundamentally resolve the shortcomings of LRM--ased scoring and are insufficient to mit- igate the overestimation problem, requiring for more human- centric evaluation process that mirrors human judgment rules for better correlations.
15000
Human
12000
Rule Based
9000
Model-Based
6000
Score
Figure Distribution of evaluation scores across different scoring paradigms.
Another remaining concern with a rule-based scorer how sensitive the results are to the choice of scoring scheme. To study this, conducted an experiment using an alternative severity weighting scheme (i.e., -3/-2/-1). Table 2 reports the average change A across all correlation metrics. We observe that although adjusting the weights does slightly shift the absolute correlation values, the differences are modest. A likely explanation is that meta-evaluation metrics are primarily sensitive to the rank order of segments. As long as the ordinal structure the penalties is preserved, the rankings remain relatively stable, supporting the robustness of the rule-based scoring approach.
3.4
"Overthinking" Process: When More is Not Better
Thinking Budgots
QuQ 32B (Sec.)
Derpwek 671B
50
Thisking #Talons
1000
Daskking Cuums
Thhinksng Tokeras
4000
Insumett
Comeet
Incernet
Cornet
30
2000
500
1000
[-20, -15) -10) (-10,5) 1-5,
[-20, (15) [-10, [-5, 0]
Model Scale (Nominal)
Human Score Range
Human Score Range
(a) Budget statics across all languages.
(b) Budget allocations across varying evaluation difficulties.
Figure 6: Analysis of thinking budget attribution across model scale and evaluation difficulty. Appendix B.4 includes all the results under different settings.
Thinking Budgets LRMs typically benefit from scaling test-time thinking budget. Intuitively, we investigate whether such scaling is both effective and efficient in MT evaluation. We quantify the thinking budget along two dimen- sions: 1) the number of tokens generated during the reasoning process. and 2) the number of reasoning turns". Additionally, since two of the used LRMs also have corresponding ceneral-purposs LLMss, we also examine their performance assess whether LRM post-training contributes to improved performance".
The
80%
36%
Results and Discussions The results in Figure 6(a) show that reasoning cost not necessarily tied to model size. For example, QwQ, despite not being the most powerful -RM-as-a-judge model, incurs the highest reasoning cost.
335
11%
6/%
Moreover, the thinking budget is also unrelated to instance difficulty. As shown in Figure 6(b), median thinking tokens remain stable across difficulty levels. An exception appears at the extremes of evaluation difficulty, where human scores are either very high or very low. The correctly aligned (i.e., model scores consistent with human judgment) predictions require more effort, while misaligned ones are cheaper. This the only case where the thinking budget shows a hint of rational allocation.
QwQ2326 RID-4200
Figure 7: Com- parison between LRM and its cor- responding LLM.
4 "mmiricalll, we observe that these LRMss use output delimiter across reasoning turns.
We exclude R1 Distill-QQenne 7B from this comparison, its original model not general- perppse LLM.
7

---

On the other hand, we compute meta- veta-evvluation metrics across different languages and evaluation levels, comparing LRMs with their corresponding general-purpose LLMs. Statistical significance testing is used to categorize outcomes as wins, ties, or losses, shown in Figure 7. "Tie" is considered as failed significance test. The results show that LRMs underperform nearly half of the evaluation settings, indicating that current general-purpose LRMs, despite their" "slow "slow-thinking" process, still struggle to consistently enhance evaluation performance.
Improving LRM via Human-Aligned Thinking Trajectory
4.1 Methodology
Motivations Our analysis reveals that standard practices of LLMs in MT evaluation are not universally optimal for LRMs. The unconstrained "thinking" processes within LRMs can be inefficient and may lead to overestimated score. Key insights derived from this analysis include: 1) Reference- free evaluation is preferable for strong LRMs, while reference- baseed evaluation remains suitable for weaker LRMs. 2) Aligning the LRM's reasoning process with specific scoring rubrics is crucial. 3) Extensive budget allocated to LRM "thinking" do not consistently improve performance.
Drawing from previous observations, we introduce Thinking-calibrated MQM (Thin- MQM) scoring method, methodology designed to adapt LRMs to emulate human evaluation process. The core idea is intuitive: generating synthetic data that mirrors the human MQM workflow, thereby calibrating and aligning the LRM's internal thinking process with the pipeline of human evaluation.
ThinMQM
Given human MQM annotations which consist of error spans E = {e,, + En and their associ- ated severity levels L = 42 12............... .n. J. we aim to model the human two-phase evaluation process. This process involves an initial error span annotation (ESA) stage, TESA : X - (E, S), followed by scoring stage based into a concise yet effective structured thinking chain, intended to serve as proxy for human annota- tion steps. The resulting synthesized data, Dsyuth = {(X vitef, [TESA (X), Tscore(TEs. (X))), adheres to the structure shown in Figure 2 (b). The LRM, denoted by Mg. would be post-trained on the dataset D Dayuth, with parameters 0 producing the output sequences. The fine-tuning process seeks to update 0 to 0' by minimizing the cross-entropy loss function LCE over all instances in D Daynthi
rubric, Tecore (E.S) - ScorearQ.- We transform this sequence
0' f- arg min
Σ (X vRof. 0), [TESA (X), (TESA (XF)((())))
0
Deynth
4.2 Experiments
Data We synthesized ThinMQM training data based on the human-annotatee MQM dataset from WMT23, which includes two evaluation tasks: English-German and Chinese-Englinn English. Synthetic data instances were constructed based on the methodology described above and the prompt templates detailed in Appendix C.4. Due an imbalance distribution across the two language pairs, we down- sampled the larger set to ensure balanced training data. The final dataset consists of approximately 5,980 instances per language pair, yielding a total of 11,960 training instances.
Model and Setups To verify the effectiveness of ThinMQM on various model sizes, we fine-tune 7B, 8B, and 32B models. Based on earlier analysis, we adopt reference-based evaluation setup (Ref.) for the 7B and 8B models in both training and inference, while the 32B model employed reference-free setup (Src.). All models are fine-tuned for epochs with a learning rate of le 5. and the total batch size is 32. Other training
are detailed in Appendix C.I.
The results in Table clearly demonstrate that post-training calibration with Thin- MQM significantly improves LRM performance under the same evaluation setups. Specifically, the 7B model shows gains of up +8.7 points in meta-evaluation metrics, while the 32B model achieves a +3.9 points improvement, reaching performance comparable to state-of-the-ar metrics such as XCOMET, despite those relying on training on large-scale MQM data and have different model architectures. Notably, within the LLM/LRM evaluation paradigm, our ThinMQM-32B model achieves superior average performance compared to the baselines, though not necessarily on every individual language pair.
Main Results
8

---

Table Performance comparison of different models. The highest value is bolded, and the second-
best is underlined. denotes significantly better (p 0.05) results based on permutation tests.
Avg.
En-De
En-Es
Ja-Zh
Metric/Model
All
SPA (%)
Acce
SPA (%)
Acceg
SPA (%)
Acceg
BLEU [6]
58.9
73.7
43.1
51.4
68.0
73.6
43.5
COMET-22 [9]
68.9
87.9
48.2
77.9
68.3
81.4
49.6
xCOMET [10]
71.9
90.6
53.0
78.9
68.8
88.9
51.0
GEMBA-ESA [13]
71.1
79.1
50.7
84.0
68.3
90.8
53.9
Gemini P.5-Pro [47]
71.0
82.3
51.2
76.9
68.0
94.8
53.1
Deepseek-RI [19]
82.1
47.4
77.8
68.0
QwQ 32B
68.3
79.8
46.8
76.1
68.0
91.9
46.9
ThinMQM
72.2.3.9
83.2.a.4
52.5.s.7
80.7.4.6
69.2 .12
91.3.0.6
56.1 19.2
RI-Distill D--Dstill-Llama--t 8B
64.9
71.8
42,9
78.5
68.0
84.7
43.5
ThinMQM
70.8,5.
85.5
48.6.5.7
81.3.22
68.2.o2
90.5.5.8
51.0,7.
RI RI-Distill-Qwen-
61.1
67.3
42.9
61.0
68.0
83.8
43.5
+ ThinMQM
69.8.s.7
84.5.172
48.5.5.6
77.8.165
68.0.ao
89.0,5.2
51.347.8
4.3
Analysis
#Oxuuenos
15000
12000
87.6%
9000
QwQQ32R
En-De
12.35%
6000
0,05%
3000
89.50%
25
20
-5
0
En-Es
10.9%%
0.21%
82.43%
FOxurenos
15000
Ja-Zh
17.55% 0.02%
12000
9000
ThinMQM-32B
6000
3000
20%
40%
60%
80%
Share of Misidentified MQM Error Types
15
10
Score
Critical
Major
Minor
Figure 8: Comparison of scoring distribu- Figure Distribution of ThinMQM-322 human judg-
tions between ThinMQM and QwBB ment discrepancies across MQM error types.
Scoring Distribution As shown Figure 8, the primary reason behind the improvement brought by ThinMQM lies in the calibrated scoring distribution Specifically, ThinMQM effectively mitigates the overestimation problem. aligning the model predicted scores more closely with the human MQM distribution, particularly in cases with non-error cases. This finding echoes the earlier observations in Figure 5, indicating that the performance gains from ThinMQM are both meaningful and justified.
Error Typology To analyze cases where ThinMQM-32B diverges from human judgments, we categorize these discrepancies according to the MQM translation error taxonomy (Critical, Major, Minor), as shown in Figure 9. The analysis shows that the largest misalignment arises from Minor- level errors. Moreover, within the Minor category, accwacyynistransllttin accounts for the highest proportion of discrepancies, highlighting areas where future improvements should be targeted.
Efficiency As illustrated in Figure (b), ThinMQM reduced the unnecessary thinking budget while maintaining high evaluation performance. This indicates that post-training alignment not only improves effectiveness but also enhances efficiency. In practice, This represents substantial decrease in the computational cost of LRM-based translation evaluation. For example, when evaluating English-Gerd German translations with QwQ 32B under the vLLM framework using four A100 GPUs, inference time is reduced from 12 minutes per 1,000 examples to 40 seconds on average.
'Please refer to Appendix B.1 for detailed language-specific distributions due space limitation.

---

Table 4: Multi-run evaluation of ThinMQM temperature 0.6.
Table 5: Performance compari- son in Hindi-Chinese MQM.
Each score is presented as Mcangea. over independent runs.
Avg.
En-De
En-Es
Ja-Zh
Model
Sys, p Seg.
Model
All
SPA (%)
Acc,,
SPA (%) Accay
SPA (%)
Accey
XCOMET-XXL
62.5
47.8
ThinMQM 32B
80.7 72.
53.1 ous
80.8,uz
68.9
92.5,c11
55.9,002
ThinMQM 32B
63.4
57.4
ThinMQM 8B
70.4
83.
48.6
81.3
90.3 68.2
01.0.00
ThinMQM 7B
51.3
49.1
ThinMQM 7B
85.4g1 p0.0
48.2.04
77.6
68.1.om om
89.3
51.4.003
ThinMQM 8B
50.3
47.5
4.4
Ablation Study
Stability Figure 10 demonstrates that ThinMQM robust to test-time hyperparameter choices. We evaluate performance under various decoding settings and compare meta- mettrcss scores. For example, QwQ-32B's system-level evaluation is sensitive under greedy decoding, whereas ThinMQM remains stable, nit with only drop the segment level. To further verify stability, conduct three runs at a fixed temperature of 0.6. As shown in Table 4, the low standard deviation confirms that ThinMQM's performance is consistent and not subject to significant random fluctuations. Overall, these results validate our chosen decoding configuration as a broadly robust setup for LRMM-based evaluation.
System-Leee evell
Segment-LLevi
QwQ X2R (Souree)
82.5
SPA
Arcou
77.5
ThinMQM 328 (Sounce)
SPA
50.0
80.0
Generalization To perform a more stringent out-of- distriiiiittionn test a low-resource language pair, we sourced recently re- leased Sindi-Chinese dataset with MQM annotations [48], which was published after our LRMs' knowledge cutoff date. Since this dataset contains translations from fewer than four systems, we system-level Pearson p and Kendall correlation evaluation metrics. As shown in Table 5, ThinMQM demonstrates low-resource scenarios, outperforming the XCOMET-XXL baseline.
Temparatuuee
Figure 10: Performance of dif- ferent models under varying tem- perature setups.
meta-
generalization capabilities under
Prompting Templates To further strengthen the comparison between our proposed ThinMQM and alternative baseline prompting templates, used GPT-4o to paraphrase our GEMBA-MQM prompt and generated three additional variants, which we denote as P1-P3. Figure 11 (details in Section B.3) shows that ThinMQM consistently maintains a per- formance advantage across model sizes. Besides, ad- ditional interesting observations emerge from these results. For LRM baselines, large models (32B) are relatively insensitive to prompt variation, exhibiting only minimal differences in performance. In contrast, smaller models (7-8B) are more sensitive to prompts. However, the resulting fluctuations limited and still do not surpass the ThinMQM.
GEMBAF. F1
F2
F3
WM114 Metrics Avg. Cort.
70
46S
60.1
1477)
98.7
60
KID-Qwwen
RID-Llama
32B
Figure 11: Comparison of ThinMQM with baselines using paraphrased prompts.
5 Conclusions
In this paper, we presented a systematic investigation into I.MMaas-a-judge for machine translation evaluation, exploring their capacity to model the process of MQM assessment task. Our analysis across various LRMs revealed that there is a need to tailor evaluation materials for evaluation and they "overthink" simple instances, exhibiting overestimation biases. To address this, we introduced simple yet effective method of calibrating LRM thinking by training them on synthetic, human- like MQM evaluation trajectories. This approach substantially reduced thinking budgets while improving evaluation performance on WMT24 Metrics benchmarks, primarily by calibrating scoring distributions and reducing overestimation. Our findings demonstrate the potential of LRMs for MT evaluation but highlight the critical need for controlled thinking and careful calibration realize their full potential in translation evaluation, paving the way for future adyancements in developing better LRM-as-22 -ajdae-a-uudge in MT evaluation. Future work will extend evaluation to more diverse languages.
10

---

Acknowledgments
This work was supported in part by the Science and Technology Development Fund of Macau SAR (Grant No. FDCT//000//222//MM, China Strategic Scientific and Technological Innovation Cooperation Project Grant No. 2022YFE0204900), the Science and Technology Development Fund of Macau SAR (Grant No. TDCT/0007//022//KKP) the Science and Technology Development Fund of Macau SAR (Grant No. FDCT/060/202//FF National Natural Science Foundation of China Grant No. 62261 606488) the UM and UMDF (Grant Nos. NYRG-GRG2023-000 D00000OSTTUUSE -GGG2022-001155FSTTTTT FST), and the National Natural Science Foundation of China (Grant No. 62266013). We would like to thank the anonymous reviewers for their insightful comments.
SF2023-00151-F55 EF2023-00090-FF EF2024-00185
Bibliography
[1] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, 2015.
[2] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. N. Vishwanathan, and Roman Garnett, editors, Advances Neural Information Processing Systems 30; Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008, 2017.
[3] Felix Stahlberg. Neural machine translation: A review. Journal of Artificial Intelligence Research, 69:343-418 2020.
[4] Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics,
2020.
[5] Marta Costa-jussà, James Cross, Onur Çelebi, Maha Elhayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. No language left behind: Scaling human-centered machine translation. ArXiv preprint, 0es/220770067222 2022.
[6] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Pierre Isabelle, Eugene Charniak, and Dekang Lin, editors, Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Association for Computational Linguistics, 2002.
[7] Maja Popović. chrF: character n- n-gram F-score for automatic MT evaluation. In Ondřej Bojar, Rajan Chatterjee, Christian Federmann, Barry Haddow, Chris Hokamp, Matthias Huck, Varvara Logacheva, and Pavel Pecina, editors, Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392-395. Association for Computational Linguistics, 2015.
[8] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Eval- uating text generation with BERT. In 8th International Conference Learning Representations, ICLR 2020. OpenReview.net, 2020.
[9] Ricardo Rei, Craig Stewart, Ana Farinha, and Alon Lavie. COMET: A neural framework for MT evaluation. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685-2702. Association for Computational Linguisties, 2020.
[10] Nuno M Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and An- dré FT Martins. xcomet: Transparent machine translation evaluation through fine- aine-griined error detection. Transactions of the Association for Computational Linguistics, 12:979-995, 2024.
[11] Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on ilm-as-a-judge. ArXiv preprint, abs/2411. 15594, 2024.

---

[12] Yann Dubois. Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-conirolled alpacaeval: simple way to debias automatic evaluators. ArXiv preprint, abs/2404.04475, 2024.
[13] Tom Koemi and Christian Federmann. GEMBA-MQM: Detecting translation quality error spans with GPT-4, In Philipp Koehn, Barry Haddow, Tom Koemi, and Christof Monz, editors, Proceedings of the Eighth Conference on Machine Translation, pages 768-775. Association for Computational Linguistics, 2023.
[14] Tom Koemi and Christian Federmann. Large language models are state-of-the-arr evaluators of translation quality. In Mary Nurminen, Judith Brenner, Maarit Koponen, Sirkku Latomaa, Mikhail Mikhailov, Frederike Schierl, Tharindu Ranasinghe, Eva Vanmassenhove, Sergi Alvarez Vidal, Nora Aranberri, Mara Nunziatini, Carla Parra Escartín, Mikel Forcada, Maja Popovic. Carolina Scarton, and Helena Moniz, editors, Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 193-20S European Association for Machine Translation, 2023.
[15] Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, André Martins, Graham Neubig, Ankush Garg, Jonathan Clark, Markus Freitag, and Orhan Firat. The devil is in the errors:
Leveraging large language models for fine-grained machine translation evaluation. In Philipp Koehn, Barry Haddow, Tom Koemi, and Christof Monz, editors, Proceedings of the Eighth Conference on Machine Translation, pages 1066-1083, 1083, Association for Computational Linguisties, 2023.
[16] Keith Frankish. Dual- proless and dual-system theories of reasoning. Philosophy Compass, 5(10):914-95) 2010.
[17] Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. Experts, errors, and context: A large-scale study of human evaluation for machine translation. Transactions of the Association for Computational Linguistics, 9: 9:1460-1474, 2021.
[18] Fengli Xu, Qianyue Hao, Chenyang Shao, Zefang Zong, Yu Li, Jingwei Wang. Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, et al. Toward large reasoning models: survey of reinforced reasoning with large language models. Patterns, 6(10), 2025.
[19] Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, ArXiv preprint, abs/2501. 2025.
[20] Zhong-ZZ Zhi Li, Duzhen Zhang, Ming-Liand Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jic Wang, Xiuyi Chen, et al. From system to system 2: A survey of reasoning large language models. ArXiv preprint, abs/2502.17419. 2025.
[21] Aljoscha Burchardt. Multidimensional quality metrics: flexible system for assessing translation quality. In Proceedings of Translating and the Computer 35. Aslib, 2013.
[22] Arle Lommel, Serge Gladkoff, Alan Melby, Sue Ellen Wright, Ingemar Strandvik, Katerina Gasova, Angelika Vaasa, Andy Benzo, Romina Marazzato Sparano. Monica Foresi, et al. The multi-range theory of translation quality measurement: Mqm scoring models and statistical quality control. ArXiv preprint, abs/2405. 16969, 2024.
[23] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men. Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong I Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. ArXiv preprint, abs/2412 151115 2024.
[24] Tom Koemi, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Chris- tian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, et al. Findings of the wmt24 general machine translation shared task: the Ilm era here but mt not solved yet. In Proceedings of the Ninth Conference Machine Translation, pages 1-46, 2024.
12

---

[25] Markus Freitag, Nitika Mathur, Daniel Deutsch, Chi-Kiu Lo, Eleftherios Avramidis, Ricardo Rei. Brian Thompson, Frédéric Blain, Tom Koemi, Jiayi Wang. al. Are 1lms breaking mi metrics? results of the wmt24 metrics shared task. In Proceedings of the Ninth Conference on Machine Translation, pages 47-81, 2024.
[26] Matthew Snover, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul. A study of translation edit rate with targeted human annotation. Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers, pages 223-231. Association for Machine Translation in the Americas, 2006.
[27] Matthew G Snover, Nitin Madnani, Bonnie Dorr, and Richard Schwartz. Ter-plus: paraphrase, semantic, and alignment enhancements to translation edit rate. Machine Translation, 23:117 127, 2009.
[28] Thibault Sellam, Dipanjan Das, and Ankur Parikh. BLEURT: Learning robust metrics for text generation. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881-7892 Association for Computational Linguistics, 2020.
[29] Juraj Juraska, Mara Finkelstein, Daniel Deutsch, Aditya Siddhant, Mehdi Mirzazadeh, and Markus Freitag. MetricX-23: The Google submission the WMT 2023 metrics shared task. In Philipp Koehn, Barry Haddow, Tom Kocmi, and Christof Monz, editors, Proceedings of the Eighth Conference on Machine Translation, pages 756-767. Association for Computational Linguistics, 2023.
[30] Eirini Chatzikoumi. How evaluate machine translation: review of automated and human metrics. Natural Language Engineering, 26(2):137-161, 2020.
[31] Tharindu Ranasinghe, Constantin Orasan, and Ruslan Mitkov. TransQuest at WMT2020: Sentence-level direct assessment. In Loic Barrault, Ondiej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Alexander Fraser, Yvette Graham, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, and Matteo Negri, editors, Proceedings of the Fifth Conference on Machine Translation, pages 1049-1055, Association for Computational Linguistics, 2020.
[32] Qingsong Ma, Johnny Wei, Ondřej Bojar, and Yvette Graham. Results of the WMT19 metrics shared task: Segment-level and strong MT systems pose big challenges. In Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, André Martins, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Matt Post, Marco Turchi, and Karin Verspoor, editors, Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers), pages 62-90. Association for Computational Linguistics, 2019.
[33] Markus Freitag, Nitika Mathur, Chi-kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thomp- son, Tom Koemi, Frederic Blain, Daniel Deutsch, Craig Stewart, Chrysoula Zerva, Sheila Castilho, Alon Lavie, and George Foster. Results of WMT23 metrics shared task: Metrics might be guilty but references are not innocent. In Philipp Kochn, Barry Haddow, Tom Koemi, and Christof Monz, editors, Proceedings of the Eighth Conference on Machine Translation, pages
Association for Computational Linguistics, 2023.
[34] Chi-kiu Lo. YiSi a unified semantic MT quality evaluation and estimation metric for languages with different levels of available resources. In Ondřej Bojar, Rajen Chatterjee, Christian Feder- mann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, André Martins, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Matt Post, Marco Turchi, and Karin Verspoor, editors, Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers), pages 507-513. Association for Computational Linguistics, 2019.
[35] Yu Wan, Dayiheng Liu, Baosong Yang, Haibo Zhang, Boxing Chen, Derek Wong, and Lidia Chao. UniTE: Unified translation evaluation. In Smaranda Muresan, Preslav Nakov, and Aline
13

---

Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pages 81 8117-8127. Association for Computational Linguistics, 2022.
[36] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng. Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al. A survey on in-context learning. ArXiv preprint, abs/2301.000 2330,,0234, 2023.
[37] Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannanch Hajishirzi, and Luke Zettlemoyer. Rethinking the role demonstrations: What makes in-context learning work? In Yoav Goldberg. Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages Association for Computational Linguistics, 2022.
[38] Shahriar Golchin and Mihai Surdeanu. Time travel in Ilms: Tracing data contamination in large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, 2024.
[39] Simone Balloccu, Patrícia Schmidtová, Mateusz Lango, and Ondrej Dusek. Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source LLMs. In Yvette Graham and Matthew Purver, editors, Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 67-93, Association for Computational Linguistics, 2024.
[40] Yihong Dong, Xue Jiang. Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang, and Ge Li. Generalization or memorization: Data contamination and trustworthy evaluation for large language models. ArXiv preprint, 1tt./2202.15938. 2024.
[41] Brian Thompson, Nitika Mathur, Daniel Deutsch, and Huda Khayrallah. Improving statistical significance in human evaluation of automatic metries via soft pairwise accuracy. In Barry Haddow, Tom Koemi, Philipp Koehn, and Christof Monz, editors, Proceedings of the Ninth Con- ference on Machine Translation, pages 1222-1234 Association for Computational Linguistics, 2024.
[42] Daniel Deutsch, George Foster, and Markus Freitag. Ties matter: Meta eetalevvaluaiing modern metrics with pairwise accuracy and tie calibration. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12914-12929, Association for Computational Linguistics, 2023.
[43] Alvin E Roth. The Shapley value: essays in honor of Lloyd S. Shapley. Cambridge University Press, 1988.
[44] Xu Huang, Zhirui Zhang, Xiang Geng, Yichao Du, Jiajun Chen, and Shujian Huang. Lost in the source language: How large language models evaluate the quality of machine translation. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACI. 2024, pages 3546-3562. Association for Computational Linguistics, 2024.
[45] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, al. Openai ol system card. ArXiv preprint, 11222242.16620, 2024.
[46] Markus Freitag, David Grangier, and Isaac Caswell. BLEU might be guilty but references are not innocent. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 61-71. Association for Computational Linguistics, 2020.
[47] Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, Luke Marris, Sam Petulla, Colin Gaffney, Asaf Aharoni, Nathan Lintz, Tiago Cardal Pais, Henrik Jacobsson, Idan Szpektor, Nan- Jiang Jiang, Krishna Haridasan, Ahmed Omran, Nikunj Saunshi, Dara Bahri, Gaurav Mishra, Eric Chu, Toby Boyd, Brad Hekman, Aaron Parisi, Chaoyi Zhang, Kornraphop Kawintiranon, Tania Bedrax- Weiss, Oliver Wang, Ya Xu, Ollie Purkiss, Uri Mendlovic, Ilai Deutel, Nam
14

---

Nguyen, Adam Langley, Flip Korn, Lucia Rossazza, Alexandre Ramé, Sagar Waghmare, Helen Miller, Nathan Byrd, Ashrith Sheshan. Raia Hadsell Sangnie Bhardwaj. Pawel Janus, Tero Rissa, Dan Horgan, Sharon Silver, Ayzaan Wahid, Sergey Brin, Yves Raimond, Klemen Kloboves, Cindy Wang. Nitesh Bharadwaj Gundavarapu, Ilia Shumailov, Bo Wang, Mantas Pajarskas, Joe Heyward, Martin Nikoltchev, Maciej Kula, Hao Zhou, Zachary Garrett, Sushant Kafle, Sercan Arik, Ankita Goel, Mingyao Yang. Jiho Park, Koji Kojima, Parsa Mahmoudich, Koray Kavukcuoglu, Grace Chen, Doug Fritz, Anton Bulyenov, Sudeshna Roy, Dimitris Paparas, Hadar Shemtov, Bo-Juen Chen, Robin Strudel, David Reitter, Aurko Roy, Andrey Vlasov, Changwan Ryu, Chas Leichner, Haichuan Yang. Zelda Mariet, Denis Vnukov, Tim Sohn, Amy Stuart, Wei Liang. Minmin Chen, Praynaa Rawlani, Christy Koh, JD Co-Reyes, Guangda Lai, Praseem Banzal, Dimitrios Vytiniotis, Jieru Mei, and Mu Cai. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. CoRR, abs/2507.06261, 2025.
[48] Jianhao Yan, Pingchuan Yan, Yulong Chen, Jing Li, Xianchao Zhu, and Yue Zhang. Benchmark- ing gpt-4 against human translators: A comprehensive evaluation across languages, domains, and expertise levels. arXiv preprint arXiv:2411.13775 2024.
15

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The first two research questions were answered in Section 3, and the last one was answered in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
16

---

Justification:
We discuss the limitations of this work in the Appendix A.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specifiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texi system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: For the purpose of approximating Shapley Value, provided justifications based on the evaluation scenarios used in the MT evaluation.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have introduced the details of data, models and hyper-parameters for all the experiments.
Guidelines:
17

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We will attached the code in supplemental materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
www.ssmssssssoossisooooooooooiii for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //n httpssscc/pubbcccuuuccceeeo
/CodeSubmissee ioabbbmmssiiin ssssnsoliccyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
18

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We have included the details of data, models and hyper- pyper--parmeeers for all the experiments in this paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Yes. We conducted statistical significance test for all comparison results and report them in the Table.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The compute infrastructure is described in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
19

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics attpss//nneut cctppublic//tiicetuuuiiiii
Answer: [Yes]
Justification:
We have followed the NeurIPS Code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: It is a foundational research in large reasoning model.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The research conducted in this paper poses no such risks.
20

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We followed the licenses for existing assets.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We tend to use Open Source License.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
21

---

Answer:
[NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification:
We reported it in the submission system.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non-standard components.
Please refer to LLM policy (https: //w/rnurip...// / for what should or should not be described.
22

---

Appendix
A Limitations
Our proposed calibration method relies on synthetic thinking trajectories designed to "human- and "fiddings-rriven. However, these synthetic datasets may not fully capture the diversity of human cognitive processes during MT evaluation. Additionally, since WMT24 includes MQM human ratings for only three language pairs, previous benchmarks have faced risks of data contamination. Our evaluations were primarily conducted using the WMT24 benchmarks, which may not represent all language pairs or domains equally. Lastly, this work follows an "understanding and then improving" approach. While we focus on analyzing the behavior of the LRM, the current calibration method primarily targets the efficiency of the thinking process (reducing "overthinking") and calibrating scoring distributions. More nuanced aspects of the reasoning process, such as the LRM'ss ability to consistently identify specific error types with fine granularity, may require more targeted advanced alignment techniques.
B
Supplementary Details
B.1 Language-specific Scoring Distributions
Figure 12 presents all the scoring distributions when evaluating instances of different languages.
English-German
English-Spanish
English-Chinge
6000
Human
5000
QwQ-32B
4000
-5
-10
6000
Human
5000
ThinMQM-328
4000
3000
2000
-10
-5
-10
Score
Score
Score
Figure 12:
Distribution of MQM scores for QwQ-32B (top row) and ThinMQM-32B (bottom row) compared to human evaluations across different language pairs.
B.2 Data Contamination Prevention
To ensure the integrity of our evaluation and prevent data contamination, we implemented the following rigorous measures. For LRMs evaluated, we rigorously verified that their official knowledge cutoff date or public release date precedes the release date of our evaluation benchmarks. This chronological separation guarantees that the models were not exposed to the test data during their original training phase. Our synthetic training data, ThinMQM, is derived from the WMT23 dataset. The source data utilized for the synthesis of the synthetic set was finalized prior to the public release of the WMT224 MQM evaluation benchmark. This temporal order ensures overlap between our training data and the final test set. To provide full transparency, we detail the relevant dates for all major components used in this study Table 6.
B.3 Supplementary Ablation Results
Effect of ICI. Demonstrations We further analyze the effects ICL on the baseline model in Table 7 using QwQ-32B model. The results indicate that ICL is generally beneficial, improving the
23

---

Table 6: Knowledge cutoff and release dates for all models, data sources, and benchmarks used in this work. This chronology confirms the prevention of data contamination.
Component
Type
Release Date / Knowledge Cutoff
WMT24 MQM
Evaluation Benchmark
Oct 4, 2024
Hindi-Chinese Expert MQM
Evaluation Benchmark
Nov 26, 2024
QwQ 32B
LRM
Sep 19, 2024 (Qwen 2.5 Base)
RI-Distill-L 8B
LRM
Dec. 2023 (Llama 3.1 Basc)
U.Disstill-wwer 7B
LRM
Sep 19, 2024 (Qwen 2.5 Base)
WMT23
Training Data Source
Aug 10, 2023
Table 7: Effects of ICL. The highest value in each block bolded.
En-De
En-Es
Ja-Zh
Avg.
Model
SPA (%)
Acce
SPA (%)
Acce
SPA (%)
Acce
SPA (%)
Acce
All
QwQ 32B
Joint
81.7
44.2
72.2
68.0
92.5
44.3
67.2
82.1
52.2
w/o ICL
80.6
42.9
68.4
68.0
90.2
43.5
65.6
79.7
51.5
Src. 79.8
46.8
76.2
68.0
91.9
46.9
68.3
82.6
53,9
w/o ICL
78.0
44.8
76.5
68.0
92.6
45.7
67.6
82.4
52.8
Ref.
84.2
42.9
68.8
68.0
94.2
43.5
66.9
82.4
51.5
+ w/o ICL
74.8
42.9
64.0
68.0
90.0
43.5
63.9
76.3
51.5
average score for both the Joint and Src. settings, with the Src. variant achieving the best overall performance. The effect on the Ref. setting is more nuanced; while ICI. significantly boosts the average SPA (%) (82.4 76.3), has discernible impact the Acce, score, resulting in identical average score of 51.5 for both configurations. These results confirm that ICL. is generally an effective strategy for improving overall model performance.
Table 8: Details of Training configuration.
Hyperparameter
Value
Batch size per device
2 for 7B/SB, 1 for 32B
Gradient accumulation steps
Learning rate
1.0 10-5
Training epochs
4.0 cosine 0.1
Learning rate scheduler
Warmup ratio
Mixed precision (bfloat16)
Enabled
Detailed Results of Prompt Variation As shown in Table the ThinMQM model family still establishes a strong performance baseline when changing the prompt templates of basclines, out- performing all other tested model variants across all the scales. Our investigation into prompt sensitivity for the QwQ and RI-Distill models reveals inconsistent effects. For the QwQ 32B model, performance is relatively stable, with prompts P1 and P3 matching the GEMBA prompt baseline. Conversely, for the RI-Distill- -waenl--ween/ 7B model, prompt P1 provides notable improvement, boosting the average score from 61.1 to 62.5. Prompt P2, however, consistently degrades performance across all models. Most strikingly, for both the 8B and 7B models, the Acci, scores remain completely static regardless the prompt, suggesting that while prompt engineering can influence SPA (%), fails to improve Acc', for these models, supporting the choice of GEMBA prompting template.
24

---

Table 9: Performance comparison of baseline using different prompts. The highest value in each model is bolded and the second- best is underlined.
En-De
En-Es
Ja-Zh
Avg.
Model SPA (%)
Accey
SPA (%)
Acce,
SPA (%)
AccEy
SPA (%)
Aeci eq
All
ThinMQM 32B
83.2
52.5
80.7
69.2
91.3
56.1
85.1
59.3
72.2
QwQ 32B (Gemba P.)
79.8
46.8
76.1
68.0
91.9
46.9
82.6
53.9
68.3
9 w/
77.8
47.4
79.3
68.0
89.8
47.4
82.3
54.3
68.3
9 w/ P2
77.0
46.9
74,4
68.0
93.7
48.4
81.7
54.4
68.1
4 w/ P3
79.3
46.9
78.2
68.0
90.9
46.4
82.8
53.8
68.3
ThinMQM 8B
85.5
48.6
81.3
68.2
90.5
51.0
85.8
55.9
70.8
RID-L. 8B (Gemba P.)
71.8
42.9
78.5
68.0
84.7
43.5
78.3
51.5
64.9
+ w/ Pl
74.5
42.9
72.3
68.0
85.7
43.5
77.5
51.5
64.5
4 w/ P2 71.1
42.9
65.5
68.0
84.3
43,5
73.6
51.5
62.6
w/ P3 70.2
42.9
72.2
68.0
83.3
43.5
75.2
51.5
63.4
ThinMQM 7B
84.5
48.5
77.8
68.0
89.0
51.3
83.8
55.9
69.8
RID-Q. 7B (Gemba P.)
67.3
42.9
61.0
68.0
83.8
43.5
70.7
51.5
61.1
- w/ P1
70.8
42.9
70.0
68.0
79.6
43.5
73.5
51.5
62.5
4 w/ P2
63.1
42.9
58.5
68.0
82.4
43.5
68.0
51.5
59.7
w/ P3
66.5
42.9
69.6
68.0
81.9
43.5
72.7
51.5
62.1
B.4 Scoring Distribution of Different Evaluation Difficulty
Figure 13 presents all the scoring distributions when evaluating instances varying difficulty level.
C Supplementary Setups
C.1 Training Setups
We train the 7B/8B models using 4 A100 GPUs and the 32B model using 8 A100 GPUs. To enhance training efficiency, we utilize the DeepSpeed (Zero3) framework' for offloading. The settings are detailed in Table 8.
25

---

Desp 671B (jin!!
Despoeck RI 6718 (nc)
Deepseck Ri 67LB (ec)
#Thimbing Takens
6000
SThiking Takens
3000
Connec
-0000
#Thinking
2000
2000
[-0 -511-11, -10) (-10.-5
[1
-5)1-15 (-10.-5)
[-30 -1501-15, -100 (-10.-5)
1-8
Human Score Ranga
Human Score Range
Human Score Range
QuQ 328 (jtet))
QuQ (su))
QwQ 320 (ee)
#Theking TAken
#Thaaking Tokens
6000
#Thokknng Tobens
6000
2000
2000
<0
212 250 35, 120 [110]
[5.0]
20
1a0 15) 130 E-1 5) [5.0]
[ 2511 15, 1.0 [:10] <0
[5.00
Euman Score Ranga
Human Score Range
Euman Scome Range
RID+ lama 68
RITS Jama aRt (ref)
RITH Jima (onr)
#Thekknng takers
STinking Tokers
-000
#Tiikingg
1000
1000
20
12 250 a5. 130 [100
[5.00
20
220
151
120 (100 (0
[5.00]
21 251 as. 1:0 [-10]
15.00)
Human Score Range
Human Score Range
Human Score Range
7B
RIDD-Qeen 7% (nel)
R113Qwwen (err)
#Thinknng lakens
#Thinking Tolkrr
-0000
6000
4000
#Thinking
4000
2000
1000
2000
+20
1-20
-15]1-15 (-1d)-5)
[5.
-220
[:20
-1531-18 -120 (-10.55
[-.
+20
1-21
-1531-15 -190 [-1d 15.00
Muman Score Range
Huuman Score Range
Muman Score Range
Figure 13: Distribution of thinking tokens under different evaluation difficulty.
26

---

C.2 LRM-as-a-judge Prompt
For different prompting templates, we present only one specific evaluation setup as a demonstration. For the others, we simply remove or add some information.
L-M-ag-a-judge Prompt (Adapted from GEMBA-MQM. Joint. Setting)
(Source Language} source:
(Source Text}
(Target Language} human reference:
[Reference Text}
{Source Language} translation:
[Translation Text}
Based on the source segment, human reference and machine translation surrounded with triple backticks, identify error types in the translation and classify them. The categories of errors are: accuracy (addition, mistranslation, omission, untranslated text), fluency (character encoding, grammar, inconsistency, punctuation, register, spelling), style (awkward), terminol- ogy (inappropriate for context, inconsistent use), non-translation, other, or no-error. Each error classified as one of three categories: critical, major, and minor. Critical errors inhibit comprehension of the text. Major errors disrupt the flow, but what the text is trying to say is still understandable. Minor errors are technically errors, but do not disrupt the flow or hinder comprehension. Strictly output error classification results in this format:
Critical:
[error type]-[error__s spans] (one per line, no-error if empty)
Major:
[error cerror_typee_ccrrr_roo [ (one per line, use no-error if empty)
Minor:
Leror__type--errrrspppaa (one per line, no-error if empty).
C.3
Auxiliary Scoring Model
LLM Post-scoring Prompt (Adapted from GEMBA-ESA Ref. Setting)
Given the translation from (Source Language} to (Target Language} and the annotated error spans, assign a score on continuous scale from 0 to 100. The scale has following reference points: O="No meaning preserved", 33=""Sme meaning preserved", 66="Most meaning preserved and few grammar mistakes", up to 100="Perfect meaning and grammar". Score the following translation from [Source Language} source: "'(Source Text)"" {Target Language} translation: " "[Translation Text]" Annotated error spans:
"" "(ESA Details]**
Score (0-100):
C.4 ThinMQM Template
ThinMQM Template (Sre. Setting)
INPUT:
{Source Language} source:
{Source Text}
[Target Language} translation:
(Translation Text)
Based on the source segment and machine translation surrounded with triple backticks, identify error types the translation, classify them and give a score. The categories of enrors are: accuracy (addition, mistranslation, omission, untranslated text), fluency (char-
27

---

acter encoding, grammar, inconsistency, punctuation, register, spelling), style (awkward), terminology (inappropriate for context, inconsistent use), non-translation, other, or no-error. Each errоr is classified one of three categories: critical, major, and minor. Critical er- rors inhibit comprehension of the text. Major errors disrupt the flow, but what the text is trying to say still understandable. Minor errors technically errors, but do not disrupt the flow or hinder comprehension. For each error span, please deduct correspond- ing points based following reference points: -25="Cirtical", -5= -I="Minor", hall=Wiioo//aens//unututtooooooo 1"", O="No-error".
OUTPUT:
<think> (n Okay, let's tackle this translation quality assessment task, First, need to analyze the translation and classify the emror..
Critical: (Critical Error ESA}
Major: {Major Error ESA}
Minor: {Minor Error ESA}
Finally, can calculate the final score: {Score Calculation Process}
<\think>
Score: (Final Score}
28


Reviews:
[{"review_id": "jnpiJTd1cY", "reviewer": "Reviewer_edhE", "summary": "This paper is the first systematic evaluation of large reasoning models (LRMs) as the judge for evaluating machine translation tasks. The authors systematically analyze LRM performance in MT evaluation and identify key insights:\n\n1) Smaller LRM models (7B–8B) benefit more from reference-based inputs, while larger ones (32B–671B) perform better with source-based information. This contradicts prior LLM findings and suggests that LRM design and scale fundamentally alter how evaluation context is processed.\n2) Although reference-based evaluation is popular, it may hurt LRM accuracy, especially at scale, due to noise or low quality in reference translations. Human MQM annotations are reference-free, so introducing references may create a mismatch with human judgment standards.\n3) Switching from rule-based to model-based re-scoring (using auxiliary models like Qwen-2.5 32B) increases computation but doesn't significantly improve alignment with human evaluations. Worse, the improvements often stem from the scorer rather than the LRM, muddying attribution.\n4) Larger LRMs often generate more tokens and reasoning steps (higher “thinking budget”), but this does not consistently correlate with better evaluation accuracy. Surprisingly, correct judgments require more effort only in edge cases, and LRMs still underperform general LLMs in nearly half of tested scenarios.\n\nTo address these issues, the paper proposes Thinking-calibrated MQM (ThinMQM) scoring, which involves training LRMs on synthetic, human-like thinking trajectories using supervised fine-tuning. Experiments on WMT24 Metrics benchmarks demonstrate that this approach reduces thinking budgets while improving evaluation performance across various LRM.", "questions": "1) How does changing the rule-based scoring schema change the correlation with human judgement? For example instead of -25, -5, and -1 if you use -3, -2, -1 or -100, -50, -1 how does that affect your evaluation and correlation with human judgement?\n\n2) You are using a quite high temperature, 0.6 for decoding, which means more randomness in the experiments, and less reproducibility. Is there any reason that you didn’t use temperature 0 or 0.1 instead?\n\n3) Why does your model perform better than XCOMET on Ja-Zh tasks but not on others? Do you think the reason is that XCOMET does not have the Chinese and Japanese knowledge or is there any other reason?", "limitations": "YES", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths: \n\n1) This paper is the first systematic evaluation of LRM as the evaluators on the machine translation tasks, which provides a lot of interesting insights.\n2) The paper is very well written and clear, make it easy to understand and follow for even less familiar to the topic researchers,\n3) This paper studies each concept in a very detailed evaluation setup and with many observations, that does not leave any questions around the topic unanswered.\n\nWeaknesses:\n\n1) The results suggest that XCOMET, an Encoder only model that is trained with a backbone encoder of size 3.5B is performing better than the proposed trained LRM method in tasks when one side of the translation is English. Additionally, since this is an encoder only model it is cheaper than LRM models as well. \n2) The paper does not study more powerful commercial LLMs such as GPT-O3 or Gemini 2.5 Pro. It would be interesting to see how they will perform in these tasks and what are the problems with those frontier LLMs.", "was_revised": true, "final_justification": "The authors responded to all my comments and have resolved my concerns", "rebuttal": "Thank you for your valuable feedback and insightful questions. Below are our point-by-point responses.\n\n> **Weaknesses 1:** The results suggest that XCOMET, an Encoder only model that is trained with a backbone encoder of size 3.5B is performing better than the proposed trained LRM method in tasks when one side of the translation is English. Additionally, since this is an encoder only model it is cheaper than LRM models as well.\n\nFirstly, we would like to respectfully clarify a crucial difference in the **training data** used. Our method was trained on ~**12k** synthetic reasoning trajectories. In contrast, XCOMET [1] utilizes much larger datasets: the WMT 17-20 Direct Assessment (DA) dataset with **1M samples** across 36 languages, and an additional **176k samples** from WMT20-22 and other MQM-annotated sources.\n\nRegarding research focus, our primary contribution is to analyze the challenges and potential improvements of using Large Reasoning Models (LRMs) for MT evaluation, rather than solely pursuing SOTA performance.\n\nInspired by your comment & Q3, we conducted an additional experiment on a **low-resource language pair** to further investigate the advantage of LRM backbones. On a most recent publicly available Hindi-Chinese MQM-annotated dataset [2], our ThinMQM model outperforms XCOMET-XXL, as shown in Table below. This suggests that while our LRM approach has a larger parameter count, it may offer valuable insights into out-of-distribution generalization and better performance on low-resource language pairs.\n\n| Model       | Segment-Level Pearson $\\rho$ | Segment-Level $\\tau$ |\n| :---------- | :--------------------------: | :------------------: |\n| XCOMET-XXL  |             62.5             |         47.8         |\n| ThinMQM 32B |           **63.4**           |       **57.4**       |\n| ThinMQM 7B  |             51.3             |         49.1         |\n| ThinMQM 8B  |             50.3             |         47.5         |\n\n\n------\n\n> **Weakness 2:** The paper does not study more powerful commercial LLMs such as GPT-O3 or Gemini 2.5 Pro. It would be interesting to see how they will perform in these tasks and what are the problems with those frontier LLMs.\n\nWe agree that evaluating frontier models like GPT-o3 or Gemini-2.5 Pro would be very interesting. However, the APIs for these models do not offer **transparent reasoning trajectories**, which are essential for the type of fair and detailed analysis central to our study. For instance, the Gemini 2.5 Pro API only returns **synthesized** reasoning trajectories [3]. Our choice of DeepSeek-R1 was therefore motivated by its transparent reasoning trajectories, making it a more suitable model for our analysis.\n\nMoreover, MT evaluation is a token-intensive task, requiring the assessment of multiple system outputs for each source sentence. Even with the relatively affordable DeepSeek-R1 API, our experiments cost nearly $800.\n\nHowever, to provide a more direct comparison, we have run an additional experiment with Gemini 2.5 Pro during the rebuttal period. As shown in the table below, its performance is superior to the GPT-4-driven GEMBA but still falls short of our ThinMQM method. We will add this result to the paper in the future.\n\n| **Model**      | **Avg. (All)** | **En-De (Sys)** | **En-De (Seg)** | **En-Es (Sys)** | **En-Es (Seg)** | **Ja-Zh (Sys)** | **Ja-Zh (Seg)** |\n| :------------- | :------------: | :-------------: | :-------------: | :-------------: | :-------------: | :-------------: | :-------------: |\n| ThinMQM 32B    |    **72.2**    |      80.8       |      53.2       |      81.1       |      68.8       |      93.3       |      55.7       |\n| GEMBA-ESA      |      71.1      |      79.1       |      50.7       |      84.0       |      68.3       |      90.8       |      53.9       |\n| Gemini-2.5 Pro |      71.0      |      82.3       |      51.2       |      76.9       |      68.0       |      94.8       |      53.1       |\n\n------\n\n> **Q1:** How does changing the rule-based scoring schema change the correlation with human judgement? For example instead of -25, -5, and -1 if you use -3, -2, -1 or -100, -50, -1 how does that affect your evaluation and correlation with human judgement?\n\nThis is an insightful question. We ran an experiment to test a different scoring scheme (i.e., -3/-2/-1). In Table below, we averaged the change in correlation metrics across all languages and evaluation levels. \nWe found that while changing the weights does affect the absolute correlation scores, the differences are not very substantial.\n\n*Note: $\\Delta$ = New scale $-$ MQM scale*\n\n|                              | R1 671B | QwQ 32B | R1-Distill-Llama 8B | R1-Distill-Qwen 7B |\n| ---------------------------: | ------- | :-----: | :-----------------: | :----------------: |\n| Avg. $\\Delta$ (Acc.$^*_{eq}$ + SPA) | +0.23 $\\uparrow$   |  +0.33  $\\uparrow$  |       -0.73  $\\downarrow$        |       -0.50 $\\downarrow$       |\n\nA possible reason is that meta-evaluation metrics are primarily sensitive to the **rank order** of the segments. As long as the ordinal relationship of the penalties is preserved (i.e., major errors receive a larger penalty than minor errors), the rankings remain relatively stable, thus ensuring the scoring's reasonability.\n\n------\n\n> **Q2:** You are using a quite high temperature, 0.6 for decoding, which means more randomness in the experiments, and less reproducibility. Is there any reason that you didn’t use temperature 0 or 0.1 instead?\n\nOur reasons for this choice are threefold:\n\n1. **Recommended Practice.** This temperature setting (=0.6 or >0.6) is often the recommended setting for complex reasoning tasks with LLMs and has been used in prior work [4-6].\n2. **Our Empirical Results.** Our preliminary experiments showed that lower temperatures (e.g., 0 or 0.1) led to a noticeable degradation in performance on this task (Figure 8b, QwQ-32B). We have also included significance tests in the paper for all comparison experiments.\n3. **Additional Multi-run Experiments.** To explicitly address the concern of randomness, we ran our ThinMQM model 3 times at temperature=0.6. The results below show a very small standard deviation, indicating that the randomness has a minimal impact on the final outcome.\n\n*Note: (Mean ± Std. of 3 runs)*\n\n| **Model**   | **Avg. (All)** | **En-De (Sys)** | **En-De (Seg)** | **En-Es (Sys)** | **En-Es (Seg)** | **Ja-Zh (Sys)** | **Ja-Zh (Seg)** |\n| :---------- | :------------: | :-------------: | :-------------: | :-------------: | :-------------: | :-------------: | :-------------: |\n| ThinMQM 32B |  72.0 ± 0.003  |  80.7 ± 0.026   |  53.1 ± 0.006   |  80.8 ± 0.002   |  68.9 ± 0.002   |  92.5 ± 0.011   |  55.9 ± 0.002   |\n| ThinMQM 8B  |  70.4 ± 0.004  |  83.1 ± 0.021   |  48.6 ± 0.003   |  81.3 ± 0.020   |  68.2 ± 0.001   |  0.903 ± 0.013  |  51.0 ± 0.001   |\n| ThinMQM 7B  |  70.0 ± 0.002  |  85.4 ± 0.011   |  48.2 ± 0.004   |  77.6 ± 0.002   |  68.1 ± 0.001   |  0.893 ± 0.004  |  51.4 ± 0.003   |\n\n------\n\n> **Q3:** Why does your model perform better than XCOMET on Ja-Zh tasks but not on others? Do you think the reason is that XCOMET does not have the Chinese and Japanese knowledge or is there any other reason?\n\nWe investigated the training data for XCOMET and confirmed that it does contain Chinese and Japanese data (English-centric), although not specifically from Japanese-Chinese MQM evaluation. Therefore, we believe XCOMET's weaker performance on this pair is not due to a complete lack of knowledge but may stem from: \n\n- a) The Ja-Zh task representing a possible **Out-Of-Distribution** scenario for XCOMET model, testing its generalization limits. \n- b) A potential **imbalance** in the multilingual capabilities of its **XLM-R backbone**.\n\nNotably, our model was also **not trained on any Ja-Zh MQM data**. Its performance likely comes from the superior multilingual and zero-shot generalization capabilities of its underlying backbone model.\n\n------\n\n**References**\n\n[1] Guerreiro, N. M., Rei, R., Stigt, D. V., Coheur, L., Colombo, P., & Martins, A. F. (2024). XCOMET: Transparent machine translation evaluation through fine-grained error detection. *Transactions of the Association for Computational Linguistics*, *12*, 979-995.\n\n[2] Yan, J., Yan, P., Chen, Y., Li, J., Zhu, X., & Zhang, Y. (2024). Benchmarking gpt-4 against human translators: A comprehensive evaluation across languages, domains, and expertise levels. *arXiv preprint arXiv:2411.13775*.\n\n[3] https://ai.google.dev/gemini-api/docs/thinking#summaries\n\n[4] https://huggingface.co/Qwen/QwQ-32B#usage-guidelines\n\n[5] Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., ... & He, Y. (2025). Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. *arXiv preprint arXiv:2501.12948*.\n\n[6] Abdin, M., Agarwal, S., Awadallah, A., Balachandran, V., Behl, H., Chen, L., ... & Zheng, G. (2025). Phi-4-reasoning technical report. *arXiv preprint arXiv:2504.21318*."}, {"review_id": "VmgeSlczmB", "reviewer": "Reviewer_EpwK", "summary": "The paper delivers the first comprehensive examination of large reasoning models (LRMs) as machine-translation judges, showing that while their “slow-thinking” paradigm can model the MQM evaluation process, they also require carefully chosen inputs, often “over-think” easy segments and systematically over-estimate quality. Building on these insights, the authors introduce ThinMQM—a post-training scheme that feeds LRMs synthetic, human-aligned error-annotation + scoring trajectories—which realigns the model’s reasoning, cuts inference-time “thinking” tokens by about 35 × and boosts human-correlation on the WMT-24 Metrics benchmark by up to +8.7 points across 7 B–32 B models, matching or surpassing state-of-the-art neural metrics without auxiliary scorers .", "questions": "The paper attributes the +3.9 – +8.7 point gains to ThinMQM but does not disentangle which design choices—trajectory length, explicit ESA tokens, or post-training itself—drive the lift. (More ablation studies would be nice.) \nThe efficiency story (35 × fewer “thinking” tokens) is promising yet lacks a cost–quality curve.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The paper has solid empirical design because it study evaluates four LRMs ranging from 7 B to 671 B parameters under a unified GEMBA-MQM protocol, reporting both system- and segment-level SPA/Acc with 1 000-sample permutation tests, which lends statistical rigor. Substantial gains and efficiency - ThinMQM lifts correlation by up to +8.7 points while slashing the “slow-thinking” budget ~35 × and cutting wall-clock inference from 12 min to 40 s per 1 k sentences. The writing is easy to follow-the introduction explicitly lists three research questions, and Figure 2 walks the reader through the evaluation pipeline and ThinMQM calibration steps, making the workflow flows well. First systematic look at “LRM-as-a-judge.” The paper is, one of the earliest comprehensive study of reasoning-tuned models for MQM evaluation, and it introduces ThinMQM—synthetic, human-aligned reasoning trajectories to calibrate scoring.\n\nThinMQM is tuned on only ~11.9 k synthetic instances covering two language pairs, raising questions about robustness beyond WMT24’s three test pairs. There are some clarity issues - there is no explanation in fig.2's caption. Moreover, the novelty in terms of using LRMs for judging the quality of a response is questionable, because there are already many papers on this. While the thin-trajectory idea is neat, the fine-tune-on-synthetic-CoT recipe resembles recent chain-of-thought distillation work; novelty lies more in application domain than in method.", "was_revised": true, "final_justification": "My concerns are well addressed and would be happy to raise my score slightly. Thank you for the experiments", "rebuttal": "Thank you for your valuable and constructive feedback. We appreciate your insights and believe they will strengthen our paper. Below are our responses.\n\n------\n\n> **Weakness 1:** ThinMQM is tuned on only ~11.9 k synthetic instances covering two language pairs, raising questions about robustness beyond WMT24’s three test pairs.\n\nWe acknowledge the concern. The fact is that high-quality, human-annotated MQM test data is scarce due to the intensive labor and cost involved. In our work, we have utilized all publicly *available test data (3 languages)* from the WMT24 MQM shared task.\n\nTo assess the generalization capabilities of ThinMQM, we have conducted two additional experiments:\n\na. **Evaluation on WMT22 En-Ru:** We tested ThinMQM models and corresponding unaligned LRMs on the WMT22 English-Russian MQM data. This pair may help (Note: The WMT22 data was released before our LRMs' knowledge cutoff date) demonstrate the robustness of the method across different data distributions within the MQM scope. Experimental results in the following table show a consistent performance improvement.\n\n*Note: ( $\\Delta$ = ThinMQM - Original LRM)*\n\n|                      |  $\\Delta$  ThinMQM 7B |  $\\Delta$  ThinMQM 8B |  $\\Delta$  ThinMQM 32B |\n| :------------------: | :----------: | :----------: | :-----------: |\n| System-Level $\\rho$  |    +17.1     |     +6.6     |     +7.6      |\n| Segment-Level $\\tau$ |    +19.9     |    +14.6     |     +4.5      |\n\n\nb. **Evaluation on a Low-Resource, Out-of-Distribution Dataset:** To perform a stricter test, we sourced a recently released Hindi-Chinese dataset with MQM annotations [1], which was published after LRMs' knowledge cutoff date. Since this dataset contains translations from fewer than four systems, we report segment-level Pearson and Kendall correlation since the amount of candidate system is quite small. As shown in the table below, ThinMQM demonstrates strong correlation, performing competitively (32B model) against the powerful XCOMET-XXL baseline.\n\n| Model       | Segment-Level Pearson $\\rho$ | Segment-Level $\\tau$ |\n| :---------- | :--------------------------: | :------------------: |\n| XCOMET-XXL  |             62.5             |         47.8         |\n| ThinMQM 32B |           **63.4**           |       **57.4**       |\n| ThinMQM 7B  |             51.3             |         49.1         |\n| ThinMQM 8B  |             50.3             |         47.5         |\n\nThese new results suggest that ThinMQM generalizes well to unseen (w.r.t. SFT data) and low-resource language pairs.\n\n------\n\n> **Weakness 2:** There are some clarity issues - there is no explanation in fig.2's caption.\n\nThank you for pointing this out. We agree that the caption for Figure 2 was too brief. We will revise it in the final version to provide a comprehensive explanation of our research framework and clearly define the notations used in the synthetic data generation process.\n\n------\n\n> **Weakness 3:** Moreover, the novelty in terms of using LRMs for judging the quality of a response is questionable, because there are already many papers on this. While the thin-trajectory idea is neat, the fine-tune-on-synthetic-CoT recipe resembles recent chain-of-thought distillation work; novelty lies more in application domain than in method.\n\nWe appreciate you recognizing our work's contribution and agree with your assessment that a key part of our novelty lies in the application domain.\n\nWhile the idea of using LRMs for quality evaluation exists, there are still significant open challenges in machine translation evaluation regarding *how* to do so effectively and efficiently. Our work addresses a piece of this puzzle by exploring tailored materials for different model capabilities, better scoring mechanisms, reducing reasoning costs while improving performance.\n\nFurthermore, we would like to respectfully clarify a distinction between our \"thin-trajectory\" and Chain-of-Thought distillation. Our trajectories are **not distilled** from the outputs of a larger teacher model [2,3]. Instead, they are **synthetically constructed** using templates that are explicitly designed to mimic the standard operating procedures and annotation guidelines followed by humans. This approach builds a reasoning trajectory from first principles based on ground-truth labels, rather than learning to imitate another model's reasoning.\n\n------\n\n> **Q1:** The paper attributes the +3.9 – +8.7 point gains to ThinMQM but does not disentangle which design choices—trajectory length, explicit ESA tokens, or post-training itself—drive the lift. (More ablation studies would be nice.) The efficiency story (35 × fewer “thinking” tokens) is promising yet lacks a cost–quality curve.\n\nThis is an excellent suggestion. To better disentangle the contributions of our design choices, we have conducted an ablation study on the Error Span Annotation (ESA) component.\n\nWe ran an experiment where we removed the ESA spans from the reasoning trajectory during inference, retaining only the final score. The results, shown below, indicate a drop in performance. This highlights that identifying error spans is a crucial component for achieving high accuracy. Furthermore, as suggested by Reviewer RoLH, retaining ESA allows for valuable, fine-grained error analysis.\n\n*Note: The 'w/o ESA' row indicates the performance change ($\\Delta$) compared to the original ThinMQM model.*\n\n|              | Avg. (All) | En-De (Sys) | En-De (Seg) | En-Es (Sys) | En-Es (Seg) | Ja-Zh (Sys) | Ja-Zh (Seg) |\n| -----------: | :--------: | :---------: | :---------: | :---------: | :---------: | :---------: | :---------: |\n|  ThinMQM 7B |            |             |             |             |             |             |             |\n|     $\\Delta$ w/o ESA |   -0.80    |    +1.00    |    -2.40    |    -0.30    |    0.00     |    -0.90    |    -2.20    |\n|  ThinMQM 8B |            |             |             |             |             |             |             |\n|     $\\Delta$ w/o ESA |   -0.60    |    +0.20    |    -1.10    |    -1.30    |    0.00     |    -1.70    |    -0.20    |\n| ThinMQM 32B |            |             |             |             |             |             |             |\n|     $\\Delta$ w/o ESA |   -0.70    |    -1.80    |    -1.60    |    +1.70    |    -0.90    |    -0.50    |    -0.70    |\n\nRegarding the cost-quality curve, we agree this would be a valuable addition to illustrate the efficiency of our model. We are preparing this visualization and will gladly incorporate it into the final version of the paper. We appreciate your understanding that we cannot attach new figures during the rebuttal phase.\n\n------\n\n**References**\n\n[1] Yan, J., Yan, P., Chen, Y., Li, J., Zhu, X., & Zhang, Y. (2024). Benchmarking gpt-4 against human translators: A comprehensive evaluation across languages, domains, and expertise levels. *arXiv preprint arXiv:2411.13775*.\n\n[2] Zhuang, X., Zhu, Z., Wang, Z., Cheng, X., & Zou, Y. (2025). Unicott: A unified framework for structural chain-of-thought distillation. In *The Thirteenth International Conference on Learning Representations*.\n\n[3] Dai, C., Li, K., Zhou, W., & Hu, S. (2024). Improve Student’s Reasoning Generalizability through Cascading Decomposed CoTs Distillation. In *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing* (pp. 15623-15643)."}, {"review_id": "0T6oTWky2U", "reviewer": "Reviewer_JKag", "summary": "This paper first shows the potential of existing large reasoning models (LRMs) in the translation quality evaluation task.\nThen, on the basis of the observations, it proposes to use synthetic data for teaching human evaluation process, i.e., identifying the error spans and their severity,\nfollowed by aggregating them into a single numeric score.\nThrough an experiment, the authors confirmed that the proposed method improves the evaluation accuracy,\nin terms of system-level and segment-level pairwise accuracies, of LRMs, which can be sometime comparable or better to existing model-based metrics, such as COMET-22 and XCOMET.", "questions": "- Could you clearly define what deserves as a large reasoning model (LRM)?  The explanation in ll.28-29 \"explicitly powered by a thinking-guided generation process\" is too vague.\n\n- Please clearly and properly explain the elements in mathematical formulations those pointed out as \"Weaknesses.\"\n\n- What does \"All\" in Figure 3 and Tables 1 and 2 mean?  Are SPE and $ACC_{eq}^{*}$ additive, while they are based on different measurement and thus have different distribution?\n\n- Tables 1 and 2 shows the results with significance testing with $p<0.05$; which configuration is compared with each one?\n\n- One of the conclusions in this paper would be \"our thinMQM models outperform both existing LLM-based and metric-based methods\" in ll.277-278, but this is not always true.\n  How did the authors justify this?\n\nFollowings are comments for making charts appropriate.\n\n- Figure 1 should also show LLM-based and metric-based methods.  While using log-scale itself is OK, using only irregular scale marks make it difficult to understand it.\n\n- Figure 2 is first mentioned in page 8, l.256, so it should be moved to page 8, or it should be clearly explained earlier since presents the proposed method.\n\n- Figure 3 should be located after Table 1, following the order of their first mentions in the running text.\n\n- Figure 4: scale for p-value is strange; it is neither linear nor log-scaled.\n\n- Figure 6 is never mentioned in the running text.\n\n- Figure 7: line charts are inappropriate, since the horizontal axis is not interval scale but nominal scale.\n\n- Figure 8: bar charts are not based on 0 but arbitrary values, incorrectly exaggerating the difference with height and area of bars.\n\nReferences have a number of errors, including inappropriate lowercasing and inconsistent sets of BiBTeX fields.", "limitations": "Yes.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n- The paper unveils several tendencies of existing LRMs in the translation quality evaluation task (Section 3).\n- It proposes an effective way to improve the accuracy of the evaluation metric based on LRM, relying on synthetic data that follow manual evaluation process (Section 4).\n\nWeaknesses:\n\n- While the analysis of tendencies of simple use of LRMs is informative, it is too detailed, so as to make the main part not self-contained.\n\n- While the proposed method improves evaluation accuracy of LRMs,\n  they are still worse than existing smaller task-specific models, e.g., XCOMET (3B or 10B).\n\n- Mathematical formulations are imprecise and/or defective.  For example, $S'$ in l.154 is never used;\n  the operation $+$ for two textual elements in l.157 is undefined; $\\phi_{s}^{approx}$ and $\\phi{r}^{approx}$ are not defined and their usages are not explained.\n  while $T_{ESA}$ and $T_{score}$ are defined as function in ll.252-253, they are used as variables in l.255 and Equation (3);\n  the loss function in Equation (3), $\\mathcal{L}$, is never defined; the returned value of $M(\\cdot;\\theta)$ is unknown.", "was_revised": true, "final_justification": "Through the rebuttal, some of my misunderstandings were resolved.\nYet, I think this paper should be judged once it is revised.\nFor fairness, I've not taken care of new results provided after the submission deadline.", "rebuttal": "Thank you for your time and detailed feedback, which is invaluable for improving our work. We would like to address the concerns raised and clarify some points.\n\n------\n\n> **Weakness 1:** While the analysis of tendencies of simple use of LRMs is informative, it is too detailed, so as to make the main part not self-contained; the method of generating the proposed synthetic data is not explained in the main part at all.\n\nThank you for acknowledging that our analysis is informative. We would like to gently clarify that our research is *analysis-driven*. The detailed analysis of potential issues in current LRM practices in MT evaluation is fundamental to motivating and deriving our final proposed method. Therefore, we consider this analytical section to be a crucial component of the paper's narrative.\n\nHowever, we understand the concern regarding the balance of the paper. We are more than willing to revise the proportions and level of detail of the analysis and method sections to improve the overall flow and focus.\n\nRegarding the method for generating synthetic data, we would like to clarify that this was included in the main paper. We provide a detailed **description in Lines 246-266**. We apologize if its placement was not prominent enough and will ensure it is clearly signposted in the revised version.\n\n------\n\n> **Weakness 2:** While the proposed method improves evaluation accuracy of LRMs, they are still worse than existing smaller task-specific models, e.g., XCOMET (3B or 10B).\n\nWe would like to clarify two key points:\n\n1. The primary contribution of our paper is the improvement of LRMs-as-Judges. Our goal is to analyze specific problems and enhance this paradigm.\n2. The comparison with XCOMET may also consider the **difference in training data**. Our method uses only ~12k synthetic trajectories (for 2 language pairs), whereas XCOMET was trained on 1M Direct Assessment (DA) human annotations (36 languages) and 176K MQM annotations (14 languages) [1].\n\nFurthermore, as we noted in our response to other reviewers, a key strength of the LRM-based approach lies in its potential for out-of-distribution and low-resource language pairs. This is demonstrated by our additional experiments on recently released Hindi-Chinese MQM data [2].\n\n\n| Model       | Segment-Level Pearson $\\rho$ | Segment-Level $\\tau$ |\n| :---------- | :--------------------------: | :------------------: |\n| XCOMET-XXL  |             62.5             |         47.8         |\n| ThinMQM 32B |           **63.4**           |       **57.4**       |\n| ThinMQM 7B  |             51.3             |         49.1         |\n| ThinMQM 8B  |             50.3             |         47.5         |\n\n\n------\n\n> **Weakness 3:** Mathematical formulations are imprecise and/or defective.\n\nThank you very much for your detailed feedback regarding the mathematical formulations. We noted that some of these concepts were defined within the paper, and we aim to clarify any misunderstandings below. In the revised manuscript, we will incorporate more detailed explanations for these notions and refine our notation to ensure greater clarity and precision.\n\n- **Implementation of $v(\\cdot)$ in l.153:** As we explained in Lines 153-154, $v(\\cdot)$ is calculated \"using system-level and segment-level metrics,\" which correspond to the SPA and $Acc{_{Eq}^*}$ metrics defined in Line 133. We will make this link more explicit.\n- **$S’$ in l.154 is never used:** This is a typo and should be the lowercase $s'$ from Equation 2. We will correct this.\n- **The $+$ operation in l.157:** The notation $r+t$ was intended to represent the standard reference-based evaluation setting (Ref.). Thus, $v(r+t)$ is the correlation score under this setting. If the $+$ symbol is confusing, we are happy to replace it with a more descriptive notation (e.g., $v([r, t])$) and define it clearly.\n- **$\\phi_{s}^{approx}$ and $\\phi_{r}^{approx}$ are not defined:** We used \"approximate\" here to distinguish from the strict definition of Shapley Value in Eq. 2. As noted in Line 154-157, a strict calculation would require evaluating all subsets. However, in the context of machine translation evaluation, evaluating a reference $v(r)$ or a translation $v(t)$ in isolation is not a valid setting. Our $\\phi_{s}^{approx}$ is the practical Shapley Value for the MT evaluation task. We will clarify this and can rename it to $\\phi_{s}^{MT}$  for better understanding.\n- **$T_{ESA}$  and $T_{score}$ usage in l.255 and Eq.3:** Our intention was to represent the *output* of these functions after the mapping. We will revise the notation to be more explicit, for instance: $[T_{ESA}(X), T_{Score}(T_{ESA}(X))]$.\n- **The loss function $\\mathcal{L}$ in Eq.3 is never defined:** We used the cross-entropy loss function. We will explicitly state this in the experimental setup (Section 4.2) in the revision.\n- **The returned value of $M(·;θ)$ is unknown:** As defined in **Line 257**, $M$ is the model. The return value $M(·;θ)$ is the **generated output sequence** from the model. We will ensure this is clear.\n\n------\n\n> **Q1:** Could you clearly define what deserves as a large reasoning model (LRM)? The explanation in ll.28-29 \"explicitly powered by a thinking-guided generation process\" is too vague.\n\nThank you for this question. We appreciate the opportunity to clarify the definition of Large Reasoning Models (LRMs) with reference to existing literature:\n\n- *A Large Reasoning Models (LRMs) is an advanced artificial intelligence model that extends the capabilities of traditional Large Language Models (LLMs) by introducing the concept of a \"thought\". A \"thought\" is defined as a sequence of tokens representing intermediate reasoning steps, designed to mimic complex human cognitive processes like tree search and reflective thinking [3].* Representative LRMs include OpenAI's o1/o3, Google’s Gemini 2.5 Pro, DeepSeek-R1, etc.\n\nWe will summarize and refine this definition in the paper to be more concrete.\n\n------\n\n> **Q2:** What does \"All\" in Figure 3 and Tables 1 and 2 mean? Are SPA and $Acc{_{Eq}^*}$ additive, while they are based on different measurement and thus have different distribution?\n\n\"All\" refers to the **mean (average) score** of all correlation metrics used. We follow the official evaluation practice of the WMT24 MQM shared task [4], which also reports the average of different correlation coefficients to provide a measure of performance.\n\n------\n\n> **Q3:** Tables 1 and 2 shows the results with significance testing with p<0.05; which configuration is compared with each one?\n\nAs described in **Lines 139-140**, the testing was conducted using the permutation-based method (1,000 resampling times), i.e., PERM-BOTH hypothesis test [5]. This also follows WMT24 official evaluation practice [4].\n\n------\n\n> **Q4:** One of the conclusions in this paper would be \"our thinMQM models outperform both existing LLM-based and metric-based methods\" in ll.277-278, but this is not always true. How did the authors justify this?\n\nWe will revise it to be more precise. The corrected statement will clarify that our ThinMQM-32B model achieves superior *average* performance compared to the baselines, though not necessarily on every individual language pair. We will also refine our conclusion to emphasize that our method shows a highly competitive performance and a much better efficiency-performance trade-off, particularly within the scope of LRM/LLM-based evaluators.\n\n------\n\n> **Response to Comments on Charts**\n\n- **Figure 1:** For part (a), we will include the available models in the scale comparison. However, some closed-source models, such as GEMBA-ESA, lack publicly available information regarding their scale. For (b), we would like to clarify that the scope is limited to LRMs as we are analyzing reasoning budgets.\n- **Figure 7:** We will add an axis break on the horizontal axis to improve readability.\n- **General:** We will review all figures to improve their clarity and adjust their placement within the paper based on the feedback.\n\n------\n\n**References**\n\n[1] Guerreiro, N. M., Rei, R., Stigt, D. V., Coheur, L., Colombo, P., & Martins, A. F. (2024). XCOMET: Transparent machine translation evaluation through fine-grained error detection. *Transactions of the Association for Computational Linguistics*, *12*, 979-995.\n\n[2] Yan, J., Yan, P., Chen, Y., Li, J., Zhu, X., & Zhang, Y. (2024). Benchmarking gpt-4 against human translators: A comprehensive evaluation across languages, domains, and expertise levels. *arXiv preprint arXiv:2411.13775*.\n\n[3] Xu, F., Hao, Q., Zong, Z., Wang, J., Zhang, Y., Wang, J., ... & Li, Y. (2025). Towards large reasoning models: A survey of reinforced reasoning with large language models. *arXiv preprint arXiv:2501.09686*.\n\n[4] Freitag, M., Mathur, N., Deutsch, D., Lo, C. K., Avramidis, E., Rei, R., ... & Lavie, A. (2024). Are LLMs breaking MT metrics? results of the WMT24 metrics shared task. In *Proceedings of the Ninth Conference on Machine Translation* (pp. 47-81).\n\n[5] Deutsch, D., Dror, R., & Roth, D. (2021). A statistical analysis of summarization evaluation metrics using resampling methods. *Transactions of the Association for Computational Linguistics*, *9*, 1132-1146."}, {"review_id": "ZLg6KWdPmm", "reviewer": "Reviewer_RoLH", "summary": "This paper investigates the capability of Large Reasoning Models (LRMs) to serve as evaluators for machine translation (MT) quality, a relatively unexplored topic. The authors identify key challenges in using LRMs for MT evaluation, such as inefficient “overthinking,” scoring overestimation, and material sensitivity. They propose a solution named ThinMQM—a thinking-calibrated MQM scoring method—that aligns LRM reasoning trajectories with human-like annotations using synthetic data. The method significantly improves correlation with human MQM ratings while reducing computation by 35x across multiple LRM scales. Empirical results on WMT24 metrics benchmarks support their claims, showing strong improvements over both traditional metrics (BLEU, COMET) and LLM-based baselines (GEMBA-MQM).", "questions": "1.The proposed ThinMQM is evaluated only under the MQM framework and WMT24 language pairs. Could the authors clarify whether the method can generalize to other translation evaluation paradigms (e.g., adequacy-only scales) or low-resource settings? Demonstrating broader applicability would increase the impact of this work.\n2.In setups where auxiliary models (e.g., Qwen-32B) are used to score, how do the authors disentangle the effect of the LRM's reasoning from the scoring model? It remains unclear whether the observed improvement is due to better annotations or just better post-processing. More controlled ablation here would strengthen the claims.\n3.Can the authors provide examples or an error taxonomy where ThinMQM still diverges significantly from human judgments? This would help clarify whether current improvements come at the cost of under- or over-penalizing certain error types, and guide future work in more targeted calibration.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Stengths:\n1.This is the first systematic study of LRM-as-a-judge for MT evaluation, bridging reasoning models and MT metrics—a timely and significant contribution.\n2.ThinMQM is a conceptually elegant and practically effective method to calibrate LRMs using synthetic reasoning-aligned supervision. This allows efficiency gains without sacrificing (and even improving) performance.\n3.The paper thoroughly analyzes failure modes, such as overestimation and overthinking, and leverages Shapley value analysis to isolate input contributions—rare in MT metric research.\n4.The paper is clearly written and well-structured, and includes useful figures/tables to illustrate thinking budgets, score distributions, and model comparisons.\nWeakness:\n1.The main training and evaluation are based on just three language pairs, and training data is synthesized from WMT23 MQM annotations. Generalizability to low-resource or morphologically rich languages is unclear.\n2.While ThinMQM reduces variance and improves scoring alignment, the diversity of human annotation styles or reasoning is not captured—raising questions about overfitting to one “style” of evaluation.\n3.When auxiliary models are used for scoring (e.g., Qwen-32B), attribution becomes blurry. It is unclear whether improvements stem from the LRM itself or the scorer, despite some efforts to address this.\n4.Although the paper shows efficiency improvements, some segment-level performance under certain decoding parameters still fluctuates slightly, suggesting calibration is sensitive to model internals.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback. We appreciate the opportunity to clarify our contributions and address the concerns raised. Below are our point-by-point responses.\n\n> **Weakness 1. & Q1.** The main training and evaluation are based on just three language pairs, and training data is synthesized from WMT23 MQM annotations. Generalizability to low-resource or morphologically rich languages is unclear.\n\nWe appreciate your concern regarding generalizability. The three language pairs we used come are all available MQM evaluation materials in the WMT24 benchmarks. To better assess the robustness of ThinMQM, we have conducted two new experiments on more diverse and out-of-distribution data.\n\na. **Evaluation on WMT22 English-Russian:** We evaluated our model on the WMT22 English-Russian language pair. Although this dataset predates the knowledge cutoff for the LRM, which could be a potential source of data leakage, the results in the table below show that ThinMQM still achieves a significant improvement.\n\n*Note: ( $\\Delta$ = ThinMQM - Original LRM)*\n\n|                      |  $\\Delta$  ThinMQM 7B |  $\\Delta$  ThinMQM 8B |  $\\Delta$  ThinMQM 32B |\n| :------------------: | :----------: | :----------: | :-----------: |\n| System-Level $\\rho$  |    +17.1     |     +6.6     |     +7.6      |\n| Segment-Level $\\tau$ |    +19.9     |    +14.6     |     +4.5      |\n\n\nb. **A Recent Low-Resource MQM Dataset (Hindi-Chinese):** To perform a more stringent test on a low-resource language pair, we sourced a recently released Hindi-Chinese dataset with MQM annotations [1], which was published after our LRMs’ knowledge cutoff date. Since this dataset contains translations from fewer than four systems, we report segment-level Pearson and Kendall correlation. As shown in the table below, ThinMQM demonstrates strong generalization capabilities, outperforming the XCOMET-XXL baseline.\n\n| Model       | Segment-Level Pearson $\\rho$ | Segment-Level $\\tau$ |\n| :---------- | :--------------------------: | :------------------: |\n| XCOMET-XXL  |             62.5             |         47.8         |\n| ThinMQM 32B |           **63.4**           |       **57.4**       |\n| ThinMQM 7B  |             51.3             |         49.1         |\n| ThinMQM 8B  |             50.3             |         47.5         |\n\n\n------\n\n> **Weakness: 2:** While ThinMQM reduces variance and improves scoring alignment, the diversity of human annotation styles or reasoning is not captured—raising questions about overfitting to one “style” of evaluation.  \n\nWe believe this point touches upon a fundamental challenge in evaluation research: the trade-off between aligning with a specific benchmark to achieve high correlation and capturing the full diversity of human judgment. A deep exploration of annotation \"style\" is a valuable research direction that extends beyond the immediate scope of this paper.\n\nOur work is situated within the MQM framework, which itself imposes a structured and detailed set of guidelines for human annotators. This inherent structure helps standardize annotations and reduce, though not eliminate, the impact of individual preferences.\n\nWe will add a discussion of this point in the revised manuscript. As a potential avenue for future work, one could collect data from annotators with diverse backgrounds using varied evaluation criteria to explicitly model and understand these stylistic differences.\n\n> **Weakness 3. & Q2.** When auxiliary models are used for scoring (e.g., Qwen-32B), attribution becomes blurry. It is unclear whether improvements stem from the LRM itself or the scorer, despite some efforts to address this.\n\nWe would like to address two (possible) misunderstandings:\n\n1. The significance test in Section 3.1 compared LRM+Auxiliary Model versus the Auxiliary Model alone, showing no significant difference between these approaches. However, the LRM+Auxiliary Model outperforms the LRM alone. This indicates that the performance improvements primarily come from the Auxiliary Model rather than the LRM itself.\n2. **ThinMQM does not use an auxiliary scorer.** We apologize if our writing was unclear. The final score in ThinMQM is generated directly and solely by the LRM as the final step in its internal reasoning trace.\n\n**Besides, for ThinMQM, the error annotation process is integral to performance.** We conducted a new ablation study. We trained the LRM to **only** predict the final score, without the intermediate error annotation steps. The results below show a drop in performance, confirming that the fine-grained error analysis is critical to ThinMQM's success.\n\n*Note: The 'w/o ESA' row indicates the performance change ($\\Delta$) compared to the original ThinMQM model.*\n\n|              | Avg. (All) | En-De (Sys) | En-De (Seg) | En-Es (Sys) | En-Es (Seg) | Ja-Zh (Sys) | Ja-Zh (Seg) |\n| -----------: | :--------: | :---------: | :---------: | :---------: | :---------: | :---------: | :---------: |\n|  ThinMQM 7B |            |             |             |             |             |             |             |\n|      $\\Delta$ w/o ESA |   -0.80    |    +1.00    |    -2.40    |    -0.30    |    0.00     |    -0.90    |    -2.20    |\n|  ThinMQM 8B |            |             |             |             |             |             |             |\n|      $\\Delta$ w/o ESA |   -0.60    |    +0.20    |    -1.10    |    -1.30    |    0.00     |    -1.70    |    -0.20    |\n| ThinMQM 32B |            |             |             |             |             |             |             |\n|     $\\Delta$ w/o ESA |   -0.70    |    -1.80    |    -1.60    |    +1.70    |    -0.90    |    -0.50    |    -0.70    |\n\n\n> **Weakness 4:** Although the paper shows efficiency improvements, some segment-level performance under certain decoding parameters still fluctuates slightly, suggesting calibration is sensitive to model internals.\n\nThank you for noticing this. In line with common practices for LRMs [2-3], we recommend using a decoding temperature of 0.6 in practice, which we have also found provides more stable results for ThinMQM models.\n\nTo formally verify the stability of this setting, we performed 3  runs at a fixed temperature of 0.6. The table below reports the mean and standard deviation. The low standard deviation indicates that performance is highly consistent and not subject to significant random fluctuations.\n\n*Note: (Mean ± Std. of 3 runs)*\n\n| **Model**   | **Avg. (All)** | **En-De (Sys)** | **En-De (Seg)** | **En-Es (Sys)** | **En-Es (Seg)** | **Ja-Zh (Sys)** | **Ja-Zh (Seg)** |\n| :---------- | :------------: | :-------------: | :-------------: | :-------------: | :-------------: | :-------------: | :-------------: |\n| ThinMQM 32B |  72.0 ± 0.003  |  80.7 ± 0.026   |  53.1 ± 0.006   |  80.8 ± 0.002   |  68.9 ± 0.002   |  92.5 ± 0.011   |  55.9 ± 0.002   |\n| ThinMQM 8B  |  70.4 ± 0.004  |  83.1 ± 0.021   |  48.6 ± 0.003   |  81.3 ± 0.020   |  68.2 ± 0.001   |  90.3 ± 0.013  |  51.0 ± 0.001   |\n| ThinMQM 7B  |  70.0 ± 0.002  |  85.4 ± 0.011   |  48.2 ± 0.004   |  77.6 ± 0.002   |  68.1 ± 0.001   |  89.3 ± 0.004  |  51.4 ± 0.003   |\n\n------\n\n> **Q3:** Can the authors provide examples or an error taxonomy where ThinMQM still diverges significantly from human judgments? This would help clarify whether current improvements come at the cost of under- or over-penalizing certain error types, and guide future work in more targeted calibration.\n\nThis is an excellent suggestion. We agree that an error analysis provides deeper insight into LRMs’ behavior. We have conducted a detailed analysis of the cases where scores from ThinMQM-32B diverge from human judgments. We categorized these discrepancies based on the MQM translation error taxonomy (Critical, Major, Minor).\n\nThe distribution of these error types is presented in the table below. This analysis reveals that the model predictions most often misalign with human judgments on **Minor-level errors.** Furthermore, when analyzing all minor error types, we found that **accuracy/mistranslation** accounts for the highest proportion of Minor-level prediction errors. These results helps pinpoint specific areas for targeted improvements in future work.\n\n|       | Critical (%) | Major (%) | Minor (%) |\n| :---: | :----------: | :-------: | :-------: |\n| En-De |    0.05     |  12.35   |  87.60  |\n| En-Es |    0.21     |  10.29   |  89.50   |\n| Ja-Zh |    0.02     |  17.55   |  82.43   |\n\n------\n\n**References**\n\n[1]  Yan, J., Yan, P., Chen, Y., Li, J., Zhu, X., & Zhang, Y. (2024). Benchmarking gpt-4 against human translators: A comprehensive evaluation across languages, domains, and expertise levels. *arXiv preprint arXiv:2411.13775*.\n\n[2] Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., ... & He, Y. (2025). Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. *arXiv preprint arXiv:2501.12948*.\n\n[3] Abdin, M., Agarwal, S., Awadallah, A., Balachandran, V., Behl, H., Chen, L., ... & Zheng, G. (2025). Phi-4-reasoning technical report. *arXiv preprint arXiv:2504.21318*."}]

Author rebuttal:

Meta-review: # Summary
This paper investigates the capability of Large Reasoning Models (LRMs) to serve as evaluators for machine translation (MT) quality, a relatively unexplored topic. The authors identify key challenges in using LRMs for MT evaluation, such as inefficient “overthinking,” scoring overestimation, and material sensitivity. They propose a solution named ThinMQM—a thinking-calibrated MQM scoring method—that aligns LRM reasoning trajectories with human-like annotations using synthetic data. The method significantly improves correlation with human MQM ratings while reducing computation by 35x across multiple LRM scales.

# Strengths
* This is the first systematic study of LRM-as-a-judge for MT evaluation, bridging reasoning models and MT metrics—a timely and significant contribution.
* It proposes an effective way to improve the accuracy of the evaluation metric based on LRM, relying on synthetic data that follow a manual evaluation process.
* The paper is clearly written and well-structured, and includes useful figures/tables to illustrate thinking budgets, score distributions, and model comparisons

# Overall
This paper presents a novel method for evaluating MT output with LRMs. The authors successfully resolved all major weaknesses and questions identified during the author-reviewer discussion period. The paper now offers sufficient contributions to be of broad interest to the research community.
Decision: Accept (poster)