Title: Are Large Reasoning Models Good Translation Evaluators? Analysis and Performance Boost
Venue: NeurIPS 2025
Accepted: yes
Keywords: large reasoning model

evaluation

machine translation

analysis
Abstract: Recent advancements in large reasoning models (LRMs) have introduced an intermediate "thinking" process prior to generating final answers, improving their reasoning capabilities on complex downstream tasks. However, the potential of LRMs as evaluators for machine translation (MT) quality remains underexplored. We provides the first systematic analysis of LRM-as-a-judge in MT evaluation. We identify key challenges, revealing LRMs require tailored evaluation materials, tend to "overthink" simpler instances and have issues with scoring mechanisms leading to overestimation. To address these, we propose to calibrate LRM thinking by training them on synthetic, human-like thinking trajectories. Our experiments on WMT24 Metrics benchmarks demonstrate that this approach largely reduces thinking budgets by ~35x while concurrently improving evaluation performance across different LRM scales from 7B to 32B (e.g., R1-Distill-Qwen-7B achieves a +8.7 correlation point improvement). These findings highlight the potential of efficiently calibrated LRMs to advance fine-grained automatic MT evaluation.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the capability of Large Reasoning Models (LRMs) to serve as evaluators for machine translation (MT) quality, a relatively unexplored topic. The authors identify key challenges in using LRMs for MT evaluation, such as inefficient “overthinking,” scoring overestimation, and material sensitivity. They propose a solution named ThinMQM—a thinking-calibrated MQM scoring method—that aligns LRM reasoning trajectories with human-like annotations using synthetic data. The method significantly improves correlation with human MQM ratings while reducing computation by 35x across multiple LRM scales. Empirical results on WMT24 metrics benchmarks support their claims, showing strong improvements over both traditional metrics (BLEU, COMET) and LLM-based baselines (GEMBA-MQM).\n\nSTRENGTHS AND WEAKNESSES:\nStengths:\n1.This is the first systematic study of LRM-as-a-judge for MT evaluation, bridging reasoning models and MT metrics—a timely and significant contribution.\n2.ThinMQM is a conceptually elegant and practically effective method to calibrate LRMs using synthetic reasoning-aligned supervision. This allows efficiency gains without sacrificing (and even improving) performance.\n3.The paper thoroughly analyzes failure modes, such as overestimation and overthinking, and leverages Shapley value analysis to isolate input contributions—rare in MT metric research.\n4.The paper is clearly written and well-structured, and includes useful figures/tables to illustrate thinking budgets, score distributions, and model comparisons.\nWeakness:\n1.The main training and evaluation are based on just three language pairs, and training data is synthesized from WMT23 MQM annotations. Generalizability to low-resource or morphologically rich languages is unclear.\n2.While ThinMQM reduces variance and improves scoring alignment, the diversity of human annotation styles or reasoning is not captured—raising questions about overfitting to one “style” of evaluation.\n3.When auxiliary models are used for scoring (e.g., Qwen-32B), attribution becomes blurry. It is unclear whether improvements stem from the LRM itself or the scorer, despite some efforts to address this.\n4.Although the paper shows efficiency improvements, some segment-level performance under certain decoding parameters still fluctuates slightly, suggesting calibration is sensitive to model internals.\n\nQUESTIONS:\n1.The proposed ThinMQM is evaluated only under the MQM framework and WMT24 language pairs. Could the authors clarify whether the method can generalize to other translation evaluation paradigms (e.g., adequacy-only scales) or low-resource settings? Demonstrating broader applicability would increase the impact of this work.\n2.In setups where auxiliary models (e.g., Qwen-32B) are used to score, how do the authors disentangle the effect of the LRM's reasoning from the scoring model? It remains unclear whether the observed improvement is due to better annotations or just better post-processing. More controlled ablation here would strengthen the claims.\n3.Can the authors provide examples or an error taxonomy where ThinMQM still diverges significantly from human judgments? This would help clarify whether current improvements come at the cost of under- or over-penalizing certain error types, and guide future work in more targeted calibration.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper first shows the potential of existing large reasoning models (LRMs) in the translation quality evaluation task.\nThen, on the basis of the observations, it proposes to use synthetic data for teaching human evaluation process, i.e., identifying the error spans and their severity,\nfollowed by aggregating them into a single numeric score.\nThrough an experiment, the authors confirmed that the proposed method improves the evaluation accuracy,\nin terms of system-level and segment-level pairwise accuracies, of LRMs, which can be sometime comparable or better to existing model-based metrics, such as COMET-22 and XCOMET.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The paper unveils several tendencies of existing LRMs in the translation quality evaluation task (Section 3).\n- It proposes an effective way to improve the accuracy of the evaluation metric based on LRM, relying on synthetic data that follow manual evaluation process (Section 4).\n\nWeaknesses:\n\n- While the analysis of tendencies of simple use of LRMs is informative, it is too detailed, so as to make the main part not self-contained.\n\n- While the proposed method improves evaluation accuracy of LRMs,\n  they are still worse than existing smaller task-specific models, e.g., XCOMET (3B or 10B).\n\n- Mathematical formulations are imprecise and/or defective.  For example, $S'$ in l.154 is never used;\n  the operation $+$ for two textual elements in l.157 is undefined; $\\phi_{s}^{approx}$ and $\\phi{r}^{approx}$ are not defined and their usages are not explained.\n  while $T_{ESA}$ and $T_{score}$ are defined as function in ll.252-253, they are used as variables in l.255 and Equation (3);\n  the loss function in Equation (3), $\\mathcal{L}$, is never defined; the returned value of $M(\\cdot;\\theta)$ is unknown.\n\nQUESTIONS:\n- Could you clearly define what deserves as a large reasoning model (LRM)?  The explanation in ll.28-29 \"explicitly powered by a thinking-guided generation process\" is too vague.\n\n- Please clearly and properly explain the elements in mathematical formulations those pointed out as \"Weaknesses.\"\n\n- What does \"All\" in Figure 3 and Tables 1 and 2 mean?  Are SPE and $ACC_{eq}^{*}$ additive, while they are based on different measurement and thus have different distribution?\n\n- Tables 1 and 2 shows the results with significance testing with $p<0.05$; which configuration is compared with each one?\n\n- One of the conclusions in this paper would be \"our thinMQM models outperform both existing LLM-based and metric-based methods\" in ll.277-278, but this is not always true.\n  How did the authors justify this?\n\nFollowings are comments for making charts appropriate.\n\n- Figure 1 should also show LLM-based and metric-based methods.  While using log-scale itself is OK, using only irregular scale marks make it difficult to understand it.\n\n- Figure 2 is first mentioned in page 8, l.256, so it should be moved to page 8, or it should be clearly explained earlier since presents the proposed method.\n\n- Figure 3 should be located after Table 1, following the order of their first mentions in the running text.\n\n- Figure 4: scale for p-value is strange; it is neither linear nor log-scaled.\n\n- Figure 6 is never mentioned in the running text.\n\n- Figure 7: line charts are inappropriate, since the horizontal axis is not interval scale but nominal scale.\n\n- Figure 8: bar charts are not based on 0 but arbitrary values, incorrectly exaggerating the difference with height and area of bars.\n\nReferences have a number of errors, including inappropriate lowercasing and inconsistent sets of BiBTeX fields.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper delivers the first comprehensive examination of large reasoning models (LRMs) as machine-translation judges, showing that while their “slow-thinking” paradigm can model the MQM evaluation process, they also require carefully chosen inputs, often “over-think” easy segments and systematically over-estimate quality. Building on these insights, the authors introduce ThinMQM—a post-training scheme that feeds LRMs synthetic, human-aligned error-annotation + scoring trajectories—which realigns the model’s reasoning, cuts inference-time “thinking” tokens by about 35 × and boosts human-correlation on the WMT-24 Metrics benchmark by up to +8.7 points across 7 B–32 B models, matching or surpassing state-of-the-art neural metrics without auxiliary scorers .\n\nSTRENGTHS AND WEAKNESSES:\nThe paper has solid empirical design because it study evaluates four LRMs ranging from 7 B to 671 B parameters under a unified GEMBA-MQM protocol, reporting both system- and segment-level SPA/Acc with 1 000-sample permutation tests, which lends statistical rigor. Substantial gains and efficiency - ThinMQM lifts correlation by up to +8.7 points while slashing the “slow-thinking” budget ~35 × and cutting wall-clock inference from 12 min to 40 s per 1 k sentences. The writing is easy to follow-the introduction explicitly lists three research questions, and Figure 2 walks the reader through the evaluation pipeline and ThinMQM calibration steps, making the workflow flows well. First systematic look at “LRM-as-a-judge.” The paper is, one of the earliest comprehensive study of reasoning-tuned models for MQM evaluation, and it introduces ThinMQM—synthetic, human-aligned reasoning trajectories to calibrate scoring.\n\nThinMQM is tuned on only ~11.9 k synthetic instances covering two language pairs, raising questions about robustness beyond WMT24’s three test pairs. There are some clarity issues - there is no explanation in fig.2's caption. Moreover, the novelty in terms of using LRMs for judging the quality of a response is questionable, because there are already many papers on this. While the thin-trajectory idea is neat, the fine-tune-on-synthetic-CoT recipe resembles recent chain-of-thought distillation work; novelty lies more in application domain than in method.\n\nQUESTIONS:\nThe paper attributes the +3.9 – +8.7 point gains to ThinMQM but does not disentangle which design choices—trajectory length, explicit ESA tokens, or post-training itself—drive the lift. (More ablation studies would be nice.) \nThe efficiency story (35 × fewer “thinking” tokens) is promising yet lacks a cost–quality curve.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is the first systematic evaluation of large reasoning models (LRMs) as the judge for evaluating machine translation tasks. The authors systematically analyze LRM performance in MT evaluation and identify key insights:\n\n1) Smaller LRM models (7B–8B) benefit more from reference-based inputs, while larger ones (32B–671B) perform better with source-based information. This contradicts prior LLM findings and suggests that LRM design and scale fundamentally alter how evaluation context is processed.\n2) Although reference-based evaluation is popular, it may hurt LRM accuracy, especially at scale, due to noise or low quality in reference translations. Human MQM annotations are reference-free, so introducing references may create a mismatch with human judgment standards.\n3) Switching from rule-based to model-based re-scoring (using auxiliary models like Qwen-2.5 32B) increases computation but doesn't significantly improve alignment with human evaluations. Worse, the improvements often stem from the scorer rather than the LRM, muddying attribution.\n4) Larger LRMs often generate more tokens and reasoning steps (higher “thinking budget”), but this does not consistently correlate with better evaluation accuracy. Surprisingly, correct judgments require more effort only in edge cases, and LRMs still underperform general LLMs in nearly half of tested scenarios.\n\nTo address these issues, the paper proposes Thinking-calibrated MQM (ThinMQM) scoring, which involves training LRMs on synthetic, human-like thinking trajectories using supervised fine-tuning. Experiments on WMT24 Metrics benchmarks demonstrate that this approach reduces thinking budgets while improving evaluation performance across various LRM.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n1) This paper is the first systematic evaluation of LRM as the evaluators on the machine translation tasks, which provides a lot of interesting insights.\n2) The paper is very well written and clear, make it easy to understand and follow for even less familiar to the topic researchers,\n3) This paper studies each concept in a very detailed evaluation setup and with many observations, that does not leave any questions around the topic unanswered.\n\nWeaknesses:\n\n1) The results suggest that XCOMET, an Encoder only model that is trained with a backbone encoder of size 3.5B is performing better than the proposed trained LRM method in tasks when one side of the translation is English. Additionally, since this is an encoder only model it is cheaper than LRM models as well. \n2) The paper does not study more powerful commercial LLMs such as GPT-O3 or Gemini 2.5 Pro. It would be interesting to see how they will perform in these tasks and what are the problems with those frontier LLMs.\n\nQUESTIONS:\n1) How does changing the rule-based scoring schema change the correlation with human judgement? For example instead of -25, -5, and -1 if you use -3, -2, -1 or -100, -50, -1 how does that affect your evaluation and correlation with human judgement?\n\n2) You are using a quite high temperature, 0.6 for decoding, which means more randomness in the experiments, and less reproducibility. Is there any reason that you didn’t use temperature 0 or 0.1 instead?\n\n3) Why does your model perform better than XCOMET on Ja-Zh tasks but not on others? Do you think the reason is that XCOMET does not have the Chinese and Japanese knowledge or is there any other reason?\n\nLIMITATIONS:\nYES", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-10"}