Title: Calibrating Translation Decoding with Quality Estimation on LLMs
Venue: NeurIPS 2025
Accepted: yes
Keywords: Translation

Quality Estimation

LLMs

Calibration
Abstract: Neural machine translation (NMT) systems typically employ maximum *a&nbsp;posteriori* (MAP) decoding to select the highest-scoring translation from the distribution. However, recent evidence highlights the inadequacy of MAP decoding, often resulting in low-quality or even pathological hypotheses as the decoding objective is only weakly aligned with real-world translation quality. This paper proposes to directly calibrate hypothesis likelihood with translation quality from a distributional view by directly optimizing their Pearson correlation, thereby enhancing decoding effectiveness. With our method, translation with large language models (LLMs) improves substantially after limited training (2K instances per direction). This improvement is orthogonal to those achieved through supervised fine-tuning, leading to substantial gains across a broad range of metrics and human evaluations. This holds even when applied to top-performing translation-specialized LLMs fine-tuned on high-quality translation data, such as Tower, or when compared to recent preference optimization methods, like CPO. Moreover, the calibrated translation likelihood can directly serve as a strong proxy for translation quality, closely approximating or even surpassing some state-of-the-art translation quality estimation models, like CometKiwi. 
Lastly, our in-depth analysis demonstrates that calibration enhances the effectiveness of MAP decoding, thereby enabling greater efficiency in real-world deployment. The resulting state-of-the-art translation model, which covers 10 languages, along with the accompanying code and human evaluation data, has been released: https://github.com/moore3930/calibrating-llm-mt.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe presented paper proposes an additional loss for machine translation fine-tuning of LLMs which directly optimizes the spearman correlation towards neural translation quality metrics. The authors demonstrate that this approach leads to a better calibrated model with superior translation quality that can also serve as a quality estimator based on its hypothesis likelihoods.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- Interesting finding that we can directly optimize Pearson correlation to not only get a better translation model but also a better QE metric.\n- Following best practices in machine translation e.g. multiple neural metrics and the experiments are comprehensive.\n- Interesting finding that scaling up the calibrated model improves translation quality but not quality estimation\n\n\n### Weaknesses\n\n- l. 150 the statement that the translation quality has not been used in previous works is not exactly accurate, see for example \"[A Preference-driven Paradigm for Enhanced Translation with Large Language Models](https://aclanthology.org/2024.naacl-long.186/)\" (Zhu et al., NAACL 2024).\n- If I understand correctly, the proposed approach uses listwise preference data through the proxy of the $L_{pearson}$ which is not really a fair comparison to e.g. CPO which only leverages a single rejected hypothesis. A more fair comparison would be towards \"[A Preference-driven Paradigm for Enhanced Translation with Large Language Models](https://aclanthology.org/2024.naacl-long.186/)\" (Zhu et al., NAACL 2024) where we leverage the metric scores as replacement for human annotations and optimize directly on listwise rejected hypothesis.\n- The presented approach is only evaluated on the sentence level and doesn't venture into paragraph or document level machine translation which may come with different learning dynamics.\n- l. 218ff. / l.333f.: I don't think it is justified to say that the quality is \"comparable to that of the current top-performing system, that is Tower-70B-v2 equipped with 100-time-sampling MBR/TRR\"as there are still significant gaps towards that from Table 1 which are as big (and sometimes even bigger) than the gain achieved through this approach compared to CPO.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper propose a method to calibrate the likelihood of translation hypotheses with their actual translation quality by optimizing the Pearson correlation between these two metrics during training, which aims to enhance the effectiveness of decoding and improve translation quality.\nThis calibration method involves sampling multiple hypotheses, computing their likelihoods and quality scores using an external metric (e.g., COMET), and minimizing the negative Pearson correlation between these scores.\n\nThe experiments conducted demonstrate significant improvements in translation quality across various metrics and languages when using this calibration method. Additionally, the calibrated likelihoods serve as a capable proxy for translation quality, surpassing some existing quality estimation models.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\nFirst, I think the idea of applying likelihood calibration with quality estimation for machine translation task is novel, the method is simple and clear, which is also orthogonal to SFT method.\n\nSecond, give credit to the author for carrying out extensive experiments across 10 languages, multiple base models (Tower, ALMA), and metrics (COMET, XCOMET, KIWI) demonstrate consistent, significant gains over strong baselines. Human evaluations (200 samples/direction) confirm quality improvements. \n\nWeakness:\n\nI think the paper bears some similarity with the paper \"Calibrating Sequence likelihood Improves Conditional Language Generation\" in ICLR2023 (which the author has already cited.) which proposed a sequence likelihood calibration (SLiC) stage after MLE to enhance the model quality.  Although the metric and task may differ a little bit (this paper focus on MT while the SLiC paper doesn't cover ), I think the core idea is very relevant. So it's kind of bring down the innovativeness a bit.  So I would like to see more analysis on this part, may be a direction comparison in experiment.\n\nAnother small concern is that the translation quality improvement of human evaluation shown in Figure 4 is far more significant than the automatic evaluation improvement in Table1. Does it mean that the automatic evaluation metric like XCOMET and KIWI-XL could not fully reflect the improvement of this method ?  If so, is it problematic (or can be improved) to use COMET as the translation quality score for likelihood calibration.\n\nQUESTIONS:\nShown in weakness part, 1 for the SLiC paper comparison, 2 for the discrepancy between automatic evaluation improvement and human evaluation improvement.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a method to enhance machine translation for LLMs by calibrating the model's hypothesis likelihood with translation quality. At small computational cost, the method substantially improves translation decoding quality. Experiments on two top-performing translation-specialized LLMs (ALMA and Tower, 7B/13B parameters) demonstrate its efficacy. Furthermore, analysis experiments indicate that the model also improves its translation assessment capability.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths：\n1. This paper proposes a simple and effective method presented in clear writing style, which is easy to follow.\n2. The method is computationally economical compared with methods like MBR, and even achieves a higher performance.\n3. Thorough experiments demonstrate significant improvement at low computational cost, reflected in both COMET-based metrics and human evaluation.\n4. Analysis experiments indicate enhanced translation quality assessment ability, further illustrating the effectiveness of translation quality data calibration.\n\nWeaknesses：\n1. The experimental baselines are insufficient, only comparing the single baseline of CPO. Fine-tuning work based on translation quality preferences, such as the work [1] referenced by Section 2, should be considered for inclusion in the main experiments.\n2. The experimental setup mentions that the presented results show the best outcomes under different learning rate settings. Sensitivity analysis regarding learning rates should be added to provide more information about the method’s robustness.\n\n[1] A Preference-driven Paradigm for Enhanced Translation with Large Language Models\n\nQUESTIONS:\n1. The comparison between the MBR of the Tower-70B model and the additionally trained 7B model are not univariate. It is suggested to supplement with an SFT fine-tuned or fine-tuned with proposed method on 7B model to perform MBR, to better control variables and demonstrate the effect of the proposed method when superimposed with MBR.\n2. Supplement experiments or extra explanations for the points mentioned in the weakness part.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addressed a well-known gap in translation decoding, where employing maximum a posteriori (MAP) search often generates lower-quality outputs, as model probability doesn't align well with human judgements. The issue is known as miscalibration issue. To resolve it, the authors proposed calibrating translation likelihoods with quality estimates, specifically, they train a lightweight calibration layer on top of LLMs, optimizing the Pearson correlation between predicted likelihood and reference quality. With just 2k instances per language direction, this quality-calibrated likelihood consistently enhances translation quality that are detected both in automatic evaluation metrics and human evaluations. Tested across 10 languages and on translation-specialized LLMs like TowerMistral, the authors confirmed the consistent gains by using the proposed approach. \n\n- the paper is mostly easy to follow.\n- looks like 2k samples suffice, but I am curious how smaller or larger sample sizes impact performance. If possible, additional experiments in mid/low-resourced directions would be interesting if the page allows.\n- do you have any insights into search dynamics of the proposed method? Compared to MAP decoding, does calibration impact beam search diversity or risk repetitive patterns or hallucination unnecessarily?\n\nSTRENGTHS AND WEAKNESSES:\nstrengths\n- the proposed method works well with minimal data of 2k examples per direction and computational resources of 1-2 H100 GPUs, making it practical and efficient.\n- Experimental results across multiple languages, metrics, and human evaluations demonstrate improvements\n- the authors will likely make available code and model checkpoints, enabling reproducibity\n\nweaknesses\n- unclear to me how well the calibration holds under domain shifts or when scaling to low-resource languages with low-quality noisier signals. It'd be interesting to report the results for mid/low-resourced language directions.\n- this work would benefit from deeper comparisons to related methods like quality-aware decoding [Fernandes et al., 2022]. Have you ever done such experiments?\n\nQUESTIONS:\nplease see the summary.\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}