MBR and QE Finetuning: Training-time Distillation of the Best and Most Expensive Decoding Methods
Authors: Mara Finkelstein
Markus Freitag
Venue: ICLR 2024 poster (2024)
Keywords: Generative models
Natural language generation
Neural machine translation
LLMs
MBR decoding
Quality estimation
Finetuning
Distillation
Primary area: generative models
TL;DR: We show that quality gains from expensive decoding methods for NLG models can be distilled at training time, both via self-distillation and using an external teacher model, while maintaining maximum efficiency at inference time.

Abstract: Recent research in decoding methods for Natural Language Generation (NLG) tasks has shown that MAP decoding is not optimal, because model probabilities do not always align with human preferences. Stronger decoding methods, including Quality Estimation (QE) reranking and Minimum Bayes' Risk (MBR) decoding, have since been proposed to mitigate the model-perplexity-vs-quality mismatch. While these decoding methods achieve state-of-the-art performance, they are prohibitively expensive to compute. In this work, we propose MBR finetuning and QE finetuning, which distill the quality gains from these decoding methods at training time, while using an efficient decoding algorithm at inference time. Using the canonical NLG task of Neural Machine Translation (NMT), we show that even with self-training, these finetuning methods significantly outperform the base model. Moreover, when using an external LLM as a teacher model, these finetuning methods outperform finetuning on human-generated references. These findings suggest new ways to leverage monolingual data to achieve improvements in model quality that are on par with, or even exceed, improvements from human-curated data, while maintaining maximum efficiency during decoding.

Published as a conference paper at ICLR 2024
MBR AND QE FINETUNING: TRAINING-TIME DISTIL- LATION OF THE BEST AND MOST EXPENSIVE DECOD- ING METHODS
Mara Finkelstein and Markus Freitag
Google
{marafin, freitag f com
ABSTRACT
Recent research in decoding methods for Natural Language Generation (NLG) tasks has shown that MAP decoding is not optimal, because model probabilities do not always align with human preferences. Stronger decoding methods, including Quality Estimation (QE) reranking and Mínimum Bayes Risk (MBR) decoding, have since been proposed to mitigate the petppeeettt-ss--uuuaale
mismatch.
While these decoding methods achieve of performance, they are pro- hibitively expensive to compute. In this work, we propose MBR finetuning and QE finetuning, which distill the quality gains from these decoding methods training time, while using an efficient decoding algorithm at inference time. Using the canonical NLG task of Neural Machine Translation (NMT), we show that even with self-training, these finetuning methods significantly outperform the base model. Moreover, when using an external LLM as a teacher model, these finetuning methods outperform finetuming human-genete suggest new ways to leverage monolingual data to achieve improvements in model quality that on par with, data, while maintaining maximum efficiency during decoding.
references. These findings
exceed, improvements from human-curated
INTRODUCTION
Beam search and greedy decoding are the most common decoding methods used for Natural Language Generation (NLG) tasks, including neural machine translation (NMT). However, Eikema & Aziz (2020) showed that maximum a posteriori (MAP) decoding methods, which approximate the most likely prediction based on model probabilities, are suboptimal for NMT due to misaligned probability distributions. They instead proposed Minimum Bayes Risk (MBR) decoding as an alternative decoding method. Unlike MAP decoding, MBR decoding does not aim to produce the prediction (i.c. translation) with the highest estimated model probability. Instead. chooses the prediction that is estimated to have the highest quality with respect to a utility metric. A follow-up study by |Freitag al. (2022a) showed that MBR decoding with neural utility metrics like BLEURT (Sellam et al.. 2020) and COMET (Rei et al. (020) significantly outperforms beam search decoding, according to expert-based human evaluation.
However, the main drawback of MBR decoding is that it is prohibitively expensive. In particular, the algorithm requires that, for every input query, large number n of candidates be generated from the model, and then an (expensive) scoring function be computed on every pair of distinct candidates (n4, n.), for a total of O(n²) computations.
Given the significant quality improvements afforded by MBR decoding relative to beam search, we propose distill the MBR quality gains at training time, without affecting decoding speed or resource usage. Despite its quality advantages, the slow inference speed of MBR decoding remains a limitation even when generating distillation data. As an alternative, we can rerank the same candidate model predictions using a neural quality estimation (QE) metric. Reranking is faster than MBR decoding, because its inference cost scales linearly with the number of candidate predictions, rather than quadratically. Fernandes et al. (2022) showed that reranking with neural metrics produces better predictions than beam search, and that has similar benefits to MBR decoding.

---

Published as a conference paper at ICLR 2024
In this work, focus on the task of NMT, and show that benefit from the quality gains of MBR decoding and QE reranking by finetuning NMT models on MBR-decoded and QE-reranked outputs generated from monolingual sources, either via self-training or using external teacher model, and then using more efficient decoding method (such as beam search) at inference time.
Our contributions can be summarized as follows:
We propose two finetuning methods, MBR finetuning and QE finetuning, each of which distills performance gains from MBR decoding and QE reranking, respectively, training time while avoiding expensive decoding at inference time.
Using the task of NMT, we show that these finetuning methods significantly outperform the base model across two language pairs, while finetuning on beam search output degrades quality.
We show that both MBR and QE finetuning on top of model finetuned on human transla- tions yields additional quality improvements.
We show that using LLM as the teacher model substantially outperforms using self- teacher for MBR and QE finetuning.
Moreover, show that both MBR and QE finetuning from the base student model using LLM teacher even outperforms finetuning on human translations.
MBR DECODING AND QE RERANKING
Broadly, both MBR decoding and QE reranking can be decomposed into two steps:
1. Given source segment, generate list of candidate model outputs. In this work, sampling to generate the candidate translations from a NMT model.
2. Choose the best output based on a utility function. In this work, we use either neural QE or neural reference-ben based metric as utility function.
2.1 CANDIDATE LIST GENERATION
The first step is identical for both decoding strategies. We generate candidate translations using epsilon sampling (Hewitt et al. 2022) with c=0.2, which was shown to be the best sampling method for MBR decoding in Freitag et al. (2023).
2.2 MINIMUM BAYES' RISK (MBR) SCORING
MBR scoring uses the set H of samples obtained from the first step both as candidate translations and as "pseudo-referenees" then uses a eeference-based utility metric to estimate the expected utility of each candidate translation with respect to the set of pseudo-referrncees The candidate with the highest expected utility is chosen the best translation.
That is, given a utility metric u(h, r) which estimates the quality of candidate translation conditioned reference translation r, we select the Minimum Bayes Risk (MBR) translation man from a set hypotheses H. as
1
Σ u(h, v)
arg REW.
H
Freitag al. (2022a) showed that neural utility metries outperform lexical overlap-basee metrics, so we use BLEURT v0.2 (Sellam et al.) 2020) as the utility function Note that passes through the utility function required to compute mmbr is quadratic the size of the candidate set H. In practice, this means that MBR decoding is prohibitively expensive.
the number of forward
2.3 SCORING WITH A QE METRIC
Alternatively, QE reranking uses reference-free utility metric to score the candidate translations, so that rather than requiring an average over all pseudo-referenees compute each candidate's utility,
2

---

Published as a conference paper at ICLR 2024
only single forward pass through the metric model is required. Thus, QE reranking is linear, rather than quadratic, in the candidate size.
Formally, a QE utility metric u(h, s) estimates the quality of candidate translation h conditioned on the source s, rather than on the reference. We select the best QE translation h of the source s from set of hypotheses H.
h = arg max u(h, 8)
BEH
There no QE version of BLEURT, so we instead use MetricX-XXL--OO XXL-QE as our utility function. This metric has the same architecture and was trained on the same human judgements data as MetricX-XXL. the winning submission to the 2M Metrics Shared Task (Freitag et al. 2022b). To make it a QE metric, we pass the source segment as input to the metric instead of the reference.
See Table 5 (Appendix A for meta-evaluation of BLEURT against MetricX-XXI--Q0 on the 22 en en MQM test set. According both segment-level and system-level meta-evaluation metrics, BLEURT and MetricX-XXL-Q6 perform comparably. Also see Figure | for a comparison of the ranking of all sampled translations of single source sentence by MBR scoring with BLEURT (as described in 22.2) versus QE scoring with MetricX-XXLL -xxeX-XXX-QEEE (as described in $2.3). and see Figure] 2 for the MBR and QE score distributions of all sampled candidates versus the top-1 candidate.
MBR AND QE FINETUNING
MBR decoding and QE reranking are both inference-time solutions to the "model pooade--prplexxty-eeeeeeee quality mismatch" problem. We propose MBR and QF finetuming to instead ameliorate this mismatch at training time via direct adaptation of the model weights, without having to incur high costs and resource usage at inference time.
Concretely, the method of MBR finetuning can be decomposed into the following steps:
1. Dataset generation: Generate MBR-decoded translations (see datasource using teacher model T.
2.2 from some monolingual
2. Finetuning:
Finetune student model S the dataset generated in Step 1.
2.1. Variant I: Self-training (T S). The finetuning is initialized from the same model
checkpoint used to generate the MBR-decoded dataset.
2.2. Variant 2: Distillation (T + S). The student model S is finetuned on the dataset generated by the teacher model T.
3. At inference time, decode with the finetuned student model S using efficient decoding method such as beam search, greedy search, or sampling.
QE finetuning is analogous to MBR finetuning, with the only difference being that QE reranking is used as the decoding strategy instead of MBR decoding during dataset creation in Step 1.
4 RELATED WORK
MBR decoding with neural metrics was first proposed by FFeitag et al. (202aa). where it was shown outperform QE reranking when using BLEURT v0.2 (Sellam (020)) and COMET-QE-2/ (Rei et al. 2021) as the QE utility metric. Fernandes et al. (2022) systematically explored different ranking strategies for so-called quality-aware decoding and also found that (top-1) QE reranking underperformed the beam search baseline. However, tuned reranking, where four different QE metrics are linearly combined with weights learned so as to maximize a reference-bast metric on validation set, showed strong improvements over beam search.
MBR utility metric
Shen et al. (2015) proposed training-time approach to directly optimize evaluation metrics, which they call Minimum Bayes Risk Training. They initialized from model already trained the maximum likelihood estimation (MLE) objective and then used an alternative loss function which aligns the model's probability space with the evaluation metric. In particular, for each source sentence, they scored N' samples from the model using the evaluation metric, with the target side of the training example used as the reference. Then the risk for each sample was calculated by multiplying the
3

---

Published as a conference paper at ICLR 2024
metric score with the sample's model probability, and the loss for this example computed as the negative sum of the risks of all samples. So rather than using the single best sample according to the evaluation metric (as in MBR finetuning), they weighted the loss by the risk of all generated samples. Unlike MBR finetuning which uses a static teacher, their (self-)teacher evolves over the course of training as model weights as their evaluation metric (and did not consider neural metrics). Their method also requires references (unlike MBR finetuning), though using Bayes Risk scoring instead would be a natural alternative. The quality improvements achieved with this method were modest, and came at the cost of unstable and expensive training.
updated. They only considered BLEU (Papineni al., 2002)
Gulcehre et al. (2023) proposed a method for offline reinforcement learning-based finetuning called ReST, which generates a finetuning dataset by sampling from the "policy" (i.e. modcl), scoring each sample with QE metric, then iteratively finetuning on these samples using reward-weighted loss based on the QE scores (after removing samples with scores below a fixed threshold, which becomes stricter at later iterations). The method of QE finetuning proposed in this paper, on the other hand, selects the top-1 sample based on QE score. Morevoer, ReST finetuning uses ancestral, rather than epsilon, sampling and samples from the original base training dataset, without exploring the effect of finetuning dataset used on model performance.
Extensive prior work in unsupervised and semi-supervised machine translation has also investigated how monolingual data can be used to improve translation model quality. The techniques of forward and backward translation have resulted from this line of research (Sennrich et al., 2016). A related line of work has focused on distilling large teacher models into smaller student models, by penalizing differences between the translations produced by the teacher and the student (Hinton et al.) 2015). This work has shown that the quality and style of teacher models can be transferred to students (Freitag & Al-Onaizan 2016). However, the use of alternative decoding algorithms to generate distillation data has not been previously explored, and distillation is traditionally performed using an external teacher model. This work addresses both of these gaps, and shows that seff-distillation is effective (only) when the strongest and most expensive decoding algorithms are used.
5
EXPERIMENTAL SETUP
5.1
DATASETS
We perform experiments two language pairs, English-German (high-resource) and English- Japanese (medium-rresuurce)
5.1.1 ENGLISH--GERMAN (EN-+DE)
Base training data The base model was trained on the en- WMT' training data (Akhbardch) et al. 2021). We used all parallel data, after filtering based on length heuristics (removing source sentences longer than 250 tokens and sentences pairs with a source/target ratio exceeding 1.5) and language id, performing deduplication, and normalizing punctuation. We also used the W1 monolingual NewsCrawl data for backtranslation (with a WMT de->en backtranslation model), After filtering, the training dataset had 89,4 million sentences.
Finetuning data We used previous WMT test sets from the years 2009-2019 as finetuning data (Akhbardeh et al.) 2021). Our finetuning dataset had a total of 30,426 sentences. The target side of this dataset is human-translated references, and we finetuned both on these targets, as well as on MBR-decoded and QE-reranked translations generated from the source-side sentences.
5.1.2 ENGLISH-JAPANER (EN (JN-JA)
Base training data The base model was trained on the en-jj WMT'22 parallel training data (Koemi et al. 2022). The data was deduplicated and filtered using CDS Wang al. 2018), where the top 30% of data by CDS score was preserved. After filtering, the training dataset had 8 million sentences.
4

---

Published as a conference paper at ICLR 2024
Finetuning data We used the 20 test set, comprised of 1000 sentences, finetuning data (Kocmi et al. 2022). As with en-*de, we finetuned both on the provided references, and on MBR and QE translations generated from this dataset's source sentences.
5.1.3 MONOLINGUAL DATA
The finetuning datasets for en and en->ja only have 30k and Ik examples, respectively. as these datasets are constrained by the requirement of high-quality human reference translations. Since MBR and QE finetuning do not require references, we also experiment with using monolingual datasources. For that, we take random sample of 200k English sentences from Common Crawl
5.1.4 MBR AND QE DATA
We generate MBR and QE translations both from the en en->de and en ej finetuning datasets 1 and ) and from the monolingual CommonCraw dataset ($3.1.3). We generate 256 candidate translations per source via epsilon sampling (setting e=0.22), and the same set of sampling translations to generate both the MBR and QE data. We use BLEURT (Sellam et al.) 2020) as the MBR utility metric and MetricX-XXI--OF. a QE version of MetricX-XXI, (Freitag al. 2022b) with the same architecture and trained on the same human judgements data, as the QE utility metric. In addition to generating MBR and QE translations as finetuning data, also generate beam search translations from the same datasets as baselines. We use beam size of 4 and length penalty as described in Equation 10 in Wu et al. (2016) with a = 0.5.
5.1.5 DEVELOPMENT AND TEST SETS
For both language pairs, newstest2021 as development set for checkpoint picking, and report all results on the generalMT2022 test set (Koemi et al. 2022).
5.2 MODELS AND TRAINING RECIPE
Student model For both language pairs (en (en->de and en->ja), used 375.4 million parameter Transformer encoder-decoder architecture, implemented in lingvo (Shen et al. 2019). The model architecture is similar the transformer-big setting in Vaswani et al. (2017). with encoder and decoder layers, model dimension of 1024, hidden dimension of 8192, and 16 multi-attention heads. The en en->de model used a bilingual vocabulary of 32k subword units and the en-vja model used multilingual vocabulary of 256k subword units (Kudo & Richardson 2018). The models were trained without label smoothing to avoid distorting the probability distributions when sampling translations for MBR and QE data generation, as it has been found that label smoothing negatively impacts model fit (Eikema & Aziz, (02T). The best (base and incremental) checkpoints were chosen maximize BLEURT on the development set.
Teacher models In addition to self-training, we also experimented with using LLM as the teacher model. Instead of using a (zero-shot or few-shot) prompt- based approach, we finetuned PaLM-2 Bison (Anil et al.] 2023) on the (reference-based) finetuning datasets (see $5.1.1 and 5.1.2 to maximize translation quality.
5.3 EVALUATION
We evaluate our models on six automatic metrics: BLEURT (Sellam et al. 2020). MeIL (Freitag al. 2022b), MetricX-23-c (which resembles MetricX-XXX but uses PaL--2 backbone; Juraska et al. )) chrF (Popović 2015), Comet20 (Rei et al. 2020). and Comer22 (Rei et al. 2022). Since the MBR data generated using BLEURT as the utility function, and the QE data is generated using MetricX-XXL-QE. the MBR- and
models may overfit to the BLEURT metric, while the QE-reraneed and QE-finetuned models may overfit to the MetricX-XN metric. So while these metrics can provide useful signals about the effectiveness of the distillation (e.g. from the MBR-decoded teacher to the MBB-ffineuned student), they cannot be trusted as unbiased measures of model quality.
'htps//coommoncrww.oorg
5

---

Published as a conference paper at ICLR 2024
To measure model quality, instead rely Comet20. Note that BLEURT and Comet20 were finetuned on the same human judgements data, as were MetricX-XXXL and MetricX-23-c, so these pairs of metrics may tend to be highly correlated with each other. Thus, also report MetricX-23-c as counterbalance to any Comef20-specific biases, which may tend to favor models optimized for BLEURT (e.g. MBR-decoded and MßR--inetuned models). We also verify the trustworthiness of Comer20 with human evaluation study (see (ection(5.3.11 Unless otherwise indicated, all evaluation results for the finetuned models are reported using beam search the decoding strategy (with the same hyperparameters as in $.1.4..
5.3.1 HUMAN EVALUATION
We hired professional translators and measured translation quality with a document context version of MQM (Lommel et al. 2014) which mimics the setup proposed in Freitag et al. (2021). This includes using the same error categories, severity levels, and error weighting schema. We assign weight of 5 to each major error and 1 to each minor error, except for minor punctuation errors which receive a score of 0.1. The final segment-level score is an average over scores from all annotators. We refer the reader to Freitag al. (202T) for the details error categories and annotator instructions.
5.4 EXPERIMENT ARMS
We performed experiments in three phases. See Table (Appendix) [B for a summary of the datasets used per phase, and see Table for the time and compute costs of generating the MBR and QE data.
Phase 1 Finetune from the base checkpoint using the finetuning datasets described in $5.1.1 (for en-*de) and $5.1.2 (for en->ja) This phase allows for a direct comparison of MBR and QE finetuning against finetuning on human-generated references. In the remaining phases, large corpora of unpaired monolingual corpora are used to generate the MBR and QE datasets, so no comparison against reference-based finetuning is possible.
Phase Finetune from both base and incremental (reference-finetueed) checkpoints using self- MBR and self-QE data generated from the monolingual Common Crawl corpus described in $5.1.3 In this phase, we investigate whether using a larger corpus of source-side data to generate MBR and QE translations yields additional quality improvements over the finetuned models from Phase 1, and the extent to which this closes the performance gap with finetuning on references.
Phase 3 Finetune from both base and incremental sefference-finttunne checkpoints using MBR and QE data generated from the PaLM-2 Bison teacher model. In this phase, we investigate whether using teacher model which is stronger than the student affords further quality gains.
RESULTS
6.1 EN->D
The Comet20 results from all experiment phases are summarized in Table 2 We observe that QE reranking (Ib) and MBR decoding (Ic) using the base checkpoint both outperform beam search decoding (1a). This establishes the premise that the MBR and QE "teachers" used to generate the MBR and QE finetuning data are stronger than the beam search bascline. Also see Table (Appendix C.I for results across all automatic metrics (as described in {5.3), see Table 1 for breakout of the Comet20 results by domain, and see Table 10 the performance of the PaLM-2 teacher.
Model
Comet20 f
MQM score
Ia) Beam search decoding (base model)
57.79
1.422
1b) QE reranking (base mudel)
59.21
1.289
Ig) Reference-finetuned
60.95
1.183
2d) Fir-OO--from-re--iitttttee
61.11
1,099
36) Pal. 1e-2--0Ffroom-rf--fieeeeee
62.35
1.085
Table 1: Results of MQM study WM test Lower MQM scores
better (indicating fewer errors).

---

Published as a conference paper at ICLR 2024
Model
COMET20
Oa) WMT 2022 Top Submission ((DExploreAcaademy)
63.26*
la) Beam search decoding (base model)
57.79
1b) QE reranking (base model)
59.21 (+1.42)
1c) MBR decoding (base model)
59.35 (+1.56)
Id) Ream-finetuned
57.08 (-0.71)
le) MBR-finetund
58.02 (+0.23)
If) QE-finetuned
59.63* 59.63*++1844)
Ig) Reference-finetuned
(+3.16)
2a) Self-MBR-froo finetfiie
59.94* (+2.15)
2b) beaa-froo-rel-fieeeeee
60.06 (-0.99)
2c) Self-QU QE-from-buse-fineeee finetunedd
60.49* (+2.70)
2d) Scll-QEEfrom-reef cefoom--rf-fnntuneddd
61.11 (+0.16)
2e) Self-MBR-from-rcf- -fn-nttuedd-
61.49 (+0.54)
3a) PaLM-2- MaLLM-2--reedy-fuee fronyy-rrorrrrrre rent-r-efffttteed
61.61 (+0.66)
3b) PaLMM22- 2--E--from: from-buse Puuse-fnnttune
62.344 (+4.55)
3c) -er------QE--too--e----nneeeee
62.35* (+1.40)
3d) PaL.M-2-MBBR-rrmmm
63.51* (+5.72)
3e) www.--2--MMR--roo------eeeentm
63.62* (+2.67)
Table 2: Results en en- WMT'22 test set. Asterisks indicate scores which significantly better than their baseline and parentheticals indicate delta from baseline. For models finetuned from the base checkpoint, their baseline is row la) and for models finetuned from the reference-finetuned checkpoint, their baseline is row 1g).
Finetuning on references g) achieves the highest Comet20 score (with a gain of 3.16 points over the base model), though MBR and QE finetuning (le and If. respectively) still achieve gains of 0.23 and 1.84 points against the base model, respectively, in contrast to beam finetuning (Id) which degrades performance.
Phase 1
Phase 2 MBR and QE finetuning from the base checkpoint on the larger Common Crawl dataset (2a and 2c, respectively) both yield gains against the base model, and outperform the Phase 1 finetuning on the smaller dataset (le and If, respectively). Moreover, performing a second round of finetuning from the reference-fineturee finetuned checkpoint either Crawl data (2d and 2e, respectively) achieves further gains in Comet20. So MBR and QE finetuning from large monolingual corpus can complement finetuning small dataset of high-quality human references. Also note that beam finetuning (using the same source-side Common Crawl dataset) degrades performance relative the initial enference-finetunee checkpoint.
or self-MBR-Common
Phase 3 Using QE and MBR finetuning data generated by the PaLM-2 Bison teacher (3b and 3d, respectively) outperforms using the self-teacher model (2c and 2a, respectively) by large margin (of 1.85 and 3.57 points, respectively) and, in fact, outperforms finetuning on references (by 1.39 and 2.56 points, respectively). Recall that the PaLM-2 Bison teacher was finetuned on references. but the student model never directly saw them. Moreover, performing a second round of QE or MBR finetuning from the reference-finetuned checkpoint using the PaLM-2 Bison teacher (3c and 3e, respectively) yields further performance improvements, with the additional large gain of 2.67 points on top of the inference-fineturne model, outperforming all other models including the winning WMT'22 submission.
model achieving an
Human Evaluation: MQM Study We perform a human evaluation to compare the "from-ref" QE finetuned systems 2d) and 3e) in Table (using self-teacher and PaLM-2 teacher, respectively), against the beam search decoded and QE-reranked base model (la and 1b, respectively). as well as the reference-finetunel model (Ig) from which QE finetuning was initialized. As shown in Table [ the ranking of the systems according to human evaluation agrees with the ranking according to Comer20. Both QE finetined systems outperform the feferrnce-finetunee baseline, and QE finetuning using the PaLM-2 teacher outperforms using the self-teacher.
6.2
EN JJA
We then investigate whether the gains from QE finetuning also extend resoure setting. Recall that the human reference-based finetuning dataset for English-Japanese only has 1K sentences,
7

---

Published as a conference paper at ICLR 2024
while monolingual English data is widely available (to use for generating Japanese MBR and QE translations). The results for all en en- experiment phases are summarized in Table 3 and the trends align with those observed for en en-
Model
COMET20
Oa) WMT 2022 Top Submission (NT3)
64.15*
Ia) Beam search decoding (base model)
47.04
1b) Reference-finetuned
50.96* (+3.92)
Ie) QE-finetuned
51.39* (+4,35)
Id) Beam- finetuned
47.04 (+0.00)
2a) Salf-QE-from-bee base-finetuned
53.73* (+6.69)
2h) Self-QF-from--e cefrro--rrfffiituneee
56.69* (+5.73)
3a) PaLM----EEffrre -E-fro---ffe--eee from-base- finettine
60.63* (+13.59)
3b) (ea-------ttto-e---iiiee
60.58* (49.62)
Table 3: Results en-*ja WMT'22 test set. Asterisks indicate scores which significantly better than their baseline and parentheticals indicate delta from baseline. For models finetuned from the base checkpoint, their baseline is row la), and for models finetuned from the diference-finetuned checkpoint, their baseline is row lb).
Phase QE finetuning (Ic) achieves 4.35 Comet20 performance improvement over the base model (la) and, even more notably, outperforms finetuning on human references (1b) by 0.43 points. The baseline of beam finetuning (Id), on the other hand, degrades performance relative to the base model. Note that for beam finetuning, the max-BLEURT checkpoint on the development set was the initial checkpoint, so the performance reported in row Id) matches that of the base model (1a), as per our checkpoint selection procedure.
Phase 2 QE finetuning from the base checkpoint (2a) on the larger Common Crawl dataset (200k sentences, versus 1 from Phase I) achieves even stronger performance (better by 2.34 points), outperforming the reference-fineture reference-finee finetuned checkpoint (2b) achieves additional large performance improvements, beating the ffference-iinttueed model by 5.73 points.
model (Ib) by 2.77 points. Moreover, QE finetuning from the
As with en end the largest performance improvements are observed when using the PaLM- 2 Bison teacher, rather than the self-teachet setup from the preceding phases. Here, we that QE finetuning from the base checkpoint using the Pal.M-2 Bison teacher (3a) dramatically outperforms finetuning references (1b) by 9.67 points, while QE finetuning from the eeference-finetune checkpoint (3b) does not afford any additional performance improvements.
Phase 3
6.3 ABLATION STUDY: WHAT IS THE EFFECT OF THE DECODING STRATEGY ON THE PERFORMANCE OF MBR FIBR-FINETUNED AND QF-FINETUNED MODELS?
We present a single ablation study here, and refer the reader to Appendix D for a comprehensive battery of ablation studies covering the effects of the type and size of the (source-side) monolingual dataset and of candidate size used for MBR and QE data generation on the performance of the finetuned model, as well as the performance implications of finetuning on back-translated MBR and QE data, mixtures of MBR and QE data, and iterative finetuning.
All MBR-finetuned and QE-finetuned model results reported so far use beam search decoding. The techniques of MBR and QE finetuning rely on using an efficient decoding algorithm, but this does introduce mismatch between the decoding algorithm used to generate the finetuning data and the decoding algorithm used by the student at inference time. Greedy search and sampling are also efficient decoding algorithms, so we compare their performance against that of beam search, using the best en ee HBR-ffietuned and QE-finetuned models from each experiment phase. We use epsilon sampling with e = 0.02 as the sampling strategy. As shown in Table 4 beam search outperforms the other decoding methods for the Phase models, while the even-more-eeficien greedy decoding is a close contender and actually outperforms beam search for the Phase 2 and Phase 3 Q0E-finetnee models. Sampling decoding, on the other hand, lags well behind beam search and greedy decoding. Also see Table in Appendix[D]ot a comparison of how MBR decoding and QE reranking compare against these more efficient decoding strategies for MBR-finetuned and QF-finetuned models.
8

---

Published as a conference paper at ICLR 2024
Model
QE-finetuned
MBR-finetuned
Beam
58.02
Phose
Greedy
59.58
57.61
Sampling
55.91
$7.69
61.11
61.49
Phose 2 (from-re) ref)
Greedy
61.19
60.93
Sampling
57.80
Beam
62.35
63.62
Phase 3 (from-reef)
Greedy Sampling 56.22
62.52
62.65
58.92
Table 4: Performance of en en-*de QE and MBR-finetuned models when using different decoding strategies.
7 DISCUSSION
High-quality datasets of transaatt references are expensive to curate. We have seen that, for both high-resource (en-xde) and medium-resource (en-xja) language pair, MBR and QE finetuning lead to significant improvements over the base model. Moreover, self-MBR and self-QE finetuning on top of the eeference-finetune model (already strong baseline) further boost performance, while performance improvements are not observed when finetuning on self-beam translations. We also showed that using an external LLM teacher to generate MBR QE finetuning data leads even larger performance improvements and, in fact, outperforms finetuning on human-translatt translated references.
Given the effectiveness of an external teacher model (which differs in architecture, training data, etc. from the student) relative to self-teacher, the success of these finetuning methods can likely not be primarily explained by mitigation of the label exposure bias problem (Schmidi (2019). Instead, the effectiveness of MBR and QE finetuning seemingly can be attributed primarily to the high quality of the translations used as finetuning data.
When comparing MBR against QE finetuning, we see that QE tends to perform at least as well as MBR finetuning using self-teacher, while MBR outperforms QE finetuning using the LLM teacher. We hypothesize that the sampling translations from the self-teacher do not closely approximate the true reference distribution, so using them as pseudo-references does not help (or even hurts), relative to using a reference-free free (QE) metric. For higher-quality models (e.g. PaL-M-2), the candidate translations are good enough approximations references that they provide a useful signal.
MBR and QE finetuning shift the MBR and QE quality gains to training time, with the expectation that an efficient decoding strategy be used inference time. When evaluating the finetuned models, we showed that beam search and greedy decoding perform comparably, and outperform sampling. Note that these decoding methods incrementally continue a partial translation hypothesis, while MBR decoding and QF reranking have the advantage that they operate on full translation hypotheses. Given that MBR and QE finetuning improve performance even using efficient decoding methods, these finetuning techniques are indeed able to help the model make better local decisions during decoding.
8 CONCLUSION
In this work, we have proposed the techniques of MBR finetuning and QE finetuming to distill the quality gains from MBR decoding and QE reranking, which are state-of-the-ar decoding methods for NLG models, but are prohibitively expensive to compute. We have shown that MBR and QE finetuning lead consistent quality improvements over the base model, even when using self- teacher, and also achieve additional performance improvements top of human reference-finetuned models. Furthermore, using a LLM as the teacher model substantially outperforms using self-teacher and, in fact, MBR and QE finetuning using LLM teacher outperforms finetuning most language pairs, (source-side) monolingual data is available in larger quantities than parallel data. MBR and QE finetuning open up the possibility of leveraging monolingual data to achieve substantial gains in model performance both through self-training and distillation from a stronger teacher model.
references. For
There are many avenues for future work, including extending MBR and QE finetuning other NLG tasks besides NMT, more comprehensively investigating the effect of the utility function used on MBR and QE finetuning performance, experimenting with few-shot generation of MBR and QE data from a LLM teacher (removing the need for finetuning the LLM), and document-level MBR and QE finetuning.

---

Published as a conference paper at ICLR 2024
9 REPRODUCIBILITY STATEMENT
This work was performed on open-source WMT datasets (Akhbardeh et al. using the open-source lingvo library (Shen et al. 2019).
2021 Kocmi et al.
2022)
REFERENCES
Farhad Akhbardeh, Arkady Arkhangorodsky, Magdalena Biesialska, Ondiej Bojar, Rajen Chatterjee, Vishrav Chaudhary, Marta R. Costa-jussa, Cristina Sspaña-Bonet, Angela Fan, Christian Feder- mann, Markus Freitag. Yvette Graham, Roman Grundkiewicz, Barry Haddow, Leonie Harter, Kenneth Heafield, Christopher Homan, Matthias Huck, Kwabena Amponsah- Kaaiaaire,aairee,ee, Jungo Ka- sai, Daniel Khashabi, Kevin Knight, Tom Koemi, Philipp Koehn, Nicholas Lourie, Christof Monz, Makoto Morishita, Masaaki Nagata, Ajay Nagesh, Toshiaki Nakazawa, Matteo Negri, Santanu Pal, Allahsera Auguste Tapo. Marco Turchi, Valentin Vydrin, and Marcos Zampieri. Findings of the 2021 conference on machine translation (WMT21). In Proceedings of the Sixth Conference on Ma- chine Translation, pp. 1-88, Online, November 2021, Association for Computational Linguistics. org 2021. wmt-1.1
URL https://aclanttholl//
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent EI Shafey, Yanping Huang, Kathy Meier-Hellsterm, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette- Choot, Aakanksha Chowdhery. Clément Crepy. Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Diaz, Nan Du Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sncha Kudugunta, Chang Lan, Katherine Lee. Benjamin Lee, Eric Li, Music Li. Wei Li. YaGuang Li, Jian Li, Hyeontack Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter. Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang. Pidong Wang, Zirui Wang. Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. Palm 2 technical report, 2023.
Daniel Deutsch, George Foster, and Markus Freitag. Ties Matter: Modifying Kendall's Tau for Modern Metric Meta-Evaluation, 2023. URL httptttsss//riv..too/ wttgg//././/...../ 14324.
Bryan Eikema and Wilker Aziz. Is map decoding all you need? the inadequacy of the mode in neural machine translation. arXiv preprini arXiv:2005.10225. 2020.
Bryan Eikema and Wilker Aziz. Sampling- -based approximations to minimum bayes risk decoding for neural machine translation. arXiv preprint arXiv:2108.047188 2021.
Patrick Fernandes, António Farinhas, Ricardo Rei, José GC de Souza, Perez Ogayo, Graham Neubig, and André FT Martins. Quality-aware decoding for neural machine translation. arXiv preprint arXiv:2205.00978 2022.
Markus Freitag and Yaser AI-Onaizan. Fast domain adaptation for neural machine translation. arXiv preprint 1FXv::6612.0699, 2016.
Markus Freitag, George Foster. David Grangier. Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey, Experts. errors, and context: A large-scale study of human evaluation for machine translation.
Transactions of the Association for Computational Linguistics, 9:1460-1474 1474, 2021. doi: 10. 1162/ tacl 000437.
wrwp/221..1100000 t.c1-1.87
10

---

Published as a conference paper at ICLR 2024
Markus Freitag, David Grangier, Qijun Tan, and Bowen Liang. High quality rather than high model probability: Minimum bayes risk decoding with neural metrics. Transactions of the Association for Computational Linguistics, 10:81 1 1-825, 2022a.
Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo. Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and André FT Martins. Results of wmt22 metrics shared task: Stop using bleu-neural metrics are better and more robust. In Proceedings of the Seventh Conference on Machine Translation (WMT), pp. 68, 2022b.
Markus Freitag, Behrooz Ghorbani, and Patrick Fernandes. Epsilon sampling rocks: Investigating sampling strategies for minimum bayes risk decoding for machine translation. arXiv preprint arXiv:2305.0986h 2023.
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308. 08998, 2023.
John Hewitt, Christopher D Manning, and Percy Liang. Truncation sampling as language model desmoothing. arXiv preprint arXiv:2210.15 15191, 2022.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint XX3v:1503.00531,, 2015.
Norman P Jouppi, Doe Hyun Yoon, George Kurian, Sheng Li, Nishant Patil, James Laudon, Cliff Young, and David Patterson. A domain-specific supercomputer for training deep neural networks. Communications of the ACM, 63(7)667-78, 2020.
Juraj Juraska, Mara Finkelstein, Daniel Deutsch, Aditya Siddhant, Mehdi Mirzazadeh, and Markus Freitag. Metricx-23: The google submission to the wmt 2023 metries shared task. In Proceedings of the Eighth Conference on Machine Translation, pp. 754-765, Singapore, December 2023. / https////acaaatooogoooooo orrgaaanntoooooooor/22233 org.2023.3222 wmt 1.63
Association for Computational Linguistics. URL https:
Tom Kocmi, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grundkiewicz. Barry Haddow, Rebecca Knowles, Philipp Koehn, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Michal Novák, Martin Popel. and Maja Popović. Findings of the 2022 conference on machine translation (WMT22). In Proceedings of the Seventh Conference on Machine Translation (WMT), pp. 1-45, Abu Dhabi, United Arab Emirates (Hybrid), December 2022. Association for Computational Linguistics. URL htttttts//claatthooo.oo/. org/ 2022. 20222wmmt1..... 1.1
Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv: 06226 2018.
Arle Lommel, Hans Uszkoreit, and Aljoscha Burchardt. Multidimenniioaal quality metrics (MQM) A framework for declaring and describing translation quality metrics) Revista Tradumâtica: tecnologies de la tradueció, 2014.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Maja Popović. chrF: character n-gram F-score for automatic MT evaluation. In Ondrej Bojar, Rajan Chatterjee, Christian Federmann, Barry Haddow, Chris Hokamp, Matthias Huck, Varvara Logacheva, and Pavel Pecina (eds.), Proceedings of the Tenth Workshop Statistical Machine Translation, pp. 392-395, Lisbon, Portugal, September 2015. Association for Computational URL/httpss//accaanthoooo wwtpt//ooooooooooo////000000 3rg/
Linguisties. doi: 10. 10.18653/v1/115-11
Ricardo Rei, Craig Stewart, Catarina Farinha, and Alon Lavie. Unbabel's participation in the wmt20 metrics shared task. arXiv preprint arXiv:2010. 15535, 2020.
Ricardo Rei, Ana C Farinha, Chrysoula Zerva, Daan van Stigt, Craig Stewart, Pedro Ramos, Taisiya Glushkova, André FT Martins, and Alon Lavie. Are references really needed? unbabel-ist 2021 submission for the metrics shared task. In Proceedings of the Sixth Conference on Machine Translation, pp. 1030-1040, 2021.
11

---

Published as a conference paper at ICLR 2024
Ricardo Rei, José GC De Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André FT Martins. Comet-22: Unbabel-ist 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference Machine Translation (WMT), pp. 578-585, 2022.
Florian Schmidt. Generalization in generation: A closer look at exposure bias. arXiv preprint arXiv:1910.00222, 2019.
Thibault Sellam, Dipanjan Das, and Ankur P Parikh. Bleurt: Learning robust metrics for text generation. arXiv preprint arXiv:2004.046688 2020.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation mod- els with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 86-96, Berlin, Germany, Au- gust 2016. Association for Computational Linguistics. doi: 10.18653/1/P16-1000 URL https://aalaaathoooy..or/////
Jonathan Shen, Patrick Nguyen, Yonghui Wu, Zhifeng Chen, Mia Chen, Ye Jia, Anjuli Kannan, Tara Sainath, Yuan Cao, Chung-Cheng Chiu, et al. Lingvo: modular and scalable framework for 2019.
sequence-to--seqence modeling. arXiv preprint
Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. Minimum risk training for neural machine translation. arXiv preprint Se5ivvv5//2.22555 2015.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017
Wei Wang, Taro Watanabe, Macduff Hughes, Tetsuji Nakagawa, and Ciprian Chelba. Denoising neural machine translation training with trusted data and online data selection. arXiv preprint arXiv:1809.00068 2018
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc Le. Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao,, Qin Gao, Klaus Macherey, al. Google's neural machine translation sys- tem: Bridging the gap between human and machine translation. arXiv preprint arXiv: /609.08144, 2016.
A MBR AND QE UTILITY FUNCTIONS
Seg-Level
Sys-Level
Metric
acc*
acc
BLEURT
57.5
76.9 76.9
METRICX- -XX--EE
59.1
Table 5: Segment-level and system-level meta-evaluation results on the WMT 22 MQM en de test set.
Table shows the eetta-vvaluation of the (reference-based BLEURT metric against the (reference- free) MetricX-XXx XXL-QE metric. We measure the metric performance with respect ground translation quality ratings using the benchmark WMT'22 en->de MQM dataset. At the segment-level, we report the "group-by-item" variant the pairwise accuracy meta-evaluation metric proposed by Deutsch et al. 2023 Note that random metric would achieve 33% accuracy. We also report system- level pairwise accuracy. According to both the segment-level and system-level meta-evaluation metrics, BLEURT and MetricX-XXL-4 perform comparably.
Figures [and] present additional comparisons of the scores generated via MBR decoding versus QE reranking.
12

---

Published as a conference paper at ICLR 2024
OE MBR
90
80
MBR: 1000
60
-6
50
30
100
150
Figure 1: QE vs MBR scores for all 256 candidate translations generated from single source sentence in the en-tde newstest2009-200 2019 dataset. The ranking of examples by QE score is not the same as the ranking by MBR score.
10P
cands
qP. bleurtx, topl_cands
cands
mbr bleurt top1 cands
10
10
107
Count
10
107
10
10P
10a
0
20
40
60
80
100
0.0
0.2
0.4
0.6
0.8
1.0
(a) QE
(b) MBR
Figure QF and MBR score distributions for all candidates versus the top-1 candidate (for candidate 2izc=256), generated from the (source-side) en-tde News 2009-2019 finetuning set.
B MBR AND QE DATASET STATISTICS
See T for summary the (source-side) datasets used to generate MBR and QE data during each of the three experimental phases. Note that during Phase and Phase 2, the "self-teacher" was used, and during Phase 3, the PaLM-2 Bison teacher was used.
Base training
Phase
Phase
Phase
WMT 2021 parallel training data (fittered
WMT 2009-2019 (newstest) test sets 30,426
Common Crawl 200,000
Common Craw! 200,000
en-ile
89.8 4 million
WMT 2022 paraltel training data (fittered
WMT test set 1,000
Common Crawl 200,000
Common Crawl 200,000
en en-jja
million
Table 6: Training and finetuning datasets used, and their respective sizes, at each experimental phase.
The bottleneck in generating data for MBR and QE finetuning is in the candidate scoring stage. Table 7 shows the time and compute requirements for performing MBR-BLEURR and QE-MetricX-XXX scoring, across all three experiment phases and across both language pairs. We TPUs for BLEURT and MetricX-XX0 XXL scoring, as described in Jouppi et al. (202)). The TPUv4 hardware is faster and newer generation than TPUv3. MBR scoring takes about 7.65 TPUv4 minutes/source sentence, while QE scoring takes about 4.2 TPUv3 seconds/source sentence. So even when using faster hardware for MBR, QE scoring 109x faster.
13

---

Published as a conference paper at ICLR 2024
Phase
Phase
Phase
en en->de
WMT 2009-2019 test sets
Common Crawl
Common Crawl
MBR
7.8 hours (500 TPUv4) 1.1 hours (32 TPUv3)
5.1 hours (5000 TPUv4) 7.3 hours (32 TPUv3)
5.1 hours (5000 TPUv4)
QE
7.3 hours (32 TPUv3)
en en-jja
WMT 2020 test set
Common Crawl
Common Crawl
QE
2.2 minutes (32 TPUv3)
7.3 hours (32 TPUv3)
7.3 hours (32 TPUv3)
Table 7: Wallclock time for scoring step during MBR and QE data generation. TPUv4 is newer and fäster generation than TPUv3 (Jouppi al. 2020).
C ADDITIONAL SYSTEM COMPARISON
C.I EN-DD RESULTS
Table shows en end results across all metrics (while the main text only reports Come120).
Model
BLEURT
MetrieX-XXI.
MetrieX-23.c
chrlf
COMET22
COMET20
On) WMT 2022 Top Submission (DEpploreAcssdmmy))
78.38
82.71
-0.61
86.04
87.41
63.26
Ja) Boam search decoding (base model)
76.78
80.52
-0.73
82.54
85.84
57.79
1b) QU reranking (base model)
78.01
82,92
.0.55
82.83
86.56
59.21
1c) MBR decoding (base model)
81.09
82.29
-0.67
82.73
86.38
59.35*
Id) Beam-finetuned
76.14
79.99
-0.77
82,76
85.65
$7.08
1e) MBR-finetined
77.34
80.36
-0.75
83.46
58.
If QE-finetined
77.24
81,25
-0.70
83.33
86.34
59.633
1g) Reference finetuned
2a) Self MBR from base finetuned
77.79
81.42
.0.99
83,51
86.25
59 94
2c) Self-QE-from-base ba---------nnteeed
77.57
81.43
-0.70
84.27
86.10
60.49
24) 1iE-from-ref-fnnttue
77.86
81.63
-0.67
83.53
86.75
61.11
2e) Sel--MBRR MBR from ref finetumed
78.50
81.72
-0.66
83.45
86.76
61.49
3a) PaLM A-2 gregddy from ref. finetuned
77.60
81.62
-0.69
83.46
86.73
61.61
3b) FaLM-2-QE-har DEE-Irom-busee base-finetuned
78.21
82.22
-0.66
84.14
86.85
62.34*
3e) M-------ftton-t---mee
78.26
82.15
-0.65
84.10
87.11
62,35"
3d) C--M---roo--bse---------------
79.35
82.82
3e) PaLM-2-MBB -------------------
79.39
82.85
-0.63
84,51
87.29
63.62
Table 8: Results on English-Germe 22 test set across metrics. Asterisks Comer20 column indicate scores which are significantly better than their baseline. For models finetuned from the base checkpoint, their baseline is row la), and for models finetuned from the Ig).
checkpoint, their baseline is row
Table shows the en- Comet20 results broken out by domain (Conversation e-Commerce, News, and Social).
Model
Conversation
e-C Commercce
News
Social
WMT 2022 Top Suhmission (IDFEspleeAAcukmy)
la) Beam search decoding (base model)
63.49
62.51
59.61
$5.69
1b) QE reranking (base model)
16) MBR decoding (hase model))
64.90
62.98
60.88
48.53
1d) Beam- finetuned
62.45
58.97
1a) MBR finetuned
62.68
62.08
60.74
46 46.58
10 OR-Risetuned
62.
64.60
60 94
49
Refirenne-finetuueel
24l Sel--MRR-Freee waie---oo--nhs-tetttteee
66.53
63.99
60.91
49 30
26) fiff-beans-f dinctuned
24) -o--foon--sse---nnntee
66.
63.99
62. 17
49 65
240 Self-QF--fru--eef---e--e
66. 61
2e) Self MBR from ref finetuned
68.7
61.76
51.16
3xi PaLM greedy from finetuned
68.1
65.39
50.61
3b) PaLM-22QQE 2--Qu--meee from hase dinetuned
67.08
52.46
3ci PaLM-2-C8 from ref finetuned
68.07
65.60
63.00 41
52.
dinetuned
3c) WW--MMBfffoo-ff-ieee
21.10
66.44
03 96
52.98
Table Comer20 scores on Gnelish-German WM test set, broken out by domain.
14

---

Published as a conference paper at ICLR 2024
Table 10 shows the performance of the en->de PaLM-2 greedy-decoded, QE-reranked, and MBR- decoded teacher models.
Model
BLEURT
MetrieX-XXI.
COMET20
Pal M-2 greedy decoding
78.81
83.59
63.23
PaLM-2 QE reranking
78.77
84.29
60.86
PaLM-2 MBR decoding 82.43
84.73
63.76
Table 10: Performance of greedy-decoded, QE-reranked, and MBR-decoded en-ide PuLM-2 teacher models on the WMT'22 test set.
Figure 3 shows comparison of the similarity of the Phase 1 en- self-MBR and self lef-QE finetuned models against their teachers, as measured by cross- cross-BLEU on the WMT test set. Note that the similarity between each teacher and respective student is lower than the similarity between the finetuned models.
100
mbr
29.8
100
22.3
29.9
27.8
29.9
29-9
nbr finetuned
98
31.2
69 6
100
91.4
99
beam, finetuned
08.4
30.1
69 7
91.7
100
99.4
mbr
fintuned
02 thituned
fictured
at finetueed
Figure 3: Cross-BLBU of en-tde models on WM test set, as a measure of model similarity.
C.2 EN-JA RESULTS
Table 11 shows en en-+ja results across all metries (while the main text only reports Comel20).
Model
BLEURT
MetrieX-XXI.
MetrieX-23-e
chrF
COMET22
COMET20
WMT 2022 Top Submission (NT5)
69.49
83.24
-0.62
95.15
89.26
64.15*
Beam search decoding (base model)
64.69
77.28
-0.95
91.19
86.03
47.04
1b) finctuned
65.70
78.13
-0.92
92.01
86.81
50.96*
1c) QH-finetuned
65.71
78.35
-0.89
94.30
86.67
51.39*
1d) Beam- -inettidd
64.69
77.28
-0.95
91.19
86.03
47.04
Self- Self-QE- -DE-rrom-base-fiee baw-----nffnedeed
65.52
78.36
-0.84
94.46
87.21
53.73*
2b) Tel---EE-rro--r--fiettteee
66.56
79.31
-0.81
94.32
87.76
56.69*
Pal M---M--------- from--ase-Heee mase-finetuned 3b) PaLM-2-QEE Pe--Q--ffo-reee from--ceffinetuued finetuned
68.00
80.51
-0.77
95.40
88.5
60.63*
67.94
80.35
-0.78
95.30
88.48
60.58*
Table 11: Results on Jappncsee WMT'22 test set across all metrics. Asterisks in the Comer20 column indicate scores which are significantly better than their baseline. For models finetuned from the base checkpoint, their bascline is row la), and for models finetuned from the reference-finctuned checkpoint, their baseline is row lb).
Table 12 shows the performance of the en -ja PaLM-2 greedy-decoded, QE-reranked, and MBR- decoded teacher models.
15

---

Published as a conference paper at ICLR 2024
Model
BLEURT
MetricX-XXL
COMET20
PaL.M2 greedy decoding
71.51
85.56
70.25
PaLM-2 QE reranking
70.17
86.37
67.52
PaLM-2 MBR decoding 75.64
86.44
72.69
Table 12: Performance of en en-+ja greedy-decoded, QE-reranked, and MBR-decoded Pa/M-2 teacher models on the WMT'22 test set.
C.3 DE RESULTS
We show Phase results for German- Englishh in Table 13 finetuning using the (source side of the) WMT 2009-2019 test sets (30,695 examples total; Akhbardeh al. (20TT), and compare this against finetuning on the references and on beam translations. We use the same model architecture as for en->de and en-ija, described in 5.2
We perform MBR finetuning and QE
Model
BLEURT
MetricX-XXL
chrF
COMET22
COMET20
Ia) Base model
72.96
78.03
76.55
84.17
51.77
1b) Beam-finetuned
72.97
78.02
75.94
84.18
51.77
Ic) MBR-finetuned
73.62
78.45
76.74
84.38
53.31
1d) Eeference-finetung
73.64
78.76
76.00
84.67
54.15
1e) QE-finetuned
73.86
78.99
76.54
84.81
55.10
Table 13: Results on de-ten 2M test set. Beam search used as the decoding strategy for all models.
We find that the trends this (into-English) language pair align with those observed for the other (outoof-English) language pairs in our study. In particular, we observe that MBR finetuning and QE finetuning both improve performance relative to the base model, while beam finetuming does not. Moreover, QE finetuning outperforms finetuning references (as also observed for en jn)).).
D ADDITIONAL ABLATION STUDIES
There are many source-side and target-side variables which may affect the performance of MBR and QE finetuning. We perform ablation studies (on en-xde) to isolate the effects of several of these variables. On the source side, we consider the effect of the type and size of monolingual dataset used for MBR and QE data generation. On the target side, we investigate the effect of candidate size used for MBR and QE data generation the performance of the finetuned model. We also explore finetuning on back teeessaasle MBR and QE data, mixtures of MBR and QE data, as well as iterative finetuning.
D.0.1 DOES THE MONOLINGUAL DATASET USED FOR MBR AND QE FINETCNING MATTER?
To probe the effect the source-side dataset on the downstream performance MBR QE finetuning, we sample 30k sentences from the NewsCrawl (Akhbardeh et al. 2021 and Common- Crawl (.3.1 datasets, then use the en en->de refcrence-finetuned checkpoint to generate MBR and QE translations. which are used for second round of finetuning. As shown in Table 14 choice of dataset matters very little. The NewsCrawl dataset has a small advantage over CommonCraw/ in Comet20 scores for the
model, while the reverse is true for the QB-finetueed model.
Model
BLEURT
MetrieX-XXI.
COMET20
MBR-finetund (NewsCrawl)
78.11
81.85
60.60
MBR-finetuned (CommonCrawl)
78.33
81.72
60.06
QE fe-fiintuned (NewsCrawl)
77.64 77.70
81.57
59.40
QF finetuned (CommonCrawl)
81.53
60.62
Table 14: Comparison finetuning results using sentences sampled from NewsCrawl versus CommonCrawl datasets.
16

---

Published as a conference paper at ICLR 2024
We also investigate the extent to which dataset size affects model performance, by finetuning the entire Common Crawl (.3.1 MBR and QE dataset (generated from the reference-finetumed checkpoint) of 200k sentences, versus finetuning in Table 15| Reducing the dataset to only 15% of its original size leads to a consistent, but small, decrease performance across all metrics.
sample of 30k sentences. The results shown
Model
BLEURT
MetricX-XXL
COMET20
MBR-finetuned (CC-200k)
78.50
81.72
61.49
MBR-finetunce (CC-30k)
78.33
81.72
60.06
QE-fiettnne (CC-200k)
77.86
81.63 81.53
61.11
QE-finetuned (CC-30k)
77.70
60.62
Table Comparison MBR and QE finetuning (from the reference-fineture checkpoint) using Common-
Crawl dataset samples of size 200k vs 30k.
D.1 How DOES THE NUMBER OF CANDIDATE TRANSLATIONS USED FOR QE-RERANKED DATASET GENERATION AFFECT DOWNSTREAM QE FINETUNING PERFORMANCE?
As shown in Figure finetuning on QE dataset generated with candidate size=256 does nor outperform candidate size=32. For QE-finetuned models across all candidate sizes (32, 64, 128, and 256), their performance exceeds not only the (beam search decoded) checkpoint from which finetuning is initialized, but also the performance their QE-reranked teacher model. So even with a small candidate size (allowing for efficient dataset generation), the performance of the teacher model can be exceeded.
en-de
QE-fnettned
QE-reranked
59.5
Come22 score
Beam- dem--decoddd
base model
59.0
98.0
32
64
128
256
Number candidates
Figure 4: Finetuned model performance as function of the number of candidates per source used generate the QE dataset. Performance the QE-reranked leacher model improves as candidate size increases, while the QE-finetuned student is more robust to candidate size, and outperforms the teacher at all candidate sizes. The lower bound of beam scarch decoding from the base model is also shown for perspective.
D.2 AT WHAT DATASET SIZE DOES QE FINETUNING ACIIEVE PARITY WITH FINETUNING ON REFERENCES?
In 1 we saw that finetuning references outperforms self-QE finetuning given the source- side dataset (of 30k sentences). But for many language pairs, high-quality human-tramm translated data suitable for incremental training is scarce. In this ablation, we finetuned subsamples from the references dataset to determine the reference dataset size at which parity with QE finetuning is achieved.
As shown in Table 16, QE finetuning achieves slightly superior performance (in terms Comer20 score) finetuning references dataset of 5k sentences. So for this particular setup (language pair, dataset, model), there is a 6x multiplier in QE dataset size relative to references dataset size needed achieve performance parity.
17

---

Published as a conference paper at ICLR 2024
Model
BLEURT
MetricX-XXI.
COMET20
Reference-finetunee (30k)
77.49
81.53
60.95
Reference-finetunee (20k)
77.31
81.27
60,48
ffferenceffiettneed (10k)
77.29
81.22
60.07
Reference-finetunedd (5k)
77.35
81.25
59.60
DE-finetuned (30k)
77.24
81.25
59.63
Table 16: Reference-finetunie subsampled datasets of different sizes. The Comet20 score of the reference- finetuned model on 5k sentences is slightly lower than that the QE-finetuned model on 30k sentences.
D.3 Is BACK-TRANSLATED QE DATA AS EFFECTIVE AS FORWARD-TRANSLATEDD
In $6.1 we showed that QE finetuning outperforms the baseline of finetuning on beam search translations. Finetuning on back-translated QE data is another useful baseline to compare against, to understand whether or not the decoder specifically benefits from learning from the quality and style of (target-side) QE translations.
To generate back-translann translated QE data, we used WMT German- German-English model with the same architec- ture described in 5.2, We generated QE translations of the NewsTest 2009-2019 German-Englin corpus, and used these translations to finetune the base English-Gerne model. As shown in Table 17 finetuning back-translated QE data doesn't 't boost model quality much finetuning forward-translated QE data does.
Model
BLEURT
MetricX-XXI
COMET20
Base model
76.78
80.52
57.79
OE-fiinetuned
77.24
81.25
59.63
QE-BT-fnetuned
76.52
80.63
59.12
Table 17: Comparison of finetuning results using forward-translated back-translated QE data.
D.4 How DOES FINETUNING MIXTURE OF MBR AND QE DATA COMPARE TO FINETUNING ON EACH DATASET SEPARATELY?
The mixture was created as a balanced combination of the MBR and QE data generated for the Phase experiments (as described in (T8). As shown in Table [8 the Comet20 score of the mixture- mixture-finetuned model falls between the scores of the MBR-finetuned and QF-finetuned models, but is closer the (higher) score of the QF-finetuned model.
Model
BLEURT MetricX-XXL
COMET20
Base model MBR-finetuned
76.78
80.52
57.79
77.34
80.36
58.02
QE-finetuned
77.24
81.25
59.63
MBR+QE-finetuned 77.42
81.19
59.40
Table 18: Comparison of finetuning on MBR data only and QE data only, versus finetuning on mixture of both datasets.
D.5 DOES ITERATIVE QE FINETUNING OUTPERFORM A SINGLE ROUND OF FINETUNING?
QE finetuning can be performed iteratively, by generating new QE data for the next finetuning round from the previous round's QE- ff-finntueed checkpoint. We experiment with two rounds of QE finetuning using the setup from Phase ($5.4). with the NewsTest 2009-2019 dataset used for both rounds. We in Table 19 that single round of QE finetuning outperforms rounds.
18

---

Published as a conference paper at ICLR 2024
Model
BLEURT
MetricX-XXI.
COMET20
QF-finetuned (after round)
77.24
59.63 81.25
QE-finetuned (after rounds)
77.19
80.48
58.06
Table 19: Iterative QE finetuning does not outperform a single round of QE finetuning.
D.6 Do MBR DECODING AND QE RERANKING IMPROVE THE PERFORMANCE OF MBR-FINNTUNEDD AND QE-FINETUNED MODELS, RELATIVE TO MORE EFFICIENT DECODING METHODS?
As shown in Table 20, after MBR finetuning, QE reranking works better than MBR decoding. Similarly, after QE finetuning, MBR decoding works better than QE reranking. This suggests that using the same decoding method both to generate the finetuning data and at inference time overfits to the metric used as the utility function. Also note that the gap between MBR decoding/QB reranking and beam search/greedy decoding is small for the MBR/QE-finetunl and of the QE-finetuned models, as well as Phase 3 of the MBR-finetuned model, beamn search or greedy decoding is actually the top-pertiee oo-performing decoding strategy.
models. In fact, for Phases
Model
QE-finetuned
MBR-finetuned
Beam
59.63
58.02
Greedy
59.58
57.61
Phase
Sampling
55.91
57.69
MBR
60.32
55,21
QE
60.13
60.03
Beam
61.11
61.49
Greedy
61.19
60,93
Phase 2 (from-ref)
Sampling
57.14
57.80
MBR
61.15
59.35
QE
60.92
61.62
Beam
62.35
63,62
Greedy
62.52
62.65
Phase 3 (from-ref)
Sampling
56.22
58,92
MBR
61.76
62,30
QE
60.56
63.05
Table 20: Comet20 performance of en-xde QE-finetunee and MBR- models when using different decoding strategies, including MBR decoding and QE reranking. Here, the sampling strategy used was epsilon sampling with 0.02.
19


Reviews:
[{"review_id": "BQLoIw3WtP", "reviewer": "Reviewer_5sbu", "summary": "The paper describes approaches to training NMT models that use different decoding methods than beam search, namely, QE reranking and MBR decoding. Using such decoding methods in inference can be quite expensive computationally but using them in training time allows for maintaining inference time efficiency while leveraging their mitigation of model-perplexity-vs-quality. This approach can be seen as an alternative to aligning MT models with human translation preferences, given by the QE or utility function used in MBR. Furthermore, using either QE reranking or MBR decoding allows for exploring monolingual data augmentation for training as neither require references for scoring translation hypotheses. Automatic and manual evaluation results indicate that QE-finetuned systems outperform the reference-finetuned baseline, and QE finetuning using a PaLM-2 teacher outperforms using the self-teacher model.", "questions": "* Is there any description of MetricX-XXL-QE in another paper? I could not find a paper describing except a paragraph in the Findings paper. How many languages does it support? This seems to be a very big QE model, 30B parameters. \n\n* How much time is added to the training process when using the MBR and QE reranking? Maybe a table illustrating the added wallclock time would be interesting to contrast with the gains in performance. \n\n* How much does the data used to train the QE model and the utility function used in MBR affect the final results for specific domains? Do yo have any insights on this? For example, for English-German, it seems most of the MT training data is similar to the data used to train the QE and BLEURT models. What if this is not the case? Do you observe the same improvements or does something change in the translation quality of the best models?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is a good contribution towards aligning MT models with human preferences as given by reference-less or reference-based metrics (QE reranking and the utility function used for MBR, respectively). It allows leveraging monolingual data relevant to the MT model application setting for improving its translation quality. Automatic evaluation is consistent with manual evaluation and the experimental setting is sound. This is an interesting alternative to reinforcement learning models that have shown to be quite unstable in MT settings. The approaches shown here are also generalizable to other natural generation tasks though no experiments have been devised to show that.", "weaknesses": "The paper is quite dense at some parts, mainly at the experimental setting. It takes a few passes on those sections to completely understand the results and the settings but it is clear. The QE approach is not really reproducible as the QE model is not publicly available. Furthermore, the best teacher model seems to be a PaLM2-Bison that is also not publicly available.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MortOlkL57", "reviewer": "Reviewer_2DMZ", "summary": "The paper proposes two new training methods called MBR finetuning and QE finetuning to distill the benefits of high-quality decoding methods like Minimum Bayes Risk (MBR) decoding and Quality Estimation (QE) reranking into the model weights, while avoiding expensive inference costs. The results demonstrates their effectiveness for machine translation including exceeding human references", "questions": "1.\tPlease report the performance of the PaLM-2 Bison model.\n2.\tFor English-German, MBR finetuning seems to outperform QE finetuning. Why was only QE finetuning evaluated for English-Japanese instead of also evaluating MBR?\n3.\tCan you report the training time and compute requirements for the QE and MBR finetuning experiments? Providing the concrete training costs for comparison would be helpful.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. MBR and QE produce superior quality but are too expensive for inference. The finetuning approach provides a way to achieve most of their benefits without sacrificing efficiency. This could enable deploying higher performance models in real applications.\n2. The proposed methods are clearly explained and technically sound. The experiments are comprehensive and rigorous, spanning various metrics, domains, language pairs, and resource settings. The results consistently validate the effectiveness of MBR and QE finetuning over strong baselines.", "weaknesses": "1.\tThe novelty of the method is unclear. Using high-quality pseudo data generated by better decoding methods is a common technique in neural machine translation. This paper does not seem to provide new insights beyond what is already known.\n2.\tWhile this paper reports significant gains with MBR and QE finetuning, the reasons for the improvements are not well analyzed. In some cases, QE finetuning performs better, while in others, MBR finetuning is superior. More details and an in-depth analysis of the advantages and limitations of each finetuning method would be expected.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PO9iVUsla5", "reviewer": "Reviewer_VTjn", "summary": "This paper uses expensive decoding methods like MBR decoding and QE using neural metrics to generate data for finetuning. After fine-tuning, the model can be used with cheaper decoding strategies like beam search/greedy search and still retain the gains of the expensive decoding methods.\nThe paper explores different setups: finetuning on references, finetuning on MBR/QE decoded reference set, finetuning on MBR/QE decoded sampled monolingual data, finetuning on MBR/QE decoded samples from a very strong teacher (finetuned LLM) and show that finetuning on MBR/QE decoded monolingual data can get additional benefits over finetuning on references. They also show that finetuning using decoded data from a very strong teacher shows the best results.\nThe paper also does ablations around the effect of candidate size/source sentences selected/forward translation vs. backward translation while generating the finetuning data.\nFinally the QE finetuned systems are evaluated against baselines in an MQM human evaluation to confirm the rankings produced by COMET-20", "questions": "Questions covered in the weakness section. No other specific questions.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strengths of the paper:\n\n1. Sound experimental setup, clear description of the proposed method and showing its effectiveness\n2. Important contribution for practitioners because currently MBR and QE decoding with neural metrics is computationally infeasible to be deployed in real time, the proposed approach can help bring some of those gains to production systems", "weaknesses": "Weaknesses of the paper:\n\n1. The paper presents a section of the results from Palm2 finetuning and show large improvements. I think that is expected because it is a much stronger model. This set of experiments doesn't add too much value to the paper. The paper should also show the beam and MBR decoded results of the Palm2 model used for finetuning, so that the readers can understand show much of the teacher model performance can be transferred to the student via this finetuning.\n2. After using MBR/QE finetuning, do we still see a difference between beam decoding and MBR/QE decoding? Table 3 summarizes the results using beam/greedy and sampling, but MBR/QE decoding results should also be included for completeness.\n3. In Table 1, we see 2a perform better than 1c) and 2c) perform better than 1b). This is a bit suprising because I would have assumed that finetuning performance will be upper bounded by MBR/QE used during run-time. I couldn't find any comment on this comparison in the paper.\n4. The paper does not have any numbers on lexical metrics like chrF. Would have been nice to include those numbers atleast in the Appendix, for interested readers", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VwRSUchbs6", "reviewer": "Reviewer_t5J4", "summary": "Previous research has shown that ranking techniques (with QE metrics or MBR) generally improve machine translation quality, being very expensive at inference time. This paper proposes MBR/QE finetuning—using QE/MBR outputs at training time, while employing efficient decoding algorithms (e.g., beam search) during inference. Their experiments suggest that (1) MBR/QE finetuning alone are worse than vanilla finetuning with human references using the same high-quality data; (2) self-MBR/QE finetuning can complement finetuning on a small dataset of high-quality human references; and (3) using MBR/QE data generated by a stronger teacher model further improves quality. A subset of the results is validated by professional translators.", "questions": "I left some questions in the weaknesses part. Other comments/questions: \n\n- MBR decoding uses BLEURT and is prohibitively expensive. Note, however, that more efficient alternatives exist. See implementation details discussed in [1], Section 4.3. Is there a reason to choose BLEURT? While efficiency at inference time is an advantage, MBR/QE finetuning is expensive at training time. It would be good to see some numbers to understand the training/inference time difference when compared to other existing approaches.\n- In the introduction you say that MBR decoding requires the generation of a large number of candidates. While this is true if you get unbiased samples from the model, it does not seem to be the case when you bias the distribution (e.g., using nucleus sampling). I suggest you see the discussions in [2] and [3] and comment. Also, see my comment above about [1]. \n- Results use COMET-20 instead of more recent versions already available online such as COMET-22. Is there a reason for using the 2020 version?\n- Human evaluation results are important and should not be in the appendix (Table 7). In fact, I think they should be more highlighted in the paper! It would also be interesting to see if they generalize for En-Ja. Can you explain the reasoning for using 9 professional translators for En-De instead of using fewer and evaluate En-Ja as well?\n- What happens if you decode with reranking techniques (QE/MBR) using a model trained with your method, instead of using beam/greedy search? Even though this would make the method very inefficient at inference time, it would be interesting to see if it further boosts the performance.\n- According to Table 3, both beam and greedy search work well. Does this mean that your method helps solve the beam search curse [4]? What happens when you increase the beam width?\n\n[1] Identifying Weaknesses in Machine Translation Metrics Through Minimum Bayes Risk Decoding: A Case Study for COMET (Amrhein & Sennrich, AACL-IJCNLP 2022)\n\n[2] Quality-Aware Decoding for Neural Machine Translation (Fernandes et al., NAACL 2022).\n\n[3] An Empirical Study of Translation Hypothesis Ensembling with Large Language Models (Farinhas et al., EMNLP 2023).\n\n[4] Six Challenges for Neural Machine Translation (Koehn & Knowles, NGT 2017).", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The motivation is clear and the paper is well written.\n- MBR/QE finetuning (their method) seems to work well with beam/greedy search, making inference faster when compared to standard reranking techniques that employ rerankers at inference time.\n- Their models are evaluated with multiple state of the art evaluation metrics for MT. Their findings are further validated by 9 professional translators using the MQM framework.", "weaknesses": "- Some parts of the paper (e.g., the first three paragraphs of the introduction) talk about NLG in general but the paper focuses on machine translation. Reranking methods, in particular, rely on good quality estimation models that exist for MT but may not exist for other tasks. Of course, MBR finetuning can be applied to other NLG tasks (e.g., summarization) but this paper does not touch this problem. I think the authors should either revise the writing and make the scope more clear from the beginning, or perform experiments in other NLG tasks. The contribution is mainly empirical and only validated for MT. \n- The study is limited in terms of language coverage. They perform experiments on 2 language pairs only (mid & high resource), both for translation out of English. It’s not clear if the findings hold for lower resource languages and when translating into English. In particular, I’m expecting quality estimation models to be worse for low resource languages, which may end up affecting the quality of the translations produced with their method. Have you tried other languages? If so, what happens in that case?\n- The related work discusses other methods for training NMT models beyond MLE (e.g., RL methods) but none of them is used as a baseline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)