SlowGuess/NAACL_2022_OCR huggingface dataset:
Patrick Fernandes*,1,2,3 António Farinhas*,2,3 Ricardo Rei2,4,5 José G. C. de Souza
Perez Ogayo<sup>1</sup> Graham Neubig<sup>1</sup> André F. T. Martins<sup>2,3,5</sup>
$^{1}$ Carnegie Mellon University  $^{2}$ Instituto Superior Técnico (Lisbon ELLIS Unit)
$^{3}$ Instituto de Telecomunicações  $^{4}$ INESC-ID  $^{5}$ Unbabel
pfernand@cs.cmu.edu antonio.farinhas@tecnico.ulisboa.pt
Despite the progress in machine translation quality estimation and evaluation in the last years, decoding in neural machine translation (NMT) is mostly oblivious to this and centers around finding the most probable translation according to the model (MAP decoding), approximated with beam search. In this paper, we bring together these two lines of research and propose quality-aware decoding for NMT, by leveraging recent breakthroughs in reference-free and reference-based MT evaluation through various inference methods like  $N$ -best reranking and minimum Bayes risk decoding. We perform an extensive comparison of various possible candidate generation and ranking methods across four datasets and two model classes and find that quality-aware decoding consistently outperforms MAP-based decoding according both to state-of-the-art automatic metrics (COMET and BLEURT) and to human assessments. Our code is available at https://github.com/deep-spin/qaware-decode.
The most common procedure in neural machine translation (NMT) is to train models using maximum likelihood estimation (MLE) at training time, and to decode with beam search at test time, as a way to approximate maximum-a-posteriori (MAP) decoding. However, several works have questioned the utility of model likelihood as a good proxy for translation quality (Koehn and Knowles, 2017; Ott et al., 2018; Stahlberg and Byrne, 2019; Eikema and Aziz, 2020). In parallel, significant progress has been made in methods for quality estimation and evaluation of generated translations (Specia et al., 2020; Mathur et al., 2020b), but this progress is, by and large, not yet reflected in either training or decoding methods. Exceptions such as minimum risk training (Shen et al., 2016; Edunov et al., 2018)
Figure 1: Quality-aware decoding framework. First, translation candidates are generated according to the model. Then, using reference-free and/or reference-based MT metrics, these candidates are ranked, and the highest ranked one is picked as the final translation.
come at a cost of more expensive and unstable training, often with modest quality improvements.
An appealing alternative is to modify the decoding procedure only, separating it into two stages: candidate generation (§2.1; where candidates are generated with beam search or sampled from the whole distribution) and ranking (§2.2; where they are scored using a quality metric of interest, and the translation with the highest score is picked). This strategy has been explored in approaches using  $N$ -best reranking (Ng et al., 2019; Bhattacharyya et al., 2021) and minimum Bayes risk (MBR) decoding (Shu and Nakayama, 2017; Eikema and Aziz, 2021; Müller and Sennrich, 2021). While this previous work has exhibited promising results, it has mostly focused on optimizing lexical metrics such as BLEU or METEOR (Papineni et al., 2002; Lavie and Denkowski, 2009), which have limited correlation with human judgments (Mathur et al., 2020a; Freitag et al., 2021a). Moreover, a rigorous apples-to-apples comparison among this suite of techniques and their variants is still missing, even though they share similar building blocks.
Our work fills these gaps by asking the question:
"Can we leverage recent advances in MT quality evaluation to generate better translations?"
If so, how can we most effectively do so?"
To answer this question, we systematically explore
Human Language Technologies, pages 1396 - 1412
NMT decoding using a suite of ranking procedures. We take advantage of recent state-of-the-art learnable metrics, both reference-based, such as COMET and BLEURT (Rei et al., 2020a; Sellam et al., 2020), and reference-free (also known as quality estimation; QE), such as TransQuest and OpenKiwi (Ranasinghe et al., 2020; Kepler et al., 2019). We compare different ranking strategies under a unified framework, which we name quality-aware decoding ( $\S 3$ ). First, we analyze the performance of decoding using  $N$ -best reranking, both fixed according to a single metric and learned using multiple metrics, where the coefficients for each metric are optimized according to a reference-based metric. Second, we explore ranking using reference-based metrics directly through MBR decoding. Finally, to circumvent the expensive computational cost of the latter when the number of candidates is large, we develop a two-stage ranking procedure, where we use  $N$ -best reranking to pick a subset of the candidates to be ranked through MBR decoding. We explore the interaction of these different ranking methods with various candidate generation procedures including beam search, vanilla sampling, and nucleus sampling.
Experiments with two model sizes and four datasets (§4) reveal that while MAP-based decoding appears competitive when evaluating with lexical-based metrics (BLEU and ChrF), the story is very different with state-of-the-art evaluation metrics, where quality-aware decoding shows significant gains, both with  $N$ -best reranking and MBR decoding. We perform a human-study to more faithfully evaluate our systems and find that, while performance on learnable metrics is not always predictive of the best system, quality-aware decoding usually results in translations with higher quality than MAP-based decoding.
2 Candidate Generation and Ranking
We start by reviewing some of the most commonly used methods for both candidate generation and ranking under a common lens.
2.1 Candidate Generation
An NMT model defines a probability distribution  $p_{\theta}(y|x)$  over a set of hypotheses  $\mathcal{V}$ , conditioned on a source sentence  $x$ , where  $\theta$  are learned parameters. A translation is typically predicted using
MAP decoding, formalized as
$$
\hat {y} _ {\mathrm {M A P}} = \underset {y \in \mathcal {Y}} {\arg \max } \log p _ {\theta} (y | x). \tag {1}
$$
In words, MAP decoding searches for the most probable translation under  $p_{\theta}(y|x)$ , i.e., the mode of the model distribution. Finding the exact  $\hat{y}_{\mathrm{MAP}}$  is intractable since the search space  $\mathcal{V}$  is combinatorially large, thus, approximations like beam search (Graves, 2012; Sutskever et al., 2014) are used. However, it has been shown that the translation quality degrades for large values of the beam size (Koehn and Knowles, 2017; Yang et al., 2018; Murray and Chiang, 2018; Meister et al., 2020), with the empty string often being the true MAP hypothesis (Stahlberg and Byrne, 2019).
A stochastic alternative to beam search is to draw samples directly from  $p_{\theta}(y|x)$  with ancestral sampling, optionally with variants that truncate this distribution, such as top- $k$  sampling (Fan et al., 2018) or  $p$ -nucleus sampling (Holtzman et al., 2020) – the latter samples from the smallest set of words whose cumulative probability is larger than a predefined value  $p$ . Deterministic methods combining beam and nucleus search have also been proposed (Shaham and Levy, 2021).
Unlike beam search, sampling is not a search algorithm nor a decision rule – it is not expected for a single sample to outperform MAP decoding (Eikema and Aziz, 2020). However, samples from the model can still be useful for alternative decoding methods, as we shall see. While beam search focus on high probability candidates, typically similar to each other, sampling allows for more exploration, leading to higher candidate diversity.
2.2 Ranking
We assume access to a set  $\bar{\mathcal{V}}\subseteq \mathcal{V}$  containing  $N$  candidate translations for a source sentence, obtained with one of the generation procedures described in  $\S 2.1$ . As long as  $N$  is relatively small, it is possible to (re-)rank these candidates in a post-hoc manner, such that the best translation maximizes a given metric of interest. We highlight two different lines of work for ranking in MT decoding: first,  $N$ -best reranking, using reference-free metrics as features; second, MBR decoding, using reference-based metrics.
2.2.1  $N$ -best Reranking
In its simplest form (which we call fixed reranking), a single feature  $f$  is used (e.g., an estimated quality
score), and the candidate that maximizes this score is picked as the final translation,
$$
\hat {y} _ {\mathrm {F - R R}} = \underset {y \in \bar {\mathcal {Y}}} {\arg \max } f (y). \tag {2}
$$
When multiple features  $[f_1, \ldots, f_K]$  are available, one can tune weights  $[w_1, \ldots, w_K]$  for these features to maximize a given reference-based evaluation metric on a validation set (Och, 2003; Duh and Kirchhoff, 2008) - we call this tuned reranking. In this case, the final translation is
$$
\hat {y} _ {\mathrm {T - R R}} = \underset {y \in \bar {\mathcal {Y}}} {\arg \max } \sum_ {k = 1} ^ {K} w _ {k} f _ {k} (y). \tag {3}
$$
2.2.2 Minimum Bayes Risk (MBR) Decoding
While the techniques above rely on reference-free metrics for the computation of features, MBR decoding uses reference-based metrics to rank candidates. Unlike MAP decoding, which searches for the most probable translation, MBR decoding aims to find the translation that maximizes the expected utility (equivalently, that minimizes risk, Kumar and Byrne 2002, 2004; Eikema and Aziz 2020). Let again  $\bar{\mathcal{V}}\subseteq \mathcal{V}$  be a set containing  $N$  hypotheses and  $u(y^{*},y)$  a utility function measuring the similarity between a hypothesis  $y\in \mathcal{V}$  and a reference  $y^{*}\in \bar{\mathcal{V}}$  (e.g., an automatic evaluation metric such as BLEU or COMET). MBR decoding seeks for
$$
\hat {y} _ {\mathrm {M B R}} = \underset {y \in \bar {\mathcal {Y}}} {\arg \max } \underbrace {\mathbb {E} _ {Y \sim p _ {\theta} (y | x)} [ u (Y , y) ]} _ {\approx \frac {1}{M} \sum_ {j = 1} ^ {M} u (y ^ {(j)}, y)}, \tag {4}
$$
where in Eq. 4 the expectation is approximated as a Monte Carlo (MC) sum using model samples  $y^{(1)}, \ldots, y^{(M)} \sim p_{\theta}(y|x)$ . In practice, the translation with the highest expected utility can be computed by comparing each hypothesis  $y \in \bar{\mathcal{V}}$  to all the other hypotheses in the set.
3 Quality-Aware Decoding
While recent works have explored various combinations of candidate generation and ranking procedures for NMT (Lee et al., 2021; Bhattacharyya et al., 2021; Eikema and Aziz, 2021; Müller and Sennrich, 2021), they suffer from two limitations:
- The ranking procedure is usually based on simple lexical-based metrics (BLEU, chrF, METEOR).
Although these metrics are well established and inexpensive to compute, they correlate poorly with human judgments at segment level (Mathur et al., 2020b; Freitag et al., 2021c).
Each work independently explores  $N$ -best reranking or MBR decoding, making unclear which method produces better translations.
In this work, we hypothesize that using more powerful metrics in the ranking procedure may lead to better quality translations. We propose a unified framework for ranking with both reference-based (§3.1) and reference-free metrics (§3.2), independently of the candidate generation procedure. We explore four methods with different computational costs for a given number of candidates,  $N$ .
Fixed  $N$ -best Reranker. An  $N$ -best reranker using a single reference-free metric (§3.2) as a feature, according to Eq. 2. The computational cost of this ranker is  $\mathcal{O}(N \times C_{\mathcal{M}^{\mathrm{QE}}})$ , where  $C_{\mathcal{M}^{\mathrm{QE}}}$  denotes the cost of running an evaluation with a metric  $\mathcal{M}^{\mathrm{QE}}$ .
Tuned  $N$ -best Reranker. An  $N$ -best reranker using as features all the reference-free metrics in §3.2, along with the model log-likelihood  $\log p_{\theta}(y|x)$ . The weights in Eq. 3 are optimized to maximize a given reference-based metric  $\mathcal{M}^{\mathrm{ref}}$  using MERT (Och, 2003), a coordinate-ascent optimization algorithm widely used in previous work. Note that  $\mathcal{M}^{\mathrm{ref}}$  is used for tuning only; at test time, only reference-free metrics are used. Therefore, the decoding cost is  $\mathcal{O}(N \times \sum_{i} C_{\mathcal{M}_{i}^{\mathrm{QE}}})$ .
MBR Decoding. Choosing as the utility function a reference-based metric  $\mathcal{M}^{\mathrm{ref}}$  (§3.1), we estimate the utility using a simple Monte Carlo sum, as shown in Eq. 4. The estimation requires computing pairwise comparisons and thus the cost of running MBR decoding is  $\mathcal{O}(N^2 \times C_{\mathcal{M}^{\mathrm{ref}}})$ .
$N$ -best Reranker  $\rightarrow$  MBR. Using a large number of samples in MBR decoding is expensive due to its quadratic cost. To circumvent this issue, we explore a two-stage ranking approach: we first rank all the candidates using a tuned  $N$ -best reranker, followed by MBR decoding using the top  $M$  candidates. The computational cost becomes  $\mathcal{O}(N \times \sum_{i} C_{\mathcal{M}_{i}} + M^{2} \times C_{\mathcal{M}^{\mathrm{ref}}})$ . The first ranking stage prunes the candidate list to a smaller, higher quality subset, making possible a more accurate estimation of the utility with less samples, and potentially allowing a better ranker than plain MBR for almost the same computational budget.
<sup>1</sup>We also consider the case where  $y^{(1)}, \ldots, y^{(M)}$  are obtained from nucleus sampling or beam search. Although the original MC estimate is unbiased, these ones are biased.
3.1 Reference-based Metrics
Reference-based metrics are the standard way to evaluate MT systems; the most used ones rely on the lexical overlap between hypotheses and reference translations (Papineni et al., 2002; Lavie and Denkowski, 2009; Popovic, 2015). However, lexical-based approaches have important limitations: they have difficulties recognizing correct translations that are paraphrases of the reference(s); they ignore the source sentence, an important indicator of meaning for the translation; and they do not always correlate well with human judgments, particularly at segment-level (Freitag et al., 2021c).
In this work, apart from BLEU (computed using SacreBLEU $^2$  (Post, 2018)) and chrF, we use the following state-of-the-art trainable reference-based metrics for both ranking and performance evaluation of MT systems:
- BLEURT (Sellam et al., 2020; Pu et al., 2021), trained to regress on human direct assessments (DA; Graham et al. 2013). We use the largest multilingual version, BLEURT-20, based on the RemBERT model (Chung et al., 2021).
COMET (Rei et al., 2020a), based on XLM-R (Conneau et al., 2020), trained to regress on quality assessments such as DA using both the reference and the source to assess the quality of a given translation. We use the publicly available model developed for the WMT20 metrics shared task (wmt20-comet-da).
These metrics have shown much better correlation at segment-level than previous lexical metrics in WMT metrics shared tasks (Mathur et al., 2020b; Freitag et al., 2021c). Hence, as discussed in §2.2, they are good candidates to be used either indirectly as an optimization objective for learning the tuned reranker's feature weights, or directly as a utility function in MBR decoding. In the former, the higher the metric correlation with human judgment, the better the translation picked by the tuned reranker. In the latter, we approximate the expected utility in Eq. 4 by letting a candidate generated by the model be a reference translation – a suitable premise if the model is good in expectation.
3.2 Reference-free Metrics
MT evaluation metrics have also been developed for the case where references are not available -
they are called reference-free or quality estimation (QE) metrics. In the last years, considerable improvements have been made to such metrics, with state-of-the-art models having increasing correlations with human annotators (Freitag et al., 2021c; Specia et al., 2021). These improvements enable the use of such models for ranking translation hypotheses in a more reliable way than before.
In this work, we explore four recently proposed reference-free metrics as features for  $N$ -best reranking, all at the sentence-level:
COMET-QE (Rei et al., 2020b), a reference-free version of COMET (§3.1). It was the winning submission for the QE-as-a-metric subtask of the WMT20 shared task (Mathur et al., 2020b).
- TransQuest (Ranasinghe et al., 2020), the winning submission for the sentence-level DA prediction subtask of the WMT20 QE shared task (Specia et al., 2020). Similarly to COMET-QE this metric predicts a DA score.
- MBART-QE (Zerva et al., 2021), based on the mBART (Liu et al., 2020) model, trained to predict both the mean and the variance of DA scores. It was a top performer in the WMT21 QE shared task (Specia et al., 2021).
- OpenKiwi-MQM (Kepler et al., 2019; Rei et al., 2021), based on XLM-R, trained to predict the multidimensional quality metric (MQM; Lommel et al. 2014). This reference-free metric was ranked second on the QE-as-a-metric subtask from the WMT 2021 metrics shared task.
We study the benefits of quality-aware decoding over MAP-based decoding in two regimes:
- A high-resource, unconstrained, setting with large transformer models (6 layers, 16 attention heads, 1024 embedding dimensions, and 8192 hidden dimensions) trained by Ng et al. (2019) for the WMT19 news translation task (Barrault et al., 2019), using English to German  $(\mathrm{EN} \rightarrow \mathrm{DE})$  and English to Russian  $(\mathrm{EN} \rightarrow \mathrm{RU})$  language pairs. These models were trained on
$^{3}$ MQM annotations are expert-level type of annotations more fine-grained than DA, with individual errors annotated.
${}^{2}$  nrefs:1|case:mixed|eff:no|tok:13a | smooth:exp|version:2.0.0
Figure 2: Values for BLEU(top) andCOMET(bottom)for  $\mathrm{EN}\rightarrow \mathrm{DE}$  as we increase the number of candidates for different generation and ranking procedures, as well as oracles with the respective metrics, for the large (left) and small(right) models. Baseline values (with beam size of 5) are marked with a dashed horizontal line.
over 20 million parallel and 100 million back-translated sentences, being the winning submissions of that year's shared task. We consider the non-ensemble version of the model and use newstest19 for validation and newstest20 for testing.
- A more constrained scenario with a small transformer model (6 layers, 4 attention heads, 512 embedding dimensions, and 1024 hidden dimensions) trained from scratch in Fairseq (Ott et al., 2019) on the smaller IWSLT17 datasets (Cettolo et al., 2012) for English to German  $(\mathrm{EN} \rightarrow \mathrm{DE})$  and English to French  $(\mathrm{EN} \rightarrow \mathrm{FR})$ , each with a little over  $200\mathrm{k}$  training examples. We chose these datasets because they have been extensively used in previous work (Bhattacharyya et al., 2021) and smaller model allows us to answer questions about how the training methodology affects ranking performance (see § 4.2.2). Further training details can be found in Appendix A.
We use beam search with a beam size of 5 as our decoding baseline because we found that it resulted in better or similar translations than larger beam sizes. For tuned  $N$ -best reranking, we use Travatar's (Neubig, 2013) implementation of MERT (Och, 2003) to optimize the weight of each feature, as described in §3.2. Finally, we evaluate each system using the metrics discussed in §3.1, along with BLEU and chrF (Popovic, 2015).
Overall, given all the metrics, candidate generation, and ranking procedures, we evaluate over 150 systems per dataset. We report subsets of this data separately to answer specific research questions, and defer to Appendix B for additional results.
4.2.1 Impact of Candidate Generation
First, we explore the impact of the candidate generation procedure and the number of candidates.
Which candidate generation method works best, beam search or sampling? We generate candidates with beam search, vanilla sampling, and nucleus sampling. For the latter, we use  $p = 0.6$  based on early results showing improved performance for all metrics. For  $N$ -best reranking, we use up to 200 samples; for MBR decoding, due to the quadratic computational cost, we use up to 100.
Figure 2 shows BLEU and COMET for different candidate generation and ranking methods for the EN  $\rightarrow$  DE WMT20 and IWSLT17 datasets, with increasing number of candidates. The baseline is represented by the dashed line. To assess the performance ceiling of the rankers, we also report results with an oracle ranker for the reported metrics, picking the candidate that maximizes it. For the fixed  $N$ -best reranker, we use COMET-QE as a metric, albeit the results for other reference-free metrics are similar. Performance seems to scale well with the number of candidates, particularly for vanilla sampling and for the tuned  $N$ -best reranker and MBR decoder. (Lee et al., 2021; Müller and Sennrich, 2021). However, all the rankers using vanilla sampling severely under-perform the baseline in most cases (see also §4.2.2). In contrast, the rankers using beam search or nucleus sampling are competitive or outperform the baseline in terms of BLEU, and greatly outperform it in terms of COMET. For the larger models, we see that the performance according to the lexical metrics degrades with more candidates. In this scenario, rankers us
4We picked nucleus sampling over top-  $k$  sampling because it allows varying support size and has outperformed top-  $k$  in text generation tasks (Holtzman et al., 2020).
Figure 3: COMET scores for EN  $\rightarrow$  DE (IWSLT17) for models trained with and without label smoothing.
ing nucleus sampling seem to have an edge over the ones that use beam search for COMET.
Based on the findings above, and due to generally better performance of COMET over BLEU for MT evaluation (Kocmi et al., 2021), in following experiments we use nucleus sampling with the large model and beam search with the small model.
4.2.2 Impact of Label Smoothing
How does label smoothing affect candidate generation? Label smoothing (Szegedy et al., 2016) is a regularization technique that redistributes probability mass from the gold label to the other target labels, typically preventing the model from becoming overconfident (Müller et al., 2019). However, it has been found that label smoothing negatively impacts model fit, compromising the performance of MBR decoding (Eikema and Aziz, 2020, 2021). Thus, we train a small transformer model without label smoothing to verify its impact in the performance of  $N$ -best reranking and MBR decoding. Figure 3 shows that disabling label smoothing really helps when generating candidates using vanilla sampling. However, the performance degrades for candidates generated using nucleus sampling when we disable label smoothing, hinting that the pruning mechanism of nucleus sampling may help mitigate the negative impact of label smoothing in sampling based approaches. Even without label smoothing, vanilla sampling is not competitive with nucleus sampling or beam search with label smoothing, thus, we do not experiment further with it.
4.2.3 Impact of Ranking and Metrics
We now investigate the usefulness of the metrics presented in §3 as features and objectives for ranking. For  $N$ -best reranking, we use all the available candidates (200) while, for MBR, due to the computational cost of using 100 candidates, we report results with 50 candidates only (we found that ranking with tuned  $N$ -best reranking with  $N = 100$  and MBR with  $N = 50$  takes about the same time). We report results in Table 1, and use them to answer
some specific research questions.