Venue: NAACL 2024

# On the True Distribution Approximation of Minimum Bayes-Risk Decoding

Atsumoto Ohashi\({}^{1*}\)  Ukyo Honda\({}^{2}\) Tetsuro Morimura\({}^{2}\) Yuu Jinnai\({}^{2}\)

\({}^{1}\)Nagoya University  \({}^{2}\)CyberAgent

ohashi.atsumoto.c@es.mail.nagoya-u.ac.jp

{honda_ukyo,morimura_tetsuro,jinnai_yu}@cyberagent.co.jp

Work done during an internship at CyberAgent.The code is available at [https://github.com/CyberAgentALLab/mbr-anomaly](https://github.com/CyberAgentALLab/mbr-anomaly).

###### Abstract

Minimum Bayes-risk (MBR) decoding has recently gained renewed attention in text generation. MBR decoding considers texts sampled from a model as pseudo-references and selects the text with the highest similarity to the others. Therefore, sampling is one of the key elements of MBR decoding, and previous studies reported that the performance varies by sampling methods. From a theoretical standpoint, this performance variation is likely tied to how closely the samples approximate the true distribution of references. However, this approximation has not been the subject of in-depth study. In this study, we propose using anomaly detection to measure the degree of approximation. We first closely examine the performance variation and then show that previous hypotheses about samples do not correlate well with the variation, but our introduced anomaly scores do. The results are the first to empirically support the link between the performance and the core assumption of MBR decoding.1

Footnote 1: The code is available at [https://github.com/CyberAgentALLab/mbr-anomaly](https://github.com/CyberAgentALLab/mbr-anomaly).

+
Footnote †: The code is available at [https://github.com/CyberAgentALLab/mbr-anomaly](https://github.com/CyberAgentALLab/mbr-anomaly).

## 1 Introduction

Minimum Bayes-risk (MBR) decoding has recently re-emerged as a better alternative to beam search in text generation such as neural machine translation (NMT), text summarization, and image captioning [8, 30, 47]. MBR decoding first samples texts from a model and then selects the text most similar to the others, considering the text samples as substitutes for references. Therefore, sampling plays an important role in MBR decoding, and previous studies have reported that the performance varies by sampling methods [5, 69, 70].

From a theoretical standpoint, the samples are assumed to approximate the _true distribution_, the distribution of human-quality translations [45, 10]. If the approximation deviates, biases can emerge in results of MBR decoding. This implies a significant link between the performance variation and approximation quality. Although previous studies explained the performance variation by some properties of samples, _e.g._, sampling bias and cumulative probability mass [5, 69], those properties have no clear relation with the true distribution. Consequently, the relation between the performance gains by sampling methods and the core assumption remains unclear.

This study aims to empirically support the link between the performance and the approximation of true distribution. To this end, we introduce measures for the degree of approximation. If the assumption for samples holds, references, which are drawn from the true distribution by definition, should not deviate from the majority of the samples. Based on this recasting, we employ _anomaly detection_ (also called _outlier or novelty detection_) for the measure. Our hypothesis is that references achieve lower anomaly scores among samples obtained with a higher-performance sampling method. We first closely examine the performance variation by sampling methods. Then, we show that the variation highly correlates with the anomaly scores but not so with the properties based on previous hypotheses. The results are the first to provide empirical evidence for the link between the performance and core assumption, which is an important step to understanding the connection between the actual performance and the theory of MBR decoding.

## 2 Preliminaries

Let \(u(y,r)\) be a utility function to measure the quality of model translation \(y\) (**candidate**; Freitag et al.,

2022) given its reference translation \(r\). Among a set of candidates \({\cal Y}\), MBR decoding selects the one that minimizes the expected error or, equivalently, maximizes the expected utility Kumar and Byrne (2002, 2004); Freitag et al. (2022):

\[y^{*}=\operatorname*{arg\,max}_{y\in{\cal Y}}\mathbb{E}_{r\sim P_{\mathrm{human }}(\cdot|x)}[u(y,r)]. \tag{1}\]

Here, \(P_{\mathrm{human}}(\cdot|x)\) is the **true distribution** over translations of an input text \(x\)Kumar and Byrne (2002, 2004), which describes human-quality translations in the space of all translations.

Since the true distribution is unknown, MBR decoding approximates Eq. (1) with finite samples drawn from a model \(r^{\prime}\sim P_{\mathrm{model}}(\cdot|x)\). That is, MBR decoding _assumes that the samples drawn from a model approximate the true distribution_ of references. The samples are called **pseudo-references**Freitag et al. (2022), which subsequently serve as alternatives to references in the computation of MBR as follows:

\[y^{*}=\operatorname*{arg\,max}_{y\in{\cal Y}}\frac{1}{|{\cal R}^{\prime}|} \sum_{r^{\prime}\in{\cal R}^{\prime}}u(y,r^{\prime}). \tag{2}\]

In practice, candidates \({\cal Y}\) and pseudo-references \({\cal R}^{\prime}\) can be the same or different sets of samples. Figure 1 shows an example of the above procedure.

## 3 Performance Variation by Sampling

Previous studies reported that performance varies by sampling methods in NMT. However, they used the same set of model translations for both candidates and pseudo-references Eikema and Aziz (2020); Fernandes et al. (2022); Freitag et al. (2023) or explored sampling methods only for candidates Eikema and Aziz (2022). These settings obscure the effect of pseudo-references, for which the true distribution is assumed, on the performance variation. This section shows the effect of pseudo-reference sampling on performance by evaluating pseudo-references separately from candidates.

### Setup

Following Fernandes et al. (2022), we use publicly-available Transformer models Vaswani et al. (2017) trained by Ng et al. (2019)2 for the WMT19 news translation task Barrault et al. (2019). The models were trained in four directions between English (en) and German (de) or Russian (ru). We conducted our experiments on the test set of WMT19 (_newstest19_), which was used as the development set in the previous work Fernandes et al. (2022). Due to the quadratic computational cost of MBR decoding, we drew 100 samples of \({\cal Y}\) and \({\cal R}^{\prime}\) for each of the 1,000 examples of _newstest19_. We employ COMET22 for the utility function \(u\), which is the state-of-the-art evaluation metric in machine translation Rei et al. (2022, 2020).3

Footnote 2: [https://github.com/facebookresearch/fairseq/blob/74@qaf7f79a7b6ddac4cbfe7cfcc715b3c1b21e/examples/translation/README.md](https://github.com/facebookresearch/fairseq/blob/74@qaf7f79a7b6ddac4cbfe7cfcc715b3c1b21e/examples/translation/README.md)

Footnote 3: COMET22 improved robustness to the deviation in numbers and named entities, which was the weakness of the previous COMET Amrhein and Sennrich (2022).

For sampling methods, we use those that have been reported to be effective: ancestral sampling Eikema and Aziz (2020); Freitag et al. (2022), beam search, nucleus sampling Eikema and Aziz (2022); Fernandes et al. (2022), and epsilon sampling Freitag et al. (2023). Ancestral sampling draws samples from \(P_{\mathrm{model}}(\cdot|x)\) without modification, while nucleus sampling restricts sampling to words with top-\(p\) probabilities Holtzman et al. (2020) and epsilon sampling truncates words with probabilities lower than \(\epsilon\)Hewitt et al. (2022). We adopt the best hyperparameters reported for \(p\) and \(\epsilon\)Fernandes et al. (2022); Freitag et al. (2023). The beam size was set to 100 to collect 100 samples.

### Results

Fixing Candidates.Since we focus on sampling for pseudo-references, we first search for the best sampling method _for candidates_ and fix it. The objective of searching for the best is to prevent the pseudo-reference's contribution to scores from being capped and obscured by the candidate's quality. To this end, we conduct the search on the same _newstest19_ as the subsequent experiments.4 Following

Figure 1: Illustrative example of MBR decoding.

 

Fernandes et al. (2022), we search for the sampling method that achieves the highest oracle score, \(\max_{y\in Y}u(y,r)\), on average. Table 1 shows that epsilon sampling achieves the highest oracle score across the language pairs. Based on these results, we fixed the sampling method of candidates to epsilon sampling in all the following experiments.

Footnote 3: [https://github.com/en-developers/datasets/](https://github.com/en-developers/datasets/)

Varying Pseudo-References.Then, we evaluate the effect of pseudo-references on performance by varying their sampling methods. Table 2 shows the results. As expected from previous studies, the performance of MBR decoding varies even when only changing the sampling methods of pseudo-references. The variation is nearly consistent across language pairs, indicating the pervasive effect of pseudo-reference on performance. The best sampling method for candidates (epsilon sampling) is not the best for pseudo-references. This shows that the desirable properties for candidates and pseudo-references are different.

Table 2 also shows the results of beam search just for the comparison with MBR decoding. Here, the beam size was set to 5. MBR decoding significantly outperforms beam search and even outperforms the ensemble model, which was the winner of WMT19 Barrault et al. (2019). Since the effectiveness of epsilon sampling was reported on the other WMT dataset Freitag et al. (2023), we have a good reason to use epsilon sampling for this comparison.

## 4 Hypotheses for Performance Variation

The previous section confirmed that the performance varies by sampling pseudo-references. The question that naturally arises in response to the results is: why does this variation occur?

### Previous Hypotheses

Eikema and Aziz (2022) hypothesized that unbiased sampling is desirable for pseudo-references. Since the biased sampling methods limit the sampling to words of high probability, we use the average log probability **(Avg. Prob.)** of samples as a continuous proxy of bias existence in sampling. Eikema and Aziz (2020) and Freitag et al. (2023) did not distinguish between candidates and pseudo-references but referred to the larger cumulative probability mass (**Cum. Prob.)** of unique samples as a desirable property because it indicates diverse and probable samples. Eikema and Aziz (2022) employed candidate sampling that achieved high expected utility. If this criterion applies to pseudo-references, performance should be higher when the expected utility against candidates (**Cad. Sim.)** or references (**Ref. Sim.)** is high.

### Our Hypothesis

Given the relaxation from Eq. (1) to Eq. (2), a better approximation of the true distribution by pseudo-references should be associated with higher performance. To examine the relation, we propose using anomaly detection to quantitatively evaluate the approximation. If a better approximation is achieved, references should deviate less from the majority of the samples since references are drawn from the true distribution by definition. This recasting allows us to use anomaly scores of anomaly detection for measuring the degree of approximation. We then hypothesize that _a higher-performance sampling method forms samples where references achieve lower anomaly scores_.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline _Candidate_ & de-en & en-de & ru-en & en-ru \\ \hline Ancestral & 85.82 & 86.32 & 82.11 & 86.13 \\ Beam & 88.47 & 89.32 & 84.16 & 89.44 \\ Epsilon (\(\epsilon=0.02\)) & **88.51** & **89.47** & **84.36** & **90.17** \\ Nucleus (\(p=0.6\)) & 88.01 & 89.12 & 83.76 & 89.96 \\ Nucleus (\(p=0.9\)) & 88.02 & 89.04 & 83.98 & 89.57 \\ \hline \hline \end{tabular}
\end{table}
Table 1: _Oracle scores_ in COMET22 to determine the sampling method _for candidates_. The results are the average of three runs with different seeds except for beam search. The best/worst scores are in **bold**/underlined.

\begin{table}
\begin{tabular}{c l r r r r} \hline \hline  & _Pseudo-Reference_ & de-en & en-de & ru-en & en-ru \\ \cline{2-5}  & Ancestral & 85.82 & 87.51 & 82.02 & 88.41 \\  & Beam & 85.62 & 87.40 & 81.64 & 87.78 \\  & Epsilon (\(\epsilon=0.02\)) & 85.89 & 87.74 & 82.01 & 88.46 \\  & Epsilon (\(\epsilon=0.02\))* & 85.87 & 87.74 & 81.98 & 88.46 \\  & Nucleus (\(p=0.6\)) & 85.69 & 87.57 & 81.76 & 88.26 \\  & Nucleus (\(p=0.9\)) & **86.04** & **87.82** & **82.18** & **88.61** \\ \hline \hline \multicolumn{5}{c}{Beam Search} & 84.38 & 86.13 & 80.76 & 85.69 \\  & Beam Search (ensemble) & 84.30 & 86.06 & 80.91 & 85.74 \\ \hline \hline \end{tabular}
\end{table}
Table 2: COMET22 scores of MBR decoding with different pseudo-references. Candidates are sampled with epsilon sampling (\(\epsilon=0.02\)). Epsilon (\(\epsilon=0.02\))* shows the results of sampling candidates and pseudo-references with the same epsilon sampling but with different seeds. The results are the average of three runs with different seeds except for beam search. The best/worst scores are in **bold**/underlined.

 

Experiments

We test the hypotheses discussed in the previous section by evaluating the correlation between the performance variation and the properties or anomaly scores.

### Setup

The setup is the same as described in Section 3.1. We run each sampling method with three different seeds and then calculate the Spearman's rank correlation coefficient \(\rho\) between their averaged properties or anomaly scores (see Section 4) with the COMET22 scores reported in Table 2.

### Anomaly Scores

To test our hypothesis, we employ three popular methods used in _unsupervised_ anomaly detection Kriegel et al. (2011); Gu et al. (2019); Ruff et al. (2021). The first uses **Mahalanobis distance** (\(d_{M}\); Mahalanobis (1936) as an anomaly score, a classical distance measure between a data point and a distribution. In our context, the distance is between reference \(r\) and pseudo-references \(\mathcal{R^{\prime}}\) in a feature space: \(d_{M}(r,\mathcal{R^{\prime}})=\sqrt{(r-\mu)\Sigma^{-1}(r-\mu)}\), where \(\mu\) and \(\Sigma^{-1}\) are mean and inverse covariance matrix of \(\mathcal{R^{\prime}}\), respectively. Mahalanobis distance assumes that the data is normally distributed, but this assumption does not necessarily hold. \(k\)**-nearest neighbors** (\(k\)NN; Angiulli and Pizzuti (2002) does not have the assumption and is applicable to other data, such as the one with multimodal distribution. \(k\mathrm{NN}\) is a simple algorithm to consider the local density of a given data point. Still, it is known to perform favorably to some state-of-the-art algorithms for anomaly detection Gu et al. (2019). Let \(N_{i}(r,\mathcal{R^{\prime}})\) be the \(i\)th nearest pseudo-reference to \(r\) in Euclidean distance. \(k\mathrm{NN}\) takes the average of Euclidean distance from \(r\) to its \(k\)-nearest pseudo-references \(\{N_{i}(r,\mathcal{R^{\prime}})\}_{i=1}^{k}\): \(k\mathrm{NN}(r,\mathcal{R^{\prime}})=\frac{1}{k}\sum_{i=1}^{k}\lVert r-N_{i}( r,\mathcal{R^{\prime}})\rVert\). **Local outlier factor** (**LOF**; Breunig et al. (2000) additionally considers the local density of the \(k\)-nearest data points themselves. LOF in our setting measures how the local density of \(r\) deviates from that of its \(k\)-nearest pseudo-references. To better illustrate the relationship with \(k\mathrm{NN}\), we show a simplified version of LOF Schubert et al. (2014) here: \(\mathrm{LOF}_{k}(r,\mathcal{R^{\prime}})=\frac{1}{k}\sum_{r^{\prime}\in\{N_{ i}(r,\mathcal{R^{\prime}})\}_{i=1}^{k}}\frac{\lVert r-N_{k}(r, \mathcal{R^{\prime}})\rVert}{|r^{\prime}-N_{k}(r^{\prime},\mathcal{R^{\prime} })||}\). See Breunig et al. (2000) for the complete formula we used.

To calculate the anomaly scores, samples needs to be represented in a feature space. We obtain the representation in the space of utility by measuring the utility of references or pseudo-references given a set of candidates \(\mathcal{Y}\). A reference \(r\) in the space is then defined as \(\left[u(r,y_{1}),\ldots,u(r,y_{|\mathcal{Y}|})\right]^{\top}\). Same for a pseudo-reference \(r^{\prime}\).

### Results

Table 3 shows the results. As expected, the anomaly scores are clearly more correlated than the properties based on previous hypotheses. Except for Cum. Prob., Cand. Sim., and Ref. Sim., the signs of \(\rho\) are all as expected, including the anomaly scores. See Table 7 in Appendix for the results used to calculate \(\rho\).

Among the anomaly scores,5\(k\mathrm{NN}\) and LOF with \(k\) = 50 stably correlate with the performance variation better than those with \(k\) = 100 and \(d_{M}\). We speculate that the significant degradation of \(k\)NN with \(k\) = 100 is caused by outliers in pseudo-references. While \(k\)NN with \(k\) < 100 can effec 

tively avoid including these outliers in the calculation of anomaly scores, \(k\)NN with \(k=100\) cannot, and its anomaly scores are likely to be distorted by the outliers. These results suggest that even if some pseudo-references are outliers against a reference, the performance tends to be higher if the rest of the pseudo-reference is close to the reference. In other words, pseudo-references do not have to be close to references in entirety to perform well.

## 6 Related Work

MBR decoding has been used in automatic speech recognition Goel and Byrne (2000), statistical machine translation Kumar and Byrne (2002); Xu and Nakayama (2017); Blain et al. (2017). Recently, MBR decoding has gained prominence again in NMT because of the following two innovations. (1) Eikema and Aziz (2020) showed that MBR decoding with stochastic sampling has a potential to outperform MAP decoding methods, including beam search; (2) Freitag et al. (2022) and Fernandes et al. (2022) explored utility functions and found that using neural reference-based metrics as the utility function significantly enhances the quality of output texts. Muller and Sennrich (2021) reported domain robustness and less hallucination in the outputs of MBR decoding. Other text generation tasks such as text summarization, image captioning, and diversity-aware text generation also benefit from MBR decoding Suzgun et al. (2023); Borgeaud and Emerson (2020); Jinnai et al. (2024). Recent studies have focused on improving the efficiency of MBR decoding Cheng and Vlachos (2023); Finkelstein and Freitag (2023); Yang et al. (2023); Jinnai et al. (2023); Jinnai and Ariu (2024).

The most related studies explored sampling methods for MBR decoding and raised hypotheses to explain the difference in performance by sampling methods Eikema and Aziz (2020, 2022); Fernandes et al. (2022); Freitag et al. (2023). We also explored sampling methods but differed in that we did it more closely by focusing on pseudo-references. Furthermore, we introduced anomaly scores that correlate with the performance variation better than previous hypotheses.

## 7 Conclusion

This study investigated the relation between the performance of MBR decoding and the core assumption about samples: samples follow the true distribution of references. We introduced anomaly scores used in anomaly detection to evaluate the approximation of the true distribution. Experimental results demonstrated that the anomaly scores correlate with the performance significantly better than the properties hypothesized to explain the performance variation in prior literature. The previous hypotheses assumed that unbiased sampling (Avg. Prob.), diverse and probable samples (Cum. Prob.), or high expected utility Cand. and Ref. Sim.) are the key properties of samples to achieve high performance. However, these properties do not have an obvious relationship to approximating the true distribution of references, in contrast to the anomaly scores we employed.

These results show the insufficiency of existing hypotheses about the properties that samples should possess. The results are also the first to empirically support the link between the actual performance and the key assumption of MBR decoding. We believe this serves as an essential step to understanding the connection between the actual performance and the theory of MBR decoding.

## 8 Limitations and Risks

The limitation of the study is that it is solely a thorough analysis of MBR decoding, not accompanied by an algorithm to improve the performance of MBR decoding. However, our analysis empirically shows that previous hypotheses about the properties of samples are insufficient and that following the assumption of the MBR decoding is the key to improving performance. We believe this is an important contribution that modifies the direction of future development of MBR decoding.

Our investigation is limited to Transformer models provided by Ng et al. (2019) and the task is limited to machine translation. Future work will extend the analysis to a wider range of models and text generation tasks. However, it is worth noting that some studies support the general applicability of MBR decoding findings obtained in NMT to other text generation tasks and models. Some hyperparameters Suzgun et al. (2023), efficiency-boosting techniques Jinnai et al. (2024), or diversity-aware extensions Jinnai et al. (2024) for MBR decoding consistently perform well across machine translation, summarization, image captioning, and data-to-text generation with different models. Bertsch et al. (2023) shows that MBR decoding works well even in open-ended 

text generation tasks.

We do not foresee any ethical concerns in our analysis.

## References

* Amrhein and Sennrich (2022) Chantal Amrhein and Rico Sennrich. 2022. Identifying weaknesses in machine translation metrics through minimum Bayes risk decoding: A case study for COMET. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, pp. 1125-1141. External Links: Link Cited by: SS3.1.
* F. Angiulli and C. Pizzuti (2002)Fast outlier detection in high dimensional spaces. In Principles of Data Mining and Knowledge Discovery, Berlin, Heidelberg, pp. 15-27. External Links: ISBN 978-3-642-4655-3 Cited by: SS1.
* L. Barrault, O. Bojar, M. R. Costa-jussa, C. Federmann, M. Fishel, Y. Graham, B. Haddow, M. Huck, P. Koehn, S. Malmasi, C. Monz, M. Muller, S. Pal, M. Post, and M. Zampieri (2019)Findings of the 2019 conference on machine translation (WMT19). In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), Florence, Italy, pp. 1-61. External Links: Link Cited by: SS1.
* A. Bertsch, A. Xie, G. Neubig, and M. R. Gormley (2023)It's mbr all the way down: modern generation techniques through the lens of minimum bayes risk. arXiv preprint arXiv:2310.01387v1. Cited by: SS1.
* F. Blain, L. Specia, and P. Madhyastha (2017)Exploring hypotheses spaces in neural machine translation. In Proceedings of Machine Translation Summit XVI: Research Track, Nagoya Japan, pp. 282-298. External Links: Link Cited by: SS1.
* S. Borgeaud and G. Emerson (2020)Leveraging sentence similarity in natural language generation: improving beam search using range voting. In Proceedings of the Fourth Workshop on Neural Generation and Translation, Online, pp. 97-109. External Links: Link Cited by: SS1.
* M. Breunig, H. Kriegel, R. T. Ng, and J. Sander (2000)LoF: identifying density-based local outliers. SIGMOD Rec.29 (2), pp. 93-104. External Links: ISSN 0001-927X, Document, Link Cited by: SS1.
* J. Cheng and A. Vlachos (2023)Faster minimum bayes risk decoding with confidence-based pruning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, pp. 12473-12480. External Links: Link Cited by: SS1.
* B. Eikema and W. Aziz (2020)Is MAP decoding all you need? the inadequacy of the mode in neural machine translation. In Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain, pp. 4506-4520. External Links: Link Cited by: SS1.
* B. Eikema and W. Aziz (2022)Sampling-based approximations to minimum bayes risk decoding for neural machine translation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Abu Dhabi, United Arab Emirates, pp. 10978-10993. External Links: Link Cited by: SS1.
* P. Fernandes, A. Farinhas, R. Rei, J. G. C. de Souza, P. Ogayo, G. Neubig, and A. Martins (2022)Quality-aware decoding for neural machine translation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Seattle, United States, pp. 1396-1412. External Links: Link Cited by: SS1.
* M. Finkelstein and M. Freitag (2023)MBR and QE finetuning: training-time distillation of the best and most expensive decoding methods. arXiv preprint arXiv:2309.10966v1. Cited by: SS1.
* M. Freitag, B. Ghorbani, and P. Fernandes (2023)Epsilon sampling rocks: investigating sampling strategies for minimum bayes risk decoding for machine translation. arXiv preprint arXiv:2305.09860v2. Cited by: SS1.
* M. Freitag, D. Grangier, Q. Tan, and Bowen Liang (2022)High quality rather than high model probability: minimum bayes risk decoding with neural metrics. Transactions of the Association for Computational Linguistics10, pp. 811-825. External Links: Link Cited by: SS1.
* V. Goel and W. J. Byrne (2000)Minimum bayes-risk automatic speech recognition. Computer Speech & Language14 (2), pp. 115-135. External Links: Link Cited by: SS1.
* X. Gu, L. Akoglu, and A. Rinaldo (2019)Statistical analysis of nearest neighbor methods for anomaly detection. In Advances in Neural Information Processing Systems, Vol. 32, pp. 3414-3427. External Links: Link Cited by: SS1.
* J. Hewitt, C. Manning, and P. Liang (2022)Truncation sampling as language model desmoothing. In Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, pp. 3414-3427. External Links: Link Cited by: SS1.
* A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi (2020)The curious case of neural text degeneration. In International Conference on Learning Representations, External Links: Link Cited by: SS1.
* Y. Jinnai and K. Ariu (2024)Hyperparameter-free approach for faster minimum bayes risk decoding. arXiv preprint arXiv:2401.02749v1.

 

Yuu Jinnai, Ukyo Honda, Tetsuro Morimura, and Peinan Zhang. 2024. Generating diverse and high-quality texts by minimum bayes risk decoding. _arXiv preprint arXiv:2401.05054v1_.
* Jinnai et al. (2023) Yuu Jinnai, Tetsuro Morimura, Ukyo Honda, Kaito Ariu, and Kenshi Abe. 2023. Model-based minimum bayes risk decoding. _arXiv preprint arXiv:2311.05263v1_.
* Kriegel et al. (2011) Hans-Peter Kriegel, Peer Kroger, Erich Schubert, and Arthur Zimek. 2011. Interpreting and unifying outlier scores. In _Proceedings of the 2011 SIAM International Conference on Data Mining (SDM)_, pages 13-24. SIAM.
* Kumar and Byrne (2002) Shankar Kumar and William Byrne. 2002. Minimum Bayes-risk word alignments of bilingual texts. In _Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP 2002)_, pages 140-147. Association for Computational Linguistics.
* Kumar and Byrne (2004) Shankar Kumar and William Byrne. 2004. Minimum Bayes-risk decoding for statistical machine translation. In _Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004_, pages 169-176, Boston, Massachusetts, USA. Association for Computational Linguistics.
* Mahalanobis (1936) Prasanta Chandra Mahalanobis. 1936. On the generalized distance in statistics. In _Proceedings of the National Institute of Science of India_, volume 12, pages 49-55. National Institute of Science of India.
* Muller and Sennrich (2021) Mathias Muller and Rico Sennrich. 2021. Understanding the properties of minimum Bayes risk decoding in neural machine translation. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing Volume 1: Long Papers_, pages 259-272, Online. Association for Computational Linguistics.
* Ng et al. (2019) Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. 2019. Facebook FAIR's WMT19 news translation task submission. In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_, pages 314-319, Florence, Italy. Association for Computational Linguistics.
* Rei et al. (2022) Ricardo Rei, Jose G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinaha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and Andre F. T. Martins. 2022. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In _Proceedings of the Seventh Conference on Machine Translation (WMT)_, pages 578-585, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
* Rei et al. (2020) Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 2685-2702, Online. Association for Computational Linguistics.
* Ruff et al. (2021) Lukas Ruff, Jacob R Kauffmann, Robert A Vandermeulen, Gregoire Montavon, Wojciech Samek, Marius Kloft, Thomas G Dietterich, and Klaus-Robert Muller. 2021. A unifying review of deep and shallow anomaly detection. _Proceedings of the IEEE_, 109(5):756-795.
* Schubert et al. (2014) Erich Schubert, Arthur Zimek, and Hans-Peter Kriegel. 2014. Local outlier detection reconsidered: a generalized view on locality with applications to spatial, video, and network outlier detection. _Data mining and knowledge discovery_, 28:190-237.
* Shu and Nakayama (2017) Raphael Shu and Hideki Nakayama. 2017. Later-stage minimum Bayes-risk decoding for neural machine translation. _arXiv preprint arXiv:1704.03169v2_.
* Stahlberg et al. (2017) Felix Stahlberg, Adria de Gispert, Eva Hasler, and Bill Byrne. 2017. Neural machine translation by minimising the Bayes-risk with respect to syntactic translation lattices. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_, pages 362-368, Valencia, Spain. Association for Computational Linguistics.
* Suzgun et al. (2023) Mirac Suzgun, Luke Melas-Kyriazi, and Dan Jurafsky. 2023. Follow the wisdom of the crowd: Effective text generation via minimum Bayes risk decoding. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 4265-4293, Toronto, Canada. Association for Computational Linguistics.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Le ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems_, volume 30. Curran Associates, Inc.
* Yang et al. (2023) Guangyu Yang, Jinghong Chen, Weizhe Lin, and Bill Byrne. 2023. Direct preference optimization for neural machine translation with minimum bayes risk decoding. _arXiv preprint arXiv:2311.08380v1_.

 



## Appendix A Results with COMET20

To support the analysis of this study with a different utility function, we conducted the same experiments as in Tables 1, 2, and 3 using COMET20 Rei et al. (2020). Tables 4, and 5, 6 show the same tendency: the performance varies by sampling methods almost consistently, and the anomaly scores achieve the best correlations with the performance variation. These results confirm the validity of our analysis even with other utility functions.





\begin{table}
\begin{tabular}{l l c r r r r r r r r r r r r r r r}  & \multicolumn{10}{c}{Prob.} & \multicolumn{3}{c}{Sim.} & \multicolumn{4}{c}{\(k\)NN\(\downarrow\)} \\ \cline{5-16}  & _Pseudo-Reference_ & COMET2\(\uparrow\) & Avg.\(\downarrow\) & Cum.\(\uparrow\) & Cand.\(\uparrow\) & Ref.\(\uparrow\) & \(d_{M}\)\(\downarrow\) & 5 & 25 & 50 & 75 & 100 & 5 & 25 & 50 & 75 & 100 \\ \hline \multicolumn{16}{|c|}{**de-en**} \\ \multirow{8}{*}{\(\sigma^{\prime}\)} & Ancestral & 85.82 & **-3.59** & 0.87 & 71.20 & 60.45 & 8.02 & 0.22 & 0.39 & 0.62 & 0.88 & 1.30 & 1.05 & 1.07 & 1.10 & 1.03 & 1.00 \\  & Beam & 85.62 & -0.76 & **1.02** & 85.44 & 83.01 & 15.10 & 0.33 & 0.41 & 0.46 & 0.50 & 0.54 & 1.75 & 1.55 & 1.47 & 1.25 & 1.00 \\  & Epsilon (\(\epsilon=0.02\)) & 85.89 & -0.89 & 0.97 & 84.81 & 82.18 & **6.94** & 0.26 & 0.35 & 0.41 & 0.46 & 0.53 & 1.08 & 1.09 & 1.08 & 1.01 & 1.00 \\  & Epsilon (\(\epsilon=0.02\))* & 85.87 & -0.89 & 0.97 & 84.70 & 82.18 & 8.61 & 0.23 & 0.33 & 0.39 & 0.45 & **0.51** & 1.10 & 1.10 & 1.09 & 1.02 & 1.00 \\  & Nucleus (\(p=0.6\)) & 85.69 & -0.70 & 0.83 & **85.63** & **83.24** & 16.47 & 0.32 & 0.39 & 0.45 & 0.49 & 0.53 & 1.62 & 1.43 & 1.37 & 1.16 & 1.00 \\  & Nucleus (\(p=0.9\)) & **86.04** & -1.50 & 0.95 & 81.66 & 78.02 & 7.38 & **0.18** & **0.27** & **0.35** & **0.42** & 0.57 & **1.04** & **1.02** & **1.01** & **0.99** & 1.00 \\ \hline \multicolumn{16}{|c|}{**en-de**} \\ \multirow{8}{*}{\(\sigma^{\prime}\)} & Ancestral & 87.51 & **-3.60** & 0.65 & 68.41 & 51.56 & 8.16 & 0.22 & 0.44 & 0.74 & 1.09 & 1.71 & 1.08 & 1.11 & 1.11 & 1.03 & 1.00 \\  & Beam & 87.40 & -0.65 & **0.74** & 87.02 & 85.06 & 15.86 & 0.30 & 0.37 & 0.41 & 0.44 & 0.47 & 2.05 & 1.80 & 1.64 & 1.36 & 1.00 \\  & Epsilon (\(\epsilon=0.02\)) & 87.74 & -0.80 & 0.71 & 86.33 & 83.96 & **6.52** & 0.23 & 0.31 & 0.36 & 0.40 & 0.45 & 1.09 & 1.11 & 1.09 & 1.01 & 1.00 \\  & Epsilon (\(\epsilon=0.02\))* & 87.74 & -0.80 & 0.71 & 86.24 & 83.96 & 8.22 & 0.21 & 0.29 & 0.35 & 0.39 & **0.44** & 1.11 & 1.13 & 1.11 & 1.02 & 1.00 \\  & Nucleus (\(p=0.6\)) & 87.57 & -0.61 & 0.62 & **87.12** & **85.10** & 14.95 & 0.29 & 0.35 & 0.39 & 0.43 & 0.46 & 1.68 & 1.49 & 1.42 & 1.21 & 1.00 \\  & Nucleus (\(p=0.9\)) & **87.82** & -1.30 & 0.70 & 83.84 & 79.38 & 7.08 & **0.16** & **0.24** & **0.31** & **0.37** & 0.49 & **1.04** & **1.02** & **1.00** & 1.00 \\ \hline \multicolumn{16}{|c|}{**ru-en**} \\ \multirow{8}{*}{\(\sigma^{\prime}\)} & Ancestral & 88.41 & **-3.75** & 0.60 & 70.67 & 59.20 & 8.13 & **0.20** & **0.35** & 0.56 & 0.83 & 1.25 & **1.04** & **1.03** & **1.03** & **1.00** & **1.00** \\  & Beam & 87.78 & -0.74 & **0.74** & 85.71 & 79.69 & 24.41 & 0.60 & 0.68 & 0.73 & 0.77 & 0.81 & 3.02 & 2.65 & 2.40 & 1.99 & 1.28 \\  & Epsilon (\(\epsilon=0.02\)) & 88.46 & -0.89 & 0.69 & 85.18 & 79.03 & 11.26 & 0.48 & 0.60 & 0.67 & 0.72 & 0.77 & 1.30 & 1.40 & 1.44 & 1.34 & **1.00** \\  & Epsilon (\(\epsilon=0.02\))* & 88.46 & -0.89 & 0.69 & 85.09 & 79.03 & 11.19 & 0.46 & 0.58 & 0.65 & 0.71 & 0.76 & 1.33 & 1.46 & 1.51 & 1.39 & **1.00** \\  & Nucleus (\(p=0.6\)) & 88.26 & -0.69 & 0.60 & **85.96** & **79.91** & 25.16 & 0.59 & 0.67 & 0.73 & 0.77 & 0.81 & 2.89 & 2.40 & 2.18 & 1.83 & 1.15 \\  & Nucleus (\(p=0.9\)) & **88.61** & -1.52 & 0.67 & 82.07 & 75.17 & **7.85** & 0.24 & 0.41 & **0.52** & **0.61** & **0.75** & 1.05 & 1.09 & 1.14 & 1.02 & **1.00** \\ \hline \multicolumn{16}{|c|}{**en-ru**} \\ \multirow{8}{*}{\(\sigma^{\prime}\)} & Ancestral & 82.02 & **-3.85** & 0.37 & 71.25 & 55.20 & 8.91 & 0.23 & 0.42 & 0.67 & 0.97 & 1.40 & 1.07 & 1.13 & 1.13 & 1.05 & **1.00** \\  & Beam & 81.64 & -0.72 & **0.44** & **87.47** & **84.73** & 20.91 & 0.38 & 0.44 & 0.48 & 0.51 & 0.54 & 2.42 & 2.04 & 1.85 & 1.51 & 1.02 \\ \cline{1-1}  & Epsilon (\(\epsilon=0.02\)) & 82.01 & -0.94 & 0.42 & 86.68 & 83.57 & **7.27** & 0.28 & 0.37 & 0.42 & 0.46 & 0.51 & 1.10 & 1.12 & 1.11 & 1.13 & 1.03 & **1.00** \\ \cline{1-1}  & Epsilon (\(\epsilon=0.02\))* & 81.98 & -0.94 & 0.42 & 86.58 & 83.57 & 9.04 & 0.26 & 0.35 & 0.40 & 0.45 & **0.50** & 1.11 & 1.14 & 11.4 & 1.05 & **1.00** \\ \cline{1-1}  & Nucleus (\(p=0.6\)) & 81.76 & -0.80 & 0.37 & 87