Epsilon Sampling Rocks: Investigating Sampling Strategies for Minimum Bayes Risk Decoding for Machine Translation
Authors: Markus Freitag
Behrooz Ghorbani
Patrick Fernandes
Venue: EMNLP 2023 Findings (2023)
Keywords: machine translation
mbr decoding
decoding strategies
bleurt
automatic evaluation
TL;DR: We explore how different sampling approaches for generating candidate lists for MBR decoding affect performance. MBR decoding based on epsilon-sampling significantly outperforms beam search and MBR decoding with all other tested sampling methods.

Abstract: Recent advances in machine translation (MT) have shown that Minimum Bayes Risk (MBR) decoding can be a powerful alternative to beam search decoding, especially when combined with neural-based utility functions. However, the performance of MBR decoding depends heavily on how and how many candidates are sampled from the model. In this paper, we explore how different sampling approaches for generating candidate lists for MBR decoding affect performance. We evaluate popular sampling approaches, such as ancestral, nucleus, and top-k sampling. Based on our insights into their limitations, we experiment with the recently proposed epsilon-sampling approach, which prunes away all tokens with a probability smaller than epsilon, ensuring that each token in a sample receives a fair probability mass. Through extensive human evaluations, we demonstrate that MBR decoding based on epsilon-sampling significantly outperforms not only beam search decoding, but also MBR decoding with all other tested sampling methods across four language pairs.

Epsilon Sampling Rocks: Investigating Sampling Strategies for Minimum Bayes Risk Decoding for Machine Translation
Markus Freitag
Behrooz Ghorbani"
Patrick Fernandes
Google Research frei Freitag@google.com
OpenAl
Carnegie Mellon University Instituto Superior Técnico pfernandēcs. cmu edu
ghorbani@openai..omm
Abstract
Next Token Probebilities According to the Model
Sampling 0.02)
Recent advances in machine translation (MT) have shown that Minimum Bayes Risk (MBR) decoding can be powerful alternative to beam search decoding, especially when combined with neural-basee utility functions. However, the performance of MBR decoding depends heavily on how and how many candidates are sampled from the model. In this paper, we ex- plore how different sampling approaches for generating candidate lists for MBR decoding affect performance. We evaluate popular sam- pling approaches, such as ancestral, nucleus, and top-k sampling. Based on our insights into their limitations, we experiment with the recently proposed epsilon sampllnn (Hewitt al., 2022) approach, which prunes away all tokens with probability smaller than epsilon. ensuring that each token in a sample receives a fair probability mass. Through extensive hu- man evaluations, demonstrate that MBR de- coding based on apsilon-sampling significantly outperforms only beam search decoding, but also MBR decoding with all other tested sampling methods across four language pairs.
Top-K Sampling (K 10)
Nuc Sampling 0.9)
222
Correct Takee
Token Probability
0.3
0.2
0.1
0.0
1.0
1.5
20
25
30
Token Index (Sorted Order)
Figure 1: Sorted next token prediction probabil- ities for an example sentence from newstest2021 English--Geman, For simplicity, we plot only top 30 tokens (out of 32k). The correct token is the second most likely token (with 19.3% probability). All but five tokens have probability of less than 0.02. However, in aggregate, these low- oww-probability tokens have 21.4% probability.
experimented with different utility functions and showed that the sampling-bar MBR decoding approach works well with neural metrics that are fine-tuned on human judgment, such as BLEURT and COMET (Sellam et al., 2020; Rei et al., 2020), significantly outperforming beam search decoding in expert-based human evaluation.
1 Introduction
MBR decoding has recently gained attention in Machine Translation (MT) as a decision rule with the potential to overcome some of the biases of beam search decoding in NMT (Eikema and Aziz, 2020; Müller and Sennrich, 2021; Eikema and Aziz, 2021; Freitag et al., 2022a; Fernandes et al., 2022). While most prior work on MBR decoding for MT is based on k-best lists obtained via beam search, Eikema and Aziz (2020) proposed to use approx- imation of MBR decoding based unbiased sam- pling overcome the shorteomings of MAP de- coding. They demonstrated that samples from the NMT model are faithful to the training data statis- tics, while beam search is not. Freitag et al. (2022a)
In this work, we continue this exploration while focusing on the sampling approach used for gen- erating the candidate lists for MBR decoding. We compare MBR decoding using BLEURT on pop- ular sampling approaches such as ancestral sam- pling, nucleus sampling, or k-best sampling and an- alyze their advantages and disadvantages. Based on these insights, explore MBR with the recently- proposed epsilon-sampling (Hewitt al., 2022) ap- proach, which instead of considering only tokens that fall within an aggregated probability mass (nu- cleus sampling) or a fixed amount of tokens (k (k-best sampling), prunes away all tokens with a probabil-
Work done while working at Google
Work done while a Student Rescarcher at Gongle

---

ity smaller than epsilon. By doing so, it ensures that every token in each sample gets a decent amount of probability mass from the model. This is not always the case when sampling with nucleus or top-k sampling and the incorrect samples can hurt MBR decoding performance especially for long sequences. In addition, we also explore the rela- tionship between the underlying sampling approach and the sampling's temperature.
(low-probability tokens) is often believed to be highly unreliable. Other sampling strategies at- tempt to alleviate this issue by systematically trim- ming the tail.
Top-k Sampling: Top-k sampling is a simple modification of ancestral sampling that steers the generation towards high probability tokens. Let St.k be the set corresponding ity tokens at time Then top-k sampling chooses token y at time with probability proportional to
highest probabil-
Our contributions can be summarized as follows:
We conduct experiments with MBR decoding based on candidate list generated by ancestral, nucleus, top-k and epsilon sampling.
Pmodel (ylæ, Vi - if y E St,k. {
otherwise.
We run extensive human evaluations directly verifying the effects of different sampling methods on the model generation quality.
Nucleus Sampling: Similar to top-k sampling, nucleus sampling (Holtzman et al., 2019) also steers the generation away from the lower trail of the model distribution. Let Qt.p be the smallest possible set of tokens that covers fraction p of the posterior model probability at time t. Then nucleus sampling chooses token y at time t with probability proportional to
We demonstrate that MBR decoding based on epsilon sampling significantly outperforms not only beam search, but also MBR decoding on all other tested sampling approaches on 4 tested language pairs.
{
Pmodei(ula, U1 11 if y E Qt.p. 0
We conduct expert-based MQM evaluation to verify the quality of our translations.
otherwise.
As Qt. is the smallest possible set covering the predefined fraction p, it includes only the upper tail of the distribution and discards p fraction of the tokens from the lower trail.
2 Methods
2.1
Sampling Approaches
In this section, we provide a brief overview of sampling methods considered in this study. In our analysis, denote random variables with bold upper-case letters and their realizations with lower-case plain letters. Let Pmodel (Y, = yr X = x, YI--1 = 91tt-1 denote the probability as- signed by the model to token Y/2 at time f, condi- tioned on the source (X = x)) and the target tokens generated so far (Y1:-1 = 91:1-)) ). To simplify the notation, when there is no room for misinter- pretation, often omit the random variables and simply write Pmodel (Vel2r, 11. We consider the following strategies for generating samples from the model:
Epsilon Sampling: Epsilon sampling (Hewitt et al., 2022) employs simple, yet effective, strat- egy for pruning unreliable, low-probability tokens. Let e S 1 be non-negative threshold. Epsilon sampling chooses token y at time / with probability proportional to
{ Pmooee(y)z: Bhat 31--1) 1/
Pmodee (ylær, Via 1)26,
otherwise.
In their study, Hewitt et al. (2022) argue that epsilon sampling breaks the relative probability principle. F.g. the prompt The should allow many valid continuations. This is particularly true for open-ended generation tasks (e.g. story genera- tion), where there are many possible continuations for given prompt. In machine translation, how- ever, epsilon sampling can be useful technique. This is because machine translation is a conditional generation task, where the output conditioned on the input. This means that there are only a limited number of possible continuations for a given input.
Ancestral Sampling: Ancestral sampling sim- ply draws y/ from ( xr Un. 1/T Here, T is the sampling temperature hyper-parameter which determines the peakedness of the distribu- tion. While this approach is simple and faithful to the model, errors present in the estimated model distribution Pmnddl- In particular, the tail of the distribution
highly sensitive to biases and

---

2.2 Minimum Bayes Risk Decoding
the underlying model distribution. The cost of sam- pling linear in the size of the set. Cross-utility can involve evaluating a large neural network as well and the cost of utility computation is gener- ally quadratic the size of the number of samples. It is important to add that we generate indepen- dent samples which implies that sentences with higher model probabilities have a higher chance to be drawn several times. By doing so and not deduping the candidate lists, we do not need incorporate (again) the model probabilities during MBR decoding.
MBR, or Minimum Bayes Risk, decoding al- gorithm that relies on two essential components: a (translation) model and utility metric. The trans- lation model minde (y/zr) estimates the probability of any target segment y given source segment The utility metric r), estimates quality of candidate translation h given reference transla- tion r.
Given a set of hypotheses H. we would like to select the best hypothesis according to its expected utility with respect to the distribution over human references in the space of all sequences Ω2 i.c.
3 Experimental Setup
A best
arg max Ey (a hEH.
(k)) (1)
3.1 Data and Model
arg max
P (rlz).
kCH
We run experiments on four language pairs: English<+Gennnn
rEs
(Ens>De) and English+Chinese (En++Zh). We use an in- ternal web-crawled dataset for training our models. We filter out noisy examples with contrastive data selection proposed by Wang et al. (2018). After filtering, have 625 million training examples for En language pair and 1.32 billion training examples for En+*Zh We use newstest2019 as our dev set to pick checkpoints and newstest2021 (Akhbardeh et al., 2021) as our test set.
Since P Phumaan(T (r|z) is unknown, we need to rely on the model estimate instead, i.e.
h model
arg max
Σ
u(h, y) (model())z) (2)
bEH
VEQ
This substitution assumes that the model provides good approximation for the true underlying (hu- man translation) distribution. As Ω2 the space of all sequences, infinite, it is impossible integrate over it, and so MBR relies on Monte-Carlo (MC) estimation, using a finite number of pseudo refer- ences Hmodel sampled from the model ( (-(z))
3.2 Model
This yields,
Our translation models are 581M parameter trans- former models (Vaswani al., 2017) with 12 en- coder and 12 decoder layers, model dimension size of 1,024, hidden dimension size of 4,096, and the number of aolti-attention heads 16. Our mod- els use a vocabulary of 32k subword units (Kudo and Richardson, 2018) trained separately for each language pair on the parallel data. We train the models until convergences for around 500,000 up- dates with batch size of 0.5M tokens. We follow the suggestion of Eikema and Aziz (2020) and train our models without label smoothing.
MBR arg max REM
u(h, y).
|mmdee yEHmodel
(3) Commonly, one relies on the same set of model hypotheses for H. (candidate pool) and Hmodel (pseudo--eferencces)) i.e. H = Hmmodel-- In that case, growing Hmodel has two beneficial effects: larger set provides a better approximation of the expected utility (reducing finite sample variance) while the maximum over a finite candidate pool obviously increases as the candidate pool grows.
Growing Hmodel is however computationally costly, both to obtain hypotheses and to evaluate their cross-utility. In all our experiments, adopt approximation to MBR decod- ing (Likema and Aziz, 2020) to generate finite set of samples from a neural machine translation model. Eikema and Aziz (2020) showed that unbi- ased sampling provides a good approximation for
We run beam search with beam size of (larger beam sizes did not improve quality) and length penalty as described in Equation 10 in Wu et al. (2016) using a=0.5. We do not use coverage penalty as this does not improve the results.
the sampling-
For MBR decoding, we generate 1,024 samples for each source sentence and use BLEURT (Sellam et al., 2020) as utility function across the paper.

---

3.3 Human Evaluation
large number of undesirable tokens, while for cases where the model is uncertain, it might prune a large set of valid tokens (Figure 2).
We hired professional translators (7 for En-+De, 5 for De->En, 4 for Zh->En, and 4 for En En-+Zh) and measured translation quality with document context version of MQM (Lommel et al., 2014) which mimics the setup proposed in Freitag al. (2021). This includes using the same error cate- gories, severity levels and error weighting schema. As suggested in the study, we weight each major error with and each minor error with 1, except for minor punctuation errors which get a score of 0.1. The final segment-level score is an average over scores from all annotators. We refer the reader to Freitag et al. (2021) for the details on error cate- gories and annotator instructions.
Aad Biir Fropabitiee (onniert Pocel
Blan Bita Modeli
10
Tepet Sampling
Sarpiig X=121
0.4
0.19
0.0
0075
0.50
98 53 03
37. 35.0
0.000
25 5.0
pwates (iiln)
Figure 2: Top-K sampling is insensitive to model confi- dence.
Nucleus sampling considers the most probable tokens within an accumulated probability mass of p. Even with smaller p values, we sometimes need to consider several hundreds of tokens. Figure 3 shows an example where we need almost 400 to- kens to get an accumulated probability mass of 0.9. Many of the 400 tokens have very low probability on their own, but as their accumulated mass is not small, there is large chance that one of the low probability tokens will be considered by nucleus sampling.
4 Experimental Results
To understand why epsilon sampling the most suitable sampling approach for machine translation, we first investigate the limitations of the commonly used sampling approaches in Section 4.1. We then explore different hyperparameter settings for all sampling approaches in Section 4.2. Since it is not feasible to run human assessment on all hyperpa- rameter settings, we select subset of the highest quality settings based on BLEURT and conduct an expert-based human evaluation to assess the final quality of this subset in Scction 4.3.
Balr (MP)
BesT BANNY Necaoiatto (TEP))
10
4.1 Distributional Properties of Sampling Approaches
10"
100
333
244
400
20:
10
107
NMT models usually generate a dense distribu- tions over the target vocabulary for each time step. Each token assigned a non-zero probability, even when it is completely unrelated to the source query. When combined with large vocabularies, this means that considerable (cumulative) proba- bility mass is assigned to undesirable tokens at the tail of the distribution. Instead of considering the full vocabulary as done in ancestral sampling, nu- cleus and top-k sampling are designed to mitigate this problem by trimming the tail of the distribu- tion (Figure 1). Even though both nucleus and top-k sampling prune away a significant number of tokens, they potentially still consider tokens that have very low probability according to the model distribution.
Figure 3: Nucleus sampling can have unpredictable behavior and lead many low-probability tokens being considered.
We believe that if tokens have a very low prob- ability according the model, we shouldn't 't con- sider them at all. This is precisely the motivation behind using epsilon sampling. By setting an ade- quate threshold e, we can exclude the undesirable low- cow-probyyylity tokens. This leads to pruned set of tokens that has both variable number of tokens and variable accumulated probability mass.
We want to highlight another interesting compar- ison of the different sampling approaches in Fig- ure 4. The cumulative probability mass is larger for epsilon sampling when sampling 1024 times from the model while the other sampling approaches converge to the same area. This might indicate that we can generate more diverse translations when us-
During top-k sampling, the number of non- trimmed tokens is independent of the model con- fidence (entropy): ) this means that for cases where the model is too confident, it might fail to prune a

---

ing epsilon sampling and, importantly, this means we can get better estimations of the true utility in Equation 3.
choose from. For larger candidate sizes, higher temperatures perform better as can be more risky and thus also consider interesting translations with lower model probability, but potentially even higher quality. In this paper, we will focus on large candidate sizes and thus choose r=1.0 for our ex- periments going forward.
Po0ocb Procapitt Conore Bergggos
0.175
0.150
0.325
0.80
0.100
0.075
0.75
0.000
0.025
Sampling
Top- Sampling IK
Nucious Samping 1p- 0.99
0.70
200
600
600
800
1000
BLEURT
Numbor of Sampliss
0.65
Figure 4: (Cumulative) probability mass sentences, covered by each sampling method as we increase the number samples.
Ancesiaa
0.60
Ancestat vas no Ft
Ancestzll Sana ky Irm 2.3:
boan owr
10a
10a
10a
10a
Number of Candidates
4.2 Hyperparameter Search
Figure 5:
Ancestral Sampling for newstest2021 English-Geenen for various temperatures.
All of the sampling approaches introduced in Sec- tion 2.1 have one or two hyperparameters. We ran large number of experiments on English-+German and used this language pair as our development language pair for hyperparameter selection. We then used the exact same hyperparameters for other language pairs to investigate how well they general- ize. All MBR decoding results used BLEURT the utility function. For each sampling approach, picked 1-2 hyperparameter settings purely based on BLEURT. the final evaluation, we will in- vestigate how well the BLEURT scores correlate with human assessment when used both as a utility function and an evaluation metric.
4.2.2 Top-K Sampling
Top-k k sampling has two parameters: and Fig- ure shows different hyperparameter settings. Sim- ilar to ancestral sampling, using a lower tempera- ture helps when the candidate size is small. Once the candidate size is large, the quality of the dif- ferent hyperparameter settings are almost identical. We decided to use the more traditional settings (k=10, 7=1.0), and (k=50, 7=1.0) for our experi- ments going forward.
4.2.1 Ancestral Sampling
0.80
Ancestral sampling has one hyperparameter, the temperature We MBR decoding can- didate lists generated via ancestral sampling with
0.76
0.74
T 0.65, 0.75, 0.85, 1.0, 1.1 and compared per- formance to the translations generated with beam search decoding. In addition to just looking at the performance of MBR decoding using all 1024 sam- ples, we also looked at how well MBR decoding performs when reducing the candidate list size.
0.72
0.70
Tex- sampling 306, Lai
0.66
Teast Sampligg A4-
0.64
In Figure 5, we observe that using lower tem- perature (7) is favorable when the candidate size is small. This is because lower temperature en- courages the model generate more likely tokens, which is important when the candidate list is small and there are only a few possible translations to
been march
109
10a
102
10a
Number of Candidates
Figure 6:
Top-K Sampling for newstest202]
English- -German for different settings.

---

4.2.3 Nucleus Sampling
but show promising results for epsilon sampling. The limitations of traditional sampling approaches, as discussed Section 4.1, I, are that they can con- sider very low-probability tokens especially when combined with a large temperature. This can lead to sharp drop in translation quality.
Nucleus sampling has two hyperparameters: 7 and p. Figure 7 illustrates the performance of differ- ent hyperparameter settings. First, we observe the same behavior that we saw with ancestral and top-k sampling: using lower temperature is favorable when the candidate size is small. This is because lower temperature encourages the model to gener- ate more likely tokens, which is important when the candidate list is small and there are only a few pos- sible translations choose from. Second, we find that using a higher temperature (r=1.5) combined with p=0.9 outperforms all other tested settings when using a large (=1024) candidate list. This is because higher temperature allows the model to generate more creative and interesting translations, while p=0.8 ensures that the model does not gen- erate many low-probability tokens. Therefore. we will use this setting (p=0.9, r=1.5) in our final evaluation.
We chose two settings for our final evaluation: The highest scoring setting (r=2.0, c=0.02) and (r=1.0, c=0.02) with the same epsilon threshold, but lower temperature measure the direct impact of a large temperature during epsilon sampling.
0.80
0.78
0.76
0.72
0.70
Gorler Sand in 6.000 Abl..1
0.68
Oo Sund io Sr EL
Caster sand iigin (--6.1) 2-2
20 50 100 200 500 10002000
Number Candidates
0.76
BLEURT w.et. re
Figure 8:
Epsilon Sampling for newstest202
English-GGerman -German for different settings.
0.72
Muceat Samsinng 16-18 15)
Nucios So rm101
4.3 Final Evaluation
Samoing 1o 2-
The experimental results of MBR decoded trans- lations with 1024 samples comparing the six sam- pling settings chosen Section 4.2 including an expert-based MQM human evaluation are summa- rized in Table Our findings are consistent across all four tested language pairs. Based on the expert- based MQM human evaluation, we come to the following conclusions:
0.68
Nccrat Stn.iiiiooii
bain soarch
10a
10a
102
10a
Number Candidates
Figure 7:
Nucleus Sampling for newstest2021
Anglish-GGemann for different settings.
4.2.4 Epsilon Sampling
MBR decoding with BLEURT outperforms beam search decoding across all four language pairs independent of the underlying sampling approach.
Epsilon sampling has two parameters: and Figure 8 shows the performance of different pa- rameter settings with MBR decoding. Similar to the other sampling strategies, low temperature (r) is needed to yield good translation quality for small candidate sizes. Increasing the temperature results higher BLEURT scores when using large candidate list.
For all language pairs, MBR Epsilon (c=0.02, 7=1.0) sampling outperforms all other MBR runs.
Higher temperature in epsilon sampling lead "reward over-fitting:; even though r=2 generally has higher BLEURT scores (the util- ity function) for most language-pairs, humans tend to prefer translation obtained from 7=1. This indicates that that high temperature might
We have highlight that very high temperatures (T 22) yield sharp drop in translation quality when used for the traditional sampling approaches!
'Ancestral, top-k k and nucleus sampling with 7=2.0 yield 0.45, 0.65 and 0.7 BLEURT for candidate list size of 1024.

---

Automatic Evaluation
Model
Human Eval
BLEU
CHRf
BLEURT
COMET20
logP
MQM 4
Human Transl, (Ref-D)
60.9
76.1
59.0
-30.6
0.55
Epsilon (c=0.02, 1.0)
34.7
63.6
78.6
61.9
-10.8
1.04
Top-k (k=10, 1=1.0)
79.0
61.9
-13.6
MBR
Top-k (A=50,==1.0)) 1=1.0)
32.6
62.3
79.1
61.3
-15.6
1.15
Ancestral (r=1.0)²
33.3
62.7
78.9
61.1
-15.0
1.16
Nuclens (p=0.9, 1=1.5)
31.9
61.9
78.9
60.8
-15.8
1.25
Epsilon (c=0.02, 1=2.0)
27.5
59.0
79.6
60.0
-20.5
1.26
Beam 4
37.1
65.1
74,4
58.0
-5.4
1.36
(a) English Gerrann (Ref-C)
Automatic Evaluation
Model
Human Eval
BLEU
CHRF
BLEURT
COMET20
logP
MQM +
Human Transl, (Ref-A)
28.0
28.2
63.5
42.7
-54.2
1.86
Epsilon (c=0.02, 1=1.0)
29.6
26.9
67.2
44.1
-13.4
3.61
Top-k (k=10, r=1.0)
29.3
26.8
67.8
44.0
-16.2
3.97
MBR
Nucleus (p=0,9,
28.2
26.1
68.6
45.6
-20.6
4.09
Top-k (k=50, 1=1.0)+
28.7
26.3
68.5
45.1
-18.9
4.10
Ancestral (r=1.0)²
28.4
26.1
68.5
44.9
-0..0
4.12
Epsilon (c=0.02, ==2.0)
26.6
25.0
68.9
45.1
-23.4
4,40
Beam 4
29.6
28.5
61.9
36.0
-7.6
4.77
(b) English CChinese (Ref-B)
Automatic Evaluation
Model
Human Fval
BLEU
CHRF
BLEURT
COMET20
logP
MQM +
Epsilon (c=0.02, r=1.0)
33.1
60.9
76.1
62.8
-6.9
1.07
MBR
Nucleus r=
31.0
59.4
76.3
61.3
-9.4
1.22
Epsilon (c=..02, r=2.0y
28.2
57.8
76.4
60.5
-11.4
1.25
Human Transl. (Ref-B)²
29.5
57.7
73.5
56.4
-26.7
1.26
Top-k (k=10, 1=1.0)*
32.1
60.4
76.3
62.4
-7.8
1.31
MBR Top-k (k=50, 1=1.0)
31.9
60.2
76.3
62.2
-8.5
1,38
Ancestral (r=1.0)²
32.0
60.4
76.2
61.8
-8.7
1,40
Beam 41
34.7
62.4
74.5
61.1
-4.7
1.43
(c) German- +English (Ref-A)
Automatic Evaluation
Model
Human Eval
BLEU
CHRF
BLEURT
COMET20
logP
MQM
Human Transl. (Ref-B)
28.2
59.6
69.6
47.3
-53.1
1.53
Epsilon (c=0.02, r=1.0)
70.2
43.4
-18.3
3,02
Top-k (k=10,, r=1.0)
70.4
43.2
Top-k (k=50, 1=1.0)
25.3
55.5
70.2
42.1
-23.1
3.29
MBR
Ancestral (f=1.0)*
25.2
55.5
70.1
41.8
-24.2
3.49
Epsilon (c=0.02, T=2.0)+
70.1
41.8
-28.4
3.38
Nucleus (p=0.9 1=1
24.1
54.5
69.8
39.9
-27.4
3.57
Beam 4t
27.2
54.5
65.8
30.2
-10.5
3.61
(d) Chinese Baanesse=Englsshh +English (Ref-A)
Table 1: Actual utility, log-likelihood (logP) and MQM score for different MBR methods and beam search on newstest2021. All MQM results labelled with are significantly worse than MBR Epsilon (c=0.02, 7=1.0) on PERM-BOTH significance testing (Deutsch et al., 2021) with p=0.001.

---

be exploiting weaknesses in the utility (Am- rhein and Sennrich, 2022).
They demonstrated that neural-based utility func- tion like BLEURT and COMET outperform lexical overlap metrics. Further, Fernandes et al. (2022) found that alternatives to ancestral sampling could lead to improved performance of MBR-based de- coding. (Amrhein and Sennrich, 2022) found that, despite promising results with neural metrics in ma- chine translation evaluation, MBR might exploit biases towards faulty translations with high scores that exist in these metrics (reward over-fitting).
Beam search consistently outperforms MBR decoding when combined with any sampling approach, when measuring by BLEU. This highlights the low correlation of BLEU with human judgement as discussed in several pub- lications before (Freitag et al., 2022b).
As observed before, the human translation for newstest2021 German-tEnglish contains many errors and is already outperformed by MBR decoding. This highlights again the importance of acquiring high level human translations when comparing humans with ma- chines.
MBR has since been extensively in used in sub- missions to various machine and speech translation shared tasks with good results (Nowakowski et al., 2022; Jon et al., 2022; Yan et al., 2022), showcas- ing its potential improve translation quality. Besides the approaches discussed in Section 2.1, other sampling approaches that attempt to fix some of the issues with vanilla sampling have been pro- posed. For example, (Meister et al., 2022) intro- duced the concept of typical sampling, which pro- posed pruning tokens whose probability deviates alot from model's (conditional) entropy (so poten- tially both high and low probability tokens), and show this reduces degenerate repetitions and im- proves generation quality.
5 Related Work
Minimum Bayes Risk (MBR) decoding stems from statistical decision theory from the principal of maximisation of expected utility (Bickel and Dok- sum, 1977; Berger, 1985). MBR has been applied parsing (Goodman, 1996; Sima' an, 2003) and speech recognition (Stoleke al., 1997; Goel and Byrne, 2000). The same idea was later applied to bilingual word alignment (Kumar and Byrne, 2002)
Conclusion
and machine translation (Kumar and Byrne, 2004). MBR was used to maximize overlap metrics such BLEU (Papineni et al., 2002) with statistical MT systems (Kumar and Byrne, 2004; Smith and Eis- ner, 2006; Tromble al., 2008).
In this paper, we investigated the impact of differ- ent sampling approaches (ancestral, nucleus, top- k, and epsilon sampling) during MBR decoding. We analysed the limitations of the traditional sam- pling approaches and proposed combining the cently proposed ahrilon-sampling with MBR. Fi- nally, we conducted human evaluations on four lan- guage pairs, and showed that MBR decoding based on apsilon-sampling significantly outperforms not only beam search but also MBR decoding with all other tested sampling approaches. We believe that the results of this study are significant for sev- eral reasons. First, they demonstrate that epsilon- sampling is a promising approach for improving the quality of MBR decoding. Second, they provide insights into the relative performance of different sampling approaches, suggesting that epsilon sam- pling is a promising direction even outside of MBR decoding and for other non open open-ended generation tasks.
After the advent of neural machine translation, most methods relied on beam search to approx- imate MAP decoding (Bahdanau et al., 2015; Gehring et al., 2017: Vaswani et al., 2017). MBR decoding has recently gained attention in MT as a decision rule with the potential to overcome some of the biases of MAP decoding in NMT (Eikema and Aziz, 2020; Müller and Sennrich, 2021; Eikema and Aziz, 2021). While most prior work on MBR decoding for MT is based on k-best lists obtained via beam search, Eikema and Aziz (2020) proposed to use an approximation of MBR decoding based on unbiased sampling to overcome the shortcomings of MAP decoding. They demon- strated that samples from the NMT model are faith- ful to the training data statistics, while beam search is not. We adopt their sampling-bant -based MBR decod- ing approximation in all our experiments. Freitag et al. (2022a); Fernandes et al. (2022) further ex- plored MBR using neural- based utility functions.
In future work, we plan to conduct further human assessment to investigate the impact of using a smaller candidate size on the performance of MBR decoding.

---

Limitations
Koehn, Nicholas Lourie, Christof Monz, Makoto Morishita, Masaaki Nagata, Ajay Nagesh, Toshiaki Nakazawa, Matteo Negri, Santanu Pal, Allahsera Au- guste Tapo, Marco Turchi, Valentin Vydrin, and Mar- cos Zampieri. 2021. Findings of the 2021 conference machine translation (WMT21). In Proceedings of the Sixth Conference on Machine Translation, pages 1- 88, Online. Association for Computational Linguis- tics.
Automatic evaluation of translations generated with MBR decoding can be challenging when the quality metric (BLEURT, COMET) and the utility funetion (in our case BLEURT) are the same or at least re- lated. This is because there may be metric honey pots (Freitag et al., 2020) that yield high metric scores, but are not necessarily of high quality. MBR decoding is particularly susceptible to this issue, as it is designed to find the translation with the highest possible metric score.
Chantal Amrhein and Rico Sennrich. 2022. Identifying weaknesses in machine translation metrics through minimum Bayes risk decoding: A case study for COMET. In Proceedings of the 2nd Conference of the Asia- -aciffcc Chapter of the Association for Computational Linguistics and the 12th International Joint Conference Natural Language Processing (Volume I: Long Papers), pages 1125-1141, Online only. Association for Computational Linguistics.
To address the problem of potential overfitting of the metric, it is necessary to run a final human assessment to verify the quality of the translations. However, human assessment is expensive and time- consuming, so it is not possible to assess all trans- lations generated as part of this study. We had to limit ourselves to subset of translations that we could send out for human assessment. It is possible that some of the translations that we did not assess are of higher quality than the translations that we did assess.
Dzmitry Bahdanau. Kyunghyun Cho, and Yoshua Ben- gio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Truck Proceedings.
James O Berger. 1985, Statistical decision theory and Bayesian analysis; 2nd ed. Springer series in statis- tics. Springer, New York.
Another limitation of MBR decoding with neural metrics is its decoding speed. In our experiments, we had to generate 1024 samples for each source sentence and each hyperparameter setting, and then score each pair of samples with BLEURT. The cost of sampling is linear in the size of the set, and the cost of utility computation is generally quadratic in the size of the number of samples. This makes MBR decoding in its current form very expensive and impractical for most use cases.
Peter J Bickel and Kjell A Doksum. 1977. Mathematical statisties: Basic ideas and selected topics. Holder- Day Series in Probability and Statistics, Holder-Day, San Francisco.
Daniel Deutsch, Rotem Dror, and Dan Roth. 2021. A statistical analysis of summarization evaluation met- ries using resampling methods. Transactions of the Association for Computational Linguistics, 9:1132- 1146.
Ethics Statement
Bryan Fikema and Wilker Aziz. 2020. Is MAP decoding all you need? the inadequacy of the mode in neural machine translation. In Proceedings of the 28th Inter- national Conference on Computational Linguistics. pages 4506 4520, Barcelona, Spain (Online). Inter- national Committee on Computational Linguistics.
The annotators were compensated fairly and did not have disclose any personal information dur- ing the annotation process. All of the test sets used in this study are publicly available, and annotators were allowed to label sensitive information nec- essary.
Bryan Eikema and Wilker Aziz. 2021. Sampling- based minimum hayes risk decoding for neural machine translation.
Patrick Fernandes, António Farinhas, Ricardo Rei, José G. C. de Souza, Perez Ogayo, Graham Neubig. and Andre Martins. 2022. Quality-aware decoding for neural machine translation. In Proceedings of the 2022 Conference of the North American Chap- of the Association for Computational Linguistics: Human Language Technologies, pages 1396-1412, Seattle, United States. Association for Computational Linguistics.
References
Farhad Akhbardeh, Arkady Arkhangorodsky, Mag- dalena Biesialska, Ondfej Bojar, Rajen Chatter- jec, Vishrav Chaudhary, Marta R. Costa-jussa, Cristina España-Bonet, Angela Fan, Christian Fe- dermann, Markus Freitag, Yvette Graham. Ro- man Grundkiewizz, Barry Haddow, Leonie Harter, Kenneth Heafield, Christopher Homan, Matthias Huck, Kwabena Amponsah- -aapoonsa-KKakyire.. Jungo Kasai, Daniel Khashabi, Kevin Knight, Tom Koemi, Philipp
Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021.

---

Experts, errors, and context: scage-scale study of human evaluation for machine translation. Transac- tions of the Association for Computational Linguis- tics, 460-14477
Shankar Kumar and William Byrne 2002. Minimum Bayes-risk word aliguments of bilingual texts. In Pro- ceedings of the 2002 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP 2002), pages 140-147. Association for Computational Lin- guistics.
Markus Freitag, David Grangier, and Isaac Caswell. 2020. BLEU might be guilty but references are not innocent. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lunguage Process- ing (EMNLP), pages 61-71, Online. Association for Computational Linguistics.
Shankar Kumar and William Byme. 2004. Minimum Bayes-risk decoding for statistical machine transla- tion. In Proceedings of the Human Language Tech- nology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 169-176, Boston, Mas- sachusetts, USA. Association for Computational Lin- guistics.
Markus Freitag, David Grangier, Qijun Tan, and Bowen Liang. 2022a. High quality rather than high model probability: Minimum Bayes risk decoding with neu- ral metrics. Transactions of the Association for Com- putational Linguistics, 1-82..
Arle Lommel, Hans Uszkoreit, and Aljoscha Burchardt, 2014. Multidimensional Quality Metrics (MQM): A Framework for Declaring and Describing Translation Quality Metries. Tradumâtica, pages 0455-463.
Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Koemi, George Foster, Alon Lavie, and André F. T. Martins. 2022b. Results of WMT22 metrics shared task: Stop using BLEU- - neural metrics are better and more robust. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 46-68 68, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
Clara Meister, Tiago Pimentel, Gian Wiher, and Ryan Cotterell. 2022. Typical decoding for natural lan- guage generation. CoRR, abs/2202.00666.
Mathias Müller and Rico Sennrich. 2021. Understand- ing the properties of minimum bayes risk decoding in neural machine translation. In Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing (ACL-IJCNLP 2021).
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017. Convolutional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine Learn- ing, volume 70 of Proceedings of Machine Learning Research, pages 1243-1252, PMLR.
Artur Nowakowski, Gabriela Palka, Kamil Guttmann, and Miko Pokrywka. 2022. Adam mickiewiez uni- versity at wmt 2022: Ner-assisted and quality-aware neural machine translation. In Conference on Ma- chine Translation.
Vaibhava Goel and William Byrne. 2000. Minimum bayes-risk automatic speech recognition. Computer Speech & Language, 14(2):11--13
Joshua Goodman. 1996. Parsing algorithms and met- rics. In Proceedings of the 34th Annual Meeting on Association for Computational Linguistics, ACL '96, page 177-183, USA. Association for Computational Linguisties.
Kishore Papineni. Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th Anmual Meeting of the Association for Compu- tational Linguistics, pages 31 1-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
John Hewitt, Christopher Manning, and Percy Liang. 2022. Truncation sampling as language model desmoothing. In Findings of the Association for Com- putational Linguistics: EMNLP 2022, pages 3414- 3427, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Process- ing (EMNLP), pages 2685-2702, Online. Association for Computational Linguistics.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi, 2019. The curious case of neural text degeneration. arXiv preprint 1rXiv:1904,0975/
Josef Jon, Martin Popel, and Ondvrej Bojar. 2022. Cuni submission in wmt22 general task. ArXiv, abs/2211.16174.
Thibault Sellam, Amy Pu, Hyung Won Chung, Sebas- tian Gehrmann, Qijun Tan, Markus Freitag, Dipanjan Das, and Ankur Parikh. 2020. Learning to evaluate translation beyond english: Bleurt submissions to the wmn metries 2020 shared task. In Proceedings of the Fifth Conference on Machine Translation, pages 921-927 Online. Association for Computational Lin- guistics.
Taku Kudo and John Richardson. 2018 SentencePiece: A simple and language independent subword tok- enizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium. Association for Computational Linguistics.
Khalil Sima' an. 2003. On maximizing metrics for syn- tactic disambiguation. In Proceedings of the Eighth

---

International Conference on Parsing Technologies. pages 183-194, Nancy, France.
A Token-Level Probabilities
We believe that the strong results of MBR decoding based on epsilon sampling are mainly due to the epsilon threshold, which avoids translations that contain at least one token with low (smaller than epsilon) token-level probability. By adding this ep- silon guard, we avoid translations where not every token backed up by at least some decent probabil- ity mass from the model. To investigate this hypoth- esis, we looked at the MQM human annotations, in particular at the major error spans annotated for the other sampling strategies, to connect error spans with token-level probabilities. Some exam- ple translations for German in Table 2. Error spans labeled by professional translators are highlighted in red. Interestingly, we can see that all major error spans contain least one token that has token-level probability smaller than 0.02. Consequently, none of the translations generated by MBR decoding based on either top-k or nucleus sampling are in the candidate pool when doing MBR decoding with epsilon sampling.
David Smith and Jason Fisner. 2006. Minimum risk annealing for training log-linear models. In Proceed- ings of the COLING/ACL 2006 Main Conference Poster Sessions, pages 787-794.
Andreas Stolcke, Yochai Konig, and Mitchel Weintraub. 1997. Explicit word enror minimization in n-best list rescoring. In Fifth European Conference on Speech Communication and Technology.
Roy Tromble, Shankar Kumar, Franz Josef Och, and Wolfgang Macherey. 2008. Lattice minimum bayes- risk decoding for statistical machine translation. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 620- 629.
can be seen
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Wei Wang, Taro Watanabe, Macduff Hughes, Tetsuji Nakagawa, and Ciprian Chelba. 2018. Denoising neural machine translation training with trusted data and online data selection. In Proceedings the Third Conference Machine Translation: Research Pa- pers, pages 133-143, Brussels, Belgium. Association for Computational Linguistics.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine trans- lation. arXiv preprint
Brian Yan, Patrick Fernandes, Siddharth Dalmia, Jia- tong Shi, Yifan Peng. Dan Berrebbi, Xinyi Wang. Graham Neubig, and Shinji Watanahe. 2022. CMU's IWSLT 2022 dialect speech tramslation system. In Proceedings of the 19th International Conference on Spoken Language Translation (IWSLT 2022). pages 298-307, Dublin, Ireland (in-per un-person and online). As- sociation for Computational Linguistics.

---

Source Reference
Ein Sicherheitsdienst überwacht das Ausgehverbot.
security service monitors the curfew.
Epsilon (c=0.02, 1=1.0)
A security guard supervises the curfew.
Top-k (k=10, t=1.0)
A security guard keeps watch over the arca.
Nucleus (p=0.9, t=1.5)
A security guard keeps watch over the premises.
(a) Example I: The major crror spans area and premises have token-level prohabilitics smaller than 0.02.
Source
Bundesarbeiisminister Hubertus Heil (SPD) will nach den nun geplanten strengen Vorschriften gegen Missstände in der Fleischindustrie auch andere Branchen überprüffe.. The Federal Minister Labor Hubertus Heil (SPD) wants to investigate other sectors after the currently planned, strict measures aguinst abuses in the meat industry.
Reference
Epsilon (s=0.02, 1=1.0)
Federal Labor Minister Hubertus (SPD) also wants check other industries accord- ing the now planned strict regulations against abuses in meat industry.
Top-k (k=10, t=1.0)
Federal Labor Minister Huhertus Heil (SPD) wants to examine other industrics following the new, strict regulations against abuses the meat industry.
Nucleus (p=0.9, t=1.5)
German Labor Minister Hubertus Heil (SPD) plans to audit other sectors as well, based on the new strict rules that would combat abuses in the meat industry.
Example The token new token level probability smaller than 0.02 in both error spans.
Source Reference
Beruflich angekommen privat noch nicht
Professionally achieved privately there yet.
Epsilon (e=0.02, 1=1.0)
Arrived professionally not privately yet
Top-k (k=10, t=1.0)
Arrived professional level privately yet
Nucleus (p=0.9, t=15))
Newly arrived in professional life but not in private life
(c) Example 3: The first token in all errors spans (Newly, professional, and .) have token-level prohabilities smaller than 0.02.
Table 2: Example translations where avoid major translation errors by using epsilon sampling instead of nucleus or top-k sampling during MBR decoding. Major translation eтrоrs are highlighted in red and were labeled by human annotators as part our MQM evaluation. Interestingly, all these major enrors were avoided when switching epsilon sampling, the error spans contain least one token with token-level probability smaller than 0.02 (the epsilon threshold used in our experiments). In all of the examples, the translations generated with MBR decoding based on epsilon sampling are error-free.


Reviews:
[{"review_id": "XlOH7xxWpZ", "reviewer": "Reviewer_YyrA", "summary": "", "questions": "1.  I'm afraid the content of this paper might be insufficient for a conference paper. The instantiation of epsilon sampling for machine translation might be too incremental. Could you help show some details of your method, especially the differences between your method and the conventional epsilon sampling (especially for MT)? This can help others quickly understand your thoughts and merits in this research.\n2. As the decoding strategy lies in the filtering on probabilites of candidates when decoding, I'm concerned that some other MT approaches which changes the learning objectives with respect to the ground truth probabilities might have different conclusions. For example, for the models whose learning objective excludes label smoothing (which is a common practice when training MT models nowadays), can your method still performs better than other baselines? If so, how would you change the hyper-parameters of your method?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot you for your review.\n\nRegarding your question 1: We use epsilon sampling to generate the candidate list for MBR decoding. We do not simply test epsilon sampling for MT. There are a couple of novel contributions which as far as we know have not been investigated:\n\nWe compare different sampling approaches for MBR Decoding. There are 2 existing papers about MBR decoding for NMT and one is using ancestral sampling, the other is using nucleus sampling. There was never a study which sampling approach performs better. Also no one ever tried MBR decoding with top-k sampling or epsilon sampling.\nBased on the limitation and failure modes of MBR decoding with the popular sampling approaches, we apply epsilon sampling to generate the candidate lists. The usage of epsilon sampling in MT is not only new, but using it for MBR decoding is novel as well. We also investigate the impact of various temperatures on the candidate list and its impact on MBR decogin.\nThe experimental results are pretty good and set a new standard in NMT. We are convinced that this is interesting for the EMNLP community and will have impact beyond MT.\nAll experiments and the choice for epsilon sampling are well motivated. In fact, we implemented epsilon sampling before it was published by others as we designed a sampling approach that should fit MBR decoding as much as possible.\nWe believe that all the points above are more than sufficient enough to justify an EMNLP paper.\n\nRegarding question 2: This is a very good question and we will add a discussion about this in the paper. Our models are actually trained without label smoothing (l 231) as this would corrupt the probability distribution for MBR decoding. This is actually a larger issue for nucleus and ancestral sampling as many incorrect tokens get some decent probability mass assigned (see Section 2 in our paper). For epsilon sampling, this is not an issue as the epsilon threshold prunes away the tokens that some decent probability mass due to label smoothing. We ran a couple of experiments with and without label smoothing and epsilon sampling is quite robust and you do not have to change the epsilon threshold. This is not the case for nucleus or ancestral sampling where we had to change the temperature to get similar results. Thanks again for pointing out this detail. We will add the discussion and the advantage of epsilon sampling to the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. An efficient method for decoding translation output using epsilon decoding.\n2. Quantitative analyses on the performance of epsilon decoding.\n3. Well-written and easy-to-follow.", "reasons_to_reject": "1. It's concerned that this work might be a simple incremental work (See Q1)\n2. A potential risk for this method lies in the generality of MT models for decoding (See Q2)", "questions_to_authors": "1.  I'm afraid the content of this paper might be insufficient for a conference paper. The instantiation of epsilon sampling for machine translation might be too incremental. Could you help show some details of your method, especially the differences between your method and the conventional epsilon sampling (especially for MT)? This can help others quickly understand your thoughts and merits in this research.\n2. As the decoding strategy lies in the filtering on probabilites of candidates when decoding, I'm concerned that some other MT approaches which changes the learning objectives with respect to the ground truth probabilities might have different conclusions. For example, for the models whose learning objective excludes label smoothing (which is a common practice when training MT models nowadays), can your method still performs better than other baselines? If so, how would you change the hyper-parameters of your method?", "paper_topic_and_main_contributions": "This paper investigates the feasibility of combining the machine translation model decoding with epsilon-sampling method, which help filter out the decoding candidates with quite low probabilities. Based on the experimental results, the proposed method can give better quality of translation outputs than other baselines, like top-k and nucleus sampling. Quantitative analyses also demonstrate the effectiveness of this method. Overall, this paper is well-written and easy-to-follow, and it is well-structured. However, the merits of this paper might be not very sufficient for top-tier conference paper.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "V5NXmKBOGm", "reviewer": "Reviewer_b3eK", "summary": "", "questions": "Do you have a suggestion on applying this method in a latency-sensitive situation?\n\nHow would you compare this MBR decoding with MAP decoding and reranking?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for your review!\n\nTo answer your questions: Q1: This is still an active research area. One big advantage of epsilon sampling is that we can use a smaller candidate list than what we used to use for ancestral and nucleus sampling generated candidate lists. Currently the bottleneck is mainly the heavy computational cost of the utility function. Distilling the utility function into a smaller model seems like a promising direction. Q2: For reranking, we need a QE metric as we do not have access to a reference translation. Based on (https://aclanthology.org/2022.tacl-1.47/), reranking underperforms MBR decoding as it sometimes prefers translations that are unrelated to the source. MBR decoding gives us the security net when compared to all the pseudo references to ensure that the translation is also accurate. MAP and MBR decoding are quite different and their output quality can not only differ, but also the translation style. MAP decoding generates very literal, boring translations (also leading to very high BLEU scores), while MBR decoding generates very natural translations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper brings up the MBR decoding to the community which is a nice alternative to the MAP decoding in some cases. Most importantly, it draws a practical conclusion that the epsilon sampling is the best with extensive experiments, which is very informative and helpful for related researchers and practitioners.", "reasons_to_reject": "Although MBR decoding is costly and won't be ideal to use in many cases, it's not the main concern of the topic of this paper. I don't see a clear reason to reject.", "questions_to_authors": "Do you have a suggestion on applying this method in a latency-sensitive situation?\n\nHow would you compare this MBR decoding with MAP decoding and reranking?", "paper_topic_and_main_contributions": "This paper systematically investigates various popular sampling strategies for MBR decoding. The conclusion is that the epsilon sampling is the most promising one. The background part provides a clear overview of all these sampling methods and the MBR decoding itself. The experiments part is comprehensive and solid, which well supports the main conclusion.", "excitement": 5, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "n/a", "Typos_Grammar_Style_and_Presentation_Improvements": "n/a"}, "extra_text": {}}, {"review_id": "mbwZvYV66X", "reviewer": "Reviewer_FrLm", "summary": "", "questions": "The authors should address the weaknesses mentioned in the above section.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for the review. Here are the answers for your questions:\nQ1: There are many use cases where clients actually prefer to get high quality output and can wait for the translations. That said, epsilon sampling is the first step in the right direction, but we need to do further research to come up with a faster, similar reliable approach.\nQ2: We agree that more language pairs would be nice. However, as we use BLEURT as a utility function in MBR decoding, we cannot rely on BLEURT as the evaluation metric any longer (we basically cherry-pick the translation with the highest expected BLEURT from the candidate list during MBR) and need to conduct a human eval. Human evals are expensive and time consuming and conduction human evals for 4 LPs is way more than what is typically conåducted in an EMNLP paper.\nQ3: We feel that it is not necessary to lay out our next follow up research projects in the paper. We will add some suggestions at the end of the paper that include distilling the utility function and further investigate how we can prune the candidate lists.\nQ4: Can you elaborate a little bit more on this. What do you want to investigate in addition to what we already have? We have more experiments that we would have loved to add to the paper, but are limited by the page limit.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper compares popular sampling approaches for generating candidate lists for MBR decoding, including ancestral, nucleus, top-k, and epsilon sampling. It provides insights into the relative strengths and weaknesses of different sampling approaches for MBR decoding.\n\n2. The authors have proposed the epsilon-sampling approach. This approach prunes all tokens with a probability smaller than a given threshold (epsilon), ensuring that each token in the sample gets a fair share of the probability mass. They argue that this approach benefits long sequences where incorrect samples can negatively affect MBR decoding performance. The paper also discusses the effect of the underlying sampling approach and the sampling's temperature.\n\n3. The paper demonstrates through experiments and human evaluations that MBR decoding based on epsilon sampling outperforms not only beam search, but also MBR decoding on all other tested sampling methods. The authors evaluate across 4 language pairs - English <-> German and English <-> Chinese. The paper verifies the quality of the translations through expert-based MQM evaluations.\n\n4. The paper investigates the hyperparameter search for each sampling method, running experiments to determine optimal settings. \n\n5. The paper is well-structured and well-written.", "reasons_to_reject": "1. The authors highlight that MBR decoding, in its current form, is computationally expensive and time-consuming, making it impractical for most use cases.\n\n2. The experiments were conducted on four language pairs. Including more language pairs could have provided a more comprehensive evaluation of the performance of the different sampling strategies.\n\n3. Although the authors have mentioned the limitations in the paper, they should provide a more detailed plan on how they plan to address these drawbacks in their future work.\n\n4. The authors should provide ablation studies that could isolate the impact of specific factors underlying epsilon sampling's strong performance.", "questions_to_authors": "The authors should address the weaknesses mentioned in the above section.", "paper_topic_and_main_contributions": "This paper investigates different sampling strategies for generating candidate lists for Minimum Bayes Risk (MBR) decoding for neural machine translation. The performance of MBR decoding depends on how candidates are sampled from the model. The authors evaluate popular sampling strategies, such as ancestral, nucleus, and top-k. To address the limitations, they have proposed the epsilon-sampling approach. This approach prunes all tokens with a probability smaller than a given threshold (epsilon), ensuring that each token in the sample gets a fair share of the probability mass. They argue that this approach benefits long sequences where incorrect samples can negatively affect MBR decoding performance. The paper also discusses the effect of the underlying sampling approach and the sampling's temperature.\n\nTo validate their proposal, the authors conduct experiments with MBR decoding based on candidate lists generated by different sampling methods, including ancestral, nucleus, top-k, and epsilon sampling. They run human evaluations to verify the effects of other sampling methods on model generation quality. The authors demonstrate that MBR decoding based on epsilon sampling significantly outperforms not only beam search but also MBR decoding based on all other tested sampling approaches across four tested language pairs. They also conduct the Multidimensional Quality Metrics (MQM) evaluation to verify the quality of their translations.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper compares different sampling strategies for MBR decoding, including a number of well-known approaches such as top-k and nucleus sampling, and the more obscure epsilon sampling, originally introduced by Hewitt et al., but dismissed as having a "key failure" in high entropy distributions. The main contribution of the paper is showing that in the constrained setting of MT, epsilon sampling is an effective sampling strategy, and in combination with MBR decoding, performs better than other strategies tested in human evaluation.

Reviewers agree that empirical insight by the paper are valuable, and potentially have practical impacts (currently limited by the ineffiency of MBR). There is more disagreement on the more subjective excitement score. On the one extreme, reviewer YyrA finds that repurposing epsilon sampling for MT is an incremental contribution. On the other extreme, reviewer FrLm gives the paper credit for "proposing the epsilon sampling approach", and b3eK for "bring[ing] up the MBR decoding to the community".

Factually, the view of YyrA is more precise, since epsilon sampling is not introduced in this paper, nor can the paper be accepted on the basis of bringing MBR to the community, since it follows a number of other papers on the issue. Still, given the previous obscurity of epsilon sampling, which was even dismissed in the paper introducing it, the empirical findings in this paper are significant and I expect it to have some impact on the field.
Decision: Accept-Findings