Wat zei je? Detecting Out-of-Distribution Translations with Variational Transformers
Authors: Tim Z. Xiao
Aidan Gomez
Yarin Gal
Venue:  (2021)
Keywords: Bayesian Deep Learning
Uncertainty
NMT
Transformer

Abstract: We detect out-of-training-distribution sentences in Neural Machine Translation using the Bayesian Deep Learning equivalent of Transformer models. For this we develop a new measure of uncertainty designed specifically for long sequences of discrete random variables—i.e. words in the output sentence. Our new measure of uncertainty solves a major intractability in the naive application of existing approaches on long sentences. We use our new measure on a Transformer model trained  with  dropout  approximate  inference.   On  the  task  of  German-English translation using WMT13 and Europarl, we show that with dropout uncertainty our measure is able to identify when Dutch source sentences, sentences which use the same word types as German, are given to the model instead of German.

Under review as a conference paper at ICLR 2021
WAT ZEI JE? DETECTING OUT-OF-DISTRIBUTION TRANSLATIONS WITH VARIATIONAL TRANSFORMERS
Anonymous authors Paper under double-blind review
ABSTRACT
We detect out-of-training distiii-titiiiiiinnnn sentences in Neural Machine Translation using the Bayesian Deep Learning equivalent Transformer models. For this we develop new measure of uncertainty designed specifically for long sequences of discrete random variables-i.e. words in the output sentence. Our new measure uncertainty solves a major intractability in the naive application of existing approaches on long sentences. We use our new measure on a Transformer model trained with dropout approximate inference. On the task of Cerman-Englinn translation using WMT13 and Europarl, we show that with dropout uncertainty our measure is able to identify when Dutch source sentences, sentences which use the same word types as German, are given to the model instead of German.
INTRODUCTION
Statistical Machine Translation (SMT, (Brown et al. 1993 Och 2003)
built top of probabilistic
modelling foundations such as the IBM alignment models (Vogel et al. 1996 Brown et al. 1993
Gal & Blunsom 2013), has largely been replaced in recent years following the emergence of Neura
Machine Trans ation approaches (NMT, (Kalchbrenner & Blunsom 2013 Bahdanau al. 2015
Luong et al. 2015: Vaswani et al. 2017)). This change has brought with it huge performance gains to the field (Sennrich et al. 2016). but at the same time we have lost many desirable properties of these models. Statistical probabilistic models can inform us when they are guessing at random on inputs they never saw before (Ghahramani 2015). This information can be used, for example, to detect out-of-training -raaaniinnddditibuuttoo examples for selective classification by referring uncertain inputs to an expert for annotation (Leibig et al. 2017). ). or for human- -ieee-----ttt--eeee approach reduce data labelling costs (Gal et al. 2017; Walmsley et al. 2019: Kirsch et al. 2019).
With new tools in machine learning we now incorporate such probabilistic foundations into deep learning NLP models without sacrificing performance. This field, known as Bayesian Deep Learning (BDL, (Neal 2012 Gal 2016)), is concerned with the development of scalable tools which capture epistemic uncertainty the model's notion of "I don't know". measure of a model's lack of knowledge e.g. due to lack of training data, or when an input is given to the model which is very dissimilar to what the model has seen before. Such BDL tools have been used extensively in the Computer Vision literature (Kendall & Gal 2017: Litjens al.) 2017), and have been demonstrated to be of practical use for applications including medical imaging (Litjens et al. 201 Nair et al. 2020), robotics (Gal et al. 2016 Chua al.) 2018), and astronomy (Hon et al. 2018 Soboczenski ct al. 2018 Hezaveh al. 2017).
In this paper we extend these tools, often used for vision tasks, to the language domain. We demonstrate how these tools can be used effectively on the task of selective classification in NMT by identifying source sentences the translation model has never seen before. and referring such source sentences to an expert for translation. We demonstrate this with state-of-the-art Transformer models, and show how model performance increases when rejecting sentences the model is uncertain about-i.e. the model's measure of epistemic uncertainty correlates with mistranslations.
For this we develop several measures of epistemic uncertainty for applications in natural language processing, concentrating on the task of machine translation ($3). We compare these measures both with standard deterministic Transformer models, as well as with Variational Transformers, a new approach we introduce to capture epistemic uncertainty in sequence models using dropout approximate inference (Gal 2016). We give extensive analysis of the methodology, and compare

---

Under review as a conference paper at ICLR 2021
the different approaches quantitatively in the www.oo--taiing-diiaiie
settings ($4), which shows our proposed uncertainty estimate BLEUVar works well for measuring the epistemic uncertainty for machine translation. We also analyse the performance of BLEUVar qualitatively from both the influence of sentence length and from the linguistic perspective. We finish with discussion in potential use cases for the new methodogy proposed.
The closest NLP task to the above problem definition is the quality estimation (QE) task in Machine Translation (Specia al. 2010 Blatz et al. 22004), which tries to solve a similar problem by predicting the quality of a translation with a score called Human Translation Error Rate (HTER, (Snover et al. 2006)). This is done by training a surrogate QE model on source sentences and their corresponding machine-generatel translations in specific domain, with the target of the surrogate predict the the percentage of edits needed to be fixed. While many methods have been shown to successfully solve the task of estimating the quality of translations (Kim et al., 2017: Martins et al. 2016 2011 7
Kreutzer et al., 2015). by definition QE crucially relies on examples mistranslations
train the surrogate. The assumption that such training data is available is often violated in practice though (e.g. in active learning), thus existing approaches in QE research cannot generally be used to detect coof-raining-ddsrriiiiii
examples (see Appendix [ for detail discussion about the differences between QE and our task, as well as other related work that is similar to ours but not solving the same problem).
BACKGROUND: UNCERTAINTY IN DEEP LEARNING
For most machine learning models, the optimisation objectives give us point estimate of the model parameters, which maximise the likelihood of the model generating the training data (i.e. p(Y|X, = w²), w E Ω s.t. S2 is the set of all possible model parameters, Y, X are the training data). Such point estimate w" gives us very good prediction when the test data follow the same distribution as the training data distribution. Given a new input x² at test time, the model prediction for the corresponding y" is
1 arg max [x" w").
(1)
(OOD) data which it never saw before. Instead, we would wish for the model to indicate its uncertainty towards such inputs. We could use p(j"">>,,"> w²) as an estimate for model uncertainty, but as we show below, it would not be well calibrated one. It might be the case that many w might give equally good predictions on the train set, but might widely disagree with their predictions on OOD data. fact, wt might give arbitrary predictions on OOD training data which is very dissimilar to previously observed inputs. Thus, high score does not distinguish whether x² is OOD or not, and whether we should trust the model's 's prediction.
However, we cannot expect the model to perform well on
2.1 BAYESIAN INFERENCE
Bayesian probabilistic models capture the notion of uncertainty explicitly. Rather than considering a single point estimate w., Bayesian models aim to capture the entire distribution of w from the training data. The resulting distribution is called posterior distribution
p(w(X,Y) p(Y|X, )(w)
(2)
x
At test time, we can make prediction about the corresponding y² by integrating out all possible - ply X,Y) - /
Using the variance of the predictive distribution p(v"|=", XY as the uncertainty measure would have taken into account the variance of w. Hence, an uncertainty measure based on this quantity could be a strong indicator for x being OOD.
2.2 APPROXIMATE INFERENCE
The difficulty in doing Bayesian inference comes from the intractability of calculating the evidence
x(
p(||) p (wW))d
(4)
2

---

Under review as a conference paper at ICLR 2021
There might be closed form solution for simple model. But for most interesting problems, it is too difficult to compute an exact solution. Therefore, approximations are often used for such inference problems.
Variational inference (VI) is pragmatic popular method for doing approximate inference (Jordan et al. 1999). The method involves defining an approximating distribution qo (w),, and trying to find the parameter & for that minimises the Kullback-Leibler (KL) divergence (Kullback & Leibler 1951
between qo (w) and the posterior
(L(g)(W)ll(()x,) = (og
90 p(w|X, XY) dw.
(5)
The resulting 0* gives us the closest approximation for the posterior in the family of distribution go (w). However, calculating the KL divergence here is also intractable we still have to do the integration for the evidence in posterior.
Fortunately, a tractable and equivalent objective for minimising the KL divergence is to maximise the evidence lower bound (ELBO, (Blei et al., 2017)) of go(w))
EL)O(gg)
qo (w)))oo (X w)dw (LL((w))))(W)))
(6)
This is tractable as we know all the distributions (both ((V))),(VVVXX.... w) and p(w) are defined by the model or by our assumptions).
2.3 BAYESIAN INFERENCE IN DEEP LEARNING
Almost all deep models treat the units in a deep neural network as deterministic functions. To adapt Bayesian methods in deep learning. we need first turn our model into a probabilistic model. can be done by modelling the weights in each unit of the network as samples from probability distributions. Such networks with Bayesian neural networks is that the integration in ELBO becomes intractable when we have more than one hidden layer (Gal 2016).
called Bayesian neural networks (Neal 2012). One major challenge
Many works have tried tackling this problem. One practical method proposed by Gal (0066) MC Dropout. Gal (2016) showed that optimising any neural networks with dropout can be viewed as an approximate inference in probabilistic model (when dropout p is tuned correctly), ),), which implies that trained neural network with dropout can be interpreted a Bayesian neural network Gal (2016). Stochastic forward passes with dropout 'turned-on' at test time then correspond to draws from the predictive distribution. Here we extend on these ideas and propose the Variational Transformer, which is based on MC Dropout applied to the original Transformer model. We perform extensive empirical evaluation with this model on the task of NMT. Representing uncertainty in a translation model is the first step towards detecting OOD data. We next discuss how to use this uncertainty effectively, and provide the main contribution of this work.
MEASURES OF UNCERTAINTY FOR NMT
Principally, we care about measuring the variance of a model's outputs around some given input point. In the context of a simple classifier model, the solution is often found by measuring the mutual information between the predicted discrete distribution and model parameters, evaluating the output's entropy, or simply computing the variance of model outputs (Gal) 2016). In the domain of language, however, there are many semantically equivalent alternatives to the same prediction, and is a difficult matter to measure the disagreement between the predicted discrete sequences, which in turn complicates the estimation of variance in the output space. Much worse, when attempting to naively use MI or entropy with long sequences or large sets discrete random variables, we quickly discover that even approximate integration over the product space becomes prohibitive (Kirsch et al. (2019) In order to capture epistemic uncertainty in the task of NMT, we propose several measures of uncertainty appropriate for long sequences of discrete variables (Beam Score and Sequence Probability are measures similar to (Wang et al. 2019 Fomicheva et al. 2020)):
1. Beam Score: we assign a confidence to output y/ generated (using beam search) from input
using the score assigned to y's beam (Wu et al. 2016), where Sringgh_penaltyyrrs.
/++bl
5+1
log (pw- ())x))
BS
(7)
(y; 0.6)
3

---

Under review as a conference paper at ICLR 2021
2. Sequence Probability: assign a confidence to the deterministic model output y generated from input x by taking the log predictive probability under the weight distribution.
SP log (Bw~gg- (w) (zlz))
(8)
length_ Lougth_penaltyytee (y; 0.6)
3. BLEU Variance: ideally, we would like to measure the variance of outputs y as the uncertainty an input z, i.e.:
Var Bunggo (w) (ulep (y u²² Eawggo
(y y¹)²
where ot = E, By--p-yylat (yla) [y). If we treat sentences as points in some high dimensional space, 1ly- y'll corresponds
distance between these two points, which is numerical value representing the difference between two sentences. Thus, any metric for measuring the difference between sentences will allow us to calculate the variance of output y. In our experiments we choose BLEU (Papineni et al. 2002). The BLEU score of candidate text the reference text is number between O and 1, with the value closer to I indicating the two texts are more similad Now; we can estimate the variance at an imput x by producing pairs of outputs from the model and measuring the squared complement of the BLEU between them, i.e. 1yy s/1l = (1 BLEU(y,"")) and we have:
BLEUVar = Eu--qa- (w)
(1 (LEU(U. (v)8
(10)
For the beam score we use the deterministic model found by gradient descent and simply take the probabilities from under its output probabilities. This will be our baseline. For sequence probability and BLEU variance we use MC Dropout (Gal 2016) and take a number of samples (N) to estimate the expectations
log
E Pws (u|z))
SP
(11)
sength_penalte (y; 0.6)
For the BLEUVar approximation, we opt for decoding outputs using beam search applied different model samples (realised by randomising the dropout masks) and measuring the complement BLEU between pairs of these examples.
N
N
BLEUVar R
(1- E
(z), decw, (z)))²
(12)
Additionally, out of the N sample sequences generated by BLEUVar, we need choose one or generate a new sequence as the result for a specific input. In regression, the mean of the samples normally chosen as the result, which is approximation for the predictive mean. In case, the 'mean' of N sentences is hard to derive or even not properly defined. Therefore, we use the sampled sequence that is the closest to the ideal 'mean' as an approximation. Since we can measure the disagreement between any two sentences using BLEU, the sequence that is the closest to the 'mean' of the N samples must have the smallest disagreement with rest of the N - samples. Hence, the final output sequence of the method BLEU Variance is:
Σ (L-BLEU(),E))) + Σ (1- (1-BLEUG/,,WD w.))
j arg min De
(13)
Vj
Vjşi
One remark for the above three methods is that the two methods BS and SP have the same resulting translation y given v, but with different values as its uncertainty estimates. For the method BLEUVar, it uses jū for the resulting translation, and a value in a different range as the uncertainty estimate. This is reflected in the plots from the experiments section. For example, in Figure (b)) BS and SP converged into the same value, while BLEUVar converged to a different value.
3.1 EVALUATING UNCERTAINTY IN SEQUENCE MODELS
A number of uncertainty evaluation metrics have been proposed for standard classifier networks such as the popular ECE and MSE metrics (Guo et al. 2017); however, for sequence modelling these maasiffeation-sseccffe metrics are not applicable. Instead, we opt for the performance versus retention curve method for evaluating our uncertainty measures following Filos et al. (2019).
A common practice in the NMT literature is to scale it up by x x100, i.e. in the range of [0, 100).
The approximations helow should have constant scaling factors, but these don't ton't impact our evaluation metric performancc-retentin curves) we leave them out for simplicity.
4

---

Under review as a conference paper at ICLR 2021
The performance-retenne curve indicates how well uncertainty measure would perform if the k% least certain outputs were deleted from the test dataset. The z-axis ranges along the fraction of data retained, while the y-axis measures some performance metric of the model on the retained data. A inrformaace-rrtenttion curve of well-cali uncertainty measure will see clear and sustained improvement in performance as low-confidence predictions are excluded from the test set; while a the well-calibrated metric's curve. In addition, also calculate the area under curve (AUC) as summary statistic for comparing the curves. A larger AUC corresponds to a higher curve on average.
model will yield curve that either lacks trend or tends to lay beneath
4 EXPERIMENTS
Experiments consist of evaluations on both in-distribution (see Appendix A and out- cut-of-ddisributtion sets. The implementation of the Transformer architecture we use is taken from the Tensor2Tensor (Vaswani et al. 2018) repository.
As discussed in the previous sections we use performance-reteniin curves to evaluate the different uncertainty estimates of our models. We also use scatter plots of the uncertainty versus pairwise BLEU of the predictions to offer another gauge of model uncertainty (see Appendix A..2) To visualise the quality of uncertainty estimates for Transformers, on their uncertainty estimates from the most confident to the most uncertain, and plot BLEU scores as a function of the fraction of data retained starting from the least uncertain oulput. For a good uncertainty estimate, are expecting to see the BLEU scores decrease when the fraction of retained data increases.
order the generated sequence based
The following datasets were used in our experiments: (1) WMT EN + DE: The training set for translation tasks between English (EN) and German (DE) composed of news-commenttryyv//3 with 284k sentences pairs, wmn 3 3-oommmocram/a/ with 2.4m sentences pairs and europarl-v7 with 1.9m sentences pairs, in total 4.6m sentences pairs. The test set was the newstest2014 with size 3k from WMT 2014. (2) WMT NL + EN: The test set for Dutch (NL) English (EN) translation was subset (size 3k sentences pairs) of news- patisscoomnntar--14...4
In the experiments below, we denote the results from methods beam score, sequence probubility, and BLEU Variance as BS, SP and BLEUVar respectively. If suffix is added such BLEUVar-10, then the suffix 10 indicates the number of samples taken during MC dropout.
4.1 OUT-OF DISTRR-DDITIIIUUION EXPERIMENTS
The in-distribution tests showed in Appendix A illustrate that when the training set is large enough, the three uncertainty estimates have similar performance if the test set is in the same domain the training set. However, when the training set only has limited data (e.g. 50k compares to 4.6m), BLEUVar outperforms BS and SP even though the test set is in the same domain as the training set. One explanation is that with the limited amount of training data, the model was not able to learn distribution that capture the data from the test set. Therefore, to some extend, the test data are slightly out-of diittriibbuuionn
In this section, we will have look of experiments under two different obviously disttiibuinon settings (which in our application is equivalent to a domain shift) for evaluating our uncertainty estimates.
The first experiment exploits the fact that the wordings and the sentence patterns might vary across different content domains in the same language. For example, sentences from a legal document and sentences from social media are different in terms of formality, even though they can both be in the same language. If we train a model using data from one domain and test with data from another domain, then such input would be out-of nut-of-distriuutio for the model.
The second experiment explores an extreme case of for translation task from language A to language B, if we test it on input from language C s.t. C + A. it would be an out- out-of-distribution input.
setting. Given trained model
4.1.1 UNCERTAINTY CAUSED BY DIFFERENT CONTENT DOMAINS
To evaluate our uncertainty measures on out-of-content--oeee -ontent--ooain test set, we trained model using only the news-commentarr-v13 data for German English (DE-EN) task. This training set has 284k:
5

---

Under review as a conference paper at ICLR 2021
SP-10
SP-10
BLEUVar-10
BLEUVaR-10
0.4
1.0
0.2
0.4
1.0
Fraction of data retained
Fraction of data retained
(a) In-domain test newstest2014 (size 3k) (b) Out-of-domain test Europarl (size 3k)
Figure I: Uncertainty measure comparisons using the same-domain test set newstex12014 (left) and out-of-rrainin--dmmii test set Europarl (right). The Transformer model was trained for DE
to EN tasks with the news-comunnttary-v/33 EN-DE training set (size 284k) using 350k steps.
Table I: Area under curve (AUC) for all plots in Figure
BS
SP-10
BLEUVar-IB
(a) Test set newstest2014
25.75
25.78
27.23
(b) Test set Europarl
27.65
28.29
29.46
samples in the domain news commentary. During test time, we used both the in-domain test data (newstest2014) and the out-of-domain test data (a subset of Europarl). newstes12014 is in the same domain as the training set. Europarl contains samples extracted from the proceedings of the European Parliament, which has different domain than news commentary.
We expect good uncertainty measure perform well in both in-domain and out-of-domain test set. As shown Figure [ and Table [] BLEUVar outperforms the other two measures by large margin for both test sets. In particular, the unstable performance of BS on the out-of-domain test data indicating such uncertainty measure might be not be reliable dut-of-ddittibuuiion data, i.e. data it never saw before.
4.1.2
UNCERTAINTY CAUSED BY DIFFERENT LANGUAGE DOMAINS
60
SP
60
SP
50
SP 50
50
B0
50
BLEUVar-10 10
BLEUV
Reference IDE to EN
40
40
30
20
0.6
0.8
0.8
1.0
t.3 64 0.3 00 0. 02 1.0
Fraction data retained
Fraction data retained
Fraction retained
(a) DE EN (size 3k)
(b) NL to (size 3k)
(c) DE+NL to EN (size 6k)
Figure 2: Uncertainty measure comparisons using the in-distribution DE-EN test set (a), out- of-distribution NL-EN test set (b) and the combined DE+NL to EN test set (c). The Reference line in (c) corresponds to the BS plot from (a). which only has 3k test data. Therefore it only reaches the fraction 0.5 in this graph. The model was trained for DE to EN task with the full
EN-DE training set (size 1.6m) using 350% steps.
Table AUC for all plots Figure
BS
SP-10
BLEU
(a) DE to EN
40.22
39.43
38.61
(b) NL EN
5.14
5.49
6.75
(c) DE+NL to EN
22.55
25.71
29.74
We trained a Transformer model on WMT13 and Europarl DE (German) to EN (English) sentence pairs (obtaining BLEU 33 on the WMT14 test set). We then evaluated the model on out-of-training- distribution input sentences in NL (Dutch), which shares a large overlapping vocabulary with German (hence input sentences look plausible to non-native speakers). One would hope that such data falling outside of the training distribution would produce model predictions with high uncertainty. A similar experiment with a different language pair (French, German to English) can be found in Appendix B
The BLEU scores differences between Figure 2 and (b) shows that feeding Dutch (NL) sentences into a German to English model does not result in meaningful translations in general. Nevertheless, BLEUVA still provide a better uncertainty estimate with Dutch input (see Figure (b). Table 2 b)).

---

Under review as a conference paper at ICLR 2021
In addition, Figure 2e) shows the performaace-retentin eurves for the combined DE+NL test set, in which BLEUVar outperforms BS and SP by large margin. The fact that BLEUVar is close to the DE-EN reference curve indicates most of the DE input has been assigned with high confidence correctly. As the left half of the BLEUVar curve, which corresponds to the most certain half of the test data, nearly resembles the result from Figure[[aa)
A more interesting result is shown in Figure 3 Given the combined DE+NL test set, BLEU'ar is able to nicely separate the test set into two clusters (DE and NI.) using only the uncertainty estimates without evaluating on the target translation.
1000
10 105 BLEUVar-10 (NL-500
85 (DB-BM ER-)
820
85 (NL-9 EN)
5.00
SP-13 (ML-C10
150
630
300
200
50
100
Uncerta 65 idenem probasiiyy
-00
-30
uncerbaiityy
-30
-25
donfidence frn 5h-10
-10
Uncoriainty /Vbrrbnoes fro BLEUVOr 10
20:00
4000
60.00
8000
(a) BS histogram on a mixed test set (Left corresponds to higher un- certainty)
(b) SP-10 histogram on mixed test set (Left corresponds to higher uncertainty)
(c) BLEUVar-10 10 histogram on mixed test set (Right corresponds to higher uncertainty)
Figure 3: Histograms show uncertainty value for DE-EI (green) and NL-EN (red). Note how BLEUVar-10 is able to clearly separate the un-distribution (green) from dfitt--distribution (red). The model was trained for DE to EN task with the full EN-DE training set (size 4. 4.6m)) using 350k steps.
Note that we do not try solve the problem of language detection here. Instead, we use it as an extreme example to show that our uncertainty metric is able to assign very high uncertainty scores towards out-of-distribution test data while being more confident towards in-distribution data. But our methodology was not trained specifically nor designed for language detection. However, being able to separate the two shows that our methods do reflect the uncertainty of the model very well, it also shows that Bayesian deep NLP models can be very powerful.
4.2 ANALYSIS OF SENTENCE LENGTH VERSUS UNCERTAINTY
Table 3: Average BLEUVar for output sentences of various lengths from Figure 2
Lengths
1-10
11-20
21-30
31-40
41-50
51+
DE-EN (In-dist)
2579.93
1867.11
1613.95
1507.85
1502.78
2794.11
NL-EN (OOD)
4772.16
5388.25
5579.82
6039.02
6705.95
7042.69
Use sentences with different lengths and uncertainties from section $4.1.2 as examples, we can see from the Table thhat that long sentences do not necessarily have consistently low/high uncertainty (BLEUVar). For the in-distribution test data, the model is very certain for sentences of length around 21-50, this is because the training sentences length distribution is centred around these sentence lengths (with 54% of the training data in this interval). For the OOD test data, the model is very uncertain across all sentence lengths, and as the sentences get longer the model becomes more uncertain. Note that average uncertainty for the shortest OOD sentences is 44700, which is much greater than the average uncertainty for the longest in-distribution sentences, ~2700.
An important point is that it is not necessary that all the short sentences must have low uncertainty, even for sentences in-distribution. Lee., our uncertainty metric is not simply measure of sentence lengths (or even correlated with it).
We further extended the experiment design and added a naive baseline which simply looks at the
sentence length to reject sentences (ordering test data using Tongeet_ Beaagree__enttee__eeeeee -enggh i.e. from short sentences to long sentences). Calculating BL.U scores at different retention rates (as in Figure aa) see Table 4), we that the curve is much lower than BLEUVar. In fact, the performance of sentence length referral worse than another naive baseline: randomly referring sentences without looking at them at all.
sentence. length
With the common practice of BLEU (i.e. x x100), the Bar value results in x x000 in the plots.
7

---

Under review as a conference paper at ICLR 2021
Table 4: BLEU scores at different retention rates under three ordering (sentence length here is system output sentence length; source sentence length behaves the same).
Retention
Sentence length
Random referral
BLBUVar
0.2
29.62
32,06
42.87
0.3
31.48
31.93
41.28
0.4
32.38
32.26
39.39
4.3 INTROSPECTION INTO THE MODEL UNCERTAINTY FROM THE LINGUISTIC PERSPECTIVE
Based on experiments section $4.1.2 below are some example sentence translations sampled from the model (and with which we estimate the model uncertainty). As a reminder, the uncertainty here (BLEUVar) is the level of disagreement between sampled sentences, as determined by pairwise-BLEE (pairwise between each model output and the other model outputs). For a very confident translation from an in-disriibution input (i.e. German), refer to Appendix translations sampled from the model are consistent with each other, and the model has no uncertainty at all. For other less certain in-distribution translations as showed in in Appendix D.2 Table I1] we see that each translation sampled from the model is inconsistent with the others in subtle ways, leading to a larger variability in pair-wise BLEU scores. The model has high uncertainty, but still lower than that of the average OOD sentence. In contrast, Table shows 3 truncated samples from OOD input (i.e. Dutch), the complete table with 5 full samples can be found in Appendix D.3 Here each translation sampled from the model is wildly inconsistent with the others, with some translations reminiscent of nonsense translations often encountered with neural systems when these are run on inputs they never saw before. We can identify these bad translations by the large variability in pair-wise BLEU scores. The model has much higher uncertainty than that of the average in-distribution sentences.
TÓ We can see that all
Table 5: Out Oft-of-distribure distribution NL source sentence from the experiment in Figure c).
Source sentence (NL):
De debiteurenlanden zouden hun concurrentiekractt terugkrijgen; hun schulden zouden in rečle termen afnemen: de dreiging van staatsbankroeten zou met de ECB onder hun controle verdwijnen, en hun leenkosten zouden dalen naar een niveau dat vergelijkbaar is met dat van het Verenigd Koninkrijk.
Reference translation (EN): (only used to compute "BLEU to reference")
Dehtor countries would regain their competitivenes;; their deht would diminish in real terms; and, with the ECB under their control, the threat of default would disappear and their borrowing costs would fall levels comparable to that in the United Kingdom.
Model predictive-mean translation (EN): (averaging over predictive probabilities during decoding) of terugkrijgen; they
The debitenlands were to compete with the were in debt in the countries of wtwtarriaaaaafffffffffffaaaaaaae
Translation "BLEU to reference"
1.9
Translation uncertainty:
8617
Translations sampled from the model: (3 shorten samples from predictive prohabilities during decoding)
1 The debitenlands were the ones to compete in their avaliivarivaaiivaliivavvvlll sect-denominnted in their afafafaaaafiffafaffn the tripthirtirrtirvenn of state Bankrbankrbankrbann with the ECB in its control the run the run-off- the run the run-run run run run run of the ECB. 2 In the bebdebbdehdedeebts were competitive in terms of law; those debt owed in debt in debt; the three of state bankrbankrhankr- with the ECB, in its, in its, in its, in its control business business the ECB, in its, in the control- the dispute, the husiness- the dispute, the dispute,
of them; they were
3 At the time of its independence, it was aaaaiiaariaaiiialivvavvvllees three-three of the bankrbankrraankrall with the ECB in its control of the ones- the ones in question- the parties in question, the countries in the future; the three of the bunkrbankrbankktaakkbbaa
one; the debts of the poor; the
FUTURE DIRECTIONS
With the new tools above we can now develop NMT systems which can be deployed in scenarios where high trust is required of the system, for example in legal applications. With the new tools proposed we can integrate expert annotation in the deployment phase of the system by referring uncertain sentences to human annotation instead of automatic one. Further, with these new tools future research could examine numan-in-the-lonn approaches NLP. Such approaches will allow us to develop NLP tools in scenarios when hand-labelling labelling of data is too costly, for example language with scarce resources.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
John Blatz, Erin Fitzgerald. George Foster, Simona Gandrabur, Cyril Goutte, Alex Kulesza, Alberto Sanchis, and Nicola Ueffing. Confidence estimation for machine translation. In Coling 2004: Proceedings of the 20th International Conference on Computational Linguistics, pp. 315-321, 2004.
David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Association, 1 112(518):859-877 2017.
Peter Brown, Vincent J Della Pietra, Stephen A Della Pietra, and Robert Mercer. The mathematics of statistical machine translation: Parameter estimation. Computational linguistics, 19(2)263-311, 1993.
Kurtland Chua, Roberto Calandra, Rowan McAllister and Sergey Levine. Deep reinforcement learning in handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, pp. 4754-4765, 2018.
Shrey Desai and Greg Durrett. Calibration of pre-trained transformers. arXiv:2003.07892, 2020.
Angelos Filos, Sebastian Farquhar, Aidan N. Gomez, Tim G. J. Rudner, Zachary Kenton, Lewis Smith, Milad Alizadeh, Arnoud de Kroon, and Yarin Gal. Benchmarking bayesian deep learning with diabetic retinopathy diagnosis. https: /ttp::://itttu com bww.:////bbb--bbearrk...e 2019.
Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. Unsupervised quality estimation for neural machine translation. Trans. Assoc. Comput. Linguistics, 8:539-555, 2020.
Yarin Gal. Uncertainty in deep learning. PhD thesis, PhD thesis, University of Cambridge, 2016.
Yarin Gal and Phil Blunsom. systematic Bayesian treatment of the IBM alignment models. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 969-977, Atlanta, Georgia, June 2013. Association for Computational Linguistics. URL hwtp://wwwwwwwww...w..o......... org/ antho 30tpll1yyy/1111111
Yarin Gal, Rowan McAllister, and Carl Edward Rasmussen. Improving pilco with bayesian neural network dynamics models. In Data-Efficient Machine Learning workshop. ICML, volume 4, 2016.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1183-1192. JMLR. org, 2017.
Zoubin Ghahramani. Probabilistic machine learning and artificial intelligence. Nature, 521(7553): 452 459, 2015.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1321-1330. JMLR. org, 2017.
Yashar D Hezaveh, Laurence Perreault Levasseur, and Philip Marshall. Fast automated analysis of strong gravitational lenses with convolutional neural networks. Nature, 558(7669)5555, 2017.
Mare Hon, Dennis Stello, and Joel C Zinn. Detecting solar-like oscillations in red giants with deep learning. The Astrophysical Journal, 859(1):64, 2018.
Michael I Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul. An introduction to variational methods for graphical models. Machine learning, 37(2):183-233 1999.

---

Under review as a conference paper at ICLR 2021
Nal Kalchbrenner and Phil Blunsom. Recurrent continuous translation models. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1700-1709, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL
https: www.wwwww.... aclweb wwtp./or.aaaa.ooggg.ooooooo
Alex Kendall and Yarin Gal. What uncertainties do we need bayesian deep learning for computer vision? In Advances in Neural Information Processing Systems, pp.
2017.
Hyun Kim, Jong-Hyeok Lee, and Seung-Hoom Na. Predictor-estimate using multilevel task learning with stack propagation for neural quality estimation. In Proceedings of the Second Conference on Machine Translation, pp. 562-568, 2017.
Andreas Kirsch, Joost van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. In Advances in Neural Information Processing Systems, pp. 7024-7035, 2019.
Julia Kreutzer, Shigehiko Schamoni, and Stefan Riezler. Quality estimation from scratch (quetch):
Deep learning for word-level translation quality estimation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pp. 316-322, 2015.
Solomon Kullback and Richard Leibler. On information and sufficiency. The annals of mathemati- cal statistics, 22(1):79-86) 1951.
Aviral Kumar and Sunita Sarawagi. Calibration of encoder decoder models for neural machine translation. arXiv:1903.00800 2019.
Christian Leibig, Vaneeda Allken, Murat Seçkin Ayhan, Philipp Berens, and Siegfried Wahl. Lever- aging uncertainty information from deep neural networks for disease detection. Scientific reports, 1716, 2017.
Geert Litjens, Thijs Kooi, Babak Ehteshami Bejnordi, Arnaud Arindra Adiyoso Setio, Francesco Ciompi, Mohsen Ghafoorian, Jeroen Awm Van Der Laak, Bram Van Ginneken, and Clara Sánchez. A survey on deep learning in medical image analysis. Medical image analysis, 42:60-88, 2017.
Minh-Thang Luong, Hieu Pham, and Christopher D Manning. Effective approaches to attention-based neural machine translation. vrXiv:1508.0402. 2015.
André FT Martins, Ramón Astudillo, Chris Hokamp, and Fabio Kepler. Unbabel's participation in the wmt16 word-level translation quality estimation shared task. Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pp. 806-81 116-811,22 2016
André FT Martins, Marcin Junezys-Dowmunt, Fabio N Kepler, Ramón Astudillo, Chris Hokamp, and Roman Grundkiewicz. Pushing the limits of translation quality estimation. Transactions of the Association for Computational Linguistics, 5:205-2188 2017.
Tanya Nair, Doina Precup, Douglas L Amold, and Tal Arbel. Exploring uncertainty measures in deep networks for multiple sclerosis lesion detection and segmentation. Medical image analysis, 59: 101557, 2020.
Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
Franz. Josef Och. Minimum error rate training in statistical machine translation. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics-Volune pp. 160-167. Association for Computational Linguistics, 2003.
Myle Ott, Michael Auli, David Grangier, and Marc' Aurelio Ranzato. Analyzing uncertainty in neural machine translation. In Proceedings of the 35th International Conference on Machine Learning, ICMI. 2018, Stockholmsmässann Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 3953-3962. PMLR, 2018.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on Association for Computational Linguistics, pp. 31 111-318. Association for Computational Linguistics, 2002.
10

---

Under review as a conference paper at ICLR 2021
Rico Sennrich, Barry Haddow, and Alexandra Birch. Edinburgh neural machine translation systems for WMT 16. In Proceedings of the First Conference on Machine Translation, WMT 2016, colocated with ACL 2016, August 11-12, Berlin, Germany, pp. 371-376. The Association for Computer Linguisties, 2016.
Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. A study of translation edit rate with targeted human annotation. Proceedings of association for machine translation in the Americas, volume 200, 2006.
Frank Soboczenski, Michael D Himes, Molly D O'Beirne, Simone Zorzan, Atilim Gunes Baydin, Adam D Cobb, Yarin Gal, Daniel Angerhausen, Massimo Mascaro, Giada N Amey, et al. Bayesian deep learning for exoplanet atmospheric retrieval. urXiv:/811 1,03390, 2018
Lucia Specia, Dhwaj Raj, and Marco Turchi, Machine translation evaluation versus quality estimation. Machine translation, 24(1):39-50, 2010.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pp. 5998-6008, 2017.
Ashish Vaswani, Samy Bengio, Eugene Brevdo, François Chollet, Aidan N. Gomez, Stephan Gouws, Llion Jones, Lukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. Tensor2tensor for neural machine translation. In Colin Cherry and Graham Neubig (eds.), Proceedings of the 13th Conference of the Association for Machine Trunslation in the Americas, AMTA 2018, Boston, MA, USA, March 17-21, 2018 Volume 1: Research Papers, pp. 193-199, Association for Machine Translation the Americas, 2018.
Stephan Vogel, Hermann Ney, and Christoph Tillmann. Hmm-based word alignment in statistical translation. In Proceedings of the 16th conference on Computational linguistics- Volume 2, pp. 836-841. Association for Computational Linguistics, 1996.
Mike Walmsley, Lewis Smith, Chris Lintott, Yarin Gal, Steven Bamford, Hugh Dickinson, Lucy Fort- son, Sandor Kruk, Karen Masters, Claudia Scarlata, al. Galaxy zoo: Probabilistic morphology through bayesian cnns and active learning. Monthly Notices of the Royal Astronomical Society 2019.
Shuo Wang, Yang Liu, Chao Wang, Huanbo Luan, and Maosong Sun. Improving mack-translation with uncertainty bbaae Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pp. 791-802. Association for Computational Linguistics, 2019.
based confidence estimation. In Proceedings of the 2019 Conference Empirical
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's ' neural machine translation system: Bridging the gap between human and machine translation. arXiv: 1609, 08100/144444 44, 2016.
Edward N Zalta, Uri Nodelman, Colin Allen, and R Lanier Anderson. Stanford encyclopedia of philosophy, 1995.

---

Under review as a conference paper at ICLR 2021
A IN-DISTRIBUTION EXPERIMENTS
This section details our experiments on data that lays within the training distribution for the WMT English (EN) - German (DE) and English (EN) + Vietnamese (VI) tasks. We explore the calibration of Transformer models in this setting and evaluate the effectiveness of using MC Dropout and the proposed methods to measure the model uncertainty.
In addition to WMT13 dataset for EN - DE tasks mentioned in the previous section, we use the IWSLT 2015 dataset for translation tasks from EN VI. There are 133k sentences pairs in the IWSLT 2015 training set and 1.3k sentences pairs in the IWSLT 2015 test set. Both the training and test data for IWSLT 2015 come from the domain of TED talks.
A.1
EVALUATING MODEL CALIBRATION
BS
BS
60
BLEUVar-10 BLEUVar-50
60
SP-10
SP-50
50
50
BLEU
40
30
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.6
1.0
Fraction of data retained
Fraction of data retained
(a) BLEUVar- 10 vs BLEUVar-50
(b) SP-10 vs SP-50
Figure 4: Uncertainty estimator comparisons for different number of samples. The model is trained for EN to DE tasks with 1.6m training data using 350k steps.
Table AUC for plots in Figure f and Figure
BS
SP-10 34.86
SP-50
BLEUVar-10
BLEUVar-50
75.78
34.93
34.14
34.68
The first question we hope to answer the quality of calibration in Transformers models and to evaluate the effectiveness of MC Dropout in improving uncertainty estimates.
The Transformer was trained on the full EN-DE training set (4.6 million samples) for 350k steps. We evaluate on the newstes12014 test set.
SP-10
BLEUVar-10
0.6
Fraction of data retained
Figure 5: BLEU scores different uncertainty estimators under various retained data rates. The model is trained for DE tasks with 4.6m training data using 350k steps.
The results from Figures and 5 suggest that the beam search score provides a well-calibrated uncertainty metric on the in-distribution test data. The second observation is that MC Dropout-based methods seem slightly under-perform beam in this setting (see Table 6), even when the
12

---

Under review as a conference paper at ICLR 2021
number of samples is increased fivefold. In this setting, our proposed metric (BLEUVar) benefits more from increasing the number of dropout samples relative sequence probability.
A.2
THE IMPACT TRAINING SET SIZE
60
BS
SP-10
SP-50
BLEUVar-10
BLEUVar-50
50
50
BLEU
30
30
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Fraction of data retained
Fraction of data retained
(a) 10 samples
(b) 50 samples
Figure 6: BLEU scores for different uncertainty estimators under various retained data rates. The model is trained for EN VI tasks with 133/ training data using 350k steps.
Table 7: AUC for plots in Figure
BS
SP-10
SP-50
BLEUVar-10
BLEUVar-50
35.55
36.25
36.33
35.66
36.92
The WMT EN-DE training set is fairly large and one would assume that most test sentences (or very similar ones) have been observed during training time. Hence we do not expect much epistemic uncertainty to exist in this testing scenario, which the experiments seem to confirm. natural question to ask is the effect of training set size the calibration of models. We explore this question by considering the WMT English to Vietnamese (EN-VI task which has 133k samples in the training set (approx. 2.6% of EN-DE), and down-sampling the EN-DE training set to 50k and 100k samples.
The performance-retention plots in Figure 6 and the AUC in Table 7 indicate that, while large training set yields curves that seem to suggest beam score is sufficient uncertainty metric, when small dataset is used the MC Dropout-based uncertainty metries begin to outperform the beam score (note the retention range 0,0 to 0.2). Moreover, in the small training set setting increasing the number of samples drawn from MC dropout results in a significant improvement for BLEUVar.
40
BS
BS
40
35
SP-10
SP-10
BLEUVar-10
BLEUVar-10
30
25
20
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Fraction of data retained
Fraction of data retained
(a) 50k training data
(b) 100k training data
Figure 7: Uncertainty estimator comparisons for models with different sizes of training set. The models were trained for EN to DE tasks with 50k and 100k training data using 350k steps.
The experiments depicted in Figures and consist of down sampling the EN-DE training set. Figure 7 and Table T demonstrates similar pattern to the above EN-VI experiment when down-sampling the EN-DE data to 504 and 100k examples. Again, in the low-data regime BLEUVar substantially nut-performs beam score and sequence probability. Figure demonstrates the impact of data size
13

---

Under review as a conference paper at ICLR 2021
Table 8: AUC for plots in Figure
BS
SP-10
BLEUVar 10
(a) 50k training data
13.97
13.61
14.89
(b) 100% training data
19.88
19.64
20.44
100
230
10
03
80
Count
Individual BLEU
60
10
Individual
03
10
28
Uncertainty BLEUVar-10
Uncertainty BLEUVar-10
(a) 50% training data
(b) 100k training data
320
00000
0000
Count
Individual BLEU
20
Uncertainty: BLEUVar-10
(c) 4.6m training data
The density of individual BLEU score versus uncertainty (BLEUVar-10) for all sentences in the same test set newstes12014 produced by models trained with various size of data set. The sentences BLEUVar-10. Following the calculation of BLEUVar, since have 10 samples, the uncertainty estimate BLEUVar- has the value range [0, 90). And scale it up x 100, which results in the x-axis has the range [0, 9000]. The models trained for EN DE tasks with 50k, 100k and 4.6m training data using 350k steps.
Figure 8:
ordered by their uncertainty from low (left) to high (right) using
on the distribution of example uncertainty versus performance. We see that low data regimes lead to a low-entropy distribution with high uncertainty across the entire test set; as data availability is increased, uncertainty decreases, and average model performance increases for all rates of data retention.
14

---

Under review as a conference paper at ICLR 2021
B ADDITIONAL OUT-OF-DISTRIBUTION EXPERIMENTS (FR+DE TO EN)
We have done similar experiments as section 44 on other language pair. Figure 9 here uses a similar experiment design Figure 2 in $4.1.2 Instead of testing DE and NL on model trained with DE-EN task, here the model trained with FR (French) EN (English) task and tests FR to EN (in-distribution), DE to EN (OOD) and FR+DE to EN test sets. The pair FR and EN has much less overlap in vocabulary than DE and NL.
The BLEU the full combined test (see Figure the best the models can do; then BLEUVar rejects German sentences until it has mostly French sentences left and has peak performance (data retained=0.3), after which it is forced to reject French sentences as well. Note that performance goes down for small retain rate because there small number of DE (OOD) data erroneously being assigned with high confidence, which our metric captures well.
This result is similar to the DE+NL experiments in $.1.22 with BLEUVar outperforms the rest by large margin in the mixed test set (see Figure (ec) and Table (c)). Further, the BLEU scores of OOD test is roughly flat compared to the amount of data retained (see Figure b)).
B5
B5
B5 SP 10
10
SP.10 BLEUWar-10
BLEUVaa-10
BLLilVee11
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.5
0.8
1.0
Fraction of data retained
Fraction of data retained
Fraction of data retained
(a) FR to EN (size 3k)
(b) DE to EN (size 3k)
(c) FR+DE EN (size 6k)
Figure 9: Uncertainty measure comparisons using the in-distribution FR-EN test (a),
of-distributton DE-EN test (b) and the combined FR+DE to EN set (c). The Reference line (c) corresponds to the BS plot from (a). which only has test data. Therefore it only reaches the fraction 0.5 this graph. The model was trained for to EN task with WMT 2014 English-Freed Freishh--reehhhhe training set (size 36m) using 350k steps.
Table AUC for all plots Figure
BS
SP-10
BLEU 10
(a) FR EN
38.18
38.02
38.53
(b) DE to EN
3.56
3.78
3.82
(c) FR+DE EN
14.70
17.67
25.10
15

---

Under review as a conference paper at ICLR 2021
C RELATED WORK
Our work might look similar to quality estimation (QE) task in MT (peciaa et al. 2010 Blatz et al. 2004), but the problem of QE is fairly different to what we do in this paper. QE assumes the existence of fixed translation system (e.g., an in-house encoder-decoder attention-based NMT system, as in WMT19's shared task in Quality Estimation). The QE models then have to determine the quality of the system's 's output. In contrast, we look at the problem of "introspection" where the system has to decide the the confidence ("quality") of its own output. This confidence can then be used for selective classification where the model can reject some uncertain translation. Further, standard approaches in QE might assume access to privileged data (e.g., the NMT translations for the source sentences and their corresponding human post-edition, as task in WMT19' QE), which we do not require. In addition, most existing approaches for QE require additional model to be trained to estimate the translation quality of MT model, while our method does not have such requirement. Therefore, our method is able to provide uncertainty estimate simply with the parallel corpus used for training the translation model without the need for additional data and training procedure.
The closest to our paper is task 3 in WMT19' QE: correlate to human judgement. We would like to stress that system's 's confidence in its own prediction does not have to be correlated to human judgement. Indeed, we demonstrate this in Appendix [A.2 where model can indicate that does not have enough training data, and requires additional data to increase its confidence (the model's 's subjective view its uncertainty does not have to correlate with empirical mistakes Bayesian epistemology (Zalta et al. 1995)).
metric score sentences is sought, which must
In addition, QE tasks are mainly focus on estimating the in-distribution translation quality, since the test sets are in the same domain as the training sets provided by WMT QE tasks (e.g. both in the IT domain for English-German WMT18 199). In contrast, the goal for our uncertainty estimate is to identify the out-of-distribution translations, rather then estimating the quality of in-distribution translation. Therefore, our tasks are fundamentally different to QE.
There have been some prior attempts at investigating the similar problem as ours. In particular, Kumar & Sarawagi (2019) investigated the calibration of various NMT models at the token level. Kumar & Sarawagi found that many models are ill-calibrated at the token level, leading to the resulting probability distribution over the vocabulary used during decoding is not good reference for model uncertainty. To correct for this, Kumar & Sarawagi design a recalibration strategy that applics an adaptive temperature to the logits, determined by the token identities, attention entropies. and other relevant components. Desaa & Durrett (2020) looked into the calibration of pre-trained Transformers, and discovered that pre-trained Transformers are well calibrated for in-distribution data but ill-calibrated for out-of- -itoo-ddistribttinn data. Such observations on NMT calibration further motivate us to design better uncertainty measures for NMT models.
Another study of uncertainty in NMT models comes from Ott et al. (2018); they found models tend to have overly high uncertainty in their output distribution over sequences. Note that both do not consider epistemic uncertainty, not OOD settings. There are some work consider epistemic uncertainty (Fomicheva et af. 2020 Wang et al.) 2019) and propose MC Dropout-based measures similar to our Sequence Probability (SP). Our work explores this direction and offers a new uncertainty estimation technique BL... Var) that empirically out-performs existing methods by a significant margin.
16

---

Under review as a conference paper at ICLR 2021
D TRANSLATION SAMPLES
D.1 IN-DISTRIBUTION CERTAIN SAMPLES
Table 10: (Low uncertainty) In-distribution DE source sentence from the experiment in Figure a).
Source sentence (DE)
Nevada hat bereits ein Pilotprojekt abgeschlossen.
Reference translation (EN): (only used to compute "BLEU to reference")
Nevada has already completed a pilot.
Model predictive-mean translation (EN): (averaging over predictive probabilities during decoding) Nevada has already completed a pilot project.
Translation "BLEU reference".
70.7
Translation uncertainty:
Translations sampled from the model: (5 samples from predictive probabilities during decoding)
1 Nevada has already completed pilot project.
2 Nevada has already completed pilot project.
3 Nevada has already completed pilot project.
Nevada has already completed a pilot project.
5 Nevada has already completed pilot project.
D.2 IN-DISTRIBUTION UNCERTAIN SAMPLES
Table 11: (High uncertainty) In-distribution DE source sentence from the experiment Figure a).
Source sentence (DE)
Im Grunde genommen sind vegane Gerichte für alle da.
Reference translation (EN) (only to compute "BLEU to reference") Essentially, vegan dishes are for everyone.
Model predictive-mean translation (EN) : (averaging over predictive probabilities during decoding)
Basically vegan dishes are there for everyone.
Translation "BLEU to reference" 34.5
Translation uncertainty:
3122
Translations sampled from the model: (5 samples from predictive probabilities during decoding)
Basically vegan dishes are for everyone.
Basically, vegan dishes are there for everyone.
Essentially, vegan dishes are available for everyone.
Basically, vegane dishes are there for all.
Basically vegan dishes are there for everyone.
17

---

Under review as a conference paper at ICLR 2021
D.3 OUT-OF-DISTRIBUTION SAMPLES
Table 12: Out-of-distribution NL source sentence from the experiment Figure c
Source sentence (NL):
De debiteurenlanden zouden hun concurrentiekracht terugkrijgen; hun schulden zouden in reële termen afnemen; de dreiging van staatsbankroeten zou met de ECB onder hun controle verdwijnen, en hun leenkosten zouden dalen naar een niveau dat vergelijkhaar is met dat van het Verenigd Koninkrijk.
Reference translation (EN) (only used to compute "BLEU to reference")
Debtor countries would regain their competitiveness; their debt would diminish in real terms; and, with the ECB under their control, the threat of default would disappear and their borrowing costs would fall to levels comparable to that in the United Kingdom.
Model predictive-mean translation (EN): (averaging over predictive probabilities during decoding)
The debitenlands were to compete with the irw.ttvaiivaaiivaaivvrrrraaaaaaa of terugkrij- gen: they were in debt the countries of dfhhhiiillacddddiiiiiiiiaaaaaaai
www.eeaiiiiffiiiiiiiiiiiiiiiiiii http:tttwwwtttttttttttttttttaaat afafafafafafaaaf
Translation "BLEU reference":
1.9
Translation uncertainty:
8617
Translations sampled from the model: samples from predictive probabilities during decoding)
1 The debitenlands were the ones compete in their aatdlvaalivaiiaaailllliaavavvaa rivalrivalrivarriaas of them; they were debt-denominated in their afafafafafafaffffen the tripthrthirthirwent of state bankrbankkrbankkraankuu with the ECB in its control- the run the run the run the run the run the run - the run the run - the run-off- - the run the run-run run run run run of the ECB. 2 In the bebbebbdebddebdebbbbbeeebde the httpo//lwbeeebbbbbbddcbebbbbbbbb
were competitive terms of law; those debt owed in debt in debt; the three of state bunkkbbankkbbnttbbkkkbaakk
krbankrbankrbaan www.aankknnnnnnabaaababe. business the ECB, in its, in the control- the dispute, the business the dispute, the dispute, the - business- the dispute, the sovereign At the time of its independence, it was wwv.aaiiaaaiaaiivaiivvvvvaaaaa
with the ECB, in its, in its, in its, in its control - business
3
one; the debts of the poor; the three-three of the bankrbankrbankrall with the ECB in its control of the ones - the ones in question the parties in question, the countries in the future; the three of the wannkkbankrbankkbbnnkbbbb... bankrbankrbankr with the ECB in its control, with the ECB in its control, in the face, the disputes, the financial crises, the financial crisis, the financial crisis, the financial crisis, the financial crisis, the financial crisis, the financial crisis, the financial crisis, the financial crisis, the financial crisis, the financial crisis, the financial crisis, the financial crisis.
4
De debitenlanden zouden hun concomncurenttervvaaiiaaall terugkrijgenn, hun levlevlevlevleevevv ev wwwppooccceeeeeeeeooeoeooooeeeee wwwpecteeeleewoooooeeeeeeeeeeeee
18

---

Under review as a conference paper at ICLR 2021
5
The debitenland hawp:///wwg.......maaaaaaaaaaaaa bgambgamble in their own countries; the wwwpa//wwnnnaaabaaaaaaaaaaaannna
19


Reviews:
[{"review_id": "dSP5G2sOrYf", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes a method for estimating a neural machine translation (NMT) system's uncertainty about its translation of a sentence that has two parts: (1) use MC Dropout as a proxy for integrating out parameters; (2) two uncertainty metrics (probability of translation summing over randomly-sampled parameters and variance in BLEU using randomly-sampled parameters). The baseline method is just to use the probability of the 1-best translation under the MLE parameters. The method is evaluated by measuring the BLEU score of a test set retaining only the most-certain fraction of the sentences.\n\nFor in-domain sentences, BLEUVar does the best, and SP is the same as the baseline. For out-of-domain sentences (train on news-commentary, test on Europarl), the baseline does dramatically worse, and both BLEUVar and SP are better. The authors also tried treating Dutch sentences as \"out-of-domain\" and got similar results (BLEUVar > SP > baseline).\n\nI don’t think I understand the authors’ differentiation of the present task from confidence estimation (e.g., Blatz et al., 2004, http://www.alexkulesza.com/pubs/confest_report04.pdf; Ueffing and Ney, 2007, https://www.aclweb.org/anthology/J07-1003.pdf). The authors write that “by definition QE crucially relies on examples of mistranslations to train the surrogate,” but the features these systems use do not necessarily rely on mistranslations; only the classifier does. The present work avoids building a classifier by evaluating the measure directly using performance-retention curves. That’s fine, of course, but I’m not seeing that the distinction between the present work and previous work on confidence estimation is that sharp. Some of the features from older work could have been used as additional baselines here (e.g., some of the ones on pages 41-42 and 45-46 of the Blatz et al. report cited above). \n\nThe paper uses a lot of space (about 1.5 pages) presenting fairly technical (to me) background on Bayesian inference, which turns out to not be needed for understanding the rest of the paper. I would suggest cutting this section down. On the other hand, the paper spends very little space explaining MC Dropout. To make the paper self-contained, I think it would be helpful to at least give a brief review of what MC Dropout is and how to do it.\n\nSections 4.2 reports some experiments/analysis on the relationship between uncertainty and sentence length. I’m uncertain what the research question here is, and what the reader should learn from this section. Section 4.3 shows one example sentence; again, I’m not sure what question this example is meant to answer.\n\nWould it be valuable to study the relationship between certainty (as measured by your measures) and quality? What would a scatterplot of BLEU vs. BLEUVar or SP look like?\n\nIn any case, I think it could be interesting to show more than a few examples, maybe 10+ examples, to give a sense for what makes the model more or less certain.\n\nOverall, I like this method and think that it has value for the MT community. I’m not totally sure that ICLR is the best place for this work to be published; the WMT conference would have been a better fit.\n\nOther comments:\n\n- Why is the baseline method called \"Beam Score\" when all three methods use beam search? Would it make sense to rename \"Beam Score\" to \"MLE Probability\" and \"Sequence Probability\" to \"MAP Probability\"?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "S0mUomiAsmC", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \n\nThe paper proposes a technique for assessing the uncertainty of a Transformer-based NMT model on a given input $x$. The technique relies on computing a variance-like estimate over a collection of translation candidates for $x$, where these candidates are obtained by perturbing the decoding mechanism through the use of dropout at test time. Experiments compare this technique with other ways of measuring the \"epistemic uncertainty\" of the NMT model. In limited training data conditions, the proposed measure is better aligned with the actual performance of the model than competing measures, and in particular is better able to detect Out-of-Domain translation requests.\n\nPros:- The paper addresses an important problem: how to detect translation requests for which a given NMT model is unlikely to produce good results, so that these requests can be handled in a proper way (e.g. referred to a human translator).\n- Doing that in a way that only relies on intrinsic properties of the NMT model without requiring a specific quality-estimation training setup, which additionally might have difficulty covering all of the possible out-of-domain cases.\n- Proposing to assess uncertainty by an original perturbation mechanism, using dropout at test time, which mimics a form of sampling over the parameter space of the model.\n- The clever way in which, in equation (9), a proxy for variance over symbolic target sequences (which is not well-defined) is obtained in terms of the BLEU distances between *pairs* of target sequences.\n\nIssues and Questions:\n- The not well-justified relationship (IMO) with Variational Inference, reflected in the title containing the term \"Variational\", and the introduction of the term \"Variational Transformer\" to characterise the approach. Section 2.1 on Bayesian Inference and the variational ELBO technique appears to be very loosely connected to the actual technique used in the paper (or at the very least the connection is not made clear). Two issues here: (1) no details at all are provided concerning the training of Transformer with the MC Dropout technique; (2) it is also not clear how fundamental for the approach it actually is to train according to this technique (presumably according to some variational principle ?), as opposed to randomly perturbing the weights of the model at test time in any simpler way --- which would also produce a sample of outputs on which the proposed BLEUVar technique could be applied.\n- Related to this point, BLEUVar could also be applied to simpler techniques for producing the target samples than sampling the parameter space, namely simply sampling the output space (hierarchical sampling over the NMT outputs, for the standard fixed parameter value). This would appear to be a rather obvious and relevant baseline. In other words: what is the relative importance in practice of a Bayesian approach to producing different outputs   as compared to the effect of using BLEUVar to measure the variance of candidate outputs ?\n- In equation (6) you seem to assume (in the KL term) that $p(\\omega) = p(\\omega|X)$ (am I correct?), in other words independence of the parameters and $X$. If this is the case, it looks counter-intuitive, because detecting that $x$ is out-of-domain would seem to be very correlated to the structure of $X$, and not only on the structure of $Y$ given $X$ ?\n- The experiments do show some superiority of your technique BLEUVar over the non-bayesian BS and the bayesian SP uncertainty measure, in terms of their \"correlation\" with BLEU, but BLEUVar use BLEU internally, while the other two measures don't, so could this explain its superiority? \n- Overall, I did not find the experiments strong. In particular the \"different languages\" experiments of section 4.1.2 look extremely artificial. You write that German and Dutch have a large overlapping vocabulary and that Dutch sentences \"look plausible [German sentences] to a non-native speaker\", which is quite an exaggeration. I do understand that your point is of course not to claim that your technique can be used as a practical language guesser, but to show that your technique is able to compute high uncertainty on Dutch sentences (for an NMT system trained on German). However, this experiment can only be seen as a toy experiment (the translations from Dutch can be immediately detected as being ridiculous, on very simple criteria), and more realistic experiments would be needed to convince the reader of the practical applicability of the technique.\n- The last section (section 5) states that \"With the new tools above we can now develop NMT systems that can be deployed in scenarios where high trust is required of the system, for example in legal applications\". Based on my understanding of the paper, this statement appears to be unwarranted at this stage.\n\n------ After rebuttal: \nI am lowering my score for the paper. I am not convinced by the responses to several of my questions, in particular to what I felt was an exaggerated insistence on the paper being about  “Variational Transformers”, the rather artificial connection to ELBO (noted by several reviewers), and the lack of self-contained description in the paper of the actual technique used, MC-Dropout, which might be explained in simple and sufficient terms on its own. \nAlso, I am disappointed that the authors did not update in any way their submission to reflect the reviewers’ comments (contrarily to misleading expressions in the rebuttals). It is therefore impossible to know whether such unconvincing claims as that made in the conclusion “With the new tools above …” would be maintained in the final version.", "extra_scores": {}, "extra_text": {}}, {"review_id": "QG-0fWCwucn", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed a Baysian method for detecting out of distribution (OOD) in machine translation. To this end, the paper introduces BLEU variance (BLEUVar) that is computed based on a number of samples from Transformer with MC Dropout. The advantage of BLEUVar is that it doesn’t require reference, instead it’s computed based on pairwise comparison of the decoded sentences.\nThe proposed BLEUVar requires decoding $n=10$ translations of an input sentence. This is not desirable for MT especially each target sentence is decoded conditioning on a particular dropout mask, so it’s not parallelable. I wonder what would be the advantage of this approach compared to a simpler approach of detecting OOD just from input sentences without running the whole translation pipeline.\n\nFor OOD experiments, I find that the experiment in section 4.4.1 is a bit unrealistic. For MT, there are many datasets for different domains that can be used as testset (e.g., [biomedical](http://www.statmt.org/wmt20/biomedical-translation-task.html), [chat](http://www.statmt.org/wmt20/chat-task.html), ...). Moreover, the training data in 4.4.1 is too small compared to typical MT training data.\n\nSimilarly, I also felt that the experiment on different language domains in 4.4.2 is unrealistic. While the authors applied their method for an extreme case for demonstration. I think there are many extreme domains in translation which are more realistic. It would be nice to see the proposed method applied for those domains (i.e., biomedical, law, IT,...)\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "0WOioaJyakd", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper adapts MC-dropout to neural machine translation in order to measure prediction uncertainty to detect OOD samples.\n\nAs far as I can tell, the main novelty of this work compared to previous research is the use of BLEU score variance to estimate uncertainty (as opposed to log probability). While the proposed method shows good results when used on out-of-domain data (or when the input data comes from a different language) the paper lacks comparison with previous work. Moreover, the paper's organization relies too much on the appendix. In particular, important discussion of related research is not included in the main text.\n\nPros:\n- The paper is clearly written and well presented\n- Good results on out-of-domain data and mixed language data\n\n\nCons:\n - Insufficient comparison with previous work. In particular Fomicheva et al. 2020 and Wang et al. 2019 propose similar methods based on the variance of the token level log probabilities (instead of BLEU here). Without these results, it is hard to tell whether the proposed use of BLEU variance is responsible for improvements over the simple \"SP\" baseline.\n - BLEUVar seems to help more when the data is from a different domain. However, it would be more interesting to see how it fares on a mixture of in-domain and out-of-domain data (after all, this is a more realistic OOD detection setting). This experiment was performed for different languages but I think it would be more important to see it on different domains of the same language (since language identification is easier than domain identification).\n - Too much material is in the appendix: related work, important results on in-domain OOD detection. Results in the appendix are heavily referenced in the main text, which makes it feel like the paper is not \"self-contained\". The reader has to jump back and forth between main text and appendix to get the whole story. \n\n\nRemarks:\n- \"Wat zei je?\": Having parts of the title in a different language is fine, especially for an MT paper, but consider adding an English translation in the introduction or as a footnote, as most of the audience does not read Dutch.\n- In light of previous work, I suggest toning down grand claims such as \"Our new measure of uncertainty solves a major intractability in the naive application of existing approaches on long sentences\" in the abstract\n- I don't understand the point of mentioning the ELBO. It takes almost half a page but is not used anywhere else in the paper. This valuable space could be used to move more relevant content from the appendix to the main text (such as related work or the experiments in appendix B)\n- In section 3, it might be worth mentioning that BLEU (or rather 1-BLEU) is not a proper distance metric. For instance, it is not symmetric, which has important implications (for instance in eq. 12 it is crucial to include both directions)\n- Compare the mixed language result with LID? Seems like a relevant baseline for this specific scenario (and more practical than running a neural model 10 times).\n- Most figures would be more readable with bars (binned by 10% increments, and with error bars for each bin) than the currebt line plots\n- In table 3: I suggest reporting the square root of BLEUVar instead so the scores have the same \"unit\" as BLEU score (ie. \"BLEUStd\")\n- Typo in 4.2: \"Use\" -> \"Using\"\n- The repeated references to the appendix made Section 4.3 in particular very hard to read. I suggest the authors focus on discussing mainly the results they can include in the main text, and only refer to the appendix shortly at the end (eg. \"see appendix [..] for examples of other cases\")\n\n---\nPost rebuttal: The authors have partially addressed my concerns with regards to experiments on actual domains. I think this is a central part of the paper and these experiments could be improved, however I am willing to augment my score to 5. I am still ambivalent about the paper but I wouldn't fight against it being accepted.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper

* adheres to the Bayesian interpretation of MC dropout and applies it to Transformer-based NMT, thus approximately sampling from the NMT model's posterior predictive distribution $Y_*|x_*, \mathcal D$
* as the NMT predictive distribution is over a discrete sample space, the authors compute variance of pairwise comparisons between the translation and other candidate outputs in a beam of likely translations (the authors call this BLEUVar).

Whereas the work is potentially interesting it does not seem ripe for publication. Here are some of the issues I'd like to highlight:

1. OOD detection. Detection in input space seems like a natural baseline. The authors argue that OOD detection in output space takes the downstream task into consideration, but going through the conditional also makes the task considerably more difficult and computationally challenging. Though we appreciate the author's point, we don't see it as a good enough reason to discard OOD detection in input space as a serious alternative. 

2. Why BDL? The motivation for Bayesian methods is clear, but BDL can at best *approximate* Bayesian reasoning, thus the question does deserve an answer. The reviewers asked for experiments that demonstrate empirically the relevance of the Bayesian formulation, for example, one reviewer suggested to compute BLEUVar in the frequentist case, and that makes perfect sense. Consider this: $q(\theta)$ likely under-estimates posterior uncertainty, so let's say that $\operatorname{Var}(\theta|\mathcal D)$ is rather small, then BLEUVar as presented is in fact not capturing posterior predictive uncertainty (due to entropy of $Y_*|x_*, \mathcal D$), but rather sampling uncertainty (due to entropy of $Y_*|x_*, \theta$). 

3. Unrealistic experiments: we all agreed that the experiments are weak. For example, we do not share the authors' excitement for the results around a foreign language as an example of OOD data point, we see it as an artificially simple case. We also expected more interesting cases of mixed domain data sets (for ideas, check tasks within WMT and IWSLT, as well as resources such as Opus and low-resource language pairs as those in FLORES) and more generally different levels of noise (e.g., synthetic data produced by other translation engines, round-trip/back-translations are very typical in low-resource settings). 

Additional remarks/suggestions:
* in my personal view, BLEUVar should *not* be based on biased statistics (beam search introduces all sorts of unknown biases); the pairwise comparison mechanism behind BLEUVar is similar to what MT researchers call minimum Bayes risk decoding (a frequentist criterion for making decisions under uncertainty).
* we do believe the setting explored in this paper *is* related to confidence estimation, and even though I agree with the authors that a direct comparison is not per se needed, CE datasets could still prove useful for evaluation;

Though the paper has been appreciated for it dispenses with quality annotation, for it attempts to quantify estimation uncertainty (or epistemic, if the authors prefer) rather than sampling uncertainty (or aleatoric), and for other technical contributions (such as BLEUVar), we think this paper needs more than subtle/careful positioning, it really needs to acknowledge the relevance of certain alternatives and evaluate against them (BDL need not win every comparison, that's not so much the issue, the issue is that the current picture is too incomplete). 

A final (personal) remark. I noticed the exchange regarding the suitability of the paper to an ML (vs NLP) venue. I personally do not think your submission is more or less appropriate to one or the other on the grounds of its technical content. The expert reviews attached suggest enough ideas for improvements, and I would imagine an improved version of the paper having a good chance at any major ML (or NLP) venue.

Decision: Reject