Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper describes a method for estimating a neural machine translation (NMT) system's uncertainty about its translation of a sentence that has two parts: (1) use MC Dropout as a proxy for integrating out parameters; (2) two uncertainty metrics (probability of translation summing over randomly-sampled parameters and variance in BLEU using randomly-sampled parameters). The baseline method is just to use the probability of the 1-best translation under the MLE parameters. The method is evaluated by measuring the BLEU score of a test set retaining only the most-certain fraction of the sentences.

For in-domain sentences, BLEUVar does the best, and SP is the same as the baseline. For out-of-domain sentences (train on news-commentary, test on Europarl), the baseline does dramatically worse, and both BLEUVar and SP are better. The authors also tried treating Dutch sentences as "out-of-domain" and got similar results (BLEUVar > SP > baseline).

I don’t think I understand the authors’ differentiation of the present task from confidence estimation (e.g., Blatz et al., 2004, http://www.alexkulesza.com/pubs/confest_report04.pdf; Ueffing and Ney, 2007, https://www.aclweb.org/anthology/J07-1003.pdf). The authors write that “by definition QE crucially relies on examples of mistranslations to train the surrogate,” but the features these systems use do not necessarily rely on mistranslations; only the classifier does. The present work avoids building a classifier by evaluating the measure directly using performance-retention curves. That’s fine, of course, but I’m not seeing that the distinction between the present work and previous work on confidence estimation is that sharp. Some of the features from older work could have been used as additional baselines here (e.g., some of the ones on pages 41-42 and 45-46 of the Blatz et al. report cited above). 

The paper uses a lot of space (about 1.5 pages) presenting fairly technical (to me) background on Bayesian inference, which turns out to not be needed for understanding the rest of the paper. I would suggest cutting this section down. On the other hand, the paper spends very little space explaining MC Dropout. To make the paper self-contained, I think it would be helpful to at least give a brief review of what MC Dropout is and how to do it.

Sections 4.2 reports some experiments/analysis on the relationship between uncertainty and sentence length. I’m uncertain what the research question here is, and what the reader should learn from this section. Section 4.3 shows one example sentence; again, I’m not sure what question this example is meant to answer.

Would it be valuable to study the relationship between certainty (as measured by your measures) and quality? What would a scatterplot of BLEU vs. BLEUVar or SP look like?

In any case, I think it could be interesting to show more than a few examples, maybe 10+ examples, to give a sense for what makes the model more or less certain.

Overall, I like this method and think that it has value for the MT community. I’m not totally sure that ICLR is the best place for this work to be published; the WMT conference would have been a better fit.

Other comments:

- Why is the baseline method called "Beam Score" when all three methods use beam search? Would it make sense to rename "Beam Score" to "MLE Probability" and "Sequence Probability" to "MAP Probability"?

review_title: A simple, interesting method for estimating uncertainty in neural machine translation

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposed a Baysian method for detecting out of distribution (OOD) in machine translation. To this end, the paper introduces BLEU variance (BLEUVar) that is computed based on a number of samples from Transformer with MC Dropout. The advantage of BLEUVar is that it doesn’t require reference, instead it’s computed based on pairwise comparison of the decoded sentences.
The proposed BLEUVar requires decoding $n=10$ translations of an input sentence. This is not desirable for MT especially each target sentence is decoded conditioning on a particular dropout mask, so it’s not parallelable. I wonder what would be the advantage of this approach compared to a simpler approach of detecting OOD just from input sentences without running the whole translation pipeline.

For OOD experiments, I find that the experiment in section 4.4.1 is a bit unrealistic. For MT, there are many datasets for different domains that can be used as testset (e.g., [biomedical](http://www.statmt.org/wmt20/biomedical-translation-task.html), [chat](http://www.statmt.org/wmt20/chat-task.html), ...). Moreover, the training data in 4.4.1 is too small compared to typical MT training data.

Similarly, I also felt that the experiment on different language domains in 4.4.2 is unrealistic. While the authors applied their method for an extreme case for demonstration. I think there are many extreme domains in translation which are more realistic. It would be nice to see the proposed method applied for those domains (i.e., biomedical, law, IT,...)

review_title: The experiments are unrealistic 

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper adapts MC-dropout to neural machine translation in order to measure prediction uncertainty to detect OOD samples.

As far as I can tell, the main novelty of this work compared to previous research is the use of BLEU score variance to estimate uncertainty (as opposed to log probability). While the proposed method shows good results when used on out-of-domain data (or when the input data comes from a different language) the paper lacks comparison with previous work. Moreover, the paper's organization relies too much on the appendix. In particular, important discussion of related research is not included in the main text.

Pros:
- The paper is clearly written and well presented
- Good results on out-of-domain data and mixed language data


Cons:
 - Insufficient comparison with previous work. In particular Fomicheva et al. 2020 and Wang et al. 2019 propose similar methods based on the variance of the token level log probabilities (instead of BLEU here). Without these results, it is hard to tell whether the proposed use of BLEU variance is responsible for improvements over the simple "SP" baseline.
 - BLEUVar seems to help more when the data is from a different domain. However, it would be more interesting to see how it fares on a mixture of in-domain and out-of-domain data (after all, this is a more realistic OOD detection setting). This experiment was performed for different languages but I think it would be more important to see it on different domains of the same language (since language identification is easier than domain identification).
 - Too much material is in the appendix: related work, important results on in-domain OOD detection. Results in the appendix are heavily referenced in the main text, which makes it feel like the paper is not "self-contained". The reader has to jump back and forth between main text and appendix to get the whole story. 


Remarks:
- "Wat zei je?": Having parts of the title in a different language is fine, especially for an MT paper, but consider adding an English translation in the introduction or as a footnote, as most of the audience does not read Dutch.
- In light of previous work, I suggest toning down grand claims such as "Our new measure of uncertainty solves a major intractability in the naive application of existing approaches on long sentences" in the abstract
- I don't understand the point of mentioning the ELBO. It takes almost half a page but is not used anywhere else in the paper. This valuable space could be used to move more relevant content from the appendix to the main text (such as related work or the experiments in appendix B)
- In section 3, it might be worth mentioning that BLEU (or rather 1-BLEU) is not a proper distance metric. For instance, it is not symmetric, which has important implications (for instance in eq. 12 it is crucial to include both directions)
- Compare the mixed language result with LID? Seems like a relevant baseline for this specific scenario (and more practical than running a neural model 10 times).
- Most figures would be more readable with bars (binned by 10% increments, and with error bars for each bin) than the currebt line plots
- In table 3: I suggest reporting the square root of BLEUVar instead so the scores have the same "unit" as BLEU score (ie. "BLEUStd")
- Typo in 4.2: "Use" -> "Using"
- The repeated references to the appendix made Section 4.3 in particular very hard to read. I suggest the authors focus on discussing mainly the results they can include in the main text, and only refer to the appendix shortly at the end (eg. "see appendix [..] for examples of other cases")

---
Post rebuttal: The authors have partially addressed my concerns with regards to experiments on actual domains. I think this is a central part of the paper and these experiments could be improved, however I am willing to augment my score to 5. I am still ambivalent about the paper but I wouldn't fight against it being accepted.
review_title: Interesting approach but insufficient comparison with previous work

final_score: rating: 3: Clear rejection
review_content: Summary: 

The paper proposes a technique for assessing the uncertainty of a Transformer-based NMT model on a given input $x$. The technique relies on computing a variance-like estimate over a collection of translation candidates for $x$, where these candidates are obtained by perturbing the decoding mechanism through the use of dropout at test time. Experiments compare this technique with other ways of measuring the "epistemic uncertainty" of the NMT model. In limited training data conditions, the proposed measure is better aligned with the actual performance of the model than competing measures, and in particular is better able to detect Out-of-Domain translation requests.

Pros:- The paper addresses an important problem: how to detect translation requests for which a given NMT model is unlikely to produce good results, so that these requests can be handled in a proper way (e.g. referred to a human translator).
- Doing that in a way that only relies on intrinsic properties of the NMT model without requiring a specific quality-estimation training setup, which additionally might have difficulty covering all of the possible out-of-domain cases.
- Proposing to assess uncertainty by an original perturbation mechanism, using dropout at test time, which mimics a form of sampling over the parameter space of the model.
- The clever way in which, in equation (9), a proxy for variance over symbolic target sequences (which is not well-defined) is obtained in terms of the BLEU distances between *pairs* of target sequences.

Issues and Questions:
- The not well-justified relationship (IMO) with Variational Inference, reflected in the title containing the term "Variational", and the introduction of the term "Variational Transformer" to characterise the approach. Section 2.1 on Bayesian Inference and the variational ELBO technique appears to be very loosely connected to the actual technique used in the paper (or at the very least the connection is not made clear). Two issues here: (1) no details at all are provided concerning the training of Transformer with the MC Dropout technique; (2) it is also not clear how fundamental for the approach it actually is to train according to this technique (presumably according to some variational principle ?), as opposed to randomly perturbing the weights of the model at test time in any simpler way --- which would also produce a sample of outputs on which the proposed BLEUVar technique could be applied.
- Related to this point, BLEUVar could also be applied to simpler techniques for producing the target samples than sampling the parameter space, namely simply sampling the output space (hierarchical sampling over the NMT outputs, for the standard fixed parameter value). This would appear to be a rather obvious and relevant baseline. In other words: what is the relative importance in practice of a Bayesian approach to producing different outputs   as compared to the effect of using BLEUVar to measure the variance of candidate outputs ?
- In equation (6) you seem to assume (in the KL term) that $p(\omega) = p(\omega|X)$ (am I correct?), in other words independence of the parameters and $X$. If this is the case, it looks counter-intuitive, because detecting that $x$ is out-of-domain would seem to be very correlated to the structure of $X$, and not only on the structure of $Y$ given $X$ ?
- The experiments do show some superiority of your technique BLEUVar over the non-bayesian BS and the bayesian SP uncertainty measure, in terms of their "correlation" with BLEU, but BLEUVar use BLEU internally, while the other two measures don't, so could this explain its superiority? 
- Overall, I did not find the experiments strong. In particular the "different languages" experiments of section 4.1.2 look extremely artificial. You write that German and Dutch have a large overlapping vocabulary and that Dutch sentences "look plausible [German sentences] to a non-native speaker", which is quite an exaggeration. I do understand that your point is of course not to claim that your technique can be used as a practical language guesser, but to show that your technique is able to compute high uncertainty on Dutch sentences (for an NMT system trained on German). However, this experiment can only be seen as a toy experiment (the translations from Dutch can be immediately detected as being ridiculous, on very simple criteria), and more realistic experiments would be needed to convince the reader of the practical applicability of the technique.
- The last section (section 5) states that "With the new tools above we can now develop NMT systems that can be deployed in scenarios where high trust is required of the system, for example in legal applications". Based on my understanding of the paper, this statement appears to be unwarranted at this stage.

------ After rebuttal: 
I am lowering my score for the paper. I am not convinced by the responses to several of my questions, in particular to what I felt was an exaggerated insistence on the paper being about  “Variational Transformers”, the rather artificial connection to ELBO (noted by several reviewers), and the lack of self-contained description in the paper of the actual technique used, MC-Dropout, which might be explained in simple and sufficient terms on its own. 
Also, I am disappointed that the authors did not update in any way their submission to reflect the reviewers’ comments (contrarily to misleading expressions in the rebuttals). It is therefore impossible to know whether such unconvincing claims as that made in the conclusion “With the new tools above …” would be maintained in the final version.
review_title: Review for Wat zei je? Detecting Out-of-Distribution Translations with Variational Transformers 

Meta-review: This paper

* adheres to the Bayesian interpretation of MC dropout and applies it to Transformer-based NMT, thus approximately sampling from the NMT model's posterior predictive distribution $Y_*|x_*, \mathcal D$
* as the NMT predictive distribution is over a discrete sample space, the authors compute variance of pairwise comparisons between the translation and other candidate outputs in a beam of likely translations (the authors call this BLEUVar).

Whereas the work is potentially interesting it does not seem ripe for publication. Here are some of the issues I'd like to highlight:

1. OOD detection. Detection in input space seems like a natural baseline. The authors argue that OOD detection in output space takes the downstream task into consideration, but going through the conditional also makes the task considerably more difficult and computationally challenging. Though we appreciate the author's point, we don't see it as a good enough reason to discard OOD detection in input space as a serious alternative. 

2. Why BDL? The motivation for Bayesian methods is clear, but BDL can at best *approximate* Bayesian reasoning, thus the question does deserve an answer. The reviewers asked for experiments that demonstrate empirically the relevance of the Bayesian formulation, for example, one reviewer suggested to compute BLEUVar in the frequentist case, and that makes perfect sense. Consider this: $q(\theta)$ likely under-estimates posterior uncertainty, so let's say that $\operatorname{Var}(\theta|\mathcal D)$ is rather small, then BLEUVar as presented is in fact not capturing posterior predictive uncertainty (due to entropy of $Y_*|x_*, \mathcal D$), but rather sampling uncertainty (due to entropy of $Y_*|x_*, \theta$). 

3. Unrealistic experiments: we all agreed that the experiments are weak. For example, we do not share the authors' excitement for the results around a foreign language as an example of OOD data point, we see it as an artificially simple case. We also expected more interesting cases of mixed domain data sets (for ideas, check tasks within WMT and IWSLT, as well as resources such as Opus and low-resource language pairs as those in FLORES) and more generally different levels of noise (e.g., synthetic data produced by other translation engines, round-trip/back-translations are very typical in low-resource settings). 

Additional remarks/suggestions:
* in my personal view, BLEUVar should *not* be based on biased statistics (beam search introduces all sorts of unknown biases); the pairwise comparison mechanism behind BLEUVar is similar to what MT researchers call minimum Bayes risk decoding (a frequentist criterion for making decisions under uncertainty).
* we do believe the setting explored in this paper *is* related to confidence estimation, and even though I agree with the authors that a direct comparison is not per se needed, CE datasets could still prove useful for evaluation;

Though the paper has been appreciated for it dispenses with quality annotation, for it attempts to quantify estimation uncertainty (or epistemic, if the authors prefer) rather than sampling uncertainty (or aleatoric), and for other technical contributions (such as BLEUVar), we think this paper needs more than subtle/careful positioning, it really needs to acknowledge the relevance of certain alternatives and evaluate against them (BDL need not win every comparison, that's not so much the issue, the issue is that the current picture is too incomplete). 

A final (personal) remark. I noticed the exchange regarding the suitability of the paper to an ML (vs NLP) venue. I personally do not think your submission is more or less appropriate to one or the other on the grounds of its technical content. The expert reviews attached suggest enough ideas for improvements, and I would imagine an improved version of the paper having a good chance at any major ML (or NLP) venue.

Decision: Reject