On NMT Search Errors and Model Errors: Cat Got Your Tongue?

Abstract: We report on search errors and model errors in neural machine translation (NMT). We present an exact inference procedure for neural sequence models based on a combination of beam search and depth-first search. We use our exact search to find the global best model scores under a Transformer base model for the entire WMT15 English-German test set. Surprisingly, beam search fails to find these global best model scores in most cases, even with a very large beam size of 100. For more than 50% of the sentences, the model in fact assigns its global best score to the empty translation, revealing a massive failure of neural models in properly accounting for adequacy. We show by constraining search with a minimum translation length that at the root of the problem of empty translations lies an inherent bias towards shorter translations. We conclude that vanilla NMT in its current form requires just the right amount of beam search errors, which, from a modelling perspective, is a highly unsatisfactory conclusion indeed, as the model often prefers an empty translation.

[0]Now at Google.
Neural machine translation BIBREF0 , BIBREF1 , BIBREF2 assigns the probability INLINEFORM0 of a translation INLINEFORM1 of length INLINEFORM2 over the target language vocabulary INLINEFORM3 for a source sentence INLINEFORM4 of length INLINEFORM5 over the source language vocabulary INLINEFORM6 via a left-to-right factorization using the chain rule: DISPLAYFORM0 
The task of finding the most likely translation INLINEFORM0 for a given source sentence INLINEFORM1 is known as the decoding or inference problem: DISPLAYFORM0 
The NMT search space is vast as it grows exponentially with the sequence length. For example, for a common vocabulary size of INLINEFORM0 , there are already more possible translations with 20 words or less than atoms in the observable universe ( INLINEFORM1 ). Thus, complete enumeration of the search space is impossible. The size of the NMT search space is perhaps the main reason why – besides some preliminary studies BIBREF3 , BIBREF4 , BIBREF5 – analyzing search errors in NMT has received only limited attention. To the best of our knowledge, none of the previous studies were able to quantify the number of search errors in unconstrained NMT due to the lack of an exact inference scheme that – although too slow for practical MT – guarantees to find the global best model score for analysis purposes.
[t!] BeamSearch INLINEFORM0 [1] INLINEFORM1 : Source sentence, INLINEFORM2 : Beam size INLINEFORM3 Initialize with empty translation prefix and zero score INLINEFORM4 INLINEFORM5 INLINEFORM6 INLINEFORM7 Hypotheses ending with INLINEFORM8 are not expanded INLINEFORM9 Add all possible continuations INLINEFORM10 Select INLINEFORM11 -best INLINEFORM12 INLINEFORM13 INLINEFORM14 
[t!] DFS INLINEFORM0 [1] INLINEFORM1 : Source sentence
 INLINEFORM0 : Translation prefix (default: INLINEFORM1 )
 INLINEFORM0 : INLINEFORM1 (default: INLINEFORM2 )
 INLINEFORM0 : Lower bound INLINEFORM1 INLINEFORM2 Trigger INLINEFORM3 update INLINEFORM4 Initialize INLINEFORM5 with dummy value INLINEFORM6 INLINEFORM7 INLINEFORM8 INLINEFORM9 INLINEFORM10 INLINEFORM11 INLINEFORM12 
In this work we propose such an exact decoding algorithm for NMT that exploits the monotonicity of NMT scores: Since the conditional log-probabilities in Eq. EQREF1 are always negative, partial hypotheses can be safely discarded once their score drops below the log-probability of any complete hypothesis. Using our exact inference scheme we show that beam search does not find the global best model score for more than half of the sentences. However, these search errors, paradoxically, often prevent the decoder from suffering from a frequent but very serious model error in NMT, namely that the empty hypothesis often gets the global best model score. Our findings suggest a reassessment of the amount of model and search errors in NMT, and we hope that they will spark new efforts in improving NMT modeling capabilities, especially in terms of adequacy.
Decoding in NMT (Eq. EQREF2 ) is usually tackled with beam search, which is a time-synchronous approximate search algorithm that builds up hypotheses from left to right. A formal algorithm description is given in Alg. SECREF1 . Beam search maintains a set of active hypotheses INLINEFORM0 . In each iteration, all hypotheses in INLINEFORM1 that do not end with the end-of-sentence symbol INLINEFORM2 are expanded and collected in INLINEFORM3 . The best INLINEFORM4 items in INLINEFORM5 constitute the set of active hypotheses INLINEFORM6 in the next iteration (line 11 in Alg. SECREF1 ), where INLINEFORM7 is the beam size. The algorithm terminates when the best hypothesis in INLINEFORM8 ends with the end-of-sentence symbol INLINEFORM9 . Hypotheses are called complete if they end with INLINEFORM10 and partial if they do not.
Beam search is the ubiquitous decoding algorithm for NMT, but it is prone to search errors as the number of active hypotheses is limited by INLINEFORM0 . In particular, beam search never compares partial hypotheses of different lengths with each other. As we will see in later sections, this is one of the main sources of search errors. However, in many cases, the model score found by beam search is a reasonable approximation to the global best model score. Let INLINEFORM1 be the model score found by beam search ( INLINEFORM2 in line 12, Alg. SECREF1 ), which is a lower bound on the global best model score: INLINEFORM3 . Furthermore, since the conditionals INLINEFORM4 in Eq. EQREF1 are log-probabilities and thus non-positive, expanding a partial hypothesis is guaranteed to result in a lower model score, i.e.: DISPLAYFORM0 
Consequently, when we are interested in the global best hypothesis INLINEFORM0 , we only need to consider partial hypotheses with scores greater than INLINEFORM1 . In our exact decoding scheme we traverse the NMT search space in a depth-first order, but cut off branches along which the accumulated model score falls below INLINEFORM2 . During depth-first search (DFS), we update INLINEFORM3 when we find a better complete hypothesis. Alg. SECREF1 specifies the DFS algorithm formally. An important detail is that elements in INLINEFORM4 are ordered such that the loop in line 5 considers the INLINEFORM5 token first. This often updates INLINEFORM6 early on and leads to better pruning in subsequent recursive calls.
We conduct all our experiments in this section on the entire English-German WMT news-test2015 test set (2,169 sentences) with a Transformer base BIBREF13 model trained with Tensor2Tensor BIBREF14 on parallel WMT18 data excluding ParaCrawl. Our pre-processing is as described by BIBREF15 and includes joint subword segmentation using byte pair encoding BIBREF16 with 32K merges. We report cased BLEU scores. An open-source implementation of our exact inference scheme is available in the SGNMT decoder BIBREF17 , BIBREF4 .
Our main result is shown in Tab. TABREF9 . Greedy and beam search both achieve reasonable BLEU scores but rely on a high number of search errors to not be affected by a serious NMT model error: For 51.8% of the sentences, NMT assigns the global best model score to the empty translation, i.e. a single INLINEFORM0 token. Fig. FIGREF10 visualizes the relationship between BLEU and the number of search errors. Large beam sizes reduce the number of search errors, but the BLEU score drops because translations are too short. Even a large beam size of 100 produces 53.62% search errors. Fig. FIGREF11 shows that beam search effectively reduces search errors with respect to greedy decoding to some degree, but is ineffective in reducing search errors even further. For example, Beam-10 yields 15.9% fewer search errors (absolute) than greedy decoding (57.68% vs. 73.58%), but Beam-100 improves search only slightly (53.62% search errors) despite being 10 times slower than beam-10.
The problem of empty translations is also visible in the histogram over length ratios (Fig. FIGREF13 ). Beam search – although still slightly too short – roughly follows the reference distribution, but exact search has an isolated peak in INLINEFORM0 from the empty translations.
Tab. TABREF14 demonstrates that the problems of search errors and empty translations are not specific to the Transformer base model and also occur with other architectures. Even a highly optimized Transformer Big model from our WMT18 shared task submission BIBREF15 has 25.8% empty translations.
Fig. FIGREF15 shows that long source sentences are more affected by both beam search errors and the problem of empty translations. The global best translation is empty for almost all sentences longer than 40 tokens (green curve). Even without sentences where the model prefers the empty translation, a large amount of search errors remain (blue curve).
To find out more about the length deficiency we constrained exact search to certain translation lengths. Constraining search that way increases the run time as the INLINEFORM0 -bounds are lower. Therefore, all results in this section are conducted on only a subset of the test set to keep the runtime under control. We first constrained search to translations longer than 0.25 times the source sentence length and thus excluded the empty translation from the search space. Although this mitigates the problem slightly (Fig. FIGREF16 ), it still results in a peak in the INLINEFORM1 cluster. This suggests that the problem of empty translations is the consequence of an inherent model bias towards shorter hypotheses and cannot be fixed with a length constraint.
We then constrained exact search to either the length of the best Beam-10 hypothesis or the reference length. Tab. TABREF18 shows that exact search constrained to the Beam-10 hypothesis length does not improve over beam search, suggesting that any search errors between beam search score and global best score for that length are insignificant enough so as not to affect the BLEU score. The oracle experiment in which we constrained exact search to the correct reference length (last row in Tab. TABREF18 ) improved the BLEU score by 0.9 points.
A popular method to counter the length bias in NMT is length normalization BIBREF6 , BIBREF7 which simply divides the sentence score by the sentence length. We can find the global best translations under length normalization by generalizing our exact inference scheme to length dependent lower bounds INLINEFORM0 . The generalized scheme finds the best model scores for each translation length INLINEFORM1 in a certain range (e.g. zero to 1.2 times the source sentence length). The initial lower bounds are derived from the Beam-10 hypothesis INLINEFORM2 as follows: DISPLAYFORM0 
Exact search under length normalization does not suffer from the length deficiency anymore (last row in Tab. TABREF19 ), but it is not able to match our best BLEU score under Beam-10 search. This suggests that while length normalization biases search towards translations of roughly the correct length, it does not fix the fundamental modelling problem.
Other researchers have also noted that large beam sizes yield shorter translations BIBREF19 . BIBREF20 argue that this model error is due to the locally normalized maximum likelihood training objective in NMT that underestimates the margin between the correct translation and shorter ones if trained with regularization and finite data. A similar argument was made by BIBREF10 who pointed out the difficulty for a locally normalized model to estimate the “budget” for all remaining (longer) translations. BIBREF21 demonstrated that NMT models are often poorly calibrated, and that that can cause the length deficiency. BIBREF5 argued that uncertainty caused by noisy training data may play a role. BIBREF22 showed that the consistent best string problem for RNNs is decidable. We provide an alternative DFS algorithm that relies on the monotonic nature of model scores rather than consistency, and that often converges in practice.
To the best of our knowledge, this is the first work that reports the exact number of search errors in NMT as prior work often relied on approximations, e.g. via INLINEFORM0 -best lists BIBREF3 or constraints BIBREF4 .
We have presented an exact inference scheme for NMT. Exact search may not be practical, but it allowed us to discover deficiencies in widely used NMT models. We linked deteriorating BLEU scores of large beams with the reduction of search errors and showed that the model often prefers the empty translation – an evidence of NMT's failure to properly model adequacy. Our investigations into length constrained exact search suggested that simple heuristics like length normalization are unlikely to remedy the problem satisfactorily.
This work was supported by the U.K. Engineering and Physical Sciences Research Council (EPSRC) grant EP/L027623/1 and has been performed using resources provided by the Cambridge Tier-2 system operated by the University of Cambridge Research Computing Service funded by EPSRC Tier-2 capital grant EP/P020259/1.

Figure and table captions:
Table 1: NMT with exact inference. In the absence of search errors, NMT often prefers the empty translation, causing a dramatic drop in length ratio and BLEU.
Figure 1: BLEU over the percentage of search errors. Large beam sizes yield fewer search errors but the BLEU score suffers from a length ratio below 1.
Figure 2: Even large beam sizes produce a large number of search errors.
Figure 3: Histogram over target/source length ratios.
Figure 5: Histogram over length ratios with minimum translation length constraint of 0.25 times the source sentence length. Experiment conducted on 73.0% of the test set.
Table 2: ∗: The recurrent LSTM, the convolutional SliceNet (Kaiser et al., 2017), and the Transformer-Big systems are strong baselines from a WMT’18 shared task submission (Stahlberg et al., 2018a).
Figure 4: Number of search errors under Beam-10 and empty global bests over the source sentence length.
Table 3: Exact search under length constraints. Experiment conducted on 48.3% of the test set.
Table 4: Length normalization fixes translation lengths, but prevents exact search from matching the BLEU score of Beam-10. Experiment conducted on 48.3% of the test set.

Questions about this paper:
what were the length constraints they set?

Answers:
They set translation length longer than minimum 0.25 times the source sentence length