(Unconstrained) Beam Search is Sensitive to Large Search Discrepancies

1 INTRODUCTION . Neural sequence models are among the most popular tools for modeling sequential data and have been applied to a range of applications including machine translation ( Gehring et al. , 2017 ) , summarization ( Chopra et al. , 2016 ) , image captioning ( Vinyals et al. , 2017 ) , and conversation modeling ( Vinyals & Le , 2015 ) . The most commonly used inference algorithm for decoding neural sequence models is beam search , a search algorithm that generates the sequence tokens one-by-one while keeping a fixed number of active candidates ( beams ) at each step . Recently , several works reported the problem of performance degradation in beam search . In machine translation , Koehn & Knowles ( 2017 ) found that beam search “ only improves translation for narrow beams and deteriorates when exposed to a larger search space ” . They chose this problem as one of six central challenges in machine translation . Ott et al . ( 2018 ) proposed the existence of training pairs in which the target is a copy of the source as an explanation for the performance degradation . For larger beams , more predictions can be classified as “ copies ” 1 and filtering these copies reduces the performance degradation . In image captioning , Vinyals et al . ( 2017 ) observed performance degradation for wider beams and highlighted the use of a narrower beam search as one of the most significant improvements in their model . They hypothesized that the performance degradation is either due to overfitting or that the objective function used in training ( likelihood ) is not aligned with human judgement . Their analysis found that wider beams exhibited more predictions that repeat training captions and fewer novel ones . They claim that this observation supports the hypothesis that the model is overfitted and therefore see the use of smaller beam width as “ another way to regularize ” . In this work , we analyze the performance of beam search across multiple tasks including machine translation , abstractive summarization , and image captioning . We present an explanatory model that 1 “ Copies ” are predictions that share at least 50 % of their unigrams with their source ( Ott et al. , 2018 ) . is based on the concept of search discrepancies ( deviations from greedy choices ) and perform an empirical study of the distribution of such discrepancies . We make the following contributions : 1 . We show that increasing the beam width leads to solutions with more and larger early discrepancies . These sequences often have lower evaluation score , leading to the observed performance degradation . As we increase the beam width , the difference between discrepancies that are associated with improved vs. degraded solutions grows substantially . 2 . We show that our explanatory model generalizes the previously observed copies and predictions that repeat training set targets and accounts for more of the degraded predictions . 3 . Exploiting the above insights , we propose a fix that is based on constraining the discrepancies considered by the beam search . An empirical analysis shows it successfully eliminates the performance degradation . 2 PRELIMINARIES . 2.1 NEURAL SEQUENCE MODELS . Given a model parameterized by θ and an input x , the problem of sequence generation consists of finding a sequence ŷ such that ŷ = argmaxy∈Y Pθ ( y | x ) , where Y is the set of all sequences . y is a sequence of tokens y = { y0 , ... yT−1 } from vocabulary V , where T is the length of the sequence y . The expression Pθ ( y | x ) can then be factored as Pθ ( y | x ) = ∏T−1 t=0 Pθ ( yt | x ; { y0 , ... , yt−1 } ) , or for convenience using log-probability as ∑T−1 t=0 logPθ ( yt | x ; { y0 , ... , yt−1 } ) . It is common to model logPθ ( yt | x ; { y0 , ... , yt−1 } ) using a Recurrent Neural Network ( RNN ) , where the sequence { y0 , ... , yt−1 } conditioned on is expressed by a fixed length hidden state ht . This hidden state is updated using a non-linear function f : ht+1 = f ( ht , yt ) . Exhaustive search to find the globally optimal sequence is not tractable . A greedy algorithm that selects the best candidate at each time step yt = argmaxy∈V logPθ ( y | x ; { y0 , ... , yt−1 } ) makes a sequence of locally optimal decisions , but can lead to a globally sub-optimal sequence . Beam search , in contrast , extends the B most probable partial solutions at each step , where B is called beam width . Following Vijayakumar et al . ( 2018 ) , we denote the set of B solutions held by the beam search at step t − 1 as Y [ t−1 ] = { y1 , [ t−1 ] , ... , yB , [ t−1 ] } . At each step , beam search selects the top scoring B candidates from the set of all possible one token extensions of its beams Yt= { y [ t ] | y [ t−1 ] ∈ Y [ t−1 ] ∧ yt ∈ V } . Formally , the beam search candidates are updated as follows : Y [ t ] = argmax y [ 1 , t ] , ... , y [ B , t ] ∈Yt ∑ b∈ [ 1 .. B ] logPθ ( yb , t | x ) s.t . yi 6= yj ∀i 6= j ; i , j ∈ [ 1 .. B ] ( 1 ) 2.2 SEARCH DISCREPANCIES IN NEURAL SEQUENCE GENERATION . In combinatorial search , a search discrepancy is a decision made by the search algorithm that is not the most highly rated one according to the heuristic ( Harvey & Ginsberg , 1995 ) . In the context of search for neural sequence generation , we define a search discrepancy as extending a partial sequence with a token that is not the most probable one ( i.e. , different than the greedy algorithm ) . More formally , a sequence y is considered to have a search discrepancy at time step t if logPθ ( yt | x ; { y0 , ... , yt−1 } ) < max y∈V logPθ ( y | x ; { y0 , ... , yt−1 } ) . ( 2 ) We denote the ratio between the most probable token and the chosen token as discrepancy gap . We measure the gap based on the difference in log-probability , i.e. , the discrepancy gap at step t is max y∈V log Pθ ( y | x ; { y0 , ... , yt−1 } ) − log Pθ ( yt | x ; { y0 , ... , yt−1 } ) . ( 3 ) 3 EXPERIMENTAL SETUP . We perform an extensive empirical evaluation over multiple tasks , models , datasets , toolkits , and evaluation metrics . Following is a description of the experimental setup for each task . Machine Translation . We use the convolutional model by Gehring et al . ( 2017 ) implemented in the fairseq-py toolkit . We present results for two models , trained on WMT ’ 14 En-Fr and En-De datasets and evaluated on newstest2014 En-Fr and En-De , respectively . Summarization . We use the abstractive summarization model by Chopra et al . ( 2016 ) implemented in OpenNMT toolkit ( Klein et al. , 2017 ) . The model is trained and evaluated using Rush et al. ’ s ( 2015 ) test split of the Gigaword corpus ( Graff et al. , 2003 ) . Image Captioning . We use the model by Vinyals et al . ( 2017 ) , trained on the MSCOCO dataset ( Lin et al. , 2014 ) . The test set includes 5000 images based on Karpathy & Fei-Fei ’ s ( 2015 ) splits . In machine translation and summarization , we apply length normalization on the hypotheses loglikelihood , as it was shown to reduce the performance degradation by not prioritizing short sentences ( Koehn & Knowles , 2017 ; Gehring et al. , 2017 ) . For image captioning , consistent with previous works , we do not use length normalization ( we also found it reduces the overall performance ) . 3.1 EVALUATION METRICS . While beam search finds the ( approximately ) most probable sequence , the quality of a sequence is evaluated based on human references using a task-specific evaluation metric . For machine translation and image captioning we use BLEU-n ( Papineni et al. , 2002 ) , a geometric average of precision over 1- to n-grams multiplied by a brevity penalty for short sentences . As in recent literature , we present results for BLEU-4 . Corpus-level BLEU is reported without smoothing , while for sentencelevel BLEU we use smoothed n-gram counts for n > 1 ( Lin & Och , 2004 ) . For image captioning , we also evaluated the performance using CIDEr ( Vedantam et al. , 2015 ) and SPICE ( Anderson et al. , 2016b ) and report these metrics in Appendix E. For summarization , we use ROUGE ( Lin , 2004 ) , the n-gram recall between candidate summary and a reference . We report the F-score of ROUGE-1 , however similar trends were observed for the F-score of ROUGE-L ( for longest common subsequence ) . 4 EMPIRICAL ANALYSIS OF SEARCH DISCREPANCIES IN BEAM SEARCH . In this section we present an empirical analysis of the search discrepancies . We analyze and compare the most likely hypotheses found by a beam search for the following beam widths : { 1 , 3 , 5 , 25 , 100 , 250 } . Due to space , we present detailed results for one of the tasks and summarize the results for the others . The results for all tasks and metrics can be found in Appendix A . 4.1 BASELINE RESULTS . Table 1 presents the performance of beam search with different beam widths , based on the chosen evaluation metrics . The performance degradation for larger beam widths appears for all tested tasks based on their task-specific evaluation metric . These results are consistent with the existing reports of such performance degradation ( Koehn & Knowles , 2017 ; Ott et al. , 2018 ; Vinyals et al. , 2017 ) . 4.2 THE DISTRIBUTION OF SEARCH DISCREPANCIES . In this section , we analyze the distribution and size of search discrepancies vs. their position ( index ) in sequence . Figure 1 shows the number of discrepancies per position for the most likely hypotheses generated by a beam search on the WMT ’ 14 En-De test set for different beam widths ( all graphs are based on the same number of solutions , however the total number of discrepancies in the generated solutions is not necessarily the same for different beam widths ) . In general , the majority of discrepancies happen in early positions . More interestingly , for larger beams , the number of early discrepancies grows significantly while the number of later discrepancies stays approximately the same . Larger beams seem to allow the search to find solutions with higher overall probability by exploring less probable early tokens , however they do not seem to lead to more probable sequences that share a prefix with solutions found for a smaller beam width . Similar results for the other tasks are reported in Appendix A . For image captioning ( MSCOCO ) , we find the majority of early search discrepancies appear on the second token due to the first token being “ a ” with high probability in almost all sentences ( in greedy search , for example , 99 % of the generated captions start with “ a ” ) . Next , we analyze the discrepancy gap vs. sequence position . Figure 2 presents the mean gap per position for WMT ’ 14 En-De for different beam widths . Again , we can see that the changes are mainly in the early positions : as we increase the beam width , the search tends to find solutions with larger early discrepancy gap , i.e. , the early tokens are relatively less likely . The discrepancy gap of the other tokens remains similar . Similar results for the other tasks are reported in Appendix A . The increase in count and size of early discrepancies for larger beams means that the search manages to find solutions with higher overall probability when starting from a large discrepancy . However , these solution are not necessarily better according to the evaluation metric . The observed performance degradation suggests that the more probable solutions found by larger beams are , in fact , worse . Identifying discrepancies that are likely to lead to a worse solution is therefore a key task in addressing the performance degradation . In the next section , we analyze the differences between discrepancies in solutions with higher evaluation vs. solutions with lower evaluation .

Summary: This paper addresses issues with the beam search decoding algorithm that is commonly applied to recurrent models during inference. In particular, the paper investigates why using larger beam widths, resulting in output sequences with higher log-probabilities, often leads to worse performance on evaluation metrics of interest such as BLEU. The paper argues that this effect is related to ‘search discrepancies’ (deviations from greedy choices early in decoding), and proposes a constrained decoding mechanism as a heuristic fix.