Title: Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models
Authors: Ashwin K Vijayakumar
Michael Cogswell
Ramprasaath R. Selvaraju
Qing Sun
Stefan Lee
David Crandall
Dhruv Batra
Venue: ICLR 2017
Keywords: Deep learning
Computer vision
Natural language processing

Abstract: Neural sequence models are widely used to model time-series data. Equally ubiquitous is the usage of beam search (BS) as an approximate inference algorithm to decode output sequences from these models. BS explores the search space in a greedy left-right fashion retaining only the top B candidates. This tends to result in sequences that differ only slightly from each other. Producing lists of nearly identical sequences is not only computationally wasteful but also typically fails to capture the inherent ambiguity of complex AI tasks. To overcome this problem, we propose Diverse Beam Search (DBS), an alternative to BS that decodes a list of diverse outputs by optimizing a diversity-augmented objective. We observe that our method not only improved diversity but also finds better top 1 solutions by controlling for the exploration and exploitation of the search space. Moreover, these gains are achieved with minimal computational or memory overhead com- pared to beam search. To demonstrate the broad applicability of our method, we present results on image captioning, machine translation, conversation and visual question generation using both standard quantitative metrics and qualitative human studies. We find that our method consistently outperforms BS and previously proposed techniques for diverse decoding from neural sequence models.

TL;DR: We introduce a novel, diversity promoting beam search algorithm that results in significantly improved diversity between decoded sequences as evaluated on multiple sequence generation tasks.

Review discussion:
Is it possible to apply the technique on the applications in Jiwei Li's papers, so you can do a fair comparison with Li?
Thank you for the question.
To be sure we are talking about the same papers, we believe you are referring to:
"Mutual Information and Diverse Decoding Improve Neural Machine Translation” and
"A Diversity-Promoting Objective Function for Neural Conversation Models”
cited as Li & Jurafsky (2016) and Li et al. (2015) respectively in our submission. We discuss these works in the final paragraphs of the Section 4 and as baselines in Section 5.
As we state in our paper, Li et al. (2015) introduce a maximum mutual information (MMI) decoding objective that encourages generated sentences to avoid generic phrasings (e.g. “I don’t know” responses in dialog). This principle is orthogonal and complementary to DBS, since the goal is not to generate multiple diverse outputs (as in DBS) but to not produce generic responses.
Li et al. (2015) evaluate on two dialog generation tasks which we do not replicate.
Li & Jurafsky (2016) build on MMI and introduce (1) a number of techniques to improve re-ranking sequences for machine translation, and (2) a diversity inducing beam search heuristic based on introducing an intra-sibling rank penalty. Since the emphasis of our paper is generating diverse lists (and not re-ranking them), we focus on (2).
Like Li & Jurafsky (2016), we evaluate on the WMT’14 English-German task.
Replicating the experimental setup of Li et al. (2015) is difficult since they did not release code. Reproducing an entire project from scratch is outside the scope of this paper.
Finally, we disagree with a premise in the question — that somehow the comparisons presented in our paper are not “fair”. For both image captioning and machine translation, we compute oracle based metrics to isolate the effect of diverse decoding irrespective of re-ranking strategies and all algorithms produce an equal number of candidate sequences. We perform these evaluations on our reimplementation of these methods as the diverse decoding code has not been made public. As shown in Tables 1 and 2, we consistently outperform the diversity decoding of Li & Jurafsky (2016) and perform comparably to the method of Li et al. (2015) despite the fact it makes uses of an additional trained language model.
NA
Can authors provide more details on how the hyperparameters of Li & Jurafsky 2016 (\lambda, \gamma, \mu...) are tuned? In L&J16, these parameters were tuned to optimize for regular BLEU (and other metrics) which differs from "oracle" BLEU (and other metrics) used in this manuscript. In the case of "oracle" BLEU @ k (k=5,10,20,...), it seems to me that a method that generates more diverse candidates will have a better luck of including a candidate with a higher score. If L&J16 were tuned to attain a higher degree of diversity (to match that of DBS), will it also lead to a higher oracle BLEU/Cider/SPICE etc?
Thank you for the question.
For each experiment reported in the paper, we tune the parameter \gamma in Li and Jurafsky, 2016 (Eq. 14, Section 4.2) exactly how we tune parameters of DBS - by performing a grid search for the best oracle performance on a held-out validation set, as mentioned in our paper. (Section 5.2 for image-captioning and Section 5.3 for machine translation )
Note that since we do not rerank the decoded diverse lists, the parameters \lambda, \gamma and \eta used in eq. 14, Section 4.3 are not applicable in our experiments.
This paper considers the problem of decoding diverge solutions from neural sequence models. It basically adds an additional term to the log-likelihood of standard neural sequence models, and this additional term will encourage the solutions to be diverse.
In addition to solve the inference, this paper uses a modified beam search.
On the plus side, there is not much work on producing diverse solutions in RNN/LSTM models. This paper represents one of the few works on this topic. And this paper is well-written and easy to follow.
The novel of this paper is relatively small. There has been a lot of prior work on producing diverse models in the area of probailistic graphical models. Most of them introduce an additional term in the objective function to encourage diversity. From that perspective, the solution proposed in this paper is not that different from previous work.
Of course, one can argue that most previous work focues on probabilistic graphical models, while this paper focuses on RNN/LSTM models.
But since RNN/LSTM can be simply interpreted as a probabilistic model, I would consider it a small novelty.
The diverse beam search seems to straightforward, i.e. it partitions the beam search space into groups, and does not consider the diversity within group (in order to reduce the search space). To me, this seems to be a simple trick. Note most previous work on diverse solutions in probabilistic graphical models usually involve developing some nontrivial algorithmic solutions, e.g. in order to achieve efficiency.
In comparison, the proposed solution in this paper seems to be simplistic for a paper.
The experimental results how improvement over previous methods (Li & Jurafsky, 2015, 2016).
But it is hard to say how rigorous the comparisons are, since they are based on the authors' own implementation of (Li & Jurasky, 2015, 2016).
update: given that the authors made the code available (I do hope the code will remain publicly available), this has alleviated some of my concerns about the rigor of the experiments. I will raise my rate to 6.
We appreciate that the reviewers effort and are glad they found our paper well-written and clear. We agree that diverse decoding in RNNs is an important but understudied task that this paper works to address.
1. Novelty
We agree there is a large body of work on producing diverse outputs in the context of probabilistic graphical models (we discuss these in Section 4) and RNN models can be viewed as a special class of PGM (specifically a T-order Markov Chain).
However, we believe a crucial problem and contribution has been missed by the reviewer, so please allow us to restate. To the best of our knowledge, all diverse M-Best methods proposed in the classical PGM literature assume access to a tractable black-box 1-best or MAP inference algorithm. Since RNNs are equivalent to a T-th order Markov Chain, exact MAP inference is intractable and thus a direct applications of classical diverse M-Best techniques is infeasible and new techniques must be developed (which is what our paper does).
As we discuss in our related work (Section 4), the closest to this goal is the work of Gimpel et al. 2013, who apply DivMBest (Batra et al. 2012) to RNNs treating beam search as a black box inference technique. Unfortunately, this method is extremely wasteful both in terms of computation and time — it makes M sequential calls to beam search (of arbitrary size B) and retains only the most likely sentence, thus wastefully discarding B-1 outputs at each stage. By integrating diversity within beam search, we overcome these shortcomings by running parallel beam searches with staggered time offsets, obtaining significant time-savings.
As a result, our method decodes diverse lists while being comparable to a single run of classical beam search in both computation and memory requirements. This is a significant contribution and improvement over classical BS (in terms of diversity) and the work of Gimpel et al. (in terms of computation and time).
2. Comparisons to our own reimplementation
We are surprised and disappointed that the reviewer expresses doubt about rigor of our experiments citing our own reimplementation of Li and Jurafsky, 2016 and Li et al., 2015. As we state in the paper (Section 5) and in our previous response to the reviewer below, this is not our choice — the implementation of the diverse decoding scheme used in either papers is not publicly available.
In the interest of transparency, our reimplementation of Li and Jurafsky, 2016 is made available here https://github.com/ashwinkalyan/dbs/tree/lj16 and of Li et al. 2015 is available here: https://github.com/ashwinkalyan/dbs/tree/li15. As stated previously, parameters of all methods are tuned using grid search to optimize for oracle accuracies on a held-out validation set — to ensure a fair comparison.
We are at a loss for what else we could do to ensure a transparent, fair, and rigorous evaluation, and are open to concrete suggestions.
[ Summary ]
This paper presents a new modified beam search algorithm that promotes diverse beam candidates. It is a well known problem —with both RNNs and also non-neural language models— that beam search tends to generate beam candidates that are very similar with each other, which can cause two separate but related problems: (1) search error: beam search may not be able to discover a globally optimal solution as they can easily fall out of the beam early on, (2) simple, common, non-diverse output: the resulting output text tends to be generic and common.
This paper aims to address the second problem (2) by modifying the search objective function itself so that there is a distinct term that scores diversity among the beam candidates.
In other words, the goal of the presented algorithm is not to reduce the search error of the original objective function.
In contrast, stack decoding and future cost estimation, common practices in phrase-based SMT, aim to address the search error problem.
[ Merits ]
I think the Diverse Beam Search (DBS) algorithm proposed by the authors has some merits. It may be useful when we cannot rely on traditional beam search on the original objective function either because the trained model is not strong enough, or because of the search error, or because the objective itself does not align with the goal of the application.
[ Weaknesses ]
It is however not entirely clear how the proposed method compares against more traditional approaches like stack decoding and future cost estimation, on tasks like machine translation, as the authors compare their algorithm mainly against L&J’s diverse LM models and simple beam search.
In fact, modification to the objective function has been applied even in the neural MT context.
For example, see equation (14) in page 12 of the following paper:
"Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation" (https://arxiv.org/pdf/1609.08144v2.pdf)
where the attention coverage term serves a role similar to stack decoding (though unlike stack decoding, the objective term is entirely re-defined, more similarly to DBS proposed in this work), and the length penalty may have an effect that indirectly promotes more informative (thus more likely diverse) responses.
Comparison against these existing algorithms would make the proposed work more complete.
Also, I have a mixed feeling about computing and reporting only *oracle* BLUE, CIDEr, METEOR, etc. Especially given how these oracle scores are very close to each other, and that developing a high performing ranking has not been addressed in this work (and that doing so must be not all that trivial), I’m somewhat skeptical how much of DBS results make a practical difference.
**** [Update after the author responses] ****
The authors addressed some of my concerns by adding a new baseline comparison against Wu et al. 2016.
Thus I will raise my score to 6.
Thank you for the feedback.
Comparison to Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation: As noted by the reviewer, this paper uses two methods for diverse decodings — (1) modified objective that includes a coverage penalty and (2) length normalization. (eq. 14)
The coverage penalty term as proposed in that paper (using attention over the inputs to define ‘coverage’) is intricately tied to the task of NMT and unlike DBS does not generalize to other sequence decoding tasks (e.g. image captioning, visual question generation, dialog, etc).
Unlike NMT, some input information can be ignored in other tasks.
In contrast, the length-normalization term to select the top-B completed beams can be easily implemented for any decoding scenario.
Thus, following your request, we have updated the paper to include comparison to this for both captioning and machine translation. Following the experimental setup in the paper, for fairness to all techniques, we tune the strength \alpha through a grid search for the oracle accuracy on a held-out validation set (0.8 and 0.6 for captioning on PASCAL-50S and 0.6 for MT). Table 1 and 2 in the updated pdf show results on captioning and MT respectively. We find that DBS length normalization helps over BS in the order of 0.4 BLEU points, but performs 0.2 BLEU points worse than DBS on MT. Thank you for this suggestion.
We also investigate the importance of the length term (in the Discussion section of the Appendix) by computing the correlation between the length and the accuracy obtained for each generated hypothesis. On the PASCAL-50S dataset, we observe that the correlation with length and SPICE is insignificant for all decoding methods - BS, DBS and L&J16.
Significance of DBS results: The oracle accuracy is a measure of the maximum accuracy that can be achieved by a list of M decoded solutions (assuming access to a perfect re-ranker). As the goal of our paper is to develop an efficient diverse decoding technique, coming up with better re-ranking methods is an entirely different problem that is beyond the scope of this work.
We perform additional analysis between the beam budget and corresponding oracle accuracy of the generated lists (presented in the Discussion section in the supplementary). We notice that DBS generates high-scoring sequences at smaller beam sizes as compared to the baselines -- meaning that it utilizes the beam budget better to explore the search space. In this context, we believe that efficient diverse decoding techniques can help us use simpler re-rankers that need to perform fewer comparisons to select the top-1 solution.
Also, we would like to point out that the gains obtained from diverse decoding on NMT are consistent with L&J16 -- oracle@20 increases by a score of 0.6 due to DBS, compared to using BS. (L&J16 obtain a score increase of 0.3 due to diverse decoding at B=200).
While it is possible that machine translation as a task does not require significant diversity in its outputs, DBS shows the potential of obtaining competitive gains provided access to a good re-ranker.
Finally, we note that oracle accuracies are a well-established performance metric, used by a line of previous work -- Batra et al., 2012, Prasad et al., 2014, Kirillov et al., 2015 and Stochastic Multiple Choice Learning for Training Diverse Deep Ensembles (Lee et al., 2016).
The paper addresses an important problem - namely on how to improve diversity in responses. It is applaudable that the authors show results on several tasks showing the applicability across different problems.
In my view there are two weaknesses at this point
1) the improvements (for essentially all tasks) seem rather minor and do not really fit the overall claim of the paper
2) the approach seems quite ad hoc and it unclear to me if this is something that will and should be widely adopted. Having said this the gist of the proposed solution seems interesting but somewhat premature.
Thank you for your feedback. We appreciate the note about the broad applicability of our work demonstrated with results on several tasks.
Can you kindly explain the terms used in the weaknesses listed by you?
The current review simply states opinion without pointing to anything concrete in the paper and as such, is not something we can use to improve the paper.
1. “Improvements are minor and do not … fit the … claim of the paper”
We present quantitative results on two applications — image captioning and machine translation. On both applications, under all standard metrics (SPICE, CIDER, BLEU, METEOR), our approach outperforms two strong baselines on all settings (oracle@1,5,10,20 using a beam size of 20 for all methods) except one setting (see table 4, M=1).
Moreover, the improvements of our work over recent competitive baselines (e.g. Li and Jurafsky, 2016) are typically larger than the improvements of the baselines over classical beam search.
Further, as we discuss in Section 5.2, results averaged over the entire test set may be a pessimistic measure of the gains due to diverse decoding because not all instances need diversity. In the context of image captioning, “simple” image scenes do not need diversity in captions (because there may not be much to say), while “complex” scenes may benefit more from diversity in generated captions. We provide relevant examples in the supplementary to illustrate this point. Interestingly, note that a more recent work by Li et al. https://arxiv.org/pdf/1611.08562v1.pdf refer to this aspect of our paper and develop techniques to control the diversity strength while decoding the output.
In this context, we would like to know what makes these improvements appear “minor”. And how do they not fit the claims of the paper?
2. “The approach is ad hoc”
We present a formal objective to optimize, describe why exact optimization of that objective is intractable even for small sequences, and provide a doubly-greedy algorithm that combines ideas from beam search (greedy in time) and diverse M-Best methods from PGM literature (greedy in solutions).
What makes our method ad hoc?
3. “gist of the proposed solution seems interesting but somewhat premature”
We developed a general beam search replacement, demonstrated its applicability and results on 4 different applications (image captioning, machine translation, dialog, visual question generation), presented parameter sensitivity plots, ablation studies with 4 different diversity measures, released our codebase (including our reimplementation of baselines), and an online interactive demo to visualize the working of the algorithm.
What makes this premature?
1) I stick with my argument that the improvements are minor. The absolute differences in the corresponding tables 1-3 are simply not impressive in my view - sorry.
after carefully rereading the claims there is maybe not that much of a mismatch - but the paper - as it is currently written - makes a big argument about the quantitative differences so that I was rather disappointed how marginal those differences are. Toning down the paper (and the response) would have been more convincing to me.
2) as described in the paper the procedure is double-greedy - at least in that sense it is ad hoc. Personally I also felt that the exact formulation is either directly drawn from literature or not well enough justified.
3) which makes me conclude that the work is premature
As neither the paper nor the comment above does seem to address any of the above points really I won't change my assessment either.
If other think this is a great approach and a good way forward I am the last to contradict.
Time will tell. And I might be simply not realizing the greatness of this approach as claimed by the authors.
I am sorry that this is turning into an opinion/impression argument.
That is not our goal, those arguments aren't productive, and we respect your view.
For the sake of scientific accuracy, we should be clear that saying a doubly-greedy approach is ad hoc is saying that
(a) beam search is adhoc and
(b) Nemhauser's classical greedy algorithm with a (1-1/e)-approximation guarantee for cardinality constrained monotone submodular maximization is adhoc.
Because those are the two greedy algorithms our approach is based on.
An approach with applicability to a variety of sequence prediction tasks and results on 4 different applications is demonstrably not ad hoc ("a solution designed for a specific problem or task, non-generalizable, and not intended to be able to be adapted to other purposes").
Thank you for your response and again, thank you for your review (as fellow reviewers, we do appreciate the time and effort that goes into these things!).
Following suggestions from Reviewer 2, we have revised our submission to include comparison to the modified beam search objective proposed by Wu et al., 2016 (Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation ).
Further details are provided in the paper and in the reply to Reviewer-2’s comments. We thank the reviewer for this suggestion.
We also conduct additional experiments that study the correlation of SPICE with caption length and the variation of oracle accuracy with beam budget.
While the first experiment suggests that longer sequences do not necessarily result in better scoring captions, the latter experiment shows that DBS utilizes the beam budget efficiently — obtaining higher oracle accuracies at much lower beam budgets compared to other decoding techniques.

Meta-review:
role: PC
data: comment: Unfortunately, even after the reviewers adjusted their scores, this paper remains very close to the decision boundary. It presents a thorough empirical evaluation, but the improvements are fairly models. The area chair is also not convinced the idea itself will be very influential and change the way people do decoding since it feels a bit ad hoc (as pointed out by reviewer 1). Overall, this paper did not meet the bar for acceptance at ICLR this year.

Decision: Reject