Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: Can authors provide more details on how the hyperparameters of Li & Jurafsky 2016 (\lambda, \gamma, \mu...) are tuned? In L&J16, these parameters were tuned to optimize for regular BLEU (and other metrics) which differs from "oracle" BLEU (and other metrics) used in this manuscript. In the case of "oracle" BLEU @ k (k=5,10,20,...), it seems to me that a method that generates more diverse candidates will have a better luck of including a candidate with a higher score. If L&J16 were tuned to attain a higher degree of diversity (to match that of DBS), will it also lead to a higher oracle BLEU/Cider/SPICE etc?
 
review_title: Li & Jurafsky comparison

final_score: rating: 4: Ok but not good enough - rejection
review_content: 
The paper addresses an important problem - namely on how to improve diversity in responses. It is applaudable that the authors show results on several tasks showing the applicability across different problems. 

In my view there are two weaknesses at this point

1) the improvements (for essentially all tasks) seem rather minor and do not really fit the overall claim of the paper

2) the approach seems quite ad hoc and it unclear to me if this is something that will and should be widely adopted. Having said this the gist of the proposed solution seems interesting but somewhat premature. 
review_title: good problem - but results are somewhat unclear

final_score: rating: null
review_content: NA
review_title: no question at this point

final_score: rating: 6: Marginally above acceptance threshold
review_content: 

[ Summary ]

This paper presents a new modified beam search algorithm that promotes diverse beam candidates. It is a well known problem —with both RNNs and also non-neural language models— that beam search tends to generate beam candidates that are very similar with each other, which can cause two separate but related problems: (1) search error: beam search may not be able to discover a globally optimal solution as they can easily fall out of the beam early on, (2) simple, common, non-diverse output: the resulting output text tends to be generic and common.

This paper aims to address the second problem (2) by modifying the search objective function itself so that there is a distinct term that scores diversity among the beam candidates. In other words, the goal of the presented algorithm is not to reduce the search error of the original objective function. In contrast, stack decoding and future cost estimation, common practices in phrase-based SMT, aim to address the search error problem.

[ Merits ]

I think the Diverse Beam Search (DBS) algorithm proposed by the authors has some merits. It may be useful when we cannot rely on traditional beam search on the original objective function either because the trained model is not strong enough, or because of the search error, or because the objective itself does not align with the goal of the application.

[ Weaknesses ]

It is however not entirely clear how the proposed method compares against more traditional approaches like stack decoding and future cost estimation, on tasks like machine translation, as the authors compare their algorithm mainly against L&J’s diverse LM models and simple beam search.

In fact, modification to the objective function has been applied even in the neural MT context. For example, see equation (14) in page 12 of the following paper:

"Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation" (https://arxiv.org/pdf/1609.08144v2.pdf)

where the attention coverage term serves a role similar to stack decoding (though unlike stack decoding, the objective term is entirely re-defined, more similarly to DBS proposed in this work), and the length penalty may have an effect that indirectly promotes more informative (thus more likely diverse) responses.

Comparison against these existing algorithms would make the proposed work more complete.

Also, I have a mixed feeling about computing and reporting only *oracle* BLUE, CIDEr, METEOR, etc. Especially given how these oracle scores are very close to each other, and that developing a high performing ranking has not been addressed in this work (and that doing so must be not all that trivial), I’m somewhat skeptical how much of DBS results make a practical difference.




**** [Update after the author responses] ****

The authors addressed some of my concerns by adding a new baseline comparison against Wu et al. 2016. Thus I will raise my score to 6. 




review_title: potentially interesting idea but lacking comparisons against other classic search techniques beyond simple beam search

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper considers the problem of decoding diverge solutions from neural sequence models. It basically adds an additional term to the log-likelihood of standard neural sequence models, and this additional term will encourage the solutions to be diverse. In addition to solve the inference, this paper uses a modified beam search.

On the plus side, there is not much work on producing diverse solutions in RNN/LSTM models. This paper represents one of the few works on this topic. And this paper is well-written and easy to follow.

The novel of this paper is relatively small. There has been a lot of prior work on producing diverse models in the area of probailistic graphical models. Most of them introduce an additional term in the objective function to encourage diversity. From that perspective, the solution proposed in this paper is not that different from previous work. Of course, one can argue that most previous work focues on probabilistic graphical models, while this paper focuses on RNN/LSTM models. But since RNN/LSTM can be simply interpreted as a probabilistic model, I would consider it a small novelty.

The diverse beam search seems to straightforward, i.e. it partitions the beam search space into groups, and does not consider the diversity within group (in order to reduce the search space). To me, this seems to be a simple trick. Note most previous work on diverse solutions in probabilistic graphical models usually involve developing some nontrivial algorithmic solutions, e.g. in order to achieve efficiency. In comparison, the proposed solution in this paper seems to be simplistic for a paper.

The experimental results how improvement over previous methods (Li & Jurafsky, 2015, 2016). But it is hard to say how rigorous the comparisons are, since they are based on the authors' own implementation of (Li & Jurasky, 2015, 2016).

---------------
update: given that the authors made the code available (I do hope the code will remain publicly available), this has alleviated some of my concerns about the rigor of the experiments. I will raise my rate to 6.
review_title: a relatively new problem, but proposed seems to be too simplistic

final_score: rating: null
review_content: Is it possible to apply the technique on the applications in Jiwei Li's papers, so you can do a fair comparison with Li?
review_title: experiment comparison

Meta-review: Unfortunately, even after the reviewers adjusted their scores, this paper remains very close to the decision boundary. It presents a thorough empirical evaluation, but the improvements are fairly models. The area chair is also not convinced the idea itself will be very influential and change the way people do decoding since it feels a bit ad hoc (as pointed out by reviewer 1). Overall, this paper did not meet the bar for acceptance at ICLR this year.
Decision: Reject