Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: This paper introduces the use of data-dependent Gaussian prior, to overcome negative diversity ignorance problem that includes the exposure bias problem for sequence generation models. In addition to the usual MLE (teacher forcing) criteria, the authors add the KL divergence between the prediction and the Gaussian PDF on the word embedding space. Experimental results show that the proposed method consistently improves the performance of the state-of-the-art methods for neural machine translation, text summarization, storytelling, and image captioning.

I lean to accept this paper. The proposed method is well motivated and shown to be effective in several tasks for language generation.

I have some major comments about the evaluation function $f(\cdot)$. The authors propose to define it as a Gaussian distribution.
- While this choice seems to be reasonable, I would like to know how its standard deviation can be defined. If it is a hyperparameter, the sensitivity of different deviations for the performance should be experimentally reported. A small valued deviation would make the KL divergence close to zero, while a large one makes its convergence slow.
- Another way to remedy the problem of KL divergence above is applying Wasserstein distance instead of KL divergence. I would like to know if the authors have investigated the use.

Minor comments:
- White space should be inserted between "sequence-to-sequence" and "(seq2seq)" on the third page.
- If the authors define a sequence using a bold and italic font as $\boldsymbol{y}$, each token can be represented using an italic font to distinguish each token and the entire sequence: $\boldsymbol{y}=<y_1,...,y_l>$. Otherwise, the sequence can be defined as $\mathcal{Y}$ if the authors like to represent each token as a vector.
- There is a typo on the fifth page. The word "dada-independent" should be "data-independent."
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: This paper proposes to add a prior/objective to the standard MLE objective for training text generation models. The prior penalizes incorrect generations/predictions when they are close to the reference; thus, in contrast with standard MLE alone, the training objective does not equally penalize all incorrect predictions. For the experiments, the authors use cosine similarity between fastText embeddings to determine the similarity of a predicted word and the target word. The method is tested on a comprehensive set of text generation tasks: machine translation, unsupervised machine translation, summarization, storytelling, and image captioning. In all cases, simply adding the proposed prior improves over a state-of-the-art model. The results are remarkable, as the proposed prior is useful despite the variety of architectures, tasks (including multi-modal ones), and models with/without pre-training.

In general, it is promising to pursue work in altering the standard MLE objective; changes to learning objective seem orthogonal to the modeling gains made in many papers (as evidenced by the gains the authors show across diverse models). This paper opens up several new directions, i.e., how can we impose even more effective priors? The authors show that it's effective to use a relatively simple fastText-based prior, but it's possible to consider other priors based on large-scale pre-trained language models or learned models. In this vein, a concurrent paper "Neural Text Generation with Unlikelihood Training" has also shown it effective to alter the standard MLE objective. I think it would be nice to discuss this paper and related works. Overall, I think the approach is quite general and elegant.

My main criticism is that the writing was unfocused or unclear at times. The intro discusses a variety of problems in generation, before explaining that the authors only intend to tackle one ("negative diversity ignorance"). It would have been more helpful to read more text in the intro that motivated the problem of negative diversity ignorance and the proposed solution. The second paragraph in the Discussion in Section 4 is rather ambiguous and hand-wavy. It would be nice to see the authors' intuition described more rigorously (i.e., explicitly describing in math how the cosine similarity score is used in the Gaussian prior, or describing in math how the central limit theorem is used). Some of the existing mathematical explanation in section 4 could be made simpler or more clear (the description of f(y) seems to be a distraction since it doesn't end up in the final loss).

I would have also appreciated more analysis. After reading the paper, I have the following questions (which the authors may be able to address in the rebuttal):
* Do off-the-shelf fastText embeddings work well? How important is it to train fastText embeddings on the data itself? If off-the-shelf embeddings worked well, that could make the method easier to use for others in practice.
* How does the gain in performance with D2GPo vary based on the number of training examples? Priors are generally more helpful in low-data regimes. If that is the case here as well, you might get even more compelling results on low-data tasks (all tasks attempted here are somewhat large-scale, as I understand)
* Qualitatively, do you notice any difference in the generations? How does the model make mistakes (are these more "semantic" somehow, i.e. swapping a different synonym in). Perhaps the gaussian prior has some failure modes, i.e., where it increases the probability of very incorrect/opposite words because they have a similar fastText representation. These kinds of intuitions would be useful to know

I also have one technical question:
* When you compare against MASS (Song et al. 2019), do you use the same code and/or pre-trained weights from MASS, or do you pre-train from scratch using the procedure from MASS? (The wording in the text is somewhat ambiguous.) I'm just wondering how comparable the results are vs. MASS, or if it would be useful to know how your version of the pre-trained model does.


Despite my above questions/concerns, I think the proposed method or its predecessors could provide improvements across a variety of text generation tasks, so I overall highly recommend this paper for acceptance.

review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: The paper introduces a new Gaussian prior objective, "D2GPo", that addresses the fact that in sequence generative models, all incorrect predictions are penalized equally by MLE, a phenomenon which the authors refer to as the negative diversity ignorance drawback. The proposed objective is simple to implement and can easily be added on top of a regular cross-entropy loss. The paper shows that the new objective shows consistent improvements across a wide range of tasks.

- Something that perturbed me a lot when reading Section 4, is that the function f should take as input 2 arguments, and not just 1. Typically, in Equation 7, from what I understand the numerator should be exp(f(y'_i, y_i)/T), and the denominator the sum over the exp(f(y'_j, y_i)/T). In other words, the value of f assigns a score to a word y'_j which depends on the target word y_i. This is also what suggests Figure 1: f depends on the target word. I think this should really be clarified in the paper, and f needs to be defined formally. In Figure 1, what is the exact value of f when the model outputs "armchair" / when the model outputs "armchairs". If I did not understand correctly, please point me out.

- The cross-entropy loss in a generative model is essentially the KL divergence loss KL(Q, P) where Q is the true distribution (i.e. a one-hot vector) and P is the model output. In Equation 7, when T -> 0, the distribution Q becomes this one-hot vector. In that case, from what I understand, the D2GPo objective is actually very similar to the initial MLE objective, except that you compute KL(P, Q) instead of KL(Q, P), and that Q is not exactly one-hot because you consider a T > 0. Can you confirm this? Also, any reason why you considered KL(P, Q) instead of KL(Q, P)?

- What value did you use for the temperature T? Did you study its impact?

- Same question for lambda in Equation 6, what is the value you considered, and did you study its impact?

- As mentioned in the introduction of the paper, one drawback with traditional neural generative, is the generation diversity, i.e. the fact that generations are generic, and an easy way to see this is to observe that generations are mostly composed of the most frequent words in the vocabulary. Did you evaluate whether a model trained with D2GPo has more diversity, and is more likely to generate rare words compared to a regular model trained with MLE only?

- Figure 1 in the appendix is helpful. It would be nice to move it to Section 4.

- In the related work, you write "the Transformer provides us with a more structured memory for handling long-term dependencies", which sounds a bit odd. There is no explicit memory component in the transformer, the ability to handle long-term dependencies comes instead from the self-attention mechanism.

Overall the paper tackles an interesting problem which I feel has received surprisingly little attention from the community. The paper is well written, and has a lot of experiments supporting the method. But I think Section 4 needs to be clarified, and some experimental details are missing. Also, more information about the impact of T and lambda (at least on one type of experiments) would be very useful to have.
review_title: Official Blind Review #1

Meta-review: This paper addresses the problem of poor generation quality in models for text generation that results from the use of the maximum likelihood (ML) loss, in particular the fact that the ML loss does not differentiate between different "incorrect" generated outputs (ones that do not match the corresponding training sequence).  The authors propose to train text generation models with an additional loss term that measures the distance from the ground truth via a Gaussian distribution based on embeddings of the ground-truth tokens.  This is not the first attempt to address drawbacks of ML training for text generation, but it is simple and intuitive, and produces improvements over the state of the art on a range of tasks.  The reviewers are all quite positive, and are in agreement that the author responses and revisions have improved the paper quality and addressed initial concerns.  I think this work will be broadly appreciated by the ICLR audience.  One negative point is that the writing quality still needs improvement.
Decision: Accept (Talk)