Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper concerns the limitation of the quality-only evaluation metric for text generation models. Instead, a desirable evaluation metric should not only measure the sample quality, but also the sample diversity, to prevent the mode collapse problem in gan-based models generation. The author presents an interesting, but not too surprising finding that, tuning the temperature beam search sampling consistently outperform all other GAN/RL-based training method for text generation models. The idea of sweeping temperature during beam search decoding is not new in the NLP community, which limits the novelty of this paper. What’s more, some parts of the experiment results is also somehow not new, in the sense that the SBLEU vs Negative BLEU tradeoff curve is also shown in [1,2,3,4].

[1] Jointly measuring diversity and quality in text generation models, 2019
[2] Training language gans from scratch, 2019
[3] On accurate evaluation of gans for language generation, 2018
[4] Towards Text Generation with Adversarially Learned Neural Outlines, 2018

I would love to increase my score if the author could address the following comments:
(1) Are the comparing methods, say MLE models and other GAN-based models, have the similar number of model parameters? It is not clear from the paper. Otherwise, one can use a 12/24 layer Transformer-XL to have dominative performance?
(2) Since this is an empirical study paper. It would be great if this paper can also present more SoTA models trained by MLE such as Transformer-XL on more challenging datasets, such as Wikitext-2 or Wikitext-103. In this kind of large vocabulary datasets, I think the RL/GAN-based training methods would easily breakdown, and far worse than MLE-based training.
(3) To make the empirical study more comprehensive, the author could perhaps evaluate with the n-gram and FED metric.

review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: Recently many language GAN papers have been published to overcome the so called exposure bias, and demonstrated improvements  in natural language generation in terms of sample quality, some works propose to assess the generation in terms of diversity, however, quality and diversity are two conflicting measures that are hard to meet. This paper is a groundbreaking work that proposes receiver operating curve or Pareto optimality for quality and diversity measures, and shows that simple temperature sweeping in MLE generates the best quality-diversity curves than all language GAN models through comprehensive experiments. It points out a good target that language GANs should aims at. 

For the experiments on long-text generation using EMNLP news 2017, it is not clear how the data is partitioned as training data, validation data and test data to get the results in Figures 4(a), moreover for the results for LeakGAN, RankGAN SeqGAN, MaliGAN, it seems that they are copied from other papers, but again how the data set is partitioned is not clear  for example in SeqGAN's paper, and most likely, the data is partitioned in a different way, so the results are not comparable. The authors should run the code and get the results on it own.

An important RL-free language GAN paper is missing, 
Zhongliang Li, Tian Xia, Xingyu Lou, Kaihe Xu, Shaojun Wang, Jing Xiao: Adversarial Discrete Sequence Generation without Explicit NeuralNetworks as Discriminators. AISTATS 2019: 3089-3098.
This paper directly and adversarially train a language model without MLE pre-training and obtains good results, it is better to compare the results.

Typo: page 1, the second line from bottom, as a computationally, not an
review_title: Official Blind Review #1

Meta-review: Main content:

Blind review #1 summarizes it well:

Recently many language GAN papers have been published to overcome the so called exposure bias, and demonstrated improvements  in natural language generation in terms of sample quality, some works propose to assess the generation in terms of diversity, however, quality and diversity are two conflicting measures that are hard to meet. This paper is a groundbreaking work that proposes receiver operating curve or Pareto optimality for quality and diversity measures, and shows that simple temperature sweeping in MLE generates the best quality-diversity curves than all language GAN models through comprehensive experiments. It points out a good target that language GANs should aims at. 

--

Discussion:

The main reservation was the originality of the idea of using temperature sweep in the softmax. However, it turns out this idea came from the authors in the first place, which they have not been able to state directly due to the anonymity requirement. Per the program chair's instruction to direct this to the area chair, I think this has been handled correctly.

--

Recommendation and justification:

This paper should be accepted. It provides readers with insight in that it illuminates a misconception of how important exposure bias has been assumed to be, and provides a less expensive MLE based way to train than GAN counterparts.
Decision: Accept (Poster)