Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper introduces a Self-Adversarial Learning (SAL) mechanism in GAN based text generation, aiming at tackling the problem of mode collapse and sparse rewards problem. Specifically, motivated by “self-play” mechanism in RL community, instead of using a binary classifier as discriminator in original GAN, SAL employs a comparative discriminator which is a pairwise classifier with three classes: “better”, “worse” and “indistinguishable”. The authors provide lots of experimental results and ablation study showing the efficiency of the proposed mechanism in comparison with previous GAN models.

Decision: weak reject. 
This paper is well motivated: clearly sparse rewards and mode collapse are two problems need to be solved in GAN based text generation, however, the following concerns prevent me from finding this paper acceptable in ICLR:
The “self-play” idea is widely used in RL. In RL, the comparison between policies can be determined directly by the game simulation results. In text generation, such comparison is more difficult to be judged. This paper appears to assume that the generated sentences are “worse” than the real samples, which is similar to the original definition of discriminator in GAN, and the generated sentences in earlier epochs are “worse” than that in later epochs, which needs further justified.
As a simple extension to GAN, I’m not convinced that the problem of mode collapse could be solved by proposed mechanism. If the generator falls into a local minimum, a collapsed mode, the proposed mechanism will never pull the generator out of that. Moreover, what is the theoretical foundation of the proposed evaluation metric on quality-diversity trade-off, NLL_{gen} + NLL_{oracle}?
The setting of the important set of hyper-parameters, reward weights, is unclear in the paper. The reward weights directly influence the reward in training, thus should play an important role of model performance. More discussion about this should be provided.
Moreover, in the paper, only comparison between proposed mechanism with GAN based models are shown. Comparison with more recent models like RelGAN should be provided. And comparison with other state-of-the-art text generation model should be discussed.



Feedbacks: 
References regarding experimental results in table are incorrect. For example, the results in synthetic data should be in “Table 2” and COCO image caption dataset should be in “Table 3”. 
Some imprecise parts, for example, in Equation (5) and (6), it should be G_\theta(Y_{1:t-1} and G_\theta(y_t | Y_{1:t-1} .
I’m curious about what the performance would be like if the weakly supervision by regarding sentences generated in later training stage are “better” than the sentences generated in earlier training stage is removed in training comparative discriminator. This is different from the CAL model in the ablation study.
What is the influence of different values of rewards weight? 
Also about the rewards weight, in the description of Scheduled rewarding, the rewards weight is described to be linearly changed with training iteration, while in Appendix C.3, the rewards weight is described to be fixed. This is very confusing.
A minor issue: for image captioning, a lot other metrics are widely used in measuring the model performance, e.g. CIDEr, SPICE and so on. Those metrics could be helpful for audience to understand how the model performs in comparison with other captioning models.
Another minor issue: for image captioning and WMT (conditional generation), the detailed model structures are not described in paper, which is not very friendly to audiences with relatively little knowledge in related areas.


review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: Summary: This paper describes a self-adversarial method to train a GAN for text generation that circumventes the problems of mode collapse and reward sparsity. They replace the traditional binary discriminator with a comparative discriminator, which provides the generator with more frequent rewards that are not always restricted to the limited number of real training examples. 

They show gains on synthetic and real generation tasks. 

Strengths:

The paper is well written with adequate details about the training objectives and the learning algorithm. I appreciate the human analysis conducted by the authors. The SAL method performs better than many strong baselines consistently across 3 datasets and on various metrics.

The authors present strong ablations, that are very insightful, particularly, regarding the role played by classifying sentences as indistinguishable.

Weaknesses and Questions: 

In general, some more description about the motivation of each metric would be helpful, rather than just stating that its from previous work.  

How is BLEU evaluated for this text generation task? Is the entire test set treated as a single reference? Do you generate the same number of tokens as the reference and then compute n-gram overlap between the reference and the prediction? What happens to the brevity penalty of BLEU?

In Table 4, does BLEU-5(F) denote only 5-gram precision, or is it the geometric mean of 1-5 gram overlaps?

How does NLL_gen serve as a measure of diversity for the synthetic dataset?

For the human evaluation, does quality mean grammaticality? Can simple memorized sentences be scored higher?

Typos in Section 4. The authors refer to tables 17 and 18. Please fix. 
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: To alleviate the issues of reward sparsity and mode collapse in most text-generation GANs with a binary discriminator, this paper proposes a self-adversarial learning (SAL) framework with a novel comparative discriminator that takes pairs of text examples from real and generated examples and outputs better, worse, or indistinguishable. Inspired by self-play in reinforcement learning, SAL employs self-play to reward the generator to generate better samples than previous samples with self-improvement signals from the comparative discriminator. It is argued that, because the comparative discriminator always produces self-improvement signals during the training and the self-improvement signal will not be very strong when generated samples are already good enough, the issues of reward sparsity and mode collapses in conventional text GANs are reduced. Experimental results on synthetic data and benchmark datasets demonstrate that SAL outperforms SeqGAN, MaliGAN, and RankGAN both quantitatively and qualitatively.

Pros:

This paper is well-written. The studied problem is well-motivated and the method is clearly presented. The SAL framework with self-play and a comparative discriminator is novel and the results are convincing.

Cons:

1) SAL has a new discriminator, which can be viewed as an architecture change. Although it has a new training strategy and is very different from recent text GANs such as LeakGAN and RelGAN, the latest results of these recent methods on the benchmark datasets should be included in Table 4 and 5 for reference.

2) The comparative discriminator is novel and well-suited for comparing pairs of samples. However, the only informative signals come from pairs of data with one from a real sample and the other from a generated sample. During the self-play process, why the signals from the comparative discriminator comparing two generated samples are always trustworthy? Do the reward signals always help train a better generator? 

3) In section 4.3, since the self-play and the comparative discriminators are shown to be the most significant, it is better to clearly show the algorithms as in Algorithm 1 for training these two baselines either without self-play or without the new discriminator. Does replacing the binary classifier with WGAN classifier help here? All these details should be included in the appendix.

4) Missing training details: It is unclear how the model architectures are chosen, and learning rate, optimizer, training epochs etc. are also missing. All these training details should be included in the appendix.

In summary, this paper proposes a novel discriminator with a new training strategy suited for the discriminator for text generation. Experimental results demonstrate the effectiveness of the proposed framework. I like the idea in the paper and am happy to vote for acceptance.

review_title: Official Blind Review #1

Meta-review: This paper proposes a method for improving training of text generation with GANs by performing discrimination between different generated examples, instead of solely between real and generated examples.

R3 and R1 appreciated the general idea, and thought that while there are still concerns, overall the paper seems to be interesting enough to warrant publication at ICLR. R2 has a rating of "weak reject", but I tend to agree with the authors that comparison with other methods that use different model architectures is orthogonal to the contribution of this paper.

In sum, I think that this paper would likely make a good contribution to ICLR and recommend acceptance.
Decision: Accept (Poster)