Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper attempts to address the difficulties of adversarial training for language generation, as discrete generator outputs make conventional GANs difficult to train.

The authors propose combining cooperative decoding proposed by Gabriel et al (2019), along with MCTS-based decoding, to train a discriminator and generator jointly. The generator outputs are trained with the results of their MCTS-based cooperative decoding procedure.

The authors demonstrate their method's performance on summarization (CNN and Daily Mail dataset) and question generation (SQuAD).

main_review: Overall, this work demonstrates a novel approach to integrating cooperative training, and is a useful contribution. The method is promising, and could be extended and applied to other discrete generation applications as well.

The comments that follow are mostly related to experiments and the paper's presentation.

Regarding experiments, as the authors point out, the "Base" discriminator metric uses the discriminator trained on MLE outputs generated by beam search. However, it's used both as a metric and for training, so the authors use a "Base+" discriminator that is further fine tuned on the final model outputs. One potential concern is that the fine tuning process may result in the finetuned model still being too similar to Base, and unable to find discriminating factors that would be found had the model been trained from scratch. Have the authors considered training a discriminator from scratch, and if so, how does it perform? Performing this experiment is not a requirement for my review: as generated natural language is difficult to evaluate automatically, human evaluations should still take precedence.

To clarify, is the generator pretrained with standard MLE before training SelfGAN/Coop-MCTS, or is the latter trained from scratch? Or is only the discriminator pretrained? How stable is the SelfGAN training?

Overall, the paper may be somewhat clearer with more equations to precisely and concisely describe the method, and/or some minor reorganization. Algorithm 1 is somewhat brief, and the overall method's description is split between Section 3 and the MCTS description in Section 4, which due to the author's setup, are also important for describing how the generator is trained.

*Related work:*

The authors cite the cooperative decoding work of Gabriel et al from 2019, but it appears they have a later version in EACL 2021, see https://arxiv.org/abs/1907.01272. This should be updated accordingly.

As the authors cite other applications of MCTS in NLP, the authors should also consider citing related MCTS decoding work in neural machine translation, such as https://arxiv.org/abs/2004.12527 and https://arxiv.org/abs/2104.05336.

*Clarifications:*

- On line 309, the sentence "SelfGAN produces sequences that form a more compact set for discriminator training …" is unclear, what does this refer to?
- On line 319, can you clarify how the metric for "colinearity of generator gradients for the generated samples from the model with those for the corresponding human references" is computed?
- Last sentence of the conclusion is awkward/unclear, what does "increasing it" refer to?

*Minor comments:*

- L111: add a caption for Algorithm 1, and reference it in the main text
- L124 and alg. 1 L5: inconsistent naming between "standard training" and "classical maximization", perhaps say something like "standard maximum likelihood with S_coop as the target"
- L124: in the first equation, shouldn't $\sum_{...} \pi_{\theta}(...)$ be $\sum_{...} \log \pi_{\theta}(...)$?
- L178: 'a in' -> 'in'
- L262: version -> versions
- Table 1 should explain "B4", "R1", and "RL" in the caption (e.g. BLEU4 ("B4"))
- L323: mean -> means?

limitations_and_societal_impact: The authors adequately discuss limitations and societal impact in their conclusions.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a method of training (or rather fine-tuning) text generation models using a GAN-like setup. Instead of using the discriminator directly to update the generator as is usually the case, the authors propose using the discriminator as a way of cooperative decoding to generate high-quality samples which are then used as the gold output to train the model in the usual teacher-forcing setup. The authors explore a sampling procedure based on MCTS to generate more diverse sentences rather than using beam-search or sampling during training and inference. On summarization and question generation, they show improvements on the standard and proposed evaluation metrics. 
main_review: Strengths:
1. Language GANs have been shown to be difficult to train due to sparse signals from the discriminator. This paper presents an interesting idea to incorporate this signal in a more stable manner. 
2. MCTS based decoding algorithm also has potential applications even for cases where the proposed training method was not used. 
3. The paper is clearly written and easy to follow for the most part. 

Weaknesses and comments:
1. The paper presents the method as a language GAN yet doesn't evaluate it on the most common task used in this area: unconditional text generation where the goal is to generate high-quality diverse text where the benefits of the proposed decoding method can be more clearly evaluated. 
2. Relating to my previous comment, a diversity measure should be added to the evaluation. Plus the quality metrics show very small improvements in most cases over baselines.  
3.  The proposed decoding process is quite slow for the improvement it offers. 


------------------------

Update: The author response addresses most of my concerns, I am updating my score to 7. 
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: To train language GANs, previous approaches use the discriminator score as a reward, and train the generator through RL. Instead, this paper proposes to use the discriminator to guide the decoding process, searching for outputs with high discriminator scores, and then train the generator to generate these outputs. To find outputs with high discriminator scores, the authors employ Monte Carlo Tree Search and use the discriminator as the value network. Experiments include question generation and summarization, and the proposed method shows improvements to MLE and ColdGAN.
main_review: While the proposed model looks fine, I think the experiments and baselines are lacking, and most importantly, the results do not match those reported in previous work. I hope the authors can validly address my concerns.

The results of MLE and ColdGAN in Table 1 are overall much worse than those reported in the ColdGAN paper. Just to give an example, T5-small (MLE) has BLUE-4 19.65 on question generation (see Table 2 in the ColdGAN paper), while this paper reports 16.5. Can the authors please explain the huge discrepancy?

In addition, the models are not tested on unconditional generation. We know that mode collapse is a serious problem of GANs. It's thus unclear whether the proposed method can generate diverse outputs in unconditional scenarios.

For cooperative decoding, the proposed MCTS approach is compared against a reranking baseline. But there're also other methods to use the classifier to guide the generation process. For example, one can combine the generator probabilities and the discriminator probabilities to sample the next word [1]. Can the authors compare with this method as well?

In Fig 2 Right, the authors report the collinearity of generator gradients for generated samples with those for human references. However, it's unclear to me why the higher the better. If so, why not just train with human references using MLE, which has a collinearity of 1?

Finally, the writing is sometimes obscure (e.g., "the sampling mechanism of sequences that have to be favored through MLE", "for each condition in the training set X"). Please revise them to avoid confusion. 

[1] Yang and Klein. FUDGE: Controlled Text Generation With Future Discriminators. NAACL 2021.
limitations_and_societal_impact: The authors have discussed the societal impact of natural language generation in the paper.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work proposes selfGAN. Instead of using RL techniques to utilize the discriminator. They do Coop-MCTS which is termed as a "cooperative decoding", which is designed to generate sequences that is better than naïve sampling or decoding. And the coop-decoded sequence is then used to train the generator in turn, in the standard teacher-forcing way. I understand this approach as the process in which the discriminator is helping the generator to refine its generations. Better results comparing to MLE and ColdGAN are shown in the experiments.

main_review: Strength of this work: (1) The idea is novel and interesting (it reminds me of self training). (2) The results are good and the comparison protocols are principled. 

Weakness: (1) There are a number of text GAN works that claim to outperform the MLE baseline, therefore only comparing with ColdGAN does not seem to be enough. (2) This paper is well written in most parts, however, I find it hard to understand the Coop-MCTS section.

In addition, I have several questions:

Line 27: "However, NLG metrics are known to not reflect well human judgement [21], which explains why the resulting models tend to be qualitatively worse than their MLE baselines [3]. " While that maybe true, it's also possible that the RL methods is simply hard to train well, and the NLG metrics is not to blame.

I did not quite understand the Coop-MCTS algorithm (I admit that I'm not familiar with PUCT). How is it able to revise a sequence? What's the inutition behind Eq.1? I hope you could rewrite this section when you revise the paper.

Is selfGAN also initialized by T5? I'm assuming it is, could the training algorithm work if it is trained from scratch?

It would be good to discuss a bit about [1], I know the two are doing different things. But the naming could be confusing.

Figure 2 left is broken?

[1] CoT: Cooperative Training for Generative Modeling of Discrete Data  https://arxiv.org/abs/1804.03782

limitations_and_societal_impact: Did you mention how fast is MCTS comparing to beam-search?
review_title: null

Meta-review: This work demonstrates the effectiveness of a novel and interesting approach to utilising GANs with cooperative training and MCTS. The paper is well written and easy to follow. Though the initial results in the paper lacked evaluation on an important task (unconditional text generation) and didn't evaluate the diversity of generated text, the strong rebuttal resolves these issues and gives convincing evidence of the effectiveness of SelfGAN and MCTS. Hopefully this paper inspires more work in this interesting direction.
Decision: Accept (Poster)