To Beam Or Not To Beam: That is a Question of Cooperation for Language GANs
Authors: Thomas Scialom
Paul-Alexis Dray
Jacopo Staiano
sylvain lamprier
Benjamin Piwowarski
Venue: NeurIPS 2021 Poster (2021)
Keywords: NLP - NLG - Language - Summarization - GAN - Generative - Adversarial - Discriminator - Cooperative - Decoding - Search - Beam

Abstract: Due to the discrete nature of words, language GANs require to be optimized from rewards provided by discriminator networks, via reinforcement learning methods. This is a much harder setting than for continuous tasks, which enjoy gradient flows from discriminators to generators, usually leading to dramatic learning instabilities.   However, we claim that this can be solved by making discriminator and generator networks cooperate to produce output sequences during training. 
These cooperative outputs, inherently built to obtain higher discrimination scores, not only provide denser rewards for training but also form a more compact artificial set for discriminator training, hence improving its accuracy and stability.
In this paper, we show that our SelfGAN framework, built on this cooperative principle, outperforms Teacher Forcing and obtains state-of-the-art results on two challenging tasks, Summarization and Question Generation.

To Beam Or Not To Beam: That is a Question of Cooperation for Language GANs
Thomas Scialom" Paul-Alexis Dray", Sylvain Lamprier',
Benjamin P Jacopo Staiano" CNRS, France
Sorbonne Université, CNRS, LIP6, F-75005 Paris, France reciTAL,, Paris, France thomas@recital.86
Abstract
Due to the discrete nature of words, language GANs require to be optimized from rewards provided by discriminator networks, via reinforcement learning methods. This is a much harder setting than for continuous tasks, which enjoy gradient flows from discriminators generators, usually leading to dramatic learning instabilities. However, we claim that this can be solved by making discriminator and generator networks cooperate to produce output sequences during training. These cooperative outputs, inherently built to obtain higher discrimination scores, not only provide denser rewards for training, but also form more compact artificial set for discriminator training, hence improving its accuracy and stability. In this paper, we show that our SelJGAN framework, built on this cooperative principle, outperforms Teacher Forcing and obtains cate-of-theeart results on two challenging tasks, Summarization and Question Generation.
Introduction
Natural Language Generation encompasses tasks such as Machine Translation, Summarization or Data To Text generation. The real life applications numerous, but require highly reliable and fluent models. Despite sigmificant advances, state-of-the-ane art models are still known to be de- generated, with outputs containing repetitions and even nonfactual information i.c. hallucination [13]
Among the culprits is a limitation of Teacher Foreing 37 :: the loss is computed at a token level while the aim is to produce complete sequences, Moreover, while single ground-truth reference is considered correct, several realizations of the same content may exist. Finally, the model is subject to Exposure Bias 227 i.e. mismatch between training and inference distributions in the latter, the model has no access to ground truth for the previously generated tokens. The literature has considered this mismatch responsible for the lower quality observed when generating longer sequences (2) [6]
To overcome such Teacher Forcing limitations, a consensus has emerged: a sequence level objective should be introduced [27] [4]] A body of work has proposed to use Reinforcement Learning (RL) with standard NLG metrics like BLEU 39 or ROUGE 22). However, NLG metries are known not
reflect well human judgement (1 which explains why the resulting models tend to be qualitatively worse than their MIEE baselines evaluate the output with learned discriminator. An ideal discriminator would not be biased w.r.t. to its training set, and could therefore be considered as perfect metric that matches human consensus. Note that discriminators are already reported to be highly accurate distinguish human written texts from machine generated ones (1 43).
To move toward less biased metrics, a natural alternative is to
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

In light of this observation, two concurrent approaches have been explored: i) at training time, using Generative Adversarial Networks [42] ; and ii) at inference time, via cooperative decoding [T] discriminator guides the search algorithm, such that the generator and the discriminator cooperate to select the generated tokens. These approaches pursue the same objective: producing texts more similar to what human writes.
Both methodologies suffer from specific limitations. Cooperative decoding algorithms rely on a discriminator that re-ranks a limited set of candidates selected by the generator ! decoding algorithms are limited by the generator ability to rank relevant tokens in good enough position. On the other hand, language GANs are learned via reinforcement learning due to the discrete nature of text. This makes them particularly unstable to train, and usually fall short compared to Teacher Forcing [3. In standard Language GANs, the discriminator provides a reward for the entire sequence, which can be difficult to exploit by the generator due to its sparsity (6]
Hence, cooperative
In this paper, we propose SelfGAN, a framework to learn language GANs in a Self-ttaiinng process where the signal from the discriminator is passed to the generator in a completely new way. We consider cooperative algorithms as a way to infuse the discriminator signal. We start from a simple observation: outputs obtained via cooperative decoding are more human-like, compared to their generator-only counterparts. Inspired by recent knowledge distillation approaches, we propose to consider cooperative outputs as targets in a Teacher Forcing training process: cooperative decoding stands as teacher we attempt to imitate through the generator network. Just like standard GAN, both the generator and the discriminator are trained each step. While the generator improves. it becomes adversarial to the discriminator, which benefits from the cooperative generation. The diseriminator, now trained on improved sequences, also contributes to improve the cooperative generation, and so forth. Note that in SelfGANs the discriminator is only used to drive the cooperative generation and never to provide a reward signal like in standard Language GANs.
SelfGAN can be implemented with any cooperative decoding algorithm. Current cooperative ap- proaches (7 rely on "myopic" algorithms like Beam Search or Sampling that generate the tokens left-to-right. The model has to always predict the next word, and can never look back and revise past choices. In some cases, despite all the candidates being judged to likely not be human by the discriminator, the model is locked in a dead-end. This behavior is quite unnatural for humans who often proofread their texts. We refer to this phenomenon as the left-to-right curse.
To address this left-to-right curse, we introduce Coop-MCTS, a new decoding algorithm based on Monte Carlo Tree Search (MCTS) 14]. We compare Coop-MCTS state-of-the- art cooperative decoding algorithms in two scenarios: i) inference time, as the decoding algorithm; and ii) during training, as the cooperative algorithm in SelfGAN. In both scenarios, we show that the respective resulting outputs are more likely to look like human texts and improve all the automatic metrics.
All in all, contributions can be summarized as follows:
1. SelfGAN We propose a new training framework based on cooperative decoding, wherein the generated sequences are used as ground truth;
2. Coop-MCTS We improve cooperative decoding with new decoding algorithm, Coop- MCTS, offering a solution to the left-to-right limitation of current search methods;
3. We show that combining SelfGAN and Coop-MCTS compare favorably to prior sfatc-of-the- art results on two challenging tasks, Summarization and Question Generation.
2 Related Work
Beyond Teacher Forcing To mitigate the limitations inherent to Teacher Forcing, various alternatives have been proposed. In Scheduled Sampling, Bengio et al. (2 proposed to condition the generation not only on the ground truth tokens, but also the generated ones. Given that only one reference is available, this introduces a new mismatch when computing the loss, this time between the generated tokens used to condition the model, and the target tokens. To take into account multiple possible
Opposed a generator that outputs probabilities for the entire vocabulary V at once, a discriminator outputs the likelihood for a specific sequence to be human- human-written or machine machine-generated. generated. Calculating the discriminator probability for every possible sequence is therefore not realistic, as the computation grows exponentially at a pace of V where V is the vocabulary size and ( the sequence length.

---

references, using sequence level metric is potential alternative. In Mixer. Ranzato et al. [27] chose BLEU, the standard metric to evaluate Machine Translation. Since is not differentiable. the task is framed in Reinforcement Learning setup where the reward corresponds to the BLEU score given sampled sequence tokens. Paulus et al. 23 applied the same method to Summarization, using this time ROUGE. While the results improved in terms of ROUGE, the human evaluation found that the generated summaries were rated worse in term of fluency than the MLE baseline. The model learns to take advantage of metric biases, while being less correct according to human judgement.
Language GANs In theory, a perfect discriminator would be able to judge if an output corresponds to the data distribution or not. Discriminators could therefore be an interesting alternative reward compared to other metrics. In practice, we need to train the discriminator jointly with the generator, framing the task as GAN. Language GANs are known to underperform MLE [3]. due to the unavoidable sparsity of discriminator reward. A large body of works have proposed denser rewards: ranking or comparative discriminators A 17 45 sequential discriminator where the rewards are More recently, Scialom et
provided at each time step of the generation
E0 proposed to stabilize the GAN training by lowering the Softmax temperature to explore more structured outputs, and closer to the generator distribution.
33 d
In this work, our proposed framework allows to propagate the discriminator signal in a cooperative way, which can be seen as an alternative solution the sparsity of the reward and the training stability.
Knowledge Distillation SelfGAN has a connection with knowledge distillation 12]. where a student trained outputs from the teacher. In particular, self distillation using only generator has shown to improve generalisation on image GANs 44 by acting as label smoothing. To the best of our knowledge, this work is the first propose the idea of augmenting the teacher by coupling a discriminator to generator. Beyond GANs, SelfGAN could serve for other applications using distillation, e.g. in semi-supervised methods that use a teacher model to create synthetic labels for unlabeled examples [33 40
Monte Carlo Tree Search in NLG Despite important successes in games 29 36 very few works
have attempted to apply MCTS to NLG. Kumagai et al. [15) proposed to employ context-free grammar rules combined with n- n-gram language model and explore the space of grammatically correct texts via a MCTS. In the context of commercial e-commerce agents, Mukherjee [99 proposed to optimise with MCTS scoring function designed to reward grammatical correctness.
3 SelfGAN
Algorithm SelfGAN
1: Input: a generator gen, a discriminator discr, and a cooperative decoding method decod
2: for epochs do
3: for X, Sref in training set do
>> Start Training
4: Scoop t- decodcoop (X, gen, discr)
5:
gen.train(srcs=X, (gts=Scoop)
b Standard maximum likelihood but with Scoop as the
target, and not Srej
6:
discr. diisertrrin(srcs=xX human_ exs= Sref, Muchine_exs=SSoee
The difficulty of GAN-based approaches for NLP tasks lies in the fact that no gradient flow can be propagated from the discriminator to the generator. As discussed above, approaches from the literature circumvent this difficulty by employing RL approaches, using discriminator scores as rewards to train the generator. However, such approaches induce great instabilities in the learning process, due to the use of a non-stationary reward function in addition to the high variance associated to monte-carlo estimations of RL.
The idea in our SelfGAN approach is to transfer the sparse signal of the discriminator, classically used as rewards for RL procedure, to the sampling mechanism of sequences that have to be favored through MLE. In that way, SelfGAN starts from a pretrained generator, that we fine-tune using sequences Scoop provided by a cooperative decoding process decodgoop for each condition in the training set X. This process, detailed in the next section, uses both the generator and a discriminator network output human-like sequences Scoop- for which we improve the generator likelihood
via classical maximization: maxe (X,,e)((X)S....) (2(2) where (s(2)z) = II!=1 Te Te(
3

---

stands for the generator probability of sequence given the conditioning input with implemented as neural architecture with softmax output function.
At each iteration of the training procedure, the discriminator network is optimized as a binary classifier on i) the human references and ii) the machine generated via the cooperative sequences:
Σ (E.
log(D(æ + G
log(1 - Dar, Scoop))
H
where x is the source input, H is the set of pairs associating with human written text Syef from the data distribution, and G is a set of pairs with generated outputs Scoop- D(x, s) stands for the probability, provided by the discriminator network, that sequence is human reference for condition x. In order to effectively guide the cooperative process at each step, the discriminator needs to be sequential: consistently with [31], use left-to-right mask during training, allowing discriminator predictions for unfinished sequences.
Please note that, by construction of the cooperative decoding process, we have with high probability at each iteration (xx, Scoop) >= Dxx, ggen) for any condition æ E X, with Scoop cooperative decoded sequence for x and Sgen a sequence directly sampled from the generator according to тT (s|z). Based on this observation, and provided that the discriminator sufficiently trained at each step, the generator is trained such that the probability of predicting human-like sequences is maximized. This process i) allows us to consider a sequence level metric, and ii) offers more stability compared to Reinforcement Learning, we observe in experiments (see section 6 Note also that, contrary to RI. approaches which have to find a good balance between discriminator and generator capacities, our approach does not suffer from Vanishing Gradient (1], since discrimination is only used for decoding, in cooperative process for generator training. We depict the SelfGAN Algorithm
Decoding Mechanisms
4.1 Standard Practices: Generator-only
At decoding time, two different approaches are commonly used in NLG: Sampling and Beam Search. They respectively correspond two different objectives.
Sampling To obtain diverse outputs, it is common to sample tokens from the model distribution. In particular, this is mandatory when there is input at all, i... Unconditional NLG, for instance GPT [24]. However, the longer the sequence, the more likely to sample a token from the tail of the distribution, causing degeneration [13]. To mitigate this issue, common practices are to lower the Softmax Temperature and keeping only the Top K tokens 10 the Top P probability mass [3]
Beam Search is the standard algorithm to approximate the sequence maximising the output probabil- ity, by maintaining K candidates each step. Its usage suits better conditional NLG tasks, where the diversity arises from the variety of conditioners inputs, e.g. in Summarization.
4.2 Cooperative Decoding: Combining a Discriminator and Generator
Subject exposure bias, neither Sampling Beam Search satisfying: the outputs produced easily identified by discriminator (11, indicating that they differ from human written text. In light of this, two concurrent works have recently proposed to use the discriminator during the decoding.
DAS Reranking Step By Step In Discriminative Adversarial Search [31 a discriminator re-ranks the sub-sequence candidates at each decoding step of Beam Search, in order to favor human-like outputs.
DAS gioall Reranking Complete Sequences In a concurrent work 7 a very similar cooperative method is proposed: this time, N complete sequences are sampled from the auto-regressive model. The N sequences are scored by discriminator, allowing to select the one with the highest probability to be human-like. Since the discriminator re-ranking is computed on a complete sequence, we refer to this method as DASglobal- opposed to DAS

---

4.3 Coop-MCTS: Cooperative Decoding beyond the Ro-ghht Curse
can happen that all sequence candidates are judged by the discriminator be machine- like rather than human-like like. In such case, the cooperative decoding is stuck in dead end; such limitation is unsatisfactory. Neither DAStecal or DASglobal have the ability to revise their previous decisions.
To cope with those limitations of myopic decoding strategies, we propose to consider an adaptation of MCTS for NLG. Just like in the context of games [36], we consider a policy network x, the generator, that outputs probability over all the possible actions (tokens) at each step of the sequence. The discriminator D corresponds the value network. In MCTS, the trajectories are explored build tree following three steps:
1. Selection starting from the root, children nodes tokens w are selected among the vocabulary V recursively w.r.t. the PUCT algorithm 28 36
No 1+ (s,
((s,w) + Cpuct (w
3 arg max WEV
where Q is the value of taking action in state in NLG, this corresponds selecting token among the vocabulary at step given the source context and the sub-sequence wo, ..., Wi- Cpuct constant, the temperature that scales the Softmax, and (s,ww) the number of times the token w has been chosen in state s. We stop the loop when a node 8, has not been expanded yet, i.e. the discriminator D has not calculated its value.
Extension Given the selected node, calculate the distribution probability from the generator (w 8.). We apply nucleus sampling [13 to filter out the less likely tokens and reduce the number of actions. The remaining tokens constitute the children nodes, associated their corresponding probability. At the same time, we calculate the value of the current state ((*)) that allows to compute the backup step.
3. Backup we update Q for all the nodes that led to 8, such that Q max (Q, D (s.)). Note that we choose to use the max instead of the average for the following reason: the value network, i.e. discriminator, becomes
accurate the candidate sequence grows (see Figure in (II,,
hence if a long sequence is judged human by the discriminator, any of its sub-sequences should be considered human-like as well. In contrast, long sequence be machine-like despite starting in very human-like manner: the beginning sub-sequence should keep its human-like score.
These three steps are computed for a restricted number of simulations. Then, the next token corre- sponds to the root child with the most visit counts. The process continues step by step to generate the next token, until reaching either the special token End Of Sentence, or the maximum length.
5 Experimental Details
5.1 Datasets
To measure the effectiveness of SelfGAN, experiment two standard conditional NLG tasks: Question Generation (QG) and Summarization, consistently with previous works [8] [0]:
Question Generation:
we used the SQuAD dataset 226, consisting of 100K triplets of Wikipedia paragraphs, factual questions, and their answers.
Summarization: we used the CNN/Daily Mail dataset (CNNDM) 20). consisting of 300K news articles, paired with their corresponding summaries. The summaries are formed of multiple sentences, making the amount of tokens to generate much larger than for Question Generation.
5.2 Models Reported
MLE the first baseline we consider is a standard model trained via teacher forcing. As for all our experiments, we initialised the seq2seq with T5 225. as detailed in Section 5.4
ColdGAN we consider as second baseline the current tate-offthe art for language GANs, ColdGAN 30). The authors proposed to lower the temperature when sampling the sequences during training. with the objective of stabilizing the training process.
5

---

SelfGAN can be based on any cooperative decoding algorithm. To train SelfGAN, we there- fore experiment the three different cooperative algorithms described in Section (DAS DAScloall- and CooP-MCTS) and report the results for the corresponding SelfGAN: Selj SelfGANpAs Global, and SelfGANccop.MCTT.. MCTS-
Decoding Method at inference time For each model, any decoding method can be applied at inference time, independently from the training scheme. Therefore, for all the models described above, we report the results given each decoding method previously described (Section : Beam Search, LoSat DASotobal, and Coop-MCTS.
To the best of our knowledge, GANs and Cooperative decoding have never been directly compared before this work. A fortiori, this is the first time that GAN model is tested with Cooperative decoding method at inference. We investigate possible distillation effects in Section 6
5.3 Metrics
To compare the different models, we report two type of metrics: n-gram based and discriminator.
N-gram based We report the standard BLEU [22 and ROUGE [18]. Both measure an overlap of n-grams between the reference and the evaluated text. They differ in that BLEU is precision oriented while ROUGE is rather recall oriented.
Discriminators Both BLEU and ROUGE suffer from the aforementioned limitations. We therefore propose to consider discriminators for model evaluation. Intuitively, they measure how model outputs are similar to what a human would have written. We consider two different discriminators:
Base is a discriminator trained on the MLE baseline outputs generated via beam search. It allows to measure the corresponding improvement from the MLE baseline. Note that corresponds to the initial discriminator in all the GANs experiments, and the discriminator used in the cooperative search for the MLE baseline.
Base+ Since the Base discriminator plays a role in all our experiments (except MLE+Beam Search), it is possible that model that makes use of this Base obtains better Base results, despite bringing new biases and de-generation behaviors. For this reason, we also report Base+, discriminator fine-tuned on all the different model outputs together. Base+ never used by any model training or inference time. It is thus more robust toward an undesirable adversarial generation mode, while still being comparable for the different experiments. We argue that a higher Base+ score indicates a real improvement beyond potential bias.
5.4 Implementation Details
For all experiments, we used the T5-small [25] architecture Using 4 Nvidia V100 SXM2 GPUs. SelfGANccoopMCT training and evaluation takes respectively 26 hours and 1 hour on CNN/DM; and 0.5 hours on SQuAD. Compared to 2 (0.5) hours to train via MLE on CNN/DM (SQuAD), we identify the computational cost the main limitation of our work.
6 Results and discussion
6.1 Conditional Text Generation
In Table we report the results for all the previously trained generators, with the different decoding algorithms presented in Section 4 By 'modell, in the following, we refer the couple composed by a trained generator and a decoding algorithm.
We report BLEU4, ROUGE-1, ROUGE-L along with scores for the discriminators Base and Base+, computed as the percentage of outputs considered as human by given discriminator model. Base was only trained on MLE+Beam Search outputs. As expected, by further training on the outputs generated by all the different models, Base+ has a higher accuracy, which consistently results in lower scores compared to Base
? As implemented in HuggingFace transformers 38

---

Generator
Question Generation
Summarization
Decoder
B4
R1
RL
Base
Base+
B4
RL
Base
Base+
MLE
BeamSearch
25
19,7
45,2
41,1
15%
15%
15,9
42,3
40,4
9%
DAStecal
31
19,9
45,2
41,1
28%
19%
16,6
43,8
40,9
17%
11%
DASgiobal
20,0
45.2
41,2
20%
17%
16,2
44,1
41,9
12%
9%
Coop-MCTS
19.8
45,3
41,5
33%
21%
16,3
42,5
40,6
20%
12%
ColdGAN
BeamSearch
30
19.9
45.2
41,4
26%
17.9%
16,3
42,8
40,7
15%
10%
DAStscal
19,8
45,3
41,1
31%
20%
15.9
42,5
42,0
19%
DASptohul
20,2
45,6
41.5
26%
18%
16,6
44,6
41,2
16%
10%
CooP-MCTS
19,9
45,4
41,2
39%
22%
15,9
44,2
41,2
23%
12%
SelIGANDAS
BeamSearch
20,2
45,4
41,6
27%
21%
16,9
44,2
42,5
16%
11%
DASiccal
20,5
45,5
41,7
30%
23%
16,9
44,4
41,9
18%
13%
DASgtabal
20,1
45,4
41,7
33%
20%
16,6
44,0
42,3
19%
11%
Coop-MCTS
20,4
45,5
41,8
39%
23%
16,4
43,8
42,8
23%
13%
SelfGANDAS
BeamSearch
20,4
45,5
41,7
24%
19%
16.9
43,0
41.5
14%
DAStocal
19.9
45,4
41,3
32%
22%
15,9
42,7
40,6
18%
12%
DASglobal
20,7
45,6
41,9
29%
20%
17,0
43,7
42,6
17%
11%
Coop-MCTS
20,0
45,3
41,4
40%
24%
16,1
43,4
42,3
23%
13%
SelfGANcoopp.
BeamSearch
20,5
46,6
42,6
34%
21%
17,0
42,8
41,5
20%
13%
DAStscal
20.6
46.7
41.7
42%
24%
16,6
43.7
42.8
25%
DASglobal
20,5
46,6
41,7
39%
21%
16,5
42,8
40,9
23%
Coop-MCTS 21,1
48,9
44,7
40%
26 %
17,5
43,5
42,3
23%
15%
Table Results of our experiments on QG (left) and Summarization (right). For each generator, we report the results with the four different decoders. The reported metrics correspond to BLEU4 (B4). ROUGE-I (R1), ROUGE-L (RL) and the discriminators Base and Base+ described in Section For Base and Base+ the scores correspond to the probability of being human, so higher is better for all the metrics. For MefGGANMCTS, we experimented with different seeds and the standard deviation is always inferior to 0.1 for BLEU4 and ROUGE, and inferior to 0.5% for Base and Base+.
We start by focusing on the MLE results to compare the different decoding mechanisms. We observe that all the cooperative searches outperform Beam Search. Regarding Base and Base+ metrics, DAS compares favorably to DASciatal- We hypothesize that invoking the discriminator to rank at each step have more impact than using it only once fully decoded sequences. Finally, our proposed Coop-MCTS obtains the best results by a large margin.
Regarding the different GANs, we first compare them given the default decoding mechanism, i.c. Beam Search. The three versions of SelfGAN compare favorably to MLE and ColdGAN on both n-gram based metrics and discriminators metrics. Among SelfGANs. SelfGAN Coop-MCSS obtains the best results: given a Beam Search decoding, obtains the best BLEU, ROUGE-1 and ROUGE-L on the two tasks (respectively 7:22; 44.3; 40.6 on QG and 12.3; 38.6; 36.7 on Summarization), The performance term Base and Base+ for SelfGAN CeoGANNCoop-MCC MCTS even more important comparison to the other models (34.1 %; 21.9% on QG and 20.2%; 12.7% on Summarization)
Both GAN training time and Cooperative decoding inference time pursue the same objective: to obtain better outputs that look like human texts. Would generator trained via GAN, coupled with Cooperative Decoding mechanism for inference result into cumulative improvement from the two methods? First, on both ColdGAN and three SelfGANs, we can observe that adding a Cooperative Decoding method allows to gain significant improvement on Base and Baset. In particular, is interesting to note that for SelfGAN an additional pattern seems to emerge: using the same cooperative decoding algorithm both during training and inference seems to provide additional gains. The best performance is achieved with the generator SelSGAN Cwop-MSSs MCTS paired with the decoding Coop-MCTS. Compared to MLE via Beam Search, it obtains final improvement superior to point term of ROUGE and BLEU. The relative improvement for Base+ is significant: from 15. 15.2% to 26.2%

---

Model
T=0.5
T=1
T=2
MLE+Sample ColdGAN+Sample MLE+CCoppMMCTS
0.42:0.29
0.31;0.11
0.18:0.07
0.47:0.21
0.33;0.08
0.22;0.06
0,45:0.22
0.34:0.10
0.21;0.06
BelIGANNCop-MCTS--Coo--
0.48:0.20
0.37:0.09
0.24:0.05
Table 2: Results on Unconditional Text Generation for samples realized at three different temperatures, terms of BLEU Vs Self-BLEU (higher better,lower better).
Sunli Baundarch
0.0
Capson + deserioucc
0.0
Se-faMMoe NOT
HOW
Traniing
Tranning Stop
Tra
Stop
Figure 1: Average difference for Summarization between human references and model outputs for the Length (Left), the Novelty (Middle), and the 3-grams repetitions (Right) during training. The closer to 0 the less differences w.rt. gold-references.
QG and from 8.6% to 15.3% Summarization. This corresponds to almost twice outputs that sound human according to the Discriminator metric.
6.2 Unconditional Text Generation
We follow the ColdGAN setup: we compared our proposed approaches on the EMNLP2017 News dataset. The evaluation takes into account both the quality and the diversity. Consistently with previous works (e.g. ColdGAN, ScratchGAN, LeakGAN), we use the following metrics:i) BLEU-5 for measuring the quality (higher better); ii) Self-BLEU-5 for measuring the diversity (lower better).
To obtain a finer comparison between models, Caccia et al. 3 proposed to draw the curve of BLEU vs self-Blle BLEU, by sampling with various temperatures at inference.
We denote as the standard method to generate text this setup, as used all the previous works we are comparing to. It is simple left to right decoding where, at each step, token is sampled among the Softmax probabilities scaled by the temperature.
In our Coop-MCTS, the probability of token is given by its visit counts during the simulations. In Conditional generation, select each step the token with the maximum number of counts. In Unconditional generation, we sample from the tokens counts distribution.
Overall, the results consistent with the experiments on Conditional Generation: the MLE generator decoded with our proposed MCTS (3rd row) obtains:
significantly better slightly lower results than ColdGAN decoded with Sample(2nd row)
2. results than when the same MLE decoded with Sample( 1st row);
3. SelfGAN decoded with MCTS (4th row) obtains the best results.
6.3 Discussion
Human-like features during training In NLG, various rules are often integrated into the Beam Search to improve the quality of the outputs, for instance a length penalty [34] or an interdiction for 3-grams repetitions [23 [) Such need hard code these rules indicates diserepancy between the human output characteristics and what the model has learned. In particular, Scialom et al. [31 reported the difference between DAS and the human reference for: i) Length: the average number of tokens per output; ii) Novelty: percentage of tokens in the output that were not present in the source text; iii) N-gram repetition: percentage of N-grams that occur more than once in the output. To measure how SelfGAN learns these features by itself, we report in Figure I the evolution of these
8

---

Generator
Decoder
Consistency
Coherence
Fluency
Relevance
MLE
BeamSearch
3.9
3.1
4.1
3.2
MLE
Coop-MCTS
3,4**
3,5**
3.8
3,6**
ColdGan
BeamSearch
3.8
3.3
4.2
3.5
ColdGan
Coop-MCTS
3,4
3,6**
4.0
3,7
SelfGAN CooS
BeamSearch
4.0
3,5**
4.3
3,9
SelfGANciop-MerT
Coop-MCTS
3.9
3,98
4.0
4.2*
Table Human Evaluation on Summarization, Two tailed t-test results are reported for each model
compared to B..EBeeaaSe (*: .01, **: p-pp.. .00)..
statistics during training: we observe that SelfGAN constantly reach statistics more similar to human references than ColdGAN.
Human Evaluation We conduct human evaluation to measure the models performances beyond au- tomatic metrics. We limit the evaluation to three generators (MLE, ColdGAN, and Sel/GANcoopMCTT) and two decoding methods (Beam Search and Coop-MCTS), for a total of different models. Three professional English speakers rated 300 sampled summaries and followed the same protocol from Fabbri et al. 19.. Four dimensions are evaluated on a Likert scale from to 5 (the higher the better):
1. Consistency: the proportion of facts in the summary correct w.r.t. the source text;
2. Coherence: how well-structured and well-organized is the summary;
3. Fluency: how fluent the summary is to read;
4. Relevance: the ratio between important and excess information in the summary.
From Table 3 observe significantly better results for BelfGANcoup- w.r.t both MLE and ColdGAN. While Coop-MCTS decoding appears overall beneficial in terms of Coherence and Relevance, but scores lower Consistency and Fluency, its combination with SelfGAN CoopGANCoop-MeTs allows to obtain significant improvements on the former two dimensions while still maintaining comparable scores the latter.
Analysis To further understand the benefits selfGAN, we propose to analyze the evolution of the first plot the
generator and discriminator networks through the learning process. In (leff, average magnitude (L2 norm) of the discriminator gradients w.r.t. its parameters. We observe that CN induces important instabilities for its discriminator gradient magnitude. Conversely, thanks to its cooperative decoding process, SelfGAN produces sequences that form more compact set for discriminator training, variance of gradient magnitude twice lower than ColdG for a comparable magnitude in average. This discriminator stability is first explanation for the improvements of the proposed approach.
time, with a highly fluctuating
second plot, given on the right of Figure 2 we report the collinearity of generator gradients for the generated samples from the model with those for the corresponding human references. Higher values indicate sampling strategies that induce a useful gradient flow for the generator. For ablation purposes, BeamSeurch approach, where we used a standard Beam Search to generate the training examples: note that it has no discriminator, hence it is not a GAN anymore. We can observe its divergence, as opposed to SelfGAN coop- which emphasizes the importance of the cooperative decoding for producing the example used to train the model. For SelfGANcooo.Mer and ColdGAN, the gradients become more co- co-linear with human references through time, indicating a convergence of the process towards the human distribution. We observe that SelfGAN produces more useful sequences for achieving this convergence.
we first report values for "SelfGAN BelfGGANeemmSerrch
Coop-MCTS as an altermative to the dead-end search When analysing the behavior for the Coop- MCTS decoding, observed in different examples that it provides effective mean to revise generations that eventually ended up to be unlikely. To illustrate this, we report in Table 6.3 the different MCTS steps for ambiguous example: the conditioned answer, Super Bowl, occurs at different places of the the input. Therefore, the model has to decide which specific mention of Super Bowl to focus on: at step 17, it considers its current generation new node (How). The final output is a question that arguably sounds better than the initial one.
dead end and decides to start
Societal Impact Reliable NLG models can have significant societal impact with beneficial appli- cations such as efficient information access automatic summarization or personalized student evaluation through question generation. Still, malicious actors can the same technology to build

---

SeNCA AGAN
64
Calimartty
33
63019
4200
2019
4000
CBick
100K
2004
4000
0000
300
Tralning
Traninn Stop
Figure 2: Left: Moving Average of the magnitude of the discriminators gradients during training. Right: collinearity of the generators gradients between the sampled texts and their corresponding human reference for CelfGANcoo--MTTS. MCTS ColdGAN and SelfGAN BeamSearch- Both on Summarization.
Conditioned Answer: Super Bowl
Context:
Super Bowl wess Americall football game determine the Ehampion of the National Football League NN for the 2015 season. The American Football Conference (AFC) chempion Denver Broncos defeated the National Footbal Conference [NFC) champlon Carolina Panthers 2456'10 10 to eam their third Super Bowl title. The game was played on February 2016, Lev's Stadium the San Francisco Bay Area Santa Clara, Califorii, As this was the 50th Super Bowl, the league various gold poid-themed initiatives, naming each Super Bovi game with Roman numerals (under which the game would have been knowi the logo prominemily feature Arabic numerals
emphasized "gokden anniversary"
temporarily suspending tradition
Supee Bowl L*) s0
Step 01: What
Step 16: What was the name of the game that would have been known as "Super Bowl
Step 17: How
Step 46: How is called the American foothall game that determines the NFL champion?
Table Progressive results obtained by our Coop-MCTS decoding method on Question Generation during a simulation. Until the 16th step, the generation is left-to-right. Then, the cooperation mechanism kicks in, allowing the model to safely abort this beam, by restarting new question with How. We report the cross-attention weights the input context for step 16 (red) and 17 (bluc).
tools detrimental to socicty, c.g. large scale creation of misleading (fake) news 24].. As argued by Zellers al. 43
keeping this research open and under public scrutiny can be effective defense.
7 Conclusion
In this paper we propose SelfGAN, a new framework to train Generative Adversarial Networks based on cooperative decoding search. To overcome the left-to-right curse that limits standard search algorithms, we propose Coop-MCTS. We conducted extensive experiments on two challenging tasks: Summarization and Question Generation, obtaining state-of-the-a art performance for SelfGAN both in terms of automatic metrics and within a human evaluation. As the stability of the discriminator looks to be crucial for language GANs, plan for future works to still focus increasing it through the definition of dynamic regularization mechanisms. Finally, we will explore how reference-less metrics, e.g. 32 be combined to help the exploration during the decoding.
8 Acknowledgments
This work was partially performed using HPC resources from GENCI-IDRIS (Grant 2021- AD011012318).
References
[1] Arjovsky, M. and Bottou, L. Towards principled methods for training generative adversarial networks. arXiv preprint arXiv:1701.04862. 2017.
[2] Bengio, S., Vinyals, O.,, Jaitly, N., and Shazeer, N. Scheduled sampling for sequence prediction with recurrent neural networks. Advances Neural Information Processing Systems, pp- 1171-1179, 2015.
10

---

[3] Caccia, M., Caccia, L., Fedus, W., Larochelle, H., Pineau, J., and Charlin, L. Language gans falling short. In International Conference on Learning Representations, 2020. URL hnttps://operrewe
[4] Che, T., Li, Y., Zhang, R., Hjelm, R. D., Li, W., Song, Y., and Bengio, Y. Maxiimum-likelihdd augmented discrete generative adversarial networks. arXiv preprint arXiv:/702.07980 2017.
[5] Coulom, R. Efficient selectivity and backup operators in conte-carlo tree search. In Interna- lional conference on computers and games, pp. 72-83. Springer, 2006.
[6] de Masson d' Autume, C., Mohamed, S., Rosca, M., and Rae, J. Training language gans from scratch. In Advances in Neural Information Processing Systems, pp. 4302 4313, 2019.
[7] Deng, Y., Bakhtin, A., Ott, M., Szlam, A., and Ranzato, M. Residual energy- energy-based models for text generation. arXiv preprint arXiv:2004. 1714, 2020.
[8] Dong, L., Yang, N., Wang, W., Wei, F., Liu, X., Wang, Y., Gao, J., Zhou, M., and Hon, H.-W. Unified language model pre-training for natural language understanding and generation. In Advances in Neural Information Processing Systems, pp. 13044,13054, 2019.
[9] Fabbri, A. R., Kryšciński, W., McCann, B., Xiong, C., Socher, R., and Radev, D. Summeval: Re-evaluating summarization evaluation. arXiv preprint arXiv:2007 2020.
[10] Fan, A., Lewis, M., and Dauphin, Y. Hierarchical neural story generation. arXiv preprint arXiv:/805.0483 048333 2018.
[11] Gabriel, S., Bosselut, A., Holtzman, A., Lo, K., Çelikyilmaz, A., and Choi, Y. Cooperative generator-discriminatoo networks for abstractive summarization with narrative flow. 2019.
[12] Hinton, G., Vinyals. O., and Dean, J. Distilling the knowledge in a neural network. arXiv 2015.
preprint arXivv/503.000
[13] Holtzman, A., Buys, J., Du, L., Forbes, M. and Choi, Y. The curious case of neural text degeneration. arXiv preprint arXiv:1904.00 09751, 2019.
[14] Koesis, L.. and Szepesvári, C. Bandit based monte-carlo planning. In European conference on machine learning, pp. 282-293. Springer, 2006.
[15] Kumagai, K., Kobayashi, I., Mochihashi, D., Asoh, H., Nakamura, T., and Nagai, T. Human- like natural language generation using monte carlo tree search. In Proceedings of the INLG 2016 Workshop on Computational Creativity in Natural Language Generation, pp. 11-18, 2016.
[16] Lamb, A., Goyal, A., Zhang, Y., Zhang, S., Courville, A., and Bengio, Y. Professor forcing: A new algorithm for training recurrent networks. arXiv preprint arXiv: 1610.09038, 2016.
[17] J., Monroe, W. Shi, T., Jean, S., Ritter, A., and Jurafsky, D. Adversarial learning for neural dialogue generation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2157-2169, 2017.
[18) Lin, C.-Y. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
[19] Mukherjee, S. An unsupervised approach to automatic response generation for conversational e-commerce agents using monte carlo tree search. 2019.
[20] Nallapati, R., Zhou, B., Gulcehre, C., Xiang, B., et al. Abstractive text summarization using pruuence-to-sequenee rnns and beyond. arXiv preprint arXiv: 060220000,, 2016.
[21] Novikova, J., Dušek, O., Cercas Curry, A., and Rieser, V. Why we need new evaluation metrics for NLG. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2241-2252, Copenhagen, Denmark, September 2017. Association for Computational Linguisties. doi: 10.1 1 1238. URL https:tpps.... www. aclweb aclveb.org/ anthology/D17-123

---

[22] Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pp. 311-318. Association for Computational Linguistics, 2002.
[23] Paulus, R., Xiong, C., and Socher, R. A deep reinforced model for abstractive summarization. arXiv preprint arXiv: /005. 00044444444 2017.
[24] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, Language models are unsupervised multitask learners. OpenAl Blog, 188):, 2019.
[25] Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, Exploring the limits of transfer learning with unified text-to-text transformer. arXiv preprint arXiv: 10683, 2019.
[26] Rajpurkar, P., Zhang. J., Lopyrev, K., and Liang, P. Squad: 100,000+ questions for machine comprehension of text. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392 2016.
[27] Ranzato, M., Chopra, S., Auli, M., and Zaremba, W. Sequence level training with recurrent neural networks. arXiv preprint
2015.
[28] Rosin, C. D. Multi-armed bandits with episode context. Annals of Mathematics and Artificial Intelligence, 6133)2003-300 2011.
[29] Schrittwieser, J.. Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., et al. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588(7839):604
2020.
[30] Scialom, T., Dray, P.-A., Lamprier, S., Piwowarski, B., and Staiano, J. Coldgans: Taming language gans with cautious sampling strategies. Advances in Neurul Information Processing Systems, 2020.
[31] Scialom, T., Dray, P.-A., Lamprier, S., Piwowarski, B., and Staiano, J. Discriminative adversarial search for abstractive summarization. arXiv preprint 10X1v:2002.10375, 2020.
[32] Scialom, T., Dray, P.-A., Gallinari, P., Lamprier, S., Piwowarski, B., Staiano, J., and Wang, A. Questeval: Summarization asks for fact-based evaluation. arXiv preprint arXiv:2103. /2693, 2021.
[33] Scudder, H. Probability of error of some adaptive pattern-recognition machines. IEEE Transac- lions on Information Theory, 11(3):363-37, 1965.
[34] See, A., Liu, P. J., and Manning, C. D. Get to the point: Summarization with pointer-generator networks. arXiv preprint arXiv: 04368, 2017.
[35] Semeniuta, S., Severyn, A., and Gelly, S. On accurate evaluation of gans for language generation. arXiv preprint 1806.000660466 2018
[36] Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, I.., Lai, M., Bolton, A., et al. Mastering the game of go without human knowledge. nature, 550(7676):354-333
2017.
[37] Williams, R. J. and Zipser, D. A learning algorithm for continually running fully recurrent neural networks. Neural computation, 1(2):270-280, 1989.
[38] Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T. Louf, R., Funtowicz, M., et al. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint 191000377 2019.
[39] Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q. Macherey, K., et al. Google's 's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint 16Xiv:1609.08144 2016.
[40] Yarowsky, D. Unsupervised word sense disambiguation rivaling supervised methods. In 33rd annual meeting of the association for computational linguistics, pp. 189-196, 1995.
12

---

[41] Yu, L., Zhang, W., Wang, J., and SeqGAN, Y. Y. Sequence generative adversarial nets with policy gradient, arxiv e-prints, page. arXiv preprint 1K9:056699 2016.
[42] Yu, L., Zhang, W., Wang. J., and Yu, Y. S. Sequence generative adversarial nets with policy gradient. 489 in. In AAAI conference artificial intelligence, volume 490, 2017.
[43] Zellers, R., Holtzman, A., Rashkin, H.,, Bisk, Y., Farhadi, A., Roesner, F., and Choi, Y. Defending against neural fake news. arXiv preprint arXiv: /26/6, 2019.
[44] Zhang, Z. and Sabuncu, M. Self-distillation as instance-specific label smoothing. Advances in Neural Information Processing Systems, 33, 2020.
[45] Zhou, W., Ge. T., Xu, K., Wei, F., and Zhou. M. Self celf-adversarial learning with comparative discrimination for text generation. In International Conference on Learning Representations, 2020. URL.tttps http://peeeeeeeeeeeee///r//ee/aa
13


Reviews:
[{"review_id": "ad4or7b-4GA", "reviewer": "Reviewer_TEo2", "summary": "This paper presents a method of training (or rather fine-tuning) text generation models using a GAN-like setup. Instead of using the discriminator directly to update the generator as is usually the case, the authors propose using the discriminator as a way of cooperative decoding to generate high-quality samples which are then used as the gold output to train the model in the usual teacher-forcing setup. The authors explore a sampling procedure based on MCTS to generate more diverse sentences rather than using beam-search or sampling during training and inference. On summarization and question generation, they show improvements on the standard and proposed evaluation metrics. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. Language GANs have been shown to be difficult to train due to sparse signals from the discriminator. This paper presents an interesting idea to incorporate this signal in a more stable manner. \n2. MCTS based decoding algorithm also has potential applications even for cases where the proposed training method was not used. \n3. The paper is clearly written and easy to follow for the most part. \n\nWeaknesses and comments:\n1. The paper presents the method as a language GAN yet doesn't evaluate it on the most common task used in this area: unconditional text generation where the goal is to generate high-quality diverse text where the benefits of the proposed decoding method can be more clearly evaluated. \n2. Relating to my previous comment, a diversity measure should be added to the evaluation. Plus the quality metrics show very small improvements in most cases over baselines.  \n3.  The proposed decoding process is quite slow for the improvement it offers. \n\n\n------------------------\n\nUpdate: The author response addresses most of my concerns, I am updating my score to 7. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "LD1R0uSCDt3", "reviewer": "Reviewer_oytt", "summary": "This paper attempts to address the difficulties of adversarial training for language generation, as discrete generator outputs make conventional GANs difficult to train.\n\nThe authors propose combining cooperative decoding proposed by Gabriel et al (2019), along with MCTS-based decoding, to train a discriminator and generator jointly. The generator outputs are trained with the results of their MCTS-based cooperative decoding procedure.\n\nThe authors demonstrate their method's performance on summarization (CNN and Daily Mail dataset) and question generation (SQuAD).\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, this work demonstrates a novel approach to integrating cooperative training, and is a useful contribution. The method is promising, and could be extended and applied to other discrete generation applications as well.\n\nThe comments that follow are mostly related to experiments and the paper's presentation.\n\nRegarding experiments, as the authors point out, the \"Base\" discriminator metric uses the discriminator trained on MLE outputs generated by beam search. However, it's used both as a metric and for training, so the authors use a \"Base+\" discriminator that is further fine tuned on the final model outputs. One potential concern is that the fine tuning process may result in the finetuned model still being too similar to Base, and unable to find discriminating factors that would be found had the model been trained from scratch. Have the authors considered training a discriminator from scratch, and if so, how does it perform? Performing this experiment is not a requirement for my review: as generated natural language is difficult to evaluate automatically, human evaluations should still take precedence.\n\nTo clarify, is the generator pretrained with standard MLE before training SelfGAN/Coop-MCTS, or is the latter trained from scratch? Or is only the discriminator pretrained? How stable is the SelfGAN training?\n\nOverall, the paper may be somewhat clearer with more equations to precisely and concisely describe the method, and/or some minor reorganization. Algorithm 1 is somewhat brief, and the overall method's description is split between Section 3 and the MCTS description in Section 4, which due to the author's setup, are also important for describing how the generator is trained.\n\n*Related work:*\n\nThe authors cite the cooperative decoding work of Gabriel et al from 2019, but it appears they have a later version in EACL 2021, see https://arxiv.org/abs/1907.01272. This should be updated accordingly.\n\nAs the authors cite other applications of MCTS in NLP, the authors should also consider citing related MCTS decoding work in neural machine translation, such as https://arxiv.org/abs/2004.12527 and https://arxiv.org/abs/2104.05336.\n\n*Clarifications:*\n\n- On line 309, the sentence \"SelfGAN produces sequences that form a more compact set for discriminator training …\" is unclear, what does this refer to?\n- On line 319, can you clarify how the metric for \"colinearity of generator gradients for the generated samples from the model with those for the corresponding human references\" is computed?\n- Last sentence of the conclusion is awkward/unclear, what does \"increasing it\" refer to?\n\n*Minor comments:*\n\n- L111: add a caption for Algorithm 1, and reference it in the main text\n- L124 and alg. 1 L5: inconsistent naming between \"standard training\" and \"classical maximization\", perhaps say something like \"standard maximum likelihood with S_coop as the target\"\n- L124: in the first equation, shouldn't $\\sum_{...} \\pi_{\\theta}(...)$ be $\\sum_{...} \\log \\pi_{\\theta}(...)$?\n- L178: 'a in' -> 'in'\n- L262: version -> versions\n- Table 1 should explain \"B4\", \"R1\", and \"RL\" in the caption (e.g. BLEU4 (\"B4\"))\n- L323: mean -> means?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately discuss limitations and societal impact in their conclusions.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "Hwm4xGFffUo", "reviewer": "Reviewer_sMqm", "summary": "To train language GANs, previous approaches use the discriminator score as a reward, and train the generator through RL. Instead, this paper proposes to use the discriminator to guide the decoding process, searching for outputs with high discriminator scores, and then train the generator to generate these outputs. To find outputs with high discriminator scores, the authors employ Monte Carlo Tree Search and use the discriminator as the value network. Experiments include question generation and summarization, and the proposed method shows improvements to MLE and ColdGAN.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While the proposed model looks fine, I think the experiments and baselines are lacking, and most importantly, the results do not match those reported in previous work. I hope the authors can validly address my concerns.\n\nThe results of MLE and ColdGAN in Table 1 are overall much worse than those reported in the ColdGAN paper. Just to give an example, T5-small (MLE) has BLUE-4 19.65 on question generation (see Table 2 in the ColdGAN paper), while this paper reports 16.5. Can the authors please explain the huge discrepancy?\n\nIn addition, the models are not tested on unconditional generation. We know that mode collapse is a serious problem of GANs. It's thus unclear whether the proposed method can generate diverse outputs in unconditional scenarios.\n\nFor cooperative decoding, the proposed MCTS approach is compared against a reranking baseline. But there're also other methods to use the classifier to guide the generation process. For example, one can combine the generator probabilities and the discriminator probabilities to sample the next word [1]. Can the authors compare with this method as well?\n\nIn Fig 2 Right, the authors report the collinearity of generator gradients for generated samples with those for human references. However, it's unclear to me why the higher the better. If so, why not just train with human references using MLE, which has a collinearity of 1?\n\nFinally, the writing is sometimes obscure (e.g., \"the sampling mechanism of sequences that have to be favored through MLE\", \"for each condition in the training set X\"). Please revise them to avoid confusion. \n\n[1] Yang and Klein. FUDGE: Controlled Text Generation With Future Discriminators. NAACL 2021.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have discussed the societal impact of natural language generation in the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5h"}, "extra_text": {}}, {"review_id": "Dq-HgW5Kvng", "reviewer": "Reviewer_iqLB", "summary": "This work proposes selfGAN. Instead of using RL techniques to utilize the discriminator. They do Coop-MCTS which is termed as a \"cooperative decoding\", which is designed to generate sequences that is better than naïve sampling or decoding. And the coop-decoded sequence is then used to train the generator in turn, in the standard teacher-forcing way. I understand this approach as the process in which the discriminator is helping the generator to refine its generations. Better results comparing to MLE and ColdGAN are shown in the experiments.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength of this work: (1) The idea is novel and interesting (it reminds me of self training). (2) The results are good and the comparison protocols are principled. \n\nWeakness: (1) There are a number of text GAN works that claim to outperform the MLE baseline, therefore only comparing with ColdGAN does not seem to be enough. (2) This paper is well written in most parts, however, I find it hard to understand the Coop-MCTS section.\n\nIn addition, I have several questions:\n\nLine 27: \"However, NLG metrics are known to not reflect well human judgement [21], which explains why the resulting models tend to be qualitatively worse than their MLE baselines [3]. \" While that maybe true, it's also possible that the RL methods is simply hard to train well, and the NLG metrics is not to blame.\n\nI did not quite understand the Coop-MCTS algorithm (I admit that I'm not familiar with PUCT). How is it able to revise a sequence? What's the inutition behind Eq.1? I hope you could rewrite this section when you revise the paper.\n\nIs selfGAN also initialized by T5? I'm assuming it is, could the training algorithm work if it is trained from scratch?\n\nIt would be good to discuss a bit about [1], I know the two are doing different things. But the naming could be confusing.\n\nFigure 2 left is broken?\n\n[1] CoT: Cooperative Training for Generative Modeling of Discrete Data  https://arxiv.org/abs/1804.03782\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Did you mention how fast is MCTS comparing to beam-search?", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work demonstrates the effectiveness of a novel and interesting approach to utilising GANs with cooperative training and MCTS. The paper is well written and easy to follow. Though the initial results in the paper lacked evaluation on an important task (unconditional text generation) and didn't evaluate the diversity of generated text, the strong rebuttal resolves these issues and gives convincing evidence of the effectiveness of SelfGAN and MCTS. Hopefully this paper inspires more work in this interesting direction.
Decision: Accept (Poster)