To Beam Or Not To Beam: That is a Question of Cooperation for Language GANs

1 Introduction . Natural Language Generation encompasses tasks such as Machine Translation , Summarization or Data To Text generation . The real life applications are numerous , but require highly reliable and fluent models . Despite significant advances , state-of-the-art models are still known to be de-generated , with outputs containing repetitions and even nonfactual information i.e . hallucination [ 13 ] . Among the culprits is a limitation of Teacher Forcing [ 37 ] : the loss is computed at a token level while the aim is to produce complete sequences . Moreover , while a single ground-truth reference is considered correct , several realizations of the same content may exist . Finally , the model is subject to Exposure Bias [ 27 ] , i.e . a mismatch between training and inference distributions – in the latter , the model has no access to ground truth for the previously generated tokens . The literature has considered this mismatch responsible for the lower quality observed when generating longer sequences [ 2 , 16 ] . To overcome such Teacher Forcing limitations , a consensus has emerged : a sequence level objective should be introduced [ 27 , 41 ] . A body of work has proposed to use Reinforcement Learning ( RL ) with standard NLG metrics like BLEU [ 39 ] or ROUGE [ 23 ] . However , NLG metrics are known to not reflect well human judgement [ 21 ] , which explains why the resulting models tend to be qualitatively worse than their MLE baselines [ 3 ] . To move toward less biased metrics , a natural alternative is to evaluate the output with a learned discriminator . An ideal discriminator would not be biased w.r.t . to its training set , and could therefore be considered as a perfect metric that matches human consensus . Note that discriminators are already reported to be highly accurate to distinguish human written texts from machine generated ones [ 31 , 43 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In light of this observation , two concurrent approaches have been explored : i ) at training time , using Generative Adversarial Networks [ 42 ] ; and ii ) at inference time , via cooperative decoding [ 11 ] : a discriminator guides the search algorithm , such that the generator and the discriminator cooperate to select the generated tokens . These approaches pursue the same objective : producing texts more similar to what a human writes . Both methodologies suffer from specific limitations . Cooperative decoding algorithms rely on a discriminator that re-ranks a limited set of candidates selected by the generator.1 Hence , cooperative decoding algorithms are limited by the generator ability to rank relevant tokens in a good enough position . On the other hand , language GANs are learned via reinforcement learning due to the discrete nature of text . This makes them particularly unstable to train , and usually fall short compared to Teacher Forcing [ 3 ] . In standard Language GANs , the discriminator provides a reward for the entire sequence , which can be difficult to exploit by the generator due to its sparsity [ 6 ] . In this paper , we propose SelfGAN , a framework to learn language GANs in a Self -training process where the signal from the discriminator is passed to the generator in a completely new way . We consider cooperative algorithms as a way to infuse the discriminator signal . We start from a simple observation : outputs obtained via cooperative decoding are more human-like , compared to their generator-only counterparts . Inspired by recent knowledge distillation approaches , we propose to consider cooperative outputs as targets in a Teacher Forcing training process : cooperative decoding stands as a teacher we attempt to imitate through the generator network . Just like a standard GAN , both the generator and the discriminator are trained at each step . While the generator improves , it becomes adversarial to the discriminator , which benefits from the cooperative generation . The discriminator , now trained on improved sequences , also contributes to improve the cooperative generation , and so forth . Note that in SelfGANs the discriminator is only used to drive the cooperative generation and never to provide a reward signal like in standard Language GANs . SelfGAN can be implemented with any cooperative decoding algorithm . Current cooperative approaches [ 7 , 31 ] rely on `` myopic '' algorithms like Beam Search or Sampling that generate the tokens left-to-right . The model has to always predict the next word , and can never look back and revise past choices . In some cases , despite all the candidates being judged to likely not be human by the discriminator , the model is locked in a dead-end . This behavior is quite unnatural for humans – who often proofread their texts . We refer to this phenomenon as the left-to-right curse . To address this left-to-right curse , we introduce Coop-MCTS , a new decoding algorithm based on Monte Carlo Tree Search ( MCTS ) [ 5 , 14 ] . We compare Coop-MCTS to state-of-the-art cooperative decoding algorithms in two scenarios : i ) inference time , as the decoding algorithm ; and ii ) during training , as the cooperative algorithm in SelfGAN . In both scenarios , we show that the respective resulting outputs are more likely to look like human texts and improve all the automatic metrics . All in all , our contributions can be summarized as follows : 1 . SelfGAN We propose a new training framework based on cooperative decoding , wherein the generated sequences are used as ground truth ; 2 . Coop-MCTS We improve cooperative decoding with a new decoding algorithm , CoopMCTS , offering a solution to the left-to-right limitation of current search methods ; 3 . We show that combining SelfGAN and Coop-MCTS compare favorably to prior state-of-theart results on two challenging tasks , Summarization and Question Generation . 2 Related Work . Beyond Teacher Forcing To mitigate the limitations inherent to Teacher Forcing , various alternatives have been proposed . In Scheduled Sampling , Bengio et al . [ 2 ] proposed to condition the generation not only on the ground truth tokens , but also the generated ones . Given that only one reference is available , this introduces a new mismatch when computing the loss , this time between the generated tokens used to condition the model , and the target tokens . To take into account multiple possible 1Opposed to a generator that outputs probabilities for the entire vocabulary V at once , a discriminator outputs the likelihood for a specific sequence to be human-written or machine-generated . Calculating the discriminator probability for every possible sequence is therefore not realistic , as the computation grows exponentially at a pace of V l where V is the vocabulary size and l the sequence length . references , using a sequence level metric is a potential alternative . In Mixer , Ranzato et al . [ 27 ] chose BLEU , the standard metric to evaluate Machine Translation . Since it is not differentiable , the task is framed in a Reinforcement Learning setup where the reward corresponds to the BLEU score given a sampled sequence of tokens . Paulus et al . [ 23 ] applied the same method to Summarization , using this time ROUGE . While the results improved in terms of ROUGE , the human evaluation found that the generated summaries were rated worse in term of fluency than the MLE baseline . The model learns to take advantage of metric biases , while being less correct according to human judgement . Language GANs In theory , a perfect discriminator would be able to judge if an output corresponds to the data distribution or not . Discriminators could therefore be an interesting alternative reward compared to other metrics . In practice , we need to train the discriminator jointly with the generator , framing the task as a GAN . Language GANs are known to underperform MLE [ 3 ] , due to the unavoidable sparsity of a discriminator reward . A large body of works have proposed denser rewards : ranking or comparative discriminators [ 4 , 17 , 45 ] , a sequential discriminator where the rewards are provided at each time step of the generation [ 35 , 6 ] . More recently , Scialom et al . [ 30 ] proposed to stabilize the GAN training by lowering the Softmax temperature to explore more structured outputs , and closer to the generator distribution . In this work , our proposed framework allows to propagate the discriminator signal in a cooperative way , which can be seen as an alternative solution to the sparsity of the reward and the training stability . Knowledge Distillation SelfGAN has a connection with knowledge distillation [ 12 ] , where a student is trained on outputs from the teacher . In particular , self distillation using only a generator has shown to improve generalisation on image GANs [ 44 ] by acting as label smoothing . To the best of our knowledge , this work is the first to propose the idea of augmenting the teacher by coupling a discriminator to a generator . Beyond GANs , SelfGAN could serve for other applications using distillation , e.g . in semi-supervised methods that use a teacher model to create synthetic labels for unlabeled examples [ 33 , 40 ] . Monte Carlo Tree Search in NLG Despite important successes in games [ 29 , 36 ] , very few works have attempted to apply MCTS to NLG . Kumagai et al . [ 15 ] proposed to employ context-free grammar rules combined with a n-gram language model and explore the space of grammatically correct texts via a MCTS . In the context of commercial e-commerce agents , Mukherjee [ 19 ] proposed to optimise with a MCTS a scoring function designed to reward grammatical correctness . 3 SelfGAN . Algorithm 1 SelfGAN 1 : Input : a generator gen , a discriminator discr , and a cooperative decoding method decodcoop 2 : for n epochs do 3 : for X , Sref in training set do . Start Training 4 : Scoop ← decodcoop ( X , gen , discr ) 5 : gen.train ( srcs=X , tgts=Scoop ) . Standard maximum likelihood but with Scoop as the target , and not Sref 6 : discr.train ( srcs=X , human_exs= Sref , machine_exs=Scoop ) The difficulty of GAN-based approaches for NLP tasks lies in the fact that no gradient flow can be propagated from the discriminator to the generator . As discussed above , approaches from the literature circumvent this difficulty by employing RL approaches , using discriminator scores as rewards to train the generator . However , such approaches induce great instabilities in the learning process , due to the use of a non-stationary reward function in addition to the high variance associated to monte-carlo estimations of RL . The idea in our SelfGAN approach is to transfer the sparse signal of the discriminator , classically used as rewards for a RL procedure , to the sampling mechanism of sequences that have to be favored through MLE . In that way , SelfGAN starts from a pretrained generator , that we fine-tune using sequences Scoop provided by a cooperative decoding process decodcoop for each condition in the training set X . This process , detailed in the next section , uses both the generator and a discriminator network to output human-like sequences Scoop , for which we improve the generator likelihood via classical maximization : maxθ ∑ ( x , s ) ∈ ( X , Scoop ) πθ ( s|x ) , where πθ ( s|x ) = ∏|s| t=1 πθ ( st|s0 : t−1 , x ) stands for the generator probability of sequence s given the conditioning input x , with πθ implemented as a neural architecture with a softmax output function . At each iteration of the training procedure , the discriminator network is optimized as a binary classifier on i ) the human references and ii ) the machine generated via the cooperative sequences : 1 |H| ∑ ( x , sref ) ∈H log ( D ( x , sref ) ) + 1 |G| ∑ ( x , scoop ) ∈G log ( 1−D ( x , scoop ) ) where x is the source input , H is the set of pairs associating x with a human written text sref from the data distribution , and G is a set of pairs with generated outputs scoop . D ( x , s ) stands for the probability , provided by the discriminator network , that sequence s is a human reference for condition x . In order to effectively guide the cooperative process at each step , the discriminator needs to be sequential : consistently with [ 31 ] , we use a left-to-right mask during training , allowing discriminator predictions for unfinished sequences . Please note that , by construction of the cooperative decoding process , we have with high probability at each iteration D ( x , scoop ) > = D ( x , sgen ) for any condition x ∈ X , with scoop a cooperative decoded sequence for x and sgen a sequence directly sampled from the generator according to πθ ( s|x ) . Based on this observation , and provided that the discriminator is sufficiently trained at each step , the generator is trained such that the probability of predicting human-like sequences is maximized . This process i ) allows us to consider a sequence level metric , and ii ) offers more stability compared to Reinforcement Learning , as we observe in our experiments ( see section 6 ) . Note also that , contrary to RL approaches which have to find a good balance between discriminator and generator capacities , our approach does not suffer from Vanishing Gradient [ 1 ] , since discrimination is only used for decoding , in a cooperative process for generator training . We depict the SelfGAN in Algorithm 1 .

Summary: To train language GANs, previous approaches use the discriminator score as a reward, and train the generator through RL. Instead, this paper proposes to use the discriminator to guide the decoding process, searching for outputs with high discriminator scores, and then train the generator to generate these outputs. To find outputs with high discriminator scores, the authors employ Monte Carlo Tree Search and use the discriminator as the value network. Experiments include question generation and summarization, and the proposed method shows improvements to MLE and ColdGAN.