To Beam Or Not To Beam: That is a Question of Cooperation for Language GANs

1 Introduction . Natural Language Generation encompasses tasks such as Machine Translation , Summarization or Data To Text generation . The real life applications are numerous , but require highly reliable and fluent models . Despite significant advances , state-of-the-art models are still known to be de-generated , with outputs containing repetitions and even nonfactual information i.e . hallucination [ 13 ] . Among the culprits is a limitation of Teacher Forcing [ 37 ] : the loss is computed at a token level while the aim is to produce complete sequences . Moreover , while a single ground-truth reference is considered correct , several realizations of the same content may exist . Finally , the model is subject to Exposure Bias [ 27 ] , i.e . a mismatch between training and inference distributions – in the latter , the model has no access to ground truth for the previously generated tokens . The literature has considered this mismatch responsible for the lower quality observed when generating longer sequences [ 2 , 16 ] . To overcome such Teacher Forcing limitations , a consensus has emerged : a sequence level objective should be introduced [ 27 , 41 ] . A body of work has proposed to use Reinforcement Learning ( RL ) with standard NLG metrics like BLEU [ 39 ] or ROUGE [ 23 ] . However , NLG metrics are known to not reflect well human judgement [ 21 ] , which explains why the resulting models tend to be qualitatively worse than their MLE baselines [ 3 ] . To move toward less biased metrics , a natural alternative is to evaluate the output with a learned discriminator . An ideal discriminator would not be biased w.r.t . to its training set , and could therefore be considered as a perfect metric that matches human consensus . Note that discriminators are already reported to be highly accurate to distinguish human written texts from machine generated ones [ 31 , 43 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In light of this observation , two concurrent approaches have been explored : i ) at training time , using Generative Adversarial Networks [ 42 ] ; and ii ) at inference time , via cooperative decoding [ 11 ] : a discriminator guides the search algorithm , such that the generator and the discriminator cooperate to select the generated tokens . These approaches pursue the same objective : producing texts more similar to what a human writes . Both methodologies suffer from specific limitations . Cooperative decoding algorithms rely on a discriminator that re-ranks a limited set of candidates selected by the generator.1 Hence , cooperative decoding algorithms are limited by the generator ability to rank relevant tokens in a good enough position . On the other hand , language GANs are learned via reinforcement learning due to the discrete nature of text . This makes them particularly unstable to train , and usually fall short compared to Teacher Forcing [ 3 ] . In standard Language GANs , the discriminator provides a reward for the entire sequence , which can be difficult to exploit by the generator due to its sparsity [ 6 ] . In this paper , we propose SelfGAN , a framework to learn language GANs in a Self -training process where the signal from the discriminator is passed to the generator in a completely new way . We consider cooperative algorithms as a way to infuse the discriminator signal . We start from a simple observation : outputs obtained via cooperative decoding are more human-like , compared to their generator-only counterparts . Inspired by recent knowledge distillation approaches , we propose to consider cooperative outputs as targets in a Teacher Forcing training process : cooperative decoding stands as a teacher we attempt to imitate through the generator network . Just like a standard GAN , both the generator and the discriminator are trained at each step . While the generator improves , it becomes adversarial to the discriminator , which benefits from the cooperative generation . The discriminator , now trained on improved sequences , also contributes to improve the cooperative generation , and so forth . Note that in SelfGANs the discriminator is only used to drive the cooperative generation and never to provide a reward signal like in standard Language GANs . SelfGAN can be implemented with any cooperative decoding algorithm . Current cooperative approaches [ 7 , 31 ] rely on `` myopic '' algorithms like Beam Search or Sampling that generate the tokens left-to-right . The model has to always predict the next word , and can never look back and revise past choices . In some cases , despite all the candidates being judged to likely not be human by the discriminator , the model is locked in a dead-end . This behavior is quite unnatural for humans – who often proofread their texts . We refer to this phenomenon as the left-to-right curse . To address this left-to-right curse , we introduce Coop-MCTS , a new decoding algorithm based on Monte Carlo Tree Search ( MCTS ) [ 5 , 14 ] . We compare Coop-MCTS to state-of-the-art cooperative decoding algorithms in two scenarios : i ) inference time , as the decoding algorithm ; and ii ) during training , as the cooperative algorithm in SelfGAN . In both scenarios , we show that the respective resulting outputs are more likely to look like human texts and improve all the automatic metrics . All in all , our contributions can be summarized as follows : 1 . SelfGAN We propose a new training framework based on cooperative decoding , wherein the generated sequences are used as ground truth ; 2 . Coop-MCTS We improve cooperative decoding with a new decoding algorithm , CoopMCTS , offering a solution to the left-to-right limitation of current search methods ; 3 . We show that combining SelfGAN and Coop-MCTS compare favorably to prior state-of-theart results on two challenging tasks , Summarization and Question Generation . 2 Related Work . Beyond Teacher Forcing To mitigate the limitations inherent to Teacher Forcing , various alternatives have been proposed . In Scheduled Sampling , Bengio et al . [ 2 ] proposed to condition the generation not only on the ground truth tokens , but also the generated ones . Given that only one reference is available , this introduces a new mismatch when computing the loss , this time between the generated tokens used to condition the model , and the target tokens . To take into account multiple possible 1Opposed to a generator that outputs probabilities for the entire vocabulary V at once , a discriminator outputs the likelihood for a specific sequence to be human-written or machine-generated . Calculating the discriminator probability for every possible sequence is therefore not realistic , as the computation grows exponentially at a pace of V l where V is the vocabulary size and l the sequence length . references , using a sequence level metric is a potential alternative . In Mixer , Ranzato et al . [ 27 ] chose BLEU , the standard metric to evaluate Machine Translation . Since it is not differentiable , the task is framed in a Reinforcement Learning setup where the reward corresponds to the BLEU score given a sampled sequence of tokens . Paulus et al . [ 23 ] applied the same method to Summarization , using this time ROUGE . While the results improved in terms of ROUGE , the human evaluation found that the generated summaries were rated worse in term of fluency than the MLE baseline . The model learns to take advantage of metric biases , while being less correct according to human judgement . Language GANs In theory , a perfect discriminator would be able to judge if an output corresponds to the data distribution or not . Discriminators could therefore be an interesting alternative reward compared to other metrics . In practice , we need to train the discriminator jointly with the generator , framing the task as a GAN . Language GANs are known to underperform MLE [ 3 ] , due to the unavoidable sparsity of a discriminator reward . A large body of works have proposed denser rewards : ranking or comparative discriminators [ 4 , 17 , 45 ] , a sequential discriminator where the rewards are provided at each time step of the generation [ 35 , 6 ] . More recently , Scialom et al . [ 30 ] proposed to stabilize the GAN training by lowering the Softmax temperature to explore more structured outputs , and closer to the generator distribution . In this work , our proposed framework allows to propagate the discriminator signal in a cooperative way , which can be seen as an alternative solution to the sparsity of the reward and the training stability . Knowledge Distillation SelfGAN has a connection with knowledge distillation [ 12 ] , where a student is trained on outputs from the teacher . In particular , self distillation using only a generator has shown to improve generalisation on image GANs [ 44 ] by acting as label smoothing . To the best of our knowledge , this work is the first to propose the idea of augmenting the teacher by coupling a discriminator to a generator . Beyond GANs , SelfGAN could serve for other applications using distillation , e.g . in semi-supervised methods that use a teacher model to create synthetic labels for unlabeled examples [ 33 , 40 ] . Monte Carlo Tree Search in NLG Despite important successes in games [ 29 , 36 ] , very few works have attempted to apply MCTS to NLG . Kumagai et al . [ 15 ] proposed to employ context-free grammar rules combined with a n-gram language model and explore the space of grammatically correct texts via a MCTS . In the context of commercial e-commerce agents , Mukherjee [ 19 ] proposed to optimise with a MCTS a scoring function designed to reward grammatical correctness . 3 SelfGAN . Algorithm 1 SelfGAN 1 : Input : a generator gen , a discriminator discr , and a cooperative decoding method decodcoop 2 : for n epochs do 3 : for X , Sref in training set do . Start Training 4 : Scoop ← decodcoop ( X , gen , discr ) 5 : gen.train ( srcs=X , tgts=Scoop ) . Standard maximum likelihood but with Scoop as the target , and not Sref 6 : discr.train ( srcs=X , human_exs= Sref , machine_exs=Scoop ) The difficulty of GAN-based approaches for NLP tasks lies in the fact that no gradient flow can be propagated from the discriminator to the generator . As discussed above , approaches from the literature circumvent this difficulty by employing RL approaches , using discriminator scores as rewards to train the generator . However , such approaches induce great instabilities in the learning process , due to the use of a non-stationary reward function in addition to the high variance associated to monte-carlo estimations of RL . The idea in our SelfGAN approach is to transfer the sparse signal of the discriminator , classically used as rewards for a RL procedure , to the sampling mechanism of sequences that have to be favored through MLE . In that way , SelfGAN starts from a pretrained generator , that we fine-tune using sequences Scoop provided by a cooperative decoding process decodcoop for each condition in the training set X . This process , detailed in the next section , uses both the generator and a discriminator network to output human-like sequences Scoop , for which we improve the generator likelihood via classical maximization : maxθ ∑ ( x , s ) ∈ ( X , Scoop ) πθ ( s|x ) , where πθ ( s|x ) = ∏|s| t=1 πθ ( st|s0 : t−1 , x ) stands for the generator probability of sequence s given the conditioning input x , with πθ implemented as a neural architecture with a softmax output function . At each iteration of the training procedure , the discriminator network is optimized as a binary classifier on i ) the human references and ii ) the machine generated via the cooperative sequences : 1 |H| ∑ ( x , sref ) ∈H log ( D ( x , sref ) ) + 1 |G| ∑ ( x , scoop ) ∈G log ( 1−D ( x , scoop ) ) where x is the source input , H is the set of pairs associating x with a human written text sref from the data distribution , and G is a set of pairs with generated outputs scoop . D ( x , s ) stands for the probability , provided by the discriminator network , that sequence s is a human reference for condition x . In order to effectively guide the cooperative process at each step , the discriminator needs to be sequential : consistently with [ 31 ] , we use a left-to-right mask during training , allowing discriminator predictions for unfinished sequences . Please note that , by construction of the cooperative decoding process , we have with high probability at each iteration D ( x , scoop ) > = D ( x , sgen ) for any condition x ∈ X , with scoop a cooperative decoded sequence for x and sgen a sequence directly sampled from the generator according to πθ ( s|x ) . Based on this observation , and provided that the discriminator is sufficiently trained at each step , the generator is trained such that the probability of predicting human-like sequences is maximized . This process i ) allows us to consider a sequence level metric , and ii ) offers more stability compared to Reinforcement Learning , as we observe in our experiments ( see section 6 ) . Note also that , contrary to RL approaches which have to find a good balance between discriminator and generator capacities , our approach does not suffer from Vanishing Gradient [ 1 ] , since discrimination is only used for decoding , in a cooperative process for generator training . We depict the SelfGAN in Algorithm 1 .

Summary: This paper presents a method of training (or rather fine-tuning) text generation models using a GAN-like setup. Instead of using the discriminator directly to update the generator as is usually the case, the authors propose using the discriminator as a way of cooperative decoding to generate high-quality samples which are then used as the gold output to train the model in the usual teacher-forcing setup. The authors explore a sampling procedure based on MCTS to generate more diverse sentences rather than using beam-search or sampling during training and inference. On summarization and question generation, they show improvements on the standard and proposed evaluation metrics.