Self-Adversarial Learning with Comparative Discrimination for Text Generation

1 INTRODUCTION . Generative Adversarial Networks ( Goodfellow et al. , 2014 ) ( GANs ) have achieved tremendous success for image generation and received much attention in computer vision . For text generation , however , the performance of GANs is severely limited due to reward sparsity and mode collapse : reward sparsity refers to the difficulty for the generator to receive reward signals when its generated samples can hardly fool the discriminator that is much easier to train ; while mode collapse refers to the phenomenon that the generator only learns limited patterns from the real data . As a result , both the quality and diversity of generated text samples are limited . To address the above issues , we propose a novel self-adversarial learning ( SAL ) paradigm for improving adversarial text generation . In contrast to standard GANs ( Figure 1 ( a ) ) that use a binary classifier as its discriminator to predict whether a sample is real or generated , SAL employs a comparative discriminator which is a pairwise classifier assessing whether the currently generated sample is better than its previously generated one , as shown in Figure 1 ( b ) . During training , SAL rewards the generator when its currently generated samples are found to be better than its previously generated samples . In the earlier training stage when the quality of generated samples is far below the real data , this self-improvement reward mechanism makes it easier for the generator to receive non-sparse rewards with informative learning signals , effectively alleviating the reward sparsity issue ; while in the later training stage , SAL can prevent a sample from keeping receiving high reward as the self-improvement for a popular mode will become more and more difficult , and therefore help the generator avoid collapsing toward the limited patterns of real data . We comprehensively evaluate the proposed self-adversarial learning paradigm in both synthetic data and real data on the text generation benchmark platform ( Zhu et al. , 2018 ) . Compared to the previous approaches for adversarial text generation ( Yu et al. , 2017 ; Che et al. , 2017 ; Lin et al. , 2017 ) , our ∗ This work was done during the first author ’ s internship at Microsoft Research Asia . approach shows a substantial improvement in terms of both the quality and the diversity of generated samples as well as better performance stability in adversarial learning . 2 BACKGROUND : ADVERSARIAL TEXT GENERATION . Adversarial text generation has drawn much attention in recent years due to its advantages ( e.g. , sequence-level guidance without the exposure bias issue ( Bengio et al. , 2015 ) ) over maximum likelihood estimation ( MLE ) for natural language generation . It formulates the learning process as a minimax game between a generator Gθ parameterized by θ and a discriminator Dφ parameterized by φ : the discriminator is trained to distinguish between the samples drawn from the real data distribution pdata and the samples generated by the generator ; while the generator is trained to generate samples that can “ fool ” the discriminator . The adversarial learning objective of the generator and the discriminator can be formulated as : min θ max φ Ex∼pdata [ logDφ ( x ) ] + Ez∼pz [ log ( 1−Dφ ( Gθ ( z ) ) ) ] ( 1 ) where x is a sample from the real data , Gθ ( z ) is a sample generated by the generator with the initialization z that is drawn from the noise distribution pz ( e.g. , standard normal distribution ) . While GANs have shown some promising results ( Yu et al. , 2017 ; Guo et al. , 2018 ) , there are two fundamental issues that impede their progress in text generation : ( i ) Reward sparsity , which is due to the fact that the discriminator tends to learn much better than the generator and thus easily recognizes generated samples as fake . In such cases , it will be difficult for the generator to receive rewards ; ( ii ) Mode collapse , which arises from the intrinsic nature of GANs and leads the adversarial models to only learn the limited patterns from the real samples . These two issues limit the ability of GANs to generate high-quality and diverse text samples , which have not been well addressed yet . 3 SELF-ADVERSARIAL LEARNING . To address the aforementioned issues , we propose a novel self-adversarial learning ( SAL ) paradigm . Inspired by self-play ( Silver et al. , 2017 ; Rennie et al. , 2017 ) in reinforcement learning , the core idea of SAL is to reward the generator if its currently generated sample is found to be better than its previously generated ones . Like AlphaGo ( Silver et al. , 2017 ) , the generator in SAL struggles to learn to generate better samples than its previously generated samples for passing the “ self-improvement ” test by a comparative discriminator , which is a pairwise classifier trained to compare the quality of two samples , as Figure 1 ( b ) shows . Compared to conventional GANs ( Figure 1 ( a ) ) , SAL has the following advantages : First , in the earlier training stage when the quality of generated samples is far below the real data , the self-improvement reward mechanism of SAL allows the generator to receive informative learning signals more easily as it makes the assessment of sample quality better adapted to the current capability of the generator , making it less likely to suffer from the issue of reward sparsity ; Second , in the later training stage when the generated samples ’ quality is high , SAL can prevent a sample from keeping receiving high reward as it will become more and more difficult to pass the “ self-improvement ” test , thus reducing the risk of the generator collapsing towards limited patterns . The self-improvement mechanism and the ’ tie ’ option in the comparative discriminator also provides a reasonable baseline which corresponds to cases where newly generated samples are found to be indistinguishable with previous ones , thus improving the training stability . We provide a more detailed qualitative analysis of why the proposed self-adversarial learning paradigm can alleviate these problems in Appendix . 3.1 COMPARATIVE DISCRIMINATOR . As introduced above , the core component for SAL is the comparative discriminator , which is a pairwise classifier comparing the quality of two samples . It learns a total order of sample quality and encodes the inductive bias that one sample is better ( > ) , worse ( < ) , or indistinguishable ( ≈ ) in terms of its quality compared to the other . For a ( text ) sample , the comparative discriminator can offer more informative learning signals than the conventional binary ( i.e. , real/fake ) classification based discriminator because the sample can receive multiple feedbacks from the comparative discriminator by comparing it with multiple other samples . For training the comparative discriminator , we construct pairwise training examples from the real and generated samples , as Figure 2 shows . For a real sample s+ and a generated sample s− , we assign the label “ better ( > ) ” to the pair ( s+ , s− ) and “ worse ( < ) ” to ( s− , s+ ) . For two samples both from real data or from the generated samples , we assign the label “ indistinguishable ( ≈ ) ” to such pairs ( i.e. , ( si+ , s j + ) and ( s i − , s j − ) ) . For a training set with n real samples and n generated samples , the comparative discrimination can construct ( 2n 2 ) pairwise training examples , allowing to enhance the generalization ability of the comparative discriminator . Moreover , to improve the model ’ s ability to distinguish between good generated samples and bad generated samples for self-play learning , we additionally select the samples generated during the later stage of MLE training as pseudo-real samples , and select those generated in the earlier epochs when the generator does not fully converge as fake sentences . We then pair the pseudo-real samples with the fake samples to construct training instances to supervise the model to compare their qualities . In this way , the comparative discriminator is prevented from being taught to always recognize two generated samples as equally bad and assign zero reward to the generator . As a result , it can become more sensitive to the quality difference in a pair of text samples and thus allow the generator to receive rewards more easily . 3.2 TRAINING . Before we formally introduce the training procedure for SAL , we first define the learning objective for the comparative discriminator Dφ and the generator Gθ in SAL : LD = −E ( x1 , x2 ) ∼ ( M∪pdata ( x ) ) 2 [ logD Q ( x1 , x2 ) φ ( x1 , x2 ) ] ( 2 ) LG = −E ( z , xr ) ∼ ( pz ( z ) , M ) [ ∑ q∈ { > , < , ≈ } wq logD q φ ( Gθ ( z ) , xr ) ] ( 3 ) In Eq ( 2 ) and Eq ( 3 ) , M is the set of previous generated samples by the generator , Q ( x1 , x2 ) ∈ { > , < , ≈ } is the true label for the pair ( x1 , x2 ) , Dqφ ( x1 , x2 ) is the probability of the comparative discriminator ’ s prediction being q ( q ∈ { > , < , ≈ } ) for the pair ( x1 , x2 ) . wq is the reward weight for the case q , which is a hyperparameter for SAL . If the generator generates a sample Gθ ( z ) that is better ( > ) than its previously generated sample xr , it receives a positive reward ; if Gθ ( z ) is worse ( < ) than xr , it receives a negative reward ; while if the quality of Gθ ( z ) is classified as similar ( ≈ ) to xr , it receives zero credit . Therefore , we have w ( > ) > 0 = w ( ≈ ) > w ( < ) . Since LG can only be directly optimized in standard continuous GAN training , we alternatively employ the policy gradient algorithm ( Sutton et al. , 2000 ) to train the generator , as previous approaches for adversarial text generation . For SAL , we define the reward for a generated sample xg compared with a reference sample xr which is a previous generated sample by the generator as the weighted reward based on the probability distribution of the prediction by the comparative discriminator : γφ ( xg , xr ) = w ( > ) D ( > ) φ ( xg , xr ) + w ( < ) D ( < ) φ ( xg , xr ) + w ( ≈ ) D ( ≈ ) φ ( xg , xr ) ( 4 ) In text generation , the generator Gθ obtains the reward only when one sample has been completely generated , which means no intermediate reward is gained . To relieve this problem , following the practice in SeqGAN ( Yu et al. , 2017 ) , we utilize the Monte Carlo rollout method to approximate intermediate rewards by sampling unknown tokens with generated prefix Y1 : t following generator policy Gθ till sample completion . Empirically , we found that the Monte Carlo rollout also helps to reduce the variance of the reference sample utilized for comparison . We calculate the expected reward as Rθ , φ ( s = Y1 : t−1 , a = yt ) = E ( xg , xr ) ∼ ( Gθ ( Y1 : t−1 ) , M ) [ γφ ( xg , xr ) ] ( 5 ) The objective of the generator is to generate a sequence to maximize its expected final reward . With likelihood ratio ( Sutton et al. , 2000 ) , we can formulate the gradient of the objective function for generator Gθ as : ∇θJ ( θ ) = T∑ t=1 EY1 : t−1∼Gθ [ ∇θ log ( Gθ ( yt|Y1 : t ) ) · Rθ , φ ( s = Y1 : t−1 , a = yt ) ] ( 6 ) To improve the training of self-adversarial learning , we borrow ideas from the field of deep reinforcement learning and propose two training techniques to improve self-adversarial training . Scheduled rewarding Similar to the exploitation-exploration trade-off in reinforcement learning ( Langford & Zhang , 2007 ) , the positive reward assigned for actions generating better samples encourage exploration while the penalty for generating worse samples makes the generator more conservative . Intuitively , in the earlier stage of self-adversarial learning , the generator should explore better policy by receiving higher rewards for relative progress ; while in the later stage , the generator should be more conservative by penalizing more for worse samples to avoid performance degradation . We simply decrease w ( > ) and increase w ( < ) linearly with training iteration and refer this technique as scheduled rewarding . Algorithm 1 Self-Adversarial Learning With Comparative Discriminator Require : Generator Gθ ; comparative discriminator Dφ ; samples of real sentences S+ ; self-adversarial learning step g ; discriminator step k ; memory bufferM for the previous generated samples 1 : Pretrain Gθ using MLE on S+ 2 : Generate samples with Gθ and store them intoM 3 : repeat 4 : for k steps do 5 : Collect a mini-batch of balanced sample pairs ( x1 , x2 ) fromM∪S+ 6 : Update Dφ via Eq ( 2 ) 7 : end for 8 : for g steps do 9 : Generate a mini-batch of samples xg ∼ Gθ 10 : Collect a mini-batch of reference samples xr fromM 11 : Update Gθ via Eq ( 6 ) 12 : end for 13 : UpdateM with Gθ 14 : until Convergence Memory replay Continuously comparing the generator with its most recent stage may suffer from the correlation between generated samples and reference samples , which makes the training process unstable . Inspired by experience replay ( Lillicrap et al. , 2015 ) , we construct a memory buffer which contains samples generated in the last K training steps . Reference samples are sampled from the memory buffer rather than samples generated in the most recent stage of the generator , which empirically helps stabilize the training process . The training process of SAL is summarized in Algorithm 3 . Self-adversarial learning with the proposed comparative discriminator achieves Nash Equilibrium when the generator models the distribution of real samples perfectly . In this case , the comparative discriminator can not successfully distinguish generated samples from real samples and tends to recognize two samples as `` indistinguishable '' . The reward received by the generator is thus zero and training converges . However , how a non-Bernoulli GAN converges to such an equilibrium is still an open problem ( Goodfellow et al. , 2014 ; Goodfellow , 2014 ) and is beyond the scope of this work .

Summary: To alleviate the issues of reward sparsity and mode collapse in most text-generation GANs with a binary discriminator, this paper proposes a self-adversarial learning (SAL) framework with a novel comparative discriminator that takes pairs of text examples from real and generated examples and outputs better, worse, or indistinguishable. Inspired by self-play in reinforcement learning, SAL employs self-play to reward the generator to generate better samples than previous samples with self-improvement signals from the comparative discriminator. It is argued that, because the comparative discriminator always produces self-improvement signals during the training and the self-improvement signal will not be very strong when generated samples are already good enough, the issues of reward sparsity and mode collapses in conventional text GANs are reduced. Experimental results on synthetic data and benchmark datasets demonstrate that SAL outperforms SeqGAN, MaliGAN, and RankGAN both quantitatively and qualitatively.