DelibGAN: Coarse-to-Fine Text Generation via Adversarial Network

1 INTRODUCTION . Building a good text generative model has always been a fundamental problem in the natural language processing field . The most common generative model is Recurrent Neural Networks ( RNN ) ( Mikolov et al. , 2011 ) , which predicts each word of a sentence conditioned on the previous word and an evolving hidden state . However , it suffers from two main drawbacks : First , RNN based models are always trained through the maximum likelihood approach , which suffers from exposure bias ( Bengio et al. , 2015 ) ; Second , the loss function used to train the model is at word level but the performance is typically evaluated at sentence level ( Wang & Wan , 2018a ) . Some recent studies have tried to solve these problems . Some approaches work indirectly by making the hidden state dynamics predictable ( Professor Forcing ( Goyal et al. , 2016 ) ) or by randomly adjusting the sampling words during training time ( Scheduled Sampling ( Bengio et al. , 2015 ) ) . But they do not directly specify the cost function on the RNN output to encourage high sample quality . Later works used Generative Adversarial Networks ( GAN ) ( Goodfellow et al. , 2014 ) or reinforcement learning to directly affect the decoding sequence , such as Gumbel-softmax distribution ( Kusner & Hernández-Lobato , 2016 ) , SeqGAN ( Yu et al. , 2017 ) , MaskGAN ( Fedus et al. , 2018 ) , etc . However , they all adopt an one-pass forward decoding , which can only see words that have been decoded . Recently , coarse-to-fine generator with multiple decoders has achieved great success on some tasks . But they either do not define the meaning of the sketch ( Xia et al. , 2017 ) , or are limited to specific tasks such as logical form parsing , code generation , SQL query generation , etc ( Lapata & Dong , 2018 ) . And they all are supervised learning methods . Inspired by Deliberation Networks ( Xia et al. , 2017 ) , we propose a novel adversarial learning framework , namely DelibGAN , for generating high-quality sentences without supervision . Our framework consists of a coarse-to-fine generator and a multiple instance discriminator , and the coarseto-fine generator contains a first-pass decoder and a second-pass decoder . Further , we propose two training mechanisms , named DelibGAN-I and DelibGAN-II . The former one uses the sentencelevel penalty predicted by the discriminator to fine-tune the second-pass decoder . And the latter one imposes the influence of word importance on the first-pass decoder , with the purpose of decoding important words as much as possible in the first pass , and thus better guiding the decoding of the second-pass decoder . The word importance is obtained through the multiple instance learning mechanism of the discriminator , which is used to directly tune the first-pass decoder and later guide the second-pass decoding . In short , for DelibGAN-I , the discriminator is used to fine-tune the second- pass decoder G2 , while for DelibGAN-II , the discriminator is further used to evaluate the importance of each word and tune G1 . Our motivation is two-fold : ( 1 ) We use a coarse-to-fine generator to generate important words ( i.e. , sketch ) as much as possible during the first decoding and utilize the global information ( i.e. , the sentence decoded in the first pass ) at the time of final decoding ; ( 2 ) The discriminator can evaluate the sentence-level quality to guide the final generation via policy gradients , moreover , it can provide word-level importance , which can be used to adjust the sketches in the first-pass generation to make sense . We compare our models with several typical and state-of-the-art unsupervised generic text generation models , including RNNLM ( Mikolov et al. , 2011 ) , SeqGAN ( Yu et al. , 2017 ) , SentiGAN ( Wang & Wan , 2018a ) , MaskGAN ( Fedus et al. , 2018 ) . Experimental results on three datasets ( a synthetic dataset , a descriptive text dataset and a sentimental text dataset ) show that our models can produce more realistic samples . The major contributions of this paper are summarized as follows : • We propose a novel adversarial learning framework - DelibGAN , which consists of a coarse-to-fine generator and a multiple instance discriminator , to generate high-quality sentences without supervision . • We propose two training mechanisms , DelibGAN-I and DelibGAN-II , to make the generator generate higher-quality sentences . • Both qualitative and quantitative results show that DelibGAN-I and DelibGAN-II can produce more realistic samples , and DelibGAN-II performs best . 2 FRAMEWORK . 2.1 ARCHITECTURE . The architecture of our proposed DelibGAN is show in Figure1 . The entire model consists of two modules : a coarse-to-fine generator and a multiple instance discriminator . The coarse-to-fine generator has a first-pass decoder G1 and a second-pass decoder G2 . The multiple instance discriminator D can evaluate both a whole sentence and each individual word in the sentence by using the multiple instance learning mechanism , and thus the two decoders can be influenced by the discriminator D in different ways . In our first training mechanism DelibGAN-I , the sentence-level penalty predicted by the discriminator is used to fine-tune the second-pass decoder G2 . In our second training mechanism DelibGAN-II , in addition to the fine-tuning of G2 , the word-level importance is predicted and used to tune G1 , to make G1 decode important words as much as possible , and thus better guide the decoding of G2 . 2.2 COARSE-TO-FINE GENERATOR . Inspired by Deliberation Networks ( Xia et al. , 2017 ) , we propose a coarse-to-fine generator with the following two purposes : The first is to enable the generator to utilize the global information ( i.e. , the sentence decoded in the first pass ) at the time of final decoding , rather than only the sequence that has been decoded ; The second is to force the generator to generate important words ( i.e. , sketch ) as much as possible during the first-pass decoding , so as to provide more useful clues for decoding in the second pass . Our coarse-to-fine generator contains a first-pass decoder G1 and a second-pass decoder G2 . Both decoders are RNN and can be easily replaced with its variants such as LSTM and GRU ( In this work , we use LSTM ) . The prior input noise z sampled from a distribution Pz ( e.g. , a normal distribution ) is used to initialize the input of G1 , then G1 will generate a first-pass sequence x̂ = { x̂1 , x̂2 · · · x̂l } , and a series of hidden states Ĥ = { ĥ1 , ĥ2 · · · ĥl } , where l is the length of the generated sequence . At step t , ĥt is calculated as ĥt = RNN ( f ( x̂t−1 ) , ĥt−1 ) , where f is a word embedding representation function . Then ĥt is fed into a softmax layer , and x̂t is sampled out from the obtained multinomial distribution . Finally , the first-pass output sequence x̂ is generated by G1 . In the second-pass decoding , we take the hidden state of the last moment of G1 as the initial hidden state of G2 . And we use an attention model in G2 , Specifically , at step t , the attention model in G2 first generates a context ct defined as follows : ct = l∑ i=1 aiĥi ; ai ∝ exp ( vTa tanh ( Wattĥi +Uattht−1 ) ) ∀i ∈ [ 1 , l ] ; l∑ i=1 ai = 1 . ( 1 ) After receiving ct , we calculate the hidden state ht as ht = RNN ( [ f ( xt−1 ) ; ct ] , ht−1 ) . Then xt is sampled in the same way of sampling x̂t , and the second-pass output sequence x is generated and its length is m . 2.3 MULTIPLE INSTANCE DISCRIMINATOR . Like Yu et al . ( 2017 ) and Wang & Wan ( 2018a ) , in order to evaluate the performance of a generator at the sentence-level , we use a discriminator to calculate rewards . Further , we want to get the importance of each word in the sentence to adjust the generator to decode the important words as much as possible in the first-pass decoding . Thus we adopt a multi-instance learning ( MIL ) mechanism in our discriminator . MIL deals with problems where labels are associated with groups of instances or bags ( sentences in our case ) , while instance labels ( word-level importance ) are unobserved ( Keeler & Rumelhart , 1991 ; Zhang et al. , 2002 ; Wang & Wan , 2018b ) . Our multiple instance discriminator D uses a structure similar to a bidirectional RNN . Supposing the input to D is a sentence w = { w1 , w2 , · · · , wn } , where n is the length of w. We apply a bidirectional LSTM on the sentence , and for each wordwt ( t ∈ n ) , we concatenate its corresponding forward and backward hidden state vectors : −→st = −−−−→ LSTM ( wt ) ; ←−st = ←−−−− LSTM ( wt ) ; st = −→st ||←−st , ( 2 ) where −−−−→ LSTM ( wt ) and ←−−−− LSTM ( wt ) represent the forward and backward LSTM hidden state vectors for wt . Then we use a softmax classifier to get the word-level probability distribution over target labels ( i.e. , real and fake ) . pt = softmax ( tanh ( Wdst + bd ) , where pt = [ p ( t ) real , p ( t ) fake ] ; p ( t ) real + p ( t ) fake = 1 . ( 3 ) p ( t ) real and p ( t ) fake represent the probability that the t-th word is predicted to be real and fake , respectively ; Wd and bd are parameters , shared across all words . Finally , we use the average of the word-level distribution as the sentence-level distribution over the target labels : pi = 1 n n∑ t=1 p ( t ) i , ∀i ∈ { real , fake } ; p = [ preal , pfake ] . ( 4 ) 2.4 TRAINING . Here we introduce the adversarial training of the generator and the discriminator in detail . In this paper , we formalize the text generation problem as a sequential decision making process ( Bachman & Precup , 2015 ) , which solves the problem that the gradient can not pass back to the generative model when the output is discrete . Moreover , we use policy gradients to update parameters of the generator . We propose two loss functions for the generator , named DelibGAN-I and DelibGAN-II . DelibGANI just forces the first-pass decoder G1 to maintain the same decoding result , while DelibGAN-II utilizes the result of the discriminator at the word level , forcing G1 to decode important words ( evaluated by the discriminator ) as much as possible . But for the second-pass decoder G2 , both of them use the penalty-based objective function ( Wang & Wan , 2018a ) to update the parameters θG2 of G2 . Specifically , at each timestep t , the existing sequence generated by G2 is x1 : t−1 = { x1 , ... , xt−1 } , and the next token xt selected in the next step is an action sampling from the policy G2 ( xt|x1 : t−1 ; θG2 ) , and we use the Monte Carlo rollouts method ( Dai et al. , 2017 ; Yu et al. , 2017 ) to simulate intermediate penalty RθG2 , θD ( xt ) , where θD is the parameters of the discriminator D. Then , the total penalty of G2 can be computed by : Ex∼G2 ( x ) = m∑ t=1 [ ( RθG2 , θD ( xt ) − b ) G2 ( xt|x1 : t−1 ; θG2 ) ] = m∑ t=1 [ ( 1 q ( q∑ j=1 ( 1−D ( xt|x ( j ) 1 : t−1 ; θD ) ) ) − b ) G2 ( xt|x1 : t−1 ; θG2 ) ] , ( 5 ) where b is the bias , x ( j ) 1 : t−1 is the q-time Monte Carlo search sampled based on the roll-out policy G2 ( xt|x1 : t−1 ; θG2 ) , and D ( xt|x ( j ) 1 : t−1 ; θD ) is the sentence probability preal given by the discriminator for sentence x ( j ) 1 : t−1||xt . DelibGAN-I : For DelibGAN-I , we just force the first-pass decoder G1 to maintain the same decoding result . Specifically , after we use ( G1 , G2 ) to sample a pair of sentences ( x̂ , x ) , we first use the discriminator to calculate the sentence-level penalties . Then in the process of using penalties to update the generator , we force the first-pass decoder G1 to decode the same result , which is x̂ . The loss of generator G1 ( the parameters are denoted as θG1 ) is defined as follows : E ( I ) x∼G1 ( x̂ ) = 1 l l∑ t=1 − log ( G1 ( x̂t|x̂1 : t−1 ; θG1 ) ) ( 6 ) we propose it with the purpose of keeping the results of G1 as constant as possible , and just finetuning G2 based on the sentence-level penalty mentioned above . Finally , the objective of the generator is to minimize : J ( I ) G1 , G2 ( θG1 , θG2 ) = E ( I ) x∼G1 ( x̂ ) + Ex∼G2 ( x ) = 1 l l∑ t=1 − log ( G1 ( x̂t|x̂1 : t−1 ; θG1 ) ) + m∑ t=1 [ ( 1 q ( q∑ j=1 ( 1−D ( xt|x ( j ) 1 : t−1 ; θD ) ) ) − b ) G2 ( xt|x1 : t−1 ; θG2 ) ] . ( 7 ) DelibGAN-II : For DelibGAN-II , we want to impose direct influence to G1 , forcing it to decode important words as much as possible in the first-pass decoding , to better guide the decoding of G2 . We tried different ways of influencing , including using sentence-level penalties like Eq5 , but finally we introduced and leveraged a multi-instance learning mechanism in the discriminator . This is because it can provide the importance of words in a sentence , and we add the word-level importance to the loss as weights . So the loss of generator G1 is changed as follows : E ( II ) x∼G1 ( x̂ ) = 1 l l∑ t=1 −p ( t ) real log ( G1 ( x̂t|x̂1 : t−1 ; θG1 ) ) ( 8 ) where p ( t ) real is the probability that the discriminator predicts x̂t as real . Therefore , the objective of the generator is to minimize the total loss : J ( II ) G1 , G2 ( θG1 , θG2 ) = E ( II ) x∼G1 ( x̂ ) + Ex∼G2 ( x ) = 1 l l∑ t=1 −preal ( x̂ ) log ( G1 ( x̂t|x̂1 : t−1 ; θG1 ) ) + m∑ t=1 [ ( 1 q ( q∑ j=1 ( 1−D ( xt|x ( j ) 1 : t−1 ; θD ) ) ) − b ) G2 ( xt|x1 : t−1 ; θG2 ) ] . ( 9 ) For discriminator D , the goal is to distinguish between real text and generated text as much as possible . It is worth noting that we only have the target label for the sentence , but do not have labels on words . The objective function of the discriminator is to minimize : JD ( θD ) = −Ex∼G log pfake − Ex∼S log preal , ( 10 ) where S is real sentences in the corpus . We perform the adversarial training of the generator and the discriminator , and train them alternately , as shown in Algorithm 1 . Algorithm 1 The adversarial training process in DelibGAN Input : Input noise , z ; Generator , ( G1 , G2 ) ; Discriminator , D ; Real text dataset , S ; Output : Well trained generator , ( G1 , G2 ) ; 1 : Initialize ( G1 , G2 ) , D with random weights ; 2 : Pre-train ( G1 , G2 ) using MLE on S ; 3 : repeat 4 : for d-steps do 5 : Generate fake texts ( { x̂ } , { x } ) using ( G1 , G2 ) ; 6 : Update D using ( { x } , S ) by minimizing Eq 10 ; 7 : end for 8 : for g-steps do 9 : Generate fake texts ( { x̂ } , { x } ) using ( G1 , G2 ) ; 10 : Calculate penalty Ex∼G2 ( x ) by Eq ( 5 ) ; 11 : For DelibGAN-I , Update ( G1 , G2 ) by minimizing Eq ( 7 ) ; 12 : For DelibGAN-II , Update ( G1 , G2 ) by minimizing Eq ( 9 ) ; 13 : end for 14 : until Model converges 15 : return ;

Summary: The main contribution of this paper is the proposed DelibGAN for text generation. The framework introduces a coarse-to-fine generator, which contains a first-pass decoder and a second-pass decoder. Instead of using a binary classifier, the discriminator is a multiple instance discriminator. Two different variants of DelibGAN are proposed, with experiments showing that DelibGAN-II performs the best.