Year: 2018

Abstract:
In this paper, we propose a novel adversarial learning framework, namely DelibGAN, for generating high-quality sentences without supervision. Our framework consists of a coarse-to-fine generator, which contains a first-pass decoder and a second-pass decoder, and a multiple instance discriminator. And we propose two training mechanisms DelibGAN-I and DelibGAN-II. The discriminator is used to fine-tune the second-pass decoder in DelibGAN-I and further evaluate the importance of each word and tune the first-pass decoder in DelibGAN-II. We compare our models with several typical and state-of-the-art unsupervised generic text generation models on three datasets (a synthetic dataset, a descriptive text dataset and a sentimental text dataset). Both qualitative and quantitative experimental results show that our models produce more realistic samples, and DelibGAN-II performs best.

heading: 1 INTRODUCTION
text: Building a good text generative model has always been a fundamental problem in the natural language processing field. The most common generative model is Recurrent Neural Networks (RNN)(Mikolov et al., 2011), which predicts each word of a sentence conditioned on the previous word and an evolving hidden state. However, it suffers from two main drawbacks: First, RNN based models are always trained through the maximum likelihood approach, which suffers from exposure bias(Bengio et al., 2015); Second, the loss function used to train the model is at word level but the performance is typically evaluated at sentence level(Wang & Wan, 2018a).
Some recent studies have tried to solve these problems. Some approaches work indirectly by making the hidden state dynamics predictable (Professor Forcing (Goyal et al., 2016)) or by randomly adjusting the sampling words during training time (Scheduled Sampling (Bengio et al., 2015)). But they do not directly specify the cost function on the RNN output to encourage high sample quality. Later works used Generative Adversarial Networks (GAN)(Goodfellow et al., 2014) or reinforcement learning to directly affect the decoding sequence, such as Gumbel-softmax distribution(Kusner & Hernández-Lobato, 2016), SeqGAN(Yu et al., 2017), MaskGAN(Fedus et al., 2018), etc. However, they all adopt an one-pass forward decoding , which can only see words that have been decoded. Recently, coarse-to-fine generator with multiple decoders has achieved great success on some tasks. But they either do not define the meaning of the sketch(Xia et al., 2017), or are limited to specific tasks such as logical form parsing, code generation, SQL query generation, etc(Lapata & Dong, 2018). And they all are supervised learning methods.
Inspired by Deliberation Networks(Xia et al., 2017), we propose a novel adversarial learning framework, namely DelibGAN, for generating high-quality sentences without supervision. Our framework consists of a coarse-to-fine generator and a multiple instance discriminator, and the coarseto-fine generator contains a first-pass decoder and a second-pass decoder. Further, we propose two training mechanisms, named DelibGAN-I and DelibGAN-II. The former one uses the sentencelevel penalty predicted by the discriminator to fine-tune the second-pass decoder. And the latter one imposes the influence of word importance on the first-pass decoder, with the purpose of decoding important words as much as possible in the first pass, and thus better guiding the decoding of the second-pass decoder. The word importance is obtained through the multiple instance learning mechanism of the discriminator, which is used to directly tune the first-pass decoder and later guide the second-pass decoding. In short, for DelibGAN-I, the discriminator is used to fine-tune the second-
pass decoder G2, while for DelibGAN-II, the discriminator is further used to evaluate the importance of each word and tune G1. Our motivation is two-fold: (1) We use a coarse-to-fine generator to generate important words (i.e., sketch) as much as possible during the first decoding and utilize the global information (i.e., the sentence decoded in the first pass) at the time of final decoding; (2) The discriminator can evaluate the sentence-level quality to guide the final generation via policy gradients, moreover, it can provide word-level importance, which can be used to adjust the sketches in the first-pass generation to make sense.
We compare our models with several typical and state-of-the-art unsupervised generic text generation models, including RNNLM(Mikolov et al., 2011), SeqGAN(Yu et al., 2017), SentiGAN(Wang & Wan, 2018a), MaskGAN(Fedus et al., 2018). Experimental results on three datasets (a synthetic dataset, a descriptive text dataset and a sentimental text dataset) show that our models can produce more realistic samples. The major contributions of this paper are summarized as follows:
• We propose a novel adversarial learning framework - DelibGAN, which consists of a coarse-to-fine generator and a multiple instance discriminator, to generate high-quality sentences without supervision.
• We propose two training mechanisms, DelibGAN-I and DelibGAN-II, to make the generator generate higher-quality sentences.
• Both qualitative and quantitative results show that DelibGAN-I and DelibGAN-II can produce more realistic samples, and DelibGAN-II performs best.

heading: 2 FRAMEWORK

heading: 2.1 ARCHITECTURE
text: The architecture of our proposed DelibGAN is show in Figure1. The entire model consists of two modules: a coarse-to-fine generator and a multiple instance discriminator. The coarse-to-fine generator has a first-pass decoder G1 and a second-pass decoder G2. The multiple instance discriminator D can evaluate both a whole sentence and each individual word in the sentence by using the multiple instance learning mechanism, and thus the two decoders can be influenced by the discriminator D in different ways. In our first training mechanism DelibGAN-I, the sentence-level penalty predicted by the discriminator is used to fine-tune the second-pass decoder G2. In our second training mechanism DelibGAN-II, in addition to the fine-tuning of G2, the word-level importance is predicted and used to tune G1, to make G1 decode important words as much as possible, and thus better guide the decoding of G2.

heading: 2.2 COARSE-TO-FINE GENERATOR
text: Inspired by Deliberation Networks(Xia et al., 2017), we propose a coarse-to-fine generator with the following two purposes: The first is to enable the generator to utilize the global information (i.e., the sentence decoded in the first pass) at the time of final decoding, rather than only the sequence that has been decoded; The second is to force the generator to generate important words (i.e., sketch) as much as possible during the first-pass decoding, so as to provide more useful clues for decoding in the second pass.
Our coarse-to-fine generator contains a first-pass decoder G1 and a second-pass decoder G2. Both decoders are RNN and can be easily replaced with its variants such as LSTM and GRU (In this work, we use LSTM). The prior input noise z sampled from a distribution Pz (e.g., a normal distribution) is used to initialize the input of G1, then G1 will generate a first-pass sequence x̂ = {x̂1, x̂2 · · · x̂l}, and a series of hidden states Ĥ = {ĥ1, ĥ2 · · · ĥl}, where l is the length of the generated sequence. At step t, ĥt is calculated as ĥt = RNN(f(x̂t−1), ĥt−1), where f is a word embedding representation function. Then ĥt is fed into a softmax layer, and x̂t is sampled out from the obtained multinomial distribution. Finally, the first-pass output sequence x̂ is generated by G1. In the second-pass decoding, we take the hidden state of the last moment of G1 as the initial hidden state of G2. And we use an attention model in G2, Specifically, at step t, the attention model in G2 first generates a context ct defined as follows:
ct = l∑ i=1 aiĥi; ai ∝ exp (vTa tanh(Wattĥi +Uattht−1))∀i ∈ [1, l]; l∑ i=1 ai = 1. (1)
After receiving ct, we calculate the hidden state ht as ht = RNN([f(xt−1); ct],ht−1). Then xt is sampled in the same way of sampling x̂t, and the second-pass output sequence x is generated and its length is m.

heading: 2.3 MULTIPLE INSTANCE DISCRIMINATOR
text: Like Yu et al. (2017) and Wang & Wan (2018a), in order to evaluate the performance of a generator at the sentence-level, we use a discriminator to calculate rewards. Further, we want to get the importance of each word in the sentence to adjust the generator to decode the important words as much as possible in the first-pass decoding. Thus we adopt a multi-instance learning (MIL) mechanism in our discriminator. MIL deals with problems where labels are associated with groups of instances or bags (sentences in our case), while instance labels (word-level importance) are unobserved (Keeler & Rumelhart, 1991; Zhang et al., 2002; Wang & Wan, 2018b).
Our multiple instance discriminator D uses a structure similar to a bidirectional RNN. Supposing the input to D is a sentence w = {w1, w2, · · · , wn}, where n is the length of w. We apply a bidirectional LSTM on the sentence, and for each wordwt(t ∈ n), we concatenate its corresponding forward and backward hidden state vectors:
−→st = −−−−→ LSTM(wt);←−st = ←−−−− LSTM(wt); st = −→st ||←−st , (2)
where −−−−→ LSTM(wt) and ←−−−− LSTM(wt) represent the forward and backward LSTM hidden state vectors for wt. Then we use a softmax classifier to get the word-level probability distribution over target labels (i.e., real and fake).
pt = softmax(tanh(Wdst + bd), where pt = [p (t) real, p (t) fake]; p (t) real + p (t) fake = 1. (3)
p (t) real and p (t) fake represent the probability that the t-th word is predicted to be real and fake, respectively; Wd and bd are parameters, shared across all words. Finally, we use the average of the word-level distribution as the sentence-level distribution over the target labels:
pi = 1
n n∑ t=1 p (t) i ,∀i ∈ {real, fake};p = [preal, pfake]. (4)

heading: 2.4 TRAINING
text: Here we introduce the adversarial training of the generator and the discriminator in detail. In this paper, we formalize the text generation problem as a sequential decision making process (Bachman & Precup, 2015), which solves the problem that the gradient cannot pass back to the generative model when the output is discrete. Moreover, we use policy gradients to update parameters of the generator.
We propose two loss functions for the generator, named DelibGAN-I and DelibGAN-II. DelibGANI just forces the first-pass decoder G1 to maintain the same decoding result, while DelibGAN-II utilizes the result of the discriminator at the word level, forcing G1 to decode important words (evaluated by the discriminator) as much as possible. But for the second-pass decoder G2, both of them use the penalty-based objective function (Wang & Wan, 2018a) to update the parameters θG2 of G2. Specifically, at each timestep t, the existing sequence generated by G2 is x1:t−1 = {x1, ..., xt−1}, and the next token xt selected in the next step is an action sampling from the policy G2(xt|x1:t−1; θG2), and we use the Monte Carlo rollouts method (Dai et al., 2017; Yu et al., 2017) to simulate intermediate penalty RθG2 ,θD (xt), where θD is the parameters of the discriminator D. Then, the total penalty of G2 can be computed by:
Ex∼G2(x) = m∑ t=1 [(RθG2 ,θD (xt)− b)G2(xt|x1:t−1; θG2)]
= m∑ t=1 [( 1 q ( q∑ j=1 (1−D(xt|x(j)1:t−1; θD)))− b)G2(xt|x1:t−1; θG2)], (5)
where b is the bias, x(j)1:t−1 is the q-time Monte Carlo search sampled based on the roll-out policy G2(xt|x1:t−1; θG2), and D(xt|x (j) 1:t−1; θD) is the sentence probability preal given by the discriminator for sentence x(j)1:t−1||xt. DelibGAN-I: For DelibGAN-I, we just force the first-pass decoder G1 to maintain the same decoding result. Specifically, after we use (G1,G2) to sample a pair of sentences (x̂,x), we first use the discriminator to calculate the sentence-level penalties. Then in the process of using penalties to update the generator, we force the first-pass decoder G1 to decode the same result, which is x̂. The loss of generator G1 (the parameters are denoted as θG1 ) is defined as follows:
E(I)x∼G1(x̂) = 1
l l∑ t=1 − log(G1(x̂t|x̂1:t−1; θG1)) (6)
we propose it with the purpose of keeping the results of G1 as constant as possible, and just finetuning G2 based on the sentence-level penalty mentioned above. Finally, the objective of the generator is to minimize:
J (I) G1,G2(θG1 , θG2) = E (I) x∼G1(x̂) + Ex∼G2(x) =
1
l l∑ t=1 − log(G1(x̂t|x̂1:t−1; θG1))
+ m∑ t=1 [( 1 q ( q∑ j=1 (1−D(xt|x(j)1:t−1; θD)))− b)G2(xt|x1:t−1; θG2)]. (7)
DelibGAN-II: For DelibGAN-II, we want to impose direct influence to G1, forcing it to decode important words as much as possible in the first-pass decoding, to better guide the decoding of G2. We tried different ways of influencing, including using sentence-level penalties like Eq5, but finally we introduced and leveraged a multi-instance learning mechanism in the discriminator. This is because it can provide the importance of words in a sentence, and we add the word-level importance to the loss as weights. So the loss of generator G1 is changed as follows:
E(II)x∼G1(x̂) = 1
l l∑ t=1 −p(t)real log(G1(x̂t|x̂1:t−1; θG1)) (8)
where p(t)real is the probability that the discriminator predicts x̂t as real. Therefore, the objective of the generator is to minimize the total loss:
J (II) G1,G2(θG1 , θG2) = E (II) x∼G1(x̂) + Ex∼G2(x) =
1
l l∑ t=1 −preal(x̂) log(G1(x̂t|x̂1:t−1; θG1))
+ m∑ t=1 [( 1 q ( q∑ j=1 (1−D(xt|x(j)1:t−1; θD)))− b)G2(xt|x1:t−1; θG2)]. (9)
For discriminator D, the goal is to distinguish between real text and generated text as much as possible. It is worth noting that we only have the target label for the sentence, but do not have labels on words. The objective function of the discriminator is to minimize:
JD(θD) = −Ex∼G log pfake − Ex∼S log preal, (10)
where S is real sentences in the corpus. We perform the adversarial training of the generator and the discriminator, and train them alternately, as shown in Algorithm 1.
Algorithm 1 The adversarial training process in DelibGAN Input: Input noise, z; Generator, (G1 , G2); Discriminator, D; Real text dataset, S; Output: Well trained generator, (G1 , G2); 1: Initialize (G1 , G2),D with random weights; 2: Pre-train (G1 , G2) using MLE on S; 3: repeat 4: for d-steps do 5: Generate fake texts ({x̂}, {x}) using (G1,G2); 6: Update D using ({x},S) by minimizing Eq 10; 7: end for 8: for g-steps do 9: Generate fake texts ({x̂}, {x}) using (G1,G2); 10: Calculate penalty Ex∼G2(x) by Eq (5) ; 11: For DelibGAN-I, Update (G1,G2) by minimizing Eq (7); 12: For DelibGAN-II, Update (G1,G2) by minimizing Eq (9); 13: end for 14: until Model converges 15: return ;

heading: 3 EXPERIMENTS

heading: 3.1 SETUP
text: In this study, we hope that our unsupervised models can generate sentences with higher quality. Without loss of generality, we evaluate our model 1 on three benchmark datasets: a synthetic dataset, a descriptive text dataset and a sentimental text dataset. We compare with several typical and stateof-the-art unsupervised generic text generation models, including RNNLM(Mikolov et al., 2011), SeqGAN(Yu et al., 2017), SentiGAN(Wang & Wan, 2018a), MaskGAN(Fedus et al., 2018). It is worth noting that pre-training was used for all selected baselines.
After training models, we let them generate 1k sentences each. Measuring the quality of generated sentences has always been a difficult problem. For synthetic data, we have an oracle model to measure the negative log-likelihood (NLL) scores. But for real texts, we use automatic quantitative indicators (e.g., Fluency, Novelty and Diversity) and human evaluation methods (e.g., Grammaticality, Topicality and Overall) to evaluate the quality of generated sentences.
For automatic quantitative indicators, same as Wang & Wan (2018a), we use a language modeling toolkit - SRILM (Stolcke, 2002) to test the fluency of generated sentences, which calculates the perplexity of generated sentences using the language model trained on respective corpus. In addition,
1We will release the source codes of our methods upon the acceptance of this paper.
given the generated sentence set A and its training corpus set B, the novelty Novelty(A|B) and the diversity Diversity(A) are defined as follows:
Novelty(A|B) = 1 |A| ∑ i∈A (1−max{ϕ(i, j); ∀j ∈ B}), (11)
Diversity(A) = 1 |A| ∑ i∈A (1−max{ϕ(i, j);∀j ∈ A\{i}}), (12)
ϕ is the Jaccard similarity function. It is worth noting that in terms of the fluency indicator, a smaller value is better, but the novelty and diversity indicators are just the opposite.
For human evaluation, we randomly extracted 100 sentences from the generated sentences, and then let five experts rate each sentence according to its “grammaticality”, “topicality” and “overall” aspects, where “topicality” indicates whether the sentence accords with the topic/genre of the dataset (i.e., happy moment and sentimental text for the two real datasets, respectively). Scores range from 1 to 5, with 5 being the best. We finally calculate the average of the scores.

heading: 3.2 SIMULATION ON SYNTHETIC DATA
text: Here we use the synthetic dataset2 used by Yu et al. (2017), which consists of a set of sequential tokens which can be seen as the simulated data comparing to the real-word language data. Moreover, it automatically evaluates the negative log-likelihood (NLL) scores of our generated sequences, which brings us convenience. We compare our model with various models on this dataset, as shown in Table 1 and Figure 2. It is worth noting that the DelibMLE method is just using our coarse-to-fine generator.
From the results, we can see that: (1) Our coarse-to-fine generator (DelibMLE) performs better and even outperforms seqGAN. (2) Our proposed methods DelibGAN-I and DelibGAN-II outperform all other competitors with a large margin which means the framework we proposed is better than
2The synthetic data and the oracle model (LSTM model) are publicly available at https://github.com/LantaoYu/SeqGAN
the other models in capturing the dependency of the sequential tokens. (3) DelibGAN-II is better than DelibGAN-I, which indicates that it is useful to use the word-level information given by the discriminator during the first-pass decoding.

heading: 3.3 RESULTS ON DESCRIPTIVE TEXT DATA
text: In this section, we validate our model on the descriptive text corpus: HappyDB(Asai et al., 2018). HappyDB is a corpus of 100,000+ crowd-sourced happy moments. Its content is mainly about events that make people happy in the past, and 83% of corpora contain only one sentence. We only use those instances that contain one sentence, and we get a total of 83711 happy moments. Table 2 shows the results on the automatic quantitative indicators, and we can see that DelibGAN-I and DelibGAN-II perform exceptionally well, with the ability of keeping the fluency of sentences, generating sentences different from that in the training corpus and generating a variety of sentences.
Further, we use human evaluation to evaluate generated sentences. Table 3 shows the results, and we can see DelibGAN-I and DelibGAN-II outperform other models, especially in topicality.

heading: 3.4 RESULTS ON SENTIMENTAL TEXT DATA
text: Here we validate our model on the sentimental text corpus: Stanford Sentiment Treebank (SSTB, Socher et al. (2013)), which contains 9613 sentimental reviews of movies. Tables 4 and 5 show the results of automatic quantitative indicators and human evaluation, respectively. Both results show that our models DelibGAN-I and DelibGAN-II produce more realistic samples of sentimental texts than other models, and DelibGAN-II performs best.

heading: 3.4.1 CASE STUDY
text: In Table 6, we show example sentences generated by DelibMLE, DelibGAN-I and DelibGAN-II. From the examples, we can see that: (1) In DelibMLE, there is no significant correlation between the first-pass decoding result G1 and the second-pass decoding result G2. (2) Compared with DelibGANI, DelibGAN-II is more likely to decode important words in G1, and those words behave like keywords in HappyDB and sentiment words in SSTB.

heading: 4 RELATED WORK
text: Unsupervised text generation models are usually based on RNN(Mikolov et al., 2011), which easily suffers from exposure bias(Bengio et al., 2015) and the inconsistency between word-level loss function and sentence-level evaluation(Wang & Wan, 2018a). Several strategies have been exploited later, including Professor Forcing (Goyal et al., 2016), Scheduled SamplingBengio et al. (2015) and MaskMLE(Fedus et al., 2018).
Generative Adversarial Networks (GAN)(Goodfellow et al., 2014) has also been applied for text generation. Due to the non-differentiable problem and more complex modes of languages, the quality of generated texts are usually not very satisfactory. Some studies apply GAN in text generation by modifying the word prediction function, such as Gumbel-softmax distribution(Kusner & HernándezLobato, 2016). Some other works apply reinforcement learning to solve the non-differentiable problem, including SeqGAN(Yu et al., 2017), LeakGAN(Guo et al., 2018), RankGAN(Lin et al., 2017), SentiGAN(Wang & Wan, 2018a) and MaskGAN(Fedus et al., 2018). However, they all adopt an one-pass forward decoding , which can only see words that have been decoded.
Recently, Deliberation networks(Xia et al., 2017), which has two decoders and produces better sequences by modifying sketches, has achieved good results on machine translation tasks. Later, Lapata & Dong (2018) also propose a similar structure and make meaningful settings for the sketches. But this is limited to specific tasks such as logical form parsing, code generation and SQL query generation. Moreover, these methods are supervised learning methods.
Multiple Instance Learning (MIL)(Keeler & Rumelhart, 1991) deals with problems where labels are associated with groups of instances or bags (sentences in our case), while instance labels (word-level importance in our case) are unobserved. The initial MIL makes a strong assumption that a bag is negative only if all of its instances are negative, and positive otherwise(Dietterich et al., 1997; Maron & Ratan, 1998; Zhang et al., 2002), and subsequent works relax this assumption and make it more suitable for the task at hand(Cour et al., 2011; Kotzias et al., 2015; Wang & Wan, 2018b).

heading: 5 CONCLUSIONS AND FUTURE WORK
text: In this paper, we propose a novel adversarial learning framework DelibGAN to generate high-quality sentences without supervision, by making use of a coarse-to-fine generator and a multiple instance discriminator. Evaluation results on various datasets verifies the efficacy of our proposed models. In future work, we will try to make the sketch more meaningful and apply our framework to more tasks.

Review summaries:
The main contribution of this paper is the proposed DelibGAN for text generation. The framework introduces a coarse-to-fine generator, which contains a first-pass decoder and a second-pass decoder. Instead of using a binary classifier, the discriminator is a multiple instance discriminator. Two different variants of DelibGAN are proposed, with experiments showing that DelibGAN-II performs the best.