Title: RELGAN: RELATIONAL GENERATIVE ADVERSARIAL NETWORKS FOR TEXT GENERATION

Abstract: Generative adversarial networks (GANs) have achieved great success at generating realistic images. However, the text generation still remains a challenging task for modern GAN architectures. In this work, we propose RelGAN, a new GAN architecture for text generation, consisting of three main components: a relational memory based generator for the long-distance dependency modeling, the Gumbel-Softmax relaxation for training GANs on discrete data, and multiple embedded representations in the discriminator to provide a more informative signal for the generator updates. Our experiments show that RelGAN outperforms current state-of-the-art models in terms of sample quality and diversity, and we also reveal via ablation studies that each component of RelGAN contributes critically to its performance improvements. Moreover, a key advantage of our method, that distinguishes it from other GANs, is the ability to control the trade-off between sample quality and diversity via the use of a single adjustable parameter. Finally, RelGAN is the first architecture that makes GANs with Gumbel-Softmax relaxation succeed in generating realistic text.

1. INTRODUCTION:

Generative adversarial networks (GANs) (Goodfellow et al., 2014) were originally designed to generate continuous data and have achieved a lot of success at generating continuous samples, such as images. Recently, GANs were extended to generate discrete data, in particular text sequences (Kusner & Hernández-Lobato, 2016; Yu et al., 2017; Zhang et al., 2017; Lin et al., 2017; Guo et al., 2017; Fedus et al., 2018) . However, this extension is not straightforward. The main issue is that outputs of GANs for the discrete data generation are not differentiable and thus the standard gradient-based techniques cannot be applied directly in these settings. To overcome this, most state-of-the-art GANs have used the REINFORCE algorithm (Williams, 1992) and its variants that originate from the reinforcement learning (RL) community to train the generator while the discriminator is still a classifier to discriminate real and generated text and provides reward signals for the generator updates. A detailed description of the related work is deferred to Appendix 4.

Although these state-of-the-art GANs have shown some promising results in text generation as compared to the conventional maximum likelihood estimation (MLE) method, they also suffer from some fundamental issues, including training instability and mode collapse. First, their performance is quite sensitive to random parameter initializations and hyperparameter choices (Semeniuta et al., 2018) . Moreover, many GANs heavily employ RL heuristics, such as Monte Carlo search (Yu et al., 2017) and hierarchical RL (Guo et al., 2017) , making the already difficult-to-train GANs more complicated and thus the individual role of adversarial training unclear. The second issue is mode collapse as the generated text sentences tend to be less diverse (Semeniuta et al., 2018; Fedus et al., 2018) and it becomes more severe when generating longer sentences. The mode collapse issue can be caused either by a lack of expressive power in the generator (since it may not be capable of covering many more complex modes in data distribution), or by a less informative guiding signal in the discriminator (as it may constrain the generator updates to within certain modes).

In this work, we propose a new GAN architecture -Relational GAN (RelGAN), whose design is motivated by the issues identified above. The RelGAN architecture mainly consists of three parts: 1) a relational memory (Santoro et al., 2018) based generator, which promises more expressive power and better ability of modeling longer-range dependencies in text; 2) Gumbel-Softmax relaxation (Jang et al., 2016; Maddison et al., 2016) for training GANs on discrete data, which simplifies our model, enabling us to stay within a classical GAN framework without intensive RL heuristics; 3) multiple embedded representations in the discriminator, enabling a more diverse and informative signal for the generator updates. We experimentally demonstrate that RelGAN outperforms most current models in terms of sample quality and diversity. Furthermore, we show via ablation studies that each part of RelGAN plays an important role in its performance improvements. A key advantage of our method, that distinguishes it from other GANs, is the ability to control the trade-off between sample quality and diversity, via the use of a single adjustable parameter. Finally, to the best of our knowledge, RelGAN is the first architecture to demonstrate that GANs with Gumbel-Softmax relaxation are capable of generating realistic text.

2. RELGAN 2.1 RELATIONAL MEMORY BASED GENERATOR:

Current dominant GANs for text generation, such as Kusner & Hernández-Lobato (2016); Yu et al. (2017) ; Lin et al. (2017) ; Guo et al. (2017) ; Fedus et al. (2018) are built using LSTM (Hochreiter & Schmidhuber, 1997) as the generator architecture. However, the LSTM-based generator might be the bottleneck of GANs from the following experimental observations: 1) The discriminator's loss value very quickly goes to near its minimum after few iterations of adversarial training. It means that the discriminator may be much more powerful than the generator and can easily distinguish between real and fake samples. 2) Mode collapse in current GANs (Fedus et al., 2018) may also partly indicate the incapacity of generator, as it may not be expressive enough to fit all the modes of data distribution. 3) Current GANs perform poorly at long sentence generation (Guo et al., 2017) , and we know that LSTM packs all information about the previous text sequences into a common hidden vector, potentially limiting its ability of modeling the long-distance dependency.

Therefore, we propose to use the more powerful module -relational memory (Santoro et al., 2018) -as the generator architecture for text generation. The basic idea of relational memory is to consider a fixed set of memory slots (e.g. memory matrix) and allow for interactions between memory slots by using the self-attention mechanism (Vaswani et al., 2017) . The empirical findings by Santoro et al. (2018) showed that relational memory performs better in the language modeling compared to LSTM. Intuitively, the use of multiple memory slots and the attention across these memories can increase the expressive power of generator and its ability of generating longer text sentences.

Formally, we assume each row of the memory M t represents a memory slot and Figure 1 shows how self-attention updates M t by incorporating new observation x t at time t. Given H heads, we have H sets of queries, keys and values via three linear transformations, respectively: For each head, we get query

Q (h) t = M t W (h) q , key K (h) t = [M t ; x t ]W (h) k and value V (h) t = [M t ; x t ]W (h) v

where [; ] denotes the row-wise concatenation. Thus, the updated memory Mt+1 is given by

Mt+1 = [ M (1) t+1 : • • • : M (H) t+1 ], M (h) t+1 = σ M t W (h) q ([M t ; x t ]W (h) k ) T √ d k [M t ; x t ]W (h) v (1)

where σ(•) denotes the softmax function which is performed on each row, d k is the column dimension of the key K (h) t

and [:] denotes the column-wise concatenation.

By following the same idea of Santoro et al. (2018) , the next memory M t+1 and output (logits) o t of the generator at time t are given by

M t+1 = f θ1 ( Mt+1 , M t ), o t = f θ2 ( Mt+1 , M t ) (2)

respectively, where the two parametrized functions f θ1 and f θ2 are combinations of skip connections, multi-layer perceptron (MLP), gated operations and/or pre-softmax linear transformations. 

Q (h) t , K (h) t

and V

(h) t denote the queries, keys and values, respectively. Note that the softmax function is performed on each row, and ⊗ denotes the dot product. The concatenation (denoted by "concat") of Mt and xt is row-wise where the embedded input is first passed through a linear layer to make xt match the row dimension of Mt.

2.2. TRAINING WITH DISCRETE DATA:



2.2.1. GUMBEL-SOFTMAX RELAXATION:

Before the introduction of Gumbel-Softmax relaxation, we first show why training GANs with discrete data is a critical issue. Assuming the vocabulary size is V , for the output logits o t ∈ R V of the generator in (2), the next generated one-hot token y t+1 ∈ R V will be obtained by sampling:

EQUATION

where similarly σ(•) denotes the softmax function which is performed on o t element-wisely and we use σ(o t ) to represent the multinomial distribution on the set of all possible tokens. As we know, the sampling operations in (3) on the multinomial distribution of the generator output are not differentiable, which implies a step function at the end of the generator. Since the derivative of a step function is 0 almost everywhere, we have ∂yt+1 ∂θ G = 0 a.e. for t = 0, • • • , T -1 where θ G denotes the generator parameters. By chain rule, the gradients of the generator loss l G w.r.t. θ G will be

EQUATION

So the gradients of the generator loss cannot pass back to the generator via the discriminator. This is the notorious "non-differentiability issue" of GANs in discrete data generation.

To deal with the non-differentiablity issue, we apply the Gumbel-Softmax relaxation technique which defines a continuous distribution over the simplex that can approximate samples from a categorical distribution (Jang et al., 2016; Maddison et al., 2016) . Formally, the Gumbel-Softmax relaxation includes two parts: 1) The Gumbel-Max trick. According to Jang et al. (2016); Maddison et al. (2016) , the sampling in (3) can be reparametrized as

EQUATION

where o

(i)

t denotes the i-th entry of o t and g (i)

t is from the i.i.d. standard Gumbel distribution, i.e. g

(i) t = -log(-log U (i) t ) with U (i) t

∼ Uniform(0, 1). 2) Relaxing the discreteness. As the arg max operation in (5) is still non-differentiable, we need further approximate the "one-hot with arg max" by softmax, which yields

EQUATION

where β > 0 is a tunable parameter called inverse temperature. As ŷt+1 in ( 6) is differentiable with respect to o t , we can use ŷt+1 instead of the one-hot token y t+1 as the input of the discriminator. Also, note that the new observation x t+1 of the generator to be concatenated with M t+1 in next time t + 1 is given by x t+1 = f θ3 (y t+1 ), where the parametrized function f θ3 is composed of an embedding layer that maps y t+1 to an embedded input and a linear layer that makes x t+1 match the row dimension of M t+1 (the embedded input and the linear layer are shown in Figure 1 ) . 

2.2.2. TEMPERATURE CONTROL:

With the larger inverse temperature β, ŷt+1 in (6) will become a better approximation of y t+1 in (3) and asymptotically as β → ∞, ŷt+1 → y t+1 . However, the issue is that the variance of gradients will be very large as we have Var( ∂ ŷt+1 ∂ot ) ∝ β 2 , and thus the parameter updates will become very sensitive to the input noise. Intuitively, this might cause poor sample quality. On the other hand, with the smaller inverse temperature β, the generator will pay more attention to making a sharp distribution of entries in ŷi+1 due to the larger (initial) approximation gap between ŷt+1 and y t+1 , which implicitly discourages its possible "exploration". Intuitively, this might be one factor that contributes to mode collapse of RelGAN on text generation.

Therefore, the larger β encourages more exploration for better sample diversity while the smaller β encourages more exploitation for better sample quality. We thus propose to increase the inverse temperature β over iterations via an exponential policy:

β n = β n/N

max , where β max denotes the maximum inverse temperature, N is the maximum number of training iterations and n denotes the current iteration. In the exponential policy, as the increase rate of inverse temperature depends on β max , β max will decide the transition time from the exploitation phase to the exploration phase. In such sense, RelGAN provides a flexibility of either generating more diverse samples with a large β max or generating better quality samples with a small β max while most current GANs cannot provide.

2.3. MULTIPLE REPRESENTATIONS IN DISCRIMINATOR:

A commonly used discriminator for text generation is a CNN-based classifier (Kim, 2014) that employs a convolutional layer with multiple filters of different sizes to capture relations of various word lengths and a max-pooling layer over the entire input sentence for each feature map (Yu et al., 2017; Zhang et al., 2017; Lin et al., 2017; Guo et al., 2017) . In this discriminator architecture, the input of the CNN-based classifier is a sentence of length T represented by a single embedded matrix X ∈ R d×T where each column xt ∈ R d is a d-dimensional embedded vector of each word.

In this work, we propose a new discriminator framework that applies multiple embedded representations for each sentence, with each representation independently passed through the above CNNbased classifier to get an individual score. The average of these individual scores will serve as the final guiding information to update the generator. Our hypothesis is that each embedded representation may capture a specific aspect of the input sentence and the discriminator which compares real and generated sentences from these different perspectives can provide more diverse and comprehensive guiding information for the generator updates. This idea resembles the use of multiple discriminators to improve GANs on image generation (Durugkar et al., 2016) , but the difference is that we only use multiple different representations of the input while still keeping a single or weight-sharing CNN-based classifier, which presumably has much less computational cost.

Formally, we assume that r t denotes the t-th one-hot real token and ŷt from (6) denotes the tth softmax-like generated token, and Figure 2 shows the proposed discriminator framework with multiple embedded representations where either the real input [r 1 :

• • • : r T ] ∈ R V ×T or the generated input [ŷ 1 : • • • : ŷT ] ∈ R V ×T will be mapped into S embedded representations by S distinct embedding matrices {W (s) e } S s=1 with W (s) e ∈ R d×V . Let X(s)

r and X(s) y be the s-th embedded representation of the real and generated input, respectively. Thus, we have

X(s) r = W (s) e [r 1 : • • • : r T ], X(s) y = W (s) e [ŷ 1 : • • • : ŷT ] (7)

and the final discriminator loss is given by

l D = 1 S S s=1 E r 1:T ∼Pr ŷ1:T ∼P θ f (D( X(s) r ), D( X(s) y )) (8)

where the expectation is taken w.r.t. both real sentence distribution P r and generated sentence distribution P θ , and the loss function f is determined by the specific GAN loss, such as vanilla GAN (Goodfellow et al., 2014) , f -GAN (Nowozin et al., 2016) and WGAN (Arjovsky et al., 2017) . Throughout this paper, the generator loss can be simply set to be l G = -l D .

2.4. TRAINING TECHNIQUES:

Choice of Loss Function. Empirically, we first compared three different standard GAN losses: standard GAN (the non-saturating version) (Goodfellow et al., 2014) , hinge loss (Nowozin et al., 2016; Zhang et al., 2018) 

EQUATION

Intuitively, the loss function in ( 9) is to directly estimate the average probability that real sentences are more realistic than generated sentences in terms of different embedded representations.

Generator Pre-training. Most current GANs for text generation need the pre-training for both generator and discriminator, such as SeqGAN (Yu et al., 2017) , and some may further heavily rely on the exclusive pre-training techniques, such as TextGAN (Zhang et al., 2017) , LeakGAN (Guo et al., 2017) and MaskGAN (Fedus et al., 2018) . Instead, the proposed RelGAN only need to pre-train the generator simply via the standard MLE training for several epochs before starting the adversarial training. Experimentally, we find that a good initialization for generator provided by the MLE pre-training is necessary for a good convergence behavior of adversarial training.

3. EXPERIMENTS:

We test RelGAN on both synthetic and real data, where the synthetic data are 10,000 discrete sequences generated by an oracle-LSTM with fixed parameters (Yu et al., 2017) and the real data include the COCO image captions (Chen et al., 2015) and EMNLP2017 WMT News, first used by Guo et al. (2017) for text generation. The experimental settings are given in Appendix A.

Evaluation Metrics. How to properly evaluate generative models remains an open research question (Theis et al., 2015; Semeniuta et al., 2018) . The key issue plaging current evaluation metrics for GANs is that they cannot measure sample quality and sample diversity simultaneously. Therefore, we use two distinct metrics: for synthetic data, we use both negative log-likelihood (called NLL gen ) and its counterpart (called NLL oracle ), defined as:

EQUATION

)

where the generated sentence distribution P θ and the real sentence distribution P r are both known by evaluating the generator and oracle-LSTM, respectively. Generally, NLL gen measures sample diversity while NLL oracle is more sensitive to sample quality (Theis et al., 2015; Arjovsky & Bottou, 2017) . For the real dataset, we also apply NLL gen to measure the sample diversity, similar to Lu et al. (2018) . However, since NLL oracle cannot be evaluated without an oracle, we instead apply the commonly-used BLEU scores (Papineni et al., 2002) to measure the sample quality and compare with the MLE baseline, along with other start-of-the-art GANs, including SeqGAN (Yu et al., 2017) , RankGAN (Lin et al., 2017) and LeakGAN (Guo et al., 2017) . Note that for BLEU score evaluation, we follow the strategy in (Yu et al., 2017; Zhu et al., 2018 ) by using the test data as the reference.

3.1. SYNTHETIC DATA:

We run the synthetic data experiments with sequence length 20 and 40, respectively. The NLL oracle results of RelGAN and other models are shown in Table 1 where we set β max = 1 for length 20 and β max = 2 for length 40. Note that "MLE" in Table 1 denotes the baseline model where LSTMs are trained with the teacher-forcing algorithm to maximize the likelihood (same with Table 2 and 3 ). We can see that RelGAN outperforms other models in both cases, and its lead in performance becomes larger with longer sequence length, demonstrating the log-distance dependency modeling ability of the proposed generator. 

3.2. COCO IMAGE CAPTIONS DATASET:

In order to test RelGAN on real-world data, we first run experiments on the COCO Image Captions dataset. By following the same data pre-processing as in Zhu et al. (2018) , the dataset includes 4,682 unique words with the maximum sentence length 37. Both the training and test data contain 10,000 sentences.

The BLEU scores of RelGAN compared with previous models are shown in Table 2 where we set β max = 100 and 1000, respectively. We can see that RelGAN is significantly and consistently better than other models in terms of all the BLEU scores, which means its ability of generating high-quality sentences of COCO Image Captions. Furthermore, the NLL gen scores of RelGAN and previous models are also shown in 0.849 ± 0.030 0.687 ± 0.047 0.502 ± 0.048 0.331 ± 0.044 0.756 ± 0.054 RelGAN (1000) 0.814 ± 0.012 0.634 ± 0.020 0.455 ± 0.023 0.303 ± 0.020 0.655 ± 0.048 Table 2: The BLEU and NLLgen scores on COCO Image Captions where βmax = 100 and 1000, respectively. RelGAN is run with 6 random seeds and the final score is obtained by taking the average of scores, and other scores are based on the same evaluation settings in Zhu et al. (2018) . For BLEU scores, the higher the better.

3.3. EMNLP2017 WMT NEWS DATASET:

The EMNLP2017 WMT News dataset consists of 5,255 unique words with the maximum sentence length 51 after applying the same data pre-processing as in Zhu et al. (2018) The BLEU and NLLgen scores on EMNLP2017 WMT News where βmax = 100 and 1000, respectively. Our model is run with 6 random seeds and the final score is obtained by taking the average of scores, and other scores are based on the same evaluation settings in Zhu et al. (2018) .

The BLEU scores of RelGAN compared with previous models are shown in Table 3 where we set β max = 100 and 1000, respectively. We can see that RelGAN also consistently outperforms previous models in terms of all the BLEU scores, demonstrating its ability of generating high-quality sentences on EMNLP2017 WMT News. Moreover, the sample diversity metric NLL gen scores of RelGAN and previous models are also shown in Table 3 . Similarly, RelGAN achieves the stateof-the-art results in terms of sample diversity. Upon visually examining generated samples (See Appendix C.2 for more details), we do not observe obvious mode collapse for β max ∈ {100, 1000}.

Finally, from Tables 2 and 3 , we can see that the sample quality and diversity trade-off with different values of maximum inverse temperature β max also exists on the real data. That is, RelGAN with β max = 100 achieves better sample quality while RelGAN with β max = 1000 achieves better sample diversity. Depending on what the underlying applications of text generation via RelGAN are, we can adjust β max properly to get either better quality or better diversity.

To further evaluate the sample quality of RelGAN and other models on EMNLP2017 WMT News, we also perform the human evaluation by using Amazon Mechanical Turk. We randomly sampled 100 sentences for each model and the real dataset, and asked 10 different people to score each sentence on a scale of 1-5. Please see Appendix A.2 for more details of human evaluation. 4 : The means and standard deviations of human scores for RelGAN and other models on EMNLP2017 WMT News by using Amazon Mechanical Turk. Note that "Real" denotes the human score on the real dataset.

The human score results are provided in Table 4 , where we can see that RelAGN generates better human-looking samples than other GANs and the MLE baseline model.

Published as a conference paper at ICLR 2019 3.4 ABLATION STUDY

3.4.1. IMPACT OF RELATIONAL MEMORY:

To show the impact of relational memory in RelGAN, we propose to replace relational memory by LSTM-32 and LSTM-512 as the generator architecture, respectively, and see how the performance differs. Here LSTM-k represents the LSTM with hidden dimension being k. We choose k = 32 because most previous GANs (Yu et al., 2017; Guo et al., 2017) have used this architecture for text generation, and also choose k = 512 because for more fair comparison, we want to keep the total memory size of LSTM to be the same with the relational memory we have used.

The results on the COCO Image Captions dataset are shown in Figure 4 (Left), where we provide the BLEU-4 score (See Appendix D.1 for all the BLEU scores). We can see that the BLEU scores of relational memory are consistently better than those of LSTM-32 and LSTM-512, which demonstrates the advantages of using relational memory as generator in RelGAN. 

3.4.2. IMPACT OF GUMBEL-SOFTMAX RELAXATION:

To show the impact of Gumbel-Softmax relaxation in RelGAN, we can instead apply the vanilla REINFORCE method to deal with the non-differentiable issue of RelGAN on text generation. In this experiment, we keep all other hyperparameters in RelGAN fixed and compare the performance of Gumbel-Softmax relaxation and the vanilla REINFORCE method.

The results are shown in Figure 4 (Right), where the BLEU-2 score is provided (See Appendix D.2 for all the BLEU scores). We can see that under the proposed RelGAN framework, Gumbel-Softmax relaxation performs much better than the vanilla REINFORCE method. During experiments, we find that the variance of generator gradients in the vanilla REINFORCE method is too large to provide any useful update for generator, which may explain why the performance of vanilla REINFORCE does not improve after the pre-training, as observed in Figure 4 (Right). The exploration of various variance reduction techniques for the REINFORCE method in RelGAN is out of scope of this paper.

3.4.3. IMPACT OF MULTIPLE REPRESENTATIONS IN DISCRIMINATOR:

To show the impact of multiple embedded representations in discriminator while keeping the expressive power of discriminator the same for fair comparison, we propose to apply S embedded presentations with each embedded vector of length d = dmax S where d max denotes the total length of representations. In this experiment, we set d max = 64, and thus for instance, if S = 1 then d = 64 for each embedded vector, and if S = 2 then d = 32 for each embedded vector, and so on. We first test RelGAN on the synthetic data with S ∈ {1, 2, 4, 8, 16, 32, 64} and the results are shown in Figure 5 (Left). We can see that as the number of embedded representations S increases, the best NLL oracle score tends to keep decreasing, yielding better sample quality. Furthermore, we test Rel-GAN on COCO Image Captions with S ∈ {1, 64} and the BLEU-3 score is shown in Figure 5 (Right). Still, we can see that the BLEU scores of RelGAN with S = 64 are consistently better than those of RelGAN with S = 1 (see Appendix D.3 for all the BLEU scores). Note that in both experi- ments, we do not see an obvious sign of mode collapse with varying number of representations. For example, the NLL gen score on the synthetic data stays around 4.4 for different values of S (close to the best NLL gen 4.2 for MLE shown in Figure 3 (Left)). Thus, these experiments demonstrate the advantages of using multiple embedded representations for discriminator in RelGAN.

4. RELATED WORK:

Since GANs are originally proposed for continuous data, extending GAN training to discrete data generation has been an active research topic. Current works focus on dealing with the nondiferentiable issue brought by the discrete data nature either by considering the RL methods or by reformulating the problem in continuous space.

A large class of GANs for text generation relies on the RL algorithm. SeqGAN ( (Yu et al., 2017) ) models the text generation as a sequential decision making process and trains the generator with policy gradient methods (Sutton et al., 2000) . MaliGAN (Che et al., 2017) proposes the co-training with a maximum-likelihood objective to reduce the gradient variance. RankGAN (Lin et al., 2017) proposes a ranking model to replace the original binary classifier as the discriminator. LeakGAN (Guo et al., 2017) designs a mechanism to provide intermediate information about text generation for generator, where the discriminator can leak its features through a manager module. MaskGAN (Fedus et al., 2018) introduces an actor-critic conditional GAN that fills in missing text conditioned on the surrounding context by resorting to a seq2seq model.

Other GANs without RL methods either approximate the discrete data or work in the continuous latent space. TextGAN (Zhang et al., 2017) provides a feature matching mechanism that matches the latent features of real and generated sentences via a kernelized discrepancy metric to alleviate the mode collapse. FM-GAN (Chen et al., 2018) proposes to match the latent feature distributions of real and synthetic sentences using the feature-movers distance. Similar to our work, both textGAN and FM-GAN apply an annealed softmax to approximate the argmax in the generator. However, they do not rely on the Gumbel-Max trick to reparametrize the sampling operations, which is the major difference with us in dealing with the non-differentiable issue. ARAE (Zhao et al., 2018) applies an additional autoencoder to embed the discrete data into a continuous latent space in which GANs can be trained properly. As for approximating the categorical distribution with Gumbel-Softmax relaxation, Gu et al. (2017) Attention mechanisms, in particular self-attention (Vaswani et al., 2017) , have gradually become a building block of many novel neural network architectures (Vaswani et al., 2017; Parmar et al., 2018; Santoro et al., 2018) due to its ability of capturing long or global dependencies and reducing computational cost via parallelization. In the context of GANs, self-attention have not been fully explored. SAGAN (Zhang et al., 2018) applies self-attention in GANs to model long range depen-dencies in images and get the state-of-the-art results on conditional image generation. In contrast, we employ self-attention in GANs for text generation by using relational memory as generator.

5. CONCLUSIONS:

We proposed a new GAN architecture called RelGAN for text generation, that outperforms most current models in terms of sample quality and diversity on both synthetic and real data. Furthermore, the trade-off between the generated sample diversity and quality can be adjusted properly in RelGAN by controlling the inverse temperature. In RelGAN, we used the relational memory based generator to improve its ability of modeling long distance dependencies and also applied multiple embedded representations in discriminator such that it can provide more diverse and informative guiding signal for generator. By applying Gumbel-Softmax relaxation to deal with the non-differentiable issue, our architecture is simple to implement without employing intensive RL heuristics. For the future directions, since we have demonstrated that GANs with Gumbel-Softmax relaxation is very promising for text generation, we would like to explore further in this direction. For example, it is interesting to make RelGAN work better without any pre-training. Also, extending RelGAN to a conditional model for many text generation related applications is another interesting direction.

A EXPERIMENTAL SETTINGS:

A.1 TRAINING DETAILS Unless stated otherwise, for the CNN-based discriminator architecture, we use filter windows of sizes {3,4,5} and 300 feature maps each. For relational memory, we set memory size to be 256, memory slots to be 1, number of heads to be 2. The batch size is set to be 64. For embedding dimensions, we set the embedding dimension of the input token for generator to be 32 and that for discriminator to be 1 with the number of embedded representations S = 64. We use Adam (Kingma & Ba, 2014) with β 1 = 0.9 and β 2 = 0.999 and gradient clipping is applied if the norm of gradients exceeds 5. We first pre-train the generator via MLE with learning rate of 1e-2 for 150 epochs and then start adversarial training with learning rate of 1e-4 for both discriminator and generator. For adversarial training, we set the maximum number of iterations N = 5000 and we perform 5 gradient descent steps on the discriminator for every step on the generator.

A.2 HUMAN EVALUATION DETAILS:

On Amazon Mechanical Turk, our instructions are given as follows:

The text quality evaluation is based on grammatical correctness and meaningfulness (i.e. if a sentence makes sense). Please ignore any text formatting problems (e.g., capitalization, punctuation, spelling errors, extra spaces between words and punctuations). Note: A very short sentence (less than 10 words) should be penalized with its score minus 1. Please see below for the detailed criteria.

Scale:

Criterion & Example

5. -Excellent:

Its grammatically correct and makes sense. For example, "if England wins the World Cup next year , it will be the most significant result the sport has seen in more than a decade ."

4. -Good:

It has some small grammatical errors and mostly makes sense. For example, "it is useful to have had a doctor who forced her to release him a couple of days before she was cleared ."

3. -Fair:

It has major grammatical errors but the whole still conveys some meanings. For example, "even then once again there ' s a sign of that stuff is going on the way to work on christmas eve ."

2. -Poor:

It has severe grammatical errors and the whole doesn't make sense, but some parts are still locally meaningful.

For example, "we go to work for the moment in life their eyes and , i have been a different race on to go ."

1. -Unacceptable:

It is basically a random collection of words. For example, "i go com com com , i on on on play can go go ."

Table 5 : The human evaluation scale from 1 to 5 with corresponding criteria and example sentences.

B RELGAN WITH VARYING HYPERPARAMETERS (Jolicoeur-Martineau, 2018) , standard GAN (the non-saturating version) (Goodfellow et al., 2014) and hinge loss (Nowozin et al., 2016; Zhang et al., 2018) , where βmax = 1000 and we use two different optimizers - a rectangle shaped wooden sitting on a lush green field .

a man standing in a picture of a kitchen with a dog watching him .

a man is carving some meat in a park .

a kitchen with a black window and a large white stove wall .

a cat is looking on a man in a bathroom .

a train is covered in the air in the city scene .

a bathroom has a toilet , and bathroom rug for a urinal or a urinal on the side .

a home kitchen with a double oven and table while a table and chairs .

a small airplane flying above an airport covered with wood .

a man in a kitchen preparing food on a table .

a large passenger jet flying in a clear blue sky .

a group of people riding motorcycles on a street .

a woman in a kitchen with her hands clasped .

a smiling woman is sitting on a green bench .

people are hiding under colorful umbrellas on a rainy day .

a photo of a small restroom in a kitchen .

many sheep graze are shown in front of a group of people .

a plane is parked next to an airplane on a runway near a control tower with two back .

a woman walking past a straw shower holding a corner .

an office desk with a row of books in the kitchen .

a cat sitting on top of a kitchen counter .

a person riding a bike through a lush green park .

a bathroom with a sink , toilet and toilet paper dispenser .

a man is on a motorcycle with a woman on the back of it .

two giraffes in a wild , lightly wooded field .

a metal tin pan filled with two different kitchen appliances .

a white airplane flying in the sky over a runway .

a car driving on a busy street at night from an airport .

Table 6 : Randomly chosen 15 generated samples trained on COCO image caption data with βmax = 100 (top row) and βmax = 1000 (bottom row). We can see that for βmax = 100, the words "man" and "kitchen" occur severe times even in these small set of generated sentences, meaning some sort of lacking sentence diversity.

For βmax = 1000, however, there is no sign of obviously poor sentence diversity by human eyes. Interestingly, Figure 10 shows that during the adversarial training, the sample diversity (measured by NLL gen ) of RelGAN first decreases and then increases, and its turning point matches well with that of the sample quality (measured by BLEU scores) of RelGAN. These training dynamics illustrate a diversity-quality transition over iterations in RelGAN: Early on in training, it learns to aggressively improve sample quality while sacrificing diversity. Later on, it turns instead to maximizing sample diversity while gradually decreasing sample quality. Intuitively, it seems to be much easier for the generator to just produce realistic samples -regardless of their diversity -to fool the discriminator in the early stage of training. As the discriminator becomes better at distinguishing samples with less diversity over iterations, the generator has to focus more on producing more diverse samples to fool the discriminator.

F RELGAN WITHOUT PRE-TRAINING:

In this section, we want to test the performance of RelGAN without pre-training for different loss functions, including standard GAN (the non-satuarating version) (Goodfellow et al., 2014) , WGAN-GP (Gulrajani et al., 2017) , hinge loss (Nowozin et al., 2016; Zhang et al., 2018) All the results are run with 6 random seeds and the final score is obtained by taking the average of scores. As a reference, we also provide the results of an untrained RelGAN which is marked as "random". Note that for the hinge loss, we get no valid results as it suffers from the vanishing gradient issue.

We can see that without pre-training, there is still a significant improvement for RelGAN compared with the random generation, in particular for the standard GAN loss, even though the improvement is inferior to the case with pre-training. In contrast, without pre-training, previous RL-based GANs for text generation, such as SeqGAN and RankGAN, always get stuck around their initialization points and are not able to improve their performance at all. This demonstrates that RelGAN may be a more promising GAN architecture to explore in order to completely get rid of the pre-training for GANs on text generation. Besides, Table 8 also shows that the evaluation results of RelGAN without pretraining vary with different loss functions and values of β max . We leave an extensive hyperparameter search to further improve the performance of RelGAN without pre-training for future work.

G MORE EXPLORATION ON TUNABLE HYPERPARAMETER β max:

For real data experiments, we have showed the advantages of RelGAN over other models by setting the maximum inverse temperature β max ∈ {100, 1000}, which are carefully chosen for a good tradeoff between sample quality and diversity. A natural question will be to explore the two extremes: what happens with the real data if β max is either too large or too small? Does it behave similarly to the synthetic data experiments in terms of the trade-off between sample diversity and quality?

To this end, we choose a broad range of β max ∈ {1, 10, 10 2 , 10 3 , 10 4 , 10 5 , 10 6 , 10 7 } and test its impact in RelGAN on the COCO Image Captions dataset. The BLEU and NLL gen scores are given in Table 9 , where we can see that both BLEU and NLL gen scores increase with the decrease of β max , and the variance of each score also consistently becomes larger for a smaller β max . For better illustration, we also plot BLEU-4 and NLL gen scores with error bars in Figure 11 .

It first confirms that similar to the synthetic data experiments, there also exists a consistent trade-off between sample quality and diversity in real data, controlled by the tunable hyperparameter β max . Besides, it reveals the failing cases at the two extremes: On the one hand, if β max is too small, i.e. All the results are run with 6 random seeds and the final score is obtained by taking the average of scores. As we can see, both BLEU and NLLgen scores increase with the decrease of βmax. Besides, the variance of each score also consistently becomes larger for a smaller βmax. main text to show the advantages of RelGAN over other models while still demonstrating its ability to control the trade-off between sample quality and diversity.

β

:



Figures and Tables:

Figure 1: The self-attention mechanism for updating the memory from Mt to Mt+1 by incorporating new observation xt, where each row of the memory matrix Mt is a memory slot, and Q

Figure 2: The proposed discriminator framework with multiple embedded representations. The input is either the real sentence [r1 : • • • : rT ] where rt denotes the t-th one-hot token, or the generated (approximate) sentence [ŷ1 : • • • : ŷT ] where ŷt is from (6). Also, S embedding matrices {W (s) e } S s=1 map each input into S embedded representations, each of which is passed through discriminator independently to get the related loss. Note that "D" is the CNN-based classifier D(•) ∈ R with weight-sharing and ⊕ denotes taking the average.

Figure 3: The training curves of NLLgen scores (left) and NLLoracle scores (right) on synthetic data of length 20 with different values of maximum inverse temperature βmax ∈ {1, 2, 5, 10, 100}. The vertical dash line represents the end of pre-training. With the increase of βmax, NLLgen becomes lower but NLLoracle becomes higher. For both the NLLgen and NLLoracle scores, the lower the better.

Figure 4: (Left) Training curves of the BLEU-4 score on COCO Image Captions with different generator architectures -relational memory (RM), LSTM-32 and LSTM-512. (Right) Training curves of the BLEU-2 score on COCO Image Captions with Gumbel-Softmax relaxation and the vanilla REINFORCE method. All the results are obtained by taking the average of 6 runs with different random seeds.

Figure 5: (Left) The best NLLoracle score on the synthetic data varies with different number of embedded presentations S = {1, 2, 4, 8, 16, 32, 64} where βmax = 10. (Right) The training curves of BLEU-3 score on COCO Image Captions with the number of embedded representations S = 1 and S = 64, respectively, where βmax = 1000. All results are obtained by taking the average of 6 runs with different random seeds.

has used it to improve the generation quality in neural machine translation. More similarly, Kusner & Hernández-Lobato (2016) provides some initial experiments of training GANs with Gumbel-Softmax relaxation on a synthetic task, but scaling them to work on real text dataset remains a challenging open problem.

Figure 6: Training curves of BLEU scores on COCO Image Captions with different loss functions: RSGAN(Jolicoeur-Martineau, 2018), standard GAN (the non-saturating version)(Goodfellow et al., 2014) and hinge loss(Nowozin et al., 2016;Zhang et al., 2018), where βmax = 1000 and we use two different optimizers -(a) Adam and (b) RMSProp. All the results are obtained by taking the average of 6 runs with different random seeds. The vertical dash line represents the end of pre-training. We can see that RelGAN works well with different commonly-used loss functions of GANs and different optimization methods. In this scenario, the performance of RSGAN and standard GAN outperforms the hinge loss version.

Figure 6: Training curves of BLEU scores on COCO Image Captions with different loss functions: RSGAN(Jolicoeur-Martineau, 2018), standard GAN (the non-saturating version)(Goodfellow et al., 2014) and hinge loss(Nowozin et al., 2016;Zhang et al., 2018), where βmax = 1000 and we use two different optimizers -(a) Adam and (b) RMSProp. All the results are obtained by taking the average of 6 runs with different random seeds. The vertical dash line represents the end of pre-training. We can see that RelGAN works well with different commonly-used loss functions of GANs and different optimization methods. In this scenario, the performance of RSGAN and standard GAN outperforms the hinge loss version.

Figure 9: Training curves of BLEU scores on COCO Image Captions with different number of embedded representations S = 1 and S = 64, where βmax = 1000. We can see that the BLEU scores of S = 64 are consistently better than those of S = 1, which demonstrates the advantages of using multiple embedded representations for discriminator in RelGAN.

Figure 10: The training curve of NLLgen in RelGAN on COCO Image Captions, where βmax = 1000. We can see that during the adversarial training, the NLLgen score first increases and then decreases after around 800 iterations. The turning point matches well with those in the training curves of BLEU scores.

Figure 11: The BLEU-4 (Left) and NLLgen (Right) scores with error bars in RelGAN on COCO Image Captions with varying maximum inverse temperature βmax.

The NLLoracle scores on synthetic data where βmax = 1 for length 20 and βmax = 2 for length 40. RelGAN is run with 6 random seeds and the final score is obtained by taking the average of scores, and other scores are from their original papers andGuo et al. (2017). Note that "Real" denotes the real data generated by the oracle-LSTM. For the NLLoracle score, the lower the better.We also evaluate the trade-off between sample quality and diversity as a function of the maximum inverse temperature β max and the results are shown in Figure3. As β max increases, NLL gen decreases, which implies better sample diversity, but NLL oracle increases, which implies worse sample quality. Especially when β max ∈ {10, 100}, the best NLL gen score of 4.4 for RelGAN is very close to the best NLL gen score of 4.2 for MLE pre-training, implying that RelGAN with a sufficiently large inverse temperature suffers little mode collapse on synthetic data.

Table content: <table><tr><td>Length</td><td>MLE</td><td colspan="3">SeqGAN RankGAN LeakGAN</td><td>RelGAN</td><td>Real</td></tr><tr><td>20</td><td>9.038</td><td>8.736</td><td>8.247</td><td>7.038</td><td colspan="2">6.680 ± 0.343 5.750</td></tr><tr><td>40</td><td>10.411</td><td>10.310</td><td>9.958</td><td>7.191</td><td colspan="2">6.765 ± 0.026 4.071</td></tr></table>



Table content: <table><tr><td>Method</td><td>BLEU-2</td><td>BLEU-3</td><td>BLEU-4</td><td>BLEU-5</td><td>NLLgen</td></tr><tr><td>MLE</td><td>0.731</td><td>0.497</td><td>0.305</td><td>0.189</td><td>0.718</td></tr><tr><td>SeqGAN</td><td>0.745</td><td>0.498</td><td>0.294</td><td>0.180</td><td>1.082</td></tr><tr><td>RankGAN</td><td>0.743</td><td>0.467</td><td>0.264</td><td>0.156</td><td>1.344</td></tr><tr><td>LeakGAN</td><td>0.746</td><td>0.528</td><td>0.355</td><td>0.230</td><td>0.679</td></tr><tr><td>RelGAN (100)</td><td/><td/><td/><td/><td/></tr></table>

. The training data contains abbout 270,000 sentences and test data contains 10,000 sentences. ± 0.013 0.705 ± 0.019 0.501 ± 0.023 0.319 ± 0.018 2.482 ± 0.031 RelGAN (1000) 0.837 ± 0.012 0.654 ± 0.010 0.435 ± 0.011 0.265 ± 0.011 2.285 ± 0.025

Table content: <table><tr><td>Method</td><td>BLEU-2</td><td>BLEU-3</td><td>BLEU-4</td><td>BLEU-5</td><td>NLLgen</td></tr><tr><td>MLE</td><td>0.768</td><td>0.473</td><td>0.240</td><td>0.126</td><td>2.382</td></tr><tr><td>SeqGAN</td><td>0.777</td><td>0.491</td><td>0.261</td><td>0.138</td><td>2.773</td></tr><tr><td>RankGAN</td><td>0.727</td><td>0.435</td><td>0.209</td><td>0.101</td><td>3.345</td></tr><tr><td>LeakGAN</td><td>0.826</td><td>0.645</td><td>0.437</td><td>0.272</td><td>2.356</td></tr><tr><td>RelGAN (100)</td><td>0.881</td><td/><td/><td/><td/></tr></table>

D MORE RESULTS ON ABLATION STUDY D.1 IMPACT OF RELATIONAL MEMORYFigure7: Training curves of BLEU scores on COCO Image Captions with different generator architecturesrelational memory (RM), LSTM-32 and LSTM-512, where βmax = 1000. We can see that the BLEU scores of relational memory are consistently better than those of LSTM-32 and LSTM-512, which demonstrates the advantages of using relational memory as generator in RelGAN.

Table content: <table/>

and RSGAN(Jolicoeur-Martineau, 2018). The BLEU and NLL gen scores evaluated on COCO Image Captions are shown in Table8. The BLEU and NLLgen scores of RelGAN without pre-training on COCO Image Captions where with a little hyperparameter tuning, we set βmax = 100 for the standard GAN loss and βmax = 1000 for other losses.

Table content: <table><tr><td>Losses</td><td>BLEU-2</td><td>BLEU-3</td><td>BLEU-4</td><td>BLEU-5</td><td>NLLgen</td></tr><tr><td>Hinge</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td colspan="6">WGAN-GP 0.330 ± 0.024 0.111 ± 0.019 0.065 ± 0.017 0.045 ± 0.013 4.063 ± 0.623</td></tr><tr><td>RSGAN</td><td colspan="5">0.460 ± 0.026 0.172 ± 0.025 0.085 ± 0.017 0.056 ± 0.013 3.065 ± 0.917</td></tr><tr><td>Standard</td><td colspan="5">0.590 ± 0.019 0.280 ± 0.020 0.141 ± 0.018 0.094 ± 0.011 2.259 ± 0.263</td></tr><tr><td>Random</td><td>0.041</td><td>0.017</td><td>0.011</td><td>0.008</td><td>8.355</td></tr></table>

max = 1, RelGAN suffers from severe mode collapse (denoted by the large NLL gen ) and training instability (denoted by high variances of scores) issues. On the other hand, if β max is too large, i.e. β max = 10 7 , the sample quality improvement of RelGAN becomes marginal (denoted by the low BLEU scores). Therefore, we have chosen the two intermediate values {100, 1000} of β max in the ± 0.121 0.791 ± 0.209 0.659 ± 0.243 0.500 ± 0.230 1.454 ± 0.121 10 0.862 ± 0.038 0.741 ± 0.060 0.604 ± 0.060 0.445 ± 0.080 1.084 ± 0.061 10 2 0.849 ± 0.030 0.687 ± 0.047 0.502 ± 0.048 0.331 ± 0.044 0.756 ± 0.054 10 3 0.814 ± 0.012 0.634 ± 0.020 0.455 ± 0.023 0.303 ± 0.020 0.655 ± 0.048 10 4 0.801 ± 0.006 0.609 ± 0.012 0.430 ± 0.019 0.288 ± 0.015 0.631 ± 0.045 10 5 0.796 ± 0.007 0.599 ± 0.012 0.417 ± 0.010 0.277 ± 0.012 0.588 ± 0.037 10 6 0.790 ± 0.009 0.588 ± 0.011 0.408 ± 0.013 0.272 ± 0.010 0.569 ± 0.039 10 7 0.775 ± 0.011 0.572 ± 0.020 0.390 ± 0.019 0.252 ± 0.016 0.547 ± 0.032 The BLEU and NLLgen scores of RelGAN with various values of βmax on COCO Image Captions.

Table content: <table><tr><td>βmax</td><td>BLEU-2</td><td>BLEU-3</td><td>BLEU-4</td><td>BLEU-5</td><td>NLLgen</td></tr><tr><td>1</td><td>0.890</td><td/><td/><td/><td/></tr></table>

Peer review (iclr2019):
 ========================== I have read the authors* response and other reviewers* comments. Thanks the authors for taking great effort in answering my questions. Generally, I feel satisfied with the repsonse, and prefer an acceptance recommendation. ========================== Contributions: The main contribution of this paper is the proposed RelGAN. First, instead of using a standard LSTM as generator, the authors propose using a relational memory based generator. Second, instead of using a single CNN as discriminator, the authors use multiple embedded representations. Third, Gumbel-softmax relaxation is also used for training GANs on discrete textual data. The authors also claim the proposed model has the ability to control the trade-off between sample quality and diversity via a single adjustable parameter. Detailed comments: (1) Novelty: This paper is not a breakthrough paper, mainly following previous work and propose new designs to improve the performance. However, it still contains some novelty inside, for example, the model choice of the generator and discriminator. I think the observation that the temperature control used in the Gumbel-softmax can reflect the trade-off between quality and diversity is interesting. However, I feel the claim in the last sentence of the abstract and introduction is a little bit strong. Though this paper seems to be the first to really use Gumbel-softmax for text generation, similar techniques like using annealed softmax to approximate argmax has already been used in previous work (Zhang et al., 2017). Since this is similar to Gumbel-softmax, I think this may need additional one or two sentences to clarify this for more careful discussion. Further, I would also recommend the authors discuss the following paper [a] to make this work more comprehensive as to the discussion of related work. [a] also uses annealed softmax approximation, and also divide the GAN approaches as RL-based and RL-free, similar in spirit as the discuss in this paper. [a] Adversarial Text Generation via Feature-Mover*s Distance, NIPS 2018. (2) Presentation: This paper is carefully written and easy to follow. I enjoyed reading the paper. (3) Evaluation: Experiments are generally well-executed, with ablation study also provided. However, human evaluation is lacked, which I think is essential for this line of work. I have a few questions listed below. Questions: (1) In section 2.4, it mentions that the generator needs pre-training. So, my question is: does the discriminator also need pre-training? If so, how the discriminator is pre-trained? (2) In Table 1 & 2 & 3, how does your model compare with MaskGAN? If this can be provided, it would be better. (3) Instead of using NLL_{gen}, a natural question is: what are the self-BLEU score results since it was used in previous work? (4) The .beta_max value used in the synthetic and real datasets is quite different. For example, .beta_max = 1 or 2 in synthetic data, while .beta_max = 100 or 1000 is used in real data. What is the observation here? Can the authors provide some insights into this? (5) I feel Figure 3 is interesting. As the authors noted, NLL_gen measures diversity, NLL_oracle measures quality. Looking at Figure 3, does this mean GAN model produces higher quality samples than MLE pretrained models, while GAN models also produces less diverse samples than MLE models? This is due to NLL_gen increases after pretraining, while NLL_oracle further decreases after pretraining. However, this conclusion also seems strange. Can the authors provide some discussion on this? (6) Can human evaluation be performed since automatic metrics are not reliable enough?