Title: SALSA-TEXT : SELF ATTENTIVE LATENT SPACE BASED ADVERSARIAL TEXT GENERATION

Abstract: Inspired by the success of self attention mechanism and Transformer architecture in sequence transduction and image generation applications, we propose novel self attention-based architectures to improve the performance of adversarial latent codebased schemes in text generation. Adversarial latent code-based text generation has recently gained a lot of attention due to its promising results. In this paper, we take a step to fortify the architectures used in these setups, specifically AAE and ARAE. We benchmark two latent code-based methods (AAE and ARAE) designed based on adversarial setups. In our experiments, the Google sentence compression dataset is utilized to compare our method with these methods using various objective and subjective measures. The experiments demonstrate the proposed (self) attention-based models outperform the state-of-the-art in adversarial code-based text generation.

1. INTRODUCTION:

Text generation is of particular interest in many natural language processing (NLP) applications such as dialogue systems, machine translation, image captioning and text summarization. Recent deep learning-based approaches to this problem can be categorized into three classes: auto-regressive or maximum likelihood estimation (MLE)-based, generative adversarial network (GAN)-based and reinforcement learning (RL)-based approaches.

MLE-based methods (such as Sutskever et al. (2014) ) model the text (language) as an auto-regressive process, commonly using RNNs. RNNs compactly represent the samples history in the form of recurrent states. In these models, text is generated by predicting next token (character, word, etc) based on the previously generated ones (Graves, 2013) .

One of the main challenges involved with auto-regressive methods is exposure bias (Bengio et al., 2015) . This problem arises due to discrepancy between the training and generation phases. In fact, ground-truth samples from the past are used in training, while past generated ones are used in generation. A number of solutions have been proposed to address this problem by modifying the training procedure including scheduled sampling (Bengio et al., 2015) , Gibbs sampling (Su et al., 2018) , and Professor forcing (Lamb et al., 2016) .

Over the past few years, researchers have extensively used GANs (Goodfellow et al., 2014) as a powerful generative model for text (Yu et al., 2017; Che et al., 2017) , inspired by the great success in the field of image generation. GANs are believed to be capable of solving the exposure bias problem in text generation raised from using MLE. The reason is that they solved a similar issue of blurry image generation in MLE-based variational autoencoders (VAEs). It is belived that the discriminator is able to guide the text generator, through their training exchange, how to generate samples similar to real (training) data. However, there are other challenges involved in GAN-based text generation.

A few of these challenges in text generation are inherent to GANs themselves, such as mode collapse and training instability. The mode collapse problem happens when the adversarially trained generator does not produce diverse texts. These issues can be mitigated by using well-known techniques such as feature matching (Zhang et al., 2017) , and entropy regularization (Shi et al., 2018) . Another challenge is due to the discrete nature of text, which causes the generator sampling to be non-differentiable over the categorical distribution of the words.

In this paper, we take advantage of Transformer self-attention mechanism (Vaswani et al., 2017) and incorporate it in two state-of-the-art adversarial latent code-based schemes proposed for text generation. More specifically:

• We incorporate the Transformer structure in the design of encoder and decoder blocks of AAE (Makhzani et al., 2015a) and ARAE (Kim et al., 2017a) setups for text generation.

• Blocks closely inspired from the Transformer's encoder layers, incorporating self-attention and element-wise fully-connected layers in a residual configuration and with positional encodings, are used along with spectral normalization to propose a novel GAN (both generator and discriminator) structure for AAE and ARAE setups.

• The performance improvement obtained from the proposed architectures is demonstrated via objective and subjective measures used in extensive experiments.

2. RELATED WORK:

2.1 SPECTRAL NORMALIZATION Spectral normalization (Miyato et al., 2018) is a weight normalization method proposed to stabilize the training of GANs. The authors show that the Lipshitz norm of a neural networks can be bounded by normalizing the spectral norm of layer weight matrices. As opposed to local regularizations used in WGAN-GP, etc., the network-wide spectral regularization stabilizes the GAN training, produces more diverse outputs and results in higher inception scores. We use spectral normalization in our adversarial setups for the same reasons.

2.2. ATTENTION MODELS:

In sequence modeling literature, attention was initially proposed by Bahdanau et al. (2014) . It recognizes the fixed-length latent representation of the input sequence as the main performance bottleneck in the seq-to-seq models and proposed using soft-attention in the decoder. Using attention, the decoder can also attend to any desired token in the input sequence besides consuming the compressed representation resulting at the end of encoding operation.

Self-attention was initially proposed for language inference in Parikh et al. (2016) . The authors named it as "intra-attention" and showed that their structure can be an effective alternative for LSTMs in the task of natural language inference (Bowman et al., 2015) , at the time achieving state of the art performance with much fewer parameters as well as requiring a training time an order of magnitude shorter. Self-attention structures have since been used to set the state of the art in a number of different tasks (Vaswani et al., 2017; Dehghani et al., 2018; Yu et al., 2018; Radford et al.; Al-Rfou et al., 2018) . They drastically reduce the path length between any two sequence inputs, making the learning of long term dependencies much easier (Vaswani et al., 2017) . They are considerably easier to parallelize, reducing the number of operations that are required to be sequential.

Recently, Zhang et al. (2018) applied self attention along with spectral normalization to the task of image generation using GANs. It showed by visualization that using attention, the generator can attend to far neighborhoods of any shape rather than close-by fixed-shape ones at each level in a hierarchical generation. The authors claim that applying spectral normalization to generator as well as discriminator helps training dynamics (stability). Similarly, we also adopt self attention and spectral normalization in our architecture designs.

Transformer (Vaswani et al., 2017) extended the use of self attention mechanism and was proved to be the state-of-the-art in sequence transduction applications such as machine translation. It dispenses convolutional and recurrent layers and relies entirely on attention-only layers and element-wise feed forward layers.

2.3. LATENT SPACE-BASED TEXT GENERATION:

One of the main challenges of the language generation task originates from the discrete nature of text. Similarly to generating other discrete tokens, the back propagation of error through argmax operator is not well-defined. To address this problem, various approaches have been proposed in the literature including continuous approximation of discrete sampling (Gulrajani et al., 2017; Jang et al., 2016) , using policy gradient from reinforcement learning (Guo, 2015; Shi et al., 2018) , etc. One of the most successful solutions is based on autoencoders with continuous latent spaces (i.e. latent code-based methods). Various training setups have been proposed for training these autoencoders including adversarial (Kim et al., 2017a) and variational (Hu et al., 2017) setups.

A recent paper (Cífka et al., 2018) performs a thorough review of the state-of-the-art latent codebased text generation methods. It studies the performance of a number of code-based text generation schemes and uses a unified rigorous evaluation protocol to evaluate them. We got inspired by their evaluation protocol to demonstrate the strength of our self attention-based approach in the context. They use a broad set of measures to perform a comprehensive study. We adopt forward and reverse perplexity as well as BLEU from their objective measures and fluency from the subjective ones.

2.4. ADVERSARIAL TEXT GENERATION:

In this section, we briefly explain two prominent baseline methods using adversarial latent code-based generation techniques and present the technical details in Section 3.

2.4.1. AAE:

Adversarial autoencoder (AAE) (Makhzani et al., 2015b) proposes an adversarial setup to train probabilistic autoencoders. It matches the aggregated posterior of the encoder output (latent codes) to an arbitrary distribution that can be easily sampled from. Although authors demonstrate the applications of their setup in semi-supervised learning, style and content disentanglement, etc, AAE decoder can be effectively used as a generative model, converting samples of the arbitrary distribution (noise) to real-like outputs. From application perspective, authors only evaluated AAE performance in vision-related applications. In this paper, we tailor AAE for text generation, following guidelines proposed by Cífka et al. (2018) and incorporate self attention and Transformer as novel parts in the model.

2.4.2. ARAE:

The adversarially regularized autoencoder (ARAE) (Kim et al., 2017b ) learns an autoencoder with continuous contracted codes that highly correlate with discrete inputs. That is, similar inputs get encoded (mapped) to nearby continuous codes. ARAE aims at exploiting GAN's ability to force the generator to output continuous codes corresponding to the code space obtained from encoding the real text data. By matching the outputs of generator and encoder, ARAE provides an implicit latent code GAN that serves as a generative model for decoding text.

3. SELF ATTENTIVE LATENT CODE-BASED MODELS:

In this section, we explain the details of our self attention-based models following the ARAE and AAE setups proposed in Cífka et al. (2018) . These setups have shown comparable to the state-of-theart results in text generation. We select similar setups to provide fair comparisons and report the best techniques/parameters based on our experiments.

In our architectures, Transformer (Vaswani et al., 2017) is used in designing all autoencoders. In both encoder and decoder, we use three blocks of Transformer. 'Block' and 'layer' names are used, respectively, instead of 'layer' and 'sub-layer' in the original paper.

Layer normalization (Ba et al., 2016) is applied on every layer (multi-head attention, masked multihead attention and feed forward layers) within each Transformer block. Multi-head attentions have eight heads and embedding layers are of size 304 (a multiple of eight). Similarly to Vaswani et al. (2017) , positional encoding is used at the very first layer of the encoder and decoder. The dimensions and encoding place were found empirically for the best objective and subjective performance.

For GAN structures, i.e. the generator and discriminator architectures, we use modified Transformer encoder layers combined with spectral normalization, as depicted in Fig. 1 (N = 3). As in the regular transformer blocks, all connections are residual. Inspired by spectral normalization successes in the 

4. EXPERIMENTS:

We study the performance of our self attentive (SALSA) architectures and compare it with that of the code-based setups studied in Cífka et al. (2018) .

4.1. EXPERIMENTAL SETUP:

The performance of the models is evaluated in sentence generation (sampling), on the Google Sentence Compression (GSC) dataset1 (as in Cífka et al. (2018) ). Training on this dataset is very challenging as the sentences are relatively long (average of 24.8 words) and diverse in terms of content, grammar, etc. GSC comprises 200, 000 training and 9, 995 test sentences. For all the trained models, we use Google's SentencePiece2 tokenizer using byte-pair encoding (BPE (Sennrich et al., 2015) ) as in Cífka et al. (2018) . -.com is launching a new album on "The Idol's.

-.

-Tuesday for a US dollar in the US, according to the US Department.

-US US markets on Wednesday as US stocks rose to a US dollar, dealers said.

-The US will open its first-day visit to next year.

SALSA-AAE:

-The world's largest car maker, said it will buy back a new US$4 million fine for the first time in three years.

-London, May 6 A 48-year-old man has been charged with raping a woman in the face of livelihood on Tuesday.

-In a bid to save money, the US government's most expensive land.

-The Governors of the city of Caterpillar, who was in talks with the world's most expensive officials.

-Israel has launched a new website that would help the Gaza Strip, and the first such incident in a deadly attack on the West Bank and the United States.

-Harrington has been found guilty of two counts of driving under the influence of a semi-final at a local court.

-A man has been arrested and charged with sexual assault on an alleged assault and assault for allegedly assaulting him to death and a child.

-A man is accused of stealing a child in her own home case of her husband.

-A man, who was accused of killing the two-year-old girl in connection with several of them who died from injuries last week.

-A man is facing charges of sexual assault for allegedly assaulting a woman and her wife.

-A man accused of killing the man in his home is being sentenced to life and is expected to be on the way.

-A man has been arrested in connection with her death and then sex with her husband, who was in critical condition and will be released on Monday.

SALSA-ARAE:

-Former PSV Ethanolve said he has "two and the title of his contract," the first-round pick of the season.

-The Queen will present a "Curprone of the Donegal Plan" in a new biopic of the forthcoming musical, and "Kalways," the school said in a statement.

-This week between the "Daily Show" flights, a year after the airing, a day before the launch of a summer vacation.

-However, the Myssenon, is planning to open its first wedding, with a move that will include a number of its customers, but they are not planning to sell their services.

-The Dallas Morning News reported that a Houston man is in "a new, a state that is leaving the Houston Rockets.

-The Dallas Morning News Corp. said it is to open a subsidiary of Houston, the largest newspaper and its staff, to be in the city.

We filter the dataset to only include sentences with a maximum of 50 PBE tokens. This only lowers the average number of words per sentence and total number of sentences to 23.1 and 183739, respectively in the training set. The test dataset is also reduced to 9254 lines with an average of 22.7 words per sentence. Samples of generated sentences from all models is listed in Section 4.3.

The input noise to the generator is of size 100 (as in Cífka et al. (2018) ). We upsample the noise to the embedding size of 304 by using a fully connected layer. The same upsampled noise is copied a number of times equal to the maximum number of steps in the sentence. We use T = 50 times in our experiments. The noise is then fed to the generator, where positional encodings are added to each step. The previously mentioned fully connected layer also serves to allow the model to learn to protect the information of the positional encodings from the noise. Positional encodings are also added at the start of each transformer encoder block. As we use fixed size sequences, the attention depth is always fixed (T bpe). Positional encodings are also added to the input of each transformer encoder block, inside of the generator.

4.2. EVALUATION METRICS:

We use various objective and subjective measures to evaluate the models. As objective measures, we use BLEU (Papineni et al., 2002) , Self-BLEU (Zhu et al., 2018) , forward and reverse perplexity.

BLEU (Papineni et al., 2002) is a widely used metric to compute the similarity of a set of generated sentences with a reference dataset. The results are described in Table 2 .

Self BLEU (Zhu et al., 2018) (Table 3 ) is a measure of diversity for generated texts. In Self-BLEU, for each generated sentence, we compute the BLEU using the sentence as hypothesis and the rest of the generated sentences as the reference. When averaged over all the references, it gives us a measure of how diverse the sentences are. Lower Self-BLEU scores are better, as high BLEU scores indicate great similarity.

In Perplexity evaluation (Table 4 ), the goal is to measure the individual quality of the sentences generated. We train an LSTM language model on the WMT News 2017 Dataset3 filtered for lines of a maximum of 50 BPE tokens (a total of 200000 sentences). The perplexity of the language model is computed over 100000 generated sentences for each model.

Reverse perplexity evaluation (Table 4 ) aims to measure variety of the generated sentences. For each model, we train an LSTM-based language model based on 100000 generated sentences, and then evaluate the perplexity on the GSC test dataset, filtered to a maximum length of 50 BPE. Diverse generated sentences that cover dataset to a good extent would result in better (lower) reverse perplexity measures resulting from the trained LSTM network (language model).

For the subjective evaluation (Table 5 ), we use Amazon mechanical Turk4 online platform. 18 sentences are sampled from each model, i.e. a total of 162 sentences. We assign 81 randomly selected sentences to 50 native English speakers (among Mechanical Turk Masters with hit approval ratings greater than 75%). The remaining 81 are assigned to another group of 50 people with the same qualifications. Each person was asked to evaluate the assigned 81 sentences in one and a half hours.

In the evaluation, the 5-point Likart scale is used to measure grammaticality, semantic consistency and overall (Fluency). The overall reflects both grammar and semantic consistency in addition to other human-specific factors. Hence, it is a good representative of "Fluency" measure used in Cífka et al. (2018) .

4.3. SAMPLES OF GENERATED SENTENCES:

In Table 1 , we list six generated sentences for each model. As seen, AAE generates rather short sentences, while the corresponding SALSA version (SALSA-AAE) has alleviated the issue to a good extent. Finally, ARAE suffers from extreme mode collapse as opposed to its SALSA counterpart.

4.4. RESULTS AND DISCUSSION:

The results of objective and subjective evaluations are presented in Tables 2 to 5 . As seen, the proposed self attention-based (SALSA) architectures consistently outperform the non-attention-based benchmarks in terms of diversity (measured by reverse perplexity). Moreover, they often show better performance in terms of output quality (measured by BLEU, self BLEU, preplexity and human evaluations) on the long and complicated sentences of the GSC dataset.

As seen in the generated samples (Table 1 ), human evaluation (Table 5 ) and objective metrics (Tables 2 to 4 ), the original AAE and ARAE setups perform very poorly on GSC with long sentences. With reverse perplexities of over 8000 and high self-BLEU scores close to 0.9, they suffer from a high level of mode collapse (repeated sentences).

Human evaluations do not account for lack of diversity. The reason is humans are presented with a number of shuffled sentences and asked to evaluate them independently (without knowing which sentence coming from which model). Hence, in our experiments for the original AAE and ARAE, a model can generate similar sentences (maybe due to mode collapse) and still receives high subjective scores.

Objective and subjective evaluation of the studied models on the GSC dataset It seems that, in our experiments, the original ARAE model suffers from mode collapse. We can see that it has slightly higher human evaluation scores, but extremely poor diversity metrics, i.e. very high reverse perplexity and self-BLEU scores. It can also be seen in the randomly selected generated sentences (Table 1 ), where all the sentences start with "A man" and invariably mention he is being arrested or accused of grievous crimes. This is likely because the sentences in the GSC dataset are long and that their structure is elaborate. SALSA-ARAE on the other hand reliably produces sentences of quality with great diversity.

SALSA-AAE has both considerably higher individual quality metrics than the original AAE and much better diversity metrics. It is the strongest pure adversarial text model. As seen in Table 5 , SALSA-AAE provides the best grammaticality, semantic consistency and Fluency performance. 

5. CONCLUSION AND FUTURE WORK:



:

https://github.com/google-research-datasets/sentence-compression

https://github.com/google/sentencepiece

http://www.statmt.org/wmt17/

https://www.mturk.com/

Figures and Tables:

Figure 1: SALSA-TEXT generator and discriminator architecture designed using Transformer encoder structure

Figure 2: SALSA-AAE for text generation. As explained in the figure, Transformer-based encoder, decoder and discriminator (with self attention) architectures are used. Decoder uses fixed-size full codes.

Figure 3: SALSA-ARAE for text generation. Similarly to SALSA-AAE, all blocks are Transformerbased and decoder uses fixed-size full codes. Generator and discriminator comprise of self-attention layers.

In this paper, we introduced SALSA-TEXT, a Transformer-based architecture for adversarial codebased text generation. It incorporates self-attention mechanism by utilizing Transformer architecture in autoencoder and GAN setups. Our extensive experiments demonstrate the better performance of our models compared to the state-of-the-art in adversarial code-based text generation (without self-attention). The proposed architectures provide diverse, long and high quality output sentences as confirmed by objective metrics and human evaluations in extensive experiments.As a future direction, it is beneficial to study the performance of self attention in other text generation methods including variational code-based and reinforcement learning-based approaches. Another interesting direction is to experiment with deeper Transformer-based autoencoders to better capture the underlying language model and perform unsupervised pre-training isnpired by the success ofAl-Rfou et al. (2018) andRadford et al..

Samples from models trained on GSC dataset

Table content: <table><tr><td>Model</td><td>Sample sentences</td></tr><tr><td>AAE</td><td>-on Thursday for an undisclosed amount of US Senate.</td></tr></table>



Table content: <table><tr><td/><td/><td colspan="2">: BLEU</td><td/></tr><tr><td colspan="5">AAE SALSA-AAE ARAE SALSA-ARAE</td></tr><tr><td>BLEU-1 0.671</td><td/><td>0.905</td><td>0.924</td><td>0.865</td></tr><tr><td>BLEU-2 0.500</td><td/><td>0.638</td><td>0.698</td><td>0.585</td></tr><tr><td>BLEU-3 0.279</td><td/><td>0.367</td><td>0.402</td><td>0.294</td></tr><tr><td>BLEU-4 0.149</td><td/><td>0.192</td><td>0.212</td><td>0.137</td></tr><tr><td>BLEU-5 0.086</td><td/><td>0.101</td><td>0.116</td><td>0.071</td></tr><tr><td/><td colspan="3">Table 3: Self-BLEU</td><td/></tr><tr><td colspan="5">AAE SALSA-AAE ARAE SALSA-ARAE</td></tr><tr><td colspan="2">Self BLEU-1 0.950</td><td>0.897</td><td>0.973</td><td>0.896</td></tr><tr><td colspan="2">Self BLEU-2 0.759</td><td>0.738</td><td>0.921</td><td>0.731</td></tr><tr><td colspan="2">Self BLEU-3 0.604</td><td>0.573</td><td>0.843</td><td>0.549</td></tr><tr><td colspan="2">Self BLEU-4 0.412</td><td>0.410</td><td>0.751</td><td>0.367</td></tr><tr><td colspan="2">Self BLEU-5 0.270</td><td>0.275</td><td>0.653</td><td>0.226</td></tr><tr><td/><td colspan="3">Table 4: Perplexity</td><td/></tr><tr><td colspan="2">AAE</td><td colspan="3">SALSA-AAE ARAE SALSA-ARAE</td></tr><tr><td colspan="2">Reverse perplexity 10309</td><td>822</td><td>8857</td><td>1008</td></tr><tr><td>Perplexity</td><td>88</td><td>61</td><td>37</td><td>106</td></tr><tr><td colspan="4">Table 5: Human Evaluations</td><td/></tr><tr><td/><td colspan="4">AAE SALSA-AAE ARAE SALSA-ARAE</td></tr><tr><td>Grammaticality</td><td>2.756</td><td>3.09</td><td>2.980</td><td>2.898</td></tr><tr><td colspan="2">Semantic consistency 2.575</td><td>2.597</td><td>2.856</td><td>2.617</td></tr><tr><td>Fluency</td><td>2.604</td><td>2.700</td><td>2.851</td><td>2.652</td></tr></table>

Peer review (iclr2019):
 Summary of the paper: This work builds on the adversarial autoencoder (AAE) proposed by Makhzani et al 2015 and the adversarially regularized auto encoder (ARAE) proposed by Kim et al. 2017. It does this by integrating Vaswani et al. 2017’s transformer model and self-attention (Parikh et al. 2016 and many others) and the AAE and ARAE methodologies. The authors call this SALSA (Self Attentive Latent Space Based Adversarial Text Generation). Basically SALSA combines these parts that are successful on their own and shows on the Google Sentence Compression dataset that ARAE and AAE can be beaten with the inclusion/injection of Transformer + self-attention. Their models show longer and higher-quality sentence generation with more diversity. For some of the BLEU/Human Eval metrics though, ARAE seems to perform better. SALSA seems to aid in alleviating mode collapse (something ARAE struggles dramatically to do). Preliminary Comments: * Paper is easy-to-read and organized. * Experimental design is well-described especially in how the dataset was preprocessed; something many papers in this area fail to do. * Evaluation metrics are discussed in detail, which is a plus. Drawbacks: * Novelty is limited: SALSA is an aggregation of already proposed methodologies. * The experimentation could’ve been more robust. - It seems as though there’s only one model that is evaluated here. Trying different models (varying HPs or just random seeds) or providing justification of how this specific model was chosen would be helpful. If I missed this, please correct me. - Varying the dimensionality of the noise vector, how often it is copied would be helpful. - Discussion on why copying the noise vector T times rather than generating a smaller dimensional noise vector (or the same dimensional noise vector) at each step would strengthen the paper. * Statistics of the human evaluation, standard errors/variances of how the humans evaluated the sentences would strengthen the results more. * The discussion of experimental results leaves a little to be desired. I’d like to know what a 0.1 improvement in grammaticality, semantic consistency, fluency etc means. Standard errors like I mentioned before and/or examples of this would go a long way to helping that. * The authors discuss how SALSA aids in ‘better performance’ on long/complex sentences. A definition of what is long and complex in conjunction with evaluation across BLEU/Self-BLEU/PPL/R-PPL/3 Human metrics would strengthen the paper dramatically. * Looking at SALSA-ARAE vs ARAE in Table 2, ARAE has much higher BLEU scores. Some discussion at why this is the case would also improve the paper. Minor Issues: * There are many typos in the text, so another proof-read would be good. * Figures and tables are not space-efficient and do not need to be so large; placed in the manners they are. This empty space could be used to add more discussion/further experimentation. Figure 1 can be rotated to give you 6-8 lines more. Figures 2 and 3 can be put side by side without reducing the font too much. * The bolding of certain numbers in the tables are confusing on first glance because you’re obviously comparing SALSA-AAE to AAE and SALSA-ARAE to ARAE, but the tables have both without a divider, so on first glance I was confused (wasn’t the best performance in the row). So including a double lined divider on each table would be helpful. * A few papers have been listed multiple times in the references (AAE and ARAE papers). * Related work section: an inclusion of a few of the more recent GAN papers for text is missing. I think some discussion of other VAE approaches to text such as Bowman et al. 2016: Generating Sentences from a Continuous Space among others would be good. I think citing Kingma and Welling 2014 Auto-Encoding Variational Bayes and Rezende et al. 2014 Stochastic Backpropagation and Approximate Inference in Deep Generative models is important in the first mention of VAEs. Concluding Thoughts: * This paper has limited model novelty and doesn’t overcome this by having great experiments or analysis. * This paper has the building blocks for great experimentation and analysis, and if those are completed well, has the ability to make strong claims regarding the impact of SALSA on a variety of metrics.