Title: ADVERSARIALLY REGULARIZED AUTOENCODERS

Abstract: While autoencoders are a key technique in representation learning for continuous structures, such as images or wave forms, developing general-purpose autoencoders for discrete structures, such as text sequence or discretized images, has proven to be more challenging. In particular, discrete inputs make it more difficult to learn a smooth encoder that preserves the complex local relationships in the input space. In this work, we propose an adversarially regularized autoencoder (ARAE) with the goal of learning more robust discrete-space representations. ARAE jointly trains both a rich discrete-space encoder, such as an RNN, and a simpler continuous space generator function, while using generative adversarial network (GAN) training to constrain the distributions to be similar. This method yields a smoother contracted code space that maps similar inputs to nearby codes, and also an implicit latent variable GAN model for generation. Experiments on text and discretized images demonstrate that the GAN model produces clean interpolations and captures the multimodality of the original space, and that the autoencoder produces improvements in semi-supervised learning as well as state-of-the-art results in unaligned text style transfer task using only a shared continuous-space representation.

1. INTRODUCTION:

Recent work on regularized autoencoders, such as variational (Kingma & Welling, 2014; Rezende et al., 2014) and denoising (Vincent et al., 2008) variants, has shown significant progress in learning smooth representations of complex, high-dimensional continuous data such as images. These codespace representations facilitate the ability to apply smoother transformations in latent space in order to produce complex modifications of generated outputs, while still remaining on the data manifold.

Unfortunately, learning similar latent representations of discrete structures, such as text sequences or discretized images, remains a challenging problem. Initial work on VAEs for text has shown that optimization is difficult, as the decoder can easily degenerate into a unconditional language model (Bowman et al., 2015b) . Recent work on generative adversarial networks (GANs) for text has mostly focused on getting around the use of discrete structures either through policy gradient methods (Che et al., 2017; Hjelm et al., 2017; Yu et al., 2017) or with the Gumbel-Softmax distribution (Kusner & Hernandez-Lobato, 2016) . However, neither approach can yet produce robust representations directly.

A major difficulty of discrete autoencoders is mapping a discrete structure to a continuous code vector while also smoothly capturing the complex local relationships of the input space. Inspired by recent work combining pretrained autoencoders with deep latent variable models, we propose to target this issue with an adversarially regularized autoencoder (ARAE). Specifically we jointly train a discrete structure encoder and continuous space generator, while constraining the two models with a discriminator to agree in distribution. This approach allows us to utilize a complex encoder model, such as an RNN, and still constrain it with a very flexible, but more limited generator distribution. The full model can be then used as a smoother discrete structure autoencoder or as a latent variable GAN model where a sample can be decoded, with the same decoder, to a discrete output. Since the system produces a single continuous coded representation-in contrast to methods that act on each RNN state-it can easily be further regularized with problem-specific invariants, for instance to learn to ignore style, sentiment or other attributes for transfer tasks.

Experiments apply ARAE to discretized images and sentences, and demonstrate that the key properties of the model. Using the latent variable model (ARAE-GAN), the model is able to generate varied samples that can be quantitatively shown to cover the input spaces and to generate consistent image and sentence manipulations by moving around in the latent space via interpolation and offset vector arithmetic. Using the discrete encoder, the model can be used in a semi-supervised setting to give improvement in a sentence inference task. When the ARAE model is trained with task-specific adversarial regularization, the model improves the current best results on sentiment transfer reported in Shen et al. (2017) and produces compelling outputs on a topic transfer task using only a single shared code space. All outputs are listed in the Appendix 9 and code is available at (removed for review).

2. RELATED WORK:

In practice unregularized autoencoders often learn a degenerate identity mapping where the latent code space is free of any structure, so it is necessary to apply some method of regularization. A popular approach is to regularize through an explicit prior on the code space and use a variational approximation to the posterior, leading to a family of models called variational autoencoders (VAE) (Kingma & Welling, 2014; Rezende et al., 2014) . Unfortunately VAEs for discrete text sequences can be challenging to train-for example, if the training procedure is not carefully tuned with techniques like word dropout and KL annealing (Bowman et al., 2015b) , the decoder simply becomes a language model and ignores the latent code (although there has been some recent successes with convolutional models (Semeniuta et al., 2017; Yang et al., 2017) ). One possible reason for the difficulty in training VAEs is due to the strictness of the prior (usually a spherical Gaussian) and/or the parameterization of the posterior. There has been some work on making the prior/posterior more flexible through explicit parameterization (Rezende & Mohamed, 2015; Kingma et al., 2016; Chen et al., 2017) . A notable technique is adversarial autoencoders (AAE) (Makhzani et al., 2015) which attempt to imbue the model with a more flexible prior implicitly through adversarial training. In AAE framework, the discriminator is trained to distinguish between samples from a fixed prior distribution and the input encoding, thereby pushing the code distribution to match the prior. While this adds more flexibility, it has similar issues for modeling text sequences and suffers from mode-collapse in our experiments. Our approach has similar motivation, but notably we do not sample from a fixed prior distribution-our 'prior' is instead parameterized through a flexible generator. Nonetheless, this view (which has been observed by various researchers (Tran et al., 2017; Mescheder et al., 2017; Makhzani & Frey, 2017) ) provides an interesting connection between VAEs and GANs.

The success of GANs on images have led many researchers to consider applying GANs to discrete data such as text. Policy gradient methods are a natural way to deal with the resulting non-differentiable generator objective when training directly in discrete space (Glynn, 1987; Williams, 1992) . When trained on text data however, such methods often require pre-training/co-training with a maximum likelihood (i.e. language modeling) objective (Che et al., 2017; Yu et al., 2017; Li et al., 2017) . This precludes there being a latent encoding of the sentence, and is also a potential disadvantage of existing language models (which can otherwise generate locally-coherent samples). Another direction of work has been through reparameterizing the categorical distribution with the Gumbel-Softmax trick (Jang et al., 2017; Maddison et al., 2017) -while initial experiments were encouraging on a synthetic task (Kusner & Hernandez-Lobato, 2016) , scaling them to work on natural language is a challenging open problem. There has also been a flurry of recent, related approaches that work directly with the soft outputs from a generator (Gulrajani et al., 2017; Sai Rajeswar, 2017; Shen et al., 2017; Press et al., 2017) . For example, Shen et al. (Shen et al., 2017) exploits adversarial loss for unaligned style transfer between text by having the discriminator act on the RNN hidden states and using the soft outputs at each step as input to an RNN generator, utilizing the Professor-forcing framework (Lamb et al., 2016) . Our approach instead works entirely in code space and does not require utilizing RNN hidden states directly.

3. BACKGROUND:

Discrete Structure Autoencoders Define X = V n to be a set of discrete structures where V is a vocabulary of symbols and P x to be a distribution over this space. For instance, for binarized images V = {0, 1} and n is the number of pixels, while for sentences V is the vocabulary and n is the sentence length. A discrete autoencoder consists of two parameterized functions: a deterministic encoder function enc φ : X → C with parameters φ that maps from input to code space and a conditional decoder distribution p ψ (x | c) over structures X with parameters ψ. The parameters are trained on a cross-entropy reconstruction loss:

L rec (φ, ψ) = -log p ψ (x | enc φ (x))

The choice of the encoder and decoder parameterization is specific to the structure of interest, for example we use RNNs for sequences. We use the notation, x = arg max x p ψ (x | enc φ (x)) for the (approximate) decoder mode. When x = x the autoencoder is said to perfectly reconstruct x.

Generative Adversarial Networks GANs are a class of parameterized implicit generative models (Goodfellow et al., 2014) . The method approximates drawing samples from a true distribution c ∼ P r by instead employing a latent variable z and a parameterized deterministic generator function c = g θ (z) to produce samples c ∼ P g . Initial work on GANs minimizes the Jensen-Shannon divergence between the distributions. Recent work on Wasserstein GAN (WGAN) (Arjovsky et al., 2017) , replaces this with the Earth-Mover (Wasserstein-1) distance.

GAN training utilizes two separate models: a generator g θ (z) maps a latent vector from some easy-to-sample source distribution to a sample and a critic/discriminator f w (c) aims to distinguish real data and generated samples from g θ . Informally, the generator is trained to fool the critic, and the critic to tell real from generated. WGAN training uses the following min-max optimization over generator parameters θ and critic parameters w,

EQUATION

where f w : C → R denotes the critic function, c is obtained from the generator, c = g θ (z), and P r and P g are real and generated distributions. If the critic parameters w are restricted to an 1-Lipschitz function set W, this term correspond to minimizing Wasserstein-1 distance W (P r , P g ). We use a naive approximation to enforce this property by weight-clipping, i.e. w = [-, ] d (Arjovsky et al., 2017) .

4. MODEL: ADVERSARIALLY REGULARIZED AUTOENCODER:

Ideally, a discrete autoencoder should be able to reconstruct x from c, but also smoothly assign similar codes c and c to similar x and x . For continuous autoencoders, this property can be enforced directly through explicit regularization. For instance, contractive autoencoders (Rifai et al., 2011) regularize their loss by the functional smoothness of enc φ . However, this criteria does not apply when inputs are discrete and we lack even a metric on the input space. How can we enforce that similar discrete structures map to nearby codes?

Adversarially regularized autoencoders target this issue by learning a parallel continuous-space generator with a restricted functional form to act as a smoother reference encoding. The joint objective regularizes the autoencoder to constrain the discrete encoder to agree in distribution with its continuous counterpart:

min φ,ψ,θ L rec (φ, ψ) + λ (1) W (P r , P g )

Above W is the Wasserstein-1 distance between P r the distribution of codes from the discrete encoder model (enc φ (x) where x ∼ P(x)) and P g is the distribution of codes from the continuous generator model (g θ (z) for some z, e.g. z ∼ N (0, I)). To approximate Wasserstein-1 term, the W function includes an embedded critic function which is optimized adversarially to the encoder and generator as described in the background. The full model is shown in Figure 1 .

To train the model, we use a block coordinate descent to alternate between optimizing different parts of the model: (1) the encoder and decoder to minimize reconstruction loss, (2) the WGAN critic function to approximate the W term, (3) the encoder and generator to adversarially fool the critic to minimize W :

1) min φ,ψ L rec (φ, ψ) 2) min w∈W L cri (w) = max w∈W E x∼Px [f w (enc φ (x))] -E c∼Pg [f w (c)] 3) min φ,θ L encs (φ, θ) = min φ,θ E x∼Px [f w (enc φ (x))] -E c∼Pg [f w (c)]

The full training algorithm is shown in Algorithm 1.

discrete struct. encoder code (P r ) decoder reconstruction loss

x ∼ P x enc φ c p ψ x L rec + z ∼ N g θ c f w W W (P g , P r )

latent var. generator code (P g ) critic regularization

Figure 1 : ARAE architecture. The model can be used as an autoencoder, where a structure x is encoded and decoded to produce x, and as a GAN (ARAE-GAN), where a sample z is passed though a generator g θ to produce a code vector, which is similarly decoded to x. The critic function fw is only used at training to help approximate W .

Algorithm 1 ARAE Training:

for number of training iterations do (1) Train the autoencoder for reconstruction

[Lrec(φ, ψ)]. Sample {x (i) } m i=1 ∼ Px and compute code-vectors c (i) = enc φ (x (i) ). Backpropagate reconstruction loss, Lrec = -1 m m i=1 log p ψ (x (i) |c (i) , [y (i) ]), and update. (2) Train the critic [Lcri(w)] (Repeat k times) Sample {x (i) } m i=1 ∼ Px and {z (i) } m i=1 ∼ N (0, I). Compute code-vectors c (i) = enc φ (x (i) ) and c(i) = g θ (z (i) ). Backpropagate loss -1 m m i=1 fw(c (i) ) + 1 m m i=1 fw(c (i) ), update, clip the critic w to [-, ] d .

(3) Train the generator and encoder adversarially to critic [Lencs(φ, θ)]

Sample

{x (i) } m i=1 ∼ Px and {z (i) } m i=1 ∼ N (0, I) Compute code-vectors c (i) = enc φ (x (i) ) and c(i) = g θ (z (i) ). Backpropagate adversarial loss 1 m m i=1 fw(c (i) ) -1 m m i=1 fw(c (i)

) and update.

Extension: Code Space Transfer One benefit of the ARAE framework is that it compresses the input to a single code vector. This framework makes it ideal for manipulating discrete objects while in continuous code space. For example, consider the problem of unaligned transfer, where we want to change an attribute of a discrete input without supervised examples, e.g. to change the topic or sentiment of a sentence. First, we extend the decoder to condition on a transfer variable denoting this attribute y which is known during training, to learn p ψ (x | c, y). Next, we train the code space to be invariant to this attribute, to force it to be learned fully by the decoder. Specifically, we further regularize the code space to map similar x with different attribute labels y near enough to fool a code space attribute classifier, i.e.:

min φ,ψ,θ L rec (φ, ψ) + λ (1) W (P r , P g ) -λ (2) L class (φ, u)

where L class (φ, u) is the loss of a classifier p u (y | c) from code space to labels (in our experiments we always set λ (2) = 1). To incorporate this additional regularization, we simply add two more gradient update steps: (2b) training a classifier to discriminate codes, and (3b) adversarially training the encoder to fool this classifier. The algorithm is shown in Algorithm 2. Note that similar technique has been introduced in other domains, notably in images (Lample et al., 2017) and video modeling (Denton & Birodkar, 2017) .

5. METHODS AND ARCHITECTURES:

We experiment with three different ARAE models: (1) an autoencoder for discretized images trained on the binarized version of MNIST, (2) an autoencoder for text sequences trained using the Stanford Natural Language Inference (SNLI) corpus (Bowman et al., 2015a) Sample {x (i) } m i=1 ∼ Px, lookup y (i) , and compute code-vectors i) , and compute code-vectors

c (i) = enc φ (x (i) ). Backpropagate loss -1 m m i=1 log pu(y (i) |c (i) ), update. (3b) Train the encoder adversarially to code classifier [max φ Lclass(φ, u)] Sample {x (i) } m i=1 ∼ Px, lookup y (

c (i) = enc φ (x (i) ). Backpropagate adversarial classifier loss -1 m m i=1 log pu(1 -y (i) | c (i) ), update.

for text transfer (Section 6.2) based on the Yelp and Yahoo datasets for unaligned sentiment and topic transfer. All three models utilize the same generator architecture, g θ . The generator architecture uses a low dimensional z with a Gaussian prior p(z) = N (0, I), and maps it to c. Both the critic f w and the generator g θ are parameterized as feed-forward MLPs.

The image model uses fully-connected NN to autoencode binarized images. Here X = {0, 1} n where n is the image size. The encoder used is a feed-forward MLP network mapping from {0, 1} n → R m , enc φ (x) = MLP(x; φ) = c. The decoder predicts each pixel in x as a parameterized logistic regression,

p ψ (x | c) = n j=1 σ(h) xj (1 -σ(h)) 1-xj where h = MLP(c; ψ).

The text model uses a recurrent neural network (RNN) for both the encoder and decoder. Here X = V n where n is the sentence length and V is the vocabulary of the underlying language. Define an RNN as a parameterized recurrent function h j = RNN(x j , h j-1 ; φ) for j = 1 . . . n (with h 0 = 0) that maps a discrete input structure x to hidden vectors h 1 . . . h n . For the encoder, we define enc φ (x) = h n = c. For decoding we feed c as an additional input to the decoder RNN at each time step, i.e. hj = RNN(x j , hj-1 , c; ψ), and further calculate the distribution over V at each time step via softmax, p ψ (x | c) = n j=1 softmax(W hj + b) xj where W and b are parameters (part of ψ). Finding the most likely sequence x under this distribution is intractable, but it is possible to approximate it using greedy search or beam search. In our experiments we use an LSTM architecture (Hochreiter & Schmidhuber, 1997) for both the encoder/decoder and decode using greedy search. The text transfer model uses the same architecture as the text model but extends it with a code space classifier p(y|c) which is modeled using an MLP and trained to minimize cross-entropy.

Our baselines utilize a standard autoencoder (AE) and the cross-aligned autoencoder (Shen et al., 2017) for transfer. Note that in both our ARAE and standard AE experiments, the encoded code from the encoder is normalized to lie on the unit sphere, and the generated code is bounded to lie in (-1, 1) n by the tanh function at output layer. We additionally experimented with the sequence VAE introduced by Bowman et al. (2015b) and the adversarial autoencoder (AAE) model (Makhzani et al., 2015) on the SNLI dataset. However despite extensive parameter tuning we found that neither model was able to learn meaningful latent representations-the VAE simply ignored the latent code and the AAE experienced mode-collapse and repeatedly generated the same samples. The Appendix 12 includes detailed descriptions of the hyperparameters, model architecture, and training regimes.

6. EXPERIMENTS:

Our experiments consider three aspects of the model. First we measure the empirical impact of regularization on the autoencoder. Next we apply the discrete autoencoder to two applications, unaligned style transfer and semi-supervised learning. Finally we employ the learned generator network as an implicit latent variable model (ARAE-GAN) over discrete sequences.

6.1. IMPACT OF REGULARIZATION ON DISCRETE ENCODING:

Our main goal for ARAE is to regularize the model produce a smoother encoder by requiring the distribution from the encoder to match the distribution from the continuous generator over a simple latent variable. To examine this claim we consider two basic statistical properties of the code space during training of the text model on SNLI, shown in Figure 2 . On the left, we see that the 2 norm of c and code c converge quickly in ARAE training. The encoder code is always restricted to be on the unit sphere, and the generated code c quickly learns to match it. The middle plot shows the convergence of the trace of the covariance matrix between the generator and the encoder as training progresses. We find that variance of the encoder and the generator match after several epochs. To check the smoothness of the model, for both ARAE/AE, we take a sentence and calculate the average cosine similarity of 100 randomly-selected sentences that had an edit-distance of at most 5 to the original sentence. We do this for 250 sentences and calculate the mean of the average cosine similarity.

Figure 2 (right) shows that the cosine similarity of nearby sentences is quite high for the ARAE than in the case for the AE. Edit-distance is not an ideal proxy for similarity in sentences, but it is often a sufficient condition.

Finally an ideal representation should be robust to small changes of the input around the training examples in code space (Rifai et al., 2011) . We can test this property by feeding a noised input to the encoder and (i) calculating the score given to the original input, and (ii) checking the reconstructions. Table 1 (right) shows an experiment for text where we add noise by permuting k words in each sentence. We observe that the ARAE is able to map a noised sentence to a natural sentence, (though not necessarily the denoised sentence). Table 1 (left) shows empirical results for these experiments.

We obtain the reconstruction error (i.e. negative log likelihood) of the original (non-noised) sentence under the decoder, utilizing the noised code. We find that when k = 0 (i.e. no swaps), the regular AE better reconstructs the input as expected. However, as we increase the number of swaps and push the input further away from the data manifold, the ARAE is more likely to produce the original sentence. We note that unlike denoising autoencoders which require a domain-specific noising function (Hill et al., 2016; Vincent et al., 2008) , the ARAE is not explicitly trained to denoise an input, but learns to do so as a byproduct of adversarial regularization.

6.2. APPLICATIONS OF DISCRETE AUTOENCODER:

Unaligned Text Transfer A smooth autoencoder combined with low reconstruction error should make it possible to more robustly manipulate discrete objects through code space without dropping off the data manifold. To test this hypothesis, we experimented with two unaligned text transfer tasks. For these tasks, we attempt to change one attribute of a sentence without aligned examples of this change. To perform this transfer, we learn a code space that can represent an input that is agnostic to this attribute, and a decoder that can incorporate the attribute (as described in Section 4). We experiment with unaligned transfer of sentiment on the Yelp corpus and topic on the Yahoo corpus (Zhang et al., 2015) . For sentiment we follow the same setup as Shen et al. (2017) and split the Yelp corpus into two sets of unaligned positive and negative reviews. We train an ARAE as an autoencoder with two separate decoders, one for positive and one for negative sentiment, and incorporate adversarial training of the encoder to remove sentiment information from the code space. We test by encoding in sentences of one class and decoding, greedily, with the opposite decoder.

Our evaluation is based on four automatic metrics, shown in Table 2 : (i) Transfer: measuring how successful the model is at transferring sentiment based on an automatic classifier (we use the fastText library (Joulin et al., 2016) ). (ii) BLEU: measuring the consistency between the transferred text and the original. We expect the model to maintain as much information as possible and transfer only the style; (iii) Perplexity: measuring the fluency of the generated text; (iv) Reverse Perplexity: measuring the extent to which the generations are representative of the underlying data distribution.1 Both perplexity numbers are obtained by training an RNN language model.

We additionally perform human evaluations on the cross-aligned AE and our best ARAE model. We randomly select 1000 sentences (500/500 positive/negative), obtain the corresponding transfers from both models, and ask Amazon Mechanical Turkers to evaluate the sentiment (Positive/Neutral/Negative) and naturalness (1-5, 5 being most natural) of the transferred sentences. We create a separate task in which we show the Turkers the original and the transferred sentences, and ask them to evaluate the similarity based on sentence structure (1-5, 5 being most similar). We explicitly ask the Turkers to disregard sentiment in their similarity assessment.

In addition to comparing against the cross-aligned AE of Shen et al. (2017) , we also compare against a vanilla AE trained without adversarial regularization. For ARAE, we experimented with different λ (1) weighting on the adversarial loss (see section 4) with λ

(1)

EQUATION

= 10. We generally set λ (2) = 1. Experimentally the adversarial regularization enhances transfer and perplexity, but tends to make the transferred text less similar to the original, compared to the AE. Some randomly selected sentences are shown in figure 6 and more samples are shown available in Appendix 9.

The same method can be applied to other style transfer tasks, for instance the more challenging Yahoo QA data (Zhang et al., 2015) . For Yahoo we chose 3 relatively distinct topic classes for transfer: A man is on the corner in a sport area .

A man is on corner in a road all .

A lady is on outside a racetrack . A lady is outside on a racetrack . A lot of people is outdoors in an urban setting .

A lot of people is outdoors in an urban setting .

A lot of people is outdoors in an urban setting .

A man is on a ship path with the woman .

A man is on a ship path with the woman . A man is passing on a bridge with the girl . A man is passing on a bridge with the girl . A man is passing on a bridge with the girl . A man is passing on a bridge with the dogs . A man is passing on a bridge with the dogs .

A man in a cave is used an escalator .

A man in a cave is used an escalator A man in a cave is used chairs . A man in a number is used many equipment A man in a number is posing so on a big rock .

People are posing in a rural area . People are posing in a rural area. A man in a tie is sleeping and clapping on balloons . ⇒ walking A man in a tie is clapping and walking dogs . A person is standing in the air beneath a criminal .

⇒ walking A person is walking in the air beneath a pickup . The jewish boy is trying to stay out of his skateboard .

⇒man

The jewish man is trying to stay out of his horse . The people works in a new uniform studio .

⇒man A man works in a new studio uniform . Some child head a playing plastic with drink .

⇒ Two Two children playing a head with plastic drink . A baby workers is watching steak with the water .

⇒ Two Two workers watching baby steak with the grass . The people shine or looks into an area .

⇒ dog The dog arrives or looks into an area . The boy 's babies is wearing a huge factory .

⇒ dog The dog 's babies is wearing a huge ears . A women are walking outside near a man .

⇒ standing Three women are standing near a man walking . The dogs are sleeping in front of the dinner .

⇒ standing Two dogs are standing in front of the dinner . A side child listening to a piece with steps playing on a table .

⇒ Several Several child playing a guitar on side with a table . Two children are working in red shirt at the cold field .

⇒ Several Several children working in red shirt are cold at the field . shown in Appendix 10). All models are of the same size to allow for fair comparison. We train an RNN language model on generated samples and evaluate on held-out data to calculate the reverse perplexity. As can be seen from Table 5 , training on real data (understandably) outperforms training on generated data by a large margin. Surprisingly however, we find that a language model trained on ARAE-GAN data performs slightly better than one trained on LM-generated/AE-generated data. We further found that the reverse PPL of an AAE (Makhzani et al., 2015) was quite high (980) due to mode-collapse.

Another property of GANs (and VAEs) is that the Gaussian form of z induces the ability to smoothly interpolate between outputs by exploiting the structure of the latent space. While language models may provide a better estimate of the underlying probability space, constructing this style of interpolation would require combinatorial search, which makes this a useful feature of text GANs. We experiment with this property by sampling two points z 0 and z 1 from p(z) and constructing intermediary points

z λ = λz 1 + (1 -λ)z 0 .

For each we generate the argmax output xλ . The samples are shown in Figure 3 (left) for text and in Figure 3 (right) for a discretized MNIST ARAE-GAN.

A final intriguing property of image GANs is the ability to move in the latent space via offset vectors (similar to the case with word vectors (Mikolov et al., 2013) ). For example, Radford et al. (Radford et al., 2016) observe that when the mean latent vector for "men with glasses" is subtracted from the mean latent vector for "men without glasses" and applied to an image of a "woman without glasses", the resulting image is that of a "woman with glasses". To experiment with this property we generate 1 million sentences from the ARAE-GAN and compute vector transforms in this space to attempt to change main verbs, subjects and modifier (details in Appendix 11). Some examples of successful transformations are shown in Figure 4 (right). Quantitative evaluation of the success of the vector transformations is given in Figure 4 (left).

7. CONCLUSION:

We present adversarially regularized autoencoders, as a simple approach for training a discrete structure autoencoder jointly with a code-space generative adversarial network. The model learns a improved autoencoder as demonstrated by semi-supervised experiments and improvements on text transfer experiments. It also learns a useful generative model for text that exhibits a robust latent space, as demonstrated by natural interpolations and vector arithmetic. We do note that (as has been frequently observed when training GANs) our model seemed to be quite sensitive to hyperparameters. Finally, while many useful models for text generation already exist, text GANs provide a qualitatively different approach influenced by the underlying latent variable structure. We envision that such a framework could be extended to a conditional setting, combined with other existing decoding schemes, or used to provide a more interpretable model of language.

8 APPENDIX: OPTIMALITY PROPERTY One can interpret the ARAE framework as a dual pathway network mapping two distinct distributions into a similar one; enc φ and g θ both output code vectors that are kept similar in terms of Wasserstein distance as measured by the critic. We provide the following proposition showing that under our parameterization of the encoder and the generator, as the Wasserstein distance converges, the encoder distribution (c ∼ P r ) converges to the generator distribution (c ∼ P g ), and further, their moments converge.

This is ideal since under our setting the generated distribution is simpler than the encoded distribution, because the input to the generator is from a simple distribution (e.g. spherical Gaussian) and the generator possesses less capacity than the encoder. However, it is not so simple that it is overly restrictive (e.g. as in VAEs). Empirically we observe that the first and second moments do indeed converge as training progresses (Section 6.1).

Proposition 1. Let P be a distribution on a compact set χ , and (P n ) n∈N be a sequence of distributions on χ . Further suppose that W (P n , P) → 0. Then the following statements hold:

(i) P n P (i.e. convergence in distribution).

(ii) All moments converge, i.e. for all k > 1, k ∈ N,

E X∼Pn d i=1 X pi i → E X∼P d i=1 X pi i for all p 1 , . . . , p d such that d i=1 p i = k

Proof. (i) has been proved in Villani (2008) Theorem 6.9.

For (ii), using The Portmanteau Theorem, (i) is equivalent to: ) ] for all bounded and continuous function f : R d → R, where d is the dimension of the random variable.

E X∼Pn [f (X)] → E X∼P [f (X

The k-th moment of a distribution is given by

E d i=1 X pi i such that d i=1 p i = k

Our encoded code is bounded as we normalize the encoder output to lie on the unit sphere, and our generated code is also bounded to lie in (-1, 1) n by the tanh function. Hence f (X) = d i=1 X qi i is a bounded continuous function for all q i > 0. Therefore, Original we came on the recommendation of a bell boy and the food was amazing . Original the people who ordered off the menu did n't seem to do much better . Cross-AE we came on the recommendation and the food was a joke . ARAE the people who work there are super friendly and the menu is good . Cross-AE we went on the car of the time and the chicken was awful . Cross-AE the place , one of the office is always worth you do a business .

E X∼Pn d i=1 X pi i → E X∼P d i=1

Original service is good but not quick , just enjoy the wine and your company . Original they told us in the beginning to make sure they do n't eat anything . ARAE service is good but not quick , but the service is horrible . ARAE they told us in the mood to make sure they do great food . Cross-AE service is good , and horrible , is the same and worst time ever . Cross-AE they 're us in the next for us as you do n't eat .

Original the steak was really juicy with my side of salsa to balance the flavor . Original the person who was teaching me how to control my horse was pretty rude . ARAE the steak was really bland with the sauce and mashed potatoes . ARAE the person who was able to give me a pretty good price . Cross-AE the fish was so much , the most of sauce had got the flavor . Cross-AE the owner 's was gorgeous when i had a table and was friendly .

Original other than that one hell hole of a star bucks they 're all great ! Original he was cleaning the table next to us with gloves on and a rag . ARAE other than that one star rating the toilet they 're not allowed . ARAE he was prompt and patient with us and the staff is awesome . Cross-AE a wonder our one came in a _num_ months , you 're so better ! Cross-AE he was like the only thing to get some with with my hair . 

:

This reverse perplexity is calculated by training a language model on the generated data and measuring perplexity on held-out, real data (i.e. reverse of regular perplexity). We also found this metric to be helpful for early-stopping based on validation data.

To "sample" from an AE we fit a multivariate Gaussian to the code space after training and generate code vectors from this Gaussian to decode back into sentence space.

The GAN training loop refers to how many times we train GAN in each entire training loop (one training loop contains training autoencoder for one loop, and training GAN for one or several).

Figures and Tables:

, and (3) an autoencoder trained Algorithm 2 ARAE Transfer Extension [Each loop additionally:] (2b) Train the code classifier [minu Lclass(φ, u)]

Figure 2: Left: 2 norm of encoder code c and generator code c during ARAE training. The encoder c is normalized by the model, whereas the generator learns to match this as training progresses. Middle: Sum of the dimension-wise variances of the encoder codes Pr and generator codes Pg compared to that of the standard AE. Right: Average cosine similarity of nearby sentences (edit-distance wise) for the ARAE and AE.

Figure 3: Sample interpolations from the ARAE-GAN. Constructed by linearly interpolating in the latent space and decoding to the output space. Word changes are highlighted in black. Results of the ARAE. The top block shows output generation of the decoder taking fake hidden codes generated by the GAN; the bottom block shows sample interpolation results.

Figure 4: Left. Quantitative evaluation of transformations. Match % refers to the % of samples where at least one decoder samples (per 100) had the desired transformation in the output, while Prec. measures the average precision of the output against the original sentence. Right. Examples (out of 100 decoder samples per sentence) where the offset vectors produced successful transformations of the original sentence. See Appendix 11 for full methodology.

Left. Reconstruction error (negative log-likelihood averaged over sentences) of the original sentence from a corrupted sentence. Here k is the number of swaps performed on the original sentence. Right. Samples generated from AE and ARAE where the input is noised by swapping words.

Table content: <table><tr><td>k</td><td>AE</td><td>ARAE</td><td>Original</td><td>A woman wearing sunglasses .</td><td>Original</td><td>They have been swimming .</td></tr><tr><td/><td/><td/><td>Noised</td><td>A woman sunglasses wearing .</td><td>Noised</td><td>been have They swimming .</td></tr><tr><td colspan="2">0 1.06</td><td>2.19</td><td>AE</td><td>A woman sunglasses wearing sunglasses .</td><td>AE</td><td>been have been swimming .</td></tr><tr><td colspan="2">1 4.51</td><td>4.07</td><td>ARAE</td><td>A woman wearing sunglasses .</td><td>ARAE</td><td>Children have been swimming .</td></tr><tr><td colspan="2">2 6.61 3 9.14</td><td>5.39 6.86</td><td>Original Noised AE</td><td>Pets galloping down the street . Pets down the galloping street . Pets riding the down galloping .</td><td>Original Noised AE</td><td>The child is sleeping . child The is sleeping . The child is sleeping is .</td></tr><tr><td colspan="2">4 9.97</td><td>7.47</td><td>ARAE</td><td>Pets congregate down the street near a ravine .</td><td>ARAE</td><td>The child is sleeping .</td></tr></table>

came on the recommendation of a bell boy and the food was amazing . the people who ordered off the menu did n't seem to do much better . ARAE we came on the recommendation and the food was a joke . ARAE the people who work there are super friendly and the menu is good . Cross-AE we went on the car of the time and the chicken was awful .Cross-AE the place , one of the office is always worth you do a business .

Table content: <table><tr><td/><td/><td/><td colspan="2">Automatic Evaluation</td><td/><td colspan="2">Human Evaluation</td></tr><tr><td/><td>Model</td><td colspan="7">Transfer BLEU PPL Reverse PPL Transfer Similarity Naturalness</td></tr><tr><td/><td>Cross-Aligned AE</td><td>77.1%</td><td>17.75</td><td>65.9</td><td>124.2</td><td>57%</td><td>3.8</td><td>2.7</td></tr><tr><td/><td>AE</td><td>59.3%</td><td>37.28</td><td>31.9</td><td>68.9</td><td>-</td><td>-</td><td>-</td></tr><tr><td/><td>ARAE, λ (1) a</td><td>73.4%</td><td>31.15</td><td>29.7</td><td>70.1</td><td>-</td><td>-</td><td>-</td></tr><tr><td/><td>ARAE, λ (1) b</td><td>81.8%</td><td>20.18</td><td>27.7</td><td>77.0</td><td>74%</td><td>3.7</td><td>3.8</td></tr><tr><td colspan="9">Table 2: Experiments on sentiment transfer. Left shows the automatic metrics (Transfer/BLEU/PPL/Reverse</td></tr><tr><td colspan="9">PPL) while right shows human evaluation metrics (Transfer/Similarity/Naturalness). Cross-Aligned AE is from</td></tr><tr><td colspan="2">Shen et al. (2017)</td><td/><td/><td/><td/><td/><td/></tr><tr><td/><td>Positive ⇒ Negative</td><td/><td/><td/><td/><td>Negative ⇒ Positive</td><td/></tr><tr><td/><td>great indoor mall .</td><td/><td/><td/><td/><td>hell no !</td><td/></tr><tr><td>ARAE</td><td>no smoking mall .</td><td/><td/><td/><td>ARAE</td><td>hell great !</td><td/></tr><tr><td>Cross-AE</td><td>terrible outdoor urine .</td><td/><td/><td/><td>Cross-AE</td><td>incredible pork !</td><td/></tr><tr><td/><td colspan="3">it has a great atmosphere , with wonderful service .</td><td/><td/><td colspan="2">small , smokey , dark and rude management .</td></tr><tr><td>ARAE</td><td colspan="2">it has no taste , with a complete jerk .</td><td/><td/><td>ARAE</td><td colspan="2">small , intimate , and cozy friendly staff .</td></tr><tr><td>Cross-AE</td><td colspan="3">it has a great horrible food and run out service .</td><td/><td>Cross-AE</td><td>great , , , chips and wine .</td><td/></tr></table>

Sentiment transfer results. Original sentence and transferred output (from ARAE and the Cross-Aligned AE) of 6 randomly-drawn examples.

Table content: <table/>

Science & Math, Entertainment & Music, and Politics & Government. As the dataset contains both Left

Table content: <table><tr><td>Model</td><td colspan="2">Medium Small</td><td>Tiny</td><td>Data for LM</td><td>Reverse PPL</td></tr><tr><td>Supervised Encoder Semi-Supervised AE Semi-Supervised ARAE</td><td>65.9% 68.5% 70.9%</td><td colspan="2">62.5% 57.9% 64.6% 59.9% 66.8% 62.5%</td><td>Real data LM samples AE samples ARAE-GAN samples</td><td>27.4 90.6 97.3 82.2</td></tr></table>

Original their menu is extensive , even have italian food . Original the restaurant did n't meet our standard though . ARAE their menu is limited , even if i have an option . ARAE the restaurant did n't disappoint our expectations though . Cross-AE their menu is decent , i have gotten italian food . Cross-AE the restaurant is always happy and knowledge . Original everyone who works there is incredibly friendly as well . Original you could not see the stage at all ! ARAE everyone who works there is incredibly rude as well . ARAE you could see the difference at the counter ! Cross-AE everyone who works there is extremely clean and as well . Cross-AE you could definitely get the fuss ! Original there are a couple decent places to drink and eat in here as well . Original room is void of all personality , no pictures or any sort of decorations . ARAE there are a couple slices of options and _num_ wings in the place . ARAE room is eclectic , lots of flavor and all of the best . Cross-AE there are a few night places to eat the car here are a crowd . Cross-AE it 's a nice that amazing , that one 's some of flavor . Original if you 're in the mood to be adventurous , this is your place ! Original waited in line to see how long a wait would be for three people . ARAE if you 're in the mood to be disappointed , this is not the place . ARAE waited in line for a long wait and totally worth it . Cross-AE if you 're in the drive to the work , this is my place ! Cross-AE another great job to see and a lot going to be from dinner .

Table content: <table><tr><td colspan="5">9 APPENDIX: SHEET OF STYLE-TRANSFER SAMPLES</td></tr><tr><td colspan="3">YELP TRANSFER</td><td/></tr><tr><td/><td colspan="2">Positive to Negative</td><td/><td>Negative to Positive</td></tr><tr><td>Original</td><td colspan="2">great indoor mall .</td><td>Original</td><td>hell no !</td></tr><tr><td>ARAE</td><td colspan="2">no smoking mall .</td><td>ARAE</td><td>hell great !</td></tr><tr><td>Cross-AE</td><td colspan="2">terrible outdoor urine .</td><td>Cross-AE</td><td>incredible pork !</td></tr><tr><td>Original</td><td colspan="2">great blooming onion .</td><td>Original</td><td>highly disappointed !</td></tr><tr><td>ARAE</td><td colspan="2">no receipt onion .</td><td>ARAE</td><td>highly recommended !</td></tr><tr><td>Cross-AE</td><td colspan="2">terrible of pie .</td><td>Cross-AE</td><td>highly clean !</td></tr><tr><td>Original</td><td colspan="2">i really enjoyed getting my nails done by peter .</td><td>Original</td><td>bad products .</td></tr><tr><td>ARAE</td><td colspan="2">i really needed getting my nails done by now .</td><td>ARAE</td><td>good products .</td></tr><tr><td>Cross-AE</td><td colspan="2">i really really told my nails done with these things .</td><td>Cross-AE</td><td>good prices .</td></tr><tr><td>Original</td><td colspan="2">definitely a great choice for sushi in las vegas !</td><td>Original</td><td>i was so very disappointed today at lunch .</td></tr><tr><td>ARAE</td><td colspan="2">definitely a _num_ star rating for _num_ sushi in las vegas .</td><td>ARAE</td><td>i highly recommend this place today .</td></tr><tr><td>Cross-AE</td><td colspan="2">not a great choice for breakfast in las vegas vegas !</td><td>Cross-AE</td><td>i was so very pleased to this .</td></tr><tr><td>Original</td><td colspan="2">the best piece of meat i have ever had !</td><td>Original</td><td>i have n't received any response to anything .</td></tr><tr><td>ARAE</td><td colspan="2">the worst piece of meat i have ever been to !</td><td>ARAE</td><td>i have n't received any problems to please .</td></tr><tr><td>Cross-AE</td><td colspan="2">the worst part of that i have ever had had !</td><td>Cross-AE</td><td>i have always the desert vet .</td></tr><tr><td>Original</td><td colspan="2">really good food , super casual and really friendly .</td><td>Original</td><td>all the fixes were minor and the bill ?</td></tr><tr><td>ARAE</td><td colspan="2">really bad food , really generally really low and decent food .</td><td>ARAE</td><td>all the barbers were entertaining and the bill did n't disappoint .</td></tr><tr><td>Cross-AE</td><td colspan="2">really good food , super horrible and not the price .</td><td>Cross-AE</td><td>all the flavors were especially and one !</td></tr><tr><td>Original</td><td colspan="2">it has a great atmosphere , with wonderful service .</td><td>Original</td><td>small , smokey , dark and rude management .</td></tr><tr><td>ARAE</td><td colspan="2">it has no taste , with a complete jerk .</td><td>ARAE</td><td>small , intimate , and cozy friendly staff .</td></tr><tr><td>Cross-AE</td><td colspan="2">it has a great horrible food and run out service .</td><td>Cross-AE</td><td>great , , , chips and wine .</td></tr><tr><td/><td/><td/><td/><td>X pi i</td></tr><tr><td colspan="2">where</td><td>d i=1 p i = k</td><td/></tr></table>

Full sheet of sentiment transfer result

Table content: <table/>

Peer review (iclr2018):
 I was asked to contribute this review rather late in the process, and in order to remain unbiased I avoided reading other reviews. I apologize if some of these comments have already been addressed in replies to other reviewers. This paper proposes a regularization strategy for autoencoders that is very similar to the adversarial autoencoder of Makhzani et al. The main difference appears to be that rather than using the classic GAN loss to shape the aggregate posterior of an autoencoder to match a chosen, fixed distribution, they instead employ a Wasserstein GAN loss (and associated weight magnitude constraint, presumably enforced with projected gradient descent) on a system where the matched distribution is instead learned via a parameterized sampler ("generator" in the GAN lingo). Gradient steps that optimize the encoder, decoder and generator are interleaved. The authors apply an extension of this method to topic and sentiment transfer and show moderately good latent space interpolations between generated sentences. The difference from the original AAE is rather small and straightforward, making the novelty mainly in the choice of task, focusing on discrete vectors and sequences. The exposition leaves ample room for improvement. For one thing, there is the irksome and repeated use of "discrete structure" when discrete *sequences* are considered almost exclusively (with the exception of discretized MNIST digits). The paper is also light on discussion of related work other than Makhzani et al -- the wealth of literature on combining autoencoders (or autoencoder-like structures such as ALI/BiGAN) and GANs merits at least passing mention. The empirical work is somewhat compelling, though I am not an expert in this task domain. The annealed importance sampling technique of Wu et al (2017) for estimating bounds on a generator*s log likelihood could be easily applied in this setting and would give (for example, on binarized MNIST) a quantitative measurement of the degree of overfitting, and this would have been preferable than inventing new heuristic measures. The "Reverse PPL" metric requires more justification, and it looks an awful lot like the long-since-discredited Parzen window density estimation technique used in the original GAN paper. High-level comments: - It*s not clear why the optimization is done in 3 separate steps. Aside from the WGAN critic needing to be optimized for more steps, couldn*t the remaining components be trained jointly, with a weighted sum of terms for the encoder? - In section 2, "This [pre-training or co-training with maximum likelihood] precludes there being a latent encoding of the sentence." It is not at all clear to me why this would be the case. - "One benefit of the ARAE framework is that it compresses the input to a single code vector." This is true of any autoencoder. - It would be worth explaining, in a sentence, the approach in Shen et al for those who are not familiar with it, seeing as it is used as a baseline. - We are told that the encoder*s output is l2-normalized but the generator*s is not, instead output units of the generator are squashed with the tanh activation. The motivation for this choice would be helpful. Shortly thereafter we are told that the generator quickly learns to produce norm 1 outputs as evidence that it is matching the encoder*s distribution, but this is something that could have just as easily have been built-in, and is a trivial sort of "distribution matching" - In general, tables that report averages would do well to report error bars as well. In general some more nuanced statistical analysis of these results would be worthwhile, especially where they concern human ratings. - The dataaset fractions chosen for the semi-supervised experience seem completely arbitrary. Is this protocol derived from some other source? Putting these in a table along with the results would improve readability. - Linear interpolation in latent space may not be the best choice here seeing as e.g. for a Gaussian code the region near the origin has rather low probability. Spherical interpolation as recommended by White (2016) may improve qualitative results. - For the interpolation results you say "we output the argmax", what is meant? Is beam search performed in the case of sequences? - Finally, a minor point: I will challenge the authors to justify their claim that the learned generative model is "useful" (their word). Interpolating between two sentences sampled from the prior is a neat parlour trick, but the model as-is has little utility. Even some speculation on how this aspect could be applied would be appreciated (admittedly, many GAN papers could use some reflection of this sort).