Title: DENOISING IMPROVES LATENT SPACE GEOMETRY IN TEXT AUTOENCODERS

Abstract: Neural language models have recently shown impressive gains in unconditional text generation, but controllable generation and manipulation of text remain challenging. In particular, controlling text via latent space operations in autoencoders has been difficult, in part due to chaotic latent space geometry. We propose to employ adversarial autoencoders together with denoising (referred as DAAE) to drive the latent space to organize itself. Theoretically, we prove that input sentence perturbations in the denoising approach encourage similar sentences to map to similar latent representations. Empirically, we illustrate the trade-off between textgeneration and autoencoder-reconstruction capabilities, and our model significantly improves over other autoencoder variants. Even from completely unsupervised training without style information, DAAE can perform various style transfers, including tense and sentiment, through simple latent vector arithmetic. 1

1. INTRODUCTION:

Autoencoder based generative models have recently become popular tools for advancing controllable text generation such as style or sentiment transfer (Bowman et al., 2016; Hu et al., 2017; Shen et al., 2017; Zhao et al., 2018) . By mapping sentences to vectors in the latent space, these models offer in principle an attractive, continuous approach to manipulating text by means of simple latent vector arithmetic. However, the success of such manipulations rests heavily on the latent space geometry and how well it agrees with underlying sentence semantics. Indeed, we demonstrate that without additional guidance, fortuitous geometric agreements are unlikely to arise, shedding light on challenges faced by existing methods.

We use adversarial autoencoders (Makhzani et al., 2015, AAEs) to study the latent space geometry. In contrast to variational autoencoders (Kingma & Welling, 2014, VAEs) , AAEs can maintain strong coupling between the encoder and decoder that the decoder does not omit the encoded input sentence (Bowman et al., 2016) . The training criterion for AAEs consists of two parts, the ability to reconstruct sentences and the additional constraint that the encoded sentences are overall indistinguishable from prior samples, typically Gaussian. We show that these objectives alone do not suffice to force proper latent space geometry for text control. Specifically, for discrete objects such as sentences where continuity assumptions no longer hold, powerful AAEs can easily learn to map training sentences into latent prior samples arbitrarily (Figure 1 , Left), while retaining perfect reconstruction. Latent space manipulations in such cases will yield random, unpredictable results.

To remedy this, we augment AAEs with a simple denoising objective (Vincent et al., 2008; Creswell & Bharath, 2018) that requires perturbed sentence with some words missing to be mapped back to the original version. We refer to our model as DAAE. We prove that the denoising criterion can eliminate disorganized solutions and drive the latent space to organize itself. As a result, similar sentences begin to be mapped to similar latent vectors (Figure 1 , Right).

Improvements in latent space geometry carry many positive consequences. Through systematic evaluations of the generation and reconstruction capabilities of various text autoencoders (Cífka et al., 2018) , we find that our proposed DAAE provides the best trade-off between producing highquality text vs. informative sentence representations. We empirically verify that DAAE has the best neighborhood preservation property, consistent with our theory. We further investigate to what extent X < l a t e x i t s h a 1 _ b a s e 6 4 = " l E x 7 P B c r k D b y M 6 j F z T W Y Y B H + h 3 E = " > A A A B 8 X i c b V D L S g M x F L 1 T X 7 W + q i 7 d B I v g q s x U Q Z d F N y 4 r 2 A e 2 Q 8 m k m T Y 0 k x m S O 0 I Z + h d u X C j i 1 r 9 x 5 9 + Y t r P Q 1 g O B w z n 3 k n N P k E h h 0 H W / n c L a + s b m V n G 7 t L O 7 t 3 9 Q P j x q m T j V j D d Z L G P d C a j h U i j e R I G S d x L N a R R I 3 g 7 G t z O / / c S 1 E b F 6 w E n C / Y g O l Q g F o 2 i l x 1 5 E c c S o J J 1 + u e J W 3 T n I K v F y U o E c j X 7 5 q z e I W R p x h U x S Y 7 q e m 6 C f U Y 2 C S T 4 t 9 V L D E 8 r G d M i 7 l i o a c e N n 8 8 R T c m a V A Q l j b Z 9 C M l d / b 2 Q 0 M m Y S B X Z y l t A s e z P x P 6 + b Y n j t Z 0 I l K X L F F h + F q S Q Y k 9 n 5 Z C A 0 Z y g n l l C m h c 1 K 2 I h q y t C W V L I l e M s n r 5 J W r e p d V G v 3 l 5 X 6 T V 5 H E U 7 g F M 7 B g y u o w x 0 0 o A k M F D z D K 7 w 5 x n l x 3 p 2 P x W j B y X e O 4 Q + c z x 8 j x p C Q < / l a t e x i t > X < l a t e x i t s h a 1 _ b a s e 6 4 = " l E x 7 P B c r k D b y M 6 j F z T W Y Y B H + h 3 E = " > A A A B 8 X i c b V D L S g M x F L 1 T X 7 W + q i 7 d B I v g q s x U Q Z d F N y 4 r 2 A e 2 Q 8 m k m T Y 0 k x m S O 0 I Z + h d u X C j i 1 r 9 x 5 9 + Y t r P Q 1 g O B w z n 3 k n N P k E h h 0 H W / n c L a + s b m V n G 7 t L O 7 t 3 9 Q P j x q m T j V j D d Z L G P d C a j h U i j e R I G S d x L N a R R I 3 g 7 G t z O / / c S 1 E b F 6 w E n C / Y g O l Q g F o 2 i l x 1 5 E c c S o J J 1 + u e J W 3 T n I K v F y U o E c j X 7 5 q z e I W R p x h U x S Y 7 q e m 6 C f U Y 2 C S T 4 t 9 V L D E 8 r G d M i 7 l i o a c e N n 8 8 R T c m a V A Q l j b Z 9 C M l d / b 2 Q 0 M m Y S B X Z y l t A s e z P x P 6 + b Y n j t Z 0 I l K X L F F h + F q S Q Y k 9 n 5 Z C A 0 Z y g n l l C m h c 1 K 2 I h q y t C W V L I l e M s n r 5 J W r e p d V G v 3 l 5 X 6 T V 5 H E U 7 g F M 7 B g y u o w x 0 0 o A k M F D z D K 7 w 5 x n l x 3 p 2 P x W j B y X e O 4 Q + c z x 8 j x p C Q < / l a t e x i t > Z < l a t e x i t s h a 1 _ b a s e 6 4 = " Y v 1 y O d p 3 y C 9 y 8 P / v i D / 6 k H / o k o U = " > A A A B 8 X i c b V D L S g M x F L 3 j s 9 Z X 1 a W b Y B F c l Z k q 6 L L o x m U F + 6 D t U D L p n T Y 0 k x m S j F B K / 8 K N C 0 X c + j f u / B s z 7 S y 0 9 U D g c M 6 9 5 N w T J I J r 4 7 r f z t r 6 x u b W d m G n u L u 3 f 3 B Y O j p u 6 j h V D B s s F r F q B 1 S j 4 B I b h h u B 7 U Q h j Q K B r W B 8 l / m t J 1 S a x / L R T B L 0 I z q U P O S M G i t 1 e h E 1 I 0 Y F 6 f R L Z b f i z k F W i Z e T M u S o 9 0 t f v U H M 0 g i l Y Y J q 3 f X c x P h T q g x n A m f F X q o x o W x M h 9 i 1 V N I I t T + d J 5 6 R c 6 s M S B g r + 6 Q h c / X 3 x p R G W k + i w E 5 m C f W y l 4 n / e d 3 U h D f + l M s k N S j Z 4 q M w F c T text can be manipulated by applying simple transformations in the learned latent space. Our model is able to perform sentence-level vector arithmetic (Mikolov et al., 2013) fairly well to change the tense or sentiment of a sentence without any training supervision. It also produces higher quality sentence interpolations than other text autoencoders, suggesting better linguistic continuity in its latent space (Bowman et al., 2016) .

E J D u f D L h C Z s T E E s o U t 1 k J G 1 F F m b E l F W 0 J 3 v L J q 6 R Z r X i X l e r D V b l 2 m 9 d R g F M 4 g w v w 4 B p q c A 9 1 a A A D C c / w C m + O d l 6 c d + d j M b r m 5 D s n 8 A f O 5 w 8 m z p C S < / l a t e x i t > Z < l a t e x i t s h a 1 _ b a s e 6 4 = " Y v 1 y O d p 3 y C 9 y 8 P / v i D / 6 k H / o k o U = " > A A A B 8 X i c b V D L S g M x F L 3 j s 9 Z X 1 a W b Y B F c l Z k q 6 L L o x m U F + 6 D t U D L p n T Y 0 k x m S j F B K / 8 K N C 0 X c + j f u / B s z 7 S y 0 9 U D g c M 6 9 5 N w T J I J r 4 7 r f z t r 6 x u b W d m G n u L u 3 f 3 B Y O j p u 6 j h V D B s s F r F q B 1 S j 4 B I b h h u B 7 U Q h j Q K B r W B 8 l / m t J 1 S a x / L R T B L 0 I z q U P O S M G i t 1 e h E 1 I 0 Y F 6 f R L Z b f i z k F W i Z e T M u S o 9 0 t f v U H M 0 g i l Y Y J q 3 f X c x P h T q g x n A m f F X q o x o W x M h 9 i 1 V N I I t T + d J 5 6 R c 6 s M S B g r + 6 Q h c / X 3 x p R G W k + i w E 5 m C f W y l 4 n / e d 3 U h D f + l M s k N S j Z 4 q M w F c T E J D u f D L h C Z s T E E s o U t 1 k J G 1 F F m b E l F W 0 J 3 v L J q 6 R Z r X i X l e r D V b l 2 m 9 d R g F M 4 g w v w 4 B p q c A 9 1 a A A D C c / w C m + O d l 6 c d + d j M b r m 5 D s n 8 A f O 5 w 8 m z p C S < / l a t e x i t > E G C(x) < l a t e x i t s h a 1 _ b a s e 6 4 = " w C Q d G N j w P 4 M Y E 5 Y P m 9 z H o y O 3 g 6 I = " > A A A B 6 3 i c b V B N T w I x E J 3 F L 8 Q v 1 K O X R m K C F 7 K L J n I k 4 e I R E w E T 2 J B u 6 U J D 2

8 K h + f d H W U K E I 7 J O K R e g i w p p x J 2 j H M c P o Q K 4 p F w G k v m L Y y v / d I l W a R v D e z m P o C j y U L G c E m k 1 r V p 8 t h u e L W 3 A X Q O v F y U o E c 7 W H 5 a z C K S C K o N I R j r f u e G x s / x c o w w u m 8 N E g 0 j T G Z 4 j H t W y q x o N p P F 7 f O 0 Y V V R i i M l C 1 p 0 E L 9 P Z F i o f V M B L Z T Y D P R q 1 4 m / u f 1 E x M 2 / J T J O D F U k u W i M O H I R C h 7 H I 2 Y o s T w m S W Y K G Z v R W S C F S b G x l O y I X i r L 6 + T b r 3 m X d X q d 9 e V Z i O P o w h n c A 5 V 8 O A G m n A L b e g A g Q k 8 w y u 8 O c J 5 c d 6 d j 2 V r w c l n T u E P n M 8 f N T a N q A = = < / l a t e x i t >

2. RELATED WORK:

Denoising is first introduced into standard autoencoders by Vincent et al. (2008, DAE) to learn robust representations. Without a latent prior, DAE requires sophisticated MCMC sampling to be employed generatively (Bengio et al., 2013) . Creswell & Bharath (2018) applied denoising with AAEs to generative image modeling. Here, we demonstrate that input perturbations are particularly useful for discrete text modeling because they encourage preservation of data structure in the latent space.

Apart from the AAE framework that our paper focuses on, another popular latent variable generative model is the variational autoencoder (Kingma & Welling, 2014, VAE) . Unfortunately, when the decoder is a powerful autoregressive model (such as a language model), VAE suffers from the posterior collapse problem where the latent representations get ignored (Bowman et al., 2016; Chen et al., 2016) . If denoising is used in conjunction with VAEs (Im et al., 2017) in text applications, then the noisy inputs will only exacerbate VAE's neglect of the latent variable. Bowman et al. (2016) proposed to weaken VAE's decoder by masking words on the decoder side to alleviate its collapse issue. However, even with a weakened decoder and combined with other techniques including KL-weight annealing and adjusting training dynamics, it is still difficult to inject significant content into the latent code (Yang et al., 2017; Kim et al., 2018; He et al., 2019) . Alternatives like the β-VAE (Higgins et al., 2017) appear necessary.

Previous work on controllable text generation has employed autoencoders trained with attribute label information (Hu et al., 2017; Shen et al., 2017; Zhao et al., 2018; Logeswaran et al., 2018; Subramanian et al., 2018) . We show that the proposed DAAE model can perform text manipulations despite being trained in a completely unsupervised manner without attribute labels. This suggests that on the one hand, our model can be adapted to semi-supervised learning when a few labels are available. On the other hand, it can be easily scaled up to train one large model on unlabeled text corpora and then applied for transferring various styles.

3. METHOD:

Define X = V m to be a space of sequences of discrete symbols from vocabulary V (with maximum length m); also define Z = R d to be a continuous latent space. Our goal is to learn a mapping between the data distribution p data (x) over X and a given prior distribution p(z) over latent space Z (following common practice, a Gaussian prior is used in our experiments, although not required by our methodology). Such a mapping allows us to easily manipulate discrete data through continuous latent representations z, and provides a generative model where data samples can be obtained by first drawing z from the prior and then sampling a corresponding sequence via p(x|z).

We adopt the adversarial autoencoder (AAE) framework, which involves a (deterministic) encoder E : X → Z, a probabilistic decoder G : Z → X , and a discriminator D : Z → [0, 1] . Both E and G are recurrent neural networks (RNNs)2 . E takes input sequence x and outputs the last hidden state as its encoding z. G generates a sequence x autoregressively, with each step conditioned on z and previous symbols. The discriminator D is a feed-forward net that outputs the probability of z coming from the prior rather than the encoder. E, G and D are trained jointly with a min-max objective:

EQUATION

with:

EQUATION

where reconstruction loss L rec and adversarial loss3 L adv are weighted via hyperparameter λ > 0.

We further introduce perturbations in X space to learn smoother representations that reflect local structure in the data, ending up with the denoising adversarial autoencoder (DAAE) model. Given a perturbation process C that stochastically maps x to nearby x ∈ X , let p(x, x) = p data (x)p C (x|x) and p(x) = x p(x, x). We change the loss functions to be:

EQUATION

Here, L rec is the loss of reconstructing x from x, and L adv is the adversarial loss evaluated on perturbed x. The objective function combines the denoising technique with the AAE (Vincent et al., 2008; Creswell & Bharath, 2018) . When p C (x|x) = 1[x = x] (i.e. there is no perturbation), the above simply becomes the usual AAE objective.

Let p E (z|x) denote the encoder distribution. With our perturbation process C, the posterior distributions of the DAAE are of the following form:

EQUATION

This enables the DAAE to utilize stochastic encodings even by merely employing a deterministic encoder network trained without any reparameterization-style tricks. Note that since q(z|x) of the form (6) is a subset of all possible conditional distributions, our model is still minimizing an upper bound of the Wasserstein distance between data and model distributions, as previously shown by Tolstikhin et al. (2017) for AAE (see Appendix A for a full proof).

4. LATENT SPACE GEOMETRY:

The latent space geometry of text autoencoders is an important yet understudied problem. Only when the latent space is smooth and regular can meaningful text manipulations be enacted via simple modifications of the corresponding latent representations. Here, we discuss in detail the posterior characteristics of the DAAE, and provide a theoretical analysis of how input perturbations help better structure the latent space geometry (all proofs are relegated to the appendix).

Assume our perturbations preserve x with some probability (i.e. p C (x|x) > 0). When the support of C(x 1 ) and C(x 2 ) do not overlap for different training examples x 1 = x 2 , the encoder can learn to assign p E (z|x) = p E (z|x) for x ∈ C(x), and we are back to the unconstrained posterior scenario q(z|x) = p E (z|x) (Eq. 6). If C(x 1 ) and C(x 2 ) do intersect, then the latent posterior of x 1 and x 2 will have overlapping components p E (z|x) for x ∈ C(x 1 ) ∩ C(x 2 ). For example, if p C (x|x) assigns a high probability to x that lies close to x (based on some metric over X ), then for similar x 1 and x 2 , the high-probability overlap between their perturbations will inherently force their posteriors closer together in the latent space. This is desirable for learning good representations z, while not guaranteed by merely minimizing the statistical divergence between p data (x) and p G

(x) = E p(z) [p G (x|z)].

noise in single-layer networks (Poole et al., 2014) , here we study high-capacity encoder/decoder networks (Schäfer & Zimmermann, 2006) with a large number of parameters that are used in modern sequence models (Devlin et al., 2018; Radford et al., 2019) . Throughout, we assume that: Assumption 1. E is a universal approximator capable of producing any mapping from x's to z's. Assumption 2. G can approximate arbitrary p(x|z) so long as it remains sufficiently Lipschitz continuous in z. Namely, there exists L > 0 such that all decoder models G obtainable via training satisfy that for all x ∈ X , z 1 , z 2 ∈ Z:

| log p G (x|z 1 ) -log p G (x|z 2 )| ≤ L z 1 -z 2 .

Following prior analysis of language decoders (Mueller et al., 2017) , we assume that G is L-Lipschitz in its continuous input z (denote this set of possible decoders by G L ). When G is implemented as a RNN or Transformer language model, log p G (x|z) will remain Lipschitz in z if the recurrent or attention weight matrices have bounded norm. This property is naturally encouraged by popular training methods that utilize SGD with early stopping and L 2 regularization (Zhang et al., 2017 ). Note we have not assumed E or G is Lipschitz in x, which would be unreasonable since x stands for discrete text, and when a few symbols change, the decoder likelihood for the entire sequence can vary drastically (e.g., G may assign a much higher probability to a grammatical sentence than an ungrammatical one that only differs by one word). Our discussion is directed to the nature of such families of log-likelihood functions with a continuous variable z and a discrete variable x.

We further assume an effectively trained discriminator that succeeds in its adversarial task:

Assumption 3. D ensures that the latent encodings z 1 , • • • , z n of training examples x 1 , • • • , x n are indistinguishable from prior samples z ∼ p(z).

For simplicity, we directly assume that z 1 , • • • , z n are actual samples from p(z) which are given a priori. Here, the task of the encoder E is to map given unique training examples to the given latent points, and the goal of the decoder p G (•|•) is to maximize -L rec under the encoder mapping. The question now is which one-to-one mapping an optimal encoder/decoder will learn under the AAE and DAAE objective (Eq. 2 and Eq. 4). We start with the following observation: Theorem 1. For any one-to-one encoder mapping

E from {x 1 , • • • , x n } to {z 1 , • • • , z n }, the optimal value of objective max G∈G L 1 n n i=1 log p G (x i |E(x i ))

is the same. Intuitively, this result stems from the fact that the model receives no information about the structure of x, and x 1 , • • • , x n are simply provided as different symbols. Hence AAE offers no preference over x-z couplings, and a random matching in which the z do not reflect any data structure is equally good as any other matching (Figure 1 , Left). Latent point assignments start to differentiate, however, once we introduce local input perturbations.

To elucidate how perturbations affect latent space geometry, it helps to first consider a simple setting with only four examples x 1 , x 2 , x 3 , x 4 ∈ X . Again, we consider given latent points z 1 , z 2 , z 3 , z 4 sampled from p(z), and the encoder/decoder are tasked with learning which x to match with which z. As depicted in Figure 1 , suppose there are two pairs of x closer together and also two pairs of z closer together. Let σ denote the sigmoid function, we have the following conclusion: Theorem 2. Let d be a distance metric over X . Suppose x 1 , x 2 , x 3 , x 4 satisfy that with some > 0: d(x 1 , x 2 ) < , d(x 3 , x 4 ) < , and d(x i , x j ) > for all other (x i , x j ) pairs. In addition, z 1 , z 2 , z 3 , z 4 satisfy that with some 0 < δ < ζ: z 1z 2 < δ, z 3z 4 < δ, and z iz j > ζ for all other (z i , z j ) pairs. Suppose our perturbation process C reflects local X geometry with:

p C (x i |x j ) = 1/2 if d(x i , x j ) < and = 0 otherwise. For δ < 1 L (2 log (σ(Lζ)) + log 2) and ζ > 1 L log 1/( √ 2 -1) , the denoising objective max G∈G L 1 n n i=1 n j=1 p C (x j |x i ) log p G (x i |E(x j

)) (where n = 4) achieves the largest value when encoder E maps close pairs of x to close pairs of z.

This entails that DAAE will always prefer to map similar x to similar z. Note that Theorem 1 still applies here, and AAE will not prefer any particular x-z pairing over the other possibilities. We next generalize beyond the basic four-points scenario to consider n examples of x that are clustered. Here, we can ask whether this cluster organization will be reflected in the latent space of DAAE. Theorem 3. Suppose x 1 , • • • , x n are divided into n/K clusters of equal size K, with S i denoting the cluster index of x i . Let the perturbation process C be uniform within clusters, i.e. p C (x i |x j ) = 1/K if S i = S j and = 0 otherwise. For a one-to-one encoder mapping

E from {x 1 , • • • , x n } to {z 1 , • • • , z n }, the denoising objective max G∈G L 1 n n i=1 n j=1 p C (x j |x i ) log p G (x i |E(x j )) is upper bounded by: 1 n 2 i,j:Si =Sj log σ(L E(x i ) -E(x j ) ) -log K.

Theorem 3 provides an upper bound of the DAAE objective that can be achieved by a particular x-z mapping. This achievable limit is substantially better when examples in the same cluster are mapped to the latent space in a manner that is well-separated from encodings of other clusters. In other words, by preserving input space cluster structure in the latent space, DAAE can achieve better objective values and thus is incentivized to learn such encoder/decoder mappings. An analogous corollary can be shown for the case when examples x are perturbed to yield additional inputs x not present in the training data. In this case, the model would aim to map each example and its perturbations as a group to a compact group of z points well-separated from other groups in the latent space.

In conclusion, our analysis shows that a well-trained DAAE is guaranteed to learn neighborhoodpreserving latent representations, whereas even a perfectly-trained AAE model may learn latent representations whose geometry fails to reflect similarity in the x space. Empirical experiments in Section 5.2 confirm that our theory holds in practice.

5. EXPERIMENTS:

We evaluate our proposed model and other text autoencoders on two benchmark datasets: Yelp reviews and Yahoo answers (Shen et al., 2017; Yang et al., 2017) . Detailed descriptions of datasets, training settings, human evaluations, and additional results/examples can be found in the appendix.

Perturbation Process We randomly delete each word with probability p, so that perturbations of sentences with more words in common will have a larger overlap. We also tried replacing each word with a <mask> token or a random word and found that they all brought improvements, but deleting words worked best. We leave it to future work to explore more sophisticated text perturbations.

Baselines We compare our proposed DAAE with four alternative text autoencoders: adversarially regularized autoencoder (Zhao et al., 2018, ARAE) , β-VAE (Higgins et al., 2017) , AAE (Makhzani et al., 2015) , and latent-noising AAE (Rubenstein et al., 2018, LAAE) . Similar to our model, the LAAE uses Gaussian perturbations in the latent space to improve AAE's latent geometry (rather than perturbations in the sentence space). However, LAAE requires enforcing an L 1 penalty (λ 1 • log σ 2 (x) 1 ) on the latent perturbations' log-variance to prevent them from vanishing. In contrast, input perturbations in DAAE enable stochastic latent representations without parametric restrictions like Gaussianity.

5.1. GENERATION-RECONSTRUCTION TRADE-OFF:

We evaluate various latent variable generative models in terms of both generation quality and reconstruction accuracy. A strong model should not only generate high quality sentences, but also learn useful latent variables that capture significant data content. Recent work on text autoencoders has found an inherent tension between these aims (Bowman et al., 2016; Cífka et al., 2018 ), yet only when both goals are met can we successfully manipulate sentences by modifying their latent representation (in order to produce valid output sentences that retain the semantics of the input).

We compute the BLEU score (Papineni et al., 2002) between input and reconstructed sentences to measure reconstruction accuracy, and compute Forward/Reverse PPL to measure sentence generation quality (Zhao et al., 2018; Cífka et al., 2018) . 4 Forward PPL is the perplexity of a language model trained on real data and evaluated on generated data. It measures the fluency of the generated text, but cannot detect the collapsed case where the model repeatedly generates a few common sentences. Reverse PPL is the perplexity of a language model trained on generated data and evaluated on real data. It takes into account both the fluency and diversity of the generated text. If a model generates only a few common sentences, a language model trained on it will exhibit poor PPL on real data.

We thoroughly investigate the performance of different models and their trade-off between generation and reconstruction. Figure 2 plots reconstruction BLEU (higher is better) vs. Forward/Reverse PPL (lower is better). The lower right corner indicates an ideal situation where good reconstruction accuracy and generation quality are both achieved. For models with tunable hyperparameters, we sweep the full spectrum of their generation-reconstruction trade-off by varying the KL coefficient β of β-VAE, the log-variance L 1 penalty λ 1 of LAAE, and the word drop probability p of DAAE. = 0.01 < l a t e x i t s h a 1 _ b a s e 6 4 = " B V 6 5 h 

V K y h 8 A E N i Y d P Q / U 2 4 E A T C s = " > A A A B 8 X i c b V B N S w M x E J 2 t X 7 V + V T 1 6 C R b B U 9 m t g l 6 E g h e P F e w H t k v J p r N t a D a 7 J F m h L P 0 X X j w o 4 t V / 4 8 1 / Y 9 r u Q V s f h D z e m 2 F m X p A I r o 3 r f j u F t f W N z a 3 i d m l n d 2 / / o H x 4 1 N J x q h g 2 W S x i 1 Q m o R s E l N g 0 3 A j u J Q h o F A t v B + H b m t 5 9 Q a R 7 L B z N J 0 I / o U P K Q M 2 q s 9 N g L 0 N A b t + p 6 / X L F f n O Q V e L l p A I 5 G v 3 y V 2 8 Q s z R C a Z i g W n c 9 N z F + R p X h T O C 0 1 E s 1 J p S N 6 R C 7 l k o a o f a z + c Z T c m a V A Q l j Z Z 8 0 Z K 7 + 7 s h o p P U k C m x l R M 1 I L 3 s z 8 T + v m 5 r w 2 s + 4 T F K D k i 0 G h a k g J i a z 8 8 m A K 2 R G T C y h T H G 7 K 2 E j q i g z N q S S D c F b P n m V t G p V 7 6 J a u 7 + s 1 G t 5 H E U 4 g V M 4 B w + u o A 5 3 0 I A m M J D w D K / w 5 m j n x X l 3 P h a l B S f v O Y Y / c D 5 / A A M J j 8 Y = < / l a t e x i t > = 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " L Y F E e f Z 6 f S F s C a D j O r s C + R M D 4 F E = " > A A A B 7 n i c b V D L S g N B E O z 1 G e M r 6 t H L Y B A 8 h d 0 o 6 E U I e P E Y w T w g W c L s p J M M m X 0 w 0 y u E J R / h x Y M i X v 0 e b / 6 N k 2 Q P m l j Q U F R 1 0 9 0 V J E o a c t 1 v Z 2 1 9 Y 3 N r u 7 B T 3 N 3 b P z g s H R 0 3 T Z x q g Q 0 R q 1 i 3 A 2 5 Q y Q g b J E l h O 9 H I w 0 B h K x j f z f z W E 2 o j 4 + i R J g n 6 I R 9 G c i A F J y u 1 u g E S v / V 6 p b J b c e d g q 8 T L S R l y 1 H u l r 2 4 / F m m I E Q n F j e l 4 b k J + x j V J o X B a 7 K Y G E y 7 G f I g d S y M e o v G z + b l T d m 6 V P h v E 2 l Z E b K 7 + n s h 4 a M w k D G x n y G l k l r 2 Z + J / X S W l w 4 2 c y S l L C S C w W D V L F K G a z 3 1 l f a h S k J p Z w o a W 9 l Y k R 1 1 y Q T a h o Q / C W X 1 4 l z W r F u 6 x U H 6 7 K t W o e R w F O 4 Q w u w I N r q M E 9 1 K E B A s b w D K / w 5 i T O i / P u f C x a 1 5 x 8 5 g T + w P n 8 A b U i j x o = < / l a t e x i t > p = 0.1 < l a t e x i t s h a 1 _ b a s e 6 4 = " G H s A G a l S K w 3 X q D 6 s S H k B D + X W D 1 k = " > A A A B 7 H i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x S J 4 C k k V 9 C I U v H i s Y N p C G 8 p m O 2 m X b j Z h d y O U 0 t / g x Y M i X v 1 B 3 v w 3 b t s c t P X B w O O 9 G W b m R Z n g 2 n j e t 7 O 2 v r G 5 t V 3 a K e / u 7 R 8 c V o 6 O m z r N F c O A p S J V 7 Y h q F F x i Y L g R 2 M 4 U 0 i Q S 2 I p G d z O / 9 Y R K 8 1 Q + m n G G Y U I H k s e c U W O l I L v 1 X L 9 X q X q u N w d Z J X 5 B q l C g 0 a t 8 d f s p y x O U h g m q d c f 3 M h N O q D K c C Z y W u 7 n G j L I R H W D H U k k T 1 O F k f u y U n F u l T + J U 2 Z K G z N X f E x O a a D 1 O I t u Z U D P U y 9 5 M / M / r 5 C a + C S d c Z r l B y R a L 4 l w Q k 5 L Z 5 6 T P F T I j x p Z Q p r i 9 l b A h V Z Q Z m 0 / Z h u A v v 7 x K m j X X v 3 R r D 1 f V e q 2 I o w S n c A Y X 4 M M 1 1 O E e G h A A A w 7 P 8 A p v j n R e n H f n Y 9 G 6 5 h Q z J / A H z u c P o o a N 3 A = = < / l a t e x i t > p = 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " h a 4 Q S 8 D D + 9 4 q q D r z e 0 U W o 2 C k k 0 o = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 m q o B e h 4 M V j R f s B b S i b 7 a R d u t m E 3 Y 1 Q Q n + C F w + K e P U X e f P f u G 1 z 0 N Y H A 4 / 3 Z p i Z F y S C a + O 6 3 0 5 h b X 1 j c 6 u 4 X d r Z 3 d s / K B 8 e t X S c K o Z N F o t Y d Q K q U X C J T c O N w E 6 i k E a B w H Y w v p 3 5 7 S d U m s f y 0 U w S 9 C M 6 l D z k j B o r P S Q 3 X r 9 c c a v u H G S V e D m p Q I 5 G v / z V G 8 Q s j V A a J q j W X c 9 N j J 9 R Z T g T O C 3 1 U o 0 J Z W M 6 x K 6 l k k a o / W x + 6 p S c W W V A w l j Z k o b M 1 d 8 T G Y 2 0 n k S B 7 Y y o G e l l b y b + 5 3 V T E 1 7 7 G Z d J a l C y x a I w F c T E Z P Y 3 G X C F z I i J J Z Q p b m 8 l b E Q V Z c a m U 7 I h e M s v r 5 J W r e p d V G v 3 l 5 V 6 L Y + j C C d w C u f g w R X U 4 Q 4 a 0 A Q G Q 3 i G V 3 h z h P P i v D s f i

= " > A A A B 7 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 h a Q S 9 C w Y v H C q Y t t K F s t p t 2 6 e 4 m 7 G 6 E E v o b v H h Q x K s / y J v / x m 2 b g 7 Y + G H i 8 N 8 P M v C j l T B v P + 3 Z K G 5 t b 2 z v l 3 c r e / s H h U f X 4 p K 2 T T B E a k I Q n q h t h T T m T N D D M c N p N F c U i 4 r Q T T e 7 m f u e J K s 0 S + W i m K Q 0 F H k k W M 4 K N l Y L 0 1 n M b g 2 r N c 7 0 F 0 D r x C 1 K D A q 1 B 9 a s / T E g m q D S E Y 6 1 7 v p e a M M f K M M L p r N L P N E 0 x m e A R 7 V k q s a A 6 z B f H z t C F V Y Y o T p Q t a d B C / T 2 R Y 6 H 1 V E S 2 U 2 A z 1 q v e X

B V 6 5 h V K y h 8 A E N i Y d P Q / U 2 4 E A T C s = " > A A A B 8 X i c b V B N S w M x E J 2 t X 7 V + V T 1 6 C R b B U 9 m t g l 6 E g h e P F e w H t k v J p r N t a D a 7 J F m h L P 0 X X j w o 4 t V / 4 8 1 / Y 9 r u Q V s f h D z e m 2 F m X p A I r o 3 r f j u F t f W N z a 3 i d m l n d 2 / / o H x 4 1 N J x q h g 2 W S x i 1 Q m o R s E l N g 0 3 A j u J Q h o F A t v B + H b m t 5 9 Q a R 7 L B z N J 0 I / o U P K Q M 2 q s 9 N g L 0 N A b t + p 6 / X L F f n O Q V e L l p A I 5 G v 3 y V 2 8 Q s z R C a Z i g W n c 9 N z F + R p X h T O C 0 1 E s 1 J p S N 6 R C 7 l k o a o f a z + c Z T c m a V A Q l j Z Z 8 0 Z K 7 + 7 s h o p P U k C m x l R M 1 I L 3 s z 8 T + v m 5 r w 2 s + 4 T F K D k i 0 G h a k g J i a z 8 8 m A K 2 R G T C y h T H G 7 K 2 E j q i g z N q S S D c F b P n m V t G p V

z v 2 1 l Z X V v f 2 C x s F b d 3 d v f 2 S w e H D Z 1 k i m G d J S J R r Z B q F F x i 3 X A j s J U q p H E o s B k O b 6 d + 8 w m V 5 o l 8 M K M U g 5 j 2 J Y 8 4 o 8 Z K j 5 0 Q D b 3 x 3 M t u q e y 5 3 g x k m f g 5 K U O O W r f 0 1 e k l L I t R G i a o 1 m 3 f S 0 0 w p s p w J n B S 7 G Q a U 8 q G t I 9 t S y W N U Q f j 2 c E T c m q V H o k S Z U s a M l N / T 4 x p r P U o D m 1 n T M 1 A L 3 p T 8 T + v n Z n o O h h z m W Y G J Z s v i j J B T E K m 3 5 M e V 8 i M G F l C m e L 2 V s I G V F F m b E Z F G 4 K / + P I y a V

R c / 9 y t 3 F + U q 5 U 8 j g I c w w m c g Q 9 X U I U 7 q E E d G M T w D K / w 5 i j n x X l 3 P u a t K 0 4 + c w R / 4 H z + A J i T j 5 A = < / l a t e x i t > p = 0.1 

< l a t e x i t s h a _ b a s e = " G H s A G a l S K w X q D s S H k B D + X W D k = " > A A A B H i c b V B N S N A E J W e t X a O X x S J C k k V C I U v H i s Y N p C G p m O m X b j Z h d y O U t / g x Y M i X v B v w b t s c t P X B w O O G W b m R Z n g n j e t O v r G t V a K e / u R c V o O m z r N F c O A p S J V Y h q F F x i Y L g R M U i Q S I p G d z O / Y R K Q + m n G G Y U I H k s e c U W O l I L v X L X q X q u N w d Z J X B q l C g a t d f s p y x O U h g m q d c f M h N O q D K c C Z y W u n G j L I R H W D H U k k T O F k f u y U n F u l T + J U Z K G z N X f E x O a a D O I t u Z U D P U y M / M / r C a + C S d c Z r l B y R a L l w Q k L Z T P F T I j x p Z Q p r i l b A h V Z Q Z m / Z h u A v v x K m j X X v R r D f V e q I o w S n c A Y X M M O E e G h A A A w P A p v j n R e n H f n Y G h Q z J / A

s V q E K r 4 7 W m F 4 b y d Q J D C K S 9 o = " > A A A B 8 X i c b V B N S 8 N A E N 3 U r 1 q / q h 6 9 B I v g K S R V 0 Y t Q 9 O K x g v 3 A N p T N d t I u 3 W z C 7 k Q o o f / C i w d F v P p v v P l v 3 L Y 5 a O u D g c d 7 M 8 z M C x L B N b r u t 1 V Y W V 1 b 3 y h u l r a 2 d 3 b 3 y v s H T R 2 n i k G D x S J W 7 Y B q E F x C A z k K a C c K a B Q I a A W j 2 6 n f e g K l e S w f c J y A H 9 G B 5 C F n F I 3 0 2 A 0 A 6 b X r e B e 9 c s V 1 3 B n s Z e L l p E J y 1 H v l r 2 4 / Z m k E E p m g W n c 8 N 0 E / o w o 5 E z A p d V M N C W U j O o C O o Z J G o P 1 s d v H E P j F K 3 w 5 j Z U q i P V N / T 2 Q 0 0 n o c B a Y z o j j U i 9 5 U / M / r p B h e + R m X S Y o g 2 X x R m A o b Y 3 v 6 v t 3 n C h i K s S G U K W 5 u t d m Q K s r Q h F Q y I X i L L y + T Z t X x z p z q / X

= " > A A A B 9 X i c b V D L S g M x F L 1 T X 7 W + q i 7 d B I v g q s x U Q T d C 0 Y 3 L C v Y B 7 V g y m U w b m k m G J K O U o f / h x o U i b v 0 X d / 6 N a T s L b T 0 Q c j j n X H J z g o Q z b V z 3 2 y m s r K 6 t b x Q 3 S 1 v b

O 7 t 7 5 f 2 D l p a p I r R J J J e q E 2 B N O R O 0 a Z j h t J M o i u O A 0 3 Y w u p n 6 7 U e q N J P i 3 o w T 6 s d 4 I F j E C D Z W e u h x G w 1 x 3 7 t y q 6 7 X L 1 f s N Q N a J l 5 O K p C j 0 S 9 / 9 U J J 0 p g K Q z j W u u u 5 i f E z r A w j n 6 1 U 7 8 7 

E 5 K v V T T B J M R H t C u p Q L H V P v Z b O s J O r F K i C K p 7 B E G z d T f E x m O t R 7 H g U 3 G 2 A z 1 o j c V / / O 6 q Y k u / Y y J J D V U k P l D U c q R k W h a A Q q Z o s T w s S W Y K G Z 3 R W S I F S b G F l W y J X i L X 1 4 m r V r V O 6 v W

X V l F q V G 7 A x V j 1 i P P G + F j b b I T p F Y = " > A A A B 9 H i c b V D L S s N A F L 2 p r 1 p f V Z d u B o v g K i R V 0 I 1 Q d O O y g n 1 A G 8 p k M m m H T i Z x Z l I o o d / h x o U i b v 0 Y d / 6 N 0 z Q L b T 0 w c

4 P q j X H d n K g V e I W p A Y F m o P q V z + I S R p R o Q n H S v V c J 9 F e h q V m h N N Z p Z 8 q m m A y x k P a M 1 T g i C o v y 4 + e o T O j B C i M p X l C o 1 z 9 P Z H h S K l p 5 J t k h P V I L X t z 8 T + v l + r w 2 s u Y S F J N B V k s C l O O d I z m D a C A S U o 0 n x q C i W T m V k R G W G K i T U 8 V U 4 K 7 / O V V 0 q 7 b 7 o V d f 7 i s N W 6 L O s p w A q d w D i 5 c Q Q P u o Q k t I P A E z / A K b 9 b E e r H e r Y 9 F t G Q V M 8 f w B 9 b n D 0 F E k R o = < /

W q C G 0 Q y a V q B 1 h T z g R t G G Y 4 b S e K 4 j j g t B U M b y Z + 6 5 E q z a S 4 N 6 O E + j H u C x Y x g o 2 V H r r c R k P c 8 6 7 c i n v e K 5 X t N Q V a J F 5 O y p C j 3 i t 9 d U N J 0 p g K Q z j W u u O 5 i f E z r A w j n I 6 L 3 V T T B J M h 7 t O O p Q L H V P v Z d O s x O r Z K i C K p 7 B E G T d X f E x m O t R 7 F g U 3 G 2 A z 0 v D c R / / M 6 q Y k u / Y y J J D V U k N l D U c q R k W h S A Q q Z o s T w k S W Y K G Z 3 R W S A F S b G F l W 0 J X j z X 1 4 k z W r F O

< l a t e x i t s h a 1 _ b a s e 6 4 = " p A O y m g h E B T Z k + Q 9 t 1 h + 6 z q W x t I s = " >:

A A A B 7 H i c b V B N S 8 N A E J 3 1 s 9 a v q k c v i 0 X w F J I q 6 k U o e v F Y w b S F N p T N d t M u 3 W z C 7 k Y o o b / B i w d F v P q D v P l v 3 L Y 5 a O u D g c d 7 M 8 z M C 1 P B t X H d b 7 S y u r a + s V n a K m / v 7 O 7 t V w 4 O m z r J F G U + T U S i 2 i H R T H D J f M O N Y O 1 U M R K H g r X C 0 d 3 U b z 0 x p X k i H 8 0 4 Z U F M B p J H n B J j J T + 9 c Z 3 L X q X q O u 4 M e J l 4 B a l C g U a v 8 t X t J z S L m T R U E K 0 7 n p u a I C f K c C r Y p N z N N E s J H Z E B 6 1 g q S c x 0 k M + O n e B T q / R x l C h b 0 u C Z + n s i J 7 H W 4 z i 0 n T E x Q 7 3 o T c X / v E 5 m o u s g 5 z L N D J N 0 v i j K B D Y J n n 6 O + 1 w x a s T Y E k I V t 7 d i O i S K U G P z K d s Q v M W X

= " > A A A B 9 X i c b V D L S g M x F L 1 T X 7 W + q i 7 d B I v g q s x U Q T d C 0 Y 3 L C v Y B 7 V g y m U w b m k m G J K O U o f / h x o U i b v 0 X d / 6 N a T s L b T 0 Q c j j n X H J z g o Q z b V z 3 2 y m s r K 6 t b x Q 3 S 1 v b O 7 t 7 5 f 2 D l p a p I r R J J J e q E 2 B N O R O 0 a Z j h t J M o i u O A

0 3 Y w u p n 6 7 U e q N J P i 3 o w T 6 s d 4 I F j E C D Z W e u h x G w 1 x 3 7 t y q 6 7 X L 1 f s N Q N a J l 5 O K p C j 0 S 9 / 9 U J J 0 In the left panel, we observe that a standard VAE (β = 1) completely collapses and ignores the latent variable z, resulting in reconstruction BLEU close to 0. At the other extreme, AAE can achieve near-perfect reconstruction, but its latent space is highly non-smooth and generated sentences are of poor quality, indicated by its large Forward PPL. Decreasing β in VAE or introducing latent noises in AAE provides the model with a similar trade-off curve between reconstruction and generation. We note that ARAE falls on or above their curves, revealing that it does not fare better than these methods (Cífka et al. (2018) also reported similar findings). Our proposed DAAE provides a trade-off curve that is strictly superior to other models. With discrete x and a complex encoder, the Gaussian perturbations added to the latent space by β-VAE and LAAE are not directly related to how the inputs are encoded. In contrast, input perturbations added by DAAE can constrain the encoder to maintain coherence between neighboring inputs in an end-to-end fashion and help learn smoother latent space.

p g K Q z j W u u u 5 i f E z r A w j n E 5 K v V T T B J M R H t C u p Q L H V P v Z b O s J O r F K i C K p 7 B E G z d T f E x m O t R 7 H g U 3 G 2 A z 1 o j c V / / O 6 q Y k u / Y y J J D V U k P l D U c q R k W h a A Q q Z o s T w s S W Y K G Z 3 R W S I F S b G F l W y J X i L X 1 4 m r V r V O 6 v W 7 s 4

r 9 e u 8 j i I c w T G c g g c X U I d b a E A T C C h 4 h l d 4 c 5 6 c F + f d + Z h H C 0 4 + c w h / 4 H z +:

A L G U k V M = < / l a t e x i t > 1 = 0.2 < l a t e x i t s h a 1 _ b a s e 6 4 = " X V l F q V G 7 A x V j 1 i P P G + F j b b I T p F Y = " > A A A B 9 H i c b V D L S s N A F L 2 p r 1 p f V Z d u B o v g K i R V 0 I 1 Q d O O y g n 1 A G 8 p k M m m H T i Z x Z l I o o d / h x o U i b v 0 Y d / 6 N 0 z Q L b T 0 w c

4 P q j X H d n K g V e I W p A Y F m o P q V z + I S R p R o Q n H S v V c J 9 F e h q V m h N N Z p Z 8 q m m A y x k P a M 1 T g i C o v y 4 + e o T O j B C i M p X l C o 1 z 9 P Z H h S K l p 5 J t k h P V I L X t z 8 T + v l + r w 2 s u Y S F J N B V k s C l O O d I z m D a C A S U o 0 n x q C i W T m V k R G W G K i T U 8 V U 4 K 7 / O V V 0 q 7 b 7 o V d f 7 i s N W 6 L O s p w A q d w D i 5 c Q Q P u o Q k t I P A E z / A K b 9 b E e r H e r Y 9 F t G Q V M 8 f w B 9 b n D 0 F E k R o = < / l a t e x i t >

The right panel in Figure 2 illustrates that Reverse PPL first drops and then rises as we increase the degree of regularization/perturbation. This is because when z encodes little information, generations from prior-sampled z lack enough diversity to cover the real data. Again, DAAE outperforms the other models which tend to have higher Reverse PPL and lower reconstruction BLEU. In subsequent experiments, we set β = 0.15 for β-VAE, λ 1 = 0.05 for LAAE, and p = 0.3 for DAAE, to ensure they have strong reconstruction abilities and encode enough information to enable text manipulations.

5.2. NEIGHBORHOOD PRESERVATION:

Figure 3 : Recall rate of 10 nearest neighbors in the sentence space retrieved by k nearest neighbors in the latent space on Yelp. ARAE is not plotted here as we find its recall significantly below other models (< 1%).

In this section, we empirically investigate whether our previous theory holds in practice. That is, in actual autoencoder models trained on real text datasets, do sentence perturbations induce latent space organization that better preserves neighborhood structure in the data space? Under our word-drop perturbation process, sentences with more words in common are more likely to be perturbed into one another. This choice of C approximately encodes sentence similarity via the normalized edit distance5 . Within the test set, we find both the 10 nearest neighbors of each sentence based on the normalized edit distance (denote this set by NN x ), as well as the k nearest neighbors based on Euclidean distance between latent representations (denote this set by NN z ). We

AAE DAAE:

Source my waitress katie was fantastic , attentive and personable . my waitress katie was fantastic , attentive and personable . my cashier did not smile , barely said hello .

the manager , linda , was very very attentive and personable . the service is fantastic , the food is great .

stylist brenda was very friendly , attentive and professional . the employees are extremely nice and helpful .

the manager was also super nice and personable . our server kaitlyn was also very attentive and pleasant .

my server alicia was so sweet and attentive . the crab po boy was also bland and forgettable . our waitress ms. taylor was amazing and very knowledgeable .

Source i have been known to eat two meals a day here . i have been known to eat two meals a day here . i have eaten here for _num_ years and never had a bad meal ever . you can seriously eat one meal a day here . i love this joint .

i was really pleased with our experience here . i have no desire to ever have it again .

ive been coming here for years and always have a good experience . you do n't need to have every possible dish on the menu .

i have gone to this place for happy hour for years . i love this arena .

we had _num_ ayce dinner buffets for _num_ on a tuesday night . Figures 3 shows that DAAE consistently gives the highest recall, about 1.5∼2 times that of AAE, implying that input perturbations have a substantial effect on shaping the latent space geometry. Tables 1 presents the five nearest neighbors found by AAE and DAAE in their latent space for example test set sentences. The AAE sometimes encodes entirely unrelated sentences close together, while the latent space geometry of the DAAE is structured based on key words such as "attentive" and "personable", and tends to group sentences with similar semantics close together.

5.3. APPLICATIONS TO CONTROLLABLE TEXT GENERATION:

5.3.1 STYLE TRANSFER VIA VECTOR ARITHMETIC Mikolov et al. (2013) previously discovered that word embeddings from unsupervised learning can capture linguistic relationships via simple arithmetic. A canonical example is the embedding arithmetic "King" -"Man" + "Woman" ≈ "Queen". Here, we use the Yelp dataset with tense and sentiment as two example attributes (Hu et al., 2017; Shen et al., 2017) to investigate whether analogous structure emerges in the latent space of our sentence-level models.

Tense We use the Stanford Parser to extract the main verb of a sentence and determine the sentence tense based on its part-of-speech tag. We compute a single "tense vector" by averaging the latent code z separately for 100 past tense sentences and 100 present tense sentences in the dev set, and then calculating the difference between the two. Given a sentence from the test set, we attempt to change its tense from past to present or from present to past through simple addition/subtraction of the tense vector. More precisely, a source sentence x is first is encoded to z = E(x), and then the tense-modified sentence is produced via G(z ± v), where v ∈ R d denotes the fixed tense vector.

Input i enjoy hanging out in their hookah lounge . had they informed me of the charge i would n't have waited . ARAE i enjoy hanging out in their 25th lounge . amazing egg of the may i actually ! β-VAE i made up out in the backyard springs salad .

had they help me of the charge i would n't have waited . AAE i enjoy hanging out in their brooklyn lounge . have they informed me of the charge i would n't have waited . LAAE i enjoy hanging out in the customized and play . they are girl ( the number so i would n't be forever . DAAE i enjoyed hanging out in their hookah lounge .

they have informed me of the charge i have n't waited .

Table 4 : Examples of vector arithmetic for tense inversion.

Input the food is entirely tasteless and slimy . the food is entirely tasteless and slimy . +v the food is entirely tasteless and slimy . the food is tremendous and fresh . +1.5v the food is entirely tasteless and slimy .

the food is sensational and fresh . +2v the food is entirely and beef . the food is gigantic .

Input i really love the authentic food and will come back again . i really love the authentic food and will come back again . -v i really love the authentic food and will come back again . i really love the authentic food and will never come back again . -1.5v i really but the authentic food and will come back again .

i really do not like the food and will never come back again .

-2v:

i really but the worst food but will never come back again . i really did not believe the pretentious service and will never go back . Tables 4 and J .2 show the results of adding or subtracting this fixed latent vector offset under different models. DAAE can successfully change "enjoy" to "enjoyed", or change the subjunctive mood to declarative mood and adjust the word order. Other baselines either fail to alter the tense, or undesirably change the semantic meaning of the source sentence (e.g. "enjoy" to "made").

Sentiment Following the same procedure used to alter tense, we compute a "sentiment vector" v from 100 negative and positive sentences and use it to change the sentiment of test sentences. Table 3 reports the automatic evaluations, and Tables 5 and J .3 show examples generated by AAE and DAAE. Scaling ±v to ±1.5v and ±2v, we find that the resulting sentences get more and more positive/negative. However, the PPL for AAE increases rapidly with this scaling factor, indicating that the sentences become unnatural when their encodings have a large offset. DAAE enjoys a much smoother latent space than AAE. Despite the fact that no sentiment labels are provided during training (a more challenging task than previous style transfer models (Shen et al., 2017) ), DAAE with ±1.5v is able to transfer sentiment fairly well.

5.3.2. SENTENCE INTERPOLATION VIA LATENT SPACE TRAVERSAL:

We also study sentence interpolation by traversing the latent space of text autoencoders. Given two input sentences, we encode them to z 1 , z 2 and decode from tz 1 + (1t)z 2 (0 ≤ t ≤ 1) to obtain interpolated sentences. Ideally this should produce fluent sentences with gradual semantic change (Bowman et al., 2016) . Table 6 shows two examples from the Yelp dataset, where it is clear that DAAE produces more coherent and natural interpolations than AAE. Table J .4 in the appendix shows two difficult examples from the Yahoo dataset, where we interpolate between dissimilar sentences. While it is challenging to generate semantically correct sentences in these cases, the latent space of our model exhibits continuity on topic and syntactic structure.

6. CONCLUSION:

This paper proposed DAAE for generative text modeling. As revealed in previous work (Devlin et al., 2018; Lample et al., 2018) , we find that denoising techniques can greatly improve the learned text representations. We provide a theoretical explanation for this phenomenon by analyzing the latent Input 1 it 's so much better than the other chinese food places in this area . fried dumplings are a must . Input 2 better than other places . the fried dumplings are a must if you ever visit this place . AAE it 's so much better than the other chinese food places in this area . fried dumplings are a must . it 's so much better than the other food places in this area .

fried dumplings are a must . better , much better .

the dumplings are a must if you worst . better than other places .

the fried dumplings are a must if you ever this place . better than other places .

the fried dumplings are a must if you ever visit this place . DAAE it 's so much better than the other chinese food places in this area . fried dumplings are a must . it 's much better than the other chinese places in this area .

fried dumplings are a must visit . better than the other chinese places in this area .

fried dumplings are a must in this place . better than the other places in charlotte .

the fried dumplings are a must we ever visit this . better than other places .

the fried dumplings are a must if we ever visit this place . space geometry arisen from input perturbations. Our proposed model substantially outperforms other text autoencoders, and demonstrates potential for various text manipulations via vector operations. Future work may investigate superior perturbation strategies and additional properties of latent space geometry to provide finer control over the text generated using autoencoder models.

A WASSERSTEIN DISTANCE:

The AAE objective can be connected to a relaxed form of the Wasserstein distance between model and data distributions (Tolstikhin et al., 2017) . Specifically, for cost function c(•, •) : X × X → R and deterministic decoder mapping G : Z → X , it holds that:

EQUATION

where the minimization over couplings Γ with marginals p data and p G can be replaced with minimization over conditional distributions q(z|x) whose marginal q(z) = E pdata(x) [q(z|x)] matches the latent prior distribution p(z). Relaxing this marginal constraint via a divergence penalty D(q(z) p(z)) estimated by adversarial training, one recovers the AAE objective (Eq. 1). In particular, AAE on discrete x with the cross-entropy loss is minimizing an upper bound of the total variation distance between p data and p G , with c chosen as the indicator cost function (Zhao et al., 2018) .

Our model is optimizing over conditional distributions q(z|x) of the form (6), a subset of all possible conditional distributions. Thus, after introducing input perturbations, our method is still minimizing an upper bound of the Wasserstein distance between p data and p G described in (7).

B PROOF OF THEOREM 1:

Theorem 1. For any one-to-one encoder mapping

E from {x 1 , • • • , x n } to {z 1 , • • • , z n }, the optimal value of objective max G∈G L 1 n n i=1 log p G (x i |E(x i )) is the same.

Proof. Consider two encoder matchings x i to z α(i) and x i to z β(i) , where both α and β are permutations of the indices {1, . . . , n}. Suppose G α is the optimal decoder model for the first matching (with permutations α). This implies

p Gα = arg max G∈G L n i=1 log p G (x i |z α(i) )

Now let p G β (x i |z j ) = p Gα (x βα -1 (i) |z j ), ∀i, j. Then G β can achieve exactly the same log-likelihood objective value for matching β as G α for matching α, while still respecting the Lipschitz constraint.

C PROOF OF THEOREM 2:

Theorem 2. Let d be a distance metric over X . Suppose x 1 , x 2 , x 3 , x 4 satisfy that with some > 0: d(x 1 , x 2 ) < , d(x 3 , x 4 ) < , and d(x i , x j ) > for all other (x i , x j ) pairs. In addition, z 1 , z 2 , z 3 , z 4 satisfy that with some 0 < δ < ζ: z 1z 2 < δ, z 3z 4 < δ, and z iz j > ζ for all other (z i , z j ) pairs. Suppose our perturbation process C reflects local X geometry with:

p C (x i |x j ) = 1/2 if d(x i , x j ) < and = 0 otherwise. For δ < 1 L (2 log (σ(Lζ)) + log 2) and ζ > 1 L log 1/( √ 2 -1) , the denoising objective max G∈G L 1 n n i=1 n j=1 p C (x j |x i ) log p G (x i |E(x j

)) (where n = 4) achieves the largest value when encoder E maps close pairs of x to close pairs of z.

Proof. Let [n] denote {1, . . . , n}, and assume without loss of generality that the encoder E maps each x i to z i . We also define A = {1, 2}, B = {3, 4} as the two x-pairs that lie close together. For our choice of C(x), the training objective to be maximized is:

EQUATION

The remainder of our proof is split into two cases:

Case 1. ||z j -z || > ζ for j ∈ A, ∈ B Case 2. ||z j -z || < δ for j ∈ A, ∈ B

Under Case 1, x points that lie far apart also have z encodings that remain far apart. Under Case 2, x points that lie far apart have z encodings that lie close together. We complete the proof by showing that the achievable objective value in Case 2 is strictly worse than in Case 1, and thus an optimal encoder/decoder pair would avoid the x, z matching that leads to Case 2.

In Case 1 where ||z jz || > ζ for all j ∈ A, ∈ B, we can lower bound the training objective (8) by choosing: 

EQUATION

| log p G (x i |z j ) -log p G (x i |z )| = 0 if j, ∈ A or j, ∈ B ≤ log ((1 -γ)/γ) otherwise and thus remains ≤ L||z j -z || when γ = σ(-Lζ) ≥ σ(-L||z j -z ||) = 1/[1 + exp(L||z j -z ||)].

Plugging the p G (x|z) assignment from ( 9) into (8), we see that an optimal decoder can obtain training objective value ≥ 8 log [σ(Lζ)/2] in Case 1 where ||z

j -z || > ζ, ∀j ∈ A, ∈ B.

Next, we consider the alternative case where ||z jz || < δ for j ∈ A, ∈ B.

For i, j ∈ A and for all ∈ B, we have:

log p G (x i |z j ) ≤ log p G (x i |z ) + L||z j -z || by Assumption 2 ≤ log p G (x i |z ) + Lδ ≤ Lδ + log 1 - k∈B p G (x k |z ) since k p G (x k |z ) ≤ 1

Continuing from (8), the overall training objective in this case is thus:

i,j∈A log p G (x i |z j ) + k, ∈B log p G (x k |z ) ≤ 4Lδ + i,j∈A min ∈B log 1 - k∈B p G (x k |z ) + k, ∈B log p G (x k |z ) ≤ 4Lδ + ∈B 2 log 1 - k∈B p G (x k |z ) + k∈B log p G (x k |z ) ≤ 4Lδ -12 log 2

using the fact that the optimal decoder for the bound in this case is:

p G (x k |z ) = 1/4 for all k, ∈ B.

Finally, plugging our range for δ stated in the Theorem 2, it shows that the best achievable objective value in Case 2 is strictly worse than the objective value achievable in Case 1. Thus, the optimal encoder/decoder pair under the AAE with perturbed x will always prefer the matching between {x 1 , . . . , x 4 } and {z 1 , . . . , z 4 } that ensures nearby x i are encoded to nearby z i (corresponding to Case 1).

D PROOF OF THEOREM 3:

Theorem 3. Suppose x 1 , • • • , x n are divided into n/K clusters of equal size K, with S i denoting the cluster index of x i . Let the perturbation process C be uniform within clusters, i.e. p C (x i |x j ) = 1/K if S i = S j and = 0 otherwise. For a one-to-one encoder mapping

E from {x 1 , • • • , x n } to {z 1 , • • • , z n }, the denoising objective max G∈G L 1 n n i=1 n j=1 p C (x j |x i ) log p G (x i |E(x j )

) is upper bounded by: 1 n 2 i,j:Si =Sj log σ(L E(x i ) -E(x j ) )log K.

Proof. Without loss of generality, let E(x i ) = z i for notational convenience. We consider what is the optimal decoder probability assignment p G (x i |z j ) under the Lipschitz constraint 2.

The objective of the AAE with perturbed x is to maximize:

1 n i j p C (x j |x i ) log p G (x i |E(x j )) = 1 nK j i:Si=Sj log p G (x i |z j )

We first show that the optimal p G (•|•) will satisfy that the same probability is assigned within a cluster, i.e. p(x i |z j ) = p(x k |z j ) for all i, k s.t. S i = S k . If not, let P sj = i:Si=s p G (x i |z j ), and we reassign p G (x i |z j ) = P Sij /K. Then G still conforms to the Lipschitz constraint if G meets it, and G will have a larger target value than G.

Now let us define P j = i:Si=Sj p G (x i |z j ) = K • p G (x j |z j ) (0 ≤ P j ≤ 1). The objective becomes: (log P i + log P j )log K Consider each term max p G (log P i + log P j ): when S i = S j , this term can achieve the maximum value 0 by assigning P i = P j = 1; when S i = S j , the Lipschitz constraint ensures that: log(1 -P i ) ≥ log P j -L z iz j log(1 -P j ) ≥ log P i -L z iz j Therefore:

log P i + log P j ≤ 2 log σ(L z iz j )

Overall, we thus have:

max p G 1 nK j i:Si=Sj log p G (x i |z j ) ≤ 1 n 2 i,j:Si =Sj log σ(L z i -z j ) -log K E DATASETS

The Yelp dataset is from Shen et al. (2017) , which has 444K/63K/127K sentences of less than 16 words in length as train/dev/test sets, with a vocabulary of 10K. It was originally divided into positive and negative sentences for style transfer between them. Here we discard the sentiment label and let the model learn from all sentences indiscriminately. Our second dataset of Yahoo answers is from Yang et al. (2017) . It was originally document-level. We perform sentence segmentation and keep sentences with length from 2 to 50 words. The resulting dataset has 495K/49K/50K sentences for train/dev/test sets, with vocabulary size 20K.

F EXPERIMENTAL DETAILS:

We use the same architecture to implement all models with different objectives. Input the staff is rude and the dr. does not spend time with you . slow service , the food tasted like last night 's leftovers . ARAE the staff is rude and the dr. does not worth two with you .

slow service , the food tasted like last night 's leftovers . β-VAE the staff was rude and the dr. did not spend time with your attitude . slow service , the food tastes like last place serves . AAE the staff was rude and the dr. does not spend time with you . slow service , the food tasted like last night 's leftovers . LAAE the staff was rude and the dr. is even for another of her entertained . slow service , the food , on this burger spot ! DAAE the staff was rude and the dr. did not make time with you .

slow service , the food tastes like last night ... .

Input they are the worst credit union in arizona . i reported this twice and nothing was done . ARAE they are the worst bank credit in arizona .

i swear this twice and nothing was done . β-VAE they were the worst credit union in my book .

i 've gone here and nothing too . AAE they are the worst credit union in arizona . i reported this twice and nothing was done . LAAE they were the worst credit union in my heart .

i dislike this twice so pleasant guy . DAAE they were the worst credit union in arizona ever .

i hate this pizza and nothing done . 

:

Our code will be made publicly available after the review process.

Transformer models(Vaswani et al., 2017) did not outperform LSTMs on our moderately-sized datasets.

We actually train E to maximize log D(E(x)) instead of -log(1 -D(E(x))), which is more stable in practice(Goodfellow et al., 2014). We also tried WGAN(Arjovsky et al., 2017) but did not notice any gains.

While some use importance sampling estimates of data likelihood to evaluate VAEs(He et al., 2019), adopting the encoder as a proposal density is not suited for AAE variants, as they are optimized based on Wasserstein distances rather than likelihoods and lack closed-form posteriors.

Normalized edit distance ∈ [0, 1] is the Levenshtein distance divided by the max length of two sentences.

Figures and Tables:

9 2 0 X S P Z 8 B e 8 e N A Y r / 4 h b / 4 b u 7 A H B V 8 y y c t 7 M 5 m Z F 8 S c a e O 6 3 0 5 h Y 3 N r e 6 e 4 W 9 r b P z g

Figure 1: Illustration of the learned latent geometry by AAE before and after introducing x perturbations. With high-capacity encoder/decoder networks, a standard AAE has no preference over x-z couplings and thus can learn a random mapping between them (Left). Trained with local perturbations C(x), DAAE learns to map similar x to close z to best achieve the denoising objective (Right).

9 a C k 8 8 c w x 8 4 n z / I R Y 1 q < / l a t e x i t > p = 0.3 < l a t e x i t s h a 1 _ b a s e 6 4 = " z 5 E v d a L 2 H U 8 e + z l n H y D y m n j N D u I

P z P 6 2 U m v g l z J t P M U E m W i + K M I 5 O g + e d o y B Q l h k 8 t w U Q x e y s i Y 6 w w M T a f i g 3 B X 3 1 5 n b T r r t 9 w 6 w 9 X t W a 9 i K M M Z 3 A O l + D D N T T h H l o Q A A E G z / A K b 4 5 0 X p x 3 5 2 P Z W n K K m V P 4 A + f z B 6 W O j d 4 = < / l a t e x i t > = 0.01 < l a t e x i t s h a 1 _ b a s e 6 4 = "

7 6 J a u 7 + s 1 G t 5 H E U 4 g V M 4 B w + u o A 5 3 0 I A m M J D w D K / w 5 m j n x X l 3 P h a l B S f v O Y Y / c D 5 / A A M J j 8 Y = < / l a t e x i t > = 0.5 < l a t e x i t s h a 1 _ b a s e 6 4 = " t T W r 4 s Y 2 i 0 v f t 1 l 4 a C 9 F d W B 0 a 4 M = " > A A A B 8 H i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x S J 4 C k l V 9 C I U v H i s Y D + k D W W z n b R L N 5 u w u x F K 6 a / w 4 k E R r / 4 c b / 4 b t 2 0 O 2 v p g 4 P H e D D P z w l R w b T

H z u c P o o a N A = = < / l a t e x i t > = 0.15 < l a t e x i t s h a 1 _ b a s e 6 4 = " b N 7 j

m l d p P H U S R H 5 J i c E o 9 c k h q 5 I 3 X S I I x I 8 k x e y Z u l r R f r 3 f q Y t x a s f O a Q / I H 1 + Q M P b o / b < / l a t e x i t > 1 = 0.01 < l a t e x i t s h a 1 _ b a s e 6 4 = " b l o H z X c I Q n y 3 d 2 F B J e j C R R 0 I S g E

7 s 4 r 9 e u 8 j i I c w T G c g g c X U I d b a E A T C C h 4 h l d 4 c 5 6 c F + f d + Z h H C 0 4 + c w h / 4 H z + A L G U k V M = < / l a t e x i t > 1 = 0.2 < l a t e x i t s h a 1 _ b a s e 6 4 = "

D j n X O 6 d 4 y e c K e 0 4 3 1 Z p b X 1 j c 6 u 8 X d n Z 3 d s / q B 4 e t V W c S k J b J O a x 7 P p Y U c 4 E b W m m O e 0 m k u L I 5 7 T j j + / m f m d C p W K x e N T T h H o R H g o W M o K 1 k b w + N 9 E A D 9 w b x 6

l a t e x i t > 1 = 0.05 < l a t e x i t s h a 1 _ b a s e 6 4 = " J a x M Z U e a A n r+ Q I W p k A v n R d R w E D 4 = " > A A A B 9 X i c b V D L S g M x F L 3 j s 9 Z X 1 a W b Y B F c l Z m q 6 E Y o u n F Z w T 6 g H U s m k 2 l D M 8 m Q Z J Q y 9 D / c u F D E r f / i zr 8 x b W e h r Q d C D u e c S 2 5 O k H C m j e t + O 0 v L K 6 t r 6 4 W N 4 u b W 9 s 5 u a W + / q W

K t e u 8 j g I c w h G c g A c X U I N b q E M D C C h 4 h l d 4 c 5 6 c F + f d + Z h F l 5 x 8 5 g D + w P n 8 A b e k k V c = < / l a t e x i t > p = 0.6

l 0 m z 5 n j n T u 3 h o l q / L e I o w T G c w B l 4 c A V 1 u I c G + E C B w z O 8 w h u S 6 A W 9 o 4 9 5 6 w o q Z o 7 g D 9 D n D 6 7 q j f E = < / l a t e x i t > 1 = 0.01 < l a t e x i t s h a 1 _ b a s e 6 4 = " b l o H z X c I Q n y 3 d 2 F B J e j C R R 0 I S g E

D j n X O 6 d 4 y e c K e 0 4 3 1 Z p b X 1 j c 6 u 8 X d n Z 3 d s / q B 4 e t V W c S k J b J O a x 7 P p Y U c 4 E b W m m O e 0 m k u L I 5 7 T j j + / m f m d C p W K x e N T T h H o R H g o W M o K 1 k b w + N 9 E A D 9 w b x 6

Figure 2: Generation-reconstruction trade-off of various text autoencoders on Yelp. The "real data" line marks the PPL of a language model trained and evaluated on real data. We strive to approach the lower right corner with both high BLEU and low PPL. The grey box identifies hyperparameters we use for respective models in subsequent experiments. Points of severe collapse (Reverse PPL > 200) are removed from the right panel.

i + log P j )log K

Examples of 5 nearest neighbors in the latent Euclidean space of AAE and DAAE on the Yelp dataset.

Table content: <table><tr><td colspan="4">Model ACC BLEU PPL</td></tr><tr><td>ARAE</td><td>17.2</td><td>55.7</td><td>59.1</td></tr><tr><td colspan="2">β-VAE 49.0</td><td>43.5</td><td>44.4</td></tr><tr><td>AAE</td><td>9.7</td><td>82.2</td><td>37.4</td></tr><tr><td>LAAE</td><td>43.6</td><td>37.5</td><td>55.8</td></tr><tr><td>DAAE</td><td>50.3</td><td>54.3</td><td>32.0</td></tr><tr><td colspan="4">β-VAE is better: 25 DAAE is better: 48</td></tr><tr><td colspan="4">both good: 26 both bad: 67 n/a: 34</td></tr></table>

Above: automatic evaluations of vector arithmetic for tense inversion. Below: human evaluation statistics of our model vs. the closest baseline β-VAE.

Table content: <table><tr><td>Model</td><td/><td colspan="3">ACC BLEU PPL</td></tr><tr><td colspan="3">Shen et al. (2017) 81.7</td><td>12.4</td><td>38.4</td></tr><tr><td>AAE DAAE</td><td>±v ±1.5v ±2v ±v ±1.5v ±2v</td><td>7.2 25.1 57.5 36.2 73.6 91.8</td><td>86.0 59.6 27.4 40.9 18.2 7.3</td><td>33.7 59.5 139.8 40.0 54.1 61.8</td></tr></table>

Automatic evaluations of vector arithmetic for sentiment transfer. Accuracy (ACC) is measured by a sentiment classifier. The model of Shen et al. (2017) is specifically trained for sentiment transfer with labeled data, while our text autoencoders are not.

Table content: <table/>

Examples of vector arithmetic for sentiment transfer.To quantitatively compare different models, we compute their tense transfer accuracy as measured by the parser, the output BLEU with the input sentence, and output (forward) PPL evaluated by a language model. DAAE achieves the highest accuracy, lowest PPL, and relatively high BLEU (Table2, Above), indicating that the output sentences produced by our model are more likely to be of high quality and of the proper tense, while remaining similar to the source sentence. A human evaluation on 200 test sentences (100 past and 100 present, details in Appendix G) suggests that DAAE outperforms β-VAE twice as often as it is outperformed, and our model successfully inverts tense for (48 + 26)/(200 -34) = 44.6% of sentences, 13.8% more than β-VAE (Table2, Below).

Table content: <table/>

Interpolations between two input sentences generated by AAE and our model on the Yelp dataset.

Table content: <table/>

2 ), where σ(•) denotes the sigmoid function. Note that this ensures i∈[4]    p G (x i |z j ) = 1 for each j ∈ [4], and does not violate the Lipschitz condition from Assumption 2 since:

Table content: <table/>

The encoder E, generator G, and the language model used to compute Forward/Reverse PPL are one-layer LSTMs with hidden dimension 1024 and word embedding dimension 512. The last hidden state of the encoder is projected into 128/256 dimensions to produce the latent code z for Yelp/Yahoo datasets J ADDITIONAL EXAMPLES Source how many gospels are there that were n't included in the bible ? 5-NN by AAE there are no other gospels that were n't included in the bible . how many permutations are there for the letters in the word _UNK ' ? anyone else picked up any of the _UNK in the film ? what 's the significance of the number 40 in the bible ? how many pieces of ribbon were used in the _UNK act ? 5-NN by DAAE there are no other gospels that were n't included in the bible . how many litres of water is there in the sea ? how many _UNK gods are there in the classroom ? how many pieces of ribbon were used in the _UNK act ? how many times have you been grounded in the last year ? Source how do i change colors in new yahoo mail beta ? 5-NN by AAE how should you present yourself at a _UNK speaking exam ? how can i learn to be a hip hop producer ? how can i create a _UNK web on the internet ? how can i change my _UNK for female not male ? what should you look for in buying your first cello ? 5-NN by DAAE how do i change that back to english ? is it possible to _UNK a yahoo account ? how do i change my yahoo toolbar options ? what should you look for in buying your first cello ? who do you think should go number one in the baseball fantasy draft , pujols or _UNK ? Table J.1: Examples of nearest neighbors in the latent Euclidean space of AAE and DAAE on Yahoo dataset.

Table content: <table/>

2: Additional examples of vector arithmetic for tense inversion.

Table content: <table/>

Peer review (iclr2020):
The paper *Denoising Improves Latent Space Geometry in Text Autoencoders* tackles the problem of text autoencoding in a space which respects text similarities. It is an interesting problem for which various attempts have been proposed, while still facing difficulties for encoding in smooth spaces. The paper proposes a simple (rather straightforward) approach based on adversarial learning, with some theoretical guarantees, which obtains good performances for reconstruction and neighborhood preservation. My main concern is about the missing of comparison with word dropout with variational encoding [Bowman et al., 2016], which also considers perturbations of the input texts to enforce the decoder to use the latent space. While the authors cite this work, I cannot understand why they did not include it in their experiments. Also, theorem 3 gives an upperbound of the achievable log-likelihood, which is *substantially better when examples in the same cluster are mapped to to points in the latent space in a manner that is well-separated from encodings of other clusters*. Ok but what does it show for the approach. If it was a lower-bound of the DAAE likelihood it would be interesting. But an upperbound ? In which sense does it indicate that it will be better than AAE ? Wouldn*t it be possible to theoretically analyze other baselines? Also, all the theoretical analysis is made based on strong assumptions. Are these verified on considered datasets? Minor questions : - In introduction of the experiments section, authors mention that they tried word removal and word masking. What is the difference ? - what is the language model used for forward and reverse ppl ?