Year: 2019

Abstract:
Neural language models have recently shown impressive gains in unconditional text generation, but controllable generation and manipulation of text remain challenging. In particular, controlling text via latent space operations in autoencoders has been difficult, in part due to chaotic latent space geometry. We propose to employ adversarial autoencoders together with denoising (referred as DAAE) to drive the latent space to organize itself. Theoretically, we prove that input sentence perturbations in the denoising approach encourage similar sentences to map to similar latent representations. Empirically, we illustrate the trade-off between textgeneration and autoencoder-reconstruction capabilities, and our model significantly improves over other autoencoder variants. Even from completely unsupervised training without style information, DAAE can perform various style transfers, including tense and sentiment, through simple latent vector arithmetic.1

heading: 1 INTRODUCTION
text: Autoencoder based generative models have recently become popular tools for advancing controllable text generation such as style or sentiment transfer (Bowman et al., 2016; Hu et al., 2017; Shen et al., 2017; Zhao et al., 2018). By mapping sentences to vectors in the latent space, these models offer in principle an attractive, continuous approach to manipulating text by means of simple latent vector arithmetic. However, the success of such manipulations rests heavily on the latent space geometry and how well it agrees with underlying sentence semantics. Indeed, we demonstrate that without additional guidance, fortuitous geometric agreements are unlikely to arise, shedding light on challenges faced by existing methods.
We use adversarial autoencoders (Makhzani et al., 2015, AAEs) to study the latent space geometry. In contrast to variational autoencoders (Kingma & Welling, 2014, VAEs), AAEs can maintain strong coupling between the encoder and decoder that the decoder does not omit the encoded input sentence (Bowman et al., 2016). The training criterion for AAEs consists of two parts, the ability to reconstruct sentences and the additional constraint that the encoded sentences are overall indistinguishable from prior samples, typically Gaussian. We show that these objectives alone do not suffice to force proper latent space geometry for text control. Specifically, for discrete objects such as sentences where continuity assumptions no longer hold, powerful AAEs can easily learn to map training sentences into latent prior samples arbitrarily (Figure 1, Left), while retaining perfect reconstruction. Latent space manipulations in such cases will yield random, unpredictable results.
To remedy this, we augment AAEs with a simple denoising objective (Vincent et al., 2008; Creswell & Bharath, 2018) that requires perturbed sentence with some words missing to be mapped back to the original version. We refer to our model as DAAE. We prove that the denoising criterion can eliminate disorganized solutions and drive the latent space to organize itself. As a result, similar sentences begin to be mapped to similar latent vectors (Figure 1, Right).
Improvements in latent space geometry carry many positive consequences. Through systematic evaluations of the generation and reconstruction capabilities of various text autoencoders (Cífka et al., 2018), we find that our proposed DAAE provides the best trade-off between producing highquality text vs. informative sentence representations. We empirically verify that DAAE has the best neighborhood preservation property, consistent with our theory. We further investigate to what extent
1Our code will be made publicly available after the review process.
text can be manipulated by applying simple transformations in the learned latent space. Our model is able to perform sentence-level vector arithmetic (Mikolov et al., 2013) fairly well to change the tense or sentiment of a sentence without any training supervision. It also produces higher quality sentence interpolations than other text autoencoders, suggesting better linguistic continuity in its latent space (Bowman et al., 2016).

heading: 2 RELATED WORK
text: Denoising is first introduced into standard autoencoders by Vincent et al. (2008, DAE) to learn robust representations. Without a latent prior, DAE requires sophisticated MCMC sampling to be employed generatively (Bengio et al., 2013). Creswell & Bharath (2018) applied denoising with AAEs to generative image modeling. Here, we demonstrate that input perturbations are particularly useful for discrete text modeling because they encourage preservation of data structure in the latent space.
Apart from the AAE framework that our paper focuses on, another popular latent variable generative model is the variational autoencoder (Kingma & Welling, 2014, VAE). Unfortunately, when the decoder is a powerful autoregressive model (such as a language model), VAE suffers from the posterior collapse problem where the latent representations get ignored (Bowman et al., 2016; Chen et al., 2016). If denoising is used in conjunction with VAEs (Im et al., 2017) in text applications, then the noisy inputs will only exacerbate VAE’s neglect of the latent variable. Bowman et al. (2016) proposed to weaken VAE’s decoder by masking words on the decoder side to alleviate its collapse issue. However, even with a weakened decoder and combined with other techniques including KL-weight annealing and adjusting training dynamics, it is still difficult to inject significant content into the latent code (Yang et al., 2017; Kim et al., 2018; He et al., 2019). Alternatives like the β-VAE (Higgins et al., 2017) appear necessary.
Previous work on controllable text generation has employed autoencoders trained with attribute label information (Hu et al., 2017; Shen et al., 2017; Zhao et al., 2018; Logeswaran et al., 2018; Subramanian et al., 2018). We show that the proposed DAAE model can perform text manipulations despite being trained in a completely unsupervised manner without attribute labels. This suggests that on the one hand, our model can be adapted to semi-supervised learning when a few labels are available. On the other hand, it can be easily scaled up to train one large model on unlabeled text corpora and then applied for transferring various styles.

heading: 3 METHOD
text: Define X = Vm to be a space of sequences of discrete symbols from vocabulary V (with maximum length m); also define Z = Rd to be a continuous latent space. Our goal is to learn a mapping between the data distribution pdata(x) over X and a given prior distribution p(z) over latent space Z (following common practice, a Gaussian prior is used in our experiments, although not required by our methodology). Such a mapping allows us to easily manipulate discrete data through continuous latent representations z, and provides a generative model where data samples can be obtained by first drawing z from the prior and then sampling a corresponding sequence via p(x|z).
We adopt the adversarial autoencoder (AAE) framework, which involves a (deterministic) encoder E : X → Z , a probabilistic decoder G : Z → X , and a discriminator D : Z → [0, 1] . Both E and G are recurrent neural networks (RNNs)2. E takes input sequence x and outputs the last hidden state as its encoding z. G generates a sequence x autoregressively, with each step conditioned on z and previous symbols. The discriminator D is a feed-forward net that outputs the probability of z coming from the prior rather than the encoder. E, G and D are trained jointly with a min-max objective:
min E,G max D Lrec(θE , θG)− λLadv(θE , θD) (1)
with: Lrec(θE , θG) = Epdata(x)[− log pG(x|E(x))] (2) Ladv(θE , θD) = Ep(z)[− logD(z)] + Epdata(x)[− log(1−D(E(x)))] (3)
where reconstruction loss Lrec and adversarial loss3 Ladv are weighted via hyperparameter λ > 0. We further introduce perturbations in X space to learn smoother representations that reflect local structure in the data, ending up with the denoising adversarial autoencoder (DAAE) model. Given a perturbation process C that stochastically maps x to nearby x̃ ∈ X , let p(x, x̃) = pdata(x)pC(x̃|x) and p(x̃) = ∑ x p(x, x̃). We change the loss functions to be:
Lrec(θE , θG) = Ep(x,x̃)[− log pG(x|E(x̃))] (4) Ladv(θE , θD) = Ep(z)[− logD(z)] + Ep(x̃)[− log(1−D(E(x̃)))] (5)
Here, Lrec is the loss of reconstructing x from x̃, and Ladv is the adversarial loss evaluated on perturbed x. The objective function combines the denoising technique with the AAE (Vincent et al., 2008; Creswell & Bharath, 2018). When pC(x̃|x) = 1[x̃ = x] (i.e. there is no perturbation), the above simply becomes the usual AAE objective.
Let pE(z|x) denote the encoder distribution. With our perturbation process C, the posterior distributions of the DAAE are of the following form:
q(z|x) = ∑ x̃ pC(x̃|x)pE(z|x̃) (6)
This enables the DAAE to utilize stochastic encodings even by merely employing a deterministic encoder network trained without any reparameterization-style tricks. Note that since q(z|x) of the form (6) is a subset of all possible conditional distributions, our model is still minimizing an upper bound of the Wasserstein distance between data and model distributions, as previously shown by Tolstikhin et al. (2017) for AAE (see Appendix A for a full proof).

heading: 4 LATENT SPACE GEOMETRY
text: The latent space geometry of text autoencoders is an important yet understudied problem. Only when the latent space is smooth and regular can meaningful text manipulations be enacted via simple modifications of the corresponding latent representations. Here, we discuss in detail the posterior characteristics of the DAAE, and provide a theoretical analysis of how input perturbations help better structure the latent space geometry (all proofs are relegated to the appendix).
Assume our perturbations preserve x with some probability (i.e. pC(x|x) > 0). When the support of C(x1) and C(x2) do not overlap for different training examples x1 6= x2, the encoder can learn to assign pE(z|x̃) = pE(z|x) for x̃ ∈ C(x), and we are back to the unconstrained posterior scenario q(z|x) = pE(z|x) (Eq. 6). If C(x1) and C(x2) do intersect, then the latent posterior of x1 and x2 will have overlapping components pE(z|x̃) for x̃ ∈ C(x1)∩C(x2). For example, if pC(x̃|x) assigns a high probability to x̃ that lies close to x (based on some metric over X ), then for similar x1 and x2, the high-probability overlap between their perturbations will inherently force their posteriors closer together in the latent space. This is desirable for learning good representations z, while not guaranteed by merely minimizing the statistical divergence between pdata(x) and pG(x) = Ep(z)[pG(x|z)]. Now we formally analyze which type of x-z mappings will be learned by AAE and DAAE, respectively, to achieve global optimality of their training objectives. Unlike previous analyses of
2Transformer models (Vaswani et al., 2017) did not outperform LSTMs on our moderately-sized datasets. 3We actually train E to maximize logD(E(x)) instead of − log(1 −D(E(x))), which is more stable in
practice (Goodfellow et al., 2014). We also tried WGAN (Arjovsky et al., 2017) but did not notice any gains.
noise in single-layer networks (Poole et al., 2014), here we study high-capacity encoder/decoder networks (Schäfer & Zimmermann, 2006) with a large number of parameters that are used in modern sequence models (Devlin et al., 2018; Radford et al., 2019). Throughout, we assume that: Assumption 1. E is a universal approximator capable of producing any mapping from x’s to z’s. Assumption 2. G can approximate arbitrary p(x|z) so long as it remains sufficiently Lipschitz continuous in z. Namely, there exists L > 0 such that all decoder models G obtainable via training satisfy that for all x ∈ X , z1, z2 ∈ Z: | log pG(x|z1)− log pG(x|z2)| ≤ L‖z1 − z2‖. Following prior analysis of language decoders (Mueller et al., 2017), we assume that G is L-Lipschitz in its continuous input z (denote this set of possible decoders by GL). When G is implemented as a RNN or Transformer language model, log pG(x|z) will remain Lipschitz in z if the recurrent or attention weight matrices have bounded norm. This property is naturally encouraged by popular training methods that utilize SGD with early stopping and L2 regularization (Zhang et al., 2017). Note we have not assumed E or G is Lipschitz in x, which would be unreasonable since x stands for discrete text, and when a few symbols change, the decoder likelihood for the entire sequence can vary drastically (e.g., G may assign a much higher probability to a grammatical sentence than an ungrammatical one that only differs by one word). Our discussion is directed to the nature of such families of log-likelihood functions with a continuous variable z and a discrete variable x.
We further assume an effectively trained discriminator that succeeds in its adversarial task: Assumption 3. D ensures that the latent encodings z1, · · · , zn of training examples x1, · · · , xn are indistinguishable from prior samples z ∼ p(z). For simplicity, we directly assume that z1, · · · , zn are actual samples from p(z) which are given a priori. Here, the task of the encoder E is to map given unique training examples to the given latent points, and the goal of the decoder pG(·|·) is to maximize −Lrec under the encoder mapping. The question now is which one-to-one mapping an optimal encoder/decoder will learn under the AAE and DAAE objective (Eq. 2 and Eq. 4). We start with the following observation: Theorem 1. For any one-to-one encoder mappingE from {x1, · · · , xn} to {z1, · · · , zn}, the optimal value of objective maxG∈GL 1 n ∑n i=1 log pG(xi|E(xi)) is the same.
Intuitively, this result stems from the fact that the model receives no information about the structure of x, and x1, · · · , xn are simply provided as different symbols. Hence AAE offers no preference over x-z couplings, and a random matching in which the z do not reflect any data structure is equally good as any other matching (Figure 1, Left). Latent point assignments start to differentiate, however, once we introduce local input perturbations.
To elucidate how perturbations affect latent space geometry, it helps to first consider a simple setting with only four examples x1, x2, x3, x4 ∈ X . Again, we consider given latent points z1, z2, z3, z4 sampled from p(z), and the encoder/decoder are tasked with learning which x to match with which z. As depicted in Figure 1, suppose there are two pairs of x closer together and also two pairs of z closer together. Let σ denote the sigmoid function, we have the following conclusion: Theorem 2. Let d be a distance metric over X . Suppose x1, x2, x3, x4 satisfy that with some > 0: d(x1, x2) < , d(x3, x4) < , and d(xi, xj) > for all other (xi, xj) pairs. In addition, z1, z2, z3, z4 satisfy that with some 0 < δ < ζ: ‖z1 − z2‖ < δ, ‖z3 − z4‖ < δ, and ‖zi − zj‖ > ζ for all other (zi, zj) pairs. Suppose our perturbation process C reflects local X geometry with: pC(xi|xj) = 1/2 if d(xi, xj) < and = 0 otherwise. For δ < 1L (2 log (σ(Lζ)) + log 2) and ζ > 1 L log ( 1/( √ 2− 1) ) , the denoising objective maxG∈GL 1 n ∑n i=1 ∑n j=1 pC(xj |xi) log pG(xi|E(xj)) (where n = 4) achieves the largest value when encoder E maps close pairs of x to close pairs of z.
This entails that DAAE will always prefer to map similar x to similar z. Note that Theorem 1 still applies here, and AAE will not prefer any particular x-z pairing over the other possibilities. We next generalize beyond the basic four-points scenario to consider n examples of x that are clustered. Here, we can ask whether this cluster organization will be reflected in the latent space of DAAE. Theorem 3. Suppose x1, · · · , xn are divided into n/K clusters of equal size K, with Si denoting the cluster index of xi. Let the perturbation process C be uniform within clusters, i.e. pC(xi|xj) = 1/K if Si = Sj and = 0 otherwise. For a one-to-one encoder mapping E from {x1, · · · , xn} to {z1, · · · , zn}, the denoising objective maxG∈GL 1n ∑n i=1 ∑n j=1 pC(xj |xi) log pG(xi|E(xj)) is
upper bounded by: 1n2 ∑ i,j:Si 6=Sj log σ(L‖E(xi)− E(xj)‖)− logK.
Theorem 3 provides an upper bound of the DAAE objective that can be achieved by a particular x-z mapping. This achievable limit is substantially better when examples in the same cluster are mapped to the latent space in a manner that is well-separated from encodings of other clusters. In other words, by preserving input space cluster structure in the latent space, DAAE can achieve better objective values and thus is incentivized to learn such encoder/decoder mappings. An analogous corollary can be shown for the case when examples x are perturbed to yield additional inputs x̃ not present in the training data. In this case, the model would aim to map each example and its perturbations as a group to a compact group of z points well-separated from other groups in the latent space.
In conclusion, our analysis shows that a well-trained DAAE is guaranteed to learn neighborhoodpreserving latent representations, whereas even a perfectly-trained AAE model may learn latent representations whose geometry fails to reflect similarity in the x space. Empirical experiments in Section 5.2 confirm that our theory holds in practice.

heading: 5 EXPERIMENTS
text: We evaluate our proposed model and other text autoencoders on two benchmark datasets: Yelp reviews and Yahoo answers (Shen et al., 2017; Yang et al., 2017). Detailed descriptions of datasets, training settings, human evaluations, and additional results/examples can be found in the appendix.
Perturbation Process We randomly delete each word with probability p, so that perturbations of sentences with more words in common will have a larger overlap. We also tried replacing each word with a <mask> token or a random word and found that they all brought improvements, but deleting words worked best. We leave it to future work to explore more sophisticated text perturbations.
Baselines We compare our proposed DAAE with four alternative text autoencoders: adversarially regularized autoencoder (Zhao et al., 2018, ARAE), β-VAE (Higgins et al., 2017), AAE (Makhzani et al., 2015), and latent-noising AAE (Rubenstein et al., 2018, LAAE). Similar to our model, the LAAE uses Gaussian perturbations in the latent space to improve AAE’s latent geometry (rather than perturbations in the sentence space). However, LAAE requires enforcing an L1 penalty (λ1 · ‖ log σ2(x)‖1) on the latent perturbations’ log-variance to prevent them from vanishing. In contrast, input perturbations in DAAE enable stochastic latent representations without parametric restrictions like Gaussianity.

heading: 5.1 GENERATION-RECONSTRUCTION TRADE-OFF
text: We evaluate various latent variable generative models in terms of both generation quality and reconstruction accuracy. A strong model should not only generate high quality sentences, but also learn useful latent variables that capture significant data content. Recent work on text autoencoders has found an inherent tension between these aims (Bowman et al., 2016; Cífka et al., 2018), yet only when both goals are met can we successfully manipulate sentences by modifying their latent representation (in order to produce valid output sentences that retain the semantics of the input).
We compute the BLEU score (Papineni et al., 2002) between input and reconstructed sentences to measure reconstruction accuracy, and compute Forward/Reverse PPL to measure sentence generation quality (Zhao et al., 2018; Cífka et al., 2018).4 Forward PPL is the perplexity of a language model trained on real data and evaluated on generated data. It measures the fluency of the generated text, but cannot detect the collapsed case where the model repeatedly generates a few common sentences. Reverse PPL is the perplexity of a language model trained on generated data and evaluated on real data. It takes into account both the fluency and diversity of the generated text. If a model generates only a few common sentences, a language model trained on it will exhibit poor PPL on real data.
We thoroughly investigate the performance of different models and their trade-off between generation and reconstruction. Figure 2 plots reconstruction BLEU (higher is better) vs. Forward/Reverse PPL (lower is better). The lower right corner indicates an ideal situation where good reconstruction accuracy and generation quality are both achieved. For models with tunable hyperparameters, we sweep the full spectrum of their generation-reconstruction trade-off by varying the KL coefficient β of β-VAE, the log-variance L1 penalty λ1 of LAAE, and the word drop probability p of DAAE.
4 While some use importance sampling estimates of data likelihood to evaluate VAEs (He et al., 2019), adopting the encoder as a proposal density is not suited for AAE variants, as they are optimized based on Wasserstein distances rather than likelihoods and lack closed-form posteriors.
In the left panel, we observe that a standard VAE (β = 1) completely collapses and ignores the latent variable z, resulting in reconstruction BLEU close to 0. At the other extreme, AAE can achieve near-perfect reconstruction, but its latent space is highly non-smooth and generated sentences are of poor quality, indicated by its large Forward PPL. Decreasing β in VAE or introducing latent noises in AAE provides the model with a similar trade-off curve between reconstruction and generation. We note that ARAE falls on or above their curves, revealing that it does not fare better than these methods (Cífka et al. (2018) also reported similar findings). Our proposed DAAE provides a trade-off curve that is strictly superior to other models. With discrete x and a complex encoder, the Gaussian perturbations added to the latent space by β-VAE and LAAE are not directly related to how the inputs are encoded. In contrast, input perturbations added by DAAE can constrain the encoder to maintain coherence between neighboring inputs in an end-to-end fashion and help learn smoother latent space.
The right panel in Figure 2 illustrates that Reverse PPL first drops and then rises as we increase the degree of regularization/perturbation. This is because when z encodes little information, generations from prior-sampled z lack enough diversity to cover the real data. Again, DAAE outperforms the other models which tend to have higher Reverse PPL and lower reconstruction BLEU. In subsequent experiments, we set β = 0.15 for β-VAE, λ1 = 0.05 for LAAE, and p = 0.3 for DAAE, to ensure they have strong reconstruction abilities and encode enough information to enable text manipulations.

heading: 5.2 NEIGHBORHOOD PRESERVATION
text: In this section, we empirically investigate whether our previous theory holds in practice. That is, in actual autoencoder models trained on real text datasets, do sentence perturbations induce latent space organization that better preserves neighborhood structure in the data space?
Under our word-drop perturbation process, sentences with more words in common are more likely to be perturbed into one another. This choice of C approximately encodes sentence similarity via the normalized edit distance5. Within the test set, we find both the 10 nearest neighbors of each sentence based on the normalized edit distance (denote this set by NNx), as well as the k nearest neighbors based on Euclidean distance between latent representations (denote this set by NNz). We
5Normalized edit distance ∈ [0, 1] is the Levenshtein distance divided by the max length of two sentences.
compute the recall rate |NNx ∩ NNz| / |NNx|, which indicates how well local neighborhoods are preserved in the latent space of different models.
Figures 3 shows that DAAE consistently gives the highest recall, about 1.5∼2 times that of AAE, implying that input perturbations have a substantial effect on shaping the latent space geometry. Tables 1 presents the five nearest neighbors found by AAE and DAAE in their latent space for example test set sentences. The AAE sometimes encodes entirely unrelated sentences close together, while the latent space geometry of the DAAE is structured based on key words such as “attentive” and “personable”, and tends to group sentences with similar semantics close together.

heading: 5.3 APPLICATIONS TO CONTROLLABLE TEXT GENERATION

heading: 5.3.1 STYLE TRANSFER VIA VECTOR ARITHMETIC
text: Mikolov et al. (2013) previously discovered that word embeddings from unsupervised learning can capture linguistic relationships via simple arithmetic. A canonical example is the embedding arithmetic “King” - “Man” + “Woman” ≈ “Queen”. Here, we use the Yelp dataset with tense and sentiment as two example attributes (Hu et al., 2017; Shen et al., 2017) to investigate whether analogous structure emerges in the latent space of our sentence-level models.
Tense We use the Stanford Parser to extract the main verb of a sentence and determine the sentence tense based on its part-of-speech tag. We compute a single “tense vector” by averaging the latent code z separately for 100 past tense sentences and 100 present tense sentences in the dev set, and then calculating the difference between the two. Given a sentence from the test set, we attempt to change its tense from past to present or from present to past through simple addition/subtraction of the tense vector. More precisely, a source sentence x is first is encoded to z = E(x), and then the tense-modified sentence is produced via G(z ± v), where v ∈ Rd denotes the fixed tense vector.
To quantitatively compare different models, we compute their tense transfer accuracy as measured by the parser, the output BLEU with the input sentence, and output (forward) PPL evaluated by a language model. DAAE achieves the highest accuracy, lowest PPL, and relatively high BLEU (Table 2, Above), indicating that the output sentences produced by our model are more likely to be of high quality and of the proper tense, while remaining similar to the source sentence. A human evaluation on 200 test sentences (100 past and 100 present, details in Appendix G) suggests that DAAE outperforms β-VAE twice as often as it is outperformed, and our model successfully inverts tense for (48 + 26)/(200− 34) = 44.6% of sentences, 13.8% more than β-VAE (Table 2, Below). Tables 4 and J.2 show the results of adding or subtracting this fixed latent vector offset under different models. DAAE can successfully change “enjoy” to “enjoyed”, or change the subjunctive mood to declarative mood and adjust the word order. Other baselines either fail to alter the tense, or undesirably change the semantic meaning of the source sentence (e.g. “enjoy” to “made”).
Sentiment Following the same procedure used to alter tense, we compute a “sentiment vector” v from 100 negative and positive sentences and use it to change the sentiment of test sentences. Table 3 reports the automatic evaluations, and Tables 5 and J.3 show examples generated by AAE and DAAE. Scaling ±v to ±1.5v and ±2v, we find that the resulting sentences get more and more positive/negative. However, the PPL for AAE increases rapidly with this scaling factor, indicating that the sentences become unnatural when their encodings have a large offset. DAAE enjoys a much smoother latent space than AAE. Despite the fact that no sentiment labels are provided during training (a more challenging task than previous style transfer models (Shen et al., 2017)), DAAE with ±1.5v is able to transfer sentiment fairly well.

heading: 5.3.2 SENTENCE INTERPOLATION VIA LATENT SPACE TRAVERSAL
text: We also study sentence interpolation by traversing the latent space of text autoencoders. Given two input sentences, we encode them to z1, z2 and decode from tz1 + (1 − t)z2 (0 ≤ t ≤ 1) to obtain interpolated sentences. Ideally this should produce fluent sentences with gradual semantic change (Bowman et al., 2016). Table 6 shows two examples from the Yelp dataset, where it is clear that DAAE produces more coherent and natural interpolations than AAE. Table J.4 in the appendix shows two difficult examples from the Yahoo dataset, where we interpolate between dissimilar sentences. While it is challenging to generate semantically correct sentences in these cases, the latent space of our model exhibits continuity on topic and syntactic structure.

heading: 6 CONCLUSION
text: This paper proposed DAAE for generative text modeling. As revealed in previous work (Devlin et al., 2018; Lample et al., 2018), we find that denoising techniques can greatly improve the learned text representations. We provide a theoretical explanation for this phenomenon by analyzing the latent
Input 1 it ’s so much better than the other chinese food places in this area . fried dumplings are a must . Input 2 better than other places . the fried dumplings are a must if you ever visit this place .
space geometry arisen from input perturbations. Our proposed model substantially outperforms other text autoencoders, and demonstrates potential for various text manipulations via vector operations. Future work may investigate superior perturbation strategies and additional properties of latent space geometry to provide finer control over the text generated using autoencoder models.

heading: A WASSERSTEIN DISTANCE
text: The AAE objective can be connected to a relaxed form of the Wasserstein distance between model and data distributions (Tolstikhin et al., 2017). Specifically, for cost function c(·, ·) : X × X → R and deterministic decoder mapping G : Z → X , it holds that:
inf Γ∈P(x∼pdata,y∼pG) E(x,y)∼Γ[c(x, y)] = inf q(z|x):q(z)=p(z) Epdata(x)Eq(z|x)[c(x,G(z))] (7)
where the minimization over couplings Γ with marginals pdata and pG can be replaced with minimization over conditional distributions q(z|x) whose marginal q(z) = Epdata(x)[q(z|x)] matches the latent prior distribution p(z). Relaxing this marginal constraint via a divergence penalty D(q(z)‖p(z)) estimated by adversarial training, one recovers the AAE objective (Eq. 1). In particular, AAE on discrete x with the cross-entropy loss is minimizing an upper bound of the total variation distance between pdata and pG, with c chosen as the indicator cost function (Zhao et al., 2018).
Our model is optimizing over conditional distributions q(z|x) of the form (6), a subset of all possible conditional distributions. Thus, after introducing input perturbations, our method is still minimizing an upper bound of the Wasserstein distance between pdata and pG described in (7).

heading: B PROOF OF THEOREM 1
text: Theorem 1. For any one-to-one encoder mappingE from {x1, · · · , xn} to {z1, · · · , zn}, the optimal value of objective maxG∈GL 1 n ∑n i=1 log pG(xi|E(xi)) is the same.
Proof. Consider two encoder matchings xi to zα(i) and xi to zβ(i), where both α and β are permutations of the indices {1, . . . , n}. Suppose Gα is the optimal decoder model for the first matching (with permutations α). This implies
pGα = arg max G∈GL n∑ i=1 log pG(xi|zα(i))
Now let pGβ (xi|zj) = pGα(xβα−1(i)|zj),∀i, j. ThenGβ can achieve exactly the same log-likelihood objective value for matching β as Gα for matching α, while still respecting the Lipschitz constraint.

heading: C PROOF OF THEOREM 2
text: Theorem 2. Let d be a distance metric over X . Suppose x1, x2, x3, x4 satisfy that with some > 0: d(x1, x2) < , d(x3, x4) < , and d(xi, xj) > for all other (xi, xj) pairs. In addition, z1, z2, z3, z4 satisfy that with some 0 < δ < ζ: ‖z1 − z2‖ < δ, ‖z3 − z4‖ < δ, and ‖zi − zj‖ > ζ for all other (zi, zj) pairs. Suppose our perturbation process C reflects local X geometry with: pC(xi|xj) = 1/2 if d(xi, xj) < and = 0 otherwise. For δ < 1L (2 log (σ(Lζ)) + log 2) and ζ > 1 L log ( 1/( √ 2− 1) ) , the denoising objective maxG∈GL 1 n ∑n i=1 ∑n j=1 pC(xj |xi) log pG(xi|E(xj)) (where n = 4) achieves the largest value when encoder E maps close pairs of x to close pairs of z.
Proof. Let [n] denote {1, . . . , n}, and assume without loss of generality that the encoder E maps each xi to zi. We also define A = {1, 2}, B = {3, 4} as the two x-pairs that lie close together. For our choice of C(x), the training objective to be maximized is:∑
i,j∈A log pG(xi|E(xj)) + ∑ k,`∈B log pG(xk|E(x`))
= ∑ i,j∈A log pG(xi|zj) + ∑ k,`∈B log pG(xk|z`) (8)
The remainder of our proof is split into two cases:
Case 1. ||zj − z`|| > ζ for j ∈ A, ` ∈ B
Case 2. ||zj − z`|| < δ for j ∈ A, ` ∈ B Under Case 1, x points that lie far apart also have z encodings that remain far apart. Under Case 2, x points that lie far apart have z encodings that lie close together. We complete the proof by showing that the achievable objective value in Case 2 is strictly worse than in Case 1, and thus an optimal encoder/decoder pair would avoid the x, z matching that leads to Case 2.
In Case 1 where ||zj − z`|| > ζ for all j ∈ A, ` ∈ B, we can lower bound the training objective (8) by choosing:
pG(xi|zj) = {
(1− γ)/2 if i, j ∈ A or i, j ∈ B γ/2 otherwise
(9)
with γ = σ(−Lζ) ∈ (0, 12 ), where σ(·) denotes the sigmoid function. Note that this ensures∑ i∈[4] pG(xi|zj) = 1 for each j ∈ [4], and does not violate the Lipschitz condition from Assumption 2 since:
| log pG(xi|zj)− log pG(xi|z`)| {
= 0 if j, ` ∈ A or j, ` ∈ B ≤ log ((1− γ)/γ) otherwise
and thus remains≤ L||zj−z`|| when γ = σ(−Lζ) ≥ σ(−L||zj−z`||) = 1/[1+exp(L||zj−z`||)]. Plugging the pG(x|z) assignment from (9) into (8), we see that an optimal decoder can obtain training objective value ≥ 8 log [σ(Lζ)/2] in Case 1 where ||zj − z`|| > ζ, ∀j ∈ A, ` ∈ B.
Next, we consider the alternative case where ||zj − z`|| < δ for j ∈ A, ` ∈ B. For i, j ∈ A and for all ` ∈ B, we have:
log pG(xi|zj) ≤ log pG(xi|z`) + L||zj − z`|| by Assumption 2 ≤ log pG(xi|z`) + Lδ
≤ Lδ + log [
1− ∑ k∈B
pG(xk|z`) ]
since ∑ k pG(xk|z`) ≤ 1
Continuing from (8), the overall training objective in this case is thus:∑ i,j∈A log pG(xi|zj) + ∑ k,`∈B log pG(xk|z`)
≤ 4Lδ + ∑ i,j∈A min `∈B log
[ 1−
∑ k∈B
pG(xk|z`) ]
+ ∑ k,`∈B log pG(xk|z`)
≤ 4Lδ + ∑ `∈B
[ 2 log ( 1−
∑ k∈B
pG(xk|z`) )
+ ∑ k∈B
log pG(xk|z`) ]
≤ 4Lδ − 12 log 2 using the fact that the optimal decoder for the bound in this case is: pG(xk|z`) = 1/4 for all k, ` ∈ B. Finally, plugging our range for δ stated in the Theorem 2, it shows that the best achievable objective value in Case 2 is strictly worse than the objective value achievable in Case 1. Thus, the optimal encoder/decoder pair under the AAE with perturbed x will always prefer the matching between {x1, . . . , x4} and {z1, . . . , z4} that ensures nearby xi are encoded to nearby zi (corresponding to Case 1).

heading: D PROOF OF THEOREM 3
text: Theorem 3. Suppose x1, · · · , xn are divided into n/K clusters of equal size K, with Si denoting the cluster index of xi. Let the perturbation process C be uniform within clusters, i.e. pC(xi|xj) = 1/K if Si = Sj and = 0 otherwise. For a one-to-one encoder mapping E from {x1, · · · , xn} to {z1, · · · , zn}, the denoising objective maxG∈GL 1n ∑n i=1 ∑n j=1 pC(xj |xi) log pG(xi|E(xj)) is
upper bounded by: 1n2 ∑ i,j:Si 6=Sj log σ(L‖E(xi)− E(xj)‖)− logK.
Proof. Without loss of generality, let E(xi) = zi for notational convenience. We consider what is the optimal decoder probability assignment pG(xi|zj) under the Lipschitz constraint 2. The objective of the AAE with perturbed x is to maximize:
1
n ∑ i ∑ j pC(xj |xi) log pG(xi|E(xj)) = 1 nK ∑ j ∑ i:Si=Sj log pG(xi|zj)
We first show that the optimal pG(·|·) will satisfy that the same probability is assigned within a cluster, i.e. p(xi|zj) = p(xk|zj) for all i, k s.t. Si = Sk. If not, let Psj = ∑ i:Si=s
pG(xi|zj), and we reassign pG′(xi|zj) = PSij/K. Then G′ still conforms to the Lipschitz constraint if G meets it, and G′ will have a larger target value than G.
Now let us define Pj = ∑ i:Si=Sj pG(xi|zj) = K ·pG(xj |zj) (0 ≤ Pj ≤ 1). The objective becomes:
max pG
1
nK ∑ j ∑ i:Si=Sj log pG(xi|zj) = max pG 1 n ∑ j log pG(xj |zj)
= max pG
1
n ∑ j logPj − logK
= max pG
1
2n2 ∑ i ∑ j (logPi + logPj)− logK
≤ 1 2n2 ∑ i ∑ j max pG (logPi + logPj)− logK
Consider each term maxpG(logPi + logPj): when Si = Sj , this term can achieve the maximum value 0 by assigning Pi = Pj = 1; when Si 6= Sj , the Lipschitz constraint ensures that:
log(1− Pi) ≥ logPj − L‖zi − zj‖ log(1− Pj) ≥ logPi − L‖zi − zj‖
Therefore:
logPi + logPj ≤ 2 log σ(L‖zi − zj‖) Overall, we thus have:
max pG
1
nK ∑ j ∑ i:Si=Sj log pG(xi|zj) ≤ 1 n2 ∑ i,j:Si 6=Sj log σ(L‖zi − zj‖)− logK

heading: E DATASETS
text: The Yelp dataset is from Shen et al. (2017), which has 444K/63K/127K sentences of less than 16 words in length as train/dev/test sets, with a vocabulary of 10K. It was originally divided into positive and negative sentences for style transfer between them. Here we discard the sentiment label and let the model learn from all sentences indiscriminately. Our second dataset of Yahoo answers is from Yang et al. (2017). It was originally document-level. We perform sentence segmentation and keep sentences with length from 2 to 50 words. The resulting dataset has 495K/49K/50K sentences for train/dev/test sets, with vocabulary size 20K.

heading: F EXPERIMENTAL DETAILS
text: We use the same architecture to implement all models with different objectives. The encoder E, generator G, and the language model used to compute Forward/Reverse PPL are one-layer LSTMs with hidden dimension 1024 and word embedding dimension 512. The last hidden state of the encoder is projected into 128/256 dimensions to produce the latent code z for Yelp/Yahoo datasets
respectively, which is then projected and added with input word embeddings fed to the generator. The discriminator D is an MLP with one hidden layer of size 512. λ of AAE based models is set to 10 to ensure the latent codes are indistinguishable from the prior. All models are trained via the Adam optimizer (Kingma & Ba, 2014) with learning rate 0.0005, β1 = 0.5, β2 = 0.999. At test time, encoder-side perturbations are disabled, and we use greedy decoding to generate x from z.

heading: G HUMAN EVALUATION
text: For the tense transfer experiment, the human annotator is presented with a source sentence and two outputs (one from each approach, presented in random order) and asked to judge which one successfully changes the tense while being faithful to the source, or whether both are good/bad, or if the input is not suitable to have its tense inverted. We collect labels from two human annotators and if they disagree, we further solicit a label from the third annotator.
H GENERATION-RECONSTRUCTION RESULTS ON THE YAHOO DATASET
= 0.01 <latexit sha1_base64="BV65hVKyh8AENiYdPQ/U24EATCs=">AAAB8XicbVBNSwMxEJ2tX7V+VT16CRbBU9mtgl6EghePFewHtkvJprNtaDa7JFmhLP0XXjwo4tV/481/Y9ruQVsfhDzem2FmXpAIro3rfjuFtfWNza3idmlnd2//oHx41NJxqhg2WSxi1QmoRsElNg03AjuJQhoFAtvB+Hbmt59QaR7LBzNJ0I/oUPKQM2qs9NgL0NAbt+p6/XLFfnOQVeLlpAI5Gv3yV28QszRCaZigWnc9NzF+RpXhTOC01Es1JpSN6RC7lkoaofaz+cZTcmaVAQljZZ80ZK7+7shopPUkCmxlRM1IL3sz8T+vm5rw2s+4TFKDki0GhakgJiaz88mAK2RGTCyhTHG7K2EjqigzNqSSDcFbPnmVtGpV76Jau7+s1Gt5HEU4gVM4Bw+uoA530IAmMJDwDK/w5mjnxXl3PhalBSfvOYY/cD5/AAMJj8Y=</latexit>
= 1 <latexit sha1_base64="LYFEefZ6fSFsCaDjOrsC+RMD4FE=">AAAB7nicbVDLSgNBEOz1GeMr6tHLYBA8hd0o6EUIePEYwTwgWcLspJMMmX0w0yuEJR/hxYMiXv0eb/6Nk2QPmljQUFR1090VJEoact1vZ219Y3Nru7BT3N3bPzgsHR03TZxqgQ0Rq1i3A25QyQgbJElhO9HIw0BhKxjfzfzWE2oj4+iRJgn6IR9GciAFJyu1ugESv/V6pbJbcedgq8TLSRly1Hulr24/FmmIEQnFjel4bkJ+xjVJoXBa7KYGEy7GfIgdSyMeovGz+blTdm6VPhvE2lZEbK7+nsh4aMwkDGxnyGlklr2Z+J/XSWlw42cySlLCSCwWDVLFKGaz31lfahSkJpZwoaW9lYkR11yQTahoQ/CWX14lzWrFu6xUH67KtWoeRwFO4QwuwINrqME91KEBAsbwDK/w5iTOi/PufCxa15x85gT+wPn8AbUijxo=</latexit>
p = 0.1 <latexit sha1_base64="GHsAGalSKw3XqD6sSHkBD+XWD1k=">AAAB7HicbVBNS8NAEJ34WetX1aOXxSJ4CkkV9CIUvHisYNpCG8pmO2mXbjZhdyOU0t/gxYMiXv1B3vw3btsctPXBwOO9GWbmRZng2njet7O2vrG5tV3aKe/u7R8cVo6OmzrNFcOApSJV7YhqFFxiYLgR2M4U0iQS2IpGdzO/9YRK81Q+mnGGYUIHksecUWOlILv1XL9XqXquNwdZJX5BqlCg0at8dfspyxOUhgmqdcf3MhNOqDKcCZyWu7nGjLIRHWDHUkkT1OFkfuyUnFulT+JU2ZKGzNXfExOaaD1OItuZUDPUy95M/M/r5Ca+CSdcZrlByRaL4lwQk5LZ56TPFTIjxpZQpri9lbAhVZQZm0/ZhuAvv7xKmjXXv3RrD1fVeq2IowSncAYX4MM11OEeGhAAAw7P8ApvjnRenHfnY9G65hQzJ/AHzucPooaN3A==</latexit>
p = 1 <latexit sha1_base64="ha4QS8DD+94qqDrze0UWo2Ckk0o=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mqoBeh4MVjRfsBbSib7aRdutmE3Y1QQn+CFw+KePUXefPfuG1z0NYHA4/3ZpiZFySCa+O6305hbX1jc6u4XdrZ3ds/KB8etXScKoZNFotYdQKqUXCJTcONwE6ikEaBwHYwvp357SdUmsfy0UwS9CM6lDzkjBorPSQ3Xr9ccavuHGSVeDmpQI5Gv/zVG8QsjVAaJqjWXc9NjJ9RZTgTOC31Uo0JZWM6xK6lkkao/Wx+6pScWWVAwljZkobM1d8TGY20nkSB7YyoGellbyb+53VTE177GZdJalCyxaIwFcTEZPY3GXCFzIiJJZQpbm8lbEQVZcamU7IheMsvr5JWrepdVGv3l5V6LY+jCCdwCufgwRXU4Q4a0AQGQ3iGV3hzhPPivDsfi9aCk88cwx84nz/IRY1q</latexit>
= 0.01 <latexit sha1_base64="BV65hVKyh8AENiYdPQ/U24EATCs=">AAAB8XicbVBNSwMxEJ2tX7V+VT16CRbBU9mtgl6EghePFewHtkvJprNtaDa7JFmhLP0XXjwo4tV/481/Y9ruQVsfhDzem2FmXpAIro3rfjuFtfWNza3idmlnd2//oHx41NJxqhg2WSxi1QmoRsElNg03AjuJQhoFAtvB+Hbmt59QaR7LBzNJ0I/oUPKQM2qs9NgL0NAbt+p6/XLFfnOQVeLlpAI5Gv3yV28QszRCaZigWnc9NzF+RpXhTOC01Es1JpSN6RC7lkoaofaz+cZTcmaVAQljZZ80ZK7+7shopPUkCmxlRM1IL3sz8T+vm5rw2s+4TFKDki0GhakgJiaz88mAK2RGTCyhTHG7K2EjqigzNqSSDcFbPnmVtGpV76Jau7+s1Gt5HEU4gVM4Bw+uoA530IAmMJDwDK/w5mjnxXl3PhalBSfvOYY/cD5/AAMJj8Y=</latexit>
p = 0.1 <latexit sha1_base64="GHsAGalSKw3XqD6sSHkBD+XWD1k=">AAAB7HicbVBNS8NAEJ34WetX1aOXxSJ4CkkV9CIUvHisYNpCG8pmO2mXbjZhdyOU0t/gxYMiXv1B3vw3btsctPXBwOO9GWbmRZng2njet7O2vrG5tV3aKe/u7R8cVo6OmzrNFcOApSJV7YhqFFxiYLgR2M4U0iQS2IpGdzO/9YRK81Q+mnGGYUIHksecUWOlILv1XL9XqXquNwdZJX5BqlCg0at8dfspyxOUhgmqdcf3MhNOqDKcCZyWu7nGjLIRHWDHUkkT1OFkfuyUnFulT+JU2ZKGzNXfExOaaD1OItuZUDPUy95M/M/r5Ca+CSdcZrlByRaL4lwQk5LZ56TPFTIjxpZQpri9lbAhVZQZm0/ZhuAvv7xKmjXXv3RrD1fVeq2IowSncAYX4MM11OEeGhAAAw7P8ApvjnRenHfnY9G65hQzJ/AHzucPooaN3A==</latexit>
1 = 0.01 <latexit sha1_base64="bloHzXcIQny3d2FBJejCRR0ISgE=">AAAB9XicbVDLSgMxFL1TX7W+qi7dBIvgqsxUQTdC0Y3LCvYB7VgymUwbmkmGJKOUof/hxoUibv0Xd/6NaTsLbT0QcjjnXHJzgoQzbVz32ymsrK6tbxQ3S1vbO7t75f2DlpapIrRJJJeqE2BNORO0aZjhtJMoiuOA03Ywupn67UeqNJPi3owT6sd4IFjECDZWeuhxGw1x37tyq67XL1fsNQNaJl5OKpCj0S9/9UJJ0pgKQzjWuuu5ifEzrAwjnE5KvVTTBJMRHtCupQLHVPvZbOsJOrFKiCKp7BEGzdTfExmOtR7HgU3G2Az1ojcV//O6qYku/YyJJDVUkPlDUcqRkWhaAQqZosTwsSWYKGZ3RWSIFSbGFlWyJXiLX14mrVrVO6vW7s4r9eu8jiIcwTGcggcXUIdbaEATCCh4hld4c56cF+fd+ZhHC04+cwh/4Hz+ALGUkVM=</latexit>
1 = 0.2 <latexit sha1_base64="XVlFqVG7AxVj1iPPG+FjbbITpFY=">AAAB9HicbVDLSsNAFL2pr1pfVZduBovgKiRV0I1QdOOygn1AG8pkMmmHTiZxZlIood/hxoUibv0Yd/6N0zQLbT0wcDjnXO6d4yecKe0431ZpbX1jc6u8XdnZ3ds/qB4etVWcSkJbJOax7PpYUc4EbWmmOe0mkuLI57Tjj+/mfmdCpWKxeNTThHoRHgoWMoK1kbw+N9EAD9wbx64PqjXHdnKgVeIWpAYFmoPqVz+ISRpRoQnHSvVcJ9FehqVmhNNZpZ8qmmAyxkPaM1TgiCovy4+eoTOjBCiMpXlCo1z9PZHhSKlp5JtkhPVILXtz8T+vl+rw2suYSFJNBVksClOOdIzmDaCASUo0nxqCiWTmVkRGWGKiTU8VU4K7/OVV0q7b7oVdf7isNW6LOspwAqdwDi5cQQPuoQktIPAEz/AKb9bEerHerY9FtGQVM8fwB9bnD0FEkRo=</latexit>
1 = 0.05 <latexit sha1_base64="JaxMZUeaAnr+QIWpkAvnRdRwED4=">AAAB9XicbVDLSgMxFL3js9ZX1aWbYBFclZmq6EYounFZwT6gHUsmk2lDM8mQZJQy9D/cuFDErf/izr8xbWehrQdCDuecS25OkHCmjet+O0vLK6tr64WN4ubW9s5uaW+/qWWqCG0QyaVqB1hTzgRtGGY4bSeK4jjgtBUMbyZ+65EqzaS4N6OE+jHuCxYxgo2VHrrcRkPc867cinveK5XtNQVaJF5OypCj3it9dUNJ0pgKQzjWuuO5ifEzrAwjnI6L3VTTBJMh7tOOpQLHVPvZdOsxOrZKiCKp7BEGTdXfExmOtR7FgU3G2Az0vDcR//M6qYku/YyJJDVUkNlDUcqRkWhSAQqZosTwkSWYKGZ3RWSAFSbGFlW0JXjzX14kzWrFO61U787Kteu8jgIcwhGcgAcXUINbqEMDCCh4hld4c56cF+fd+ZhFl5x85gD+wPn8AbekkVc=</latexit>
1 = 0.01 <latexit sha1_base64="bloHzXcIQny3d2FBJejCRR0ISgE=">AAAB9XicbVDLSgMxFL1TX7W+qi7dBIvgqsxUQTdC0Y3LCvYB7VgymUwbmkmGJKOUof/hxoUibv0Xd/6NaTsLbT0QcjjnXHJzgoQzbVz32ymsrK6tbxQ3S1vbO7t75f2DlpapIrRJJJeqE2BNORO0aZjhtJMoiuOA03Ywupn67UeqNJPi3owT6sd4IFjECDZWeuhxGw1x37tyq67XL1fsNQNaJl5OKpCj0S9/9UJJ0pgKQzjWuuu5ifEzrAwjnE5KvVTTBJMRHtCupQLHVPvZbOsJOrFKiCKp7BEGzdTfExmOtR7HgU3G2Az1ojcV//O6qYku/YyJJDVUkPlDUcqRkWhaAQqZosTwsSWYKGZ3RWSIFSbGFlWyJXiLX14mrVrVO6vW7s4r9eu8jiIcwTGcggcXUIdbaEATCCh4hld4c56cF+fd+ZhHC04+cwh/4Hz+ALGUkVM=</latexit> 1 = 0.1
<latexit sha1_base64="syeDM2EVxd5nOmJkFAh9HY/R1JA=">AAAB9HicbVDLSsNAFL2pr1pfVZduBovgKiRV0I1QdOOygn1AG8pkMmmHTiZxZlIood/hxoUibv0Yd/6N0zQLbT0wcDjnXO6d4yecKe0431ZpbX1jc6u8XdnZ3ds/qB4etVWcSkJbJOax7PpYUc4EbWmmOe0mkuLI57Tjj+/mfmdCpWKxeNTThHoRHgoWMoK1kbw+N9EAD9wbx3YH1ZpjOznQKnELUoMCzUH1qx/EJI2o0IRjpXquk2gvw1Izwums0k8VTTAZ4yHtGSpwRJWX5UfP0JlRAhTG0jyhUa7+nshwpNQ08k0ywnqklr25+J/XS3V47WVMJKmmgiwWhSlHOkbzBlDAJCWaTw3BRDJzKyIjLDHRpqeKKcFd/vIqaddt98KuP1zWGrdFHWU4gVM4BxeuoAH30IQWEHiCZ3iFN2tivVjv1sciWrKKmWP4A+vzBz/AkRk=</latexit>
= 0.1 <latexit sha1_base64="lQ3dfPYIjt1Rujnm9oRIE6ahP7g=">AAAB8HicbVBNS8NAEJ34WetX1aOXxSJ4CkkV9CIUvXisYD+kDWWznbRLN5uwuxFK6a/w4kERr/4cb/4bt20O2vpg4PHeDDPzwlRwbTzv21lZXVvf2CxsFbd3dvf2SweHDZ1kimGdJSJRrZBqFFxi3XAjsJUqpHEosBkOb6d+8wmV5ol8MKMUg5j2JY84o8ZKj50QDb32XL9bKnuuNwNZJn5OypCj1i19dXoJy2KUhgmqddv3UhOMqTKcCZwUO5nGlLIh7WPbUklj1MF4dvCEnFqlR6JE2ZKGzNTfE2Maaz2KQ9sZUzPQi95U/M9rZya6CsZcpplByeaLokwQk5Dp96THFTIjRpZQpri9lbABVZQZm1HRhuAvvrxMGhXXP3cr9xfl6k0eRwGO4QTOwIdLqMId1KAODGJ4hld4c5Tz4rw7H/PWFSefOYI/cD5/AJdTj5w=</latexit>
p = 0.2 <latexit sha1_base64="WwDnrEuQH6Z7JXaD8hcdJoK6m0s=">AAAB7HicbVBNS8NAEJ3Ur1q/qh69LBbBU0iqoBeh6MVjBdMW2lA22027dLNZdjdCCf0NXjwo4tUf5M1/47bNQVsfDDzem2FmXiQ508bzvp3S2vrG5lZ5u7Kzu7d/UD08auk0U4QGJOWp6kRYU84EDQwznHakojiJOG1H47uZ336iSrNUPJqJpGGCh4LFjGBjpUDeeG69X615rjcHWiV+QWpQoNmvfvUGKckSKgzhWOuu70kT5lgZRjidVnqZphKTMR7SrqUCJ1SH+fzYKTqzygDFqbIlDJqrvydynGg9SSLbmWAz0sveTPzP62Ymvg5zJmRmqCCLRXHGkUnR7HM0YIoSwyeWYKKYvRWREVaYGJtPxYbgL7+8Slp1179w6w+XtcZtEUcZTuAUzsGHK2jAPTQhAAIMnuEV3hzhvDjvzseiteQUM8fwB87nD6jaje0=</latexit>
= 0.4 <latexit sha1_base64="22+9GVtaNf2f2jPffZehEOFqgDM=">AAAB8HicbVBNSwMxEJ2tX7V+VT16CRbB07JbC3oRil48VrAf0i4lm2bb0CS7JFmhLP0VXjwo4tWf481/Y9ruQVsfDDzem2FmXphwpo3nfTuFtfWNza3idmlnd2//oHx41NJxqghtkpjHqhNiTTmTtGmY4bSTKIpFyGk7HN/O/PYTVZrF8sFMEhoIPJQsYgQbKz32QmrwtefW+uWK53pzoFXi56QCORr98ldvEJNUUGkIx1p3fS8xQYaVYYTTaamXappgMsZD2rVUYkF1kM0PnqIzqwxQFCtb0qC5+nsiw0LriQhtp8BmpJe9mfif101NdBVkTCapoZIsFkUpRyZGs+/RgClKDJ9Ygoli9lZERlhhYmxGJRuCv/zyKmlVXf/Crd7XKvWbPI4inMApnIMPl1CHO2hAEwgIeIZXeHOU8+K8Ox+L1oKTzxzDHzifP5vfj58=</latexit>
p = 0.5 <latexit sha1_base64="NcRyqU1q7HvM5CG6xRdiEyR/sWI=">AAAB7HicbVBNS8NAEJ31s9avqkcvi0XwFJKq6EUoevFYwbSFNpTNdtMu3WzC7kYoob/BiwdFvPqDvPlv3LY5aOuDgcd7M8zMC1PBtXHdb7Syura+sVnaKm/v7O7tVw4OmzrJFGU+TUSi2iHRTHDJfMONYO1UMRKHgrXC0d3Ubz0xpXkiH804ZUFMBpJHnBJjJT+9cZ3LXqXqOu4MeJl4BalCgUav8tXtJzSLmTRUEK07npuaICfKcCrYpNzNNEsJHZEB61gqScx0kM+OneBTq/RxlChb0uCZ+nsiJ7HW4zi0nTExQ73oTcX/vE5mousg5zLNDJN0vijKBDYJnn6O+1wxasTYEkIVt7diOiSKUGPzKdsQvMWXl0mz5njnTu3holq/LeIowTGcwBl4cAV1uIcG+ECBwzO8whuS6AW9o4956woqZo7gD9DnD61mjfA=</latexit>
Figure H.1: Generation-reconstruction trade-off of various text autoencoders on Yahoo. The “real data” line marks the PPL of a language model trained and evaluated on real data. We strive to approach the lower right corner with both high BLEU and low PPL. The grey box identifies hyperparameters we use for respective models in subsequent experiments. Points of severe collapse (Reverse PPL > 300) are removed from the right panel.

heading: I NEIGHBORHOOD PRESERVATION

heading: J ADDITIONAL EXAMPLES
text: Source how many gospels are there that were n’t included in the bible ?
5-NN by AAE there are no other gospels that were n’t included in the bible . how many permutations are there for the letters in the word _UNK ’ ? anyone else picked up any of the _UNK in the film ? what ’s the significance of the number 40 in the bible ? how many pieces of ribbon were used in the _UNK act ?
5-NN by DAAE there are no other gospels that were n’t included in the bible . how many litres of water is there in the sea ? how many _UNK gods are there in the classroom ? how many pieces of ribbon were used in the _UNK act ? how many times have you been grounded in the last year ?
Source how do i change colors in new yahoo mail beta ?
5-NN by AAE how should you present yourself at a _UNK speaking exam ? how can i learn to be a hip hop producer ? how can i create a _UNK web on the internet ? how can i change my _UNK for female not male ? what should you look for in buying your first cello ?
5-NN by DAAE how do i change that back to english ? is it possible to _UNK a yahoo account ? how do i change my yahoo toolbar options ? what should you look for in buying your first cello ? who do you think should go number one in the baseball fantasy draft , pujols or _UNK ?
Table J.1: Examples of nearest neighbors in the latent Euclidean space of AAE and DAAE on Yahoo dataset.
Input the staff is rude and the dr. does not spend time with you . slow service , the food tasted like last night ’s leftovers . ARAE the staff is rude and the dr. does not worth two with you . slow service , the food tasted like last night ’s leftovers . β-VAE the staff was rude and the dr. did not spend time with your attitude . slow service , the food tastes like last place serves . AAE the staff was rude and the dr. does not spend time with you . slow service , the food tasted like last night ’s leftovers . LAAE the staff was rude and the dr. is even for another of her entertained . slow service , the food , on this burger spot ! DAAE the staff was rude and the dr. did not make time with you . slow service , the food tastes like last night ... .
Input they are the worst credit union in arizona . i reported this twice and nothing was done . ARAE they are the worst bank credit in arizona . i swear this twice and nothing was done . β-VAE they were the worst credit union in my book . i ’ve gone here and nothing too . AAE they are the worst credit union in arizona . i reported this twice and nothing was done . LAAE they were the worst credit union in my heart . i dislike this twice so pleasant guy . DAAE they were the worst credit union in arizona ever . i hate this pizza and nothing done .
Table J.2: Additional examples of vector arithmetic for tense inversion.
AAE DAAE
Input this woman was extremely rude to me . this woman was extremely rude to me . +v this woman was extremely rude to me . this woman was extremely nice . +1.5v this woman was extremely rude to baby . this staff was amazing . +2v this woman was extremely rude to muffins . this staff is amazing .
Input my boyfriend said his pizza was basic and bland also . my boyfriend said his pizza was basic and bland also . +v my boyfriend said his pizza was basic and tasty also . my boyfriend said his pizza is also excellent . +1.5v my shared said friday pizza was basic and tasty also . my boyfriend and pizza is excellent also . +2v my shared got pizza pasta was basic and tasty also . my smoked pizza is excellent and also exceptional .
Input the stew is quite inexpensive and very tasty . the stew is quite inexpensive and very tasty . −v the stew is quite inexpensive and very tasty . the stew is quite an inexpensive and very large . −1.5v the stew is quite inexpensive and very very tasteless . the stew is quite a bit overpriced and very fairly brown . −2v the – was being slow - very very tasteless . the hostess was quite impossible in an expensive and very few customers .
Input the patrons all looked happy and relaxed . the patrons all looked happy and relaxed . −v the patrons all looked happy and relaxed . the patrons all helped us were happy and relaxed . −1.5v the patrons all just happy and smelled . the patrons that all seemed around and left very stressed . −2v the patrons all just happy and smelled . the patrons actually kept us all looked long and was annoyed .
Table J.3: Additional examples of vector arithmetic for sentiment transfer.
Input 1 what language should i learn to be more competitive in today ’s global culture ? Input 2 what languages do you speak ?
AAE what language should i learn to be more competitive in today ’s global culture ? what language should i learn to be more competitive in today ’s global culture ? what language should you speak ? what languages do you speak ? what languages do you speak ?
DAAE what language should i learn to be more competitive in today ’s global culture ? what language should i learn to be competitive today in arabic ’s culture ? what languages do you learn to be english culture ? what languages do you learn ? what languages do you speak ?
Input 1 i believe angels exist . Input 2 if you were a character from a movie , who would it be and why ?
AAE i believe angels exist . i believe angels - there was the exist exist . i believe in tsunami romeo or <unk> i think would it exist as the world population . if you were a character from me in this , would we it be ( why ! if you were a character from a movie , who would it be and why ?
DAAE i believe angels exist . i believe angels exist in the evolution . what did <unk> worship by in <unk> universe ? if you were your character from a bible , it will be why ? if you were a character from a movie , who would it be and why ?
Table J.4: Interpolations between two input sentences generated by AAE and our model on the Yahoo dataset.

Review summaries:
The paper "Denoising Improves Latent Space Geometry in Text Autoencoders" tackles the problem of text autoencoding in a space which respects text similarities. It is an interesting problem for which various attempts have been proposed, while still facing difficulties for encoding in smooth spaces. The paper proposes a simple (rather straightforward) approach based on adversarial learning, with some theoretical guarantees, which obtains good performances for reconstruction and neighborhood preservation. 

This paper presented a denoising adversarial autoencoder for sentence embeddings. The idea is that by introducing perturbations (word omissions, etc) the embeddings are more meaningful and less "memorized". Evaluations include measuring sentence perplexity in generation/reconstruction, tense changing via vector arithmetic, sentiment changes via negative/positive vector additions, and sentence interpolations.