Denoising Improves Latent Space Geometry in Text Autoencoders
Authors: Tianxiao Shen
Jonas Mueller
Regina Barzilay
Tommi Jaakkola
Venue:  (2020)
Keywords: controllable text generation
autoencoders
denoising
latent space geometry

Abstract: Neural language models have recently shown impressive gains in unconditional text generation, but controllable generation and manipulation of text remain challenging. In particular, controlling text via latent space operations in autoencoders has been difficult, in part due to chaotic latent space geometry. We propose to employ adversarial autoencoders together with denoising (referred as DAAE) to drive the latent space to organize itself. Theoretically, we prove that input sentence perturbations in the denoising approach encourage similar sentences to map to similar latent representations. Empirically, we illustrate the trade-off between text-generation and autoencoder-reconstruction capabilities, and our model significantly improves over other autoencoder variants. Even from completely unsupervised training, DAAE can successfully alter the tense/sentiment of sentences via simple latent vector arithmetic.

Under review as a conference paper at ICLR 2020
DENOISING IMPROVES LATENT SPACE GEOMETRY IN TEXT AUTOENCODERS
Anonymous authors Paper under double-blind review
ABSTRACT
Neural language models have recently shown impressive gains in unconditional text generation, but controllable generation and manipulation of text remain challenging. In particular, controlling text via latent space operations in autoencoders has been difficult, in part due chaotic latent space geometry. We propose employ adversarial autoencoders together with denoising (referred as DAAF) to drive the latent space to organize itself. Theoretically, we prove that input sentence perturbations in the denoising approach encourage similar sentences to map to similar latent representations. Empirically, we illustrate the trade-off between text- capabilities, and our model significantly
generation and cuttoenncderr-econstttutttte improves over other autoencoder variants. Even from completely unsupervised training without style information, DAAE can perform various style transfers, including tense and sentiment, through simple latent vector arithmetic.!
INTRODUCTION
Autoencoder based generative models have recently become popular tools for advancing controllable text generation such as style or sentiment transfer (Bowman al., 2016; Hu et al., 2017; Shen et al., 2017; Zhao et al., 2018). By mapping sentences to vectors in the latent space, these models offer in principle an attractive, continuous approach manipulating text by means of simple latent vector arithmetic. However, the success of such manipulations rests heavily on the latent space geometry and how well it agrees with underlying sentence semantics. Indeed, we demonstrate that without additional guidance, fortuitous geometric agreements are unlikely to arise, shedding light on challenges faced by existing methods.
We adversarial autoencoders (Makhzani al., 2015, AAEs) study the latent space geometry. In contrast to variational autoencoders (Kingma Welling, 2014. VAEs), AAEs can maintain strong coupling between the encoder and decoder that the decoder does not omit the encoded input sentence (Bowman et al., 2016). The training criterion for AAEs consists of two parts, the ability to reconstruct sentences and the additional constraint that the encoded sentences overall indistinguishable from prior samples, typically Gaussian. We show that these objectives alone do suffice force proper latent space geometry for text control. Specifically, for discrete objects such as sentences where continuity assumptions no longer hold, powerful AAEs can easily learn to map training sentences into latent prior samples arbitrarily (Figure 1, Left), while retaining perfect reconstruction. Latent space manipulations in such cases will yield random, unpredictable results.
To remedy this, we augment AAEs with a simple denoising objective (Vincent al., 2008; Creswell & Bharath, 2018) that requires perturbed sentence with some words missing to be mapped back to the original version. We refer to our model as DAAE. We prove that the denoising criterion can eliminate disorganized solutions and drive the latent space to organize itself. As a result, similar sentences begin to be mapped to similar latent vectors (Figure 1, Right).
Improvements in latent space geometry carry many positive consequences. Through systematic evaluations of the generation and reconstruction capabilities of various text autoencoders (Cifka et al., 2018), find that our proposed DAAE provides the best trade-off between producing high- quality text vs. informative sentence representations. We empirically verify that DAAE has the best neighborhood preservation property, consistent with our theory. We further investigate to what extent
Our code will be made publicly available after the review process.

---

Under review as a conference paper at ICLR 2020
X
Σ
X
E
((2)
E
Figure I: Illustration of the learned latent geometry by AAE before and after introducing. perturbations. With high-capacity encoder/decoder networks, a standard AAE has no preference over x=2 couplings and thus can learn a random mapping between them (Left). Trained with local perturbations C(z), DAAB learns to map similar æ to close to best achieve the denoising objective (Right).
text can be manipulated by applying simple transformations in the learned latent space. Our model is able to perform sentence-level vector arithmetic (Mikolov et al., 2013) fairly well to change the tense sentiment of sentence without any training supervision. It also produces higher quality sentence interpolations than other text autoencoders, suggesting better linguistic continuity in its latent space (Bowman et al., 2016).
RELATED WORK
Denoising is first introduced into standard autoencoders by Vincent al. (2008, DAE) learn robust representations. Without a latent prior, DAE requires sophisticated MCMC sampling to be employed generatively (Bengio al., 2013). Creswell & Bharath (2018) applied denoising with AAES to generative image modeling. Here, we demonstrate that input perturbations are particularly useful for discrete text modeling because they encourage preservation of data structure in the latent space.
Apart from the AAE framework that our paper focuses on, another popular latent variable generative model is the variational autoencoder (Kingma & Welling, 2014, VAE). Unfortunately, when the decoder is a powerful autoregressive model (such as a language model), VAE suffers from the posterior collapse problem where the latent representations get ignored (Bowman et al., 2016; Chen et al., 2016). If denoising is used in conjunction with VAEs (Im et al., 2017) in text applications, then the noisy inputs will only exacerbate VAE's neglect the latent variable. Bowman et al. (2016) proposed weaken VAE's decoder by masking words on the decoder side to alleviate its collapse issue. However, even with a weakened decoder and combined with other techniques including KL-weight annealing and adjusting training dynamics, it is still difficult to inject significant content into the latent code (Yang et al., 2017; Kim et al., 2018; He et al., 2019). Alternatives like the 3-VAE (Higgins et al., 2017) appear necessary.
Previous work on controllable text generation has employed autoencoders trained with attribute label information (Hu al., 2017; Shen et al., 2017: Zhao al., 2018; Logeswaran et al., 2018; Subramanian et al., 2018). We show that the proposed DAAE model can perform text manipulations despite being trained in a completely unsupervised manner without attribute labels. This suggests that on the one hand, our model can be adapted to semi-supervised learning when few labels are available. On the other hand, can be easily scaled up to train one large model on unlabeled text corpora and then applied for transferring various styles.
3 METHOD
Define X' = ym to be space of sequences of discrete symbols from vocabulary V (with maximum length m); also define = Rd to be a continuous latent space. Our goal is to learn a mapping between the data distribution Paata over X' and a given prior distribution p(z) over latent space Z (following common practice, a Gaussian prior is used in our experiments, although not required by methodology). Such mapping allows to easily manipulate discrete data through continuous latent representations 2, and provides a generative model where data samples can be obtained by first drawing from the prior and then sampling corresponding sequence via p(x[2)).
2

---

Under review as a conference paper at ICLR 2020
We adopt the adversarial autoencoder (AAE) framework, which involves (deterministic) encoder E X' + 2, a probabilistiic decoder G : 2 + X, and discriminator Z + [0, 1]. Both E and G' are recurrent neural networks (RNNs)?. E' takes input sequence z and outputs the last hidden state as its encoding& 2. G generates a sequence x: autoregressively, with each step conditioned on 2 and previous symbols. The discriminator D is feed-forward net that outputs the probability of coming from the prior rather than the encoder. E, G and D are trained jointly with a min-max objective:
min max Lre (0 0G) (f, E.G D
(1)
with: Lice (0, 0c) = p loggo (o)
(2)
Lad (0 0p) Ep(a) log [=))] + TE pen log(1 - ((E())))
(3)
where reconstruction loss fee and adversarial loss Ladv are weighted via hyperparameter À 0.
We further introduce perturbations in X' space to learn smoother representations that reflect local structure in the data, ending up with the denoising adversarial autoencoder (DAAE) model. Given perturbation process C' that stochastically maps x to nearby ā E X', let p(æ, .) = Pasa (z)c(() and p(.) () = Eg p(x, 8). We change the loss functions to be:
(0g, 0g) = Ep(a,a) log (x) E())]
(4)
(0 0D) = Eptn)- log Eps -ogg D(E()))
(5)
Here, Free is the loss of reconstructing x from ã, and Latv is the adversarial loss evaluated on perturbed x. The objective function combines the denoising technique with the AAE (Vincent et al., 2008; Creswell & Bharath, 2018). When pc = [ x| (i.e. there is no perturbation), the above simply becomes the usual AAE objective.
Let (z(2zz) denote the encoder distribution. With our perturbation process C, the posterior distribu- tions of the DAAE are of the following form:
(<1|z) Σ pe pc(2)z)p=(2li)
(6)
This enables the DAAE to utilize stochastic encodings even by merely employing a deterministic encoder network trained without any reparameterizattionssyy tricks. Note that since q(=|ær) of the form (6) is a subset of all possible conditional distributions, our model is still minimizing an upper bound of the Wasserstein distance between data and model distributions, as previously shown by Tolstikhin et al. (2017) for AAE (see Appendix A for full proof).
4 LATENT SPACE GEOMETRY
The latent space geometry of text autoencoders is an important yet understudied problem. Only when the latent space is smooth and regular meaningful text manipulations be enacted via simple modifications of the corresponding latent representations. Here, we discuss in detail the posterior characteristics of the DAAE, and provide a theoretical analysis of how input perturbations help better structure the latent space geometry (all proofs are relegated to the appendix).
Assume perturbations preserve with some probability (i.e. po(x))r) > 0). When the support of C(I1) and C(z2) do not overlap for different training examples æ1 # I2. the encoder can learn to assign (=(²)
(=(x(x) for E C(x), and we are back to the unconstrained posterior scenario q<>[x)) = Pe a|x) (Eq. 6). If ((z1) and C(x2) do intersect, then the latent posterior of x1 and z2 will have overlapping components PE (. for à E C(xzg) For example, if pe (ā(ær) assigns a high probability to . that lies close to x (based on some metric over X)), then for similar xi and 222 the high-probability overlap between their perturbations will inherently force their posteriors closer together in the latent space. This is desirable for learning good representations z, while not guaranteed by merely minimizing the statistical divergence between Pdata (z) and PG(x) = Ep(z) (P)(2)))
Now we formally analyze which type of z-z mappings will be learned by AAE and DAAE, re- spectively, to achieve global optimality of their training objectives. Unlike previous analyses of
2 Transformer models (Vaswani et al., 2017) did not outperform LSTMs on our moderately-sized datasets. We actually train E to maximize log D (FF())) instead of log(1 D)F(x))), which is more stable in practice (Goodfellow et al., 2014). We also tried WGAN (Arjovsky al., 2017) but did notice any gains.
3

---

Under review as a conference paper at ICLR 2020
noise in single-layer networks (Poole et al., 2014), here we study high-capacity encoder/decoder networks (Schäfer & Zimmermann, 2006) with a large number of parameters that are used in modern sequence models (Devlin et al., 2018; Radford et al., 2019). Throughout, we assume that:
Assumption 1. E is a universal approximator capable of producing any mapping from x' e's to 2 z's. Assumption 2. G can approximate arbitrary p(x[zz so long as it remains sufficiently Lipschitz continuous in z, Namely, there exists L > 0 such that all decoder models G obtainable via training satisfy that for all I E X, 21, 22 E 2: log (2| - log Pc S L||z1 22Il-
Following prior analysis of language decoders (Mueller et al., 2017), we assume that G is L-Lipschitz in its continuous input (denote this set of possible decoders by GL). When is implemented as a RNN or Transformer language model, log Pc 22) will remain Lipschitz in if the recurrent or attention weight matrices have bounded norm. This property is naturally encouraged by popular training methods that utilize SGD with early stopping and L2 regularization (Zhang al., 2017). Note we have not assumed E or G is Lipschitz in v, which would be unreasonable since x stands for discrete text, and when few symbols change, the decoder likelihood for the entire sequence can vary drastically (e.g.. G may assign a much higher probability to grammatical sentence than an ungrammatical one that only differs by one word). Our discussion directed to the nature of such families of oggllikel functions with a continuous variable and discrete variable x.
We further assume an effectively trained discriminator that succeeds in its adversarial task: Assumption 3. D ensures that the latent encodings "1,"" Z of training examples Ti," Tn are indistinguishable from prior samples p(z).
For simplicity. we directly assume that 21. .... ; 2n are actual samples from p(z) which are given a priori. Here, the task of the encoder E is to map given unique training examples to the given latent points, and the goal of the decoder PG (-I-) is to maximize -fre: under the encoder mapping. The question now is which one-to-one mapping optimal encoder/decoder will learn under the AAE and DAAE objective (Eq. 2 and Eq. 4). We start with the following observation:
Theorem 1. For any one-to-one encoder mapping E from {z1.... .nn) to {21, ;, the optimal value of objective maxcess A Ei= log pG (2; [((x)) is the same.
Intuitively, this result stems from the fact that the model receives no information about the structure of æ, and ,,, Fr are simply provided different symbols. Hence AAE offers no preference over z-z couplings, and a random matching in which the = do not reflect any data structure is equally good as any other matching (Figure 1, Left). Latent point assignments start to differentiate, however, once we introduce local input perturbations.
To elucidate how perturbations affect latent space geometry, helps to first consider a simple setting with only four examples I 22 I3, I4 E X. Again, we consider given latent points 21, Z2, 23, 24 sampled from p(2), and the encoder/decoder are tasked with learning which x to match with which 2. As depicted in Figure 1, suppose there are two pairs of x closer together and also two pairs of closer together. Let a denote the sigmoid function, we have the following conclusion:
Theorem 2. Let d be a distance metric over X. Suppose I1, I2, I3, I4 satisfy that with some e, d(x3,44) < 6, and d(z;, xz)) > e for all other (Zi,xz) pairs. In addition, 21, k2, 23,32, 24 satisfy that with some 0 < 8 < Ç: || - =2ell < 6, 1|zg- -zall < ô, and [|z;- =zzll zyll > Ç for all other (22, 22) pairs. Suppose our perturbation process C reflects local X geometry with:
e >> O: d(x1,
(x,(xx) = 1/2 if d(x,, < and = 0 otherwise. For d < 1 (2 log (o(L)) + log 2) and Ç > L log (1/(v2 - 1)). the denoising objective maxgeş, 1 E Σド= E (z,) log pa (E(x,) (where n = 4) achieves the largest value when encoder E maps close pairs of x to close pairs of z.
1
This entails that DAAE will always prefer to map similar x to similar 2. Note that Theorem I still applies here, and AAE will not prefer any particular z-2 pairing over the other possibilities. We next generalize beyond the basic four- oour-points scenario to consider n examples of x that are clustered. Here, we can ask whether this cluster organization will be reflected in the latent space of DAAE.
Theorem 3. Suppose "i,'." ITn are divided into n/ K clusters of equal size K, with St denoting the cluster index of vi. Let the perturbation process C be uniform within clusters, i.e. po(a;(u;)) = 1/K if S. = Sj and = O otherwise. For a one-to-one encoder mapping E from {ar,---, ,nn In}
to -zi,.... 3. the denoising objective maxGess 1 E Ej=1 Po(x (x:) log po (x)])((xx)) is upper bounded by: 1 Eij.s.s, log [T. (((,))) F(xr))]) log K.
4

---

Under review as a conference paper at ICLR 2020
Theorem provides upper bound of the DAAE objective that can be achieved by particular 2-2 mapping. This achievable limit is substantially better when examples in the same cluster are mapped to the latent space in manner that is well-separated from encodings of other clusters. In other words, by preserving input space cluster structure in the latent space, DAAE can achieve better objective values and thus is incentivized to learn such encoder/decoder mappings. An analogous corollary can be shown for the case when examples z are perturbed to yield additional inputs I not present in the training data. In this case, the model would aim map each example and its perturbations as group to compact group of points well-separated from other groups in the latent space.
In conclusion, our analysis shows that a well-traines DAAE guaranteed to learn neighborhood- preserving latent representations, whereas even a perfectly-trainee AAE model may learn latent representations whose geometry fails to reflect similarity in the æ space. Empirical experiments in Section 5.2 confirm that our theory holds in practice.
5 EXPERIMENTS
We evaluate our proposed model and other text autoencoders on two benchmark datasets: Yelp reviews and Yahoo answers (Shen al., 2017; Yang et al., 2017). Detailed descriptions of datasets, training settings, human evaluations, and additional results/examples can be found in the appendix.
Perturbation Process We randomly delete each word with probability p. so that perturbations of sentences with more words in common will have a larger overlap. We also tried replacing each word with a <mask> token or a random word and found that they all brought improvements, but deleting words worked best. We leave it to future work to explore more sophisticated text perturbations.
Baselines We compare our proposed DAAE with four alternative text autoencoders: adversarially reg- ularized autoencoder (Zhao et al., 2018, ARAE), 3- B-VAE (Higgins et al., 2017), AAE (Makhzani et al., 2015), and notentgnissing AAE (Rubenstein et al., 2018, LAAE). Similar to our model, the LAAE uses Gaussian perturbations in the latent space to improve AAE's latent geometry (rather than perturbations in the sentence space). However, LAAE requires enforcing an La penalty (A) || log (3(x))
(x+)]l) II on the latent perturbations' log-variance to prevent them from vanishing. In contrast, input perturbations in DAAE enable stochastic latent representations without parametric restrictions like Gaussianity.
5.1 GENVERATIONRRECONSTUUUITITI
RECONSTRUCTION TRADE-OFF
We evaluate various latent variable generative models in terms of both generation quality and reconstruction accuracy. A strong model should not only generate high quality sentences, but also learn useful latent variables that capture significant data content. Recent work on text autoencoders has found an inherent tension between these aims (Bowman al., 2016; Cifka et al., 2018), yet only when both goals are met can we successfully manipulate sentences by modifying their latent representation (in order to produce valid output sentences that retain the semanties of the input).
We compute the BLEU score (Papineni et al., 2002) between input and reconstructed sentences to measure reconstruction accuracy, and compute Forward/Reverse PPL to measure sentence generation quality (Zhao et al., 2018; Cífka et al., 2018).* 4 Forward PPI. is the perplexity of a language model trained on real data and evaluated on generated data. It measures the fluency of the generated text, but cannot detect the collapsed case where the model repeatedly generates a few common sentences. Reverse PPL is the perplexity of a language model trained on generated data and evaluated on real data. It takes into account both the fluency and diversity of the generated text. If model generates only a few common sentences, language model trained on it will exhibit poor PPL on real data.
We thoroughly investigate the performance of different models and their trade-off between generation and reconstruction. Figure 2 plots reconstruction BLEU (higher better) vs. Forward/Reverse PPL (lower is better). The lower right corner indicates ideal situation where good reconstruction accuracy and generation quality are both achieved. For models with tunable hyperparameters, we sweep the full spectrum of their inneration-reconstuueiin of B- VAE, the oog-variance L1 penalty X1 of LAAE, and the word drop probability p of DAAE.
trade-off by varying the KL coefficient B
4 While some use importance sampling estimates of data likelihood to evaluate VAEs (He et al., 2019). adopting the encoder as proposal density is not suited for AAE variants, as they are optimized hased on Wasserstein distances rather than likelihoods and lack closed-form posteriors.
5

---

Under review as a conference paper at ICLR 2020
real data
#=
0.2
400
90
350
PAVAE
80
3 0.0
À 0.01
Forward PPL
LAKE
neverse IYL
70
250
DAAE
À 0.01
00
200
A 0.05
8
50
150
0.3
0.1
100
50
0.2
1
%
20
20
40
60
30 100
20
40
60
80
100
istruction
Reconstruction
Figure 2: Generation- trade-off of various text autoencoders Yelp. The "real data" line marks the PPL. of language model trained and evaluated real data. We strive approach the lower right corner with both high BLEU and low PPL. The grey box identifies hyperparameters we use for respective models in subsequent experiments. Points of severe collapse (Reverse PPL > 200) are removed from the right pancl.
In the left panel, we observe that a standard VAE (3 1) completely collapses and ignores the latent variable 2, resulting in reconstruction BLEU close 0. At the other extreme, AAE achieve near-perfect reconstruction, but its latent space is highly non-smooth and generated sentences are of poor quality, indicated by its large Forward PPL. Decreasing / in VAE introducing latent noises in AAE provides the model with a similar trade-off curve between reconstruction and generation. We note that ARAE falls on above their curves, revealing that it does not fare better than these methods (Cifka et al. (2018) also reported similar findings). Our proposed DAAE provides a trade-off' curve that is strictly superior other models. With discrete æ and a complex encoder, the Gaussian perturbations added to the latent space by 3- 3-VAE and LAAE are not directly related to how the inputs are encoded. In contrast, input perturbations added by DAAE can constrain the encoder to maintain coherence between neighboring inputs in an end-to-end fashion and help learn smoother latent space.
The right panel in Figure illustrates that Reverse PPL first drops and then rises as we increase the degree of rggularization/perturaaiooo This is because when encodes little information, generations from prior-samppl sampldd z lack enough diversity to cover the real data. Again, DAAE outperforms the other models which tend to have higher Reverse PPL and lower reconstruction BLEU. In subsequent experiments, we set 3 = 0.15 for VAE,,, A1 = 0.05 for LAAE, and p = 0.3 for DAAE, to ensure they have strong reconstruction abilities and encode enough information enable text manipulations.
5.2 NEIGHBORHOOD PRESERVATION
In this section, we empirically investigate whether our previous theory holds in practice. That is, in actual autoencoder models trained on real text datasets, do sentence perturbations induce latent space organization that better preserves neighbor- hood structure in the data space?
B-VAE
LAAE
DAME
30
Recail
25
Under our word-drop perturbation process, sen- tences with more words in common are more likely to be perturbed into another. This choice of approximately encodes sentence similarity via the normalized edit distance Within the test set, we find both the 10 nearest neighbors of each sentence based on the normalized edit distance (denote this set by NN,), as well as the & nearest neighbors based on Euclidean distance between la- tent representations (denote this set by NN.). We
20
15-
10 20
50
300
Figure 3: Recall rate of 10 nearest neighbors in the sentence space retrieved by nearest neighbors in the latent space on Yelp. ARÁE not plotted here as we find its recall significantly below other models (< 1%).
Normalized edit distance [0, 1] is the Levenshtein distance divided the length of two sentences.

---

Under review as a conference paper at ICLR 2020
AAE
DAAE
Source
my waitress katie was fantuxtic, , attentive and personable.
my waitress katie was fantastic attentive and personable. manager, , linda, , was very very ittentive personuble. very friendly. attemtive and professional.
my cashier smile. harely hello
the service fantastic, the is great...
stylist
the employees extremely nice and helpful.
the manager also super nice and personable.
server kaidyy also attentive and pleasant..
my server alicia
sweet and attentive.
the crab buy was also bland and forgettable.
our waitress ms. tayler was amazing and very knowledgeable..
Source have been known
meals day
have
to eat meals a day
have
for num_ years and never had had meal
you seriously
meal day here.
love this joint.
was really pleased with our experience here
have no desire to ever have again
ive bocn coming here for years and always have a good experience.
you da n't need to have every possible dish on the menu
have gone to this place for happy hour for years..
love prena.
we pum, ayce dinner buffets for nom mesday night
Table Examples of nearest neighbors in the latent Euclidean space of AAE and DAAE on the Yelp dataset.
Model
ACC
BLEU
PPL
Model
ACC
BLEU
PPL
ARAE
17.2
55.7
59.1
Shen al. (2017)
81.7
12.4
38.4
8-VAE
49.0
43.5
44.4
tv
7.2
86.0
33.7
AAE
9.7
82.2
37.4
AAE
11.5v
25.1
59.6
59.5
LAAE
43.6
37.5
55.8
+.2v
57.5
27.4
139.8
DAAE
50.3
54.3
32.0
EV
36.2
40.9
40.0
DAAE
+1.50
73.6
18.2
54.1
+22
91.8
7.3
61.8
B-VAE is better: 25
DAAE is better: 48
both good: 26
both bad: 67 n/a: 34
Table 3: Automatic evaluations of vector arithmetic for sentiment transfer. Accuracy (ACC) is measured by a sentiment classifier. The model of Shen et al. (2017) is specifically trained for sentiment transfer with labeled data, while our text autoencoders are not.
Table 2: Above: automatic evaluations of vector arith- metic for tense inversion. Below: human evaluation statistics of our model vs. the closest baseline 3- 3-VAE.
compute the recall rate [NN, n NNN [NN,]. which indicates how well local neighborhoods are preserved in the latent space of different models.
Figures 3 shows that DAAE consistently gives the highest recall, about 1.5~~2 times that of AAE, implying that input perturbations have substantial effect shaping the latent space geometry. Tables presents the five nearest neighbors found by AAE and DAAE in their latent space for example test set sentences. The AAE sometimes encodes entirely unrelated sentences close together, while the latent space geometry of the DAAE is structured based on key words such as "attentive" and "personable", and tends group sentences with similar semantics close together.
5.3 APPLICATIONS TO CONTROLLABLE TEXT GENERATION
5.3.1 STYLE TRANSFER VIA VECTOR ARITHMETIC
Mikolov et al. (2013) previously discovered that word embeddings from unsupervised learning can capture linguistic relationships via simple arithmetic. A canonical example is the embedding arithmetic "King" "Man" + "Woman" "Queen". Here, we use the Yelp dataset with tense and sentiment as two example attributes (Hu et al., 2017; Shen et al., 2017) to investigate whether analogous structure emerges in the latent space of our sentence-level models.
Tense We use the Stanford Parser to extract the main verb of sentence and determine the sentence tense based on its part-of-speech tag. We compute single "tense vector" by averaging the latent code separately for 100 past tense sentences and 100 present tense sentences in the dev set, and then calculating the difference between the two. Given a sentence from the test set, we attempt to change its tense from past to present or from present past through simple addition/subbractionn of the tense vector. More precisely, a source sentence x is first encoded to = E(x), and then the tense-modified sentence is produced via G(z + v), where E Rd denotes the fixed tense vector.
7

---

Under review as a conference paper at ICLR 2020
Input
i enjoy hanging their bookah lounge.
had they informed of the charge i would n't have waited
ARAE
enjoy banging out their 25th lounge.
amazing egg of the may actually!
V-VAF
i made out in the hackyard springs salad..
had they help me of the charge would n't have waited.
AAE
i enjoy banging out in their brooklyn lounge.
have they informed me of the charge i would n't have waited.
LAAE
enjoy banging out customized and play.
they are girl the number so would ntt forever.
DAAE
enjoyed hanging out their bookaah lounge...
they informed me the charge have waited.
Table 4: Examples of vector arithmetic for tense inversion.
AAE
DAAB
Input
the food is entirely tasteless and slimy.
the food entirely tasteless and slimy.
food catirely tasteless slimy-
the food is tremendous fresh.
-1.50
the food entirely tasteless and slimy.
the food is sensational and fresh.
2z
the food is entirely and beef..
the food is gigantic.
Input
really love the authentic food
come again.
really the authentic and will come back ugain.
really love the authentic food and will come back again.
really love the authentic food and will come hack again.
-1.5v
really but the authentic food and will come back aguin. really do not like the food and will never come back again. i really did not believe the pretentious service and will never go back.
20
really but the wond food but will never come back again.
Table 5: Examples of vector arithmetic for sentiment transfer.
To quantitatively compare different models, we compute their tense transfer accuracy as measured by the parser, the output BLEU with the input sentence, and output (forward) PPL. evaluated by language model. DAAE achieves the highest accuracy, lowest PPL, and relatively high BLEU (Table 2, Above), indicating that the output sentences produced by our model are more likely to be of high quality and of the proper tense, while remaining similar to the source sentence. A human evaluation on 200 test sentences (100 past and 100 present, details in Appendix G) suggests that DAAE outperforms V-V twice as often as it is outperformed, and our model successfully inverts tense for (48 + 26)/ - 34) = 44.6% of sentences, 13.8% more than 3-VAE (Table 2, Below). Tables 4 and J.2 show the results of adding or subtracting this fixed latent vector offset under different models. DAAE can successfully change "enjoy" to "enjoyed",, or change the subjunctive mood to declarative mood and adjust the word order. Other baselines either fail alter the tense, or undesirably change the semantic meaning of the source sentence (e.g. "enjoy" to "made").
Sentiment Following the same procedure used alter tense, we compute "sentiment vector" v from 100 negative and positive sentences and use it to change the sentiment of test sentences. Table 3 reports the automatic evaluations, and Tables and J.3 show examples generated by AAE and DAAE. Scaling - to +1.5v and =2v, we find that the resulting sentences get more and more positive/negative. However, the PPI. for AAE increases rapidly with this scaling factor, indicating that the sentences become unnatural when their encodings have a large offset. DAAE enjoys much smoother latent space than AAE. Despite the fact that no sentiment labels are provided during training (a more challenging task than previous style transfer models (Shen et al., 2017)), DAAE with 1.5v is able to transfer sentiment fairly well.
5.3.2 SENTENCE INTERPOLATION VIA LATENT SPACE TRAVERSAL
We also study sentence interpolation by traversing the latent space of text autoencoders. Given two input sentences, we encode them to 21, 22 and decode from 121 + (1 l) (0 < S 1) to obtain interpolated sentences. Ideally this should produce fluent sentences with gradual semantic change (Bowman et al., 2016). Table 6 shows two examples from the Yclp dataset, where it is clear that DAAE produces more coherent and natural interpolations than AAE. Table J.4 in the appendix shows two difficult examples from the Yahoo dataset, where we interpolate between dissimilar sentences. While it is challenging to generate semantically correct sentences in these cases, the latent space of our model exhibits continuity on topic and syntactic structure.
6 CONCLUSION
This paper proposed DAAE for generative text modeling. As revealed in previous work (Devlin et al., 2018; Lample et al., 2018), we find that denoising techniques can greatly improve the learned text representations. We provide theoretical explanation for this phenomenon by analyzing the latent
8

---

Under review as a conference paper at ICLR 2020
much better than the other chinese food places this Input better than other places.
fried dumplings
the fried dumplings must if ever visit this place.
places
dumplings are must.
food places
fried dumplings are must.
better, much better
the dumplings must you worst.
better than other places. better than other places.
the fried dumplings must if you ever this place
the fried dumplings must if you ever visit this place.
DAAB
chinese food places much hetter than the other chinese places this better than the other chinese places this area...
much
fried dumpling
fried dumplings
fried dumplings must in this place the fried dumplings are must we over visit this the fried dumplings are must we ever visit this place
better than the other places in charlotic.
hetter than other places.
Table 6: Interpolations between two input sentences generated AAE and model the Yelp dataset.
space geometry arisen from imput perturbations. Our proposed model substantially outperforms other text autoencoders, and demonstrates potential for various text manipulations via vector operations. Future work may investigate superior perturbation strategies and additional properties of latent space geometry to provide finer control over the text generated using autoencoder models.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International Conference on Machine Learning, pp. 214-223, 2017.
Yoshua Bengio, Li Yao, Guillaume Alain, and Pascal Vincent. Generalized denoising auto-encoders as generative models. In Advances in Neural Information Processing Systems, pp. 899-907, 2013.
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from continuous space. In Conference Computational Natural Language Learning, 2016.
Xi Chen, Diederik P Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, and Pieter Abbeel. Variational lossy autoencoder. arXiv preprint arXiv: 1et::11112000 1.02731, 2016.
Ondiej Cifka, Aliaksei Severyn, Enrique Alfonseca, and Katja Filippova. Eval all, trust a few, do wrong to none: Comparing sentence generation models. arXiv preprint arXiv: /804 2018.
Antonia Creswell and Anil Anthony Bharath. Denoising adversarial autoencoders. IEEE tronsactions neural networks and learning systems, (99):1-17, 2018.
Jacob Devlin, Ming-Wai Weig-Wei&o Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- tion processing systems, pp. 2672-2680, 2014.
Junxian He, Daniel Spokoyny, Graham Neubig, and Taylor Berg- Kerg-Kirkpattrck. Lagging inference networks and posterior collapse in variational autoencoders. arXiv preprint urXiv:1911005534 2019.
Trina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations, volume 3, 2017.
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric Xing. Toward controlled generation of text. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1587-1596. JMLR. org, 2017.
Daniel Im Jiwoong Im, Sungjin Ahn, Roland Memisevic, and Yoshua Bengio. Denoising criterion for variational auto-encoding framework. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
Yoon Kim, Sam Wiseman, Andrew C Miller, David Sontag, and Alexander M Rush. Semi-amortized variational autoencoders. arXiv preprint arXiv: /ttl::/8022222220 802.02550, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Mare' Aurelio Ranzato. Phrase- based & neural unsupervised machine translation. arXiv preprint arXiv:1804.0777 07755,7 2018.
Lajanugen Logeswaran, Honglak Lee, and Samy Bengio. Content preserving text generation with attribute controls. In Advances in Neural Information Processing Systems, pp. 5103-5113, 2018.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders. arXiv preprint arXiv: 15/ 2015.
10

---

Under review as a conference paper at ICLR 2020
Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 746-751, 2013.
Jonas Mueller, David Gifford, and Tommi Jaakkola. Sequence better sequence: continuous revision of combinatorial structures. In Proceedings of the 34th International Conference on Machine Learning-Voii Volume 70. pp. 2536 2336-2544. JMLR. org, 2017.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting association for
computational linguistics, pp. 31 311-318. Association for Computational Linguistics, 2002.
Ben Poole, Jascha Sohl-Dickstein, and Surya Ganguli. Analyzing noise in autoencoders and deep networks. arXiv preprint arXiv:1406. 1831, 2014.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1:8, 2019.
Paul K Rubenstein, Bernhard Schoelkopf, and Ilya Tolstikhin. On the latent space of wasserstein auto-encoders, arXiv preprint arXiv: /802.03661, 2018.
Anton Maximilian Schäfer and Hans Georg Zimmermann. Recurrent neural networks are universal approximators. In International Conference on Artificial Neural Networks, pp. 632-640. Springer, 2006.
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. Style transfer from non-parallel text by cross-alignment. Advances in neural information processing systems, pp. 6830-684- 2017.
Sandeep Subramanian, Guillaume Lample, Eric Michael Smith, Ludovic Denoyer, Marc' Aurelio Ran- zato, and Y-Lan Boureau. Multiple-attriuut text style transfer. arXiv preprint 1110022811.00552, 2018.
Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto-encoders. arXiv preprint arXiv: 171100155. 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096-1103, ACM, 2008.
Zichao Yang, Zhiting Hu, Ruslan Salakhutdinov, and Taylor Berg- Berg-Kirkpatrick. Improved variational autoencoders for text modeling using dilated convolutions. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 3881 1881-3899. JMLR. org, 2017.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understand- ing deep learning requires rethinking generalization. In International Conference on Learning Representations, 2017.
Junbo Zhao, Yoon Kim, Kelly Zhang, Alexander M Rush, Yann LeCun, al. Adversarially regularized autoencoders. Proceedings of the 35th International Conference on Machine Learning, 2018

---

Under review as a conference paper at ICLR 2020
A WASSERSTEIN DISTANCE
The AAE objective can be connected to a relaxed form of the Wasserstein distance between model and data distributions (Tolstikhin et al., 2017). Specifically, for cost function c(::) X X R
and deterministic decoder mapping G + X, it holds that:
(EP(pr-Pau.---P B(ay))r(c(x, (c(x, <(=)=)={(=)=<p()
inf
inf E (a) (a) (c(a, G(=))]
where the minimization over couplings I' with marginals Pdat and pa can be replaced with minimiza- tion over conditional distributions g(=|2) whose marginal g(z) Expn(z) [=(==))(((())))))))))))))))))[ matches the latent prior distribution p(z). Relaxing this marginal constraint via a divergence penalty p({())lp(z)) estimated by adversarial training, recovers the AAE objective (Eq. 1). In particular, AAE on discrete x with the cross-entropy loss is minimizing an upper bound of the total variation distance
between Pdata and pc, with chosen as the indicator cost function (Zhao al., 2018).
Our model is optimizing over conditional distributions g(>|x) ()) of the form (6), subset of all possible conditional distributions. Thus, after introducing input perturbations, our method is still minimizing an upper bound of the Wasserstein distance between Pdata and PG described in (7).
B PROOF OF THEOREM
Theorem 1. For any one-to-one encoder mapping E. from (Z1, .nn } to (21, ,Zn), , the optimal value of objective maxgeg, 1 n Ei= lo (x,))((())) the same.
Proof. Consider two encoder matchings x; to Ra(i) and x; to =a(1). where both and 3 are permu- tations of the indices [I....,n} n.). Suppose G., is the optimal decoder model for the first matching (with permutations o.). This implies
PG. = arg max GEG,
log tog (x)l=(())))))
Now let PG, (zg(2z) = PG, ( objective value matching 3 as G. for matching a, while still respecting the Lipschitz constraint.
(1) 122). V. Then Gg can achieve exactly the same log-likelihood
C PROOF OF THEOREM? 2
Theorem 2. Let d be distance metric over X. Suppose I1, I2,
32, 33, a4 satisfy that with some
6 <> O: 2(21, < e, d(x3,, 24) <,6, 6, and d(sr,, > for all other (Ii,3;) pairs. In addition, 21, 22, 24 satisfy that with some <, 8 Ç: 1|21-122 1 22ll < 6, ||23 -zall ô. and ||z - 2yll > for all other (2,, 2z) pairs. Suppose our perturbation process C reffects local geometry with: pc(x,;(13) = 1/2 if d(x,, xz) < and = 0 otherwise. For 5 < (olog (o(L)) + log 2) and Ç T, log (1/(v2- 1)). the denoising objective maxGEsL 1 E E/=1 PC(F, (x;) log log E(xx)) (where n = 4) achieves the largest value when encoder E maps close pairs of x to close pairs of a.
Proof. Let [n] denote [l....,n.] n). and assume without loss of generality that the encoder E maps each to z;. We also define A = [1,2], B {3, 4] the two x-pairs that lie close together. For our choice of C(x), the training objective to be maximized
Σ iJEA
log (
Σ K.EEB
log (ogpo(x)(E(z))
Σ
logPo(1.12z)
+
Σ
log logPa(ryla))
(8)
K
The remainder of our proof is split into two cases:
Case 1. ||ا =
for E A, B
12

---

Under review as a conference paper at ICLR 2020
Case 2. ||2 - < 8 for A, B
Under Case 1, z points that lie far apart also have 2 encodings that remain far apart. Under Case 2, points that lie far apart have encodings that lic close together. We complete the proof by showing that the achievable objective value in Case 2 is strictly worse than Case 1, and thus an optimal encoder/decoder pair would avoid the H, 2 matching that leads Case 2.
In Case 1 where ||, - zel1 Ç for all j E A, E B, we can lower bound the training objective (8)
by choosing:
11-1 if E A i, pa (a 1/22
otherwise
with 7 d(-LC) (0, 1), where a--) denotes the sigmoid function. Note that this ensures
Σ pa(()1z;) for each j [4], and does not violate the Lipschitz condition from Assumption ve[4]
since:
llog log (xsp2())
{
=
i E A j, B
log ((1 7)/)
otherwise
and thus remains < 11|2 =z|| when 7 = - 2 a ( = = 1/(1 + exp([||2z =x[i]
Plugging the pa 2) assignment from (9) into (8), we see that an optimal decoder can obtain training objective value 2 8 log [o ([)) in Case 1 where ||= - =ell 5. Vj A, B.
Next, we consider the alternative case where ||z zell 8 for A, B.
For i,j E A and for all e E B, we have:
log (xxgl2z)) log Pa Pa (2,)22) T.|l2
by Assumption
< log (rr (2) + L5
< LE + log - Σ
pa(
since Ex ((rallz))) S
KEB
Continuing from (8), the overall training objective in this case is thus:
Σ
log loglo(<)12;)
+
Σ logpo(zz/22)
R
4L8
min log FEB
Σ
(E(
Σ
log pa (ze)
AJEA
REB
N.ER
< 4L8 +
log
1
Σ
+
log (2
EEB
KEB
KEB
< 4L8 12 log
using the fact that the optimal decoder for the bound in this case is: (=(2))2)) = 1/4 for all k, E B.
Finally, plugging range for å stated in the Theorem 2, it shows that the best achievable objective value Case 2 is strictly worse than the objective value achievable in Case Thus, the optimal encoder/decoder pair under the AAE with perturbed ø will always prefer the matching between (#1...... 24] and { (21....,24 } that ensures nearby 2; are encoded to nearby 2, (corresponding to Case 1).
D PROOF OF THEOREM
Theorem 3. Suppose "1,"." Ttn are divided into n/ K clusters of equal size K, with St denoting the cluster index of wi. Let the perturbation process C be uniform within clusters, i.e. po(x;(æ;) 1/K K if S, = Sj and = 0 otherwise. For a one-to-one encoder mapping E from {- Fn to -.1,... 2n the denoising objective maxGes. 1 > Ej= Po(x,) log E((,) is upper bounded by: 1 Eisss.s, logo F((.)) F(xr))l) log K.
13

---

Under review as a conference paper at ICLR 2020
Proof. Without loss of generality, let Exx;) 2, for notational convenience. We consider what is the optimal decoder probability assigmment pa (2x|2z) under the Lipschitz constraint 2.
The objective of the AAE with perturbed is to maximize:
1 ΣΣ (a,)) (ogpot nI
Σ K:S.Ss
log
nk
We first show that the optimal po(-).) will satisfy that the same probability is assigned within cluster, i.e. p(zz|zz) = p())(z)) for all i, k s.t. S; = Sy. If not, let Paj - m Pa (z)l))))) and we reassign Par (za12z) ) = / K. Then G' still conforms to the Lipschitz constraint if G meets it, and G' will have larger target value than G.
Now let us define P3 = Eis; po(zilz)) = K K.PGK-PPPAA (P(()))))) (0SP) S 1). The objective becomes:
max PG
1 nk
Σ
logpo(z)lz))
max
Σ log (og,(()))))))
pG
8: S3
Σ log Pj llog K
max
Pa n
max
Σ Σ
(log P. + log Pj) - log K
pa 2n 2
$ 2n
Σ
max(log P, log Pj) - log K
PC
Consider each term maXpa (log P, log P;): when S, Sj. this term can achieve the maximum
value by assigning P. = P; = 1; when S; * Sj. the Lipschitz constraint ensures that:
log(1 P.) log P3 L|| zz
log(1 - Pÿ) 2 log P: - L||z - -z
Therefore:
log P: + log P3 < 2 log o (Lllz 23ll)
Overall, we thus have:
max Σ Σ PC nk
loggo((;|25)
Σ
logo((Ll22 =zill) loog K
n² fJ
E DATASETS
The Yelp dataset is from Shen al. (2017). which has 444K/63K//27K sentences of less than 16 words in length as train/dev/test sets, with vocabulary of 10K. It was originally divided into positive and negative sentences for style transfer between them. Here discard the sentiment label and let the model learn from all sentences indiscriminately. Our second dataset of Yahoo answers is from Yang et al. (2017). It was originally document-level. We perform sentence segmentation and keep sentences with length from 2 to 50 words. The resulting dataset has 495K/49K/50K sentences for train/dev/test sets, with vocabulary size 20K.
F EXPERIMENTAL. DETAILS
We use the same architecture to implement all models with different objectives. The encoder E, generator G, and the language model used to compute Forward/Reverse PPL one-layer LSTMs with hidden dimension 1024 and word embedding dimension 512. The last hidden state of the encoder is projected into 128/256 dimensions to produce the latent code for Yelp/Yahoo datasets
14

---

Under review as a conference paper at ICLR 2020
respectively, which is then projected and added with input word embeddings fed to the generator. The discriminator D is an MLP with one hidden layer of size 512. À of AAE based models is set to 10 to emsure the latent codes indistinguishable from the prior. All models are trained via the Adam optimizer (Kingma & Вa, 2014) with learning rate 0.0005, 31 = 0.5, $2 = 0.999. At test time, encoder-side perturbations disabled, and greedy decoding to generate from
G HUMAN EVALUATION
For the tense transfer experiment, the human annotator is presented with source sentence and two outputs (one from each approach, presented in random order) and asked to judge which one successfully changes the tense while being faithful to the source, whether both good/bad, if the input is not suitable to have its tense inverted. We collect labels from two human annotators and if they disagree, further solicit label from the third annotator.
H CENNEATTION-EECONSSSO
RESULTS ON THE YAHOO DATASET
reel data
220
ARAE
0.01
p
RAVAE
400
AAE
130
Fonward PPL
LAAE
À
0.01
Reverrs PPL
150
140
Ax 0.1
0.01
0.1
200
0.03
17 0.1
0.1
100
100
60
Reconstruction BLEU
Reconstruction BLEU
Figure H.I: 1: cimeraatinn-ecoostnuuee
trade-off of various text autoencoders on Yahoo. The "real data" line marks the PPL. of language model trained and evaluated on real data. We strive to approach the lower right corner with both high BLEU and low PPL. The grey box identifies hyperparameters subsequent experiments. Points of collapse (Reverse PPL 300) are removed from the right pancl.
use for respective models
I NEIGHBORHOOD PRESERVATION
Yelp
Yahoo
40
P-VAE
AAE
30
LAAE
Recall P8
DAAE
Untrained Encoder
20
15
DAE
Figure I.2: Recall rate of 10 nearest neighhors in the sentence space retrieved by & nearest neighbors in the latent space on the Yelp and Yahoo datasets. Here include non-generative models AE and DAR. We find that untrained RNN encoder from random initialization has good recall rate, and suspect that SGD training vanilla towards only the reconstruction improves neighborhood preservation in this case. Also note that DAAE has the highest recall rate among all generative models that have a latent prior imposed.
will not overturn this initial bias. Note that denoising still
15

---

Under review as a conference paper at ICLR 2020
J ADDITIONAL EXAMPLES
Source
how many gospells are there that were n't included in the bible?
5.NN by AAE
there are no other guspels that were n'l included in the bible
how many permutations anyone picked up any the _LNNKK the film what 's the significacee the number 40 the bible ? how many pieces of ribhon were used in the .UNK act
for the letters in word _J_JUU ?
5-NN by DAAB
there are no other guspels that were n'l included in the bible how many litres water there sea? many .UNK gods
classreom
how many of ribbon used the _UNK act how many times have you been grounded in the last year ?
how change colors new yahoo
5-NN by AAE
how should you present yourself __NNK speaking exam
how can i learn to he hip hop producer?
how can create a .UN web on the internet how can change my _UNK female not male ?
what should you look in buying your cello??
5-NN by DAAE
how do change that back to english? ? possible .UNK a yahoo account
how do change my yahoo toolhar options
what should you look for in baying your first cello
who you should go number one basehall fantasy draft. pujols _UNK?
Table J.1: Examples of nearest neighbors in the latent Euclidean space of AAE and DAAE on Yahoo dataset.
Input
not spend with you
slow service,
tasted
night 's leftovers.
ARAE
the staff rude and the dr. does worth two with you.
slow service, the food tasted like last night leftovers.
8. VAE
the staff rude and the dr. did spend time with your attitute.
slow service the food tastes like last place
AAE
the staff was nude and the dr. does not spend time with you.
slow service, the food tasted like last night 's leftovers.
LAAE
the staff was rude and the dr. is even for another of her entertained
slow service, the food, on this burger spot
DAAE
the siuff was rude and the dr. did not make time with you.
slow service the food tastes like last night
Input
they the worst credit union in arizona..
reported this twice and nothing done.
ARAE
they the warst bank credit in arizona.
this twice and nothing done
9. VAE
they were the worst credit union in my book.
''e gone here and nothing too.
AAE
they are the worst credit union in arizona.
reported this twice and nothing was done
LAAE
they were the wonst credit union in my heart..
dislike this twice so pleasant guy.
DAAE
they were worst union arizoma ever
this pizza and noihing done.
Table J.2: Additional examples of vector arithmetic for tense inversion.
16

---

Under review as a conference paper at ICLR 2020
AAE
DAAE
Input
extremelyy rude this woman extremely rude me.
this
this extremety rude
+
this this staff was umazing.... this staff is amazing.
extremely nice.
+1.5v
this woman was extremely rude to baby..
+22
this woman was extremely rude to muftins.
Toput
my hoyfriend said his pizza husic and bland also.
my boyfriend said his pizza hasic and bland also. boyfriend said his pizza is also excellent.
4-2
boyfriend said his pizza was basic and tasty also..
+1.5v
shared suid friday pizza was basic and lasty also...
bayfriend and pizza is excellent also..
+2
my shared got pizza pasia was basic and lusty also
my smoked pizza is excellent and also exceptional
Input
the stew is quite inexpensive and very lasty.
the stew is quite inexpensive and very tasty
the stew quite inexpeasive and tasty
the stew is quite an inexpensive and very large
-1.5v
the stew is quite inexpeasive and very very tasteless.
the stow is quite bit overpriced and very fairly brown..
2v
the - was being slow very very tasteless.
the hostess was quite impossible in an expensive and very few customers.
Input
the patrons all looked happy and relaxed.
the patrons all looked happy and reluxed. the patrons all helped us were happy and relaxed
the patrons all looked happy and relaxed.
1.50
the patrons all just happy and smelled.
the patrons that seemed around and left very stressed the putrons actually kept us all looked long and was annoyed.
2v
the patrons all just happy and smelled.
Table J.3: Additional examples of vector arithmetic for sentiment transfer.
Inpuit language should learn to more competitive today glohal culture? Input
what languages you speak ?
AAE
what language should learn to be more competitive in today global culture ?
what language should learn to be more competitive today global culture what language should you speak
what languages you speak:
what languages you speak?
DAAE
what language should i learn to be more competitive in today global culture ?
what language should learn to be competitive today in arabic 's culture? ?
what languages do you learn to be english culture what languages you what languages you speak?
Input
believe angels exist.
Input
you were character from movic, , who would be and why? ?
AAF
helieve angels exist.
believe angels there was the exist exist, believe in tsunami romeo or <unk>- i think would exist as the world population.
you were a character from me in this, would we he ( why you were a charncter from movie, who would it why
DAAR
helieve angels exist.
believe angels exist in the evolution. what did <unk> worship by in sunk> universe ?
you were your character from hible it will he why
you were charncter from movie, , who would
why
Table J.4: Interpolations between two input sentences generated by AAE and model the Yahoo dataset.
17


Reviews:
[{"review_id": "r1gBOZnutS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presented a denoising adversarial autoencoder for sentence embeddings. The idea is that by introducing perturbations (word omissions, etc) the embeddings are more meaningful and less \"memorized\". Evaluations include measuring sentence perplexity in generation/reconstruction, tense changing via vector arithmetic, sentiment changes via negative/positive vector additions, and sentence interpolations. \n\nStrengths: I thought the idea is nice, and the results do seem to show improvements in a number of interesting tasks. \n\nWeaknesses: I don't really think the explanation, especially in Theorem 1, makes a lot of sense. Qualitatively speaking, it's true that \"memorization\" in autoencoders (where the latent space has a 1-1 mapping with the input space) is problematic when the autoencoders are too powerful, but it is not always the case, and it is too far to say that the probability in theorem 1 is ALWAYS agnostic to encoding. The fact is word2vec works just fine with no perturbations, and there is no mathematical reason why sentence embeddings are fundamentally different. What is more accurate to say is that there is a tradeoff between model complexity and latent space representation usefulness, which is also related to the regularization/overfitting tradeoff in supervised learning. Here, injecting noise in the exact same fashion proposed in this paper is a well-accepted practice. While I think it's interesting that it works well here, I wouldn't really frame it as such a novelty, in that case, and I believe the other works on denoising autoencoders should be compared against in the experiments. In general, I find the mathematical claims a bit dubious, as a main assumption seems to be that the autoencoder itself is so overparametrized that it isn't really functioning as a representation-learning tool anyway. \n\nI also feel that the last experiment (referenced in the appendix) needs to go in the main text if we're to see it as a contribution. There is some wording that can be tightened in the main text, to make more room. \n\nOverall, I would improve my rating if the paper refocused more on the experiments, included more baselines (like other denoising autoencoders) and tasks that measure something besides perplexity (such as actual sentiment prediction, or machine translation, or other somewhat unrelated downstream tasks), and decreased the emphasis on the theoretical analysis--unless there is something I am significantly misunderstanding, it does not seem to be a particularly powerful theoretical contribution. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "H1x_OYsitH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper \"Denoising Improves Latent Space Geometry in Text Autoencoders\" tackles the problem of text autoencoding in a space which respects text similarities. It is an interesting problem for which various attempts have been proposed, while still facing difficulties for encoding in smooth spaces. The paper proposes a simple (rather straightforward) approach based on adversarial learning, with some theoretical guarantees, which obtains good performances for reconstruction and neighborhood preservation. \n\nMy main concern is about the missing of comparison with word dropout with variational encoding [Bowman et al., 2016], which also considers perturbations of the input texts to enforce the decoder to use the latent space. While the authors cite this work, I cannot understand why they did not include it in their experiments. \n\nAlso, theorem 3 gives an upperbound of the achievable log-likelihood, which is \"substantially better when examples in the same cluster are mapped to to points in the latent space in a manner that is well-separated from encodings of other\nclusters\". Ok but what does it show for the approach. If it was a lower-bound of the DAAE likelihood it would be interesting. But an upperbound ? In which sense does it indicate that it will be better than AAE ? Wouldn't it be possible to theoretically analyze other baselines? Also, all the theoretical analysis is made based on strong assumptions. Are these verified on considered datasets? \n\n\nMinor questions : \n          - In introduction of the experiments section, authors mention that they tried word removal and word masking. What is the difference ? \n           - what is the language model used for forward and reverse ppl ? \n\n   ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BylI49ChFr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper argues that adding noise to the inputs of an adversarial autoencoder for text improves the geometry of the learned latent space (in terms of mapping similar input sentences to nearby points in the latent space). The authors present a mathematical argument for why adding noise to the inputs would enforce latent space structure while a vanilla autoencoder would have no preference over x-z mappings.\n\nOverall, the paper addresses an important problem of improving autoencoder based generative models of text, presents a simple solution to do so and mathematically and empirically demonstrates its effectiveness. While I think that the benchmarks are somewhat artificial with small sentences and vocabulary sizes, I think the improvements demonstrated are substantial enough.\n\nI have a few questions & comments\n\n1) I’m curious about whether the authors have an intuition for why input space noise is better than latent space noise? Poole et al 2014 [1] showed that additive latent space gaussian noise in autoencoders is equivalent to a contractive autoencoder penalty and contractive autoencoders have an *explicit* penalty to encourage minimal change in z when changing x (i.e.) penalizing the norm of ||dz/dx||. Additive latent space noise appears to be a key ingredient to getting the ARAE and similar work like in Subramanian et al 2018 [2] to work. Was the LAAE implemented in the same framework as your DAAE?\n2) It would be great to see Forward / Reverse PPL results on bigger datasets like the BookCorpus or WMT similar to [2].\n3) You may be able to get similar reconstruction vs sample quality trade-offs with ARAEs by varying the variance of the gaussian noise, similar to LAAEs.\n4) In Figure 3, I would really like to see how an autoencoder that isn’t a generative model performs. How well would a vanilla autoencoder or vanilla DAE perform? This is a cool setup to evaluate latent space representation quality - you could even consider running some of the SentEval probing tasks on these representations.\n5) Could you use something like gradient-based latent space walks like in [2] to characterize the latent space geometry? https://arxiv.org/abs/1711.08014 also use similar gradient-based walks to characterize latent space smoothness in deep generative models. For example, if it takes 10 latent space gradient steps with a fixed learning rate for model “a” to turn sentence “x” into a *similar* sentence “y” but 20 steps for model “b”, then maybe “a” has smoother latent space geometry.\n6) Theorem 1 - What if the set of zs isn’t unique and there is some sort of encoder collapse? Does this theorem still hold? (i.e.) there exists some set of points x_1, x_2 .. x_i \\in x, that all map to z_k (and even potentially in the limit that all points in x map to the same point in z space).\n7) It would be good to point out that the model presented in this work is far from SOTA on sentiment style transfer benchmarks like Yelp.\n\n[1] Analyzing noise in autoencoders and deep networks - https://arxiv.org/pdf/1406.1831.pdf\n[2] Towards Text Generation with Adversarially Learned Neural Outlines - https://papers.nips.cc/paper/7983-towards-text-generation-with-adversarially-learned-neural-outlines.pdf", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work presents a simple technique for improving the latent space geometry of text autoencoders. The strengths of the paper lie in the simplicity of the method, and results show that the technique improves over the considered baselines. However, some reviewers expressed concerns over the presented theory for why input noise helps, and did not address concerns that the theory was useful. The paper should be improved if Section 4 were instead rewritten to focus on providing intuition, either with empirical analysis, results on a toy task, or clear but high level discussion of why the method helps. The current theorem statements seem either unnecessary or make strong assumptions that don't hold in practice. As a result, Section 4 in its current form is not in service to the reader's understanding why the simple method works. 
Finally, further improvements to the paper could be made with comparisons to additional baselines from prior work as suggested by reviewers.
Decision: Reject