Denoising Improves Latent Space Geometry in Text Autoencoders

1 INTRODUCTION . Autoencoder based generative models have recently become popular tools for advancing controllable text generation such as style or sentiment transfer ( Bowman et al. , 2016 ; Hu et al. , 2017 ; Shen et al. , 2017 ; Zhao et al. , 2018 ) . By mapping sentences to vectors in the latent space , these models offer in principle an attractive , continuous approach to manipulating text by means of simple latent vector arithmetic . However , the success of such manipulations rests heavily on the latent space geometry and how well it agrees with underlying sentence semantics . Indeed , we demonstrate that without additional guidance , fortuitous geometric agreements are unlikely to arise , shedding light on challenges faced by existing methods . We use adversarial autoencoders ( Makhzani et al. , 2015 , AAEs ) to study the latent space geometry . In contrast to variational autoencoders ( Kingma & Welling , 2014 , VAEs ) , AAEs can maintain strong coupling between the encoder and decoder that the decoder does not omit the encoded input sentence ( Bowman et al. , 2016 ) . The training criterion for AAEs consists of two parts , the ability to reconstruct sentences and the additional constraint that the encoded sentences are overall indistinguishable from prior samples , typically Gaussian . We show that these objectives alone do not suffice to force proper latent space geometry for text control . Specifically , for discrete objects such as sentences where continuity assumptions no longer hold , powerful AAEs can easily learn to map training sentences into latent prior samples arbitrarily ( Figure 1 , Left ) , while retaining perfect reconstruction . Latent space manipulations in such cases will yield random , unpredictable results . To remedy this , we augment AAEs with a simple denoising objective ( Vincent et al. , 2008 ; Creswell & Bharath , 2018 ) that requires perturbed sentence with some words missing to be mapped back to the original version . We refer to our model as DAAE . We prove that the denoising criterion can eliminate disorganized solutions and drive the latent space to organize itself . As a result , similar sentences begin to be mapped to similar latent vectors ( Figure 1 , Right ) . Improvements in latent space geometry carry many positive consequences . Through systematic evaluations of the generation and reconstruction capabilities of various text autoencoders ( Cífka et al. , 2018 ) , we find that our proposed DAAE provides the best trade-off between producing highquality text vs. informative sentence representations . We empirically verify that DAAE has the best neighborhood preservation property , consistent with our theory . We further investigate to what extent 1Our code will be made publicly available after the review process . text can be manipulated by applying simple transformations in the learned latent space . Our model is able to perform sentence-level vector arithmetic ( Mikolov et al. , 2013 ) fairly well to change the tense or sentiment of a sentence without any training supervision . It also produces higher quality sentence interpolations than other text autoencoders , suggesting better linguistic continuity in its latent space ( Bowman et al. , 2016 ) . 2 RELATED WORK . Denoising is first introduced into standard autoencoders by Vincent et al . ( 2008 , DAE ) to learn robust representations . Without a latent prior , DAE requires sophisticated MCMC sampling to be employed generatively ( Bengio et al. , 2013 ) . Creswell & Bharath ( 2018 ) applied denoising with AAEs to generative image modeling . Here , we demonstrate that input perturbations are particularly useful for discrete text modeling because they encourage preservation of data structure in the latent space . Apart from the AAE framework that our paper focuses on , another popular latent variable generative model is the variational autoencoder ( Kingma & Welling , 2014 , VAE ) . Unfortunately , when the decoder is a powerful autoregressive model ( such as a language model ) , VAE suffers from the posterior collapse problem where the latent representations get ignored ( Bowman et al. , 2016 ; Chen et al. , 2016 ) . If denoising is used in conjunction with VAEs ( Im et al. , 2017 ) in text applications , then the noisy inputs will only exacerbate VAE ’ s neglect of the latent variable . Bowman et al . ( 2016 ) proposed to weaken VAE ’ s decoder by masking words on the decoder side to alleviate its collapse issue . However , even with a weakened decoder and combined with other techniques including KL-weight annealing and adjusting training dynamics , it is still difficult to inject significant content into the latent code ( Yang et al. , 2017 ; Kim et al. , 2018 ; He et al. , 2019 ) . Alternatives like the β-VAE ( Higgins et al. , 2017 ) appear necessary . Previous work on controllable text generation has employed autoencoders trained with attribute label information ( Hu et al. , 2017 ; Shen et al. , 2017 ; Zhao et al. , 2018 ; Logeswaran et al. , 2018 ; Subramanian et al. , 2018 ) . We show that the proposed DAAE model can perform text manipulations despite being trained in a completely unsupervised manner without attribute labels . This suggests that on the one hand , our model can be adapted to semi-supervised learning when a few labels are available . On the other hand , it can be easily scaled up to train one large model on unlabeled text corpora and then applied for transferring various styles . 3 METHOD . Define X = Vm to be a space of sequences of discrete symbols from vocabulary V ( with maximum length m ) ; also define Z = Rd to be a continuous latent space . Our goal is to learn a mapping between the data distribution pdata ( x ) over X and a given prior distribution p ( z ) over latent space Z ( following common practice , a Gaussian prior is used in our experiments , although not required by our methodology ) . Such a mapping allows us to easily manipulate discrete data through continuous latent representations z , and provides a generative model where data samples can be obtained by first drawing z from the prior and then sampling a corresponding sequence via p ( x|z ) . We adopt the adversarial autoencoder ( AAE ) framework , which involves a ( deterministic ) encoder E : X → Z , a probabilistic decoder G : Z → X , and a discriminator D : Z → [ 0 , 1 ] . Both E and G are recurrent neural networks ( RNNs ) 2 . E takes input sequence x and outputs the last hidden state as its encoding z. G generates a sequence x autoregressively , with each step conditioned on z and previous symbols . The discriminator D is a feed-forward net that outputs the probability of z coming from the prior rather than the encoder . E , G and D are trained jointly with a min-max objective : min E , G max D Lrec ( θE , θG ) − λLadv ( θE , θD ) ( 1 ) with : Lrec ( θE , θG ) = Epdata ( x ) [ − log pG ( x|E ( x ) ) ] ( 2 ) Ladv ( θE , θD ) = Ep ( z ) [ − logD ( z ) ] + Epdata ( x ) [ − log ( 1−D ( E ( x ) ) ) ] ( 3 ) where reconstruction loss Lrec and adversarial loss3 Ladv are weighted via hyperparameter λ > 0 . We further introduce perturbations in X space to learn smoother representations that reflect local structure in the data , ending up with the denoising adversarial autoencoder ( DAAE ) model . Given a perturbation process C that stochastically maps x to nearby x̃ ∈ X , let p ( x , x̃ ) = pdata ( x ) pC ( x̃|x ) and p ( x̃ ) = ∑ x p ( x , x̃ ) . We change the loss functions to be : Lrec ( θE , θG ) = Ep ( x , x̃ ) [ − log pG ( x|E ( x̃ ) ) ] ( 4 ) Ladv ( θE , θD ) = Ep ( z ) [ − logD ( z ) ] + Ep ( x̃ ) [ − log ( 1−D ( E ( x̃ ) ) ) ] ( 5 ) Here , Lrec is the loss of reconstructing x from x̃ , and Ladv is the adversarial loss evaluated on perturbed x . The objective function combines the denoising technique with the AAE ( Vincent et al. , 2008 ; Creswell & Bharath , 2018 ) . When pC ( x̃|x ) = 1 [ x̃ = x ] ( i.e . there is no perturbation ) , the above simply becomes the usual AAE objective . Let pE ( z|x ) denote the encoder distribution . With our perturbation process C , the posterior distributions of the DAAE are of the following form : q ( z|x ) = ∑ x̃ pC ( x̃|x ) pE ( z|x̃ ) ( 6 ) This enables the DAAE to utilize stochastic encodings even by merely employing a deterministic encoder network trained without any reparameterization-style tricks . Note that since q ( z|x ) of the form ( 6 ) is a subset of all possible conditional distributions , our model is still minimizing an upper bound of the Wasserstein distance between data and model distributions , as previously shown by Tolstikhin et al . ( 2017 ) for AAE ( see Appendix A for a full proof ) . 4 LATENT SPACE GEOMETRY . The latent space geometry of text autoencoders is an important yet understudied problem . Only when the latent space is smooth and regular can meaningful text manipulations be enacted via simple modifications of the corresponding latent representations . Here , we discuss in detail the posterior characteristics of the DAAE , and provide a theoretical analysis of how input perturbations help better structure the latent space geometry ( all proofs are relegated to the appendix ) . Assume our perturbations preserve x with some probability ( i.e . pC ( x|x ) > 0 ) . When the support of C ( x1 ) and C ( x2 ) do not overlap for different training examples x1 6= x2 , the encoder can learn to assign pE ( z|x̃ ) = pE ( z|x ) for x̃ ∈ C ( x ) , and we are back to the unconstrained posterior scenario q ( z|x ) = pE ( z|x ) ( Eq . 6 ) . If C ( x1 ) and C ( x2 ) do intersect , then the latent posterior of x1 and x2 will have overlapping components pE ( z|x̃ ) for x̃ ∈ C ( x1 ) ∩C ( x2 ) . For example , if pC ( x̃|x ) assigns a high probability to x̃ that lies close to x ( based on some metric over X ) , then for similar x1 and x2 , the high-probability overlap between their perturbations will inherently force their posteriors closer together in the latent space . This is desirable for learning good representations z , while not guaranteed by merely minimizing the statistical divergence between pdata ( x ) and pG ( x ) = Ep ( z ) [ pG ( x|z ) ] . Now we formally analyze which type of x-z mappings will be learned by AAE and DAAE , respectively , to achieve global optimality of their training objectives . Unlike previous analyses of 2Transformer models ( Vaswani et al. , 2017 ) did not outperform LSTMs on our moderately-sized datasets . 3We actually train E to maximize logD ( E ( x ) ) instead of − log ( 1 −D ( E ( x ) ) ) , which is more stable in practice ( Goodfellow et al. , 2014 ) . We also tried WGAN ( Arjovsky et al. , 2017 ) but did not notice any gains . noise in single-layer networks ( Poole et al. , 2014 ) , here we study high-capacity encoder/decoder networks ( Schäfer & Zimmermann , 2006 ) with a large number of parameters that are used in modern sequence models ( Devlin et al. , 2018 ; Radford et al. , 2019 ) . Throughout , we assume that : Assumption 1 . E is a universal approximator capable of producing any mapping from x ’ s to z ’ s . Assumption 2 . G can approximate arbitrary p ( x|z ) so long as it remains sufficiently Lipschitz continuous in z. Namely , there exists L > 0 such that all decoder models G obtainable via training satisfy that for all x ∈ X , z1 , z2 ∈ Z : | log pG ( x|z1 ) − log pG ( x|z2 ) | ≤ L‖z1 − z2‖ . Following prior analysis of language decoders ( Mueller et al. , 2017 ) , we assume that G is L-Lipschitz in its continuous input z ( denote this set of possible decoders by GL ) . When G is implemented as a RNN or Transformer language model , log pG ( x|z ) will remain Lipschitz in z if the recurrent or attention weight matrices have bounded norm . This property is naturally encouraged by popular training methods that utilize SGD with early stopping and L2 regularization ( Zhang et al. , 2017 ) . Note we have not assumed E or G is Lipschitz in x , which would be unreasonable since x stands for discrete text , and when a few symbols change , the decoder likelihood for the entire sequence can vary drastically ( e.g. , G may assign a much higher probability to a grammatical sentence than an ungrammatical one that only differs by one word ) . Our discussion is directed to the nature of such families of log-likelihood functions with a continuous variable z and a discrete variable x . We further assume an effectively trained discriminator that succeeds in its adversarial task : Assumption 3 . D ensures that the latent encodings z1 , · · · , zn of training examples x1 , · · · , xn are indistinguishable from prior samples z ∼ p ( z ) . For simplicity , we directly assume that z1 , · · · , zn are actual samples from p ( z ) which are given a priori . Here , the task of the encoder E is to map given unique training examples to the given latent points , and the goal of the decoder pG ( ·|· ) is to maximize −Lrec under the encoder mapping . The question now is which one-to-one mapping an optimal encoder/decoder will learn under the AAE and DAAE objective ( Eq . 2 and Eq . 4 ) . We start with the following observation : Theorem 1 . For any one-to-one encoder mappingE from { x1 , · · · , xn } to { z1 , · · · , zn } , the optimal value of objective maxG∈GL 1 n ∑n i=1 log pG ( xi|E ( xi ) ) is the same . Intuitively , this result stems from the fact that the model receives no information about the structure of x , and x1 , · · · , xn are simply provided as different symbols . Hence AAE offers no preference over x-z couplings , and a random matching in which the z do not reflect any data structure is equally good as any other matching ( Figure 1 , Left ) . Latent point assignments start to differentiate , however , once we introduce local input perturbations . To elucidate how perturbations affect latent space geometry , it helps to first consider a simple setting with only four examples x1 , x2 , x3 , x4 ∈ X . Again , we consider given latent points z1 , z2 , z3 , z4 sampled from p ( z ) , and the encoder/decoder are tasked with learning which x to match with which z . As depicted in Figure 1 , suppose there are two pairs of x closer together and also two pairs of z closer together . Let σ denote the sigmoid function , we have the following conclusion : Theorem 2 . Let d be a distance metric over X . Suppose x1 , x2 , x3 , x4 satisfy that with some > 0 : d ( x1 , x2 ) < , d ( x3 , x4 ) < , and d ( xi , xj ) > for all other ( xi , xj ) pairs . In addition , z1 , z2 , z3 , z4 satisfy that with some 0 < δ < ζ : ‖z1 − z2‖ < δ , ‖z3 − z4‖ < δ , and ‖zi − zj‖ > ζ for all other ( zi , zj ) pairs . Suppose our perturbation process C reflects local X geometry with : pC ( xi|xj ) = 1/2 if d ( xi , xj ) < and = 0 otherwise . For δ < 1L ( 2 log ( σ ( Lζ ) ) + log 2 ) and ζ > 1 L log ( 1/ ( √ 2− 1 ) ) , the denoising objective maxG∈GL 1 n ∑n i=1 ∑n j=1 pC ( xj |xi ) log pG ( xi|E ( xj ) ) ( where n = 4 ) achieves the largest value when encoder E maps close pairs of x to close pairs of z . This entails that DAAE will always prefer to map similar x to similar z . Note that Theorem 1 still applies here , and AAE will not prefer any particular x-z pairing over the other possibilities . We next generalize beyond the basic four-points scenario to consider n examples of x that are clustered . Here , we can ask whether this cluster organization will be reflected in the latent space of DAAE . Theorem 3 . Suppose x1 , · · · , xn are divided into n/K clusters of equal size K , with Si denoting the cluster index of xi . Let the perturbation process C be uniform within clusters , i.e . pC ( xi|xj ) = 1/K if Si = Sj and = 0 otherwise . For a one-to-one encoder mapping E from { x1 , · · · , xn } to { z1 , · · · , zn } , the denoising objective maxG∈GL 1n ∑n i=1 ∑n j=1 pC ( xj |xi ) log pG ( xi|E ( xj ) ) is upper bounded by : 1n2 ∑ i , j : Si 6=Sj log σ ( L‖E ( xi ) − E ( xj ) ‖ ) − logK . Theorem 3 provides an upper bound of the DAAE objective that can be achieved by a particular x-z mapping . This achievable limit is substantially better when examples in the same cluster are mapped to the latent space in a manner that is well-separated from encodings of other clusters . In other words , by preserving input space cluster structure in the latent space , DAAE can achieve better objective values and thus is incentivized to learn such encoder/decoder mappings . An analogous corollary can be shown for the case when examples x are perturbed to yield additional inputs x̃ not present in the training data . In this case , the model would aim to map each example and its perturbations as a group to a compact group of z points well-separated from other groups in the latent space . In conclusion , our analysis shows that a well-trained DAAE is guaranteed to learn neighborhoodpreserving latent representations , whereas even a perfectly-trained AAE model may learn latent representations whose geometry fails to reflect similarity in the x space . Empirical experiments in Section 5.2 confirm that our theory holds in practice .

Summary: This paper presented a denoising adversarial autoencoder for sentence embeddings. The idea is that by introducing perturbations (word omissions, etc) the embeddings are more meaningful and less "memorized". Evaluations include measuring sentence perplexity in generation/reconstruction, tense changing via vector arithmetic, sentiment changes via negative/positive vector additions, and sentence interpolations.