Denoising Improves Latent Space Geometry in Text Autoencoders

1 INTRODUCTION . Autoencoder based generative models have recently become popular tools for advancing controllable text generation such as style or sentiment transfer ( Bowman et al. , 2016 ; Hu et al. , 2017 ; Shen et al. , 2017 ; Zhao et al. , 2018 ) . By mapping sentences to vectors in the latent space , these models offer in principle an attractive , continuous approach to manipulating text by means of simple latent vector arithmetic . However , the success of such manipulations rests heavily on the latent space geometry and how well it agrees with underlying sentence semantics . Indeed , we demonstrate that without additional guidance , fortuitous geometric agreements are unlikely to arise , shedding light on challenges faced by existing methods . We use adversarial autoencoders ( Makhzani et al. , 2015 , AAEs ) to study the latent space geometry . In contrast to variational autoencoders ( Kingma & Welling , 2014 , VAEs ) , AAEs can maintain strong coupling between the encoder and decoder that the decoder does not omit the encoded input sentence ( Bowman et al. , 2016 ) . The training criterion for AAEs consists of two parts , the ability to reconstruct sentences and the additional constraint that the encoded sentences are overall indistinguishable from prior samples , typically Gaussian . We show that these objectives alone do not suffice to force proper latent space geometry for text control . Specifically , for discrete objects such as sentences where continuity assumptions no longer hold , powerful AAEs can easily learn to map training sentences into latent prior samples arbitrarily ( Figure 1 , Left ) , while retaining perfect reconstruction . Latent space manipulations in such cases will yield random , unpredictable results . To remedy this , we augment AAEs with a simple denoising objective ( Vincent et al. , 2008 ; Creswell & Bharath , 2018 ) that requires perturbed sentence with some words missing to be mapped back to the original version . We refer to our model as DAAE . We prove that the denoising criterion can eliminate disorganized solutions and drive the latent space to organize itself . As a result , similar sentences begin to be mapped to similar latent vectors ( Figure 1 , Right ) . Improvements in latent space geometry carry many positive consequences . Through systematic evaluations of the generation and reconstruction capabilities of various text autoencoders ( Cífka et al. , 2018 ) , we find that our proposed DAAE provides the best trade-off between producing highquality text vs. informative sentence representations . We empirically verify that DAAE has the best neighborhood preservation property , consistent with our theory . We further investigate to what extent 1Our code will be made publicly available after the review process . text can be manipulated by applying simple transformations in the learned latent space . Our model is able to perform sentence-level vector arithmetic ( Mikolov et al. , 2013 ) fairly well to change the tense or sentiment of a sentence without any training supervision . It also produces higher quality sentence interpolations than other text autoencoders , suggesting better linguistic continuity in its latent space ( Bowman et al. , 2016 ) . 2 RELATED WORK . Denoising is first introduced into standard autoencoders by Vincent et al . ( 2008 , DAE ) to learn robust representations . Without a latent prior , DAE requires sophisticated MCMC sampling to be employed generatively ( Bengio et al. , 2013 ) . Creswell & Bharath ( 2018 ) applied denoising with AAEs to generative image modeling . Here , we demonstrate that input perturbations are particularly useful for discrete text modeling because they encourage preservation of data structure in the latent space . Apart from the AAE framework that our paper focuses on , another popular latent variable generative model is the variational autoencoder ( Kingma & Welling , 2014 , VAE ) . Unfortunately , when the decoder is a powerful autoregressive model ( such as a language model ) , VAE suffers from the posterior collapse problem where the latent representations get ignored ( Bowman et al. , 2016 ; Chen et al. , 2016 ) . If denoising is used in conjunction with VAEs ( Im et al. , 2017 ) in text applications , then the noisy inputs will only exacerbate VAE ’ s neglect of the latent variable . Bowman et al . ( 2016 ) proposed to weaken VAE ’ s decoder by masking words on the decoder side to alleviate its collapse issue . However , even with a weakened decoder and combined with other techniques including KL-weight annealing and adjusting training dynamics , it is still difficult to inject significant content into the latent code ( Yang et al. , 2017 ; Kim et al. , 2018 ; He et al. , 2019 ) . Alternatives like the β-VAE ( Higgins et al. , 2017 ) appear necessary . Previous work on controllable text generation has employed autoencoders trained with attribute label information ( Hu et al. , 2017 ; Shen et al. , 2017 ; Zhao et al. , 2018 ; Logeswaran et al. , 2018 ; Subramanian et al. , 2018 ) . We show that the proposed DAAE model can perform text manipulations despite being trained in a completely unsupervised manner without attribute labels . This suggests that on the one hand , our model can be adapted to semi-supervised learning when a few labels are available . On the other hand , it can be easily scaled up to train one large model on unlabeled text corpora and then applied for transferring various styles . 3 METHOD . Define X = Vm to be a space of sequences of discrete symbols from vocabulary V ( with maximum length m ) ; also define Z = Rd to be a continuous latent space . Our goal is to learn a mapping between the data distribution pdata ( x ) over X and a given prior distribution p ( z ) over latent space Z ( following common practice , a Gaussian prior is used in our experiments , although not required by our methodology ) . Such a mapping allows us to easily manipulate discrete data through continuous latent representations z , and provides a generative model where data samples can be obtained by first drawing z from the prior and then sampling a corresponding sequence via p ( x|z ) . We adopt the adversarial autoencoder ( AAE ) framework , which involves a ( deterministic ) encoder E : X → Z , a probabilistic decoder G : Z → X , and a discriminator D : Z → [ 0 , 1 ] . Both E and G are recurrent neural networks ( RNNs ) 2 . E takes input sequence x and outputs the last hidden state as its encoding z. G generates a sequence x autoregressively , with each step conditioned on z and previous symbols . The discriminator D is a feed-forward net that outputs the probability of z coming from the prior rather than the encoder . E , G and D are trained jointly with a min-max objective : min E , G max D Lrec ( θE , θG ) − λLadv ( θE , θD ) ( 1 ) with : Lrec ( θE , θG ) = Epdata ( x ) [ − log pG ( x|E ( x ) ) ] ( 2 ) Ladv ( θE , θD ) = Ep ( z ) [ − logD ( z ) ] + Epdata ( x ) [ − log ( 1−D ( E ( x ) ) ) ] ( 3 ) where reconstruction loss Lrec and adversarial loss3 Ladv are weighted via hyperparameter λ > 0 . We further introduce perturbations in X space to learn smoother representations that reflect local structure in the data , ending up with the denoising adversarial autoencoder ( DAAE ) model . Given a perturbation process C that stochastically maps x to nearby x̃ ∈ X , let p ( x , x̃ ) = pdata ( x ) pC ( x̃|x ) and p ( x̃ ) = ∑ x p ( x , x̃ ) . We change the loss functions to be : Lrec ( θE , θG ) = Ep ( x , x̃ ) [ − log pG ( x|E ( x̃ ) ) ] ( 4 ) Ladv ( θE , θD ) = Ep ( z ) [ − logD ( z ) ] + Ep ( x̃ ) [ − log ( 1−D ( E ( x̃ ) ) ) ] ( 5 ) Here , Lrec is the loss of reconstructing x from x̃ , and Ladv is the adversarial loss evaluated on perturbed x . The objective function combines the denoising technique with the AAE ( Vincent et al. , 2008 ; Creswell & Bharath , 2018 ) . When pC ( x̃|x ) = 1 [ x̃ = x ] ( i.e . there is no perturbation ) , the above simply becomes the usual AAE objective . Let pE ( z|x ) denote the encoder distribution . With our perturbation process C , the posterior distributions of the DAAE are of the following form : q ( z|x ) = ∑ x̃ pC ( x̃|x ) pE ( z|x̃ ) ( 6 ) This enables the DAAE to utilize stochastic encodings even by merely employing a deterministic encoder network trained without any reparameterization-style tricks . Note that since q ( z|x ) of the form ( 6 ) is a subset of all possible conditional distributions , our model is still minimizing an upper bound of the Wasserstein distance between data and model distributions , as previously shown by Tolstikhin et al . ( 2017 ) for AAE ( see Appendix A for a full proof ) . 4 LATENT SPACE GEOMETRY . The latent space geometry of text autoencoders is an important yet understudied problem . Only when the latent space is smooth and regular can meaningful text manipulations be enacted via simple modifications of the corresponding latent representations . Here , we discuss in detail the posterior characteristics of the DAAE , and provide a theoretical analysis of how input perturbations help better structure the latent space geometry ( all proofs are relegated to the appendix ) . Assume our perturbations preserve x with some probability ( i.e . pC ( x|x ) > 0 ) . When the support of C ( x1 ) and C ( x2 ) do not overlap for different training examples x1 6= x2 , the encoder can learn to assign pE ( z|x̃ ) = pE ( z|x ) for x̃ ∈ C ( x ) , and we are back to the unconstrained posterior scenario q ( z|x ) = pE ( z|x ) ( Eq . 6 ) . If C ( x1 ) and C ( x2 ) do intersect , then the latent posterior of x1 and x2 will have overlapping components pE ( z|x̃ ) for x̃ ∈ C ( x1 ) ∩C ( x2 ) . For example , if pC ( x̃|x ) assigns a high probability to x̃ that lies close to x ( based on some metric over X ) , then for similar x1 and x2 , the high-probability overlap between their perturbations will inherently force their posteriors closer together in the latent space . This is desirable for learning good representations z , while not guaranteed by merely minimizing the statistical divergence between pdata ( x ) and pG ( x ) = Ep ( z ) [ pG ( x|z ) ] . Now we formally analyze which type of x-z mappings will be learned by AAE and DAAE , respectively , to achieve global optimality of their training objectives . Unlike previous analyses of 2Transformer models ( Vaswani et al. , 2017 ) did not outperform LSTMs on our moderately-sized datasets . 3We actually train E to maximize logD ( E ( x ) ) instead of − log ( 1 −D ( E ( x ) ) ) , which is more stable in practice ( Goodfellow et al. , 2014 ) . We also tried WGAN ( Arjovsky et al. , 2017 ) but did not notice any gains . noise in single-layer networks ( Poole et al. , 2014 ) , here we study high-capacity encoder/decoder networks ( Schäfer & Zimmermann , 2006 ) with a large number of parameters that are used in modern sequence models ( Devlin et al. , 2018 ; Radford et al. , 2019 ) . Throughout , we assume that : Assumption 1 . E is a universal approximator capable of producing any mapping from x ’ s to z ’ s . Assumption 2 . G can approximate arbitrary p ( x|z ) so long as it remains sufficiently Lipschitz continuous in z. Namely , there exists L > 0 such that all decoder models G obtainable via training satisfy that for all x ∈ X , z1 , z2 ∈ Z : | log pG ( x|z1 ) − log pG ( x|z2 ) | ≤ L‖z1 − z2‖ . Following prior analysis of language decoders ( Mueller et al. , 2017 ) , we assume that G is L-Lipschitz in its continuous input z ( denote this set of possible decoders by GL ) . When G is implemented as a RNN or Transformer language model , log pG ( x|z ) will remain Lipschitz in z if the recurrent or attention weight matrices have bounded norm . This property is naturally encouraged by popular training methods that utilize SGD with early stopping and L2 regularization ( Zhang et al. , 2017 ) . Note we have not assumed E or G is Lipschitz in x , which would be unreasonable since x stands for discrete text , and when a few symbols change , the decoder likelihood for the entire sequence can vary drastically ( e.g. , G may assign a much higher probability to a grammatical sentence than an ungrammatical one that only differs by one word ) . Our discussion is directed to the nature of such families of log-likelihood functions with a continuous variable z and a discrete variable x . We further assume an effectively trained discriminator that succeeds in its adversarial task : Assumption 3 . D ensures that the latent encodings z1 , · · · , zn of training examples x1 , · · · , xn are indistinguishable from prior samples z ∼ p ( z ) . For simplicity , we directly assume that z1 , · · · , zn are actual samples from p ( z ) which are given a priori . Here , the task of the encoder E is to map given unique training examples to the given latent points , and the goal of the decoder pG ( ·|· ) is to maximize −Lrec under the encoder mapping . The question now is which one-to-one mapping an optimal encoder/decoder will learn under the AAE and DAAE objective ( Eq . 2 and Eq . 4 ) . We start with the following observation : Theorem 1 . For any one-to-one encoder mappingE from { x1 , · · · , xn } to { z1 , · · · , zn } , the optimal value of objective maxG∈GL 1 n ∑n i=1 log pG ( xi|E ( xi ) ) is the same . Intuitively , this result stems from the fact that the model receives no information about the structure of x , and x1 , · · · , xn are simply provided as different symbols . Hence AAE offers no preference over x-z couplings , and a random matching in which the z do not reflect any data structure is equally good as any other matching ( Figure 1 , Left ) . Latent point assignments start to differentiate , however , once we introduce local input perturbations . To elucidate how perturbations affect latent space geometry , it helps to first consider a simple setting with only four examples x1 , x2 , x3 , x4 ∈ X . Again , we consider given latent points z1 , z2 , z3 , z4 sampled from p ( z ) , and the encoder/decoder are tasked with learning which x to match with which z . As depicted in Figure 1 , suppose there are two pairs of x closer together and also two pairs of z closer together . Let σ denote the sigmoid function , we have the following conclusion : Theorem 2 . Let d be a distance metric over X . Suppose x1 , x2 , x3 , x4 satisfy that with some > 0 : d ( x1 , x2 ) < , d ( x3 , x4 ) < , and d ( xi , xj ) > for all other ( xi , xj ) pairs . In addition , z1 , z2 , z3 , z4 satisfy that with some 0 < δ < ζ : ‖z1 − z2‖ < δ , ‖z3 − z4‖ < δ , and ‖zi − zj‖ > ζ for all other ( zi , zj ) pairs . Suppose our perturbation process C reflects local X geometry with : pC ( xi|xj ) = 1/2 if d ( xi , xj ) < and = 0 otherwise . For δ < 1L ( 2 log ( σ ( Lζ ) ) + log 2 ) and ζ > 1 L log ( 1/ ( √ 2− 1 ) ) , the denoising objective maxG∈GL 1 n ∑n i=1 ∑n j=1 pC ( xj |xi ) log pG ( xi|E ( xj ) ) ( where n = 4 ) achieves the largest value when encoder E maps close pairs of x to close pairs of z . This entails that DAAE will always prefer to map similar x to similar z . Note that Theorem 1 still applies here , and AAE will not prefer any particular x-z pairing over the other possibilities . We next generalize beyond the basic four-points scenario to consider n examples of x that are clustered . Here , we can ask whether this cluster organization will be reflected in the latent space of DAAE . Theorem 3 . Suppose x1 , · · · , xn are divided into n/K clusters of equal size K , with Si denoting the cluster index of xi . Let the perturbation process C be uniform within clusters , i.e . pC ( xi|xj ) = 1/K if Si = Sj and = 0 otherwise . For a one-to-one encoder mapping E from { x1 , · · · , xn } to { z1 , · · · , zn } , the denoising objective maxG∈GL 1n ∑n i=1 ∑n j=1 pC ( xj |xi ) log pG ( xi|E ( xj ) ) is upper bounded by : 1n2 ∑ i , j : Si 6=Sj log σ ( L‖E ( xi ) − E ( xj ) ‖ ) − logK . Theorem 3 provides an upper bound of the DAAE objective that can be achieved by a particular x-z mapping . This achievable limit is substantially better when examples in the same cluster are mapped to the latent space in a manner that is well-separated from encodings of other clusters . In other words , by preserving input space cluster structure in the latent space , DAAE can achieve better objective values and thus is incentivized to learn such encoder/decoder mappings . An analogous corollary can be shown for the case when examples x are perturbed to yield additional inputs x̃ not present in the training data . In this case , the model would aim to map each example and its perturbations as a group to a compact group of z points well-separated from other groups in the latent space . In conclusion , our analysis shows that a well-trained DAAE is guaranteed to learn neighborhoodpreserving latent representations , whereas even a perfectly-trained AAE model may learn latent representations whose geometry fails to reflect similarity in the x space . Empirical experiments in Section 5.2 confirm that our theory holds in practice .

Summary: The paper "Denoising Improves Latent Space Geometry in Text Autoencoders" tackles the problem of text autoencoding in a space which respects text similarities. It is an interesting problem for which various attempts have been proposed, while still facing difficulties for encoding in smooth spaces. The paper proposes a simple (rather straightforward) approach based on adversarial learning, with some theoretical guarantees, which obtains good performances for reconstruction and neighborhood preservation.