On Variational Learning of Controllable Representations for Text without Supervision

1 INTRODUCTION . High-dimensional data , such as images and text , are often causally generated through the interaction of many complex factors , such as lighting and pose in images or style and content in texts . Recently , VAEs and other unsupervised generative models have found successes in modelling the manifold of natural images ( Higgins et al. , 2017 ; Kumar et al. , 2017 ; Chen et al. , 2016 ) . These models often discover controllable latent factors that allow manipulation of the images through conditional generation from interpolated or extrapolated latent codes , often with impressive quality . On the other hand , while various attributes of text such as sentiment and topic can be discovered in an unsupervised way , manipulating the text by changing these learned factors have not been possible with unsupervised generative models to the best of our knowledge . Cı́fka et al . ( 2018 ) ; Zhao et al . ( 2018 ) observed that text manipulation is generally more challenging compared to images , and the successes of these models can not be directly transferred to texts . Controllable text generation aims at generating realistic text with control over various attributes including sentiment , topic and other high-level properties . Besides being a scientific curiosity , the possibility of unsupervised controllable text generation could help in a wide range of application , e.g. , dialogues systems ( Wen et al. , 2016 ) . Existing promising progress ( Shen et al. , 2017 ; Fu et al. , 2018 ; Li et al. , 2018 ; Sudhakar et al. , 2019 ) all relies on supervised learning from annotated attributes to generate the text in a controllable fashion . The high cost of labelling large training corpora with attributes of interest limits the usage of these models , as pre-existing annotations often do not align with some downstream goal . Even if cheap labels are available , for example , review scores as a proxy for sentiment , the control is limited to the variation defined by the attributes . In this work , we examine the obstacles that prevent sequence VAEs from performing well in unsupervised controllable text generation . We empirically discover that manipulating the latent factors for typical semantic variations often leads to latent codes that reside in some low-density region of the aggregated posterior distribution . In other words , there are vacant regions in the latent code space ( Makhzani et al. , 2015 ; Rezende & Viola , 2018 ) not being considered by the decoding network , at least not at convergence . As a result , the decoding network is unable to process such manipulated latent codes , yielding unpredictable generation results of low quality . In order to mitigate the latent vacancy problem , we propose to constrain the posterior mean to a learned probability simplex and only perform manipulation within the probability simplex . Two regularizers are added to the original objective of VAE . The first enforces an orthogonal structure of the learned probability simplex ; the other encourages this simplex to be filled without holes . Besides confirming that latent vacancy is indeed a cause of failure in previous sequence VAEs ’ , it is also the first successful attempt towards unsupervised learning of controllable representations for text to the best of our knowledge . Experimental results on text style transfer show that our approach significantly outperforms unsupervised baselines , and is competitive with strong supervised approaches across a wide range of evaluation metrics . Our proposed framework also enables finer-grained and more flexible control over text generation . In particular , we can switch the topic in the middle of sentence generation , and the model will often still find a way to complete the sentence in a natural way . 2 BACKGROUND : VARIATIONAL AUTOENCODERS . The variational autoencoder ( VAE ) ( Kingma & Welling , 2013 ) is a generative model defined by a prior p ( z ) and a conditional distribution pθ ( x|z ) . The VAE is trained to optimize a tractable variational lower bound of log pθ ( x ) : LVAE ( x ; θ , φ ) = Ez∼qφ ( z|x ) [ log pθ ( x|z ) ] −DKL ( qφ ( z |x ) ||p ( z ) ) , ( 1 ) where qφ ( z |x ) is a variational distribution parameterized by an encoding network with parameters φ , and pθ ( x|z ) denotes the decoding network with parameters θ . This objective tries to minimize the reconstruction error to generate the data , and at the same time regularizes qφ ( z |x ) towards the prior p ( z ) . In this paper , p ( z ) is chosen as N ( 0 , I ) . For text modelling , the input x is some observed text . Both the encoding and decoding network are usually recurrent neural networks . Note that during learning , the decoding network pθ only learns to decode conditioned on z that are sampled from qφ ( z |x ) . In other words , the decoding network only learns to process z sampled from the aggregated posterior distribution qφ ( z ) = Ex∼pd ( x ) qφ ( z |x ) , where pd ( x ) is the data distribution . If qφ ( z ) has regions of low density , there is no guarantee that pθ would decode well in such regions . This is an important intuition that will become central to our analysis in Sec . 3 . 3 LATENT VACANCY PREVENTS EFFECTIVE MANIPULATION . In this section , we take a deeper look into the aggregated posterior latent space of sequence VAE trained on text , and provide justification for the alternative solution we propose in Section 4 . 3.1 OBSERVATIONS FROM UNSUPERVISED SENTIMENT MANIPULATION . As pointed out by Bowman et al . ( 2015 ) , one of the motivations to apply VAEs on text is to allow generation of the sentences conditioned on extrinsic features by controlling the latent codes . Without annotated labels , no previous methods have successfully learned controllable latent factors as mentioned in Sec . 1 . To understand what is missing , we conduct exploratory experiments to use VAE for unsupervised sentiment manipulation . We use the Yelp restaurant reviews dataset and the same data split following Li et al . ( 2018 ) . We train a β-VAE ( Higgins et al. , 2017 ) 1 with a latent space of 80 dimensions , an LSTM encoder , and an LSTM decoder . Details about this experiment are described in Appendix A.1 . By inspecting the accuracy on the validation set , we find that there exists one dimension of latent code achieving higher than 90 % sentiment classification accuracy by its value alone , while other latent codes get accuracy around 50 % . Further details can be found in Appendix A.2 . It means that this latent dimension is an effective sentiment indicator . Similar phenomena have been observed in 1We also try state-of-the-art techniques ( He et al. , 2019 ) on VAE w.r.t . optimizing ELBO but the KL term trained with those techniques are too small to capture the details of the source sentence . large-scale language models ( Radford et al. , 2017 ) . However , the direct influence on the generative process of the model observed in Radford et al . ( 2017 ) does not apply on the VAE . When we try to perform sentiment manipulation by modifying this latent dimension2 , the decoding network fails to generate the desired outputs most of the time , as evidenced by the poor quantitative evaluation in Table . 1 , and poor samples shown in Appendix A.3 . 3.2 LATENT VACANCY IN TEXT MODELLING . One possible reason for the failure is that the decoding network is never trained on codes like the manipulated ones . This is the case if the aggregated posterior has holes or regions of low density , and the manipulated codes fall into such vacant regions . Supposing the aggregated posterior latent space possesses a shape as shown in Fig . 1 , the direct manipulated latent codes will fall out of the aggregated posterior latent space for most input samples . Such latent codes are never seen by the model during training and possess a low density under the aggregated posterior distribution , leading to unpredictable behaviours during decoding . To verify our hypothesis demonstrated in Fig . 1 , we empirically estimate the density of sentimentmanipulated codes under the aggregated posterior distribution of our trained VAE . Here , we approximate the data distribution pd ( x ) with the empirical distribution over all the training samples . As a result , the estimated aggregated posterior distribution is a large mixture of Gaussian distribution . For all 1000 test samples , we move the dimension of code capturing sentiment from µ− 2σ to µ+ 2σ where µ and σ are the mean and the standard deviation estimated on all the training samples and 2Different strategies are attempted , see Appendix A.4 for details . measure the averaged negative log-likelihood ( NLL ) under the aggregated posterior distribution . As depicted in Fig . 3 ( a ) , the NLL plotted in blue dot curve rises sharply when moving away from µ even if there is only one dimension of code is changing , indicating the existence of the vacancy in the aggregated posterior latent space . In addition , we draw the histogram of all the test samples ’ NLL considering their original latent codes and modified ones in Fig . 3 ( b ) . The histogram shows that there is a large divergence in NLL between the original latent codes and the modified ones . Also , the modified latent codes have two separate modes , confirming the irregular shape of the aggregated posterior latent space . In order to resolve this issue , the approach proposed in this work is to constrain the posterior in a way that the manipulation only happens in a learned simplex , as depicted in Fig . 2 . In this constrained subspace , the phenomenom of low density holes of aggregated posterior is significantly reduced , as Fig . 3 ( a ) and ( c ) empirically show that there is little change in NLL of original versus modified codes . The details of our method is presented in the next section . 4 METHOD . 4.1 OVERVIEW . The experiments conducted in Sec . 3 validates the existence of vacancy in the aggregated posterior latent space . One potential way to resolve the problem is to better match the aggregated posterior with the prior ( Makhzani et al. , 2015 ; Tomczak & Welling , 2017 ; Zhao et al. , 2018 ) . However , in terms of unsupervised learning of controllable representation for text , these previous methods have not shown successes ; Zhao et al . ( 2018 ) only attempted supervised text style transfer , and also reported negative results from the AAE ( Makhzani et al. , 2015 ) . Another way to resolve the vacancy issue is to directly enforce that the aggregated posterior itself has no vacant region anywhere where we would like to perform latent code manipulation . We propose to map the posterior Gaussian mean to a constrained space , more specifically a learned probability simplex , where we can encourage the constrained latent space to be filled without vacancy , and perform manipulation to be within this simplex . As illustrated in Fig . 2 , we add an additional mapping function as part of the encoding network which maps the mean of the Gaussian posterior to a constrained space . Two regularization terms are introduced later to ensure the learned simplex is not degenerate and that this subspace is well filled . In addition , we separately model the relevant factors that we wish to control and the irrelevant factors by splitting z into two parts , z ( 1 ) and z ( 2 ) , following prior work ( Bao et al. , 2019 ) . The first part captures the relevant factors that are dominant in the data without an inductive bias from external signals , while the second part learns to encode the remaining local information that is useful for reconstructing the source sentences . As a result , qφ ( z |x ) is decomposed into qφ1 ( z ( 1 ) |x ) qφ2 ( z ( 2 ) |x ) where φ = φ1 ∪ φ2 . With diagonal covariances the KL divergence term in Eq . 1 splits into two separate KL terms . In practice , we use a MLP encoding network to parametrize z ( 1 ) with some sentence representations as the input ( e.g. , averaging GloVe embeddings ( Pennington et al. , 2014 ) over the input tokens ) and a LSTM encoding network to parametrize z ( 2 ) . We only constrain the posterior of z ( 1 ) and z ( 2 ) is optimized the same way as the traditional VAE .

Summary: This paper presents a method for controlled text generation by using a new loss function (standard VAE loss with auxiliary losses added on). The method is tested on style transfer datasets: Yelp and Amazon. The central hypothesis is that when manipulating latent codes of a VAE, you can end up in low-density regions of the aggregated posterior. Such latent codes are rarely seen by the decoder so that quality of generation is low. To address this problem, they constrain the posterior mean to a learnt probability simplex and try to ensure that the simplex is densely filled. They do this by adding 2 regularizing losses to the VAE loss.