Unsupervised Discovery of Interpretable Latent Manipulations in Language VAEs

1 INTRODUCTION . Transformer-based models yield state-of-the-art results on a number of tasks , including representation learning ( Devlin et al. , 2019 ; Liu et al. , 2019 ; Clark et al. , 2020 ) and generation ( Radford et al . ; Raffel et al. , 2019 ; Lewis et al. , 2020 ) . Notably , large language models have been reported to produce outputs nearly indistinguishable from human-written texts ( Brown et al. , 2020 ) . Although the predictions of autoregressive language models are fluent and coherent , it is not clear how to manipulate the model to get samples with desired properties . For example , make them shorter , more formal or more positive , or , alternatively , use the same model to rewrite human-written texts in a different tone . Current approaches often rely on external labels of target attributes and require modifications to the model . This involves retraining for new attributes or changing the decoding procedure , which is usually expensive . In contrast , models with explicit latent spaces have the innate ability to manipulate text attributes by moving along latent directions . They , however , gained limited traction . One reason is that training a VAE on text data poses a number of optimization challenges , which have been tackled with a varying degree of success ( He et al. , 2019 ; Fu et al. , 2019 ; Zhu et al. , 2020 ) . Additionally , language VAEs are mostly small LSTM-based models which goes against the current trend of using large pretrained Transformers . The first large-scale language VAE model is the recently introduced OPTIMUS ( Li et al. , 2020 ) : it uses BERT as the encoder and GPT-2 as the decoder , and sets a new record on benchmark datasets . Differently from texts , latent space models for images , especially GANs , achieve state-of-the-art generation results . Therefore , these models have been the focus of the research community , and the properties of latent spaces are well-learned . For example , even early works on generative adversarial networks for images report that it is possible to have smooth interpolations between images in the latent space ( Goodfellow et al. , 2014 ) . More recent studies show that the latent space directions corresponding to human-interpretable image transformations ( from now on , ” interpretable directions ” ) can be discovered in an unsupervised way ( Härkönen et al. , 2020 ; Voynov & Babenko , 2020 ; Peebles et al. , 2020 ) . In this paper , we show that for the language domain , much alike the well-studied visual domain , a sufficiently “ good ” latent space allows to manipulate sample attributes with relative ease . To avoid the known difficulties associated with training language GANs , we experiment with VAEs ; more specifically , with the current state-of-the-art model OPTIMUS . We show that for this model , not only it is possible to produce meaningful and “ smooth ” interpolations between examples and to transfer specific properties via arithmetic operations in the latent space , but it is also possible to discover the interpretable latent directions in an unsupervised manner . We propose a method based on the PCA of latent representations of the texts in the training dataset . According to human evaluation , the proportion of interpretable directions among the ones found by our method is consistently larger than the proportion of interpretable directions among canonical co-ordinates or random directions in the latent space . The meaningful directions found by this method include , for example , subject age , subject gender , verb tense , and sentence length . Some of the directions , e.g . sentence length , are potentially useful : the ability to expand or shrink a text while preserving its content may be useful for tasks like summarization . Note that the proposed method is simple and fast . The method is simple because it requires only the forward pass of the encoder , without backpropagating through decoding steps . This is very important for the language domain , where backpropagation through samples is significantly more difficult than for images . Namely , generation is non-differentiable , and previous attempts to overcome this issue relied on noisy or biased gradient estimates , which is less reliable than the standard MLE training . Instead , we do not rely on generated samples at all : we operate directly in the latent space . Additionally , since sampling directly from the prior does not yield diverse samples in case of OPTIMUS , we use the representations of the training data without running a decoding procedure - this maked the method fast . To summarize , our contributions are as follows : 1 . We propose the first method for unsupervised discovery of interpretable directions in latent spaces of language VAEs . 2 . This method is simple and fast : it is based on PCA of latent representations for texts in the training dataset . 3 . This method is effective : the proportion of interpretable directions among the ones found by our method is consistently larger than that of canonical co-ordinates or random directions in the latent space . 4 . Our work lays foundations for two important areas : first , it allows to compare models in terms of latent space interpretability , and second , it provides a baseline for unsupervised latent controls discovery . 2 RELATED WORK . Finding interpretable directions in latent spaces of language VAEs is related to three lines of work . First , latent variable models for text and , more specifically , properties of latent spaces : for interpretable directions to exist , latent space has to be smooth ( i.e . allow coherent interpolations ) . Then , since great part of the motivation for finding interpretable directions is manipulating generated texts , we discuss works on controllable text generation for different types of models , both VAE and standard autoregressive . Finally , we mention recent works trying to discover interpretable directions in image GANs . 2.1 LATENT VARIABLE MODELS FOR TEXT . Latent variable models encode information about text into a probability distribution . In addition to sampling new sentences from the prior distribution , they potentially allow to explicitly encode specific properties of text , such as sentiment or style . Even early works on VAEs show that a latent space obtained with the VAE objective can result in coherent interpolations ( Bowman et al. , 2016 ) . While this is encouraging , training good VAEs with smooth and expressive latent spaces is challenging . Specifically , for interpretable directions to exist , we need a model which ( i ) does not ignore latent variable – to produce good samples , ( ii ) has continuous latent space – to allow controllable manipulation . Ignoring latent variable is a known problem of VAEs . It arises because of the the KL vanishing problem : over the course of training , the KL divergence part of the loss may drop to 0 , which indicates that the model ignores the latent variable . There exist many ways to alleviate this issue ( Yang et al. , 2017 ; Fang et al. , 2019 ; Fu et al. , 2019 ; Zhu et al. , 2020 ) ; one of the simpler ways is adjusting the weight of the KL loss component according to a specific schedule . Another problem is the latent vacancy problem : differently from images , not all regions of the latent space are occupied by the posterior distribution ( Xu et al. , 2020 ) . In simple words , text latent spaces tend to have “ holes ” where the decoding network fails to generalize . As a result , when the latent codes are manipulated , the modified codes often land in these holes or vacant regions in the posterior latent space . If this happens , a model can not decode properly . In light of the above , discovery of interpretable directions in text latent spaces is possible only with a strong model . Therefore , we use the current state-of-the-art model OPTIMUS ( Li et al. , 2020 ) . It is a recent large-scale variational autoencoder which initializes the encoder with BERT ( Devlin et al. , 2019 ) and the decoder with GPT2 ( Radford et al. ) . In addition to the model ’ s high capacity , we use it because of the available checkpoints and reported results on latent space manipulation . 2.2 CONTROLLABLE GENERATION FOR TEXT DATA . Latent variable models . A natural way to achieve text generation with required attributes is using latent variable text generation models . The idea is that information about the attribute value is encoded in the latent code , and to obtain samples with the desired property one has to fix the corresponding component ( direction ) of the code . For example , several works learn latent spaces with disentangled representations of content and style ( Hu et al. , 2017 ; Logeswaran et al. , 2018 ; Lample et al. , 2019 ; Yang et al. , 2018 ; Shen et al. , 2017 ; John et al. , 2019 ) . After that , to generate sentences in a specific style , the style vector is fixed . Depending on the approach , this style vector can either be estimated by encoding sentences with the desired attribute or be directly produced by specifying the structured latent code ( e.g . one-hot encoding of an attribute ) . Another line of research shows that it is possible to achieve attribute manipulation by moving in the latent space along specific vectors . These vectors , however , are found using data labelled with the attribute , i.e . with supervision . For example , Shen et al . ( 2020 ) change tense of a sentence by adding to its latent representation a “ tense vector ” computed as a difference of averaged representations of sentences with different tenses ; Wang et al . ( 2019 ) use gradients of the attribute classifier . One of the first successful methods that learns a disentangled latent space is the work by Xu et al . ( 2020 ) : they use basis vectors in the constrained latent space ; however , this involves training a model with a structured latent space , which is rather complicated . Autoregressive models . Controllable generation for standard autoregressive language models is usually achieved by either prepending an attribute to the input sequence as a prompt ( Keskar et al. , 2019 ) , training an additional component of the model ( Chan et al. , 2020 ) or adjusting the decoding result with additional attribute-specific language models ( Dathathri et al. , 2020 ) . A more thorough comparison of approaches to controlled text generation can be read in Prabhumoye et al . ( 2020 ) . Note that all these approaches require supervision and substantial changes to either training or generation procedures , whereas our approach is applicable to any variational autoencoder . 2.3 INTERPRETABLE DIRECTIONS MINING IN IMAGE GANS . To the best of our knowledge , there are only three works which discover interpretable latent directions in an unsupervised way , and all of them operate with GANs for images . Two of them are not applicable to texts directly . In Voynov & Babenko ( 2020 ) , the interpretable directions are trained . These directions are the ones which can be recognized by a separate reconstructor based on two samples , from the original and a shifted latent vector . Peebles et al . ( 2020 ) propose to learn disentangled attributes by minimizing the sum of squared off-diagonal terms of the generator Hessian matrix . Both approaches require backpropagation through sampling and therefore are not applicable directly for texts : unlike images , generated texts are not differentiable with respect to their latent representations . The last approach , Härkönen et al . ( 2020 ) , show that interpretable controls for image synthesis can be identified by finding principal components from layer outputs of the generator network for several samples from the prior . In our more challenging language domain , instead of sampling from the generator distribution , we take advantage of the availability of the encoder in VAEs and perform PCA on training data representations .

Summary: This paper proposes a simple approach to discover interpretable latent manipulations in trained text VAEs. The method essentially involves performing PCA on the latent representations to find directions that maximize variance. The authors argue that this results in more interpretable directions. The method is applied on top of a VAE model (OPTIMUS), and the authors argue that different directions discovered by PCA correspond to interpretable concepts.