Bag-of-Vectors Autoencoders for Unsupervised Conditional Text Generation

1 INTRODUCTION . In conditional text generation , we would like to produce an output text given an input text . Hence , parallel input-output pairs are required to train a good supervised machine learning model on this type of task . Large-scale pretraining ( Peters et al. , 2018 ; Devlin et al. , 2019 ; Lewis et al. , 2020 ) can alleviate the necessity for training examples to some extent , but even this requires a substantial number of annotations ( Yogatama et al. , 2019 ) . This is an expensive process and can introduce unwanted artifacts itself , which are henceforth learned by the model ( Gururangan et al. , 2018 ) . For these reasons , there is substantial interest in unsupervised solutions . Text autoencoders ( AEs ) don ’ t require labeled data for training , and are therefore a popular model for unsupervised approaches to many tasks , such as machine translation ( Artetxe et al. , 2018 ) , sentence compression ( Févry & Phang , 2018 ) and sentiment transfer ( Shen et al. , 2017 ) . The classical text AE ( Bowman et al. , 2016 ) embeds the input text into a single fixed-size vector via the encoder , and then tries to reconstruct the input text from the single vector via the decoder . Single-vector embeddings are very useful , because they allow to perform conditional text generation through simple mappings in the embedding space , e.g . by adding a constant offset vector to change attributes such as sentiment ( Shen et al. , 2020 ) . Recently , Mai et al . ( 2020 ) proposed Emb2Emb , a method that can learn these mappings directly in the embedding space of any pretrained single-vector AE . This is a powerful framework , because the AE can then be pretrained on virtually infinite amounts of unlabeled data before applying it to any downstream application . This concept , transfer learning , is arguably one of the most important drivers of progress in machine learning in the recent decade : These so-called Foundation Models ( Bommasani et al. , 2021 ) have revolutionized natural language understanding ( e.g , BERT ( Devlin et al. , 2019 ) ) and computer vision ( e.g , DALL-E ( Ramesh et al. , 2021 ) ) , among others . Since Emb2Emb was designed to work with any pretrained AE , it was an important step towards their scalability . However , as Bommasani et al . ( 2021 ) point out , another crucial model property is expressivity , the ability to represent the data distribution it is trained on . In this regard , single-vector representations are fundamentally limited ; they act as a bottleneck , causing the model to increasingly struggle to encode longer text ( Bahdanau et al. , 2015 ) . In this paper , we extent Emb2Emb from single-vector bottleneck AEs to Bag-of-Vector Autoencoders ( BoV-AEs ) , which encode text into a variable-size representation where the number of vectors grows with the length of the text . This gives BoV- enc dec Text Autoencoder Pretraining Inference enc dec enc Task Training Figure 2 : High-level view of the Emb2Emb framework . Text Autoencoder Pretraining : An autoencoder is trained on an unlabeled corpus , i.e. , the encoder enc transforms an input text x into a continuous embedding zx , which is in turn used by the decoder dec to predict a reconstruction x̂ of the input sentence . Task Training : The encoder is frozen ( grey ) , and a mapping Φ is trained ( green ) on input embeddings zx to output predictions ẑy such that it minimize some loss L ( ẑy ) . Inference : To obtain textual predictions ŷ , the encoder is composed with Φ and the decoder . AEs the same kind of representations as attention-based models . But this added expressivity comes with additional challenges , as illustrated in Figure 1 . In the single-vector case , an operation Φ in the vector space consists of a simple vector-to-vector mapping . But with BoV-AEs , Φ needs to map a bag of vectors onto another bag of vectors , which requires more complicated operations . In this paper , we demonstrate how such a mapping can be learned in the context of the Emb2Emb framework by making the following novel contributions : ( i ) We propose a regularization scheme for BoV-AEs , ( ii ) a neural mapping architecture Φ for Emb2Emb , and ( iii ) a suitable training loss . Empirically , we show on two unsupervised conditional text generation tasks , sentiment transfer ( Shen et al. , 2017 ) and sentence summarization ( Rush et al. , 2015 ) , that BoV-AEs perform substantially better than standard AEs if the text is too long to be captured by one vector alone . Our ablation studies confirm that our technical contributions are crucial for this success . In the following section , we review the Emb2Emb framework , before we introduce BoV-AE ( Section 3 ) and its integration within Emb2Emb ( Section 4 ) . 2 BACKGROUND : Emb2Emb Embedding-to-Embedding ( Emb2Emb ) was introduced by Mai et al . ( 2020 ) as both a supervised and an unsupervised framework for conditional text generation . The core idea is to disentangle the transition from the discrete text space to a continuous latent space from the specific task , allowing for larger-scale pretraining with unlabeled data . The workflow of the framework is depicted in Figure 2 . First , a text AE A = dec ◦ enc is trained to map an input sentence from the discrete text space X to an embedding space Z via the encoder enc : X → Z , and back to X via a decoder dec : Z → X , such thatA ( x ) = x , typically trained via negative log-likelihood , Lrec = NLL ( A ( x ) , x ) . In contrast to other methods , A can in principle be any AE , opening the possibility for large-scale AE pretraining with unlabeled data . Second , taskspecific training is performed only in the embedding space Z of the AE . To this end , the encoder is frozen , and a new mapping layer Φ : Z → Z is introduced , which is trained to transform the embedding of the input zx into the embedding of the predicted output ẑy . The concrete loss L ( ẑy ) depends on the type of task . In the supervised case , the true output is also encoded into space Z , and the distance between the true embedding and the predicted embedding is minimized . In the unsupervised case , the loss needs to be defined for the specific task at hand . For example , for sentiment transfer , where the goal is to transform a negative review into a positive review while retaining as much of the input as possible , Mai et al . ( 2020 ) compose the loss as a combination of two loss terms1 , L ( ẑy ) = Lsim ( zx , ẑy ) + λstyLsty ( ẑy ) . Lsty encourages ẑy to be classified as a positive review according to a separately trained sentiment classifier . Lsim encourages the output to be close to the input in embedding space , e.g . via euclidean distance . λsty is a hyperparameter that controls the importance of changing the sentiment of the predicted output . A main question in Emb2Emb is how to choose the embedding spaceZ . Mai et al . ( 2020 ) use a single continuous vector to encode all the information of the input , i.e . Z = Rd . This choice simplifies the mapping Φ to an MLP and the training loss to vector space distances , which is relatively easy to train . On the other hand , it limits the model in fundamental ways : The representation is fixed-sized , i.e. , the representation can not grow in size . Sequence-to-sequence models with a fixed-size bottleneck struggle to encode long text sequences ( Bahdanau et al. , 2015 ) , which is a key reason why attention-based models are now standard practice in sequence-to-sequence models . Hence , it would be desirable to adapt Emb2Emb in such a way that Z contains variable-sized embeddings instead . 3 BAG-OF-VECTORS AUTOENCODER . We propose Bag-of-Vectors Autoencoders ( BoV-AEs ) to be used with Emb2Emb . Following the naming convention by Henderson ( 2020 ) , we refer to a bag of vectors as a ( multi ) -set of vectors that ( i ) can grow arbitrarily large , and ( ii ) where the elements are not ordered ( a basic property of sets ) . A type of BoV representation that is used very commonly is found in Transformer ( Vaswani et al. , 2017 ) encoder-decoder models , where there is one vector to represent each token of the input text , and the order of the vectors does not matter when the decoder accesses the output of the encoder . In this work , we also rely on Transformer models as the backbone of our encoders and decoders . However , in principle , any encoder and decoder can be used , as long as the encoder produces a bag as output and the decoder takes a bag as input . Formally , Z = ( Rd ) + , so the encoder produces a bag-of-vectors X = { z1 , ... , zn } : = enc ( x ) , where n is the number of vectors in the input bag . 3.1 REGULARIZATION . The fact that we use a BoV-based AE presents a major challenge : AEs have to be regularized to prevent them from learning a simple identity mapping where the input is merely copied to the output , which does not result in a meaningful embedding space . In fixed-size embeddings , this is for example achieved through under-completeness ( choosing a latent dimension that is smaller than the input dimension ) or through injection of noise , either at the input or in the embedding space . While there exists a lot of research on regularizing fixed-sized AEs , it is not clear how to achieve the same goal in a BoV-AE . Here , regularizing the capacity of each vector is not enough . As long as each vector can store a ( constant ) positive amount of information , a bag of unlimited size can still store infinite information . However , it is not clear to what extent the size of the bag needs to be restricted . By default , a standard Transformer model produces as many vectors as there are input tokens , but this is likely too many , as it makes copying from the input to the output trivial . Hence , we want the encoder to output fewer vectors . In the following we explain how this is achieved in BoV-AEs . L0Drop Ideally , we want the model to decide for itself on a per-example basis which vectors it needs to retain for reconstruction . To this end , we adopt L0Drop , a differentiable approximation to L0 regularization , which was originally developed by Zhang et al . ( 2021 ) for the purpose of speeding up a model through sparsification . The model computes scalar gates gi = g ( zi ) ∈ [ 0 , 1 ] ( which can be exactly zero or one ) for each encoder output . After the gates are computed , we multiply them with their corresponding vector . Vectors whose gates are near zero ( i.e. , smaller than some > 0 ) are removed from the bag entirely . An additional loss term , LL0 ( X ) = λL0 ∑n i gi encourages the model to close as many gates as possible , where the hyperparameter λL0 controls the sparsity rate implicitly . However , in initial experiments , we found λL0 difficult to tune , as it is very sensitive with respect to other hyperparameters . We instead employ a modified loss that seeks to explicitly match 1Their total loss includes an adversarial component that encourages the outputs of the mapping to stay on the latent space manifold . We leave adaptation of this component to the BoV scenario for future work . a certain target ratio r of open gates . Similar to the free-bits objective that is used to prevent the posterior collapse problem in VAEs ( Kingma et al. , 2016 ) , the objective becomes LL0 ( X ) = λL0 max ( r , 1n ∑n i gi ) . ( 1 ) By setting λL0 to a large enough value ( empirically , λL0 = 10 ) , we find that this objective reaches the target ratio r reliably for different r while at the same time reducing the reconstruction loss . This allows to compare different strengths of regularization while reducing the tuning effort substantially .

Summary: This paper extends previous work, Emb2Emb, by replacing the LSTM autoencoder with a Transformer autoencoder. The authors propose some techniques to handle the difficulty of this replacement. They also conduct some experiments to support the claim.