Bag-of-Vectors Autoencoders for Unsupervised Conditional Text Generation

1 INTRODUCTION . In conditional text generation , we would like to produce an output text given an input text . Hence , parallel input-output pairs are required to train a good supervised machine learning model on this type of task . Large-scale pretraining ( Peters et al. , 2018 ; Devlin et al. , 2019 ; Lewis et al. , 2020 ) can alleviate the necessity for training examples to some extent , but even this requires a substantial number of annotations ( Yogatama et al. , 2019 ) . This is an expensive process and can introduce unwanted artifacts itself , which are henceforth learned by the model ( Gururangan et al. , 2018 ) . For these reasons , there is substantial interest in unsupervised solutions . Text autoencoders ( AEs ) don ’ t require labeled data for training , and are therefore a popular model for unsupervised approaches to many tasks , such as machine translation ( Artetxe et al. , 2018 ) , sentence compression ( Févry & Phang , 2018 ) and sentiment transfer ( Shen et al. , 2017 ) . The classical text AE ( Bowman et al. , 2016 ) embeds the input text into a single fixed-size vector via the encoder , and then tries to reconstruct the input text from the single vector via the decoder . Single-vector embeddings are very useful , because they allow to perform conditional text generation through simple mappings in the embedding space , e.g . by adding a constant offset vector to change attributes such as sentiment ( Shen et al. , 2020 ) . Recently , Mai et al . ( 2020 ) proposed Emb2Emb , a method that can learn these mappings directly in the embedding space of any pretrained single-vector AE . This is a powerful framework , because the AE can then be pretrained on virtually infinite amounts of unlabeled data before applying it to any downstream application . This concept , transfer learning , is arguably one of the most important drivers of progress in machine learning in the recent decade : These so-called Foundation Models ( Bommasani et al. , 2021 ) have revolutionized natural language understanding ( e.g , BERT ( Devlin et al. , 2019 ) ) and computer vision ( e.g , DALL-E ( Ramesh et al. , 2021 ) ) , among others . Since Emb2Emb was designed to work with any pretrained AE , it was an important step towards their scalability . However , as Bommasani et al . ( 2021 ) point out , another crucial model property is expressivity , the ability to represent the data distribution it is trained on . In this regard , single-vector representations are fundamentally limited ; they act as a bottleneck , causing the model to increasingly struggle to encode longer text ( Bahdanau et al. , 2015 ) . In this paper , we extent Emb2Emb from single-vector bottleneck AEs to Bag-of-Vector Autoencoders ( BoV-AEs ) , which encode text into a variable-size representation where the number of vectors grows with the length of the text . This gives BoV- enc dec Text Autoencoder Pretraining Inference enc dec enc Task Training Figure 2 : High-level view of the Emb2Emb framework . Text Autoencoder Pretraining : An autoencoder is trained on an unlabeled corpus , i.e. , the encoder enc transforms an input text x into a continuous embedding zx , which is in turn used by the decoder dec to predict a reconstruction x̂ of the input sentence . Task Training : The encoder is frozen ( grey ) , and a mapping Φ is trained ( green ) on input embeddings zx to output predictions ẑy such that it minimize some loss L ( ẑy ) . Inference : To obtain textual predictions ŷ , the encoder is composed with Φ and the decoder . AEs the same kind of representations as attention-based models . But this added expressivity comes with additional challenges , as illustrated in Figure 1 . In the single-vector case , an operation Φ in the vector space consists of a simple vector-to-vector mapping . But with BoV-AEs , Φ needs to map a bag of vectors onto another bag of vectors , which requires more complicated operations . In this paper , we demonstrate how such a mapping can be learned in the context of the Emb2Emb framework by making the following novel contributions : ( i ) We propose a regularization scheme for BoV-AEs , ( ii ) a neural mapping architecture Φ for Emb2Emb , and ( iii ) a suitable training loss . Empirically , we show on two unsupervised conditional text generation tasks , sentiment transfer ( Shen et al. , 2017 ) and sentence summarization ( Rush et al. , 2015 ) , that BoV-AEs perform substantially better than standard AEs if the text is too long to be captured by one vector alone . Our ablation studies confirm that our technical contributions are crucial for this success . In the following section , we review the Emb2Emb framework , before we introduce BoV-AE ( Section 3 ) and its integration within Emb2Emb ( Section 4 ) . 2 BACKGROUND : Emb2Emb Embedding-to-Embedding ( Emb2Emb ) was introduced by Mai et al . ( 2020 ) as both a supervised and an unsupervised framework for conditional text generation . The core idea is to disentangle the transition from the discrete text space to a continuous latent space from the specific task , allowing for larger-scale pretraining with unlabeled data . The workflow of the framework is depicted in Figure 2 . First , a text AE A = dec ◦ enc is trained to map an input sentence from the discrete text space X to an embedding space Z via the encoder enc : X → Z , and back to X via a decoder dec : Z → X , such thatA ( x ) = x , typically trained via negative log-likelihood , Lrec = NLL ( A ( x ) , x ) . In contrast to other methods , A can in principle be any AE , opening the possibility for large-scale AE pretraining with unlabeled data . Second , taskspecific training is performed only in the embedding space Z of the AE . To this end , the encoder is frozen , and a new mapping layer Φ : Z → Z is introduced , which is trained to transform the embedding of the input zx into the embedding of the predicted output ẑy . The concrete loss L ( ẑy ) depends on the type of task . In the supervised case , the true output is also encoded into space Z , and the distance between the true embedding and the predicted embedding is minimized . In the unsupervised case , the loss needs to be defined for the specific task at hand . For example , for sentiment transfer , where the goal is to transform a negative review into a positive review while retaining as much of the input as possible , Mai et al . ( 2020 ) compose the loss as a combination of two loss terms1 , L ( ẑy ) = Lsim ( zx , ẑy ) + λstyLsty ( ẑy ) . Lsty encourages ẑy to be classified as a positive review according to a separately trained sentiment classifier . Lsim encourages the output to be close to the input in embedding space , e.g . via euclidean distance . λsty is a hyperparameter that controls the importance of changing the sentiment of the predicted output . A main question in Emb2Emb is how to choose the embedding spaceZ . Mai et al . ( 2020 ) use a single continuous vector to encode all the information of the input , i.e . Z = Rd . This choice simplifies the mapping Φ to an MLP and the training loss to vector space distances , which is relatively easy to train . On the other hand , it limits the model in fundamental ways : The representation is fixed-sized , i.e. , the representation can not grow in size . Sequence-to-sequence models with a fixed-size bottleneck struggle to encode long text sequences ( Bahdanau et al. , 2015 ) , which is a key reason why attention-based models are now standard practice in sequence-to-sequence models . Hence , it would be desirable to adapt Emb2Emb in such a way that Z contains variable-sized embeddings instead . 3 BAG-OF-VECTORS AUTOENCODER . We propose Bag-of-Vectors Autoencoders ( BoV-AEs ) to be used with Emb2Emb . Following the naming convention by Henderson ( 2020 ) , we refer to a bag of vectors as a ( multi ) -set of vectors that ( i ) can grow arbitrarily large , and ( ii ) where the elements are not ordered ( a basic property of sets ) . A type of BoV representation that is used very commonly is found in Transformer ( Vaswani et al. , 2017 ) encoder-decoder models , where there is one vector to represent each token of the input text , and the order of the vectors does not matter when the decoder accesses the output of the encoder . In this work , we also rely on Transformer models as the backbone of our encoders and decoders . However , in principle , any encoder and decoder can be used , as long as the encoder produces a bag as output and the decoder takes a bag as input . Formally , Z = ( Rd ) + , so the encoder produces a bag-of-vectors X = { z1 , ... , zn } : = enc ( x ) , where n is the number of vectors in the input bag . 3.1 REGULARIZATION . The fact that we use a BoV-based AE presents a major challenge : AEs have to be regularized to prevent them from learning a simple identity mapping where the input is merely copied to the output , which does not result in a meaningful embedding space . In fixed-size embeddings , this is for example achieved through under-completeness ( choosing a latent dimension that is smaller than the input dimension ) or through injection of noise , either at the input or in the embedding space . While there exists a lot of research on regularizing fixed-sized AEs , it is not clear how to achieve the same goal in a BoV-AE . Here , regularizing the capacity of each vector is not enough . As long as each vector can store a ( constant ) positive amount of information , a bag of unlimited size can still store infinite information . However , it is not clear to what extent the size of the bag needs to be restricted . By default , a standard Transformer model produces as many vectors as there are input tokens , but this is likely too many , as it makes copying from the input to the output trivial . Hence , we want the encoder to output fewer vectors . In the following we explain how this is achieved in BoV-AEs . L0Drop Ideally , we want the model to decide for itself on a per-example basis which vectors it needs to retain for reconstruction . To this end , we adopt L0Drop , a differentiable approximation to L0 regularization , which was originally developed by Zhang et al . ( 2021 ) for the purpose of speeding up a model through sparsification . The model computes scalar gates gi = g ( zi ) ∈ [ 0 , 1 ] ( which can be exactly zero or one ) for each encoder output . After the gates are computed , we multiply them with their corresponding vector . Vectors whose gates are near zero ( i.e. , smaller than some > 0 ) are removed from the bag entirely . An additional loss term , LL0 ( X ) = λL0 ∑n i gi encourages the model to close as many gates as possible , where the hyperparameter λL0 controls the sparsity rate implicitly . However , in initial experiments , we found λL0 difficult to tune , as it is very sensitive with respect to other hyperparameters . We instead employ a modified loss that seeks to explicitly match 1Their total loss includes an adversarial component that encourages the outputs of the mapping to stay on the latent space manifold . We leave adaptation of this component to the BoV scenario for future work . a certain target ratio r of open gates . Similar to the free-bits objective that is used to prevent the posterior collapse problem in VAEs ( Kingma et al. , 2016 ) , the objective becomes LL0 ( X ) = λL0 max ( r , 1n ∑n i gi ) . ( 1 ) By setting λL0 to a large enough value ( empirically , λL0 = 10 ) , we find that this objective reaches the target ratio r reliably for different r while at the same time reducing the reconstruction loss . This allows to compare different strengths of regularization while reducing the tuning effort substantially .

Summary: This paper proposes a text autoencoder with a bag-of-vectors embedding, which is more capable of encoding longer text than a single-vector embedding. To train such a model, the authors first encode the input sequence into a vector sequence of equal length, and then add a gate to each vector and encourage the model to keep a small number of open gates.  The proposed BoV-AE can be used in (non-parallel) seq2seq tasks by learning a mapping within the embedding space, similar to the Emb2Emb method proposed by Mai et al. The difference is that the mapping is now between two bags of vectors instead of between two vectors. The authors use a Transformer with a copy mechanism and an offset vector to produce a bag of vectors. To align two bags of vectors, the authors adjust the average Hausdorff distance to a soft version to make training smoother.  Experiments are conducted on a sentiment transfer task and a sentence summarization task. BoW-AE shows better performance compared to AE with a single-vector embedding.