A Bilingual Generative Transformer for Semantic Sentence Embedding

1 INTRODUCTION . Learning useful representations of language has been a source of recent success in natural language processing ( NLP ) . Much work has been done on learning representations for words ( Mikolov et al. , 2013 ; Pennington et al. , 2014 ) and sentences ( Kiros et al. , 2015 ; Conneau et al. , 2017 ) . More recently , deep neural architectures have been used to learn contextualized word embeddings ( Peters et al. , 2018 ; Devlin et al. , 2018 ) which have enabled state-of-the-art results on many tasks . We focus on learning semantic sentence embeddings in this paper , which play an important role in many downstream applications . Since they do not require any labelled data for fine-tuning , sentence embeddings are useful for a variety of problems right out of the box . These include Semantic Textual Similarity ( STS ; Agirre et al . ( 2012 ) ) , mining bitext ( Zweigenbaum et al. , 2018 ) , and paraphrase identification ( Dolan et al. , 2004 ) . Semantic similarity measures also have downstream uses such as fine-tuning machine translation systems ( Wieting et al. , 2019a ) . There are three main ingredients when designing a sentence embedding model : the architecture , the training data , and the objective function . Many architectures including LSTMs ( Hill et al. , 2016 ; Conneau et al. , 2017 ; Schwenk & Douze , 2017 ; Subramanian et al. , 2018 ) , Transformers ( Cer et al. , 2018 ; Reimers & Gurevych , 2019 ) , and averaging models ( Wieting et al. , 2016a ; Arora et al. , 2017 ) have found success for learning sentence embeddings . The choice of training data and objective are intimately intertwined , and there are a wide variety of options including next-sentence prediction ( Kiros et al. , 2015 ) , machine translation ( Espana-Bonet et al. , 2017 ; Schwenk & Douze , 2017 ; Schwenk , 2018 ; Artetxe & Schwenk , 2018 ) , natural language inference ( NLI ) ( Conneau et al. , 2017 ) , and multi-task objectives which include some of the previously mentioned objectives ( Cer et al. , 2018 ) as well as additional tasks like constituency parsing ( Subramanian et al. , 2018 ) . Surprisingly , despite ample testing of more powerful architectures , the best performing models for many sentence embedding tasks related to semantic similarity often use simple architectures that are mostly agnostic to the interactions between words . For instance , some of the top performing techniques use word embedding averaging ( Wieting et al. , 2016a ) , character n-grams ( Wieting et al. , 2016b ) , and subword embedding averaging ( Wieting et al. , 2019b ) to create representations . These simple approaches are competitive with much more complicated architectures on in-domain data and generalize well to unseen domains , but are fundamentally limited by their inability to capture word order . Training these approaches generally relies on discriminative objectives defined on paraphrase data ( Ganitkevitch et al. , 2013 ; Wieting & Gimpel , 2018 ) or bilingual data ( Wieting et al. , 2019b ) . The inclusion of latent variables in these models has also been explored ( Chen et al. , 2019 ) . Intuitively , bilingual data in particular is promising because it potentially offers a useful signal for learning the underlying semantics of sentences . Within a translation pair , properties shared by both sentences are more likely semantic , while those that are divergent are more likely stylistic or language-specific . While previous work learning from bilingual data perhaps takes advantage of this fact implicitly , the focus of this paper is modelling this intuition explicitly , and to the best of our knowledge , this has not not been explored in prior work . Specifically , we propose a deep generative model that is encouraged to perform source separation on parallel sentences , isolating what they have in common in a latent semantic embedding and explaining what is left over with language-specific latent vectors . At test time , we use inference networks ( Kingma & Welling , 2013 ) for approximating the model ’ s posterior on the semantic and source-separated latent variables to encode monolingual sentences . Finally , since our model and training objective are generative , our approach does not require knowledge of the distance metrics to be used during evaluation,1 and it has the additional property of being able to generate text . In experiments , we evaluate our probabilistic source-separation approach on a standard suite of STS evaluations . We demonstrate that the proposed approach is effective , most notably allowing the learning of high-capacity deep transformer architectures ( Vaswani et al. , 2017 ) while still generalizing to new domains , significantly outperforming a variety of state-of-the-art baselines . Further , we conduct a thorough analysis by identifying subsets of the STS evaluation where simple word overlap is not able to accurately assess semantic similarity . On these most difficult instances , we find that our approach yields the largest gains , indicating that our system is modeling interactions between words to good effect . We also find that our model better handles cross-lingual semantic similarity than multilingual translation baseline approaches , indicating that stripping away language-specific information allows for better comparisons between sentences from different languages . Finally , we analyze our model to uncover what information was captured by the source separation into the semantic and language-specific variables and the relationship between this encoded information and language distance to English . We find that the language-specific variables tend to explain more superficial or language-specific properties such as overall sentence length , amount and location of punctuation , and the gender of articles ( if gender is present in the language ) , but semantic and syntactic information is more concentrated in the shared semantic variables , matching our intuition . Language distance has an effect as well , where languages that share common structures with English put more information into the semantic variables , while more distant languages put more information into the language-specific variables . Lastly , we show outputs generated from our model that exhibit its ability to do a type of style transfer . 2 MODEL . Our proposed training objective leverages a generative model of parallel text in two languages ( e.g . English ( en ) and French ( fr ) ) that form a pair consisting of an English sentence xen and a French sentence xfr . Importantly , this generative process utilizes three underlying latent vectors : languagespecific variation variables ( language variables ) zfr and zen respectively for each side of the translation , as well as a shared semantic variation variable ( semantic variable ) zsem . In this section we will first describe the generative model for the text and latent variables . In the following section we will describe the inference procedure of zsem given an input sentence , which corresponds to our core task of obtaining sentence embeddings useful for downstream tasks such as semantic similarity . Further , by encouraging the model to perform this source separation , the learned semantic encoders will more crisply represent the underlying semantics , increasing performance on downstream semantic tasks . 1In other words , we don ’ t assume cosine similarity as a metric , though it does work well in our experiments . The generative process of our model , the Bilingual Generative Transformer ( BGT ) , is depicted in Figure 1 and its computation graph is shown in Figure 2 . First , we sample latent variables 〈zfr , zen , zsem〉 , where zi ∈ Rk , from a multivariate Gaussian prior N ( 0 , Ik ) . These variables are then fed into a decoder that samples sentences ; xen is sampled conditioned on zsem and zen , while xfr is sampled conditioned on zsem and zfr . Because sentences in both languages will use zsem in generation , we expect that in a well- trained model this variable will encode semantic , syntactic , or stylistic information shared across both sentences , while zfr and zen will handle any language-specific peculiarities or specific stylistic decisions that are less central to the sentence meaning and thus do not translate across sentences . In the following section , we further discuss how this is explicitly encouraged by the learning process . Decoder Architecture . Many latent variable models for text use LSTMs ( Hochreiter & Schmidhuber , 1997 ) as their decoders ( Yang et al. , 2017 ; Ziegler & Rush , 2019 ; Ma et al. , 2019 ) . However , state-of-the-art models in neural machine translation have seen increased performance and speed using deep Transformer architectures . We also found in our experiments ( see Appendix C for details ) that Transformers led to increased performance in our setting , so they are used in our main model . We use two decoders in our model , one for modelling p ( xfr|zsem , zfr ; θ ) and one for modeling p ( xen|zsem , zen ; θ ) . These decoders are depicted on the right side of Figure 2 . Each decoder takes in two latent variables , a language variable and a semantic variable . These variables are concatenated together prior to being used by the decoder for reconstruction . We explore four ways of using this latent vector : ( 1 ) Concatenate it to the word embeddings ( Word ) ( 2 ) Use it as the initial hidden state ( Hidden , LSTM only ) ( 3 ) Use it as you would the attention context vector in the traditional sequenceto-sequence framework ( Attention ) and ( 4 ) Concatenate it to the hidden state immediately prior to computing the logits ( Logit ) . Unlike Attention , there is no additional feedforward layer in this setting . We experimented with these four approaches , as well as combinations thereof , and report this analysis in Appendix A . From these experiments , we see that the closer the sentence embedding is to the softmax , the better the performance on downstream tasks evaluating its semantic content . We hypothesise that this is due to better gradient propagation because the sentence embedding is now closer to the error signal . Since Attention and Logit performed best , we use these in our Transformer experiments . 3 LEARNING AND INFERENCE . Our model is trained on a training set X of parallel text consisting of N examples , X = { 〈x1en , x1fr〉 , . . . , 〈xNen , xNfr〉 } , and Z is our collection of latent variables Z = ( 〈z1en , z1fr , z1sem〉 , . . . , 〈zNen , zNfr , zNsem〉 ) . We wish to maximize the likelihood of the parameters of the two decoders θ with respect to the observed X , marginalizing over the latent variables Z. p ( X ; θ ) = ∫ Z p ( X , Z ; θ ) dZ Unfortunately , this integral is intractable due to the complex relationship between X and Z . However , related latent variable models like variational autoencoders ( VAEs ( Kingma & Welling , 2013 ) ) learn by optimizing a variational lower bound on the log marginal likelihood . This surrogate objective is called the evidence lower bound ( ELBO ) and introduces a variational approximation , q to the true posterior of the model p. The q distribution is parameterized by a neural network with parameters φ. ELBO can be written for our model as follows : ELBO =Eq ( Z|X ; φ ) [ log p ( X|Z ; θ ) ] − KL ( q ( Z|X ; φ ) ||p ( Z ; θ ) ) This lower bound on the marginal can be optimized by gradient ascent by using the reparameterization trick ( Kingma & Welling , 2013 ) . This trick allows for the expectation under q to be approximated through sampling in a way that preserves backpropagation . We make several independence assumptions for q ( zsem , zen , zfr|xen , xfr ; φ ) . Specifically , to match our goal of source separation , we factor q as q ( zsem , zen , zfr|xen , xfr ; φ ) = q ( zsem|xen , xfr ; φ ) q ( zen|xen ) q ( zfr|xfr ; φ ) , with φ being the parameters of the encoders that make up the inference networks , defined in the next paragraph . Lastly , we note that the KL term in our ELBO equation encourages explaining variation that is shared by translations with the shared semantic variable and explaining language-specific variation with the corresponding language-specific variables . Information shared by the two sentences will result in a lower KL loss if it is encoded in the shared variable , otherwise that information will be replicated and the overall cost of encoding will increase . Encoder Architecture . We use three inference networks as shown on the left side of Figure 2 : an English inference network to produce the English language variable , a French inference network to produce the French language variable , and a semantic inference network to produce the semantic variable . Just as in the decoder architecture , we use a Transformer for the encoders . The semantic inference network is a bilingual encoder that encodes each language . For each translation pair , we alternate which of the two parallel sentences is fed into the semantic encoder within a batch . Since the semantic encoder is meant to capture language agnostic semantic information , its outputs for a translation pair should be similar regardless of the language of the input sentence . We note that other operations are possible for combining the views each parallel sentence offers . For instance , we could feed both sentences into the semantic encoder and pool their representations . However , in practice we find that alternating works well and leave further study of this to future work .

Summary: This paper addresses the problem of constructing a sentence embedding using a generative transformer model which encodes semantic aspects and language-specific aspect separately. They use transformers to encode and decode sentence embedding, and the objective reconstructs input with a latent variables (language variables for each language and semantic language).  These latent variables are sampled from multivariate Gaussian prior, and the learning uses evidence lower bound (ELBO) for variational approximation of the joint distribution of latent variables and input.