Universal Sentence Representations Learning with Conditional Masked Language Model

1 INTRODUCTION . Sentence embeddings map sentences into a vector space . The vectors capture rich semantic information that can be used to measure semantic textual similarity ( STS ) between sentences or train classifiers for a broad range of downstream tasks ( Conneau et al. , 2017 ; Subramanian et al. , 2018 ; Logeswaran & Lee , 2018b ; Cer et al. , 2018 ; Reimers & Gurevych , 2019 ; Yang et al. , 2019a ; d ; Giorgi et al. , 2020 ) . State-of-the-art models are usually trained on supervised tasks such as natural language inference ( Conneau et al. , 2017 ) , or with semi-structured data like question-answer pairs ( Cer et al. , 2018 ) and translation pairs ( Subramanian et al. , 2018 ; Yang et al. , 2019a ) . However , labeled and semi-structured data are difficult and expensive to obtain , making it hard to cover many domains and languages . Conversely , recent efforts to improve language models include the development of masked language model ( MLM ) pre-training from large scale unlabeled corpora ( Devlin et al. , 2019 ; Lan et al. , 2020 ; Liu et al. , 2019 ) . While internal MLM model representations are helpful when fine-tuning on downstream tasks , they do not directly produce good sentence representations , without further supervised ( Reimers & Gurevych , 2019 ) or semi-structured ( Feng et al. , 2020 ) finetuning . In this paper , we explore an unsupervised approach , called Conditional Masked Language Modeling ( CMLM ) , to effectively learn sentence representations from large scale unlabeled corpora . CMLM integrates sentence representation learning into MLM training by conditioning on sentence level representations produced by adjacent sentences . The model therefore needs to learn effective sentence representations in order to perform good MLM . Since CMLM is fully unsupervised , it can be easily extended to new languages . We explore CMLM for both English and multilingual sentence embeddings for 100+ languages . Our English CMLM model achieves state-of-the-art performance on SentEval ( Conneau & Kiela , 2018 ) , even outperforming models learned using ( semi- ) supervised signals . Moreover , models training on the English Amazon review data using our multilingual vectors exhibit strong multilingual transfer performance on translations of the Amazon review evaluation data to French , German and Japanese , outperforming existing multilingual sentence embedding models by > 5 % for non-English languages and by > 2 % on the original English data . We further extend the multilingual CMLM to co-training with parallel text ( bitext ) retrieval task , and finetuning with cross-lingual natural language inference ( NLI ) data , inspired by the success of prior work on multitask sentence representation learning ( Subramanian et al. , 2018 ; Yang et al. , 2019a ) and NLI learning ( Conneau et al. , 2017 ; Reimers & Gurevych , 2019 ) . We achieve performance 1.4 % better than the previous state-of-the-art multilingual sentence representation model ( see details in section 4.2 ) . Language agnostic representations require semantically similar cross-lingual pairs to be closer in representation space than unrelated same-language pairs ( Roy et al. , 2020 ) . While we find our original sentence embeddings do have a bias for same language sentences , we discover that removing the first few principal components of the embeddings eliminates the self language bias . The rest of the paper is organized as follows . Section 2 describes the architecture for CMLM unsupervised learning . In Section 3 we present CMLM trained on English data and evaluation results on SentEval . In Section 3 we apply CMLM to learn sentence multilingual sentence representations . Multitask training strategies on how to effectively combining CMLM , bitext retrieval and cross lingual NLI finetuning are explored . In Section 5 , we investigate self language bias in multilingual representations and how to eliminate it . The contributions of this paper can be summarized as follows : ( 1 ) A novel pre-training technique CMLM for unsupervised sentence representation learning on unlabeled corpora ( either in monolingual and multilingual ) . ( 2 ) An effective multitask training framework , which combines unsupervised learning task CMLM with supervised learning Bitext Retrieval and cross-lingual NLI finetuning . ( 3 ) An evaluation benchmark for multilingual sentence representations . ( 4 ) A simple and effective algebraic method to remove same language bias in multilingual representations . 2 CONDITIONAL MASKED LANGUAGE MODELING . We introduce Conditional Masked Language Modeling ( CMLM ) as a novel architecture for combining next sentence prediction with MLM training . By “ conditional , ” we mean the MLM task for one sentence depends on the encoded sentence level representation of the adjacent sentence . This builds on prior work on next sentence prediction that has been widely used for learning sentence level representations ( Kiros et al. , 2015 ; Logeswaran & Lee , 2018b ; Cer et al. , 2018 ; Yang et al. , 2019a ) , but has thus far produced poor quality sentence embeddings within BERT based models ( Reimers & Gurevych , 2019 ) . While existing MLMs like BERT include next sentence prediction tasks , they do so without any inductive bias to try to encode the meaning of a sentence within a single embedding vector . We introduce a strong inductive bias for learning sentence embeddings by structuring the task as follows . Given a pair of ordered sentences , the first sentence is fed to an encoder that produces a sentence level embedding . The embedding is then provided to an encoder that conditions on the sentence embedding in order to better perform MLM prediction over the second sentence . This is notably similar to Skip-Thought ( Kiros et al. , 2015 ) , but replaces the generation of the complete second sentence with the MLM denoising objective . It is also similar to T5 ’ s MLM inspired unsupervised encode-decoder objective ( Raffel et al. , 2019 ) , with the second encoder acting as a sort of decoder given the representation produced for the first sentence . Our method critically differs from T5 ’ s in that a sentence embedding bottleneck is used to pass information between two model components and in that the task involves denoising a second sentence when conditioning on the first rather than denoising a single text stream . Fig . 1 illustrates the architecture of our model . The first sentence s1 is tokenized and input to a transformer encoder and a sentence vector v 2 Rd is computed from the sequence outputs by average pooling.1 The sentence vector v is then projected into N spaces with one of the projections being the identity mapping , i.e . vp = P ( v ) 2 Rd⇥N . Here we use a three-layer MLP as the projection P ( · ) . The second sentence s2 is then masked following the procedure described in the original BERT paper , including random replacement and the use of unchanged tokens . The second encoder shares the same weights with the encoder used to embed s1 2 . Tokens in the masked s2 are converted into word vectors and concatenated with vp . The concatenated representations are provided to the transformer encoder to predict the masked tokens in s2 . At inference time , we keep the first encoding module and discard the subsequent MLM prediction . In Section 5.2 , we explore various different configurations of CMLM , including the number of projection spaces , and how the projected vectors are connected to the embeddings of the second sentence . 3 LEARNING ENGLISH SENTENCE REPRESENTATIONS WITH CMLM . For training English sentence encoders with CMLM , we use three Common Crawl dumps . The data are filtered by a classifier which detects whether a sentence belongs to the main content of the web page or not . We use WordPiece tokenization and the vocabulary is the same as public English uncased BERT . In order to enable the model to learn bidirectional information , for two consecutive sequences s1 and s2 , we swap their order for 50 % of the time . This order-swapping process echos with the preceding and succeeding sentences prediction in Skip-Thought ( Kiros et al. , 2015 ) . The length of s1 and s2 are set to be 256 tokens ( the maximum length ) . The number of masked tokens in s2 are 80 ( 31.3 % ) , moderately higher than classical BERT . This change in the ratio of masked tokens is to make the task more challenging , due to the fact that in CMLM , language modeling has access to extra information from adjacent sentences . We train with batch size of 2048 for 1 million steps . The optimizer is LAMB with learning rate of 10 3 , 1 = 0.9 , 2 = 0.999 , warm-up in the first 10,000 steps and linear decay afterwards . We explore two transformer configurations , base and large , same as in the original BERT paper . The number of projections N is 15 by experimenting with multiple choices . 3.1 EVALUATION . We evaluate the sentence representations on the following tasks : ( 1 ) classification : MR ( movie reviews Pang & Lee ( 2005 ) ) , binary SST ( sentiment analysis , Socher et al . ( 2013 ) ) , TREC ( question-type , Voorhees & Tice ( 2000 ) ) , CR ( product reviews , Hu & Liu ( 2004 ) ) , SUBJ ( subjectivity/objectivity , Pang & Lee ( 2004 ) ) . ( 2 ) Entailment : SNLI ( Bowman et al. , 2015 ) and SICK dataset for entailment ( SICK-E , Marelli et al . ( 2014 ) ) . The evaluation is done using SentEval ( Conneau & Kiela , 2018 ) which is a prevailing evaluation toolkit for sentence embeddings . The classifier for the downstream is logistic regression . For each task , the encoder and embeddings are fixed and only downstream neural structures are trained . The baseline sentence embedding models include SkipThought ( Kiros et al. , 2015 ) , InferSent ( Conneau et al. , 2017 ) , USE ( Cer et al. , 2018 ) , QuickThought ( Logeswaran & Lee , 2018a ) and English BERT using standard pre-trained models from TensorFlow Hub website ( Devlin et al . ( 2019 ) , and SBert ( Reimers & Gurevych , 2019 ) . To evaluate the possible improvements coming from training data and processes , we train standard BERT models ( English BERT base/large ( CC ) ) on the same Common Crawl Corpora that CMLM is trained on . Similarly , we also train QuickThought , a competitive unsupervised sentence representations learning model , on the same Common Crawl Corpora ( denoted as “ QuickThought ( CC ) ” ) . To further address the possible advantage from using Transformer encoder , we use a Transformer encoder as the sentence encoder in QuickThought ( CC ) . The representations for BERT are computed by averaging pooling of the sequence outputs 1One can equivalently choose other pooling methods , such as max pooling or use the vector output corresponding to a special token position such as the [ CLS ] token . 2The dual-encoder sharing encoder weights for different inputs can be also referred as “ siamese encoder ” ( we also explore options including [ CLS ] vector and max pooling and the results are available in the appendix ) . 3.2 RESULTS . Evaluation results are presented in Table 1 . CMLM outperforms existing models overall , besting MLM ( both English BERT and English BERT ( CC ) ) using both base and large configurations . The closest competing model is SBERT , which uses supervised NLI data rather than a purely unsupervised approach . Interestingly , CMLM outperforms SBERT on the SICK-E NLI task .

Summary: The authors present conditional masked language modeling (CMLM), a new method for unsupervised pretraining, in which the skip-thought notion of conditioning on neighboring sentences is adopted for masked language modeling. The upshot of the proposed approach is that it generates single sentence embeddings that perform competitively on SentEval. In the multilingual setting, the authors combine their CMLM method with a bitext retrieval objective (selecting a sentence’s translation from the other sentences of the language in the batch) that increases performance on a version of the SentEval tasks translated into 14 other languages. In their analysis, the authors make further claims about multilingual embeddings capturing language ID information in their first principle components, a conclusion somewhat substantiated by their results. The authors provide a small amount of ablation experiments for experimental/model design choices.