Universal Sentence Representations Learning with Conditional Masked Language Model
Authors: Ziyi Yang
Yinfei Yang
Daniel M Cer
Jax Law
Eric Darve
Venue:  (2021)
Keywords: multilingual representations
sentence embeddings

Abstract: This paper presents a novel training method, Conditional Masked Language Modeling (CMLM), to effectively learn sentence representations on large scale unlabeled corpora. CMLM integrates sentence representation learning into MLM training by conditioning on the encoded vectors of adjacent sentences. Our English CMLM model achieves state-of-the-art performance on SentEval, even outperforming models learned using (semi-)supervised signals. As a fully unsupervised learning method, CMLM can be conveniently extended to a broad range of languages and domains. We find that a multilingual CMLM model co-trained with bitext retrieval~(BR) and natural language inference~(NLI) tasks outperforms the previous state-of-the-art multilingual models by a large margin. We explore the same language bias of the learned representations, and propose a principle component based approach to remove the language identifying information from the representation while still retaining sentence semantics.

Under review as a conference paper at ICLR 2021
UNIVERSAL SENTENCE REPRESENTATIONS LEARN- ING WITH CONDITIONAL MASKED LANGUAGE MODEL
Anonymous authors Paper under double-blind review
ABSTRACT
This paper presents novel training method, Conditional Masked Language Mod- eling (CMLM), to effectively learn sentence representations on large scale un- labeled corpora. CMLM integrates sentence representation learning into MI.M training by conditioning on the encoded vectors of adjacent sentences. Our En- glish CMLM model achieves state-of-the-at performance SentEval (Conneau & Kiela 2018), even outperforming models learned using (semi-) semee-suuerriiedd sig- nals. As a fully unsupervised learning method, CMLM can be conveniently ex- tended to a broad range of languages and domains. We find that a multilingual CMLM model co-trained with bitext retrieval (BR) and natural language infer- ence (NLI) tasks outperforms the previous state-of-the-art multilingual models by large margin. We explore the same language bias of the learned representations, and propose a principle component based approach to remove the language identi- fying information from the representation while still retaining sentence semantics.
INTRODUCTION
Sentence embeddings map sentences into vector space. The vectors capture rich semantic infor- mation that can be used to measure semantic textual similarity (STS) between sentences or train
classifiers for broad range of downstream tasks (Conneau al. 2017 Subramanian et al. 2018
Logeswaran & Lee 2018b Cer et al. 2018: Reimers & Gurevych] [019] Yang et (l..1122011g 9a d Giorgi
et 20200. State- oh-the-at models are usually trained on supervised tasks such as natural language inference (Conneau et al. 2017), or with semi-structured data like question-answer pairs (Cer et al., 2018 and translation pairs (Subramanian et al. 2018 Yang et al. 2019a). However, labeled and semi-structured data are difficult and expensive to obtain, making it hard to cover many domains and languages. Conversely, recent efforts to improve language models include the development of masked language model (MLM) pre-training from large scale unlabeled corpora (Devlin et al., 2019 Lan et al.] 2020 Liu et al. 2019). While internal MLM model representations are helpful when fine-tuning downstream tasks, they do not directly produce good sentence representations, without further supervised (Reimers & Gurevych] 2019) or semi-structured (Feng et al. 2020) fine-
tuning.
In this paper, we explore an unsupervised approach, called Conditional Masked Language Modeling (CMLM), effectively learn sentence representations from large scale unlabeled corpora. CMLM integrates sentence representation learning into MLM training by conditioning on sentence level representations produced by adjacent sentences. The model therefore needs learn effective sen- tence representations in order to perform good MLM. Since CMLM is fully unsupervised, it can be easily extended to new languages. We explore CMLM for both English and multilingual sentence embeddings for 100+ languages. Our English CMLM model achieves state-of-ther the---rrtttttttt performance on SentEval (Conneau Kiela, 2018), even outperforming models learned using (semi- jsperviupe signals. Moreover, models training on the English Amazon review data using multilingual vec- tors exhibit strong multilingual transfer performance on translations of the Amazon review evalua- tion data French, German and Japanese, outperforming existing multilingual sentence embedding models by > 5% for non- -on-Ennlish languages and by > 2% on the original English data.
We further extend the multilingual CMLM to co-training with parallel text (bitext) retrieval task, and finetuning with cross-lingual natural language inference (NLI) data, inspired by the success of prior work multitask sentence representation learning (Subramanian et al. 2018 Yang al. 2019a)

---

Under review as a conference paper at ICLR 2021
know, gonna
projection
Transformer Encoder
P(v)
pooling
Transformer Encoder
LLife box of chocolates.
"You never [MASK] what you're [MASK] get
Figure 1: The architecture of Conditional Masked Language Modeling (CMLM).
and NLI learning (Coneeau al. 2017 Reimers & Gurevych 2019). We achieve performance 1.4% better than the previous sfatc-of-the- art multilingual sentence representation model (see details section 4.2) Language agnostic representations require semantically similar cross-lingual pairs to be closer in representation space than unrelated same-language pairs (Roy et al. 2020). While we find our original sentence embeddings do have bias for same language sentences, we discover that removing the first few principal components of the embeddings eliminates the self language bias.
The rest of the paper is organized as follows. Section describes the architecture for CMLM un- supervised learning. In Section we present CMLM trained on English data and evaluation results on SentEval. In Section 3 we apply CMLM learn sentence multilingual sentence representations. Multitask training strategies on how to effectively combining CMLM, bitext retrieval and cross lin- gual NLI finetuning are explored. In Section
5 we investigate self language bias in multilingual
representations and how to eliminate it.
The contributions of this paper can be summarized as follows: (1) A novel pre-training technique CMLM for unsupervised sentence representation learning unlabeled corpora (either in monolin- gual and multilingual). (2) An effective multitask training framework, which combines unsupervised learning task CMLM with supervised learning Bitext Retrieval and cross-lingual NLI finetuning. (3) An evaluation benchmark for multilingual sentence representations.
(4) A simple and effective al- gebraic method remove same language bias in multilingual representations.
2 CONDITIONAL MASKED LANGUAGE MODELING
We introduce Conditional Masked Language Modeling (CMLM) novel architecture for com- bining next sentence prediction with MLM training. By "conditional," we mean the MLM task for sentence depends the encoded sentence level representation of the adjacent sentence. This builds on prior work on next sentence prediction that has been widely used for learning sentence level representations (Kiros al. 2015 Logeswaran & Lee 20186 Cer et al. 2018 Yang al. 2019a), but has thus far produced poor quality sentence embeddings within BERT based models Reimers & Gurevych 2019).
While existing MLMs like BERT include next sentence prediction tasks, they do so without any inductive bias to try to encode the meaning sentence within single embedding vector. We introduce a strong inductive bias for learning sentence embeddings by structuring the task as follows. Given a pair of ordered sentences, the first sentence fed to an encoder that produces sentence level embedding. The embedding is then provided to encoder that conditions on the sentence embedding in order to better perform MLM prediction over the second sentence. This notably similar to Skip-Thought (Kiros et al.) 2015), but replaces the generation of the complete second sentence with the MLM denoising objective. It also similar to T5's MLM inspired unsupervised decoder objective (Raffel al. 2019) with the second encoder acting sort of decoder given the representation produced for the first sentence. Our method critically differs from T5's in that sentence embedding bottleneck is used pass information between model components
2

---

Under review as a conference paper at ICLR 2021
and in that the task involves denoising second sentence when conditioning the first rather than denoising a single text stream.
Fig. illustrates the architecture of our model. The first sentence $1 is tokenized and input to a transformer encoder and a sentence vector v E gd is computed from the sequence outputs by average pooling | The sentence vector is then projected into N spaces with one of the projections being the identity mapping, i.e. vp = P(v) E RdxN Here we use a three-layer MLP as the projection P(-).
The second sentence 82 is then masked following the procedure described in the original BERT paper, including random replacement and the use of unchanged tokens. The second encoder shares the same weights with the encoder used to embed il Tokens in the masked $2 are converted into word vectors and concatenated with Up. The concatenated representations are provided to the transformer encoder to predict the masked tokens in $2. At inference time, we keep the first encoding module and discard the subsequent MLM prediction. In Section 5.2] we explore various different configurations of CMLM, including the number of projection spaces, and how the projected vectors are connected to the embeddings of the second sentence.
3 LEARNING ENGLISH SENTENCE REPRESENTATIONS WITH CMLM
For training English sentence encoders with CMLM, we use three Common Crawl dumps. The data are filtered by classifier which detects whether a sentence belongs to the main content of the web page or not. We use WordPiece tokenization and the vocabulary is the same as public English uncased BERT. In order to enable the model to learn bidirectional information, for two consecutive sequences 81 and $2, we swap their order for 50% of the time. This order-swapping process echos with the preceding and succeeding sentences prediction in Skip-Thought (Kiros al. 2015). The length of 8 and 82 are set to be 256 tokens (the maximum length). The number of masked tokens in 82 are 80 (31.3%), moderately higher than classical BERT. This change in the ratio of masked tokens is to make the task more challenging, due to the fact that in CMLM, language modeling has access to extra information from adjacent sentences. We train with batch size of 2048 for million steps. The optimizer is LAMB with learning rate of 10-3 31 = 0.9, B2 = 0.999, warm-up in the first 10,000 steps and linear decay afterwards. We explore two transformer configurations, base and large, same as in the original BERT paper. The number of projections N is 15 by experimenting with multiple choices.
3.1 EVALUATION
We evaluate the sentence representations the following tasks: (1) classification: MR (movie
reviews Pang & Lee (200)). binary SST (sentiment analysis. Socher et al. (2013)). TREC (question-lype. Voorhees & Tice (2000). CR (product reviews, Hu & Liu (2004). SUBJ (sub-
jectivity/objectiiity, |ang & Lee (204)). (2) Entailment: SNLI (owmma et al. 2015) and SICK dataset for entailment (SICK-E, Marelli al. (2014). The evaluation is done using SentEval (Con-) neau & Kiela 2018) which is prevailing evaluation toolkit for sentence embeddings. The classifier for the downstream logistic regression. For each task, the encoder and embeddings are fixed and only downstream neural structures are trained.
The baseline sentence embedding models include SkipThought (Kiros al. 2015). InferSent (Con- neau et al. 2017). USE (Cer et al., 2018). QuickThought (Logeswaran & Lee 2018a) and English BERT using standard pre-trained models from TensorFlow Hub website (Devlin et al. 2019), and SBert (Reimers & Gurevych 2019). To evaluate the possible improvements coming from train- ing data and processes, we train standard BERT models (English BERT base/large (CC)) on the same Common Crawl Corpora that CMLM is trained
Similarly, we also train a competitive unsupervised sentence representations learning model, on the same Common Crawl Corpora (denoted as "QuickThought (CC)"). To further address the possible advantage from ing Transformer encoder, we use a Transformer encoder as the sentence encoder in QuickThought (CC). The representations for BERT are computed by averaging pooling of the sequence outputs
OOn can equivalently choose other pooling methods, such as max pooling or use the vector output corre- sponding to special token position such as the [CI.S] token.
"The dual-encoder sharing encoder weights for different inputs can be also referred as "siamese encoder"
3

---

Under review as a conference paper at ICLR 2021
(we also explore options including [CLS] vector and max pooling and the results available in the appendix).
3.2 RESULTS
Evaluation results are presented in Table 1 CMLM outperforms existing models overall, besting MLM (both English BERT and English BERT (CC)) using both base and large configurations. The closest competing model is SBERT, which uses supervised NLI data rather than purely unsuper- vised approach. Interestingly, CMLM outperforms SBERT on the SICK-E NLI task.
Model
MR
CR
SUBJ
MPQA
SST
TREC
MRPC
SICK-E
SICK-R
Avg.
ShipThought
76.5
80.1
93.6
87.1
82.0
92.2
82.3
85.8
InferSent
81.6
86.5
92.5
90.4
84.2
88.2
75.8
80.3
83.7
84.7
USE
80.1
85.2
94.0
86.7
86.4
93.2
70.1
82.4
85.9
84.9
QuickThought (CC)
75.7
81.9
94.3
84.7
79.7
83.0
70.4
75.0
78.5
80.4
English BERT base
81.6
87.4
95.2
87.8
85.8
90.6
71.1
79.3
80.5
84.3
English BERT base (CC)
82.5
88.5
95.6
87.3
88.0
91.4
72.0
79.3
79.0
84.6
CMLM base (ours)
83.6
89.9
96.2
89.3
88.5
91.0
69.7
82.3
83.4
86,0
English BERT lurge
84.3
88.9
95.7
86.8
88.9
91.4
71.8
75.7
77.0
84.5
English BERT large (CC)
85,4
89.0
95.7
86.9
90.5
91.2
75.5
74.3
77.0
85.0
SBERT (MNLI SNLI)
84.8
90.0
94.5
90.3
90.7
87.4
76.0
74.9
84.2
85.9
CMLM large (ours)
85.6
89.1
96.6
89.3
91.4
92.4
70.0
82.2
84.5
86.8
Table 1: Transfer learning test set results on SentEval for English models. Baseline models include BERT-based (BERT and SBERT) and non-BERT models (Skip Though,,, InferSent and USE).
4 LEARNING MULTILINGUAL SENTENCE REPRESENTATIONS WITH CMLM
As a fully unsupervised method, CMLM can be conveniently extended to multilingual modeling even for less well resourced languages. Learning good multilingual sentence representations more challenging than learning monolingual ones, especially when attempting to capture the se- mantic alignment between different languages. As CMLM does explicitly address cross-lingual alignment, we explore several modeling approaches besides CMLM: (1) Co-training CMLM with bitext retrieval task; (2) Fine-tuning with cross-lingual NLJ data.
4.1 MULTILINGUAL CMLM
We follow the same configuration used to learn English sentence representations with CMLM, but extend the training data to include more languages. Results below will show that CMLM again exhibits competitive performance as a general technique to learn from large scale unlabeled corpora.
4.2 MULTITASK TRAINING WITH CMLM AND BITEXT RETRIEVAL
Besides the monolingual pretraining data, we collect a dataset of bilingual translation pairs {(s b)) using bitext mining system Feng et al. 2020). The source sentences {s:) are in English and the target sentences {t,} covers over 100 languages. We build retrieval task with the translation parallel data, identifying the corresponding translation of the input sentence from candidates in the same batch. Concretely, incorporating Additive Margin Softmax (Yang et al. 2019b), we compute bitext retrieval loss Li for the source sentences as:
1 B
(I)
(as m BB Ex eo(erit))
Above ( ( denotes the the inner products of sentence vectors of (()) and 1.99 (embedded by the transformer encoder); m and B denotes the additive margin and the batch size respectively. Note the way generate sentence embeddings is the same in CMLM. We compute the bitext
4

---

Under review as a conference paper at ICLR 2021
retrieval loss for the target sentences Cir by normalizing source sentences, rather than target sentences, in the denominator P
The final bitext retrieval loss for is given as Lor = Lúr + forr-
There are several ways to incorporate the monolingual CMLM task and bitext retrieval (BR). We explore the following multistage and multitask pretraining strategies:
S1. CMLM+BR: Train with both CMLM and BR from the start;
S2. CMLM - BR: Train with CMLM in the first stage and then train with on BR:
S3. CMLM + CMLM+BR: Train with only CMLM in the first stage and then with both tasks.
When training with both CMLM and BR, the optimization loss is weighted sum of the language modeling and the retrieval loss Cer. i.e. L LCMLM aChr- We empirically find 0.2 works well. As shown in Table B S3 is found to be the most effective. Unless otherwise denoted, our models trained with CMLM and BR follow S3. We also discover that given pre-trained transformer encoder, e.g. mBERT, we can improve the quality of sentence representations by finetuning the transformer encoder with CMLM and BR. As shown in Table 2 and Table S the improvements between "mBERT" and "f-mBERT" (finetuned mBERT) are significant.
4.3 FINETUNING WITH CROSS LINGUAL NATURAL LANGUAGE INFERENCE
Finetuning with NLI data has proved to be an effective method to improve the quality of embeddings for English models. We extend this to the multilingual domain. Given a premise sentence u and hypothesis sentence v, we train 3-way classifier on the concatenation of [u, v, /u v). * v]. Weights of transformer encoders are also updated in the finetuning process. Different from previous work also using multilingual NLI data (angg et al. 2019a), the premise and hypothesis here in different languages. The cross lingual NLI data are generated by translating Multi-Genre NLI Corpus (Williams al. 2018) into 14 languages using Google Translate API.
4.4 CONFIGURATIONS
Monolingual training data for CMLM are generated from 3 versions of Common Crawl data in 113 languages. The data cleaning and filtering is the same as the English-only ones. A new cased vocabulary is built from the all data sources using the WordPiece vocabulary generation library from Tensorflow Text. The language smoothing exponent from the vocab generation tool is set to 0.3, as the distribution of data size for each language is imbalanced. The final vocabulary size is 501,153. The number of projections N is set to be 15, the batch size B is 2048 and the positive margin is 0.3. For CMLM only pretraining, the number steps is million. In multitask learning, for S1 and S3, the first stage is of 1.5 million and the second stage is of million steps; for S2, number of training steps is million. The transformer encoder uses the BERT base configuration. Initial learning rate and optimizer chosen are the same as the English models.
4.5 EVALUATIONS
4.5.1 XEVAL: MULTILINGUAL BENCHMARKS FOR SENTENCE REPRESENTATIONS EVALUATION
Evaluations in previous multilingual literature focused on the cross lingual transfer learning abil- ity from English other languages. However, this evaluation protocol that treats English as the "anchor" does not equally assess the quality of non-English sentence representations with English ones. In order to address the issue, we prepare a new benchmark for multilingual sentence vectors, XEVAL, by translating SentEval (English) to other 14 languages with industrial translation API.
Results models trained with monolingual data are shown Table 2 Baseline models include mBERT (Devlin et al. 2019). XLM-R (Ruder al. 2019) and transformer encoder trained with MLM on the same Common Crawl data (MLM(CC), again this is to control the effects of training data). The method to produce sentence representations for mBERT and XLM-R is chosen to be averaging pooling after exploring options including [CLS) representations and max pooling. The
i... by swapping the and subscripts the last term of the denominator.
5

---

Under review as a conference paper at ICLR 2021
Model
ar
bg
de
cl
cn
65
fr
hi
SW
th
tr
ur
vi
zh
Avg.
MBERT
76.3
76.1
71.7
76.1
80.1
78.5
78.7
75.6
77.3
70.5
73.6
75.7
74.2
78.8
78.7
76.5
MLM (CC)
79.2
81.7
79.9
84.4
82.1
82.2
79.2
81.2
70.3
76.9
79.0
81.3
79.4
XLM-R
78.1
78.0
76.2
78.2
82.8
812
80.4
77.2
80.2
71.0
77.5
79.7
76.7
80.3
80.8
78.5
CMLM
80.6
81.2
82.6
81.4
85.0
82.3
83.4
80.0
82.3
76.2
78.8
81.0
78.5
81.6
81.2
Table 2: Performance (accuracy) of multilingual models trained with monolingual data on XEVAL. Highest numbers are highlighted bold.
Model
bg
de
el
fr
th
tr
vi
zh
Avg.
LASER
82.1
81.2
81.7
78.1
82.3
81.0
80.8
78.9
82.2
75.8
80.3
81.8
77.2
81.6
82.1
80.4
mUSE
80.4
74.0
82.2
65.0
83.3
82.7
62.3
68.1
81.6
80.3
68.8
68.0
76.2
SI
78.3
78.9
79.3
78.
81.0
78.7
79.5
78.0
79.0
76.6
77.8
78.6
77.7
79.0
78.6
S2
81.3
81.0
83.0
81.4
85.6
83.0
83.6
80.4
82.3
77.6
80.1
81.0
79.8
82.4
82.3
81.6
82.6
83.0
84.0
81,8
85.8
84,2
84.0
79.3
81.2
82.7
81.2
83.0
S3+NLI
84.2
83.7
85.0
83.4
87.0
85.9
85.8
83,0
85.6
79.6
83.0
84,2
81.2
84.2
84.4
84.0
MBERT
76.3
76.1
77.7
76.1
80.1
78.5
78.7
75.6
77.3
70.5
73.6
75.7
74.2
78.8
78.7
76.5
i-BERT
77.2
78.5
79.7
76.7
81.4
80.0
80.3
77.2
79.1
73.3
76.1
77.1
76.9
79.8
80.4
78.3
Table 3: Performance (accuracy) of models trained with cross lingual data on XEVAL. mUSE only supports 16 languages, underline indicates the language is not supported by mUSE. We test with multiple strategies for multitask pretraining: [S1]: CMLM BR; [S2]: CMLM+BR; [S3]: CMLM CMLM+BR. [f-mBERT] denotes finetuning mBERT with CMLM and BR.
multilingual model CMLM on monolingual data outperform all basclines in 12 out of 15 languages and the average performance.
Results of models trained with cross lingual data are presented in Table Baseline models for comparison include LASER (Artetxe & Schwenk 2019), trained with parallel data) and multilingual USE Yang et al. 2019a), trained with cross lingual NLD. Our model (S3) outperforms LASER in 13 out of 15 languages. Notably, finetuning with NLI in the cross lingual way produces significant improvement (S3 NLI v.s. S3) and it also outperforms mUSE by significant margins Multitask learning with CMLM and BR can also be used to increase the performance of pretrained encoders, e.g. mBERT. mBERT trained with CMLM and BR (f-mBERT) has significant improvement upon mBERT.
4.5.2 AMAZON REVIEWS
We also conduct zero-shot transfer learning evaluation Amazon reviews dataset (Prettenhofer Stein 2010p. Following Chidambaram et al. 2019). the original dataset is converted to clas- sification benchmark by treating reviews with strictly more than 3 stars as positive and negative otherwise. We split 6000 English reviews in the original training set into 90% for training and 10% for development. The two-way classifier, upon the concatenation of [u, v. uu v/. u v) (follow- ing previous works e.g. |Reimers Gurevych (2019). is trained on the English training set and then evaluated English, French, German and Japanese test sets (each has 6000 examples). Note the same multilingual encoder and classifier are used for all the evaluations. We also experiment with whether freezing the encoder weights or not during training. As presented in Table 4 CMLM alone has already outperformed baseline models. Training with BR and cross lingual NLI finetuning further boost the performance.
4.6 TATOEBA: SEMANTIC SEARCH
To directly assess the ability of our models on capturing semantics, we test on Tatoeba dataset pro- posed in Artetxe & Schwenk (2019). Tatoeba dataset include up to 1,000 English-aligned sentence pairs for each evaluated language. The task the other language by cosine similarity. The experiments is conducted on the 36 languages sent as in XTREME benchmark (Hu et al. 2020 and the evaluation metric retrieval accuracy. We test
find the nearest neighbor for the query sentence in
Note ml USE only supports 16 languages, the best CMLM model is still significantly hetter if only consid- ering the mUSE supported languages (underline in table indicates the unsupported languages mUSE)

---

Under review as a conference paper at ICLR 2021
Models
English
French
German
Japanese
Encoder parameters frozen during finetaning
Eriguchi ct al. (2018
83.2
81.3
Chidambaram
2019
en-fr en-de
87.4
82.3
hhdambaram al.
2019
87.1
81.0
mBEKT
80.0
73.1
70.4
71.7
XLM-R
85.3
81.5
82.5
CMLM
88.4
88.2
87.5
83.7
CMLM+ BR
88.3
87.2
86.4
83.2
CMLM+ BR NLI
89.4
88.8
88.4
82,8
Encoder parameters are trained during finetuming
mBERT
89.3
83.5
79.4
74.0
CMLM
93.4
92.4
92.1
88.6
CMLM+ BR
93.6
93.1
92.3
88.1
CMLM+ BR + NLI
93.7
92.4
93.5
86.8
Table 4: Classification accuracy on the Amazon Reviews dataset. The experiments examine the zero- shot cross-lingual ability of multilingual models. We explore both freezing/updating the weights of the multilingual encoder during training on English data.
our models with configuration CMLM+BR and CMLM+BR+NNL Basclines (results collected from Hu et al. (2020): Artetxe & Schwenk (219)) include mBERT, LASER, XLM, XLM-R. Results are presented in Table 5 Our model CMLM+BR outperforms all baseline models in 30 out of 36 lan- guages and has the highest average performance. One interesting observation is that finetuning with NLI actually undermines the model performance on semantic search, in contrary with the significant improvements from CMLM+BR CMLM+BR+NLI XEVAL (Table 3
We speculate this be- cause unlike semantic search, NLI inference is not a linear process. Finetuning with cross-lingual NLI not expected to help the linear retrieval by nearest neighbor search.
Lang.
af
ar
bg
de
cl
64
e1
21
f6
he
it
ja
MBEKT
42.7
25.8
49.3
17
71.2
29.8
68.7
20.3
25.5
46.1
39
66.3
41.9
34.8
38.7
54.6
58.4
42
XLM
43.2
18.2
13.5
66.2
25.6
58.4
24.8
17.1
32.2
32.2
54.5
32.1
26.5
30.1
45.9
56.5
40
XLM-R
58.2
CMLM-BR
96.3
90.6
91.2
97.7
95.4
98.1
92.0
95.6
96.1
92.8
97.6
95.6
94.2
95.6
CMLM-BR+NLI
90.
83.6
92.6
30.4
97.6
91.6
9.5
$2.6
70.3
90L7
$3.9
93.5
80.8
94.6
89.6
91.7
90.4
$8.4
3v
1a
83
3o
ml
me
ml
P
TU
SW
Uh
0
Ur
vi
zh
Mean
MBEKT
13.6
20.5
27.1
38.5
19.8
20.9
68
69.9
61.2
11.5
14.3
16.2
13.7
16
34.8
31.6
62
71.6
38.7
XLM
22.4
22.9
17.9
25.5
20
13.9
59.6
63.9
44.8
12.6
30.3
12.4
31.8
14.8
26.3
18.1
47.1
42.2
32.6
XLMER
14.1
26.4
$7.3
CMLM-BR+NLI
60.9
33.1
803
$5.6
94.9
90.7
93.2
92.3
91.7
76.7
$3.6
92.8
94.7
$2.0
94.3
34.7
94.3
93.1
53.8
Table Tatoeba results (retrieval accuracy) for each language. Our model CMLM+BR achieves the best results on 30 out of 36 languages.
5 ANALYSIS
5.1 LANGUAGE AGNOSTIC PROPERTIES
Language Agnosticism has been a property of great interest for multilingual representations. How- ever, there has not been qualitative measurement rigid definition for this property. Here we propose that "language agnostic" refers to the property that sentences representations are neutral w.r.t their language information. For example, two sentences with similar semantics should close in embedding space whether they are of the same languages not. Another case is that given one query sentence in language 1 and two candidate sentences with the identical meanings (different from the query sentence) in languages 4 and l2, the l input sentence should not be biased towards the 1 candidate sentence. To capture this intuition, we convert the PAWS-X dataset (Yang et al. 2019c) retrieval task to measure the language agnostic property. Specifically, PAWS-X dataset

---

Under review as a conference paper at ICLR 2021
consists of English sentences and their translations in other six languages (r-axis labels in Fig. Given a query, we inspect the language distribution of the retrieved sentences (by ranking cosine similarities). In Fig. 2 query sentences tations of mBERT (first row) have a strong self language bias, i.e. sentences in the language match- ing the query dominant. In contrast, the bias is much weaker in model trained with CMLM and BR (the third column), probably due to the cross lingual retrieval pretraining. We discover that removing the first principal component of each monolingual space from sentence representations effectively eliminate the self language bias. As shown in the second and the fourth column in Fig, ! with principal component removal (PCR), the language distribution is much more uniform. We fur- ther explore PCR by experimenting on the Tatoeba dataset. Table shows the retrieval accuracy of multilingual model with and w/o PCR. PCR increases the overall retrieval performance for both two models. This suggests the first principal components in each monolingual space primarily encodes language identification information.
in German, French and Chinese for each
Represen-
mBERT
mBERT PCR
PCR
de
0.2
0.c
1.5
0.1
fr
0..
zh
0.5
0.4
0.2
Figure 2: Language distribution of retrieved sentences. The first and third columns are mBERT and our models. Our model already general has more uniform distribution than mBERT. The second and fourth columns mBERT and our model with PCR.
fra
cmn
dcu
rus
ita
tur
hun
jpn
nld
Avg.
mBERT
60.2
60.2
62.8
65.9
53.8
55.7
32.4
62.4
31.9
39.0
56.2
52.8
mBERT PCR
59.9
64.3
61.7
67.5
57.4
56.2
33.3
64.4
36.5
42.3
61.1
54.8
ours
96.1
95.6
98.1
97.7
94.9
94.2
98.6
95.3
96.5
95.6
97.3
96.3
ours + PCR
95.5
96.0
98.2
97.9
95.1
94.1
98.5
95.8
96.6
95.3
97.2
96.4
Table Average retrieval accuracy on 36 languages of multilingual representations model with and without principal component removal (PCR) Tatoeba dataset.
We also visualize the sentence representations in Tatoeba dataset in Fig. Our model (the first row) shows both weak and strong semantic alignment (Roy et al.) 2020). Representations are close to others with similar semantics regardless of their languages (strong alignment), especially for French and Russian, where representations form several distinct clusters. Also representations from the same language tend to cluster (weak alignment). While representations from mBERT generally exhibit weak alignment.
5.2 ABLATION STUDY
In this section, we explore different configurations of CMLM, including the number of spaces in the projection N and CMLM architecture. As shown in Table projecting the sentence vector into N 15 produces highest overall performance. We also try modification to CMLM architecture. Besides the concatenation with token embeddings of before input to the transformer encoder, the projected
8

---

Under review as a conference paper at ICLR 2021
0.5
ours
0.00
-0.2
-0.5
-02-
0.4
0.4
MBERT
0.2
0.2
0.2
-0.7
-0.-
-04
fra
deu
eng
spa
Figure 3: Visualizations of sentence embeddings in Tatoeba dataset in 2D. The target languages are all English and the source languages are French, German, Russian and Spanish from left to right columns. The first and second rows are our model and mBERT respectively.
vectors are also concatenated with the sequence outputs of 82 for the masked token prediction. This version of architecture is denoted "skip" and model performance actually becomes worse.
Note that the projected vector can also used to produce the sentence representation v.- For example, one way is to use the average of projected vectors, i.e. Us N Σ, e Recall (8) is the ith projection. This version is denoted as "proj" Table Sentence representations produced in this way still yield competitive performance, which further confirm the effectiveness of the projection.
Model
MR
CR
SUBJ
MPQA
SST
TREC
MRPC
SICK-E
SICK-R
Avg.
82.3
89.7
95.8
88.8
87.6
90.4
71.5
80.8
83.4
85.5
N = 5
83.7
90.0
95.5
89.0
89.4
86.6
69.5
79.3
81.7
85.0
N 10
83.4
89.0
96.1
88.9
88.2
90.2
68.5
79.7
81.5
84.9
= 15
83.6
89.9
96.2
89.3
88.5
91.0
69.7
82.3
83.4
86.0
N = 20
81.1
89.5
95.8
88.9
85.9
89.8
69.7
80.2
85.0
85.1
skip
80.3
86.8
94.5
87.5
84.9
86.0
69.2
72.8
74.7
81.9
proj
82.6
89.7
96.0
87.3
87.5
89.2
70.5
81.7
83.8
85.4
Table Ablation study of CMLM designs, including the number of projection spaces, architecture and sentence representations. The experiments are conducted on SentEval.
CONCLUSION
We present novel sentence representation learning method Conditional Masked Language Model- ing (CMLM) for training on large scale unlabeled corpus. CMLM outperforms the previous state-of- the-art English sentence embeddings models, including those trained with (semi-)sung sumerviiee signals. For multilingual representations learning, we discover that co-training CMLM with bitext retrieval and cross-lingual NI.I finetuning achieves state-of-the-arr performance. We also discover that mul- tilingual representations have the same language bias and principal component removal (PCR) can climinate the bias by separating language identity information from semantics.
REFERENCES
Mikel Artetxe and Holger Schwenk. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7:597-610, 2019.
Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods Natural Language Processing, pp. 632-642, 2015.

---

Under review as a conference paper at ICLR 2021
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhommi St John, Noah Con- stant, Mario Sunajardo-Cespeder Steve Yuan, Chris Tar, et al. Universal sentence encoder. arXiv preprint arXiv:1803.1111 2018.
Muthu Chidambaram, Yinfei Yang, Daniel Cer, Steve Yuan, Yunhsuan Sung, Brian Strope, and Ray Kurzweil. Learning cross-lingual sentence representations via multi-task dual-encoder model. In Proceedings of the 4th Workshop on Representation Learning for NLP RRpPLANLP-2019) pp. 250-259, Florence, Italy, August 2019. Association for Computational Linguisties. doi: 10. 16653/v1/W1--3300. URL https: htt.pp::/wwww.c...
wwwwwwwww...oo........w...oowwo/ a web. web.orr//anthhooog///// 9-4300
Alexis Conneau and Douwe Kiela. Senteval: An evaluation toolkit for universal sentence repre- sentations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), 2018.
Alexis Conneau, Douwe Kicla, Holger Schwenk, Loic Barrault, and Antoine Bordes. Supervised learning of universal sentence representations from natural language inference data. In Proceed- ings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 670-680, 2017.
Jacob Devlin, Ming-Woi Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171 1177,, 2019.
Akiko Eriguchi, Melvin Johnson, Orhan Firat, Hideto Kazawa, and Wolfgang Macherey. Zero- shot cross-lingual classification using multilingual neural machine translation. arXiv preprint arXiv:1809.04686, 2018
Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. Language- agnostic bert sentence embedding, arXiv preprint arXiv:2007.01852, 2020.
John M Giorgi, Osvald Nitski, Gary D Bader, and Bo Wang. Declutr: Deep contrastive learning for unsupervised textual representations. arXiv preprint Ttt::20006036555 2020.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. Xtreme: massively multilingual multi-task benchmark for evaluating cross-lingual generaliza- tion. arXiv preprint arXiv:2003. 1080, 2020.
Minqing Hu and Bing Liu. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 168-177, 2004.
Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Tor- ralba, and Sanja Fidler. Skip-thought vectors. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems 28, pp. 3294 3302. Curran Associates, Inc., 2015. URL http::/papezs. nips. 5950- -tto-=-kii-ttooctt--eetttttoooo
pdE
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations, 2020.
www.eewwweeeeeeeeeeeeeeeeeeeeeee forum?
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907.19977.19999 2019.
Lajanugen Logeswaran and Honglak Lee. An efficient framework for learning sentence representa- tions. In International Conference on Learning Representations, 2018a.
Lajanugen Logeswaran and Honglak Lee, An efficient framework for learning sentence representa- tions. In International Conference Learning Representations, 2018b.
10

---

Under review as a conference paper at ICLR 2021
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Berarrii, Roberto Zam- parelli, et al. A sick cure for the evaluation of compositional distributional semantic models. In LREC, pp. 216-223, 2014.
Bo Pang and Lillian Lee. A sentimental education: Sentiment analysis using subjectivity summa- rization based on minimum cuts. In Proceedings of the 42nd Annual Meeting of the Association for Compulational Linguistics (ACL-04), pp. 271-278, 2004.
Bo Pang and Lillian Lee. Seeing stars: exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguisties, pp. 115-124, 2005.
Peter Prettenhofer and Benno Stein. Cross-langun lrass-la text classification using structural correspon- dence learning. In Proceedings of the 48th annual meeting of the association for computational linguistics, pp. 1118-1 127, 2010.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints, 2019.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 3973-3983, 2019.
Uma Roy, Noah Constant, Rami Al-Rfou, Aditya Barua, Aaron Phillips, and Yinfei Yang. Lareqa: Languace answer retrieval from multilingual pool. arXiv preprint arXiv:2004.05453 2020.
Sebastian Ruder, Anders Søgaard, and Ivan Vulić. Unsupervised cross- pooss-lingual representation learn- ing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguis- fies: Tutorial Abstructs, pp. 31-38, Florence, Italy, July 2019. Association for Computational Linguisties. doi: 10. 10.18653/v1/P19-110 219-4007
URL https: /www C web. www.oorg//attooooooooooooo
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language pro- cessing, pp. 1631-1642. 2013. 2013.
Sandeep Subramanian, Adam Trischler, Yoshua Bengio, and Christopher J Pal. Learning general purpose distributed sentence representations via large scale multi-task learning. In International Conference Learning Representations, 2018. URLhttps:: /owpr///eeeeeeeeeeeeeeeee net forum?
id=B1BMgG-CZ
Ellen M Voorhees and Dawn M Tice. Building question answering test collection. In Proceed- ings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, pp. 200, 2000.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1112-1122, 2018.
Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-Hsuan Sung, et al. Multilingual universal sentence encoder for semantic retrieval. arXiv preprint arXiv: /90iv:1907.044 /907.04307 2019a.
Yinfei Yang, Gustavo Hermandez Abrego, Steve Yuan, Mandy Guo, Qinlan Shen, Daniel Cer, Yun- hsuan Sung, Brian Strope, and Ray Kurzweil. Improving multilingual sentence embedding using bi-directional dual encoder with additive margin softmax. In Proceedings of the Twenty-Eight Artificial Intelligence, 1JCAI-19, pp. Joint Conferences on Artificial Intelligence Organization, 2019b. doi: 10.24963/jcuu.20119773 URL htttps://do .rgg 219.2493333 caai cca.9/19/72200///44
International Joint Conference
International
11

---

Under review as a conference paper at ICLR 2021
Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. Paws-x: A cross-linguss adversarial dataset for paraphrase identification. In Proceedings of the 2019 Conference on Empirical Meth- ods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNI.P-JCNIIP) pp. 3678-3683, 2019c.
Ziyi Yang, Chenguang Zhu, and Weizhu Chen. Parameter-free sentence embedding via orthogonal basis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP- (JCNLP), pp. 638 648, 2019d.
12


Reviews:
[{"review_id": "geo0tdb3NW-", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors present conditional masked language modeling (CMLM), a new method for unsupervised pretraining, in which the skip-thought notion of conditioning on neighboring sentences is adopted for masked language modeling. The upshot of the proposed approach is that it generates single sentence embeddings that perform competitively on SentEval. In the multilingual setting, the authors combine their CMLM method with a bitext retrieval objective (selecting a sentence’s translation from the other sentences of the language in the batch) that increases performance on a version of the SentEval tasks translated into 14 other languages. In their analysis, the authors make further claims about multilingual embeddings capturing language ID information in their first principle components, a conclusion somewhat substantiated by their results. The authors provide a small amount of ablation experiments for experimental/model design choices.\n\nThe underlying idea is worth pursuing, the execution and description could be improved, people will be interested in the results that are present (but then have questions).\n\n\nOverall\n\nWhy only a subset of SentEval for the English experiments (3.2) but then the full SentEval in the multilingual XEval experiment (4.5.1)?  Especially if you are trying to make a single-sentence encoder but then evaluating on SICK-E instead of SICK-R, which is arguably a more applicable eval set. \n\nWhy no XLM-R in the amazon reviews analysis (4.5.2)?\n\nSec 1.\nFig 1: box of chocolate*s*\n\n\nSec 2.\n--“conditional”, →  “conditional,”\n\n--no quantitative comparison of using max vs mean pool vs CLS embedding\n\n--first sentence is feed → first sentence is fed\n\n--three-layer neural network → three-layer MLP\n\n--refer to using the same set of encoder weights for different inputs as siamese networks, as done in the sentence-bert paper https://en.wikipedia.org/wiki/Siamese_neural_network\n\n-- v_d is used but not defined\n\n\nSec 3.\n--Skip-thought originally used a sentence to predict the generation of both the preceding and succeeding sentences. This is functionally equivalent to your flip-flopping the order of the consecutive sequences. I would make the point that these steps are equivalent.  Note that this is also not necessarily making the task “more challenging” (and moreover I am not sure why “more challenging” equates with “better pretraining method for language understanding” -- and an ablation of this step is not included to show that it is in fact necessary and useful).\n\n-- similarly, no analysis of masking rate, nor explanation for why ‘more challenging’ is better.\n\n-- “We explore two transformer configurations, base and large, same as in the original BERT paper.” – fragmented\n\n-- The number of projections N = 15. – fragmented\n\n-- SentBERT → SentenceBERT or SBERT\n\n-- On the specific subset of SentEval tasks you’ve selected, the majority of the performance discrepancy is in the SICK-E  task--otherwise, the overall #’s are rather interchangeable. How does this change if you add in the rest of the SentEval tasks, and why were they omitted? Analysis/exploration for why you get such a performance boost only on SICK-E would also be useful.\n\n-- “the length … set to be 256 tokens”: please clarify whether the “length” refers to the maximum length, or each sentence is a fixed-length chunk consisting of 256 tokens\n\n--typo:  “we also exploring”\n\n\nSec 4.\n--If your introduced bitext retrieval objective uses batch size, experiments comparing the effect of batch size is necessary. \n\n-- Please specify the value of the margin m being used in the experiments\n\n--Choice of number of projections is also not motivated (and in fact contradicted by the ablation  experiment finding that 15 is better)\n\n-- the motivation and contribution for XEVAL are great-- the explanation of the dataset is lacking. What translation API was used? How was the XEVAL score computed for each language? Is it the full set of SentEval downstream tasks?\n\n-- cite precedent for using the concatenation of u,v, u-v and u*v. (or show its effect via ablation)\n\n-- BR  → CMLM+BR configuration not evaluated\n\n-- choice of different training step #’s in each configuration is not particularly motivated.\n\n-- “after exploring options including [CLS] representations and max pooling.” what was the performance drop?\n\n-- typo: “has a significant upon mBERT”\n\n\nSec 5.\n--It is not clear that you can make the claim that the first PC *only* encodes language-identification information?\n\n--I assume Figure 3 is 2-dimensional TSNE (needs a cite), which comes with its own set of caveats as a visualization tool. Quantitative clustering analysis such as silhouette coefficient might be more appropriate than a plot.  If Figure 3 is not t-SNE, please specify the meaning of X and Y axes.\n\n-- did not try higher than n=15 projections but claimed it was optimal\n\n-- The description of the “skip” ablation is unclear: please clarify what is meant by “concatenated with the sequence outputs of s2”.\n\n-- typos: “removing the first principal component … effectively eliminate”, “for both two models”, “representations … generally exhibits”", "extra_scores": {}, "extra_text": {}}, {"review_id": "I_H2kz-72M", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I appreciate the response from the authors to my review, as well as to others. \n\nMy concerns on the intuition are most not solved. Although in this DNN dominating era, we cannot expect the explainability as we had before, I still believe that a solid work should be grounded on a reasonable basis, which could be in a high level, such as BERT and SBERT. Let's refer to the example given in the model architecture. The projection of the sentence vector of \"Life is a box of chocolates\" is left-concatenated with the masked embeddings of the second sentence. This operation is very much lacking in intuition, how come the projection of a sentence representation can be concatenated with the embeddings? In addition, \"The second encoder shares the same weights with the encoder used to embed s1\", considering their inputs are very different, weight sharing for the two encoders are also problematic.\n\nAnother point I just noticed, although the authors claimed that their model is better than SBERT, and did a comparison with SBERT-large, they did not compare with SBERT-base, which makes the conclusion unreliable.\n\n---------------------------------------------------------------------------------------------------------------------------\nThis paper proposes a method called \"Conditional Masked Language Model\" for unsupervised sentence representation learning. The method involves two-sentence encoder, where one sentence depends on the encoded sentence level representation of the adjacent sentence. The experimental results are good overall, as the proposed method tends to give the best results across monolingual and multilingual benchmark datasets.\n\nThere are still some concerns about the novelty of this paper.  First, I think the explanations for the intuition of the proposed model can be clarified, especially in the introduction section. Second, the baselines used for comparison are not complete, which makes me concern about the effectiveness of the model. The proposed model is the combination of Skip-Thought (Kiros et al., 2015) and BERT masked LM (Devlin et al., 2019).  Their experimental results show a detailed comparison of BERT but ignore much about the Skip-Thought. Although the authors mentioned the results of the Skip-Thought model on the SentEval benchmark, the encoder used in the Skip-Thought (Kiros et al., 2015) is RNN while the author used the Transformer Encoder.  I would appreciate a better and fair comparison of the Skip-Thought model by using same transformer encoder and same training corpus.\n\nI am not sure why you used the concatenation when you do the masked LM.  Are there any other ways to do that?  It can be more convincing to see some analysis or results here. Additionally, there is another work titled “DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations”, which also focuses on unsupervised sentence representation learning.  Although it is from arxiv, it would be nice that the authors can mention this work.\n\nIt seems good that the authors performed many experiments over many different datasets across monolingual and multilingual.  The exploration of the same language bias of the learned representations is also very interesting.\n\nTo summarize, the paper is a bit limited in terms of technical contribution, particularly considering that there is not a good intuition explanation, but some analysis in this paper looks good.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "FdjrveSXho", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents Conditional Masked Language Modeling (CMLM), which integrates sentence representation learning into MLM training by conditioning on the encoded vectors of adjacent sentences. It is shown that the English CMLM model achieves strong performance on SentEval, and outperforms models learned using (semi-)supervised signals. It is also found that a multilingual CMLM model co-trained with bitext retrieval (BR) and natural language inference (NLI) tasks outperforms the previous state-of-the-art multilingual models by a large margin. The paper further proposes a principle component based approach to remove the language identifying information from the representation while still retaining sentence semantics.\n\n-Strengths\n\nLearning sentence representations on large scale unlabeled corpora is an important research problem. This paper presents a heavily empirical study, with a series of experiments to evaluate the proposed sentence representation learning method. Multilingual experiments are conducted, with interesting results on language agnostic.\n\nThe proposed method, as shown in Figure 1, is somewhat new.\n\n-Weaknesses\n\nThe study is mainly empirical.\nThe authors should provide more details about the three-layer neural network as the projection P (·).\nAnother concern is that the contribution of this paper to research community may be weak, if the code is not released and the results are not easily reproduced.\n\n--------update after reading the response-----------\n\nThanks for the authors' response. Mainly empirical and limited in methodology novelty. So I tend to keep the score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "1jVUti2kkB4", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "----------\n\nI appreciate the response from authors and the additional experiments. I do think the semantic search task adds value to the paper. However, the paper continues to be centered around the SentEval benchmark results. While SentEval is a useful benchmark to evaluate sentence representations, it doesn't reflect well how these representations will be used in practice. A fine-tuned BERT model will likely perform strongly on these tasks. The paper would be far more compelling if the authors can provide strong evidence that the sentence embeddings do well on tasks where using a BERT model is either less effective due to performance or computational reasons. \n\nI prefer to keep my score. \n\n------------\n\nThis work proposes a self-supervised training objective called CMLM (conditional masked language model) for learning sentence representations. An encoder produces multiple fixed length representations of a given sentence and a decoder reconstructs the adjacent sentence given it’s masked version and the encoded representations. CMLM performs well on the SentEval benchmark. CMLM is further extended to the multilingual setting via bi-text retrieval contrastive training and training on NLI data. The multilingual version is shown to work well on multiple translated versions of the SentEval benchmark (SentEval data translated into other languages using an off the shelf translation system) and Amazon reviews (sentiment classification).   \n\n\nPros\n* This work addresses the important problem of (unsupervised) sentence representation learning. Extracting fixed-length sentence representations from popular language model based encoders is a non-trivial problem and this work attempts to provide a solution.\n* Experiments go beyond the standard English setting and evaluate sentence representations in the multilingual setting as well. \n* Interesting modeling approaches.\n\nCons\n* Experiments are weak. It is unclear to what extent the tasks + evaluation protocol considered here are reflective of language understanding. I don’t think strong baselines were considered. Some of the evaluation benchmarks considered seem arbitrary.\n* Model is largely based on prior work. The main contribution is not clear. There are many settings considered in the paper and it is unclear if the proposed contributions are truly significant due to weak baselines are differences in data used for training different methods.\n\nThere are several issues with the experimental setup.\n* Evaluation protocol: It is unclear if the evaluation protocol considered is measuring language understanding capability well. Representations from the encoders are held fixed and linear classifiers are trained on top of these fixed representations on downstream tasks using labelled data. To me, this is not a setting that demands sentence vectors. It only shows that the sentence vectors capture useful features. I would suggest focusing on a setting where the advantage of the sentence vectors can be demonstrated such as a retrieval problem.\n* Baselines: It is unfair to compare the proposed method against baselines like BERT which are not designed to produce fixed length encodings. \n* Data used for pre-training: It is difficult to gauge how good the method is in comparison to other models due to differences in the data used for pre-training. Ideally, there should be a table comparing models that use the exact same resources. In Table 1, although BERT-base/large is trained on the same data, it is not a strong baseline since mean pooled representations from the encoder are treated as a sentence representation. Ideally, the model should be compared against a skip-thought baseline or an unsupervised sentence representation learning method that uses the same resources for training. \n\nI would have expected the authors to evaluate multilingual representations on existing benchmarks as well. I don’t find the proposed benchmark XEVAL very convincing. Claims would have been stronger if authors had also included results on existing benchmarks. \n\nThe authors need to make it clear exactly what resources are used for training each method. \n\nPresentation can be improved, especially the organization of the paper. It is difficult to follow the paper and identify the main contributions in the current presentation. \n\nThe paper touches upon several things - conditional masked language modeling, bitext retrieval, NLI training, language agnosticism, etc, and I find the paper quite incoherent. I suggest the authors make a focused contribution and provide strong experimental evidence to support that contribution. Right now there’s too many things which makes it hard to make sense of the paper as whole.\n\nWhile the approaches considered in the paper have some merit, the significance of this work is unclear due to issues in the evaluation. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a Conditional Masked Language Modeling (CMLM) method to enhance the MLM by conditioning on the contextual information. 

All of the reviewers think the results are good. However, the reviewers also think the intuition and experiments are not so convincing. The responses and revisions still not satisfy all the reviewers' major concern.
Decision: Reject