On Learning Universal Representations Across Languages
Authors: Xiangpeng Wei
Rongxiang Weng
Yue Hu
Luxi Xing
Heng Yu
Weihua Luo
Venue: ICLR 2021 Poster (2021)
Keywords: universal representation learning
cross-lingual pretraining
hierarchical contrastive learning

Abstract: Recent studies have demonstrated the overwhelming advantage of cross-lingual pre-trained models (PTMs), such as multilingual BERT and XLM, on cross-lingual NLP tasks. However, existing approaches essentially capture the co-occurrence among tokens through involving the masked language model (MLM) objective with token-level cross entropy. In this work, we extend these approaches to learn sentence-level representations and show the effectiveness on cross-lingual understanding and generation. Specifically, we propose a Hierarchical Contrastive Learning (HiCTL) method to (1) learn universal representations for parallel sentences distributed in one or multiple languages and (2) distinguish the semantically-related words from a shared cross-lingual vocabulary for each sentence. We conduct evaluations on two challenging cross-lingual tasks, XTREME and machine translation. Experimental results show that the HiCTL outperforms the state-of-the-art XLM-R by an absolute gain of 4.2% accuracy on the XTREME benchmark as well as achieves substantial improvements on both of the high resource and low-resource English$\rightarrow$X translation tasks over strong baselines.

Published as a conference paper at ICLR 2021
ON LEARNING UNIVERSAL REPRESENTATIONS ACROSS LANGUAGES
Xiangpeng Wei Rongxiang Weng', Yue Hu Luxi Xing Heng Yu', Weihua Luo IIstitute of Information Engineering, Chinese Academy of Sciences, Beijing, China School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China {weixi (weixiangped angpeng, huyue, xi xingluxi /@iie. aae........... ac.cn
Machine Intelligence Technology Lab, Alibaba Group, Hangzhou, China (wengrx, yuheng. . yh, wei hua. uowhh @ iiaaa-ba---- ieeba-innc innc. com
ABSTRACT
Recent studies have demonstrated the overwhelming advantage of cross-lingual pre-trained models (PTMs), such as multilingual BERT and XI.,, on cross- lingual NLP tasks. However, existing approaches essentially capture the co- occurrence among tokens through involving the masked language model (MLM) objective with token-level cross entropy. In this work, we extend these approaches to learn sentence-level representations and show the effectiveness on cross- lingual understanding and generation. Specifically, we propose a Hierarchical Contrastive Learning (HICTL) method to (1) learn universal representations for parallel sentences distributed one or multiple languages and (2) distinguish the semantically-relat words from a shared cross-lingual vocabulary for each sen- tence. We conduct evaluations on two challenging cross-lingual tasks, XTREME and machine translation. Experimental results show that the HICTL outperforms the state-of-the-arr XLM-R by an absolute gain of 4.2% accuracy on the XTREME benchmark as well achieves substantial improvements both of the high- resource and low-resource English->X +X translation tasks over strong baselines.
INTRODUCTION
Pre-trained models (PTMs) like ELMo (Peters al.) 2018), GPT (Radford al. 2018 and BERT (Devlin al., 2019) have shown remarkable success of effectively transferring knowledge learned from large-scale unlabeled data downstream NLP tasks, such text classification (Socher ct al. 2013) and natural language inference (Bowman et al. 2015) Williams et al.] (018). with lim- ited no training data. To extend such pegeraiining-fmettniin endeavors such as multilingual BERT (Devlin et al., 2019) and XLM (Conneau & Lample! 2019) have been made for learning cross-lingual representation. More recently, Conneau et al. (2020) present XLM-R to study the effects of training unsupervised cross-lingual representations at huge scale and demonstrate promising progress on cross-lingual tasks.
paradigm to multiple languages, some
However, all of these studies only perform a masked language model (MLM) with token-level (i.e., subword) cross entropy. which limits PTMs capture the co-occurrence among tokens and con- sequently fail to understand the whole sentence. It leads to two major shortcomings for current cross-lingual PTMs, i.e., the acquisition of sentence-level representations and semantic alignments among parallel sentences in different languages. Considering the former, [Dellin et al. (2019) in- troduced the next sentence prediction (NSP) task to distinguish whether two input sentences are continuous segments from the training corpus. However, this simple binary classification task is not enough to model sentence- representations (Joshi et al. 2020 Yang et al. [2019] Liu et al. 2019 Lan al. 2020 Conneau et al. 2020). For the latter, (Huang et al. (2119) defined the cross- lingual paraphrase classification task, which concatenates two sentences from different languages as input
"Wor done at Alibaba Group. Yue Hu and Heng Yu are the corceresppnniing authors. We also made an of- ficial submission to XTREME (https:/ /tt///siitt /es..s.t............eee..eeeeeee research, googl e/ xxtee)),, with several improved
techniques used in (Fang et 2020 Luo 2020).

---

Published as a conference paper at ICLR 2021
and classifies whether they are with the same meaning. This task learns patterns of sentence- well but fails to distinguish the exact meaning of each sentence.
In response to these problems, we propose to strengthen PTMs through learning universal represen- tations among annaaniialll-cquivalleee sentences distributed in different languages. We introduce a novel Hierarchical Contrastive Learning (HICTL) framework to learn language invariant sentence representations via self-supervised non-paramen con-paammttric instance discrimination. Specifically, we use a BERT-style model to encode two sentences separately, and the representation of the first token (e.g., [CLS] in BERT) will be treated as the sentence representation. Then, we conduct instance-wise comparison both sentence-level and word-level, which are complementary each other. At the sentence level, we maximize the similarity between two parallel sentences while minimizing which among non-parallel ones. At the word-level, we maintain bag-of-words for each sentence-pais each word in which is considered ones. To reduce the space of negative samples, we conduct negative sampling for word-level con- trastive learning. With the HICTL framework, the PTMs are encouraged to learn language-agnon representation, thereby bridging the semantic discrepancy among cross-lingual sentences.
positive sample while the rest words in vocabulary are negative
The HICTL is conducted on the basis of XLM-R (Conneau et al., 2020) and experiments are performed on several challenging cross-lingual tasks: language understanding tasks (e.g.. XNLI, XQuAD, and MLQA) in the XTREME (Hu et al.] 2020) benchmark, and machine translation in the IWSLT and WMT benchmarks. Extensive empirical evidence demonstrates that our approach achieve consistent improvements over baselines various tasks of both cross- lonsull language understanding and generation. In more detail, our HICTL obtains absolute gains of 4.2% (up to 6.0% on zero-shot sentence retrieval tasks, e.g. BUCC and Tatoeba) accuracy on XTREME over XLM-R. For machine translation, our HICTL achieves substantial improvements over baselines on both low-resuurce (IWSLT English-*X] and nigh-resource (WMT English-*X] translation tasks.
RELATED WORK
Pre-trained Language Models. Recently, substantial work has shown that pre-trained models (PTMs) (Peters et al. 2018 Radford et al. 2018: Devlin al. 2019) the large corpus are ben- eficial for downstream NLP tasks. The application scheme is to fine-tune the pre-trained model using the limited labeled data of specific target tasks. For cross-lingual pre- tre-training, both Devlin jet al. (219)) and Conneau & Lample| (2019) trained a transformer-basee model on multilingual Wikipedia which covers various languages, while XLM-R (Conneau al. 2020) studied the effects of training unsupervised cross-lingual representations on a very large scale.
For sequence-to-sequence pre-training, UniLM (Dong et al. 2019) fine-tuned BERT with an ensem- ble of masks, which employs shared Transformer network and utilizing specific self-attention mask control what context the prediction conditions on. (Song by jointly training the mncoder-decoder framework. XLNet (Yang et al. 2019) trained by predicting masked tokens auto-regressively in a permuted order, which allows predictions to condition on both left and right context. Raffel et al. (2019) unified every NLP problem pre-trained denoising sequence-to-sequenee 2020) pre-trained a denoising gequence-to-sqquene model, in which spans are masked from the input but the complete output is regressseryssll predicted.
(2019) extended BERT-style models
text-to-text problem and model at scale. Concurrently, BART (Lewis et al.)
Previous works have explored using pre-trained models to improve text generation, such pre- training both the encoder and decoder on several languages (Song et al. 2019 Conneau & Lample)
2019 Raffel et al.] 2019) using pre-trained models initialize encoders (Edunov et al. 2019
Zhang et al. 2019a Guo et al. 2020). Zhu et al.] (2020) and Weng et al. (2020) proposed a BERT-
fused NMT model, which the representations from BERT are treated as context and fed into all
layers of both the encoder and decoder: Zhong et al. (2020) formulated the extractive summarization
Siamese-BERR architecture to compute the similarity between the source document and the candidate summary, which leverages the gra-trained BERT Siamese network structure. Our approach also belongs to the contextual pre-training so it could be applied to various downstream ULU and NLG tasks.
task semantic text matching problem and proposed
Contrastive Learning. Contrastive learning (CTL) Saunshi et al. 2019) aims at maximizing the similarity between the encoded query y and its matched key h+ while keeping randomly sampled
2

---

Published as a conference paper at ICLR 2021
Contrastive
Contrastive loss
s(q.k)
s(q,
q
(lt Ki, Kr. Kzn-2)
Uc ki, kzz ..,, km)
Embedding Lookup
XLM-R M(-)
XLMR M(-
XLM-R MC
Negative Sampling
Wr E
X
O Ux
X; o
V
(a) Sentence-Level CTL
(h) Word-Level CTT.
Figure/ 1: Illustration of Hierarchical Contrastive Learning (HICTL). n is the batch size, m denotes the number negative samples for word-level contrastive learning. B and V indicates the bag-of- words of the instance (rt, Ut) and the overall vocabulary of all languages, respectively.
keys , k faraway from it. With similarity measured by a score function s(g. k). form of contrastive loss function, called InfoNCE (Oord et al. 2018 is considered in this paper:
(sp(s(q.k+)) Lat -og (xp(s(q,k+)) + +, exp(s(q,
(1)
where the score function k(q.k) essentially implemented the cosine similarity - T&T q and are often encoded by a learnable neural encoder, such as BERT (Devlin et al. 2019) or ResNet (He et al. 2016). A and k- are typically called positive and negative samples. In addition to the form illustrated in Eq. (T). contrastive losses can also based on other forms, such as margin-
loses (Hadsell al. (0066) and variants of NCE losses (Mnih & Kavukcuoglu, 2013).
Contrastive learning is at the core of several recent work on unsupervised or self-supervised learning from computer vision (Wu et al. 2018 Oord et al. 2018) Ye et al. 22019; He et al. 2019: Chen al.
2020
Tian et al. 2020) to natural language processing (Mikolov et al. [2013] Mnih & Kavukcuogli
2013
Devlin et al.] 2019 Clark al. 2020b Feng al.) 2020: Chi al.)
2020). Kong et al.
(2020)
improved language representation learning by maximizing the mutual information between
masked sentence representation and local n-gram spans. Clark al. (2020b) utilized discrimi-
nator to predict whether a token is replaced by generator given its surrounding context. [ter et al. (2020) proposed to pre-train language models with contrastive sentence objectives that predict the surrounding sentences given an anchor sentence. In this paper, we propose HICTI to encourage par- allel cioss-lingual sentences to have the identical semantic representation and distinguish whether word is contained in them as well, which can naturally improve the capability of cross-lingus understanding and generation for PTMs.
METHODOLOGY
3.1 HIERARCHICAL CONTRASTIVE LEARNING
We propose hierarchical contrastive learning (HICTL), a novel comparison learning framework that unifies cross- lingull sentences as well as related words. HICTL can learn from both non- non-parallel and parallel multilingual data, and the overall architecture of HICTL is illustrated in Figure [ We rep- resent a training batch of the original sentences as x = {01, 72. Fn} and its aligned counterpart is denoted as y {y1, ...,.., --. Ya ). where is the batch size. For each pair (a;, Vi), yi is either the
translation in the other language of z; when using parallel data or the perturbation through reorder- ing tokens in x; when only monolingual data is available.x* X is denoted as a modified version of x where the i-th instance is removed.
Sentence-Level CTL. As illustrated Figure Ia we apply the XLM-R as the encoder to represent sentences into hidden representations. The first token of every sequence is always special token
3

---

Published as a conference paper at ICLR 2021
d' = -Ql12
k
k+
Figure 2: Illustration of constructing hard negative samples (HNS). A circle (the radius is d+ =Il k 9 l12) in the embedding space represents manifold near in which sentences are semantically equivalent. We can generate coherent sample (i.e., k-) that interpolate between known pair g and k The synthetic negative À can be controlled adaptively with proper difficulty during training. The curly brace in green indicates the walking range of hard negative samples, the closer to the circle the harder the sample is.
(e.g., (CLS)), and the final hidden state corresponding to this token is used as the aggregate sen- tence representation for pre-training, that is, Va (Mg(M()) where g()) (-) is the aggregate function and f.)) is a linear projection, denotes the composition of operations. To obtain universal repre- sentation among semantically-equiilleee sentences, encourage Jz, (the query, denoted as g) to be as similar as possible to Ty. (the positive sample, denoted as k+) but dissimilar to all other instances (i.e., X U x considered as a series of negative samples, denoted as {ki k2 , training batch. Formally, the sentence-level contrastive loss for æ; is defined
... k in
exp os k*)
(I,)) = - log
(2)
exp os(g. k+)
Σド= y'uu' exp os(q, k,))
Symmetrically, also expect Ty. (the query, denoted q) to be similar possible to Tx, (the positive sample, denoted k-t) but dissimilar all other instances in the same training batch, thus,
exp os( A+)
Lestt (yt) = log
exp os(g, k+) + > MyxM exp os k7)
The sentence-level contrastive loss over the training batch can formulated as
Ls
Σ (Fecu()))) Lscu(y)))-
(4)
For sentence-level contrastive learning, we treat other instances contained in the training batch as negative samples for the current instance. However, such randomly selected negative samples are often uninformative, which poses a challenge of distinguishing very similar but nonequivalent sam- ples. To address this issue, we employ smoothed linear interpolation (Bowman et al.) 2016 [Zheng] jet al., (201)) between sentences in the embedding space alleviate the lack of informative sam- ples for pre-training, as shown in Figure 2 Given training batch {(r;, u.) Jn where n is the batch size. In this context, having obtained the embeddings a triplet, an anchor q and positive k+ as well as negative k- (supposing q, k+ and k- representations of sentences H,, Ys and v x V, respectively). construct harder negative sample k to replace k
g A(k 9). X
dt +
if d
k-
if d dt.
where d =|1 ++ q lle and d =|| -- - 9 1l2. For the first condition, the hardness of A increases when À becomes smaller. To this end, we intuitively set X as
cotte
d
A
, (0,1)
(6)
d
4

---

Published as a conference paper at ICLR 2021
where Pang TOU 1001-100.-11 e-23" - is the average log- probability over the last 100 training batches and Ls formulated Eq. 44) the sentence-level contrastive loss of one training batch. During pre-training. when the model tends to distinguish positive samples easily, which means neg- ative samples are not informative already. At this time, Pang and d' d L. which leads negative samples are adaptively synthesized in the following training steps, vice versa. As hard neg- ative samples usually result in significant changes of the model parameters, we introduce the slack coefficient Ç to prevent the model from being trained in the wrong direction, when it accidentally switch from random negative samples to very hard ones. In practice, we empirically set Ç 0.9.
and harder
Word-Level CTL. Intuitively, predicting the related words in other languages for each sentence can bridge the representations of words in different languages. As shown in Figure Tb] we concatenate the sentence pair (z,, y1) 2; y4 [CIS] 2; [SEP] ] ys [SEP] and the bag-of-words of which is denoted as B. For word-level contrastive learning, the final state of the first token is treated as the query (γ), each word WA E B is considered as the positive sample and all the other words (V' ie., the words in V that are not in B where V indicates the overall vocabulary of all languages) negative samples. As the vocabulary usually with large space, we propose to only use subset S C V sampled according to the normalized similarities between q and the embeddings of the words. As result, the subset 8 naturally contains the hard negative samples which are beneficial for learning
representations (Ye et al. 2019). Specifically, the word-level contrastive loss for (zi, yi) is defined as
exp (o(())((ww))
Lmeett (Ti, (Ti,
log espos(q. e(wx)) + Emzes exp expos(q. e(w;))
B
where e(-) is the embedding lookup function and [B] is the number of unique words the concate- nated sequence 2; y/a The overall word-level contrastive loss can be formulated as:
12 LW
Σ 72
(xs yk).
(8)
Multi-Task Pre-training, Both MLM and translation language model (TLM) are combined with HICTL by default, as the prior work (Conneau & Lample, 2019) has verified the effectiveness of them in XLM. In summary, the model can be optimized by minimizing the entire training loss:
= + Ls + Lw.
where LIM is implemented as either the TLM when using parallel data or the MLM when only monolingual data is available to recover the original words of masked positions given the contexts.
3.2 CROSS-LINGUAL FINE-TUNING
Language Understanding. The representations produced by HICTL can be used in several ways for language understanding tasks whether they involve single text or text pairs. Concretely, (i) the [CLS]] representation of single-sentence in sentiment analysis sentence pairs in paraphrasing and entailment is fed into an extra output-layer for classification. (iz) The pre-trained encoder can be
used assign POS tags to each word or to locate and classify all the named entities in the sentence for structured prediction, as well as (úii) to extract answer spans for question answering.
Language Generation. We also explore using HICTL to improve machine translation. In the previous work, Conneau & Lample (2019) has shown that the pre-trained encoders can provide a better initialization of both super- vised and unsupervised NMT systems. [Liu et al. (2020b) has shown that NMT models can be improved by incorpo- rating pre-trained eequence-to-sequene models on vari- ous language pairs but highest-resource settings. As illus- trated in Figure3] we use the model pre- are-trained by HICTL as the encoder, and add new set of decoder parame- ters that are learned from scratch. To prevent pre-trained weights from being washed out by supervised training,
Yo
Y1
Ya
eos
cts
Pre- trained Encodor
Randomly Initialized Decoder
cls 20 X1
X2
sep
bos
Yo
Y1 3/
y'3
Figure 3: Fine-tuning on NMT task.
5

---

Published as conference paper at ICLR 2021
Table
Overall results on XTREME benchmark. Results of mBERT (Devlin et al. 2019
XLM (Conneau & Lample 2019) and XLM-R (Conneau et al. 2020) are from XTREME (Hu et al. Results are from our in-house replication. HNS is short for "Hard Negative Samples".
2020
Pair sentence
Structured prediction
Question answering
Sentence retrieval
Model
XNLI
PAWS-X
POS
NER
XQuAD
MLQA
TyDiQA-GoldP
BUCC
Tatocha
Avg.
Metrics
Acc.
Acc.
FI
FI
FI EM
FI EM
FI/EM
FI
Avc.
Crosr-Fingull somt tranffer [models am trained on Euglish dara)
65.4
64.5
61.4
59.7
56.7
38.7
XLM
69.1
80.9
70.1
61.2
54.3
48.5 32.6
43.6/ 29.1
56.8
32.6
55.5
XLM-RBoc
76.2
63.7
HICTLBax
77.3
84.5
71.4
64.1
58.5
65.8 /47.6
61.97 42.88
XLM-R
79.2
86.4
73.8
65.4
76.6
60.8
53.6/
65.1 45.0
66.0
57.3
68.2
HICTL
87.5
66.2
77.9
72.8
54.5
66.0 48.7
68.4
$9.7
69.6
Translate-trainn-di mmodels trained English training data and tronslated data the longuage)
MBERT
75.1
88.9
72.4/58.33
67.6 /9.88
64.21 49.3
XLM-R
82.9
90.1
74.6
66.8
80.4 65.6
72.4 54.7
66.2 48.2
67.9
59.1
70.6
92.2
.
74.4
71.8
HNS
84.7
92.8
77.2
69.0
82,9 82.9/67.4 67.4
74.8
71.1/53.2
77.6
69.1
74.8
we train the encoder-decoder model in two steps. In the first step, we freeze the pre-trained encoder and only update the decoder. In the second step, we train all parameters for a relatively small number of iterations. In both cases, compute the similarities between the [CLS] representation of the encoder and all target words advance. Then we aggregate them with the logits before the softmax of each decoder step through element-wise additive operation. The encoder-decoder model is optimized by maximizing the log-likelihood bitext at both steps.
EXPERIMENTS
We consider two evaluation benchmarks: nine cross-lingual language understanding tasks in the XTREME benchmark and machine translation tasks (IWSLT' 14 Englishe+German, IWSLT" 14 16 Romanian-<Englin WMT' 14 English-~(Gemman, French))). In this section, describe the data and training details, and provide detailed evaluation results.
IWSLT' English~ Chinese] and
4.1 DATA AND MODEL
During pre-training, follow Conneau et al. (2020) build Common-Crave Corpus using the
CCNet (Wenzek et al. 2019) too for monolingual texts. Table 7 (see appendix A) reports the
language codes and data size in our work. For parallel data, we use the same (English-to-X) MT
dataset as (Conneau & Lample 2019). which are collected from MultiUN (Eisele & Yu 2010) for French, Spanish, Arabic and Chinese, the IIT Bombay corpus (Kunchukuttan et al., 2018a)
for Hindi, the OpenSubtitles 2018 for Turkish, Vietnamese and Thai, the EUbookshop corpus for German, Greek and Bulgarian, Tanzil for both Urdu and Swahili, and VoicesVoices for Swahili. Table (see appendix A) shows the statistics of the parallel data.
We adopt the Soansformer-Enooder (Vawwani et al. 2017) as the backbone with 12 layers and 768 hidden units for HICTI Base- and 24 layers and 1024 hidden units for HICTI. We initialize the parameters of HICTL with XLM-R (Conncau et al. 2020). Hyperparameters for pre-training and fine-tuning shown in Table (see appendix B). We run the pre-training experiments on 8 V100 GPUs, batch size 1024. The number of negative samples m=512 for word-level contrastive learning.
4.2 EXPERIMENTAL EVALUATION
Cross-lingual Language Understanding (XTREME) There nine tasks in XTREME that be grouped into four categories: (i) sentence classification consists of Cross-lingual Natural Lan- guage Inference (XNLI) (Conneau et al. 2018) g and Cross-lingual Paraphrase Adversaries from

---

Published as a conference paper at ICLR 2021
Table 2: Comparison with existing methods on XTREME tasks.
Structured prediction
Question answering
Model
XNLI
PAWS-X
POS
NER
XQUAD
MIQA
TyDiQA-GoldP
Metrics
Acc.
Acc.
FI
FI
F1 /EM
EI /EM
FI/EM
Tramstate-treie-all
FILTER
83.9
91.4
76.2
67.7
82.4/
76.2 57.7
68.3 50.9
VECO
83.0
91.1
75.1
65.7
79.9/ 66.3
73.1/54.9
75.0/ 58.9
HICTL
84.7
92.8
77.2
69.0
82.9/67.4
74.3
71.1/53.2
Table 3: Ablation study on XTREME tasks.
XNLI
PAWS-X
POS
NER F1
XQuAD FI/EM
MLQA FI/EM
TyDiQA-GoldP FI/EM
BUCC
Tatocha
Acc.
Acc.
F1
F1
Acc.
FULL MODEL
84.7
92.8
77.2
69.0
/7.4
57.3
71.1
77.6
69.1
74.8
w/o Sentence-CTL
82.9
90.5
75.9
67.8
82.3 66.7
74.3/ 56.5
52.3
71.4
62.6
72.4
w/o Word CTL
84.3
92.1
76.3
68.4
82.5 66.9
74.1/56.7
70.2//52.5
76.8
68.4
74.2
w/o MT data
84.2
92.4
76.6
68.2
82.6/67.0
74.5 56.8
70.1 /
74.7
66.8
73.8
Word Scrambling (PAWS-X) (Zhang al. 2019b). (+i) Structured prediction includes POS tag- ging and NER. We use POS tagging data from the Universal Dependencies v2.5 (Nivre et al.) 2018) treebanks. Each word is assigned one of 17 universal POS tags. For NER, we use the Wikiann dataset (Pan et al., 2017). (iii) Question answering includes three tasks: Cross-lingual Question Answering (XQuAD) (Artetxe et al. 2019), Multilingual Question Answering (MIQA) (Lewis et al. 2019), and the gold passage version of the Typologically Diverse Question Answering dataset (TydiQA-GoldP) (Clark et al. 2020a). (iv) Sentence retrieval includes two tasks: BUCC (Zwegeen- baum al. 2017) and Tatoeba (Artetxe & Schwenk 2019), which aims to extract parallel sentences between the English corpus and target languages. As XTREME provides no training data, thus we directly evaluate pre-trained models on test sets.
Table provides detailed results on four categories in XTREME. First, compared to the state of the art XL.MR baseline, HICTI. further achieves significant gains of 1.43% and 2.80% aver- age on nine tasks with cross-lingual zero-shot transfer and translate-train-all settings, respectively. Second, mining hard negative samples via smoothed linear interpolation play an important role in contrastive learning, which significantly improves accuracy by 1.6 points on average. Third, HICTL with hardness aware augmentation delivers large improvements on zero-shot sentence retrieval tasks (scores 5.8 and 6.0 points higher on BUCC and Tatoeba, respectively). Following (Hu et al. 2020), we directly evaluate pre-trained models on test sets without any extra labeled data or fine-tuning techniques used in (Fang et al., 2020 Luo et al. 2020). These results demonstrate the capacity of HICTI on learning crosu-lingual representations. We also compare our best model with two exist- ing models: FILTER (Fang et al.) 2020) and VECO (Luo et al. 2020). The results demonstrate that HICTL achieves the best performance on most tasks with less monolingual data.
Ablation experiments are present Table 3 Comparing the full model, we can draw several con- clusions: (1) removing the sentence-level CTL objective hurts performance consistently and sig- nificantly, (2) the word-level CTL objective has least drop compared to others, and (3) the parallel (MT) data has large impact on zero-shot multilingual sentence retrieval tasks. Moreover, Table provides the comparisons between HICTI and existing methods.
Machine Translation The main idea of HICTL is to summarize cross-lingual parallel sentences into shared representation that we term as semantic embedding, using which semantically re- lated words can be distinguished from others. Thus it is natural to apply this global embedding to text generation. We fine-tune the pre-trained HICTL with the base setting on machine trans- lation tasks with both low-resource and high- nigh-resource settings. For the low-resource scenario, choose IWSLT' 14 English (En++Dee IWSLT' 14 English SSpanishh (En-EEs) 16
We split 7k sentence pairs from the training dataset for validation and concatenate dev2010, dev2012, lst2010, tst201 1, tst2012 as the test set.

---

Published as a conference paper at ICLR 2021
Table 4: BLEU scores [%) on high-resource tasks. Results with f and are from VECO (Luo et al. 2020) and our in-house implementation, respectively. In our implementation, we use XLM-R and the best version HiCTL (pre-traind with CCNet-100 and hard negative samples) to initialize the encoder, respectively.
Layers
WMT'
Model
Encoder
Decoder
En->De
En-+Fr
Randomly Initialize
Transformer-Bi Big (Vaswani et al.
201
28.4
41.0
heep-Transfommer (Luu ct al.
2020a
60
12
30.1
43.8
Deep MSC Model (We et al.
2020
18
6
30.56
Pre-trained Models Initialize
CTNMT (Yang et al.
2020
18
6
30.1
42.3
BERT-fused NMT Zhu
2020
30.75
43.78
mBART (Liu et al.
2020b
12
12
30.0
43,2
VECO (Luo et al.
2020)
24
6
31.5
44.4
XLM-R
24
6
30.91
43.27
HICTL
24
6
31.74
43.95
Table 5: BLEU scores [%] on low-resource tasks. Results with are from our in-house imple- mentation. We provide additional experimental results (to follow experiments in Zhu et al. (2020) on IWSLT' 14 English-sSpanish (En->Es task. HICTLBase represents the BASE sized model that is pre-trained on CCNet- with hard negative samples.
IWSLT'14 14
WMT' 16 Ro-
IWSLT'17
Model
En-+De
De De->En
En-*Es
En-+Fr
En En->Zh
Transformer Vaswani
2
28.64
34.51
39.3
33.51
35.8
26.5
BERT-fused NMT (Zhu el al. 2020
30.45
36.11
41.4
39.10
38.7
28.2
HiCTL
31.88
37.96
42.1
39.88
40.2
29.9
Comaaiia--Engllinn (Ro-EEn), IWSLT'17 English- (En->Fr) and English-
(En-*Zh) translation There are 160k, 183k. 236k, 235k, 0.6M bilingual sentence pairs for Ent+De, En-+Es. En-+Fr, En and Ro-=En tasks. For the rich-resouree scenario, we work on WMT' 14 En-+ Fr). the corpus sizes are 4.5M and 36M respectively. We concatenate newstest 2012 and newstest 2013 as the validation set and use newstest 20/4 as the test set.
During fine-tuning, we use the pre-trained model to initialize the encoder and introduce randomly initialized decoder. We develop a shallower decoder with identical layers to reduce the computa- tion overhead. At the first fine-tune step, we concatenate the datasets of all language pairs in either low-resource or high-resource settings to optimize the decoder only until convergence" Then we tune the whole cncoder-decoder model using per-language corpus the second step. The ini- tial learning rate 2c-5 and inverse sqrt learning rate (Vaswani et al. (017)) scheduler is also adopted. For WMT" 14 En En, we beam search with width 4 and length penalty 0.6 for infer- ence. For other tasks, we use width and length penalty of 1.0. We use mult it-ble perl evaluate IWSLT' 14 En and WMT tasks, but sacreBLEU for the remaining tasks, for fair comparison with previous work.
Results both high-resource and low-resource tasks reported in Table 4 and Table respec- tively. We implemented standard Transformer (apply the base and big setting for IWSLT and WMT tasks respectively) as baseline. The proposed HICTL can improve the BLEU scores of the eight tasks by 3.34, 2.95, 3.24, 3,45, 2.8, 6.37, 4.4, and 3,4. In addition, our approach also out- performs the BERT-fused model Yang et al. 2020 a method treats BERT as an extra context
https: / t. Fbk. eu/mt. php?re ease- 2000e-201--111 -1------eeees
Zhao al. (020) conducted a theoretical investigation on learning universal representations for the task of multilingual MT, while we directly use shared encoder and decoder across languages for simplicity.
8

---

Published as a conference paper at ICLR 2021
Table BLEU scores [%) sero MT via Language Transfer. We bold the highest trans- ferring score for each language family.
Fine-tuning Languages
Test Languages
Cn
Hi-+En
mBART
HiCTL
mBART HiCTL
Cs-En
21.6
22.4
Ro-=En
19.5
19.0
It=En
16.7
18.6
NI-$En
17.0
18.1
Hi Hi-+En
23.5
25.2
Ne-tEn
14.5
16.0
Si+EEn
13.0
14.7
Gu-Enn
0.0
0.1
and fuses the representations extracted from BERT with each encoder and decoder layer. Note we achieve new state-of-the-an results IWSLT' 14 En->ee, IWSLT' 17 En->(Fr, Zh} translations. These improvements show that mapping different languages into a universal representation space is beneficial for both low-resource and high-resource translations.
We also evaluate our model on tasks where no bi-text is available for the target language pair. Fol- lowing mBART (Liu et al., 2020b), adopt the setting of language transfer. That is, no bi-text for the target pair is available, but there bi-text for translating from some other language into the target language. For explanation, supposing there is parallel data for the target language pair Itngish (It+=En,, but we can transfer knowledge learned from Czech-tEnglish (Cs+>En, high-resource language pair) to It It-En. We consider X-=nn translation, covering Indic languages (Ne, Hi, Si, Gu) and European languages (Ro, It, Cs, NI). For European languages, we fine-tune on Cs->En translation, the parallel data is from WMT' 19 that contains 1M sentence pairs. We test on fCs, Ro, It, -n, in which test sets are from previous WMT (Cs, Ro) or IWSLT (It, NI) competitions. For Indic languages, we fine-tune on Hi-=E translation (1.56M sentence pairs are from IITB Kunchukuttan et al. 2018 2018b)), and test on (Ro, It, Cs, NII-En translations.
Results are shown in Table 6 We can always obtain reasonable transferring scores low-resource pairs over different fine-tuned models. However, experience shows that the randomly initialized models without pre-training always achieve near BLEU. The underlying scenario that multilin- gual pre-training produces universal representations across languages that once the model leams to translate one language, it learns to translate all languages with similar representations. Moreover, failure happened in Gu->En translation, conjecture that only 0.3GB monolingual data for pre-training, which difficult to learn informative representations for Gujarati.
5 CONCLUSION
We have demonstrated that pre-trained language models (PTMs) trained to learn commonsense knowledge from large-scale unlabeled data highly benefit from hierarchical contrastive learning (HICTL), both in terms of cross-lingual understanding and generation. Learning universal repre- sentations at both word-level and sentence-level bridges the semantic discrepancy across languages. As result, HICTL sets a new level of performance among cross-lingual PTMs, improving on the state of the art by a large margin.
ACKNOWLEDGMENTS
We would like to thank the anonymous reviewers for the helpful comments. We also thank Jing Yu for the instructive suggestions. This work is supported by the National Key R&D Program China under Grant
and No. 2018YFB 1403202.

---

Published as a conference paper at ICLR 2021
REFERENCES
Mikel Artetxe and Holger Schwenk. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7:597-610, 2019.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. On the cross- looss-lingual transferability of mono- lingual representations. arXiv preprint /9X0v 2019.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large an- notated corpus for learning natural language inference. In Proceedings of the 2015 Confer- ence on Empirical Methods in Natural Language Processing, Pp. 632-642, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10.18655/11/DS--107.. URL https: /7 www ac web. wrwbboorg/attooooo www.aannoooooooooooooegee ogy/D15-1072
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Józefowicz. and Samy Bengio. Generating sentences from continuous space. In Proceedings the 20th SIGNLL Con- ference on Computational Natural Language Learning, CONLL 2016, Berlin, Germany, August 11-12, 2016, pp. 10-21, 2016. URL URL
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of Machine Learning and Systems 2020, pp. 10719 2020.
Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. Infoxlm: An information-theoreiic framework for cross- lingual language model pre-training. CoRR, 0bs/2007.07834 2020. URL https://arxiv.
erq/abs. 2007. 07834
Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. Tydi qa: A benchmark for information-seeking question answering typologically diverse languages. arXiv preprint arXiv:2003. 2020a.
Kevin Clark, Minh-Thang Luong, Quoc Le, and Christopher Manning. ELECTRA: pre- training text encoders as discriminators rather than generators. In 8th International Conference on Learning Representations, ICLR 2020. OpenReview.net, 2020b.
Alexis Conneau and Guillaume Lample. Cross-lingual language model pretraining. In
Proc. of NIPS 2019, pp. 7059-7069, 2019. URL http://paperrs nips. cc/paper/ 8928 cross Lingual -ao---nnguage model www.-ppeeeraannnnnneeeee
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. XNLI: Evaluating Pross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp- 2475-2485 Brussels, Belgium, October-November 2018. Association for Computational Lin- guistics. doi: 10.18653/v1/D18-112 1269. URL https::t//// //t::///wwwwww aalwwelweee www.:aawwwbb.ge...eee...... www.bb///ogottoooogo..ooooo 218-1269
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8440-8451, Online, July 2020. As- sociation for Computational Linguistics. URL https: http://wwwwwwwwbb////p/pt www..ntholog/../////oooooooo..// acl-ma.c 747
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long and Short Papers), pp. 4171-4186 Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/119-11 1423. URL https: 7/ 7/www. web. www..oogg//aahhooooo//iiill/////
10

---

Published as a conference paper at ICLR 2021
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao- Wuen Hon. Unified language model pre-training for natural language understanding and generation. In Advances in Neural Information Processing Systems 32, NeurIPS 2019, pp- 13063-13075. Curran Associates, Inc., 2019.
Sergey Edunov, Alexei Baevski, and Michael Auli. Pre-trained language model representations for language generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Papers), pp. 4052-400 4059, Minneapolis, Minnesota, June 2019. Association for Com- putational Linguistics. doi: 10.18653/11/119-44099
URL https: wwww www.ww...brrr.o....... org/
Sathologyy/NI914000
Andreas Eisele and Chen Yu. Multiun: A multilingual corpus from united nation documents. In International Conference Language Resources & Evaluation, 2010.
Yuwei Fang, Shuohang Wang, Zhe Gan, Siqi Sun, and Jingjing Liu. FILTER: an enhanced fusion
method for cross-lingual language understanding. CoRR, abs/2009.05166, 2020. https:
//wxxxv.org/aab/////// /0tpgg//aa///00...0.. 05166
Fangxiaoyu Feng, Yinfei Yang. Daniel Cer, Naveen Arivazhagan, and Wei Wang. Language-
agnostic BERT sentence embedding. CoRR, abs/2007.01852, 2020. URL htthhttp///aaat
hrg/abs/20011 01852
Junliang Guo, Zhirui Zhang, Linli Xu, Hao-Ran Wei, Boxing Chen, and Enhong Chen. Incorporat- ing bert into parallel sequence decoding with adapters. In H. Larochelle, M. Ranzato, Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 10843-10854. Curran Associates, Inc., 2020. URL /RLLhtttts://proeeeee ngs
htw.:/ceeeddnns..u.ii....e neurips. Paper. Saperrpor pdd
Cc
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recogni- tion (CVPR 2006), pp. 1735-1742. IEEE Computer Society, 2006.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, pp. 770-778. IEEE Computer Society, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. CoRR, abs/1911.00570 2019.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual general- ization. CoRR, abs/2003. 1080, 2020. URL http://axvvv.. org abwa/b//ooo 11080
Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjum Shou, Daxin Jiang, and Ming Zhou. Unicoder: A universal language encoder by pre-training with multiple cross- coos-lingual tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNIP--JCNILP), pp. 2485-2494 Hong Kong, China, November 2019, Association for Computational Linguistics. URL https: / /wwww wwwwwb.w.b........ www.:wwww.g......... wtt.://ggggttoooy////aoao 1252
Dan Iter, Kelvin Guu, Larry Lansing, and Dan Jurafsky. Pretraining with contrastive sentence ob- jectives improves discourse performance of language models. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4859 4870, Online, 2020. As- sociation for Computational Linguistics. URL htttt::///// petttp:://ww... /wwww acl www.. orgganthology/
acl.acllmaaa. 439.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. Spanbert: Improving pre-training by representing and predicting spans. Trans. Assoc. Comput. Linguistics, 8:64-77, 2020. URL Inttps://transatt org/ //t////////ddx...///// /i///ddx..... http//iee...pptt..ll..
article/view
11

---

Published as a conference paper at ICLR 2021
Lingpeng Kong, Cyprien de Masson d' Autume, Lei Yu, Wang Ling, Zihang Dai, and Dani Yo- gatama. A mutual information maximization perspective of language representation learning. In 8th International Conference Learning Representations, ICLR 2020. OpenReview.net, 2020.
Anoop Kunchukuttan, Pratik Mehta, and Pushpak Bhattacharyya. The IIT bombay english-hindi parallel corpus. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation, LREC 2018. European Language Resources Association (ELRA). 2018a.
Anoop Kunchukuttan, Pratik Mehta, and Pushpak Bhattacharyya. The IIT Bombay English- Hindi parallel corpus. In Proceedings of the Eleventh International Conference Language Resources and Evaluation (LREC 2018), 2018b. URL https:/ //op:://wwwww http://wwwclllclc.cc aclweb. perggannhologyy/
L18 118-1548
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite BERT for self-supervised learning of language representations. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia. April 26-30, 2020. OpenReview.net, 2020. URL hnttps://openreeeee ew net/ / forum? i H1eA7ABEVS
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequencc pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7880 Associa- tion for Computational Linguistics, 2020.
Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. Mlqa: Evaluating cross-lingual extractive question answering. arXiv preprint 19Xiv: 2019.
Xiaodong Liu, Kevin Duh, Liyuan Liu, and Jianfeng Gao. Very deep transformers for neural ma- chine translation. arXiv preprint arXiv:2008.0777 2020a.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov, Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692 2019,
Yinhan Liu, Jiatao Gu. Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. CoRR, abs/2001.082100 2020b.
Fuli Luo, Wei Wang, Jiahao Liu, Yijia Liu, Bin Bi, Songfang Huang, Fei Huang, and Luo Si. VECO: variable encoder-decoder pre-training for cross lingull understanding and generation. CoRR, abs/2010. 16046, 2020. URL htt.tt://aaaxiiiise www.:i.oooroaaaa.. wwtp:///ggaab/////////// 20100 16046
Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. Distributed rep- resentations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26, pp. 3111-3119, 2013.
Andriy Mnih and Koray Kavukeuoglu. Learning word embeddings efficiently with noise-contrastive estimation. In Advances in Neural Information Processing Systems 26, pp. 2265-2273, 2013.
Joakim Nivre, Mitchell Abrams, Zeljko Agic, Lars Ahrenberg, Lene Antonsen, and al. Universal Dependencies 2.2. 2018. URL https:/ / h/wahhaaa.. arallarchh. www.aarchiiisss.. www.rciiiisoovvtteeeeeeeeee fr/
ha1-01930733 LINDAT/CLARIN digital library at the Institute of Formal and Applied Lin- guisties (ÚFAL), Faculty of Mathematics and Physics, Charles University.
Aäron van den Oord, Yazhe Li. and Oriol Vinyals. Representation learning with contrastive predic- tive coding. CoRR, abs/1 607.03748, 2018. URL htttttt/p/////ii www.ooggg/////... 1807 03748
Xiaoman Pan, Boliang Zhang, Jonathan May. Joel Nothman, Kevin Knight, and Heng Ji. Cross-
lingual name tagging and linking for 282 languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 1946-1955 Vancou- ver, Canada, July 2017. Association for Computational Linguistics. doi: 18.186 1/P17 1178. URL https:: www. a c web. web.orgg/attol wgw.tttohoooyyyyyoooop
12

---

Published as a conference paper at ICLR 2021
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Con- ference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume (Long Papers), pp. 2227-2237, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18633/1//118-12000 URL
https: http://wwcoeee..oogaaaaaoooooo/a -
Alec Radford, Karthik Narasimhan, Tim Salimans, and llya Sutskever. Improving language un- derstanding by generative pre-training. URI. https://s3-us-wests amazonaws. com/openai- understanding paper. pdf, 2018.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: 10683, 2019.
Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoretical analysis of contrastive unsupervised representation learning. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pp. (6628-6637 Long Beach, California, USA 09-15 Jun 2019. PMLR. URL http:///proceedings mlr. press/ v97/sauns 9a. ht.ll
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. pp. 1631-1642, Seattle, Washington, USA, October 2013. Association for Computa- tional Linguistics. URL helphttss://// //ww::::/www.... acl org/ ant htt.//ooooooooooo 1170
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. MASS: masked sequence sequence pre-training for language generation. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, volume 97, pp. 5926-5936. PMLR, 2019.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision - ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI, volume 12356 of Lecture Notes in Computer Science, pp. 776-794. Springer. 2020. doi: 10.1007/ 45. URL https:://////. /ttp://ooi...e 0ttp:///100100011---000000000000
45
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances Neural Infor- mation Processing Systems 30, NIPS 2017, pp. 5998-5998999 6008. Curran Associates, Inc., 2017. URL nips cc wttpcc//ppeeee wwtpcccpppppppp..ceece http://71111attttne. attaattttoonnini
you--oou-eeeeee need. pdd]
Xiangpeng Wei, Heng Yu, Yue Hu, Yue Zhang, Rongxiang Weng, and Weihua Luo. Multiscale collaborative deep models for neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 414-426, Online, July 2020. As- sociation for Computational Linguistics. URL htttps::/wwwtt wwwwwb.wwwcccccoco how.aa/oooo./at/oogggooo
2220 2010.acl-oo maiin. 40
Rongxiang Weng, Heng Yu, Shujian Huang, Shanbo Cheng, and Weihua Luo. Acquiring knowledge from pre- trained model to neural machine translation. In Proceedings of the AAAI Conference on Artificial Intelligence, pp.
2020.
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzman, Armand Joulin, and Edouard Grave Cenet: Extracting high quality monolingual datasets from web crawl data, arXiv preprint arXiv:1911.0000 2019.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-com cover-co challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, Volume I (Long Papers). pp. 1112-1122, New Orleans, Louisiana, June 2018. Association
13

---

Published as a conference paper at ICLR 2021
for Computational Linguistics. doi: 10.18653//1/188-1UU URL https: www acwweb.
orq/ hate/anntholooy//1B-1100
Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, pp. 3733-3742. IEEE Computer Society, 2018.
Jiacheng Yang, Mingxuan Wang, Hao Zhou, Chengqi Zhao, Weinan Zhang, Yong Yu, and Lei Li. Towards making the most of BERT in neural machine translation. In The Thirty-Fourth Fourth AAAI Conference Artificial Intelligence, AAAI 2020, pp. 9378-9385. AAAI Press, 2020. URL
https://aaai: wwwoorg//jo.... httpex..pt//AA./aatiicl////es/s 6479
Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neu- ral Information Processing Systems 32, NeurIPS 2019, pp. 5753-5763. Curran Associates, Inc., 2019.
Mang Ye, Xu Zhang, Pong C. Yuen, and Shih-Fu Chang. Unsupervised embedding learning via invariant and spreading instance feature. In JEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, pp. 6210-6219. Computer Vision Foundation IEEE, 2019.
Xingxing Zhang, Furu Wei, and Ming Zhou HIBERT: Document level pre-training of hierar-
chical bidirectional transformers for document summarization. In Proceedings of the 57th An- nual Meeting of the Association for Computational Linguistics, pp. 5059-5069, Florence, Italy, July 2019a. Association for Computational Linguistics. doi: 10. 10.18653/1//11914999 URL www. ac web. www.g//aacooooooooohchhoooooool
Yuan Zhang, Jason Baldridge, and Luheng He. PAWS: Paraphrase adversaries from word scram- bling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Pa- pers), pp. 1298-1308, Minneapolis, Minnesota, June 2019b. Association for Computational Lin- guisties. doi: 10.18653/1I/N11-1113. URL httph:tt/////... //www. ac N19-1131
Han Zhao, Junjie Hu, and Andrej Risteski. On learning language-invarin invariant representations for uni- versal machine translation. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learn- ing Research, pp. 11352-11364. PMLR, 13-18 Jul 2020. URL URU/attp//ppcccedingg mir. 0tsss//v19//hao2000
Wenzhao Zheng, Zhaodong Chen, Jiwen Lu, and Zhou Jie. Hardness-award deep met- IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA. USA, June 16-20, 2019, pp. 72-81, 2019. URL wwtp://..coo/o/cecctteeee content Bertent__VPRRRR CVPR 2019/html/ htth////m/////gggggg//
ric learning.
http:///penacces.. thecvf. com/ Hardness-Aware Deep _ee___eeeee____ieeeeeeeee Metric Searrric__earnnnnnn CVPR_ 2019 2_ppp__pp_______1____11___ html
Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. Extractive
summarization as text matching. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp. 6197-6208. Association for Computational Linguistics, 2020.
URL http:::tt:///tt acl web. org/ ant how.hhholyyy/a/ 2020 acl main. 552
Jinhua Zhu, Yingce Xia, Lijun Wu, Di He, Tao Qin, Wengang Zhou, Houqiang Li, and Tie- Liu. Incorporating BERT into neural machine translation. In 8th International Conference on Learning Representations, ICLR 2020. OpenReview.net, 2020.
net
Pierre Zweigenbaum, Serge Sharoff, and Reinhard Rapp. Overview of the second BUCC shared task: Spotting parallel sentences in comparable corpora. In Proceedings of the 10th Work- shop on Building and Using Comparable Corpora, pp. 60-67, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10.18653/w1/117-25522 URL https:
w/www. ac www.wwww....oooc wwt.:/wwww..r....ttoooooo////aaa
14

---

Published as a conference paper at ICLR 2021
Table The statistics of CCNet corpus used for pretraining.
Code
Size (GB)
Code
Size (GB)
Code
Size (GB)
Code
Size (GB)
Code
Size (GB)
af
1.3
6.1
ja
24.2
mt
0.2
sq
3.0
am
0.7
2.0
jv
0.2
0,9
Sr
5.1
20.4
21.6
ka
3,4
2.6
0.1
as
0.1
19.2
kk
2.6
15.8 SV
10.8
az
3.6
fr
46.5
km
1.0
по
3.7 SW
1.6
be
3.5
0.2 kn
1.2
0.1
ta
8.2
bg
22.6
ga
0.5 ko
17.2
0.6
tc
2.6
bn
7.9
gd
0.1 ku
0.4
pa
0.8 th
14.7
br
0.1
2.9 ky
1.2
pl
16.8
d
0.8
bs
0.1
gu
0.3 la
2.5
ps
0.7
17.3
ca
10.1
lo
0.6
pt
15.9
ug
0.4
cs
16.3
he
6.7 It
7.2
TO
8.6
uk
9.1
cy
0.8
hi
20.2 lv
6.4
48.1
5.0
da
15.2
hr
5.4
mg
0.2
0.3
UZ
0.7
de
46.3
9.5
mk
1.9
sd
0.4
vi
44.6
el
29.3
hy
5.5
ml
4.3
si
2.1
xh
0.1
49.7
10.6
1.7
sk
4.9
yi
0.3
eo
0.9
mr
1.3
2.8
zh
36.8
es
44.6
19.8
ms
3,2
S0
0.4
Table 8: Parallel data used for pre-training.
Code
Sentence Pair (#millions)
Code
Sentence Pair (#millions)
en-ar
9.8
en-ru
11.7
en-bg en-de
0.6
en-sw
0.2
9.3
en-th
3,3
en-el
4.0
en-tr
0.5
en-es
11.4
en-ur
0.7
en-fr
13.2
en-vi
3.5
en-hi
1.6
en-zh
9.6
A PRE-TRAINING DATA
During pre-training, follow Conneau et al. (2020) to build Common- Crawl Corpus using the CCNet (Wenzek et al. 2019) too for monolingual texts. Table 7 reports the language codes and data size in our work. For parallel data, we the same (English-to--X MT dataset (Conneau & Lample 2019), which are collected from MultiUN (Eisele & Yu 2010) for French, Spanish, Arabic and Chinese, the IIT Bombay corpus (Kunchukuttan et 201 for Hindi, the OpenSubtitles 2018 for Turkish, Vietnamese and Thai, the BUbookshop corpus for German, Greek and Bulgarian, Tanzil for both Urdu and Swahili, and GlobalVoices for Swahili. Table Biie@sshwww the statistics of the parallel
data.
B HYPERPARAMETERS FOR PRE-TRAINING AND FINE-TUNING
As shown in Table 9 we present the hyperparameters for pre-training HICTL. We the same vocabulary as well as the sentence-piece model with XLM-R (Conneau et al. (020). During fine- tuning on XTREME, we search the learning rate over (5e-6, le-5, 1.5e-5, 2c-5, 2.5e-5, 3e-5) and batch size over {16, 32) for BASE-siZe models. And we select the best LARGE-size model by searching the learning rate over (3e-6, 5e-6, le-5) as well as batch size over (32, 64).
15

---

Published as a conference paper at ICLR 2021
Table Hyperparameters used for pre-training.
Hyperparameters
BASE
LARGE
Number of layers
12
24
Hidden size
768
1024
FFN inner hidden size
3072
4096
Attention heads
12
16
Mask percent (monooliguual/illiguuaa)
15%/25%
15%/25%
Adam
le-6
le-6
Adam
(0.9, 0.98)
(0.9, 0.999)
Learning rate
2.5e-4 linear
le-4
Learning rate schedule
linear
Warmup steps
10,000
10,000
Attention dropout
0.1
0.1
Dropout
0.1
0.1
Max sequence length (monolingual/bllinguaa)
256
256
Batch size
1024
1024
Training steps
200k
200k
Table 10: Results Cross-lingual Natural Language Inference (XNLI) for each language. We
report the accuracy on each of the 15 XNLI languages and the average accuracy of our HICTL as well five baselines: BiLSTM (Conneau et al. 2018), mBERT (Devlin al. 2019), XLM (Conneau &
Lample, 2019), Unicoder (Huang et al. 2019) and XLM-R (Conneau et al. 2020). Results of $ are from our in-house replication.
MODEL
en
fr
de
bg
SW
ur
Avg
Evaluation of crasy lingual sentence
(Cross lingual transfer)
BiL.TMM
73.7
67.7
68.7
67.7
68.9
67.9
65.4
64.2
64.8
66.4
64.1
65.8
64.1
55.7
58.4
65.6
mBERT
81.4
74.3
70.5
62.1
63.8
58.3
XLM
85.0
78.7
78.9
77.8
76.6
77.4
75.3
72.5
73.1
76.1
73.2
76.5
69.6
68.4
67.3
75.1
Unicoder
85.1
79.0
79.4
77.8
77.2
77.2
76.3
72.8
73.5
76.4
73.6
76.2
69 4
69.7
66.7
75.4
XLM-RBae
85.8
80.7
78.7
68.3
76.2
HICTLuse
86.3
80.5
81.3
79.5
78.9
80.6
79.0
75.4
74.8
77.4
75.7
77.6
73.1
69.9
69.7
77.3
Machine mumslate at training (Tramslate-troin))
BiLSTM
73.7
68.3
68.8
66.5
66.4
67.4
66.5
64.5
66.0
62.8
67.0
58.2
56.6
65.4
mBERT
81.9
77.8
75.9
70.7
76.6
61.6
XLM
85.0
80.2
80.8
80.3
78.1
79.3
78.1
74.7
76.5
76.6
75.5
78.6
72.3
70.9
63.2
76.7
Unicuder
85.1
80.0
81.1
79.9
77.7
80.2
77.9
75.3
76.7
76.4
75.2
79.4
71.8
71.8
64.5
76.9
HICTLasee
85.7
81.3
82.1
80.2
81.4
81.0
80.5
79.7
77.4
78.2
77.5
80.2
75.4
73.5
72.9
79.1
Fine-Tume multitingual model on all training sets (Tromsiate-trabn-oll)
XLM
81.3
79.1
68.5
Unicoder
85.6
$1.1
82,3
80.9
79.5
81
79.7
76.8
78.2
77.9
77.1
80.5
73.4
73.8
69.6
78.5
XLM-RRaaee
81.4
82.2
80.3
80.4
81.3
79.7
77.3
76.
73.1
73.0
79.
HICTLBux
86.5
82,3
83.2
80.8
81.6
82.2
81.3
80.5
78.1
80.4
78.6
80.7
76.7
73.8
73.9
80.0
XLM-R
89.1
85.1
86.6
85.7
85.3
85.9
83.5
83.2
83.1
83.7
81.5
83.7
81.6
78.0
78.1
83.6
XLM-R
88.9
84.7
86.2
84.8
85.0
85.3
82.4
82.7
82.4
82.8
80.9
83.0
80.2
77.3
77.2
82.9
HICTE
89.3
85.5
86.9
86.1
85.7
86.1
83.7
83.9
83.3
83.5
81.8
84.2
81.0
78.4
77.9
83,8
C RESULTS FOR EACH DATASET AND LANGUAGE
Below, we provide detailed results for each dataset and language on XTREME, as shown in Table 10 14 Results of XLM-R from our implementation.
D VISUALIZATION OF SENTENCE EMBEDDINGS
We collect 10 sets of samples from WMT' 14-19, each of them contains 100 parallel sentences distributed in 5 languages. As the t-SNE visualization Figure set of sentences under the same meaning clustered more densely for HICTI than XLM-R, which reveals the strong capability
16

---

Published as a conference paper at ICLR 2021
Table 11: PAWS-X accuracy scores for each language.
Model
dc
fr
ja
ko
zh
avg
Trunslate-train-all
XLM-R
95.7
92.2
92.7
92.5
84.7
85.9
87.1
90.1
HICTL, Wiki-15 + MT
96.6
93.2
93.3
92.9
86.5
87.3
88.6
91.2
HICTL, CCNet-100 + MT
96,9
93.8
94.4
94.3
88.0
88.2
89.4
92.2
+HARD NEGATIVE SAMPLES
97.4
94.2
95.0
94.2
89.1
89.5
90.2
92.8
Table 12: POS results (Accuracy) for each language.
Model
af
de
cl
ct
Eo.
1i
fr
he
hi
hu
id
it
traniaat--rrin-all
XLM-R
HICTI, WIRL-5
74.8
HICTI, CONG--100 MT
01.8
70.2
90.7
90.8
80.0
98.3
89.7
90.1
76.2
73.0
88.
90.2
70.7
50.0
86.4
74.5
92.0
&HARD NEGATIVE SAMPLES
92.2
71.0
91.5
91.3
90.0
97.7
91.0
89.4
75.7
73.5
88.8
50.1
71.1
79.7
85.4
75.1
91.7
y0
db
Tramiate-tmin--tll
HICTL, Wiki-1 a MT
28.4
79.2
54.2
80.7
90.9
88.4
90.5
67.3
89.1
48.7
92.2
77.6
72.0
58.8
27.2
27.1
75.5
HiCT,, CCNet-100 a MT
30.2
80.4
55.1
82.1
91.2
90.2
90..7
68.1
90.1
50.3
95.2
78.7
73.3
59.2
27.8
27.9
76.8
+HARD NEGATIVE SAMPLES
31,9
80.9
57.0
83.5
91.7
91.0
91.2
69.5
90.8
50.3
94.8
79.4
73.4
59.5
28.6
28.7
77.2
of HICTL on learning universal representations across different languages. Note that the t-SNE visualization of HICTL still demonstrates some noises, we attribute them to the lack of hard negative examples for sentence-level contrastive learning and leave this to future work for consideration.
17

---

Published as a conference paper at ICLR 2021
Table 13: NER results (FI) for each language.
XLM-R
86.8
K1.4
55.2
82.9
81.1
79.1
$1.5
81.1
61.3
4066
64.1
80.6
K3.2
A8
76.1
79.4
55.2
80.7
22.7
63.9
HICTL Wike-5 MT
87.0
K2.3
55.2
84.7
70.0
81.2
50.1
81.5
79.5
61
82.8
R0.5
604
74.6
79.5
54.8
83.5
24.9
66.1
HiCTL CCN--1101 MT
88.6
10-9
55.4
88.6
81.8
K7D
$2.5
81.2
60.5
66-2
61-2
K3,8
77.8
54.4
55.8
WX.7
26-0
66
HEARD NEGATIVE SAMPLES
58.9
$2.0
56.6
$5.7
83.4
$2.5
$4.
83.0
83.8
65.4
65.4
R2.0
$2.6
60.5
74.7
$1.5
S8L1
84.7
27.9
65.9
8k
m!
m06
my
11
P1
1W
53
10
1h
17
11
30
W
75.0
HICTL OCNa-100 MF
72.8
57.6
64.6
70.4
71.5
61.4
5901
81.7
88.1
90.3
24.3
60.6
$7.9
5.6
77.5
79.0
59.8
83.7
53.5
96.9
+BEARD NEGATIVE SAMPLESS
76.8
60.9
65.0
71.4
72.5
90.0
56.3
85.9
84.5
71.4
75.6
62.9
58.8
3.9
77.7
50.4
59.1
83.6
37.7
37.2
Table 14: Tatoeba results (Accuracy) for each language
Model
bg
bn
de
dl
Fa
6
fr
he
hi
hu
id
it
Translate mom ol
M-M
90.7
50.5
72.2
45.4
89.5
61.3
77.6
51.7
38.6
71.7
72.8
76.9
66.3
73.1
65.
77.5
68.5
63.1
[ICTL, WiL: MT
61.5
51.4
76.1
92.1
63.4
80.5
55.9
37.8
74.5
76.7
78.0
68.4
74.5
80.4
70.2
63.9
HICTL, CCNer-100 100 MT 4HARD
50.9
76.8
98.5
68.8
80.9
59.3
41.5
77.3
78.2
80.3
70.2
77.9
81.3
73.7
57.7
82.9
p5
13
36.7
HICTL, Wi6--15
61.2
28.6
41.6
71.2
80.2
HICTL, CCNer-100
57.3
$1.6
68.0
21.8
62.0
88.1
68.9
77.7
26.1
32.9
39.5
32.9
43.2
21.2
27.8
79.9
74.7
4HARD NEGATIVE SAMPLUS
27.3
63.0
61.5
72.6
75.3
67.8
92.8
92.8
85.4
32.0
36.7
47.8
41.5
49.8
77.0
34.3
84.3
81.3
60
English
English
30
French
French
German
Russian Spanish
Spanish
20
10
0.
-10
-20
-40
-15 1.0
15 20
-20 -10
10
20
Figure 4: Visualizations (t-SNE projection) of sentence embeddings output by HICTL (left) and XLM-R (right). We collect 10 sets of samples from WMT'144199, each of them contains 100 parallel sentences distributed in languages (i.e., English, French, German, Russian, and Spanish). Each set is identified by color and different languages marked by different shapes. We can that a set of sentences under the same meaning are clustered more densely for HICTL than XLM-R, which reveals the strong capability of HICTL learning universal representations across different languages.
18


Reviews:
[{"review_id": "ras1CtT7zq", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a pre-trained language model variant which extends XLM-R (multilingual masked model) with two new objectives. The main difference to most other models is that the new losses are contrastive losses (however, as pointed out by the authors, other contrastive losses had been used before in e.g. ELECTRA). The first additional loss is a sentence-level one - where a [CLS] token is trained to be close to the positive sample, the paired sentence, with other sentences as negative samples. The same is done at word level, where the bag of words constructed from two sentences becomes the set of positive samples and other vocabulary words are negative samples. \nContrastive losses are promising and the paper shows positive results when adding them to the previously proposed XLM-R model. The review of previous work is thorough and very helpful to place the work proposed in the existing literature. However I had difficulties understanding the impact of the changes proposed and disentangling the different factors that may have led to the results. At the end of reading this paper, I am not sure if implementing what the authors proposed, versus other variations of existing models, would have given the same improvements: While these improvements can be seen across many data sets, they are often modest. The proposal does not offer any other advantages, such as computational efficiency. \nFor the NMT experiments, additional experiments on En-Es and En-Ro (to follow experiments in Zhu et al 2020), and/or back-translation experiments would have made the impact of the method clearer. Given that the main contribution of the paper is empirical (none of the ideas are new), better and more comprehensive experimental results would have strengthened this work.\nThe following are clarification questions/comments: \n- The query and key terminology used in section 2 is confusing: why not use negative/positive sample notation from the Saunshi et al, 2019 and  Oord et al, 2018 papers? Section 3.1 introduces r_x, which is yet another notation for the query q. \n- Figure 1: please clarify the notation used in the caption (e.g. the set B is defined only later, similarly n, m, V). \n- The losses in equations (2) and (3) are symmetric: if the data pairs are symmetric, which seems to be the case, why distinguish between queries and keys at all and define two identical, but symmetric losses? \n- First paragraph in Word-Level CTL in Section 3.1: This should be rephrased in order to clarify the motivation for the word level loss. \n- I couldn't find details regarding the negative samples for the sentence loss: no of negative samples, how are they obtained, etc.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "MC9UvNq1Qp", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThe work applies and adjusts contrastive learning in the subject area of pre-training language models. The work first identifies the challenges with the current landscape of Masked Language Models with limits to learning sentence-level representations and semantic alignments in sentences of different languages. To take care of these gaps, the authors propose using HCTL as an approach that can learn more universal representations for sentences across different languages. The work builds on top of the BERT models, with the adjusted contrastive learning objective goal.\n\nStrengths\n\nThe paper is well written. The authors work to clearly identify shortcomings in the current literature. The identification of contrastive learning as a possible approach to learn better representations. \n\nQuestions\n\n- Using the [CLS} token has been one of the ways to capture the sentence embedding for BERT, correct? I do understand that there are other ways to try to extract sentence embedding from pre-trained BERT, but your approach is not new in this sense?\n\n- I am not sure I missed it, but I do not have an objective view on the computational overhead (especially on having to sample the right-hand side of the non-similar keys). Simply, how much more time do I spend using this type of training as compared to the prior approaches? \n\n- With the prior question, the results, show consistent improvements (your biggest strength along with the representation itself being comparable) but are a slight improvement in individual metrics (a point or two). Given the goal of developing universal cross-lingual representations, why is it that if we do learn these better representations are we not doing Very Very well?\n\n- Might it be that the prior models, already have these representations already embedded in their models and your work just extracted them? Could there be another approach to XML-R for example, that extracts these representations?\n\n- Thank you for including different sizes of corpora and also some low-resource languages. Kiswahili is the smallest parallel data that you have, and I wondered what would happen with a smaller language? For low-resource languages, it would be insightful to also have examples of where the failures happen and why.\n\n- Please expand the future 3 captions. it is hard to understand what it represents from the caption. Just make sure the caption is completely descriptive. The figure is also very small and hard to read. \n\nI think the contrastive learning approach is very interesting for this use-case and made my reading and evaluation of this paper more interesting. I look forward to the responses from the authors. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "3TbJ4NBxa_j", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper presents HICTL which enables models to learn sentence level representations and uses contrastive learning to force better language agnostic representations for large multilingual encoders. They obtain significant gains on the XTREME benchmark and also on standard MT benchmarks.\n\nReasons for score:\nI score this paper a 6. The constrastive losses introduced in the paper are interesting. The sentence-level and word-level CTL definitely seem to have an impact on the downstream performance. The improvements on XTREME over XLM-R is pretty strong. However, the improvements in MT are not that convincing. I would have also liked to see more ablations. Finally, the authors initialize from XLM-R and fine-tune on 15 languages with both monolingual and parallel data. I would have liked to see this number (i.e. number of languages) be much larger.\n\nCons:\n- Why did you only choose to fine-tune on the 15 languages from XNLI? \n- Can you present the breakdown of results in different tasks by language as an appendix? Specifically, I would like to see if the improvements are only coming in the 15 languages you fine-tune on or on others as well. Do you have an answer for this?\n- I would like to see ablations for XTREME where you only did sentence-CTL or only did word-CTL. Other interesting ablations without parallel data would have also added more value to the paper. Ablations of the amount of data used would also be interesting.\n- Writing could have been clearer. I found a lot of grammar mistakes in the paper which need to be corrected.\n- Why did you not make an official submission to the XTREME leaderboard? That would have been a more fair evaluation of your system.\n\n\nMinor comments:\n- Word-level CTL: change \"are in two folds\" to \"are two fold\".\n- Table 2: It would be good to have an average column which will make it easier for the reader.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents two new representation learning tasks (losses) based on contrastive learning that---when combined with a language modeling loss---result in a better multilingual model. Experiments on machine translation and XTREME demonstrate the benefit of the proposed method compared to strong baselines.

I think this is an interesting paper that advances multilingual representation learning. The authors have incorporated many suggestions from the reviewers to improve the paper during the rebuttal period. I recommend to accept the paper, but also strongly suggest the authors to make an official submission to XTREME to validate their results.
Decision: Accept (Poster)