EMMA-X: An EM-like Multilingual Pre-training Algorithm for Cross-lingual Representation Learning
Authors: Ping Guo
Xiangpeng Wei
Yue Hu
Baosong Yang
Dayiheng Liu
Fei Huang
Jun Xie
Venue: NeurIPS 2023 poster (2023)
Keywords: cross-lingual pretraining;language-agnostic representation

Abstract: Expressing universal semantics common to all languages is helpful to understand the meanings of complex and culture-specific sentences. The research theme underlying this scenario focuses on learning universal representations across languages with the usage of massive parallel corpora. However, due to the sparsity and scarcity of parallel data, there is still a big challenge in learning authentic ``universals'' for any two languages. In this paper, we propose Emma-X: an EM-like Multilingual pre-training Algorithm, to learn Cross-lingual universals with the aid of excessive multilingual non-parallel data. Emma-X unifies the cross-lingual representation learning task and an extra semantic relation prediction task within an EM framework. Both the extra semantic classifier and the cross-lingual sentence encoder approximate the semantic relation of two sentences, and supervise each other until convergence. To evaluate Emma-X, we conduct experiments on xrete, a newly introduced benchmark containing 12 widely studied cross-lingual tasks that fully depend on sentence-level representations. Results reveal that Emma-X achieves state-of-the-art performance. Further geometric analysis of the built representation space with three requirements demonstrates the superiority of Emma-X over advanced models.

EMMA-X: An EM-like Multilingual Pre-training Algorithm for Cross-lingual Representation Learning
Ping Guo
Xiangpeng Weit.*
Yue Hu
Baosong Yang!
Dayiheng Liu" Fei Huang"
Jun Xie
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China School Cyber Security, University of Chinese Academy of Sciences, Beijing, China Machine Intelligence Technology Lab, Alibaba DAMO Academy, Hangzhou, China Kguoping, huyue)@iie att.:/@ieea..cooo ac. cn, pemyuel pemyue1@gmail.c com
Abstract
Expressing universal semantics common to all languages is helpful in understand- ing the meanings of complex and culture-specific sentences. The research theme underlying this scenario focuses on learning universal representations across lan- guages with the usage of massive parallel corpora. However, due the sparsity and scarcity of parallel data, there is still a big challenge in learning authentic "uni- versals" for any two languages. In this paper, we propose EMMA- X: an EM-like Multilingual pre-tram trainininng Algorithm, to learn (X)Cross-lingue universals with the aid of excessive multilingual non- non-parallel data. EMMA-X unifies the cross-lingual representation learning task and an extra semantic relation prediction task within EM framework. Both the extra semantic classifier and the cross-lingual sentence encoder approximate the semantic relation of two sentences, and supervise each other until convergence. To evaluate EMMA-X, we conduct experiments on XRETE, newly introduced benchmark containing 12 widely studied cross-lingual tasks that fully depend on sentence-level representations. Results reveal that EMMA-X achieves state-of-the-art performance. Further geometric analysis of the built repre- sentation space with three requirements demonstrates the superiority of EMMA-X over advanced models ?
Introduction
Research on how express universal semantics for natural languages (metaphorically as "alphabet of human thoughts" by Leibniz and von Leibniz [199]) has lasted long time. Usually, these universal meanings underlying all human natural languages are referred to as irreducible semantic cores [Wierzbicka] 1999]. These common cores across languages can serve as bridge, to help better understand the exact meanings of complex sentences different languages.
In the context of computational linguistics, various works [Huang et al. 2019 Conneau al. 2020 Chi et al. 2021 Wei et af., [022 Lee et al., 2022 Li et al. 2023] Chen et al.] 2023] have led to great improvements
learning cross-lingual universal representations with the usage of parallel corpora, and verify that multilingual universality contributes a major performance on cross- lingual understanding. However, due the sparsity and scarcity of parallel data, these advanced techniques face a big challenge in learning real universality among all languages. For instance, among the widely-available top 100 languages that theoretically can build 4950 language pairs, only about 200 language pairs have considerable parallel data Aharoni et al. 2019 Bapna et al.
Conresponding Author.
"Codes and datasets of the XRETE benchmark:
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Recently, Large Language Models (LLMs) (e.g.. PaLM [howwdhery et al., 2022]. OPT (Zhang al. 2022b], BLOOMZ (Workshop al. 2023]. ChatGPT, etc.) have reached a milestone in the held of Natural Language Processing, for their promising capability at understanding and following complex natural language instructions different languages. By modeling wide variety of sentence samples in discrete sentence space, LLMS can capture some universal linguistic phenomena to gain cross-lingual transferability. This is consistent with our goal of building a universal basement that supports all languages. The difference lies in that we achieve it through learning universal continuous representations across different languages.
2022
Concretely, we propose EMMA-X tackle the above challenge from a continuous perspective. EMMA-X can learn cross-lingual universal sentence representations with excessive non-parallel multilingual data by unifying two highly dependent tasks in an EM (Moon] [996] framework: semantic relation classification and cross-lingual universal representation learning. For the former, we introduce a Gaussian Mixture Model (Everitt and Hand 1981] classifier (GMM classifier) to deal with the key challenge of forming positive sentence pairs for non non-paralle multilingual corpora, by annotating the semantic relationship of sentence pairs in any two arbitrary languages on the fly. For the latter, we employ cross-lingual encoder to learn universal sentence representations via contrastive learning. where positive pairs are chosen by GMM classifier. Further, we construct training signals according to the output of the gross-lingual encoder, to inversely supervise GMM classifier. From the perspective of EM algorithm, in E-step, both modules try to approximate the semantic relationship given a sentence pair sampled from two arbitrary languages. One module is supervised by the approximation of the other to build its own expectation. In M-step, two modules update their parameters by maximizing expectations, respectively. We give theoretical justification about how these two tasks can be interpreted from an EM perspective (Section 4
To incentivize the research of universal sentence representation learning, we form a Cross-lingual RlEpresentation Transfer Evaluation (XRETE) benchmark, which includes 12 cross-lingual tasks cov- ering more than 50 languages. XRETE fully depends on sentence-level representations. Experimental results demonstrate that EMMA-X significantly outperforms pre-trained language models (Conneau ctt al. 2020 Chi et al.) 2021] by 32% most on XRETE. We also perform an evaluation of ChatGPT on XRETE to explore its multilingual performance. Detailed analysis also shows that EMMA-X can mitigate the representation discrepancy between head and massive long-tail languages. We further conduct geometric analysis directly on representation space from three perspectives: Invariance [Abend and Rappoport 2017], Canonical Form [Teller 2000) and Isotropy [Mu and Viswanath 2018], which provides a further understanding of the cross- frogs-lingual transferability of these models.
2 Preliminaries
Cross-lingual representation learning aims at mapping sentences from different languages into a unified continuous space, where synonyms across different languages are pulled closer. Given a sentence x, the representation is formulated as
(x) [[(((( (x))],
where M(: O.)) denotes the encoder network with set of trainable parameters O. which is typically implemented as a transformer encoder architecture [Vaswani et al. 2017 Devlin et al., 2019 Lee et al.. 2022 Feng al. 2022 f()) is L-2 normalization and g(-) is the aggregate function. We take the final hidden states of [CLS] token as the aggregate sentence representation.
To learn reasonable representations that can express universal semantics across different languages, various well-designed techniques have been applied to (x)) A predominant one is to build contrastive learning (CTL.) [Saunshi et al. [019] objective with parallel corpora. The basic idea to maximize the similarity between representations (i.e., ((xx and y(x)) of two cemantically-equivalenn sentences (x, y). while keep randomly sampled irrelevant ones 7(0") away. Formally, assume B to be a batch of multilingual parallel bitexts, the contrastive loss under InfoNCE [Oord et al., 2018 formulation is
LCTL= -log (0()(0),)0)) + Ey'eB ,((1)))
(2)
where s(-) is implemented as the cosine similarity s(y((), = | typically called positive and negative samples.
y and y' are
2

---

,00
7/4
yira)
Pai
Con (0m)
+
Psi Pa
M-Step
((ra)
8 - 0 x V VagCrax 0.
du (000)))))
1420, am
000
(rs)
y(
000
000
00o
0
E-Step
GMM Classifier
Cross-Lingual Encoder
A
f
A
y(0)
(n))
(x) (bs)
y(x) (rrs)
Y2
Ya
Figure 1: Illustration of EMMA-X, involving two modules (GMM classifier and Cross-lingual Encoder) that are reciprocated from each other and are updated alternatively. means the current instance, {y1, Y2. Ya.... ...) are samples in various languages for comparison. -y(x) is the continuous representation given discrete sentence x. Cy and cg formulate the semantic ranks approximated according to Eq. and Eq. to supervise the GMM classifier and cross loossll encoder, respectively.
3 Methodology
We propose EMMA-X that fully exploits massive monolingual data to learn cross-lingual universal representations. As illustrated in Figure I EMMA-X consists of modules: 1) A GMM classifier G(: θg) ) to approximate the semantic relation of non-parallel sentences. 2) A cross-lingual encoder M(: M)) convert multilingual sentences into universal representations. For optimization, EMMA- X unifies these two modules in an EM framework with dual supervision. In this section, we begin with definition of the semantic relation rank ($3.T). Then, we introduce model initialization ($3.2) and the proposed training paradigm ($ (3.3), followed by dual supervision strategy ($3.4). For clearer presentation, an Algorithm of EMMA-X is shown Algorithm
3.1 Rank of Semantic Relations
Mainstream methods model semantic relations with a strict binary separation: positives and negatives. However, the boundary between positives and negatives is blurry, and many samples cannot be clearly classified either positives or negatives. So it cannot maximize the potential of models perceive more subtle semantic changes. Also, binary separation will lead to far more negative samples than positive ones (imbalanced data). To more accurately capture the semantic relation between two sentences and alleviate imbalanced problem, we subdivide the relation into N semantic ranks, where semantic similarity of each rank decreases N increases, e.g., c denotes two sentences are paraphrases of each other, while N implies sentences are irrelevant. In practice, we set N to 4.
3.2 Model Initialization
In EMMA-X, the GMM classifier G-: Og) and cross-lingual encoder M(:00M) are initialized by training with massive parallel corpora, respectively.
Initialization of Cross-lingual Encoder: is initialized with XLM-R Conneau et al. 2020 and then continuously trained with InfoNCE (Oord et al., 2018 loss by Eq Following HICTL [Wei et al.) 202T] and INFOXLM [Chi et al. 2021], we treat the parallel sentence pairs as the query sentence x and its positive counterpart y, while treating the randomly sampled sentence negative one y'.
Initialization of GMM Classifier. reasonable solution warm up the GMM classifier to use the available cross-lingual parallel corpora training signals. Suppose x and y are parallel sentences,

---

and y' is an outlier. We set the semantic ranks for 1,(0),,000 and ,(0,))))))))) as = 1 and c = N,
respectively, according the definition described in $ $3.1 To obtain the fine-grained semantic ranks. we design a linear interpolation strategy similar to Wei et al. [2022] and mixup (Zhang et al. 2018).
which provides virtual training examples for each semantic rank. Formally,
(8)
A) -7 +
(3)
where À [0, 1] is sampled from uniform distribution. We compute r ((1-))-c-- N)] as the soft semantic rank for (y(x), ,(0)), where [-] means the least integer greater than equal to the input. The virtual training examples are grouped together with the real parallel corpora to optimize the GMM classifier:
[MLE log Pa(c 1/x, 04) - log Po(c NIx,y':00)) -
Σ log Pole = r (4)
The posterior probability (-(-) is formulated
No - 700011000000 (p...))
= ra(c=rix,y:0) Σ E TTy N5(7000 7(0) peg, (3))
(5)
where the Gaussian form Mf is assigned with a prior probability TT, mean Hr, and standard deviation or that are all parameterized by trainable variables, thus Og = [(TT, |r []]. See Appendix G for the specific calculation of Gaussian form N,.
3.3 The EM Iteration Framework
After initialization, EMMA-X further trains the GMM classifier (g: and cross-lingual encoder M(: θM) with only multilingual non-parallel data with an EM framework.
E-Step. For optimization, represent training batch of multilingual non-parallel sentences as X X2,....... x/ accompanied by a queue of random sentences as y = (y1.y2..... Yk) for instance comparison. Formally, the expectation for GMM classifier is:
LMLE(X V; 0g) -Ex,-x Bys-y llog Pş(c ci cxlxx,y;:
(6)
where EM E [1, N] represents an approximated semantic rank for the combination of anchor and another random sentence yı. based on the cosine similarity among representations (i.e., - (xx) and
((x))) produced by the cross-lingual encoder (i.e., M(:; ²x))) Please refer to $3.4 for details.
Correspondingly, the expectation for the cross-lingual encoder can be calculated with contrastive learning, where the positive samples are established by the maximum a posteriori approximation (argmax prediction) cg given by the GMM classifier. Specifically, we apply ranking InfoNCE (Hoff mann et al. 2022) as the training objective, which recursively takes parallel sentence pairs in each rank (e.g, Cg) as positives and ranks that are larger than og as negatives. Formally,
Ey--V e010000 (00)
Everyigan (66,00)),,0
(x.))X))0Om) -ExiNx log Exe ExeryegeuNN (01) + log
Exe EyeWeegea Vegeeeea E02.N (100)
Σ EoLndigEN- edp000, ((00))
t... + log ExInDeeetN-I. (slr foon
where cp E N1 represents a semantic rank approximated by the posteriori of GMM classifier 334.. For simplicity, we omit the temperature term in Eq. 7 and please see Appendix G for details.
M-Step. We use gradient descent algorithm to update the parameters of each module by optimizing its expectation. At each time step t, where 17 and nf are learning rates for each expectation, formally,
e + 0/ - x V (MLE(X
(8)
@t -- 0i - o x Co

---

Algorithm 1: EMMA-X X Training Algorithm
Input: multilingual parallel and non-parallel corpora
Output: M(:0m) and G (g)
Phase
/* Warm-up two modules with multilingual parallel corpora */
while not convergence do
Sample batch of multilingual bitexts B;
for (x.)) E B do
Sample y' B, yy. Build virtual training examples y with Eq. Compute LCTI with Eq. to update M and LMLE with Eq. to update θg: end end
Phase 2: /+ EллA-X training with EM framework using only non non-par con-parallel corpora */
while not convergence do
E-Step
Sample batch of multilingual anchors & Queue batch of random sentences y;
for yx) E X' x )' do
Approximate semantic rank with Eq. and eMm with Eq. 10
end
Compute LMI. Ob) according Eq. and (x (M) according Eq.
M-Step
Update 9M and Og according Eq. end
3.4 Dual Supervision
The approximated semantic ranks cg and eM are critical in EMMA-X training algorithm. To preserve their quality, propose dual supervision. predictions from one module fed the other to calculate the expectation. In this section, we explain in detail how we approximate the semantic ranks for GMM classifier and lingull encoder, respectively.
Approximate Semantic Rank with GMM classifier. The way to obtain semantic rank with semantic classifier is straightforward. The semantic rank corresponding to the highest probability among multiple Gaussian distributions chosen as the prediction, which is further used to supervise the cross-lingual encoder M(: @M), illustrated in Eq. Formally,
6g argmaax (a r
(9)
Approximate Semantic Rank with Cross-lingual Encoder. One common way calculate sentence relation is to measure the similarity between two real-valued representations. Assuming 8, (a scalar initialized as {)) reflect the general similarity score in semantic rank sentence pair yx), if its similarity score is close to sy, the sentence pair is likely to belong to rank r. Cross-lingual encoder M(: O.) determines the semantic relation for each pair according to
Given random
cM argmin [s() (5.) (0),),00))) ((x)) - srl
(10)
where refers absolute value. Symmetrically, cM is used supervise GMM classifier (Eq. 6)
During the training process, the general similarity score for cach semantic rank may vary. Thus, propose average strategy to adaptively adjust the value of to simulate this change.
Specifically, at time step f, s, is updated by cosine similarity all the sentence pairs, which are currently categorized into the rank r according the cross-lingual encoder in Eq.
- c + (1 c) (x)) ((x)))
if y& Voger.
(11)
Here E [0, 1] is momentum coefficient to make 8n evolve smoothly during training.
4 Theoretical Analysis
In this section, we provide theoretical justification for EMMA-X and demonstrate the mutual influence between two modules with rigorous interpretation from EM algorithm perspective. We show that
5

---

under dual supervision, minimizing the positive terms in Eq. S iintrinsicaly maximizes the objective of classical clustering algorithm. For simplicity, we assume that each semantic rank has the same number of sentence pairs and represents model parameters with G In EMMA-X, we model the semantic relation of sentence pair (t) through a joint distribution P(x;; with the semantic rank as a latent variable. Let ((c) be prior distribution over the possible values of semantic ranks. That is >, Q(c= = Q(c)= 2 O. The training goal is to maximize the following likelihood:
N
argmax e
Σ
Σ
log yex, = argmax Σ
log xeXYyac
Σ
Σ
@
(12)
N log
2 argmax E
ΣEΣ Y
Q(e=
Q(c=r)
E-Step. To make the inequality hold with equality, we have:
y4, c= Q(c=r)= د yk, Pic= p(c=r|x, 9),
(13)
which is the posterior probability and is approximated by the prediction from GMM classifier. Since each sentence pair (x) belongs to only one semantic rank, we approximate Q(c = r) 11(c r), which is a one-hot distribution.
M-Step. We try to maximize the likelihood in Eq. 12 under the semantic rank cg
N Σ r=1
argmax ΣE
Q(r log
P(x.yyE. ((r)
argmax 6
ΣE
Σ
log
XEE YEE
X.EX r=
(14)
N
argmax n(n 1)
The above derivation uses the assumption that P(x;, yelc and o, being the mean value and standard deviation of the Euclidean distance between sentence pairs in semantic rank r. Detailed proof of Eq. 14is is in Appendix
O) ~ Nr (x: yuljr, ā.), with jr,
Next, we prove that minimizing the positive terms in expectation LCTL (X, (,M) actually equal to maximizing a lower bound of Eq. 14 As we apply dual supervision, data in the contrastive label space also follows the distribution N, (x: ykslrr, Tr). Hence, under mild assumptions, we can get:
N-1
(X n
Σ
n(n 1) Σ 2
Σ
Σ log ye
(15)
YVEV
where LOTI. (.) means the positive terms. In the derivation, we the intrinsic property of semantic ranks (ū1 < U22 < ... < (N)). Detailed proof is in Appendix G Eq, 15 demonstrates that with dual supervision, minimizing the contrastive loss can partially maximize the likelihood in Eq. 12
5 Experiments
To thoroughly evaluate the performance of EMMA-X. X, we conduct experiments on XRETE benchmark to verify the transfer ability of EMMA-X on various cross- lingull downstream tasks with strong baselines (pre-trained models: XLM-R R [Conneau et al., 2020]. INFOXLM (Chi et al. [202T). HICTL (Wei et al. 2021], sentence models: LaBSE [Feng et al. 2022]. S-BERT [Reimers and Gurevych 2020)) and ChatGPT in Section 5.2 See Appendices C and D for details. We further conduct
geometric analysis Section 5.3 to better interpret the cross- lingull transferability in EMMA-X.
5.1 Setup
Corpus & Model. We collect parallel corpora from CCAligned (El-Kishky et al. 2020 CCMatrix
[Schwenk et al. (02T], WMT [Akhbardeh al.] (02T). and MultiUN Ziemski et al. 2016), involving 94 languages with 3.2 billion sentence pairs. In addition, we add CC-100 [Conneau et al. 2020

---

Model
XSLI
Inference
ANIJ
MuliiSSS
Simibrity
QF
LAReQA
Mossli-X
Nutriecal
BUDC
Tatecha
XCDRA
MolIFURLES
Clasiiiction
MullAC
PAWS-X
Menicc
Acc. (1)
Acc. (1)
Spearman
Peasina (1)
mA 20
mAPQ
C1
Auc. (D)
Acc. (1)
Acc. (D)
MAE
Acc. (7)
g
55.8
6
38.6
56.3
90.0
56.1
67.4
XLM
77.8
56.8*
50.9*
XLM-E
40.12"
61.5*
40 7
66.0
80.2"
65.6
58.0"
HICTL"
77.8'
92.5"
Olacr oxplooossntionn towurinee moal (mosiets nutted Engiltsh rreining data on date munuiittd target langangee
XiMag!
92.8
65.9
63.2
40.3
67.9
71.2
90.1
INOXLM
56.2
49.11
K1.2
64.1
64.9
57.1
774
74.0
67.7
16.2
950
HierL"
35.1
49.02
$1.5
64.9
45.1
54.8
77.6
65.8
74.8
68.3
38.2
92.8
RMMA-X
88.1
50.21
$7.3
07.2
58.6
59.6
87.1
82.5
78.2
71.4
32.7
98.2
Table 1: Results on the XRETE benchmark. denotes the results from previous literature, HHu et al.
2020] Conneau et al. (2020) Wee et al. [2021) "Ruder et al. (2021). ddeootes the results from the original paper. & denotes results from re-trained models with the same model size and training corpora as EMMA-X. denotes the zero-shot performance. Refer to Appendix F for greater details on each task and language.
Model
MulaSTS
Similarity
QE
LAReQA
Retriesal MewiliX
BUCC
Talocbu
Model
[lead
FLORES-200
Long sail
Ilead
Tatocha
Long-tail
Spearmes (7)
Pesewor (1)
mAPP:20001 (7)
2APP0-20 (1)
(?)
(T)
Metrics
Acc. (?)
(?)
(1)
Asc. (')
S BERT
$4.0
39.3
31.8
14.4
88.5
83.6
S-BERT
87.5
$1.6
91.4
57,8
LaBsi
74.4
31.6
12.8
11.2
93.2
83.6
LaBSE
99.9
82.5
95.7
77.9
ISPOXLM
55.2
49.4
16.9
23.9
77.4
66.2
INPOXIMM
83.4
53.8
87.8
56.1
XLMER
25.0
10.4
16.1
11.7
67.9
59.1
XLM-R R
68.2
45.5
06.3
55.7
EMMA-N
6299
54.7
19.4
29.4
87.1
K2.5
EMMA-X
94.5
84.2
91.9
78.1
Table 2: Zero-shot Results on Similarity and Retrieved tasks. Results of LaBSE use Customized Vocab setting. Re- sults of S-BERT are from XLM-r+-SBEL SaEERTTparpphrasss.. The bold font denotes the best results.
Table 3: Retrieval results on FLORES-
200 and Tatoeba in xx - En direction. The bold font denotes the best results.
as the large-scale monolingual corpus with about 800 billion sentences that covers 94 languages. The cross-lingual encoder starts from the well-trained XLM-R large model [Conneau et al. 2020]. The GMM classifier is implemented prior ı E R1, mean μ E R 1024 and a standard deviation a E R.1044, all are trainable variables. We set the total semantic ranks as N 4. The statistics of all data and dyper-parameee are shown in Appendix
mixture of Gaussian forms, each of which consists of
5.2 XRETE Evaluation
XRETE includes 12 cross-lingual tasks divided into different categories. We report the "translate- train" performance in Table T on most tasks but zero-shot performance on BUCC and Tatoeba following Ruder et al. 2021] Table 2 presents zero-shot comparisons with sentence models.
Comparisons with Pre-trained Models. In Table I EMMA-X consistently outperforms all baseline models (XLM-R [Conneau et al. 2020]. HICTL Wei al., 202T] and INFOXLM [Chi et al. 88.1%
2)21)) with 7.97% improvements on average. Specifically. EMMA-X achieves 88. 1% accuracy on XNLI [Conneau et al., [0188) and 50.21% accuracy on ANLI [Ebrahimi al.) 2022] with up to 6.4% improvements than baselines. On the MultiSTS [Reimers and Gurevych] 2020 task, EMMA-X
achieves an 87.3 correlation score, outperforming several strong baselines by 5. 21.4, and even achieves comparable performance in the cross-lingual and the monolingual settings (see Appendix F for language-specificc results). Furthermore, EMMA-X obtains 67.2 Pearson score on QE (Specia ct al. 2021] task, which is comparable to the winner the leaderboard") without any specific finetuning techniques. As for sentence retrieval, EMMA-X consistently outperforms previous strong baselines among all 4 tasks [Ruder et al. [2021), and demonstrates 2.6%~~39.6 improvements over these baselines. Similar results can be found in sentence classification tasks. EMMA-X obtains an 81.3% top-1 accuracy averaged on XCOPA [Ponti et al., 2020), MultiEURLEX (Chalkidis et al. 2021] and PAWS-X (Yang et al.] [019b] tasks, outperforming XLM-R, INFOXLM and HICTL by 7.0%, 3.9% and 3.5% improvements, respectively. On MultiARC [Keung et al.) 2020 task, EMMA-X shows the lowest error rates among all models. The consistent improvements on all tasks reveal that
EMMA-X obtain better universal representations for different natural languages with various
html

---

topics and domains. We further conduct experiments with ChatGPT on XRETE tasks without 4 Retrieval tasks. We list the prompts for each task in Appendix [ ChatGPT's zero-shot performance is worse than fine-tuned pre-trained models and the performance gap is very large on most tasks.
Zero-shot comparisons with sentence models. Compared with XLM-R and INFOXLM,, which adopt the same amount of training data as EMMA-X, EMMA-X consistently outperforms XLM-R and INFOXLM by 73.2% and 25.1% on average, as shown in Table 2 The results further prove the effectiveness of our pre-training technique. Through the reciprocation between GMM classifier and cross- lingual encoder, EMMA-X - can generate reliable semantic rank for multilingual non-pann con-paralle corpora, which can provide more supervision signals than previous baselines. EMMA-X even achieves comparable results with strong supervised methods: LaBSE [Feng et al.) al.]|2222. [022] and S-BERT [Reimers) and Gurevych, 2020], which both trained on supervised data. LaBSE is trained on fnne-filtered bilingual corpus with 6B translation pairs (2 times larger than EMMA-X), while S-BERT is distilled from a S-BERT model fine-tuned on English NLI, STS datasets, and 50M English paraphrase pairs. Compared with these two methods, EMMA-X can achieve the best results on QE and Mewsli-X by outperforming S-BERT and LaBSE by 71.7% and 1 117.8% averaged. EMMA-X performs worse than these baselines on MultiSTS and BUCC, for these two tasks only contain rich-resource languages, which already have great deal of parallel data.
Model
Invariance
Canonical Form
Hend Langs.
Isotropy
Invariance
Long-tail Langs. Canonical Form
Isotropy
Invariance
All Langs. Canonical Form
Isatropy
Metrics
KL-D ())
CH-I(t)
PR (R)
KL-D (})
CH-(())
(R)
KL)
CH-I())
PR (1)
XLM-R (cls)
0,7356
30.19
0.3681
2,0042
7,96
0.3686
16501
20.80
0.3683
INPOXIM (els)
0.4991
38.82
0.4478
1,555
13.02
0.4406
1.4777
31.51
04466
S-BERT (mean)
0.1115
108.22
0,45199
1,3122
44,32
0.4414
0,9782
102.36
0.4467
EMMA-X (cls)
0,3603
43.52
0,3963
46.53
0,5732
1.19H4
0.5918
Table 4: Comparisons with existing methods on FLORES dataset for geometric analysis. "cls" and "mean" " represent different pooling strategies to obtain sentence representations.
Performance on Loig-tail Languages. One goal of EMMA-X is to learn universal sentence representations accommodated for more languages. To better prove this, we report the retrieval accuracy on FLORES-200 200 (Costa-jussà et al., [022] and Tatoeba [Artetxe and Schwenk] 2019). We reformulate FLORES-20K which contains manual translations in 204 languages (totaling 3001 sentences) to perform retrieval tasks in the same way as Tatoeba and report the performance in terms of language data scale Table Details about the separation of languages and FLORES-200 are shown in Appendices A and B. On head languages, EMMA-X performs worse than LaBSE by about 4.6% but outperforms S-BERT by 3.5%. On the long-tail languages, EMMA-X can surpass S-BERT by 4.3% averaged on two tasks. EMMA-X can even exceed the strongest LaBSE by 2.1% on FLORES. One reason for the superior results on long-tai languages is that for those long-tail languages that have only bi-lingual parallel data with rich-resource languages (often English), EMMA-X can provide multi-lingual semantic relation signals for them with arbitrary languages through dual supervision.
5.3 Geometric Analysis
To interpret the advantages of EMMA-XX X. we evaluate the geometric characteristics of it on FLORES- 200 dataset [Costa-jussà et al. 2022) without any fine-tuning. The criteria of three requirements are Invariance, measured with KL Ki-divergence (KL-D) [Kullback and Leibler
1951), Canonical Form, measured with Calinski-Harabans Index (CH-I) [Calinski and Harabasz 1974 and Isotropy, measured with principal ratio (PR) Mu and Viswanath 2018 Details of them are shown in Appendix B We report the numerical results in Table and visualize each characteristic in Figure 2
Invariance & Canonical Form aim to measure how languages are aligned in the representation space. If the sentence representations are universal, then sentences in different languages should follow& a similar distribution, which is measured by invariance in KL-divergence. Similarly, canonical form measures how well semantic equivalent sentences are grouped into one cluster, with clustering metric (CH-I). In Table 4 S-BERT outperforms other baselines in "Invariance" and "Canonical Form" on head languages. However, EMMA-X shows better performance on long-tail languages in these two metrics, which is consistent with Table 3. Figure presents similar results. Among the 20 languages we randomly sampled from FLORES. EMMA-X can align 17 languages as shown in Figure 2 with "xh, co, ur" as outliers. In Figure 2c
2i and 2h different colors represent different languages. So
8

---

(a) XLM-R (Invariance)
INFOXLM (Invariance)
(c) S-BERT (Invariance)
(d) EMMA-X X (Invariance)
(e) XLM-R R (Canonical)
(f) INFOXLM (Canonical)
(g) S-BERT (Canonical)
(h) EMMA-X X (Canonical)
(i) XLM-R (Isotropy)
(i) INFOXIMM (Isotropy)
(k) S-BERT (Isotropy)
(l) EMMA-X (Isotropy)
Figure 2: Visualization of representations from EMMA-X, XLM-R, INFOXLM and S-BERT. We use t-SNE van der Maaten and Hinton 2008 to visualize each geometric metrics.
a cluster with only one color means this language is isolated from other languages and not aligned well in representation space. Figure 2h shows that EMMA-X performs well in most languages.
Isotropy measures how expressive representation space is since high-dimensional representation space can easily deteriorate into low-dimensional manifold. From Table EMMA-X achieves the best results in isotropy. The Isotropy of S-BERT (k) different from other methods. We conjecture the reason is that S-BERT removes MLM tasks during fine-tuning, token embeddings will only receive sentence-level supervision, resulting identical sentence representations for different languages. The abnormal results observed on the high KL-divergence, depicted in Table 4 and Figure 2c], can be attributed to the representation space for S-BERT deteriorating into a oww-dimnnsionl manifold (low isotropy score in Table and Figure 2K). and different languages not distributed uniformly across the whole representation space, which limits the expressive ability.
ModelU ModellLanguage
af
bg
bn
de
el
fa
Fr
he
bi
bu
id
it
Phase
87.8
84.3
85.2
84.9
88.7
+ fixed GMM
86.2
84.3
89.6
79.4
92.6
87.5
93.9
92.6
$4.5
$7.9
92.1
91.4
84.4
92.6
87.1
90.9
87.1
82.4
EMMA-X
ModelLLngguage
jv
ka
kk
ko
ml
mn
ml
SW
te
th
11
tr
ur
vi
xh
+ fixed GMM
37.9
91.8
67.1
87.6
91.8
85.3
93.1
88.9
90.8
75.4
72.4
82.3
86.7
68.1
91.6
89.3
94.9
88.8
Table 5: Ablation Study about EMMA-X on 36 languages in Tatoeba. We mark the 14 long-tail languages with color red. We highlight the best results with bold font and the second-best results with underlines.
5.4 Ablation Analysis
The primary goal of EMMA-X is to acquire universal semantic representations for multitude of languages. For the limited number of parallel data, EMMA-X strives to leverage non-parallel data to extend language coverage for universal semantic representations and to enhance representation performance those languages. To provide additional evidence supporting our claim, we propose an ablation experiment in Table The results in Table 5 demonstrate that EMMA-X X significantly

---

Setting
Acc. (1)
XNL.
Aec.
AND
VatiiSi Speaman (17
Peurson
QF
mAP420
BARIQA
2AP0220
MonaliX
BUOC
F1
Aoc. (7)
Tatedha
Acc.
XCKIPA
MUMPERLEE FX
Acc. (7)
MalBARC
MAE (L)
PAWS-X
Acc. (*)
(FUN, N =
56.6
46.01
570
16.1
46.6
54.1
81.1
76-7
76.2
08.5
$4.9
14.8
(GMM,
55.2
49.53
M47
67.3
4788
56.6
80.3
77.0
70.1
386
M0.9
(GMM. V7 47
S8.1
50.21
873
672
5066
39.6
$7.1
80.5
78.2
714
32.7
94.2
(GMM,
67.
30.88
6.1
49.2
82.8
7888
71.3
32.5
98.1
Table 6: Effect of different settings on the XRETE benchmark. means results with default settings (i.e., GMM classifier and N 4). "FCN" means the semantic classifier is developed as fully connected networks.
improves the retrieval accuracy by 8.1 when compared to solely using parallel data. Moreover, on 16 long-tail languages, the retrieval accuracy further increases from 76.2 to 90.7, representing a substantial improvement of 19.0%. These findings support our claim that EMMA-X can indeed learn effective universal semantic representations from an abundance of non-parallel data.
5.5 Semantic Rank and GMM Analysis
We assessed EMMA-X's performance on XRETE benchmark with various settings in Table The evaluation focuses on two aspects: the classifier type and the value of semantic similarity rank.
Type of the classifier. Without direct supervision, we propose each semantic rank follow a Gaussian distribution. Comparing "GMM" with "FCN" in Table 6 the performance drops from 70.7 to 68.1. This decline is attributed to the handling of outliers: the "GMM" handles them better due to its soft assignment approach, giving outliers low probabilities and thereby minimizing their influence.
Value of Semantic Similarity Rank. Different semantic similarity ranks were evaluated Table 6 The peak performance at N = 4 implies that refined similarity rank ensures balanced distribution of semantic ranks, which helps alleviate data imbalance. However, increasing the number of semantic ranks complicates the learning process for both "GMM" and the cross-lingual encoder. N 4 strikes a balance between data distribution and learning complexity.
Related Work
Cross-lingual Representation Pre-training In recent years, various studies (Devlin et al. 2019
Conneau et al. 2020) have shifted the monolingual pre-training procedure to multilingual scenarios. Most of them often rely heavily on parallel data to learn cross-lingual sentence representations, with several improved techniques, such as language modeling [Conneau and Lample 2019 and contrastive learning [Chi et al., 2021 Hu et al. 2021 Wei et al.) 2021]. Recently, some endeavors incorporate monolingual data into parallel corpora, by translating monolingual corpora into pseudo parallel corpora Kvapilíkováá et al. 2020 Ouyang et al. 2021]. or computing the semantic similarity
in monolingual corpora off-the-shel and using supervision signals [Goswami al. 202T
Contrastive Learning has become a popular paradigm in NLP. Besides constructing the positives and negatives through parallel corpora (Zhang et al., [222]] other labeled data [Gunel et al.) 2021
Ni et al. 2022]. researchers also adopt sulf-supervised methods, which build positives and negatives
by corrupting sentences [Gao al., 2021 Chuang et al. 2022] or data augmentation methods (Zhang et al. 2022a Wu et af.] 2002]] Another line of research improves the quality of negatives by preserving memory queue Yang et al. 202T Wang et al. 2021 generating high-quality negatives (Zhang et al. 2022c Wei et al. 2022
7 Conclusion
In this paper, we study the problem of learning cross-lingual universal representations. Major contributions are fourfold: 1) We propose a novel paradigm EMMA-X, which can make full use of massive monolingual data learn universities for any two languages. 2) We summarize three requirements for the universal representation space among all languages and verify the superiority of EMMA-X towards strong baselines. 3) To incentivize the research of cross-lingual universal representation learning, we form novel benchmark (XRETE) with 12 cross-lingual tasks fully depending on sentence-level representations. 4) Experiments on XRETF demonstrate that EMMA-X achieved state-of-the results over strong baselines.
10

---

Acknowledgements
We thank the anonymous reviewers for their constructive comments. This work is supported by the National Natural Science Foundation of China (Grant No. No.U21B2009), This research is also supported by the Strategic Priority Research Program of the Chinese Academy of Science, Grant No.XDDC02030400.
References
Omri Abend and Ari Rappoport. The state of the art in semantic representation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 77-89, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 11118653/11/111-110
Roce Aharoni, Melvin Johnson, and Orhan Firat, Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 3874 -384, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: N..18653/1//N11-111 URL https:
org. N19 1388
Farhad Akhbardeh, Arkady Arkhangorodsky, Magdalena Biesialska, Ondiej Bojar, Rajen Chatterjee, Vishrav Chaudhary, Marta R. Costa-jussa, Cristina España-Bonet, Angela Fan. Christian Feder- mann, Markus Freitag. Yvette Graham, Roman Grundkiewiez, Barry Haddow, Leonie Harter, Kenneth Heafilld Christopher Homan, Matthias Huck, Kwabena Amponsah-KKaayire,,, Jungo Ka- sai, Daniel Khashabi, Kevin Knight, Tom Kocmi, Philipp Koehn, Nicholas Lourie, Christof Monz, Makoto Morishita, Masaaki Nagata, Ajay Nagesh, Toshiaki Nakazawa, Matteo Negri, Santanu Pal, Allahsera Auguste Tapo, Marco Turchi, Valentin Vydrin, and Marcos Zampieri. Findings of the 2021 conference on machine translation (WMT21). Proceedings of the Sixth Conference Machine Translation, pages 1-88, Online, November 2021. Association for Computational Linguistics. URL
org/ 2021 wmt- 1.1
Mikel Artetxe and Holger Schwenk. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7:597-610, 2019. doi: 10. a 00288. Uttppptts://acaaatttoo
org 2t///Q9911000
Mikel Artetxe, Sehastian Ruder, and Dani Yogatama. On the cross-lingual transferability of monolin- gual representations. In Proceedings of the 58th Annual Meeting of the Association for Computa- tional Linguistics, pages 4623-4637, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v 122333/112222cclllc acl-man..411
URL httpp:///aclanthologyy org/2020. 2020. acl-main
421
Ankur Bapna, Isaac Caswell, Julia Kreutzer, Orhan Firat, Daan van Esch, Aditya Siddhant, Mengmeng Niu, Pallavi Nikhil Baljekar, Xavier Garcia, Wolfgang Macherey, Theresa Breiner, Vera Saldinger Axelrod, Jason Riesa, Yuan Cao, Mía Chen, Klaus Macherey, Maxim Krikun, Pidong Wang, Alexander Gutkin, Apu Shah, Yanping Huang, Zhifeng Chen, Yonghui Wu, and Macduff Richard Hughes. Building machine translation systems for the next thousand languages. Technical report, Google Research, 2022.
Jan A. Botha, Zifei Shan, and Daniel Gillick. Entity Linking in 100 Languages. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 7833-7845, Online, November 2020. Association for Computational Linguistics. doi: 18653/ 11/2000.emnlp-mannn
/tL/
2020. emnlp-main. 630
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10.18653//1/D15--10007 URL https
7/www.aclwed org/ wtt.:///aooooogg///----00000
T. Caliński and J Harabasz. A dendrite method for cluster analysis. Communications in Statistics, 3(1)11227, 1974, doi: 10. 10800003600927708888777 URL /ttp:htttt/////t htpp:://www.... tandf doi/abs/ 1080 0ttp://0000000000000000000000000
conndfoniinn onl online...... com/

---

Daniel Cer, Mona Diab, Eneko Agirre, lñigo Cazezo and Lucia Specia. SemEval- 2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-20117) pages 1-14, Vancouver, Canada, August 2017. Association for Computational Linguisties. doi: 10. 1/11666211/177-0001 URL atttps:/aclaanthooog. www./oi- 2001
Ilias Chalkidis, Manos Fergadiotis, and Ion Androutsopoulos. MuliEURLEX multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 6974-6999 Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics, doi: 10. 10.18653//1/22221.emm
URL httpss:/acllanthooog..oooo
temnlp-mati mapp.main..5 559
Nuo Chen, Linjun Shou, Ming Gong, Jian Pei, Bowen Cao, Jianhui Chang, Daxin Jiang, and Jia Li. Alleviating over-smoothing for unsupervised sentence representation. 2023.
Zewen Chi, Li Dong, Furu Wei, Nan Yang. Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling framework for cross- lingual language model pre- pre-training. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3578, Online, June 2021. Association for Computational Linguistics. doi: 10. 10653/3/v1/ 2021.naacl-main.2 280. URL httphhtt//aaclaatttoooooo htwphooooogyoogg/2222/ org/2021 naacl-main 280
Mao, Heyan Huang, and Ming Zhou. InfoXLM: An information-theoreet
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer. Vinodkumar Prabhakaran, Emily Reif, Nan Du Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Brica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways, 2022.
Douglas Eck, Jeff Dean,
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic. Shane Legg, and Dario Amodei. Deep rein- forcement learning from human preferences. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL wttp/hcpps:////roeeeeeeee neurips............e............ cc/
Paper.pdffffff pdf
Yung-Sung Chuang, Rumen Dangovski, Hongyin Luo, Yang Zhang, Shiyu Chang, Marin Soljacic, Shang- Wen Li, Scott Yih, Yoon Kim, and James Glass. DiffCSE: Difference- contrastive learning for sentence embeddings. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4207-4218, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10. 18653/v1/2022.naae wwwp:/0m22m.acmmmmmm.1111111 URL https: pttttp:://aalaantolloggo org/ 2022. naacl-mainn wwi.-mmin.33.. 311
Alexis Conneau and Guillaume Lample. Cross-lingual language model pretraining. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bus E. Fox, and R. Gar- nett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019, URL https: http:p//cccocc.o......a.a neurips cctpppppee//////iiiicicc Paper. pdf
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475-2485, Brussels, Belgium, Detober-Novemie 2018. Association for Computational Linguistics. doi: 10.18653/v 1/D18- 1269. URL https: / hwwp://www..// aclweb. org/ anthology/DD8 1269.
12

---

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440-8451, Online, July 2020. Association for Computational Linguistics. doi: 10118655/1/2022..cclmmmii ittps://aclanthooogyy org /tw//2200.aaa0 acl mollm.mann.7.. 747
URL
Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. No language left behind: Scaling human-centered machine translation. arXiv preprint Nxiv:2207..0667. 2022.
Jacob Devlin, Mfng-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume 1 (Long and Short Pupers), pages 4171-4186, Minneapolis, Minnesota, URLhttps
June 2019. Association for Computational Linguistics. doi: 1/119-143/ //aclanthology org/ 01t//N191100 1423
Abteen Ebrahimi, Manuel Mager, Arturo Oncevay, Vishrav Chaudhary, Luis Chiruzzo, Angela Fan, John Ortega, Ricardo Ramos, Annette Rios, Ivan Vladimir Meza Ruiz, Gustavo Giménez- Lugo, Elisabeth Mager, Graham Neubig, Alexis Palmer, Rolando Coto-Solano, Thang Vu, and Katharina Kann. AmericasNLI: Evaluating zero-shot natural language understanding of pretrained multilingual models in truly low-resource languages. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pages 6279 Dublin, Ireland. May 2022. Association for Computational Linguistics. doi: 10.18653/w1/2022 acl-long 435. URL httptttta/aclaatttoooooo
org 202g/2202..c0 acl 1eal-long.444 435
Ahmed Ei-Kishky, Vishrav Chaudhary, Francisco Guzmán, and Philipp Koehn. CCAligned: A massive collection of cross-lingual web-document pairs. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5960-5969, Online, November 2020. Association for Computational Linguistics. doi: 10.1 10553//1/222.emllpp..
480. URL https: / http///aaaaaattoooooooooo axg/2020/ emnlp-main, 480
BS Everitt and DJ Hand. Finite mixture distributions. Monographs on Applied Probability and Statistics, 1981.
Fangxiaoyu Feng, Yinfci Yang. Daniel Cer, Naveen Arivazhagan, and Wei Wang. Language-agnostiee BERT sentence embedding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pages 2022. Association for Computational Linguistics. doi: 1118653/11/2022.ac-bo0 https://aclantholle .rg wttgg/222.accco acl-long. 62
Dublin, Ireland, May URL
Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6910, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/112221..emmm main.552. URL https
//aclanthology. org/ 2021 emnlp main. 552
Andrew Gordon, Zornitsa Kozareva, and Melissa Roemmele. SemEval-2012 task Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 394-398, Montréal, Canada, 7-8 June 2012. Association for Computational Linguistics. BRLhtttp... http:htpt///acllaa htapp:s//aclaathoooog.o
org/ www//oggg//./. 02///1111100 1052
Koustava Goswami, Sourav Dutta, Haytham Assem, Theodorus Fransen, and John P. McCrae. Cross- lingual sentence embedding using multi-task learning. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9099-91 13, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 18653/ org/2021. 2021. emnlp-main. 716
1/202 1/220leemmllppeee manlp-mmin.771e 16.. URL https http://aaaaaattog...aaaao org/
13

---

Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Mare' Aurelio Ranzato, Francisco Guzmán, and Angela Fan. The Flores- 101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association doi: 10.1162/tac1 a 00474. URL https
for Computational Linguistics, 10:522-538, 2022.
//aclanthology. org/ wrg//022.......ooooo. 2022. wawtttallllae
Beliz Gunel, Jingfei Du, Alexis Conneau, and Veselin Stoyanov. Supervised contrastive learning for pre-trained language model fine-tuning. International Conference on Learning Representations, 2021. URL httpp::/ nwt. wowummidd-uu/UiOhia
Mandy Guo, Qinlan Shen, Yinfei Yang, Heming Ge, Daniel Cer, Gustavo Hernandez Abrego, Keith Stevens, Noah Constant, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. Effective parallel corpus mining using bilingual sentence embeddings. In Proceedings of the Third Con- ference on Machine Translation: Research Papers, pages 165-176, Brussels, Belgium, Oc- tober 2018. Association for Computational Linguistics. doi: 10. (0.18653/1//118-63333 URL https: haappss//ccla org/ 1ww.gggy.rrggo1/-111111 W18-
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9726-9735. IEEE Computer Society, 2020.
David T Hoffmann, N Behrmann, Gall, Thomas Brox, and M Noroozi. Ranking info noise contrastive estimation: Boosting contrastive learning via ranked positives. In AAAI Conference on Artificial Intelligence, 2022.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisa- tion. In Hal Daumé IIII and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 1 119 of Proceedings of Machine Learning Research, pages 441 1- 4421. PMLR, 13-18 Jul 2020. URL
mll wwwpppeesssiiiieee hatpss//v19hhhuee html
Junjie Hu, Melvin Johnson, Orhan Firat, Aditya Siddhant, and Graham Neubig. Explicit alignment objectives for multilingual bidirectional encoders. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages
Online, June 2021. Association for Computational Linguistics. doi: 10. 18653/v 1/202 mwwcclll..n..anlmemmmmeaama mww.l-main.n2111 m2...284. URL https::/accaattaiiet
org/ 2021 nwwcl-aaallomolclm main. 284
Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, and Ming Zhou. Unicoder: universal language encoder by pre-training with multiple cross-lingual tasks. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNE pages 2485 2494, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19 1252. URL /attps:/aaclantiot www.chhoooooyoo//11e.e 1252
Jeannie Irwin, Henk Harkema, Lee M Christensen, Titus Schleyer, Peter Haug, and Wendy W Chapman. Methodology to develop and evaluate a semantic representation for nip. In AMIA Annual Symposium Proceedings, volume 2009, page 271. American Medical Informatics Association, 2009.
Fabio Kepler, Jonay Trénous, Marcos Treviso, Miguel Vera, and André F. T. Martins. OpenKiwi: open source framework for quality estimation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 117-122, Florence, Italy, July 2019, Association for Computational Linguistics. doi: 10.18653/v1//19-3303 URL https://aclanthol// htw.c//aaooooogoooo/111-------
Phillip Keung, Yichao Lu, György Szarvas, and Noah A. Smith. The multilingual Amazon reviews corpus. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Online, November 2020. Association for Computational Processing (EMNLP), pages http://220..mmmmmma.m.m.amm URL Linguistics. doi: 10. 10.1186//1//2200
emnlp www..ainn... 369
14

---

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, San Diego. CA, 2015. URL URL http://arxiv org/ abs/ 1412.689
Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium, November 2018, Association for Computational Linguisties. doi: 10. 1865vv1l/v D18-2012. URL http:tttcccaaatttooossee http://claatttooooo.../////11111
S. Kullback and R. A. Leibler. On Information and Sufficiency. The Annals of Mathematicaa Statistics, 22(1):79 86, 1951. doi: 10.1214/amm//117779966 URL httpss/s:://do org/10. 10. 1214/ //oms///m///////////////// 1177729694
Ivana Kvapilíková, Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Ondřej Bojar. Unsupervised multilingual sentence embeddings for parallel corpus mining. In Proceedings the 58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 255-262, Online, July 2020. Association for Computational Linguisties. doi: 10.18653/v acl-srw.34. URL httpt:::s://aaattattta
org/2020 acl arl-srw... srw.34 34
Seonghyeon Lee, Dongha Lee, Seongbo Jang, and Hwanjo Yu. Toward interpretable semantic textual similarity via optimal transport-based contrastive sentence learning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5969-5979, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 121186533112002..o. acc-oong..lln-nn44 412. URL https::/
org/2022 2022. acl long. 412
Gottfried Wilhelm Leibniz and Gottfried Wilhelm Freiherr von Leibniz. Leibniz: New essays on human understanding. Cambridge University Press, 1996.
Ziheng Li, Shaohan Huang, Zihan Zhang, Zhi-Hong Deng, Qiang Lou, Haizhen Huang, Jian Jiao, Furu Wei, Weiwei Deng, and Qi Zhang. Dual-alignment pre-training for cross- cross-lingual sentence embedding, 2023.
Fuli Luo, Wei Wang, Jiahao Liu, Yijia Liu, Bin Bi, Songfang Huang, Fei Huang, and Luo Si. VECO: Variable and flcxible linsull pre-traing vee-traiiing for language understanding and generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I Ith International Joint Conference on Natural I anguage Processing ( (Volume I: Long Papers), pages 3980-3994, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v 12202.llooo/......oooooo acl-long.330. URL www.aaanttooooooooooooeoooooooee 0rg/2021..o. acl acl-long. 308
T.K. Moon. The expectation-maxiiizzae algorithm. IEEE Signal Processing Magazine, 13(6): 40, 1996. doi: 10.1109/79.54395.5
Jiaqi Mu and Pramod Viswanath. All-but-the-top: Simple and effective postprocessing for word representations. In International Conference on Learning Representations, 2018. URL https kkk
//openreview
Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith Hall, Daniel Cer, and Yinfei Yang. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1864-1874, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. http:///..2..innnaaallllll. URL httpps://acaaathhloogg///// wwwp://..o.o./.......... wwwp:///w...iiinnncannanc.....e. 146
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR, http//1070077777 2018 htt.:/tpp//aari.ooroaaaa////////
03748
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.
15

---

Xuan Ouyang, Shuohuan Wang, Chao Pang, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. ERNIE-M Enhanced multilingual representatinn by aligning cross- coss--ingual semantics with mono- lingual corpora. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 27-38, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. (/665//1/2021eemmlppoo org/2021. 2021. emnlp-main.3
URL
https://aclaathoology org/
Serge Sharoff Pierre Zweigenbaum and Reinhard Rapp. Overview of the third buce shared task: Spotting parallel sentences in comparable corpora. In Reinhard Rapp. Pierre Zweigenbaum. and Serge Sharoff, editors, Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Paris, France, may 2018, European Language Resources Association (ELRA). ISBN 979-10-95546-07-8
Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. XCOPA: A multilingual dataset for causal commonsense reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2362-2376. Online, November 2020. Association for Computational Linguisties. doi: 10. 10.16633/11/220
emnlp- maip. 185. URL btattps://aaclatthlll0// httphhoooogo...g//222/o2//e// org/2020.0m0 wmnnlmmaaaeteeeeeeeeee main. 185
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10. 10.18653/v1/D16- 1264. URL btttps://cllanthologs
org/D16-110
Nils Reimers and Iryna Gurevych. Making monolingual sentence embeddings multilingual us- ing knowledge distillation. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 4512-4525, Online, November 2020. Associ- ation for Computational Linguistics. doi: 10. 2665////2020.emmmmmmaaii //aclanthology. wwg.2020..o.0.o//oo/ 2020. emnlp-main. met.maain.355. 365
URL https
Uma Roy, Noah Constant, Rami AI-Rfou, Aditya Barua, Aaron Phillips, and Yinfei Yang. LAReQA: Language-agnostic answer retrieval from a multilingual pool. In Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing (EMNLP), pages 5919-5930, Online, November 2020. Association for Computational Linguistics. doi: 10. http:///220..m/ppp.e 477. URL httpp::/aacaanthologyy wrg//020...c....oo
2020. emnlp-main. 477
Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Dan Garrette, Graham Neubig, and Melvin Johnson. XTREME-R: Towards more challenging and nuanced multilingual evaluation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pages 00215-10245. Online and Punta Cana, Dominican Republic. November 2021. Association for Computational Linguistics. doi: 10.18653) v1/2021. v1/2021.emnlp-mn main.802............. URL
org/ 2021 emwlp-main..... 802
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Com- monsense reasoning about social interactions. In Proceedings of the 2019 Conference on Em- pirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-LJCNLL) pages 4463 4473, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: D0.18653/v1/D19..0 https::/aclanthologyy hwwphhhooogooooog//... Dt9- 1454
URL
Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoretical analysis of contrastive unsupervised representation learning. In International Confer- PMLR, 2019.
ence on Machine Learning, pages
Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, Armand Joulin, and Angela Fan. CCMatrix: Mining billions of high-qualie parallel sentences on the web. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1thh International Joint Conference on Natural Language Processing (Volume I: Long Pupers), pages 1490-65000 Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/202 lonl-
507. ht http://::////aaaaooollso//// org/ acl long..n.00 507
16

---

Lucia Specia, Frédéric Blain, Marina Fomicheva, Chrysoula Zerva, Zhenhao Li, Vishrav Chaudhary. and André F. T. Martins. Findings of the WMT 2021 shared task on quality estimation. In Proceedingss of the Sixth Conference on Machine Translation, pages 684-725, Online, November 2021. Association for Computational Linguisties. URI. htttp:://aalatttolllggooo wmt-1.71
Virginia Teller. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguisties, and Speech Recognition. Computational Linguistics, 26(4):638-6) 641, 12 2000. ISSN 0891-2017. doi: 10. (0.1166200091100075100057 URL htt.htttss//eee org/10 1162/009122100500000977
Nattapong Tiyajamom, Tomoyuki Kajiwara, Yuki Arase, and Makoto Onizuka. Language- agnostic representation from multilingual sentence encoders for cross- gross-lingual similarity esti- mation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7764-7774, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguisties. doi: 10. 116653/11/2021 emnlp-main.612. URL
ltttps://aclanthologg. org/oogg/ 2201/2211-a2 tomlp-mainn 612
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Ma- chine Learning Research, 9866)2257-22055 2008. URL http:://jmle org papprs/v9/
vaandemaatenO8a.cr html
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems 30, NIPS 2017 4-9 December 2017, Long Beach, CA. USA, pages 5998-6000 2017. URL http:: nips ccc/paper/7181-- atteetionnttttteee -iteeettionissslllll all--alluu---eee
Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2495-2506 June 2021.
Liang Wang, Wei Zhao, and Jingming Liu, Aligning cross-lingual sentence representations with dual momentum contrast. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 3807-3815, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. (0018653/v122221emmm// main-.mainn00 URL
org/2021. 2021. camnlpmmate mwt-maain 309
org/
Xiangpeng Wei, Rongxiang Weng, Yue Hu, Luxi Xing, Heng Yu, and Weihua Luo. On learning uni- versal representations across languages. In International Conference on Learning Representations, 2021. URL.iattps://ppereviie
Xiangpeng Wei, Heng Yu, Yue Hu, Rongxiang Weng, Weihua Luo, and Rong Jin. Learning to generalize to more: Continuous semantic augmentation for neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume
I: Long Papers), pages 7930-7944, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 12.1865//1/2022..cc--oo
URL. http:: /atpp::/acclatthoooog..rro
acl acl-1ong.544
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4003-4012, Marseille, France, May 2020, European Language Resources Association. ISBN 979-10-95546-3444 URL htttps://aciaathhologg// wwwpolloooyyyggg//000/ org/2020 1rec- 494
Anna Wierzbicka. Emotions across languages and cultures: Diversity and universals. Cambridge university press, 1999.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume 1 (Long Pupers), pages 1112-1122, New Orleans, Louisiana, June 2018. Association for Computational Linguisties. doi: 10. 10.18653/11/N18- 1101. URL. https: httpss:/aaaaaahholgg.o org/N18-//-- NN1//N18111 1101
17

---

BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana llié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, and et al. Bloom: 176b-parameter open-access multilingual language model, 2023.
Xing Wu, Chaochen Gao, Zijia Lin, Jizhong Han, Zhongyuan Wang, and Songlin Hu. Infocse: infornation-aggrrggaeed contrastive learning of sentence embeddings, 2022. URL https:// baxiv.org//ae 2210. 2210.06432
Nan Yang. Furu Wei, Binxing Jiao, Daxing Jiang, and Linjun Yang. xMoCo: Cross momentum contrastive learning for open-domain question answering. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing ( (Volume I: Long Papers), pages 6120-6129, Online, August 2021. Association for Computational Linguisties. doi: 11116633/112021..cllloooggg https://aclantholle .wg.g//oooo//ooo wogg/022/ acl-long.477
URL
Yinfei Yang, Gustavo Hernandez Abrego, Steve Yuan, Mandy Guo, Qinlan Shen, Daniel Cer, Yun- hsuan Sung, Brian Strope, and Ray Kurzweil. Improving multilingual sentence embedding using bi-directional dual encoder with additive margin softmax. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19. pages 5370-5378, International Joint Conferences on Artificial Intelligence Organization, 7 2019a. doi: 2.244966/jaaii22199774 URL htttps:/ddi..ccc org/ 10. 24963/ i jjcai. 2019/746
Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. PAWS-X: A cross-lingual adversar- ial dataset for paraphrase identification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP), pages 3687-3692, Hong Kong, China, Novem- ber 2019b. Association for Computational Linguistics. doi: 10. 10.118653/11/D11113338 URL attpss:/aclantthlooy) org/D19-10 1382
Dejiao Zhang. Shang- Wen Li, Wei Xiao, Henghui Zhu, Ramesh Nallapati, Andrew Arnold, and Bing Xiang. Pairwise supervised contrastive learning of sentence representations. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 5786-5798, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 1tt66/////2222..mmmmmmmiii emmlp www..min.5.... 467
URL http:: /attas://allantholoogy org/
Dejiao Zhang, Wei Xiao, Henghui Zhu, Xiaofei Ma, and Andrew Amold. Virtual augmenta- tion supported contrastive learning of sentence representations. In Findings of the Associa- tion for Computational Linguistics: ACL 2022, pages 864-876, Dublin, Ireland, May 2022a.
Association for Computational Linguisties. doi: htttps:/acclatthol.gg org 2022. Findings-acl.77
acl.70. URL
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018. URL https
//openreview.coo www.:eeettfeeee forttt/orrmmmt Doberumnid-rrDddp---
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li. Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022b.
Yanzhao Zhang, Richong Zhang, Samuel Mensah, Xudong Liu, and Yongyi Mao. Unsupervised sentence representation via contrastive learning with mixing negatives. Proceedings of the AAAI Conference on Artificial Intelligence, wwtpiidde..ppppp...aae 21428. URL hutp::: aaai org/index..eieeoooooooooogeeeeee
1738, Jun. 2022c. doi: 10. 1609/aaai. v3.i0..
Yuan Zhang, Jason Baldridge, and Luheng He. PAWS: Paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Humun Language Technologies, Volume I (Long and Short Papers), pages 1298-1308. Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v 1/N19-1131.
URL htttps://aclanthoollot org/ N19 1131
18

---

Michal Ziemski, Marcin Junezys- Dowmmut, and Bruno Pouliquen. The United Nations parallel corpus v1.0. In Proceedings of the Tenth International Conference on Language Resources and Evaluation 16), pages 3530-3534 Portorož, Slovenia, May 2016. European Language Resources Association (ELRA). URL,
Pierre Zweigenbaum, Serge Sharoff, and Reinhard Rapp. Overview of the second BUCC shared task: Spotting parallel sentences in comparable corpora. In Proceedings of the 10th Workshop on Building and Using Comparable Corpora, pages 60-67, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10.1 10.18663/1/WW1-255222. URL https: //www
aclueb. org/ /
19

---

A Training Corpora and Hyper-parameters
A.1 Training Corpora
As for monolingual data, we follow Conneau et al. 2020 to build a Common Crawl Corpus using
the CCNet [Wenzek et al. 2020] too which is widely used in the literature Huang et al. 2019).
Luo et al. 202T]. Chi et al. (02T]. Wei et al. 202T]. Further, we collect parallel corpora from
CCAligned El- Ki-Kks al. 2020 , CCMatrix Schwenk al. (202T). WMT Akhbardeh et al. [022]]
and MAultiUN/Zemmaa
et al. [2016], involving 94 languages with more than 4.8 billion sentence
pairs. We use the OpusFilter" tool to remove noisy bitexts, which results in 3.2 billion sentence pairs.
Table [shoos the statistics for both monolingual and parallel data. We apply subword tokenization
directly on raw text data using Sentence Piece Model Kudo and Richardson [0188] without any additional preprocessing. To better support our motivation that EMMA-X can cover more languages than previous cross-lingual sentence representations, we divide Tatoeba Artetxe and Schwenk (2019) into two subsets: "Head", containing languages usually covered in previous methods, and "Long-tail", with other languages. We treat the 36 languages contained in XTREME Ruder al. [2021] as head
languages, which are: "ar, he, vi, id, jv, tl, eu, ml, ta, te, af, nl, en, de, el, bn, hi, mr, ur, fa, fr, it, pt, es, bg, ru, ja, ka, ko, th, sw, zh, kk, tr, et, fi, hu, az, It, pl, uk, ro".. The remaining 76 languages in Tatoeba are treated as long-tail ones.
A.2 Hyper-parameters
The parameters of EMMA-X are first initialized with XLM-R, with 24 layers of Transformer (Vaswani
eet al.] 2017) encoder, 1024 hidden states, and 16 attention heads. We set the total semantic ranks as 4.
The GMM classifier is implemented as mixture of Gaussian forms, each of which consists of prior
E R1, mean μl E R 1024 and variance a E IR 1024 all are trainable variables. We optimize the GMM classifier with Adam =3, 3>=0.999) Kingma and Ba [2055] using a batch size of 1024 and learning rate of 3e-5. For cross-lingual encoder, we apply the same training setting as MoCo He al. 22200), with the momentum queue to be 256 and temperature as 0.04. We set the momentum coefficient to 0.999 and use the Adam optimizer with a cosine decay learning rate whose peak is 5c-4.
B FLORES-200 Dataset and Geometric Analysis
B.1 FLORES-200 dataset
FLORES-200 Goyal et al. [2022] Costa-jussà al. 2022 is a many-to-many multilingual bench- mark, which consists of 3001 sentences in 204 total languages. FLORES-20 200 sourced all sentences from English WikiMedia and translated these English sentences into 204 languages by human trans- lators. In particular, sentences in FLORES-201 200 have a much larger breadth of topics. for they are collected from three different sources: WikiNews WikiJunior and WikiVoyagd We summa- rize the basic statistics of all languages in FLORES- and Schwenk, 2 we treat English data "eng Latn" as retrieval labels and report the retrieval accuracy using the same scripts as Tatoeba in XTREME (Ruder et al., 2021). We set the 68 lan- guages: "bel_ Cyrl, bos Latn, hun Latn, epo Latn, khm Khmr, urd Arab, srp_Cyrl, Lat, slk Latn,
in Table R Similar to Tatoeba [Artetxe]
hye_Armn, gla_Latn, por Latn, Latn, _ mar Deva, hrv Latn, hin_ Deva, _eorr BengBeng, fin Latn, cym Latn, oci Latn, cat cat_ Latn, fao Latn, xho Latn, spa Latn, ron Latn, amh Ethi, ces Latn, swe Latn, nld Latn, tat Cyrl, kor Hang, glg Latn, fra Latn, eus Latn, ind Latn, dan Latn, tha Thai, deu Lain, tel Telu, afr Latn, pol Latn, est Latn, uig Arab, ukr Cyrl, uzn Latn, heb Hebr, kaz. Cyrl, nob Latn, rus_Cyrl, vie Latn, arb wob_Arab, zho Hans, tuk Latn, khk_ -yrl, jan_njjnaaaaa Jpan, ell Grek, isl_ Latn, tam Taml, sly Latn, tur Latn, mkd Cyrl, tgl Latn, gle Latn" as "Head" languages, and the remaining 135 languages (excluded English data) as "Long-tail" ones.
ita_ nno Latn,
httpa://en.vikiboott http:/boookk.orgg///k//////i/i/m http:://///aaaiiib.aa..ooo.//...
https://en.wwiiivoygge http::/www...ooggoooooo./aaaaaaa
20

---

(GB)
(GB)
19
2.0
2.8
37
06
by
17.
O
mg
5.5
49
1.3
Table The statisties of CC-100 and the collected parallel corpora used for training. We report the list of 94 languages and include the size of the monolingual data (in GiB) and the number of sentence pairs (in Millions, which denotes the number of sentence pairs between the specific language and English) in parallel corpora for each language. "_" means the number of sentence pairs is less than 0.1 million.
Number of Sentences
3001
Average Words per Sentence
21
Number of Articles
842
Average Number of Sentences per Article
3.5
Domain
Articles
Sentences
WikiNews
309
993
WikiJunior
284
1006
WikiVoyage
249
1002
Sub-Topic
Articles
Sentences
Crime
155
313
Disasters
27
65
Entertainment
28
68
Geographt
36
86
Health
27
67
Nature
17
45
Politics
171
341
Science
154
325
Sports
154
162
Travel
505
1529
Table Basic Statistics of FLORES--200
B.2 Three measurements in Geometric Analysis
Invariance Measurement implies whether the semantic distributions of all languages are simi-
lar [Abend and Rappoport 2017). We adopt a Gaussian form Ni(pu,o?) where μ Excr and 2001
approximate the semantic distribution of each language L, Further, compute the mean averaged KI.-divergence (KL-D for short) (Kullback and Leibler] 1951] among all language pairs the overall Invariance I. with L the total number of languages:
KLING (|Ar,)) + KL(N,,II,.
(16)
LX (L
11382
Canonical Form Measurement Previous works Teller 2000 Irwin et al. 2009 have demonstrated that a good multilingual space should distribute sentence representations based on their semantic similarities rather than language families. To measure this in quantity, we focus on Calinski-Harabasz Index (CH-I) [allinski and Harabasz 1974). which measures how similar object is its cluster compared to other clusters. We group all semantically equivalent sentences a cluster, which leads to 3001 clusters and each obsesses 204 sentences in 204 different languages. Assuming e₁ and e are the centroid of cluster l: and the whole dataset s, respectively. The CH-I Ca is defined as:
C₁ = 204 Σ lice |
Σ
cx
(17)
21

---

Task category
Task
Train
Dev
Text
Lang.
Metric
Domain
Inference
AmericasNLI
392,702
222-743
738-750
10
Accuracy
Misc.
XNLI
392,702
2,490
5,010
Accuracy
Semantic Similarity
Muhi-STS
150,152
10,000-1,500
250
Spearman
Misc.
WMT2IQETaskI
7,000
1,000
1,000
7 ((1))
Pearson
News
LAReQA
87,599
10,579
1,190
11
mA mAP@20 mAP@20
Wikipedia
Mewsli-X
116,093
10,252
428-1,482
(50)
News
Sentence Retrieval
BUCC
1,896-14,330
F1
Wiki/News
Talueba
(122)
Accuracy
XCOPA
33,410:400
100
500
Accuracy
Misc.
Classification
MulIEURLEX
55,000
5,000
5,000
23
Accuracy
Legal
MulrARC
200,000
5,000
5,000
MAE
Reviews
PAWS-X
49,401
2,000
2,000
Accuracy
Wiki/Quora
Table 9: Overview of XRETE tasks. For tasks that have training and dev sets in other languages, we only report the number of sentences in English sets. We report the number of test examples per language.
The higher the CH-I is, the better the semantically equivalent sentences are clustered.
Isotropy Measurement A high-dimensional embedding space often demonstrates poor isotropy, and deteriorates into a low-dimensional manifold that greatly limits the expressive ability of the embedding space. We adopt principal ratio (PR) [Mu and Viswanath 2018 to measure isotropy. Let e the sentence representation matrix, V be the set of the eigenvectors of e, the Isotropy I.a is
Igo
min
exp(pu
/e/ee//m /max DEV
exp((Te)
(18)
CEE
cEE
The closer I, is to 1, the more isotropic the representation space is.
C XRETE: Cross-lingual Representation Transfer Evaluation
XRETE consists of 12 tasks that fall into four different categories. In our trranslate-train-al setting, we individually fine-tune models with English training set and its translated training sets on each task. Then we report the performance of our fine-tuned model. We give an overview in Table and describe the task details as follows.
The Cross-lingual Natural Language Inference corpus Conneau et al. [018 tasks the systems with reading two sentences and determining whether one entails the other, contradicts it, or neither (neutral). A crowdsourcing- -based procedure is used for collecting English examples, which are later translated into ten target languages for evaluation. Training data stays consistent with the English training data of MultiNLI Williams et al.) 2018)). J. For evaluation, we concatenate two sentences input and apply a new classification head to distinguish sentence relationships. We perform "translate-traie train--laai-lll" evaluation, where the model is first tne-tuned on English training data and its translated data in other languages, then evaluated test sets.
XNLI
AmericasNLI (ANLI) The AmericasNLII Ebrahimi et al. 2022 is an extension of XNLI task to 10 Indigenous languages of the Americas. All of these languages are truly low-resource languages and serve as good testbed for zero-shot cross-lingual transferabilityy As Spanish is more relative to the target languages, the Spanish version of XNLI subset is translated for evaluation. For training, both English and Spanish versions of MultiNLI training data are provided. We evaluate on ANLI following the same settings as in XNLI.
MultiSTS The Multilingual Semantic Textual Similarity dataset Cer et al. 2017 Reimers and Gurevych (2020) aims to assign a semantic similarity score for a pair of sentences. The MultiTSS dataset contains 7 cross-lingual sentence pairs and 3 monolingual pairs. Stanford NLI Bowman et al. 20155 and English STS Cer et al. (017)] are provided as training sets. We report the results after first fine-tuning on English training set using a Siamese network structure [Reimers and Gurevych 2020
Then we compute the cosine similarity between the sentence pairs and compute Spearman s rank correlation between the predicted score and gold score following Reimers and Gurevych 2020
22

---

WMT21QETassk (QE) The WMT21 Quality Estimation Task 1 Sentence-level Direct Assessment Specia et al. [202T] aims at testing the translation quality and this task has been applied to test the sensitivity of language models to semantic similarity Tiyajamorn et al. [2021]. The training and evaluation sets are collected from Wikipedia by translating sentences using state-of-the-art translation models to 6 languages and annotated by professional translators. In WMT21,, 4 new language pairs with no training data are given to test zero-shot cross-lingual transferabilityy Our evaluation setting on QE is similar to that on MultiSTS, but we report Pearson's 's rank correlation [Kepler et al. 2019
2020 is QA retrieval task where models are required to retrieve all relevant answers in different languages over a large multilingual pool. The dataset is constructed on XQuAD Artetxe et al. (2020) and question is linked with answer sentences in different languages. The training set of SQuAD v vI.1 Rajpurkar et al. (016) is used to fine-tune the model adapt QA retrieval task. During the evaluation, sentence embeddings are also obtained by siamese network, and we retrieve the sentences with the highest cosine similarity as predictions.
LAReQA The Language-Agnostii Retrieval Question Answering/Roy et.
Mewsli-X Mewsli (Multilingual Entities in News, linked) requires linking an entity mention to
its entry in a language-agnostic knowledge base Bothh et al. (2020). Mewsli-X[Rudee Ruder et al. [202T) features 15k mentions
languages. For each mention, Mewsli-X offers entity descriptions
candidate pool containing 1M candidates across 50 languages. Fine-tuning is done on predefined
set of English-only mention-entity pairs from English Wikipedia hyperlinks. Our evaluation setting is identical to LAReQA.
BUCC The second and third shared task of the workshop on Building and Using Parallel Corpora Zweigenbaum et al. [2017]. Pierre Zweigenbaum and Rapp (2018) aims to examine the ability of models to detect parallel sentence pairs in pair of monolingual corpora. The dataset provides train and test splits in 5 languages. Following XTREME Hu et al. [2020]. we directly evaluate on BUCC without fine-tuning and retrieve sentences with the highest cosine similarity.
Tatoeba The goal of the Tatoeba dataset Artette and Schwenk [019 is to find the nearest neighbor for each sentence in the other language according to cosine similarity and compute the error rate. The dataset consists of up to 1,000 English-aligned sentence pairs covering 122 languages. Following XTREME Hu et al. 22200) we directly evaluate on Tatoeba without fine-tuning and retrieve sentences with the highest cosine similarity.
XCOPA In the Cross-lingual Choice of Plausible Alternatives dataset Ponti et al. 2020], each XCOPA instance corresponds to a premise and two alternatives. The task formulated
binary classification to predict the more plausible choice. The English COPA [Gordon et al.) (0012) training set and Social Sap et al. [2019] training data are used for fine-tuning, while the validation and test sets of English COPA are translated and re-annotated into 11 languages for evaluation.
MultEURLEX The MultiEURLEX dataset|Chalkinn Chalkidis et al. 2021 is legal topic classification task that comprises 65k European Union (EU) laws in 23 official EU languages. The dataset provides muuli-graaullr labels per document. The dataset is split into training. development, and test subsets chronologically, resulting in 55k training documents for languages, and 5k each for development and test subsets in all 23 languages.
The Multilingual Amazon Reviews Corpus Keuug et al. (020 is large-scale collection of Amazon reviews for multilingual text classification in 6 languages. Different languages are directly gathered from the marketplaces in different countries. The goal is predict the reviewer's rating on the 5-star scale using the test of the review as input. The data is clearly split into training (200,000 reviews), development (5,000 reviews), and test sets (5,000 reviews) for each language.
MultiARC
The Cross-lingual Paraphrase Adversaries from Word Scrambling Yang et al. (019b) dataset requires identifying whether two sentences are paraphrases. A subset of the evaluation pairs in English PAWS et (2019) are sunan-translated into 6 typologically distinct languages for evaluation, while the English PAWS training set is used for training.
PAWS-X
23

---

Model
ar
bg
de
el
fr
hi
SW
th
tr
vi
zh
Avg.
XLM-R
88.6
84.5
86.7
84.6
85.2
84.7
82.0
82.5
82.6
82.4
80.6
83.1
80.3
17.3
77.2
82.8
INFOXLM
90.4
83.9
85.8
83.5
83.3
82.7
84.2
HICTL
90.6
86.8
88.2
87.4
87.0
87.4
85.0
81.9
83.3
84.8
83.1
85.7
82.8
79.7
80.9
85.1
ChaGPT
70.4
61.0
64.5
64.8
62.8
65.7
66.3
51.5
63.4
55.7
53.0
61.6
47.9
61.6
62.6
60.9
EMMA-X
86.7
Table 10: XNLI results (accuracy) for each language.
Model
aym
bzd
cni
heh
nah
oto
quy
shp
tar
Avg.
XLM-R
49.01
50.61
41.72
58.34
42,46
54.63
35.57
59.29
51.62
41.54
48.48
INFOXLM
49.87
51.29
42.41
58.83
43.07
55.25
36.14
59.87
52.20
42.12
49.10
HICTL
49.65
51.22
42.36
58.82
43.09
55.13
36.04
59.61
52.17
42.08
49.02
ChatGPT
42.0
43.6
40.8
40.4
40.0
43.8
41.1
43.1
42.0
40.0
41.7
EMMA-X
51.19
52.50
43.62
59.88
44.31
55.44
39.16
60.14
52.84
43.10
50.21
Table 11: AmericasNLI (ANLI) results (top-1 accuracy) across different input languages.
Model
en-de
en-fr
en-es
en-fr
en-it
en-nl
Avg.
XLM-R
50.2
63.7
45.8
59.6
68.0
63.4
69.6
87.7
82.5
68.5
65.9
INFOXLM
81.7
80.3
79.9
79.1
80.6
83.4
81.2
86.7
87.2
81.7
82.2
HICTL
80.4
81.8
78.3
80.6
81.2
80.9
79.3
88.4
86.1
79.6
81.6
EMMA-X
86.6
85.0
87.1
84.4
85.2
89.4
88.3
90.9
92.0
84.5
87.3
Table 12: MultiSTS results (Spearman) across different input languages.
D Baseline Methods
To fairly evaluate the performance of EMMA-X, we choose XLM-R Conneau and Lample le and its several derivatives as our baselines, which contain: (1) XLM-R, R,, which applies multi multilingual MLM tasks as pre-training objectives on CCNet-100 corpus; (2) HICTL Wei al. 2021], which continues training on XLM-R using hierarchical contrastive learning: and (3) INFOXLM, which is initialized with XLM-R and trains with cross-lingual contrast, multilingual MLM and TLM. Also, we compare EMMA-X to strong sentence models: (1) S-BERT [Reimers and Gurevych 2020). which adopts multilingual knowledge distillation to extend monolingual sentence representations to multilingual. We use the strongest baseline, XLM-R <- SBERT parrpppaaappraee, proposed in the original paper as a baseline. (2) LaBSE [Feng et al. 2022]. which systematically combines several best methods, including masked language modeling, translation language modeling [Conneau and Lample 2019], dual encoder translation ranking [Guo et al. 2018), and additive margin softmax Yang et al. 2019a], learn cross-lingual sentence representations. It filters 17B monolingual sentences and 6B translation pairs for sentence representation learning. We take the best model, LaBSE with Customized Vocab our baseline. We further report the zero-shot results Large Language Model (LLM), ChatGPT, which is trained on a wide variety of multilingual sentences and instruction tuning based Reinforcement Learning with Human Feedback [Christiano al. 2017 2022
2019
Ouyang al.
E Prompts for ChatGPT
In this section, we show the input prompts of ChatGPT each task in Table 13
Results of each Language
We show the details for tasks and all languages in Tables TO (XNLI. AmericasNL.D, 12 Muulii-
STS), 14 (E), 15 (LAR&QA), [6 MMewsli-X), 17 (XCOPA), 18 BUCC) and 19 (PAWS-X)

---

G Equations and Theoretical Analysis
G.1 Details of Equations
Details of Gaussian Form In EMMA-X, GMM classifier is introduced determine the semantic rank of sentence pairs. The posterior probability Pa(-) of GMM classifier is already discussed Eq. 5. We show the explicit calculation of Gaussian form N, (-)(i),, 7(82))
as:
( [1)15))) ro]" daolo, 3(65(10)--11191 -
-g (p, (2x)(d/2) do.)(
T
(19)
where d is the dimension of hidden states of 7x(xx) and 7(3r)),
Details of contrastive learning The training objective of cross-lingual encoder in EMMA-X the ranking InfoNCE loss. We show the explicit expansion of this loss (Eq.
(c -Ex,Nx
E Eyn~Ve- 2/ (xg) ((60))))
log
EyiDigga
e
71
+ Eyiny../
E Σyr~M
22
(20)
+ log
2/7 12
2/y (
Exirdeza
Eye
Eye
72
Σye~
r3
log
0g
aly(at) (00) Σ
Σ Eyi
13
where T, represents the temperature term. As small temperature tends to be less tolerant similar samples, and large 7 tends to cluster similar samples together [Wang and Liu 2021], we empirically
set Ti < < < 74, which remains the same as Hoffmann et al. 2022]
G.2 Theoretical Analysis
In this section, we provide detailed proof for Eq. 14 and Eq. 15. Next, we prove the feasibility of our dual supervision. GMM classifier clusters sentence pairs terms of Euclidean distance, while cross-lingual encoder minimizes the covariance of each semantic relation rank via cosine distance. Finally, we prove that these two metrics are actually equivalent to each other in the unit hypersphere of the embedding space.
25

---

Proof of Eq. 14. We provide the derivation of Eq. 14. With the assumption that P(x,,
() Nr (x: Yk yr, ), we have,
Σ Σ Σ
(r) log ye. Q(r)
Σ
log
Vexi. =r,
X EV y=1
xEEX YNEY r=1
Σ
log (2/)) (2.12))61_111 (o.11/22)
YNE
+
1
[x:
-Yk) [..] o,- [(x: yka) =Ir])
2
Σ
Σ
(xi yx]]]² 2 Σ ΣΣ65 [x:-- (x:- (x)- mp²
X.ER NEY
XEEAY
Σ n T npp
N
nnn 1)
(21)
with n denoting the number of sentence pairs in semantic rank Here, ignore the impact of o
Proof of Eq. 15. As we apply dual supervision, data in the contrastive label space also follows the distribution (x; yr (r), Hence, under mild assumptions, we can get:
(XTL(X = Ex
log
yE:
X YNEY
(22)
= Σ Σ
(x: (x)) -2
2
X, yxEV
N-1
2
Σ
Based on the definition of semantic ranks, we have jn < júz <.. < juN. Empirically, the number of sentence pairs in each rank n is larger than the number of semantic ranks N. Hence, it can be derived that:
N-1
(x)
Σ
n Σ
1 + ni
AN
n
(23)
= n(n - 1) Σ 2
Σ Σ Σ XEE Q(r) log y(,
Q(r)
Therefore, prove that minimizing the positive terms LT (X, (,M) in contrastive learning is equivalent to maximizing a lower bound of the likelihood in Eq. 12.
According the definition of semantic ranks, the approximated semantic rank eg from GMM classifier should satisfy the following restriction,
Ey 11. (x.) (xx)]] < Eyr 11 (x)) - -- (xeel] < ... < Byr Eyr (000))) (24)
Byz~>/-==211>
-
26

---

Similarly, the approximated semantic rank eiu from cross- tross-lingual encoder should satisfy the following restriction.
Eya-Ves -3 (07))x)), ,(0u)) > Byr~De -07 y/ora) > Byandinn - - ,
(25)
Next, we prove that these two restrictions are interchangeable with each other in a unit hypersphere. For simplicity, consider only two ranks, but extending the explanation to more ranks is trivial. As the Euclidean distance is always larger than O, we have:
Ey, By.~V/g-1111 | - -300 < By- (7(0)) - 10001
+ Eyemy 9 (7 - -0101 < By -2 (7 -0002
+ Ey o -1 (2 (111)))))01)) < Ey v/ 3 -2 -2(2 2 (30)))))(oo)
(26)
By,,,y 1- s/2 (x.) ,g(rا)) By v 2/7 (x:)) ,
Ey Ey (x)) |(0 Ey, V // (xx) 7(00)].
From the above analyses, we can tell that the approximated semantic rank from one module can provide reasonable supervision signal to guide the training of the other module. Hence, all sentence pairs will be uniformly distributed according unified ranking semantic similarity in the embedding space.
27

---

Basic Prompt for XNL./ANLI
Task Description: Read the following and determine the relationship between Hypothesis and Premise.
Choose relation from "contradiction", "neutral". "entailment".
Hypothesis: Yo... no puedo pensar por qué deherías hablarme así, dijo ella, con menos de lo que le había ascgurado antes.
Premise: Blla era una buena amiga de él, por esto lc dolía que le hablara así.
Basic Prompt for MultiSTS
Task Description: Read the following sentences and measure the real-valued meaning similarity between these sentences. You can choose the meaning similarity score, ranging from for meaning overlap for meaning equivalence.
Sentencel: A person is baseball team.
Sentence2: Fine Person spielt in einem Team Baskethall.
Basic Prompt for QF
Task Description: Read the Source sentence and its Translation, and estimate the quality of the Translation. You can rate the translation from 0-1 according to the perceived translation quality.
Source: in Framta a inceput stagnarea demografică de lungä duratä, refacerea durånd generatic.
Translation:
In France, long-term demographic stagnation has started, restoring generation.
Basic Prompt for XCOPA
Task Description: Read the Premise and determine which choice is the effect(or cause) of the Premise
Choose from "Choicel" " or "Choice2".
Premise: Kuki kurukuna wasiman haykurqanku.
Choicel: Kuki kurukunaqa wasimanta chinkarqanku.
Choice2: Kuki kuruqa wasip kurkunta mikhurqanku.
Basic Prompt for MultEURLEX
Task Description: Read the following sentences and determine the legal topic of the given sentence. The legal topic should choose from 'international organisations', 'social questions', 'production'. 'technology and research', 'environment", 'energy', 'transport', 'law", 'finance', 'education and communications,, 'trade', "agriculture', 'forestry and fisheries', 'economies', "agri-foodstuffs' 'EUROPEAN UNION', 'science', "politics', 'international relations', 'industry'. 'geography', 'business and competition', 'employment and working conditions'.
Sentence: NEUVOSTON ASETUS (EU) N:o 1390/2013, annettu päivänä joulukuuta 2013, Euroopan unionin ja Komorien liiton kesken näiden välisessä bulastukuumpanuussoimuukkuss määrätyjen kalastus- mahdollisuuksienn ja taloudellisen korvauksen vahvistamisesta hyväksytyn pöytäkirjan mukaisten kalastus- mahdollisuuksien jakamisesta...
Basic Prompt for MultiARC
Read the following review and predict 5-star scale rating (1 means the poorest experience and 5 represents excellent or outstanding performance) that can best match the review.
Task Description:
Review: no me llego el articulo me lo mando por correos normal sin seguimiento y nunca me llego tota un desastre
Basic Prompt for PAWS-X
Task Description: Read the following sentences and determine whether two sentences are paraphrases.
Return yes or no.
Sentencel: La excepción fue entre fines de 2005 y 2009 cuando jugó en Suecia con Carlstad United BK
Serbia con FK Borae Čačak y el FC Terek Grozny de Rusia.
Sentence2: La excepción dio entre fines del 2005 y 2009, cuando jugó Suecia el Carlstad United BK, Serbia con el FK Borac Čačak y el FC Terek Grozny de Rusia.
Table 13: Prompts of ChatGPT on each task.
28

---

Model
en-de
en-2h
et-en
ne-en
ro-ch
ru-en
si-en
eu-cs
en-ja
km-en
Avg.
XLM-R
0.412
0.566
0.797
0.812
0.891
0.774
0.578
0.547
0.335
0.612
0.635
0.632
INFOXLM
0.517
0.534
0.775
0.834
0.890
0.788
0.581
0.564
0.325
0.635
0.616
0.641
HICTL
0.495
0.579
0.792
0.835
0.904
0.787
0.575
0.556
0.342
0.625
0.648
0.649
EMMA-X
0.580
0.589
0.809
0.854
0.897
0.829
0.593
0.577
0.370
0.641
0.651
0.672
Table 14: WMTT1--QE-Task results (Pearson) across different input languages.
Model
de
el
hi
th
tr
vi
zh
Avg.
XLM-R
34.1
42.4
39.3
44.8
44.0
37.3
41.7
38.6
40.9
40.4
39.5
40.3
INFOXLM
39.7
52.6
39.2
55.1
36.8
51.0
28.5
41.1
48.9
47.3
HICTL
40.3
53.2
41.7
56.3
54.3
39.6
51.7
30.1
42.8
48.9
48,5
46.1
EMMA-X
45.1
58.4
45.4
60.6
59.8
41.4
56.3
34.7
47.1
54.6
53.4
50.6
Table 15: LAReQA results (mean average precision @ 20, mAP@ 20) across different input languages.
Model
de
fa
pl
ta
tr
uk
Avg.
XLM-R
34.6
66.0
62.6
64.8
27.1
47.8
64.8
33.7
17.8
62.3
53.2
48.6
INFOXLM
40.8
71.6
66.3
68.7
48.7
61.0
66.7
39.2
42.0
64.6
58.1
57.1
HICTL
41.7
68.5
64.2
65.6
45.6
51.9
67.6
40.4
32.8
65.5
58.9
54.8
EMMA-X
50.2
78.7
69.1
63.7
47.9
59.6
70.0
50.2
43.5
68.0
60.9
59.6
Table 16: Mewsli-X results (mean average precision @20, mAP@ 20) across different input languages.
Model
et
ht
id
it
ta
th
tr
vi
zh
Avg.
XLM-R
73.8
67.4
77.8
72.2
52.3
70.9
72.1
74.6
73.4
73.2
75.7
71.2
INFOXLM
75.1
73.4
78.3
80.7
65.6
69.1
72.7
73.9
76.9
77.8
77.5
74.6
HICTL
75.9
73.1
77.8
81.2
65.5
73.8
72.6
73.2
76.1
75.4
78.0
74.8
ChatGPT
80.6
64.1
85.6
89.2
47.4
75.9
56.4
67.3
82.2
81.5
85.8
74.2
EMMA-X
76.8
74.0
77.6
79.8
76.2
74.4
77.8
74.2
77.6
82.6
89.6
78.2
Table 17: XCOPA results (accuracy) across different input languages.
Model
de
zh
Model
fr
XLM-R
76.1
72.3
62.3
60.8
67.9
XLM-R
95.7
92.2
92.7
92.5
84.7
85.9
87.1
90.1
INFOXLM
81.3
78.2
76.0
74,2
77.4
INFOXLM
91.7
94.6
95.2
95.1
88.9
89.0
90.2
93.0
HiCT.
97.4
94.2
95.0
94.2
89.1
89.5
90.2
92.8
80.5
76.0
ChatGPT
67.0
EMMA-X
85.1
78.3
ESMA-X
97.3
95.6
91.7
96.0
92.9
89.8
93.0
94.2
Table 18: BUCC results (F1) across dif-
Table 19: PAWS-X results (accuracy) for each
ferent languages.
language.
29


Reviews:
[{"review_id": "fluWDbToVU", "reviewer": "Reviewer_LAMb", "summary": "This paper proposes to apply the EM framework to realize unified sentence representation learning with non-parallel multilingual data. This framework consists of two modules, a GMM classifier and a cross-language encoder, which are responsible for semantically related classification and cross-language unified representation. However, this is achieved on the premise that the parallel multilingual data initializes the models. In addition, the author conducted a theoretical analysis and forms a new benchmark, based on which the effectiveness of the framework was proved.", "questions": "1.\tWhy choose the GMM model as the Semantic Relations model?\n2.\tWhy form a XRETE benchmark, what are its innovations and necessity?\n3.\tThis article introduces monolingual data from a continuous perspective, so what are its advantages compared with discrete methods(e.g. Back-translation)\n", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper applies the EM algorithm to realize the optimization of the multilingual unified representation parallel-trained model under non-parallel data.", "weaknesses": "1.\tWhy choose the GMM model as the Semantic Relations model?\n2.\tWhy form a XRETE benchmark, what are its innovations and necessity?\n3.\tThis article introduces monolingual data from a continuous perspective, so what are its advantages compared with discrete methods(e.g. Back-translation)\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive comments. \n\n**Q1: The reason to use GMM model as Semantic Relation Model.**\n\nPlease refer to the response to Q2 included in the “**General responses to all Reviewers**” part for details.\n\n**Q2: The reason to form an XRETE benchmark.**\n\nThe formation of the XRETE benchmark is driven by the following reasons:\n1. Inclusion of recent, dedicated, and challenging benchmarks: XRETE incorporates more up-to-date benchmarks that specifically focus on universal sentence representations. These newly added benchmarks, such as Legal topics in MultiEURLEX [1], Review topics in Multilingual Amazon Review corpus [2], WMT Quality Estimation [3], and Americas NLI [4], offer increased difficulty in evaluating the performance of models.\n2. Coverage of more low-resource languages: XRETE goes beyond its predecessors by encompassing a broader range of low-resource languages. For instance, it includes 10 Indigenous languages of the Americas, which were not part of the XTREME [5] or XGLUE [6] benchmarks. This expansion ensures that the XRETE benchmark provides a more comprehensive evaluation of models' capabilities across diverse linguistic backgrounds.\n\n**Q3: Advantages of EMMA-X compared with Back-Translation.**\n\nEMMA-X offers several advantages over discrete methods like back-translation:\n1. Faster parallel corpus construction: EMMA-X outperforms back-translation in constructing parallel corpora at a faster pace. The process of generating supervision signals in EMMA-X involves predicting the semantic relation rank between two sentences, which is a more efficient approach compared to back-translation's auto-regressive generation of pseudo-parallel sentences. \n2. Higher quality parallel corpora on low-resource languages: In the context of low-resource languages, EMMA-X excels in constructing parallel corpora with higher quality compared to back-translation. Back-translation may yield erroneous results for some languages, or unavailable for some languages such as pa, ps, and others. Conversely, EMMA-X retrieves real sentences from a monolingual, low-resource corpus, ensuring the production of more reliable and accurate parallel data.\n3. Justification of semantic relation rank predictions during pre-training: EMMA-X uniquely unifies the prediction of semantic relation rank with the learning process of universal semantic representations within an Expectation-Maximization (EM) framework, which allows for the justification of semantic relation rank predictions during pre-training. In contrast, discrete methods like back-translation often rely on fixed translation models for pseudo-parallel corpus construction, leading to limited improvement in sentence representations' quality.\n\n**References:**\n\n[1] Chalkidis I, Fergadiotis M, Androutsopoulos I. MultiEURLEX--A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer.\n\n[2] Phillip Keung, Yichao Lu, György Szarvas, and Noah A. Smith. The multilingual Amazon reviews corpus.\n\n[3] Lucia Specia, Frédéric Blain, Marina Fomicheva, Chrysoula Zerva, Zhenhao Li, Vishrav Chaudhary, and André F. T. Martins. Findings of the WMT 2021 shared task on quality estimation.\n\n[4] Ebrahimi A, Mager M, Oncevay A, et al. Americasnli: Evaluating zero-shot natural language understanding of pretrained multilingual models in truly low-resource languages.\n\n[5] Hu J, Ruder S, Siddhant A, et al. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization.\n\n[6] Liang Y, Duan N, Gong Y, et al. XGLUE: A new benchmark dataset for cross-lingual pre-training, understanding and generation.\n"}, {"review_id": "gz3B8f7J8s", "reviewer": "Reviewer_FMvk", "summary": "The paper proposes EMMA-X, an EM-like approach to pretrain multilingual models. It learns the cross-lingual representation learning task and semantic relation prediction task within EM. They propose a new benchmark, XRETE, to evaluate the experiments with 12 cross-lingual tasks. The training involves two stages: 1) pretraining using parallel data (bitext) and 2) training with EM Framework with non-parallel data. The proposed approach outperforms the baselines.", "questions": "**Suggestion**\n- Figure 1 is not easily understandable. It would be great if there is a clear step-by-step procedure in the caption.\n", "limitations": "No. The authors can have a section to address the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed method outperforms other existing pre-trained models, showing the solid multilingual / cross-lingual ability.\n- The proposed method is backed by a strong theoretical analysis \n- The paper proposes a new benchmark from existing datasets to evaluate the cross-linguality.", "weaknesses": "- The main paper does not provide enough information about the model. I would suggest moving the model details from the Appendix to the main paper.\n- The benchmark does not provide the model parameters. The comparison may not be fair since the model sizes are different.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive reviews.\n\n**Q1: Moving model details from Appendix to Main paper.**\n\nDue to page limitations, certain model details have been included in Appendix A. We will incorporate the model details to the main paper in the next version.\n\n**Q2: Lack of information about model parameters in the benchmark.**\n\nWe have provided the details of our baseline model in Appendix D and will move this part to main pages in the revision. Accully, most of our baseline models share identical model parameters with EMMA-X, as they are also further trained based on the XLM-R [1] model. The only exception is LaBSE [2], which is further trained based on the BERT model, with the only difference being the size of the vocabularies. \n\n**Q3: Adding a step-by-step procedure in the caption of Figure 1.**\n\nIn Section 3.3, we have already introduced EMMA-X step-by-step, and further illustrated the procedure through Algorithm 1. To provide readers with a better understanding of EMMA-X, we will include a step-by-step procedure in the caption of Figure 1.\n\n**References:**\n\n[1] Conneau A, Khandelwal K, Goyal N, et al. Unsupervised cross-lingual representation learning at scale.\n\n[2] Feng F, Yang Y, Cer D, et al. Language-agnostic BERT sentence embedding"}, {"review_id": "mzOIMuVlIK", "reviewer": "Reviewer_s5pr", "summary": "This paper proposes a new approach to learn cross-lingual representation learning as a universal alignment solution for any two languages without parallel data. The proposed approach resembles EM-like algorithm, which consists of a classifier to quantify the semantic similarity of two non-parallel sentences, as well as a sentence encoder that aligns the embedding representation of different languages. During the training process, the two components receive supervision from each other to reinforce the capability to recognize the accurate semantic similarity, where the paper also provides the theoretical justification to demonstrate the mutual influence being positive and effective. A new benchmark is also proposed that comprises four existing cross-lingual transfer tasks; the proposed approach is compared against other recent baselines, as well as ChatGPT. The empirical results show that new state-of-the-art performance can be obtained by this new approach with good margins.", "questions": "What is the primary reason for adopting GMM as the similarity classifier? Can we use a normal classifier (e.g. using CLS on Transformers) and simply use their softmax probability for each rank?", "limitations": "It would be good to discuss how much degradation could happen if there is not enough parallel corpus for the warm-up.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- A new approach is proposed based on a novel EM-like algorithm with mutual inference and supervision by two components. Especially, the approach proposes to rank the similarity of a sentence pair, rather than optimizing the traditional binary positive/negative contrastive objective, which is indeed interesting and effective shown by the results.\n  \n- State-of-the-art performance is achieved by this approach on the new benchmark consisting different types of existing cross-lingual transfer tasks, comparing against strong baselines such as XLM-R.\n  \n- A theoretical analysis is provided, in addition to the superior empirical results to justify the effectiveness of the proposed approach.", "weaknesses": "- It might be over-claiming that the proposed approach operates to align the representation of different languages without parallel data, since Algorithm 1 clearly shows that parallel corpus is needed to bootstrap both the similarity classifier and the sentence encoder, and all experiments are based on this warm up that leverages parallel corpus.\n  \n- There lacks more in-depth analysis regarding the ablation of the proposed approach. In particular:\n  \n  - There could be a simplified version of this approach without iterative optimization: we can first train a similarity classifier based on the proposed mixed-up method, and generate similarity rankings of sentence pairs on non-parallel data, which can then be used to optimize the contrastive objective directly for training the encoder. In this way, we can verify whether the improvement mostly comes from the fine-grained similarity ranking, and how much this iterative optimization mechanism contributes to the final performance.\n    \n  - There could also be another experiment that downgrades the similarity ranking to binary, as if the traditional positive/negative pair. Thus, it could show again that whether this EM-like algorithm is really necessary and how much this process could improve upon the plain contrastive objective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive reviews.\n\n**Q1: Over-claiming EMMA-X operates without parallel data.**\n\nThe primary goal of EMMA-X is to acquire universal semantic representations for a multitude of languages. However, it is important to note that only a limited number of languages (4%) possess parallel data. Therefore, EMMA-X strives to leverage non-parallel data to extend language coverage for universal semantic representations and to enhance representation performance for those languages that have limited available resources.\n\n**Q2: Adding an ablation study about the EM iterative optimization.**\n\nThanks for your constructive advice. We apply an iterative paradigm to optimize the GMM classifier since the parallel data is not sufficient to cover so many languages to train a classifier and the semantic rank expectation from GMM classifier can be biased. The iterative optimization can help GMM classifier to learn better on long-tail languages. \nTo provide additional evidence supporting our claim, we compare EMMA-X with a new baseline that applies a fixed GMM classifier to generate similarity ranks to optimize the encoder, referred to as \"Phase 1 + Fixed GMM\". The results are shown in the ablation experiments (Table 5) in \"General Response to All Reviewers.\" From Table 5, we can see that EMMA-X improves retrieval accuracy by 6.6 compared with \"Phase 1 + Fixed GMM\". On 16 long-tail languages, the retrieval accuracy gap further raises to 10.8. This reveals that the iterative optimization mechanism can further improve the quality of the GMM classifier, especially for long-tail languages.\n\n**Q3: Adding an experiment that downgrades the similarity ranking to binary.**\n\nPlease refer to the response to Q3 included in the “**General responses to all Reviewers**” part for details.\n\n**Q4: The reason to adopt GMM as classifier.**\n\nPlease refer to the response to Q2 included in the “**General responses to all Reviewers**” part for details.\n"}, {"review_id": "YHAXwHZ04U", "reviewer": "Reviewer_oxNi", "summary": "This paper proposes an EM-like pre-training algorithm called EMMA-X to learn cross-lingual representations. The authors unify semantic relation classification and universal representation learning into the framework. To this end, a GMM classifier and a cross-lingual encoder is jointly trained in the algorithm. Finally, the authors validate their method by conducting experiments on 12 cross-lingual tasks including inference, similarity, retrieval, and classification.", "questions": "- Consider removing the comma after each equation\n- Line 69: Why do we have to use L-2 normalization? There are multiple ways to perform normalizations, L-2 is just one of them. Therefore, I would rather say $f(\\cdot)$ is a normalization, e.g., L-2.\n- Figure 1 in caption: \"{y1, and y2, y3, ...}\" -> {$y\\_1, y\\_2, y\\_3, \\cdots$}\n- In Figure 1, I assume $\\gamma^{(x)} - \\gamma^{(y_1)}$ is missing\n- The number of semantic ranks is set to 4. This sounds to me like a random selection. Did you do experiments to justify the choice?\n- The paper claims that the model can learn to learn cross-lingual universals with the aid of massive multilingual non-parallel data, but the cross-lingual encoder is continued trained on infoNCE on some parallel sentence pairs. I would assume this continue-training step is even more important than the method proposed. I would suggest the author justify the initialization by simply using the model weights of XLM-R.\n- Line 188-190: The number of sentence pairs is 3.2 billion, but what is the exact number of non-parallel sentences? That should also be mentioned in the main content of the paper.\n- Line 192: $\\mathbb{R}^1$ -> $\\mathbb{R}$\n- Table 1: the authors should also use special characters to denote which results (BUCC and Tatoeba?) are obtained by zero-shot for EMMA-X.\n- Figure 2 and Table 4: according to the results, S-BERT performs very well (even better) than the proposed method, especially on long-tail languages. In this case, what would be the benefit of applying EMMA-X to those languages instead of simply using S-BERT?\n", "limitations": "The paper does not include a limitation or broader impact section, therefore not applicable.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-structured and easy to follow.\n- The EM-like pre-training, which includes a semantic relation rank task, is somewhat novel.\n- The experiments are extensive, and the results generally indicate the proposed method is effective.\n- The geometric analysis section is interesting and informative, where the authors visualize the representations from four different models. ", "weaknesses": "- Although the authors claim that EMMA-X can learn good representations with excessive non-parallel, the encoder, as far as I can see, is continue-trained on parallel sentence pairs with InfoNCE on parallel corpora for initialization. Therefore, this initialization weakens the claim.\n\n- The author does not verify the choice of a GMM classifier. At least, the author should give some simple intuition to the readers. \n\n- Semantic relation task is very important in the proposed framework. The authors claim that a binary separation is not good, but do not give further justification for the choices of N=4 semantic ranks.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive comments.\n\n**Q1: Initialization with parallel corpora weakens motivation.**\n\nThe primary goal of EMMA-X is to acquire universal semantic representations for a multitude of languages. However, it is important to note that only a limited number of languages (4%) possess parallel data. Therefore, EMMA-X strives to leverage non-parallel data to extend language coverage for universal semantic representations and to enhance representation performance for those languages that have limited available resources.\nTo provide additional evidence supporting our claim, we propose an ablation experiment in the General response to all authors. The results in Table 5 demonstrate that EMMA-X significantly improves the retrieval accuracy by 8.1 when compared to solely using parallel data. Moreover, on 16 long-tail languages, the retrieval accuracy further increases from 76.2 to 90.7, representing a substantial improvement of 19.0%. These findings support our claim that EMMA-X can indeed learn effective universal semantic representations from an abundance of non-parallel data.\n\n**Q2: Comparison between GMM classifier and other simple ones.**\n\nPlease refer to the response to Q2 included in the “**General responses to all Reviewers**” part for details.\n\n**Q3: Missing justification for the choice of semantic similarity ranks.**\n\nPlease refer to the response to Q3 included in the “**General responses to all Reviewers**” part for details.\n\n**Q4: The reason to use L2-Normalization.**\n\nWe measure the semantic similarity between sentence via cosine similarity, which requires an L2-normalization to the output of embeddings of encoder. We also apply L2-Normalization following the common configuration in MoCo [1], HiCTL [2], and InfoXLM [3]. \n\n**Q5: The exact number of non-parallel sentences.**\n\nAs shown in Appendix A, we use 800B of non-parallel data covering 94 languages, which is about 67 times larger than the parallel data. We will add the statistics to the main content of the paper in the next version.\n\n**Q6: The benefit of using EMMA-X rather than S-BERT, based on results from Figure 2 and Table 4.**\n\nTable 4 demonstrates that EMMA-X exhibits superior performance on low-resource languages compared to S-BERT [4], surpassing it by 69.5%, 5.0%, and 29.5% in three geometric criteria, respectively. Similarly, Figure 2 provides visual evidence supporting this claim. Specifically, EMMA-X showcases smaller language-specific clusters on four low-tail languages (tk, xh, tt, and eo) when compared to S-BERT.\n\nThe abnormal results observed on the high KL-divergence, as depicted in Table 4 and Figure 2(k), can be attributed to the representation space for S-BERT deteriorating into a low-dimensional manifold (low isotropy score in Table 4 and Figure 2k), and different languages are not distributed uniformly across the whole representation space, which limits the expressive ability. \n\n**Q7: Typo and expression errors.**\n\nThanks for pointing out the errors. We will proofread the content again to fix the wrong expressions and typos.\n\n**References:**\n\n[1] He K, Fan H, Wu Y, et al. Momentum contrast for unsupervised visual representation learning.\n\n[2] Wei X, Weng R, Hu Y, et al. On learning universal representations across languages.\n\n[3] Chi Z, Dong L, Wei F, et al. InfoXLM: An information-theoretic framework for cross-lingual language model pre-training.\n\n[4] Reimers N, Gurevych I. Sentence-bert: Sentence embeddings using siamese bert-networks.\n"}, {"review_id": "L1gv4oIHQK", "reviewer": "Reviewer_tN44", "summary": "This paper proposed EM-like multilingual pre-training algorithm using both supervised parallel data and unsupervised monolingual data to train an encoder model such that it can produce language agnostic representations for multiple languages. That is, embedding space of sentences of similar meanings will be the same across languages. The method uses an EM procedures that employs an GMM classifier, which is pre-trained with parallel data to produce a similarity rank between 2 input sentence, and then train the encoder model both with parallel data and ranks produced by GMM with unsupervised data.\nThe method achieves higher scores across many inference, similarity and retrieval tasks compared to unsupervised XLM-R.", "questions": "NA", "limitations": "No limitation mentioned.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* The methodology seems original and new, especially for the new rank-based classifier.\n* The paper presentation and method description is clear and understandable.", "weaknesses": "There are multiple concerns with the evaluation and comparison with the method.\n\n* Despite utilize large-scale parallel data from multiple languages heavily in the pretraining stage of encoder and GMM classififer, the paper compares its method with unsupervised models like XLM-R. This make the comparison in main table 1 invalid and incomparable. This method is more semi-supervised with supervised pretraining and unsupervised finetuning stages.\n* There are some comparison with supervised models like LaBSE, but apparently the proposed method do not score well with supervised models. Moreover, the writting seems to suggest the paper is comparing unsupervised vs supervised but in fact this is supervised vs supervised.\n* XLM-R is not designed for language agnostic representation, but there are notable baselines that achieve the same language-agnostic object with simpler methodology: [LASER](https://github.com/facebookresearch/LASER) which use parallel data and [CRISS](https://arxiv.org/abs/2006.09526) which does not use parallel data. Not only they are not compared, they are also not mentioned in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments.\n\n**Q1: Unfair comparison between EMMA-X and unsupervised method (XLM-R).**\n\nAs a semi-supervised method, we have undertaken a comprehensive evaluation by comparing our results with four supervised methods (InfoXLM [1], HiCTL [2], LaBSE [3], and S-BERT [4]) in Tables 1, 2, and 3, leading to substantial improvements. The reason for including XLM-R [5] in our comparisons is that EMMA-X is initialized from XLM-R. **To ensure a fair assessment, we additionally retrained XLM-R, InfoXLM, HiCTL using the same parallel data as EMMA-X, and EMMA-X can outperform retrained baselines by 7.97% on average as presented in Table 1 with the symbol \\ddag.**\n\n**Q2: Poor performance on EMMA-X when compared with supervised models (LaBSE).**\n\nWe have not claimed that EMMA-X is an unsupervised method and explicitly emphasize the usage of supervised data to initialize the model in both Section 3.2 and Algorithm 1. As you said, EMMA-X is a semi-supervised method. EMMA-X can outperform two supervised baselines on all tasks and even outperform the state-of-the-art supervised baseline (LaBSE) on two tasks and all long-tail languags. It is hard to directly compare EMMA-X with LaBSE, as LaBSE uses a large pre-training dataset containing CC-100, Wikipedia, and a fine-selected 6B translation corpus that are much better than EMMA-X in both quantity and quality. However, the notable improvements exhibited by EMMA-X on WMT21QE [6], Mewsli-X [7], and LaREQA [8] in Table 2 and on low-resource languages of Tatoeba [9] and FLORES-200 [10] in Table 3 can further demonstrate the effectiveness of EMMA-X. \n\n**Q3: Lacking baselines and citations.**\n\nWhile LASER [11] is a notable baseline, we do not compare with it because we have chosen a stronger and more recent baseline, LaBSE. We have cited LASER in our paper from the perspective of its data **in line 235**, and we will add a more comprehensive discussion about LASER, LaBSE, S-BERT, and EMMA-X in the next version of our paper. \nEMMA-X and its baselines are encoder-only model designed for learning representations, while CRISS [12] is an unsupervised seq2seq model that mainly performs on machine translation. We will mention CRISS in the next version. \n\n**References:**\n\n[1] Chi Z, Dong L, Wei F, et al. InfoXLM: An information-theoretic framework for cross-lingual language model pre-training.\n\n[2] Wei X, Weng R, Hu Y, et al. On learning universal representations across languages.\n\n[3] Feng F, Yang Y, Cer D, et al. Language-agnostic BERT sentence embedding.\n\n[4] Reimers N, Gurevych I. Sentence-bert: Sentence embeddings using siamese bert-networks.\n\n[5] Conneau A, Khandelwal K, Goyal N, et al. Unsupervised cross-lingual representation learning at scale.\n\n[6] Lucia Specia, Frédéric Blain, Marina Fomicheva, Chrysoula Zerva, Zhenhao Li, Vishrav Chaudhary, and André F. T. Martins. Findings of the WMT 2021 shared task on quality estimation.\n\n[7] Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Dan Garrette, Graham Neubig, and Melvin Johnson. XTREME-R: Towards more challenging and nuanced multilingual evaluation.\n\n[8] Uma Roy, Noah Constant, Rami Al-Rfou, Aditya Barua, Aaron Phillips, and Yinfei Yang. LAReQA: Language-agnostic answer retrieval from a multilingual pool.\n\n[9] Mikel Artetxe and Holger Schwenk. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond.\n\n[10] Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. No language left behind: Scaling human-centered machine translation.\n\n[11] Mikel Artetxe and Holger Schwenk. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond.\n\n[12] Tran C, Tang Y, Li X, et al. Cross-lingual retrieval for iterative self-supervised training."}]

Author rebuttal:
**General Response to all Reviewers:**

Thank all reviewers for their time and efforts.

**Q1: Clarification about “Unsupervised” / “without parallel data” in EMMA-X.**

In EMMA-X, we explicitly show the use of parallel data for initializing the model in Section 3.2 and Algorithm 1. The primary goal of EMMA-X is to acquire universal semantic representations for a multitude of languages. However, it is important to note that only a limited number of languages (4%) possess parallel data. Therefore, EMMA-X strives to leverage non-parallel data to extend language coverage for universal semantic representations and to enhance representation performance for those languages that have limited available resources.
To further prove our claims, we provided an extra ablation study to show how each part in EMMA-X can affect performance and the results are shown in Table 5.

**Q2: The reason to use GMM classifier.**

Given the absence of direct supervision signals, we intuitively hypothesize that each semantic rank follows a Gaussian distribution. To effectively cluster each sentence pair into its corresponding semantic rank, we employ Gaussian Mixture Model (GMM) as the classifier. Moreover, the GMM classifier offers several advantages:
1. Addressing imbalanced data: The GMM classifier is particularly advantageous in handling imbalanced datasets. It can model clusters of varying sizes, ensuring that minority class data points are not overshadowed by the majority class. By allowing data points to contribute to multiple clusters with different probabilities, the GMM classifier mitigates the imbalance issue effectively.
2. Effective handling of outliers: With its soft assignment approach, the GMM classifier can effectively deal with outliers. Outliers are assigned low probabilities of belonging to any cluster, reducing their impact on shifting the cluster centers or covariances significantly.

To further substantiate our claims, we plan to include an additional experiment that compares the performance of the GMM classifier with that of a normal classifier in the forthcoming version.

**Q3: Choice of semantic similarity rank.**

The selection of the semantic similarity rank was made heuristically, and we chose a rank of 4 based on the following two reasons:
1. Addressing data imbalance: A fine-grained similarity rank allows for a smoother distribution of semantic ranks. In the case of binary classification of semantic relations, there tends to be a significant imbalance between negative (non-semantically similar) and positive samples. By employing a fine-grained semantic rank, we can better distribute sentence pairs originally classified as negatives into different ranks, thus mitigating the data imbalance issue.
2. Learning complexity consideration: Conversely, defining a higher number of semantic ranks poses challenges for both the GMM classifier and the cross-lingual encoder to learn. Therefore, we heuristically arrived at the choice of N=4 as it represents a good trade-off between addressing the data imbalance problem and managing learning complexity.

To provide further support for our approach, we are currently conducting experiments on varying the number of semantic ranks, and we intend to include these findings in the next version of our paper.


Meta-review: This is a solid and well-written paper. Experiments are extensive and show that the proposed approach is effective and gives sota results. The authors should ensure they address all of the reviewers' concerns in the camera ready, in particular the comments about claims (ie, use of parallel data to bootstrap the model); the reasons behind choosing GMMs; the choice of ranking vs a binary approach and choice of semantic ranks; clarification wrt supervised vs unsupervised vs semi-supervised and the appropriateness of (and differences between) baselines and methods. The authors promise to add certain additional experiments in the next version of the paper and they are highly encouraged to do so in the camera ready, as well as Table 5 presented here.
Decision: Accept (poster)