Title: EMMA-X: An EM-like Multilingual Pre-training Algorithm for Cross-lingual Representation Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: cross-lingual pretraining;language-agnostic representation
Abstract: Expressing universal semantics common to all languages is helpful to understand the meanings of complex and culture-specific sentences. The research theme underlying this scenario focuses on learning universal representations across languages with the usage of massive parallel corpora. However, due to the sparsity and scarcity of parallel data, there is still a big challenge in learning authentic ``universals'' for any two languages. In this paper, we propose Emma-X: an EM-like Multilingual pre-training Algorithm, to learn Cross-lingual universals with the aid of excessive multilingual non-parallel data. Emma-X unifies the cross-lingual representation learning task and an extra semantic relation prediction task within an EM framework. Both the extra semantic classifier and the cross-lingual sentence encoder approximate the semantic relation of two sentences, and supervise each other until convergence. To evaluate Emma-X, we conduct experiments on xrete, a newly introduced benchmark containing 12 widely studied cross-lingual tasks that fully depend on sentence-level representations. Results reveal that Emma-X achieves state-of-the-art performance. Further geometric analysis of the built representation space with three requirements demonstrates the superiority of Emma-X over advanced models.

EMMA-X: An EM-like Multilingual Pre-training Algorithm for Cross-lingual Representation Learning

Abstract

Expressing universal semantics common to all languages is helpful in understanding the meanings of complex and culture-specific sentences. The research theme underlying this scenario focuses on learning universal representations across languages with the usage of massive parallel corpora. However, due to the sparsity and scarcity of parallel data, there is still a big challenge in learning authentic 'universals' for any two languages. In this paper, we propose EMMA-X: an EM -like M ultilingual pre-training A lgorithm, to learn (X)C ross-lingual universals with the aid of excessive multilingual non-parallel data. EMMA-X unifies the cross-lingual representation learning task and an extra semantic relation prediction task within an EM framework. Both the extra semantic classifier and the cross-lingual sentence encoder approximate the semantic relation of two sentences, and supervise each other until convergence. To evaluate EMMA-X, we conduct experiments on XRETE , a newly introduced benchmark containing 12 widely studied cross-lingual tasks that fully depend on sentence-level representations. Results reveal that EMMA-X achieves state-of-the-art performance. Further geometric analysis of the built representation space with three requirements demonstrates the superiority of EMMA-X over advanced models 2 .

1 Introduction

Research on how to express universal semantics for natural languages (metaphorically as 'alphabet of human thoughts' by Leibniz and von Leibniz [1996]) has lasted a long time. Usually, these universal meanings underlying all human natural languages are referred to as irreducible semantic cores [Wierzbicka, 1999]. These common cores across languages can serve as a bridge, to help better understand the exact meanings of complex sentences in different languages.

In the context of computational linguistics, various works [Huang et al., 2019, Conneau et al., 2020, Chi et al., 2021, Wei et al., 2021, Lee et al., 2022, Li et al., 2023, Chen et al., 2023] have led to great improvements on learning cross-lingual universal representations with the usage of parallel corpora, and verify that multilingual universality contributes a major performance on crosslingual understanding. However, due to the sparsity and scarcity of parallel data, these advanced techniques face a big challenge in learning real universality among all languages. For instance, among the widely-available top 100 languages that theoretically can build 4950 language pairs, only about 200 language pairs have considerable parallel data [Aharoni et al., 2019, Bapna et al.,

2022]. Recently, Large Language Models (LLMs) (e.g., PaLM [Chowdhery et al., 2022], OPT [Zhang et al., 2022b], BLOOMZ [Workshop et al., 2023], ChatGPT, etc.) have reached a milestone in the field of Natural Language Processing, for their promising capability at understanding and following complex natural language instructions in different languages. By modeling a wide variety of sentence samples in discrete sentence space, LLMs can capture some universal linguistic phenomena to gain cross-lingual transferability. This is consistent with our goal of building a universal basement that supports all languages. The difference lies in that we achieve it through learning universal continuous representations across different languages.

Concretely, we propose EMMA-X to tackle the above challenge from a continuous perspective. EMMA-X can learn cross-lingual universal sentence representations with excessive non-parallel multilingual data by unifying two highly dependent tasks in an EM [Moon, 1996] framework: semantic relation classification and cross-lingual universal representation learning. For the former, we introduce a Gaussian Mixture Model [Everitt and Hand, 1981] classifier ( GMMclassifier ) to deal with the key challenge of forming positive sentence pairs for non-parallel multilingual corpora, by annotating the semantic relationship of sentence pairs in any two arbitrary languages on the fly. For the latter, we employ a cross-lingual encoder to learn universal sentence representations via contrastive learning, where positive pairs are chosen by GMM classifier. Further, we construct training signals according to the output of the cross-lingual encoder, to inversely supervise GMM classifier. From the perspective of EM algorithm, in E-step, both modules try to approximate the semantic relationship given a sentence pair sampled from two arbitrary languages. One module is supervised by the approximation of the other to build its own expectation. In M-step, two modules update their parameters by maximizing expectations, respectively. We give a theoretical justification about how these two tasks can be interpreted from an EM perspective (Section 4).

To incentivize the research of universal sentence representation learning, we form a Cross-lingual REpresentation Transfer Evaluation ( XRETE ) benchmark, which includes 12 cross-lingual tasks covering more than 50 languages. XRETE fully depends on sentence-level representations. Experimental results demonstrate that EMMA-X significantly outperforms pre-trained language models [Conneau et al., 2020, Chi et al., 2021] by 32% at most on XRETE. We also perform an evaluation of ChatGPT on XRETE to explore its multilingual performance. Detailed analysis also shows that EMMA-X can mitigate the representation discrepancy between head and massive long-tail languages. We further conduct geometric analysis directly on representation space from three perspectives: Invariance [Abend and Rappoport, 2017], Canonical Form [Teller, 2000] and Isotropy [Mu and Viswanath, 2018], which provides a further understanding of the cross-lingual transferability of these models.

2 Preliminaries

Cross-lingual representation learning aims at mapping sentences from different languages into a unified continuous space, where synonyms across different languages are pulled closer. Given a sentence x , the representation is formulated as

where M ( · ; Θ M ) denotes the encoder network with a set of trainable parameters Θ M , which is typically implemented as a transformer encoder architecture [Vaswani et al., 2017, Devlin et al., 2019, Lee et al., 2022, Feng et al., 2022]. f ( · ) is L-2 normalization and g ( · ) is the aggregate function. We take the final hidden states of ' [CLS] ' token as the aggregate sentence representation.

To learn reasonable representations that can express universal semantics across different languages, various well-designed techniques have been applied to γ ( x ) . Apredominant one is to build contrastive learning (CTL) [Saunshi et al., 2019] objective with parallel corpora. The basic idea is to maximize the similarity between representations (i.e., γ (x) and γ (y) ) of two semantically-equivalent sentences ( x , y ) , while keep randomly sampled irrelevant ones γ (y ′ ) away. Formally, assume B to be a batch of multilingual parallel bitexts, the contrastive loss under InfoNCE [Oord et al., 2018] formulation is

̸

where s ( · ) is implemented as the cosine similarity s ( γ ( x ) , γ ( y ) ) = γ ( x ) ⊤ γ ( y ) ∥ γ ( x ) ∥·∥ γ ( y ) ∥ , y and y ′ are typically called positive and negative samples.

E-Step

Cross-Lingual Encoder

GMM Classifier

M-Step

3 Methodology

We propose EMMA-X that fully exploits massive monolingual data to learn cross-lingual universal representations. As illustrated in Figure 1, EMMA-X consists of two modules: 1) A GMM classifier G ( · ; Θ G ) to approximate the semantic relation of non-parallel sentences. 2) A cross-lingual encoder M ( · ; Θ M ) to convert multilingual sentences into universal representations. For optimization, EMMAX unifies these two modules in an EM framework with dual supervision. In this section, we begin with a definition of the semantic relation rank (§3.1). Then, we introduce model initialization (§3.2) and the proposed training paradigm (§3.3), followed by a dual supervision strategy (§3.4). For a clearer presentation, an Algorithm of EMMA-X is shown in Algorithm 1.

3.1 Rank of Semantic Relations

Mainstream methods model semantic relations with a strict binary separation: positives and negatives. However, the boundary between positives and negatives is blurry, and many samples cannot be clearly classified as either positives or negatives. So it cannot maximize the potential of models to perceive more subtle semantic changes. Also, a binary separation will lead to far more negative samples than positive ones (imbalanced data). To more accurately capture the semantic relation between two sentences and alleviate imbalanced problem, we subdivide the relation into N semantic ranks, where the semantic similarity of each rank decreases as N increases, e.g., c = 1 denotes two sentences are paraphrases of each other, while c = N implies sentences are irrelevant. In practice, we set N to 4.

3.2 Model Initialization

In EMMA-X, the GMM classifier G ( · ; Θ G ) and cross-lingual encoder M ( · ; Θ M ) are initialized by training with massive parallel corpora, respectively.

Initialization of Cross-lingual Encoder. It is initialized with XLM-R [Conneau et al., 2020] and then continuously trained with InfoNCE [Oord et al., 2018] loss by Eq.2. Following HICTL [Wei et al., 2021] and INFOXLM [Chi et al., 2021], we treat the parallel sentence pairs as the query sentence x and its positive counterpart y , while treating the randomly sampled sentence as a negative one y ′ .

Initialization of GMM Classifier. Areasonable solution to warm up the GMM classifier is to use the available cross-lingual parallel corpora as training signals. Suppose x and y are parallel sentences,

and y ′ is an outlier. We set the semantic ranks for ( γ ( x ) , γ ( y ) ) and ( γ ( x ) , γ ( y ′ ) ) as c = 1 and c = N , respectively, according to the definition described in § 3.1. To obtain the fine-grained semantic ranks, we design a linear interpolation strategy similar to Wei et al. [2022] and mixup [Zhang et al., 2018], which provides virtual training examples for each semantic rank. Formally,

where λ ∈ [0 , 1] is sampled from a uniform distribution. We compute r = ⌈ (1 -λ ) · ( c = 1)+ λ · ( c = N ) ⌉ as the soft semantic rank for ( γ ( x ) , γ ( ˜ y ) ) , where ⌈·⌉ means the least integer greater than or equal to the input. The virtual training examples are grouped together with the real parallel corpora to optimize the GMM classifier:

The posterior probability P G ( · ) is formulated as

where the Gaussian form N r is assigned with a prior probability π r , mean µ r and standard deviation σ r that are all parameterized by trainable variables, thus Θ G = { ( π r , µ r , σ r ) | r ∈ [1 , N ] } . See Appendix G for the specific calculation of Gaussian form N r .

3.3 The EM Iteration Framework

After initialization, EMMA-X further trains the GMM classifier G ( · ; Θ G ) and cross-lingual encoder M ( · ; Θ M ) with only multilingual non-parallel data with an EM framework.

E-Step. For optimization, we represent a training batch of multilingual non-parallel sentences as X = { x 1 , x 2 , ..., x I } accompanied by a queue of random sentences as Y = { y 1 , y 2 , ..., y K } for instance comparison. Formally, the expectation for GMM classifier is:

where c ∗ M ∈ [1 , N ] represents an approximated semantic rank for the combination of anchor x i and another random sentence y k , based on the cosine similarity among representations (i.e., γ (x i ) and γ (y k ) ) produced by the cross-lingual encoder (i.e., M ( · ; Θ M ) ). Please refer to §3.4 for details.

Correspondingly, the expectation for the cross-lingual encoder can be calculated with contrastive learning, where the positive samples are established by the maximum a posteriori approximation (argmax prediction) c ∗ G given by the GMM classifier. Specifically, we apply ranking InfoNCE [Hoffmann et al., 2022] as the training objective, which recursively takes parallel sentence pairs in each rank (e.g, c ∗ G ) as positives and ranks that are larger than c ∗ G as negatives. Formally,

where c ∗ G ∈ [1 , N ] represents a semantic rank approximated by the posteriori of GMM classifier (§3.4). For simplicity, we omit the temperature term in Eq. 7, and please see Appendix G for details.

M-Step. We use gradient descent algorithm to update the parameters of each module by optimizing its expectation. At each time step t , where η and η ′ are learning rates for each expectation, formally,

̸

3.4 Dual Supervision

The approximated semantic ranks c ∗ G and c ∗ M are critical in EMMA-X training algorithm. To preserve their quality, we propose dual supervision : predictions from one module are fed to the other to calculate the expectation. In this section, we explain in detail how we approximate the semantic ranks for GMM classifier and cross-lingual encoder, respectively.

Approximate Semantic Rank with GMM classifier. The way to obtain semantic rank with semantic classifier is straightforward. The semantic rank corresponding to the highest probability among multiple Gaussian distributions is chosen as the prediction, which is further used to supervise the cross-lingual encoder M ( · ; Θ M ) , as illustrated in Eq. 7. Formally,

Approximate Semantic Rank with Cross-lingual Encoder. One common way to calculate sentence relation is to measure the similarity between two real-valued representations. Assuming s r (a scalar initialized as r N ) can reflect the general similarity score in semantic rank c = r . Given a random sentence pair ( x i , y k ) , if its similarity score is close to s r , the sentence pair is likely to belong to rank c = r . Cross-lingual encoder M ( · ; Θ M ) determines the semantic relation for each pair according to

where | · | refers to absolute value. Symmetrically, c ∗ M is used to supervise GMM classifier (Eq. 6).

During the training process, the general similarity score for each semantic rank may vary. Thus, we propose a moving-average strategy to adaptively adjust the value of s r to simulate this change. Specifically, at time step t , s r is updated by cosine similarity of all the sentence pairs, which are currently categorized into the rank c = r according to the cross-lingual encoder in Eq. 9.

Here ϵ ∈ [0 , 1] is a momentum coefficient to make s n evolve smoothly during training.

4 Theoretical Analysis

In this section, we provide theoretical justification for EMMA-X and demonstrate the mutual influence between two modules with rigorous interpretation from an EM algorithm perspective. We show that

under dual supervision, minimizing the positive terms in Eq. 7 intrinsically maximizes the objective of a classical clustering algorithm. For simplicity, we assume that each semantic rank has the same number of sentence pairs n and represents model parameters with Θ . In EMMA-X, we model the semantic relation of sentence pair ( x i , y k ) through a joint distribution P ( x i , y k ) with the semantic rank c as a latent variable. Let Q ( c ) be a prior distribution over the possible values of semantic ranks. That is ∑ r Q ( c = r ) = 1 , Q ( c ) ≥ 0 . The training goal is to maximize the following likelihood:

E-Step. To make the inequality hold with equality, we have:

which is the posterior probability and is approximated by the prediction from GMM classifier. Since each sentence pair ( x i , y k ) belongs to only one semantic rank, we approximate Q ( c = r ) = I ( c ∗ G = r ) , which is a one-hot distribution.

M-Step. We try to maximize the likelihood in Eq. 12 under the semantic rank c ∗ G :

The above derivation uses the assumption that P ( x i , y k | c ∗ G = r, Θ) ∼ N r ( x i -y k | ˜ µ r , ˜ σ r ) , with ˜ µ r and ˜ σ r being the mean value and standard deviation of the Euclidean distance between sentence pairs in semantic rank r . Detailed proof of Eq. 14 is in Appendix G .

Next, we prove that minimizing the positive terms in expectation L CTL ( X , Y ; Θ M ) actually equal to maximizing a lower bound of Eq. 14. As we apply dual supervision, data in the contrastive label space also follows the distribution N r ( x i -y k | ˜ µ r , ˜ σ r ) . Hence, under mild assumptions, we can get:

where L + CTL ( · ) means the positive terms. In the derivation, we use the intrinsic property of semantic ranks ( ˜ µ 1 < ˜ µ 2 < ... < ˜ µ N ). Detailed proof is in Appendix G . Eq. 15 demonstrates that with dual supervision, minimizing the contrastive loss can partially maximize the likelihood in Eq. 12.

5 Experiments

To thoroughly evaluate the performance of EMMA-X, we conduct experiments on XRETE benchmark to verify the transfer ability of EMMA-X on various cross-lingual downstream tasks with strong baselines (pre-trained models: XLM-R [Conneau et al., 2020], INFOXLM [Chi et al., 2021], HICTL [Wei et al., 2021], sentence models: LaBSE [Feng et al., 2022], S-BERT [Reimers and Gurevych, 2020]) and ChatGPT in Section 5.2. See Appendices C and D for details. We further conduct geometric analysis in Section 5.3 to better interpret the cross-lingual transferability in EMMA-X.

5.1 Setup

Corpus & Model. We collect parallel corpora from CCAligned [El-Kishky et al., 2020], CCMatrix [Schwenk et al., 2021], WMT [Akhbardeh et al., 2021], and MultiUN [Ziemski et al., 2016], involving 94 languages with 3.2 billion sentence pairs. In addition, we add CC-100 [Conneau et al., 2020]

Model

Metrics

S-BERT

LaBSE

INFOXLM

XLM-R

EMMA-X

FLORES-200

Head

Long-tail

Acc. (

↑

87.5

99.9

83.4

68.2

94.5

Acc. (

↑

)

51.6

82.5

53.8

45.3

Tatoeba

Head

Long-tail

Acc. (

↑

91.4

95.7

87.8

66.3

Acc. (

↑

)

57.8

77.9

56.1

55.7

84.2

91.9

78.1

as the large-scale monolingual corpus with about 800 billion sentences that covers 94 languages. The cross-lingual encoder starts from the well-trained XLM-R large model [Conneau et al., 2020]. The GMM classifier is implemented as a mixture of Gaussian forms, each of which consists of a prior π ∈ R 1 , a mean µ ∈ R 1024 and a standard deviation σ ∈ R 1024 , all are trainable variables. We set the total semantic ranks as N = 4 . The statistics of all data and hyper-parameters are shown in Appendix A .

5.2 XRETE Evaluation

XRETE includes 12 cross-lingual tasks divided into 4 different categories. We report the 'translatetrain' performance in Table 1 on most tasks but zero-shot performance on BUCC and Tatoeba following [Ruder et al., 2021]. Table 2 presents zero-shot comparisons with sentence models.

Comparisons with Pre-trained Models. In Table 1, EMMA-X consistently outperforms all baseline models (XLM-R [Conneau et al., 2020], HICTL [Wei et al., 2021] and INFOXLM [Chi et al., 2021]) with 7.97% improvements on average. Specifically, EMMA-X achieves 88.1% accuracy on XNLI [Conneau et al., 2018] and 50.21% accuracy on ANLI [Ebrahimi et al., 2022] with up to 6.4% improvements than baselines. On the MultiSTS [Reimers and Gurevych, 2020] task, EMMA-X achieves an 87.3 correlation score, outperforming several strong baselines by 5.1 ∼ 21.4, and even achieves comparable performance in the cross-lingual and the monolingual settings (see Appendix F for language-specific results). Furthermore, EMMA-X obtains a 67.2 Pearson score on QE [Specia et al., 2021] task, which is comparable to the winner on the leaderboard 3 without any specific finetuning techniques. As for sentence retrieval, EMMA-X consistently outperforms previous strong baselines among all 4 tasks [Ruder et al., 2021], and demonstrates 2.6% ∼ 39.6% improvements over these baselines. Similar results can be found in sentence classification tasks. EMMA-X obtains an 81.3% top-1 accuracy averaged on XCOPA [Ponti et al., 2020], MultiEURLEX [Chalkidis et al., 2021] and PAWS-X [Yang et al., 2019b] tasks, outperforming XLM-R, INFOXLM and HICTL by 7.0%, 3.9% and 3.5% improvements, respectively. On MultiARC [Keung et al., 2020] task, EMMA-X shows the lowest error rates among all models. The consistent improvements on all tasks reveal that EMMA-X can obtain better universal representations for different natural languages with various

)

)

topics and domains. We further conduct experiments with ChatGPT on XRETE tasks without 4 Retrieval tasks. We list the prompts for each task in Appendix E . ChatGPT's zero-shot performance is worse than fine-tuned pre-trained models and the performance gap is very large on most tasks.

Zero-shot comparisons with sentence models. Compared with XLM-R and INFOXLM, which adopt the same amount of training data as EMMA-X, EMMA-X consistently outperforms XLM-R and INFOXLM by 73.2% and 25.1% on average, as shown in Table 2. The results further prove the effectiveness of our pre-training technique. Through the reciprocation between GMM classifier and cross-lingual encoder, EMMA-X can generate reliable semantic rank for multilingual non-parallel corpora, which can provide more supervision signals than previous baselines. EMMA-X even achieves comparable results with strong supervised methods: LaBSE [Feng et al., 2022] and S-BERT [Reimers and Gurevych, 2020], which both trained on supervised data. LaBSE is trained on a fine-filtered bilingual corpus with 6B translation pairs (2 times larger than EMMA-X), while S-BERT is distilled from a S-BERT model fine-tuned on English NLI, STS datasets, and 50M English paraphrase pairs. Compared with these two methods, EMMA-X can achieve the best results on QE and Mewsli-X by outperforming S-BERT and LaBSE by 71.7% and 117.8% averaged. EMMA-X performs worse than these baselines on MultiSTS and BUCC, for these two tasks only contain rich-resource languages, which already have great deal of parallel data.

Table 4: Comparisons with existing methods on FLORES dataset for geometric analysis. 'cls' and 'mean' represent different pooling strategies to obtain sentence representations.

Performance on Long-tail Languages. One goal of EMMA-X is to learn universal sentence representations accommodated for more languages. To better prove this, we report the retrieval accuracy on FLORES-200 [Costa-jussà et al., 2022] and Tatoeba [Artetxe and Schwenk, 2019]. We reformulate FLORES-200, which contains manual translations in 204 languages (totaling 3001 sentences) to perform retrieval tasks in the same way as Tatoeba and report the performance in terms of language data scale in Table 3. Details about the separation of languages and FLORES-200 are shown in Appendices A and B . On head languages, EMMA-X performs worse than LaBSE by about 4.6% but outperforms S-BERT by 3.5%. On the long-tail languages, EMMA-X can surpass S-BERT by 4.3% averaged on two tasks. EMMA-X can even exceed the strongest LaBSE by 2.1% on FLORES. One reason for the superior results on long-tail languages is that for those long-tail languages that have only bi-lingual parallel data with rich-resource languages (often English), EMMA-X can provide multi-lingual semantic relation signals for them with arbitrary languages through dual supervision.

5.3 Geometric Analysis

To interpret the advantages of EMMA-X, we evaluate the geometric characteristics of it on FLORES200 dataset [Costa-jussà et al., 2022] without any fine-tuning. The criteria of three requirements are Invariance, measured with KL-divergence (KL-D) [Kullback and Leibler, 1951], Canonical Form, measured with Calinski-Harabasz Index (CH-I) [Cali´ nski and Harabasz, 1974] and Isotropy, measured with principal ratio (PR) [Mu and Viswanath, 2018]. Details of them are shown in Appendix B . We report the numerical results in Table 4 and visualize each characteristic in Figure 2.

Invariance & Canonical Form aim to measure how languages are aligned in the representation space. If the sentence representations are universal, then sentences in different languages should follow a similar distribution, which is measured by invariance in KL-divergence. Similarly, canonical form measures how well semantic equivalent sentences are grouped into one cluster, with a clustering metric (CH-I). In Table 4, S-BERT outperforms other baselines in 'Invariance' and 'Canonical Form' on head languages. However, EMMA-X shows better performance on long-tail languages in these two metrics, which is consistent with Table 3. Figure 2 presents similar results. Among the 20 languages we randomly sampled from FLORES, EMMA-X can align 17 languages as shown in Figure 2d, with 'xh, eo, ur' as outliers. In Figure 2e, 2g, 2f and 2h, different colors represent different languages. So

Frequency of the same plots

100

50

0

50

100

0.00

0.05

0.10

0.15

0.20

0.25

ml

eo

cs

es

mr

ug

sr

te

ta

tk

tt

ur

fr

xh

hu

bs

sk

en

(a) XLM-R (Invariance)

100

50

0

50

100

0.00

0.05

0.10

0.15

0.20

ml

eo

cs

es

mr

ug

sr

te

ta

tk

tt

ur

fr

xh

hu

bs

sk

en

(b) INFOXLM (Invariance)

-100

-50

0

50

100

0.00

0.02

0.04

0.06

0.08

ml

eo

cs

es

mr

ug

sr

te

ta

tk

tt

ur

fr

xh

hu

bs

sk

en

(c) S-BERT (Invariance)

100

50

0

50

100

0.00

0.05

0.10

0.15

0.20

ml

eo

cs

es

mr

ug

sr

te

ta

tk

tt

ur

fr

xh

hu

bs

sk

en

(d) EMMA-X (Invariance)

100

50

0

50

100

100

50

0

50

100

ml

eo

cs

es

mr

ug

sr

te

ta

tk

tt

ur

fr

xh

hu

bs

sk

en

(e) XLM-R (Canonical)

100

50

0

50

100

100

50

0

50

100

ml

eo

cs

es

mr

ug

sr

te

ta

tk

tt

ur

fr

xh

hu

bs

sk

en

(f) INFOXLM (Canonical)

-80

-60

-40

-20

0

20

40

60

80

-80

-60

-40

-20

0

20

40

60

ml

eo

cs

es

mr

ug

sr

te

ta

tk

tt

ur

fr

xh

hu

bs

sk

en

(g) S-BERT (Canonical)

80

60

40

20

0

20

40

60

80

100

100

50

0

50

100

ml

eo

cs

es

mr

ug

sr

te

ta

tk

tt

ur

fr

xh

hu

bs

sk

en

(h) EMMA-X (Canonical)

0.05

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.02

0.00

0.02

0.04

0.06

0.08

0.10

0.12

50

100

150

200

250

300

350

400

Frequency of the same plots

(i) XLM-R (Isotropy)

0.2

0.0

0.2

0.4

0.6

0.8

0.4

0.2

0.0

0.2

0.4

20

40

60

80

100

120

140

Frequency of the same plots

(j) INFOXLM (Isotropy)

-1.0

-0.5

0.0

0.5

1.0

1.5

2.0

1e7

-2000

0

2000

4000

6000

8000

5000

10000

15000

20000

25000

30000

35000

Frequency of the same plots

(k) S-BERT (Isotropy)

0.2

0.1

0.0

0.1

0.2

0.3

0.4

0.3

0.2

0.1

0.0

0.1

0.2

20

40

60

80

100

(l) EMMA-X (Isotropy)

a cluster with only one color means this language is isolated from other languages and not aligned well in representation space. Figure 2h shows that EMMA-X performs well in most languages.

Isotropy measures how expressive a representation space is since high-dimensional representation space can easily deteriorate into a low-dimensional manifold. From Table 4, EMMA-X achieves the best results in isotropy. The Isotropy of S-BERT (k) is different from other methods. We conjecture the reason is that S-BERT removes MLM tasks during fine-tuning, so token embeddings will only receive sentence-level supervision, resulting in identical sentence representations for different languages. The abnormal results observed on the high KL-divergence, as depicted in Table 4 and Figure 2c, can be attributed to the representation space for S-BERT deteriorating into a low-dimensional manifold (low isotropy score in Table 4 and Figure 2k), and different languages are not distributed uniformly across the whole representation space, which limits the expressive ability.

5.4 Ablation Analysis

The primary goal of EMMA-X is to acquire universal semantic representations for a multitude of languages. For the limited number of parallel data, EMMA-X strives to leverage non-parallel data to extend language coverage for universal semantic representations and to enhance representation performance for those languages. To provide additional evidence supporting our claim, we propose an ablation experiment in Table 5. The results in Table 5 demonstrate that EMMA-X significantly

improves the retrieval accuracy by 8.1 when compared to solely using parallel data. Moreover, on 16 long-tail languages, the retrieval accuracy further increases from 76.2 to 90.7, representing a substantial improvement of 19.0%. These findings support our claim that EMMA-X can indeed learn effective universal semantic representations from an abundance of non-parallel data.

5.5 Semantic Rank and GMM Analysis

We assessed EMMA-X's performance on XRETE benchmark with various settings in Table 6. The evaluation focuses on two aspects: the classifier type and the value of semantic similarity rank.

Type of the classifier. Without direct supervision, we propose each semantic rank follow a Gaussian distribution. Comparing ' GMM ' with ' FCN ' in Table 6, the performance drops from 70 . 7 to 68 . 1 . This decline is attributed to the handling of outliers: the ' GMM ' handles them better due to its soft assignment approach, giving outliers low probabilities and thereby minimizing their influence.

Value of Semantic Similarity Rank. Different semantic similarity ranks were evaluated in Table 6. The peak performance at N = 4 implies that a refined similarity rank ensures a balanced distribution of semantic ranks, which helps alleviate data imbalance. However, increasing the number of semantic ranks complicates the learning process for both ' GMM ' and the cross-lingual encoder. N = 4 strikes a balance between data distribution and learning complexity.

6 Related Work

Cross-lingual Representation Pre-training In recent years, various studies [Devlin et al., 2019, Conneau et al., 2020] have shifted the monolingual pre-training procedure to multilingual scenarios. Most of them often rely heavily on parallel data to learn cross-lingual sentence representations, with several improved techniques, such as language modeling [Conneau and Lample, 2019] and contrastive learning [Chi et al., 2021, Hu et al., 2021, Wei et al., 2021]. Recently, some endeavors incorporate monolingual data into parallel corpora, by translating monolingual corpora into pseudo parallel corpora [Kvapilíková et al., 2020, Ouyang et al., 2021], or computing the semantic similarity in monolingual corpora off-the-shelf and using it as supervision signals [Goswami et al., 2021].

Contrastive Learning has become a popular paradigm in NLP. Besides constructing the positives and negatives through parallel corpora [Zhang et al., 2021] or other labeled data [Gunel et al., 2021, Ni et al., 2022], researchers also adopt self-supervised methods, which build positives and negatives by corrupting sentences [Gao et al., 2021, Chuang et al., 2022] or data augmentation methods [Zhang et al., 2022a, Wu et al., 2022]. Another line of research improves the quality of negatives by preserving a memory queue [Yang et al., 2021, Wang et al., 2021] or generating high-quality negatives [Zhang et al., 2022c, Wei et al., 2022].

7 Conclusion

In this paper, we study the problem of learning cross-lingual universal representations. Major contributions are fourfold: 1) We propose a novel paradigm EMMA-X, which can make full use of massive monolingual data to learn universities for any two languages. 2) We summarize three requirements for the universal representation space among all languages and verify the superiority of EMMA-X towards strong baselines. 3) To incentivize the research of cross-lingual universal representation learning, we form a novel benchmark (XRETE) with 12 cross-lingual tasks fully depending on sentence-level representations. 4) Experiments on XRETE demonstrate that EMMA-X achieved state-of-the-art results over strong baselines.

References

Omri Abend and Ari Rappoport. The state of the art in semantic representation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 77-89, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v1/P17-1008. URL https://aclanthology.org/P17-1008 .

Roee Aharoni, Melvin Johnson, and Orhan Firat. Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 3874-3884, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1388. URL https://aclanthology.org/N19-1388 .

Farhad Akhbardeh, Arkady Arkhangorodsky, Magdalena Biesialska, Ondˇ rej Bojar, Rajen Chatterjee, Vishrav Chaudhary, Marta R. Costa-jussa, Cristina España-Bonet, Angela Fan, Christian Federmann, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Leonie Harter, Kenneth Heafield, Christopher Homan, Matthias Huck, Kwabena Amponsah-Kaakyire, Jungo Kasai, Daniel Khashabi, Kevin Knight, Tom Kocmi, Philipp Koehn, Nicholas Lourie, Christof Monz, Makoto Morishita, Masaaki Nagata, Ajay Nagesh, Toshiaki Nakazawa, Matteo Negri, Santanu Pal, Allahsera Auguste Tapo, Marco Turchi, Valentin Vydrin, and Marcos Zampieri. Findings of the 2021 conference on machine translation (WMT21). In Proceedings of the Sixth Conference on Machine Translation , pages 1-88, Online, November 2021. Association for Computational Linguistics. URL https://aclanthology.org/2021.wmt-1.1 .

Mikel Artetxe and Holger Schwenk. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics , 7:597-610, 2019. doi: 10.1162/tacl_a_00288. URL https://aclanthology.org/Q19-1038 .

Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 4623-4637, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.421. URL https://aclanthology.org/2020.acl-main. 421 .

Ankur Bapna, Isaac Caswell, Julia Kreutzer, Orhan Firat, Daan van Esch, Aditya Siddhant, Mengmeng Niu, Pallavi Nikhil Baljekar, Xavier Garcia, Wolfgang Macherey, Theresa Breiner, Vera Saldinger Axelrod, Jason Riesa, Yuan Cao, Mia Chen, Klaus Macherey, Maxim Krikun, Pidong Wang, Alexander Gutkin, Apu Shah, Yanping Huang, Zhifeng Chen, Yonghui Wu, and Macduff Richard Hughes. Building machine translation systems for the next thousand languages. Technical report, Google Research, 2022.

Jan A. Botha, Zifei Shan, and Daniel Gillick. Entity Linking in 100 Languages. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 7833-7845, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/ v1/2020.emnlp-main.630. URL https://aclanthology.org/2020.emnlp-main.630 .

Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing , pages 632-642, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10.18653/v1/D15-1075. URL https: //www.aclweb.org/anthology/D15-1075 .

Cali´ nski and J Harabasz. A dendrite method for cluster analysis. Communications in Statistics , 3(1):1-27, 1974. doi: 10.1080/03610927408827101. URL https://www.tandfonline.com/ doi/abs/10.1080/03610927408827101 .

Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017) , pages 1-14, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10.18653/v1/S17-2001. URL https://aclanthology.org/S17-2001 .

Ilias Chalkidis, Manos Fergadiotis, and Ion Androutsopoulos. MultiEURLEX - a multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 6974-6996, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.559. URL https://aclanthology.org/2021. emnlp-main.559 .

Nuo Chen, Linjun Shou, Ming Gong, Jian Pei, Bowen Cao, Jianhui Chang, Daxin Jiang, and Jia Li. Alleviating over-smoothing for unsupervised sentence representation, 2023.

Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. InfoXLM: An information-theoretic framework for crosslingual language model pre-training. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 3576-3588, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/ 2021.naacl-main.280. URL https://aclanthology.org/2021.naacl-main.280 .

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways, 2022.

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/ paper_files/paper/2017/file/d5e2c0adad503c91f91df240d0cd4e49-Paper.pdf .

Yung-Sung Chuang, Rumen Dangovski, Hongyin Luo, Yang Zhang, Shiyu Chang, Marin Soljacic, Shang-Wen Li, Scott Yih, Yoon Kim, and James Glass. DiffCSE: Difference-based contrastive learning for sentence embeddings. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 4207-4218, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10. 18653/v1/2022.naacl-main.311. URL https://aclanthology.org/2022.naacl-main.311 .

Alexis Conneau and Guillaume Lample. Cross-lingual language model pretraining. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper/2019/file/ c04c19c2c2474dbf5f7ac4372c5b9af1-Paper.pdf .

Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 2475-2485, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1269. URL https://www.aclweb.org/anthology/D18-1269 .

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 8440-8451, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.747. URL https://aclanthology.org/2020.acl-main.747 .

Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672 , 2022.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL https: //aclanthology.org/N19-1423 .

Abteen Ebrahimi, Manuel Mager, Arturo Oncevay, Vishrav Chaudhary, Luis Chiruzzo, Angela Fan, John Ortega, Ricardo Ramos, Annette Rios, Ivan Vladimir Meza Ruiz, Gustavo GiménezLugo, Elisabeth Mager, Graham Neubig, Alexis Palmer, Rolando Coto-Solano, Thang Vu, and Katharina Kann. AmericasNLI: Evaluating zero-shot natural language understanding of pretrained multilingual models in truly low-resource languages. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 6279-6299, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022. acl-long.435. URL https://aclanthology.org/2022.acl-long.435 .

Ahmed El-Kishky, Vishrav Chaudhary, Francisco Guzmán, and Philipp Koehn. CCAligned: A massive collection of cross-lingual web-document pairs. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 5960-5969, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main. 480. URL https://aclanthology.org/2020.emnlp-main.480 .

BS Everitt and DJ Hand. Finite mixture distributions. Monographs on Applied Probability and Statistics , 1981.

Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. Language-agnostic BERT sentence embedding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 878-891, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.62. URL https://aclanthology.org/2022.acl-long.62 .

Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 6894-6910, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.552. URL https: //aclanthology.org/2021.emnlp-main.552 .

Andrew Gordon, Zornitsa Kozareva, and Melissa Roemmele. SemEval-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics - Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012) , pages 394-398, Montréal, Canada, 7-8 June 2012. Association for Computational Linguistics. URL https://aclanthology.org/S12-1052 .

Koustava Goswami, Sourav Dutta, Haytham Assem, Theodorus Fransen, and John P. McCrae. Crosslingual sentence embedding using multi-task learning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 9099-9113, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/ v1/2021.emnlp-main.716. URL https://aclanthology.org/2021.emnlp-main.716 .

Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc'Aurelio Ranzato, Francisco Guzmán, and Angela Fan. The Flores-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computational Linguistics , 10:522-538, 2022. doi: 10.1162/tacl_a_00474. URL https: //aclanthology.org/2022.tacl-1.30 .

Beliz Gunel, Jingfei Du, Alexis Conneau, and Veselin Stoyanov. Supervised contrastive learning for pre-trained language model fine-tuning. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=cu7IUiOhujH .

Mandy Guo, Qinlan Shen, Yinfei Yang, Heming Ge, Daniel Cer, Gustavo Hernandez Abrego, Keith Stevens, Noah Constant, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. Effective parallel corpus mining using bilingual sentence embeddings. In Proceedings of the Third Conference on Machine Translation: Research Papers , pages 165-176, Brussels, Belgium, October 2018. Association for Computational Linguistics. doi: 10.18653/v1/W18-6317. URL https://aclanthology.org/W18-6317 .

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9726-9735. IEEE Computer Society, 2020.

David T Hoffmann, N Behrmann, J Gall, Thomas Brox, and M Noroozi. Ranking info noise contrastive estimation: Boosting contrastive learning via ranked positives. In AAAI Conference on Artificial Intelligence , 2022.

Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 44114421. PMLR, 13-18 Jul 2020. URL https://proceedings.mlr.press/v119/hu20b.html .

Junjie Hu, Melvin Johnson, Orhan Firat, Aditya Siddhant, and Graham Neubig. Explicit alignment objectives for multilingual bidirectional encoders. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 3633-3643, Online, June 2021. Association for Computational Linguistics. doi: 10. 18653/v1/2021.naacl-main.284. URL https://aclanthology.org/2021.naacl-main.284 .

Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, and Ming Zhou. Unicoder: A universal language encoder by pre-training with multiple cross-lingual tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 2485-2494, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1252. URL https://aclanthology.org/D19-1252 .

Jeannie Y Irwin, Henk Harkema, Lee M Christensen, Titus Schleyer, Peter J Haug, and Wendy W Chapman. Methodology to develop and evaluate a semantic representation for nlp. In AMIA Annual Symposium Proceedings , volume 2009, page 271. American Medical Informatics Association, 2009.

Fabio Kepler, Jonay Trénous, Marcos Treviso, Miguel Vera, and André F. T. Martins. OpenKiwi: An open source framework for quality estimation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations , pages 117-122, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-3020. URL https://aclanthology.org/P19-3020 .

Phillip Keung, Yichao Lu, György Szarvas, and Noah A. Smith. The multilingual Amazon reviews corpus. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 4563-4568, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.369. URL https://aclanthology.org/2020. emnlp-main.369 .

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, San Diego, CA , 2015. URL http://arxiv.org/abs/ 1412.6980 .

Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 66-71, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10.18653/v1/ D18-2012. URL https://aclanthology.org/D18-2012 .

Kullback and R. A. Leibler. On Information and Sufficiency. The Annals of Mathematical Statistics , 22(1):79 - 86, 1951. doi: 10.1214/aoms/1177729694. URL https://doi.org/10.1214/aoms/ 1177729694 .

Ivana Kvapilíková, Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Ondˇ rej Bojar. Unsupervised multilingual sentence embeddings for parallel corpus mining. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop , pages 255-262, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020. acl-srw.34. URL https://aclanthology.org/2020.acl-srw.34 .

Seonghyeon Lee, Dongha Lee, Seongbo Jang, and Hwanjo Yu. Toward interpretable semantic textual similarity via optimal transport-based contrastive sentence learning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 5969-5979, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.412. URL https://aclanthology.org/2022.acl-long.412 .

Gottfried Wilhelm Leibniz and Gottfried Wilhelm Freiherr von Leibniz. Leibniz: New essays on human understanding . Cambridge University Press, 1996.

Ziheng Li, Shaohan Huang, Zihan Zhang, Zhi-Hong Deng, Qiang Lou, Haizhen Huang, Jian Jiao, Furu Wei, Weiwei Deng, and Qi Zhang. Dual-alignment pre-training for cross-lingual sentence embedding, 2023.

Fuli Luo, Wei Wang, Jiahao Liu, Yijia Liu, Bin Bi, Songfang Huang, Fei Huang, and Luo Si. VECO: Variable and flexible cross-lingual pre-training for language understanding and generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 3980-3994, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.308. URL https://aclanthology.org/2021.acl-long.308 .

T.K. Moon. The expectation-maximization algorithm. IEEE Signal Processing Magazine , 13(6): 47-60, 1996. doi: 10.1109/79.543975.

Jiaqi Mu and Pramod Viswanath. All-but-the-top: Simple and effective postprocessing for word representations. In International Conference on Learning Representations , 2018. URL https: //openreview.net/forum?id=HkuGJ3kCb .

Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith Hall, Daniel Cer, and Yinfei Yang. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. In Findings of the Association for Computational Linguistics: ACL 2022 , pages 1864-1874, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.findings-acl.146. URL https://aclanthology.org/2022.findings-acl.146 .

Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR , abs/1807.03748, 2018. URL http://arxiv.org/abs/1807.03748 .

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.

Xuan Ouyang, Shuohuan Wang, Chao Pang, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. ERNIE-M: Enhanced multilingual representation by aligning cross-lingual semantics with monolingual corpora. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 27-38, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.3. URL https://aclanthology.org/2021.emnlp-main.3 .

Serge Sharoff Pierre Zweigenbaum and Reinhard Rapp. Overview of the third bucc shared task: Spotting parallel sentences in comparable corpora. In Reinhard Rapp, Pierre Zweigenbaum, and Serge Sharoff, editors, Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018) , Paris, France, may 2018. European Language Resources Association (ELRA). ISBN 979-10-95546-07-8.

Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vuli´ c, and Anna Korhonen. XCOPA: A multilingual dataset for causal commonsense reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 2362-2376, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020. emnlp-main.185. URL https://aclanthology.org/2020.emnlp-main.185 .

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing , pages 2383-2392, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v1/D16-1264. URL https://aclanthology. org/D16-1264 .

Nils Reimers and Iryna Gurevych. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 4512-4525, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.365. URL https: //aclanthology.org/2020.emnlp-main.365 .

Uma Roy, Noah Constant, Rami Al-Rfou, Aditya Barua, Aaron Phillips, and Yinfei Yang. LAReQA: Language-agnostic answer retrieval from a multilingual pool. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 5919-5930, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main. 477. URL https://aclanthology.org/2020.emnlp-main.477 .

Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Dan Garrette, Graham Neubig, and Melvin Johnson. XTREME-R: Towards more challenging and nuanced multilingual evaluation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 10215-10245, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/ v1/2021.emnlp-main.802. URL https://aclanthology.org/2021.emnlp-main.802 .

Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 4463-4473, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1454. URL https://aclanthology.org/D19-1454 .

Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoretical analysis of contrastive unsupervised representation learning. In International Conference on Machine Learning , pages 5628-5637. PMLR, 2019.

Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, Armand Joulin, and Angela Fan. CCMatrix: Mining billions of high-quality parallel sentences on the web. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 6490-6500, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long. 507. URL https://aclanthology.org/2021.acl-long.507 .

Lucia Specia, Frédéric Blain, Marina Fomicheva, Chrysoula Zerva, Zhenhao Li, Vishrav Chaudhary, and André F. T. Martins. Findings of the WMT 2021 shared task on quality estimation. In Proceedings of the Sixth Conference on Machine Translation , pages 684-725, Online, November 2021. Association for Computational Linguistics. URL https://aclanthology.org/2021. wmt-1.71 .

Virginia Teller. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition. Computational Linguistics , 26(4):638-641, 12 2000. ISSN 0891-2017. doi: 10.1162/089120100750105975. URL https://doi.org/10. 1162/089120100750105975 .

Nattapong Tiyajamorn, Tomoyuki Kajiwara, Yuki Arase, and Makoto Onizuka. Languageagnostic representation from multilingual sentence encoders for cross-lingual similarity estimation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 7764-7774, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.612. URL https://aclanthology.org/2021.emnlp-main.612 .

Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research , 9(86):2579-2605, 2008. URL http://jmlr.org/papers/v9/ vandermaaten08a.html .

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30, NIPS 2017 4-9 December 2017, Long Beach, CA, USA , pages 5998-6008, 2017. URL http://papers.nips.cc/paper/7181-attention-is-all-you-need .

Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2495-2504, June 2021.

Liang Wang, Wei Zhao, and Jingming Liu. Aligning cross-lingual sentence representations with dual momentum contrast. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 3807-3815, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.309. URL https://aclanthology.org/2021.emnlp-main.309 .

Xiangpeng Wei, Rongxiang Weng, Yue Hu, Luxi Xing, Heng Yu, and Weihua Luo. On learning universal representations across languages. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=Uu1Nw-eeTxJ .

Xiangpeng Wei, Heng Yu, Yue Hu, Rongxiang Weng, Weihua Luo, and Rong Jin. Learning to generalize to more: Continuous semantic augmentation for neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 7930-7944, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.546. URL https://aclanthology.org/2022. acl-long.546 .

Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the Twelfth Language Resources and Evaluation Conference , pages 4003-4012, Marseille, France, May 2020. European Language Resources Association. ISBN 979-10-95546-34-4. URL https://aclanthology.org/2020.lrec-1.494 .

Anna Wierzbicka. Emotions across languages and cultures: Diversity and universals . Cambridge university press, 1999.

Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) , pages 1112-1122, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18-1101. URL https://aclanthology.org/N18-1101 .

BigScience Workshop, :, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili´ c, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, and et al. Bloom: A 176b-parameter open-access multilingual language model, 2023.

Xing Wu, Chaochen Gao, Zijia Lin, Jizhong Han, Zhongyuan Wang, and Songlin Hu. Infocse: Information-aggregated contrastive learning of sentence embeddings, 2022. URL https:// arxiv.org/abs/2210.06432 .

Nan Yang, Furu Wei, Binxing Jiao, Daxing Jiang, and Linjun Yang. xMoCo: Cross momentum contrastive learning for open-domain question answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 6120-6129, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.477. URL https://aclanthology.org/2021.acl-long.477 .

Yinfei Yang, Gustavo Hernandez Abrego, Steve Yuan, Mandy Guo, Qinlan Shen, Daniel Cer, Yunhsuan Sung, Brian Strope, and Ray Kurzweil. Improving multilingual sentence embedding using bi-directional dual encoder with additive margin softmax. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19 , pages 5370-5378. International Joint Conferences on Artificial Intelligence Organization, 7 2019a. doi: 10.24963/ijcai.2019/746. URL https://doi.org/10.24963/ijcai.2019/746 .

Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. PAWS-X: A cross-lingual adversarial dataset for paraphrase identification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 3687-3692, Hong Kong, China, November 2019b. Association for Computational Linguistics. doi: 10.18653/v1/D19-1382. URL https://aclanthology.org/D19-1382 .

Dejiao Zhang, Shang-Wen Li, Wei Xiao, Henghui Zhu, Ramesh Nallapati, Andrew O. Arnold, and Bing Xiang. Pairwise supervised contrastive learning of sentence representations. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 5786-5798, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.467. URL https://aclanthology.org/2021. emnlp-main.467 .

Dejiao Zhang, Wei Xiao, Henghui Zhu, Xiaofei Ma, and Andrew Arnold. Virtual augmentation supported contrastive learning of sentence representations. In Findings of the Association for Computational Linguistics: ACL 2022 , pages 864-876, Dublin, Ireland, May 2022a. Association for Computational Linguistics. doi: 10.18653/v1/2022.findings-acl.70. URL https://aclanthology.org/2022.findings-acl.70 .

Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations , 2018. URL https: //openreview.net/forum?id=r1Ddp1-Rb .

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022b.

Yanzhao Zhang, Richong Zhang, Samuel Mensah, Xudong Liu, and Yongyi Mao. Unsupervised sentence representation via contrastive learning with mixing negatives. Proceedings of the AAAI Conference on Artificial Intelligence , 36(10):11730-11738, Jun. 2022c. doi: 10.1609/aaai.v36i10. 21428. URL https://ojs.aaai.org/index.php/AAAI/article/view/21428 .

Yuan Zhang, Jason Baldridge, and Luheng He. PAWS: Paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 1298-1308, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1131. URL https://aclanthology.org/N19-1131 .

Michał Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. The United Nations parallel corpus v1.0. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16) , pages 3530-3534, Portorož, Slovenia, May 2016. European Language Resources Association (ELRA). URL https://aclanthology.org/L16-1561 .

Pierre Zweigenbaum, Serge Sharoff, and Reinhard Rapp. Overview of the second BUCC shared task: Spotting parallel sentences in comparable corpora. In Proceedings of the 10th Workshop on Building and Using Comparable Corpora , pages 60-67, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10.18653/v1/W17-2512. URL https://www. aclweb.org/anthology/W17-2512 .

A Training Corpora and Hyper-parameters

A.1 Training Corpora

As for monolingual data, we follow Conneau et al. [2020] to build a Common-Crawl Corpus using the CCNet [Wenzek et al., 2020] tool 4 , which is widely used in the literature Huang et al. [2019], Luo et al. [2021], Chi et al. [2021], Wei et al. [2021]. Further, we collect parallel corpora from CCAligned El-Kishky et al. [2020], CCMatrix Schwenk et al. [2021], WMT Akhbardeh et al. [2021], and MultiUN Ziemski et al. [2016], involving 94 languages with more than 4.8 billion sentence pairs. We use the OpusFilter 5 tool to remove noisy bitexts, which results in 3.2 billion sentence pairs. Table 7 shows the statistics for both monolingual and parallel data. We apply subword tokenization directly on raw text data using Sentence Piece Model Kudo and Richardson [2018] without any additional preprocessing. To better support our motivation that EMMA-X can cover more languages than previous cross-lingual sentence representations, we divide Tatoeba Artetxe and Schwenk [2019] into two subsets: 'Head', containing languages usually covered in previous methods, and 'Long-tail', with other languages. We treat the 36 languages contained in XTREME Ruder et al. [2021] as head languages, which are: 'ar, he, vi, id, jv, tl, eu, ml, ta, te, af, nl, en, de, el, bn, hi, mr, ur, fa, fr, it, pt, es, bg, ru, ja, ka, ko, th, sw, zh, kk, tr, et, fi, hu, az, lt, pl, uk, ro' . The remaining 76 languages in Tatoeba are treated as long-tail ones.

A.2 Hyper-parameters

The parameters of EMMA-X are first initialized with XLM-R, with 24 layers of Transformer [Vaswani et al., 2017] encoder, 1024 hidden states, and 16 attention heads. We set the total semantic ranks as 4. The GMM classifier is implemented as a mixture of Gaussian forms, each of which consists of a prior π ∈ R 1 , a mean µ ∈ R 1024 , and a variance σ ∈ R 1024 , all are trainable variables. We optimize the GMMclassifier with Adam ( β 1 =0.9, β 2 =0.999) Kingma and Ba [2015] using a batch size of 1024 and a learning rate of 3e-5. For cross-lingual encoder, we apply the same training setting as MoCo He et al. [2020], with the momentum queue K to be 256 and temperature as 0.04. We set the momentum coefficient to 0.999 and use the Adam optimizer with a cosine decay learning rate whose peak is 5e-4.

B FLORES-200 Dataset and Geometric Analysis

B.1 FLORES-200 dataset

FLORES-200 Goyal et al. [2022], Costa-jussà et al. [2022] is a many-to-many multilingual benchmark, which consists of 3001 sentences in 204 total languages. FLORES-200 sourced all sentences from English WikiMedia and translated these English sentences into 204 languages by human translators. In particular, sentences in FLORES-200 have a much larger breadth of topics, for they are collected from three different sources: WikiNews 6 , WikiJunior 7 and WikiVoyage 8 . We summarize the basic statistics of all languages in FLORES-200 in Table 8. Similar to Tatoeba [Artetxe and Schwenk, 2019], we treat English data 'eng_Latn' as retrieval labels and report the retrieval accuracy using the same scripts as Tatoeba in XTREME [Ruder et al., 2021]. We set the 68 languages: 'bel_Cyrl, bos_Latn, hun_Latn, epo_Latn, khm_Khmr, urd_Arab, srp_Cyrl, jav_Latn, hye_Armn, gla_Latn, por_Latn, lit_Latn, bul_Cyrl, slk_Latn, mal_Mlym, ita_Latn, nno_Latn, mar_Deva, hrv_Latn, hin_Deva, kat_Geor, ben_Beng, fin_Latn, cym_Latn, oci_Latn, cat_Latn, fao_Latn, xho_Latn, spa_Latn, ron_Latn, amh_Ethi, ces_Latn, swe_Latn, nld_Latn, tat_Cyrl, kor_Hang, glg_Latn, fra_Latn, eus_Latn, ind_Latn, dan_Latn, tha_Thai, deu_Latn, tel_Telu, afr_Latn, pol_Latn, est_Latn, uig_Arab, ukr_Cyrl, uzn_Latn, heb_Hebr, kaz_Cyrl, nob_Latn, rus_Cyrl, vie_Latn, arb_Arab, zho_Hans, tuk_Latn, khk_Cyrl, jpn_Jpan, ell_Grek, isl_Latn, tam_Taml, slv_Latn, tur_Latn, mkd_Cyrl, tgl_Latn, gle_Latn' as 'Head' languages, and the remaining 135 languages (excluded English data) as 'Long-tail' ones.

B.2 Three measurements in Geometric Analysis

Invariance Measurement implies whether the semantic distributions of all languages are similar [Abend and Rappoport, 2017]. We adopt a Gaussian form N l ( µ l ,σ 2 l ) where µ l = ∑ x ∈ l γ ( x ) 3001 and σ 2 l = ∑ x ∈ l ( γ ( x ) -µ l )( γ ( x ) -µ l ) T , to approximate the semantic distribution of each language l . Further, we compute the mean averaged KL-divergence (KL-D for short) [Kullback and Leibler, 1951] among all language pairs as the overall Invariance score I v with L as the total number of languages:

̸

Canonical Form Measurement Previous works [Teller, 2000, Irwin et al., 2009] have demonstrated that a good multilingual space should distribute sentence representations based on their semantic similarities rather than language families. To measure this in quantity, we focus on Calinski-Harabasz Index (CH-I) [Cali´ nski and Harabasz, 1974], which measures how similar an object is to its own cluster compared to other clusters. We group all semantically equivalent sentences in a cluster, which leads to 3001 clusters and each obsesses 204 sentences in 204 different languages. Assuming c k and c are the centroid of cluster k and the whole dataset S , respectively. The CH-I C h is defined as:

Table 9: Overview of XRETE tasks. For tasks that have training and dev sets in other languages, we only report the number of sentences in English sets. We report the number of test examples per language.

The higher the CH-I is, the better the semantically equivalent sentences are clustered.

Isotropy Measurement A high-dimensional embedding space often demonstrates poor isotropy, and deteriorates into a low-dimensional manifold that greatly limits the expressive ability of the embedding space. We adopt principal ratio (PR) [Mu and Viswanath, 2018] to measure isotropy. Let E be the sentence representation matrix, V be the set of the eigenvectors of E , the Isotropy I so is

The closer I so is to 1 , the more isotropic the representation space is.

C XRETE: Cross-lingual Representation Transfer Evaluation

XRETE consists of 12 tasks that fall into four different categories. In our 'translate-train-all' setting, we individually fine-tune models with English training set and its translated training sets on each task. Then we report the performance of our fine-tuned model. We give an overview in Table 9 and describe the task details as follows.

XNLI The Cross-lingual Natural Language Inference corpus Conneau et al. [2018] tasks the systems with reading two sentences and determining whether one entails the other, contradicts it, or neither (neutral). A crowdsourcing-based procedure is used for collecting English examples, which are later translated into ten target languages for evaluation. Training data stays consistent with the English training data of MultiNLI Williams et al. [2018]. For evaluation, we concatenate two sentences as input and apply a new classification head to distinguish sentence relationships. We perform 'translate-train-all' evaluation, where the model is first fine-tuned on English training data and its translated data in other languages, then evaluated on test sets.

AmericasNLI (ANLI) The AmericasNLI Ebrahimi et al. [2022] is an extension of XNLI task to 10 Indigenous languages of the Americas. All of these languages are truly low-resource languages and serve as a good testbed for zero-shot cross-lingual transferability. As Spanish is more relative to the target languages, the Spanish version of XNLI subset is translated for evaluation. For training, both English and Spanish versions of MultiNLI training data are provided. We evaluate on ANLI following the same settings as in XNLI.

MultiSTS The Multilingual Semantic Textual Similarity dataset Cer et al. [2017], Reimers and Gurevych [2020] aims to assign a semantic similarity score for a pair of sentences. The MultiSTS dataset contains 7 cross-lingual sentence pairs and 3 monolingual pairs. Stanford NLI Bowman et al. [2015] and English STS Cer et al. [2017] are provided as training sets. We report the results after first fine-tuning on English training set using a Siamese network structure [Reimers and Gurevych, 2020]. Then we compute the cosine similarity between the sentence pairs and compute Spearman's rank correlation between the predicted score and gold score following Reimers and Gurevych [2020].

WMT21QETask1 (QE) The WMT21 Quality Estimation Task 1 Sentence-level Direct Assessment Specia et al. [2021] aims at testing the translation quality and this task has been applied to test the sensitivity of language models to semantic similarity Tiyajamorn et al. [2021]. The training and evaluation sets are collected from Wikipedia by translating sentences using state-of-the-art translation models to 6 languages and annotated by professional translators. In WMT21, 4 new language pairs with no training data are given to test zero-shot cross-lingual transferability. Our evaluation setting on QE is similar to that on MultiSTS, but we report Pearson's rank correlation [Kepler et al., 2019].

LAReQA The Language-Agnostic Retrieval Question Answering Roy et al. [2020] is a QA retrieval task where models are required to retrieve all relevant answers in different languages over a large multilingual pool. The dataset is constructed on XQuAD Artetxe et al. [2020] and a question is linked with answer sentences in different languages. The training set of SQuAD v1.1 Rajpurkar et al. [2016] is used to fine-tune the model to adapt to QA retrieval task. During the evaluation, sentence embeddings are also obtained by a siamese network, and we retrieve the sentences with the highest cosine similarity as predictions.

Mewsli-X Mewsli ( M ultilingual Entities in N ews , li nked) requires linking an entity mention to its entry in a language-agnostic knowledge base Botha et al. [2020]. Mewsli-X Ruder et al. [2021] features 15k mentions in 11 languages. For each mention, Mewsli-X offers entity descriptions candidate pool containing 1M candidates across 50 languages. Fine-tuning is done on a predefined set of English-only mention-entity pairs from English Wikipedia hyperlinks. Our evaluation setting is identical to LAReQA.

BUCC The second and third shared task of the workshop on Building and Using Parallel Corpora Zweigenbaum et al. [2017], Pierre Zweigenbaum and Rapp [2018] aims to examine the ability of models to detect parallel sentence pairs in a pair of monolingual corpora. The dataset provides train and test splits in 5 languages. Following XTREME Hu et al. [2020], we directly evaluate on BUCC without fine-tuning and retrieve sentences with the highest cosine similarity.

Tatoeba The goal of the Tatoeba dataset Artetxe and Schwenk [2019] is to find the nearest neighbor for each sentence in the other language according to cosine similarity and compute the error rate. The dataset consists of up to 1,000 English-aligned sentence pairs covering 122 languages. Following XTREME Hu et al. [2020], we directly evaluate on Tatoeba without fine-tuning and retrieve sentences with the highest cosine similarity.

XCOPA In the Cross-lingual Choice of Plausible Alternatives dataset Ponti et al. [2020], each XCOPA instance corresponds to a premise and two alternatives. The task is formulated as a binary classification to predict the more plausible choice. The English COPA Gordon et al. [2012] training set and Social IQa Sap et al. [2019] training data are used for fine-tuning, while the validation and test sets of English COPA are translated and re-annotated into 11 languages for evaluation.

MultiEURLEX The MultiEURLEX dataset Chalkidis et al. [2021] is a legal topic classification task that comprises 65k European Union (EU) laws in 23 official EU languages. The dataset provides multi-granular labels per document. The dataset is split into training, development, and test subsets chronologically, resulting in 55k training documents for 7 languages, and 5k each for development and test subsets in all 23 languages.

MultiARC The Multilingual Amazon Reviews Corpus Keung et al. [2020] is a large-scale collection of Amazon reviews for multilingual text classification in 6 languages. Different languages are directly gathered from the marketplaces in different countries. The goal is to predict the reviewer's rating on the 5-star scale using the test of the review as input. The data is clearly split into training (200,000 reviews), development (5,000 reviews), and test sets (5,000 reviews) for each language.

PAWS-X The Cross-lingual Paraphrase Adversaries from Word Scrambling Yang et al. [2019b] dataset requires identifying whether two sentences are paraphrases. A subset of the evaluation pairs in English PAWS Zhang et al. [2019] are human-translated into 6 typologically distinct languages for evaluation, while the English PAWS training set is used for training.

D Baseline Methods

To fairly evaluate the performance of EMMA-X, we choose XLM-R Conneau and Lample [2019] and its several derivatives as our baselines, which contain: (1) XLM-R, which applies multilingual MLM tasks as pre-training objectives on CCNet-100 corpus; (2) HICTL Wei et al. [2021], which continues training on XLM-R using hierarchical contrastive learning; and (3) INFOXLM, which is initialized with XLM-R and trains with cross-lingual contrast, multilingual MLM and TLM. Also, we compare EMMA-X to strong sentence models: (1) S-BERT [Reimers and Gurevych, 2020], which adopts multilingual knowledge distillation to extend monolingual sentence representations to multilingual. We use the strongest baseline, XLM-R ← SBERT-paraphrase , proposed in the original paper as a baseline. (2) LaBSE [Feng et al., 2022], which systematically combines several best methods, including masked language modeling, translation language modeling [Conneau and Lample, 2019], dual encoder translation ranking [Guo et al., 2018], and additive margin softmax [Yang et al., 2019a], to learn cross-lingual sentence representations. It filters 17B monolingual sentences and 6B translation pairs for sentence representation learning. We take the best model, LaBSE with Customized Vocab as our baseline. We further report the zero-shot results on Large Language Model (LLM), ChatGPT, which is trained on a wide variety of multilingual sentences and instruction tuning based on Reinforcement Learning with Human Feedback [Christiano et al., 2017, Ouyang et al., 2022].

E Prompts for ChatGPT

In this section, we show the input prompts of ChatGPT on each task in Table 13.

F Results of each Language

We show the details for tasks and all languages in Tables 10 (XNLI), 11 (AmericasNLI), 12 (MultiSTS), 14 (QE), 15 (LAReQA), 16 (Mewsli-X), 17 (XCOPA), 18 (BUCC) and 19 (PAWS-X).

G Equations and Theoretical Analysis

G.1 Details of Equations

Details of Gaussian Form N r In EMMA-X, GMM classifier is introduced to determine the semantic rank of sentence pairs. The posterior probability P G ( · ) of GMM classifier is already discussed in Eq. 5. We show the explicit calculation of Gaussian form N r ( γ ( x i ) , γ ( y k ) ) as:

where d is the dimension of hidden states of γ ( x i ) and γ ( y k ) .

Details of contrastive learning The training objective of cross-lingual encoder in EMMA-X is the ranking InfoNCE loss. We show the explicit expansion of this loss (Eq. 7) as:

where τ r represents the temperature term. As small temperature τ tends to be less tolerant to similar samples, and large τ tends to cluster similar samples together [Wang and Liu, 2021], we empirically set τ 1 < τ 2 < τ 3 < τ 4 , which remains the same as Hoffmann et al. [2022].

G.2 Theoretical Analysis

In this section, we provide detailed proof for Eq. 14 and Eq. 15. Next, we prove the feasibility of our dual supervision. GMM classifier clusters sentence pairs in terms of Euclidean distance, while cross-lingual encoder minimizes the covariance of each semantic relation rank via cosine distance. Finally, we prove that these two metrics are actually equivalent to each other in the unit hypersphere of the embedding space.

Proof of Eq. 14. We provide the derivation of Eq. 14. With the assumption that P ( x i , y k | c ∗ G = r, Θ) ∼ N r ( x i -y k | ˜ µ r , ˜ σ r ) , we have,

with n denoting the number of sentence pairs in semantic rank r . Here, we ignore the impact of ˜ σ r .

Proof of Eq. 15. As we apply dual supervision, data in the contrastive label space also follows the distribution N r ( x i -y k | ˜ µ r , ˜ σ r ) . Hence, under mild assumptions, we can get:

Based on the definition of semantic ranks, we have ˜ µ 1 < ˜ µ 2 < ... < ˜ µ N . Empirically, the number of sentence pairs in each rank n is larger than the number of semantic ranks N . Hence, it can be derived that:

Therefore, we prove that minimizing the positive terms L + CTL ( X , Y ; Θ M ) in contrastive learning is equivalent to maximizing a lower bound of the likelihood in Eq. 12.

According to the definition of semantic ranks, the approximated semantic rank c ∗ G from GMM classifier should satisfy the following restriction,

Similarly, the approximated semantic rank c ∗ M from cross-lingual encoder should satisfy the following restriction,

Next, we prove that these two restrictions are interchangeable with each other in a unit hypersphere. For simplicity, we consider only two ranks, but extending the explanation to more ranks is trivial. As the Euclidean distance is always larger than 0 , we have:

From the above analyses, we can tell that the approximated semantic rank from one module can provide a reasonable supervision signal to guide the training of the other module. Hence, all sentence pairs will be uniformly distributed according to a unified ranking semantic similarity in the embedding space.

Basic Prompt for XNLI/ANLI

Task Description: Read the following and determine the relationship between Hypothesis and Premise. Choose relation from 'contradiction', 'neutral', or 'entailment'.

Hypothesis: Yo... no puedo pensar por qué deberías hablarme así, dijo ella, con menos de lo que le había asegurado antes.

Basic Prompt for MultiSTS

Task Description: Read the following sentences and measure the real-valued meaning similarity between these two sentences. You can choose the meaning similarity score, ranging from 0 for no meaning overlap to 5 for meaning equivalence.

Sentence1: A person is on a baseball team.

Sentence2: Eine Person spielt in einem Team Basketball.

Basic Prompt for QE

Task Description: Read the Source sentence and its Translation, and estimate the quality of the Translation. You can rate the translation from 0-1 according to the perceived translation quality.

Source: În Frant , a a început stagnarea demografic˘ a de lung˘ a durat˘ a, refacerea durând o generat , ie.

Translation: In France, long-term demographic stagnation has started, restoring a generation.

Basic Prompt for XCOPA

Task Description: Read the Premise and determine which choice is the effect(or cause) of the Premise . Choose from 'Choice1' or 'Choice2'.

Choice2: Kuki kuruqa wasip kurkunta mikhurqanku.

Basic Prompt for MultiEURLEX

Task Description: Read the following sentences and determine the legal topic of the given sentence. The legal topic should choose from 'international organisations', 'social questions', 'production', 'technology and research', 'environment', 'energy', 'transport', 'law', 'finance', 'education and communications', 'trade', 'agriculture', 'forestry and fisheries', 'economics', 'agri-foodstuffs', 'EUROPEAN UNION', 'science', 'politics', 'international relations', 'industry', 'geography', 'business and competition', 'employment and working conditions'.

Sentence: NEUVOSTON ASETUS (EU) N:o 1390/2013, annettu 16 päivänä joulukuuta 2013, Euroopan unionin ja Komorien liiton kesken näiden välisessä kalastuskumppanuussopimuksessa määrätyjen kalastusmahdollisuuksien ja taloudellisen korvauksen vahvistamisesta hyväksytyn pöytäkirjan mukaisten kalastusmahdollisuuksien jakamisesta ...

Basic Prompt for MultiARC

Task Description: Read the following review and predict a 5-star scale rating (1 means the poorest experience and 5 represents excellent or outstanding performance) that can best match the review.

Review: no me llego el articulo me lo mando por correos normal sin seguimiento y nunca me llego tota un desastre

Basic Prompt for PAWS-X

Task Description: Read the following sentences and determine whether two sentences are paraphrases. Return yes or no.

Sentence1: La excepción fue entre fines de 2005 y 2009 cuando jugó en Suecia con Carlstad United BK, Serbia con FK Borac ˇ Caˇ cak y el FC Terek Grozny de Rusia.

Sentence2: La excepción se dio entre fines del 2005 y 2009, cuando jugó con Suecia en el Carlstad United BK, Serbia con el FK Borac ˇ Caˇ cak y el FC Terek Grozny de Rusia.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed EM-like multilingual pre-training algorithm using both supervised parallel data and unsupervised monolingual data to train an encoder model such that it can produce language agnostic representations for multiple languages. That is, embedding space of sentences of similar meanings will be the same across languages. The method uses an EM procedures that employs an GMM classifier, which is pre-trained with parallel data to produce a similarity rank between 2 input sentence, and then train the encoder model both with parallel data and ranks produced by GMM with unsupervised data.\nThe method achieves higher scores across many inference, similarity and retrieval tasks compared to unsupervised XLM-R.\n\nSTRENGTHS:\n* The methodology seems original and new, especially for the new rank-based classifier.\n* The paper presentation and method description is clear and understandable.\n\nWEAKNESSES:\nThere are multiple concerns with the evaluation and comparison with the method.\n\n* Despite utilize large-scale parallel data from multiple languages heavily in the pretraining stage of encoder and GMM classififer, the paper compares its method with unsupervised models like XLM-R. This make the comparison in main table 1 invalid and incomparable. This method is more semi-supervised with supervised pretraining and unsupervised finetuning stages.\n* There are some comparison with supervised models like LaBSE, but apparently the proposed method do not score well with supervised models. Moreover, the writting seems to suggest the paper is comparing unsupervised vs supervised but in fact this is supervised vs supervised.\n* XLM-R is not designed for language agnostic representation, but there are notable baselines that achieve the same language-agnostic object with simpler methodology: [LASER](https://github.com/facebookresearch/LASER) which use parallel data and [CRISS](https://arxiv.org/abs/2006.09526) which does not use parallel data. Not only they are not compared, they are also not mentioned in the paper.\n\nQUESTIONS:\nNA\n\nLIMITATIONS:\nNo limitation mentioned.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an EM-like pre-training algorithm called EMMA-X to learn cross-lingual representations. The authors unify semantic relation classification and universal representation learning into the framework. To this end, a GMM classifier and a cross-lingual encoder is jointly trained in the algorithm. Finally, the authors validate their method by conducting experiments on 12 cross-lingual tasks including inference, similarity, retrieval, and classification.\n\nSTRENGTHS:\n- The paper is well-structured and easy to follow.\n- The EM-like pre-training, which includes a semantic relation rank task, is somewhat novel.\n- The experiments are extensive, and the results generally indicate the proposed method is effective.\n- The geometric analysis section is interesting and informative, where the authors visualize the representations from four different models.\n\nWEAKNESSES:\n- Although the authors claim that EMMA-X can learn good representations with excessive non-parallel, the encoder, as far as I can see, is continue-trained on parallel sentence pairs with InfoNCE on parallel corpora for initialization. Therefore, this initialization weakens the claim.\n\n- The author does not verify the choice of a GMM classifier. At least, the author should give some simple intuition to the readers. \n\n- Semantic relation task is very important in the proposed framework. The authors claim that a binary separation is not good, but do not give further justification for the choices of N=4 semantic ranks.\n\nQUESTIONS:\n- Consider removing the comma after each equation\n- Line 69: Why do we have to use L-2 normalization? There are multiple ways to perform normalizations, L-2 is just one of them. Therefore, I would rather say $f(\\cdot)$ is a normalization, e.g., L-2.\n- Figure 1 in caption: \"{y1, and y2, y3, ...}\" -> {$y\\_1, y\\_2, y\\_3, \\cdots$}\n- In Figure 1, I assume $\\gamma^{(x)} - \\gamma^{(y_1)}$ is missing\n- The number of semantic ranks is set to 4. This sounds to me like a random selection. Did you do experiments to justify the choice?\n- The paper claims that the model can learn to learn cross-lingual universals with the aid of massive multilingual non-parallel data, but the cross-lingual encoder is continued trained on infoNCE on some parallel sentence pairs. I would assume this continue-training step is even more important than the method proposed. I would suggest the author justify the initialization by simply using the model weights of XLM-R.\n- Line 188-190: The number of sentence pairs is 3.2 billion, but what is the exact number of non-parallel sentences? That should also be mentioned in the main content of the paper.\n- Line 192: $\\mathbb{R}^1$ -> $\\mathbb{R}$\n- Table 1: the authors should also use special characters to denote which results (BUCC and Tatoeba?) are obtained by zero-shot for EMMA-X.\n- Figure 2 and Table 4: according to the results, S-BERT performs very well (even better) than the proposed method, especially on long-tail languages. In this case, what would be the benefit of applying EMMA-X to those languages instead of simply using S-BERT?\n\nLIMITATIONS:\nThe paper does not include a limitation or broader impact section, therefore not applicable.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new approach to learn cross-lingual representation learning as a universal alignment solution for any two languages without parallel data. The proposed approach resembles EM-like algorithm, which consists of a classifier to quantify the semantic similarity of two non-parallel sentences, as well as a sentence encoder that aligns the embedding representation of different languages. During the training process, the two components receive supervision from each other to reinforce the capability to recognize the accurate semantic similarity, where the paper also provides the theoretical justification to demonstrate the mutual influence being positive and effective. A new benchmark is also proposed that comprises four existing cross-lingual transfer tasks; the proposed approach is compared against other recent baselines, as well as ChatGPT. The empirical results show that new state-of-the-art performance can be obtained by this new approach with good margins.\n\nSTRENGTHS:\n- A new approach is proposed based on a novel EM-like algorithm with mutual inference and supervision by two components. Especially, the approach proposes to rank the similarity of a sentence pair, rather than optimizing the traditional binary positive/negative contrastive objective, which is indeed interesting and effective shown by the results.\n  \n- State-of-the-art performance is achieved by this approach on the new benchmark consisting different types of existing cross-lingual transfer tasks, comparing against strong baselines such as XLM-R.\n  \n- A theoretical analysis is provided, in addition to the superior empirical results to justify the effectiveness of the proposed approach.\n\nWEAKNESSES:\n- It might be over-claiming that the proposed approach operates to align the representation of different languages without parallel data, since Algorithm 1 clearly shows that parallel corpus is needed to bootstrap both the similarity classifier and the sentence encoder, and all experiments are based on this warm up that leverages parallel corpus.\n  \n- There lacks more in-depth analysis regarding the ablation of the proposed approach. In particular:\n  \n  - There could be a simplified version of this approach without iterative optimization: we can first train a similarity classifier based on the proposed mixed-up method, and generate similarity rankings of sentence pairs on non-parallel data, which can then be used to optimize the contrastive objective directly for training the encoder. In this way, we can verify whether the improvement mostly comes from the fine-grained similarity ranking, and how much this iterative optimization mechanism contributes to the final performance.\n    \n  - There could also be another experiment that downgrades the similarity ranking to binary, as if the traditional positive/negative pair. Thus, it could show again that whether this EM-like algorithm is really necessary and how much this process could improve upon the plain contrastive objective.\n\nQUESTIONS:\nWhat is the primary reason for adopting GMM as the similarity classifier? Can we use a normal classifier (e.g. using CLS on Transformers) and simply use their softmax probability for each rank?\n\nLIMITATIONS:\nIt would be good to discuss how much degradation could happen if there is not enough parallel corpus for the warm-up.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper proposes EMMA-X, an EM-like approach to pretrain multilingual models. It learns the cross-lingual representation learning task and semantic relation prediction task within EM. They propose a new benchmark, XRETE, to evaluate the experiments with 12 cross-lingual tasks. The training involves two stages: 1) pretraining using parallel data (bitext) and 2) training with EM Framework with non-parallel data. The proposed approach outperforms the baselines.

STRENGTHS:
- The proposed method outperforms other existing pre-trained models, showing the solid multilingual / cross-lingual ability.
- The proposed method is backed by a strong theoretical analysis 
- The paper proposes a new benchmark from existing datasets to evaluate the cross-linguality.

WEAKNESSES:
- The main paper does not provide enough information about the model. I would suggest moving the model details from the Appendix to the main paper.
- The benchmark does not provide the model parameters. The comparison may not be fair since the model sizes are different.

QUESTIONS:
**Suggestion**
- Figure 1 is not easily understandable. It would be great if there is a clear step-by-step procedure in the caption.

LIMITATIONS:
No. The authors can have a section to address the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

comments: SUMMARY:
This paper proposes to apply the EM framework to realize unified sentence representation learning with non-parallel multilingual data. This framework consists of two modules, a GMM classifier and a cross-language encoder, which are responsible for semantically related classification and cross-language unified representation. However, this is achieved on the premise that the parallel multilingual data initializes the models. In addition, the author conducted a theoretical analysis and forms a new benchmark, based on which the effectiveness of the framework was proved.

STRENGTHS:
The paper applies the EM algorithm to realize the optimization of the multilingual unified representation parallel-trained model under non-parallel data.

WEAKNESSES:
1.	Why choose the GMM model as the Semantic Relations model?
2.	Why form a XRETE benchmark, what are its innovations and necessity?
3.	This article introduces monolingual data from a continuous perspective, so what are its advantages compared with discrete methods(e.g. Back-translation)

QUESTIONS:
1.	Why choose the GMM model as the Semantic Relations model?
2.	Why form a XRETE benchmark, what are its innovations and necessity?
3.	This article introduces monolingual data from a continuous perspective, so what are its advantages compared with discrete methods(e.g. Back-translation)

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07