SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \underset {\Theta} {\operatorname {a r g m a x}} \sum_ {\mathbf {x} _ {i} \in \mathcal {X}} \sum_ {\mathbf {y} _ {k} \in \mathcal {Y}} \sum_ {r = 1} ^ {N} Q (r) \log \frac {P \left(\mathbf {x} _ {i} , \mathbf {y} _ {k} , r \mid \Theta\right)}{Q (r)} \approx \underset {\Theta} {\operatorname {a r g m a x}} \sum_ {\mathbf {x} _ {i} \in \mathcal {X}} \sum_ {\mathbf {y} _ {k} \in \mathcal {Y}} \sum_ {r = 1} ^ {N} \log P \left(\mathbf {x} _ {i}, \mathbf {y} _ {k} \mid c _ {\mathcal {G}} ^ {*} = r, \Theta\right) \tag {14} \\ \geq \operatorname *{argmax}_{\Theta}n(n - 1)\sum_{r = 1}^{N}\tilde{\mu}_{r}^{2}, \\ \end{array}
$$
The above derivation uses the assumption that  $P(\mathbf{x}_i, \mathbf{y}_k | c_{\mathcal{G}}^* = r, \Theta) \sim \mathcal{N}_r(\mathbf{x}_i - \mathbf{y}_k | \tilde{\mu}_r, \tilde{\sigma}_r)$ , with  $\tilde{\mu}_r$  and  $\tilde{\sigma}_r$  being the mean value and standard deviation of the Euclidean distance between sentence pairs in semantic rank  $r$ . Detailed proof of Eq. 14 is in Appendix G.
Next, we prove that minimizing the positive terms in expectation  $\mathcal{L}_{\mathbf{CTL}}(\mathcal{X},\mathcal{Y};\Theta_{\mathcal{M}})$  actually equal to maximizing a lower bound of Eq. 14. As we apply dual supervision, data in the contrastive label space also follows the distribution  $\mathcal{N}_r(\mathbf{x}_i - \mathbf{y}_k|\tilde{\mu}_r,\tilde{\sigma}_r)$ . Hence, under mild assumptions, we can get:
$$
\mathcal {L} _ {\mathbf {C T L}} ^ {+} (\mathcal {X}, \mathcal {Y}; \Theta_ {\mathcal {M}}) = n ^ {2} \sum_ {r = 1} ^ {N - 1} \tilde {\mu} _ {r} ^ {2} <   n (n - 1) \sum_ {r = 1} ^ {N} \tilde {\mu} _ {r} ^ {2} \leq \sum_ {\mathbf {x} _ {i} \in \mathcal {X}} \sum_ {\mathbf {y} _ {k} \in \mathcal {Y}} \log P (\mathbf {x} _ {i}, \mathbf {y} _ {k} | \boldsymbol {\Theta}), \tag {15}
$$
where  $\mathcal{L}_{\mathbf{CTL}}^{+}(\cdot)$  means the positive terms. In the derivation, we use the intrinsic property of semantic ranks ( $\tilde{\mu}_1 < \tilde{\mu}_2 < \ldots < \tilde{\mu}_N$ ). Detailed proof is in Appendix G. Eq. 15 demonstrates that with dual supervision, minimizing the contrastive loss can partially maximize the likelihood in Eq. 12.
To thoroughly evaluate the performance of EMMA-X, we conduct experiments on XRETE benchmark to verify the transfer ability of EMMA-X on various cross-lingual downstream tasks with strong baselines (pre-trained models: XLM-R [Conneau et al., 2020], INFOXLM [Chi et al., 2021], HICTL [Wei et al., 2021], sentence models: LaBSE [Feng et al., 2022], S-BERT [Reimers and Gurevych, 2020]) and ChatGPT in Section 5.2. See Appendices C and D for details. We further conduct geometric analysis in Section 5.3 to better interpret the cross-lingual transferability in EMMA-X.
Corpus & Model. We collect parallel corpora from CCAligned [El-Kishky et al., 2020], CCMatrix [Schwenk et al., 2021], WMT [Akhbardeh et al., 2021], and MultiUN [Ziemski et al., 2016], involving 94 languages with 3.2 billion sentence pairs. In addition, we add CC-100 [Conneau et al., 2020]
Model	Inference	Similarity	Retrieval	Classification
XNLI	ANLI	MultiSTS	QE	LAReQA	Mewsi-X	BUCC	Tatoeba	XCOPA	MultiEURLEX	MultiARC	PAWS-X
Metrics	Acc. (†)	Acc. (†)	Spearman (†)	Pearson (†)	mAP@20 (†)	mAP@20 (†)	F1 (†)	Acc. (†)	Acc. (†)	Acc. (†)	MAE (↓)	Acc. (†)
MBERT*	75.1a	-	55.8s	-	21.6d	38.6d	56.7a	39.0a	56.1d	67.4s	48.2s	81.9a
XLM*	77.8b	-	-	-	-	-	56.8a	32.6a	-	-	-	80.9a
XLM-R*	83.6b	49.12s	61.5s	58.7s	40.7d	45.7d	66.0a	57.7a	69.2d	66.6a	-	88.9a
HICTL*	84.7c	-	-	-	-	-	77.6c	69.1c	-	-	-	92.8c
CHATGPT†	60.9	41.7	68.6	60.9	-	-	-	-	74.2	68.7	40.2	64.2
Ours re-implementation, translate-train-all (models are trained on English training data and on its data translated to the target language)
XLM-R‡	82.8	48.48	65.9	63.2	40.3	48.6	67.9	59.1	71.2	66.9	44.9	90.1
INFOXLM‡	84.2	49.10	82.2	64.1	44.9	57.1	77.4	66.2	74.6	67.7	36.2	93.0
HICTL‡	85.1	49.02	81.6	64.9	46.1	54.8	77.6	65.8	74.8	68.3	38.2	92.8
EMMA-X	88.1	50.21	87.3	67.2	50.6	59.6	87.1	82.5	78.2	71.4	32.7	94.2
Table 1: Results on the XRETE benchmark. * denotes the results from previous literature,  ${}^{a}\mathrm{{Hu}}$  et al. [2020]  ${}^{b}$  Conneau et al. [2020]  ${}^{c}$  Wei et al. [2021]  ${}^{d}$  Ruder et al. [2021].  ${}^{s}$  denotes the results from the original paper.  ${}^{ \ddagger  }$  denotes results from our re-trained models with the same model size and training corpora as EMMA-X.  ${}^{ \dagger  }$  denotes the zero-shot performance. Refer to Appendix F for greater details on each task and language.
Model	Similarity	Retrieval
MultiSTS	QE	LAReQA	Mewsi-X	BUCC	Tatoeba
Metrics	Spearman (↑)	Pearson (↑)	mAP@20 (↑)	mAP@20 (↑)	F1 (↑)	Acc. (↑)
S-BERT	84.0	39.3	31.8	14.4	88.5	68.6
LaBSE	74.4	31.6	12.8	11.2	93.2	83.6
INFOXLM	55.2	49.4	16.9	23.9	77.4	66.2
XLM-R	25.0	10.4	16.1	11.7	67.9	59.1
EMMA-X	62.9	54.7	19.4	29.4	87.1	82.5
Table 2: Zero-shot Results on Similarity and Retrieved tasks. Results of LaBSE use Customized VOCab setting. Results of S-BERT are from XLM-r←SBERT-paraphrases. The bold font denotes the best 2 results.
Model	FLORES-200	Tatoeba
Head	Long-tail	Head	Long-tail
Metrics	Acc. (↑)	Acc. (↑)	Acc. (↑)	Acc. (↑)
S-BERT	87.5	51.6	91.4	57.8
LaBSE	99.9	82.5	95.7	77.9
INFOXLM	83.4	53.8	87.8	56.1
XLM-R	68.2	45.3	66.3	55.7
EMMA-X	94.5	84.2	91.9	78.1
Table 3: Retrieval results on FLORES-200 and Tatoeba in xx  $\rightarrow$  En direction. The bold font denotes the best results.
as the large-scale monolingual corpus with about 800 billion sentences that covers 94 languages. The cross-lingual encoder starts from the well-trained XLM-R large model [Conneau et al., 2020]. The GMM classifier is implemented as a mixture of Gaussian forms, each of which consists of a prior  $\pi \in \mathbb{R}^1$ , a mean  $\mu \in \mathbb{R}^{1024}$  and a standard deviation  $\sigma \in \mathbb{R}^{1024}$ , all are trainable variables. We set the total semantic ranks as  $N = 4$ . The statistics of all data and hyper-parameters are shown in Appendix A.
5.2 XRETE Evaluation
XRETE includes 12 cross-lingual tasks divided into 4 different categories. We report the "translate-train" performance in Table 1 on most tasks but zero-shot performance on BUCC and Tatoeba following [Ruder et al., 2021]. Table 2 presents zero-shot comparisons with sentence models.
Comparisons with Pre-trained Models. In Table 1, EMMA-X consistently outperforms all baseline models (XLM-R [Conneau et al., 2020], HICTL [Wei et al., 2021] and INFOXLM [Chi et al., 2021]) with  $7.97\%$  improvements on average. Specifically, EMMA-X achieves  $88.1\%$  accuracy on XNLI [Conneau et al., 2018] and  $50.21\%$  accuracy on ANLI [Ebrahimmi et al., 2022] with up to  $6.4\%$  improvements than baselines. On the MultiSTS [Reimers and Gurevych, 2020] task, EMMA-X achieves an  $87.3$  correlation score, outperforming several strong baselines by  $5.1 \sim 21.4$ , and even achieves comparable performance in the cross-lingual and the monolingual settings (see Appendix F for language-specific results). Furthermore, EMMA-X obtains a  $67.2$  Pearson score on QE [Specia et al., 2021] task, which is comparable to the winner on the leaderboard without any specific finetuning techniques. As for sentence retrieval, EMMA-X consistently outperforms previous strong baselines among all 4 tasks [Ruder et al., 2021], and demonstrates  $2.6\% \sim 39.6\%$  improvements over these baselines. Similar results can be found in sentence classification tasks. EMMA-X obtains an  $81.3\%$  top-1 accuracy averaged on XCOPA [Ponti et al., 2020], MultiEURLEX [Chalkidis et al., 2021] and PAWS-X [Yang et al., 2019b] tasks, outperforming XLM-R, INFOXLM and HICTL by  $7.0\%$ ,  $3.9\%$  and  $3.5\%$  improvements, respectively. On MultiARC [Keung et al., 2020] task, EMMA-X shows the lowest error rates among all models. The consistent improvements on all tasks reveal that EMMA-X can obtain better universal representations for different natural languages with various
<sup>3</sup>https://www.statmt.org/wmt21/quality-estimation-task_results.html
topics and domains. We further conduct experiments with ChatGPT on XRETE tasks without 4 Retrieval tasks. We list the prompts for each task in Appendix E. ChatGPT's zero-shot performance is worse than fine-tuned pre-trained models and the performance gap is very large on most tasks.
Zero-shot comparisons with sentence models. Compared with XLM-R and INFOXLM, which adopt the same amount of training data as EMMA-X, EMMA-X consistently outperforms XLM-R and INFOXLM by  $73.2\%$  and  $25.1\%$  on average, as shown in Table 2. The results further prove the effectiveness of our pre-training technique. Through the reciprocation between GMM classifier and cross-lingual encoder, EMMA-X can generate reliable semantic rank for multilingual non-parallel corpora, which can provide more supervision signals than previous baselines. EMMA-X even achieves comparable results with strong supervised methods: LaBSE [Feng et al., 2022] and S-BERT [Reimers and Gurevych, 2020], which both trained on supervised data. LaBSE is trained on a fine-filtered bilingual corpus with 6B translation pairs (2 times larger than EMMA-X), while S-BERT is distilled from a S-BERT model fine-tuned on English NLI, STS datasets, and 50M English paraphrase pairs. Compared with these two methods, EMMA-X can achieve the best results on QE and Mewsli-X by outperforming S-BERT and LaBSE by  $71.7\%$  and  $117.8\%$  averaged. EMMA-X performs worse than these baselines on MultiSTS and BUCC, for these two tasks only contain rich-resource languages, which already have great deal of parallel data.
Model	Head Langs.	Long-tail Langs.	All Langs.
Invariance	Canonical Form	Isotropy	Invariance	Canonical Form	Isotropy	Invariance	Canonical Form	Isotropy
Metrics	KL-D (↓)	CH-I (↑)	PR (↑)	KL-D (↓)	CH-I (↑)	PR (↑)	KL-D (↓)	CH-I (↑)	PR (↑)
XLM-R (cls)	0.7356	30.19	0.3681	2.0042	7.96	0.3686	1.6501	20.80	0.3683
INFOXLM (cls)	0.4491	38.82	0.4478	1.8555	13.02	0.4406	1.4747	31.51	0.4665
S-BERT (mean)	0.1115	108.22	0.4519	1.3112	44.32	0.4414	0.9782	102.36	0.4467
EMMA-X (cls)	0.3603	43.52	0.5318	0.3963	46.53	0.5732	1.1904	48.70	0.5918
Table 4: Comparisons with existing methods on FLORES dataset for geometric analysis. "cls" and "mean" represent different pooling strategies to obtain sentence representations.
Performance on Long-tail Languages. One goal of EMMA-X is to learn universal sentence representations accommodated for more languages. To better prove this, we report the retrieval accuracy on FLORES-200 [Costa-jussà et al., 2022] and Tatoeba [Artetxe and Schwenk, 2019]. We reformulate FLORES-200, which contains manual translations in 204 languages (totaling 3001 sentences) to perform retrieval tasks in the same way as Tatoeba and report the performance in terms of language data scale in Table 3. Details about the separation of languages and FLORES-200 are shown in Appendices A and B. On head languages, EMMA-X performs worse than LaBSE by about  $4.6\%$  but outperforms S-BERT by  $3.5\%$ . On the long-tail languages, EMMA-X can surpass S-BERT by  $4.3\%$  averaged on two tasks. EMMA-X can even exceed the strongest LaBSE by  $2.1\%$  on FLORES. One reason for the superior results on long-tail languages is that for those long-tail languages that have only bi-lingual parallel data with rich-resource languages (often English), EMMA-X can provide multi-lingual semantic relation signals for them with arbitrary languages through dual supervision.
5.3 Geometric Analysis
To interpret the advantages of EMMA-X, we evaluate the geometric characteristics of it on FLORES-200 dataset [Costa-jussà et al., 2022] without any fine-tuning. The criteria of three requirements are Invariance, measured with KL-divergence (KL-D) [Kullback and Leibler, 1951], Canonical Form, measured with Calinski-Harabasz Index (CH-I) [Calinski and Harabasz, 1974] and Isotropy, measured with principal ratio (PR) [Mu and Viswanath, 2018]. Details of them are shown in Appendix B. We report the numerical results in Table 4 and visualize each characteristic in Figure 2.
Invariance & Canonical Form aim to measure how languages are aligned in the representation space. If the sentence representations are universal, then sentences in different languages should follow a similar distribution, which is measured by invariance in KL-divergence. Similarly, canonical form measures how well semantic equivalent sentences are grouped into one cluster, with a clustering metric (CH-I). In Table 4, S-BERT outperforms other baselines in "Invariance" and "Canonical Form" on head languages. However, EMMA-X shows better performance on long-tail languages in these two metrics, which is consistent with Table 3. Figure 2 presents similar results. Among the 20 languages we randomly sampled from FLORES, EMMA-X can align 17 languages as shown in Figure 2d, with "xh, eo, ur" as outliers. In Figure 2e, 2g, 2f and 2h, different colors represent different languages. So
(c) S-BERT (Invariance)
(a) XLM-R (Invariance)
(b) INFOXLM (Invariance)
(d) EMMA-X (Invariance)
(e) XLM-R (Canonical)
(i) XLM-R (Isotropy)
Figure 2: Visualization of representations from EMMA-X, XLM-R, INFOXLM and S-BERT. We use t-SNE [van der Maaten and Hinton, 2008] to visualize each geometric metrics.
(f) INFOXLM (Canonical)
(j) INFOXLM (Isotropy)
(g) S-BERT (Canonical)
(k) S-BERT (Isotropy)
(h) EMMA-X (Canonical)
(1) EMMA-X (Isotropy)
a cluster with only one color means this language is isolated from other languages and not aligned well in representation space. Figure 2h shows that EMMA-X performs well in most languages.
Isotropy measures how expressive a representation space is since high-dimensional representation space can easily deteriorate into a low-dimensional manifold. From Table 4, EMMA-X achieves the best results in isotropy. The Isotropy of S-BERT (k) is different from other methods. We conjecture the reason is that S-BERT removes MLM tasks during fine-tuning, so token embeddings will only receive sentence-level supervision, resulting in identical sentence representations for different languages. The abnormal results observed on the high KL-divergence, as depicted in Table 4 and Figure 2c, can be attributed to the representation space for S-BERT deteriorating into a low-dimensional manifold (low isotropy score in Table 4 and Figure 2k), and different languages are not distributed uniformly across the whole representation space, which limits the expressive ability.
Model\Language	af	ar	bg	bn	de	el	es	et	eu	fa	fi	fr	he	hi	hu	id	it	ja
+ Phase 1	86.7	84.9	87.8	80.0	96.6	89.8	91.9	90.3	81.5	84.3	88.2	88.3	85.2	84.9	89.1	93.7	88.7	86.2
+ fixed GMM	86.2	84.3	89.6	79.4	92.6	87.5	93.9	92.6	84.5	87.9	92.1	91.4	84.4	92.6	87.1	90.9	87.1	82.4
EMMA-X	93.9	90.2	91.4	92.1	95.9	92.0	95.7	94.3	96.4	96.0	94.7	92.1	90.3	98.5	92.0	93.4	91.6	91.3
Model\Language	jv	ka	kk	ko	ml	mr	nl	pt	ru	sw	ta	te	th	tl	tr	ur	vi	zh
+ Phase 1	22.6	88.3	61.6	85.0	93.9	82.0	94.7	92.1	88.7	70.5	70.9	81.5	84.9	59.2	88.2	85.8	97.1	91.7
+ fixed GMM	37.9	91.8	67.1	87.6	91.8	85.3	93.1	88.9	90.8	75.4	72.4	82.3	86.7	68.1	91.6	89.3	94.9	88.8
EMMA-X	56.7	93.2	86.0	91.4	96.7	94.8	93.6	92.7	91.6	87.2	87.8	95.4	96.9	85.7	96.5	93.4	96.1	92.7
Table 5: Ablation Study about EMMA-X on 36 languages in Tatoeba. We mark the 14 long-tail languages with color red. We highlight the best results with bold font and the second-best results with underlines.
The primary goal of EMMA-X is to acquire universal semantic representations for a multitude of languages. For the limited number of parallel data, EMMA-X strives to leverage non-parallel data to extend language coverage for universal semantic representations and to enhance representation performance for those languages. To provide additional evidence supporting our claim, we propose an ablation experiment in Table 5. The results in Table 5 demonstrate that EMMA-X significantly
Setting	XNLI Acc. (†)	ANLI Acc. (†)	MultiSTS Spearman (†)	QE Pearson (†)	LAReQA mAP@20 (†)	Mewsli-X mAP@20 (†)	BUCC F1 (†)	Tatoeba Acc. (†)	XCOPA Acc. (†)	MultiEURLEX Acc. (†)	MultiARC MAE (↓)	PAWS-X Acc. (†)
(FCN, N=4)	86.4	46.63	87.0	66.1	46.6	54.2	81.1	78.2	76.2	68.5	34.9	90.8
(GMM, N=2)	85.2	49.52	84.7	67.3	47.8	56.6	85.6	82.3	77.0	70.1	30.6	90.9
(GMM, N=4)‡	88.1	50.21	87.3	67.2	50.6	59.6	87.1	82.5	78.2	71.4	32.7	94.2
(GMM, N=8)	87.9	50.58	86.3	67.4	49.2	59.5	85.7	82.5	78.8	71.3	32.5	95.1
Table 6: Effect of different settings on the XRETE benchmark.  $\ddagger$  means results with default settings (i.e., GMM classifier and  $N = 4$ ). "FCN" means the semantic classifier is developed as fully connected networks.
improves the retrieval accuracy by 8.1 when compared to solely using parallel data. Moreover, on 16 long-tail languages, the retrieval accuracy further increases from 76.2 to 90.7, representing a substantial improvement of  $19.0\%$ . These findings support our claim that EMMA-X can indeed learn effective universal semantic representations from an abundance of non-parallel data.
5.5 Semantic Rank and GMM Analysis
We assessed EMMA-X's performance on XRETE benchmark with various settings in Table 6. The evaluation focuses on two aspects: the classifier type and the value of semantic similarity rank.
Type of the classifier. Without direct supervision, we propose each semantic rank follow a Gaussian distribution. Comparing "GMM" with "FCN" in Table 6, the performance drops from 70.7 to 68.1. This decline is attributed to the handling of outliers: the "GMM" handles them better due to its soft assignment approach, giving outliers low probabilities and thereby minimizing their influence.
Value of Semantic Similarity Rank. Different semantic similarity ranks were evaluated in Table 6. The peak performance at  $N = 4$  implies that a refined similarity rank ensures a balanced distribution of semantic ranks, which helps alleviate data imbalance. However, increasing the number of semantic ranks complicates the learning process for both "GMM" and the cross-lingual encoder.  $N = 4$  strikes a balance between data distribution and learning complexity.
Cross-lingual Representation Pre-training In recent years, various studies [Devlin et al., 2019, Conneau et al., 2020] have shifted the monolingual pre-training procedure to multilingual scenarios. Most of them often rely heavily on parallel data to learn cross-lingual sentence representations, with several improved techniques, such as language modeling [Conneau and Lample, 2019] and contrastive learning [Chi et al., 2021, Hu et al., 2021, Wei et al., 2021]. Recently, some endeavors incorporate monolingual data into parallel corpora, by translating monolingual corpora into pseudo parallel corpora [Kvapilíková et al., 2020, Ouyang et al., 2021], or computing the semantic similarity in monolingual corpora off-the-shelf and using it as supervision signals [Goswami et al., 2021].
Contrastive Learning has become a popular paradigm in NLP. Besides constructing the positives and negatives through parallel corpora [Zhang et al., 2021] or other labeled data [Gunel et al., 2021, Ni et al., 2022], researchers also adopt self-supervised methods, which build positives and negatives by corrupting sentences [Gao et al., 2021, Chuang et al., 2022] or data augmentation methods [Zhang et al., 2022a, Wu et al., 2022]. Another line of research improves the quality of negatives by preserving a memory queue [Yang et al., 2021, Wang et al., 2021] or generating high-quality negatives [Zhang et al., 2022c, Wei et al., 2022].
In this paper, we study the problem of learning cross-lingual universal representations. Major contributions are fourfold: 1) We propose a novel paradigm EMMA-X, which can make full use of massive monolingual data to learn universities for any two languages. 2) We summarize three requirements for the universal representation space among all languages and verify the superiority of EMMA-X towards strong baselines. 3) To incentivize the research of cross-lingual universal representation learning, we form a novel benchmark (XRETE) with 12 cross-lingual tasks fully depending on sentence-level representations. 4) Experiments on XRETE demonstrate that EMMA-X achieved state-of-the-art results over strong baselines.
We thank the anonymous reviewers for their constructive comments. This work is supported by the National Natural Science Foundation of China (Grant No.U21B2009). This research is also supported by the Strategic Priority Research Program of the Chinese Academy of Science, Grant No.XDC02030400.
Michał Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. The United Nations parallel corpus v1.0. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 3530-3534, Portoroz, Slovenia, May 2016. European Language Resources Association (ELRA). URL https://aclanthology.org/L16-1561.
Pierre Zweigenbaum, Serge Sharoff, and Reinhard Rapp. Overview of the second BUCC shared task: Spotting parallel sentences in comparable corpora. In Proceedings of the 10th Workshop on Building and Using Comparable Corpora, pages 60-67, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10.18653/v1/W17-2512. URL https://www.aclweb.org/anthology/W17-2512.
A Training Corpora and Hyper-parameters
A.1 Training Corpora
As for monolingual data, we follow Conneau et al. [2020] to build a Common-Crawl Corpus using the CCNet [Wenzek et al., 2020] tool<sup>4</sup>, which is widely used in the literature Huang et al. [2019], Luo et al. [2021], Chi et al. [2021], Wei et al. [2021]. Further, we collect parallel corpora from CCAligned El-Kishky et al. [2020], CCMatrix Schwenk et al. [2021], WMT Akhbardeh et al. [2021], and MultiUN Ziemski et al. [2016], involving 94 languages with more than 4.8 billion sentence pairs. We use the OpusFilter<sup>5</sup> tool to remove noisy bittexts, which results in 3.2 billion sentence pairs. Table 7 shows the statistics for both monolingual and parallel data. We apply subword tokenization directly on raw text data using Sentence Piece Model Kudo and Richardson [2018] without any additional preprocessing. To better support our motivation that EMMA-X can cover more languages than previous cross-lingual sentence representations, we divide Tatoeba Artetxe and Schwenk [2019] into two subsets: "Head", containing languages usually covered in previous methods, and "Long-tail", with other languages. We treat the 36 languages contained in XTREME Ruder et al. [2021] as head languages, which are: "ar, he, vi, id, jv, tl, eu, ml, ta, te, af, nl, en, de, el, bn, hi, mr, ur, fa, fr, it, pt, es, bg, ru, ja, ka, ko, th, sw, zh, kk, tr, et, fi, hu, az, lt, pl, uk, ro". The remaining 76 languages in Tatoeba are treated as long-tail ones.
The parameters of EMMA-X are first initialized with XLM-R, with 24 layers of Transformer [Vaswani et al., 2017] encoder, 1024 hidden states, and 16 attention heads. We set the total semantic ranks as 4. The GMM classifier is implemented as a mixture of Gaussian forms, each of which consists of a prior  $\pi \in \mathbb{R}^1$ , a mean  $\mu \in \mathbb{R}^{1024}$ , and a variance  $\sigma \in \mathbb{R}^{1024}$ , all are trainable variables. We optimize the GMM classifier with Adam ( $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.999$ ) Kingma and Ba [2015] using a batch size of 1024 and a learning rate of 3e-5. For cross-lingual encoder, we apply the same training setting as MoCo He et al. [2020], with the momentum queue  $K$  to be 256 and temperature as 0.04. We set the momentum coefficient to 0.999 and use the Adam optimizer with a cosine decay learning rate whose peak is 5e-4.
B FLORES-200 Dataset and Geometric Analysis
B.1 FLORES-200 dataset