SlowGuess/NeurIPS_2023_OCR huggingface dataset:
FLORES-200 Goyal et al. [2022], Costa-jussà et al. [2022] is a many-to-many multilingual benchmark, which consists of 3001 sentences in 204 total languages. FLORES-200 sourced all sentences from English WikiMedia and translated these English sentences into 204 languages by human translators. In particular, sentences in FLORES-200 have a much larger breadth of topics, for they are collected from three different sources: WikiNews $^6$ , WikiJunior $^7$  and WikiVoyage $^8$ . We summarize the basic statistics of all languages in FLORES-200 in Table 8. Similar to Tatoeba [Artetxe and Schwenk, 2019], we treat English data "eng_Latn" as retrieval labels and report the retrieval accuracy using the same scripts as Tatoeba in XTREME [Ruder et al., 2021]. We set the 68 languages: "bel_Cyrl, bos_Latn, hun_Latn, epo_Latn, khm_Khmr, urd_Arab, srp_Cyrl, jav_Latn, hye_Armn, gla_Latn, por_Latn, lit_Latn, bul_Cyrl, slk_Latn, mal_Mlym, ita_Latn, nno_Latn, mar_Deva, hrv_Latn, hin_Deva, kat_Geor, ben_Beng, fin_Latn, cym_Latn, oci_Latn, cat_Latn, fao_Latn, xho_Latn, spa_Latn, ron_Latn, amh_Ethi, ces_Latn, swe_Latn, nth_Latn, tat_Cyrl, kor_Hang, glg_Latn, fra_Latn, eus_Latn, ind_Latn, dan_Latn, tha_Thai, deu_Latn, tel_Telu, afr_Latn, pol_Latn, est_Latn, uig_Arab, ukr_Cyrl, uzn_Latn, heb_Hebr, kaz_Cyrl, nob_Latn, rus_Cyrl, vie_Latn, arb_Arab, zho_Hans, suk_Latn, khk_Cyrl, jpn_Jpan, ell_Grek, isl_Latn, tam_Taml, svL_Latn, tur_Latn, mkd_Cyrl,tgl_Latn, gle_Latn" as "Head" languages, and the remaining 135 languages (excluded English data) as "Long-tail" ones.
<sup>4</sup>https://github.com/facebookresearch/cc_net
<sup>5</sup>https://github.com/Helsinki-NLP/OpusFilter
$^{6}$ https://en.wikipedia.org/wiki/MainPage
<sup>7</sup>https://en.wikipedia.org/wiki/WikiJunior
<sup>8</sup>https://en.wikipedia.org/wiki/Main_Page
Code	Size (GB)	Sent. (M)	Code	Size (GB)	Sent. (M)	Code	Size (GB)	Sent. (M)	Code	Size (GB)	Sent. (M)	Code	Size (GB)	Sent. (M)
af	1.3	-	et	6.1	22.3	ja	24.2	89.2	mt	0.2	-	sq	3.0	-
am	0.7	-	eu	2.0	0.81	jv	0.2	-	my	0.9	-	sr	5.1	-
ar	20.4	72.3	fa	21.6	7.5	ka	3.4	2.0	ne	2.6	-	su	0.1	-
as	0.1	-	fi	19.2	92.8	kk	2.6	2.8	nl	15.8	66.0	sv	10.8	74.2
az	3.6	0.82	fr	46.5	331.5	km	1.0	0.84	no	3.7	-	sw	1.6	1.7
be	3.5	0.51	fy	0.2	0.13	kn	1.2	-	om	0.1	-	ta	8.2	2.79
bg	22.6	47.2	ga	0.5	-	ko	17.2	79.3	or	0.6	-	te	2.6	-
bn	7.9	7.52	gd	0.1	0.05	ku	0.4	-	pa	0.8	-	th	14.7	13.1
br	0.1	-	gl	2.9	0.77	ky	1.2	-	pl	16.8	79.7	tl	0.8	-
bs	0.1	-	gu	0.3	-	la	2.5	-	ps	0.7	-	tr	17.3	93.8
ca	10.1	14.9	ha	0.3	-	lo	0.6	-	pt	15.9	247.6	ug	0.4	-
cs	16.3	108.4	he	6.7	47.1	lt	7.2	11.0	ro	8.6	60.4	uk	9.1	0.78
cy	0.8	-	hi	20.2	3.2	lv	6.4	0.37	ru	48.1	134.9	ur	5.0	1.15
da	15.2	8.0	hr	5.4	-	mg	0.2	-	sa	0.3	-	uz	0.7	-
de	46.3	283.4	hu	9.5	55.2	mk	1.9	-	sd	0.4	-	vi	44.6	15.3
el	29.3	95.1	hy	5.5	1.7	ml	4.3	1.07	si	2.1	0.60	xh	0.1	-
en	49.7	-	id	10.6	184.6	mn	1.7	0.19	sk	4.9	-	yi	0.3	-
eo	0.9	0.18	is	1.3	-	mr	1.3	-	sl	2.8	9.8	zh	36.8	379.4
es	44.6	279.6	it	19.8	179.3	ms	3.2	2.1	so	0.4	-	-	-	-
Table 7: The statistics of CC-100 and the collected parallel corpora used for training. We report the list of 94 languages and include the size of the monolingual data (in GiB) and the number of sentence pairs (in Millions, which denotes the number of sentence pairs between the specific language and English) in parallel corpora for each language. “-” means the number of sentence pairs is less than 0.1 million.
Number of Sentences	3001
Average Words per Sentence	21
Number of Articles	842
Average Number of Sentences per Article	3.5
Domain	Articles	Sentences
WikiNews	309	993
WikiJunior	284	1006
WikiVoyage	249	1002
Sub-Topic	Articles	Sentences
Crime	155	313
Disasters	27	65
Entertainment	28	68
Geography	36	86
Health	27	67
Nature	17	45
Politics	171	341
Science	154	325
Sports	154	162
Travel	505	1529
Table 8: Basic Statistics of FLORES-200.
B.2 Three measurements in Geometric Analysis
Invariance Measurement implies whether the semantic distributions of all languages are similar [Abend and Rappoport, 2017]. We adopt a Gaussian form  $\mathcal{N}_l(\mu_l,\sigma_l^2)$  where  $\mu_{l} = \frac{\sum_{\mathbf{x}\in l}\gamma^{(\mathbf{x})}}{3001}$  and  $\sigma_l^2 = \sum_{\mathbf{x}\in l}(\gamma^{(\mathbf{x})} - \mu_l)(\gamma^{(\mathbf{x})} - \mu_l)^T$  to approximate the semantic distribution of each language  $l$ . Further, we compute the mean averaged KL-divergence (KL-D for short) [Kullback and Leibler, 1951] among all language pairs as the overall Invariance score  $\pmb{\pi}_{v}$  with  $L$  as the total number of languages:
$$
\mathcal {I} _ {v} = \frac {1}{L \times (L - 1)} \sum_ {l _ {1} \neq l _ {2}} \frac {\mathbf {K L} \left(\mathcal {N} _ {l _ {1}} \mid \mid \mathcal {N} _ {l _ {2}}\right) + \mathbf {K L} \left(\mathcal {N} _ {l _ {2}} \mid \mid \mathcal {N} _ {l _ {1}}\right)}{2}. \tag {16}
$$
Canonical Form Measurement Previous works [Teller, 2000, Irwin et al., 2009] have demonstrated that a good multilingual space should distribute sentence representations based on their semantic similarities rather than language families. To measure this in quantity, we focus on Calinski-Harabasz Index (CH-I) [Calinski and Harabasz, 1974], which measures how similar an object is to its own cluster compared to other clusters. We group all semantically equivalent sentences in a cluster, which leads to 3001 clusters and each obsesses 204 sentences in 204 different languages. Assuming  $c_k$  and  $c$  are the centroid of cluster  $k$  and the whole dataset  $s$ , respectively. The CH-I  $c_h$  is defined as:
$$
\mathcal {C} _ {h} = \left[ 2 0 4 \times \sum_ {k = 1} ^ {K} \| c _ {k} - c \| ^ {2} \right] / \left[ \sum_ {k = 1} ^ {K} \sum_ {s \in \mathcal {S}} \| s - c _ {k} \| ^ {2} \right]. \tag {17}
$$
Task category	Task	Train	Dev	Test	Lang.	Metric	Domain
Inference	AmericasNLI	392,702	222-743	738-750	10	Accuracy	Misc.
XNLI	392,702	2,490	5,010	15	Accuracy	Misc.
Semantic Similarity	Multi-STS	550,152+5,749	10,000+1,500	250	7	Spearman	Misc.
WMT21QETask1	7,000	1,000	1,000	7 (11)	Pearson	News
Sentence Retrieval	LAReQA	87,599	10,579	1,190	11	mAP@20	Wikipedia
Mewsli-X	116,093	10,252	428-1,482	11 (50)	mAP@20	News
BUCC	-	-	1,896-14,330	5	F1	Wiki/News
Tatoeba	-	-	1,000	36 (122)	Accuracy	Misc.
Classification	XCOPA	33,410+400	100	500	11	Accuracy	Misc.
MultiEURLEX	55,000	5,000	5,000	23	Accuracy	Legal
MultiARC	200,000	5,000	5,000	6	MAE	Reviews
PAWS-X	49,401	2,000	2,000	7	Accuracy	Wiki/Quora
Table 9: Overview of XRETE tasks. For tasks that have training and dev sets in other languages, we only report the number of sentences in English sets. We report the number of test examples per language.
The higher the CH-I is, the better the semantically equivalent sentences are clustered.
Isotropy Measurement A high-dimensional embedding space often demonstrates poor isotropy, and deteriorates into a low-dimensional manifold that greatly limits the expressive ability of the embedding space. We adopt principal ratio (PR) [Mu and Viswanath, 2018] to measure isotropy. Let  $\varepsilon$  be the sentence representation matrix,  $\nu$  be the set of the eigenvectors of  $\varepsilon$ , the Isotropy  $\tau_{so}$  is
$$
\mathcal {I} _ {s o} = \min  _ {v \in \mathcal {V}} \sum_ {e \in \mathcal {E}} \exp (v ^ {\mathrm {T}} e) / \max  _ {v \in \mathcal {V}} \sum_ {e \in \mathcal {E}} \exp (v ^ {\mathrm {T}} e). \tag {18}
$$
The closer  $\pmb{\tau}_{so}$  is to 1, the more isotropic the representation space is.
C XRETE: Cross-lingual Representation Transfer Evaluation
XRETE consists of 12 tasks that fall into four different categories. In our "translate-train-all" setting, we individually fine-tune models with English training set and its translated training sets on each task. Then we report the performance of our fine-tuned model. We give an overview in Table 9 and describe the task details as follows.
XNLI The Cross-lingual Natural Language Inference corpus Conneau et al. [2018] tasks the systems with reading two sentences and determining whether one entails the other, contradicts it, or neither (neutral). A crowdsourcing-based procedure is used for collecting English examples, which are later translated into ten target languages for evaluation. Training data stays consistent with the English training data of MultiNLI Williams et al. [2018]. For evaluation, we concatenate two sentences as input and apply a new classification head to distinguish sentence relationships. We perform "translate-train-all" evaluation, where the model is first fine-tuned on English training data and its translated data in other languages, then evaluated on test sets.
AmericasNLI (ANLI) The AmericasNLI Ebrahimi et al. [2022] is an extension of XNLI task to 10 Indigenous languages of the Americas. All of these languages are truly low-resource languages and serve as a good testbed for zero-shot cross-lingual transferability. As Spanish is more relative to the target languages, the Spanish version of XNLI subset is translated for evaluation. For training, both English and Spanish versions of MultiNLI training data are provided. We evaluate on ANLI following the same settings as in XNLI.
MultiSTS The Multilingual Semantic Textual Similarity dataset Cer et al. [2017], Reimers and Gurevych [2020] aims to assign a semantic similarity score for a pair of sentences. The MultiSTS dataset contains 7 cross-lingual sentence pairs and 3 monolingual pairs. Stanford NLI Bowman et al. [2015] and English STS Cer et al. [2017] are provided as training sets. We report the results after first fine-tuning on English training set using a Siamese network structure [Reimers and Gurevych, 2020]. Then we compute the cosine similarity between the sentence pairs and compute Spearman's rank correlation between the predicted score and gold score following Reimers and Gurevych [2020].
WMT21QETask1 (QE) The WMT21 Quality Estimation Task 1 Sentence-level Direct Assessment Specia et al. [2021] aims at testing the translation quality and this task has been applied to test the sensitivity of language models to semantic similarity Tiyajamorn et al. [2021]. The training and evaluation sets are collected from Wikipedia by translating sentences using state-of-the-art translation models to 6 languages and annotated by professional translators. In WMT21, 4 new language pairs with no training data are given to test zero-shot cross-lingual transferability. Our evaluation setting on QE is similar to that on MultiSTS, but we report Pearson's rank correlation [Kepler et al., 2019].
LAReQA The Language-Agnostic Retrieval Question Answering Roy et al. [2020] is a QA retrieval task where models are required to retrieve all relevant answers in different languages over a large multilingual pool. The dataset is constructed on XQuAD Artetxe et al. [2020] and a question is linked with answer sentences in different languages. The training set of SQuAD v1.1 Rajpurkar et al. [2016] is used to fine-tune the model to adapt to QA retrieval task. During the evaluation, sentence embeddings are also obtained by a siamese network, and we retrieve the sentences with the highest cosine similarity as predictions.
Mewsli-X Mewsli (Multilingual Entities in News, linked) requires linking an entity mention to its entry in a language-agnostic knowledge base Botha et al. [2020]. Mewsli-X Ruder et al. [2021] features 15k mentions in 11 languages. For each mention, Mewsli-X offers entity descriptions candidate pool containing 1M candidates across 50 languages. Fine-tuning is done on a predefined set of English-only mention-entity pairs from English Wikipedia hyperlinks. Our evaluation setting is identical to LAReQA.
BUCC The second and third shared task of the workshop on Building and Using Parallel Corpora Zweigenbaum et al. [2017], Pierre Zweigenbaum and Rapp [2018] aims to examine the ability of models to detect parallel sentence pairs in a pair of monolingual corpora. The dataset provides train and test splits in 5 languages. Following XTREME Hu et al. [2020], we directly evaluate on BUCC without fine-tuning and retrieve sentences with the highest cosine similarity.
Tatoeba The goal of the Tatoeba dataset Artetxe and Schwenk [2019] is to find the nearest neighbor for each sentence in the other language according to cosine similarity and compute the error rate. The dataset consists of up to 1,000 English-aligned sentence pairs covering 122 languages. Following XTREME Hu et al. [2020], we directly evaluate on Tatoeba without fine-tuning and retrieve sentences with the highest cosine similarity.
XCOPA In the Cross-lingual Choice of Plausible Alternatives dataset Ponti et al. [2020], each XCOPA instance corresponds to a premise and two alternatives. The task is formulated as a binary classification to predict the more plausible choice. The English COPA Gordon et al. [2012] training set and Social IQa Sap et al. [2019] training data are used for fine-tuning, while the validation and test sets of English COPA are translated and re-annotated into 11 languages for evaluation.
MultiEURLEX The MultiEURLEX dataset Chalkidis et al. [2021] is a legal topic classification task that comprises 65k European Union (EU) laws in 23 official EU languages. The dataset provides multi-granular labels per document. The dataset is split into training, development, and test subsets chronologically, resulting in 55k training documents for 7 languages, and 5k each for development and test subsets in all 23 languages.
MultiARC The Multilingual Amazon Reviews Corpus Keung et al. [2020] is a large-scale collection of Amazon reviews for multilingual text classification in 6 languages. Different languages are directly gathered from the marketplaces in different countries. The goal is to predict the reviewer's rating on the 5-star scale using the test of the review as input. The data is clearly split into training (200,000 reviews), development (5,000 reviews), and test sets (5,000 reviews) for each language.
PAWS-X The Cross-lingual Paraphrase Adversaries from Word Scrambling Yang et al. [2019b] dataset requires identifying whether two sentences are paraphrases. A subset of the evaluation pairs in English PAWS Zhang et al. [2019] are human-translated into 6 typologically distinct languages for evaluation, while the English PAWS training set is used for training.
Model	en	ar	bg	de	el	es	fr	hi	ru	sw	th	tr	ur	vi	zh	Avg.
XLM-R	88.6	84.5	86.7	84.6	85.2	84.7	82.0	82.5	82.6	82.4	80.6	83.1	80.3	77.3	77.2	82.8
INFOXLM	90.4	83.9	85.8	86.0	85.6	87.8	86.9	83.9	83.5	83.3	81.2	84.6	82.7	81.6	75.7	84.2
HICTL	90.6	86.8	88.2	87.4	87.0	87.4	85.0	83.9	83.3	84.8	83.1	85.7	82.8	79.7	80.9	85.1
ChatGPT	70.4	61.0	64.5	64.8	62.8	65.7	66.3	51.5	63.4	55.7	53.0	61.6	47.9	61.6	62.6	60.9
EMMA-X	91.9	89.2	90.1	89.6	89.5	90.3	88.7	86.7	85.4	88.5	86.7	89.6	87.7	83.6	83.9	88.1
Table 10: XNLI results (accuracy) for each language.
Model	aym	bzd	cni	gn	hch	nah	oto	quy	shp	tar	Avg.
XLM-R	49.01	50.61	41.72	58.34	42.46	54.63	35.57	59.29	51.62	41.54	48.48
INFOXLM	49.87	51.29	42.41	58.83	43.07	55.25	36.14	59.87	52.20	42.12	49.10
HICTL	49.65	51.22	42.36	58.82	43.09	55.13	36.04	59.61	52.17	42.08	49.02
ChatGPT	42.0	43.6	40.8	40.4	40.0	43.8	41.1	43.1	42.0	40.0	41.7
EMMA-X	51.19	52.50	43.62	59.88	44.31	55.44	39.16	60.14	52.84	43.10	50.21
Table 11: AmericasNLI (ANLI) results (top-1 accuracy) across different input languages.
Model	en-ar	en-de	en-tr	en-es	en-fr	en-it	en-nl	ar-ar	en-en	es-es	Avg.
XLM-R	50.2	63.7	45.8	59.6	68.0	63.4	69.6	87.7	82.5	68.5	65.9
INFOXLM	81.7	80.3	79.9	79.1	80.6	83.4	81.2	86.7	87.2	81.7	82.2
HICTL	80.4	81.8	78.3	80.6	81.2	80.9	79.3	88.4	86.1	79.6	81.6
EMMA-X	86.6	85.0	87.1	84.4	85.2	89.4	88.3	90.9	92.0	84.5	87.3
Table 12: MultiSTS results (Spearman) across different input languages.
D Baseline Methods
To fairly evaluate the performance of EMMA-X, we choose XLM-R Conneau and Lample [2019] and its several derivatives as our baselines, which contain: (1) XLM-R, which applies multilingual MLM tasks as pre-training objectives on CCNet-100 corpus; (2) HICTL Wei et al. [2021], which continues training on XLM-R using hierarchical contrastive learning; and (3) INFOXLM, which is initialized with XLM-R and trains with cross-lingual contrast, multilingual MLM and TLM. Also, we compare EMMA-X to strong sentence models: (1) S-BERT [Reimers and Gurevych, 2020], which adopts multilingual knowledge distillation to extend monolingual sentence representations to multilingual. We use the strongest baseline, XLM-R  $\leftarrow$  SBERT-paraphrase, proposed in the original paper as a baseline. (2) LaBSE [Feng et al., 2022], which systematically combines several best methods, including masked language modeling, translation language modeling [Conneau and Lample, 2019], dual encoder translation ranking [Guo et al., 2018], and additive margin softmax [Yang et al., 2019a], to learn cross-lingual sentence representations. It filters 17B monolingual sentences and 6B translation pairs for sentence representation learning. We take the best model, LaBSE with Customized VOCab as our baseline. We further report the zero-shot results on Large Language Model (LLM), ChatGPT, which is trained on a wide variety of multilingual sentences and instruction tuning based on Reinforcement Learning with Human Feedback [Christiano et al., 2017, Ouyang et al., 2022].
E Prompts for ChatGPT
In this section, we show the input prompts of ChatGPT on each task in Table 13.
F Results of each Language
We show the details for tasks and all languages in Tables 10 (XNLI), 11 (AmericasNLI), 12 (Multi-STS), 14 (QE), 15 (LAReQA), 16 (Mewsli-X), 17 (XCOPA), 18 (BUCC) and 19 (PAWS-X).
G Equations and Theoretical Analysis
G.1 Details of Equations
Details of Gaussian Form  $\mathcal{N}_r$  In EMMA-X, GMM classifier is introduced to determine the semantic rank of sentence pairs. The posterior probability  $P_{\mathcal{G}}(\cdot)$  of GMM classifier is already discussed in Eq. 5. We show the explicit calculation of Gaussian form  $\mathcal{N}_r(\gamma^{(\mathbf{x}_i)},\gamma^{(\mathbf{y}_k)})$  as:
$$
\mathcal {N} _ {r} \left(\gamma^ {\left(\mathbf {x} _ {i}\right)} - \gamma^ {\left(\mathbf {y} _ {k}\right)} \mid \mu_ {r}, \sigma_ {r}\right) = \frac {\pi_ {r}}{(2 \pi) ^ {(d / 2)} \left| \operatorname {d i a g} \left(\sigma_ {r}\right) \right|} \cdot e ^ {\left(- \frac {1}{2} \left[ \left(\gamma^ {\left(\mathbf {x} _ {i}\right)} - \gamma^ {\left(\mathbf {y} _ {k}\right)}\right) - \mu_ {r} \right] ^ {T} \operatorname {d i a g} \left(\sigma_ {r} ^ {- 2}\right) \left[ \left(\gamma^ {\left(\mathbf {x} _ {i}\right)} - \gamma^ {\left(\mathbf {y} _ {k}\right)}\right) - \mu_ {r} \right]\right)}, \tag {19}
$$
where  $d$  is the dimension of hidden states of  $\gamma^{(\mathbf{x}_i)}$  and  $\gamma^{(\mathbf{y}_k)}$ .
Details of contrastive learning The training objective of cross-lingual encoder in EMMA-X is the ranking InfoNCE loss. We show the explicit expansion of this loss (Eq. 7) as:
$$
\begin{array}{l} \mathcal {L} _ {\mathbf {C T L}} (\mathcal {X}, \mathcal {Y}; \Theta_ {\mathcal {M}}) = - \mathbb {E} _ {\mathbf {x} _ {i} \sim \mathcal {X}} \left[ \right. \\ \underbrace {\log \frac {\sum_ {\mathbf {y} _ {k} \sim y _ {c _ {\mathcal {G}} ^ {*} = 1}} e ^ {\frac {s _ {[ \gamma (\mathbf {x} _ {i}) , \gamma (\mathbf {y} _ {k}) ]}}{\tau_ {1}}}} _ {\ell_ {1}} + \sum_ {\mathbf {y} _ {t} \sim y _ {c _ {\mathcal {G}} ^ {*} = 2}} e ^ {\frac {s _ {[ \gamma (\mathbf {x} _ {i}) , \gamma (\mathbf {y} _ {t}) ]}}{\tau_ {1}}} + \ldots + \sum_ {\mathbf {y} _ {t} \sim y, y _ {c _ {\mathcal {G}} ^ {*} = 4}} e ^ {\frac {s _ {[ \gamma (\mathbf {x} _ {i}) , \gamma (\mathbf {y} _ {t}) ]}}{\tau_ {1}}} \\ + \log \frac {\sum_ {\mathbf {y} _ {k} \sim \mathcal {Y} _ {c _ {\mathcal {G}} ^ {*}} = 2} e ^ {\frac {s [ \gamma (\mathbf {x} _ {i}) , \gamma (\mathbf {y} _ {k}) ]}{\tau_ {2}}}}{\underbrace {\sum_ {\mathbf {y} _ {t} \sim \mathcal {Y} _ {c _ {\mathcal {G}} ^ {*}} = 2} e ^ {\frac {s [ \gamma (\mathbf {x} _ {i}) , \gamma (\mathbf {y} _ {t}) ]}{\tau_ {2}}} + \sum_ {\mathbf {y} _ {t} \sim \mathcal {Y} _ {c _ {\mathcal {G}} ^ {*}} = 3} e ^ {\frac {s [ \gamma (\mathbf {x} _ {i}) , \gamma (\mathbf {y} _ {t}) ]}{\tau_ {2}}} + \sum_ {\mathbf {y} _ {t} \sim \mathcal {Y} _ {c _ {\mathcal {G}} ^ {*}} = 4} e ^ {\frac {s [ \gamma (\mathbf {x} _ {i}) , \gamma (\mathbf {y} _ {t}) ]}{\tau_ {2}}}}}} _ {\ell_ {2}}} \tag {20} \\ + \underbrace {\log \frac {\sum_ {\mathbf {y} _ {k} \sim \mathcal {Y} _ {c _ {\mathcal {G}} ^ {*}} = 3} e ^ {\frac {s _ {[ \gamma (\mathbf {x} _ {i}) , \gamma (\mathbf {y} _ {k}) ]}}{\tau_ {3}}}} _ {\ell_ {3}} + \underbrace {\sum_ {\mathbf {y} _ {t} \sim \mathcal {Y} _ {c _ {\mathcal {G}} ^ {*}} = 3} e ^ {\frac {s _ {[ \gamma (\mathbf {x} _ {i}) , \gamma (\mathbf {y} _ {t}) ]}}{\tau_ {3}}} + \sum_ {\mathbf {y} _ {t} \sim \mathcal {Y} _ {c _ {\mathcal {G}} ^ {*}} = 4} e ^ {\frac {s _ {[ \gamma (\mathbf {x} _ {i}) , \gamma (\mathbf {y} _ {t}) ]}}{\tau_ {3}}}} _ {\ell_ {3}} \Biggr ] , \\ \end{array}
$$
where  $\tau_{r}$  represents the temperature term. As small temperature  $\tau$  tends to be less tolerant to similar samples, and large  $\tau$  tends to cluster similar samples together [Wang and Liu, 2021], we empirically set  $\tau_{1} < \tau_{2} < \tau_{3} < \tau_{4}$ , which remains the same as Hoffmann et al. [2022].
G.2 Theoretical Analysis
In this section, we provide detailed proof for Eq. 14 and Eq. 15. Next, we prove the feasibility of our dual supervision. GMM classifier clusters sentence pairs in terms of Euclidean distance, while cross-lingual encoder minimizes the covariance of each semantic relation rank via cosine distance. Finally, we prove that these two metrics are actually equivalent to each other in the unit hypersphere of the embedding space.
Proof of Eq. 14. We provide the derivation of Eq. 14. With the assumption that  $P(\mathbf{x}_i, \mathbf{y}_k | c_{\mathcal{G}}^* = r, \Theta) \sim \mathcal{N}_r(\mathbf{x}_i - \mathbf{y}_k | \tilde{\mu}_r, \tilde{\sigma}_r)$ , we have,
$$
\begin{array}{l} \sum_ {\mathbf {x} _ {i} \in \mathcal {X}} \sum_ {\mathbf {y} _ {k} \in \mathcal {Y}} \sum_ {r = 1} ^ {N} Q (r) \log \frac {P (\mathbf {x} _ {i} , \mathbf {y} _ {k} , r | \boldsymbol {\Theta})}{Q (r)} \approx \sum_ {\mathbf {x} _ {i} \in \mathcal {X}} \sum_ {\mathbf {y} _ {k} \in \mathcal {Y}} \sum_ {r = 1} ^ {N} \log P (\mathbf {x} _ {i}, \mathbf {y} _ {k} | c _ {\mathcal {G}} ^ {*} = r, \boldsymbol {\Theta}) \\ = \sum_ {\mathbf {x} _ {i} \in \mathcal {X}} \sum_ {\mathbf {y} _ {k} \in \mathcal {Y}} \sum_ {r = 1} ^ {N} \left(\log \left(\frac {1}{(2 \pi) ^ {(d / 2)} | \tilde {\sigma} _ {r} | ^ {1 / 2}}\right) \right. \\ \left. + \frac {1}{2} \left[ \left(\mathbf {x} _ {i} - \mathbf {y} _ {k}\right) - \tilde {\mu} _ {r} \right] ^ {T} \tilde {\sigma} _ {r} ^ {- 1} \left[ \left(\mathbf {x} _ {i} - \mathbf {y} _ {k}\right) - \tilde {\mu} _ {r} \right]\right) \\ \geq \sum_ {r = 1} ^ {N} \left[ \sum_ {\mathbf {x} _ {i} \in \mathcal {X}} \sum_ {\mathbf {y} _ {k} \in \mathcal {Y}} \left(\mathbf {x} _ {i} - \mathbf {y} _ {k}\right) \right] ^ {2} - 2 \tilde {\mu} _ {r} \sum_ {\mathbf {x} _ {i} \in \mathcal {X}} \sum_ {\mathbf {y} _ {k} \in \mathcal {Y}} \left(\mathbf {x} _ {i} - \mathbf {y} _ {k}\right) + n \tilde {\mu} _ {r} ^ {2} \\ = \sum_ {r = 1} ^ {N} n ^ {2} \tilde {\mu} _ {r} ^ {2} - n \tilde {\mu} _ {r} ^ {2} \\ = n (n - 1) \sum_ {r = 1} ^ {N} \tilde {\mu} _ {r} ^ {2}, \tag {21} \\ \end{array}
$$
with  $n$  denoting the number of sentence pairs in semantic rank  $r$ . Here, we ignore the impact of  $\tilde{\sigma}_r$ .
Proof of Eq. 15. As we apply dual supervision, data in the contrastive label space also follows the distribution  $\mathcal{N}_r\big(\mathbf{x}_i - \mathbf{y}_k|\tilde{\mu}_r,\tilde{\sigma}_r\big)$ . Hence, under mild assumptions, we can get: