SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathcal {L} _ {\mathbf {C T L}} ^ {+} (\mathcal {X}, \mathcal {Y}; \Theta_ {\mathcal {M}}) = \mathbb {E} _ {\mathbf {x} _ {i} \sim \mathcal {X}} \sum_ {r = 1} ^ {N - 1} \log \sum_ {\mathbf {y} _ {k} \sim \mathcal {Y} _ {c _ {\mathcal {G}} ^ {*}} = r} e ^ {s [ \gamma (\mathbf {x} _ {i}), \gamma (\mathbf {y} _ {k}) ]} \\ = \sum_ {\mathbf {x} _ {i} \in \mathcal {X}} \sum_ {\mathbf {y} _ {k} \in \mathcal {Y}} \sum_ {r = 1} ^ {N - 1} s \left(\mathbf {x} _ {i}, \mathbf {y} _ {k}\right) \tag {22} \\ = \sum_ {\mathbf {x} _ {i} \in \mathcal {X}} \sum_ {\mathbf {y} _ {k} \in \mathcal {Y}} \sum_ {r = 1} ^ {N - 1} \frac {\left(\mathbf {x} _ {i} - \mathbf {y} _ {k}\right) ^ {2} - 2}{2} \\ = n ^ {2} \sum_ {r = 1} ^ {N - 1} \tilde {\mu} _ {r} ^ {2}. \\ \end{array}
$$
Based on the definition of semantic ranks, we have  $\tilde{\mu}_1 < \tilde{\mu}_2 < \ldots < \tilde{\mu}_N$ . Empirically, the number of sentence pairs in each rank  $n$  is larger than the number of semantic ranks  $N$ . Hence, it can be derived that:
$$
\begin{array}{l} \mathcal {L} _ {\mathbf {C T L}} ^ {+} (\mathcal {X}, \mathcal {Y}; \Theta_ {\mathcal {M}}) = n ^ {2} \sum_ {r = 1} ^ {N - 1} \tilde {\mu} _ {r} ^ {2} \\ <   n ^ {2} \sum_ {r = 1} ^ {N - 1} \tilde {\mu} _ {r} ^ {2} + n ^ {2} \tilde {\mu} _ {N} ^ {2} - n \sum_ {r = 1} ^ {N} \tilde {\mu} _ {r} ^ {2} \tag {23} \\ = n (n - 1) \sum_ {r = 1} ^ {N} \tilde {\mu} _ {r} ^ {2} \\ \leq \sum_ {\mathbf {x} _ {i} \in \mathcal {X}} \sum_ {\mathbf {y} _ {k} \in \mathcal {Y}} \sum_ {r = 1} ^ {N} Q (r) \log \frac {P (\mathbf {x} _ {i} , \mathbf {y} _ {k} , r | \Theta)}{Q (r)}. \\ \end{array}
$$
Therefore, we prove that minimizing the positive terms  $\mathcal{L}_{\mathbf{CTL}}^{+}(\mathcal{X},\mathcal{Y};\Theta_{\mathcal{M}})$  in contrastive learning is equivalent to maximizing a lower bound of the likelihood in Eq. 12.
According to the definition of semantic ranks, the approximated semantic rank  $c_{\mathcal{G}}^{*}$  from GMM classifier should satisfy the following restriction,
$$
\mathbb {E} _ {\mathbf {y} _ {k} \sim y _ {c _ {\mathcal {G}} ^ {*}} = 1} | | \gamma^ {(\mathbf {x} _ {i})} - \gamma^ {(\mathbf {y} _ {k})} | | <   \mathbb {E} _ {\mathbf {y} _ {k} \sim y _ {c _ {\mathcal {G}} ^ {*}} = 2} | | \gamma^ {(\mathbf {x} _ {i})} - \gamma^ {(\mathbf {y} _ {k})} | | <   \dots <   \mathbb {E} _ {\mathbf {y} _ {k} \sim y _ {c _ {\mathcal {G}} ^ {*}} = N} | | \gamma^ {(\mathbf {x} _ {i})} - \gamma^ {(\mathbf {y} _ {k})} | |. \tag {24}
$$
Similarly, the approximated semantic rank  $c_{\mathcal{M}}^*$  from cross-lingual encoder should satisfy the following restriction,
$$
\mathbb {E} _ {\mathbf {y} _ {k} \sim \mathcal {Y} _ {c _ {\mathcal {M}} ^ {*}} = 1} s [ \gamma^ {(\mathbf {x} _ {i})}, \gamma^ {(\mathbf {y} _ {k})} ] > \mathbb {E} _ {\mathbf {y} _ {k} \sim \mathcal {Y} _ {c _ {\mathcal {M}} ^ {*}} = 2} s [ \gamma^ {(\mathbf {x} _ {i})}, \gamma^ {(\mathbf {y} _ {k})} ] > \dots > \mathbb {E} _ {\mathbf {y} _ {k} \sim \mathcal {Y} _ {c _ {\mathcal {M}} ^ {*}} = N} s [ \gamma^ {(\mathbf {x} _ {i})}, \gamma^ {(\mathbf {y} _ {k})} ]. \tag {25}
$$
Next, we prove that these two restrictions are interchangeable with each other in a unit hypersphere. For simplicity, we consider only two ranks, but extending the explanation to more ranks is trivial. As the Euclidean distance is always larger than 0, we have:
$$
\begin{array}{l} \mathbb {E} _ {\mathbf {y} _ {k} \sim y _ {c _ {\mathcal {G}} ^ {*}} = 1} | | \gamma^ {(\mathbf {x} _ {i})} - \gamma^ {(\mathbf {y} _ {k})} | | <   \mathbb {E} _ {\mathbf {y} \sim y _ {c _ {\mathcal {G}} ^ {*}} = 2} | | \gamma^ {(\mathbf {x} _ {i})} - \gamma^ {(\mathbf {y} _ {k})} | | \\ \Leftrightarrow \mathbb {E} _ {\mathbf {y} _ {k}} \sim \mathcal {Y} _ {c _ {\mathcal {G}} ^ {*} = 1} \left(\gamma^ {(\mathbf {x} _ {i})} - \gamma^ {(\mathbf {y} _ {k})}\right) ^ {2} <   \mathbb {E} _ {\mathbf {y} _ {k}} \sim \mathcal {Y} _ {c _ {\mathcal {G}} ^ {*} = 2} \left(\gamma^ {(\mathbf {x} _ {i})} - \gamma^ {(\mathbf {y} _ {k})}\right) ^ {2} \\ \Leftrightarrow \mathbb {E} _ {\mathbf {y} _ {k} \sim \mathcal {Y} _ {c _ {\mathcal {G}} ^ {*} = 1}} (2 - 2 \gamma^ {(\mathbf {x} _ {i})} \gamma^ {(\mathbf {y} _ {k})}) <   \mathbb {E} _ {\mathbf {y} _ {k} \sim \mathcal {Y} _ {c _ {\mathcal {G}} ^ {*} = 2}} (2 - 2 \gamma^ {(\mathbf {x} _ {i})} \gamma^ {(\mathbf {y} _ {k})}) \tag {26} \\ \Leftrightarrow \mathbb {E} _ {\mathbf {y} _ {k} \sim \mathcal {Y} _ {c _ {G} ^ {*}} = 1} s \left[ \gamma^ {(\mathbf {x} _ {i})}, \gamma^ {(\mathbf {y} _ {k})} \right] > \mathbb {E} _ {\mathbf {y} _ {k} \sim \mathcal {Y} _ {c _ {G} ^ {*}} = 2} s \left[ \gamma^ {(\mathbf {x} _ {i})}, \gamma^ {(\mathbf {y} _ {k})} \right] \\ \Leftrightarrow \mathbb {E} _ {\mathbf {y} _ {k} \sim \mathcal {Y} _ {c _ {\mathcal {M}} ^ {*}} = 1} s [ \gamma^ {(\mathbf {x} _ {i})}, \gamma^ {(\mathbf {y} _ {k})} ] > \mathbb {E} _ {\mathbf {y} _ {k} \sim \mathcal {Y} _ {c _ {\mathcal {M}} ^ {*}} = 2} s [ \gamma^ {(\mathbf {x} _ {i})}, \gamma^ {(\mathbf {y} _ {k})} ]. \\ \end{array}
$$
From the above analyses, we can tell that the approximated semantic rank from one module can provide a reasonable supervision signal to guide the training of the other module. Hence, all sentence pairs will be uniformly distributed according to a unified ranking semantic similarity in the embedding space.
Basic Prompt for XNLI/ANLI
Task Description: Read the following and determine the relationship between Hypothesis and Premise. Choose relation from "contradiction", "neutral", or "entailment". Hypothesis: Yo... no=puede pensar por qué deben laslabarme asi, bajo ella, con menos de lo que le había asegurado antes. Premise: Ella era una buena amiga de él, poresto le dolía que le hablara asi.
Basic Prompt for MultiSTS
Task Description: Read the following sentences and measure the real-valued meaning similarity between these two sentences. You can choose the meaning similarity score, ranging from 0 for no meaning overlap to 5 for meaning equivalence.Sentence1: A person is on a baseball team.Sentence2: Eine Person spiel in einem Team Basketball.
Basic Prompt for QE
Task Description: Read the Source sentence and its Translation, and estimate the quality of the Translation. You can rate the translation from 0-1 according to the perceived translation quality.Source: In Franta ainceput stagnarea demografica de lunga duratá, refacerea durànd o generatie.Translation: In France, long-term demographic stagnation has started, restoring a generation.
Basic Prompt for XCOPA
Task Description: Read the Premise and determine which choice is the effect(or cause) of the Premise .Choose from "Choice1" or "Choice2".Premise: Kuki kurukuna wasiman haykurqanku.Choice1: Kuki kurukunaqa wasimanta chinkarqanku.Choice2: Kuki kuruqa wasip kurkunta mikhurqanku.
Basic Prompt for MultiEURLEX
Task Description: Read the following sentences and determine the legal topic of the given sentence. The legal topic should choose from 'international organisations', 'social questions', 'production', 'technology and research', 'environment', 'energy', 'transport', 'law', 'finance', 'education and communications', 'trade', 'agriculture', 'forestry and fisheries', 'economics', 'agri-foodstuffs', 'EUROPEAN UNION', 'science', 'politics', 'international relations', 'industry', 'geography', 'business and competition', 'employment and working conditions'.Sentence: NEUVOSTON ASETUS (EU) N:o 1390/2013, annettu 16 pääivänä joulukuuta 2013, Europonan unionin ja Komorian liiton kesken näiden välissessä kalastuskumpppanuussopimukssessa määrätyjen kalastus-mahdollisuuksien ja taloudellisen korvauksen vahvistamisesta hyväksytyn pöytäkirjan mukaisten kalastus-mahdollisuuksien jakamisesta ...
Basic Prompt for MultiARC
Task Description: Read the following review and predict a 5-star scale rating (1 means the poorest experience and 5 represents excellent or outstanding performance) that can best match the review.Review: no meiego el articulo me lo mando por correos normal sin seguemiento y nunca melegoto tata un desastre
Basic Prompt for PAWS-X
Task Description: Read the following sentences and determine whether two sentences are paraphrases.Return yes or no.Sentence1: La exception fue entre fines de 2005 y 2009 cuandojugó en Suecia con Carlstad United BK,Serbia con FK Borac Čačak y el FC Terek Grozny de Rusia.Sentence2: La exception se dio entre fines del 2005 y 2009, cuandojugó con Suecia en el Carlstad United BK, Serbia con el FK Borac Čačak y el FC Terek Grozny de Rusia.
Table 13: Prompts of ChatGPT on each task.
Model	en-de	en-zh	et-en	ne-en	ro-en	ru-en	si-en	en-cs	en-ja	km-en	ps-en	Avg.
XLM-R	0.412	0.566	0.797	0.812	0.891	0.774	0.578	0.547	0.335	0.612	0.635	0.632
INFOXLM	0.517	0.534	0.775	0.834	0.890	0.788	0.581	0.564	0.325	0.635	0.616	0.641
HICTL	0.495	0.579	0.792	0.835	0.904	0.787	0.575	0.556	0.342	0.625	0.648	0.649
EMMA-X	0.580	0.589	0.809	0.854	0.897	0.829	0.593	0.577	0.370	0.641	0.651	0.672
Table 14: WMT21-QE-Task1 results (Pearson) across different input languages.
Model	ar	de	el	en	es	hi	ru	th	tr	vi	zh	Avg.
XLM-R	34.1	42.4	39.3	44.8	44.0	37.3	41.7	38.6	40.9	40.4	39.5	40.3
INFOXLM	39.7	52.6	39.2	55.1	53.4	36.8	51.0	28.5	41.1	48.9	47.3	44.9
HICTL	40.3	53.2	41.7	56.3	54.3	39.6	51.7	30.1	42.8	48.9	48.5	46.1
EMMA-X	45.1	58.4	45.4	60.6	59.8	41.4	56.3	34.7	47.1	54.6	53.4	50.6
Table 15: LAReQA results (mean average precision@20, mAP@20) across different input languages.
Model	ar	de	en	es	fa	ja	pl	ro	ta	tr	uk	Avg.
XLM-R	34.6	66.0	62.6	64.8	27.1	47.8	64.8	33.7	17.8	62.3	53.2	48.6
INFOXLM	40.8	71.6	66.3	68.7	48.7	61.0	66.7	39.2	42.0	64.6	58.1	57.1
HICTL	41.7	68.5	64.2	65.6	45.6	51.9	67.6	40.4	32.8	65.5	58.9	54.8
EMMA-X	50.2	78.7	69.1	63.7	47.9	59.6	70.0	50.2	43.5	68.0	60.9	59.6
Table 16: Mewsli-X results (mean average precision@20, mAP@20) across different input languages.
Model	et	ht	id	it	qu	sw	ta	th	tr	vi	zh	Avg.
XLM-R	73.8	67.4	77.8	72.2	52.3	70.9	72.1	74.6	73.4	73.2	75.7	71.2
INFOXLM	75.1	73.4	78.3	80.7	65.6	69.1	72.7	73.9	76.9	77.8	77.5	74.6
HICTL	75.9	73.1	77.8	81.2	65.5	73.8	72.6	73.2	76.1	75.4	78.0	74.8
ChatGPT	80.6	64.1	85.6	89.2	47.4	75.9	56.4	67.3	82.2	81.5	85.8	74.2
EMMA-X	76.8	74.0	77.6	79.8	76.2	74.4	77.8	74.2	77.6	82.6	89.6	78.2
Table 17: XCOPA results (accuracy) across different input languages.
Model	de	fr	ru	zh	Avg.
XLM-R	76.1	72.3	62.3	60.8	67.9
INFOXLM	81.3	78.2	76.0	74.2	77.4
HICTL	80.5	79.2	76.0	74.8	77.6
EMMA-X	85.1	82.8	81.3	78.3	81.9
Table 18: BUCC results (F1) across different languages.
Model	en	de	es	fr	ja	ko	zh	Avg.
XLM-R	95.7	92.2	92.7	92.5	84.7	85.9	87.1	90.1
INFOXLM	97.7	94.6	95.2	95.1	88.9	89.0	90.2	93.0
HICTL	97.4	94.2	95.0	94.2	89.1	89.5	90.2	92.8
ChatGPT	71.9	67.8	67.9	67.0	58.3	54.7	61.4	64.2
EMMA-X	97.3	95.6	94.7	96.0	92.9	89.8	93.0	94.2
Table 19: PAWS-X results (accuracy) for each language.