MEXMA: Token-level objectives improve sentence representations
Authors: João Maria Janeiro, Benjamin Piwowarski, Patrick Gallinari, Loic Barrault
Accepted at: acl25

MEXMA: Token-level objectives improve sentence representations

João Maria Janeiro1,2    Benjamin Piwowarski2    Patrick Gallinari2,3    Loïc Barrault1
1Meta AI   2Sorbonne Université, CNRS, ISIR, F-75005 Paris, France
3Criteo AI Lab, Paris, France
{jooajaneiro}@meta.com

Abstract

Cross-lingual sentence encoders (CLSE) create fixed-size sentence representations with aligned translations. Current pre-trained CLSE approaches use sentence-level objectives only. This can lead to loss of information, especially for tokens, which then degrades the sentence representation. We propose MEXMA, a novel approach that integrates both sentence-level and token-level objectives. The sentence representation in one language is used to predict masked tokens in another language, with both the sentence representation and all tokens directly updating the encoder. We show that adding token-level objectives greatly improves the sentence representation quality across several tasks. Our approach outperforms current pre-trained cross-lingual sentence encoders on bitext mining as well as several downstream tasks. We also analyse the information encoded in our tokens, and how the sentence representation is built from them.

1 Introduction

Creating general-purpose multilingual embeddings has attracted significant attention from the research community in recent years, driven by the growing need for efficient and effective cross-lingual representations. Cross-Lingual Sentence Encoders (CLSE) create fixed-size sentence representations that are able to capture the relevant information in a sentence, and are aligned across languages. By capturing relevant sentence information in a shared multilingual space, these aligned representations enable efficient comparison and retrieval based on distance measures, thereby facilitating their effective utilization in various downstream applications.

Current CLSE (Duquenne et al., 2023; Feng et al., 2022) typically build upon pre-trained encoders, often language models (Conneau et al., 2020; Devlin et al., 2019) or translation models (NLLB Team et al., 2022). These pre-trained encoders have been trained using objectives that focus on individual words or tokens, i.e. token-level objectives. Examples of such objectives include unmasking, where the model is required to predict each token individually, and all predictions are used to update the encoder directly. However, Muennighoff et al. (2023); Hu et al. (2020) show that pre-trained encoders without objectives that consider entire sentences, i.e. sentence-level objectives, do not create good sentence representations. This means that CLSE need to train using sentence-level objectives, in order to effectively capture the relevant information of the sentences.

Although CLSE start from encoders pre-trained with token-level objectives, they are commonly trained with sentence-level objectives that only update the encoder through the sentence representation (Duquenne et al., 2023; Feng et al., 2022; Yang et al., 2019; Artetxe and Schwenk, 2019b), without any objective for each token individually. We hypothesize that token-level objectives should be kept during the training of CLSE, coupled with the sentence-level objectives, to better update the encoder and improve sentence representation quality and alignment. The intuition is that only using sentence-level objectives leads to a degradation of token-level information, especially lexical, which, in turn, can impact the sentence representation.

Recently, approaches have explored the use of both token-level and sentence-level objectives for better sentence representations. In DAP (Li et al., 2023), the token-level objective is only used to update the token representations in the encoder, without influencing directly the sentence representation. In RetroMAE (Xiao et al., 2022), the tokens are not directly updated with the same token-level objective as the sentence representation.

To effectively combine token and sentence-level objectives, we propose MEXMA, a new approach that uses the sentence representation in one language to predict masked tokens in another lan-
guage, and uses both the sentence and tokens’ information to update the encoder. This token-level objective is combined with a sentence-level objective to enforce sentence alignment across languages.

Our approach outperforms state-of-the-art pre-trained cross-lingual sentence encoders, namely LaBSE and SONAR, on several key tasks including bitext mining, classification, and pair classification. Specifically, we report notable gains on the xsim++ benchmark computed over the FLORES200 test set, where MEXMA achieves an error rate of 9.6%, surpassing SONAR’s 12.1%. Additionally, in classification tasks evaluated on MTEB and SentEval, MEXMA achieves an accuracy of 65.4% compared to SONAR’s 63.0%. The larger supervision in MEXMA enables training smaller models with better alignment than LaBSE (\( \approx 2\times \)) and close to SONAR’s performance (\( \approx 3\times \)).

Our main contributions are:
• We introduce a novel architecture leveraging both sentence-level and token-level objectives outperforming current approaches.
• We perform ablation studies that show the impact of token-level objectives on the sentence-level representations performance.
• We thoroughly analyze our model’s inner workings by examining token content and sentence embedding construction.
• We show our approach can be coupled with existing alignment approaches, specifically contrastive learning, and improve its quality.
• Our code is available at: github.com/facebookresearch/mexma and our model at huggingface.co/facebook/MEXMA.

2 Related Work

Sentence embeddings have been well studied in the last decade. Initially, recurrent networks were trained to predict previous and next sentence (Kiros et al., 2015) or sentence entailment (Conneau et al., 2017). Universal Sentence Encoder (Cer et al., 2018) trains a transformer network on both tasks. Reimers and Gurevych (2019) propose to continue the training of a BERT model to include a sentence-level objective. CLSE are multilingual extensions of these works that align the representations across languages.

UPDATE VIA SENTENCE REPRESENTATION
Most current CLSE approaches only update their encoder via the sentence representation objective, without having any token-level objective in the output of the encoder to update each token individually (Guo et al., 2018; Yang et al., 2019; Feng et al., 2022; Artetxe and Schwenk, 2019b; Duquenne et al., 2023; Heffernan et al., 2022). They are most commonly based on contrastive learning (Hadsell et al., 2006), that aims to reduce the distance between positive pairs (translations) and increase the distance between negative pairs (non-translations) (Guo et al., 2018; Yang et al., 2019; Feng et al., 2022). Notably, LaBSE (Feng et al., 2022) uses the contrastive loss, with the additive margin softmax approach of Yang et al. (2019). Non-contrastive approaches reduce the distance between positive pairs (translations) only, being prone to collapse. A common solution to collapse is to use an auto-regressive decoder to prevent it. For CLSE, it is common to use translation (Artetxe and Schwenk, 2019b; Duquenne et al., 2023) with a fixed-size sentence representation after the encoder (bottleneck), assuming that a model can translate a sentence into many languages only if a good sentence-level representation is learned. The bottleneck, however, prevents gradients from the decoder to directly update the individual token representations of the encoder, which we hypothesize leads to a degradation of token level information.

UPDATE VIA SENTENCE AND TOKEN REPRESENTATIONS
Recent approaches (Li et al., 2023; Xiao et al., 2022; Wei et al., 2021; Fan et al., 2022) have shown that combining token and sentence level objectives can improve sentence representations. RetroMAE (Xiao et al., 2022), is an Information Retrieval method that uses fixed-size sentence representations to guide token unmasking, demonstrating its effectiveness in enhancing sentence representation quality. The encoder itself is only updated by its own Masked Language Modeling (MLM) loss, and via the gradients coming from the sentence representation, but not from the direct token-level gradients of the heavy unmasking with the sentence representation as context. Wei et al. (2021) combines MLM with a contrastive loss. However, the alignment between sentences is performed on masked sentences, and the unmasking is not done with a cross-lingual sentence context. DAP (Li et al., 2023) proposes to jointly align tokens and sentence representations. It performs unmasking with all tokens of the other language as context. However, it relies exclusively on the contrastive loss to update the sentence representations.
In our work, we show that sentence and token-level objectives can be much more intertwined. Both individual tokens and the sentence representations update the encoder, leading to improved performance. Detailed diagrams of the described architectures are provided in Appendix H.

3 Methodology

We propose MEXMA, a novel cross-lingual sentence encoder trained with both token- and sentence-level objectives. The goal is to create a sentence representation that is able to encode the syntactic, semantic and lexical information in a sentence, with representations well aligned across languages. To achieve this goal, inspired by monolingual masked auto-encoding techniques (Xiao et al., 2022), we use the sentence representation in one language to unmask the tokens in another language, updating both the sentence and individual tokens. The masking allows us to use a non-contrastive loss to align sentence representations, since it prevents the collapse. Our architecture is depicted in Figure 1, and is composed of several components that we describe now. In the following, we refer to inputs, models and outputs that have no masking as clean, and masked for their masked counterparts. We consider two languages, language \( \mathcal{A} \) and language \( \mathcal{B} \), and two clean parallel sentence representations \( S_{\mathcal{A}} \) and \( S_{\mathcal{B}} \).

Cross-unmasking To ensure that our sentence vector captures the meaningful information of the sentence, we mask a significant portion of the input tokens in language \( \mathcal{A} \). This makes it challenging for the MLM head to recover the missing tokens without any additional context. To overcome this challenge, we provide the unmasking head with the sentence vector \( S_{\mathcal{B}} \), derived from the clean sentence in language \( \mathcal{B} \). This forces the model to leverage the information in \( S_{\mathcal{B}} \) to predict the masked tokens in language \( \mathcal{A} \). By doing so, we encourage the sentence vector to capture the essential information of the sentence that is useful across languages. We formulate this component into a symmetrical cross-entropy loss (CE), applied over the outputs of the encoders:

\[
\mathcal{L}_{mlm} = CE([S_{\mathcal{B}}, \hat{A}], A) + CE([S_{\mathcal{A}}, \hat{B}], B),
\]

where \( \hat{A} \) and \( \hat{B} \) are the outputs of the masked encoders without the CLS embedding, A and B the masked tokens’ targets, and \([X; Y]\) represents the concatenation of X and Y.

Alignment loss The cross-unmasking generates an implicit alignment due to the switching of languages to perform the unmasking. However, it does not strongly enforce the same sentence representations in two different languages to be close in the embedding space. Following SONAR (Duquenne et al., 2023), to further reinforce the spatial proximity of semantically equivalent sentences across languages, we use an additional non-contrastive alignment objective. We formulate this component as a Mean Squared Error (MSE) loss between sentence representations:

\[
\mathcal{L}_{alignment} = MSE(S_{\mathcal{A}}, S_{\mathcal{B}}),
\]

The two losses, unmasking and alignment, complement each other to provide both aligned and meaningful vector representations of sentences in multiple languages.

Symmetrical architecture To align all languages and maximize data usage, we adopt a symmetrical approach that unmasks the tokens of language \( \mathcal{A} \) with \( S_{\mathcal{B}} \), and vice versa, simultaneously. We thus create four instances of the encoder (with shared parameters). For each language, we have two versions of each sentence: one heavily masked and one clean. This allows us to generate two clean sentence vectors, \( S_{\mathcal{A}} \) and \( S_{\mathcal{B}} \), which is essential for aligning representations between languages. The clean encoders generate the clean embeddings that are further used to align the multilingual representations.

KoLeo loss In preliminary experiments, we noticed that our representations exhibited more anisotropy than those learned with contrastive approaches. This has been shown to impact the quality of the representations (Godey et al., 2024). Inspired by DINOv2 (Oquab et al., 2024), we employ the KoLeo loss (Beirlant et al., 1997; Sablayrolles et al., 2019) to encourage sentence representations to spread out evenly in the latent space (details in Appendix A).

Our training loss is a weighted combination of all previous losses:

\[
\mathcal{L}_{MEXMA} = \alpha \cdot \mathcal{L}_{alignment} + \beta \cdot \mathcal{L}_{mlm} + \gamma \cdot \mathcal{L}_K
\]

where \( \alpha \), \( \beta \) and \( \gamma \) are hyper-parameters that control the weight of each loss term. To show that MEXMA can be used on top of existing alignment approaches, we provide, in Section 5.2, experimental results when replacing the MSE alignment loss in MEXMA with a contrastive loss.
Figure 1: MEXMA architecture. Given two translations, we create two views for each, a masked and a clean version (symmetrical architecture), and use the sentence representations from one language to unmask the other (cross-unmasking). We align the clean sentence representations via the alignment loss, and increase the usage of the space with the KoLeo loss.

3.1 Experimental setup

ENCODER BACKBONE As our encoder, we utilize a modified version of the XLM-RoBERTa model (Conneau et al., 2020) provided by HuggingFace that uses a more efficient attention (details in Appendix A). Our sentence representation from the encoder is obtained via the CLS embedding of the last layer, without any further processing.

TRAINING DATA Our training dataset is a subset of the NLLB-200 corpus (NLLB Team et al., 2022), which comprises 200 languages. We cover 81 languages, utilizing only publicly available data, all sourced from Opus (Tiedemann, 2012). The specific languages used are listed in Appendix C. We always train using one sentence in English associated with its translation in one of the remaining 80 languages, as done in SONAR. The dataset consists of a combination of human-translated and synthetic data, where we attempt to impose a minimum of 15 million sentences per language. For languages with limited human-annotated data, we supplemented the dataset with mined data from NLLB (Schwenk et al., 2020; Fan et al., 2020; NLLB Team et al., 2022) to reach the 15 million sentence threshold. Conversely, to ensure that our dataset is somewhat balanced across languages, for languages with abundant human-annotated data, we capped the dataset at 25 million sentences per language. The datasets used are in Table 15.

We provide details about the parameters and configurations of our model in Appendix A.

4 Results

To assess the quality and alignment of our embeddings, we evaluate them on a range of tasks. These tasks fall into two categories: mining tasks and other downstream tasks. Mining tasks measure how aligned our representations are across languages, while downstream tasks evaluate the generalization power and overall quality of our embeddings.

4.1 Multilingual alignment through mining

We evaluate our model on three alignment tasks: xsim\(^1\), xsim++ (Chen et al., 2023), and BUCC (Zweigenbaum et al., 2018, 2017). Both xsim and BUCC involve retrieving the correct translation of a query sentence from multilingual datasets. xsim++ adds complexity by introducing hard negatives in English sentences. Following Heffernan et al. (2022), we exclude Tatoeba due to limited data and low-quality translations.

xsim and xsim++ use a margin-based similarity approach (Artetxe and Schwenk, 2019a), while

\footnotetext{1https://github.com/facebookresearch/LASER/tree/main/tasks/xsim}
<table>
  <tr>
    <th>Model</th>
    <th>xsim ↓</th>
    <th>xsim++ ↓</th>
    <th>BUCC ↑</th>
  </tr>
  <tr>
    <td>DAP</td>
    <td>2.90</td>
    <td>32.82</td>
    <td>98.68</td>
  </tr>
  <tr>
    <td>SONAR</td>
    <td>0.04</td>
    <td>10.55</td>
    <td>98.25</td>
  </tr>
  <tr>
    <td>LaBSE</td>
    <td>0.26</td>
    <td>14.51</td>
    <td>98.75</td>
  </tr>
  <tr>
    <td>MEXMA</td>
    <td><b>0.02</b></td>
    <td><b>8.26</b></td>
    <td><b>98.93</b></td>
  </tr>
</table>

Table 1: Results in mining (%). xsim and xsim++ are computed on the 34 languages (FLORES200 dataset, X-eng pairs) supported by all models. BUCC is computed with F1 in its 4 languages.

BUCC employs cosine similarity. xsim and xsim++ scores are the error rate of misaligned sentences, whereas BUCC uses the F1 score, evaluated with the MTEB benchmark (Muennighoff et al., 2023).

BUCC covers German, French, Russian and Chinese. We report the results for xsim and xsim++ on the FLORES200 dataset, for the 34 languages common to all models. Results per language, and on larger sets of languages are available in Appendix E.

The results are shown in Table 1. MEXMA outperforms previous SOTA on all benchmarks, showcasing improved alignment in our approach. The improvements in xsim and BUCC suggest that our approach improves the semantic alignment of the embeddings. The large improvement in xsim++ (+2.48% absolute against the previous best model SONAR) also indicates the increased robustness of our model with regard to hard negatives, likely due to handling better lexical information. For more thorough comparisons using the same data and backbones see Appendix B.4.

4.2 Downstream tasks

To understand the quality of our embeddings and how generic they are, we evaluate them on several tasks from the MTEB benchmark (Muennighoff et al., 2023). We report the averaged results for each language. For the full list of results for each task, see Appendix E.

SINGLE SENTENCE CLASSIFICATION We evaluate our model’s classification performance on two benchmarks. The SentEval (SE) suite (Conneau and Kiela, 2018) is used to assess the performance across various tasks in English and the MTEB benchmark for the multilingual classification capabilities. Table 2 shows the aggregated results. We can see that MEXMA outperforms all baseline models on average, and more specifically gains +2.33% when compared with SONAR.

<table>
  <tr>
    <th>Model</th>
    <th>Class</th>
    <th>SE</th>
    <th>PC</th>
    <th>STS</th>
  </tr>
  <tr>
    <td>DAP</td>
    <td>61.80</td>
    <td>78.18</td>
    <td>66.01</td>
    <td>59.39</td>
  </tr>
  <tr>
    <td>SONAR</td>
    <td>63.02</td>
    <td>85.82</td>
    <td>69.70</td>
    <td>58.04</td>
  </tr>
  <tr>
    <td>LaBSE</td>
    <td>62.77</td>
    <td>85.63</td>
    <td>68.47</td>
    <td>64.65</td>
  </tr>
  <tr>
    <td>MEXMA</td>
    <td><b>65.35</b></td>
    <td><b>86.38</b></td>
    <td>71.55</td>
    <td>63.99</td>
  </tr>
</table>

Table 2: Averaged results on Classification (Class), SentEval (SE), Pair Classification (PC) and Semantic textual similarity (STS), across several languages.

PAIRWISE SENTENCE CLASSIFICATION We further evaluate on the pair classification task. This task consists in classifying sentence pairs, e.g. determining if two sentences are duplicates or not. The metric, as reported in MTEB, is the Average Precision (AP) based on the distance between sentence representations. The results are in Table 2. MEXMA consistently outperforms all baselines on average, by at least +1.85%. These results, combined with our single sentence classification results, suggest that our model can effectively encode the relevant information in the sentence vectors.

SEMANTIC TEXTUAL SIMILARITY (STS)
The STS task evaluates the model’s ability to replicate human judgments on sentence similarity. The metric, as reported in MTEB, is the Spearman correlation based on distance. The results are in Table 2. We can see that LaBSE outperforms all other methods, and in particular MEXMA by 0.66%. MEXMA outperforms SONAR (+5.95%) and DAP (+4.6%). The results indicate that the contrastive loss better suits the STS task, given this is the only task where DAP is able to outperform SONAR, and where LaBSE outperforms MEXMA.

5 Ablations and Analyses

In this section, we conduct a comprehensive analysis of our MEXMA architecture, examining the impact of its individual components, how it scales with varying model and data sizes, and its potential to improve other alignment approaches. We also examine the characteristics of token embeddings and sentence representations learned by our model.

5.1 Model components

In Table 3 we ablate the impact of having direct token-level gradients in MEXMA. The goal is to understand the relevance of the gradients that update the encoder: either only from the sentence, or from the sentence and all tokens. Model (1) includes all of MEXMA’s components, as covered
<table>
  <tr>
    <th>component</th>
    <th>xsim ↓</th>
    <th>xsim++ ↓</th>
    <th>SE ↑</th>
  </tr>
  <tr>
    <td>Sentence grads (1)</td>
    <td>0.15</td>
    <td>11.37</td>
    <td>85.06</td>
  </tr>
  <tr>
    <td>+ Token grads (2)</td>
    <td>0.10</td>
    <td>9.67</td>
    <td>85.98</td>
  </tr>
  <tr>
    <td>+ KoLeo loss (3)</td>
    <td>0.06</td>
    <td>9.60</td>
    <td>86.38</td>
  </tr>
</table>

Table 3: Ablation study of the components of the model. All experiments are conducted with the final hyperparameters of the model, as reported in Section 3.1.

<table>
  <tr>
    <th>Model</th>
    <th>xsim ↓</th>
    <th>xsim++ ↓</th>
    <th>SE ↑</th>
  </tr>
  <tr>
    <td>C XLM-R</td>
    <td>0.13</td>
    <td>33.30</td>
    <td>85.5</td>
  </tr>
  <tr>
    <td>C-MEXMA detach</td>
    <td>0.13</td>
    <td>12.78</td>
    <td>85.86</td>
  </tr>
  <tr>
    <td>C-MEXMA</td>
    <td>0.12</td>
    <td>10.93</td>
    <td>85.94</td>
  </tr>
</table>

Table 4: Using contrastive loss (refered as "C") as the alignment loss in MEXMA and XLM-R.

in Section 3, except the KoLeo loss. However, the gradients from the unmasking task are only back-propagating through the sentence representations to the encoder, and are deactivated for the individual tokens coming from the encoder, i.e. in the \( \mathcal{L}_{mlm} \) mentioned in Section 3, \( A/B \) have no gradients flowing back to the encoder. This model already achieves results that are competitive with current state of the art models, and if we allow the gradients to flow through the tokens directly (model (2)), we are able to outperform them. As we hypothesized, adding updates on the tokens directly, coupled with the sentence updates largely improves results across all tasks. Additionally, we also show that adding the KoLeo loss also improves results across all tasks (model (3)). The ablation on all components of the model is provided in Appendix B.

5.2 Contrastive alignment loss

To further assess the improvements given by the direct token updates in MEXMA, and understand MEXMA’s scalability compared to other alignment approaches, we replaced our MSE alignment loss with a contrastive loss and dropped the KoLeo loss. The results are presented in Table 4. We used a siamese network with XLM-RoBERTa-large trained on the symmetric cross-entropy loss (InfoNCE from (van den Oord et al., 2019)) as the baseline model, "C XLM-R", having an architecture similar to LaBSE (Feng et al., 2022). Our training used a batch size of 1.2k tokens, with the rest of the parameters the same as reported in Section 3.1. Our baseline model performs well on xsim and SentEval but struggles with xsim++. The MEXMA architecture without token-level gradients ("C-MEXMA detach"), similar to model (1) in Section 5.1, improves the performance. Adding token-level gradients, the full MEXMA architecture with contrastive loss ("C-MEXMA"), similar to model (2) in Section 5.1, results in competitive performance, outperforming previous approaches in SentEval and xsim++. This demonstrates the positive impact of direct token-level gradients and shows that MEXMA can be easily integrated within existing alignment approaches, such as contrastive learning, to improve their results.

<table>
  <tr>
    <th>Model</th>
    <th>#par</th>
    <th>xsim ↓</th>
    <th>xsim++ ↓</th>
    <th>SE ↑</th>
  </tr>
  <tr>
    <td>DAP</td>
    <td>277M</td>
    <td>2.90</td>
    <td>32.82</td>
    <td>78.18</td>
  </tr>
  <tr>
    <td>MEXMA-b</td>
    <td>277M</td>
    <td>0.06</td>
    <td>11.01</td>
    <td>85.30</td>
  </tr>
  <tr>
    <td>LaBSE</td>
    <td>471M</td>
    <td>0.26</td>
    <td>14.51</td>
    <td>85.63</td>
  </tr>
  <tr>
    <td>MEXMA</td>
    <td>559M</td>
    <td>0.02</td>
    <td>8.26</td>
    <td>86.38</td>
  </tr>
  <tr>
    <td>SONAR</td>
    <td>766M</td>
    <td>0.04</td>
    <td>10.55</td>
    <td>85.82</td>
  </tr>
</table>

Table 5: Model size comparison. MEXMA-b is based on the XLM-RoBERTa-base, and MEXMA is based on XLM-RoBERTa-Large.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">xsim ↓</th>
    <th colspan="2">xsim++ ↓</th>
    <th colspan="2">SentEval ↑</th>
  </tr>
  <tr>
    <th>81</th>
    <th>90</th>
    <th>81</th>
    <th>90</th>
    <th>81</th>
    <th>90</th>
  </tr>
  <tr>
    <td>SONAR</td>
    <td>0.09</td>
    <td>0.05</td>
    <td>12.08</td>
    <td>11.42</td>
    <td>85.82</td>
    <td>85.82</td>
  </tr>
  <tr>
    <td>MEXMA</td>
    <td><b>0.06</b></td>
    <td><b>0.05</b></td>
    <td><b>9.60</b></td>
    <td><b>9.06</b></td>
    <td><b>86.38</b></td>
    <td><b>86.64</b></td>
  </tr>
</table>

Table 6: Training data size comparison. We train MEXMA on either 81 languages, or 90 languages. See Appendix C for the list of covered languages.

MEXMA architecture without token-level gradients ("C-MEXMA detach"), similar to model (1) in Section 5.1, improves the performance. Adding token-level gradients, the full MEXMA architecture with contrastive loss ("C-MEXMA"), similar to model (2) in Section 5.1, results in competitive performance, outperforming previous approaches in SentEval and xsim++. This demonstrates the positive impact of direct token-level gradients and shows that MEXMA can be easily integrated within existing alignment approaches, such as contrastive learning, to improve their results.

5.3 Model and data sizes

Table 5 shows how our model’s results depending on the model size. We train two models, MEXMA-b with 277M parameters, based on XLM-RoBERTa-base, and MEXMA with 559M parameters, based on XLM-RoBERTa-large. We observe that even the smaller model (277M parameters) outperforms LaBSE (471M parameters), on both xsim and xsim++, and gets close SentEval results, with a 0.3% decrease in performance, with 59% of the size. This smaller model gets surprisingly close to the results of SONAR, which has 766M parameters, i.e. \( \approx 2.77 \) times its size. These results show that our approach works on smaller and larger models, and it seems to enable quite powerful small models, due to our stronger training signal. Our larger model, MEXMA, with \( \approx 73\% \) the size of SONAR, is able to largely outperform it across all tasks.

To investigate the impact of training data, we
<table>
  <tr>
    <th>Model</th>
    <th>% other</th>
    <th>% SL</th>
    <th>% SS</th>
    <th>% T</th>
  </tr>
  <tr>
    <td>XLM-R</td>
    <td>1.19</td>
    <td>63.89</td>
    <td>2.65</td>
    <td>32.27</td>
  </tr>
  <tr>
    <td>LaBSE</td>
    <td>0.00</td>
    <td>0.13</td>
    <td>42.33</td>
    <td>57.54</td>
  </tr>
  <tr>
    <td>DAP</td>
    <td>0.00</td>
    <td>0.66</td>
    <td>20.11</td>
    <td>79.23</td>
  </tr>
  <tr>
    <td>NT-MEXMA</td>
    <td>0.13</td>
    <td>0.40</td>
    <td>11.90</td>
    <td>87.57</td>
  </tr>
  <tr>
    <td>NLLB</td>
    <td>0.40</td>
    <td>3.17</td>
    <td>1.72</td>
    <td>94.71</td>
  </tr>
  <tr>
    <td>SONAR</td>
    <td>0.00</td>
    <td>0.13</td>
    <td>0.20</td>
    <td>99.67</td>
  </tr>
  <tr>
    <td>MEXMA</td>
    <td>0.26</td>
    <td>1.33</td>
    <td>0.53</td>
    <td>97.88</td>
  </tr>
</table>

Table 7: Result of the token matching analysis. Where SL: Same Language, SS: Same Sentence and T: Translation.

conducted experiments using two different language subsets of the FLORES200 dataset. We trained separate MEXMA models on each subset, using the same hyperparameters as reported in Section 3.1. For comparison, we evaluated the publicly available SONAR model, which was trained on all available 200 languages, on both language subsets. The results, presented in Table 6, demonstrate that MEXMA outperforms SONAR on both subsets, highlighting the adaptability and robustness of our approach to varying training data.

5.4 Masking ratio

NLP models typically use masking percentages around 15%, whereas vision papers have explored much higher masking ratios, ranging from 40% in BEiT (Bao et al., 2022) to as high as 90% in MAE (He et al., 2022) and V-JEPA (Bardes et al., 2024), usually aligning augmentations. For text, there is less redundancy and the representations are more information-dense. In our case, we are aligning the same sentence in several languages, which can be viewed as augmentations of a pivot sentence, i.e. the sentence in English. We need to know how much we can mask, to make the unmasking task hard, but to not deteriorate the performance of our encoder. More information is provided in Appendix B. The range 30%-60% seems to be the best operating region. We selected 40% for all experiments conducted in this paper, since it has the best balance between alignment and classification.

5.5 Token embeddings analysis

Sentence vectors are pooled representations of their tokens. In this section, we investigate the information encoded in the tokens from the last layer across different models. Our goal is to determine whether the tokens primarily convey semantic, lexical, and/or contextual information. Although these categories can be intertwined, understanding the dominant characteristics of each model’s tokens provides valuable insights into their behavior.

To gain insight into the information encoded in individual tokens, we examined their nearest neighbors in the embedding space. We categorized these neighboring tokens into four groups based on the sentence they belong to. Same language: the matched token is the same token in a different sentence in the same language, which means that it encodes lexical information. Same sentence: the token matches another one in the same sentence, meaning the tokens’ representations are heavily influenced by its context. Translation: the token matches its equivalent in a translation of the original sentence. It means that the tokens’ representations are aligned across languages. Other: tokens that do not belong to previous classes.

We conducted these experiments by encoding all tokens from all sentences of the 81 languages (see Appendix C for the list) on the FLORES200 test set using each model. We randomly select three tokens among each of the first 250 English sentences of the dataset as query, and for each query, we retrieve the five closest tokens among all tokens of all sentences (but itself). We analyze the properties of the sentence encoders as well as some respective backbones. XLM-RoBERTa (used to initialize MEXMA) and NLLB-200 encoder (used in SONAR). For the sake of comparison, we also examine "no-tok-MEXMA", a variant of MEXMA that does not use token-level gradients during training. The statistics are shown in Table 7.

Our analysis identifies distinct model characteristics and clusters them into three behaviors. XLM-RoBERTa exhibits strong lexical relationships (high same language percentage) but weaker semantic and contextual relations. LaBSE, DAP and no-tok-MEXMA show higher semantic capabilities as shown by the larger translation rate. However, we can also observe a high percentage of matches with adjacent tokens (same sentence column), indicating that those models encode a very large amount of context in their tokens.

NLLB, SONAR and MEXMA have strong cross-lingual semantic capability as shown by the very high percentage in the translation column. This is expected as they were trained to perform translation or cross-lingual unmasking.

Note also that LaBSE and DAP are the only models trained with a sentence-level contrastive loss, and even though DAP has an additional loss
to enforce the semantic alignment of the tokens, it does not manage to achieve the same alignment as SONAR and MEXMA.

Notably, comparing the backbones NLLB and XLM-RoBERTa, we can see that the former exhibits more semantical tokens than the latter, as shown by its higher translation rate and lower same sentence rate, which can be attributed to its translation-based pre-training that enhances semantic properties and cross-lingual alignment. SONAR, which starts from NLLB, also matches translated tokens with a high rate, >99%, but does not encode a lot of lexical information (low same language rate). MEXMA also matches translated tokens very frequently, but additionally displays more lexicality (higher same language rate) and increased semantic robustness (higher other rate). To assess the latter, we verified MEXMA’s other matches. The matched tokens belong to sentences in other languages that are not translations of the original one, with the matched token being the translated token. We believe that MEXMA produces sentence representations that inherit the above properties, allowing it to outperform other models on downstream tasks. We provide examples to illustrate the behavior of the models, also experiments with SimAlign (Jalili Sabet et al., 2020), in Appendix F.

5.6 Sentence vector analysis

Sentence representations are created by combining token representations in various ways (average or CLS attention pooling). The previous section examined properties encoded in tokens, and this section explores how these representations are combined to form the sentence embedding.

To create the sentence representation, SONAR averages the tokens embeddings (uniform distribution) while MEXMA and LaBSE both use a CLS token to perform pooling over the tokens.

We analyse the attention distributions by computing their average entropy in the last layer given by the CLS token, on FLORES200 test set, for the languages supported by both LaBSE and MEXMA. LaBSE gets an entropy of \( \approx 3.4 \), while MEXMA gets an entropy of \( \approx 2.5 \). Although the absolute entropy values themselves are difficult to interpret, the relative difference is informative. LaBSE has a more uniform distribution of attention probabilities as exhibited by the higher entropy compared to MEXMA. We provide examples of the distributions in Appendix G.

We perform an additional analysis to understand the importance and impact of the attention distribution for MEXMA and LaBSE. Concretely, we create sentence representations by averaging the tokens embeddings, i.e. using a completely uniform distribution. The results are provided in Table 8. The deltas are computed in terms of relative change from the uniform to the CLS representation. We can see that for all tasks, MEXMA has a larger change in performance compared to LaBSE, showing that indeed since our representations are more skewed, we suffer more from an increase in uniformity of the distribution. For those tasks, it is noticeable that MEXMA having a uniform distribution, will lose its ability to focus on the important tokens, decreasing its results. For LaBSE the decrease is not as accentuated, since it was already not focusing as much on the important tokens with its more uniform CLS pooling.

<table>
  <tr>
    <th>Model</th>
    <th>xsim ↓</th>
    <th>xsim++ ↓</th>
    <th>STS ↑</th>
    <th>Class ↑</th>
  </tr>
  <tr>
    <td>Uni-LaBSE</td>
    <td>2.02</td>
    <td>20.73</td>
    <td>63.50</td>
    <td>58.03</td>
  </tr>
  <tr>
    <td>Uni-MEXMA</td>
    <td>0.19</td>
    <td>18.21</td>
    <td>54.24</td>
    <td>56.98</td>
  </tr>
  <tr>
    <td>CLS-LaBSE</td>
    <td>0.92</td>
    <td>18.65</td>
    <td>64.65</td>
    <td>62.77</td>
  </tr>
  <tr>
    <td>CLS-MEXMA</td>
    <td>0.06</td>
    <td>9.60</td>
    <td>63.99</td>
    <td>65.35</td>
  </tr>
  <tr>
    <td>Δ LaBSE</td>
    <td>-119.65</td>
    <td>-11.19</td>
    <td>+1.78</td>
    <td>+7.55</td>
  </tr>
  <tr>
    <td>Δ MEXMA</td>
    <td>-212.50</td>
    <td>-89.73</td>
    <td>+15.24</td>
    <td>+12.81</td>
  </tr>
</table>

Table 8: Results for LaBSE and MEXMA, using a uniform (Uni-) or the CLS attention distribution. The last two rows provide the relative delta between the uniform and CLS distributions. STS and Classification results are across all datasets mentioned under Appendix E.

6 Conclusion

We introduced MEXMA, a novel multilingual alignment technique that leverages both token-level and sentence-level objectives. We show that integrating token-level objectives into the training of cross-lingual sentence encoders (CLSE) greatly improves sentence representation quality, outperforming current state-of-the-art pre-trained CLSE in bitext mining and other downstream tasks. We additionally validate these improvements via ablations. Notably, MEXMA also achieves strong token alignment across languages and effectively encodes meaningful information within each token. Since the sentence representation is built from these tokens, as we analysed, this leads to better sentence representations. Looking ahead, we plan to explore MEXMA’s scalability to more languages, and potentially modalities.
7 Limitations

This work covers 80 languages, which limits its usability compared to other approaches, namely SONAR (Duquenne et al., 2023), that have a wider language coverage. In this work we did not explore using other backbones supporting more languages, focusing only on using BERT-like models, since those are pre-trained with Masked Language Modeling, which is the task we also selected for our method. For future work, it would be relevant to carefully tune other backbones supporting more languages with our presented method.

In this work, the focus was on the method and architecture, consequently no specific data processing was performed. We opted to use the NLLB data (NLLB Team et al., 2022) as-is, without any additional processing or cleaning, which could be advantageous. To train our approach 300k steps with 1200 batch size was needed, a total of 360M sentences processed. There could be a focus on making the approach less reliant on large scales of data, however, since it is not a common trend on other approaches, we also decided not to make it a focus in our approach. It is, however, an issue worth addressing.

Comparing approaches using different backbones and different training data is not straightforward, and although in the main text we just compare our best model against other best models, in Appendix B.4, we provide fairer comparisons using the same backbones and training data.

References

Ahmed Abdelali, Francisco Guzman, Hassan Sajjad, and Stephan Vogel. 2014. The AMARA corpus: Building parallel language resources for the educational domain. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 1856–1862, Reykjavik, Iceland. European Language Resources Association (ELRA).

Antonios Anastasopoulos, Alessandro Cattelan, Zi-Yi Dou, Marcello Federico, Christian Federmann, Dmitriy Genzel, Francisco Guzmán, Junjie Hu, Macduff Hughes, Philipp Koehn, Rosie Lazar, Will Lewis, Graham Neubig, Mengmeng Niu, Alp Öktem, Eric Paquin, Grace Tang, and Sylwia Tur. 2020. TICO-19: the translation initiative for COVID-19. In Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020, Online. Association for Computational Linguistics.

Mikel Artetxe and Holger Schwenk. 2019a. Margin-based parallel corpus mining with multilingual sentence embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3197–3203, Florence, Italy. Association for Computational Linguistics.

Mikel Artetxe and Holger Schwenk. 2019b. Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond. Transactions of the Association for Computational Linguistics, 7:597–610.

Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. 2022. BFit: BERT pre-training of image transformers. In International Conference on Learning Representations.

Adrien Bardes, Quentin Garrido, Jean Ponce, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nicolas Ballas. 2024. Revisiting feature prediction for learning visual representations from video. arXiv:2404.08471.

Jan Beirlant, E. Dudewicz, L. Gyor, and E.C. Meulen. 1997. Nonparametric entropy estimation: An overview. International Journal of Mathematical and Statistical Sciences, 6.

Steven Cao, Nikita Kitaev, and Dan Klein. 2020. Multilingual alignment of contextual word representations. In International Conference on Learning Representations.

Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. 2018. Universal sentence encoder for English. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 169–174, Brussels, Belgium. Association for Computational Linguistics.

Mingda Chen, Kevin Heffernan, Onur Çelebi, Alexandre Mouchachko, and Holger Schwenk. 2023. xSIM++: An improved proxy to bitext mining performance for low-resource languages. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 101–109, Toronto, Canada. Association for Computational Linguistics.

Christos Christodoulopoulos and Mark Steedman. 2015. A massively parallel corpus: the bible in 100 languages. Language resources and evaluation, 49:375–395.

Alexis Conneau, Kartik Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451, Online. Association for Computational Linguistics.
Alexis Conneau and Douwe Kiela. 2018. Senteval: An evaluation toolkit for universal sentence representations. arXiv preprint arXiv:1803.05449.

Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670–680, Copenhagen, Denmark. Association for Computational Linguistics.

Timothée Darcel, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. 2024. Vision transformers need registers. Preprint, arXiv:2309.16588.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.

Paul-Ambroise Duquenne, Holger Schwenk, and Benoît Sagot. 2023. Sonar: Sentence-level multimodal and language-agnostic representations. Preprint, arXiv:2308.11466.

Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Çelebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. 2020. Beyond english-centric multilingual machine translation. Preprint, arXiv:2010.11125.

Shuai Fan, Chen Lin, Haonan Li, Zhenghao Lin, Jinsong Su, Hang Zhang, Yeyun Gong, Jian Guo, and Nan Duan. 2022. Sentiment-aware word and sentence level pre-training for sentiment analysis. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4984–4994, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Ariazhagan, and Wei Wang. 2022. Language-agnostic BERT sentence embedding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 878–891, Dublin, Ireland. Association for Computational Linguistics.

Nathan Godey, Éric Clergerie, and Benoît Sagot. 2024. Anisotropy Is Inherent to Self-Attention in Transformers. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 35–48. Association for Computational Linguistics.

Mandy Guo, Qinlan Shen, Yinfei Yang, Heming Ge, Daniel Cer, Gustavo Hernandez Abrego, Keith Stevens, Noah Constant, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018. Effective parallel corpus mining using bilingual sentence embeddings. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 165–176, Brussels, Belgium. Association for Computational Linguistics.

Raia Hadsell, Sumit Chopra, and Yann LeCun. 2006. Dimensionality reduction by learning an invariant mapping. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), pages 1735 – 1742.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. 2022. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16000–16009.

Kevin Heffernan, Onur Çelebi, and Holger Schwenk. 2022. Bitext mining using distilled sentence representations for low-resource languages. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2101–2112, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 4411–4421. PMLR.

Masoud Jalili Sabet, Philipp Dutter, François Yvon, and Hinrich Schütze. 2020. SimAlign: High quality word alignments without parallel training data using static and contextualized embeddings. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 1627–1643, Online. Association for Computational Linguistics.

Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2015. Skip-thought vectors. Preprint, arXiv:1506.06726.

Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of machine translation summit x: papers, pages 79–86.

Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, Daniel Haziza, Luca Wehrstedt, Jeremy Reizenstein, and Grigory Sizov. 2022. xformers: A modular and hackable transformer modelling library. https://github.com/facebookresearch/xformers.
Ziheng Li, Shaohan Huang, Zihan Zhang, Zhi-Hong Deng, Qiang Lou, Haizhen Huang, Jian Jiao, Furu Wei, Weiwei Deng, and Qi Zhang. 2023. Dual-alignment pre-training for cross-lingual sentence embedding. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3466–3478, Toronto, Canada. Association for Computational Linguistics.

Pierre Lison and Jörg Tiedemann. 2016. OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 923–929, Portorož, Slovenia. European Language Resources Association (ELRA).

David Mareček. 2008. Automatic alignment of teckogrammatical trees from czech-english parallel corpus. Master’s thesis, Charles University, MFF UK.

Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. MTEB: Massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2014–2037, Dubrovnik, Croatia. Association for Computational Linguistics.

NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejía Gonzalez, Prangthip Hansanti, John Hoffman, Semaier Jarret, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human-centered machine translation. Preprint, arXiv:2207.04672.

Franz Josef Och and Hermann Ney. 2000. Improved statistical alignment models. In Proceedings of the 38th annual meeting of the association for computational linguistics, pages 440–447.

Maxime Oquab, Timothée Darcet, Théo Moutakamni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. 2024. DINOV2: Learning robust visual features without supervision. Transactions on Machine Learning Research.

Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. CoRR, abs/1908.10084.

Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.

Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, and Hervé Jégou. 2019. Spreading vectors for similarity search. In International Conference on Learning Representations.

Tal Schuster, Ori Ram, Regina Barzilay, and Amir Globerson. 2019. Cross-lingual alignment of contextual word embeddings, with applications to zero-shot dependency parsing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1599–1613, Minneapolis, Minnesota. Association for Computational Linguistics.

Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, and Armand Joulin. 2020. Cematrix: Mining billions of high-quality parallel sentences on the web. Preprint, arXiv:1911.04944.

Ralf Steinberger, Andreas Eisele, Szymon Kloczek, Spyridon Pilos, and Patrick Schlüter. 2012. DGT-TM: A freely available translation memory in 22 languages. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 454–459, Istanbul, Turkey. European Language Resources Association (ELRA).

Mingjie Sun, Xinlei Chen, J. Zico Kolter, and Zhuang Liu. 2024. Massive activations in large language models. Preprint, arXiv:2402.17762.

Jörg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 2214–2218, Istanbul, Turkey. European Language Resources Association (ELRA).

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2019. Representation learning with contrastive predictive coding. Preprint, arXiv:1807.03748.

Jesse Vig. 2019. A multiscale visualization of attention in the transformer model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 37–42, Florence, Italy. Association for Computational Linguistics.

Xiangpeng Wei, Rongxiang Weng, Yue Hu, Luxi Xing, Heng Yu, and Weihua Luo. 2021. On learning universal representations across languages. Preprint, arXiv:2007.15960.

Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. Efficient streaming language models with attention sinks. Preprint, arXiv:2309.17453.
Shitao Xiao, Zheng Liu, Yingxia Shao, and Zhao Cao. 2022. RetroMAE: Pre-training retrieval-oriented language models via masked auto-encoder. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 538–548, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Yinfei Yang, Gustavo Hernandez Abrego, Steve Yuan, Mandy Guo, Qinlan Shen, Daniel Cer, Yun hsuan Sung, Brian Strope, and Ray Kurzweil. 2019. Improving multilingual sentence embedding using bi-directional dual encoder with additive margin softmax. Preprint, arXiv:1902.08564.

Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Ci-hang Xie, Alan Yuille, and Tao Kong. 2022. ibot: Image bert pre-training with online tokenizer. International Conference on Learning Representations (ICLR).

Michal Ziemski, Marcin Junczys-Downmunt, and Bruno Pouliquen. 2016. The United Nations parallel corpus v1.0. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 3530–3534, Portorož, Slovenia. European Language Resources Association (ELRA).

Pierre Zweigenbaum, Serge Sharoff, and Reinhard Rapp. 2017. Overview of the second BUCC shared task: Spotting parallel sentences in comparable corpora. In Proceedings of the 10th Workshop on Building and Using Comparable Corpora, pages 60–67, Vancouver, Canada. Association for Computational Linguistics.

Pierre Zweigenbaum, Serge Sharoff, and Reinhard Rapp. 2018. Overview of the Third BUCC Shared Task: Spotting Parallel Sentences in Comparable Corpora. In Workshop on Building and Using Comparable Corpora, Miyazaki, Japan.
A  Experimental Setup

A.1 KoLeo loss
We define below the KoLeo loss, \( L_{KoLeo} \), for a set of \( n \) representations, as well as the symmetrical version, \( L_K \), we use to train our models:

\[
\mathcal{L}_K = \mathcal{L}_{KoLeo}(S_A) + \mathcal{L}_{KoLeo}(S_B),
\]
with
\[
\mathcal{L}_{KoLeo} = - \frac{1}{n} \sum_{i=1}^n \log(d_{n,i})
\]
where \( d_{n,i} = min_{j \neq i} \| x_i - x_j \| \) is the distance between \( x_i \) and its nearest point in the batch.

A.2 Encoder backbone
The available implementation of XLM-RoBERTa in HuggingFace employs an inefficient attention mechanism, which we have modified to incorporate the memory-efficient attention from xFormers (Lefaudeux et al., 2022). This modification was necessary due to the random batching process used in our training, which results in a significant amount of padding and increased computational cost. To address this issue and eliminate padding, we have employed the BlockDiagonal-Mask \( ^2 \), which through custom CUDA kernels, avoids computations in padding altogether. With this change we are able to increase our batch size in each GPU by a factor of \( \approx 8 \).

A.3 Unmasking head
For the unmasking head, we use 6 transformer layers, also leveraging the memory-efficient attention.

A.4 Compute and training length
Our models were trained on a single node of 8 A100 GPUs. Each GPU had a batch size of 150, totalling 1,200 batch size across all GPUs. We accumulated two gradients, making our effective batch size 2,400. We trained our models for 300k steps.

A.5 Losses
Our models were trained with \( \alpha = 1, \beta = \frac{1}{2} \) and \( \gamma = \frac{0.01}{2} \).

A.6 Training parameters
We utilize the AdamW optimizer for our training process. The learning rate is linearly increased from 1e-5 for the 300k steps. To optimize memory usage, we employ mixed precision training, where the model is stored in float32, while most computations are performed in float16. The maximum sequence length for our input data is set to 200 tokens. Finally, we apply a masking ratio of 40% to the input data.

\( ^2 \)https://facebookresearch.github.io/xformers/components/ops.html#xformers.ops.fmha.attn_bias.BlockDiagonalMask
B Ablations

B.1 Model components
In Table 9, we ablate the different components of our architecture described in Section 3. We briefly explain each entry in the table. Model (1) has only two encoder instances, one for each language, where one of the inputs is masked, and the other is left clean. The token unmasking is performed with the clean sentence representation as context. The languages are randomly swapped at every new sample, to eliminate potential biases. The gradients from the unmasking task are only propagated back to the encoder via the sentence representation, and there is no gradient propagation from the individual tokens back to the encoder. There is also neither alignment nor kolo losses. Model (2) adds two additional encoder instances, totalling four instances, two for each language, where now each language has its clean and masked input. This allows to unmask language \( \mathcal{A} \) with language \( \mathcal{B} \), and vice-versa, and will also allow (once added) to align two clean sentence representations. Model (3) adds the alignment loss, but it is performed between the masked sentence representation of language \( \mathcal{A} \) and the clean sentence representation of language \( \mathcal{B} \), to better emphasize the advantages of having a symmetrical architecture with an alignment loss between two clean representations. Model (4) then changes the alignment loss to be performed between the two clean sentence representations of each language. In model (5) we allow the gradients from the unmasking to be propagated to the encoder via each individual token, as well as its sentence representation. Finally, model (6) adds the KoLeo loss.

The results indicate that each component always enhances performance on at least two out of the three tasks. Notably, the alignment loss, (3)-(4), and token-level gradients, (5), emerge as the most critical components. More precisely, the alignment loss yields substantial improvements on two tasks while also resulting in a notable decline in performance on another task. In contrast, the token-level gradients consistently provide significant performance gains across all three tasks.

B.2 Cross-linguality
In Table 10 we ablate the importance of cross-linguality in the unmasking. To conduct this experiment, we performed the unmasking using as context the sentence representation in the same language of the tokens being unmasked, instead of the representation in the opposite language. The large gap in the results shows the importance of doing the unmasking cross-lingually, as motivated in Section 3. The experiments were conducted using the same hyperparameters.

B.3 Masking ratio
Classical NLP masked encoders like BERT use a small masking percentage, usually \( \approx 15\% \), without aligning any augmentations. Recent vision approaches use much higher masking percentages. BEiT (Bao et al., 2022) was one of the first masked image modelling (MIM) approaches, in a BERT-style training, and masked 40%. MAE (He et al., 2022) is another BERT-like model for images, and masks 75%, but shows that even masking 80% or 90% still achieves good results. DINO v2 (Oquab et al., 2024) and I-BOT (Zhou et al., 2022) mask between 10%-50% in a block-wise masking scenario, aligning augmentations. I-BOT can use 65%-75% masking ratio, if randomly masking (instead of block-wise masking). For videos, V-JEPA (Bardes et al., 2024) masks with a very high percentage of 90%. Recent textual approaches, namely RetroMAE experiment with masking percentages of up to 50~70%, but this task will not update the actual encoder.

For MEXMA, since these masking gradients are updating our encoder, we need to strive for a balance where unmasking is hard, and cannot be done by the encoder and head, but also not too much that will deteriorate the representations of the encoder. Table 11 shows the results we obtained for the different masking ratios.

B.4 Fairer comparisons
In this section, we conduct additional experiments to enhance the fairness of our method comparisons, ensuring that all models are trained on the same data as MEXMA. Results are reported in Table 12. To facilitate a more equitable comparison with LaBSE, we re-implemented LaBSE using the same backbone as MEXMA, i.e. XLM-RoBERTa (XLM-R). The model was trained with the same hyperparameters reported in the original paper, specifically a 4k batch size (compared to the 1k batch size used in MEXMA). This different backbone and data led to improved performance over LabSE on the xsim and SentEval benchmarks, although it resulted in a significant decrease in performance on the xsim++ benchmark. For a more accurate
<table>
  <tr>
    <th>component</th>
    <th>xsim ↓</th>
    <th>xsim++ ↓</th>
    <th>SentEval ↑</th>
  </tr>
  <tr>
    <td>Non-symmetrical (1)</td>
    <td>0.09</td>
    <td>14.75</td>
    <td>84.68</td>
  </tr>
  <tr>
    <td>+ Symmetrical architecture (2)</td>
    <td>0.09<sup>0.00</sup></td>
    <td>14.39<sup>0.36</sup></td>
    <td>84.83<sup>0.15</sup></td>
  </tr>
  <tr>
    <td>+ Alignment loss (clean to dirty alignment) (3)</td>
    <td>0.21<sup>10.12</sup></td>
    <td>12.09<sup>4.23</sup></td>
    <td>85.61<sup>7.08</sup></td>
  </tr>
  <tr>
    <td>+ Clean to clean alignment (4)</td>
    <td>0.15<sup>0.06</sup></td>
    <td>11.37<sup>0.72</sup></td>
    <td>85.06<sup>0.55</sup></td>
  </tr>
  <tr>
    <td>+ Token-level grads (5)</td>
    <td>0.10<sup>0.05</sup></td>
    <td>9.67<sup>1.7</sup></td>
    <td>85.98<sup>0.92</sup></td>
  </tr>
  <tr>
    <td>+ KoLeo loss (6) - MEXMA</td>
    <td>0.06<sup>0.04</sup></td>
    <td>9.60<sup>0.07</sup></td>
    <td>86.38<sup>0.04</sup></td>
  </tr>
</table>

Table 9: Ablation study of the different components of the model. All experiments are conducted with the final parameters of the model, as reported in Section 3.1.

<table>
  <tr>
    <th>component</th>
    <th>xsim ↓</th>
    <th>xsim++ ↓</th>
    <th>SentEval ↑</th>
  </tr>
  <tr>
    <td>Same language unmasking</td>
    <td>21.83</td>
    <td>73.78</td>
    <td>80.34</td>
  </tr>
  <tr>
    <td>Cross lingual unmasking</td>
    <td>0.06<sup>21.77</sup></td>
    <td>9.60<sup>64.18</sup></td>
    <td>86.38<sup>16.04</sup></td>
  </tr>
</table>

Table 10: Ablation study of the importance of cross-lingual unmasking. All experiments are conducted with the final parameters of the model, as reported in Section 3.1.

<table>
  <tr>
    <th>Masking %</th>
    <th>xsim ↓</th>
    <th>xsim++ ↓</th>
    <th>SentEval ↑</th>
  </tr>
  <tr>
    <td>20%</td>
    <td>0.06</td>
    <td>10.50</td>
    <td>85.87</td>
  </tr>
  <tr>
    <td>30%</td>
    <td>0.06</td>
    <td>9.82</td>
    <td>86.00</td>
  </tr>
  <tr>
    <td>40%</td>
    <td>0.06</td>
    <td>9.60</td>
    <td>86.38</td>
  </tr>
  <tr>
    <td>50%</td>
    <td>0.07</td>
    <td>9.56</td>
    <td>86.37</td>
  </tr>
  <tr>
    <td>60%</td>
    <td>0.08</td>
    <td>9.79</td>
    <td>86.13</td>
  </tr>
  <tr>
    <td>70%</td>
    <td>0.09</td>
    <td>10.65</td>
    <td>86.41</td>
  </tr>
  <tr>
    <td>80%</td>
    <td>0.10</td>
    <td>12.81</td>
    <td>85.85</td>
  </tr>
  <tr>
    <td>90%</td>
    <td>0.11</td>
    <td>14.62</td>
    <td>84.99</td>
  </tr>
</table>

Table 11: The model performance across different masking ratios.

<table>
  <tr>
    <th>Model</th>
    <th>xsim ↓</th>
    <th>xsim++ ↓</th>
    <th>SentEval ↑</th>
  </tr>
  <tr>
    <td>XLM-R LaBSE</td>
    <td>0.10</td>
    <td>33.82</td>
    <td>86.08</td>
  </tr>
  <tr>
    <td>NLLB-MEXMA</td>
    <td>0.11</td>
    <td>23.36</td>
    <td>85.20</td>
  </tr>
</table>

<table>
  <tr>
    <th colspan="4">Baselines</th>
  </tr>
  <tr>
    <td>MEXMA</td>
    <td>0.06</td>
    <td>9.60</td>
    <td>86.36</td>
  </tr>
  <tr>
    <td>SONAR</td>
    <td>0.09</td>
    <td>12.08</td>
    <td>85.82</td>
  </tr>
  <tr>
    <td>LaBSE</td>
    <td>0.92</td>
    <td>18.65</td>
    <td>85.63</td>
  </tr>
</table>

Table 12: Fairer comparisons with same backbone to compare different strategies, all trained on the same data.

comparison with SONAR, we replaced the XLM-R backbone in MEXMA with the NLLB encoder. This approach is more straightforward than training SONAR with XLM-R, as SONAR requires a pre-trained translation decoder. The results, however, were inferior to those of MEXMA across all tasks, with larger gaps than SONAR. This outcome is expected, given that the NLLB model was not originally trained for masked language modeling.
C  Language information appendix

In this section, we cover the languages used by our model. The list of languages used to train our model is reported in Table 13. The list used to conduct the experiments with 90 languages is available in Table 14.
<table>
  <tr>
    <th>FLORES200 code</th>
    <th>Language</th>
    <th>FLORES200 code</th>
    <th>Language</th>
  </tr>
  <tr>
    <td>acm_Arab</td>
    <td>Mesopotamian Arabic</td>
    <td>kan_Knda</td>
    <td>Kannada</td>
  </tr>
  <tr>
    <td>aeb_Arab</td>
    <td>Tunisian Arabic</td>
    <td>kat_Geor</td>
    <td>Georgian</td>
  </tr>
  <tr>
    <td>afr_Latn</td>
    <td>Afrikaans</td>
    <td>kaz_Cyrl</td>
    <td>Kazakh</td>
  </tr>
  <tr>
    <td>amh_Ethi</td>
    <td>Amharic</td>
    <td>khm_Khmr</td>
    <td>Khmer</td>
  </tr>
  <tr>
    <td>ary_Arab</td>
    <td>Moroccan Arabic</td>
    <td>kir_Cyrl</td>
    <td>Kyrgyz</td>
  </tr>
  <tr>
    <td>arz_Arab</td>
    <td>Egyptian Arabic</td>
    <td>kor_Hang</td>
    <td>Korean</td>
  </tr>
  <tr>
    <td>asm_Beng</td>
    <td>Assamese</td>
    <td>lao_Lao0</td>
    <td>Lao</td>
  </tr>
  <tr>
    <td>azb_Arab</td>
    <td>South Azerbaijani</td>
    <td>mal_Mlym</td>
    <td>Malayalam</td>
  </tr>
  <tr>
    <td>azj_Latn</td>
    <td>Azerbaijani</td>
    <td>mar_Deva</td>
    <td>Marathi</td>
  </tr>
  <tr>
    <td>bel_Cyrl</td>
    <td>Belarusian</td>
    <td>mkd_Cyrl</td>
    <td>Macedonian</td>
  </tr>
  <tr>
    <td>ben_Beng</td>
    <td>Bengali</td>
    <td>mya_Mymr</td>
    <td>Burmese</td>
  </tr>
  <tr>
    <td>bos_Latn</td>
    <td>Bosnian</td>
    <td>nld_Latn</td>
    <td>Dutch</td>
  </tr>
  <tr>
    <td>bul_Cyrl</td>
    <td>Bulgarian</td>
    <td>nno_Latn</td>
    <td>Norwegian</td>
  </tr>
  <tr>
    <td>cat_Latn</td>
    <td>Catalan</td>
    <td>nob_Latn</td>
    <td>Norwegian Bokmål</td>
  </tr>
  <tr>
    <td>ces_Latn</td>
    <td>Czech</td>
    <td>npi_Deva</td>
    <td>Nepali</td>
  </tr>
  <tr>
    <td>ckb_Arab</td>
    <td>Central Kurdish</td>
    <td>pol_Latn</td>
    <td>Polish</td>
  </tr>
  <tr>
    <td>cym_Latn</td>
    <td>Welsh</td>
    <td>por_Latn</td>
    <td>Portuguese</td>
  </tr>
  <tr>
    <td>dan_Latn</td>
    <td>Danish</td>
    <td>ron_Latn</td>
    <td>Romanian</td>
  </tr>
  <tr>
    <td>deu_Latn</td>
    <td>German</td>
    <td>rus_Cyrl</td>
    <td>Russian</td>
  </tr>
  <tr>
    <td>ell_Grek</td>
    <td>Greek</td>
    <td>san_Deva</td>
    <td>Sanskrit</td>
  </tr>
  <tr>
    <td>eng_Latn</td>
    <td>English</td>
    <td>sin_Sinh</td>
    <td>Sinhala</td>
  </tr>
  <tr>
    <td>epo_Latn</td>
    <td>Esperanto</td>
    <td>slk_Latn</td>
    <td>Slovak</td>
  </tr>
  <tr>
    <td>est_Latn</td>
    <td>Estonian</td>
    <td>slv_Latn</td>
    <td>Slovenian</td>
  </tr>
  <tr>
    <td>eus_Latn</td>
    <td>Basque</td>
    <td>snd_Arab</td>
    <td>Sindhi</td>
  </tr>
  <tr>
    <td>fin_Latn</td>
    <td>Finnish</td>
    <td>som_Latn</td>
    <td>Somali</td>
  </tr>
  <tr>
    <td>fra_Latn</td>
    <td>French</td>
    <td>spa_Latn</td>
    <td>Spanish</td>
  </tr>
  <tr>
    <td>gla_Latn</td>
    <td>Scottish Gaelic</td>
    <td>srp_Cyrl</td>
    <td>Serbian</td>
  </tr>
  <tr>
    <td>gle_Latn</td>
    <td>Irish</td>
    <td>sun_Latn</td>
    <td>Sundanese</td>
  </tr>
  <tr>
    <td>glg_Latn</td>
    <td>Galician</td>
    <td>swe_Latn</td>
    <td>Swedish</td>
  </tr>
  <tr>
    <td>guj_Gujr</td>
    <td>Gujarati</td>
    <td>swh_Latn</td>
    <td>Swahili</td>
  </tr>
  <tr>
    <td>hau_Latn</td>
    <td>Hausa</td>
    <td>tam_Taml</td>
    <td>Tamil</td>
  </tr>
  <tr>
    <td>heb_Hebr</td>
    <td>Hebrew</td>
    <td>tel_Telu</td>
    <td>Telugu</td>
  </tr>
  <tr>
    <td>hin_Deva</td>
    <td>Hindi</td>
    <td>tha_Thai</td>
    <td>Thai</td>
  </tr>
  <tr>
    <td>hrv_Latn</td>
    <td>Croatian</td>
    <td>tur_Latn</td>
    <td>Turkish</td>
  </tr>
  <tr>
    <td>hun_Latn</td>
    <td>Hungarian</td>
    <td>uig_Arab</td>
    <td>Uyghur</td>
  </tr>
  <tr>
    <td>hye_Armn</td>
    <td>Armenian</td>
    <td>ukr_Cyrl</td>
    <td>Ukrainian</td>
  </tr>
  <tr>
    <td>ind_Latn</td>
    <td>Indonesian</td>
    <td>urd_Arab</td>
    <td>Urdu</td>
  </tr>
  <tr>
    <td>isl_Latn</td>
    <td>Icelandic</td>
    <td>vie_Latn</td>
    <td>Vietnamese</td>
  </tr>
  <tr>
    <td>ita_Latn</td>
    <td>Italian</td>
    <td>xho_Latn</td>
    <td>Xhosa</td>
  </tr>
  <tr>
    <td>jav_Latn</td>
    <td>Javanese</td>
    <td>zho_Hant</td>
    <td>Chinese (Traditional)</td>
  </tr>
  <tr>
    <td>jpn_Jpan</td>
    <td>Japanese</td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 13: 81 languages set.
<table>
  <tr>
    <th>FLORES200 code</th>
    <th>Language</th>
    <th>FLORES200 code</th>
    <th>Language</th>
  </tr>
  <tr><td>afr_Latn</td><td>Afrikaans</td><td>kmr_Latn</td><td>Kurdish (Kurmanji)</td></tr>
  <tr><td>als_Latn</td><td>Albanian</td><td>kor_Hang</td><td>Korean</td></tr>
  <tr><td>amh_Ethi</td><td>Amharic</td><td>lao_Lao0</td><td>Lao</td></tr>
  <tr><td>arb_Arab</td><td>Arabic</td><td>lit_Latn</td><td>Lithuanian</td></tr>
  <tr><td>asm_Beng</td><td>Assamese</td><td>lvv_Latn</td><td>Latvian</td></tr>
  <tr><td>azj_Latn</td><td>Azerbaijani</td><td>mal_Mlym</td><td>Malayalam</td></tr>
  <tr><td>bel_Cyril</td><td>Belarusian</td><td>mar_Deva</td><td>Marathi</td></tr>
  <tr><td>ben_Beng</td><td>Bengali</td><td>mkd_Cyril</td><td>Macedonian</td></tr>
  <tr><td>bos_Latn</td><td>Bosnian</td><td>mya_Mymr</td><td>Burmese</td></tr>
  <tr><td>bul_Cyril</td><td>Bulgarian</td><td>nld_Latn</td><td>Dutch</td></tr>
  <tr><td>cat_Latn</td><td>Catalan</td><td>nno_Latn</td><td>Norwegian</td></tr>
  <tr><td>ces_Latn</td><td>Czech</td><td>npi_Deva</td><td>Nepali</td></tr>
  <tr><td>cym_Latn</td><td>Welsh</td><td>ory_Orya</td><td>Oriya</td></tr>
  <tr><td>dan_Latn</td><td>Danish</td><td>pan_Guru</td><td>Punjabi</td></tr>
  <tr><td>deu_Latn</td><td>German</td><td>pbt_Arab</td><td>Pashto</td></tr>
  <tr><td>ell_Grek</td><td>Greek</td><td>plt_Latn</td><td>Malagasy</td></tr>
  <tr><td>eng_Latn</td><td>English</td><td>pol_Latn</td><td>Polish</td></tr>
  <tr><td>epo_Latn</td><td>Esperanto</td><td>por_Latn</td><td>Portuguese</td></tr>
  <tr><td>est_Latn</td><td>Estonian</td><td>prs_Arab</td><td>Persian</td></tr>
  <tr><td>eus_Latn</td><td>Basque</td><td>ron_Latn</td><td>Romanian</td></tr>
  <tr><td>fin_Latn</td><td>Finnish</td><td>rus_Cyril</td><td>Russian</td></tr>
  <tr><td>fra_Latn</td><td>French</td><td>san_Deva</td><td>Sanskrit</td></tr>
  <tr><td>gaz_Latn</td><td>Oromo</td><td>sin_Sinh</td><td>Sinhala</td></tr>
  <tr><td>gla_Latn</td><td>Gaelic</td><td>slk_Latn</td><td>Slovak</td></tr>
  <tr><td>gle_Latn</td><td>Irish</td><td>slv_Latn</td><td>Slovenian</td></tr>
  <tr><td>glg_Latn</td><td>Galician</td><td>snd_Arab</td><td>Sindhi</td></tr>
  <tr><td>guj_Gujr</td><td>Gujarati</td><td>som_Latn</td><td>Somali</td></tr>
  <tr><td>hau_Latn</td><td>Hausa</td><td>spa_Latn</td><td>Spanish</td></tr>
  <tr><td>heb_Hebr</td><td>Hebrew</td><td>srp_Cyril</td><td>Serbian</td></tr>
  <tr><td>hin_Deva</td><td>Hindi</td><td>sun_Latn</td><td>Sundanese</td></tr>
  <tr><td>hrv_Latn</td><td>Croatian</td><td>swe_Latn</td><td>Swedish</td></tr>
  <tr><td>hun_Latn</td><td>Hungarian</td><td>swh_Latn</td><td>Swahili</td></tr>
  <tr><td>hye_Armn</td><td>Armenian</td><td>tam_Taml</td><td>Tamil</td></tr>
  <tr><td>ind_Latn</td><td>Indonesian</td><td>tel_Telu</td><td>Telugu</td></tr>
  <tr><td>isl_Latn</td><td>Icelandic</td><td>tha_Thai</td><td>Thai</td></tr>
  <tr><td>ita_Latn</td><td>Italian</td><td>tur_Latn</td><td>Turkish</td></tr>
  <tr><td>jav_Latn</td><td>Javanese</td><td>uig_Arab</td><td>Uyghur</td></tr>
  <tr><td>jpn_Jpan</td><td>Japanese</td><td>ukr_Cyril</td><td>Ukrainian</td></tr>
  <tr><td>kan_Knda</td><td>Kannada</td><td>urd_Arab</td><td>Urdu</td></tr>
  <tr><td>kat_Geor</td><td>Georgian</td><td>uzn_Latn</td><td>Uzbek</td></tr>
  <tr><td>kaz_Cyril</td><td>Kazakh</td><td>vie_Latn</td><td>Vietnamese</td></tr>
  <tr><td>khk_Cyril</td><td>Mongolian</td><td>xho_Latn</td><td>Xhosa</td></tr>
  <tr><td>khm_Khmr</td><td>Khmer</td><td>ydd_Hebr</td><td>Yiddish</td></tr>
  <tr><td>kir_Cyril</td><td>Kyrgyz</td><td>zho_Hans</td><td>Chinese (Simplified)</td></tr>
  <tr><td>zsm_Latn</td><td>Malay</td><td>zho_Hant</td><td>Chinese (Traditional)</td></tr>
</table>

Table 14: 90 languages set
D  Datasets

In this section we report the data used to train our models. Table 15 reports all the datasets used to train the models.
<table>
  <tr>
    <th>Dataset</th>
    <th>Source</th>
    <th>Origin</th>
  </tr>
  <tr>
    <td>bible-uedin</td>
    <td>Opus</td>
    <td><a href="https://www.aclweb.org/anthology/W15-4623">Christodouloupoulos and Steedman (2015); Tiedemann (2012)</a></td>
  </tr>
  <tr>
    <td>DGT</td>
    <td>Opus</td>
    <td>Steinberger et al. (2012); Tiedemann (2012)</td>
  </tr>
  <tr>
    <td>ECB</td>
    <td>Opus</td>
    <td>Tiedemann (2012)</td>
  </tr>
  <tr>
    <td>EMEA</td>
    <td>Opus</td>
    <td>Tiedemann (2012)</td>
  </tr>
  <tr>
    <td>EUbookshop</td>
    <td>Opus</td>
    <td>Tiedemann (2012)</td>
  </tr>
  <tr>
    <td>infopankki</td>
    <td>Opus</td>
    <td>Tiedemann (2012)</td>
  </tr>
  <tr>
    <td>memat</td>
    <td>Opus</td>
    <td>Tiedemann (2012)</td>
  </tr>
  <tr>
    <td>OpenSubtitles</td>
    <td>Opus</td>
    <td>Lison and Tiedemann (2016); Tiedemann (2012), <a href="https://opus.nlpl.eu/OpenSubtitles.php">Link: opus.nlpl.eu/OpenSubtitles.php</a></td>
  </tr>
  <tr>
    <td>QED</td>
    <td>Opus</td>
    <td>(Abdelali et al., 2014; Tiedemann, 2012)</td>
  </tr>
  <tr>
    <td>Tanzil</td>
    <td>Opus</td>
    <td>Tiedemann (2012), <a href="https://tanzil.net/trans">Link: tanzil.net/trans</a></td>
  </tr>
  <tr>
    <td>Tatoeba</td>
    <td>Opus</td>
    <td>Tiedemann (2012)</td>
  </tr>
  <tr>
    <td>Ted20</td>
    <td>Opus</td>
    <td>Reimers and Gurevych (2020); Tiedemann (2012)</td>
  </tr>
  <tr>
    <td>Tico19</td>
    <td>Opus</td>
    <td>Anastasopoulos et al. (2020); Tiedemann (2012)</td>
  </tr>
  <tr>
    <td>UNPC</td>
    <td>Opus</td>
    <td>Ziemski et al. (2016); Tiedemann (2012)</td>
  </tr>
  <tr>
    <td>Wikimedia</td>
    <td>Opus</td>
    <td>Tiedemann (2012)</td>
  </tr>
  <tr>
    <td>NLLB mined</td>
    <td>Opus</td>
    <td>Schwenk et al. (2020); Fan et al. (2020); Tiedemann (2012)</td>
  </tr>
</table>

Table 15: Datasets used to train our models.
E  Detailed results

In this section, we report the detailed scores of the MTEB benchmark reported in Section 4, as well as the remaining benchmarks. We report the scores per task, with every dataset used per task, and per language. MEXMA is able to outperform the previous SOTA results on mining, while also improving the downstream results on classification and pair classification. LaBSE outperforms all other models on STS.

E.1 Bitext Mining

Results for mining are in Table 16, for the BUCC dataset. We report the scores on the four available languages: German, French, Russian and Chinese. We evaluate our model using xsim and xsim++ on the FLORES200 dataset, covering the 81 languages supported by our model (listed in Appendix C). For fairer comparison, we also report results for the 72 languages supported by LaBSE, SONAR, and MEXMA ("o-xsim"), and separately for the 34 languages common to DAP and the other models ("d-xsim"). The results are available in Table 17. Results on all languages covered by MEXMA for xsim and xsim++ on FLORES200 are provided in Tables 18 and 19, respectively.

<table>
  <tr>
    <th>LP</th>
    <th>DAP</th>
    <th>SONAR</th>
    <th>LaBSE</th>
    <th>MEXMA</th>
  </tr>
  <tr>
    <td>de-en</td>
    <td>99.45</td>
    <td>98.82</td>
    <td>99.35</td>
    <td>99.52</td>
  </tr>
  <tr>
    <td>fr-en</td>
    <td>98.58</td>
    <td>98.09</td>
    <td>98.72</td>
    <td>98.98</td>
  </tr>
  <tr>
    <td>ru-en</td>
    <td>97.74</td>
    <td>97.37</td>
    <td>97.78</td>
    <td>98.06</td>
  </tr>
  <tr>
    <td>zh-en</td>
    <td>98.96</td>
    <td>98.72</td>
    <td>99.16</td>
    <td>99.18</td>
  </tr>
</table>

Table 16: BUCC results for each language pair (LP).

E.2 Classification

Classification results for English are available in Table 20, for SentEval, and in Table 21 for the English MTEB classification datasets. Classification results for Chinese, French, Danish, Norwegian and Polish are reported in Table 22, Table 23, Table 24, Table 25, Table 26, respectively. MEXMA outperforms all other models on average.

E.3 Pair Classification

Pair classification results for English, French and Chinese are reported in Table 27, Table 28, and Table 29, respectively. MEXMA outperforms all other models on average.

E.4 Semantic Textual Similarity (STS)

Semantic Textual Similarity (STS) results are reported in Table 30, Table 32, Table 33 and Table 31 for English, French, Polish and Chinese, respectively. LaBSE outperforms MEXMA and the remaining models on STS. MEXMA and LaBSE outperform SONAR by large margins.
<table>
  <tr>
    <th>Model</th>
    <th>xsim ↓</th>
    <th>xsim++ ↓</th>
    <th>BUCC ↑</th>
    <th>o-xsim ↓</th>
    <th>o-xsim++ ↓</th>
    <th>d-xsim ↓</th>
    <th>d-xsim++ ↓</th>
  </tr>
  <tr>
    <td>DAP</td>
    <td>-</td>
    <td>-</td>
    <td>98.68</td>
    <td>-</td>
    <td>-</td>
    <td>2.90</td>
    <td>32.82</td>
  </tr>
  <tr>
    <td>SONAR</td>
    <td>0.09</td>
    <td>12.08</td>
    <td>98.25</td>
    <td>0.08</td>
    <td>11.68</td>
    <td>0.04</td>
    <td>10.55</td>
  </tr>
  <tr>
    <td>LaBSE</td>
    <td>0.92</td>
    <td>18.65</td>
    <td>98.75</td>
    <td>0.31</td>
    <td>16.21</td>
    <td>0.26</td>
    <td>14.51</td>
  </tr>
  <tr>
    <td>MEXMA</td>
    <td><b>0.06</b></td>
    <td><b>9.60</b></td>
    <td><b>98.93</b></td>
    <td><b>0.05</b></td>
    <td><b>9.01</b></td>
    <td><b>0.02</b></td>
    <td><b>8.26</b></td>
  </tr>
</table>

Table 17: Results in mining (%). xsim and xsim++ are computed on 81 languages (FLORES200 dataset, X-eng pairs), with o-… columns showing results for 72 supported languages from LaBSE and d-… columns showing results for 34 languages supported by DAP. BUCC is computed with F1 in its 4 languages.

<table>
  <tr>
    <th>Language</th>
    <th>SONAR</th>
    <th>LaBSE</th>
    <th>MEXMA</th>
    <th>DAP</th>
    <th>Language</th>
    <th>SONAR</th>
    <th>LaBSE</th>
    <th>MEXMA</th>
    <th>DAP</th>
  </tr>
  <tr>
    <td>acm_Arab</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>-</td>
    <td>kan_Knda</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>-</td>
  </tr>
  <tr>
    <td>aeb_Arab</td>
    <td>0.10</td>
    <td>0.10</td>
    <td>0.10</td>
    <td>-</td>
    <td>kat_Geor</td>
    <td>0.40</td>
    <td>0</td>
    <td>0</td>
    <td>7.41</td>
  </tr>
  <tr>
    <td>afr_Latn</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>0.10</td>
    <td>kaz_Cyril</td>
    <td>0.20</td>
    <td>0.20</td>
    <td>0.20</td>
    <td>44.96</td>
  </tr>
  <tr>
    <td>amh_Ethi</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>-</td>
    <td>khm_Khmr</td>
    <td>0</td>
    <td>2.08</td>
    <td>0</td>
    <td>-</td>
  </tr>
  <tr>
    <td>ary_Arab</td>
    <td>0.79</td>
    <td>1.09</td>
    <td>0.89</td>
    <td>-</td>
    <td>kir_Cyril</td>
    <td>0.10</td>
    <td>0</td>
    <td>0</td>
    <td>-</td>
  </tr>
  <tr>
    <td>arz_Arab</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>-</td>
    <td>kor_Hang</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>asm_Beng</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>-</td>
    <td>lao_Laoo</td>
    <td>0</td>
    <td>2.77</td>
    <td>0</td>
    <td>0.20</td>
  </tr>
  <tr>
    <td>azb_Arab</td>
    <td>1.68</td>
    <td>9.58</td>
    <td>0.99</td>
    <td>-</td>
    <td>mal_Mlym</td>
    <td>0.10</td>
    <td>0.10</td>
    <td>0.10</td>
    <td>1.48</td>
  </tr>
  <tr>
    <td>azj_Latn</td>
    <td>0.20</td>
    <td>0.10</td>
    <td>0.10</td>
    <td>-</td>
    <td>mar_Deva</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>1.38</td>
  </tr>
  <tr>
    <td>bel_Cyril</td>
    <td>0.30</td>
    <td>0</td>
    <td>0</td>
    <td>-</td>
    <td>mkd_Cyril</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>-</td>
  </tr>
  <tr>
    <td>ben_Beng</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>mya_Mymr</td>
    <td>0.20</td>
    <td>0.30</td>
    <td>0.20</td>
    <td>-</td>
  </tr>
  <tr>
    <td>bos_Latin</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>-</td>
    <td>nld_Latin</td>
    <td>0.10</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>bul_Cyril</td>
    <td>0.10</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>nno_Latin</td>
    <td>0.10</td>
    <td>0</td>
    <td>0.10</td>
    <td>-</td>
  </tr>
  <tr>
    <td>cat_Latn</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>-</td>
    <td>nob_Latin</td>
    <td>0.10</td>
    <td>0.10</td>
    <td>0.10</td>
    <td>-</td>
  </tr>
  <tr>
    <td>ces_Latin</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>-</td>
    <td>npi_Deva</td>
    <td>0.40</td>
    <td>0.30</td>
    <td>0.30</td>
    <td>-</td>
  </tr>
  <tr>
    <td>ckb_Arab</td>
    <td>0.10</td>
    <td>49.11</td>
    <td>0</td>
    <td>-</td>
    <td>pol_Latin</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>-</td>
  </tr>
  <tr>
    <td>cym_Latin</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>-</td>
    <td>por_Latin</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>-</td>
  </tr>
  <tr>
    <td>dan_Latin</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>-</td>
    <td>ron_Latin</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>-</td>
  </tr>
  <tr>
    <td>deu_Latin</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>rus_Cyril</td>
    <td>0.10</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>ell_Grek</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>0.10</td>
    <td>san_Deva</td>
    <td>0.50</td>
    <td>0.79</td>
    <td>0.40</td>
    <td>-</td>
  </tr>
  <tr>
    <td>epo_Latin</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>-</td>
    <td>sin_Sinh</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>-</td>
  </tr>
  <tr>
    <td>est_Latin</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>slk_Latin</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>-</td>
  </tr>
  <tr>
    <td>eus_Latin</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>sly_Latin</td>
    <td>0.10</td>
    <td>0</td>
    <td>0</td>
    <td>-</td>
  </tr>
  <tr>
    <td>fin_Latin</td>
    <td>0.10</td>
    <td>0.10</td>
    <td>0.10</td>
    <td>0.10</td>
    <td>snd_Arab</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>-</td>
  </tr>
  <tr>
    <td>fra_Latin</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>som_Latin</td>
    <td>0.10</td>
    <td>0.20</td>
    <td>0.10</td>
    <td>-</td>
  </tr>
  <tr>
    <td>gla_Latin</td>
    <td>0.10</td>
    <td>0.10</td>
    <td>0.10</td>
    <td>-</td>
    <td>spa_Latin</td>
    <td>0.10</td>
    <td>0.10</td>
    <td>0.10</td>
    <td>0.10</td>
  </tr>
  <tr>
    <td>gle_Latin</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>-</td>
    <td>srp_Cyril</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>-</td>
  </tr>
  <tr>
    <td>glg_Latin</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>-</td>
    <td>sun_Latin</td>
    <td>0.10</td>
    <td>0.10</td>
    <td>0.10</td>
    <td>-</td>
  </tr>
  <tr>
    <td>guj_Gujr</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>-</td>
    <td>swe_Latin</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>-</td>
  </tr>
  <tr>
    <td>hau_Latin</td>
    <td>0.30</td>
    <td>0.30</td>
    <td>0.30</td>
    <td>-</td>
    <td>swh_Latin</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>heb_Hebr</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>tam_Taml</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>28.26</td>
  </tr>
  <tr>
    <td>hin_Deva</td>
    <td>0.10</td>
    <td>0</td>
    <td>0</td>
    <td>0.10</td>
    <td>tel_Telu</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>2.77</td>
  </tr>
  <tr>
    <td>hrv_Latin</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>-</td>
    <td>tha_Thai</td>
    <td>0</td>
    <td>5.53</td>
    <td>0.10</td>
    <td>0.10</td>
  </tr>
  <tr>
    <td>hun_Latin</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>tur_Latin</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>hye_Armn</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>-</td>
    <td>uig_Arab</td>
    <td>0.10</td>
    <td>0.10</td>
    <td>0.10</td>
    <td>-</td>
  </tr>
  <tr>
    <td>ind_Latin</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>ukr_Cyril</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>-</td>
  </tr>
  <tr>
    <td>isl_Latin</td>
    <td>0.20</td>
    <td>0.10</td>
    <td>0.10</td>
    <td>-</td>
    <td>urd_Arab</td>
    <td>0.10</td>
    <td>0.10</td>
    <td>0.10</td>
    <td>0.30</td>
  </tr>
  <tr>
    <td>ita_Latin</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>vie_Latin</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>jav_Latin</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>11.17</td>
    <td>xho_Latin</td>
    <td>0.10</td>
    <td>0.10</td>
    <td>0.10</td>
    <td>-</td>
  </tr>
  <tr>
    <td>jpn_Jpan</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>zho_Hant</td>
    <td>0.10</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
  </tr>
</table>

Table 18: xsim results for each language in FLORES200 covered by MEXMA.
<table>
  <tr>
    <th>Language</th>
    <th>SONAR</th>
    <th>LaBSE</th>
    <th>MEXMA</th>
    <th>DAP</th>
    <th>Language</th>
    <th>SONAR</th>
    <th>LaBSE</th>
    <th>MEXMA</th>
    <th>DAP</th>
  </tr>
  <tr>
    <td>acm_Arab</td>
    <td>13.54</td>
    <td>28.56</td>
    <td>12.35</td>
    <td>-</td>
    <td>kan_Knda</td>
    <td>16.21</td>
    <td>18.38</td>
    <td>10.77</td>
    <td>-</td>
  </tr>
  <tr>
    <td>aeb_Arab</td>
    <td>14.23</td>
    <td>35.38</td>
    <td>14.82</td>
    <td>-</td>
    <td>kat_Geor</td>
    <td>16.01</td>
    <td>18.48</td>
    <td>11.66</td>
    <td>69.66</td>
  </tr>
  <tr>
    <td>afr_Latn</td>
    <td>6.62</td>
    <td>9.39</td>
    <td>5.63</td>
    <td>20.75</td>
    <td>kaz_CyrI</td>
    <td>12.55</td>
    <td>15.32</td>
    <td>8.89</td>
    <td>89.72</td>
  </tr>
  <tr>
    <td>amh_Ethi</td>
    <td>11.56</td>
    <td>19.07</td>
    <td>7.51</td>
    <td>-</td>
    <td>khm_Khmr</td>
    <td>14.72</td>
    <td>20.55</td>
    <td>9.39</td>
    <td>-</td>
  </tr>
  <tr>
    <td>ary_Arab</td>
    <td>15.91</td>
    <td>44.47</td>
    <td>25.59</td>
    <td>-</td>
    <td>kir_CyrI</td>
    <td>15.12</td>
    <td>20.55</td>
    <td>13.04</td>
    <td>-</td>
  </tr>
  <tr>
    <td>arz_Arab</td>
    <td>13.93</td>
    <td>31.03</td>
    <td>13.24</td>
    <td>-</td>
    <td>kor_Hang</td>
    <td>14.82</td>
    <td>18.58</td>
    <td>9.19</td>
    <td>-</td>
  </tr>
  <tr>
    <td>asm_Beng</td>
    <td>17.98</td>
    <td>41.11</td>
    <td>13.44</td>
    <td>-</td>
    <td>lao_Laoo</td>
    <td>10.18</td>
    <td>18.77</td>
    <td>7.41</td>
    <td>42.19</td>
  </tr>
  <tr>
    <td>azb_Arab</td>
    <td>45.26</td>
    <td>69.17</td>
    <td>33.00</td>
    <td>-</td>
    <td>mal_Mlym</td>
    <td>13.14</td>
    <td>19.96</td>
    <td>11.17</td>
    <td>54.35</td>
  </tr>
  <tr>
    <td>azj_Latn</td>
    <td>17.69</td>
    <td>17.69</td>
    <td>12.35</td>
    <td>-</td>
    <td>mar_Deva</td>
    <td>10.97</td>
    <td>15.42</td>
    <td>8.00</td>
    <td>54.45</td>
  </tr>
  <tr>
    <td>bel_CyrI</td>
    <td>20.26</td>
    <td>21.94</td>
    <td>13.44</td>
    <td>-</td>
    <td>mkd_CyrI</td>
    <td>7.51</td>
    <td>11.86</td>
    <td>6.42</td>
    <td>-</td>
  </tr>
  <tr>
    <td>ben_Beng</td>
    <td>13.83</td>
    <td>17.79</td>
    <td>8.70</td>
    <td>33.60</td>
    <td>mya_Mymr</td>
    <td>19.66</td>
    <td>28.06</td>
    <td>15.91</td>
    <td>-</td>
  </tr>
  <tr>
    <td>bos_Latn</td>
    <td>7.61</td>
    <td>8.10</td>
    <td>5.24</td>
    <td>-</td>
    <td>nld_Latn</td>
    <td>13.34</td>
    <td>13.34</td>
    <td>10.08</td>
    <td>20.45</td>
  </tr>
  <tr>
    <td>bul_CyrI</td>
    <td>9.19</td>
    <td>9.19</td>
    <td>5.53</td>
    <td>17.89</td>
    <td>nno_Latn</td>
    <td>16.80</td>
    <td>13.24</td>
    <td>8.30</td>
    <td>-</td>
  </tr>
  <tr>
    <td>cat_Latn</td>
    <td>6.03</td>
    <td>8.79</td>
    <td>5.04</td>
    <td>-</td>
    <td>nob_Latn</td>
    <td>15.51</td>
    <td>11.56</td>
    <td>7.41</td>
    <td>-</td>
  </tr>
  <tr>
    <td>ces_Latn</td>
    <td>8.20</td>
    <td>11.76</td>
    <td>6.72</td>
    <td>-</td>
    <td>npi_Deva</td>
    <td>14.53</td>
    <td>13.74</td>
    <td>7.61</td>
    <td>-</td>
  </tr>
  <tr>
    <td>ckb_Arab</td>
    <td>13.64</td>
    <td>93.97</td>
    <td>14.03</td>
    <td>-</td>
    <td>pol_Latn</td>
    <td>11.17</td>
    <td>12.65</td>
    <td>8.70</td>
    <td>-</td>
  </tr>
  <tr>
    <td>cym_Latn</td>
    <td>7.61</td>
    <td>14.03</td>
    <td>5.43</td>
    <td>-</td>
    <td>por_Latn</td>
    <td>5.93</td>
    <td>9.09</td>
    <td>6.32</td>
    <td>14.53</td>
  </tr>
  <tr>
    <td>dan_Latn</td>
    <td>6.03</td>
    <td>8.10</td>
    <td>4.84</td>
    <td>-</td>
    <td>ron_Latn</td>
    <td>8.10</td>
    <td>8.40</td>
    <td>5.73</td>
    <td>-</td>
  </tr>
  <tr>
    <td>deu_Latn</td>
    <td>6.13</td>
    <td>7.61</td>
    <td>6.13</td>
    <td>15.22</td>
    <td>rus_CyrI</td>
    <td>7.91</td>
    <td>9.98</td>
    <td>6.23</td>
    <td>19.17</td>
  </tr>
  <tr>
    <td>ell_Grek</td>
    <td>10.57</td>
    <td>16.40</td>
    <td>8.99</td>
    <td>26.58</td>
    <td>san_Deva</td>
    <td>24.41</td>
    <td>51.09</td>
    <td>22.33</td>
    <td>-</td>
  </tr>
  <tr>
    <td>epo_Latn</td>
    <td>6.13</td>
    <td>9.49</td>
    <td>5.63</td>
    <td>-</td>
    <td>sin_Sinh</td>
    <td>12.15</td>
    <td>16.01</td>
    <td>7.91</td>
    <td>-</td>
  </tr>
  <tr>
    <td>est_Latn</td>
    <td>8.10</td>
    <td>11.46</td>
    <td>5.93</td>
    <td>18.87</td>
    <td>slk_Latn</td>
    <td>8.99</td>
    <td>10.77</td>
    <td>7.51</td>
    <td>-</td>
  </tr>
  <tr>
    <td>eus_Latn</td>
    <td>10.87</td>
    <td>15.32</td>
    <td>8.30</td>
    <td>25.20</td>
    <td>slv_Latn</td>
    <td>9.58</td>
    <td>11.56</td>
    <td>6.62</td>
    <td>-</td>
  </tr>
  <tr>
    <td>fin_Latn</td>
    <td>8.99</td>
    <td>13.44</td>
    <td>8.50</td>
    <td>20.55</td>
    <td>snd_Arab</td>
    <td>13.64</td>
    <td>28.85</td>
    <td>9.68</td>
    <td>-</td>
  </tr>
  <tr>
    <td>fra_Latn</td>
    <td>5.93</td>
    <td>7.61</td>
    <td>5.34</td>
    <td>17.59</td>
    <td>som_Latn</td>
    <td>15.81</td>
    <td>30.93</td>
    <td>14.92</td>
    <td>-</td>
  </tr>
  <tr>
    <td>gla_Latn</td>
    <td>17.19</td>
    <td>23.62</td>
    <td>12.25</td>
    <td>-</td>
    <td>spa_Latn</td>
    <td>9.49</td>
    <td>11.07</td>
    <td>7.71</td>
    <td>20.55</td>
  </tr>
  <tr>
    <td>gle_Latn</td>
    <td>10.57</td>
    <td>15.81</td>
    <td>9.68</td>
    <td>-</td>
    <td>srp_CyrI</td>
    <td>6.92</td>
    <td>9.98</td>
    <td>5.34</td>
    <td>-</td>
  </tr>
  <tr>
    <td>glg_Latn</td>
    <td>7.51</td>
    <td>8.40</td>
    <td>5.63</td>
    <td>-</td>
    <td>sun_Latn</td>
    <td>15.02</td>
    <td>16.50</td>
    <td>10.38</td>
    <td>-</td>
  </tr>
  <tr>
    <td>guj_Gujr</td>
    <td>11.56</td>
    <td>15.12</td>
    <td>8.30</td>
    <td>-</td>
    <td>swe_Latn</td>
    <td>8.00</td>
    <td>8.99</td>
    <td>6.03</td>
    <td>-</td>
  </tr>
  <tr>
    <td>hau_Latn</td>
    <td>16.40</td>
    <td>25.99</td>
    <td>13.44</td>
    <td>-</td>
    <td>swh_Latn</td>
    <td>7.11</td>
    <td>15.71</td>
    <td>8.89</td>
    <td>29.05</td>
  </tr>
  <tr>
    <td>heb_Hebr</td>
    <td>6.92</td>
    <td>15.02</td>
    <td>7.51</td>
    <td>26.28</td>
    <td>tam_Taml</td>
    <td>15.61</td>
    <td>18.48</td>
    <td>11.26</td>
    <td>81.32</td>
  </tr>
  <tr>
    <td>hin_Deva</td>
    <td>9.58</td>
    <td>10.97</td>
    <td>6.92</td>
    <td>29.74</td>
    <td>tel_Telu</td>
    <td>13.83</td>
    <td>15.12</td>
    <td>10.87</td>
    <td>57.02</td>
  </tr>
  <tr>
    <td>hrv_Latn</td>
    <td>8.20</td>
    <td>9.09</td>
    <td>6.52</td>
    <td>-</td>
    <td>tha_Thai</td>
    <td>10.57</td>
    <td>28.16</td>
    <td>8.20</td>
    <td>30.83</td>
  </tr>
  <tr>
    <td>hun_Latn</td>
    <td>9.09</td>
    <td>13.74</td>
    <td>7.91</td>
    <td>17.79</td>
    <td>tur_Latn</td>
    <td>8.60</td>
    <td>10.87</td>
    <td>7.51</td>
    <td>18.38</td>
  </tr>
  <tr>
    <td>hye_Armn</td>
    <td>7.51</td>
    <td>12.94</td>
    <td>9.09</td>
    <td>-</td>
    <td>uig_Arab</td>
    <td>16.70</td>
    <td>23.12</td>
    <td>13.74</td>
    <td>-</td>
  </tr>
  <tr>
    <td>ind_Latn</td>
    <td>6.23</td>
    <td>9.09</td>
    <td>5.73</td>
    <td>14.92</td>
    <td>ukr_CyrI</td>
    <td>10.08</td>
    <td>12.25</td>
    <td>7.61</td>
    <td>-</td>
  </tr>
  <tr>
    <td>isl_Latn</td>
    <td>10.38</td>
    <td>14.43</td>
    <td>8.50</td>
    <td>-</td>
    <td>urd_Arab</td>
    <td>12.25</td>
    <td>16.70</td>
    <td>10.08</td>
    <td>47.13</td>
  </tr>
  <tr>
    <td>ita_Latn</td>
    <td>9.98</td>
    <td>9.49</td>
    <td>6.23</td>
    <td>16.11</td>
    <td>vie_Latn</td>
    <td>7.41</td>
    <td>12.15</td>
    <td>7.61</td>
    <td>18.58</td>
  </tr>
  <tr>
    <td>jav_Latn</td>
    <td>13.74</td>
    <td>17.09</td>
    <td>9.88</td>
    <td>63.04</td>
    <td>xho_Latn</td>
    <td>11.96</td>
    <td>31.42</td>
    <td>15.61</td>
    <td>-</td>
  </tr>
  <tr>
    <td>jpn_Jpan</td>
    <td>15.22</td>
    <td>17.79</td>
    <td>10.08</td>
    <td>27.17</td>
    <td>zho_Hant</td>
    <td>17.89</td>
    <td>24.60</td>
    <td>12.55</td>
    <td>28.56</td>
  </tr>
</table>

Table 19: xsim++ results for each language in FLORES200 covered by MEXMA.
<table>
  <tr>
    <th>Task</th>
    <th>DAP</th>
    <th>SONAR</th>
    <th>LaBSE</th>
    <th>MEXMA</th>
  </tr>
  <tr>
    <td>Average</td>
    <td>78.18</td>
    <td>85.82</td>
    <td>85.63</td>
    <td>86.38</td>
  </tr>
  <tr>
    <td>MR</td>
    <td>74.33</td>
    <td>81.23</td>
    <td>78.89</td>
    <td>80.14</td>
  </tr>
  <tr>
    <td>SST2</td>
    <td>81.88</td>
    <td>86.49</td>
    <td>83.64</td>
    <td>86.16</td>
  </tr>
  <tr>
    <td>TREC</td>
    <td>75.00</td>
    <td>95.00</td>
    <td>92.80</td>
    <td>94.80</td>
  </tr>
  <tr>
    <td>CR</td>
    <td>78.70</td>
    <td>85.67</td>
    <td>86.44</td>
    <td>84.43</td>
  </tr>
  <tr>
    <td>SUBJ</td>
    <td>91.83</td>
    <td>93.70</td>
    <td>93.14</td>
    <td>94.27</td>
  </tr>
  <tr>
    <td>MPQA</td>
    <td>78.86</td>
    <td>89.38</td>
    <td>89.66</td>
    <td>89.41</td>
  </tr>
  <tr>
    <td>MRPC</td>
    <td>66.67</td>
    <td>69.28</td>
    <td>74.84</td>
    <td>75.42</td>
  </tr>
</table>

Table 20: SentEval results.

<table>
  <tr>
    <th>Dataset</th>
    <th>DAP</th>
    <th>SONAR</th>
    <th>LaBSE</th>
    <th>MEXMA</th>
  </tr>
  <tr>
    <td>Average</td>
    <td>66.35</td>
    <td>65.63</td>
    <td>66.75</td>
    <td>68.20</td>
  </tr>
  <tr>
    <td>AmazonCounterfactualClassification</td>
    <td>77.16</td>
    <td>81.49</td>
    <td>75.93</td>
    <td>78.06</td>
  </tr>
  <tr>
    <td>AmazonPolarityClassification</td>
    <td>65.73</td>
    <td>62.73</td>
    <td>68.95</td>
    <td>64.96</td>
  </tr>
  <tr>
    <td>AmazonReviewsClassification</td>
    <td>34.03</td>
    <td>31.55</td>
    <td>35.80</td>
    <td>32.77</td>
  </tr>
  <tr>
    <td>Banking77Classification</td>
    <td>71.83</td>
    <td>73.50</td>
    <td>69.85</td>
    <td>75.14</td>
  </tr>
  <tr>
    <td>ImdbClassification</td>
    <td>62.06</td>
    <td>55.75</td>
    <td>62.04</td>
    <td>62.08</td>
  </tr>
  <tr>
    <td>MTOPDomainClassification</td>
    <td>85.54</td>
    <td>89.92</td>
    <td>86.06</td>
    <td>89.85</td>
  </tr>
  <tr>
    <td>MTOPIntentClassification</td>
    <td>64.17</td>
    <td>70.85</td>
    <td>63.03</td>
    <td>75.18</td>
  </tr>
  <tr>
    <td>MasakhaNEWSClassification</td>
    <td>77.95</td>
    <td>55.42</td>
    <td>77.77</td>
    <td>72.28</td>
  </tr>
  <tr>
    <td>MassiveIntentClassification</td>
    <td>63.48</td>
    <td>64.37</td>
    <td>61.46</td>
    <td>66.64</td>
  </tr>
  <tr>
    <td>MassiveScenarioClassification</td>
    <td>68.75</td>
    <td>69.05</td>
    <td>66.41</td>
    <td>70.38</td>
  </tr>
  <tr>
    <td>ToxicConversationsClassification</td>
    <td>59.14</td>
    <td>67.28</td>
    <td>66.90</td>
    <td>62.85</td>
  </tr>
</table>

Table 21: MTEB English classification results.

<table>
  <tr>
    <th>Dataset</th>
    <th>DAP</th>
    <th>SONAR</th>
    <th>LaBSE</th>
    <th>MEXMA</th>
  </tr>
  <tr>
    <td>Average</td>
    <td>67.46</td>
    <td>63.13</td>
    <td>68.69</td>
    <td>66.25</td>
  </tr>
  <tr>
    <td>AmazonReviewsClassification (zh)</td>
    <td>34.35</td>
    <td>31.91</td>
    <td>32.98</td>
    <td>33.40</td>
  </tr>
  <tr>
    <td>MassiveIntentClassification (zh-CN)</td>
    <td>71.99</td>
    <td>62.08</td>
    <td>63.86</td>
    <td>74.41</td>
  </tr>
  <tr>
    <td>MassiveScenarioClassification (zh-CN)</td>
    <td>65.45</td>
    <td>68.88</td>
    <td>70.85</td>
    <td>65.28</td>
  </tr>
  <tr>
    <td>JDRreview</td>
    <td>71.54</td>
    <td>69.59</td>
    <td>79.13</td>
    <td>70.73</td>
  </tr>
  <tr>
    <td>MultilingualSentiment</td>
    <td>62.03</td>
    <td>57.69</td>
    <td>65.52</td>
    <td>60.34</td>
  </tr>
  <tr>
    <td>OnlineShopping</td>
    <td>85.03</td>
    <td>75.64</td>
    <td>85.62</td>
    <td>80.09</td>
  </tr>
  <tr>
    <td>Waimai</td>
    <td>81.82</td>
    <td>76.12</td>
    <td>82.85</td>
    <td>79.52</td>
  </tr>
</table>

Table 22: MTEB Chinese classification results.

<table>
  <tr>
    <th>Dataset</th>
    <th>DAP</th>
    <th>SONAR</th>
    <th>LaBSE</th>
    <th>MEXMA</th>
  </tr>
  <tr>
    <td>Average</td>
    <td>63.76</td>
    <td>61.88</td>
    <td>62.05</td>
    <td>66.07</td>
  </tr>
  <tr>
    <td>AmazonReviewsClassification</td>
    <td>35.60</td>
    <td>34.91</td>
    <td>38.52</td>
    <td>35.62</td>
  </tr>
  <tr>
    <td>MTOPDomainClassification</td>
    <td>84.43</td>
    <td>86.19</td>
    <td>84.14</td>
    <td>86.70</td>
  </tr>
  <tr>
    <td>MTOPIntentClassification</td>
    <td>65.78</td>
    <td>66.75</td>
    <td>62.01</td>
    <td>74.12</td>
  </tr>
  <tr>
    <td>MassiveIntentClassification</td>
    <td>64.51</td>
    <td>58.55</td>
    <td>60.47</td>
    <td>65.59</td>
  </tr>
  <tr>
    <td>MassiveScenarioClassification</td>
    <td>68.50</td>
    <td>63.02</td>
    <td>65.1</td>
    <td>68.31</td>
  </tr>
</table>

Table 23: MTEB French classification results.
<table>
  <tr>
    <th>Dataset</th>
    <th>DAP</th>
    <th>SONAR</th>
    <th>LaBSE</th>
    <th>MEXMA</th>
  </tr>
  <tr>
    <td>Average</td>
    <td>52.27</td>
    <td>54.01</td>
    <td>49.53</td>
    <td>55.38</td>
  </tr>
  <tr>
    <td>DanishPoliticalCommentsClassification</td>
    <td>36.44</td>
    <td>37.59</td>
    <td>38.69</td>
    <td>38.75</td>
  </tr>
  <tr>
    <td>LccSentimentClassification</td>
    <td>58.27</td>
    <td>54.27</td>
    <td>50.07</td>
    <td>52.40</td>
  </tr>
  <tr>
    <td>MassiveIntentClassification (da)</td>
    <td>58.74</td>
    <td>62.03</td>
    <td>58.25</td>
    <td>65.75</td>
  </tr>
  <tr>
    <td>MassiveScenarioClassification (da)</td>
    <td>66.15</td>
    <td>67.76</td>
    <td>65.24</td>
    <td>69.26</td>
  </tr>
  <tr>
    <td>NordicLangClassification</td>
    <td>41.73</td>
    <td>48.40</td>
    <td>35.38</td>
    <td>50.74</td>
  </tr>
</table>

Table 24: MTEB Danish classification results.

<table>
  <tr>
    <th>Dataset</th>
    <th>DAP</th>
    <th>SONAR</th>
    <th>LaBSE</th>
    <th>MEXMA</th>
  </tr>
  <tr>
    <td>Average</td>
    <td>51.58</td>
    <td>55.59</td>
    <td>50.76</td>
    <td>58.08</td>
  </tr>
  <tr>
    <td>MassiveIntentClassification</td>
    <td>55.85</td>
    <td>59.90</td>
    <td>57.91</td>
    <td>64.48</td>
  </tr>
  <tr>
    <td>MassiveScenarioClassification</td>
    <td>62.67</td>
    <td>65.81</td>
    <td>64.29</td>
    <td>68.22</td>
  </tr>
  <tr>
    <td>NoRecClassification</td>
    <td>46.06</td>
    <td>48.25</td>
    <td>45.44</td>
    <td>48.88</td>
  </tr>
  <tr>
    <td>NordicLangClassification</td>
    <td>41.73</td>
    <td>48.40</td>
    <td>35.38</td>
    <td>50.74</td>
  </tr>
</table>

Table 25: MTEB Norwegian classification results.

<table>
  <tr>
    <th>Dataset</th>
    <th>DAP</th>
    <th>SONAR</th>
    <th>LaBSE</th>
    <th>MEXMA</th>
  </tr>
  <tr>
    <td>Average</td>
    <td>53.03</td>
    <td>55.09</td>
    <td>56.00</td>
    <td>57.09</td>
  </tr>
  <tr>
    <td>AllegroReviews</td>
    <td>31.58</td>
    <td>29.62</td>
    <td>34.89</td>
    <td>31.09</td>
  </tr>
  <tr>
    <td>MassiveIntentClassification (pl)</td>
    <td>58.53</td>
    <td>65.86</td>
    <td>59.71</td>
    <td>66.85</td>
  </tr>
  <tr>
    <td>MassiveScenarioClassification (pl)</td>
    <td>63.05</td>
    <td>69.99</td>
    <td>64.58</td>
    <td>70.20</td>
  </tr>
  <tr>
    <td>PAC</td>
    <td>67.97</td>
    <td>73.87</td>
    <td>68.11</td>
    <td>73.31</td>
  </tr>
  <tr>
    <td>PolEmo2.0-IN</td>
    <td>61.75</td>
    <td>52.80</td>
    <td>64.00</td>
    <td>59.10</td>
  </tr>
  <tr>
    <td>PolEmo2.0-OUT</td>
    <td>35.32</td>
    <td>38.40</td>
    <td>44.72</td>
    <td>42.00</td>
  </tr>
</table>

Table 26: MTEB Polish classification results.

<table>
  <tr>
    <th>Dataset</th>
    <th>DAP</th>
    <th>SONAR</th>
    <th>LaBSE</th>
    <th>MEXMA</th>
  </tr>
  <tr>
    <td>Average</td>
    <td>63.87</td>
    <td>70.73</td>
    <td>69.75</td>
    <td>74.39</td>
  </tr>
  <tr>
    <td>PawsX</td>
    <td>55.30</td>
    <td>75.05</td>
    <td>54.07</td>
    <td>73.18</td>
  </tr>
  <tr>
    <td>SprintDuplicateQuestions</td>
    <td>72.47</td>
    <td>77.08</td>
    <td>89.26</td>
    <td>86.89</td>
  </tr>
  <tr>
    <td>XNLI</td>
    <td>63.83</td>
    <td>60.06</td>
    <td>65.92</td>
    <td>63.10</td>
  </tr>
</table>

Table 27: MTEB English pair classification results.

<table>
  <tr>
    <th>Dataset</th>
    <th>DAP</th>
    <th>SONAR</th>
    <th>LaBSE</th>
    <th>MEXMA</th>
  </tr>
  <tr>
    <td>Average</td>
    <td>73.03</td>
    <td>77.57</td>
    <td>73.70</td>
    <td>78.13</td>
  </tr>
  <tr>
    <td>PawsX (fr)</td>
    <td>55.57</td>
    <td>71.36</td>
    <td>54.63</td>
    <td>71.07</td>
  </tr>
  <tr>
    <td>Opusparcus (fr)</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>100.00</td>
  </tr>
  <tr>
    <td>XNLI</td>
    <td>63.52</td>
    <td>61.34</td>
    <td>66.48</td>
    <td>63.32</td>
  </tr>
</table>

Table 28: MTEB French pair classification results.

<table>
  <tr>
    <th>Dataset</th>
    <th>DAP</th>
    <th>SONAR</th>
    <th>LaBSE</th>
    <th>MEXMA</th>
  </tr>
  <tr>
    <td>Average</td>
    <td>61.12</td>
    <td>60.80</td>
    <td>61.95</td>
    <td>62.12</td>
  </tr>
  <tr>
    <td>PawsX(zh)</td>
    <td>56.20</td>
    <td>65.35</td>
    <td>54.26</td>
    <td>63.68</td>
  </tr>
  <tr>
    <td>Cmnli</td>
    <td>69.29</td>
    <td>61.86</td>
    <td>72.67</td>
    <td>67.45</td>
  </tr>
  <tr>
    <td>Ocnli</td>
    <td>57.86</td>
    <td>55.18</td>
    <td>58.91</td>
    <td>55.23</td>
  </tr>
</table>

Table 29: MTEB Chinese pair classification results.
<table>
  <tr>
    <th>Dataset</th>
    <th>DAP</th>
    <th>SONAR</th>
    <th>LaBSE</th>
    <th>MEXMA</th>
  </tr>
  <tr>
    <td>Average</td>
    <td>67.45</td>
    <td>67.24</td>
    <td>70.93</td>
    <td>70.62</td>
  </tr>
  <tr>
    <td>BIOSSES</td>
    <td>70.51</td>
    <td>79.11</td>
    <td>78.70</td>
    <td>75.97</td>
  </tr>
  <tr>
    <td>SICK-R</td>
    <td>69.18</td>
    <td>62.94</td>
    <td>69.99</td>
    <td>66.00</td>
  </tr>
  <tr>
    <td>STS12</td>
    <td>64.69</td>
    <td>65.46</td>
    <td>65.08</td>
    <td>67.32</td>
  </tr>
  <tr>
    <td>STS13</td>
    <td>63.50</td>
    <td>62.79</td>
    <td>67.98</td>
    <td>67.05</td>
  </tr>
  <tr>
    <td>STS14</td>
    <td>61.49</td>
    <td>57.54</td>
    <td>64.03</td>
    <td>62.73</td>
  </tr>
  <tr>
    <td>STS15</td>
    <td>75.38</td>
    <td>74.25</td>
    <td>76.59</td>
    <td>75.72</td>
  </tr>
  <tr>
    <td>STS16</td>
    <td>68.00</td>
    <td>75.73</td>
    <td>72.98</td>
    <td>76.93</td>
  </tr>
  <tr>
    <td>STS17 (en-en)</td>
    <td>77.03</td>
    <td>79.94</td>
    <td>79.45</td>
    <td>80.97</td>
  </tr>
  <tr>
    <td>STS22 (en)</td>
    <td>53.38</td>
    <td>47.12</td>
    <td>60.97</td>
    <td>57.11</td>
  </tr>
  <tr>
    <td>STSBenchmark</td>
    <td>69.39</td>
    <td>67.39</td>
    <td>72.25</td>
    <td>73.53</td>
  </tr>
  <tr>
    <td>STSBenchmarkMultilingualSTS (en)</td>
    <td>69.39</td>
    <td>67.39</td>
    <td>72.25</td>
    <td>73.53</td>
  </tr>
</table>

Table 30: MTEB English STS results.

<table>
  <tr>
    <th>Dataset</th>
    <th>DAP</th>
    <th>SONAR</th>
    <th>LaBSE</th>
    <th>MEXMA</th>
  </tr>
  <tr>
    <td>Average</td>
    <td>45.31</td>
    <td>42.15</td>
    <td>47.50</td>
    <td>51.56</td>
  </tr>
  <tr>
    <td>ATEC</td>
    <td>28.01</td>
    <td>26.18</td>
    <td>26.61</td>
    <td>29.68</td>
  </tr>
  <tr>
    <td>BQ</td>
    <td>40.01</td>
    <td>37.66</td>
    <td>42.60</td>
    <td>44.37</td>
  </tr>
  <tr>
    <td>LCQMC</td>
    <td>54.97</td>
    <td>50.11</td>
    <td>52.19</td>
    <td>61.34</td>
  </tr>
  <tr>
    <td>PAWSX</td>
    <td>12.99</td>
    <td>32.75</td>
    <td>10.23</td>
    <td>27.77</td>
  </tr>
  <tr>
    <td>STS22(zh)</td>
    <td>52.05</td>
    <td>52.82</td>
    <td>63.02</td>
    <td>63.49</td>
  </tr>
  <tr>
    <td>STSB</td>
    <td>63.67</td>
    <td>50.18</td>
    <td>68.38</td>
    <td>65.75</td>
  </tr>
  <tr>
    <td>STSBenchmarkMultilingualSTS (zh)</td>
    <td>65.46</td>
    <td>45.33</td>
    <td>69.50</td>
    <td>68.55</td>
  </tr>
</table>

Table 31: MTEB Chinese STS results.

<table>
  <tr>
    <th>Dataset</th>
    <th>DAP</th>
    <th>SONAR</th>
    <th>LaBSE</th>
    <th>MEXMA</th>
  </tr>
  <tr>
    <td>Average</td>
    <td>67.74</td>
    <td>65.60</td>
    <td>74.33</td>
    <td>70.10</td>
  </tr>
  <tr>
    <td>SICKFr</td>
    <td>66.84</td>
    <td>66.1</td>
    <td>69.94</td>
    <td>65.94</td>
  </tr>
  <tr>
    <td>STS22 (fr)</td>
    <td>64.44</td>
    <td>61.72</td>
    <td>77.95</td>
    <td>72.19</td>
  </tr>
  <tr>
    <td>STSBenchmarkMultilingualSTS (fr)</td>
    <td>71.92</td>
    <td>68.99</td>
    <td>75.1</td>
    <td>72.17</td>
  </tr>
</table>

Table 32: MTEB French STS results.

<table>
  <tr>
    <th>Dataset</th>
    <th>DAP</th>
    <th>SONAR</th>
    <th>LaBSE</th>
    <th>MEXMA</th>
  </tr>
  <tr>
    <td>Average</td>
    <td>57.06</td>
    <td>57.17</td>
    <td>65.82</td>
    <td>63.67</td>
  </tr>
  <tr>
    <td>CDSC-R</td>
    <td>74.12</td>
    <td>85.77</td>
    <td>85.53</td>
    <td>85.95</td>
  </tr>
  <tr>
    <td>SICK-R-PL</td>
    <td>60.63</td>
    <td>62.98</td>
    <td>65.90</td>
    <td>64.31</td>
  </tr>
  <tr>
    <td>STS22 (pl)</td>
    <td>28.16</td>
    <td>25.31</td>
    <td>39.28</td>
    <td>32.51</td>
  </tr>
  <tr>
    <td>STSBenchmarkMultilingualSTS (pl)</td>
    <td>65.31</td>
    <td>54.62</td>
    <td>72.58</td>
    <td>71.93</td>
  </tr>
</table>

Table 33: MTEB Polish STS results.
F  Token level analysis

In this section, we illustrate the behaviour of each model by visualizing the closest tokens in the space. We observe that MEXMA matches tokens in translations but also different contexts when tokens are used with the same meaning. This is further broken down in Table 7, which distinguishes between two types of matches MEXMA does: (1) "same language" matches, where the model identifies the same token used in a different context (monolingual), and (2) "other" matches, where it recognizes translated tokens in a sentence in another language that is not a translation (multilingual). We observe that SONAR primarily matches tokens across translations, but does not tend to match the same token when it appears in different sentences within the same language. Examples of MEXMA and SONAR comparisons of matching the same token in other sentences is in Figure 5, and both models matching translations in Figure 6. In both figures, we show the three closest tokens to the selected token, denoted as query on the green box, with the blue text. The closest tokens are in the purple boxes with the pink text. Additionally, we show examples of how LaBSE and MEXMA without direct token-level gradients (no-tok MEXMA), match adjacent tokens in the same sentence regularly. These are shown for LaBSE in Figure 2, and for no-tok MEXMA in Figure 3. Lastly, we show how XLM-RoBERTa mostly matches the same tokens in other sentences in the same language, presented in Figure 4. For these last three models, we show the top-2 closest tokens, with the same color scheme as mentioned above. Each image has two examples for the given model.

<table>
  <tr>
    <th>Dataset</th>
    <th>XLM-R</th>
    <th>SONAR</th>
    <th>LaBSE</th>
    <th>MEXMA</th>
  </tr>
  <tr>
    <td>Average</td>
    <td>56.90</td>
    <td>70.66</td>
    <td>67.53</td>
    <td>73.03</td>
  </tr>
  <tr>
    <td>eng-deu</td>
    <td>61.50</td>
    <td>77.10</td>
    <td>72.90</td>
    <td>80.50</td>
  </tr>
  <tr>
    <td>eng-fra</td>
    <td>71.20</td>
    <td>89.20</td>
    <td>84.40</td>
    <td>91.50</td>
  </tr>
  <tr>
    <td>eng-ces</td>
    <td>38.00</td>
    <td>45.70</td>
    <td>45.30</td>
    <td>47.10</td>
  </tr>
</table>

Table 34: SimAlign results using different models as backbone for the token-level alignment on different language pairs.

2020; Schuster et al., 2019). In order to further validate the improved alignment of our tokens, we use XLM-RoBERTa, SONAR, LaBSE and MEXMA as the backbone for SimAlign (Jalili Sabet et al., 2020). We test the models across 3 language pairs, on the datasets reference in SimAlign, English-German (Koehn, 2005), English-French (Och and Ney, 2000) and English-Czech (Mareček, 2008). The alignment created by the models is compared a reference word alignment to compute a F1 score. The results are provided in Table 34, and it is possible to see that the alignment created by MEXMA achieves better F1 scores. All results were achieved using the itermax method, taking the word representations from the last layer of each model.

F.1 Verifying the token alignment quality through SimAlign

Several approaches have showed that aligned tokens across languages lead to better performing sentence representations (Li et al., 2023; Cao et al.,
Figure 2: Example of LaBSE’s token matching. The token in blue is the query token, the tokens in pink are the closest tokens to the query token in the space.

Figure 3: Example of MEXMA no token-level grad’s token matching. The token in blue is the query token, the tokens in pink are the closest tokens to the query token in the space.

Figure 4: Example of XLM-RoBERTa token matching. The token in blue is the query token, the tokens in pink are the closest tokens to the query token in the space.
He built a WiFi door bell, he said.

The find also grants insight into the evolution of feathers in birds.

Token: bell

WiFi ile çalışan bir kapı zili yaptığıını söyledi.

WiFi ഉപയോഗിച്ച് WiFi കുടീരത്തിന്റെ വേദിയിൽ ഒരു പട്ടികയുണ്ടായി.

Previously, Ring’s CEO, Jamie Siminoff, remarked the company started when his doorbell wasn’t audible from his shop in his garage.

WiFi ഉപയോഗിച്ച് നഗരത്തിലെ ഒരു പട്ടികയായി.

Dheweke mbangun bel lawang WiFi, jarene.

জিনি জানা যে তিনি একটি ওয়াই-ফাই ডোর বেল তৈরি করেছিলেন।

Token: birds

But there are a lot of things about birds that still look like a dinosaur.

The area is also home to an extremely wide variety of animal and bird species.

The researchers suggested that, even though this is the tail of a young dinosaur, the sample shows adult plumage and not a chick’s down.

The number of people present was so large that it was not possible for everybody to gain access to the funeral in St. Peter’s Square.

Token: young

The feathers’ structure suggests that they were not used in flight but rather for temperature regulation or display. The researchers suggested that, even though this is the tail of a young dinosaur, the sample shows adult plumage and not a chick’s down.

Els investigadors suggerien que, tot i que es tracta de la cua d’un dinosaure jove, la mostra revela un plomatge adult, i no pas plomissol.

Ceretătorii au sugerat că, deși aceasta este coada unui dinozaur tânăr, esantionul arată un penaj adult și nu puiul unui pui.

Os pesquisadores sugeriram que, apesar do rabo ser de um dinossauro jovem, a amostra revela uma plumagem adulta, não uma penugem.

Token: funeral

The number of people present was so large that it was not possible for everybody to gain access to the funeral in St. Peter’s Square.

Fue tanta la cantidad de gente que se concentró, que no todos pudieron acceder al funeral en la Plaza de San Pedro.

O número de pessoas presentes era tão grande que não foi possível que todos tivessem acesso ao funeral na Praça de São Pedro.

Over four million people went to Rome to attend the funeral.

O número de pessoas presentes era tão grande que não foi possível que todos tivessem acesso ao funeral na Praça de São Pedro.

Hi havia tanta gent present que no tots van aconseguir accedir al funeral a la Plaça de Sant Pere.

Foi tal o número de pessoas que acudiu que non todo o mundo pudo acceder ao funeral na praza de San Pedro.

Figure 5: Comparison of SONAR and MEXMA token matching. MEXMA displays the ability to match a token in another sentence in the same language. SONAR matches a translated token. The token in blue is the query token, the tokens in pink are the closest tokens to the query token in the space. MEXMA is on the left, SONAR on the right.
"We now have 4-month-old mice that are non-diabetic that used to be diabetic," he added.

He did not set a figure for the cuts, saying they will be made based on China's economic output.

Siminoff said sales boosted after his 2013 appearance in a Shark Tank episode where the show panel declined funding the startup.

Liberal criticism of the reconstruction effort has focused on the awarding of reconstruction contracts to perceived Washington insiders.

Figure 6: Comparison of SONAR and MEXMA on translated tokens in translations.
G  Attention distribution over tokens

In this section, we provide some examples of MEXMA and LaBSE’s attention probabilities given by the CLS token to the word tokens. The examples are provided in Figures 7, 8, 9 and 10. Across all figures, it is possible to see that LaBSE tends to be more uniform across all tokens, while MEXMA tends to focus more attention on a smaller subset of the tokens. All examples are taken from the FLORES200 test set with the xsim++ extension, where some words in the original sentences are replaced, and the models have to be able to still match the correct translation, and not a sentence with a small change. From Figure 7 to Figure 8 "nineteen" is replaced with "twenty nine". From Figure 9 to Figure 10 the word "white" is replaced with "black".

Figure 7 shows the attention placed by MEXMA and LaBSE on the same sentence in English and Portuguese. It is possible to see that MEXMA in Portuguese places most of the attention in two tokens, "governador" and "19", where the token in "19" is very relevant here since it is the one needed to distinguish the examples in xsim++. LaBSE seems to have many tokens with a lot of attention, and does not have "19" as one of the tokens with the most attention.

In Figure 8, we have the English example with nineteen (as previously shown in Figure 7) compared to the same sentence with nineteen replaced by twenty-nine. Interestingly, LaBSE places more attention on the "#teen" token than the "nine" token, but similar attention to the "twenty", "-" and "nine" tokens. MEXMA places similar attention in all nineteen tokens, and in twenty nine it places a small amount of attention on the irrelevant "-", with a higher degree of attention in "nine" and a smaller amount of attention in "twenty". MEXMA also seems to do a good job ignoring irrelevant tokens like "of", while LaBSE places a lot of attention in it.

Figure 9 has the same sentence in English and Portuguese, where, in xsim++ the models need to be able to match the color "white" instead of other colors. It is possible to see that, for LaBSE, white is not one of the most relevant tokens in English, but for MEXMA it is, along with "television". In Portuguese the behavior is similar, the token "bran" in "esbranquiçada" has a large degree of attention from MEXMA, while for LaBSE is it not a token with a lot of attention, and "cada" which is a token that does not convey the idea of white, is the one with the most attention out of the 4 tokens of the word, for LaBSE. In Portuguese it is also noticeable that MEXMA gives a small amount of attention to most of the less relevant tokens, while LaBSE seems to have a lot more tokens with a high degree of attention.

Figure 10 shows the same English sentence as Figure 9, with the word white replaced with the word black. Interestingly, MEXMA’s attention remains the same with black and white, while for LaBSE the token "black" seems to get less attention than the token "white". The remaining tokens get similar attention in both models.

Additionally, Figure 11, provides a comparison for MEXMA and LaBSE with the probabilities of all heads, and all tokens, using BertViz (Vig, 2019). It is possible to see that MEXMA places a lot of attention on the EOS token, 
, which is used as an attention dump, i.e. an irrelevant token that receives a very large attention probability, a common phenomena in transformers, as explored in Xiao et al. (2024); Darcel et al. (2024); Sun et al. (2024). This happens frequently with MEXMA. It is, again, possible to see the difference in uniformity for MEXMA and LaBSE, with LaBSE having a more uniform attention in the figure. If we remove the BOS and EOS tokens from the entropy computation, we now get an entropy of \( \approx 3.5 \) and \( \approx 3 \) for LaBSE and MEXMA, respec-
tively. MEXMA’s entropy increases, while LaBSE stays mostly similar, which shows that MEXMA indeed frequently uses the EOS token as a dump. However, MEXMA still has a lower entropy and a more skewed distribution over its word tokens, with or without BOS and EOS, as shown by the lower entropy and the Figures 7-10.
L: The governor 's office said nine ##teen of the injured were police officers .
M: _The _governo r ' s _office _said _ni nete en _of _the _in ju red _were _police _officer s .
L: O gabinete do governador afirmou que 19 dos feridos eram agentes policiais .
M: _O _gabinet e _do _governador _afirmou _que _19 _dos _fer idos _eram _agentes _policiais

Figure 7: Comparison of LaBSE and MEXMA’s probabilities distribution over the tokens. In this example, the models had to match the sentence with "19" in Portuguese and English. LaBSE’s entries are preceeded with "L:", and MEXMA’s with "M:".

L: The governor 's office said nine ##teen of the injured were police officers .
M: _The _governo r ' s _office _said _ni nete en _of _the _in ju red _were _police _officer s .
L: The governor 's office said twenty - nine of the injured were police officers .
M: _The _governo r ' s _office _said _twenty - nine _of _the _in ju red _were _police _officer s .

Figure 8: Comparison of LaBSE and MEXMA’s probabilities distribution over the tokens. In this example, the models had to distinguish the sentence with "19" and "29" in Portuguese and English. LaBSE’s entries are preceeded with "L:", and MEXMA’s with "M:".

L: Television reports show white smoke coming from the plant .
M: _Television _reports _show _white _smo ke _coming _from _the _plant .
L: Reportage ##ns televisiva ##s divulga ##m a fum ##aça es ##bran ##qui ##çada saindo da planta .
M: _Report agens _televisi vas _divulga m _a _fum a ca _es bran qui ça da _sa indo _da _planta

Figure 9: Comparison of LaBSE and MEXMA’s probabilities distribution over the tokens. In this example, the models had to match the sentence with "white" in Portuguese and English. LaBSE’s entries are preceeded with "L:", and MEXMA’s with "M:".

L: Television reports show white smoke coming from the plant .
M: _Television _reports _show _white _smo ke _coming _from _the _plant .
L: Television reports show black smoke coming from the plant .
M: _Television _reports _show _black _smo ke _coming _from _the _plant .

Figure 10: Comparison of LaBSE and MEXMA’s probabilities distribution over the tokens. In this example, the models had to distinguish the sentence with "white" and "black" in Portuguese and English. LaBSE’s entries are preceeded with "L:", and MEXMA’s with "M:"
Figure 11: Attention distribution of MEXMA and LaBSE across all heads, and all tokens. On the left is LaBSE, on the right is MEXMA. MEXMA uses the EOS token as an attention dump, and has a more skewed distribution, while LaBSE has a more uniform distribution.
H  Baseline architectures

We report SONAR, LaBSE’s, DAP’s and RetroMAE’s architectures in Figures 12b, 12a, 12c and 12d, respectively for easier comparison. LaBSE employs a slightly modified contrastive loss, to increase separation, and SONAR is based on translation. DAP uses token-level objectives, but it does not leverage them to update the sentence representation. RetroMAE uses the sentence in the heavy unmasking, but that unmasking does not update the tokens outputted by the encoder, it is monolingual, and the sentence representation does not come from an unmasked input. MEXMA is based on cross unmasking and has direct token level gradients updating its internal representations.
(a) LaBSE’s architecture.

The car is red.    El coche es rojo.

(b) SONAR’s architecture.

El coche es rojo.    [BOS] The car is red.

(c) DAP’s architecture.

The car is red.    El coche es rojo.

(d) RetroMAE’s architecture.

The car is red.    The car is red.

Figure 12: Architecture of the baselines.

Reviewer paper summaries: The authors present a novel approach for extracting cross-lingual sentence level embeddings that utilizes a combination of sentence and token-level objectives. They conducted extensive ablations in different areas: training data, model size, combination of training objectives, etc. They additionally compare and achieve SOTA performance against existing current SOTA CLSE extraction methods.
Summary of strengths: 1. Improved performance: The authors have combined existing methodologies (with extensive ablation and justification) to develop a novel SOTA approach for CLSE with proper references to the original work.

2. Thorough Ablation: The authors reported thorough ablations and hypotheses for their results. 

3. Open-source: Code attachment guarantees reproducibility and broader research impact.
Summary of weaknesses: 1. Fair comparisons conducted with existing methods are not all better.

2. The tables need better formatting.
Comments, suggestions and typos: Comments: 
The paper reads and flows well without typos, includes appendix for further clarifications and most tables and figures have self-contained captions.

Suggestions:

  " The results indicate that the contrastive loss better suits the STS task, given this is the only task where DAP is able to outperform SONAR, and where LaBSE outperforms MEXMA. " --
Since the authors perform ablation with contrastive loss instead of mlm, it is curious if this modification achieves SOTA performance for STS task (Table 2 experiments) since the authors hypothesize CL is better suited for STS.

   Include in Tables 2,5,7 direction arrows to show optimal score direction.
   Use latex package booktabs for better table formatting.
   Tables in the appendix needs proper formatting and alignment for better readability. Refer to https://people.inf.ethz.ch/markusp/teaching/guides/guide-tables.pdf
Overall assessment scores: 4