Arctic-Embed 2.0: Multilingual Retrieval Without Compromise
Authors: Puxuan Yu
Luke Merrick
Gaurav Nuti
Daniel F Campos
Venue: COLM 2025 (2025)
Keywords: Multilingual Retrieval
Dense Retrieval
Cross-lingual Transfer
TL;DR: A multilingual embedding model that also performs well on English retrieval — with probing experiments to understand why.

Abstract: This paper presents the training methodology of Snowflake Arctic-Embed 2.0, a set of open-source text embedding models built for effective and efficient multilingual retrieval. While prior works have suffered from degraded English retrieval quality, Arctic-Embed 2.0 delivers competitive retrieval quality on multilingual and English-only benchmarks, and supports Matryoshka Representation Learning (MRL) for efficient embedding storage with significantly lower compressed quality degradation compared to alternatives. Beyond describing the design and implementation details, we highlight critical research questions encountered during development, including the mechanisms of cross-lingual transfer in retrieval pre-training and what we term the "English performance gap" - the systematic quality difference between specialized English-only models and multilingual alternatives. Through targeted experiments addressing these questions, we derive insights from both positive and negative results, contributing to a broader understanding of multilingual embedding models and aiming to stimulate further research on improving cross-lingual representation quality while maintaining strong monolingual performance.

Published as a conference paper at COLM 2025
Arctic-Embed 2.0: Multilingual Retrieval Without Compro- mise
Puxuan Yu Snowflake Inc. puxuan. yueeionnu@usoffllakcc com
Luke Merrick"
Gaurav Nuti
DatologyAI
Snowflake Inc.
contact@lukemerridde com
gaurav. www.intowwwwllae.ccoo
Daniel Campos
Snowflake Inc.
daniel. campos@snow
Abstract
This paper presents the training methodology of Snowflake Aretic-Embed 2.0, a set of open-source text embedding models built for effective and efficient multilingual retrieval. While prior works have suffered from de- graded English retrieval quality, Arctic-Embed 2.0 delivers competitive retrieval quality on multilingual and English-onty benchmarks, and sup- ports Matryoshka Representation Learning (MRL) for efficient embedding storage with significantly lower compressed quality degradation compared alternatives. Beyond describing the design and implementation details, we highlight critical research questions encountered during development, including the mechanisms of cross-lingual transfer in retrieval pre-training and what we term the "English performance gap" - the systematic qual- ity difference between specialized English-only models and multilingual alternatives. Through targeted experiments addressing these questions, we derive insights from both positive and negative results, contributing to a broader understanding of multilingual embedding models and aiming to stimulate further research on improving cross-lingual representation quality while maintaining strong monolingual performance.
1 Introduction
Transformer-based embedding models have become a cornerstone of various information retrieval (IR) applications (e.g,, search engines and retrieval-augmeeted generation systems). Although many efforts have focused on English-only retrieval (Merrick et al., 2024; Wang et al., 2024a; Günther et al., 2024; Nussbaum et al., 2024), considerable efforts have also been directed toward developing multilingual embedding models (Wang al., 2024b; Zhang et al., 2024; Chen et al., 2024; Sturua et al., 2024). By learning to map queries and documents from multiple languages into a shared representation space, these multilingual text embedding models enable non-English monolingual retrieval as well as cross-lingual retrieval.
We develop Arctic-Embed 2.0 to deliver frontier retrieval quality while addressing two key limitations observed in current multilingual embedding models:
Many ttateoff-the-rtt models that deliver high retrieval effectiveness require a large number of parameters and generate large embedding vectors (Wang et al., 2024a; Sturua et al., 2024). These increase both computational and economic costs of dense retrieval, presenting challenges when dealing with large corpora.
Efficiency Losses:
* Work done while Snowilake.
'hee open-source model weights available under the Apache 2.0 License: snowflake-arctic- embed-m-v22 and snowflake-arttieemmee----22

---

Published as a conference paper at COLM 2025
Snowflake Arctic-Embed-22.0
0.54
Snowflake Arctic-Embed-2
avg. nDCG@10 on CLEF (Multilingua)
0.52
MGTE
0.46
0.44
mE5-Base
mES-Large
BGE-M3
avg. nDCG@10 MTEB-R (English)
Figure 1: Single-vector dense retrieval performance of open-source multilingual embed- ding models with fewer than 1B parameters. Scores are average nDCG@10 on MTEB Retrieval (Muennighoff et al., 2023) and the subset of CLEF (ELRA, 2006) covering English, French, Spanish, Italian, and German.
Compromised English Retrieval Quality: It is common for multilingual models to un- derperform their English-only counterparts on English retrieval evaluations (e.g., MTEB Retrieval (Muennighoff et al., 2023)). This trade-off has been a significant pain point in deploying multilingual systems.
Arctic-Embert 2.0 outperforms leading open-source alternatives, making it highly versatile solution for both English and non-English contexts. Additionally, our two-stage approach to integrating Matryoshka Representation Learning (MRL) (Kusupati et al., 2022) drastically mitigates quality degradation during compression compared other models supporting dimensionality reduction. Our contributions are as follows:
We introduce Arctic-Embed 2.0, multilingual embedding models that achieve competitive retrieval quality on both English and multilingual benchmarks and support size-efficient embeddings via MRL.
We investigate potential causes of reduced English retrieval quality in prior multilingual models. We empirically refute the hypothesis that pretraining on distant languages harms English performance, and we propose alternative explanations for future investigation.
We reveal that pretrained checkpoint evaluations often fail to predict fine-tuned perfor- mance, highlighting the need for improved pretraining evaluation methods.
We show that while fine-tuning generally enhances cross-lingual transfer, contrastive pretraining in multilingual settings can lead to negative tross-lingual transfer.
2 Methodology
We follow three-stage training framework inspired by prior works (Merrick et al., 2024; Wang et al., 2024b; Nussbaum et al., 2024; Chen et al., 2024; Zhang et al., 2024): pretraining via masked language modeling, contrastive pretraining, and contrastive finetuning.
2

---

Published as a conference paper at COLM 2025
2.1 Masked Language Modeling
We use two open-source pretrained encoder models: et al., 2024) for medium size and bge-m3-retromat (Chen et al., 2024) for large size. Both models employ the XLM-R tokenizer (Conneau et al., 2020). Further details the selection of these base models can be found in Appendix A.
(Zhang
2.2 Contrastive Training Data
Guided by end-user applications, we focused on European languages: English, French, Spanish, German, Italian, Portuguese, and Polish.
Contrastive pretraining data consists of loosely related pairs that are not human-labeled, such title-webpage pairs. For English data, we followed Arctic-Embed (Merrick et al., 2024). For multilingual data, we used mC4 (Habernal et al., 2016), CC News (treating page titles queries and bodies documents), and multilingual Wikipedia (titles and section headings concatenated as queries, section texts as documents) following Wang et al. (2024b). NLLB (Team et al., 2022) was excluded due to its limited resemblance to query-document tasks and negligible empirical benefit in early tests. For mC4, CC News, and mWiki, we subsetted to our target languages, including English.
CC News
pt
Arctic Embed
18.7%
6.9%
10.3%
5.7%
21.0%
75.8%
de
fr
mC4
Figure 2: Breakdown of 1.41B contrastive pretraining samples by data source and language.
For finetuning data, we follow the data mix of Merrick et al. (2024) for English, adding MIRACL (Zhang et al., 2023b) training set for high-quality multilingual training samples. We exclude Mr. Tydi (Zhang et al., 2021) due to overlap with MIRACL but use all MIRACL languages (not just target ones), as we observe no negative transfer to retrieval target languages.
2.3 Methods of Data Filtering and Training
We apply heuristic and consistency quality filters to English-only pretraining data. For multilingual pretraining data, we adopt retrieval-based consistency filtering as in several prior works (Nussbaum et al., 2024; Günther et al., 2023; Wang et al., 2024a; Dai et al., 2022), using the small multilingual E5 model (Wang et al., 2024b) to embed queries and documents. Each dataset is partitioned into even shards of approximately 3 million query- document pairs, and pairs are filtered out the pair's ' document ranks below rank 20 by vector similarity within all documents in its shard. This results in approximately 1.41 billion unsupervised query-document pairs. A detailed breakdown this combined dataset by source and language is shown in Figure 2.
3

---

Published as a conference paper at COLM 2025
MIEE Retrivval hDC0010
0.554
0.550
0.548
GTE Large
Stella 1.50
95%
96%
97%
98%
99%
Random
Aug Min Neg
Model Used Ta Mine Negatives
False Positive Threshold
Margin
Score
Score
Curriculum Approach
Figure 3: Hard-negative mining ablation studies. A stronger teacher embedding model and well-tuned false-positive cutoff led to improved downstream performance, while random order of examples performed just well as various approaches to creating easy-to-hard curricula.
For pretraining, we use the standard InfoNCE contrastive loss (van den Oord et al., 2018) with a temperature 7 0.02 as our contrastive pretraining objective. We use random in- batch negatives and follow the approach of Nussbaum et al. (2024) and Merrick et al. (2024), sampling all training mini-batches from single data source at a time. For multilingual sources, different--anguage subsets are treated as distinct sources during batch construction. We use a batch size of 32,768, a maximum query length of 32, and maximum document length of 256. We use peak learning rates of 3e-5 and le-4 for the large and medium models, respectively, following linear warmup-stable-decyy (WSD) schedule (Hu et al., 2024) for 3 epochs. To accommodate the large batch size and dataset scale, we employ activation checkpointing and use 32 H100 GPUs in a distributed data-parallel training setup.
In the finetuning stage, we train using the same InfoNCE loss function but rely on smaller, high-quality datasets and carefully curated negatives instead of random in-batch negatives. We also extend the maximum sequence length for queries and documents to 512 tokens, adjusting the batch size to 256 sets of 1 query, positive doc, and 10 negative docs, changing the learning rate to 1e-5 and 5e-6 for medium and large models, respectively, and adjusting our WSD learning rate schedule to have no warmup and perform linear decay for 6,000 out of a total of 9,342 steps.
2.4 Hard Negative Mining for Finetuning
To select the most effective hard negatives, we adopt the strategy from NV Retriever (Mor- eira et al., 2024): documents scored as most relevant by a teacher embedding model are used as negatives, but any negative with a relevance score exceeding a specified percent- age of the known-positive's score is discarded stella-en-1. 1 lingual data. Using ste-large-en-te finding that stronger teacher models yield higher-quality fine-tuning datasets (Figure left). Rather than adhering to the 95% dise--ositive filtering threshold suggested in prior work, however, we experimented with varying thresholds and observed improvement at higher thresholds (Figure 3, middle).
potential false negative. We used the English teacher model, and mul tiitinngulll---lrgge for multi-
for comparison, we confirm Moreira et al. (2024)'s
We also explored curriculum learning for hard negative mining, inspired by Merrick al. (2024). Specifically, we ordered data by increasing negative hardness during training, using metrics like the average margin between relevance scores of negatives and known-positives, average negative relevance score, and minimum negative relevance score. However, as shown in Figure (right), random data ordering produced comparable or better results than any curriculum-basee approach.
2hotps: hatpan::/hhgggnnnaaee htt.annnggngcc/euuuuaannsssslll

---

Published as a conference paper at COLM 2025
Model Name
Multilingual?
#Params
Emb. Dim.
MTEB-R
CLEF
E5 Base v2
86M
768
0.502
ME5 Base
yes
86M
768
0,489
0.432
GTE Base En v1.5
no
113M
768
0,540
GTE Multilingual Base
yes
113M
768
0,511
0.479
Aretic-Embed 1.0-M
86M
768
0.549
Aretic-Embed 2.0-M
yes
113M
768
0,554
0.534
truncation
yes
113M
256
0.549
0.522
E5 Large v2
303M
1024
0.506
ME5 Large
yes
303M
1024
0.514
0.431
BGE Large En
no
303M
1024
0.521
BGE M3
yes
303M
1024
0,488
0.410
Aretic-Embed 1.0-L
303M
1024
0,560
Aretic-Embed 2.0-L
yes
303M
1024
0.556
0.541
truncation
yes
303M
256
0.547
0.530
OpenAl Text Emb. 3 Large truncation
yes
unknown
3072
0.554
0.565
yes
unknown
256
0.517
0.510
Google Text Emb. 4
no
1.2B
768
0,557*
truncation
1.2B
256
0.524*
Voyage Multilingual1 2
yes
unknown
1024
0.569
Table 1: nDCG@10 performance of models in the evaluations, grouped by size. The best- performing model in each group highlighted in bold, while the second-best underlined. #Params: the count of non-embedding parameters, except for Google's model, which only reports total parameters. Asterisks denote results from Lee et al. (2024).
2.5 Matryoshka Representation Learning
While the modest model sizes of our models enable inference with low latency and high throughput on modern GPU hardware, scalability downstream retrieval systems often depends optimizing the memory footprint of embedding vectors, since retrieval costs typically scale with the total memory consumed by embeddings (Aguerrebere al., 2023). Merrick (2024) showed that combining MRL (Kusupati et al., 2022) with scalar quantization an effective method for compressing embeddings with minimal retrieval degradation. We emulate this approach, applying MRL loss during both pretraining and finetuning stages at a single truncated dimensionality of 256. This enables the medium and large models to achieve 3x and 4x compression, respectively, while ensuring a homogeneous distribution of components that facilitates aggressive quantization for further compression.
3 Benchmarking
3.1 Evaluation Data Sets
We evaluate English-only and multilingual retrieval using the widely adopted MTEB Re- trieval benchmark (Muennighoff et al., 2023), and several languages from the CLEF 2000- 2003 test suite (ELRA, 2006). Details of CLEF can be found in Appendix B. We choose CLEF over MIRACL (Zhang et al., 2023b) because MIRACL relies exclusively on multilingual Wikipedia, and its training dataset is widely used for both language model (LM) pretraining and embedding model training (including ours). Consequently, MIRACL does not effec- tively assess model's ability to generalize beyond the Wikipedia domain. In contrast, the CLEF dataset, lacking a training set and not being freely accessible online, is derived from the news domain, making it crucial tool for evaluating out-of-domain multilingual retrieval. We still present and discuss results on MIRACL in Appendix C. For per-task performance on MTEB Retrieval, see Appendix D.

---

Published as a conference paper at COLM 2025
3.2 Results
The evaluation results, measured in nDCG@10, for MTEB-R and CLEF are presented in Table We run all open-source models ourselves while including some results pub- lished by Lee et al. (2024) for closed-source models. We additionally run OpenAl's text Bextteebbediing-3-1 for additional points of reference.
model and Voyage Al's voyage-mul tulg-miiggunnllll--- models on CLEF
Overall, our models achieve the best performance in their respective size categories on MTEB-R and CLEF, consistently outperforming same-sized competitors and delivering results comparable flagship closed-soure offerings while being at least a magnitude smaller.
Among models trained with MRL, Aretic-Embed 2.0 pulls ahead as the clear leader when em- beddings are truncated to 256 dimensions, In this setting, our medium size model outscores the best competitor (Google moxt-embedding-000 0.549 to 0.524 on MTEB-R despite having far fewer model parameters, and medium and large models retain 99% and 98% of the original MTEB-R scores, respectively - substantially better than Google text- eeweddmbedd 004 (94%) (Lee et al., 2024) and OpenAl Text Embedding 3 Large (93%) (OpenAI, 2024). We postulate that this stronger relative performance under truncation is a result of applying MRL to contrastive pretraining as well as contrastive finetuning, as Lee al. (2024) indicate that MRL was only applied in the finetuning stage for Google's text- eext-dmbedding -044 model.
4 Research Questions From The Journey
Several empirical observations arose during the development of Arctic-Embed 2.0 which lead to interesting and relevant research questions. Here we present two of these research questions which we explored through additional experimentation. Though our experiments shed some light on the situation, both questions remain open as important lines for future study.
4.1 RQ1: Cross-lingual Transfer
How much does our large-scale contrastive pretraining benefit retrieval for languages not represented in the pretraining data? Cross-lingual transfer (CLT) is phenomenon wherein language-agnostt task knowledge is transferred from resource-rich source languages to target languages with limited full MIRACL benchmark (including on languages not covered by our contrastive pretrain- ing), we focus CLT in pretraining, though this phenomenon has also been studied in multilingual retrieval during the finetuning stage (Zhang et al., 2023a).
resources. After observing strong scores across the
4.1.1 Experiments
We evaluate checkpoints of our medium model during contrastive pretraining at 2K-step intervals up 10K steps, then 10K-step intervals thereafter, assessing their performance on MIRACL. We evaluate in two ways: (1) direct evaluation of the checkpoint, and (2) finetune the checkpoint, then evaluate it.
4.1.2 Results
The evaluation results on MIRACL are shown in Figure 4. Results with and without finetuning are represented by solid and dotted lines, respectively. From this plot, we observe the following:
Evaluation before finetuning be misleading. For instance, without finetuning, the 130K-step checkpoint appears 18.3% worse than the 8K-step checkpoint on MIRACL's English subset. Áfter finetuning, however, it performs 3.1% better.
We find little CLT in contrastive pretraining. We observe negative trends in pre- and post-finetuned evaluation scores across most language families beyond those represented

---

Published as a conference paper at COLM 2025
Fclmesarj
0200044
022
0202220
022
204
0000000
0.2
0.
10
tr Mays
Figure 4: MIRACL performance (in nDCG@10) at different points during contrastive pre- training. Languages are grouped by linguistic families provided by Zhang et al. (2023b). Dotted lines represent non non-finetuned runs, while solid lines represent finetuned runs. The relative improvement or deterioration of model performance at the end (130K steps) com- pared to the 8K-step checkpoint is reported for both non-finetuned and finetuned runs.
by the pretraining data. On evaluations with finetuning, the benefits of pretraining appear to peak within the first 10K steps, after which performance begins to decline for languages not represented by the pretraining data. We observe negative CLT effects particularly in Chinese (-13.3%), Japanese (-6.9%), Russian (-6.8)), Finnish (-6.7%), Korean (-5.9%), and, surprisingly, French (-2.9%), which makes up 13.8% of the pretraining data (Figure 2).
4.2
RQ2: English Performance Gap
Why do many multilingual embedding models perform worse on English retrieval than English-only variants? As shown in Table 1, transitioning from English-only to multilingual models results drops of 1.3, 2.9,and and 3.3 points on MTEB-R for E5 Base, GTE Base, and BGE Large, respectively, with several closed-source embedding models providers also providing paired models with similar score gaps. However, despite this precedent, we observe strong English-language retrieval quality in our models. To understand why the degradation seen in other works appears absent in results, first conduct pilot experiments (Appendix E) to confirm that we do not observe this language gap in our training.
Initial hypothesis. Unable to induce English score gap in our training regimen (see Appendix E), we look to the multilingual pretraining data used by other works to explain their English score gaps. Since our training data focus European languages and we observe negative transfer to certain non-European languages like Chinese in our RQ1 experiments, we hypothesize that certain languages may act "adversaries" to English in retrieval tasks (i.e., training on these languages strongly diminishes English-language retrieval performance and vice versa).
Experimental design. To test this hypothesis, we paired English pretraining data with data from German, Spanish (controls), or Chinese (treatment). For each language, we sampled
600 batches (19.6M examples) from their respective corpora: web crawl for English, CC News for Spanish and German, and C-MTP (Xiao et al., 2024) for Chinese. An English-only run was also trained for 16 epochs, while paired runs were trained for 8 epochs, totaling 314M samples per run. The English-only run was evaluated at the midpoint ("en") to isolate
At the time of writing, Google, Voyage AT, and Cohere offer such English-multilingual model pairs.
7

---

Published as a conference paper at COLM 2025
data addition effects, and at completion ("en+en") to examine partial data replacement. All runs were fine-tuned on the same data before evaluation.
Results and analysis. Figure 5 presents the outcomes of these experiments. Starting with the MTEB-R benchmark, it is evident that incorporating the Chinese pretraining data actually improves English retrieval performance, whether of English data. This finding directly contradicts our initial hypothesis. Additionally, we see that this Chinese pretraining data outperforms Spanish and German CC News in improving retrieval across MTEB-R, MIRACL, and, in some cases, CLEF, which notably includes evaluation datasets for German and Spanish but not Chinese. Finally, we note that these findings corroborate the trend observed in Figure 4, where repeated training on English pretraining data primarily benefits English retrieval but provides limited improvement for other languages.
addition to partial replacement
MTEB-R R
CLEF
0.455
0.5300
0.529
0.450
0.450
0.448
0.526
0.525
0.523
0.445
0.444
0.443
nPOG010
0.519
0.436
0.435
0.430
0.5125
0.510
0.425
en
en+enn
en
en+es
en+zh*
en
en+een
en+de
entes
en+
MIRACL
MIRACL ZH
0.605
0.0050
0.58
0.576
0.8025
0.601
0,600
0.6000
0.57
0.5975
0.562
0.558
0.593
0.593
0.552
0,550
5875
0.50
en
en+en
en+de
en+es
en+z2h*
en
en+en
en+de
entes
en++zh+
Figure 5: The impact of adding equal amounts of English, German, Spanish, or Chinese data to the existing English pretraining baseline ("en") on downstream retrieval performance. For Chinese data, error bars indicate the standard deviation across consistency filtering levels (top-(1, (1, 5, 10, 20, 30) out of 3M), reflecting the effect of varying data quality.
Alternative hypotheses. One hypothesis suggested by our results is that data quality plays more critical role than language itself, with lower-quality multilingual training data accounting for much of the performance gap observed in prior works. Another hypothesis relates to model capacity: the issue may not stem from specific languages but rather from the total number of languages trained simultaneously, potentially exceeding the model's capacity. This could lead a trade-off where English performance is marginalized to achieve slight improvements across many other languages.
8

---

Published as a conference paper at COLM 2025
4.3 Reflections On The Journey
To summarize some key findings from our model development process, probing experi- ments, and our takes on interesting and important future directions:
Data quality matters more than quantity. We follow the advice of Merrick et al. (2024) to emphasize data quality, deliberately rejecting lower-quality multilingual training data sources, performing retrieval-based consistency filtering, and carefully mining the best neg- ative examples possible for fine-tuning, Though we do not extensively study lower-quality data, we rule out several other possible causes of lower retrieval performance observed in other works, and so we hypothesize that is this focus on quality that explains why we do not observe degradation in English retrieval performance. In other words, the English score gap observed in other multilingual models may simply reflect the challenge of ac- quiring high-quality retrieval training data in certain non- non-English languages. Numerous empirical results from this paper lend credence to this quality-centrr view, including our finetuning ablations in Section 2.4 and the strong results across languages under reduced training budget in Appendix E.
No clear formula for successful cross-lingual transfer in multilingual retrieval models. In this work, we take a step toward improving the generalization of multilingual embedding models to unseen languages and domains, particularly by evaluating non-English retrieval beyond Wikipedia using CLEF (ELRA, 2006). While we demonstrate the potential for multilingual training to enhance existing benchmark scores (e.g., better MTEB Retrieval scores with multilingual approaches in Appendix E) and expand language support without penalty (e.g,, overall benchmark improvements from adding Chinese in Figure 5), the actual process of model development is constrained by several factors: data quality (which is challenging to quantify), the availability of out-of-domain retrieval evaluation benchmarks in more languages, and the need to avoid exceeding model capacity (a concept still not fully understood). Faced with these limitations and uncertainties, incrementally and carefully incorporating mon-English data into proven English-language training mix has turned out to be the most effective strategy available for training useful multilingual embedding models.
Model "knowledge" and task-calibration are both important yet possibly orthogonal. As evidenced by the declining un-finetuned English retrieval scores in Figure 4, entirely possible for pretraining to show the embedding model hundreds of millions more highly- filtered query-document pairs yet actually induce degradation in downstream retrieval after certain point (20K steps out of 130K). Add on the finetuning step, however, and we find the performance trend reversed, with an extended pretraining regimen responsible for driving a 3% increase to the final nDCG@10 score! This flip-flop not only highlights the importance of measuring final downstream performance, but also demonstrates the intriguing possibility of some amount of useful knowledge being imparted "silently" into the model by contrastive training (somewhal hat analogously to how non-contrastive MLM pretraining improves the language model without making it fit for retrieval out-of-the-box). In hindsight, it appears that the success of Aretic-Embed 2.0 may stem from both giving the model substantial mount of "knowledge" through large-scale MLM and contrastive pretraining steps and from carefully "recalibrating" the model with the best and most denoised positive and hard-negative examples.
Conclusion
In addition detailing the training process for Aretic-Embed 2.0, we investigate linguistic transfer in embedding models, revealing that prolonged contrastive pretraining does not always enhance cross-lingual transfer, though high-quality pretraining data from languages distant to English can surprisingly do so in some settings. We further discuss how these experiments also uncover concrete evidence of the finetuning step of training "reversing" the negative impact of prolonged pretraining downstream retrieval performance, surprising result that indicates new direction for future scientific inquiry.

---

Published as a conference paper at COLM 2025
References
Cecilia Aguerrebere, Ishwar Bhati, Mark Hildebrand, Mariano Tepper, and Ted Willke. Similarity search in the blink of an eye with compressed indices, 2023. URL https:
wwrrkiv.org/abee /bs////4. 04759.
Jianly Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. BGE M3- Embedding: Multi-Lingual,, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation, June 2024. URL http:://axxiiv..rrr wrw.vvorg//aaoe arXiv:2402.33216 [cs].
wttp://...w.g/b//////////// http:/bbs/2222....// 03216.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised Cross-lingual Representation Learning at Scale. ar 1r1::1111.02211
[csl, April 2020. URL //arp///axiv.... wrgg orgg/bs//9111..1.// 01111111111111111111111111111111
02116. 16. arXiv: 1911.02116.
Zhuyun Dai, Vincent Y. Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith B. Hall, and Ming-Wet Chang. Promptagator: Few-shot dense retrieval from examples, 2022.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. arXiv preprint ar arXiv:240..21783 2024.
ELRA. The clef test suite for the clef 2000-2003 campaigg, 2006. URL https://catalogue. elra.
Michael Günther, Louis Milliken, Jonathan Geuter, Georgios Mastrapas, Bo Wang, and Han Xiao. Jina embeddings: A novel set of high-performance sentence embedding models, 2023. URL https: / http://///axiv.o/oabbbbese/ orgg 2307. 11224.
Michael Günther, Jackmin Ong, Isabelle Mohr, Alaeddine Abdessalem, Tanguy Abel, Mo- hammad Kalim Akram, Susana Guzman, Georgios Mastrapas, Saba Sturua, Bo Wang, Maximilian Werk, Nan Wang, and Han Xiao. Jina embeddings 2: 8192-token general- purpose text embeddings for long documents, 2024.
Ivan Habernal, Omnia Zayed, and Iryna Gurevych. C4Corpus: Multilingual Web-size Corpus with Free License. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelník, Bente Maegaard, Joseph Mariani, Helene Mazo, Asun- cion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Tenth Inferna- tional Conference on Language Resources and Evaluation (LREC'16), pp. 914-922, Por- torož, Slovenia, May 2016. European Language Resources Association (ELRA). URL https://aclanthhologyy wwt.:nthhhooggo.ggl111----- org/L16-1145
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, lie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. Minicpm: Unveiling the potential of small language models with scalable training strategies, 2024. URL
https://arxiv. httg/bbs/44000 24395
Zhiqi Huang, Puxuan Yu, and James Allan. Improving Cross-lingual Information Retrieval on Low-Resource Languages via Optimal Transport Distillation. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, Pp. 1048-1056, Singapore Singapore, February 2023. ACM. ISBN 978-1-4503-9407-9, doi: 10.1145/ 3539597.3570468. URL https: //dl. acm.
wtt.:aa......o....... org//oi//0.oooooooooo /145/3539597.... 3570468.
Zhiqi Huang, Puxuan Yu, Shauli Ravfogel, and James Allan. Language Concept Erasure for Language-invariant Dense Retrieval. In Yaser Al-Onaizan, Mohit Bansal, and Yun- Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 13261-13273, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653 vv1 ht/p//112222222 htt.1///ememmmmmmamaa///// URL https:
//acl //acllanthologyorrg///// emnlp-main, 736.
10

---

Published as a conference paper at COLM 2025
Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, and Ali Farhadi. Matryoshka Representation Learning. Ad- vances in Neural Information Processing Systems, 55:30233-302444 December 2022. URL 72221994888047711177779999999999 oobrreeeeeeebttaaaa--ooeeeeeeeee htmll
https:///pooeedinnnnneeiiigge cc/paprr. http://ppppfflles/eeeeer ht1le//paper/222//haaa
Jinhyuk Lee, Zhuyun Dai, Xiaoqi Ren, Blair Chen, Daniel Cer, Jeremy R. Cole, Kai Hui, Michael Boratko, Rajvi Kapadia, Wen Ding, Yi Luan, Sai Meher Karthik Duddu, Gus- tavo Hernandez Abrego, Weiqiang Shi, Nithi Gupta, Aditya Kusupati, Prateek Jain, Siddhartha Reddy Jonnalagadda, Ming- Wei Chang, and Iftekhar Naim. Gecko: Versatile text embeddings distilled from large language models, 2024.
Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, Daniel Haziza, Luca Wehrstedt, Jeremy Reizenstein, and Grigory Sizov. xformers: A modular and hackable transformer modelling library. xformers, 2022.
com http://ccoooccccboooeeeerrre
Luke Merrick.
Snowflake arctic embed m v1.5: Hitting the roi sweet spot for enterprise retrieval, Jul 2024. URL https://www.snww/// comp/engineering-bbog
Luke Merrick, Danmei Xu, Gaurav Nuti, and Daniel Campos. Arctic-Embed: Scalable, Efficient, and Accurate Text Embedding Models, May 2024. URL http:://///ii.oorr http:/iiivorgg/s/////// 2405.05374, arXiv:2405.05374 version: 1.
Gabriel de Souza P. Moreira, Radek Osmulski, Mengyao Xu, Ronay AK, Benedikt Schifferer, and Even Oldridge. NV-Retriever: Improving text embedding models with effective hard- negative mining, July 2024. URL http: org/abs//48)) 15831. arXiv:2407. 15831.
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. Mteb: Massive text embedding benchmark, 2023.
Suraj Nair, Eugene Yang, Dawn Lawrie, James Mayfield, and Douglas W. Oard. BLADE: Combining Vocabulary Pruning and Intermediate Pretraining for Scaleable Neural CLIR. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '23, pp. 1219-1229, New York, NY, USA, July 2023. Association for Computing Machinery. ISBN 978-144503994441 9400-66.99066...0000000 doi: 10.1145/353961..555 11.91111....944444 URL https://dl. acm. org htg//oi//0n 10. 1145/353911 8. 3591644.
Zach Nussbaum, John X. Morris, Brandon Duderstadt, and Andriy Mulyar. Nomic embed: Training a reproducible long context text embedder, 2024.
OpenAL, New embedding models and api updates, Jan 2024. URL /ttpss/s://oppeaiii cowp fidex/nnew-embeddin moodeelll sodeels-and-api--e api--
Saba Sturua, Isabelle Mohr, Mohammad Kalim Akram, Michael Günther, Bo Wang, Markus Krimmel, Feng Wang, Georgios Mastrapas, Andreas Koukounas, Andreas Koukounas, Nan Wang, and Han Xiao. jina-embedddngs--3: Multilingual Embeddings With Task LoRA, September 2024. URL http:////xxxii.or///aaass
10173. arXiv:2409.10173 [cs].
NLLB Team, Marta R. Costa-jussă, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Bar- rault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. No Language Left Behind: Scaling Human-Cern Centered Machine Translation, August 2022. URL http: //txxaax...////////////ea./aiav www.:orgg/////.eee 04672. arXiv:2207.044. 04672..
11

---

Published as a conference paper at COLM 2025
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. ArXiv, abs/1807 0037488, 2018. URL https://apt httppppp.smmaaicscchl...rrr
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improving Text Embeddings with Large Language Models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meefing of the Association for Compututional Linguistics (Volume 1: Long Papers), Pp. 11897-11916, Bangkok, Thailand, August 2024a. Association for Computational Linguistics. URL https: aacaathooann........ org/2024. acl- long. 642.
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Multilingual E5 Text Embeddings: A Technical Report, February 2024b. URL http: arXiv:2402.05672 [cs].
//arxiv wttp:/o///bbb//....eeeg
Shitao Xiao, Zheng Liu, Yingxia Shao, and Zhao Cao. RetroMAE: Pre-Training Retrieval- oriented Language Models Via Masked Auto-Encoder. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 538-548, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1/22 2t2/3//11/2222.mmmmmaa...33 URL ptattts://cclaathhooii
org/ 2022. emn emw.p-main..3 35.
Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Yun Nie. C-Pack: Packed Resources For General Chinese Embeddings. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieoal, SIGIR '24 pp. 649, New York, NY, USA, July 2024. Association for Computing Machinery. ISBN 9798400704314. doi: 10.1155/666672236577888 76266626.66777777777
htLLstts::/// acm.
org/doi /10.
Puxuan Yu and James Allan. A Study of Neural Matching Models for Cross- IR. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1637-1640, Virtual Event China, July 2020. ACM. ISBN 978- 1-4503-8016-4 doi: 2..145//399221.3400133 URL https: acm. org/ 1145/ 3397271 3401322.
Puxuan Yu, Hongliang Fei, and Ping Li. Cross-lingual Language Model Pretraining for Retrieval. In Proceedings of the Web Conference 2021, WWW 211, pp. 1029-1039, New York, NY, USA, June 2021. Association for Computing Machinery. ISBN 978-1-4503-8812-7, doi: URL https: /dl. acm. org//oi//i 11
00.1145/3442381..
Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang, I'enguun Xie, Fei Huang, Meishan Zhang, Wenjie Li, and Min Zhang. mGTE: Generalized Long-C CnngCtttextt Text Representation and Reranking Models for Multilingual Text Retrieval, July 2024. URL htap:// org/ abs///bs1 19669. arXiv:2407.199 196.9999.99... [cs].
Xinyu Zhang, Xueguang Ma, Peng Shi, and Jimmy Lin. Mr. TiDi: A Multi-lingual Bench- mark for Dense Retrieval. In Duygu Ataman, Alexandra Birch, Alexis Conneau, Orhan Sebastian Ruder, and Gozde Gul Sahin (eds.), Proceedings of the 1st Workshop on Multi- lingual Representation Learning, PP. 127-137, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653. 10.18653/v1/2022 211/221.mml-11.. URL httpps:/aaclantholoogy/ wwwpooooooooog/2222/0 1..-1.
Xinyu Zhang, Kelechi Ogueji, Xueguang Ma, and Jimmy Lin. Toward Best Practices for Training Multilingual Dense Retrieval Models. ACM Trans. Inf. Syst., 42(2):39:1-39:33, September 2023a. ISSN 1046-8188. doi: 10.1145/3613447. URL httpp::/ddl acm. http:///d..ccmoor/ddooo.. org/doi/ 10. 1145/3613447
Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso- Hermelo, Xiaoguang Li, Qun Liu, Mehdi Rezagholizadeh, and Jimmy Lin. MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages. Transactions of the Association for Computational Linguistics, 11:1114-1131 September 2023b. ISSN 2307-387X. doi: 10.1162/tac 00595. URL https://doi. org/ 1162/tacl 00595.
12

---

Published as a conference paper at COLM 2025
Language
#
D
# Rels
# Rel/Q
English
246
113,005
4,769
19.4
French
185
129,689
3,022
16.3
Italian
176
144,040
2,626
14.9
German
184
153,496
3,066
16.7
Spanish
156
452,027
5,759
36.9
Table 2: Statistics of the CLEF evaluation datasets: number of queries (# Q), corpus size (#
D), number of relevance judgments (# Rels), and average annotations per query (# Rel/Q).
Model Name
#Params
Emb. Dim.
MIRACL
MIRACL-O
CLEF
ME5 Base
86M
768
0.608
0.509
0,432
GTE Multilingual Base
113M
768
0.621
0,523
0.479
Arctic-Embed 2.0-M
113M
768
0.592
0,552
0.534
+ truncation
113M
256
0.578
0.545
0.522
ME5 Large BGE M3
303M
1024
0,651
0.540
0.431
303M
1024
0,678
0.568
0,410
Arctic-Embed 2.0-I.
303M
1024
0.649
0.558
0.541
truncation
303M
256
0.638
0.547
0.530
OpenAl Text Emb. Large
unknown
3072
0,549*
0.565
Google Text Emb. 4 Multilingual Voyage Multilingual
1.2B
768
0.562*
unknown
1024
0.569
Table 3: nDCG@10 performance of models on MIRACL grouped by size. Asterisks denote results from Lee al. (2024).
A Selection of Base Models
We aim to develop two model sizes to balance efficiency and effectiveness trade-offs. Based on pilot experiments with small-scale data, we selected the M..--prerriieed m-GTE check- point (Zhang et al., 2024) to initialize our medium model and the RetroMAA-pretrriiner BGE-M3 checkpoint (Chen et al., 2024) for the large model. The m-GTE architecture lever- ages unpadding and xFormers acceleration (Lefaudeux et al., 2022) for greater efficiency, while the BGE-M3 model benefits from RetroMAE pretraining (Xiao et al., 2022), retrieval- focused objective critical to its performance.
Both checkpoints use the XLM-R tokenizer. We also tested the newer Llama3 tok- enizer (Dubey et al., 2024), designed for multilingual large language models. To accommo- date this, we randomly reinitialized model weights, including the embedding layer, and pretrained both models directly without MLM pretraining. However, the Llama3 tokenizer showed no significant effectiveness gains and added efficiency overhead.
B CLEF Dataset Details
The statistics of the CLEF datasets used to evaluate the out-of-domain generalizability of multilingual models are reported Table 2. These datasets have been widely adopted in the literature on non-English monolingual retrieval (Huang et al., 2023; 2024) and cross-lingual retrieval (Yu & Allan, 2020; Yu et al., 2021; Nair et al., 2023) as a reliable benchmark. Since CLEF includes long documents beyond 512 tokens, enable the maximum token limit for all models during evaluation on this dataset 512 tokens for E5, 8192 tokens for all other models.
13

---

Published as a conference paper at COLM 2025
C Evaluation Results on MIRACL
Built upon the multilingual Wikipedia collection, a corpus widely used for language model pretraining, the MIRACL multilingual retrieval benchmark (Zhang et al., 2023b) presents scenario susceptible to in-domain testing and even data leakage, particularly due to its substantial training set.
In Table 3, compare multilingual models' performance the full MIRACL set (18 languages), MIRACL-O (an English, French, Spanish, and German subset overlapping with languages in training data), and the average CLEF performance Our results indicate that while Arctic-Ember 2.0 lags behind the top open-source models by roughly points the general MIRACL benchmark, it maintains competitive standing on both the languages included in our training (as reflected in MIRACL-O) and domains outside our training (as reflected in CLEF). Furthermore, the "low MIRACL but high CLEF" phenomenon observed in closed-source multilingual models suggests potential overfitting issue among our open-source competitors specifically towards the MIRACL dataset, which could result in poorer generalization to unseen data.
reference point.
D Per-dataset Results on MTEB Retrieval
Dataset
AE2L
AE2M
mGTE
BGE-M3
mE5L
mE5B
mE5S
AE155
Average
0.5565
0.5538
0.5107
0.4878
0.5140
0.4887
0.4646
0.5509
ArguAna
0.5917
0.5803
0.5845
0.4411
0.5437
0.4421
0.3910
0.5953
ClimateFEVER
0.4351
0.4381
0.3514
0.2971
0.2572
0.2326
0.2170
0.3686
CQADupStack
0.4610
0.4720
0.4308
0.3995
0.3967
0.4003
0.3706
0.4499
DBPedia
0.4338
0.4388
0.3972
0.3402
0.3907
0.3700
0.3757
0.4556
FEVER
0,9192
0.9155
0.9206
0.8094
0.8280
0.7942
0.7526
0.8841
FiQA2018
0.4546
0.4394
0.4468
0.4108
0.4378
0.3328
0.3328
0.4240
HotpotQA
0.6814
0.7240
0.6304
0.6944
0.7121
0.6856
0.6508
0.7221
MSMARCO
0.4485
0.4404
0.3933
0.3831
0.4370
0.4229
0.4097
0.4203
NFCorpus
0.3533
0.3588
0.3638
0.3167
0.3406
0.3246
0.3100
0.3622
NQ
0.6365
0.6404
0.5990
0.5894
0.6302
0.5994
0.5623
0.6246
QuoraRetrieval
0.8901
0.8873
0.8802
0.8857
0.8817
0.8763
0.8694
0.8659
SCIDOCS
0,2024
0,2034
0.1821
0.1642
0.1746
0.1719
0.1346
0.2149
SciFact
0.7058
0.7180
0.7300
0.6419
0.7042
0.6950
0.6770
0.7159
Touche2020
0.2950
0.2977
0.2346
0.2213
0.2314
0.2135
0.2117
0.3143
TREC-COVID
0.8387
0.8028
0.5722
0.5568
0.7123
0.6971
0.7119
0.8462
Table 4: Retrieval performance on MTEB Retrieval task (English). AE stands for Arctic- Embed.
E Replication of "Language Gap" with Fewer Pretraining Samples
We perform comparison of several variations of our training recipe to confirm our intuition that we are not observing any significant "language gap" in our training. In a shortened training procedure with only 13K contrastive pretraining steps, we vary the following factors:
MLM base model variants: English-only (En-GTE) vs. multilingual (mGTE);
Pretraining data:
English-only (English portion of our pretraining data only) vs. multilin- gual (original data mix);
Fine-tuning data: English-only (English portion of our fine-tuning data) vs. multilingual (English fine-tuning data plus non-English MIRACL training set).
As shown by the results tabulated in Table 5, none of the multilingual treatments we attempted induce sizable decrease in MTEB Retrieval score compared to the all-English
14

---

Published as a conference paper at COLM 2025
Evaluation Sets
MLM
PT
FT
MTEB-R
CLEF
MIRACL
MIRACL-O
En
0.526
0.327
0.114
0.271
En
Mul
0.532
0.340
0.268
0.361
En
En
0.524
0.439
0.486
0.478
Mul
0.532
0.442
0.588
0.517
Mul
En
0.525
0.451
0.530
0.521
Mul
Mul
0.529
0.452
0.594
0.538
Table 5: Impact of masked language modeling (MLM) base model, pretraining (PT), and finetuning (FT) data configurations with either English-only (En) or multilingual (Mul) content, showing their impact on downstream retrieval evaluations.
baseline. In fact, we actually observe slight positive effect from including the non-English MIRACL finetuning datasets even on the English MLM base model.
15


Reviews:
[{"review_id": "UU2sbY7JYc", "reviewer": "Reviewer_wM18", "summary": "The purpose of this paper is to propose a multilingual biencode that is efficient and effective. While the paper neglects to identify it, this is a first-stage retrieval that operates as a bi-encoder. In particular a new single-vector per passage model is trained that consists of fewer than 1B parameters. They gather 1.4B pre-training samples from 4 sources - Arctic Embed, mWiki, CC News, and MC4.  Some of the query-document pairs are filtered when the documents are what are associated with the queries is ranked lower than 20. The model is trained with contrastive loss. Hard negative mining is included.Embeddings are compressed with Matryoshka Representation Learning and scalar quantization.\n\nThe approach is evaluated on MTEB-R for English retrieval performance and CLEF 2000-2003 for cross language performance. Relative to the other single vector per passage approaches, this new model is competitive with the English performance and outperforms on CLEF.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Single-vector per passage approaches have been weaker in CLIR. Determining an effective approach to train such as a model is an important contribution.\n\nThe approach used to train the model for retrieval is clearly described.", "reasons_to_reject": "The paper neglects to compare against state-of-the-art baselines for CLIR such as ColBERT with PLAID-X and Learned Sparse Representations with SPLADE. Both of these techniques are quite effective for both CLIR and monolingual retrieval and cannot be ignored. It is also surprising that NeuCLIR was not chosen as a benchmark over CLEF since it is much newer and better judged. Finally, nDCG is not the standard metric used on CLEF, which makes it challenging to compare the results reported to other prior work. MAP is the standard metric used for CLEF. It is also unclear why MSMARCO was not used as training data. Many CLIR approaches rely on mMarco and neuMARCO for such training.\n\nThe paper attempts to draw conclusions on English performance ability for cross-language by comparing scores; however, such a comparison is not particularly meaningful because even among the same collection documents different sets of queries will produce different scores.", "questions_to_authors": "Q1 What is the performance on CLEF in terms of MAP?\n\nQ2 Do the results observed on CLEF hold up on NeuCLIR?", "extra_scores": {}}, {"review_id": "CcHu3UbgBm", "reviewer": "Reviewer_HHZh", "summary": "The paper presents a well-engineered, open-source multilingual embedding model and provides valuable insights into multilingual training dynamics. The authors conduct a thorough investigation of training strategies, data filtering methods, and cross-lingual transfer effects, and also offer practical guidance for future work. While the empirical analysis is extensive, the exposition could be clearer, and the discussion of their proposed research questions would benefit from greater depth and alignment with the presented results.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The Open-Multilingual-Embed-L model achieves strong multilingual retrieval results on the CLEF benchmark while retaining competitive performance on English retrieval tasks, such as those in MTEB-R. This balance addresses the issue in multilingual embedding models, which often degrade in English-specific scenarios.\n\n2. The authors’ commitment to open-source development enhances transparency and supports reproducibility, which is critical for further research in this domain.\n\n3. The training pipeline is clearly explained, including detailed data filtering techniques, robust contrastive learning, and the integration of Matryoshka Representation Learning.", "reasons_to_reject": "1. In Table 3, the empirical analysis could be more compelling with a fuller comparison across all benchmarked models, including OpenAI’s Text Embedding 3, Google’s Text Embedding 4, and Voyage Multilingual 2 on MIRACL, MIRACL-O, and CLEF. This is particularly important given the paper’s emphasis on generalization and the noted overfitting tendencies of some models to MIRACL.\n\n2. The paper lacks a cohesive narrative that connects its methodology and main results to the research questions posed later on. Clarifying how the investigations into the \"English performance gap\" and \"cross-lingual transfer\" influence overall performance and inform methodological improvements would deepen the paper's contribution.", "questions_to_authors": "1. Adding per-task results for individual MTEB retrieval tasks in the appendix would increase transparency and enable more fine-grained analysis.\n\n2. Although not critical to the current work, it would be interesting to explore whether multilingual retrieval quality can be improved using small LLM-based retrievers (e.g., 1.5B parameter scale).", "extra_scores": {}}, {"review_id": "DugdDXm3yp", "reviewer": "Reviewer_XDLM", "summary": "The authors release Open-Multilingual-Embed (OME), two open-source dense‐retrieval encoders (113M and 303M params) trained with a three-stage recipe—MLM, large-scale contrastive pre-training, and hard-negative fine-tuning using high-quality European-language data plus retrieval-based filtering—and compressed with Matryoshka Representation Learning (MRL). On MTEB-Retrieval and CLEF, OME either matches or surpasses open baselines and remains competitive with closed-source APIs.\n\nThe paper also reports exploratory studies on cross-lingual transfer (CLT) and the English performance gap. They found that evaluation during pre-training cannot accurately indicate performance after fine-tuning. They also found that after the first ~10 k steps, CLT declines for most families not in the pre-training data. For English performance gap, the authors demonstrate that in contrast to other models, OME does not exhibit the English performance gap (performance on English dropping after training on multilingual data). They speculate it is due to higher data quality that they curated via filtering, hard negative mining, stronger teacher etc.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. OME achieves competitive accuracy without sacrificing English language performance, surpassing mE5-Base and BGE-M3 on retrieval benchmarks while avoiding the usual ≥3-pt English drop.\n\n2. The paper consists of thorough and complete ablations -- Hard-negative mining thresholds, curriculum vs. random ordering, and cross-lingual transfer curves etc. This provides guidance to those looking to build their own multilingual encoders.\n\n3. The commitment to open-source facilitates reproducibility and further research.\n\n4. Given the complexity and nuances of multi-stage pipeline involved in training an encoder, the authors did a really good job of clearly communicating complex ideas. The paper is very well-written and reads nicely, with each component well motivated and linked back to prior work.", "reasons_to_reject": "The work is primarily a tidy integration of known tricks. Every core component—MLM warm-start, InfoNCE contrastive pre-training, NV-Retriever style hard negatives, MRL compression is known techniques already in use for training encoders. The exploratory research questions (CLT and English gap) are good initial studies with speculative explanations but lack deep rigorous scientific exploration.\n\nDue to this, I believe the paper is better suited as an industry or system track paper, rather than a main conference paper", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Cwz7KYDrvI", "reviewer": "Reviewer_aVcC", "summary": "- The paper introduces Open-Multilingual-Embed models which report strong English (0.554 nDCG@10 on MTEB-R) and multilingual (0.534 nDCG@10 on CLEF) performance with a 113M parameter model\n- It also reports that these models support Matryoshka Representation Learning (MRL), which allows for up to 4x compression with small quality degradation (retains about 99% of MTEB-R scores after truncation to 256 dimensions from the original 768)\n- The models are trained on a dataset of 1.41B samples in 8 European languages (English, French, Spanish, German, Italian, Portuguese, and Polish) which the authors filter from various sources. They experiment with two models used for Hard Negative Mining and find that `gte-large-en-v1.5` works better than `stella-en-1.5B-v5` for this purpose\n- Further experiments provide additional insights, such as \n  1. refuting the view that pretraining on distant languages like Chinese harms English retrieval, since adding Chinese data actually improved MTEB-R English scores\n  2. that pretrained checkpoint evaluations are shown to poorly predict final performance, because prolonged pretraining initially degraded but ultimately improved scores after finetuning\n  3. that contrastive pretraining offers limited cross-lingual transfer to unrepresented languages, but fine-tuning significantly improves their performance after this pretraining phase\n- A big portion of the paper is devoted to the \"English performance gap\", which the authors suggest may ultimately boil down to the lack of high quality multilingual training data.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- A solid practical contribution with a lot of potential impact\n- A blueprint for training similar multilingual models\n- Interesting experiments", "reasons_to_reject": "1. The evaluation primarily relies on MTEB-R and CLEF and lacks ablation studies for various design choices such as the effect of different data filtering strategies and their parameters\n2. Cross-lingual transfer is only measured for a narrow set of target languages (often high-resource), and the model's performance on an actual low-resource, unseen, or code-mixed languages is not investigated further\n3. The experiments lack statistical significance tests, and it is hence difficult to assess to what extent can the gains be attributed to chance\n4. Some of the important claims (e.g. adding adding Chinese data improved MTEB-R English scores) are not explored further and so they may be relatively easily attributed to dataset artifacts rather than to a new phenomena\n5. The paper does not provide a rigorous error analysis or qualitative examples to demonstrate where the model fails or underperforms, which weakens the understanding of its limitations", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eSztO4pNzn", "reviewer": "Reviewer_P4sz", "summary": "This paper introduces Open-Multilingual-Embed, compact multilingual dense retrievers trained using a three-stage pipeline (MLM, massive multilingual contrastive pretraining, and high-quality fine-tuning). The models integrate Matryoshka Representation Learning (MRL) for embedding compression, allowing a 3–4× smaller footprint with minimal performance loss. They achieve state-of-the-art open-source multilingual and English retrieval performance.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Effective compression with MRL, significantly reducing embedding dimensions (768/1024→256).\n- Strong empirical results across multilingual (CLEF) and English (MTEB-R) benchmarks.\n- Training pipeline with massive multilingual data (1.41 B pairs).", "reasons_to_reject": "- Limited language diversity (pretraining covers seven European languages).\n- Limited ablations on embedding dimension choices or stage-specific MRL impact.\n- High computational requirements (32 × H100 GPUs) limit reproducibility.", "questions_to_authors": "- How does your method generalize to (non-European) non-latin or low-resource languages?\n- Have you tested compression beyond 256 dimensions? Is 256 dims optimal?\n- Do you plan to release code/scripts for reproducibility?", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents Open Multilingual Embed (OME), a compact multilingual bi-encoder for first-stage retrieval with under 1B parameters. Trained on 1.4B filtered query-document pairs using a three-stage pipeline and compressed with Matryoshka Representation Learning, OME achieves strong performance on both English (MTEB-R) and cross-lingual (CLEF) benchmarks. It outperforms comparable open models and avoids the typical English performance drop in multilingual training. The paper also provides insights into training strategies and cross-lingual transfer, though the exposition could be clearer and more tightly connected to its research questions.

The paper received positive feedback from the reviewers, who highlighted that the Open Multilingual Embed L model demonstrates strong performance on multilingual retrieval benchmarks such as CLEF, while also maintaining competitive results on English tasks in MTEB-R. This is particularly noteworthy given that multilingual models often experience a drop in English performance. The reviewers also commended the authors' commitment to open-source development, which enhances transparency and facilitates reproducibility—both crucial for progress in this field. Furthermore, the training pipeline is well-documented, with clear descriptions of data filtering strategies, contrastive learning, and the application of Matryoshka Representation Learning for compact and efficient embeddings. Given the paper’s solid contributions and well-executed methodology, I recommend it for acceptance.

Suggestions for improvements:
- As mentioned by one reviewer, please revise the paper to include statistical significance testing to strengthen the validity of our findings.
Decision: Accept