MILCO: Learned Sparse Retrieval Across Languages via a Multilingual Connector
Authors: Thong Nguyen
Yibin Lei
Jia-Huei Ju
Eugene Yang
Andrew Yates
Venue: ICLR 2026 Poster (2026)
Keywords: learned sparse retrieval
multilingual retrieval
cross-lingual retrieval
neural lexical search
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: MILCO aligns queries and documents into a shared English lexical space with two-stage training and a LexEcho head, preserving key source tokens while ensuring strong sparse multilingual performance and efficiency via pruning.

Abstract: Learned Sparse Retrieval (LSR) combines the efficiency of bi-encoders with the transparency of lexical matching, but existing approaches struggle to scale beyond English. We introduce MILCO, an LSR architecture that maps queries and documents from different languages into a shared English lexical space via a multilingual connector. MILCO is trained with a specialized two-stage regime that combines Sparse Alignment Pretraining with contrastive training to provide representation transparency and effectiveness while mitigating semantic collapse. Motivated by the observation that uncommon entities are often lost when projected into English, we propose a new LexEcho head, which enhances robustness by augmenting the English lexical representation with a source-language view obtained through a special [ECHO] token. MILCO achieves state-of-the-art multilingual and cross-lingual LSR performance, outperforming leading dense, sparse, and multi-vector baselines such as BGE-M3 and Qwen3-Embed on standard multilingual benchmarks, while supporting dynamic efficiency through post-hoc pruning. Notably, when using mass-based pruning to reduce document representations to only 30 active dimensions on average, MILCO 560M outperforms the similarly-sized Qwen3-Embed 0.6B with 1024 dimensions, while achieving 3$\times$ lower retrieval latency and 10$\times$ smaller index size.

Published as a conference paper at ICLR 2026
MILCO: LEARNED SPARSE RETRIEVAL ACROSS LANGUAGES VIA A MULTILINGUAL CONNECTOR
Thong Nguyen, Yibin Lei & Jia-Huei Ju University of Amsterdam (t (t..guyen2,> y. lei, j J @uva n1
Eugene Yang & Andrew Yates Johns Hopkins University. HLTCOE feugene. yang, andrew yates jhu.
ABSTRACT
Learned Sparse Retrieval (LSR) combines the efficiency of bi-encoders with the transparency of lexical matching, but existing approaches struggle scale be- yond English. We introduce MILCO, an LSR architecture that maps queries and documents from different languages into shared English lexical space via mul- tilingual connector. MILCO is trained with specialized two-stage regime that combines Sparse Alignment Pretraining with contrastive training to provide repre- sentation transparency and effectiveness while mitigating semantic collapse. Mo- tivated by the observation that uncommon entities are often lost when projected into English, we propose new LexEcho head, which enhances robustness by aug- menting the English lexical representation with a source-language view obtained through special [ECHO] token. MILCO achieves oftth--rrrr-tt multilingual and cross-lingual LSR performance, outperforming leading dense, sparse, and multi- vector baselines such BGE-M3 and Qwen3-Emhed standard multilin- gual benchmarks, while supporting dynamic efficiency through post-hoc pruning. Notably, when using mass-based pruning reduce document representations only 30 active dimensions on average, MILCO 560M outperforms the similarly- sized --eee- 0.6B with 1024 dimensions, while achieving 3x lower re- trieval latency and 10x smaller index size
INTRODUCTION
Learned Sparse Retrieval (LSR) represents queries and documents sparse lexical embeddings and retains the scalability benefits of bi-encoders (MacAvancy et al., 2020; Formal et al. 2021 Nguyen al. 2023) Unlike dense methods, LSR aligns representation with a natural language vocabulary, yielding transparent representations that facilitate error tracing and bias inspection. LSR naturally supports dynamic post-hoc pruning at inference time (Bruch et al. 2024), providing Matryoshka-like like latency control (Kusupati et al.) 2022) without requiring auxiliary training objectives. Empirically, benchmarks like BEIR (Thakur et al. 2021) and MTEB (Enevoldsen et al. 20255.. Theoretically. recent work shows sparse lexical embeddings exhibit higher representational capacity than dense embeddings, which illustrated by their superior performance on the LIMIT benchmark (Weller et al. 2025) where even state-of-the-ar
LSR (Lassance et al. 2024 Lei et al.] (025) is competitive
dense models fail catastrophically.
Thus far, LSR progress has been driven primarily by English (Formal et al. 2022 Shen et al. 2025
Nardini et al., (025), where models such as SPLADE (Lassance al. al.] 2024 deliver strong zero shot effectiveness and have seen wide adoption in production systems (e.g., OpenSearch, ElasticSearch. Sentence Transformers). Extensions beyond English remain fragmented: BGE-M3 (Chen et al. 2024) combines dense, sparse, and multi-vector vector heads under shared backbone, but its sparse com- ponent underperforms and lacks cross-lingual support; conversely, SPLADE-X (Nair et al. 2022b)
Our code is available http:://githbb..cm///noooottt

---

Published as a conference paper at ICLR 2026
and BLADE (Nair et al. 2023) target cross-lingual retrieval only and rely on training separate mod- els for each language pair, limiting their applications.
A straightforward multilingual LSR approach is to attach multilinguua MLM head to multilingual base encoder, projecting inputs into the full multilingual vocabulary. However, directly optimizing such models lead to severe semantic collapse (Nguyen et al. 2024), where representations lose interpretable term semantics, resulting in significant degradation the model's transparency and effectiveness. This behavior demonstrated both qualitatively and quantitatively in Section
18
125
85
24
+81
phone
English View
transter
Weight
Token
Weight
step
LexEcho Head
8
1.45
mo
1.18
import
import
phane
0.86
0.88
Import
music
18:
0.87
Yandler
0.72
Multilinuual Connector
0.65
songe
song
0.49
285
0.38
how
Multiiigual Encoder
0.38
f How import Momo Live music to mobie phone? (Tranelsion)
74
How import Momo Live music to muble phone? (Tranelatio)
@ Non-English Input (Chinase)
Figure 1: MILCO's LexEcho head produces two lexical views: (1) pivot (English) view supporting cross-lingual and multilingual retrieval, and (2) a source view for robustness uncommon entities.
To overcome those challenges, we introduce MILCO, illustrated in Figure | an LSR architecture that uses multilingual connector between multilingual base encoder and an English MI.M head, mapping text from all languages into a shared English vocabulary space. MILCO collapses the multilingual vocabulary English to create universal representation, which also reduces memory and computation during training. This approach enables one single MILCO model to support both multilingual and cross-lingual retrieval across many languages.
MILCO Training. We adopt two-stage training procedure. First, we propose Sparse Align- ment Pretraining (SAP), which maps multilingual inputs to English lexical targets, in contrast to prior dense alignment methods that operate in fow-dimensional latent space (Reimers & Gurevych, 2020). SAP leverages widely available bitext corpora instead scarce multilingual relevance la- bels, enabling large-scale multilingual pretraining. Alignment pretraining enables the model to then be fine-tuned with contrastive training using distillation (Lassance et al.) 2024). which enhances retrieval effectiveness while preserving grounding. Crucially, SAP is prerequisite: without align- ment, contrastive training leads to semantic collapse, harming effectiveness.
LexEcho Head. We observe that uncommon entities, especially from non-Latin languages, are often lost when projected into English. To address this, we introduce LexEcho, a dual-view LSR head illustrated in Figure [] The pivot (English) view is obtained by max-pooling over the logit matrix of an English head, with our multilingual connector enabling it to operate across many languages. The source view selectively echoes input tokens through special [ECHO] token, pre- serving entities that the English view fails to capture and assigning higher scores to more important tokens. This approach allows the model to represent entities it has never seen before or cannot translate.
Across 39 languages, our 560M MILCO model sets a new state of the art for Learned Sparse Re- trieval in both multilingual and cross-lingual settings. On MIRACL, our best model surpasses BGE- Sparse, BGE-Dense, and Qwen3- Embbe 8B by +34.1% %, +4.5%, and +3.6% nDCG@ 10, respec- tively, while also providing transparent representations, Experiments also show that the proposed LexEcho head enhances robustness to tail entities, yielding an +4.2% overall improvement on MIR- ACL. Like Matryoshka Representation Learnin, MILCO supports controllable efficiency via post-
2

---

Published as a conference paper at ICLR 2026
hoc pruning, surpassing Ewen3-Embed 0.6B with only 30 active dimensions per document, while achieving 3x lower retrieval latency and 10x smaller index size.
Our Contributions:
We introduce MILCO, multilingual connector architecture that maps queries and documents into a shared English lexical space, unifying multilingual and cross-lingual retrieval within a single model. Its LEXECHO head provides dual lexical views, enhancing robustness unseen or uncommon entities or concepts.
We introduce a new Sparse Aligument Pretraining (SAP) pretraining strategy tailored to multilin- gual LSR that addresses semantic collapse and provides the foundation for contrastive training, leading to an effective and transparent model.
Through comprehensive experiments on multilingual and cross- lonss-lnngaal benchmarks across 39 languages, we demonstrate that the MILCO architecture and Sparse Alignment Pretraining are key to achieving state-of-the- of-th-t-t multilingual and cross-lingual sparse retrieval.
2 RELATED WORK
Learned Sparse Retrieval (LSR). Zamani et al. (2018) first proposed SNRM, an n-gram neural model for learning sparse representations compatible with inverted indexes, though its representa- tions remained latent. Subsequent work (MacAvaney et al. 2020 Formal et al. (021) replaced SNRM with Transformer architectures that map text directly into the English lexicon, yielding more transparent and effective models. Nguyen et al. (2023) categorize LSR architectures into three groups: Binary Encoders, which assign binary weights to tokens and enable efficient inference-free query encoding with modest effectiveness trade-off's (Nardini et al., 2025 Shen al. 2025); MLP
Encoders, which score tokens by contextual importance (Macevaney et al. 2020 Lin & Ma (02T]; and MLM Encoders, used in state-of-the. tfftthe-art methods like Splade (Formal et al. 2021), which pro- vide differentiable query weighting and expansion. Beyond architecture, training protocols such as hard negative mining and distillation (e.g., from cross-encoders)
key to narrowing the gap with dense and hybrid systems (Formal et al. 2022 Lassance et al., 2024). In this work, we introduce MILCO, a new LSR architecture with LexEcho head for multilingual sparse retrieval.
Multilingual/Cros--nn language Retrieval A central challenge in cross-language IR is the language mismatch between queries and documents. Existing approaches address this either through transla- tion pipelines multilingual encoders that map text from different languages into shared latent space for cross- lonsulin matching. Representative efforts include dense encoder methods (Zhang al. 2024 Wang et al., 2024 iZhang et al.)
2025) and multi-vector methods with multilingual pre-
training (Louis et al. [2024] Yang et al.
2024a). Community benchmarks such as MIRACL (Zhang al. b) and NeuCLIR (Lawrie et al. 2024) provide standardized evaluation across many lan- guages, while studies on translationese highlight biases introduced by translated text (Gellerstam,
1986 Riley et al. 2020 Nair et al., [022a Zhang al., a). For sparse retrieval, BGE-M3 (Chen et al. 2024) combines dense, sparse, and multi- heads for multilingual retrieval, but its sparse component underperforms and offers limited cross-language support. Other sparse models such as SPLADE-X (Nair et al. 2022b) and BLADE (Nair et al. (023) focus on cross-language re- trieval with language-specific models. In contrast, our sparse model, MILCO,, supports both multi- lingual and cross-language retrieval within single model while substantially outperforming prior approaches.
Aligmment Pretraining. Previous work highlights the importance of multilingual pre-training for building shared cross-language semantic spaces (Conncau et al.) 2020; Chi et al.; Feng et al. 2022]] Yang et al. 2022). For retrieval, pre-training directly on relevance objectives has been explored, often using in-batch negatives and hard-negative mining (Zhang et al. 2024). Another direction focuses on distilling efficient models, where cross-encoder or ensemble teachers guide bi-encoder students to produce retrillal-friendll embeddings (Kim et al. 2023 Campos et al. (023). In multi- lingual IR, distillation also yields compact, language-agnose agnostic dense embeddings for scalable cross- language retrieval (Reimers & Gurevych 2020: Yang al.) 2024a). While prior work has mainly focused on dense models, we are the first to explore multilingual sparse alignment and introduce a sparse alignment pre-training method that enables LSR perform well multilingual data.
3

---

Published as a conference paper at ICLR 2026
3 PROPOSED METHODOLOGY
3.1 MILCO ARCHITECTURE
MILCO consists of three main components: (i) Multilingual Encoder, (ii) a Multilingual Con- nector, and (iii) LexEcho Head.
MILCO processing the Chinese input: "How to import Momo Live music to a mobile phone?" Let denote the set supported languages. For an imput text in language E L, we first tokenize it into a sequence of souree tokens:
(81....,8p)
Multilingual Encoder. A transformer- Multilingual Encoder Enc(() maps the input tokens g(4) into sequence of hidden states of dimension de in multilingual embedding space:
H i Enc(s(()) E Rxdd
where H represents the contextualized embeddings for the n input tokens. For conciseness, omit the superscript (8) whenever the language space of the variable is unambiguous, making it H.
Multilingual Connector. The Multilingual Connector then projects these multilingual hidden states H into Z of dimension de, which live in the embedding space of the pivot language:
Z LayerNorm (Linear(ø(H) (H)) Raxd, where <(.) Raxde - Rexd.
For simplicity, implement the connector with Multi-Layer Perceptron. This projection unifies representations across different languages through English as the pivot, allowing our LexEcho Head to project them into shared English lexicon. While architecturally there is restriction the selection of the pivot language, we select English because of its rich resources and the availability of LSR teacher models for alignment, which discuss later in this section.
LexEcho Head. The LexEcho head produces a dual-view lexical representation from the projected states Z. generates two complementary sparse views: 1 an Pivot (English) View that captures semantic concepts in English and @ Source View that preserves important source input tokens.
@ Pivot (English) View: The English lexical representation is generated by an English MLL head, in LSR models like SPLADE (Lassance et al. 20244), but our multilingual connector extends this to the 39+ languages supported by our base model.
Multilingual representations Z (Eq. 3 are linearly refined and decoded onto the English vocabulary
V. via embedding matrix E E R Velxde and bias b,, yielding logits that score each source token against every English token.
T log (1 + ReLU (Dec(Z))) E IR R where Dec(x) xE + by-
Here, we define the log saturatioo effect function, introduced by MacAvaney al. 2020 Formal et al. (2021) as LogSat(-) for simplicity,
LogSat(x) = log(1 + ReLU(x))
(5)
Next, max-pooling across source tokens (n) yields the final English lexical representation:
(c) = max T(e) i1 max (ee) max r( where i E [1,
(6)
This English view ((e) is sparse and includes not only direct translations (live, music, phone) but semantically related terms (song, stream, step) that supports semantic retrieval.
Source View: The connector maps common concepts into English but can fail on uncommon unseen entities, especially in non-Latin scripts (e.g., Momo in Figure D. when names differ across languages (e.g.. Douyin vs. TikTok). Scaling model size alone cannot solve this, as new entities continually appear.
4

---

Published as a conference paper at ICLR 2026
Our LexEcho head tackles this by selectively echoing key tokens from the source. A dedicated [ECHO] token in the MLM head, denoted DEqECHOI(.),, produces a weight vector w for each source token to ensure crucial tokens selected:
LogSat (DeqjEchO) (Z)) E R
By combining the English t(e) and the weighted source views {s wi).= MILCO produces dual-view representation o = {t((=, s(1) w that leverages cross-lingual projection form a unified lexical view preserving crucial source-language tokens that would otherwise be lost in translation.
3.2 TRAINING: SPARSE ALIGNMENT AND CONTRASTIVE REFINEMENT
We propose a two-stage training recipe for MILCO: Sparse Alignment Pretraining to ground mul- tilingual text to English lexical space, followed by Sparse Contrastive Training to refine alignment and optimize retrieval effectiveness, with sparsity enforced throughout.
Sparse Alignment Pretraining (SAP). To ensure the English view t(e) is grounded in the English lexicon, we leverage widely available parallel (xx-en) sentences to align the English view of non-English sentence the representation of its corresponding English sentence. Given a pair of tokenized parallel sentences (s((), (())) in language and English, we employ an oracle teacher English LSR model, such as SPLADEv3 (Lassance al. 2024). denoted as LSR", to produce the target English sparse representation t².
We design sparse-aware MSE (SMSE) loss, specifically to minimize the difference between two sparse vectors. Since most coordinates are zero, the learning signal should concentrate on the few active ones. Also, with the (-) activation, negative pre-activation values yield zero gradients. Therefore, we compute the loss directly on the decoded logits, i.e. Dec(Z), which were the input to LogSat(-). with max pooling across the input tokens and restrict it to coordinates where at least one side positive, For clarity, we denote such augmented representations as ((e) and E*, Formally, the SMSE loss can be written as
IVal EK 1
(E(e) V Ej
LSMSF (11)
IVl E/Ed 1 ((() V 1j 15>0
where 1(-) denotes the indicator function. This SMSE objective mitigates gradient dilution and focuses training on informative lexical coordinates, yielding more stable alignment. During training, apply SMSE over batches flattened into single vectors.
Sparse Contrastive Training (SCT). Alignment pretraining grounds multilingual inputs in shared English lexicon but is not directly optimized for retrieval. To improve effectiveness, we further train MILCO with a LexEcho head using contrastive objective retrieval datasets. Fol- lowing Lassance et al.) (2024). , we use KL distillation loss (details in Section A.7) trans- fer knowledge from cross-encoder to MILCO. To promote sparsity, add fn-orrm regular- ization on query and document representations q and p. Concretely, the training objective is Leontrastive LLLD + agl19l11 + aul|pl/ı, where the no-mms are implemented as means over the training batch.
4 EXPERIMENTAL SETTINGS
Pretraining, Training and Evaluation Data. For Sparse Alignment Pretraining. we use 594M bitext pairs from diverse domains collected with Sentence Transformers (Reimers Gurevychj 2019) where each pair contains an English sentence and its translation. Dataset statistics are shown in Table 14 For Sparse Contrastive Training, we adopt the 1.4M multilingual queries released by Chen et al. (2024), with positive/negative documents and teacher scores obtained from bge-reranker- v2.3 reranker. More details are Table
bgg-rermnkrr2255gemm2llliggge
5

---

Published as a conference paper at ICLR 2026
Following Chen al. (2024). evaluate MILCO four benchmarks: MIRACL (Zhang al. b). a large-scale multilingual retrieval benchmark covering 18 languages with high-quality human annotations; MTEB v2 (Enevoldsen et al. 2025) for large-scale multilingual retrieval: MLDR CChen et al. 2024), a multilingual long-documedt retrieval benchmark in 13 languages; and MKQA Long- pre al. 202T), cross-lingual benchmark with English documents and queries in 25 languages. Additional results on BEIR (Thakur et al. (2021), NeuCLIR (Lawrie et al., (2024) and LIMIT (Weller
al. 2025) are also included in the Appendix. Our evaluation spans 39 languages total.
Table Multilingual passage retrieval performance on the MIRACL dev set (measured by nDCG @ 10). Superscript ": results obtained from Lassance (2023).
Model
Siae
Dono, muttissncior and lyyirid baselines
nfES.
560M
66.6
16.0
75.9
52.0
$2.0
59.0
77.8
54.5
62.0
52.0
70.5
66.5
67.4
74.9
84.6
50.2
56.0
$6.4
78.3
E5 E
7,118
63.4
73.3
70.3
57.3
52.2
52.1
34.7
55.2
52.1
52.7
96.8
61.8
62.3
68.4
73.9
74.0
$4.0
54.1
79.7
MJ Dense
360M
692
75.4
80.0
56.9
56.1
60.9
790
28.5
39.5
26.1
72.8
699
78.1
25.7
86.2
82.6
82.7
26.7
51.8
Mi-Mallivve:
SM0M
70.5
79.6
81.0
59.3
57.8
62.0
90.1
59.4
61.5
58.3
74.5
71.2
71.2
29.1
87-9
850
83.7
58.0
$2.4
S60M
70.6
79.6
80.7
58.8
S8.1
62.3
79-3
56.0
62.9
58.9
73.9
21.2
498
78.5
87.2
83.1
63.5
57.7
85.3
M3-DaaeeeSperee Soprro-Mullivaaee
560M
71.5
90.2
81.5
59.6
90.7
63.4
$0.4
61.2
63.3
59.0
75.2
72.1
71.7
79.6
88.1
53.7
64.9
99.8
83.5
FLAID X (bBatovector)
560M
15.5
65.9
68.01
46.4
51.4
48.5
52.5
61.9
42.8
56.8
14.5
61.2
03.4
61.2
32.9
75.6
72.0
44.5
49.1
Qwen Limbed 0.60
5WM
00.5
69.9
06.3
51.5
54.3
52.7
69.7
54.4
51.3
51.
63.5
60.1
59.7
08.6
71.2
733
58.3
52.9
74.0
Qwnn Sor---tmeed Eimbed 315
7,571
69.8
75.2
78.3
59.8
99.6
60.5
190
61.0
63.1
56.1
74.3
67.5
73.5
73.2
84.5
81.5
61.3
60.5
56.5
MIl CDdems plign diailty
560M
67.9
77.0
76.6
58.3
87.5
6012
77.1
59.0
60.5
55.5
70.5
70.9
67.5
74.4
86.1
50.6
67.5
57.6
72.8
MILCO dense (disa))
560M
70.9
79.5
80.2
56.8
60.4
63.4
78.5
62.6
62.
58.4
74.7
79.5
72.1
79.5
87.0
52.9
64.2
59.5
53.2
baseliees
Splade
3.4B
ISPLADDEID""
OpenSvont
187M
54.2
51.4
36-7
48.6
58.3
65.9
60.7
66.8
24.00
56.2
M3-3qume-
550M
34.5
65.4
55.3
46.2
48.9
56.1
61.5
44.5
70-1
70.0
I MILED ISAP SCTKD- Teafchho
S6IM
72.1
98.4
82.6
61.4
68.9
62.3
$1.2
61.7
64.4
60.9
77.2
72.1
74.6
W9.3
H7.9
84.2
65.5
61.4
K3A
@ MILCO SCTxD- MLMas?
560M
69.4
77.3
79.5
$7.6
59.7
55.5
79.8
60.5
63.4
57.7
72.5
67.7
72.6
78.1
82.3
50.4
66.5
59.7
51.2
3 MILCO ISAR SCT, LoEeko)
560M
70.1
79.4
80.8
57.6
57.2
6016
90.1
57.7
63.3
28.4
75.2
71.0
72.6
77.2
81.2
52.7
06.5
59.3
51.6
# MILCO (SAR MLMen)
560M
36.5
59.8
59.7
57.0
56.0
44.9
66.0
65.2
55.6
48.8
24,9
59.4
51.2
47.8
55.0
55.
16.7
48.5
62.2
8 (SCTTD Milaa
360M
59.2
72:7
72.3
47.7
47.6
50.9
12.5
65.8
50.4
51.8
64.0
62.7
53.6
62.3
71.7
72.8
46.2
44.8
66.2
6 a*MILCC (CCTgD- MLMss
360M
50.7
65.5
62.0
39.7
39.4
42.0
67.0
39.5
36.9
44.9
56.1
52.9
47.3
58.5
71.0
67.0
41.8
34.6
46.9
Baselines. We compare MILCO against two group of baselines: Dense/Mullti-vettor and Sparse methods. For dense/multi vector baselines, we include recent state-of the-ar art methods, including multilingual E5 (Wang et al., 2024). BGE-M3 (Chen et al.) 2024), PLAID-X (Yang et al. 2024b).
Qwen3 Embeddings (Zhang al. 2025). For sparse buselines, we include unsupervised BM25, M3-Sparse (Chen et al. (004) and also OpenSearch (Shen et al. (025), T-Splade (Lassance 2023).
mSplade (Lassance 2023). Among these, T-Splade is the approach that translates text into English and encodes the translated text by the Splade model (Formal et al. 2021).
MILCO configurations. We consider the following configurations in experiments:
MILCO (SAP, SCTKD- LexEcho): Our strongest sctup, which combines alignment with contrastive distillation training and the LexEcho head, producing dual-view lexical representations.
MILCO (SAP, SCTxD, MLMca): Similar to (1), but the source view is removed from LexEcho's output, producing only English lexical representations.
MILCO (SAP, SCT, LexEcho): Similar to (1), but without distillation. Instead, the InfoNCE loss (Oord et al. 2018) with in-batch negatives is used for Sparse Contrastive Training. MILCO (SAP, MLM Similar to (2), but without Sparse Contrastive Training.
MILCO (SCTRd. MLMen): Similar to (2), but without Sparse Alignment Pre-training. noMILCO (SCTkD- MLM.): A baseline model trained directly with the full multilingual MLM head (without our multilingual connector).
We initialized MILCO from the bge-m3-une tamm----uupeevseed multilingual base encoder and initialized the LexEcho head with Splade-v3's English MLM head (Lassance et al., (2024). We use Splade- v3 representations of English text for alignment pretraining. More details on hyperparameters and hardware are provided in Section A.8 of the Appendix.

---

Published as a conference paper at ICLR 2026
5 RESULTS AND DISCUSSION
RQI: How does MILCO perform compared to state-of-the-art baselines? Table reports the performance of MILCO and baselines on the MIRACL benchmark (18 languages). Overall, the MILCO @ model, trained with our two-stage pipeline and LexEcho head, achieves the highest effectiveness with an average nDCG@ 10 of 72.3.
Against sparse baselines, MILCO outperforms M3-Sparse (Chen et al. 2024) by 34.1%, T- Splade (Lassance 2023) by 32.7%, and MSpladesTok (Lassance) 2023) by 13.1%. Against dense baselines, MILCO still shows substantially higher effectiveness, though with smaller margins. Compared to models of similar size, outperforms Qwen3 0.6B (Zhang al., (025) and M3- Dense (Chen et al.] (024) by 19.5% and 4.5%, respectively, on MIRACL. This advantage gen- cralizes to 39 languages on MTEB v2 cross-lingual and multilingual retrieval (Table 3 Despite being ~44 smaller, it outperforms E5-Mistral 7B (Wang et al. 2023) and Qwen3 8B on MIRACL, though Qwen3 8B performs better on MTEBv2 where it better leverages -pecifie instructions. We additionally train two dense baselines using the same backbone and training data MILCO. The first, MII.CO-dense (align + distill), which uses dense alignment to then] per/gteeregeeeeee se and distillation, achieves average nDCG@ @ 10 of 67.9 on MIRACI. A variant trained with distil- lation only performs better, reaching 70.9 nDCG@ 10 on MIRACL, However, both dense baselines still substantially underperform our best sparse MILCO@ trained with the two-stage recipe.
Table 2: Performance on Multilingual Long
Table 3: Performance on multilingual and cross- lingual retrieval tasks Multilingual MTEBv2. (39 languages). More details in Table
Document Retrieval (nDCG@ 10, 13 languages).
More language-specifff details in T
Model
Size
Max Length
Avg
Model
Size
Avg
Dense vesti-vector and lyybrid boselines
Large Models (21B)
mEStegs
560M
512
34.2
Qwen3-Emmbed-88
8B
75.59
7B
8192
42.6
jinz-embeedding--
3.88
73.84
M3-Dense
560M
52.5
retrieeee vi
7.1B
ME1-Muliivecttor
560M
8192
57.6
SFR TEReFinbedddngg-iine
7.IB
68.50
Dense-Sparse
goc-Qwen2-7BB
7B
67.22
M3-All
560M
8192
65.0
inf-vetriever-v1-1.55
1.5B
65.34
IGTE-TRM Dense
304M
8192
56.9
gto-Qwen2-1.5B-in
1.58
65.12
BOTEE-TMM Dense Sparse
304M
8192
71.3
GriLLM-7B
7B
PLAID-X (Multi-vactor)
560M
512
74.2
NV EW-emnedd-22
7,9B
58.65
Dwon3-Embed-0068
0.6B
32768
NV-Fmbhdd-x
7,9B
Qwen3 Eann--EEeedd--e 8B
8B
32678
59.1
Small Models (<.IB)
Sparze baselines
guo-mulilingull-aaee
305M
64.72
BM25
8192
53.6
Qwen3-Embeed Embedd-0.6
0.6B
M3-Sparse
560M
8192
62.2
bge m3
560M
62.02
mGTE-TRM Spurse
granite-77mm-mult
278M
55.80
granite- 107m-multi
107M
49.88
MILCO (SAP, SCTgD, LexEcho)
560M
512
74.4
2 MILCO (SAR SCTgp, MLM
560M
512
69.9
MILCO SCTgD- Lexficho)
560M
66.83
We further evaluate MILCO on the Multilingual Long Document Retrieval (MLDR) benchmark (Table 2). Because MILCO is trained with a 512-token limit, we split long documents into 512- token passages and score documents by their best passage. Under this setup, MILCO achieves an average nDCG @ 10 of 74.4, which is 14% higher than M3-All, the dense--spare--mutiivvved ensemble, and substantially surpasses Qwen3 0.6B and 8B with native long-context support.
In the Appendices, we report results on LIMIT Test (Weller et al. 2025) (Table IT] and BEIR Thakur et al., 2021) (Table I0p. On LIMIT, MILCO substantially outperforms all dense basclines regardless of size. On BEIR (English), it trails Qwen3 0.6B slightly, but scales better on large collections.
RQ2: What is the effect of sparse alignment and contrastive training in MILCOO? We observe that Sparse Alignment Pretraining is crucial to ensure that the model's output is grounded in the English vocabulary. In Figure 2 show two examples of MILCO's output under three training setups. Without SAP, contrastive training leads to semantic collapse, where the model produces
'then pemniper/gteen per/ is similar in size and BEIR (English) performance to our SPLADE-v3 sparse English teacher (GTF-base: 52.61 nDCG@ 10, SPLADF-v3: $1.69 nDCG@ 10).
7

---

Published as a conference paper at ICLR 2026
completely random and unexplainable (latent) output tokens with clear relation to the input. We observe the same effect when we train noMILCO @ a multilingual LSR model without the multilingual connector (similar to Splade training). With alignment pretraining, MILCO produces understandable and semantically equivalent English tokens as demonstrated in the figure. However, observe that both Alignment-only and Contrastive-only result in medioere multilingual retrieval effectiveness. On MIRACL results in Table ! Alignment-onyy MILCO @ and Contrastive-only MILCO @ only achieve the average nDCG@ @ 10 of 54.5 and 59.2 respectively. Direct training without our connector (noMILCO @)) leads to larger drop in performance, resulting in nDCG@ 10 of 50.7.
To further improve retrieval effectiveness, we finetune MILCO on retrieval data with a contrastive objective. We experiment with two contrastive losses: InfoNCE with dataset-provided labels (MILCO ))) and KI. divergence for knowledge distillation (MIL.CO @). With an InfoNCE loss, MILCO E's average nDCG@ 10 improves by 28.62%, from 54.5 with only alignment to 70.1, be- coming competitive to AGG--M3Deense and Multi-vector models. Adding distillation further boosts effectiveness, increasing nDCG @ 10 to 72.3 and making MILCO subtantially outperform all base- lines, including the hybrid BGE-M3 dense-sparse-mullivee multivector model and Qwen3 models.
Does the proposed LexEcho head improve robustness? Unlike dense or multi-vector methods, the transparency of MILCO's sparse, lexicalized representations make errors traceable. When analyzing the English view, we found that representations often miss uncommon entities like Momo in Figure leading reduced retrieval accuracy. In the figure, Doc2 (score 9.64) is ranked below Doc1 (score = 9.89), despite being more relevant.
RQ3:
The LexEcho head addresses this with dual-view representation composed of English view and a source view. When an important entity is missing from the English view, MILCO can fall back to the source view for source-token matching. In Figure 3 LexEcho seems recognize the model's missing knowledge of Momo in English and assigns a high weight to Ri in the Chinese view. In contrast, for Apple, the model relies primarily on English representations (assigning them the highest weights in Docl and Doc3) while assigning T (Apple) a low weight in the Chinese view.
On MIRACL (Table D. MILCO ① with a LexEcho head consistently outperforms MILCO @ with only an English view across all 18 languages, achieving an average nDCG@ 10 of 72.3 (+4.17% over 69.4). The largest gains oceur non-Latin languages such as Chinese (zh: +8.09%), Telugu (te: +6.8%), Farsi (fa: +6.5%), Korean (ko: +6.5%), and Japanese (ja: +6.04%), where mapping entities into English is particularly difficult since entities could be named differently in English. The benefits of the LexEcho head also extend to long-document retrieval: on MLDR (Table MILCO with LexEcho achieves 74.4 nDCG@ 10, a 6.43% improvement over MILCO with only an English view (69.5). These highlight the broader robustness of our approach.
RQ4: Can MILCO perform zero-shot cross-lingual retrieval? MILCO uses the multilingual connector to maps text across languages into a unified English lexical view. This allows MILCO to perform zero-shot cross-lingual retrieval, which is not possible with sparse models like M3-Sparse that rely on only a source view. We benchmark the cross-lingual capability of MILCO (zero-shot) and baselines on MKQA with R@ 100 in Table
Prior sparse methods (c.g., BM25, BGE-M3-Sparse) generate source-view representations including input tokens with scalar weights. Their vocabularies are language-specifie, so inputs in Chinese
Alignment Only
Alignment Contrastive
Contrastive Only
(de): Baltimore Maryland großartigste Stadt Ameria (Datimore Maryland the greatest city America)
baltimore (2.22), maryland (1.80 (1,86), city (1.40).
baltimore (1.77), city (1.52), maryland (1.23), america (1.18), greatest (0.89), usa (0.81).
governing (1.17), past (1.07), match (0.95), worn (0.86), sky (0.65), gas (0.52), boot (0.34), mayor (0.31)
greatest 25), biggest (0.98), garrison (0.35),
geography (0.31), tourism (0.19) ...
(0.82), urban (0.26) ...
Input (vi]: Gia in tai sān rōng pãa Testa la bao nhiéu? (What is Testa's is net worth?)
tesia (3.36), worth (2.1)), price (1.84), net (1.52).
tesia (3.02), worth (1.89), price (1 (1.47),net (1.37), wealth (0.71), asset 63), stock (0.42). company (0.41)
relative (0.97), drinks (0.75), gaelic (0.75). contaminated (0.73), sigh (0.57), weab (0.5) dust (0.46). (1.22)
salary (1. (1)14), money (0.80), mining (0.35).
generation (0:33)...
Figure 2: Sparse representations with different training strategies. Alignment only produces many grounded tokens (green) but also distantly relevant tokens (orange), Contrastive further prunes and refines. Contrastive-onty suffers from semantic collapse, drifting toward ungrounded tokens (red).
8

---

Published as a conference paper at ICLR 2026
Input
Tranalation
(1) Leaticho [English View
(2) LacEcho (Source Viea?)
Score (1)
Score (1)112)
Query:
How import Nomo Live Music mobila phoow?
ronsfort (crg)6.40) phorma)8.46). alwaarsl. 44)
comgso0 (vel).01)
210,44) 10 25 A10.355,
M80 W1 <pe10.30)
se),
Doc 1.
FC 00.38
Userra can downloww: Apple Must:
and then import tham thair phones
songg1 1000), canjt 1.00) suvee 1,001 spoto 96)...
downloadd shoned
play' 14),
(0.00)
BRH (0 28), (0,000227 88-0.00
1000 1975
34), 1H1000.34)
Rark
9.09
Rank 2
Doc2: 16 il 18 (8 (1
Stngge from Momo Live cann be to your phone using the nowe function
songlt musti (o)) storaget 01), liwep cargo n1), tranefer B1), atmommj..77) 78))
phonett
(6.56). M960.44) (0.53). 1310
19 1675
9.64 Rark2
Rank
Figure 3: The tail entity Momo is missing in the English view of the query and Doc2, reducing Doc2's s score despite its higher relevance. The LexEcho head resolves this by selectively retaining missing entities from source tokens, correctly ranking Doc2 on top.
yield only Chinese tokens. This causes vocabulary mismatch and poor cross-lingual retrieval, with average R@ 100 scores of just 39.9 and 45.3 on MKQA. In contrast, MILCO avoids this issue with a shared English lexical space. Despite not being trained for gross-lingual retrieval, MILCO achieves strong results on MKQA, with a zero-shot R 100 of 76.6, improving 91.9% and 69.1 % over BM25 and BGE-M3-Sparse, respectively.
Dense and multi-vector methods operate in a latent space, so they do not suffer from vocabulary mismatch and per- form reasonably well on MKQA. BGE-Dense and multi- vector models
Table 4: Cross-lingual retrieval perfor- mance MKQA, averaged across 25 languages. More details in Table
among the strongest baselines, with 100 of around 75. While these methods
Model
Avg (R 100)
an average R @ outperform sparse baselines (e.g.. BM25 or BGE-M3- Sparse), MILCO achieves about 1. 1.7-1.9% higher @@ 100 than the BGE dense and multi-vector baselines, while also retaining the transparency that facilitate model anal- ysis and error tracing. MILCO is about 9% and 12.8% better than E5-Mistral 7B and Qwen3 8B, respectively, despite having only 560M parameters.
Baselines
ES lurge
70.9
ES misstral-hh
70.1
BGE-M3 Dense
75.1
BGE-M3 Multi-Vee
75.3
BGE-M3 Demse+Sparse
PLAID X (multivector)
73.4
Qwen3-Embel Emmeed0.66
54.4
Qwen3 Embed SB
67.9
BM25
39.9
BGE-M3 Sparse
45.3
6 EFFICIENCY
MIL.O (SAP. SCTgD: LasFalo)
AND EFFECTIVENESS TRADEOFFS
Model Size vs. Effectiveness. In Figure we plot
MILCO's effectiveness against model size compared to basclines. We observe that MILCO, with 560M parameters, is the most effective model within its size range and even substantially outper- forms larger models (e.g.. Qwen3-8B and E5-Mistral-7BB across all 18 MIRACL languages. With the same model size, the BGE-M3-Multivedd model underperforms MILCO despite producing multiple dense vectors for each query/document.
Sparsity vs. Effectiveness. Dense retrieval models like Matryoshka representations (Kusupati et al. 2022) support truncating embeddings for efficiency, but require additional Matryoshka training losses. MILCO and LSR methods naturally allow post-hoe pruning (Lei et al., 2025 Wen et al. 2025 Bruch al. 2024), because LSR encodes queries and documents weighted tokens that can be ranked and truncated at inference. Unlike Matryoshka, which applies the same truncation size to all inputs, LSR supports variable i: (e.g., fewer tokens for shorter texts). | two pruning strategies: top-k pruning and mass-based pruning, which removes the p-tail percentile of token weights, yielding variable tokens per document. Exact numbers are included in the Appendix 12 Mass-based pruning delivers slightly more favorable trade-off than top-k pruning. At = 95, it averages only 30 tokens/document yet already surpasses Owen3-Embed 0.6B on nDCG@ 10 (62.2). It reaches 96% of full performance at p 86 (86.4 tokens/doc) and achieves SOTA at 300 tokens, with only marginal gains beyond. With LexEcho's vocabulary of 280k terms, activating just 300 tokens (0.1%) yields representations that are 99.9% sparse, transparent, and highly effective.
Sparsity vs. Efficiency. We report index statisties and retrieval latency with MILCO under an inverted-index setting. We use Seismic (Bruch et al., 2024). an ANN method built on top of an inverted index. All results on MIRACL (Eiglish subset, about 32M passages), with retrieval run

---

Published as a conference paper at ICLR 2026
70
M3-Multivver M33-Dense
MILCO MLM
LexEcho
Qwen3
70
67.4
70.6
ESlarge
nDCG@10
ESmistral 78
nDCG@10
60
Owen3 0.68
60
T-Splade
M3-Sparse
50
Tap-K Pruning
Mass-band base- Pruning
Model Typ-
M3 MB--Spasse (33.9)
Owen3-0.668 23)
m mContrieeer mDPR
40
M3 Multivector (70.5)
0.1
0.2
0.5
2
10
0
100
200
300
400
Model Size (Billion Parameters, log scale)
Tokens Kept
Figure 4: Model size versus effectiveness on MIRACL. MILCO is lightweight (560M params), while being highly effective.
Figure 5: Effectiveness (nDCG@ 10, MIRACL) of MILCO with varying sparsity levels obtained post-hoc pruning methods.
Table 5: Retrieval Latency of MILCO sparse retrieval with Seismic and Qwen3- Saaendd-mbdddin-O- 0.6B dense retrieval with Faiss.
Model
Index
Avg Latency (ms)
P95 Latency (ms)
QPS
nDCG@
Index Size
Qwen3-Embee Embed-0.6B
HNSW
1.29
1.47
777
50.4
134G
MILCO (p=0)
Seismic
1.85
4.32
538
56.4
61G
MILCO (p=10)
Seismic
1.29
2.92
774
56.3
40G
MILCO (p=30)
Seismic
0.61
1,26
1647
54.4
25G
MILCO (p=50)
Seismic
0.65
1.33
1545
53.3
16G
MILCO (p=60)
Seismic
0.44
0.82
2265
50.3
12G
on single AMD EPYC 7763 CPU core. build several indexes of the pivot view with hyper- parameters (n.postings=1500)
query.cut=10, deap-factor:0.9 and different amounts of post-hoc pruning. The unpruned index (p = 0) has an average posting-list length of 4636.09, resulting in a 61 GB index and an average retrieval latency of 1.85 ms/query. We then prune the lowest-weight dimensions whose cumulative weights account for E {10, 30, 50, 60 of the total, which shrinks the inverted index and speeds up retrieval. Results shown in Table 5
Regarding index size, post-hoc pruning substantially shrinks the index: 40 GB at 10, 25 GB at p 30, 16 GB at p 50, and 12 GB at p 60. Thus, the most aggressive pruning level, we duce index size by roughly 80% while keeping strong retrieval effectiveness. To contextualize these index sizes, we compare against Qwen3-Enes Eandn---mmdddn-.....
with Faiss dense HNSW index (Douze et al. (2224 (M 32, ef = 64) on the same hardware and collection. This dense baseline has an index size of 134 GB, whereas MILCO's pruned Seismie index is already smaller at p = (40 GB) and becomes 5-10x smaller at higher pruning levels (25 GB at 30, 12 GB at p 60).
Regarding latency, pruning also yields consistent improvements over the full representation: p 10 reduces average latency from 1.85 ms to 1.29 ms (~30% speed-up) with virtually no loss in nDCG@ 10 (56.4 56.3), p = 30 makes queries about 3x faster (0.61 ms, nDCG@ 10 = 54.4), and p 60 achieves For comparison, the 1eee33--mmmdddnnmmmm-eee B Faiss HNSW achieves an average latency of 1.29 ms and nDCG@ 10 50.4. Seismic's ANN inverted index with pruning therefore allows MILCO to (i) match this latency at = 10 while achieving substantially higher effectiveness (nDCG@ 10 = 56.3), and (ii) further reduce latency 0..44 ms at p 60, while remaining least effective overall.
4x speed-up (0.44 ms) while remaining competitive (nDCG@ @ 10 50..).
CONCLUSION
We introduced MILCO, a novel multilingual learned sparse retriever that connects 39 languages to shared English lexical space through lightweight connector. Alignment pretraining enables the use of contrastive training, whereas the LexEcho preserves entities lost during cross-lingual projection. MILCO delivers strong zero-shot cross-lingual retrieval, showing competitive performance with- out cross-ling cingull retrieval training. Overall, MILCO achieves tfate-oftthe-ee ahtt multilingual retrieval results, while offering transparent representations and efficient post-hoe pruning.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We have taken several measures to ensure the transparency and reproducibility of our work.
Datasets. All datasets used for pretraining and training MILCO models are publicly available. Table 14 (Appendix) reports statistics on the number and sources parallel sentences used for Sparse Alignment Pretraining, while Table [5 (Appendix) describes the datasets used for Sparse Contrastive Training. These datasets are widely adopted in prior work on dense retrieval (Wang al. 2024 Chen et al. 2024 ei al.). We note, however, that some recent models such as Qwen3- Embed (Zhang et al. 2025) do not disclose their training data, which makes direct comparisons not
strictly fair.
dlppe--arrametes and hardware. All hyper-parameter and hardware specifications used for Sparse Alignment and Sparse Contrastive Training are described in Section A.8 (Appendix). Any hyper-parameters not explicitly listed are set to the default values provided in HuggingFace's Trainer (Wolf et al., 2019), which we use to train our models. During pretraining and training, we hard-coded the random seed to 42.
Code. Our code is available at:
Models and evaluation. MILCO is trained on 63 languages and evaluated on 39 languages, as detailed in Section A.9 (Appendix). Trained MILCO checkpoints are released at: hhtpp::/ github. catiiuu..com/ttt www//tooonnitttccccc 99 mmimimiii milco To illustrate the multilingual capabilities of our model, we provide example inputs and their corresponding sparse representations across multiple languages in Section A.I (Appendix).
ETHICS STATEMENT
We present MILCO, multilingual learned sparse retrieval method supporting 39 languages. All datasets and models used to train MILCO are publicly available, and we do not introduce any pro- prietary sensitive data. Since work builds on public data, it may reflect biases present in those sources. We aims to broaden access to multilingual information retrieval research, especially for underrepresented languages.
ACKNOWLEDDGENT
This research supported by the Hybrid Intelligence Center 10-year program funded by the Dutch Ministry of Education, Culture and Science through the Netherlands Organisation for Scien- tific Research, project VT. Vidi.22.. 166 of the NWO Talent Programme which is (partly) financed by the Dutch Research Council (NWO). We acknowledge the Dutch Research Council for awarding this project access the LUMI supercomputer, owned by the EuroHPC Joint Undertaking, hosted by CSC (Finland) and the LUMI consortium through project number NWO-2024.050.
REFERENCES
Luiz Bonifacio, Vitor Jeronymo, Hugo Queiroz Abonizio, Israel Campiotti, Marzich Fadace, Roberto Lotufo, and Rodrigo Nogueira. mmarco: A multilingual version of the ms marco passage ranking dataset. arXiv:2108. 13897, 2021.
Sebastian Bruch, Franco Maria Nardini, Cosimo Rulli, and Rossano Venturini. Efficient inverted indexes for approximate retrieval over learned sparse representations. In Proc. of SIGIR, pp- 152-162, 2024.
Daniel Campos, Alessandro Magnani, and Chengxiang Zhai. Quick dense retrievers consume KALE: Post training KullbackLeibler alignment of embeddings for asymmetrical dual encoders. In Proc. of ACL Workshop on SustaiNLP, pp. 59-77, 2023.
Jianly Chen. Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Multi-linge Lanti-lingual, multi funtiiiiunccionalitt.. multi multi-granularity text embeddings through eelf-knowledge dis- tillation. arXiv:2402.00) 03200200332222 16, 2024.
11

---

Published as a conference paper at ICLR 2026
Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. InfoXLM: An information-theortiicc framework for cross- lingual language model pre-training. In Proc. of NAACL-HLT.
Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S Weld. Specter: Document- level representation learning using citation-informed transformers. In Proc. of ACL, pp. 2270- 2282, 2020.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning at scale. In Proc. ACAC, pp. 8440-8451, 2020.
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre- Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. 2024.
Kenneth Enevoldsen, Isaac Chung, Imene Kerboua, Márton Kardos, Ashwin Mathur, David Stap, Jay Gala, Wissam Siblini, Dominik Krzemiński, Genta Indra Winata, et al. Mmteb: Massive multilingual text embedding benchmark. arXiv:2502.13595, 2025.
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. Elis: Long form question answering. In Proc. of ACI., pp. 3558-3567, 2019.
Fangxiaoyu Feng, Yinfei Yang. Daniel Cer, Naveen Arivazhagan, and Wei Wang. Language- agnostic bert sentence embedding. In Proc. of ACL (Volume I: Long Papers), pp. 878-891, 2022.
Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. Splade: Sparse lexical and expan- sion model for first stage ranking. In Proc. of SIGIR, pp. 2288-2228222 2292,,2922222 2021.
Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. From distillation to hard negative sampling: Making sparse neural ir models more effective. Proc. of SIGIR, pp. 2353-2355 2022.
Martin Gellerstam. Translationese in swedish novels translated from english. Translation studies in Scandinavia, 288-95,11 1986.
Wei He, Kai Liu, Jing Liu, Yajuan Lyu, Shiqi Zhao, Xinyan Xiao, Yuan Liu, Yizhong Wang, Hua Wu, Qiaoqiao She, et al. Dureader; a chinese machine reading comprehension dataset from real- 2017.
world applications. arXiv:1711.05077
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. Proc. of ACL (Volume I: Long Pa- pers), 2017.
Seungyeon Kim, Ankit Singh Rawat, Manzil Zaheer, Sadeep Jayasumana, Veeranjaneyula Sad- hanala, Wittawat Jitkrittum, Aditya Krishna Menon, Rob Fergus, and Sanjiv Kumar. Embeddistill:: A geometric knowledge distillation for information retrieval. arXiv:2301.12000 2023.
Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ra- manujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, et al. Matryoshka representation learning. Proc. of NewrIPS, 35:30233-30249, 2022.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019.
Carlos Lassance. Extending english ir methods to multi-lingual ir. arXiv:2302.14723. 2023.
Carlos Lassance, Hervé Déjean, Thibault Formal, and Stéphane Clinchant. Splade-v3: New base- lines for splade. arXiv:2403.06789, 2024.
Dawn Lawrie, Sean MacAvaney, James Mayfield, Paul McNamee, Douglas W Oard, Luca Soldaini, and Eugene Yang. Overview of the trec 2023 neuclir track. 2024.
12

---

Published as a conference paper at ICLR 2026
Yibin Lei, Tao Shen, Yu Cao, and Andrew Yates. Enhancing exxicon-based text embeddings with large language models. In Proc. of ACL ( (Volume 1: Long Papers), 2025.
Chaofan Li, Minghao Qin, Shitao Xiao, Jianlyu Chen, Kun Luo, Defu Lian, Yingxia Shao, and Zheng Liu. Making text embedders few-shot learners. In Proc. of ICLR.
Jimmy Lin and Xueguang Ma. few brief notes on deepimpact, coil, and a conceptual framework for information retrieval techniques. arXiv:2106. 14807, 2021.
Shayne Longpre, Yi Lu, and Joachim Daiber. MKQA: A linguisticallyy diverse benchmark for mul- tilingual open domain question answering. Trans. of ACI., 9, 2021.
Antoine Louis, Vageesh Saxena, Gijs van Dijck, and Gerasimos Spanakis. ColBERT-XM: A modular multi-vector representation model for zero-shot multilingual information retrieval. arXiv:2402.15059, 2024.
Sean MacAvaney, Franco Maria Nardini, Raffaele Perego, Nicola Tonellotto, Nazli Goharian, and Ophir Frieder. Expansion via prediction of importance with contextualization. Proc. of SIGIR, pp. 1573-1576, 2020.
Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. Www'18 18 open challenge: financial opinion mining and question answer- ing. In In Proc. WWW. pp. 1941-1942, 2018
Suraj Nair, Eugene Yang, Dawn Lawrie, Kevin Duh, Paul McNamee, Kenton Murray, James May- field, and Douglas W Oard. Transfer learning approaches for building cross-language dense retrieval models. In Proc. of ECIR, pp. 382-396. Springer, 2022a.
Suraj Nair, Eugene Yang, Dawn Lawrie, James Mayfield, and Douglas W Oard. Learning a sparse representation model for neural clir. In DESIRES, pp. 53-64, 2022b.
Suraj Nair, Eugene Yang, Dawn Lawrie, James Mayfield, and Douglas W Oard. Blade: combining vocabulary pruning and intermediate pretraining for scaleable neural clir. In Proc. of SIGIR, pp- 1219-1229, 2023.
Franco Maria Nardini, Thong Nguyen, Cosimo Rulli, Rossano Venturini, and Andrew Yates. Effec- tive inference-free retrieval for learned sparse representations. In Proc. of SIGIR, pp. 2936-2940, 2025.
Thong Nguyen, Sean MacAvaney, and Andrew Yates. A unified framework for learned sparse retrieval. In In Proc. of ECIR, pp. 101-116. Springer, 2023.
Thong Nguyen, Mariya Hendriksen, Andrew Yates, and Maarten de Rijke. Multimodal learned sparse retrieval with probabilistic expansion control. In In Proc. of ECIR, pp. 448-464. Springer, 2024.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. Ms marco: A human-generated machine reading comprehension dataset. arXiv:1611.09266 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv: 1807 003748377.... 2018
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In Proc. of EMNLP, pp. 2383 2392, 2016.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proc. of EMNLP-IJCNLP, pp. 3982-3992, 2019.
Nils Reimers and Iryna Gurevych. Making monolingual sentence embeddings multilingual using knowledge distillation. Proc. of EMNLP (EMNLP), pp. 4512-4525, 2020.
Parker Riley, Isaac Caswell, Markus Freitag, and David Grangier. Translationese as a language in "multilingual" NMT. In Proc. of ACI,, 2020.
13

---

Published as a conference paper at ICLR 2026
Lakshay Sharma, Laura Graesser, Nikita Nangia, and Utku Evci. Natural language understanding with the quora question pairs dataset. arXiv:1907.01041, 2019.
Xinjie Shen, Zhichao Geng, and Yang Yang. Exploring 10 sparsification for inference-free sparse retrievers. In Proc. of SIGIR, pp. 2572-2576, 2025.
Nandan Thakur, Nils Reimers, Andreas Rúcklé Abhishek Srivastava, and Iryna Gurevych. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Proc. of NeurlPS, 2021.
James Thome, Andreas Vlachos, Christos Christodoulopoulls, and Arpit Mittal. FEVER: large- scale dataset for fact extraction and VERification. Proc. of NAACL-HLT (Volume I: Long Papers), 2018
Henning Wachsmuth, Shahbaz Syed, and Benno Stein. Retrieval of the best counterargument with- out prior topic knowledge. In Proc. of ACL (CC(Voll (Volume I: Long Papers), pp. 241-251, 2018.
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improv- ing text embeddings with large language models. arXiv:2401.00368, 2023.
Liang Wang. Nan Yang, Xiaolong Huang, Linjun Yang. Rangan Majumder, and Furu Wei. Multi- lingual e5 text embeddings: technical report. axXiv:2402.05672 2024.
Orion Weller, Michael Boratko, Iftekhar Naim, and Jinhyuk Lee. On the theoretical limitations of embedding-basee retrieval. arXiv:2508.21033 2025.
Tiansheng Wen, Yifei Wang, Zequn Zeng. Zhong Peng, Yudi Su, Xinyang Liu, Bo Chen, Hong- wei Liu, Stefanie Jegelka, and Chenyu You. Beyond matryoshka: Revisiting sparse coding for adaptive representation. arXiv:2503.01776, 2025.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: State-of-the-att natural language processing. arXiv:1910.03771. 2019.
Xiaohui Xic, Qian Dong, Bingning Wang, Feiyang Lv, Ting Yao, Weinan Gan, Zhijing Wu, Xiang- sheng Li, Haitao Li, Yiqun Liu, et al. T2ranking: A large-scale chinese benchmark for passage ranking. In Proc. of SIGIR, pp. 2681-2690. 2023.
Eugene Yang, Suraj Nair, Ramraj Chandradevan, Rebecca iglesias-Flores, and Douglas W Oard. C3: Continued pretraining with contrastive weak supervision for cross language ad-hoe retrieval. In Proc. of SIGIR, pp. 2507-2512, 12, 2022.
Eugene Yang. Dawn Lawrie, and James Mayfield. Distillation for multilingual information retrieval. In Proc. of SIGIR, pp. 2368-2373, 2024a.
Eugene Yang, Dawn Lawrie, James Mayfield, Douglas W. Oard, and Scott Miller. Translate-distill Learning cross-language dense retrieval by translation and distillation. In Proc. of ECIR, 2024b.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: dataset for diverse, explainable multi-hop question answering. In Proc. of EMNLP. pp. 2369-2380, 2018.
Hamed Zamani, Mostafa Dehghani, W Bruce Croft, Erik Learned-Miller, and Jaap Kamps. From neural re-ranking to neural ranking: Learning sparse representation for inverted indexing. In Proc. of CIKM, pp. 497-506, 2018.
Crystina Zhang, Jing Lu, Vinh Q Tran, Tal Schuster, Donald Metzler, and Jimmy Lin. Tomato, tomahto, tomate: Do multilingual language models understand based subword-level semantic concepts? In Proc. of NAACL (Findings), a.
Xinyu Zhang, Nandan Thakur, Odumayo Ogundepo, Ehsan Kamalloo, David aoguang Li, Qun Liu, Mehdi Rezagholizadeh, and Jimmy Lin. MIRACL: multilingual retrieval dataset covering 18 diverse languages. Trans. Assoc. Comput. Linguist., b.
Xi-
14

---

Published as a conference paper at ICLR 2026
Xinyu Zhang, Xueguang Ma, Peng Shi, and Jimmy Lin. Mr. tydi: A multi-lingual benchmark for dense retrieval. arXiv:2108.08787, 2021.
Xinyu Zhang, Kelechi Ogueji, Xueguang Ma, and Jimmy Lin. Toward Best Practices for Training Multilingual Dense Retrieval Models. ACM Trans. Inf. Syst., 42(2):1-33, 2024.
Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, et al. Qwen3 embedding: Advancing text embedding and reranking through foundation models. 211:22006.05776, 2025.
15

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 DEMONSTRATION EXAMPLES
In Figure we present list of demonstration examples. The imputs are in different languages, while the outputs are bag-of-words words English tokens produced by our ! csac SCTxD- Lestchh)) model. These examples illustrate that MILCCO generates transparent representations, making possible for humans to interpret, inspect, and trace potential errors or biases.
10
Language
Input Text
Sparse Representation (English View)
[hosle": "worthh: 2.72, "nar" 1.7, "prioe" 1..5. "salory" 0.8. "nkkola" 0.78, ".7p'oo""" "stock": 0.58, "electric" 0.52, "eart: 0.40, "paneration" 0.49 "moonyy: 0.48, "lov?: 0.42. "mining" 0.31, "compery" 0.27, "safe" 0.2, "milon: 0.12, "edmund": 0.12. "reverue"" 0.06)
1
en
what tasla not worth?
(Tssia" 3.35, "orth:: 2.79. "naf: 1.73, "price": "salary": 1.08, "monay": "generation 0.51,
टеs 6 gea airiit wमा t??
"nikola": 0.49, "elecric" 0.4, "car":0.11 0.31, "campary" 0299 "minnng" 0.29, "lvi" 0.25, 0.13
0.17.
16, "sue"" 0:11,
(Teeaa 3.35, Worrfh: 2.71, "price": "ret": 1.78, "asiery" 1.35, "monuy": 0 0,85,
zh
10 20 #
wbook": 0.79. "company"" 0.56, Welectic:: 1..00.5. 0.54 "nkdd"" (0.d 0.54, "gereration" 0.49.
"oar": 0.34, "mining" 0.32. 0.19 "motoes" 0.15, "otall 0.03)
0.19
Tosia" 3.34, Worth: 2.60, "nef": 1,34, "price:: 1.51, "stock" 0.78, "salary" 0.88,
1
da
Was ist Tesiss Natooert??
0.25, "avs" 0.22. 'campayn" 0.19, "miring" 0.16, "mian" 0.11, "weathh" 0.11,
"maney" 0.62, 'electric" 0.48, "nikaa" 0.46. "generatior"" 0.43 "sale" 0.34, "car"
"odmund": 0.11, "ournntly" 0.5, "tavanue" 0.13
(Teela *wort": 2.75
1,80, "prica" 1666, "swarry" 0.68, Waock" 0.87,
is de netloaarde van Tosia?
"sale" 0.32, "lesf: 0.27 "mining" 0.23, "oar": 0.22. "revenue" 0.15, "admund" 0.12, "ourreily" 0.11, "weelth": 0000. "imesmmen" 0.08)
"money" 0.81, "electric
"nikela" 0.48, "camosny" 0.41, "ganeration" 0.41,
(hosia" 3.35, "worh": 2.84, "naft: 1,81, "price
"salary" 0.85, Sttck"
12u العاي الللا
"nikalat 0.28
"money":0.55 "campany" 0.53, "el.cleec" 0.48, "av""
26,
0.42 0.22. "sale" 0.18, 0.16 notar:
"invesmmet"
"oumently" 0:03,
0.03)
(Tesle 3.36, "wrth: 251, "price": 1.84, "tat" 1.52, "salary" 1.14, 'money": 0.8
GlA tài sln rónng cla Tadia là bao nhill?
"stock": 0.7. "electrcc 0.62, "nikolaa: 0.45, "mining:: 0.35 "ganeration 0.33, "sale:: 0.27, "corgary"" 0.25, "mian" 0.21, "lor" 0.18, "woll"": 0.18. "rvestment" 0.14, "lev": 0.09 "rvveue"" 0.08. "otoll 0.05)
["aahmmre" 2.47, "marylane" 2.07, "greatost" 1.68, "oty": 1.59, "biggest: 1.38,
2
an
Battmore Maryland the groatest city in America
"emeriaa"" 0.47, "us": 0.45, "urban" 0.41, "birthpknns" 0.41, "was" 0.34, "amous":
"md": 1.88 "emerica: 1.11. "uee": 1.00. "greet" 0.98, "clies" "lowr" 0.7.
0.29. "garrison" 0.25, bbauttfff: 0.23: 0.23, "gingrapmy" 0.21)
"ballimoe" 2.65, "maryland" 2 28.8 "argesi": 1.7. 7.yt"y": 11.5 1.55. "biggeut: 1.5, "me":
wndhhtt #hhils graftet er ve (रार
"usa": 1,14, "clies" 1006. 118h 0.78, "Iown" 0.65, "wmarica"
"popusation= 0.64, "arhann: 0.48, "merrpollsss""
"cansus" 0.31, "garrson": 0.3, "big" 0.19)
gengraphy" 0.29, "sia" 0.26, "hoadqurrtrss"
biggesf: "usa": 1.58. 2.07, "graateat" "oly" "mct: 1.65,
(halimmm" 66,
1.29, "diles": 1.25, 1.02. "america" 0.91,
0.81, "bast" "rrhan": "gaograohy" "locatio" 0.48)
"bithplaca"": 0.57.
"beauifu": "garison" 0.51, "reuqqusten":
("balimore" 2.2. "maryland"; 1.56, "cay" 1.4. "greateff: 1.25, "md" 1.16,
2
de
Battmore Maryland de großarägste Stadt Amerika
"besuifuf:005 "moel" 0.52, "mmerican" 0.49, 'ua" 0.48, "urben" 0.43, 'bes": 0.37. "garrion"" 0.35. "gengraphy" 0.31, "arrhpaaw"" 0.20)
"bigposf: 0.98, "great" 0.96, "cllas": 0.56, "usa", 0.34, "amarica" 0.91, "tonn" 0.68,
["bahmmor" 2.35, "marylano" 2.0, "bost" 18,4 "oty: 1.51, "md": 1,27, "usa": 1.17,
2
ml
Battmore Maryland de beste stad in Amorika
"emerica" 1.11, "besutia" 1.05. "chies" 1 1,02 "tiggest" 0.80, "lown":077
"emerican" 0.61, "ux" 0.54 "urban" 0.49, "birtholace" 0.57. "garrison" 0.35,
"poputar::33. 0.3. "ooriam" 0.29 "location" 0.28, "Teadquetens" 0.27)
["batimore: 2.57, "maryland" 2.08, "oay:: 1.63, "greatesf: 1.55, "md": 1.53,
"biggesth 1.38, "great" 1,24, "usa": 121. "ollwe" 1.15, "america" 1.14, "tonn" 0.72.
"ganaooe 0.66, 0.61 "toaalon" 57 "urban" 8.57, "haadqurrers" 0.53,
american" 0.52, "bas": 0.47, "besutitu" 0.43, "sirhpiaca"" 0.42)
hhllimmm" 2545, "maryland" "ollee" 0.95, "argass" "us": "smarica" 0.80.
"cll". 1.47, biogest" 1.38, 'md": 1.28,
Batsmore Maryland thanh phó nhát My
"reatess" "uaa".
"geat" 0.47, Turban:
"waadqurters" 0.34, "matroallann 0.3,
"parisso"" 29, "populatio" 0,27, "melmpolis' 0.22. "ameriann" 0.21)
(Tmport" 1.6, "music" 1 146, "ave": 1. 1.17. "phone" 1.16, "mmprreed: 0.95, "songs":
3
2th
94 1 2.
0.86, "sop": 0.84, "stag": 0.71, "ountbb"" 0.68, "phonas": 0666 "atream" 0.63, "no":
0.62. "bulton" 0.62, Vlonkoed" 0.58, "song". 0.48, "pandora' 0.48, "mpp3" 0.47.
"payer" 0.46, "syne' 0.48, "Irnnsler" 0.45)
Alan Smitho is a psoudonym for a fictional director responsible for timms which the actual diractor coes
I'alan": 3.02. "etee": 3.0. "smit:: 2.95, "psoudonym" 2:15, "ctrector". 2.05,
"droctora" 1.93, "Idiionl"" 1.9, "Haga" 1888 "guid"" 1,85, "responltie" 1.61, to":
de
not want ha name escocieled with the work From 1968 to 2000, was recommendad by the Directons Guild of America (DGA) for such ataatoos..
1.58, "who": 1.40, "Min" 1 1.34 "alas" 1.29, "dtractoo": 1.26. "#toss"" 1.2, "actual"
1.09, "1968": 1.07, "ecommanded"" 1.05, "ctracting. 1.04)
Paul Jules Antoine Mellet, is Moutin (Aliee) al mort A Channaumelant (Char), linuistee français des pramiéres dacanniss aussi phiblogue
[mma) 2.0. "eeler" 2.0.ee 2.73, "Juds"" "poud" 2.57, "antninw" 2.46 "WHdi"" 2.41.
prinipal
Wllee: "inguior" 2.21. "fwnnc"" "char": 2.04, "chaloai" 1.84 "franca": 1.77. "HMmaT: 1.72, "mo"" "wh"" 1.69, "Iinguisics": 1.68, "delan": 1.6, "born
est
1.51. "cled": 1.39. "Wiet". 1.34)
R &. a. & 200 @ th 1199
2289E 120 to)) de 1947 244119
('dynasty 2. 18, "emperor" 2.03, "uing:: "ming": 1.98, "sima: 1.0, 'imm:: 1.69,
8
zh
T 1h8 M. CO
soot": 1.67. "morgelan": 1.84, 18", 1.61, "charr 1.34, "horea": 1.45, "age": 1.44. "etchang" 1.46, "dyresties" 1.42, "date": 1,35. "whot: 1.30, "diec": 1,32, "china":
C F 77 ta E SET Whth 99 2
1.31, "raign:: 1.31, 'ompire":
Figure 6: Examples of MILCO's output representations (English view) on different languages.
16

---

Published as a conference paper at ICLR 2026
A.2 DETAILED MULTILINGUAL/CROSS--II RETRIEVAL RESULTS
In this section, we show the detailed language-specifie results of MILCO and baselines the fol- lowing multilingual and cross-lingual retrieval datasets. The result on Multilingual Long Document Retrieval (MLDR) is shown in Table The result on MTEBv2 (multilingual and cross- longull tasks) is shown in Table The result the NeuCLIR benchmark (cross-lingus retrieval) is shown in Table The result on MKQA (cross-lingual retrieval) is shown in Table
Table 6: Multilingual (long) document retrieval on the MLDR (measured by nDCG@ 10).
Moded
Max Length
Aug
de
fr
bi
ko
th
zh
bareliney
512
34.2
33.0
26-9
33.0
51.1
49.5
21.0
45.1
19.9
27.1
58.7
42.4
15.9
13.2
ES ES
$192
42.6
29.5
40.6
43.3
70.2
60.5
23.2
55.3
41.5
32.7
60.5
52.4
18.2
16.8
Mi-Dense
8192
52.5
47.5
46.1
68.9
74.8
738
40.7
62.3
30.9
42.9
74.4
99.5
33.6
26.0
ME 155-Mullissse
$192
57.6
56.5
50.4
55.8
79.5
71.2
45.6
66.8
52.8
48.5
77.5
64.2
39.4
32.7
Spunee
8192
64.8
63.0
56.4
64.2
88.7
84.2
52.3
758
58.5
53.1
$6.0
75.6
42.9
42.0
M1
64.7
57.9
83.9
52.2
75.5
55.7
$5.4
64.7
60.0
PLAID (Muli vector)
74.2
73.5
65.5
90.9
87.5
64.0
67.3
66.9
43.7
62.7
Embed 0.68
Qwenn-Fambsl--e
Sporse baselines
RM25
8192
53.6
45.1
53.6
57.0
78.0
75.7
45.7
70.9
96.2
25.7
$3.6
61.3
33.6
$4.6
MB M3-Spune
8192
62.2
58.7
53.0
62.1
87.4
82.7
49.6
74.7
53.9
47.9
85.2
72.9
40.3
40.5
ap Loddoo
Table 7: Performance of embedding models MTEBv2's multilingual and cross-lingual retrieval
tasks (Eeevllssnn et al. 2025). MILCO outperforms other models with similar sizes (e.g. Qwen3-
0.6B, BGE-M3), while under-performs larger models, such
(M Multilin-
gual, C= Cross-lingual). We evaluate English-only retrieval tasks separately in Section A.3
Model
Avg.
Belebele (M)
MIRACL-HN (M)
MLQA (C)
Statean (M)
Twitter (M)
Wiki (C)
gue-mliilinuual-base
64.72
77.60
64.17
72.19
21.74
68.92
83.69
bge-m3
62.02
78. 16
69.59
74.81
21.86
37.82
89.87
granite-27mm-muli
55.80
62.20
59.45
62.99
30.14
85.06
gramite-125m-eng
26.99
33.37
16.35
22.90
30.71
5.92
52.70
granite-10mmmmuultt
49.88
55.12
$7.25
60.47
27.50
17.06
$1.88
gte-Qwen2-7B-ins
67.22
77.54
51,58
78.69
37,87
68.64
88.97
1o-Owen2--1.B-ing
65.12
66.59
63.23
72.89
33.25
67.01
87.77
NV-Embed-v2
55.54
70.61
19.55
inf retreeeiivv
71.21
77.37
60.93
80,31
37.30
79.30
92.02
jinaemmeddding-vvd
73.84
74.29
62.95
74.90
58.07
84.38
88.46
inf ref-reeriever-v vi-
62.35
72.93
70.46
88.93
Qwen3-Emhed 0168
63.93
68.74
61.23
72.79
33.63
611014
87.13
Swen3-Embedd
75.59
70.58
83,55
40.46
78.20
91.96
ww.. scTap Lankes
72.65
Table 8: Results NeuCLIR cross-lingual benchmarks (Lawrie et al. 2024) three languages (Chinese, Persian, and Russian). The Avg. MLIR score (nDCG@2) 20) is the mean across the two years. Our 560M MILCO model outperforms Qwen3 0.6B, but falls behind PLAID-X and Qwen3- Embed 4B/8B. PLAID-X focuses exclusively on the test languages,
Model
2023 MLIR (C)
2024 MLIR (C)
Avg. MLIR (C)
SPLADE v3 (transl. does)
0.420
0.440
0.430
PLAID-X
0.404
0.468
0.436
Qwen3-Embed 0.6B
0.317
0.311
0.314
Owen3-Embed 4B
0.440
0.415
0.428
Qmenn-Embed 8B
0.434
0.419
0.427
o MILCO (SAP. SCTxp- LexFaho)
0.395
0.427
0.411
17

---

Published as a conference paper at ICLR 2026
Table 9: Cross-lingual retrieval performance MKQA (Recall @ 100). Abbreviations: mContriever, OA3 = OpenAl-3, PLD = PLAID-X, Q0.6 Qwen3-wennccmmm (.wen3-0.6B. Q8 = Qwen3-8B
mCtr
Spurse
Lang
mDPR
unC.ie
E:S-L.
E5-M7B
OA3
M3-D
M3-1IV
M3-DS
M3-AII
PL.D
Q106
Q8
BM22
M3-8
48.2
56.2
65.7
98.6
65.6
71.1
71.4
71.1
11.5
64.2
46.8
64.75
18.9
23.5
74.9
67.4
77.4
17.6
77.0
55.4
77.9
65.8
76.2
35.4
66.8
76.7
75.1
75.7
73.6
51.1
76.6
68.2
76.5
77.0
76.2
76.2
76.6
76.6
61.8
51.9
774
49.7
6322
09.0
67.2
58.1
72.4
72.9
12.5
13.0
70.5
39.0
62.58
26.9
$1.1
74.3
60.4
69.7
74,7
75
71.2
74.7
74.6
74,9
75.0
72.2
43.7
65.07
38.1
44.6
758
66.0
72.3
76.8
77.1
73.6
76.0
76.4
76.3
76.5
75.2
61.3
69
45.2
52.5
77.5
60.3
64.8
71,5
65.1
7L.9
75,0
75.1
75.0
75.2
75.2
57.2
09,45
24.5
31.3
77.2
20.5
26
33.4
34.3
33.9
63.6
69.1
68.8
69.2
63.7
24.6
$2.76
20.6
30.1
7044
ko
50,9
59.7
99
73.3
71.
71.6
31.8
70.7
65.51
27.9
31.4
73.9
59
62.4
77,1
77.4
77.4
57.9
gl
67.0
71
73.3
76.3
76.3
76.4
48.0
50.5
769
pt
65.5
72.6
73,5
77.5
73.7
96.3
76.4
76.5
76.4
74.4
61.0
46.9
50.9
274
tU
62.7
70.8
73.5
72.0
76.2
76,4
76.2
76.5
76,2
58.9
09.03
33.2
36,9
77.4
66.9
73.2
77.6
78.3
74.0
76.9
77.2
77.
77.4
76.5
55.8
69.71
$4.6
50.6
76.2
53.8
66.9
76.0
63.4
55.2
75.6
75.9
76.0
76.6
76.2
56.3
69.72
31.8
45.0
77.9
39.1
71.1
74,3
74.9
75.2
73.6
75.9
76.0
76.0
72.0
52.1
66:57
45.8
51.
77.6
61.4
70.9
75.4
77.0
71.1
76.5
76.7
76.8
76.9
74.3
57.6
69.08
46.6
51.8
77.9
zh
63.7
70.7
74.6
14.7
75.0
31.0
35.4
39.8
Ave
60.5
67.9
70.9
70.1
89.5
75.1
75.3
75.3
75.5
73.4
54.4
67.9
39.9
45.3
76.6
Table 10: Performance comparison on BEIR English retrieval benchmark.
Larp Madeh 1o poran))
Seal Modeic pareent
Deledt
Nee
te NITLIN
AcaiMaa
3-0
83.
62.7
484
62-
48.
4500
46.
423
41.9
411
339
18.
36.
$350
08.3
992
58.4
336
32.
713
11.
315
13.
31.0
746
36.8
701
Tra
3530
303
359
7511
35.4
$2.3
25.3
731
005
73-1
7122
24.
332
28.
10
474
42.4
47.4
42.3
200
25.8
445
41.
2611
26.2
40.
257
38.8
433
11.2
233
308
PENER
3.0
11.
103
5L5
909
143
12.
342
915
16.3
948
12
632
192
5511
75.
134
MEMARCO
3.5M
455
465
423
41.
810
43.
438
312
436
436
433
16.
41.
80
Nigilaag
66.
6.
68.4
50.
53
654
600
51.
565
164
16
545
55.8
933
51.
A.3 ENGLISH RETRIEVAL RESULTS (BEIR)
In Table 10 report the performance of MILCO and baselines various retrieval tasks evaluated on BEIR English benchmark (Thakur et al. 2021).
On average across BEIR benchmarks, MILCO attains 54.4, slightly behind the dense competitor Qwen3-0.6B (56.2: - 1.88.. This gap is expected, as Qwen3-0.6B benefits from instruction tuning, which the Qwen3 paper (Zhang tions. Compared to other English-only sparse baselines, MILCO is clearly stronger than Splade -3 (+3.3) and marginally ahead of opensearch-gte (+0.6), while still trailing LENS-d4K (-7.3), a much larger sparse model. It is worth noting, however, that the comparison to Splade V3 is not entirely fair: Splade only trained on MSMARCO, while MILCO (and most other baselines) trained on much larger data, including BEIR's in-domain training sets, which naturally favors transfer to the BEIR evaluation benchmark.
2025) reports adds +1-5%, while MILCO does not use instruc-
When focusing on the more challenging large-collection datasets (2 1M documents), MILCO shows its main strength. achieves an average 57.3, surpassing Qwen3-0.6B (54.5; +2.8), Splade V3 (53.3; +4.0), and opensearch-gte (55,8; +1.5). MILCO's improvements are particularly pronounced on datasets such as HotpotQA (77.7 vs. 65.7; +12.0) and NQ (64.9 vs. 53.5; +11.4). Although still falls behind because such scenarios are the most relevant to real-world search applications, where corpora often contain millions of documents.
(65.4; -,,,), the strong performance large collections is noteworthy
18

---

Published as a conference paper at ICLR 2026
Table 11: Performance of MILCO compared to dense baselines the LIMIT benchmark.
Model
Dim
2
Recall 10
@ecall 100
BM25
default default
85.7
90.4
93.6
23.1
34.6
54.8
ES-Mistral 7B
32
0.5
E5-Mistral 7B
64
0.1
0.4
ES-Mistral 7B
128
0.1
0.3
1.0
E5 E5-Mistral 7B
256
0.4
0,9
1.9
E5-Mistral 7B
512
0.7
3.8
E5-Mistral 7B
768
0.9
1.7
4,3
E5-Mistral 7B
1024
0.9
1.8
5.9
E5-Mistral 7B
2048
1.0
1.9
6.8
E5-Mistral 7B
3072
1.3
2.0
7.7
E5-Mistral 7B
4096
8.3
MritLM 7B
32
0.8
GriLLM 7B
64
0.1
0.3
GritLM 7B
128
0.1
0.3
1.3
GritLM 7B
256
0.1
0.4
2.8
GritLM 7B
512
0.6
1.8
6.5
GritLM 7B
768
1.5
3.1
8.7
GritLM 7B
1024
1.8
3.5
10.6
GritLM 7B
2048
2.3
4.3
11.8
GritLM 7B
3072
2.0
12,9
GritLM 7B
4096
2.4
4.1
12.9
Qwen 3mbbed
32
0.1
1.1
Owee3-FFmhed
64
0.2
1.0
Qwen3-
128
0.3
0.4
1.8
Qwen3-Fmber
256
3.2
Qwen3-Emiee embbed
512
0.6
1.3
3.3
@wen3-Embee
768
0.7
1.5
3,8
Owen3-Ember
1024
0.7
1.6
4,6
D0we3-EEbbr
2048
0.9
4.7
Qwen
3072
0.8
1.6
4,8
Twen3-Embbe
4096
0.8
1.8
4,8
Gemini-Embed
2
0.1
Gemini-Embed
0.0
Gemini-Embed
0.0
Gemini-Emmed
16
0.0
Gemini Embed
32
0
0.0
Gemini-Embed
64
0.3
Gemini-Embed
0.3
Gemini-Embed
256
0.1
1.2
Cemini-EEmber
512
0.2
1.1
3.6
Gemini- Embed
768
0.9
2.5
7.6
Gemini-Ember
1024
1.3
2.7
8.1
Comini-Embee
8.5
Gemini-Ember
3072
1.6
3.5
10.0
L MILCO ISAP. SCTgD LexEcho)
280,524
47.0
73.5
A.4 EMBEDDING LIMIT TEST
Table II shows the advantage of MILCO over strong state-of-the-art dense baselines on the LIMIT test (Weleer et al. 2025). MILCO achieves an R@ 100 of 73.5, whereas dense models such as Gemini-Embed and Qwen3-Embed nearly collapse to zero.
A.5 CORRELATION BETWEEN TEXT LENGTH AND VECTOR SPARSITY
In Figure 7 show strong correlation between input length and the sparsity of vectors produced by MILCO. Unlike dense retrieval methods, which always generate fixed-length vectors for all queries and documents, LSR methods, including MII.c,, adaptively determine the optimal sparsity
19

---

Published as a conference paper at ICLR 2026
Vector sparsity (number of non-zeros)
800
600
400
200
2000
Text length (# of characters)
Figure 7: MILCO: Correlation between input text length and the sparsity of output vectors.
Table 12: MILCO: Effectiveness at different TopK (tokens). (nDCG@ 10, MIRACL)
Toplk (tolens))
Avg
a5
bn
de
C0
C5
1a
ja
10
04
5M
1e
1h
30
zh
0/
03.55
53.11
87005
300
62.75
83.00
72.6
86
80.02
87.92
66.85
72.7
62.14
87.95
64.41
1000
71.7
M0.11
13
62.83
67.09
62.17
63.04
80.6
62.74
65.12
60.99
77.78
71.94
14.89
80.36
87.95
64.48
83.
66.04
Table 13: MILCO: Effectiveness at different pruning percentile P. (nDCG@ 10, MIRACL)
hi
39
512.5
12.6
80.8
K3.0
62.7
62.1
62.0
50.6
63.8
65.1
69.9
77.3
22.0
74.9
80.2
67.8
84.4
83.0
66.8
20 455.8
72.6
80.7
81.8
62.8
62.0
61.
63.2
30.6
63.9
643
61.0
77.3
72.0
74.0
80.2
67.6
84.5
81.7
66.8
50 285.6
72.3
80.7
81.8
61.7
61.1
61.7
63.1
$0.5
62.6
64.3
60.8
76.9
71.9
74.6
80.
0..5
84.2
90.4
66.4
70 172.2
71.7
80.3
$1.5
60.9
60.1
69.5
62.5
79.5
61.3
64.2
60.1
76.6
70.5
74.2
79.4
57.5
83.8
90.7
65.2
80
115.1
10.6
79.6
M0.7
609
59.0
59.6
61.3
783
608
623
58.
754
68.
73.0
78. 3
86.9
85.7
75.2
64.0
85
86.4
69.5
76.6
70.5
59.0
59.2
58.5
60.1
78.1
96.
61.7
S8.
16.2
67.5
71.3
77.2
96.4
82.8
77.7
63.0
90 $3.9
67.4
76.6
77.8
5788
53.0
56
$6.1
76.7
$7.4
586
96.4
72.1
65.7
68.0
76 8
64.5
81.4
75.0
60.7
95
20-2
62.2
71.8
71.8
52.5
48.7
31.2
48.9
718
$3.6
52.6
$3.5
65.5
63.3
63.3
71.3
78.1
76.2
70.5
55.7
97
17.7
56.0
66.2
53.9
46.2
43.4
46.
44.2
66.0
49.4
450
45.5
57.4
36.
56.1
65.0
67.4
68.6
67.3
50.0
39.3
48.38
48.14
32.18
29.04
3531
2835
50.3
31.56
28.06
31.17
39.98
42.21
31.82
43004
43.00
44.59
57.06
34.13
(i.e.. the number of non-zero elements) based on the content density of the input text, as approximi- ated by its length. This allows MILCO to allocate fewer tokens for short texts and more for longer ones, while maintaining the same average sparsity overall.
we show the results of two different post-hoc pruning methods (Top-k Pruning and Mass-based Pruning) applied on MILCO.
On Table 12 and Table 13
A.6 ALIGNMENT AND CONTRASTIVE TRAINING DATA
The sources and statisties of the data used for our Sparse Alignment Pretraining are reported in Table [4] In total, the corpus consists of 594 million bi-text pairs, each containing one English sentence and one non-English sentence with the same semantic meaning.
The statistics of the data used for our contrastive training are shown in Table 15 This dataset contains 1.4M queries collected from 16 datasets, covering English, Chinese, and 16 additional languages from Mr.TYII (Zhang et al. 2021) and MIRACL (Zhang al., [b). Similar prior work (Chen et al. 2024 Li et al. Lei et al. 2025). our training data also includes many in-domain datasets from BER Thakur et al. 2021).
20

---

Published as a conference paper at ICLR 2026
Table 14: Pretraining datasets: Parallel Sentences collected from OPUS by Reimers & Gurevych
2019).
Dataset Name
#Pairs
mmarco (passages)
115M
wikititles
14M
wikimatrix
19M
europarl
50M
opensubtitles
274M
talks
20M
tatoeba
8M
jw300
92M
news-commentary
2M
Total
594M
Table 15: Contrastive Training Data obtained from
Chen et al. (0244)
Dataset
#Samples
en.msmarco Nguyen et 2016
485,823
en.eli5 (Fan et al. 2019
150,000
zh. zh.mmarco zh (Bonitacio et al.
2021
100,000
zh a2ranknn (ie et 2023
90,467
en squqqud (Rajpurkar et al. 20166
87,599
hotpotqa (Yagg et 2018
84,516
zh_dureader (He et al 2017
80,416
en.riivia (Joshı al. (017))
60.315
en quora (Sharma et al. 2019
60,202
en nq (Kwiatkowkki al. 2019
58,568
multilingual imrtydi (Zhang et al.
2021
48,729
aettllingualmmiraa (Zhang et al.
40,203
en.fever (Thorne et al. 2018)
29,096
en_fiqa (Mara al. 2018)
5,500
en-arguana Wachsmmuhh et al.
2018
4,065
en scidocs (Cohan et al.
2020
884
Total
1,386,383
A.7 CONTRASTIVE TRAINING: DISTILLATION
For each query qs. we consider a document candidate set consisting of one positive dt and a set of negatives D The precomputed teacher scores are from the cross-encoder, denoted as fc q). The student scores funicoo (d, q) are estimated via the dot product of MILCO's lexical representations. These scores converted into distributions the candidate set with softmax:
exp (d..
(d(()) Exe exp (0, (d)
where (g. d' denotes either the teacher or student scoring function of the given query and docu- ment. The distillation objective is then defined as the KL divergence between the teacher's 's and the student's distributions across a batch of B queries:
B
LRLD =
KL (XLLe (Cee))
(10)
B
21

---

Published as a conference paper at ICLR 2026
A.8 TRAINING CONFIGURATIONS AND HYPER-PAREE HYPER-PARAMETERS PARAMETERS
The hyperparameters for pretraining and training are reported in Table] Table 17 Both training Instinct stages are conducted on 16 GPU nodes, each equipped with AMD Instinct MI250X GPU dies. We instantiate the Multilingual Connector with simple randomly -nitiiiiiee MLP layer with GELU activation function. For sparse regularization, we set the regularization weight to 1e-5 for both queries and documents during contrastive training. We train MILCO using the HuggingFace framework (Wolf' et al.) (019). Hyperparameters not listed in Table 16 and Table 17 are set to the default values defined HuggingFace's 's TrainingArguments.
Table 16: MILCO: Hyperparameters for Sparse Alignment Pre-training tr-irrniiin..
Hyperparameter
Value
training.lyp lype
aligmment
mypel.type
bert
Isr www.eennerr....ee w.l.enodder.checkoiiee
checkpoint
naver/splade-v3
_ewplleeeeeeeeeeeeeeeeeeeeeeeeee _ train_datasets
multilingual
14.A/hheemm3-uuspprriicc
mmarco, wikititles, wikimatrix, europarl, opensubtitles, talks, tatocba.jwv jw300, news-commentary
seed
42
max_length
256
per_device_trait htrec_trraaaathhaeeeeeaeee batch_size
per_device_eva htt.ii/eeelabaachhseeeeee batch-size
128
num train epochs
2
save total limit
2
warmup_steps st.scheduler.Iyp
10000
cosine
dataloader pwm_ workers w
8
learning_rate
2e-5
bfl6
True
logging_steps
500
save_steps
20000
pooling
max
False
Synamic_length
True
Table 17: MILCO: Hyperparameters for Sparse Contrastive Training.
Hyperparameter
Value
trainingtyye
distillation
model_type
bert
navcr/splade-v3
multilingual encoderreeeeeeeeeereeeeeee
EAAA/fgeemm--unsuprrvisd
train.group.sifg group_size
8
lambda_q
1c-3
lambda.d
le-5
train datasets
bge
seed
42
max max_Jength
512
per.device_trait waaiceeettraiiieeeeeeeeeeeeeeeee wtt.:ttaat..niiiiiitee
8
32
train epochs
8
save total limit
2
warmup ratio
0.03
Ir scheduler type
cosine
dataloader workers
learning_rate
2e-5
bf16 logging-steps
True
500
22

---

Published as a conference paper at ICLR 2026
A.9
LIST OF LANGUAGES SUPPORTED BY MILCO
Table 18: Datasets and their supported languages.
Dataset
Languages (standardized)
#laguagees
MIRACL
ar. bn, de, en, es. fa, fi, fr. hi, id, ja, ko, ru, sw, te, th, zh
18
MLDR
de, en, es, fr, hi, it, ja, ko, pt, ru, th, zh
13
MKQA
da, de, fi, fr, he, hu, it, ja, km, ko, nl, no, pl, pt,
255
ru, sv, th, ir, vi, zh-en, zh-hk, zh-tw
BelebelcRetrieval
aem, af,
3
MI MI.QARetrieval
ar, de, en, es, hi, vi
TwitterkjermRRetriivaal
dan
WikipcdaReeiiecaMulllngnggu
bg. bn, cs, da, de, en, fa, hi, it, nl, no, pt, ro, St, sv
16
WikiMatrix
ar, bg, ca, cs, da, de, cl, es, et, fa, fi, fr. gl, he, hi, hr, hu, hy, id, it, ja, ka, ko, It, lv, mk, ms, nl, pl, pt, ro, ru, sk, sl, sq, Sr, sv, th, tr, uk, ur, vi, zh-on
41
purallelseniennesspeennbbbbbll
ar, bg, ca, cs, da, de, el, es, et. fa, fi, fr. gl. he, hi, hr, hu. id. it. ja, ka, ko, It, mk, mr, nl. pl, pl. ro, ru, sk, sl, sq, sr, sv, tr, uk, vi, zh
38
parallel-senttenees-tttoobb
ar, bg, ca, cs, da, de, el, es, et, fa, fi, fr, gl. gu, he, hi, hr, hu, hy. id, it, ja, ka, ko, ku, It, Iv, mk, mn,
46
my, nb,
nl, pl, pt, ro, sk, sl, sq. ST, th, uk, ur, vi, zh
puralle-senteneesglloaavvvveesss
bg, ca, da, de, el, es, fa, fr, he, hi, hu, id, it, ko, mk. my, al, pl. pt. ro, sq, st,
27
paralle--smmiencccsseurpnaat
bg, es, da, de, cl, es, et, hu, sl,
nl, pl, pt, ro, sk,
20
surillll-settenneestalllss
ar. bg, ca, cs, da, de, el. es, et, fa, fi, fr. fr-ca, gl. gu, hi, hr, hu, by, id, it, ja, ka, ko, ku, mk, mn, mr, ms. my, nb, nl, pl. pl, pt-br, ro, ru, sk, sl, sq, sr, sv, th, ir, uk, ur, vi, zh-en, zh-tw
50
parallee--entenneesjw.30000
ar, bg, cs, da, de, el, es, et, fa, fi, fr, gu, he, hi, hr, hu, hy, id, it, ja. ka, ko, It, Iv, mk. mn, mr, my, nl, pl. pt, ro, ru, sk, sl, sr, sv, th, tr, uk, ur, vi
43
pualleeseeeeeessewwwwccmmamaaaar
ar, cs, de, es, fr. it, ja, nl, pt, ru
10
mmarco
ar. zh. nl. en, fr. de, hi. id, it, ja. pt. es. vi
14
All Test Datasets
acm, af, ar, bg, bn, es, da. de, id, km, ko, ms, nl, pl, pt, ro, ST, tc, th, tr, vi, yo, zh, zh-en, zh-hk, zh-tw
fla fi, fr,, he, hi, hu,
39
All (Pretrain/Train Test) datasets
aem, af, an, bg, bn, ca, da, de, el, en, es, fa, fi, fr. fr-ca, gl. gu, he, hi, hr. hu, hy. id. it, ja, ka, km, ko, ku, It, Iv, mi, mn, mr, ms, my, nl, no, pl, pt, pt-br, ro, ru, sk, sl, sq. sr, sv, sw, te, tr. uk, ur, vi, yo, zh, zh-en, zh-hk, zh-tw
63
A.10 ABLATIONS ON LEXECHO HEAD
To provide additional insights on our LexEcho head, we perform an ablation that interpolates be- tween the pivot (English) and source views in LexEcho. Specifically, multiply the English view by a and the source view by (1 - a),, with a E {0.0, 0.2...................... 1.0]. and report MIRACL results in Table 19
23

---

Published as a conference paper at ICLR 2026
Table 19: MILCOO performance under different pivot-source weighting schemes the MIRACI. (hard negatives) benchmark.
ai
In
411
es
fn
6
fr
14
ko
FB
SW
te
th
2h
de
y0
Arg
0.00
5.21
9.51
3.56
0.99
1.74
8.18
0.14
1.56
4.1
3.35
11:14
2.19
2.75
31. 35
14,90
0.00
0.21
0.27
5.61
0.20
11.01
17.51
6.16
0.91
4.25
17.80
1.51
3.95
9.55
7.16
19.09
5.35
6.95
43.42
22.47
0.11
1.34
4.49
10.19
0.40
79.38
79.30
56.55
55.59
57.29
79.72
60.70
57.46
58.46
15.36
71.99
73.10
77.52
96.77
82.64
56.95
59.67
63.23
68.43
0.50
50.79
82.00
62.08
02.06
63.08
90.64
62.73
65.06
60.91
77.25
72,00
74.96
80.18
57.82
54.46
06.30
62.83
82.14
72.06
0.60
80.08
81.43
62.03
62.00
62. 34
79.91
63.14
94.94
00.14
76.64
71.13
74. 34.30
79.61
57.11
83.81
65.54
61.73
81.72
72.09
0.80
78.19
80.23
60. 18
61.27
60.36
78.81
62.16
64.00
58.99
74.66
69.12
72.95
78.49
84.05
81.47
63.63
60.58
80.86
70.52
1.00
73.77
79.35
59.53
60.98
99.75
78.43
62.19
63.31
57.97
73.80
68.52
72.74
78.04
90
30.72
63.33
60.55
80.46
70,02
We find that MILCO performs very poorly with only the source view (a 0.0: average nDCG@10 = 5.61), indicating that the English pivot is essential. With only the English view (a = 1.0), MILCO is already strong (average nDCG@10 70.02) but not optimal. The best result obtained with roughly balanced fusion (a = 0.5). reaching an average nDCG@10 of 72.66 (around 3-4% relative improvement over 1.0). Performance for a between 0.5 and 0.6 is very similar, suggesting that LexEcho is not overly sensitive to the exact weighting as long as both views contribute.
A.11 EFFECT LANGUAGE COVERAGE IN ALIGNMENT PRETRAINING
We now investigate how language coverage in the parallel corpus used for MILCO's sparse align- ment pretraining affects the final retrieval performance across MIRACL languages. To this end, we compare MILCO (alignment distillation) against our best-performing dense bascline (distillation only), trained with the same data and compute, and report per-language AnDCG@10. We then aggregate languages by their share of alignment data analyze the relationship between coverage and effectiveness (Tables 20 and
Table 20: Average AnDCG@10 as a function alignment coverage. Languages are grouped into well-represented (P 1%) and under-represented (P < 1%) buckets based on their proportion in the parallel corpus.
Percentage category: P2 1
P<]
Avg. AnDCG@10
1.59
0.77
Table 21 lists nDCG@10 and AnDCG@10 for all 18 MIRACL languages, together with their pro- portion in the parallel corpus. Aggregating by coverage (Table 20), we observe an average gain of +1.59 AnDCG@11 for well-represented languages (P > 1%) and smaller but still positive aver- age gain +0.77 for undrr--epeesented languages (P < 1%), including languages with 0 parallel samples (sw, te, yo, bn). The main exception is fa (-1.09), which plan analyze further.
These results indicate that higher coverage in the alignment corpus amplifies the gains from MILCO, but is not strictly required: even languages that weakly covered entirely absent from the alignment corpus still benefit on average. This behavior highlights MILCO's multilingual alignment pretraining effective mechanism for knowledge transfer and sharing across languages.
24

---

Published as a conference paper at ICLR 2026
Table 21: MILCO effectiveness vs. alignment data distribution across languages (MIRACL Hard Negatives.). We report nDCG@10, per-language AnDCG@10 (MILCO vs. dense baseline), and the number and proportion of parallel corpus samples used for alignment pretraining.
Lang nDCG@ 10
AnDCG@ 10
Samples
Percentage
ar bn en
80.4
0.919
19002229
5.5
82.6
2.387
60.4
3.585
es
60.9
0.487
28470451
8.23
fa
62.3
-1.09
646913
0.19
81.2
2.676
4958793
1.43
fr
61.7
-0.922
22574836
6.53
hi
64.4
2.158
9626940
2.78
id
60.9
2.498
17591514
5.09
ja
77.2
2.524
11542221
3.34
ko
72.1
1.548
3648265
1.05
ru
74.6
2.458
16592711
4.8
80.3
0.734
0
te
87.9
0.87
th
84.2
1.295
1668858
0.48
zh
65.5
1.328
9006690
2.6
de
61.4
1.823
24431450
7.07
yo
83.6
0.41
25


Reviews:
[{"review_id": "qTTl2g7wlU", "reviewer": "Reviewer_ej1c", "summary": "This paper proposes a novel method for a learned sparse retriever (LSR) model targeting multilingual and cross-lingual retrieval tasks, an area where LSRs typically struggle. Dense embedding retrievers usually perform better on such tasks due to cross-lingual lexical and entity mismatches, which make sparse methods harder to optimize. The authors introduce a LexEcho head that maps sparse representations into an English pivot space while preserving the original language tokens with predicted weights for indexing. This multilingual fusion representation is shown to be crucial through ablation studies. The final model is trained using language alignment pre-training and contrastive learning with distillation. Experimental results on both multilingual and cross-lingual benchmarks demonstrate the effectiveness of the proposed LSR method.", "questions": "- How does the proposed model compare to the dense retrieval model under the same training setup?\n- How does the sparse retriever model with token pruning compare to the dense model running on GPU on latency?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method is creative and effective.\n2. The ablation studies on the modeling components are insightful and clearly demonstrate the importance of each part of the method.", "weaknesses": "1. The paper evaluates efficiency only through token pruning. However, it does not report the actual retrieval latency when using the sparse index, nor compare it to the latency of dense embedding models. Fewer tokens or sparsity does not necessarily guarantee faster retrieval, since inverted indices may produce very long posting lists in multilingual settings.\n2. A baseline dense embedding model trained with the same language alignment pre-training and contrastive learning with distillation (from the same cross-encoder and data) would strengthen the comparison of sparse vs dense. The paper presents an effective LSR model for multilingual retrieval. However, the weaknesses highlight missing experimental comparisons that make it difficult to determine whether sparse or dense models are ultimately more suitable for this task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OY7K0PDJCN", "reviewer": "Reviewer_LS1r", "summary": "This paper introduces MILCO, a LSR architecture for multilingual and cross-lingual retrieval. MILCO maps queries and documents into a shared English lexical space through the integration of a multilingual connector and a LexEcho head, enhancing robustness to uncommon entities. The paper also proposes a SAP strategy, which addresses the entity loss issue and improves retrieval effectiveness. Experimental results show that MILCO outperforms existing methods on multilingual and cross-lingual benchmarks, demonstrating high efficiency and accuracy.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper identifies that during cross-lingual projection, entities from non-Latin languages (e.g., Chinese or Arabic proper nouns) are often lost. To address this, it introduces the LexEcho head, which generates dual sparse representations — one from the English view and one from the source-language view — significantly improving robustness to rare entities and enhancing cross-lingual consistency.\n\n2. The paper introduces Sparse Alignment Pretraining (SAP), a novel pretraining strategy specifically designed for multilingual LSR. Unlike prior methods that align multilingual inputs within dense semantic spaces, SAP directly aligns multilingual inputs with English lexical targets at the vocabulary level.\n\n3. The paper is well-written.", "weaknesses": "Although MILCO improves training efficiency by mapping multilingual inputs into a unified English lexical space via the “Multilingual Connector,” this approach may introduce an English-centric bias. For languages with syntactic structures or lexical mappings that differ substantially from English, such monolingual alignment could lead to semantic distortion or representational misalignment, raising concerns about its generalization and fairness across diverse linguistic systems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dN6qzMwpzA", "reviewer": "Reviewer_4uhs", "summary": "This paper proposes MILCO, a multilingual learned-sparse retriever that projects inputs from many languages into a shared English lexical space via a lightweight multilingual connector. The model is trained in two stages: Sparse Alignment Pretraining (SAP) using large bitext corpora and an English LSR teacher, followed by Sparse Contrastive Training (SCT) with KL-distillation and InfoNCE. To preserve rare source-language entities lost in projection, the authors propose LexEcho, a dual-view head combining an English view with a weighted source view via an ECHO token. The method is empirically evaluated on MIRACL, MTEBv2, MLDR, MKQA and shows strong performance, supporting LSR post-hoc pruning and interpretability.", "questions": "1. Please answer **Weakness 1** by clarifying the motivation for mapping multilingual tokens to English tokens.\n2. Please answer **Weakness 2** by clarifying the necessity of your design except for cross-lingual bi-text retrieval. Any related comparison results or experiments are welcome.\n3. Please provide a more detailed training budget (GPU type/count, total GPU hours) and distillation/teacher details (which teacher checkpoints, how produced).\n4. How does MILCO perform on truly low-resource languages not well represented in parallel corpus? Can the authors report per-language numbers for the lowest-performing languages?\n5. What are the index size and query latency numbers (both full representation and after pruning) using typical inverted index settings (e.g. Anserini with Lucene9) and hardware?\n6. Can you provide more ablations on LexEcho fusion (e.g., varying weight between English and source views) and present error types where LexEcho hurts ranking?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. **Addresses an important practical challenge**: multilingual sparse retrieval with interpretability.\n2. **Good technical contributions**: SAP (sparse-aware MSE on logits) and LexEcho (dual view + [ECHO]) address semantic collapse of bi-text logits mapping.\n3. **Strong empirical results** across multiple benchmarks, with efficiency/accuracy tradeoff analysis (post-hoc pruning).", "weaknesses": "1. **Motivation needs to be strengthened**: why choosing to map multilingual tokens to English tokens? Could the authors clarify the rationale for specifically choosing English as the shared lexical space? What makes the English vocabulary a preferable or necessary choice in this framework?\n2. **Necessity outside of cross-lingual bi-text retrieval**: Does the proposed alignment (via the English teacher model) offer empirical or conceptual benefits that would not hold if the shared space were defined in another language or script? For purely monolingual retrieval scenarios (e.g., Chinese-to-Chinese), is the English-space projection still beneficial, or does it primarily target cross-lingual retrieval only?\n3. **Experimental comparison could be more rigorous**: disclosure of training compute, teacher model details, and dataset overlaps with baselines is incomplete. Without that, claims of beating much larger models risk being influenced by differing training data/compute.\n4. **Robustness to low-resource languages and domain shifts is not fully analyzed**: SAP relies on parallel corpora which may bias alignment.\n5. **Engineering/operational metrics (index size, latency, end-to-end retrieval QPS)** are not fully reported but are crucial for claims about effectiveness and deployability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)