KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Model
Authors: Xinping Zhao
Xinshuo Hu
Zifei Shan
Shouzheng Huang
Yao Zhou
Xin Zhang
Zetian Sun
zhenyu liu
Dongfang Li
Xinyuan Wei
Youcheng Pan
Yang Xiang
Meishan Zhang
Haofen Wang
Jun Yu
Baotian Hu
Min Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Embedding Model
LLMs
Retriever
Primary area: foundation or frontier models, including LLMs

Abstract: Recent advancements in Large Language Models (LLMs)-based text embedding models primarily focus on data scaling or synthesis, yet limited exploration of training techniques and data quality, thereby constraining performance. In this work, we propose KaLM-Embedding-V2 from the Lychee-KaLM team, a series of versatile and compact embedding models, systematically incentivizing advanced embedding capability in LLMs by superior training techniques and high-quality data. For model architecture, we implement the models on a 0.5B compact size with simple mean-pooling to produce fixed-length embeddings and remove the causal attention mask to enable fully bidirectional representation learning. For training techniques, we propose a progressive multi-stage training pipeline: pre-training on weakly supervised large-scale datasets, fine-tuning with supervised high-quality datasets, and contrastive distillation with fine-grained soft signals, integrated with focal-style reweighting and online hard-negative mixing to emphasize difficult samples and enrich hard negatives, respectively. For training data, we curate over 20 categories for pre-training and 100 categories for fine-tuning and contrastive distillation, to improve both performance and generalization, leveraging task-specific instructions, hard-negative mining, and example-based multi-class labeling to ensure high quality. Combining these techniques, our KaLM-Embedding-V2 series achieves state-of-the-art performance on the Massive Text Embedding Benchmark, outperforming models of comparable size and rivaling models 3-26x larger, setting a new standard for versatile and compact embedding models under 1B parameters. The code, data, and models are available at https://kalm-embedding.github.io/.

Published as a conference paper at ICLR 2026
KALM-EMBEDDING-VO SUPERIOR TRAINING TECH- NIQUES AND DATA INSPIRE A VERSATILE EMBEDDING MODEL
Xinping Zhao' Xinshuo Hu Zifei Shan', Shouzheng Huang', Yao Zhou', Xin Zhang, Zetian Sun, Zhenyu Liu, Dongfang Li, Xinyuan Wei, Youcheng Pan, Yang Xiang, Meishan Zhang, Haofen Wang", Jun Yu, Baotian Hu Min Zhang' Shenzhen Loop Area Institute (SLAI); Teencent, Shenzhen, China; Tongji University kinpingzhao@slai edu.. en, sinshaohu@tencent com
shouzzeenghuunngg222gmaa com, yoozhou@tencent. com
zi sifeishanetencent com, (baotianhu, minzhang] @ssaii edu.
httpss////ggggggceeeeeooo KaLM httpess///ggggiinaeee....o/////d Embedding
ABSTRACT
Recent advancements in Large Language Models (LLMs)-based text embedding models primarily focus on data scaling or synthesis, yet limited exploration training techniques and data quality, thereby constraining performance. In this work, we propose KaLM-Embeede Embedding -22 from the Lychee-KaLM team, a series of versatile and compact embedding models. systematically incentivizing advanced embedding capability in LLMs by superior training techniques and high-quality data. For model architecture, we implement the models on a 0.SB compact size with simple -aannppooing to produce fixed-length embeddings and remove the causal attention mask to enable fully bidirectional representation learning. For training techniques, propose progressive multi-stage training pipeline: pre- training on weakly supervised large-scale datasets, fine-tuning with supervised high-quality datasets, and contrastive distillation with fine-graie ine-grained soft signals, integrated with focal-style reweighting and online hard-negative mixing to em- phasize difficult samples and enrich hard negatives, respectively. For training data, we curate over 20 categories for pre-training and 100 categories for fine- tuning and contrastive distillation, to improve both performance and generalization, leveraging task-specific instructions, hard-negative mining, and example-based multi-class labeling to ensure high quality. Combining these techniques, our KaLM-EEbeddd.ng series achieves state-of-the-an art performance on the Mas- sive Text Embedding Benchmark, outperforming models of comparable size and rivaling models 3-26x larger, setting new standard for versatile and compact embedding models under IB parameters. The code, data, and models are available at https: -eelmmmmmeedinnnn qi github.ii. io/.
INTRODUCTION
Text embedding encapsulates text semantics and serves as fundamental infrastructure in numerous natural language processing (NLP) tasks (Muennighoff et al., 2023a; Xiao et al., 2024), including retrieval (Nguyen et al., 2016), reranking (Liu et al., 2018b), classification (McAuley & Leskovec, 2013), and semantic textual similarity (STS) (Agirre et al., 2012), etc. Recently, eotrieval-augment generation (RAG) has gained increasing attention in LLMs (Wu et al., 2022; Gao et al., 2023; Huang Huang, 2024; Zhao al., 2024; 2025; Rao al., 2025: Chen et al., 2025a), where embedding models play crucial role in RAG. It enables the efficient retrieval of external information to complement LLMs' outdated, incomplete, or inaccurate internal knowledge. With the advancement of LLMs, embedding models have become the primary bottleneck for improvement within the
Equual contribution
1 "ooresponndig Author

---

Published as a conference paper at ICLR 2026
MTTBBleng.oo vl), meen score Lasks
MTEBIcmn, vil, all tasks
2m
71
KILM Ewiendiigg 42.5
71.
aye syenttioguaaggo germa
KJLM KJLM Enbcoding:
Spe Bllnnual gammma
Mean (Tacio Score
69
*
Mean (Thak) Score
5)
gie 2ww.0o21.000-maee...
KILM Emocoding V2
&
Dower--Cbeerrggg et"
Kou Embeddinng/
cle musti owbbnrding33
63
spe-m
e5 mistr Ibirstnut
TuBl mu.biguul--oo sb-lagee
bgg m3
59
57
18
106
18
10B
Number Perameters (log scole)
Number of Parameters flog scale)
Figure 1: (Left) Comparison between the S-LM--Emeeddin ng series and other models on MTEB, The red dashed line depicts the logarithmic trendline fitted to the performance data of all the baseline models. The colors represent models with comparable parameter scales, with each group of models sharing the same parameter scale assigned a consistent color. (Right) Radar charts show our models achieve SOTA performance in a wide array of tasks.
RAG framework (Setty et al., 2024), which leads to the emergence of numerous text embedding models (Zhang et al., 2025e; Lee et al., 2025a;b;b; 2024; Huang et al., 2024; Xiao et al., 2024; Li et al., 2023; Vera et al., 2025).
Although numerous text embedding models have been built on massive or synthetic data (Zhang et al. 2025c; Lee et al., 2025b; 2024), they fall short in exploring superior training techniques and high-quality data, as well as how different training techniques, architecture designs, and data curation strategies can be systematically orchestrated to maximize the full potential of embedding capabilities in LLMs. Furthermore, most state-of-the-aa (SOTA) embedding models originate from industry. where proprictary data, closed training code, commercial restrictions, and limited reproducibility pose challenges for academic research. To this end, it necessary and valuable to establish new standards for open-source embedding models, emphasizing versatility and compactness two crucial properties demanded in real-world scenarios where accuracy and efficiency are paramount. By fully open- sourcing models, code, and data with commercial use permitted, we aim ensure transparency and reproducibility. thereby facilitating academic research and enabling widespread practical applications.
In this work, we propose KaLM-Embed Embedding-Ve a series of versatile and compact general purpoose text embedding models, enhanced with the well-designed model architecture, superior training techniques, and high-quality data curation, which aim incentivize advanced Knowledge in large Language Models into Embedding Models. Specifically, we make the following four innovations:
For model architecture, our BaLM--Embedding--- size, with simple yet effective mean- aaaan-pooiig layer to produce fixed-length embeddings. To further improve representation learning, we remove the causal attention mask of decoder-only LLMs and enable bidirectional attention during training well inference, which has been proven to be more effective for representation learning (Lee et al., 2025a;b; Sturua et al., 2024; Li et al., 2023).
series are implemented upon a 0.5B compact
For training recipe, we implement a progressive multi-stage training pipeline, starting with the Qwen2-0.5B (Yang et al., 2024). Specifically, the training begins with pre- pre-training on large-scale weakly supervised datasets that may include noise, then fine-tuning on relatively smaller, high- quality, supervised datasets, followed by contrastive distillation on fine-grained soft signals that capture nuanced differences. The multi-stage training pipeline progressively incentivizes advanced embedding capabilities in LLMs from coarse-graine goasse-rrained fine-grained representation learning.
For training objective, previous works (Lee et al., 2025a; Hu et al., 2025) equally treat each training sample, making the optimization direction dominated by the majority of easy samples. Inspired by (Lin et al., 2017), we introduce a focal-style reweighting mechanism to emphasize difficult samples. However, as training progresses, offline mined hard negatives become less challenging. To provide continual informative hard negatives, we propose synthesizing new hard ones via online pair-wise or list-wise mixing. Unlike offline mining, our online hard negative mixing blends features of existing hard negatives to generate new ones, significantly reducing computational cost.
For training data, curate over 20 categories of data for pre- pre-training and 100 categories of data for fine-tuning and distillation. We present a comprehensive recipe for curating high-quality training data, including dataset-specific construction, task-specific instructions, hard-negative mining, and
2

---

Published as a conference paper at ICLR 2026
example- based multi-clas, labeling. This allows the research community to reproduce the model and considerably lowers the entry barrier, facilitating the development of embedding models.
series obtains impressive performance on the Massive Text Embedding Benchmark (MTEB) English (eng) (Muennighoff et al., 2023a) and Chinese (cmn) (Xiao et al., 2024), significantly outperforming models of comparable size, as shown in Figure 1. Remarkably, even at a 0.5B size, the Ka Ka wiw./mmmddddiid22ddmmm series competes with 3-26x larger models. Out-of-domain (OOD) evaluation (Appendix C), matryoshka embedding evaluation (Appendix D), case study (Appendix E), visualization analysis (Appendix F), and multilingual evaluation (Appendix G) are provided in Appendices due to the page limit. In nutshell, the proposed model exhibits strong OOD generalization, competing with the 15x larger model in real-world retrieval scenarios; maintains robust performance with matryoshka embeddings at smaller dimensions, e.g., 256; case studies show its enhanced discriminative capacity in distinguishing positive passages from hard negatives; visualization analysis reveals superior intra- class compactness and inter-class separability clusters; and multilingual evaluation show that its performance is comparable to SOTA multilingual embedding models, even though it was not trained on large-scale multilingual corpora.
Combining these innovative techniques, our
RELATED WORK
Text embedding models. Text embeddings (Zhang et al., 2025b), which are vectors encapsulating text semantics, are fundamental for NLP tasks such as retrieval (Nguyen et al., 2016), reranking (Liu et al., 201 20) and classification (McAuley & Leskovec, 2013). BERT (Devlin et al., 2018) marked a significant milestone, using masked language modeling to pre-train deep bidirectional Transformer encoders for powerful contextual modeling. A breakthrough for sentence similarity tasks was Sentence-BERT (SBERT) (Reimers & Gurevych, 2019), which fine-tuned BERT-like models with pairs to generate semantically meaningful sentence embeddings directly comparable via similarity. Another prominent example is the Text-to-Tex. Transfer Transformer (T5) (Raffel et al., 2019) which follows fully encoder-decoder architecture and reframes all NLP tasks as text-to-text generation. While not initially designed for text embedding, the encoder portion of T5 can be used to generate powerful sentence representations. To systematically assess the robustness, generalization, and transfrrabbbbttt of such embedding models, comprehensive benchmarks like the Massive Text Embedding Benchmark (MTEB) (Muennighoff' et al., 2023a; Xiao et al.. 2024; Enevoldsen et al., 2025) have emerged. These benchmarks provide critical insight into how well embedding models perform in real-world, diverse scenarios, driving further research in text embedding.
LLMs embedding models. Pioneering studies explored the feasibility of leveraging LLMs for representation learning by adapting generative or encoder-decoder architectures into embedding models. E5 (Wang al., 2022) unified retrieval, classification, and III tasks under contrastive framework using in-batch negatives. GTR (Ni et al., 2022) fine-tuned T5 models for dual-encoder retrieval tasks. INSTRUCTOR (Su al., 2023) introduced instruction tuning for embeddings, enabling task-specific representation via natural language prompts. Recently, LLMs, characterized by their massive scale and remarkable capacity, have become prevailing paradigm in generating high- quality text embeddings. Many embedding models using LLMs as the backbone, e.g., BGE (Li et al., 2025), NV-Emb (Lee et al., 2025a), E5-Mistral (Wang et al., 2024a), GTE (Li et al., 2023; Zhang et al., 2025c), Jina (Sturua et al., 2024; Akram et al., 2026), as well as (Hu et al., 2025), mainly initialized from the Mistral or Qwen, etc, have achieved substantial improvements over earlier encoder- based models such as BERT and T5. Adapting LLMs into embedding models requires sophisticated training strategies, e.g., contrastive pre-training to draw semantically similar inputs together (Gao et al., 2021), instruction tuning to tailor embeddings for downstream tasks (Su et al., 2023), contrastive distillation for compression (Rao et al., 2023; Zhang et al., 2025a; Akram et al., 2026), and herd-negative mining enforce fine-grained distinctions. Although studied for ages, systematic research of superior training techniques and high-quality data curation is still underexplored.
3 METHOD
In this section, we present comprehensive technical details of the KaLM-Embon Bel.---Embedding--e series, including model architecture designs, training objectives, training recipes, and data curation strategies.
3

---

Published as a conference paper at ICLR 2026
allc
11. Glesrganc Loas
alla
Paaling Liger
Pastzing Lover
KaUM Embedding V2
KaEM BaLM-Embedding-V V2
KaLM Embodding-V2
Teacher Embedding Model
COOD $000 Bunk untrrtiin mplane) Seary?
OOO0 0000
Ohasage
Utank insinatieess Speryl//Pussgee)
(teak Untintiiess (pery)) (Passage)
Figure 2: The overall training workflow of the
series. The left illustrates
the workflow of contrastive learning, while the right shows that of contrastive distillation.
3.1 MODEL ARCHITECTURE
The KaI.M-Mbedda ng-V2 series is initialized from Qwen2-0.5B (Yang et al., 2024) and further tuned, which enables our embedding models to leverage the vast knowledge already encoded in its parameters. While causal attention masks are commonly used LLMs for language modeling, they are not well-suited for representation learning, thereby hindering embedding capacity (Lee et al., 2025a;b; Sturua al., 2024; Li al., 2023). To address this, we remove the causal attention mask and enable fully bidirectional attention. For text embedding, an imput sequence T of length L is processed by KaLM Embedding -V- denoted as K((). to produce token embeddings T emb I Rxxd A pooling layer P-)) is then applied to obtain single embedding E E IR representing the entire input:
Temb- = K(T), E = -P(TTeml)
(1)
where d is the hidden dimension. Following prior works (Lee al., 2025a;b; Hu et al., 2025), set P()) as the simple yet effective mean pooling. The input consists of the task instruction (optional) and the query/passage, described in $3.4. The overall training workflow is illustrated in Figure
3.2 TRAINING OBJECTIVE
Contrastive Learning. The KaLMM-mbeddi series mainly trained with the contrastive loss, specifically InfoNCE: (Gutmann & Hyvärinen, 2010), which maximizes the agreement of positive pairs while minimizing that of negative pairs. The workflow of contrastive learning illustrated on the left side of Figure 2. Generally, a training batch is organized as {4i q. P ....,, Pi, J where is the batch size. Each sample consists of a task instruction I,, a query a positive target pt and (optionally) M hard negatives {Pr, .P..2s-.>PP.... Before loss computation, the query q and passages (p, and pi.z) are encoded as vectors:
(2) where @ denotes concatenation. For most tasks, the instruction is prepended only to the query, while for symmetric tasks, it is also prepended to the passages, detailed in Table 1. Having established the embedding vectors of queries, positive targets, and hard negatives, for each mini-batch of size N, we optimize the contrastive learning objective with in-batch negatives and in-batch hard negatives as:
= P(K(T, @ ()), p =(C(())) P P(K(P()),,))),,
<q...p;))) )/T
N
N
M eaaa. (3)
L = E. - log iEN
Z; = c
+ Σ (a(an)))
Σ
Zi
where s(-) measures the similarity between two embedding vectors, which is set as the cosine similarity function; is the temperature coefficient; the three terms in the denominator Z; represent (1) the positive target, (2) in- in-batch negatives, and (3) in-batch hard negatives, respectively.
Focal-style Reweighting Mechanism. While effective, the above training objective treats each sample equally, making the optimization direction dominated by the majority of easy samples. Inspired by (Lin et al., 2017), we re-weight each sample according to its difficulty, where the more difficult the sample, the larger the weight, thereby focusing on learning difficult samples. The loss weight and the optimized training objective are defined as follows:
W = (qE 1/T )",
(q..))// W; log
L
E N
(4)
Z,
Z:
where y [0, +c) is focusing parameter controlling the skewness of the weighting scheme. When 0, the objective reduces to the standard form with uniform weighting. As increases, the loss pays attention to the difficult samples than the easy ones.
4

---

Published as a conference paper at ICLR 2026
Base Model
Stage Pre-training on Large-scale Weakly Supervised Datasets
Stage Fine-tuning High-quality, Supervised Datasets
Stage
Kal FalM-FmmbedWWie
Contrastive Distille
tion on Fine-grained Soft Signals
KoLM--Embed-V2
Figure 3: Multi-stage training pipeline of the KaIMM-Ebbddd ng-V2 series.
Online Hard Negative Mixing Strategy. As training progresses, offline mined hard negatives become less difficult after several training iterations. provide continual informative hard negatives throughout the training, previous works typically re-mines hard negatives after every fixed number of steps (e.g., 1000), which largely reduces training efficiency. To this end, we propose an online hard negative mixing strategy that synthesizes new informative hard negatives via pair sair-wise/list-we wise mixing, in favor of effectiveness and efficiency. The pair-wise/list-wei wise mixing can be formulated as:
h
h = Ap. + (1 A) i k. J. [1
(5)
8
AmPr.
Σ Am =1
(6)
where hy and s denote pair-wist and list-wise synthetic hard negatives, respectively; || the 12-norm; Pig and Pi,k are randomly drawn from the hard negative set (Pr.1.-.-,Pii without replacement; ~ Beta(a = 2 = 2), X E (0, 1); and Am = e(q..p .T. /E/ c((q. The mixing incurs negligible overhead. After synthesis, h and s, incorporated into the denominator Zi as additional hard negatives for query qs:
e<(q.pp
2, Z + Σ
da 1/5 + Σ (l C -=
-w, log
(7)
Z,
where multiple synthetic negatives can be applied, though only one is illustrated here for clarity.
Contrastive Distillation. Unlike previous works trained solely with coarse-grained hard signals, we further perform contrastive distillation by distilling fine-grained soft signals, i.e., the normalized distribution of temperature-scaled cosine similarity scores from a stronger teacher model (Qwen3- Embedding-88D (Zhang et al., 2025c)). This encourages the embedding model to capture nuanced differences between the positive and negative. Specifically, the training objective minimizes the discrepancy between the teacher's ' and the student's 's distributions. Formally, following (Hinton et al., 2015), we employ the Kullback-Leibler (KI.) divergence as the contrastive distillation objective:
Pi(i) P.O Po(i)
GRaa/T
CKL- DDK((PPPP)) - P((1) log
Egews P.(1)
(8)
Ejetosh
where Pt and P. represent the teacher's and student's distribution of similarity scores, respectively: Pt( and (i) denote the i-th entry; 2x, represents the i-th similarity score. We find that continual training with contrastive distillation yields substantial improvements over further fine-tuning with contrastive learning. It worth mentioning that the proposed contrastive distillation is model-agnostic and can be applied to any embedding models. The working flow of contrastive distillation is shown on the right of Figure 2.
Matryoshka Representation Learning (MRL). We incorporate MRL (Kusupati al., 2022) into both the contrastive (Equation 7) and Kl. (Equation 8) to enable flexible-dimensional embeddings, which leads the best overall performance with matryoshka embeddings as shown in Appendix D.
3.3 TRAINING RECIPE
To progressively incentivize embedding capabilities in LLMs, introduce multi stage training pipeline that smoothly transitions from coarse-grasd searse-grained to fin-graiied representation learning: (1)
Pre-training, (2) Fine-tuning, and (3) Contrastive distillation, described below.

---

Published as a conference paper at ICLR 2026
Table The task instruction query for training and evaluation.
Task Type
Instruction
Example
Given a query, retrieve documents that answer the query. \n Query: [query)
Instruct:
Retrieval, Reranking
General
Asymmetric
Instruct: Categorizing the given news title 'n Query: [query)
Classification, Clustering
Specific
Instruct: Retrieve semantically similar text.
Symmetric
STS, Pair Classification
General
In Query: {query)
Pre-training. The Ka Embedding -V series is first pre-trained on large-scale, weakly super- vised datasets spanning over 20 categories (refer to Table 16 for details) to learn general-puryy purposs representations. This stage employs the training objective in Equation 3, using only in- in-batch negatives. The comprehensive pre-training endows the model with strong generalization.
Fine-tuning. Next, the model is fine-tuned on over 100 categories of high-quality supervised datasets covering both retrieval and non-retrieval tasks, such STS and classification (referring to Table 17). This stage uses the training objective in Equation with a relatively small batch size to alleviate in-batch false negatives, further improving the overall model performance.
Contrastive Distillation. Finally, instead of further fine-tuning only with coarse coarse-grained hard signals, the model distills fine-grat geeeggrined soft knowledge from a stronger teacher model, using supervised high- quality data. The student trained align its normalized temperature-scine distribution with that of the teacher. This stage employs the training objectives in Equation 8 and Equation further improve the model capacity that captures nuanced semantic differences.
cosine similarity
The overall workflow of the multi-stage training pipeline is illustrated in Figure 3. The model obtained after pre-training followed by fine-tuning denoted KaLM- WiL.M---edddig-- V2, and further applying contrastive distillation produces KaL Ww..M--Ebbediiig- V2.5.
3.4 TRAINING DATA
We curate around 470M samples over 20 categories large-scale weakly supervised data for pre- training, and about 6M samples over 100 categories of high-quality supervised data for fine-tuning as well as contrastive distillation, with detailed statistics presented in Table 16 and Table 17. Our training datasets cover both retrieval and non-retrieval tasks, including retrieval, classification, clustering, STS, and pair classification. To ensure embeddings with specific task cneruuctionfollowinnn abilities, prepend specific task instructions to the queries. The instructed query is formulated as follows:
Cinst Instruct: (task instruction! Query: q-
Instructions for different task types are summarized in Table 1, and a detailed task instruction list is provided in Table 18. For symmetric tasks (e.g.. STS and Pair Classification), task instructions also prepended to the passages, whereas for asymmetric tasks, passages remain unchanged.
3.4.1 RETRIEVAL DATASETS
We collect diverse and comprehensive retrieval datasets for both pre-training and fine-tuning (see Table 16 and Table 17), and further enrich them via hard negative mining and persona-based synthesis.
Hard Negative Mining. As mentioned in $3.2, the training objective is to maximize the similarity between a query and its positive while minimizing similarity to negatives, especially hard negatives. However, most retrieval datasets only provide query-positive pairs. To address this, we mine hard negatives manually. Specifically, a previously trained model is used to retrieve candidate passages, from which we sample negatives ranked between positions 50 and 100.
Fersona-based Synthetic Data. Following (Wang et al., 2024a), we generate 550k synthetic samples using Qwen2-72B 7 spanning six task types with 40k unique instructions. To further enhance diversity, incorporate randomly sampled personas from Persona Hub (Chan et al., 2024) as system prompts during instruction generation, thereby enriching domain coverage while avoiding role conflicts in subsequent data generation (Tan et al., 2024).

---

Published as a conference paper at ICLR 2026
3.4.2 MON-RETRIEVAL DATASETS
In addition retrieval datasets, we also collect large-scale non-retrieval datasets covering four task types: (I) classification, (2) clustering, (3) semantic textual similarity (STS), and (4) pair classification (see Table 16 and Table 17). To ensure compatibility with contrastive learning, all datasets are reformulated into unified retrieval-style format: query q. positive target p*, and hard negatives {pi +P2 ......PM
PM )- To accommodate the different formats of these tasks, we process STS and pair classification symmetrically, and clusteringcclssificaaiii asymmetrically,
detailed below.
Symmetric Data Processing. To construct training samples for STS and pair classification datasets, we collect any pair of texts with the corresponding relevance score in the range [0. 5], L... (L', sccore, where we create two positive pairs (q = p'ppp = (") and (g - '". = (') if score 4. Besides, for the dataset with binary labels (0 or 1), create two positive pairs (g = t',p' = t") and (g === t", p = )') if score = 1. Hard negatives are mined from the candidate pool of other texts using the method proposed in $3.4.1. 1. Task instructions prepended both queries, positive targets, as well as hard negatives, because STS and pair classification are symmetric tasks, as shown in Table 1.
Asymmetric Data Processing. For clustering and classification datasets, training samples con- structed from text-label pairs (1, label) as (q= 1.pt = label). Hard negatives are first drawn from other labels within the dataset; if fewer than M, additional negatives clustering or classification datasets, mitigating the issue of having too few label categories in certain individual datasets. Task instructions are prepended to queries only in this situation. Inspired by (Lee et al., 2025a), we further apply example-based multi-class labeling: positives are randomly sampled examples from the same cluster/class, while negatives are sampled from other clusters/classes. In this symmetric setting, task instructions are prepended to both the queries, positives, and hard negatives.
sampled from labels across all
4 EXPERIMENT
Experimental details, including implementation details, comparison baselines, and evaluation, provided in Appendix B. The full MTEB results for all tasks, and the statistics of datasets as well as the detailed task instructions, are provided in Appendix H and Appendix respectively.
4.1 MAIN RESULTS
Table 2 presents the overall comparison of 18 models, reporting the average MTEB scores across all tasks and task types. From the results, we have several key observations: (1) Large-scale open-source models (> IB parameters) such as Bwenn-Emeddding-B8 NV-Embed-v2 and bge-multiliggual- gemma2 achieve strong results but at a high computational cost. (2) Among models with 1B parameters, KaLM Eiw..dd-Embddiig--. V2 achieves notable improvements over competitive baselines (e.g.. Owen3-Fimbeddingg
and (ina-mmbeddding--)), improving over v1 by +4.37 MTK (cmn) and +2.53 MTK (eng). (3) Kal.--Embdd V2.5 further advances SOTA among models with < 1B parameters, with average scores of 70.13 MTK (avg) and 69.16 MTY (avg), competing with billion- scale models while maintaining efficiency. Overall, these results manifest both effectiveness and compactness of the KaLM Embedding V2 series, making it an economical choice for deploying online applications.
Table 3 and Table 4 report detailed task results, where Class., Clust., PairCL., Reran., Retri., STS, and Summ. denote Classification, Clustering. Pair Classification, Reranking, Retrieval, Semantic Textual Similarity, and Summarization. Among models with < 1B parameters, achieves best or second-best results in 6/6 cases on MTEB (cmn, vl) 11 and 4/7 cases on MTEB (eng. vl). Compared to models with > IB parameters, performance across all tasks on both MTEB (cmn, v1) and MTEB (eng, vl), substantially advancing the development of downstream applications. These results manifest the versatility and compactness of the MaLM-EEmeddda V2 series again. Notably, the LaLM-Embeddingg tuned and distilled on just 2-4 GPUs with about 6M samples, compared to Swen3-Embeddingg 19M samples, indicating the effectiveness of our superior training techniques and data engineering.
-.w.eddmbbddnng--- V2.5
HELLM-Embedding-- -2.. achieves competitive
series is fine-
7

---

Published as a conference paper at ICLR 2026
Evaluation results MTEB Chinese (cmn) and English (eng). The best results boldfaced and the second-best ones are underlined (only considering models with < IB parameters). The KaLM-Embedeen Embedaing-V/ series achieves SOTA performance among competitive embedding models with <1B parameters, serving as economical choice for building online applications, e.g., RAG systems. 'M' and 'B' denote million and billion, respectively. MTK refers to Mean (Task),
Table 2:
MTY to Mean (Type). Results are mainly sourced from MTEB leaderboard (accessed Sep 10, 2025).
MTEB (emn, v1) MTEB (eng, v1) Avg
Model
Size
Dim
MTK
MTY
MIK
MTY
MTK MTY
Commercial embedding API services
-ett--ebeddinn--eeee 3-large (2024)
3072
64.52
62.33
Cohere-emin mutbeeligulllliaallll v3.0 (2023)
1024
64.01
62.09
Open-Source Embedding Models > 1B parameters
GRITLM 8X78 (T3B active) (2024)
T3B
4096
65.50
63.01
gge-mulilingual-gemme (2024)
9B
3584
67.64
68.52
69.88
66.11
68.76
67.32
NV-Fmbed-v2 (2025a)
7B
4096
72.31
67.97
Owen3-Embeddingg (2025c)
8B
4096
73.84
75.00
e5-mistral-7b-inst 7b-instuct (2022)
7B
4096
59.92
60.51
66.46
64.22
63.19
62.37
Qwen3-Embeediing-te (2025c)
2560
72.26
73.50
gte-
ste-Qween--,55-imstee (2023)
1.5B
1536
67.12
67.83
67.19
64.44
67.16
66.14
Open-Source Embedding Models < 1B parameters
#wenn--mmbedding-0..6 (2025c)
596M
1024
66.33
67.44
66.76
63.62
66.55
65.53
mnn-embbddings-v3 (Multi-LoRA) (2024)
572M
1024
61.82
61.61
65.51
62.76
63.67
62.19
muliiliggualee--arge (2024b)
560M
1024
58.08
58.24
60.89
59.48
59.49
58.86
bge-m3 (Dense) (2024)
560M
1024
60.34
61.23
59.84
58.98
60.09
60.11
Saaphhaase-ML--mpe---bbeee (2019)
278M
768
42.89
48.36
54.64
55.46
48.77
51.91
gte-multilingual base (Dense) (2024)
305M
768
62.94
63.92
61.40
60.10
62.17
62.01
KaLM Embedding series
KaLM- BLLL--Embddiing-- VI SaLM-Embeddinng Kal.M-Fmheddiin-VV V2.5
494M
896
63.78
64,56
64,94
61.49
64.36
63.03
494M
68.15
69.28
67.47
64.14
67.81
66.71
494M
896
70.93
72.46
69.33
65.83
70.13
69.16
Table 3: Detailed model performance on MTEB (cmn, vl) derived from C-MTEB (Xiao et al., 2024).
MTEB (emn, v1)
Model
Size
MTK
MTY
Class.
Clust.
PairCl.
Reran.
Retri.
STS
bge- poo-multilingual--eee
2uutilinguallgemma2 gemma2
9B
67.64
68.52
75.31
59.30
79.30
68.28
73.73
55.19
Soenn--Ebeedding-e
73.84
75.00
76.97
80.08
84.23
66.99
78.21
63.53
e5-mistral- 35-mistral-7-innstrue
7B
59.92
60.51
72.96
52.30
66.31
61.38
61.75
48.34
Bree3-Embeddinng44
4B
72.26
73.50
75.46
77.89
83.34
66.05
77.03
61.26
(Qwen2-1.5B-ingg
1.5B
67.12
67.83
72.53
54.61
79.50
68.21
71.86
60.25
0.6B
596M
66.33
67.44
71.40
68.74
76.42
62.58
71.03
54.52
jina-
saaa-embeddings--t (Multi-LoRA)
572M
61.82
61.61
70.47
50.22
67.22
60.72
68.54
$2.46
multiinguullee--argg
560M
58.08
58.24
69.80
48.23
64.52
$7.45
63.65
45.81
bge-m3 (Dense)
560M
60.34
61.23
70.52
45.75
73.98
62.88
65.43
48.79
paraphrase-Mi ML-mppet-base-t
278M
42.89
48.36
65.88
39.67
80.90
44.91
22.92
35.85
gte-multilingual-bo (Dense)
305M
62.94
63.92
66.84
47.48
78.34
68.17
71.95
50.75
Kal.M-Fmbccddnn- VI
494M
63.78
64.56
73.89
57.54
72.94
64.48
70.12
48.41
Kal.M-Embedding- V2
494M
68.15
69.28
75.14
69.76
77.91
65.16
72.15
55.58
KaLM MLLM-EEmeddiig... V2.5
494M
70.93
72.46
77.48
73.09
84.09
66.90
73.42
59.80
4.2
IN-DEPTH ANALYSIS
We next investigate how different key settings influence model performance, including (1) focal- style reweighting, (2) online hard negative mixing, (3) bidirectional attention, (4) example-based multi-class labeling, (5) contrastive distillation, and (6) the temperature coefficient.
Ablation Study Training Techniques. Table 5 and Figure 4 present the ablation results both MTEB (eng. vl) and MTEB (cmn, v1). We observe that removing focal-style reweighting leads the largest performance drop, with MTK dropping from 69.33 to 68.70 on eng and from
8

---

Published as a conference paper at ICLR 2026
Table 4: Detailed embedding model performance on MTEB (eng, v1) (Muennighoff al., 2023a).
Results on MTEB (eng, v2) (Enevoldsen et al., 2025) are provided in Table 12.
MTEB (eng, v1)
Model
Size
MTK MTY Class. Clust. PairCI Reran. Retri. STS Summ.
get-embeedding---lagee
64,52 62.33 75.12 49.01
85.81
59.16
55.43 81.73 30.05
Cohere-embed taaubeddmuitlliggal--a
64.01 62.09 76.01 46.60
86.15
57.86
53.84 83.15 30.99
GRITLM 8x7B
13B
65.50 63.01 77.69
50.14
85.23
59.80
55.13
83.26
29.82
gge-multlinguallgemmet
9B
69.88 66.11 88.08
54.65
85.97
59.72
59.24
83.88
31.20
NV-Embed-v2
7B
72.31 67.97 90.37
58.46
88.67
60.65
62.65
84.31
30.7
c5-mistral-7b-inse
7B
66.46
64.22 77.37
50.26
88.42
60.21
57.07
84.65
31.53
1.5B
67.19
64.44 82.53
48.75
87.52
59.98
58.29
82.81
31.17
Qwen3--Embeddding-O..
596M
66.76
63.62 82.61
49.87
84.29
57.96
54.32
86.97
29.23
jina tiie--mmeeddngg---ee v3 (Multi-
572M
65.51
62.76
82.58
45.21
84.01
58.13
53.88
85.81
29.71
multilingual-e5-lan
560M
60.89
59.48
71.77
41.23
84.75
55.96
51.40
81.62
29.64
bge-m3 (Dense)
560M
59.84
58.98
74.08
37.27
84.50
55.28
48.82
81.37
31.55
paraphrase-Me -a-paee-------nee---------e-
278M
54.64
55.46
67.46
38.50
80.81
53.80
35.34
80.77
31.57
gte- gte-multilingual- base (Dense)
305M
61.40
60.10
70.89
44.31
84.23
57.47
51.08
82.11
30.58
KaLM- WLLLM---edddingg-- V1
494M
64.94
61.49
84.74
47.82
83.26
55.41
51.65
82.24
25.23
Kal B-bee-nnnd V2
494M
67.47
64.14
87.19
56.05
86.18
56.74
51.67
82.61
28.51
Kal B-leM-Emmeddiig--- V2.5
494M
69.33
65.83
88.34
56.59
86.60
57.84
55.00
85.27
31.18
MTEB (eng V13
MTED leng. V1]
MTEA (cmn, v11
MIEU (cmn, vi)
66.0
Woon (hyes)
80.0
70
64.5
66
65
en Roua-apps
begetter Rivin
Hord
Aussition
53.5
Assk Serinting
ML.3
Rootighting
leod tepiltir Monin
wll Bldesatioal
Aa Eolendenn
42.8
aeptioe Maaig
Herd
wit Bisrectonnl
50
Pleeggningg
with
teep Mavg
oir Butnetiend
MTK
MTY
(c) MTK
MTY
Figure 4: Ablation study on focal-style reweighting, hard negative mixing, and bidirectional attention.
70.93 to 69.41 on cmn, indicating that it plays key role in improving general performance. On the other hand, climinating hard negative mixing or bidirectional attention yields smaller but consistent declines, demonstrating that hard negative mixing supplements informative hard negatives throughout training, while embeddings generated with bidirectional attention are more effective than those generated with causal attention. Overall, these results confirm that the proposed training techniques are complementary and jointly contribute to the performance of the KaIM- Eee--dimeddnnngg V2 series.
Example-based v.s. Label-based Labeling, Ta- ble 6 presents the comparison results between us- ing class/clust and sampled examples as positives and negatives. Note that, in the setting of 'Exam- ple', both example-based and label-based labeling data are used for training. The results demonstrate that example- based labeling leads considerable improvements, especially on the clustering task, demonstrating the effect of supplementing the class.
Table 6: Effect of example- based labeling.
MTEB (cmn, v1)
MTEB (eng, v1)
Setting
Class.
Clust.
Class.
Clust.
Example
77.48
73.09
88.34
56.59
Label
76.90
64.71
87.03
52.71
and clust, data with example-based multi-class labeling.
Effectiveness of Contrastive Distillation. Dur- ing the contrastive distillation stage, the KaLM- Embedding-V V2 is further optimized using the training objectives in Equation (denoted as 'KL') and Equation 7 (denoted as 'CL'), Implementation details can be seen in Appendix B. To assess the contribution of each objective, conduct ab-
Table 7: Effect of contrastive distillation.
MTEB (cmn, vl)
MTEB (eng, v1)
Setting
MTK
MTY
MTK
MTY
CI.
70.93
72.46
69.33
65.83
only KL
70.72
72.48
68.63
65,29
only CI.
68.31
69.88
67.67
64.37

---

Published as a conference paper at ICLR 2026
Table 5: Detailed ablation study results several key components, including focal-style reweighting.
hard negative mixing, and bidirectional attention.
MTEB (eng, v1)
Row
Setting
MTK
MTY Class.
Clust.
PairCI.
Reran.
Retri.
STS
Summ.
1
MoLM-Emmbeddingg V2.5
69.33
65.83
88.34
56.59
86.60
$7.84
55.00
85.27
31.18
2
w/o Focal-style Reweighting
68.70
65.19
87.68
55.40
86.62
57.66
54.82
84.31
29.86
3
w/o Hard Negative Mixing
68.91
65.39
87.88
55.81
86.67
57.46
54.91
84.64
30.38
4
w/o Bidirectional Attention
68.94
65.05
88.51
56.10
85.40
57.65
54.70
84.55
28.43
MTEB (cmn, v1)
V2.5 w/o Focal-style Reweighting
70.93
72.46
77.48
73.09
84.09
66.90
73.42
59.80
2
69.41
70.51
76.31
70.07
79.66
65.58
71.73
59.71
3 w/o Hard Negative Mixing
70.54
72.08
76.71
72.02
84.28
66.50
73.26
59.70
4 w/o Bidirectional Attention
70.50
71.95
77.41
72.71
82,87
66.40
73.01
59.27
lation study, as shown in Table 7. The results show that combining CL and KL achieves the best performance. Using only CL leads to the largest drop, while using only KL yields smaller but consis- tent declines, especially in MTEB (eng, v1). This means that KL serves as the primary learning signal, while CL provides the auxiliary learning one, and their combination yields the best performance.
Sensitivity of Temperature Coefficient. KL- divergence is sensitive to the temperature coef- ficient (coef) (Hinton al., 2015). Table 8 shows the performance in terms of different T under 0.01 (Low), т = 0.05 (Mid), and T 0.1 (High). We can observe that Mid leads to the best performance, since setting T to a too small value (e.g., 0.01) makes the teacher distribution overly skewed, while too large (such as I) oversmooths it, both reducing the informativeness of the learning signals.
Table Sensitivity of temperature coef т.
MTEB (cmn, v1)
MTEB (eng, v1)
Setting
MTK
MTY
MTK
MTY
Low
68.06
69.54
67.85
64.80
the 'only KL' setting, where
Mid
70.72
72.48
68.63
65.29
High
67.10
68.28
66.60
63.72
5 CONCLUSION
In this work, we propose BaMi---Ebedding-22 a series of versatile and compact embedding models that achieve SOTA performance on MTEB (cmn, vl) and MTEB (eng, vl) among competitive embedding models with < IB parameters. The strong performance stems from several systematized innovative designs. For model architecture, we remove the causal attention mask to enable more effective representation learning. For training techniques, we introduce multi-stage training pipeline that progressively incentivizes advanced embedding capabilities in LLMs. For training objectives, we introduce focal-style reweighting mechanism to emphasize difficult samples, and online hard- negative mixing strategy to enrich hard negatives. For training data, we collect over 20 categories of data for pre-training and 100 categories of data for fine-tuning well distillation, leveraging task- specific instructions, hard-negative mining, example- based multi-class labeling, etc, to carefully curate data. By combining superior training techniques and high-quality data, KaLM-Embo Embeeddin ing-22 significantly outperforms others of comparable size and even competes with 3x to 26x larger models.
ACKNOWLEDGMENTS
This work jointly supported by grants: Natural Science Foundation of China (No. 62422603, No. 62376067) and Guangdong Basic and Applied Basic Research Foundation (No. 2024B0101050000
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our work and to facilitate a clearer understanding of our contributions, we provide extensive supporting materials. In the main text, we describe our proposed method in $3 and present the detailed benchmark results in $4. In the Appendix B and Appendix I, we provide further detailed information, including implementation details, training details, and evaluation details, statistics of datasets, and task instructions used in evaluations, ensure our results reproducible.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Eneko Agirre, Daniel M. Cer, Mona T. Diab, and Aitor Gonzalez-Agirre. Semeval-2012 task 6: A pilot on semantic textual similarity. In SemEval@ pp. 385-393. The Association for Computer Linguistics, 2012. URL https::///// https:///ccaatthhooogg.o wtwp:gg...rrrgrgee org/ 2011-1.
Mohammad Kalim Akram, Saba Sturua, Nastia Havriushenko, Quentin Herreros, Michael Günther, Maximilian Werk, and Han Xiao, jin=emmbeddings--55teexx Task-targeted embedding distillation, 2026. URL https::///..mm waw.i.iiiivv////ii//..../.aaaava
wwtp:/i...rgga..ooooo.o org/ /200 15547.
Luiz Henrique Bonifacio, Israel Campiotti, Roberto A. Lotufo, and Rodrigo Frassetto Nogueira. mmarco: A multilingual version of MS MARCO passage ranking dataset. CoRR, abs/2108.1389 2021. URL https::ttp.............tt http//ixvvcoo/ org/ abs / 08 13897.
Vera Boteva, Demian Gholipour Ghalandari, Artem Sokolov, and Stefan Riezler. A full-text learning to rank dataset for medical information retrieval. In Nicola Ferro, Fabio Crestani, Marie-Francine Moens, Josiane Mothe, Fabrizio Silvestri, Giorgio Maria Di Nunzio, Claudia Hauff, and Gianmaria Silvello Advances in Information Retrieval 38th European Conference on IR Research, ECIR 2016, Padua, Italy, March 20-23, 2016. Proceedings, volume 9626 of Lecture Notes in Computer Science, pp. 716-722. Springer, 2016. doi: 10.
1 URL
https: /ttp//oi..o///// org.i..rgg/ae / 1007 9978- 319
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In Lluís Mårquez, Chris Callison-Buut Jian Su, Daniele Pighin, and Yuval Marton (eds.). Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pp. 632-642. The Association for Computational Linguistics, 2015. doi: 10.18653/V1/D15--007 URL htw.t:tpp///mm /tw.//oiiiio oro.orgg/// 11w53/866llle v115/ 1075.
lñigo Casanueva, Tadas Temeinas, Daniela Gerz, Matthew Henderson, and Ivan Vulic. Efficient intent detection with dual sentence encoders. CoRR, abs/2003.00807, 2020. URL https:
// //arxiv. atg/abs/////eeeeeooeee.....aaaar 2003 04807.
Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. Scaling synthetic data creation with 1,000,000,000 personas. CoRR, abs/2406.20099 2024. doi: 10445500ARXXV.220022000 URL https: /wo. /rg//0...//...ooooooooo...eeeo.. 48500 arxiv. 2406.20094 20094.
Huiyao Chen, Yi Yang, Yinghui Li, Meishan Zhang, Baotian Hu, and Min Zhang. Beyond chunking: Discourse-aware hierarchical retrieval for long document question answering. arXiv preprint arXiv:2506.06313, 2025a.
Jianly Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. BGE m3 m3-embedding: Multi-lingual, multi-functionaalty. multi-granun granularrtuuty text embeddings through self-knowledge distillation. CoRR, abs/22402.0326, 2024. doi: 1044850/AAXII2240000316 URL https: //doi. woogorrgo0... 10. 08855//aaiiii 2402 03216.
Jianluu Chen, Nan Wang, Chaofan Li, Bo Wang, Shitao Xiao, Han Xiao, Hao Liao, Defu Lian, and Zheng Liu. Air-bench: Automated heterogeneous information retrieval benchmark. In ACL (1), pp. 19921-20022. Association for Computational Linguistics, 2025b.
Jing Chen, Qingcai Chen, Xin Liu, Haijun Yang, Daohe Lu, and Buzhou Tang. The BQ corpus: A large-scale domain-specific chinese corpus for sentence semantic equivalence identification. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun 'ichi Tsujii (eds.). Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pp. 4946 Association for Computational Linguistics, 2018. URL
Cohere. Cohere-ebbddmmlliiigggaaa--- 2023. URL Colerre-mmbeddmm..i1inggggal
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman. Holger Schwenk, and Veselin Stoyanov. XNLI: evaluating lingual sentence representations. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun' Jun'ichi Tsujii (eds.), Proceedings of the 2018
11

---

Published as a conference paper at ICLR 2026
Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018. pp. 2475-2485, Association for Computational Linguistics, 2018. doi: (0.18653/V1/D18. 1269. URL https:://///..
wttpo//.t....//////////oo/.ooo 11t.1165////111111555.11 1269.
Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Y. Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mcjia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. No language left behind: Scaling human- centered machine translation. CoRR, abs/2207 04672, 2022. doi: 00448500AAXII.2207.04565 / 0tt.ol////.....////../..... arxiv. 2207. 04672.
URL https:://///
48550/
Yiming Cui, Ting Liu, Wanxiang Che, Li Xiao, Zhipeng Chen, Wentao Ma, Shijin Wang, and Guoping Hu. A span-extraction dataset for chinese machine reading comprehension. In Kentaro Inui. Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.). Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. EMNLP-IJCNLP 2019, Hong Kong. China, November 3-7, 2019. pp. 5882-5888. Association for Computational Linguistics, 2019. doi: 10. 16653/V1/D11-16000 URL https::/p//. /doi. org /0rg/11..0 1865663//// /0./vlll//// 9- 1600.
DataCanary, hilfialkaff,, Lili Jiang, Meg Risdal, Nikhil Dandekar, and tomtung. Quora question pairs, 2017. URL https:ttp//...........tt //kgg/aglllee e. com compet tttps/oooaooogggoueeeesee on- to-pal rs.
Jacob Devlin, Ming- Weng Chang, Kenton Lee, and Kristina Toutanova. BERT: pre- vre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018 URL http:://axixa org/
org/abs/////oooooooet...//..eerr abs 18111 04805.
Matthew Dunn, Levent Sagun, Mike Higgins, V. Ugur Güney, Volkan Cirik, and Kyunghyun Cho. Searchqa: A new q&a dataset augmented with context from a search engine. CoRR, abs//704.05177 2017. URL http: / /arxivv.vvvv.////////aaa//aavtvv org/abs/ 1704. 05179.
Kenneth C. Enevoldsen, Isaac Chung, Imene Kerboua, Márton Kardos, Ashwin Mathur, David Stap, Jay Gala, Wissam Siblini, Dominik Krzeminski, Genta Indra Winata, Saba Sturua, Saiteja Utpala, Mathieu Ciancone, Marion Schaeffer, Diganta Misra, Shreeya Dhakal, Jonathan Rystrøm, Roman Solomatin, Ömer Veysel Çagatan, Akash Kundu, and al. MMTEB: massive multilingual text embedding benchmark. In ICLR. OpenReview.net, 2025.
Anthony Fader, Luke Zettlemoyer, and Oren Etzioni. Open question answering over curated and extracted knowledge bases. In Sofus A. Macskassy, Claudia Perlich, Jure Leskovec, Wei Wang, and Rayid Ghani (eds.), The 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '14, New York, NY, USA August 24 27, 2014, pp. 1156 1-55-1165, 1165. ACM, 2014. URL https: do org/ 10. 1 2 2623330 2623677
doi: 10. 012114//662330.222666777
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. ELI5: long form question answering. In Anna Korhonen, David R. Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Itly, July 28- August 2, 2019, Volume I: Long Papers, pp. 3558-3567. Association for Computational I. Linguistics, 2019. doi: 1111653/V11/11--3346 URI htt.t: //tp:::///oii org/10. 18653/
Jack FitzGerald, Christopher Hench, Charith Peris, Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron Nash, Liam Urbach, Vishesh Kakarala, Richa Singh, Swetha Ranganath, Laurie Crist, Misha Britan, Wouter Leeuwis, Gökhan Tür, and Prem Natarajan. MASSIVE: A Im-example multilingual natural language understanding dataset with 51 typologically-divese languages. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14. 2023, pp. 43022. Association for Computational Linguistics, 2023. doi: 10.18653/V AC2-222233.LLL0000 235. URL https: // //ooi. 10. 1too.oogg/00..853/1111 2023 acl aallllloooen 235.
12

---

Published as a conference paper at ICLR 2026
Wikimedia Foundation. Wikimedia downloads, 2024. URL BtLhttt:://// http:pp::////pps.p.ppp wwkimedia.oo... Accessed: 2024-05-01.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. In Maaii-Frrnccin Moens, Xuanjing Huang. Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings the 2021 Conference on Empirical Methods in Natural Language Processing. EMNLPP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 6894 6910. Association for Computational Linguisties, 2021. doi: 10. 10.186//11/2211.11 /ttp://////222.MM.... EMNLP-MAIN, 552. URL https::///// /dii htt.:/ii...gg..gg//// 10. www.11166////22/// 20211 emw.p-mmmppppp. i.mainn55.
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Qianyu Guo, Meng Wang, and Haofen Wang. Aarrieval-augmente generation for large language models: A survey. CoRR, abs/2312.10991 2023. doi: (048550/ARXXV2232219999 URL https: bittps://dot http:///doo.oo/....... 48550/ arxiv. 2312 10997.
Gregor Geigle, Nils Reimers, Andreas Rücklé, and Iryna Gurevych. TWEAC: transformer with extendable QA agent classifiers. CoRR, abs/2104.07081, 2021. URL abs/2104 07081.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Yee Whye Teh and D. Mike Titterington (eds.). Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2010, Chia Laguna Resort, Sardinia, Italy, May 13-15, 2010, volume of JMLR Proceedings, pp- ngs wgw.mlm... wwwpseeerrr.... /atpss/////gttmnnn////// gutmgnn1 0a
297-304. JMLR.org, 2010. htt.http///pooot/ html.
Felix Hamborg, Norman Meuschke, Corinna Breitinger, and Bela Gipp. news-please - A generic news crawler and extractor. Maria Gäde, Violeta Trkulja, and Vivien Petras (eds.), Everything Changes, Everything Stays the Same? Understanding Information Spaces. Proceedings of the 15th International Symposium of Information Science, ISI 2017, Berlin, Germany, March 13-15, 2017, volume 70 of Schriften zur Informaationswissenscchaa,, pp. 218 223.. Verlag Werner Hülsbusch, 2017. doi: 10.18452/1447. URL https: /doiiiioooooo/ wor.i..org//... 10. 22442/11444
Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Samin Mubasshir, Yuan-Fang Li, Yong- Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. XI-sum: Large-scale multilingual abstractive summarization for 44 languages. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Findings of the Association for Computational Linguistics: ACI/IJCNLP 2021, Online Event, August 1-6, 2021, volume ACL/IICNLP 2021 of Findings of ACL, pp. 4693-4703. ACL.413.
Association for Computational Linguistics, 2021. doi: 1116653/V1222111INDD
URL https:://///mm /ooi org/ / 10. 11653/v1/20000 http:222....iinnnieeeee findings- 413.
Wei He, Kai Liu. Jing Liu, Yajuan Lyu, Shiqi Zhao, Xinyan Xiao, Yuan Liu, Yizhong Wang, Hua Wu, Qiaoqiao She, Xuan Liu, Tian Wu, and Haifeng Wang. Dureader: a chinese machine reading comprehension dataset from real-world applications. In Eunsol Choi, Minjoon Seo, Danqi Chen, Robin Jia, and Jonathan Berant (eds.), Proceedings of the Workshop on Machine Reading for Question Answering @ ACL 2018, Melbourne, Australia, July 19, 2018, pp. 37-46. Association for Computational Linguistics, 2018 doi: 10.18653/V 1- URL https: //aclanthology / -2605.
Kevin Hefferman, Onur Çelebi, and Holger Schwenk. Bitext mining using distilled sentence representations for low-resource languages. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 2101-2112. Association for Compu- tational Linguistics, 2022. doi: 10. //doi wrogr/rg1000 0. 18653/166vlll /2022. f findi indinngnnnnnnnnn p. 154.
URL https:
Geoffrey F. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. CoRR, 503.0533,.,, 2015.
Doris Hoogeveen, Karin M. Verspoor, and Timothy Baldwin. Cqadupstack: A benchmark data set for community question-answering research. In ADCS, pp. 3:1-3:8. ACM, 2015.
13

---

Published as a conference paper at ICLR 2026
Yupeng Hou, Jiacheng Li, Zhankui He, An Yan, Xiusi Chen, and Julian J. McAuley. Bridging language and items for retrieval and recommendation. CoRR, abs/2403.03952, 2024. doi: 10. 48500/ARXIV.2403000555
URL https: : /dii. wrg./o..//...... 48550 wtt.:./..vvvv...... 2403. 03952.
Baotian Hu, Qingcai Chen, and Fangze Zhu. LCSTS: A large scale chinese short text summarization dataset. In Lluís Marquez, Chris Callison-Burch, Jian Su, Daniele Pighin, and Yuval Marton (eds.), Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pp. 1967-1972. The Association for Computational I Linguisties, 2015. doi: 1111118////DDD---1 1229. URL https: /doii org/10. 18653/vl 00153/v1161--111 2110/1151122211
Hai Hu, Kyle Richardson, Liang Xu, Lu Li, Sandra Kübler, and Lawrence S. Moss. OCNLI: original chinese natural language inference. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings the Association for Computational Linguistics: EMNLP 2020, Online Event, 16- 20 November 2020, volume EMNLP 2020 of Findings of ACL, pp. 3512-3526. Association for Computational Linguisties, 2020. doi: 10. 1111888/1/222001INDDD HINNIIIIE-NNNNNGSSLLLLL URL
https: http://doi.ooooo
18w../1653/vvv wwwp////22... http://2222......nni...a. findings eem.ldinnnnseell.....m.mmmmmm... 314.
Xinshuo Hu. Zifei Shan. Xinping Zhao, Zetian Sun, Zhenyu Liu, Dongfang Li, Shaolin Ye, Xinyuan Wei, Qian Chen, Baotian Hu, Haofen Wang, Jun Yu, and Min Zhang. aalm-embedding: Superior training data brings A stronger embedding model. CoRR, 0tss/2001.00022 2025. doi: 10.48550/ ARXIV.2501.010 010028. URL https: /oi. org/10......oooo.......oo 00. 48550/ arxiv 2501. 01028.
Xuming Hu, Zhijiang Guo, Guanyu Wu, Aiwei Liu, Lijie Wen, and Philip S. Yu. CHEF: A pilot chinese dataset for wvddence-bssed fact-checking. In Marine Carpuat, Matie-Catherine de Marneffe, and Iván Vladimir Meza Ruíz (eds.). Proceedings of the 2022 Conference of the North American Chapter the Association for Computational Linguisties: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pp. 3362-3376. Association for Computational Linguistics, 2022. doi: A.11661/V122222NAAL--MMMAA
URL hhttps://dol. org/10.
18653/v /0153/v1/20 2022. naacl main. 246.
Junqin Huang, Zhongjie Hu, Zihao Jing, Mengya Gao, and Yichao Wu. Piccolo2: General text embedding with multi-task hybrid loss training. CoRR, wwwp224550093322 2024. doi: 10.48550/ ARXIV.2405.0697. URL https::///// //tp:///oii///oo org/10. 48550/0/5.///........
Yizheng Huang and Jimmy Huang. A survey retrieval-auggmentée text generation for large language models. CoRR, 10s82404.19981, 2024. doi: 10.48550/ARXXIV.4401.
URL
https: /tw.///oii/// orgg 100. 48550/ wwt.0//rrivvvv/.. 2404 10981.
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Mare Brockschmidt. Code- searchnet challenge: Evaluating the state of semantic code search. CoRR, abs/1909.0944 2019. URL http: /ett:://arxix aawivv.vivv.. org/ abs /1909 09436.
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W. Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan (eds.). Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. EMNLP-IJCNLP 2019, Hong Kong. China, November 3-7, 2019, pp. 2567-2577 Association for Computational Linguistics, 2019. doi: 10. 1/D99 URL https: 1259.
//doi. ooooi.org//0.. www.11863vvl//
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min-Yee Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 August Volume 1: Long Papers, pp. 1601-1611. Association for Computational Linguistics, 2017. doi: 10.18653/V1/P1--111 URL. https: //doi.orgg 10. /653vvlllll/ /
Daniel Khashabi, Amos Ng, Tushar Khot, Ashish Sabharwal, Hannanch Hajishirzi, and Chris Kallison-Burchh Gooaq: Open question answering with diverse answer types. In Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic,
14

---

Published as a conference paper at ICLR 2026
16-20 November. 2021, pp. 421-433. Association for Computational Linguistics, 2021. doi: 10. 18653N11202.FIIDIINGSS -ENNNG.... URL http://:.....r11...11/33/1///11/ findings-emnip. 38.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015. URL http: //arxvaaaivv.v//aa/./. org abss/////////// /1412. 6980..
Yuta Koreeda and Christopher D. Manning. Contractnli: A dataset for document-leves natural language inference for contracts. Mrrie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Findings of the Association for Compulational Linguistics: EMNLP 2021, Virtual Event Punta Cana, Dominicun Republic, 16-20 November, 2021, pp. 1907-1919. Association for Computational Linguistics, 2021. doi: 111166331122221FIIIIIGGSGS URL https:/ doi org / 11653/v1/222// f findings per.anngg-een.....mmm 164.
Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-SSyder, Kaifeng Chen, Sham M. Kakade, Prateek Jain, and Ali Farhadi. Matryoshka representation learning. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural In- formation Processing Systems 35: Annual Conference on Neural Information Process- ing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 December 2022, 2022. URL http: http:/ppprrr...... nips. cc/ pttpppppe__fili Feeeeeellaie papps///p/22222 2tt////222/hhh//// c32319944862476115550 211. wate:ea22aaaatttttt-a-a-aa-- wtw.:ratttt-ooofeeneeeee html.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Keig-Wei&&e Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: benchmark for question answering research. Trans. Assoc. Comput. Linguistics, 7:452-466, 2019. doi: 10.1 (01162//TACILAAL0022200 URL https::///doo. org/10. 11 1162/tacll 600216
Ken Lang. Newsweeder: Learning to filter netnews. In Armand Prieditis and Stuart Russell (eds.), Machine Learning, Proceedings of the Twelfth International Conference on Machine Learning, Tahoe City, California, USA, July 9-12, 1995, pp. 331-339. Morgan Kaufmann, 1995. doi: 10.1016/B978- 1--58660377--6500444 b978 55860 377 6. 50048-7. 7.
URL https:
1016/
Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Ny-embed: Improved techniques for training iims generalist embedding models. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025a. URL https: /openrevieww... net forum?ide 1gsyLSsDRe.
Jinhyuk Lee, Zhuyun Dai, Xiaoqi Ren, Blair Chen, Daniel Cer, Jeremy R. Cole, Kai Hui, Michael Boratko, Rajvi Kapadia, Wen Ding, Yi Luan, Sai Meher Karthik Duddu, Gustavo Hernández Ábrego, Weiqiang Shi, Nithi Gupta, Aditya Kusupati, Prateek Jain, Siddhartha Reddy Jonnalagadda, Ming-Wei Chang, and Iftekhar Naim. Gecko: Versatile text embeddings distilled from large language models. CoRR, abs/2403.20327. 2024. URL https: //tpi////iiii/////// wtt.:i..rgr11....0 arXiv. 2403 20327.
Jinhyuk Lee, Feiyang Chen, Sahil Dua, Daniel Cer, Madhuri Shanbhogue, Iftekhar Naim, Gus- tavo Hernández Ábrego, Zhe Li, Kaifeng Chen, Henrique Schechter Vera, Xiaoqi Ren, Shanfeng Zhang, Daniel Salz, Michael Boratko, Jay Han, Blair Chen, Shuo Huang, Vikram Rao, Paul Sugan- than, Feng Han, Andreas Doumanoglou, Nithi Gupta, Fedor Moiseev, Cathy Yip, Aashi Jain, Simon Baumgartner. Shahrokh Shahi, Frank Palma Gomez, Sandeep Mariserla, Min Choi, Parashar Shah, Sonam Goenka, Ke Chen, Ye Xia, Koert Chen, Sai Meher Karthik Duddu, Yichang Chen, Trevor Walker, Wenlei Zhou, Rakesh Ghiya, Zach Gleicher, Karan Gill, Zhe Dong. Mojtaba Seyedhos- seini,
Sung, Raphael Hoffmann, and Tom Duerig. Gemini embedding: Generalizable embeddings from gemini. CoRR, 0as/2503..789/) 2025b, doi: 1044850/ARXIV22503.7755 www................. 2503 07891.
URL https: // //w...iii////////////// .erii.org/// 48550/ arxiv
15

---

Published as a conference paper at ICLR 2026
Patrick S. H. Lewis, Yuxiang Wu, Linqing Liu, Pasquale Minervini, Heinrich Küttler, Aleksandra Piktus, Pontus Stenetorp, and Sebastian Riedel. PAQ: 65 million probably-asked questions and what you can do with them. Trans. Assoc. Comput. Linguisties, 9:1098-1115, 2021. doi: 10.1 112/TAACLLALLOOO 000455 URL https: : /ooi htt//oooooroooo org/orgg///.
62/tacll 1t0o://aacl__015.5..11
Chaofan Li, Minghao Qin, Shitao Xiao, Jianlyu Chen, Kun Luo, Defu Lian, Yingxia Shao, and Zheng Liu. Making text embedders few-shot learners. The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL https: wwwp/eeeeeewe...m net/ / forum? d= jQ0u.
Haoran Li, Abhinav Arora, Shuohui Chen, Anchit Gupta, Sonal Gupta, and Yashar Mehdad. MTOP:
A comprehensive multilingual task-oriented semantic parsing benchmark. In Paola Merlo, Jörg Tiedemann, and Reut Tsarfaty (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 23, 2021, pp. 2950-2962. Association for Computational I Linguistics, 2021. doi: 10. 10.18663/11/0011 EACL-MAIN.257 URL https: /oii org 186538653322 22021 eacl 257.
Yudong Li, Yuqing Zhang, Zhe Zhao, Linlin Shen, Weijie Liu, Weiquan Mao, and Hui Zhang. CSL: A large-scale chinese scientific literature dataset. In Nicoletta Calzolari, Chu-Ren Huang, Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin-Hsi Chen, Lucia Donatelli, Heng Ji, Sadao Kurohashi, Patrizia Paggio, Nianwen Xue, Seokhwan Kim, Younggyun Hahm, Zhong He, Tony Kyungil Lee, Enrico Santus, Francis Bond, and Seung-Hoon Na (eds.), Proceedings of the 29th International Conference Computational Linguistics, COLING 2022, Gyeongju, Republic of Korea, October 12-17, 2022, pp. 3917-3923, International Committee on Computational Linguistics, 2022. URL https: http:///aaaaattoooooooee www.hhhllooo......ooooeo 2022. wwt.:22.....nnoo coling-1. 344.
Zehan Li, Xin Zhang. Yanzhao Zhang, Dingkun Long, Pengiun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning. CoRR, abs/2300.00328, 2023. doi: 10.4850/ARXIV..308..252 033881. URL 03281.
Bw/8/5500055.. 2308.
Tsung- Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In IEEE International Conference Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pp. 2999-3007. IEEE Computer Society, 2017, doi: 10. 10.1109/CCV22217.3224 URL https::///// //cii.//////i//// org/10 10. 1109/ ICCV 2017..2... 324.
Hongcheng Liu, Yusheng Liao, Yutong Meng, and Yuhao Wang. Xiezhi: Chinese law large language model, 2023, URL https: / htt//gittuuuu cww htmm/LiuHC44288 BALWWAAWPPP.
Xin Liu, Qingcai Chen, Chong Deng, Huajun Zeng, Jing Chen, Dongfang Li, and Buzhou Tang. LCQMC: A large-scale chinese question matching corpus. In Emily M. Bender, Leon Derezynski, and Pierre Isabelle (eds.). Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, August 20-26, 2018, pp. 1952-1962. Association for Computational Linguistics, 2018a. URL https: / /alaccaantin aeclannhology org/ c18-1166/
Xueqing Liu, Chi Wang, Yue Leng, and ChengXiang Zhai Linkso: a dataset for learning to retrieve similar question answer pairs software development forums. In NL4SE ESEC/SIGSOFT FSE, pp. 2-5. ACM, 2018b.
Dingkun Long, Qiong Gao, Kuan Zou, Guangwei Xu, Pengjun Xie, Ruijie Guo, Jian Xu, Guanjun Jiang. Luxi Xing, and Ping Yang. Multi-cpr: multi domain chinese dataset for passage retrieval. In Enrique Amigó, Pablo Castells, Julio Gonzalo, Ben Carterette, J. Shane Culpepper, and Gabriella Kazai (eds.), SIGIR '22: The 45th International ACM SIGIR Conference on Research and Develop- ment in Information Retrieval, Madrid, Spain, July 11 15, 2022, pp. 3046-3056 3056. ACM, 2022. doi: 10.1 09.11455/47749553333773 URL hwwps:ttp..........t. doi org/ 0. 1 1 3477495 35317365 7366.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Dekang Lin, Yuji Matsumoto, and Rada Mihalcea (eds.), The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, 19-24 June, 2011, I, Portland,
16

---

Published as a conference paper at ICLR 2026
Oregon, USA, pp. 142-150. The Association for Computer Linguisties, 2011. URL https: //aclanthology wttp://oo////.....//
Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. Www' 18 open challenge: Financial opinion mining and question answer- ing. In Pierre-Antoine Champin, Fabien Gandon, Mounia Lalmas, and Panagiotis G. Ipeirotis (eds.), Companion of the The Web Conference 2018 on The Web Conference 2018, WWW 2018, Lyon, France, April 23-27, 2018, pp. 1941-1942. ACM, 2018. doi: 10.114553184558.11923300 URL https: /oi. org 10 1145/ 3184555 3192301.
Chaitanya Malaviya, Subin Lee, Sihao Chen, Elizabeth Sieber, Mark Yatskar, and Dan Roth. Expertqa: Expert-curated questions and attributed answers. In Kevin Duh, Helena Gómez-Adorno, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Compulational Linguistics: Human Language Technologies (Volume Long Pupers), NAACL. 2024, Mexico City, Mexico, June 16-21, 2024, pp. 3025-3045. Association for Computational Linguistics, 2024. doi: 1L.8633/12022,MAALLLLLOO //doi. org 10. (00/653/v1/200
URL https:
nww.l paaacl-longe 1on.lllooon.... 167.
Julian J. McAuley and Jure Leskovec. Hidden factors and hidden topics: understanding rating dimensions with review text. In Qiang Yang, Irwin King, Qing Li, Pearl Pu, and George Karypis (eds.), Seventh ACM Conference on Recommender Systems, RecSys '13, Hong Kong. China, October 12-16, 2013, pp. 165-172. ACM, 2013, doi: 10.1145/2507155.250777 20w.57771777277116.1 2507163.
URL https:
//doi. 10.
Sepideh Mollanorozy, Mare Tanti, and Malvina Nissim. Cross-lingual transfer learning with persian. In Proceedings of the 5th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, pp. 89-95, 2023.
Niklas Muennighoff, Nouamane Tazi, Loie Magne, and Nils Reimers. MTEB: massive text embedding benchmark. In Andreas Vlachos and Isabelle Augenstein (eds.), Proceedings of the 17th Conference of the European Chapter of the Association for C Computational Linguistics, EACL 2023, Dubrovnik, Croatia, May 2-6, 2023, pp. 2006-2029, Association for Computational Linguistics, 2023a. doi: 10.18653/V1/2023 BB005///1//2223AC-LMMMMM EACL--AIAAALLLIIIIA 148. URL httph::t///aat .rr.
bacl-main. 148.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M. Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang. Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. Crosslingual generalization through multitask finetuning. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9- 2023, pp. 15991 1-16111, Association for Computational Linguisties, URL https: http:/idi.igo..11...e.
2023b. doi: 10.18653/V 2023 acl- long. 891.
Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. Generative representational instruction tuning. CoRR, 0bs/24422.09906 2024. doi: 10.4850/ARXXI.240000009 URL https: /tt.///o..oooo/oo// org/10..rg1....0 48550/arXiv....//////// 2402. 09906.
Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of GPT4, CoRR, abs/2306.022077, 2023. doi: 10448500/ARXW.220000022 48550/arXiv 2306 02707.
URL https://doi... org/10.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted question-answeree with human feedback. CoRR, abs/21 12. 29332, 2021. URL https://aras arg/abs 22... 09332.
17

---

Published as a conference paper at ICLR 2026
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. MS MARCO: A human generated machine reading comprehension dataset. In Tarek Richard Besold, Antoine Bordes, Artur S. d' Avila Garcez, and Greg Wayne (eds.), Proceed- ings the Workshop on Cognitive Computation: Integrating neural and symbolic approaches 2016 co-located with the 30th Annual Conference
Neural Information Processing Systems (NIPS 2016), Barcelona, Spain, December 9, 2016, volume 1773 of CEUR Workshop Proceedings. CEUR- WS.org 2016. URL https: haops://ceus /ceur ws . wwog/oogoo.........ooooooooo Noooooo7777 C11/3///CCONPPP 2016 pdf.
/ttp//CCCCNCCC1CopaParana paper..
Jianmo Ni, Jiacheng Li, and Julian J. McAuley. Justifying recommendations using distantly-labeled reviews and fine-grained aspects. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLE 2019, Hong Kong, China, November 3-7, 2019, pp. 188-197. Association for Computational Linguisties, 2019. doi: 10. 1653/V1/D119 1018. 8. URL https : / org/ 100 1188663//2 /D19-1//1 8.
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernández Ábrego. Ji Ma, Vincent Y. Zhao, Yi Luan, Keith B. Hall, Ming- Wei Chang, and Yinfei Yang. Large dual encoders are generalizable retrievers. In Yoav Goldberg, Zormitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 9844-9855 Association for Computational Linguistics, 2022. doi: 18653/v1/2024 e0m..mm2.....m....m main. 669.
669, URL https:
James O'Neill, Polina Rozenshtein, Ryuichi Kiryo, Motoko Kubota, and Danushka Bollegala. I wish would have loved this one, but I didn't A multilingual dataset for counterfactual detection in product review. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic. 7-11 November, 2021, pp. 7092- 7108. Association for Computational Linguistics, 2021. doi: 10. v 568. URL www.thtttt/./...... /tt:://ooii..o org 10. / emn www.mmmnlmmmmmmmmmmmmmmmmmmmmmmm 1.
OpenAl. rexteembedding---larg... 2024. new www.eeminnnemdeeeeeeeeeeeeeeeeee and- updates/.
URL htttp::/oppeai.coom index/
Yujia Qin, Zihan Cai, Dian Jin, Lan Yan, Shihao Liang, Kunlun Zhu, Yankai Lin, Xu Han, Ning Ding, Huadong Wang, Ruobing Xie, Fanchao Qi. Zhiyuan Liu, Maosong Sun, and Jie Zhou. Webepm: Interactive web search for chinese long-form question answering. In Anna Rogers, Jordan L. Boyd- Graber, and Naoaki Okazaki (eds.). Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Pupers), ACL 2023, Toronto, Canada, July 9-14, 20233 pp. ACL-LONG.499 URL https: /doi wttp//.........//////...
Association for Computational Linguisties, 2023. doi: 2..1853//1/2025 acl -aail-oon...000 499.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR, Abs/1910.10683, 2019. URL http: / arxivv org. abs / 191119911111111 10683.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100, 000+ questions for machine comprehension of text. In Jian Su, Xavier Carreras, and Kevin Duh (eds.). Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pp. 2383-2392. The Association for Computational Linguisties, 2016. doi: 10. N0.18653/V1/D16-- 1264. URL https: / /tt//ooooooooooooo org / 10. 18653 (01655/vv/d/ 6-16446,66666
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad. In Iryna Gurevych and Yusuke Miyao (eds.). Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 2: Short Papers, pp. 784-789. Association for Computational Linguistics, 2018. doi: 10.18653/V /t///P18122.-- URL https: httpp::/acaaaato.l////// hatpa/aaaanntooooooooooo
org/ wwt.oorrg//////e/ P18-2124/
Jun Rao, Liang Ding, Shuhan Qi, Meng Fang, Yang Liu, Li Shen, and Dacheng Tao. Dynamic contrastive distillation for image-text retrieval. IEEE Trans. Multim., 25:8383-8384 2023.
18

---

Published as a conference paper at ICLR 2026
Jun Rao, Zepeng Lin, Xuebo Liu, Xiaopeng Ke, Lian Lian, Dong Jin, Shengjum Cheng, Jun Yu, and Min Zhang. Apt: Improving specialist Ilm performance with weakness case acquisition and iterative preference training, 2025. URL http::////axxx.../g/.oo
2200. 03483.
Chandan K. Reddy, Lluís Mårquez, Fran Valero, Nikhil Rao, Hugo Zaragoza, Sambaran Bandyopad- hyay, Arnb Biswas, Anlu Xing, and Karthik Subbian. Shopping queries dataset: A scage ESCI benchmark for improving product search. CoRR, fabbs/2206006588, 2022. doi: 10.48550/
ARXIV.2206.06558 URL https::/p//............t /doi. org/
htwp:arwvvv.220...... 06588
Nils Reimers. Reddit (title, body) pairs, 2021. URL htwps: hitp: htw.tttpps///ugggggggac.....o sentence-t trattrrnnsorrroo -ttp://eeeeedddiieeeeeee-dddoddd
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks.
In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNE 2019, Hong Kong. China, November 3-7, 2019, pp. 3980-3992 Association for Computational Linguistics, 2019. doi: 10.18653/V1/D19- 1410. URL https::/p//...cc..t. /tt.///o.i.// httgg//....63//1//11.. 9414.
Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen. CARER: contextualized affect representations for emotion recognition. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun't ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Brussels, Belgium, October 31 November 4, 2018, pp. 3687-3697. Association for Computational Linguistics, 2018. doi: 10. 10.18663/V1/118-111 1404. URL https: /oii. .rgg 10. 186538863//0 vl / 8- 1404.
Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, Armand Joulin, and Angela Fan. Cematrix: Mining billions of nigh-quality parallel sentences on the web. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pp. 6490-6500. Association for Computational Linguistics, 2021. doi: 20118653/V1/222 ACL-LONG.507.. URL https: /oi. org / 10. 18 11w53/ http://222....111.11 aal-lcclllonn. 507.
Spurthi Setty, Katherine Jijo, Eden Chung, and Natan Vidra. Improving retrieval for RAG based question answering models on financial documents. CoRR, abs/2404.07211,, 2024. doi: 10.48550/ ARXIV.2404.07221. URL htwps:ttt////. /wwii org/10. 4054444555//11 wwtp:xiii..4..722.
Chih-Chieh Shao, Trois Liu, Yuting Lai, Yiying Tseng, and Sam Tsai. DRCD: a chinese machine reading comprehension dataset. CoRR, abs/1806.00920 2018. URL http: http:///axxxiii aariiv.xxi..////// abs/1800 00920.
Zhihong Shao, Minlie Huang, Jiangtao Wen, Wenfei Xu, and Xiaoyan Zhu. Long and diverse text generation with planning-based hierarchical variational model. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. EMNLP-IJCNLP 2019, Hong Kong. China, November 3-7, 2019, pp. Association for Computational Linguistics, 2019. doi: 10. 10.18653/V 1/D19-11/1 URL https: //doi wrw/orrg.0000 100.//111000 / /D1 1321.
Shivalika Singh, Freddie Vargus, Daniel D'souza, Börje Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura O' MaMooyy Mike Zhang, Ramith Het- tiarachchi, Joseph Wilson, Marina Machado, Luisa Souza Moura, Dominik Krzeminski, Hakimeh Fadaei, Irem Ergün, Ifeoma Okoh, Aisha Alaagib, Oshan Mudannayake, Zaid Alyafeai, Minh Vu Chien, Sebastian Ruder, Surya Guthikonda, Emad A. Alghamdi, Sebastian Gehrmann, Niklas Muennighoff, Max Bartolo, Julia Kreutzer, Ahmet Üstün, Marzich Fadaee, and Sara Hooker. Aya dataset: An open-access collection for multilingual instruction tuning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computationnll Linguistics (Volume 1: Long Papers). ACL 2024, Bangkok, Thailand, August 11-16. 2024, pp. 11521-11567. Association for Computational Linguistics, 2024, doi: 10. 1188633////222/ ACI-LLONG.620. URL https::/ /ow.i org / 10. httgg//...11/5//// 181538653vll1 /202 acl- long. 62.0
19

---

Published as a conference paper at ICLR 2026
Inc. Stack Exchange. Stackexchange (title, body) pairs, 2021. URL Utthhtps:///huggggaaaaa c0/ //datassettt flax- wwweeeeeeeeeeenneeeeeeeeeeeeeeee http:///nnnnssscccccccccccaae wtw.aackeeccaggetilleeeee jsonl.
Saba Sturua, Isabelle Mohr, Mohammad Kalim Akram, Michael Günther, Bo Wang, Markus Krimmel, Feng Wang, Georgios Mastrapas, Andreas Koukounas, Nan Wang, and Han Xiao. jina-embeddings- v3: Multilingual embeddings with task lora. CoRR, 10177, 2024, doi: 10.48550/ARXIV. 1409.1 URL https: doi. wrg/oc.goooooooooooooooo (1555///aXiii 2409. 10173.
Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang. Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. One embedder, any task: iearucttio-finetuned text embeddings. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Associaation for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 1102- 1121. Association for Computational Linguistics, 2023. doi:
71. URL https: doi. org/ 186538663///0 212053/v1/200 aac..//00......iingaaacc........
Maosong Sun, Jingyang Li, Zhipeng Guo, Yu Zhao, Yabin Zheng, Xiance Si, and Zhiyuan Liu. org/
Thucte: An efficient chinese text classifier, 2016. URL
Accessed: 2024-05-01.
Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang. Amrita Bhattacharjee, Man- sooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. Large language models for data annotation and synthesis: A survey. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.). Proceed- ings of the 2024 Conference on Empirical Methods in Natural Language Processing. EMNLP 2024, Miami, FL, USA. November 12-16, 2024, pp. 930-957. Association for Computational Linguistics, 2024. URL https: http://aaaaaatoooooooooooo
org 2224. emnlp- main. 54.
Hongxuan Tang, Hongyu Li, Jing Liu, Yu Hong, Hua Wu, and Haifeng Wang. Dureader_robur A chinese dataset towards evaluating robustness and generalization of machine reading compre- hension in real-world applications. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. ACL/JCNLP 2021, (Volume 2: Short Papers), Virtual Event, August 1-6, 202 I, pp. 955-963. Associa- tion for Computational Linguistics, 2021. doi: 20.18653/V1/00211 ACL-2.....--SSOO.TITI https: //o::///ooiio org. 10. 186.3/6663300 11//2221.acl achssrrr. 120.
URL
short.
Shancheng Tang, Yunyue Bai, and Fuyu Ma. Chinese semantic text similarity trainning dataset, 2016. Accessed: 2024-05-01.
URL /aatts::/githhee com/
Nandan Thakur, Nils Reimers, Andreas Rücklé Abhishek Srivastava, and Iryna Gurevych. BEIR: A heterogeneous benchmark for zero-shot evaluation of informa- tion retrieval models. In Joaquin Vanschoren and Sai-Kit Yeung (eds.). Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. URL https: patess:/ddttaae bttp///ttteeeee-bbeaaaake. preprccdmrrrk-pprrrdiingg.g www.eeeiios..eeeeeeeeeo nwwrips cc/ hash/
wwtperiis...ppppee/eeeeee cc/pappr/202// www.aaatt----stttt--eaeeaeaeeeee Abstract- rownctt-ooond..rr htmll.
Nandan Thakur, Jianmo Ni, Gustavo Hernández Ábrego, John Wieting, Jimmy Lin, and Daniel Cer. Leveraging Ilms for synthesizing training data across many languages in multilingual dense retrieval. In Kevin Duh, Helena Gómez-Adorno, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies (Volume 1: Long Papers), NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pp. 7699-7724, Association for Computational Linguistics, 2024. doi: 1111866311222AMAAAL-LLOOO naacl-long. 4
URL https::/ org/ 111/100.1633/1/222222
James Thorne, Andreas Vlachos, Christos Christodoulopoulos. and Arpit Mittal. FEVER:
large-scale dataset for fact extraction and verification. In Marilyn A. Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL- HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume (Long Papers), pp. 809- 819. Association for Computational Linguistics, 2018. doi: 10.18653/V WINIINNNN-1-111 1074. URL https: / //o.. .org/10 /0. 1tt.//1...11//// 11653/ --10//07..111004....
20

---

Published as a conference paper at ICLR 2026
Ustinian. Law question wnnshooingnn dataset, 2020. URL https:://wwtt htttp:s//www.hyyhalle.coom
dataset http://eeeee333eeeeee10000000000
Henrique Schechter Vera, Sahil Dua, Biao Zhang, Daniel Salz, Ryan Mullins, Sindhu Raghuram Panyam, Sara Smoot, Iftekhar Naim, Joe Zou, Feiyang Chen, Daniel Cer, Alice Lisak, Min Choi, Lucas Gonzalez, Omar Sanseviero, Glenn Cameron, Ian Ballantyne, Kat Black, Kaifeng Chen, Weiyi Wang. Zhe Li, Gus Martins, Jinhyuk Lee, Mark Sherwood, Ju-yeong Ji, Renjie Jingxiao Zheng. Jyotinder Singh, Abheesht Sharma, Divyashree Sreepathihalli, Aashi Jain, Adham Elarabawy, AJ Co, Andreas Doumanoglou. Babak Samari, Ben Hora, Brian Potetz, Dahun Kim, Enrique Alfonseca, Fedor Moiseev, Feng Han, Frank Palma Gomez, Gustavo Hernández Abrego, Hesen Zhang, Hui Hui, Jay Han, Karan Gill, Ke Chen, Koert Chen, Madhuri Shanbhogue, Michael Boratko, Paul Suganthan, Sai Meher Karthik Duddu, Sandeep Mariserla, Setareh Ariafar, Shanfeng Zhang, Shijie Zhang. Simon Baumgartner, Sonam Goenka, Steve Qiu, Tanmaya Dabral, Trevor Walker, Vikram Rao, Waleed Khawaja, Wenlei Zhou, Xiaoqi Ren, Ye Xia, Yichang Chen, Yi-Ting Chen, Zhe Dong, Zhongli Ding, Francesco Visin, Gaël Liu, Jiageng Zhang, Kathleen Kenealy, Michelle Casbon, Ravin Kumar, Thomas Mesnard, Zach Gleicher, Cormac Brick, Olivier Lacombe, Adam Roberts, Qin Yin, Yun-Hsuan Sung. Raphael Hoffmann, Tris Warkentin, Armand Joulin, Tom Duerig, and Mojtaba Seyedhosseini. Embeddinggemma:: Powerful and lightweight text representations. CoRR, abs/2509.203355 2025.
William R. Hersh, Kyle Ellen M. Voorhees, Tasmeer Alam, Steven Bedrick, Dina TREC-COVID: constructing a pandemic Lo, Kirk Roberts, Ian Soboroff, and Lucy Lu Wang. information retrieval test collection. SIGIR Forum, 54(1):1:1-1:12, 2020. doi: 10. 10.1145/3451944. 3451965. URL https: / / /oiiiiiioo.//// org/ 10. 11411 111144/34599444 3451965.
David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. Fact or fiction: Verifying scientific claims. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 7534-7550. 609.
Association for Computational Linguistics, 2020. doi: mia.ppmmainn.mmmmmmmmmmmm 609.
URL https:tt////.c / /tazz///0..00 11w.............00 http://w122222..m.11...mm ma
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Ma- jumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-traiming. CoRR, abs/2212.035333, 2022. doi: (0.48550//RXX1.2222.000 48550/arxis 2212 03533..
URL https: http://ooooooog///////
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improving text embeddings with large language models. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 1 1187-1191 1916. Association for Computational Linguistics, 2024a. doi: 10.18665/112224.A1111LOGCG ht.tps: / /ooi .org/10...... 10. 0653/vvll vl 120022221l12220.. ong 642.
URL.
Liang Wang. Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Multi- lingual E5 text embeddings: A technical report. CoRR, abs/2402.05672. 2024b. doi: 10.48550/ ARX1V.2402.05672. URL https: /doii org/10. 48554 arxiv. . 2402. 05672.
Lucy Lu Wang, Kyle Lo, Yoganand Chandrasekhar, Russell Reas, Jiangjiang Yang, Darrin Eide, Kathryn Funk, Rodney Kinney, Ziyang Liu, William Merrill, Paul Mooney, Dewey A. Murdick, Devvret Rishi, Jerry Sheehan, Zhihong Shen, Brandon Stilson, Alex D. Wade, Kuansan Wang, Chris Wilhelm, Boya Xie. Douglas Raymond, Daniel S. Weld, Oren Etzioni, and Sebastian Kohlmeier. CORD-19: the covid-19 19 open research dataset. CoRR, abs/2004.10706, 2020. URL https://arxis org/ abs// 200//00044110
Adina Williams, Nikita Nangia, and Samuel R. Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Marilyn A. Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018. Volume (Long Papers), pp. 1112-1122. As- doi: 10.18653/VI/N188 1101. URL https:
sociation for Computational Linguistics, 2018.
21

---

URL Published as a conference paper at ICLR 2026
Yuxiang Wu, Yu Zhao, Baotian Hu, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. An efficient memory-augmented transformer for seowledge-intessiig NLP tasks. In EMNLP, pp. 5184-5196. Association for Computational Linguisties, 2022.
Chaojun Xiao, Haoxi Zhong, Zhipeng Guo, Cunchao Tu, Zhiyuan Liu, Maosong Sun, Tianyang Zhang, Xianpei Han, Zhen Hu, Heng Wang, and Jianfeng Xu. CAIL2019-SCM, A dataset of similar case matching legal domain. CoRR, 0ts//9111.89960 2019. URL attp://arae org/abs/ / 911 08962.
Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. C-pack: Packed resources for general chinese embeddings. In Grace Hui Yang, Hongning Wang, Sam Han, Claudia Hauff, Guido Zuccon, and Yi Zhang (eds.), Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024, Washington DC, USA, July 14-18, 2024 pp. 641 649, ACM, 2024. doi: 10.1145/362677723357777 3657878.
URL https:://///mm / / org/ 10.
Xiaohui Xie, Qian Dong, Bingning Wang, Feiyang Lv. Ting Yao, Weinan Gan, Zhijing Wu, Xi- angsheng Li, Haitao Li, Yiqun Liu, and Jin Ma. T2ranking: A large-scale chinese benchmark for passage ranking. In Hsin-Hsi Chen, Wei-Jou (Edward) Duh, Hen-Hsen Huang, Makoto P. Kato, Josiane Mothe, and Barbara Poblete (eds.), Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2023, Taipei, Taiwan, July 23-27, 2023, pp. 2681 2681-2690, ACM, 2023. doi: 00.1145/3539618.339111 https: / doi. wwwgooggo///oo. /0. 114 145 35333333966 5366189668661 61866118111 3591874.
Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, Yin Tian, Qianqian Dong, Weitang Liu, Bo Shi, Yiming Cui, Junyi Li, Jun Zeng, Rongzhao Wang, Weijian Xie, Yanting Li, Yina Patterson, Zuoyu Tian, Yiwen Zhang, He Zhou, Shaoweihua Liu, Zhe Zhao, Qipeng Zhao, Cong Yue, Xinrui Zhang. Zhengliang Yang, Kyle Richardson, and Zhenzhong Lan. CLUE: A chinese language understanding evaluation benchmark. In Donia Scott, Núria Bel, and Chengqing Zong (eds.), Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pp. 47644477777777 International Committee on Computational Linguistics, 2020. doi: 10. (0066////2200.OLLNG-AAAAA coling main. 419.
URL https:/
org/ wwt.ooo..gg.......... 18...11/1111222000
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang. Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang. Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng. Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yun- fei Chu, Yuqiong Liu, Zeyu Cui. Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report. CoRR, abs/2407. 2024. doi: 1044850/ARXXI22407.16666 https: /oii.....////// wrt./0..//100....... 48550/ arxiv. 2407 10671.
URL
Dongjie Yang, Ruifeng Yuan, Yuantao Fan, Yifei Yang. Zili Wang, Shusen Wang, and Hai Zhao. Refgpt: Dialogue generation of gpt. by gpt, and for GPT. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pp. 2511 Association for Computational LiniiaalLiiiiiiiiii Linguistics, 2023. doi: FIW.:W/WI2222FINDINGGEEEEEEEEEE 2023. f wen.innnsnennll.......m 165.
URL https::::/ //opp::://diiieee htt/://....gg.o..1.....
Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. PAWS-X: dataset for paraphrase identification. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pp. 3685-3690. Association for Computational Linguistics, 2019. doi: 10.18653/V 1/D19 1382 URL https: / /ooi. . 10. 18553//1...55/11/ D1 13382.
A cross-lingual adversarial
22

---

Published as a conference paper at ICLR 2026
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question an- swering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 2018. pp. 2369-23800 Association for Computational Linguistics, 2018. URL https: /tw.//iiio/// wtt.:o.../....././.... 18653/11/11111 d18 1259.
doi: 10. /LDI86////IDDII-II
Sha Yuan, Hanyu Zhao, Zhengxiao Du, Ming Ding, Xiao Liu, Yukuo Cen, Xu Zou, Zhilin Yang. and Jie Tang. Wudaocorpora: A super large-scale chinese corpora for pre-training language models. Al Open, 2:65-68, 2021. doi: /111///.A/P.EN.... 1016/j aiopen 2021. 06. 001.
URL https: / //t////d////// //dpi.org/10....../... org/10.
Dun Zhang, Jiacheng Li, Ziyang Zeng, and Fulong Wang. Jasper and stella: distillation of sota embedding models, 2025a. URL https: //axivv//ravvva//aaa/aaaeaa org/ abs 2bss/2 22l...222.... 19048.
Meishan Zhang, Xin Zhang, Xinping Zhao, Shouzheng Huang, Baotian Hu, and Min Zhang. On the role of pretrained language models in pereral- text embeddings: A survey. CoRR, abs/2507.20783, 2025b. URL https: //toi//doii/// org/10 /tt.g//.....///////// (1858///a2ii1. 2507.
Sheng Zhang, Xin Zhang, Hui Wang, Lixiang Guo, and Shanshan Liu. Multi-scale attentive interaction networks for chinese medical question answer selection. IEEE Access, 01774061774077 2018. doi: 10.1009//CCESS.00 2018. 2883637.
201...8263882282333 2883637. URL ht.ps:: httpst:::///iii http:://doo...ggg./ 1109 110./11CC1.1.1.. ACCESS 2018.
Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang. Pengjun Xie, Fei Huang, et al. mgte: Generalized long-content text representation and reranking models for multilingual text retrieval. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pp. 1393-1412, 2024. URL https: //o::///oooii/// org/10 /83866333v20 30/33/11/22/22/ emn.1p inndistrry...... 103.
Xinyu Zhang. Xueguang Ma, Peng Shi, and Jimmy Lin. Mr. tydi: A multi-lingus benchmark for dense retrieval. CoRR, abs/2108.08787 2021. URL https: / arxiv. org/ abs 2108. 08787.
Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermeloo Xiaoguang Li, Qun Liu, Mehdi Rezagholizadeh, and Jimmy Lin. MIRACL: A multilingual retrieval dataset covering 18 diverse languages. Trans. Assoc. Comput. Linguistics, 11:1114-1131 2023. doi: 10.1 10.1162/TACIALA AALACCALALLL0000000 00555. URL https: /oii. org 0 1.1162/aacc___000055555 _005955
Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embedding: Advanc- ing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176, 2025c. URL https: doi. org/ 10. 4 48550/ arxiv 2506. 05176.
Xinping Zhao, Dongfang Li, Yan Zhong, Boren Hu, Yibin Chen, Baotian Hu, and Min Zhang. SEER: self-aligned evidence extraction for retrieval-augmennted generation. In EMNLP, pp. 3027-3041. Association for Computational Linguistics, 2024. URL https: //do:i////i... orciiorrg/e wtt..//gg............ 2024. ewnlp-main 178.
Xinping Zhao, Yan Zhong, Zetian Sun, Xinshuo Hu, Zhenyu Liu, Dongfang Li, Baotian Hu, and Min Zhang. Funnelrag: A coarse-to-fine progressive retrieval paradigm for RAG. In NAACL (Findings), pp. 3029-3046. Association for Computational Linguistics, 2025. URL https: findings naacl. 166.
//doi. org 10. 11w53/
Zhen Zhao, Yuqiu Liu, Gang Zhang, Liang Tang, and Xiaolin Hu. The winning solution to the iflytek challenge 2021 cultivated land extraction from high-resolution remote sensing image. CoRR, 109/2 2022. URL https: wwwaaa.vvvvo......... www.or////////////////ooo/eaeee/ /2202. 10974.
Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. Opencodeimterpreter:: Integrating code generation with execution and refinement, In ACL (Findings), pp. 12834-12859 Association for Computational Linguistics, 2024.
23

---

Published as a conference paper at ICLR 2026
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettle- moyer, and Omer Levy. LIMA: less is more for alignment. In Alice Oh, Tristan Nau- mann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances Neural Informa-
in Neural Information Processing Systems 36: Annual Conference tion Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http:/ http::///ppper.eee nips pc/paper__ieeeee hwwplleeeepapeeeeeeeeee pttpp//ppppp///////e/////// hash/ html.
Wei Zhu. Chatmed-dattset: An gpt generated medical query-response datasets for medcial large language models, 2023, URL https: / /ggthtb.th.c../ www.ehhw....omiiioooi.oo..o wwwoo/mmccaaii http:///w..ccaaaaaaaaaaahahaaaaa wwhh/Chatted.
24

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS
In this work, we utilized LLMs solely for the purpose of polishing writing, The LLMs were not used for content generation, and all research, analysis, and conclusions presented are the result of our own work and independent thought.
B EXPERIMENTAL DETAILS
Implementation Details. We adopt InfoNCE loss (Gutmann Hyvärinen, 2010) and KL-divergence loss (Hinton et al., 2015) as training objectives, with temperature coefficients set to 0.01 and 0.05, respectively. Qwen2-0.5 (Yang al., 2024) serves as the base decoder-only LLM backbone. combined with a simple yet effective mean pooling. To enable fully bidirectional modeling, we remove the causal attention mask from the decoder-only L.M.. The embedding dimension is 896, with maximum input length of 512 tokens. The model is fully fine-tuned with all parameters updated, using mixed precision with Biloat 16, Matryoshka Representation Learning (MRL) (Kusupati et al., 2022) is applied to both InfoNCE and KL-divergence losses with embedding dimensions of 896, 512, 256, 128, and 64, weighted by 1.0, 0.3, 0.2, 0.1, and 0.1, respectively. The model is optimized by the Adam optimizer (Kingma & Ba, 2015).
Based on the above common configurations, we detail the settings for each training stage. (1) Pre- training: We exclusively use in-batch negatives for training efficiency. Pre-training is conducted on 6 nodes (8 GPUs each) for epoch, corresponding to approximately 19k steps, with a per-GPU batch size of 512 and learning rate of le-4. (2) Fine-tuning: We incorporate hard negatives by sampling M = 7 examples from ranks 50 to 100 within the candidate pool. Training is conducted for I epoch, approximately 12k steps, with a per- GPU batch size of 120 and learning rate of 2e-5. The focusing parameter 7 in Equation 4 is set to 0.5. For each sample, pair-wise and a fist-wise hard negative is mined. Fine-tuning performed on GPUs, requiring approximately 220 GPU hours for epoch. (3) Contrastive distillation: The model is jointly optimized with contrastive and KL-divergence losses, weighted at 0.3 and 0.7, respectively. Qwen3- Eennndd-mdddingg-- -BB (Zhang et al., 2025c) is used as the teacher model, where teacher embeddings for all training samples pre-computed and cached to accelerate training. Training is run for epoch, approximately 24k steps, with a per-GPU batch size of 120 and learning rate of 1e-5. Distillation is performed on just 2 GPUs, requiring about 280 GPU hours for epoch. The detailed hyperparameter settings adopted in the experiments are presented in Table 9.
Table 9: Hyperparameters used in the experiments. For batch size, training steps, learning rate, and so on, the three values correspond pre-training, fine-tuning, and contrastive distillation, respectively.
Parameter
Value
Batch size (per GPU)
512/120/120
GPU used
48/4/2
Training Steps
19k/12kk24k
Training Data Size
470M/6M/6M
Warm-up steps
10% 200 200
Learning Rate
le-4/2e-5/le-5
Epochs
(all stages)
Base model
Qwen2-0.5 (bidirectional) Mean pooling 896
Pooling strategy
Embedding dimension
Maximum Input Length
512
MRL Dimensions
896, 512, 256, 128, 64
MRL Weights
1.0, 0.3, 0.2, 0.1, 0.1 0.5
Focusing Parameter
Hard negatives
M = 7, ranks 50-100 Adam Bfloat16
Optimizer
Precision
Contrastive Learning- 0.01
Temperature Coefficients
Contrastive Distillation 0.05
Teacher model
Qwen3-Ennee Saann--Emeeddiig-33
25

---

(Cohere, Published as a conference paper at ICLR 2026
Baselines. We compare the KaLY-Embedcing--V series with the following compet- itive general-purpose and multilingual open-source text embedding models and commer- cial embedding API services. The open-source models include: paraphrase-multiingae (ML.)-mpnet-base-22 (Reimers & Gurevych, 2019), jin-eembedding--33 (Sturua et al., 2024), Qwenn3-Emmeddinng hatp:/dnnnnnn//ooeeeeennnennnnee (Zhang et al., 2025c; Li et al., 2023; Zhang
(Chen al., 2024; Xiao al., 2024), EmbeddingGemma- 300M (Vera et al., 2025), mulilinggull-e--lan
al., 2024),
instruct (Wang et al., 2024b; 2022), GRITLM 8x7B (Muennighoff al., 2024) sparse mixture-of- experts embedding model with 13B active parameters during inference), NV-Embed-v2 (Lee et al., 2025a), and Gal.M-Embedding-VV (Hu et al., 2025). The commercial embedding services include rext-embedding---large (OpenAl, 2024) from OpenAl and Cohere-embed-mulliinggan 2023).
Evaluation. We evaluate the KaLM-Emmeedeee Embedding V2 series and the competitive baseline embedding models on MTEB (Muennighoff et al., 2023a; Xiao et al., 2024) for both Chinese (cmn) and English (eng). For Chinese, use MTEB (cmn v1), derived from C-MTEB (Xiao al., 2024), which comprises 35 tasks across task types. For English, we adopt MTEB (eng v1) (Muennighoff al., 2023a), covering 56 tasks across 7 task types, providing broader evaluation scope than v2, which contains only 41 tasks across the same number of task types. Following the MTEB (cmn, v1) leaderboard, we exclude from the classification task, as well as STS22 from the STS task, resulting in 31 tasks. This setup slightly differs from the original C-MTEB (Xiao et al., 2024). For evaluation, we evaluate our Ka LM-MMEMT LLLM-Mmbedding---- ensure fair comparison with previous works. For models without officially reported results on the MTEB leaderboards, we evaluate them using the task instructions summarized in Table 18 to ensure fair comparison. We also provide results on MTEB (eng, v2) (Enevoldsen al., 2025) Table 12.
saazzonReeiewsCassficttti
MassivennennCCassficatiee and
VlasiveScenarioClasiiii.e
series using maximum length of 512 tokens
Table 10: OOD Evaluation on real-world industrial scenarios. Recall K measures whether the positive item appears in the top-K retrieved items. MRR @ K denotes mean reciprocal rank and further measures the ranking quality. It reciprocally discounts the position.
Customer Service EAQ Retrieval
Model
Size
MRR@1 1
MRR Ø
MRR @ 10
Recall @
Recall@5
Recall @ 10
Swen3-Embeddune
7.57B
44.49
57.79
58,91
44.49
78.44
86.69
Owen3-Embedding-O..ff
596M
40.36
53.60
54.61
40.36
75.22
82.56
bge-m3 (Dense)
560M
34.40
46.68
48.19
34.40
68.80
79.81
gcc-muttilinuual-be base (Dense)
305M
39.90
50.44
51.47
39.90
67.43
75.68
KaL BoLLM-Emeddding- V2.5
494M
45.87
56.96
58.05
45.87
77.06
85.32
Game Documentation
Search
8B
7.57B
23.61
35.64
37.52
23.61
56.55
70.45
Fwen3-EEmeedding-O..m
596M
20.70
31.40
33.14
20.70
50.23
63.28
bge-m3 (Dense)
560M
20.02
30.62
32.47
20.02
49.04
62.70
gtc-multilingual base (Dense)
305M
18.10
27.50
29.02
18.10
43.86
55.14
KaLM-Embeddee V2.5
494M
23.82
36.36
38.24
23.82
58.23
72.22
C OUT-OF-DOMAIN GENERATION
To comprehensively assess robustness and generalization in real- industrial applications, we conducted out-of-domain (OOD) evaluations in two Chinese retrieval scenarios, with sizes ranging from thousands to tens of thousands. The first involves customer service FAQ retrieval, where all queries originate from real user interactions, with relevance labels manually annotated by human experts. The second targets game documentation search in a vertical domain, utilizing real user- generated queries; relevant documents were filtered and selected based user click-through data. None of the models has been trained on these datasets, ensuring genuine OOD evaluation. We choose embedding models widely used in industries from GTE and BGE baselines. From the results shown Table 10, KaLM- Ee..ed-iiddiing--.. V2.5 achieves SOTA performance compared to models of comparable size. Furthermore, despite being 15 times smaller in size than Owen3-Embedden
26

---

Published as a conference paper at ICLR 2026
Table 11: Matryoshka emhedding performance, where 'Full' denotes the maximum dimension, specifically 896 for the KaLM-Embedding series.
MTEB (eng, vl)
Model
Dim
MTK
MTY Class.
Clust.
PairCL
Reran.
Retri.
STS Summ.
Full
69.33
65.83
88.34 56.59
86.60
57.84
55.00 85.27 31.18
512
69.13 (-0.288%
65.65
88.35 56.52
86.53
57.76
54.44
85.32
30.65
KaL.M-Embedding-VW. V2.5
256
68.80 (-0.764%)
65.43
88.29
56.37
86.35
57.45
53.47
85.12
30.95
128
68.05 (-1. -1)8446)
64.95
88.14
56.29
85.83
56.64
51.25
84.95
31.57
64
66.44 (4.16)%)
63.63
87.87
56.06
84.96
56.04
46.63
84.13
29.71
Full
69.36
65.86
88.55
56.18
86.86
57.86
55.14
85.36
31.07
512
69.02 (-0.490%)
65.71
88.71
56.66
86.92
58.13
53.67
84.77
31.11
Mal.M-Embedding- V2.5
256
68.40 (-1.384%)
65.34
88.69
56.63
86.58
57.64
52.10
83.93
31.80
(w/o MKT.)
128
67.36 (-2.884%)
64.40
88.61
56.45
85.59
56.61
49.40
83.29
30.84
64
65.36 (-5.767%)
63.01
88.36
56.00
84.39
55.84
43.76
82.05
30.68
Full
67.47
64.14
87.19
56.05
86.18
56.74
51.67
82.61
28.51
512
67.23 (-0.356%)
63.98
87.14
56.04
86.11
56.49
50.90
82.62
28.57
KaLM-Embedding-- V2
256 66.76 (-1.052%)
63.76
87.18
56.03
85.83
56.09
49.55
82.19
29.43
128 65.65 (-2.697%)
62.83
86.98
55.80
84.94
55.09
46.39
81.92
28.67
64 63.73 (-5.543%)
61.56
86.72
55.53
83.63
54.21
40.83
80.79
29.19
Full
64.94
61.49
84.74
47.82
83.26
55.41
51.65
82.24
25.23
512 64.48 (-0.708%
61.14
84.60
47.49
82.92
54.72
50.74
81.90
25.61
KaL.M-Emnediing-Y V1
256 63.85 (-1.678%)
60.85
84.29
47.21
82.74
53.94
49.01
81.90
26.89
128 62.13 13 (-4.327%)
59.35
83.71
46.44
81.09
52.05
44.83
81.40
25.96
59.69
(-8.084%)
57.71
82.68
45.49
78.54
50.41
38.61
80.60
27.64
MTEB (emn, v1)
Full
70.93
72.46
77.48
73.09
84.09
66.90
73.42
59.80
512
70.80 (-0.183%) 72.36
77.48
73.07
84.05
66.83
72.96
59.79
Kal.M-Emmedding-V V2.5
256
70.43 (-0.705%)
72.09
77.38
73.06
84.21
66.20
71.94
59.73
128
69.76 (-1.650%)
71.62
77.38
73.37
84.05
65.68
69.60
59.61
64
68.10 (-3.990%)
70.32
76.98
73.17
83,95
63.60
65.06
59.13
Full
70.91
72.46
77.44
72.80
84.53
66.74
73.45
59.79
512
70.45 (-0.649%) 71.84
77.73
72.26
82.38
66.59
72.96 59.12
Kal.M---mbedding V2.5
256
69.89 (-1.438%)
71.38
77.67
72.25
82.21
65.80
71.65 58.67
(w'o MKL)
128
68.75 (-3.046%)
70.36
77.50
72.03
81.25
64.30
68.98
58.08
64
66.89 (-5.669%)
68.91
77.17
71.83
80.48
63.01
63.80
57.14
Full
68.15
69.28
75.14
69.76
77.91
65.16
72.15 55.58
512
67.85 (-0.440%)
69.01
75.04
69.35
77.64
65.09
71.46 55.50
KaLMM-Emeedinng-- V2
256
67.37 (-1.145%)
68.64
74.96
69.32
77.77
64.80
69.65 55.31
128
66.38 (-2.597%)
67.88
74.85
69.41
76.93
64.15
66.92
55.02
64
64.13 (-5.899%)
66.14
74.62
69.35
76.33
61.99
60.43
54.12
Full
63.78
64.56
73.89
57.54
72.94
64.48
70.12
48.41
512
63.39 (-0.611 %)
64.18
73.58
57.26
72.54
63.98
69.39
48.35
K-l.M--mmedding-- V1
62.82 (-1.505%
63.77
73.71
57.20
72.56
63.50
67.50 48.17
128
61.59 (-3.434%)
62.75
73.51
57.52
71.62
62.08
63.97 47.82
58.98 (-7.526%) 60.74
72.85
56.58
71.27
60.22
56.72 46.82
8B, KaL.M-EEmeddingg- -2.5 still outperforms it in 8/12 cases. These results demonstrate that KaLM I-M-Embeddi V2-V2 models not only achieve state-of-the-an performance on MTEB, but also exhibit strong generalization and robustness in wolld industrial applications.
D MATRYOSHKA EMBEDDING
To enable dinieiio-i embeddings, incorporate MRL into both contrastive and KL loss. Unlike previous works, we also optimize matryoshka embeddings using the matryoshka KL. objective, referred to MKL'. To verify the effectiveness of matryoshka embeddings and MKL,
Note that the teacher model still computes soft signals using the full dimension.
27

---

Published as a conference paper at ICLR 2026
(a) BaL.M-Embedding-YY VI.
(b) GGLMM-Emeddee V2.5.
Probercch lrgakers
(c)
Figure 5: Comparison of discriminative capacity between positive and hard negatives. Cases are randomly sampled from the HotpotQA dataset, where the task instruction "Instruct: Given query, retrieve documents that answer the query. in Query: (query }".
conduct dimensionality reduction experiments along with MKL ablation studies, as shown in Table 11. From the results, we mainly have the following observations. Firstly, for tasks such as Class., Clust., PairCI., STS, and Summ., performance degrades only slightly when using matryoshka embeddings of smaller sizes, whereas tasks like Reran. and Retri. exhibit more substantial drops. This indicates that semantic matching tasks (e.g., Class., Clust., and PairCI.) can be effectively handled even with low-dimensional matryoshka embeddings, whereas retrieval and reranking tasks demand higher-dimensional embeddings preserve performance. Secondly, compared with KaLM- Embedding V2.5 (w/o MKL), V2, and VI, KaLM- Embedding- V2.5 demonstrates consistently smaller performance degradation as embedding dimensionality decreases. For example, on MTEB (cmn, v1), the performance drop from full dimension to 64 dimensions is only -3.99% for K-L Embedding- V2.5, compared to -5.67% for its counterpart without MKL. We find that the superior robustness of Kal.M-Embeddi V22. using matryoshka embeddings of smaller sizes mainly stems from its relatively smaller performance degradation on Reran. and Retri. tasks compared to others. These results show that MKL makes under small embedding dimensions. Thirdly, retrieval tasks exhibit the largest performance drops as embedding dimensions decrease, showing they rely heavily on high- eigh-dimensiona embedding. This also explains why small, low-dimensional embedding models lag behind larger, high-dimensional ones on retrieval tasks, illustrated in Table 4. Overall, these results indicate that matryoshka embeddings provide flexible, compact representations that maintain strong performance on semantic matching tasks, while retrieval and reranking tasks benefit from higher-dimensional embeddings.
V2.5 more robust, with smaller drops
E
CASE STUDY
To provide more intuitive and qualitative understanding of our model's discriminative capacity, we conduct a case study on randomly sampled examples from the HotpotQA, a representative retrieval dataset. For each case, we compute similarity scores between a query, its ground-truth positive, and 7 hard negatives. To visualize the results, the score between the query and the positive is plotted as single point, i.e., the red star. The seven scores between the query and the hard negatives are used to generate a box plot. An ideal embedding model should assign a significantly higher score to the positive compared to all hard negatives, placing the red star well above the corresponding box plot. This visualization provides a clear comparison of how effectively each model can distinguish the positive passages from hard negative ones. From the results shown in Figure 5,
28

---

Published as a conference paper at ICLR 2026
Table 12: Detailed embedding model performance MTEB (eng, v2) (Enevoldsen et al., 2025).
MTEB (eng, v2)
Model
Size
MTK
MTY Class.
Clust.
PairCl. Reran. Retri.
STS
Summ.
-B
8B
75.22
68.70
90.43
58.57
87.52
51.56
69.44
88.58
34.83
NV-Embed-v2
69.81
65.00
87.19
47.66
88.69
49.61
62.84
83.82
35.21
Bren3--mbedding-B3
4B
74.60
68.09
89.84
57.51
87.01
50.76
68.46
88.72
34.39
gte- -w-----..5---istmeeee
1.5B
67.20
63.26
85.84
53.54
87.52
49.25
50.25
82.51
33.94
0.6B 0.6B
596M
70.70
64.88
85.76
54.05
84.37
48.18
61.83
86.57
33.43
multilinguual-C0 large-instruct
560M
65.53
61.21
75.54
49.89
86.24
48.74
53,47
84.72
29.89
bge-large-en-v1.5
335M
65.89
61.87
78.34
48.01
87.13
48.26
55,44
82.79
33.13
EmbeddingGGemma 300M
307M
69.67
65.11
87.55
56.55
87.29
47.43
55.69
83.61
37.64
KaLM EeeM--Emedding-- V2.5
494M
71.29
65.31
90.50
58.12
86.63
47.42
58.45
84.82
31.21
Table 13: Detailed AIR-Bench QA results (NDCG@ 10 scores) on AIR benchmark 24.05, across seven languages.
RI-BBench QA
Model
Size
MTK
en
zh
es
fr
ja
de
ru
bge-multiinguullgeemna.
9B
51.77
46.25
49.34
60.76
49.69
60.02
49.77
54.97
gte-Qwen2-7B-ineee innstruuttt
7B
49.33
51.87
47.12
55.18
43.04
54.76
44.91
52.65
gte-Qwen2-1e 1.55-inntrrct
1.5B
45.72
48.03
43.13
50.26
40.37
50.04
41.25
50.73
jina-emeeddings-v3 (Multi-LoRA)
572M
45.97
45.07
44.76
52.19
39.94
50.11
43.62
51.70
multilinguale5--aagee
560M
44.54
43.91
43.60
50.84
35.94
52.84
41.93
50.44
hge-m3 (Dense)
560M
49.30
48.78
47.45
53.73
44.66
54.23
46.71
54.55
--M----deddin--222 V2.5
494M
49.02
49.86
48.69
54.43
43.05
52.80
46.00
52.43
we observe that KaLM-Embedding-V22 demonstrates the superior discriminative capacity in all cases, while KaLM-Embedding-V and Swenn-Embedding-O..e cases. Besides, the distance between the red star and the median (the green line) of the box plot for SaLLMMEEmbeddigg V2.5 is consistently larger than the corresponding distance for both KaLM- Embedding- V1 and Soen3-Embedding their hard negative scores is close the positive, meaning their limited ability to distinguish subtle yet critical differences. The large and consistent margin maintained by KaLM-EEmeeddie V2.5 demonstrates the effectiveness its improved training techniques, especially the Focal-style Reweighting Mechanism, which focuses on learning hard samples and leads to the large margin observed in the visualization. conclusion, the qualitative results provide intuitive evidence that aligns with high quantitative benchmark performance, solidifying the model's effectiveness.
perform poorly in the 1st and 3rd
in most cases. This indicates that the distribution of
F VISUALIZATION ANALYSIS
To better understand the relationship between embedding quality and downstream task performance, conduct visualization analysis of different models clustering and classification datasets, covering intent recognition, category identification, and topic classification, with both English and Chinese data included. As shown in Figure 6, we project embeddings into 2D by UMAP (Uniform Manifold Approximation and Projection), with colors indicating the corresponding labels of the data points. From the results, the embeddings produced by Kmbb--mbedding- V2.5 exhibit more compact and separated clusters compared to KaLM Bee.M--Emmddddii-y.d V1 and In the ReddiiClustering and CL.SClusteringPP2P semantically similar samples are tightly grouped under V2.5, while inter-class boundaries become more distinct, aligning with its superior clustering performance. contrast, Qwen3-Embedding -066 displays overlapping regions between categories, suggesting a weaker capability in modeling fine-grained semantic distinctions. The results of the Banking77Classificatee further confirm this conclusion. WeLLM---meedingg.. V2.5 forms separated clusters, whereas V1 and -rreendimmddiingg.. 0.6B embeddings remain entangled. Overall, the improved intra-class compactness and inter-class separability of BoLLM-Embedddingg V2.5 provide strong support for its superior results these tasks.
-..6..
29

---

Published as a conference paper at ICLR 2026
G MULTILINGUAL EVALUATION
To evaluate multilingual and OOD generalization, we adopt AIR-Bench QA (Chen et al., 2025b) (version of 24.05), which provides more comprehensive test than static benchmarks. AIR-Bench is automatically generated to avoid data leakage, spans diverse tasks, domains, and languages. Table 13 shows the evaluation results AIR-Benen Bench QA, including seven languages: en (English), zh (Chinese), (Spanish), fr (French), ja (Japanese), de (German), and ru (Russian). KaLM- E-bbedinddd V2.5 evaluated using general retrieval instruction: "Insiruct: Given query, retrieve documents that answer the query: VI Query: (query)" Despite not being trained on large-scale multilingual corpora, Kal Jal.M-Embedding V2.5 demonstrates competitive performance across all seven languages. It performs on par with or close substantially larger 7B-9B models. For example, its average score (49.02) is nearly identical to that of the much larger Bec-Qwen2-7B-img lower-resource languages, its performance is comparable strong multilingual embedding baselines, such as bge-m3. These results demonstrate that
model (49.33). In
V2.5 generalizes well beyond its primary English-Chinese training focus, exhibiting robust retrieval performance across wide range of multilingual and low-resource language settings.
H FULL MTEB RESULTS
Table 14 and Table 15 show the full METB results for each dataset.
I DATASETS AND INSTRUCTIONS
Table 16 and Table 17 show the detailed dataset list used for pre-training, and fine-tuning as well as distillation, respectively. Table 18 presents the task instructions used in the MTEB evaluation.
30

---

Published as a conference paper at ICLR 2026
KO1 Eniboeddn VI
Onand Enteodiing 0.65
15
5-
-10
10
15
Label
AFL.bt
Attanta bet
China tet Chratianity
amazzn not artmmuttt
sustrall.cst actettonctos
balogyaet
concerbe
dassicalmusc net
Advice bdt
Austin ba
trancertstt
canuit
comebe
Anurchiambet
Autocitet
Coffee.eet
apple bo archiasturr..ot
bartimonebe bd
booka.:st
cabuist
collegoento be
Antralsset
Seoging.tst
advertbiig bet
besionate
chembbryled
collegecct
Ailad.
buorggge
unsall Lo
wwwroheggyttt athe ser
trasibeet
Chocuial
comeddy un
buodhhmmta
amsteumacioooo:
bicycling.ot
conadaaet
Checago bo
comkloookss
Actronomyyit
Comnolctio.ztt
(a) ReddicClustering, where the task instruction is "Instruct: Identify the topic theme of Reddit posts based on the titles Query: [query }"
15
10
10
12
10
Label
Saiwen
Schectied
Bend
2:9 Stremmi
888 Minorol
Outural Scieme)
(b) CUSClusteringP22 where the task instruction is "Instruct: Tdentify the main category of scholar papers hased the titles and abstracts Query: (query]".
Kast 1nt.hsennteeding
Kell M.Ertteeddiigp.3..
Queen Dewr33--tmbooddn---00
20
20
15.
10-
10
-5
20
-15
-10
5
10
15
20
25
20
25
J000
(c) Banking77Claassfcctton. where the task instruction "Instruct: Given online banking query, find the corresponding intents Query: [query]". J".
Figure 6: Embedding distribution comparisons between KaLM- BaLL--Emeeddin... V1, KaL EeLMddiibddiigg.. V2.5, and Diven--Embedding
31

---

Published as a conference paper at ICLR 2026
Table 14: Results for each dataset MTEB (eng, v1). 'Emb' is the abbreviation of 'Embedding'
Dataset
KaLM-Emb- KaLM-Emb-v2 KaLM-Bmb-V2.
Amaazo@Counerrfatuuaalaasaiiiiia
91.73
95.25
94.75
AmazonPolarity Classiitttaalaiiiaaatiooooo
96.56
96.67
97.03
61.42
57.89
64.15
Bunking77Classiffican
84.54
89.48
90.31
EmnotionClassifiaation
86.90
92.50
83.80
Classification
ImdbClassificttion
94.93
95.16
95.91
72.52
77.80
83.24
MassiveSccearrioCllsfffiii
79.32
86.00
89.35
97.54
98.86
98.69
MrOPIntennCCassiicctiioo
85.76
88.77
91.10
ToxiCConversattonsSlaaiiicccaaa
89.28
89.34
91.70
76.35
78.60
80.08
ArxivChlsteringgPPP
49.68
51.16
52.11
ArxivClusteringSSS
42.21
43.70
45.10
Biorxiv
BiorxisClusteringPPPP
43.84
47.69
48.51
BiorxisClusteringSSS
37.31
41.93
42.75
MedraivClusteringPPP
39.91
43.72
43.09
Clustering
MedrxisClusteringS22
36.79
40.56
40.43
RedditClustering
55.47
76.52
76.89
CeadiiCCusteringgPPP
65.96
73.05
72.84
StuckFExcangeeClusterngg
66.38
78.40
80.22
Stack
StucAFcchangeeClusseiinngg.
39.19
45.41
47.26
TwentyyNewsgroussstuueeeeng
49.33
74.44
73.26
SprintDappicateeuuusiinss
92.65
95.88
96.00
Pair Classification
TwitterScmEva/2015
71.44
76.72
77.15
TwitterUUI.Corpus
85.69
85.95
86.66
Ask AskUbunnuuuuQuestiin
60.35
62.13
62.39
MindSmaallReranknng
31.92
32.04
32.45
Reranking
SciDoesRR
80.99
82.25
84.68
StekOOvrrfowwuuyuussttts
48.38
50.54
51.82
ArguAna
58.63
57.42
60.15
ClimateFEVER
25.85
25.07
34.50
CQADupstack
41.83
44.19
47.20
DBPedia
38.99
40,26
42.62
FEVER
86.54
83,00
87.89
FiQA2018
44.74
45.23
47.10
HotpotQA
67.58
70.14
71.76
Retrieval
MSMARCO
34.59
36.20
40.62
NFCorpus
35.33
35.17
37.11
NQ
47.50
48.10
58.61
QuoraRetrieval
87.47
89.81
89.57
SCIDOCS
19,97
20.81
21.62
SciFact
72.89
71.98
74.38
TRECCOVID
83.72
79.27
82.98
Touche2020
29.15
28.43
28.93
BIOSSES
86. 14
84.16
84.02
SICK-R R
79.73
79.85
83.20
STS12
80.17
82.27
81.90
STS13
83.86
85.96
89.52
STS14
80.57
83,50
85,99
STS15
87.34
86,44
90.33
STS16
84.83
85.70
87.74
STS17
86.16
92.34
STS22
69.21
66.95
68.76
STSBenchmark
84.12
85.07
88.88
Summarization SummEval
25.23
28.51
31.18
Mean (Task)
64,94
67.47
69.33
Mean (Type)
61.49
64.14
65.83
32

---

Published as a conference paper at ICLR 2026
Table 15: Results for each dataset on MTEB (cmn, v1).
Dataset
KaLM- 2MMM-1
RaLM Enb-V2
KaLM-Emb-V22
IFlyTck
48.54
51.01
56.59
JDReview
83.02
86.87
88.82
MultiingualSentiment
78.25
79.16
81.26
Classification
OnlineShopping
93.08
94.40
95.02
TNews
51.59
50.75
53.27
Waimai
88.85
88.67
89.91
CLSClustering
46.92
62.95
66.25
CLSClusteringSS2
44.67
59.44
62.73
Clustering
ThuNewsChhsstringgP2
72.87
80.79
84.64
ThuNewsCCusteringSS5
65.68
75.87
78.75
Cmnli
76.67
78.08
86.07
Pair Classification
Ocnli
69.22
77.73
82.12
CMedQAVI-reraniinn
82.34
83.65
84.58
CMeiaAAV2rerraninn
83.12
84.25
85.78
Reranking
MMarroReranking
25.75
26.04
29.64
12Reranking
66.73
66.69
67.60
CmedgaRetrieval
42.12
44.81
45.87
CovidRetrieval
82.40
83,30
83.57
DuRetrieval
82.19
83.17
86.14
EcomRetrieval
62.56
65.10
66.68
Retrieval
MedicalRetrieval
56.89
59.81
60.46
MMrrcoRetricval
78.96
80.59
82.23
T2Retrieval
84.06
84,88
85.97
VideoRetrieval
71.82
75.51
76.44
AFQMC
38.02
44.18
48.78
ATEC
46.19
49.75
52.45
BQ
54.48
61.22
69.74
STS
LCQMC
70.81
73.83
77.50
PAWSX
16.32
43.38
47.90
QBQTC
35.28
37.61
39.83
STSB
77.80
79.10
82.38
Mean (Task)
63.78
68.15
70.93
Mean (Type)
64.56
69.28
72.46
Table 16: Pre-training data list.
Source
Language
Pairs
Amazon-Reviews Reviews (Hou al., 2024)
Amazon-
multilingual
23M
CC-News (Hamborg et al., 2017)
multilingual
100M
NLLB (Costa-jussà et al., 2022; Hefferman et al., 2022; Schwenk et al., 2021)
multilingual
2M
Wikipedia (Foundation, 2024)
multilingual
100M
XP3 (Muennighoff al., 2023b)
muitilingual
19M
XL-Sum (Hasan al., 2021)
multilingual
SWIM-IR (Monolingual) (Thakur et al., 2024) SWIM-IR (Cross-lingual) (Thakur et al., 2024)
multilingual
3M
multilingual
15M
CSL (Li et al., 2022)
zh
0.4M
Wudao (Yuan et al., 2021)
zh
44M
THUCNews (Sun et al., 2016)
zh
0.8M
Zhihu-KOL
zh
0.8M
CodeSearchNNt (Husain et al., 2019)
en
1M
PAQ (Lewis et al., 2021)
9M
Reddit
en
100M
StackExchange
en
14M
S2ORC
en
41M
33

---

Published as a conference paper at ICLR 2026
Table 17: Fine-tuning data list.
Swarce
Type
Cites
Lagssse
Pstrs
PaireBiterd)
Codereatbeck 22lan 20.4)
Remeeaal
s4p
50000
22000
ELIS (hu ct... 2019)
Reticval
s2p
100000
75408
ExpenQA (Malargg et 2004)
Retieval
s2p
1261
1252
Gasaa) (Khahalk e 20213
Rethoval
<2p
50000
49533
MEDNERGE (ifermngaotf ei 2004; Sa 2023)
Retneval
x2p
100000
71700
Opertrea (Maakazie stal, 2023)
Retizval
s2p
40000
38623
RAQ cLews ei.. 2021)
Reticcall
s2p
50000
4954)
PulMellQ (inn dt.
Retieeal
s2p
50000
79054
SearMQ (Dous et 2017)
Retieva
<2p
1000
5995
ande go
Retieal
<2p
23397
17927
CT-Ness [Banbirgg et al. 7017)
Retinaal
<pp
10000
79700
cerp-coonD./errooccaa..... M030 Wangeral, at. 1000)
Ratiral
<p
50050
Dippddt-Fiaitty (Thaceretsl. 2001)
Ranixal
<p
100000
ESCI etal.. 1022)
Ranosal
<2
10000
FEVER (Thsen et 2018)
Retesal
<2p
67855
87246
RQA (Maia 2013)
Retiesal
1Ap
5400
HogooQA (Yass ct..
Retiesal
x4p
184057
150153
MLDRR (Class 20224)
Retiesal
x2p
41434
MSMARCO (t al. 2016)
Retiesal
x2p
175133
174120
MEMARCO-22 (0pARC00022pp et 2010)
Retieea
x2p
277144
NiToges (Botri 201.
Reteesa
<2p
11674
1100
<7p
<7p
SQUAD [loguetar 2016)
stsl. 2001)
Natua (Keizkonski 2019)
Retizval
S9622
Yakco
Retiival
30000
21724
CQADisStad (Hoogevezs 2015)
Retieval
24045
7356
ComnocNLI (Koceda Muniing 2021)
STS
3195
MukiNLI (Willams 20000
STS
66674
65701
NELB (Comm-jess al. 2002: Hefferann al. 2007)
STS
16000
Quera (Dura 2)
STS
09674
Wiecknssors (Fokkr at.l., 201-4)
STS
50020
SinCSE NEI (Gan al, 2021)
STS
12a
251597
217009
(Boorman al. 20151
STS
12x
24655
16400
12. pb
15000
14529
Bionay
Clasfiatiio
pb
6562
6767
Modeiss
Caedicaioon
2012
1945
Reddi Canteing (Geogl 2021)
Chelizztoe
x2x
12300)
21400
Reddi Cantemas Otrimass 2011)
Caedizaioe
p2
1230495
42440
Stacknchang Clasrnng (Grigle et 2023)
x2a
1014825
505.00
Stackeshange Clastrrap P2P (Suck Exchange. 2021)
Cassiaaiio
25333327
45600
Tweaty Clastring (Larg, 1995)
Clasfiatioon
923
11314
0233
Amnomhilziit (Mcluley & Lolkooo, 2013)
Clasiiatioon
+25
10000
IMDB (Miss et 2011
Clasilatioon
10000
5355
barking77 (Casarzesa 2039)
Clastlationn
62s
10020
RinoiitClesillattinn (Sarwis ol.
Clastlation
62s
10000
10000
Chelliatio
67x
10000
10000
Chelceaio
+2
7016
7500
Adhettstaan (it 2019)
Ranosal
<p
2h
20000
175.55
CHEF (R) et 2062)
Reriesa
<2p
4952
4024
ChalMed Darase (2ho, 2003)
Reriesal
<7p
25
20000
CMRC 2015 (Cai ml 7019)
Reninal
<7p
26
10000
0753
(Sto) al. 2013)
Rrnosal
<7p
28
5000
4714
LCSTS (T al, 2015)
Ranosal
<tp
28
20000
10555
LIMA (2haa at. 20235
Ranzval
<tp
28
2008
1991
Mals-CHi (Lang et al. 2012)
Retizval
<2p
28
267951
254587
RAWS-X (Xing ct 2019)
Retizval
x0p
28
40401
19259
R:GUPT (Yang st... 2013)
Retizval
x2p
24
50000
40596
T2Ruzkau (Ke at 2023)
Reticval
s2p
26
199412
158606
THUCCNes::Sus et al, 2016)
Retieval
s2p
26
20000
19288
UMETRIP.QA
Retieval
<2p
2647
2537
WelCPM (Oin 2013)
Retirval
<3p
1605
1602
oCDIIIN-Niss
Ratenal
<7p
50001
cMaak)4-N1 (Zhngg 2018)
Ratreal
<tp
231851
100
CSL fi al, 2002)
Ranoaal
<tp
20000
Dakkeader (lk stal. 2008)
Ranosal
<2p
aale Clang stal. 2021)
Renesal
<2p
19992
lon gat lovabidoo (Uainias, 2020)
2003)
Retnesal
12p
500
500
Retiesal
x2p
9000
67884
#MARCO (Botitacoo et al, 2021)
Retiesal
x1p
400000
3774770
level dat, webg
Retiesa
x1p
12765
30551
Retiesal
<2p
5000
50577
5112
CINLID
ChineseSTS Clang
CMNILI al... 2000)
110029
<Chen 2018, Lie al,,, 2018a, etal, 2019)
215587
185787
OCNLI
12957
QBQTC
51620
4735-
StxCIL
STS
544004
204059
XNLI (oto (Conneax al, 2018)
STS
51000
CSL 2002)
Classication
15000
122.0
THUCNews (Sus 2016)
Classicaionn
10000
1950
TNews
Classikaiin
10000
JDResiew
Clastiation
1252
1955
IFyTe e2ha 2007)
Clastkation
10000
CallnwShuppiig
Chedisviion
7850
TW60
Weimal
Chedicaioo
+24
7164
1936
Apa batsur d al, 3134)
Ronesal
mubillepai
203002
MIRACL Uhhng sl. 2023)
Banexal
mubiingual
40151
30046
Mr. Tylx (2hrg st. 2021)
Reneeaal
42p
mubiinguul
46729
46997
PAWSXX DSagg al., 2019)
SIS
subiingua
125435
128396
Ammankeeiign (Si 2019)
Classiiation
subiingua
10020
7721
AunaonrCinttaattuuu (O'Neil al. 2021)
Classiiation
625
aubiiiguu
10000
5323
StokliguddSeiiieen (Mollaronnz at. 2023)
Clastiiation
$25
anhilingul
10000
1924
Amnzon Mamive inar (FioGzraad et al, 20239
Confecation
82a
subiinguul
10020
1981
Amaannaaeeeeeeee (FiuCirndd 2023)
Clasfiation
523
subiingu
10020
7076
SETOPINmmn (Li et 2021)
Classfiation
525
subiingu
10000
0610
SETOPlaeat (Li al, 2021)
Classfiation
$25
aubiinguu
10000
7052
34

---

Published as a conference paper at ICLR 2026
Table 18: Detailed task instruction list for MTEB evaluation. Pair Classification", Reranking". Retrieval", and STS* indicate we use the same instructions for all the respective remaining tasks.
Task Name
Instruction
Classifcation
Instruct: Given Amzzon review, judge whether it countefaactual. Query: (query)
AmaconPolarity
AmscmmPolrryyCavviietieeee Classification
Instruct: Chasilying Amazon reviews into pusilive negative sentimett in Query: [qurr)
Amaronfteviov wtteeviiiioolleiiiiiiiii
Instruct: Classifyigg the given Amazon review into its appooriate noing category In Query: (query
Banking7PClassifee fiClasiifictiin
Instruct: Given a online banking query. find the correqooning intents Query: [query)
Instract: Classifying the caocion expressed the given Twintrr message the emations: anger, fear, Joy. love, sadness, and surprise Query: [query)
ImuthClossiiiaation
Instract: Classifying the senriment expressed in given movie roview the IMDR dattaser Query: [query]
MassinelaentCCassileeen
Instruet: Given utternnce as query, find the inteuts Va Query: (query)
Instruct: Given a user utterance as query, find the user scenarios la Query: (query)
MrToPPoensinCteeeifiii
Instruct: Classi Sying the intest domain of the given uiternnee in teik-eiiented comversation Query: Iquery)
Instruct: Classitying the intest the given otterance mek-oriented conversution in Query: Iquery)
Instruet: Classitying the given comments either tosie toxic Qnery: (query)
Instruct: Classitying the scatimen a gaven tweet as either positive, negative, or neutral Query. [query]
TNews
Instruct: Categorizing the given news tite in Query: (quesy]
WFlTek
Instruct: Given App description text, find the appcopriate fine grained casegory in Query: !query]
MolsiingulSenimees
Instruct: Chasslling semimen: of the customer review into pusitive, neutral. negative Query: !query)
JDReview
Instruct: Chasilying sentimen of the cusiomer review for iPfone into positive or negative in Query: Iquery)
OnlineShopping
Instruct: Classifying semtmmet the customer review into positive or negative Query: (quaryy
Waimai
Instruct: Clusidy customer review a food takcoway platform into positive or negative Query: (query)
Clastering
ArshCClsseringP2
Instrutt: Idemtily the main und secondary category of Arsiv pupers based on the tiles and abstracts 'n Query: (quiry) I
ArriCClasering555
Instruct: Menitiy the main and secondary category of Arxie pupers baved on the tilles Quary. (query)
BlorsiCChostringPPPP
Instruct: Montify main entegory Biorsiv papers hased the and abstnees ta Query: (querr)
BlonivChustering525
Instruct: Mestifyy the main entegory of Blordi papers hased on the ritles Query: [query]
MedraicClusteiugeepP
Instrut: Identity the main category of Medoxiv papers based on the titee and abstrects la Query: (query)
MedrxivClusterngNS25
Instruct: Identdy the main category of Medixiv papers based on titles la Query: Iquery)
RoddiiCasteerin
Instruct: Identiy the topic theme Reddi posts based the titls Query: (queryl
RediiiCusseringg2P
Instruct: Identily the topic or theme of Reddis posts based un the tites and posts 'n Query: (query)
StaekExchangeChhserrigg
Instrut: Mdnniiy the topie or theae of Stacktxccangee posts based on the tirles 'n Query: (query)
Instruct: Ideutiy the topic or theme Stackeeschange posts based the given paragraphi tn Query: (quary)
Instruct: Identiy topic theme the given news articles Query: (query)
CLSClusterngS22
Instruct: Identily the main category of scholar popers based on the tides In Query: (query)
CLSClustering22P
Instruct: Identity the main category of scholar pupers based on the tides and absinats la Query: (query)
ThuNNewCClaseernng555
Instruct: Idemtify topic or theme the given news articles based the rilee Query: [query)
ThuNeesCCuuserrnggPPPP
Instruct: Mentify the topic theme of the given news articles based on the tillex and contents' In Query: [query)
Pair Classification
Pair Classification
Instruct: Retrieve semantically similar text. la Query: (query)
Sprinthaullaatteuusiiiss
Instruet: Retrieve semantcaally similar questions. Query: [query]
Reranking
Reranking"
Instrict: Given a query, retrieve docoments that ancwor the query. 'n Query: (quey)
Instruct: Retrieve semantically similar questions. Query: [query]
Instriet: Retrieve semantically similar questions. Query: [query]
SoiDoeKK
Instruct: Retrivee relevant paper titles In Queey: [queey]
Retrieval
Retrieval
Instruct: Given a query. retieve documents that answer the query. Query: (query)
QuoraRetrieval
Instruct: Retrieve semuntically similar questions. Query: [query)
CQADupauck
Instrutt: Given a question, rettieve desiled question descriptions from Suckeschinge that ure duplicates to the given question Query: (query)
STS
STS"
Instruct: Retrieve semuticallly similar text, le Query: (querr))
Summarization
SummFval
Instrict: Retrieve semantically similar summories, In Query: [query]
35


Reviews:
[{"review_id": "89v5saTZOk", "reviewer": "Reviewer_U1wE", "summary": "This paper presents techniques behind the KaLM text embedding model. It applies various training techniques such as focal-style re-weighing to upweigh loss on harder examples, online hard negative mixing, contrastive distillation from a bigger model, and multi-stage training.", "questions": "Where does the high performance with the small 0.5B model come from exactly? Is it the:\n1) base model 2) training data 3) contrastive distillation 4) the two techniques of focal style re-weighing, hard-negative mixing.\nI ask this because based on Table 5, removing both the techniques (4) would still create a strong text embedding model for 0.5B standards.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* The final model performs well on benchmarks with fewer parameters.\n* Focal-style reweighing, online hard-negative mixing and contrastive distillation are novel in the text embedding space.\n* Very detailed and sounds experimental results leave few questions unanswered.", "weaknesses": "* The most impactful technique of focal-style reweighing is commonly used in machine learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l4o38NRxAi", "reviewer": "Reviewer_4fB5", "summary": "This paper proposes KaLM-Embedding-V2, a 0.5B LLM-based text embedding model. The approach modifies a decoder-style architecture by removing the causal attention mask and applying mean pooling, enabling more effective bidirectional representation learning. Training follows a three-stage progressive pipeline. As a result, the model achieves SOTA on English and Chinese MTEB.", "questions": "Suggestions:\n- it is not clear what represents different colors on figure 1\n- Some citations are duplicated (MTEB, Sentence BERT, NV-Embed)\n- While MTEB (v1) is larger than  MTEB (v2), MTEB v2 implements multiple fixes on existing MTEB tasks reducing estimation noise. It also removed a few faulty tasks, either due to frequent overfitting/leakage or outright bugs, e.g., SummEval has a known bug that makes its estimates unreliable (wrong). This mteb software will raise a warning about the fact. I would at least remove SummEval if not entirely replace MTEB(v1) with MTEB(v2)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Methodical system design. Detailed explanation of architecture, data, training all contribute\n- Ablations support most claims (focal loss, hard-negative mixing, distillation components)\n- Description of training data and pipeline for it’s creation", "weaknesses": "- The distillation stage is not fully justified, as the paper does not explore whether applying distillation earlier or interleaving phases could be equally or more effective.\n- Instruction dependence remains unclear, since the paper does not evaluate performance without instructions or under instruction mismatch.\n- Causal-vs-bidirectional attention choice insufficiently validated, lacking direct comparison with strong causal-mask baselines from the same model family.\n- No comparison of performance to teacher (Qwen3-Embedding-8B) and no comparison to smaller (to teacher) Qwen3-4b model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jjFLZz33zT", "reviewer": "Reviewer_4xdV", "summary": "This paper presents KaLM-Embedding-V2, an embedding model that leverages Qwen2-0.5B and different training tricks to improve the embedding representation and accuracy on MTEB tasks. The model surpasses most of the selected small models and even selected larger models on MTEB English and Chinese v1.", "questions": "1 - How does the model perform on other languages than English and Chinese?\n\n2 - Why not evaluate on MTEB(eng, v2) [1]?\n\n\n[1] Enevoldsen, Kenneth, et al. \"Mmteb: Massive multilingual text embedding benchmark.\" arXiv preprint arXiv:2502.13595 (2025).", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1 - New embedding models that surpass some SoTA LLM-based and encoder-based models with very few parameters.  \n\n2 - Authors give extensive details about the training recipe and the datasets used for it.  \n\n3 - The model is compared to multiple existing models and evaluated on both English and Chinese.  \n\n4 - Good open-source contribution if the training code is released, as it goes beyond most recent models that are open-weights only.", "weaknesses": "1 - The originality of this work is not very clear, as it combines existing training recipes that were used for training embedding models (see NV-Embed models and Qwen3 embedding models).   \n\n2 - Table 2 is clearly missing SoTA models on MTEB(eng, v1). Checking the MTEB leaderboard (on MTEB(eng, v2)), I see for example that Qwen3-4B and 8B achieve good performance on the benchmark but are not listed in the >1B models list.  They outperform KaLM-v2 but they are larger, it would be great to list them.  \n\n3 - Table 5 is very dense and hard to read, probably a plot of the average score on MTEB with the ablations could be better.  \n\n4 - The model is fine-tuned on many training sets of the datasets that are used for MTEB evaluation. MTEB uses the test sets, but there is a distribution similarity between the sets that may help the model get better on the benchmark. Evaluating on MTEB(eng, v2) could have fixed some of these issues, as for example it removes MS MARCO that is used for this model finetuning. Also, evaluating on AIRBench could give more insights on how the model performs on out-of-domain and out-of-distribution data (see NV-embed paper)  \n\n5 - The model is not evaluated in a multilingual setting (2 high resource languages are not enough) but is compared to multilingual models. It would be great to add a multilingual evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IQiFmwomfW", "reviewer": "Reviewer_kXYw", "summary": "This paper presents KaLM-Embedding-V2, a compact (0.5B parameters) text embedding model trained with improved data curation and training techniques.", "questions": "1. The teacher model is Qwen3-Embedding-8B. Why did the authors choose to initialize KaLM-Embedding-V2 from Qwen2-0.5B instead of using an existing Qwen-Embedding model (e.g., Qwen-Embedding-0.6B)? Wouldn’t that potentially eliminate the need for the expensive pre-training stage? Have the authors compared starting from Qwen2-base versus Qwen-Embedding models to verify whether the pre-training stage is truly necessary or beneficial?\n2. Whether the proposed distillation strategy generalizes across different teacher families or if it only works within the Qwen series.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The model achieves state-of-the-art results among models under 1B parameters and performs competitively with much larger models.\n2. The paper includes detailed ablation studies, comparisons, and analyses across multiple benchmarks, which make the results convincing.\n3. Open-source release", "weaknesses": "The contribution is mainly the integration of multiple known techniques, rather than a fundamentally novel algorithm.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)