Phased Training for LLM-powered Text Retrieval Models Beyond Data Scaling
Authors: Xin Zhang
Yanzhao Zhang
Wen Xie
Dingkun Long
Mingxin Li
Pengjun Xie
Meishan Zhang
Wenjie Li
Min Zhang
Venue: COLM 2025 (2025)
Keywords: Text Retrieval
Text Embedding
Reranking
LLM-based Embedding
TL;DR: Training powerful general-purpose text embedding and reranking models by a multi-stage training framework and efficient data synthesis.

Abstract: Current efforts in building large language models (LLMs) based general-purpose text retrieval models primarily focus on architectural design and training data scaling. However, significant challenges remain in effectively modeling diverse retrieval tasks and domains, including multi-task conflict, data imbalance, and training efficiency. To address these challenges, we propose a novel phased training framework for text retrieval, featuring:  (1) robust foundation modeling with core relevance data, (2) progressive specialization through modular task adaptation, and (3) knowledge fusion via weight interpolation based model merging. This framework simultaneously optimizes both embedding and reranking models through a unified architecture. We also present an efficient yet scalable data synthesis pipeline to expand training data, based on open-source LLMs. These synthetic data can be efficiently incorporated into the phased training framework, enhancing model performance. We identify five distinct types of retrieval tasks, \ie basic relevance retrieval, code retrieval, tool retrieval, complex instruction-based retrieval, as well as reasoning-intensive retrieval, conducting extensive experiments. Our method achieves the best performance across MTEB and various retrieval benchmarks of the five task types. Further analysis demonstrates the effectiveness and efficiency of our proposed training framework and data synthesis pipeline.

Published as a conference paper at COLM 2025
Phased Training for LLM-powered Text Retrieval Models Be- yond Data Scaling
Xin Zh Yanzhao Zhang, Wen Xie', Dingkun Long, Mingxin Penjun Xie, Meishan / Wenjie Li?, Min Zhang'
'Harbin Institute of Technology, Shenzhen 2hh Hong Kong Polytechnic University zhangxin2023estu. hit. edu. cn {zhangmei shan, zhangmin2021 J edu. cn Project: hatbb//c///ecccilyyeeeeeeeee
Model:
Abstract
Current efforts in building large language models (LLMs) based general- purpose text retrieval models primarily focus architectural design and training data scaling. However, significant challenges remain in effectively modeling diverse retrieval tasks and domains, including multi-task conflict, data imbalance, and training efficiency. To address these challenges, we propose novel phased training framework for text retrieval, featuring: (1) robust foundation modeling with core relevance data, (2) progressive specialization through modular task adaptation, and (3) knowledge fusion via weight interpolation based model merging. This framework simultane- ously optimizes both embedding and reranking models through a unified architecture. We also present an efficient yet scalable data synthesis pipeline expand training data, based on open-source LLMs. These synthetic data can be efficiently incorporated into the phased training framework, en- hancing model performance. We identify five distinct types of retrieval tasks, i.e., basic relevance retrieval, code retrieval, tool retrieval, complex instruction-based retrieval, as well as reasoning-intensiin ducting extensive experiments. Our method achieves the best performance across MTEB and various retrieval benchmarks the five task types. Fur- ther analysis demonstrates the effectiveness and efficiency of our proposed training framework and data synthesis pipeline.
retrieval, con-
06.6
08.5
78.9
$0.5
22.5 38.5
35.3
NV-Embed-22
BGEnuttega
GTE in
6.a
Lychee-embbedyy
00.3
C2.7
Lychee-reranke
22.3
57.6
MMTEB Retricval
MTTEB-Code Code Retrieval
ToolBench Teal Retrieval
FallowiR Instruction Retrieval
BRIGHT Reasoning Retrieval
Basic Relevance Retrieval
Figure 1: The performance our Lychee models and baselines on different retrieval tasks.
Introduction
Text retrieval serves as foundational technology for numerous downstream applica- tions, including web search (Huang et al., 2020; Liu et al., 2021) and question answering (Karpukhin et al., 2020). Modern competitive text retrieval systems typically adopt multi- stage pipeline: dense vector retrieval via embedding models for initial candidate recall, followed by cross-encoder rerankers for fine-grained relevance scoring (Zhao al., 2024).
* *Corresponding author.

---

Published as a conference paper at COLM 2025
Recently, the emergence of Large Language Models (LLMs) has profoundly reshaped re- trieval tasks. LLMs introduce dual advancements: (1) enhanced semantic understanding capabilities that enable superior embedding and reranking models, and (2) novel appli- cation paradigms such as Retrieval-Augment which demand new specialized retrieval capabilities beyond traditional semantic matching, including tool or APl retrieval (Xu et al., 2024), instruction-based retrieval (Weller et al., 2025a), and peasoning-intenslv retrieval (Su et al., 2025). These emerging scenarios require models to not only recognize basic relevance but also comprehend functional compatibility, procedural logic, and task-specific constraints.
Generation (RAG) and Agent Systems,
To build such general-purpose retrieval models, existing approaches primarily use LLMs as backbone models, scaling training data through public sources and L.M--riven synthetic dataset (Wang et al., 2024; Lee et al., 2025a;b). The training strategy still inherits BERT-era methodologies, where diverse data are naively mixed or sampled to perform contrastive learning. However, previous schemes suffer from critical challenges in building versatile models for various basic and specialized retrieval tasks. In the aspect of data synthesis, prior research relies expensive commercial LLM services for data generation and filtering to ensure the quality (Wang et al., 2024; Lee et al., 2025b), which severely restrict the efficiency and scalability of data scaling. Moreover, the directly mixing of data in single-step training approach presents several obvious issues: (1) conflict in multi-task retrieval learning (Yu et al., 2020; Li et al., 2024), (2) data imbalance across tasks, domains and languages (Wang et al., 2024), and (3) inefficient training and developing process ($4.3).
To address these challenges, we propose a new phased training framework with adaptive model merging, designed to progressively enhance retrieval capabilities. Our framework first categorizes training datasets into clusters based on task similarity and data volume, then builds the model in a phased manner: (1) establishing foundational relevance competencies across tasks on sampled warm-up data, (2) separately training specialized task models for each cluster, avoiding the impact of data imbalance and task conflicts, (3) merging all task models into a composite one with adaptive parameter interpolation (Li et al., 2024) to fuse multi-task knowledge, and (4) fine-tuning the merged model to optimize adaptability across tasks. This framework offers a more efficient way compared to the complex and expensive process of tuning data sampling ratios in traditional multi-task training. In addition, we introduce a cost-effective data synthesis pipeline leveraging role-aware LLM prompting with open-source LLMs, generating high-quality multi-domain query-document pairs without reliance on commercial APIs. More importantly, these synthetic data can be quickly integrated into the final model through the separate training and merging paradigm in the proposed framework, enhancing the performance across multiple retrieval tasks.
In this work, we identify five types of retrieval tasks (64.1), including basic relevance retrieval, code retrieval, tool retrieval, complex instruction-baser retrieval, and reasoning- intensive retrieval. Built upon powerful Qwen2.5 (Yang al., 2024) LLMs, our framework produces a suite of general text embedding and reranking models, Lychee series, achieving new state-of-the-ar results across multiple evaluation benchmarks of the five types ($4.2). We further demonstrate the effectiveness and efficiency of our training framework and data synthesis pipeline through extensive analysis (64.3). Our contributions are threefold:
Phased Training Paradigm: We propose a multi-stage training strategy with adaptive model merging that enables efficient knowledge transfer across tasks, outperforming tradi- tional single-stage approaches in extensibility and final performance.
We present a scalable, cost-effective method for generating high-quality retrieval data, which is validated to match exceed effectiveness of existing data synthesis pipelines.
Scalable LLM-Driven Data Synthesis:
Open-Source Release: We will open-source our general-purpose Lychee embedding and reranking models and training protocols to facilitate advancements in LLW--ooered retrieval systems.
2

---

Published as a conference paper at COLM 2025
up
(2) Learning
LLM
LLM
Merp
Musk
Traning
Relevance
Musk
Data of All Tasks
(Equally sampled)
Warm-up Data
One Step Cost
Tool Retrieevll
Pefformance
M Misk
Sampled
Retriever
Retriever
(4) Enhance
M
Murrrge
(3) Merge
M
task
Total Cost
III
Intermaciane
Conventional Training
Our Phased Training Framework
Figure 2: Illustration of conventional training and our phased training framework. The conventional approach typically mixes all different types of data together and performs single-stage multi-task training, which faces challenges of data imbalance and task conflicts. Moreover, any adjustment requires re-running the entire training process. To address these challenges, our framework first learns each retrieval task independently, followed by model merge and enhance phases, enabling efficient training process (analysis in $4.3).
2 Related Work
Text-centric Retrieval Text-centric information retrieval tasks have evolved substantially in recent years. Traditional text retrieval tasks primarily focus on information-seeking scenarios, matching user queries with relevant passages containing direct answers (Nguyen et al., 2016; Thakur al., 2021). This can be extended to multilingual retrieval across language boundaries (Zhang et al., 2023b) and code retrieval for finding relevant code snippets (Husain al., 2019). Beyond information-seeking, with the development of LLMs, more specialized retrieval scenarios have emerged. For example, (1) instruction following (Su et al., 2023; Weller et al., 2025a) where the search needs to follow specific instructions, (2) tool retrieval that focuses (Zhang et al., 2023a; Xu et al., 2024), and (3) saasoning-intensii complex reasoning to better understand the query (Xiao et al., 2024a; Su et al., 2025). While these tasks have typically been addressed separately, our work explores a unified approach capable of handling diverse scenarios while maintaining competitive performance.
finding appropriate tools or APIs for specific functionalities
retrieval that requires
Multi-stage Retrieval with Embedding and Reranking Retrieval tasks are primarily addressed through a multi-stage approach, typically consisting first-stage retrieval for efficiently selecting candidates (e.g., top 1,000) from large corpus, followed by reranking for precise relevance assessment (Zhao et al., 2024). Modern first-stage retrieval has shifted to dense retrieval approaches (Karpukhin et al., 2020), where an embedding model encodes queries and documents into dense vectors for semantic matching. The reranking stage commonly employs transformer-based cross-encoder models for fine-grained scoring of query-candidate pairs (Zhao et al., 2024), enabling richer interaction between query and candidate compared first-stage retrievers. With the widespread adoption of RAG ap- plications, pre-trained general-purpose embedding and reranking models have garnered significant attention (Wang al., 2022; Chen et al., 2024; Zhang al., 2024). In particular, LLM based approaches have consistently pushed the boundaries of retrieval performance (Wang et al., 2024; Muennighoff et al., 2025; BehnamGhader et al., 2024; Lee et al., 2025a; Li et al., 2025). The success of these models can be attributed to their effective single-stage large- scale training on data from diverse domains and tasks. However, simple data combination can lead to multi-task conflict issues (Yu et al., 2020), which limits the model training (Sturua embedding and
et al., 2024). In this work, we aim to improve LLLM-based general-purpen reranking models, proposing novel phased training framework with model merging (Li et al., 2024) to tackle the challenge in multi-task/ domain training.
LLM-driven Retrieval Data Synthesis Beyond architectural and training improvements, scaling training data is crucial for building powerful general-purpose retrieval models.
3

---

Published as a conference paper at COLM 2025
However, the scarcity of authentic data presents significant challenge (Villalobos et al., 2024). To address this, using LLMs to synthesize retrieval data has proven to be an effective solution, such synthesizing multi-task and multilingual retrieval data (Wang et al., 2024) and long-context retrieval pairs (Chen et al., 2024). Furthermore, some researchers propose using synthetic retrieval data as evaluation sets, offering more cost-effective (Rahmani et al., 2025) and dynamic model assessment (Chen et al., 2025). However, existing retrieval data generation pipelines often rely heavily on expensive proprietary LLMs (Wang al., 2024) or involve complex workflows (Chen et al., 2025), making the data synthesis process either prohibitively expensive or inefficient. In this paper, we present novel role-based data generation pipeline that enables efficient and cost-effective data scaling.
3 Method
3.1 Implementation of Embedding and Reranking
Our embedding and reranking models are both designed within an instruction retrieval framework, where each query 9 is accompanied by a task instruction I to guide the retrieval of the relevant document d' (a.k.a., positive). Each training instance comprises I, 9, d and a list of irrelevant documents [dj, d2, dn] (i.e,, negatives).
Embedding We utilize the LLM backbone encode the input text, where the final hidden state corresponding to the [EOS] token is extracted as the final text embedding. Concretely, the query embedding computed as eg = (LM I, q, [EOS]] [-1]] and the doc embedding ea = (LM({d, [EOS]]) (EdS))
[-1]. The model is trained by the following contrastive loss:
exp(cos(e,,e)//t
c = log
(1)
jj=1 exp(cos(e,,,,)//) + exp(cos(e,,e)) 7T))
where eq. et c,, represents the embedding of query, positive document and the j-th negative document separately, and N is the number of hard negatives. 7 is the temperature parameter.
Reranking We transform the relevance judgment task into a binary classification problem. Specifically, we use softmax function on the probabilities of the model outputs for "yes" and "no" to obtain the final ranking score. We optimize the following loss:
P(g, d) (prefix) Instruct: Query:q Document: d
(2)
Lyank (ogg((((P().)))
where p(-1+) denotes the LLM probability output by the LLM, and the (prefix) is fixed prompt for relevance judgment: "Given the instruction and query, judge whether the document is relevant to the query (yes/no). The label / is "yes" for positive documents and "no" for negatives.
3.2 Phased Training Framework
To develop a general model for multiple retrieval tasks, straightforward approach to combine the training data from all tasks into a single dataset. However, this naive method often leads to data imbalance issues, as the volume of available training data typically varies significantly across different tasks. For instance, general relevance retrieval tasks usually have substantially more training data compared specialized tasks such as tool retrieval. While ensuring balanced task contribution during training is crucial for achieving strong performance across all tasks, finding the optimal data composition through experimental search can be computationally expensive. Moreover, simply combining data from different tasks may lead to multi-tast conflict issues (Yu et al., 2020; Li et al., 2024).
To address these challenges, propose phased training framework (Figure 2). Given datasets from multiple tasks, we first classify them into T clusters (C,...., CT} based on the amount of available data and task similarity. The framework consists of four steps:
4

---

Published as a conference paper at COLM 2025
(1) Warm-up: We sample small and equal proportions of data from each cluster to build wa training data Dwip. Then we train an initial model Mwrp on these data, establishing fundamental relevance capabilities across all tasks.
(2) Task Learning: For each cluster C, we continue train Mwrp on the full-scale data to obtain the specialized task model M Mask This step ensures that each model develops expertise in its respective task cluster.
(3) Merge: All task models {M (ask M task are merged into a single composite model
Mmerge- The merging method is detailed in $3.3.
(4) Enhance: Finally, to optimize the model's adaptability across different tasks while maintaining task-specific com competencies, we fine-tune Mmerge on the warm-up data Dwrp-
3.3 Adaptive Model Merging
Our method is inspired by findings from Li et al. (2024) and employ the SLERP merging approach', which utilizes spherical linear interpolation for model merging and sampled tiny dataset for hyperparameter search. Given two models (i.e,, M; and M,) trained from an identical base model Mo, SLERP first calculates task vectors (v, and vj) which represent the parameter differences between the respective trained models and the original base model, egg, 2; ((M,) - ((M)). Let aij denote the angle between the task vectors of and zy. The merged task vector Vij = ( is computed as:
sin (1 - t)ai))
sin(taii) sin(a;)) Dj
faterp (a 0j)
2j +
(3)
sin():;)
where is a hyperparameter. Extending this approach to N models, the final task vector V derived as follows:
V Σ Ssterp (Uk Vi-1),
where V; 1 is the merged task vector of N 1 models. Thus, the final model parameters is 0 = M()) + x x V, where À is hyperparameter controlling task vector scaling, The search for optimal merging hyperparameters could be posed as an optimization problem:
(16) = argmin ((NN))))
argmin((u)),,)
1 Dr
C(1:0) + pAA) JED.
Σ
(5)
where μ is hyperparameter used to prevent overfitting in this optimization problem, and D, is tiny sample set for efficient hyperparameter search.
3.4 Scalable LLM-Driven Data Synthesis
We present a simple but effective data synthesize method by LLM (i.e,, Qwen2.5-72B 1eenn..5-7BB-n-iiee Instrrcct Yang et al. (2024)), which involves a two-step generation process guided by simulated user roles and query categories (Figure 3). This approach aims to diversify and enrich the training data for downstream tasks. Our data synthesis begins with comprehensive collection from diverse sources, including news articles, Wikipedia articles, academic papers, and technology publications (Appendix SA.3). The variety of source material helps ensure the generated data covers broad spectrum of topics and writing styles.
corpus
Role Candidates Selection Inspired by AIR-Bench (Chen et al., 2025) and Persona Hub (Ge et al., 2024), we assign specific roles to each document to simulate the persona of user that might pose queries about the document. This injection of user perspective enhances the diversity and realism of the synthetic queries. Specifically, we leverage retrieval model? to identify the top 5 role candidates for each document from Persona Hub.
Refer to https: //gg:::////ttbbb Stt. SLE-M---LERPMMM--M- 2https: wwt..//aaaba-..//ttteeoeaaaaaa.. Qwen2-1 1.5B 5B instruct

---

Published as a conference paper at COLM 2025
Retrieval
Bake college stadeer prdlan
Role Pool
Role
Document
00
lst incrt Synthetic Query
LLM
-Dilisitt levat Linvering
LLM
Select
Leeghh los
lit treapeatied
Langaage: Exshis
isdes Somple...
Corpus
Document
Config
Data Preparation
Generation Step 1: Configuration
Generation Step 2: Query
Figure 3: Our two-step generation pipeline for synthesizing retrieval data.
Generation Step 1 Configuration In this step, the sampled document and its role can- didates are provided to a prompt, instructing the LLM to output the most appropriate role-configuration combination for query generation. The query configuration consists of 4 dimensions: (1) Type: we define five query types, [Keywords, Factual, Summarization, Yes/No, Background}; (2) Difficulty Level: we set three levels of difficulty, {High School, University, PhD); (3) Length: we specify different length option lists for different question types; and (4) Language: English or Chinese. The combination of these dimensions allows for fine-grained control over the characteristics of the generated queries.
Generation Step 2- Query Generation The selected document and generated query config- uration serve as the input for this step. The LLM is prompted with the following structured template to generate queries that adhere to the defined type, difficulty, length, and language constraints, maintaining consistency and relevance to the original document.
You need to start from the perspective of the following (Role), generate a query based on the (Document) and the (Requirements). **Role:**
**Document:**
**Requirements::
Type: (
Difficulty level:
Length: the length of the generated sentences should be 0 words;
Language: the language of generation should be (
Filtering We implement a filtering process with two checks to ensure the quality and relevance of the generated queries: (1) Format validation: Any queries that do not conform to the predefined JSON format are discarded during the parsing phase; (2) Consistency filtering: We employ an embedding model? to assess the relevance between generated queries and their corresponding documents. Queries that fail retrieve their associated document within the top-20 results are filtered out. This quality assurance step ensures that only high-quality, relevant queries are retained in the final dataset.
Experiments
4.1 Setup
Evaluation Tasks We explore five types of retrieval tasks: (1) Basic Relevance Retrieval, we set three categories: English, Chinese, and Multilingual, and evaluate on MTEB (Muen- nighoff al., 2023), CMTEB (Xiao al., 2024b), MMTEB (Enevoldsen
al., 2025) and MLDR (Chen et al., 2024), respectively; (2) Code Retrieval, we evaluate on MTEB-Code (Enevoldsen et al., 2025), which composed entirely of code retrieved data; (3) Tool Retrieval, on T (Qin et al., 2024); (4) Complex Instruction Retrieval, on FollowIR (Weller et al., 2025a); and (5) Reasoning Intensive Retrieval, on BRIGHT (Su et al., 2025). For MTEB,
FFor example, Keywords corresponds the length list (less than 5, less than 7, to 13), and Summar commarization corresponds the length list (13 to 21, 20 to 50, more than 80 but less than 100). We the processed version by Zhang et al. (2023a) for retrieval.

---

Published as a conference paper at COLM 2025
Task Type (5)
Description
Data Cluster (4)
#Train
Evaluation Benchmark
MTEB (Muennighoff et al., 2023)
EN & ZH
4M
Basic Relevance Retrieval
CMTEB" (Xiao et al., 2024b)
BR
MMTEB (Enevoldsen et al., 2025)
Multilingual
0.5M
MLDR (Chen et al., 2024)
CR
Code Retrieval
MTEB-Code (Enevoldsen et al., 2025)
Code Tool
1.2M
TR
Tool Retrieval
ToolBench (Qin et al., 2024)
Complex Instruction-
CIR
based Retrieval
CIR
1M
FollowIR (Weller et al., 2025a)
Reasoning Intensive
RIR
BRIGHT (Su al., 2025)
Retrieval
Table 1: Our text-centric retrieval experiment setting. For reasoning intensive retrieval, we do not have training data as there are no appropriate datasets publicly available. "We evaluate on the full set for embedding models and the retrieval subset for reranking.
BR
CR
TR
CIR
RIR
Model
Param
MTEB CMTEB MMTEB MLDR MTEB-Code
ToolBench FollowIR
BRIGHT
BGEmultiingual
9,24B
69.88e
68.44
61.257
49.107
62.047
63.65%
-2.137
17.687
NV-Embed-v2
7,85B
72.31
56.25
63.747
50.547
1.04
19,287
GritLM-7B
7.24B
66.8*
60.93
73.6'
35.42
3.45
20.63
mittall-7
7.11B
66.6*
59.92
60.28
69.2"
31.79
-0.62
17.54
GTEQwen2-78
7.62B
69.88
71.62
62.51
56.531
62.17%
59.487
4.94
22.89
Lyche--embedqwen255
7,62B
73.00
73.43
65.12
57.90
79.93
85.16
6.91
24.50
GTEQwen2-155 58
1.54B
67.19
67.12
59.47
52.11
61.987
62.577
0.74
18.477
Lychee-embedowwez3.e
1.54B
68.39
69.77
58.43
53.85
72.54
86.35
5.74
19.47
BGE-M3 (Dense) Jina-v3 pyche-embeedqwwe25a.
0.56B
59.847
61.79%
59.54"
52.50"
58.22
58.457
-3.11
11.947
0.57B
65.524
63.077
58.37*
40.717
58.857
59.647
-1.34
11.347
0.49B
65.73
69.26
56.81
51.44
71.06
85.43
2.24
13.25
Table 2: Evaluation for embedding models. Scores are reported from the MTEB leaderboard by default to ensure consistency and comparability across evaluations. "Taken from the corresponding paper or model card. 'Our run. "From Enevoldsen et al. (2025).
CMTEB, MMTEB, we follow their scoring methods and report the overall average. For MLDR, ToolBench, BRIGHT, we nDCG@10. For FollowIR, use its defined p-MRR.
Training Data We select public training datasets for each retrieval task type, then man- ually categorize into four clusters based language and task relevance for training framework. Table presents the distribution of different data types used throughout the model training process (detailed datasets in Appendix Table 5). Due to limited public data for tool retrieval and overlap with code retrieval, they are combined for training. Moreover, we employ 2M LLM synthesized data (the pipeline elaborated in $3.4, statistics in Table 8) to enhance the retrieval training.
Implementation To support these diverse tasks with multilingual input, we choose the Qwen2.5 LLMs (Yang et al., 2024) as our backbone. We adopt Qwen2.5 models in three sizes: 0.5B, 1.5B, and 7B, which yield embeddings of 1024, 1536, and 3584 dimensions, respectively. We employ LoRA (Hu et al., 2022) with the rank of 64 and the alpha of 32. In training, we use the Adam W (Loshchilov & Hutter, 2019) optimizer with the dynamic batch size strategy to improve the training efficiency of different length texts (details refer to Appendix Table 6). We train the models for approximately 10,000 steps during the Warm-up and Enhance stage. The number of steps for the Task learning stage varies depending on the clustering data size: around 200,000 steps for the English and Chinese (EN & ZH) cluster, 50,000 steps for the Multilingual, 100,000 steps for Tode&Tool, and 20,000 steps for CIR. In the Merge stage, we search for the merging sypee-parameterss using roughly 100 steps. The learning rate is set to 1 with warm-up ratio of 0.1 in all training.
7

---

Published as a conference paper at COLM 2025
BR
CR
TR
CIR
RIR
Model
Param
BEIR
CMTEB-R MMTEB-R MLDR MTEB-Code ToolBench FollowIR
BRIGI
wychee-embedqwen25-11
1.54B
57.16
72.98
39.28
53.85
72.54
86.35
5.74
19.47
278M
54.61
70.18
54.43
50.32
46.32
67.80
-0.69
16.69
BC
568M
55,36
71.82
57.13
60.80
50.81
62.52
-0.06
15.87
BGFpranker-2-gemma
9.24B
60.81
71.74
69.80
49.10
68.63
68.14
-2.13
17.68
lychee-rerankkwwen...n
0.49B
56.13
69.79
58.09
59.74
69.86
86.79
6.29
11.58
Lychee-rerankgwena... Q-we2..-11.8
1.54B
59.56
76.37
62.47
64.09
78.03
90.82
7.38
16.92
Lychee-rerank Qwen2.5-7B
7.628
62.85
79.06
68..7
64.50
80.78
95.50
12.11
25.31
Table 3: Evaluation results for reranking models. We use the retrieval subsets of MTEB, CMTEB and MMTEB, which are BEIR, CMTEB-R and MMTEM-R. The rest are all retrieval tasks. All scores are our runs based on the retrieval top-100 results from the first row.
4.2 Main Results
Embedding In Table 2, we present the evaluation results for embedding models. We compare our models with three types of representative baselines: (1) LLM-powered English- centered versatile embedders, i.e., NV-Embed-v2 (Lee et al., 2025a), GritLM-7B (Muennighoff et al., 2025); (2) LLM-based multilingual embedders, ie.. multtiiiln (Chen et al., 2024), GTE (Li et al., 2023), and E5 mistral-7b (Wang et al., 2024); and (3) strong BERT-based multilingual models (for comparation with our 0.5B model), i.e., BGE-M3 (Chen et al., 2024) and Jina-v3 (Sturua et al., 2024). Our models achieve the best performance across all tasks, including all levels of retrieval (basic relevance and specialized), demonstrating the effectiveness of our approach. Detailed results of each benchmark are provided in the Appendix, along with ablation studies on our training framework and data in $4.3.
Reranking In Table 3, we report the evaluation results for reranking models, where we use the retrieval subset of MTEB (i.e., BEIR (Thakur et al., 2021)), CMTEB (CMTEB-R) and MMTEB (MMTEB-R) for evaluation. To make fair comparison, we take the retrieval as candidates to all rerankers. Similar to the embedding evaluation, we compare our rerankers with: (1): LLM-powered general-purpose rerankers, Le., pennperanees22-eemmm (2) BERT-based rerankers, egg, ge Similar to the embedding evaluation, out models demonstrate superior performance compared to previous models of comparable size.
top-100 results from Ours - Qwen2.5-1.58 e
4.3 Analysis
Training Efficiency There are two commonly adopted baseline training strategies in current practice: (1) naive training, which directly combines training data from different tasks; (2) naïve training with up-sampl aasmmamlingg ing, which increases the probability of appearance for tasks with smaller data scales before conducting native multi-task training. Since they both are single-stage training, any adjustments to data hyper-parameters would require to run the entire training process again, which is computationally expensive. Our framework, in contrast, learns different data clusters separately, allowing for individual adjustments with greater efficiency. Based on the experimental settings in Table 1, we analyze and compare the efficiency gaps. Following the conventional setting of training for one epoch, use the amount of training data consumed as a proxy for training cost. Assuming we make one modification to the CIR data (1M) during training. For naïve training, the cost of one training iteration is (4 0.5 +15++1.. 1.2 + 1) = 6.7, resulting in a total cost of 13.4 for two training iterations. For up-sampling with a coefficient of 1.3 for small-scale data, one training iteration costs 4 - (0.5 + +) * 1.3 7.51, leading to total training cost of 15.02. In our framework, Warm-up and Enhance steps each consume 0.1, while Merge is virtually cost-free. The task learning phase consumes 6.7, making the initial training cost 6.9. Since need to readjust CIR, the additional costs of and 0.1 are required for re-running the CIR task learning and Enhance, bringing the total cost to 6.9 +1+0.11 8, which is significantly lower than both 13.4 and 15.02. This demonstrates the efficiency advantages of our framework. For reference, we also list the time required for the 1.5b embedding model when training
8

---

Published as a conference paper at COLM 2025
CR
TR
CIR FollowIR
RIR BRIGHT
Row Setting
BEIR CMTEB-R MMTEE-R MLDR
MTEB-Code ToolBench
Lyche-eebbedlwaa25000
56.03
70.69
57.70
49.10
71.06
85.43
2.24
13.25
2
w/o Enhance (i.e,, Mmeepe)
53.99
70.10
56.71
48.26
72.01
84.97
2.25
13:16
w/o Merge (split Task models)
56.12
70.12
49.77
71.10
85.43
1.56
17.39
4
w/o Task carning (Murp)
50.69
67.38
52.38
45.76
51.20
64.98
0.69
10:39
5 w/o synthetic data
53,30
69:20
55.78
44.79
70.26
85.45
1.99
13.12
Naive
52.91
70.10
46.71
40.26
71.05
67.12
1.15
12.16
Naive w/ up-sampling
55.47
70,50
57.28
48.65
70.79
85.63
2,16
13.39
Ablation study. For brevity, we report results from our 0.5B embedding model, with other model variants showing consistent trends. Row 6 and 7 are trained on the same data with row 1, including the synthetic data. The naïve refers to simply combining all data to train the model. The up-sampling method involves increasing the probability of appearance for data subsets with smaller scales.
Table 4:
32 GPUs, where our method takes about 7 days, the naïve method takes about 11.5 days, and the up-sampling takes about 13.5 days.
To conclude, our framework yields similar strong performance compared to basic up- sampling strategy, but allows more efficient model iteration and adjustment.
Ablation study of the training framework. Table 4 on the embedding model to evaluate the performance of intermediate models, analyzing the impact of each step in our training framework. Our framework consists of four steps, producing three types of intermediate models: the warm-up model Mmerge, task-specific models, and the merged model Mwrp. First, during the warm-up step, we train the LLM on small-scale data to acquire a fundamental retrieval model Mwrp- As shown row of Table 4, Mwrp performs relatively weak across various retrieval tasks. Then, in the lask learning step, we derive specialized models for each task from Mwrp. Row demonstrates that all task-specific models achieved significant improvements compared to Mwrp (row 4). Next, we merge multiple task models into single model Mmerge, whose performance (row 2) generally remains comparable to task-specific models (row 3), with slight degradation in some cases. Finally, after the enhance step, we obtain the final model. Comparing it (row 1) with Mmerge (row 2), we observe further improvements across all tasks. Additionally, we train the model using two baseline strategies without our framework (rows 6 & 7). Their inferior performance compared our final model validates that our training framework effectively enhances the development of multi-task retrieval models. For comparisons with other model merging methods, we recommend referring to Li et al. (2024), which provides comprehensive analysis of various merging methods on text embedding.
We conduct ablation experiments
Data synthesis quality evaluation. Table 4 & Figure We pro- posed an efficient data generation method ($3.4) to create extensive synthetic data for both Chinese and English Basic Relevance Re- trieval. We validate its effectiveness from two perspectives. First, we perform a direct ablation by removing synthetic data from the training process. As shown in 5 of Table 4, the model trained without synthetic data shows lower performance compared to our final model (row 1), demonstrating the beneficial impact of syn- thetic data. Second, we conduct controlled experiments comparing different data generation pipelines, including AIR-Bench (Chen et al., 2025) and Promptagator (Dai et al., 2023). Using the same 400k corpus from MS MARCO, we generate synthetic datasets of equal size using three different methods, train three small mod- els, and evaluate performance on BEIR. As illustrated in Figure our method achieves the best performance, outperforming the two baselines, while providing higher efficiency and more diverse synthetic data.
BEIR Scores
36.77
35.80
35
31.56
30
AIR-Bench
Promptagator
Our
Figure 4: Comparison of different synthesis methods. We generate data from the same cor- pus, train small mod- els, and evaluate on BEIR.

---

httt:/ooummmddggm44\\00 Published as a conference paper at COLM 2025
5 Conclusion
In this work, we aim to develop enhanced LLM-based general-purpose text embedding and reranking models, focusing on five retrieval tasks: basic relevance retrieval, code retrieval, tool retrieval, complex instruction retrieval, and reasoning-intensivee intensive retrieval. We propose a novel phased training framework to address multi-task learning conflicts and data imbalance through model merging and task-specific learning. Additionally, introduce an efficient data synthesis method for scaling, employing a two-step generation process based on user roles and query configurations. With these advancements, we build series of models of varying sizes based on robust multilinguul LLMs. Our models demonstrate superior performance across eight benchmarks covering all five tasks, underscoring the effectiveness of our approach. We conduct ablation studies to confirm that our training framework significantly enhances multi-task retrieval learning and our data synthesis method is more effective than previous approaches.
Acknowledgments
We sincerely thank the anonymous reviewers for their valuable feedback and suggestions. This work receives partial support from the Natural Science Foundation of China (under Grant 624B2048) and Research Grant Council of Hong Kong (PolyU/152099724) and the Shenzhen Science and Technology Program (under Grant 2BSY5222332226000000008
References
Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. LLM2vec: Large language models are secretly powerful text encoders. In First Conference on Language Modeling, 2024. URL https://openrevieer net/ forum?1d==I11RR7VEB.
Luiz Bonifacio, Vitor Jeronymo, Hugo Queiroz Abonizio, Israel Campiotti, Marzieh Fadaee, Roberto Lotufo, and Rodrigo Nogueira. mMARCO: A multilingual version of the ms marco passage ranking dataset. arXiv preprint ar Xiv:22108. 1it:29108.1897,, 2021. URL https:// wwxiv.org/abs//111 13897.
Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. M3- embedding: Multi-linguality, multifunncionnaliy, multi-granularity text embeddings through self-knowledge distillation. In Findings of the Association for Computational Linguistics: ACL 2024, pp. 2335, Bangkok, Thailand, August 2024. URL https: //aclanthology org. findings acl. 137/.
Jianlyu Chen, Nan Wang, Chaofan Li, Bo Wang, Shitao Xiao, Han Xiao, Hao Liao, Defu Lian, and Zheng Liu. AIR-bench: Automated heterogeneous information retrieval benchmark. In Proc. of the ACL, pp. 19991-20022, Vienna, Austria, July 2025. URL https://aclanthologr. wrt.ey..rgg////// 2025. acl-long. 982/
Zhuyun Dai, Vincent Y Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith Hall, and Ming-Wei Chang. Promptagator: Few-shot dense retrieval from 8 examples. In The Fleventh International Conference on Learning Representations, 2023. URL https://openreviiee
Kenneth Enevoldsen, Isaac Chung, Imene Kerboua, Márton Kardos, Ashwin Mathur, David Stap, Jay Gala, Wissam Siblini, Dominik Krzemiñski, Genta Indra Winata, al. MMTEB: Massive multilingual text embedding benchmark. In The Thirteenth International Conference on Learning Representations, 2025. URL httpp::/ http://wwwwwee.e.eeee///ee/////a
Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sen- tence embeddings. In Proc. of the EMNLP, pp. 6894-6910, Online and Punta Cana, Domini- can Republic, November 2021. URL https:://acleet h/tptt/s///claaahooogg...o. org/2021
emnlp-main. 552.
10

---

Published as a conference paper at COLM 2025
Tao Ge, Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. Scaling synthetic data creation with 1,000,000,000 personas. arviv preprint ar 0rii::2406.20094 2024. URL
Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference Learning Representations, 2022. URL /ospes::///prreeeee..neee Forum?ld=nneeWeeeFYB
Jui-Ting Huang, Ashish Sharma, Shuying Sun, Li Xia, David Zhang, Philiip Pronin, Janani Padmanabhan, Giuseppe Ottaviano, and Linjun Yang. Embedding- retrieval in facebook search. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2553-2561 Virtual Event, CA, USA, 2020. URL https://doi, org/10 10. 01145/39444/ 3403305.
Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, and Lichao Sun. Metatool benchmark for large language models: Deciding whether use tools and which to use. In The Twelffh International Conference on Learning Representations, 2024. URL R0c2qtalgG.
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. Codesearchnet challenge: Evaluating the state of semantic code search. arXiv preprint 1iXit:1909.09436, 2019. URL https://arxitn wwwpeiiww...ooabiaeeeee www.oogg//////o /1t///99...00 09436.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proc. of the ACL, pp. 1601-1611, Vancouver, Canada, July 2017. URL https::/acllanthooegg org/P17-1147.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proc. of the EMNLP, pp- 6769-6781, Online, November 2020. URL https://aclanthologgy hwwphhologo.oog///880 wtw. http:/22.emmmmmmmaaa....mm emnlp-main. 550/.
Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder, Kaifeng Chen, Sham M. Kakade, Prateek Jain, Neural Information fm355.
and Ali Farhadi. Matryoshka representation learning. In Advances leamnig
Processing Systems, 2022. URL https: //openrv///ee wwtpeeewwwwwweeeeeeeeeeeeeeeeeee met/forumiiceeeee
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszko- reit, Quoc Le, and Slav Petrov. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistias, 7:452-466, 2019, URL https://aclanthologgy hwwp:nnthhooooooooogggoooe 900
Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. NV-embed: Improved techniques for training LLMs as general- ist embedding models. In The Thirleenth International Conference on Learning Representations, 2025a. URL https: /ttp::://pnnrv//////// wtt.peerreviee.netooru/m..22e www.net/orumm2ddlgglsslllll/
Jinhyuk Lee, Feiyang Chen, Sahil Dua, Daniel Cer, Madhuri Shanbhogue, Iftekhar Naim, Gustavo Heránnde Ábrego, Zhe Li, Kaifeng Chen, Henrique Schechter Vera, al. Gemini embedding: Generalizable embeddings from gemini. ar preprint ar XxXo:2203.00881, 2025b. URL https://arxii org/abs. 2503. 0789.17891.
Chaofan Li, Minghao Qin, Shitao Xiao, Jianlyu Chen, Kun Luo, Defu Lian, Yingxia Shao, and Zheng Liu. Making text embedders fhot learners. In The Thirteenth International Conference on Learning Representations, 2025. URL https: /ttp///eeeeeeeeeee/e/ee/ee///.ep net ff wfLuiDjj@u.
11

---

Published as a conference paper at COLM 2025
Mingxin Li, Zhijie Nie, Yanzhao Zhang, Dingkun Long, Richong Zhang, and Pengjun Xie. Improving general text embedding model: Tackling task conflict and data imbalance through model merging, 2024. URL http:///aaxxiv..o/bbbbbe
15035.
Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang.
Towards general text embeddings with multi-stage contrastive learning. ar arv preprint arXiv:2308.0328/ 2023. URL
Yiding Liu, Weixue Lu, Suqi Cheng, Daiting Shi, Shuaiqiang Wang, Zhicong Cheng, and Dawei Yin. Pre-trained language model for web-scale retrieval in baidu search. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, Pp. 3365--377,, Virtual Event, Singapore, 2021. URL https: //to:. orgo //1. 1145/344444444 3467149.
Zuxin Liu, Thai Quoc Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Shirley Kokane, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng, Rithesh R N, Liangwei Yang, Silvio Savarese, Juan Carlos Niebles, Huan Wang, Shelby Heinecke, and Caiming Xiong. APIGen: Au- tomated Plpeline for generating verifiable and diverse function-calling datasets. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks sttt/forum?id=ff3vvaaaia
Track, 2024. URL https: httpss////ppeeeeeeeeeeee
Dingkun Long, Qiong Gao, Kuan Zou, Guangwei Xu, Pengjun Xie, Ruijie Guo, Jian Xu, Guanjun Jiang, Luxi Xing, and Ping Yang. Multi-cpr: A multi domain chinese dataset for passage retrieval. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 9046-3056 Madrid, Spain, 2022. URL https://doi. wwt.///oooooooooooooooe org/10 /0. 1145/3477495. 3531736.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International wet.ooruunfoff
Conference on Learning Representations, 2019. URL Bkg6RiCqY7.
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. MTEB: Massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 2014-2037, Dubrovník, Croatia, May 2023.
URL /atps////lllaahloogoooooo org/ 2023. eacl-main. 148/.
Niklas Muennighoff, Hongjin SU, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. Generative representational instruction tuning. In The Thirteenth International Conference on Learning Representations, 2025. URL Forum?id-BC41IIfSSN
net//
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. MS MARCO: A human generated machine reading comprehension dataset. In Proceedings of the Workshop on Cognitive Computation: Integrating neural and symbolic appronches 2016 co-located with the 30th Annual Conference Neural Information Processing Systems (NIPS 401 Barcelona, Spain, December 9, 2016, volume 1773 of CEUR Workshop 2016 pwp.prr........ .ddf.
Proceedings, 2016. URL https: wttp:///eerwwss.eeee www.ww.w....g.....ooo org//ol-1111
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, dahai li, Zhiyuan Liu, and Maosong Sun. ToolLLM: Facilitating large language models to master 16000+ real-world APIs. In The Tweefth International Conference on Learning Representations, 2024. URL WRR.https://operreiieee
net/foruue d=d jr.
Yifu Qiu, Hongyu Li, Yingqi Qu, Ying Chen, QiaoQiao She, Jing Liu, Hua Wu, and Haifeng Wang. DuReader-retrieval A large-scale Chinese benchmark for passage retrieval from web search engine. In Proc. of the EMNLP, pp. org/2022 ea0.222mmmm-mmm...3.3
Abu Dhabi, United Arab
Emirates, December 2022. URL httts:://aclaaee https://acllathhllogy .acyy.org/a
Hossein A. Rahmani, Xi Wang, Emine Yilmaz, Nick Craswell, Bhaskar Mitra, and Paul Thomas. Syndl: A large-scale synthetic test collection for passage retrieval. In Companion Proceedings of the ACM on Web Conference 2025, pp. 781-784, Sydney NSW, Australia, 2025. URL https://doi.
12

---

Published as a conference paper at COLM 2025
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proc. of the EMNLP, PP. 2383-2392, Austin, Texas, November 2016. URL https://aclanttollee
org/D16-1222 1264.
Saba Sturua, Isabelle Mohr, Mohammad Kalim Akram, Michael Günther, Bo Wang, Markus Krimmel, Feng Wang, Georgios Mastrapas, Andreas Koukounas, Nan Wang, et al. jina- embeddings--3:: Muliilingual embeddings with task lora. arXiv preprint ar arXio:2409.10173, 2024. URL https: //rxiv http://aaaxi...orrb///////// wtt./br////////// 2409. 10173.
Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. One embedder, any task: Instruction- finetuned text embeddings. In Findings of the Association for Computationnl Linguistics: ACL. org/ gr2/3023.
2023, pp. 1102-1121, Toronto, Canada, July 2023. URL findings-acl.71 71/.
Hongjin Su, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han yu Wang, Liu Haisu, Quan Shi, Zachary S Siegel, Michael Tang, Ruoxi Sun, Jinsung Yoon, Sercan O Arik, Danqi Chen, and Tao Yu. BRIGHT: A realistic and challenging benchmark for reasoning- intensive retrieval. In The Thirteenth International Conference on Learning Representations, 2025. URL https: /ttp///eppeeeeeeeeeeeeeeee httt/forummid=ykuucuaa
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL https://oppenreiiet net//forumiidwwCuST5TfF
James Thorne, Andreas Vlachos, Christos Christodooloppoulos and Arpit Mittal. FEVER: large-scale dataset for fact extraction and VERification. In Proc. of the NAACL-HLT, pp. 1074.
809-819, New Orleans, Louisiana, June 2018. URL htttt::://.llanthllgg....oo
Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. Position: Will we run out of data? Limits of LLM scaling based on human- generated data. In Proceedings of the 41st International Conference on Machine Learning, volume 235, pp. 19523-49544 21-27 Jul 2024. URL https://proceedding wwwpeediiiig.mm...........
v235/villalobos24e html.
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by suakly-superrised contrastive pre-training. Xrv preprint arXiv:2212.03533, 2022. URL
http://rrxx....rra/////2222 03533.
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improving text embeddings with large language models. In Proc. of the ACL, pp. 11897- 11916, Bangkok, Thailand, August 2024. htthttps:/ //acl///ccca http//allaalll 642/.
Orion Weller, Benjamin Chang, Sean MacAvaney, Kyle Lo, Arman Cohan, Benjamin Van Durme, Dawn Lawrie, and Luca Soldaini. FollowIR: Evaluating and teaching information retrieval models to follow instructions. In Proc. of the NAACL-HLT, pp- 11926-11942, Albuquerque, New Mexico, April 2025a. URL https://acllathhoogg org/ 2025. maacl-long -97/.
Orion Weller, Benjamin Van Durme, Dawn Lawrie, Ashwin Paranjape, Yuhao Zhang, and Jack Hessel. Promptriever: Instruction-trainet retrievers can be prompted like language models. In The Thirteenth International Conference on Learning Representations, 2025b. URL https://openreview. nett/forum/ddoodvsjjj4
Chenghao Xiao, G Thomas Hudson, and Noura Al Moubayed. Rar-b: Reasoning as retrieval benchmark. Xrv preprint ar XrXio:2404.06347, 2024a. URL httpps://aaxxv..rr//// 2404. 06347.
Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Yun Nie. C-pack: Packed resources for general chinese embeddings. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 641-649, Washington DC, USA, 2024b. URL httph: //doi. org/10 1145/3626722.6677889
13

---

Published as a conference paper at COLM 2025
Xiaohui Xie, Qian Dong, Bingning Wang, Feiyang Lv, Ting Yao, Weinan Gan, Zhijing Wu, Xiangsheng Li, Haitao Li, Yiqun Liu, and Jin Ma. T2ranking: large-scale chinese benchmark for passage ranking. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2681-26900 Taipei, Taiwan, China, 2023. URL https: http:://ooooo/oo/o//o/ org/10. 1145/3539618 3591874.
Qiancheng Xu, Yongqi Li, Heming Xia, and Wenjie Li. Enhancing tool retrieval with iterative feedback from large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 9609-9619, Miami, Florida, USA, November 2024. URL https://aclanthologgy www.://...rg//22/ we..0..f.innninn indings-emmlp.500 561/.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. technical report. arXiv preprint or XXXi:22412.15115 2024. URL https://arxiv.cr wttpr/iv.orr//b///22222 15115.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhut- dinov, and Christopher D. Manning. HotpotQA: dataset for diverse, explainable multi-hop question answering. In Proc. of the EMNLP, pp. 2369-2380, Brussels, Belgium, October-November 2018. URL httts:://acaaneee httpss//llcaanooooogo.ooooo org 0D10/ 1259.
Tianhe Yu, Saurabh Kumar, Abhishek Gerge, Levine, Karol Hausman, and Chelsea Finn. Gradient Surgery for Multi-Task Learning. In Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL https: http://oooeeing.n.eeeee newrips..eeipp....o cc/ paper/2022 hash/ haa.saanneaaaaacfaaaaaaaaa3222a0 Abstrraaaaaacaattttttcccacaaaace Abstract html.
Peitian Zhang, Shitao Xiao, Zheng Liu, Zhicheng Dou, and Yun Nie. Retrieve anything to augment large language models. arv preprint arXi::2310.07554, 2023a. URL https: //arxiv.org/abs//3100 07554.
Sheng Zhang, Xin Zhang, Hui Wang, Lixiang Guo, and Shanshan Liu. Multi-scale attentive interaction networks for chinese medical question answer selection. IEEE Access, 74061-74071, 2018. URL
ieee. www..eeeeeeeeeeeee.......... //ccmmmet///44665.
Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang, Pengjun Xie, Fei Huang, Meishan Zhang, Wenjie Li, and Min Zhang, mGTE: Generalized long-context text representation and reranking models for multilingual text retrieval. In Proc. of the EMNLP: Industry Track, pp. 1393-1412, Miami, Florida, US, November 2024. URL /atp:://aclantholoogo org/ 2024. emwlp-ineeeeeeeeeee industry.103..
Xinyu Zhang, Xueguang Ma, Peng Shi, and Jimmy Lin. Mr. TyDi: A multi-lingual bench- mark for dense retrieval. In Proceedings of the 1st Workshop on Multilingual Representa- tion Learning, pp. 127-137, Punta Cana, Dominican Republic, November 2021. URL https://aclanthology/ org/2021 mr1-1.12/.
Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso- Hermelo, Xiaoguang Li, Qun Liu, Mehdi Rezagholizadeh, and Jimmy Lin. MIRACL: A multilingual retrieval dataset covering 18 diverse languages. Transactions of the Association for Computational Linguistics, 11:1114-1131, 2023b. URL https://acllattholoo o02/2023. tac1-1.63/.
Wayne Xin Zhao, Jing Liu, Ruiyang Ren, and Ji-Rong Wen. Dense text retrieval based on pretrained language models: A survey. ACM Thans. Inf. Syst., 42(4), February 2024. ISSN 1046-8188. URL httph:tts////ttt //tp:::///ooooooooo wttp:/i....g/.... 11111..11/336888111
14

---

Published as a conference paper at COLM 2025
Appendix
A Implementation Details
A.1 Training Data
To build strong general-purpose retriever models, collect various public datasets for training and employ our efficient data synthesis pipeline to generate around 2M synthetic data for basic relevance retrieval. Table presents the detailed datasets.
Cluster
Instances
MS MARCO(2016), Natural Questions (NQ)2201)) TriviaQA HltpotQQA(2018). SQuAD(2016), FEVER(2018), MEDI(2029),
En
AIINLI (2021), DuReader (2022), mMARCO-zh (2021), 12-Ranking(2023),
CmedQAv2((018),, SimCLUE", Multi-CPP((2222), Our synthesized data (2M)
Multilingual
0.5M
Mr. TyDi (2021), MIRACL (2023b), MLDR (2024)
CodeSeerchNeet(211)) MetaTooll(0244))
Code & Tool
1.2M
APIGen(2024) (oolBench(2024)
CIR
1M
Promptriever (2025b)
Table 5: Details of the training data.
A.2 Training Setup
In the Warm-up stage, we sample 25,000 instances from each cluster, totaling 100, 000 instances for training. During the Task Learning stage, the model is trained using the complete dataset from each cluster. In the Model-Merging stage, we sample 1,000 instances from all clusters. For the Enhance stage, fine-tune the model using the same 100,000 instances from the Warm-up stage.
We use dynamic batch size strategy in the training process to speed up the training with long-context instances. Table 6 shows the batch size setting for different models.
For the embedding model, we add MRL loss (Kusupati al., 2022) in the Enhance step to enable the model to generate elastic embeddings, providing the flexibility to adapt the embedding dimensionality based on computational constraints or task requirements.
length
BS
S-BS
BS(R)
S-BS(R)
0-500
512/64/32
128/64
/28/64/32
128/96/64
128/64/32
500-1000
64/32/24
128/64/8
96/48/32
128/64/8
1000-2000
8/4/4
8/4/4
12/6/4
8/4/4
2000-3000
4/4/4
6/3/2
8/6/4
6/3/2
3000-8000
4/2/2
4/2/1
6/4/2
4/2/1
Table 6: Batch size (BS) and sub batch size (S-BS) of different length for embedding (E) and reranker (R) model across the 0.5B, 1.5B, and 7B parameter models
A.3 Data Generation
In this subsection, describe the details of efficient data synthesis pipeline and present a concrete two-step generation example in Figure 5.
Corpus collection We employ doo2query-style method to synthesize relevance retrieval data. To ensure the synthesized data spans various domains and subjects, we collect passages in both Chinese and English from diverse fields. For the Chinese data, refer
15

---

Published as a conference paper at COLM 2025
Query Type
Definition
Length
Keywords
The query is composed of separate set of words that are the center of the passage. Such as: Federal learning, Al security;
(less than less than 7,7 13}
Factual
Ask questions about a sentence in the passage, such as: what, whe.weeeewwwwwh.yy..
(5 20, to 27, more than 20 but less than 50}
Summarization
Summarize the main content of the passage, such
(13 to 21, 20 to 50, more than 80 but less than 100}
as: This passage tells about..:
Yes/No
The query can be answered by yes or no, such as: Is... ?;
(5 to 20, to 27, more than 20 but less than 50}
Background
This type of query consists of background, where the character describes a background (his state, what is being done) and the problem he has encoun- tered, and the provided passage can help him. For example, an intern who is owed wages describes the situation and asks if he can get paid, and the relevant legal documents can help him;
{5 to 20, 25 to 80, more than 80}
Table 7: Five query types, definitions and the optional length
to IndustryCorpus" sampling passages from 17 domains such as Programming, Law, and Sports. For English, select passages from datasets including finwebedu, wiki_6M, arxiv, ccnews, and cnn. The passages are categorized by length and evenly sampled across five intervals: 0-512, 512-1024, 1024-2048, 2048-4096, and 4096-8192 tokens. We collect 2M passages in both Chinese and English, ensuring comprehensive corpus for study.
Role preparation We choose the open-source Personallub dataset as our candidate set for personas, which contains 375,000 detailed role types. These include roles such as 'A scientist who studies the ocean and its ecosystems, focusing on the aphotic zone and its unique creatures. They are likely be interested in the depths of the ocean, the effects of light on marine life, and the survival strategies of creatures in the aphotic zone.' Using a dense retrieval model, we identify the Top-5 personas most likely to be interested in a given passage. Subsequently, we prompt the LLM to select the most suitable persona from these five candidates. The LLM adopts the perspective of this chosen persona to pose a question about the passage. This approach yields more detailed and varied outcomes than directly generating associated personas, thereby enhancing the diversity of the synthesized data.
Query type definition To further enhance the diversity of synthesized data, define five query types: Keywords, Factual, Summarization, Yes/No, Background. Each query type is assigned different optional length, with specific definitions and lengths detailed in Table 7.
Generation details We use the -.wen22.5-72-Innsttree steps, deploying it locally using the vLLM acceleration framework. For generation pa- rameters, we set the temperature to 0.7, top-p to 0.8, and top k to 50. Additionally, to further accelerate inference, we enable prefix caching (enable cenable.prefix.... srrefx_caching-Tiee True) to cache task descriptions and examples within the prompt. Synthesizing queries for two million passages takes approximately 3,000 GPU hours on A100 80G units.
model for the generation in both
A.4 Data statistics
We provide statistics on the synthesized query data in Table 8. The table includes the number of queries generated for each query type, as well as their average lengths.
http:////.....ooaaasasc///////pp whttps://hf.cco//ee////// h-tp:///wwe222----1111----
16

---

Published as a conference paper at COLM 2025
Given a "Passage" and "Character", select the appropriate option from three fields: Character, Question type, Difficulty, and return the result in JSON format. First, select the Character who are likely to be interested the Passage from the candidates. Then select the Question type that the Character might ask about Passage; Finally, choose the Difficulty possible question based the Passage, the Character, the Question type.
Question type: (Keywords Factual Summarization Yes or No Background) Difficully: (High School University PhD)
Here are few examples for your reference: (omitted some space)
Now the "Passage" and the "Character" you with, your output must always JSON object only.
"Passage"
204. (a) All wages, e.medd by any person in any employment are due and payable twice during each calendar month,; ... (Amended by Stats, 2019, Ch. 508. (SB 6omittd some for space) "CCarracte List""
[" new employee, fresh out school, with a strong interest workplace drama." ...." a migrant worker works hard to improve his family life secondary vocational student who just started working Callfomia hotel young employee who struggling the workplace
"Character": secondary vocational student who just started working California hotel "Question type": "Background", "Difficulty": "PhD"
new session
You need to start from the perspective of the following "Charrater"", generate a question or summary based on the "*Passage" and the "Requiment"
Here examples for your reference: ... (omitted
space)
Now you need to generate according to the "CCharacter", "Passage" and "Requirement". "Character"
secondary vocational student who just started working in a Califomia hotel
"Passage"
and payable twice during each calendar month; (Amended by Stats. 2019, Ch. 508. (SB (omitted some for space) Requiment**
204. (a) wages, eamed any person any employment
-Tpe: Background;
Difficulty level: PhD:
-Lnggth the length of the generated sentences should be 20 words;
LLaggunng:: the language which the results are generated should be English;
am secondary vocational student, and now am working hotel California. During the probation period, have been working for 7 days, my salary monthly. resign now, can still get the 7 days' salary?'
Figure A example of synthesizing query data.
Query Type
Keywords
Factual Summarization
Yes/No
Background
Num.
397,079 9.02
400,339 17.29
400,111 88.14
402553
399918
Avg. Token Length
14.15
90.31
Table Statistics the synthesized queries.
17

---

Published as a conference paper at COLM 2025
MTEB English
Param.
Avg.
Avg.
Class.
Clust.
PairC. 3
Rerank. 4
Reir. 15
STS
Summ.
#Datasets (-)
56
12
11
10
1
BGEmalilinuun
9.24B
69.88
66.09
88.08
54.65
85.84
59.72
59.24
83.88
31.20
NV-Embed-v2
7.85B
72.31
67.97
90.37
58.46
88.67
60.65
62.65
84.31
30.7
GritLM-7B
724B
66.80
64.14
79.5
50.6
872
60.3
57.4
83.4
30.4
B5 mistral77
7.11B
66.60
64.31
78,5
50.3
88.3
60.2
56.9
84.6
31.4
GTEqwen2-78
7.62B
69.88
66.30
86.58
56.92
85.90)
61.42
58.86
83.16
31.35
yyche-embeedqwea25555
7.62B
73.00
68.69
90.48
88.96
63.55
62.97
85.38
30.23
GTEQwen2-158
1.54B
67.19
64.43
82.53
48.75
87.52
59.98
58.29
52.81
31.17
Lychee-embedqwwn.....
1.54B
68,39
65.19
86.07
52.80
86.80
61.03
57.15
82.45
30.09
BGE-M37 (Dense)
0.56B
59.84
56.44
73.09
37.27
84.50
55.29
48.73
81.46
31.55
Jina-v3
0.57B
65.52
62.76
82 62.58
45.27
84.01
58.13
53.87
85.80
29.71
Lyche--embedqwen25...88
0.49B
65.73
63.27
82.65
49.80
84.43
59.02
54.49
81.23
31.29
Table 9: Results on MTEB English (Muennighoff et al., 2023). We compare models from the online leaderboard. "Taken from the corresponding paper or model card. "Our run. Avg. means average by task type.
C-MTEB
Param.
Avg. 35
Avg.
Class.
Clust.
PairC.
Rerank.
Retr.
STS
#Datasets
BGE BGEmuliiligual
9.24B
68.4 68.44
69.
74.11
59.30
86.67
68.28
73.73
56.87
mistral77
7.11B
59.92
60.51
72.96
52.30
66.31
61.38
61.75
48.34
GTEqwen2-77
7.62B
71.62
72.19
75.77
66.06
81.16
69.24
75.70
65.20
Lychee-embedowen...
7.62B
73.43
74.91
76.95
80.80
82.43
68.35
76.37
63.58
GT 1.5 Jychee-embedggwen..55
1.54B
67.12
67.79
72.53
54.61
79.50
68.21
71.86
60.05
1.54B
69.77
69.94
74.54
79.13
68.03
66.24
73.00
58.71
BGE-M3" (Dense)
0.56B
61.79
62.94
67.99
45.84
73.98
74.64
65.27
49.94
Lychee-embedlwwen2.05s. 0.49B
69.26
70.52
72.13
76.14
78.08
64.96
71.07
60.70
Table 10: Results on C-MTEB (Xiao et al., 2024b) (MTEB Chinese). We compare models from the online leaderboard. " "Taken from the corresponding paper or model card. 'Our runs. Avg. means average by task type.
B Evaluation Results
For MTEB, CMTEB, MMTEB, MTEB-code, and FollowIR, we we employ the evaluation protocol from the mteb toolkit. For MLDR, ToolBench, and BRIGHT, We list the retrieval instructions for all evaluation subsets in Table 19.
implementations.
B.1 Embedding Benchmarks
Following the practice of build versatile embedding models, we evaluation our embedders and baseline models the Massive Text Embedding Benchmark (Muennighoff et al., 2023) (Table 9) as well as its Chinese (Xiao et al., 2024b) (Table 10) and Multilingual (Enevoldsen et al., 2025) (Table 11) extensions. There retrieval subset are used for reranker evaluation.
B.2 Retrieval Benchmarks
In this section, present the detailed results all retrieval benchmarks evaluated in this paper, i.e., BTTEB-Retrieval/BRE
(Table 12), CMTEB-Retrieval (Table 13), MMTEB-Retrieval (Table 14), MLDR (Chen et al., 2024) (Table 15), WTEB-TTBBBBOOOOO Code (Table 16), FollowIR (Weller et al., 2025a) (Table 17) and BRIGHT' (Su et al., 2025) (Table 18).
18

---

Published as a conference paper at COLM 2025
M-MTEB
Param.
Avg.
Avg.
Btxt.
Class. 43
Clust. 17
InstR.
MultiC. 5
PairC.
Rerank.
Retr. 18
STS
#Datasets (-))
132
13
11
16
9.248
61.25
51,25
67 85
67 67
49.60
-7.12
21.18
80.83
57.80
53.19
73.47
NV-Embed-v2
7,83B
56.25
49.64
57. 37.84
29
41.38
1.04
18.63
78.94
63.82
56.72
71.10
Grit.M-7BB
7.24B
60.93
53,83
70.53
61.83
50.46
3.45
22.77
79.94
63.78
58.31
73.33
E3memmthhh
7:118
53.18
70.58
60.31
51.39
-0.62
22.20
55.75
74,02
GTEgwend 78
7.62B
62.51
36,00
73.92
61.35
$3.36
4.94
25.48
85.13
63.55
60.08
73.98
lychee-embedgwweza.
7.628
65.12
58.20
74.69
63.58
56.26
6.92
26.73
84.23
66.75
66.20
75.21
GTEowent-1.1 26
1.548
52.75
62.5
52.59
81
sechee-embedqwes25...
1.54B
58.43
52.43
59:93
57 67
51.85
5.74
22.45
78.85
64.38
59.28
69.79
BGE-M3 (Dense)
0,56B
59.54
59.28
79:11
60.35
41.79
-3.11
20.10
80.76
62.79
54.59
74.12
Jina
58.77
18.38
lychee-embeddueesz....
0.49B
56.81
50.78
48
54.17
51.07
3.74
21.97
79.12
62.32
57.71
68 94
Table 11:
Results on MMTEB (Enevoldsen et al., 2025) (Multilingual MTEB). We compare models from the online leaderboard. "Taken from the corresponding paper or model card. Denote our runs, Avg." T means average by task type.
BEIR
Aug.
Angu-
Ana
mate-
Rever
Cli-
CQA-
Dup- Stack
DB- Pedia
Fever
FKQA
Hetpot- QA
MAR-
00
NR- Corpus
NQ
Quora
does
Sci-
Sci-
fact
Touche-
2020
Covid
Trec-
BGE
59.24
77.57
28.37
17.94
51.37
90.38
60.01
63.26
65.71
38.11
71.45
90.01
26.93
72.05
30.26
64.27
EV-Embed
70.07
45.39
58.40
53.50
93.75
63.73
85.48
45.63
73.37
89.08
21.90
80.11
31.78
88.85
GTFQmer2 79
42.66
90.10
23.48
79.06
Lychee-embedduueee
62.97
80.02
49.98
52.00
03.13
94.55
71.11
82.20
43.75
43.67
73,86
90.71
38.84
82.32
23.51
62.85
Jychee-rerankkennt55
62.85
80.38
49.83
48.77
55.
92
74.83
65.17
47.15
36 48
70.43
89.31
36.29
82.03
25 68.68
73:45
GTEgwer3:10
$8.29
69.2 72
42,91
44.76
48.69
91.57
54.70
68, 95
43.36
39.38 34
63.99
89
24.98
78
27 89
85.38
Lychee-embedl
57:15
69.25
45.04
47.53
50.35
90.99
53.71
75.50
41.57
36.77
64.91
89.84
26.27
78.33
26
$6.95
Sychee-rerankgwnntss
25.26
BGE-MB (Dense)
Jina-13
gyghee-embedquenakee
$4.49
79.00
41.46
40.83
43.88
91
47.69
69.96
40.24
34.44
59
89
22.88
74 24
20.33
61.61
pychee-rerankkeentie
56.03
60.30
40.82
43.04
48
91.83
46.04
82.67
44.08
31.07
60.47
88
62
21.52
72.42
29.
79.29
Table 12: BEIR benchmark (Thakur et al., 2021). We report nDCG@10 scores.
CMTEB-R
Avg.
Cmedqa Covid
Du
Ecom MMarco Medical
T2
Video
BGE
73.73
42.21
77.46
90.46 69.30
84.70
62.02
86.26
77.40
E5 mistralb7
61.75
34.23
73.11
87.04 45.95
74.84
52.83
80.68
45.34
GTEQWen2-7B
75.71
48.69
81.04
87.44 71.15
85.16
65.59
87.73
78.84
Ours surs:.en2.5-78
76.37
48.64
81.07
88.83 73.99
85.24
66.83
88.03
78.32
Jychee-rerankqwen22-77
79.06
48.46
92.33
91.00 73.77
88.31
68.69
88.88 79.58
GTEQwen2-1.55
71.86
46.97
80.79
89.40 62.51
83.01
58.65
85.47
68.11
Lychee-embedowenn...5.
73.00
43.16
79.02
88.74 69.11
82.03
62.76
83.93
75.26
Jychee-rerankkQwen....55
76.37
45.32
90.84
89.09 69.42
85,59
64.78
84.63
78.24
BGE-M3 (Dense)
65.27
33.76
77.57
83.97 58.44
77.28
54.22
81.38
56.93
Jina-v3
68.54
35.84
78.55
83.13 60.89
79.69
56.64
83.16
70.43
Lychee-embedowen.50ss
71.07
42.71
77.43
87.10
64.55
80.00
59.10
82.73
74.96
Lychee-rerankkgwen-50055
69.79
35.11
90.92
80.15
63.20
79.06
55.83
78.90
75.21
Table 13: The retrieval performance C-MTEB (Xiao et al., 2024b). We report nDCG@10
scores.
19

---

Published as a conference paper at COLM 2025
MMTEB-R
Aug.
AlLA--
Stat-
titee
Argu-
Ada
Bele- bol
Covid
grid
Ha-
LEMB-
Pass-
Key
Legal-
Bench
MIR-
ACE.
MLQQA
docs
Sd-
Spar-
12A
Sock-
Over-
fow
Stat-
dial
TREC. COVID
Temp
Rua-
onL.
Tws Bter
Wikdi
WinG
77.37
92.95
76:90
NV-Embass-c2
70.17
42.75
21:00
92.93
49:74
7:16
34.54
61.65
73.11
98
30.75
94.60
58.03
15.51
91.02
33.41
90.26
39.51
33.77
54.57
81.04
99,05
38.50
81.58
23.48
64.35
2.15
68.64
66.81
Lychee Sydhe--erankkowwasss peaakkesss
Iychee
M6.1
68.77
33.30
48.32
87.50
80.35
77.5
78.30
92.32
61.47
98.90
98.52
39.10
40.01
95.87
99.53
68.75
66.25
72.77
70.00
38.64
35.29
70:46
57.36
95.34
96.40
46.33
$9.10
73.45
63.80
9.25
5.20
09.91
89.52
92.30
90.01
74.9)
73.71
CTEquee in
60.76
34,59
69.72
56.39
50.79
98.69
38.25
94.84
63.23
72.89
21.98
18.63
90.27
33.25
55.35
1.33
67,01
87.31
65.81
Lycher embelguents
99-28
35.05
69.25
60.21
78.36
98.79
38,60
93.55
58.10
60.45
25.41
5,90
93.78
32.27
59.17
0.61
62.56
57.35
38,97
62.47
69.15
98.15
38.50
65.51
65.90
29.07
4.67
09.56
9033
60143
(Dennee
19.92
0.55
73,02
18.57
53.71
V6.57
79.00
58 07
75.6
98.87
38.00
93.67
62.09
49.54
22:99
90.96
22.92
64.04
1.06
90.43
61.36
52.50
58.79
$8.25
60.30
65.76
90,92
98.16
38.50
94.12
89.57
62.51
21.52
21
90.01
36.25
79.26
2.73
46.67
89.64
49.73
Table 14: The retrieval performance on MMTEB (Enevoldsen et al., 2025). We report
nDCG@10 scores.
Max Length
Avg.
de
hi
11
ja
k0
th
zh
BGEmuliligual
8192
49.10
41.97
42.59
44.89
75.97
69. 27
34.80
58.97
50.83
35.88
55.01
32.14
21.89
32768
56.53
53.10
53.91
51.45
78.96
78.23
35.81
67.15
53.55
47.76
78.66
63.88
39.22
31.70
Lychee-embedgwwa..a.
32768
57.90
57.14
53.95
57
84.00
74.28
3188
69.67
55.
48.48
81.05
67 49
38.89
33.30
Jychee-rerankQwm22555
32768
64.50
67.23
59.15
72.46
88.19
83.36
34
74.10
61.42
58.43
84.43
71.37
38.71
45.65
GTE GTEqwen2-18
32768
52.11
50.07
50.91
52
78.67
73.79
23
14
64
19
48.70
42.35
74.40
36
32.53
26.04
sychet-embedgwwnt...
32768
53.85
49.58
51.15
56.11
78.77
71
71.26
69
54
46.91
79.05
63.04
34.34
29.84
Lychee-rerankgwenzzt
32768
64.09
64.04
57.19
61.80
87.01
83 27
33.30
75.62
63.88
59.01
85.31
75.65
40.44
46.54
BGE-M3 (Dense)
46.1
48.9
62.7
59.5
33.6
Jina-v3
8192
40.71
34.39
38.19
29.10
62.05
59
85
25
53
82
38.45
32.34
63.29
49.51
25.55
17.20
Jychee-embeddwwazzaaaa
32768
61.18
26.10
Jychee-eerankgweez50..3
32768
59.74
59.86
53.85
68.43
81.82
82.04
27 82
69
09
57.02
52.51
83.17
64.20
35.02
41.87
Table 15: Evaluation of multilingual long-doc retrieval on the MLDR (Chen et al., 2024)
testset (measured by nDCG@10).
MTBR(Code)
Arg.
Apps
COIR- CodiSourdh
Code-
Code- Fortbacko
Code- Foutbakk
Code- ScardhNat-
SearchNet
Ocetit
Code
Trans-
Code- Trans-
CosQA
Stack Overllow
Synthwtic- Text25QL
Contest
RGF
60.04
22:93
68:14
60.48
60.52
96.70
73.23
83.45
86.54
32.64
27.05
92.05
56.67
Embod/v2
61.85
68.52
88.14
33.40
62.17
25.39
71,79
$7.66
85.15
66.24
81.03
32.17
31.26
kycheesebellguwee
79.53
95.20
89.70
79.50
94.58
89.48
88.53
92.54
94.37
36.97
40.59
96.40
65.05
Jychet-rennkkewa220
M0.28
94.85
90.93
82.87
95.90
90.00
97.00
91.96
94.67
36.97
37.63
95.54
62.93
61.98
25.91
71.56
59.00
49.92
$1.92
72.08
91.08
75:02
32.73
32.25
50:27
54.85 89
OunGeedsiss
72.54
80.11
95.70
34.04
lyche-arrnnkemmss..u
78,05
74.14
88.96
83.91
94.31
54.73
33.13
BGE-MB (Dense)
58,07
53.35
25.37
Ima-93
$8.85
28,99
67.83
57.21
$1.17
77.37
30.91
90.29
kychoesembedlg...
71.06
68.16
35.64
60.0
88.70
85.70
80.21
85.86
62.00
33.98
58.55
96.96
60.63
lychee-rerank
(9.20)
61.52
85.11
72.50
73.01
19.07
90369
09.25
70.84
25.37
36.75
90.10
60.76
Table 16: Performance on MCode) (Enevoldsen et al., 2025). We report nDCG@10 scores.
Follow1R
Avg.
Core17 News21
Robust04
BGEmultilnngall Y
-2.13
-2.99
-0.29
3349
NV-Embed-r2
1.04
3.21
1.58
-1.67
GritLM-7B
3.45
6.70
1.22
2.44
E5 mistrallt
-0.62
3.76
0.74
-6.35
GTEQwen2-78
4.94
6.78
4.11
3.93
Lychee-embeslgwen255.
6.91
8,50
5.71
6.55
Jychee-rerankggent.577
12.11
15.70
16.29
4.35
GTF GTee2-e221158
0.74
2.84
2.95
-3.55
Lychee-embedl
5.74
8.60
4.88
3.73
Lychee-rerankowwnz.. Qwen2.5-1.58
7.38
9.68
6.97
4.73
BGE-M3 (Dense)
-3.11
-1.25
-1.39
-6.69
Jina-v3
-1.34
-0.06
2.36
-6.31
OursQwen25500588 2.24
3.38
2.12
1.22
lychee-rerankkqwe225oos
6.29
9.58
6.45
2.85
Table 17: The retrieval performance on FollowIR (Weller et al., 2025a). We report p-MRR scores.
20

---

Published as a conference paper at COLM 2025
StackExchange
Coding
Theorem-baser
Avg.
Bio.
Earth.
Econ.
Psy.
Rob.
Stack.
Sus.
Leet.
Pony
aops
TheoQ.
Theol]
BGF muutiiilnnll 7
16.42 23.13
15.44
23.62
14.19
14.2 25.96
17.43
14.38
9.52
23.43
14.49
17.66
NV-Embed-v27
22.59 33.38
23.45
26.72
18.29
21.20
19.17
27.52
6.0
9.32
16.77
6.91
19.28
GritLM-7B
25.04 32.77
19.0
19.93
17.31
11.62
18.04
29.85
21.98
8.91
23.34
19.75
20.63
18.84 25.96
15.49
15.79
16.37
9.83
18.52
28.72
4.81
7.1
23.94
25.09
17:54
CTEQwen2-78
32.09 40.66
16.18
26.58
12.82
20.82 13.95
31.07
1.25
15.1
29.9
34.22
22.89
Lychee-embedgwennze
28.05 34.94
23.27
28.58
14.41
29.01 14.46
7.9
18.44
13.52
39.69
41.53
24.
Bechee-rerankownn2...
28.79 34.49
23.16
28.08
18.45
18.93
18.08
39.74
41.68
25.31
GTE 58"
17.56 29.38
25.14
24.8
10.38
10.8
26.44
7.07
16.39
14.87
20.1
18.73
188.47
syche--ebeelowen231..
18.45
30.46
26.25
26.7
11.67
11.99
27.07
17.54
11.55
22.87
21.97
19.47
sychee-rerankqqwns5-11
17.76
19.12
10.64
13.89
15.47
11.62
24.65
2.13
13.15
16.71
25.69
32.53
16.92
BGE-M37 (Dense)
9.47
15.26
11.71
13.42
13.05
9.23
10.12
24.79
14.83
4.56
12.62
4.25
11.94
Jina-v37
13.47
11.46
14.48
8.67
11.57
5.93
8.83
18.07
3.97
13.47
22.79
16.91
11.34
Lychee-embeddwwen2550
8.35
13.20
10.21
8.17
11.46
8.53
21.16
12.57
10.26
13.87
22.57
18:46
13.25
Lychee-rerankkgwnnsssss
8.52
12.24
6.61
8.18
9.97
5.03
18.65
4.26
6.91
13.17
21.58
23.86
11.58
Table 18: scores.
The retrieval performance BRIGHT (Su et al., 2025). We report nDCG@10
Task Name
Instruction Template
BornholmBitettMiningg
Retrieve parallel sentences.
Retrieve parallel sentences
BUCC.v2
Retrieve parallel sentences
DiaBlaBitextMining
Retrieve parallel sentences
HloresiitetMMining
Retrieve parallel sentences
INZ2GenBitextMiinng
Retrieve parallel sentences
MediGennBenchHoorsSttttMMiiigi
Retrieve parallel sentences
NollySentiBtextMMinnng
Retrieve parallel sentences
NowvegaanCouttoiieettiiinnng
Retrieve parallel sentences in Norwegian Bokmál and Nynorsk
NTREXBitextMiiingg
Retrieve parallel sentences
NusaTTransaaiooniittttiiinin
Retrieve parallel sentences
NusaXBitextMining
Retrieve parallel sentences
latoeba
Retrieve parallel sentences
Classify user reviews into positive or negative sen- timent
Classify product reviews into positive negative sentiment
GrveklegalCoddcllastfictto
Given greek legal text, classify its topic Given a Wikipedia articles, categorized it into classes based on its DBpedia ontology. Given financial news, categorized by sentiment into positive, negative, neutral Gvien poem, categorized by sentiment into posi- tive, no impact, negative or mixed Classify the given comments either toxic or not toxic
DBpediaClassificationn
FinancalPhnnssaakkCaasiiatttot
PoemSeentieenCCasstiiccttoo
TixieCooneesaiionClaaiiiccooo
TveefTlpicSnggeClassticctoo
Gvien twitter, classify topic
EstonianValenneClllssiitcaaa
Given news, classify its topic Gvien shopreview, classify its type articles, classify ist topic
GujaratiVeevSllassfictttoo
Given Gujarati
SentimentAnallyisHinddi
Given hindi text, categorized by sentiment into positive, negative or neutral
Inddmsinni@ChsstaiiiCCssssfhhii
Given an Indonesian news headlines, classify its into clickbait or non-clickbait
IaCCaseholdClassffccttion
Given judgments, classify its topic
KorSarcasmClassffcattiio
Given twitter, categorized it into sareasm or
Continued on next page
21

---

Published as a conference paper at COLM 2025
Task Name
Instruction Template
KurisshSnnimeenClaaiiffttti
Given text, categorized by sentiment into positive or negative
NacedoriaavveessniieennCCCssiiaa
Given a Macedonian tweet, categorized by senti- ment into positive, negative, or neutral Given a text, categorized by sentiment into positive, negative, or neutral
AfriSentiClassiiiattio
AmazzuCCountrrfccuulCCCafaffataa
Classify a given Amazon customer review text as either counterfactual or nof-counterfactual
CataloniallveeCClastiiitton
Given tweet, categorized by sentiment into AGAINST, FAVOR NEUTRAL
Cyclliclursicaaagglaaaiiittot
Given text, classify its language Given text, classify its language Classify the News in the given texts into one of the seven category: politics, sports, health, business, entertainment, technology, religion
IndicLangClassfication
MasahaNNEWSClassficattoo
MassivintentCllasficattton
Given user utterance query, find the user tents
MulliHateClassffcatioon
Given text, categorized by sentiment into hate or non-hate
NordicLangClaastication
Classify texts based on language
Given paragraph, classify emotion
NusaX-senti
Given a text, categorized by sentiment into positive negative
ScalaClassificttion
Classify passages in Scandinavian Languages based on linguistic acceptability
SwiisjudgemennCCaaffffttttin
Given a text, categorized into approval or dis- missal
NepaliNeessClassficcttion
Given new, categorized it into business, entertain- OT sports
OdiaNewsClassifiation
Given new, categorized ment or sports
business, entertain-
PurjabiNewsCCassffcttioon
Given new, categorized it into two-classes Classify the sentiment of out-of-domaal (products and school) online reviews
PolEmo2.0-OUT
PAC
Classify the sentence into one of the two types: "BEZPÍECZNE, WWTZPPECZNE.POSTANOWWN WWw.PPTTTTTTTTEELOOOOOO and "KLAUZUU ALBUZZYNAA"
SinhalaNewsClassficatooon
Given new, categorized it into political, business, technology, sports and Entertainment
Given a movie review, categorized it by its senti- ment
SiswatiNewsslassfffcctton
Given new, classify its topic Given movie review, categorized into positive or negative
SwahillNewsClassfficttto
Given new, classify its domain
DalajClassification
Classify texts based on linguistic acceptability in Swedish
IswanaNewsClasstication
Given new, classify its topic
siZuluNNesClaasifccttion
Given a new, classify its topic Identify of Wikipedia articles of cities by country Identify the topic or theme of the given news arti- cles based the titles Identify verses from the Bible in Kalderash Romani by book.
WikiCitiesClusteiing
MasakhaNEWWSCluttrrnnng
RomaniBibleChastering
ArJiVHicnachicaCllsseeinggg
Identify the main and secondary category of Arxiv papers based the titles and abstracts Identify the main and secondary category Arxiv papers based on the titles Identify the category of documents from the Big Patent dataset
ArVitHicaacchccCCltsttinggg
BigPatentCCustering.2
Continued on next page
22

---

Published as a conference paper at COLM 2025
Task Name
Instruction Template
BiorxiiCllsterinng222222
Identify the main category of Biorxiv papers based on the titles and abstracts Identify the main category of Medrxiv papers based on the titles and abstracts
MedrxiiChhaterrnng2222222
StacEExchangeelussterrnggzz
Identify the topic or theme of StackExchange posts based the titles
HALCClsstering5255.22
Identify the topic titles from HAL Identify the category of documents Identify the category of wiki passages Identify categories in Norwegian lexicon Identify the category of tiblestabstracts from Li- brary of Science
SIB20NClusterringg225
WikiClusterrigg/222222
SNJHHienachhcclClussteenngg]"]]"
PlscClustering2222222
SweddnClsserrnggP2P
Identify news categories in Swedish passages Identify the main category of scholar papers based the titles and abstracts
CLSClustering?2Pxx2
StackOverflowOA
Given a question about coding, retrieval code or passage that can solve user's 's question
IwiiterifjerneRettieval
Retrieve answers to questions asked in Danish tweets
ArguAna SCIDOCS
Given claim, find documents that refute the claim Given a scientific paper title, retrieve paper ab- stracts that are cited by the given paper Given the following spatial reasoning question, trieve the right answer.
SpartQA
IempReasonL1
Given the following question about time, retrieve the correct answer.
TRECCOVID
Given query on COVID-19, retrieve documents that answer the query
WinoGrande
Given the following sentence, retrieve an appropri- ate answer to fill Given question on COVID-19, retrieve news arti- cles that answer the question
missing underscored part.
CovidRetrieval
KortkaeSpeccMLLCasssccttiin
Given Korean online news comments, classify its fine-grained hate speech classes Given maltese new, classify its topic
MalteseNewsCClasificatie
Given text, classify its topic Given tweet, classify its topic
CEDRClassification
Given comment as query, find expressed emotions (joy, sadness, surprise, fear, and anger)
CTKFactsNLI
Retrieve semantically similar text
SprintDuplicaeeuuusiios
Retrieve duplicate questions from Sprint forum Retrieve tweets that are semantically similar to the given tweet
TwitterURLCCorpus
PrappaaCP
Retrieve semantically similar text
indonli
Retrieve semantically similar text Retrieve semantically similar text Retrieve semantically similar text Retrieve semantically similar
OpusparcusPC
PawsXPaiiClassification
RTE3
XNLI
Retrieve semantically similar text
PpcPC
Retrieve semantically similar text
TERRa
Given premise, retrieve hypothesis that is en- tailed by the premise
AlloprofReranking
Given a question, retrieve passages that answer the question
VoyageMMMacoRRrankinng
Given Japanese search query, retrieve web pas- sages that answer the question
RuBQReranking
Given question, retrieve Wikipedia passages that answer the question.
12Reranking
Given a Chinese search query, retrieve web pas- sages that answer the question
GermanSTTSBencharrk
Retrieve semantically similar text
Continued on next page
23

---

Published as a conference paper at COLM 2025
Task Name
Instruction Template
SICK-R
Retrieve semantically similar text
STS12
Retrieve semantically similar text
STS13
Retrieve semantically similar text
STS14
Retrieve semantically similar text
STSBenchmark
Retrieve semantically similar text
FaroeseSTS
Retrieve semantically similar text
FinParaSTS
Retrieve semantically similar text
JSICK
Retrieve semantically similar text
IndicCrosslingualSSS
Retrieve semantically similar text
SemRel24STS
Retrieve semantically similar text
STS17
Retrieve semantically similar text
STS22.v2
Retrieve semantically similar text
STSES
Retrieve semantically similar text
STSB
Retrieve semantically similar text
AmazonPolarityylaasficcttoo
Classify Amazon reviews into positive negative sentiment
AmazzoneevieveeClssftttttoo
Classify the given Amazon review into its appro- priate rating category
ArguAna
Given claim, find documents that refute the claim Identify the main and secondary category Arxiv papers based on the titles and abstracts Identify the main and secondary category of Arxiv papers based on the titles Retrieve duplicate questions from AskUbuntu fo- rum
ArxivClusteringE2P
ArxivClustering525
AsSUbuntuuDupQuustions
Banking77Classificttion
Given a online banking query, find the correspond- ing intents
BiorxivClusteringg2P
Identify the main category of Biorxiv papers based on the titles and abstracts Identify the main category of Biorxiv papers based on the titles
BiorxivClusteringS2
ClimateFEVER
Given claim about climate change, retrieve docu- ments that support refute the claim Given query, retrieve relevant entity descriptions from DBPedia
DBPedia
EmottinCClasiiication
Classify the emotion expressed in the given Twitter message into one of the six emotions: anger, fear, joy, love, sadness, and surprise Given claim, retrieve documents that support or refute the claim
FEVER
FIQA2018
Given a financial question, retrieve user replies that best answer the question
HotpotQA
Given a multi-hop question, retrieve documents that can help answer the question Classify the sentiment expressed in the given movie review text from the IMDB dataset Classify the intent domain of the given utterance in lask-oriented conversation Classify the intent the given utterance task- oriented conversation
ImdbClassification
MTOPPomainClassifcattoo
MIOPIntenCCaassiicaattoon
MassivelntenCClassiffcttton
Given user utterance query, find the user in- tents
MassiveScenariiClassficcccoo
Given user utterance as query, find the user sce- narios
MedrxivClusstrrngg27
Identify the main category of Medrxiv papers based the titles and abstracts Identify the main category of Medrxiv papers based on the titles
MedrxivClusteringSSS
MindSmallReranking
Retrieve relevant news articles based on user brows- ing history
Continued on next page
24

---

Published as a conference paper at COLM 2025
Task Name
Instruction Template
NFCorpus
Given question, retrieve relevant documents that best answer the question
NQ
Given question, retrieve Wikipedia passages that answer the question
QuoraRetrieval
Given question, retrieve questions that are seman- tically equivalent to the given question Identify the topic or theme Reddit posts based on the titles
RedditClustering
ReddiiCllsteringgP2
Identify the topic or theme of Reddit posts based the titles and posts
SCIDOCS
Given a scientific paper title, retrieve paper ab- stracts that are cited by the given paper
SICK-R
Retrieve semantically similar text
STS12
Retrieve semantically similar text
STS13
Retrieve semantically similar text
STS14
Retrieve semantically similar text
STSBenchmark
Retrieve semantically similar text
SciDocsRR
Given title of scientific paper, retrieve the titles of other relevant papers
SciFact
Given a scientific claim, retrieve documents that support or refute the claim
SprineDaapicatequessiioss
Retrieve duplicate questions from Sprint forum Tdentify the topic theme of StackFxchange posts based on the titles
StackFachangeClustering
SackkEchangeCUustternngg?
Identify the topic or theme of StackExchange posts based on the given paragraphs Retrieve duplicate questions from StackOverflow forum
StackOverfowDDppQueetttnss
TRECCOVID
Given query on COVID-19, retrieve documents that answer the query
Touche2020
Given question, retrieve detailed and persuasive arguments that answer the question Classify the given comments either toxic not toxic
loxcCCnversatonnCaasffiettttn
IveefSettienntxxrcttooolssssiaii
Classify the sentiment of a given tweet as either positive, negative, or neutral Identify the topic or theme of the given news arti- cles
TventtyeewsrrorpsCCustrrinn
IwitterSSemEval20115
Retrieve tweets that are semantically similar the given tweet
TwiterURRLCorpus
Retrieve tweets that are semantically similar to the given tweet Given a web search query, retrieve relevant pas- sages that answer the query
MSMARCO
AmazzunCoumerricuaalCaaaccccttta
Classify a given Amazon customer review text as either counterfactual or not-counterfactual
STS17
Retrieve semantically similar text Retrieve semantically similar text
SIS22 12Retrieval
Given a Chinese search query, retrieve web pas- sages that answer the question Given web search query, retrieve relevant pas- sages that answer the query Given a Chinese search query, retrieve web pas- sages that answer the question Given question on COVID-19, retrieve news arti- cles that answer the question
MMarcoRetrieval
DuRetrieval
CovidRetrieval
CmedqaRetrieval
Given Chinese community medical question, re- trieve replies that best answer the question Given a user query from an e-commerce website,
EcomRetrieval
retrieve description sentences of relevant products
Continued on next page
25

---

Published as a conference paper at COLM 2025
Task Name
Instruction Template
MedicalRetrieval
Given medical question, retrieve user replies that best answer the question
VideoRetrieval
Given a video search query, retrieve the titles of relevant videas
T2Reranking
Given Chinese search query, retrieve web pas- sages that answer the question
MMarcoReranking
Given a Chinese search query, retrieve web pas- sages that answer the question
CMedQAv1-reranking
Given Chinese community medical question, re- trieve replies that best answer the question Given Chinese community medical question, re- trieve replies that best answer the question Retrieve semantically similar text. Retrieve semantically similar text.
CMedQAv2-rerankinn
Ocnli
Cmnli
CLSClusteringS25
Identify the main category of scholar papers based on the titles
CLSClustering!2P
Identify the main category of scholar papers based on the titles and abstracts Identify the topic or theme of the given news arti- cles based on the titles Identify topic or theme of the given news arti- cles based on the titles and contents
ThuNewsClusteringSSS
ThuNewsClusteringPP
LCQMC
Retrieve semantically similar text Retrieve semantically similar text Retrieve semantically similar Retrieve semantically similar text
PAWSX
AFQMC
QBQIC
TNews
Classify the fine-grained category of the given news title
FFlyTek
Given an App description text, find the appropriate fine-grained category
Waimai
Classify the customer review from a food takeaway platform into positive or negative Classify the customer review for online shopping into positive or negative
OnlineShopping
JDReview
Classify the customer review for iPhone on e- commerce platform into positive or negative Classify sentiment of the customer review into pos- itive, neutral, negative Retrieve semantically similar text Retrieve semantically similar text
Multilingullenntmment
BQ
STSB
MultilingualSentmment
Classify sentiment of the customer review into pos- itive, neutral, negative Given a question about coding, retrieval code or passage that can solve user's question Given a question about coding, retrieval code or passage that can solve user's 's question Given question about coding, retrieval code or passage that can solve user's 's question Given a question about coding, retrieval code or passage that can solve user's 's question Given question about coding, retrieval code passage that can solve user's 's question Given a question about coding, retrieval code or passage that can solve user's question Given a question about coding, retrieval code or passage that can solve user's question Given a question about coding, retrieval code or passage that can solve user's 's question Given a question about coding, retrieval code or passage that can solve user's 's question
AppsRetrieval
CodefEditSearchRettievalll
CodeFeedbackMIT
CodeFeedbackST
CodeSSencchNeeCCRReiieaall
CodeeSearchNeeRetriivalll
CodeITransOccannCnntee
CodeTransOCeanDD..
CosQA
Continued on next page
26

---

Published as a conference paper at COLM 2025
Task Name
Instruction Template
CoURCoadeSaacciieeeeeteevlll
Given question about coding, retrieval code passage that can solve user's question Given a question about coding, retrieval code or passage that can solve user's 's question Given a question about coding, retrieval code or passage that can solve user's 's question Identify the topic of document titles from Allo Prof dataset
StacKOverflowOA
SyntheticTeex25QL.
AlooProChuustrrnng22.22
AlLAStatutes
Identifying the most relevant statutes for given situation
HagridRetrieval
Retrieval the relevant passage for the given query Retrieval the relevant passage for the given query Retrieval the relevant passage for the given query Retrieval the relevant passage for the given query Retrieval the relevant passage for the given query Retrieval the relevant passage for the given query Retrieval the relevant passage for the given query Retrieval the relevant passage for the given query Retrieval the relevant passage for the given query Retrieval the relevant passage for the given query Retrieval the relevant passage for the given query Retrieval the relevant passage for the given query Retrieve semantically similar text Retrieve semantically similar text Retrieve semantically similar text Given a question, retrieve detailed question descrip- tions from Stackexchange that are duplicates to the given question
DggalBenchCorporatel.bbyyy
LEMBPasskeyRetrriva
BelebeleRetrieval
MLQARRetrieval
SatkenDDalouuDDtasseReriiiaa
iikiiettaaetriivalltttllnngggg
Corel7/nstructionkeerieeaal
News22lnsstructionRettiiaaal
Robuss@Annstrrctionnketevaal
Webb
WikipediakerrnningMMttlinnngn
STS15
MIIRACILettievalHHadNNgggggg
BIOSSES
CQADupstackRetrieval
STS15
Retrieve semantically similar text Retrieve semantically similar text Retrieve semantically similar text Retrieve semantically similar text Given Math problem, retrieve relevant examples that help answer the problem
STS16
SummEval
ATEC
aops
biology
Given a post, retrieve relevant passages that help answer the post
earth_science
Given post, retrieve relevant passages that help answer the post Given economics post, retrieve relevant passages that help answer the post
economics
leetcode
Given a coding problem, retrieve relevant examples that help answer the problem
pony
Given question about pony program language, re- trieve relevant passages that helpanswer the ques- tion
psychology
Given a psychology post, retrieve relevant passages that help answer the post
theoremqa questions
Given Math problem, retrieve relevant examples that help answerthe problem Given Math problem, retrieve relevant theorems that help answer theproblem
shooremaattheorentt
robotics
Given robotics post, retrieve relevant passages that help answer the post
stackoverflow
Given stackoverflow past, retrieve relevant pas- sages that help answerthe post Given sustainable living post, retrieve relevant passages that help answer the post
sustainable. living
Table 19: The instruction we used on the evaluation benchmarks.
27


Reviews:
[{"review_id": "0fjJx5nXcq", "reviewer": "Reviewer_uJyM", "summary": "This paper presents a four-stage training framework for adapting LLM-based text retrieval to diverse tasks, including relevance, code, and tool retrieval. It further introduces a role-guided data synthesis method to enhance retrieval performance. Experiments on eight benchmarks across five task types, using Qwen2.5 models (0.5B, 1.5B, 7B), demonstrate better performance over baseline methods in both retrieval and reranking. Ablation studies also show the effectiveness of each training stage and the data synthesis approach.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is clearly written with well-designed and detailed experiments, reflecting the authors' strong efforts in advancing open-source work.\n\n2. The proposed methods demonstrate strong performance improvements and are easy to implement. They are likely to be valuable to the community.", "reasons_to_reject": "The proposed multi-stage training approach—comprising Warm-up, Task Learning, Merge, Enhance, and data synthesis—is not entirely novel, but the authors effectively integrate these components into a cohesive text retrieval pipeline.", "questions_to_authors": "1. Using the number of training iterations to estimate training efficiency is somewhat unconvincing, as it’s not very intuitive. I suggest using GPU time or resource usage as a more accurate and meaningful metric.\n\n2. Do the naive baselines in Table 4 also use the same synthetic data? It would be helpful to include results for the baselines with the synthetic data incorporated for a fair comparison.", "extra_scores": {}}, {"review_id": "odyKatuPtc", "reviewer": "Reviewer_5b9A", "summary": "This paper introduces a novel \"Phased Training Framework\" and an efficient LLM-driven data synthesis pipeline to build general-purpose text retrieval models (embedding and reranking). The framework tackles multi-task conflict and data imbalance via a four-step process: warm-up, task learning, model merging, and enhancement. The methodology leads to state-of-the-art results on a wide variety of retrieval tasks for both embedding and reranking, contributing valuable techniques and practices to the field.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Innovative Unified Architecture: A key contribution of this work is a novel model architecture that natively supports both embedding and reranking functionalities,\n- Effective Phased Training: The proposed multi-stage training addresses multi-task learning challenges in retrieval.\n- Efficient Data Synthesis: Offers a scalable, cost-effective method for high-quality synthetic data generation using open-source LLMs (Qwen-2.5-72B-Instruct).\n- State-of-the-Art Results: Demonstrates superior performance across numerous benchmarks and task types.", "reasons_to_reject": "- Incremental Novelty: The proposed framework largely combines existing techniques (multi-stage training, model merging, synthetic data), potentially lacking fundamental novelty beyond an engineering contribution.\n- Insufficient Comparison Baselines: To showcase the advantage in terms of training efficiency, the paper compares mainly against simple \"naïve\" multi-task training and up-sampling, failing to benchmark against more sophisticated existing strategies, making superiority claims less convincing.", "questions_to_authors": "- What is the reason for “we evaluate on the full set for embedding models and the retrieval subset for reranking”? How does reranking mode perform on non-retrieval tasks?\n- Can you confirm if it is intended that in Eq4 that you sum up the weights of all task vectors instead of averaging?\n- Could you elaborate on the specific methodology for clustering datasets based on \"task similarity and data volume\"?\n- How were the SLERP merging hyperparameters (e.g., t, lambda) determined? Additionally, how does SLERP merging compare to simply averaging the model weights?", "extra_scores": {}}, {"review_id": "mFDITwgac5", "reviewer": "Reviewer_n97a", "summary": "The paper studies the problem of text retrieval across various domains. It aims to resolve challenges in data imbalance and efficiency in training retrieval models. The paper studying namely 5 retrieval tasks spanning relevance retrieval, code retrieval, tool retrieval, complex instruction, and reasoning-intensive retrieval. The paper proposes training the model in a phased manner, which involves:\n1. Warming up: using equal-sized subsets of data.\n2. Task learning, which is analogous to task-specific learning. This is done on the same model across tasks, creating multiple copies of the model.\n3. Merge: the various merged models are then merged using the SLEPT merging technique into a single model.\n4. Enhancement: This involves again training the model over the task-specific data from the warmup stage for inference. \n\nThe authors further also propose generating of synthetic data that can aid in improving the training data. They do this using various query types (keywords, facts, summarization, classification / yes/no , and background. They further set different discovery levels for the query and then make use of an LLM to synthesize the synthetic query data. They also filter the data and consider checks for format validation and consistency using an embedding model. \n\nThe authors provide emperical evidence of outperforming various other models that are in the same parameter count for both tasks, retrieval and reranking.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The problem studied is interesting and shows that phased training provides benefits as compared to other techniques. Further the authors also provide a synthetic dataset that is composed of 2M passages both in english and chinese which can be helpful towards the broader research community.", "reasons_to_reject": "While the paper shows that the proposed technique outperforms the other studied baselines, it lacks the following:\n1. comparison with a pretrained embeddings from the studied model (Qwen-2.5 0.5B, 1.5B, and 7B)\n2. Further, it still remains to be seen if the technique generalizes over other families of models as well. \n3. In Table 4, there is no discussion on how significant the improvements of the proposed phase training framework given that in many cases naive upsampling also does a good job.\n5. The reranker often underperforms the embedding model (Table 3) and only the 7B reranker outperforms the embedder baseline. \n6. Given that there is a factor of difficulty for queries, there could have been a small user study  / semi automated verification done towards accessing that the LLM produces varied difficulty of questions. \n7. Quantitative statistics for the synthetic dataset (like number of tokens per query and other parameters) are missing, which may raise concerns about the synthetic data.", "questions_to_authors": "The paper could improve on the writing, mainly explaining:\n\n1. More on how the cost per training iteration is computed  in section 4.3\n2. Including more details on the synthesis pipeline and statistics regaring question quality and across the various tasks.\n3. Whar metric is used in table 2 ? Is it ndcg@10 ? \n4. The main text should mention that Qwen-2.5-72B instruct was used for synthetic data generation. \n5. There are various models that are trained on the MSMARCO set and evaluated on BIER, why do the authors just compare against the Jina, BGE (gemma and m3) on the reranking task? \n6. What are the insights in the BGE with gemma outperforming in table 3 for the MMTEB-R subset. \n7. For completeness, can the numbers in table 2 be filled in for the CMTEB MLDR, MTEB Code for NV-Embed-v2, GritLM-7B, and E5 mistral?\n8. Given that the phased training setup is more efficient, for the same cost it would be interesting to see the naive baseline numbers.\n9. It would be interesting to see what happens when sematic classes can be clustered together for phased training\n\nPlease also read the weaknesses for other concerns.", "extra_scores": {}}, {"review_id": "ozLuTVFonu", "reviewer": "Reviewer_G5Ni", "summary": "This work proposes a phased training approach to be able to train a model that is successful at first stage retrieval and reranking for many domains. Rather than training all data for all tasks, the paper investigates first warming up the model on a sample of all tasks, then undertaking task learning, and finally merging the models to obtain a single model for retrieval. The tasks are basic relevance, tool retrieval, code retrieval, and instruction-based retrieval.\n\nIn addition to the phased training, the authors utilize synthetic data for training. Synthetic data is generated with a two-step approach. After identifying a document, a representative user is chosen from PersonaHub to describe the searcher. Attributes of the desired query are also specified include the type of query, the difficulty level, the length, and the language to generate the query in. Once the query is generated, it is discarded if it is formatted incorrectly or does not retrieve the source document among the top 20 results.\n\nThe approach is evaluated on 5 tasks including both monolingual retrieval and multilingual retrieval. Given the interest in multilingual retrieval, it is surprising that mFollowIR is not included.\n\nThe paper is well written and generally easy to follow. The approach demonstrates robustness over the various datasets, especially the largest Qwen model that was trained, Qwen2.5-7B.\n\nA surprising result is how ineffective other models are at re-ranking the first-state retrieval result of the author's first stage retrieval (Table 3). Even the largest re-ranker used (BGE_reranker-v2-gemma) frequently performed worse than the run it was reranking. This makes the reader question whether weak baseline systems were selected or whether the documents ranked highly by this approach are substantially different from the approaches where those rerankers perform well. \n\nWhen considering the ablations, the authors appear to brush over the naive approach with up-sampling, but it appears to perform as well as their approach (sometimes a bit better, sometimes a bit more depending on the dataset).   This really needs to be acknowledged in the text. In addition it is unclear if the up-sampling approach utilizes the synthetic data and where the cost of producing that data belongs when considering the training efficiency.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ Phased training is a novel approach to training a robust first stage retriever\n+ The use of personas is synthetic data generation is novel\n+ A large variety of benchmarks were included to the results", "reasons_to_reject": "- The reranking baselines appear to be very weak and likely inflate the gain in effectiveness of this approach\n- The strong performance of up-sampling is overlooked in the prose", "questions_to_authors": "Additional multilingual experiments over NeuCLIR (both news and Chinese technical documents) and well as mFollowIR would strengthen the multilingual results in the paper.", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors propose a method for training a retriever that can handle a disparate array of retrieval tasks: basic relevance retrieval, code retrieval, tool retrieval, complex instruction based retrieval, and reasoning-intensive retrieval. To do so, they propose a "phased training" approach that entails training separate retrievers for each retrieval task and then merging the weights. The authors show that this outperforms baselines and they conduct an ablation study to tease out the source of the improvements. All reviewers appreciated the paper and recommended acceptance. I encourage the authors to take the reviewers' feedback into account for their next iteration, including all of the promised changes and in particular expanding the baselines and clarifying the writing. There have also been some recent retrievers that are not included in the paper (reasonably so, because of the concurrent submission policy) but could be compared to in the next version.
Decision: Accept