NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models
Authors: Chankyu Lee
Rajarshi Roy
Mengyao Xu
Jonathan Raiman
Mohammad Shoeybi
Bryan Catanzaro
Wei Ping
Venue: ICLR 2025 Spotlight (2025)
Keywords: LLM
embedding model
retriever
Primary area: foundation or frontier models, including LLMs

Abstract: Decoder-only large language model (LLM)-based embedding models are beginning to outperform BERT or T5-based embedding models in general-purpose text embedding tasks, including dense vector-based retrieval. In this work, we introduce the NV-Embed model, incorporating architectural designs, training procedures, and curated datasets to significantly enhance the performance of LLM as a versatile embedding model, while maintaining its simplicity and reproducibility.For model architecture, we propose a latent attention layer to obtain pooled embeddings, which consistently improves retrieval and downstream task accuracy compared to mean pooling or using the last <EOS> token embedding from LLMs. To enhance representation learning, we remove the causal attention mask of LLMs during contrastive training. For training algorithm, we introduce a two-stage contrastive instruction-tuning method. It first applies contrastive training with instructions on retrieval datasets, utilizing in-batch negatives and curated hard negative examples. At stage-2, it blends various non-retrieval into instruction tuning, which not only enhances non-retrieval task accuracy but also improves retrieval performance. For training data, we utilize the hard-negative mining, synthetic data generation and existing public available datasets to boost the performance of embedding model. By combining these techniques, our NV-Embed- v1 model secured the No.1 position on the Massive Text Embedding Benchmark (MTEB) (as of May 24, 2024), across 56 embedding tasks. NV-Embed-v2 has reclaimed and maintained the top spot on MTEB since August 30, 2024, demonstrating the sustained effectiveness of the proposed methods over time.  Also, it achieved the highest scores in the Long Doc section and the second-highest scores in the QA section of the AIR Benchmark, which covers a range of out-of-domain information retrieval topics beyond those in MTEB.  We further provide the analysis of model compression techniques for generalist embedding models.  We open-source the model at: https://huggingface.co/nvidia/NV-Embed-v2 .

Published as a conference paper at ICLR 2025
NV-EMBED: IMPROVED TECHNIQUES FOR TRAINING LLMS AS GENERALIST EMBEDDING MODELS
Chankyu Lee +1
Rajarshi Roy
Mengyao Xu
Jonathan Raiman
Mohammad Shoeybi
Bryan Catanzaro
Wei Ping
NVIDIA
ABSTRACT
Decoder-only large language model (LLM))based embedding models are begin- ning to outperform BERT or T5-based embedding models general-purpon text embedding tasks, including dense vector-based retrieval. In this work, we introduce the NV- Embed model, incorporating architectural designs, training procedures, and curated datasets significantly enhance the performance of LLM as versatile embedding model, while maintaining its simplicity and reproducibility. For model architecture, propose latent attention layer to obtain pooled embeddings, which consistently improves retrieval and downstream task accuracy compared to mean pooling using the last <EOS> token embedding from L.Mss. To enhance representation learning, we remove the causal attention mask of LLMs during contrastive training. For training algorithm, introduce two-stage contrastive instruction-tun tuning method. It first applies contrastive training with instructions on retrieval datasets, utilizing in-batch negatives and curated hard negative examples. At stage-2, it blends various non-retrieval into instruction tuning, which not only enhances non-retrieval task accuracy but also improves retrieval performance. For training data, we utilize the hard-negative mining, synthetic data generation and existing public available datasets to boost the performance of embedding model. By combining these techniques, our NV-Embed-v v and NV-Embed-v v2 models obtained the No.1 position on the Massive Text Embedding Benchmark (MTEB) (as of May 24, 2024 and August 30, 2024, respectively) across 56 embedding tasks, demonstrating the sustained effectiveness of the proposed methods over time, Also, it achieved the highest scores in the Long Doc section and the second-highest scores in the QA section of the AIR Benchmark, which covers range of out-of of-domain in- formation retrieval topics beyond those in MTEB. We further provide the analysis of model compression techmiques for generalist embedding models. We open-source the model at: https: /ttp://hu/ggggge ng face. co/ nvii a /VV Embed-va v2.
INTRODUCTION
Embedding or dense vector representation of text (Mikolov et al., 2013; Devlin et al., 2018) encodes its semantic information and can used for many downstream applications, including retrieval, rerank- ing, classification, clustering, and semantic textual similarity tasks. The embedding- also a critical component for strieval-augmennt allows LLMs to access the most up-to-date external proprietary knowledge without modifying the model parameters (Liu et al., 2024; Guu et al., 2020; Shi et al., 2023; Wang et al., 2023a).
retriever generation (RAG) (Lewis et al., 2020), which
The embedding models built on bidirectional language models (Devlin al., 2018; Raffel al., 2020) have dominated the landscape for years (e.g., Reimers & Gurevych, 2019; Gao et al., 2021; Wang al., 2022; Izacard et al., 2021; Ni al., 2021), although one notable exception is Neelakantan et al. (2022). The recent work by Wang et al. (2023b) demonstrates that decoder-only LLMs can outperform frontier bidirectional embedding models (Wang et al., 2022; Ni al., 2021; Chen et al., 2023) in retrieval and general-purposs purpose embedding tasks.
"Correspondence Chankyu Lee echankyu@@nviia.comm @nvidia.com,, Wei Ping <wping @inpinn@nvidiiccom....

---

Published as a conference paper at ICLR 2025
Table 1: Top MTEB leaderboard models as of ICLR submission date (2024-10-01). We use the original model
names on the leaderboard for clarity.
Embedding Task
Retrioval (15)
Rerank (4)
Cluster. (11)
PairClass, (3)
Class. (12)
STS (10)
Summ. 1)
(56)
Mentric
nDCG@
MAP
V-Meas.
Acc.
Spear.
Spear.
NV 22bbeba
62.65
60.65
58,46
$8.67
90.37
84.31
30.7
72.31
Bge age-en-scl (zero shot)
61.67
59, 39.66
57.51
$6.93
88.62
83.74
30.75
71.24
Stella-1 58-x5
61.01
61.21
57.60
88.07
87.63
84.51
31.49
71.19
SER- 2RR
60.18
60.14
56.17
$8.07
$9.05
81.26
30.71
70.31
Qwen2 inninstrct
60.25
61.42
56.92
$5,79
86.58
83.04
31.35
70.24
-V-bbed--vl
59.36
61.59
52,80
86.91
K7.35
82.84
31.2
69.32
Bge-melilinguul-ggoaa
59.24
59.72
54.65
85.84
$8.08
83.88
31.2
69.88
Voyage large 2innstract
58.28
60.09
53.35
89.24
$1.49
84.58
30.84
68.28
SFR- ERR-Eebedding
59.00
60.64
51.67
88.54
78.33
85.05
31.16
67.56
M-B
57.41
60.49
50.61
67.16
79.46
83.35
30.37
66.76
25-miistaal-7b-nnstre
56.9
01.21
78.47
84.66
Bott-ombed-3-aarg (OpenAI)
55.44
59.16
49.01
85.72
75.45
81.73
29.92
64.59
In this work, we introduce NV-Embed, a generalist embedding model that significantly enhances the performance of decoder-only LLMs for embedding and retrieval tasks. Specifically, we make the following contributions:
1. For model architecture, we propose novel latent attention layer to obtain pooled embeddings for a sequence tokens. In contrast to the popular average pooling in bidirectional embedding mod- els (e.g., Wang et al., 2022) and last <EOS> token embedding in decoder-only LLMs (Neclakantan et al., 2022; Wang et al., 2023b), our proposed pooling technique consistently improves accuracy of retrieval and other downstream tasks. To further enhance representation learning, we remove causal attention mask during contrastive training of decoder-only LLM, resulting in solid improvements. Our design is simpler yet more effective compared to related work (BehnamGhader et al., 2024; Muennighoff et al., 2024), which involves an additional training phase with masked token prediction mixed training objective.
2.
For model training, we introduce two-stage contrastive instruction pertuuctionttninng method, starting with the pretrained Mistral-7B (Jiang et al., 2023). In the first stage, we apply contrastive training with instructions on retrieval datasets, utilizing in-hatch negative and curated hard-negative examples. In the second stage, we blend carefully curated non-retrieval datasets into the stage-one training data. Since in-batch negative samples are mislcading for non-retrieval tasks in some cases, we disable in-batch negative training in stage two. This design not only improves the accuracy of classification, clustering, and semantic textual similarity tasks, but also surprisingly enhances retrieval performance. Note, our model is also not fine-tuned from existing embedding models'.
3. Training data is one of the most crucial factors in achieving state-of-the-an results. We provide a detailed recipe on the curation of training datasets, including dataset-specific information, the positive-aware aard-negative mining technique to enhance contrastive training, the synthetic data generation and example-based multi-class labeling. This enables the community to casily reproduce and even surpass our model, ultimately advancing the development of the embedding models.
4. Our NV-Embed-vl model obtained the No. position on the Massive Text Embedding Benchmark (MTEB) (as of May 24, 2024) (Muennighoff al., 2022) across 56 embedding tasks. By improving the curation of the training data, NV- Ti--Embed-y model set a new record high score of 72.31 and reclaimed the No. 1 spot (as of Aug 30, 2024) the highly competitive MTEB leaderboard, further demonstrating the sustained effectiveness of our approach. Note that our model also attains the highest scores in 15 retrieval tasks (commonly referred to as BEIR (Thakur et al., 2021)). 11 clustering tasks, and 12 classification tasks in the MTEB benchmark. See Table for detailed information. Additionally, it secured the highest scores in Long Doc section and the second scores in QA section on the AIR-Benchmark which covers a range of out-of- information retrieval topics beyond those MTEB
5. We study the model compression techniques, including pruning, quantization and knowledge- distillation, for LLM-based embedding models. Through the comparison with smaller embedding models directly built on Llama3.2-3B, Qwen2.5-3B, and Minitron-4B, we demonstrate that our model compression approach achieves superior accuracy and quantization robustness.
We organize the rest of the paper in the following. In $ 2, we discuss the related work. We present the architectural and training method $ 3. We provide detailed recipe of training data curation in $ 4. We present the experiment results in $ 5 and conclude the paper in & 6. Model compression techniques and results are presented in $ A due to the page limit.
results are shown in B.
For example, SFR--Emeedding and Ling- Ling-Embed are fine-tuned from Th-istrutt
2

---

Published as a conference paper at ICLR 2025
2
RELATED WORK
2.1 BIDIRECTIONAL EMBEDDING MODELS
BERT (Devlin et al., 2018) or T5 (Raffel et al., (0200)bassed embedding models have long been the dominant approaches for
embedding tasks. Early examples include Sentence- BERT (Reimers & Gurevych, 2019) and SimCSE (Gao et al., 2021), which finetune BERT on natural language inference (NLI) datasets. In general, these embedding models are first initialized from pre-trained BERT (Wang al., 2022: Izacard et al., 2021) or T5 encoders (Ni et al., 2021). Then, they are further pre-trained with contrastive learning on curated unsupervised (Izacard et al., 2021) or weakly--uperrissed text pairs (Wang et al., 2022). Finally, the embedding models (Li et al., 2023; Wang et al., 2022; Ni et al., 2021; Chen et al., 2023) are fine-tuned on a variety of supervised data, including MS MARCO (Nguyen al., 2016). for retrieval and other downstream tasks. Note that all the state-of-the-ar embedding models are trained in this supervised manner. Some of the most recent frontier models this category include mxbea--embed-large-- (Lee et al.. 2024b) (MTEB: 64.68). UAE-Large- V1 (Li & Li, 2023) (MTEB: 64.64), and voyage-large-2-mme 2-intreet (Voyage-Al 2024) (MTEB: 68.28).
2.2 DECODER-ONLY LLM-BASED EMBEDDING MODELS
Decoder-only LLMs (Brown et al., 2020) were believed to underperform bidirectional models on general-purpose embedding tasks for years, because: i) unidirectional attention limits the representa- tion learning capability, and ii) the scaling of LLMs leads to very high-dimension embeddings, which may suffer from the curse of dimensionality.
The early work by Neelakantan et al. (2022) initializes embedding models using pre-trained, decoder- only GPT-3 models (Brown et al., 2020) and applies continued contrastive training. The hidden state from the final layer, corresponding to the special token <EOS> at the end of the sequence, is used as the embedding for the input sequence. Its latest successor, text-embedding-3-larre
achieves an MTEB score of 64.59 (OpenAI, 2024). Most recently, E5-Mistral (Wang al., 2023b) (MTEB: 66.63) applies contrastive learning with task-specific instructions on Mistral 7B (Jiang et al., 2023). begins to outperform the state-of-the-art bidirectional models on comprehensive embedding benchmarks (Muennighofff et al., 2022) by utilizing massive amount of synthetic data from the proprietary GPT-4 model. LLM2Vec (BehnamGhader et al., 2024) (MTEB score: 65.01) tries to build the embedding model from LLMs while only using public available data, but it is still worse than E5-Mistaal.
Given the success of E5-Mistral, SFR- TRR-EEbeddding-iie Mistral (Meng et al., 2024b) (MTEB: 67.56) and TiR-Embeddiing22 (Meng et al., 2024a) (MTEB: 70.31) further fine-tunes this model on the blend of non-retrieval and retrieval datasets for improved accuracy on both tasks, which is closely related to our WV-Embedd However, there are the following key differences: 1) NV-Embed is trained from scratch Mistral 7B LLM directly using public available data, and not dependent embedding model or proprictary synthetic data. Consequently, we introduce a new architecture that eliminates unnecessary causal attention mask and further improves the sequence pooling mechanism with latent attention layer. 2) FERR-Embedding-MM -Mistral uses mask-homogeneous batching, which constructs batches consisting exclusively of samples from a single task. In contrast, our NV-Embed uses well-blended batches consisting samples from all tasks to avoid potential "zigzag" gradient updates, which leads a new record high score on both full MTEB and retrieval tasks compared to SFR- BeR---meddiig--issee
other
Over the past year, MTEB has become one of the most competitive leaderboards across all AI categories. leading to significantly increased competition among participants. Many of the recent top-performing models (e.g., stella-1.5BB-y gte- Qwwn2-7B-instrueeeeee Soyage-large22-instruct, and text-embed text-embed-3- for reproduction, particularly the blend of training data used. Among the recently disclosed works, GritLM (Muennighoff et al., 2024) (MTEB: 65.66) unifies text embedding and generation into single LLM model. In addition, bge-en-icl (Li et al., 2024) (MTEB: 71.24) enhances query embeddings by introducing few-shot examples the query side, utilizing the in-context learning (ICL) capabilities in text embedding tasks. This approach introduces an overhead by supplying task relevant examples to the query during the training process. To maintain zero-shot evaluation accuracy, both zero-shot
bge- sueemuutilingual-geee have not disclosed key technical details necessary
3

---

Published as a conference paper at ICLR 2025
Latent Array
V
K
Decoder Model Output
Mean
Pool
Inout Token Emb.
Output
[Bidrectional]
Latent Attention
Decoder-only LLM
Block
Figure 1: Proposed architecture design comprising of decoder-only LLM followed by latent attention layer. Latent attention layer functions as form of cross-attention where the decoder-only LLM output serves queries (Q) and trainable latent array passes through the key-value inputs, followed by MLP. Blue dotted lines indicate the two matrix multiplications involved in QKVattenntoons.
and few-shot samples are included during training. our paper, we focus comparing the zero-shot evaluation accuracy of the bge-en-icl model to ensure the fair comparisons during the evaluation phase.
Another area of research focuses on improving data curation processes to enhance the accuracy of fine-tuning retrieval embedding models. Gecko (Lee al., 2024a) (MTEB: 66.31) attempts to distill smaller bidirectional embedding model from a decoder-only LLM (Gemini et al., 2023) by generating synthetic paired data. It refines the data quality by retrieving a set of candidate passages for each query and relabeling the positive and hard negative passages using the LLM. tiig--emedmiiin et al., 2024) utilized LLMs to refine data by generating, filtering, and mining negative samples. Meanwhile, NV-Retriever (Moreira et al., 2024) introduced positive-aware hard-negative mining technique that considers positive relevance scores to more effectively eliminate false negatives. In this work, apply this positive-aware hard- -egativee technique curate the samples and enhance the contrastive training.
(Kim
METHODS
In this section, describe architecture designs and two-stage Instruction-tuning method.
3.1 BIDIRECTIONAL ATTENTION
The causal attention mask in decoder-only LLMs is introduced for next-token prediction task (Vaswani al.. 2017). In principle, causal mask in decoder blocks prevents information leakage by allowing the decoder to attend only to previous positions during auto-regressive text generation. However, it observed that unidirectional attention limits the model's representation power, as evidenced by the poor performance of GPT models compared to similarly sized BERT or T5 models on natural language understanding benchmarks (e.g., Wang et al., 2019). In recent, LLM2Vec (BehnamGhader et al., 2024) introduces additional training phase with specially designed masked token prediction to warm-up the bidirectional attention. GRIT (Muennighoff et al., 2024) utilizes a hybrid objective simply
with both bidirectional representation learning and causal generative training. In contrast, remove the causal attention mask of decoder-only LLM during the contrastive learning and find works compellingly well demonstrated by our results. As result, go with simple solution.
4

---

Published as a conference paper at ICLR 2025
3.2 LATENT ATTENTION LAYER
There two popular methods to obtain the embedding for sequence of tokens: i) mean pooling, and iî) the last <EOS> token embedding. Previous bidirectional embedding models typically use mean pooling (Wang et al., 2022; Izacard et al., 2021), while the last <EOS> token embedding is more popular for decoder-only LLM based embedding models. However, both methods have certain limitations. Mean pooling simply takes the average of token embeddings and may dilute the important information from key phrases, meanwhile the last <EOS> token embedding may suffer from recency bias, relying heavily on the output embedding of last token.
In this work, we propose latent attention layer inspired by Jaegle et al. (2021) to achieve more expressive pooling of the sequences for general-purpose embedding tasks. Specifically, we denote the last layer hidden from decoder as the query Q E Rixd where is the length of sequence, and d is the hidden dimension. They are sent to attend the latent array K V e Rexd which are trainable "dictionary" used to obtain better representation, where r is the number of latents in the dictionary. The output this cross-attention is O Igt
softmax(QK"))
(I)
which is followed by regular MLP consists of two linear transformations with GELU activation in between. Our model uses latent attention layer with r of 512 and the number of heads as 8 for multi-head attention. Finally, we apply mean pooling after MLP layers obtain the embedding of whole sequences. See Figure I for an illustration. It is worth mentioning here that our approach follows the spirit of dictionary learning to obtain better representation (e.g., Wang et al., 2018), which is different from the Perceiver IO architecture. We compare the proposed latent attention layer with normal self-attention and find consistent improvements our ablation study.
3.3 TWO-STAGE INSTRUCTION- TUNNNG
Instruction tuunonn has been widely applied for training LLM to follow instructions (Wei et al., 2021; Ouyang et al., 2022) and to perform retrieval-augmentee generation (Wang et al., 2023a; Liu et al., 2024). has also been recently applied for training retrievers and general-purpose embedding models that can adapt their output embeddings with different instructions and task types (Asai al., 2022; Wang et al., 2023b).
To obtain generalist embedding model that can appropriately perform on retrieval and non-retrieval tasks (e.g., classification, clustering). we need take the characteristics of different tasks into account. For example, the use of in-batch negatives has been demonstrated to be highly efficient for training bense-embeddinng-baee based retrievers (e.g., Karpukhin et al., 2020), because it allows to reuse the computation and effectively train on B² question/passage pairs for each mini-batch with only B questions and corresponding positive passages. However, applying in-batch negatives trick can mislead the embedding model for classification or clustering task, the "passages" in the mini-batch may come from the the class and are not negatives.
Given these considerations, we introduce two-stage instruction tuning method which first conducts contrastive training with instructions on a variety of retrieval datasets (details are in section 4.1), utilizing in-batch negatives and curated hard-negative examples. the second stage, we perform contrastive instruction
on combination of retrieval and non-retrieval datasets (details are in section 4.2) without applying the trick of in-batch negatives. is worth mentioning here that retrieval task presents greater difficulty compared to the other tasks so that our training strategy focuses on fine-tuning the model for retrieval initially. In second stage, we blend the remaining embedding tasks into the cioruction-tuningg
4 TRAINING DATA
For training data, we employ public retrieval and non-retrieval datasets and synthetically generated samples to demonstrate model's capability in embedding tasks. Our training procedure incorpo- rates both retrieval and non-retrieval tasks including classification, clustering, and semantic textual similarity datasets.
5

---

Published as a conference paper at ICLR 2025
Given relevant query-document pair, the instructed query follows the instruction template follows:
Oins = Instruct (task_definitionn -
Query : q*
(2)
The instruction templates for each (task_definise sel_eiiinniionn are provided in Table 12 for training and Table 13 for evaluation. Note, we mask out the instruction tokens in the output embeddings during both training and evaluation, although they still impact the output due to self-attention. We do not add any instruction prefix to document corpus.
4.1 PUBLIC RETRIEVAL DATASETS
We adopt the retrieval datasets as follows: MSMARCO (Bajaj et al., 2016), HotpotQA (Yang et al., 2018), Natural Question (K (Kwiatkowski et al., 2019), PAQ (Lewis al., 2021), Stack Exchange (Stack- 2023), Natural Language Inference (Group et al., 2022), SQuAD (Rajpurkar al., 20, ArguAna (Wachsmuth al., 2018), BioASQ (Tsatsaronis et al., 2015), FiQA (Maia et al., 20, FEVER (Thorne et al., 2018), HoVer (Jiang et al., 2020), SciFact (Wadden et al., 2022), NFCorpus, MIRACL (Zhang et al., 2023) and Mr. Mr.TyDi (Zhang al., 2021).
It is important to note that certain datasets (e.g., MSMARCO) are training splits of the MTEB Benehmark, which we follow the existing practices established by leading generalist embedding models (Meng et al., 2024b; Wang et al., 2023b; BehnamGhader et al., 2024; Muennighoff et al., 2024). Table 12 further provides the number of samples used for training. We demonstrate the zero-shot generalization capability of NV-Embed on AlR-bench in B.
4.1.1 HARDNEGATIVE MINING TECHNIQUE
Embedding models are trained using contrastive learning (Gao et al., 2021), aiming to increase the similarity between the embeddings of query and its relevant passages (positives) while reducing the similarity with irrelevant passages (negatives). Public retrieval datasets typically only contains the positive query-passage pairs but do not contain its own hardnegatives, making it necessary to mine of such negative examples. To address this, we apply the recently proposed positive- aware hard-negative technique (Moreira al., 2024) that considers the positive relevance scores for better false negatives removal. Following the ablation studies in Moreira et al. (2024), we use E5 E5-misttal--7--insrrut (Wang et al., 2023b) as teacher retrieval model to identify the optimal hardnegative passages relevant to the query. We set the maximum threshold for negative scores based on a percentage the positive score (TopXPercPos) with 95% margin, described as follows: max wegative_scoeeeeoeoeeeeeooooeooe wtt.sscoretthshhhlleo threshold pos_score percentage_marre
4.2 PUBLIC NON-RETRIEVAL DATASETS
Besides retrieval datasets, we utilize public non-retrieval datasets mainly from three sub-tasks in MTEB benchmark: classification, clustering and semantic similarity (STS). We pre-process the format of these datasets to become the compatible with retrieval datasets for contrastive training: d,
query gt. positive document d+ and hard negative documents (d,,...,,,,)
For classification, we utilize the English training splits of various datasets from MTEB Huggingface datasets (Muennighoff al.. 2022: Lhoest al., 2021). The classification datasets that we use are as follows: AmazonRevicws (McAuley & Leskovec, 2013a), AmazonCCunterraatuua (O (O'Neill al., 2021), Banking77 (Casanueva et al., 2020), Emotion (Saravia et al., 2018), IMDB (Maas et al., 2011), TTToPDomai//MTOPinieee (Li et al., 2021), ToxicConversations (Adams et al., 2019), TweetSentimennEXxtaaiio (Maggie, 2020), AmazonPolarig (McAuley & Leskovee, 2013b), Mas- (FitzGerald et al., 2022). For the Emotion and AmazonCounterfactuul classification datasets we use BM25 (Robertson et al., 2009) similarity thresholds to filter out training data that is similar to the MTEB evaluation set.
diveScenario/Massieeehe
For clustering datasets, we utilize the raw_arxiv, raw biorxiv and raw raw_medrxiv datasets from MTEB Huggingface datasets,
(Lang, 1995), Reddit (Geigle et al., 2021), StackEx- change (Geigle et al., 2021), RedditP2P (Reimers, 2021b) and StackExchangeP2P (Reimers, 2021a) We filter out any training data that match the MTEB evaluation set.
The classification and clustering datasets provide examples and corresponding class/cluster labels. The example texts extracted from the appropriate teal/lätlellabstrad field used for the query

---

Published as a conference paper at ICLR 2025
q². For binary classification tasks the label texts used documents d',dd .d" For multi-class classification and clustering tasks, a randomly sampled example from the ground-truth class/cluster is used for the positive document d' and randomly sampled examples from other classes/clusters are used for negative documents dz- We will present ablation experiments supporting this approach in section 5.2.4.
For semantic textual similarity datasets, we use the training splits of three semantic similarity datasets STSI2 (Agirre et al., 2012), STS22 (Chen et al., 2022), #TS-Benchmark (Cer et al., 2017) from MTEB Huggingface datasets. For any pair of texts with associated relevance scores (fa, la, score), we create two examples (q tu, dt = to) and (q to. d tu if score 2 4. We mine the hard negatives d from the pool of other texts using the same technique as section 4.1.1. Task instructions are appended to d+, d since they are symmmetric with the query.
4.3 SYNTHETIC TASKS DATASET
Due to the limited variety of subjects and tasks in public training datasets, the available instruction templates for training are also restricted. To enhance task-wise generalization, we employ the Mixtral-8x22B- Intttu--ctttt--O.. model (MistralAI) to create a dataset consisting of 120,000 synthetic examples across 60,000 synthetic tasks. Following a two-step prompting approach proposed by E5-mistral-7b-innttu (Wang et al., 2023b), we adjust the prompts for Mixtral-8x22B-- InstBB-Istruct-OO. and English text. We generate only the short-long, long-short, and short-short examples (40,000 of each), as we use public STS datasets and do not assess bitext retrieval tasks. Example prompts for synthetic data generation can be found in Appendix 15 and 16.
5 EXPERIMENTS
Training and inference experiment details are illustrated in Appendix C.
5.1 MTEB RESULTS
We evaluate the proposed NV-Embed model on the full MTEB benchmark (Muennighoff et al., 2022) across 56 tasks. Table summarizes averaged MTEB scores for seven sub-category tasks compared frontier models on MTEB leaderboard Our initial model, namely NV WW-mmbed-l get the score of 69.32 and obtain the No.1 position on the MTEB as of May 24, 2024 (detailed benchmark scores available in Table 2). We then further improve the model through the curation of training dataset, including adding more retrieval datasets, applying positive-aware using synthetic data generation process and constructing example-based multi-class labels. As a result, our NV-Embed-v2 model sets a new record high score of 72.31 and reclaimed No. 1 (as of Aug 30, 2024) on highly competitive MTEB leaderboard, further highlighting the sustained effectiveness of the proposed methods. In following sub-section 5.2, we will present ablation studies design choices regarding the model architecture, training algorithm and the curation training data.
mining technique,
Based on quantitative leaderboard results, we compare our NV-Embed with the recent frontier embedding models. The e5-mistral-77 -b-isttrtt (Wang et al., 2023b) and google-gecko (Lee et al., 2024a) utilize proprictary synthetic data to train their model in single stage manner. In contrast, we recognize that retrieval task presents greater difficulty compared to the other embedding tasks and prioritizes our training strategy on fine-tuning the model for retrieval first, followed by blending the remaining sub-tasks into instuccioontuning leading to substantially improved BEIR and overall MTEB results.
SFR-Emmeedding-2R (Meng et al., 2024b) demonstrates competitive scores on the MTEB (70.31) and BEIR (60.18) benchmarks by continuing finetune the e5-mistral-7b-innte model (Wang et al., 2023b). However, it remains largely constrained by the architectural limitations of its parent model, such as the causal attention mask and the last token pooling method. In contrast, our NV-Embed model is trained starting from the Mistral 7B LLM (Jiang et al., 2023) rather than finetuning e5- (Wang al., 2023b). It features new architecture that removes the unnecessary causal attention mask and further improves the sequence pooling mechanism with latent attention layer. Table 3 and 14 provides a detailed scores BEIR and MTEB benchmarks.
Phtpps: /tttps://git thub. com embedd i ngs benchma rk /mteb
7

---

Published as a conference paper at ICLR 2025
Table 2: Averaged MTEB scores seven tasks after first and second stage training using only the publically available data and before applying the positive-aware hardnegative mining, synthetic data and example-bas basel multi-class labeling. The averaged score 69.32 corresponds NV-Embed-vl.
First stage training
Pool Type
EOS
Mean
Latent-attention
Self-attention
Mask Type
bidirect
causal
bidirect
causal
bidirect
causal
bidirect
causal
(1)))
57.70
56.42
58.42
57.55
59.00
57.65
57.89
57.21
Rerank (4)
59.76
57.21
60.02
59.35
59.59
59.72
59.73
59.51
Clustering (11)
44.75
40.83
45.97
45.42
45.44
45.61
45.19
45.07
PairClass. (3)
86.17
83.63
87.45
84,46
87.59
82.02
86.51
85.74
Classification (12)
73.17
69.22
74.62
72.48
73.93
72.74
73.54
73.32
STS (10)
74.96
73.45
77.47
73.60
79.07
78.65
76.89
77.55
Summar, (1)
29.28
28.4
29.72
30.89
30.16
30.94
30.22
31,59
Average (56)
62.68
60.06
64.00
62.32
64.18
63.39
63.27
63.11
Second stage training
Pool Type
EOS
Mean
Latent-attention
Self-attention
Mask Type
bidirect
causal
hidirect
causal
bidirect
causal
bidirect
causal
Retrieval (15)
58.39
56.59
58.71
57.88
59.36
58.33
58.64
57.71
Rerank (4)
60.37
59.23
60.77
60.27
60.54
60.57
60.5
60.38
Clustering (11)
51.43
49.81
52.80
51.58
52.80
51.7
53.34
51,51
PairClass. (3)
84.06
80.99
87.45
82.89
86.91
83.45
86.12
84.44
Classification (12)
85.85
85.04
87.06
86.08
87.35
86.58
86.76
86.25
STS (10)
79.55
79.12
82,53
81.74
82.84
81.94
82.38
81,52
Summar, (1)
30.36
29.12
30.49
31.82
31,20
31.87
30.105
31.4
Average (56)
67.85
66.50
68.97
68.13
69.32
68.47
69.10
68,16
Table 3: Averaged MTEB scores on seven embedding tasks after two stage training after applying the positive-aware hardnegative mining, synthetic data and example- based multi-clas labeling. Note, the averaged score 72.31 corresponds to NV-Embed-v2.
Pool Type
EOS
Mean
Latent- -ttenttnt
Self-attention
Mask Type
bidirect
causal
bidirect
causal
bidirect
causal
bidirect
causal
Retrieval (15)
62.13
60.30
61.81
61.01
62.65
61.15
61.17
60.53
Rerank (4)
60.02
59.13
60.65
59.10
60.65
59.36
60.67
59.67
Clustering (11)
58.24
57.11
57.44
57.34
58.46
57.80
58.24
57.11
PairClass, (3)
87.69
85.05
87.35
87.35
88.67
87.22
87.69
85.05
Classification (12)
90. 10
90.01
89.49
89.85
90.37
90.49
90.10
90.01
STS (10)
82.27
81.65
84.35
84.35
84.31
84.
84,22
83,81
Summar. (1)
30.25
32.75
30.75
30.88
30.70
30.90
30.93
31.36
Average (56)
71.63
70.85
71.71
71.38
72.31
71.61
71.61
70.6
5.2 ABLATION STUDY
We conduct ablation studies to compare several training, architecture and data curation design choices: two-stage training, bidirectional attention, iatent-attention pooling method, synthetic data and example-bart based multi-class labeling.
5.2.1 TWO-STAGE TRAINING
We compare the two-stage and single-stage training with and without the use of the in-batch negative technique, as shown in Table 4. We observe that our proposed two-stage training surpasses single- stage training because it allows the use of beneficial in-batch negatives for retrieval tasks in the first stage, while disabling the in-batch technique for non-retrieval tasks in the second stage. In contrast, single-stage training with in-batch negatives leads to significantly lower MTEB performance, especially in the classification sub-task. This accuracy degradation occurs because many classification tasks involve few-class labels (such binary labels like True/False), meaning that the inbatch negative labels in the batch can actually be the positive label. While single-stage training without in-batch negatives produces more comparable results (MTEB scores: 72.31 for two-stage training vs. 71.94 for single-stage without in bathh, two-stage training significantly outperforms in the retrieval sub-tasks (BEIR scores: 62.65 for two-stage training vs. 61.37 37 for single-stage without in-batch). It is worth
8

---

Published as a conference paper at ICLR 2025
Table 4: Averaged MTEB scores ablation studies for NV-Embed-v2: two stage training, multi- class data labeling, positive-aware hardnegattiv mining and synthetically generated dataset. In the third part of the table, HN represents hardnegative mining technique, AD means adding public retrieval datasets and SD refers to adding synthetically generated data. In the fourth part of the table, we also include NV-Embed-vl, which omits HN, AD, and SD in stage-one training and uses label-based approach in stage-two training.
Section 5,3.1
Two stage training
Embedding Task
Retrivaal
Rerank
Cluster.
PairClass.
Class.
STS
Summ.
Avg.
Single Stage (Inbatch Enabled)
61.25
57.67
83.7
30.75
Single Stage (Inbstch Disabled)
61.37
60.81
58.31
88.3
90.2
84.5
30.96
71.94
Two Stage Training
62.65
60.65
58.46
88.67
90.37
84.31
30.70
72.31
Reversed Two Stage
61.91
60.98
58.22
88.59
90.26
83.07
31.28
71.85
Section 5,3, Multi-class Classifiation and Clustering Labels stage-two training
Embedding lask
Retreval
Kerank
Cluster.
PairClass.
Class.
STS
Summ.
Avg.
Label-based approach Example-based approach
62.40
59.7
53.04
88.04
89.17
84.25
30.77
70.82
62.65
60.65
58.46
88.67
90.37
84.31
30.70
72.31
Section 5.3.5 Hard-negativv mining and Syuthetically Generated Dataset in stage-ome training
Embedding Task
Retrieval
Rerank
Cluster.
PairClass.
Class.
STS
Summ.
Avg.
[80] Without HN, Without AD, Without SD
59.22
59.85
57.95
85.79
90.71
81.98
29.87
70.73
[8 HN, Without Without SD
61.52
58.01
88.56
90.31
84.26
30.36
With HN, With Without SD
62.28
60.45
58.16
88.38
90.34
84.11
29.95
72.07
[3] With HN, With AD, With SD
62.65
60.65
58.46
88.67
90.37
84.31
30.70
72.31
NV NV-Embed-vl
Label-based approach [SD]
87.35
82.84
highlighting here that the retrieval is considered the most crucial sub-category for the advancement of RAG technology across the MTEB embedding tasks.
Lastly, we explore another research question: what happens the order of two-stage training is reversed? To examine this, further finetune the Single Stage (Inbatch disabled) model using only the retrieval datasets with enabling the inbatch negative technique and present the MTEB results in Table 4. While the retrieval score increased from 61.37 61.91 after the reversed two-staged training, it remains lower than the retrieval score of 62.65 achieved with our proposed two-stage training method. Furthermore, the scores on other embedding tasks, such as Clustering and STS, declined compared to the Single Stage (Inbatch disabled) approach. Consequently, the overall MTEB score for Reversed Two Stage (score: 71.85) lower than our proposed Two-Stage Training (score: 72.31) as well as the Single Stage with Inbatch disabled (score: 71.94).
5.2.2 CAUSAL ATTENTION VS. BIDIRECTIONAL ATTENTION
To examine the impact of self-attention masks in decoder-only LLM models for embedding applica- tions, we conducted experiments comparing bidirectional and causal mask types. As illustrated in Tables 2 and 3, the bidirectional mask consistently outperforms the causal mask based on the average MTEB scores across 56 tasks for all pooling types. This indicates that embeddings generated with causal attention masks are significantly less effective than those produced with bidirectional attention masks.
5.2.3 POOLING METHODS
To examine the impact of different pooling methods on embedding models, we conducted experiments comparing <ROs>-last, mean, latent-attention, and self-attention pooling types. As depicted in Tables and 3, mean pooling consistently outperforms <ROS>-last token embedding based on the average MTEB scores across 56 tasks. This difference may be due to the last <EOS> token embedding being influenced by recency bias, showing an excessive dependence on the output of the final token.
To enhance performance beyond mean pooling, we experimented with adding the proposed latent- attention or self-attention layer (both followed by MLP) before mean pooling to address the issue of important information from key phrases being diluted. According to Tables 2, self-attention does not provide additional accuracy improvements for the embedding capabilities of decoder-only LLMs (i.e., mean pooling 68.97 vs. self-attention 69. 10 MTEB tasks). It even slightly reduces accuracy

---

Published as a conference paper at ICLR 2025
on 15 retrieval tasks (i.e., mean pooling 58.71 vs. self-attention 58.64). Table 3 also shows the similar trends of NV-Embed-v2. This is not surprising. as the LLM already has many self-attention layers to learn the representation, and adding an additional one does not bring significant additive value.
In contrast, the latent-attention layer proved beneficial for majority of embedding tasks, as shown in Table 2 and 3. Specifically, the nDCG@ 10 accuracy of the more challenging 15 retrieval tasks improved (i.e., mean pooling 61.82 vs. latent-attention 62.65) in Table 3. We hypothesize that this is due to the "dictionary learning" " provided by the latent array, which offers more expressive representation. The latent-attention layer effectively learns output embedding representations from decoder-only LLMs, mitigating the information dilution caused by averaging the output embeddings.
5.2.4 MULTI-CLASS CLASSIFICATION AND CLUSTERING LABELS
We compare the effect of using two possible tech- niques for constructing positive and negative docu- ments for multi-class classification and clustering tasks. In label-based approach, the ground-truth class/cluster label corresponding to the example in the query is used the positive document, and other class/cluster labels are sampled for negative approach, another
Table 5: Ablation study on using class/cluster
labels vs. sampled class/cluster examples as positive and negative documents for multi-class classification and clustering tasks.
+/- Document Format
Labels
Examples
Emotion- Aenotion-Claseee Classification
90.83
93.38
Massivelntent- Classnsitaassiitiiionn
84.94
86.10
documents. In example-base example from the same class/cluster as the exam- ple in the query is used as the positive document, and examples from other clusters are sampled for negative documents. We use random sampling to get broad coverage across labels and exam- ples. In this work, all 11 clustering datasets and muti-class classification datasets constructed as example-based approach. As shown in Table 4, the example-based approach leads to significant improvements over the label-based approach for both classification and clustering. Table 5 further shows the detailed ablation study of label-based and example- based labels for classification and clustering multi-class samples.
MassiveScenario www.sSeeearrioCsssicaaaiii
90.18
92.17
MTOPDomain Classfiaacasiiiitiiion
98
99.25
MTOOIinenn-Claasificcciioo
88.55
94.37
Arxiv-Clusse Setr--Custerrng-222
53.01
55,80
Arxiv- Seii-CCustering-SS
49.19
51.26
45.38
$4.09
Biorxiv- Seri--Cluteerin-SS55
42.67
49.60
Medrxiv- Medrxiv-Clustering- Clussteiitnn--222
37.58
46.09
Medrxiv-Clustenin
36.82
44.86
Reddit- ModiitClustering
59.83
71.10
Reddit- Caadii-Cluteerng-PPP.
72.58
74.94
StackExchange Eechaage---Cuusteriing
79.37
10
48.59
48.36
TwentyNewsgrouppe Clusttrrig
58.41
64.82
Average (16)
64.80
69.27
5.2.5 HARDNEGATIVE MINING AND SYNTHETICALLY GENERATED DATASET
We provide step-by-step curation of training dataset, incorporating the hard negative mining technique (S1), additional public retrieval data (S2), and synthetically generated data (S3). As shown in Table 4, the first step of adding the hard negative mining technique significantly boosted retrieval accuracy, with the BEIR score increasing from 59.22 to 61.52. In the next step (S2), we included more public retrieval datasets (HoVer, SeiFact, Nfcorpus, MIRACI, Mr.Tydi) followed by synthetically generated data. Adding the public retrieval datasets further increased the retrieval score by 0.7 points. Finally, incorporating the synthetic dataset (S3) leads a modest improvement in the overall MTEB scores, raising them by 0.24 points.
6 CONCLUSION
We introduced the NV-Embed model, a decoder-only LLM designed to outperform existing bidi- rectional models in general-purpose text embedding tasks. For model architecture, we propose a latent attention layer to obtain expressive pooled embeddings and remove the unnecessary causal attention mask of decoder-only LLMs. For training algorithm, we introduce a two-stage contrastive instruction-tuning scheme to sequentially improve the embedding tasks. By leveraging carefully curated datasets, hard-negative mining, synthetic data generation and example- cramele-based multi-class labeling, our approach achieve the superior accuracy across diverse embedding tasks. As result, the series of NV NV-Embed models achieved and maintained the No.1 1 ranking on the MTEB leaderboard and also demonstrated superior accuracy in out-of-dommin tasks in AIR Benchmark.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGMENT
We would like to extend our sincere gratitude to the NVIDIA Merlin team for their valuable collabo- ration and insightful discussions on building embedding and retriever models. We especially wish to thank Benedikt Schifferer, Gabriel de Souza P. Moreira, Radek Osmulski, Mengyao Xu, Ronay Ak, and Even Oldridge for providing the data from NV-Retriever (Moreira et al., 2024).
REFERENCES
C.J. Adams, Daniel Borkan, Jeffrey Sorensen, Lucas Dixon, Lucy Vasserman, and Nithum Thain. Jig- saw unintended bias in toxicity classification, 2019. URL pestps://kaggLaa com/ competitioniin in wtw.--ttoiittt----aaalatatataaae
A.ggaw--uninteeded-bile
Encko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. SemEval-2012 task 6: A pilot on semantic textual similarity. In Eneko Agirre, Johan Bos, Mona Diab, Suresh Manandhar, Yuval Marton, and Deniz Yuret (eds.), *SEM 2012: The First Joint Conference on Lexical and Computational Semantics - Volume I: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pp. 385 393, Montréal, Canada, 7-8 June 2012. Association for Computational Linguistics. URL ht sttp::/ htt//aaaanattttaeeaeaaaaaaaatata Logy erg/s12-1 1051.
Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard, Sebastian Ricdel, Hannanch Hajishirzi, and Wen-tau Yih. Task-aware retrieval with instructions. arXiv preprint arXiv:2211.099 122222111.920 1.09260, 2022.
Payal Bajaj. Daniel Campos, Nick Craswell, Li Deng. Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. Ms marco: A human generated machine reading comprehension dataset. arxiv preprint /6/ 1.09268, 2016.
Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. LIm2vec: Large language models are secretly powerful text encoders. arXiv preprint arXiv:2404.05500 055661 2024.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
Inigo Casanueva, Tadas Temcinas, Daniela Gerz, Matthew Henderson, and Ivan Vulic. Efficient intent detection with dual sentence encoders. In Proceedings of the 2nd Workshop on NLP for ComvAl ACI. 2020, mar 2020. URL https: //ap:::://///// http///xxvvvrr///e. orgg http//b//200000 .00033.4488000 04807. Data available at https//githuh..om/Pllaa/l/ LDN/task Baa/aask-speciffe--daaasse
Daniel Cer, Mona Diab, Encko Agiree, lñigo Lopez-Gazzing and Lucia Specia. SemEval-201 7 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Steven Bethard, Marine Carpuat, Marianna Apidianaki, Saif M. Mohammad, Daniel Cer, and David Jurgens (eds.). Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pp. 1-14, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10. 1/11866///1111 URL https: acl ant.oantoooogoooooooooooooo ogy. org/s17-200 2001.
Jianly Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge mnd-embedding: Multi-lingual, multi-fun multiifunccioonaliyy multi-granuulriity text embeddings through self- seff-knowledge distillation, 2023.
Xi Chen, Ali Zeynali, Chico Camargo, Fabian Flöck, Devin Gaffney, Przemyslaw Grabowicz. Scott Hale, David Jurgens, and Mattia Samory. SemEval- 2022 task 8: Multilingual news article similarity. In Guy Emerson, Natalie Schluter, Gabriel Stanovsky, Ritesh Kumar, Alexis Palmer, Nathan Schneider. Siddharth Singh, and Shyam Ratan (eds.). Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval- 2022), pp. 1094-1106, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10. 1/265///1/202222smeealll 1.155. URL https: / / /t///caaaaaa http//ccnnnccoooccccoooooooo.oot hwtptthhhhoooooo.o....eoo org/2022. semeval--eeeeeeeeeee 1 -1. 155.
Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- treitraniig of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04800 2018
Jack FitzGerald, Christopher Hench, Charith Peris, Scout Mackie, Kay Rottmann, Ana Sanchez, Aaron Nash
Liam Urbach, Vishesh Kakarala, Richa Singh, et al. Massive: A Im-example multilingual natural language understanding dataset with S1 typologically-divers languages. arXiv preprint arXiv:2204.0 085882, 2022.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can he accurately pruned in one-shot. In International Conference on Machine Learning, pp. 10323-10337. PMI.R, 2023.
11

---

Published as a conference paper at ICLR 2025
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arX1::2210.77323, 2022.
Tianyu Gao, Xingcheng Yao, and Dangi Chen. Simese: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.08821 2021.
Gregor Geigle, Nils Reimers, Andreas Rücklé, and Iryna Gurevych. Tweac: transtormer with extendable qa
agent classifiers. arXiv preprint axXiv:2104.0708) 2021.
Team Gemini, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- Sann-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint 2311: 1805, 2023.
Stanford NLP Group al. The stanford natural language inference (snli) corpus, 2022.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In International conference on machine learning, pp. 3929-3938. PMLR, 2020.
Edward J Hu, Yclong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09683 09685, 2021.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. arXiv preprint arXiv:21 2021.
Andrew Jacgle, Schastian Borgeaud, Jean-Raptiste Alayrac Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, al. Perceiver io: general architecture for structured inputs & outputs. arXiv preprint urXiv:2107.14995 2021.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, al. Mistral 7b. arXiv preprint 1rXiv:2310006825, 2023.
Yichen Jiang, Shikha Bordia, Zheng Zhong, Charles Dognin, Maneesh Singh, and Mohit Bansal. Hover: A dataset for many-hop fact extraction and claim verification. arXiv preprint arXiv:2011.00 000000800 2020.
Vladimir Karpukhin. Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retricval for open-domain question answering. arXiv preprint arXiv:2004.04900 2020.
Junscong Kim, Scolhwa Lec, Jihoon Kwon, Sangmo Gu, Yejin Kim, Minkyung Cho, Jy yong Sohn, and Chanycol Choi. time-e-dedmmittall Elevating text retrieval with improved gpt data through task specifcc control and quality refinement. linq ai research blog. 2024. URL https: //tt::/gttlllne com/bllgg ling ding-embed--e mirraaa/.........
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein. Illia Polosukhin. Jacob Devlin. Kenton Lee, et al. Natural questions: benchmark for question answering research. Transactions the Association for Computational Linguistics, 7:453 4663-466.11 2019.
Ken Lang. Newsweeder: Learning to filter netnews. In Machine learning proceedings 1995, pp. 331-339. Elsevier, 1995.
Jinhyuk Lee, Zhuyun Dai, Xiaoqi Ren, Blair Chen, Daniel Cer, Jeremy R Cole, Kai Hui, Michael Boratko, Rajvi Kapadia, Wen Ding, al. Gecko: Versatile text embeddings distilled from large language models. arXiv preprint arXiv:2403.20322 2024a.
Sean Lee. Aamir Shakir. Darius Koenig, and Julius Lipp. Open source strikes bread new fluffy embeddings model, 2024b. URL https: / www.//ww...mm mi xedbread. ai /bllg/////////o htt////aaiieeee serii-embed-111 1a0--arree lareevl..
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459 9474, 2020.
Retrieval-augmentee generation for knowledge-
Patrick Lewis. Yuxiang Wu, Linging Liu, Pasquale Minervini, Heinrich Küttler, Aleksandra Piktus, Pontus Stenetorp, and Sebastian Riedel. Paq: 65 million probably-asked questions and what you can do with them. Transactions of the Association for Computational Linguisties, 9:1098-1115, 2021.
12

---

Published as a conference paper at ICLR 2025
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Saško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis. Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan- Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Gochringer, Vietor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 175-184, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. URL htwps: at//aclaaaa www.caaannooooooooooooooooooolol , org / 2021 emnll-m----.m demo 21.
Chaofan Li, MingHao Qin, Shitao Xiao, Jianlyu Chen, Kun Luo, Yingxia Shao, Defu Lian, and Zheng Liu. Making text embedders few-shot learners. arXiv preprint arXi::2409.1577 2024.
Haoran Li. Abhinav Arora, Shuohui Chen, Anchit Gupta, Sonal Gupta, and Yashar Mehdad. MTOP: A com- prehensive multilingual trientrie semantic parsing benchmark. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 2950-2962, Online, April 2021. Association for Computa- tional Linguistics, doi: 12118663/112222.ea eacl-main.251 257.
URL https: http:p::///alaatooooooooooo
Xianming Li and Jing Li. Angle-optimized text embeddings. arXiv preprint https://hugginge
2022,
2023. URL
co/mii/.//... xedbread //wpbaixxiiiie/ www-eeebeeddaee large--a
Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.0322 032881 2023.
Zihan Liu, Wei Ping, Rajarshi Roy, Peng Xu, Mohammad Shoeybi, and Bryan Catanzaro. ChatQA: Surpassing GPT-4 conversational QA and RAG. arXiv preprint arXiv:2401. 10222 2024.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. (42-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http: // http://wwww/// www.ww.wwebboooo org. hwwggggganloogyyggggrrrr P11-1015.
Wei Chen Maggie, Phil Culliton. Tweet sentiment extraction, 2020. URL https::/kkggle..coom
Macedo Maia, Siegfried Handschuh. André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. Www'll 18 open challenge: financial opinion mining and question answering. In Companion proceedings of the the web conference 2018, pp. 1941-1942, 2018
Julian McAuley and Jure Leskovec. Hidden factors and hidden topics: understanding rating dimensions with review text. In Proceedings of the 7th ACM Conference on Recommender Systems, RecSys '13, pp. 165-172, New York, NY. USA, 2013a. Association for Computing Machinery. ISBN 9781450024090. doi: 151144/5777577555777
URL
wtt.:/oooooooooo//oooooooooooee. .rgg /00. 1145/2507155 .5071 63.
Julian McAuley and Jure Leskovec. Hidden factors and hidden topics: understanding rating dimensions with review text. In Proceedings of the 7th ACM conference on Recommender systems, pp. (65-172, 2013b.
Rui Meng. Ye Liu, Shafiq Rayhan Joty, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. Sfr-embeddingg 2: Advanced text embedding with multi-stage stage training, 2024a. URL https: https::///uggnngaaceeee co/
Sa 1 les force/St SFF R- kmbeddi ng- k_k.
Rui Meng, Ye Liu, Shafiq Rayhan Joty, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. Strembedding-mittre enhance text retrieval with transfer learning. Salesforce Al Research Blog, 3, 2024b.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 2013.
MistralAI. Mixtral 8x22b. URL https: / /mi stral. a hww///w////// wttee///mmxttrl--x2222
Gabriel de Souza P Morcira, Radek Osmulski, Mengyao Xu, Ronay Ak, Benedikt Schifferor, and Even Oldridge.
NV-Retriever: Improving text embedding models with effective hard-negative mining. arXiv preprint 1rXiv:2407.15831, 2024.
Niklas Muennighoff, Nouamane Tazi, Loie Magne, and Nils Reimers. MTEB: Massive text embedding benchmark. arXiv preprint 16..6122 2022.
13

---

Published as a conference paper at ICLR 2025
Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. Generative representational instruction tuning. arXiv preprint arXiv:2402. 09906, 2024.
Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, et al. Text and code embeddings by contrastive pre-training. arXiv preprint arXiv:2201. 2022.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. MS MARCO: A
machine reading comprehension datasct. 2016.
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernández Ábrego, Ji Ma. Vincent Zhao, Yi Luan, Keith B Hall, Ming-Wei Chang, al. Large dual encoders generalizable retrievers. arXiv preprint arXiv: 12.07899, 2021.
James Neill, Polina Rozenshtein, Ryuichi Kiryo, Motoko Kubota, and Danushka Bollegala. wish would have loved this one, but i didn't I- multilingual dataset for counterfactual detection in product reviews. arXiv preprint arXiv:2 104. 06893, 2021.
OpenAL. New embedding models and api updates, 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances neural information processing systems, 2022.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lec, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(140):1-61 2020.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Perey Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.0522 2016.
Nils Reimers. Stackexchange (title, body) pairs, 2021a. URL
Nils Reimers. Reddit (title, body) pairs, 2021b. URL https///uggingfaec.c//aatsttttt
centence-transtoonee wttp::/eeeeeeettttttteeeeeeeeeee
Nils Reimers and Tryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv: 10084, 2019.
Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Founda- tions and Trends in Information Retrieval, 3(4):333-389 2009.
Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang. Junlin Wu, and Yi-Shin Chen. CARER: Contextualized affect representations for emotion recognition. In Ellen Riloff, David Chiang. Julia Hockenmaier, and Jun'ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3687-3697, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10. 1..18853v1/D1... 1-01--104, URL https: a sclanthology, www.ooggg....gg.... org/D18 1404.
Weijia Shi, Sewon Min, Michihiro Yasunaga. Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoerr and Wen-tau Yih. Replug: Retrieval-augment black-box language models. arXiv preprint wxXiv:2301.12652 2023.
Haack-Exhange-Commani Stack exchange data dump, 2023.
Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.1 1695, 2023.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. Beir: heterogenous benchmark for zero-shot evaluation information retrieval models. arXiv preprint arXiv:2104 0.066663, 2021.
James Thome,, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. Fever: a large-scale dataset for fact extraction and verification. arXiv preprini arXiv:1803.05355, 2018
George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopoulos. et al. An overview of the bioasq large-scale biomedical semantic indexing and question answering competition. BMC hioinformatics, 16:1-28, 2015.
14

---

Published as a conference paper at ICLR 2025
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Voyage-AI. AL. voyage-large-2-nnstee 2-instrutt: Instruction-tuned and rank 1 mteb, 2024.
Henning Wachsmuth, Shahbaz Syed, and Benno Stein. Retrieval of the best counterargument without prior topic knowledge. In Proceedings the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 251, 2018.
David Wadden, Kyle Lo, Bailey Kuehl, Arman Cohan, Iz Beltagy, Lucy Lu Wang, and Hannaneh Hajishirzi. Scifact-open: Towards open-domain scientific claim verification. arXiv preprint arXiv:2210.1377 2022.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: stickier benchmark Advances neural information processing systems, 32, 2019.
caneral-purpoon language understanding systems.
Boxin Wang. Wei Ping. Lawrence McAfee, Peng Xu, Li. Mohammad Shoeybi, and Bryan Catanzaro. Instructretro: Instruction tuning post retrieval-augmentt pretraining. arXiv preprint arXiv:2310.07713, 2023a.
Liang Wang. Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly- weakly-supervised contrastive pre-training, arXiv preprint arXiv:2212.03533 2022.
Liang Wang. Nan Yang, Xiaolong Huang. Linjun Yang. Rangan Majumder, and Furu Wei. Improving text 2023b.
embeddings with large language models. arXiv preprint
Yuxuan Wang, Daisy Stanton, Yu Zhang, RJ-Skerry Ryan, Eric Battenberg, Joel Shor, Ying Xiao, Ye Jia, Fei Ren, and Rif A Saurous. Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis. In International conference on machine learning, pp. 5180 5189, PMLR, 2018.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester. Nan Du, Andrew M Dai, and Quoe Lc. Finetuned language models are zero-shot learners, arXiv preprint arXiv:210..01662, 2021.
Zhilin Yang. Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christo- pher Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv: 889.09600, 2018.
Xinyu Zhang, Xueguang Ma, Peng Shi, and Jimmy Lin. Mr. tydi: A multi- munti-lingual benchmark for dense retrieval. arXiv preprint arXiv:2108.088: 08787, 2021.
Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Mionns-HHemmeoo Xiaoguang Li. Qun Liu, Mchdi Rezagholizadch, and Jimmy Lin. Miracl: multilingual retrieval dataset covering 18 diverse languages. Transactions of the Association for Computational Linguistics, 11:1114-1131, 2023.
15

---

Published as a conference paper at ICLR 2025
A COMPREHENSIVE STUDY OF MODEL COMPRESSION TECHNIQUES FOR NV-EMBED
Increasing computational and memory demands of LLM-based embedding model present the chal- lenges for the deployment, limiting their scalability and accessibility. In this appendix section, we provide the analysis of post-training model compression techniques (i.e., pruming and quantization) for generalist embedding models. Our analysis demonstrates that these compression methods en- hance the accuracy and robustness of LLM-based embedding models, surpassing the performance of smaller-sized embedding models based on Llama3.2-3B. Qwen2.5-3B and Minitron-4B.
In model compression process, we first perform pruning the NV-Embed-vy -2 model, reducing its size from 8 billion parameters to 3.5 billion (i.e., pruning the main decoder-only blocks and removing the latent attention block). Next, we apply quantization to lower its precision 8-bit weights including integer and floating (F4M3, ESM2) formats. Finally, we perform continual re-training using fine- tuning (PEFT) method known as low-rank adaptation (LoRA) to restore the model's accuracy. For evaluation, we evaluate our model on MTEB benchmark (Muennighoff et al., 2022).
A.1 PRUNING
In order to find better pruning techniques, we apply three methods (magnitude- WANDA(Sun et al., 2023), SparseGPT(Frantar & Alistarh, 2023)) for semi-structured (2:4 and 4:8) and unstructured approaches. Note, unstructured pruning strategy removes the network elements from individual weights, while the structured strategy removes the blocks of nonzero weights in higher granularity ways such as row/columns of weight metrics. sparsity), ensuring that N weights remain non-zero within every group of M weights. For example, 4:8 semi-structured pruning prunes four out of every eight elements in a weight tensor. This semi- structured sparsity reduces the size of the weight matrices and computational cost, while maintaining certain regularity for efficient hardware utilization. The literature presents various criteria for deter- mining which weights prune. The simplest approach is magnitude-basee pruning, which retains weights with higher absolute values and removes the rest. Another approach is WANDA (Sun et al., 2023) which introduces a pruning technique that considers both weights and activations. SparseGPT (Frantar & Alistarh, 2023) identifies the non-critical connections by utilizing the approximate hessian based optimization method.
is the hardware friendly way (N:M
Table 6 summarizes the averaged MTEB scores for different model pruning, respectively. Among these techniques, SparseGPT generally delivers the best results, while magnitude-based and WANDA methods produce comparable performance both during pruning and after retraining as shown in Table Notably, semi-structured (2:4) pruning yields the lowest scores but demonstrates the greatest accuracy recovery following retraining for MTEB benchmarks. Based on these findings, we focus on SparseGPT pruning for subsequent ablation studies.
Table 6: Pruning MTEB benchmark
Semi-strectured
Pruning Criterion
Unstructured
4:8
Magnitude
Pruning
64.62
67.6
69.18
Re-train
69.96
70.46
70.84
Pruning
64.26
67.87
70.19
Wanda
Re-traine
69.74
70.42
70.81
Pruning
68.48
70.11
71.33
SparseGPT
Re-train
70.41
70.9
71.18
A.2 KNOWLEDGE DISTILLATION
In traditional accuracy recovery approaches after model compression, ground truth labels are utilized for continual retraining. To improve this retraining process, we leverage a knowledge distillation loss term, where the uncompressed model serves as the teacher, transfering the knowledge of the more advanced teacher model to smaller and simpler student model. To enable the student model mimic the teacher's 's behavior, we introduce mean-squared error losses for both the output state (S.) and the intermediate states (S) S
16

---

Published as a conference paper at ICLR 2025
For this knowledge distillation process, the the uncompressed embedding model serves the teacher, while the compressed version acts as the student. We remove the latent attention block and compensate the accuracy degradation with knowledge distillation. The knowledge distillation loss is defined as Lid - De (S [ + MSE ,() where Lkd knowledge distillation loss, O is the number of layers, n is layer number, MSE represents the mean- squared function, S, is student state and St is the teacher state. Based on this, the total loss function is sum of contrastive and knowledge distillation loss as: Ltotal Lenntrastive + Lid where a is weight term.
As presented in Table 7, incorporating knowledge distillation ("GT+KD") consistently outperforms using only ground truth labels ("GT"" across different approaches for MTEB benchmarks. Among the methods, 2:4 semi-structured pruning yields the worst results but benefits the most from knowledge distillation, achieving improvements of 0.76 on the MTEB benchmark.
Table 7: Knowledge Distillation MTEB benchmark
semi-structured
Label Types
Unstructured
2:4
4:8
GT
70.41
70.90
71.18
GT+KD
71.17
71.22
71.48
A.3 QUANTIZATION
For weight quantization stage, we adopt GPTQ (Frantar et al., 2022), post-training weight quantiza- tion method that utilizes approximate Hessian information to reduce the precision the weights. To evaluate compressed embedding models, compare them against three smaller LLM-based embedding models-Llama3, 2.BBB Qwen2.5-3B and Minitron-4BB -Bhhhhchh which have varying numbers of weight parameters. Table 8 provides the averaged MTEB scores for compressed models (pruning and quantization), respectively.
A key observation is that compressed models demonstrates superior robustness in low precision settings compared to their smaller counter parts. For example, NV-Embed quantized to INT8 maintains nearly identical MTEB scores (0.0% for 2:4 semi-structured, 0.01 % for 4:8 semi-structured, and 0.01% for unstructured) compared to the performance drops observed in smaller models such as Llama-3B (-0.47%). Qwen-3B (-0.14%), and Minitron-4B (-0.84%). This trend remains consistent across different 8 bit precision cases as well.
Compared to integer format which has uniform numerical distribution, floating point format can also represent the same number of discrete points, covering larger numerical range and non-uniform distributions (high precision for small values and lower precision for large values). There are two primary FP8 format: E4M3 (4-bit exponent, 3-bit mantissa), E5M2 (5-bit exponent, 2-bit mantissa) where bit represents the signed bit. Table shows that 8 bit floating point (E4M3 and E5M2) achieve comparable MTEB scores to the INT8 format.
Table 8: Quantization MTEB benchmark
Precision
16bit
INTS
FP8 (E4M3)
FP8TE5M2
Score
71.17
71.17
70.94
71.14
NV-Embed (2:4)
Diff (%)
0.00%
-0.34%
0.03%
Score
71.22
71.23
71.28
71.48
NV-Embed (4:8)
Diff (%)
0.01%
0.08%
0.37%
Score
71.48
71.49
71.55
71.75
NV-Emhed (Unstr)
Diff (%)
0.01%
0.09%
0.37%
Score
70.31
69.98
70.05
70.06
Llama3.2-3b
Diff (%)
-0.47%
-0.36%
-0.35%
Qwen2.5-3b
Score
69.77
69.70
69.70
69.67
Diff
-0.1%
-0.1%
-0.4%%
Minitron-4b
Score
70.68
70.09
69.97
69.97
Diff (%)
-0.84%
-1.0%
-1.02%
17

---

Published as a conference paper at ICLR 2025
B AIR BENCHMARK
In this appendix section, we present AIR-Bench' (version of 24.04) that is newly released information retrieval benchmark, incorporating the diverse and comprehensive domains such healthcare, law, news, book, arxiv, finance and synthetically generated samples using diverse L.MMs. Importantly, AIR-Bench can help us to understand the generalization capability of the embedding/retrieval model, because the majority of different domain samples do not appear in MTEB benchmarks. Moreover, the AIRBBench is designed as a closed- booeddoookkkoo benchmark whose ground truth is kept confidential. As a result, the benchmark score can be only obtained through the HuggingFace Hub platform.
In AIR-Benchmart 24.04 version, there two tasks: QA and Long- Doc. We evaluations English datasets in QA task and 15 English datasets on the Long-Doc task. As shown in Table 9, EV-2mbed-v2 achieves the second highest scores in QA section. As described in Table 10, our NV-Embed-v2 attained the highest scores of 74.78 on the Long-Doc section, surpassing the Bge-en-icl model that requires overheads adding in-context examples to query during training. is important to highlight that the NV Embed-v2 model, which achieved higher MTEB accuracy scores, also demonstrates improved accuracy on both QA and Long-Doc tasks in the AIR-Bench compared to NV-Embed-vl. Interestingly, this is not always observed in the literature, where model performing better on MTEB does not necessarily outperform on the AlR-Bench. For example, while FFR-Embeedding28 substantially outperforms WERR-Embedding-Mo -istral in MTEB scores (SFR- Embeddiing2RR 70.31, SFR- Embedding- Mistral: 67 67.56), falls short in AlR-Bencis performance both in QA (SFR-Ea) SFR--mmedddnng- 2R: 49.4 SFR Embedding- Mistral:: 51.58) and Long-doc FERR-Embedding-22 2R: 67.45, SERR--meedding-Miii Mistrall: 69.0).
Table QA (nDCG scores) on AIR benchmark 24.04
Domain
Wiki
Web
News
Healthcare
Finance
Arxiv
Msmarco
Avg
Bge-en-icl (zero-shot)
64.61
54.40
55.11
57.25
25.10
54.81
48.46
63.71
32.93
WW-Embedee
65.19
$2.58
53.13
59.56
25.00
53.04
48.94
60.8
$2.28
SFR- Re----mbeddingBee Mistral
63.46
51.27
52.21
58.76
23.27
56.94
47.75
58.99
51.58
Stella-I (.5iaal1.BBBo 1.5B-v5
61.99
$0.88
53.87
58.81
23.22
57.26
44.81
61.38
51.53
Ste-Qwen2--7Biintteee
63.46
51.20
54.07
54.20
22.31
58.20
40.27
58.39
50.26
NV ----ed-ld
62.84
50.42
51.46
58.53
20.65
49.89
46.10
60.27
50.02
Ling--mbed-Mistte
61.04
48.41
49.44
60.18
20.34
50.04
47.56
49.69
SFR E-beddingg----
63.72
48.77
51.14
55.86
20.98
54.78
42.84
57.66
49.47
15-mmsstral-7-iinstue
61.67
44.41
48.18
56.32
19.32
54.79
44.78
59.03
48.56
Table 10: Long-document (Recall @ 10 scores) on AIR benchmark 24.04
Domain
Arxiv (4)
Book (2)
Healthcare (5)
Law (4)
Avg. (15)
NV- Embed-v2
79.27
77.46
73.01
71.18
74.78
Bge-en-iel (zero-shot)
78.30
78.21
73.65
67.09
73.75
IV-Rmbed-vl
77.65
75.49
72.38
69.55
73.45
71.77
76.46
73.96
70.86
72.88
Ling umhed-Mistre
75.46
73.81
71.58
68.58
72.11
Stella- 1.5B-v5
73.17
70.02
69.32
71.25
UTR-Embedding Mistrall
72.79
72.41
67.94
64.83
69.0
Text-embed-3- (OpenAl)
74.53
73.16
65.83
64.47
68.77
E5 E5-mistral-70h -D-insrutt
72.14
72,44
68.44
62.92
68.49
SFR- Embedding-22 2R
70.51
70.22
67.60
62.82
67.45
hhtpp///gituu..comAARRRRRR Aaaa
18

---

Published as a conference paper at ICLR 2025
C EXPERIMENTAL DETAILS AND INSTRUCTION TEMPLATES FOR TRAINING AND EVALUATION
In this section, we describe our detailed experimental setups. We use parameter-effiient finetun- ing (PEFT) method denoted as low-rank adaptation (LoRA) (Hu et al., 2021) to efficiently finetune our proposed NV-Embed model. We chose Mistral 7B (Jiang et al., 2023) as the base decoder-only LLM. We replace the attention mask from causal to bidirectional, and integrate the latent attention layer with 512 latents, 4096 hidden dimension size, and 8 multi-head attentions.
We train Mistral 7B LLM model end-to-end with a contrastive loss using LoRA with rank 16, alpha 32 and dropout rate of 0.1. We use Adam optimizer with 50 warm-up steps and learning rate 2e-5 for first stage and 1.5e-5 for second stage with linear decay. The optimizer hyperparameters are included in Table 11. We restart the optimizer with the same 50 warm-up steps and lower learning rate for the second stage. The model is finetuned with 128 batch size, where each batch is composed of query paired with 1 positive and 7 hard negative documents. Training samples from different datasets in Table 12 are uniformly sampled. We train using Bfloat16, and set the maximum sequence length as 512 tokens. The special <BOS> and <EOS> tokens are appended at the start and end of given query and documents. The whole training is conducted in two stages where the model is initially trained on retrieval datasets utilizing in-batch negative technique. Subsequently, the model is trained with blended datasets with both retrieval and non-retrieval embedding tasks.
For evaluation, we assess our model using a maximum length of 512 tokens to ensure fair comparisons with prior work (Wang et al., 2023b), which also provides evaluation results based on $12 token limits. Evaluation instructions templates are available Table 13.
Table 11: Parameters used in the experiments
Parameter
Value
Batchsize
128
Number of Hardnegatives
7
Warm-up Steps
50
First stage 20k
Training Steps
Second stage 18k
First stage 2e-5
Learning Rate
Second stage 1.5e-5
Rank 16
LoRA Params
Alpha 32
Dropout - 0.1
Weight Decay
0.03
Optimizer
Adam
Padding Side
right
Number Latents (r)
512
Latent Width (d)
4096
Multi- Attention Heads
8
19

---

Published as a conference paper at ICLR 2025
Table 12: Instructions and number of samples used for each training dataset.
Task Name
Instructoon Jemplase
Number of Samples
ArguAma
Given a clam, retreve docummnts that support refote the claim
16k.
Retneve semantically similar text
Natural Language Inference
Given premise, retrieve a hypothosis that is entailed by the premise
270k
question retrieve paesapes that passages question
ueh
query,
quary
PAQ. MSMARCO
500k,
question, retrieve documents can help answer question Gaven question, retrieve passages that answer the question
SQUAD
873.
ScackExchange
Given web search quary, cotieve relevant passages that answer the query
8tk
Natural Question
Given quesiion retrieve presages that answer the question
100L
HorporQA
Given multi hop question, retreev documens that can help answer the question Given claim, retrieve documenss support refute the claim
170k
FEVER
140K
FIQA2018
passages that help answer question claim, retriese dacuments support refute daam Given question, retrieve relevam documents that answer the question Given guestion, retrieve passages that answer the question
finsneial question,
the query
5k
BioASQ
quary, retrieve ducuments
2/4L
HoVer
Nicurpus
3.6L
MIRACL
2K
Mr TyDi
Given question retrieve presages that answer the question
2X
SeiFact
Given scienifiic claim, rerreve documents that support or refute the claim Retrive semuntically similar test. Classify givea
0.9K
STS12, STS22, STS STSBenchmark
1.5k, 0.3K, 2.7k
Musteooouttrratuu Classifccctii
either counterfactual not-coumerfactual
6k
AmszonPolarity Classi
Classlfy
Classify the given Amazon review negative sentiment
20L
AmeoeRevieessCCue Clssffitiooo
appropriatc rating cstegory
40k
Bonking7--Chssliiin
Given unline banking query. find the correpunding insenis
1OL.
Emotion Emotion-Clasifican Classifiatiion
Classify the emotion expressed the gives Twitter message into one of the six emotionsanger,
16k
fear, joy, love, sadness, and suppris
Inddh Classificttion
Classify the semtiment expressed the piven movic review text from the IMDB datsset
24k
MTOPlntem: Clasification
Classify the inient the ullerance unienteeeeed comversation
15L
MI OPDDmmaan Classification
Classify the inteut domain
given
task oricnted
uset usterance query.
user
Miseive Classificationn
niet utterance quay.
uset secnacios
Tosie Cmni@CamerssiommCCCCCC Chnssiiiioo
Clussify the given comments is either toxic or loxic
50L.
Classify the sentmeet of given tweet as oither posilive, negative, or nentral Identify the main and secondary category of Artz papers based on the titles and abstracts
27L
Anty Clustering F2P
50K
Antiv
Clusteiing-SS
Identit the main and secondary eategory Arti papers based on the titles
50k
Bicesiv Clestrrigg P2P
Iomiify the main calegory of Biorsis Identify the categrey Binesix popers based
based the titles and sistmetts
1SL
Ricrhy- Clasering-525
tirles
Chasering
Identify
calegory
papers based
Medixi ChasterrssinnnSS.
Identity
category Medexin popers based
2.3k
Redkin-Clusatte Claszring
Identify the main caeegery of Medrxiv pupers based on the titles abstracts Idemiif the main categrry of Madrsiv papers hascd on the tifles and absumnnts Identify the main catlgorr of Medraiv papers based un the titles and abstracts Identity the main category of Medravv papers based the titles and abstracts Identify the
50k
Doddi-Chheering-S5.
40K
Stackeschange Clustering
50K
Stackeschange- Cla.tee-igiitggggggeo $28
40k
New.rruu Classering
articles
1.7k
D LATENT-ATTENTON VISUALIZATION
Latent attention AmazomReieeeCllsssffcttooo reviews
Classify the given Amazon review into its appropriate rating category
POS
NEG
Head 0
Head
Head
Head 3
Head
Head
Head 6
Head
Figure 2: Attention over 4096 latents across 8 heads (columns) are visualized for 10 positive and 10 negative reviews (rows) from the AmazoonevieesCCasiffcccii are mean pooled across tokens. The attention weights reveal that the latents specialize in learning features queries. The latent indicated by the arrows specialized in learning the positivity of reviews. It has high attention across the positive reviews and low attention across the negative reviews.
dataset. The attention weights
20

---

Published as a conference paper at ICLR 2025
Table 13: Instructions used for evaluation on the MTEB benchmark. "STS*" indicates we use the same instructions for all the STS tasks.
Task Name
Instruction Template
ArguAna
Given a claim, reirieve documenis that suppon refute the claim
ClimaleFEVER
Given claim about climate change, retrieve documents that support or refute the claim
DBPodia
Given query. retrieve relevant entity descriptions from DBPedia
FEVER
Given claim, retrieve documents that support refute claim
FiQA2018
Given financial question, retrieve user replics that best answer the question
HotporQA
Given multi-hop question, retrieve documents that can help answer the question Given web search query, retrieve relevant passages that answer the query Given question, retrieve relevant documents
MSMARCO
NFCCrpus
the question
Natural Question
Given question, rerrieve passages that answer the question Given question, retrieve questions
QuoraRetrieval
are semunically equivalent
given question
SCIDOC'S
Given scieutific paper üitl, retrieve paper abstracts that are cited by the given paper Given scicutific claim, retrieve documents that support refute the Given question, retrieve passages that answer the question Given query COVID-19 Retrieve semantically similar text.
SciFact
Touche2020
TREC-COVID
documents answer the query
STS
SummnEyal
Given summary, retrieve other semantically similar summaries
CmazonCounterfeetti FanettaClnnssiiaaiiiin
Classify given Amazon customer review text as either counterfactual or not-counterfactual Classify Amazon reviews positive negative sentiment
AmlzooPolarityy Clussification
Classily the given Amazon review into its appropriate rating category
Banking77C CCClsssiaatoiooooottioon
Given online banking query, find the conesponding intents
AmntionClassificaten
Classify emotion expressed
given Twitter message into
six emotions:anger,
fear. joy, love, sadness, and surprisc
ImdhClassification
Classify sentiment expressed in the given movie review from the IMDB
MasiieIntentCaasiiiie Inssiiicaiin
Given user utterance query, find the user intents
MassiveScenrriCCI Clavsiiictiii
Given utterance query. lind
scenarios
CoOPoommisCaasiiii fication
Classify the intent domain of the given utterance in task-oriented conversation
MTOPIntentC lassificiiinii
Classify
given utlerance
conversation
Toxic onw.erstioosslaasstaaaiiii
Classify the given comments as either toxic or not toxic
Classify sentiment given
positive, negative, neutral
Azxis ClusteringPPP
Identify the main and secondary category of Arxiv papers based on the tites and abstracts Identify the
ArxisClustering ClusteringS2S
and secondary category of Arxiv papers based the lidntify the main category Biarxiv papers hased on the ritles and abstracts
Bionxiv ChnnstChuse
Bionxi Bionav ClusteringSS2
Identify the main category Biarsiv pupers based the tiles
Medrii MeirassChsteringP22
Identify the main category Medrsiv papers hased on the titles and abstracts Identify the category Medriiv pupers bused the titles
Medrxis ChasseiCusss
ReddiCClusterian
Identify the topic or theme of Reddit posts based on the titles Identify the topic theme of Reddit posts based on the titles and posts
CldieclusteringPPP
Stack Elehnngeeeelluttteenn
Identify the topic or theme of StuckExchange posts based Identify the topic theme of StackExchange posts based the given paragraphs Identify topic
tites
BacckExehnggCC ExchangoChsteringePP
Twenty NewagroupcChustrrin
given news articles
Ask JuUUuuuuuuuuustiinn
Retrieve duplicate questions from AskUbuntu forum Retrieve zelevant news
Minn&SmellReranninn
on browsing history
SciDocsRR
Given title of scientific paper, retrieve the titles of other relevant papers Retrieve duplicate questions
Sverlow DupQuestions
StuckOverflow Retrieve duplicate questions from Sprint forum Retrieve tweets that semantically similar the tweet
SprintDoplicattuussiee Questioon
SwiterSemEFal2211
CRrCoups
Retrieve tweets that are semantically similar to the given twect
21

---

Published as a conference paper at ICLR 2025
Table 14: Full BEIR and MTEB benchmark
Bge-multiln gual-gomma2
Gre-Qwen2- TB-instruct
SFR-Embe dding-2R
bge-en-ici
NV-Embed--v
NV-Embed-v2
1.5B-v5
(2croshot)
ArguAna
ClimatFFEVER
39.37
45.88
34.43
46.11
45.35
34.72
45.39
CQADupStack
50.24
DEPEDIA
51.37
52.42
51.21
52.28
50.42
48.29
$3,50
FEVER
95.11
FIQA2018
60.04
62.03
61.77
60.48
$8.77
63.1
63.73
HotporQQA
MSMARCO
45.71
45.98
42.18
45.22
46.72
46.49
43.63
NFCorpus
Natural
71.45
67
73,96
71.8
73.85
71.22
73.57
QuoraRetrieval
90.04
90.09
89,58
90.03
91.02
89.21
89.04
SCIDOCS
26.93
28.
24.87
26.64
25.25
20.19
21.90
SciFact
72.05
79.06
85,91
80.09
78.33
78.43
80.13
To2che2020
30.25
30.57
28.
29.94
29.67
28.38
31.78
TREC-COVID
64.27
82.26
87,28
85.98
78.
85.88
88.44
BIOSSES
85.74
81.37
87.6
83.11
86.35
85.59
87.42
SICK-R
66
79.28
77.01
82.89
83.87
82.8
82.15
STS12
77.71
79.55
75.67
80.09
77.73
76.22
77.89
STS13
87.45
88.83
82.4
89.68
85,98
86.3
88.30
STS14
83.48
83.87
79,93
85.07
82.34
82.09
84,30
SISI5
87.63
88.54
85.82
89.39
87.35
87.24
89.04
STS16
86.49
87.15
84.77
86.77
STS17
91.18
88.73
88.93
91.35
91.25
87.42
90.67
STS22
69.02
66.88
67.1
68.1
68.08
69.85
68.
STSBenchmark
87.25
86.85
83.6
88.23
87.92
86.14
88.41
SummEval
31.2
31.35
30.71
30.75
31.2
30.70
SprintDuplientenussiinn
90.94
92.82
97.62
96.04
95.06
95.94
97.02
Twitter SemEaa22011
79.64
77.96
78.57
80.58
78.54
78.73
81.11
TwitterURICooppe
86.95
86.59
88.03
87 58
87.19
86.05
87.87
MazzonCounteraatuull
AmazonPolarity
96.9
97.5
97.31
97.10 16
96.86
97.14
97, 74
Revien
Banking?7
92.53
87.57
90.02
89.79
91.42
90.34
92.42
Imib
96.66
96.75
96.8
96.66
96.91
97.06
97.
Mussivelntent
MusiveSSemario
84.4
89.77
90.61
90.2
83.92
81.74
97.17
99.25
MTOPIntent
95.51
91.88
91.3
92.78
93.56
89.77
94.37
ToxicConversations
78.85
72.58
79.7
74,84
79.9
80.6
80.87
Arxi--22
50.28
51.74
48.82
50.66
49.17
49,59
$1.26
Biorxiv-P2P
$4.09
Biorxiv-S2S
49.2
46.65
46.57
46.87
48.38
44,74
49.60
46.23
Medrxiv-S25
44.11
44.13
44.18
44.65
44.2
36.98
44,86
Reddit
56.03
73.55
72.86
71.2
63.2
71.10
Roddit-P2P
65.83
74.13
72.74
75.27
72.17
68.01
74,94
StackFxchange
66.21
79.86
76.48
80.29
81.29
74.99
82.10
StackExxhanne P2P
45.74
49.41
48.29
49.57
45.53
42.04
48.36
TwentyNewssroups
70.44
53.91
66.42
61.43
68.51
60.13
64,82
Ask A.kUhamuuugQuussionn
64.59
67.58
66.71
67.33
64.8
67.5
67.46
MindSmallRerank
31.79
33.36
31.26
33.05
30.6
30.82
31,76
SciDocsRR
87.6
89.09
87.29
89.2
$6.9
87.26
87.59
StackOverlow
54.9
55.66
55.32
55.25
$6.32
56.58
55.79
MTEB Average (56)
69.88
70.24
70.31
71.19
71.24
69.32
72.31
22

---

Published as a conference paper at ICLR 2025
Table 15: Prompt template for short-long matching subgroup.
Brainstorm list of potentially useful text retrieval tasks.
Here are few examples for your reference:
Given web search query, retrieve relevant passages that answer the query Given claim about climate change, retrieve documents support refute the claim
Given job title, search for job descriptions that provide information about the role
Please adhere to the following guidelines:
Specify the type query and the type desired texts.
Fach retrieval task should
wide range of queries, and should
specific.
Cover wide range of query types desired text types.
Your output must always he a JSON list of strings only, with about 40 elements, and each element corresponds to a distinct retrieval task in one sentence, Do not explain yourself or outpul anything else, Be creative!
You have been assigned a retricval task: [task]
retrieval example for this task in JSON format. The JSON object must contain the following keys:
Your mission write
"user_query" quorquurrr : string, random example of what is provided as specified by the task description.
"positive_ Dositive_docuneee document": string, relevant document for the user query.
"hurd_ "hurd_negaun negative
negatiive_docummell documenil"; : a string, a negative document that is imeeevant appears relevant the query.
"hard_ cerd__ngativ__ednne document2": string, another hard negative document that irrelvvant but appears relevant to the query.
Please udhere to the following guidelines:
The "user_ aiser_qerry" should be (query_lyper type], length), (clarity ). and diverse in topic. The "user_query" should not restate task and just contain what the task description says provided.
All documents must created independent
query. Avoid copying query verbatim, acceptable not lopically related the query.
parts the "positive_ document"
All documents should be at least [num_words] words long.
The "hard negative document!" may contain little useful information, but it should be less usefuul or comprehensive compared to "positive document".
The "hard negative document2" may should be about related but different topic.
Do provide any explanation in any document why relevant relevant the query.
Both the query and documents require (difficulty) level education to understand.
Your output must always he a JSON object only, do not explain yourself or output anything else, Be creative!"""
Placcholders:
"[query_type Jype|" [extremely long-tail, long-tail, common)
jquery_Jength]] [less than 5 words, to words, least 10 words] "[ "(difficulty)" E (high school, college. PhD)
"(clarity)" (clear, understandable with some effort, ambiguous)
"[num_words]" E (50, 100, 200, 300, 400, 500)
23

---

Published as a conference paper at ICLR 2025
Table 16: Prompt template for long-short matching subgroup.
Braintorr list of potentially useful text classification tasks.
Please adhere to the following guidelines: Tasks should cover diverse range of domains and task types.
Your output always JSON list of strings only, with ahout elements, each element corresponds distinct classification task one sentence. Do explain yourself output anything else. Be creative! You have been ussigned text classification task: [lask]
Your mission is to write one text classification example for this task in JSON format. The JSON object must contain the following keys:
"imput_tews text": a string, the imput text specified by the classification task.
"label": string, the correct
label of the input text.
laallladdng_abeelll a string, an incorrect label that is related to the task.
Please adhere to the following guidelines:
The "input_tess" text" should he mum_wordss words and diverse in expression.
The Imislcading_label" must be valid label for the given task, but not appropriate the "label" for the "input_text"
Avoid including the values of the "label" and "misleading label" fields in the "input_text", that would make the task easy.
TTe "input_text" text" [clarity) ) and requires (difficulty) level education comprehend.
Your output must always JSON object only, not explain yourself or output anything else. creative! Placeholders:
(num_ _oodss E ("less than 10", "at least 10", "at least 50", "at least 100", "at least 200")
(difficultyy ) E (high school, college, PhD)
{clarity) ) E (clear, understandahle with some effort, ambiguous
24


Reviews:
[{"review_id": "THdbu5zx1W", "reviewer": "Reviewer_MUEH", "summary": "This paper gives a summary of the NV-Embed model that achieved the top performance in the MTEB benchmark. \nThe techniques used are\n1. latent attention layer that achieves better pooling/combination of the last layer embeddings. Causal attention mask is removed during contrastive learning. \n2. a two-stage contrastive instruction tuning method. First step tuning with in-batch negative and hard negative on retrieval datasets, and the second step tuning on non-retrieval datasets.\n3. large amount of efforts on training data curation.", "questions": "The v2 model largely outperformed v1: \n\"We then further improve the model through the curation\nof training dataset, including adding more retrieval datasets, applying positive-aware hard-negative\nmining technique, using synthetic data generation process and constructing example-based multi-class\nlabels.\" \nIt would be nice to have more discussions and ablations specifically regarding the v2 vs. v1.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work gives a good summary of the STOA NV-Embed model that leads the MTEB benchmark. I think the community will very much appreciate this paper.\n2. Impressive experimental results with good ablations to justify the design choices.\n3. Clear presentation.", "weaknesses": "The NV-Embed model is a result of a combination of methods/tricks/datasets etc. \nThere does not seem to be single innovative algorithm piece. This by itself is not a weakness of the work. However, as a result, the technical depth of the work is limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PlqwpPld5J", "reviewer": "Reviewer_rURm", "summary": "This paper presents techniques for leveraging pretrained decoder-only large transformers in retrieval tasks, achieving state-of-the-art results on the standard retrieval benchmark (MTEB). The core methods include:\n* A latent attention layer for creating pooled embeddings, which surpasses traditional mean pooling and last-token embedding approaches.\n* A two-stage training process: the first stage focuses on retrieval datasets, while the second integrates non-retrieval tasks for broader versatility.\n* The use of curated datasets (e.g. hard-negative mining) to further refine embedding quality.", "questions": "* What would happen if the order of the two-stage training were reversed?\n* Could you expand on the points raised in the weaknesses section, offering more discussion on these limitations?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* Provides very strong empirical results. This work achieves state-of-the-art results, securing the #1 position on the MTEB benchmark. This demonstrates the effectiveness of the proposed approach.\n* Provides nice ablation studies (e.g., Table 4) to analyze the effects of various design choices, including single-stage vs. two-stage, hard negative mining (HN), public retrieval set (AD), and synthetic data generation (SD).\n* Provides detailed descriptions of the data curation process, including specific techniques like hard negative mining and synthetic data generation. This transparency allows for better understanding and potential reproducibility of the work.", "weaknesses": "* Using decoder-only pretrained models for retrieval is not entirely novel (e.g., as seen in GritLM [1] and discussed in Section 2.2). While the authors note differences in their specific training approach, this limits the novelty of this aspect.\n* Given this, the main contributions center on the latent attention layer and two-stage instruction tuning. However, the performance improvement from latent attention over mean pooling appears modest (see Table 2, bidirectional columns), raising questions about the added complexity for minimal gains. \n* Although the authors provide empirical support for the benefits of two-stage training, there is limited explanation or intuition behind why this approach works effectively.\n\n[1] Muennighoff, Niklas, et al. \"Generative representational instruction tuning.\" arXiv preprint arXiv:2402.09906 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yjEMJVU9Ck", "reviewer": "Reviewer_9J2v", "summary": "This paper proposes a new embedding model: NV-embedder. The model uses a latent attention layer to obtain pooled embeddings instead of the common EOS token. The authors also propose a two-stage training method and employ hard example mining and data synthesis.", "questions": "no", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The authors tested the model on a leaderboard outside of MTEB. Since the training data is related to MTEB, out-of-domain testing is necessary. \n- The authors conducted extensive ablation studies to verify the effectiveness of each module.", "weaknesses": "The biggest drawback is the complexity of the entire training process, which includes multi-stage training, hard example mining, and data synthesis. This involves too many operations, making it difficult to reproduce.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s52MsSZtMA", "reviewer": "Reviewer_wm7z", "summary": "This paper presents the NV-Embed-v1 and v2 systems for general purpose text embeddings. The paper presents the learning algorithm along with architecture variations (latent attention), ywhich describes how the system achieves #1 on the MTEB leaderboard at the time of submission. This paper documents the high performing system, which is likely to have gathered much attention given its position in the leaderboard.", "questions": "For each stage of training, do you use the same optimizer? Restarting it for each stage?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper severs a very clear and important purpose -- documenting the current top performing system on the extremely competitive text embedding benchmark, MTEB. The paper presents an approach, which like other approaches on the MTEB leaderboard, takes a pretrained transformer model and trains using a mixture of datasets/tasks, including a hard negative mining pipeline. The authors also introduce a simple architectural change, latent attention layer.\n\nThe key strength of the paper, is its empirical gains and empirical analysis. The authors provide benchmark performance on a wide range of settings with highly performing models. This shows the capabilities of models on a wide variety of tasks, including important ablations regarding attention type, stages of training, etc.", "weaknesses": "As this paper is primarily the documentation of a very highly performing empirical system, main weakness I would point out is about innovation. On one level the paper is ground breaking because of its empirical gains, on another, the core methodological techniques are well known, only that they are more effectively performed and analyzed here.\n\nThe latent attention mechanism, while a nice architectural change, only has a small effect on the average performance (Table 2). The hard negative mining pipeline, a more established technique, on the other hand makes much more of a difference (Table 4).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)