Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrieval (2024)

Abstract: Dense retrieval calls for discriminative embeddings to represent the semantic relationship between query and document. It may benefit from the using of large language models (LLMs), given LLMs’ strong capability on semantic understanding. However, the LLMs are learned by auto-regression, whose working mechanism is completely different from representing whole text as one discriminative embedding. Thus, it is imperative to study how to adapt LLMs properly so that they can be effectively initialized as the backbone encoder for dense retrieval. In this paper, we propose a novel approach, called Llama2Vec, which performs unsupervised adaptation of LLM for its dense retrieval application. Llama2Vec consists of two pretext tasks: EBAE (Embedding-Based Auto-Encoding) and EBAR (Embedding-Based Auto-Regression), where the LLM is prompted to reconstruct the input sentence and predict the next sentence based on its text embeddings. Llama2Vec is simple, lightweight, but highly effective. It is used to adapt LLaMA-2-7B on the Wikipedia corpus. With a moderate steps of adaptation, it substantially improves the model’s fine-tuned performances on a variety of dense retrieval benchmarks. Notably, it results in the new state-of-the-art performances on popular benchmarks, such as passage and document retrieval on MSMARCO, and zero-shot retrieval on BEIR. The model and source code will be made publicly available to facilitate the future research. Our model is available at https://github.com/FlagOpen/FlagEmbedding.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3490–3500 August 11-16, 2024 ©2024 Association for Computational Linguistics
In this paper, we propose a novel approach, called Llama2Vec, which performs unsupervised adaptation of LLM for its dense retrieval application. Llama2Vec consists of two pretext tasks: EBAE (Embedding-Based Auto-Encoding) and EBAR (Embedding-Based Auto-Regression), where the LLM is prompted to reconstruct the input sentence and predict the next sentence based on its text embeddings. Llama2Vec is simple, lightweight, but highly effective. It is used to adapt LLaMA-2-7B on the Wikipedia corpus. With a moderate steps of adaptation, it substantially improves the model’s fine-tuned performances on a variety of dense retrieval benchmarks. Notably, it results in the new state-of-the-art performances on popular benchmarks, such as passage and document retrieval on MSMARCO, and zero-shot retrieval on BEIR. The model and source code will be made publicly available to facilitate the future research. Our model is available at https://github.com/FlagOpen/FlagEmbedding.

heading: 1 Introduction
text: Dense retrieval is a new paradigm of IR empowered by deep neural networks. It represents query and document as embeddings within the same latent space, where the semantic relationship between
†. Co-first authors and major contributors of techniques ∗. Major contributor of engineering ‡. Corresponding author.
query and document can be reflected by their embedding similarity. Nowadays, dense retrieval has been a critical component in many real-world scenarios, like open-domain QA, fact verification, and retrieval-augmented generation (Karpukhin et al., 2020; Thorne et al., 2018; Lewis et al., 2020; Wang et al., 2022a; Dolci et al., 2023).
The capacity of text encoder is a critical factor of dense retrieval. In the past few years, the pretrained language models, e.g., BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), T5 (Raffel et al., 2020), were widely applied to generate high-quality representations for query and document, which substantially contributed to the accuracy and generalizability of dense retrieval. Besides, it was also found that dense retrieval’s performance can further benefit from the continual growth of model size and training scale (Ni et al., 2021; Izacard et al., 2021; Wang et al., 2022c; Xiao et al., 2023). Recently, large language models (LLMs) have emerged as a unified foundation for general NLP tasks (Brown et al., 2020; Wei et al., 2021; Chowdhery et al., 2023). Given the LLMs’ superior capability on semantic understanding, it will be promising to take advantage of such powerful models as new backbones for dense retrieval. With this consideration, there have been pioneering efforts where LLMs are trained to generate discriminative text embeddings to facilitate the retrieval tasks in many different scenarios (Muennighoff, 2022; Neelakantan et al., 2022; Ma et al., 2023; Zhang et al., 2023).
Despite the preliminary progress, it remains an open problem to fully unleash the LLM’s underlying potential for dense retrieval. Particularly, the typical LLMs are pre-trained by text generation tasks, especially auto-regression, where the text embeddings are learned to predict the next tokens. Consequently, the LLMs’ text embeddings will focus on representing the local (i.e. the next step) semantic of the context. However, dense retrieval calls for text embeddings to represent the global
3490
semantic about the query and document. Such a big discrepancy will severely restrict the direct application of LLMs for dense retrieval.
To address the above problem, we propose a novel approach Llama2Vec (Figure 1), which performs unsupervised adaptation of LLMs to facilitate their usage in dense retrieval. Llama2Vec works as a continuation of pre-training. On top of the tailored unsupervised learning tasks, it transforms the LLM’s text embeddings to represent the global semantic about the input text, which makes the LLM a better initialized encoder for dense retrieval. Llama2Vec is made up of two pretext training tasks: EBAE (embedding-based auto-encoding) and EBAR (embedding-based autoregression). On top of EBAE, the LLM is prompted to generate the inductive text embedding, which can be used to predict the vocabularies of the input sentence itself. With EBAR, the LLM is prompted to generate the deductive text embedding, which can be used to predict the vocabularies for the next sentence.
The joint conduct of the above pretext tasks bring forth two benefits. Firstly, the text embeddings from LLM can be adapted from local semantic representation (predicting the next tokens) to global semantic representation (predicting the sentencelevel features), which aligns with the expected property of dense retrieval. Secondly, by learning to generate inductive and deductive text embeddings with different prompts, the LLM-based retriever can flexibly handle diversified semantic relationships about correlation (e.g., QA) and paraphrasing
(e.g., NLI), which presents a strong foundation to develop versatile retrieval models. It’s worth noting that the prediction is realized in the form of multi-classification, where the LLM’s text embedding is the input and the vocabularies within the target sentence are employed as the labels. Therefore, Llama2Vec is extremely lightweight and simple to realize based on the existing auto-regression pipeline.
We apply Llama2Vec for LLaMA-2-7B (base) (Touvron et al., 2023) over the Wikipedia corpus, where it substantially improves the LLM’s downstream retrieval performance. With standard finetuning, the well-adapted model is able to notably outperform all existing dense retrieval methods, where it establishes new state-of-the-art performances on a variety of popular benchmarks, including the supervised tasks like passage and document retrieval of MSMARCO (Nguyen et al., 2016), and the zero-shot retrieval of BEIR (Thakur et al., 2021).
To summarize, our work is highlighted by the following technical contributions. 1) We propose a new unsupervised learning method Llama2Vec. To the best of our knowledge, this is the first research work which explores the adaptation of LLMs for dense retrieval. 2) Llama2Vec is designed with simple but effective pretext tasks, which substantially improves the quality of LLM-based dense retriever in a cost-effective way. 3) The empirical studies verify the effectiveness of Llama2Vec, where substantial improvements can be achieved for both supervised and zero-shot retrieval tasks.

heading: 2 Related Works
text: In this section, the related works are discussed from two perspectives: the background of dense retrieval, and the previous efforts on leveraging the LLMs for dense retrieval applications. • Dense retrieval. Dense retrieval is to represent query and document as embeddings within the same latent space, where relevant documents can be retrieved by embedding similarity. Nowadays, it is widely utilized in many important applications, such as open-domain QA and retrieval-augmented generation (Karpukhin et al., 2020; Lewis et al., 2020). The performance of dense retrieval is influenced by many factors. For example, dense retrieval models are learned by contrastive learning, where the discriminativeness of text embeddings are largely influenced by the scale and hardness of negative sample (Qu et al., 2020; Izacard et al., 2021; Xiong et al., 2020). Besides, the learning of dense retrieval models can benefit from knowledge distillation, where fine-grained teacher labels are derived from the ranking models (Hofstätter et al., 2021; Ren et al., 2021). Apart from the above training algorithms, the backbone architecture is one decisive factor for dense retrieval. In the past few years, pre-trained language models (PLMs), like BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), T5 (Raffel et al., 2020), have been widely adopted for the encoding of query and documents. Thanks to the large-scale model architecture and pre-training, PLMs were able to produce fine-grained semantic representation of input data, which substantially benefit the quality of dense retrieval. Besides, it was found that with the expansion of model and training scale, and the optimization of pre-training algorithm, the accuracy and generality of the PLM-based dense retrieval can be further improved. (Ni et al., 2021; Izacard et al., 2021; Wang et al., 2022c; Xiao et al., 2023; Gao and Callan, 2021; Liu and Shao, 2022; Liu et al., 2023; Wang et al., 2022b). • Dense retrieval with LLM. The LLMs have been a unified foundation for many NLP tasks because of its superior capabilities. As a result, it is instinctive to leverage such powerful models to facilitate dense retrieval. LLMs can substantially contribute to many critical aspects of dense retrieval. For example, it can help to model the complex relationship between query and document considering LLMs’ strong semantic understanding capability (Brown et al., 2020; Chowdhery et al., 2023; Touvron et al., 2023). Besides, it will benefit the learning of multi-task retrievers because of LLMs’ versatility and instruction following capability (Wei et al., 2021; Chung et al., 2022). It also presents a powerful foundation to develop long-document retrievers, given its dramatically extended context lengths. Recently, there have been several preliminary works which made important progresses on LLM-based dense retrieval (Muennighoff, 2022; Neelakantan et al., 2022; Ma et al., 2023; Zhang et al., 2023). However, the existing methods simply made direct use of LLMs. Because of the discrepancy between language modeling and text embedding, much of the LLMs’ underlying potential is unexploited. In fact, it is still an open problem to study the proper adaptation of LLMs so that they can better contribute to dense retrieval.

heading: 3 Llama2Vec

heading: 3.1 Preliminary
text: Dense retrieval utilizes a text embedding model to produce the query and document’s embedding: eq and ed. The relevance of query and document is reflected by their embedding similarity: ⟨eq, ed⟩. As such, the relevant documents for the query (Dq) can be retrieved via the ANN search within the embedding space: Dq ← Top-k({d : ⟨eq, ed⟩|D}). The pre-trained language models used to be the backbone architecture of the embedding model. Take BERT as an example. The input text is tokenized as the sequence T : [CLS], t1, ..., tN, [EOS]. Then, the tokenized sequence is encoded by BERT, where the output embeddings are integrated as the text embedding. There are two common options to perform the integration: [CLS], or mean-pooling: et ← BERT(T )[CLS] or AVG ( BERT(T ) ) . When using LLMs as the encoding architecture, the text embedding needs to be generated in a different way. Since the existing LLMs mainly use the decoder-only architecture (Brown et al., 2020; Chowdhery et al., 2023; Touvron et al., 2023), the global context can only be accessed by the very last tokens of the input sequence. Therefore, the output embedding from the special token ⟨\s⟩ or [EOS] is utilized to represent the input text (Zhang et al., 2023; Ma et al., 2023). Taking LLaMA (Touvron et al., 2023) as the example, we have the following updated form of text embedding: et ← LLaMA(T )[⟨\s⟩]. (1)

heading: 3.2 Unsupervised Adaptation
text: As introduced, the LLM’s output embedding tends to foucs on local semantic, that is the prediction of the next token, because of its language modelingbased pre-training. To facilitate the application in dense retrieval, we perform unsupervised adaptation of LLM, where the LLM’s text embedding can be transformed into a semantic representation of the global context. • Objective. The retrieval tasks can be roughly divided into two types. One is to find the correlated data, e.g., question-answering (QA). The other one is to identify paraphrasing data, e.g., natural language inference (NLI). To confront these tasks, text embeddings are expected to fulfill two objectives:
• Induction. The representation of global semantic about the input text itself.
• Deduction. The representation of global semantic for the correlated text of the input.
• Pretext tasks. With the above objectives, we present two pretext training tasks of Llama2Vec. One is called EBAE (Embedding-based AutoEncoding), where the text embedding et is used to induct the sentence-level feature about input sentence itself. The other one is called EBAR (Embedding-based Auto-Regression), where the text embedding et is used to deduct the sentencelevel feature for the next sentence of the input. We argue that the strong induction and deduction capability of text embedding will be sufficient to handle the diversified retrieval scenarios, considering that arbitrary correlations can always be abstracted into the general form of input (e.g., question), retrieval purpose (e.g., get its answer), next sentence (e.g, answer). • Text Embedding. The LLM is prompted by two different templates to generate the text em-
beddings for EBAE and EBAR (Figure 1). For EBAE, the LLM is prompted by: “[Placeholder for input] SELF ⟨\s⟩”, where the inductive text embedding is generated by the following function:
eαt ← LLaMA(T, SELF, ⟨\s⟩)[−1]. (2) SELF stands for the prompt of EBAE: “The input sentence is:”. For EBAR, the LLM is prompted by template: “[Placeholder for input] NEXT ⟨\s⟩”, based on which the deductive text embedding is generated by the following function:
eβt ← LLaMA(T,NEXT, ⟨\s⟩)[−1], (3) In this place, NEXT stands for the prompt of EBAR: “The next sentence is:”.
The direct computation of the two embeddings will lead to substantial unnecessary costs because the input text T is repetitively processed for two times. To alleviate this problem, we propose to compute eαt and e β t in one pass. Particularly, the prompts of EBAE and EBAR are merged into one joint prompt for LLM: “[Placeholder for input] SELF ⟨\s⟩ NEXT ⟨\s⟩”. Because the two text embeddings need to be computed independently, we modify the typical attention mask of casual language modeling, where “SELF ⟨\s⟩” and “NEXT ⟨\s⟩” are mutually invisible (Figure 2). Now, the output embeddings of the first and second ⟨\s⟩ tokens are used for eαt and eβt , respectively. Given that the input text T account for the majority of length for the joint prompt, such an operation will save almost 50% of the computation cost compared with the naive method. • Training. As introduced, the text embeddings of Llama2Vec are adapted to capture the global semantic of the input sentence itself and the next sentence of the input. In this place, we propose a simple but effective training objective to accomplish such an adaptation. We argue that if one embedding is able to accurately predict the vocabularies in a specific context all by itself, the embedding must be a strong representation of the global semantic for the corresponding context. Based on this fundamental principle, the training of text embedding is formulated as a multi-classification problem. It linearly projects the text embedding into the vocabulary space, where the vocabulary IDs of all tokens within the target context are predicted. Specifically, the objective function of this problem is derived as:
min .− 1|T | ∑
t∈T log exp (eTW t)∑ v∈V exp (e TW v) . (4)
In this place, W ∈ Rd×|V | is the projection head of LLM; V indicates the vocabulary space; T stands for the collection of tokens of the target context (input text itself for eαt , the next sentence for e β t ). The above training objective is lightweight and simple to realize, which can be directly conducted based on the typical language modeling pipeline. • Fine-Tuning. The well-adapted LLM from Llama2Vec is fine-tuned for dense retrieval applications through contrastive learning. Because the majority of the fine-tuning datasets for dense retrieval are collected for correlation scenarios, such as QA (Nguyen et al., 2016) and Natural Question (Kwiatkowski et al., 2019), which are made up of tuples of (query, answer), we can derive the following general form of objective function:
min ∑
q
− log exp(⟨eαq , eβa⟩)∑
a′∈A′ exp(⟨eαq , e β a′⟩)
, (5)
where eαq is the query’s embedding prompted by NEXT, and eβa is the answer’s embedding prompted by SELF. Despite the fixed formulation during training, the prompt scheme can be flexibility adjusted for each individual downstream scenario. Particularly, when dealing with the correlation relationships, e.g., question-answering, we hold on to NEXT and SELF to prompt the query and answer’s embeddings. However, when handling other situations about paraphrasing relationships, the prompt scheme is as follows. To analyze the paraphrasing relationship between two long documents, we use
SELF to prompt the query and answer’s embeddings given its nature of summarizing the semantic of complex input. Meanwhile, we employ NEXT as the prompt for both inputs when dealing with two short sentences because of its nature of deducting the semantic for the related texts.

heading: 4 Experiment

heading: 4.1 Settings
text: The experimental study is performed to explore three important issues: 1) Llama2Vec’s retrieval performance after fine-tuning, 2) Llama2Vec’s generalization across diversified scenarios, 3) the impact of technical factors in Llama2Vec. With such objectives, we use the MS MARCO (Nguyen et al., 2016) as our fine-tuning dataset, and perform the evaluation on the passage retrieval and document retrieval task. We also take advantage of the BEIR benchmark (Thakur et al., 2021), where the finetuned retriever from MS MARCO is evaluated under the zero-shot setting to analyze its generalization capability. • Training. Llama2Vec is applied to the LLaMA-2-7B (base) model. The unsupervised adaptation is performed based on the unlabeled corpus of Wikipedia curated by DPR (Karpukhin et al., 2020). We perform 10,000 steps of Llama2Vec adaptation in total, with a batch size of 256, a sequence length of 1024, and a learning rate of 1e5. Llama2Vec is fine-tuned based on the training recipe presented by RepLLaMA (Ma et al., 2023):
it leverages LoRA (Hu et al., 2021) for the parameter efficient training of LLM, and simply relies on the ANN hard negatives (Xiong et al., 2020) to fine-tune the embedding model with contrastive learning.

heading: 4.2 Supervised Performance
text: First of all, we analyze the supervised retrieval quality of Llama2Vec, where the model is finetuned with the training queries from MS MARCO passage and document retrieval, respectively. • Passage Retrieval. The experiment results on MS MARCO passage retrieval are shown in Table 1. We make comparison with a wide variety of baseline methods on passage retrieval, which include the following categories: 1) basic lexical retriever: BM25 (Lin et al., 2021); 2) dense retrievers fine-tuned from BERT or RoBERTa: ANCE (Xiong et al., 2020), ADORE (Zhan et al., 2021), AR2+SimANS (Zhou et al., 2022), RocketQAv2 (Ren et al., 2021); 3) dense retrievers finetuned from the enhanced PLMs: Condenser (Gao and Callan, 2021), coCondenser (Gao and Callan, 2022), RetroMAE (Liu and Shao, 2022; Liu et al., 2023), SimLM (Wang et al., 2022b); 4) dense retrievers based on LLMs: GTR-XXL based on T54.8B (Ni et al., 2021), SGPT (Muennighoff, 2022) and OpenAI-Ada-002 (Neelakantan et al., 2022) based on GPT, RepLLaMA (Ma et al., 2023) based on LLaMA-2-7B. RepLLaMA is the closest baseline to our method, which directly fine-tunes the original LLaMA-2-7B backbone without any adaptation. There are two different fine-tuning methods (FT.): one is based on hard-negative sampling (hard): which is simple and low-cost; the other one is based on knowledge distillation (distill), which is accurate but expensive due to its demand of a precise ranker and complicated training process.
The primary observations are presented as follows. First of all, Llama2Vec achieves a superior retrieval performance in every evaluation scenario. Remarkably, it achieves a MRR@10 of 43.1 and a Recall@1000 of 99.5, which notably improves the performance of the baselines and presents a new state-of-the-art result on the large-scale dev set. Its performance is also highly competitive on DL’19 and DL’20, though it’s slightly lower on DL’19 due to the randomness of the small test set. Besides, it leads to a notably improvement over the closest baseline RepLLaMA (based on the same backbone but without adaptation), which indicates the effect introduced by the adaptation of Llama2Vec. Finally, we can observe the the LLMbased retrievers’ overwhelming advantages in comparison with the previous ones based on smaller PLMs, despite that the they are usually fine-tuned with a relatively simple approach (hard). Compared with the best PLM baseline fine-tuned by hard negatives, RetroMAE+hard and SimLM+hard, the switch to Llama2Vec brings forth almost +4% gains in MRR@10. Such a dramatic improvement validates the LLMs’ huge potential for dense retrieval, and with proper adaptation, such a potential can be exploited more effectively. • Document Retrieval. We report the evaluation results on MS MARCO document retrieval in Table 2. We make comparison with popular document retrieval methods, including BM25 (Lin et al., 2021), ADORE (Zhan et al., 2021), ANCE first-p and max-p (Xiong et al., 2020), PROP (Ma et al., 2021a), B-PROP (Ma et al., 2021b), COIL (Gao et al., 2021), COSTA (Ma et al., 2022), RepLLaMA (Ma et al., 2023), which fall into the same categories as the passage retrievers.
Our observations on document retrieval is very similar with our previous result on passage retrieval.
In particular, Llama2Vec achieves a superior empirical performance in every evaluation, where it notably improves the previous BERT-based methods by +5.7% point in MRR@100. Both LLM-based retrievers, RepLLaMA and Llama2Vec, are able to dominate the PLM-based baselines. Besides, Llama2Vec continues to outperform RepLLaMA, with a +2.3% improvement in MRR@100 on the large-scale dev set and consistent advantages on DL’19 and DL’20. The above observations further affirm our previous conclusions about the advantage of LLM backbone and the benefit from Llama2Vec. It’s worth noting that the LLM presents a powerful backbone to support document retrieval, because of not only its high expressiveness but also its long context, which enables the input document to be fully encoded instead of chunked into smaller segments.

heading: 4.3 Zero-shot Performance
text: We further investigate Llama2Vec’s impact on the generalization. We leverage the BEIR benchmark and Llama Index for the evaluation of zero-shot performances, where the fine-tuned model from MS MARCO is directly applied to its included datasets (Table 3 and Table 4).
The major observations about the BEIR evaluation result are presented as follows. First of all, Llama2Vec exhibits a remarkable performance on BEIR, where it achieves a average performance of 56.4. Such a performance is not only much higher than the rest of the baselines, but also establishes a new state-of-the-art result on BEIR (zeroshot). More impressively, it maintains the leading (bold) or the 2nd-place performance in almost every dataset, which indicates its superior versatility across different scenarios. Besides, Llama2Vec substantially outperforms RepLLaMA in most of the scenarios (12/14), which indicates the comprehensive improvement of the retriever’s generalization. It is also worth to emphasize the comparison between BM25 and dense retrievers. When the BEIR benchmark was first launched two years ago, none of the dense retrievers (BERT and many others) were able to outperform BM25 despite their competitive performances in the supervised scenarios. However, the previous situation has been largely overturned with the adoption of LLM-based text encoders, as Llama2Vec outperforms BM25 on 13/14 datasets and goes beyond its average performance by 31% relatively. The dramatic improvement can attribute to three merits of LLMs: 1) the superior expressiveness to model complex semantics, 2) the expanded context to handle long inputs, and 3) the rich knowledge to understand commonsense relationships.
Meanwhile, Llama2Vec exhibits exceptional performance on the Llama Index. Notably, Llama2Vec surpasses bge-m3 (Chen et al., 2024), OpenAI’s text-embedding-3 in both small and large embed-
ding sizes, indicating that Llama2Vec is highly competitive compared to the universal models in the community. When compared to RepLLaMa, which is only fine-tuned on the MS MARCO passage dataset, Llama2Vec also demonstrates stronger generalization abilities, showing better performance in both MRR and Hit Rate.

heading: 4.4 Technical Factors
text: Further study is made for three factors: unsupervised adaptation, prompt scheme, embedding size. • Adaptation. Our previous experiments verify the effectiveness of unsupervised adaptation given its substantial improvement over LLaMA-2-7B. In this place, we focus on exploring the the underlying reason of the empirical advantage. As introduced, the unsupervised adaptation is performed to transform the text embedding such that it can become a global semantic representation and make accurate predictions for the vocabularies within the target context (§3.2). We perform the following experiment on MS MARCO to evaluate the adaptation effect. Firstly, the query and answer embeddings are projected into distributions in the vocabulary space: dq ∈ R|V |×1 ← eαq TW ,da ∈ R|V |×1 ← eβa T W (W is the decoding head of the LLM). Then, the top-N vocabs are predicted for the query and answer: vq ← top-N(dq),va ← top-N(da). If the transformation works, there will be an increased lexical similarity between query and answer. In this place, we use BM25 to compute the similarity score: BM25(vq,va). As shown in Table 6, the adaptation (Adapt) exhibits a much higher vocab similarity over the initial LLaMA-2 backbone (Initial). Interestingly, we also observe that the lexical similarity can be improved by fine-tuning as well.
In fact, the improved lexical similarity is beneficial to the performance of dense retrieval, as studied by previous works on query and document expansion (Nogueira et al., 2019; Mao et al., 2020). With the unsupervised adaptation, such a transformation is implicitly accomplished for the text embedding. • Prompt. The well-adapted LLM encoder is able to make adaptive usage of SELF and NEXT prompt to handle different correlation and paraphrasing relationships (Ada*). Particularly, NEXTSELF (N2S) is used for correlation, SELF-SELF (S2S) is used for paraphrased documents, and NEXT-NEXT (N2N) is used for paraphrased short texts. The impact of prompt scheme is analyzed with BEIR, which contains retrieval tasks of diversified semantic relationships: for correlation, we have DBPedia (DB), FIQA (FQ), HotpotQA (HQ), NFCorpus (NF), NQ, SCIDOCS (SD), Touch (TO); for long-paraphrasing, we have Arguana (AR); for short paraphrasing, we have Climate-FEVER (CF), FEVER (FV), Quora (QR), SciFact (SF), CQADupStack(CQ). The result is shown in Table 5, where the prompt utilization exerts a major impact. The majority of tasks are about correlation; thus, N2S works the most effectively in those scenarios. In other paraphrasing cases, S2S (e.g., Arguana) and N2N (e.g., Quora) can help to achieve a better result. • Dimension. The LLMs are more expressive than smaller PLMs; however, they also come with higher costs in many perspectives. In this place, we focus on the impact from embedding size, i.e. dimension, which not only affects the computation but also determines the space cost of the vector database. We evaluate alternative dim reduction methods where the embedding size is gradually
reduced from 4096 to 768 (Table 7). One is to jointly learn the LLM and a projection head during fine-tuning (DimRed); another one is to fixed the well fine-tuned LLM retriever and learn the projection head via distillation (Liu et al., 2022) (DimRed*). Unfortunately, both methods suffer from notably performance loss after dim reduction. We further replace linear projection with sparsification (Formal et al., 2021), where the top-N entries are selected for the embedding (Sparse). Compared with the first two options, sparsification turns out to be more effective in preserving retrieval performance. The above observations also suggest the necessity of light-weight processing of LLM-based retrievers in the future.

heading: 5 Conclusion
text: In this paper, we present Llama2Vec to enhance the LLM-based dense retrieval via unsupervised adaptation. Llama2Vec is composed of two pretext tasks, EBAE and EBAR, where the LLM is prompted to reconstruct the input sentence and predict the following sentence purely with its text embeddings. On top of the unsupervised adaptation, the LLM’s text embedding can be transformed into a strong representation of global context. By using suitable prompts, it can flexibly support the semantic matching of different correlation and paraphrasing relationships. The effectiveness of Llama2Vec is verified by comprehensive experiments, where the adapted LLM achieves new state-of-the-art performances in both supervised and zero-shot evaluations, indicating the substantial improvements on both accuracy and generalization capability of the retrieval model.

heading: 6 Limitation
text: While Llama2Vec has made a substantial progress in adapting the LLM as a strong dense retriever, the current work can still be improved in the following ways. Firstly, the current method is only applied to a 7B model, it remain to explore its impact on larger LLMs. Secondly, the current model is for English centric scenario, it is necessary to make extensions for other languages. Thirdly, it is also important to find effective ways to maintain an efficient running cost for such large-scale embedding models.

heading: 7 Ethical Consideration
text: Llama2Vec is built upon LLaMA-2, it inherits potential biases, toxicity, and other problems present
in the underlying LLM. Therefore, we do not recommend utilizing Llama2Vec for retrieval purposes in sensitive contexts. Moreover, the embedding may be influenced by the training data, potentially leading to biased or discriminatory results.

heading: 8 Acknowledgements
text: This work is supported by the National Natural Science Foundation of China (Nos. 62272054, 62192784), Beijing Nova Program (No. 20230484319), Xiaomi Young Talents Program, and National Science and Technology Major Project (2023ZD0121504).

heading: Future Work
text: The above observations also suggest the necessity of light-weight processing of LLM-based retrievers in the future