LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders

Authors: parishad  behnamghader
vaibhav  adlakha
marius  mosbach
dzmitry  bahdanau
nicolas  chapados
siva  reddy
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 22:02:29

Abstract: Large decoder-only language models (LLMs) are the state-of-the-art models on most of today's NLP tasks and benchmarks. Yet, the community is only slowly adopting these models for text embedding tasks, which require rich contextualized representations. In this work, we introduce LLM2Vec, a simple unsupervised approach that can transform any decoder-only LLM into a strong text encoder. LLM2Vec consists of three simple steps: 1) enabling bidirectional attention, 2) masked next token prediction, and 3) unsupervised contrastive learning. We demonstrate the effectiveness of LLM2Vec by applying it to 4 popular LLMs ranging from 1.3B to 8B parameters and evaluate the transformed models on English word- and sequence-level tasks. We outperform encoder-only models by a large margin on word-level tasks and reach a new unsupervised state-of-the-art performance on the Massive Text Embeddings Benchmark (MTEB). Moreover, when combining LLM2Vec with supervised contrastive learning, we achieve state-of-the-art performance on MTEB among models that train only on publicly available data (as of May 24, 2024). Our strong empirical results and extensive analysis demonstrate that LLMs can be effectively transformed into universal text encoders in a parameter-efficient manner without the need for expensive adaptation or synthetic GPT-4 generated data.

Main paper: 
 introduction: 
 Text embedding models aim to encode the semantic content of natural language text in vector representations which then facilitate various natural language processing (NLP) tasks, such as semantic textual similarity, information retrieval, and clustering. For many years, the dominating paradigm for building such models relied on pre-trained bidirectional encoders or encoder-decoders such as BERT (Devlin et al., 2019) and T5 (Raffel et al., 2020) , which are typically adapted for text embedding tasks by following a multi-step training pipeline consisting of weakly-and fully-supervised contrastive training (Ni et al., 2022; Li et al., 2023a; Xiao et al., 2023, inter alia) . Only recently, the community started to adopt decoder-only LLMs for embedding text (Muennighoff, 2022; Ma et al., 2023; Wang et al., 2023; Springer et al., 2024; Li & Li, 2024) .We speculate that the slow adoption of decoder-only LLMs for text embedding tasks is partly due to their causal attention mechanism, which inherently limits their ability to produce rich contextualized representations. At any given layer, causal attention limits token interactions, ensuring that the representation of a token at position i is influenced solely by the representations of preceding tokens at positions 0, 1, . . . , i -1. Although this limitation is necessary for generative capabilities, it is sub-optimal for text embeddings as it prevents the representations from capturing information across the entire input sequence.Overcoming this architectural limitation of decoder-only LLMs for text embedding tasks is highly appealing as these models come with several advantages compared to their encoder-Masked Next Token Prediction Unsupervised Contrastive LearningFigure 1 : The 3 steps of LLM2Vec. First, we enable bidirectional attention to overcome the restrictions of causal attention (Bi). Second, we adapt the model to use bidirectional attention by masked next token prediction training (MNTP). Third, we apply unsupervised contrastive learning with mean pooling to learn better sequence representations (SimCSE).only counterparts.1 During pre-training, decoder-only LLMs learn from all input tokens and not just a small percentage2 , which-given the same amount of training data-makes them much more sample-efficient than encoder-only models (Clark et al., 2020) . Moreover, there exists a rich ecosystem around these models, with extensive tooling and well tested pre-training recipes, which has resulted in continuous improvement of these models by the community. Lastly, recent work on instruction fine-tuning and learning from human preferences has resulted in decoder-only LLMs that excel at instruction following (Wang et al., 2022b; Ouyang et al., 2022) , making them an ideal choice for building universal text embedding models that generalize across a large variety of tasks using instructions.In this work, we provide a simple unsupervised approach, termed LLM2Vec, which can be used to transform any pre-trained decoder-only LLM into a (universal) text encoder. As shown in Figure 1 , LLM2Vec consists of three simple steps: 1) enabling bidirectional attention, 2) masked next token prediction, and 3) unsupervised contrastive learning. Crucially, LLM2Vec does not require any labeled data and is highly data-and parameter-efficient.We apply LLM2vec to 4 decoder-only LLMs ranging from 1.3B to 8B parameters (S-LLaMA-1.3B, LLaMA-2-7B, Mistral-7B, Meta-LLaMA-3-8B) and evaluate the resulting models on word-and sequence-level tasks. On word-level tasks (chunking, named-entity recognition, and part-of-speech tagging), LLM2Vec-transformed models outperform strong encoder-only models by a large margin, demonstrating its effectiveness for producing rich contextualized token representations. On the Massive Text Embeddings Benchmark (MTEB), LLM2Vec-transformed models set a new state-of-the-art for unsupervised models, with our best model reaching a score of 56.8. Additionally, we combine LLM2Vec with supervised contrastive training and achieve a new state-of-the-art performance among models that train only on publicly available data. Beyond our strong empirical results, we provide an extensive analysis of how LLM2Vec affects the representations of the underlying model and reveal an intriguing property of Mistral-7B, which can handle bidirectional attention without any fine-tuning.Overall, our work demonstrates that decoder-only LLMs are indeed capable of producing universal text embedding and only very little adaptation is required to reveal this ability. Our code and pre-trained models is publicly available at https://github.com/McGill-NLP/ llm2vec 2 LLM2Vec 
three simple ingredients: 
 Enabling bidirectional attention The first step of the LLM2Vec approach is to replace the causal attention mask of decoder-only LLMs by an all-ones matrix (see Appendix B.1 for background on the self-attention). This gives each token access to every other token in the sequence, converting it into a bidirectional LLM (Devlin et al., 2019; Liu et al., 2019) . However, it is not a priori clear, why this should lead to better sequence representations. After all, the decoder-only LLM was not trained to attend to future tokens and therefore, this naive approach might even lead to worse representations. As we show, simply enabling bidirectional attention does indeed decrease in embedding performance for most models.We can however easily adapt a model to make use of its bidirectional attention.We use a simple strategy to make the model aware of its bidirectional attention by adapting it via masked next token prediction (MNTP). MNTP is a training objective that combines next token prediction with masked language modeling (Lv et al., 2023) . Given an arbitrary sequence x = (x 1 , x 2 , . . . , x N ) as input, we first mask a fraction of the input tokens and then train the model to predict the masked tokens based on the past and future context. Crucially, when predicting a masked token at position i, we compute the loss based on the logits obtained from the token representation at the previous position i -1, not the masked position itself (see Figure 1 ).While the previous two steps of the LLM2Vec recipe can transform any decoder-only LLM into an encoder for word-level tasks, they might not be sufficient for sequence representations. Unlike bidirectional encoders that include a next sentence prediction objective in their pre-training objectives (Devlin et al., 2019) , decoderonly LLMs are not explicitly trained to capture the context of the entire sequence. To fill this gap, we apply unsupervised contrastive learning via SimCSE (Gao et al., 2021) . Specifically, given an input sentence, it is passed through the model twice with independently sampled dropout masks, resulting in two different representations for the same sentence. The model is trained to maximize the similarity between these two representations while minimizing the similarity with representations of other sentences in the batch. Crucially, this step does not require any sentence pair data and can be applied using any collection of sentences.We use a pooling operation on the word representations to get the sentence representation (more details in Section 3.2).Models For most of our results, we experiment with 3 different decoder-only LLMs ranging from 1.3B to 7B parameters: Sheared-LLaMA-1.3B (S-LLaMA-1.3B, Xia et al., 2023) , Llama-2-7B-chat (LLaMA-2-7B, Touvron et al., 2023) , and Mistral-7B-Instruct-v0.2 (Mistral-7B, Jiang et al., 2023a) . In Tables 1 and 2 , we provide additional results for the recently released Meta-Llama-3-8B-Instruct model (Meta-LLaMA-3-8B, AI@Meta, 2024).We perform both the MNTP and the unsupervised SimCSE step using data from English Wikipedia. We select data from Wikipedia as it is presumably included in the pre-training mixture of all the models we experiment with. It is therefore fair to assume that these two adaptation steps are not teaching the model any new knowledge beyond how to attend to future tokens and how to construct sequence representations. Specifically, we use the Wikitext-103 dataset (Merity et al., 2017) for the MNTP step and a subset of Wikipedia sentences released by Gao et al. (2021) for the unsupervised SimCSE step.We follow established practice from the masked language modeling literature and randomly mask a fraction of the tokens from the input sequence (Devlin et al., 2019; Liu et al., 2019) . We use the underscore ( ) as the mask token, since the models we experiment with do not have a special token for masking. We fine-tune the model using LoRA (Hu et al., 2022) to predict the masked token using the representation of the previous token to maximally align our training objective with the pre-training setup of decoder-only LLMs. For all models, we trained for 1000 steps with a batch size of 32 on a single 80GB A100 GPU. For 7B and 8B models, this training takes only 100 minutes. We provide additional details of our training setup and hyperparameters in Appendix D.1.1.For the contrastive training, we apply the unsupervised SimCSE approach by Gao et al. (2021) . The positive examples are constructed by applying LLM's dropout twice on the same input sequence, whereas the other sequences in the batch act as in-batch negatives. We merge the MNTP LoRA weights into the base model and initialize new LoRA parameters before starting the SimCSE training, which ensures that the models retains the knowledge learned in the previous step. Similar to the MNTP step, we train for 1000 steps. For 7B and 8B models, this training takes 3 hours on a single 80GB A100 GPU with a batch size of 128. We provide additional details of our training setup and hyperparameters in Appendix D.1.2.3 LLM2Vec-transformed models are strong unsupervised text embedders 
evaluation on word-level tasks: 
 We start by evaluating on word-level tasks to demonstrate that LLM2Vec is successful at improving the contextual representations constructed by decoder-only LLMs.Setup We evaluate three word-level tasks: chunking, named-entity recognition (NER), and part-of-speech tagging (POS), using the CoNLL-2003 benchmark (Tjong Kim Sang & De Meulder, 2003) . We embed each input sentence and train a task-specific linear classifier on top of the frozen representations. This is akin to the linear probing setup commonly used in the language model analysis literature (Belinkov, 2022) . We compare the LLM2Vectransformed models to DeBERTa-v3-large (He et al., 2023) , the current state-of-the-art encoder-only model. Additional details about our setup are provided in Appendix D.1.3.Figure 2 shows the results of our evaluation (a detailed breakdown of the results is provided in Table 4 ). On each of the three tasks, constructing token representations with causal attention (Uni) already outperforms the encoder-only baseline. This is not surprising, given that the models we experiment with are significantly larger and have been pretrained on more data. As expected, naively applying bidirectional attention dramatically hurts performance in most cases. Interestingly, for Mistral-7B, enabling bidirectional attention hurts performance much less compared to S-LLaMA-1.3B and LLaMA-2-7B. For NER, Mistral's performance even improves by 0.6% with bidirectional connections.Focusing on the LLM2Vec-transformed models, we observe that for all models and tasks, adapting via MNTP improves performance. For instance, in the chunking task, we see improvements for S-LLaMA-1.3B (by 5%), LLaMA-2-7B (by 4%), and Mistral-7B (by 4%).Combining MNTP with SimCSE, however, performs worse than just applying MNTP. This is expected for word-level tasks, as SimCSE adapts the representations for sequence-level tasks. Figure 3 : Unsupervised results on our 15 task subset of the MTEB dataset. We ablate three different pooling choices: EOS, mean pooling, and weighted mean pooling. LLM2Vec is compatible with all three approaches and works best with mean pooling.Next, we evaluate on the Massive Text Embedding Benchmark (MTEB), a collection of 7 diverse embedding task categories covering a total of 56 datasets (Muennighoff et al., 2023) . To select the best-performing pooling method for each method, we perform ablations on a 15 task subset consisting of representative tasks from each of the MTEB categories. We provide additional details and justification for how we chose this subset in Appendix C.1.Setup Following previous work (Su et al., 2023; Wang et al., 2023; Springer et al., 2024) , we evaluate with task-specific instructions. For a fair comparison, we use the same set of instructions as Wang et al. (2023) which are also used by Springer et al. (2024) . The instructions are only added to queries and can be found in Table 10 of Appendix C.2. For symmetric tasks, the same instruction will be used for the query and the document. When applying (weighted) mean pooling (Muennighoff, 2022) , we exclude the instruction tokens.As a baseline, we compare to the unsupervised BERT models obtained from Gao et al. (2021) . Additionally, we compare to Echo embeddings, a concurrent approach by Springer et al. (2024) , which we run with the same models and instructions (see Appendix E.1 for more details on our implementation of Echo embeddings). Echo duplicates the input and takes the pooling over the second occurrence to address the limitation of causal information flow.Results on our 15 task subset of MTEB Figure 3 shows the impact of various pooling methods for all three models on the subset of MTEB tasks. We can clearly observe that applying causal attention is sub-optimal when constructing text embeddings. The dominant paradigm of applying the EOS pooling for models with causal attention is outperformed by (weighted) mean pooling. Enabling bidirectional attention without any training harms performance for S-LLaMA-1.3B and LLaMA-2-7B. Similar to our word-level results, the performance of Mistral-7B improves with bidirectional attention, even without any training.For LLM2Vec-transformed models, applying MNTP training improves the performance of S-LLaMA-1.3B and Mistral-7B. Moreover, applying SimCSE further boosts the performance of S-LLaMA-1.3B, LLaMA-2-7B, and Mistral-7B by 49.8%, 23.2%, and 37.5% compared to the best causal baseline on the MTEB subset. We further conduct an ablation of each component of LLM2Vec recipe in Appendix D.2.2 (Table 5 ). 1 shows the results of the best performing models, which we select based on the ablation above, on the full MTEB dataset. After the first two steps of LLM2Vec-bidirectional attention and MNTP-we observe a considerable improvement in performance for all four models (e.g., 16.4% improvement for Mistral-7B).When comparing to Echo embeddings, LLM2Vec (the first two steps only)3 leads to improved performance for S-LLaMA-1.3B, LLaMA-2-7B, and Meta-LLaMA-3-8B, and performs almost on par for Mistral-7B. However, compared to Echo embeddings, LLM2Vec is much Table 1 : Unsupervised results on MTEB. We compare S-LLaMA-1.3B, LLaMA-2-7B, Mistral-7B, and Meta-LLaMA-3-8B with and without LLM2Vec to the unsupervised BERT models of Gao et al. (2021) as well as Echo embeddings (Springer et al., 2024) .more efficient as Echo embeddings repeat the input and therefore double the sequence length which makes inference considerably slower (we provide a runtime comparison in Appendix E.2). Adding the final step of the LLM2Vec recipe-unsupervised SimCSEfurther boosts all three models by a large margin, making our LLM2Vec Mistral-7B SOTA among all unsupervised models with the score of 56.80.Interestingly, Meta-LLaMA-3-8B with LLM2Vec (w/o SimCSE) outperforms echo embeddings by a larger margin compared to the other models. Adding SimCSE again boosts performance, but does not outperform LLM2Vec applied to Mistral-7B.Overall, our results highlight that LLM2Vec is successful at transforming decoder-only LLMs into strong text embedding models which outperform previous unsupervised approaches on the challenging MTEB leaderboard.4 How does LLM2Vec affect a model? 
llm2vec helps models to capture information from future tokens: 
 To analyze the extent to which LLM2Vec-transformed models incorporate information from future tokens, we adopt the analysis of Springer et al. (2024) and test how well the model performs at judging the similarity between sentences that share the same prefix.Setup We evaluate on a synthetic dataset collected by Springer et al. (2024) , which consists of 35 sentence triples {(q i , s+ i , s - i )} 35 i=1 with q i = (A i , B i ), s + i = (A i , C i ), and s - i = (A i , D i ), where B i and C i have a similar meaning but B i and D i don't. We compute a sequence representation for each of these sentences by pooling only over the first part of the sentence, i.e., A i . We then compute the cosine similarity between the resulting embeddings. A model that incorporates information from future tokens (B i , C i , or D i ) in the representations of the prefix A i should assign a higher similarity to the positive example.Figure 4 shows the results of our analysis for S-LLaMA-1.3B and Mistral-7B.Results for LLaMA-2-7B, which show the same trends, and a comparison to Echo embeddings are provided in Appendix F. For S-LLaMA-1.3B, we observe that enabling bidirectional attention and training with the MNTP objective are sufficient to establish a clear separation between the positive and negative examples. For Mistral-7B, all setups lead to a larger cosine similarity between the query and positive than the query and negative examples.Our empirical results so far as well as the analysis above share an intriguing observation: enabling bidirectional attention works well for Mistral-7B, even without any training. Below, we investigate this surprising behavior by analyzing how bidirectional attention impacts the representations of a model.We feed a single input sequence (a random paragraph from Wikipedia) to each model and compute the hidden representations of every token at every layer l with causal (H c l ) and bidirectional attention (H bi l ). For every layer, we compute the cosine similarity between the representations constructed using causal and bidirectional attention, i.e., sim(H c l , H bi l ). For most layers, we expect this similarity to be low, as enabling bidirectional attention without any training should lead to substantially different representations.Figure 5 shows that as expected, for S-LLaMA-1.3B and LLaMA-2-7B, enabling bidirectional attention without training has a profound impact on the representations, leading to low cosine similarity across almost all layers and token positions. For Mistral-7B, on the other hand, the representations have very high cosine similarity throughout. Based on these findings (we replicate these results for other inputs and other variants of Mistral in Appendix F) and the strong unsupervised results for Mistral-7B with bidirectional attention, we speculate that Mistral models are pre-trained with some form bidirectional attention, e.g., prefix language modeling (Raffel et al., 2020) -at least for some parts of its training. We leave a more detailed investigation of this intriguing behavior for future work. 
combining llm2vec with supervised contrastive learning: 
 The final piece of our evaluation combines LLM2Vec with supervised contrastive learning.Setup For supervised training, we train on a replication of the public portion of the E5 dataset (Wang et al., 2023) curated by Springer et al. (2024) . The dataset consists of approximately 1.5M samples and we provide details on its compilation in Appendix G.1. We follow standard practice and train the models with contrastive learning using hard negatives and in-batch negatives. We use LoRA fine-tuning for supervised setting as well.The MNTP LoRA weights are merged into the base model, and the trainable LoRA weights are initialized with SimCSE weights. For LLM2Vec models that use just MNTP, the LoRA weights are randomly initialized. The training is performed for 1000 steps with a batch size of 512. We detail other hyperparameters in Appendix G.2.For a fair comparison, we only compare to models trained on publicly available data and provide a comparison to the top entries on the MTEB leaderboard in Appendix G.3.Table 2 shows the results of our evaluation. For all models, transforming a model with LLM2Vec leads to improved performance over the strong Uni + weighted mean baseline. As expected, performing unsupervised SimCSE is less crucial for supervised training, and even leads to slightly worse performance for LLaMA-2-7B, Mistral-7B, and Meta-LLaMA-3-8B compared to just performing the MNTP step of LLM2Vec (LLM2Vec w/o SimCSE). However, as we will show in Section 5.2, LLM2Vec with MNTP and SimCSE is much more sample-efficient, and therefore crucial in compute or data-constrained settings. Notably, our best model, Meta-LLaMA-3-8B + LLM2Vec (w/o SimCSE) leads to a new state-of-the-art performance among models trained only on publicly available data.Setup To demonstrate the sample-efficiency of LLM2Vec-transformed models, we save a checkpoint every 25 training steps and evaluate them on our 15 task subset of MTEB.As shown in Figure 6 , LLM2Vec-transformed models reach better performance earlier in training. This observation is consistent across all three models. For S-LLaMA-1.3B, the smallest of our three models, even performing just MNTP leads to a considerably improved sample-efficiency. These results are particularly encouraging for settings where it is hard to acquire high quality labeled data, a setting which we leave for future work. 
related work: 
 Supervised text encoders Initially, supervised methods primarily relied on tasks such as natural language inference or sentence similarity to train BERT-like models for producing sentence embeddings (Conneau et al., 2017; Reimers & Gurevych, 2019) . Subsequently, BERT-like models have also been adapted to tasks like retrieval (Karpukhin et al., 2020; Khattab & Zaharia, 2020) . More recent methods have further improved these representations through a complex multi-stage learning pipeline that consists of large-scale weakly supervised contrastive training followed by multi-task fine-tuning (Ni et al., 2022; Wang et al., 2022a; Li et al., 2023a; Xiao et al., 2023) Recent approaches have focused on enhancing the generalization and transferability of text embeddings using instructions (Su et al., 2023; Asai et al., 2023) .Another line of work has explored training text embedders in an unsupervised manner using only a set of unordered sentences. These unsupervised approaches typically create two different representations of the same sentence for contrastive learning. The methods vary in how they form these representations -perturbing the input sentence (Wu et al., 2020) , or using different model instances (Carlsson et al., 2021) . SimCSE (Gao et al., 2021) , the approach used in this work, generates two representations of the same sentence by passing it through the model twice with different dropout masks.Turning decoder-only LLMs into text encoders While decoder-only LLMs have outperformed bidirectional encoders across a large variety of language understanding tasks (Brown et al., 2020; Touvron et al., 2023; Jiang et al., 2023a, inter alia) , their impact on sentence representation learning remains limited. The most common approaches in literature use the final hidden state of the last token as the sentence embedding (Neelakantan et al., 2022; Ma et al., 2023; Wang et al., 2023) .There are few works that explore the limitations of using a causal attention mask when adapting decoder-only LLMs for text classification and sentence representation tasks. Li et al. (2023b) experiment with removing the causal mask of Llama-2 during supervised fine-tuning for text classification and NER tasks. Similarly, Dukić & Šnajder (2024) enable bidirectional attention for a group of layers during supervised fine-tuning on NER and chunking. In the context of sentence representation learning, Li & Li (2024) explore enabling bidirectional attention in the last layer of a decoder-only model during supervised contrastive fine-tuning on STS tasks.Concurrent to our work, several works have focused on converting decoder-only-LLMs to text encoders in supervised and unsupervised manner. Jiang et al. (2023b) and Lei et al. (2024) prompt the language model to summarize the input text in one word, and take the last layer's hidden embedding for the last token as the text's representation. Muennighoff et al. (2024) perform multi-task full fine-tuning using a combination of self-supervised language modeling with causal attention and supervised contrastive learning with bidirectional attention. In contrast, our proposed approach is much more computationally efficient, as it requires only parameter-efficient fine-tuning and 1000 gradient steps. Closest to our work is the concurrent work of Springer et al. (2024) . They propose to copy the input sequence and append it to itself, which addresses the contextualization issue of causal attention as tokens in the copy of the input can now attend to "future" tokens in the previous sequence. While this performs well in practice, it significantly increases the computational cost at inference time, which can be particularly problematic for encoding longer documents. Our approach outperforms Springer et al. (2024) , without inducing any additional computational overhead at inference time. 
conclusion: 
 We present LLM2Vec, a strong unsupervised approach to transform any decoder-only LLMs into a (universal) text embedder. We perform an extensive evaluation on word-and sequence-level tasks and demonstrate the effectiveness of LLM2Vec in both unsupervised and supervised settings. Applying LLM2Vec to Mistral-7B achieves a new state-of-the-art performance on MTEB among unsupervised approaches. When combining LLM2Vec with supervised contrastive fine-tuning, Meta-LLaMA-3-8B achieves SOTA performance among approaches that train only on publicly available data (as of May 24, 2024). Beyond our strong empirical contributions, we provide an extensive analysis of how LLM2Vec impacts the underlying model and reveal an intriguing property of Mistral-7B, which explains its strong out of the box performance with bidirectional attention. The simplicity of our approach, as well as its compute and sample-efficiency, makes LLM2vec a promising solution for low-resource and compute constrained scenarios and opens up several interesting avenues for future work. 
 
 Appendix: Large size of decoder-only LLMs Recent years have seen a increasing trend towards training very large decoder-only LLMs, with model sizes up to 540B parameters (Brown et al., 2020; Chowdhery et al., 2023) . The parameter size of the model has a direct impact on the training and inference latency. Additionally, the large output embedding dimension of these models (e.g., 4096 for Mistral-7B compared to 768 for BERT) also makes them more memory and compute intensive for creating vector indexes for large document collections. While some of these limitations can be offset by recent advances in improving the training and inference efficiency of large models (Hu et al., 2022; Dao, 2024) , these techniques can technically be applied to smaller bidirectional models as well.The advantages of small bidirectional encoders come at the cost of complex and computationally intensive training regimes (Li et al., 2023a; Xiao et al., 2023; Li et al., 2023a) . In contrast, decoder-only models are much more sample-efficient and do not require large-scale contrastive pre-training (Wang et al., 2023) . Moreover, the instruction following capabilities of decoder-only models make them strong contenders for building text embedding models that generalize to a wide range of tasks and domains without the need for expensive adaptation.While smaller models can be more practical for some applications, the sample-efficiency, the instruction following capabilities, and the widespread use of these models in the community motivates the need to explore the potential of decoder-only LLMs for text embedding tasks.As our supervised data contains train splits of publicly available datasets, there is an extremely low chance of test set contamination with the MTEB benchmark. However there is a possibility of contamination from the pre-training data of LLaMA-2-7B and Mistral-7B models (S-LLaMA-1.3B was distilled from LLaMA-2-7B). As the complete details of the pre-training data are not publicly available, we cannot be certain about the extent of contamination. However, to reliably compare with other works, we stick to our choice of model and evaluation benchmark. We leave it to future work to investigate the performance of these models on newly designed benchmarks that are not part of their pre-training data.In this work, we have implemented and evaluated our proposed methodology -LLM2Vec -using only English text corpora and benchmarks. However, the methodology is language-agnostic and can be easily extended to other languages using just unstructured text collections. We leave it to future work to investigate the performance of LLM2Vec on other languages. 2018). Given a sequence of N tokens, the token representation at any given transformer layer (x 1 , x 2 , . . . , x N ) with x i ∈ R d are stacked into a matrix X ∈ R N×d . Given this matrix, the self-attention mechanism computes the query, key, and value matrices Q, K, V ∈ R N×p via a learned linear transformation.EQUATIONEQUATION)The output of the self-attention layer is then computed as a linear combination of the values, weighted by the normalized inner product between keys and queries:EQUATION)This output is then passed through a feed-forward network and added to the residual stream to obtain the token representations at the next layer. Crucially, in the case of decoder-only LLMs, the attention mask M {j≤i} prevents accessing token embeddings to the right of the current token. 2024). In the supervised setup, we have a set of positive pairs D = {(q i , d + i )} n i=1 , and a set of negative documents that can include hard or in-batch negatives. The model is trained to maximize the similarity (i.e., usually cosine similarity) of positive pairs and minimize the similarity of negative pairs, i.e., we optimize the following objective:EQUATION)where s is a similarity metric, λ a temperature value, and N all the negative documents for query q.Unsupervised contrastive learning In unsupervised contrastive learning, no positive or hard negative pairs are available. Most unsupervised approaches construct two different representation for the same sample, using either model or input perturbations. SimCSE (Gao et al., 2021) , the unsupervised approach used in this work, creates two different representations of the same input by using independently sampled dropout masks in the intermediate model representations and train the model with in-batch negatives.C.1 MTEB subset details MTEB consists of diverse small and large embedding tasks. To speed up the evaluation4 , we consider a representative subset of 15 tasks from MTEB for our analyses, presented in Table 3 .To make sure that our ablation and analyses are not biased towards one specific category or task, this subset includes tasks from each category with almost the same proportion compared to the full MTEB5 .When evaluating on MTEB, we use the same instructions as Wang et al. (2023) . The list of instructions for each task is listed in Table 10 . The second step of LLM2Vec includes MNTP training. We follow established practice from the encoder-only literature for choosing our masking strategy. For example, (Devlin et al., 2019) mask 15% of the tokens in the input. 10% of the masked tokens are then replaced with a random token from the vocabulary, while another 10% are unmasked again, but still considered when computing the loss. As another example, RoBERTa (Liu et al., 2019 ) also masks 15% of the input tokens but applies no further post-processing to the masked tokens.For our models, we perform a hyperparameter search to select the percentage of the masked tokens in a sequence choosing from 20%, 40%, 60%, 80%, and 90%. For each model, we take the best setup (i.e., masking probability and BERT vs. RoBERTa approach) based on the performance on SICK-R (Agirre et al., 2014) task from the MTEB dataset. This results in the following choices: for S-LLaMA-1.3B, LLaMA-2-7B, and Meta-LLaMA-3-8B, we apply BERT's masking strategy with masking probability of 20%. For Mistral-7B, we apply RoBERTa's masking strategy with probability of 80%.We train all the models for 1000 steps with LoRA r = 16 and α = 32, and we follow the same training parameters as RoBERTa MNTP training. When training large 7B and 8B models, we apply brain floating point (bfloat16) quantization, as well as flash attention 2 and gradient checkpointing.The last step of LLM2Vec involves unsupervised contrastive learning with SimCSE. Our initial experiments indicated that the low value of dropout probability (0.1) typically used by bidirectional encoders (Gao et al., 2021) does not lead to optimal performance for larger decoder-only LLMs. Therefore, we use a higher dropout probability of 0.3 for all models.Similar to MNTP, we train all models with LoRA r = 16 and α = 32 for 1000 steps.For LLaMA-2-7B, Mistral-7B, and Meta-LLaMA-3-8B, we train with a batch size of 128.For S-LLaMA-1.3B, we use a batch size of 32. Additionally, when training LLaMA-2-7BMistral-7B, and Meta-LLaMA-3-8B, we apply brain floating point (bfloat16) quantization, flash attention 2, and gradient checkpointing.We evaluate on three popular word embedding tasks: chunking, named-entity recognition (NER), and part-of-speech (POS) tagging. We train a linear classifier using dropout with a dropout probability of 0.1 on top of the frozen representations obtained from the last hidden layer of a model.We use data from CoNLL-2003, consisting of roughly 14,000 training, 3,250 validation, and 3,450 test samples (Tjong Kim Sang & De Meulder, 2003) . We train the classifier for 1,500 steps with a learning rate of 5e -4 and a batch size of 8. For experiments with Mistral-7B models that have been tuned with MNTP, we use the variant which is trained with BERT's masking strategy and masking probability of 20% (please see D.1.1 for more details). Although 80% masking helps with the performance in sentence-level tasks, it prevents the model from learning proper token representations essential for word-level tasks.Since the models we experiment with have sub-token based vocabularies, we calculate the embedding of a word by averaging the representations of all its sub-tokens. For example, for a sentence "w 1 w 2 w 3 " which is tokenized as "BOS t 11 t 12 t 21 t 22 t 23 t 31 ", the representation of w 1 , w 2 , and w 3 will be computed ase 1 = 1 2 (e 11 + e 12 ) , e 2 = 1 3 (e 21 + e 22 + e 23 ) , e 3 = e 31 .Here, e . is the final representation of token t . or word w . . Moreover, for the models that have gone through MNTP, we calculate the representation based on sub-tokens of the previous word. Using the same example as above, for models trained with MNTP, the representation of words w 1 , w 2 , and w 3 will be computed as:e 1 = 1 2 (e BOS + e 11) , e 2 = 1 3 (e 12 + e 21 + e 22 ) , e 3 = e 23 .We present the detailed breakdown of the performance of LLM2Vec-transformed models on the word-level tasks in Table 4 . Our results show that applying MNTP training to decoderonly LLMs helps them take advantage of the enabled bidirectional attention which boosts their performance on word-level tasks.Table 5 presents the results on MTEB subset for all models across different pooling methods.Results show that While weighted mean works the best for causal (i.e., Uni) models, mean pooling performs the best for LLM2Vec approach.In Table 11 , we additionally present a breakdown of the unsupervised performance of LLM2Vec-transformed models on MTEB.E.1 Reproducibility Concurrent to our work, Springer et al. (2024) proposed Echo embeddings, a simple approach to convert decoder-only LLMs into text embedders by copying the input sequence and appending it to itself. For evaluation, they follow a prompt sampling procedure for the task instruction. However, they report that the exact wording or template used as a prompting strategy does not have a strong effect on the performance. For a fair comparison to our proposed models, we implement Echo embeddings using the instructions in our evaluation setup (Appendix C.2). To do a sanity check on our implementation, as well as to see the impact of exact wording of instructions, we evaluate Representation analysis Figure 8 provides additional plots for the representation analysis using two different Wikipedia paragraphs. The trends closely follow those reported in Section 4. Figure 9 shows that the same behavior we observe for Mistral-7B-instruct-v0.2 Given a web search query, retrieve relevant passages that answer the query MSMARCO Document Given a web search query, retrieve relevant documents that answer the query NQ Given a question, retrieve Wikipedia passages that answer the question QuoraDuplicates Given a question, retrieve questions that are semantically equivalent to the given question Find questions that have the same meaning as the input question SQuAD Retrieve Wikipedia passages that answer the question T2RankingGiven a Chinese search query, retrieve web passages that answer the question TriviaQA Retrieve Wikipedia passages that answer the question Table 8 : Instructions used for each of the E5 datasets. also holds true for other variants of the Mistral-7B model. We take this as additional evidence that the Mistral-7B base model was trained with some for of bidirectional attention.The dataset consists of ELI5 (sample ratio 0.1) (Fan et al., 2019) , HotpotQA (Yang et al., 2018) , FEVER (Thorne et al., 2018) , MIRACL (Zhang et al., 2023) , MS-MARCO passage ranking (sample ratio 0.5) and document ranking (sample ratio 0.2) (), NQ (Karpukhin et al., 2020) , NLI (Gao et al., 2021) , SQuAD (Rajpurkar et al., 2016) , TriviaQA (Joshi et al., 2017) , Quora Duplicate Questions (sample ratio 0.1) (DataCanary et al., 2017) , Mr-TyDi (Zhang et al., 2021) , DuReader (He et al., 2018) , and T2Ranking (sample ratio 0.5) (Xie et al., 2023) . The instruction used for each dataset can be found in Table 8 .All models are trained with LoRA r = 16 and α = 32, brain floating point (bfloat16) quantization, gradient checkpointing, and flash attention 2 (Dao, 2024) to optimize GPU memory consumption. We train on 8 NVIDIA A100 GPUs with an effective batch size of 512 for 1000 steps using a maximum sequence length of 512 tokens. We use the Adam optimizer with a learning rate of 2e -4 and a linear learning rate warm-up for the first 300 steps. Table 2 presents the performance of applying Bi + MNTP and Bi + MNTP + SimCSE with mean pooling on MTEB benchmark. We also compare the performance of our models with recent and popular models trained with only publicly available data. We further report the current top-10 models in the MTEB leaderboard, including LLM2Vec Mistral-7B in Table 9 .Our models achieve the 6th score in the MTEB leaderboard and the 1st among the models trained with only public data.We present the detailed performance of supervised LLM2Vec-transformed models on full MTEB in Table 12 . Here, we only report the Bi + MNTP transformed models as we showed they perform the best after supervised fine-tuning. Given a financial question, retrieve user replies that best answer the question HotpotQA Given a multi-hop question, retrieve documents that can help answer the question MSMARCO Given a web search query, retrieve relevant passages that answer the query NFCorpus Given a question, retrieve relevant documents that best answer the question NQ Given a question, retrieve Wikipedia passages that answer the question QuoraRetrieval Given a question, retrieve questions that are semantically equivalent to the given question SCIDOCS Given a scientific paper title, retrieve paper abstracts that are cited by the given paper SciFact Given a scientific claim, retrieve documents that support or refute the claim Touche2020Given a question, retrieve detailed and persuasive arguments that answer the question TRECCOVID Given a query on COVID-19, retrieve documents that answer the query STS* Retrieve semantically similar text. BUCC/Tatoeba Retrieve parallel sentences. SummEval Given a news summary, retrieve other semantically similar summaries We acknowledge that there are also several challenges associated with the large size of these models and provide a discussion in Appendix A.Encoder-only models are typically pre-trained by masking 15% of the tokens in the input sequence(Devlin et al., 2019).We only directly compare the performance after the first two steps of LLM2Vec to Echo embeddings as applying SimCSE involves learning sequence representation, which makes the comparison unfair.Full evaluation on MTEB takes more than 40h for Mistral-7B on 8x A100 GPUs.Since the MTEB's SummEval category includes only one dataset, we skip this category in our small-scale evaluation. 


Decision: This paper introduces LLM2Vec, an unsupervised method to transform decoder-only LLMs into powerful text encoders. LLM2Vec employs three core steps: enabling bidirectional attention, predicting masked next tokens, and applying contrastive learning without labels. The authors demonstrate this method's effectiveness on various word-level and sequence-level tasks.

Pros:
- The method is easy to understand, and the paper is well-structured and readable.
- The authors provide substantial empirical evidence showcasing the model's superior performance.

Cons:
- Reviewers primarily criticized the paper's novelty. It combines well-known techniques—bidirectional attention, masked next token prediction (MNTP), and unsupervised contrastive learning.

I personally believe that integrating established techniques in a new framework is an excellent approach to creating a simple yet effective method.

Reviews: review: paper_summary: paper_summary: This paper introduces a method to transform any decoder-only LLM into a bidirectional encoder.  The paper evaluates the method on three different LLMs and results show the effectiveness of the method.
strength_weakness: strength_weakness: However, there are some disadvantages in this paper.

1. Authors claim that the encoder building by decoder-only LLM can generate rich contextualized representations for NLP tasks. Although experiments compare the encoder with LLMs, LLMs may perform better with other prompts. It is better to compare models with other prompts.

2. The method is described clearly in this paper while the method is similar to MLM from BERT and other models. Authors should give some ablations to show the effectiveness of MNTP.

3. As discussed above, LLMs can label input sequences by generating sequences of labels. Author should make it clear that LLMs labels input sequences by generating or directly labeling tokens.
paper_summary: paper_summary: This paper introduces an unsupervised approach to convert decoder-only LLMs into robust text encoders. This approach, named LLM2Vec, is structured around three main steps: enabling bidirectional attention, masked next token prediction (MNTP), and unsupervised contrastive learning. The authors claim that LLM2Vec outperforms existing encoder-only models on various NLP tasks and establishes new benchmarks on the Massive Text Embeddings Benchmark (MTEB) using publicly available data.
strength_weakness: strength_weakness: 1. **Innovation and Originality**: While the paper skillfully integrates several established techniques—bidirectional attention, masked next token prediction (MNTP), and unsupervised contrastive learning—into the LLM2Vec framework, it does not introduce fundamentally new methodologies. The core contribution lies in the novel application and combination of these existing methods rather than in the development of new techniques. This approach might raise questions about the paper's novelty within the field, particularly in a research landscape that highly values groundbreaking innovations. A more detailed discussion on the unique configurations or optimizations made to these existing techniques within LLM2Vec could enhance the perceived novelty of the work.

2. **Fairness in Comparative Analysis**: The paper implements a transformation from unidirectional to bidirectional attention in decoder-only LLMs to enhance text embedding capabilities. However, the comparative analysis appears limited as the baseline configurations in the unidirectional setting (labeled as Uni+Mean) do not include enhancements comparable to those applied in the bidirectional setting (Bi+MNTP+SimCSE+Mean). A more equitable comparison would involve aligning the enhancements across both settings, for instance, comparing Uni+MNTP+SimCSE+Mean with Bi+MNTP+SimCSE+Mean. This would provide a clearer assessment of the impact of changing the attention mechanism independently of other variables.
questions: questions: How does LLM2Vec impact the interpretability of the embeddings? Given that the model transformations introduce changes in how tokens are encoded, does this affect the ability to trace back and understand decisions made by the model?
paper_summary: paper_summary: The article presents LLM2Vec, a novel technique to transform large, decoder-only language models into efficient text encoders without the need for expensive adaptation or the creation of synthetic GPT-4 data.      
This approach consists of three simple steps: enabling bidirectional attention, masked next token prediction (MNTP), and unsupervised contrastive learning.      
The authors evaluate the effectiveness of LLM2Vec on various English-language word- and sequence-level tasks.
strength_weakness: strength_weakness: **Language Limitation**: The evaluation is performed solely on English datasets, which does not guarantee that the method will generalize across other languages.
questions: questions: It would be beneficial if future work could address the transferability of the LLM2Vec method to models trained in other languages. This would strengthen the generalizability of the approach.
paper_summary: paper_summary: This paper studies how to adapt LLMs to embedding models. The proposed method converts an pre-trained auto-regressive language models by changing causal attention into bidirectional attention, and then finetune the model with masked next token prediction and unsupervised contrastive learning. Experimental results show that the proposed method achieves strong performance under both unsupervised and supervised settings.
strength_weakness: strength_weakness: 1. It can hardly be considered a secret that LLM can be strong embedding models, as previous works like E5-mistral-instruct have already demonstrated similar findings.
2. Under supervised setting, the proposed bidirectional attention and unsupervised training provides little improvements over the replicated E5-mistral-instruct model.

Comments: comment: I have read the response and review comments from others and updated my score.
I have read the response as well as review comments from others and will keep my score unchanged.