LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders
Authors: Parishad BehnamGhader
Vaibhav Adlakha
Marius Mosbach
Dzmitry Bahdanau
Nicolas Chapados
Siva Reddy
Venue: COLM (2024)
Keywords: text embeddings, large language models, adaptation
TL;DR: We provide LLM2Vec, a simple recipe to transform decoder-only LLMs into powerful text embedding models.

Abstract: Large decoder-only language models (LLMs) are the state-of-the-art models on most of today's NLP tasks and benchmarks. Yet, the community is only slowly adopting these models for text embedding tasks, which require rich contextualized representations. In this work, we introduce LLM2Vec, a simple unsupervised approach that can transform any decoder-only LLM into a strong text encoder. LLM2Vec consists of three simple steps: 1) enabling bidirectional attention, 2) masked next token prediction, and 3) unsupervised contrastive learning. We demonstrate the effectiveness of LLM2Vec by applying it to 4 popular LLMs ranging from 1.3B to 8B parameters and evaluate the transformed models on English word- and sequence-level tasks. We outperform encoder-only models by a large margin on word-level tasks and reach a new unsupervised state-of-the-art performance on the Massive Text Embeddings Benchmark (MTEB). Moreover, when combining LLM2Vec with supervised contrastive learning, we achieve state-of-the-art performance on MTEB among models that train only on publicly available data (as of May 24, 2024). Our strong empirical results and extensive analysis demonstrate that LLMs can be effectively transformed into universal text encoders in a parameter-efficient manner without the need for expensive adaptation or synthetic GPT-4 generated data.

Published as a conference paper at COLM 2024
LeM2 Large Language Models Are Secretly Powerful Text Encoders
Parishad BehnamGhader
Vaibhav Adlakha*st Marius Mosbach
Dzmitry Bahdanau'
Nicolas Chapados'
Siva Reddy
McGGill University, Mila *ServiceNow Research Faceboook CIFAR AI Chair
(parishad. behnaghader, vaibhav. aalphaa..dllea akha, marius mosbach massbach@mmi @mi la. quebec
Abstract
Large decoder-only language models (LLMs) are the state-of-the-art models on most of today's 's NLP tasks and benchmarks. Yet, the community is only slowly adopting these models for text embedding tasks, which require rich contextualized representations. In this work, we introduce LLM2Vec, sim- ple unsupervised approach that can transform any decoder-only LLM into strong text encoder. LLM2 consists of three simple steps: 1) enabling bidirectional attention, 2) masked next token prediction, and 3) unsuper- vised contrastive learning. We demonstrate the effectiveness of LLM2 by applying it to 4 popular LLMs ranging from 1.3B to 8B parameters and evaluate the transformed models on English word- and sequence-level tasks. We outperform encoder-only models by large margin on word-level tasks and reach new unsupervised state-of-the-ar performance the Massive Text Embeddings Benchmark (MTEB). Moreover, when combining LLM2Vec with supervised contrastive learning, we achieve state-of-the-an performance MTEB among models that train only on publicly available data (as of May 24, 2024). Our strong empirical results and extensive analy- sis demonstrate that LLMs can be effectively transformed into universal text encoders in a parameter-efficient manner without the need for expensive adaptation or synthetic GPT-4 generated data.
Introduction
Text embedding models aim to encode the semantic content of natural language text in vector representations which then facilitate various natural language processing (NLP) tasks, such semantic textual similarity, information retrieval, and clustering. For many years, the dominating paradigm for building such models relied on pre-trained bidirectional encoders encoder-decoders such BERT (Devlin et al., 2019) and T5 (Raffel al. 2020), which are typically adapted for text embedding tasks by following a multi-step training pipeline consisting of weakly- and fully-supervised contrastive training (Ni et al. 2022 Li
et al.) 2023a: Xiao et al. 2023) infer alia). Only recently, the community started to adopt
decoder-only LLMs for embedding text (Muennighoff. 2022; Ma et al. 2023 Wang et al.,
2023 Springer et al. 2024 & 2024).
We speculate that the slow adoption of decoder-only LLMs for text embedding tasks is partly due to their causal attention mechanism, which inherently limits their ability to produce rich contextualized representations. At any given layer, causal attention limits token interactions, ensuring that the representation of token at position is influenced solely by the representations of preceding tokens at positions 0, 0, - 1. Although this limitation is necessary for generative capabilities, it sub-optimal for text embeddings as it prevents the representations from capturing information across the entire input sequence.
Overcoming this architectural limitation of decoder-only LLMs for text embedding tasks is highly appealing as these models come with several advantages compared to their encoder-
"qqull contribution.

---

Published as a conference paper at COLM 2024
Enabling Bidirectional Attention
Masked Next Token Prediction
Unsupervised Contrastive Learning x(, C ,(2,2.) + (E
C
P(wa 111, w2, [M] ws)
wyg
6
003
MA
[MASK]
Figure 1: The 3 steps of LLM2Vec. First, enable bidirectional attention to overcome the restrictions of causal attention (Bi). Second, we adapt the model to use bidirectional attention by masked next token prediction training (MNTP). Third, apply unsupervised contrastive learning with mean pooling to learn better sequence representations (SimCSE).
only counterparts ! During pre-training, decoder-only LLMs learn from all input tokens and not just small percentage whichh-given the same amount of training data data-makes them much more sample-efficient than encoder-only models (Clark et al. 2020). Moreover, there exists rich ecosystem around these models, with extensive tooling and well tested pre-training recipes, which has resulted in continuous improvement these models by the community. Lastly, recent work on instruction fine-tuning and learning from human preferences has resulted in decoder-only LLMs that excel at instruction following (Wang) et al., 2022b Ouyang et al. 2022), making them an ideal choice for building universal text embedding models that generalize across a large variety of tasks using instructions.
In this work, we provide a simple unsupervised approach, termed LLM2Vec, which can be used to transform any pre-trained decoder-only LLM into (universal) text encoder. As shown in Figure LLM2Vec consists three simple steps: 1) enabling bidirectional atten- tion, 2) masked next token prediction, and 3) unsupervised contrastive learning. Crucially, LLM2Vec does not require any labeled data and is highly data- and parameter-efficient.
We apply LLM2vec to 4 decoder-only LLMs ranging from 1.3B to 8B parameters (S-LLaMA-1.3B, LLaMA-2-7B, Mistral-7B, Meta-LLaMA-3-8B) and evaluate the resulting models word- and sequence-level tasks. On word-level tasks (chunking, named-entity recognition, and part-of-speech tagging), encoder-only models by large margin, demonstrating its effectiveness for producing rich contextualized token representations. On the Massive Text Embeddings Benchmark (MTEB), ILM2Vec-transformel models set new state-of-the-art for unsupervised models, with our best model reaching a score of 56.8. Additionally, we combine LLM2Vec with supervised contrastive training and achieve new state-of-the-art performance among models that train only on publicly available data. Beyond our strong empirical results, we provide an extensive analysis of how LLM2Vec affects the representations of the underlying model and reveal an intriguing property of Mistral-7B, which can handle bidirectional attention without any fine-tuning.
models outperform strong
Overall, our work demonstrates that decoder-only LLMs are indeed capable of producing universal text embedding and only very little adaptation is required to reveal this ability. Our code and pre-trained models is publicly available at https: //tgt//hgttttttt// cot.a//ww...o//ttill--MMM. 11m2vec
We acknowledge that there also several challenges associated with the large size of these models and provide discussion Appendix Fncoder-only models are typically pre-trained by masking 15% of the tokens in the input sequence Devlin et al. 2019).
2

---

Published as a conference paper at COLM 2024
2 LLM2Vec
2.1 Three simple ingredients
Enabling bidirectional attention The first step of the LLM2Vec approach is to replace the causal attention mask of decoder-only LLMs by an all-ones matrix (see Appendix [.1 for background on the self-attention). This gives each token access to every other token in the sequence, converting it into a bidirectional LLM (Devlin et al. 2019 Liu et al. 2019). However, it is not a priori clear, why this should lead to better sequence representations. After all, the decoder-only LLM was not trained attend to future tokens and therefore, this naive approach might even lead to worse representations. As we show, simply enabling bidirectional attention does indeed decrease in embedding performance for most models. We can however easily adapt model to make use of its bidirectional attention.
Masked next token prediction We use a simple strategy to make the model aware of its bidirectional attention by adapting it via masked next token prediction (MNTP). MNTP is training objective that combines next token prediction with masked language modeling (Lv al. 2023). Given an arbitrary sequence x (I1, x2, XN) as input, we first mask traction of the input tokens and then train the model to predict the masked tokens based on the past and future context. Crucially, when predicting a masked token at position i, we compute the loss based on the logits obtained from the token representation at the previous position - 1, not the masked position itself (see Figure
Unsupervised contrastive learning While the previous two steps of the LLM2Vec recipe can transform any decoder-only LLM into an encoder for word-level tasks, they might not be sufficient for sequence representations Unlike bidirectional encoders that include a next sentence prediction objective in their pre-training objectives (Devlin et al. 2019), decoder- only LLMs are not explicitly trained to capture the context of the entire sequence. To fill this gap, apply unsupervised contrastive learning via SimCSE [Gao et al.) 2021). Specifically, given an input sentence, is passed through the model twice with independently sampled dropout masks, resulting in two different representations for the same sentence. The model is trained to maximize the similarity between these two representations while minimizing the similarity with representations of other sentences in the batch. Crucially, this step does not require any sentence pair data and can be applied using any collection of sentences. We use a pooling operation on the word representations to get the sentence representation (more details in Section 3.2)
2.2 Transforming decoder-only LLMs with LLM2Vec
Models For most of our results, we experiment with 3 different decoder-only LLMs ranging from 1.3B to 7B parameters: Sheared-LIaMA1
(S-LLaMA-1. 3B, Xia et al., 2023), Llama-2-7B-chat (LLaMA-2-7E) (1LaM-2--7B_Touvvo Touvron et al.) 2023), , and Mistral-7B-Insttrut--0]] Jiaan
(Mistral-7B, Jiang et al. 2023a). In Tables 1 and we provide additional results for the recently released
model (Meta-LLaMA-3-8B) - Al@Meta, 2024).
Training data We perform both the MNTP and the unsupervised SimCSE step using data from English Wikipedia. We select data from Wikipedia as presumably included in the pre-training mixture of all the models we experiment with. It is therefore fair to assume that these two adaptation steps are not teaching the model any new knowledge beyond how to attend to future tokens and how to construct sequence representations. Specifically, use the Wikitext- dataset (Merity et al. 2017) for the MNTP step and subset of Wikipedia sentences released by Gao et al. 2021 for the unsupervised SimCSE step.
Masked next token prediction We follow established practice from the masked language modeling literature and randomly mask a fraction of the tokens from the input sequence (Devlin et al.) 2019 Liu et al. 2019). We use the underscore (-) the mask token, since the models we experiment with do not have a special token for masking. We fine-tune the model using LoRA (Hu et al. 2022) to predict the masked token using the representation
3

---

Published as a conference paper at COLM 2024
100.0
100.0
1000
Bi- MXTP
UN
Bi MRTP
B1+ KWTP
05.0
Ri- MXTP STiCEE
Bi NNTP SS
57.5
Bi WKTP SWACF
97.5
Recuracy
Accursty
05.0
05.0
82.5
05.04
90.5
90.0
SLLLMA-338
LLMM--78
S:LLM6-13
(a) Chunking
(b) NER
(c) POS
Figure 2: Evaluation of JLLM2Ve--ranssorme models on word-level tasks. Solid and dashed horizontal lines show the performance of Uni and AaBERTa-v3-larg respectively.
of the previous token to maximally align our training objective with the pre-training setup of decoder-only LLMs. For all models, we trained for 1000 steps with batch size of 32 on a single 80GB A100 GPU. For 7B and 8B models, this training takes only 100 minutes. We provide additional details of our training setup and hyperparameters in Appendix D.1.1
Unsupervised contrastive learning For the contrastive training, apply the unsuper- vised SimCSE approach by Gao et al. (2021). The positive examples are constructed by applying LLM's dropout twice on the same input sequence, whereas the other sequences in the batch act as in-batch negatives. We merge the MNTP LoRA weights into the base model and initialize new LoRA parameters before starting the SimCSE training, which ensures that the models retains the knowledge learned in the previous step. Similar to the MNTP step, train for 1000 steps. For 7B and 8B models, this training takes 3 hours single 80GB A100 GPU with a batch size of 128, We provide additional details of our training setup and hyperparameters in Appendix D.1.2
3
LLM2Vec-transformed models are strong unsupervised text embedders
3.1 Evaluation word-level tasks
We start by evaluating on word-level tasks to demonstrate that LLM2Vec is successful at improving the contextual representations constructed by decoder-only LLMss.
Setup We evaluate three word-level tasks: chunking, named-entity recognition (NER), and part-of-speech tagging (POS), using the CONLLL-2003 benchmark (Tjong Kim Sang & De Meulder 2003). We embed each input sentence and train a task-specific linear classifier top of the frozen representations. This is akin to the linear probing setup commonly used in the language model analysis literature (Belinkov) 2022). We compare the LLM2Vec- transformed models to DeBERTa-V3-11 arge (He et al. 2023), the current sate-ooftheeart encoder-only model. Additional details about our setup are provided in Appendix D.1.3
Results Figure shows the results of our evaluation (a detailed breakdown of the results provided in Table 4 On each of the three tasks, constructing token representations with causal attention (Uni) already outperforms the encoder-only baseline. This is not surprising, given that the models we experiment with are significantly larger and have been pre- trained on more data. As expected, naively applying bidirectional attention dramatically most cases. Interestingly, for Mistral-7B, enabling bidirectional attention hurts performance much less compared to S-LLaMA-1 -1.B and LLaMA-2-7, For NER, Mistral's performance even improves by 0.6% with bidirectional connections.
hurts performance
Focusing on the LLM2Vec--ranssfrmee models, we observe that for all models and tasks, adapting via MNTP improves performance. For instance, in the chunking task, we see improvements for S-LLaMA-2 3B (by 5%), LLaMA-2-7B (by 4%), and Mistral-7B (by 4%). Combining MNTP with SimCSE, however, performs worse than just applying MNTP. This is expected for word-level tasks, as SimCSE adapts the representations for sequence-level tasks.
4

---

Published as a conference paper at COLM 2024
HUTP MULLP StnLS
MIEER waree (atben)
MTEB score (waeee))
MTEE: score (watee))
10
(0
60
FO5
FOS
FO5
(a) 100--11 S-LLaMA-1.33
(b) Llama-2-7B
(c) Mistral-7B
Figure 3: Unsupervised results on our 15 task subset of the MTEB dataset. We ablate three different pooling choices: EOS, mean pooling, and weighted mean pooling. LLM2Vec is compatible with all three approaches and works best with mean pooling.
3.2 Evaluation on sequence-level tasks
Next, we evaluate on the Massive Text Embedding Benchmark (MTEB), collection of
diverse embedding task categories covering total of 56 datasets (Muennighoff et al. (023).
To select the best-performing pooling method for each method, we perform ablations on
a 15 task subset consisting of representative tasks from each of the MTEB categories. We provide additional details and justification for how we chose this subset in Appendix C.1
Setup Following previous work (Su et al., 2023: Wang et al. 2023 Springer et al. 2024), we evaluate with task-specific instructions. For a fair comparison, we use the same set of instructions as Wang al. (2023) which are also used by Springer al. (2024). The instructions are only added to queries and can be found in Table Appendix C.2 For symmetric tasks, the same instruction will be used for the query and the document. When applying (weighted) mean pooling [Muennighoff 2022) we exclude the instruction tokens.
As baseline, we compare to the unsupervised BERT models obtained from Gao et al. (2021). Additionally, we compare to Echo embeddings, concurrent approach by Springer et al. (2024), which we run with the same models and instructions (see Appendix F.1 for more details our implementation of Echo embeddings). Echo duplicates the input and takes the pooling over the second occurrence to address the limitation of causal information flow.
Results our 15 task subset of MTEB Figure 3 shows the impact of various pooling methods for all three models on the subset of MTEB tasks. We can clearly observe that applying causal attention sub-optimal when constructing text embeddings. The dominant paradigm of applying the EOS pooling for models with causal attention is outperformed by (weighted) mean pooling. Enabling bidirectional attention without any training harms performance for S S-LLaMA-1 3B and LLaMA-2-7B. Similar to our word-level results, the per- formance of Mistral-7B improves with bidirectional attention, even without any training.
For LLM models, applying MNTP training improves the performance of S-LLaMA-1. 3B and Kistral-7B. Moreover, applying SimCSE further boosts the performance of S-LLMMA111 1.3,, LLaMA-2-7B, and Mistral-7B by 49.8%, 23.2%, and 37.5% compared to the best causal baseline on the MTEB subset. We further conduct an ablation each component of LLM2Vec recipe in Appendix D.2.2 (Tb
Results on full MTEB Table I shows the results of the best performing models, which we select based on the ablation above, on the full MTEB dataset. After the first two steps of LLM2Vec sit-Veec-bdiirettonn attention and MNTP-wo we observe considerable improvement in performance for all four models (e.g., 16.4% improvement for Mistral-7B).
When comparing to Echo embeddings, LLM2Vec (the first two steps only leads im- proved performance for S-LLaMA-1 3B, LLaMA-2-77 and Meta-LLaMAA3-B88 and performs almost on par for Mistral-P7 However, compared Echo embeddings, LLM2 Vec much
We only directly compare the performance after the first two steps of M.M. to Echo embeddings applying SimCSE involves learning sequence representation, which makes the comparison unfair.

---

Published as a conference paper at COLM 2024
Categories
Retr. 15
Rerank. 4
Clust. 11
PairClass.
Class. 12
STS 10
Summ.
Avg
of datasets
56
ancoder-only
BERT
10.59
43.44
30.12
56.33
61.66
54.36
29.82
38.33
BERT SimCSE
20.29
46.47
29.04
70.33
62.50
74.33
31.15
45.45
8 B-LLaMA-1.33
Uni + w. Mean
9.47
38
28.02
42.19
59.79
24.98
35.05
LLM2Vec (w/o SimCSE)
15.48
40.99
31.83
50.63
64.54
62.06
26.82
41.43
LLM2Vec
25.93
47.70
37.45
72.21
71.61
49
Echo
10.36
41.52
30.03
52.08
63.75
59.36
22.79
39.10
LLaMA-2-7B
Ui + w. Mean
15.16
46.94
36.85
61.41
69.05
63.42
26.64
44.54
LLM2Vec (w/o SimCSE)
19.86
44.74
35.31
61.60
67.99
66.74
26.83
45.70
LLM2Vec
36.75
52.95
40.83
77.89
71.57
76.41
31.38
55.36
Echo
16.16
46.84
34.25
63.54
69.82
67.95
25.57
45.36
Mistral-7B
Uni + w. Mean
10.43
45.11
35.82
60.28
71.14
58.59
26.57
42.46
Bi + Mean
15.84
47.40
35.55
66.53
72.18
71.04
29.93
46.86
LLM2Vec (w/o SimCSE)
19.74
50.43
40.06
70.95
72.51
71.90
27.84
49.43
LLM2Vec
38.05
53.99
40.63
80.94
74.07
78.50
30.19
56.80
Echo
22.68
51.07
36.78
75.87
72.69
73.60
29.54
50.26
Met.a LLaMA 3-88B
Uni Mean
15.17
46.22
36.84
60.94
67.41
62.80
25.51
43.98
Bi Mean
3.90
34.56
14.27
42.71
57.89
51.15
23.26
30.56
LLM2Vec (w/o SimCSE)
24.75
49.20
39.74
65.91
69.00
67.85
25.59
48.84
LLM2Vec
39.19
53.09
41.99
78.01
71.88
75.86
31.45
56.23
Echo
12.58
49.79
36.32
68.95
70.22
67.43
26.44
45.32
Table 1: Unsupervised results on MTEB. We compare S-LLaMA-1.3B, LLaMA-2-7B. Mistral-7B, and Meta-LLaMA-3-4 with and without LLM2Vec to the unsupervised BERT models of Gao et al. 2021) as well as Echo embeddings (Springer et al. 2024).
more efficient as Echo embeddings repeat the input and therefore double the sequence length which makes inference considerably slower (we provide runtime comparison in Appendix E.2). Adding the final step of the LLM2Vec recipe-une unpppeuuuueee SimCSE further boosts all three models by large margin, making our LLM2Vec Mistral-78 SOTA among all unsupervised models with the score of 56.80.
Interestingly, Meta-LLaMA-3-2 with LLM2Vec (w/o SimCSE) outperforms echo embed- dings by a larger margin compared to the other models. Adding SimCSE again boosts performance, but does outperform LLM2Vec applied to Mistral-7B.
Overall, our results highlight that LLM2 Vec is successful at transforming decoder-only LLMs into strong text embedding models which outperform previous unsupervised approaches on the challenging MTEB leaderboard.
How does LLM2Vec affect a model?
4.1 LLM2Vec helps models to capture information from future tokens
To analyze the extent to which LLMMVee-transformed models incorporate information from future tokens, we adopt the analysis Sfringee et al. (2024) and test how well the model performs judging the similarity between sentences that share the same prefix.
Setup We on a synthetic dataset collected by Springer et al. (0024), which consists of 35 sentence triples {q,, sit s7 ) with 9; = (A, B,), s. = (A,, G), and s; = (A, D,), where B, and C have similar meaning but B, and D, don't. We compute sequence

---

Published as a conference paper at COLM 2024
tarig)
B MXTP
@ jac traning)
Bi- MNTP
Sinis, 4
Siils
Sinls Sinje
Sinte,
Senjp,
Sinte
10A1
605
Sivibaty)
(ooise Sayloty
(locee sindurtt)
[cokiee sealorty
(a) S-LLLMM-1...
(b) Mistral-7B
Figure 4: Cosine similarity between query (g) and negative (s ) as well as positive examples (sT). Plots for LLaMA-2 2-BB and other approaches are shown in Appendix
L Layor
Layer
Layer
Coane sindlaryy
similrity
Crsine similrity
0.75
0.75
0.75
0.50
0.50
0.50
0.055
0.95
3.00
0.00
0.00
50
75
126
130
175
25
125 Token position
150
175
125
Taken position
Token position
(a) S-LLaMA-1.3. 3B
(b) Llama- 2-BB
(c) Mistral-7B
Figure 5: Cosine similarities at different token positions at layers when comparing represen- tations constructed with causal attention to those constructed with bidirectional attention (without training). Additional plots are shown in Appendix
representation for each of these sentences by pooling only over the first part of the sentence, ie., A₁ We then compute the cosine similarity between the resulting embeddings. model that incorporates information from future tokens (B, Ci, D;) in the representations of the prefix A, should assign a higher similarity to the positive example.
Results Figure 4 shows the results of our analysis for S-LLaMA- and Mistral-7B. Results for LLaMA-2-7B, which show same trends, and comparison to Echo embeddings are provided in Appendix F. For S-LLaMA-1. we observe that enabling bidirectional attention and training with the MNTP objective are sufficient to establish clear separation between the positive and negative examples. For Mistral-7BB all setups lead to a larger cosine similarity between the query and positive than the query and negative examples.
4.2
Why does bidirectional attention without training work for Mistral models?
Our empirical results so far as well as the analysis above share an intriguing observation: enabling bidirectional attention works well for Mistral-7B, even without any training. Below, we investigate this surprising behavior by analyzing how bidirectional attention impacts the representations of model.
Setup We feed a single input sequence (a random paragraph from Wikipedia) to each model and compute the hidden representations of every token every layer I with causal (Hf) and bidirectional attention (H For every layer, we compute the cosine similar- ity between the representations constructed using causal and bidirectional attention, i.e., sim H For most layers, we expect this similarity be low, as enabling bidirectional attention without any training should lead to substantially different representations.
Results Figure 5 shows that as expected, for S-LLaMA-1 3B and LLaMA-2-74 enabling bidirectional attention without training has profound impact the representations, leading to low cosine similarity across almost all layers and token positions. For Mistral-7B, the other hand, the representations have very high cosine similarity throughout.
7

---

Published as a conference paper at COLM 2024
Categories of datasets +
Retr. 15
Rerank. 4
Clust. 11
PairClass. 3
Class. 12
STS 10
Summ. 1
Avg 56
Previous work w/ public data only
Instructor-x
49.26
57.29
44.74
86.62
73.12
83.06
32.32
61.79
BGF IGngegge----1115
54.29
60.03
46.08
87.12
75.97
83.11
31.61
64.23
GritL-MMistta/7--- + public
53.10
61.30
48.90
86.90
77.00
82.80
29.40
64.70
Mistrll--7---1 + public data
52.78
60.38
47.78
88.47
76.80
83.77
31.90
64.56
EchoMistta-77-v11
55.52
58.14
46.32
87.34
77.43
82.56
30.73
64.68
S-LLaNA-1-38
Uni + w. Mean
51.02
54.65
39
83.57
71.64
82.16
30.05
60.44
MLM2Vec SimCSE) MLM2
51.44
55.38
43.57
86.20
72.21
83.58
30.01
61.85
51.49
55.58
43.24
85.80
72.98
83.62
30.12
61.96
LLaMA-2-7B
Uni w. Mean
54.33
58.01
40.57
87.01
75.60
83.47
29.68
62.96
LLM2Vec (w/o SimCSE)
54.60
57.38
45.24
88.03
76.33
83.73
28.49
64.14
LLM2Vec
54.34
57.70
45.04
87.87
76.53
83.43
28.82
64.04
Mistral-78
Uni Mean
54.81
57.37
41.07
86.05
76.01
83.44
30.74
63.20
LLM2Vec (w/o SimCSE)
55.99
58.42
45.54
87.99
76.63
84.09
29.96
64.80
LLM2Vec
56.U5
58.59
45.12
88,18
76.72
83.69
30.66
64.72
Uni w. Mean
55.42
58.60
43.19
86.29
75.56
83,95
30.59
63.87
MLVe (w/o SimCSE) LLM2Vec
56.63
59.68
46.45
87.80
75.92
83.58
30,94
65.01
56.71
59.02
45.86
87.95
76.67
82.98
29.67
64.90
Table 2: Supervised results full MTEB benchmark. The best performing LLM2Vec model Bita--LaMA-3--8- BB+ + LLM2Vec (w /wo SimCSE) achieves new SOTA performance among models trained only on publicly available data.
Based on these findings (we replicate these results for other inputs and other variants of Mis- tral in Appendix F.) and the strong unsupervised results for Mistral-7B with bidirectional attention, we speculate that Mistral models are pre-trained with some form bidirectional attention, e.g., prefix language modeling (Raffel et al., 2020) least for some parts of its training. We leave more detailed investigation of this intriguing behavior for future work.
5
Combining LLM2Vec with supervised contrastive learning
The final piece of our evaluation combines LLM2Vec with supervised contrastive learning.
5.1
LLM2Vec leads to strong performance on the MTEB leaderboard
Setup For supervised training, we train on a replication of the public portion of the E5 dataset (Wang al., 2023) curated by Springer et al. (2024). The dataset consists of approximately 1.5M samples and we provide details on its compilation in Appendix G.1 We follow standard practice and train the models with contrastive learning using hard negatives and in-batch negatives. We use LoRA fine-tuning for supervised setting as well. The MNIP LoRA weights are merged into the base model, and the trainable LoRA weights are initialized with SimCSE weights. For LLM2Vec models that use just MNTP, the LoRA weights are randomly initialized. The training is performed for 1000 steps with a batch size of 512. We detail other hyperparameters in Appendix G.2
For a fair comparison, we only compare to models trained on publicly available data and provide a comparison to the top entries on the MTEB leaderboard in Appendix G.3
Results Table shows the results of our evaluation. For models, transforming model with LLM2Vee leads to improved performance the strong Uni + weighted mean baseline. As expected, performing unsupervised SimCSE less crucial for supervised training, and even leads to slightly worse performance for LLaMA-2-7B, Mistral-7B. and
8

---

Published as a conference paper at COLM 2024
MTEB (subset)
700
MTEB score (sulose))
70
MTEB score (subsat)
70
611
60'
40-
60
80
30
25
75
30
75
30
Steps
Steps
Steps
(a) S S-LLLMA-1- ...
(b) Llama 1aama-22-77 227B
(c) Mistral-7B
Figure Results on the 15 task subset of MTEB during training of S-LLaMA-1.3B
LLaMA-2-7B. and Mistral-7B. For all three models, applying LLM2Vec before supervised training leads to better performance with less steps.
Meta-LLaMA-3-S0 compared to just performing the MNTP step of LLM2Vec (LLM2Vec w/o SimCSE). However, as we will show in Section 5.2 LLM2Vec with MNTP and SimCSE is much more sample-efficiet, and therefore crucial in compute or data-constrained settings. Notably, our best model, Maa-LLaMA-3-88 LLM2Vec (w/o /o SimCSE) leads to new state-of-the-art performance among models trained only on publicly available data.
5.2 LLM2Vec leads to more sample-efficient training
Setup To demonstrate the sample-eficiency L.M2Vec-transformee models, we save checkpoint every 25 training steps and evaluate them on our 15 task subset of MTEB.
Results As shown in Figure 6 LLM2Vee-transformedd models reach better performance earlier in training. This observation is consistent across all three models. For S-LLaMA-1. the smallest of our three models, even performing just MNTP leads to a considerably improved sample-efficienee These results are particularly encouraging for settings where is hard to acquire high quality labeled data, setting which we leave for future work.
Related Work
Supervised text encoders Initially, supervised methods primarily relied on tasks such as natural language inference or sentence similarity train BERT-like models for producing sentence embeddings (Conneau et al., 2017: Reimers & Gurevych) 2019). Subsequently, BERT-like models have also been adapted to tasks like retrieval (Karpukhin et al., 2020
Khattab & Zaharia 2020). More recent methods have further improved these representa- tions through a complex multi-stage learning pipeline that consists of large-scale weakly supervised contrastive training followed by multi-task fine-tuning (Ni et 2002 Wang et al., 2022a Li et al., 2023a Xiao et al. 2023) Recent approaches have focused on enhancing the generalization and transterability of text embeddings using instructions (Su al. 2023
Asai et al. 2023).
Unsupervised text encoders Another line of work has explored training text embedders in an unsupervised manner using only set of unordered sentences. These unsupervised approaches typically create two different representations of the same sentence for contrastive learning. The methods vary in how they form these representations perturbing the input sentence (Wu et al. 2020), or using different model instances (Carlsson et al., 2021). ). Sim (Gao et al. | 2021), the approach used in this work, generates two representations of the same sentence by passing it through the model twice with different dropout masks.
Turning decoder-only LLMs into text encoders While decoder-only LLMs have out- performed bidirectional encoders across large variety of language understanding tasks (Brown et al., 2020 Touvron et al. 2023 Jiang et al. 2023a inter alia), their impact on sentence representation learning remains limited. The most common approaches in literature the final hidden state of the last token as the sentence embedding (Neelakantan et al. 2022 Ma
et al. 2023
Wang et al. 2023).

---

Published as a conference paper at COLM 2024
There are few works that explore the limitations of using causal attention mask when adapting decoder-only LLMs for text classification and sentence representation tasks. et al. (2023b) experiment with removing the causal mask of Llama-2 during supervised fine-tuning for text classification and NER tasks. Similarly, Dukić & Snajder (2024) enable bidirectional attention for group of layers during supervised fine-tuning on NER and chunking. In the context of sentence representation learning, [i Li (2024) explore enabling bidirectional
attention in the last layer of a decoder-only model during supervised contrastive fine-tuning
on STS tasks.
Concurrent to our work, several works have focused on converting decoder-only-LLMMM to text encoders in supervised and unsupervised manner. Jiang et al. (2023b) and Lei et al. (0224) prompt the language model to summarize the input text in one word, and take the last layer's s hidden embedding for the last token as the text's representation. Muennighoff et al. (2024) perform multi-task full fine-tuning using combination of self-supervised language modeling with causal attention and supervised contrastive learning with bidirectional attention. In contrast, our proposed approach is much more computationally efficient, as requires only parameter-efficient fine-tuning and 1000 gradient steps. Closest to our work is the concurrent work ofSrnnne et al. (0024). They propose to copy the input sequence and append it to itself, which addresses the contextualization issue causal attention as tokens in the copy of the input can now attend to "future" tokens in the previous sequence. While this performs well in practice, it significantly increases the computational cost at inference time, which can be particularly problematic for encoding longer documents. Our approach outperforms Springer et al. (2024), without inducing any additional computational overhead at inference time.
7 Conclusion
We present LLM2Vec, strong unsupervised approach transform any decoder-only LLMs into a (universal) text embedder. We perform an extensive evaluation on word- and sequence-level tasks and demonstrate the effectiveness of LLM2 in both unsupervised and supervised settings. Applying LLM2Vec to Mistral- achieves a new state-of-the-art performance on MTEB among unsupervised approaches. When combining LLM2Vec with supervised contrastive fine-tuning, Meta- achieves SOTA performance among approaches that train only on publicly available data (as of May 24, 2024). Beyond our strong empirical contributions, we provide extensive analysis of how LLM2Vec impacts the underlying model and reveal an intriguing property of Mistral-7B, which explains its strong out of the box performance with bidirectional attention. The simplicity of our approach, as well as its compute and sample-efficiency makes LLM2vec a promising solu- tion for low-resource and compute constrained scenarios and opens up several interesting avenues for future work.
Acknowledgements
We thank the members of SR's research group for providing feedback throughout the project. Furthermore, we thank Jacob Mitchell Springer for providing the supervised training data used in Springer et al. (2024). PB is supported by the Mila-Intel Grant program. MM is partly funded by the Mila P2v5 Technology Maturation Grant and the Mila-Samsung grant. SR is supported by a Facebook CIFAR AI Chair and NSERC Discovery Grant program.
References
Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. SemEval-2014 task 10: Multilingual semantic textual similarity. In Preslav Nakov and Torsten Zesch (eds.), Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pp. 81-91, Dublin, Ireland, August 2014. Association for Computational Linguistics. doi: 1111155/11114201111 httpsttss:///cclla 00t1////oooooor/1111110000
10

---

Published as a conference paper at COLM 2024
Al@Meta. Llama 3 model card. 2024. URL httpgs///ggtuubbcco///aaaaaaaaaa
blob/main/MDDELL MORDEOAAEDD......AEEEE mdd
Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard, Sebastian Riedel, Hannaneh Hajishirzi, and Wen-tau Yih. Task-aware retrieval with instructions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, Pp. 3650-3675 Toronto, Canada, July 2023. Asso- ciation for Computational Linguistics. doi: 10. 10.18653 wt////2...nddiiiiieee findings--cl...22 acl. URL atttgs://aclaathhooog...orr/
findings-annn acl. .2..252
Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Com- putational Linguistics, 48(1):207-219, March 2022. doi: 10.1162/coli.a.000 URL
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss: Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Informationn Processing Systems, volume 33, pp- 1877-1901 Curran Associates, Inc., 2020. URL URL/Bstps:/ppoceeeiigg..uuu wttps/cceaaplpeeeeeeaaeeaaaeeeee
Fredrik Carlsson, Amaru Cuba Gyllensten, Evangelia Gogoulou, Erik Ylipää Hellqvist, and Magnus Sahlgren. Semantic re-tuning with contrastive tension. In International Conference on Learning Represenlations, 2021. htppttt/////oeeeeeeeee wttp::/wwe//eotmmmmmm Ov_sMNau-PF
Jianpeng Cheng, Li Dong, and Mirella Lapata. Long short-term memory-network, for machine reading. In Jian Su, Kevin Duh, and Xavier Carreras (eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 551-561, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 2211653/v1/D116111 URL
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Hen- ryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiri- donov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Bren- nan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellsterm, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. PaLM: Scaling language mod- eling with pathways. Journal of Machine Learning Research, 24(240):1 2023. URL http://jmlr. pttpppapess//2//2222222222 1144. html
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. Electra: Pre- training text encoders discriminators rather than generators. In International Conference on Learning Representations, 2020. URL /ttpp:://peerree
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. Su- pervised learning of universal sentence representations from natural language infer- ence data. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.), Proceedings of the 2017 Conference Empirical Methods in Natural Language Processing, Pp. 670-680, Copenhagen, Denmark, September 2017, Association for Computational Linguistics. doi: wtll//ttppe///ccaaattooooooooggp
11

---

Published as a conference paper at COLM 2024
Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning, In The Twelfth International Conference on Learning Representations, 2024. URL https
hilfialkaff DataCanary, Jiang Lili, Risdal Meg, Dandekar Nikhil, and tomtung. Quora ques- tion pairs. 2017. URL https::/kaggle..... com/ http:///wgeeeeeccccc/epeeetttttt wttp///ppeeiitoiuuoooooooooootet wotpa///touoouuusooooooooaooooae question-paiiee
Jacob Devlin, Ming- Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics; Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10.18653/ 11/111-1NN1111 URLhttps: htattpp://accanthoolog.. 1423
David Dukić and Jan Snajder. Looking right is sometimes right: Investigating the capabilities of decoder-only Ilms for sequence labeling. arXivo preprint, 2024. URL https: //axxiv org/abs/2401. 14556
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. ELI5: Long form question answering. In Anna Korhonen, David Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3558-3567, Florence, Italy, July 2019. Association for Computational Lin- guistics. doi: 10.18653/ 111/53//w1P1911344
wtt/ 1346
Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 6894-6910, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/ w1/2021..22ml// emmllp-miin.552. URL https: httpt:::s//alaaahooooooooo .rg.ggg...o..ooooooo............ 2021. emnlp maap-mmain... 552
Pengcheng He, Jianfeng Gao, and Weizhu Chen. DeBERTav3: Improving deBERTa us- ing ELECTRA-style pre-training with disettandd embedding sharing. In The Eleventh International Conference on Learning Representations, 2023. URL https wtt.:n/to/oooommst//. XhLxHA
Wei He, Kai Liu, Jing Liu, Yajuan Lyu, Shiqi Zhao, Xinyan Xiao, Yuan Liu, Yizhong Wang, Hua Wu, Qiaoqiao She, Xuan Liu, Tian Wu, and Haifeng Wang. DuReader: a Chinese machine reading comprehension dataset from real-world applications. In Eunsol Choi, Minjoon Seo, Danqi Chen, Robin Jia, and Jonathan Berant (eds.), Proceedings of the Workshop on Machine Reading for Question Answering, pp. 37-46, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 11111865////// 2605. URL https wwwpe/thooooyyoog/1111----
Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference Learning Representations, 2022. URL https: pottps://oi //oper/eeee//.eeee////m net/
Vorum?id=nZeVVeeFFVz
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William Sayed. Mistral 7B. aiv preprint, 2023a. URL //tp///axvvvv// htt.a//cc.//aaaaaooo/////////.00
06825
Ting Jiang, Shaohan Huang, Zhongzhi Luan, Deqing Wang, and Fuzhen Zhuang. Scaling sentence embeddings with large language models. 2023b. URL https //arxxiv org/ abs/2307 16645.
12

---

Published as a conference paper at COLM 2024
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer: TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzi- lay and Min Kan (eds.), Proceedings of the 55th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601-1611, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v1/P11/
URL
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answer- ing. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6769-6781, Online, November 2020. Association for Computational Linguistics. doi: 10. (0.18853/vv /v11 4020.emmll--min.55
org/2020 emnlp-main 550
Omar Khattab and Matei Zaharia. ColBERT Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '20, pp. 39-48 New York, NY, USA, 2020. Association for Computing Machinery, ISBN 9781450380164. doi: 011145/339727.34400
URL httpt:htpp///tt htt.ddoioorggg//// org/10..//00 1145//1977773111111 3401075
Yibin Lei, Di Wu, Tianyi Zhou, Tao Shen, Yu Cao, Chongyang Tao, and Andrew Yates. Meta-task prompting elicits embeddings from large language models. 2024. URL https
//arxiv.org/ abs/ 2202.22222.222 18458
Xianming Li and Jing Li. BeLLM: Backward dependency enhanced large language model for sentence embeddings. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceed- ings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 792 192-804, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: N0.18653/v11222 naacl-long.45. URL httpp://acclantiooogg.oooo otg. naac- -
Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang.
Towards general text embeddings with multi-stage contrastive learning. arxiv preprint, 2023a. URL htttpptss////xiit/
abs//308. 03281
Zongxi Li, Xianming Li, Yuzhang Liu, Haoran Xie, Jing Li, Fu lee Wang, Qing Li, and Xiaoqin Zhong. Label supervised Ilama finetuning. Xrv preprint, 2023b. URL https
//arxiv. saaxiv.org/ae wwwp////......... 01208
01208
Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. A structured self-attentive sentence embedding. In International Conference on Learning Representations, 2017. URL BJC_jUqxe
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arx preprint, 2019. URL http:://arxiv.org/aaa 11692
Ang Lv, Kaiyi Zhang, Shufang Xie, Quan Tu, Yuhan Chen, Ji-Rong Wen, and Rui Yan. Are we falling in middle-intellignnce trap? an analysis and mitigation of the reversal curse. arv preprint, 2023. URL /attps://arxix.
www.rrvvv...gr..eeoo farg/abl abs/ /211 07468
Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. Fine-tuning LLaMA for multi-stage text retrieval. arXiv preprint, 2023. URL httts:::/arxivoog//uteet 08319
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017. URL https //openreviev www.:/eoooooodddyyyy///oo..
Niklas Muennighoft. SGPT: GPT sentence embeddding for semantic search. ar Xiv preprint, 2022.
13

---

Published as a conference paper at COLM 2024
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. MTEB: Massive text embedding benchmark. In Andreas Vlachos and Isabelle Augenstein (eds.), Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 2014-2037, Dubrovnik, Croatia, May 2023. Association for Computational I Linguistics. doi: 10.18653/v11
URL https: hatpps:/aclaanhhooogg..ooo org/ wtwp:///......m...aa.aeeea.aaaa. eacl-main
148
Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. Generative representational instruction tuning. ar Xiv preprint, 2024. /ttps://s::// org/ abaaaass//aa 2402. 09906
Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, Johannes Heidecke, Pranav Shyam, Boris Power, Tyna Eloundou Nekoul, Girish Sastry, Gretchen Krueger, David Schnurr, Felipe Petroski Such, Kenny Hsu, Madeleine Thompson, Tabarak Khan, Toki Sherbakov, Joanne Jang, Peter Welinder, and Lilian Weng. Text and code embeddings by contrastive pre-training. ar Xio preprint, 2022. URL bttppts/://axiii http:////xxivooo//////// 2201 10005
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vin- cent Zhao, Yi Luan, Keith Hall, Ming-Wer Wei Chang, and Yinfei Yang. Large dual en- coders are generalizable retrievers. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 9844-9855, Abu Dhabi, United Arab Emirates, December 2022. Asso- ciation for Computational Linguistics. doi: (0.11853//1122222coo mainnlm.m.i.666. URL https://aclanthology) org.g/22.gg2222222 emnlp-maaaeeeeeee main..miinccmmm 669
Long Ouyang, Jeff Wu, Xu Jiang. Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. arXiv preprint, 2022. URL
02155
Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive summarization. In International Conference on Learning Representations, 2018. URL https
wttprreewweiineeffffoooeeee id=HAACQ
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL http://milr..orgpppeerr//s////
html
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Jian Su, Kevin Duh, and Xavier Carreras (eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, Austin, Texas, November 2016. Association for Computational Linguistics. URL
doi: 10.18653 1264
Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLPP), pp. 3982- 3992, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.15653/11/D19-1410 UKLpppps///bllattoogggoog////-
Jacob Mitchell Springer, Suhas Kotha, Daniel Fried, Graham Neubig, and Aditi Raghunathan. Repetition improves language model embeddings. arXiv preprint, 2024. URL https
//arxiv.org. absa 2ea.b///220..... 15449
Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. One embedder, any task: Instruction- finetuned text embeddings. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 1102-1121,
14

---

Published as a conference paper at COLM 2024
Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/ /v1/ org/ 2023. findings-acl.. ac1.
2023 2023. www.diignn--cl.i. acl.71.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. FEVER: A large-scale dataset for fact extraction and VERification. In Marilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers), pp. 809-819, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/vv /1/11111111100 1074. URL
Erik F. Tjong Kim Sang and Fien De Meulder. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pp. 142-147, 2003. URL httpp://www acwbwe/ebbew...weebbbb.. hwtgg//ntollgy//// 043-
aclweb.
Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, D. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, A. Hartshom, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. preprint, 2023. /ttttsss: org/ abs/ 09288
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systemss, volume 30. Curran Associates, Inc., 2017. URL htttps::/pprcceeeingggg wtwpeeeeeeddineeenennseeeeeeeeee neurips...... wttparrppppcppppe...eeeeeee////e FE5ee4/5577ee11110055CCCCC833353
faraper_filss wwwps//paeeee///////// paale////apee/21//ffieee
pdf
Liang Wang. Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings supelyys contrastive pre-training. Xrv preprint, 2022a.
23333
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improving text embeddings with large language models. arx preprint, 2023. URL
https://arxiv.cm org/ abs/ halss//44100000 00368
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunku- mar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Puro- hit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, Generalization via declarative instruc-
and Xudong Shen. Super-Naturallnstruttii tions on 1600+ NLP tasks. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 5085-5109, Abu Dhabi, United Arab Emirates, December 2022b. Ássoci- ation for Computational Linguistics. doi: 10.18653/v1 20mp///002.mmmpmmaa...a 340. URL www.lloo..ooogoooooooeoe org/ 2022.22022ee00 wwn.emmain.neeee
Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. CLEAR: Contrastive learning for sentence representation. arviv preprint, 2020. URL ht
arxiv. seciv.org/abbe 2bs/ 15466
15

---

Published as a conference paper at COLM 2024
Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared LLaMA: Acceler- ating language model pre-training via structured pruning. In Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization htt///orrmmmdd667/////
(WANT@NeurIPS 2023), 2023. URL https://openrevieet
Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-Pack: Packaged resources to advance general chinese embedding. arXiv preprint, 2023. URL https
//arxiv.org wtwgg//b//09..000//// 07597
Xiaohui Xie, Qian Dong, Bingning Wang, Feiyang Lv, Ting Yao, Weinan Gan, Zhijing Wu, Xiangsheng Li, Haitao Li, Yiqun Liu, and Jin Ma. T2ranking: large-scale chinese benchmark for passage ranking. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '23, pp. 2681-2690, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9781450394086. doi: (0.1145/3539618.. 833998844. URL https://dbi.
1141/35396661118 3591874
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhut- dinov, and Christopher D. Manning. HotpotQA: dataset for diverse, explainable multi-hop question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Brussels, Belgium, October-November 2018. As-
Language Processing, pp. sociation for Computational Linguistics. doi: 10.18653/v1/D18- 1259. URL https
//aclanthology org/D18-10 1259
Xinyu Zhang, Xueguang Ma, Peng Shi, and Jimmy Lin. Mr. TyDi: A multi-lingual bench- 1
mark for dense retrieval. In Duygu Ataman, Alexandra Birch, Alexis Conneau, Orhan Firat, Sebastian Ruder, and Gozde Gul Sahin (eds.), Proceedings of the 1st Workshop on Mulfi- lingual Representation Learning, pp. 127-137, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/ /01655/vv/2000 v122021 mr1-1.12. URL https://aclanthologgs org.orggg//0 mr1-1.12
Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso- Hermelo, Xiaoguang Li, Qun Liu, Mehdi Rezagholizadeh, and Jimmy Lin. MIRACL:
A Multilingual Retrieval Dataset Covering 18 Diverse Languages. Transactions of the
Association for Computational Linguistics, 11:1114-1131, 09 2023. ISSN 2307-387X. doi: 10.1162 tacl 00595. Rtl.atttss//boi..r//10..... wwt.://////....////eeeeeeeeeee htcl//ecceccc..e a 00595
16

---

Published as a conference paper at COLM 2024
A Limitations
Large size of decoder-only LLMs Recent years have seen increasing trend towards training very large decoder-only LLMs, with model sizes up to 540B parameters (Brown et al., 2020/Chowdhery et al. 2023). The parameter size of the model has direct impact the training and inference latency. Additionally, the large output embedding dimension of these models (e.g., 4096 for Mistral-7B compared to 768 for BERT) also makes them more memory and compute intensive for creating vector indexes for large document collections. While some of these limitations can be offset by recent advances improving the training and inference efficiency of large models (Hu et al. 2022 Dao 2024), these techniques can technically be applied to smaller bidirectional models as well.
The advantages of small bidirectional encoders come at the cost of complex and compu- tationally intensive training regimes (Li et al. 2023a Xiao et al. 2023 Li et al. (2023)). In contrast, decoder-only models are much more sample-efficient and do not require large-scale contrastive pre-training (Wang al. 2023 Moreover, the instruction following capabilities of decoder-only models make them strong contenders for building text embedding mod- els that generalize to a wide range of tasks and domains without the need for expensive adaptation.
not
While smaller models can be more practical for some applications, the sample-eficienccy the instruction following capabilities, and the widespread use of these models in the community motivates the need to explore the potential of decoder-only LLMs for text embedding tasks.
Data contamination from pre-training As our supervised data contains train splits of publicly available datasets, there is extremely low chance of test set contamination with the MTEB benchmark. However there is a possibility of contamination from the pre-training data of LLaMA-2-7B and Mistral-7B models (s- MLLLLMMA-1... As the complete details of the pre-training data are not publicly available, we cannot be certain about the extent of contamination. However, to reliably compare with other works, we stick to our choice of model and evaluation benchmark. We leave it to future work to investigate the performance of these models on newly designed benchmarks that are not part of their pre-training data.
was distilled from LLaMA-2-7B)
Extending to other languages In this work, we have implemented and evaluated our proposed methodology LLM2Vec using only English text corpora and benchmarks. However, the methodology is language-agnostic and can be easily extended to other lan- guages using just unstructured text collections. We leave to future work to investigate the performance of LLM2Vec on other languages.
B Background
B.1 Self-attention
The self-attention mechanism is crucial component of decoder-only LLMs Cheng et al. (2016); Lin al. (2017); Vaswani al. (2017); Paulus al. (2018). Given a sequence of N tokens, the token representation at any given transtormer layer X2, .,)) with E Rd are stacked into matrix X E Nxd anism computes the query, key, and value matrices Q, V E NNp via learned linear transformation.
xd Given this matrix, the self-attention mech-
Q = XWO
(1)
K XW
(2)
= XWV.
(3)
The output of the self-attention layer is then computed as a linear combination of the values, weighted by the normalized inner product between keys and queries:
17

---

Published as a conference paper at COLM 2024
M Miss OK?
0 - =oofmaxx
V.
(4)
This output is then passed through a feed-forward network and added to the residual stream to obtain the token representations at the next layer. Crucially, in the case of decoder-only LLMs, the attention mask (isi)) prevents accessing token embeddings to the right of the current token.
B.2 Contrastive learning
Contrastive learning is popular paradigm to learn text representations Karpukhin et al. (2020); Gao et al. (2021]; Su et al. (2023); Wang et al. (2023); Springer et al. (2024). In the supervised setup, we have set of positive pairs D = {g: d; } 1. and set of negative documents that can include hard or in-batch negatives. The model trained to maximize the similarity (i.e., usually cosine similarity) of positive pairs and minimize the similarity of negative pairs, ie,, we optimize the following objective:
essqqat))
C eslgdd)) + Es EN eds(gdt))
(5)
where is a similarity metric, temperature value, and N all the negative documents for query
Unsupervised contrastive learning In unsupervised contrastive learning, no positive or hard negative pairs available. Most unsupervised approaches construct two different representation for the same sample, using either model or input perturbations. SimCSE (Gao et al. 2021), the unsupervised approach used in this work, creates two different representations of the same input by using independently sampled dropout masks the intermediate model representations and train the model with in-batch negatives.
C Massive Text Embeddings Benchmark (MTEB)
C.1 MTEB subset details
MTEB consists of diverse small and large embedding tasks. To speed up the evaluatior we consider a representative subset of 15 tasks from MTEB for our analyses, presented Table To make sure that our ablation and analyses are not biased towards one specific category or task, this subset includes tasks from each category with almost the same proportion compared to the full 5
C.2 MTEB instructions
When evaluating MTEB, we use the same instructions as Wang et al. (2033. The list of instructions for each task is listed in T1
FFul evaluation on MTEB takes more than 40h for Mistral-7B on GPUs. SSince the MTEB's SulmFval category includes only dataset, we skip this category in our small-scale evaluation.
18

---

Published as a conference paper at COLM 2024
Category
Dataset
SciFact
Retrieval (3)
ArguAna
NFCorpus
StadOOverlowDupQQuetttos
Reranking (2)
SciDocsRR
BiorxivClusteringg25
Clustering (3)
MedraivCClustrring525
[wenty/Vewsgoupsslusserigg
Pair Classification (1)
SprintDDapliaaeeuuusttinss
Bankinng7CCaasffication
Classification (3)
EmottonClassffictionn
MassivelntennClassfficttton
STS17
STS (3)
SICK-R
STSBenchmark
SummEval (0)
Overall
15 datasets
Table 3: Subset of MTEB tasks used for our ablations and analysis.
D Details on unsupervised results
D.1
Training details
D.1.1
MNTP training details
The second step of LLM2Vec includes MNIP training. We follow established practice from the encoder-only literature for choosing our masking strategy. For example, (Devlin al. 2019) mask 15% of the tokens in the input. 10% of the masked tokens are then replaced with a random token from the vocabulary, while another 10% are unmasked again, but still considered when computing the loss. As another example, RoBERTa (Liu et al. 2019) also masks 15% of the input tokens but applies no further post- proce-socsssing to the masked tokens.
For our models, perform hyperparameter search to select the percentage of the masked tokens in a sequence choosing from 20%, 40%, 60%, 80%, and 90%. For each model, we take the best setup (i.e., masking probability and BERT RoBERTa approach) based on the performance on SICK-R (Agirre et al. 2014) task from the MTEB dataset. This results in the following choices: for S-LLaMA-1 3B, LLMM-2--77 and Meta-LLaMA 3-BB, we apply BERT's masking strategy with masking probability of 20%. For Mistral -7B, we apply RoBERTa's masking strategy with probability of 80%.
We train all the models for 1000 steps with LoRA 16 and a = 32, and we follow the same training parameters as RoBERTa MNTP training. When training large 7B and 8B models, we apply brain floating point (bfloat16) quantization, as well as flash attention and gradient checkpointing.
D.1.2 SimCSE training details
The last step of LLM2Vec involves unsupervised contrastive learning with SimCSE. Our initial experiments indicated that the low value of dropout probability (0.1) typically used by bidirectional encoders (Gao et al.) 2021) does not lead to optimal performance for larger decoder-only LLMs. Theretore, we use a higher dropout probability of 0.3 for all models.
Similar to MNTP, we train all models with LoRA 16 and For LLaMA-2-7B, Mistral-7B, and Ssta-LLaMA-3-955 we train with a batch size of 128. For S-LLaMA-1. 3B, we use batch size of 32. Additionally, when training LLaMA-2-7B
32 for 1000 steps.
19

---

Published as a conference paper at COLM 2024
Mistral-7B, and Meta-LLaMA-3- 3--88B, apply brain floating point (bfloat16) quantization, flash attention 2, and gradient checkpointing.
D.1.3 Word-level training details
We evaluate on three popular word embedding tasks: chunking, named-entity recognition (NER), and part-of-specch (POS) tagging. We train a linear classifier using dropout with dropout probability of 0.1 on top of the frozen representations obtained from the last hidden layer of model.
We use data from CoNLL-2003, consisting of roughly 14,000 training, 3,250 validation, and 3,450 test samples (Tjong Kim Sang & De Meulder 2003). We train the classifier for 1,500 steps with a learning rate of 5e and a batch size of 8. For experiments with Mistral-7B models that have been tuned with MNTP, use the variant which is trained with BERT's masking strategy and masking probability of 20% (please see D.1.1 for more details). Although 80% masking helps with the performance in sentence-level tasks, it prevents the model from learning proper token representations essential for word-level tasks.
Since the models we experiment with have sub-token based vocabularies, we calculate the embedding of word by averaging the representations of all its sub-tokens. For example, for sentence "wi W2 W3" which is tokenized as "BOS t11t12 12122222 f31" the representation of Wh w2, and wy will be computed as
e1 = (e11 + e12), e2 = (e21 + 622 + e23), E3 631
Here, is the final representation of token f. or word wo Moreover, for the models that have gone through MNTP, we calculate the representation based on sub-tokens of the previous word. Using the same example as above, for models trained with MNTP, the representation of words W1, W2, and W3 will be computed as:
1
e1 = (eRos + tnn) 62 = (e12 + e21 + 622), e3 = 023.
D.2 Additional results
D.2.1 Word-level task results
We present the detailed breakdown of the performance of LLMMVVe--rransfomeed models on the word-level tasks in Table Our results show that applying MNTP training to decoder- only LLMs helps them take advantage of the enabled bidirectional attention which boosts their performance on word-level tasks.
D.2.2 Sentence-level task results
Table presents the results on MTEB subset for all models across different pooling methods. Results show that While weighted mean works the best for causal (i.e., Uni) models, mean pooling performs the best for M.M22ec approach.
In Table 11 we additionally present breakdown of the unsupervised performance of -LM2Vec-transformed models on MTEB.
Comparison with Echo embedding
E.1 Reproducibility
Concurrent to our work, Springer et al. (0244) proposed Echo embeddings, simple ap- proach to convert decoder-only LLMs into text embedders by copying the input sequence and appending it to itself. For evaluation, they follow prompt sampling procedure for the task instruction. However, they report that the exact wording or template used as prompting strategy does not have strong effect on the performance.
20

---

W. mean Published as a conference paper at COLM 2024
Model
Chunking
NER
POS tagging
Encoder-only
BERT-large DeBERTa-large
71.77
90.09
75.12
85.74
94.97
86.49
S- S-LLaMA-1.33
Uni
86.10
96.09
90.89
Bi
76.50
92.17
89.18
Bi MNTP
90.51
96.59
92.04
SimCSE
75.93
91.45
89.22
MNTP SimCSE
89.33
95.90
90.38
LLaMA-2-78
Uni
88.23
96.59
91.53
78.24
92.31
90.62
MINP
91.61
97.16
92.61
SimCSE
77.75
91.96
90.48
Bi MNTP SimCSE
89.66
96.05
90.53
Mistral-78
87.53
96.52
90.86
85.66
97.14
90.70
Bi + MNTP
91.17
97.18
92.35
Bi SimCSE
86.91
97.15
92.12
MNTP SimCSE
90.69
96.87
92.08
Table 4: Unsupervised results on the word-level tasks for different models.
Model
EOS
Mean
S-LLaMA-1.23
Uni
27.72
33.03
34.99
Bi
21.16
30.26
30.20
Bi MNTP
29.16
42.10
38.67
Uni + SimCSE
37.44
44.95
47.13
Bi SimCSE
40.43
44.46
44.83
Bi MNTP + SimCSE
45.57
52.40
50.23
LLaMA-2-7B
Uni
33.23
45.83
47.85
Bi
34.47
38.22
37.50
Bi + MMTP
32.66
48.00
44.30
Uni + SimCSE
38.47
52.03
53.55
Bi + SimCSE
40.37
44.13
44.08
Bi MNTP SimCSE
50.61
58.97
55.75
Mistral-7B
Uni
22.12
43.00
44.01
Bi
25.17
50.07
45.20
Bi + MNIP
26.54
53.89
48,93
Uni + SimCSE
34.60
52.04
53.95
Bi + SimCSE
49.73
60.29
56.56
MNTP SimCSE
53.67
60.50
57.55
Table 5: Unsupervised results on MTEB subset for different models.
For fair comparison to proposed models, implement Echo embeddings using the instructions in our evaluation setup (Appendix (C2). To do a sanity check on our implementation, well as to see the impact of exact wording of instructions, evaluate
21

---

Published as a conference paper at COLM 2024
Echo embedding the same subset of 26 MTEB tasks that was chosen in their work. We run this evaluation using the are directly comparable to theirs.
model to ensure that the results
The unsupervised Echo model based on our implementation and instructions achieved score of 55.22 the 26 task subset, whereas their reported score is 55.07. This result validates our implementation and confirms an observation made by Sprinnger et al. (2024) the exact wording or template used does not have a strong effect the pertormance.
E.2 Efficiency
In Table report the approximate evaluation time it took (in hours) to evaluate each of the models on MTEB using 8x 80GB A100 GPUs. Given that Echo embeddings rely on copying the input text, evaluation takes much longer compared to our approach. We note that the increased inference time of Echo embeddings is especially problematic for the encoding of large retrieval corpora in MTEB benchmark.
Model
LLM2Vec
Echo embeddings
S S-LLaMA-1.. 1B
2 30 hrs
~ 40 hrs
LLaMA-2-7B
< 42 hrs
~ 63 hrs
Mistral-7B
44 hrs
64 hrs
Table 6: Evaluation time of Echo Embeddings compared to LLM2Vec in hours on 8x 80GB A100 GPUs.
More analysis results
Data used for our analysis Table 7
shows examples of the data used for our cosine
similarity analysis in Section
q: the query
s the positive sample
s the negative sample
She loves to travel in sum- mer, especially to cold des- tinations, avoiding hot and crowded places.
She loves to travel in sum- chilly
She loves to travel in summer, but prefers visit hot and bustling tourist spots.
mer, specifically locations, steering clear of warm, populous areas.
The cat often sits by the window, dreaming of chas- ing birds and enjoying the warm sunshine.
The cat often sits by the win- dow, imagining bird pur- suits and basking in the sunlight.
The cat often sits by the win- dow, but is too lazy to dream of chasing anything.
He reads books every night, finding solace in fiction and escaping from the stresses of daily life.
He reads books every night, seeking comfort in stories and evading everyday ten- sions.
He reads books every night, yet he feels that non-fiction is more engaging and infor- mative.
She paints landscapes week- ends, expressing her creativ- ity through vibrant colors and abstract forms.
She paints landscapes week- ends, showcasing her artis- tic flair with lively hues and unconventional shapes.
She paints landscapes on week- ends, preferring realistic and detailed depictions of nature.
Table 7: Toy data used for the analysis in Section by Springer et al. (2024).
4 These sentences were originally collected
Cosine similarity analysis Figure cosine similarity results for all three models. In addition to our LLLMVoee-tr models, we also provide results for Echo emebddings.
22

---

Published as a conference paper at COLM 2024
E1 Additional plots
fiew triiing!
Frho exbeddings
Siedg. 1
Sinie x
Sinis 17
Sinie x
Danday
Sirla
(110 (raine dirivity)
0.e ciabviy)
San (cosine
(a) G-LLaMA-1.33
mokieg
NNTP
NNTP SmrCS
Edho unbuldin
Sivdg
Sindg 1
Sinfe )
Sinte x
Sinte
019
0.50
6:96
0.80
0.55
3.10
0.35
0:30 9.5
(:500 0.05
Sain (cridie deviarg tyl
Skn (cisie siniarty)
Shn (rnsie Siriaity)
Skr (ciske sevibiit)
(b) LLaMA-2-7B
Bi Eoo necknng
HNTP
NNTPP
Edoo erbedding
Simdq
Sinto
Sinip
Sinte ]
5in (cosine
Skr siriaitt)
(ctske siriarty)
(cosiee
(c) Mistral-7B
Figure 7: Cosine similarity between query and negative as well as positive examples for S-LLaMA-1.3B 3B, LLaMA-2-7B, and Mistral-7B.
Layer
Layer
Layer 16
32
1.30
1.00
L.00
Cosine similarity
0.75-
similritt
0.76
Cosine similartty
1,775
0.50
0,50
0.50
0.25
0.25
0.22
0.00
25
50
75
100 125
150 175 200
0.00
25
50
775
100
125
150
175 200
0.00
50
75 Token position
100
125
150 175 200
Taken position
Token pasition
(a) S S-LLaMA- -1. 3B
(b) L1 ama -2-BB
(c) Mistral-7B
Layer 12
Layer 16
Layer 36
20
32
1.00
1.00
1.00
Cosine similaity
simitortty
Cosine similerity
0.75-
0.75
0.50
0,50
0.50
0.25-
0.25
0.005
25
50
125 Taken position
75
100
150
175
25
50
75
100
125
150
175
25
50
75
100
125
150
178
Taken position
Token postiion
(d) S-LLaMA-1.3B
(e) 1.lama-2-77
Mistral-7B
Figure 8: Cosine similarities at different token positions at layers when comparing represen- tations constructed with causal attention to those constructed with bidirectional attention (without training).
Representation analysis using two different Wikipedia paragraphs. The trends closely follow those reported in Sec- tion 4 Figure& that the same behavior we observe for Mistral-7B-instree
additional plots for the representation analysis
23

---

Published as a conference paper at COLM 2024
Layer 16
Layer 16
Layer
22
36
22
1.30
1.00
1,00
Cotine similarty
similority
Cosine simileity
0.50
0.50
0.30
0.25
0.25
0.25
0.00
0
25
50
75
100
125
150
175
0.00
25
50
75
.00
125
150
175
0.000
25
50
75
178 100
125
150
Taken
Taken position
Takon
(a) Mistral-7B-vo.1
(b) Miistaalls------ -isstral-7--Instruct-000
Figure 9: Cosine similarities at different token positions at layers when comparing repre- sentations Mistral models constructed with causal attention to those constructed with bidirectional attention (without training).
Dataset
Instruction(s)
NLI
Given premise, retrieve hypothesis that entailed by the premise Retrieve semantically similar text
Retrieve
DuReader
Given a Chinese search query, retrieve web passages answer question
ELJ5
Provided user question, retrieve the highest voted answers Reddit ELI5 forum Given a claim, retrieve documents that support or refute the claim
FEVER
HotpotQA
Given multi-hup quesfion, retrieve documents that can help answer the question Given a question, retrieve Wikipedia passages that answer the question question,
MIRACL
MrTyDi
Wikipedia passages that answer the question Given weh search query, retrieve relevant passages that answer the query Given web search query, retrieve relevant documents that answer query Given question, retrieve Wikipedia passages that answer the question
MSMARCO Passage
MSMARCO Document
NQ
QupraDaplicates
Given a question, retrieve questions that are semantically equivalent to the given question Find questions that have the same meaning the input question
9QuAD
Retrieve Wikipedia passages that answer the question
12Ranking
Chinese search query, retrieve passages answer question
TriviaQA
Retrieve Wikipedia passages that answer the question
Table 8: Instructions used for each of the E5 datasets.
also holds true for other variants of the Mistral-7B model. We take this as additional evidence that the Mistral-7B base model was trained with some for of bidirectional attention.
G Details on supervised results
G.1
E5 dataset
The dataset consists of EL.5 (sample ratio 0.1) (Fan et al. 2019), HotpotQA (Yang et al. 2018),
FEVER (Thorne al. 2018), MIRACL (Zhang et al. 2023), MS-MARCO passage ran ranking
(sample ratio 0.5) and document ranking (sample ratio 0.2) (), NO (Karpukhin et al. 2020),
NLI (Gao et al. 2021,, SQuAD (Rajpurkar et al. 2016), TriviaQA (Joshi et al. 2017 Quora
Duplicate Questions (sample ratio 0.1) (DataCanary et al. 2017), Mr- TyDi (Zhang et al.
2021), DuReader (He et al. 2018), and T2Ranking (sample ratio 0.5) (Xie et al. 2023) The instruction used for each dataset can be found in Table
G.2 Training details
All models are trained with LoRA 16 and 32, brain floating point (bfloat16)
quantization, gradient checkpointing, and flash attention 2 (Dao) 2024 to optimize GPU memory consumption. We train 8 NVIDIA A100 GPUs with effective batch size of 512 for 1000 steps using maximum sequence length of 512 tokens. We use the Adam optimizer with learning rate of 2e 4 and a linear learning rate warm-up for the first 300 steps.
24

---

Published as a conference paper at COLM 2024
Rerank.
Avg
Dala
9FR Engeedddddd---iii Mistral
14.22
X
4056
99.00
60.66
51.67
58.54
78. 28:33
85.05
31.16
67.
noyage--ite---instnnnd i2-issrrrt
2,45
X
1024
55.60
56.24
52.42
86.57
79.25
85.79
31.01
67.
GriLLM-BB
14.08
X
4396
57.41
60.49
50.61
87.36
79.46
83.35
30.37
66.76
<5-mistral77h-innnee -7hinsimttt
14.22
4/96
56,89
60.21
50.26
88.34
78.47
84.63
31.40
66-63
GritLM-&x7B
93.41
1056
55.09
58,80
50.14
81.97
78.53
83.26
29.82
65.66
MNTIP
mvbsi-embee--hrgee
1024
$1.39
60.11
16.71
73.64
32.71
61.68
Large
1.34
1024
54.66
88
46.73
87.25
73.58
84.54
32.03
64.64
10
testembbedding--lrre
3072
55.44
59.16
49.01
85.72
75.45
81.73
29.92
64.59
Table 9: Top-10 models on the MTEB leaderboard of 2024-03-29. LLM2Vec achieves the 6th rank overall, and the top rank among models trained with only publicly available data.
G.3 Results
Table presents the performance of applying MNTP and Bi MNTP + SimCSE with mean pooling on MTEB benchmark. We also compare the performance of our models with recent and popular models trained with only publicly available data. We further report the current top-10 models in the MTEB leaderboard, including Our models achieve the 6th score in the MTEB leaderboard and the 1st among the models trained with only public data.
in Table
We present the detailed performance supervised LLMMVVc-ttransfrmmed models on full MTEB in Table 12. Here, we only report the Bi + MNTP transformed models as we showed they perform the best after supervised fine-tuning.
25

---

Published as a conference paper at COLM 2024
Iask Name
Instruction
Classify a given Amazon customer review either counterfactual or nol-counttrfactuul
AmazunPolarityy Classification
Classify Amazon reviews into positive ur negative sentiment
AmazonReviewed Seweevviesaassssaattiiiii
Classify the given Amazon review into appropriate rating category
Booking7CClsssficatte CmnotinCCasssficatinn
Given a online banking query, find the corresponding intents Classify the emotion expressed in the given Twitter message into one of the six emotions:
anger, tear, joy, love, sadness, and surprise
ImdbClassification
Classify the sentiment expressed in the given movie review text from the IMDB dataset
Messivelnten(Classfiina
Given
utterance as query, find user intents
Given a user utterance as query, find the user scenarios
MTOPDomain lassification
Classify the intent domain the given utterance in task-oriented conversation
Clussify the intent of the given utferance in task-uriented conversation
Toxie ConversationsClassif
Classify the given comments as either toxic or not toxic
Classify the sentiment Identify the main and secondary calegory of Arxi papers based on the titles and abstracts
given
either positive, negative, neutral
Andiv Arxi<ClusteringP22
Arxiv ClusteringS22
Identify the main and secondary categury of Arxiv papers based on the titles
Biorxiv HioriiCCustteringe Clustering!2P
Identify the main category of Biorxiv papers based on the titles and abstracts Identify the main category of Biorxiv papers based on titles
Biorxiv Clusteriigg
MedrxiyClustering
Identify the main categury of Medrxiv papers based the titles and abstracts Identify the main category of Medrivv papers based on the titles Identify the topic theme of Reddit posts based the titles Identify the topic or theme of Reddit posts based on the titles and posts Identify the tupic or theme StackExchange posts based the titles
Medrxiv CinCtsstii
ReddiCClsttering
Redda Clusterrinn
Stack ChangCClusteringg
Stack FacckFxchann Exchange Chusterrnnnntr
Identify the topic or theme of StackFxchange posts based on the given paragraphs Identify the topic or theme of the given news articles Retrieve duplicate questions from Sprint forum
Evvate55emEva22115
Retrieve tweets that are semantically similar to the given tweet Retrieve
WiiterURRCorppu
that semantically similar the given tweet
AskL AskU/buntuDDn Dupuntuuuuuuuution
Retrieve duplicate questions from AskUbuntu forum Retrieve relevant news articles based user browsing history Given a title of scientitic paper, retrieve the titles of other relevant papers Retrieve duplicate questions from StackOverflow forum
Mind5mall MindSkallRerankiing
SeiDoesRR
StacKhverloowuuppuestiee
ArguAna
Given claim, find documents that refute the claim
ClimateFEVER
Given a claim about climate change, retrieve documents that support or refute the claim Given question, retrieve detailed question descriptions from Stackexchange that are duplicates to the given question
COADupstackRettieve
DBPedia
Given a query, retrieve relevant entity descriptions from DRPedia Given a claim, retrieve documents that support refute the claim Given a financial question, retrieve user replies that best answer the question
FEVER
RQA20188
HotpotQA
Given multi-hop question, retrieve ducuments that help answer the question Given a web search query, retrieve relevant passages that answer the query Given a question, retrieve relevant documents that best answer the question Given a question, retrieve Wikipedia passages that answer the question Given a question, retrieve questions that are semantically equivalent Given a scientific paper title, retrieve paper abstracts that are cited by the given paper Given a scientific claim, retrieve documents that support or refute Given question, retrieve detailed and persuasive arguments that answer the question Given a query on COVID-19, 19, retrieve documents that answer the query Retrieve semantically similar Retrieve parallel sentences. Given a news summary, retrieve other semantically similar summaries
MSMARCO
NFCorpus
NQ
Qutravetrieval
the given question
SCIDOCS
SciFact
claim
Touchu2020
TRECCOVID
BUCC/Tatocha
SummEval
Table 10: Instructions used for evaluation on the MTEB benchmark. "STS"" refers to all the
STS tasks.
26

---

Published as a conference paper at COLM 2024
Task
S-LLaMA-11..
1LaMA-2-77
Mistral-7B
72.93
76.91
76.94
AmazonPolarttyCCnasicttttoo
74.28
79.05
85.29
AnazznnerieessClssiiccttton
36.14
40.08
47.09
ArguAna
43.64
47.09
51.00
Arxiv ExxicClustering12P
42.91
47.81
47.56
ixiyClusteringS25
35.20
40.53
39.92
AsUUbuntuDupQuestions
52.70
55.56
38.60
BIOSSES
75.12
82,41
83.29
Banking77CCaassffcation
79.00
84.65
86.16
BiorxivClusteringP22
35.02
38.12
36.14
BiorxivClustering525
27.21
31.25
30.26
COADupstackRetrieval
18.50
30.78
33.37
Climate/EVER
18.95
20.67
22.97
DBPedia
13.21
25.81
25.48
EmottonClassffiationn
42.85
46.58
48.88
FEVER
16.96
43.48
45.11
FiQA2018
16.99
24.62
27.24
HotpotQA
22.64
48.46
54.54
IndbClassification
71.92
75.68
77.95
MSMARCO
7.03
18.81
19.13
MTOOPDomaiCCLassiicattnn
91.24
94.33
95.48
MIOPIItennCClassifcatioon
74.08
79.54
82.84
MassivelntentCClassiccttto
69.99
73.84
76.65
MasiivesenaariCllassictttoo
75.15
79.17
79.99
MedraiiCChustering222
30.15
30.94
30.11
MedrwivClusteringS22
26.96
28.04
26.93
MindSmallKerankingg
29.52
30.86
29.73
NFCorpus
15.73
26.81
27.16
NQ
17.96
33.21
34.16
QuoraRetrieval
78.23
86.15
84.40
RedditClustering
38.67
42.84
41.83
ReddiiClusteriggP22
53.42
60.10
62.08
SCIDOCS
5.53
10.00
15.35
SICK-R
69.34
71.77
75.55
STS12
60.09
65.39
67.65
STS13
72.52
79,26
83,90
STS14
66.70
72.98
76.97
STS15
77.69
82.72
83.80
STS16
75.94
81.02
81.91
STS17
81.67
86.70
85.58
STS22
63.70
63.47
65.93
STSBenchmark
73.36
78.32
80.42
SciDocsRR
67.76
77.62
77.81
SciFact
38.31
64.48
68.67
SprintDuphicaeeeuustiio
77.36
87.57
91.30
StackExchangeChsssrringg
59.35
65.12
67.34
StadEExchaneeClusstrinng2ZZ
31.47
33.61
34.50
StackOverflow DupQussionsssssssss
40.82
47.77
49.80
SummEval
31.23
31.38
30.19
TRECCOVID
56.04
60.67
55.66
Touche2020
19.17
10.18
6.54
loxi@CanveraatoonsClaiiiccciai
68.41
71.81
70.71
56.08
57.17
60.90
"weetty\lwwgrrouusCCuuseerng
31.54
30.76
30.26
Tvil2er5SemEva22015
61.54
65.14
68.76
TwitterURI.Corput
77.73
80.94
82.76
Average
49.42
55.36
56.80
Table 11: Unsupervised results of LLM2Vec transformed models on MTEB.
27

---

Published as a conference paper at COLM 2024
Task
S-LLaMA-1..8
LLaMA-2-7B
Mistral-7B
77.42
82.22
77.58
AmazonPolarttyCCnasicttttoo
82.05
89.69
91.12
AnazznnerieessClssiiccttton
40.81
48,47
49.97
ArguAna
51.66
56.53
57.48
Arxiv
ExxicClusteringP22
43.47
43.14
42.81
ixiyClusteringS25
39.85
42.38
44.24
AsUUbuntuDupQuestion
60.71
63.13
63.98
BIOSSES
85.88
82.13
85.24
Banking77CCaassffcation
86.01
88.17
88.31
BiorxivClusteringP22
37.10
35.88
34.27
BiorxivClustering525
34.28
34.81
35.53
COADupstackRetrieval
41.73
45.94
48.84
Climate/EVER
33.49
30.70
35.19
DBPedia
43.58
48.42
49.58
EmmotooCCassifiaation
48.38
51.71
52.05
FEVER
86.81
89.93
89.40
FiQA2018
41.00
51.28
53.11
HotpotQA
63.85
72,99
74.07
IndbClassification
75.33
85.78
87.42
MSMARCO
38.32
41.45
42.17
MTOOPDomaiCCLassiicattnn
94.09
95.57
96.04
MIOPIItennCClassifcatiion
77.05
82.81
84.77
MassivelntentCClassiccttto
75.58
78.06
79.29
MassivesenaaiooChssifttttton
79.16
81.35
81.64
MedraiiCChustering222
33.55
32.23
31.07
MedrwivClusteringS22
31.11
31.37
31.27
MindSmallKerankingg
31.96
31.34
31.50
NFCorpus
37.12
40.33
39.33
NQ
53.89
61.24
61.70
QuoraRetrieval
87.37
85.59
87.75
RedditClustering
53.02
61.10
60.24
ReddiiClusteriggP22
60.47
64.52
64.12
SCIDOCS
17.96
21.05
22.50
SICK-R
82.25
83.01
83.70
STS12
78.28
78.85
78.80
STS13
85,52
86.84
86.37
STS14
82.49
84.04
84.04
STS15
88.76
88.72
88.99
STS16
87.11
86.79
87.22
STS17
90.10
90.63
90.19
STS22
68.25
67.55
67.68
STSBenchmark
87.16
88.72
88.65
SciDocsRR
79. 79.23
84.03
83.80
SciFact
72.08
77.30
78.86
SprintDupiicaeeuuustiio
96.25
96.83
96.82
StackExchangeChsssrringg
63.04
67.98
70.73
StakkEchanggCCluseinng2ZZZ
34.01
33.20
34.50
StackOverflow DupQussions
49.61
51.02
54.41
SummEval
30.01
28.49
29.96
TRECCOVID
80.41
79,25
77.69
Touche2020
22.31
16.92
22.18
"bxiiCCanveraattnnlllsiiaaaatt IweetSeniieetkkktaaiinnslsssiiit
69.92
71.01
69.26
60.76
61.11
62.14
"weetty\lwwgrrouusCCuuseeenn
49.37
51.04
52.18
Evil2er5SemEva22015
76.14
80.70
80.60
TwitterURI.Corpin
86.23
86.56
86.56
Average
61.85
64.14
64.80
Table 12: Supervised results of LLM2Vec (only Bi + MNTP) models on MTEB,
28


Reviews:
[{"review_id": "SfMqob0vIT", "reviewer": "Reviewer_pAEP", "summary": "This paper introduces a method to transform any decoder-only LLM into a bidirectional encoder.  The paper evaluates the method on three different LLMs and results show the effectiveness of the method.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to thank the reviewer for their positive feedback as they find our paper simple to understand, providing an effective method, significant experiments, and clear analysis. We address the concerns raised by the reviewer below.\n\n**Using other prompts**: Springer et al. (2024) have shown that LLMs for text embeddings are robust to minor prompt variations. To ensure a fair comparison between our method and the baselines, we did perform our experiments using the best performing instructions which are also used in previous work (Echo-embeddings, E5, and GritLM) for all models. \n\nAdditionally, we conducted an experiment to quantify the change in performance resulting from prompt variations. We compare the evaluation results for two sets of instructions – the ones we use in our paper (referred to as E5 inst) and those used in Su et al. (2022) (referred to as Instructor inst). In general, E5 inst are more detailed than Instructor inst. We ran the evaluation on the subset of 15 MTEB tasks described in Appendix C.1. We compare our best unsupervised approach without any contrastive learning (Bi + MNTP + Mean)  with Echo. \n\nWe observe that for both approaches, the performance drops by 2-3% when switching from E5 inst to Instructor inst. So as expected both approaches exhibit highly similar behavior when changing prompts (this is expected as they use the same underlying LLM). Finally, we want to emphasize again that the prompts we are using in the paper (E5 inst) are the currently best performing prompts in the literature. We will make sure to include these results in the final version.\n\n**Ablations for MNTP**: We would like to point out that the ablation the reviewer is asking for is already provided in the paper. Table 5 in the appendix shows the improvements gained by each step, including MNTP. To be more precise, the impact of MNTP can be observed by comparing Bi vs. Bi+MNTP and Bi+SimCSE vs. Bi+MNTP+SimCSE models when mean pooling is applied. For instance, applying MNTP before SimCSE improves Llama-2’s performance by 34% in the unsupervised setting.\n\nWe kindly ask the reviewer to elaborate more on their third point so that we can address their concern.\n\nWe hope that the reviewer considers revising their score based on our answer and we remain available for further clarifications during the discussion period.\n\nSu et al. (2022) - One Embedder, Any Task: Instruction-Finetuned Text Embeddings\nSpringer et al. (2024) - Repetition Improves Language Model Embeddings", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The method is simple to understand and the writing is clear. The results of the experiments are significant. The analysis is  also clear.", "reasons_to_reject": "However, there are some disadvantages in this paper.\n\n1. Authors claim that the encoder building by decoder-only LLM can generate rich contextualized representations for NLP tasks. Although experiments compare the encoder with LLMs, LLMs may perform better with other prompts. It is better to compare models with other prompts.\n\n2. The method is described clearly in this paper while the method is similar to MLM from BERT and other models. Authors should give some ablations to show the effectiveness of MNTP.\n\n3. As discussed above, LLMs can label input sequences by generating sequences of labels. Author should make it clear that LLMs labels input sequences by generating or directly labeling tokens.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WzjLXTnPiN", "reviewer": "Reviewer_1qid", "summary": "This paper introduces an unsupervised approach to convert decoder-only LLMs into robust text encoders. This approach, named LLM2Vec, is structured around three main steps: enabling bidirectional attention, masked next token prediction (MNTP), and unsupervised contrastive learning. The authors claim that LLM2Vec outperforms existing encoder-only models on various NLP tasks and establishes new benchmarks on the Massive Text Embeddings Benchmark (MTEB) using publicly available data.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and are happy to see that they found that our paper provides extensive empirical results with data- and parameter-efficient methods which is particularly relevant for practical aspects. We address the reviewers' concerns below.\n\n**Innovation and originality**: We believe that showing how to combine already-established methods in a novel setting, i.e., the application of LLMs for general text representations, is a novel contribution. We will follow the reviewers’ suggestion and will update the final version with more discussion on the adaptations we made to make these methods work in our setting. For instance, we substituted the masked token prediction of MLM with masked next token prediction due to the nature of auto-regressive language models, i.e., predicting the next instead of the current token.\n\n**Fairness in comparative analysis**: MNTP is specifically designed for bidirectional models and to adapt them to make use of future tokens, which is why we do not use it in the unidirectional case. While in theory MNTP could be applied to unidirectional models, it would still suffer from the limitation of only having access to the past, and therefore we would not expect it to provide any benefit over the original LM objective. That said, we already provide additional configurations (Uni and Uni+SimCSE for three different pooling methods) for unidirectional approaches in Table 5 in the Appendix, where we also ablate the importance of each component of LLM2Vec recipe.\n\n**Regarding the question about interpretability**: Existing LM interpretability methods for input attribution or for analyzing attention patterns could be applied on top of LLM2Vec transformed models. However, this is beyond the scope of our paper and we leave such investigations for future work. That said, section 4 of our paper already provides an extensive analysis of how bidirectionality and the LLM2Vec recipe affect the representations of LLMs.  \n\nWe hope that our response is sufficient to address the concerns raised by the reviewer and that they consider revisiting their score in light of our clarifications. We remain available for discussions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors provide extensive empirical evidence demonstrating that LLM2Vec enhances the performance of LLMs on both word-level and sequence-level embedding tasks, surpassing existing state-of-the-art models.\n\n2. LLM2Vec is described as data- and parameter-efficient, which is particularly important for practical applications where computational resources may be limited.", "reasons_to_reject": "1. **Innovation and Originality**: While the paper skillfully integrates several established techniques—bidirectional attention, masked next token prediction (MNTP), and unsupervised contrastive learning—into the LLM2Vec framework, it does not introduce fundamentally new methodologies. The core contribution lies in the novel application and combination of these existing methods rather than in the development of new techniques. This approach might raise questions about the paper's novelty within the field, particularly in a research landscape that highly values groundbreaking innovations. A more detailed discussion on the unique configurations or optimizations made to these existing techniques within LLM2Vec could enhance the perceived novelty of the work.\n\n2. **Fairness in Comparative Analysis**: The paper implements a transformation from unidirectional to bidirectional attention in decoder-only LLMs to enhance text embedding capabilities. However, the comparative analysis appears limited as the baseline configurations in the unidirectional setting (labeled as Uni+Mean) do not include enhancements comparable to those applied in the bidirectional setting (Bi+MNTP+SimCSE+Mean). A more equitable comparison would involve aligning the enhancements across both settings, for instance, comparing Uni+MNTP+SimCSE+Mean with Bi+MNTP+SimCSE+Mean. This would provide a clearer assessment of the impact of changing the attention mechanism independently of other variables.", "questions_to_authors": "How does LLM2Vec impact the interpretability of the embeddings? Given that the model transformations introduce changes in how tokens are encoded, does this affect the ability to trace back and understand decisions made by the model?", "extra_scores": {}}, {"review_id": "sV5B3Qa571", "reviewer": "Reviewer_ihtZ", "summary": "The article presents LLM2Vec, a novel technique to transform large, decoder-only language models into efficient text encoders without the need for expensive adaptation or the creation of synthetic GPT-4 data.      \nThis approach consists of three simple steps: enabling bidirectional attention, masked next token prediction (MNTP), and unsupervised contrastive learning.      \nThe authors evaluate the effectiveness of LLM2Vec on various English-language word- and sequence-level tasks.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive assessment of our work. We are happy to see that the reviewer points out the simplicity of our proposed approach as well as its strong performance. \n\n**Regarding the language limitation**: We want to point out that we are aware of this limitation and also discuss it in the limitation section of our paper (see Appendix A on page 16). We believe that our paper, as is, makes a strong contribution to LLM-based text representation and given the simplicity of our approach, we are hopeful that it can be extended to more languages. In fact, we are already working on this.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "**Innovation**: LLM2Vec introduces a new methodology to leverage pre-trained decoder-only language models, paving the way for better text embedding performance.    In particular, MNTP task, well adapted to the encoder and decoder.\n\n **Simplicity**: The approach is straightforward with only three steps, making it easy to replicate and understand.\n\n**Performance**: The method demonstrates superior performance in unsupervised settings, as well as in comparisons with supervised contrastive learning, establishing new state-of-the-art results.", "reasons_to_reject": "**Language Limitation**: The evaluation is performed solely on English datasets, which does not guarantee that the method will generalize across other languages.", "questions_to_authors": "It would be beneficial if future work could address the transferability of the LLM2Vec method to models trained in other languages. This would strengthen the generalizability of the approach.", "extra_scores": {}}, {"review_id": "7TwRFJViL9", "reviewer": "Reviewer_cJbi", "summary": "This paper studies how to adapt LLMs to embedding models. The proposed method converts an pre-trained auto-regressive language models by changing causal attention into bidirectional attention, and then finetune the model with masked next token prediction and unsupervised contrastive learning. Experimental results show that the proposed method achieves strong performance under both unsupervised and supervised settings.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their efforts and for pointing out that our paper provides an easy-to-follow method which leads to strong empirical results. We address the concerns raised by the reviewer below.\n\n**Not a secret**: The reviewer is correct in pointing out that the E5-Mistral model trained by Wang et al. (2024) is also based on a decoder-only LLM. However, there are a few important points worth highlighting. First, part of our motivation for saying “secretly” in the title is the strong performance of our method in the unsupervised setting and the fact that only little adaptation is needed to convert decoder-only LLMs into strong embedders. This hasn’t been extensively explored yet and the simplicity of our approach shows that you don’t necessarily need supervised contrastive fine-tuning to convert LLMs into strong embedding models. Second, the paper by Wang et al. 2024 is in fact concurrent work to ours (it was released less than 3 months before the COLM deadline). That said, we are open to suggestions for how to update our title. \n\n**Regarding the improvements in the supervised setting**: We highlight that the MTEB scores are average scores across 56 tasks. Therefore, even a small improvement corresponds to improvement over various tasks in different categories, as shown in category-detailed results.  In Table 1 and 2. Stated differently, this is a property of the MTEB benchmark. To provide evidence for this, we computed the performance improvement for each of the top-10 models on the MTEB leaderboard provided in Table 9 in the Appendix. It can be seen that each entry reports a “small” improvement over the previous approach and LLM2Vec also follows this pattern: 10) text-embedding-3-large (64.59) – +0.077% – 9) UAE-Large-V1 (64.64) – +0.062% – 8) mxbai-embed-large-v1 (64.68) – + 0.0% – 7) echo-mistral-7b-instruct-lasttoken (64.68) – +0.186% – 6) LLM2Vec (Bi+MNTP+Mean) (64.80) – +1.327% – 5) GritLM-8x7B (65.66) – +1.477% – 4) e5-mistral-7b-instruct (66.63) – +0.195% – 3) GritLM-7B (66.76) – +0.554% – 2) voyage-lite-02-instruct (67.13) – +0.641% – 1) SFR-Embedding-Mistral (67.56). Note that the improvements from rank 5) onwards are expected to be bigger compared to models on ranks 10) to 6) as these models rely on synthetic data for supervised fine-tuning.  \n\nWe hope that we could clarify the issues raised by the reviewer and remain available for further clarifications.   \n\nWang et al. (2024) - Improving Text Embeddings with Large Language Models", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper presents a easy-to-follow recipe to get strong embedding models from pre-trained LLMs.\n2. The empirical results are strong.", "reasons_to_reject": "1. It can hardly be considered a secret that LLM can be strong embedding models, as previous works like E5-mistral-instruct have already demonstrated similar findings.\n2. Under supervised setting, the proposed bidirectional attention and unsupervised training provides little improvements over the replicated E5-mistral-instruct model.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces LLM2Vec, an unsupervised method to transform decoder-only LLMs into powerful text encoders. LLM2Vec employs three core steps: enabling bidirectional attention, predicting masked next tokens, and applying contrastive learning without labels. The authors demonstrate this method's effectiveness on various word-level and sequence-level tasks.

Pros:
- The method is easy to understand, and the paper is well-structured and readable.
- The authors provide substantial empirical evidence showcasing the model's superior performance.

Cons:
- Reviewers primarily criticized the paper's novelty. It combines well-known techniques—bidirectional attention, masked next token prediction (MNTP), and unsupervised contrastive learning.

I personally believe that integrating established techniques in a new framework is an excellent approach to creating a simple yet effective method.
Decision: Accept