Repetition Improves Language Model Embeddings
Authors: Jacob Mitchell Springer
Suhas Kotha
Daniel Fried
Graham Neubig
Aditi Raghunathan
Venue: ICLR 2025 Poster (2025)
Keywords: embeddings
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: Repeat the input twice for high quality embeddings

Abstract: Bidirectional models are considered essential for strong text embeddings. Recent approaches to adapt autoregressive language models (LMs) into strong text embedding models have largely had the requirement to modify the LM architecture to be bidirectional. We challenge this premise by introducing ``echo embeddings'' which converts autoregressive LMs into high quality text embedding models \emph{without} changing the architecture or requiring fine-tuning. By repeating the input and extracting embeddings from the repeated tokens—which have access to all original tokens—echo embeddings improve over classical LM embeddings by over 5\% in zero-shot settings. Our zero-shot embeddings nearly match those obtained by bidirectionally-converted LMs that undergo additional masked-language modeling training. Echo embeddings are also compatible with supervised fine-tuning, matching or outperforming bidirectionally-converted LMs in an apples-to-apples comparison, even with an identical compute budget during training and inference. Overall, repetition is a simple and effective strategy to circumvent the need for bidirectional attention in embedding models, paving the way towards a unified architecture for all NLP tasks.

Published as a conference paper at ICLR 2025
REPETITION IMPROVES LANGUAGE MODEL EMBEDDINGS
Jacob Mitchell Springer Aditi Raghunathan Camegie Mellon University
Suhas Kotha
Daniel Fried Graham Neubig
ABSTRACT
Bidirectional models are considered essential for strong text embeddings. Recent approaches to adapt autoregressive language models (LMs) into strong text em- bedding models have largely had the requirement to modify the LM architecture be bidirectional. We challenge this premise by introducing "echo embeddings" which converts autoregressive L.Ms into high quality text embedding models with- out changing the architecture or requiring fine-tuning, By repeating the input and extracting embeddings from the repeated tokens-which have access to all orig- inal tokens echo embeddings improve over classical LM embeddings by over 5% in zero-shot settings. Our zero-shot embeddings nearly match those obtained by bidirectionally-coneers
LMs that undergo additional masked- lankkugg mod- eling training. Echo embeddings are also compatible with supervised fine-tuning, matching or outperforming bidirectionally-convees comparison, with identical compute budget during training and inference. Overall, repetition is a simple and effective strategy to circumvent the need for bidirectional attention in embedding models, paving the way towards unified ar- chitecture for all NLP tasks.
LMs an apples-to-apples
INTRODUCTION
Neural text embeddings have crucial role in modern approaches to information retrieval (IR), semantic similarity estimation, classifica- tion, and clustering (Ni al.) (202b;; Muen- nighoff et al.] 2022). For example, document retrieval often leverages low-dimensional em- beddings for efficient lookup: when queries and documents are encoded as vectors where se- mantic relationships are described by similar- ity in some metric space, a query lookup can be reduced an approximate nearest-neighbor search in embedding space (Johnson et al. 2019 Vanderkam al. 2013).
Autoregressive embeddings do not encode context from later tokens
classical
embeddings
Repetition enables embeddings to encode context
from later tokens
echo embeddings
Figure Overview of echo embeddings.
Until recently, the dominant pretrained language model paradigm for neural embeddings have been masked language models with bidirectional attention (Ni al. 2021a Raffel et al. 2020 Izac-
ard et al. 2021 Wang et al., 2022; Jiang et al., 2022 [Su et al. 2022] Xiao et 2023a Li et al.] 2023). Very recent literature has made progress generating high quality embeddings from au- toregressive language models. However, these approaches either require large-scale synthetic data (Wang et al. (023) or for the architecture of the base model to be "cast" to have bidirectional atten- tion SrehnamGhader et al. 2024 Muennighoff et al. 2024). The requirement to change the archi- tecture prevents the extraction of high quality embeddings without additional training. Further, this requirement is an obstacle to the development of truly unified models that generate text and pro- duce embeddings simultaneously without changing the architecture between encoding text for em- beddings and generating tokens.

---

Published as a conference paper at ICLR 2025
Is bidirectional attention crucial for high quality embeddings? Intuitively, autoregressive models fail to capture information across the entire imput due to the causal attention mask. We show this concretely in toy setup in Section B In this work, present surprisingly effective way to overcome this limitation withoul requiring bidirectional attention or fine-tuning.
Our method, "echo embeddings" is simple: we prompt the model with the input twice and extract embeddings from the second occurrence of the input. Even with causal attention, the embeddings of the second occurrence can access the entire input in the first occurrence. The second key ingredient is to prompt the language model to repeat the input (or, for example, to fix, rephrase, or otherwise reconstruct the input). This simple change enables the model to extract bidirectional information. As a result, echo embeddings offer strong performance in the entirely zero-shot setting-without any additional fine-tuning.
without
In comparison with the classical method of extracting embeddings, the zero-shot performance of echo embeddings around 5% higher. This nearly matches the performance of the recent more expensive method of casting to bidirectional attention followed with an addititional unsupervised MLM training (BehnamGhader et al., 2024). Even in comparison to the strong zero-shot baseline of PromptEOL. (Jiang et al.] 2023b) method in which the model is prompted to summarize the input in a single token echo embeddings outperform by a significant margin, and is less sensitive to the exact choice of prompt. We emphasize that echo embeddings are simple, cheap, and flexible.
The current best embeddings still require additional fine-tuning with a supervised objective. We find that echo embeddings slightly outperform the classical approach to extracting embeddings, even when the architecture is cast to be bidirectional, and even when echo embeddings is given an identical compute budget. We speculate that these gains are because echo embeddings better preserve and leverage large-scale pretraining because there are no architectural changes. This is in contrast to the bidirectional relaxation approach, which requires the model to adjust its weights to perform well with bidirectional attention.
Echo embeddings come with a lot of benefits: simple to apply, works zero-shot without additional training, allows a unified model to be used for various tasks, and offers (slightly) better performance than casting to bidirectional attention offers. One apparent drawback that repetition doubles the compute cost. To account for this, in all our experiments, we report "compute matched" results for both inference time and training time (when applicable). While the gains are now slightly smaller, we see that echo embeddings still perform well. In total, we believe that the echo embedding ap- proach, by virtue of its simplicity and effectiveness, is an important step in bridging the gap between traditionally bidirectional embedding models and the current predominant paradigm of autoregres- sive language models.
2 PRELIMINARIES
Our goal is to extract text embeddings that map sentence x to vector (xx) E I2 such that the se- mantic similarity between sentences is captured as similarity between their embeddings. In practice, we use the cosine similarity between embeddings as a similarity metric (detailed in Appendix C).
Embeddings from language models. We are primarily interested in the embeddings extracted from autoregressive language models, which typically have causal attention masking and are trained on a next-token objective. For brevity, we drop the term "autoregressive" in the following.
As is standard, we extract embeddings from the activations of the final hidden layer. Each input token zj at position j is associated with a contextualized token embedding which is the hidden layer representation Φ (x)). We can pool the embeddings across all the tokens in different ways. this work, we focus on two common strategies which have been considered by prior work (Reimers & Gurevych 2019 Muennighoff 2022 Zhang et al. 2023a Wang et al. 2023).
Pooling. A mean token embedding over a set of indices A, refers the mean contextualized token embeddings at indices in A (A(Z) = L Erea (x(2). A last-token embedding refers to the contextualized token embedding of the last token in the input sequence, written (=)
2

---

Published as a conference paper at ICLR 2025
A.
All toy data (S1+ +-2)
Clas +
Clas
Sim(g ) Simig.
Classical
10
Accuracy
Accuracy
Density
Structure 2
0.5
00
3.0
Clas. Mean
Clas.
Echo
51
52
51
52 51
52
0.70
0.76
0..2
0.88
0.94
1.00
Last
Sim (cosine similarity)
Figure 2: A. Accuracy of classical embeddings (mean and last-token pooling) and echo embeddings on the entire toy dataset. B. Accuracies of classical and echo embeddings split by the different structures of the data (SI and S2, see text). C. We take the echo embeddings of only the first parts of query q = [A, B] and sentences s , and plot the distribution of cosine similarities, showing that echo embeddings encode later information in earlier tokens.
Classical embeddings. Traditionally, embeddings are computed by simply passing the sentence to the model and extracting some pooling (e.g. mean or last-token) of the contextualized embeddings corresponding to the input sentence. We will refer to embeddings created in this way as "classical embeddings". Additionally, one might first prompt the language model with an explanation of the task of interest followed by the sentence (Su et al. 2022).
ECHO EMBEDDINGS FROM LLMS
In this section, describe general methodology to extract high quality embeddings from autore- gressive language models without bidirectional attention. We start with simple synthetic dataset that demonstrates why causal attention might inhibit embeddings from capturing information reli- ably across the entire context (Section 3.T). We then present our proposed method of Echo Em- beddings in Section 3.2, and test that our simple and intuitive approach fact addresses the failure mode identified in the synthetic setup without introducing bidirectional attention.
3.1 FAILURE OF CLASSICAL EMBEDDINGS WITH CAUSAL ATTENTION
High-quality embeddings should effectively capture various text structures, including where simi- larity between inputs is strongly affected by either the early parts the later parts of the inputs. We create dataset with two "opposite" structures (S1 and S2) and show the classical embeddings with causal attention cannot capture both simultaneously.
The dataset we construct consists of a query and a similar sentence s and a dissimilar sentence 8T. We measure the performance of embedding Φ(-) by computing the fraction where Sim st) > Sim(g. s Each query and sentence is a concatenation of two strings, allowing us to vary whether early (or later) parts are similar (or discriminatory). Notationally, A, At, B, B are semantically similar, while A, A- B, B- semantically dissimilar. We query GPT-4 to generate these various realistic but synthetic strings (see Section C.
S1 (Early discriminatory; late redundant). For query [A, B). in this structure, the early tokens provide information to discriminate between 8t 8 while the later tokens are all similar. In other words, s [A+, B+] and s- [A=, B). where the discriminatory parts are bolded for convenience.
S2 (Early redundant; late discriminatory). Analogously, consider second "opposite" struc- ture where the early tokens are all similar but the later tokens can discriminate between st, s For query q [A, B), have s [A+,B+] and s [A,B B where the discriminatory parts bolded for convenience.
We show the results of zero-shot classical embeddings extracted from Figure P (A) on dataset that has mixture both structures. We see that neither mean nor last token pooling allows classical embeddings to achieve beatero performance.
in
3

---

Published as a conference paper at ICLR 2025
To get some insight into the failure, plot the performance each substructure separately in Fig- ure 2 (B). Last-token pooling disproportionately weights the later parts of the sentence thereby missing discriminatory information in the early tokens in S1. In general, last-token pooling does not work well in a zero-shot manner on real data as well, as we see in the next section.
Mean-token pooling is better suited the zero-shot setting and allow us extract information from all parts of the input. This works well structure S1. However, the causal attention produces different issue: the contextual embedding at position k, dhx (x) cannot encode information about the later tokens 2R+1, 2h+22 This causes mean-token embeddings to exaggerate early token similarity and dilute discriminatory information in the later parts because of causal atfention. This effect is especially exacerbated when the early tokens are similar, as in S2. In total, classical embeddings fail when the dataset contains inputs of both instances- where the discriminatory infor- mation could be in the early or later tokens. We show that this failure mode of classical embeddings can be overcome in a completely zero-shot manner using simple method which we describe next.
3.2 ECHO EMBEDDINGS: A GENERAL FRAMEWORK
We saw in Section 3.1 that while mean-token pooling aggregates information across all tokens, the contextualized representations of early tokens are suboptimal because they cannot attend to the later tokens due to the causal attention. Replacing the causal with bidirectional attention is a natural solution, and common intuition seems to be that this is necessary. We propose a new general embedding framework-eche embeddings-that captures contextualized information from all tokens in the input with only causal attention:
Echo embeddings method: Prompt the language model to act as autoencoder, e.g., by asking the model to frepeat, rephrase, fix, } the input; feed the sentence æ to the language model twice; pool the contextualized embeddings of the second occurrence of x.
Ex: "Rewrite the sentence: rewritten sentence: x"; embeddings are pooled from the bolded x.
The second occurrence can attend to everything, and hence extracting contextualized representations from the second occurrence could capture information from the entire input. Furthermore, in order to encourage the second occurrence to actually "encode" information about the first, we instruct the language model to perform a generic task that requires using this information, e.g., "rewrite" or "repeat". The exact prompt, template, task used to extract embeddings can be varied to suit the model and downstream task. We find empirically that the performance of echo embeddings are not particularly sensitive to the exact phrasing of the prompt (Section 4.2). We provide a list of example prompts in Appendix C
While echo embeddings is simple and intuitive, does it actually effectively capture bidirectional information?
We apply echo embeddings with mean-token pooling the synthetic dataset introduced in Sec- tion 3 and present the results in F Echo embeddings with mean-token pooling works signif- icantly better than classical embeddings and works well on both structures in the whether the discriminatory information is in the early or later tokens.
In particular, let focus on structure $2 where the discriminatory information appears in the later tokens. Echo embeddings with mean-token pooling work well, while classical embeddings did not because the early tokens could not attend to the later ones. This implies that echo embeddings can successfully address this failure and the representations of early tokens can capture information "bidirectionally" about the entire input without changing causal bidirectional attention. To test this further, we measure whether echo embeddings can meaningfully distinguish st and s- using just the information in the A portions despite the discriminatory information solely appearing in the later B- parts. Recall that echo embeddings extract embeddings from the second occurrence. We plot the cosine similarities Sim(q. s*) and Sim (q. s in Figure (C). We find that Sim s*) is typically larger than Sim(g, s ). Since any distinction between 8 s must come from the echo embeddings of A capturing information from the later parts of the sentence.
only pooling the echo embeddings of the A portion,
4

---

Published as a conference paper at ICLR 2025
Table 1: Main results in the zero-shot setting: Apples-to-apples comparison of zero-shot echo and classical embeddings with mean pooling, and other baselines on MTEB (56 datasets). For the autore- gressive embeddings in this table, we extract embeddings from the Mistral-7B-Ii BB-Isstrucct..... model. Bolding only zero-shot models.
Categories
Clas.
Clus.
P. Cls.
Rera.
Retr.
STS
Su.
Average
# of datasets
4
10
Echo (Ours)
72.01
33.80
72.41
47.56
20.82
73.74
30.70
48.64
Echo (compute matched)
71.63
33.51
72.31
47.43
22.85
73.64
31.02
49.02
Classical
64.87
34.14
57.71
43.60
18.26
57.07
27.02
42.38
PromptEOL
73.84
27.53
55.47
48.44
13.18
67.14
28.33
43.69
BERT
61 1,.47
29.48
56,33
43,44
10,44
52.98
29.82
37.87
RoBERTa
62.63
29.05
56.95
41.92
8.62
55.24
28.64
37.86
M.L22 (requires fine tuning)
72.51
40.06
70.95
50.43
19.74
71.90
27.84
49.43
Summary. We construct synthetic dataset that showcases failure mode in extracting embed- dings from causal language models in the classical manner. While ours is a simplistic construction, we demonstrate that the failure mode we outline does oceur on real data as well in Appendix C and could explain the poor performance of classical embeddings in practice. However, echo embed- dings, where we repeat the input and take embeddings from the second occurrence, can successfully address this failure mode. We now test echo embeddings variety of real-world settings.
EXPERIMENTS
In this section, we describe how we implement and evaluate echo embeddings on real data.
4.1 MASSIVE TEXT EMBEDDING BENCHMARK
MTEB & MTEB-MINI. Our main evaluation dataset is the English-language subset of the Mas- sive Text Embedding Benchmark (MTEB) (Muennighoff' al. 2022). MTEB collection of 56 datasets that are grouped into different embedding tasks: classification, clustering, pair classifica- tion, reranking, retrieval, sentence similarity (STS), and summarization, with the goal of evaluating embeddings broadly. However, the name suggests, MTEB is massive, and requires multiple days to evaluate a 7B-parameter model on 8 A 100 GPUs. Thus, we adopt a smaller 28-dataset subset of MTEB that spans all categories except summarization, which we call MTEB-MINI We provide detailed description of MTEB and MTEB-MING in Appendix A.I
4.2 EVALUATION OF ZERO-SHOT EMBEDDINGS
We begin with the zero-shot setting. where embeddings are extracted from pre-trained model with- additional training. Our goal is show that without any additional training, echo embeddings outperform prior entirely-zero-shot methods, and can perform similarly to unsupervised approaches (which require additional fine-tuning with unsupervised data), such as LLM2 VeMMueee-unsueevvieee (BehnamGhader et al. 2024). While supervised fine-tuning is often used to improve embedders, zero-shot embeddings have the advantage that no further (potentially expensive) training is required.
Constructing zero-shot echo embeddings. Following the echo embeddings method described in Section to construct echo embeddings we need to pick prompt, a pooling strategy, and language model. We evaluate the impact of these choices the performance of the embeddings on the Massive Text Embedding Benchmark (MTEB) (Muennighoff et al. 2022).
Choosing prompt: As specified in Section 3 must prompt the language model to act as an autoencoder. Sclar et al. (2023) argues that on many tasks, the exact formatting and wording of the prompt can strongly influence performance. Thus, we investigate: does the choice of prompt affect embeddings quality? As we shall discuss later, we find that, the choice prompt does not substantially affect the performance of echo embeddings (Section 4.3). This implies that prompt
5

---

Published as a conference paper at ICLR 2025
Table 2: Role of scale and base model in the zero- shot setting: Average MTEB score (56 datasets) for LLaMA-7B-2-Inste and S-LLaMA-1.33
Table 3: Casting bidirectional atten-
tion: We evaluate MTEB-MINI (28 datasets) and compare classical with causal attention (CI.+Uni) and with bidirectional attention (CI.+Bi).
Model
Echo
Classical
PromptEOL
-MA-1.33
40.45
34.31 40.29
37.50
LLaMA-7B
45.84
42.84
Model
Echo
Cl.+Uni
Cl++Bi
LLaMA-2-7B
56,99
47.27
43.03
S-LLaMA-1.38
49.55
40.14
35.77
Mistral-7B
59.78
49.03
58.24
tuning is unnecessary, and any reasonable prompt will perform well. For our main results, we adopt the prompt Rtewrite the following paragraph: S. The rewri tten paragraph: 5" for echo embeddings and "Write a paragraph: S" for classical embeddings.
Choosing pooling strategy: We evaluate both mean and last- token pooling for both classical and echo embeddings 4 Con- sistent with our toy-data analysis in Section we find that mean token pooling is required for good embeddings and thus use this for further zero-shot experiments (Table
Table 4: Role of pooling in the zero-shot setting: Average MTEB Score (56 datasets) for mean and last token pooling.
Choosing language model: We evaluate the performance of echo embeddings on three language models: Mistral-7B- Instruct-v0.1 (Jiang et al. 2023a). ALLMM-2-7B--nstru (Tou- vron et al. (023a)), and S-LLaMA-L.3B (Xia et al.) (023) (sometimes abbreviated). We select the instruction-fiettunee
Mean
Last-token
Echo Classical
48.64
31.55
42.38
31.94
model for each of them. Refer to Appendix A.2 for additional information on the base models.
Compute-matching echo embeddings. Naively, echo embed- dings would require twice the compute of classical embed- dings. To account for this, we also consider compute-matched echo embeddings in which we halve the length of the input.
0.6
Score (M (MTEB-min) mini)
Alternative approaches to constructing embeddings from autoregressive models. In addition our results, we com- pare to two recently proposed methods to convert autore- gressive models into embedders without supervised fine- tuning: PromptEOL (zero-shot) and the unsupervised variant of LLM2Vec (requires additional fine-tuning). PromptEOL. prompts the model to summarize the input in a single token, and extracts embeddings from the predicted token (Jiang et al.) 2023b) (see Appendix D for additional details). The unsu- pervised variant of LLM2 Vec LLLMMVec-unsupervisee incor- porates additional musked-language--me modeling fine-tuning step. Importantly, LLM2Vec is not zero-shot because it in- volves this extra fine-tuning step.
0.2
0.0
Classical
Echo
PromptEOL
Figure 3: Role of prompt in the zero-shot setting: Variance of zero- shot performance across different prompts and formats on MTEB- MINI (28 datasets). Refer to Ap- pendex D for details on the exact prompts and formats.
4.3 ZERO-SHOT RESULTS
Echo embeddings yield strong embeddings, and without
additional compute. We present the main zero-shot results in which we evaluate echo embeddings on MTEB in [ We find that echo embeddings successfully converts causal language models into strong encoders. For Mistral-7B-Instruett we find, surprisingly, that compute-matcher echo em- beddings, achieves slightly higher performance than full-compute echo embeddings. The difference small enough that unclear to what degree we would observe this improvement in other set- be adapted to outperform classical embeddings with-
tings, but suggests that echo embeddings out any additional compute. We extend this analysis compute in Section 4.6, We also evaluate echo embeddings multiple models and scales in Table and find consistent results.

---

Published as a conference paper at ICLR 2025
Table 5: Main fine-tuning results: We report MTEB scores for echo embeddings with full compute and compute-matched and classical embeddings with causal and with casted bidirectional archi- tecture. For echo embeddings, we use Mistral-7B as the backbone model, and last-token pooling. For the compute matched echo embeddings, FT+IT refers to both fine-tuning and inference-time compute matching.
Categories
Clas.
Clus.
Cls.
Rera.
Retr.
STS
Su.
Average
# datasets
12
4
10
Echo (ours)
77.43
46.32
87.34
58.14
55,52
82.56
30.73
64.68
Echo (FT+IT compute matched)
77.39
46.27
87.49
58.08
55.07
83.18
30.73
64.66
Classical
76.57
45.78
86.37
56.71
54.87
82.03
31.02
63.98
Classical bidirectional attn.
76.70
45.94
88.15
57.23
54.96
82.42
29.32
64.23
Comparison to PromptEOL. We find that echo embeddings outperform PromptEOL on average across the tested MTEB datasets by 5%. In addition to this improvement, echo embeddings is additionally much less sensitive to the exact wording and formatting the prompt (Figure 3).
Comparison to prior encoder models. We evaluate the zero-shot MTEB performance of exist- ing bidirectional (encoder) models, BERT-large and RoBERTa-large, which are often used as the backbone of fine-tuned embedders. We find that echo embeddings outperform both BERT-large and RoBERTa-large in the zero-shot setting. We suspect that the performance improvement is due to Mistral-7B being significantly more powerful than BERT and RoBERTa. This highlights the impor- tance of our method: by using echo embeddings, we can leverage the full power of autoregressive language models, which generally dominant over bidirectional models on other NLP tasks.
Comparison to LLM2Vec. We find that entirely zero-shot echo embeddings can perform similarly LLM2Vcc-unsupervieed despite the fact that LLMMVec-unsupervisee is not zero-shot method, and requires additional fine-tuning on top of the pre-trained base model. Echo embeddings thus offers simple alternative to LLM2Vec that does not require additional training.
Relaxing to bidirectional attention. We find that changing the attention mask from causal to bidi- rectional often substantially underperforms the causal architecture, but not for Mistral- (Table Regardless, echo embeddings consistently outperforms this baseline. A priori, we might expect that, without further fine-tuning, such a modification the architecture would harm performance, we would expect that the model has not been trained to accommodate the additional attention. For LLaMA-2-7B and S-LLaMA- 1.3B, this is the case. However, for Mistral-7B, casting to bidirec- tional attention does not harm performance and instead yields performance similar to echo embed- dings. We suspect that this may arise from ntn-standard pre-training methodology. as discussed by BehnamGhader al. (2024).
Ablations. We perform multiple ablations to evaluate the role of each component:
1. We test the role of pooling strategy in Table 4 Consistent with our analysis in Section find that mean pooling outperforms last-token pooling by a wide margin for both echo and classical embeddings. This suggests that mean pooling is required for echo embeddings in the zero-shot setting.
2. We evaluate the role of the exact wording and formatting of the prompt in Figure We find that the performance of echo embeddings has low variance with respect to the exact choice of prompt. This implies that echo embeddings does not require prompt tuning to perform well. Further, we find that all tested echo embeddings prompts outperform every tested classical embeddings prompt.
4.4 EVALUATION OF FINE-TUNED EMBEDDINGS
Echo embeddings also perform well in the fine-tuning setting, We will show that echo embeddings can outperform classical embeddings, even when the attention is converted to bidirectional. Fur- ther, echo embeddings match the performance of other methods that require much more compli- cated fine-tur teninunnng methodology, with identical fine-tuning and inference- seerence-time compute, and without modifying the architecture.
7

---

Published as a conference paper at ICLR 2025
Table 7: Comparisons recent baselines: MTEB score for models trained on similar data. Echo embeddings matches the performance of models trained on similar data, but without re- quiring bidirectional attention. FT+IT refers to the setting in which echo embeddings are compute- compute-matched matched at train and inference time.
Table 6: Role of base model in the fine-tuning setting: We compare echo and classical embed- dings for S-LLaMA-1.3B. FT+IT refers the setting in which echo embeddings are compute- matched at train and inference time.
Methods
MTEB Score
Method
MTEB Score
Echo
62.01
Echo (FT+IT compute matched)
61.65
Echo (FT+IT compute matched)
64.66
Classical
61.26
LLM2Vee-supervis
64.80
GritLM-7B (public data)
64.70
Constructing fine-tuned echo embeddings. To construct fine-tuned embeddings, must again choose a prompt, pooling strategy, and a language model. In addition, we need to choose a con- trastive learning objective and collection of training datasets.
Choosing a prompl: We adopt minimal prompt that differs for queries and documents. Prior work has shown that the addition of an instruction can improve the quality of the embed- dings (Wang et al., (202), and thus we include an instruction in the prompt. The prompts are: Echo (docs)
Echo (queries)
Classical (queries) Instruct: (instruction) Document: S
Classical (docs)
Instruct: (instruction)
Document: S
Query:
Document again: S
Query: S
Query again: 5"
We provide list of the instructions in Appendix
Choosing pooling strategy: We evaluate both mean and last-token pooling for both classical and echo embeddings. In practice, we find that last-token pooling performs slightly better than mean pooling in this setting (see Section 4.5)
Choosing language model: We train and evaluate the performance of echo embeddings on Mistral- IB-Instruct-v0O.1 and S-LLaMA-1.31 (Tiang et al.) [2023]] Xia et al., (023). We choose Mistral- 7B as the best-performing model in the zero-shot setting, and 1.LLMA-1.3BB to test the broad applicability of our method across model class and scale.
Training datasets and optimization; We train collection of publicly available datasets that are standard training datasets in the embedding literature. We list and describe each of the datasets in Appendix F To fine-tune the model, optimize the SimCSE loss with in-batch and mined hard negatives. Since this is standard, we defer discussion of this to Appendix Each batch is constructed of training dataset, and then collecting examples from only this dataset. We use GradCache to train with large batch size (2048) with limited GPU memory (Gao al. aa. We train with LoRA instead of full finetuning, with 16 and 16. We choose 1750 and a learning rate of 8 x 10-4
by sampling a dataset from
Compute-matching echo embeddings. Echo embeddings require twice the compute of classical em- beddings. Similar to the zero-shot setting, we reduce the inference-time cost of echo embeddings by halving the length of the input. However, in the training setting, we additionally must match the train compute. We opt to train for half as many steps as classical embeddings. this is the most straightforward way to match the compute. The compute-matched echo embeddings in Table 5 re- fer to the setting where we match compute for both training and inference.
Classical embeddings with bidirectional attention. In addition to our comparison to classical embeddings with causal attention, we train model for which we replace the causal attention with bidirectional attention.
4.5 FINE-TUNING RESULTS
Echo embeddings yield high quality embeddings the fine-tuning setting, We present the main fine-tuning results in Table Similar the zero-shot setting, mattute echo embeddings
8

---

Published as a conference paper at ICLR 2025
outperform classical embeddings with both architectures: causal and bidirectional. When additional compute available, echo embeddings improve performance further. This suggests that the funda- mental gap between classical and echo embeddings that we identified in Section perisssts even after fine-tuning. We find consistent results when evaluating across model classes and scales (Table 66.
Comparison to LLM2Vec. We compare to the supervised variant of Vec. The LLM2 Vec method relaxes the at- tention of the base autoregressive model to bidirectional and requires two fine-tuning stages: (1) fine-tuning with an un- supervised lankkeee-angggge--ee modeling objective (2) super- vised fine-tuning for embeddings. Despite the complexity of the L.M2Vec method and additional fine-tuning, echo embeddings nonetheless approximately matches the perfor- mance with identical fine-tuning and inference-time com- pute (Table 7 Echo embeddings thus offers itself much simpler method that does not require modifying the architecture nor additional fine-tuning stages, with nearly identical performance.
Echo
MTEB-min
Classical
Classical Bi
50
128
256
512
Budget (tokens)
Figure 4: Zero-shot setting: MTEB- mini score of zero-shot echo em- beddings and classical embeddings with bidirectional attention with fixed compute (token) budget. We the zero-shot Mistral 7B from Table I The standard token budget for MTEB is 512.
Comparison to GritLM. We also compare to GritLM MMuennighoff al. 2024) in Table This approach, which simultaneously trains embedder and generative model, finds the bidirectional attention relaxation to be crucial. We find that compute-matched echo embeddings perform sim- ilarly the variant of GritLM that has been trained on simi- lar data as ours (public data), even though echo embeddings use causal architecture.
The role of pooling, We investigate the role of pooling in Appendix the choice of pooling strategy has much weaker effect than in the zero-shot setting. Consistent with prior work (Wang al., 2023) ZZhng et al. (023a). we find that last-token pooling. with the addition of a trainable end-of-sentence token embedding, slightly outperforms mean pooling.
the fine-tuning setting,
4.6 ADDRESSING THE INFERENCE- -IME CoST OF ECHO EMBEDDINGS
One potential drawback of echo embeddings is the compu- tational cost of the method at both training and inference time. Naively, echo embeddings requires approximately twice the compute of classical embeddings, as the input is repeated. We have so far seen that in an entirely compute- matched training and inference time setting, echo embed- dings outperforms classical embeddings (Table (5). In this section, we expand this analysis to consider how echo em- beddings perform under different inference-time compute budgets. We explore the role of training under different budgets in Appendix
MTEB-mm mini score
72
71
Echo
Classical-Bi
70
512
Budget (tokens)
Figure 5: Fine-tuning setting: MTEB- mini score of fine-tuned echo em- beddings and classical embeddings with bidirectional attention with fixed compute (token) budget. We use the trained Mistral-7B models from Ta- ble 5 The standard token budget for MTEB is 512.
4.7 THE ROLE OF COMPUTE
The role of inference-time compute in the zero-shot set- ting. In Section 4.2 we demonstrated that by halving the number of input tokens that are encoded by echo embed- dings, thus matching compute with the classical approach to emebddings, we achieve stronger performance than clas- sical embeddings. Here, we investigate the role of compute in further detail by plotting the performance of echo embed- dings as a function of the total number of tokens encoded (Figure 4h. We find that when the budget is greater than approximately 64 tokens, echo embeddings outperforms Mistral-7B classical embeddings with bidirectional attention. For very small compu- tational budgets (fewer than 64 tokens), find that Mistral-7B classical embeddings with bidirec-

---

Published as a conference paper at ICLR 2025
tional outperform echo embeddings. Since the standard budget for MTEB is 512 tokens, echo em- beddings likely offer stronger performance in most settings. Over all compute budgets, echo em- beddings outperform causal attention classical embeddings.
The role of inference-time compute in the fine-tuning setting. The fine-tuning setting exhibits a similar trend: for very small computational budgets, echo embeddings perform worse than classical embeddings, but as the budget increases echo embeddings outperform classical embeddings (Fig- ure 5). In the fine-tuning setting, echo embeddings outperform classical embeddings for budgets greater than 128 tokens, which is still much smaller than the standard MTEB budget (512 tokens).
RELATED WORK
Sentence embeddings. Dense low-dimensional vectors representing textual semantics has been widely studied and applicd. Early approaches involved computing embeddings for individual words
(Hinton
1984 Rumelhart et al. 1986 Elman 1990 Mikolov et al.
[013 Pennington al. 2014).
Later work aims to compute dense vectors representing the semantics of entire sequences by com-
bining or composing word vectors (Le & Mikolov 2014 [yyer et al.
2015 Kiros et al. 2015 Socher
ct al. 2011 Tai et al. 2015 Wang et al. 2016 Wieting et al. 2015). Khattab & Zaharia (2020) pro-
pose use late interaction between document and query vectors to improve retrieval performance.
Reimers & Gurevych (2019) propose S-BERT which takes a pretrained BERT (Devlin et al. 2018)
and trains with a triplet loss. More recent approaches typically adopt this approach different pre- trained models (Raffel et al., 2020 Ni et al. 202 Raffel et al. 2020) and contrastive objectives (Oord et al., 2018: Gao et al., (021b). Other papers propose including prompts to improve task- specific embedding performance (Jiang et al. 2022 Su et al. 2022). Some work combines multiple of these training objectives and approaches (Xiao et al. 2023a Li al. 2023). Most recent sentence embeddings research has focused on improving finetuning. Reimers & Gurevych (2019) demon- strates that without finetuning, BERT has low-quality embeddings. To our knowledge, Jiang et al.. 2023b) is the only paper that constructs zero-shot embeddings for autoregressive language models.
Next-token language modeling for embeddings. A series of papers aim to construct high qual- ity embeddings from autoregressive large language models. Most closely related, [Behnam Ghader et al. (2024) proposed LLM2Vec which adds an additional unsupervised training step on a masked- language-modeling
Shjjetive.Muutipph papers apply the fine-tuning approach of S-BERT to language models but using a trained GPT (Radford et al. 2018) as the backbone architecture (Muennighoff
2022 Zhang et al. (023a). Ma (2023) adopts this approach but for LLaMA-2 (Touvron al.,
2023b). Diang et al. (2023b) extracts embeddings by asking a language model to summarize the in- put sentence, Wang et al. (2023) improves embeddings by adding synthetic training data and trains on Mistral (Jiang et al. (023a). Muennighoff et al. (2024) adds an additional objective to maintain
generation performace, but requires bidirectional attention. Li& & Li (003b) modifies the architec-
ture of causal language model to be bidirectional.
Language modeling with bidirectional information. Xu et al. 2023) finds that repetition can improve the performance of autoregressive language models for reasoning tasks. Arora et al. (2024) finds that repetition can be useful to improve recall for recurrent architectures.
6 CONCLUSION
We have demonstrated that the common assumption that bidirectional architectures are crucial for high quality embedding models is false. Our method, echo embeddings, prompts an autoregressive model to act as an autoecoder, which enables the model to encode bidirectional information. Our results highlight the importance of bidirectional information-we find that the inability to encode bidirectional information leads to the failure of classical-style embeddings in autoregressive models. Further, our method demonstrates that by carefully choosing the task that the language model is prompted to perform at inference time is promising approach for improving embeddings. We hope that our work will inspire future research towards understanding how inference-time changes can impact embeddings.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This material is based upon work supported by the National Science Foundation Graduate Research Fellowship under Grant No. DGE2140739. Any opinion, findings, and conclusions recommen- dations expressed in this material are those of the authors(s) and do not necessarily reflect the views of the National Science Foundation.
This research was supported by the Center for Al Safety Compute Cluster. Any opinions, findings, and conclusions recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the sponsors.
This work was supported in part by the A12050 program at Schmidt Sciences (Grant #G2264481).
We gratefully acknowledge the support of Apple.
REFERENCES
Simran Arora, Aman Timalsina, Aaryan Singhal, Benjamin Spector, Sabri Eyuboglu, Xinyi Zhao, Ashish Rao, Atri Rudra, and Christopher Ré. Just read twice: closing the recall gap for recurrent language models. arXiv preprint urXiv:2407.05-88 2024.
Payal Bajaj, Daniel Campos. Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Ma- jumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. Ms marco: A human generated machine reading comprehen- sion dataset, 2018.
Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapa- dos, and Siva Reddy. LLM2Vec: Large language models are secretly powerful text encoders. arXiv preprint, 2024. URL httthttss////xxi org/ http//b//2000 2404. 05961
DataCanary, hilfialkaff, Lili Jiang, Meg Risdal, Nikhil Dandekar, and tomtung. Quora question pairs, 2017. URL fttts:://kaggle.com//cooo com/ competi http://eee....coootooooooooo//aa
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.0480 2018.
Jeffrey L Elman. Finding structure in time. Cognitive science, 14(2):179-211, 1990.
Angela Fan, Yaeine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. Eli5:
Long form question answering, 2019.
Luyu Gao, Yunyi Zhang, Jiawei Han, and Jamie Callan, Scaling deep contrastive learning batch size under memory limited setup. arXiv preprint arXiv:2 100.0083,,, 2021a.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104. 08821, 2021b.
Geoffrey E Hinton. Distributed representations. 1984.
Mohit lyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daumé III. Deep unordered compo- sition rivals syntactic methods for text classification. In Proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference nat- ural language processing (volume 1: Long papers). pp. 1681-1691, 2015.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning.
arXiv preprint 2X.0:2//2.00/111 2021.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06225, 2023a.
Ting Jiang, Jian Jiao, Shaohan Huang. Zihan Zhang, Deqing Wang, Fuzhen Zhuang, Furu Wei, Haizhen Huang, Denvy Deng. and Qi Zhang. Promptbert: Improving bert sentence embeddings with prompts. arXiv preprint arXiv:2201.04433 2022.

---

Published as a conference paper at ICLR 2025
Ting Jiang, Shaohan Huang, Zhongzhi Luan, Deqing Wang, and Fuzhen Zhuang. Scaling sentence embeddings with large language models. arXiv preprint arXiv:2307.1664. 2023b.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. JEEE Transactions Big Data, 7(3):535-547, 2019.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen tau Yih. Dense passage retrieval for open-domain question answering, 2020.
Omar Khattab and Matei Zaharia. Colbert:
Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference research and development in Information Retrieval, pp. 39 48, 2020.
Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip-thought vectors. Advances in neural information processing systems, 28, 2015.
Quoe Le and Tomas Mikolov. Distributed representations sentences and documents. Interna- tional conference machine learning, pp. 1188-1196. PMLR, 2014.
Xianming Li and Jing Li. Angle-optimized text embeddings. arXiv preprint M.Xi::330..18777 2023a.
Xianming Li and Jing Li. Deelm: Dependency-enhancee large language model for sentence embed-
dings. arXiv preprini wrXiv:2311.05290 2023b.
Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.0328. 2023.
Xueguang Ma, Liang Wang, Nan Yang. Furu Wei, and Jimmy Lin. Fine-tuning Ilama for multi-stage text retrieval. arXiv preprint arXiv:23 10. 10.08319, 2023.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. arXiv preprint arXiv:1301.378) 2013.
Niklas Muennighoff. Sgpt: Gpt sentence embeddings for semantic search. arXiv preprint www.22202.88904, 2022.
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. Mteb: Massive text embed- ding benchmark. arXiv preprint arXiv:2210.07316, 2022. doi: 10:8550/AR31222200071.666
URL https: .rgg abs/ /210. 07316
Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. Generative representational instruction tuning. arXiv preprint arXiv:2402. 2024.
Jianmo Ni, Gustavo Hernández Ábrego, Noah Constant, Ji Ma, Keith B Hall, Daniel Cer, and Yin- fei Yang. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. arXiv preprint arXiv:2108. 08807777... 2021a.
Jianmo Ni, Chen Qu Jing Lu, Zhuyun Dai, Gustavo Hernández Ábrego, Ji Ma, Vincent Y Zhao, Yi Luan, Keith B Hall, Ming- Wei Chang, et al. Large dual encoders generalizable retrievers. 2021b.
arXiv preprint 2
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv: 2018
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 20/4 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543. 2014.
Yifu Qiu, Hongyu Li, Yingqi Qu, Ying Chen, Qiaoqiao She, Jing Liu, Hua Wu, and Haifeng Wang. Dureader-retrieval A large-scale chinese benchmark for passage retrieval from web search en- gine, 2022.
12

---

Published as a conference paper at ICLR 2025
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language under- standing by generative pre- tre-training. 2018.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu, Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-5551 2020.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. arXiv preprini arXiv: 10084, 2019.
David E Rumelhart, Geoffrey Hinton, and Ronald Williams. Learning representations by back- propagating enrors. nature, (238668)):333-533 1986.
Melanie Selar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models' sensi- tivity to spurious features in prompt design or: How learned to start worrying about prompt for- matting. arXiv preprint arXiv:2310.113222 2023.
Richard Socher, Eric Huang, Jeffrey Pennin, Christopher D Manning, and Andrew Ng. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. Advances in neural infor- mation processing systems, 24, 2011.
Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A Smith, Luke Zettlemoyer, and Tao Yu. One embedder, any task: Instruction-finetunee text embeddings. arXiv preprint arXiv:2212.09741 2022.
Kai Sheng Tai, Richard Socher, and Christopher D Manning. Improved semantic representations from tree-structured long short-term memory networks. arXiv preprint 1rXiv:1503.00075 2015.
James Thome, Andreas Vlachos, Christos Christodoulopoulls,, and Arpit Mittal. Fever: a large-
scale dataset for fact extraction and verification, 2018
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 2rXiv:2302.13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi. Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov. Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023b.
Dan Vanderkam, Rob Schonberger, Henry Rowley, and Sanjiv Kumar. Nearest neighbor search in google correlate, Technical report. Google, 2013. URL http:://ww.googg/ com/ trends/ porreelate/mnsaarchhh pdf
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang. Daxin Jiang, Rangan Ma- jumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.0355: 2022.
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improv- ing text embeddings with large language models. arXiv preprint arXiv:2401.00368 2023.
Liang Wang. Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Multi- lingual e5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672, 2024.
13

---

Published as a conference paper at ICLR 2025
Yashen Wang, He- Yan Huang, Chong Feng, Qiang Zhou, Jiahui Gu, and Xiong Gao. Cse: Concep- tual sentence embeddings based on attention model. In Proceedings of the 54th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 505-515, 2016.
John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. Towards universal paraphrastic sentence embeddings. arXiv preprint wxxiv1551.08/990 2013.
Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared llama: Accelerating language model pre-training via structured pruning. arXiv preprint arXiv:2310.06694 2023.
Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighof. C-pack: Packaged resources to advance general chinese embedding. arXiv preprini arXiv:2309.0759) 2023a.
Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged resources to advance general chinese embedding, 2023b.
Xiaohui Xie, Qian Dong, Bingning Wang, Feiyang Lv, Ting Yao, Weinan Gan, Zhijing Wu, Xiang- sheng Li, Haitao Li, Yiqun Liu, and Jin Ma. T2ranking: A large-scale chinese benchmark for passage ranking, 2023.
Xiaohan Xu, Chongyang Tao, Tao Shen, Can Xu, Hongbo Xu, Guodong Long, and Jian-guang Lou. Re-reading improves reasoning in language models. arXiv preprint arXiv:2309.062773 2023.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. Hotpotqa: dataset for diverse, explainable multi-hop question answering, 2018.
Xin Zhang, Zehan Li, Yanzhao Zhang, Dingkun Long, Pengjun Xie, Meishan Zhang, and Min Zhang. Language models are universal embedders. arXiv preprint arXiv:2310.0823: 2023a.
Xinyu Zhang, Xueguang Ma, Peng Shi, and Jimmy Lin. Mr. tydi: multi-lingual benchmark for dense retrieval, 2021.
Xinyu Zhang. Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso -ironso-Hermelo, Xi- aoguang Li, Qun Liu, Mehdi Rezagholizadeh, and Jimmy Lin. MIRACL: Multilingual Re- trieval Dataset Covering 18 Diverse Languages. Transactions of the Association for Computa- tional Linguistics, 11:1 11:1114-1111, 09 2023b. ISSN 2307-387X, doi: 10. a_05955, URL https://doi. wrg/ 1162/0..11111800 wwt.//111ccc..alee (.855
A GENERAL INFORMATION FOR REPRODUCIBILITY
In this section we include information that might aid reprodueibility that is not specific any specific setting in the paper.
A.1 MASSIVE TEXT EMBEDDING BENCHMARK
The Massive Text Embedding Benchmark (MTEB) is collection of datasets from seven cate- gories: classification, clustering, pair classification, reranking, retrieval, sentence similarity (STS), and summarization. The leaderboard published https://huggingaccs ctt spaces//tteb leaderboard, The list of datasets and their descriptions can be found at Muennighoff et al. (2022) in Appendix A.
A.2
BASE MODEL HUGGINGFACE IDs
In this paper, we use the following models:
Mistral 7B instruction-tuned: mistrala/Miisral-BBBlnsssccu
LLaMA 7B instruction-tuned. mataallama//llmaa227hhhhh
S-LLaMA 1.3B: priineeonni//hhheellllLL.M......
LLaMA 13B instruction-tuned: meta-llamaa/llama- 1 33--chat-hi

---

Published as a conference paper at ICLR 2025
B SOCIETAL CONSIDERATIONS
In this section we discuss the societal impact of our work and the safeguards we have taken to mitigate negative impacts.
B.1 IMPACT
We intend for and believe that this work will have a positive societal impact. In particular, language models are known to suffer from limitations such as hallucinations and biases. the context of reurieeal-augmeened generation, we believe that higher quality embeddings can lead to more accurate retrievals, which can in turn lead to more accurate generations. This can help mitigate the impact of hallucinations and biases in language models. We also believe that the improved embeddings can be used in a variety of downstream tasks, such as information retrieval, question answering, and summarization, which can have positive impact on society. Nonetheless, acknowledge that not all uses of our work will be necessarily positive. For example, our work can be used to improve the performance of language models in generating fake news misinformation. Second, release a training dataset that has been aggregated from a set of other open source datasets that might nonetheless contain harmful content. We believe that the positive impacts of work outweigh the negative impacts, and we are committed to working towards ensuring that our work is used for positive purposes.
C ECHO EMBEDDINGS: ADDITIONAL INFORMATION
In this section, we aim describe the additional details that were omitted from Section and
Cosine Similarity. As discussed in Section 2 we often use the cosine similarity to measure the similarity in embeddings. Recall that given two sentences x and y, we wish determine the degree to which they are semantically similar. Cosine similarity.
(Ø(z), ((v))
Sim (2, y) [((x))ll
measures the similarity between the embeddings of and y for any embedding function Φ: X'' - R The cosine similarity is used for our experiments in Sections and as the similarity function for training in[4.4 All MTEB datasets use cosine similarity compute similarity with the exception of the classification datasets, in which similarity is not explicitly measured, and the clustering datasets, which use Fuclidean distance,
Sim v) = [1Ø(z) - <(y))l.
(2)
metric.
Prompts for Section 3.
For these experiments, we only evaluate with single prompting strategy. For classical embeddings, we encode a sentence using the prompt:
Write sentence: S
We take the pooled embedding to the mean token embedding (s()). For echo embeddings, we encode sentence S using the prompt:
Rewrite the following sentence:
The rewritten sentence: S'
where S' = S and we let our pooled embedding be the mean token embedding 6s: (z). We do not evaluate with the last-token pooling strategy in this Section.
General Prompting Guidelines. Throughout the paper, we variety of different prompts to construct embeddings. Section D. we demonstrate that for zero-shot embeddings, the exact wording template used prompting strategy does not have strong effect on the performance of MTEB tasks, with the exception of for the summarization approach. This implies, in general, that classical embeddings and echo embeddings should be robust to the exact choice of prompts. The
15

---

Published as a conference paper at ICLR 2025
important component of echo embeddings is instead the structure: the input text should be repeated twice when computing embeddings, and the embeddings should be taken over the second occurrence of the input text.
Example classical embedding structures:
Say the sentence:
5
Write the phrase: S
Complete the query: S
Explain the text: S
Example echo embedding structures:
Repeat the sentence: S
The sentence again: 5*
Rephrase the query: S
The query rephrased: S"
Fill in the blank: S The blanks filled in: S'
Rewrite the text: S
The sentence rewritten: S
Toy data. We provide a subset of the toy data from Section For Structure 1, the data is given in Table 8 For Structure 2, the data is given in Table 9 For Structure 3, the data given in Ta- ble TO. In all cases, the data is generated by GPT-4. The data from Structure the following GPT-4 prompt, and the other structures are generated from minor variations on this: Together, we need to generate sentence triplets. Each triplet will have the following form: Sentence can be anything, be creative here.
generated from
Sentence must represent something opposite to sentence 1, however, it is important that the first half of the sentence is exactly the same the first half of sentence 2. The only difference in word- ing can be in the second half the sentence.
Sentence should be extremely similar sentence and semantically equivalent, but slightly worded.
Here is an example:
"sentencel": : "I like to eat apples and bananas but I really hate almost every other fruit.", "sentence2": "I like to eat apples and bananas and also enjoy also every other fruit", "sentence3": "I like two fruits: apples and bananas but I hate nearly all fruits other than these.".
The first half the sentence should be relatively short, less than 10 words, but the second half should be long, least 10 words. Give more examples, and write them in json format. Be creative!
16

---

Published as a conference paper at ICLR 2025
q
She loves to travel in sum- mer, especially to cold des- tinations, avoiding hot and crowded places
She loves to travel summer, but prefers to visit hot and bustling tourist spots
In summer, she adores travel- ing, specifically chilly loca- tions, steering clear of warm, populous areas
The cat often sits by the window, dreaming of chasing birds and enjoying the warm sunshine
The cat often sits by the win- dow, but is too lazy to dream of chasing anything
Frequently, the cat lounges near the window, imagining bird pursuits and basking the sunlight
He reads books every night, finding solace in fiction and escaping from the stresses of daily life
He reads books every night, yet he feels that non-fiction is more engaging and informa- tive
Nightly, he immerses himself in books, seeking comfort in stories and evading everyday tensions
They play music loudly in the evening, filling their home with energetic beats and vi- brant melodies
They play music loudly in the evening, but only soothing classical tunes to relax
In the evenings, they blast tunes, their house resonating with lively rhythms and bright harmonies
She paints landscapes on weekends, expressing her cre- ativity through vibrant colors and abstract forms
She paints landscapes on weekends, preferring realistic and detailed depictions of na- ture
On weekends, she engages in landscape painting, show- casing her artistic flair with lively hues and unconven- tional shapes
The children eagerly await winter, dreaming of snowball fights and building snowmen
The children eagerly await winter, yet they dislike the cold and prefer staying in- doors
During winter, the kids are excited, imagining snow bat- tles and constructing snow figures He often jokes at parties, but tends alienate others with his sarcasm
He often jokes at parties, be- coming the center of attention with his witty humor
At social gatherings, he fre- quently makes jokes, capti- vating the crowd with his clever wit
She collects antique vases, adoring their unique designs and historical significance
She collects antique vases, but is indifferent to their history and focuses on their resale value
Her hobby is gathering old vases, cherishing their distinct patterns and the stories they hold
The band plays rock mu- sie loudly, thrilling audiences with energetic performances and powerful lyrics
The band plays rock mu- sic loudly, but often re- ceives complaints for being too noisy
Performing rock loudly, the band excites crowds with dy- namic shows and impactful words
He prefers working night, enjoying the quiet and focus- ing better without distractions
He prefers working night, despite feeling more tired and less productive
Nighttime is his preferred work period, appreciating the tranquility and concentrated environment
She writes poetry in her free time, pouring her emotions and experiences into each verse
She writes poetry in her free time, but struggles to find in- spiration and motivation
During her leisure, she crafts poems, infusing her feelings and life stories into every line
Table 8: Examples of Structure I from Section
17

---

Published as a conference paper at ICLR 2025
On sunny days, often find myself longing for the cool breeze of the ocean and the sound of waves crashing, as enjoy outdoor activities
During rainy days, usually prefer the warmth and quiet of my home, as I enjoy outdoor activities
When the sun is shining, tend crave the refresh- ing sea air and the rhythmic sound of the ocean, since relish spending time outdoors
As lover of classical mu- sic, I spend hours listen- ing to Beethoven and Bach, reveling in the complexity of their compositions, though I'm fond of playing the guitar
Despite my preference for rock music, rarely spend time music other than playing my favorite tunes on the guitar, though m fond of playing the guitar
Being enthusiast of clas- sical melodies, I often in- dulge in lengthy sessions of Beethoven and Bach, appre- ciating the intricacies of their work, as delight in guitar playing
In the world of literature, I have insatiable appetite for mystery novels and spend countless evenings unraveling their plots, but adore reading poetry
Contrary to my usual tastes, rarely delve into mystery nov- els and prefer lighter reading materials, but adore reading poetry
fervent reader, my passion lies in the twists and turns of mystery stories, which I often explore dur- ing long nights, yet I cherish reading poetry
Growing up bustling city, I've always been surrounded by the constant hum of activ- ity and the bright city lights, which makes me appreciate quiet countryside walks
Having been raised in tran- quil rural area, I'm more ac- customed the sounds na- ture and open fields, which makes me appreciate quiet countryside walks
Raised in the lively atmo- sphere of an urban environ- ment, I'm used to the never- ending city noise and glow- ing nights, leading me to en- joy the serenity of rural strolls
Ever since was child, fas- cinated by the vastness of the universe, would spend countless nights gazing at the stars through my telescope, dreaming of exploring distant galaxies, yet still find solace in simple nature hikes
Though I've always been more interested in the imme- diate world around me, pre- ferring to focus on the tangi- ble and the present, rarely look at the night sky, yet I still find solace simple na- ture hikes
From my early years, capti- vated by the infinity of space, I devoted many nights to star- gazing and imagining inter- stellar journeys, but also en- joy the peace of nature walks
Growing up with passion for culinary arts, experiment- ing with exotic ingredients and complex recipes, and of- ten spending whole days in the kitchen perfecting new dishes, also have deep ap- preciation for classic litera- ture
Despite my lack of interest cooking and a preference for simple, quick meals that require minimal preparation, I'm not one to spend time the kitchen, also have a deep appreciation for classic litera-
Since childhood, I've been enthusiastic about cook- ing, often trying out un- usual ingredients and intri- cate recipes, dedicating entire days to refining my culinary creations, and equally cher- ish classic literary works
ture
Table 9: Examples of Structure 2 from Section B
18

---

Published as a conference paper at ICLR 2025
s
SShe loves travel in sum- mer, especially to cold des- tinations, avoiding hot and crowded places
She loves travel in summer, but prefers to visit hot and bustling tourist spots
She loves travel in summer, specifically to chilly loca- tions, steering clear of warm, populous areas
The cat often sits by the window, dreaming of chasing birds and enjoying the warm sunshine
The cat often sits by the win- dow, but too lazy to dream of chasing anything
The cat often sits by the win- dow, imagining bird pursuits and basking in the sunlight
He reads books every night, finding solace in fiction and escaping from the stresses of daily life
He reads books every night, yet he feels that non-fiction more engaging and informa- tive
He reads books every night, seeking comfort in stories and evading everyday tensions
They play music loudly in the evening, filling their home with energetic beats and vi- brant melodies
They play music loudly in the evening, but only soothing classical tunes to relax
They play music loudly in the evening, their house resonat- ing with lively rhythms and bright harmonies
She paints landscapes on weekends, expressing her cre- ativity through vibrant colors and abstract forms
She paints landscapes on weekends, preferring realistic and detailed depictions of na- ture
She paints landscapes on weekends, showcasing her artistic flair with lively hues and unconventional shapes
The children eagerly await winter, dreaming of snowball fights and building snowmen
The children eagerly await winter, yet they dislike the cold and prefer staying in- doors He often jokes at parties, be- coming the center of attention with his witty humor
The children eagerly await winter, imagining snow bat- tles and constructing figures He often jokes at parties, but tends to alienate others with his sarcasm
He often jokes at parties, cap- tivating the crowd with his clever wit
She collects antique vases, adoring their unique designs and historical significance
She collects antique vases, but is indifferent to their history and focuses their resale value
She collects antique vases, cherishing their distinct pat- terns and the stories they hold
The band plays rock mu- sie loudly, thrilling audiences with energetic performances and powerful lyries
The band plays rock mu- sic loudly, but often re- ceives complaints for being too noisy
The band plays rock mu- sic loudly, the band excites crowds with dynamic shows and impactful words
He prefers working at night, enjoying the quiet and focus- ing better without distractions
He prefers working at night, despite feeling more tired and less productive
He prefers working at night, appreciating the tranquility and concentrated environ- ment
She writes poetry in her free time, pouring her emotions and experiences into each verse
She writes poetry in her free time, but struggles to find in- spiration and motivation
She writes poetry in her free time, infusing her feelings and life stories into every line
Table 10: Examples of Structure from Section
19

---

Published as a conference paper at ICLR 2025
Most improved
Least improved
Sentence 1
Sentence 2
Score
Sentence 1
Sentence 2
Score
The best thing you can do is to know your stuff.
The best thing to do is to overcome the fussiness.
0.0
Sometime if you really want it you might need pay agency to get the place for you.
You could prob-
2.0
ably get tour
agency to do
for you but it would cost you.
It really doesn't matter.
It doesn't matter unless it is really far off.
3.0
There are three options:
There only 5.0 three options:
think it's fine to ask this question.
think okay to ask the ques- tion.
5.0 Bremer said one initiative is to launch US$70 million nation- wide program in the next weeks to clean up neighborhoods and build com- munity projects.
Bremer said he would launch $70-million pro- gram in the next two weeks to clean up neigh- borhoods across Iraq and build community projects, but gave no details.
3.6
What kind of in- sulation it?
What kind of floors are above?
0.0
"Tony's s not feel- ing well," Spurs coach Gregg Popovich said.
We're thrilled to be up 3-2," Coach Gregg Popovich said Wednesday.
1.6
depends tirely on your company and your contract.
guess it depends on the nature of your contract.
4.0 Shares of Man- dalay closed down eight cents to $29.42, before the earnings were announced.
Shares of Man- dalay closed down 8 cents at $29.42 Thursday.
4.0
You need to read a lot to know what you like and what you don't.
You have to know what you want to do.
0.0
Singapore ported no sus- pected SARS cases Wednes- day, but officials quarantined 70 people who had contact with the Taiwanese pa- tient.
3.0 Still, Singapore quarantined 70 people who had been in close contact with the scientist.
I would say you can do it, but it wouldn't be ad- vised.
Personally, I would say not unless it suits you.
2.0
The dollar
Against the Swiss franc the
1.333
at 117.85 yen against the Japanese cur-
dollar
1.3289 francs, up 0.5 percent on the day.
rency, up 0.1 per-
cent.
Table 1I: Example sentences from STSBenchmark which zero-shot echo embeddings with Mis-
tral 7B most improve (left) and least improve (right).
20

---

Published as a conference paper at ICLR 2025
D ADDITIONAL ZERO-SHOT RESULTS
Recent literature suggests that the performance of language models on zero-shot tasks can be highly variable depending on the exact wording and template of the prompts (Sclar et al. (023). Thus, in order to test the degree to which the exact prompt matters for each of the embedding strategies we consider, we perform prompi randomization where we sample prompts by randomizing the exact wording, punctuation, and capitalization of the prompt.
Prompt sampling procedure. Here we describe the prompt sampling procedure and then provide the prompts that we use for the zero shot:
1. Choose an instruction. For classical embeddings, choose from (Write, Say, Complete, Explain]. For echo embeddings, we choose from {Repeat, Rewrite, Rephrase, Fill in the blank). For PromptEOL, we choose from (Summarize. Categorize. Understand, Analyze).
2. Choose wording for the instruction. For example, if we chose "Say" the instruction, then we would choose from {Say a sentence, Say a paragraph, Say something, Say a re- sponse, Say a query, Say prompt). For PromptEOL, we also choose a second part of the wording, as the PromptEOL strategy requires that the summary be in one word: {in one word, with a single word, succinetly with one word, in a unique one-word way, in a single word, in word].
3. Choose separator, which include colons, commas, newlines.
4. Choose prefix, which includes markers indicate the first and appearance of the input.
5. Classical prompts have the form: "(instruction) (separator) {prefix} S".
6. Echo prompts have the form: "(instruction) (separator) {prefix0} S {separator) {prefix1).S"*
7. PromptEOL prompts have the form: "finstructionO) {separator) {prefix} S (instruction1) {separator)",
Classical embeddings.
For classical, we choose the prompts:
Write sentence 1] S
Write prompt! (I) S
Write some text PROMPT-S
SAY PARAGRAPH SENTENCE 0) S
Say a query QUERY: S
Say a sentence! [A] S
COMPLETE THE PROMPT Text (1) S
COMPLETE THE QUERY text 0) S
Complete the query SENTENCE 0) S
Complete the sentence:-S
Explain a query text 0 S
Explain a prompt Sentence 1 S
21

---

Published as a conference paper at ICLR 2025
EXPLAIN A SENTENCE Prompt (1) S
Echo embeddings.
For echo, we choose the prompts:
Repeat The Paragraph.
query 1) S. query 2)
Repeat the response. S.
AGAIN 2) S
REPEAT THE SENTENCE PROMPT : RESPONSE
S'
Rewrite the query QUERY (A) (B) S
Rewrite the text. SENTENCE A) S. S"
Rewrite the response query A] S query B] S"
Rephrase the sentence: StAgain@S'
Rephrase The Sentence! Text iú S! Answer is S
REPHRASE THE QUERY Sentence a) S Answer b) S'
Fill in the blank in the prompt: Query a) S: Query b) S
FILL IN THE BLANK IN THE RESPONSE - Sentence A) S Sentence B) S'
Fill in the blank in the paragraph. Text-= S.
Response- - S"
PromptEOL embeddings. For PromptEOL, the prompts:
SUMMARIZE THE QUERY.
Prompt: S'IN A WORD.
Summarize the sentence! PROMPT زا S' S'Succinetly With One Word!
SUMMARIZE THE PARAGRAPH. PROMPT (0) S'IN A WORD.
CATEGORIZE THE PROMPT query
S With single word
Categorize the query prompt [1] S in word
CATEGORIZE THE SENTENCE.
Prompt iL S IN A WORD.
Understand the sentence @ In single word
22

---

Published as a conference paper at ICLR 2025
Understand The Prompt:QUERY [0] S in single word:
UNDERSTAND THE TeRAGRAPH:Tex Succinetly with one word:
Analyze the sentence.
Sentence S In Unique One-word Way.
Analyze the response! query ag S IN A UNIQUE WNE-WORD WAY!
Analyze The Prompt Sentence ag. In a unique one-word way
Subset of MTEB for zero-shot sensitivity evaluation. Given that we evaluate on hundreds of prompts, we choose a subset of MTEB to reflect each of the MTEB categories but with a sub- stantially reduced computational requirement for evaluation. We evalaute on the following sub- set of MTEB: FiQA2018, SCIDOCS, SciFact, NFCorpus, TwitterSemmvva12015. TwitterURL- Corpus, ImdbClassification, AmazouRriieessCassiiittttoo TiveeSentimeeE..trattttCCCCCCii tion. MTOPDomainCCassiiican ivClustcringS2S, StackOverflow Dupuuuttoss,s AskUUumuDupuuustionn. SciDocsRR, BIOSSES, STS12, STS13, STS14, STS15, STS16, STS17, STS22, STSBenchmark, and SICK-R. Note that this excludes the largest retrieval datasets in MTEB, on which we find PromptEOL is most likely to fail.
TweatyNewssrruupsClssscci
BiorxivClusteringS22 Medrx-
Mcasuring the sensitivity of different embedding strategies to prompting. We plot the sensi- tivity of classical, repetition, and PromptEOL to different choices of prompts for different models in Figures 7 8 and | We also extend to plotting on all tested datasets individually in Figures [0] [T and 12 We observe that PromptEOL is highly sensitive to the exact prompt used. However, neither classical nor echo were particularly sensitive. Consistently, mean token pooling outperformed last token pooling by large factor.
E ToY VERIFICATION
In Section 3 we demonstrate that classical embeddings overestimate the similarity between exam- ples which superficially similar based on tokens that appear early in the sequence. In this sec- tion, we investigate whether this failure mode generalizes to real data. We provide qualitative and quantitative evidence that this failure mode does, in fact, generalize real data.
E. QUALITATIVE COMPARISON OF CLASSICAL AND ECHO EMBEDDINGS.
To build intuition that this applies to realistic data, we present the sentence pair from STSBench- mark, sentence similarity task from MTEB, in which echo embeddings reduce error the most:
The best thing you can do is to know your 21: stuff.
The best thing to do overcome the 02 fussiness.
which has a ground-truth score of 0 (out of 5) similarity. The sentence pair for which echo embed- dings reduces error the least is:
Sometime you really want it you might ne- y1: ed to pay agency to get the place for you.
You could probably get a tour agency to do it for you but it would cost you.
U
which has ground-truth similarity of 2 (out of 5). We provide examples in the Appendix Table
For these examples, the sentence pair (#1, 22) which echo embeddings improve emor the most has exactly the property we identify as a failure mode for classical embeddings: the sentence is superficially similar for the first few tokens. On the other hand (V1, y2) does not have this property.
23

---

Published as a conference paper at ICLR 2025
BIOSSES
SICKR
Echo
basical
Lcho
Cossical
-21 Unceresiimatio
Zin Overestimation
Underistimation
Underestimaion
Overestimation
Undersstimation
STS19
STSL3
Echo
Clasical
Echo
Clasiial
10:0
100
X001
100 Undernstimation
1000
180
Underacimation
Coerestination
Ownedimation
Underndimation
Osenelimation
Undernsimmaion
Donrssimation
ST514
STS15
Frhn
Clavera
Fcbe
Chosical
-230 -140
Jin 301 300 -4i -200 -100
30
-100 Unstiontimatoon
120 Overestimttonn
-
-200 Underestination
10.0 Overestination -?
Underestination
Overestimation +t
Underestimation
Overestimazion
STS16
STS1/
Lcho
Classica
Echo
Clavical
-.30
200 Overistimation -2 Underestiationn
5M
20
10
100
120
10
100
Underssination
Ostrasimation
mation
Ostroximacion
4-
Understimatioon
Ovanotimation
STSBerdhmark
Edho
Clnaical
a Echo
Clinkial
300
-20e
20.0
-000 - Undenntimationn
X/0
JBM
-300
-100
-00 Underesiination
10:0
+- Undernnimation
Overestimarion
Oveestimation
Undroest imation
Ovenetination
Overenfimatinn (mrean)
Sentence is simitr
Second holf (meon)
Secord half
Figure 6: We plot the histogram distribution of the difference between the predicted rank and the ground truth rank sentence pairs in STS datasets. When predicted rank larger than the ground truth rank-wher the rank difference is positive-ther then the embedding has overestimated the simi- larity this pair. Similarly, negative values imply that the the rank is underestimated, We plot the distribution of these ranks for both classical and echo embeddings where split the data into two groups: one which sentences are similar the first part of the sentence (top 10% by first-half similarity), and another in which sentences are similar in the second part of the sentence (top 10% by second-hall half similarity).
24

---

Published as a conference paper at ICLR 2025
E.2 QUANTITATIVE EVALUATION OF THE FAILURE MODE.
In order test this hypothesis quantitatively, that are superficially similar in their early tokens, and examples that do not have this property. We compare the behavior of classical embeddings on these datapoints.
design experiment to identify real examples
To conduct this experiment, we extract a set of examples from the STS datasets included in the MTEB benchmark in which the first half of the sentence is similar, and measure the degree which the similarity is overestimated. We also select points which are similar in the second half of the sentence, and measure the degree to which similarity overestimated. By comparing the degree to which sentences which are similar in the first half are overestimated in similarity, and the degree to which sentences which are similar in the second half are overestimated, then we can identify classical embeddings overestimate similarity in specifically sentences which are similar in the first half. Thus, under our hypothesis, we expect that, for classical embeddings, sentences which are similar in the first half overestimated in similarity more than sentences that are similar in their second half. On the other hand, we expect that, for echo embeddings, the degree to which similarity is over- underestimated is independent of whether the sentences similar in the first second half of the sentence.
Identifying examples based on similarity in the first/second part of the sentence. We aim to determine which sentences are most similar in the first half of the sentence in the second half of the sentence. For each sentence pair I, we split the sentences in half by number of words, yielding æ [ 222, and y //]] We select sentences which are most similar in the first waaeodd-lnnngge--mon
embedding model age-base-en-vy 1.5 (Xiao et al. 2023b). To select sentences that are similar in the first half, we measure the cosine similarit Sim V1) and take the top 10% of sentence pairs 2, y which have the highest cosine similarity. Similarly, to select sentences which are similar in the second half, we collect the top 10% of examples by Sim(22, U2). We collect examples from each of the STS datasets in MTEB.
half by using the off-the-shelf
Measuring sentence similarity estimation error. We must determine the degree to which classi- cal and echo embeddings overestimate similarity. The STS datasets contain sentences pairs which are ranked by similarity: the sentences which are most similar have the highest ground-truth rank- ing, and the least similar sentences have the lowest. We will denote the ranking of sentence pair i as r,. We compute estimated ranking (fr} } by ranking sentence pairs by the cosine similarity be- tween their embeddings. We can compare the error our estimated ranking by taking the rank dif- ference Err, = A; rę- When Err, 0, we say that the ith sentence pair is overestimated in simi- larity, and similarly underestimated when Err; < O.
Results. We plot the the distribution over rank differences for sentences which are similar in the first half and sentences which are similar in the second half for echo and classical embeddings, from all STS datasets (Figure ( We also highlight the means of the distributions. In accordance with our hypothesis, we observe that for classical embeddings, sentences which are similar in the first half are generally overestimated in similarity more than sentences which are similar in the second half of the sentence, suggesting that classical embeddings fail particularly on sentences that are similar in early tokens. Further, we generally observe no difference between the estimation error distributions for echo embeddings, which demonstrates that echo embeddings recover from this particular failure mode.
There are some notable counterexamples: BIOSSES does not exhibit this trend, but has few exam- ples and thus the results may arise from noise alone. Further, STS22 exhibits identical distributions in estimation error between sentences which are similar in the first half and sentences which are sim- ilar in the second half, for both classical and echo embeddings. It is unclear why this trend fails to hold for STS22. Nonetheless, the trend holds for every other dataset, suggesting that the conceptual failure of classical embeddings that identified in Section generalizees to real data.
Qualitative examples. In addition, we provide qualitative examples of sentence pairs from STS- Benchmark where echo embeddings reduce error most, and where echo embeddings reduce error least, in comparison to classical embeddings. More precisely, plot the top and bottom 7 exam-
25

---

Published as a conference paper at ICLR 2025
ples ranked by | Err, ). where Erri represents the rank difference of the ith example of classical embeddings, and Err, is similar but for echo embeddings (Table 11).
E.3 FULL ZERO-SHOT
Here we present omitted results from the main paper. First, the role of scale and base model is shown T2
Table 12: Role of scale and base model in the zero-shol setling: We study the role of scale and base model using different 7B parameter model: LLaMA-7B and 1.3B parameter model: S-LLaMA- 1.3B.
Categories
Clas.
Clus. 11
P. Cls.
Rera. 4
Retr.
STS
Su.
Average
# of datasets
12
15
10
56
Echo
1.3B
64.85
30.62
47.30
42.35
13.23
61.68
23.80
40.45
Classical
1.3B
59.41
27.50
41.01
38.09
8.53
47.91
23.36
34.31
PromptEOL
1.3B
68.86
24.52
36.52
43.81
8.34
56.64
27.36
37.50
Echo
7B
70.75
30.68
65.15
45.72
18.03
70.19
29.81
45.84
Classical
7B
63.54
31.99
53.78
42.82
15.07
55.69
26.21
40.29
PromptEOL
74.02
26.93
58.04
47.48
10.11
66.94
29.71
42.84
Classification
Pair Classification
70
C+
50
mB
say
compl-
expla
say
write
compl
expla
repea
rephr
fill
rewri
repea
rephr
summa
analy
categ
under-
say
compl
expla
say
write-
compl.
expla
repica
rephr
fill
rewri
repea
rephr
summa
analy
catog
under-
Clustering
Retrieval
121
8
-3+
+7
50y
compl
expla
say
write
compl
expla
rephr
fill
rewn
repea
rephr
summa
analy
categ
under
say
write
compl
expla-
say
write
compl
expla
rephr
fill
rewn
repea
rephr
summa
analy
catog
under
STS
Reranking
Score
8
rephr- HTI
-3
4T
say
write
compl
expla
say
write
compl
expla
fill
rewn
repea
rephr
summa
amaly-
categ
under
say
write
compl-
expla-
say-
write
compl
expla
repea
rephr-
fill-
rewni
repea
rephr-
summa
analy-
catog
under-
Average
Embedding strategy
Score
Classical (last)
Repetition (last)
Classical (mean)
Repetition (mean)
expla 0i
rephr +i
rewri- a
Summarization
say
compl
Say
write-
compl
expla-
repea
fill-
repea-
rephr
summa
analy-
categ-
under
Figure 7: Variance over different prompting strategies for zero-shot Mistral-7B.
26

---

Published as a conference paper at ICLR 2025
Classification
Pair Classification
40
TH
90
8 65
++-
+TM
H
compl
expla
say
write
compl
expla
repea
rephr-
fill
rewri
repea
rephr
fill
summa
analy
categ
under
compl
expla
say
write-
compl
expla
repca
rephr-
fill
rewri
repea
rephr
fill
summa
analy
categ
under
Clustering
Retrieval
0
analy 8
say
compl
expla
say
write
compl
expla
rephi:
fill
rewn
repea-
rephr
summa
analy catog under
say
write
compl
expla-
say
write-
compl
expla
repea
rephi-
fill
rewn
repea-
rephr
summa
catog
under
STS
Reranking
85 5 35 58
rephr -IH
do
say
compl
expla
say
write
compl
expla
fill
rewni
repea
rephr
summa
analy
categ
under
say
compl
expla
5ay
write
compl
expla
repea
rephr
fil
rewni
repea
rephr
summa
analy
cattgg
under
Average
80
Embedding strategy
Scort
Classical (last)
Repetition (last)
Classical (mean)
Repetition (mean)
80
reghr -1
to
Summarization
say
compl
expla
say
write
compl
expla.
repea
fill
rewn
repea
rephr
summa
analy
categ
under
Figure 8: Variance over different prompting strategies for zero-shot LLaMa-2-7B.
27

---

Published as a conference paper at ICLR 2025
Classification
Pair Classification
70
+
ott
80
18
50
*
88
say
comple
expla
say
write
compl
expla
repea
rephr-
fill
rewri
repea
rephi
fill
summa
analy
categ
under
compl
expla
say
write-
compl
expla
repca
rephr-
fill
rewri
repea
rephr
fill
summa
analy
categ
under
Clustering
Retrieval
+
Score
0
th
rewn -To
expla di
say
write
comple
say
write
compl
expla
rephi-
fill
repea
rophr
summa
analy
catog
under
say
write
compl
expla-1
say
write-
compl
expla
repea
rephir-
fill
rewn
repea-
rephr
summa
analy
catog
under
STS
Reranking
8 55
88 8
F
-44
rephr- oBo
say
compl
expla
say
write
compl
expla
rephr
fil
rewn
repea
rephr
summa
analy
categ
under
say
compl
expla
say
write-
compl
expla
repea
fill
rewni
repea
rephr
summa
analy
categg
under
Average
Embedding strategy
Classical (last)
Repetition (last)
Classical (mean)
Repetition (mean)
afo
reghr- 40
Summarization
say
write
comple
expla
say
write
compl
expla.
fill
rewni
repea
rephr
summa
analy
categ
under
Figure 9; Variance over different prompting strategies for zero-shot LLaMa-2-13BB --BB..
28

---

Published as a conference paper at ICLR 2025
FiQ42008 (wetrevs))
SCIDOCS (netreza)
Salfart (vetnincl)
Spon
Scom
3
0
come
with
con
50
wrth
inde
w1
cono
1000
303y
NFComuu (vetinad))
Twinte Corpas fww. stediirtiio)))
12
30
Score
com
wibh
comp
wibh
campl
50
comp
comp
1000
IndbC CusiCtitttttoonn (dessfication)
Ponesea Fication (dessiittio)
aadiiaa atdiiattion (dea Fra un)
Son 1
*
Sore
8
wh
wik
compl
expee
mampl
whe
c0mpl
unde
00mp
comp
reat
2000
MTOFDemanC Ceaffaaaeae (dearfictiin)
TorrsNeeepuupcesstii (clatoin))
Blonncl (dustore))
1.
corp
14:00
50
com
reph
undee
comp
comp
12:0
30d5
uni
Matni-atttiuSSE (duaterine)
StackO Wow.oooseenn (strenking)
0 2
AuUuarnDDppuustcn (reankinn)
Seor
23
min
12000
50
min
1200
snita
12:00
500
01000
Siloesiit (eeorinng)
STS32 S1932 (cs)
Score
121
with
60
wib
corp
1
with
50
with
18
man'
10p0
comp
with
comgl
1000
1
sank
ST513 (se))
ST334 (185)
ST335 (Pg
Sooe
5
comi
cony
egg
1000
Dy
comat
agg
caNS
0100
STS16 (mm)
STS1T ((ba)
STS22 (ms)
Scon
5 1
21
rego
inde-
comal
1000
myh--
3000
STSBenchaakk (cts)
SICK-R (es)
Embedding stategy
Classcal (lax)
Repotitor (ust)
HU
(mear) Summariatton
Repetitor (rean)
177
comp
will
won
will
esepp
comp
want
220 inde
Figure 10: Variance over different prompting strategies for all evaluated datasets for zero-shot Mistral-7B.
29

---

Published as a conference paper at ICLR 2025
FiQ42018 (wetress))
SCIDOCS (netreza)
Salfart (vetninol))
Spon
Scor
-
C
60
oF
with
con
353
comp
50
wrtee
indr
with
cono
1000
303
calog
NFComuu (vetinad)
TwittessnnFoeiis Ira craredan hortion
Twinte Corpar (poettiiieeeitiiie
KD
Scor
1
comple
wibh
comp
wib
camp
50
wide
1rde
1000
IndbC Cusittittttioonn (Jussfication)
Amaacee Focation (clesiicatio)
WeddGntiinttnaatiiCCCC assficationn (dlas Featin)
Sone
0
comp
223h
nampl
cxple
who
camp
unde
wit
oomp
reant
MTOFDemanCuuffine csifictioon (dearfcation
Cow.ooeeeeoeoeeeeseetit (clatoiig)
Blonncl. (dustore))
12:00
evep-
50
Comp
reph
comp
1200
rewd"
unb
Modni-CattriuS2 (duaterive)
StackO BowD. AQuusrinnaaaaaa (reranking)
AaUlsaaDDpQuustoo (rwanking)
Seor
1 0
Srore
mit
12000
50
12000
12:00
500
Siloesii (weoriing)
BOOSES (At))
STS12 (cs)
$
Score
x
10
with
wib
with
mampl
50
with
10p0
will
with
comgl
1000
1
3153
0100
5T513 (26)
ST334 (18s)
STS35 (P9)
Scoe
181
.TH
con'
eg
cato
comst
agg
STS16 (mm)
STSIT (1bs)
ST522 (ex)
1009
comal
repo
myr
calg
Inde
conal
myhe-
3000
5T58mcchark (ets)
SICK- (mm)
Emtedding stategy
Soor
Classcal (lax)
Repotitor (ust)
10
(mear) Summariation
Repetitor (rean)
comp
wit
will
comp
11
wan
con inder
Figure 11: Variance over different prompting strategies for all evaluated datasets for zero-shot M-
30

---

Published as a conference paper at ICLR 2025
FiQ42008 (wetrevs))
SCIDOCS (netriza))
Salfart (vetrinca))
Spm
Scom
x3
with
w1th
cmmp
50
1000
inde
with
cono
10000
300y
calog
NFCommu ((ntinad))
Twinte Corpar ppoke steeiiiiire
F
Score
3
7
D
wil
com
wibh
comp
wib
campl
1rde
compl
IndbC Iusittiitatioon (dussiication
Poaceee Fication (dlsiiicaii)
TeadGatiinattnaaaiiCCCC aasiiattin (diaa Frat
Sone 3.3
MEC
50
wh
compl
12
w9b
mampl
whe
00mp
unde
oomp
udd
www.oeoooooooceeeee (clatorin))
MTOFDeman seifaaten (dearfiction)
Blonkcl. (dusterne)
+
righ
Inte
evep
comp
regh-
regh
GUAA unde
comp
1200
reph
rewe"
20dp
unb
MctnirCateriuSE (duaterie)
StackO www.ooooo..s........
Auuasimm (stranking)
AulUsanDDspausstee (rwanking)
g
Seor
Store
2
12000
50
1200
12500
300
SclooiifR (eroniing)
HOSSIS (1)))
STS12 (cs)
2
with
with
corp
18
5193
with
mamp
50
wilh
18
1000
wile
withr
corgl
10p00
CANS-
ST513 (16))
STS14 ST534(000 (185)
ST335 (FS)
500
Sooe
1
con'
egg
530
cans
Dy
coma
agg
STS16 (mx)
STS (m))
ST522 (mx)
U.S.
5100
inde
comal
1000
3000
ST5Bmnchmrk (its)
SICK-R (es)
Emtedding stategy
Soor
Class (lax)
Repotitor (ist)
(mear)
Repetitor (rean)
Summariatton
comp
with
comp
n1 vont
(1
COs inde
Figure 12: Variance over different prompting strategies for all evaluated datasets for zero-shot M-33-.
31

---

Published as a conference paper at ICLR 2025
F ADDITIONAL FINETUNING RESULTS
Table 13: MTEB Leaderboard: We compare to other recent models on MTEB, For more complete
list, please visit: https://huggingfaeee co/ ct//sppces//tt//eeeeeerooo/o/o
Categories
Clas.
Clus.
Cls.
Rera.
Retr.
STS
Average 56
# of datasets
12
11
15
10
Main fine-tuning results trained with publicly available data:
Echo (ours)
77.43
46.32
87.34
58.14
55.52
82.56
30.73
64.68
Echo + compute matched
77.39
46.27
87.49
58.08
55.07
83.18
30.73
64.66
Classical
76.57
45.78
86.37
56.71
54.87
82.03
31.02
63.98
Classical bidirectional attn.
76.70
45.94
88.15
57.23
54.96
82.42
29.32
64.23
UAB-Large-V (MLM)
75.58
46.73
87.25
59.88
54.66
84.54
32.03
64.64
multilingual-e--arr (MLM)
77.56
47.10
86.19
58.58
52.47
84.78
30.39
64.41
bge-large-on-v1.5 (MLM)
75.97
46.08
87.12
60.03
54.29
83.11
31.61
64.23
udever-bloom-7b (autoregr.)
72.13
40.81
85.40
55.91
49.34
83.01
30.97
60.63
sgpt-5.8b (autoregr.)
68.13
40.34
82,00
56.56
50.25
78.10
31.46
58.93
Prior work trained with closed-source data (see Section 5.2):
c5-mistral-7b (autoregr.)
78.47
50.26
88.34
60.21
56.89
84.63
31.40
66.63
GritLM-7B (autoregr.)
79.46
50.61
87.16
60.49
57.41
83.35
30.37
66.76
In this section, we address the omitted details from the finetuning results of the main paper.
Training Datasets. We follow the setup of Waan et al. (2023). and use the following datasets: ELI5 (sample ratio 0.1) (Fan al. 2019). HotpotQA (Yang et al. 2018), FEVER Thorre et al.) 2018), MIRACL (Zhang et al. 2023b), MS-MARCO passage ranking (sample ratio 0.5) and docu- ment ranking (sample ratio 0.2) (Bajaj et al. (0188). NQ (Karpukhin et al. 2020). NLI (Gao et al.)
2021b), SQuAD (Karpukhin et al. 2020). TriviaQA (Karpukhin al. 2020). Quora Duplicate Ques- tions (sample ratio 0.1) DDataCanary et al. 2017), Mr- TyDi (Zhang et al. 2021), DuReader (Qiu et al. (022) and T2Ranking (sample ratio 0.5) (Xie et al. 2023 We use approximately 1.5M train- ing examples.
GPUs. Training model takes approximately two days four A100 GPUs. Evaluating model on the MTEB benchmark can be completed in parallel in approximately two days with eight A100s.
Instructions for finetuning datasets.
We also follow the setup of Wang et al. (2023). and use the instructions in Table 15 For evaluation, we use the instructions found in Table 16
Models on MTEB leaderboard. We compare our implementation of classical and echo embed- dings state-of-ther ofrtth-arttt approaches on MTEB. Namely, display results for UAE- MAE-Large-V V1 (Li 2023a). multilingualle5-laan (Wang et al. (2024). bge-large-en-vl.5 (Xiao et al., (023b). udever- bloom-7b (Zhang al.) 2023a), 5gpt-5.8b (Muennighofi 2022), e5-mistral-7b (concurrent work)
Wang et al. 2003).
Pooling, We investigate the role of pooling for fine-tuning (Table 14). We find that last-token, with a trainable end-of-sentence token embedding, is optimal.
Additional ablations. We plot additional ablations, including ablating the role of instructions dur- ing training and evaluation, well providing evaluation at step 280 (out of 720 total steps), which is approximately 40% of the duration of training (Table 17h. We note that echo embeddings still outperform classical embeddings in this setting.
Comparisons to the MTEB leaderboard. board in Table 13
We include some recent models on the MTEB leader-
32

---

Published as a conference paper at ICLR 2025
Table 14: Role of pooling in the fine-tuning setting: Average MTEB Score (56 datasets) for differ- ent pooling strategies. We evaluate mean and last-token pooling after fine-tuning with Mistral-7B- Instruct-vÔ.
Categories datasets
Clas.
Clus.
P. Cls.
Rera.
Retr.
STS
Su.
Average
10
Echo last token pooling
77.43
46.32
87.34
58.14
55,52
82.56
30.73
64.68
Echo + mean pooling
77.00
44.94
87.73
58.30
55.11
82.52
29.46
64.22
Classical last token pooling
76.57
45.78
86.37
56.71
54.87
82.03
31.02
63.98
Classical mean pooling
76.26
42.68
86.31
57.58
53.75
81.53
30.19
62.96
NLI
Given a premise, retrieve hypothesis that is entailed by the premise Retrieve semantically similar text
NLI
DuReader
Given Chinese search query, retrieve web passages that answer the question Provided
ELIS
question, retrieve the highest voted answers Reddit ELIS
forum
FEVER
Given a claim, retrieve documents that support or refute the claim Given a multi- hhop question, retrieve documents that can help answer the ques- tion
HotpotQA
MIRACL
Given question, retrieve Wikipedia passages that answer the question Given question, retrieve Wikipedia passages that answer question Given a web search query, retrieve relevant passages that answer the query Given web search query, retrieve relevant documents that answer the query Given question, retrieve Wikipedia passages that answer the question Given question, retrieve questions that are semantically equivalent the given question
MrTyDi
MSMARCO Passage
MSMARCO Document
NQ
QuoraDuplicates
QuoraDuplicates
Find questions that have the same meaning as the input question Retrieve Wikipedia passages that answer the question
Squad
T2Ranking
Given Chinese search query, retrieve weh passages that answer the question Retrieve Wikipedia passages that answer question
TriviaQA
Table 15:
Instructions for finetuning datasets.
Performance over training time. We plot the performance over the duration of training for subset of MTEB tasks in Figure 13 Surprisingly, task performance decreases over training for many tasks.
All results. We plot the results for every MTEB dataset for echo embeddings, for classical embed- dings, and for bidirectional embeddings in Table 18
Training objective. For the training objective, we use the SimCSE loss (Gao et al. 202 is defined,
exp (Sim (hs, /T) Σ/=1 exp (Sim (h,, k,)) /T)
& log
In this loss function, h; represents query (or reference sentence when the data is symmetric), h represents a positive example associated with h; and {hz } represents the set of negatives associated with the example, including mined hard negatives.
33

---

Published as a conference paper at ICLR 2025
FiQA2218
SCIDOCS
Soiaac
Noropuss
Soore
0.555
0.21
0.76
0.39
0.71
0:45
0.20
0.39
70
330
630
770
210
190
630
770
210
350
490
630
770
210
400
770
QutraRetrieval
1witterrSmEva22015
TiittrrURL Corpus
ImdlaC ClassClaaiiacaa lassilccation
Soxe
0.85
0.87
0.900
70
210
350
190
630
770
70
210
350
190
630
770
70
210
350
190
630
770
70
210
350
190
630
770
weeerSentmmnt Exteriiieeettttiiiiinni www.reerrrreccncnnnnneiiooo
0.50
0.965
Score
0.62
0.960
0.50
0.46
0.60
0.955
0.45
70
210
350
190
630
770
70
210
350
190
630
770
70
210
350
190
630
770
70
210
350
190
630
770
Biarsi-Clnntering5555
MedrriiCClaaeernnSS55
Scon
0.30
0.52
0.63
0.28
0.62
370
630
770
210
400
770
350
630
770
70
210
400
770
RRiDousRR
BIOSSES
STS12
ST513
0.76
0.85
Score
0.86
0.74
0.84
0.84
0.83
0.82
630
630
770
STS14
STS15
STS16
STS17
0.87
Score
0.81
0.86
0.915
0.80
0.965
0.910
0.85
0.905
0.860
70
210
350
400
630
770
70
210
370
400
630
770
70
210
350
400
630
770
70
210
350
400
6:0
770
STS22
STSBenchmark
SICK-R
SummEval
Scon
0.70
0. 0.865
0.31
0.68
0.855
70
210
3370
400
630
770
70
210
350
400
630
770
70
210 350 400) 630 770
70
210 350
400)
6.3
770
Step
Step
Step
0.68
Embodding strategy
Repetition
0.67
Classical
0.66
210
350
490
630
770
Step
Figure 13: Performance of the evaluated MTEB datasets for finetuning over the number of finetuning steps.
34

---

Published as a conference paper at ICLR 2025
Amuzon Comteeetttl.
Classify a given Amazon customer review text as cither counterfactaal or not counterfactual
AmazonPPlanityCli
Classity Amazon reviews into positive or aegative sentiment
AmaponReriwssCle
Classiy the given Amazon
appoupriate raiigg calegoey
Banking77Cls.
online bunking query,
corroponding intents
EmotionChs.
Classify the emosion expressed in the given Twiteer message into one of the six emotions: anger, fear, 30%, love.
sadness, and supprise
ImdbCls
Classiyy the sentiment expressed the given muvic review from the IMDB datleset
MassivelnteuCls
user utterance query.
user inents
Given user ulterance as query. lind the user scenamios
M MTOPDamainCss
Classity the inteat domain of the given utterance in task-orizuted conversation
MTOPInicniCss
Classify the inteat the given
Classify the given comments either tusk oriented conversation
or tuxic
Classify the sentiment a gaven tweet as cither positive, negative, or noutral
AairCCustee
Identify the main und secondary casegory of Arsiv papers based on the lides and abstracts
Clusterriss
Identify the und secondary cosegury Arsh papens based on the litles
Chusering'22
Identify main category Bloexi pupers based
abstracts
Biorsi ChaseringS22
Identify the main category of Bicesiv pupers based on the titles
Caateenhhsseinng P2P
Identify the main categecy of Medixiv papers based the titles and abstroets Identify the category Medisiv pupers based the titles
MedranChsserinng
Clusteiuu
Identify tupic theme of Reddit posts
on
Raddi Pus
Identity the topic theme of Reddit posts based on the tites and posts Identify the tupic theme of StackEschinge posts based on the titles Identify the tupic theme of StuckEschnnge posts based Ideatify the topic theme Retrieve duplicate questions from Spcint tocum
StackEnchangee Clstsinnngg
StackEachangee Clusterin
given paragraphs
Twenty Newsgroupe Chastinnnn
given news articles
SprimDuplicc QueeiiiituuQution
Twitte SemE5al201
Retrieve tweets that are semontically similar the given rweet Retrieve that semeatically similar the given Retrive duplicate questions from AskUbuntu forum Retrieve relevast news articles bised on user browsing history Given tite of scientific
Twimet CoCcou
MindSmmllll Krrnkinn
SciDeesRR
retrieve the tities of other relevant papers Retneve duplicate questions from Stackverrlow forum
SackOverrlow
ArguAno
Given clain, find documents refute
VE
Given claim about climate shange, retrivve documeats that support or recute the claim
OoADepeaacAnnniidRe AndwidRetr.
Given question, rerrieve detailed question deserijtions from Staketchange that are duplicates the given quest Given question,
given given question Given quection, retrieve detailed question deseriptions from Stackeschange that are duplicates to the given question Given question, retrieve detailed question deserijtians from Stacketchange that are duplicates the given questian detailed question descriptions from Stackexchange that duplicates the given question Given question, retrieve desalled question descriptions from Stac kexccancn that are duplicates the given question Given question, retrieve detailed question deseriptions from Stac Backkeshhnnge that are duplicates to the given question Given question, retrivve detailed question descriptions from Stackeschange that are duplicates the given question Given question retrieve desailed question deseriytians Stac leckhanhn that duplicates the given question Given question, retrieve detailed question descriptions from Stackexchange that are duplicates the given question Given quesioo, retrieve dessileduussionn deseriptions from Stakexchange that are duplicates to the given question Given query. retriese relevart emity descriptions from DRPedis documents suppor refute the claim
detailed question deseriytions Stackeschange that duplicates
ADDuerackGannnngkk
Given question, retrieve detalled question descriptions from Stackesccange that are duplicates
CQAeeeeack Gisseir.
Given question
CQA PagranmosRRt.
COADepeackStnsseer
COADuperack Ree.
CQADopetck BereeaaaeWooeeeseeeoee RettrressRerr.
DBPadia
REVER
Given
F
Given finuncial question, netrive neer replics that seower the question Given multi-hpp quesion, retrieve documents that can help answer the question Given web search query, retreve relevant pissages that answer the query quesfice, retriave relevani documents that Given question, retrieve Wikipedia passages answer question
HоtprQQ
MSMARCD
NFCorpus
the
NQ
QueraRetr.
Given queation, retriove quesioos that are semantically equivalent to the given question Given scientifie paper tifee, retrieve paper shttmess that clied by the given paper Given scientifie
SCIDOCS
SciFact
that suppoct refute the claim
Touche2020
Given question, retrieve detailed and persussive seguments that answer the question Given query on COVID- 19, retrieve doenments that answer the query
TRECCDVID
BUOSSES
Retrivve sensanically similar text
SICK-R
Retrieve senamically similar
STS12
Retrieve sonuntically similar
STS13
Retrieve semamically similar text
STS14
Retrieve semamically similar text
STSI5
Retrieve conamically similar
STS16
Retrieve semumically similar text
STS17
Retrieve senamically similar text
ST522
Retrieve sensniically similar text
S7SBenchmak
Retrieve semamically similar
SummFoal
Given news summary, retriese other semontically similar summories
Table 16: MTEB instructions for evaluation of finetuned models.
35

---

Published as a conference paper at ICLR 2025
Table 17: Additional ablations for finetuning. All ablations were performed using Mistral-7B backbone. We include ablations to compare the effect of using instructions, the effect of using the mean last token, and the effect of using the last token at a different step.
Categories datasets
Clas. 12
Clus.
P. Cls.
Rera.
Retr. 15
STS
Su.
Avg 56
11
Fcho (w/ instruct., mean)
77.00
44.94
87.73
58.30
55.11
82.52
29.46
64.22
Echo (w/ instruct., last) Classical (w/ instruct., mean)
77.43
46.32
87.34
58.14
55.52
82.56
30.73
64.68
76.26
42.68
86.31
57.58
53.75
81.53
30.19
62.96
Classical (w/ instruct., last)
76.57
45.78
86.37
56.71
54.87
82.03
31.02
63.98
Echo (w/out instruct., mean)
75.26
42.93
86.95
57.05
55.65
81.40
30.62
63.28
Echo (w/out instruct., last) Classical (w/out instruct., mean)
75.30
42.94
86.31
57.31
54.18
80.92
31.00
62.80
75.23
41.79
85.24
56.31
53.24
80.97
30.64
62.19
Classical (w/out instruct., last)
75.01
42.70
85.69
56.64
53.29
80.92
30.91
62.37
Echo (w/ instruct., mean, step 280)
76.84
45.76
87.72
59.33
53.55
82.64
30.33
64.04
Echo (w/ instruct., last, step 280)
76.41
46.70
87.17
59.10
54.84
82.98
31.09
64.50
Classical (w/ instruct., mean, step
76.18
42.99
85.44
57.63
53.96
82.53
29.94
63.19
280)
Classical (w/ instruct., last, step 280)
76.54
46.22
86.70
57.79
53.73
82.22
30.13
63.87
36

---

Published as a conference paper at ICLR 2025
Table 18: Results from all MTEB datasets for finetuning with Mistral-7B.
Echo (last)
Echo (mean)
Classical (last)
Classical (mean)
Bidirectional (last)
Dataset
Amazon Caatraoneoouttratttuaaaaaaaiiiie
82.97
82.91
80.82
82.21
83.07
AmazoonPlarityCCassiiii Classification
90.98
92.55
90.37
90.83
aunazonReviewssClaificccaai
48.71
49.41
48.75
46.76
47.94
Banking77Classiicattion
88.15
88,06
87.95
87.69
88.17
EmottonClassificationn
52.18
51.51
50.66
49.23
52.09
ImdhClassification
87.42
84.80
83.18
82.53
83.02
MussivennemtClaassccttion
79.67
79.70
78.60
79.15
78.93
MassiveScenarioolassiiicco
82.82
82.74
81.71
81.46
81.80
MTOPDonainCCasiiicaa
96.16
96.10
95.92
95.54
96.14
MTOPInnennCCaassffaaaiinn
85.75
85.87
85.96
85.86
85.98
71.91
72.21
71.19
72.21
71.46
62.40
62,46
61.60
62.07
60.97
ArxitClustriiggP2P
47.02
45.52
46.73
45.80
47.03
CxitelusteringS25
43.52
42.32
43.99
40.73
42.14
Biorxiv BiotxicClusteringgPPP
35.53
35.24
36.50
35,42
36.21
BiorxivClusteringgS
35.34
33.70
34.87
32.03
34.77
MedrxxiCClusteringP2PP
30.27
29.68
30.67
29.74
31,06
Medrxiv Clustrinnus
29.67
27.73
29.75
27.97
30.12
RedditClustering
61.77
59.12
61.17
54.79
62.50
ReddicClusteringPP
66.01
65.44
64.84
63.68
65,45
Stack BlackExchangeeCluserinng
72.04
71.21
71.87
66.99
71.58
StacKExchaangeCussternnggPP
35.29
34.07
33.08
31.47
34.98
TwentyNowsgruupscChseeingg
53.04
50.29
50.07
40.91
49.53
SprintDuphicalleuussttooss
94.59
95.05
94.38
95.29
96.26
TwitterSemEvva2215
79.93
80.73
77.18
75.98
80.80
TwitterURLCCrpus
87.50
87.40
87.56
87.67
87.38
Ask
AskUhuntuDupQuusstionn
64.13
64.44
62.24
63.32
62.65
MindSmall
32.92
32.11
32.68
32.52
32.53
SciDocsRR
83,68
84.15
81.60
83.01
82.36
StackOverlowDDuuuuestiooss
51.84
52.51
50.33
51.48
51.35
ArguAna
58.52
56.52
57.22
51.14
57.27
ClimateFEVER
34.56
37.07
31.10
30.31
32.73
CQADupstackRRttrevvlll
46.91
46.48
45.11
43.30
46.52
DBPedia
46.83
48.19
45.18
46.80
46.76
FEVER
91.22
91.14
90.30
90.63
91.66
FiQA2018
54.51
54.11
50.31
48.94
53.06
HotpotQA
76.41
75.75
72.95
68.50
75,30
MSMARCO
43.25
43.11
42.31
41.49
43.38
NFCorpus
39.55
37.18
39.32
38.53
38.61
NQ
62.31
61.51
62.07
60.65
63.69
QuoraRetrieval
89.34
89.33
89.04
88,94
89.57
SCIDOCS
20.17
17.73
19.34
19.88
19.69
SciFact
73,99
73.57
74.22
75.39
75.83
Touche2020
18.52
18.92
24.46
19.44
15.79
TRECCOVID
76.66
76.02
80.17
82.30
74.50
BIOSSES
86.54
86.78
85.73
83.31
85.38
STS12
76.13
75.89
75.84
76.23
75.50
STS13
83.19
82,90
83.41
82.61
83.44
STS14
80.60
80.99
79.80
79.89
81.35
STS15
87.16
87.16
86.99
86.68
87.43
STS16
85.16
84.93
83.93
84.18
85.34
STS17
90.88
90.78
91.12
90.14
90.99
STS22
67.04
67.21
66,27
65.99
66.32
STSBenchmark
85.67
85,87
84.96
85.20
85,45
SICK-R
83.23
82.70
82.22
81.11
82.97
SummEval
30.73
29.46
31.02
30.19
29.32
37


Reviews:
[{"review_id": "iRtmnjcAEh", "reviewer": "Reviewer_gzEy", "summary": "The paper proposes a very easy method to improve the language model embedding. They introduce echo embedding by repeating the input and extracting embedding from the repeated tokens. The experiment show that the echo embedding improve over classical LM embeddings by over 5% in zero-shot settings. Echo embeddings are also compatible with supervised fine-tuning, even with an identical compute during training and inference.", "questions": "Have you tried tripling or increasing the input length even further? I'm curious to know if this would result in any additional improvements.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The echo embedding method is both easy and effective. While previous studies have demonstrated that repetition is beneficial for reasoning tasks and recurrent language models, this paper shows that it is also effective for causal language model embedding.\n2. The paper is clearly written and easy to understand.\n3. The use of a simple synthetic dataset to analyze why causal attention might inhibit embeddings from reliably capturing information across the entire context is interesting.", "weaknesses": "The echo embedding method will inevitably double the input length. Although experiments show that reducing the input length and training steps by half still yields good results, this approach may not be suitable in cases where important information is located in the latter half of the input context. For example, the S2 (Early redundant; late discriminatory) cases described in Section 3.1 of the paper. Additionally, because self-attention has a computational complexity of O(n^2) with respect to input length, training for only half as many steps may not necessarily equate to the training cost of the original baseline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VBJ4WNz7qs", "reviewer": "Reviewer_PgPq", "summary": "This paper proposes a simple strategy to extract high-quality sentence representation from causal LMs, i.e., repeating the input and extracting embeddings from the repeated tokens. They argue that their method requires neither bi-directional attention nor supervised fine-tuning. They first conduct experiments on a toy dataset, where they can control the information type of part of the inputs. Then, they clearly show the limitation of classical embeddings (mean pooling or last token) and the advantage of their method. Experiments on the MTEB dataset show that their method has advantages over classical approaches and recent alternatives. However, I hold some concerns with the experiment setting, see weakness parts. Overall, I find the paper's idea interesting, but the practical aspects need further discussion.", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. I appreciate the toy experiment, which clearly supports their claim about the limitation of classical embeddings and the advantages of echo embeddings.\n\n2. The results on the MTEB dataset show clear improvements over classical embedding extraction settings, achieving comparable results with LLM2Vec, which needs backbone changes and unsupervised finetuning.\n\n3. The method itself is very simple and insightful, requiring no changes to the backbone.", "weaknesses": "1. The setting of the most relevant baseline, promptEOL, does not seem to exactly align with that in the original paper. The results of PromptEOL appear significantly different from those reported in the original paper. In the original study, PromptEOL achieved an average score of 72.10 across seven STS tasks using the OPT-6.7B model. However, in your paper, PromptEOL only obtains an average of 67.14 on ten STS tasks. I didn't expect such a big performance discrepancy. Is this because of the three additional STS tasks in the evaluation?\n\n2. I found the prompt used in this paper differs from that in the original paper. How about the performance of promptEOL that uses exactly the same prompt?  \n\n\n3. I found the \"halve input\" setting weird to me. For a prompt like \"rewrite S, rewritten S'\", does it mean halving both S and S' to extract embedding? If so, I will read the results as the \"nature\"/\"issue\" of the problem itself, rather than an advantage of echo embedding. E.g., how about also halving the input for PromptEOL?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z0oC2P36fe", "reviewer": "Reviewer_HXgu", "summary": "Causal attention in autoregressive models limits the use of LLMs in text embedding tasks because tokens in a sequence cannot reference subsequent tokens. This paper introduces an \"echo embedding\" method to transform autoregressive language models into high-quality text embedding models without altering the causal attention architecture or requiring fine-tuning. By repeating the input sequence twice and extracting embeddings from the second occurrence, the tokens can attend to the full input from the first occurrence. Experimental results demonstrate that the proposed method can nearly match the performance of bi-directional and fine-tuned models using zero-shot embeddings.", "questions": "1. I am curious about the influence of number of repetition times, which is not important though.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is simple by repeating the input sentence twice to get the text embeddings.\n\n2. The toy example design is interesting.\n\n3. The results of zero-shot settings is impressive.", "weaknesses": "1. The motivation regarding causal attention seems questionable. Although LLM2Vec utilizes causal attention, it still performs exceptionally well in extracting text embeddings.\n\n2. After fine-tuning the model, the performance gap between \"echo embedding\" and other models is minor. However, \"echo embedding\" requires the input sentence to be repeated twice, increasing computational costs. This limitation confines the proposed method to zero-shot settings only.\n\n3. At least one illustrative example should be included in the main article, rather than relegating all examples to the appendix.\n\n4. Typo: line 122, \"encode can\" -> \"can encode\"\nline 283: \"embedding 4\" -> \"embedding in Table 4\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SYZXZzTDFC", "reviewer": "Reviewer_qkAp", "summary": "The paper presents a simple and effective method called \"echo embeddings\" that enhances text embeddings generated by autoregressive language models without finetuning. This technique involves prompting the model with the input twice and extracting embeddings from the repeated tokens. The authors demonstrate that echo embeddings significantly improve the quality of embeddings in zero-shot settings, achieving nearly the same performance as bidirectionally-converted LMs that undergo additional masked-language modeling training. Echo embeddings are also shown to be compatible with supervised fine-tuning, matching or outperforming the bidirectionally-converted LMs in direct comparisons.", "questions": "- See above, performance on generation tasks.\n\n- Can you provide more details on the impact of different prompts on the performance of echo embeddings? Have you explored variations in the prompt structure, and how sensitive are the results to these variations?\n\n- Have you tested the echo embeddings method on a wider range of autoregressive models beyond those reported in the paper? How does the method perform on models with different architectures and training paradigms? For example, non-gpt models.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces a novel and straightforward method to enhance autoregressive embeddings without the need for architectural changes or additional fine-tuning. This simplicity is a significant advantage, making the approach easily applicable to existing models. The results indicate that echo embeddings provide a substantial performance boost in zero-shot settings compared to classical embeddings. \n\n- Echo embeddings can be applied in both zero-shot and fine-tuned settings, demonstrating flexibility and robustness across different tasks and benchmarks. The technique also shows consistent results across multiple models and scales. While repetition doubles the compute cost, the paper provides compute-matched results showing that echo embeddings still outperform classical embeddings even when adjusting for compute. This suggests that the method is efficient and scalable.\n\n- The paper presents extensive experiments on a variety of datasets and tasks such as classification, clustering, reranking, retrieval, and sentence similarity. This comprehensive evaluation supports the validity and generalizability of the proposed method.", "weaknesses": "While the method shows strong performance across various benchmarks, it is possible that the specific tasks and datasets used in the evaluation may favor the proposed approach. Further validation on different types of data and tasks would strengthen the claim of general applicability. And also, does this technique helps some generation tasks? For example, if echoes some important information in the prompt, will it boost the performance on downstream generation tasks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)