Repetition Improves Language Model Embeddings

Authors: Jacob Mitchell Springer, Suhas Kotha, Daniel Fried, Graham Neubig, Aditi Raghunathan
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: embeddings

Abstract:
Bidirectional models are considered essential for strong text embeddings. Recent approaches to adapt autoregressive language models (LMs) into strong text embedding models have largely had the requirement to modify the LM architecture to be bidirectional. We challenge this premise by introducing ``echo embeddings'' which converts autoregressive LMs into high quality text embedding models \emph{without} changing the architecture or requiring fine-tuning. By repeating the input and extracting embeddings from the repeated tokens—which have access to all original tokens—echo embeddings improve over classical LM embeddings by over 5\% in zero-shot settings. Our zero-shot embeddings nearly match those obtained by bidirectionally-converted LMs that undergo additional masked-language modeling training. Echo embeddings are also compatible with supervised fine-tuning, matching or outperforming bidirectionally-converted LMs in an apples-to-apples comparison, even with an identical compute budget during training and inference. Overall, repetition is a simple and effective strategy to circumvent the need for bidirectional attention in embedding models, paving the way towards a unified architecture for all NLP tasks.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730288394554

thread_type:
official_review

<thread_object>
creator: reviewer_qkap

<content>
summary:
The paper presents a simple and effective method called "echo embeddings" that enhances text embeddings generated by autoregressive language models without finetuning. This technique involves prompting the model with the input twice and extracting embeddings from the repeated tokens. The authors demonstrate that echo embeddings significantly improve the quality of embeddings in zero-shot settings, achieving nearly the same performance as bidirectionally-converted LMs that undergo additional masked-language modeling training. Echo embeddings are also shown to be compatible with supervised fine-tuning, matching or outperforming the bidirectionally-converted LMs in direct comparisons.

strengths:
- The paper introduces a novel and straightforward method to enhance autoregressive embeddings without the need for architectural changes or additional fine-tuning. This simplicity is a significant advantage, making the approach easily applicable to existing models. The results indicate that echo embeddings provide a substantial performance boost in zero-shot settings compared to classical embeddings. 

- Echo embeddings can be applied in both zero-shot and fine-tuned settings, demonstrating flexibility and robustness across different tasks and benchmarks. The technique also shows consistent results across multiple models and scales. While repetition doubles the compute cost, the paper provides compute-matched results showing that echo embeddings still outperform classical embeddings even when adjusting for compute. This suggests that the method is efficient and scalable.

- The paper presents extensive experiments on a variety of datasets and tasks such as classification, clustering, reranking, retrieval, and sentence similarity. This comprehensive evaluation supports the validity and generalizability of the proposed method.

weaknesses:
While the method shows strong performance across various benchmarks, it is possible that the specific tasks and datasets used in the evaluation may favor the proposed approach. Further validation on different types of data and tasks would strengthen the claim of general applicability. And also, does this technique helps some generation tasks? For example, if echoes some important information in the prompt, will it boost the performance on downstream generation tasks?

questions:
- See above, performance on generation tasks.

- Can you provide more details on the impact of different prompts on the performance of echo embeddings? Have you explored variations in the prompt structure, and how sensitive are the results to these variations?

- Have you tested the echo embeddings method on a wider range of autoregressive models beyond those reported in the paper? How does the method perform on models with different architectures and training paradigms? For example, non-gpt models.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730612621611

thread_type:
official_review

<thread_object>
creator: reviewer_hxgu

<content>
summary:
Causal attention in autoregressive models limits the use of LLMs in text embedding tasks because tokens in a sequence cannot reference subsequent tokens. This paper introduces an "echo embedding" method to transform autoregressive language models into high-quality text embedding models without altering the causal attention architecture or requiring fine-tuning. By repeating the input sequence twice and extracting embeddings from the second occurrence, the tokens can attend to the full input from the first occurrence. Experimental results demonstrate that the proposed method can nearly match the performance of bi-directional and fine-tuned models using zero-shot embeddings.

strengths:
1. The proposed method is simple by repeating the input sentence twice to get the text embeddings.

2. The toy example design is interesting.

3. The results of zero-shot settings is impressive.

weaknesses:
1. The motivation regarding causal attention seems questionable. Although LLM2Vec utilizes causal attention, it still performs exceptionally well in extracting text embeddings.

2. After fine-tuning the model, the performance gap between "echo embedding" and other models is minor. However, "echo embedding" requires the input sentence to be repeated twice, increasing computational costs. This limitation confines the proposed method to zero-shot settings only.

3. At least one illustrative example should be included in the main article, rather than relegating all examples to the appendix.

4. Typo: line 122, "encode can" -> "can encode"
line 283: "embedding 4" -> "embedding in Table 4"

questions:
1. I am curious about the influence of number of repetition times, which is not important though.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730714550191

thread_type:
official_review

<thread_object>
creator: reviewer_pgpq

<content>
summary:
This paper proposes a simple strategy to extract high-quality sentence representation from causal LMs, i.e., repeating the input and extracting embeddings from the repeated tokens. They argue that their method requires neither bi-directional attention nor supervised fine-tuning. They first conduct experiments on a toy dataset, where they can control the information type of part of the inputs. Then, they clearly show the limitation of classical embeddings (mean pooling or last token) and the advantage of their method. Experiments on the MTEB dataset show that their method has advantages over classical approaches and recent alternatives. However, I hold some concerns with the experiment setting, see weakness parts. Overall, I find the paper's idea interesting, but the practical aspects need further discussion.

strengths:
1. I appreciate the toy experiment, which clearly supports their claim about the limitation of classical embeddings and the advantages of echo embeddings.

2. The results on the MTEB dataset show clear improvements over classical embedding extraction settings, achieving comparable results with LLM2Vec, which needs backbone changes and unsupervised finetuning.

3. The method itself is very simple and insightful, requiring no changes to the backbone.

weaknesses:
1. The setting of the most relevant baseline, promptEOL, does not seem to exactly align with that in the original paper. The results of PromptEOL appear significantly different from those reported in the original paper. In the original study, PromptEOL achieved an average score of 72.10 across seven STS tasks using the OPT-6.7B model. However, in your paper, PromptEOL only obtains an average of 67.14 on ten STS tasks. I didn't expect such a big performance discrepancy. Is this because of the three additional STS tasks in the evaluation?

2. I found the prompt used in this paper differs from that in the original paper. How about the performance of promptEOL that uses exactly the same prompt?  


3. I found the "halve input" setting weird to me. For a prompt like "rewrite S, rewritten S'", does it mean halving both S and S' to extract embedding? If so, I will read the results as the "nature"/"issue" of the problem itself, rather than an advantage of echo embedding. E.g., how about also halving the input for PromptEOL?

questions:
See weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730723045030

thread_type:
official_review

<thread_object>
creator: reviewer_gzey

<content>
summary:
The paper proposes a very easy method to improve the language model embedding. They introduce echo embedding by repeating the input and extracting embedding from the repeated tokens. The experiment show that the echo embedding improve over classical LM embeddings by over 5% in zero-shot settings. Echo embeddings are also compatible with supervised fine-tuning, even with an identical compute during training and inference.

strengths:
1. The echo embedding method is both easy and effective. While previous studies have demonstrated that repetition is beneficial for reasoning tasks and recurrent language models, this paper shows that it is also effective for causal language model embedding.
2. The paper is clearly written and easy to understand.
3. The use of a simple synthetic dataset to analyze why causal attention might inhibit embeddings from reliably capturing information across the entire context is interesting.

weaknesses:
The echo embedding method will inevitably double the input length. Although experiments show that reducing the input length and training steps by half still yields good results, this approach may not be suitable in cases where important information is located in the latter half of the input context. For example, the S2 (Early redundant; late discriminatory) cases described in Section 3.1 of the paper. Additionally, because self-attention has a computational complexity of O(n^2) with respect to input length, training for only half as many steps may not necessarily equate to the training cost of the original baseline.

questions:
Have you tried tripling or increasing the input length even further? I'm curious to know if this would result in any additional improvements.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes “echo embedding,” a method to improve language model embeddings by repeating the input and extracting embeddings from the repeated tokens. It aims to address the limitations of causal attention in autoregressive models for text embedding tasks without architectural changes or fine-tuning.  All reviewers vote for accptance and  there is an overall agreement that the method is simple, effective in zero-shot settings, and the experiments are relatively comprehensive. However, each reviewer also raises valid concerns that need to be addressed, e.g., input length and computational complexity, which are well discussed during the rebuttal. Overall, this is a good contribution to the conference.