ESE: Espresso Sentence Embeddings

Authors: Xianming LI, Zongxi Li, Jing Li, Haoran Xie, Qing Li
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: sentence embeddings, semantic textual similarity, information retrieval, retrieval-augmented generation

Abstract:
High-quality sentence embeddings are fundamental in many natural language processing (NLP) tasks, such as semantic textual similarity (STS) and retrieval-augmented generation (RAG). However, most existing methods leverage fixed-length sentence embeddings from full-layer language models, which lack the scalability to accommodate the diverse available resources across various applications. Viewing this gap, we propose a novel sentence embedding model Espresso Sentence Embeddings (ESE) with two learning processes. First, the learn-to-express process encodes more salient representations to shallow layers. Second, the learn-to-compress process compacts essential features into the initial dimensions using Principal Component Analysis (PCA). This way, ESE can scale model depth via the former process and embedding size via the latter. Extensive experiments on STS and RAG suggest that ESE can effectively produce high-quality sentence embeddings with less model depth and embedding size, enhancing inference efficiency. The code is available at https://github.com/SeanLee97/AnglE/blob/main/README_ESE.md.

Decision: accept (poster)
Reviewer scores: 5, 6, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1730099962084

thread_type:
official_review

<thread_object>
creator: reviewer_3a5x

<content>
summary:
This paper is concerned about sentence embeddings. This paper claims that previous optimizations on the efficiency of sentence embeddings largely lie in the embedding size yet ignore the model depth. However, both the model depth and the embedding size could have a large impact on the efficiency. So in this paper, espresso sentence embeddings are proposed to consider both two facets in a joint optimization framework including both learning-to-express and learning-to-compress. The learning-to-express attempts to compress the model depth while the learning-to-compress attempts to compress the embedding size. Experimental results show that espresso sentence embedding enjoy better performance-efficiency tradeoffs than compared baselines.

strengths:
- The proposed espresso sentence embeddings are indeed the first of its kind in this area.
- The experimental results denote good performance-efficiency tradeoffs have been achieved.

weaknesses:
- The idea, though not previous studied, is not that novel. And both two facets, i.e., the mode depth and embedding size, have been thoroughly studied in existing literature.

questions:
- Why, in Figure 4a, spearman correlation is degraded along the increment of embedding size?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730623449935

thread_type:
official_review

<thread_object>
creator: reviewer_xogj

<content>
summary:
The paper proposes "Espresso Sentence Embedding" (ESE) for effective sentence embedding learning with two key processes "learn-to-express" and "learn-to-compress.” In the "learn-to-express" phase, the loss is computed using both the final layer and intermediate layer embeddings. In "learn-to-compress," ESE focuses on compressing embeddings by applying the softmax of the outer product of attention values. Experimental results demonstrate that ESE outperforms baselines like RAW (original model) and MRL (Matryoshka embedding; Kusupati et al., 2022) on STS and QA benchmarks. Furthermore, ablation studies confirm the robustness of ESE across various language models and embedding sizes.

strengths:
**Motivation**

- The proposed work addresses the scalability challenges in sentence embedding that arise from fixed embedding sizes and a fixed number of layers. The training objective (learn-to-compress and learn-to-express) is well-motivated and effectively demonstrated throughout the paper.

**Methods and Experiments**

- The approach of directly training shallow layer embeddings (Section 3.2) significantly enhances embedding quality. Experimental results in Figure 3 show that even embeddings from shallow layers can achieve performance comparable to deeper layers in other models, highlighting the effectiveness of this approach.
- The novel compression method (Section 3.3) successfully captures interactions between vector dimensions, improving upon traditional PCA techniques, as shown in Table 2.

**Practicality**

- The framework presents a representation learning method that can be applied to low-resource scenarios.

**Writing**

- The paper's motivation is clearly written and easy to follow. The figures, particularly Figure 2, are well-designed and effectively illustrate the proposed methodology.

weaknesses:
The experimental setup is generally well constructed, but I have little concerns about following issues:

1. Kusupati et al. (2022) conduct extensive experiments across various classification and retrieval tasks using the ImageNet-1K dataset. In contrast, the paper under review **restricts its experimental scope solely to STS tasks,** which raises **concerns about the generalizability of the proposed method.** If the authors intended to focus exclusively on the NLP domain, I recommend including experiments in areas where sentence embeddings are widely applicable, such as information retrieval (e.g. BEIR benchmarks[1]). Although the paper includes results from HotpotQA (Table 4), it omits key metrics for evaluating the quality of embeddings from the retriever, such as supporting facts EM and F1. As a result, it is challenging to evaluate the sole impact on the retriever (i.e., embedding) within the overall pipeline.
2. It is uncertain whether **comparing ESE’s intermediate layer performance with that of other models is entirely fair.** Obviously, these models are intended to use the final layer embedding for inference, so a comparison based on shallow-layer performance would be inappropriate. For practical significance, it would be better to **emphasize that “non-final-layer embeddings (e.g., <12th or <24th layer) of ESE outperform the final-layer embeddings (e.g., 12th or 24th layer) of MRL or RAW” ** This would highlight ESE’s potential time-saving benefits by reducing forward passes while guaranteeing the fair comparison. However, in the BGE-based results shown in Figure 3, ESE’s performance at layer 11 does not appear to surpass that of MRL or ESE’s own performance at layer 12, which raises concerns about claimed practical advantages of  ESE.

[1] Thakur, Nandan, et al. "Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models." arXiv preprint arXiv:2104.08663 (2021).

questions:
1. Why do MRL and RAW show consistent results across compression sizes in Figure 5? Does compression not cause any performance change for these models?
2. In Section 3.3, the compression dimension $k$, to which the vector is reduced, appears to be fixed. However, compressed embedding sizes vary in Figures 3 and 4. How is this possible? Are these models using different compression dimensions at training and testing time, or are they trained with varying k values each time?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730640878841

thread_type:
official_review

<thread_object>
creator: reviewer_876w

<content>
summary:
This paper follows the motivation of Matryoshka Representation Learning (MRL) to learn sentence representation at different granularities, allowing a single embedding to adapt to the computational constraints of downstream tasks. The main idea is to use the PCA compressed representation to supervise representation learning at each grain. They show some positive results on STS tasks, more specifically, compared to previous works, their method makes the embedding extracted in the inner layers better, which is aligned with their design. Also, some ablation studies are provided to support the plausibility of their design. Basically, the paper is clear and well-written. However, I hold some concerns with their motivation and design, see the weaknesses part.

strengths:
1. The paper is well-written, showing clearly the gains from the inner layers upon raw and MRL design.
2. I appreciate their experiments with different model scales and compression strategies, which clearly demonstrate the ablation and provide realistic comparison results.

weaknesses:
1) The design is complex: there are a lot of hyperparameters to tune with, like the weight for Loss_le, Loss_lc, the combination of MSE and KL-div, and joint training. It is hard to provide insights into where the main gain comes from.

2) Is embedding dimension k also a hyperparameter? If so, does this mean that achieving different types or lengths of compressed representations requires multiple training iterations?

3) I doubt real-world applications for this method. Their main motivation is to reduce inference latency, where the most straightforward way is to use a smaller backbone. However, as shown in Table 5, the gain compared with BERT_small is marginal.

questions:
1) Is embedding dimension k also a hyperparameter? If so, does this mean that achieving different types or lengths of compressed representations requires multiple training iterations?

2) Does "PCA on sentence embeddings" in Table 2 mean directly applying PCA on the high-dim representations? If so, from the storage angle, the advantage of the proposed method is limited?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730656310981

thread_type:
official_review

<thread_object>
creator: reviewer_527o

<content>
summary:
This work introduces a method for fine-tuning sentence embeddings while simultaneously providing flexibility in model size and embedding size. The method consists of two parts: learn-to-express, which trains different model depths, and learn-to-compress, which focuses on compressing information into a sub-embedding. The authors provide extensive experimental results on STS datasets and RAG on QA, including visualizations for the quality of the embeddings as well as results on inference efficiency. For the ablation studies, they investigate PCA on different dependencies, the effect of embedding sizes for PCA, and weighting present in the loss function.

strengths:
1. The authors fuse two existing ideas (model truncation and embedding truncation) into a single training process, which is a somewhat limited novelty. However, this provides much-needed flexibility for real-world use cases. 
2. The method is backed up by empirical results on seven STS datasets, one RAG dataset, and ablation studies that justify method design. 
3. The paper uses figures to better explain concepts, which is nice. However, I often found it lacking details for experiments and ablation studies.
4. This paper introduces the idea of training multiple configurations of a model at once. This is very useful for efficiency-critical and memory-constrained environments. The method can be handy for engineering use cases.

weaknesses:
## 1. Unfocused contribution statement

> To the best of our knowledge, we are the first to learn sentence embedding with information compression, presenting scalable embedding inference to both model depths and embedding dimensions

> We are the first to employ the information compression technique to scale sentence embeddings"

If by "information compression" the authors are referring to model depth, there are already existing papers on this topic, some of which are even cited (e.g., "BeLLM: Backward Dependency Enhanced Large Language Model for Sentence Embeddings" and "Are ELECTRA's Sentence Embeddings Beyond Repair? The Case of Semantic Textual Similarity"). Additionally, other cited works focus on reducing the embedding size (e.g., "Matryoshka Representation Learning"). Please clarify your wording and specify precisely how your method differs from these approaches.

## 2. Incomplete related work

The related work section is quite incomplete, as it includes only a few well-known references, most of which are not closely connected to the paper's topic. For example, a few papers that address closely related topics (excluding the aforementioned references) are:

- Interpreting Pretrained Contextualized Representations via Reductions to Static Embeddings
- WhiteningBERT: An Easy Unsupervised Sentence Embedding Approach
- What does BERT learn about the structure of language?
- How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings
- BERT Has More to Offer: BERT Layers Combination Yields Better Sentence Embeddings

## 3. Unmotivated design choices

The paper is very empirical in nature, which is not a bad thing. While most of the method's design is supported by intuitive explanations, I find the rationale for choosing the "align" loss function insufficiently explained. Ideally, a more theoretical approach and justification for introducing the loss functions would significantly strengthen the paper.

> Specifically, we use Euclidean distance, i.e., IndexFlatL2 in faiss, as the similarity measurement

Why use the Euclidean distance if you are optimizing for cosine similarity?

> Note that we leave the last layer unweighted due to its critical role in capturing sentence-level semantics

The reference paper mentioned only tests this on LLMs, so what is the design justification for using it on MLMs?

## 4. Potentially incorrect/misleading conclusions

> It allows the direct use of the first k-dimensional sub-embeddings at inference without repeatedly performing PCA and improves inference efficiency

I am not convinced by the second part of this statement. I can see that this improves efficiency ONLY if you consider using PCA. If you don't use any compression, the efficiency would be roughly the same, wouldn't it?

Furthermore, Figures 3, 4, and ≺ Avg. columns for Tables 1 and 2 provide an unfair comparison -- RAW and MRL do not tune the inner transformer layers. This comparison is repeated a couple of times throughout the paper as an insight: 
 
> RAW and MRL exhibit inferior performance in the shallow layers

> This shows the effectiveness of ESE's learn-to-express process in encoding the more important information into shallow layers

> Notably, ESE achieves a significant improvement at shallow layers, demonstrating its superior scalability

This leads an unwary reader to an incorrect conclusion. It would make more sense to tune each layer separately and compare it with your method.

Furthermore, Table 5 is a somewhat unfair comparison. Truncating the model depth does not mean the model has the same number of parameters as BERT_smal or BERT_tiny (due to different hidden dimensions). I would advise the authors to add the information on number of parameters and mention this while commenting on the results.

The comparison of distances after using TSNE is debatable. TSNE preserves the global structure, but measuring distances after TSNE does not make sense (cf. [https://datascience.stackexchange.com/questions/17314/are-t-sne-dimensions-meaningful](https://datascience.stackexchange.com/questions/17314/are-t-sne-dimensions-meaningful)). For more rigor, I would instead use MDS for this visualization and comparison.

## Minor remarks

- Figure 7 seems to lack details on which layer was used to create the embeddings on the subfigures and which 128 dimensions were chosen for the RAW method.

- For Table 1, I would advise to put the links into the footnotes.

- I know you use CLS pooling in your experiments, but the implementation for mean pooling is incorrect.

```
    if pooling_strategy == 'avg':
        outputs = torch.sum(
            outputs * inputs["attention_mask"][:, :, None], dim=1) / torch.sum(inputs["attention_mask"])
```

- Is Equation 6 missing a sum over k?

questions:
## Q1:

> The possible reason is the ESE's high-quality embeddings in varying scaled sizes allow easier and more effective indexing, thus improving RAG efficiency

I do not see how this affects the indexing process. Can you elaborate a bit more?

## Q2:

Could you share the code for the inference used in the case of MRL or point me to the part of the code that does this in the anonymized repository?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731142046178

thread_type:
official_review

<thread_object>
creator: reviewer_5kse

<content>
summary:
The paper proposes an approach for generating sentence embeddings that are scalable both in terms of model depth and embedding size. The proposed Espresso Sentence Embeddings (ESE) model incorporates two key processes: the learn-to-express process and the learn-to-compress process. The learn-to-express process aims to encode crucial latent representations in shallow layers, while the learn-to-compress process uses Principal Component Analysis (PCA) to condense essential features into the initial dimensions of the embeddings. The paper demonstrates that ESE can produce high-quality embeddings that are competitive with or surpass existing methods on standard benchmarks for semantic textual similarity (STS) and retrieval-augmented generation (RAG) tasks, particularly at reduced model depths and embedding sizes.

strengths:
1. The paper is well-organized and easy to follow. 
2. Experimental results are clearly presented, demonstrating the effectiveness of the proposed ESE method.

weaknesses:
1. Lack of Comparison with Strong Baselines like Llama 3.1
2. The paper does not include comparisons with OpenAI's embedding models, which are considered significant benchmarks in the field of NLP.

questions:
Please refer to the weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732233376119

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Reponse to All Reviewers

comment:
Dear All Reviewers,

Thank you for your thorough review and valuable feedback on our work.

We have carefully addressed all comments and concerns raised during the review process. We would greatly appreciate your evaluation of our responses and welcome any additional feedback to enhance the quality of our work further.

Thank you for your time and consideration.

Best regards,

Authors.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732499872910

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Look forward to your reply!

comment:
Dear Reviewers **5KSE**, **876W**, and **527o**:

Thank you for your valuable comments on our work. As the rebuttal deadline approaches, **we would greatly appreciate hearing from reviewers 5KSE and 876W regarding whether our responses have adequately addressed their concerns.**

We are particularly grateful to reviewer 527o for the comprehensive and professional review and detailed suggestions that have significantly improved our work. **We look forward to receiving feedback from 527o on our responses to the follow-up questions.**

Below is a summary table of our revisions addressing the concerns raised by reviewers 5KSE, 876W, and 527o:

| **Update**                                                                    |         **Section**        | **Reviwers** |
|---------------------------------------------------------------------------|:----------------------:|----------|
| Compare with LLaMA3.1                                                     | Appendix Section A     | 5KSE     |
| Compare with OpenAI                                                       | Openreview Comment Box | 5KSE     |
| Explanation for model design and real-world applications                      | Openreview Comment Box | 876W     |
| Addressing the followed-up concerns related to efficiency and scalability | Openreview Comment Box | 527o     |



- **5KSE:** [llama3.1](https://openreview.net/forum?id=plgLA2YBLH&noteId=89U0K3XzpH) and 
[openai](https://openreview.net/forum?id=plgLA2YBLH&noteId=nKbNepAOke)

- **876W:** [part 1](https://openreview.net/forum?id=plgLA2YBLH&noteId=rTRAsIZ2cH)
and [part 2](https://openreview.net/forum?id=plgLA2YBLH&noteId=x7Q5rzEYUn)

- **527o follow-up replies:** [part 1](https://openreview.net/forum?id=plgLA2YBLH&noteId=VVOSWg9Bvj)
and [part 2](https://openreview.net/forum?id=plgLA2YBLH&noteId=DzYR7V90uP)

---

Your continued engagement in this discussion is crucial for further improving our work. We welcome any additional feedback or suggestions you may have. Thank you so much!
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes Espresso Sentence Embeddings (ESE) for scalable sentence embedding learning, aiming to address the challenges of fixed embedding sizes and model depths in existing sentence embedding methods. 
Reviewers generally recognized the value of the paper's core contribution, which is enabling scalable sentence embeddings. Multiple reviewers gave positive comments on the presentation and experiments, as well as the clear research objectives and well-designed training methodology.  Although this paper is on the borderline, most of the concerns are fixed in the discussion period. Two negative reviewers do not give any response after rebuttal, and after reading their review, I regard them as invalid.