ESE: Espresso Sentence Embeddings
Authors: Xianming LI
Zongxi Li
Jing Li
Haoran Xie
Qing Li
Venue: ICLR 2025 Poster (2025)
Keywords: sentence embeddings
semantic textual similarity
information retrieval
retrieval-augmented generation
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We are the first to learn sentence embedding with information compression, presenting scalable embedding inference to both model depths and embedding sizes.

Abstract: High-quality sentence embeddings are fundamental in many natural language processing (NLP) tasks, such as semantic textual similarity (STS) and retrieval-augmented generation (RAG). However, most existing methods leverage fixed-length sentence embeddings from full-layer language models, which lack the scalability to accommodate the diverse available resources across various applications. Viewing this gap, we propose a novel sentence embedding model Espresso Sentence Embeddings (ESE) with two learning processes. First, the learn-to-express process encodes more salient representations to shallow layers. Second, the learn-to-compress process compacts essential features into the initial dimensions using Principal Component Analysis (PCA). This way, ESE can scale model depth via the former process and embedding size via the latter. Extensive experiments on STS and RAG suggest that ESE can effectively produce high-quality sentence embeddings with less model depth and embedding size, enhancing inference efficiency. The code is available at https://github.com/SeanLee97/AnglE/blob/main/README_ESE.md.

Published as a conference paper at ICLR 2025
ESE: ESPRESSO SENTENCE EMBEDDINGS
Xianming Li Zongxi Li
Jing Li 1,3,, Haoran Xie Qing Li
Department of Computing,
Research Centre on Data Science Artificial Intelligence, The Hong Kong Polytechnic University, Hong Kong SAR
2 School of Data Science, Lingnan University, Hong Kong SAR
hk,
hexiee hk, {jing-amelia. 1i, ging- wtt.elllwpppy..ccmmmmmmm edu.
ABSTRACT
High-quality sentence embeddings fundamental in many natural language processing (NLP) tasks, such as semantic textual similarity (STS) and retrieval- augmented generation (RAG). However, most existing methods leverage fixed- length sentence embeddings from full-layer language models, which lack the scalability to accommodate the diverse available resources across various ap- plications. Viewing this gap, we propose novel sentence embedding model Espresso Sentence Embeddings (ESE) with two learning processes. First, the learn-to-express process encodes more salient representations to shallow lay- ers. Second, the learn-to-compress process compacts essential features into the initial dimensions using Principal Component Analysis (PCA). This way. ESE can scale model depth via the former process and embedding size via the lat- ter. Extensive experiments STS and RAG suggest that ESE can effectively produce high-quality sentence embeddings with less model depth and embed- ding size, enhancing embedding inference efficiency. The code available http:////htttu...commm/aaaaa////
READDEE ESE md
INTRODUCTION
Sentence embedding learning (Cer al. 2018 Reimers & Gurevych, 2019 Gao et al. 2021 Li
& Li. 2024a b) is a crucial yet challenging task in the NLP research. It aims to capture essen- tial semantic and syntactic information in language, benefiting various scenarios such clustering
(Reimers & Gurevych 2019), semantic textual similarity (STS) (Li Li 2023 Zhang et al. 2024
and cetrreval-auumeetter generation (RAG) (Gao et 20233
In the common deployment practices, the pipeline of applying sentence embeddings unfolds in two typical stages: (i) computing the sentence embeddings via forward pass and (ii) employing these
embeddings in downstream tasks. Existing work (Reimers & Gurevych] 2019 Gao et al., 2021 Li
Li 2024a] inter alia.) typically adopts entire Transformer (Vaswani et al. (017) layers and full embedding sizes for all tasks to ensure optimal performance, regardless of the varying resources and requirements across applications. It can result computational redundancy and fails to scale well to the diverse resources available in downstream scenarios (Kusupati et al. 2022). To address this challenge, Matryoshka Representation Learning (MRL) concurrently trains multiple embed- dings with cascading dimensions to enable scalable embedding sizes while preserving maximum semantics (Kusupati et al., 2022). However, MRL employs full Transformer layers for embedding inference, in which high computational costs persist when using Large Language Models (LLMs)
with very deep architectures (Wang et al. 2023 & Li 2024a b
Lee et al. 2024).
Corresponding Author.

---

Published as a conference paper at ICLR 2025
Traditioaal Scriince Enthelling
Mairyesaka Embbedding
Esprcseo Scntence Emtedding
Trankraer Hackhan
Endwddding Qualiy
Traadirme Rackbaar
Fedndding Quality
trankener Badkeac
Tadedding
Nidle kywer
bxlooaus Spuni Skee stonboaigggg
lekkezz Spadt Sae
belecau Sausi
The comparison of traditional (left), MRL (middle) (Kusupati et al. 2022). and the pro- posed ESE (right) sentence embedding models. The gray blocks represent Transformer layers that fine-tuned in the full model setting, while the coffee-colored blocks indicate layers used in scal- able settings. In traditional and MRL approaches, while the final layer produces high-quality embed- dings, the intermediate layers yield poor-quality embeddings. In contrast, the proposed ESE enables flexible scaling along both model depth and embedding dimension.
Figure 1:
We propose novel Espresso Sentence Embeddings (ESE) to further improve inference efficiency by enabling scalability on model depth. ESE consists of two key processes, First, the learn-to- express process allocates more crucial latent representations to shallow layers by weighting embed- dings at different levels. Second, the learn-to-compres process condenses essential features into initial dimensions by exploring the inner dependencies of embedding dimensions through Princi- pal Component Analysis (PCA). This mechanism allows the firs---diimenson sub-en sub-embeddings to exhibit robust performance, offering flexibility in dynamic settings with varying embedding capac- ity requirements but with guaranteed performance by adjusting the value of k. Figure [ illustrates ESE's overview and its differences from the MRL (Kusupati et al. 2022) and traditional embed- dings (Reimers& & Gurevych [019 Gao et al. [021 Li & Li| 2024a). As depicted, ESE is designed to encode more salient features into shallow layers and small embedding dimensions simultane- ously. Thus, it enables scalable sentence embeddings that adapt various settings of model depth and embedding size, allowing for more flexible accommodation of diverse computing resources. In contrast, MRL focuses solely on scaling embedding size, while traditional models lack scalability. Additionally, the PCA implementation of ESE at various layers of embedding learning can help organize the learned features in order according to their significance; it allows easier training than MRL by jointly training multiple varying-dimension dimension embeddings to scale embedding sizes.
To the best of our knowledge, we are the first to learn sentence embedding with information com- pression, presenting scalable embedding inference to both model depths and embedding dimensions.
We extensively experiment across STS and RAG tasks to evaluate ESE. First, the main results on the STS benchmarks show that ESE performs competitively in full settings than non-trivial baselines and shows significantly better results with shallow model depth or smaller embedding size. For in- stance, ESF enhances BGE's (Xiao et al. 2023) shallow-layer embeddings from 45.60 to 66.27, showcasing the effectiveness of our method. Then, ablation studies indicate that all modules pos- itively affect ESE, and its performance is sensitive to smaller PCA compression sizes. Moreover, the RAG experiments show that ESE improves retrieval across varying embedding sizes and model depths, underscoring the potential in various application scenarios. Finally, we further discuss ESE and present the following findings: (i) ESE shows better inference efficiency on both STS and RAG tasks; (ii) In visualization, ESE's scaled embeddings exhibit greater overlap with the unscaled ones, indicating its effectiveness in information compression and superiority in scaling embeddings.
In summary, our contributions are as follows:
We are the first to employ the information compression technique to scale sentence embeddings.
Our novel ESE model allows for scalable embeddings in both model depth and embedding size.
Extensive experiments on STS and RAG show ESE's superiority in producing effective embed- dings with reduced model depth and embedding size, enhancing inference efficiency.
2

---

Published as a conference paper at ICLR 2025
Embrdding Learning
Espresso Sentence Embedding
Vanilla
Espresso
SHFAT SinCSF [Aagle
Information Squaoe
PCA Compression
Appa
X,
#2
*2
#1
PCC
The ESE framework. The left part is the vanilla Transformer backbone, where each layer is not scalable. The center part is the ESE training with the learn-to-express (to scale model depths) and compprss (to scale embedding sizes) processes. The right part the trained Espresso Transformer backbone, where each layer is scalable (marked with double-ended arrows).
Figure 2:
RELATED WORK
Our work is line with the sentence embedding learning research. While early efforts primarily focused on word embeddings (Mikolov et al., 2013), the recent trend has shifted towards sentence embeddings as they can capture representations from richer contexts. In the training methods, many
studies adopted supervised approaches align embeddings to human senses (Conneau al. 2017
Cer et al. [0188] Reimers & Gurevych| 2019 Li & Li. (2024)) And contrastive learning techniques
(Carlsson et al., [2200] Gao al. [202]] Chuang et al. 2022 Xu et al.
2023
Liu 2024) have
recently become popular to involve in-batch comparison of positive and negative pairs. As for
model architectures, with the LLMs breakthrough in NLP (OpenAT 2022 Touvron et al. 2023)
LLM-based sentence embedding models have drawn growing attention (Li & Li 2024a
Wang
al.
2023
Li
Li 2024b Muennighoff al., 2024) for more effective context exploration.
However, the common practices deploy embeddings with full model layers and predefined embed- ding sizes. It inevitably constrains the scalability of sentence embedding use in downstream applica- tions, particularly in scenarios where resources are limited. To address this constraint, Matryoshka Representation Learning (MRL) was introduced to allow scalable embedding sizes (Kusupati et al.) 2022). Yet, it still relies on full Transformer layers for embedding inference. In contrast, ESE has dual scalability in both model depth and embedding size, largely enhancing the efficiency of embed- ding inference. Moreover, MRL trains multiple embeddings in varying sizes, whereas ESE employs PCA for embedding size compression. Here, the PCA in training can help organize the features in order and allow more effective embedding learning. While some other methods (Zhu et al.) 2018 Gupta et al.) 2019 Zhao et al.] 2022) applied PCA-alike compression to pre- or post-process trained embeddings, ESE is the first to integrate PCA into sentence embedding training.
ESPRESSO SENTENCE EMBEDDINGS
This section describes how ESE learns scalable sentence embeddings by elaborating the learn-to- express and learn-to comppess processes. Figure depictts its framework.
3.1 ENCODER
We start the discussion with how we process the input, where the pretrained language model are used as an encoder transform the text into dense sentence embeddings. Here, BERT (Devlin
et al. 2019) or LLMs (Touvron et al. 2023 Bai et al. 2023) serves as the backbone to encode text as follows:
Pooling(Enccder*** (x)) e Rx1
(1)
where i E n. is first-i layers of the n-layer Transformer backbone, and ls E [1, d denotes first- k-dimensional sub-embeddings of the full d-dimensional embeddings. For BERT-based models,
3

---

Published as a conference paper at ICLR 2025
the "CLS" pooling strategy following Li & Li (024a). For LLM-based models, follow previous practices et al. 2023 & Li 2024b) to transform all layers into bidirectional and adopt the mean pooling. Here, select BERT and LLMs like Qwen and LLaMA the encoder since they are widely used Transformer-based models for sentence embeddings. It allows for evaluating the effectiveness and generalization of the proposed ESE across various model scales.
3.2 LEARN TO EXPRESS
The encoded embeddings X will then go through a leare-to-express process to allow scalable model depths. Our idea to encode more essential representations into shallow layers. Concretely, we cache each layer's sentence embeddings obtained by Eq. for i E [1. 1] and then jointly train their firs----dimension sub-embeddings by a weighted loss. The loss is computed as follows:
Die
Wy * loss (o.s x.9) loss (X.s((;,,, 1 g).
Here w, denotes the weight for i-th layer. We provide two implementations for the weights w;, The first implementation is to parameterize so that the weights can be adjusted dynamically during the model training. The second implementation sets w, = 1
I++n((]- This way, the weights decrease
as the layer depth increases, allowing the shallow layers to capture more crucial information for embedding scalability in model depths. Note that we leave the last layer unweighted due to its critical role in capturing sentence-level semantics (Li & Li 2024b). The second implementation is applied by default as it does not introduce any additional parameters, making the implementation more efficient as the efficiency is always an important factor for the proposed ESE. The loss( can be any loss function for sentence embedding learning, e-g., contrastive loss (Gao et al. [02TT) or AnglE. loss (Li & Li 2024a). We use the latter one by default. G is the positive or negative sample indicators for loss computation.
3.3 LEARN COMPRESS
The Caam-to-eepress method (Section 3.2)
focuses scaling embeddings across layers for model depths. Then, the learn-to-compress aims to enable scalable embedding sizes within each layer while preserving maximum semanties. The goal is allow ESE condense more essential infor- mation into sub-vectors of the initial embedding dimensions, as shown in the "Information Squeeze" box in Figure 2 shows information distributions, where the left-hand-side part contains higher "information density" and richer representation than the right-hand-side
To achieve the above, we use Principal Component Analysis (PCA) to reduce embeddings' dimen- sionality from size d to k while preserving the crucial semantics, where d and k denote full and compressed embedding sizes, respectively. We compress the inner dependencies within embedding dimensions instead of directly compressing the sentence embeddings. This step aims to enable ESE to capture nuanced pross-dimension interactions without oversimplifying the embedding structure.
Concretely, we first compute the embedding inner-dependency matrix by scaled dot product:
Xdd -xx Vd
A softmax
Rdxd
where i denotes first-i Transformer layers. X Rdx1 are full sentence embeddings obtained from Eq. [ The design pertains to the self-attention of Transformer Vaswani et al. (2017) reflecting the weights across varying dimensions. Then, we employ PCA, implemented with Singular Value Decomposition (SVD), to compress its information. It thus results in A = U, v based on which we obtain top-k principal components (i.e. dependencies) for each embedding, as follows:
A
U U.ddd1kk.
1sk,1 dx
where 1 < k < d. 1 d, 1: means the selection of the first-d rows and first-k columns, and A indicates the compressed anner-depnddecyy matrix. The diagonal matrix Σ; displays the singular values Ti,,, FiA & on its diagonal, arranged in descending order according their magnitude.
4

---

Published as a conference paper at ICLR 2025
Subsequently, we apply the top-k inner-dependency matrix A (with compressed self-attention-alike weights) to the original X It results in compressed sentence embeddings as follows:
X = X E RAXI pew
(5)
Although the above process works only in training, PCA may cause higher computational costs in embedding inference. To tackle this issue, we align the truncated first k-dimensional sub- embeddings to the PCA-compressed ones to reduce the cost. The aligmment involves mean squared error and Kullback-Leible divergence losses (Kim et al. 2021).
As an innovative approach, we employ a weighted strategy across layers, alignment with the learn- to-express process. to coherently coordinate the compression of embedding sizes across scalable model depths, as follows:
wi align(X3 XX. x align (X.,, x;),, pon
pen
(6)
align(z,)) MSF(z, y) KIDivv(r,))
where MSE(.) and KLDiv(.) denote the mean squared error and Kullback-Leibler divergence, re- spectively. w; means the weight for the i-th layer. It has the same setting as Eq. 2 By optimizing Lie. the truncated first k-dimensional sub-embedding will aligned to PCA-compressed ones. It allows the direct use of the first k-dimensional sub-embeddings at inference without repeatedly performing PCA and improves inference efficiency.
3.4 JOINT LEARNING
Finally, we jointly train the learn-to-expee -eeeresss and learn-to- com--es- processes within a unified frame- work. Their respective objectives are seamlessly integrated into the overall learning objective of the proposed ESE, as follows:
L = aLie + BLis
where a and A are the weights trading off the two processes. We set both to 1 by default. By jointly optimizing the two objectives, ESE trains its sentence embeddings be scalable in both model depths and embedding sizes.
4 EXPERIMENT
In the experiments, we first present the main intrinsic results of STS in Section 4.1 Then, we probe into more details of ESE output via an ablation study in Section Finally, we further discuss ESE from varying perspectives in Section 4.4 to provide more insight.
4.1 STS EXPERIMENTS
Following common practices (Gao et al. 2021 &i Li 2024a), we adopt STS for intrinsic evalua- tion assess the quality of trained sentence embeddings. It measures how well embeddings capture semantic similarity between sentences, indicating their capacity for effective text representation.
Setup. the comparison, we include two baselines: the scalable embeddings MRL (with scalable embedding sizes) (Kusupati et al. 2022) and RAW embeddings (without any scaling operations).
For the datasets, we train sentence embeddings on MultiNI. (Williams al. 2018 and SNLI (Bow- man et al. 2015) datasets following previous studies. Evaluation of model performance is conducted on the STS benchmark computed by SentEval (Conneau & Kiela 2018), where Spearman's 's corre- lation in the "all" setting is reported the evaluation metric. To enable comprehensive evaluation, this benchmark comprises seven widely used STS datasets: STS 2012-2016 (Agirre et al. 2012 2013 2014 2015: 2016), SICK-R (Marelli et al. 2014), and STS-B B (Cer et al. 2017).
For model settings, we examine two popular BERT based backbones, bge-base-en-vl.5 (iio et al, 2023) and UAE-Large-V/ (Li & Li 2024a) and M- backbone, Qwen1.5 (Bai et al.
5

---

Published as a conference paper at ICLR 2025
Table 1: STS benchmark results. The last column (< Avg.) the average results of shallow layers (except the last one), while the remaining correspond to the last-layer results. Avg.: average results over varying benchmark datasets. RAW: the original model: MRL: Kusupati et al. (2022). The coffee-colored cells: the best results for each backbone model; boldfaced numbers: the overall best results. For < Avg.. ESE performs significantly better than baselines: p-value < 0.05 (paired t-test).
Model
STS12
STS13
STS14
STS15
STS16
STS-B
SICK-R
Avg.
Avg.
bge-base-en-vI.5
(iao al.
2023
RAW
78.03
81.18
82.27
87.96
85.47
86.41
79.88
83.46
45.60
MRI. 75.90
87.87
83.97
88.92
85.07
87.17
79.18
84.01
46.18
+ ESE
77.70
86.97
83.57
89.43
86.16
87.27
80.32
84.49
66.27
UAE-Large-V
(Li Li (0224))
RAW
79.09
89.62
85.02
89.51
86.61
89.06
82. 10
85.86
44.80
MRL
78.26
90.19
84.91
89.48
86.17
88.49
79.28
85.25
44.97
ESE
79.64
90.40
85.76
90.33
86.64
88.54
81.09
86.06
59.12
Qwen1.5-0.5B
(Bai al. 2023)
RAW
75.91
83.77
80.04
86.05
82.91
85.32
78.98
81.85
56.59
MRL 76.30
85.04
80.68
86.15
83.12
85.65
79.45
82.34
58.22
+ ESE 76.43
85.70
81.75
86.30
83.67
85.76
80.16
82.82
59.99
2023). We use the recent popular AnglE (Li & Li
2024a) loss as the sentence embedding training loss. The initial learning rates set to 5e 5 and 2e 4 to train BERT- and LLM-basedd
models, respectively. For efficient LLM fine-tuning, we utilize LoRA (Hu et al. 2021 Dettmers
al. 2024) with parameters lora.r 32, lora.alpha
32, and lora_dropot
0.1. For the
ESE setup, the compression dimension & is set to 128 by default, and the weights and B for joint
learning (Eq. are set to
Main Results In the main STS experiments, we extract embeddings from the last 12 Transformer layers for comprehensive analyses. Then, we follow common practices report results from the last layer of all benchmark datasets, including their average (Avg.). Moreover, we report the average results from the shallow layers (excluding the last one) assess the scalability of ESE across model depths (< Avg.). The results are shown in Table I where we can draw the following observations.
First, ESE shows competitive and even better performance compared to RAW across every backbone in the last layer results. It indicates that the scalability learning of ESE does not adversely impact its last-layer performance. The marginal performance gain could be attributed to the use of PCA in organizing features, which potentially facilitates more effective embedding learning even in an unscaled setup. In contrast, MRL fails to outperform RAW in UAE's last layer, possibly due to chaos introduced by its multi-embeddiig learning strategy. Second, ESE significantly outperforms baselines in the < Avg. setup, with 20.67 gain to BGE. indicates ESE's capability of effectively scaling embeddings across different layers. The improvement is less notable in L.-M-based models than in BERT-basce models, possibly due to LoRA's limitation on updating parameters in LLMs.
To further examine the scalability ESE model depths and embedding sizes, we show the STS results over embedding sizes from every layer in Figure While all the models achieve competitive results using the full-sized embeddings from the last layer, RAW and MRL exhibit inferior perfor- mance in the shallow layers. They even demonstrate performance fluctuations as the layers deepen. In contrast, ESE can significantly improve the performance of the shallow layers. For example, the BGE achieves result over 60.00 at the 11th layer, while BGE-MRL requires ten layers to reach that level. In contrast, BGE-ESE only requires two layers and consistently improves until the last layer, reaching score of 84.49. This shows the effecctiveness of ESE's learn-to-express process in encoding the more important information into shallow layers.
Moreover, ESE consistently performs better over embedding sizes at each layer, indicating the ef- fectiveness of its can--to-compres process. The performance gain is more significant at shallow layers, potentially because they contain richer information, resulting in more effective compres- sion. Notably, ESE's stable performance since the 128 dimension suggests that a -28-dimensional embedding may capture the majority of salient features.

---

Published as a conference paper at ICLR 2025
BGE Based
RAW
MRL
ESE
Owen Based
layer
layer
layer
layer
layr
0-0
40
20
250
500
750
250
500
150
250
2003
150
250
500
750
240
2003
250
750
1eyer
layer
layer
layer 36
Layce
17
layer
Spenmman's
40
laver
layer
layer
250
500
750
250
300
750
250
500
750
250
500
750
250
500
7.00
250
500
750
bayer 10
loyrer
Jayer
laye: -
loger 23
layer 24
60
Embedding
Figure 3: Results of the STS benchmark for each of the last 12 layers of BGE- backbone (left part) and Owen-based backbone (right part). For each layer's result, the x-axis shows the embedding size, and the y-axis shows the average Spearman's 's correlation over varying benchmark datasets.
84
Spearman''
Spearnan's
83
1000
2000
3000
4000
1000
2000
4000
Embedding Size
Fmbeeding Size
(a) 1ayer=16
(b) layer=32
Figure 4: The average STS benchmark results of various embedding sizes for the half layer (first 16 layers) and last layer (full 32 layers) of LLaMA2 7B (Touvron al. 2023) with the RAW, MRL., and ESE training.
Furthermore, considering the Qwenl.5 0.5B is relatively small LLM (Lu et al. 2024), we also evaluate the effectiveness ESE on larger L.MSs by testing LLaMA2 7B (Tovrron et 2023) on the STS benchmark, as shown in Figure [ Here, we report the results of two typical layers, i.e. the middle layer (1ayer=16) and the last layer (layer=32), outperforms baselines across different model depths and embedding sizes. Notably, ESE achieves a significant improvement at shallow layers, demonstrating its superior scalability. We can also see that there is performance drop observed the middle layer for baselines and ESE. This could be attributed to the LoRA's limitation in sentence embedding learning. Nevertheless, ESE can mitigate this limitation, evidenced by its lower drop rate than baselines.
evident that the proposed ESE consistently
7

---

Published as a conference paper at ICLR 2025
4.2 ABLATION STUDY
While Section 4.1 shows the overall effectiveness of ESE, we conduct ablation studies of ESE on the STS benchmark to assess the effectiveness of each component. The results are presented in Table
We first evaluate different compression strategies to scale embedding sizes: the default PCA on embedding dependencies, PCA on sentence embeddings directly (i.e., PCA on X the max-k embedding dependencies (i.e., max k of dependency matrix A in Eq. without PCA), and the ablation without any compression. All components contribute positively to the learning of both the last-layer and the shallow-layer embeddings. Interestingly, the performance drops substantially without compression, suggesting that compression can also help the learning of scaling across layers.
As to the weighting strategies, compare the performance of without, dynamic, and default layer weighting. The results indicate that without layer weight performs worse than dynamic and default both the last layer and shallow layer learning, particularly the latter. It suggests that assigning dif- ferent layer weights allows shallow layers to capture more essential information. While the dynamic layer weighting strategy outperforms the default strategy in the last layer, performs worse for shal- low layers. This might be because dynamic weighting places more importance the last layer of learning, which negatively affects the shallow layers' performance, thereby hurting the scalability of model depth. The default layer weighting strategy consistently improves the performance of both the last and shallow layers. This could be because the performance improvements in the shallow layers propagate to the deeper layers, resulting in an overall performance improvement.
Furthermore, we assess the impact of different PCA compression sizes (k) during training. We train ESE NLI datasets with varying compression sizes and evaluate their STS performance in the full setting. For comparison with the baselines, we report ESE's and baseline's results in Figure 5 The results indicate that ESE consistently outperforms the baselines across different compression sizes, highlighting the effectiveness of information compression in enhancing sentence embeddings. However, there exhibits a wwwrease-thenndeeeeeee because the model with large compression sizes may overlook critical details, whereas that with small compression sizes might not have adequate capacity retain all essential information.
trend (peaked at 128). This is
Table 2: Ablation study of ESE on STS bench- mark. Avg. and < Avg. denote the aver- age Spearman's correlation of the last layer and shallow layers (except the last), respectively.
ESEE
84.4
MRL
Ax0h Spearman's
RAW
84.2
84.0
Model
Avg.
Avg.
Compression Strategies
83.8
PCA dependencies (default)
84.49
66.27
83.6
PCA on sentence embeddings
84.30
65.69
max-k & embedding dependencies
84.22
65.47
32
64
128
256
384
512
640
768
none (w/o compression)
84.13
50.18
compression size
Layer Weighting Strategies
Figure 5: The STS benchmark results of ESE trained on different PCA compression sizes. Dashed lines indicate baseline results, serving reference points.
without weight dynamic weight
84.25
64.56
84.63
65.94
default weight
84.49
66.27
4.3 RAG EXPERIMENTS
The previous discussions center on the intrinsic evaluation STS. We conduct an extrinsic eval- uation to examine how ESE helps downstream retrieval and generation deployment. We compare different sentence embedding models based on bge-base-en-vl.5 (Xiao et al. 2023) on HotpotQA dataset. It contains 113k Wikipedia factoid question-answer pairs. We employ faiss (Johnson al.
'The Raw setting does not support embedding compression, and supports embedding truncation rather than compression. Hence, we depict their highest performance as light-colored lines in Figure
8

---

Published as a conference paper at ICLR 2025
2019 to index and retrieve documents based sentence embeddings. Since sentence embed- dings are normalized, cosine similarity or Euclidean distance can used. We follow the common practice (Douze et al. 2024) of using Euclidean distance, i.c., IndexFlatL2 in faiss, the similarity measurement. For each query, we retrieve the most relevant context from all Wikipedia documents in HotporQA and generate responses using LLLMM2 7B (Touvron et al. 2023) via a prompt in Table For the QA evaluation, we adopt the popular metric Exact Match.
The RAG results of the last layer with vari- ous embedding sizes are listed in Table B We observe that ESE outperforms baselines in all embedding sizes. The improvements are more evident with smaller embedding sizes, for ex- ample, 2.42% and 1.80% higher than RAW and MRL with 64-dimensional embedding, re- spectively. Moreover, we compare the perfor- mance when using the sentence embeddings from the half layer (n = 6). ESE shows 0. 0.72% RAW
Table RAG exact matching scores with dif- ferent embedding sizes of different sentence embedding models on HotporQA. bge-base-en- vI.5 serves the backbone sentence embed- ding model (marked with RAW).
Model
embedding size
RAW
MRL
ESE
and 0.44% improvement compared and MRL, respectively. These results demon- strate that effective ESE sentence embeddings, scalable in terms of both embedding sizes and model depths, can further benefit RAG.
641
29.86
30.48
32.28
128
38.85
38.90
39.28
256
42.05
42.20
42.50
512
44.16
44.20
44.44
768
45.06
45.09
45.31
Table The prompt used LLaMA2 7B for the RAG task. The "(context)". and "[question]" are the placeholders for the input context and question.
As an advanced QA system, your role is to provide accurate and straightforward answers based the provided context. Utilize the following information answer the given question, directly output the straightforward answer, and do not explain
Context: [context).
Question: {question).
Requirement: For 'yes/no' question, directly output 'yes' or 'no'. For the 'who' question, directly output the people's names. For the 'when' question, directly output the date or time, so and forth.
Output:
4.4 DISCUSSION
Efficiency. We compare ESE with MRL (Kusupati et al. 2022) in terms of efficiency and effec- tiveness on STS, as shown in Figure ba the same, the performance gaps between MRL's and ESE's embeddings are significant. This shows that our approach can effectively improve the shallow layer's 's task-specific representation, underscor- ing the scalability of the ESE model. We also examine how ESE affects RAG efficiency. Figure 6bb illustrates the time consumed by the encoding and retrieval stages of the RAG pipeline with various embedding sizes. The results indicate that smaller embedding sizes are more efficient than larger ones, and ESE is slightly more efficient than baselines. It is because the ESE's high-quality embed- dings in varying scaled sizes allow easier and more effective indexing, improving RAG efficiency.
Although the inference times at different layers are largely
Compression Quality. To intuitively compare the information compression quality between ESE and baselines, use t-SNE (Van der Maaten & Hinton 2008) to visualize 768-dimensional (full) and 22s-iimnsioonll (scaled) sentence embeddings in 2D space, as shown in Figure 7 ESE exhibits higher overlapping rate between the dark- and light-color dots than baselines, showing ESE's high fidelity in the compressing process, i.c., ESE's s first-128 sub-embeddings carry most key information from their full embeddings. To quantitatively measure the difference, calculate the average

---

Published as a conference paper at ICLR 2025
Encoding and retriieal time (sevonds)
Layer
MRL Tayer 12
-10
ESE Layer 12
Inforesos Time 60
16-
150
125
100
ESE Time
75
ESE STS
10
12
200
400
600
800
Layor
Embedding dimensions
(b)
Figure 6: (a) Inference time and STS performance vs layers. (b) The encoding and retrieval time with different embedding sizes on the HotpotQA dataset.
-23
(a) RAW (Avg. (istance=32.63)
(b) MRL (Avg. distance=25.75)
(c) ESE (Avg. distance=10.36)
Figure 7: The I-SNE visualization of sentence embeddings the STS-B test bge-base-en- vI.5 serves as the backbone for RAW, MRL, and ESE models. The dark-color dots denote the full fi8-dimensional sentence embeddings, while light-color ones represent the first 128-dimension sub-embeddings.. The Avg. means the average distance between the 768-dimensional and the 128- dimensional embedding points.
distance between the 768- and 128-dimensional embedding points and put the numbers in Figure 7 is evident that the proposed ESE achieves the lowest average distance among the baselines. effectively condense essential information into the initial embedding
This indicates that ESE dimensions by the learn-to-come comppeos process. It can reduce information loss when using small first- k sub-embeddings for downstream applications.
CONCLUSION
In this paper, we have presented a novel sentence embedding model called Espresso Sentence Embeddings (ESE). It is the first work to support both model depth and embedding size scal- ing with two novel processes. First, the learn-to-eppress enables model depth scaling by allowing more important features to be captured by shallow layers. Second, the lear-tt-ccoppress allows embedding size scaling by employing information compression compress embeddings' inner de- pendencies. The dual scalability of ESE enables it to be truncated into different scales, making adaptable to varying computational resource requirements. Extensive experiments STS and RAG have suggested that ESE can effectively produce high-quality embeddings with less model depth and embedding size, enhancing embedding inference efficiency.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
Xianming Li and Jing Li's work has been supported by a grant from the Research Grants Coun- cil of the Hong Kong Special Administrative Region, China (Project No. PolyU/252008211), the Innovation and Technology Fund (Project No. PRP/047/22FX), and PolyU Internal Fund from RC- DSAI (Project No. 1-CEIE). Zongxi Li's work has been supported by Faculty Research Grants (SDS24A2) of Lingnan University. Hong Kong, and the Faculty Development Scheme (Project No. UGC/FDS 6/E10/23), of Hong Kong Research Grants Council; Haoran Xie's work has been sup- ported by the Faculty Research Grants (SDS24A8) and the Direct Grant (DR25E8) of Lingnan Uni- versity. Hong Kong; Qing Li's work has been supported by Hong Kong Research Grants Council through Research Impact Fund (project no. R1015-23).
Here, we sincerely thank the reviewers and ACs for their valuable input, which has greatly improved work.
REFERENCES
Eneko Agimr,, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agire. SemEval-2012 task 6: A pilot on semantic textual similarity. In *SEM 2012: The First Joint Conference on Lexical and Computa- fional Semantics Volume I: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pp. 385-393, Montréal, Canada, 7-8 June 2012. Association for Computational Linguisties. URL https::/aaclantholloyy.. .rg/ 1051
Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo. *SEM 2013 shared task: Semantic textual similarity. In Second Joint Conference on Lexical and Computational Se- mantics (*SEM), Volume I: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity, pp. 22.43. Association for Computational Linguistics, 2013.
Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Nonzalez-Agires Wei- wei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. SemEval-201 4 task 10: Multilingual semantic textual similarity. In Proceedings of the 8th International Workshop on Semantic Eval- uation (SemEval 2014), pp. 81-91, Dublin, Ireland, August 2014. Association for Computational wtwp://ccr/////111e11
Linguistics. doi: 10.3 15v 1/$14-200
Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Wei- wei Guo, Iñigo Lopez-Gazpio. Montse Maritxalar, Rada Mihalcea, German Rigau, Larraitz Uria, and Janyce Wiebe. SemEval-2015 task 2: Semantic textual similarity, English, Spanish and pilot on interpretability. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), 5), pp. 252 252-263, Denver, Colorado, June 2015. Association for Computational Lin- guistics. doi: 10.18653/v /S555/15552..55 htL / a www.hllonhhooooooooooooooooooooo org/ 111--015255
Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe. SemEval-2016 task 1: Semantic textual similarity, monolin- gual and cross-lingual evaluation. In Proceedings of the 10th International Workshop on Seman- tic Evaluation (SemEvaal20016, pp. 497-511, San Diego California, June 2016. Association for Computational Linguistics. doi: 10.18653/w1/$16-11 1081.
URL [RL[]sttpe::/aaLaanoooeee org/
S16 1081
Jinze Bai, Shuai Bai, Yunfei Chu, et al. Qwen technical report. arXiv preprint arXiv:2309.16609 2023.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large an- notated corpus for learning natural language inference. In Proceedings of the 2015 Confer- ence on Empirical Methods in Natural Language Processing, pp. 632 642, Lisbon, Portugal, September 2015, Association for Computational Linguistics. doi: 10.15653//1/D15--0777 URL
httts:://acl anthol ooy. wwwpp//.gg/lll... 1075]
Fredrik Carlsson, Amaru Cuba Gyllensten, Evangelia Gogoulou, Erik Ylipää Hellqvist, and Magnus Sahlgren. Semantic re-tuning with contrastive tension. In International conference on learning representations, 2020.
11

---

Published as a conference paper at ICLR 2025
Daniel Cer, Mona Diab, Eneko Aginre, lñigo Sopez-Gapiio, and Lucia Specia. SemEval- 2017 task I: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2011) 7), pp. 1-14, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 1053/////11/172222 URL htRl:httss://att onthology...rgg/111122
Daniel Cer, Yinfei Yang. Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Con- stant, Mario
Cespeeees, Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. Uni- versal sentence encoder for English. In Proceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing: System Demonstrations, pp. 169-174, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10.18653/v1/DI8-202 URL ottess//acclancholooo.ooogg 2029
Yung-Sung Chuang, Rumen Dangovski, Hongyin Luo, Yang Zhang. Shiyu Chang, Marin Soljacic, Shang-Wen Li, Scott Yih, Yoon Kim, and James Glass. DifICSE: Difference-based contrastive learning for sentence embeddings. can Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4207 -01074218..0 Association for Computational Linguistics, July 2022. doi: 11118653/1/20222. naacl-main.311 m1in.311. httl:/tss///aaaaaaoooooooo.///// www.:aattw...oaaooooo////e///eee www..aaaccllac
Proceedings of the 2022 Conference of the North Ameri-
Alexis Conneau and Douwe Kiela. SentEval: An evaluation toolkit for universal sentence repre- sentations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May 2018. European Language Resources Associa- org /L18 -2269
tion (ELRA). URL
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. Supervised learning of universal sentence representations from natural language inference data. In Proceed- ings of the 2017 Conference on Empirical Methods Natural Language Processing, pp. 670- 680, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10. 10.18653/v1/D11-1100
anthol wwwp:/hooooyy.orr11111111000
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techhologies, pp. 4171-4186, 2019.
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre- Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. arXiv preprint arXiv:2401.0888 008281 2024.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6894-6910. Association for Computational Linguistics, 2021.
Simple contrastive learning of sentence
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai. Jiawei Sun, and Haofen Wang. Retricval- auggented generation for large language models: A survey. arXiv preprint arXiv:2312.10997 2023.
Rohit Gupta, Patrik Lambert, Raj Patel, and John Tinsley. Improving robustness in real-world neural machine translation engines. In Proceedings of Machine Translation Summit XVII: Translator, Project and User Tracks, pp. 142-148, Dublin, Ireland, August 2019. European Association for Machine Translation. httptt:::/// /BBLitttps//ccc.atttooooy...r
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2 (xXiv22166099660 096809 2021.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion scale similarity search with GPUs. JEEE Transactions Big Data, 73):5335-547, 2019
12

---

Published as a conference paper at ICLR 2025
Tachyeon Kim, Jaehoon Oh, Nakyil Kim, Sangwook Cho, and Se- Young Yun. Comparing kullback- leibler divergence and mean squared error loss in knowledge distillation. In Proceedings of the Thirtieth International Joint Conference Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021, pp. 2628-2633 ijcai.org, 2021.
Aditya Kusupati. Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ra- manujan, William Howard-Snyder, et al. Matryoshka representation learning. In Advances in Neural Information Processing Systems, volume 35, pp. -2233. Curran Associates, www..nuuu..pappppeeeesseee ccppaper siles/paperr2222//11..
Inc., 2022. URL http::t:t/ppoceeddnneeeee www..rooo....................... www.inu.r......
Jinhyuk Lee, Zhuyun Dai, Xiaoqi Ren, Blair Chen, Daniel Cer, Jeremy R Cole, Kai Hui, Michael Boratko, Rajvi Kapadia, Wen Ding, et al. Gecko: Versatile text embeddings distilled from large language models. arXiv preprint arXiv:2403.20322 2024.
Xianming Li and Jing Li. Angle-optimized text embeddings. arXiv preprint arXiv:2309.18771, 2023.
Xianming Li and Jing Li. Aoe: Angle-optimizee embeddings for semantic textual similarity. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 1825-1839, 2024a.
Xianming Li and Jing Li. Bellm: Backward dependency enhanced large language model for sen- tence embeddings. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, 2024b.
Zongxi Li, Xianming Li, Yuzhang Liu, Haoran Xie, Jing Li, Fu-lee Wang. Qing Li, and Xiaoqin Zhong. Label supervised llama finetuning. arXiv preprins arXiv:2310.01200 2023.
Kang Liu. SelCSE: Set operations using contrastive learning of sentence embeddings. In The Twelfth International Conference on Learning Representations, 2024. URL
net / htt////rrmmmddeee (ii-rrunndd-EEGSSHH...
Zhenyan Lu, Xiang Li, Dongqi Cai, Rongjie Yi, et al. Small language models: Survey, measure- ments, and insights. arXiv preprint arXiv:2409.15 15790, 2024.
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. A SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation 14), pp. 216-223. European Language Resources Association (ELRA), May 2014. URL Bacta__aperr_eee pdff
http: //ww: 1 con. www.:/ot.ooooooooooooeoooooooo http://oceeinnnn//e////e////////
Tomás Mikolov, Πla Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality. In 27th Annual Conference Neural Information Processing Systems 2013., pp. 31 3111-3119, 2013.
Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, al. Generative representa- tional instruction tuning. arXiv preprint arXiv:2402.09906, 2024.
OpenAl. Introducing chatgpt, 2022. URL
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, pp. -9900 Association for Computational Linguistics, 2019.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(1), 2008.
13

---

Published as a conference paper at ICLR 2025
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improv- ing text embeddings with large language models. arXiv preprint urXiv:2401.00366 2023.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. Proceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Compulational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1112-1122. Association for Computational Linguistics, June 2018.
doi: 10.18653/v 1N18-1101.
https httttps::/aaaaattollosseee org wowg////-/1100001
Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged resources to advance general chinese embedding, 2023.
Jiahao Xu, Wei Shao, Lihui Chen, and Lemao Liu. DistilICSE: Distilled contrastive learning for sentence embeddings. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 8153-815 3166..15.. Association for Computational Linguistics, December 2023.
Marina Zhang, Owen Skipper Vallis, Aysegul Bumin, Tanay Vakharia, and Elic Bursztein. RET- Sim: Resilient and efficient text similarity. In The Twelfth International Conference on Learning Representations, 2024. htRpsst/t:////// httttps://openreviee. wwt..e/eooruumm---222000....
Xuandong Zhao, Zhiguo Yu, Ming Wu, and Lei Li. Compressing sentence representation for mantic retrieval via homomorphic projective distillation. In Findings of the Association for Com- putational Linguisties: ACL 2022, pp. 774-781, 2022.
Yonghua Zhu, Xuejun Zhang, Ruili Wang, Wei Zheng, and Yingying Zhu. Self-representationn and pea embedding for unsupervised feature selection. World Wide Web, 11:1675-168,, 201 8.
14


Reviews:
[{"review_id": "LfYhtmLwW4", "reviewer": "Reviewer_5KSE", "summary": "The paper proposes an approach for generating sentence embeddings that are scalable both in terms of model depth and embedding size. The proposed Espresso Sentence Embeddings (ESE) model incorporates two key processes: the learn-to-express process and the learn-to-compress process. The learn-to-express process aims to encode crucial latent representations in shallow layers, while the learn-to-compress process uses Principal Component Analysis (PCA) to condense essential features into the initial dimensions of the embeddings. The paper demonstrates that ESE can produce high-quality embeddings that are competitive with or surpass existing methods on standard benchmarks for semantic textual similarity (STS) and retrieval-augmented generation (RAG) tasks, particularly at reduced model depths and embedding sizes.", "questions": "Please refer to the weakness section.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-organized and easy to follow. \n2. Experimental results are clearly presented, demonstrating the effectiveness of the proposed ESE method.", "weaknesses": "1. Lack of Comparison with Strong Baselines like Llama 3.1\n2. The paper does not include comparisons with OpenAI's embedding models, which are considered significant benchmarks in the field of NLP.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Gwf9nk7DNk", "reviewer": "Reviewer_527o", "summary": "This work introduces a method for fine-tuning sentence embeddings while simultaneously providing flexibility in model size and embedding size. The method consists of two parts: learn-to-express, which trains different model depths, and learn-to-compress, which focuses on compressing information into a sub-embedding. The authors provide extensive experimental results on STS datasets and RAG on QA, including visualizations for the quality of the embeddings as well as results on inference efficiency. For the ablation studies, they investigate PCA on different dependencies, the effect of embedding sizes for PCA, and weighting present in the loss function.", "questions": "## Q1:\n\n> The possible reason is the ESE's high-quality embeddings in varying scaled sizes allow easier and more effective indexing, thus improving RAG efficiency\n\nI do not see how this affects the indexing process. Can you elaborate a bit more?\n\n## Q2:\n\nCould you share the code for the inference used in the case of MRL or point me to the part of the code that does this in the anonymized repository?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The authors fuse two existing ideas (model truncation and embedding truncation) into a single training process, which is a somewhat limited novelty. However, this provides much-needed flexibility for real-world use cases. \n2. The method is backed up by empirical results on seven STS datasets, one RAG dataset, and ablation studies that justify method design. \n3. The paper uses figures to better explain concepts, which is nice. However, I often found it lacking details for experiments and ablation studies.\n4. This paper introduces the idea of training multiple configurations of a model at once. This is very useful for efficiency-critical and memory-constrained environments. The method can be handy for engineering use cases.", "weaknesses": "## 1. Unfocused contribution statement\n\n> To the best of our knowledge, we are the first to learn sentence embedding with information compression, presenting scalable embedding inference to both model depths and embedding dimensions\n\n> We are the first to employ the information compression technique to scale sentence embeddings\"\n\nIf by \"information compression\" the authors are referring to model depth, there are already existing papers on this topic, some of which are even cited (e.g., \"BeLLM: Backward Dependency Enhanced Large Language Model for Sentence Embeddings\" and \"Are ELECTRA's Sentence Embeddings Beyond Repair? The Case of Semantic Textual Similarity\"). Additionally, other cited works focus on reducing the embedding size (e.g., \"Matryoshka Representation Learning\"). Please clarify your wording and specify precisely how your method differs from these approaches.\n\n## 2. Incomplete related work\n\nThe related work section is quite incomplete, as it includes only a few well-known references, most of which are not closely connected to the paper's topic. For example, a few papers that address closely related topics (excluding the aforementioned references) are:\n\n- Interpreting Pretrained Contextualized Representations via Reductions to Static Embeddings\n- WhiteningBERT: An Easy Unsupervised Sentence Embedding Approach\n- What does BERT learn about the structure of language?\n- How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings\n- BERT Has More to Offer: BERT Layers Combination Yields Better Sentence Embeddings\n\n## 3. Unmotivated design choices\n\nThe paper is very empirical in nature, which is not a bad thing. While most of the method's design is supported by intuitive explanations, I find the rationale for choosing the \"align\" loss function insufficiently explained. Ideally, a more theoretical approach and justification for introducing the loss functions would significantly strengthen the paper.\n\n> Specifically, we use Euclidean distance, i.e., IndexFlatL2 in faiss, as the similarity measurement\n\nWhy use the Euclidean distance if you are optimizing for cosine similarity?\n\n> Note that we leave the last layer unweighted due to its critical role in capturing sentence-level semantics\n\nThe reference paper mentioned only tests this on LLMs, so what is the design justification for using it on MLMs?\n\n## 4. Potentially incorrect/misleading conclusions\n\n> It allows the direct use of the first k-dimensional sub-embeddings at inference without repeatedly performing PCA and improves inference efficiency\n\nI am not convinced by the second part of this statement. I can see that this improves efficiency ONLY if you consider using PCA. If you don't use any compression, the efficiency would be roughly the same, wouldn't it?\n\nFurthermore, Figures 3, 4, and ≺ Avg. columns for Tables 1 and 2 provide an unfair comparison -- RAW and MRL do not tune the inner transformer layers. This comparison is repeated a couple of times throughout the paper as an insight: \n \n> RAW and MRL exhibit inferior performance in the shallow layers\n\n> This shows the effectiveness of ESE's learn-to-express process in encoding the more important information into shallow layers\n\n> Notably, ESE achieves a significant improvement at shallow layers, demonstrating its superior scalability\n\nThis leads an unwary reader to an incorrect conclusion. It would make more sense to tune each layer separately and compare it with your method.\n\nFurthermore, Table 5 is a somewhat unfair comparison. Truncating the model depth does not mean the model has the same number of parameters as BERT_smal or BERT_tiny (due to different hidden dimensions). I would advise the authors to add the information on number of parameters and mention this while commenting on the results.\n\nThe comparison of distances after using TSNE is debatable. TSNE preserves the global structure, but measuring distances after TSNE does not make sense (cf. [https://datascience.stackexchange.com/questions/17314/are-t-sne-dimensions-meaningful](https://datascience.stackexchange.com/questions/17314/are-t-sne-dimensions-meaningful)). For more rigor, I would instead use MDS for this visualization and comparison.\n\n## Minor remarks\n\n- Figure 7 seems to lack details on which layer was used to create the embeddings on the subfigures and which 128 dimensions were chosen for the RAW method.\n\n- For Table 1, I would advise to put the links into the footnotes.\n\n- I know you use CLS pooling in your experiments, but the implementation for mean pooling is incorrect.\n\n```\n    if pooling_strategy == 'avg':\n        outputs = torch.sum(\n            outputs * inputs[\"attention_mask\"][:, :, None], dim=1) / torch.sum(inputs[\"attention_mask\"])\n```\n\n- Is Equation 6 missing a sum over k?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "039Hveh0Fn", "reviewer": "Reviewer_876W", "summary": "This paper follows the motivation of Matryoshka Representation Learning (MRL) to learn sentence representation at different granularities, allowing a single embedding to adapt to the computational constraints of downstream tasks. The main idea is to use the PCA compressed representation to supervise representation learning at each grain. They show some positive results on STS tasks, more specifically, compared to previous works, their method makes the embedding extracted in the inner layers better, which is aligned with their design. Also, some ablation studies are provided to support the plausibility of their design. Basically, the paper is clear and well-written. However, I hold some concerns with their motivation and design, see the weaknesses part.", "questions": "1) Is embedding dimension k also a hyperparameter? If so, does this mean that achieving different types or lengths of compressed representations requires multiple training iterations?\n\n2) Does \"PCA on sentence embeddings\" in Table 2 mean directly applying PCA on the high-dim representations? If so, from the storage angle, the advantage of the proposed method is limited?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper is well-written, showing clearly the gains from the inner layers upon raw and MRL design.\n2. I appreciate their experiments with different model scales and compression strategies, which clearly demonstrate the ablation and provide realistic comparison results.", "weaknesses": "1) The design is complex: there are a lot of hyperparameters to tune with, like the weight for Loss_le, Loss_lc, the combination of MSE and KL-div, and joint training. It is hard to provide insights into where the main gain comes from.\n\n2) Is embedding dimension k also a hyperparameter? If so, does this mean that achieving different types or lengths of compressed representations requires multiple training iterations?\n\n3) I doubt real-world applications for this method. Their main motivation is to reduce inference latency, where the most straightforward way is to use a smaller backbone. However, as shown in Table 5, the gain compared with BERT_small is marginal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Kpv29xhkho", "reviewer": "Reviewer_XogJ", "summary": "The paper proposes \"Espresso Sentence Embedding\" (ESE) for effective sentence embedding learning with two key processes \"learn-to-express\" and \"learn-to-compress.” In the \"learn-to-express\" phase, the loss is computed using both the final layer and intermediate layer embeddings. In \"learn-to-compress,\" ESE focuses on compressing embeddings by applying the softmax of the outer product of attention values. Experimental results demonstrate that ESE outperforms baselines like RAW (original model) and MRL (Matryoshka embedding; Kusupati et al., 2022) on STS and QA benchmarks. Furthermore, ablation studies confirm the robustness of ESE across various language models and embedding sizes.", "questions": "1. Why do MRL and RAW show consistent results across compression sizes in Figure 5? Does compression not cause any performance change for these models?\n2. In Section 3.3, the compression dimension $k$, to which the vector is reduced, appears to be fixed. However, compressed embedding sizes vary in Figures 3 and 4. How is this possible? Are these models using different compression dimensions at training and testing time, or are they trained with varying k values each time?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "**Motivation**\n\n- The proposed work addresses the scalability challenges in sentence embedding that arise from fixed embedding sizes and a fixed number of layers. The training objective (learn-to-compress and learn-to-express) is well-motivated and effectively demonstrated throughout the paper.\n\n**Methods and Experiments**\n\n- The approach of directly training shallow layer embeddings (Section 3.2) significantly enhances embedding quality. Experimental results in Figure 3 show that even embeddings from shallow layers can achieve performance comparable to deeper layers in other models, highlighting the effectiveness of this approach.\n- The novel compression method (Section 3.3) successfully captures interactions between vector dimensions, improving upon traditional PCA techniques, as shown in Table 2.\n\n**Practicality**\n\n- The framework presents a representation learning method that can be applied to low-resource scenarios.\n\n**Writing**\n\n- The paper's motivation is clearly written and easy to follow. The figures, particularly Figure 2, are well-designed and effectively illustrate the proposed methodology.", "weaknesses": "The experimental setup is generally well constructed, but I have little concerns about following issues:\n\n1. Kusupati et al. (2022) conduct extensive experiments across various classification and retrieval tasks using the ImageNet-1K dataset. In contrast, the paper under review **restricts its experimental scope solely to STS tasks,** which raises **concerns about the generalizability of the proposed method.** If the authors intended to focus exclusively on the NLP domain, I recommend including experiments in areas where sentence embeddings are widely applicable, such as information retrieval (e.g. BEIR benchmarks[1]). Although the paper includes results from HotpotQA (Table 4), it omits key metrics for evaluating the quality of embeddings from the retriever, such as supporting facts EM and F1. As a result, it is challenging to evaluate the sole impact on the retriever (i.e., embedding) within the overall pipeline.\n2. It is uncertain whether **comparing ESE’s intermediate layer performance with that of other models is entirely fair.** Obviously, these models are intended to use the final layer embedding for inference, so a comparison based on shallow-layer performance would be inappropriate. For practical significance, it would be better to **emphasize that “non-final-layer embeddings (e.g., <12th or <24th layer) of ESE outperform the final-layer embeddings (e.g., 12th or 24th layer) of MRL or RAW” ** This would highlight ESE’s potential time-saving benefits by reducing forward passes while guaranteeing the fair comparison. However, in the BGE-based results shown in Figure 3, ESE’s performance at layer 11 does not appear to surpass that of MRL or ESE’s own performance at layer 12, which raises concerns about claimed practical advantages of  ESE.\n\n[1] Thakur, Nandan, et al. \"Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models.\" arXiv preprint arXiv:2104.08663 (2021).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GYXYu9ebOE", "reviewer": "Reviewer_3a5X", "summary": "This paper is concerned about sentence embeddings. This paper claims that previous optimizations on the efficiency of sentence embeddings largely lie in the embedding size yet ignore the model depth. However, both the model depth and the embedding size could have a large impact on the efficiency. So in this paper, espresso sentence embeddings are proposed to consider both two facets in a joint optimization framework including both learning-to-express and learning-to-compress. The learning-to-express attempts to compress the model depth while the learning-to-compress attempts to compress the embedding size. Experimental results show that espresso sentence embedding enjoy better performance-efficiency tradeoffs than compared baselines.", "questions": "- Why, in Figure 4a, spearman correlation is degraded along the increment of embedding size?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed espresso sentence embeddings are indeed the first of its kind in this area.\n- The experimental results denote good performance-efficiency tradeoffs have been achieved.", "weaknesses": "- The idea, though not previous studied, is not that novel. And both two facets, i.e., the mode depth and embedding size, have been thoroughly studied in existing literature.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)