Length-Induced Embedding Collapse in PLM-based Models
Authors: Yuqi Zhou, Sunhao Dai, Zhanshuo Cao, Xiao Zhang, Jun Xu
Accepted at: acl25

Length-Induced Embedding Collapse in PLM-based Models

Yuqi Zhou, Sunhao Dai, Zhanshuo Cao, Xiao Zhang, Jun Xu*
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China
{yuqizhou, sunhaodai, caozhanshuo117, zhangx89, junxu}@ruc.edu.cn

Abstract

Text embeddings from PLM-based models enable a wide range of applications, yet their performance often degrades on longer texts. In this paper, we introduce a phenomenon we call Length Collapse, where embeddings of longer texts tend to cluster together. This clustering results in a distributional inconsistency between the embeddings of short and long texts. We further investigate how these differences contribute to the performance decline observed with longer texts across various downstream tasks. Through a rigorous theoretical analysis of the self-attention mechanism, which acts as a low-pass filter in PLM-based models, we demonstrate that as text length increases, the strength of low-pass filtering intensifies, causing embeddings to retain more low-frequency components. As a result, input token features become more similar, leading to clustering and ultimately the collapse of embeddings for longer texts. To address this issue, we propose a simple method, TempScale, which mitigates the Length Collapse phenomenon. By narrowing the gap in low-pass filtering rates between long and short texts, TempScale ensures more consistent embeddings across different text lengths. This approach leads to performance improvements of \(0.94\%\) on MTEB and \(1.10\%\) on LongEmbed, which focuses specifically on long-context retrieval, providing strong evidence for the validity of our analysis. The source code is available at https://github.com/Yuqi-Zhou/Length_Collapse.

1 Introduction

Text embeddings—dense vectors that capture the semantic information of texts—are essential for many NLP applications, including text analysis (Aggarwal and Zhai, 2012; Angelov, 2020), question answering (Tan et al., 2023; Xu et al., 2024), web search (Zhao et al., 2024; Yates et al., 2021), and retrieval-augmented generation (Gao et al., 2023; Fan et al., 2024). Typically, embeddings are generated by pre-trained language models (PLMs), which produce fixed-dimensional vectors regardless of text length. In practice, we expect PLMs to perform consistently across texts of varying lengths in downstream tasks.

Unfortunately, popular PLM-based embedding models perform poorly on longer texts. As shown in Figure 1a, we evaluate mainstream models on the IMDB classification task from the Massive Text Embedding Benchmark (MTEB)(Muennighoff et al., 2023), using test sets grouped by text length. The results show that models with varying capabilities degrade as text length increases. For example, the BGE (Xiao et al., 2023) model’s accuracy drops from 75.6% in the [0, 100] token range to 59.0% in the [400, 500] range, a 16.6% point decline.

We attribute this performance degradation to a systematic tendency in embedding models: as the text length \(n\) increases, the cosine similarity between embeddings of different texts also increases—a phenomenon we refer to as Length Collapse. To verify this, we conduct controlled experiments shown in Figures 1b and 1c. Figure 1b shows that embeddings of longer texts cluster more densely near the origin in the reduced embedding space, indicating a collapse that reduces variance. Figure 1c further shows that embeddings of longer texts have higher cosine similarity to each other, leading to smaller differences. This collapse causes distributional inconsistency between embeddings of different lengths. Furthermore, we analyze how this distributional inconsistency in embeddings of different text lengths leads to performance degradation in downstream tasks such as classification, retrieval, and STS, particularly affecting the performance of longer texts.

To study how text length affects embedding distributions, we analyze the self-attention mechanism in Fourier space (Wang et al., 2022b) (\(§\) 2.2). Build-
Figure 1: (a) Performance of embedding models on IMDb classification across length intervals [0, 100] to [400, 500]. The bluer ■ a cell, the higher the classification accuracy. (b) t-SNE visualization of embeddings from the BGE on NFCorpus dataset, with ● for the original dataset and ▲ and × for LLM-summarized versions, retaining semantic meaning with varying lengths. L indicates average text length, and D denotes mean distance to the origin. (c) Mean pairwise cosine similarity of embeddings from the BGE model across text length intervals on the corpus from NFCorpus, with an X-axis for length intervals and a Y-axis for average pairwise similarity.

ing on the findings that cascading self-attention blocks act as repeated low-pass filters, we prove that the attenuation rate of this filter is proportional to the largest singular value \( \sigma_a \) of the high-frequency components in the self-attention matrix. Assuming Gaussian-distributed input keys and query tokens, we show that \( \sigma_a \) decreases with increasing text length, causing longer texts to retain more of the Direct Component (DC) in token signals. This results in embeddings collapsing to a narrow space, as seen in Figure 1b.

To validate our theoretical analysis, we propose a method, Temperature Scaling (TempScale), to mitigate Length Collapse. TempScale manipulates the attention map by dividing the scores by a parameter called temperature (smaller than 1) before applying the softmax(\( \cdot \)) operator. This reduces the gap in \( \sigma_a \) between long and short texts in the self-attention matrix, minimizing the distributional gap between their embeddings. As shown in Figure 1c, a smaller temperature enables embeddings to exhibit lower pairwise cosine similarity, resulting in a more even distribution and alleviating Length Collapse. Additionally, TempScale improves performance by **0.94%** and **1.10%** in MTEB and LongEmbed (Zhu et al., 2024), respectively. All results confirm the validity of our analysis.

Our contributions are as follows: (1) We are the first to identify **Length Collapse**, where embeddings of longer texts cluster together, and explain how this uneven distribution degrades overall performance, especially for long texts. (2) We provide a rigorous theoretical analysis in the spectral domain, demonstrating that Length Collapse occurs due to the increasing low-pass filtering strength of self-attention as sequence length grows, causing token signals to retain only their DC component. (3) To validate this theoretical analysis, we propose a simple method, TempScale, to mitigate Length Collapse. Empirical results show a **0.94%** performance improvement across the MTEB and a **1.10%** improvement on LongEmbed, validating the correctness of our theoretical analysis.

2 **Length Collapse**

In this section, we define the problem and introduce our notations. We then present the Transformer structure in PLM-based models and briefly explain the Fourier transform from (Wang et al., 2022b). Using this framework, we show that the attention mechanism functions as a low-pass filter, with longer input sequences amplifying this effect, resulting in increasingly similar representations.

2.1 **Preliminaries and Background**

**Notations.** Let \( X \in \mathbb{R}^{n \times d} \) denote the input feature matrix, where \( n \) is the number of tokens and \( d \) is the embedding dimension. Let \( x_i \in \mathbb{R}^d \) represent the vector for the \( i \)-th token, and \( z_j \in \mathbb{R}^n \) represent the token sequence for the \( j \)-th dimension, where \( i \in \{1, \ldots, n\} \) and \( j \in \{1, \ldots, d\} \).

**Transformer Architecture.** Most modern embedding models (Chen et al., 2024a; Xiong et al., 2021) use a bidirectional transformer architecture with attention mechanisms. These models typically consist of three components: an embedding layer, a stack of transformer encoder blocks with Multi-Head Self-Attention (MSA) and Feed-Forward Net-
works (FFN), and a pooling layer to generate the final embedding. The Self-Attention (SA) module encodes each token by aggregating information from other tokens based on attention scores, as defined by the equation (Waswani et al., 2017):

\[
\text{SA}(\mathbf{X}) = \operatorname{softmax}\left( \frac{\mathbf{X} W_Q (\mathbf{X} W_K)^T}{\sqrt{d}} \right) \mathbf{X} W_V,
\]

where \( W_K, W_Q, \) and \( W_V \) are the key, query, and value weight matrices. The function \( \operatorname{softmax}(\cdot) \) normalizes the attention scores. Multi-Head Self-Attention (MSA) aggregates the outputs of multiple SA heads, projected back to the hidden dimension:

\[
\text{MSA}(\mathbf{X}) = [\text{SA}_1(\mathbf{X}) \quad \cdots \quad \text{SA}_H(\mathbf{X})] \mathbf{W}_O,
\]

where \( H \) is the number of heads, and \( \mathbf{W}_O \) projects the combined outputs to the hidden dimension.

**Fourier Analysis.** Inspired by the recent work (Wang et al., 2022b), which reveals that self-attention in Vision Transformers acts as a low-pass filter and causes feature degradation when scaling up depth, we employ Fourier analysis to further explore this phenomenon. We use the Fourier transform as the primary analytic tool, following (Wang et al., 2022b). Let \( \mathcal{F} : \mathbb{R}^n \to \mathbb{C}^n \) represent the Discrete Fourier Transform (DFT), with its inverse \( \mathcal{F}^{-1} : \mathbb{C}^n \to \mathbb{R}^n \). Applying \( \mathcal{F} \) to a token sequence \( z \) corresponds to multiplying by a DFT matrix, where the \( k \)-th row corresponds to the Fourier basis at frequency \( f_k = [e^{2\pi j(k-1)\cdot 0}, \ldots, e^{2\pi j(k-1)\cdot(n-1)}]^T / \sqrt{n} \in \mathbb{R}^n \), with \( j \) as the imaginary unit. Let \( \tilde{z} = \mathcal{F}z \) denote the spectrum of \( z \), with \( \tilde{z}_{dc} \in \mathbb{C} \) and \( \tilde{z}_{hc} \in \mathbb{C}^{n-1} \) as the DC and high-frequency components, respectively. We define the Direct-Current (DC) component as \( \mathcal{DC}[z] = \tilde{z}_{dc} f_1 \in \mathbb{C}^n \), and the high-frequency component as \( \mathcal{HC}[z] = [f_2, \ldots, f_n] \tilde{z}_{hc} \in \mathbb{C}^n \). In signal processing, a low-pass filter preserves low-frequency components while attenuating high frequencies. In this paper, we define a low-pass filter that retains only the DC component \( \mathcal{DC}[z] \), while diminishing the high-frequency components \( \mathcal{HC}[z] \). A precise definition is given in Definition 1.

**Definition 1.** *Let \( f : \mathbb{R}^n \to \mathbb{R}^n \) be an endomorphism with \( f^t \) denoting \( f \) applied \( t \) times. The function \( f \) acts as a low-pass filter if and only if \( \lim_{t \to \infty} \frac{\| \mathcal{HC}[f^t(z)] \|_2}{\| \mathcal{DC}[f^t(z)] \|_2} = 0 \) for all \( z \in \mathbb{R}^n \).*

For additional details, refer to Appendix A.

**2.2 Theoretical Analysis on Length Collapse**

**Overview.** This subsection demonstrates that increasing text length \( n \) accelerates low-pass filtering in the attention matrix, making text embeddings for longer texts more similar. We justify this by analyzing self-attention’s spectral-domain effect. Building on **Lemma 1**, we show that the largest singular value \( \sigma_a \) of \( \mathcal{HC}[\mathbf{A}] \) influences the filtering rate, with a smaller \( \sigma_a \) indicating greater high-frequency loss (**Theorem 2**). Our analysis further reveals that longer texts result in smaller \( \sigma_a \) values, causing longer text embeddings to lose feature expressiveness (**Theorem 3**). Consequently, we infer that longer texts yield more similar representations (**Corollary 4**), leading to **Length Collapse**.

Formally, the following lemma demonstrates that the attention matrix generated by a softmax function acts as a low-pass filter, independent of the specific token features or context window.

**Lemma 1.** (*Attention Matrix is A Low-pass Filter*) *Let \( \mathbf{A} = \operatorname{softmax}(\mathbf{P}) \), where \( \mathbf{P} \in \mathbb{R}^{n \times n} \). Then \( \mathbf{A} \) must be a low-pass filter. For all \( z \in \mathbb{R}^n \),*

\[
\lim_{t \to \infty} \frac{\| \mathcal{HC}[\mathbf{A}^t z] \|_2}{\| \mathcal{DC}[\mathbf{A}^t z] \|_2} = 0.
\]

Lemma 1 follows from the Perron-Frobenius theorem (Meyer, 2000). Since all elements of the self-attention matrix are positive and each row sums to 1, the largest eigenvalue is 1. Repeated application of the self-attention matrix represents the forward process of the embedding model, and as the number of layers increases, the output retains only the DC component. A more detailed proof can be found in Theorem 1 of Wang et al. (2022b).

Understanding that self-attention matrices act as low-pass filters, we are interested in the extent to which an SA layer suppresses high-frequency components. Additionally, we provide a filter rate to illustrate the speed at which these high-frequency components are eliminated.

**Theorem 2.** (*Filter Rate of SA*) *Let \( \sigma_a \) be the largest singular value of \( \mathcal{HC}[\mathbf{A}] \), and \( \text{SA}(\mathbf{X}) = \mathbf{A} \mathbf{X} \mathbf{W}_V \) the self-attention output. We have:*

\[
\| \mathcal{HC}[\text{SA}(\mathbf{X})] \|_F \leq \sigma_a \| \mathbf{W}_V \|_2 \| \mathcal{HC}[\mathbf{X}] \|_F. \tag{1}
\]

Theorem 2 suggests the high-frequency intensity ratio to the pre- and post- attention aggregation is upper bounded by \( \sigma_a \| \mathbf{W}_V \|_2 \). When \( \sigma_a \| \mathbf{W}_V \|_2 < 1 \), \( \mathcal{HC}[\mathbf{X}] \) converges to zero exponentially. We further present Figure 6 in Appendix C.2 to justify our results, showing that the
upper bound is consistent with the trend observed in the actual values. The proof of Theorem 2 can be found in Appendix B.1.

Based on the fact that a lower \( \sigma_s \) leads to a higher filter-pass rate, we prove in the following theorem that \( \sigma_s \) decreases as the input length \( n \) increases.

**Theorem 3. (Filter Rate of Different Input Length n)** Let \( XW_Q \) and \( XW_K \) be a Gaussian matrix, where elements \( q_{ij} \sim \mathcal{N}(0, \sigma_q^2) \) and \( k_{ij} \sim \mathcal{N}(0, \sigma_k^2), \forall i, j \). Let \( p_{ij} = q_i^\top k_j / \sqrt{d} \) the attention score of pair \( i, j \), whose variance can be expressed as \( \sigma_a^2 = \sigma_q^2 \sigma_k^2 + C_{cross} \), where \( C_{cross} \) is the cross-covariance of the squared queries and keys (Goodman, 1960). Then we have

\[
\sigma_a \leq \frac{n}{\sqrt{2 \sqrt{1 + \frac{1}{e^{2\pi^2}}(n-1)^2} + 1}},
\]

where \( \sigma_a \) decreases with \( n \) increasing.

The proof of Theorem 3 is in Appendix B.2. Building on the work of Fenton (1960); Nahshan et al. (2024) on log-normal variables, Theorem 3 shows that as input length \( n \) increases, \( \sigma_a \) decreases, suppressing high-frequency information and reducing feature expressiveness due to the self-attention matrix’s low-pass filtering effect. To validate this, we sample texts of varying lengths and plot the \( \sigma_a \) values from the final layer’s attention matrix, as shown in Figure 7 in Appendix C.3. The results confirm that \( \sigma_a \) decreases with text length, leading to a higher filtering rate. To facilitate further analysis, we define the temperature of the SA defined in Nahshan et al. (2024) as:

\[
\tau_s = \frac{1}{\sigma_s} = \frac{1}{\sqrt{\sigma_q^2 \sigma_k^2 + C_{cross}}}.
\]

Then denote \( \tilde{p}_{ij} = p_{ij}/\sigma_s \) and each element in attention matrix \( A \) can be rewritten as follows:

\[
A_{ij} = \frac{e^{\tilde{p}_{ij}/\tau_s}}{\sum_{k=1}^n e^{\tilde{p}_{ik}/\tau_s}},
\]

where \( \tilde{p}_{ij} \sim \mathcal{N}(0, 1) \) and \( \sigma_a \) increases with \( \tau_s \) decreases. This implies that with a lower temperature \( \tau_s \), the self-attention (SA) mechanism preserves more high-frequency components in the token signals, thereby preventing collapse in long texts.

**Corollary 4. (Length Collapse in Text Embeddings)** As the text length \( n \) increases, the cosine similarity of their embeddings tends to increase.

![Mean cosine similarity of text embeddings across length intervals, with embeddings averaged from the model’s word embedding matrix.](page_1012_180_377_246.png)

Figure 2: Mean cosine similarity of text embeddings across length intervals, with embeddings averaged from the model’s word embedding matrix.

The proof of Corollary 4 in Appendix B.3 is based on the assumption that the mean of word embeddings in natural language texts remains consistent, with the formal definition and proof detailed therein. As shown in Figure 2, we compute text embeddings by averaging word embeddings from the BGE model’s embedding matrix and then assess similarity across length intervals. The results show that as text length increases, embedding similarity rises, supporting the assumption. However, we must emphasize that the increased low-pass filtering with length is the primary cause of Length Collapse. In Appendix C.4, we demonstrate that repeating two different tokens and computing the similarity between the sequences shows that the cosine similarity of their embeddings increases with length, even without any overlapping tokens.

2.3 Discussion

**Other Components in Transformer.** After discussing how self-attention contributes to Length Collapse, we further examine the impact of other Transformer components, including multi-head attention, residual connections, and feed-forward networks (FFN). Fortunately, prior work (Wang et al., 2022b) has analyzed whether these modules can alleviate the low-pass filtering effect. The findings show that while they help preserve high-frequency signals, they do not change the fact that the MSA block, as a whole, behaves like a low-pass filter. Moreover, their ability to retain high-frequency information depends entirely on internal parameters and architecture, rather than input length. Therefore, these components do not affect our analysis of
Length Collapse in practical Transformer models.

Difference from Over-Smoothing in Deeper Layers. In previous research (Wang et al., 2022b), it has been noted that a self-attention module acts as a low-pass filter, causing input feature maps to gradually lose high-frequency signals as the model layers go deeper. Furthermore, other studies (Oono and Suzuki, 2020; Cai and Wang, 2020; Peng et al., 2024b) indicate that the node features of Graph Convolutional Networks (GCNs) can become exponentially trapped in the null space of the graph Laplacian matrix. The root cause of this phenomenon is that both graph Laplacian matrices and self-attention matrices consistently exhibit a dominant eigenvector, commonly referred to as the DC component. While these studies address over-smoothing in deeper layers, we focus on how the low-pass filtering process changes as the input sequence lengthens, specifically examining over-smoothing in longer sequences.

3 How does Length Collapse Lead to Performance Degradation

Building on our analysis of how the self-attention mechanism contributes to Length Collapse, this section investigates its impact on model performance across the MTEB benchmark. The tasks in MTEB can be broadly categorized into classification/clustering and matching tasks. For classification and clustering, a classifier is trained on text embeddings. In matching tasks, performance is evaluated by computing the similarity between embeddings using cosine similarity or the dot product. These matching tasks can be further divided into retrieval tasks, which typically involve texts of significantly different lengths, and STS/summarization tasks, which involve texts of similar lengths. We examine how Length Collapse affects model performance across these different task types.

Impact on Classification and Clustering Tasks. As shown in Figure 3 (Left), Length Collapse causes long-text embeddings to cluster at the center, while short-text embeddings spread around the periphery. In KNN classification, this brings clustering centers closer to long texts, resulting in a biased influence and reducing performance.

Impact on Retrieval Tasks. As shown in Figure 3 (Middle), long documents are clustered in a smaller central space, while shorter queries are more dispersed, with richer contextual representations (Ethayarajh, 2019). While centrally-positioned long documents have higher similarity with all embeddings, their representational space is more limited. This can lead to shorter noise documents (Green Square) appearing more relevant to the query (Purple Circle) due to their broader representational space. Analysis of the NFCorpus dataset, comparing rankings of the top 10% longest and shortest relevant documents (Figure 4), shows that short documents follow an inverted U-shaped distribution, whereas long documents exhibit a more uniform distribution. This is because shorter documents have a larger representational space, making it easier for them to appear at the beginning or end of the ranking list.

Impact on STS Tasks. As shown in Figure 3 (Right), Length Collapse causes long-text embeddings to cluster in a narrower space, where unrelated embeddings may show higher similarity than relevant pairs. Noise sentences (Green Square) in the long-text space may appear more similar than related sentences (Purple Square). In contrast, short-text embeddings are more spread out, resulting in better performance due to greater separation between noise and relevant sentences.

4 Mitigating Length Collapse via Temperature Scaling

As discussed in § 2.2, the self-attention matrix applies stronger low-pass filtering to longer texts, resulting in distributional differences between long and short text embeddings, which subsequently leads to a decline in long-text performance across various tasks in the MTEB benchmark. To address this issue and validate our analysis, a straightforward solution is to reduce the filtering rate differ-

![A 3D toy example explains why Length Collapse leads to performance degradation. Three diagrams labeled (a) Classification and Clustering, (b) Retrieval, and (c) STS, showing clusters and distances for different text types.](page_186_120_1077_355.png)

Figure 3: A 3D toy example explains why Length Collapse leads to performance degradation.
Figure 4: The count of relevant documents at different ranking positions, where a smaller ranking position indicates a document is more relevant to the query.

Figure 5: Two extreme cases of TempScale. The darker the color, the higher the attention score.

ence between long and short texts, thereby mitigating the disparity in their embeddings. To achieve this, we propose a scaling technique called Temperature Scaling (TempScale), which directly adjusts the attention map by multiplying \( \tau_s \) by a constant temperature \( \tau \) less than 1, thereby reducing the difference in filtering rates \( \sigma_a \) between long and short texts.

Specifically, a larger \( \sigma_s \) will result in a smaller factor \( 2\sqrt{1 + \frac{1}{e^{2\tau_s^2}}} \) in Eqn. 2, which means that as the text length \( n \) increases, \( \sigma_a \) will decrease more slowly, thereby reducing the difference in filtering rates between long and short texts. Therefore, by increasing \( \sigma_s \), or equivalently decreasing \( \tau_s \) based on Eqn. 3, we can mitigate the distributional difference between long and short text embeddings.

Formally, let \( A = \text{softmax} \left( \frac{XW_Q(XW_K)^T}{\sqrt{d}} \right) \) denote a self-attention matrix. To decrease the \( \tau_s \) of \( A \) based on Eqn. 4, we apply a temperature coefficient \( \tau \) to the logits before performing the softmax operation. Specifically, for each row \( p_i \) in the attention score matrix \( \frac{XW_Q(XW_K)^T}{\sqrt{d}} \), we compute the scaled logits by dividing by a temperature \( \tau \in (0, 1] \), and then apply the softmax function to obtain the attention weights:

\[
A = \text{softmax} \left( \frac{XW_Q(XW_K)^T}{\tau \sqrt{d}} \right),
\]

where a lower \( \tau \) results a smaller \( \tau_s \).

Another Intuitive Explanation. We illustrate the effects of temperature scaling using two extreme cases to highlight how TempScale works from a different perspective. As shown in Figure 5, when scaling the matrix \( A \) with a relatively large \( \tau \), the elements of \( A \) become nearly equal, causing the matrix to filter out all high-frequency information and resulting in identical token embeddings. In contrast, when scaling with a smaller temperature, \( A \) no longer acts as a weighted sum of token representations, but selects a specific token’s representation, preserving more high-frequency information. Viewing the attention matrix as an adjacency matrix, a higher temperature leads to a denser graph, enhancing information exchange between nodes but losing high-frequency details (Oono and Suzuki, 2020; Cai and Wang, 2020). A lower temperature, on the other hand, creates a sparser graph, preserving more high-frequency information and preventing over-smoothing between nodes.

5 Experiments

In this section, we first conduct experiments to validate the effectiveness of our TempScale on MTEB and LongEmbed. Then we analyze how different tasks can benefit from TempScale to validate our theoretical analysis.

5.1 TempScale Benefits Embedding Models

Experiment Settings. We evaluate embedding models on long-context retrieval using 4 real-world tasks from LongEmbed (Zhu et al., 2024), focusing on factuality in both short-query and long-document settings. For other tasks, we select 39 datasets from MTEB (Muenninghoff et al., 2023), covering classification, clustering, summarization, STS, retrieval, and reranking. Specifically, for classification, we only downloaded 10 datasets due to open-source restrictions. For retrieval, we select the 5 datasets with the longest documents to better assess Length Collapse. To comprehensively evaluate TempScale, we select several representative PLM-based embedding models, includ-
<table>
  <tr>
    <th rowspan="2">Num. Datasets (→)</th>
    <th colspan="3">Class.</th>
    <th colspan="3">Clust.</th>
    <th colspan="3">Summ.</th>
    <th colspan="3">STS</th>
    <th colspan="3">BeirRetr.</th>
    <th colspan="3">Rerank.</th>
    <th colspan="3">LongEmbRetr.</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>8</th><th>11</th><th>1</th>
    <th>8</th><th>11</th><th>1</th>
    <th>8</th><th>11</th><th>1</th>
    <th>8</th><th>11</th><th>1</th>
    <th>8</th><th>11</th><th>1</th>
    <th>8</th><th>11</th><th>1</th>
    <th>8</th><th>11</th><th>1</th>
  </tr>
  <tr>
    <td colspan="24"><i>window=512</i></td>
  </tr>
  <tr>
    <td>ANCE</td><td>55.27</td><td>33.04</td><td>29.58</td><td>66.32</td><td>26.95</td><td>49.09</td><td>34.02</td><td>42.04</td>
  </tr>
  <tr>
    <td>+TempScale</td><td>55.44</td><td>33.28</td><td>29.59</td><td>66.47</td><td>27.39</td><td>49.25</td><td>34.07</td><td>42.21</td>
  </tr>
  <tr>
    <td>Relative Improv. (%)</td><td>0.29 ▲</td><td>0.73 ▲</td><td>0.06 ▲</td><td>0.22 ▲</td><td>1.63 ▲</td><td>0.32 ▲</td><td>0.17 ▲</td><td>0.49 ▲</td>
  </tr>
  <tr>
    <td>GTR</td><td>55.10</td><td>38.65</td><td>29.67</td><td>70.11</td><td>33.00</td><td>54.23</td><td>37.33</td><td>45.44</td>
  </tr>
  <tr>
    <td>+TempScale</td><td>55.59</td><td>39.52</td><td>29.83</td><td>70.26</td><td>33.56</td><td>54.22</td><td>37.33</td><td>45.76</td>
  </tr>
  <tr>
    <td>Relative Improv. (%)</td><td>0.88 ▲</td><td>2.26 ▲</td><td>0.54 ▲</td><td>0.21 ▲</td><td>1.69 ▲</td><td>-0.01 ▼</td><td>0.01 ▲</td><td>0.80 ▲</td>
  </tr>
  <tr>
    <td>GIST</td><td>64.75</td><td>44.77</td><td>31.14</td><td>75.61</td><td>37.55</td><td>58.55</td><td>38.21</td><td>50.08</td>
  </tr>
  <tr>
    <td>+TempScale</td><td>65.12</td><td>44.66</td><td>32.17</td><td>75.61</td><td>37.82</td><td>58.60</td><td>38.35</td><td>50.33</td>
  </tr>
  <tr>
    <td>Relative Improv. (%)</td><td>0.56 ▲</td><td>-0.25 ▼</td><td>3.31 ▲</td><td>-0.01 ▼</td><td>0.71 ▲</td><td>0.08 ▲</td><td>0.36 ▲</td><td>0.68 ▲</td>
  </tr>
  <tr>
    <td>BGE</td><td>64.79</td><td>45.80</td><td>31.03</td><td>75.88</td><td>38.14</td><td>58.87</td><td>37.46</td><td>50.28</td>
  </tr>
  <tr>
    <td>+TempScale</td><td>64.91</td><td>45.79</td><td>31.87</td><td>75.68</td><td>38.46</td><td>58.97</td><td>38.73</td><td>50.63</td>
  </tr>
  <tr>
    <td>Relative Improv. (%)</td><td>0.19 ▲</td><td>-0.01 ▼</td><td>2.71 ▲</td><td>-0.26 ▼</td><td>0.84 ▲</td><td>0.17 ▲</td><td>3.41 ▲</td><td>1.01 ▲</td>
  </tr>
  <tr>
    <td colspan="24"><i>window=4k</i></td>
  </tr>
  <tr>
    <td>E5</td><td>61.72</td><td>38.82</td><td>30.58</td><td>71.77</td><td>29.77</td><td>53.12</td><td>56.01</td><td>48.83</td>
  </tr>
  <tr>
    <td>+TempScale</td><td>62.15</td><td>40.62</td><td>31.26</td><td>72.17</td><td>30.28</td><td>53.47</td><td>56.88</td><td>49.55</td>
  </tr>
  <tr>
    <td>Relative Improv. (%)</td><td>0.7 ▲</td><td>4.62 ▲</td><td>2.25 ▲</td><td>0.55 ▲</td><td>1.69 ▲</td><td>0.65 ▲</td><td>1.56 ▲</td><td>1.72 ▲</td>
  </tr>
  <tr>
    <td>Avg Improv. (%)</td><td>0.53 ▲</td><td>1.47 ▲</td><td>1.77 ▲</td><td>0.14 ▲</td><td>1.31 ▲</td><td>0.24 ▲</td><td>1.10 ▲</td><td>0.94 ▲</td>
  </tr>
</table>

Table 1: Average of the main metric (see Appendix F) per task on MTEB English subsets and LongEmb. Relative Improv. means percentage increase over the performance without TempScale and improvements are highlighted with ▲ while decreasing values are denoted by ▼.

ing (1) ANCE (Xiong et al., 2021); (2) GTR (Ni et al., 2022); (3) GIST (Solatorio, 2024); (4) BGE (Xiao et al., 2023); (5) E5 (Zhu et al., 2024). These models are fine-tuned from various pretrain language models, including BERT (Kenton and Toutanova, 2019), RoBERTa (Liu et al., 2019), and T5 (Chung et al., 2024). More descriptions of the datasets and models can be found in Appendix G. When evaluating the embedding models, we set the same \( \tau \) on the softmax function for the attention modules across all layers within the range of \( \{0.1, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0\} \). The metrics used for different tasks are consistent with MTEB and can be found in Appendix F.

Results. We select the optimal temperature \( \tau \) for each model based on their performance across a single task and present the results in Table 1. The findings demonstrate that our proposed method, TempScale, improves performance across a range of general tasks, with an average increase of 0.94%. For long-context retrieval tasks, the average improvement is 1.10%. Text length varies widely across tasks—for example, STS uses very short texts, whereas LongEmbed includes much longer ones (1000+ tokens). TempScale proves effective for both preventing long-text collapse and enhancing short-text embeddings, which improves downstream task performance. Additionally, larger context windows yield greater performance gains, with E5 showing the highest improvement of 1.72%.

This may be due to larger windows offering more data for adjustment. Beyond the excellent performance of TempScale, more importantly, it validates the correctness of our theoretical analysis.

5.2 Further Analysis

How do the Classification and Clustering Tasks Benefit from TempScale? In § 3, we attribute the performance decline in classification and clustering tasks to the distributional differences between long and short texts, which lead the model to assign greater weight to long text embeddings during classifier training. TempScale addresses this by adjusting the longer texts to align with the same space as short texts, as described in Figure 1c, ensuring that both contribute equally during training. More experiments can be found in Appendix C.6.

How do Retrieval Tasks Benefit from TempScale? In § 3, we attribute the performance decline in retrieval tasks to the fact that short texts have more contextual embeddings. Therefore, TempScale improves the performance of long texts by adjusting the temperature to increase high-frequency information, enabling a more contextual distribution. To validate this, we select NFCorpus and SciFact to investigate the impact. As shown in Table 2, we record the average ranking position of the longest positive documents after applying TempScale at different temperatures, where a lower value indicates the documents are ranked higher.
The experimental results show that a lower temperature causes the model to rank relevant long documents higher. Our method enables long texts to become more contextual, thereby reducing the bias introduced by length. In summary, TempScale enhances the performance of long documents by introducing TempScale. Similar phenomena on more models and datasets are provided in Appendix C.7.

How do the STS Tasks Benefit from TempScale? TempScale enhances performance by giving long sentence embeddings the same spatial representation as short texts. To verify this, we record the cosine similarity between random sentences, related sentences, and unrelated sentences at different temperatures, as shown in Table 3. The results show that as the \( \tau \) in TempScale decreases, the similarity between related sentences remains relatively high, while the similarity between unrelated sentences is further reduced. Specifically, although the similarity between random sentences increases with \( \tau \) in STS13 and STS14, the increase in the unrelated part is not as large as that of the random sentences. This indicates that the distance between unrelated sentences is increasing.

Can the Temperature be Set Based on the Length of the Texts? As shown in Table 2, the ranking of relevant long documents in retrieval tasks improves as the temperature decreases. This indicates that using a smaller temperature \( \tau \) for long texts can better align the embeddings of long and short texts within the same distribution. For longer texts, better performance can be achieved by setting a smaller \( \tau \), and the discussions can be found in Appendix C.7.

6 Related Work

Text Embedding Models. Embeddings are generated by pre-trained language models (PLMs), which produce fixed-dimensional embeddings regardless of the input text length, laying the foundation of numerous NLP applications. Early works on text embeddings lack context awareness and are thus commonly labeled as word embedding models (Pennington et al., 2014). Modern embedding models (Wang et al., 2022a; Xiao et al., 2023) incorporate context awareness into language models through self-attention mechanisms, serving as the foundation for the latest embedding models. These models, based on Transformer architectures like BERT (Kenton and Toutanova, 2019) and RoBERTa (Liu et al., 2019), are first pre-trained on large-scale weakly supervised text pairs using contrastive loss (Gao et al., 2021), then fine-tuned on small scale but high-quality datasets. More recently, (Muenninghoff et al.) investigates the integration of generative and embedding tasks in large language models, introducing transformer-based GritLM, which enhances performance in both areas. Among the diverse range of pre-trained transformers (Wolf et al., 2020), self-attention plays a crucial role, and this paper focuses on how this module contributes to length collapse.

Context Window Extension for Embedding Models. Despite transformer-based embedding models excelling in generating vector representations, they are typically constrained by a narrow context window of around 512 input tokens (Wang et al., 2022a; Xiao et al., 2023; Ni et al., 2022). This limitation significantly restricts their use in scenarios that require processing long inputs, such as extensive Wikipedia entries or meeting transcripts (Saad-Falcon et al., 2024; Zhu et al., 2024). Previous work attributes this poor performance to limited context windows and attempts to extend the window size. Current initiatives to develop long-context embedding models generally begin with acquiring a long-context backbone model, either by pre-training from scratch with long inputs (Günther et al., 2023; Nussbaum et al., 2024; Chen et al., 2024a) or utilizing existing models (Wang et al., 2024; Zhu et al., 2024). This is followed by training the backbone model to generate embeddings. However, in this paper, we discover that regardless of the model’s context window size, the model consistently performs worse on longer texts than on shorter texts due to length collapse. Therefore, we aim to improve the performance of long texts across all context window size models by analyzing and addressing length collapse, rather than simply expanding context window size.

7 Discussion

LLM-based Embedding Models: While our main focus is on PLM-based models, we also observe similar signs of Length Collapse in LLM-based models, which represent another major class of current mainstream models. We provide a detailed discussion of these observations in Appendix D.

Contrastive Learning: Length Collapse increases similarity between long and short text embeddings. Although contrastive learning mitigates high similarity, performance degradation in long texts is
<table>
  <tr>
    <th rowspan="2">Temperature \( \tau \)</th>
    <th colspan="3">1.0</th>
    <th colspan="3">0.9</th>
    <th colspan="3">0.8</th>
  </tr>
  <tr>
    <th>ANCE</th>
    <th>GTR</th>
    <th>GIST</th>
    <th>BGE</th>
    <th>E5</th>
    <th>ANCE</th>
    <th>GTR</th>
    <th>GIST</th>
    <th>BGE</th>
    <th>E5</th>
  </tr>
  <tr>
    <td>1.0</td>
    <td>1,306.2</td>
    <td>1,132.8</td>
    <td>1,111.0</td>
    <td>998.3</td>
    <td>1,193.3</td>
    <td>1,291.6</td>
    <td>1,120.5</td>
    <td>1,112.8</td>
    <td>978.9</td>
    <td>1,172.3</td>
    <td>1,278.1</td>
    <td>1,113.1</td>
    <td>1,113.8</td>
    <td>965.1</td>
    <td>1,162.8</td>
  </tr>
  <tr>
    <td>0.9</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>0.8</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

(a) NFCorpus

<table>
  <tr>
    <th rowspan="2">Temperature \( \tau \)</th>
    <th colspan="3">1.0</th>
    <th colspan="3">0.9</th>
    <th colspan="3">0.8</th>
  </tr>
  <tr>
    <th>ANCE</th>
    <th>GTR</th>
    <th>GIST</th>
    <th>BGE</th>
    <th>E5</th>
    <th>ANCE</th>
    <th>GTR</th>
    <th>GIST</th>
    <th>BGE</th>
    <th>E5</th>
  </tr>
  <tr>
    <td>1.0</td>
    <td>80.7</td>
    <td>81.7</td>
    <td>14.4</td>
    <td>13.3</td>
    <td>66.8</td>
    <td>78.7</td>
    <td>76.6</td>
    <td>12.4</td>
    <td>12.3</td>
    <td>47.5</td>
    <td>81.5</td>
    <td>72.4</td>
    <td>11.3</td>
    <td>12.4</td>
    <td>38.9</td>
  </tr>
  <tr>
    <td>0.9</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>0.8</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

(b) SciFact

Table 2: Average ranking position with 20% longest document across different temperature \( \tau \).

<table>
  <tr>
    <th rowspan="2">Temperature</th>
    <th colspan="3">STS12</th>
    <th colspan="3">STS13</th>
    <th colspan="3">STS14</th>
  </tr>
  <tr>
    <th>Random</th>
    <th>Related</th>
    <th>Unrelated</th>
    <th>Random</th>
    <th>Related</th>
    <th>Unrelated</th>
    <th>Random</th>
    <th>Related</th>
    <th>Unrelated</th>
  </tr>
  <tr>
    <td>1.0</td>
    <td>0.9097</td>
    <td>0.9611</td>
    <td>0.7973</td>
    <td>0.8707</td>
    <td>0.9470</td>
    <td>0.7931</td>
    <td>0.8877</td>
    <td>0.9494</td>
    <td>0.7889</td>
  </tr>
  <tr>
    <td>0.9</td>
    <td>0.9096</td>
    <td>0.9612</td>
    <td>0.7937</td>
    <td>0.8721</td>
    <td>0.9481</td>
    <td>0.7942</td>
    <td>0.8886</td>
    <td>0.9499</td>
    <td>0.7893</td>
  </tr>
  <tr>
    <td>0.8</td>
    <td>0.9097</td>
    <td>0.9612</td>
    <td>0.7926</td>
    <td>0.8741</td>
    <td>0.9490</td>
    <td>0.7969</td>
    <td>0.8897</td>
    <td>0.9501</td>
    <td>0.7912</td>
  </tr>
</table>

Table 3: Average cosine similarities across different temperature settings.

also due to distribution differences between long and short texts, not just similarity. Further analysis of why contrastive learning fails to fully mitigate Length Collapse is provided in Appendix E.1. Comparison with Other Methods: We compare our method with post-processing approaches (Flow Function (Li et al., 2020), Whitening (Su et al., 2021)) and long-text methods (Chiang and Cholak (2022), YaRN (Peng et al., 2024a)) in Appendix E.2 and E.3. Despite similar forms, these methods fail to address Length Collapse effectively.

8 Conclusion

In this paper, we identify the phenomenon of Length Collapse, where the embeddings of longer texts tend to cluster together, and provide a Fourier domain analysis to explain this behavior. Our theoretical findings suggest that Self-Attention inherently performs stronger low-pass filtering as the text length increases, leading to patch uniformity issues in longer sentences. Furthermore, the distributional differences between short and long text embeddings contribute to the performance decline of longer texts. To address this, we propose TempScale, which effectively balances the filtering rates for short and long texts. Our extensive experiments validate the accuracy of our analysis and the effectiveness of our method, significantly improving the performance of general embedding models on the MTEB and LongEmbed benchmarks.

Limitations

Limitations include: 1) LLM-based embedding model: Although we have observed a Length Collapse in LLM-based embedding models, further analysis is needed to investigate how unidirectional attention mechanisms specifically contribute to this phenomenon. Additionally, LLMs also exhibit a performance drop on long texts during text generation, which can be seen as Length Collapse. However, in this work, we only focus on PLM-based models across various tasks in MTEB. Future work could attempt to explain the reasons behind the performance decline of LLMs on long text generation from the perspective of low-pass filtering; 2) Tuning method: The work in this paper relies on existing models and pre-trained parameters without using a training dataset. In future work, we will focus on tuning the temperature for additional improvements; 3) Analysis on more modules: We primarily investigate the impact of the self-attention module on Length Collapse in this paper. Moving forward, we plan to explore the effects of additional modules in transformers such as LayerNorm and FFN.

Acknowledgements

This work was funded by the National Key R&D Program of China (2023YFA1008704), the National Natural Science Foundation of China (62472426,62376275). Supported by fund for building world-class universities (disciplines) of Renmin University of China. Work partially done at Beijing Key Laboratory of Research on Large Models and Intelligent Governance, and Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE.
References

Charu C Aggarwal and ChengXiang Zhai. 2012. A survey of text clustering algorithms. Mining text data, pages 77–128.

Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012. Semeval-2012 task 6: A pilot on semantic textual similarity. In* SEM 2012: The First Joint Conference on Lexical and Computational Semantics–Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012).

Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo. 2013. * sem 2013 shared task: Semantic textual similarity. In SEM, pages 32–43.

Dimo Angelov. 2020. Top2vec: Distributed representations of topics. arXiv preprint arXiv:2008.09470.

Anil Bandhakavi, Nirmalie Wiratunga, P Deepak, and Stewart Massie. 2014. Generating a word-emotion lexicon from# emotional tweets. In SEM), pages 12–21.

Ergun Biçiçi. 2015. Rtm-deu: Predicting semantic similarity with referential translation machines. In Proceedings of the 9th international workshop on semantic evaluation (SemEval 2015), pages 56–63.

Alexander Bondarenko, Maik Fröbe, Meriem Beblouci, Lukas Gienapp, Yamen Ajjour, Alexander Panchenko, Chris Biemann, Benno Stein, Henning Wachsmuth, Martin Potthast, et al. 2020. Overview of touché 2020: argument retrieval. In Experimental IR Meets Multilinguality, Multimodality, and Interaction: 11th International Conference of the CLEF Association, CLEF 2020, Thessaloniki, Greece, September 22–25, 2020, Proceedings 11, pages 384–395. Springer.

Vera Boteva, Demian Gholipour, Artem Sokolov, and Stefan Riezler. 2016. A full-text learning to rank dataset for medical information retrieval. In ECIR, pages 716–722.

Chen Cai and Yusu Wang. 2020. A note on over-smoothing for graph neural networks. arXiv preprint arXiv:2006.13318.

Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020. Efficient intent detection with dual sentence encoders. arXiv preprint arXiv:2003.04807.

Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024a. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. arXiv preprint arXiv:2402.03216.

Mingda Chen, Zewei Chu, Sam Wiseman, and Kevin Gimpel. 2021. Summscreen: A dataset for abstractive screenplay summarization. arXiv preprint arXiv:2104.07091.

Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. 2024b. Longlora: Efficient fine-tuning of long-context large language models. In ICLR.

David Chiang and Peter Cholak. 2022. Overcoming a theoretical limitation of self-attention. In ACL, pages 7654–7664.

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2024. Scaling instruction-finetuned language models. JMLR, 25(70):1–53.

Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S Weld. 2020a. Specter: Document-level representation learning using citation-informed transformers. arXiv preprint arXiv:2004.07180.

Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S. Weld. 2020b. Specter: Document-level representation learning using citation-informed transformers. In ACL.

Slawomir Dadas, Michał Perelkiewicz, and Rafał Poświata. 2020. Evaluation of sentence representations in Polish. In LREC, pages 1674–1680.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Kawin Ethayarajh. 2019. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings. In EMNLP. Association for Computational Linguistics.

Alexander R Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. Summeval: Re-evaluating summarization evaluation. TACL, pages 391–409.

Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. 2024. A survey on rag meeting llms: Towards retrieval-augmented large language models. In SIGKDD, pages 6491–6501.

Lawrence Fenton. 1960. The sum of log-normal probability distributions in scatter transmission systems. IRE Transactions on communications systems, 8(1):57–67.

Jack FitzGerald, Christopher Hench, Charith Peris, Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron Nash, Liam Urbach, Vishesh Kakarala, Richa Singh, et al. 2022. Massive: A 1m-example multilingual natural language understanding dataset with 51 typologically-diverse languages. arXiv preprint arXiv:2204.08582.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simcse: Simple contrastive learning of sentence embeddings. In EMNLP, pages 6894–6910.

Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997.

Gregor Geigle, Nils Reimers, Andreas Rücklé, and Iryna Gurevych. 2021. Tweac: transformer with extendable qa agent classifiers. arXiv preprint arXiv:2104.07081.

Leo A Goodman. 1960. On the exact variance of products. Journal of the American statistical association, 55(292):708–713.

Michael Günther, Jackmin Ong, Isabelle Mohr, Alaed-dine Abdessalem, Tanguy Abel, Mohammad Kalim Akram, Susana Guzman, Georgios Mastrapas, Saba Sturua, Bo Wang, et al. 2023. Jina embeddings 2: 8192-token general-purpose text embeddings for long documents. arXiv preprint arXiv:2310.19923.

Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6609–6625.

Ziyan Jiang, Xueguang Ma, and Wenhui Chen. 2024. Longrag: Enhancing retrieval-augmented generation with long-context llms. arXiv preprint arXiv:2406.15319.

Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171–4186.

Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018. The narrativeqa reading comprehension challenge. TACL, pages 317–328.

Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2024. Nv-embed: Improved techniques for training llms as generalist embedding models. arXiv preprint arXiv:2405.17428.

Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. 2020. On the sentence embeddings from pre-trained language models. In EMNLP, pages 9119–9130.

Xueqing Liu, Chi Wang, Yue Leng, and ChengXiang Zhai. 2018. Linkso: a dataset for learning to retrieve similar question answer pairs on software development forums. In Proceedings of the 4th ACM SIGSOFT International Workshop on NLP for Software Engineering, pages 2–5.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv.

Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In ACL, pages 142–150.

Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. 2018. Www’18 open challenge: financial opinion mining and question answering. In Companion proceedings of the the web conference 2018, pages 1941–1942.

Carl D Meyer. 2000. Matrix analysis and applied linear algebra, volume 71. SIAM.

Niklas Muennighoff, SU Hongjin, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. Generative representational instruction tuning. In ICLR 2024 Workshop: How Far Are We From AGI.

Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. Mtcb: Massive text embedding benchmark. In EACL, pages 2014–2037.

Yury Nahshan, Joseph Kampeas, and Emir Haleva. 2024. Linear log-normal attention with unbiased concentration. In ICLR.

Preslav Nakov, Alan Ritter, Sara Rosenthal, Fabrizio Sebastiani, and Veselin Stoyanov. 2016. SemEval-2016 task 4: Sentiment analysis in Twitter. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016).

Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, et al. 2022. Large dual encoders are generalizable retrievers. In EMNLP, pages 9844–9855.

Zach Nussbaum, John X Morris, Brandon Duderstadt, and Andriy Mulyar. 2024. Nomic embed: Training a reproducible long context text embedder. arXiv preprint arXiv:2402.01613.

Kenta Oono and Taiji Suzuki. 2020. Graph neural networks exponentially lose expressive power for node classification. In ICLR.

Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2024a. Yarn: Efficient context window extension of large language models. In ICLR.

Jie Peng, Runlin Lei, and Zhewei Wei. 2024b. Beyond over-smoothing: Uncovering the trainability challenges in deep graph neural networks. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pages 1878–1887.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In EMNLP, pages 1532–1543.

Nils Reimers, Philip Beyer, and Iryna Gurevych. 2016. Task-oriented intrinsic evaluation of semantic textual similarity. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 87–96.

Andrew Rosenberg and Julia Hirschberg. 2007. V-measure: A conditional entropy-based external cluster evaluation measure. In EMNLP, pages 410–420.

Jon Saad-Falcon, Daniel Y Fu, Simran Arora, Neel Guha, and Christopher Re. 2024. Benchmarking and building long-context retrieval models with loco and m2-bert. In ICML.

Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen. 2018. Carer: Contextualized affect representations for emotion recognition. In EMNLP, pages 3687–3697.

Gizem Sogancioglu, Hakime Öztürk, and Arzucan Özgür. 2017. Biosses: a semantic sentence similarity estimation system for the biomedical domain. Bioinformatics, pages i49–i58.

Aivin V Solatorio. 2024. Gistembed: Guided in-sample selection of training negatives for text embedding fine-tuning. arXiv preprint arXiv:2402.16829.

Jianlin Su, Jiarun Cao, Weijie Liu, and Yangyiwen Ou. 2021. Whitening sentence representations for better semantics and faster retrieval. arXiv preprint arXiv:2103.15316.

Yiming Tan, Dehai Min, Yu Li, Wenbo Li, Nan Hu, Yongrui Chen, and Guilin Qi. 2023. Evaluation of chatgpt as a question answering system for answering complex questions. arXiv.

David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylem, Arman Cohan, and Hamaneh Hajishirzi. 2020. Fact or fiction: Verifying scientific claims. arXiv preprint arXiv:2004.14974.

Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022a. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533.

Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Improving text embeddings with large language models. arXiv preprint arXiv:2401.00368.

Peihao Wang, Wenqing Zheng, Tianlong Chen, and Zhangyang Wang. 2022b. Anti-oversmoothing in deep vision transformers via the fourier domain analysis: From theory to practice. In ICLR.

Tongzhou Wang and Phillip Isola. 2020. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In UCNK, pages 9929–9939. PMLR.

A Waswani, N Shazeer, N Parmar, J Uszkoreit, L Jones, A Gomez, L Kaiser, and I Polosukhin. 2017. Attention is all you need. In NIPS.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierrette Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38–45.

Fangzhao Wu, Ying Qiao, Jiun-Hung Chen, Chuhuan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wu, et al. 2020. Mind: A large-scale dataset for news recommendation. In ACL, pages 3597–3606.

Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muenninghoff, Defu Lian, and Jian-Yun Nie. 2023. C-pack: Packaged resources to advance general chinese embedding. arXiv preprint arXiv:2309.07597.

Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In ICLR.

Shicheng Xu, Liang Pang, Huawei Shen, Xueqi Cheng, and Tat-Seng Chua. 2024. Search-in-the-chain: Interactively enhancing large language models with search for knowledge-intensive tasks. In WWW, pages 1362–1373.

Andrew Yates, Rodrigo Nogueira, and Jimmy Lin. 2021. Pretrained transformers for text ranking: Bert and beyond. In WSDM, pages 1154–1156.

Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. NIPS.

Wayne Xin Zhao, Jing Liu, Ruixiang Ren, and Ji-Rong Wen. 2024. Dense text retrieval based on pretrained language models: A survey. ACM Transactions on Information Systems, 42(4):1–60.

Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, et al. 2021. Qmsum: A new benchmark for query-based multi-domain meeting summarization. arXiv preprint arXiv:2104.05938.

Dawei Zhu, Liang Wang, Nan Yang, Yifan Song, Wen-hao Wu, Furu Wei, and Sujian Li. 2024. Longembed: Extending embedding models for long context retrieval. arXiv preprint arXiv:2404.12096.
A  Background Information about Fourier Analysis

In this appendix, we provide additional background information on Fourier analysis. Specifically, consider the discrete Fourier transform (DFT) in the real-valued domain, denoted as \( \mathcal{F} : \mathbb{R}^n \to \mathbb{C}^n \). The DFT can be expressed in matrix form as shown below:

\[
DFT = \frac{1}{\sqrt{n}} \begin{bmatrix}
1 & 1 & \cdots & 1 \\
1 & e^{2\pi j} & \cdots & e^{2\pi j(n-1)} \\
\vdots & \vdots & \ddots & \vdots \\
1 & e^{2\pi j(k-1)} & \cdots & e^{2\pi j(k-1)(n-1)} \\
\vdots & \vdots & \ddots & \vdots \\
1 & e^{2\pi j(n-1)} & \cdots & e^{2\pi j(n-1)^2}
\end{bmatrix}
\]

and inverse discrete Fourier transform is \( DFT^{-1} = DFT^\top = DFT \). In signal processing, we can regard matrices as multi-channel signals. For example, \( X \in \mathbb{R}^{n \times d} \) means \( d \)-channel \( n \)-length signals. When the DFT and inverse DFT are applied to multi-channel signals, each channel is transformed independently. That is,
\[
\mathcal{F}(X) = [\mathcal{F}(x_1) \quad \cdots \quad \mathcal{F}(x_d)] = DFT \cdot X.
\]
Hereby, we can independently operate \( \mathcal{DC}[\cdot] \) and \( \mathcal{HC}[\cdot] \) on the echo channel using the matrices in Eqn. 6. Then we can write \( \mathcal{DC}[\cdot] \) as below:
\[
\mathcal{DC}[x] = DFT^{-1} \operatorname{diag}(1, 0, \cdots, 0) DFT x
= \frac{1}{n} 11^T x.
\]
Conversely, we can denote \( \mathcal{HC}[\cdot] \) as:
\[
\mathcal{HC}[x] = DFT^{-1} \operatorname{diag}(0, 1, \cdots, 1) DFT x
= DFT^{-1} (I - \operatorname{diag}(1, 0, \cdots, 0)) DFT x
= (I - \frac{1}{n} 11^T) x.
\]

B  Detailed Proofs

B.1  Proof of Theorem 2

We start our analysis by providing a lemma.

Lemma 5. The following holds \( \|AB\|_F \leq \|A\|_2 \|B\|_F \) and \( \|AB\|_F \leq \|A\|_F \|B\|_2 \).

Proof. Denote \( B = (b_1 \ \cdots \ b_n) \) and we have \( AB = (Ab_1 \ \cdots \ Ab_n) \). From the definition of the spectral norm, we have: \( \|A\|_2 \geq \frac{\|Ab_i\|_2}{\|b_i\|_2} \). Taking the average of the right-hand side, we obtain: \( \|A\|_2^2 \geq \sum_{i=1}^n \frac{\|b_i\|_2^2 \|Ab_i\|_2^2}{\|b_i\|_2^2} \). This implies: \( \|A\|_2^2 \|B\|_F^2 \geq \sum_{i=1}^n \|Ab_i\|_2^2 = \|AB\|_F^2 \). Finally, the last step utilizes the result \( \|A\|_F^2 = \sum_{i,j} |a_{ij}|^2 = \sum_{j=1}^n \|a_j\|_2^2 \). Because both the spectral norm and the Frobenius norm of a matrix remain unchanged under transposition, we have \( \|AB\|_F = \|B^\top A^\top\|_F \leq \|B^\top\|_2 \|A^\top\|_F = \|A\|_F \|B\|_2 \).

Theorem 6. (Filter Rate of SA) Let \( \sigma_a \) be the largest singular value of \( \mathcal{HC}[A] \). Define \( \mathrm{SA}(X) = AXW_V \) as the output of a self-attention module, then
\[
\|\mathcal{HC}[\mathrm{SA}(X)]\|_F \leq \sigma_a \|W_V\|_2 \|\mathcal{HC}[X]\|_F. \tag{6}
\]
Proof. First, we write \( X = \mathcal{DC}[X] + \mathcal{HC}[X] = \frac{1}{n} 11^T X + H \), where \( H = \mathcal{HC}[X] \) represents the remaining part of the original signals.
\[
\mathcal{HC}[\mathrm{SA}(X)] = \left(I - \frac{1}{n} 11^T\right) A X W_V \tag{7}
= \left(I - \frac{1}{n} 11^T\right) A (\frac{1}{n} 11^T X + H) W_V \tag{8}
= \frac{1}{n} \left(I - \frac{1}{n} 11^T\right) A 11^T X W_V \tag{9}
+ \left(I - \frac{1}{n} 11^T\right) A H W_V \tag{10}
= \left(I - \frac{1}{n} 11^T\right) A H W_V \tag{11}
\]
Therefore,
\[
\|\mathcal{HC}[\mathrm{SA}(X)]\|_F = \left\| \left(I - \frac{1}{n} 11^T\right) A H W_V \right\|_F \tag{12}
\leq \left\| \left(I - \frac{1}{n} 11^T\right) A \right\|_2 \|W_V\|_2 \|H\|_F \tag{13}
= \sigma_a \|W_V\|_2 \|H\|_F \tag{14}
\]
The Eqn. 13 leverages inequality in Lemma 5.

B.2  Proof of Theorem 3

Theorem 7. (Filter Rate of Different Input Length n) Let \( X W_Q \) and \( X W_K \) be a Gaussian matrix, where elements \( q_{ij} \sim \mathcal{N}(0, \sigma_q^2) \) and \( k_{ij} \sim \mathcal{N}(0, \sigma_k^2), \forall i, j \). Let \( x_{ij} = q_i^\top k_j / \sqrt{d} \) the attention score of pair \( i, j \), whose variance can be expressed as \( \sigma_s^2 = \sigma_q^2 \sigma_k^2 + C_{cross} \), where \( C_{cross} \) is the cross-covariance of the squared queries and keys (Goodman, 1960). Then we have
\[
\sigma_a \leq \sqrt{\frac{n}{2 \sqrt{1 + \frac{1}{e^{2\pi s}} (n-1)^{\frac{3}{2}} + 1}}}, \tag{15}
\]
where \( \sigma_a \) decreases with \( n \) increasing.
Proof. First, we have
\[
\sigma_a = \left\| \left( I - \frac{1}{n} \mathbf{1}\mathbf{1}^T \right) A \right\|_2 \tag{16}
\]
\[
\leq \left\| I - \frac{1}{n} \mathbf{1}\mathbf{1}^T \right\|_2 \|A\|_2 \tag{17}
\]
\[
\leq \|A\|_F \tag{18}
\]

The Eqn. 18 leverages \( \|I - \frac{1}{n} \mathbf{1}\mathbf{1}^T\|_2 = 1 \) and \( \|A\|_2 \leq \|A\|_F \). Now we need to upper bound \( \|A\|_F \). Generally, the product of two independent Gaussian variables has a density in the form of a modified Bessel function of the second kind (Nahshan et al., 2024). When the vector dimensions are sufficiently large, the Central Limit Theorem implies that the distribution of the dot product between \( q_i \) and \( k_j \) can be approximated by a Gaussian distribution with zero mean and variance \( \sigma_s^2 \). As mentioned in Theorem 3, the variance of \( q_i^\top k_j \) can be expressed as \( \sigma^2 = \sigma_q^2 \sigma_k^2 + C_{\text{cross}} \), where \( C_{\text{cross}} = \operatorname{Cov}(q^2, k^2) - \operatorname{Cov}(q, k)^2 \) is the cross-covariance of the squared queries and keys (Goodman, 1960). Thus we can suppose that each element \( p_j \sim \mathcal{N}(0, \sigma_s) \) in the matrix \( X W_Q W_K^T X^T \) is independent, where \( j \in (1, \cdots, n) \):

\[
\|A\|_F = \sqrt{\sum_{j=1}^n \left( \frac{\sum_{i=1}^n e^{x_i j}}{\sum_{i=1}^n e^{x_i}} \right)^2} \tag{19}
\]
\[
= \sqrt{\frac{\sum_{j=1}^n e^{2 x_j}}{2 \sum_{i=1}^n \sum_{j=1, j \neq i}^n e^{x_i + x_j} + \sum_{j=1}^n e^{2 x_j}}} \tag{20}
\]
\[
= \frac{n e^{n + 2 \sigma^2 - \frac{1}{2} \ln \frac{e^{4 \sigma^2} - 1}{n}}}{2 e^{\ln(n-1)+\sigma^2 - \frac{1}{2} \ln \frac{e^{2 \sigma^2} - 1}{n-1}} + e^{\ln n + 2 \sigma^2 - \frac{1}{2} \ln \frac{e^{4 \sigma^2} - 1}{n}}} \tag{21}
\]
\[
= \sqrt{\frac{n}{2 \sqrt{1 + \frac{1}{e^{2 \sigma^2}} (n-1)^2} + 1}}. \tag{22}
\]

The derivation in Eqn. 21 primarily relies on the theorem from Fenton (1960), which addresses the sum of log-normal variables. First, we have \( e^x \sim \operatorname{LogNormal}(0, \sigma_x^2) \) and \( e^{2x} \sim \operatorname{LogNormal}(0, 4\sigma_x^2) \). Additionally, we assume that \( e^{x_i} \) and \( e^{x_j} \) are independent, leading to \( e^{x_i + x_j} \sim \operatorname{LogNormal}(0, 2\sigma_x^2) \). Now, considering the sum of log-normal variables, Fenton (1960)'s theorem provides that, for moderate values of \( \sigma^2 \), the sum of zero-mean i.i.d. log-normal variables can be approximated by another log-normal distribution with mean \( \mu_\Sigma \) and variance \( \sigma_\Sigma^2 \), where:
\[
\sigma_\Sigma^2 = \ln \left( \frac{1}{n} \left( e^{\sigma^2} - 1 \right) + 1 \right);
\]
\[
\mu_\Sigma = \ln n + (\sigma^2 - \sigma_\Sigma^2)/2.
\]

For moderate values of \( n \) and \( \sigma^2 \), the variance \( \sigma_\Sigma^2 \) can be approximated as:
\[
\sigma_\Sigma^2 \approx \ln \left( \frac{1}{n} \left( e^{\sigma^2} - 1 \right) \right).
\]

Thus, the sum \( \sum_{j=1}^n e^{2 x_j} \) follows a log-normal distribution:
\[
\sum_{j=1}^n e^{2 x_j} \sim \operatorname{LogNormal} \left( \ln n + 2 \sigma^2 - \frac{1}{2} \ln \left( \frac{1}{n} \left( e^{\sigma^2} - 1 \right) \right), \ln \left( \frac{1}{n} \left( e^{\sigma^2} - 1 \right) \right) \right).
\]
Similarly, the sum \( \sum_{i=1}^n \sum_{j=1, j \neq i}^n e^{x_i + x_j} \) follows:
\[
\sum_{i=1}^n \sum_{j=1, j \neq i}^n e^{x_i + x_j} \sim \operatorname{LogNormal} \left( \ln n(n-1) + \sigma^2 - \frac{1}{2} \ln \left( \frac{e^{2 \sigma^2} - 1}{n(n-1)} \right), \ln \left( \frac{e^{2 \sigma^2} - 1}{n(n-1)} \right) \right).
\]
From these, Eqn. 21 follows naturally. Finally, we have
\[
\sigma_a \leq \sqrt{\frac{n}{2 \sqrt{1 + \frac{1}{e^{2 \sigma^2}} (n-1)^2} + 1}},
\]
where \( \sigma_a \) decreases with \( n \) increasing.

B.3 Proof of Corollary 4

Theorem 8. (Length Collapse in Text Embeddings)
Let \( x_1, x_2 \in \mathbb{R}^d \) be text embeddings of two documents with length \( n \), decomposed as:
\[
x_i = DC x_i + HC x_i
\]
Assume \( \|HC x_i\|_2 \leq \beta(n) \) where \( \beta(n) = O(1/\sqrt{n}) \), and \( \|DC x_i\|_2 \geq \alpha > 0 \) is a constant. Then the cosine similarity satisfies:
\[
\lim_{n \to \infty} \cos(x_1, x_2) = 1
\]
Proof. Given the two texts embeddings \( x_1 \) and \( x_2 \), we have
\[
\cos(x_1, x_2) = \frac{(\mathcal{H}C[x_1] + \mathcal{D}C[x_1]) (\mathcal{H}C[x_2]^T + \mathcal{D}C[x_2]^T)}{\|x_1\|_2 \|x_2\|_2} \tag{23}
\]
\[
= \frac{\mathcal{H}C[x_1] \mathcal{H}C[x_2]^T + \mathcal{D}C[x_1] \mathcal{D}C[x_2]^T}{\|x_1\|_2 \|x_2\|_2} \tag{24}
\]
\[
\geq \frac{\alpha^2}{\sqrt{\alpha^2 + \alpha_1^2} \sqrt{\alpha^2 + \alpha_2^2}}, \tag{25}
\]
where \( \alpha_1 \) and \( \alpha_2 \) represent the maximum values in the frequency domain of \( \mathcal{H}C[x_1] \) and \( \mathcal{H}C[x_2] \) and \( \alpha \) represent the value of \( \mathcal{D}C[x_1] \) and \( \mathcal{D}C[x_2] \), respectively, after applying the discrete Fourier transform. Eqn. 24 leverages that \( \mathcal{H}C[\cdot] \) and \( \mathcal{D}C[\cdot] \) are orthogonal. Eqn. 25 leverages the assumption that the mean of word embeddings in natural language texts maintains a relatively consistent representation. Thus \( \mathcal{H}C[x_1] \) and \( \mathcal{H}C[x_2] \) have the same
value \( \alpha \) after applying the discrete Fourier transform. Finally, according to Theorem 3, \( \alpha_1 \) and \( \alpha_2 \) will gradually decrease with \( n \) grows, leading to a higher cosine similarity between \( x_1 \) and \( x_2 \). \( \square \)

C More Experiments and Analysis

C.1 Rewriting Process
To investigate the differences in embedding distributions across texts of varying lengths, we use the Llama3 (i.e., Llama-3.1-8B-Instruct) (Dubey et al., 2024) model to rewrite the texts. Specifically, we used two prompts, “Please express the given text in one sentence. No more than 10 tokens. [[original text]]” and “Please use few sentences to summarize the given text. [[original text]]”, to summarize the texts. This rewriting allows the texts to retain the same semantics while having shorter lengths. By studying the differences in texts of varying lengths, we conclude that the cause of the Length Collapse phenomenon is that longer texts cluster to each other in embedding space.

C.2 Details on Figure 6
To verify Theorem 2, we illustrate the high-frequency intensity of each layer’s output along with its theoretical upper limit. Our visualization is based on the official checkpoint of 12-layer ANCE, GIST, and BGE. We use a logarithmic scale for the purpose of a better view. Let \( X_l \) denote the output of the \( l \)-th layer. For red line, we directly calculate \( \log(\| \mathcal{H}C [X_{l+1}] \|_F / \| \mathcal{H}C [X_l] \|_F ) \) at each layer. In practice, models typically employ multi-head attention, so we replace \( \| W_V \|_2 \) in Eqn. 1 with \( \sigma_1 H \), where \( \sigma_1 = \max_{h=1}^H \| W_{Vh}^b \|_2 \). For the blue line, we obtain the coefficient \( \sigma_1 \) with respect to network parameters and apply the logarithmic scale. To summarize, Figure 6 implies a convergence rate, which is consistent with our Theorem 2.

C.3 Details on Figure 7
To verify Theorem 3, we visualize the value of \( \sigma_a \) across different text lengths. Our visualization is based on the texts from NFCorpus. We sample 100 samples for each bin from 0 to 500 with a bin size of 50. The \( \sigma_a \) value is computed as the average of the \( \sigma_a \) based on the attention of all heads before the output of the final layer. To summarize, Figure 7 implies that \( \sigma_a \) shows a decreasing trend as \( n \) increases, which is consistent with our Theorem 3. Moreover, \( \sigma_a \) also increases as \( \tau \) decreases, further validating our proposed method, TempScale. E5 model shows an increasing \( \sigma_a \) with length, which may be due to anomalies in the deep layer attention patterns.

C.4 More Analysis about Assumption in Theorem 4
In Corollary 4, we hypothesize and verify that all-natural language sequences tend to have a relatively consistent representation. As a result, different texts tend to exhibit consistent low-pass signals after losing high-frequency information, leading to ultimately consistent text embeddings. This leads to an increase in cosine similarity for longer texts. However, as shown in Figure 8, we repeat the words “dog” and “word” \( n \) times and calculate the similarity between these two texts. The results show that even when the sequences do not overlap, the text embeddings tend to converge to similar representations as the sequence length increases. This further demonstrates that Length Collapse causes completely different sequences to converge toward similarity.

C.5 More Discussion about Other Works
Other Components in Transformer. After discussing how self-attention contributes to Length Collapse, we proceed to examine the influence of other modules in the transformer, such as multi-head, residual, and FFN. Fortunately, previous work (Wang et al., 2022b) has talked about whether these components can effectively alleviate the low-pass filtering drawbacks. The proof demonstrates that while these components help preserve high-frequency signals, they do not alter the fact that the MSA block, as a whole, functions solely with the representational power of a low-pass filter. Furthermore, the ability of these models to preserve high-frequency signals is solely determined by their internal parameters and architecture, independent of the input text length. As a result, these modules do not impact our analysis of Length Collapse in practical models.

Difference from Over-Smoothing in Deeper Layers. In previous research (Wang et al., 2022b), it has been noted that a self-attention module acts as a low-pass filter, causing input feature maps to gradually lose high-frequency signals as the model layers go deeper. Furthermore, other studies (Oono and Suzuki, 2020; Cai and Wang, 2020) indicate that the node features of Graph Convolutional Networks (GCNs) can become exponentially trapped in the null space of the graph Laplacian matrix. The
Figure 6: Visualization of the intensity of high-frequency components and their theoretical upper bounds. The blue line is defined by \( \log(\|\mathcal{H}C[\mathbf{X}_{t+1}]\|_F / \|\mathcal{H}C[\mathbf{X}_t]\|_F) \), and the red line is estimated using the results in Theorem 2.

(a) ANCE        (b) GTR        (c) GSIT        (d) BGE        (e) E5

Figure 7: \( \sigma_a \) of \( \mathcal{H}C[\mathbf{A}] \) before the last layer across different text length under different \( \tau \) setting.

root cause of this phenomenon is that both graph Laplacian matrices and self-attention matrices consistently exhibit a dominant eigenvector, commonly referred to as the DC component. While these studies address over-smoothing in deeper layers, we focus on how the low-pass filtering process changes as the input sequence lengthens, specifically examining over-smoothing in longer sequences.

C.6 More discussions on Classification and Clustering Tasks

For data grouping tasks like classification and clustering, in the case of \( N \)-class tasks, we can think of the model as learning \( N \) classification boundaries. The farther the text embedding is from the boundary, the closer the output probability approaches 1 or 0. As shown in Figure 9 (left), we plot the entropy of the model output probabilities across different length intervals. The model outputs higher entropy for longer texts, which may be because the embeddings of longer texts are positioned closer to the center of the space as described in Figure 1b, resulting in a shorter distance to various classification boundaries. Meanwhile, in Figure 9 (right), we can also observe that accuracy and entropy follow the same trend: the model achieves higher accuracy when it has lower entropy. In other words, the model performs better if the text embeddings are farther from the boundary. After applying TempScale, a decreased entropy will result in increased accuracy. This further supports the relationship between entropy and accuracy in classification tasks. Moreover, if the model exhibits a more severe Length Collapse phenomenon, meaning a greater performance drop on longer texts, the more performance improvement it experiences after applying TempScale. This suggests that one possible reason TempScale is effective on shorter text datasets is that it adjusts the distribution differences between texts of varying lengths. As shown in Figure 1c, after applying TempScale, the distribution of text embeddings across different length intervals becomes more uniform, which facilitates the model in learning length-agnostic parameters.

C.7 More Results on Longer Texts

Can the temperature be set based on the length of the text? In the previous experiments, we use the same temperature \( \tau \) for scaling all texts in the same task. However, our analysis indicates that texts of different lengths have varying filtering rates, so a natural idea is to use different temperatures for texts of different lengths. As shown in Figure 12, we plot the performance trend for texts under the same settings in Figure 9 as the temperature varies. The results indicate that a higher temperature is optimal for short texts, while a lower temperature is preferable for long texts, as confirmed by the results in Table 2. When performing retrieval tasks, we can also set different temperatures for queries and documents to achieve better performance. As shown in Figure 11, on the QMSum dataset, we can consistently achieve better performance by setting a lower temperature for queries. Moreover, as shown in Figure 13, compared to the QMSum dataset, SummScreenFD requires a lower temperature for scaling the document due to its longer length. This further supports the conclusion that longer texts require a lower temperature for scaling. In Figure 10, we observe a similar phenomenon across other models. Except for ANCE, the performance of other models on long queries
Figure 8: The cosine similarity of embeddings of texts generated by repeating words “dog” and “cat”.

Figure 9: Probability entropy and classification accuracy of models across different length intervals on Amazon-Polarity dataset. Each model has bars representing intervals of 100 in length, with 500 text samples per interval, covering a range from 0 to 300. Bars represent raw outputs, with green and red hatching indicating increases and decreases after TempScale, respectively.

decreases as the temperature decreases. This suggests that long queries require a lower temperature to mitigate the Length Collapse phenomenon. However, ANCE’s performance degradation with decreasing temperature is likely attributable to its inherent limitations in processing long texts.

D Length Collapse in LLM-based Embedding Models

To explore whether Length Collapse also occurs in long-context LLMs, we select three widely used LLM-based embedding models from the MTEB benchmark—bge-multilingual-gemma2 (Chen et al., 2024a), NV-Embed-v2 (Lee et al., 2024), and e5-mistral-7b-instruct (Wang et al., 2022a)—all of which rank within the top 20 on the MTEB leaderboard. We conduct experiments using three commonly used long-text datasets, including nq and hotpotqa in LongRAG (Jiang et al., 2024) and LongAlpaca-12k (Chen et al., 2024b), which are selected based on relevance to the keyword “long” and chosen for their evenly distributed text lengths. Specifically, we analyze shifts in embedding space across different text length intervals by calculating the average Euclidean distance of each embedding from the central embedding (the mean of all embeddings) and computing cosine similarity between each pair of embeddings as in Table 4 and Table 5.

The experimental results show that as text length increases, the embeddings from different LLM-based embedding models exhibit a gradual convergence trend, with the average Euclidean distance between embeddings decreasing and pairwise cosine similarity increasing. This indicates that even mainstream long-context LLM embedding models tend to experience embedding convergence (Length Collapse) and reduced distinctiveness in long-text processing due to the low-pass filtering effect.

E More Discussions about TempScale

E.1 Relationship with Contrastive Learning

Background and Motivation. Contrastive learning is widely used to address embedding space anisotropy, reducing high similarity between random text samples of varying lengths by maximizing distances among negative pairs and aligning positive pairs. While it improves embedding quality in many applications, its impact on Length Collapse remains unexplored. This section examines how contrastive learning affects Length Collapse, assessing its contributions and limitations.

As shown in previous works (Wang and Isola, 2020), the InfoNCE loss, when scaled to a large number of negative samples, can be decomposed into two primary components: Alignment and Uniformity. Alignment ensures that positive pairs—texts with similar content—are close in the embedding space, while Uniformity spreads embeddings of negative pairs to prevent them from clustering excessively. Mathematically, as the number of negative samples \( M \to \infty \), the normalized InfoNCE loss can be expressed as:
Figure 10: Results about performance difference between long query and short query across varying temperature \( \tau \) on SummScreenFD dataset.

Figure 11: Relative performance compared to the raw results with varying query (Q Temperature) and document (D Temperature) temperatures using different models on QMSum.

<table>
  <tr>
    <th>Dataset</th>
    <th>Model</th>
    <th colspan="3">0-1000</th>
    <th colspan="3">1000-2000</th>
    <th colspan="3">2000-3000</th>
  </tr>
  <tr>
    <td rowspan="3">nq</td>
    <td>bge-multilingual-gemma2</td>
    <td>0.94</td>
    <td>0.92</td>
    <td>0.91</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>NV-Embed-v2</td>
    <td>0.98</td>
    <td>0.91</td>
    <td>0.89</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>e5-mistral-7b-instruct</td>
    <td>0.68</td>
    <td>0.64</td>
    <td>0.64</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="3">hotpotqa</td>
    <td>bge-multilingual-gemma2</td>
    <td>0.93</td>
    <td>0.88</td>
    <td>0.87</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>NV-Embed-v2</td>
    <td>0.95</td>
    <td>0.84</td>
    <td>0.79</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>e5-mistral-7b-instruct</td>
    <td>0.70</td>
    <td>0.61</td>
    <td>0.57</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="3">LongAlpaca-12k</td>
    <td>bge-multilingual-gemma2</td>
    <td>0.71</td>
    <td>0.48</td>
    <td>0.49</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>NV-Embed-v2</td>
    <td>0.89</td>
    <td>0.86</td>
    <td>0.87</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>e5-mistral-7b-instruct</td>
    <td>0.69</td>
    <td>0.53</td>
    <td>0.53</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 4: Euclidean distance results for different datasets and LLM-based embedding models.

<table>
  <tr>
    <th>Dataset</th>
    <th>Model</th>
    <th colspan="3">0-1000</th>
    <th colspan="3">1000-2000</th>
    <th colspan="3">2000-3000</th>
  </tr>
  <tr>
    <td rowspan="3">nq</td>
    <td>bge-multilingual-gemma2</td>
    <td>0.39</td>
    <td>0.42</td>
    <td>0.45</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>NV-Embed-v2</td>
    <td>0.32</td>
    <td>0.47</td>
    <td>0.59</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>e5-mistral-7b-instruct</td>
    <td>0.75</td>
    <td>0.77</td>
    <td>0.78</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="3">hotpotqa</td>
    <td>bge-multilingual-gemma2</td>
    <td>0.51</td>
    <td>0.52</td>
    <td>0.59</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>NV-Embed-v2</td>
    <td>0.48</td>
    <td>0.59</td>
    <td>0.69</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>e5-mistral-7b-instruct</td>
    <td>0.75</td>
    <td>0.81</td>
    <td>0.85</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="3">LongAlpaca-12k</td>
    <td>bge-multilingual-gemma2</td>
    <td>0.76</td>
    <td>0.90</td>
    <td>0.89</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>NV-Embed-v2</td>
    <td>0.48</td>
    <td>0.65</td>
    <td>0.68</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>e5-mistral-7b-instruct</td>
    <td>0.78</td>
    <td>0.87</td>
    <td>0.87</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 5: Pair cosine similarities results for different datasets and LLM-based embedding models.
Figure 12: Results about performance difference between long query and short query across varying temperature \( \tau \) on AmazonPolarity dataset.

Figure 13: Relative performance compared to the raw results with varying query (Q Temperature) and document (D Temperature) temperatures using different models on SummscreenFD

\[
\lim_{M \to \infty} L(f, \tau) - \log M = -\frac{1}{2} E_{(x, x^+) \sim p_{data}} \left[ f(x)^T f(x^+) \right] \\
+ E_{x \sim p_{data}} \left[ \log E_{x^+ \sim p_{data}} \left[ e^{f(x)^T f(x^-)/\tau} \right] \right].
\]

These properties make contrastive learning especially effective in tasks such as retrieval, where maximizing inter-sample variance is crucial.

E.1.1 Why Contrastive Learning Cannot Fully Address Length Collapse
While contrastive learning alleviates high similarity issues across all text lengths, it does not entirely resolve the Length Collapse issue inherent in PLM-based models. Length Collapse arises from self-attention’s tendency to push embeddings for longer texts toward a concentrated representation space. This characteristic is unaffected by contrastive learning’s alignment or uniformity mechanisms because contrastive learning optimizes the relative positioning of positive and negative pairs rather than mitigating the length-induced clustering trend.

E.2 Comparison with Other Post-processing Techniques
In addition to TempScale, several post-processing methods have been proposed to address high similarity in embeddings, such as the Flow Function (Li et al., 2020) and Whitening (Su et al., 2021). For comparative analysis, we implemented the last2avg version of the Flow Function, as the full flow version requires additional training, and the Whitening method is based on its original formulation. The results, summarized in Table 6, highlight that these methods do not perform as effectively in our specific application. These results indicate that Flow and Whitening, originally developed for standard BERT embeddings, are less effective with the fine-tuned Transformer model used in this paper, which includes additional normalization layers. While TempScale reduces similarity in long-sequence embeddings, achieving distributional consistency across different sequence lengths is more critical for performance. Therefore, lowering similarity alone may not significantly improve downstream tasks.

E.3 Comparison with Other Similar Long-Text Methods
To thoroughly evaluate our TempScale approach, we compare it with two related attention scaling methods: softmax \( \left( \frac{\log Q K^T}{\sqrt{d}} \right) V \) (Method1) (Chiang and Cholak, 2022) and YaRN (Peng et al., 2024a). In YaRN, the scaling formula for the attention matrix is given as softmax \( \left( \frac{1}{\tau \sqrt{d}} Q K^T \right) V \), where \( \frac{1}{\tau} = 0.1 \ln s + 1 \), and \( s = \frac{L'}{L} \), with \( L' \) representing the extended context window length and \( L \) the original context window length. These methods propose different scaling strategies, but they differ in their theoretical motivations, applications, and experimental outcomes.

While Method1, YaRN, and TempScale share a similar structural approach, they tackle different challenges. Method1 is for binary classification
<table>
  <tr>
    <th>Model</th>
    <th>Rerank.</th>
    <th>Summ.</th>
    <th>Class.</th>
    <th>Clust.</th>
    <th>LongEmbdRetr.</th>
    <th>STS</th>
    <th>BeirRetr.</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>ANCE</td>
    <td>49.09</td>
    <td>29.58</td>
    <td>55.27</td>
    <td>33.04</td>
    <td>34.02</td>
    <td>66.32</td>
    <td>36.87</td>
    <td>43.45</td>
  </tr>
  <tr>
    <td>+Flow</td>
    <td>48.49</td>
    <td>29.57</td>
    <td>56.34</td>
    <td>32.11</td>
    <td>31.92</td>
    <td>66.02</td>
    <td>36.57</td>
    <td>43.00</td>
  </tr>
  <tr>
    <td>+Whitening</td>
    <td>25.16</td>
    <td>21.76</td>
    <td>25.82</td>
    <td>3.21</td>
    <td>1.29</td>
    <td>2.78</td>
    <td>0.79</td>
    <td>11.54</td>
  </tr>
  <tr>
    <td>GTR</td>
    <td>54.23</td>
    <td>29.67</td>
    <td>55.10</td>
    <td>38.65</td>
    <td>37.33</td>
    <td>70.11</td>
    <td>44.98</td>
    <td>47.15</td>
  </tr>
  <tr>
    <td>+Flow</td>
    <td>37.49</td>
    <td>28.09</td>
    <td>41.98</td>
    <td>21.87</td>
    <td>7.65</td>
    <td>35.55</td>
    <td>1.35</td>
    <td>24.85</td>
  </tr>
  <tr>
    <td>+Whitening</td>
    <td>24.63</td>
    <td>20.69</td>
    <td>25.71</td>
    <td>3.06</td>
    <td>1.35</td>
    <td>-3.02</td>
    <td>0.63</td>
    <td>10.44</td>
  </tr>
  <tr>
    <td>GIST</td>
    <td>58.55</td>
    <td>31.14</td>
    <td>64.75</td>
    <td>44.77</td>
    <td>38.21</td>
    <td>75.61</td>
    <td>52.77</td>
    <td>52.26</td>
  </tr>
  <tr>
    <td>+Flow</td>
    <td>58.16</td>
    <td>30.56</td>
    <td>65.68</td>
    <td>44.67</td>
    <td>35.83</td>
    <td>74.98</td>
    <td>51.40</td>
    <td>51.61</td>
  </tr>
  <tr>
    <td>+Whitening</td>
    <td>24.08</td>
    <td>24.26</td>
    <td>20.06</td>
    <td>9.83</td>
    <td>1.78</td>
    <td>-4.31</td>
    <td>0.81</td>
    <td>10.93</td>
  </tr>
  <tr>
    <td>BGE</td>
    <td>58.87</td>
    <td>31.03</td>
    <td>64.79</td>
    <td>45.80</td>
    <td>37.46</td>
    <td>75.88</td>
    <td>55.29</td>
    <td>52.73</td>
  </tr>
  <tr>
    <td>+Flow</td>
    <td>58.48</td>
    <td>30.73</td>
    <td>64.77</td>
    <td>45.28</td>
    <td>36.95</td>
    <td>74.43</td>
    <td>54.52</td>
    <td>52.16</td>
  </tr>
  <tr>
    <td>+Whitening</td>
    <td>24.75</td>
    <td>19.61</td>
    <td>16.50</td>
    <td>4.23</td>
    <td>2.39</td>
    <td>1.60</td>
    <td>0.47</td>
    <td>9.94</td>
  </tr>
  <tr>
    <td>E5-4K</td>
    <td>53.12</td>
    <td>30.58</td>
    <td>61.72</td>
    <td>41.01</td>
    <td>56.01</td>
    <td>71.77</td>
    <td>47.22</td>
    <td>51.63</td>
  </tr>
  <tr>
    <td>+Flow</td>
    <td>52.48</td>
    <td>29.82</td>
    <td>62.33</td>
    <td>39.95</td>
    <td>44.71</td>
    <td>68.55</td>
    <td>30.68</td>
    <td>46.93</td>
  </tr>
  <tr>
    <td>+Whitening</td>
    <td>24.94</td>
    <td>22.14</td>
    <td>22.73</td>
    <td>4.04</td>
    <td>1.85</td>
    <td>1.96</td>
    <td>0.56</td>
    <td>11.17</td>
  </tr>
</table>

Table 6: Performance comparison of different post-processing methods across various tasks

tasks, specifically determining if the first string in a sequence of binary strings is a “1”. It focuses on resolving straightforward decision-making problems in binary data. YaRN, by contrast, extends the context window of LLMs without requiring retraining, particularly adjusting the attention mechanism to handle larger contexts. TempScale addresses Length Collapse in embedding models for long texts, improving performance without retraining by preserving distinct representations as text length increases. Moreover, Method1 offers a solution using a specially designed Transformer example, but its motivation is limited by the specificity of this example. In contrast, YaRN introduces an empirically derived scaling formula. TempScale is based on a rigorous low-pass filtering analysis, giving it a strong theoretical foundation and an intuitive explanation.

To effectively compare the performance of the above methods, we tested them on the MTEB benchmark. Since we are not modifying the context window length, we adapt YaRN as softmax \( \left( \frac{\tau}{\sqrt{d}} Q K^T \right) V \), with \( \tau = \lambda \log n + 1 \). We explore values for \( \lambda \) in the range \( \{0.0001, 0.001, 0.01, 0.1, 1\} \). This setup is reasonable, as our findings indicate that longer texts generally benefit from a smaller temperature scale. The experimental results are as shown in Table 7.

The experimental results indicate that neither Method1 nor YaRN effectively adapts to the embedding model scenario. (Although YaRN shows slight improvement when the \( \lambda \) value is small, in this case, Method 2 degenerates into TempScale.) Some potential reasons are as follows: A plausible explanation is that while both methods perform finer scaling on the attention matrix, applying different temperature adjustments across varying text lengths may lead to embeddings from different lengths falling into distinct distributions, which is unfavorable for downstream tasks. Moreover, in generation tasks, Method1 and YaRN succeed likely because of differences in output requirements. For these tasks, the model outputs a probability distribution and samples from it. Minor perturbations generally don’t affect the token output significantly; even if one sequence’s token distribution changes, it doesn’t impact the output of other sequences. In contrast, any substantial change in a single embedding for an embedding model can directly affect the entire downstream performance. For instance, in classification tasks, a classifier model relies on embeddings as input, and in retrieval tasks, a change in embedding impacts document ranking. Overall, for embedding tasks, simply applying a single temperature adjustment better maintains the overall embedding distribution, helping mitigate Length Collapse and achieve better results across various downstream applications.
<table>
  <tr>
    <th>Model</th>
    <th>Rerank.</th>
    <th>Summ.</th>
    <th>Class.</th>
    <th>Clust.</th>
    <th>LongEmbRetr.</th>
    <th>STS</th>
    <th>BeirRetr.</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>ANCE</td>
    <td>49.09</td>
    <td>29.58</td>
    <td>55.27</td>
    <td>33.04</td>
    <td>34.02</td>
    <td>66.32</td>
    <td>36.87</td>
    <td>43.45</td>
  </tr>
  <tr>
    <td>+Method1</td>
    <td>41.80</td>
    <td>29.21</td>
    <td>48.21</td>
    <td>20.72</td>
    <td>6.23</td>
    <td>53.40</td>
    <td>7.74</td>
    <td>29.61</td>
  </tr>
  <tr>
    <td>+YaRN(\( \lambda = 0.0001 \))</td>
    <td>49.09</td>
    <td>29.58</td>
    <td>55.62</td>
    <td>33.01</td>
    <td>34.02</td>
    <td>66.32</td>
    <td>36.86</td>
    <td>43.50</td>
  </tr>
  <tr>
    <td>+YaRN(\( \lambda = 0.001 \))</td>
    <td>49.08</td>
    <td>29.57</td>
    <td>55.62</td>
    <td>33.05</td>
    <td>33.96</td>
    <td>66.32</td>
    <td>36.87</td>
    <td>43.49</td>
  </tr>
  <tr>
    <td>+YaRN(\( \lambda = 0.01 \))</td>
    <td>49.10</td>
    <td>29.31</td>
    <td>55.65</td>
    <td>32.95</td>
    <td>33.92</td>
    <td>66.27</td>
    <td>36.87</td>
    <td>43.44</td>
  </tr>
  <tr>
    <td>+YaRN(\( \lambda = 0.1 \))</td>
    <td>48.95</td>
    <td>29.30</td>
    <td>55.45</td>
    <td>32.65</td>
    <td>32.31</td>
    <td>65.72</td>
    <td>35.77</td>
    <td>42.88</td>
  </tr>
  <tr>
    <td>+YaRN(\( \lambda = 1 \))</td>
    <td>45.71</td>
    <td>29.28</td>
    <td>52.80</td>
    <td>26.16</td>
    <td>10.59</td>
    <td>60.15</td>
    <td>13.34</td>
    <td>34.00</td>
  </tr>
  <tr>
    <td>GTR</td>
    <td>54.23</td>
    <td>29.67</td>
    <td>55.10</td>
    <td>38.65</td>
    <td>37.33</td>
    <td>70.11</td>
    <td>44.98</td>
    <td>47.15</td>
  </tr>
  <tr>
    <td>+Method1</td>
    <td>38.12</td>
    <td>28.44</td>
    <td>40.34</td>
    <td>14.46</td>
    <td>3.61</td>
    <td>47.09</td>
    <td>0.82</td>
    <td>24.70</td>
  </tr>
  <tr>
    <td>+YaRN(\( \lambda = 0.0001 \))</td>
    <td>54.23</td>
    <td>29.68</td>
    <td>55.06</td>
    <td>38.30</td>
    <td>37.42</td>
    <td>70.11</td>
    <td>44.98</td>
    <td>47.11</td>
  </tr>
  <tr>
    <td>+YaRN(\( \lambda = 0.001 \))</td>
    <td>54.23</td>
    <td>29.71</td>
    <td>55.08</td>
    <td>38.29</td>
    <td>37.56</td>
    <td>70.11</td>
    <td>44.97</td>
    <td>47.13</td>
  </tr>
  <tr>
    <td>+YaRN(\( \lambda = 0.01 \))</td>
    <td>54.23</td>
    <td>29.71</td>
    <td>55.04</td>
    <td>38.63</td>
    <td>37.39</td>
    <td>70.06</td>
    <td>45.00</td>
    <td>47.15</td>
  </tr>
  <tr>
    <td>+YaRN(\( \lambda = 0.1 \))</td>
    <td>54.12</td>
    <td>29.34</td>
    <td>54.86</td>
    <td>34.07</td>
    <td>36.53</td>
    <td>69.86</td>
    <td>43.06</td>
    <td>45.98</td>
  </tr>
  <tr>
    <td>+YaRN(\( \lambda = 1 \))</td>
    <td>55.19</td>
    <td>29.00</td>
    <td>47.27</td>
    <td>19.06</td>
    <td>3.99</td>
    <td>61.87</td>
    <td>1.46</td>
    <td>31.12</td>
  </tr>
  <tr>
    <td>GIST</td>
    <td>58.55</td>
    <td>31.14</td>
    <td>64.75</td>
    <td>44.77</td>
    <td>38.21</td>
    <td>75.61</td>
    <td>52.77</td>
    <td>52.26</td>
  </tr>
  <tr>
    <td>+Method1</td>
    <td>58.64</td>
    <td>28.26</td>
    <td>51.05</td>
    <td>28.11</td>
    <td>5.42</td>
    <td>62.43</td>
    <td>7.51</td>
    <td>34.49</td>
  </tr>
  <tr>
    <td>+YaRN(\( \lambda = 0.0001 \))</td>
    <td>58.55</td>
    <td>31.14</td>
    <td>64.26</td>
    <td>44.75</td>
    <td>38.21</td>
    <td>75.61</td>
    <td>52.78</td>
    <td>52.19</td>
  </tr>
  <tr>
    <td>+YaRN(\( \lambda = 0.001 \))</td>
    <td>58.55</td>
    <td>31.12</td>
    <td>64.28</td>
    <td>44.75</td>
    <td>38.20</td>
    <td>75.61</td>
    <td>52.78</td>
    <td>52.18</td>
  </tr>
  <tr>
    <td>+YaRN(\( \lambda = 0.01 \))</td>
    <td>58.53</td>
    <td>31.26</td>
    <td>64.24</td>
    <td>44.78</td>
    <td>38.05</td>
    <td>75.60</td>
    <td>52.79</td>
    <td>52.18</td>
  </tr>
  <tr>
    <td>+YaRN(\( \lambda = 0.1 \))</td>
    <td>58.45</td>
    <td>30.36</td>
    <td>63.86</td>
    <td>44.62</td>
    <td>37.34</td>
    <td>75.31</td>
    <td>52.10</td>
    <td>51.72</td>
  </tr>
  <tr>
    <td>+YaRN(\( \lambda = 1 \))</td>
    <td>55.75</td>
    <td>26.76</td>
    <td>58.79</td>
    <td>36.95</td>
    <td>11.85</td>
    <td>69.00</td>
    <td>23.59</td>
    <td>40.38</td>
  </tr>
  <tr>
    <td>BGE</td>
    <td>58.87</td>
    <td>31.03</td>
    <td>64.79</td>
    <td>45.80</td>
    <td>37.46</td>
    <td>75.88</td>
    <td>55.29</td>
    <td>52.73</td>
  </tr>
  <tr>
    <td>+Method1</td>
    <td>37.78</td>
    <td>29.17</td>
    <td>37.80</td>
    <td>14.36</td>
    <td>2.10</td>
    <td>43.72</td>
    <td>0.84</td>
    <td>23.68</td>
  </tr>
  <tr>
    <td>+YaRN(\( \lambda = 0.0001 \))</td>
    <td>58.86</td>
    <td>31.04</td>
    <td>64.78</td>
    <td>45.77</td>
    <td>37.45</td>
    <td>75.88</td>
    <td>55.29</td>
    <td>52.72</td>
  </tr>
  <tr>
    <td>+YaRN(\( \lambda = 0.001 \))</td>
    <td>58.86</td>
    <td>30.96</td>
    <td>64.78</td>
    <td>45.75</td>
    <td>37.47</td>
    <td>75.87</td>
    <td>55.28</td>
    <td>52.71</td>
  </tr>
  <tr>
    <td>+YaRN(\( \lambda = 0.01 \))</td>
    <td>58.85</td>
    <td>31.02</td>
    <td>64.73</td>
    <td>45.61</td>
    <td>37.26</td>
    <td>75.86</td>
    <td>55.22</td>
    <td>52.65</td>
  </tr>
  <tr>
    <td>+YaRN(\( \lambda = 0.1 \))</td>
    <td>58.86</td>
    <td>30.90</td>
    <td>64.51</td>
    <td>45.19</td>
    <td>36.55</td>
    <td>75.55</td>
    <td>54.96</td>
    <td>52.36</td>
  </tr>
  <tr>
    <td>+YaRN(\( \lambda = 1 \))</td>
    <td>52.65</td>
    <td>29.09</td>
    <td>50.51</td>
    <td>25.50</td>
    <td>2.36</td>
    <td>64.20</td>
    <td>2.14</td>
    <td>32.35</td>
  </tr>
  <tr>
    <td>E5-4K</td>
    <td>53.12</td>
    <td>30.58</td>
    <td>61.72</td>
    <td>41.01</td>
    <td>56.01</td>
    <td>71.77</td>
    <td>47.22</td>
    <td>51.63</td>
  </tr>
  <tr>
    <td>+Method1</td>
    <td>40.90</td>
    <td>24.11</td>
    <td>42.85</td>
    <td>13.64</td>
    <td>3.24</td>
    <td>41.62</td>
    <td>1.05</td>
    <td>23.92</td>
  </tr>
  <tr>
    <td>+YaRN(\( \lambda = 0.0001 \))</td>
    <td>53.12</td>
    <td>30.57</td>
    <td>61.78</td>
    <td>40.77</td>
    <td>56.01</td>
    <td>71.77</td>
    <td>47.22</td>
    <td>51.61</td>
  </tr>
  <tr>
    <td>+YaRN(\( \lambda = 0.001 \))</td>
    <td>53.11</td>
    <td>30.55</td>
    <td>61.78</td>
    <td>40.75</td>
    <td>55.98</td>
    <td>71.77</td>
    <td>47.22</td>
    <td>51.59</td>
  </tr>
  <tr>
    <td>+YaRN(\( \lambda = 0.01 \))</td>
    <td>53.07</td>
    <td>30.42</td>
    <td>61.73</td>
    <td>40.61</td>
    <td>55.53</td>
    <td>71.71</td>
    <td>47.21</td>
    <td>51.47</td>
  </tr>
  <tr>
    <td>+YaRN(\( \lambda = 0.1 \))</td>
    <td>52.64</td>
    <td>30.20</td>
    <td>61.51</td>
    <td>40.19</td>
    <td>48.39</td>
    <td>70.68</td>
    <td>46.36</td>
    <td>50.00</td>
  </tr>
  <tr>
    <td>+YaRN(\( \lambda = 1 \))</td>
    <td>45.00</td>
    <td>29.01</td>
    <td>50.81</td>
    <td>17.99</td>
    <td>2.91</td>
    <td>57.27</td>
    <td>1.46</td>
    <td>29.21</td>
  </tr>
</table>

Table 7: Average main metric on MTEB and LongEmb across Method1 and YaRN.

F Datasets and Evaluation Metrics

Table 8 provides an overview of the datasets used in our experiments. Next, we give a brief description of the tasks involved in the experiments and the corresponding datasets and evaluation metrics they include.

F.1 Classification

In general, we use the provided embedding model to obtain a training set and a test set. The embeddings of the training set are used to train a logistic regression classifier with a maximum of 100 iterations, which is then scored on the test set. The main evaluation metrics are accuracy, average precision, and the F1 score.

AmazonPolarity (Zhang et al., 2015) consists of Amazon customer reviews, each labeled as either “positive” or “negative.”

Banking77 (Casanueva et al., 2020) dataset consists of online banking user queries labeled with one of 77 specific intents.

Emotion (Saravia et al., 2018) comprises Twitter messages categorized by six fundamental emotions: anger, fear, joy, love, sadness, and surprise.

IMDb (Maas et al., 2011) consists of extensive movie reviews categorized as either positive
<table>
  <tr>
    <th>Type</th>
    <th>Name</th>
    <th>Categ.</th>
    <th>#Lang.</th>
    <th>Train Samples</th>
    <th>Dev Samples</th>
    <th>Test Samples</th>
    <th>Train avg. chars</th>
    <th>Dev avg. chars</th>
    <th>Test avg. chars</th>
  </tr>
  <tr>
    <td rowspan="6">BEIR Retrival</td>
    <td>NFCorpus</td>
    <td>s2p</td>
    <td>1</td>
    <td>0</td>
    <td>0</td>
    <td>3,956</td>
    <td>0</td>
    <td>0</td>
    <td>4,462.7</td>
  </tr>
  <tr>
    <td>SciFact</td>
    <td>s2p</td>
    <td>1</td>
    <td>0</td>
    <td>0</td>
    <td>5,483</td>
    <td>0</td>
    <td>0</td>
    <td>1,422.3</td>
  </tr>
  <tr>
    <td>SciFact</td>
    <td>s2p</td>
    <td>1</td>
    <td>0</td>
    <td>0</td>
    <td>5,483</td>
    <td>0</td>
    <td>0</td>
    <td>1,422.3</td>
  </tr>
  <tr>
    <td>SCIDOCs</td>
    <td>s2p</td>
    <td>1</td>
    <td>0</td>
    <td>0</td>
    <td>26,657</td>
    <td>0</td>
    <td>0</td>
    <td>1161.9</td>
  </tr>
  <tr>
    <td>FiQA2018</td>
    <td>s2p</td>
    <td>1</td>
    <td>0</td>
    <td>0</td>
    <td>58,286</td>
    <td>0</td>
    <td>0</td>
    <td>760.4</td>
  </tr>
  <tr>
    <td>Touche2020</td>
    <td>s2p</td>
    <td>1</td>
    <td>0</td>
    <td>0</td>
    <td>382,594</td>
    <td>0</td>
    <td>0</td>
    <td>1720.1</td>
  </tr>
  <tr>
    <td rowspan="7">Classification</td>
    <td>AmazonPolarityClassification</td>
    <td>p2p</td>
    <td>1</td>
    <td>3,600,000</td>
    <td>0</td>
    <td>400,000</td>
    <td>431.6</td>
    <td>0</td>
    <td>431.4</td>
  </tr>
  <tr>
    <td>Banking77Classification</td>
    <td>s2s</td>
    <td>1</td>
    <td>10,003</td>
    <td>0</td>
    <td>3,080</td>
    <td>59.5</td>
    <td>0</td>
    <td>54.2</td>
  </tr>
  <tr>
    <td>EmotionClassification</td>
    <td>s2s</td>
    <td>1</td>
    <td>16,000</td>
    <td>2,000</td>
    <td>2,000</td>
    <td>96.8</td>
    <td>95.3</td>
    <td>96.6</td>
  </tr>
  <tr>
    <td>InAdClassification</td>
    <td>s2p</td>
    <td>1</td>
    <td>25,000</td>
    <td>0</td>
    <td>25,000</td>
    <td>1,325.1</td>
    <td>0</td>
    <td>1,293.8</td>
  </tr>
  <tr>
    <td>MassiveIntentClassification</td>
    <td>s2s</td>
    <td>51</td>
    <td>11,514</td>
    <td>2,033</td>
    <td>2,034</td>
    <td>33.4</td>
    <td>34.8</td>
    <td>34.6</td>
  </tr>
  <tr>
    <td>MassiveScenarioClassification</td>
    <td>s2s</td>
    <td>51</td>
    <td>11,514</td>
    <td>2,033</td>
    <td>2,034</td>
    <td>33.4</td>
    <td>34.8</td>
    <td>34.6</td>
  </tr>
  <tr>
    <td>ToxicConversationsClassification</td>
    <td>s2s</td>
    <td>51</td>
    <td>50,000</td>
    <td>0</td>
    <td>50,000</td>
    <td>298.8</td>
    <td>0</td>
    <td>296.6</td>
  </tr>
  <tr>
    <td rowspan="1">TweetSentimentExtractionClassification</td>
    <td>s2s</td>
    <td>1</td>
    <td>27,481</td>
    <td>0</td>
    <td>3,534</td>
    <td>68.3</td>
    <td>0</td>
    <td>67.8</td>
  </tr>
  <tr>
    <td rowspan="11">Clustering</td>
    <td>ArxivClusteringP2P</td>
    <td>p2p</td>
    <td>1</td>
    <td>0</td>
    <td>0</td>
    <td>732,723</td>
    <td>0</td>
    <td>0</td>
    <td>1,009.9</td>
  </tr>
  <tr>
    <td>ArxivClusteringS2S</td>
    <td>s2s</td>
    <td>1</td>
    <td>0</td>
    <td>0</td>
    <td>732,723</td>
    <td>0</td>
    <td>0</td>
    <td>74.0</td>
  </tr>
  <tr>
    <td>BioRxivClusteringP2P</td>
    <td>p2p</td>
    <td>1</td>
    <td>0</td>
    <td>0</td>
    <td>75,000</td>
    <td>0</td>
    <td>0</td>
    <td>1,666.2</td>
  </tr>
  <tr>
    <td>BioRxivClusteringS2S</td>
    <td>s2s</td>
    <td>1</td>
    <td>0</td>
    <td>0</td>
    <td>75,000</td>
    <td>0</td>
    <td>0</td>
    <td>101.6</td>
  </tr>
  <tr>
    <td>MedRxivClusteringP2P</td>
    <td>p2p</td>
    <td>1</td>
    <td>0</td>
    <td>0</td>
    <td>37,500</td>
    <td>0</td>
    <td>0</td>
    <td>1,981.2</td>
  </tr>
  <tr>
    <td>MedRxivClusteringS2S</td>
    <td>s2s</td>
    <td>1</td>
    <td>0</td>
    <td>0</td>
    <td>37,500</td>
    <td>0</td>
    <td>0</td>
    <td>114.7</td>
  </tr>
  <tr>
    <td>RedditClustering</td>
    <td>s2s</td>
    <td>1</td>
    <td>0</td>
    <td>424,664</td>
    <td>420,464</td>
    <td>64.7</td>
    <td>64.7</td>
    <td>64.7</td>
  </tr>
  <tr>
    <td>RedditClusteringP2P</td>
    <td>p2p</td>
    <td>1</td>
    <td>0</td>
    <td>0</td>
    <td>459,399</td>
    <td>0</td>
    <td>0</td>
    <td>727.7</td>
  </tr>
  <tr>
    <td>StackExchangeClustering</td>
    <td>s2s</td>
    <td>1</td>
    <td>0</td>
    <td>417,060</td>
    <td>373,850</td>
    <td>56.8</td>
    <td>57.0</td>
    <td>57.0</td>
  </tr>
  <tr>
    <td>StackExchangeClusteringP2P</td>
    <td>p2p</td>
    <td>1</td>
    <td>0</td>
    <td>0</td>
    <td>75,000</td>
    <td>0</td>
    <td>0</td>
    <td>1,090.7</td>
  </tr>
  <tr>
    <td>TwentyNewsgroupsClustering</td>
    <td>s2s</td>
    <td>1</td>
    <td>0</td>
    <td>0</td>
    <td>59,545</td>
    <td>0</td>
    <td>0</td>
    <td>32.0</td>
  </tr>
  <tr>
    <td rowspan="4">LongEmb Retrival</td>
    <td>LEMBNarrativeQARetrieval</td>
    <td>s2p</td>
    <td>1</td>
    <td>0</td>
    <td>0</td>
    <td>10,804</td>
    <td>0</td>
    <td>0</td>
    <td>326,753.5</td>
  </tr>
  <tr>
    <td>LEMBQMSumRetrieval</td>
    <td>s2p</td>
    <td>1</td>
    <td>0</td>
    <td>0</td>
    <td>1,724</td>
    <td>0</td>
    <td>0</td>
    <td>53,335.8</td>
  </tr>
  <tr>
    <td>LEMBSummScreenFDRetrieval</td>
    <td>s2p</td>
    <td>1</td>
    <td>0</td>
    <td>672</td>
    <td>0</td>
    <td>0</td>
    <td>30,854.3</td>
    <td>0</td>
  </tr>
  <tr>
    <td>LEMBWikiQARetrieval</td>
    <td>s2p</td>
    <td>1</td>
    <td>0</td>
    <td>0</td>
    <td>500</td>
    <td>0</td>
    <td>0</td>
    <td>37,445.6</td>
  </tr>
  <tr>
    <td rowspan="4">Reranking</td>
    <td>AskUbuntuDupQuestions</td>
    <td>s2s</td>
    <td>1</td>
    <td>0</td>
    <td>0</td>
    <td>2,255</td>
    <td>0</td>
    <td>0</td>
    <td>52.5</td>
  </tr>
  <tr>
    <td>MindSmallRanking</td>
    <td>s2s</td>
    <td>1</td>
    <td>231,530</td>
    <td>0</td>
    <td>107,968</td>
    <td>69.0</td>
    <td>0</td>
    <td>70.9</td>
  </tr>
  <tr>
    <td>SciDocsRR</td>
    <td>s2s</td>
    <td>1</td>
    <td>0</td>
    <td>19,594</td>
    <td>19,599</td>
    <td>0</td>
    <td>69.4</td>
    <td>69.0</td>
  </tr>
  <tr>
    <td>StackOverflowDupQuestions</td>
    <td>s2s</td>
    <td>1</td>
    <td>23,018</td>
    <td>3,467</td>
    <td>3,467</td>
    <td>49.6</td>
    <td>49.8</td>
    <td>49.8</td>
  </tr>
  <tr>
    <td rowspan="8">STS</td>
    <td>BIOSSES</td>
    <td>s2s</td>
    <td>1</td>
    <td>200</td>
    <td>200</td>
    <td>200</td>
    <td>156.6</td>
    <td>156.6</td>
    <td>156.6</td>
  </tr>
  <tr>
    <td>SICK-R</td>
    <td>s2s</td>
    <td>1</td>
    <td>19,854</td>
    <td>19,854</td>
    <td>19,854</td>
    <td>46.1</td>
    <td>46.1</td>
    <td>46.1</td>
  </tr>
  <tr>
    <td>STS12</td>
    <td>s2s</td>
    <td>1</td>
    <td>4,468</td>
    <td>0</td>
    <td>6,216</td>
    <td>100.7</td>
    <td>0</td>
    <td>64.7</td>
  </tr>
  <tr>
    <td>STS13</td>
    <td>s2s</td>
    <td>1</td>
    <td>0</td>
    <td>0</td>
    <td>3,000</td>
    <td>0</td>
    <td>0</td>
    <td>54.0</td>
  </tr>
  <tr>
    <td>STS14</td>
    <td>s2s</td>
    <td>1</td>
    <td>0</td>
    <td>0</td>
    <td>7,500</td>
    <td>0</td>
    <td>0</td>
    <td>54.3</td>
  </tr>
  <tr>
    <td>STS15</td>
    <td>s2s</td>
    <td>1</td>
    <td>0</td>
    <td>0</td>
    <td>6,000</td>
    <td>0</td>
    <td>0</td>
    <td>57.7</td>
  </tr>
  <tr>
    <td>STS16</td>
    <td>s2s</td>
    <td>1</td>
    <td>0</td>
    <td>0</td>
    <td>2,372</td>
    <td>0</td>
    <td>0</td>
    <td>65.3</td>
  </tr>
  <tr>
    <td>STS17</td>
    <td>s2s</td>
    <td>11</td>
    <td>0</td>
    <td>0</td>
    <td>500</td>
    <td>0</td>
    <td>0</td>
    <td>43.3</td>
  </tr>
  <tr>
    <td>STS22</td>
    <td>p2p</td>
    <td>18</td>
    <td>0</td>
    <td>0</td>
    <td>8,060</td>
    <td>0</td>
    <td>0</td>
    <td>1,992.8</td>
  </tr>
  <tr>
    <td>STSBenchmark</td>
    <td>s2s</td>
    <td>1</td>
    <td>11,498</td>
    <td>3,000</td>
    <td>2,758</td>
    <td>57.6</td>
    <td>64.0</td>
    <td>53.6</td>
  </tr>
  <tr>
    <td>Summarization</td>
    <td>SummEval</td>
    <td>p2p</td>
    <td>1</td>
    <td>0</td>
    <td>0</td>
    <td>2,800</td>
    <td>0</td>
    <td>0</td>
    <td>359.8</td>
  </tr>
</table>

Table 8: Statistics of the experimental datasets used in the work.

or negative.

MassiveIntent (FitzGerald et al., 2022) is a multilingual dataset featuring a diverse array of utterances from Amazon Alexa, each labeled with one of 60 different intents across 51 languages.

MassiveScenario (FitzGerald et al., 2022) dataset comprises a diverse collection of Amazon Alexa user utterances, each labeled with one of 60 thematic intents, and supports 51 languages.

ToxicConversations¹, sourced from a Kaggle competition, comprises comments from the Civil Comments platform, complete with annotations indicating whether each comment is toxic.

¹ToxicConversations

TweetSentimentExtraction², a dataset from a Kaggle competition, focuses on classifying tweets into three categories: neutral, positive, and negative sentiments.

F.2 Clustering

Clustering aims at grouping a given set of sentences or texts into meaningful clusters by training a mini-batch k-means model on the text embeddings. The model is scored using the v-measure (Rosenberg and Hirschberg, 2007). Since the V-measure does not depend on the cluster labels, the arrangement of the labels will not affect the score.

ArxivClusteringS2S, ArxivClusteringP2P,
<table>
  <tr>
    <th>Model Name</th>
    <th>Publicly Available Link</th>
  </tr>
  <tr>
    <td>ANCE</td>
    <td>https://huggingface.co/sentence-transformers/msmarco-roberta-base-ance-firstp</td>
  </tr>
  <tr>
    <td>GTR</td>
    <td>https://huggingface.co/sentence-transformers/gtr-t5-base</td>
  </tr>
  <tr>
    <td>GIST</td>
    <td>https://huggingface.co/avsolatorio/GIST-small-Embedding-v0</td>
  </tr>
  <tr>
    <td>BGE</td>
    <td>https://huggingface.co/BAAI/bge-base-en-v1.5</td>
  </tr>
  <tr>
    <td>E5</td>
    <td>https://huggingface.co/dzwu/e5-base-4k</td>
  </tr>
</table>

Table 9: Embedding models used in the experiments.

BiorxivClusteringS2S, BiorxivClusteringP2P, MedrxivClusteringP2P, MedrxivClusteringS2S (Muennighoff et al., 2023). These datasets are tailored for MTEB, utilizing titles or a combination of titles and abstracts from arXiv, bioRxiv, and medRxiv, with clustering labels derived from human-assigned categories, emphasizing both main and secondary classification levels.

RedditClustering (Geigle et al., 2021), a dataset that consists of titles from 199 subreddits, is organized into 25 splits, each featuring 10 to 50 classes, with every class containing between 100 and 1,000 sentences.

RedditClusteringP2P (Muennighoff et al., 2023), developed for the MTEB, consists of Reddit posts combined with their titles, organized into ten splits featuring 10 and 100 clusters each, with a total of 1,000 to 100,000 posts, aimed at clustering based on subreddit affiliation.

StackExchangeClustering (Geigle et al., 2021), a dataset consisting of titles from 121 Stack Exchange communities, is organized into 25 subsets, each containing 10 to 50 categories, with 100 to 1,000 sentences per category.

StackExchangeClusteringP2P (Muennighoff et al., 2023), designed for MTEB, comprises 10 splits of posts from StackExchange, each containing 5,000 to 10,000 entries, clustered by subreddit based on the combined content of titles and posts.

TwentyNewsgroupsClustering 3 consists of article titles from 20 different newsgroups, designed for clustering tasks, and includes 10 splits with each split featuring between 1,000 and 10,000 titles across the 20 categories.

F.3 Reranking

Reranking involves inputting a query along with a series of relevant and irrelevant reference texts, then sorting the results based on their relevance to the query. The provided model embeds the reference texts, which are compared to the query using cosine similarity. Each query is scored, and the average score across all queries is used to generate the final ranking. The evaluation metrics are MRR@k and MAP, with MAP serving as the primary metric.

AskUbuntuDupQuestions 4 dataset comprises questions sourced from AskUbuntu, accompanied by manually annotated labels that indicate whether pairs of questions are similar or dissimilar.

MindSmall (Wu et al., 2020) dataset is a comprehensive English resource designed for research in news recommendation, focusing on ranking news articles based on the title of a currently read article to suggest related content.

SciDocsRR (Cohan et al., 2020a) is a dataset designed for ranking related scientific papers using their titles as the primary basis for assessment.

StackOverflowDupQuestions (Liu et al., 2018) dataset focuses on identifying whether questions tagged with Java, JavaScript, and Python on Stack Overflow are duplicates of existing queries.

F.4 Retrieval

In the retrieval task, each dataset consists of a corpus, queries, and a mapping of each query to relevant documents. The task goal is to find these relevant documents based on a given query. When evaluating, we first use the provided model to embed queries and corpus documents and then calculate the cosine similarity to obtain relevance scores and rank the corpus documents for each query based on these scores. The evaluation metrics consist of nDCG@k, MRR@k, MAP@k, precision@k, and recall@k, with nDCG@10 as the primary metric.

F.4.1 Beir Retrival

NFCorpus (Boteva et al., 2016) is a dataset that includes natural language queries sourced from NutritionFacts, paired with annotated medical documents from PubMed, utilizing the original splits

3https://scikit-learn.org/0.19/datasets/twenty_newsgroups.html
4https://github.com/taolei87/askubuntu
from various types of content from NF, such as videos, blogs, and Q&A posts.

SciFact (Wadden et al., 2020) dataset evaluates scientific claims by matching them with evidence sourced from research literature, specifically utilizing a set of 300 test queries and the complete document collection from the original dataset.

SCIDOCS (Cohan et al., 2020b) is a benchmark dataset for evaluating scientific document embeddings, featuring seven tasks such as citation prediction, document classification, and recommendation.

FiQA2018 (Maia et al., 2018) is a financial question answering dataset built by crawling StackExchange posts under the Investment topic from 2009 to 2017, containing a knowledge base of 57,640 answer posts and 17,110 training question-answer pairs, with 531 testing question-answer pairs.

Touche2020 (Bondarenko et al., 2020) is a benchmark for argument retrieval, designed to support research in finding and evaluating arguments on various topics.

F.4.2 LongEmbed Retrival
LongEmbed (Zhu et al., 2024) includes 4 real-world retrieval tasks curated from long-form QA and summarization. The document in LongEmbed is much longer compared to BEIR. Thus, it can effectively evaluate the capability of the embedding model on long texts.

LEMBNarrativeQARetrieval (Kočiský et al., 2018) is a question-answering dataset featuring lengthy narratives, averaging 50,474 words, that challenge models to comprehend and extract information about characters and events dispersed throughout the stories.

LEMBQMSumRetrieval (Zhong et al., 2021) dataset focuses on generating summaries of meetings based on specific queries, necessitating the extraction and synthesis of relevant information from various segments of the conversation that cover multiple topics and participants.

LEMBSummScreenFDRetrieval (Chen et al., 2021) dataset consists of pairs of transcripts from TV series and their corresponding human-crafted summaries, requiring the integration of dispersed plot elements into concise narrative descriptions.

LEMBWikimQARetrieval (Ho et al., 2020) dataset is a complex question-answering resource that includes questions requiring up to five reasoning steps, designed using specific templates to encourage deep understanding rather than simple retrieval of information.

F.5 Semantic Textual Similarity (STS)
The task goal is to determine the similarity between a pair of sentences, where continuous scores serve as labels, with higher values indicating greater similarity. The provided model embeds the sentences, and their similarity is calculated using cosine similarity. The primary evaluation metric is the Spearman correlation (Reimers et al., 2016).

STS12, STS13, STS14, STS15, STS16, STS17, STS22, STSBenchmark (Agirre et al., 2012, 2013; Bandhakavi et al., 2014; Biçiçi, 2015; Nakov et al., 2016) 5 6 7 are collections of sentence pairs designed to evaluate semantic textual similarity, with the former set focused on monolingual English pairs and the latter two incorporating cross-lingual comparisons across multiple languages.

BIOSSES (Soğancıoğlu et al., 2017) comprises 100 pairs of sentences specifically focused on the biomedical domain.

SICK-R (Dadas et al., 2020), which stands for Sentences Involving Compositional Knowledge, comprises 100,000 diverse sentence pairs that exhibit rich lexical, syntactic, and semantic characteristics.

F.6 Summarization
The input consists of a set of summaries written by humans and machines. The goal is to score the machine-generated summaries. Use the provided model to embed the summaries. Calculate the distance between each machine summary and all human summary embeddings. Retain similar scores to the model score for each individual machine-generated summary. Calculate the Spearman correlation based on cosine similarity (Reimers et al., 2016) as the main metric.

SummEval (Fabbri et al., 2021) consists of summaries produced by advanced summarization models trained on CNN and DailyMail articles.

G Embedding Models
Table 9 provides the models used in the experiments and their publicly available links. Below is a brief introduction to these models.

ANCE (Xiong et al., 2021) enhances dense retrieval by selecting challenging negative samples

5https://alt.qcri.org/semeval2017/task1/
6https://competitions.codalab.org/competitions/33835
7https://github.com/PhilipMay/stsb-multi-mt/
from the entire corpus and asynchronously updating the Approximate Nearest Neighbor (ANN) index with each training iteration, using a context window size of 512.

GTR (Ni et al., 2022) improves dual encoder performance for retrieval tasks by scaling up model size while keeping a fixed bottleneck embedding, leading to significant improvements in out-of-domain generalization, all within a context window size of 512.

GIST (Solatorio, 2024) consistently improves performance across different model sizes by leveraging the strengths of large, resource-intensive models to enhance smaller ones, making advanced AI technologies more accessible and cost-effective, all within a context window size of 512.

BGE (Xiao et al., 2023) offers a range of well-trained embedding models based on a BERT-like architecture, enabling users to balance performance and efficiency for various applications while also allowing easy fine-tuning. In our experiments, we use the gte-base-en-v1.5 model, which operates with a context window size of 512.

E5 (Zhu et al., 2024) is a long-context embedding model fine-tuned to support 4k token inputs while maintaining the original performance for shorter contexts, designed to advance research in long-context embedding technologies. It uses a context window size of 4k.

Reviewer paper summaries: This paper identifies the length collapse phenomenon in PLM-based embedding models and introduces an interesting perspective through theoretical analysis in the spectral domain. Specifically, it finds out that as sequence length increases, the self-attention mechanism exhibits stronger low-pass filtering effects, which in turn leads to anisotropic embeddings. Building on this theoretical insight, the authors propose the TempScale method to mitigate length collapse and further explore its impact on various downstream text embedding tasks. The study is well-supported by both theoretical foundations and extensive empirical evaluations.
Summary of strengths: 1. The paper provides a theoretical explanation for the length collapse phenomenon and offers a well-grounded proof.
2. It comprehensively discusses how this phenomenon occurs and how temperature scaling affects text embedding performance in various downstream tasks.
3. The findings are supported by extensive experiments, which validate the paper's claims.
Summary of weaknesses: 1. Although the identified phenomenon offers a valuable perspective on why embeddings perform worse on long texts, the proposed temperature scaling method achieve limited improvements on downstream tasks.
2. The study is based on PLM-based models, without considering recent embedding models, particularly LLM-based embeddings with decoder architectures. Due to differences in the attention mechanism, these models may encounter different phenomena.
Comments, suggestions and typos: Since the study focuses on the length collapse phenomenon, the embedding models used in the experiments should include more large-window models, as this would make the findings more relevant and insightful.
Overall assessment scores: 3