Enhancing Lexicon-Based Text Embeddings with Large Language Models
Authors: Yibin Lei, Tao Shen, Yu Cao, Andrew Yates
Accepted at: acl25

Enhancing Lexicon-Based Text Embeddings with Large Language Models

Yibin Lei1, Tao Shen2, Yu Cao3, Andrew Yates1,4
1University of Amsterdam 2University of Technology Sydney
3Tencent IEG 4Johns Hopkins University, HLTCOE
y.lei@uva.nl, tao.shen@uts.edu.au
rainyucao@tencent.com, andrew.yates@jhu.edu

Abstract

Recent large language models (LLMs) have demonstrated exceptional performance on general-purpose text embedding tasks. While dense embeddings have dominated related research, we introduce the first lexicon-based embeddings (LENS) leveraging LLMs that achieve competitive performance on these tasks. LENS consolidates the vocabulary space through token embedding clustering to handle the issue of token redundancy in LLM vocabularies. To further improve performance, we investigate bidirectional attention and various pooling strategies. Specifically, LENS simplifies lexical matching with redundant vocabularies by assigning each dimension to a specific token cluster, where semantically similar tokens are grouped together. Extensive experiments demonstrate that LENS outperforms dense embeddings on the Massive Text Embedding Benchmark (MTEB), delivering compact representations with dimensionality comparable to dense counterparts. Furthermore, LENS inherently supports efficient embedding dimension pruning without any specialized objectives like Matryoshka Representation Learning. Notably, combining LENS with dense embeddings achieves state-of-the-art performance on the retrieval subset of MTEB (i.e., BEIR).1

1 Introduction

Text embeddings are vector representations of text that power a wide range of applications, including retrieval, question answering, semantic textual similarity, and clustering. Recent advances in LLMs have shown that a single model can generate embeddings excelling across diverse tasks, highlighting their versatility (Wang et al., 2023; Li et al., 2024; BehnamGhader et al., 2024; Lei et al., 2024; Muennighoff et al., 2024; Meng et al., 2024; Lee et al., 2024a).

1Our code and models are available at https://github.com/Yibin-Lei/LENS.

![A diagram showing the redundancy and noise in LLM tokenizers, as well as the absence of bidirectional dependencies in causal LLMs, motivating LENS.](page_1016_370_388_312.png)

Figure 1: The redundancy and noise in LLM tokenizers, as well as the absence of bidirectional dependencies in causal LLMs motivate LENS.

While dense embeddings that encode texts into low-dimensional, real-valued latent semantic spaces dominate recent research, lexicon-based embeddings (Formal et al., 2021a,b; Shen et al., 2023a; Lassance et al., 2024) offer distinctive advantages. These high-dimensional representations, where each dimension corresponds to a specific token of the vocabulary, align more closely with the pre-training objectives of language models due to their shared use of the vocabulary space and the language modeling head (Shen et al., 2023a). Recent studies have demonstrated that lexicon-based embeddings can surpass their dense counterparts, utilizing masked language models under specific control (Déjean et al., 2023). Additionally, lexicon-based embeddings can offer better transparency, providing clearer insights into the representations via the weight of each token. Moreover, the combination of dense and lexicon-based embeddings has also been shown to be promising in prior studies, as they effectively complement each other (Lin, 2021; Shen et al., 2023b; Gao et al., 2021).

Despite these benefits, lexicon-based embeddings remain underexplored beyond retrieval tasks.
To effectively apply them to other tasks, it is essential to address the challenges posed by LLMs, as shown in Fig. 1. The first issue is the inherent redundancy of LLM vocabularies. Since most modern tokenizers rely on subword tokenization (e.g., “education” is split into “edu” and “cation”), it fragments the entire vocabulary space (Soler et al., 2024). More importantly, semantically equivalent tokens can appear in multiple forms in the tokenizer (e.g., “what”, “What”, “ what” and “review”, “reviews”), introducing inconsistencies and difficulties in lexicon matching (e.g., the choice of whether “what” and/or “ what” appears in the lexical representation should be consistent along with their associated weights). Consequently, recent studies indicate that replacing the original tokenization of BM25 (Robertson et al., 1995) with the XLM-R tokenizer (Conneau et al., 2020) can lead to a significant performance drop due to the noisier vocabulary (Chen et al., 2024). The second challenge is that LLMs typically employ unidirectional attention during pre-training, where tokens only attend to preceding tokens. This limitation prevents each token from fully leveraging the surrounding context, which is crucial as lexicon-based embeddings are derived from the outputs of all tokens.

To address these challenges, we first explore the potential of LLMs generating embeddings where each dimension corresponds to a token cluster instead of the traditional single token, with each cluster grouping tokens that share similar meanings or stem from the same lexeme. To achieve this, we utilize a simple yet effective approach that directly clusters the token embeddings and leverages the centroids of these clusters as the new token embeddings for the language modeling head. As shown in Table 4, the resulting clusters naturally group tokens with similar meanings, forming more coherent and compact embeddings. These cluster-based embeddings can achieve the equivalent feature size as dense embeddings from the same LLM (e.g., 4,000d), which is much smaller than previous lexicon-based embeddings. Such a property i) facilitates the integration of LENS into existing dense frameworks like FAISS, reducing the need for sparsity constraints, and also ii) eliminates computational overhead in tasks such as clustering and classification, where inverted indices are not a natural fit.

Given recent studies highlighting the significant impact of attention mechanisms and pooling strategies on dense embeddings (Li et al., 2024; Muenighoff et al., 2024; BehnamGhader et al., 2024; Lee et al., 2024a), we incorporate variants of these two factors in our framework to examine how they affect lexicon-based embeddings. Contrary to prior findings (Li et al., 2024), which suggest that preserving the original architecture of LLMs typically yields optimal performance for dense embeddings, our results indicate that bidirectional attention is critical for achieving superior performance with lexicon-based embeddings.

Built on these techniques, we introduce LENS, a framework designed to generate low-dimensional lexicon-based embeddings that achieve impressive results across a variety of tasks. Specifically, our experiments demonstrate that LENS outperforms dense embeddings on the Massive Text Embedding Benchmark (MTEB) (Muenighoff et al., 2023), achieving state-of-the-art (SOTA) zero-shot performance among models trained exclusively on public data, as of December 1, 2024. These results demonstrate that sparse lexicon-based embeddings can perform competitively with dense embeddings across a diverse range of tasks. Qualitative examples also illustrate that LENS produces grounded and meaningful representations. Our analysis also shows that LENS, even when using 2000 clusters, still outperforms embeddings that leverage the original vocabulary space. Efficiency can be further improved through Top-K pruning: by retaining only the top 256 of 4000 dimensions at inference time, LENS maintains high performance with minimal degradation. This sparsification requires no specialized training (e.g., Matryoshka Representation Learning (Kusupati et al., 2022)) and arises naturally from its lexicon-based design. Finally, combining LENS with dense embeddings achieves SOTA performance on the retrieval subset of MTEB (specifically, BEIR).

2 Related Work

Lexicon-Based Embeddings. Lexicon-based embeddings assign each dimension of the embedding vector to a specific token in the vocabulary. With advances in masked language models, recent studies have demonstrated that lexicon-based embeddings (Mallia et al., 2021; Lin and Ma, 2021; Zhuang and Zuccon, 2021; Formal et al., 2021a,b; Shen et al., 2023a; Nguyen et al., 2023b; Lassance et al., 2024; Nguyen et al., 2023a, 2024; Lupart et al., 2024) can deliver strong performance. Among these approaches, SPLADE (Formal et al.,
2021b,a; Lassance et al., 2024) stands out as one of the most effective methods, even outperforming comparable dense embeddings (Déjean et al., 2023). Moreover, lexicon-based embeddings have been shown to complement dense embeddings, with their combination yielding substantial performance improvements (Chen et al., 2024; Shen et al., 2023b; Lin, 2021; Gao et al., 2021). Despite these advances, research on lexicon-based embeddings has largely focused on retrieval tasks, leaving other applications relatively such as clustering and classification underexplored.

LLM-Based Embeddings. As decoder-only LLMs continue to advance, recent work has investigated their potential for generating dense text embeddings capable of performing well across different tasks. To align LLMs with text embedding tasks, LLM2Vec (BehnamGhader et al., 2024) enables bi-directional attention with masked next-token prediction training and unsupervised contrastive learning, while LLaRA (Li et al., 2023a) leverages an auto-encoding objective to enhance embedding quality. Recent efforts, such as E5-Mistral (Wang et al., 2023) and Gecko (Lee et al., 2024b), focus on improving embedding models by using LLMs to generate diverse training data. Additionally, GRIT (Muennighoff et al., 2024) explores the combination of contrastive learning and language modeling objectives to train a single LLM that performs well on both embedding and generation tasks. Meanwhile, studies (Muennighoff et al., 2024; Lee et al., 2024a; BehnamGhader et al., 2024; Li et al., 2024) highlight the significant influence of architectural choices on embedding model performance, with findings (Li et al., 2024) indicating that retaining the original unidirectional attention often yields the best results.

Research on leveraging LLMs for lexicon-based embeddings remains limited. PromptReps (Zhuang et al., 2024) uses prompt engineering to generate lexicon-based embeddings from LLMs, but performs substantially worse than dense counterparts and approaches that involve fine-tuning like Mistral-SPLADE (Doshi et al., 2024). Furthermore, these efforts are limited to retrieval tasks. We explore how general-purpose lexicon-based embeddings can be produced.

3 Methodology

In this section, we first introduce preliminaries for a better understanding of the design of our framework, then formally describe the details of LENS.

3.1 Preliminaries

3.1.1 Lexicon-Based Embeddings Using Masked Language Models

SPLADE (Formal et al., 2021b,a; Lassance et al., 2024) is a representative method that utilizes Masked Language Models (MLMs) and regards the logits from the masked language modeling head as lexicon-based embeddings, leveraging the bidirectional attention. The MLM produces a sequence of logits \( L = (l_1, l_2, \ldots, l_n), l_i \in \mathbb{R}^{|V|} \) given the input sequence, where \( |V| \) is the vocabulary size. Each logit value \( l_{ij} \) represents the likelihood of the vocabulary token \( j \) being relevant to the position \( i \). Specifically, these scores are produced by the language modeling head, which maps the output hidden states to the vocabulary space using the token embedding matrix.

To obtain the lexicon-based embeddings, SPLADE first applies a log-saturation transformation to the logits to scale the weight and enforce it as non-negative,

\[
w_{ij} = \log \left( 1 + ReLU(l_{ij}) \right).
\]

Then it performs max-pooling across logits of all tokens to derive the final weight for each vocabulary token,

\[
w_j = \max_{i \in n} w_{ij}.
\]

Despite its proven effectiveness, former research on lexicon-based embeddings using MLMs primarily focused on small-scale models, leaving the performance of larger models mostly unexplored.

3.1.2 Lexicon-Based Embeddings Using Causal Language Models

Motivated by the growing capability of larger-scaled models, recent works have begun to use causal language models with significantly more parameters, such as LLaMA (Touvron et al., 2023) and Mistral (Jiang et al., 2023), to derive lexicon-based embeddings. Two notable methods are PromptReps (Zhuang et al., 2024) and Mistral-SPLADE (Doshi et al., 2024), which employ prompts to alleviate the limitations brought by the unidirectional attention.

PromptReps enables LLMs to generate both dense and lexicon-based embeddings through carefully designed prompts such as "This sentence [INPUT] means in one word:". Dense embeddings
are derived from the hidden states of the final token ", and lexicon-based ones are the logits for the next token prediction. Nevertheless, such a method relying solely on prompt causes a substantial performance drop of lexicon-based embeddings compared to their dense counterparts, e.g., MRR@10 of 34.15 vs. 41.86 on the MS MARCO dataset.

Mistral-SPLADE adapts SPLADE to large causal models like Mistral by using echo prompting (Springer et al., 2024). It enables full-context visibility of each token by duplicating the input sequence and regards the representations of the second occurrence as the output. Despite getting advancements on the BEIR benchmark, it still lags behind dense embeddings like ES-Mistral (Wang et al., 2023) and LLM2Vec (BehnamGhader et al., 2024), demonstrating that lexicon-based embeddings using large models cannot solely rely on prompting.

Hence, LENS systematically investigate the architecture of LLMs, including attention mechanisms and pooling methods, rather than exterior prompting. We investigate lexicon-based embeddings from LLMs, not only on retrieval tasks that have been widely examined before, but also on clustering and classification tasks which remain unexplored.

3.1.3 Tokenization in LLMs
LLM tokenizers, though designed to cover all possible text forms for the language modeling objective, may hinder the effectiveness of lexicon-based embedding. i) High redundancy can be introduced under the same lexeme and further affect the token matching. E.g., “What”, “what”, and “ what” can be regarded as distinct tokens due to differences in case or whitespace, even though they represent the same word, and may have different weights in the embedding. ii) Subword fragmentation (Soler et al., 2024) split a common word into pieces like “education” into “edu” and “cation”, posing additional matching complexity. iii) Tokenizers trained on large corpora often include rare tokens, which inflate the vocabulary size and make the embedding larger and slower to match.

Therefore, instead of directly using the original language modeling head, we simply cluster original tokens to form clusters and use their centroid embeddings to replace the original token embeddings of the language modeling head. This approach reduces the redundancy by merging related tokens and decreases the size of embeddings by using a smaller clustered vocabulary.

3.2 Framework of LENS
After discussing the background, we introduce the framework of our method, as shown in Fig. 2.

![The model framework of LENS.](page_1012_172_393_312.png)

Figure 2: The model framework of LENS.

3.2.1 Architecture Design
Language Modeling Head. Motivated by the redundancy and noise in LLM tokenizer mentioned above, LENS assigns weights to groups of tokens with similar meanings, whose effectiveness has been verified in Zhang et al. (2024). Specifically, we apply K-means clustering (Hartigan and Wong, 1979) to the token embeddings from the language modeling head, where \( k \) is our desired lexicon-based embedding size. Then the original token embeddings in the LM head are replaced by the cluster centroids, while the input token embeddings remain unchanged. This substitution reduces the dimensionality of the lexicon-based embeddings, as the logits now represent scores over clusters rather than the large original vocabulary. See Table 4 and Appendix A.1 for detailed cluster results.

Attention Mechanism. Given the former illustration of the limitations of unidirectional attention in typical causal LLMs, we emphasize it restricts the visibility of each token to the entire context. Hence, unlike previous works that rely on non-fundamental solutions like prompt engineering, we address this issue by directly modifying attention to be bidirectional during fine-tuning, which makes prompt design easier and inference more efficient.

3.2.2 Representation Generation
Following Wang et al. (2023) and Li et al. (2024), given a raw query-passage pair \((q, p)\) for a specific embedding task, we first construct the instructed
query input text as

\[
q_{ins} = \langle \text{Instruct}\rangle \{ \text{task\_definition}\} \langle \text{query}\rangle \{q\}.
\]

Here task\_definition refers to the definition of the specific embedding task, guiding the model to adapt towards that task. On the other hand, the input of the passage part is solely the original text. Following Wang et al. (2023) and Li et al. (2024), a [EOS] token is also appended to the end of the sequence.

We then feed this input into the modified LLM, and derive a series of logits vectors \( L = (l_1, l_2, \ldots, l_n), l_i \in \mathbb{R}^k \), where \( n \) is the sequence length and \( k \) is our clustering size. To obtain the final embeddings, following Formal et al. (2021a), log-saturation and max-pooling are applied to \( L \) along the sequence dimension as in Eq. 1. and 2.

We only employ tokens corresponding to the original query \( q \) to derive the output of the query, avoiding the noise brought by task\_definition tokens, inspired by BehnamGhader et al. (2024). Moreover, considering the autoregressive nature of LLMs, where each logit is used for the prediction of the subsequent position, we shift the logits during pooling. Specifically, we regard the logit corresponding to the neighbor on the left of each token as its feature during computation.

3.2.3 Training

Recent research has explored various complex methods for training embedding models. For example, NV-Embed-v2 (Lee et al., 2024a) employs a two-stage training pipeline while also incorporating positive-aware hard-negative mining and synthetic data generation. In contrast, for simplicity and fair comparison, the training of LENS strictly adheres to the training procedure of BGE-en-ICL (Li et al., 2024), a SOTA LLM-based dense embedding model. It uses a single-stage training process and relies exclusively on publicly available data.

Given a processed input pair \((q_{ins}, p)\), we utilize the InfoNCE loss as our objective,

\[
\mathcal{L} = - \log \frac{\exp(\operatorname{sim}(q_{ins}, p)/\tau)}{\exp(\frac{\operatorname{sim}(q_{ins}, p)}{\tau}) + \sum_{j=1}^{N} \exp(\frac{\operatorname{sim}(q_{ins}, p_j^-)}{\tau})}
\]

Here \( p_j^- \) and \( N \) denote the negative passage and the number of negative passages, respectively, \( \operatorname{sim}() \) is the cosine similarity function, defined as \( \operatorname{sim}() = \cos(h_{q_{ins}}, h_p) \), where \( h_{q_{ins}} \in \mathbb{R}^k \) and \( h_p \in \mathbb{R}^k \) are the lexicon-based embeddings from the LLM for the instructed query and passage. The temperature \( \tau \) is set to 0.02 in our experiments.

4 Experiments

4.1 Setup

To ensure a fair comparison between dense embeddings and LENS, we strictly adhere to the training recipe of the SOTA dense model, BGE-en-ICL.

Model Setup. The Mistral-7B-v0.1 (Jiang et al., 2023) model is used as the backbone in LENS, in line with recent works such as BGE-en-ICL (Li et al., 2024), E5-Mistral (Wang et al., 2023), NV-Embed-v2 (Lee et al., 2024a), and LLM2Vec (BehnamGhader et al., 2024). To investigate the effect of different clustering sizes to consolidate the output token embeddings, we set \( k \) in KMeans clustering to 4,000 and 8,000 clusters, referred to as LENS-4000 and LENS-8000, respectively. LENS-4000 can output 4000-d embeddings, which is comparable to the 4096-d dense embeddings produced by the same backbone LLM.

Training Data. We directly utilize the publicly available training data provided by BGE-en-ICL. This dataset is a mixture of retrieval, reranking, clustering, classification, and semantic textual similarity (STS) tasks. Details about the training data can be found in Appendix A.2. We use the same set of task instructions as BGE-en-ICL, refer to Appendix A.3 for details.

Training Configurations. Following BGE-en-ICL, our model is trained for one epoch using LoRA (Hu et al., 2021), where the LoRA rank is 32 and the alpha is 64, and the learning rate is set to 1e-4. Each training sample is composed of 1 positive and 7 hard negatives. For retrieval tasks, we use a batch size of 512, whereas a batch size of 256 is used for the rest tasks. All data are drawn from the same dataset within the same batch. In retrieval tasks, we employ in-batch negatives and apply a KL-divergence loss to distill ranking scores from the BGE-reranker model. The maximum length for both the query and passage is set to 512. We deviate from BGE-en-ICL by omitting in-context learning samples during training and concentrate on zero-shot scenarios solely, in order to evaluate performance without extraneous signals.

Evaluations. We evaluate the performance of various embedding models using MTEB (Muenninghoff et al., 2023) and AIR-Bench (Zeng et al.,
<table>
  <tr>
    <th>Task # of datasets →</th>
    <th>#Dims</th>
    <th>Retr. 15</th>
    <th>Rerank. 4</th>
    <th>Clust. 11</th>
    <th>PairClass. 3</th>
    <th>Class. 12</th>
    <th>STS 10</th>
    <th>Summ. 1</th>
    <th>Avg. 56</th>
  </tr>
  <tr>
    <th colspan="10">Non-Fully Public Training Data</th>
  </tr>
  <tr>
    <td>E5-mistral-7b-instruct</td>
    <td>4096</td>
    <td>56.90</td>
    <td>60.21</td>
    <td>50.26</td>
    <td>88.34</td>
    <td>78.47</td>
    <td>84.66</td>
    <td>31.40</td>
    <td>66.63</td>
  </tr>
  <tr>
    <td>Linq-Embed-Mistral</td>
    <td>4096</td>
    <td>60.19</td>
    <td>60.29</td>
    <td>51.42</td>
    <td>88.35</td>
    <td>80.20</td>
    <td>84.97</td>
    <td>30.98</td>
    <td>68.17</td>
  </tr>
  <tr>
    <td>voyage-large-2-instruct</td>
    <td>1024</td>
    <td>58.28</td>
    <td>60.09</td>
    <td>53.35</td>
    <td>89.24</td>
    <td>81.49</td>
    <td>84.31</td>
    <td>30.84</td>
    <td>68.23</td>
  </tr>
  <tr>
    <td>stella_en_400M_v5</td>
    <td>8192</td>
    <td>58.97</td>
    <td>60.16</td>
    <td>56.70</td>
    <td>87.74</td>
    <td>86.67</td>
    <td>84.22</td>
    <td>31.66</td>
    <td>70.11</td>
  </tr>
  <tr>
    <td>gte-Qwen2-7B-instruct</td>
    <td>3584</td>
    <td>60.25</td>
    <td>61.42</td>
    <td>56.92</td>
    <td>85.79</td>
    <td>86.58</td>
    <td>83.04</td>
    <td>31.35</td>
    <td>70.24</td>
  </tr>
  <tr>
    <td>SFR-Embedding-2_R</td>
    <td>4096</td>
    <td>60.18</td>
    <td>60.14</td>
    <td>56.17</td>
    <td>88.07</td>
    <td>89.05</td>
    <td>81.26</td>
    <td>30.71</td>
    <td>70.31</td>
  </tr>
  <tr>
    <td>stella_en_1.5B_v5</td>
    <td>8192</td>
    <td>61.01</td>
    <td>61.21</td>
    <td>57.69</td>
    <td>88.07</td>
    <td>87.63</td>
    <td>84.51</td>
    <td>31.49</td>
    <td>71.19</td>
  </tr>
  <tr>
    <td>NV-Embed-v2</td>
    <td>4096</td>
    <td>62.65</td>
    <td>60.65</td>
    <td>58.46</td>
    <td>88.67</td>
    <td>90.37</td>
    <td>84.31</td>
    <td>30.70</td>
    <td>72.31</td>
  </tr>
  <tr>
    <th colspan="10">Fully Public Training Data</th>
  </tr>
  <tr>
    <td>LLM2Vec-Mistral-supervised</td>
    <td>4096</td>
    <td>55.99</td>
    <td>58.42</td>
    <td>45.54</td>
    <td><b>87.99</b></td>
    <td>76.63</td>
    <td>84.09</td>
    <td>29.96</td>
    <td>64.80</td>
  </tr>
  <tr>
    <td>GritLM-7B</td>
    <td>4096</td>
    <td>57.41</td>
    <td>60.49</td>
    <td>50.61</td>
    <td>87.16</td>
    <td>79.46</td>
    <td>83.35</td>
    <td>30.37</td>
    <td>66.76</td>
  </tr>
  <tr>
    <td>NV-Embed-v1</td>
    <td>4096</td>
    <td>59.36</td>
    <td>60.59</td>
    <td>52.80</td>
    <td>86.91</td>
    <td>87.35</td>
    <td>82.84</td>
    <td>31.20</td>
    <td>69.32</td>
  </tr>
  <tr>
    <td>bge-multilingual-gemma2</td>
    <td>3584</td>
    <td>59.24</td>
    <td>59.72</td>
    <td>54.65</td>
    <td>85.84</td>
    <td>88.08</td>
    <td>83.88</td>
    <td>31.20</td>
    <td>69.88</td>
  </tr>
  <tr>
    <td>BGE-en-ICL (zero-shot)</td>
    <td>4096</td>
    <td>61.67</td>
    <td>59.66</td>
    <td>57.51</td>
    <td>86.93</td>
    <td><b>88.62</b></td>
    <td>83.74</td>
    <td>30.75</td>
    <td>71.24</td>
  </tr>
  <tr>
    <td>LENS-4000 (<b>Ours</b>)</td>
    <td>4000</td>
    <td>60.76</td>
    <td>60.86</td>
    <td>57.92</td>
    <td>87.93</td>
    <td>88.13</td>
    <td>84.35</td>
    <td>31.56</td>
    <td>71.22</td>
  </tr>
  <tr>
    <td>LENS-8000 (<b>Ours</b>)</td>
    <td>8000</td>
    <td><b>61.86</b></td>
    <td><b>60.91</b></td>
    <td><b>58.02</b></td>
    <td>87.98</td>
    <td><b>88.43</b></td>
    <td><b>84.67</b></td>
    <td>29.54</td>
    <td><b>71.63</b></td>
  </tr>
</table>

Table 1: Top-performing models on the MTEB leaderboard as of December 1, 2024 compared to LENS. #Dims refers to the embedding dimensions. Abbreviations: Retr. = Retrieval; Rerank. = Reranking; Clust. = Clustering; PairClass. = Pair Classification; Class. = Classification; STS = Semantic Textual Similarity; Summ. = Summarization. The best and the second best results using public data are in bold and underlined font, respectively.

<table>
  <tr>
    <th>Domain # of datasets →</th>
    <th>#Dims</th>
    <th>wiki 1</th>
    <th>web 1</th>
    <th>news 1</th>
    <th>healthcare 1</th>
    <th>law 1</th>
    <th>finance 1</th>
    <th>arxiv 1</th>
    <th>msmarco 1</th>
    <th>Avg. 8</th>
  </tr>
  <tr>
    <td>E5-mistral-7b-instruct</td>
    <td>4096</td>
    <td>61.67</td>
    <td>44.41</td>
    <td>48.18</td>
    <td>56.32</td>
    <td>19.32</td>
    <td>54.79</td>
    <td>44.78</td>
    <td>59.03</td>
    <td>48.56</td>
  </tr>
  <tr>
    <td>Linq-Embed-Mistral</td>
    <td>4096</td>
    <td>61.04</td>
    <td>48.41</td>
    <td>49.44</td>
    <td><b>60.18</b></td>
    <td>20.34</td>
    <td>50.04</td>
    <td>47.56</td>
    <td>60.50</td>
    <td>49.69</td>
  </tr>
  <tr>
    <td>NV-Embed-v1</td>
    <td>4096</td>
    <td>62.84</td>
    <td>50.42</td>
    <td>51.46</td>
    <td>58.53</td>
    <td>20.65</td>
    <td>49.89</td>
    <td>46.10</td>
    <td>60.27</td>
    <td>50.02</td>
  </tr>
  <tr>
    <td>gte-Qwen2-7B-instruct</td>
    <td>3584</td>
    <td>63.46</td>
    <td>51.20</td>
    <td>54.07</td>
    <td>54.20</td>
    <td>22.31</td>
    <td><b>58.20</b></td>
    <td>40.27</td>
    <td>58.39</td>
    <td>50.26</td>
  </tr>
  <tr>
    <td>stella_en_1.5B_v5</td>
    <td>8192</td>
    <td>61.99</td>
    <td>50.88</td>
    <td>53.87</td>
    <td>58.81</td>
    <td>23.22</td>
    <td><b>57.26</b></td>
    <td>44.81</td>
    <td>61.38</td>
    <td>51.53</td>
  </tr>
  <tr>
    <td>SFR-Embedding-Mistral</td>
    <td>4096</td>
    <td>63.46</td>
    <td>51.27</td>
    <td>52.21</td>
    <td>58.76</td>
    <td>23.27</td>
    <td>56.94</td>
    <td>47.75</td>
    <td>58.99</td>
    <td>51.58</td>
  </tr>
  <tr>
    <td>NV-Embed-v2</td>
    <td>4096</td>
    <td>65.19</td>
    <td>52.58</td>
    <td>53.13</td>
    <td>59.56</td>
    <td>25.00</td>
    <td>53.04</td>
    <td><b>48.94</b></td>
    <td>60.80</td>
    <td>52.28</td>
  </tr>
  <tr>
    <td>BGE-en-ICL (zero-shot)</td>
    <td>4096</td>
    <td>64.61</td>
    <td><b>54.40</b></td>
    <td><b>55.11</b></td>
    <td>57.25</td>
    <td><b>25.10</b></td>
    <td>54.81</td>
    <td>48.46</td>
    <td><b>63.71</b></td>
    <td><b>52.93</b></td>
  </tr>
  <tr>
    <td>LENS-4000 (<b>Ours</b>)</td>
    <td>4000</td>
    <td>62.60</td>
    <td>52.06</td>
    <td>52.49</td>
    <td>57.23</td>
    <td>24.08</td>
    <td>48.87</td>
    <td>43.78</td>
    <td>61.17</td>
    <td>50.28</td>
  </tr>
  <tr>
    <td>LENS-8000 (<b>Ours</b>)</td>
    <td>8000</td>
    <td><b>65.50</b></td>
    <td><b>54.52</b></td>
    <td><b>55.16</b></td>
    <td>58.20</td>
    <td><b>25.62</b></td>
    <td>54.57</td>
    <td>45.45</td>
    <td><b>63.00</b></td>
    <td><b>52.75</b></td>
  </tr>
</table>

Table 2: QA performance on AIR-Bench 24.04 (English) across different models, where nDCG@10 is used as the metric. #Dims refers to the embedding dimensions. The best and the second best results across all models are in bold and underlined font, respectively.

2024). MTEB is a comprehensive text embedding benchmark encompassing seven task types across a total of 56 datasets. AIR-Bench, on the other hand, spans diverse domains for retrieval tasks, including law, healthcare, and books, having no overlap with MTEB. Notably, the ground truth for the test set in AIR-Bench is hidden, and we use the 24.04 version to assess the model’s out-of-domain capabilities.

We compare LENS to numerous baselines, including E5-mistral-7b-instruct (Wang et al., 2023), NV-Embed-v1/v2 (Lee et al., 2024a), gte-Qwen2-7B-instruct (Li et al., 2023b), LLM2Vec (BehnaghamGhaderi et al., 2024), SFR-Embedding-2_R (Meng et al., 2024), GritLM-7B (Muenninghoff et al., 2024), and BGE-en-ICL (Li et al., 2024). The results of PromptReps and Mistral-Splade are excluded, as they are designed specifically for retrieval tasks and their performance falls below of the weakest baseline, namely LLM2Vec-Mistral-supervised. Some of the baselines use private data during training or involve in-context learning. To ensure a fair comparison, we focus on zero-shot scenarios where no few-shot sample is included in the prompt, e.g., BGE-en-ICL.

4.2 Main results

MTEB. Table 1 demonstrates the results of a variety of models on MTEB. LENS-8000 achieves the highest average performance among all models trained on fully public data. Notably, LENS-8000 outperforms BGE-en-ICL, its dense embedding counterpart trained with the same data and hyperparameters, with consistently higher perfor-
<table>
  <tr>
    <th>Text</th>
    <th>Top-weighted clusters</th>
  </tr>
  <tr>
    <td>most dependable affordable cars (cars, Cars), (cheap, affordable), (reliable, reli), (depend, depends), (aff, afford)</td>
    <td></td>
  </tr>
  <tr>
    <td>fastest growing bonsai trees</td>
    <td>(faster, fastest), (grow, growing), (fast, Fast), (tree, trees), (quickly, rapid)</td>
  </tr>
  <tr>
    <td>causes of hypoxia in adults</td>
    <td>(adult, adults), (oxygen, oxy), (cause, caused), (hyp, yp), (ox, Ox)</td>
  </tr>
  <tr>
    <td>weather in lisbon april</td>
    <td>(Portug, Portuguese), (bon, Bon), (weather, rather), (Spring, spring), (AP, #AP)</td>
  </tr>
  <tr>
    <td>other hot flashes causes</td>
    <td>(hot, Hot), (cause, causes), (flash, Flash), (flush, #flush), (heat, Heat)</td>
  </tr>
</table>

Table 3: Qualitive examples of LENS-8000. For each example, the top-5 clusters with the largest weights in the embeddings are shown, with two tokens from each cluster included. # denotes a whitespace character.

<table>
  <tr>
    <th>Clusters</th>
    <th></th>
  </tr>
  <tr>
    <td>quickly, rapid, rapidly, swift</td>
    <td></td>
  </tr>
  <tr>
    <td>cannot, impossible, Unable, Cannot, Unable</td>
    <td></td>
  </tr>
  <tr>
    <td>shows, shown, showed, showing</td>
    <td></td>
  </tr>
  <tr>
    <td>review, Review, reviews, reviewed, Reviews</td>
    <td></td>
  </tr>
  <tr>
    <td>educ, education, Educ, Education, educational, Edu</td>
    <td></td>
  </tr>
</table>

Table 4: Cluster examples of LENS-8000. Each row presents tokens belonging to a single cluster. More cluster examples are provided in Appendix A.1.

mance in 6 out of 7 categories of tasks. LENS-4000 yields comparable performance to BGE-en-ICL and both produce representations of similar sizes, but our lexicon-based method can deliver better transparency. Furthermore, LENS-8000 ranks second among all models in overall average performance. The leading model, NV-Embed-v2, attains high performance through a significantly more complex training pipeline, which includes a two-stage training pipeline, positive-aware hard-negative mining, and synthetic data generation. By contrast, LENS uses fully public data and adopts a simpler training procedure following BGE-en-ICL.

AIR-Bench. We also evaluate LENS along with baselines on the QA tasks of AIR-Bench. As shown in Table 2, LENS-8000 outperforms the top-performing model on MTEB, NV-Embed-v2, demonstrating its promising generalization capabilities. Despite slightly lagging behind its dense counterpart BGE-en-ICL, LENS still remains competitive in several sub-tasks. However, LENS-4000 performs less competitively, potentially because a smaller number of clusters may result in over-generalized clusters and information loss.

4.3 Qualitative Examples

We present some clustering results in Table 4. These examples illustrate that: i) LENS groups semantically equivalent tokens (e.g., “rapid” and “quickly”, “cannot” and “impossible”); ii) it groups morphologically similar tokens (e.g., “shows” and “showed”); and iii) it groups uppercase/lowercase variants and whole-word/subword forms (e.g., “review” and “Review”, “Edu” and “education”). Such an observation highlights the effectiveness of clustering in reducing the redundancy and noise of the tokenizer.

In addition, qualitative examples from MS MARCO of LENS-8000 embeddings are given in Table 3. The top-5 clusters with the largest weights in the embeddings are presented for each sample, where two tokens from each cluster are included. Obviously, these clusters are highly semantically relevant to the input texts, which can be regarded as some keywords. There are also interesting findings that the embeddings show some deep understanding of the text such as “oxygen” in response to the input “causes of hypoxia in adults”, and some knowledge expansion capabilities like “Portuguese” and “spring” for the input “weather in lisbon april”. These qualitative samples demonstrate that lexicon-based embeddings from LLMs capture more than shallow token meanings.

5 Analysis

In this section, we conduct a detailed investigation of LENS design decisions. For the sake of computational resources, we reduce the training data of each dataset to 10% of its original size and limit both query and passage lengths to a maximum of 128 tokens. Besides, we use the same MTEB subset as Jiang et al. (2024) for faster evaluation, as it correlates well with the overall performance of MTEB (details in Appendix A.4).

5.1 Influence of the Number of Clusters

We investigate how the number of clusters \( k \) affects performance, as shown in Figure 3. The configuration with 32,000 clusters corresponds to retaining the original token embeddings without applying clustering. Reducing the number of clusters generally improves performance, with notable gains observed at 8,000 and 16,000 clusters. We also show the performance of models trained without the distillation loss, using only contrastive learning.
<table>
  <tr>
    <th rowspan="2">Task # of datasets →</th>
    <th colspan="1">Retr.</th>
    <th colspan="1">Rerank.</th>
    <th colspan="1">Clust.</th>
    <th colspan="1">PairClass.</th>
    <th colspan="1">Class.</th>
    <th colspan="1">STS</th>
    <th colspan="1">Summ.</th>
    <th colspan="1">Avg.</th>
  </tr>
  <tr>
    <th>1</th>
    <th>1</th>
    <th>1</th>
    <th>1</th>
    <th>1</th>
    <th>1</th>
    <th>1</th>
    <th>7</th>
  </tr>
  <tr>
    <th colspan="9">Unidirectional Attention</th>
  </tr>
  <tr>
    <td>Last-token pooling</td>
    <td>73.84</td>
    <td>65.19</td>
    <td>60.46</td>
    <td>96.69</td>
    <td>58.66</td>
    <td>89.26</td>
    <td>30.05</td>
    <td>67.73</td>
  </tr>
  <tr>
    <td>Sum-pooling</td>
    <td>72.46</td>
    <td>59.57</td>
    <td>50.55</td>
    <td>89.90</td>
    <td>54.64</td>
    <td>80.55</td>
    <td>29.70</td>
    <td>62.48</td>
  </tr>
  <tr>
    <td>Max-pooling</td>
    <td>75.18</td>
    <td>59.68</td>
    <td>50.93</td>
    <td>92.06</td>
    <td>57.58</td>
    <td>82.74</td>
    <td>30.89</td>
    <td>64.15</td>
  </tr>
  <tr>
    <th colspan="9">Bidirectional Attention</th>
  </tr>
  <tr>
    <td>Last-token pooling</td>
    <td>76.89</td>
    <td>64.21</td>
    <td>61.57</td>
    <td>96.62</td>
    <td>58.33</td>
    <td>88.72</td>
    <td>30.72</td>
    <td>68.15</td>
  </tr>
  <tr>
    <td>Sum-pooling</td>
    <td>75.65</td>
    <td>63.64</td>
    <td>61.77</td>
    <td>96.97</td>
    <td>60.05</td>
    <td>89.58</td>
    <td>30.98</td>
    <td>68.38</td>
  </tr>
  <tr>
    <td>Max-pooling</td>
    <td>76.19</td>
    <td>64.53</td>
    <td>63.05</td>
    <td>97.03</td>
    <td>62.30</td>
    <td>88.92</td>
    <td>31.49</td>
    <td>69.07</td>
  </tr>
</table>

Table 5: Influence of attention mechanisms and pooling methods.

![Line plot showing influence of number of clusters on subset average MTEB score, with and without distillation.](page_186_624_482_312.png)

Figure 3: Influence of the number of clusters. The configuration with 32,000 clusters retains the original token embeddings without clustering. w/o. distillation refers to training with only the contrastive loss, excluding the distillation loss.

Across all values of \( k \) except 32,000, distillation provides a consistent benefit. A configuration of 8,000 clusters strikes a good balance between effectiveness and efficiency (dimensionality). Consequently, we employ \( k = 8,000 \) in the subsequent experiments.

5.2 Influence of Model Architecture

We investigate the effects of the attention mechanism and pooling methods, as illustrated in Table 5. For attention, we examine both unidirectional and bidirectional attention. Regarding pooling strategies, we assess max-pooling, sum-pooling, and last-token pooling. The results highlight the critical role of bidirectional attention in achieving strong performance with lexicon-based embeddings, as evidenced by its superiority across all pooling methods. Among these pooling methods, max-pooling emerges as the most effective strategy. This finding partially explains the poor performance of lexicon-based embeddings from PromptReps, which relies on last-token pooling with unidirectional attention.

5.3 Top-K Pruning and Natural Sparsity

Table 6 shows the effect of Top-K pruning on LENS embeddings. Each embedding dimension in LENS corresponds to a distinct lexical cluster, resulting in disentangled representations that naturally support sparsification. We apply Top-K pruning at inference time by retaining only the \( K \) dimensions with the highest activation values in the embedding, setting the rest to zero. We use LENS-4000 in this analysis due to its relatively low dimensionality, which allows Top-K pruning to be particularly effective. This approach requires no retraining and produces compact, sparse representations. We compare LENS to Gecko (Lee et al., 2024b), which is trained with Matryoshka Representation Learning (Kusupati et al., 2022) to explicitly support multiple embedding sizes. In contrast, LENS is trained once and pruned post hoc.

<table>
  <tr>
    <th>Model</th>
    <th>#Active Dims</th>
    <th>BEIR Avg.</th>
  </tr>
  <tr>
    <td>BGE-en-ICL</td>
    <td>4096</td>
    <td>61.67</td>
  </tr>
  <tr>
    <td>Gecko</td>
    <td>768</td>
    <td>55.67</td>
  </tr>
  <tr>
    <td>Gecko</td>
    <td>256</td>
    <td>52.43</td>
  </tr>
  <tr>
    <td>LENS-4000</td>
    <td>/</td>
    <td>60.76</td>
  </tr>
  <tr>
    <td>LENS-4000</td>
    <td>768</td>
    <td>60.01</td>
  </tr>
  <tr>
    <td>LENS-4000</td>
    <td>512</td>
    <td>59.33</td>
  </tr>
  <tr>
    <td>LENS-4000</td>
    <td>256</td>
    <td>57.19</td>
  </tr>
</table>

Table 6: Top-K pruning results on BEIR. LENS-4000 is evaluated by retaining only the top-K activated dimensions at inference time, without retraining. When pruning from 768 to 256 dimensions, LENS exhibits a smaller performance drop than Gecko, despite Gecko is explicitly trained to support multiple embedding sizes.

5.4 Hybrid Lexicon-Dense Embeddings

Previous studies have demonstrated that lexicon-based embeddings and dense embeddings are complementary, and combining them can lead to significant performance improvements. In this section, we explore the effectiveness of combining LENS with BGE-en-ICL, both trained on the same data but representing different types of embeddings. To evaluate general-use cases, we concatenate the two embeddings into a single embedding, without applying any additional operations. We hypothesize that enhanced performance could be achieved by tuning the combination weights of the two embed-
<table>
  <tr>
    <th>Dataset</th>
    <th>ARG</th>
    <th>CLI</th>
    <th>CQA</th>
    <th>DBP</th>
    <th>FEV</th>
    <th>FIQ</th>
    <th>HOT</th>
    <th>MSM</th>
    <th>NFC</th>
    <th>NQ</th>
    <th>QUO</th>
    <th>SCD</th>
    <th>SCF</th>
    <th>TOU</th>
    <th>COV</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>BGE-en-ICL</td>
    <td>82.76</td>
    <td>45.35</td>
    <td>47.23</td>
    <td>50.42</td>
    <td>91.96</td>
    <td>58.77</td>
    <td>84.98</td>
    <td>46.72</td>
    <td>40.69</td>
    <td>73.85</td>
    <td>91.02</td>
    <td>25.25</td>
    <td>78.33</td>
    <td>29.67</td>
    <td>78.11</td>
    <td>61.67</td>
  </tr>
  <tr>
    <td>LENS-8000 (<b>Ours</b>)</td>
    <td>76.02</td>
    <td>45.77</td>
    <td>48.67</td>
    <td>49.75</td>
    <td>92.32</td>
    <td>61.57</td>
    <td>85.71</td>
    <td>47.24</td>
    <td>40.61</td>
    <td>74.64</td>
    <td>90.79</td>
    <td>28.54</td>
    <td>79.75</td>
    <td>23.74</td>
    <td>77.18</td>
    <td>61.86</td>
  </tr>
  <tr>
    <td>NV-Embed-v2</td>
    <td>70.07</td>
    <td>45.39</td>
    <td>50.24</td>
    <td>53.50</td>
    <td>93.75</td>
    <td>65.73</td>
    <td>85.48</td>
    <td>45.63</td>
    <td>45.17</td>
    <td>83.57</td>
    <td>89.04</td>
    <td>21.90</td>
    <td>80.13</td>
    <td>31.78</td>
    <td>88.44</td>
    <td>62.65</td>
  </tr>
  <tr>
    <td><b>LENS (Ours)</b> + BGE</td>
    <td><b>81.37</b></td>
    <td><b>47.14</b></td>
    <td><b>48.57</b></td>
    <td><b>51.79</b></td>
    <td><b>93.12</b></td>
    <td><b>62.00</b></td>
    <td><b>87.12</b></td>
    <td><b>47.66</b></td>
    <td><b>41.55</b></td>
    <td><b>75.81</b></td>
    <td><b>91.07</b></td>
    <td><b>28.41</b></td>
    <td><b>80.19</b></td>
    <td><b>30.51</b></td>
    <td><b>78.72</b></td>
    <td><b>63.00</b></td>
  </tr>
</table>

Table 7: Results in terms of nDCG@10 on the retrieval subset (i.e., BEIR) of MTEB. We use the first three letters of each dataset’s name as its abbreviation, except SCIDOCs (abbreviated as SCD) and SciFact (abbreviated as SCF). <b>Bolded values</b> indicate datasets where the combinations outperform both LENS-8000 and BGE-en-ICL individually. On 12 out of 15 datasets, combining LENS-8000 and BGE-en-ICL results in improved performance.

<table>
  <tr>
    <th>Text</th>
    <th>Problematic Top-10 Weighted Clusters</th>
  </tr>
  <tr>
    <td>how much does an average person make for tutoring (tuple, Tuple)</td>
    <td>(count, (count), (count), (SSL, ssl))</td>
  </tr>
  <tr>
    <td>what county is rossville, ga in</td>
    <td>(tom, (Tom, Tommy), (rott, rot))</td>
  </tr>
  <tr>
    <td>what causes tomato dry rot in tomatoes</td>
    <td>(har, #har, (anger, angers), (lang)</td>
  </tr>
  <tr>
    <td>what is a hangar</td>
    <td>(har, #har, (anger, angers), (lang)</td>
  </tr>
</table>

Table 8: Examples of failure cases from LENS-8000, where some of the top-10 weighted dimensions are associated with clusters that are lexically similar to the query terms but semantically unrelated. Each cluster is illustrated with up to two representative tokens.

dings.

The results are presented in Table 7. Combining LENS-8000 with BGE-en-ICL yields a substantial performance improvement, increasing from 61.67/61.86 to 63.00, which surpasses NV-Embed-v2 and achieves SOTA results on the retrieval subset of MTEB as of December 1, 2024. Furthermore, such an improvement is consistent, as evidenced by performance gains on 12 out of 15 datasets.

5.5 Failure Cases of LENS

The lexicon-grounded nature of LENS enables inspection of failure cases by examining the most influential token clusters for each input text. Table 8 shows several examples from LENS-8000 where the top-weighted dimensions are dominated by clusters that are lexically similar but semantically irrelevant.

Many of these issues stem from the subword tokenization used in LLMs, where words are broken into fragments that may overlap in form but not in meaning. For example, the text “how much does an average person make for tutoring” activates the cluster (“tuple”, “Tuple”), likely due to subword overlap with “tutoring” rather than any meaningful connection. Similarly, “what is a hangar” triggers clusters such as (“har”, “#har”) and (“anger”, “angers”), which are unrelated in meaning but happen to resemble parts of the input string.

To address these limitations, a promising direction is to shift from token-level to entity-level representations (Nguyen et al., 2024), associating embedding dimensions with semantically meaningful whole words or phrases rather than isolated sub-word fragments. Another complementary direction is to explicitly down-weight or drop high-frequency tokens that tend to appear in many clusters across the corpus, as these often reflect common morphemes or generic terms that contribute little to semantic discrimination but introduce noise into the learned dimensions.

6 Conclusion

In this work, we introduce LENS, a simple yet effective framework for generating lexicon-based text embeddings using LLMs. Our approach leverages token embedding clustering to address the redundancy challenges inherent in LLM tokenizers, while also enabling bidirectional attention to fully unlock the potential of LLMs. Extensive experiments demonstrate the promising effectiveness and generalization capabilities of LENS compared to SOTA dense embeddings. Qualitative examples illustrate that LENS produces embeddings that are grounded and demonstrate a deep understanding of the input. Further analyses show the superiority of fusing lexicon-based LENS and dense embeddings, which surpasses each individual model on the retrieval subset of MTEB (i.e., BEIR).

Limitations

We acknowledge the following limitations of our work. First, our training and evaluation are limited to English, leaving multilingual datasets, such as Miracl (Zhang et al., 2023), unexplored. This restricts the generalizability of our findings to non-English contexts. Second, we applied LENS exclusively to the widely used Mistral-7B model, leaving other models unexplored. Additionally, compared to previous lexicon-based models like SPLADE, utilizing LLMs as the backbone significantly increases computational costs.
Acknowledgements

This research was supported by the Hybrid Intelligence Center, a 10-year program funded by the Dutch Ministry of Education, Culture and Science through the Netherlands Organisation for Scientific Research, https://hybrid-intelligence-centre.nl, and project VI.Vidi.223.166 of the NWO Talent Programme which is (partly) financed by the Dutch Research Council (NWO).

References

C.J. Adams, Daniel Borkan, Jeffrey Sorensen, Lucas Dixon, Lucy Vasserman, and Nithum Thain. 2019. Jigsaw unintended bias in toxicity classification.

Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012. SemEval-2012 task 6: A pilot on semantic textual similarity. in* sem 2012: The first joint conference on lexical and computational semantics–volume 1: Proceedings of the main conference and the shared task, and volume 2: Proceedings of the sixth international workshop on semantic evaluation (semeval 2012). Association for Computational Linguistics.

Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2018. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268.

Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. 2024. Llm2vec: Large language models are secretly powerful text encoders. arXiv preprint arXiv:2404.05961.

Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulic. 2020. Efficient intent detection with dual sentence encoders. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 38–45.

Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. arXiv preprint arXiv:1708.00055.

Jianyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In Findings of the Association for Computational Linguistics: ACL 2024.

Xi Chen, Ali Zeynali, Chico Camargo, Fabian Flöck, Devin Gaffney, Przemyslaw Grabowicz, Scott Hale, David Jurgens, and Mattia Samory. 2022. SemEval-2022 task 8: Multilingual news article similarity. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 1094–1106.

Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S Weld. 2020. Specter: Document-level representation learning using citation-informed transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2270–2282.

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.

DataCanary, hilfialkaff, Lili Jiang, Meg Risdal, Nikhil Dandekar, and tomtung. 2017. Quora question pairs.

Hervé Déjean, Stephane Clinchant, Carlos Lassance, Simon Lupart, and Thibault Formal. 2023. Benchmarking middle-trained language models for neural search. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’23.

Meet Doshi, Vishwajeet Kumar, Rudra Murthy, Vignesh P. and Jaydeep Sen. 2024. Mistral-splade: Llms for better learned sparse retrieval. arXiv preprint arXiv:2110.01529.

Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. Eh5: Long form question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3558–3567.

Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. 2021a. Splade v2: Sparse lexical and expansion model for information retrieval. arXiv preprint arXiv:2109.10086.

Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. 2021b. Splade: Sparse lexical and expansion model for first stage ranking. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’21.

Luyu Gao, Zhuyun Dai, Tongfei Chen, Zhen Fan, Benjamin Van Durme, and Jamie Callan. 2021. Complement lexical retrieval model with semantic residual embeddings. In 43rd European Conference on IR Research, ECIR 2021, ECIR ’21, page 146–160.

John A Hartigan and Manchek A Wong. 1979. Algorithm as 136: A k-means clustering algorithm. Journal of the royal statistical society. series c (applied statistics), 28(1):100–108.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825.

Albert Q. Jiang, Alicja Ziariko, Bartosz Piotrowski, Wenda Li, Mateja Jamnik, and Piotr Miłoś. 2024. Repurposing language models into embedding models: Finding the compute-optimal recipe. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, and Ali Farhadi. 2022. Matryoshka representation learning. In Advances in Neural Information Processing Systems, volume 35, pages 30233–30249. Curran Associates, Inc.

Ken Lang. 1995. Newswedeer: Learning to filter net-news. In Machine learning proceedings 1995, pages 331–339. Elsevier.

Carlos Lassance, Hervé Déjean, Thibault Formal, and Stéphane Clinchant. 2024. Splade-v3: New baselines for splade. arXiv preprint arXiv:2403.06789.

Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2024a. Nv-embed: Improved techniques for training llms as generalist embedding models. arXiv preprint arXiv:2405.17428.

Jinhyuk Lee, Zhuyun Dai, Xiaoqi Ren, Blair Chen, Daniel Cer, Jeremy R. Cole, Kai Hui, Michael Boratko, Rajiv Kapadia, Wen Ding, Yi Luan, Sai Meher Karthik Duddu, Gustavo Hernandez Abrego, Weiqiang Shi, Nithi Gupta, Aditya Kusupati, Prateek Jain, Siddhartha Reddy Jonnalagadda, Ming-Wei Chang, and Iftekhar Naim. 2024b. Gecko: Versatile text embeddings distilled from large language models. arXiv preprint arXiv:2403.20327.

Yibin Lei, Di Wu, Tianyi Zhou, Tao Shen, Yu Cao, Chongyang Tao, and Andrew Yates. 2024. Meta-task prompting elicits embeddings from large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10141–10157. Association for Computational Linguistics.

Chaofan Li, Zheng Liu, Shitao Xiao, and Yingxia Shao. 2023a. Making large language models a better foundation for dense retrieval. arXiv preprint arXiv:2312.15503.

Chaofan Li, MingHao Qin, Shitao Xiao, Jianlyu Chen, Kun Luo, Yingxia Shao, Defu Lian, and Zheng Liu. 2024. Making text embedders few-shot learners. arXiv preprint arXiv:2409.15700.

Haoran Li, Abhinav Arora, Shuhui Chen, Anchit Gupta, Sonal Gupta, and Yashar Mehdad. 2021. Mtop: A comprehensive multilingual task-oriented semantic parsing benchmark. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2950–2962.

Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023b. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281.

Jimmy Lin. 2021. A proposed conceptual framework for a representational approach to information retrieval. arXiv preprint arXiv:2110.01529.

Jimmy Lin and Xueguang Ma. 2021. A few brief notes on deepimpact, coil, and a conceptual framework for information retrieval techniques. arXiv preprint arXiv:2106.14807.

Xueqing Liu, Chi Wang, Yue Leng, and ChengXiang Zhai. 2018. Linkso: a dataset for learning to retrieve similar question answer pairs on software development forums. In Proceedings of the 4th ACM SIGSOFT International Workshop on NLP for Software Engineering, pages 2–5.

Simon Lupart, Mohammad Aliannejadi, and Evangelos Kanoulas. 2024. Disco: Llm knowledge distillation for efficient sparse retrieval in conversational search. arXiv preprint arXiv:2410.14609.

Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pages 142–150.

Wei Chen Maggie, Phil Culliton. 2020. Tweet sentiment extraction.

Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. 2018. Www’18 open challenge: financial opinion mining and question answering. In Companion proceedings of the the web conference 2018, pages 1941–1942.

Antonio Mallia, Omar Khattab, Torsten Suel, and Nicola Tonellotto. 2021. Learning passage impacts for inverted indexes. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, page 1723–1727.

Julian McAuley and Jure Leskovec. 2013. Hidden factors and hidden topics: understanding rating dimensions with review text. In Proceedings of the 7th ACM conference on Recommender systems, pages 165–172.
Rui Meng, Ye Liu, Shafiq Rayhan Joty, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. 2024. Sfr-embedding-2: Advanced text embedding with multi-stage training.

Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. 2024. Generative representational instruction tuning.

Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. Mtbe: Massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2014–2037.

Thong Nguyen, Shubham Chatterjee, Sean MacAvaney, Iain Mackie, Jeff Dalton, and Andrew Yates. 2024. DyVo: Dynamic vocabularies for learned sparse retrieval with entities. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 767–783. Association for Computational Linguistics.

Thong Nguyen, Sean MacAvaney, and Andrew Yates. 2023a. Adapting learned sparse retrieval for long documents. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’23, page 1781–1785. Association for Computing Machinery.

Thong Nguyen, Sean MacAvaney, and Andrew Yates. 2023b. A unified framework for learned sparse retrieval. In Advances in Information Retrieval; 45th European Conference on Information Retrieval, ECIR 2023, Dublin, Ireland, April 2–6, 2023, Proceedings, Part III, pages 101–116. Springer.

James O’Neill, Polina Rozenshtein, Ryuichi Kiryo, Motoko Kubota, and Danushka Bollegala. 2021. I wish i would have loved this one, but i didn’t—a multilingual dataset for counterfactual detection in product review. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7092–7108.

Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, et al. 1995. Okapi at trec-3. Nist Special Publication Sp.

Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen. 2018. Carer: Contextualized affect representations for emotion recognition. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 3687–3697.

Tao Shen, Xiubo Geng, Chongyang Tao, Can Xu, Xiaolong Huang, Binxing Jiao, Linjun Yang, and Daxin Jiang. 2023a. LexMAE: Lexicon-bottlenecked pre-training for large-scale retrieval. In The Eleventh International Conference on Learning Representations.

Tao Shen, Xiubo Geng, Chongyang Tao, Can Xu, Guodong Long, Kai Zhang, and Daxin Jiang. 2023b. Unifier: A unified retriever for large-scale retrieval. KDD ’23.

Aina Garí Soler, Matthieu Labeau, and Chloé Clavel. 2024. The impact of word splitting on the semantic content of contextualized word representations. Transactions of the Association for Computational Linguistics.

Jacob Mitchell Springer, Suhas Kotha, Daniel Fried, Graham Neubig, and Aditi Raghunathan. 2024. Repetition improves language model embeddings. arXiv preprint arXiv:2402.15449.

James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. Fever: a large-scale dataset for fact extraction and verification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 809–819.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.

Henning Wachsmuth, Shahbaz Syed, and Benno Stein. 2018. Retrieval of the best counterargument without prior topic knowledge. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 241–251.

Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2023. Improving text embeddings with large language models. arXiv preprint arXiv:2401.00368.

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369–2380.

Yi Zeng, Yu Yang, Andy Zhou, Jeffrey Ziwei Tan, Yuheng Tu, Yifan Mai, Kevin Klyman, Minzhou Pan, Ruoxi Jia, Dawn Song, et al. 2024. Airbench 2024: A safety benchmark based on risk categories from regulations and policies. arXiv preprint arXiv:2407.17436.

Xinyu Zhang, Jing Lu, Vinh Q. Tran, Tal Schuster, Donald Metzler, and Jimmy Lin. 2024. Tomato, tomato, tomate: Measuring the role of shared semantics among subwords in multilingual language models. arXiv preprint arXiv:2411.04530.

Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermelo, Xiaoguang Li, Qun Liu, Mehdi Rezagholizadeh, and
Jimmy Lin. 2023. Miracl: A multilingual retrieval dataset covering 18 diverse languages. Transactions of the Association for Computational Linguistics, 11:1114–1131.

Shengyao Zhuang, Xueguang Ma, Bevan Koopman, Jimmy Lin, and Guido Zuccon. 2024. PromptReps: Prompting large language models to generate dense and sparse representations for zero-shot document retrieval. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.

Shengyao Zhuang and Guido Zuccon. 2021. Fast passage re-ranking with contextualized exact term matching and efficient passage expansion. arXiv preprint arXiv:2108.08513.

A Appendix

A.1 Clustering results

<table>
  <tr>
    <th>Clusters</th>
  </tr>
  <tr>
    <td>impact, Impact, impacts</td>
  </tr>
  <tr>
    <td>Entity, entity, #Entity, Entities, entities</td>
  </tr>
  <tr>
    <td>TV, television, tv, Television, televis</td>
  </tr>
  <tr>
    <td>comfort, comfortable, #comfort</td>
  </tr>
  <tr>
    <td>beautiful, lovely, gorgeous, handsome, beautifully</td>
  </tr>
  <tr>
    <td>guy, guys, Guy, dude</td>
  </tr>
  <tr>
    <td>fit, FIT, fits, fitting, fitted</td>
  </tr>
  <tr>
    <td>recomm, recommend, recommended, recommendation</td>
  </tr>
  <tr>
    <td>star, stars, Stars</td>
  </tr>
  <tr>
    <td>reach, reached, reaching, reaches, #reach</td>
  </tr>
</table>

Table 9: Cluster examples of LENS-8000. Each row presents tokens belonging to a single cluster.

A.2 Training data details

We leverage the public training data provided by BGE-en-ICL (Li et al., 2024). Specifically, the training data is a mixture of retrieval, reranking, classification, clustering, and STS data.

• Retrieval: ELI5 (Fan et al., 2019), HotpotQA (Yang et al., 2018), FEVER (Thorne et al., 2018), MSMARCO passage and document ranking (Bajaj et al., 2018), NQ, NLI, SQuAD, TriviaQA, Quora Duplicate Questions (DataCanary et al., 2017), Arguana (Wachsmuth et al., 2018), FiQA (Maia et al., 2018).

• Reranking: SciDocsRR (Cohan et al., 2020), StackOverFlowDupQuestions (Liu et al., 2018).

• Classification: AmazonReviews-Classification (McAuley and Leskovec, 2013), AmazonCounterfactual-Classification (O’Neill et al., 2021), Banking77-Classification (Casanueva et al., 2020), Emotion-Classification (Saravia et al., 2018), TweetSentimentExtraction-Classification (Maggie, 2020), MTOPIntent-Classification (Li et al., 2021), IMDB-Classification (Maas et al., 2011), ToxicConversations-Classification (Adams et al., 2019).

• Clustering: TwentyNewsgroups-Clustering (Lang, 1995), {Arxiv/Biorxiv/Medrxiv/Reddit/StackExchange}-Clustering-{S2S/P2P}
• STS: STS12 (Agirre et al., 2012), STS22 (Chen et al., 2022), STS-Benchmark (Cer et al., 2017).

A.3 Task Instructions
We present the task instructions we used in Table 10.

A.4 MTEB Subset Details
Following Jiang et al. (2024), for each task category, we select one dataset for evaluation. The chosen dataset is determined based on the model results presented in the original MTEB paper, focusing on the dataset with the highest correlation to the category’s average performance.

• Classification: EmotionClassification

• Clustering: TwentyNewsgroupsClustering

• Pair classification: SprintDuplicateQuestions

• Reranking: AskUbuntuDupQuestions

• Retrieval: SciFact

• Semantic text similarity: STS15

• Summarization: SummEval

A.5 Detailed MTEB Results
We present the detailed MTEB results in Table 11.
<table>
  <tr>
    <th>Task Name</th>
    <th>Instruction</th>
  </tr>
  <tr>
    <td>ArguAna</td>
    <td>Given a claim, find documents that refute the claim.</td>
  </tr>
  <tr>
    <td>ClimateFEVER</td>
    <td>Given a claim about climate change, retrieve documents that support or refute the claim.</td>
  </tr>
  <tr>
    <td>CQADupStack</td>
    <td>Given a question, retrieve detailed question descriptions from Stackexchange that are duplicates to the given question.</td>
  </tr>
  <tr>
    <td>DBPedia</td>
    <td>Given a query, retrieve relevant entity descriptions from DBPedia.</td>
  </tr>
  <tr>
    <td>FEVER</td>
    <td>Given a claim, retrieve documents that support or refute the claim.</td>
  </tr>
  <tr>
    <td>FiQA2018</td>
    <td>Given a financial question, retrieve user replies that best answer the question.</td>
  </tr>
  <tr>
    <td>HotpotQA</td>
    <td>Given a multi-hop question, retrieve documents that can help answer the question.</td>
  </tr>
  <tr>
    <td>MSMARCO</td>
    <td>Given a web search query, retrieve relevant passages that answer the query.</td>
  </tr>
  <tr>
    <td>NFCorpus</td>
    <td>Given a question, retrieve relevant documents that best answer the question.</td>
  </tr>
  <tr>
    <td>Natural Question</td>
    <td>Given a question, retrieve Wikipedia passages that answer the question.</td>
  </tr>
  <tr>
    <td>QuoraRetrieval</td>
    <td>Given a question, retrieve questions that are semantically equivalent to the given question.</td>
  </tr>
  <tr>
    <td>SCIDOCS</td>
    <td>Given a scientific paper title, retrieve paper abstracts that are cited by the given paper.</td>
  </tr>
  <tr>
    <td>SciFact</td>
    <td>Given a scientific claim, retrieve documents that support or refute the claim.</td>
  </tr>
  <tr>
    <td>Touche2020</td>
    <td>Given a question, retrieve detailed and persuasive arguments that answer the question.</td>
  </tr>
  <tr>
    <td>TREC-COVID</td>
    <td>Given a query, retrieve documents that answer the query.</td>
  </tr>
  <tr>
    <td>STS*</td>
    <td>Retrieve semantically similar text.</td>
  </tr>
  <tr>
    <td>SummEval</td>
    <td>Given a news summary, retrieve other semantically similar summaries.</td>
  </tr>
  <tr>
    <td>AmazonCounterfactualClassification</td>
    <td>Classify a given Amazon customer review text as either counterfactual or not-counterfactual.</td>
  </tr>
  <tr>
    <td>AmazonPolarityClassification</td>
    <td>Classify Amazon reviews into positive or negative sentiment.</td>
  </tr>
  <tr>
    <td>AmazonReviewsClassification</td>
    <td>Classify the given Amazon review into its appropriate rating category.</td>
  </tr>
  <tr>
    <td>Banking77Classification</td>
    <td>Given a online banking query, find the corresponding intents.</td>
  </tr>
  <tr>
    <td>EmotionClassification</td>
    <td>Classify the emotion expressed in the given Twitter message into one of the six emotions: anger, fear, joy, love, sadness, and surprise.</td>
  </tr>
  <tr>
    <td>ImdbClassification</td>
    <td>Classify the sentiment expressed in the given movie review text from the IMDB dataset.</td>
  </tr>
  <tr>
    <td>MassiveIntentClassification</td>
    <td>Given a user utterance as query, find the user intents.</td>
  </tr>
  <tr>
    <td>MassiveScenarioClassification</td>
    <td>Given a user utterance as query, find the user scenarios.</td>
  </tr>
  <tr>
    <td>MTOPDomainClassification</td>
    <td>Classify the intent domain of the given utterance in task-oriented conversation.</td>
  </tr>
  <tr>
    <td>MTOPIntentClassification</td>
    <td>Classify the intent of the given utterance in task-oriented conversation.</td>
  </tr>
  <tr>
    <td>ToxicConversationsClassification</td>
    <td>Classify the given comments as either toxic or not toxic.</td>
  </tr>
  <tr>
    <td>TweetSentimentExtractionClassification</td>
    <td>Classify the sentiment of a given tweet as either positive, negative, or neutral.</td>
  </tr>
  <tr>
    <td>ArxivClusteringP2P</td>
    <td>Identify the main and secondary category of Arxiv papers based on the titles and abstracts.</td>
  </tr>
  <tr>
    <td>ArxivClusteringS2S</td>
    <td>Identify the main and secondary category of Arxiv papers based on the titles.</td>
  </tr>
  <tr>
    <td>BiorxivClusteringP2P</td>
    <td>Identify the main category of Biorxiv papers based on the titles and abstracts.</td>
  </tr>
  <tr>
    <td>BiorxivClusteringS2S</td>
    <td>Identify the main category of Biorxiv papers based on the titles.</td>
  </tr>
  <tr>
    <td>MedrxivClusteringP2P</td>
    <td>Identify the main category of Medrxiv papers based on the titles and abstracts.</td>
  </tr>
  <tr>
    <td>MedrxivClusteringS2S</td>
    <td>Identify the main category of Medrxiv papers based on the titles.</td>
  </tr>
  <tr>
    <td>RedditClustering</td>
    <td>Identify the topic or theme of Reddit posts based on the titles.</td>
  </tr>
  <tr>
    <td>RedditClusteringP2P</td>
    <td>Identify the topic or theme of Reddit posts based on the titles and posts.</td>
  </tr>
  <tr>
    <td>StackExchangeClustering</td>
    <td>Identify the topic or theme of StackExchange posts based on the titles.</td>
  </tr>
  <tr>
    <td>StackExchangeClusteringP2P</td>
    <td>Identify the topic or theme of StackExchange posts based on the given paragraphs.</td>
  </tr>
  <tr>
    <td>TwentyNewsgroupsClustering</td>
    <td>Identify the topic or theme of the given news articles.</td>
  </tr>
  <tr>
    <td>AskUbuntuDupQuestions</td>
    <td>Retrieve duplicate questions from AskUbuntu forum.</td>
  </tr>
  <tr>
    <td>MindSmallReranking</td>
    <td>Retrieve relevant news articles based on user browsing history.</td>
  </tr>
  <tr>
    <td>SciDocsRR</td>
    <td>Given a title of a scientific paper, retrieve the titles of other relevant papers.</td>
  </tr>
  <tr>
    <td>StackOverflowDupQuestions</td>
    <td>Retrieve duplicate questions from StackOverflow forum.</td>
  </tr>
  <tr>
    <td>SprintDuplicateQuestions</td>
    <td>Retrieve duplicate questions from Sprint forum.</td>
  </tr>
  <tr>
    <td>TwitterSemEval2015</td>
    <td>Retrieve tweets that are semantically similar to the given tweet.</td>
  </tr>
  <tr>
    <td>TwitterURLCorpus</td>
    <td>Retrieve tweets that are semantically similar to the given tweet.</td>
  </tr>
  <tr>
    <td>AIR-Bench</td>
    <td>Given a question, retrieve passages that answer the question.</td>
  </tr>
</table>

Table 10: Task instructions for MTEB and AIR-Bench benchmarks.
<table>
  <tr>
    <th>Dataset</th>
    <th>gte-Qwen-7B-instruct</th>
    <th>SFR-Embedding-2_R</th>
    <th>stella_en_1.5B_v5</th>
    <th>BGE-en-ICL (zero-shot)</th>
    <th>NV-Embed-v2</th>
    <th>LENS -4000</th>
    <th>LENS -8000</th>
  </tr>
  <tr><td>ArguAna</td><td>64.27</td><td>62.34</td><td>65.27</td><td>82.76</td><td>70.07</td><td>77.32</td><td>76.02</td></tr>
  <tr><td>ClimateFEVER</td><td>45.88</td><td>34.43</td><td>46.11</td><td>45.35</td><td>45.39</td><td>44.62</td><td>45.77</td></tr>
  <tr><td>CQADupStack</td><td>46.43</td><td>46.11</td><td>47.75</td><td>47.23</td><td>50.24</td><td>47.39</td><td>48.67</td></tr>
  <tr><td>DBPEDIA</td><td>52.42</td><td>51.21</td><td>52.28</td><td>50.42</td><td>53.50</td><td>50.10</td><td>49.75</td></tr>
  <tr><td>FEVER</td><td>95.11</td><td>92.16</td><td>94.83</td><td>91.96</td><td>93.75</td><td>92.37</td><td>92.32</td></tr>
  <tr><td>FiQA2018</td><td>62.03</td><td>61.77</td><td>60.48</td><td>58.77</td><td>65.73</td><td>60.43</td><td>61.57</td></tr>
  <tr><td>HotpotQA</td><td>73.08</td><td>81.36</td><td>76.67</td><td>84.98</td><td>85.48</td><td>85.07</td><td>85.71</td></tr>
  <tr><td>MSMARCO</td><td>45.98</td><td>42.18</td><td>45.22</td><td>46.72</td><td>45.63</td><td>46.95</td><td>47.24</td></tr>
  <tr><td>NFCorpus</td><td>40.60</td><td>41.34</td><td>42.00</td><td>40.69</td><td>45.17</td><td>41.64</td><td>40.61</td></tr>
  <tr><td>Natural Question</td><td>67.00</td><td>73.96</td><td>71.80</td><td>73.85</td><td>73.57</td><td>73.13</td><td>74.64</td></tr>
  <tr><td>QuoraRetrieval</td><td>90.09</td><td>89.58</td><td>90.03</td><td>91.02</td><td>89.04</td><td>90.84</td><td>90.79</td></tr>
  <tr><td>SCIDOCS</td><td>28.91</td><td>24.87</td><td>26.64</td><td>25.25</td><td>21.90</td><td>27.51</td><td>28.54</td></tr>
  <tr><td>SciFact</td><td>79.06</td><td>85.91</td><td>80.09</td><td>78.33</td><td>80.13</td><td>78.39</td><td>79.75</td></tr>
  <tr><td>Touche2020</td><td>30.57</td><td>28.18</td><td>29.94</td><td>29.67</td><td>31.78</td><td>25.86</td><td>29.34</td></tr>
  <tr><td>TREC-COVID</td><td>82.26</td><td>87.28</td><td>85.98</td><td>78.11</td><td>88.44</td><td>69.73</td><td>77.18</td></tr>
  <tr><td>BIOSSES</td><td>81.37</td><td>87.60</td><td>83.11</td><td>86.35</td><td>87.42</td><td>84.47</td><td>85.83</td></tr>
  <tr><td>SICK-R</td><td>79.28</td><td>77.01</td><td>82.89</td><td>83.87</td><td>82.15</td><td>83.81</td><td>83.30</td></tr>
  <tr><td>STS12</td><td>79.55</td><td>75.67</td><td>80.09</td><td>77.73</td><td>77.89</td><td>79.07</td><td>80.99</td></tr>
  <tr><td>STS13</td><td>88.83</td><td>82.40</td><td>89.68</td><td>85.98</td><td>88.30</td><td>86.54</td><td>87.34</td></tr>
  <tr><td>STS14</td><td>83.87</td><td>79.93</td><td>85.07</td><td>82.34</td><td>84.30</td><td>84.32</td><td>84.39</td></tr>
  <tr><td>STS15</td><td>88.54</td><td>85.82</td><td>89.39</td><td>87.35</td><td>89.04</td><td>89.69</td><td>89.75</td></tr>
  <tr><td>STS16</td><td>86.49</td><td>84.50</td><td>87.15</td><td>86.54</td><td>86.77</td><td>87.23</td><td>87.63</td></tr>
  <tr><td>STS17</td><td>88.73</td><td>88.93</td><td>91.35</td><td>91.25</td><td>90.67</td><td>91.55</td><td>90.87</td></tr>
  <tr><td>STS22</td><td>66.88</td><td>67.10</td><td>68.10</td><td>68.08</td><td>68.12</td><td>68.69</td><td>68.09</td></tr>
  <tr><td>STSBenchmark</td><td>86.85</td><td>83.60</td><td>88.23</td><td>87.92</td><td>88.41</td><td>88.22</td><td>88.47</td></tr>
  <tr><td>SummEval</td><td>31.35</td><td>30.71</td><td>31.49</td><td>30.75</td><td>30.70</td><td>31.55</td><td>29.54</td></tr>
  <tr><td>SprintDuplicateQuestions</td><td>92.82</td><td>97.62</td><td>96.04</td><td>95.06</td><td>97.02</td><td>96.98</td><td>97.00</td></tr>
  <tr><td>TwitterSemEval2015</td><td>77.96</td><td>78.57</td><td>80.58</td><td>78.54</td><td>81.11</td><td>79.31</td><td>79.56</td></tr>
  <tr><td>TwitterURLCorpus</td><td>86.59</td><td>88.03</td><td>87.58</td><td>87.19</td><td>87.87</td><td>87.50</td><td>87.37</td></tr>
  <tr><td>AmazonCounterfactual</td><td>91.31</td><td>92.72</td><td>92.87</td><td>92.88</td><td>94.28</td><td>93.61</td><td>93.69</td></tr>
  <tr><td>AmazonPolarity</td><td>97.50</td><td>97.31</td><td>97.16</td><td>96.86</td><td>97.74</td><td>97.05</td><td>97.07</td></tr>
  <tr><td>AmazonReviews</td><td>62.56</td><td>61.04</td><td>59.36</td><td>61.28</td><td>63.96</td><td>62.83</td><td>63.61</td></tr>
  <tr><td>Banking77</td><td>87.57</td><td>90.02</td><td>89.79</td><td>91.42</td><td>92.42</td><td>90.43</td><td>90.19</td></tr>
  <tr><td>Emotion</td><td>79.45</td><td>93.37</td><td>84.29</td><td>93.31</td><td>93.38</td><td>92.33</td><td>91.87</td></tr>
  <tr><td>Imdb</td><td>96.75</td><td>96.80</td><td>96.66</td><td>96.91</td><td>97.14</td><td>97.12</td><td>97.00</td></tr>
  <tr><td>MassiveIntent</td><td>85.41</td><td>85.97</td><td>85.83</td><td>82.26</td><td>86.10</td><td>79.65</td><td>81.14</td></tr>
  <tr><td>MassiveScenario</td><td>89.77</td><td>90.61</td><td>90.20</td><td>83.92</td><td>92.17</td><td>81.97</td><td>83.53</td></tr>
  <tr><td>MTOPDomain</td><td>99.04</td><td>98.58</td><td>99.01</td><td>97.99</td><td>99.25</td><td>97.49</td><td>97.44</td></tr>
  <tr><td>MTOPIntent</td><td>91.88</td><td>91.30</td><td>92.78</td><td>93.56</td><td>94.37</td><td>92.59</td><td>92.81</td></tr>
  <tr><td>ToxicConversations</td><td>85.12</td><td>91.14</td><td>88.76</td><td>93.16</td><td>92.74</td><td>92.29</td><td>92.37</td></tr>
  <tr><td>TweetSentimentExtraction</td><td>72.58</td><td>79.70</td><td>74.84</td><td>79.90</td><td>80.87</td><td>80.17</td><td>80.42</td></tr>
  <tr><td>Arxiv-P2P</td><td>54.46</td><td>54.02</td><td>55.44</td><td>54.42</td><td>55.80</td><td>54.87</td><td>54.81</td></tr>
  <tr><td>Arxiv-S2S</td><td>51.74</td><td>48.82</td><td>50.66</td><td>49.17</td><td>51.26</td><td>50.25</td><td>50.14</td></tr>
  <tr><td>Biorxiv-P2P</td><td>50.09</td><td>50.76</td><td>50.68</td><td>52.32</td><td>54.09</td><td>52.39</td><td>52.48</td></tr>
  <tr><td>Biorxiv-S2S</td><td>46.65</td><td>46.57</td><td>46.87</td><td>48.38</td><td>49.60</td><td>48.35</td><td>48.52</td></tr>
  <tr><td>Medrxiv-P2P</td><td>46.23</td><td>46.66</td><td>46.87</td><td>46.13</td><td>46.09</td><td>46.35</td><td>46.38</td></tr>
  <tr><td>Medrxiv-S2S</td><td>44.13</td><td>44.18</td><td>44.65</td><td>44.20</td><td>44.86</td><td>44.54</td><td>44.89</td></tr>
  <tr><td>Reddit</td><td>73.55</td><td>62.92</td><td>72.86</td><td>71.20</td><td>71.10</td><td>72.32</td><td>72.37</td></tr>
  <tr><td>Reddit-P2P</td><td>74.13</td><td>72.74</td><td>75.27</td><td>72.17</td><td>74.94</td><td>73.20</td><td>73.89</td></tr>
  <tr><td>StackExchange</td><td>79.86</td><td>76.48</td><td>80.29</td><td>81.29</td><td>82.10</td><td>81.70</td><td>81.60</td></tr>
  <tr><td>StackExchange-P2P</td><td>49.41</td><td>48.29</td><td>49.57</td><td>45.53</td><td>48.36</td><td>43.73</td><td>44.41</td></tr>
  <tr><td>TwentyNewsgroups</td><td>53.91</td><td>66.42</td><td>61.43</td><td>68.51</td><td>64.82</td><td>69.44</td><td>68.78</td></tr>
  <tr><td>AskUbuntuDupQuestions</td><td>67.58</td><td>66.71</td><td>67.33</td><td>64.80</td><td>67.46</td><td>65.45</td><td>65.74</td></tr>
  <tr><td>MindSmallRerank</td><td>33.36</td><td>31.26</td><td>33.05</td><td>30.60</td><td>31.76</td><td>31.92</td><td>31.46</td></tr>
  <tr><td>SciDocsRR</td><td>89.09</td><td>87.29</td><td>89.20</td><td>86.90</td><td>87.59</td><td>87.92</td><td>87.63</td></tr>
  <tr><td>StackOverflowDupQuestions</td><td>55.66</td><td>55.32</td><td>55.25</td><td>56.32</td><td>55.79</td><td>58.15</td><td>58.79</td></tr>
  <tr><th>MTEB Average (56)</th><th>70.24</th><th>70.31</th><th>71.19</th><th>71.24</th><th>72.31</th><th>71.22</th><th>71.63</th></tr>
</table>

Table 11: Detailed MTEB results.

Reviewer paper summaries: - The authors propose a novel lexicon-based text embedding models with SOTA embedding performances compared with popular dense embedding models.
- More specifically, the authors introduce token clustering and bi-directional attentions to improve the performance of lexicon-based text embedding models.
- The authors provide extensive experiments with compelling results and additional ablation studies as well as qualitative analyses.

This paper introduces LENS, a novel framework for generating lexicon-based text embeddings using large language models (LLMs). LENS addresses two main challenges: tokenizer redundancy and unidirectional attention limitations. To tackle these issues, LENS consolidates the vocabulary space through token embedding clustering and incorporates bidirectional attention. The framework assigns each dimension to a specific token cluster rather than individual tokens, reducing embedding dimensionality while maintaining semantic relationships. Extensive experiments demonstrate that LENS outperforms dense embeddings on the Massive Text Embedding Benchmark (MTEB), achieving state-of-the-art zero-shot performance among models trained on public data.

The work introduces Lexicon-based EmbeddiNgS (LENS), a novel framework leveraging large language models (LLMs) to create competitive text embeddings. While dense embeddings have been the focus of recent research, LENS addresses limitations in traditional causal LLMs, such as tokenization redundancy and unidirectional attention.

Key innovations of LENS include:

1. **Token Clustering**: LENS consolidates the vocabulary by clustering token embeddings, grouping semantically similar tokens together. This reduces redundancy and enhances lexicon matching.

2. **Bidirectional Attention**: The framework incorporates bidirectional attention, allowing each token to utilize surrounding context, which is crucial for generating effective lexicon-based embeddings.

3. **Pooling Strategies**: LENS explores various pooling strategies to optimize the embedding process.

Experiments show that LENS outperforms dense embeddings on the Massive Text Embedding Benchmark (MTEB) and achieves state-of-the-art performance in zero-shot scenarios. Additionally, combining LENS with dense embeddings yields top results in the retrieval subset of MTEB (BEIR). Overall, LENS promises compact, meaningful feature representations while addressing challenges posed by existing LLM architectures.
Summary of strengths: - The proposed method is quite novel and intuitive, especially for the token clustering part.
- The experimental section is quite detailed and well organized. It is quite convincing.
- The paper is well written and easy to follow.

- The idea of lexicon matching by assigning each dimension to a specific token cluster and grouping semantically similar tokens to reduce the dimensionality of lexicon-based retrieval is interesting. 
- LENS demonstrates impressive results on comprehensive benchmarks like MTEB and AIR-Bench, often outperforming state-of-the-art dense embedding models.
- The experiments expanding the number of clusters provide evidence for the effectiveness of the proposed method to consolidate the vocabulary space.

1. Clear demonstration.
2. Thorough comparison.
Summary of weaknesses: - It is unclear what are the benefits of the proposed method compared with the dense embedding models. 
- Could we claim that bi-directional attention part is a novel contribution?

- The paper lacks comprehensive ablation studies, particularly for the token clustering method. Distilling ranking scores from the BGE-reranker model should be ablated to investigate the effectiveness of the proposed clustering approach independently.
- While the paper provides some examples of clusters, a more thorough visualization of the clusters would be beneficial. For instance, showing the top-k tokens nearest to each cluster centroid could offer deeper insights into the semantic meaning and coherence of the clusters.

1. The performance improvements appear to be somewhat subtle.
2. Why not include few-shot comparisons? This would allow for a more comprehensive evaluation against models like BGE-en-ICL.
Comments, suggestions and typos: 1. LENS vs sparse embedding vs dense embedding
- One of the main advantage of sparse embedding is that the embedding itself can be represented as the sparse vector.
- This is highly beneficial on the retrieval latency. It would be great if the authors also provide some latency analyses with LENS vs dense embedding models.

2. Combining with dense embedding
- As described in Table 6, it shows the performance improvements when combining LENS and one dense embedding.
- But unlike what authors explained in the introduction, the impact is quite small. In that point of view, can we really say that LENS and dense embeddings are somewhat complementary? 
- Personally, the current results in Table 6 is somewhat like ensembling effects where we use two model ensembles. (maybe we can get similar performance improvements when we combine NV-embed-2 and BGE)

N/A

As in weaknesses.
Overall assessment scores: 3.5

3.5

2.5
Metareview score: 4
Metareview: This research introduces LENS, a novel framework for generating lex embeddings using large language models (LLMs). LENS addresses challenges faced by lexicon-based embeddings like token redundancy and unidirectional attention limitations by clustering semantically similar tokens and employing bidirectional attention.  Experimental results demonstrate that LENS outperforms existing dense embeddings on the Massive Text Embedding Benchmark (MTEB) and achieves state-of-the-art retrieval performance when combined with dense embeddings. The study also analyzes the impact of various architectural choices, such as attention mechanisms and pooling strategies, on the performance of lexicon-based embeddings.
Summary of reasons to publish: * This work addresses key challenges in constructing lexicon-based embeddings using LLMs.  Lexicon-based embeddings offer improved interpretability, and the demonstrated performance gains have significant implications for real-world applications where both quality and transparency are critical.
* The ablation study thoroughly evaluates the contribution of each proposed component, including token clustering, bidirectional attention, and pooling strategy.
* In author response, additional experimental results show that LENS can be further optimized with top-k pruning to improve the effciency. 
* The paper is nicely written and easy to follow.