LDIR: Low-Dimensional Dense and Interpretable Text Embeddings with Relative Representations
Authors: Yile Wang, Zhanyu Shen, Hui Huang
Accepted at: acl25

LDIR: Low-Dimensional Dense and Interpretable Text Embeddings with Relative Representations

Yile Wang, Zhanyu Shen, Hui Huang*
College of Computer Science and Software Engineering, Shenzhen University
wangyile@szu.edu.cn, zhanyushen.s@gmail.com, hhzhiyan@gmail.com

Abstract

Semantic text representation is a fundamental task in the field of natural language processing. Existing text embedding (e.g., SimCSE and LLM2Vec) have demonstrated excellent performance, but the values of each dimension are difficult to trace and interpret. Bag-of-words, as classic sparse interpretable embeddings, suffers from poor performance. Recently, Benara et al. (2024) propose interpretable text embeddings using large language models, which forms “0/1” embeddings based on responses to a series of questions. These interpretable text embeddings are typically high-dimensional (larger than 10,000). In this work, we propose Low-dimensional (lower than 500) Dense and Interpretable text embeddings with Relative representations (LDIR). The numerical values of its dimensions indicate semantic relatedness to different anchor texts through farthest point sampling, offering both semantic representation as well as a certain level of traceability and interpretability. We validate LDIR on multiple semantic textual similarity, retrieval, and clustering tasks. Extensive experimental results show that LDIR performs close to the black-box baseline models and outperforms the interpretable embeddings baselines with much fewer dimensions. Code is available at https://github.com/szu-tera/LDIR.

<table>
  <tr>
    <th>Embeddings</th>
    <th>Type</th>
    <th>Interp.</th>
    <th>Examples</th>
  </tr>
  <tr>
    <td>SimCSE</td>
    <td>Dense Emb</td>
    <td></td>
    <td>0.71, -0.03, ..., 0.13<br>(uninterpretable)</td>
  </tr>
  <tr>
    <td>LLM2Vec</td>
    <td>(768–4,096 dim.)</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Bag-of-Words</td>
    <td>Sparse Emb (~30,000 dim.)</td>
    <td></td>
    <td>12, 5, ..., 0<br>(occurrence of <word>_s)</td>
  </tr>
  <tr>
    <td>QAEmb-MBQA</td>
    <td>0/1 Emb (~10,000 dim.)</td>
    <td></td>
    <td>1, 0, ..., 1<br>("yes/no" to <question>_s)</td>
  </tr>
  <tr>
    <td>CQG-MBQA</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>LDIR (Ours)</td>
    <td>Dense Emb (~500 dim.)</td>
    <td></td>
    <td>0.16, 0.83, ..., 0.35<br>(relatedness to <anchor>_s)</td>
  </tr>
</table>

Table 1: Comparing existing text embedding with LDIR. In particular, our text embedding is relatively low-dimensional and dense, while maintaining interpretable.

Seminal text embedding models mainly fall into two categories. The first is based on lightweight pre-trained models such as BERT (Devlin et al., 2019), including Sentence-BERT (Reimers and Gurevych, 2019), SimCSE (Gao et al., 2021), etc. The second is based on current large language models such as LLaMA-3 (Dubey et al., 2024), including LLM2Vec (BehnamGhader et al., 2024), AngIE (Li and Li, 2024), etc. The text embeddings usually have 768 to 4096 dimensions correlated with the hidden size of the model and perform well across tasks. However, the values on each dimension are difficult to trace and interpret directly.

In contrast, bag-of-words is a classical sparse and interpretable text representation where each dimension’s value represents the frequency of a specific word. However, its representational capability is limited. Recent QAEmb-MBQA (Benara et al., 2024) and CQG-MBQA (Sun et al., 2025) generate interpretable text embeddings based on large language models, where the value on each dimension is restricted to “0/1”, reflecting “yes/no” answers to different crafted questions, as shown in Table 1. These works pave the way for new developments in interpretable text embedding.

In the aforementioned interpretable text embeddings, the value on each dimension are explicitly defined. However, due to the limitations of their sparse or “0/1” representations, they often require
a high dimensionality. For example, bag-of-words is associated with the word vocabulary size, which typically reaches around 30K sizes in current models. QAEmb-MBQA and CQG-MBQA rely on designing a large number of questions, usually around 10K. In this work, we aim to mitigate the limitations of sparse or “0/1” representation and the requirement of high dimensionality.

Inspired by Moschella et al. (2023), we propose building low-dimensional, dense, and interpretable text embeddings with relative representations (LDIR). Specifically, the value on each dimension represents the relatedness to different “anchor texts” by automatic farthest point sampling, which can be floating-point numbers instead of “0/1”, thereby leading a overall flexible and dense embeddings. We hope the dimensionality of vectors can be significantly reduced in this way, achieving a balance among semantic expressiveness, representational efficiency, and interpretability.

We conduct extensive experiments to validate the semantic expressiveness of LDIR, including evaluations on seven semantic textual similarity tasks, six retrieval tasks, and seven clustering tasks. Experimental results show that our embeddings, with only 500 embedding dimensions, achieve comparable performance to black-box embeddings, and outperforms interpretable embedding baselines of QAEmb-MBQA and CQG-MBQA.

2 Related Work

Semantic Text Embedding. For pre-trained neural model based text embeddings, Sentence-BERT (Reimers and Gurevych, 2019) and SimCSE (Gao et al., 2021) use siamese network or contrastive learning to improve the semantic representations of BERT (Devlin et al., 2019). For large language model based text embeddings, MetaEOL (Lei et al., 2024) elicit the text embeddings from a single-token through prompting LLaMA (Touvron et al., 2023). AngIE (Li and Li, 2024) optimizes angle differences instead of cosine similarity for better modeling semantics. LLM2Vec (BehnamGhader et al., 2024) enables bidirectional attention and avoid the limitations of decoder-only models in encoding texts. NV-Embed (Lee et al., 2025) propose latent attention layer to enhance the general-purpose embedding tasks of decoder-only models. Although these methods produce dense text embedding and show strong performance on benchmarks, however, it’s difficult to interpret each dimension of the resulted dense representations.

Interpretable Text Embedding. Bag-of-words (BoW) is one of the most classic and interpretable text embedding, where each dimension represents the frequency of occurrence for each word in the vocabulary. However, its representation capacity is largely limited. LISA (Patel et al., 2023) distills linguistically interpretable style attribute embeddings for author style representation, which is not general semantics oriented. QAEmb-MBQA (Benara et al., 2024) proposes designing multiple questions and use the “yes/no” answers by language models as “0/1” embeddings for semantic representations. CQG-MBQA (Sun et al., 2025) further improves the question generation module and reduce the costs for obtaining the “0/1” embeddings. Unlike these approaches, we develop low-dimensional dense text embeddings which can improve the general semantic expressiveness while preserving a certain of interpretability.

Relative Representation. Moschella et al. (2023) observe that the latent similarity between each sample and a fixed set of anchors in neural networks shows invariance for different training settings. They call such latent similarity as relative representation and verify the findings on the space of word embeddings (Mikolov, 2013; Bojanowski et al., 2017) or model stitching (Bansal et al., 2021). Norelli et al. (2023) use relative representations to align images and text for zero-shot classification. Maniparambil et al. (2024) also leverage such cross-modal alignment for caption matching and retrieval tasks. Overall, the above studies primarily illustrate the potential of relative representation for feature alignment, rather than utilizing it for sentence-level semantic representations.

3 Method

3.1 QA-Embedding Baseline

Formally, we define the semantic text embedding \( e(t) \) of a given a text \( t \) as:

\[
e(t) = \mathcal{E}(t) = [v_1, v_2, ..., v_d],
\]

where \( \mathcal{E} \) is an embedding model, \( v_1, v_2, ..., v_d \) are the values in each dimension. For neural model \( \mathcal{E} \), the embedding \( [v_1, v_2, ..., v_d] \) generally encode some high-level semantic features. However, it is difficult to interpret the specific meanings of each value \( v_i \) directly.
Figure 1: Comparison between baselines (top) and our method (bottom). Benara et al. (2024) and Sun et al. (2025) uses the generated questions and “yes/no” answers to build high-dimensional “0/1” text embeddings. We use anchor texts via farthest point sampling to calculate the relatedness and build low-dimensional dense text embeddings.

Benara et al. (2024) propose generating interpretable QA-Embedding to make the values in each dimension meaningful. In particular, they first generate k questions \( q_1, q_2, ..., q_k \) by leveraging GPT-4 with specific prompts, then obtain the k “yes” or “no” responses \( \{r_j\}_{j=1}^k \) by using large language models \( \mathcal{M} \) (e.g., GPT-4 or LLaMA-3):

\[
r_1, ..., r_k = \mathcal{M}(q_1 \oplus t), ..., \mathcal{M}(q_k \oplus t),
\]

where \( \oplus \) indicates concatenation operation, and the answer \( r_j \) is either “yes” or “no” to the question \( q_j \).

The interpretable text embedding is defined as the “0/1” sequence of according to the answers:

\[
\begin{aligned}
e_{0/1}^{\text{interp}}(t) &= [\hat{r}_1, \hat{r}_2, ..., \hat{r}_k], \\
\hat{r}_j &= \begin{cases}
1, & \text{if } r_j = \mathcal{M}(q_j \oplus t) \text{ is "yes"}, \\
0, & \text{otherwise}.
\end{cases}
\end{aligned}
\]

Sun et al. (2025) further improves the QA-Embedding by reducing the costs of calling LLMs through applying external trainable question generation module. We show an example of QA-Embedding in the top of Figure 1.

Both Benara et al. (2024) and Sun et al. (2025) rely on high-quality questions \( \{q_j\}_{j=1}^k \) generated via GPT-4 or trained models, followed by extra filtering. On the other hand, since \( \hat{r}_j \) can only be 0 or 1, the number of questions is extremely large, often reaching 9000~10000 (i.e., the number \( k \)).

3.2 “0/1” Embedding to Dense Embedding

Although the above values \( \{\hat{r}_j\}_{j=1}^k \) of the embedding \( e^{\text{interp}}(t) \) is interpretable, the expressive power is greatly limited because all values are either 0 or 1. Therefore, we consider using dense representations with floating-point number of each dimension. The overview of our dense and interpretable embedding is illustrated in the bottom of Figure 1.

Instead of generating \( k \) questions \( q_1, q_2, ..., q_k \), we first select \( n \) representative texts \( a_1, a_2, ..., a_n \) which we called “anchor texts” (detailed in the following subsection). Then, for each anchor text \( a_j \), we compute the relatedness between \( a_j \) and \( t \).

\[
s_j = \text{REL}(a_j, t),
\]

where \( \text{REL}(a_j, t) \) indicates the relatedness score of \( a_j \) and \( t \), which can be computed through general encoder or trained text encoders using widely used cosine similarity (Mikolov, 2013; Reimers and Gurevych, 2019; Ethayarajh, 2019):

\[
\text{REL}(a_j, t) = \frac{\text{ENC}(a_j) \cdot \text{ENC}(t)}{||\text{ENC}(a_j)|| \cdot ||\text{ENC}(t)||},
\]

where ENC can be an pre-trained encoder such as SimCSE (Gao et al., 2021), ModernBERT (Warner et al., 2024), and AngIE (Li and Li, 2024), without further fine-tuning.

After obtaining the relatedness scores \( \{s_j\}_{j=1}^n \), our dense and interpretable embeddings is:

\[
\begin{aligned}
e^{\text{dense}}_{\text{interp}}(t) &= [s_1, ..., s_n] \\
&= [\text{REL}(a_1, t), ..., \text{REL}(a_n, t)].
\end{aligned}
\]
Figure 2: Example of extracting four anchor texts using farthest point sampling. Each dot represents a text embedding through an encoder, with the blue dots indicating the extracted anchor texts.

Comparing Eq. 6 with Eq. 3, our embeddings \( e_{\text{interp}}^{\text{dense}}(t) \) are represented based on floating-point numbers, which can alleviate the requirement for high dimensionality. In practice, we find that 200~500 (i.e., the number \( n \)) anchor texts are enough to achieve better performance.

3.3 Anchor Texts Selection

In contrast to other interpretable embedding methods that rely on the extensive general knowledge of large language models (e.g., GPT-4) to derive and filter explanatory questions, we propose to automatically sample anchor texts from the corpus without generation and filtering. Selecting representative texts \( a_1, a_2, ..., a_n \) as anchor texts are important. For example, if all the anchor texts are similar to each other, then values in the final embeddings in Eq. 6 will also be close, thus the representativeness of overall embedding is limited. In practice, we use the farthest point sampling algorithm to select the representative anchor texts and compare with other sampling methods such as uniform sampling and K-Means in Section 5.1.

Farthest point sampling (FPS) is a widely used method that identifies representative subset within the complete feature space. In our workflow, the generation process begins by applying an encoder to the corpus texts, producing \( N \) embeddings \( \{e_1, e_2, ..., e_N\} \). Then a subset \( \{\hat{e}_1, \hat{e}_2, ..., \hat{e}_n\} \) is sampled gradually such that the distance between any two embedding vectors \( (\hat{e}_i, \hat{e}_j) \) are farther apart than that of any embedding vector pair in the remaining set \( \{e_1, e_2, ..., e_{N-n}\} \). Therefore, the text corresponding to the collected subset of embeddings constitutes the anchor texts \( a_1, a_2, ..., a_n \) required. An example of FPS procedure is shown in Figure 2. Benara et al. (2024) uses AngIE (Li and Li, 2024) to encode texts in corpus for generating questions by GPT-4. We follow them and set AngIE as the text encoder.

<table>
  <tr>
    <th>Embeddings</th>
    <th>Type</th>
    <th>Dim.</th>
    <th>Ext. Costs</th>
    <th>Interp.↑</th>
    <th>Perf.↑</th>
  </tr>
  <tr>
    <td>SimCSE<br>LLM2Vec</td>
    <td>Dense<br>Dense</td>
    <td>Medium<br>Medium</td>
    <td></td>
    <td>Low<br>Low</td>
    <td>High<br>High</td>
  </tr>
  <tr>
    <td>Bag-of-Words</td>
    <td>Sparse</td>
    <td>Large</td>
    <td>Low</td>
    <td>High</td>
    <td>Low</td>
  </tr>
  <tr>
    <td>QAEmb-MBQA<br>CQG-MBQA</td>
    <td>0/1 Emb.<br>Large</td>
    <td>Large</td>
    <td>High</td>
    <td>High</td>
    <td>Medium</td>
  </tr>
  <tr>
    <td>LDIR (Ours)</td>
    <td>Dense</td>
    <td>Small</td>
    <td>Low</td>
    <td>Medium</td>
    <td>High</td>
  </tr>
</table>

Table 2: Overall comparison between existing embeddings and LDIR. †: The external costs indicate the costs for external training or calling large language models.

3.4 Overall Comparison

We make an overall comparison to existing embedding methods in Table 2. Comparing from different perspectives, we achieve **low-dimensional and dense embeddings with low costs**. Specifically, SimCSE and LLM2Vec require additional contrastive learning to fine-tune models, QAEmb-MBQA and CQG-MBQA require extra calls on GPT-4 or training question generation modules. As comparison, LDIR primarily employs a automatically farthest point sampling strategy to extract anchor texts, and the calculation of embeddings does not require prompting large language models.

We also notice a **trade-off between interpretability and performance** across embeddings. For example, SimCSE and LLM2Vec exhibit strong semantic representation capabilities, but their interpretability is relatively low. On the other hand, bag-of-words offers high interpretability but performs poorly. Compared to QAEmb-MBQA and CQG-MBQA, LDIR sacrifices some interpretability (as the overall semantic relatedness is less direct and explicit than responses to questions) but achieves better performance, comparable to embeddings like SimCSE (see experimental results in Section 4.2). In Section 5.3, we further discuss how to obtain fine-grained relatedness to enhance the interpretability of our approach.

4 Experiments

4.1 Settings

Corpus. We use the same texts resource MEDI2\(^1\) and pre-processing steps (merging and filtering) by Sun et al. (2025), which results in a final corpus of approximately 6.8 million sentences for task-agnostic anchor texts extraction.

Tasks and Datasets. For semantic textual similarity (STS) tasks, we evaluate on SemEval STS

\footnotetext{1https://huggingface.co/datasets/GritLM/MEDI2}
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Dim.</th>
    <th rowspan="2">Type</th>
    <th colspan="6">Spearman Correlation (STS)</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>STS12</th>
    <th>STS13</th>
    <th>STS14</th>
    <th>STS15</th>
    <th>STS16</th>
    <th>STS-B</th>
    <th>SICK-R</th>
  </tr>
  <tr>
    <td colspan="9">(Black-Box Embeddings)</td>
    <td></td>
  </tr>
  <tr>
    <td>BERT<sub>base</sub> (Devlin et al., 2019)</td>
    <td>768</td>
    <td>Dense</td>
    <td>38.78</td>
    <td>57.98</td>
    <td>57.98</td>
    <td>63.15</td>
    <td>61.06</td>
    <td>46.35</td>
    <td>58.40</td>
    <td>54.81</td>
  </tr>
  <tr>
    <td>GloVe (Pennington et al., 2014)</td>
    <td>300</td>
    <td>Dense</td>
    <td>54.64</td>
    <td>69.16</td>
    <td>60.81</td>
    <td>72.31</td>
    <td>65.34</td>
    <td>61.54</td>
    <td>55.43</td>
    <td>62.74</td>
  </tr>
  <tr>
    <td>USE (Cer et al., 2018)</td>
    <td>512</td>
    <td>Dense</td>
    <td>64.49</td>
    <td>67.80</td>
    <td>64.61</td>
    <td>76.83</td>
    <td>73.18</td>
    <td>74.92</td>
    <td>76.69</td>
    <td>71.22</td>
  </tr>
  <tr>
    <td>LLM2Vec (BehnamGhader et al., 2024)</td>
    <td>4,096</td>
    <td>Dense</td>
    <td>61.60</td>
    <td>79.71</td>
    <td>72.11</td>
    <td>82.18</td>
    <td>79.41</td>
    <td>77.44</td>
    <td>72.16</td>
    <td>74.94</td>
  </tr>
  <tr>
    <td>SimCSE<sub>emb</sub> (Gao et al., 2021)</td>
    <td>768</td>
    <td>Dense</td>
    <td>66.05</td>
    <td>81.49</td>
    <td>73.61</td>
    <td>79.72</td>
    <td>78.12</td>
    <td>76.52</td>
    <td>72.24</td>
    <td>75.39</td>
  </tr>
  <tr>
    <td>SBERT<sub>ori</sub> (Reimers and Gurevych, 2019)</td>
    <td>768</td>
    <td>Dense</td>
    <td>74.53</td>
    <td>77.00</td>
    <td>73.18</td>
    <td>81.85</td>
    <td>76.82</td>
    <td>79.10</td>
    <td>74.29</td>
    <td>76.68</td>
  </tr>
  <tr>
    <td>SimCSE<sub>emb</sub> (Gao et al., 2021)</td>
    <td>768</td>
    <td>Dense</td>
    <td>75.30</td>
    <td>84.67</td>
    <td>80.19</td>
    <td>85.40</td>
    <td>80.82</td>
    <td>84.25</td>
    <td>68.38</td>
    <td>79.86</td>
  </tr>
  <tr>
    <td>WhitenedCSE (Zhuo et al., 2023)</td>
    <td>768</td>
    <td>Dense</td>
    <td>74.65</td>
    <td>85.79</td>
    <td>77.49</td>
    <td>84.71</td>
    <td>80.33</td>
    <td>81.48</td>
    <td>75.34</td>
    <td>79.97</td>
  </tr>
  <tr>
    <td>SBERT<sub>new</sub> (Reimers and Gurevych, 2019)</td>
    <td>768</td>
    <td>Dense</td>
    <td>73.08</td>
    <td>82.13</td>
    <td>76.73</td>
    <td>85.58</td>
    <td>80.23</td>
    <td>83.09</td>
    <td>79.32</td>
    <td>80.02</td>
  </tr>
  <tr>
    <td>text-embedding-ada-002 (OpenAI, 2022)</td>
    <td>1,536</td>
    <td>Dense</td>
    <td>72.84</td>
    <td>86.10</td>
    <td>81.15</td>
    <td>88.49</td>
    <td>85.08</td>
    <td>83.56</td>
    <td>79.00</td>
    <td>82.31</td>
  </tr>
  <tr>
    <td>ModernBERT<sub>emb</sub> (Warner et al., 2024)</td>
    <td>1,024</td>
    <td>Dense</td>
    <td>80.67</td>
    <td>87.87</td>
    <td>83.80</td>
    <td>88.59</td>
    <td>86.82</td>
    <td>87.40</td>
    <td>80.31</td>
    <td>85.07</td>
  </tr>
  <tr>
    <td>AngIE (Li and Li, 2024)</td>
    <td>1,024</td>
    <td>Dense</td>
    <td>79.09</td>
    <td>89.62</td>
    <td>85.02</td>
    <td>89.51</td>
    <td>86.61</td>
    <td>89.06</td>
    <td>82.62</td>
    <td>85.93</td>
  </tr>
  <tr>
    <td colspan="9">(Interpretable Embeddings)</td>
    <td></td>
  </tr>
  <tr>
    <td>Bag-of-Words (with BERT vocabulary)</td>
    <td>28,996</td>
    <td>Sparse</td>
    <td>44.75</td>
    <td>52.06</td>
    <td>54.78</td>
    <td>68.65</td>
    <td>60.59</td>
    <td>54.85</td>
    <td>57.87</td>
    <td>56.22</td>
  </tr>
  <tr>
    <td>QAEmb-MBQA (Benara et al., 2024)</td>
    <td>10,654</td>
    <td>0/1 Emb.</td>
    <td>39.40</td>
    <td>63.19</td>
    <td>57.68</td>
    <td>69.29</td>
    <td>63.18</td>
    <td>71.33</td>
    <td>72.33</td>
    <td>65.20</td>
  </tr>
  <tr>
    <td>CQG-MBQA (Sun et al., 2025)</td>
    <td>9,614</td>
    <td>0/1 Emb.</td>
    <td>69.21</td>
    <td>80.19</td>
    <td>73.91</td>
    <td>80.66</td>
    <td>78.30</td>
    <td>82.69</td>
    <td>78.21</td>
    <td>77.60</td>
  </tr>
  <tr>
    <td>LDIR (with SBERT<sub>new</sub>)</td>
    <td>200</td>
    <td>Dense</td>
    <td>74.44</td>
    <td>77.73</td>
    <td>73.58</td>
    <td>81.93</td>
    <td>77.98</td>
    <td>80.82</td>
    <td>78.42</td>
    <td>77.84</td>
  </tr>
  <tr>
    <td>LDIR (with SBERT<sub>new</sub>)</td>
    <td>500</td>
    <td>Dense</td>
    <td>72.51</td>
    <td>79.92</td>
    <td>74.93</td>
    <td>82.93</td>
    <td>78.44</td>
    <td>81.88</td>
    <td>79.34</td>
    <td>78.55</td>
  </tr>
  <tr>
    <td>LDIR (with ModernBERT<sub>emb</sub>)</td>
    <td>200</td>
    <td>Dense</td>
    <td>72.99</td>
    <td>79.63</td>
    <td>75.09</td>
    <td>82.52</td>
    <td>79.93</td>
    <td>82.61</td>
    <td>78.43</td>
    <td>78.74</td>
  </tr>
  <tr>
    <td>LDIR (with ModernBERT<sub>emb</sub>)</td>
    <td>500</td>
    <td>Dense</td>
    <td>72.30</td>
    <td>81.97</td>
    <td>75.48</td>
    <td>82.86</td>
    <td>79.88</td>
    <td>83.22</td>
    <td>80.72</td>
    <td>79.49</td>
  </tr>
  <tr>
    <td>LDIR (with AngIE)</td>
    <td>200</td>
    <td>Dense</td>
    <td><u>79.28</u></td>
    <td>82.16</td>
    <td><u>80.96</u></td>
    <td>84.64</td>
    <td><u>83.68</u></td>
    <td>85.32</td>
    <td>77.92</td>
    <td>81.99</td>
  </tr>
  <tr>
    <td>LDIR (with AngIE)</td>
    <td>500</td>
    <td>Dense</td>
    <td>78.85</td>
    <td><u>84.35</u></td>
    <td>80.93</td>
    <td><u>84.79</u></td>
    <td>83.61</td>
    <td><u>86.31</u></td>
    <td><u>80.85</u></td>
    <td><u>82.82</u></td>
  </tr>
</table>

Table 3: Comparison between baselines and LDIR on semantic textual similarity (STS) tasks. Among the interpretable embeddings, the best results in each column are in bold, and the second-best results are underlined.

tasks 2012-2016 (Agirre et al., 2012, 2013, 2014, 2015, 2016), STS Benchmark (Cer et al., 2017), and SICK-Relatedness (Marelli et al., 2014).

For retrieval tasks, we use 1% of the samples from the MS MARCO (Bajaj et al., 2016) development set, ArguAna (Wachsmuth et al., 2018), FiQA-2018 (FQA; Maia et al., 2018), NFCorpus (NFC; Boteva et al., 2016), SCIDocs (Cohan et al., 2020), and SciFact (Wadden et al., 2020).

For clustering tasks, we use TwentyNewsgroups (TNG), StackExchange (SE-P2P), Biorxiv (BR-P2P, BR-S2S), Medrxiv (MR-P2P, MR-S2S), and Reddit (RD-P2P) from massive text embedding benchmark (MTEB; Muennighoff et al., 2023).

Baselines. We use the baselines in Sun et al. (2025), including GloVe (Pennington et al., 2014), USE (Cer et al., 2018), BERT (Devlin et al., 2019), SBERT (Reimers and Gurevych, 2019), SimCSE (Gao et al., 2021), API-based text-embedding-ada-002 (OpenAI, 2022), Whitened-CSE (Zhuo et al., 2023), AngIE (Li and Li, 2024), Bag-of-Words, BM25 (Robertson et al., 2009), QAEmb-MBQA (Benara et al., 2024). We also use LLM2Vec (BehnamGhader et al., 2024), ModernBERT (Warner et al., 2024), and CQG-MBQA (Sun et al., 2025) as additional baselines.

Evaluations. We use the official evaluation suite of BEIR (Thakur et al., 2021) for MS MARCO and MTEB (Muennighoff et al., 2023) for other datasets. Spearman correlation, nDCG@10, and V-measure metrics are used as metrics for STS, retrieval, and clustering tasks, respectively.

4.2 Main Results

We apply different encoders ENC in Eq. 5 and sample different number of anchor texts (\( n = 200 \) and \( n = 500 \)) for comparison.

Semantic Textual Similarity. The results are shown in Table 3. LDIR demonstrates significant performance improvements among interpretable text embedding models while achieving competitive results compared to state-of-the-art black-box models. For example, LDIR outperforms the best interpretable embeddings CQG-MBQA (82.82 vs. 77.60), and also surpasses black-box models including SBERT<sub>new</sub> (80.02) and OpenAI’s text-embedding-ada-002 (82.31).

Comparing different dimensional settings, the 500-dimensional embeddings perform slightly better than the 200-dimensional ones. Among different backbone encoder models, AngIE gives the best results, while SBERT performs the worst, reflecting their differences in semantic encoding capabilities.

Retrieval. The results are shown in Table 4. LDIR achieves a nDCG@10 score of 46.68 with AngIE encoder and 500 dimensions, outperforming the traditional retrieval algorithm BM25 (43.23) and significantly outperforms the interpretable embed-
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Dim.</th>
    <th rowspan="2">Type</th>
    <th colspan="7">nDCG@10 (Retrieval)</th>
  </tr>
  <tr>
    <th>MS MARCO</th>
    <th>ArguAna</th>
    <th>FQA</th>
    <th>NFC</th>
    <th>SCI Docs</th>
    <th>SciFact</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td colspan="9">(Black-Box Embeddings)</td>
  </tr>
  <tr>
    <td>BERT<sub>base</sub> (Devlin et al., 2019)</td>
    <td>768</td>
    <td>Dense</td>
    <td>16.86</td>
    <td>28.29</td>
    <td>2.19</td>
    <td>4.30</td>
    <td>2.82</td>
    <td>13.34</td>
    <td>11.30</td>
  </tr>
  <tr>
    <td>SimCSE<sub>aug</sub> (Gao et al., 2021)</td>
    <td>768</td>
    <td>Dense</td>
    <td>44.63</td>
    <td>38.34</td>
    <td>9.84</td>
    <td>9.88</td>
    <td>5.50</td>
    <td>25.72</td>
    <td>22.32</td>
  </tr>
  <tr>
    <td>GloVe (Pennington et al., 2014)</td>
    <td>300</td>
    <td>Dense</td>
    <td>44.27</td>
    <td>36.30</td>
    <td>10.09</td>
    <td>13.87</td>
    <td>8.04</td>
    <td>29.58</td>
    <td>23.69</td>
  </tr>
  <tr>
    <td>SimCSE<sub>aug</sub> (Gao et al., 2021)</td>
    <td>768</td>
    <td>Dense</td>
    <td>47.86</td>
    <td>39.33</td>
    <td>10.41</td>
    <td>12.42</td>
    <td>7.53</td>
    <td>29.59</td>
    <td>24.52</td>
  </tr>
  <tr>
    <td>LLM2Vec (BehnamGhaader et al., 2024)</td>
    <td>4,096</td>
    <td>Dense</td>
    <td>63.48</td>
    <td>51.73</td>
    <td>28.56</td>
    <td>26.29</td>
    <td>10.39</td>
    <td>66.36</td>
    <td>41.14</td>
  </tr>
  <tr>
    <td>SBERT<sub>new</sub> (Reimers and Gurevych, 2019)</td>
    <td>768</td>
    <td>Dense</td>
    <td>88.74</td>
    <td>47.13</td>
    <td>37.27</td>
    <td>32.25</td>
    <td>21.82</td>
    <td>62.64</td>
    <td>48.31</td>
  </tr>
  <tr>
    <td>ModernBERT<sub>emb</sub> (Warner et al., 2024)</td>
    <td>1,024</td>
    <td>Dense</td>
    <td>87.39</td>
    <td>46.56</td>
    <td>45.18</td>
    <td>33.99</td>
    <td>21.40</td>
    <td>69.98</td>
    <td>50.75</td>
  </tr>
  <tr>
    <td>AngIE (Li and Li, 2024)</td>
    <td>1,024</td>
    <td>Dense</td>
    <td>90.43</td>
    <td>66.15</td>
    <td>44.84</td>
    <td>38.65</td>
    <td>22.98</td>
    <td>74.07</td>
    <td>56.19</td>
  </tr>
  <tr>
    <td>text-embedding-ada-002 (OpenAI, 2022)</td>
    <td>1,536</td>
    <td>Dense</td>
    <td>92.18</td>
    <td>58.05</td>
    <td>55.00</td>
    <td>42.07</td>
    <td>23.11</td>
    <td>77.77</td>
    <td>58.03</td>
  </tr>
  <tr>
    <td colspan="9">(Interpretable Embeddings)</td>
  </tr>
  <tr>
    <td>Bag-of-Words (with BERT vocabulary)</td>
    <td>28,996</td>
    <td>Sparse</td>
    <td>29.79</td>
    <td>34.25</td>
    <td>3.99</td>
    <td>21.51</td>
    <td>6.79</td>
    <td>47.36</td>
    <td>23.95</td>
  </tr>
  <tr>
    <td>BM25 (Robertson et al., 2009)</td>
    <td>N/A</td>
    <td>Sparse</td>
    <td>68.42</td>
    <td>49.28</td>
    <td>25.14</td>
    <td>32.08</td>
    <td>15.78</td>
    <td>68.70</td>
    <td>43.23</td>
  </tr>
  <tr>
    <td>QAEmb-MBQA (Benara et al., 2024)</td>
    <td>10,654</td>
    <td>0/1 Emb.</td>
    <td>40.51</td>
    <td>34.75</td>
    <td>8.23</td>
    <td>3.87</td>
    <td>3.74</td>
    <td>12.01</td>
    <td>17.19</td>
  </tr>
  <tr>
    <td>CQG-MBQA (Sun et al., 2025)</td>
    <td>9,614</td>
    <td>0/1 Emb.</td>
    <td>62.21</td>
    <td>47.75</td>
    <td>18.63</td>
    <td>9.74</td>
    <td>8.67</td>
    <td>32.80</td>
    <td>29.97</td>
  </tr>
  <tr>
    <td>LDIR (with ModernBERT<sub>emb</sub>)</td>
    <td>200</td>
    <td>Dense</td>
    <td>57.65</td>
    <td>33.62</td>
    <td>21.06</td>
    <td>16.05</td>
    <td>11.27</td>
    <td>49.59</td>
    <td>31.54</td>
  </tr>
  <tr>
    <td>LDIR (with ModernBERT<sub>emb</sub>)</td>
    <td>500</td>
    <td>Dense</td>
    <td>64.75</td>
    <td>37.91</td>
    <td>26.34</td>
    <td>18.06</td>
    <td>13.87</td>
    <td>52.13</td>
    <td>35.84</td>
  </tr>
  <tr>
    <td>LDIR (with SBERT<sub>new</sub>)</td>
    <td>200</td>
    <td>Dense</td>
    <td>78.24</td>
    <td>42.25</td>
    <td>25.94</td>
    <td>25.35</td>
    <td>18.34</td>
    <td>55.42</td>
    <td>40.92</td>
  </tr>
  <tr>
    <td>LDIR (with SBERT<sub>new</sub>)</td>
    <td>500</td>
    <td>Dense</td>
    <td>82.01</td>
    <td>44.89</td>
    <td>30.52</td>
    <td>26.06</td>
    <td>19.74</td>
    <td>56.25</td>
    <td>43.24</td>
  </tr>
  <tr>
    <td>LDIR (with AngIE)</td>
    <td>200</td>
    <td>Dense</td>
    <td><b>83.11</b></td>
    <td>57.11</td>
    <td>30.86</td>
    <td>27.06</td>
    <td>15.44</td>
    <td>59.85</td>
    <td>45.57</td>
  </tr>
  <tr>
    <td>LDIR (with AngIE)</td>
    <td>500</td>
    <td>Dense</td>
    <td>82.00</td>
    <td><b>60.10</b></td>
    <td><b>33.34</b></td>
    <td>24.67</td>
    <td>17.72</td>
    <td><b>62.26</b></td>
    <td><b>46.68</b></td>
  </tr>
</table>

Table 4: Comparison between baselines and LDIR on retrieval tasks. †: BM25 is a lexical retrieval algorithm and does not generate fixed-dimensional vectors.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Dim.</th>
    <th rowspan="2">Type</th>
    <th colspan="7">V-Measure (Clustering)</th>
  </tr>
  <tr>
    <th>TNG</th>
    <th>SE-P2P</th>
    <th>BR-P2P</th>
    <th>BR-S2S</th>
    <th>MR-P2P</th>
    <th>MR-S2S</th>
    <th>RD-P2P</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td colspan="9">(Black-Box Embeddings)</td>
  </tr>
  <tr>
    <td>SimCSE<sub>aug</sub> (Gao et al., 2021)</td>
    <td>768</td>
    <td>Dense</td>
    <td>23.21</td>
    <td>28.50</td>
    <td>24.90</td>
    <td>19.55</td>
    <td>23.60</td>
    <td>21.97</td>
    <td>45.14</td>
    <td>26.70</td>
  </tr>
  <tr>
    <td>GloVe (Pennington et al., 2014)</td>
    <td>300</td>
    <td>Dense</td>
    <td>25.83</td>
    <td>28.51</td>
    <td>29.27</td>
    <td>19.18</td>
    <td>26.12</td>
    <td>20.38</td>
    <td>35.82</td>
    <td>26.44</td>
  </tr>
  <tr>
    <td>BERT<sub>base</sub> (Devlin et al., 2019)</td>
    <td>768</td>
    <td>Dense</td>
    <td>23.35</td>
    <td>26.55</td>
    <td>30.12</td>
    <td>24.77</td>
    <td>26.09</td>
    <td>23.60</td>
    <td>43.32</td>
    <td>28.26</td>
  </tr>
  <tr>
    <td>SimCSE<sub>aug</sub> (Gao et al., 2021)</td>
    <td>768</td>
    <td>Dense</td>
    <td>34.86</td>
    <td>29.45</td>
    <td>30.15</td>
    <td>24.67</td>
    <td>26.25</td>
    <td>24.12</td>
    <td>47.74</td>
    <td>31.03</td>
  </tr>
  <tr>
    <td>LLM2Vec (BehnamGhaader et al., 2024)</td>
    <td>4,096</td>
    <td>Dense</td>
    <td>32.02</td>
    <td>36.36</td>
    <td>38.39</td>
    <td>31.31</td>
    <td>31.47</td>
    <td>27.87</td>
    <td>61.67</td>
    <td>37.01</td>
  </tr>
  <tr>
    <td>SBERT<sub>new</sub> (Reimers and Gurevych, 2019)</td>
    <td>768</td>
    <td>Dense</td>
    <td>47.47</td>
    <td>33.13</td>
    <td>36.99</td>
    <td>33.21</td>
    <td>34.25</td>
    <td>32.24</td>
    <td>54.80</td>
    <td>38.87</td>
  </tr>
  <tr>
    <td>ModernBERT<sub>emb</sub> (Warner et al., 2024)</td>
    <td>1,024</td>
    <td>Dense</td>
    <td>51.26</td>
    <td>34.73</td>
    <td>39.47</td>
    <td>34.67</td>
    <td>34.40</td>
    <td>31.63</td>
    <td>64.68</td>
    <td>41.55</td>
  </tr>
  <tr>
    <td>AngIE (Li and Li, 2024)</td>
    <td>1,024</td>
    <td>Dense</td>
    <td>51.72</td>
    <td>36.72</td>
    <td>39.38</td>
    <td>37.23</td>
    <td>33.22</td>
    <td>31.18</td>
    <td>65.35</td>
    <td>42.11</td>
  </tr>
  <tr>
    <td>text-embedding-ada-002 (OpenAI, 2022)</td>
    <td>1,536</td>
    <td>Dense</td>
    <td>58.14</td>
    <td>38.03</td>
    <td>36.53</td>
    <td>32.70</td>
    <td>31.27</td>
    <td>67.96</td>
    <td>43.07</td>
  </tr>
  <tr>
    <td colspan="9">(Interpretable Embeddings)</td>
  </tr>
  <tr>
    <td>Bag-of-Words (with BERT vocabulary)</td>
    <td>28,996</td>
    <td>Sparse</td>
    <td>8.52</td>
    <td>17.64</td>
    <td>4.70</td>
    <td>3.32</td>
    <td>11.39</td>
    <td>13.05</td>
    <td>15.67</td>
    <td>10.61</td>
  </tr>
  <tr>
    <td>QAEmb-MBQA (Benara et al., 2024)</td>
    <td>10,654</td>
    <td>0/1 Emb.</td>
    <td>36.72</td>
    <td>25.68</td>
    <td>24.66</td>
    <td>21.16</td>
    <td>25.53</td>
    <td>22.85</td>
    <td>46.57</td>
    <td>29.02</td>
  </tr>
  <tr>
    <td>CQG-MBQA (Sun et al., 2025)</td>
    <td>9,614</td>
    <td>0/1 Emb.</td>
    <td>40.00</td>
    <td>28.22</td>
    <td>34.88</td>
    <td>31.13</td>
    <td>31.02</td>
    <td>28.71</td>
    <td>54.40</td>
    <td>35.48</td>
  </tr>
  <tr>
    <td>LDIR (with ModernBERT<sub>emb</sub>)</td>
    <td>200</td>
    <td>Dense</td>
    <td>29.72</td>
    <td>37.03</td>
    <td>25.93</td>
    <td>21.06</td>
    <td>31.12</td>
    <td>27.65</td>
    <td>52.65</td>
    <td>32.17</td>
  </tr>
  <tr>
    <td>LDIR (with ModernBERT<sub>emb</sub>)</td>
    <td>500</td>
    <td>Dense</td>
    <td>30.18</td>
    <td>37.36</td>
    <td>28.40</td>
    <td>23.53</td>
    <td>32.42</td>
    <td>29.27</td>
    <td>52.77</td>
    <td>33.42</td>
  </tr>
  <tr>
    <td>LDIR (with AngIE)</td>
    <td>200</td>
    <td>Dense</td>
    <td>36.93</td>
    <td><b>40.60</b></td>
    <td>34.27</td>
    <td>31.61</td>
    <td>33.62</td>
    <td><b>58.88</b></td>
    <td>38.39</td>
  </tr>
  <tr>
    <td>LDIR (with AngIE)</td>
    <td>500</td>
    <td>Dense</td>
    <td>37.63</td>
    <td>38.82</td>
    <td>34.82</td>
    <td>33.05</td>
    <td>34.79</td>
    <td>33.63</td>
    <td>57.92</td>
    <td>38.67</td>
  </tr>
  <tr>
    <td>LDIR (with SBERT<sub>new</sub>)</td>
    <td>200</td>
    <td>Dense</td>
    <td>36.72</td>
    <td>39.44</td>
    <td>35.81</td>
    <td>30.81</td>
    <td>36.20</td>
    <td>33.98</td>
    <td>52.60</td>
    <td>37.94</td>
  </tr>
  <tr>
    <td>LDIR (with SBERT<sub>new</sub>)</td>
    <td>500</td>
    <td>Dense</td>
    <td><b>40.43</b></td>
    <td>37.86</td>
    <td><b>38.07</b></td>
    <td><b>33.32</b></td>
    <td><b>37.63</b></td>
    <td><b>35.63</b></td>
    <td>52.54</td>
    <td><b>39.36</b></td>
  </tr>
</table>

Table 5: Comparison between baselines and LDIR on clustering tasks.

dings baseline CQG-MBQA (29.97). This shows that, although the relatedness scores are computed through general and symmetric semantic similarity, LDIR can still encode the unsymmetrical features of relationship between the query and the document in retrieval tasks.

We also notice that the traditional sparse representation method BM25 performs well in some tasks, such as achieving nDCG@10 results of 32.08 and 68.70 on NFC and SciFact, respectively. This indicates that embeddings based on sparse representation are still useful in some specific retrieval scenarios and have better interpretability. Nevertheless, our best averaged results are still slightly better than BM25 algorithm.

Clustering. The results are shown in Table 5. Different from STS and retrieval tasks, LDIR with SBERT<sub>new</sub> encoder gives the best results (39.36), outperforming the CQG-MBQA baseline (35.48) by 10.9%. It also achieve comparable results to the embeddings by the backbone model SBERT<sub>new</sub> (38.87) and ModernBERT (41.55), validating the superior representation compactness of LDIR in unsupervised clustering. For ModernBERT and AngIE, the performance of LDIR with these encoders decrease, which may due to the trans-
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Dim.</th>
    <th rowspan="2">Type</th>
    <th rowspan="2">Perf.</th>
    <th colspan="7">Cognitive Load ↓</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>STS12</th>
    <th>STS13</th>
    <th>STS14</th>
    <th>STS15</th>
    <th>STS16</th>
    <th>STS-B</th>
    <th>SICK-R</th>
  </tr>
  <tr>
    <td>Bag-of-Words (with BERT vocabulary)</td>
    <td>28,996</td>
    <td>Sparse</td>
    <td>56.22</td>
    <td>8</td>
    <td>4</td>
    <td>6</td>
    <td>5</td>
    <td>8</td>
    <td>7</td>
    <td>6</td>
    <td>6</td>
  </tr>
  <tr>
    <td>QAEmb-MBQA (Beruara et al., 2024)</td>
    <td>10,654</td>
    <td>0/1 Emb.</td>
    <td>65.20</td>
    <td>1,626</td>
    <td>1,571</td>
    <td>1,625</td>
    <td>1,443</td>
    <td>1,577</td>
    <td>1,408</td>
    <td>1,018</td>
    <td>1,467</td>
  </tr>
  <tr>
    <td>CQG-MBQA (Sun et al., 2025)</td>
    <td>9,614</td>
    <td>0/1 Emb.</td>
    <td>77.60</td>
    <td>481</td>
    <td>439</td>
    <td>458</td>
    <td>426</td>
    <td>478</td>
    <td>446</td>
    <td>413</td>
    <td>449</td>
  </tr>
  <tr>
    <td>CQG-MBQA (Sun et al., 2025)<sup>†</sup></td>
    <td>11,000</td>
    <td>0/1 Emb.</td>
    <td>75.24</td>
    <td>48</td>
    <td>44</td>
    <td>43</td>
    <td>41</td>
    <td>44</td>
    <td>42</td>
    <td>37</td>
    <td>43</td>
  </tr>
  <tr>
    <td>LDIR (with AnglE, binarization)</td>
    <td>500</td>
    <td>0/1 Emb.</td>
    <td>75.87</td>
    <td>15</td>
    <td>13</td>
    <td>14</td>
    <td>12</td>
    <td>14</td>
    <td>13</td>
    <td>14</td>
    <td>14</td>
  </tr>
  <tr>
    <td>LDIR (with AnglE, binarization)</td>
    <td>200</td>
    <td>0/1 Emb.</td>
    <td>71.34</td>
    <td>6</td>
    <td>5</td>
    <td>5</td>
    <td>4</td>
    <td>5</td>
    <td>5</td>
    <td>5</td>
    <td>5</td>
  </tr>
  <tr>
    <td>LDIR (with AnglE, no binarization)</td>
    <td>500</td>
    <td>Dense</td>
    <td>82.82</td>
    <td>50</td>
    <td>52</td>
    <td>48</td>
    <td>47</td>
    <td>31</td>
    <td>43</td>
    <td>38</td>
    <td>47</td>
  </tr>
  <tr>
    <td>LDIR (with AnglE, no binarization)</td>
    <td>200</td>
    <td>Dense</td>
    <td>81.99</td>
    <td>15</td>
    <td>16</td>
    <td>15</td>
    <td>14</td>
    <td>15</td>
    <td>13</td>
    <td>12</td>
    <td>14</td>
  </tr>
</table>

Table 6: Evaluation on cognitive load using STS tasks. †: The corresponding performance and cognitive load with 1000 dimensional embeddings are not provided and we estimate them from the curves in Sun et al. (2025).

formation of relative representations in the low-dimensional space leads to a weakening of clustering characteristics for the high-dimensional representation space of these two embeddings.

4.3 Evaluation on Cognitive Load

We follow Sun et al. (2025) by using the “cognitive load” for measuring the interpretability. Formally, cognitive load is defined as the inner product of two binary embedding vectors \( \mu \) and \( v \):

\[
\text{cognitive load} = \langle \mu, v \rangle = \sum_{i=1}^{m} \mu_i v_i,
\]

where a smaller cognitive load indicates stronger model interpretability, which means we only need to focus on fewer but more representative anchors to understand the text representation. This metric is originally used for binary embeddings and we transfer LDIR to binary embeddings by setting top-k (\( k = 25 \) according to the average) high values as 1 and the other values in different dimensions as 0.

The results are shown in Table 6. Firstly, bag-of-words gives a very cognitive load with 6, showing the highest interpretability. CQG-MBQA exhibits a cognitive load of 449 with 9000 dimensions, which reduces to 43 with 1000 dimensions, with a sacrifice of overall performance. Our LDIR with dimensions of 500 and 200, giving cognitive load values of 14 and 5 after binarization, respectively, showing a favorable performance (71.34~75.87) while maintaining a low cognitive load.

For LDIR without binarization, we find that it gives 47 and 14 cognitive load directly through Eq. 7, although this metric is not applicable to non-binary embeddings. Nevertheless, how to define the “interpretability” for text embeddings and design more suitable metrics that reflects interpretability for dense text embeddings still remains a valuable and open question.

![Comparison of different anchor texts sampling methods (a) and different settings of anchor texts length (b). The horizontal axis represents the different number (20~800) of different anchor texts.](page_1012_1012_388_246.png)

Figure 3: Comparison of different anchor texts sampling methods (a) and different settings of anchor texts length (b). The horizontal axis represents the different number (20~800) of different anchor texts.

5 Analyses and Discussions

In our main experiments, we remove the long texts in corpus and use FPS to extract the anchor texts. We analysis the impact of sampling methods (Section 5.1) and the length of anchor texts (Section 5.2) below. Then we discuss how to obtain a more interpretable dense embeddings with the calculation of fine-grained relatedness (Section 5.3), compare different metrics on relatedness (Section 5.4). Finally, we show a case study (Section 5.5) and discuss the usage of relative representation based embeddings (Section 5.6).

5.1 Anchor Texts Sampling Methods

We compare three anchor texts sampling methods: FPS, uniform sampling, and K-Means . The results are shown in Figure 3(a). The trends of the three methods across different numbers of anchors were similar in general. FPS generally performed the best, with its performance converging with uniform sampling at 800 dimensions, while K-Means performed relatively poorly. The advantage of FPS in the low-dimensional range stems from its maximum diversity sampling strategy. The convergence of uniform sampling with FPS at 800 dimensions may be related to the properties of high-dimensional spaces when the dimensionality is sufficiently high, random uniform sampling can
approximately cover the boundary regions of the semantic space. Notably, the K-Means method, constrained by the high similarity of cluster centers, resulted in insufficient semantic diversity in the anchor set compared to the other two methods, leading to suboptimal performance.

5.2 The Impact of Length of Anchor Texts

We investigate the impact of different length of anchor texts, including short (less than 20 tokens), medium (20~100 tokens), and long (longer than 100 tokens) texts. The results are shown in Figure 3(b). We observe that for different settings of length, the performance improved rapidly using less than 200 anchor texts, after which the improvement is largely limited. The performance gap between short texts and medium texts narrowed and almost converged with 240 or more anchor texts, with both significantly outperforming long texts. We hypothesize that this may be attributable to the more intricate semantics and the potential inclusion of greater noise within longer texts in the corpus.

5.3 Fine-grained Relatedness Calculation

In Eq. 5, the relatedness score is rather general, which reflects the overall semantic correlation between anchor text \( a_j \) and text \( t \). We further propose calculate more fine-grained relatedness score by applying different encoders tuned on different tasks, as shown in Figure 4 (bottom).

<table>
  <tr>
    <th>Model</th>
    <th>STS12</th>
    <th>STS13</th>
    <th>STS14</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>Bag-of-Words (with BERT vocabulary)</td>
    <td>44.75</td>
    <td>52.06</td>
    <td>54.78</td>
    <td>50.53</td>
  </tr>
  <tr>
    <td>QAEmb-MBQA (Boura et al., 2024)</td>
    <td>59.40</td>
    <td>63.19</td>
    <td>57.68</td>
    <td>60.09</td>
  </tr>
  <tr>
    <td>CQG-MBQA (Sun et al., 2023)</td>
    <td>69.21</td>
    <td>80.19</td>
    <td>75.91</td>
    <td>74.44</td>
  </tr>
  <tr>
    <td>LDIR (with fine-grained relatedness)</td>
    <td><b>57.88</b></td>
    <td><b>67.33</b></td>
    <td><b>58.96</b></td>
    <td><b>61.39</b></td>
  </tr>
</table>

Table 7: The results of LDIR using the concatenated 800 dimensional embeddings with fine-grained relatedness.

<table>
  <tr>
    <th>Metrics/Distance</th>
    <th>Type</th>
    <th>STS Results in Avg.</th>
  </tr>
  <tr>
    <td>Edit Distance</td>
    <td>surface-based</td>
    <td>17.99</td>
  </tr>
  <tr>
    <td>Jaccard</td>
    <td>surface-based</td>
    <td>22.34</td>
  </tr>
  <tr>
    <td>Sokalsneath</td>
    <td>vector-based & binary</td>
    <td>79.04</td>
  </tr>
  <tr>
    <td>Jaccard</td>
    <td>vector-based & binary</td>
    <td>79.49</td>
  </tr>
  <tr>
    <td>Hamming</td>
    <td>vector-based & binary</td>
    <td>80.00</td>
  </tr>
  <tr>
    <td>Dice</td>
    <td>vector-based & binary</td>
    <td>80.07</td>
  </tr>
  <tr>
    <td>Chebyshev</td>
    <td>vector-based & dense</td>
    <td>79.33</td>
  </tr>
  <tr>
    <td>Euclidean</td>
    <td>vector-based & dense</td>
    <td>80.76</td>
  </tr>
  <tr>
    <td>Manhattan</td>
    <td>vector-based & dense</td>
    <td>81.19</td>
  </tr>
  <tr>
    <td>Cosine (this work)</td>
    <td>vector-based & dense</td>
    <td><b>82.82</b></td>
  </tr>
</table>

Table 8: The results of LDIR using different surface-based and vector-based metrics.

We use tasks in the general language understanding evaluation (GLUE; Wang et al., 2019) benchmark, which tests the text understanding ability from different perspectives. In particular, we select CoLA, STS-B, SST-2, and QNLI subtasks such that the fine-tuned models can encode syntactic, semantic, sentiment, and natural language inference features in fine-grained levels. Different models are fine-tuned on the training data and serve as encoders for calculating relatedness. The relatedness scores via 200 anchor texts are concatenated as a final 800-dimensional dense embeddings.

The results on STS12, STS13, and STS14 are shown in Table 7. We find that LDIR with fine-grained relatedness still outperforms the QAEmb-MBQA baseline (61.39 vs. 60.09). However, its overall performance decreases in STS tasks, under-performing the CQG-MBQA baseline (61.39 vs. 74.44). This show that, although the fine-grained relatedness scores have improved interpretability, their overall performance will not be enhanced in general semantic similarity tasks, which also reflects the trade-off between performance and interpretability, as discussed in Table 2.

5.4 Surface-based and Vector-based Metrics

We further try surface-based metrics as well as other various vector-based metrics for calculating the relatedness, and the results for 500 dimensional embeddings are shown in Table 8. We find that surface-based distances cannot effectively represent semantic relationships, where the results are very poor. On the other hand, other vector-based distance (even when applied to LDIR after binarization) consistently show close results, despite differences in distances calculation against the cosine distances in downstream tasks.
<table>
  <tr>
    <th>Anchor Text #1</th>
    <td>A novel, high-performing architecture for end-to-end named entity recognition and coreference resolution in main</td>
  </tr>
  <tr>
    <th>Anchor Text #2</th>
    <td>Predicting affective states experienced through an emotive-aided procedure from annotator’s mixed-initiative dialogue.</td>
  </tr>
  <tr>
    <th>Anchor Text #3</th>
    <td>Planar High-Gain Dielectric-Loaded Ampiual Linearly Tapered Slot Antenna for E- and U-Band Gigahertz Point-to-Point Wireless Services</td>
  </tr>
  <tr>
    <th>Anchor Text #4</th>
    <td>Design of a Monopulse Antenna Using a Dual V-Type Linearly Tapered Slot Antenna (DVLYSA)</td>
  </tr>
</table>

<table>
  <tr>
    <th>Texts (1) and LDIR Embeddings (4)</th>
    <th>Dim #1</th>
    <th>Dim #2</th>
    <th>Dim #3</th>
    <th>Dim #4</th>
  </tr>
  <tr>
    <th>Text A</th>
    <td>Morphological Embeddings for Named Entity Recognition in Morphologically Rich Languages</td>
    <td>0.7554</td>
    <td>0.5312</td>
    <td>0.3801</td>
    <td>0.3607</td>
  </tr>
  <tr>
    <th>Text B</th>
    <td>Design Approach to a Novel Dual-Mode Wideband Circular Sector Patch Antenna</td>
    <td>0.3941</td>
    <td>0.3373</td>
    <td>0.6874</td>
    <td>0.6661</td>
  </tr>
</table>

Table 9: LDIR embeddings of text A and text B. We only show values on four dimensions for brevity.

5.5 Case Study

Table 9 shows a case of four-dimensional LDIR embeddings of two title texts from SCIDocs. Text A has the highest relevance to anchor text 1, while text B is more related to anchor texts 3 and 4, resulting in the difference in their LDIR embeddings.

5.6 The Usage of Relative Representation

Our approach is based on transforming relative representations using an existing backbone model, where we consider its potential applications here briefly. For instance, some studies have shown that it is possible to recover original training data or sensitive information from embeddings themselves (Morris et al., 2023; Li et al., 2023; Huang et al., 2024). Therefore, directly exposing embedding interfaces may lead to data privacy or leakage issues. Consider a scenario where an embedding provider (such as text-embedding-Ada series by OpenAI) does not want directly provide users with high-dimensional raw embeddings outputs. Instead, they could offer low-dimensional relative representation vectors based on anchor text (which can be provided by the provider or defined by the user). This approach 1) can help avoid some data privacy or leakage concerns; 2) performs well in certain downstream applications; 3) allows for obtaining more text representations with the same storage and computational cost with low-dimensionality; 4) provides interpretability and reliability with the flexibility of anchor text.

6 Conclusion

We propose LDIR, a low-dimensional dense and interpretable text embeddings with relative representations. The numerical values in LDIR represent the correlation between the text and the anchor texts automatically obtained through farthest point sampling, which improves the expressiveness at a low cost compared with QA-based 0/1 embeddings. Across multiple tasks and datasets, LDIR demonstrates better semantic expressiveness compared to multiple interpretable embedding baselines. We also discuss the need for more suitable interpretability metrics and how to further enhance the interpretability of LDIR, which can serve as our future work for interpretable text embeddings.

Limitations

Interpretable text embeddings is a relatively new research topic and the baselines of interpretable text embeddings we used is limited. Also, as we mentioned, there is still room for improvement in the interpretability of our embeddings. According to our approach, the anchor texts are sampled from the entire corpus, and we can further optimize these anchor texts for different downstream tasks for better task-related expressiveness. Additionally, the human evaluation and application scenarios of interpretable text embeddings can be further explored, which is also crucial for explainable AI.

Acknowledgments

This work was supported in parts by NSFC (62306161, U21B2023), Shenzhen Science and Technology Program (KJZD20240903100022028, KQTD20210811090044003, RCJC20200714114435012), and Scientific Development Funds from Shenzhen University.

References

Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Iñigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, German Rigau, Larraitz Uria, and Janyce Wiebe. 2015. SemEval-2015 task 2: Semantic textual similarity, English, Spanish and pilot on interpretability. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 252–263, Denver, Colorado. Association for Computational Linguistics.

Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2014. SemEval-2014 task 10: Multilingual semantic textual similarity. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 81–91, Dublin, Ireland. Association for Computational Linguistics.

Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2016. SemEval-2016
task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 497–511, San Diego, California. Association for Computational Linguistics.

Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012. SemEval-2012 task 6: A pilot on semantic textual similarity. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 385–393, Montréal, Canada. Association for Computational Linguistics.

Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo. 2013. *SEM 2013 shared task: Semantic textual similarity. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity, pages 32–43, Atlanta, Georgia, USA. Association for Computational Linguistics.

Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268.

Yamini Bansal, Preetum Nakkiran, and Boaz Barak. 2021. Revisiting model stitching to compare neural representations. Advances in neural information processing systems, 34:225–236.

Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. 2024. LLM2vec: Large language models are secretly powerful text encoders. In First Conference on Language Modeling.

Vinamra Benara, Chandan Singh, John Xavier Morris, Richard Antonello, Ion Stoica, Alexander Huth, and Jianfeng Gao. 2024. Crafting interpretable embeddings for language neuroscience by asking LLMs questions. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146.

Vera Boteva, Demian Gholipour, Artem Sokolov, and Stefan Riezler. 2016. A full-text learning to rank dataset for medical information retrieval. In Advances in Information Retrieval, pages 716–722, Cham. Springer International Publishing.

Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14, Vancouver, Canada. Association for Computational Linguistics.

Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. 2018. Universal sentence encoder for English. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 169–174, Brussels, Belgium. Association for Computational Linguistics.

Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel Weld. 2020. SPECTER: Document-level representation learning using citation-informed transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2270–2282, Online. Association for Computational Linguistics.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Kawin Ethayarajh. 2019. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 55–65, Hong Kong, China. Association for Computational Linguistics.

Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Yu-Hsiang Huang, Yuche Tsai, Hsiang Hsiao, Hong-Yi Lin, and Shou-De Lin. 2024. Transferable embedding inversion attack: Uncovering privacy risks in text embeddings without model queries. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4193–4205, Bangkok, Thailand. Association for Computational Linguistics.
Vladimir Karpukhin, Barlas Ouzg, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Online. Association for Computational Linguistics.

Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2025. NV-embed: Improved techniques for training LLMs as generalist embedding models. In The Thirteenth International Conference on Learning Representations.

Yibin Lei, Di Wu, Tianyi Zhou, Tao Shen, Yu Cao, Chongyang Tao, and Andrew Yates. 2024. Meta-task prompting elicits embeddings from large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10141–10157, Bangkok, Thailand. Association for Computational Linguistics.

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474.

Haoran Li, Mingshi Xu, and Yangqiu Song. 2023. Sentence embedding leaks more information than you expect: Generative embedding inversion attack to recover the whole sentence. In Findings of the Association for Computational Linguistics: ACL 2023, pages 14022–14040, Toronto, Canada. Association for Computational Linguistics.

Xianming Li and Jing Li. 2024. AoE: Angle-optimized embeddings for semantic textual similarity. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1825–1839, Bangkok, Thailand. Association for Computational Linguistics.

Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. 2018. Www’18 open challenge: Financial opinion mining and question answering. In Companion Proceedings of the The Web Conference 2018, WWW ’18, page 1941–1942, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.

Mayug Maniparambil, Raiymbek Akshulakov, Yasser Abdelaziz Dahou Djilali, Mohamed El Amine Seddik, Sanath Narayan, Karttikeya Mangalam, and Noel E. O’Connor. 2024. Do vision and language encoders represent the world similarly? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14334–14343.

Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014. A SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 216–223, Reykjavik, Iceland. European Language Resources Association (ELRA).

Tomas Mikolov. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 3781.

John Morris, Volodymyr Kuleshov, Vitaly Shmatikov, and Alexander Rush. 2023. Text embeddings reveal (almost) as much as text. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12448–12460, Singapore. Association for Computational Linguistics.

Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodola. 2023. Relative representations enable zero-shot latent space communication. In The Eleventh International Conference on Learning Representations.

Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. MTEB: Massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2014–2037, Dubrovnik, Croatia. Association for Computational Linguistics.

Antonio Norelli, Marco Fumero, Valentino Maiorca, Luca Moschella, Emanuele Rodola, and Francesco Locatello. 2023. Asif: Coupled data turns unimodal models to multimodal without training. Advances in Neural Information Processing Systems, 36:15303–15319.

OpenAI. 2022. Openai api embeddings.

Ajay Patel, Delip Rao, Ansh Kothary, Kathleen McKown, and Chris Callison-Burch. 2023. Learning interpretable style embeddings via prompting LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15270–15290, Singapore. Association for Computational Linguistics.

Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.

Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.

Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389.
Yiqun Sun, Qiang Huang, Yixuan Tang, Anthony Kum Hoe Tung, and Jun Yu. 2025. A general framework for producing interpretable semantic text embeddings. In The Thirteenth International Conference on Learning Representations.

Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajijwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Henning Wachsmuth, Shahbaz Syed, and Benno Stein. 2018. Retrieval of the best counterargument without prior topic knowledge. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 241–251, Melbourne, Australia. Association for Computational Linguistics.

David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or fiction: Verifying scientific claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7534–7550, Online. Association for Computational Linguistics.

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations.

Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, et al. 2024. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference. arXiv preprint arXiv:2412.13663.

Wenjie Zhuo, Yifan Sun, Xiaohan Wang, Linchao Zhu, and Yi Yang. 2023. WhitenedCSE: Whitening-based contrastive learning of sentence embeddings. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12135–12148, Toronto, Canada. Association for Computational Linguistics.

A Checkpoints of Baseline Models

In Table 10, we list the checkpoints of baseline models we used in experiments.
<table>
  <tr>
    <th>Model</th>
    <th>Checkpoint</th>
  </tr>
  <tr>
    <td>BERT</td>
    <td>https://huggingface.co/google-bert/bert-base-uncased</td>
  </tr>
  <tr>
    <td>GloVe</td>
    <td>https://huggingface.co/sentence-transformers/average_word_embeddings_glove.6B.300d</td>
  </tr>
  <tr>
    <td>SimCSE<sub>unsup</sub></td>
    <td>https://huggingface.co/princeton-nlp/sup-simcse-bert-base-uncased</td>
  </tr>
  <tr>
    <td>SimCSE<sub>sup</sub></td>
    <td>https://huggingface.co/princeton-nlp/unsup-simcse-bert-base-uncased</td>
  </tr>
  <tr>
    <td>WhitenedCSE</td>
    <td>https://huggingface.co/Supstar2h/whitenedcse-bert-base</td>
  </tr>
  <tr>
    <td>SBERT<sub>ori</sub></td>
    <td>https://huggingface.co/sentence-transformers/all-mnpt-base-v1</td>
  </tr>
  <tr>
    <td>SBERT<sub>new</sub></td>
    <td>https://huggingface.co/sentence-transformers/all-mnpt-base-v2</td>
  </tr>
  <tr>
    <td>ModernBERT<sub>mb</sub></td>
    <td>https://huggingface.co/lightnail/modernbert-embed-large</td>
  </tr>
  <tr>
    <td>text-embedding-ada-002</td>
    <td>https://openai.com/index/new-and-improved-embedding-model/</td>
  </tr>
  <tr>
    <td>AngIE</td>
    <td>https://huggingface.co/WhereIsAI/UAE-Large-VI</td>
  </tr>
  <tr>
    <td>LLM2Vec</td>
    <td>https://huggingface.co/McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mnpt-unsup-simcse</td>
  </tr>
  <tr>
    <td>BM25</td>
    <td>https://github.com/xhluca/bm25s</td>
  </tr>
</table>

Table 10: Checkpoints of baseline models we used in experiments.

Metareview score: 3
Metareview: This paper introduces LDIR, a novel approach for generating low-dimensional, dense, and interpretable text embeddings. LDIR represents text semantically with each dimension linked to the relevance of specific anchor texts. Experiments show LDIR achieves performance comparable to black-box models but with reduced dimensionality and realized interpretability.
Summary of reasons to publish: Novel approach to interpretability of text embeddings. These embeddings have lower dimensionality than prior work. Only a small hit in performance compared to blackbox baseline models.