A General Framework for Producing Interpretable Semantic Text Embeddings

## Abstract

Abstract Semantic text embedding is essential to many tasks in Natural Language Processing (NLP).
While black-box models are capable of generating high-quality embeddings, their lack of interpretability limits their use in tasks that demand transparency.
Recent approaches have improved interpretability by leveraging domain-expert-crafted or LLM-generated questions, but these methods rely heavily on expert input or well-prompt design, which restricts their generalizability and ability to generate discriminative questions across a wide range of tasks.
To address these challenges, we introduce CQG-MBQA (Contrastive Question Generation - Multi-task Binary Question Answering), a general framework for producing interpretable semantic text embeddings across diverse tasks.
Our framework systematically generates highly discriminative, low cognitive load yes/no questions through the CQG method and answers them efficiently with the MBQA model, resulting in interpretable embeddings in a cost-effective manner.
We validate the effectiveness and interpretability of CQG-MBQA through extensive experiments and ablation studies, demonstrating that it delivers embedding quality comparable to many advanced black-box models while maintaining inherently interpretability. Additionally, CQG-MBQA outperforms other interpretable text embedding methods across various downstream tasks.

## 1 Introduction

Text embedding is a cornerstone of Natural Language Processing (NLP), transforming texts—whether sentences, paragraphs, or full documents—into embedding vectors that capture their semantic meaning.
In semantic embedding spaces, the similarity between texts is represented by the proximity of their embedding vectors, typically measured using distance measures like Euclidean distance, cosine distance, or inner product.
The closer the vectors, the more semantically similar the texts.
These embeddings are foundational to many downstream NLP tasks, including Semantic Textual Similarity (STS) (Agirre et al., ; ), Information Retrieval (Karpukhin et al., ; Thakur et al., ), Clustering (Aggarwal & Zhai, ), and more recently, Retrieval Augmented Generation (RAG) (Lewis et al., ; Guu et al., ; Asai et al., ).

Black-box text embedding methods, such as Sentence-BERT (Reimers & Gurevych, ), SimCSE (Gao et al., ), WhitenedCSE (Zhuo et al., ), and AnglE (Li & Li, ), excel at generating high-quality embeddings by training on vast amounts of data.
These models are highly effective at capturing semantic similarities, making them indispensable for a variety of NLP tasks (Muennighoff et al., ).
However, their black-box nature leaves the embeddings opaque to human users.
These models do not provide insight into why certain texts are deemed similar, which becomes problematic for tasks that require transparency, especially in applications involving high-stakes decision-making, such as legal and medical domains, or in cases requiring explanations for regulatory compliance.

Interpretability in machine learning is the ability of humans to understand the reasoning behind a model’s results (Miller, ), which is essential not only for building trust and ensuring safety but also for detecting biases and debugging models (Molnar, ).
Recent advances have enhanced interpretability by leveraging inherently interpretable models such as Decision Tree (Breiman et al., ) and Generalized Additive Models (Hastie & Tibshirani, ), as well as model-agnostic methods like LIME (Ribeiro et al., ) and SHAP (Lundberg & Lee, ).
However, these interpretable approaches lose effectiveness when applied on top of non-interpretable features generated by black-box text embedding models.
Consequently, the challenge remains: how can we create interpretable text embeddings without sacrificing performance?

Recent efforts have sought to address the challenge of creating interpretable embeddings by using questions as interpretable dimensions.
For instance, ChiLL (McInerney et al., ) employs yes/no questions crafted by domain experts to classify patient clinical notes, but its reliance on costly expert annotation limits its generalizability to different datasets.
QAEmb (Benara et al., ) advances this concept by using task-specific prompts with examples to automatically generate yes/no questions via Large Language Models (LLMs), achieving notable success in the fMRI prediction task (Huth et al., ; LeBel et al., ; Tang et al., ).

Nonetheless, QAEmb requires meticulously crafted prompts and uses six distinct prompt templates to generate questions for the fMRI prediction task, which complicates its usage in general settings due to the need for prompt engineering expertise.
Furthermore, this example-based question generation approach often produces generic, less discriminative questions, limiting its effectiveness in broader applications.
Given the importance of question quality in creating effective interpretable embeddings, there is a pressing need for a systematic approach that can automatically generate meaningful and discriminative questions across various text embedding tasks.

To address this gap, we introduce CQG-MBQA (Contrastive Question Generation - Multi-task Binary Question Answering), a *general* framework for producing interpretable semantic text embedding, which matches the performance of many black-box models and surpasses existing interpretable baselines across various text embedding tasks.
CQG-MBQA harnesses contrastive learning principles to prompt LLMs to generate highly discriminative binary yes/no questions, which form the dimensions of the embedding space.
These questions not only capture the semantic nuances between texts but also offer human-interpretable insights.
The main contributions of this work are as follows:

- •
We propose CQG-MBQA, the first general framework that tackles the challenge of generating interpretable text embeddings for a broad range of tasks, offering a practical and scalable solution for text representation.
- •
Our Contrastive Question Generation (CQG) method produces highly discriminative questions that offer high interpretability while minimizing cognitive load for users, ensuring that the semantic relationships between texts can be easily understood.
- •
The Multi-task Binary Question Answering (MBQA) model processes these binary questions efficiently at scale, significantly reducing the inference costs typically associated with LLMs, making the framework cost-effective for real-world applications.
- •
We validate the effectiveness of CQG-MBQA through extensive experiments and ablation studies, demonstrating its robustness and practical applicability across multiple benchmarks and downstream tasks.

## 2 Related Work

Text embedding is a fundamental NLP task that transforms texts into vector representations that capture their semantic meanings.
Generally, it can be classified into two types: black-box embedding and interpretable embedding.

#### Black-box Embedding

Early methods for text embedding, such as GloVe (Pennington et al., ) and Word2Vec (Mikolov et al., ), typically pool word embeddings to create low-dimensional semantic representations.
However, these methods, which rely on individual word embeddings, often fail to capture the full context of a text.
For example, the sentences “Most people in the world like Apple.” and “Most people in the world do not like Apple.” share high lexical overlap but have opposite meanings, highlighting the limitations of such methods, which struggle to capture deeper semantic differences beyond surface-level word similarity.

To produce context-aware text embeddings, Universal Sentence Encoder (USE) (Cer et al., ) employs a transformer model (Vaswani et al., ) trained on a combination of unsupervised tasks and supervised fine-tuning using the Stanford Natural Language Inference (SNLI) corpus (Bowman et al., ).
BERT (Devlin et al., ), a transformer network pre-trained on large-scale unlabeled text, can generate sentence embeddings by pooling its output representations.
Subsequent models have further refined BERT using contrastive learning and other semantic-related objectives.
For instance, Sentence-BERT (SBERT) (Reimers & Gurevych, ) pioneers the Siamese network structure for Semantic Textual Similarity (STS), while SimCSE (Gao et al., ) develop a contrastive learning framework for both unsupervised and supervised settings.
WhitenedCSE (Zhuo et al., ) enhances embedding uniformity and alignment with shuffled group whitening, and AnglE (Li & Li, ) optimizes angle differences to overcome cosine similarity limitations.
Despite these advances, black-box models produce embeddings that are opaque and difficult to interpret. In this work, we target generating interpretable dimensions for text embedding.

#### Interpretable Embedding

The challenge of creating interpretable embeddings has been persisted, especially with the rise of dense word embeddings.
Early efforts focus on transforming word embeddings to improve interpretability.
Jha et al. () apply categorical knowledge in the biomedical domain to convert pre-trained embeddings into interpretable dimensions, while Senel et al. () quantify interpretability by analyzing latent semantic structures.
Models like SPINE (Subramanian et al., ) employ auto-encoders to create interpretable embeddings from non-interpretable ones like GloVe (Pennington et al., ) and Word2Vec (Mikolov et al., ), and Word2Sense (Panigrahi et al., ) creates interpretable dimensions based on specific word senses.

Despite progress, developing context-aware, interpretable dimensions remains difficult.
Recent research has shifted towards indirectly understanding embedding spaces.
For instance, Lee et al. () introduce token pair contribution heatmaps to enhance interpretability in sentence similarity, while Simhi & Markovitch () propose transforming embedding spaces into comprehensible conceptual representations.
Recent advancements like ChiLL (McInerney et al., ) generate interpretable binary features from health records by querying pre-trained LLMs with *expert-crafted* yes/no questions for patient classification.
QAEmb (Benara et al., ) extends this by prompting LLMs to *automatically* generate questions using examples of texts and questions, demonstrating its efficacy in the fMRI prediction task.
Inspired by QAEmb, we propose a cost-effective framework that generates high-quality questions and uses them as interpretable dimensions for text embedding.

## 3 Interpretable Text Embedding Framework

We introduce an interpretable text embedding framework named CQG-MBQA, which uses questions as interpretable dimensions to represent text.
By posing a set of carefully designed yes/no questions about a given text, the answers form an interpretable embedding vector that captures the text’s core semantics.
For instance, consider three questions: “Is the article about AI?”, “Is the article about sports?”, and “Is the article about food?”.
For the text “Apple is a technology company.”, querying an LLM yields the answers: [“yes”, “no”, “no”], resulting in the embedding vector $[1,0,0]$, which reflects the text’s key features.
Applying the same set of questions across all texts in a corpus produces a consistent embedding matrix that encodes the semantic information of the entire dataset.

As shown in Figure 1, the CQG-MBQA framework consists of two phases: question generation and question answering.
To generate high-quality, discriminative questions, we develop a method called Contrastive Question Generation (CQG), which harnesses pre-trained dense text embedding models and generative LLMs for question generation. Details of this method are outlined in Section 3.1.
Once the questions are generated, their corresponding answers form the text’s embedding vector.
Yet, generating answers through LLMs at scale is both time-consuming and expensive. To address this, we propose a Multi-task Binary Question Answering (MBQA) model. Trained with a multi-task binary classification objective, this model can generate interpretable embeddings efficiently, requiring far fewer LLM API calls.
Further details on this model are provided in Section 3.2.

Figure: Figure 1: An overview of the CQG-MBQA framework.
Refer to caption: x1.png

### 3.1 Question Generation

#### Motivations

To effectively represent texts using binary question answers, we need highly discriminative questions that capture the subtle semantic differences between texts in the corpus.
Existing methods, such as QAEmb (Benara et al., ), generate questions by prompting LLMs with dataset descriptions, example texts, and sample questions.
However, this example-based approach presents two significant limitations:

- (1)
Insufficient Specificity:
The generated questions are often too generic, resulting in embeddings that fail to capture fine semantic nuances.
- (2)
Interpretability Issue:
A significant proportion of questions consistently yield simple “yes” answers, which can make the resulting embeddings more challenging to interpret and analyze.

These limitations reduce the effectiveness of the embeddings in capturing fine-grained semantic differences, which in turn hinders their practical utility in downstream tasks.
To overcome these challenges, we introduce Contrastive Question Generation (CQG), a novel method that leverages the creative potential of LLMs to generate more discriminative questions.

#### Contrastive Question Generation (CQG)

The CQG method applies contrastive learning principles, using *positive*, *hard negative*, and *easy negative* samples to guide LLMs in generating high-quality questions.
These questions are designed to effectively differentiate positive samples from negative ones, especially hard negatives, which are semantically similar (Robinson et al., ).
The goal is to generate questions that elicit a “yes” answer only for a specific group of texts while excluding others, even those that are closely related.

###### Example 1 :

Consider a toy example with four groups of texts for animals: $\mathbb{G}_{1}=\{{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\pgfsys@color@cmyk@stroke{1}{1}{0}{0}\pgfsys@color@cmyk@fill{1}{1}{0}{0}\textit{Whale}}},{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\pgfsys@color@cmyk@stroke{1}{1}{0}{0}\pgfsys@color@cmyk@fill{1}{1}{0}{0}\textit{Dolphin}}}\}$, $\mathbb{G}_{2}=\{{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\pgfsys@color@cmyk@stroke{1}{1}{0}{0}\pgfsys@color@cmyk@fill{1}{1}{0}{0}\textit{Shark}}},{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\pgfsys@color@cmyk@stroke{1}{1}{0}{0}\pgfsys@color@cmyk@fill{1}{1}{0}{0}\textit{Ray}}}\}$, $\mathbb{G}_{3}=\{{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\pgfsys@color@cmyk@stroke{1}{1}{0}{0}\pgfsys@color@cmyk@fill{1}{1}{0}{0}\textit{Giraffe}}},{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\pgfsys@color@cmyk@stroke{1}{1}{0}{0}\pgfsys@color@cmyk@fill{1}{1}{0}{0}\textit{Deer}}}\}$, and $\mathbb{G}_{4}=\{{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\pgfsys@color@cmyk@stroke{1}{1}{0}{0}\pgfsys@color@cmyk@fill{1}{1}{0}{0}\textit{Eagle}}},{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\pgfsys@color@cmyk@stroke{1}{1}{0}{0}\pgfsys@color@cmyk@fill{1}{1}{0}{0}\textit{Hawk}}}\}$.
The objective is to generate questions that can effectively distinguish $\mathbb{G}_{1}$ from other groups.
At first, broad questions such as “Does it live in water?” or “Is it a mammal?” might seem useful.
While these questions correctly yield “yes” for Whale and Dolphin, they also apply to other groups.
For example, Shark and Ray also live in water, and Giraffe and Deer are also mammals.
To better differentiate $\mathbb{G}_{1}$, a more precise question would be “Does the animal use echolocation to navigate and hunt?”, which yields a “yes” only for Whale and Dolphin, effectively distinguishing them from the other groups.
Furthermore, this question could also generalize to other animals, such as Bat, that were not part of the original groups, highlighting the method’s potential to apply to unseen examples.
$\triangle$

As depicted in Figure 2, the CQG method begins by identifying semantically similar groups of texts.
This is accomplished by encoding the text corpus into embedding vectors and clustering these vectors to form distinct groups.
For each cluster, we design a strategic sampling technique: selecting $n_{p}$ positive texts from within the cluster, $n_{h}$ hard negative samples from neighboring clusters, and $n_{e}$ easy negative samples from the global corpus.
The LLM is then prompted to generate questions under a key constraint: the questions must elicit “yes” answers exclusively for the positive samples and “no” answers for all negative samples. The detailed prompt for CQG is provided in Appendix A.1.

Figure: Figure 2: Illustration of the CQG method.
Refer to caption: x2.png

This strategic sampling technique serves two main purposes:

- (1)
Discriminative Power:
Contrasting positive samples with hard negative samples encourages the LLM to generate highly discriminative questions tailored to each cluster.
- (2)
Broader Relevance:
Including easy negative samples ensures that the generated questions maintain broader relevance across the entire corpus.

A well-tuned negative-to-positive ratio, i.e., $(n_{h}+n_{e})/n_{p}$, encourages the LLM to craft precise, discriminative questions for the positive cluster, leading to sparser and more interpretable embedding dimensions.
This process is repeated across all clusters, resulting in a comprehensive set of LLM-generated questions that capture the unique characteristics of each cluster while maintaining global relevance, forming the foundation of our interpretable text embedding framework.

###### Example 1 :

#### Post-Processing

LLMs may encounter two common issues when generating questions: (1) failing to consistently provide “yes” answers only for positive samples and (2) generating similar questions across different clusters.
To address these challenges, we implement a post-processing step to filter and select the highest-quality, non-redundant questions.

We introduce a probing mechanism to evaluate and refine the generated questions.
For each question, we randomly sample $p_{p}$ positive probes from the originating cluster, $p_{h}$ hard negative probes from neighboring clusters, and $p_{e}$ easy negative probes from other clusters.
The LLM answers these questions, and we calculate the quality of each question using the following formula:

$$ $\text{quality}=\frac{\text{\# {``{\color[rgb]{1,0.58,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.58,0}\pgfsys@color@cmyk@stroke{0}{0.42}{1}{0}\pgfsys@color@cmyk@fill{0}{0.42}{1}{0}{yes}}''}{} for positive probes}}{p_{p}}-\frac{\text{\# {``{\color[rgb]{1,0.58,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.58,0}\pgfsys@color@cmyk@stroke{0}{0.42}{1}{0}\pgfsys@color@cmyk@fill{0}{0.42}{1}{0}{yes}}''}{} for negative probes}}{p_{h}+p_{e}}$ (1) $$

Equation 1 measures the difference between the percentage of “yes” answers for positive probes and negative probes.
A higher quality value indicates that the question is more discriminative for the cluster, as it correctly identifies more positive samples while filtering out negatives.

To construct the final set of questions, we iteratively select the top $t$ most discriminative questions from each cluster, based on their quality values, and ensure that no two questions are highly semantically similar.
Here, the similarity between questions is measured using cosine similarity between their corresponding embedding vectors, which are generated using a pre-trained encoding model.
If the cosine similarity of two questions exceeds a predefined threshold $\theta$, they are considered duplicates, and the latter question is excluded.

This post-processing step helps filter out hallucinated and/or low-quality questions generated by the LLM.
The final set comprises $m$ questions, forming a diverse and highly discriminative collection that effectively captures the semantic structure of the entire corpus.

### 3.2 Question Answering

#### Motivations

Generating answers to questions using LLMs can be prohibitively expensive, especially when scaling up to large datasets with numerous questions.
Table 1 presents the cost analysis for embedding the MS MARCO dataset (Bajaj et al., ), revealing that the expense of LLM-generated answers becomes untenable when dealing with thousands of questions (dimensions) across million-scale datasets.
LLM-based Question Answering (QA), which leverages LLMs to answer 10,000 questions for approximately 8.8 million articles in the MS MARCO dev set, requires about 4.4 billion LLM inference passes and processes 1.5 trillion tokens. This incurs a substantial cost of 244,551 USD, even with a cost-effective model (GPT-4o-mini) and a token-efficient prompting approach (grouping 20 questions per prompt).
Further details are available in Appendix C.

While increasing the number of questions improves performance (see Section 4.5), the cost associated with LLM-based QA renders it impractical for large-scale real-world applications.
To address this, we propose the Multi-task Binary Question Answering (MBQA) model as a cost-effective alternative.

#### Multi-task Binary Question Answering (MBQA) Model

The MBQA model is designed to leverage LLM-generated answers from a smaller subset of texts to train a multi-task binary classification model.
This model consists of a pre-trained encoding model and multiple classification heads. The encoder converts the input text into an embedding vector, while the classification heads predict binary scores for each question.
Formally, the MBQA model $\mathcal{M}$ is defined as:

$$ $\mathcal{M}=(Enc,\{C_{1},C_{2},\cdots,C_{m}\}),$ (2) $$

where $Enc:\mathcal{T}\rightarrow\mathbb{R}^{d}$ represents the encoding model, and $C_{i}:\mathbb{R}^{d}\rightarrow[0,1]$ is the $i$-th Multi-Layer Perceptron (MLP) classification head.
For a given input text $\bm{t}\in\mathcal{T}$, the MBQA model generates a binary embedding vector $\bm{y}=[y_{1},y_{2},\cdots,y_{m}]\in\{0,1\}^{m}$ as follows:

$$ $\displaystyle\bm{e}$ $\displaystyle=Enc(\bm{t}),$ (3) $\displaystyle y_{i}$ $\displaystyle=\bm{1}[\sigma(C_{i}(\bm{e}))>\tau],~\text{for}~i\in\{1,2,\cdots,m\},$ (4) $$

where $\sigma$ is the Sigmoid function, $\bm{1}[\cdot]$ is the indicator function, and $\tau$ is the threshold for binary classification.
During training, the encoder $Enc$ is frozen and only the classification heads $\{C_{1},C_{2},\cdots,C_{m}\}$ are optimized using weighted Binary Cross-Entropy (BCE) Loss (Bishop, ) on the available LLM-generated question-answer pairs.

#### Remarks

The MBQA model achieves 96% accuracy in reproducing LLM-generated answers for CQG questions with just a single pass through the encoding model, substantially reducing costs compared to running a pre-trained LLM for each text.
Our model only requires training data from as few as 1,000 articles per question, resulting in 10 million text-question pairs for 10,000 questions, costing just 31 USD using GPT-4o-mini.
The training process takes 36 hours, and embedding the entire MS MARCO dev set requires 90 hours on a single GTX 1080 Ti, which is an inexpensive GPU.
Consequently, encoding the same MS MARCO dev set with the MBQA model costs around 41 USD–just 0.017% of the original cost with GPT-4o-mini.
This model allows us to scale up the number of questions (dimensions) efficiently, providing interpretable embeddings at a fraction of the cost of LLM-based answering. For more details on training and evaluation, see Appendix B.

**Table 1: Estimated cost for embedding the MS MARCO dev set using LLM-generated answers.**
| Model | Cost for Number of Questions |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| 2,000 | 4,000 | 6,000 | 8,000 | 10,000 |  |
| GPT-4o-mini | $48,859 | $97,792 | $146,725 | $195,570 | $244,551 |
| GPT-4o | $1,454,000 | $2,910,487 | $4,366,946 | $5,820,517 | $7,278,566 |
| MBQA | $13 | $20 | $27 | $34 | $41 |

## 4 Experiments

In this section, we present a comprehensive evaluation of the CQG-MBQA framework by addressing four essential questions aimed at understanding its performance and applicability:

- •
Embedding Quality: How well does our framework generate high-quality interpretable embeddings comparable to advanced black-box models? (Section 4.3)
- •
Interpretability: Does our framework improve the human interpretability of embeddings over existing methods? (Section 4.4)
- •
Question Efficiency: Can the CQG method generate a limited number of highly discriminative questions that maintain strong performance? (Section 4.5)
- •
Flexibility: Can the MBQA model be tuned to strike a balance between embedding quality and interpretability? (Section 4.6)

To rigorously evaluate the framework, we conduct experiments on three core downstream tasks in text embedding: STS, retrieval, and clustering. These experiments allow us to benchmark the CQG-MBQA framework against both black-box and interpretable models.

### 4.1 Metrics

#### Embedding Quality Measurement

For evaluating embedding quality, we adopt the metrics that are widely used in the MTEB benchmark (Muennighoff et al., ) for a comprehensive comparison.
For STS tasks, we use Spearman correlation (Spearman, ) on cosine similarity between embeddings as the evaluation metric. In retrieval tasks, we assess the performance using Normalized Discounted Cumulative Gain at Top 10 (nDCG@10) (Wang et al., ). For clustering tasks, we evaluate the results using V-Measure (Rosenberg & Hirschberg, ).

#### Interpretability Measurement

Since both STS and retrieval tasks measure pairwise text similarity using cosine similarity, we focus on interpreting the cosine similarity scores produced by CQG-MBQA.
With inherently interpretable dimensions, we can offer insights to users by highlighting the dimensions that contribute most to the similarity between two texts.
Building on COGAM (Abdul et al., ), we suggest that interpretability should account for the cognitive load imposed on users.
In COGAM, cognitive load is assessed by counting the number of visual cognitive chunks.
Similarly, we measure it by the number of questions a user must consider to understand the similarity between two texts, corresponding to the dimensions where both embedding vectors have a value of 1.
Formally, for any two binary embedding vectors $\bm{u}=[u_{1},u_{2},\cdots,u_{m}]$ and $\bm{v}=[v_{1},v_{2},\cdots,v_{m}]$, the cognitive load is defined as the inner product of $\bm{u}$ and $\bm{v}$:

$$ $\text{cognitive load}=\langle\bm{u},\bm{v}\rangle=\sum_{i=1}^{m}u_{i}v_{i}.$ (5) $$

Quantifying cognitive load allows us to assess the interpretability of our CQG-MBQA framework’s embeddings.
A lower value indicates that fewer dimensions are involved, making the interpretation easier to understand, thus enhancing both interpretability and user-friendliness.

**Table 2: STS results measured by Spearman correlation. Evaluated on seven popular datasets: SemEval STS tasks 2012-2016 (STS12–STS16) (Agirre et al., ; ; ; ; ), STS Benchmark (STS-B) (Cer et al., ), and SICK-Relatedness (SICK-R) (Marelli et al., ) using the MTEB evaluation suite (Muennighoff et al., ).**
| Type | Model | Spearman Correlation (STS) |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| STS12 | STS13 | STS14 | STS15 | STS16 | STS-B | SICK-R | Avg. |  |  |
| Black-box | BERT | 38.78 | 57.98 | 57.98 | 63.15 | 61.06 | 46.35 | 58.40 | 54.81 |
| GloVe | 54.64 | 69.16 | 60.81 | 72.31 | 65.34 | 61.54 | 55.43 | 62.74 |  |
| USE | 64.49 | 67.80 | 64.61 | 76.83 | 73.18 | 74.92 | 76.69 | 71.22 |  |
| SimCSE (Unsup.) | 66.05 | 81.49 | 73.61 | 79.72 | 78.12 | 76.52 | 72.24 | 75.39 |  |
| SBERT (Ori.) | 74.53 | 77.00 | 73.18 | 81.85 | 76.82 | 79.10 | 74.29 | 76.68 |  |
| SimCSE (Sup.) | 75.30 | 84.67 | 80.19 | 85.40 | 80.82 | 84.25 | 68.38 | 79.86 |  |
| WhitenedCSE | 74.65 | 85.79 | 77.49 | 84.71 | 80.33 | 81.48 | 75.34 | 79.97 |  |
| SBERT (New) | 73.08 | 82.13 | 76.73 | 85.58 | 80.23 | 83.09 | 79.32 | 80.02 |  |
| OpenAI | 72.84 | 86.1 | 81.15 | 88.49 | 85.08 | 83.56 | 79.00 | 82.31 |  |
| AnglE | 79.09 | 89.62 | 85.02 | 89.51 | 86.61 | 89.06 | 82.62 | 85.93 |  |
| Interp. | Bag-of-Tokens | 44.75 | 52.06 | 54.78 | 68.65 | 60.59 | 54.85 | 57.87 | 56.22 |
| QAEmb-MBQA | 59.40 | 63.19 | 57.68 | 69.29 | 63.18 | 71.33 | 72.33 | 65.20 |  |
| CQG-MBQA | 69.21 | 80.19 | 73.91 | 80.66 | 78.30 | 82.69 | 78.21 | 77.60 |  |

**Table 3: Retrieval results evaluated by nDCG@10. Evaluated on seven diverse datasets: MS MARCO (Bajaj et al., ), NewsSpectrum (NSP) (Sun et al., ), ArguAna (Wachsmuth et al., ), FiQA-2018 (FQA) (Maia et al., ), NFCorpus (NFC) (Boteva et al., ), SCIDOCS (Cohan et al., ), and SciFact (Wadden et al., ). MS MARCO is evaluated on a 1% sample of its dev set, while NewsSpectrum uses news titles as queries with corresponding articles as targets. The remaining datasets are assessed using the MTEB evaluation suite.**
| Type | Model | nDCG@10 (Retrieval) |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MS MARCO | NSP | ArguAna | FQA | NFC | SCIDOCS | SciFact | Avg. |  |  |
| Black-box | BERT | 16.86 | 10.64 | 28.29 | 2.19 | 4.30 | 2.82 | 13.34 | 11.21 |
| SimCSE (Unsup.) | 44.63 | 38.04 | 38.34 | 9.84 | 9.88 | 5.50 | 25.72 | 24.56 |  |
| GloVe | 44.27 | 30.84 | 36.30 | 10.09 | 13.87 | 8.04 | 29.58 | 24.71 |  |
| SimCSE (Sup.) | 47.86 | 46.29 | 38.33 | 10.41 | 12.42 | 7.53 | 29.59 | 27.49 |  |
| SBERT (New) | 88.74 | 70.64 | 47.13 | 37.27 | 32.25 | 21.82 | 62.64 | 51.50 |  |
| AnglE | 90.43 | 82.87 | 66.15 | 44.84 | 38.65 | 22.98 | 74.07 | 60.00 |  |
| OpenAI | 92.18 | 86.77 | 58.05 | 55.00 | 42.07 | 23.11 | 77.77 | 62.14 |  |
| Interp. | Bag-of-Tokens | 29.79 | 23.95 | 34.25 | 3.99 | 21.51 | 6.79 | 47.36 | 23.95 |
| BM25 | 68.42 | 76.42 | 49.28 | 25.14 | 32.08 | 15.78 | 68.70 | 47.97 |  |
| QAEmb-MBQA | 40.51 | 16.07 | 34.75 | 8.23 | 3.87 | 3.74 | 12.01 | 17.03 |  |
| CQG-MBQA | 62.21 | 32.30 | 47.75 | 18.63 | 9.74 | 8.67 | 32.80 | 30.30 |  |

**Table 4: Clustering results assessed by V-Measure. Evaluated on seven commonly-used datasets: TwentyNewsgroupsClustering (TNG), StackExchangeClusteringP2P (SE-P2P), BiorxivClusteringP2P (BR-P2P), BiorxivClusteringS2S (BR-S2S), MedrxivClusteringP2P (MR-P2P), MedrxivClusteringS2S (MR-S2S), and RedditClusteringP2P (RD-P2P) from the MTEB evaluation suite.**
| Type | Model | V-Measure (Clustering) |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| TNG | SE-P2P | BR-P2 | BR-S2S | MR-P2P | MR-S2S | RD-P2P | Avg. |  |  |
| Black-box | SimCSE (Unsup.) | 23.21 | 28.50 | 24.90 | 19.55 | 23.60 | 21.97 | 45.14 | 26.70 |
| GloVe | 25.83 | 28.51 | 29.27 | 19.18 | 26.12 | 20.38 | 35.82 | 26.44 |  |
| BERT | 23.35 | 26.55 | 30.12 | 24.77 | 26.09 | 23.60 | 43.32 | 28.26 |  |
| SimCSE (Sup.) | 34.86 | 29.45 | 30.15 | 24.67 | 26.25 | 24.12 | 47.74 | 31.03 |  |
| SBERT (New) | 47.47 | 33.13 | 36.99 | 33.21 | 34.25 | 32.24 | 54.80 | 38.87 |  |
| AnglE | 51.72 | 36.72 | 39.38 | 37.23 | 33.22 | 31.18 | 65.35 | 42.11 |  |
| OpenAI | 58.14 | 36.88 | 38.03 | 36.53 | 32.70 | 31.27 | 67.96 | 43.07 |  |
| Interp. | Bag-of-Tokens | 8.52 | 17.64 | 4.70 | 3.32 | 11.39 | 13.05 | 15.67 | 10.61 |
| QAEmb-MBQA | 36.72 | 25.68 | 24.66 | 21.16 | 25.53 | 22.85 | 46.57 | 29.02 |  |
| CQG-MBQA | 40.00 | 28.22 | 34.88 | 31.13 | 31.02 | 28.71 | 54.40 | 35.48 |  |

### 4.2 Models

#### Interpretable Models

We evaluate CQG-MBQA against existing interpretable baselines to provide a thorough comparison.
The implementation details of CQG-MBQA are provided in Appendix D.1.
To make a fair comparison and highlight the benefits of our CQG method, we introduce QAEmb-MBQA. We replicate the example-based question generation approach of QAEmb (Benara et al., ) and integrate the QAEmb-generated questions with our MBQA model.
Additionally, we include Bag-of-Tokens, a simple baseline that uses the BERT tokenizer to produce interpretable embeddings.
For the retrieval task, we also compare against the rule-based sparse retriever BM25 (Robertson et al., ), implemented using BM25S (Lù, ).

#### Black-box Models

To benchmark the embedding quality of CQG-MBQA, we compare it with several advanced black-box text embedding models. These include
GloVe (Pennington et al., ; Reimers & Gurevych, ), USE (Cer et al., ), BERT (Devlin et al., ), the Original (Ori.) and Up-to-date (New) versions of Sentence-BERT (SBERT) (Reimers & Gurevych, ), the Supervised (Sup.) and Unsupervised (Unsup.) versions of SimCSE (Gao et al., ).
Additionally, we compare our framework with WhitenedCSE (Zhuo et al., ), the OpenAI API, and AnglE (Li & Li, ).
Implementation details for all baseline models are outlined in Appendix D.2.

### 4.3 Embedding Quality

Tables 2–4 present the embedding quality results for the STS, retrieval, and clustering tasks, highlighting CQG-MBQA’s competitive performance.
In the STS tasks (Table 2), CQG-MBQA achieves comparable embedding quality to advanced dense embedding models like SimCSE and SBERT (New) while preserving inherent interpretability.
It outperforms earlier methods such as GloVe, USE, and BERT, as well as all interpretable baselines.
For retrieval tasks (Table 3), CQG-MBQA surpasses models like SimCSE, GloVe, and BERT.
While it trails state-of-the-art black-box models such as AnglE and OpenAI, it consistently outperforms all interpretable baselines except for BM25, which is a rule-based model optimized for retrieval.
In clustering tasks (Table 4), CQG-MBQA outperforms several black-box models (SimCSE, GloVe, and BERT) and surpasses all interpretable baselines.
Its performance closely approaches that of more recent models like SBERT (New).
Additionally, the comparison between CQG-MBQA and QAEmb-MBQA underscores the efficacy of our CQG algorithm in generating high-quality, discriminative questions that capture semantic nuances.

### 4.4 Interpretability

#### Cognitive Load

Table 5 displays the cognitive load required to interpret embeddings across different interpretable models, evaluated through the STS task, which computes pairwise similarity of texts.
This provides a clear measure of how much effort is needed to understand the embeddings.
CQG-MBQA achieves a 2.5$\sim$3.6$\times$ lower cognitive load than QAEmb-MBQA, indicating that the CQG method significantly enhances interpretability.
This technique produces more “no” answers and fewer “yes” answers, making the embeddings easier to interpret.
For Bag-of-Tokens, it has a much lower cognitive load due to its lexical nature, but this advantage comes at the cost of significantly reduced embedding quality.
The trade-off between interpretability and embedding quality can be adjusted by tuning the binary classification threshold $\tau$, as further discussed in Section 4.6.

**Table 5: STS results measured by cognitive load (the lower the better).**
| Model | Cognitive Load (STS) |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| STS12 | STS13 | STS14 | STS15 | STS16 | STS-B | SICK-R | Avg. |  |
| Bag-of-Tokens | 8 | 4 | 6 | 5 | 8 | 7 | 6 | 6 |
| QAEmb-MBQA | 1,626 | 1,571 | 1,625 | 1,443 | 1,577 | 1,408 | 1,018 | 1,467 |
| CQG-MBQA | 481 | 439 | 458 | 426 | 478 | 446 | 413 | 449 |

Figure: Figure 3: Case study.
Refer to caption: x3.png

Figure: Figure 4: Spearman correlation and cognitive load vs. the number of dimensions $m$. Higher Spearman correlation signals better embedding quality; lower cognitive load implies greater interpretability.
Refer to caption: x4.png

Figure: Figure 5: Spearman correlation and cognitive load vs. the binary classification threshold $\tau$.
Refer to caption: x5.png

#### Case Study

Figure 3 showcases a pair of texts from our training corpus, focusing on nine specific questions (dimensions) where at least one text yield a “yes” answer.
This illustrates how CQG-MBQA generates relevant and insightful dimensions.
For instance, question ID 1153, which asks if the text is related to personal health or well-being, receives a “yes” for both Text A and Text B, accurately reflecting their shared focus on health topics.
Similarly, question ID 4634 inquires whether the text aligns with scientific standards, and both texts–discussing evidence-based findings on substance effects–obtain “yes” answers, showcasing the relevance of generated questions.

Despite the texts’ similarity, CQG-MBQA captures subtle semantic differences through fine-grained questions.
For example, question ID 3400 asks if the research targets neurological conditions.
Text A, which discusses cognitive function, receives a “yes”, indicating a connection to neurological conditions, whereas Text B, focusing on blood pressure and insulin sensitivity, acquires a “no”, highlighting a clear distinction in their subject matter.
This case study highlights how CQG generates interpretable, relevant, and discriminative questions that effectively capture nuanced semantic differences, while MBQA accurately predicts the answers, reinforcing the framework’s practicality and reliability.

### 4.5 Effect on Number of Questions

We explore how varying the number of questions (dimensions) $m$ impacts the quality and interpretability of embeddings produced by CQG-MBQA.
To adjust $m$, we reduce the output length of the final binary embedding vector.
Figure 4 illustrates the relationship among embedding quality, interpretability, and $m$.
As $m$ grows, the Spearman correlation increases and stabilizes around 3,000 dimensions, indicating better embedding quality.
This highlights the need for an efficient QA model to manage computational costs while scaling up dimensions for optimal embedding quality.

However, a trade-off arises: as $m$ increases, cognitive load increases, and the interpretability declines due to a higher proportion of “yes” in the embeddings.
This inverse relationship between embedding quality and interpretability emphasizes the importance of balancing dimensions based on the task’s requirements.
Figure 4 also demonstrates the effectiveness of the CQG algorithm in generating high-quality, discriminative embeddings with approximately 3,000 questions, achieving a balance between embedding quality and interpretability without an excessive number of questions.

### 4.6 Trade-off between Embedding Quality and Interpretability

To further investigate the balance between embedding quality and interpretability, we vary the binary classification threshold $\tau$ that determines the final binary embedding vector.
Figure 5 depicts a clear trade-off between embedding quality and interpretability.
Increasing $\tau$ improves the interpretability, but this comes at the cost of reduced embedding quality, as the Spearman correlation decreases.
This is due to fewer active dimensions, leading to reduced values in the cognitive load, which are easier to interpret but may lose subtle semantic distinctions.

More importantly, this trade-off presents an opportunity for user-driven customization.
Depending on different scenarios, users of our framework can dynamically tune the desired $\tau$ based on the cognitive load to meet their needs.
For instance, in scenarios requiring rapid decision-making or where cognitive resources are constrained, users might prioritize interpretability by opting for a higher threshold.
On the other hand, in situations where nuanced analysis is crucial and resources are abundant, a lower threshold could be chosen to maximize embedding quality.
This flexibility makes CQG-MBQA highly adaptable to different scenarios and user requirements.

## 5 Conclusion

In this paper, we introduce CQG-MBQA, a novel general framework for generating interpretable semantic text embeddings by systematically creating binary questions and using the answers as interpretable embedding dimensions.
Our CQG method effectively addresses the challenges of generalizability and quality issues during the question generation phase, while the MBQA model provides an efficient, scalable solution for answering these questions, significantly reducing costs.
Through extensive experiments on STS, retrieval, and clustering tasks, we demonstrate that our framework delivers performance comparable to advanced black-box models while being inherently interpretable.
Moreover, CQG-MBQA consistently outperforms other interpretable text embedding models across various downstream tasks, further validating its effectiveness.

## Appendix A Prompts

### A.1 Prompt: Contrastive Question Generation

We present the prompt template used for the Contrastive Question Generation (CQG) algorithm. The input to the prompt template consists of two lists of texts: positive_examples and negative_examples. The LLM is explicitly instructed to generate questions that yield “yes” answers for the positive examples and “no” answers for the negative examples.
To enhance the quality of the generated questions and the potential generalizability to new texts, we prompt the LLM to avoid complex sentence structures. Initial experiments revealed that if we do not require the LLM to avoid complex sentence structures, the LLM tended to create discriminative questions by simply combining two conditions to ensure “yes” answers for the positives, rather than identifying deeper relationships between the positives and negatives.
Additionally, we include formatting instructions at the end of the prompt template to improve result parsing accuracy.

### A.2 Prompt: QAEmb Question Generation

The Question Generation Prompt #5 used in the QAEmb paper (Benara et al., ) is the most general form prompt in the list, making it suitable for adopting it to generate questions for general-purpose text embedding. It uses two lists of texts as input: example narrative sentences and example yes/no questions. The original prompt instructs the LLM to "Generate a bulleted list of 100 specific, non-overlapping yes/no questions that ask about aspects of the example narrative sentences that are important for classifying them." This was originally designed for the task of fMRI prediction with narrative sentences.

Following this approach, we designed a prompt template for experiments on QAEmb question generation, also using two lists of texts as inputs: reference_articles and example_questions. The example questions are sourced from the original QAEmb paper, while the reference articles are randomly drawn from the training dataset.

### A.3 Prompt: Multi-task Binary Question Answering

This section details the prompt template used to generate LLM answers for training the Multi-task Binary Question Answering (MBQA) model. The prompt takes two inputs: the text_chunk, which is the training article sample, and a list of questions to be answered. To optimize token usage for the article sample and instructions, we group up to 20 questions in a single prompt.

## Appendix B Training and Evaluation of the MBQA Model

To ensure that the MBQA model produces faithful answers to the questions, we evaluate its question-answering performance on a 10% held-out document set that was not used for training.

#### Data Collection

For each question generated in the previous question generation step, we randomly sample 500 in-cluster samples, 300 neighboring cluster samples from 5 nearest clusters, and 200 random samples from the entire corpus.
We use the pre-trained LLM (GPT-4o-mini) to generate answers for each question across these samples. The LLM-generated answers are batched in groups of 20 questions to train the multi-task binary classification model.
Refer to Appendix A.3 for the prompt used to collect answers.
This approach allows us to gather data from a larger number of text samples, thereby increasing the generalizability of our model.

#### Training

We train the MBQA model using the Adam optimizer with a learning rate $\alpha$ of 1e-4 and a batch size of one text sample.
For each step, we calculate the loss based on all available questions with answers from the previous data collection phase. The model is trained using the BCEWithLogitsLoss function, where the weight is the ratio of “yes” answers to “no” answers in the training data.(^1^11[https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html](https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html))
Specifically, for CQG-MBQA, we set this weight to 7.5127, and for QAEmb-MBQA, the weight is set to 4.9608.
The model is trained for 3 million steps, at which point performance begins to converge.

#### Evaluation

The classification results (with threshold $\tau=0.5$) on the held-out test set for CQG-MBQA and QAEmb-MBQA are presented in Tables 6 and 7, respectively.
The CQG-MBQA model achieves an accuracy of 96% and a macro F1 score of 91%, while the QAEmb-MBQA model attains a high 93% accuracy and 89% macro F1 score.
These results demonstrate that MBQA can accurately predict LLM-generated question answers, serving as a cost-effective substitute for the more expensive LLM model in generating embeddings.

**Table 6: Question answering performance of the CQG-MBQA model.**
| Class | Precision | Recall | F1-score | Support |
| --- | --- | --- | --- | --- |
| “no” | 1.00 | 0.96 | 0.97 | 846,089 |
| “yes” | 0.74 | 0.97 | 0.84 | 112,645 |
| Macro Avg | 0.87 | 0.96 | 0.91 | 958,734 |
| Weighted Avg | 0.97 | 0.96 | 0.96 | 958,734 |
| Accuracy | 0.96 |  |  |  |

**Table 7: Question answering performance of the QAEmb-MBQA model.**
| Class | Precision | Recall | F1-score | Support |
| --- | --- | --- | --- | --- |
| “no” | 0.99 | 0.93 | 0.96 | 886,749 |
| “yes” | 0.72 | 0.94 | 0.82 | 178,591 |
| Macro Avg | 0.85 | 0.93 | 0.89 | 1,065,340 |
| Weighted Avg | 0.94 | 0.93 | 0.93 | 1,065,340 |
| Accuracy | 0.93 |  |  |  |

## Appendix C Cost Analysis for LLM-based QA vs. MBQA

We estimate the cost of LLM-based QA and MBQA for producing question answers for interpretable embeddings for the entire MS MARCO dev set.

#### LLM-based QA

Using LLMs to answer questions for document embedding is prohibitively expensive. Table 1 shows the cost of running LLM-based QA on the MS MARCO dev set for various numbers of questions across different models.
We assume grouping 20 questions into one prompt, using the format in Appendix A.3.
Using the Batch API, the cost per 1 million tokens for GPT-4o is 2.5 USD for input tokens and 7.5 USD for output tokens, while for GPT-4o-mini, it’s 0.075 USD for input tokens and 0.3 USD for output tokens.(^2^22Costs obtained from [https://openai.com/api/pricing](https://openai.com/api/pricing))

#### MBQA

The cost of running our MBQA model comprises two key components: (1) LLM API cost for training data collection and (2) GPU runtime expenses.
For the first component, an upfront cost of approximately 31 USD is required to collect training data using GPT-4o-mini on the MEDI2 dataset. This cost covers 1,000 text-question pairs for each of 10,000 questions, resulting in a total of 10 million text-question pairs.
The cost scales down proportionally for fewer questions.
For the second component, we measure the training and inference time of our model on a single GTX 1080 Ti GPU.
We estimate the cost based on a rental rate of 0.08 USD per hour.(^3^33Costs obtained from [https://vast.ai/pricing/gpu/GTX-1080-TI](https://vast.ai/pricing/gpu/GTX-1080-TI))
Training for 3 million steps took around 36 hours, and inference times for the MS MARCO dataset varied by the number of dimensions: 48 hours for 2,000 dimensions, 63 hours for 4,000, 73 hours for 6,000, 79 hours for 8,000, and 90 hours for 10,000 dimensions.

## Appendix D Implementation Details

**Table 8: Hyperparameters used in our experiments.**
| Description | Symbol | Value |
| --- | --- | --- |
| Encoding model | $Enc$ | UAE-Large-V1 |
| Generation model | $LLM$ | GPT-4o-mini-2024-07-18 |
| Number of clusters | $k$ | 5,000 |
| Positive samples per cluster | $n_{p}$ | 6 |
| Hard negative samples per cluster | $n_{h}$ | 18 |
| Easy negative samples per cluster | $n_{e}$ | 18 |
| Positive probe samples per question | $p_{p}$ | 5 |
| Hard negative probe samples per question | $p_{h}$ | 3 |
| Easy negatives probe samples per question | $p_{e}$ | 2 |
| Deduplication threshold | $\theta$ | 0.8 |
| Top questions per cluster | $t$ | 4 |
| Learning rate of the MBQA Model | $\alpha$ | 1e-4 |
| Binary classification threshold | $\tau$ | 0.5 |

### D.1 Our CQG-MBQA Model

In the experiments, we train the proposed CQG-MBQA framework using the MEDI2 dataset (Muennighoff et al., ).
Detailed information about the model configuration and hyperparameters used in our framework is provided in Table 8.

#### Data Pre-processing

We use the MEDI2 dataset, downloaded from the HuggingFace repository at GritLM/MEDI2.(^4^44[https://huggingface.co/datasets/GritLM/MEDI2/tree/main](https://huggingface.co/datasets/GritLM/MEDI2/tree/main))
We filter out files starting with task, as they are unsuitable for the training corpus.
From the remaining files, we extract both positive and negative instances of each data line.
Since the MEDI2 dataset contains instructions for queries and documents, we remove the instruction part, leaving only the content. We merge all positive and negative instances from the filtered corpus and run a simple exact deduplication to produce the final training text corpus.

#### Contrastive Question Generation

The pre-processed training corpus contains 6.8 million unique texts. We encode these texts using the AnglE encoding model (UAE-Large-V1) and normalize the embeddings.
We then run KMeans clustering (Arthur & Vassilvitskii, ) with $k=5,000$ clusters and default parameters, utilizing using the scikit-learn library,(^5^55[https://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html)) accelerated by Intel(R) Extension for scikit-learn.(^6^66[https://github.com/intel/scikit-learn-intelex](https://github.com/intel/scikit-learn-intelex))
Once clustering is complete, we generate questions for each cluster according to the process described in Section 3.1.
We sample random positive examples from within the cluster, hard negatives from the three nearest clusters, and easy negatives from the remaining clusters. This process yields 9,614 deduplicated questions, which serve as the final embedding dimensions.

#### Multi-Task Binary Question Answering

We train the MBQA model following the setup described in Appendix B.

**Table 9: Model checkpoints used in our experiments.**
| Model | Checkpoint |
| --- | --- |
| BERT | bert-base-uncased |
| GloVe | average_word_embeddings_glove.6B.300d |
| SimCSE (Unsup.) | unsup-simcse-bert-base-uncased |
| SimCSE (Sup.) | sup-simcse-bert-base-uncased |
| SBERT (New) | all-MiniLM-L12-v2 |
| OpenAI | text-embedding-ada-002 |
| AnglE | UAE-Large-V1 |
| BM25 | bm25s |

### D.2 Baseline Models

#### QAEmb-MBQA

To ensure a fair comparison with QAEmb (Benara et al., ), we modify our framework by replacing the CQG step with the example-based prompting method adopted from QAEmb, while keeping all other parameters the same, except for setting the deduplication threshold $\theta=0.925$.
Detailed prompts used for this version are provided in Appendix A.2.
This modification results in a total of 10,654 questions.
The MBQA model is trained using the same approach as in our CQG-MBQA framework.

#### Black-box Models

For STS tasks, the results for SBERT (Ori.) and USE are sourced from (Reimers & Gurevych, ), and the results for WhitenedCSE are taken from the best-performing model in (Zhuo et al., ), all evaluated using the same metric.
Table 9 shows the model checkpoints used for each black-box model.
For STS, retrieval (excluding MS MARCO and NewsSpectrum), and clustering tasks, the results for BERT, GloVe, SimCSE (Unsup.), SimCSE (Sup.), SBERT (New), OpenAI, and AnglE are retrieved from the MTEB leaderboard.(^7^77[https://huggingface.co/spaces/mteb/leaderboard](https://huggingface.co/spaces/mteb/leaderboard))
The retrieval results for BM25 (excluding the datasets MS MARCO and NewsSpectrum) are also retrieved from the MTEB leaderboard, while our own experiments were conducted for MS MARCO and NewsSpectrum.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1) In order to achieve performance of regular dense representation models, which aspects of the framework and the implementation details do the authors think worth scaling up? Is there any interesting evidence? e.g., better datasets for generating questions; more questions to form the embedding dimensions, etc..
2) From my understanding in the appendix, the paper uses UAE-large-v1 as the encoding model, why? What happens if the encoding model is some better models - does it help with the performance of the final interpretable embeddings?
soundness: 3
strengths: 1) The question generation component of the framework concerns generating questions that are both discriminative and general. It groups similar texts by clustering for the generation of questions, such that nuanced questions can be asked for each group, as opposed to simple questions in the baseline method. The concept is analogous to leveraging hard negatives in regular training of embedding models.
2) The authors show good understanding at related work; the implementation and the evaluation are sound from the perspective of sentence embeddings.
summary: The paper proposes CQG-MVQA, an intepretable embedding framework. The framework generates questions and binary answers about texts, and trains binary classifiers on each question. Each of the prediction to these questions forms a dimension of an interpretable embedding. It is shown that these interpretable achieves decent performance on MTEB and a reduced cognitive load for interpretability compared to baseline methods in interpretable embeddings.
weaknesses: 1) Performance ablations about setups in the framework can be very interesting although currently missing (e.g., performance across different dimensionality, question difficulties, different encoding models, etc..).
2) Implementation details can be moved more to the main paper as they are mostly in appendices.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Does the CQG-MBQA framework need to generate a new set of yes/no questions every time it is applied to a different dataset? If so, is there a way to enhance the generalizability of the CQG-MBQA model? In other words, could a more universal set of yes/no questions be designed to handle multiple tasks/datasets, rather than creating a separate set tailored to each specific task/dataset?

2. Figure 4 shows that with around 3,000 questions, CQG-MBQA can achieve high-quality text embeddings on STS tasks, and using additional questions does not improve embedding quality; instead, it decreases interpretability. Does this imply that the yes/no questions generated by CQG have semantic overlap or inclusion relationships? In other words, is there a large number of semantically similar questions within the set of yes/no questions?
soundness: 4
strengths: 1. CQG-MBQA focus on producing interpretable embeddings, which is important for domains requiring transparency.
2. Compared with QAEmb,  CQG produces more discriminative questions.
3. By integrating MBQA, the framework achieves cost-effective embeddings compared to LLM-based alternatives.
4. This paper conducts extensive experiments on semantic textual similarity, retrieval, and clustering tasks, showcasing its utility and competitiveness.
summary: This paper introduces CQG-MBQA (Contrastive Question Generation - Multi-task Binary Question Answering), a general framework to create interpretable semantic text embeddings for NLP tasks. This framework emphasizes interpretability, which is essential for tasks requiring transparency, such as legal and medical applications. Traditional black-box text embedding methods, while effective, lack interpretability, limiting their utility in such cases. By comparison, CQG-MBQA is able to generate interpretable semantic text embeddings via binary yes/no questions. To be concrete, this framework first generates binary yes/no questions through contrastive question generation (CQG) using GPT-4o-mini for the entire corpus. Then, it fine-tunes a multi-task binary question-answering (MBQA) model by distilling knowledge from GPT-4o-mini. In this way, one can use MBQA to create the interpretable embeddings for text, without relying on LLMs, thus reducing the API costs. The experimental results show that CQG-MBQA performs comparably to advanced black-box models, and outperforms other interpretable text embedding models across various downstream tasks.
weaknesses: Please refer to the Questions.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - Did you evaluate your model on the fMRI task presented in the QAEmb paper for a direct performance comparison?
- How do variations in the number of initial clusters (k) or the use of different clustering methods affect the performance of CQG?
soundness: 3
strengths: The ideas behind CQG and MBQA are novel and effective, supported by thoughtful experiments and ablation studies. The paper is clearly written and well-structured. Given the increasing demand for model transparency, CQG-MBQA could have significant implications and represent a meaningful approach that would be of interest to the ICLR audience.

- The paper builds upon QAEmb with important innovations: a contrastive approach to question generation that improves discrimination using positive, hard negative, and easy negative samples for fine-grained specificity, and a multi-task model for efficient inference without the use of LLMs.
- The technical quality is demonstrated through comprehensive empirical evaluation across multiple tasks and strong baselines including both interpretable and black-box models, with clear cost analysis showing significant efficiency gains through MBQA during inference. For reproducibility, detailed implementation specifics and code are provided.
summary: The paper introduces CQG-MBQA, a framework designed to produce interpretable semantic text embeddings for diverse NLP tasks. The framework uses Contrastive Question Generation (CQG) to automatically generate meaningful yes/no questions without domain experts. The Multi-task Binary Question Answering (MBQA) model answers these questions, producing embeddings with human-interpretable dimensions at a much lower cost than answering with LLMs, while maintaining comparable accuracy. The authors validate CQG-MBQA through experiments, comparing it to black-box and interpretable models across STS, retrieval, and clustering. The experimental results show that CQG-MBQA offers better embedding quality than existing interpretable models and provides comparable embedding quality to several black-box models, maintaining high interpretability and efficiency.
weaknesses: - In retrieval tasks, there is a significant performance gap compared to black-box models, and the performance is also lower than BM25. Therefore, additional performance comparisons are needed when applying them to various downstream tasks such as sentiment classification and retrieval. 
- Lack of ablation studies to assess the efficacy of the proposed approach
  - lack of comparison between different models in Figure 4 and 5, and lack of comparison between the MBQA method and directly using the LLM’s outputs.
  - comparison between vanilla CQG with positive and hard/easy negative, and CQG with positive and negative samples
  - comparison between having and not having the probing mechanism to refine the generated questions
- Also, because the cognitive load is defined using the dot product, this measure would be directly influenced by the total number of questions. A normalized version (e.g., dot product divided by number of questions) would provide a fairer comparison across different interpretable models in Table 5.
- Having cost analysis would be beneficial as MBQA requires significant LLM inferences (or API calls) during training time and even more may be required during Post-processing (probing). 
- Including a case study on bad examples would also be beneficial—for instance, displaying cases where two texts result in similar embeddings even when those two texts do not necessarily have similar semantics. Are they completely off? Or how could one improve your approach?

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - What inspired this measurement of cognitive load?
- How much inference time does it take to run on the retrieval datasets?
- How were the retrieval datasets chosen?
- How much does the QA model quality affect embedding quality?
soundness: 2
strengths: - This paper tackles the important problem of creating interpretable text embeddings
- Some of the steps laid out in the "framework" explanation will be useful for other practitioners
- The source code is available and could be used by other researchers and engineers to build systems for interpretable embeddings
- Consideration of the tradeoff between interpretability and quality is interesting – although I have qualms with the "cognitive load" measurement of interpretability, which are mentioned below.
summary: This work proposes a "framework" for creating interpretable semantic embeddings. They tackle the important and relevant problem of creating embeddings that are useful for search & clustering but also understandable to humans.
weaknesses: - I find it important to point out that this paper isn't really proposing a "framework" in a very general sense; it's much closer to a method (and in fact the authors interchange the two words liberally throughout the paper). For this reason I object to calling it a framework at all and would prefer the paper to be about CQG-MBQA, which is an interesting and apparently effective method for interpretable
 text embeddings.
- As a related point, the organization is confusing. Currently the paper mixes together the "framework" part (which should be a general process for producing interpretable embeddings) with the "method" part (about CQG-MBQA) and also some "experimental setup" and "results"  (all in Section 3). As one example of the murky sectional boundaries, is post-processing really a necessary step of the framework?
- I'm also not sure if the Case Study is exactly a Case Study.
- The cognitive load metric seems unprincipled and lacks grounding in real cognitive science. 
- Cognitive load is simply the number of overlapping "yes" answers (or 1s in the embeddings) between the representations of a pair from an STS dataset. It is highly dependent on dimensionality and sparsity (Figure 4 & 5). It also doesn't really make sense because the interpretability of an embedding should depend on how many yes's there are for a pair *compared to other pairs*; embeddings cannot be understood simply by looking at the inner product of a pair of embeddings.  
- Many of the important design decisions in the framework are not ablated. Is filtering important? How much does choosing positive and negative samples matter, or clustering? How much does training a surrogate model affect performance?
- This is not necessary to me for accepting the paper, but a real human study could be crucial for arguing that these embeddings are in fact more interpretable
- Due to the complicated system and lack of ablations, it is not easy to understand why these embeddings outperform other interpretable embeddings such as QAEmb
- Unclear cost analysis of running this method on a downstream dataset

I think this citation could be relevant:
- Learning Interpretable Style Embeddings via Prompting LLMs (Patel et al., EMNLP Findings 2023)

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: Can the authors provide a comparison with the baseline including the sparsity penalty? Maybe showing the performance as a function of the number of questions kept?
soundness: 3
strengths: - The authors study an interesting and important problem
- They obtain strong performance results with reasonable efficiency
- They evaluate interpretability and cognitive load well, in addition to more standard performance metrics
summary: The authors introduce CQG-MBQA, a general framework for producing interpretable semantic text embeddings. It builds these embeddings from a set of yes/no questions that are designed to be highly discriminative (by separating text clustered by a pre-trained embedding model). To improve efficiency, the answers to these yes/no questions are distilled into a smaller model. The CQG-MBQA model reveals improvements relative to baseline interpretable models.
weaknesses: - The main issue seems to be the authors’ treatment of related work: the CGQ method generates questions through prompting and filters them based on their discriminative ability. The baseline QA-Emb also generates questions through prompting but filters them with a sparsity penalty. From their description, the authors don’t seem to implement the sparsity penalty, which likely skews the comparisons.
- The authors should discuss this [2023 style embeddings paper](https://arxiv.org/abs/2305.12696), which was an early precursor to the work here
- The authors should clarify whether distilling the yes/no answers into a single model is a novel contribution — the style embeddings paper & QA-Emb paper both seem to do this as well

Meta-review: This paper introduces a framework called CQG-MBQA to create interpretable text embeddings, which automatically produces yes/no questions that capture the differences between texts. 

The reviewers recognized the authors’ contributions to their proposed framework for generating discriminative questions while maintaining cost efficiency  (Reviewer SSgS, sYsh, RC4T). Also, the effectiveness of CQG-MBQA was acknowledged by the reviewers for comprehensive experiments across multiple tasks and strong baselines (Reviewer SSgS, sYsh, RC4T, vj78).

Concerns about limited or missing ablation studies on key design choices (e.g., question dimensionality, question difficulty, …) were raised (Reviewer SSgS, RC4T, vj78, 8TCb). In addition, the reviewer sYsh raised concerns about generalizability, specifically whether the method requires generating new sets of yes/no questions for each dataset and if a more universal question set could be designed. 

During the rebuttal, the authors made great efforts to address the reviewers’ concerns. For the main concern about lacking ablation studies, the authors added extensive experiments to demonstrate the performance of CQG-MBQA. For the concerns about generalizability, the authors also conducted experiments on four additional downstream tasks during the rebuttal period. Considering all factors, the AC therefore recommends acceptance of this paper.

Decision: Accept (Poster)