Crafting Interpretable Embeddings for Language Neuroscience by Asking LLMs Questions

Authors: vinamra  benara
chandan  singh
john x morris
richard j antonello
alexander g huth
jianfeng  gao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 14:03:32

Abstract: Large language models (LLMs) have rapidly improved text embeddings for a growing array of natural-language processing tasks. However, their opaqueness and proliferation into scientific domains such as neuroscience have created a growing need for interpretability. Here, we ask whether we can obtain interpretable embeddings through LLM prompting. We introduce question-answering embeddings (QA-Emb), embeddings where each feature represents an answer to a yes/no question asked to an LLM. Training QA-Emb reduces to selecting a set of underlying questions rather than learning model weights.

We use QA-Emb to flexibly generate interpretable models for predicting fMRI voxel responses to language stimuli. QA-Emb significantly outperforms an established interpretable baseline, and does so while requiring very few questions. This paves the way towards building flexible feature spaces that can concretize and evaluate our understanding of semantic brain representations. We additionally find that QA-Emb can be effectively approximated with an efficient model, and we explore broader applications in simple NLP tasks.

Main paper: 
 introduction: 
 Text embeddings are critical to many applications, including information retrieval, semantic clustering, retrieval-augmented generation, and language neuroscience. Traditionally, text embeddings leveraged interpretable representations such as bag-of-words or BM-25 [1] . Modern methods often replace these embeddings with representations from large language models (LLMs), which may better capture nuanced contexts and interactions [2] [3] [4] [5] [6] [7] . However, these embeddings are essentially black-box representations, making it difficult to understand the predictive models built on top of them (as well as why they judge different texts to be similar in a retrieval context). This opaqueness is detrimental in scientific fields, such as neuroscience [8] or social science [9] , where trustworthy interpretation itself is the end goal. Moreover, this opaqueness has debilitated the use of LLM embeddings (for prediction or retrieval) in high-stakes applications such as medicine [10] , and raised issues related to regulatory pressure, safety, and alignment [11] [12] [13] [14] .To ameliorate these issues, we introduce question-answering embeddings (QA-Emb), a method that builds an interpretable embedding by repeatedly querying a pre-trained autoregressive LLM with a set of questions that are selected for a problem (Fig. 1 ). Each element of the embedding represents Does the sentence include numerical information? Is the sentence a command? Is the sentence grammatically complex? Does the sentence mention a specific location or place? ….. Does the sentence include dialogue? Does the input describe a sensory experience? Does the sentence describe a relationship between people? Does the input mention time? Figure 1: QA-Emb produces an embedding for an input text by prompting an LLM with a series of yes/no questions. This embedding can then be used in downstream tasks such as fMRI response prediction or information retrieval.the answer to a different question asked to an LLM, making the embedding human-inspectable. For example, the first element may be the answer to the question Does the input mention time? and the output would map yes/no to 1/0. Training QA-Emb requires only black-box access to the LLM (it does not require access to the LLM internals) and modifies only natural-language prompts, rather than LLM parameters. The learning problem is similar to the optimization faced in natural-language autoprompting [15, 16] or single-neuron explanation [17, 18] , but seeks a set of questions rather than an individual prompt.We focus on a single neuroscience problem in close collaboration with neuroscientists. Grounding in a neuroscience context allows us to avoid common pitfalls in evaluating interpretation methods [19, 20] that seek to test "interpretability" generally. Additionally, this focus allows to more realistically integrate domain knowledge to select and evaluate the questions needed for QA-Emb, one of its core strengths. Nevertheless, QA-Emb may be generally applicable in other domains where it is important to meaningfully interpret text embeddings.In our neuroscience setting, we build QA-Emb representations from natural-language questions that can predict human brain responses measured by fMRI to natural-language stimuli. This allows for converting informal verbal hypotheses about the semantic selectivity of the brain into quantitative models, a pressing challenge in fields such as psychology [21] . We find that predictive models built on top of QA-Embs are quite accurate, providing a 26% improvement over an established interpretable baseline [22] and even slightly outperforming a black-box BERT baseline [23] . Additionally, QA-Emb yields concise embeddings, outperforming the interpretable baseline (that consists of 985 features) with only 29 questions.We investigate two major limitations of QA-Emb in Sec. 5. First, with regards to computational efficiency, we find that we can drastically reduce the computational cost of QA-Emb by distilling it into a model that computes the answers to all selected questions in a single feedforward pass by using many classification heads. Second, we evaluate the accuracy of modern LLMs at reliably answering diverse yes/no questions. Finally, Sec. 6 explores broader applications for QA-Emb in a simple information retrieval setting and text-clustering setting. 
methods: 
 QA-Emb is an intuitive method to generate text embeddings from a pre-trained autoregressive LLM (Fig. 1 ). Given a text input, QA-Emb builds an interpretable embedding by querying the LLM with a set of questions about the input. Each element of the embedding represents the answer to a different question asked to an LLM. This procedure allows QA-Emb to capture nuanced and relevant details in the input while staying interpretable.Learning a set of yes/no questions QA-Emb requires specifying a set of yes/no questions Q ∈ Q yes/no that yield a binary embedding v Q (x) ∈ {0, 1} d for an input string x. The questions are chosen to yield embeddings that are suitable for a downstream task. In our fMRI prediction task, we optimize for supervised linear regression: given a list of n input strings X and a multi-dimensional continuous output Y ∈ R nxd , we seek embeddings that allow for learning effective ridge regression models:EQUATION)where θ is a learned coefficient vector for predicting the fMRI responses and λ is the ridge regularization parameter.Directly optimizing over the space of yes/no questions is difficult, as it requires searching over a discrete space with a constraint set Q yes/no that is hard to specify. Instead, we heuristically optimize the set of questions Q, by prompting a highly capable LLM (e.g. GPT-4 [24] ) to generate questions relevant to our task, e.g. Generate a bulleted list of questions with yes/no answers that is relevant for {{task description}}. Customizing the task description helps yield relevant questions. The prompt can flexibly specify more prior information when available. For example, it can include examples from the input dataset to help the LLM identify data-relevant questions. Taking this a step further, questions can be generated sequentially (similar to gradient boosting) by having the LLM summarize input examples that incur high prediction error to generate new questions focused on those examples. While we focus on optimizing embeddings for fMRI ridge regression in Eq. ( 1), different downstream tasks may require different inner optimization procedures, e.g. maximizing the similarity of relevant documents for retrieval.Post-hoc pruning of Q. The set of learned questions Q can be easily pruned to be made compact and useful in different settings. For example, in our fMRI regression setting, a feature-selection procedure such as Elastic net [25] can be used to remove redundant/uninformative questions from the specified set of questions Q. Alternatively, an LLM can be used to directly adapt Q to yield taskspecific embeddings. Since the questions are all in natural language, they can be listed in a prompt, and an LLM can be asked to filter the task-relevant ones, e.g. Here is a list of questions:{{question list}} List the subset of these questions that are relevant for {{task description}}.Limitations: computational cost and LLM inaccuracies. While effective, the QA-Emb pipeline described here has two major limitations. First, QA-Emb is computationally intensive, requiring d LLM calls to compute an embedding. This is often prohibitively expensive, but may be worthwhile in high-value applications (such as our fMRI setting) and will likely become more tenable as LLM inference costs continue to rapidly decrease. We find that we can dramatically reduce this cost by distilling the QA-Emb model into a single LLM model with many classification heads in Sec. 5.1. Otherwise, LLM inference costs are partially mitigated by the ability to reuse the KV-cache for each question and the need to only generate a single token for each question. While computing embeddings with QA-Emb is expensive, searching embeddings is made faster by the fact that the resulting embeddings are binary and often relatively compact.Second, QA-Emb requires that the pre-trained LLM can faithfully answer the given yes-no questions.If an LLM is unable to accurately answer the questions, it hurts explanation's faithfulness. Thus, QA-Emb requires the use of fairly strong LLMs and the set of chosen questions should be accurately answered by these LLMs (Sec. 5.2 provides analysis on the question-answering accuracy of different LLMs).Hyperparameter settings For answering questions, we average the answers from Mistral-7B [26] (mistralai/Mistral-7B-Instruct-v0.2) and LLaMA-3 8B [27] (meta-llama/Meta-Llama-3-8B-Instruct) with two prompts. All perform similarly and averaging their answers yields a small performance improvement (Table A2 ). For generating questions, we prompt GPT-4 [24] (gpt-4-0125-preview). Experiments were run using 64 AMD MI210 GPUs, each with 64 gigabytes of memory, and reproducing all experiments in the paper requires approximately 4 days (initial explorations required roughly 5 times this amount of compute). All prompts used and generated questions are given in the appendix or on Github.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Yes, limitations are discussed in the methods section (Sec. 2) as well as the entirety of Sec. 5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
related work: 
 Text embeddings Text embeddings models, which produce vector representations of document inputs, have been foundational to NLP. Recently, transformer-based models have been trained to yield embeddings in a variety of ways [2] [3] [4] [5] [6] [7] , including producing embeddings that are sparse [28] or have variable lengths [29] . Recent works have also leveraged autoregressive LLMs to build embeddings, e.g. by repeating embeddings [30] , generating synthetic data [6, 31] , or using the lasttoken distribution of an autoregressive LLM as an embedding [32] . Similar to QA-Emb, various works have used LLM answers to multiple prompts for different purposes, e.g. text classification [33] [34] [35] , learning style embeddings [36] , or data exploration [37] .Interpreting representations A few works have focused on building intrinsically interpretable text representations, e.g. word or ngram-based embeddings such as word2vec [38] , Glove [39] , and LLM word embeddings. Although their dimensions are not natively interpretable, for some tasks, such as classification, they can be projected into a space that is interpretable [40] , i.e. a word-level representation. Note that it is difficult to learn a sparse interpretable model from these dense embeddings, as standard techniques (e.g. Elastic net) cannot be directly applied.When instead using black-box representations, there are many post-hoc methods to interpret embeddings, e.g. probing [41, 42] , categorizing elements into categories [43] [44] [45] [46] , categorizing directions in representation space [47] [48] [49] [50] , or connecting multimodal embeddings with text embeddings/text concepts [51] [52] [53] [54] [55] . For a single pair of text embeddings, prediction-level methods can be applied to approximately explain why the two embeddings are similar [56, 57] .Natural language representations in fMRI Using LLM representations to help predict brain responses to natural language has recently become popular among neuroscientists studying language processing [58] [59] [60] [61] [62] [63] (see [64, 65] for reviews). This paradigm of using "encoding models" [66] to better understand how the brain processes language has been applied to help understand the cortical organization of language timescales [67, 68] , examine the relationship between visual and semantic information in the brain [69] , and explore to what extent syntax, semantics, or discourse drives brain activity [22, [70] [71] [72] [73] [74] [75] [76] [77] 18] . The approach here extends these works to build an increasingly flexible, interpretable feature space for modeling fMRI responses to text data. 
main results: fmri interpretation: 
 A central challenge in neuroscience is understanding how and where semantic concepts are represented in the brain. To meet this challenge, we extend the line of study that fits models to predict the response of different brain voxels (i.e. small regions in the brain) to natural language stimuli. Using QA-Emb, we seek to bridge models that are interpretable [1, 22] with more recent LLM models that are accurate but opaque [58] [59] [60] .Dataset We analyze data from two recent studies [78, 79] (released under the MIT license), which contain fMRI responses for 3 human subjects listening to 20+ hours of narrative stories from podcasts.We extract text embeddings from the story that each subject hears and fit a ridge regression to predict the fMRI responses (Eq. ( 1)). Each subject listens to either 79 or 82 stories (consisting of 27,449 time points) and 2 test stories (639 time points); Each subject's fMRI data consists of approximately 100,000 voxels; we preprocess it by running principal component analysis (PCA) and extracting the coefficients of the top 100 components.We fit ridge regression models to predict these 100 coefficients and evaluate the models in the original voxel space (by applying the inverse PCA mapping and measuring the correlation between the response and prediction for each voxel). We deal with temporal sampling following [22, 60] ; an embedding is produced at the timepoint for each word in the input story and these embeddings are interpolated using Lanczos resampling. Embeddings at each timepoint are produced from the ngram consisting of the 10 words preceding the current timepoint. We select the best-performing hyperparameters via cross-validation on 5 time-stratified bootstrap samples of the training set. We select the best ridge parameters from 12 logarithmically spaced values between 10 and 10,000. To model temporal delays in the fMRI signal, we also select between adding 4, 8, or 12 time-lagged duplicates of the stimulus features.Generating QA-Emb questions To generate the questions underlying QA-Emb, we prompt GPT-4 with 6 prompts that aim to elicit knowledge useful for predicting fMRI responses (precise prompts in Appendix A.3). This includes directly asking the LLM to use its knowledge of neuroscience, to brainstorm semantic properties of narrative sentences, to summarize examples from the input data, and to generate questions similar to single-voxel explanations found in a prior work [18] . This process yields 674 questions (Fig. 1 and Table A1 show examples, see all questions on Github). We perform feature selection by running multi-task Elastic net with 20 logarithmically spaced regularization parameters ranging from 10 -3 to 1 and then fit a Ridge regression to the selected features. 2 See extended details on the fMRI experimental setup in Appendix A.1 and all prompts in Appendix A.3.Baselines We compare QA-Emb to Eng1000, an interpretable baseline developed in the neuroscience literature specifically for the task of predicting fMRI responses from narrative stories [22] . Each element in an Eng1000 embedding corresponds to a cooccurence statistic with a different word, allowing full interpretation of the underlying representation in terms of related words. We additionally compare to embeddings from BERT [23] (bert-base-uncased) and LLaMA models [81, 27] . For each subject, we sweep over 5 layers from LLaMA-2 7B (meta-llama/Llama-2-7b-hf, layers 6, 12, 18, 24, 30), LLaMA-2 70B (meta-llama/Llama-2-70b-hf, layers 12, 24, 36, 48, 60), and LLaMA-3 8B (meta-llama/Meta-Llama-3-8B, layers 6, 12, 18, 24, 30), then report the test performance for the model that yields the best cross-validated accuracy (see breakdown in Table A3 ).We find that QA-Emb predicts fMRI responses fairly well across subjects (Fig. 2A ), achieving an average test correlation of 0.116. QA-Emb significantly outperforms the interpretable baseline Eng1000 (26% average improvement). Comparing to the two transformer-based baselines (which do not yield straightforward interpretations), we find that QA-Emb slightly outperforms BERT (5% improvement) and worse than the best cross-validated LLaMA-based model (7% decrease). Trends are consistent across all 3 subjects.To yield a compact and interpretable model, Fig. 2B further investigates the compressibility of the two interpretable methods (through Elastic net regularization). Compared to Eng1000, QA-Emb improves performance very quickly as a function of the number of features included, even outperforming the final Eng1000 performance with only 29 questions (mean test correlation 0.122 versus 0.118).Table A1 shows the 29 selected questions, which constitute a human-readable description of the entire model.Fig. 2C -D further break down the predictive performance across different brain regions for a particular subject (S03). The regions that are well-predicted by QA-Emb (Fig. 2C ) align with language-specific areas that are seen in the literature [59, 82] . They do not show any major diversions from transformerbased encoding models (Fig. 2D ), with the distribution of differences being inconsistent across subjects (see Fig. A1 ).The QA-Emb representation enables not only identifying which questions are important for fMRI prediction, but also mapping their selectivity across the cortex. We analyze the QA-Emb model which uses 29 questions and visualize the learned regression weights for different questions. Fig. 3 shows example flatmaps of the regression coefficients for 3 of the questions across the 2 best-predicted subjects (S02 and S03). Learned feature weights for the example questions capture known selectivity and are highly consistent across subjects. In particular, the weights for the question "Does the sentence involve a description of a physical environment or setting?" captures classical place areas including occipital place area [83] and retrosplenial complex [84] , as well as intraparietal sulcus [85] . The weights for the question "Is the sentence grammatically complex?" bear striking similarity to the language network [82, 86] , which is itself localized from a contrast between sentences and nonwords. Other questions, such as "Does the sentence describe a physical action?", which has strong right 5 Evaluating the limitations of QA-Emb 
improving computational efficiency via model distillation: 
 To reduce the computational cost of running inference with QA-Emb, we explore distilling the many LLM calls needed to compute QA-Emb into a single model with many classification heads. Specifically, we finetune a RoBERTa model [87] (roberta-base) with 674 classification heads to predict all answers required for QA-Emb in a single feedforward pass. We finetune the model on answers from LLaMA-3 8B with a few-shot prompt for 80% of the 10-grams in the 82 fMRI training stories (123,203 examples), use the remaining 20% as a validation set for early stopping (30,801 examples), and evaluate on all 10-grams in the 2 testing stories (4,594 examples). We finetune using AdamW [88] with a learning rate of 5 • 10 -5 .When evaluated on the fMRI prediction task, the distilled model (QA-Emb (distill, binary) in Table 1 ) yields a performance only slightly below the original model. If we relax the restriction that the finetuned model yields binary embeddings and instead use the predicted probability for yes, the performance rises slightly to nearly match the original model (0.113 instead of 0.114 average test correlation) and maintains a significant improvement over the Eng1000 baseline. Note that the distilled model achieves an 88.5% match for yes/no answers on 10-grams for the test set. Nevertheless, the fMRI prediction for any given timepoint is computed from many questions and ngrams, mitigating the effect of individual errors in answering a question.We evaluate the faithfulness of our question-answering models on a recent diverse collection of 54 binary classification datasets [89, 90] (see data details in Table A4 ). These datasets are difficult, as they are intended to encompass a wider-ranging and more realistic list of questions than traditional NLP datasets.Fig. 4 shows the classification accuracy for the 3 LLMs used previously along with GPT-3.5 (gpt-3.5-turbo-0125). On average, each of the LLMs answers these questions with fairly high accuracy, with GPT-4 slightly outperforming the other models. However, we observe poor performance on some tasks, which we attribute to the task difficulty and the lack of task-specific prompt engineering. For example, the dataset yielding the lowest accuracy asks the question Is the input about math research?. While this may seem like a fairly simple question for an LLM to answer, the examples in the negative class consist of texts from other quantitative fields (e.g. chemistry) that usually contain numbers, math notation, and statistical analysis. Thus the LLMs answer yes to most examples and achieve accuracy near chance (50%). Note that these tasks are more difficult than the relatively simple questions we answer in the fMRI experiments, especially since the fMRI input lengths are each 10 words, whereas the input lengths for these datasets are over 50 words on average (with some inputs spanning over 1,000 words). 6 Secondary results: evaluating QA-Emb in simple NLP tasks 6.1 Benchmarking QA-Emb for information retrievalIn this section, we investigate applying QA-Emb to a simplified information retrieval task. We take a random subset of 4,000 queries from the MSMarco dataset ( [91] , Creative Commons License) and their corresponding groundtruth documents, resulting in 5,210 documents. We use 25% of the queries to build a training set and keep the remaining 75% for testing. For evaluation, we calculate the cosine similarity match between the embeddings for each query and its groundtruth documents using mean reciprocal rank and recall.To compute QA-Emb, we first generate 2,000 questions through prompting GPT-4 based on its knowledge of queries in information retrieval (see prompts in the Github). We use a regex to slightly rewrite the resulting questions for queries to apply to documents (e.g. Is this query related to a specific timeframe? → Is this text related to a specific timeframe?). We then answer the questions both for each query and for each corpus document, again using LLaMA-3 8B. Rather than fitting a ridge regression as in Eq. ( 1), we use the training set to learn a scalar for each question that multiplies its binary output to change both its sign and magnitude in the embedding (optimization details in Appendix A.4).Table 2 shows the information retrieval results. Combining BM-25 with QA-Emb achieves a small but significant improvement over the interpretable baselines. QA-Emb on its own achieves modest performance, slightly improving slightly over a bag-of-words representation, but significantly underperforming BM-25. Nevertheless, its size is considerably smaller than the other interpretable baselines making it quicker to interpret and to use for retrieval.  
zero-shot adaptation in text clustering: 
 We now investigate QA-Emb in a simplified text clustering setting. To do so, we study 4 textclassification datasets: Financial phrasebank ( [92] , creative commons license), Emotion [93] (CC BY-SA 4.0 license), AGNews [94] , and Rotten tomatoes [95] . For each dataset, we treat each class as a cluster and evaluate the clustering score, defined as the difference between the average inter-class embedding distance and the average intra-class embedding distance (embedding distance is measured via Euclidean distance). A larger clustering score suggests that embeddings are well-clustered within each class.In our experiment, we build a 100-dimensional embedding by prompting GPT-4 to generate 25 yes/no questions related to the semantic content of each dataset (e.g. for Rotten tomatoes, Generate 25 yes/no questions related to movie reviews). We then concatenate the answers for all 100 questions to form our embedding. These general embeddings do not yield particularly strong clustering scores (Table 3 top), as the questions are diverse and not particularly selective for each dataset.However, simply through prompting, we can adapt these general embeddings to each individual dataset. We call GPT-4 with a prompt that includes the full list of questions and ask it to select a subset of questions that are relevant to each task. The result embeddings (Table 3 bottom) yield higher clustering scores, suggesting that QA-Emb can be adapted to each task in a zero-shot manner (in this simplified setting). Moreover, the resulting task-specific embeddings are now considerably smaller. 
discussion: 
 We find that QA-Emb can effectively produce interpretable and high-performing text embeddings.While we focus on a language fMRI setting, QA-Emb may be able to help flexibly build an interpretable text feature space in a variety of domains, such as social science [9] , medicine [10] , or economics [96] , where meaningful properties of text can help discover something about an underlying phenomenon or build trust in high-stakes settings. Alternatively, it could be used in mechanistic interpretability, to help improve post-hoc explanations of learned LLM representations.As LLMs improve in both efficiency and capability, QA-Emb can be incorporated into a variety of common NLP applications as well, such as RAG or information retrieval. For example, in RAG systems such as RAPTOR [97] or Graph-RAG [98] , explanations may help an LLM not only retrieve relevant texts, but also specify why they are relevant and how they may be helpful.Learning text questions rather than model weights is a challenging research area, furthering work in automatic prompt engineering [15, 16] . Our approach takes a heuristic first step at solving this problem, but future work could explore more directly optimizing the set of learned questions Q in Eq. ( 1) via improved discrete optimization approaches and constraints. One possible approach may involve having LLMs themselves identify the errors the current model is making and improving based on these errors, similar to general trends in LLM self-improvement and autoprompting [99] [100] [101] [102] .Another approach may involve improving the explanation capabilities of LLMs to help extract more questions more faithfully from data [103, 104] .Broader Impacts QA-Emb seeks to advance the field of LLM interpretation, a crucial step toward addressing the challenges posed by these often opaque models. Although LLMs have gained widespread use, their lack of transparency can lead to significant harm, underscoring the importance of interpretable AI. There are many potential positive societal consequences of this form of interpretability, e.g., facilitating a better understanding of scientific data and models, along with a better understanding of LLMs and how to use them safely. Nevertheless, as is the case with most ML research, the interpretations could be used to interpret and potentially improve an LLM or dataset that is being used for nefarious purposes. Moreover, QA-Emb requires substantial computational resources, contributing to increased concerns over sustainability. Optimization details When fitting our QA-Emb model for information retrieval, we learn a single scalar per-question that is multiplied by each embedding before computing a similarity. To learn these scalars, we minimize a two-part loss. The first loss is the negative cosine similarity between each query and its similar documents. The second loss is the cosine similarity between each query and the remaining documents. We weight the first loss as 10 times higher than the second loss and optimize using Adam [105] with a learning rate of 10 -4 . We run for 8 epochs, when the training loss seems to plateau.Table A4 : 54 binary classification datasets along with their underlying yes/no question and corpus statistics from a recent collection [89, 90] . Justification: The claims are clearly stated that QA-Emb can generate flexible embeddings using a pre-trained LLM (described in the Sec. 2) and that this experimentally improves performance primarily for an fMRI prediction problem (Sec. 4).Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: Mentioned in the methods section.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Justification: We have reviewed the NeurIPS Code of Ethics and found that the current paper conforms to it.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: Given in the discussion section.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks beyond those discussed in the Broader Impacts above, as it releases no new data or models.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We list the licenses in the main text for all datasets where they can be found (the main fMRI data is released under the MIT license).Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.We run Elastic net using the MultiTaskElasticNet class from scikit-learn[80]. 
 
 Appendix:  


Decision: The paper proposes an approach QA-Emb for generating interpretable embeddings (with the primary application being fMRI response prediction where interpretability is highly valued).


They create this embedding based on prompting an LLM with a set of yes/no questions. This yes/no question strategy has been used before in fMRI but the authors' contribution is that they use an LLM to generate these questions and also automatically prune them (so they do not have to be manually constructed). 
 
I think this is a nice application of LLM prompting the fMRI and so support accepting the paper contigent on the authors' proposed title change. (The NLP results of the paper are not strong but I think the authors intended the main application to be about fMRI).

Reviews: review: paper_summary: paper_summary: The authors introduce question-answering embeddings (QA-Emb), where each feature in the embedding corresponds to an answer to a yes/no question asked to an LLM (e.g., LLaMA-3 8B). QA-Emb significantly outperforms an established interpretable baseline in predicting fMRI voxel responses to language stimuli. The authors also evaluate QA-Emb on information retrieval and text clustering tasks. Additionally, computational efficiency can be improved by distilling the QA-Emb model into a RoBERTa-base model.
strength_weakness: strength_weakness: (1) Motivation:
The paper aims to extract meaningful text embeddings from LLMs. However, the entire QA-Emb pipeline seems to rely less on LLMs than suggested, as these questions could also be answered by a much smaller QA model, such as RoBERTa-base, which the authors have explored in Section 5.1.

(2) Baselines:
While there are existing methods that extract text embeddings from LLMs [1,2], these methods are not compared in the paper. It is reasonable to expect that QA-Emb might perform worse than these non-interpretable methods, but a comparison with these methods would provide a clearer understanding of QA-Emb's performance.

(3) Results:
The authors claim that QA-Emb is a high-performing embedding method. However, results on the MS-MARCO tasks indicate otherwise. QA-Emb significantly lags behind BM25, while a recent LLM-based embedding method like PromptReps [1] is able to match the performance of BM25 on retrieval tasks.

[1] Ting Jiang, Shaohan Huang, Zhongzhi Luan, Deqing Wang, Fuzhen Zhuang. Scaling Sentence Embeddings with Large Language Models. Arxiv.

[2] Shengyao Zhuang, Xueguang Ma, Bevan Koopman, Jimmy Lin, Guido Zuccon. PromptReps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document Retrieval. Arxiv.
questions: questions: Please see my comments in Weakness.
limitations: limitations: The authors provide detailed discussions on limitations in Sections 2 and 5.
paper_summary: paper_summary: The paper proposes a method of prompting LLMs with a list of yes or no questions to obtain binary embeddings for texts. The list of questions is generated by prompting the LLM with task-specific knowledge. The proposed method, QA-Emb, is evaluated primarily on predicting fMRI voxel responses to texts. QA-Emb outperforms an established interpretable baseline but underperforms compared to a black-box LLM embedding baseline. Additional experiments in information retrieval indicate that QA-Emb significantly underperforms BM25 but performs better than simple n-grams.
strength_weakness: strength_weakness: 1. The method of prompting LLMs to generate interpretable features is not entirely novel. For example, "CHiLL: Zero-shot Custom Interpretable Feature Extraction from Clinical Notes with Large Language Models" also prompts LLMs to ask questions for a classification task. A deeper investigation into the question generation phase would be more interesting and challenging. The paper could benefit from a more detailed discussion on systematically selecting and validating these questions to ensure robustness across various tasks.

2. While the method performs well in predicting fMRI voxel responses, it significantly underperforms in the information retrieval task compared to BM25. This raises doubts about the method's generalizability to other domains and tasks. If the method only excels in predicting fMRI voxel responses, it may be more accurately described as a specialized model for this task rather than a general "interpretable embedding" model.

3. The QA-Emb pipeline is computationally intensive, requiring multiple LLM calls to compute an embedding, which can be prohibitively expensive.
questions: questions: 1. More experiments on different domains and tasks could help to show the generalizability of the method. Especially, on embedding related benchmarks, and against some interpretable and black-box baselines. 

2. What strategies can be employed to mitigate the computational cost of QA-Emb?
limitations: limitations: Yes.
paper_summary: paper_summary: The paper explores obtaining interpretable embeddings through LLM prompting. To address the opaque nature of text embeddings, the authors introduce question-answering embeddings (QA-Emb) by asking LLMs a set of yes/no questions. Specifically, these questions are generated by GPT-4 using predefined prompts. Then, Elastic net is adopted for feature selection. The validity of QA-Emb is tested primarily in a fMRI interpretation problem. Moreover, the paper explores the computational efficiency of the method by finetuning a RoBERTa model and extends the QA-Emb on information retrieval and text clustering.
strength_weakness: strength_weakness: - The generation of questions for QA-Emb relies on manually crafted prompts, raising concerns about the robustness of the method. It remains unclear whether the efficacy of QA-Emb would persist with different sets of prompts. Moreover, The inconsistency in the number of questions generated from each prompt (500, 100, 200, 100, 100 respectively) introduces additional complexity. It remains unclear how to determine the optimal number of generated questions for each prompt.
- The method is computationally intensive, as it requires prompting LLMs with hundreds of questions for each sample. While the authors attempt to mitigate this by fine-tuning RoBERTa, it is questionable whether a smaller model like RoBERTa can maintain performance across diverse applications with more complex questions. Furthermore, the fine-tuning process itself depends on prompting LLMs over substantial examples. 
- The demonstrated effectiveness of QA-Emb is limited to a single neuroscience problem. Its performance in information retrieval and text clustering appears underwhelming. The approach would gain more credibility if shown effective across more scenarios.
questions: questions: Could you provide more details on the method and prompt for the posthoc pruning of generated questions in the experiments?
limitations: limitations: Limitations are included.
paper_summary: paper_summary: This study proposes QA-embedding for natural language text for multiple downstream tasks. Early embedding methods like bag-of-words and BM-25 cannot capture nuanced semantic feature of a sentence, and recent works primarily utilize the hidden states of large language models (LLM) as text embeddings, which lack of interpretability. Comparing to these embedding architectures, QA-embedding construct interpretable representations of texts, which can ameliorate the influence of opaqueness in scientific fields. In QA-embedding model procedures, it collects evaluation questions which is relevant to specific task firstly, which is achieved by prompting GPT-4 with corresponding information and eradicate redundancy through an Elastic. Then, QA-embedding model builds a {0,1} vectors by transforming yes/no responses to 1/0 which are obtained from asking a QA model with evaluation questions. Moreover, QA-embeddings are transformed into representations suitable for downstream tasks through parameters fitted by ridge regression. To evaluate QA-embedding, they perform experiment on fMRI, information retrieval, text clustering task and outperforms baselines. Meanwhile, they mention the limitation of QA-embedding, such as computational cost and LLM inaccuracies. To comprehend the limitation in depth, they also conduct analysis and put forward possible improvements.
strength_weakness: strength_weakness: 1. Some implementation procedures in line 149-165 should be clearly described, including PCA, inverse PCA and the sampling process, to clarify the task setup.
2. Although computational cost can be alleviated by distilling, QA model’s reliability still debilitate the interpretability of QA-embedding. According to experiment result in 5.2, merely 80% dimensions of embedding vectors are reliable, which means QA-embedding cannot consistently control the accuracy of each dimension.
questions: questions: 1. Is there an analysis on the time efficiency of QA-embedding? Although you proposed solution to decrease the computational cost, a comparison among before and after distilling and other embedding methods will make the limitation clearer.
2. In section 5.2, did you test the classification accuracy on your distilled model RoBERTa? It seems that you treat distilled RoBERTa-base as the QA model in previous experiment, and you prove distilled RoBERTa’s ability on fMRI task, thus its classification skill or behaviour on other tasks can demonstrate the effect of your distillation technique.
3. Since QA-embedding is strongly interpretable, have you tried to make error analysis on downstream tasks? The aptitude of tracing back to the essence of model’s prediction errors will make QA-embedding competitive.
limitations: limitations: Yes

Comments: title: Response to the Rebuttal
Thanks for your recommendations
General response
comment: I appreciate the authors’ responses during the rebuttal period. However, most of my concerns remain unresolved. The performance appears to vary in a wide range based on the choice of prompt, and the computational cost is still a concern. Additionally, I am skeptical about the decision to shift the focus of the paper towards neuroscience. While this shift may mitigate QA-Emb’s performance limitations in other tasks, it also diminishes the contribution of this work in generating interpretable embeddings for a broader context.

Overall, this paper tackles the important issue of generating interpretable embeddings by LLMs, but it also presents clear weaknesses. Therefore, I decide to maintain my original assessment, which reflects my intention to encourage the exploration in this underexplored direction.
Thanks for your response.
Thank you for your time and recommendations.

1. **Re: LLM-answering accuracy.** Indeed, QA-Emb requires that the pre-trained LLM can faithfully answer the given yes-no questions. The result showing that different LLMs get ~82% accuracy (in Sec 5.2) is on a set of datasets that are both difficult and noisy. We anecdotally have two of our paper’s authors answer 100 questions at random on this dataset and they achieve 84% and 78%. For the main fMRI setting we focus on, these difficulties are rare & answering these questions is much simpler, especially since the fMRI input lengths are each 10 words, whereas the input lengths for the datasets in Sec 5.2 are over 50 words on average (with some inputs spanning over 1,000 words).
2. **Re: averaging LLMs.** When averaging embeddings from multiple LLMs, we allow for non-binary values, which then represent a probability of a "yes" answer (e.g. 1/3 if averaging a single "yes" and two "no"s). 
    1. We compare using the final-layer embeddings from our finetuned RoBERTa model to the QA-Emb model in Table 1. We find that they yield about the same performance: 0.112 for finetuned RoBERTa vs 0.113 for QA-Emb.
3. **Re: batching.** Indeed we use batching to speed up computing embeddings (both over different questions and examples), we will note this in the Methods.
We thank the reviewers for their thoughtful comments. We have made several changes to the paper in the revised version to incorporate the feedback, resulting in a much improved paper. The biggest change is to better focus the paper on the fMRI results, which are the main experiments and motivation for the paper. In the fMRI literature, interpretable encoding models are extremely desirable, but have largely been displaced by LLM-based black-box models that yield superior predictive performance. The QA-Emb approach is an effective way of yielding interpretable models that roughly match SOTA fMRI prediction performance and is a wholly novel approach in the encoding model space .

Our changes include edits throughout the paper and a title change to “Crafting Interpretable Embeddings **for Language Neuroscience** by Asking LLMs Questions”. These changes are relatively minor framing changes, as the results / experiments throughout are already focused on this application (everything but Sec 6). For example, the introduction reads:

> Grounding in a neuroscience context allows us to avoid common pitfalls in evaluating interpretation methods [19, 20] that seek to test “interpretability” generally. Additionally, this focus allows to more realistically integrate domain knowledge to select and evaluate the questions needed for QA-Emb, one of its core strengths.

The reviewers seem to agree that the proposed method is a useful contribution to the area of fMRI language neuroscience. We hope that in this case, the paper is suitable for publication at NeurIPS, as NeurIPS remains a major venue for neuroscience work, and this particular paper is a submission to the “Primary Area: Neuroscience and cognitive science” with Keywords “fMRI, Encoding models, Neuroscience, Language neuroscience, Interpretability, Large language models, Explainability, Brain mapping”.