Crafting Interpretable Embeddings for Language Neuroscience by Asking LLMs Questions
Authors: Vinamra Benara
Chandan Singh
John Xavier Morris
Richard Antonello
Ion Stoica
Alexander Huth
Jianfeng Gao
Venue: NeurIPS 2024 poster (2024)
Keywords: fMRI
Encoding models
Neuroscience
Language neuroscience
Interpretability
Large language models
Explainability
Brain mapping
Primary area: neuroscience_and_cognitive_science

Abstract: Large language models (LLMs) have rapidly improved text embeddings for a growing array of natural-language processing tasks. However, their opaqueness and proliferation into scientific domains such as neuroscience have created a growing need for interpretability. Here, we ask whether we can obtain interpretable embeddings through LLM prompting. We introduce question-answering embeddings (QA-Emb), embeddings where each feature represents an answer to a yes/no question asked to an LLM. Training QA-Emb reduces to selecting a set of underlying questions rather than learning model weights.

We use QA-Emb to flexibly generate interpretable models for predicting fMRI voxel responses to language stimuli. QA-Emb significantly outperforms an established interpretable baseline, and does so while requiring very few questions. This paves the way towards building flexible feature spaces that can concretize and evaluate our understanding of semantic brain representations. We additionally find that QA-Emb can be effectively approximated with an efficient model, and we explore broader applications in simple NLP tasks.

Crafting Interpretable Embeddings for Language Neuroscience by Asking LLMs Questions
Vinamra Benara* UC Berkeley
Chandan Singh Microsoft Research
John Morris Cornell University
Richard J. Antonello UT Austin
Ion Stoica UC Berkeley
Alexander G. Huth Austin
Jianfeng Gao Microsoft Research
*Equal contribution
Abstract
Large language models (LLMs) have rapidly improved text embeddings for grow- ing array of natural-language processing tasks. However, their opaqueness and proliferation into scientific domains such as neuroscience have created growing need for interpretability. Here, ask whether beddings through LLM prompting. We introduce question-ans (QA-Emb), embeddings where each feature represents answer to yes/no ques- tion asked to an LLM. Training QA-Emb reduces to selecting set of underlying questions rather than learning model weights.
obtain interpretable em- embeddings
We use QA-Emb to flexibly generate interpretable models for predicting fMRI voxel responses to language stimuli. QA-Eml significantly outperforms an established interpretable baseline, and does so while requiring very few questions. This paves the way towards building flexible feature spaces that concretize and evaluate our understanding of semantic brain representations. We additionally find that QA-Emb can be effectively approximated with an efficient model, and we explore broader applications in simple NL.P tasks.!
Introduction
Text embeddings critical many applications, including information retrieval, semantic clustering, retrieval-ane aarmentl- generation, and language neuroscience. Traditionally, text embeddings leveraged interpretable representations such bag-of-words BM-25 [1]. Modern methods often replace these embeddings with representations from large language models (LLMs),, which may better capture nuanced contexts and interactions [2-7]. However, these embeddings
essentially block-box representations, making it difficult to understand the predictive models built on top of them (as well why they judge different texts to be similar in retrieval context). This opaqueness is detrimental in scientific fields, such as neuroscience [8] or social science [9], where trustworthy interpretation itself is the end goal. Moreover, this opaqueness has debilitated the use of LLM embeddings (for prediction or retrieval) in high-stakes applications such as medicine [10], and raised issues related to regulatory pressure, safety, and alignment [11-14].
To ameliorate these issues, we introduce iresttio--answeingg embeddings (QA-Emb), a method that builds interpretable embedding by repeatedly querying pre-trained autoregressive LLM with set of questions that are selected for problem (Fig. 1). Each element of the embedding represents
All code for QA-Emb is made available on Github at ? gutta.ccm/cciisssneeeeeeeemeeeee
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Answer yes/no to the following question:
Embedding
Actual response
Does the sentence include numerical information?
M Way
the sentence command?
D
Predict fMRI response
sentence gremmistically complex?
Input text
Does the sentence mention specific location place?
0
Predicted response
I started to walk in besutilu!
Map yes/no
1/0
+
reighborhoed. It
LLM
10:30 night
Does the seritence include dislogue?
Does the input describe a sensory experience?
Retrieve
Does sentence describe relationship between people?
Does input montlon time?
Figure 1:
QA-Emb produces an embedding for an input text by prompting an LLM with series of yes/no questions. This embedding can then be used in downstream tasks such as fMRI response prediction or information retrieval.
the answer to a different question asked to an LLM, making the embedding human-inspectable For example, the first element may be the answer to the question Does the input mention time? and the output would map yes/no to 1/0. Training QA-Emb requires only black-box access to the LLM (it does not require access to the LLM internals) and modifies only natural- -lanuuae prompts, rather than LLM parameters. The learning problem is similar to the optimization faced in natural- languae autoprompting [15, 16] or single-neuron explanation [17, 18], but seeks a set of questions rather than an individual prompt.
We focus on a single neuroscience problem in close collaboration with neuroscientists. Grounding in neuroscience context allows us to avoid common pitfalls in evaluating interpretation methods [19, 20] that seek test "interpretability" generally. Additionally, this focus allows to more realistically integrate domain knowledge to select and evaluate the questions needed for QA-Emb, one of its core strengths. Nevertheless, QA-Emb may be generally applicable other domains where is important to meaningfully interpret text embeddings.
In neuroscience setting, build QA-Emb representations from natural-language questions that can predict human brain responses measured by fMRI to natural-language stimuli. This allows for converting informal verbal hypotheses about the semantic selectivity of the brain into quantitative models, a pressing challenge in fields such as psychology [21]. We find that predictive models built on top of QA-Embs are quite accurate, providing a 26% improvement over an established interpretable baseline [22] and even slightly outperforming a black-box BERT baseline [23]. Additionally, QA- Emb yields concise embeddings, outperforming the interpretable baseline (that consists of 985 features) with only 29 questions.
We investigate two major limitations of QA-Emb in Sec. 5. First, with regards to computational efficiency, we find that we can drastically reduce the computational cost of QA-Emb by distilling it into model that computes the answers to all selected questions in single feedforward pass by using many classification heads. Second, we evaluate the accuracy of modern L.MMs at reliably answering diverse yes/no questions. Finally, Sec. 6 explores broader applications for QA-Emb in a simple information retrieval setting and text-clustering setting.
2 Methods
QA-Emb is an intuitive method to generate text embeddings from a pre-trained autoregressive LLM (Fig. 1). Given a text input, QA-Emb builds an interpretable embedding by querying the LLM with a set of questions about the input. Each element of the embedding represents the answer to a different question asked to an L.M. This procedure allows QA-Emb capture nuanced and relevant details in the input while staying interpretable.

---

Learning a set of yes/no questions QA-Emb requires specifying set of yes/no questions Q E Qyesho that yield a binary embedding fo(z) E 10, for an input string z. The questions are chosen to yield embeddings that are suitable for downstream task. In our fMRI prediction task, we optimize for supervised linear regression: given list of n input strings X and a multi-dimensional continuous output Y E IR we seek embeddings that allow for learning effective ridge regression models:
Q arggmnn
min BER
Σ ( 0" ox ||
1110112
QE Syesho
where is a learned coefficient vector for predicting the IMRI responses and À is the ridge regulariza- tion parameter.
Directly optimizing over the space of yes/no questions is difficult, as requires searching over a discrete space with constraint set Qyesho that is hard to specify. Instead, we heuristically optimize the set of questions Q, by prompting a highly capable LLM (e.g. GPT-4 [24]) to generate questions relevant to our task, e.g. Generate bulleted list of questions with yes/no answers that is relevant for [[task description].. Customizing the task description helps yield relevant questions. The prompt can flexibly specify more prior information when available. For example, it can include examples from the input dataset to help the LLM identify data-relevant questions. Taking this a step further, questions can be generated sequentially (similar to gradient boosting) by having the LLM summarize input examples that incur high prediction error to generate new questions focused on those examples. While we focus optimizing embeddings for fMRI ridge regression in Eq. (I), different downstream tasks may require different inner optimization procedures, e.g. maximizing the similarity of relevant documents for retrieval.
Post-hoc pruning of Q. The set of learned questions Q can be easily pruned to be made compact and useful different settings. For example, in our fMRI regression setting, a feature-selection procedure such as Elastic net [25] can be used to remove redundan/uninformatiee questions from the specified set of questions Q. Alternatively, an LLM can be used to directly adapt Q to yield task- specific embeddings. Since the questions
all in natural language, they can be listed in a prompt, and an LLM can be asked to filter the task-relevant ones, e.g. Here is a list of questions: (aatiiio:::(qusstiie [[question ] List the subset of these questions that are relevant for ((task description)).
Limitations: computational cost and LLM inaccuracies. While effective, the QA-Emb pipeline described here has two major limitations. First, QA-Emb is computationally intensive, requiring d LLM calls to compute an embedding. This is often prohibitively expensive, but may be worthwhile in high-value applications (such as our fMRI setting) and will likely become more tenable as LLM inference costs continue to rapidly decrease. We find that we can dramatically reduce this cost by distilling the D--Emb model into single LLM model with many classification heads in Sec. 5.1. Otherwise. LLM inference costs are partially mitigated by the ability to reuse the KV-cache for each question and the need to only generate a single token for each question. While computing embeddings with Qm-Emb is expensive, searching embeddings is made faster by the fact that the resulting embeddings are binary and often relatively compact.
Second, QA-Emb requires that the pre-trained LLM can faithfully answer the given yes-no questions. If an LLM is unable to accurately answer the questions, it hurts explanation's 's faithfulness. Thus, QA-Emb requires the use of fairly strong LLMs and the set of chosen questions should be accurately accuracy of different
answered by these LLMs (Sec. 5.2 provides analysis on the LLMs).
Hyperparameter settings For answering questions, we average the answers from -ttpa//issaaaa-----ssse--
Mistral-7B [26] (meta- 1aae--laaaMMMtt--L -- Instruucst with two prompts. All perform similarly and averaging their answers yields small performance improvement (Table A2). For generating
2) and LLaMA-3 8B [27] questions, we prompt GPT-4 [24] (gpt-4-0125-previa
Experiments were run using 64 AMD
MI210 GPUs, each with 64 gigabytes of memory, and reproducing all experiments in the paper requires approximately days (initial explorations required roughly 5 times this amount of compute). All prompts used and generated questions are given in the appendix on Github.
3

---

3
Related work
Text embeddings Text embeddings models, which produce vector representations of document inputs, have been foundational to NLP. Recently, transformer-basee models have been trained to yield embeddings
variety of ways [2-7], including producing embeddings that are sparse [28] or have variable lengths [29]. Recent works have also leveraged autoregressive LLMs to build embeddings, e.g. by repeating embeddings [30], generating synthetic data [6, 31J, using the last- token distribution of an autoregressive LLM as an embedding [32]. Similar to QA-Emb, various works have used LLM answers to multiple prompts for different purposes, e.g. text classification [33-35], learning style embeddings [36], or data exploration [37].
Interpreting representations A few works have focused on building intrinsically interpretable text representations, e.g. word or ngram-based embeddings such as word2vec [38], Glove [39], and LLM word embeddings. Although their dimensions are not natively interpretable, for some tasks, such as classification, they can be projected into a space that is interpretable [40], i.e. a word-level representation. Note that it is difficult to learn sparse interpretable model from these dense embeddings, as standard techniques (e.g. Elastic net) cannot be directly applied.
When instead using black-box representations, there many post-hoc methods to interpret embed- dings, e-g. probing [41, 42], categorizing elements into categories [43-46], categorizing directions in representation space [47-50], connecting multimodal embeddings with text embeddings/text concepts [51-55 (51-55]. For a single pair of text embeddings, prediction-leve methods can be applied to approximately explain why the two embeddings are similar [56, 57].
Natural language representations in fMRI Using LLM representations to help predict brain responses to natural language has recently become popular among neuroscientists studying language processing [58-63] (see [64, 65] for reviews). This paradigm of using "encoding models" [66] to better understand how the brain processes language has been applied to help understand the cortical organization of language timescales [67, 68), examine the relationship between visual and semantic information in the brain [69], and explore what extent syntax, semantics, discourse drives brain activity [22, 70-77, 18]. The approach here extends these works to build an increasingly flexible, interpretable feature space for modeling fMRI responses to text data.
4 Main results: fMRI interpretation
central challenge in neuroscience is understanding how and where semantic concepts are represented in the brain. To meet this challenge, we extend the line of study that fits models to predict the response of different brain voxels (i.e. small regions in the brain) to natural language stimuli. Using QA-Emb, we seek to bridge models that are interpretable [1, 22] with more recent LLM models that are accurate but opaque [58-60].
4.1 fMRI experimental setup
Dataset We analyze data from two recent studies [78, 79] (released under the MIT license). which contain fMRI responses for human subjects listening to 20+ hours of narrative stories from podcasts. We extract text embeddings from the story that each subject hears and fit ridge regression to predict the fMRI responses (Eq. (I)). Each subject listens to either 79 or 82 stories (consisting of 27,449 time points) and 2 test stories (639 time points); Each subject's fMRI data consists of approximately 100,000 voxels; we preprocess by running principal component analysis (PCA) and extracting the coefficients of the top 100 components.
Regression modeling We fit ridge regression models predict these 100 coefficients and evaluate the models in the original voxel space (by applying the inverse PCA mapping and measuring the correlation between the response and prediction for each voxel). We deal with temporal sampling following [22, 60]; an embedding is produced at the timepoint for each word in the input story and these embeddings are interpolated using Lanczos resampling. Embeddings at each timepoint are produced from the ngram consisting of the 10 words preceding the current timepoint. We select the best-perforning hyperparameters via cross-validation on 5 time-stratified bootstrap samples of the training set. We select the best ridge parameters from 12 logarithmically spaced values between 10

---

and 10,000. To model temporal delays in the fMRI signal, we also select between adding 4, 8, or 12 time-lagged duplicates of the stimulus features.
Generating QA-Emb questions To generate the questions underlying QA-Emb, we prompt GPT-4 with 6 prompts that aim to elicit knowledge useful for predicting fMRI responses (precise prompts in Appendix A.3). This includes directly asking the LLM to use its knowledge of neuroscience, to brainstorm semantic properties of narrative sentences, to summarize examples from the input data, and to generate questions similar single-voxes explanations found in prior work [18]. This process yields 674 questions (Fig. 1 and Table A show examples, see all questions on Github). We perform feature selection by running multi-task Elastic net with 20 logarithmically spaced regularization parameters ranging from 10-3 to 1 and then fit a Ridge regression to the selected features." 2 See extended details on the IMRI experimental setup in Appendix A. and all prompts in Appendix A.3.
Baselines We compare QA-Emb to Eng1000, an interpretable baseline developed in the neuro- science literature specifically for the task of predicting fMRT responses from narrative stories [22]. Each element in an Eng 1000 embedding corresponds to cooccurence statistic with different word, allowing full interpretation of the underlying representation in terms of related words. We additionally compare embeddings from BERT [23] (bert-bbse-uuceee
and LLaMA models [81, 27]. For aaaammmmammaaaa-
layers layers 12, 24, 36, 48, 60), 6, 12, 18, 24, 30), LLaMA-2 70B --a--m---m-- layers 6, 12, 18, 24, 30), then report the test and LLaMA-3 8B (meta-llamd) Mamma/MMta-Laat performance for the model that yields the best crass-validated accuracy (see breakdown in Table A3).
each subject, we sweep over 5 layers from LLaMA-2 7B
4.2 [MRI predictive performance
We find that QA-Emb predicts fMRI responses fairly well across subjects (Fig. 2A), achieving an average test correlation of 16. QA-Emb significantly outperforms the interpretable baseline Eng 1000 (26% average improvement.. Comparing to the two transformer-based baselines (which do not yield straightforward interpretations), we find that QA-Emb slightly outperforms BERT (5% improvement) and worse than the best vrlld-validated LLaMA-based model (7% decrease). Trends are consistent across all subjects.
To yield a compact and interpretable model, Fig. 2B further investigates the compressibility of the two interpretable methods (through Elastic net regularization). Compared Eng QA-Emb improves performance very quickly as a function of the number of features included, even outperforming the final Eng1000 performance with only 29 questions (mean test correlation 0. 122 versus 0. 18). Table A1 shows the 29 selected questions, which constitute a human-readable description of the entire model.
Fig. 2C-D further break down the predictive performance across different brain regions for a particular subject (S03). The regions that are well-predien predicted by QA-Emb (Fig. 2C) align with sanguage-speciffe areas that are seen in the literature [59, 82). They do not show any major diversions from transformer- based encoding models (Fig. 2D), with the distribution of differences being inconsistent across subjects (see Fig. A1).
4.3 Interpreting the fitted representation from QA-Emb
The QA-Emb representation enables not only identifying which questions are important for IMRI prediction, but also mapping their selectivity across the cortex. We analyze the QA-Emb model which uses 29 questions and visualize the learned regression weights for different questions. Fig. shows example flatmaps of the regression coefficients for 3 of the questions across the 2 best-predicted subjects (S02 and S03). Learned feature weights for the example questions capture known selectivity and are highly consistent across subjects. In particular, the weights for the question "Does the sentence involve a description of physical environment or setting?" captures classical place areas including occipital place area [83] and retrosplenial complex [84], as well as intraparietal sulcus [85]. The weights for the question "Is the sentence grammatically complex?" bear striking similarity to the language network [82, 86), which is itself localized from contrast between sentences and nonwords. Other questions, such as "Does the sentence describe a physical action?", which has strong right
We run Elastic net using the MulttiaskklatttcNeet class from scikit-learn [80].
5

---

A
B
Eng1000
QA-Emb
0.14
BERT
0.12
LLaMA
Test conelation
0.13
0.10
0.12
Test
0.08
0.11
QA-mbs
0.06
Eng1000
0.10
S01
AVG
600
C
Subject
D
Number of features
-044 -012
0.0
0.4
0.6
-0.55 -0.00 -0.5
6d5 61s 0.15
Test correlation (QA-Emb)
Test correlation (QA-Emb BERT)
Figure 2: Predictive performance for QA-Emb compared to baselines. (A) Test correlation for QA- Emb outperforms the interpretable Eng 1000 baseline, is on par with the black-box BERT baseline, and is worse than the best-perforning LLaMA model. as a function of number included questions. (C) Test correlation per voxel for QA-Emb. (D) Difference in the test correlation per voxel for subject between QA-Emi and BERT. Error bars for (A) and (B) (standard error of the mean) are within the points (all are below 0.001). (B), (C), and (D) show results for subject S03.
(B) Test correlation for method quickly grows
Table 1: Mean test correlation when comparing QA-Emb computed via many LLM calls to QA-Emb computed via a single distilled model. Distillation does not significantly degrade performance. All standard errors of the mean are below 10
QA-Emb
QA-Emb (distill, binary)
QA-Emb (distill, probabilistic)
Eng1000
UTS01
0.081
0.083
0.080
0.077
UTS02
0.124
0.118
0.118
0.096
UTS03
0.136
0.132
0.142
0.117
AVG
0.114
0.111
0.113
0.097
laterality, do not have a strong basis in prior literature. These questions point to potentially new insights into poorly understood cortical regions.
Evaluating the limitations of QA-Emb
5.1 Improving computational efficiency via model distillation
To reduce the computational cost of rumning inference with QA-Emb, we explore distilling the many LLM calls needed to compute QA-Emb into single model with many classification heads. Specifically, we finetune a RoBERTa model [87] (roberta-base) with 674 classification heads to predict all answers required for QA-Emb in single feedforward pass. We finetune the model on answers from ILaMA-3 8B with few-shot prompt for 80% the 10-grams in the fMRI training stories (123,203 examples), use the remaining 20% validation set for early stopping (30,801

---

Does the sentence involve cleccription of a physical environment or setting?
Does the sentence describe physical action?
Is the sentence grammatica ily comp ex?
Kiwe
RESC
UPS
S02
ordon
ptamp
plómp
OPA
RFC
IPG
Prenp
S03
Rescaled Feature Weight
-2.0 -025 000
100
Figure 3: Learned feature weights for example questions capture known selectivity and are consistent across subjects. All feature weights are jointly rescaled to the range (-1, 1) for visualization. Abbreviations: Pr precuneus, pTemp posterior temporal cortex, PFC = prefrontal cortex, IPS intraparietal sulcus, RSC = retrosplenial complex, OPA = occipital place area, PPA = parahippocampal place area, Broca- Broca's area, sPMv superior premotor ventral speech area, AC auditory cortex.
examples), and evaluate on all 10- 10-grams in the testing stories (4.594 examples). We finetune using AdamW [88] with learning rate of 5 10-5.
When evaluated on the fMRI prediction task, the distilled model (QA-Emb (distill, binary) in Table 1) yields a performance only slightly below the original model. we relax the restriction that the finetuned model yields binary embeddings and instead use the predicted probability for yes, the performance rises slightly to nearly match the original model (0.113 instead of 0.114 average test correlation) and maintains significant improvement over the Eng 100 baseline. Note that the distilled model achieves an 88.5% match for yes/no answers on 10-grams for the test set. Nevertheless, the fMRI prediction for any given timepoint is computed from many questions and ngrams, mitigating the effect of individual errors in answering a question.
5.2 Evaluating question-answering l'aithfulness
We evaluate the faithfulness of our pnestion-answerre models on a recent diverse collection of 54 binary classification datasets [89, 90] (see data details in Table A4). These datasets are difficult, as they are intended to encompass a wider-ranging and more realistic list of questions than traditional NLP datasets.
Fig. 4 shows the classification accuracy for the 3 LLMS used previously along with GPT-3.5 (gpt-3. 6ppt-3.5-tum -0120. On average, each of the LLMs answers these questions with fairly high accuracy, with GPT-4 slightly outperforming the other models. However, we observe poor perfor- mance on some tasks, which we attribute to the task difficulty and the lack of task-specific prompt engineering. For example, the dataset yielding the lowest accuracy asks the question Is the input about math research?. While this may seem like fairly simple question for an LLM to answer, the examples in the negative class consist of texts from other quantitative fields (e.g. chemistry) that usually contain numbers, math notation, and statistical analysis. Thus the LLMs answer yes to most examples and achieve accuracy near chance (50%). Note that these tasks are more difficult than the relatively simple questions we answer in the fMRI experiments, especially since the fMRI input lengths are each 10 words, whereas the input lengths for these datasets are over 50 words on average (with some inputs spanning over 1,000 words).
7

---

1.0
0.8
0.804
0.820 0.822
Accuracy
0.6
0,4
0.2
0.0
LLaMA-3 (8B)
Mistral (7B)
GPT-3.5
GPT-4
Question-answwring LLM
Figure 4: Performance of question-answerring for underlying LLMs on the D3 collection of binary classification datasets. Each point shows an individual dataset and error bars show the 95% confidence interval.
Table 2: Information retrieval results for different interpretable embedding models. QA-Emb in combination with BM-25 achieves a slight improvement over the interpretable baselines. QA-Emb additionally yields reasonably strong performance compared to its embedding size. Note that QA-Emb embeddings are binary, so the raw number of dimensions overrepresents the embedding's size relative to other methods. Error bars show standard error of the mean.
Mean reciprocal rank
Recall @
Recall@5
Size
Bag of words Bag bigrams
0.37-1.01
0.28
0.42-1.02
27,677
0,391.01
0.30#.02
0.44#102
197,924
Bag of trigrams
0,39:.02
0.30=.02
0.44:.02
444,403
QA-Eit Emb BM-25
0,45#.01
0.34#.01
0.50#.01
12,000
0.77#.01
0.69#.01
0.82#1011
27,677
BM-25 + QA-Emb
0.80+t.01
0.71
0.84+1.01
29,677
Secondary results: evaluating QA-Emb in simple NLP tasks
6.1 Benchmarking QA-Emb for information retrieval
In this section, investigate applying QA-Emb simplified information retrieval task. We take random subset 4,000 queries from the MSMarco dataset ([91), Creative Commons License) and their corresponding groundtruth documents, resulting in 5,210 documents. We use 25% of the queries to build training set and keep the remaining 75% for testing. For evaluation, we calculate the cosine similarity match between the embeddings for each query and its groundtruth documents using mean reciprocal rank and recall.
To compute QA-Emb, we first generate 2,000 questions through prompting GPT-4 based its knowledge queries information retrieval (see prompts in the Github). We use regex to slightly rewrite the resulting questions for queries to apply to documents (e.g. Is this query related to specific timeframe? - Is this text related to a specific timeframe?). We then answer the questions both for each query and for each corpus document, again using LLaMA-3 8B. Rather than fitting ridge regression as in Eq. (I), we use the training set learn a scalar for each question that multiplies its binary output to change both its sign and magnitude in the embedding (optimization details in Appendix A.4).
Table 2 shows the information retrieval results. Combining BM-25 with QA-Emb achieves small significant improvement over the interpretable baselines. QA-Emb on its own achieves modest performance, slightly improving slightly over bag-of-words representation, but significantly underperforming BM-25. Nevertheless, its size considerably smaller than the other interpretable baselines making quicker to interpret and to for retrieval.

---

Table 3: Clustering scores before and after zero-shot adaptation (higher is better). Errors give standard error of the mean.
Embedding size (AVG)
Rotten tomatoes
AG News
Emotion
Financial phrasebank
AVG
Original Adapted
0.61100111 0.248=00001
0.124+0.007 0.166=0.012
0.046=+00007 0.057=00000
0.084+0.000 008 0.292-+0.017
0.095
100
0.191
25.75=+0.95
6.2 Zero-shot adaptation in text clustering
We now investigate QA-Emb in a simplified text clustering setting. To do so, we study 4 text- classification datasets: Financial phrasebank ([92), creative commons license), Emotion [93] (CC BY-SA 4.0 license), AGNews [94], and Rotten tomatoes [95]. For each dataset, we treat each class as cluster and evaluate the clastering score, defined the difference between the average inter-class embedding distance and the average intra-class embedding distance (embedding distance is measured via Euclidean distance). A larger clustering score suggests that embeddings are well-clustered within each class.
In our experiment, we build 100-dimensional embedding by prompting GPT-4 to generate 25 yes/no questions related to the semantic content of each dataset (e.g. for Rotten tomatoes, Generate 25 yes/no questions related to movie reviews). We then concatenate the answers for all 100 questions to form our embedding. These general embeddings do not yield particularly strong clustering scores (Table 3 top), as the questions are diverse and not particularly selective for each dataset.
However, simply through prompting, we can adapt these general embeddings to each individual dataset. We call GPT-4 with prompt that includes the full list of questions and ask it to select subset questions that are relevant to each task. The result embeddings (Table 3 bottom) yield higher clustering scores, suggesting that QA-Emb can be adapted each task in zero-shot manner (in this simplified setting). Moreover, the resulting task-specific embeddings are now considerably smaller.
Discussion
We find that QA-Emb can effectively produce interpretable and tigh-peerformig text embeddings. While we focus on a language fMRI setting, QA-Emb may be able to help flexibly build an inter- pretable text feature space in variety of domains, such as social science [9], medicine [10], or economics [96], where meaningful properties of text can help discover something about an underlying phenomenon or build trust in high-stakes settings. Alternatively, it could be used in mechanistic interpretability, to help improve post-hoe explanations of learned LLM representations.
As LLMs improve in both efficiency and capability, QA-Emb can be incorporated into variety of common NLP applications as well, such as RAG or information retrieval. For example, in RAG systems such as RAPTOR [97] or Graph-RAC [98], explanations may help an L.M not only retrieve relevant texts, but also specify why they are relevant and how they may be helpful.
Learning text questions rather than model weights is a challenging research area, furthering work in automatic prompt engineering [15, 16]. Our approach takes a heuristic first step at solving this problem, but future work could explore more directly optimizing the set of learned questions Q in Eq. (1) via improved discrete optimization approaches and constraints. One possible approach may involve having LLMs themselves identify the errors the current model is making and improving based on these errors, similar to general trends in LLM self-improvement and autoprompting [99-102]. Another approach may involve improving the explanation capabilities of L.LMs to help extract more questions more faithfully from data [103, 104].
Broader Impacts QA-Emb seeks to advance the field of LLM interpretation, crucial step to- ward addressing the challenges posed by these often opaque models. Although LLMSs have gained widespread use, their lack of transparency can lead to significant harm, underscoring the importance of interpretable AI. There are many potential positive societal consequences of this form of inter- pretability, e.g., facilitating a better understanding of scientific data and models, along with better understanding of I.LMs and how to use them safely. Nevertheless, as the case with most MI. research, the interpretations could be used to interpret and potentially improve LLM or dataset

---

that is being used for nefarious purposes. Moreover, QA-Emb requires substantial computational resources, contributing to increased concerns over sustainability.
References
[1] Stephen Robertson, Hugo 7aragoza, et al. The probabilistic relevance framework: Bm25 and heyond. Foundations and Trends@ in Information Retrieval, 3(4):333-389 2009.
[2] Nils Reimers and Tryna Gurevych. Sentence-hert: Sentence embeddings using siamese hert-networks.
arXiv preprint arXiv: 1908. 10084, 2019.
[3] Omar Khattab and Matei Zaharia. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR Conference Research and Development Information Retrieval, SIGIR '20, New York, NY, USA, 2020. Association for Computing Machinery.
[4] Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embed- dings. arXiv preprint arXiv:2104.0882 2021.
[5] Niklas Muennighoft. Sgpt: Gpt sentence embeddings for semantic search. arXiv preprint
FrXiv:220.08800 2022.
[6] Liang Wang, Nan Yang. Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improving
text embeddings with large language models. arXiv preprint arXiv:2401.0036 2023.
[7] Zehan Li, Xin Zhang. Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards
general text embeddings with multi-stage contrastive learning, arXiv preprint arXiv:2308. 032-810,,,381,, 2023,
[8] Shailee Jain, Vy A Vo, Leila Wehbe, and Alexander G Huth. Computational language modeling and the promise in silico experimentation. Neurobiology of Language, 5(1) 1800106, 2024.
[9] Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. Can large language models transform computational social science? arXiv preprint arXiv:2305. 035504444 2023.
[10] Xiao Zhang. Dejing Dou, and Ji Wu. Learning conceptual-comtextual embeddings for medical text. In Proceedings of the AAAI Conference on Artificial Intelligenee volume 34, pages 9579-958788 2020.
[11] Bryce Goodman and Seth Flaxman. European union regulations on algorithmic decision-making and a" right explanation". arXiv preprint arxiv:1660 08813, 2016.
[12) Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint arXiv:1066.000 06565,. 2016.
[13] Iason Gabriel. Artificial intelligence, values, and alignment. Minds and machines, 30 1-437, 2020.
[14) Chandan Singh, Jeevana Priya Inala. Michel Galley, Rich Caruana, and Jianfeng Gao. Rethinking interpretability in the era of large language models. arXiv preprint arXiv:2402.00 007 2024.
[15] Yongchao Zhou, Andrei loan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. arXiv preprint arX.122211.01910 2022.
[16] Chandan Singh, John X Morris, Jyoti Aneja, Alexander M Rush, and Jianfeng Gao. Explaining patterns in data with language models via interpretable autoprompting. arXiv preprint arXiv:22 10.00122 018,0 2022. [17] Steven Bills, Nick Cammarata, Dan Mossing, William Saunders, Jeff Wu, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, and Jan Leike. Language models can explain neurons in language models. https: /apenaipubblic blob. core. windows. met/neuree
explainer/pel perper/index.. /index..x.... htmml 2023.
[18] Chandan Singh, Aliyah R Hsu, Richard Antonello, Shailee Jain, Alexander G Huth, Bin Yu, and Jianfeng
Gao. Explaining black box text modules in natural language with language models. arXiv preprint arXiv:2305.0m 09863, 2023.
[19] Julius Adebayo. Justin Gilmer, Michael Muelly. Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Advances in Neural Information Processing Systems, pages 9505-9515, 2018.
[20] Finale Doshi Velezi-elezann and Been Kim. A roadmap for a rigorous science of interpretability. arXiv preprint
17X,::770200660) 2017.
[21] Tal Yarkoni. The generalizability crisis. Behavioral and Brain Sciences, 45:e1, 2022.
[22] Alexander G Huth, Wendy A De Heer, Thomas L. Griffiths, Frédéric E Theunissen, and Jack L Gallant.
Natural speech reveals the semantic maps that tile human cerebral cortex. Nature, 532(7600):45334555 2016.
[23] Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training deep bidirectional transformers for language understanding, arXiv preprint 18Xiv:1810004805, 2018
[24] OpenAL. GPT-4 technical report. arXiv:2303. 230.00873 08774, 2023.
[25]] Hui Zou and Trevor Hastie. Regularization and variable selection via elastic net. Journal of the Royal Statistical Society Series B: Statistical Methodology, 67(2):301-320, 2005.
[26] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, LÂ@lio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang. TimothA@e Lacroix, and William El Sayed. Mistral 7h, 2023.
[27] AI@ Meta. Llama 3 model card. 2024.
10

---

[28]
Kyoung-Re Roo Jang, Junmo Kang, Giwon Hong. Sung-Hyon Myaeng, Joohee Park. Taewon Yoon, and Heeechol Seo. Ultra-high dimensional sparse representations with binarization for efficient text retrieval. arXiv preprint arXiv:2104.07199 2021.
[29] Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan,
William Hloward-Snyder. Kaifeng Chen, Sham Kakade, Prateek Jain, et al. Matryoshka representation
learning. Advances Neural Information Processing Systems, 35:30233-30249, 2022.
[30] Jacob Mitchell Springer, Suhas Kotha, Daniel Fried, Graham Neubig, and Aditi Raghunathan. Repetition improves language model embeddings. arXiv preprint arXiv: 15449, 2024.
[31]
Jinhyuk Lee, Zhuyun Dai, Xiaoqi Ren, Blair Chen, Daniel Cer, Jeremy R. Cole, Kai Hui, Michael Boratko, Rajvi Kapadia, Wen Ding, Luan, Sai Meher Karthik Duddu, Gustavo Hemannde Abrego, Weiqiang Shi. Nithi Gupta, Aditya Kusupati, Prateek Jain, Siddhartha Reddy Jonnalagadda, Ming- Wei Chang, and Iftekhar Naim. Gecko: Versatile text embeddings distilled from large language models, 2024.
[32] Shengyao Zhuang. Xueguang Ma. Bevan Koopman, Jimmy Lin, and Guido Zuccon. Promptreps: Prompting large language models to generate dense and sparse representations for zero-shot document retrieval, 2024.
[33] Denis Jered McInerney. Geoffrey Young, Jan- Willem van de Meent, and Byron C Wallace. Chill: zero- shot custom interpretable feature extraction from clinical notes with large language models. arXiv preprint arXiv:2302. 123433 2023.
[34] Josh Magnus Ludan, Qing Lyu, Yue Yang, Liam Dugan, Mark Yatskar, and Chris Callison- Burchh Interpretable-b--deiie
text classification with iteratively generated concept bottleneck. arXiv preprint arXiv:2310. 19660, 2023.
[35] Chandan Singh, John Morris, Alexander M Rush, Jianfeng Gao, and Yuntian Deng. Tree prompting: Efficient task adaptation without fine-tuning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6253-6267, 2023.
[36] Ajay Patel, Delip Rao, Ansh Kothary, Kathleen McKcown, and Chris Callison-Burch Learning inter- pretable style embeddings via prompting 1lms. arXiv preprint arXiv:2305. /2696, 2023.
[37] Letian Peng. Yuwei Zhang. Zilong Wang, Jayanth Srinivasa, Gaowen Liu, Zihan Wang, and Jingbo Shang. Answer is all you need: Instruction-follont following text embedding via answering the question. arXiv preprint
arXiv:2402.090 09..0,9622222 2024.
[38] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations
in vector space. arXiv preprint arXiv:/30.3378, 2013.
[39] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)), pages 1,332-1533, 2014.
[40] Chandan Singh, Armin Askari, Rich Caruana, and Jianfeng Gao. Augmenting interpretable models with large language models during training. Nature Communications, 14(1):7913, 2023.
[41] Alexis Conneau, German Kruszewski, Guillaume Lample. Loic Barrault, and Marco Baroni. What you can cram into a single vector: Probing sentence embeddings for linguistic properties. arXiv preprint a8Xiv:8805.0 00000, 2018 8.
[42] Frederick Liu and Besim Avei. Incorporating priors with feature attribution on text classification. arXiv preprini arXiv: pet.:9906000000 08286, 2019.
[43] David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quanti- fying interpretability of deep visual representations. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 6541-6549, 2017.
[44] David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolci Zhou, and Antonio Torralba. Under- standing the role of individual units in deep neural network. Proceedings of the National Academy of Sciences, 30771333077-330778 2020.
[45] Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas.
Finding ncurons in haystack: Case studies with sparse probing, 2023.
[46] Nicholas Bai, Rahul Iyer. Tuomas Oikarinen, and Tsui-Wei Weng. Describe-and-ddiseett Interpreting
neurons vision networks with language models. arXiv preprint arXiv:2403.1377 2024.
[47] Sarah Schwettmann, Evan Hernandez, David Bau, Samuel Klein, Jacob Andreas, and Antonio Torralba. Toward visual concept vocabulary for gan latent space. In Proceedings of the IEEE/CVP International Conference on Computer Vision, pages 6804-68 12, 2021.
[48] Ruochen Zhao, Shafiq Joty. Yongji Wang, and Tan Wang. Explaining language models' predictions with high-impact concepts. arXiv preprint 2.05522300 160 2023.
[49] Yibo Jiang, Bryon Aragam, and Victor Veitch. Uncovering meanings of embeddings via partial orthogo- nality, Advances Neural Information Processing Systems, 36. 2024.
[50] Juri Opitz. and Anette Frank. Sbert studies meaning representations: Decomposing sentence emheddings
into explainable semantic features. arXiv preprint arXiv:2206.07000 2022.
[51] Tuomas Oikarinen and Tsui- Wei Weng. Clip-dissect: Automatic description of neuron representations in
deep vision networks. arXiv preprint 2rx:4:2204. 10965, 2022.
[52] Tuomas Oikarinen, Subhro Das, Lam M Nguyen, and Tsui- Wei Weng. Label-free concept bottleneck models. arXiv preprint arXiv:2304.00 302.0619..9........ 2023.

---

[53] Usha Bhalla, Alex Oesterling, Suraj Srinivas. Flavio P Calmon, and Himaaiindu Lakkaraju. Interpreting clip with sparse linear concept embeddings (splice). arXiv preprini arXiv:2402.10:: 10376, 2024.
[54] Yue Yang. Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison- Burch,, and Mark Yatskar. Language in a bottle: Language model guided concept bottlenecks for interpretable image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19187-
19197. 2023.
[55] Aya Abdelsalam Ismail, Julius Adebayo, Hector Corrada Bravo, Stephen Ra, and Kyunghyun Cho. Con- cept bottleneck generative models. In The Twelfth International Conference on Learning Representations, 2023.
[56] Ruoyu Chen, Jingzhi Li, Hua Zhang, Changchong Sheng, Li Liu, and Xiaochun Cao. Sim2word: Explaining similarity with representative attribute words via counterfactual explanations. ACM Trans. Multimedia Comput. Commun. Appl.. 19(6). jul 2023.
[57] Karthikeyan Natesan Ramamurthy. Amit Dhurandhar, Dennis Wei, and Zaid Bin Tariq, Analogies and feature attributions for model agnostic explanation of similarity Iearners. arXiv preprint arXiv:2002.00 153, 2022.
[58] Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A Hosseini, Nancy Kan- wisher, Joshua Tenenbaum, and Evelina Fedorenko. The neural architecture of language: Integra- tive modeling converges on predictive processing. Proceedings of the National Academy of Sciences, 118(45)e21056446 18, 2021.
[59] Richard Antonello, Aditya Vaidya, and Alexander Huth. Scaling laws for language encoding models in firi. Advances in Neural Information Processing Systems, 36, 2024.
[60] Shailee Jain and Alexander Huth. Incorporating context into language encoding models for fmri. Advances in neural information processing systems, 31, 2018.
[61] Leila Wehhe, Ashish Vaswani, Kevin Knight, and Tom Mitchell. Aligning context-hased statistical models of language with brain activity during reading. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (FMNLP), pages 233-243, Doha, Qatar, October
2014. Association for Computational Linguistics.
[62] Mariya Toneva and Leila Wehbe. Interpreting and improving natural- languugg processing (in machines) with natural aegguag--poceesinge (in the brain). In H. Wallach, H. Larochelle, A. Beygelzimer, F, d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 32.
Curran Associates, Inc., 2019.
[63] Aricl Goldstein, Zaid Zada, Eliav Buchnik, Mariano Schain, Amy Price, Bobbi Aubrey, Samuel A. Nastase, Amir Feder. Dotan Emanuel, Alon Cohen, Aren Jansen, Harshvardhan Gazula, Gina Choe, Aditi Rao, Catherine Kim, Colton Casto, Lora Fanda, Werner Doyle, Daniel Friedman, Patricia Dugan, Lucia Melloni, Roi Reichart, Sasha Devore, Adeen Flinker, Liat Hasenfratz, Omer Levy, Avinatan Hassidim, Michael Brenner, Yossi Matias, Kenneth A. Norman, Orrin Devinsky, and Uri Hasson. Shared computational principles for language processing in humans and deep language models. Nature Neuroscience, 25(3):369- 380, March 2022. Number: 3 Publisher: Nature Publishing Group.
[64] John T. Hale, Luca Campanelli, Jixing Li, Shohini Bhattasali, Christophe Pallier, and Jonathan R. Brennan. Neurocomputational models of language processing. Annual Review of Linguistics, 8(1):2774446, 2022. [65] Shailee Jain, Vy A. Vo, Leila Wehbe, and Alexander G. Huth. Computational Language Modeling and the Promise of in Silico Experimentation. Neurobiology of Language, pages 1-27, March 2023.
[66] Michael C.-K. Wu, Stephen V. David, and Jack Gallant. Complete functional characterization of sensory neurons by system identification. Annual Review of Neuroscience, 29:477-505, 2006.
[67] Shailee Jain, Vy Vo, Shivangi Mahto, Amanda LeBel, Javier S Turek, and Alexander Huth. Interpretable multi timescale models for predicting fmri responses to continuous natural speech. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 13738 13749. Curran Associates, Inc., 2020.
[68] Catherine Chen, Tom Dupré la Tour, Jack Gallant, Daniel Klein, and Fatma Deniz. The cortical representation of language timescales is shared between reading and listening. bioRxiv, pages 2023-01, 2023.
[69] Sara F Popham, Alexander G Huth, Natalia Y Bilenko, Fatma Deniz, James S Gao, Anwar O Nunez- Elizalde, and Jack L Gallant. Visual and linguistic semantic representations are aligned at the border of human visual cortex. Nature neuroscience, 24(11):: (6111):662-1661 1636, 2021.
[70] Charlotte Caucheteux. Alexandre Gramfort, and
King. Disentangling syntax and semantics in the brain with deep networks. In Proceedings of the 38th International Conference on Machine Learning, pages 1336-1348. PMLR, July 2021. ISSN: 2640-3498.
[71] Carina Kauf, Greta Tuckute, Roger Levy, Jacob Andreas, and Evelina Fedorenko. Lexical semantic content, not syntactic structure, is the main contributor to ann-brain similarity of fmri responses in the language network. bioRxiv, pages 2023-05, 2023.
[72] Aniketh Janardhan Reddy and Leila Wehbe. Can fMRI reveal the representation of syntactic structure in the brain? preprint, Neuroscience, June 2020.
[73] Alexandre Pasquiou, Yair Lakretz, Bertrand Thirion, and Christophe Pallier. Information-Reste Restricted
Neural Language Models Reveal Different Brain Regions" Sensitivity to Semanties, Syntax and Context, February 2023, MrXiv:2302.14389 [cs].
12

---

[74] Khai Loong Aw and Mariya Toneva. Training language models for deeper understanding improves brain alignment, December 2022. arXiv:2212.1066 10898 [cs, q-bio].
[75] Sreejan Kumar, Theodore R. Sumers, Takateru Yamakoshi, Ariel Goldstein, Uri Hasson, Kenneth A. Norman, Thomas L. Griffiths, Robert D. Hawkins, and Samuel A. Nastase. Reconstructing the cascade of language processing in the brain using the internal computations of a transformer-based language model. Technical report, bioRxiv, June 2022. Section: New Results Type: article.
[76] Subba Reddy Oota, Manish Gupta, and Mariya Toneva. Joint processing of linguistic properties in brains and language models, December 2022. arXiv:2212.0888
[es, q-bio].
[77] Richard Antonello, Chandan Singh, Shailee Jain, Aliyah Hsu, Jianfeng Gao, Bin Yu, and Alexander Huth.
A generative framework to bridge data-driven models and scientific thcories in language ncuroscience, 2024.
[78] Amanda LcBel, Lauren Wagner, Shailee Jain, Ancesh Adhikari- Desai, Bhavin Gupta, Allyson Morgenthal, Jerry Tang, Lixiang Xu, and Alexander G Huth. A natural language fmri dataset for voxelwise encoding models. bioRxiv, pages 2022-09, 2022.
[79] Jerry Tang. Amanda LeBel, Shailee Jain, and Alexander G Huth, Semantic reconstruction of continuous language from non-invasive brain recordings. Nature Neuroxcience, pages 1-9, 2023.
[80] Fabian Pedregosa, Ga Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-learn Machine learning in python. the Journal of machine Learning research, 12(0c)):225-28830 2011.
[81] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tume fine-tuned chat models. arXiv preprint arXiv:2307 232.08830,, 2023.
[82] Evelina Fedorenko, Anna A Ivanova, and Tamar I Regev. The language network natural kind within the broader landscape of the human brain. Nature Reviews Neuroscience, pages 1-24, 2024.
[83] Joshua B Julian, Jack Ryan, Roy H Hamilton, and Russell A Epstein. The occipital place area is causally involved in representing environmental boundaries during navigation. Current Biology, 26(8):1 2016.
[84] Anna S Mitchell, Rafal Czajkowski, Ningyu Zhang, Kate Jeffery, and Andrew JD Nelson. Retrosplenial cortex and its role in spatial cognition. Brain and neuroscience advances, 2:23982128 12818757098 2018.
[85] lenni Salsano, Valerio Santangelo, and Emiliano Macaluso. The lateral intraparictal sulcus takes viewpoint changes into account during memory- memory-guided attention in natural scenes. Brain Structure and Function, 226(4):989-0066 2021.
[86] Saima Maril-MMraaled, Dima Ayyash, Jeanne Gallée, Josef Affourtit, Malte Hoffmann, Zachary Mincroff, Olessia Jouravley, and Evelina Fedorenko. An investigation across 45 languages and 12 language families reveals universal language network. Nature Neuroscience, 25(8):0014-001 2022.
[87] Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zeulemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint ar0i: 1692, 2019.
[88] Ilya Loshchilov and Frank Hutter.
Decoupled weight decay regularization.
arXiv preprint
arXiv:1711.05000 2017.
[89] Ruiqi Zhong, Kristy Lee, Zheng Zhang, and Dan Klein. Adapting language models for zero-shot learning by poeta-tunnng on dataset and prompt collections. arXiv preprint arXiv:2104. 04670, 2021.
[90] Ruiqi Zhong, Charlie Snell, Dan Klein, and Jacob Steinhardt. Describing differences between text distributions with natural language. In International Conference on Machine Learning, pages 27099-
27116. PMLR, 2022.
[91] Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. Ms marco: A summan-geneaated machine reading comprehension dataset. 2016.
[92] P. Malo, A. Sinha, P. Korhonen, J. Wallenius, and P. Takala. Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology, 65, 2014.
[93] Elvis Saravia, Hsien Shien-Chi Toby Liu, Yen- Hao Huang, Junlin Wu, and Yi-Shin Chen. CARER: Contextual- ized affect representations for emotion recognition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3687-3697, Brussels, Belgium, October-November
2018. Association for Computational Linguistics.
[94] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In C. Cortes. N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28, Curran Associates, Inc., 2015.
[95] Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL '05), pages 115 124, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics.
[96] Anton Korinek. Language models and cognitive automation for economic research. Technical report. National Bureau of Economic Research, 2023.
13

---

[97] Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie. and Christopher D Manning- Raptor: Recursive abstractive processing for tree-organized retrieval. arXiv preprini arXiv:2401.18059 2024.
[98] Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson, From local to global: A graph rag approach to query-focusee summarization. arXiv preprint arXiv:2404. 16130, 2024.
[99]
Ruiqi Zhong. Peter Zhang, Steve Li, Jinwoo Ahn, Dan Klein, and Jacob Steinhardt. Goal driven discovery distributional differences via language descriptions. arXiv preprint arXiv:2.00. /4233, 2023.
[100] Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le. Denny Zhou, and Xinyun Chen. Large language models as optimizers. arXiv preprint arXiv:2309.088 03399,34499 2023.
[101] Xinyuan Wang, Chenxi Li, Zhen Wang, Fan Bai, Haotian Luo, Jiayou Zhang. Nebojsa Jejie, Eric P Xing. and Zhiting Hu. Promptagent: Strategic planning with language models enables expert-level prompt optimization. arXiv preprint Xr3: 16427, 2023.
[102] Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401. 10020, 2024.
[103] Afra Feyza Akyürek, Ekin Akyürek, Leshem Choshen, Derry Wijaya, and Jacob Andreas. Deduc- live closure training of language models for coherence, accuracy, and updatability. arXiv preprint
arXiv:2401.0833 2024.
[104] Yanda Chen, Chandan Singh, Xiaodong Liu, Simiao Zuo, Bin Yu, He He, and Jianfeng Gao. To- wards consistent natural-language explanations via poplaanation-conssteeeyy finetuning. arXiv preprint arXiv:2401. 2024.
[105] Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 2014.
14

---

A
Appendix
A.1 fMRI question details
Table A1: Questions list for model with 29 questions. Importance denotes the average absolute coefficient for each question (normalized by the importance of the top question).
Question
Importance
Is the sentence expressing skepticism or disbelief towards something someone?
1.000
Does sentence include dialogue?
0.983
Does the sentence describe relationship between people?
0.924
Does the sentence involve the mention of a specific object or item?
0.900
Does the sentence include technical specialized terminology?
0.882
Does the sentence contain proper noun?
0.861
Does input involve planning or organizing?
0.861
Does the sentence include numerical information?
0,850
Is time mentioned in the input?
0.844
Is the sentence grammatically complex?
0.815
Does the sentence include dialogue thoughts directed towards another character?
0.811
Does the sentence describe physical action?
0,809
Does the sentence include conditional clause?
0.782
Does the sentence describe visual experience or scene?
0.771
Does the input include philosophical reflective thought?
0.759
sentence conveying the narrator's physical movement or action detail?
0.749
Does the sentence describe physical sensation?
0.744
Does the sentence involve discussion about personal social values?
0.739
Does the sentence reference specific time date?
0.719
Does the sentence express philosophical existential query observation?
0.705
Does the sentence involve description physical environment or setting?
0.693
Does the input describe sensory experience?
0.688
Does the sentence involve planning or decision-making? Is the sentence command?
0.684
0.682
Does the sentence describe specific sensation or feeling?
0.672
Does the sentence contain cultural reference?
0.667
Does the input include dialogue between characters?
0.594
Does the sentence mention specifie location place?
0.547
Does the sentence reference specific location place?
0.545
15

---

A.2
IMRI prediction results extended
S01
S03
S02
-24 -h0
65
corralation (QA-Emb)
Test correlation (QAEmb)
Test corrlatioon (QA
(QA Emb)
S01
S03
S02
-12
211
0.2
635 an a.6 a.0 ans 61s 615 Test correlation (QA-Emb -BEEET
Test correaaiion (QA-Emi Emb- BERT)
Test correlation IQA-Emb BURT:
Figure A1: Predictive performance for QA-Emb (top row) and the difference between QA-Emb and BERT (bottom row).
Table A2: Mean test correlation for QA-Emb with different settings: varying the underlying prompts to source questions and the LLM used to answer the questions (fixing the number time-lagged delays to 8). Ensemble generally provides small boost over other models and Mistral slightly underperforms LLaMA-3 (8B).
Ensemble
LLaMA-3 (8B)
LLaMA-3 (8B)-fewshot
Mistral (7B)
Subject
Questions
S01
Prompts 1-3 (376 questions)
0.081
0.078
0.078
0.076
Prompts 1-5 (518 questions)
0.089
0.085
0.085
0.082
Prompts 1-6 (674 questions)
0.084
0.081
0.085
0.076
S02
Prompts 1-3 (376 questions) Prompts 1-5 (518 questions)
0.120
0.112
0.119
0.112
0.118
0.120
0.121
0.114
Prompts 1-6 (674 questions)
0.124
0.119
0.121
0.108
S03
Prompts 1-3 (376 questions) Prompts 1-5 (518 questions)
0.132
0.131
0.127
0.126
0.137
0.136
0.135
0.129
Prompts 1-6 (674 questions)
0.141
0.136
0.136
0.132
AVG
Prompts 1-3 (376 questions) Prompts 1-5 (518 questions)
0.111
0.107
0.108
0.104
0.115
0.114
0.114
0.108
Prompts 1-6 (674 questions)
0.116
0.112
0.114
0.105
16

---

Table A3: Mean test correlation for different baseline models as a function of hyperparameters (number lane- delays and layer for extracting embeddings)
Subject
S01
S02
S03
AVG
Delays
4
12
BERT
0.084
0.080
0.075
0.114
0.108
0.107
0.136
0.139
0.136
0.111
0.109
0.106
Eng1000
0.079
0.067
0.077
0.096
0.092
0.082
0.110
0.117
0.116
0.095
0.092
0.092
LLaMA-2 (70B) (lay 12)
0.055
0.055
0.054
0.101
0.095
0.085
0.143
0.144
0.130
0.100
0.098
0.089
LLaMA-2 (70B) (lay 24)
0.075
0.059
0.049
0.097
0.104
0.092
0.149
0.153
0.152
0.107
0.105
0.098
LLaMA-2 (70B) (lay
0,058
0.068
0.057
0.131
0.101
0.084
0.153
0.156
0.152
0.114
0.108
0.098
LLaMA-2 (70B) (lay 48)
0.093
0.060
0.052
0.114
0.094
0.091
0.148
0.151
0.149
0.118
0.102
0.098
LLaMA-2 (70B) (lay 60)
0.095
0.048
0.050
0.119
0.089
0.088
0.148
0.152
0.150
0.121
0.097
0.096
LLaMA-2 (7B) (lay 06)
0.074
0.067
0.039
0.120
0.088
0.084
0.138
0.144
0.133
0.111
0.100
0.085
LLaMA-2 (7B) (lay 12)
0.097
0.058
0.053
0.116
0.111
0.087
0.150
0.155
0.152
0.121
0.108
0.097
LLaMA-2 (7B) (lay 18)
0.079
0.076
0.042
0.123
0.103
0.090
0.143
0.153
0.150
0.115
0.111
0.094
LLaMA-2 (7B) (lay 24)
0.088
0.057
0.068
0.129
0.100
0.106
0.144
0.148
0.149
0.120
0.102
0.108
LLaMA-2 (7B) (lay 30)
0.057
0.045
0.045
0.130
0.098
0.099
0.139
0.149
0.148
0.109
0.097
0.097
LLaMA-3 (8B) (lay 06)
0.071
0.066
0.054
0.122
0.119
0.095
0.144
0.147
0.148
0.112
0.111
0.099
LLaMA-3 (8B) (lay 12)
0,089
0.073
0.050
0.110
0.099
0.095
0.146
0.151
0.153
0.115
0.108
0.099
LLaMA-3 (8B) (lay 18)
0.073
0.052
0.052
0.125
0.102
0.096
0.153
0.154
0.155
0.117
0.103
0.101
ILaMA-3 (8B) (lay 24)
0.090
0.053
0.047
0.106
0.113
0.095
0.146
0.149
0.148
0.114
0.105
0.097
ILaMA-3 (8B) (lay 30)
0.082
0.066
0.060
0.120
0.117
0.101
0.147
0.151
0.148
0.117
0.111
0.103
A.3 Prompts
A.3.1 Prompts for question generation
Generate bulleted list of 500 diverse, noe-overlg questions that can be used to classify an input based on its semantic properties. Phrase the questions in diverse ways.
Prompt 1
Here are some example questions:
I| [examples]]
Return only a bulleted list of questions and nothing else
Prompt 2
Generate bulleted list of 100 diverse, non-overlapping questions that can be used to classify sentences from a prrs-pperson story. Phrase the questions in diverse ways.
Here are some example questions:
[[examples)]
Return only a bulleted list of questions and nothing else
Prompt 3
Generate bulleted list of 200 diverse, non-overlapping questions that can be used to classify sentences from a first- pess--pesoon story. Phrase the questions in diverse ways.
Here are some example questions:
[[examples)]
Return only a bulleted list of questions and nothing else
Based on what you know from the neuroscience and psychology literature, generate bulleted list of 100 diverse, non-overlapping yes/no questions that ask about properties of a sentence that might be important for predicting bruin activity.
Prompt 4
Return only a bulleted list of questions and nothing else
Prompt Example narrative sentences
[[example sentences from dataset)
# Example yes/no questions ((example questions already asked))
17

---

Generate a bulleted list of 100 specific, non-overlapping yes/no questions that ask about aspects of the example narrative sentences that are important for classifying them. Focus on the given narrative sentences and form questions that combine shared properties from multiple sentences above. Do not repeat information in the example questions that are already given above. Instead, generate complementary questions that are not covered by the example questions. Return only bulleted list of questions and nothing else.
Prompt Generate more diverse questions that may occur for single sentence in a first-person narrative story
See exact prompts with examples in the Github repo
A.3.2 Prompts for question answering
Standard prompt <User>: Input text: (example]
Question: [question]
Answer with yes or no, then give an explanation.
Few-shot prompt <System>: You are concise, helpful assistant. <User>: Input text: and just kept laughing because it was so Question: Does the input mention laughter? Answer with Yes No.
<Assistant>: Yes
< User>> Inpul text: what crazy day things just kept on happening Question: Is the sentence related to food preparation? Answer with Yes No.
Assisttant>: No
User>> Input text: i felt like a fly on the wall just waiting for Question: Does the text use metaphor or figurative language? Answer with Yes No.
<Assistant>: Yes
<User> Input text: he takes too long in there getting the pans from Question: Is there reference to sports?
Answer with Yes or No.
Answer with Yes No.
AAsisttant>: No
< User> Input text: was silent and lovely and there sound except Question: Is the sentence expressing confusion or uncertainty?
Answer with Yes No.
<Assistant>: No <seer> Input text: (example)
Question: (question)
Answer with Yes No.
<sssistant>:
See exact prompts with examples in the Github repo.
A.4 Information retrieval details
Optimization details When fitting our QA-Emb model for information retrieval, we learn single scalar per-question that is multiplied by each embedding before computing similarity. To learn these scalars, we minimize a two-part loss. The first loss is the negative cosine similarity between each query and similar documents. The second loss the cosine similarity between each query and the remaining documents. We weight the first loss as 10 times higher than the second loss and optimize using Adam [105] with learning rate of 10-4 We run for epochs, when the training loss seems to platcau.
18

---

Table A4: 54 binary classification datasets along with their underlying yes/no question and corpus statistics from a recent collection [89, 90].
Datasct
Datasct topic
Underlying yesoo question
Examples
Unique unigrams
O-irony
sarcism
contains irony
590
3897
l-ojecttvve
unbiased
objective description of what happened
739
5628
2-subjective
subjective
contains suhjective opinion
757
5769
3-god
religions
believes god
164
1455
4-atheism
atheistic
is against religion
172
1472
5-evacuate
evacuation
involves need people evacuate
2670
16605
6-tererrism
terrorism
describes situation that involves terrorism involves crime
2640
16608
7-crime
crime
2621
16333
8. shelter
shelter
describes simation where people need shelter
2620
16347
9-food
hunger
related to food security
2642
16276
10-infrastructure
infrastrcture
related infrastructure describes regime change
2664
16548
11 11-regime change
regime change
2670
16382
12-medical
health
related to medical situation
2675
16223
13-water
water
involves situation where people need clean water
2619
16135
14-seurch
rescue
involves a search/rescue situation
2628
16131
utiiiiiyy
utility
expresses need for utility, energy sanitation is against Hillary
2640
16249
16-hillary
Hillary
224
1693
17-hillary
Hillary
supports hillary
18 offensive
derogatory
contains oflensive content
652
6109
19-offeusive
toxic
insult women or immigrants
2188
18839
20-pro 20-pro-life
pro-life
pro-life
213
1633
21 pro choice
abortion
supports aburtion
209
1593
22-physics
physies
about physics
10360
93810
23-computer science
computers
is related to computer seience
10441
93947
24-stalistics
statistics
about statisties
9286
86874
25 math
math
is about math research
8898
85118
26-grammnar
ungrammetical
ungrammatical
834
2217
27 grammnar
grammatical
grammatical
826
2236
28. sexis
sexist
offensive to women supports feminism is about world news about sports news is related to husiness related to technology
209
1641
29-sexis
feminism
1710
30-nows
world
S778
13023
31 31-sports
sports news
5674
12849
32. business
business.
5699
12913
33-1ech
technology
5727
12927
34. had
negative
contains had movie review
357
16889
35-good
good
thinks the movie is good asks quantity
380
17497
36-quantity
quantity
1901
5144
37-1ocation
location
asks about a location person asks about entity
1925
$236
person
person
19 19-entity
entity
1896
5180
40-abbrevation
abbreviation
about an abbreviation
1839
5045
41 41-defin
definition
contains definition
651
4508
42-cnvironment
anvirommentalism
against environmentalist
124
1117
43-envireament
cuvironmentalism
envisommentalist
44 44-spam
spam
span
360
2470
fact
factual information
11449
46-opinion
opinion
asks for opinion
719
11709
47-math
science
related math science
7514
$3973
48.health
health
related to health
7485
$3986
49-computer
computers
related to computer or internet
7486
$4256
5U-sport
sports
sports
51 entertainment
entertainmentt
is about entertainment
7461
53573
family
relationships
about family relationships is related to politics or govemment
7438
53-politic
polities
7410
53393
19

---

A.5 Details on question-answering evaluation datasets
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claims are clearly stated that QA-Emb can generate flexible embeddings using a pre-trained LLM (described in the Sec. 2) and that this experimentally improves performance primarily for an fMRI prediction problem (Sec. 4).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Yes, limitations are discussed in the methods section (Sec. as well as the entirety of Sec. 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
20

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper includes no theoretical results.
Guidelines:
The answer NA means that the paper does include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented
by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Yes, all details are fully provided in the paper (including extra details such as prompts in the supplementary material).
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers have some path reproducing verifying the results.
21

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: All code and data made openly available Github at at https:// anonymous. 4open wwwssceereeeeeeeeeeeceeee /atncc////ateeerrtbbllee httpo//eeeeeedddddnnooood/////// md.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ /nipp. wwtpccccccggcuiiieoocooooococcce guwde// /Coeee// serrissioonn wwtpobmmssioooooiiiiiiioo (eeonnooicyy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes all specifications are given in the Methods and the experimental setup sections in the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Yes we give details about error bars in our results section. Note that Fig. doesn't t directly show error bars since they are small enough that they are within the points (see Fig. caption).
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
22

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Mentioned in the methods section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: /at::///nnuuiiie ///urinuus...// weabblic/Ethissuiiddeeoe
Answer: [Yes]
Justification: We have reviewed the NeurIPS Code of Ethics and found that the current paper conforms
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Given in the discussion section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
23

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks beyond those discussed in the Broader Impacts above, releases no new data or models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We list the licenses in the main text for all datasets where they can be found (the main fMRI data is released under the MIT license).
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/ comddataset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's ' creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No human subjects are involved in this study. The fMRI data analyzed here is collected in previous studies following the appropriate IRB protocols.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No human subjects are involved in this study. The fMRI data analyzed here collected in previous studies following the appropriate IRB protocols.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
25

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "xTS1MCwId0", "reviewer": "Reviewer_wvR8", "summary": "The authors introduce question-answering embeddings (QA-Emb), where each feature in the embedding corresponds to an answer to a yes/no question asked to an LLM (e.g., LLaMA-3 8B). QA-Emb significantly outperforms an established interpretable baseline in predicting fMRI voxel responses to language stimuli. The authors also evaluate QA-Emb on information retrieval and text clustering tasks. Additionally, computational efficiency can be improved by distilling the QA-Emb model into a RoBERTa-base model.", "questions": "Please see my comments in Weakness.", "limitations": "The authors provide detailed discussions on limitations in Sections 2 and 5.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "(1) This paper focuses on the critical problem of extracting interpretable embeddings from large language models (LLMs).\n\n(2) Extensive experiments are conducted on three different tasks: fMRI interpretation, information retrieval, and text clustering.\n\n(3) The authors demonstrate that the model's efficiency can be improved through distillation.", "weaknesses": "(1) Motivation:\nThe paper aims to extract meaningful text embeddings from LLMs. However, the entire QA-Emb pipeline seems to rely less on LLMs than suggested, as these questions could also be answered by a much smaller QA model, such as RoBERTa-base, which the authors have explored in Section 5.1.\n\n(2) Baselines:\nWhile there are existing methods that extract text embeddings from LLMs [1,2], these methods are not compared in the paper. It is reasonable to expect that QA-Emb might perform worse than these non-interpretable methods, but a comparison with these methods would provide a clearer understanding of QA-Emb's performance.\n\n(3) Results:\nThe authors claim that QA-Emb is a high-performing embedding method. However, results on the MS-MARCO tasks indicate otherwise. QA-Emb significantly lags behind BM25, while a recent LLM-based embedding method like PromptReps [1] is able to match the performance of BM25 on retrieval tasks.\n\n[1] Ting Jiang, Shaohan Huang, Zhongzhi Luan, Deqing Wang, Fuzhen Zhuang. Scaling Sentence Embeddings with Large Language Models. Arxiv.\n\n[2] Shengyao Zhuang, Xueguang Ma, Bevan Koopman, Jimmy Lin, Guido Zuccon. PromptReps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document Retrieval. Arxiv.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and thoughtful comments - they have helped us to improve the paper.\n\n(1) Re: Motivation – Yes indeed, the entire pipeline at inference-time can use only smaller models (e.g. RoBERTA). We see this is a major strength, as it drastically reduces the inference cost of applying QA-Emb. However, at training time, a high-performing LLM (e.g. LLaMA-3) is still required to provide the answers that can be used to finetune the smaller model.\n\n(2) Re: Baselines – Thanks for sharing these baselines, we have added references to them in the paper. Indeed there are many non-interpretable baselines that we can compare to. For our main setting (fMRI prediction), state-of-the-art prediction empirically comes from embeddings obtained by taking the last token of state-of-the-art open source models such as LLaMA [1]. We do compare directly to these across many layers of recent LLaMA models (Fig 2A). We will add these comparisons for the information retrieval setting in Table 2 as well (we do expect the non-interpretable embeddings to perform better in this setting).\n\n(3) Re: Results – Thanks for pointing this out, indeed QA-Emb is a strong embedding method in terms of interpretability/compactness of the representation but yields a small drop in performance compared to black-box methods. This tradeoff is worthwhile in our fMRI setting, where the goal is to yield a succinct model for scientific understanding. In our main result (Fig 2B), QA-Emb yields very strong fMRI prediction performance using only a 29-dimensional binary representation that then enables manual inspection for scientific understanding (Fig 3). Note that we have re-worked the paper to be focused on the fMRI setting (see “General response” comment). We hope that NeurIPS reviewers for a paper in the “Neuroscience and cognitive science” track can appreciate the paper’s results.\n\nRepresentations from baseline methods are often black-box or very large. PromptReps embeddings do yield high performance, but contain an element for each token in an LLM’s vocabulary, yielding a representation with more than 100k dimensions in the PromptReps paper’s main results (with LLaMA-3). \n\n[1] “Scaling laws for language encoding models in fMRI” R Antonello, A Vaidya, A Huth, NeurIPS 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X9sHYDje1m", "reviewer": "Reviewer_Z19r", "summary": "The paper proposes a method of prompting LLMs with a list of yes or no questions to obtain binary embeddings for texts. The list of questions is generated by prompting the LLM with task-specific knowledge. The proposed method, QA-Emb, is evaluated primarily on predicting fMRI voxel responses to texts. QA-Emb outperforms an established interpretable baseline but underperforms compared to a black-box LLM embedding baseline. Additional experiments in information retrieval indicate that QA-Emb significantly underperforms BM25 but performs better than simple n-grams.", "questions": "1. More experiments on different domains and tasks could help to show the generalizability of the method. Especially, on embedding related benchmarks, and against some interpretable and black-box baselines. \n\n2. What strategies can be employed to mitigate the computational cost of QA-Emb?", "limitations": "Yes.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper introduces a creative perspective on text embedding by leveraging the QA capabilities of LLMs to achieve interpretability.\n2. The paper is well-written and easy to follow. It includes extensive experiments demonstrating the effectiveness and limitations of the method in various scenarios.\n3. This paper represents a initial effort in crafting an interpretable embedding space for tasks like regression, information retrieval, and clustering.", "weaknesses": "1. The method of prompting LLMs to generate interpretable features is not entirely novel. For example, \"CHiLL: Zero-shot Custom Interpretable Feature Extraction from Clinical Notes with Large Language Models\" also prompts LLMs to ask questions for a classification task. A deeper investigation into the question generation phase would be more interesting and challenging. The paper could benefit from a more detailed discussion on systematically selecting and validating these questions to ensure robustness across various tasks.\n\n2. While the method performs well in predicting fMRI voxel responses, it significantly underperforms in the information retrieval task compared to BM25. This raises doubts about the method's generalizability to other domains and tasks. If the method only excels in predicting fMRI voxel responses, it may be more accurately described as a specialized model for this task rather than a general \"interpretable embedding\" model.\n\n3. The QA-Emb pipeline is computationally intensive, requiring multiple LLM calls to compute an embedding, which can be prohibitively expensive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and thoughtful comments - they have helped us to improve the paper.\n\n**1) Re: question generation** – We wholeheartedly agree question generation is important and have added new experiments extending the results in Table A2 to analyze the process of question generation. We find that results are fairly stable to the choices of different prompts. We report the mean fMRI test correlation, averaged across the test set for all three subjects. See the exact prompts in Sec A.3.1.\n\nWe find that performance does not vary drastically based on the choice of prompt:\n\n|Prompts    |1    |2    |3    |4    |5   |6   |\n|-----------|-----|-----|-----|-----|----|----|\n|Performance|0.088|0.092|0.085|0.079|0.90|0.96|\n|Questions  |90   |98   |88   |97   |45  |156 |\n\nWe perform feature selection as done in the main paper (running multi-task Elastic net with 20 logarithmically spaced regularization parameters ranging from $10^{−3}$ to 1 and then fitting a Ridge regression to the selected features) and report results for the model with number of features closest to 29 (which is the main model we analyze in the paper). We find that performance again does not vary drastically based on the choice of prompt:\n\n|Prompts    |1    |2    |3    |4    |5   |6   |\n|-----------|-----|-----|-----|-----|----|----|\n|Performance|0.062|0.065|0.059|0.061|0.072|0.074|\n|Questions  |20   |24   |17   |33   |26  |34  |\n\nFinally, we find that performance does not vary drastically as the number of questions becomes large. Table A2 provides further breakdowns for this table by underlying LLM.\n\n|Prompts    |1-3  |1-5  |1-6  |\n|-----------|-----|-----|-----|\n|Performance|0.111|0.115|0.116|\n|Questions  |376  |518  |674  |\n\nWhile the method of “prompting LLMs to generate interpretable features” may not be entirely novel, we believe our method and application are. Unlike the ChiLL paper, the LLM itself (i) writes the questions and (ii) a feature selection process prunes them. This allows for the method to discover a small set of underlying important features, rather than a human specifying them. This is critical in the fMRI setting, where specifying a small, interpretable encoding model has long been pursued without success. In contrast, QA-Emb can yield very strong performance with only 29 questions (that would have been difficult for the domain experts to explicitly select in the first place). We have added a discussion of the ChiLL paper and clarified that QA-Emb’s novelty lies in (i), (ii), and the new insights they reveal in the fMRI setting.\n\n\n**2) Re: embedding generality** - Thanks, we take this point well and have revised the manuscript and generality claims to focus on the fMRI setting (see our “General response” comment). This fMRI problem is the main motivation for us, where interpretability is crucial and a minor performance drop is acceptable. We hope that NeurIPS reviewers for a paper in the “Neuroscience and cognitive science” track can appreciate the paper’s results. In settings outside of fMRI, the method does underperform baselines, although it still succeeds in improving the baseline through concatenation (and also provides very small embeddings, which may be useful in niche settings).\n\n**3) Re: computational efficiency** - The pipeline is indeed computationally expensive, but we propose and evaluate a solution: model distillation. The intro/methods briefly describe it, e.g. “we find that we can drastically reduce the computational cost of QA-Emb by distilling it into a model that computes the answers to all selected questions in a single feedforward pass by using many classification heads.” This makes computing an embedding inexpensive at inference time (see Sec 5.1) with a negligible drop in performance in our fMRI setting (see Table 1, where mean test correlation drops from 0.114 to 0.113 after distillation).\n\nDistillation also does not substantially hurt the interpretability of the underlying embedding elements. In a new experiment, we measure the distilled model’s ability to mimic the original LLaMA-based full model: we directly replace the embedding obtained from the full model with the embedding obtained from the distilled model *without changing the linear weights learned on top of the embeddings* for the fMRI setting. We find that performance drops from a mean test correlation of 0.114 to 0.112. This extremely small drop suggests that the distilled embeddings are a drop-in replacement for the original interpretable embeddings that preserve the correct, interpretable values for different questions in the embedding.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i1ZCQbnyfb", "reviewer": "Reviewer_5mep", "summary": "The paper explores obtaining interpretable embeddings through LLM prompting. To address the opaque nature of text embeddings, the authors introduce question-answering embeddings (QA-Emb) by asking LLMs a set of yes/no questions. Specifically, these questions are generated by GPT-4 using predefined prompts. Then, Elastic net is adopted for feature selection. The validity of QA-Emb is tested primarily in a fMRI interpretation problem. Moreover, the paper explores the computational efficiency of the method by finetuning a RoBERTa model and extends the QA-Emb on information retrieval and text clustering.", "questions": "Could you provide more details on the method and prompt for the posthoc pruning of generated questions in the experiments?", "limitations": "Limitations are included.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Generating interpretable embeddings using LLMs is an important and promising issue.\n- QA-Emb can potentially provide new insights, demonstrated by the interesting findings in mapping the questions' sensitivity across the cortex.\n- QA-Emb shows promising results in the fMRI interpretation problem.", "weaknesses": "- The generation of questions for QA-Emb relies on manually crafted prompts, raising concerns about the robustness of the method. It remains unclear whether the efficacy of QA-Emb would persist with different sets of prompts. Moreover, The inconsistency in the number of questions generated from each prompt (500, 100, 200, 100, 100 respectively) introduces additional complexity. It remains unclear how to determine the optimal number of generated questions for each prompt.\n- The method is computationally intensive, as it requires prompting LLMs with hundreds of questions for each sample. While the authors attempt to mitigate this by fine-tuning RoBERTa, it is questionable whether a smaller model like RoBERTa can maintain performance across diverse applications with more complex questions. Furthermore, the fine-tuning process itself depends on prompting LLMs over substantial examples. \n- The demonstrated effectiveness of QA-Emb is limited to a single neuroscience problem. Its performance in information retrieval and text clustering appears underwhelming. The approach would gain more credibility if shown effective across more scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and thoughtful comments - they have helped us to improve the paper.\n\n**(1) Re: prompt sensitivity** – Indeed prompting LLMs for this application requires some manual choices, although this can be a good thing for helping to inject domain knowledge into the problem (different numbers of desired questions can themselves be put into the prompts). We conduct new experiments extending the results in Table A2 to assess prompt sensitivity and find that results are fairly stable to the choices of different prompts. We report the mean fMRI test correlation, averaged across the test set for all three subjects. See the exact prompts in Sec A.3.1.\n\nWe find that performance does not vary drastically based on the choice of prompt:\n\n|Prompts    |1    |2    |3    |4    |5   |6   |\n|-----------|-----|-----|-----|-----|----|----|\n|Performance|0.088|0.092|0.085|0.079|0.90|0.96|\n|Questions  |90   |98   |88   |97   |45  |156 |\n\nWe perform feature selection as done in the main paper (running multi-task Elastic net with 20 logarithmically spaced regularization parameters ranging from $10^{−3}$ to 1 and then fitting a Ridge regression to the selected features) and report results for the model with number of features closest to 29 (which is the main model we analyze in the paper). We find that performance again does not vary drastically based on the choice of prompt:\n\n|Prompts    |1    |2    |3    |4    |5   |6   |\n|-----------|-----|-----|-----|-----|----|----|\n|Performance|0.062|0.065|0.059|0.061|0.072|0.074|\n|Questions  |20   |24   |17   |33   |26  |34  |\n\nFinally, we find that performance again does not vary drastically as the number of questions becomes large. Table A2 provides further breakdowns for this table by underlying LLM\n\n|Prompts    |1-3  |1-5  |1-6  |\n|-----------|-----|-----|-----|\n|Performance|0.111|0.115|0.116|\n|Questions  |376  |518  |674  |\n\n\n**(2) Re: computational cost** – The proposed method is computationally intensive, although for the main application we study here (the fMRI results), distilling RoBERTa successfully cut these costs and should similarly work in situations where questions are easy to answer. In the long run, we hope costs will be mitigated by (1) the rapidly decreasing costs of LLM inference and (2) the improvement of small models which can be used for distillation.\n\n**(3) Re: neuroscience focus** – We agree this result is specific to the neuroscience problem and have revised the manuscript and generality claims to focus on the fMRI setting (see our “General response” comment); this approach is a strong fit for the fMRI problem and reveals new insights into language neuroscience. We hope that NeurIPS reviewers for a paper in the “Neuroscience and cognitive science” track can appreciate the paper’s results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T4Y5Q0yVPZ", "reviewer": "Reviewer_s4gE", "summary": "This study proposes QA-embedding for natural language text for multiple downstream tasks. Early embedding methods like bag-of-words and BM-25 cannot capture nuanced semantic feature of a sentence, and recent works primarily utilize the hidden states of large language models (LLM) as text embeddings, which lack of interpretability. Comparing to these embedding architectures, QA-embedding construct interpretable representations of texts, which can ameliorate the influence of opaqueness in scientific fields. In QA-embedding model procedures, it collects evaluation questions which is relevant to specific task firstly, which is achieved by prompting GPT-4 with corresponding information and eradicate redundancy through an Elastic. Then, QA-embedding model builds a {0,1} vectors by transforming yes/no responses to 1/0 which are obtained from asking a QA model with evaluation questions. Moreover, QA-embeddings are transformed into representations suitable for downstream tasks through parameters fitted by ridge regression. To evaluate QA-embedding, they perform experiment on fMRI, information retrieval, text clustering task and outperforms baselines. Meanwhile, they mention the limitation of QA-embedding, such as computational cost and LLM inaccuracies. To comprehend the limitation in depth, they also conduct analysis and put forward possible improvements.", "questions": "1. Is there an analysis on the time efficiency of QA-embedding? Although you proposed solution to decrease the computational cost, a comparison among before and after distilling and other embedding methods will make the limitation clearer.\n2. In section 5.2, did you test the classification accuracy on your distilled model RoBERTa? It seems that you treat distilled RoBERTa-base as the QA model in previous experiment, and you prove distilled RoBERTa’s ability on fMRI task, thus its classification skill or behaviour on other tasks can demonstrate the effect of your distillation technique.\n3. Since QA-embedding is strongly interpretable, have you tried to make error analysis on downstream tasks? The aptitude of tracing back to the essence of model’s prediction errors will make QA-embedding competitive.", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. Text embedding are widely used in both research and applications, and highly interpretable embeddings can also enhance the reliability of models.\n2. QA-embedding is an intuitive method that can extract the semantic feature of a sentence.\n3. The proposed method is intriguing and may inspire more studies collaborating LLMs to interpretability.", "weaknesses": "1. Some implementation procedures in line 149-165 should be clearly described, including PCA, inverse PCA and the sampling process, to clarify the task setup.\n2. Although computational cost can be alleviated by distilling, QA model’s reliability still debilitate the interpretability of QA-embedding. According to experiment result in 5.2, merely 80% dimensions of embedding vectors are reliable, which means QA-embedding cannot consistently control the accuracy of each dimension.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and thoughtful comments - they have helped us to improve the paper.\n\n**W1. Re: implementation procedures** - thanks, we have added some notes on the task setup as well as an additional section A.3 describing the full fMRI data collection details and preprocessing.\n\n**W2. Re: computational cost** - this is a good point, the QA model’s interpretability relies on (i) the underlying LLM being sufficiently capable and (ii) the questions being asked to construct the embedding not being too difficult. The experiments in Sec 5.2 show that different LLMs get ~80% on a set of datasets that are both difficult and noisy. We anecdotally have two of our paper’s authors answer 100 questions at random on this dataset and they achieve 84% and 78%. We describe one of the difficulties in the dataset in Sec 5.2: \n\n“the dataset yielding the lowest accuracy asks the question Is the input about math research?. While this may seem like a fairly simple question for an LLM to answer, the examples in the negative class consist of texts from other quantitative fields (e.g. chemistry) that usually contain numbers, math notation, and statistical analysis. Thus the LLMs answer yes to most examples and achieve accuracy near chance (50%).”\n\nFor the main fMRI setting we focus on, these difficulties are rare & answering these questions is much simpler, especially since the fMRI input lengths are each 10 words, whereas the input lengths for the datasets in Sec 5.2 are over 50 words on average (with some inputs spanning over 1,000 words).\n\n**Q1. Re: time efficiency** – Thanks for pointing this out, we have added some discussion on time efficiency to Section 2 (the methods). Generally, if an embedding requires answering *n* questions, the cost of the full autoregressive approach is *n* times the inference cost of the large autoregressive model. The cost of the distilled approach is the inference cost of the smaller distilled model (plus the negligible cost of a single linear layer for classifying the answer to each question), thus reducing the total cost by more than a factor of *n*.\n\n**Q2. Re: distilled model classification accuracy** – We did not directly test the classification accuracy of the distilled RoBERTa model (although many prior studies have shown that RoBERTa can be distilled to mimic larger teacher models). In a new experiment, we measure the distilled model’s ability to mimic the original LLaMA-based full model. Specifically, we directly replace the embedding obtained from the full model with the embedding obtained from the distilled model *without changing the linear weights learned on top of the embeddings*. We find that fMRI prediction performance drops from a mean test correlation of 0.114 to 0.112. This very small drop suggests that the distilled embeddings are a drop-in replacement for the original interpretable embeddings that preserve the correct, interpretable values for different questions in the embedding.\n\n**Re: Flag for Ethics review** – the fMRI dataset we study is publicly available and comes from two recent studies (LeBel et al., 2022; Tang et al., 2023). In both cases, the experimental protocol was approved by the Institutional Review Board at the University of Texas at Austin and written informed consent was obtained from all subjects.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes an approach QA-Emb for generating interpretable embeddings (with the primary application being fMRI response prediction where interpretability is highly valued).


They create this embedding based on prompting an LLM with a set of yes/no questions. This yes/no question strategy has been used before in fMRI but the authors' contribution is that they use an LLM to generate these questions and also automatically prune them (so they do not have to be manually constructed). 
 
I think this is a nice application of LLM prompting the fMRI and so support accepting the paper contigent on the authors' proposed title change. (The NLP results of the paper are not strong but I think the authors intended the main application to be about fMRI).
Decision: Accept (poster)