Answer is All You Need: Instruction-following Text Embedding via Answering the Question (2024)

Abstract: This work aims to build a text embedder that can capture characteristics of texts specified by user instructions. Despite its tremendous potential to deploy user-oriented embeddings, none of previous approaches provides a concrete solution for it. This paper offers a new viewpoint, which treats the instruction as a question about the input text and encodes the expected answers to obtain the representation accordingly. Intuitively, texts with the same (implicit) semantics would share similar answers following the instruction, thus leading to more similar embeddings. Specifically, we propose INBEDDER that instantiates this embed-via-answering idea by only fine-tuning language models on abstractive question answering tasks. INBEDDER demonstrates significantly improved instruction-following capabilities according to our proposed instruction awareness tests and instruction robustness tests, when applied to both large language models (LLMs) (e.g., llama-2-7b) and smaller encoder-based LMs (e.g., roberta-large). Additionally, our qualitative analysis of clustering outcomes, achieved by applying different instructions to the same corpus, demonstrates a high degree of interpretability.

heading: 1 Introduction
text: Text embedders play a crucial role in large-scale textual data analysis and management. While existing models (Reimers and Gurevych, 2019a; Gao et al., 2021; Ni et al., 2022a,b; Wang et al., 2022; Xiao et al., 2023) demonstrate strong effectiveness in representing texts in general, they lack the ability to address user-specific objectives. This limitation hinders their application in more intricate scenarios where the embedding task requires the model to represent particular characteristics of the texts (Wang et al., 2023; Zhang et al., 2023b). Consider Figure 1, where a single set of reviews is required to be
∗The first two authors contributed equally to this work. †Corresponding authors.
clustered in three distinct manners to derive meaningful insights. In response, we attempt to equip the text embedders with instruction-following capability in this paper.
One straightforward solution is to embed the concatenated instruction and input. Nonetheless, generic textual embeddings represent the texts in a form that can be used in textual similarity tasks, search and clustering, etc, rather than following instructions. Even for those that are trained with multi-task contrastive objective (Su et al., 2023), there are no guarantee to generalize to new instructions due to the inevitably restricted diversity of training instructions written by humans.
We offer a novel viewpoint, which treats the instruction as a question about the input text and encodes the expected answers. Specifically, using the instructed input as the prompt to generative language models, we argue that the generated answers can be natively utilized to model semantic similarity under different instructions. For instance, given the sentences “I love cats” and “I love dogs”, the instruction “Do they love animals?” will lead to a uniform response of “Yes/Certainly/...”; Conversely, distinct answers would be generated in response to “What animals do they love?”. Therefore, we believe that the expectation of answer representations given the prompt can serve as an instructionfollowing embedding. We support this hypothesis by our empirical observations in Section 4.2 on existing instruction-tuned LLMs (Ouyang et al., 2022; Chung et al., 2022; Zhang et al., 2023a; Touvron et al., 2023a,b) 1 which have demonstrated that hidden states corresponding to the generated answers show considerably better instruction-awareness compared to those derived from the prompt.
Our observations indicate that function words and phrases in the answers do not contribute to better embedding quality. For instance, the introduc-
1For simplicity, we use LLMs to refer to instruction-tuned LLMs for the rest of the paper.
459
tory phrase “Sure! Based on the input provided...” is irrelevant to the answers and is commonly found across various inputs. This redundancy can lead to inefficiency due to an increased decoding length, emphasizing the importance of answer brevity.
To effectively instantiate the embed-viaanswering idea, we propose an Instructionfollowing Embedder framework (INBEDDER), which is compatible with both large language models (LLMs) and smaller encoder-based LMs such as RoBERTa. Specifically, INBEDDER fine-tunes the LM on a union of 11 abstractive question answering (QA) datasets with ∼ 200, 000 paragraph-question-answer triplets where the answers are usually short and informative. To facilitate the model to learn (implicit) semantics, we choose abstractive QA in particular, as the answers cannot be directly extracted. We further simplify the answers by removing all the stopwords, resulting in an average answer length of 2.89 tokens.
Due to the scarcity of evaluations focusing on instruction-following capabilities in the literature, we introduce a suite of tasks aimed at testing the ability of embedders to be instruction-aware, including (1) a triplet task that selects the closer sentence to the anchor sentence based on two different instructions, (2) an instruction-following sentence similarity task, and (3) a task for clustering the same corpus under various instructions. Furthermore, we evaluate INBEDDER s’ robustness to the instructions by testing it on clustering datasets with either correct, implicit, or incorrect instructions. Our model is compared with both traditional text embedders as well as LLM-based embedders. The results demonstrate that our model can effectively process user instructions while generating highquality embeddings. Moreover, we empirically observe that the hidden states corresponding to the
first generated token can already effectively follow instructions, which makes it as efficient as traditional embedder methods by only requiring one forward pass of the LM. Finally, we propose to interpret the embedding clusters via post-processing on the generations of INBEDDER, and we observe that the clusters can reflect instruction-following capability when applying multiple instructions to the same corpus.
Our contributions are the following: • We address a novel and challenging problem:
instruction following of text embeddings and propose a framework, INBEDDER, to handle it by learning to answer user questions given inputs. • We provide a comprehensive assessment for instruction-following text embedders, including instruction awareness tests and instruction robustness tests, which intuitively reflect the models’ instruction-following capability. • We propose an approach for extracting explanations from embedding clusters. We show that these explanations further reflect instructionfollowing capability. • We open source our code, datasets, and model checkpoints to facilitate future research: https: //github.com/zhang-yu-wei/InBedder.

heading: 2 Related Works

heading: 2.1 Text Embedder
text: Text embedders empower modern natural language processing systems with a wide variety of abilities like clustering (Aggarwal and Zhai, 2012) and information retrieval (Karpukhin et al., 2020). In the representation space of text embedders, similar texts are embedded close to each other. Thus, Siamese networks (Reimers and Gurevych, 2019b) and contrastive learning (Gao et al., 2021) are proposed to learn the relative position of texts in the latent embedding space. Text embedders are fur-
ther strengthened by incorporating more weakly supervised text similarity annotations (Wang et al., 2022; Xiao et al., 2023) and architecture variants (Ni et al., 2022b). However, these mainstream text embedders only process general textual similarity, ignoring the changing view on textual similarity based on user demands. Instructor (Su et al., 2023) explores an instruction-based text embedder by concatenating instructions before the input texts. Our INBEDDER shows a substantially stronger instruction-following text embedder in instruction following by using expected answer distributions instead of concatenated instruction-text pairs as the representation.

heading: 2.2 Instruction Tuning
text: Instruction-following (Zhang et al., 2023a) of LLMs is one of the core abilities for them to capture the user intents, which makes LLMs popular among users. InstructGPT (Ouyang et al., 2022) is a first trial on instruction-following LLMs, which unearths the potential of LLMs to complete tasks under instructions from users. With an outstanding instruction-following ability from reinforcement learning with human feedback (RLHF), ChatGPT (OpenAI, 2023) achieves great success inside or outside the natural language community. The open-source instruction-following LLMs, like LLaMA (Touvron et al., 2023a,b), also provide valuable resources for researchers to study the instruction-following abilities of LLMs. Our study extends the idea of instruction-following from language modeling to text embeddings. Previously, it has been discovered that LLM can explore and manipulate various attributes of texts (Peng et al., 2023). Moreover, LLM hidden states can effectively represent space and time (Gurnee and Tegmark, 2023), an aspect of texts such as honesty (Zou et al., 2023) or a task defined by inputoutput pairs (Todd et al., 2023). Despite the potential, it is still unknown how to effectively aggregate these hidden states to produce a high-quality representation.

heading: 2.3 Goal-Driven Clustering
text: With the recent advancements of instructionfollowing LLMs, goal-driven clustering has been proposed to group text corpora according to a personalized goal (Wang et al., 2023). In order to address such a challenging yet novel problem, GoalEX (Wang et al., 2023) applies a two-step pipeline that first proposes cluster explanations with GPT-4
and then selects clustering assignments with another LLM. A user-oriented goal is included in the proposed step. Zhang et al. (2023b) proposes another method that can incorporate user instructions to first determine sentence relationships via a triplet selection task and then produce clusters via fine-tuning. These produced clusters can be helpful for multi-document summarization (Coavoux et al., 2019; Fabbri et al., 2019; Lu et al., 2020), especially those that are personalized. Our paper on the other hand directly produces embeddings that are shaped according to different instructions applied which does not require calling APIs of LLMs and potentially saves costs.

heading: 3 Problem Formulation

heading: 3.1 Instruction-following Embedder
text: We introduce the definition of instruction-following embedder in this section. A vanilla text embedder (denoted Emb(·) : X → Z) (Reimers and Gurevych, 2019a; Gao et al., 2021; Ni et al., 2022a,b; Wang et al., 2022; Xiao et al., 2023) embeds texts from token sequence space X into a D-dimensional vector space Z ⊆ RD, where similarities between two pieces of texts can be measured by a certain metric Sim(·, ·) : Z × Z 7→ R. These embeddings are usually designed to generically represent texts, i.e., they aim to capture the overall meaning. Such an approach, while versatile, often fails to align with a specific downstream application, e.g., grouping a corpus according to a particular interest or customizing a search engine with a targeted aspect. In this paper, we assume these goals can be specified by a user instruction I and then used to shape the embedding space without any fine-tuning to the text embedder. Under this circumstance, the similarity scores are conditional, i.e., Sim(·, ·|I).
The most straightforward approach is to just embed the concatenated instruction and input, which we will hereafter refer to as prompt,
Sim(X,X ′|I) = Sim(Emb(I ⊕X),Emb(I ⊕X ′))
where X,X ′ ∈ X and ⊕ is concatenation. In order to assess the instruction-following ability, we will present a series of tasks in Section 3.2 and 3.3 that require the model to understand the instructions. Instructor (Su et al., 2023), a previous work, utilized a contrastive objective alongside multi-task learning to develop a more general text embedder. Our experiments in Section 3.2 demonstrate
that Instructor does not adequately comprehend instructions. This is not surprising given the limited instruction diversity and the lack of encouragement to follow instructions during training. Our hypothesis. We hypothesize the responses of LLMs (Touvron et al., 2023a,b; OpenAI, 2023; Chung et al., 2022) can be embedded to produce instruction-following embedding. Specifically, the LLMs are prompted to generate a response Y ,
Y = LLM(I ⊕X)
where both X,Y ∈ X are from token sequence space. LLM(·) : X → X is a function that maps prompts to responses. Usually, there could be multiple valid Y for a given prompt. In order to accommodate instruction-following embedding, we offer a novel viewpoint, which treats the instruction I as a question about the input text X and encodes the expected answers (i.e. the responses to the question). In this paper, we study how to effectively embed expected answers.

heading: 3.2 Instruction Awareness Tests
text: Traditional generic embedding evaluation benchmarks, such as MTEB (Muennighoff et al., 2023) and SentEval (Conneau and Kiela, 2018), lack the ability to assess instruction awareness. In this work, we propose a set of new tasks specifically designed to comprehensively evaluate the capabilities of embedding models in this regard. We discuss the task formulations below and leave the detailed dataset creation procedures in Appendix A. IntentEmotion. Inspired by previous works (Zhang et al., 2023b, 2024), we employ triplet tasks with two contrasting criteria, i.e. the intent and emotion of an utterance. A triplet task is composed of three different user utterances {u1, u2, u3} where u1, u2 have the same intent but different emotions while u1, u3 have the same emotion but different intents, or vice versa. A success is defined under instruction Iint if d(zint1 , z int 2 ) < d(z int 1 , z int 3 ) where z represents the embeddings. On the other hand, it is said to be a success for instruction Iemo if d(zemo1 , z emo 2 ) > d(z emo 1 , z emo 3 ). Notice that the rankings are always reverted under the two criteria. We use the harmonic mean of two success rates as our metric. InstructSTSB. The original Semantic Textual Similarity (STS) Benchmark (Cer et al., 2017) lacks a definitive criterion for annotators to rely on, resulting in the subjectivity of the ratings. In contrast,
we create a STS task where the two sentences are similar or dissimilar based on different instructions. We measure the Spearman correlation from cosine similarities. Notice that a similar dataset was first proposed in Deshpande et al. (2023). The main differences are that (1) our dataset is created directly from the original test set of STSB 2 via brainstorming instructions; (2) our dataset only involves two ratings 0 and 1 indicating same or different, unlike the 1 ∼ 5 rating scale in their case, reducing subjectivity in the evaluation. NYTClustering. We present the clustering results for the New York Times (NYT) dataset (Sandhaus, 2008), which is categorized according to two annotations: topic and location of the news articles. The results are reported using the harmonic mean of the V-measure for both clustering types.

heading: 3.3 Instruction Robustness Tests
text: We further introduce an evaluation task specifically designed to assess the robustness of embedding models to various instructions. We employ clustering tasks to evaluate model performance in response to correct, implicit, and incorrect instructions. For each clustering task, 10 correct instructions are generated by instructing GPT-4 to paraphrase the original task instructions. Similarly, 10 implicit instructions and 10 incorrect instructions are produced by GPT-4 through either rephrasing of the instructions to convey the meanings implicitly or to diverge from the original task objective. Examples of these instructions are illustrated in Figure 9. The difference in average performance between correct and incorrect instructions is denoted as ∆ci, and the difference in average performance between implicit and incorrect instructions is denoted as ∆ii. See Appendix B for details.

heading: 4 Methodology
text: In this section, we introduce INBEDDER that is derived from observations on LLMs. We first define several ways to acquire sentence embeddings from LLMs in Section 4.1. Subsequently, we illustrate early observations in Section 4.2. Finally, we introduce a framework that fine-tunes an LLM to an instruction-following embedder, INBEDDER, in Section 4.4.
2https://huggingface.co/datasets/mteb/ stsbenchmark-sts/viewer/default/test

heading: 4.1 Encoding Methods
text: Contemporary LLMs are usually composed of one (encoder-/decoder-only) or two (encoder-decoder) transformer architectures with L layers. The input of the transformer is a sequence of embeddings [h10, · · · , hN0 ] where N is the length of prompt (I ⊕ X). Each layer will then produce an intermediate hidden state hl until the last layer which is used to predict the (N + 1)th output token. We first introduce two strategies to acquire a single aggregated embedding from an off-the-shelf LLM. Direct Encoding directly utilizes LLM hidden states. Since it is not obvious which hidden states contain the most relevant information to the prompt, we explore 5 aggregation methods for each layer: 1) The average of generation Y ’s hidden states with generation length Ng,
Embavg-genl = 1
Ng + 1
Ng∑
j=0
h (N+j) l ,
2) The average of prompt hidden states. This will serve as a direct comparison to “avg-gen”.
Embavg-pptl = 1 N − 1 N−1∑
i=1
hil,
3) The hidden states used to predict the first token in generations,
Emb1st-genl = h N l ,
4) The last generation hidden states,
Emblast-genl = h N+Ng l ,
5) The average of all hidden states,
Embavg-alll = 1
N +Ng (
N∑
i=1
hil +
Ng∑
j=1
hN+jl ),
In practice, we adjust the aggregation methods with regard to the uniqueness of each architecture, for example avg-all does not make sense in the context of encoder-decoder models. We provide more details on this issue in Appendix C. When Ng = 1 or |Y | = 1, all the above aggregation methods possess the same efficiency since we only have one forward pass.
While direct encoding is commonly applied for conventional encoders (Wang et al., 2022; Su et al., 2023), using only the input information might not
reveal the implicit features that can be inducted by answering the prompt. Thus, we propose Reencoding, which is a two-step approach that first produces the responses Y based on the prompts and then re-encode them using another embedder EmbR. Mathematically,
Embre-enc = EP (Y |I⊕X)[EmbR(Y )]
We then re-write the above with an empirical estimation,
Embre-enc = 1 |SY | ∑
Y∼SY EmbR(Y )
where SY is sampled from response distribution P (Y |I⊕X). We choose EmbR to be a (relatively) light-weight sentence transformer, thus the efficiency of re-encoding is similar to that of avg-gen.

heading: 4.2 Answer Speaks Louder
text: In this section, we show some early observations that guide us towards the design of INBEDDER. With the definitions in the previous section, we show the performance comparison among various aggregation methods on an existing LLM in Figure 2 left. To assess performance, we use Instruction Awareness Tests introduced in Section 3.2 that requires the embedders to comprehend not only the
raw texts but also the instructions. It is evident that hidden states derived from generations (avg-gen) consistently surpass those from prompts (avg-ppt). Additionally, averaging all hidden states, denoted as avg-all, does not enhance performance. Finally, an examination of three distinct models in Figure 2 reveals that re-encoding consistently outperforms all direct encoding methods, while increasing the sample size |SY | will further boost performance. These observations manifest our hypothesis that answers are more important for instructionfollowing embedder, in other words “answers speak louder”.

heading: 4.3 Answer Brevity Matters
text: One notable issue for using LLMs as embedders is their propensity to produce content that, while enhancing readability for humans, may not be directly relevant to the task at hand. For instance, llama-2-7b-chat frequently initiates responses with introductory phrases such as “Based on the input provided...” or “The topic of the news article is...” which are common across various requests. It is thus plausible to conjecture that the hidden states responsible for generating these superfluous contents contribute no useful information to the embedding task. Following this intuition, we conducted a simple experiment to validate the impact of filtering out hidden states associated with such content. Specifically, we compiled a list of candidate tokens for exclusion, which includes tokens present in the instruction, stopwords, and common phrases like “Based on”. While calculating “avg-gen”, we disregard hidden states linked to the generation of tokens from this list. The outcomes, depicted as yellow bars in Figure 3, indicate a marginal improvement in the performance of the three evaluated models upon the removal of non-informative content, thereby validating the assumption that these hidden states are indeed redundant.

heading: 4.4 Our INBEDDER
text: In order to effectively instantiate the embedvia-answering, we propose a novel fine-tuning framework leveraging existing curated questionanswering (QA) datasets. Specifically, we collect a set of 11 abstractive QA datasets 3, which sum up to ∼ 200, 000 paragraph-question-answer triplets. As discussed in Section 3.3, we treat the paragraph as the input, the question as the instruction, and train the model to generate the answers. Note that, we pre-process the answers so that all the stopwords are removed, which results in an average response length of 2.89. As will be demonstrated in the experiments, such a pre-processing step significantly contributes to our method. We then fine-tune the LM with an autoregressive objective.
We emphasize three inherent advantages of INBEDDER: (1) QA datasets usually have concise outputs that will promote the LLMs to respond eagerly without considering too much about readability. Refer to Figure 4 for an example. (2) Compared to multi-task datasets introduced in Su et al. (2023), our dataset offers significantly greater diversity in instructions, attributed to the variety of questions associated with each input paragraph, unconstrained by question format. And most importantly, the datasets are publicly available without any extra costs. (3) The auto-regressive objective induces better interpretability of generated embeddings via mining explanations from its generations.

heading: 5 Experiments
text: We introduce experimental setup in Section 5.1 and Section 5.2. We then present results on proposed instruction awareness tests and instruction robustness tests in Section 3.2 and Section 5.4. Finally, we show a comparison of generic embedding tasks in Section 5.5.
3We also include several multiple-choice QA datasets but remove all the wrong choices.

heading: 5.1 Implementations
text: We fine-tune INBEDDER from various kinds of language models such as (1) roberta-large, (2) opt-1.3b, (3) opt-2.7b, and (4) llama-2-7b 4. For masked language modeling-based robertalarge, we adapt our framework by appending mask tokens behind prompts with the same length as target tokens and then training with mask token prediction loss. During testing, we append 3 mask tokens to represent the answer. We consistently train for 1 epoch with a learning rate of 2 × 10−5. For INBEDDER, we always employ the same pattern to feed the inputs to the models, i.e. “### Input:\n{input}\n\n### Instruction:\n{instruction}\n\n### Response:”. For llama-2 chat models, we provide an extra prefix to induce shorter answers: “Your task is to give an answer according to the instruction and input. Please keep your answer short.”.
At test time, we allow the maximum generation length to be 40 for llama-2 chat models and 3 for our INBEDDER. We exclude hidden states corresponding to special tokens. We consistently use e5-large-v2 (Wang et al., 2022) as our re-encoder. Lastly, we set the maximum prompt length to be 512 (including instruction, input, and the tokens in the pattern). We further show that increasing prompt length in certain task can improve the performance in Appendix D. We train and evaluate these models with at most 4×A100 (PCIe).

heading: 5.2 Compared Methods
text: We compare with generic sentence embedding models: E5 (Wang et al., 2022) and Instructor (Su
4huggingface ids: “roberta-large”, “facebook/opt-1.3b”, “facebook/opt-2.7b”, “meta-llama/Llama-2-7b-hf”.
et al., 2023) 5. For E5, we diable the instructions since this model does support instructions natively. For Instructor, we enable the instructions with the prompt pattern “instruction: input”. We also compare with instruction-tuned models: llama-2 chat models (Touvron et al., 2023b) that are fine-tuned with RLHF 6 with prompt pattern “
 Your task is to give an answer according to the instruction and input. Please keep your answer short.\n\nInput: input\n\nInstruction: instruction\n\n### Your Answer: 
”. For roberta-large and opt models we compare with those checkpoints tuned on Alpaca (Taori et al., 2023). For Alpaca fine-tuning, we follow the dataset and hyperparameters of the original implementation 7, except that we simplified the prompt pattern.

heading: 5.3 Instruction Awareness Tests Results
text: In Figure 2 right, quite unexpectedly, we observe that using 1st-gen in INBEDDER achieves the best performance and it outperforms the other encoding methods by a significant amount. We hypothesize that although 1st-gen is utilized solely for decoding the first token in the generations, it may contain the most relevant information due to the model being trained on concise outputs. Further qualitative analysis in Table 6 shows that the first generated tokens usually correspond to the answer. We then present comparisons across various models in Table 1. Fine-tuning INBEDDER appears to be effective across a range of model sizes, from
5huggingface ids: “intfloat/e5-large-v2”, “hkunlp/instructor-large”
6huggingface id: “meta-llama/Llama-2-7b-chat-hf” and “meta-llama/Llama-2-13b-chat-hf”
7https://github.com/tatsu-lab/stanford_alpaca/ tree/main
the 355M model roberta-large to the 1.3/2.7b OPT and the 7b llama-2. We can also observe that without pre-processing, the performance will be significantly degraded on instruction-awareness according to llama-2-w/o-process, which further validates that conciseness of outputs is important.

heading: 5.4 Instruction Robustness Tests Results
text: Figure 5 presents the results obtained across three models, and see more datasets in Figure 8. Compared to instructor-large and llama-2-7b-chat, our model demonstrates larger values of ∆ci and ∆ii in general, as well as superior average performance when applying correct instructions. This indicates that our model exhibits a better understanding of correct or implicit instructions and possesses greater robustness against incorrect instructions.

heading: 5.5 Generic Sentence Embedding Tasks
text: Finally, we also compare performances on generic sentence embedding tasks. We choose a subset of tasks from the original MTEB (Muennighoff et al., 2023) benchmark, including: “TwentyNewsgroupsClustering”, “AskUbuntuDupQuestions”, “SciDocsReranking” and “StackOverflowDupQuestions”. The first task is a clustering task with Vmeasure as a metric. The last three tasks are reranking tasks that require the model to correctly identify the ones that are close to the query with cosine similarity metrics. We use the “mean average precision (MAP)” as our metric which is reported in MTEB. For each task, we design a task-level prompt that describes the requirements. We observe in Table 2 that our INBEDDER has a close performance to state-of-the-art embedders E5 (Wang et al., 2022) and Instructor (Su et al., 2023) than other LLMbased embedders, even though it was not trained
with a contrastive objective as in the state-of-the-art sentence transformers.

heading: 6 Embedder Clustering Interpretation
text: Interpreting neural embeddings has long been an aspiration in numerous research endeavors (Panigrahi et al., 2019; Trifonov et al., 2018). We show in this section that INBEDDER naturally possesses interpretability due to its instruction-following training objective. Specifically, we propose a method to “extract answers” from semantic clusters produced by the embedder.

heading: 6.1 Interpretation Methods
text: We directly post-process the generated sequences of INBEDDER to collect identifiable information about a cluster. To differentiate clusters, we initially collect outputs from each cluster following K-means clustering and concatenate these outputs into a single document per cluster. Subsequently, we employ Term Frequency-Inverse Document Frequency (Tf-idf) to vectorize these K documents, resulting in K feature vectors. The dimensions of each vector denote the relative frequency of a word’s occurrence in one document compared to its occurrence in others. Hence, we rank feature words according to the corresponding value in the feature vector, which will then be designated as cluster keywords.

heading: 6.2 Results
text: Table 3 presents explanations derived from llama-2-7b-InBedder. When compared to the label components of each cluster, the top words
collected effectively capture the unique characteristics of each cluster. To showcase the instructionfollowing capability of INBEDDER, cluster explanations are further illustrated with three distinct instructions in Figure 6 using the Yelp review dataset (Zhang et al., 2015) (originally designed for sentiment analysis). The top words distinctly delineate the differences between clusters, in accordance with the provided instructions. For example, under “Instruction B” various different products that the customers are reviewing about are produced from clusters, while on the other hand, under “Instruction C”, variations in average sentence lengths are observed, indicating the degree of detail present in the review.

heading: 7 Conclusions and Future Work
text: Our work addresses a novel problem, text embedding with instruction-following. We propose INBEDDER to produce desirable embeddings from LLMs via generating expected answers. The method is inspired by observations on existing LLMs. Our text embedder model llama-2-7bINBEDDER outperforms both traditional sentence transformers and aggregated embeddings from LLMs on instruction-awareness tests, and instruction robustness tests and achieves close performance on traditional generic tasks. We also show that INBEDDER inherently is applicable for embedding cluster explanation which will significantly facilitate user-oriented dataset analysis. We encourage future works to investigate more efficient solutions which is important in large-scale retrieval systems.
Limitations
Efficiency. Our model is not sufficiently efficient for large-scale retrieval tasks. In retrieval, corpus is usually encoded as vector embeddings beforehand, the only operation conducted is to encode the query and to compute the cosine similarities between the query and corpus. However, INBEDDER requires encoding the entire corpus w.r.t. each user query which results in significant latency. However, one possible solution is to first select the most similar candidates and then use INBEDDER as a querydependent reranker. Effectiveness on generic tasks. The results in Table 2 show that INBEDDER does not surpass traditional sentence transformers on especially generic reranking tasks. (1) Our ambition is to provide an instruction following embedder that could potentially facilitate user-oriented tasks rather than optimizing for high-performing sentence embedding and we leave the exploration on that dimension in future works. (2) INBEDDER might benefit from better prompt design or task description which we have discussed in Section 5.4.

heading: Acknowledgement
text: Our work is sponsored in part by NSF CAREER Award 2239440, NSF Proto-OKN Award 2333790, Cisco-UCSD Sponsored Research Project, as well as generous gifts from Google, Adobe, and Teradata. Any opinions, findings, and conclusions or recommendations expressed herein are those of the authors and should not be interpreted as necessarily representing the views, either expressed or implied, of the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for government purposes not withstanding any copyright annotation hereon.

heading: C Details on Direct Encoding
text: The direct encoding proposed in Section 4.1 are all compatible with decoder-only transformers.
For encoder-decoder models such as flan-t5, because of the two separated models, we remove avg-all since the hidden states are not in the same space. Besides, we extract avg-ppt from encoder and avg-gen&1st-gen&last-gen from decoder respectively. Notice that for 1st-gen, we use the hidden states for the BOS token in the decoder side. For encoder-only models, we remove 1st-gen and last-gen. We implement the sentence embedding function by generating tokens first 8 and then cache the intermediate hidden states for further compute. Considering the efficiency, avg-ppt&1st-gen only require single forward pass while the others require iterative generations and thus depending on the generation length.
D Impact of Maximum Prompt Length
Our method is orthogonal to the maximum prompt length being used. We test INBEDDER with maximum prompt length beyond 512 on NYTClustering. From Table 4, we observe that the performance significantly increases when using longer inputs. Thus we argue that the model still has the potential to work with a longer prompt, and we leave further investigation to future work.
8Notice that, in huggingface (Wolf et al., 2019), both decoder-only and encoder-decoder model can use “generation” function: https://huggingface.co/docs/transformers/ main_classes/text_generation. For encoder-only, we simply concatenate the “[MASK]” tokens after the prompts for generation.
Feedbacks
cluster 1 (lowest entropy) 2 3 (highest entropy)
compoents Structure, Coherence:229 Inclusion of Main Points:9 Content Accuracy:7 Content Accuracy:148 Inclusion of Main Points:134 Structure, Coherence:10 Inclusion of Main Points:109 Content Accuracy:97 Structure, Coherence:13
top words
sentence,better, structure,written, improved,flow, unclear,read...
dislike,accurate, author,generated, mention,machine, accuracy,inaccurate...
like,feedback, post,likes, good,human, advice,relationship...
FewRel
cluster 1 (lowest entropy) 2 3 63 64 (highest entropy)
compoents taxon rank:24 heritage designation:70 located in the administrative...:1 location:1 taxon rank:46 instance of:1 said to be the same as:1 country of citizenship:1 movement:9 religion:6 work location:6 said to be the same as:5 language of work or name:23 said to be the same as:16 followed by: =10 applies to jurisdiction:7
top words family,species, gastropod,marine, sea,urothoidae, psolidae,feed... historic,registe, places,listed, national,historical, house,significance...
family,subfamily, order,genus, families,tribe, sent,orders...
friends,beowulf∗, personalities,jewish, slave,personality, independence,owner... language,minor, wikipedia,candela, french,translation, flag,major...
FewNerd

heading: Future Work
text: Thus we argue that the model still has the potential to work with a longer prompt, and we leave further investigation to future work. 8Notice that, in huggingface (Wolf et al. , 2019), both decoder-only and encoder-decoder model can use “generation” function: https://huggingface. co/docs/transformers/ main_classes/text_generation. For encoder-only, we simply concatenate the “[MASK]” tokens after the prompts for generation. Feedbacks
cluster 1 (lowest entropy) 2 3 (highest entropy)
compoents Structure, Coherence:229 Inclusion of Main Points:9 Content Accuracy:7 Content Accuracy:148 Inclusion of Main Points:134 Structure, Coherence:10 Inclusion of Main Points:109 Content Accuracy:97 Structure, Coherence:13
top words
sentence,better, structure,written, improved,flow, unclear,read. dislike,accurate, author,generated, mention,machine, accuracy,inaccurate. like,feedback, post,likes, good,human, advice,relationship. FewRel
cluster 1 (lowest entropy) 2 3 63 64 (highest entropy)
compoents taxon rank:24 heritage designation:70 located in the administrative. :1 location:1 taxon rank:46 instance of:1 said to be the same as:1 country of citizenship:1 movement:9 religion:6 work location:6 said to be the same as:5 language of work or name:23 said to be the same as:16 followed by: =10 applies to jurisdiction:7
top words family,species, gastropod,marine, sea,urothoidae, psolidae,feed. historic,registe, places,listed, national,historical, house,significance. family,subfamily, order,genus, families,tribe, sent,orders. friends,beowulf∗, personalities,jewish, slave,personality, independence,owner. language,minor, wikipedia,candela, french,translation, flag,major. FewNerd