One Embedder, Any Task: Instruction-Finetuned Text Embeddings
Year: 2023

Abstract:
We introduce INSTRUCTOR, a new method for computing text embeddings given task instructions: every text input is embedded together with instructions explaining the use case (e.g., task and domain descriptions). Unlike encoders from prior work that are more specialized, INSTRUCTOR is a single embedder that can generate text embeddings tailored to different downstream tasks and domains, without any further training. We first annotate instructions for 330 diverse tasks and train INSTRUCTOR on this multitask mixture with a contrastive loss. We evaluate INSTRUCTOR on 70 embedding evaluation tasks (66 of which are unseen during training), ranging from classification and information retrieval to semantic textual similarity and text generation evaluation. INSTRUCTOR, while having an order of magnitude fewer parameters than the previous best model, achieves state-of-the-art performance, with an average improvement of 3.4% compared to the previous best results on the 70 diverse datasets. Our analysis suggests that INSTRUCTOR is robust to changes in instructions, and that instruction finetuning mitigates the challenge of training a single model on diverse datasets. Our model, code, and data are available at https: //instructor-embedding.github.io.

heading: 1 Introduction
text: Text embeddings represent discrete text inputs (e.g., sentences, documents, and code) as fixed-sized vectors that can be used in many downstream tasks. These tasks include semantic textual similarity (Agirre et al., 2012; Marelli et al., 2014; Cer et al., 2017; Lin et al., 2018), information retrieval (Mitra et al., 2017; Karpukhin et al., 2020; Izacard et al., 2022), automatic text evaluation (Zhang et al., 2020; Sellam et al., 2020; Hessel et al., 2021), prompt retrieval for in-context learning (Liu et al., 2022; Rubin et al., 2022; Su et al., 2022), and beyond. Recently, we have seen dramatic advances
∗Equal contribution.
dings, depending on the end task (e.g., duplicate question detection, information retrieval, and topic classification).
in learning text embeddings (Kiros et al., 2015; Conneau et al., 2017; Logeswaran and Lee, 2018; Reimers and Gurevych, 2019; Gao et al., 2021; Ni et al., 2021, 2022) that perform well on their intended tasks or datasets.
However, most existing embeddings can have significantly degraded performance when applied to new tasks or domains (Thakur et al., 2021; Muennighoff et al., 2022). For example, DPR (Karpukhin et al., 2020) is stronger for retrieval than text similarity tasks, and vice versa for SimCSE (Gao et al., 2021). Moreover, existing embeddings usually perform poorly when applied to the same type of task but in different domains such as medicine and finance. A common method to address this issue is to further finetune the embeddings on datasets in downstream tasks and domains, which often requires a lot of annotated data (Guru-
1102
rangan et al., 2020). In this paper, we hypothesize that text embeddings (even for the same text input) can be adjusted to different downstream applications using task and domain descriptions, without further task- or domain-specific finetuning.
We introduce INSTRUCTOR (Instruction-based Omnifarious Representations), a single multitask model that generates task- and domain-aware embeddings given a text input and its task instructions. It achieves state-of-the-art performance on massively many downstream embedding tasks without any training. At the core of our approach is instruction-based finetuning (Zhong et al., 2021; Min et al., 2022; Sanh et al., 2022; Wei et al., 2022): we embed every input together with its end task and domain instruction, departing from prior approaches to embeddings that only take text input. INSTRUCTOR embeds the same input into different vectors for different end goals (e.g., Who sings the song “Love Story”? is embedded into three different vectors for different tasks in Fig. 1). As shown in Fig. 2, INSTRUCTOR is trained on MEDI, our new collection of 330 text embedding datasets newly annotated with human-written task instructions (§2.3). We train INSTRUCTOR with a contrastive loss over all datasets that maximizes the similarity between semantically related text pairs while minimizing unrelated pairs.
We extensively evaluate INSTRUCTOR on diverse domains (e.g., finance, medicine, and news)
and a variety of downstream applications (a total of 70 embedding evaluation datasets, including 66 not seen during training), spanning classification, semantic textual similarity, information retrieval, text generation evaluation, and prompt retrieval for incontext learning. INSTRUCTOR significantly outperforms prior state-of-the-art embedding models by an average of 3.4% over the 70 diverse datasets. INSTRUCTOR also outperforms a variant that is trained without task instructions (§4), demonstrating the importance of instructions to create taskaware embeddings. Our analysis shows that instruction finetuning addresses the challenge of training a single model on diverse datasets (§4.1). Further, we demonstrate that the task diversity of MEDI makes the performance of INSTRUCTOR particularly robust to paraphrases in instructions (§4.2). Overall, these results strongly suggest that instruction finetuning should be adopted broadly for text embeddings, which we support by sharing all of our models and code.

heading: 2 INSTRUCTOR
text: INSTRUCTOR encodes inputs together with task instructions, thereby providing task-specific representations that can be used for many downstream language tasks, without any additional training. Here we introduce the architecture of INSTRUCTOR (§2.1), present how we perform multitask instruction-based finetuning (§2.2), and describe
how we collect and annotate the MEDI training data (§2.3). By default, we refer "task" to a dataset, and use them interchangeably throughout the paper, while a "task category", such as Retrieval, includes many tasks.

heading: 2.1 Embedding Architecture
text: We build INSTRUCTOR, based on the single encoder architecture (Izacard and Grave, 2021; Ni et al., 2021, 2022). Following prior work (Ni et al., 2021, 2022), we use GTR models as the backbone encoder (GTR-Base for INSTRUCTOR-Base, GTRLarge for INSTRUCTOR, GTR-XL for INSTRUCTOR-XL). The GTR models are initialized from T5 models, pretrained on a web corpus, and finetuned on information search datasets. The availability of different sizes in the GTR model family allows us to explore the scaling behaviors of instructionfinetuned embedding models. Given an input text x and a task instruction Ix, INSTRUCTOR encodes their concatenation Ix ⊕ x. We then generate a fixed-sized, task-specific embedding EI(Ix, x) by applying mean pooling to the last hidden representations over the tokens in x.

heading: 2.2 Training Objective
text: INSTRUCTOR is trained by formulating a wide variety of tasks as a text-to-text problem of distinguishing good/bad candidate outputs y ∈ {y+, y−i } given an input x, where a training sample corresponds to the tuple (x, Ix, y, Iy), with Ix and Iy being instructions associated with x and y, respectively. For example, in a retrieval task, x is a query, and good/bad y is a relevant/irrelevant document from some document collection. For a textual similarity task, the input and output have a similar form and typically come from the same source collection. For a classification task, training samples can be formed by choosing y as text sequences associated with the same vs. different classes for good vs. bad examples (Details about pair construction are in §2.3). The input and output instructions depend on the task. For symmetric tasks such as textual similarity, where the input and output have the same form and encoding objective, the instructions are the same. For asymmetric tasks such as retrieval, where the input is a single sentence query and the output is a document, the instructions reflect that difference.
The goodness of candidate y for input x is given by similarity s(x, y) that is the cosine between
their INSTRUCTOR embeddings: s(x, y) = cos(EI(Ix ⊕ x),EI(Iy ⊕ y)) Following Ni et al. (2021), we maximize the similarity between positive pairs (x, y+) and minimize negative pairs {(x, y−i )}ki=1, where k denotes the number of negative pairs per positive pair. Specifically, our training objective is:
L = e s(x,y+)/γ
∑y∈B es(x,y)/γ , where γ is the softmax temperature and B is a union of (x, y+) and {(x, y−i )}ki=1. Further following Ni et al. (2021), we compute the same loss with x and y swapped and add it to the previous loss (i.e., bidirectional in-batch sampled loss).

heading: 2.3 MEDI: Multitask Embedding Data with Instructions
text: There are no existing datasets that consist of a variety of tasks for embedding training with instructions. We thus construct a collection of 330 datasets with instructions across diverse task categories and domains: Multitask Embeddings Data with Instructions (MEDI).
Data Construction We build MEDI by combining 300 datasets from SuperNaturalInstructions (super-NI; Wang et al., 2022) with 30 datasets from existing collections designed for embedding training.
The super-NI datasets come with natural language instructions, but positive and negative pairs are not provided. We construct these pairs by using Sentence-T5 embeddings (Ni et al., 2022),1 denoted with E(⋅). For the classification datasets, we calculate the pairwise cosine similarity between examples based on input text embeddings cos(E(xi),E(xj)). An example xi with a high similarity to xj is used to create a positive pair if both examples have the same class label (y+j = yi), and a negative pair if the labels differ (y−j ≠ yi). For the remaining tasks where the output labels are text sequences, the following scores are first computed:
spos = cos(E(xi),E(xj)) + cos(E(yi),E(yj)) sneg = cos(E(xi),E(xj)) − cos(E(yi),E(yj))
1We do not include instruction for Sentence-T5 as it is not fine-tuned with instructions.
We select example pairs with the highest spos as positive pairs and highest sneg as hard negative pairs. We use one hard negative together with inbatch sampled negatives in the training. Our later analysis shows that the training data from super-NI particularly improve the instruction robustness in evaluation due to the diverse task definitions (§4.2).
The other 30 embedding training datasets come from the Sentence Transformers embedding data,2 KILT (Petroni et al., 2021), and MedMCQA (Pal et al., 2022). These 30 datasets already contain positive pairs; a few of them, such as MSMARCO (Bajaj et al., 2016) and Natural Questions (Kwiatkowski et al., 2019), also contain hard negative pairs. Following Ni et al. (2021), we use four negative pairs (hard or in-batch negatives) during the model finetuning process. Since all of these datasets do not have instructions, we develop a unified instruction template and manually write a specific prompt for each dataset, as described next.3 We release these instructions together with our MEDI data.
Instruction Annotation Each training instance from MEDI is a tuple (x, Ix, y, Iy), where the nat-
2 https://huggingface.co/datasets/
sentence-transformers/embedding-training-data. 3All prompts are reviewed by multiple authors independently to make sure they consistently follow our template.
ural language instructions Ix and Iy describe how the embeddings of x and y are used for the task. For example, in open-domain QA (e.g., Natural Questions in Table 1), Ix is “Represent the Wikipedia question for retrieving supporting documents; Input: ,” and Iy is “Represent the Wikipedia document for retrieval; Input: .”
To make instructions consistent across all datasets in MEDI, we design a unified instruction format that consists of the following parts (see Table 4 in the appendix for instances of each part):
• Text Type specifies the type of input text that we encode using the embedding model. For example, for an open-domain QA task, the input type of the query is a question, while the input type of the target is a document. • Task Objective (Optional) describes the objective of how the input text is used in a task. For example, for a classification task, the task objective is to classify the sentence into some category, while the task objective of the retrieval is to retrieve a relevant document. Because not all sentences are associated with a specific task (e.g., STS targets general encoding), we make this part optional. • Domain (Optional) describes the task domain. For example, for NewsIR, the domain of the task is news. Because not all tasks spec-
ify a domain (e.g., STS deals with general statements),this part is also optional.
The final instruction takes the following format: “REPRESENT THE (DOMAIN) TEXT TYPE FOR TASK OBJECTIVE:." Appendix 8 shows instructions for each dataset in MEDI.

heading: 3 Experiments
text: We train INSTRUCTOR on the MEDI data and evaluate it on a wide range of 70 downstream tasks. Specifically, we use the MTEB benchmark from recent work (Muennighoff et al., 2022), which consists of 56 datasets over 7 diverse task categories, such as classification, reranking, and information retrieval. We then further apply INSTRUCTOR to prompt retrieval for in-context learning and text generation evaluation. In all three settings, INSTRUCTOR achieves the state-of-the-art performance. See Appendix §A and §B for our detailed settings.

heading: 3.1 Main Results
text: Table 2 presents the results from INSTRUCTOR and the baselines over the three benchmarks: MTEB, Billboard, and prompt retrieval. We conduct headto-head comparison between INSTRUCTOR and
GTR models with the same size. We also include the performance of other representative models for reference, while they are not meant for direct comparison.
INSTRUCTOR achieves the best performance on all three benchmarks on average. Compared to GTR-Large (335M), from which INSTRUCTOR is initialized, instruction finetuning enhances the performance by 5.7%, 18.3%, and 5.7% in MTEB, Billboard, and prompt retrieval respectively. Specifically, among all task categories, INSTRUCTOR (335M) demonstrates large improvements over GTR-Large on the text evaluation (18.3%), classification (10.1%), and clustering tasks (8.9%). Particularly noteworthy is INSTRUCTOR’s performance compared to the previous stateof-the-art model, Sent-T5-XXL (58.4 vs. 56.5 on average), despite the fact that INSTRUCTOR has one order of magnitude fewer parameters (335M vs. 4.8B).
As expected, the retrieval-based models (e.g., GTR-XXL) show strong performance on retrieval and reranking but significantly lag behind on STS and classification. Conversely, similarity-based models (e.g., Sent-T5-XXL) perform well on STS, classification, and text evaluation, but not on re-
trieval. It suggests that these baselines tend to generate specialized embeddings that only excel at certain tasks, while INSTRUCTOR provides universal embeddings that perform well on diverse task categories.

heading: 4 Analysis and Ablations
text: We demonstrate INSTRUCTOR enables universal text embeddings for many diverse tasks. Here we analyze our results from various perspectives: the importance of instructions (§4.1), instruction robustness (§4.2) and complexity (§4.3), model sizes (§4.4), domain shifts (§4.5), and qualitative analysis (§4.6). By default, we report average performance across all categories.

heading: 4.1 Instructions Enable Diverse Training
text: Here we analyze the importance of instructions when training data are diverse. We first split MEDI into symmetric (e.g., text similarity) and asymmetric groups (e.g., open-domain QA), as defined in §2.3 (see Table §5 in the appendix for details about the symmetric and asymmetric groups). We then train INSTRUCTOR with or without instructions on each group separately.
As shown in Fig. 3, INSTRUCTOR finetuned without instructions yields performance similar to or better than the original GTR model (dotted line), if the data are symmetric or asymmetric only.
However, INSTRUCTOR suffers if finetuned without task instructions on the combination of both types of data (entire MEDI). In contrast, finetuning with instructions enables the model to benefit from the combination of symmetric and asymmetric data (see that the rightmost bar gets additive performance gains from the asymmetric and symmetric tasks). This result demonstrates the importance of instruction finetuning when diverse data are used for embedding training. Note that training on symmetric tasks only without instructions is similar to Sent-T5. Similarly, training on asymmetric tasks only without instructions is similar to GTR, which is also trained on asymmetric opendomain QA datasets. Departing from these prior methods, instruction-based finetuning enables diverse training on both types.

heading: 4.2 Instruction Robustness
text: Previous work (Sanh et al., 2022; Zhou et al., 2022) shows that instruction-finetuned language models are not robust to paraphrased instructions. Here we measure INSTRUCTOR’s robustness to variation in human-written instructions.
Specifically, we write five paraphrased instructions for all evaluation datasets (Table 6 in Appendix) and measure INSTRUCTOR’s performance gap between the best-performing and the worstperforming instructions. Fig. 4 shows that inclusion of 300 super-NI datasets is critical to the robustness of INSTRUCTOR. Removing these datasets from
training (w/o super-NI) substantially increases the performance gap between the best- and worstperforming instructions, suggesting that super-NI’s diverse instructions help the model handle different formats and styles.

heading: 4.3 Complexity of Instructions
text: Here we further analyze the role of instructions over varying degrees of their complexity. Specifically, we consider four levels of instruction complexity: N/A (no instructions), dataset tags, simple instructions, and detailed instructions (the original instruction format, §2.3). In the dataset tag setup, each example is prepended with its dataset name. For instance, on the Natural Questions dataset, the query is formatted as "Natural Questions; Input: who sings the song Love Story"). In the simple instruction setup, we use one or two words to describe the domain (e.g., for Natural Questions, the input query is Wikipedia Questions; Input: who sings the song Love Story). Fig. 5 shows their average performances across all task categories. Even with trivial dataset tags, INSTRUCTOR outperforms the original GTR model, illustrating the effectiveness of instructions for diverse training. As more information is provided in the instruction (from tag to simple and from simple to detail), we observe consistent improvements.

heading: 4.4 Model Sizes and Instruction Finetuning
text: Fig. 6 studies the influence of model sizes. Specifically, we use GTR-Base (0.1B), GTR-Large (0.3B),
and GTR-XL (1.5B). They are pretrained on the same corpus and differ only in the encoder size (the embedding sizes are the same). We compare models of various sizes and report the average performance across all the categories. As the encoder transformer model scales up, the performance continues to increase for both GTR and INSTRUCTOR. Nonetheless, the improvement in INSTRUCTOR is more pronounced, perhaps because embeddings with instructions benefit from larger capacities. This implies that large models are more generalizable to compute texts in various domains and task types, providing embeddings for general purposes. Further scale-ups are left to future work.

heading: 4.5 Instructions Mitigate Domain Shifts
text: One advantage of instruction-based finetuning is that it improves models’ ability to generalize to unseen domains and tasks. To demonstrate this effectiveness, we found three unseen domains that INSTRUCTOR was not trained on: geography, biology, and civil comments. As shown in Table 3, INSTRUCTOR largely improves (above the average improvement) GTR-Large’s performance on all three domains, indicating that instructions can help more when applying models to unseen or uncommon domains.

heading: 4.6 Qualititive analysis
text: In this qualitative analysis, we use T-SNE (van der Maaten and Hinton, 2008) to visualize two example of classification with and without instructions. The desired outcome is, for pairs with the same sentiment to be closer together, and pairs with different
sentiment to be farther apart. As shown in Fig. 7, without instructions, the green dot pairs (different sentiment) are closer together in the embedding space, while the red dot pairs (same sentiment) are farther apart. However, with instructions, our method (INSTRUCTOR) successfully encodes the red dot pairs into close embeddings and correctly classifies the pairs. The distance between the green dot pairs with different sentiment is also larger in the embedding space with instructions.

heading: 5 Related Work
text: Text Embeddings Text embeddings are useful in many applications such as information retrieval (Thakur et al., 2021), text similarity (Gao et al., 2021), prompt retrieval for in-context learning (Su et al., 2022), classification (Reimers and Gurevych, 2019), and beyond. Much prior work develops different embedding models for different applications. For example, SBERT (Reimers and Gurevych, 2019) and SimCSE (Gao et al., 2021) are applied solely to text similarity and classifi-
cation tasks, while DPR (Karpukhin et al., 2020) and Contriever (Izacard et al., 2022) focus on information retrieval. Different from Sentence-T5 trained only on symmetric data or GTR trained only on asymmetric data, we combine both groups of datasets and build MEDI, which is then used to train INSTRUCTOR with instructions. Muennighoff et al. (2022) introduced the massive text embedding benchmark, which can be used to evaluate embedding models on a variety of embedding tasks, spanning reranking, classification, information retrieval, bitext mining, pair classification, STS, and summarization. Their benchmark shows that models performing well on one task may not perform well on other tasks. The poor zero-shot transfer abilities of existing embedding models make it difficult to use them in applications where only few labeled data are available. This motivates us to develop a single embedding model that is applicable to a variety of tasks and has better generalization to unseen tasks.
Instruction Finetuning Recent work demonstrated that instruction-finetuned language models could perform new tasks given a natural language instruction (Mishra et al., 2022; Zhong et al., 2021; Min et al., 2022; Sanh et al., 2022; Wei et al., 2022; Wang et al., 2022; Ouyang et al., 2022). Nonetheless, instruction finetuning has yet to be studied in the context of broadly-applicable embeddings. In this work, we explore finetuning embedding models to follow human instructions where the instruction specifies eventual use cases. Concurrent work demonstrated that instructions could facilitate information retrieval (Asai et al., 2022), which is related to our INSTRUCTOR design. They used instructions to build a task-aware retrieval system and conducted evaluations on the retrieval task; we build a general-purpose embedding model with instructions that can be applied to 8 tasks categories (Fig. 2), including retrieval, text similarity, clustering, and text evaluation.

heading: 6 Conclusion
text: We introduced INSTRUCTOR, a single model that creates broadly-applicable text embeddings using natural language instructions. We constructed MEDI, a collection of diverse datasets, to finetune INSTRUCTOR with instructions. Our extensive experiments showed that INSTRUCTOR achieves state-of-the-art performance on text embedding benchmarks, as well as prompt retrieval for few-
shot in-context learning. We hope that researchers and practitioners will benefit from our embeddings or our datasets for tasks of their interest.

heading: 7 Limitations
text: Although INSTRUCTOR significantly improves the baseline GTR performance, we were only able to use four negative examples during the model finetuning process due to computation constraints. However, negative examples have been shown to play an important role in contrastive learning (Robinson et al., 2021). We hope that future work will scale up the number of negatives used during finetuning and investigate various methods for mining hard negatives. Additionally, we do not have enough computation resources to apply multitask instruction finetuning to GTR-XXL (4.8B parameters), which is also an area for future exploration.
At the core of INSTRUCTOR is the instruction design. While our current unified instruction format has demonstrated effectiveness, future research can explore other instructional elements to further improve performance. For example, previous work (Wang et al., 2022) have shown that incorporating demonstration examples and explanations can be beneficial for instruction-finetuned language models.

heading: Acknowledgements
text: We thank Akari Asai, Jack Lin, Minghan Li, and the ARK group at UW for their helpful feedback on this work.

heading: A Training Setups
text: Minibatch Sampling Training is performed on a combination of all training datasets in MEDI. Since the number of examples in each dataset is different in orders of magnitude, we downsample large ones. Details for the downsampled numbers of examples on each dataset are shown in Table 5 in the appendix. At each step, we first randomly select a dataset and then construct a minibatch only using the examples from that dataset. In this way, we ensure that in-batch negatives are sampled from the same dataset, thereby preventing the model from using task differences to predict the negative label. We use the maximum batch size that fits the machine memory and run all our experiments on 40GB A100 GPUs.
Training We initialize INSTRUCTOR with the GTR-Large model (Ni et al., 2021, 335M parameters)4 and finetune it on MEDI using the AdamW optimizer with learning rate 2 × 10−5 and warmup ratio 0.1. We use a softmax temperature of 0.01 and finetune INSTRUCTOR for 20K steps.
Baselines We use the official MTEB benchmark for comparisons, but here we highlight several strong baselines with the following two types. The first class of baselines is embedding models specializing in information retrieval: ContrieverMS (Izacard et al., 2022), GTR (Ni et al., 2021), and coCondenser-MS (Gao and Callan, 2022). They are all trained on open-domain QA datasets such as MS MARCO (Bajaj et al., 2016). The second class of baselines focuses on semantic textual similarity: SimCSE (Gao et al., 2021), SentT5 (Ni et al., 2022), and SGPT-NLI (Muennighoff, 2022). They are mainly trained on symmetric paraphrase datasets such as NLI (Williams et al., 2018) and the Quora question pairs.5 All of these baselines are based on pretrained language models, achieving strong performance on the MTEB leaderboard. In particular, Sent-T5-XXL and GTR-XXL (both with 4.8B parameters) achieve the first and second best average performances.
4 https://huggingface.co/sentence-transformers/
gtr-t5-large. 5 https://www.quora.com/q/quoradata/.

heading: B Embedding Evaluations
text: Here we provide a high-level summary of the evaluation tasks (Table 1). Following MTEB (Muennighoff et al., 2022), Billboard (Kasai et al., 2022a), and prompt retrieval (Su et al., 2022), we split 70 evaluation datasets into 9 categories by task objectives. Out of the 70 evaluation tasks, 66 are unseen during training (See Table 5 for datasets included during training), Table 1 for examples and instructions for the evaluation datasets.
B.1 Massive Text Embedding Benchmark
MTEB (Muennighoff et al., 2022) is a comprehensive embedding evaluation benchmark that aims to provide a holistic view of current embedding models’ performance and to discover universal text embeddings applicable to a wide range of tasks. It combines several conventional benchmarks (e.g., BEIR, Thakur et al., 2021, and STS, Cer et al., 2017) and spans a wide range of domain-specific datasets, including science, biology, and medicine. Following Muennighoff et al. (2022), we also report the average performance over 56 datasets. For each task family, we briefly describe the task objective, evaluation metric, and how embeddings are used.
Retrieval Given a query q and a corpus D ={p1, p2...pn}, retrieval aims to find the most relevant documents pi in D for query q. The embedding model is used to embed q and p1...pn into fixed-sized vectors, and then the similarity between q and pi is measured by their embedding cosine similarity. There are 14 diverse datasets (e.g., Natural Questions, Scifact, and NFCorpus) together with the community question-answering (CQA) benchmark (Hoogeveen et al., 2015). We use NDCG@10 (Normalized Discounted cumulative gain at rank position 10) to measure the performance.
Reranking Reranking ranks a list of documents based on their relevance to a query. Given a query q and a list of documents D = {p1, p2...pn}, the embedding model computes embeddings of both the query and documents, which are then used to rank the documents based on their cosine similarities. We use MAP (mean average precision), a standard metric in reranking, to measure performance.
Clustering The goal of clustering is to group similar documents into meaningful clusters. Given a
set of documents, the encoder maps each document into an embedding. The k-means clustering algorithm is then used to partition the embedded documents into clusters. The clustering performance is measured by the v-measure that is independent of the permutations of clustering labels (Rosenberg and Hirschberg, 2007).
Pair Classification Pair classification tasks aim to predict a binary label for a pair of texts. An example of this task is paraphrase identification, where the goal is to predict whether two sentences are paraphrases of each other. Given a sentence pair(t1, t2), the embedding model encodes t1 and t2 separately. The cosine similarity between the two embeddings is then used to predict the label. The average precision score is measured for evaluation.
Classification Classification is a popular way to evaluate the quality of embeddings (Conneau and Kiela, 2018). For each example in the classification dataset, the embedding of the input text is used as features to a classifier. The classifier is trained on the training data while sentence embedings are kept frozen. We report the classification accuracy on the test set as the evaluation metric.
STS Semantic textual similarity (STS) tasks evaluate the similarity between two sentences. Given a sentence pair (t1, t2), the embedding model maps t1 and t2 into embeddings separately, and then the similarity between t1 and t2 is measured by their embedding cosine similarity. The evaluation metric is Spearman’s rank correlation, which measures the correlation between the similarity scores and human judgements.
Summarization Automatic summarization evaluation aims to evaluate the quality of a machinegenerated summary given a reference summary. While human evaluations are considered more accurate, automatic evaluations allow for fast, inexpensive development cycles (Khashabi et al., 2022). Given a reference summary r and a machinegenerated summary t, the embedding model maps them into embeddings separately, and we compute the cosine similarity between r and t. Spearman’s rank correlation is reported between human judgements and automatic scores.
B.2 Prompt Retrieval
Large language models have demonstrated the ability of in-context learning, where the model can
perform downstream tasks by conditioning generation on a few task demonstrations (Liu et al., 2021). Su et al. (2022) introduce the prompt retrieval task, where the goal is to retrieve a few in-context learning (i.e., demonstration) examples from annotated examples given a test instance. The embedding model is used to encode all annotated examples and to find the few most similar examples to the test instance based on the cosine similarity. Following Su et al. (2022), we use the retrieved examples for in-context learning on GPT-J (Wang and Komatsuzaki, 2021) over 11 diverse downstream tasks (e.g., classification, multiple choice, and textto-SQL) that are not included in MEDI (thus zeroshot settings). We compare different embedding methods by measuring the average performance on these downstream tasks.
B.3 Automatic Evaluation for Generation
Similar to summarization evaluation in MTEB, we use the Billboard benchmark (Kasai et al., 2022a) to apply INSTRUCTOR to automatic evaluations for three additional text generation tasks: MSCOCO image captioning (Lin et al., 2014; Kasai et al., 2022b), CNN/DailyMail news summarization (Fabbri et al., 2021), and WMT21 Chinese-to-English translation (Barrault et al., 2020; Freitag et al., 2021). Following Kasai et al. (2022a), we measure the cosine similarity between the generated text and each reference text and take the maximum similarity score over all references available (Zhang et al., 2020). We evaluate all embedding models by the Pearson correlation with the human judgments, again following Kasai et al. (2022a). We then report the average correlation scores over the three datasets. Note that we do not use the English-toGerman dataset in Billboard because our models are trained only on English data.

heading: C Full instructions
text: We list all instructions for each dataset in MEDI in Table 7 and Table 8

heading: D Full Results
text: We provide the detailed evaluation scores in MTEB, Billboard and prompt retrieval benchmarks in Table 9 & 10.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: section 7, after conclusion.
7 A2. Did you discuss any potential risks of your work? Models and datasets are all open-sourced and used consistently with their intended use. We do not see potential risks beyond these open-sourced artifacts.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: See the introduction and abstract
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3. We open-source our code. All of our pipeline is based on open-sourced artifacts from prior work that is properly cited.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: See section 3 and the rest of the paper. All papers are properly cited.

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: See section 3. All experiments are based on publicly available datasets and models.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? See section 3. All experiments are consistent.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. We only use publicly available data that come with appropriate documentations.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 3 provides details. We also provide a codebase with full documentations.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. We report experimental settings in Section 2,3 and the appendix.
C 3 Did you run computational experiments? Section 3. See the appendix for more details

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 3 and the appendix for more detail.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 3 and the appendix for more detail

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 3 and the appendix for more detail

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 3 and the appendix for more detail
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? See section 2 for more detail.

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? See section 2 for more detail.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. All the annotations are performed by authors without extra payment.

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? See section 2 for more detail.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
7 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? All the annotations are performed by authors.

heading: Future Work
text: We hope that future work will scale up the number of negatives used during finetuning and investigate various methods for mining hard negatives. Additionally, we do not have enough computation resources to apply multitask instruction finetuning to GTR-XXL (4. 8B parameters), which is also an area for future exploration. At the core of INSTRUCTOR is the instruction design. While our current unified instruction format has demonstrated effectiveness, future research can explore other instructional elements to further improve performance. For example, previous work (Wang et al. , 2022) have shown that incorporating demonstration examples and explanations can be beneficial for instruction-finetuned language models