Task-aware Retrieval with Instructions
Year: 2023

Abstract:
We study the problem of retrieval with instructions, where users provide explicit descriptions of their intent along with their queries to guide a retrieval system. Our solution is a generalpurpose task-aware retrieval system, trained using multi-task instruction tuning and can follow human-written instructions to find relevant documents to a given query. We introduce the first large-scale collection of 37 retrieval datasets with instructions, BERRI, and present TART, a single multi-task retrieval system trained on BERRI with instructions that can adapt to a new task without any parameter updates. TART advances the state of the art on two zero-shot retrieval benchmarks, BEIR and LOTTE, outperforming models up to three times larger. We further introduce a new evaluation setup, X-Retrieval, to better reflect real-world scenarios in which diverse domains and tasks are pooled. TART significantly outperforms competitive baselines in this setup, further highlighting the effectiveness of guiding retrieval with instructions.1

heading: 1 Introduction
text: Information retrieval (IR) is the task of finding relevant documents from a large collection of texts to fulfill a user’s information need, typically expressed in the form of a textual query (Singhal et al., 2001). The notion of relevance from the user’s perspective (i.e., intent) can be amorphous (Mizzaro, 1998), and a query alone may not fully capture user information needs (Ruthven and Lalmas, 2003; Taylor, 1962). As illustrated in Figure 1 (top), given the same query, “implementing batch normalization,” users’ intents can be diverse (e.g., find code snippets or paragraph-length answers).
Most existing work tries to learn those implicit intents from labeled data (e.g., pairs of queries and relevant documents), yielding separate models for
1Code and models are available at https://github.com/ facebookresearch/tart.
different intents, as shown in the bottom left of Figure 1. These approaches usually require a vast number of annotated examples to train a model to capture the task-specific notion of relevance, while they could benefit from the abundance of data available from related tasks. Additionally, having separate models leads to complicated pipelines.
This paper advocates for a new problem formulation, retrieval with instructions (Figure 1 bottom right), to explicitly model a user’s intent by providing a natural language description of the search task (a.k.a. instruction). The goal of retrieval systems is to retrieve documents that are both relevant to the query and well-suited to the instructions (taskaware). Explicitly defining the user intent with natural language instructions provides additional flexibility that enables unifying diverse retrieval tasks during training.
Despite active research in language models (LMs), instruction-following has not been systematically explored in retrieval, partly due to the lack of annotated resources. To facilitate research
3650
in retrieval with instructions, we introduce BERRI (Bank of Explicit RetRieval Instructions), a collection of approximately 40 retrieval datasets with diverse instructions in a unified format, covering 10 diverse domains. Each task has on average 3.5 diverse instructions annotated by experts, following our novel instruction schema for retrieval tasks.
We showcase the benefit of BERRI to train TART (Task-aware ReTriever), a multi-task retrieval system that learns to follow instructions to perform diverse tasks. We employ two widely explored architectures: TART-dual, a dense dual-encoder architecture that retrieves documents based on the similarity of independently encoded query succeeded by instructions and document embeddings; TART-full, a cross-encoder architecture that calculates probabilities of a document being relevant to the query according to the instruction. We train TART leveraging hard negative samples and new instruction-unfollowing negative samples.
The TART models, particularly TART-full yields state-of-the-art results on two popular zero-shot retrieval benchmarks, BEIR (Thakur et al., 2021) and LOTTE-pooled (Santhanam et al., 2022), outperforming systems using three times more parameters (Nogueira et al. 2020; Ni et al. 2021; Muennighoff 2022) as well as task-specific retrievers trained on millions of automatically generated examples (Dai et al., 2022; Wang et al., 2022a).
We further introduce a new evaluation setup, X2-Retrieval (Cross-task Cross-domain Retrieval), where a system needs to handle queries with diverse intents and find relevant documents from a largescale, cross-domain pooled corpus, simulating challenges in real-world retrieval applications. TART outperforms other state-of-the-art methods, demonstrating its effectiveness in this under-explored setting, leveraging explicit textual intents. In summary, our contributions are as follows:
• Retrieval with instructions, a new formulation to model users’ intent explicitly (Section 3.1).
• BERRI, a new collection of about 40 retrieval datasets with instructions (Section 3.3).
• TART, a task-aware retriever trained on BERRI that advances state of the art on zeroshot and cross-task retrieval (Section 4).

heading: 2 Background and Related Work
text: Zero-shot training of retrievers. Recent neural retrievers (Karpukhin et al., 2020; Lee et al., 2019; Khattab and Zaharia, 2020) show their superiority
over term-based retrievers (e.g., BM25; Robertson and Zaragoza 2009) across domains when training data is abundant (Luo et al., 2022; Asai et al., 2021; Petroni et al., 2021). Due to the high annotation cost, improving neural retrievers in zero-shot settings is an active area of study. Pre-training neural retrievers (Izacard et al., 2022) and training a single retriever on large-scale supervised datasets such as MS MARCO (Bajaj et al., 2016) show improvements in transferring to related retrieval tasks (Khattab and Zaharia, 2020; Nogueira et al., 2020; Chen et al., 2022), while they often struggle with tasks unlike those used for training (Dai et al., 2022). To address this, several work (Wang et al., 2022a; Dai et al., 2022) trains customized retrievers for each task using unlabeled corpora, leveraging another model to automatically generate training data (Wang et al., 2022a). It often requires running massive LMs and training separate retrievers, resulting in slow and costly adaptation. Concurrent to our work, Su et al. (2022) trains a single dual-encoder model trained on embedding tasks including retrieval tasks with instructions.
Instruction tuning. Training LMs with instructions or demonstrations on many tasks has proven to be very effective for zero- or few-shot transfer (Wei et al., 2022a; Sanh et al., 2022; Ouyang et al., 2022; Min et al., 2022; Wang et al., 2022b; Mishra et al., 2022; Chung et al., 2022). Yet, such instruction tuning has not been systematically explored in retrieval for several reasons. First, large-scale instruction-annotated datasets (Bach et al., 2022; Wang et al., 2022b) exclude retrieval tasks. Second, instruction-following LMs are encoder-decoder or decoder-only models with tens of billions of parameters, which are difficult to be adapted for retrieval tasks requiring encoding millions of documents. Our work is inspired by this line of work and addresses those challenges.
Retrieval with descriptions. The problem of retrieval with descriptions (e.g., TREC 2004 Robust Track; Voorhees 2005) incorporate querydependent descriptions that describe information needs for query disambiguation (e.g., desirable documents), unlike query-independent instructions in this work. Early work shows that concatenating descriptions only marginally helps (Walker et al., 1998), while Dai and Callan (2019, 2020) suggests that powerful BERT encoders (Devlin et al., 2019) could better incorporate such rich context.
instruction defines intent, domain and unit. The full list of instructions are in Appendix A.5.

heading: 3 Formulation and Data Collection

heading: 3.1 Problem Formulation
text: This work introduces a new problem formulation, retrieval with instructions (Figure 1 bottom right). We are given a large collection of N documents D = {d1, . . . ,dN}, a search task instruction t and a query q. The problem of retrieval with instructions is to find a document d ∈ D that is relevant to q according to the instruction t. Compared to the standard retrieval setting (e.g., Figure 1 bottom left), the difference is the explicit definition of relevance in the instruction t as additional input to the system and a retrieval system needs to be task-aware—changing their relevance measure by attending to the instruction. This new formulation brings both new research challenges and opportunities. For instance, a retriever is now required to modify its search behavior according to the instructions. On the plus side, different datasets can be naturally grouped to train a single retriever, yielding benefits from cross-task interdependence. Instructions provide extra flexibility and enable zero-shot transfer via natural language instructions, unlike training with fixed task tags (Maillard et al., 2021). A single task-aware retriever obviates the need to host multiple task-specific retrievers.
Multi-task training with instructions has not been studied in the area of retrieval due to the lack of resources and dedicated models. To facilitate the research on retrieval with instructions, we introduce BERRI, a large-scale retrieval benchmark with expert-written annotations (Section 3.3) in a unified format (Section 3.2), and subsequently the multitask instruction-following retrievers (Section 4).

heading: 3.2 Unified Task and Instructions Schema
text: Task format. Each task T in BERRI consists of a corpus D, queries Q = {q1, . . . , qK}, and an instruction t, where K is the number of the queries included in the task. An instance of each task in-
cludes a query q, gold (relevant) documents d+, and negative (irrelevant) documents d−. For each task, an explicit intent t is given.
Instruction schema for retrieval. We introduce a novel schema to define an informative instruction for retrieval tasks, which have not been studied in prior instruction-following literature. An instruction that sufficiently describes an arbitrary retrieval task should include: intent, domain and unit. Specifically, intent describes how the retrieved text relates to the query, such as whether the text answers a question in the query or paraphrases it. Domain is the expected source or type of retrieved text, such as Wikipedia or PubMed articles. Unit defines the text block to retrieve, such as a sentence or a paragraph. Table 1 shows examples of instructions, and Appendix A.5 shows the full list.

heading: 3.3 Dataset: BERRI
text: Dataset selection and unification. We manually collect datasets from (1) KILT (Petroni et al., 2021), (2) the Sentence-Transformers Training Data for Text Embedding Models2, and (3) manual searches in ACL anthologies and huggingface datasets3 to cover diverse tasks and domains. Except for a few domains (e.g., Wikipedia) many domains do not have retrieval datasets while there are a few datasets for other NLP tasks that can be cast as retrieval (e.g., sentence paraphrasing). Re-purposing those non-retrieval tasks as retrieval tasks enables the diversity of the domains as well as the instructions in BERRI. From initial collections of more than 60 datasets, we conduct manual dataset inspection and select 37 datasets (Figure 2) covering diverse domains (e.g., Wikipedia, scientific papers) and tasks (e.g., fact verification, dialogue response retrieval, QA). See Appendix A.1 for more details.
2https://huggingface.co/datasets/ sentence-transformers/embedding-training-data
3https://huggingface.co/docs/datasets/index
Negative documents selection. Negative samples are crucial for training retrieval systems (Zhan et al., 2021; Qu et al., 2021). In addition to randomly sampled negative samples (random negative documents), we introduce two types of challenging negative samples: denoised hard negative documents dHD and instruction-unfollowing negative documents dUF. Figure 3 shows examples of gold documents and those negative samples.
For hard negatives dHD we run Contriever (Izacard et al., 2022) and then filter out false negative documents by running an off-the-shelf reranker4 and keeping passages with low scores (smaller than 0.1). We further introduce a new negative sampling strategy, instruction-unfollowing negative samples dUF, to make systems learn to retrieve documents that are well-suited to the instructions. As shown in Figure 3, given an instruction “find an informative dialogue response”, a system should not retrieve a Wikipedia paragraph about armadillos, even though that is highly relevant to the query. To obtain such negative documents, we retrieve documents from a different task’s target corpus using Contriever and consider all those documents to be negatives since they do not satisfy the instruction. Details are in Appendix Section C.3.
4https://huggingface.co/cross-encoder/ ms-marco-MiniLM-L-12-v2

heading: 4 TART: Multi-task Instructed Retriever
text: We now present TART (TAsk-aware ReTriever) trained on BERRI via multi-task instruction-tuning, leveraging our unified task-aware schema.

heading: 4.1 Model Architecture
text: TART-dual. TART-dual adopts a dual-encoder architecture to independently encode queries with instructions and documents. It uses maximum inner product search (MIPS) over the embeddings (Karpukhin et al., 2020). The similarity between a query q and a document d, given an instruction t, is calculated as follows:
s(t, q,d) = E([t; q])TE(d), (1)
where E(·) is the embedding function5 and [t; q] is the concatenation of the instruction and query. For this model, document embeddings can be computed offline, improving inference efficiency at the cost of storage space (Yamada et al., 2021).
TART-full. The dual-encoder architecture is known to be less expressive due to its limited querydocument interactions (Khattab and Zaharia, 2020). To address this issue, we also explore a crossencoder architecture (Nogueira and Cho, 2019),
5We use a shared encoder since having separate encoders gave no additional gains in preliminary experiments.
which computes the relevance between a query and each document by jointly encoding them with cross-attention. A cross-encoder model is often prohibitively expensive to scale up to millions of documents, so we first run a lightweight off-theshelf dual-encoder retriever to retrieve the top documents. For each of these documents, TART-full computes the similarity score as:
s(t, q,d) = FFN(E([t; q;d])), (2)
where FFN represents an additional feed-forward network that predicts whether the document follows the instruction and is related to the query.
We initialize TART-full with encoders of T5based instruction-following pretrained models, namely T0-3B (Sanh et al., 2022) and FLAN-T53B (Chung et al., 2022) for their empirical competitiveness, as found in prior work (Sachan et al., 2022). We follow the EncT5 approach (Liu et al., 2021) and prepended each sequence with a start-ofsequence token. The token representation is then fed to a newly initialized feed-forward network. Unlike MonoT5 (Nogueira et al., 2020), we use their encoders only for parameter efficiency, reducing the number of the parameters to half.

heading: 4.2 Training TART
text: We train TART-dual and TART-full using the positive documents and three types of negative documents in BERRI with instructions (Figure 3). Training TART-dual. We train TART-dual using annotated positive and negative documents in BERRI as well as in-batch negatives as follows:
L = − log e s(t,q,d+)
∑ d∈B e s(t,q,d) ,
where B denotes all documents in the same minibatch (Karpukhin et al., 2020). Training TART-full. Following prior work (Nogueira and Cho, 2019), TART-full is trained with the cross entropy loss as: L = − ∑
d∈d+ log s(t, q,d)−
∑
d∈d− log(1−s(t, q,d)).
Knowledge distillation from TART-full to TARTdual. The default hard negatives in BERRI rely on off-the-shelf models fine-tuned on MS MARCO; for some domains, the hard negative samples mined by those models can be less reliable. For a smaller dual-encoder model, those false positive and negative samples can diminish performance (Qu et al.,
2021). We apply hard knowledge distillation with TART-full (Qu et al., 2021). We first train TARTfull on the annotated gold documents and the negative documents in BERRI, and then update hard negative documents and positive documents as in Section 3.3 with TART-full, with instructions.

heading: 5 Experiments
text: We evaluate TART on zero-shot retrieval (Section 5.1) and our new more challenging evaluation setup, X2-Retrieval (Section 5.2).

heading: 5.1 Zero-shot Retrieval Evaluations
text: We run experiments on two popular zero-shot retrieval benchmarks: BEIR (Thakur et al., 2021) and LOTTE (Santhanam et al., 2022). None of the evaluation datasets overlap with BERRI.
BEIR is a collection of diverse retrieval tasks in multiple domains where the retrieval target is restricted to the target corpus in a single domain. We used publicly available datasets.6 LOTTE-Search samples GooAQ (Khashabi et al., 2021) questions whose answers come from certain forums in StackExchange. We evaluate our model in the pooled setup, where documents come from forums in diverse domains (e.g., cooking, technical). GooAQ is not included in our training set. In LOTTE, our instructions specify which forum our system should retrieve evidence from (e.g., “Retrieve a cooking StackExchange forum post”). Metrics. Following Thakur et al. (2021), for BEIR, we use NDCG@10 as our primary metric on BEIR. For LOTTE-pooled, we use Success@5 (= Recall@5) as our primary metric, as in the original paper (Santhanam et al., 2022).
5.2 X2-Retrieval Evaluation Users’ intents can be diverse, requiring searching in an open-domain environment (Piktus et al., 2021), which is currently under-explored. We introduce a more realistic evaluation setup, X2-Retrieval (Cross-task Cross-domain Retrieval), where several retrieval tasks with different intents are pooled to form a single retrieval target containing diverse documents. This requires a system not only to adapt to a new task in a zero-shot manner but also to model users’ intents expressed in natural languages to meet their information needs.
6Following Dai et al. (2022), we exclude Natural Questions, MS MARCO, HotpotQA, FEVER, and CQADupStack from our evaluation targets for fair comparison since they are included either in encoders’ pretraining or in BERRI.
Tasks and queries. Our X2-Retrieval evaluation covers six datasets across three domains, namely, Wikipedia, Science, and Technical (Table 2) domains. The key challenge here includes datasets with different search intents that may not always be obvious from the queries alone.
A pooled corpus. For the primary pooled setup, we combine all documents from different tasks and the BEIR NQ Wikipedia corpus to form a single retrieval corpus, consisting of approximately 3.7 million documents. We also report the simplified closed setup performance as an oracle setup, where a system retrieves only from the original corpus.
Metrics. We report NDCG@10 on both pooled and closed setups for each task. In addition, we evaluate the performance gap between the closed and pooled setups and refer to it as robustness. A smaller gap means that the model is distracted less by the documents from undesirable corpora.

heading: 5.3 Baselines
text: We compare TART with various state-of-the-art methods. The first group is unsupervised models that are not trained or trained on unlabeled text; these include Contriever (Izacard et al., 2022) and BM25. We also compare TART with UPR (Sachan et al., 2022), which reranks the Contriever results using a pretrained T0-3B. The second group trains retrievers and rerankers on MS MARCO or a few large-scale datasets and directly transfers them to new tasks with no adaptations, including MonoT5 (Nogueira et al., 2020), Contriever-MS MARCO and ContrieverMS MARCO + Cross Encoder (CE), ColBERT v2 (Santhanam et al., 2022), and SGPT6.8B (Muennighoff, 2022). The final group of models is specialized retrievers trained for each task on automatically generated task data. Promptagator (Dai et al., 2022) generates large amount of in-domain data using FLAN (Wei et al., 2022a),
and GPL (Wang et al., 2022a) generates them using DocT5Query (Nogueira et al., 2019). We also compare TART with their counterparts trained on BERRI and evaluated without instructions, TARTdual w/o I and TART-full w/o I.

heading: 5.4 Experimental Settings
text: We initialize TART-full from the T0-3B (Sanh et al., 2022) and FLAN-T5 encoder (Chung et al., 2022). We sample positive and negative passages with a 1:4 ratio. We initialize TART-dual from ContrieverMS MARCO (Izacard et al., 2022), which is based on BERT-base.7 Per-GPU batch size is 16, and for each positive document, we sample in total 5 negative passages, where 90% of them are randomly sampled from D, and 10% are sampled from dHD and dUF. We use top 100 Contriever-MS MARCO results as the TART-full initial candidates.8 Table 9 shows instructions for evaluations. More details are in Appendix C.1.

heading: 6 Results and Analysis

heading: 6.1 Results
text: Zero-shot evaluation. As shown in Table 3, TART-full and TART-dual largely outperform their counterparts trained and tested without instructions, demonstrating the effectiveness of instructiontuning for better zero-shot retrieval. TART-full significantly outperforms larger models and customized models trained on millions of synthetically generated in-domain data, advancing the state of the art on BEIR and LOTTE. Unlike prior methods that require additional data generation, TART only requires a single human-written instruction to
7We also tried larger models such as SGPT-1.3B (Muennighoff, 2022), but observed large performance drop on some datasets, resulting in lower average than Contriever-based TART-dual. Therefore, we use Contriever-based TART-dual.
8We found that combining TART-full with the original Contreiver performs better than combining TART-full with TART-dual, possibly because TART-full uses the hard negative samples retrieved by Contriever’s top-retrieved results.
adapt to a new task. Compared to other methods using cross-encoder-based reranking models (e.g., BM25+MonoT5), TART-full uses a much smaller number of paragraphs to be re-ranked, which significantly reduces latency caused by reranking at test time. The large performance gain from Contriever (MS) to TART-dual on six out of the nine BEIR tasks (e.g., SciFact, Arguana) shows the effectiveness of instructions and knowledge distillations. However, for the other three datasets (e.g., Touche2020), TART-dual shows large performance deterioration. We hypothesize that model capacity (i.e., BERT-base) and limited interactions between the query and document embeddings could be major bottlenecks. Prior work on instruction training in large LMs has shown that smaller models often do not get as much benefit as larger ones from instructions and increasing dataset size, possibly due to their limited model capacities (Chung et al., 2022). Su et al. (2022) also observe more significant gain from instruction tuning when they use larger encoder models (i.e., GTR-base v.s. GTR-XL), reporting performance deterioration in retrieval tasks when they instruction tune 335 million parameter base model. Future work can investigate efficient architectures that enable more rich interaction between queries with instructions and documents. X2-Retrieval evaluation. Table 4 shows the models’ X2-Retrieval performance. Contriever and Contriever+CE show competitive closed performance in the closed setup, as in BEIR, but they struggle in the pooled setup due to their inability to handle human instructions. Especially Contriever+CE shows a large performance drop on AmbigQA-pooled by retrieving documents instead of queries due to the biases from fine-tuning on a
QA dataset (i.e., MS MARCO) only. TART-full shows the best-closed performance and pooled performance, indicating its strong zeroshot adaptation and cross-task abilities. We found that a model can flexibly change its behavior based on the instructions, as shown in Table 11. TARTdual shows strong performance on the pooled setup, indicating that smaller models can be also guided by explicit instructions.

heading: 6.2 Analysis
text: Ablating instructions. We compare TART-full with three variants: (a) train without instructions, test with instructions prepends instructions at test time only to test if the models just exploit keyword matching only at test time; (b) train with instructions, test without instructions uses TART-full without instructions at test time; (c) train without instructions, test without instructions does not use instructions at all during training and test time. Figure 4 shows the performance of those baselines. On all benchmarks, ablating instructions during training or test time causes a notable performance drop. We also see that a model trained with instructions but given no instruction at test time still yields a few performance improvements over the model trained completely without instructions, indicating the effectiveness of multi-task instruction tuning.
Robustness toward instructions. Figure 5 shows the performance variance given multiple different instructions. Instructions significantly improve model performance without instructions (the blue circles). Although different instructions give small performance variance, TART often outperforms other baselines when informative instructions are given. See Table 15 for individual instructions.
Dataset scale. Following prior work on instruction tuning for LMs (Wang et al., 2022b; Wei et al., 2022a), we conduct dataset ablation, where we reduce the number of training datasets. Figure 6a shows the average BEIR performance of TART-full
trained on randomly sampled 5, 10, and 20 datasets. Increasing the number of the training datasets helps TART to perform better. In addition to domain and task diversity, the diversity of instructions observed during training may also improve performance, as in Appendix Section E.3.
Effects of negative sampling. We analyze the effectiveness of negative samples by ablating them during training. Figure 7 shows the performance of the models trained without negative samples on BEIR and X2-Retrieval. Adding more challenging negative documents (i.e., dHD and dUF) during training largely improves the model performance on BEIR. Moreover, the model trained without
instruction-following samples (w/o dUF) results in lower X2-Retrieval performance, although this model performs on par with the original TART-full on BEIR. This indicates that our new instructionunfollowing negative documents largely contribute to improving the ability to distinguish instructions and are thus crucial to build a robust task-aware retrieval system.
Model scale. We test different TART-full sizes to see how model scale affects final performance. Prior work has shown that scaling up re-ranking models often improves reranking performance (Rosa et al., 2022), and models’ instruction-following abilities improve as models get larger (Wang et al., 2022b; Sanh et al., 2022; Wei et al., 2022b). We investigate how model scale affects the ability to generalize to new tasks and follow instructions. For a fair comparison, we train TART-full using different T5 LM-Adapt (base, large, and XL) and evaluate performance using them to rerank the top 100 Contriever results. Figure 6b shows TART-full’s average performance across different model scales. We observe clear performance improvements by increasing model size as observed in prior work on large LM.

heading: 7 Conclusion
text: This paper lays the foundation for building a general-purpose task-aware retriever that can follow natural language instructions. We introduced a new setup, retrieval with instructions, to model users’ intents explicitly. We presented BERRI, the first large-scale retrieval dataset with expert-written annotations. Building upon BERRI, we trained the first instruction-following retrieval system by massive multi-task instruction-tuning, TART advances the state of the art on two zero-shot retrieval benchmarks BEIR and LOTTE as well as on our newly introduced challenging evaluation setup.
Limitations
Although our TART-full model shows the effectiveness of instruction-tuning for retrieval, on some datasets TART-dual shows large performance degradation from its non-instruction-following counterpart. We hypothesize that a smaller model size (i.e., 110 million parameters) and limited interactions between query and document embeddings are the main factors. We conduct primarily experiments training larger dual-encoder models such as SGPT (Muennighoff, 2022) on BERRI but
still observe some notable performance drop on some datasets, which indicate only scaling up encoders may not significantly improve instructionfollowing retrieval systems. Future work can study the better approach to train larger-scale dualencoder models as well as explore modeling architectures that enable rich interactions but are still more efficient than the cross-encoder, such as ColBERT-v2 (Santhanam et al., 2022).
Retrieval tasks are excluded in prior work on instruction-following of LLMs. This work is the first to explore instruction tuning in the area of retrieval, and we annotate more than 100 instructions for approximately 40 tasks, and we demonstrate the effectiveness of the dataset scale in retrieval. Yet, recent work (Wang et al., 2022b; Chung et al., 2022) show that scaling up the number of the training datasets improves LLMs’ ability to adapt to new task via instructions, and the current dataset scale might not be optimal. We open-source our instruction data and call for community efforts to collect more retrieval tasks and human-written instructions as in instruction-following for LMs (Wang et al., 2022b; Bach et al., 2022), to investigate whether further increasing the number of the datasets lead to improvements.
Ethical Considerations Although instruction-tuning using many datasets enable better zero-shot transfer, TART does not always retrieve documents that perfectly align with users’ expectations. Applying TART to safetycritical domains requires extra attention. BERRI includes approximately 40 tasks covering diverse domains. Although the data has been automatically filtered, and we have examined the data, there may still be harmful or privacy-sensitive contents. We will release all of the data and preprocessing scripts for follow-up work to inspect those dataset issues and the effects of those data.

heading: Acknowledgements
text: We thank Allen School NLP and Meta AI researchers for their insightful discussions and Jeff Dalton, Mike Lewis, Sheng-Chieh Lin, Sandy Kaplan, and Yizhong Wang for their helpful feedback on this paper and discussions.

heading: A Further BERRI Details
text: A.1 Detailed Dataset Creation Process
Manual dataset selection. From an initial list of more than 60 datasets, we assess whether it is suitable for repurposing as a retrieval task. Specifically, we sample 20 instances from the candidate dataset and check if the queries are self-contained.9 If the majority of queries fail this test, we exclude the corresponding dataset. Consequently, we use 37 datasets, including more than 5 million instances in total. For datasets that are orders of magnitude larger than other datasets (e.g., PAQ; Lewis et al. 2021), we randomly sample up to 300k instances, except for MS MARCO.10 As a result, BERRI covers diverse domains (e.g., Wikipedia, scientific papers) and tasks (e.g., fact verification, dialogue response retrieval, QA). See Appendix A.3 for more details.
Unification and instruction annotations. For retrieval datasets such as MS MARCO, we use the annotated gold documents as positive documents d+ to a given query q. Regarding non-retrieval tasks, we use the original input sequence as a query q and the original output or given context as d+. For instance, given a summarization dataset we use a source text and a summary as a query and a gold document, respectively. More details about the dataset unification are available in Section A.2.
For datasets without preprocessed retrieval targets,11 we gather all positive and negative documents provided by the original dataset to build a single task-specific retrieval corpus D.
A.2 Details of Dataset Unification
As shown in Table 5, some datasets were not originally retrieval datasets (e.g., summarization datasets). We describe how we convert these into the unified retrieval task format.
QA. For QA datasets, where each instance consists of a query, a gold context, and answers, we assume the original gold context is the gold document used as a positive sample during training.
9For examples, finding a corresponding review text for the review title “I love this!” is under-specified.
10Prior work has shown that MS MARCO can be beneficial to many downstream retrieval tasks (Izacard et al., 2022).
11For example, KILT datasets such as FEVER or NQ use the unified Wikipedia corpus.
For some exceptional datasets, we performed additional preprocessing. We found that ReCoRD instances are occasionally self-containing due to the nature of the cloze-style QA; therefore, for ReCoRD, we replace the original placeholder with the gold answer and use this original question with the answer as the query and the original context as a gold document. For MedMCQA, we use the source exam question as the query and the answer evidence as the positive document.
Summarization. For summarization datasets, we use target summarizations as the gold document and source text as the query.
Text simplifications. For text simplification datasets, we use source (often more complex) sentences as the query and simplified sentences as the gold document.
Code search. We use the source comment as the query and the corresponding implication as the gold document. We exclude the python subset from BERRI as we use it for X2-Retrieval.
A.3 BERRI Statistics
We conduct analyses on BERRI to understand its domain and intent diversities.
Intents. Open-ended intents are diverse and hard to classify into fixed sets of categories. As a proxy
for intents, Figure 8 shows the distributions of the source task categories. QA is the most representative category, while summarization and question duplication detection is also common due to their abundance in large-scale datasets. On the other hand, around 50 % of the tasks do not belong to those top three categories, such as code search or caption generations, which contribute to the diversity of BERRI. We also find that traditional nonretrieval tasks, such as sentence simplification or dialogue, can be repurposed as retrieval tasks.
Domains. Our dataset covers diverse domains. Figure 9 shows that Wikipedia (e.g., NQ), web (e.g., MS MARCO), Community QA (e.g., Quora), News(e.g., CNN/Daily) dominate, while we also have some expert domains (e.g., medical, legal, technical). We found that although many expert domain datasets are smaller than the ones in general domains like Wikipedia, adding those high-quality expert domain datasets helps the system learn to adapt to those domains or unseen expert domains with a similar writing style (e.g., scientific papers).
A.4 Dataset List
Table 5 shows all datasets we used in BERRI. Table 6 provides references for these datasets.
A.5 Instructions for BERRI
Table 7 shows the full list of the instructions in BERRI. Note that we present only one instruction for each dataset. A full list of the instructions will be released in our repository.
B Further Detail about the X2-Retrieval
Query and corpus creations. For AmbigQA, we use the official development split, including 1,172 queries, as the official test split annotations are not publicly available. We use all paraphrased questions for all train and development sets to form the retrieval corpus. For WIKIQA, we combine the development split and test split available at the huggingface datasets,13 and we use the question and answer sentence pairs that are labeled as 1 as the queries for evaluations, and use the answer sentences as the gold documents. Regarding the retrieval target, we use all sentences available in the WIKIQA dataset, including the sentences that are labeled as 0. For LinkSO, we use the original datasets’ test split for the python domain and
13https://huggingface.co/datasets/wiki_qa
sample 1,000 queries.14 We find questions that are labeled as duplicated and use their corpus as our retrieval target. For GooAQ-technical, we sample 1,000 GooAQ questions whose answers are from stackoverflow.com. As 20% of the sampled GooAQ tech queries share the same answer posts, we remove the duplicated paragraphs. For CodeSearchNet-Python, we use the comments describing the codes as queries and the corresponding python codes as positive documents. We sample 1,000 queries from the test split.
Examples. Examples of X2-Retrieval are shown in Table 8. As shown, queries themselves often do not fully indicate the users’ intents. By specifying users’ intents as explicit textual instructions, our model can effectively perform multi-task retrieval over a single pooled corpus.
Human evaluations of quality. To access the possibility of having false negative passages, we run an off-the-shelf retrieval system to retrieve the top 10 documents for randomly sampled 20 questions for each task, and we evaluate if any of the negative passages, especially from the non-target corpus, are indeed positive. We found that the false negative ratio is less than 10%.

heading: C Modeling Details
text: C.1 Hyperparameters of TART
TART-dual. We set the learning rate to be 1 × 10−5 and warm-up steps to be 1,000. The softmax temperature is set to 0.05. The batch size is 1024. We use 7 negative samples per instance; 10% of the time we use hard negative or instructionunfollowing negatives, while 90% of the time we use negative documents that are randomly sampled from the same target corpus. The maximum document chunk length is set to 256.
TART-full. To train a cross-encoder using the T03B encoder, we set the maximum sequence length to 512 and the batch size to 1, increasing the gradient accumulation steps to 8. We set the dropout rate to 0.1 and the learning rate to 1 ×10−5.
C.2 Instructions for Evaluations
Table 9 lists the instructions used for the BEIR and X2-Retrieval evaluation.
14https://sites.google.com/view/linkso
C.3 Negative Sampling Mining hard negatives. To mine hard negative documents for BERRI, we retrieve top documents
from the target corpus using Contriever (Izacard et al., 2022) and then add new documents whose normalized scores predicted by a cross-encoder model, ms-marco-MiniLM-L-12-v215 are below 0.1 as hard negative documents.
Mining instruction-unfollowing samples. To sample instruction-unfollowing samples, given a query from a target dataset, we retrieve the top 20 documents from another task’s corpus using
15https://huggingface.co/cross-encoder/ ms-marco-MiniLM-L-12-v2
Contriever-MS MARCO. For instance, given a PubMedQA, a system should not retrieve a document from a Wikipedia paragraph. A list of source target task and retrieval corpus combinations is shown in Table 10.
Sampling d− for TART-full training. Challenging negative samples help a system to effectively learn the task. On the other hand, prior work also shows that it can lead to large performance drops in out-of-domain datasets, and having both randomly sampled negative documents and carefully
designed negative documents is a key to building a system that is competitive in both in-domain and out-of-domain retrieval (Ni et al., 2021). To effectively combine the negative documents during training, we first combine random samples and hard negative samples, and then we randomly sample 4 negative documents per one positive document. The number of instruction-unfollowing documents, if applicable, is limited to less than 20% of the negative documents, and we set the maximum number of instruction-unfollowing samples from certain combinations listed in Table 10 up to 10k.

heading: D More Experimental Details
text: in addition to the in-batch negative documents. We use 8 GPUs to train TART-full and 64 GPUs to train TART-dual. We train TART-full up to 10k steps and TART-dual up to 30k steps and take the checkpoint with the best development performance. We use 64 GPUs to train TART-dual and 8 GPUs to train TART-full.

heading: E Further Results and Analyses
text: E.1 Qualitative Results on X2-Retrieval
Table 11 shows the qualitative examples given different instructions on X2-Retrieval, and Table 12 compares TART-full with Contriever MS MARCO.
E.2 Analysis of Instruction Effectiveness
Full results of instruction ablations. Table 13 shows the full BEIR results of ablating instructions and Table 14 shows the ones on LOTTE and X2Retrieval. On all of the benchmarks, removing instructions at training or test time largely hurts the performance, indicating the effectiveness of instructions.
Examples of prompts with performance. Table 15 shows the instructions and TART-full performance on three BEIR datasets. We also provide a comparison of the model performance when uninformative instructions are given in Table 16. We see that more informative and related instructions often result in a strong performance, while irrelevant instructions degrade it.
E.3 Analysis on Model and Dataset Scale
Task diversity. As shown in Figure 10, task diversity is a key to improving models’ zero-shot transfer performance. QA only struggles on Arguana, where the tasks significantly differ from QA.
Domain diversity. Figure 10 shows that having more diversity in training datasets’ domains is also crucial, especially when the target datasets are in non-general domains. For instance, a model trained only on Wikipedia datasets struggles on Touche2020 or SciFact, where documents come from argument websites and scientific papers, respectively.
Per-dataset performance breakdown. Table 17 shows the NDCG@10 across different model scales. We compare the TART-full initialized with
different sizes of T5-LM-adapt for a fair comparison. We see in general that larger models perform better.
Table 18 shows the full BEIR results of TARTfull trained on varying numbers of datasets. We see that as we increase the number of datasets used during training, model performance often improves, which is consistent with previous work on instruction-tuning in LLMs (Wang et al., 2022b).
E.4 Analysis on Different Pre-trained Models
Our TART-full is initialized with the T0-3B encoder. We experiment with more recent pretrained instruction-following models: FLAN-T5XL (Chung et al., 2022) and Tk-Instruct (Wang et al., 2022b), which are trained on the order of magnitude of more datasets. We analyze TARTfull performance when we initialize encoders using different pre-trained encoder models, including the ones that are released recently. Table 19 shows the results of TART-full, when the encoder is initialized with three different recent instructionfollowing pretrained models, T0-3B, FLAN-T5XL (Chung et al., 2022) and Tk-Instruct-3B (Wang et al., 2022b). FLAN-T5 shows the best average BEIR performance, outperforming TART-full
by 0.7 NDCG@10. Tk-Instruct shows a notable performance drop on some datasets (e.g., TREC COVID), resulting in slightly lower performance than the original TART-full (T0-3B).
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: The limitation section.

heading: 3 A2. Did you discuss any potential risks of your work?
text: The ethical consideration section.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: The Abstract and Introduction.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3.3

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Sections A.1 Table 6.

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Sections A.1 and 3.3.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Sections A.1 and 3.3.

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Sections A.1 and 3.3.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Sections 3.3 and A.5.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Sections 3.3 and A.5.
C 3 Did you run computational experiments? Sections 4 and 5.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Sections 4 and 5.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Sections 4, 5, C, and D.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Not applicable. Left blank.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 3.3.

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Section 3.3.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
7 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? The annotators of the instructions are the authors of the papers, so we cannot disclose our basic demographics due to the risk of anonymity violations.

heading: Future Work
text: Future work can study the better approach to train larger-scale dualencoder models as well as explore modeling architectures that enable rich interactions but are still more efficient than the cross-encoder, such as ColBERT-v2 (Santhanam et al. , 2022). Retrieval tasks are excluded in prior work on instruction-following of LLMs. This work is the first to explore instruction tuning in the area of retrieval, and we annotate more than 100 instructions for approximately 40 tasks, and we demonstrate the effectiveness of the dataset scale in retrieval. Yet, recent work (Wang et al. , 2022b; Chung et al. , 2022) show that scaling up the number of the training datasets improves LLMs’ ability to adapt to new task via instructions, and the current dataset scale might not be optimal. We open-source our instruction data and call for community efforts to collect more retrieval tasks and human-written instructions as in instruction-following for LMs (Wang et al. , 2022b; Bach et al. , 2022), to investigate whether further increasing the number of the datasets lead to improvements. Ethical Considerations Although instruction-tuning using many datasets enable better zero-shot transfer, TART does not always retrieve documents that perfectly align with users’ expectations. Applying TART to safetycritical domains requires extra attention. BERRI includes approximately 40 tasks covering diverse domains. Although the data has been automatically filtered, and we have examined the data, there may still be harmful or privacy-sensitive contents. We will release all of the data and preprocessing scripts for follow-up work to inspect those dataset issues and the effects of those data