Data-Efficient Finetuning Using Cross-Task Nearest Neighbors
Authors: Hamish Ivison
Noah A. Smith
Hannaneh Hajishirzi
Pradeep Dasigi
Venue: Submitted to ICLR 2023 (2023)
Keywords: multitasking
retrieval
few-shot
efficiency
nlp

Abstract: Language models trained on massive prompted multitask datasets like T0 (Sanh et al., 2021) or FLAN (Wei et al., 2021) can generalize to tasks unseen during training. We show that training on a carefully chosen subset of instances can outperform training on all available data on a variety of datasets. We assume access to a small number (250-1000) of unlabeled target task instances, select their nearest neighbors from a pool of multitask data, and use the retrieved data to train target task specific models. Our method is more data-efficient than training a single multitask model, while still outperforming it by large margins. We evaluate across a diverse set of tasks not in the multitask pool we retrieve from, including those used to evaluate T0 and in addition, more complex tasks including legal and scientific document QA. We retrieve small subsets of P3 (the collection of prompted datasets from which T0’s training data was sampled) and finetune T5 models that outperform the 3-billion parameter variant of T0 (T0-3B) by 8-30% on 11 out of 12 evaluation datasets while using at most 2% of the data used to train T0-3B. These models also provide a better initialization than T0-3B for few-shot finetuning on target-task data, as shown by a 3-23% relative improvement over few-shot finetuned T0-3B models on 8 datasets.

Under review as a conference paper at ICLR 2023
DATA-EFFICIENT FINETUNING USING CROSS-TASK NEAREST NEIGHBORS
Anonymous authors Paper under double-blind review
ABSTRACT
Language models trained on massive prompted multitask datasets like TO (Sanh) et al. 2021) or FLAN (Wei et al. (0211) can generalize to tasks unseen during training. We show that training on a carefully chosen subset of instances can out- perform training on all available data on a variety of datasets. We assume access to a small number (250- 1000) of unlabeled target task instances, select their near- est neighbors from a pool of multitask data, and use the retrieved data to train target task specific models. Our method is more data-efficient than training a sin- gle multitask model, while still outperforming it by large margins. We evaluate across a diverse set of tasks not in the multitask pool we retrieve from, including those used to evaluate TO and additional complex tasks including legal and scien- tific document QA. We retrieve small subsets of P3 (the collection of prompted datasets from which TO's training data was sampled) and finetune T5 models that outperform the 3- 3-billion parameter variant of TO (TO-3B) by 3-30% on 12 out of 14 evaluation datasets while using most 2% of the data used to train TO-3B. These models also provide a better initialization than TO-3B for few-shot finetun- ing on target-task data, as shown by a 2-23% relative improvement over few-shot finetuned TO-3B models on 8 datasets.
INTRODUCTION
Finetuning large models with data from a diverse set of tasks, augmented to include brief descrip- tions of the tasks (i.e.. prompts) has been shown to help models generalize to unseen tasks (Wei ett al., [221aa] Sanh et al., 2021). This cross-task generalization capability is particularly helpful in expensive to collect labeled target task training sets. Prior work trained single
cases where models with as much prompted data as possible for example, Sanh et al. (202T) train model on roughly 11 million instances (counting different prompt variations). The training datasets were selected without using any information about the target tasks with the goal of allowing models to generalize to new tasks from instructions alone, making the evaluation "zero-shot". However, is unclear if all the training data is required for doing well any given target task. Furthermore, given that neural network models have previously been shown to suffer from negative interference (where certain downstream tasks) in multitask setups (Aribandi et al.] (022) and benefit from pretraining on domain-relevant data (Gururangan al.) 22200] Phang al.) 2018), it is possible that training only on relevant prompted data could further improve task generalization.
in training on more datasets results in worse performance
Based on this hypothesis, we seek find small subsets of relevant training data in the massive pool of multitask data that cause the models to generalize better to a given target task than the rest of the pool. Manually finding relevant training data a massive pool of data is infeasible since is not obvious which of the source tasks are relevant for a given target task, and which instances are most relevant for target task generalization within a source task dataset (see Section 5. Hence we rely on a simple method to automatically select these subsets. Additionally, as only some samples within a given dataset may be relevant to a target task, we select per-instance rather than per-dataset, unlike prior work, which tries to identify useful datasets for transfer learning (Aribandi al.) 2022; Phang et al. 2018) and train on all data within the chosen datasets. We use a setup similar to contemporary work examining retrieval-augee augmented cross-task generalization (Lin et al. 2022): we assume access to a small number of unlabeled target task instances and use these to retrieve cross-task nearest neighbors labeled instances from the massive pool of data most similar
unlabeled target

---

Under review as a conference paper at ICLR 2023
Pool of Prompted Tasks
Pretrained LM
TREC
Keep only instances most similar to larget lasks
Cosré
Full finetuning on selected data
Ropes
DuoRC
QuoRef
MRPC
SaQ
DEFT
Adversarial
QA
Optional few-shot finetuning
WikiHop
Unlabelled Target Task
DEFT-Few
Instances
(e.g., DROP questions)
Figure 1: Overview of the DEFT method. Given some unlabeled target-task instances, find the most similar instances in a large pool of multitask data. We train a model on these instances. If we have access labeled data, optionally few-shot finetune the DEFT model.
task instances. The similarity is computed as the distance between the representations produced by the encoder of pretrained seq2seq model. Unlike prior work, we then finetune target task specific models on these neighbors alone, without using any target task specific labeled data any extra data from the pool of multitask data. We hope that the similarity between the cross-task neighbors and our target task data will enable greater generalization to our target task, with dissimilar examples that may cause interference removed from the training mixture. We also aim produce models that perform at least as well as models trained on the entire multitask pool despite being trained on a fraction of data, greatly reducing the cost of training.
We run experiments with T5 (Raffel et al. 2020) models, and use Public Pool of Prompts (P3) (Sanh) et al. 2021 the pool of prompted multitask data from which to retrieve cross-task nearest neigh- bors. We evaluate on the datasets originally used to evaluate TO (a collection of natural language understanding and commonsense tasks), well as 3 additional datasets with varied domains (e.g., legal, NLP domains). Our findings are as follows:
For all the target tasks, we find that their cross-task nearest neighbors in P3 are much more relevant as training data than the rest of the ppol-training T5 models, sometimes even variants smaller than TO-3B, on these subsets yields models with performance 3-30% better than TO-3B evaluated zero-shot across 12 out of datasets.
For some target tasks which TO-3B performs close to random chance, T5 models of the same size trained using cross-task nearest neighbors perform significantly above chance, confirming our hypothesis that massive multitask prompted training could lead negative interference between tasks.
When target task labeled data available for few-shot finetuning, we find that T5 models trained with cross-task nearest neighbors provide better initialization for parameter-efficient finetuning methods than TO-3B, performing 2-23% better than TO-3B with few-shot fine- tuning across 10 out of 11 datasets.
An analysis of what relevant data gets retrieved shows that most of the tasks in the massive pool of multitask data are not retrieved for any target tasks, confirming our hypothesis that only a small subset of data within the pool relevant to any given target task.
These findings suggest that training these models on all the available multitask prompted data re- sults in negative interference, even in relatively large (3 billion parameter) models. Furthermore,
2

---

Under review as a conference paper at ICLR 2023
they show that training large models on all available multitask data is not optimal, with higher per- formance possible through cetter-curated and much smaller training sets.
RELATED WORK
Multi-task Transfer Models Training on large multi-task mixtures is a common trend within NLP, with most existing approaches first training pretrained language model on a large collection of tasks, and then evaluating these models in either zero or few-shot settings on a collection of held-
out datasets (Wei et al.) 2021a [anh al., 2021 Khashabi et al. 2020 Mishra al. 2021 Aribandi
iet al., 2022). Most approaches do not customise their task selection to downstream tasks and assume no knowledge of the target tasks ahead of time, instead focusing on building a single model most applicable to any arbitrary evaluation task. In contrast, we show that if assume access to unlabeled target task instances, we can make much better use of the multitask data, selecting only instances useful to a given task. Relatedly, Vu et al. (2020) propose method for using gradients from labelled task data to construct task embeddings for predicting task transferability. Our method instead uses unlabelled data, which is much cheaper and easier to collect, and does not use gradients. making it much easier to scale to large models such as T5-XL.
Retrieval-based methods for NLP Adding retrieval components to language models has been shown (Khandelwal et al., 2019 Gau al. 2020, Lewis et al. (020) to augment their general- ization capabilities by externalizing memorization. In contrast to prior work in this direction that mostly focused on language modeling as the end task, we evaluate on a variety of language un- derstanding tasks. The work from Shi et al. (2022) used retrieval-based methods for classification tasks by heuristically mapping the label space of the end-tasks to that of the predicted next words of the nearest neighbors from language model. We instead finetune the models the nearest neighbors. Lin et al.] (2022) also use unlabelled examples to retrieve relevant data for improving performance, but focus on farther augmenting malti-task models, and limit themselves to small query sets and small retrieval sets. The authors claim that using a multi-task model is crucial for good retrieval performance. In contrast, we assume no access multi-task models, showing that using non- salltttssk-triinee models still work well for retrieval, and focus on pruning the amount of multi-task data used during training. Additionally, we use larger query sets (up to 1000 unlabelled examples) and retrieve thousands of examples (as opposed to hundreds).
Parameter-efficient fine-tuning methods In contrast work that focused finetuning fewer parameters in large models to adapt them to new tasks (Houlsby et al.) 2019: Hu et al. [021] Liu al., 2022), our proposal is a data-efficient pretraining method for obtaining task-specific models without using any target task labels. Our method is complementary to parameter-effiieen methods, and they can both be used in conjunction, as shown in section 4.3
Instance attribution Our approach works by identifying the most relevant training examples for given data point, which is called instance atrribution. Prior work (Koh & Liang. 2017: Yeh al.) 2018 Pruthi et al. 2020) Han & T'svekkov] (022) used instance attribution methods to interpret pre- dictions of neural network models. These methods generally relied on the gradients of the model to identify the effect specific data points, either in pretraining or finetuning stage, have on the model's predictions. Our method for identifying cross-task neighbors is simpler because we do not use gra- dients and we do not even rely on the labels of the data. Results from Pezeshkpour et al. (2021) show that instance attribution based on similarity between the model's representations are comparable to gradient-based approaches in terms of finding the most important training data points.
3 DATA EFFICIENT FINETUNING ACROSS MULTIPLE TASKS
Given a large collection of labeled prompted data (i.e., data converted into a text-to-text form, with task instructions included in the plain-text input, e.g., P3), our core hypothesis is that some tasks in this massive pool of data are more similar to given target task than others. Given target task, assume we have access to a small amount of unlabeled target task data, which is often much easier and cheaper to collect than labeled data (see section 5.2 Our aim is to find relevant subset of data
3

---

Under review as a conference paper at ICLR 2023
from our pool given single target task, ideally allowing us to train model using this subset that outperforms a similar model trained on the entire pool of data.
Manually identifying the relevant subsets of these datasets is not feasible since task boundaries are usually not clearly defined in NLP, and it is hard to interpret what skills get transferred when model is trained one dataset and evaluated other. Hence, we use similarity between the pretrained model's 's representations to compute relevance. We encode all instances in the large pool of multitask data with pretrained language model (e.g., T5), and build search index over the resulting representations. Given small amounts of unlabeled target task data, we retrieve relevant multitask subsets from the index, which we call cross-task nearest neighbors of the target tasks. We then build task task-specific models by finetuning the pretrained models on the cross-tase tasks neighbors. We refer to this approach as Data-Efficient FineTunning or DEFT for short.
We evaluate our approach both in cases where no labeled data is available, and when few (20-70) annotated labels are available. In the former case, we simply use the unlabeled data for retrieval and evaluate the resulting DEFT model 'zero-shot' on the target task. In the latter case, we first train DEFT model and then perform parameter-effiiiet few-shot tuning using IA3 (Liu et al., 2022) to make use of the labeled data.
Retrieving cross-task nearest neighbors To retrieve the most similar instances to a given set of target task instances, first build index over the massive pool of multi-task data for efficient retrieval, encoding samples using a pretrained encoder. Then, given a set of query instances Q. we retrieve our subset of similar data by computing union of the k-nearest neighbors all g E Q. Note that there may be an overlap between the sets of nearest neighbors retrieved for different queries, and hence R| [Q] * k. where R the retrieved subset. Empirically, we find |R] tends to be 5-50x smaller than Q * k due to this overlap.
Data-Efficient Training Given retrieved set of data R, we can then simply train a pretrained language model on this mixture of data using cross-entropy loss, as all data is in unified text-to- text prompted format. This training is similar to the multitask prompted training of TO (Sanh et al.] 2021 We refer to models trained on R as DEFT models. If we have no labeled data available, we directly evaluate these models on our target tasks.
Parameter-efficient Few-shot Finetuning For the case where few annotated labels are available, we make use of parameter-efficin few-shot finetuning. For this, we take our multi-task trained DEFT checkpoints and finetune them using IA3 (Liu et al., (022) on task-specific few-shot data. Concretely, given a trained transformer model, we introduce three new vectors &. h, and lg into the attention and feed-forward mechanisms of each layer as follows:
Attention(Q. (Q, K, V) = softmax (QU (MOOK")
(4 (Lon)
(1)
Vdh
Feed-forward(z)) = (Ie O f(Wiz))W2
(2)
We initialize these vectors with all ones and only update them during the few-shot finetuning. This provides an efficient method of further training our DEFT models on task-speeific data and has been shown to outperform full finetuning in the few-shot setting (Liu et al. 2022
4 EXPERIMENTS
4.1 SETUP & HYPERPARAMETERS
We construct an index of P3 data using FAISS (Johnson et al. 2019). a library for efficient similarity search over dense vectors. We use a Hierarchical Navigable Small World in- dex (Makkov& & Yashunin 2020) to approximate the k-nearest neighbor search. To keep the size of the index manageable, we use Product Quantization Jegou et al. (2010) and reduce the dimension- ality of the encoded representations using an Optimized Product Quantization transform Ge et al. (2013). We encode our instances using the T5 v1.1 model with extra language model pretraining introduced by Lester et al. (202T). For all experiments, we match the size the encoder used to index data and the size of downstream models trained on this data (e.g., if we train T5-XI. sized
Indexing P3
4

---

Under review as a conference paper at ICLR 2023
model, we use T5-XL to index and retrieve the data). We use the subset of P3 used to train TO as our pool of multitask data unless otherwise stated.
DEFT Training Following TO, we start with the T5 v1.1 model with extra language model pre- training. Unless otherwise stated, we use the 'XL' variant with 3 billion parameters across our experiments. When training on cross-task nearest neighbors, we train for 5 epochs with a batch size of 8 using the Adam optimizer (Kingma & Ba 2015) and learning rate of 5e-5. We use linear warmup schedule for the first 10% of the total training steps and linear decay for the rest of training.
Few-shot training We follow the settings suggested by Liu et al. (0222: : training for 1000 steps with a batch size of We use the Adafactor optimizer with a maximum learning rate of 3e-3 and a linear decay schedule with 60 warmup steps. We only update the IA3 vectors during training.
Evaluation Datasets We evaluate on the set of 11 datasets used to evaluate TO (RTE, ANLI R1/2/3, CB, HellaSwag, Story Cloze, WinoGrande, WSC, COPA, WiC), which include natural lan- guage inference and commonsense reasoning datasets. In addition to the TO evaluation datasets, we also evaluate on three additional datasets from diverse domains: CaseHold (Chalkidis et al., 2022 Zheng et al. 2021), a legal QA dataset, DROP (Dua et al. 2019). QA dataset that requires discrete operations, and subtask of Qasper (Dasigi et al., 2021), a QA dataset over entire NLP papers. Qasper has two subtasks seleectng paragraphs in the paper that provide evidence for answering the questions, and generating the answers. We focus on the former because it was shown to more difficult of the two, and convert it into binary classification task where the inputs combinations of questions and single paragraphs. We refer to this subtask as Qasperfvidence henceforth, and evaluate model performance in terms of document-level F1 as described in Dasigi al. (202T). ). For evaluation and few-shot training, we convert all datasets to prompted text-to-text forma eithee us- ing the prompt templates from P3 for the TO evaluation datasets or an original prompt for the other datasets. For CaseHold, DROP, and QasperEvidence we randomly split out 1000 examples from the existing validation sets to use for retrieval, and use the remaining data for evaluation. For all other datasets, we simply retrieve using up to 1000 randomly chosen examples from the training splits (if a dataset has less than 1000 training examples, we use all available training data for retrieval). We provide list of these prompts and further dataset details in Appendix A
Model Evaluation Following Sanh et al. (2021) and Brown et al. 2020 we calculate accuracy on all datasets except DROP using rank classification, where we pick the answer with lowest loss across possible answer choices given the instance input as the model prediction. As DROP is QA dataset that requires selecting spans or generating numbers, and does not have answer choices, we simply generate the prediction using greedy decoding.
Baselines For zero-shot evaluation, we primarily compare against 4 baselines: 1) TO-3B, trained on about 10% of the P3 data3] 2) Random, a model trained on a random selection of P3 data the same size as the subsets selected by DEFT, 3) T5-XL not finetuned any further, and 4) BM25, using B Robertson & Zaragoza, 2009) for retrieval instead of dense representations. For few-shot settings, we compare TO-3B with additional few-shot training with DEFT checkpoints trained on subsets chosen using (a) 1000 unlabeled instances and (b) the instances used in the few-shot training without labels. This means (b) uses no additional data, labeled or unlabeled, compared to TO-3B with shw-shot finetuning.
4.2 DATA-EFFICIENT FINE-TUNING VS. MASSIVE MULTITASK TRAINING
We first assume we have access only to unlabeled task-specific data, and so cannot train on any target task labeled data. We sample 1000 unlabeled instances per dataset and retrieve the 500 nearest
1 For example, ANLI instances were converted to '(premise} Question: (hypothesis) Truc, False, or Nei- ther?', with the answers as 'truc', 'false", or "neither'.
Sanh al. 2021 report that they train T5-XI. on most 500K instances per prompted dataset in P3, which amounts to about 10% of the pool.
We use Pyserini (Lin et al. 2021) with default settings for building and retrieving from the BM25 index.
We restrict the amount of data retrieved to same size as the subsets retrieved by DEFT.
5

---

Under review as a conference paper at ICLR 2023
DRFT-XL
T0-38
Rand-XI,
Rand-Bal
TS-XI.
DEFT-base
Rand-base
T5-hase
Maj. Class
CaseHold
$7.2
30.9
19.0
38.7
11.4
27.9
18.9
17.5
11.4
6.6
DROP
31,0
27.4
24.3
27.6
11.3
22.6
21.3
18.0
4.0
Quspertv
19.9
8.2
RTE
79.0
70.3
78.5
78.0
53.1
79.3
61.7
61.0
5200
33.1
ANLI RI
39.8
35.0
35.3
40,0
32.9
31.5
29.6
33.3
32.9
33.4
ANLI
37.5
32.6
35.3
36,9
33.5
36,9
32.5
22.3
33.5
33.4
32.7
CB
60.7
58.9
60.7
55.4
44.5
50.0
31.6
50.0
48.2
44.6
50.0
HellaSwag
33.1
28.2
27.4
29.3
23.0
28.7
26.0
25,9
25.0
23.0
25.7
StoryCloas
95.3
86.5
79.1
94,1
53.0
62.3
83.5
57.1
53.0
51.4
WinoGrande
50.0
WSC
39.4
50.0
47.1
46.2
36.3
36.5
61.4
42.3
36.5
36.3
63.5
COPA
95.0
74.0
80.0
88.0
60.0
79.0
66.0
44,0
60.0
55.0
WIC
54.9
51.1
51.4
57.5
51.7
$1.9
50.6
49.4
51.7
50.0
Average
35.3
Table 1: Performance of XL (3B) and base size (~250 million) models across datasets. 'Rand' refers to performance of models trained on randomly chosen P3 subsets of equivalent size to the chosen by DEFT, with 'Rand-bal' using uniform random sampling across tasks for subset selection. "T5' refers to performance of a non-finetuned T5 model. 'BM25' refers to models trained on subsets of equivalent size DEFT subsets from P3 retrieved using BM25, DROP and QasperEv. Results are Fl scores, CaseHold micro Fl, all else accuracy. Datasets below dashed line were used evaluate TO in |ann et al. (202T). * ReCross results reported by Lin al. (2022) and mean performance across multiple prompts.
neighbors* of each instance. We then train dataset-specific models each of the retrieved sets. As seen in Table []] our DEFT-XI. models consistently outperform TO-3B and other baselines, with median relative improvement of 13% over TO-3B. We also see that T5-base sized models also improve over baselines in Table - the DEFT-base models have a median relative improvement of 8% over the random baseline. All DEFT models are trained on subsets P3 consisting of 0.1- 2% of all P3 data. This confirms our hypothesis that training on well-chosen subset of P3 is more beneficial for target task performance than training on a uniform sample all available data. We also note that using dense representations is crucial to our approach, as using BM25 for retrieval instead overall underperforms TO-3B and even the random baselines. We also note DEFT-XL outperforms ReCross by large margins for 4/6 tasks. Our results suggest that general language model encoder can still retrieve relevant cross-tast neighbors, contrary to the claims made by Lin et al. (2022).
Remarkably, DEFT-XL outperforms the majority baselines on two target datasets (QasperEvidenee and ANLI R2) where TO-3B does not, and DEFT-base on one (COPA). This observation further confirms that multitask models trained uniformly sampled data might be suffering from negative interference between tasks.
We note our approach underperforms compared TO-3B and Rand-XI. for WSC and RTE respec- tively. However, both datasets have small evaluation sets and large variance (see Appendix B}. lead- ing us believe these differences are not significant. Relatedly, TO-3B underperforms majority class baseline on WSC.
4.3 FEW-SHOT FINETUNING OF DEFT MODELS
Next, we assume we are able to label small number task-specific examples, and further train DEFT models. We reuse the XL-size models trained in section 4.2 and further train them using the parameter-efficient IA3 on the few-shot data used by Liu et al. (2022). As seen in table DEFT models with few-shot finetuning ('DEFT-Few (IkQ)') perform models with few-shot finetuning ("TO-3B+IA3'), with statistically significant gains on datasets. This shows that DEFT models serve better starting points for few-shot finetuning than TO-3B, providing similar or better performance across all datasets despite being exposed to much less train- ing data. Notably, DEFT-Few significantly outperforms TO-3B+IA3 on WinoGrande, for which zero-shot DEFT did not significantly outperform zero-shot TO-3B. These results show that DEFT models are more amenable to few-shot fine-tuning than TO-3B.
average 7% better than TO-3B
We retrieve 2500 nearest neighbours for T5-base, as carly experiments suggested it performed worse with only 500 neighbors.

---

Under review as a conference paper at ICLR 2023
TO-3B+IA3
TS+1A3
Rand+IA3
DEFT-Fow (IkQ)
DEFT-Few (20-70Q)
RTE
77.52.0
57.04.3
79.4
81.31.
ANLI RI
44.950
39.61.8
43.3
47.31.4
47.3
ANLI R2
39.51.7
36.51.4
40.31.6
40.8ga
42.2g.7
ANLI R3
40.22.2
34.81.1
39.32.3
44.32.1
42.91
CB
78.9s.9
67.92.6
81.4a.5
82.52.6
84.6
HellaSwag
34.70.6
27.51.1
38.11.1
42.52.1
45 91
StoryCloze
93.00..
83.03.1
92.6n.
96.20.2
96. 5.
WinoGrande
50.61.
49.8o.8
51.42.3
55.9g.0
55 2g.1
WSC
64.83.6
51.01.0
55.8g.0
63.3g,2
59.6g.8
COPA
82.02.
61.64.2
86.61.7
95.41.5
92 62.2
WIC
54.91.
56.63.0
54.52.4
57.729
07.42.9
Average
60.1
51.4
60.6
64.1
64.1
Table 2: Performance of IA3 few-shot finetuned models using XL-size checkpoints. For all models we report the mean over runs with the standard deviation as subscript. We report performance for DEFT-Few models using 1000 unlabeled queries ('IkQ") and few-shot queries ("20-70Q'). See section 4.3 for details. Bolded numbers indicate that value is statistically significantly better than TO-3B+IA (p .05 two-tailed paired t-test).
Few-shot Retrieval In this experiment, we evaluate DEFT in a setting where we have access only to small number of target-task labeled examples (exactly what is available to T0-3B+A33), and no additional unlabeled examples. We construct few-shot sets for each dataset, and for each set retrieve cross-task neighbors using the few-shot data, finetune T5 models on the retrieved data, and then finally finetune using IA3 on the labeled few-shot data itself. In order to make up for the smaller query set, we retrieve the closest 2000 neighbors per query instance. As seen Table 2 this still results in model that outperforms TO-3B with few-shot tuning ('DEFT-Few (20-70Q) ), and overall achieves similar performance followed by few-shot finetuning may be a better alternative to few-shot finetuning TO-3B even when both methods have exactly the same target-task information available.
DEFT-Few (IkQ). Crucially, this shows that DEFT
ANALYSIS
5.1 CROSS- TASK RETRIEVAL
What gets retrieved? We show what source datasets get selected during retrieval for each evalu- ation dataset in figure 2 For most target datasets, the majority of source datasets are not selected, further strengthening given target task, and no single mixture of datasets is optimal for all target tasks. We additionally find that no more than 27% of all instances within any source dataset retrieved, suggesting that our approach is also effective at finding relevant subsets of data within large datasets.
hypothesis that much of the massive multitask pool is not relevant
Retrieval Hyperparameters When retrieving cross-task data, the amount and quality of data re- trieved is highly dependent on the query size (i.e. the number of task-specific instances used for retrieval) and number of neighbors (i.e. the number cross-task samples retrieved per task-specific instance). In figure 3 we show the effect of varying both query size (sweeping from 32 to all train- ing data) and the number neighbors (sweeping from 5000) on dataset performance on RTE and CaseHold respectively. We find that increasing the amount of data retrieved, whether through increasing the number of neighbors or query set size, results improved performance up to point, and then either plateaus decreases, providing evidence for our hypothesis that using 'too much' data can result in reduced downstream performance due to negative interference.
What model should you use for retrieval? To determine the effect of of model size on indexing and retrieval, we train models using the cross-task neighbors retrieved by base and XL-size models when the query size and number of neighbors is held constant. We find that using larger (XL size) indexing model generally results in better performance, but this gap is much larger when training a base size model (8%) than when training XL-size models (1%), suggesting that smaller models benefit more from larger retrieval models. We provide detailed results in Appendix
7

---

Under review as a conference paper at ICLR 2023
oasehaldd
helnawwng
atory dkoe
winegrande
ani rl
ani /2
ard rd
gasper
cxpa
dop
wic
adversanial ca
0.28
0.28
0.11
0.16
0.04
0.00
0.47
0.07
0:24
0.08
0.00
0.01
0.00
0.03
0.02
nouns
0.02
0.03
0.15
0.00
0.03
0.00
0.01
0.00
0.00
0.7
0.00
0.00
0.00
0.00
0.04
amazon polerily
0.04
0.03
0.03
0.01
0.02
0.00
0.00
0.08
0.04
0.02
0.00
0.00
0.00
0.00
0.04
app_reviews
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.04
3.0.
0.81
0.00
0.08
0.27
0.02
0.00
0.05
0.05
0.01
0.08
0.01
0.00
0.00
0.00
0.04
common_gee
0.30
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.03
0.05
0.01
0.04
e_v1.11
0.00
0.00
0.00
0.00
0.02
0.53
0.00
0.01
0.01
0.00
0.00
0,01
0.02
0.01
0.01
0.00
0.00
0.07
0.00
0,26
0.02
0.00
0.22
0.01
0.00
0.32
0.00
0.02
0.18
0.04
dbpedia_14
0.10
0.00
0.00
0.00
0.00
0.04
0.00
0.00
0.16
0.02
0.00
0.00
duore_ ParaphraseRi
0.00
0.01
0.00
0.00
0.00
0.01
0.00
0.00
0.00
0.00
0.00
0.01
0.04
duoro SelIRC
0.01
0.01
0.00
0.04
0.00
0.00
0.00
0.01
0.00
0.00
0.00
0.00
0.00
0.00
0.04
gigaword
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.01
0.00
0.00
0.00
0.00
0.04
glue_mpe
0.00
0.00
0.01
0.00
0.00
0.00
0.00
0.00
0.00
0.08
0.00
0.05
0.02
0.02
0.00
glue glue_oap
0.00
0.00
0.01
0.00
0.01
0.01
0.00
0.01
0:21
0.03
0.00
0.60
0.00
0.01
0.04
imdb
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.04
kitt tasks hotpotqa
0.18
0.19
0.06
0.00
0.00
0.00
0.00
0.000
0.00
0.01
0.00
0.00
0.00
0.00
0.04
mulli news
0.00
0.00
0.02
0,30
0.01
0.00
0.02
0.02
0.02
0.03
0,00
0,00
0,00
0.00
0.03
labeled final
0.08
0.06
0.03
0.00
0.02
0.00
0.00
0.00
0.06
0.08
0.00
0.11
0.09
0.13
0.04
qaec
0.00
0.01
quail
0.00
0.08
0.03
quarel
0.02
0.00
0.01
0.01
0.00
0.00
0.03
quartz
0.01
0.00
0.00
0.00
0.00
0.00
0.01
0.01
0.00
quoref
0.01
0.01
0.00
0.03
0.00
0.00
0.21
0.01
0.01
0.00
0.00
0.00
0.00
0.00
0.02
ropes
0.02
0.02
0.04
0.01
0.02
0.00
0.04
0.27
0.08
0.01
0.07
0.00
0.01
0.03
0.01
rotten_ rotten_tomabin tometoes
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.01
samsum
0.00
0.00
0.01
0.00
0.12
0.00
0.00
0.00
0.00
0.00
0.05
0.00
0.00
0.00
0.01
sciq
0.01
0.00
0.01
0.00
0.00
0.01
0.00
0.02
0.02
0.00
0.00
0.01
0.00
0.01
0.01
social Lgga
0.00
0.00
0.05
0.00
0.17
0.39
0.00
0.05
0.01
0.01
0.52
0.03
0.82
0.32
0.02
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.03
0.00
0.00
0,01
0,00
0.00
0,00
wikibbio
0.00
0.01
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.04
wik_hhop_oiggnnll
0.04
0.01
0.01
0.00
0.00
0.01
0.04
qa
0.11
0.02
0.00
wiga
0.00
0.00
0.00
0.01
xsum
0.01
0.01
0.13
0.07
0.01
0.00
0.07
0.01
0.01
0.15
0.00
0.00
0.00
0.00
0.04
yelp_ yelp_review, www.reevewwwwee Jull
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.01
0.00
0.00
0.00
0.00
0.00
0.00
0.04
Figure 2: Proportion of the retrieved training data for each evaluation dataset (columns) that comes from each dataset in P3 (rows). The final column shows these values for all P3.
60
awaries
10
10a
10'
16"
auofess
40
335
1000
2101
30
Figure 3: (above) RTE accuracy by number of P3 samples retrieved for DEFT-XL. across a varying number of neighbors. (below) CaseHold micro F1 by number P3 samples retrieved for DEFT-XL. across varying number of neighbors.
Are prompts useful for retrieval? All P3 data is in a prompted format, where the input made up of (a) the input instance and (b) prompt that contains information about the task. Training on prompted data greatly aids zero-shot generalisation (Wei et al., 2021b) Sanh al. [002]), but is unclear how useful it is for retrieval. To examine this, we run experiments using Super-Natural
8

---

Under review as a conference paper at ICLR 2023
60
50
No retrieval
29630
Majority Class
Doc
40
R|=100
|R|-1000
R|-00000
10300
[R]-29690
30
RR
20
10
101
10a
10a
10a
10a
Budget (number of unlabelled examples)
Figure 4: Performance of DEFT-XL and full-finetune foll-finetuning methods with the same annotation budget used for obtaining cither labeled or unlabeled data for QusperEvidence. Unless one has a large annotation budget, collecting unlabelled examples is superior to collecting labelled ones. |R] and [T] refer to the size the retrieval and train sets respectively. See section 5.2 for details.
Instructions Waang et al. 2022h, recent instruction tuning dataset. We index and retrieve the data with and without instructions in the input and compare the performance after training on retrieved subset&]] We find that retrieving without instructions outperforms retrieving with instructions by a small margin (49.2 vs 49.5 average performance), suggesting that DEFT relies more on instance information rather than task information for retrieval. We provide more details Appendix D
5.2 PRACTICALITY OF ASSUMING ACCESS TO UNLABELED DATA
Contrary to prior work, our approach assumes access to unlabeled data. This is practical assump- tion given that unlabeled data is often readily available or is far cheaper to acquire than labeled data. This is especially true for tasks such as Qasper CaseHold, which require experts to carefully read (sometimes quite long) texts to provide labels. We argue that DEFT's use of unlabeled data make it cost-efficient method obtain well-performing task-specifie model when the data labeling budget is limited.
We examine this by studying scenario where QasperEvidence data collected and assume have access to P3 and DEFT to make efficient use of it. Obtaining labeled instances for QasperEvi- dence cost 3.25 times acquiring unlabeled (question-paragraph) instanced We compare (Figure 4h performance on the test set of a T5-XL model trained on a varying number of labeled instances with DEFT-XL model trained on cross-task nearest neighbors of 3.25 as many unlabeled instances. DEFT yields better results for smaller annotation budgets (< 1000 labelled examples), and under- performs models trained on few thousand labelled examples. This confirms our suggestion that DEFT is preferable to regular finetuning for limited budgets. We also note that the DEFT setup makes easy to use target-task labeled data when available, as shown in section 4.3
CONCLUSION
In this work, we propose Data-Efficient FineTuning, a novel method for efficiently using multitask data by training task-specific models using only small amount of unlabeled target task data. We use the unlabeled data to select subsets of the multitask data, and train models on these subsets. The resulting models outperform same-sized models trained all available data (e.g.. TO), suggesting that negative interference is still present even in large massively multi-task models. Our approach also results in models that work as better starting points for few-shot finetuning, outperforming models trained on more data. Our approach still performs strongly even when as few as only 20 unlabeled examples are available. Overall, our results strongly suggest that training all available data, even with large models, is not the optimal choice for most downstream tasks, and that instead focusing on ways to better curate datasets into higher-qualite but smaller, mixtures is potentially better path forward.
'We add instructions back into samples without them in order to isolate the effect of instructions retrieval separate from their effect during finetuning. 'Based on an estimate provided by the authors of the dataset. Questions were written after only the abstracts of papers were read, and evidence selection required reading entire papers.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Vamsi Aribandi, Yi Tay. Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mchta, Honglei Zhuang, Vinh Q. Tran, Dara Bahri, Jianmo Ni, Jai Gupta, Kai Hui, Sebastian Ruder, and Donald Metzler. Ext5: Towards extreme multi-task scaling for transfer learning. In International Conference on Learning Representations, 2022. URL/tttps::/ppeeeeeeee net/ forum?
1d=vzh1BFUCi
Roy Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and
Idan Szpektor. The second PASCAL recognising textual entailment challenge. 2006.
Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. The fifth PASCAL recognizing textual entailment challenge. 2009.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Asso- ciates, Inc., 2020. URL htets:://proceedings.eees neurips. http:/g......rrrss....cc///////a hctpppapee/2222/111eee 1457c0d6bf cb4 9679674496447777 8ofb8ac 12 42w642--42aaaa 42aeaa--aarrrrrr.r..a pdf
Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Mar- tin Katz, and Nikolaos Aletras. Lexglue: A benchmark dataset for legal language understanding in english. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, Dubin, Ireland, 2022.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual entailment challenge. In Machine learning challenges. evaluating predictive uncertainty, visual object clas- sification, and recognising tectual entailment, pp. 177-190. Springer. 2006.
Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. A dataset of information-seeking questions and answers anchored in research papers. In Proceed- ings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pp. 4599 Online, June 2021. Asso- 1atp:////mmmmaammmmmm maacl-main.365. URL https:
ciation for Computational Linguistics. doi: 10. 8653/v1/2021
7/acl Mocclanthollooooo org.2011. naacl - ma 365
De Marneffe, Mandy Simons, and Judith Tonhauser. The CommitmentBank: In- vestigating projection in naturally occurring discourse. 2019. To appear in proceedings of Sinn und Bedeutung 23. Data can be found at http://wwwuucoc//m//n/o///mtennt
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp- 2368 2378, Minneapolis, Minnesota, June 2019. Association for Computational I Linguistics. doi: 10. 10.18653/v1/N19. 1246. URI. http::ttpss........... ac antho org/N1911
Tiezheng Ge, Kaiming He, Qifa Ke, and Jian Sun. Optimized product quantization for approximate nearest neighbor search. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2946-2954 2013.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pp. 1-9. Association for Computational Linguisties, 2007.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don't stop pretraining: Adapt language models to domains and tasks. In
10

---

Under review as a conference paper at ICLR 2023
Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pp-
Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/ 2020.acl- main.740. URL htspssss///aclattoo..ory http://o.......ooo//o....aa.a--. amiiinn.. 740
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In International Conference on Machine Learning, pp. 3929-3938. PMLR, 2020.
Xiaochuang Han and Yulia Tsvetkov. Orea: Interpreting prompted language models via locating supporting data evidence in the ocean of pretraining data. arXiv preprint 1tti:2205.12600 2022.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pp. 2790-2799. PMLR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09683 2021.
Herve Jegou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. IEEE transactions on pattern analysis and machine intelligence, 33(1):117-128, 2010.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Transactions Big Data, 7(3):535-547, 2019.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. arXiv preprint arXiv:1911.00177 2019.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannanch Hajishirzi. UNIFIEDQA: Crossing format boundaries with single QA system. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1896-1907, Online, November 2020. Association for Computational Linguistics. doi: 10. 10.18653/w1/2220 fi nin - emnlp.
findings-emnl..711. URL http:://////cattttooooooo.o/////
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization, ICLR (Poster), 2015. URL http://////w.....ro///////////// wttp////g/////////ee/e//
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pp. 1885-188 1894. PMLR, 2017.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efffcient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 3045-3059, Online and Punta Cana, Dominican Republic, November 2021. Associatinn for Computational Linguistics. doi: 10. 10.1655/11/222..emmgg mappppmii..... URL org/
https://aclanthologg 2ttp://cl..nnollooo...rg2222211. emalp-ma 24.
Hector J Levesque, Ernest Davis, and Leora Morgenstern. The Winograd schema challenge. In AAA/ Spring Symposium: Logical Formalizations of Commonsense Reasoning, volume 46, pp. 47, 2011.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval -ugmeente genera- tion for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33: 9459-9474, 2020.
Bill Yuchen Lin, Kangmin Tan, Chris Miller, Beiwen Tian, and Xiang Ren. Unsupervised cross-task generalization via retrieval augmentation. ArXiv, abs/2204.07931 2022.
11

---

Under review as a conference paper at ICLR 2023
Jimmy Lin, Xueguang Ma, Sheng-Chich Lin, Theng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. Pyserini: A Python toolkit for reproducible information retrieval research with sparse and dense representations. In Proceedings of the 44th Annual International ACM SIGIR Con- ference on Research and Development in Information Retrieval (SIGIR 2021), pp. 2356-2362, 2021.
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learn- ing. arXiv preprint arXiv:2205. 05638, 2022.
Yu A. Malkov and D. A. Yashunin. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(4):824-836 2020. doi: 21099TPPAML2018.28884447
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Natural instruc- tions: Benchmarking generalization new tasks from natural language instructions. CoRR, abs/2104.08773, 2021. URL
wwwpaiiww.r..gr/eooooooe abs 22t.//114....8
Nasrin Mostafazadeh, Michael Roth, Annie Louis, Nathanael Chambers, and James Allen. Lsdsem 2017 shared task: The story cloze test. In Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics, pp. 46-51, 2017.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adver- sarial NLI: A new benchmark for natural language understanding. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguisties, 2020.
Pouya Pezeshkpour, Sarthak Jain, Byron C Wallace, and Sameer Singh. An empirical comparison of instance attribution methods for nlp. arXiv preprint arXiv:2104.04/ 04128, 2021.
Jason Phang. Thibault Févry, and Samuel R. Bowman. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. ArXiv, abs/181 2018
Mohammad Taher Pilehvar and Jose Camacho-Collados. WiC: The word-in-context dataset for
evaluating context-sensitive meaning representations. In Proceedings of NAACI-HLTT, 2019.
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33: 19920-19930, 2020.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020.
Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: Bm25 and beyond. 3(4):333-389, apr 2009. ISSN 1554-0669, exc/10.1561/5000000
doi: 10.1561/1500000019, URL httpp:://oi..
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series, 2011.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An ad- versarial winograd schema challenge at scale. Commun. ACM, 64(9))99-106, aug 2021. ISSN 0001-0782. doi: 10.1145/3474381. htt.://ppa//ooo...//0000.....
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, An- toine Chaflin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. arXiv preprint 100002211. 2021.
Weijia Shi, Julian Michael, Suchin Gururangan, and Luke Zettlemoyer. Nearest neighbor zero-shot inference. arXiv preprint arXiv:2205. 2022.
12

---

Under review as a conference paper at ICLR 2023
Tu Vu, Tong Wang, Tsendsuren Munkhdalai, Alessandro Sordoni, Adam Trischler, Andrew Mattarella-Minn Micke, Subhransu Maji, and Mohit lyyer. Exploring and predicting transferability across NLP tasks. In Proceedings of the 2020 Conference Empirical Methods in Natural Lan- guage Processing (EMNLIP), pp. 7882-7926, Online, November 2020. Association for Computa- tional Linguistics. doi: 20.18665/1122222emmmllpp 635.. URL httphttts///caattttiit
org/2020. emnip main. 635
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman, SuperGI.UE: stickier benchmark for general-purpose language understanding systems. arXiv preprint / 1905.00537, 2019.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi. Yeganeh Kordi, Amirreza Mirzaci, An- jana Arunkumar, Arjun Ashok, Arut Selvan Dhanasckaran. Atharva Naik, David Stap, et al. via declarative instructions 1600+ tasks. In EMNLP,
Suppe=nnttualinstttiioognnnnnnnn 2022.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2 169.0162, 2021a.
Jason Wei, Chengyu Huang, Soroush Vosoughi, Yu Cheng, and Shiqi Xu. Few-shot text clas- sification with triplet networks, data augmentation, and curriculum learning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5493-5500, Online, June 2021b. Associa-
tion for Computational Linguistics. doi: 10. 10.18653/v1/202 1aae22nnaall-aaae maaellmain.444.
URL https:
#/aclanthology. org 2021, naacl seaacl-maingg www..annn..mmmmmm........amm 434
Chih-Kuan Ych. Joon Kim, Ian En-Hsu Yen, and Pradeep K Ravikumar, Representer point selection for explaining deep neural networks. Advances in neural information processing systems, 31, 2018.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, 2019.
Lucia Zheng, Neel Guha, Brandon R. Anderson, Peter Henderson, and Daniel E. Ho. When does pretraining help? assessing self-supervised learning for law and the casehold dataset of 53,000+ legal holdings. New York, NY, USA. 2021. Association for Computing Machinery. ISBN URL https: http:://ee..rg/10.1100000
9781450385268. doi: 10.1455/462255.34466688
0462257.3346002
13

---

Under review as a conference paper at ICLR 2023
Dataset
Retrieval
Eval
#Shots
Retrieval from
CaseHold (Zheng et al.
2021
1000
2900
Validation
DROP (Dua et
2019
1000
8535
Validation
QasperEvidence Dasigi et al.
2021
1000
43673
Validation
RTE*
1000
277
32
Train
ANLIRI (Ne et al.
2020
1000
1000
50
Train
ANLIR2 NNe
2020
1000
1000
50
Train
R3 (Nie et al
2020
1000
1000
50
Train
CB (De Marneffe et al.
2019)
2019
250
56
Train
HellaSwag (Zellers et al.
2019)
1000
10003
20
Train
StoryCloze (Mostatazadeh et a.
2017
1000
1871
70
Train
WinoGrande (Sakaguchı et al.
2021
1000
1767
50
Train
WSC (Levesque et al.) 2011)
554
104
32
Train
COPA RRemmele al. 2011
400
100
32
Train
WiC Pilehvar & Camacho Camacho-Collados 2019)
1000
638
32
Train
Table Size of splits used for experiments across datasets. "#Shots' indicates the number of shots
used in few-shot experiments, and 'retrieval from' indicates which split we selected retrieval data
from. * Following SuperGLUE Wang et al. (2019). RTE data is from RTE 1/2/3/5 (Dagan et al.
2006
Bar Haim et al.
2006 Giampiccolo et al. 2007 Bentivogli et al.
2009
A
DATASET DETAILS
Sizes and Splits For each dataset used, we provide the number of retrieval and validation examples used in Table 3 We also indicate if the retrieval data was split from the validation or training split. Note any data used retrieve is held out of the validation split avoid information leakage. We additionally provide the number of shots used for each dataset. We follow the number of splits used Liu et al. (2022) and use the data shared by the authors (available htttp://ggthhub.coom wwwpslu...ommttteee//.. few/tree/masste//dtta//ii shot).
Prompts We list the prompts used for each dataset. {x} indicates a space that is filled in by instance data.
CaseHold: What the correct holding statement for the following text? Text: fcontext) (A): {ending 1) (B): {ending 2} (C): {ending 3} (D): {ending 4) (E): {ending 5}
DROP: Passage: {passage} Question: {question] Answer:
QasperEvidence: Question: {question) } Paragraph: (paragraph) the answer to the ques- tion in the paragraph? Answer Yes or No.
RTE: {premise] Question: Does this imply that "(hypothesis ! Yes or no?
ANLI: {premise} Question: [bypothesis) True, False, or Neither?
CB: (premise) Question: (hypothesis) True, False, Neither?
HellaSwag: Complete the description with appropriate ending: First, (context a} Then, (context b) (a) (ending 1} (b) (ending 2) (c) (ending 3} (d) {ending 4)
StoryCloze: {input sentence }} (input sentence 2} (input sentence 3} (input sentence 4} What is a possible continuation for the story given the following options (answer 1} {answer 2}
WinoGrande: {sentence] What does the in the above sentence refer to? [option1] } or (option2)?
WSC: Passage: (text) Question: In the passage above, does the pronoun '(span 1)' refer to "(span 2}" Answer:
COPA: (premise) a consequence... Help me pick the more plausible option: {choice }} (choice 2}
WiC: {sentence 1} (sentence 2} Question: the word ([word)' used in the same sense in the two sentences above? Yes, No?
14

---

Under review as a conference paper at ICLR 2023
Task
DEFT-Few (20-70Q)
RTE
73.24.0
ANLI R1
36.13.0
ANLI R2
34.10.9
ANLI R3
40.62.0
CB
58.2
HellaSwag
34.1u.7
StoryCloze
95.10.3
WinoGrande
50.61.2
WSC
51.05.1
COPA
87.81.1
WIC
50.81.7
Average
55.6
Table 4: Performance of XL size models trained using DEFT with few-shot queries. We report the mean and standard deviation over runs.
Train Model Size
Base
XL
Index Model Size
Base
XL
Base
XL
CaseHold
14.8
15.8
32.6
37.2
DROP
20.8
21.3
30.4
31.0
Qasper
15.7
18.0
23.3
28.5
RTE
53.4
61.7
77.3
74.0
ANLI RI
33.3
33.3
39.5
39.8
ANLI R2
33.4
32.8
35.3
37.5
ANLI R3
33.2
33.3
42.5
41.4
CB
50.0
50.0
75.0
60.7
HellaSwag
26.0
27.9
31.7
33.1
StoryCloze
74.0
76.8
94.4
95.3
WinoGrande
49.5
50.4
51.4
50.6
WSC
41.4
42.3
43.3
39.4
CОРA
63.0
60.0
85.0
95.0
WiC
48.8
48.3
49.5
54.9
Average
39.8
42.8
50.8
51.3
Table 5: Performance of DEFT models trained on cross-task neighbors retrieved using different-size models.
B FEW-SHOT RESULTS WITHOUT IA3
For 'DEFT-Few (20-70Q)' in Table we trained 5 models using DEFT (as we used 5 few-shot sets per dataset). In Table we report the performance of these models without IA3 training. Note we did not train few-shot models for CaseHold, QasperEvidence, or DROP, and so do not report results on these datasets. Notably, RTE, CB, and WSC all have quite large standard deviation (> 3.0), which suggests our improvements (or deterioration, for WSC) over TO-3B for these datasets may not be significant.
C INDEX MODEL SIZE EXPERIMENTS
We explored mismatching the index model sizes, training XI. size models on cross-task neighbor splits indexed and retrieved using T5-base, and vice-versa. We query size of 1000 and retrieve 500 neighbors per query instance. We present the results in Table
15

---

Under review as a conference paper at ICLR 2023
Evaluation Category
DEFT-XL (instruct)
DEFT-XL (no instruct)
Rand-XL
Tk-instruct
Answerability
48.0
48.0
49.0
47.0
Cause Effect Classification
83.3
83.3
84.7
87.7
Coreference
61.0
51.0
43.0
83.0
Data Text
34.0
34.4
33.4
37.9
Dialogue Act Recognition
65.0
61.0
59.0
68.0
Entailment
50.0
68.0
13.0
19.0
Grammar Error Correction
86.3
84.8
84.7
84.8
Keyword Tagging
17.4
17.6
19.2
Overlap
17.7
20.2
22.3
17.8
Question Rewriting
45.8
64.0
59.9
68.8
Title Generation
21.4
20.9
20.3
20.4
Word Analogy
60.0
41.0
60.0
61.3
Average
49.2
49.5
45.7
50.7
Table 6: Performance of XL-size models on 12 tasks from evaluation categories Wang et al. 2002).. All results are in RougeL. 'Instruct' and 'no instruct" variants of DEFT-XL refer to models trained using subsets of Super-Natural Instructions that were retrieved using instructions and without using instruction respectively.
D SUPER-NATURAL INSTRUCTIONS EXPERIMENTS
We use version 2.7 of the Super-Natural Instructions dataset and use the official splits provided, with 100 samples per train and evaluation tasks. This results pool of 75.317 train examples. For evaluation, randomly select one task per evaluation category in Table 5 of [aang et al. (2022). Task names are given in Table 7 We then generate two indices for retrieval: one where each sample is encoded including the task instruction, and one where each sample is encoded without any instruction. We then retrieve using the 100 unlabelled test instances from each chosen evaluation task, matching the format used for the index (i.e., if we retrieve from the index with instructions, encode our query data with instructions included). In order to isolate the effect of instructions on retrieval, after retrieving examples, always train the corresponding examples with instructions included (i.e., when we retrieve examples without using instructions, we add the instructions back into the inputs before finetuning). On average, retrieve 3.5k training examples, roughly 5% of the total training data. Additionally, we finetune T5-XL model using all available training data ('Tk-instruct') ), and a random baseline using random subsets of the training data of the same size as the retrieved subsets ( ('Rand-XL')
We present our results Table We find that the instruction auggeentd and no no-instruction retrieval DEFT models achieve similar performance on average, although the no-instruction variant performs slightly higher. Both DEFT models significantly outperform the Rand-XI. baseline, suggesting that the retrieval is still effective prompts. However, we find that neither DEFT model significantly outperforms Tk-instruct, which hypothesise is related to the significantly smaller size of Super-Natural Instructions compared to P3. However, we note that our DEFT-XL models are trained on significantly less data than Tk- instruct, and training all 12 DEFT models is still cheaper than training the Tk-instruct model, using roughly 42,000 examples overall, roughly 56% of the data used to train Tk-instruct.
when using large pool of multitask data without instructions or
E RETRIEVED EXAMPLES
For a single query from each dataset, we present the top two closest datapoints retrieved below. Con- tent warning: some of these datapoints reference sensitive topics. Queries chosen randomly. Answers are in italics.
16

---

Under review as a conference paper at ICLR 2023
Evaluation Category
Task
Answerability
(10x0000mmcaace
Cause Effect Classification
Coreference
taax:33Saaaioonnnl...eereeeeeees tax359..2P..ataatttteett
Data to Text
Dialogue Act Recognition
Entailment
Las:9977.fffesibbeeeeaaaaaaaaldd antallaaaennnnnnnnneeinel Iment
Grammar Error Correction
Keyword Tagging
Las:6l33llaackkeyoorddt.ggg... task039
Overlup
Question Rewriting
Title Generation
taski
Word Analogy
taski task1 2ask11155.barrd.worddaaaaa.
Table 7: List of tasks used for each evaluation category given Table
RTE
Query: Thanks to a global ban on the ivory trade that was passed in 1989 by the Convention on International Trade in Endangered Species of Wild Fauna and Flora (CITES), the African elephant population may be reversing its spiral toward extinction),n Question: Does this imply that "The ban on ivory trade has been effective in protecting the elephant from extinction."" Yes no?
Retrieved #1: Title Dissappointed n Review: The software works OK, but haven't gotten any more than three numbers on a draw six lottery after 8 months of trying. The biggest thing to watch for is support, lack of. If you rebuild your computer or buy new one and have re-install their software, you have to get another product ID from them. It took me almost two weeks of begging and phone call (just an answering machine on their end) to get response from them. am coming up week of trying to get response from them for product ID for my new computer. Funny, because they responded the next day when first baught the program and they had my money in hand! i Does this product review convey negative positive sentiment? Negative Retrieved #2: You are considering whether to buy a product. You look at the reviews. Would the following review decrease or increase the chances of you buying the product? Review title: Amazon Rip Off Product review: What a huge waste of money. paid $$$ on this very site not but month ago, it is $S. Got it home, followed the instructions and the silly thing will not get but about foot off the ground if that, and then it just falls over and beats itself into the ground. Don't waste your cash this, give your kid fifty dollar bill and let them light it fire, they') IIl have for fun. decrease
ANLI RI
Query: Secrets of the Cryptkeeper's Haunted House was childrens Saturday-morning game show that ran on CBS. It premiered on September 14, 1996 and lasted until August 23, 1997. It featured the Cryptkeeper of "Tales from the Crypt" (with John Kassir as the voice) now serving as an announcer. It is the last TV series in the "Tales From the Crypt" franchise. \n Question: The Secrets of the Crypt Keepers House television show aired on CBS until 1997, and then was picked up and aired on NBC for an additional season. True, False, or Neither?
Retrieved #1: Is there negative or positive tone to this product review? ==)n Title: Not quite as good as some others in Review: This is a fair book, but it is not near as good as Peter O. Steiner's "Thursday Night Poker." Andy Nelson's book can't decide whether it is for beginners or advanced, so tries to fit advanced technique into too short of space. It barely scratches the surface of any of the topics it brings up. When it doesn't do that, it simply says, "Play tight that you don't even have to think. Fold 99% of your hands." " That does not make for a fun night, in my opinion. (n Answer: Negative
Retrieved #2: a delegation from the islamic resistance movement -Irb- hamas -rrb- left the gaza strip monday morning, heading for egypt hear israel 's response regarding a cairo mediated ceasefire. In nutshell, hamas leaders leave to cairo for final ceasefire discussions
17

---

Under review as a conference paper at ICLR 2023
ANLI R2
Query: The Sea Wall (French: Un barrage contre le Pacifique ) is a 2008 film by Cambodian director Rithy Panh in a TrenchhCamboddian/Beliiia co-production. The film opened on January 2009 in France. It was adapted from the 1950 novel "The Sea Wall" by Marguerite Duras. The novel had previously been adapted as "This Angry Age" by René Clément in 1958. \n Question: Marguerite Duras directed the film. True, False, or Neither?
Exactly what I had been looking for!'n Review: I've gone through two other iPod FM transmitters that ended up giving away because the quality was less than desirable. After seeing this one pop up in my Quick Picks last week decided to give it try. used it the very first evening I received it and I'm happy to say my search is over. As others noted, use a low FM frequency for the best results (87.9 in my area works well). don't 't receive any interference and the music on my iPod comes through just like I expected. For the price, this is definitely the best deal out there. in Is this product review negative? No
Retrieved #1: Title:
Retrieved #2: Based on this review, would the user recommend this ponduct?(n===iil) Review: My friend tried to commit suicide, and while he was bleeding to death, he was watching mtv, and the video for "Hold On" was playing, and he was like "yeah" and after he was done rocking out he got all inspired and called for an ambulance. And now he's still here, and he takes pills that make him tired, and everyone is careful to be very nice to him and be his best friend, even though all secretly pity him. Thank you so much. n Answer: No
ANLI R3
Query: Well, I think during the campaign, particularly now during this difficult period, we ought to be speaking with one voice, and appreciate the way the administration has worked hard to calm the tensions. Like the vice president, I call on Chairman Arafat to have his people pull back to make the peace. \n Question: Chairman Arafat needs to pull back his people during this difficult time. True, False, or Neither?
Retrieved #1: Title: clinton pushes for greater diversity on wall street' nn-=- nnn Write an article with the given title: u.s. president bill clinton urged wall street brokers to pursue business in america 's economically distressed cities, saying it 's untapped market with more buying power than mexico.
Retrieved #2: You considering whether to buy product. You look at the reviews. Would the following review decrease or increase the chances of you buying the product? n Review title: Mistake',n n Product review: didn't 't want to "purchase" Bars and Tones". It was mistake to click on it. This review doesn't It deserve so many words. n decrease
WiC
Query: It may rain in which case the picnic will be canceled. in A window case. \n Question: Is the word 'case' used in the same sense in the two sentences above? Yes, No?
Retrieved #1: Title: remains of exhumed from mass graves in eastern croatial,n \n=== n An Write an article with the given title: thirty bodies believed be croats killed by ethnic serbs at the outbreak of the serbo-croatian war in former yugoslavia have been exhumed from two mass graves in eastern croatia, an official said tuesday.
Retrieved #2: You are considering whether to buy a product. You look at the reviews. Would the following review decrease or increase the chances of you buying the product?),n n Review title: For the 50-cent table',n Product review: My favorite author has run out of steam! His co-author does not, repete, does not have the Paterson style. After sampling this "tandemly" "eriiell"-wwitten wrriten book. becomes obvious that this is time-waster Even the editing is bad. I didn't feel guilty about not finishing it. It's headed for the community library's 's monthly book sale- sale-fifty cent table. n decrease
18

---

Under review as a conference paper at ICLR 2023
COPA
Query: The woman filed restraining order against the man. As consequence... \m Help me pick the more plausible option:'n- The man called her. in- The man stalked her.
Retrieved #1: First sentence of the article: when christopher darden got a recent early-morning call from his publisher that his book in contempt had become no. on the new york times best-seller list he mumbled something like ok then rolled over and went back to sleep Ann Title: contempl does n't fill christopher darden Retrieved #2: "Extract the answer to the following question from the movie plot. If the question isn't answerable, please output "Can't t answer". n Question: Who is the toy's leader and Andy's s favorite toy?',n Title: Toy Story',n Movie plot: A boy called Andy Davis (voice: John Morris) uses his toys to act out bank robbery. The bank (voice: Don Rickles) assisted by Slinky Dog (voice: Jim Varney,, and the bystanders include Bo Peep (voice: Annie Potts) and her sheep. The day is saved by cowboy doll Woody (voice: Tom Hanks) playing the sheriff, with help from Rex the dinosaur (voice: Wallace Shawn). Woody is the only toy who gets to say his own lines because he has a pull-string that makes him say things like "Reach for the sky!" and "You're my favorite deputy!"Durin Randy Newman's 's "You've Got a Friend in Me"), Andy takes Woody downstairs to find his mother (voice: Laurie Metcalf) decorating the dining room for his birthday party. He asks if they the decorations up until they move, and his mom agrees. She says the guests will arrive soon and sends him back upstairs to get his baby sister Molly (voice: Hannah Unkrich), whose crib is in his room. Andy tosses Woody onto his bed before he pulls Molly out of her crib and carries her away. Woody and the other toys have seemed limp and inanimate up to this point, but as soon as Andy leaves the room, Woody sits up and expresses surprise that the birthday party is today. He calls "Ok, everybody, the coast clear," and the other toys come to life too. Woody calls a staff meeting and tells Slinky Dog to spread the word. Within few minutes (during which Bo Peep makes a date with Woody for that evening), all the toys are assembled. Woody starts by reminding them all to find moving buddy house, which will happen in a week. Then he tries to downplay the news that Andy's 's birthday party is happening today, but it causes commotion as the toys know that Andy's s actual birthday isn't 't till next week. Rex worries that someone will give Andy another dinosaur, and many of the toys have similar concerns. Woody points out that it makes sense have the party... \n Woody
cardboard box, the robber is Mr. Potato Head
the opening credits (soundtrack:
leave
they don't get lost when the Davis family moves to their new
WSC
Query: Passage: Dan took the rear seat while Bill claimed the front because his "Dibs!" was quicker. \n Question: In the passage above, does the pronoun "his" refer to Dan?' Answer:
Retrieved #1: Title: I want to READ it on my Kindle'n Review: Why can't get the read- able version of night for my kindle? I don't 't want the auidio version...Help! downloaded it thinking that I would have the choice to read it or to listen to it but that was not the at all. I I'm extremely disappointed. ( Does this product review convey a negative or positive sentiment? Negative Retrieved #2: You considering whether to buy product. You look at the reviews. Would the following review decrease or increase the chances of you buying the product?' n Review title: Look weird- feel great! n Product review: These look weird and also feel weird when you first put them on but they are so much fun. I love them for my yoga class, and sometimes wear them at night watching TV because the separation they give your toes is good for your feet overall. Try them... you'll Il become fan too! increase
19

---

Under review as a conference paper at ICLR 2023
WinoGrande
Query: The phone of Donald is a lot better than Adam's because paid extra for his phone. An What does the in the above sentence refer to? Donald or Adam?
Retrieved #1: Title: more than you expect',n n Product review: The thing about these tillers that they do things you might not think about. For instance, they're great for dealing with long-rooted weeds. You can hack your way down to the root, then pull up the plant and not leave huge hole in the ground.),n \n Would you say this review depicts the product in flattering or unflattering light?', n flatering
Retrieved #2: Title: purported statement from al-qaida- linked group says ultimatum against italy ends threatens attacks braacksnniiue (n(n==)ninn Write an article with the given title: a statement released sunday in the name of an al-qaida-linke -iinke group said the itaiian government has " dug its grave by its own hands" after it ignored warning to withdraw its troops from iraq by aug.
HellaSwag
Query: Complete the description with an appropriate ending: n First, [header] How to make a butterfly out of plastic spoons [title] Gather the materials you will need for this project, listed below. [title] Put craft cloth or some newspaper down on your working surface. [title] Cut the top portion of the four spoons off (leaving about half an inch of the handle left. Then, ...) [step] ) the top portion is the flat spoon part. [title] Using the cut-off handle of one of the spoons, make that the body" of the butterfly. (n(b) [step] As the buttons stretch they will process the solution. [title] Put one of the red plastic buttons (left over for the butterfly) on top of the red plastic spoon. \n(c) [step] And when done, you should have 8 small, square, circular spoons. [title] Use these four spoons as either mirror, or butterfly. (nd)) [step] [title] Thread thin needle into cast iron ring with four ends attached. [step] Thread the needle to the ring, and then sew the top portion onto the handle of the spoon at the top, or the side which should be facing away from the spoon.
Retrieved #1: Title: hmm... I Review: I bought this costume in hopes of wearing for Hal- loween ( last year). I had even separately purchased the duster (which I am now using to really dust things). Uhh... I tried it on got X-Small) and its just big... the net piece ( part of the dress with the dots) go all the way down to almost my knees. Which makes it awkward and not sexy at all- its just weird I tried tucking the net part in to my undies to hold it, but it just becomes supper puffy- again looks weird. I never wore it and its still brand new sitting in my closet just for my body- I am not sure, but the material isn't as great either compared to the picture, Def. does not look anything close to how the model looks in Sorry- this was not a good buy at all. The model sure looks good in it. in Does this product review convey a negative or positive sentiment? Negative
waaeeweere.aabbee its
Retrieved #2: What type of details about adolf heeb n can be gathered from the following bio? n Bio: adolf heeb -Irb- born 11 july 1940 -rrb- is a former eyclist and politician from liechtenstein An he competed in the individual road race at the 1960 summer olympics An he later served as a member of the landtag of liechtenstein and leader of the patriotic union party An /niviy updated in'n'nn n n nnyyn' nn'nn- n- birth date (r'yn' Mnn'n'ye yn name vn''n''''n birth place \n
20

---

Under review as a conference paper at ICLR 2023
CB
Query: B: boy, he's a big one. A: he's pretty big. That's why it really surprises me, you know, that he hasn't come back, because, like said, he's never gone away like this before, and, I would think, you know, mean, he might could get hurt by a car or something. I don't know that he could really get killed that easily because he so big. An Question: he could really get killed that easily True, False, or Neither?
Retrieved #1: Summarize this document: Glen Water Limited also paid costs of |u00031,6600 to restore fish stocks in the Tall River near Richhill. \n About 250 metres of the river was affected when untreated sewage was discharged into it. \n It caused what was described as a moderatefish kill'nn Inspectors found plume of untreated sewage coming from discharge pipe at Richhill waste water treatment works in 2014. In An investigation found that an üninterruptable power sourceät the plant had failed. (n In addition, power cut the alarm system meant staff were unaware of the problem. |n Glen Water Limited is based at Dartford in Kent. \n Under a 25-year public private partnership it has the contract for 25% of Northern Ireland's waste water treatment capacity.operates and maintains nine treatment works or pumping stations to 2032 in return for monthly payments. in Summary: A company which treats sewage for NI Water under public private partnership contract has been fined \ ,000a32.500 for polluting County Armagh river. Retrieved #2: Title: Good n Review: Well, I'd say all of these songs are well constructed, dope lyrics whatever... but wth? all the basslines sound the same or what? Personally prefer Violent By Design over this. in this product review negative? No
StoryCloze
Query: Andy had always wanted big kids bike. When he turned six Year's old he asked for bike for his birthday. He did not know how to ride a bike. On Andy's birthday his mother gave him a bike. What is possible continuation for the story given the following options ? Andy cried for h His dad taught him how to ride it.
Retrieved #1: Based on this review, would the user recommend this product?) ==)n Re-
view: love most Neil Young but every fan knows that about one three of his albums really sucks. After Greendale and Greatest hits, I'm very disapointed. (n Answer: No
Retrieved #2: hong kong share prices rose a mere percent on late overseas buying thursday despite early profit-taking dealers said An \n ===)n \n Given the above sentence, write its title: hong kong shares close. percent firmer
21

---

Under review as a conference paper at ICLR 2023
CaseHOLD
What is the correct holding statement for the following text?'n Text: component of the res judicata doctrine. The Ohio Supreme Court held that the original criminal proceedings in Krahn were insufficient to invoke collateral estoppel in the later malpractice case because the claimed error by Krahn's criminal lawyer in plea negotiations was not 'actually and necessarily litigated and determined in the denial of her motion to vacate the criminal judgment against her." Krahn, 43 Ohio St.3d at 108, 538 N.E. 2d 1058, quoting Goodson v. McDonough Power Equip.. Inc. (1983), 2 Ohio St.3d 193, 195, 2 OBR 732, 443 N.E.2d 978. The Supreme Court by no means suggested that collateral estoppel was completely inapplicable in the context of a criminal conviction when, as here, matters genuinely were litigated and determined. Id. at 107, 538 N.E.2d 1058 (JHOLDING;). Decisions in Ohio other than Krahn relative \n (A): recognizing the doctrine of collateral estoppel in agency proceedings\p in (B): holding that the facts prevent the invocation of collateral estoppel as bar to krahns cause of action in this case (C): holding collateral estoppel elements met considering changed circumstances in the context of an exception to the general rule of collateral estoppell,n n (D): recognizing the cause of action' n (E): holding that collateral estoppel applies to 1983 claims
Query:
Retrieved #1: Is there a negative or positive tone to this product review?' ==)n Title: Too steep',n Review: I bought this for my dog who had back problems, it was way too steep and my dog had to jump about 3/4's of the way up to my bed because the measurement of the ramp on the description was incorrect. It totally defeated the purpose of my dog having to not jump. I had to go back to the stairs had been using'n nn Answer: Negative
Retrieved #2: Write a title for this sentence: the fate of president barack obama 's top domestic priority remake of the u.s. health care system now rests in the hands of pivotal but deeply divided senate committee.. 'n \n Title: loughest test coming up for health care overhaul
DROP
Query: Passage: Coming off their overtime win at San Diego, the Broncos traveled to the Mall of America Field at the Hubert H. Humphrey Metrodome for interconference duel with the Minnesota Vikings. The game's first points came from the Vikings, when defensive end Jared Allen tackled running back Willis McGahee in the end zone for safety. The Broncos grabbed the lead when linebacker Mario Haggan returned an interception off Vikings' quarterback Christian Ponder 16 yards for a touchdown. Vikings' kicker Ryan Longwell made a 40-yard field goal, then the Vikings reclaimed the lead in the second quarter, on 19 1arydrd touchdown pass from Ponder to tight end Kyle Rudolph, followed by another field goal by Longwell, this time from 25 yards, just before halftime. The Broncos narrowed Minnesota's 's lead in the third quarter, with 21 21-yard touchdown pass from quarterback Tim Tebow to wide receiver Demaryius Thomas, but the Vikings countered with 52-yard touchdown pass from Ponder to wide receiver Percy Harvin. The Broncos responded with a 41-yard touchdown pass from Tebow to Thomas. Ponder then threw a 48-yard touchdown pass to Harvin to give Minnesota a 29-21 lead. On the Broncos' next possession, McGahee rushed 24 yards for a touchdown and Tebow scrambled for a two-point conversion to tie the game at 29. The Vikings subsequently reclaimed the lead on Longwell's 39-yard field goal with 3:06 left in the game. The Broncos answered with kicker Matt Prater's 's 46-yard field goal with 1:33 left to tie the game at 32. On the Vikings' ensuing possession, Broncos' cornerback Andr233; Goodman returned an interception off Ponder to the Vikings' 15-yard line. Six plays later, Prater nailed the game-winning winning 23-yard field goal as time expired to give the Broncos their fifth consecutive win. \n Question: how many yards did longwell make? n Answer:
Retrieved #1: Make title for this article: andy roddick hit record-breaking mph -Irb- kph -rrb- serve friday in a lopsided win over stefan koubek as the united states took a davis cup lead over austria in \n roddick ginepri give united states lead over austria
Retrieved #2: Orton does not start against Ohio State Purdue quarterback Kyle Orton did not start Saturday 39:s game against Ohio State, though he was listed available play. Orton has been bothered by a right hip injury for the last month. \n |n Which of the following sections of newspaper would this article likely appear in? World News, Sports, Business, or Science and Technology? Sports
22

---

Under review as a conference paper at ICLR 2023
Qasper
Query: Question: How big is Augmented LibriSpeech dataset? Paragraph: We introduce multilingual speech-to-text translation corpus, CoVoST, for 11 languages into English, diversified with over 11,000 speakers and over 60 accents. also provide baseline results, including, to our knowledge, the first end-to-end many-to-one multilingual model for spoken language translation. CoVoST is free to use with a CCO license, and the additional Tatoeba evaluation samples are also CC-licensed. Is the answer the question in the paragraph? Answer Yes or No.
Write article with the given title: you have less than a week to get your fourth of july cookout menu set and we thought 'd help
Retrieved #1: Title: make your july celebration sizzle) n I= An
Retrieved #2: Title: A good idea... i Review: that went terribly bad. I cannot comprehend how some of these "artists" were chosen for this. "Atlantic City" and "State Trooper" are embarrasing to say the least, but they sadly showcase what is now Nashville's finest. If Johnny Cash and Dar Williams recordings had not appeared on this CD, one star would have been too many. Thankfully, these mostly pathetic renderings cannot tarnish the greatness of Mr. Springsteen or his amazing album. Go get the original. You won't 't be sorry. n Does this product review convey a negative or positive sentiment? Negative
23


Reviews:
[{"review_id": "w5AZUQoMGa", "reviewer": "Reviewer_zbKf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a method for selecting subsets of prompted multitask training data for fine-tuning  a language model (Data-Efficient FineTuning, or DEFT). Given a small set of unlabeled task examples, the paper uses nearest neighbors to retrieve a targeted task subset from a larger pool of prompted examples. Using this method to build a more data-efficient training set for fine-tuning, the paper reports improvements using a T5 model an equiv. 3B T0 model (outperforming in 11/12 tasks) while using <= 2% of the same P3 training data. Further improvements are reported for few-shot fine-tuning. ", "summary_of_the_review": "I think this is a reasonable empirical paper, though I have concerns about how well results would generalize outside of P3-style prompts (which tend to be shorter than other instruction tuning type datasets). This combined with the extreme simplicity of the method undercuts some of my enthusiasm for the acceptance, though I think the empirical work is generally solid.", "strength_and_weaknesses": "Strengths\n- Optimizing fine-tuning for improved data efficiency (or more targeted task improvement) is an important research area.\n- Nice connections to retrieval augmented language models with corresponding retrieval experiments.\n\nWeaknesses\n- The approach is extremely simple. This could also be a strength, but I wonder how much using only a single source (P3) of prompts is biasing results, since many templates used for creating that dataset are very similar across tasks. Expanding results across prompted datasets would be a good series of additional experiments. \n- Given the hypotheses around the the role of negative interference in hurting performance (and the resulting benefit of querying more similar examples for fine-tuning), the paper would be much stronger if there were direct experiments exploring this\n- Some of the model naming conventions are confusing/inconsistent (DEFT-T5-base vs DEFT-base). It wasn't clear to me what DEFT-base was defined as (vs XL which was the 3B param variants).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "q4amroMHGu5", "reviewer": "Reviewer_g3S4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a simple data-efficient fine-tuning technique, where the few-shot data for a test task is augmented by retrieving similar examples from a multi-task training set (which does not contain the exact test task). The test task training set is augmented by taking the union of the training examples retrieved for each example of the test task. The authors operate in a setting where only unlabeled data for a test task is available to do retrieval and a setting where also labeled data is available. The method demonstrates gains on the test datasets used to evaluate T0, when compared to multi-task baselines that are either applied zero-shot to test tasks (e.g., T0) or are fine-tuned on the few-shot test task data (e.g., T-Few).\n", "summary_of_the_review": "This paper proposes a simple and effective idea for improving zero-shot / few-shot task generalization. The reported results are convincing. Unfortunately, the overall contribution of this paper is diminished by the limited novelty with respect to ReCross (to appear at NeurIPS, 2022, and published on arXiv in April). The authors cite this paper in the related work, but I feel that the similarity is quite understated. It is questionable whether a paper published in April should be considered related work or not and whether this should impact my review. In the current context, I would suggest the authors to do a small extra effort to 1) highlight similarities with ReCross and 2) differentiate their contribution w.r.t. ReCross, for example, by reporting ReCross performance and ablating some of the ReCross components (e.g. the reranker and the fact that ReCross uses the multi-task model as base model). Right now, the paper appears to be written as if it was the first in proposing this retrieval idea. I didn't discover the similarity until I read the ReCross work in details. All in all, I find the idea exciting: the strong similarity to a previous work, despite this being quite recently published online, requires the paper to offer new perspectives that are not present just yet; experimental setting could be made stronger by adding an additional dataset (e.g. NIv2), to show more convincingly which aspects of the method are responsible for improved performance.", "strength_and_weaknesses": "Strengths:\n- *The approach is simple and effective*: The main idea is simple and effective - perform a \"task-attention\" on a set of training tasks to augment the test task few-shot data. It is quite interesting to see that a naive example retrieval approach such as taking the union of the retrieved training examples can lead to significant gains (64.6 vs 60.4 on T0 test tasks for the few-shot setting).\n\n- *The paper is clear and well-written*\n\nWeaknesses:\n\n- *Limited novelty w.r.t. related work*: One of the main weaknesses of this paper is the close similarity to ReCross (Li et al., 2022) to appear in NeurIPS 2022, and published on ArXiv last April. ReCross introduces the same idea: retrieving from a set of upstream tasks training data to benefit the downstream task data. The authors acknowledge this similarity in the related works, by stating that their approach differ from ReCross because ReCross \"*make use of an additional re-ranker and multi-task trained model to retrieve useful instances.*\". Per se, not using a modeling choice of a previously published paper is *not* a significant novelty: the novelty and contribution would come from showing that the modeling choice used in ReCross *is not necessary* or even *harmful* for achieving good performance in a different setting. These conclusions are currently missing from the paper. Given the good scores the paper reports on the T0 suite, I can imagine that the reranker might be only marginally useful, but this should be clearly stated and empirically supported. The community goes fast and probably the authors started their work before having noticed the relevant related work. Nevertheless, at the moment, this paper presents the idea as totally new. It would be more suitable, for example, to acknowledge the similarity to Li et. al, 2022 directly in the intro and maybe make an extra effort to offer a simplification/ablation of some components of the aforementioned approach. Some of the questions I can think of that would give a bit of novelty and require a small additional effort are:\n1) Is the reranker used in ReCross useful?\n2) What happens if you fine-tune a multi-task trained model (T0-3B in your case, similar to ReCross), rather than a \"generalist\" model (T5-XL-LM)?\n\n- *Experimental setting can be stronger*: Differently from ReCross, the authors only test on the T0 evaluation suite and build heavily on top of IA3 (Liu et al, 2022). I do believe that another benchmark is necessary. It would be interesting for example to explore the proposed method with Natural Instructions V2 dataset. For example, this would give a better idea if the method works due to the nature of the prompts format in P3 (e.g . it retrieves examples with similar prompts) or due to other mechanisms.\n\n- *Claims about task interference*: The authors claim that their result show that there is negative task interference. I would think that an important experiment to substantiate this claim is to report the results for 2) above, where the method is applied on top of a multi-task model trained on the multi-task training set.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "gFlEAg2S0ot", "reviewer": "Reviewer_SJym", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is interested in building language models that can generalize to new tasks when there is only unlabeled data for that task. The authors introduce a method called DEFT, where unlabeled instances for the task at hand are used to retrieve the most similar labeled prompted data in a pool of prompted instances of NLP tasks (such as the one used to train T0). The retrieved labeled prompted instances are then used to fine-tune an encoder-decoder language model (more specifically T5 LM), and the model is evaluated on the target task. Experiments show that the resulting DEFT model outperforms T0 at the same model size by a strong margin. When a few labeled data is available for the target task, the DEFT models provide a better initialization for few-shot fine-tuning than T0.\n\nThe authors use these data points to highlight that only a subset of the pool of prompted instances is required to achieve high performance on the target task and to highlight negative transfer between tasks when doing massively prompted fine-tuning (i.e. FLAN/T0).\n", "summary_of_the_review": "Strong paper with well-executed experiments and a somehow novel story. Numbers are relatively strong.\nSome minor doubts and questions about the experiments.\nRecommend accept.", "strength_and_weaknesses": "Strengths:\n- The problem is well posed, the method is clearly explained while being relatively novel. \n- Numbers are strong and experiments well conducted and results are interesting.\n\nWeaknesses:\n- I have a few doubts about some of the stronger claims in the paper (in particular the negative transfer argument, see questions).\n- Few-shot fine-tuning has a long history in NLP. I’d like to see a fairer comparison to stronger baselines. IA3 is particular in the sense that it only fine-tunes a fraction of the parameters. Using a parameter efficient fine-tuning baseline needs a better justification (especially since DEFT-Few seems to be fully fine-tuned).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1TBCiR-ZXW", "reviewer": "Reviewer_fvhK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new approach to multitask learning by retrieving relevant training examples from P3 and train the model on it. ", "summary_of_the_review": "This paper explores a simple yet effective way to facilitate multitask data but may lack novelty and practicality.", "strength_and_weaknesses": "### Strength\n1. This paper proposes a simple yet effective idea on how to use existing resources (i.e., P3) to facilitate unseen tasks.\n2. The paper is well-written and easy to follow.\n3. The evaluation is good - it considers both 0-shot and few-shot settings.\n\n### Weaknesses\n1. The idea is not completely novel. Similar approaches exist like kNN-LM. Also, the proposed approach is also related to Mixture-of-Experts.\n2. It is unknown if the task retrieval mechanism in the paper is optimal. For example, what if we use BM25 for retrieval? Also is task embedding relevant here?\n3. The proposed method has its drawback. It requires retrieval and training for each task. In other words, for 3 tasks there are 3 models trained instead of the open-box T0. In Table 1, DEFT outperforms T0 but the improvement is somewhat limited. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject