Unsupervised Cross-Task Generalization via Retrieval Augmentation
Authors: Bill Yuchen Lin
Kangmin Tan
Chris Scott Miller
Beiwen Tian
Xiang Ren
Venue: NeurIPS 2022 Accept (2022)

Abstract: Humans can perform unseen tasks by recalling relevant skills acquired previously and then generalizing them to the target tasks, even if there is no supervision at all. In this paper, we aim to improve this kind of cross-task generalization ability of massive multi-task language models, such as T0 and FLAN, in an unsupervised setting. We propose a retrieval-augmentation method named ReCross that takes a few unlabelled examples as queries to retrieve a small subset of upstream data and uses them to update the multi-task model for better generalization. ReCross is a straightforward yet effective retrieval method that combines both efficient dense retrieval and effective pair-wise reranking. Our results and analysis show that it significantly outperforms both non-retrieval methods and other baseline methods.

Unsupervised Cross-Task Generalization via Retrieval Augmentation
Bill Yuchen Lint
Kangmin Tan' Chris Miller! Beiwen Tian' Xiang Ren
University of Southern California
Tsinghua University
(yuchen. lin, kangmint, mint,,miiin millerrss xiangren| @usc. edu
Abstract
Humans can perform unseen tasks by recalling relevant skills acquired previously and then generalizing them to the target tasks, even if there is supervision at all. In this paper, we aim to improve this kind of cross- task generalization ability of massive multi-task language models, such TO and FLAN, in an unsupervised setting. We propose a vetrieval-augmentatin method named ReCross that takes a few unlabelled examples as queries to retrieve small subset of upstream data and uses them to update the multi-task model for better generalization. ReCross is a straightforward yet effective retrieval method that combines both efficient dense retrieval and effective pair-wise reranking. Our results and analysis show that it significantly outperforms both non-retrieval methods and other baseline methods.
1 Introduction
Advances in pre-training techniques for large language models (LMs) have considerably improved natural language processing (NLP) models on various important tasks via fine-tuning with labeled data. While these fine-tuned models are impressive in their target tasks, they can hardly generalize to unseen tasks. This thus makes it difficult to approach the general linguistic intelligence that we ultimately want an NLP model to enjoy. A promising avenue is to train a massively multi-task model that learns a large of NLP tasks. However, in real-world applications, multi-task NLP model can also perform unseen tasks that they are interested in. These users may only be able to provide few unlabeled examples (i.e., the input-only data) of the target tasks with langual instructions. How can we generalize the multi-task model to unseen tasks without labels? This desirable ability is dubbed "unsupervised cross-task generalization."
often expect
Recent studies show that multi-task prompted training makes language models better in cross-task generalization, especially when natural-language instructions are used for formatting the training data (Ye et al., 2021; Sanh et al., 2021; Wei et al., 2021). The general recipe is to first fine-tune a text- to-text language model such as T5 (Raffel et al., 2020) formats. We use the term upstream learning to refer to
multi-task mixture of diverse NLP datasets
that are converted to seuuence-to-sequenee this multi task training stage. Given target task that is unseen during upstream learning, want the upstream multi-tase task model to also perform well on it via reusing the previously acquired knowledge. FLAN (Wei et al., 2021) and TO (Sanh et al., 2021) both use natural language (NL) instructions as prompts to reformat the data of various NL.P tasks for upstream learning and generalization. Their results suggest that NI instructions keys unsupervised cross-task generalization.
Despite of the exciting results from Wei et al. (2021) and Sanh et al. (2021), their studies are limited to the analysis of the task generalization performance of the frozen, turget-agnostic upstream models (i.e., FLAN and TO). We argue that the generalization performance can be further improved if we can exploit the unlabeled data of target tasks as hints for adjusting the upstream model to a more
Our data, code, and supplementary materials are https: // /t///innaabb./a///./ usc. edu /
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

upstream tasks
multi-task NLP Model
Unsupervised Cross-Task Generalization
U an unseen task w/ a few unlabeled data
Retrieval Augmentation
Figure I:
The unsupervised cross-task generalization problem. In the upstream training stage, we train a multi-task NLP model, .M, with diverse collection of upstream tasks. In the generalization stage, given unseen task U1 with few unlabeled examples Q. we want to update the upstream model (via retrieval augmentation) such that it can generalize to the target task.
dedicated, target-aware model. Intuitively, the upstream examples that share similar skills with the target task should help the task generalization if the upstream model could recap these skills via retrieving, Motivated by this idea, we propose to further improve the cross-task generalization ability of upstream models via retrieval augmentation from the upstream data.
The key challenge of such retrieval augmentation is to predict the example-Iewe utility for cross-task generalization, which introduce with details in Sec. 2. To address the challenges, we present two-stage
framework, ReCross, for unsupervised cross-task generalization pre-compute dense index by encoding all upstream data dense vectors. Given a set of unlabeled examples, we first use them to retrieve an initial set of upstream data by using encoded queries to efficiently search over the dense index. Then, we apply the reranking module to carefully analyze the utility of each candidate example. To get such reranker, we fine- tune cross- eessser model with distant supervision mined by novel algorithm. Finally, we take lop-ranking retrieved data to fine-tune the upstream model for a few steps and use this updated model for inference on the target task the future (i.e., the retrieval augmentation and model update is one-time procedure for each unseen task).
in Section 3. Specifically,
To more efficiently evaluate generalization methods without losing the generality, we train a variant of TO-like models, named BARTO, which has comparable performance with TO-3B yet is 8x smaller. Our extensive experiments show that the proposed ReCross outperforms the baseline methods by large margin. For example, ReCross improves the non-retrieval methods by 4 points on the overall performance 10 target tasks and similarly on few BigBench tasks. We also analyze the distribution of the retrieved data to understand the behavior of -arievaallagmmttttiion methods better and find that ReCross has very different distribution compared to semantic retrieval baselines.
2 Problem Formulation
Language Models. To build a general NLP model that can serve wide range of real-world downstream applications, it important to train a massively multi-task upstream model. We assume there are N different upstream tasks (e.g., sentiment analysis of IMDB reviews), dubbed as (TI,.... TN): }. We use D denote the collection of all labeled data for these upstream tasks (i.e., the upstream data), which are then used for training a massive multi-task model M (e.g., BART, T5, and other Transformer-based models). The datasets of these upstream tasks are all converted to a shared text-to-fext format using natural-language instruction templates such as PromptSource (Bach et al., 2022) to reformat data of different NI.P tasks. This pipeline has become a common approach, adopted by several recent massive multi-task models for NLP, such as TO (Sanh et al., 2021), FLAN (Wei et al., 2021), and CrossFit (Ye et al., 2021).
Massively Multi-Task
Unsupervised Cross-Tase Generalization. In real-world scenarios, it is very common that users to want a general multi-task model to perform tasks of their interest, even if their target tasks are never seen before by the upstream model. For these unseen target tasks, users usually can provide only few unlabeled examples (i.e., the input-only data) of them for specifying the task instructions. This the reason why need to study how to generalize multi-task LM unseen tasks with only few

---

Qry Examples of
R
Qi
R'
li copy
Reranker
topk
Encoders
learn
Final
Dense Index
Retrieved
Data
MIPS
rank
Instance
Task Instances
pair-wise score
distant sup.
Embedding
D
Reranker
Encoders
Alg.
query
cand.
Figure 2: ReCross is retrieval-augmentatie method for unsupervised cross-task general- ization. We reuse the encoder layers of the upstream model (green) to build dense index, which consists of vectors of the upstream examples D. We also propose algorithm to generate distant supervision for training reranker, which takes pair of examples as input and outputs score. During the evaluation, we encode query examples Q; for querying the index to get initial ranking results and then pair them with the queries again for reranking. Finally, we take the top-K results (i.e., Rt) for generalizing the upstream model M to the unseen task U4.
unlabeled examples, i.e., unsupervised cross-task generalization. For instance, in Fig. 1, the unseen task U, is a coreference-resolltiin task that is not covered by the upstream training (the top-right box in Fig. 1). We have only a few inputs for it as the "hints" for cross-task generalization, which we call query examples Q; Our objective is
the query examples Q: enhance the performance of upstream model M on the unseen task U4 For evaluating such unsupervised cross- cross-task generalization methods, we test the enhanced model with held-out labeled data of each target task.
Challenges. Standard fine-tuning approaches (with or without meta-learning designs) for few-shot cross-task generalization are not feasible here. We have to adjust the upstream model based on only few input-only examples for the unseen task. Intuitively, upstream examples that share similar skills with the target task U4 should be more beneficial than other upstream data. Thus, naive idea is to first estimate the utility of each upstream example for 24 and then re-train dedicated model M,, via weighted learning method (c.g., examples of more utility trained with larger loss).
However, such a target-aware weighted re-training method cannot scale, because the upstream data is usually very large and there can be a large number of unseen tasks from users in real-world applications. In addition, it is particularly challenging estimate the utility scores of upstream data for given unseen task, we do not have ground-truth annotations for learning this. Although there are some existing studies on task-to-task relatedness and transferability (Vu al., 2020; Lange et al., 2021; Padmakumar et al., 2022), most of them are not designed for unsupervised settings and few are done with multi-task (prompted) upstream models. Moreover, these prior analyses are mainly limited to the task-level analysis and they may not directly generalize to studying example-level utility, which particularly important for the problem setup of this work.
3 ReCross: Retrieval Augmentation for Cross-Tash Generalization
3.1 Overview
To address the above challenges for unsupervised cross-task generalization, we propose retrieval- augmentation method named ReCross. The ReCross method also based on the simple idea that we should exploit the upstream examples that enjoy better utility for a given unseen target task. Instead

---

of costly re-training from scratch, our method first retrieves a small subset of the upstream data for each unseen task. It then uses them efficiently fine-tune the upstream model such that the updated model is generalized. This can ensure scalability to great extent and benefit upstream models from re-learning target-specific acquired knowledge for cross-task generalization.
Ideally, we aim to retrieve the upstream examples that are the most beneficial ones for generalizing the upstream model towards a particular unseen task ranking the upstream data by their example-level utility. To achieve this goal while preserving the efficiency, we first use the query examples to retrieve initial candidates via efficient maximum inner product search (MIPS) over a dense index, which consists of embedding vectors of all upstream examples (Section 3.2).
Based on the candidates from dense retrieval, we learn reranking module for further improving the retrieval results (Section 3.3). The reranker is based on the cross-encoder architecture that takes a query-candidate pair of examples and outputs a more curated score of utility. Recall that we do not have any annotation for such example-level utility scores, and the only allowed resources are the upstream data and model. Therefore, we propose an algorithm to mine distant supervision from the upstream data for learning the reranker (Section 3.4). The overview of ReCross is shown in Fig. 2.
3.2 Dense Retrieval
To efficiently estimate the example-Ievel utility for generalization, we propose to first employ a dense retrieval module that ensures high scalability. Specifically, we build matrix D E R xd where each upstream example in D is encoded with a dense vector. Based on this dense index, we can now estimate the utility of an upstream example with its cosine distances to the encoded query examples in Q. That is to say. the upstream examples that are the nearest neighbors of query examples, are more likely to be beneficial for generalizing the upstream model M to the unseen target task.
To retrieve the candidate set R', we use MIPS to search for the top- examples for each query example in Q, so K [ (We introduce the details and other aggregation strategies in Appendix.) This dense-retrieval process is very efficient as we pre-compute the upstream index and perform MIPS for querying the candidates over the index on-the-fly during the generalization stage. We use the FAISS library (Johnson et al., 2019) in our implementation.
Instance embeddings. The example encoder is a key component of the dense-retrieval pipeline. An ideal example encoder is supposed to represent the underlying skills behind an example such that we can use the distances in the result embedding space to estimate utility for cross-task generalization. As we do not have annotations of utility scores for training an encoder, one may want to use pre-trained sentence embedding models such as BeRT (Reimers and Gurevych, 2019). Our empirical results show that such semantics-based encoders cannot lead to much improvement over random retrieval results. We think there are two reasons for this failure. First, the semantic similarities between examples are not suitable for estimating the utility for generalization. Second, the external encoding modules do not reflect the nature of the upstream model which we want to generalize.
To address these two issues, we propose to use the encoding layers of upstream model M for computing the example embeddings. Without loss of generality, let assume M to be text-to-text Transformer that has multiple layers for both encoders and decoders such as BART. We encode an example by first obtaining the hidden representation of each token at the last encoder layer (i.e.. a sequence of token vectors), and then performing mcan-pooling over them to get single dense vector to represent this example. By doing this, the produced example embeddings reflect the internal features of the upstream model, which are more relevant to the "thinking process" of the upstream model for the examples instead of the shallow semantic information.
3.3 Reranking Module
Weakness of the dense retrieval. Although dense retrieval is very efficient thanks to the MIPS support, the retrieval performance is limited by its two major weakness. First, is a dual-encoder architecture that encodes the candidate example and the query example separately, which ignores informative features behind token-to-token attention across a pair of examples. Second, it too costly to frequently update the example encoder, which prevents us from learning to refine the retrieval results with distant supervision (if any). Therefore, we design a re-ranking stage where we train a cross-encoder to further enhance the dense-retrieval results with mined distant supervision (Sec. 3.4).

---

Encoding query-candidate pairs. The cross-encoder architecture has been widely used in sentence- pair classification tasks such as natural language inference and paraphrase detection. We here use a cross-encoder to encode the concatenation of a query example and a candidate example. Specifically, we fine-tune a RoBERTa (Liu et al., 2019) model to classify whether an example pair is positive or negative match. The confidence of classifying such pair to be positive can thus be used as the utility score of the candidate upstream example for this query example. On top of this, we then develop a reranking module for further improving retrieval performance as follows.
Scoring paired data. To re-rank the initially retrieved data by the dense retriever, we apply the cross- coss-encoder on all pairs of query examples Q and candidate retrieved examples R', producing scores of all IQ) * |RR" query-candidate pairs. For each candidate example r E R2, we use the average of all cross-encoder scores involving as its utility score. Finally, we take the top- K examples based on this new ranking of candidate examples in R' as the final retrieved data R. We use upsampling ratio jc to denote the ratio between R' and R, i... / = [
3.4 Mining Distant Supervision for Reranking
How do we train such a re-ranking module? Recall that we only have access to the upstream data D and must not use any data from the unseen tasks at this stage. Inspired by meta-learning works, we propose an algorithm (Alg. 1) mine distant supervision data for creating a training-at training-as-testinnn as-testinng environment for learning the reranker. Our key motivation is to examine the utility scores of candidate examples by assessing the generalization performance of updated models that are fine-tuned with these candidates as if we use them for real unseen tasks. Such more realistic estimation of utility scores can thus help us train a reranker to predict.
Algorithm 1: Distant Supervision Creation
Specifically, we define a data point of such dis-
tant supervision as tuple 2 (Zg. Zp, Zn): 1) Z, is a set of query examples of a particular task Ty: 2) Zyp is the set of positive examples from other tasks; 3) Zn, is the set of negative examples from other tasks. We expect that Z,, is of more utility for generalization than Z, if Zy would be query set for the target task Tg. To this end, we first randomly sample an upstream task Ty and use small subset of its training data as the Zy, Here, we also sample a larger held-out set H, examples of task T, to facilitate utility estimation. Then, we apply the dense re- triever using Z,, as the query examples and get the retrieval results Rz. This Rz is thus the can- didate pool where we create Zp and Zn. That is, Zp Rz and Z,, Rz. We discard examples that are from the TT. so that the generated tuples closer to the real scenarios where we use the reranker on the query sets of unseen tasks.
Input: M: D: T Output: Z = -(Z,, 2, Ln)
Dr, + fIr E D/x is an example of Taj
Zq + Sample( ): Ha + ST,
Rz f DenseRetrrive Zg D)
1 Delete retrieved examples
same task as queries. +/
Rz t Rz. discard(DT.)
foreach round do
R2- shuffle()
1e Split retrieved examples into groups */
(Gi, G.) t- Rz. spli((
foreach C; [GI,... G do
M + cM.cop)(
M' (ine_tune(G.)
L + --M'.calc, [ilc_los((() (H₂,)
foreach x E G, do scores (z). append (f)
18 Score each in the group w/ the loss. */
/ Use mean group score as score for single examples */
foreach x E R2 do L score(x] & mean(scoresss]))
Our criteria to select Zp and 2n from Rz is motivated by the hypothesis that a more suitable set of retrieved examples should improve the performance M on T, after fine-tuning with it. Therefore, we iteratively sample a small subset from Rz, then fine-tune M with it, and finally, use the fine-tuned model to evaluate on 2y. The performance of such temporarily fine-tuned model can be seen as the utility score- well this subset can help generalize M to the unseen task Tg- procedures, we can thus score each example in
Soet Rs by score in increasing order. #/
Rz. sort sort(key: (key: score, order: increasing)
2. + First W items of Rz
2. Last W items of R.z
Through multiple rounds of such cumple-train-ter Rz by taking the average of all test results where it is involved. With such a new ranking of examples in Rz, we take the best W examples as Z, and the worst W as Zn.
With such distant supervision, we then can create pair of query-positive instances and neery-ne instances via pairing Z, with Zp and Zn respectively. Now we can fine-tune RoBERTa-bees model
5

---

by concatenating each pair and learning a binary-clann caaaysycasiiiiationn objective. The output logits of this trained model will be used for the reranking procedure as shown in Sec. 3.3.
3.5 Re-learning via FiineTTuning with Retrieved Data
When we have the final retrieved data Rg for certain query set Qi can now enhance the upstream model M for the unseen task 24, We use a small learning rate to continually fine-tune M with the retrieved upstream examples R, for small number of steps. We find that the learning rate has to be very small so that this step can be seen as a natural continuation of the finished upstream training and avoid overfitting the retrieved data. We acknowledge that there could be more effective methods to reuse the query examples Q as guidance for fine-tuning, and we leave this as future work. Please find more discussion on the hyper-parameter selection and configuration in our appendix.
Evaluation
In this section, we first introduce the experimental setups, including the task distribution, upstream learning details, and the configurations of the main experiments. We present experimental results and reveal some non- con-trivial findings with extensive analysis that justify the effectiveness of ReCross.
4.1 Evaluating Unsupervised Cross-Task Generalization
We follow Sanh et al. (2021) to use the templates from PromptSource (Bach al., 2022) for converting data of different types of NLP tasks to text-to-text formats. In total, we have 36 upstream tasks and 10 target unseen tasks for our main experiments. The upstream tasks are the same the ones that the TO models used for upstream learning. We follow the evaluation protocol proposed by Sanh et al. (2021) and select the target tasks that are significantly different from the upstream tasks. Besides, also include 5 additional tasks from the BIG-bench project (Srivastava et al., 2022) to create an even more out-of-distribution set of unseen tasks for analysis.
Metric. When we apply the natural-language templates for the test examples, we only keep the templates that can be evaluated with an exact match (classification, question answering. answer selection, etc.) so that it is fcasible to exact-match for evaluating all tasks. To allow smoother grading, our metric also counts the cases when outputs and truths are sub-strings of each other, which we call SoftEM. The only difference between SoftEM and the standard EM is that it also counts the sub-string matches. We observe that sometimes even though TO-like models (including ours) answer the input questions correctly, their raw outputs not exactly the same as the truth outputs generated by the PromptSource templates. In particular, the ground truth outputs for multiple-choice QA tasks are often in the form of "[ "[A/B/C/D]: [answer]". while the models often only output the id of the correct choice (e.g., "A") or the text of the answer. We also find that the model can output some noise (such additional punctuation) after the answer (e.g., "True" vs "True.") "). The standard EM will discard such matches and cause inaccurate measurements. Although SoftEM might add false positives due to substring matches, we found is very rare according to our manual inspection of the 10 tasks. Therefore, we choose to use SoftEM for a more precise evaluation. We report the results with the standard EM Table that also supports our findings.
4.2 BARTO: Upstream Learning with a Smaller LM
The T0(p)) models are all very huge, and the smallest version, TO-3B (3 billion parameters), is still too large to be fine-tuned on popular affordable GPUs We need a parameter-efficien alternative that makes the study cross-task generalization more accessible to broader community while keeping the generality. Thus, we fine-tune a BART-large (Lewis et al., 2020a) (0.4 billion parameters) following the recipe of training TO. Specifically, we sample 50k examples at most from each upstream task to build a large upstream dataset consisting of 1.7 million examples (i.e., |D] = 1. 1.7m), and then fine-tune BART-large with 22k steps with this upstream dataset. Finally, we use the fine-tuned checkpoint as our upstream model M and name it BARTO. Surprisingly. we find that BARTO and TO-3B have comparable sero-shot performance on the unseen target tasks, even though TO-3B is about 8x larger than BARTO. More implementation details are shown in Appendix.
4.3 Setup and Configurations
In our main experiments, we use |Q,|= = query examples for each unseen task U4 and retrieve [R.] 512 examples for augmenting BART. In the fine-tuning stage, use a learning rate of 1e-6 and

---

Target Task
TO-3B
BART
Random
SBERT
ReCross
ReCross
A
anli r3
26.00
30.50
35.34 11.52
32.64 12.53
36,70 10.53
35.7610.30
5.26
h-swag
34.40
39.40
33.84-5.59
30.92+7.00 82
44.3643.07
47.28
7.88
cb
53.93
39.64
4..07
48.00
44.50+4.20
44.79+3.36
5.15
wic
45.70
46.70
21.04
46. 46.78.2.22
49.90+4.60
50.58
3.88
wsc
50.00
57.88
52.50.2.29
52.69:1.133
59.27 11.96
11.46::1.47
3.58
winogrande
47.60
51.10
52.68 00.83
52.18
54.60
55.46.0088
4.36
arc-chan.
41.30
35.70
33.28 1.50
37.90+1.22
37.78
38.4440.99
2.74
obqa 38.50
34,40
28. 28.72+2.46
33.28
36.98
39.58
5.18
piqa 45.30
36.10
37. =7.00
38.54
41.34 11.75
41.42
5.32
squadv2 30.60
32,40
29.86.55.46
29 29.4641084
30.26
30.58 1.61
-1.82
All mean
41.33
40.38
39.13 13.0
40.24
40.57
44.53+0.42
4.15
@ median 41.33
40.38
39.93
40.91
43,43
44.31
3.93
@ min 41.33
40.38
35.66
38.28
42.65
44.16
3.77
@ max 41.33
40.38
40.59
41.76
44.51
45.07
4.69
Table 1:
The main experimental results (%) for unsupervised cross-task generalization in SoftEM. Each result in the upper section is the average (and the std) performance of using 5 different query sets for a task. The lower section of this table reports the mean, max, min, and median of the overall performance (i.e., the average performance all tasks) of these five rounds.
batch size of to continually fine-tune all layers of BARTO for epochs. As for re-ranking, we set the upsampling ratio μ 2, meaning that first retrieve 1024 examples for reranking and use the top 512 ones as the final retrieved data. To obtain more convincing evaluation results, we average the scores of all target tasks to show the general zero-shot performance. For each task U4 use five
different query sets, (Q ..... !!55), ). to conduct five individual rounds of retrieval, thus resulting in five average scores for all tasks. To get comprehensive assessment, we report the mean, std, median, min, and max of these five overall scores in the lower part Table We present an ablation study on prpammerrretre configurations in Table and include more details in Appendix.
4.4 Experimental Results
BARTO vs TO-3B, As mentioned earlier, we find that BARTO is comparable with the much larger unseen tasks (41.33 vs 40.38). As use BARTO TO-3B in terms of their zero-shot performance augmentation methods, its overall performance 40.38 as our base model for testing different retrieval-augee what want setrevallauggeent for non-retrieval zero-shot inference, they do not use any information from the query examples, so their mean, median, min, and max are always the same.
methods to beat. Note that when using BARTO and TO-3B
Random Retrieval. The Random column shows the results when we randomly sample R2 from the upstream data D without using any information from Q
SBERT and ReCross We use SentenceBERT (SBERT) as a strong baseline method to create dense index of the upstream data, compared with our proposed indexing method, ReCross (i.e.. ReCross without reranking). We can see that ReCrossf always outperforms the other methods. Even its minimum performance the five rounds (42.65) better than the maximum of the SBERT (41.76). Besides, the standard deviation also becomes much smaller (1.61- - 0.68), which means that improvement by the ReCross is more consistent under different query sets.
The SBERT indexing relies mainly on the semantic similarities between a query example and the upstream data. Instead, our proposed ReCross' uses the hidden representations inside the upstream model M for representing examples. We believe using such an indexing method can better help us find examples that share similar reasoning skills acquired by the upstream model.
ReCross = ReCross + Reranking. The full version of our ReCross with reranking can further
improve the performance substantially on multiple dimensions. Both all @ mean and median are improved by 1 point from the ReCross", and the std is also reduced from 0.68 to 0.42. The last column (A) Table shows improvement compared to the base model BARTO, and we can see that ReCross consistently outperforms non-retrieval methods (c.g., BARTO) by significant gap.
7

---

SBERT
ReCross
cb
0.1
10W30
obop
plge
squad
wrggnd
wec
Let Tomane
Roodg
mett
welaa daa
IVe shape
de 1i0
dae
Figure 3: The mapping between unseen tasks (as rows) and upstream tasks (as columns). The darker upstream tasks take more percentage retrieved data. For example, for the task WIC, ReCross retrieves a plurality of examples from CQP (about 30% of the retrieved examples).
To explore the potential benefits of augmentttan methods such as our ReCross, we also conduct the same experiments on five tasks selected from the BIG-Bench project. The results are shown in Table 2, where we can see that ReCross still outperforms the non-retrieval methods. An interesting case is the movie lorve_ddaals task, where the prompt in the template requires model to output "same" or "different." " However, both TO-3B and BARTO fail to follow the prompt instruction, and only output "yes/no." Only when retrieval-augmentttio methods, there are performance improvement on this task.
4.5 Analysis & More Findings.
More configurations. We have used a partic- ular configuration in main experiments that are in Table 1, which is [Q|=16, /||512, and |u|=2. In Table 3, we explore more configura- tions as ablation studies. The "Main Exp." row refers to the results shown Table 1, and the configurations of other rows are only changed with one factor time. Even using single query example, ReCross is better than BARTO. Table 2: Results on a subset of BigBench tasks. However, when increasing the query size 32, we find that the performance starts to decrease, meaning that there could be an optimal query size for certain R =512. We find that increasing | is generally beneficial, while the mean decreases when [R] is changed from 512 to 1024, although the max and the median slightly increased. Finally, we see that increasing j increases the std. and does not improve the overall performance.
Task
TO-3B
BARTO
ReCross
hindu Hindu_knnwledde
24.75
23.48
24.87
known known_unknower _unknowns
47.83
43.48
47.17+1.65
logic_grid_punee
23.60
20.70
17.1226.20
strategyqa
47.70
48.30
49.76
movie_dialog
0.00
4.40
37.22
Mean
28.78
28.07
35.23 12.85
Retrieved data distribution. Figure 3 presents the difference between the methods in terms of their retrieved data. We draw the distribution of the retrieved data among different upstream tasks for each unseen task
SetupiAll@
Mean
std.
Min
Max
Median
Main Exp.
44.53
0.42
44.16
45.07
44.31
[Q|=1
43.20
0.83
42.58
44.58
42,88
[Q]=8
43.67
0,90
42.09
44.32
43.90
individually. From the heatmap, we can see that ReCross tends have more dominant retrieved tasks (i.e., darker cells), while SBERT's results are more sparse. They both can identify that squad is most similar to the adversarial qa tasks. Their behaviors are very different too. Taking the unseen task wi winogrande (wngrnd) as an example, we can see that the SBERT retrieves from multiple upstream tasks such as paws-x and cosmosQA, but ReCross mainly retrieves from socia al -iaa wi i --q, and cos-e. The experimental results in Table show that ReCross produces a better performance than SBERT (i.c., 55.46 52)18), while
1Q]=32
42.52
1.17
40.52
43.40
42.96
RR|2556
40.80
0.83
39.45
41.68
40.96
1024
44.02
1.43
42.26
45.35
44.59
1=3
43.92
0.58
43.08
44.57
43.89
43.91
0.99
42.76
45.10
44.26
Table
The ablation study of ReCross.
8

---

it is not clear how we can predict such task correlation in advance. This suggests that we should explore more about the utility of instances and tasks in future work.
More analysis. In the appendix, we further presented some analysis to help understand "how" and "when" the retrieval augmentation works: Table 4, Table 5, Appendix A. A.2, and Appendix B. We investigate whether the utility of upstream examples in retrieval augmentation is related to the similarity in terms of the task formats. From Appendix A.I, we found some counterintuitive results. For example. if removing MCQA upstream tasks from the upstream index, then the ARC target task can have an even better performance, although it is an MCQA-formatte hypothesize that similarity in terms of reasoning types is more important than format similarity for retrieval augmentation. After all, the upstream model has been already trained to work with these basic task formats. Re-learning the tasks of the same format might lead the model to overfit the seen domains. Additionally, to provide a more concrete analysis, we also present case studies with two specific tasks (CB and SQUADv2) in Appendix B.
task. Thus, we
Moreover, we conjecture the natural language instructions in the templates are necessary for ReCross to get impressive results. Therefore, we investigated two ways of perturbing the instructions and monitoring the performance changes in Appendix A.2. We find it is indeed true that perturbations of the instructions will lead to much worse performance. We believe that a rigorous, principled way of analyzing the correlation between query and retrieval examples will be great future direction, given the strong evidence that ReCross works well such simple method.
5
More Discussion
5.1 Practicality of unsupervised setting.
Cost of obtaining task labels The unsupervised setting in the paper does not require any human annotation of labels. For some tasks (NLG tasks in particular, e.g., summarization), the expected output (label) are open-ended and possibly lengthy and thus human annotation much more expensive and time-consuming. Also, few-shot learning must ask humans to label examples for each new task, and is thus less practical when there are a large number of emerging tasks from the users. Meanwhile, ReCross requires only a natural- languagg task template, which does not require potentially expensive manual annotation or domain expertise.
Scalability & Real-Time response Deploying the ReCross pipeline is one-time process. All we need to do is to pre-compute the upstream index with LM and configure the reranker (a simple masked LM) by running our script. In production, once the users input the examples with NL instructions, we do not need to wait for any human annotations anymore, so it is much more efficient in the long run. In the scenarios where users only provide one query example and want get its label from the model, ReCross also shows great performance (i.e., IQ|=1 in Table 1). It is then impractical to assume there are few labeled data from the users too in such cases.
5.2 Empirical studies
The unsupervised ReCross performance is comparable to few-shot learning with label annotations. In Appendix D.2, we report the performance of directly fine-tuning BARTO with the labeled query examples. Although it is an unfair comparison with our previous ReCross results, we found that they are comparable. More importantly, the ReCross framework does not conflict with the sew-shot setting. Given a labeled query set for a target task, retrieved examples from the ReCross can still improve few-shot learning as additional training data. We designed two simple methods for applying ReCross under the few-shot setting and report the empirical results in Appendix D.2. It turns out that ReCross can also boost the performance under the few-shot setting by about 3 points.
Related Work
Multi-task training for task generalization. Text-to-text Transformer language models such as T5 enable us to train a multi-task NI.P model with a more straightforward recipe: mixing the data of multiple tasks into a unified seq2seq format, and then fine-tuning text-to-text LMs for implicit multi-task learning. UnifiedQA (Khashabi et al., 2020) is among the first works in this direction.

---

Although it shows great generalization performance within QA tasks, it can hardly generalize to other NLP tasks. Recent works, such as CrossFit (Ye al., 2021), ExT5 (Aribandi et al., 2022), FLAN (Wei et al., 2021), TO (Sanh et al., 2021), and InstructGPT Ouyang et al. (2022) focus on how to generalize a massively multi-task model across task boundaries in a much broader context.
Particularly, in the CrossFit framework (Ye et al., 2021), cross-task generalization requires a small number of labeled instances of the target task for fine-tuning. It because the templates of CrossFit use the task names as the hard prefixes. Therefore, it is necessary to fine-tune the upstream model with a few examples that have the target task names as prefixes (i.e., few-shot learning), but this largely limits the application scenarios of these multi-task NLP models in practice. We instead focus on unsupervised cross-tas task generalization, where there is no labeled data of an unseen task (i.e.. zero-shot learning). Using natural- language instructions as prompts, both FLAN and TO show that it promising to perform zero-shot cross-task generalization.
In this work, we also focus on such an unsupervised setting for cross-task generalization, while our problem setup is bit different from the ones used in TO and FLAN. As for the assumption about the unlabeled data, their setups can be seen as a special case of ours when [Q] = for all unseen tasks. The evaluation protocols of TO and FLAN assess the generalization performance of the upstream model as it is, and thus their evaluation is more about the quality of templates and the upstream training tricks. In contrast, our evaluation protocol can also study how to efficiently adjust the upstream model such that the updated models can generalize to new tasks without labeled data, Thus, we believe ours is a more general setup for studying unsupervised cross-task generalization.
Retrieval augmentation in NLP. We aim to retrieve useful examples from the upstream data and re-learning them for cross-task generalization. The proposed ReCross pipeline is inspired by open-ended QA methods such DPR (Karpukhin et al., 2020), DrFact (Lin et al., 2021), and RAG (Lewis et al., 2020b). Retrieval augmentation also shows great performance in pre-training LMs (Guu et al., 2020). Besides, Wang et al. (2022) shows that learning with similar data via retrieval augmentation can improve the performance of a task-specific model. Rubin et al. (2022) show that retrieving better demonstration examples is also helpful for in-context few-shot learning of GPT-3 style language models (Brown et al., 2020). The key challenge in the problem setup of this work is to predict the utility of the examples for unseen tasks with the consideration of efficiency and scalability. We have discussed more details about this challenge and related works in Sec. 2.
7 Conclusion & Future Directions
We demonstrate that retrieval augmentation can largely improve the cross-task generalization ability to multitask LMs in unsupervised settings. Our proposed method, ReCross, is straightforward yet effective retrieval method that combines both efficient dense retrieval and effective pair-wise reranking. Our empirical results show that it significantly outperforms both non-retrieval methods and other baseline methods. We perform ablation studies showing the impact of changing query sizes, retrieval sizes, upsampling ratios, etc. We also find the distribution of retrieved data for analyzing the behavior differences between ReCross and others. We believe that our paper will spur further research on retrieval-aughe augmentttnnnnni methods for cross-task generalization. Interesting future directions include: 1) improve the re-learning stage by including more information from query examples, 2) extend the distant supervision mining process as a self- traininiin procedure, 3) rigorously analyze the correlation between upstream data and target tasks, etc.
Acknowledgments
This research is supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via Contract No. 2019-1905 1199-1995600007, 660007,, the DARPA MCS program under Contract No. N660011924033 the Defense Advanced Research Projects Agency with awards W91INF- 19-20271, NSF IIS 2048211, and gift awards from Google, Amazon, JP Morgan, and Sony. We thank all collaborators in USC and the NeurlPS 2022 reviewers for their constructive feedback on the work.
References
Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Honglei Zhuang, Vinh Q Tran, Dara Bahri, Jianmo Ni, Jai Gupta, Kai Hui, Sebastian Ruder,
10

---

and Donald Metzler. 2022. Ext5: Towards extreme multi-task scaling for transfer learning. In International Conference on Learning Representations.
Stephen Bach, Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-david, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Fries, Maged Al-shaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang. Dragomir Radev, Mike Tian-jian Jiang, and Alexander Rush. 2022. PromptSource: An integrated development environment and repository for natural language prompts. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 93-104, Dublin, Ireland. Association for Computational Linguisties.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. Retrieval augmented language model pre-training.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with gpus. IEEE Transactions on Big Data.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing Online. Association for Computational Linguistics.
(EMNLP), pages
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020. UNIFIEDQA: Crossing format boundaries with a single QA system. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1907, Online. Association for Computational Linguistics.
Lukas Lange, Jannik Strötgen, Heike Adel, and Dietrich Klakow. 2021. To share or not to share:
Predicting sets of sources for model transfer learning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8744-8753, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020a. BART: Denoising sequence-to-seqeen pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih. Tim Rocktäschel, Sebastian Riedel, and Douwe generation for knowledge-intene intensive NLP tasks. In Advances in
Kiela. 2020b. Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6- 2020, virfual.
Bill Yuchen Lin, Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Xiang Ren, and William Cohen. 2021. Differentiable open-ended commonsense reasoning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 461 1-625, Online. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: robustly optimized bert pretraining approach. ArXiv preprint, abs/

---

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. ArXiv, abs/2203.02) 152555...
Vishakh Padmakumar, Leonard Lausen, Miguel Ballesteros, Sheng Zha, He He, and George Karypis. 2022. Exploring the role of task transferability in large-scale multi-task learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2550, Seattle, United States. Association for Computational Linguisties.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Nils Reimers and Iryna Gurevych. 2019. Centence-BERT. Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2675, Seattle, United States. Association for Computational Linguistics.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu. Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Stella Biderman, Leo Gao, Tali Bers, Thomas Wolf, and Alexander M. Rush. 2021. Multitask prompted training enables zero-shot task generalization.
Aarohi Srivastava, Abhinav Rastogi, Abhishek B Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ameet Annasaheb Rahane, Anantharaman S. Iyer, Anders Johan An- dreassen, Andrea Santilli, Andreas Stuhlmuller, Andrew M. Dai, Andrew D. La, Andrew Kyle Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong. Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal. Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakacs, Bridget R. Roberts, Bao Sheng Loe, Barret Zoph, Bartlomiej Bojanowski, Batuhan Ozyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill Yuchen Lin, Blake Stephen Howald, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, C'esar Ferri Ram' irez, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burchh Chris Waites, Christian Voigt, Christopher D. Manning, Christopher Potts, Cindy Tatiana Ramirez, Clara Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Daniel H Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Gonz' alez, Danny Hernandez, Danqi Chen, Daphne Ippolito. Dar Gilboa, David Dohan, D. Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin. Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong-Ho Lee, Ekate- rina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth P. Donoway, Ellie Pavlick, Emanuele Rodolà, Emma FC Lam, Eric Chu, Eric Tang. Erkut Erdem, Ernie Chang, Ethan A. Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhskii, Fan Xia, Fatemeh Siar, Fernando Mart" ilemee, Francesca Happ's e, François Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambatiita Parascandolo, Giorgio Mariani, Gloria Wang, Gonzalo Jaimovitch-L'opee L'opez, Gregor
12

---

Betz, Guy Gur-Ari, Hana Galijasevic, Han Sol Kim, Hannah Rashkin, Hanna Hajishirzi, Harsh Mehta, Hayden Bogar, Henry Shevlin, Hinrich Schütze, Hiromu Yakura, Hongming Zhang, Hubert Wong, lan Aik-Soon Ng. Isaac Noble, Jaap Jumelet, Jack Geissinger, John Kernion, Jacob Hilton, Jaehoon Lee, Jaime Fernández Fisac, J. Brooker Simon, James Koppel, James Zheng, James Zou, Jan Koco'n n, Jana Thompson, Jared Kaplan, Jarema Radom, Jascha Narain Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jenni Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Oluwadara Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang. Jane W Waweru, John Burden, John Miller, John U. Balis, Jonathan Berant, Jorg Frohberg. Jos Rozen, José Heráández-Orallo, Joseph Boudeman, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Ka- terina Ignatyeva, Katja Markert, Kaustubh D. Dhole, Kevin Gimpel, Kevin Ochieng' Omondi, Kory Wallace Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contreras- Ochando, Louis-Philippe Morency, Luca Moschella, Luca Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros Col'on, on, Luke Metz, Lutfi Kerem eSenel, Maarten Bosma, Maarten Sap, Maartje ter Hoeve, Madotto Andrea, Maheen Saleem Farooqi. Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, M Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthat, Matthew Leavitt, Matthias Hagen, M'aty' as Schubert, Medina Baitemirova, Melissa Amaud, Melvin Andrew McElrath. Michael A. Yee, Michael Cohen, Mi Gu, Michael I. Ivanitskiy, Michael Starritt, Michael Strube, Michal Swkedrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Monica Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, T MukundVarma, Nanyun Peng. Nathan Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas S. Roberts, Nicholas Doiron, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang. Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy Liang, Peter W. Chang. Peter Eckers- ley, Phu Mon Htut, Pi-Bei Hwang. P. Milkowski, Piyush S. Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, QING LYU, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ram' on Risco Delgado. Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymackers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan Le Bras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Ruslan Salakhutdinov, Ryan Chi, Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang, Sahib J. Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Sam Bowman, Samuel S. Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster. Sepideh Sadeghi, Shadi S. Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixi- ang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi. Siva Reddy, Sneha Priscilla Makini. Soo hwan Lee. Spencer Bradley Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ernon, Stella Rose Biderman, Stephanie C. Lin, Stephen Prasad, Steven T. Piantadosi, Stuart M. Shicber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq A. Ali, Tatsuo Hashimoto, Te-Lin Wu, Theo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili,, Timo Schick, T. N. Kornev, Timothy Telleen-Lawton, Titus Tunduny, Tobias Gerstenberg, Trenton Chang. Trishala Neeraj. Tushar Khot, Tyler O. Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Venkatesh Ramasesh, Vinay Uday Prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang, W Vossen, Xiang Ren, Xiaoyu F Tong, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yang Song, Yasaman Bahri, Ye Ji Choi. Yichi Yang. Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yu Hou, Yushi Bai, Zachary Seid, Zhao Xinran, Zhuoye Zhao, Zi Fu Wang, Zijie J. Wang, Zirui Wang, Ziyi Wu, Sahib Singh, and Uri Shaham. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. ArXiv preprint, abs/2206.04615.
Tu Vu, Tong Wang, Tsendsuren Munkhdalai, Alessandro Sordoni, Adam Trischler, Andrew Mattarella- Micke, Subhransu Maji, and Mohit lyyer. 2020. Exploring and predicting transferability across NI.P tasks. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7882-7926, Online. Association for Computational Linguistics.
13

---

Shuohang Wang. Yichong Xu, Yuwei Fang, Yang Liu, Siqi Sun, Ruochen Xu, Chenguang Zhu, and Michael Zeng. 2022. Training data is more valuable than you think: A simple and effective method by retrieving from training data. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 3170-3179, Dublin, Ireland. Association for Computational Linguistics.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2021. Finetuned language models are zero-shot learners, ArXiv preprint, abs/2109.01652.
Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. 2021. CrossFit: A few-shot learning challenge for cross-task generalization in NLP. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pages 7163-7189, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Please see the experiment analysis sections and the appendix.
(c) Did you discuss any potential negative societal impacts of your work? [Yes]] Please check the appendix.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URI.)? [Yes] We include our code, data, and instructions presented in the paper. As the data is quite large, we also include a script to download it.
our supplemental material for reproducing all results
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] We have specified the details of the base model (i.c., BARTO) and the configurations for our retrieval-ane cerieval-auugmentatie augmentation methods. Please check the experiment section and the appendix respectively.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] All our results are based five different rounds, where each we use different set of query examples. We also report multiple dimensions of the statistics in our table (c.g., the median, min, max, std, and mean).
(d) Did you include the total amount compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [Yes] We show these details in appendix.
4. If you are using existing assets (c.g., code, data, models) or curating/relcasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] The data we used are all open-source and publicly available via the "datasets" library from HuggingFace.
14

---

(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] Please refer our appendix and the links to the used datasets.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "pV4bG3CeeiWR", "reviewer": "Reviewer_NtmR", "summary": "This paper considers the problem unsupervised cross-task generalization of multi-task LMs such as T0, where only a few unlabeled examples from the target task are available. It proposes to retrieve, from the multi-task training corpus, samples that are likely to help the target task using the unlabeled target task inputs, and fine-tune the LM on these retrieved samples. \nTo retrieve helpful samples for a target task, they propose a retriever-ranker pipeline, where the more efficient retriever first retrieve, from the entire training set, an initial set of candidate samples, using the last-layer encoder representations from the upstream LM as embeddings for retrieval. It then trains a reranker using a meta-learning inspired approach, aiming to find training samples that are more likely to help an unseen target task.\nThey show that this retrieval-augmented approach achieves better cross-task generation than the original LM, as well as a naive retrieval approach based on semantic similarity (using SentenceBERT embeddings).", "questions": "Q1: Does this work assume the availability of prompt templates for the unseen target tasks? If I understand correctly, these templates are part of the \"input\" of the target task, which are assumed to exist? If this is correct, then it amplifies my main concern in the Weakness section. It is arguably more difficult / expensive to come up with these natural language prompts that work well for the upstream LM for each unseen task than to label a few samples.", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The paper is clearly written. The problem setting and proposed approach is lucidly presented.\n\n- Given that no target task labels are available in the \"unsupervised\" setting, the proposed method is sound, and produces non-trivial gains over the applying the vanilla upstream LM to unseen tasks.\n\nWeaknesses:\n\n- My main concern is the practicality or usefulness of this \"unsupervised\" setting. Given that only a few target task samples are required (the paper uses 16), the cost of obtaining the inputs vs. (inputs + labels) would probably not differ that much in the real world. It would at least be helpful to compare to the \"supervised\" cross-task generalization methods to show how much the gap is. And if this approach lags significantly behind a few-shot supervised one, it is arguably less expensive / time-consuming to simply label a few samples than to apply a complicated retriever-ranker pipeline.\n\nEDIT after rebuttal: The added Appendix D.2 in the few-shot setting is welcome, and to some extent alleviates my concern here. Though it does not fully address this issue: Collecting labels is more expensive on certain tasks for sure, but this work only considers the tasks where the answer can be evaluated via exact match (e.g. multiple choice, classification, etc.). Arguably the cost for collecting 16 inputs vs. 16 (inputs + labels) does not differ that much. In addition, the concern over the availability of prompt templates for unseen tasks remains. Few-shot methods can transfer to new tasks with a few inputs + labels. I'm not convinced that this is inferior to the proposed paradigm of requiring a prompt template and a few inputs.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "n6vt8rRpASL", "reviewer": "Reviewer_fTmN", "summary": "This paper presents an approach called ReCross for fine-tuning an LLM by retrieving seen-task data that is particularly relevant to unlabelled inputs from an unseen task. The seen-task examples that are most relevant are retrieved from an index which contains representations from a model that has been fine-tuned on the seen-task data. Following the retrieval, a re-ranker is used to score each retrieved example-query pair, and the top K are used for fine-tuning the model before evaluating on the unseen target task. Experiments on a range of NLU tasks including classification, QA etc. show that ReCross outperforms instruction-tuning baselines.", "questions": "Please see the previous section which includes suggestions as well.", "limitations": "Missed the section discussing limitations, where is it included?", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:  \nThis paper shows that carefully selecting data from related tasks for fine-tuning before zero-shot evaluations can help to improve performance. This is sort of a test-time adaptation of the model parameters at a coarser granularity than at an example-level. This means that the retrieval cannot be based on semantic similarity which is why the model used for creating the retrieval index has been fine-tuned on a large pre-selected subset of the seen-task data. The entire setup being explored is an important and high-impact problem to study, and retrieval augmentation provides a compelling solution worth exploring as done in this paper.   \nAlso, the Bart0 vs. T0-3B result is quite interesting. \n\nWeaknesses:  \n- The paper is missing a qualitative discussion on what sorts of examples are retrieved and what sorts of features might be in play for retrieving useful similar examples. Figure 3 doesn’t seem to be providing too many intuitions for this. This makes it a bit hard to understand where the improvements are coming from.\n- The role of the reranker is unclear. Improvements over and above the simpler version of ReCross without the reranker seem small, mostly non-existent given the standard deviation, which is bizarre especially given how complex the whole distant supervision setup is. Would it be better to just remove this from the main paper and replace it with more analysis, data-related ablations and discussion on where the improvements are coming from? ReCross without the distant supervision is still interesting and perhaps a bit simpler. \n- The conclusion from the random retrieval baseline that says “This suggests that it is promising to study better retrieval methods to get consistent improvement in overall performance” seems to only hold true for winogrande..maybe it isn't the right conclusion? Random retrieval is a control experiment to remove confounds and understand how to interpret the improvement from ReCross. E.g. h-swag, anli_r3, squadv2 do not benefit from ReCross. \n- The paper, while comprehensive, can sometimes be verbose. Perhaps some revisions would help to pare it down. Figures 1 and 2, as is, may not be helping the reader to understand the approach. Is the example in Figure 1 an actual retrieval from the amazon polarity and gigawords tasks? These don’t seem like they should help, should they? Maybe cherry-picking a clearer example could help with this one? Figure 2 might benefit from being broken down into stages.   \n\nEDIT: Updating the score from 4-->6 after the author response. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "a1AesqcUEA", "reviewer": "Reviewer_YUN8", "summary": "The authors present ReCross, a retrieval augmentation method to improve cross-task generalization of seq2seq models. Starting with a T5-like model (\"upstream model\") trained on a diverse set of tasks (in seq2seq format), the training data is then embedded using the encoder's top layer and stored in a dense index. To evaluate on an unseen task, the query is used to retrieve training data examples that might be helpful in evaluating this new example (using a two-stage retrieve+rerank process). These helpful examples are used to fine-tune the upstream model, and the model is finally used to perform the target task.", "questions": "* How does ReCross compare to the other methods of Table 1 if using exact match as the metric? I see it's implemented in the code, so I assume this was tested?", "limitations": "The authors adequately address in section 3.5 the fact that their continuous fine-tuning approach is quite simple.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* This is a simple but effective method for allowing a model to retrieve helpful previously seen training data, and re-learn from it. As the authors say, there are probably approaches that would be more effective than continuous fine-tuning. However, it's impressive how well such a simple method already works.\n* The authors provide a reasonable set of baselines and an informative choice of ablations (including those in the Appendix). We see the importance of choosing a way of embedding examples that is compatible with the upstream model's internal representation. We also see the contribution of the reranker to the overall performance, and the interactions of various central hyperparameters.\n\nWeaknesses:\n* ~One crucial issue I see with this work is that the authors create a new metric (SoftEM) and use it throughout the paper without properly motivating it. They refer to the Appendix, stating \"We discuss more on this selection with illustrative examples in Appendix\". No such discussion or examples are found in the Appendix, as far as I can tell. It's not clear whether the proposed method would still perform as well using a more standard metric.~ EDIT: Fixed in the latest draft", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept