Efficiently Enhancing Zero-Shot Performance of Instruction Following Model via Retrieval of Soft Prompt
Authors: Seonghyeon Ye
Joel Jang
Doyoung Kim
Yongrae Jo
Minjoon Seo
Venue: EMNLP 2023 Findings (2023)
Keywords: natural language processing
zeroshot language models
prompt tuning
TL;DR: Retrieval of soft prompt efficiently enhances the zero-shot performance of instruction-tuned language model.

Abstract: Enhancing the zero-shot performance of instruction-following models requires heavy computation, either by scaling the total number of training datasets or the model size. In this work, we explore how retrieval of soft prompts obtained through prompt tuning can efficiently assist hard prompts in zero-shot task generalization. Specifically, we train soft prompt embeddings for each prompt through prompt tuning, store the samples of the training instances mapped with the prompt embeddings, and retrieve the corresponding prompt embedding of the training instance closest to the query instance during inference. While only adding 0.007% additional parameters, retrieval of soft prompt enhances the performance of T0 on unseen tasks by outperforming it on 10 out of 11 datasets as well as improving the mean accuracy of T0 on BIG-bench benchmark by 2.39% points. Also, we report an interesting finding that retrieving source embeddings trained on similar answer choice formats is more important than those on similar task types.

Efficiently Enhancing Zero-Shot Performance of Instruction Following Model via Retrieval of Soft Prompt
Seonghyeon Ye
Joel Jang
Doyoung Kim KAIST
Yongrae Jo Minjoon Seo
aongee w @ @kaist.ac.kr
Abstract
Prompt Library
Output
Keys
Enhancing the zero-shot performance of instruction-followingg models requires heavy computation, either by scaling the total number of training datasets the model size. In this work, we explore how retrieval of soft prompts obtained through prompt tuning assist hard prompts in pero task general- ization. Specifically, we train soft prompt em- beddings for each prompt through prompt tun- ing, store the samples of the training instances mapped with the prompt embeddings, and re- trieve the correspouding prompt embedding of the training instance closest to the query in- stance during inference. While only adding 0.007% additional parameters, retrieval of soft prompt enhances the performance of TO on un- seen tasks by outperforming datasets as well as improving the mean accu- racy TO on BIG-hench benchmark by 2.39% points. Also, we report interesting finding that retrieving source embeddings trained on similar answer choice formats more impor- tant than those on similar task types.'
Instance
Instanca
Instance
Embedding A..
TO
retriever
Instance
Evakastion Input
Evaluation Inpur
efficiently
During zero-shot inference, RoSPR selects similar training instances with the given input from the Source Prompt Library and retrieves the prompt embeddings corresponding to the selected training instances.
Figure 1:
instruction-tuning stage involves explicit, multi- task prompted learning various tasks, enabling LMs to quickly adapt to unseen tasks at inference. two
To maximize the effect of instruction-tunm approaches have been widely explored: (1) scal- ing the number training datasets, and (2) scaling the model size (Wang et al., 2022b; Chung et al., 2022). However, both approaches require heavy computation, not applicable with an academic bud- get. Specifically, the first approach requires updat- ing the whole parameters of the model every time training dataset added, showing limitations in terms of scalability. the other hand, the second approach requires heavy memory requirements to load and train a massive LLM.
10 of I1
1 Introduction
Training Large Language Models (LLMs) on huge amounts of data has enabled LMs to perform down- stream tasks without any fine-tuning with the aid of natural prompts or concatenation
To enhance the zero-shot performance of instructionn-following model efficiently, we intro- duce Retrieval of Soft Prompt (RoSPR), which is easily scalable and requires minimal computa- tion by only adding 0.007% parameters the main model during inference. As shown in Figure 1, by training prompt embeddings (soft prompt) for each given hard prompt through prompt tuning, we construct a Source Prompt Library consisting of samples of training instances mapped with their corresponding prompt embeddings. Then, during inference, by using simple, off-the-shelf dense retriever model, we search for training instances similar to the given query instances and retrieve their corresponding prompt embeddings. Because the backbone LM is frozen, it allows the retrieved
few demon- stration instances (Brown et al., 2020; Rac et al., 2021; Kojima et al., 2022; Chowdhery et al., 2022). Additionally, recent works have shown that adding instruction funing stage, additional training step that helps pretrained LMS understand prompts and demonstrations results in a significant perfor- mance boost on zero-shot task generalization even for moderate-
LMs (Min et al., 2021; Sanh et al., 2021: Wei et al., 2021; Wang et al., 2022b; Ye et al., 2022; Chung et al., 2022). This extra
Model checkpoints and code implementation are available at gtgghhh..oo//ccogggooooyeeoeeeee

---

embeddings to serve as adapters assisting hard prompts. While ROSPR can be applied any LM, this work, we use TO (Sanh al., 2021) as our initial backbone LM and perform prompt tuning on the tasks used during the instruction-tuning stage. While adding only 0.007% additional param- eters, RoSPR outperforms TO on 10 out of 11 evaluation datasets and outperforms efficient fine- tuning baselines without any target task fine-tuning, RoSPR is also effective for challenging tasks such as tasks from BIG-bench (Srivastava et al., 2022), outperforming TO by 2.39% mean accuracy. Fur- thermore, we provide several interesting findings: (I) Variants of ROSPR that include interpolation of multiple prompt embeddings and scoring method that considers the answer choice distribution during retrieval further increases the effect of RoSPR (2) Also, we provide analysis of which factors attribute to the performance of RoSPR and show that, sim- ilarly to the role of demonstrations in in in-context learning (Min et al., 2022), heuristic features such as answer choice format are more important than the similarity of the source task.
adaptation. For parameter-efficient fine-tuning, Vu et al. (2022); Su et al. (2022) retrieve source prompt embedding that is similar to the target prompt em- bedding and obtain a better initialization point for prompt tuning. Instead of utilizing a single prompt embedding, recent works show a mixture of multi- ple prompt embeddings to be effective (Asai et al., 2022; Qin and Eisner, 2021).
For instruction-tuning. Lin al. (2022) retrieve training instances that are similar to the query through a dense retriever and fine-tune the model using the retrieved examples. For in-contexi Iearn- ing, Rubin et al. (2021); Liu et al. (2022b); Wang et al. (2023) retrieve training data that could be used for demonstrations. Wang et al. (2022c) show the effect of retrieving prompt embeddings in a continual learning setting. Although our proposed method is related to these works, the novelty of our work lies in applying souree task retrieval in the zero-shot setting and retrieving soft prompts instead of training instances.
3 Method
In this section, we introduce Retrieval of Prompt Tuning (ROSPR) for zero-shot task generalization. A detailed overview is shown Figure 2. We first train source prompt embeddings of LM for each hard prompt given a source task using prompt tun- ing (Section 3.1). Then, we save training instance samples along with their prompt embeddings in the Source Prompt Library and use it to retrieve embed- dings at inference to perform tasks in zero-shot manner (Section 3.2). We additionally introduce in- terpolation of multiple source prompt embeddings (Section 3.3) and variance-bai based ranking (Section 3.4) to increase robustness and accuracy.
2 Related Work
2.1 Task Generalization with Instruction-Tuin -Tnnin
Prompts and demonstrations are essential for task generalization since proper explanations are re- quired for LMs to understand an unseen task (Ko- jima et al., 2022; Wei et al., 2022; Lampinen et al., 2022). Instruction-tuninng, which is explicit multi- task prompted training on various downstream tasks, a simple but effective way to achieve this, resulting in improved zero-shot capabilities. Zhong et al. (2021) first introduced the method of instruction-tuning by converting various tasks format and finetuning
3.1 Training Source Prompt Embeddings
into a questiion-nsswrii the model on the aggregated dataset. Following works (Mishra al., 2022; Min al., 2021; Sanh et al., 2021; Wei al., 2021; Wang et al., 2022b; Xu et al., 2022: Ouyang et al., 2022; Ye et al., 2022; Chung et al., 2022) extended this approach on a larger scale and show that zero-shot task generaliza- tion could be enhanced with more diverse prompts, a larger number of training downstream tasks, and a larger L.M.
Even though RoSPR may be used to augment any type of LM, we use TO (Sanh et al., 2021) the backbone LM for this paper. For training of soft prompts,
utilize the source tasks and prompts used for the instruction-tuning phase of TO. While TO was trained in a multi-task learning manner, we freeze the initial TO parameters and train only soft prompts (source prompt embeddings) for each hard prompt of the source task.
Prompt Tuning Among various parameter- efficient fine-tuning methods, we follow prompt tuning proposed by Lester et al. (2021) because the number of trainable parameters is extremely small
2.2 Source Task Retrieval
Retrieving a source task that is relevant to the target task has shown to result in faster and better task

---

Source Prompt Training
Inference
kyO%)
Source Prompt Library
Ranking
RoSPr
TO
No
No
Keys
Values
Freq
Var
MIPS
kg(%)p
8
0.2
RoSPr Inter
Interpolate
Fi
k)(())
TO
by(kn)
2
k(xg)
10
1
RoSPr
No
+ Var
anaoeer tanyarts
be
"What lunyard""
Bupposee don't think they should abolishii. Cann um infer that "they should abollsh
ky(%,))
Figure overview of ROSPR. For each hard prompt of source datascts, soft prompts are trained via prompt tuning. After storing training instances as keys and corresponding prompt embedding as values, RoSPR searches training instances similar to query sel Q. retrieves the corresponding prompt embeddings. and selects the most frequently retrieved candidate for inference. Variants of selection strategy are also shown: ROSPK+INTER interpolates between multiple related source embeddings and ROSPR+ VAR ranks candidate embeddings considering both frequency and variance.
(~204K parameters per prompt), which implics that the memory overhead parameter retrieval at inference is negligible.
We first construct Source Prompt Library, con- sisting of sentence-level representations of train- ing instance inputs as keys and the corresponding source prompt embedding as the values. For each available source prompt embedding, number of samples stored in the library. The sentence- level representations are obtained by getting the mean representation of hidden states of the last layer of the dense retriever. We use a TO-small encoder as a dense retriever, replicated based on Sanh et al. (2021) with smaller model size.
For each source training dataset D; (i = 1,.., where T is the total number of souree datasets, we train source embeddings Eij (j I,.,, M,) where M, is number of hard prompts D;, making soft prompt embeddings for each individual hard prompts. Specifically, given training instance { Vik (k: = 1, K) from D; where K is the number of sampled training instances per dataset, first convert it into its hard prompted version (Ta)), hy(wis) where hy()) denotes adding the j-th hard prompt 2 Next, we train the LM with the following objective:
At inference, we first randomly sample QQ query instances from the target task, following Lin et al. (2022). After obtaining sentence-leve representa- tions for each query through TO-small encoder, retrieve top-N examples for each query instance using MIPS (maximum inner product search) op- cration on our Source Prompt Library, retrieving a total of Q * N prompt embeddings. As the de- fault methodology, among the retrieved embedding candidates, we select the most frequently retrieved prompt embedding our designated seft prompt for the given target task and concatenate the em- bedding with each of the target task instances be- fore feeding our backbone LM. In the next two subsections, we explain different strategies for calculating the target embedding from the Q * prompt embedding candidates.
max (1ax)) Eigi (x(4))) Bij
(1)
where all the parameters of the underlying back- bone LM are frozen and only Ei is trainable. In short, given D: perform M, number of prompt tumings for each hard prompts, resulting in ΣT M; total number of source prompt embed- dings. For training efficiency, we only train on K 5000 training instances for a single epoch for each source prompt embedding.
3.2 Zero-Shot Embedding Retrieval
After source prompt embedding training, we re- trieve the most related source embeddings and se- lect one from the retrieved candidates to be used at inference (right part of Figure 2).
3.3 Interpolation of Prompt Embeddings
When retrieving only a single prompt embedding for a given task (Section 3.2), it may result in high variance across evaluation prompts when the se- lected prompt embedding does not fit well with the
FFrr cach instances, the input and output are converted into its prompted version using the promptsource toolkit (Bach ct al., 2022). An example is given in Appendix G.
FFor all indexing and searching, we use FAISS (Johnson et al., 2019) for fast source prompt embedding retrieval.

---

given task. Recent works on prompt embedding re- trieval have shown that the interpolation of prompt embeddings effectively transfers to the target task (Asai et al., 2022; Vu et al., 2022). We also explore calculating the target embedding through interpola- tion of multiple souree embeddings instcad of just using a single embedding. Among Q N prompt candidates searched in Section 3.2, we select top- N' candidate embeddings based on the frequency of the search. Then, we calculate the weighted sum of the candidate embeddings, where the interpo- lation weight for each source embedding is based on the proportion of frequency. While Asai et al. (2022); Vu et al. (2022) require fine-tuning the tar- get embeddings on the target task to calculate the weights for interpolation, our approach does not re- quire any target task fine-tuning, enabling zero-shot task transfer.
explain our evaluation setting during zero-shot in- ference and bascline models. We provide detailed experiment configurations in Appendix F.
4.1 Source Tasks
For training soft prompts through prompt tuning, use the subset of source tasks used for the initial TO instruction-tunin source task, we use the prompts for each dataset in TO, resulting in a total of 230 prompts. For Source Prompt Library construction, we sample only n 100 training instances per source embed- ding to minimize the inference latency. We show variation of n and different methods to sample n training instances in Appendix D.
(Sanh et al., 2021)* 4 For each
4.2 Evaluation Tasks
Following Sanh et al. (2021), we evaluate on the validation set of 4 held-out tasks (natural language inference, sentence completion, coreference res- olution, word sense disambiguation) resulting in total of 11 evaluation datasets. We also follow Sanh et al. (2021) and evaluate on 14 different datasets from the BIG-hench benchmark (Srivas- tava et al., 2022) 5. We use rank classification evaluation method by selecting the output option with higher log-likelihood following Brown et al. (2020); Sanh et al. (2021). For all evaluation tasks, accuracy as evaluation metric and report the mean accuracy and standard deviation of all of the evaluation prompts per given dataset (average of 10 prompts per evaluation dataset) 6, For BIG- bench tasks, we do not report standard deviation because only one prompt is provided per task.
3.4 Variance-based Ranking
Similar the scoring and calibration method of Lu et al. (2022); Zhao et al. (2021), we introduce a scoring method applicable to zero-shot classi- fication tasks that allows ranking the Q * re- trieved prompt embedding candidates by consid- ering the answer choice distribution of the given target task as extra cues together with the original frequency cues. To accomplish this, perform forward pass with the concatenation of each can- didate prompt embeddings together with the given hard prompt (only including the instruction, ex- cluding the input instance) of the target task and give a higher score to the embedding candidate that results in lower variance. Ideally, the combination of soft and hard prompts should result in equal probability among the answer choices because the actual context of the task is not included.
4.3 Baseline Models
Zero-shot Baseline For zero-shot baseline mod- els, we show the results of TO (3B) together with a times larger TO (1IB) instruction-tuned model. We also compare with GPT-3 (175B) model which is 60 times larger than TO (3B).
Specifically, when given a target task with k-th hard prompt hg. for each candidate embedding Eij. we calculate the scoring as follows:
(hk Eng))
Score(he, Eij)
Fine-tuning Baseline We also compare with efficient fine-tuning baseline models that utilize prompt tuning. These models require target task prompt tuning, indicating that zero-shot transfer
V Varl[ Eey h4)
where refers to the available output options of the target task.
"We use 29 out of 38 datasets that used to train TO. We explain the training task selection rationale in Appendix H.I 'We provide the full list evaluation datasets in Appendix H.
4 Experimental Settings
In this section, we explain the experimental settings of training of source prompt embedding and con- struction of our Source Prompt Library. We also
"For methods based on RoSPR, we report the performance average of 3 runs with different random seeds for the sampling of evaluation queries used for the prompt retrieval.

---

Method
of Param
NLI
Sentence Completion
Coreference Reselot.
WSD
Total Avg.
(Rase/Tainableg
RTR
CB
AN. RI
AN. R2
AN. R3
COPA
Hellasw.
StoryC.
Winogr.
WSC
WIC
Mvon
570
64.55
33.81
23.11
75.88
84,03
50.97
50.69
51.22
3.62
PT (PT)
38/204K 204K
63.14
44.52
53.07
31.44
52.94
73,00)
26.39
50.67
46,73
50.02
ATTEMPT (FT)
/6141K
68.95
44.88
36.19
34.73
34,81
74.38
26.76
51.33
64,90
51.05
TU-ROSPR
3B/ 204K
71.54
49.48
37,05
34.61
33.92
78.75
26.97
85.52
51.50
64.52
51.76
53.24
3.62
w/ INTER
38/ 204K
70:71
52.30
37.30
34.34
33.89
78.25
26.94
85.62
51,10
64,52
50.73
53.24
3.30
w/
3R/ 204K
21.78
50.36
57.07
34.55 58
35,90
78.88
27.01
85,52
51.45
64,94
51.94
53.38
3.38
w/ VAR & INTER
3D /04K
72.60
51.98
37,25
34.31
33,95
77,83
26.84
85,58
50.93
64,97
51.18
53.40
3.47
ORACLE
3R 204K
73.79
58.10
37.65
34.92
34.91
81.13
27.75
87.57
52,36
68.17
26
35.60
3.07
TO
11B/0
80.83
70.12
43.56
38. 68
41,26
90.02
33.58
92.40
59,94
61.45
56.58
60.76
GPT-3
175B 0
63.50
46.40
34,60
35:40
34,50
91.00
78.90
83,20
70.20
63.40
45.92
59.00
Table 1: ROSPR refers our main proposed method, w/ INTER refers to applying interpolation of multiple source embedding candidates, w/ VAR refers retrieval through variance-based ranking. w/ VAR & INTER refers applying both interpolation and variance-based ranking where the interpolation weight based the variance-base based ranking score, and ORACLE refers performance when the most optimal source embedding retrieved from the candidates, acting as an upper bound performance for retrieval. FT refers fine-tuned models the target tasks. For FT models, we exclude StoryCloze due the absence of training instances. The best and second-best performance is shown in bold and underline respectively. Comparison with hard prompt optimization techniques and visualization of the results shown Appendix A and Appendix F, respectively.
is infeasible. Similar to our source prompt tun- ing process, we train each target prompt for sin- gle epoch with maximum of 5,000 training in- stances. The first baseline model is naive prompt tuning on the target tasks without any prompt re- trieval, referred to as PT (Lester et al., 2021). The second model is ATTEMPT (Asai et al., 2022), which trains the target soft prompts through at- tentional mixtures of source prompts. Because StoryCloze (Mostafazadeh et al., 2016) does not contain training instances, we exclude the dataset for fine-tuning. More training details of fine-tuning baseline are specified in Appendix C.
ROSPR also outperforms finetuning baselines even without utilizing any training instances of the target task. We first observe that PT harms the performance of the backbone model, which aligns with the result of Liu al. (2022a); Gu et al. (2022) that prompt tuning is unstable when the training in- stances or the training steps are small. By compar- ing ATTEMPT with ROSPR, ROSPR outperforms ATTEMPT on 7 out of 10 tasks and 1.21 % points on the mean accuracy of 10 tasks. This shows that RoSPR is more applicable for efficient adaptation because it requires times fewer additional param- eters compared to ATTEMPT and also does not require any further fine-tuning of the target task.
5 Experimental Results
INTER and VAR enhance the performance of ROSPR. We also analyze the effect of in- troducing variants of RoSPR: interpolation of soft prompts (INTER) and variance-based ranking (VAR) in Table 1. First, applying INTER shows sim- ilar accuracy compared to ROSPR. However, as shown in the last column of Table 1, INTER reduces the standard deviation of TO and RoSPR by 8.84% while improving the mean accuracy of TO, indicat- ing increased robustness to different surface forms of evaluation prompts. This indicates that interpo- lation of multiple source embeddings outperforms a single source embedding retrieval, aligning with the result of Asai et al. (2022). Applying VAR with TO+RoSPR improves both zero-shot accuracy and robustness of TO+RoSPR, showing that consider- ing the answer choice distribution is beneficial for zero-shot setting, aligned with results from Zhao et al. (2021); Shi al. (2022). Moreover, applying both VAR+INTER results in the highest overall av- erage accuracy, outperforming TO by 2.18% points by largely reducing the gap between larger LLMs.
ROSPR enhances the performance of TO ef- ficiently. Table shows the performance of the 11 evaluation datasets. TO+ROSPR outper- forms TO on 10 datasets among the 11 evaluation datasets. Specifically, TO+RoSPR outperforms TO on RTE (+6.99% points), CB (+4.12 1.4 points), ANLI RI (+3.24% points), and COPA (+2.87% points). This shows that soft prompt retrieval as- sists hard prompts for zero-shot task generaliza- tion with negligible number of additional param- eters (0.007%) 7. Also, while TO outperforms GPT-3 on 3 datasets (RTE, StoryCloze, WiC), TO+RoSPR additionally outperforms GPT-3 on 2 datasets (ANLI R1 and CB) and enlarging the score gap for RTE, StoryCloze and WiC.
"One exception is WSC, which is a binary classification task (yes/no) predicting whether the reference of the pronoun is conrect. We observed that the evaluation data of this dataset has unbalanced labels. containing over 60% of "No" labels. This might be the reason why TO-11B underperforms TO-3B only on this dataset (Sanh et al., 2021). Indeed, predicting only "No" on this datasct outperforms TO-1 1B (63.46>>61 61.4).

---

1.0
9.61
0.045
0.015
9.8
45
Mean Acc
Target Thisk
40
To+RoSPr 0-shet
O-shtt
Congf
6.062
0.31
0
0.25
0.25
0.12
0
9.4
35
GPT:3 0-shor
L-shet
0.
GPT33 3-shot
VSDD
6.2
0.8
30
10
10
Senk
Para
Toplc
Sum
52T
MOA
EDA
CBOA
-0.0
Model Parameters
Source Task
Figure 3: Mean accuracy of datasets of BIG-bench, evaluate on a single prompt following Sanh (2021). By only adding 0.007% parumeters TO-3B, TO+RoSPR largely reduces the performance gap between times larger TO-11B, The full result is provided in Appendix
Figure 4: Frequency of source lask lypes (x-axis) that maxi-
mizes (i.c. ORACLE) the accuracy of each target task (y-axis).
in the best performance (ORACLE) for the given prompts of the target tasks (visualized in Figure 4), From this figure, we can observe a few patterns: paraphrase task assists NLI and word sense disam- biguation task while multi-choice QA (MQA) task assists sentence completion task. For coreference resolution task, various source task types (para- phrase, summarization, multi-choice QA) assist the target task.
The effect of ROSPR generalizes to challenging tasks. ROSPR is also effective for challenging tasks such tasks from BIG-hench benchmark. As shown in Figure 3, TO+ RoSPR improves the mean accuracy performance of TO-3B by 2.39% points while only adding 0.007% additional parameters. TO+RoSPR also outperforms 60 times larger zero- shot and 1-shot GPT-3 and largely reduces the performance gap between 4 times larger TO-1 1B (1.84% points) or 60 times larger 3-shot GPT-3 (0.53% points). Applying INTER with TO+RoSPR results in additional mean accuracy enhancement, outperforming TO-3B by 2.67% points
Misaligned
Aligned
RoSPr
80
80
66
78
62
80
74
6 Analysis of ROSPR
(a) RTE
(b) COPA
(c) WiC
Zero-shot task adaptation of LMs is often seen as a problem of task location, locating the target task to where the model can solve it using the intrinsic ability obtained at pretraining stage with the aid of prompts and demonstrations (Reynolds and Mc- Donell, 2021). In this section, we analyze which factors contribute to the performance enhancement in the perspective of identifying better task location. We find that although the target task performance depends on the source task types, heuristic features such as the answer choice format are more impor- tant. This agrees with previous findings that a meta- trained LM focuses simple features such as the label space, the input distribution, and sequence format, instead of complex semantics (Webson and Pavlick, 2021; Mín al., 2022).
Figure 5: Effect of answer choice format alignment across different target datasets (RTE, COPA, WiC). We report the mean accuracy of the evaluation prompts and the performance of TO is shown in green dotted line.
A Anser choice format is important for task loca- tion. Wc also analyze the effect of using different answer choice formats with the same source task. Answer choice format decides how the available answer choices are given to the LM through the input. For example, a prompt that requires classi- fying movie review into good/bad has different answer choice format from classifying it into posi- tive/negative.
We experiment on 3 datasets (RTE, COPA, WiC) which correspond to different tasks (NI.I, sentence completion, word sense disambiguation) respec- tively. For each dataset, we select source dataset that is retrieved the most for ORACLE. Among the source prompts of the selected source dataset, we select a prompt that has the same answer choice format as the target task (ALIGNED) and another prompt that has different answer choice format (MISALIGNED). Figure shows the effect of an- swer choice format alignment on the target task
Target task performance depends on source task types. To analyze the effect of different source task types on each target task, we measure the fre- quency ratio of each source task type that results
WWo observe that applying VAR results in the same per- formance as TO=RoSP& because frequency of retrieval has much more influence than variance for BIG-bench tasks.

---

performance by comparing ALIGNED and MIs- ALIGNED. The result shows that for all datasets, ALIGNED significantly outperforms MISALIGNED. This result non non-trivial considering that the two prompt embeddings are trained on the same source training dataset and the same training configuration, with the only difference in the given answer choice format, implying that how the answer choices are given to solve specific task is more important than the content of the training data for task location. ROSPR is mostly comparable to ALIGNED em- bedding, implying that retrieving a source prompt embedding by searching for similar input instances results in retrieving a source embedding with simi- lar answer choice formats.
that embeddings from source tasks requiring simi- lar knowledge the target task may be important. Counterintuitively, for both RTE and WiC target tasks, when the answer choice format is aligned, the task source embedding of sentiment classifi- cation, which is known to be irrelevant to RTE and WiC (Pruksachatkun et al., 2020), outperforms other embeddings sourced from datasets that are more relevant the target datasets (paraphrase and multi-choice QA) (Appendix G). This implies that for retrieval of source embedding for task location, answer choice format is more important than con- taining similar knowledge required to solve the target task.
Role of ROSPR is similar to in-context learning. From the findings explained in previous paragraphs, we can conclude that although the source task types influence the target task performance, retrieving a similar answer choice format is more important for task location. Indeed, source tasks containing similar knowledge can help target tasks only if the answer choice formats are aligned to the tar- get task. These findings support Min al. (2022); Webson and Pavlick (2021) that meta-trained LM "takes less effort" to understand the input: models exploit the simple aspects prompts and demon- strations such as the format and distribution instead of complex semantics. Especially, for in-context learning, Xie et al. (2021): Min et al. (2022) show that the role of demonstrations lies in providing the shared concept and distribution hints of the target task. From this aspect, the role of ROSPR is similar demonstrations. However, it is more efficient than including demonstrations because it avoids heavy computation at inference from long sequence lengths (Liu et al., 2022a; Choi et al., 2022) since RoSPR prepends a fixed length of pre- fix tokens regardless of the task. Also, ROSPR is free from the instability of in-context learning coming from different orderings of demonstrations (Lu et al., 2022; Zhao et al., 2021). Lastly, we conjecture that ROSPR also has the benefits of soft prompts (Li and Liang, 2021) such having more expressiveness.
Paraphrase
Sentiment
QA
RTE
49
Misaligned
Aligned
(b) WIC
Figure 6: RTE (Top) and WiC (Bottom) evaluation result by retrieving MISALIGNED and ALIGNED answer choice format across various source tasks. We report the mean accuracy of the evaluation prompts and the performance of TO is shown in green dotted line.
We additionally analyze the effect of answer choice formats on RTE and WiC datasets by re- trieving prompt embeddings trained on various source tasks. Both target datasets have the answer choices of cither yes/no. Similar to the previous experiments, retrieve ALIGNED (yes/no format) and MISALIGNED prompt embeddings across three source tasks: paraphrase, sentiment classification, and multi-choice QA. As shown in Figure 6, for both target datasets, ALIGNED outperforms MIS- ALIGNED across all three source tasks. This shows that aligning to the answer choice format of the tar- get task is crucial regardless of the retrieved source task.
7 Ablation Studies
In this section, analyze the effect of the number of (1) prompts, (2) source datasets, and (3) queries sampled for evaluation. We evaluate variations of our proposed methods on 4 datasets: RTE for NLI, COPA for sentence completion, Winogrande
Answer choice format is more important than task similarity. From Figure 6, we can see that all three source tasks benefit from aligning to the target task answer choice format. One may think

---

04-
the number of task clusters is sufficient some extent, scaling the number of souree prompts per dataset is more crucial than scaling the number of source datasets per task cluster.
Noun ALz
42
Hean
60
Datasct
59-
More sampled queries improve the performance. We also analyze the effect of the number of query instances sampled at inference for retrieval. As seen in Figure 7b, increasing the number of queries results in higher mean accuracy. This is different from the analysis of Lin et al. (2022) that sampling more queries leads to better performance only to some point. Because we use the frequency of each prompt embedding candidate the default metric for retrieval, utilizing more query instances would represent the evaluation data more accurately, sulting in reduced number of wrong retrievals.
Inttracion
11200
22000
Source Prampt Libuary Size
Number of Querics
(a) Datasets vs Prompts
(h) Query variation
Figure 7: Result of various ablation settings RoSPR. (a) compares the effect of sealing the number of datasets with scaling the number of prompts and (b) shows the effect of the number sampled queries at inference. Additional ablation results are shown in Appendix D.
for coreference, and WiC for word sense disam- biguation task. We report the average of the mean accuracy of all evaluation prompts for each dataset by running 3 different runs.
Conclusion
Scaling number of prompts vs. number of datasets. Recent works on meta-trained LMs show that the number of source datasets and prompts is an important factor for zero-shot task generalization (Sanh al., 2021; Wei et al., 2021; Wang et al., 2022b; Chung ct al., 2022). We also show ablations for ROSPR and measure how the zero-shot generalization performance changes when we vary the number of prompts and datasets available during the prompt tuning stage (shown in Figure 7a). First, we vary (1) the total number of source prompts by 60, 120, and 230 by increasing the number of prompts per dataset and (2) the num- ber of datasets by 8. 16, and 30 by increasing the number of datasets per task cluster." Note that in (1), the total number of datasets is fixed while in (2), we use all available prompts for each dataset while varying the number of datasets per task cluster. In contrast to (I), (2) does not always lead to linearly increasing performance boost; the per- formance saturates as more source datasets are in- cluded. By comparing the effect of scaling datasets and scaling prompts for similar Source Prompt Li- brary sizes, we observe that the number of prompts has more impact on the accuracy of the target task (Figure 7a).
In this paper, we introduce ROSPR, method that efficiently enhances zero-shot generalization capa- bilities of meta-trained LM by retrieving prompt- specific source prompt embeddings (soft prompts) for a given target task. We accomplish this by first training the soft prompts for hard prompt of the source tasks. After training source prompt embeddings, we construct the Source Prompt Li- brary by storing the mean representation of training instances keys and the corresponding prompt embeddings as values. At inference, we search for training instances stored the library similar to sample instances from the target task, retrieve the corresponding prompt embedding, select the most frequently retrieved embedding, and append it to each of the target task instances for prediction. Our results show that RoSPR efficiently enhances the zero-shot performance of the backbone model while introducing minimal additional parameters during inference. We additionally provide analy- sis of which factors attribute to the performance of RoSPR and find that heuristic cues such as the answer choice format are critical for generaliza- tion performance, implying that it may play a role similar to demonstrations in in-context learning.
This ablation study also supports the analysis of the previous section; diverse answer choice for- mats of prompts, which are mostly influenced by the total number of source prompts, are more im- portant than source task types which are influenced by the number of source datasets." 10 Therefore,
Limitations
Although we show the effectiveness of RoSPR by applying it on TO-3B (Sanh et al., 2021), we did
the number of datasets increases, we find that answer choice formats are similar across the same task type, meaning that the diversity of answer choice formats is not inereased by increasing the numher of datascts per task clusters.
"Task cluster defined as a cluster of the same task types. 10 Althhough the total number of prompts also increases as

---

not evaluate our method on different model scales such as the TB-1 variant and other LM archi- tectures such as decoder-only LMs due to limited computational resources. This leaves future works on applying RoSPR to even larger LMs and diverse LM architectures (Wang et al., 2022a). Moreover, it is hard to apply VAR to target tasks without answer choices such as free-form generation because vari- ance among options cannot be obtained. However, RoSPR and ROSPR+INTER can still be utilized and we leave applying ROSPR on zero-shot task location of frre-form generation as future work (Scialom et al., 2022).
Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901.
Eunbi Choi, Yongrae Jo, Joel Jang, and Minjoon Seo. 2022. Prompt injection: Parameterization of fixed inputs.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sehastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint wwXiv:224.02311.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddharthha Brahma, al. 2022. Scaling eestruction-inetunee language models. arXiv preprint arXiv:2210. 1416.
Acknowledgements
We thank Sohee Yang and Eunbi Choi for help- ful feedback on the paper. This work was partly supported by Institute of Information & communi- cations Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) NNo.2022-0-00266 Comprehensive Video Under- standing and Generation with Knowledge-based Deep Logic Neural Network, 80%: No.2019-0- 00075, Artificial Intelligence Graduate School Pro- gram (KAIST), 20%).
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment chal- lenge. In Machine learning challenges workshop, pages 177-190. Springer.
Marie-Catherinn De Marneffe, Mandy Simons, and Ju- dith Tonhauser. 2019. The commitmentbank: Inves- tigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung, volume 23, pages 107-124.
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Alek- sander Wawer. 2019. SAMSum corpus: A human- annotated dialogue dataset for abstractive summa- rization. In Proceedings of the 2nd Workshop on New Frontiers Summarization, pages 70-79, Hong Kong, China. Association for Computational Linguis- tics.
References
Akari Asai, Mohammadreza Salehi, Matthew E Peters. and Hannaneh Hajishirzi. 2022. Attentional mix- tures of soft prompt tuning for parameter-efficient multi-task knowledge sharing. arXiv preprint arXiv:2205.1196. 11961.
Stephen Bach, Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai. Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-david, Canwen Xu, Gun- jan Chhablani, Han Wang, Jason Fries, Maged AI- shaibani. Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang. Dragomir Radev, Mike Tian-jian Jiang, and Alexander Rush. 2022. Prompt- Source: An integrated development environment and repository for natural language prompts. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstra- tions, pages 93-104, Dublin, Ireland. Association for Computational Linguistics.
David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. 2003. English gigaword. Linguistic Data Consor- fium, Philadelphia, 4(1):34.
Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang. 2022. PPT: Pre-trained prompt tuning for few-shot learning. In Proceedings of the 60th Anmual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8410-8423, Dublin, Ireland. Association for Computational Linguistics.
Lifu Huang. Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Cosmos QA: Machine reading comprehension with contextual commonsense rea- soning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-LJCN) 2391 2201, Hong Kong, China. Association for Com- putational Linguistics.
Max Bartolo, Alastair Roberts, Johannes Welbl, Sebas- tian Riedel, and Pontus Stenetorp. 2020. Beat the AI: Investigating adversarial human annotation for read- ing comprehension. Transactions of the Association for Computational Linguistics, 8:662-678.
pages
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with gpus. IEEE Trunsactions on Big Data, 733:5335-547.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind

---

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shol reasoners. arXiv preprint arXiv:2205.1 11916
Chonghua Liao, Yanan Zheng, and Zhilin Yang. 2022.
Zero-label prompt selection.
Bill Yuchen Lin, Kangmin Tan, Chris Miller, Beiwen Tian, and Xiang Ren. 2022. Unsupervised cross- task generalization via retrieval augmentation. arXiv preprint arX1v:2204.07937.
Andrew K Lampinen. Ishita Dasgupta, Stephanie CY Chan, Kory Matthewson, Michael Henry Tessler, Antonia Creswell, James L McClelland, Jane X Wang, and Felix Hill. 2022. Can language models learn from explanations in context? arXiv preprint arXiv:2204.02339.
Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen. Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. 2020. CommonGen: A constrained text gen- eration challenge for generative commonsense rea- soning. In Findings of the Association for Computa- tional Linguistics: EMNLP 2020, pages 1823-1840, Online, Association for Computational Linguistics.
Rémi Lebret, David Grangier, and Michael Auli. 2016. Neural text gencration from structured data with ap- plication to the biography domain. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 1203-1213, Austin, Texas. Association for Computational Linguistics.
Kevin Lin, Oyvind Tafjord, Peter Clark, and Matt Gard- ner. 2019. Reasoning over paragraph effects in situ- ations. In Proceedings of the 2nd Workshop on Ma- chine Reading for Question Answering, pages 58-62, Hong Kong, China, Association for Computational Linguistics.
Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, D. Kontokostas, Pablo N. Mendes, Scbastian Hell- mann, M. Morsey, Patrick van Kleef, S. Auer, and C. Bizer. 2015. Dbpedia- a large-scale, multilingual knowledge base extracted from wikipedia. Semantic Web, 6:96
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mo- hta, Tenghao Huang, Mohit Bansal, and Colin Raffel. 2022a. Few-shot parameter-efficien fine-tuning is better and cheaper than in-context learning, arXiv preprint arXiv:2205.05638
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficiee prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022b. What makes good in-context examples for GPT-3? In Proceedings of Deep Leurning Inside Oul (DeeLIO 2022): The 3rd Workshop on Knowledge Extrac- tion and Integration for Deep Learning Architectures. pages 100-1 14, Dublin, Treland and Online. Associa- tion for Computational Linguistics.
Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thir- teenth international conference on the principles of knowledge representation and reasoning.
Yao La, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few- shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 1086-8098, Dublin, Ireland. Association for Compu- tational Linguistics.
Quentin Lhoest, Albert Villanova del Moral, Yacine Jemniee, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Saško, Gun- jan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger. Clément Delangue, Théo Matus- sière, Lysandre Debut, Stas Bekman, Pierric Cis- tac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. 2021. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing: System Demonstrations, pages 184, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Andrew L. Maas. Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. 201 I. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon. USA. Association for Computational Lin- guistics.
Julian J. McAuley and Jure Leskovec. 2013. Hidden factors and hidden topics: understanding rating di- mensions with review text. In Seventh ACM Confer- ence on Recommender Systems, RecSys '13, Hong Kong, China, October 12-16, 2013, pages 165-172. ACM.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Longuage Processing \Volume I: Long Papers), pages 4582- 4597, Online. Association for Computational Lin- guistics.
Sewon Min, Mike Lewis, Luke Zettlemoyer, and Han- naneh Hajishirzi. 2021. Metaicl: Learning to learn in context. arXiv preprint arXiv:21 15943.

---

Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis. Hannaneh Hajishirzi, and Luke Zettle- moyer. 2022. Rethinking the role of demonstra- tions: What makes in-context learning work? arXiv preprint arXiv:2202. 128377
Mohammad Taher Pilehvar and Jose Camacho-Colladee 2019. WiC: the word-in word-in-contee in-context dataset for evalu- ating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguisties: Humun Language Technologies, Volume 1 (Long and Short Papers), pages 1267-1273, Minneapolis, Minnesota. Association for Computa- tional Linguisties.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022. Cross-task generaliza- tion via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 47703387,, Dublin, Ireland. Association for Computational Linguisties.
Yada Pruksachatkun. Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang. Clara Vania, Katharina Kann, and Samuel R. Bow- man. 2020. Intermediate-task transfer learning with pretrained language models: When and why does it work? In Proceedings of the 58th Annual Meeting of the A Association for Computational Linguistics, pages 5231-5247, Online. Association for Computational Linguistics.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839-849, San Diego, California. Association for Computational Linguis- tics.
Guanghui Qin and Jason Eisner. 2021. Learning how to ask: Querying LMs with mixtures of soft prompts. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 5203-5212, Online, Association for Computa- tional Linguistics.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for ex- treme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1797-1807, Brussels, Bel- gium. Association for Computational Linguistics.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susan- nah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXh::2112.11446
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language under- standing. In Proceedings of the 58th Annual Meet- ing of the Association for Computationnal Linguistics, pages 1885-49011 Online. Association for Computa- tional Linguisties.
Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Explain your- self! leveraging language models for commonsense reasoning. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 4932 29944992, Florence, Italy, Association for Computational Linguistics.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow in- structions with human feedback. arXiv preprint a5Xiv:2203.02155.
Laria Reynolds and Kyle McDonell. 2021. Prompt pro- gramming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Comput- ing Systems, pages 1-7.
Bo Pang and Lillian Lee. 2005. Seeing stars: Exploit- ing class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd Annual Meeting of the Association for Compu- tational Linguistics (ACI. '05), pages l 155-144 Ann Arbor, Michigan. Association for Computational Lin- guistics.
Melissa Roemmele, Cosmin Adrian Bejan, and An- drew S Gordon, 201 Choice of plausible alter- natives: An evaluation of commonsense causal rea- soning. In AAAI spring symposium: logical formal- izations of commonsense reasoning, pages 90-95.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis. Majid Yazdani, Nicola De Cao, James Thore,, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2021. KILT: benchmark for knowledge intensive language tasks. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2523-2544, Online. Association for Computational Linguistics.
Anna Rogers, Olga Kovaleva, Matthew Downey, and Anna Rumshisky. 2020. Getting closer to ai complete question answering; A set of prerequisite real tasks, Proceedings of the AAAI Conference on Artificial Intelligence, 34(05)88722-873).
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2021. Learning to retrieve prompts for in-context learning. arXiv preprint arXiv:21

---

Amrita Saha, Rahul Aralikatte, Mitesh M. Khapra, and Karthik Sankaranarayanan, 2018. DuoRC: Towards complex language understanding with paraphrased reading comprchension. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 1683- 1693, Melhourne, Australia. Association for Compu- tational Linguistics.
9th International Joint Conference on Natural Lan- guage Processing JEMNLP-JJCNLL pages 6076- 6085, Hong Kong, China. Association for Computa- tional Linguistics.
Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou', and Daniel Cer. 2022. SPoT: Better frozen model adaptation through soft prompt transfer. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguisticy (Volume I: Long Pa- pers), pages 5039-5059, Dublin, Ireland. Association for Computational Linguistics.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavar- ula, and Yejin Choi. 2021. Winogrande: An adver- sarial winograd schema challenge at scale. Commu- nications of the ACM, 64(9)999-106.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman, 2018. GLUB: A multi-task benchmark and analysis platform for nat- ural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium. Association for Com- putational Linguistics.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven L. Scao, Arun Raja, et al. 2021. Multitask prompted training en- ables zero-shot task generalization. arXiv preprint 10.00077.
Maarten Sap. Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. Social IQa: Com- monsense reasoning about social interactions. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Procexsing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP), pages 4463- 4473, Hong Kong, China. Association for Computa- tional Linguistics.
Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, and Colin Raffel. 2022a. What language model architecture and pretraining objective work best for zero-shot generalization? arXiv preprint arXiv:2204.0558
Xinyi Wang, Wanrong Zhu, and William Yang Wang. 2023. Large language models are implicitly topic models: Explaining and finding good demon- strations for in-context learning. arXiv preprint arXiv:2301.119166
Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan. 2022. Continual-tO: to: Progressively instruct- ing 50+ tasks to language models without forgetting. arXiv preprint arXiv:2205.123395
Yizhong Wang. Swaroop Mishra, Pegah Alipoor- molabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap. et al. 2022b. Benchmarking generalization via in-context instructions on 1, 1,600+ language tasks, arXiv preprint xxXv:2204..77055
Weijia Shi, Julian Michael, Suchin Gururangan, and Luke Zettlemoyer. 2022. Nearest neighbor zero-shot inference. arXiv preprint arXiv:2205.133:
Aarohi Srivastava, Abhinay Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso,
al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint 00X6v52206.06655.
Zifeng Wang, Zizhao Zhang. Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. 2022c. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 139-149.
Yusheng Su, Xiaozhi Wang, Yujia Qin. Chi-Min Chan, Yankai Lin, Huadong Wang, Kaiyue Wen, Zhiyuan Liu, Peng Li, Juanzi Li, Lei Hou, Maosong Sun, and Jie Zhou. 2022. On transfcrability of prompt tuning for natural language processing. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages Seattle, United States. Association for Computational Linguistics.
Albert Webson and Ellie Pavlick. 2021. Do prompt- based models really understand the meaning of their prompts? arXiv preprint 17..02210..01247.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu. Brian Lester, Nan Du, An- drew M Dai, and Quoe V Le. 2021. Finetuned lan- guage models are zero-shot learners. arXiv preprint arxiv:2109.01652.
Niket Tandon, Bhavana Dalvi, Keisuke Sakaguchi, Pe- ter Clark, and Antoine Bosselut. 2019. WIQA: A dataset for wwhat if..." reasoning over procedural text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le. and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.

---

Johannes Welbl, Nelson Liu, and Matt Gardner. 2017. Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy User- generated Text, pages 94-106, Copenhagen, Den- mark. Association for Computational Linguistics.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In In- ternational Conference on Machine Learning, pages 12697-12706. PMLR.
Ruiqi Zhong, Kristy Lee, Zheng Zhang, and Dan Klein. 2021. Adapting language models for zero-shot leam- ing by meta-tuning dataset and prompt collections. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2856-2878, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Constructing datasets for multi-hop reading comprehension across documents. Transactions of the Association for Computational Linguistics, 6:287- 302.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2021. An explanation of in-context learn- ing implicit bayesian inference. arXiv preprint arXiv:2111.02000.
Yongchao Zhou, Andrei Toan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023. Large language models are human-level prompt engineers.
Hanwei Xu, Yujun Chen, Yulun Du, Nan Shao, Yang- gang Wang, Haiyu and Zhilin Yang. 2022. Zero- prompt: Scaling prompt-based pretraining to 1,000 tasks improves zero-shot generalization. arXiv preprint arXiv:2011.06910
Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. WikiQA: A challenge dataset for open-domain ques- tion answering. In Proceedings the 2015 Con- serence on Empirical Methods in Natural Language Processing, pages 2013-2018, Lishon, Portugal. As- sociation for Computational Linguistics.
Seonghyeon Ye, Doyoung Kim, Joel Jang, Joongbo Shin, and Minjoon Seo. 2022. Guess the instruction! making language models stronger zero-shot learners. arXiv preprint arXiv:2210.02669
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a ma- chine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 4791-4800, Florence, Italy. Association for Computational Linguistics.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015a. Character-level convolutional networks text clas- sification. In Advances in Neural Information Pro- cessing Systems, volume 28, Curran Associates, Inc.
Xiang Zhang. Junbo Jake Zhao, and Yann LeCun. 2015b. Character-level convolutional networks for text clas- sification. In Advances in Neural Information Pro- cessing Systems 28: Annual Conference Neural In- formation Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 649-657.
Yuan Zhang, Jason Baldridge, and Luheng He. 2019. PAWS: Paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 1298- 1308, Minneapolis, Minnesota. Association for Computa- tional Linguistics.

---

A Comparison with Hard Prompt Optimization
D Additional Ablation Studies
We have conducted additional experiments to com- pare our method with 3 different hard prompt op- timization techniques including (1) RoHPr which utilizes the same search technique as RoSPr but retrieves 4 corresponding training instances for in- context learning instead of the corresponding soft prompt, and (2) APE (Zhou al., 2023) which is an automatic prompt engineering method that utilizes the generation results of an LLM (Ouyang et al., 2022). known to show better performance than human instructions, and (3) ZPS (Liao et al.. 2022) which selects an optimal hard prompt from prompt pool using prompt ensemble and pseudo- labels. As shown in the result of Table 2, on av- erage, RoSPr performs the best average, show- ing the benefits of using soft prompts over hard prompts for task generalization. While RoSPr shows improvement on 10 out of datasets, other hard prompt optimization methods do not show consistent improvements.
64
63
62
Mean Acc
61
60
59
+oSPr+Var
RoSPr
58
RoSPr+ Var+
116
230
Figure 8: Variation of number of prompts by increasing the number of prompts per dataset.
64
62
Mean Acc
61-
B Full Result of BIG-bench Evaluation
60
RoSPr
RoSPr+Var
We provide the task generalization performance result of 14 tasks from BIG-bench. shown in Table 3. Applying ROSPR largely improves the perfor- mance for 3 datasets (Hindu Knowledge, Novel Concepts, Misconceptions): (+17.72% +3.12% +1.82%) compared
59
RoSPr+ Inter
RoSPr+Var+Inee
58
16
29
Figure Variation of number of datasets by increasing the number of datasets task cluster.
TO-3B and (+13.72% +3.12%, +2.05%) compared to TO-11B. For mean accuracy of 14 tasks, TO+RoSPR outperforms TO- 3B by 2.39% points, reducing the gap between times larger TO-1 IB to 1.84% points. Moreover, applying INTER to TO+RoSPR enhances the per- formance of TO+RoSPR for most tasks, indicating that interpolation of multiple embeddings is effec- tive for challenging tasks.
We provide detailed result of variation of the number of prompts (Figure 8) and the number of datasets (Figure 9). We additionally analyze the effect of (1) different sampling methods for constructing the Source Prompt Library, (2) the number of instances sampled for constructing the Source Prompt Library, (3) the number of top-N retrieval for embedding retrieval, and (4) the num- ber of multiple source embeddings to interpolate. Same Section 7, we report the mean accuracy of 4 evaluation datasets: RTE, COPA, Winogrande, and WiC with 3 runs with different random seeds for the sampling evaluation queries.
Fine-tuning Baseline Details
For fine-tuning baseline models (PT and AT- TEMPT), we follow the training configuration of source prompt tuning. We train each target prompt with a single epoch using a maximum of 5,000 training instances. For fine-tuning, we use batch size of 32 and learning rate of lc-3. Also, we ran- domly select hard prompts (templates) of the train- ing dataset during fine-tuning. For ATTEMPT. we randomly sample one source prompt per task cluster, resulting in interpolation between soft prompts.
D.1 Sampling Methods for Source Prompt Library
We experiment three different methods to sample instances for constructing Source Prompt Library and analyze the effect of each method. By default, we choose RANDOM method, where we sample

---

RTE
CB
AN.RI
AN.R2
AN.R3
COPA
Hellasw.
StoryC.
Winogr.
WSC
WIC
AVG
TO
64.55
45.36
33.81
33.11
33.33
75.88
26.60
84.03
50.97
65.10
50.69
51.22
RoHPr
47.10
45.00
33,53
33.30
33.09
73.38
26.65
86.93
50.51
64.42
50.00
49.45
APE (Zhou et al. 2023)
68.19
48.33
35.82
35.37
34.27
68.00
25,76
78.73
50.43
58.75
50.85
50.41
ZPS (Liao et al,, 2022)
58.48
60.71
36.60
34,40
33.33
76.00
28.49
87.39
51.78
64,42
50.63
52.93
RoSPr
71.54
49.48
37.05
34.64
33.92
78.75
26.97
85,52
51.50
64.52
51.76
53.24
Table 2: Comparison result of RoSPr with different hard prompt optimization techniques.
T0-3B
RoSPR
INTER
TO-1
all average as d, in increasing order, sample {dl di 41/10000 d d1+2-N/100 -.. N1+99~N/100). as- suming there are total N training instances in a dataset.
Strategy.
52.79
52.05
52.23
52.75
Movie D.
52.85
51.45
52.23
53.69
Known Un.
47.83
47.83
47.83
58.70
Logic Grid
41.10
37,00
37.40
38.30
As shown in Figure 10, RANDOM method outper- forms CLUSTERING and DISTRIBUTED methods. Interestingly, CLUSTERING method significantly hurts the performance all 4 proposed methods, suggesting that storing similar instances per prompt results in retrieval failures more often. Also for DISTRIBUTED method, most of the methods sig- nificantly underperform RANDOM, except INTER and VARP*+VARINNTER From these results, we can conclude that random sampling represents the source dataset most effectively.
Hindu Kn.
25.71
43.43
45.71
29.71
Code
46.67
45.00
40.00
43.33
Concept
45.52
67.61
67.61
69.29
Language
14.84
13.68
14,40
20.20
Vitamin
58.88
53.71
54.53
64.73
Syllogism
52.94
50.64
51.34
51.81
Misconcept.
50.23
52.05
52.05
Logical
46.64
54,86
54,86
54,86
Winowhy
44.29
44.33
44.29
52.11
Novel Con.
15.63
15.63
18.75
15.63
AVG
42.56
44.95
45.23
46.79
Table 3: Evaluation result of 14 tasks from BIG-bench (Srivastava et al., 2022).
64
63-
62
Mean Acc
RoSPr Inter W/ Var & Inter
82
60
60
RoSPr
59
RudPv
RusPy Var-re
59
Random
Clustering
Distributed
58
300
500
Figure 10: Different instance sampling methods for constructing Source Prompt Library.
Figure 11: Variation of number of instances sampled for constructing Source Prompt Library. Default setting is 100.
100 instances by random for each prompt by as- suming that each random 100 queries of instances can represent the whole prompt. Second, we exper- iment CLUSTERING method, where we sample top 100 instances which has the closest mean represen- tation from its overall average for each prompt. If we say each instance has distance of mean repre- sentation from its overall average as d; in increas- ing order, sample {d1, d2, .... d1n0}. The last method
D.2 Number of Instances Sampled for
Constructing Source Prompt Library
We analyze the effect of size of the Source Prompt Library by varying the number of instances n to sample for each hard prompt by 100, 300, 500. Therefore, n x (number of total hard prompts) would be the size of the Souree Prompt Library. As shown in Figure 11, increasing the number of sam- pled instances does not increase the performance; it hurts the performance for most cases. This suggests that only few number of training instances are enough to represent the distribution of prompted
is DISTRIBUTED method, where we sample 100 instances in distributed way with respect its distance of mean representation from its overall average. we say each instance has a distance mean representation from its over-

---

64
63
63
Mean Acc
Mean Acc
61
61-
RoSPr
60
RoSPr+Var
RoSPr+ Inter
59
RoSPr+Inter Inter
Var+
RoSPr+ Var+
58
10
20
58
1
Figure 12: Variation of number of Top N instances for embedding retrieval. Default setting 10.
Figure 13: Variation of number of source embeddings for interpolation based methods. Default setting K' 3.
input (hard prompt input instances) for each hard prompt and increasing the number sometimes hurt the performance by adding noise the distribution. This also supports the importance of heuristic cues in Section 6 by showing that adding more training instances per hard prompt does not increase the performance. Instead, adding hard prompts with diverse answer choice formal is more important.
in Table 1. Mean accuracy would increase if we add other datasets for evaluation that benefit from interpolation such as WSC and CB.
By comparing among various N' values, we find that for ROSPR+INTER, the accuracy substantially decreases for K' = 2, implying that the possibil- ity of a wrong retrieval varies depending on the value of N'. In contrast, RoSPR+ VAR+INART is robust to the value of K', showing that suriance-based ranking icreasses robustness dif- ferent numbers of source embeddings for interpola- tion as well.
D.3
Number of Top N instances for Embedding Retrieval
We vary the number of top-N instances that are re- trieved given each query through MIPS search. As shown in Figure 12, varying the number of top-N instances does have much effect compared in-
E Visualization of Results
creasing the number of sampled queries (Figure 7b). This implies that if the size of the evaluation set of the target task is large, sampling more queries is effective than searching for more similar instances per query. This is important for variance-bass methods because the number of forward passes needed before evaluation is proportional Q N. Therefore, we can reduce the time latency by re- ducing the number of instances retrieved per query without hurting the performance much.
We show the visualization of the evaluation result 11 datasets in Figure 14. Methods based RoSPR not only show higher accuracy, but lower variance for many datasets.
Experimental Configurations
As mentioned in the previous sections, we use TO- 3B our backbone meta-trained L.M. For prompt tuning, fix the prefix length 100 and the em- beddings are initialized from 5,000 most common vocabulary following Lester al. (2021). We train each source embedding for a single epoch with a learning rate of 0.1 and a batch size of 32. We use the Adam optimizer with weight decay of le- 5. For retrieval, we randomly sample Q - 32 query instances and retrieve top K 10 exam- ples for each query. We train TO-small variant (~ 35M params) as dense retriever by multi- task prompted training on T5+LM model (Lester et al., 2021), replicating the original training setting of TO by training T5+LM for epochs using the same training instances of Sanh et al. (2021) with
D.4 Number of Source Embeddings for Interpolation
We analyze the effect of number of source embed- dings for interpolation by varying top-N' interpola- tion from 1 (no interpolation) to shown in Figure 13. By comparing between single prompt embed- ding retrieval (N' multiple embeddings (N' > 1), the mean accu- racy drops by adding multiple source embeddings for retrieval because interpolation-bant based methods underperform on tasks such as COPA as shown
and the interpolation of

---

RTE
(b) CB
(c) ANLI
(d) ANLI R2
10
(e) ANLI
COPA
(g) Hellaswag
(h) StoryCloze
11
(i) Winogrande
(i) WSC
(k) WiC
TO
RoSPr
RoSPr+Inter
RoSPr+Var
RoSPr+Var+inter
Figure 14:
Visualization of evaluation results of 11 datasets.
learning rate of le-3, input sequence length 512, output sequence 128, and batch size of 1024. We select model checkpoint by early stopping based on validation accuracy. We use meta-trained LM instead of naive pretrained model (e.g. Sentence- BERT) because meta-trained LM is shown to be more effective for retrieval (Lin et al., 2022). For the interpolation experiment, we set K' top-K' prompt embedding candidates. For training source prompt embeddings, used 8 V100 GPUs.
intuitively, paraphrase task is more similar the task of word sense disambiguation task or NLL, implying their task similarity, while the task of sen- timent classification very different. However, our counterintuitive result shows the saft prompt to show the best performance in Figure 6 Section 6, bolstering the claim that similar source task types are not major factor for evaluation performance.
for
H
Full List of Source Training and Evaluation Datasets
G Examples of Applying Prompts, Answer Choice Format and Source Task Types
All of our training and evaluation datasets are a subset of datasets used in Sanh et al. (2021). We use Huggingface version of each dataset (Lhoest et al., 2021).
Figure 15 shows an example applying prompt through Promptsource (Bach et al., 2022) as men- tioned Section 3.1.
H.1 Training Datasets
We assert that answer choice format is more important than task similarity in Section 6. We further provide details of the input instances of the mentioned tasks: paraphrase, NLI, word sense dis- ambiguation, and sentiment classification in Figure 16. As supported in Pruksachatkun et al. (2020),
Following Sanh al. (2021), use 8 task clus- ters for training of source prompt embedding: sen- timent classification, paraphrase, topic classifica- tion, summarization, struc-to-text, multiple-choice QA, extractive QA, and closed book QA. We use imdb (Maas et al., 2011). amazon_polarity

---

hx(x
Question 1: What are lanyards used for?
Can an answer to "What are lanyards used for?" Also be used to answer "What lanyard?"?
Question2: What is lanyard?
PromptSource
+
+
0
No h
Vik
Figure 15: Example of applying prompt given instance through Promptsource (Bach al., 2022).
Determine if following two sentences
He got a bang on head. They got great bang of it.
the makes strong case for the importance the musicians creating the motown sound. Did the reviewer enjoy the movie?
paraphrase each other or not.
Question: the word 'bang' used in the same sense in the two sentences above?
Sent 1: In 1951 he died and retired in 1956.
Similar
Not Similar
Sent 2: He died in 1951 and retired in 1956.
Paraphrase
WSD
Sentiment
Determine if the following
Mangla was summoned after Madhumita's 's sister
the film makes a strong case for the importance of the musicians in creating the motown sound. Did the reviewer enjoy the movie?
two sentences
paraphrase each other not.
Nidhi Shukla, who was the first witness in the case. Can we infer that 'Shukla is related to Mangla." "? Yes no?
Sent 1: In 1951 he died and retired 1956. Sent 2: He died 1951 retired 1956.
Similar
Not
Similar
Paraphrase
NLI
Sentiment
Figure 16: Examples of instances of different source tasks.

---

(McAuley and Leskovec, 2013), rotten__oone tomatoes (Pang and Lee, 2005), yelp_rre yelp_review friiewfllll full (Zhang et al., 2015b) for sentiment, glue/qqp (Wang et al., 2018), paws/labeled_iie final (Zhang et al., 2019) for paraphrase, ag_news (Zhang et al., 2015a), dbpedia_ 14 (Lehmann et al., 2015) for topic classification, gigaword (Graff et al., 2003), samsum (Gliwa al., 2019), xsum (Narayan al., 2018) for summarization, common_gent (Lin al., 2020), wiki_bio (Lebret et al., 2016) for struct-to-text, cos_e/v1.11 (Rajani et al., 2019), quail (Rogers et al., 2020), social_i_ge (Sap et al., 2019), wiqa (Tandon et al., 2019), cosmos_gp (Huang et al.. 2019), sciq (Welbl et al., 2017), wiki_ hopporriiii (Welbl et al., 2018) for multi-choice QA, adver- sarial_ quiiid_a// adversarial _qaibbert,,_ adversar- ial qa/droberta, quoref (Bartolo al., 2020), ropes (Lin al., 2019), duorc/SelIRC, duorc/Paraphrase IdentificationRC (Saha et al., 2018) for extractive QA, and kilt_ tasks/hotpotqa (Petroni et al., 2021 I), wiki_qa (Yang et al., 2015) for closed book QA.
I Full List of Retrieved Prompt Embeddings
We provide full list of retrieved prompt embed- dings of RoSPR and ORACLE for all prompts of evaluation datasets. We report retrieval results of single random seed (Table 4 ~ Table 14).
multi news Mieffbbri-ettll20019--e
exclude 6 datasets (MRPC, TREC, DREAM, QuaRTz, QASC, QuaRel) that have small training sets because leads to task imbalance, which is crit- ical for training our small dense retriever (~~ 35M params). We also exclude CNN Daily Mail, App Reviews, and WikiQA dataset due to dataset down- load issues, absence of any test or validation data, and unbalanced label distribution, respectively.
H.2 Evaluation Datasets
Following Sanh et al. (2021), include 11 evalu- ation datasets as follows: RTE (Dagan al., 2005), CB (De Marneffe et al., 2019), ANLI (Nie al., 2020) for natural language inference task, COPA (Roemmele et al., 2011), 1), Hellaswag (Zellers et al., 2019), Storycloze (Mostafazadeh et al., 2016) for sentence completion task, Winogrande (Sakaguchi et al., 2021), WSC (Levesque et al., 2012) for coreference resolution task, and WiC (Pilehvar and Camacho-Collados, 2019) for word sense disam- biguation task.
For BIG-bench tasks, evaluate 14 tasks, following Sanh et al. (2021): Known Unknown, Logic Grid, Strategy Hindu Knowledge. Movie Dialog, Code Description, Conceptual, Language ID, Vitamin C, Syllogisms, Misconccptions, Logi- cal Deduction, Winowhy and Novel Concepts.

---

Prompe Name
RoSPt
Rerioved Finbokinng
Oaalee
Retrioval Emnhalding
GPT-3 syle
61.37
74.01
pansslbeele, MOIPAAS--NLL GPTS
74.01
MNL crowchource
63.81
70.6
powsiibbeled finalecutestt question
72.20
pueshbelldd imsloonies--ppeiin
haaed the previous passge
68.23
76.53
pesstiikclo, fiullooteet-eeeiii
76.53
Saasiullaal_uuu/PPWWSSS.. GPT)
cun we inter follow that
59.57
73.24
aleeloppineennigg
75.24
pawytabeted_
Sww.PAWS GPP!
71.86
pawsstabcled foallconeett question
71884
pawstiabeldd
www./oonteet--eeoooocc qpestoo label
does this imply
64.62
65.59
pawadibbeled fnullcumeed-quesiico
71.48
guaeaneed
75.81
pussidelad Baw.dlls...//PWW-ANA.LLL GPT)
should assume
06.43
71.12
aleeloppinenung
76,33
paestabeled
www.://ddds///WWWW//// ANLI GPE3
jueified in saying
61.01
58.12
powslibeled fruulppeppraeesiee
71:12
powelabeled Fmilloonienii-uusiinn
must
passstubcled foall question
75,09
pwwstubclol_ffuu/PWWWa GPT3-ao-laed
Arg.
64.55
71.30
13.79
Table 4: List of retrieved source prompts of ROSPR and ORACLE for each evaluation prompts of RTE.
Pronpe Name
TO
RuShy
Ratioved Finhokking
Omcle
Remioved Embeokinn
cal we infer
55.36
51.79
social LoalStow cholces generate indes
67.86
santonn/Was Wine dialogue Shat mutch summary
based on the provicus passuge
64.61
58.95
sccial quiShow cheices and generate indes
69
smmum//riie dallgge that match this summary
claiss ouunlaaeeineesslliiee does follow that
50.00
67.86
socid LggShow choices gencrate index
62.64
64.29
50.00
social
L satStow choices and generate indes
67.86
sansun/Summaite:
justified in arying
53.57
50.00
sucial guSShw cheiassomi and gmurate indes
62.50
smmum chalogue that mutch this simmary
30.29
41.07
secial Leg cholces gesicrat: index
41.07
social SqalShow chokees genecate index
GPT-3 style
51.79
67.86
socid I gusShoww choices and generate indes
69.64
social SSa chokees and genevate unswer
consider bereeseeeeelieeeeeeerr
35.71
35.71
Lalbhtow chotces generate index
39.29
social Ganeine arower
gioranicual Inse
48.21
50.00
sccil guShow cheisssson and genizate indos
64.29
whiil spVieeenne snswer
53.57
50.00
secid sa/Stow cholces geserate index
64.29
social Laaiineeatee
graranieed guermicee/possiieiiin pesieeiieeeeeeeeusselle
8.93
8.9
scial SatShow choices and geterate indes
893
-all same)
does this imply
58,93
51.79
sccid guiShow choians and gencrate indes
66.7
MNLI crondsource
8.95
39.29
social Lealsoww choices generate indes
42.66
cos, _.v. Lllquestion pettestttooootttoo___oo dextrppptoooddddcriiiioooo
should ascune
57.14
50.00
secial LoaSStow choices and generate indes
66.07
cos po__oll1, wwwllllllleedddwee witpt_tt___moooo commonn sense
fullowing
50.00
quSShow choices generate index
71.43
oce Iddescription question opticn
Ang.
45.36
49.40
58.10
Table 5: List of retrieved source prompts of RoSPR and ORACLE for each evaluation prompts of CB.
Pronpe Name
TO
ReSPi
Retrizved Firbokinng
Ocalle
Retrieval Embhskinng
infer
33,90
38.90
pewsstbbeeed_ maalcootest-uueiiaa
29,40
puostatelel_ SHSPAWS--NLI GPT3
based on the previnus passage
33,90
36.50
paws/labeled finalcommest question
38.50
juoshbeeldd fina/WAWS ANLI GPT)
clain mulfaleeinommiiiiee
35.60
36.70
powstibcied SIAUPARS-ANII GPTS
39.10
paw fecillsmnett-ppoiion
dons follow
36.00
40,50
powsiibbled foolcomttssggeeiiinn
40.50
puvvishlal fmsNPAWS--NT GPTS
justiteed in siways/smmeimnehheeee
33.10
38.10
passitbbeeed Inallooaeettttussti questicn
28,50
posshaletsd_aassllonaaaa questiionnnnnooooooo no-label 33.4)
33.40
33:40
possiibeledd fnallpoomlemse--stt
GPT-3 syle
33,80
37.30
powdibbeedd FNLLPAAS.ANLIICO GPTS
38.50
pa PowshalllaL WWNSS GPT3-an4abe)
consider dwwayssomellimeeeeeeee
33.20
33.40
pawsdabeled finARAAW ANLI GPT3
33.50
31.4)
gurrnieed tmse
33.70
38.50
powsiibledd fiollcomeed-ggresiinn
38.00
puwhlsshld fmoMPAWS ANI. GPT)
must be true
34.40
39.60
powstibeeee Iinallooseetttuueriiia
29.70
33,30
33.30
powstibeled JNaUPAAWS.ANLIGFP
33.30
indi/Test Exprevsed Seatiment
does this imply
33,60
38.20
pawsitabeled fnallcozzest questicn
28.20
pawsiisheted fmaticoment question no label
MNI
33,60
33.80
35
depulia 14igiven what category dhes peragneh boling
should aosune
33.20
38.80
pewstabeied_ fnallooustt-guuriii
29.00
paostibbledd www.ttts---NAA GPT3-ao-labe)
take following truth
32,00
37.10
pawsshabelled finalPAws ANLI
38.70
pawshsbeled finallonteet suailionntt-uesttionn question label
33.81
37.07
37.65
Table 6: List of retrieved souree prompts of RoSPR and ORACLE for each evaluation prompts of ANLI
Prompt Name
zo
RoSPt
Xessieved Embedding
Oracle
Betrieved Embedding
infer
30.40
35 30
paws/bbbeled imalkcotedd-gueeee question
35:30
powe/lsheled innlAWS GPT3
based the previous passage
31.40
35.40
powstubbled_ maaleaneettgeeetti
35.40
claim melattennnoncussiee
$1.90
33.10
pawsitabeled InaPPAWS GPTS GPT3
35.90
powsllabeled wwwaloonnattttaaoooeooooooo.. question
fellow
34.00
36
powelsheled finl/AAW--AII
justifeed in saying
33,50
35.70
powstabeied fialloouestggueiiii
35.70
rotion Reo.eieeeeeeReeeeeeeee Enjoyment Yes No
33,00
33:00
pawssabblled finalpsmpleseeeee lack
33.50
adversinal gwddibertbbbedddcoomm
syle
33,50
34.90
pows/ibbled ANTT
35.00
powelibeled finllomeest-gisiiin--iiillll
comsdder wwwny//ammetiinn//eeee
33.70
33:40
ahyylii, 14givem c_cbioo,oof of catoguries
34.50
powelsheldd finslPAAS APP.GGGTTT
gus coteed true
32.90
34.00
passtabeled_ Maaliootest-uueriii.
34.30
must be true
35,10
34.60
pawsiiibeled finalcooest question
35.10
powellbbeed finalAwws ANLIGIT
33.30
33.30
peosdibcied, findkcntiet.gpeninnnn---e
33.30
indloW-iicr Expeccasi Sentment
does this imply
32.70
33.90
pawshabeled_ finallooueet cusssio
34.10
pows/lablled powslabbled_fnailcoeeees www.//ioooeeeeeeetttie
MNL J crnwehonace
33.40
34.20
dhpodin 14ggvmm choior of catepriis
34.00
aepedia 14tgieen chonce of calegories
should ascume
32.40
35.10
pawstubeled_ fnalkenteet-uennia
35.10
powellbeled pow.taachss_Lnuuuuww....o www.s:/w.w...............
tike the following truth
31.60
35.70
powsiibbiled maallpapheessssakk
35.70
33:11
34.70
34.92
Table 7: List of retrieved source prompts of ROSPR and ORACLE for each evaluation prompts of ANLI R2.

---

Prompe Name
TO
ReSit
Retrivved Libeedingg
Omoee
Recrivee Lmbeddinn
con wi infer
33,00
34.75
34.75
penaftheild, Bunleortest-ueeiieee
based
passage
33.53
34.08
pewstakeeed_ hnal question
35.33
powstabelee Boalooonnneeeeeea question label ere elv lbllgged with commun soner
cliim munelaeeiineecuuiiie
$2.83
35 58
pawsiibeledd fnolppmphrase-ist
35.92
does follow that
54,05
34
powsitbblle, fnallconeeet-ueeiii
35.35
amsroo_ Barryy/eeerrrreeeeeeee produut amazon, polazzy! User recommend shis, product
justiteed in saying
33.55
33.00
pawstabeled, pnatppmmphrse task
35.42
33.42
33:42
pawsitabeled hualt task
33.50
ponstabeled jial/parnghr uakk
GPT-3 style
33.33
34.00
poosdbblldd findPAWS ANT
34.92
powslbbuled finlPAW ANLJ cos 2VV1. Hiweeiieeeessseeocpppiia woppoooppoosseeeeo
consider dieaybhonncineeleee
23.05
32:42
copespllinnnn www..ooooddooooddddoo
33.67
52.58
34.0%
pawsstabeled fnnllcontestt question
31.85
powslabeeee fnall question
must
33.83
33.75
pawsfibcild, frublpemprrae--at
35.42
meien Reviice/eeevieee Fnjoyment
gon
33.50
33.50
pawsitabeied panphppngghssse-aaa
33,55
social L SaaSSoow chokces and gesecate unswer
this imply
32.92
33.58
glue/oggmmeennngg
$1.50
powslabeled fnalicmeest question
MNLI crondbource
33,75
33.67
ronen, tomanees// Text Lxperssed Sentiment
31,00
dhpedia, Higives J. chotce of congories
should
33.25
34.67
powddibeel, findlcontest-gpeciee
34.92
powsiibelldd finllinteettuuusiin---
tike the following as truth
33.42
32:75
social L colStow choices and generate indes
36.67
gonshabeled__inee foall cosoosesssssusssssaasse zo label
Aug.
33,33
33.86
34.91
Table 8: List of retrieved source prompts of ROSPR and ORACLE for each evaluation prompts of ANLI R3.
Prompe Name
TO
RoSPr
Retrieval Fmivalling
Onclle
Reriowe Fmnrrading
esereise
80.00
79,00
con.oV1. LoLee...suusstonn iiguesion_oodeen jrtpe_oooooodddsiiiiioooo. descriptooneees
50.00
cos_elv1.1 cos_ehv.11lqueetiig Liqueettoooootiinn detpsiiiootiittdddcriiii.t text
plansble alternstives
$4.00
K3,00
11 option description test
83.00
elv.. LAquestion option desemptiinn
or C21 promise, wxbecuse...
61.00
67.00
LquChekk a random anwer
not
75.00
oo._ IMMecrigtion_pqooiii cesttiioootttoo option,
hosLoption
70.00
76.00
social Shene chricss and genente answer
79.00
elv1.
Aquuettion option deseription lext
more likely
79.00
cos, sV1. & Lgguction_optinn deser_ddscriitioo,
85.00
cos_e/v1. Lildesccipinn questonnoptiio option text
canse effect choose
74.00
78,0 78,00
cuc ev1 Isquestion option description test
83.00
common gen/tadoom task template prompe
K2,00
Iiguesion_ootion, owsttonnessciiiiiinnnnoo
hesitainn
79.00
K2.00
co1_6 111.sstittittiinnn.mmmm Iyspestion_pppenndde pestooppeeecereeeeeooooeo description,eet
$2.00
cos elv1. 1iqquqstooo descripiioscciiiooot.. text
Agg.
75.88
78.75
81.13
Table 9: List of retrieved source prompts of ROSPR and ORACLE for each evaluation prompts of COPA.
Procrpt Name
TO
RuSPy
Retrievad Embholingg
Oncice
Retrieved Fishokling
Predict with biat
26.83
27.70
sociaL Lqu/hhow choices and genzeate index
29.11
cos_etct. 1os_e/tt.1/ppeeto Liquesttoonnotiii http::oooosiiidddcisii....o
Ransomized prompes template
26.87
26.83
social qu/Show choices
index
27.84
wiqu/wbat is www.s/eeessss_eeecmmm Aist__t
complete first then if I..ogiiss_Bor how../owwwwee continues
28.37
27.35
repes/pmumpt botiom no hini
28.35
cos efct. A'pusstion option description lest
25.28
26.58
LuuShow choices and generate index
26.58
oos_ettl. Lip.siiioottoo..oooo optiooopppooooooooooo dlsscripiion sext
how_onds
25.67
26,86
qu/bt choices genzrate index
26.86
cos_ervt. Mew..liiqueeooooopii desccsscppptttttcccttttiai description Jeax
Ang.
36.60
27.06
27.35
Table 10: List of retrieved source prompts of ROSPR and ORACIE for each evaluation prompts of Hellaswag.
Promje Name
To
RaSP:
Renieved Emieedding
Onicle
Ratrival Embeeking
Answer Givean optinns
96.48
57.76
sociaLi_gu chouces gesemtte answer
87.76
social, LqoShow choices generate unser oos elv). 1 dquesion option descrittion sest cosmos. catto prompt
Chouse Story Ending
85.91
88.24
sxial guSSoo chonees and promie onswer
85.24
Movie What llappens Next
social quiShow chonces penernte answer
Stary Continuaion and Option
83.59
$5.89
social L.gu/how choises and genenate snswer
86.55
social social Lqu/Show choices and generate anower social, qo'Stow choices generase index
Novel Conect Lidding
85.09
57.63
Selall_oa/hho chouees genemte
87.97
Ang.
84.03
85.52
87.57
Table 11: List of retrieved source prompts of ROSPR and ORACLE for each evaluation prompts of StoryCloze.
Prompe Name
To
RuSPy
Retrieval Finhedling
Onele
Rusioved Emnbolding
undecscoce refor
49.25
$1.54
coi_eh1.1
[oL_@h_.1Suusttonn peggetton__pcoondsssiooe descriptionnne
51.78
Bosslabele__nnaa/AAA ANLI
sland for
50.51
50.59
cos e 11quustion option description text
$2.09
ropew/plain no backgronnl
undersooee refor to
50.43
50,59
coi_evl Iiguestionootiinnn pisiio_deecriiionn..e
$2.33
copesipmantt bettommoo noo hist
fill blank
52.17
51.38
Iesposiinndescrigtiinn oytionntte text
$2.01
oie, olv1. Elpectionn catpeciiiiooooiiiiiioo
Replace
52.49
53.59
cos_ov1.11
www.uuutioonnoooeooo cotpoin___ptoooedeee rrsion_decciptonnoor tost
33,59
pansiabeied hnaalpaamphhuseeeee 12kk
30.97
$1.54
$2.36
Table 12: List of retrieved source prompts of ROSPR and ORACLE for each evaluation prompts of Winogrande.

---

Prompt Name
TD
ReSPy
Retreved Embedding
Uraclo
Retreved Enibedding
does the refor
68.27
66.4
socill qhiCkek a ninoo
valld
66.35
powsfihheled fiulPAWS-ANL GPTS
by they mean
62.50
66.3
social quiCheck a nandon mower is valid or not
69.23
glurlygndalliicee not
other wonds
67.31
67.51
socil LquShon choices and genemie answer
72.12
sumsum Write diakgue fhe munch this summary
68.27
sociaL LquShow choices generate answer
Sccial.suuGrneerac answer
replaced with
64.42
99.2
soiaa spiCheck af ninken snowrr is wilid or not
66.
weial selShow choiors and peterate indes
64.42
Su choices genenue
g shoiees and
indes
the pronoun refens
64.42
64.42
socia LquSSow choices and generate index
67:31
social ChaCk if candom answer valid or
Whos what belonz
64.42
63.46
weidl quiShon choices
index
65.38
emmam this dislig
dosss p stonal
67.31
63.46
sxiil was wondkingg
semamm Wrile dialogue
summary
GPT- Style
60.58
67.31
Lquht_ choices and generite index
67.31
Innnnninmttteeeeiiii Expecssed Sentiment
65.10
65.10
68.17
Table 13: List of retrieved source prompts of RoSPR and ORACLE for each evaluation prompts of WSC.
Prompe Nime
RuSP+
Retioved Fimbolling
Onccle
Retrieve Embeeking
guesEnn-coseette www.enttmmannnggeeee.. mwwning-nneeenn was.i---ennnn---ii-llleee with-llbbl
50.31
53.76
gluciegrinccinng
53.76
paustabited, hun.eesateet.peeiioooooooo
rupesprompt, botoo,
57,05
pausiabeled, finaPAWS GPTI retien Jeviioe Frjeymant
grumnsrr hemowurk
49.84
meposprompt botton,
afflmatioo_ toee_oo_at Salse
49.69
51.57
powsllabeldd Roorie lahel
$3.92
pawsiabeted, Meal/Maaiing no label
GPP-3-prompt
Show powellabeled Rinrr
penerate
pawsisbeled fini/AWS
same setse
09.81
52.82
55.17
paws/bbeted finaltask, description no label
ancia i SpoCh aandam
valid
palarityls product rovics posifive shel/gap/menning
GPT-3- with label
30.47
52.19
jlurigg/mmcunngg
52.66
polysemous
50.00
52.82
pows/lbbeled finatRewrii
$3.29
prusibbeled med/Rewwit-----lle
similarsense
Secialiig///h Show choices and gonerate
prosttbcted Beaitakk decippiiitddd
Arg.
50.69
$1.88
55.26
Table 14: List of retrieved source prompts of ROSPR and ORACLE for each evaluation prompts of WiC.


Reviews:
[{"review_id": "vB3ufP5IOQ", "reviewer": "Reviewer_YaNx", "summary": "", "questions": "* Could you please specify the number of queries employed per test data instance when reporting the results in Table 1? If this count exceeds one, it prompts the question of whether a fair comparison with the baselines is maintained.\n* Is there a possibility to adapt the T0-3B soft prompt library to accommodate the T0-11B scenario? Understanding the adaptability of the approach across different library sizes is essential.\n* Could you provide insights into the time constraints, with respect to a specific hardware setup, associated with the retrieval of relevant soft prompts from the library? This information would contribute to a more comprehensive understanding of the method's practicality.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper. We appreciate your recognition of our key arguments that 1) a retriever-based source prompt library demonstrates a straightforward and effective approach and 2) an evolving source prompt library holds the potential for continuous enhancement of the performance. \n\nWe think that all concerns or questions could be addressed. \n\nFirst, we provide the response to the 2 Reasons To Reject.\n\n---\n**The presentation of the paper is weak.**\n\nWe agree that adding a preliminary background subsection introducing the conventional instruction-tuning process and explaining how RoSPr could be applied to instruction-tuned models would clarify the process of RoSPr. We will add the corresponding subsection to the paper. Also, for Sections 5 and 6, we will reduce redundancy and verbosity to effectively convey our results and findings in our revised version. \n\n---\n**Figures 3 and 14 lack comparisons with PT and ATTEMPT.**\n\nThank you for your suggestion. For Figure 3, since the official Big-bench is a zero-shot benchmark that does not provide any training datasets, it is inapplicable to measure the performance of PT and ATTEMPT since both methods require target task training datasets. This again indicates the wide applicability of RoSPr, since it could be applied to target tasks where the training data does not exist, unlike PT and ATTEMPT.\n\nFor Figure 14, we provide the (Min, Median, Max) values for PT, ATTEMPT, and RoSPr. We will also add the results in the paper.\n\n|         |          RTE          |          CB          |        ANLI R1        |        ANLI R2        |        ANLI R3        |          COPA         |       Hellaswag       |       Winogrande      |          WSC          |          WiC          |\n|---------|:---------------------:|:--------------------:|:---------------------:|:---------------------:|:---------------------:|:---------------------:|:---------------------:|:---------------------:|:---------------------:|:---------------------:|\n| PT      | 57.76 / 63.54 / 67.87 | 8.93 / 48.21 / 58.93 | 32.10 / 33.00 / 34.40 | 30.10 / 31.00 / 33.40 | 31.83 / 33.08 / 34.08 | 64.00 / 74.50 / 78.00 | 25.56 / 26.59 / 27.20 | 49.33 / 50.59 / 52.49 | 36.54 / 48.08 / 53.85 | 49.84 / 50.00 / 50.31 |\n| ATTEMPT | 65.70 / 71.03 / 73.65 | 8.93 / 50.00 / 62.50 | 33.30 / 36.70 / 39.00 | 32.67 / 34.58 / 36.50 | 32.67 / 34.58 / 36.50 | 64.00 / 76.00 / 81.00 | 25.92 / 26.65 / 28.16 | 51.07 / 51.22 / 51.70 | 63.46 / 63.46 / 68.27 | 49.06 / 49.92 / 56.74 |\n| RoSPr   | 58.12 / 72.57 / 76.53 | 8.93 / 50.00 / 67.86 | 33.30 / 38.10 / 40.50 | 33.30 / 34.90 / 36.00 | 32.42 / 33.75 / 35.58 | 67.00 / 80.50 / 83.00 | 26.58 / 26.96 / 27.70 | 50.59 / 51.38 / 53.59 | 59.62 / 65.38 / 68.27 | 50.00 / 51.88 / 53.76 |\n\n\nNext, we address the 3 Questions for the Authors.\n\n---\n**Q1: The number of queries employed per test data instance.**\n\nThank you for your question. Note that the queries employed during inference for retrieval are not per test data instance but per **test dataset**. Following [ReCross (Lin et al., NeurIPS 2022)](https://arxiv.org/abs/2204.07937), we sample 32 test instances randomly and search for the optimal soft prompt at the beginning of the zero-shot adaptation of each dataset. This approach does not require a ground truth label for the queries employed or a separate validation set for each test dataset but leads to better performance than fine-tuning baselines (PT and ATTEMPT) which require a labeled training set. Also, instead of searching for the optimal soft prompt for each single query, the retrieval process only occurs once for a test dataset for this approach, reducing the retrieval latency. \n\n---\n**Q2: Adapting the T0-3B soft prompt library to accommodate the T0-11B scenario**\n\nAlthough we have not tested on the T0-11B scenario due to computational limitations, the soft prompt library could be adapted to the T0-11B scenario as well either by 1) training soft prompts of T0-11B through source prompt tuning, or 2) learning a linear layer to adapt the prompt library constructed from T0-3B to the dimension size of T0-11B embeddings. We leave adaptation to larger language models as future work.\n\n---\n**Q3: Time constraints of the retrieval of relevant soft prompts from the library.**\n\nThank you for your question. For a single A6000 GPU, our dense retrieval process for each target task takes 2.07 seconds on average. We choose a 35M-sized dense retrieval model (T5-small encoder) to reduce the latency during inference, which is much smaller than other retrievers such as [ReCross (Lin et al., NeurIPS 2022)](https://arxiv.org/abs/2204.07937) (400M) or [Setence-bert (Reimers & Gurevych, EMNLP 2019)](https://arxiv.org/abs/1908.10084) (80M). Also, we only store 100 examples per training prompt for efficient search.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The utilization of a retriever-based source prompt library demonstrates a straightforward and effective approach, leading to the SOTA performance.\n* The concept of an evolving source prompt library holds the potential for continuous enhancement of zero-shot performance, capitalizing on the expanding library size.", "reasons_to_reject": "* The presentation of the paper is weak. In particular, Section 3 lacks clarity and should provide preliminary information rather than solely referencing external sources. Sections 5 and 6 suffer from verbosity and should be revised to more effectively convey findings.\n* Figures 3 and 14 lack necessary comparisons with baselines (PT and ATTEMPT), which is crucial for contextualizing the contributions.", "questions_to_authors": "* Could you please specify the number of queries employed per test data instance when reporting the results in Table 1? If this count exceeds one, it prompts the question of whether a fair comparison with the baselines is maintained.\n* Is there a possibility to adapt the T0-3B soft prompt library to accommodate the T0-11B scenario? Understanding the adaptability of the approach across different library sizes is essential.\n* Could you provide insights into the time constraints, with respect to a specific hardware setup, associated with the retrieval of relevant soft prompts from the library? This information would contribute to a more comprehensive understanding of the method's practicality.", "paper_topic_and_main_contributions": "The paper under consideration focuses on the enhancement of zero-shot task generalization through retrieval-augmented soft prompting. The authors introduce ROSPR, a method that initiates by storing soft prompt embeddings, followed by their retrieval using a dense retriever. The retrieved soft prompts are subsequently combined based on the Freq./Var. criteria. Across a spectrum of 11 datasets, ROSPR consistently achieves state-of-the-art performance, as substantiated by comprehensive benchmarking. Detailed analysis and ablations are conducted.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "N3kArFtWjn", "reviewer": "Reviewer_eWKn", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper. We appreciate your recognition of the strengths of the paper: proposing a method to efficiently improve the model's ability to comprehend and follow instructions without the need for explicit target task fine-tuning, thus contributing to advancing the broader field of zero-shot learning.\nWe think that all concerns or questions could be addressed. Our response is provided below. \n\n---\n**The conceptual expression in this paper is unclear.**\n\nThank you for your feedback. We think that the concept of instruction-tuning was not clearly delivered. Unlike conventional prompt learning, instruction-tuning fine-tunes a base language model $&theta;$ on **multiple** downstream tasks (multi-task instruction learning) $D$={$D_{1}$, .., $D_{T}$}, and performs inference on **held-out tasks**  $G$={$G_{1}$, .., $G_{R}$} that are unseen during fine-tuning. We ensure that $G_{i}$ &notin; $D$. \n\nInstruction Tuning: \nFor each iteration, \n1. Sample $i$ &in; {1, 2,.., $T$}\n2. Sample {$x_{ik}$, $y_{ik}$} from $D_{i}$ = ($x_{i1}$, $y_{i1}$), .., ($x_{iN}$, $y_{iN}$)\n3. $max_{&theta;}$ $P(h_{ij}(y_{ik})|h_{ij}(x_{ik}))$ \n\nwhere $h_{ij}$ denotes applying the $j$th hard prompt of the dataset $D_{i}$.\n\n\nPrevious works have shown that scaling the number of fine-tuning datasets ($T$) or scaling the model size ($&theta;$) leads to improved generalization to unseen tasks ($G$). However, since these approaches require fine-tuning the full parameters of the model with increased scale which leads to heavy computation, we explore an efficient alternative to enhance the performance by freezing the parameter of an instruction-tuned model and fine-tuning only the soft prompts for each source task. We will clarify this part by adding a preliminary background subsection in the paper.\n\n---\t\n**Comparison with [RetroPrompt (Chen et al., NeurIPS 2022)](https://arxiv.org/abs/2205.14704)**\n\nThank you for the missing reference. We will make sure to include the citation in our revised version. However, we want to point out some differences between RetroPrompt and our method. \n- Although RetroPrompt also adopts a semi-parametric approach to enhance the few-shot/zero-shot performance of LMs for prompt learning, RetroPrompt retrieves representations from the unlabeled training data of the **target task**, while RoSPr ensures a strict held-out setting for **cross-task generalization** by retrieving from source tasks that belong to the different task type (e.g. retrieving paraphrase task soft prompt for entailment tasks). \n- Also, since the labels of the source tasks are normally different from the answer choice list of the target task for our setting, the KNN approach of RetroPrompt which assumes that the retrieved representation has the same answer choice (label choice) as the target task for probability calculation is unapplicable without converting the task into a synthetic set-up (manually matching the label choices between source & target task). \n\nThese two reasons (cross-task generalization & unaligned answer choices) are why it is difficult to make an apple-to-apple comparison between RoSPr and RetroPrompt.\n\n---\n**The novelty of the technique**\n\nAlthough the idea of retrieval of soft prompts has been explored in previous works ([SPoT (Vu et al., ACL 2022)](https://arxiv.org/abs/2110.07904), [ATTEMPT (Asai et al., EMNLP 2022)](https://arxiv.org/abs/2205.11961)), our work has technical novelty in the following aspects:\n- Our work is the first to observe the effect of retrieval of soft prompt on the **zero-shot** task generalization setting where the training data of the target task does not exist by applying the retrieval process to an instruction-tuned model and showing better performance than hard prompt optimization techniques (see the additional table below) and previous approaches that require target task fine-tuning (PT and ATTEMPT performance on Table 1 of the paper).\n- Other than the retrieval of soft prompt, we additionally introduce Var (variance-based ranking) and Inter+Var (interpolation between soft prompts searched through variance-based ranking) which improve the mean accuracy and reduce prompt sensitivity (STD) of RoSPr shown in Table 1.\n\n\n\nAdditional Table 1: Comparison with different hard prompt optimization methods: \n- RoHPr: While the search process is the same as RoSPr, RoHPr retrieves 4 in-context examples (hard prompts) instead of soft prompts.\n- [APE (Zhou et al., ICLR 2023)](https://arxiv.org/abs/2211.01910): Automatic prompt engineering technique utilizing the generation results of an LLM (InstructGPT) that is known to show better performance than human instructions.\n- [ZPS (Liao et al., 2022)](https://arxiv.org/abs/2211.04668): Prompt selection method that chooses the optimal prompt from a prompt pool using prompt ensemble and pseudo-labels.\n\n|       |  RTE  |  CB   | ANLI R1 | ANLI R2 | ANLI R3 |  COPA | Hellaswag | StoryCloze | Winogrande |  WSC  |  WiC  |  AVG  | # Improved |\n|-------|:-----:|:-----:|:-------:|:-------:|:-------:|:-----:|:---------:|:----------:|:----------:|:-----:|:-----:|:-----:|:----------:|\n| T0    | 64.55 | 45.36 |  33.81  |  33.11  |  33.33  | 75.88 |    26.60   |    84.03   |    50.97   |  65.10 | 50.69 | 51.22 |      -     |\n| RoHPr |  47.10 |   45.00  |  33.53  |   33.3  |  33.09  | 73.38 |   26.65   |    86.93   |    50.51   | 64.42 |   50.00  | 49.45 |      3     |\n| APE   | 68.19 | 48.33 |  35.82  |  35.37  |  34.27  |   68.00  |   25.76   |    78.73   |    50.43   | 58.75 | 50.85 | 50.41 |      6     |\n| ZPS   | 58.48 | 60.71 |   36.60  |   34.40  |  33.33  |   76.00  |   28.49   |    87.39   |    51.78   | 64.42 | 50.63 | 52.93 |      7     |\n| RoSPr | 71.54 | 49.48 |  37.05  |  34.64  |  33.92  | 78.75 |   26.97   |    85.52   |    51.50    | 64.52 | 51.76 | **53.24** |     10     |\n\n\nLet us know if there are further questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1、This paper aligns with the growing interest in zero-shot learning, showcasing an approach that not only addresses performance limitations but also contributes to advancing the broader field of zero-shot learning.\n\n2、This paper introduces Retrieval of Soft Prompt (ROSPR), which is easily scalable and requires minimal computation by only adding 0.007% parameters to the main model during inference.", "reasons_to_reject": "1、The conceptual expression in this paper is unclear. Although instruction-following is essentially a form of prompt, the way it is presented in this paper can easily become entangled with the concepts of contemporary LLMs. For instance, the explanation in line 42-52 is not clear.\n\n2、Prompt embedding refers to the representation of instances within the context of prompts, while the concept of zero-shot entails utilizing instances from the training samples' retrieval to enhance downstream tasks. This method has been discussed in [1], yet the authors neither cite it in the related work nor compare their approach to it. \n\n3、This paper is somewhat incremental in its technical approach, lacking significant innovation and contributions.\n\n[1] Decoupling Knowledge from Memorization: Retrieval-augmented Prompt Learning. NeurIPS 2022\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper addresses the enhancement of zero-shot performance in instruction following models through the incorporation of a soft prompt retrieval approach. The main contribution lies in proposing a method to efficiently improve the model's ability to comprehend and follow instructions without the need for explicit fine-tuning. The use of soft prompt retrieval aims to bridge the gap between traditional fine-tuning and zero-shot learning, thereby contributing to more versatile and adaptable instruction following models.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ONQCQXf7Ht", "reviewer": "Reviewer_GnhD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for making the effort to review our paper. We appreciate your recognition of our key arguments that 1) RoSPr consistently outperforms its baselines while adding a small number of parameters and 2) the paper includes a comprehensive ablation. Let us know if there are some improvements that are needed to increase the excitement of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper provides a well-structured and clear explanation of the proposed framework. The proposed ROSPR method, as evidenced by experimental results, consistently outperforms its baselines on a variety of tasks, only with the inclusion of a relatively small number of parameters. Furthermore, the paper includes a comprehensive ablation study which sheds light on the intricacies of the method.", "reasons_to_reject": "I find no reason to reject this paper.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper explores the challenge of enhancing zero-shot performance in instruction-following models without extensive computational overhead. By training soft prompts embeddings for hard prompts of source tasks and creating a vector store, ROSPR retrieves relevant prompt embeddings during inference to aid predictions on target tasks. The experiment results highlight the proposed method's efficacy in improving the zero-shot capabilities. Supplementary insights are presented in the analytical section.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OmWrmWmlfI", "reviewer": "Reviewer_dsfs", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for making the effort to review our paper. We appreciate your recognition of the strengths of the paper.\nWe think that all concerns or questions could be addressed. Our response is provided below. \n\n---\n**The novelty of the technique**\n\nAlthough the idea of retrieval of soft prompts has been explored in previous works [[SPoT (Vu et al., ACL 2022)](https://arxiv.org/abs/2110.07904), [ATTEMPT (Asai et al., EMNLP 2022)](https://arxiv.org/abs/2205.11961)], our work has technical novelty in the following aspects:\n- Our work is the first to observe the effect of retrieval of soft prompt on the **zero-shot** task generalization setting where the training data of the target task does not exist by applying the retrieval process to an instruction-tuned model and showing better performance than hard prompt optimization techniques (see the additional table below) and previous approaches that require target task fine-tuning (PT and ATTEMPT performance on Table 1 of the paper).\n- Other than the retrieval of soft prompt, we additionally introduce Var (variance-based ranking) and Inter+Var (interpolation between soft prompts searched through variance-based ranking) which improve the mean accuracy and reduce prompt sensitivity (STD) of RoSPr shown in Table 1.\n\n---\n**A comparison should be made with other methods that utilize retrieval hard prompts/examples for in-context learning.**\n\nThank you for your suggestion to add more prompt engineering & ICL baselines. We have conducted additional experiments to compare our method with 3 different hard prompt optimization techniques including (1) RoHPr which utilizes the same search technique as RoSPr but retrieves 4 corresponding training instances for in-context learning instead of the corresponding soft prompt, and (2) [APE (Zhou et al., ICLR 2023)](https://arxiv.org/abs/2211.01910) which is an automatic prompt engineering method that utilizes the generation results of an LLM (InstructGPT), known to show better performance than human instructions, and (3) [ZPS (Liao et al., 2022)](https://arxiv.org/abs/2211.04668) which selects an optimal hard prompt from a prompt pool using prompt ensemble and pseudo-labels.\n\n\nAs shown in the below table, on average, RoSPr performs the best on average, showing the benefits of using soft prompts over hard prompts for task generalization. While RoSPr shows improvement on 10 out of 11 datasets, other hard prompt optimization methods do not show consistent improvements. We will include these additional results in the revised version.\n\n|       |  RTE  |  CB   | ANLI R1 | ANLI R2 | ANLI R3 |  COPA | Hellaswag | StoryCloze | Winogrande |  WSC  |  WiC  |  AVG  | # Improved |\n|-------|:-----:|:-----:|:-------:|:-------:|:-------:|:-----:|:---------:|:----------:|:----------:|:-----:|:-----:|:-----:|:----------:|\n| T0    | 64.55 | 45.36 |  33.81  |  33.11  |  33.33  | 75.88 |    26.60   |    84.03   |    50.97   |  65.10 | 50.69 | 51.22 |      -     |\n| RoHPr |  47.10 |   45.00  |  33.53  |   33.30  |  33.09  | 73.38 |   26.65   |    86.93   |    50.51   | 64.42 |   50.00  | 49.45 |      3     |\n| APE   | 68.19 | 48.33 |  35.82  |  35.37  |  34.27  |   68.00  |   25.76   |    78.73   |    50.43   | 58.75 | 50.85 | 50.41 |      6     |\n| ZPS   | 58.48 | 60.71 |   36.60  |   34.40  |  33.33  |   76.00  |   28.49   |    87.39   |    51.78   | 64.42 | 50.63 | 52.93 |      7     |\n| RoSPr | 71.54 | 49.48 |  37.05  |  34.64  |  33.92  | 78.75 |   26.97   |    85.52   |    51.50    | 64.52 | 51.76 | **53.24** |     10     |\n\n\n---\n**Fine-tuning the backbone model to generate soft prompt embeddings poses a challenge for large-sized language models.**\n\nAlthough fine-tuning the same backbone model is needed to generate soft prompt embeddings, we think that the advantages outweigh the disadvantages for the following two reasons:\n\n- Recent studies, such as [Med-PaLM (Singhal et al., Nature 2023)](https://www.nature.com/articles/s41586-023-06291-2), have demonstrated that training soft prompts on much larger models, such as the 540B PaLM model, can significantly boost the performance in clinical knowledge tasks, showing the feasibility of training soft prompts for >500B LLMs which implies that RoSPr is also scalable.\n- In terms of practicality, [QLoRA (Dettmers et al., 2023)](https://arxiv.org/abs/2305.14314) has shown that the parameter-efficient fine-tuning of 65B LLM is feasible with a single 40GB GPU via 4-bit precision without performance degradation, allowing efficient gradient updates. Especially, since our approach only requires updating 0.007% of the parameters for 150 steps for each soft prompt, we expect that quantization techniques would enable the fine-tuning of large-sized language models with consumer GPUs. \n\nConsidering these aspects, we believe that RoSPr can be also practically effective for larger models. \n\nWe hope the authors have provided adequate additional experiments and responses to address the weaknesses identified by the reviewer and kindly ask the reviewer to reconsider their initial assessment of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-crafted, providing clear and easily understandable logic.\n\nThe implementation details are presented clearly.", "reasons_to_reject": "1. The idea of retrieval soft prompts is not novel.\n2. The benefits of employing retrieval soft prompt embeddings to enhance zero-shot generalization have not been explicitly clarified. A comparison should be made with other methods that utilize retrieval hard prompts/examples for in-context learning.\n3. The process of collecting a source prompt library necessitates fine-tuning the same backbone model to generate soft prompt embeddings. However, this poses a challenge for large-sized language models as fine-tuning becomes unfeasible, thus limiting the applicability of the method.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper aims to retrieve soft prompts to assist hard prompts in zero-shot task generalization. The authors train soft prompts for each hard prompt through prompt tuning, resulting in the creation of a source prompt library. During inference, the soft prompt embedding of the training instance that exhibits the highest similarity to the query instance is retrieved. The experimental evaluation encompasses a range of tasks, including natural language inference, sentence completion, coreference resolution, and word sense disambiguation. The paper also assesses the performance on BIG-Bench tasks.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work effectively leverages soft prompt retrieval to improve the zero-shot performance of instruction-following models and provides an easy method to improve instruction-following capabilities without fine-tuning. The reviewers raised some concerns about unclear writing and we encourage the authors to improve the presentation.
Decision: Accept-Findings