Automated Few-Shot Classification with Instruction-Finetuned Language Models
Authors: Rami Aly
Xingjian Shi
Kaixiang Lin
Aston Zhang
Andrew Gordon Wilson
Venue: EMNLP 2023 Findings (2023)
Keywords: few-shot classification
prompt automation
large language models
TL;DR: Automation of prompts for large instruction-tuned encoder-decoder language models on few-shot classification tasks.

Abstract: A particularly successful class of approaches for few-shot learning combines language models with prompts - hand-crafted task descriptions that complement data samples. However, designing prompts by hand for each task commonly requires domain knowledge and substantial guesswork.  We observe, in the context of classification tasks, that instruction finetuned language models are remarkably robust towards some dimensions of a prompt's design. We subsequently propose a simple method to eliminate the need for handcrafted prompts, named AuT-Few.
This approach consists of (i) a prompt retrieval module that selects suitable task instructions from the instruction-tuning knowledge base, and (ii) the generation of two distinct, semantically meaningful, class descriptions and a selection mechanism via cross-validation. Over 12 datasets, spanning 8 classification tasks, we show that AuT-Few outperforms current state-of-the-art few-shot learning methods. Moreover, AuT-Few is the best ranking method across datasets on the RAFT few-shot benchmark. Notably, these results are achieved without task-specific handcrafted prompts on unseen tasks.

Automated Few-shot Classification with Instruction-Finttune Language Models
Rami Aly'; Xingjian Shi?, Kaixiang Lin", Aston Zhang', Andrew Gordon Wilson 'University of Cambridge Boson AI
3 Amazon Web Services "New York University
rami aly@cl. cam ac. uk, xshiabeconnect. ust. ukk (kaixianl, as astonz Jeamazon. com, andrewgwecims. .yw.....e..
.nw.e.iiiiinnnmm.eoo.o.me. edu
Abstract
Prompts, instructions that describe the tasks in natural language, are crucial to successful fine- tuning on many tasks. Typically, prompts con- sist of two components: task templates and an- swer choices. Task templates are textual instruc- tions about the task. Answer choices are seman- tic descriptions of the categorical labels. Super- vised training prompted samples, shown in Figure helps PLMs generalize when instructed via prompts on a new problem (here natural lan- guage inference). Following Lin et al. (2022), we use the term upstream model for these instruction- finetuned PLMs. These prompted upstream models provide otate-of-the-art few-shot learning (Liu et al., 2022), yet they still rely on strenuous manual intervention from manually crafted prompts, de- signed by experts with domain knowledge about the underlying tasks.
particularly successful class of approaches for few-shot learning combines language mod- els with prompts haad-crrftedd task descrip- tions that complement data samples. However, designing prompts by hand for each task com- monly requires domain knowledge and substan- tial guesswork. We observe, in the context of classification tasks, that instruction finetuned language models are remarkably robust towards some dimensions of prompt's design. We sub- sequently propose a simple method to eliminate the need for handcrafted prompts, named AuT- Few. This approach consists of (i) prompt re- trieval module that selects suitable task instruc- tions from the instruction-tuning knowledge base, and (ii) the generation of two distinct, se- mantically meaningful, class descriptions and selection mechanism via cross validatii,,,,n,, Over 12 datasets, spanning 8 classification tasks, we show that AuT-Few outperforms current state- of-the-art few-shot learning methods. More- over, AuT-FFew is the best ranking method across datasets on the RAFT few-shot bench- mark. Notably, these results are achieved with- out task-specific handcrafted prompts on un- seen tasks.
Answer Choices
Task Template
Instruction Tuning Prompts
We came here Saturday night [..]] How does the reviewer feel about the movie?
Review:
0: very nega- tive [...]
Sentiment
5: very posi- tive
1 Introduction
Last year, Comeast [..]] Is that paraphrase of the sentence Comcast has about [...]
0: Yes I: No
Collecting annotated data is time-consuming and expensive. The goal of few-shot learning is to ad- dress this limitation by developing models that gen- eralize from small number of training examples.
Paraph.
Unseen Target Task Prompts
Given Oil prices fall back as Yukos oil threat lifted. can we guarantee that Oil prices rise. is true?
A now dominant paradigm in few-shot learn- ing involves pre-training a large language model (PLM) on unsupervised language modelling objec- tives, combined with supervised fine-tuning (Ka- plan et al., 2020; Wei et al., 2022b). Fine-tuning on a variety of classification tasks improves gen- eralization to new unseen tasks even further (Sanh al., 2022; Wei et al., 2022b; Chung al., 2022).
0: Yes
NLI
1: No
Instruction-tuning uses prompts to specify the task via templates (blue) and label descriptions via answer choices (magenta). Fine-tuning on multiple in- structed tasks improves generalization to new ones.
Figure I:
In this paper, we are concerned with an auto- mated few-shot classification regime, where the algorithm can only access the training samples
Work done while interning at Amazon Web Services. Work done when author was working at Amazon Weh Services.

---

Ooatation Answer
$4.1 Template Automation Retrieval
Poraphrase Iden
Instruction-tuiing Templates
Quastion Answering
Structure-
Summark zation
Oil prices fall back as Yukos oil threat lifted.
e A
is that paraphrase of the following sentence:
Sentiment
Toplc Classific.
Identificat.
Oil prices rise.
Instruction-tuned Language Model (Upstream Model)
NLI
3 A
Premise: "Oll prices fall back Yukos threat lifted."
$4.2 Answer Choice Automation
Generation
0: Yes
1: No
Hypothesis: "Oil prices rise."
Template-Tailored
Yes
V
Topie-Specific
journalist
Label: (Entailment)
(Dataset Default)
Entailment
not Entail.
Selection
Training Samples
Figure 2: A schematic view of our prompt automation method, AuT-Few, consisting of: the retrieval of templates from the instruction tuning collection ($4. I), and the generation of lemplate-tailoned and topic-specific answer choices and the configuration amongst them (and optionally the default dataset label text) ($4.2).
and their categorical labels. While efforts have been made to automate prompting, these methods are not directly transferable upstream models. Most techniques target prompted masked language models (i.e. encoder-only models, that make pre- dictions over continuous embeddings via its mask token (Gao et al., 2021, inter alia). Automation methods for models with a discrete output space (i.c. a decoder over the vocabulary) are costly and limited to the automation of the task template, still relying on handcrafted descriptions of labels (Liu et al., 2021: Zhou et al., 2023).
identification. To automate answer choices, AuT- Few generates label descriptions tailored to the re- trieved templates (e.g., Yes/No for polar question, as for the illustrated paraphrase identification tem- plate) and descriptions that capture a class' overall topic (e.g. Enron/purchase for Enron spam clas- sification). AuT-Few selects the most appropriate configuration via cross-validation.
AuT-Few outperforms strong baselines, includ- ing T-Few (Liu et al., 2022), by 2.1 points over total of 12 datasets, spanning 8 tasks, without any task-specific handcrafted prompts. All but task are unseen the upstream models, indicating AuT- Few's strong generalization capabilities. Moreover, by applying AuT-Feew to a small upstream model (BARTO (Lin et al., 2022)), we achieve competitive performance and efficiency to the current state-of- the-art prompt-free method, SetFit (Tunstall et al., 2022). Furthermore, AuT-Few achieves the best average rank across datasets the few-shot RAFT benchmark (Alex et al.. 2021). An ablation justifies the components of our automation method.'
To automate few-shot learning with upstream models, we analyse the role of prompts across various classification tasks and observe that upstream models exhibit low variability towards task-unspecific templates. In contrast, the selection of suitable answer choices can be important, yet answer choices do not need be tailored to the spe- cific instruction (e.g. Yes/No for polar question). These insights confirm observations by Webson and Pavlick (2022) in a broader context and they motivate simple few-shot learning automation method for upstream models, named AuT-Few.
Background and Related Work
AuT-Few builds on the state-of fate--of-thr of-t learn- ing method T-Few (Liu et al., 2022), but crucially does not use any task task-specific handcrafted prompts. AuT--Few automatically finds the most relevant tem- plates our target task from the collection prompts used instruction- tune the upstream model. As illustrated in Figure 2, given an NLI task, AuT- Few might retrieve templates written for paraphrase
2.1 Instruction-Finetunee Language Models
A language model is instruction-finetunees on prompted samples D from various tasks, such as summarization question answering. by au- toregressively generating the target answer choice through standard maximum likelihood training.
Code at: https: / thub. com/RaldirrAuT-FFe.

---

struction tuning not only improves generalization for large decoder-only models (Wei et al., 2022a), but also for comparably smaller encoder-decoder models, like TO (Sanh et al., 2022) or BARTO (Lin et al., 2022). Prompt knowledge bases (KB), like PromptSource (Bach ct al., 2022), contain prompt instructions for hundreds of tasks. Flan-T55 (Chung et al., 2022) is an improved upstream model scaled thousands of tasks (Wang al., 2022b).
Gao al., 2021; Zhou al., 2023), combines both via semi-parametris prompt tuning (Bari al., 2022). However, prompt tuning is brittle to opti- mize (Hu et al., 2022a; Liu et al., 2022), and the generation of discrete instructions requires substan- tial computational resourees, a particular concern with upstream models as they typically have bil- lions of parameters. The retrieval of instructions is limited to the retrieval of trained soft prompts and samples (Ye et al., 2022), prompt initialization (Vu et al., 2022), or the retrieval of multiple prompt mixtures (Qin and Eisner, 2021; Asai et al., 2022).
Inference. We are interested using upstream models for an unseen few-shot binary or multi- class classification task Diss- A prediction ý with an upstream model 0 is made by computing the length-normalized log probabilities for each class y E y, conditioned on the sample z, handcrafted template Φj E & (i.e. task description and sample input formatting), and on the associated answer choices bh E V (textual descriptions of labels):
Answer Choice Automation. Methods au- tomate label representations are targeting BERT- like masked language models (Devlin et al., 2019), which enables optimization of the output descrip- tions on continuous vector representation. Shin et al. (2020) train a logistic classifier embed- dings score tokens in the vocabulary by how well they predict the task labels. Gao et al. (2021) com- pute the probability for a token to be the masked classification token, by computing the dot product between both embeddings. Wang et al. (2022a) additionally ensure that label tokens belong only to single class. Alternatively to such discrete search is learning soft output representations of labels via gradient descent (Hambardzumyan et al., 2021; Cui et al., 2022; Hu et al., 2022b; Karimi Mahabadi et al., 2022), or combining both (Ma et al., 2022). Tunstall al. (2022) propose a fully prompt-free method using Sentence Transformers (Reimers and Gurevych, 2019).
1 Σ log pa (w)))()) xr O5 (w)))))
argmax, y
with T being the length of the answer choice of y. Since the use of single prompt might model the expectation over all possible prompts poorly, most systems handcraft multiple prompts for a target task. The expectation then modelled by randomly drawing a template and its answer choices.
Parameter-Efficee Finetuning. Adapting up- stream models to new task or domain on few train via full model finetuning is often infeasible as these models consist of bil- lions of parameters. Parameter-efficienn finetuning adds or updates only a small subset of parameters OPEFT < 0, and largely retains the fine-tuning performance (Karimi Mahabadi et al., 2021; Zhang et al., 2021; Chen et al., 2023). Liu et al. (2022) pro- posed T-Few and showed that parameter-efficient finetuning upstream model with T-Few performs better than in-context learning with GPT-3 in the few-shot learning setting. T-Few learns attention and activation re-scaling vectors by optimizing the maximum likelihood estimation and complements it with an unlikelihood loss.
available samples Dist
Novelty. Prior works on prompt automation are computationally intensive, brittle to optimize, or assume a continuous output representation for each token. By contrast, our proposed approach auto- mates prompts for upstream models, which operate over a discrete output space. We do not insert any additional trainable parameters for automating tem- plates. Instead, our work the first to use retrieved instruction eestruction-fieeuunnng templates for an unseen task directly and to use them to optimize the answer choices via the generation of distinct, semantically meaningful, answer choice configurations.
2.2 Prompt Automation
How Much Does the Design of Prompts Matter for Upstream Models?
Template Automation. To automate the instruc- tions as input to the model, previous work uses soft representation in the input via prompt tuning (Liu et al., 2021; Hambardzumyan et al., 2021), generates discrete instructions (Shin et al., 2020;
To automate prompts, we need understand their role in few-shot classification. While previous re- search suggests that the wording of instructions for

---

Templates
100
80
Score
60
0
RTE
WSC
WIC
ANU-r1 ANUI-2
ANLI-13
CB
Emotion Enron
Amazon
CR
SST-5
Average
Answer Choices
Score
60
40
0
WSC
WIC
ANU-r1 ANLI-1 ANLI-1
CB
Emotion
Enron
Amazon
CR
SST-5
Average
Figure 3: An analysis of prompts used in PEFT of upstream models (here TO), broken down into templates (top) and answer choices (bottom). Experiments span 12 datasets and tasks. Error bars indicate one standard deviation across runs. General task-unspecifie templates perform surprisingly well and instruction esstuctioonindependenn single answer choice configurations (i.e. dataset and best-single) outperform handcrafted prompts.
masked language models is crucial, Webson and Pavlick (2022) observe that the semantic relevance of prompt is not strong performance indicator for upstream models. However, their analysis re- stricted to natural language inference whilst using the PET (Schick and Schütze, 2021) algorithm to train the model. Yet, results in Schick and Schütze (2022) suggest that templates do matter in principle. but PET is robust when correctly configured.
of 62.8, 62.9, 64.0, 64.2, for each setting, respec- tively, we observe that simple task-unspecifie tem- plates perform surprisingly well, only perform- ing slightly worse than more complex handcrafted ones. Templates that are not well-formed or lack an instruction entirely perform substantially worse than handcrafted ones. Note that results differ heav- ily between datasets. While some datasets (Enron and CR) are virtually unaffected by the design of the template, performance is strongly affected by the template for some other (e.g. RTE, WSC, Amazon).
These results raise questions regarding the role prompts for upstream models the context of automated few-shot learning on unseen tasks. We conduct systematic ablation study for both tem- plates & and answer choices V. We use T-Few with the TO upstream model and 32 samples per class. We evaluate 12 datasets, spanning tasks. For details the datasets, see Appendix A.
Answer Choices. Similarly, for answer choices we run four experiments: reversed handcrafted an- swer choices (reversed), uniform sampling of a ran- dom word from the training vocabulary (random), label text as presented in a dataset itself, such as Entailment in Figure 2 (dataset), and handcrafted choices. Different handerafted templates for the same task might have different answer choices, de- pending on the instruction. In contrast, there ex- ists only single answer choice configuration for dataset answer choices (i.e. mapping from categori- cal label to text), which we use across all templates.
Templates. We design four experiments under- stand the importance of accurate task descriptions (i.e. semantics) increasing order: concatenation sample's content without any additional text (null), uniform sampling of words from the training vocabulary (random), general purpose instructions (c.g. Given .... the answer is ...) that are not tai- lored to the task (general), handcrafted instructions (bandcrafted).. We use the same handcrafted answer choices and templates across all settings (and vice the same templates across experiments for answer choice experiments).
We observe that unlike templates, the selection of answer choices makes large difference in performance. However, datasets that were particu- larly robust regarding template design appear be also robust here. Moreover, despite dataset choices (c.g. entailment, not_ jot__nnallmen)) not matching a template's ' instruction (e.g. "Given does fol-
As seen in Figure 3 (top), with a mean score

---

low? Yes or No?"), and only having one configu- ration of choices, we observe comparable perfor- mance handcrafted ones. Thus neither template- tailored answer choices nor multiple distinct an- swer choice configurations are needed. By man- ually selecting a single configuration of answer choices from both dataset and handcrafted choices (best-single), we easily achieve the highest average score with 66.2. An automated selection mecha- nism of single configuration can subsequently perform favourably over multiple distinctly hand- crafted prompts.
in Figure 1 for sentiment classification and para- phrase identification. We then aim to find templates A C Dir from the collection that are related to our downstream task. For instance, given the NLI sample from Figure 2, we rather want to retrieve templates about paraphrase identification than sen- timent classification. The former is both semanti- cally and structurally more similar to NLI, as both have two arguments in their input. For NLI they are hypothesis and premise while for paraphrase iden- tification these are the two compared sentences.
To find suitable templates, we first filter the col- lection 4r to templates that match the target task format the most. We achieve this by matching the number of underlying arguments of our target task, against the number of arguments of individual tem- plates in IT. We then do semantic search via an efficient retrieval system: we query a concatena- tion of a sample's argument descriptions (e.g. the strings hypothesis and premise) against all suitable templates in 4/r by encoding both query and ev- ery template in the collection with a lightweight bi-encoder (Reimers and Gurevych, 2019). If the field descriptions are uninformative (e.g. numbers), instead use the averaged representations of all samples in D'sl as the query. Using cosine similar- ity, we then select the top R templates. Finally, we adjust the retrieved templates to the downstream task via regular expressions to obtain 4
4 AuT--Few: Automated Few-shot Classification with Upstream Models
AuT-Feww is a simple. yet efficient, algorithm to automate prompts for upstream models, drawing from the insights gained from Section 3. Figure shows an illustration of AuT-Few's template and answer choice automation. AuT-Few deploys a lightweight template automation approach since accurate task templates are not essential to perfor- mance. selects suitable templates from the collec- tion of prompts the upstream model was instruction- finetuned on (Section 4.1).
On the other hand, the selection of answer choices has substantial impact on performance. Searching over all possible answer choices is in- tractable for large upstream models and also impre- cise due to the small training size. Thus, AuT--Few only considers two distinct types of answer choices (Section 4.2). One is tailored to the retrieved tem- plates by measuring the log- ligelhooo on the train- ing data (template-tailoredd) The other is based on capturing the topic of samples belonging to the same class (topic-specific).
4.2 Automated Selection of Answer Choices Generation of Answer Choice Candidates. Apart from the label descriptions that appear in the dataset, which may not be meaningful, we consider the generation of two distinct types of answer choices given the retrieved templates: template-tailored and topic-specific answer choices. Demplate-tailoree answer choices are generated by finding individual tokens for each class that max- imize the conditional likelihood over the training data of that class Dirain given the retrieved tem- plates & E A computed via the upstream model:
We select the most appropriate template and answer choice configurations via validaaiiit The automated prompts are then used for training and inference of our upstream model, where we largely follow T-Few (c.f. Section 1.1 for details).
4.1 Automated Templates via Retrieval
C. Σ
Σ
log Po(v x, Ø),
We retrieve templates that are used in instruction tuming the upstream models. This enables us to (i) adhere closely to instructions the model is fa- miliar with and has already learned (ii) exploit the associated inductive bias on answer choices for candidate generation in the next step. Specifically, we consider the collection of all prompts used for instruction tuning, prr, such as the ones shown
DED Prain 4E4A
with E \/ being token of the subword vocabu- lary of the upstream model. Tokens unspecific to individual class might be ranked high across mul- tiple classes. Thus, we further compute for every token how far its likelihood deviates from the mean CC Eeeo La. We finally select the top-ranked dis-

---

tinct tokens across all classes that maximize the sum of these scores.
et al. (2021). For baselines, and implementation specifies, including hyperparameters, see Appendix B. For used datasets, see Appendix A.
Relying exclusively on the likelihood signal (and the retrieved templates) to find answer choices might amplify the inductive bias of the model and it restricts other potentially viable answer choices" Since our analysis indicates that answer choices not tailored to the templates can still perform strongly, we additionally consider topic-specific an- swer choices not generated via our upstream model. We use the high quality contextual representations of Sentence Transformers to find single-wort word (not token) representations that semantically express the underlying content for each class, For each sen- tence Se for particular class, we obtain contex- tual representation of the sentence and each word. For every class and over the training vocabulary we then compute the cosine similarity between each sentence and word. We remove words that occur across different classes and finally use the top word for each class as the topic-specific choices.
We conduct experiments 12 text classification datasets, spanning total of 8 tasks. This collection is in essence combination of evaluation datasets used in Liu al. (2022) and Tunstall et al. (2022), minus datasets that we con- sider not traditional classification tasks, e.g. sen- tence completion, where the meaning of the class changes per instance.
Datasets.
total of
Implementation Details. AuT-Few largely fol- lows T-Few (Liu et al., 2022) for finetuning, with some modifications to training and inference to increase robustness for our automated few-shot method. Instead of only learning rescaling vectors of the upstream model's weights ((TA)), we addi- tionally learn and re-scale decomposition matrices (LoRA), as proposed by Hu al. (2022a). (IA)² and LoRA are complementary and the gradient up- dates from both methods can be made persistent to the model's weights after training without inquir- ing additional inference costs over the upstream model itself. Another limitation of T-Few is its in- ference algorithm. T-Few selects single template at random (c.f. Section 2) and it can be poor ap- proximation of the overall expectation, especially with noisy templates as used with AuT-Few. We instead run a Monte-Carlo approximation over all retrieved templates, computing a weighted average over the probabilities computed via each template.
Selection of Best Answer Choice Configura- tion. We are now tasked to find the best rep- resentation for the given task. For cach choice option, we consider joint signal derived from a supervised evaluation, i.c. F1 score, on subset of the training data Dtrain. and from a measure of the overall log probabilities on the test data Drest- The assumption for the latter is that representative answer choices better esti- mate the task's distribution, resulting in overall higher log probabilities unseen data of the target task: Ey Eeneo, a ExED ( Eloo log pa (v) xt, Φ, (t) with thp being the current answer choices configuration. We compute the final score for each candidate by summing the normalized scores of each metric over 3-fold cross- validaaal....
Baselines. In addition to the current -fate-of-the art few-shot learning method T-Few, we consider SctFit (Tunstall et al., 2022) (with a RoBERTA backbone), which is of particular relevance in our context, since it is the state-of-the-ar efficient prompt-free few-shot method. We also compare against a finetteede RoBERTaLARGE model, based the baseline in Tunstall al. (2022). The majority baseline is based on the class distribution in the test data.
5 Evaluation
5.1 Experimental Setup
This section provides an overview of our experi- mental setup. We are sampling K training samples for each class y: E Y, for total of K [ train- ing samples?, We do not consider a validation set to exist for hyperparameter-tuning following Alex
5.2 Results
Results for K = 32 samples per class are shown in Table 1. Both T-Few and AuT-Few use TO-3B as the upstream model. We report accuracy on all datasets with the exception of Amazon-CI, where we report Matthew's 's correlation coefficient due to the skewed distribution, following Tunstall et al. (2022).
"For example input prompt and samples might have been encountered for NLI tasks, focusing on options working particularly well for this scenario.
"While in Liu et al. (2022) samples are drawn randomly. i.c. not stratified, we largely adhere to the traditional N- N-Way- K-shot classification setting, as data imhalance in training is an aspect to be explored separately.

---

Majority
Zero-shot
Finetune
SetFit
Rand. T-Few
T-Few
AuT--Few
RTE
52.7
65.61.2
56.45.6
51.41.8
65.23.6
82.5
81.42.4
WSC
63.5
62.13.9
49.27.1
50.3
49.66.6
70.23.1
59.21.5
WiC
50,0
51.30.6
53.95.1
55.05.1
55.35.2
55.94.4
58.4s.1
ANLI-RI
33.4
35.60.8
32.11.9
32.91.6
45.24.9
52.92.0
49.13.7
ANLI-R2
33.4
33.60.7
33,41.6
34.01.7
40.62.0
12.5
42.01
ANLI-R3
33.5
34.20.8
31.51.6
32.71.0
36.93.4
44.21.2
43.53.0
CB
50.0
57.50.8
86.16.6
84.36.0
77.56.1
91.43.2
93.91.6
Emotion
35.2
42.10.8
3.5
71.93.2
48.7a.5
65.42.a
72.62.5
Enron
50.9
53.30.4
92.22.4
95.11.2
96.90.6
96.50.4
95.50.5
Amazon-CF
0.00
0.04u.T
40.59.9
60.13.0
35.710.6
24.07.5
59.0s.2
CR
64.2
88.90.4
84.84.3
90.71.7
93.6a.5
93.70.2
92.51.1
SST-5
26.3
38.91.0
42.13.4
49.2u.9
47.23.9
51.51.1
48.62.3
Avcrage 41.1
47.31.0
55.0
59.02.6
57.74.4
64.22.4
66.32.5
Table I: Main results with 32 samples per class, averaged over five runs. AuT-Few adopts TO the upstream model. Rand. T-Few uses randomly selected answer choices. Statistically significant differences between Fuw--Few and T-Few are marked with using two-sided Monte-Carlo permutation test with 10000 repetitions (p < 0.01). AuT-Few has the highest average score across datasets without the of handerafted task prompts while maintaining comparable standard deviation to T-Few and SetFit.
AuT-FFew outperforms T-Few (61.2 + 2.1) and SetFit (59.0 2.6), with an average score of 66.3 2.5. trivial T-Few automation strat- egy that randomly draws answer choices from the training data (c.f Section 3) performs substantially worse than AuT-Few with much higher variability (57.7 + 4.4). While AuT-Few has higher average score than T-Few, the latter wins against AuT-FFew 8 out of 12 datasets. However, observe tistically significant difference* on only 4 datasets. Out of these four datasets where we observe statis- tical significance,, AuT-Few outperforms T-Few in three them (WiC, Emotion, Amazon-CF). More- over, we would like to emphasise that performing even comparable against T-Few is already win since the latter uses multiple diverse handcrafted prompts for each target task while AuT-Few does not require any manual involvement by the user to optimize the prompt while maintaining comparable standard deviation.
these datasets have been used train TO (see App. C.1 for detailed scores).
We note that the automated prompts are not al- ways semantically coherent. As shown in Appendix D, automation choices for some datasets, such as mp3player and ipod for CR, appear odd, yet the model still achieves very high score them. This observation can be explained by our findings in section 3, identifying that some datasets such as CR and EnronSpam are particularly robust to- wards the task description and the answer choices. For CR, AuT-Few's ' cross- cross-validation strategy for selecting the best answer choice subsequently mea- sures almost identical scores for all three choice configurations (90.1, 89.8, 90.4 for the dataset, template-tailored, tailored, and topic-specific choices, spectively), resulting in the seemingly erroneously answer-choice selection.
Results across Upstream Models & Efficiency. Results of AuT-Few with different upstream mod- els, namely BARTO, TO, and Flan-T5 are seen in Table The results in the table are computed without Monte-Carlo approximation, resulting a minor performance decline, yet simplifying the efficiency comparison. Datasets that are part of the instruction-tuning corpus of Flan-T5 or BARTO have been excluded (greyed out). BARTO being about 8 times smaller than TO performs substan- tially worse, but still substantially outperforms T-Few with BARTO and maintains higher average score than SctFit. Flan-T5 performs on average the best on its unseen datasets, indicating the improved
On the blind test set with the best variant of TO (TO++, 1IB parameters) AuT-Few achieves average score of 71.3 versus 70.5 for T-Few (with the same backbone), excluding WiC and WSC, as
"We the two-sided Monte-Carlo permutation with 10000 repetitions (p-valuc 0.01). Significance for datasct holds iff results significant across all seeds.
"Notably, the performance difference between AuT-Few and T-Few on WSC, the only dataset where AuT-Few performs substantially worse. is not statistically significant given our test: this can be explained by the very small sample size of the dataset's s evaluation data of only 104 samples. Liu et al. (2022) also observed "unstable results" on WSC, see the discussion on this Github issue.

---

T-Few
AuT-Few
80
70
# Param.
Tr. Param.
330M
330M
400M
0.1M
0.3M
3B
400M
1.9
10.5M
3B
10.5M
3B
19
58.1
56.
lnf.
19c10
1.8el1
50
47.3
Tr. FLOP
8.5e14
4.1e14
39e15
3.6e15
2.7e16
2,7e16
RTE
S1.48
804s.
82.524
71.3%,
79.3g,5
90.118
40
WSC
50.3g4
61.2g
70.2a1
52.9%
58.316
73.1
80
WIC
55.0%.1
59.41.
55.12.
59.7g.
67 62.
70
64.2
66.3
32.916
34.7a.7
52.9g
33.4
47.8g.6
67.1g.
60.5
ANLI-R2
34.01.7
34.71.0
42.514
36.11.7
42.11
$3.326
ANLI-33
32.716
36.9ts
44.21,
36.2
42.12.9
52.128
50-
47.3
CB
$1.360
78.67,8
91.4g9
85.74.6
93.61,
91.01,8
Emetion
71.9g2
42.0g,5
65.425
63.9%,
72.12m"
74.31.
40
95.112
54.3
96.5p4
92.81,5
95.61.s
96.1a.
80
CR
Amazom-CT
90.71.
60.1a0
0.02
91.76.s
24.07.
93.7u2
55.011.
90.6aa
59.4us"
92.01.5
93:2as
62.716
70
60.7
66.2
62.2
SST-5
49.2um
42.4.5
47
47.7
60
Average
59.9g2
49.83.0
64.5pg
61.24a
67.1go
50
47.3
40
Table 2: Results and computational costs using different upstream models, 32 samples per class. All results are computed without Monte-Carlo approx. Datasets that appear in an upstream model's training are greyed out. WiC and WSC were excluded from all averages.
Zero shot (78)
Switt
Treew (d))
Aut (Bant]]
Autifow (10)
Figure 4: Average scores when finetuned on 16, 32, and 64 samples per class. AuT--Few performs better relative to the baselines with more training samples.
capabilities of the model's much larger and diverse instruction-tuning These results highlight the ef- fectiveness of AuT-Feew across upstream models of varying sizes.
32, and 64 samples, respectively. With 16, we observe slightly worse performance than T-Few with AuT-Few. The provided signal from only 16 samples is too noisy for our automation pipeline. With an increase in number of training samples fol- lows larger lead of AuT-Few over other models. While AuT-Few (TO) is on average 2.1 points better than T-Few with 32 samples, this lead increases 3.1 for K 64. Similar observation is made when comparing AuT-FFew (BARTO) with SetFit.
'Tee computational costs for training and infer- ence are listed in Table We follow the approach adopted by Liu et al. (2022) and Tunstall et al. (2022) to measure computational costs, namely FLOPs-per-token (Kaplan et al., 2020). AuT-Few requires about 7x the training cost of T-Few, yet remains computationally accessible, taking only few hours to train on single A10G GPU, since the number of training steps for few-shot PEFT is overall small. Similarly, while AuT-Few with BARTO takes 4.2x longer than SetFit, it still takes less than an hour of total training time. Impor- tantly, during inference, AuT-Few is as efficient as T-Few (excluding Monte-Carlo approximation, otherwise scaling linearly with the number of re- trieved templates). AuT-Few with BARTO is even more efficient than SetFit during inference, requir- ing only 60% of its computation while maintaining a competitive score.
Real-world evaluation: RAFT. RAFT (Alex et al., 2021) a benchmark targeted towards eval- uating few-shot classification methods. consists of 11 datasets, from various domains, such as the legal or medical domain. In RAFT 50 randomly sampled training samples are provided, with po- tentially imbalanced label distribution. We submit- ted predictions of AuT-Few with the 11B Flan-T5 backbone, with handcrafted prompts provided by RAFT (AuT-FFew (H)), as well with our au- tomated prompts (AuT-Few). We do make any manual dataset adjustments, with the exception of Banking_77 as only a subset of the classes appears in its training data, c.f. App. C.2.
We emphasize that while T-Few takes somewhat less computation than AuT-Few, T-Few requires significantly more human intervention, and human time is much more valuable than computer time. The difference of couple hours of computer time is negligible when it can save orders of magnitude more human time and associated costs.
Results are shown in Table 3. Our method with handcrafted prompts and the Flan-T5 upstream model achieves rank-1 with the overall highest average score. Our automated version achieves scores slightly below T-Few (the previously 2nd ranked system). This is largely due to AuT-Few 's poor performance on a single dataset, Tweet-Fvail -val--
Varying sample sizes. Figure 4 shows the perfor- mance of our baselines as well as Aut-Few over 16,

---

Hate, result of improper selection of answer choices. However, AuT-Few has the best aver- age rank across five models with 2.15. wins against T-Few on 7 out of 11 datasets. Furthermore, it has the highest overall win rate, winning against all other models we considered (including our ap- proach with handcrafted prompts) on 4 out of 11 datasets, see Table 7. These results highlight AuT- Few's robustness and generalizability to real-world classification tasks.
Sctup
Avg. Score
AuT-FFew
66.32.6
retrieved template (randomized)
65.72.9
Template
w/ entire Collection
65.62.9
only dataset
65.52.7
Choices
only lemplate-tailored
63.3g.4
only iopic-specific
62.24.8
Monte-Carlo approximation
65.83.0
Improw:
only LoRA
63.4g.4
only (IA)*
65.2g.5
Table 4: Ablation for AuT-Few with 32 samples per randomized indicates randomly selected tem- plates, entire Coll. considers PromptSource prompts.
class:
Rank
Method
Avg. Score
Avg. Rank
AuT-Few (H)
77.3
2.82
and tasks, and the best average rank across datasets on the RAFT benchmark. Machine learning, es- pecially few-shot learning, is about automation. Although T-Few takes less computation, requires hand-designed prompts which involves significant human intervention and expertise. Human-tinn time is profoundly more valuable than computer time, and Few--Few saves this valuable human time while still retaining computational tractability. Future work includes the identification of causes for the obser- vations made in section 3, particularly for datasets that are completely unaffected by the prompt's de- sign (e.g Enronspam and CR).
AuT-Few
74.7
2.45
1
yiwise
76.8
2.55
2.
T-Few
75.8
2.82
12
SctFit
71.3
4.27
5
Human baseline
73.5
Table 3: Results on the RAFT henchmark as of Octo- ber 2023. Avg. Rank is reported across the shown models. Our method with handcrafted prompts achieves rank-1 with the overall highest average score while AuT- Few has the best average rank and highest win rate.
Ablation. Results our ablation study for AuT- Few with 32 samples per class are shown in Ta- ble 4. We ablate our template retrieval method by considering randomly selected templates from the instruction tuning KB, as well template re- trieval from the entire PromptSource collection of prompts, As seen both settings perform worse than AuT-Few, with higher standard deviation across seeds. While retrieving from the entire collection performs slightly better for tasks that appear in it (e.g. NLI, emotion classification), it strongly under- performs unseen ones (e.g. WiC, Amazon-CFi Further, the ablation of the choice options shows that each definition of answer choices by itself per- forms worse than AuT-Few (including the label descriptions that appear in the dataset). Finally, we see that our modifications to T-Few's inference and training are effective, with both LoRA and (IA)3 PEFT performing worse individually. Note that AuT-Few still outperforms T-Few even when us- ing only (IA)3, indicating AuT-Few's superiority without any architectural adjustments.
Limitations
This work and the automation pipeline is con- strained to classification tasks in English. The role of templates and answer choices is necessarily dif- ferent for tasks such as natural language generation (e.g. summarization or question answering) where a single textual class representation does not exist. The proposed automated few-shot approach is not expected to work well under extremely low data regime or when training samples are highly imbal- anced (i.e. < 8 samples per class) some data signal is required for optimizing the choice space. While our evaluation aims to cover a diverse range of classification tasks, the list of evaluation tasks is not exhaustive. Subsequently, there is no guarantee that AuT-Few performs equally well on every un- seen tasks, particular ones that divert strongly from tasks the model has seen during instruction tuning.
Ethics Statement
Our paper makes state-of-the-al few-shot classifi- cation methods more accessible non-experts for real-world problems. The goal of this work not to replace human involvement the deployment AI systems but instead to shift human resources to
6 Conclusion
AuT-Few replaces hand-designed task-specific prompts with automated templates, and achieves state-of-the-ar results on a wide range of datasets

---

other essential aspects of model deployment such as the analysis of data, biases, or system errors. We discussed the computational costs of our automa- tion approach and show that they are comparable at similar model size with the most efficient few-shot systems, which themselves again are computation- ally much more efficient than full-data and full model fine-tuning, or in-context learning.
Valerio Basile, Cristina Bosco, Elisabetta Fersini, Deb- ora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. 2019. Semeval-2019 task 5: Multilingual detection of hate speech against immigrants and women in twitter. In Proceedings of the 13th international workshop on semantic evaluation, pages 54-63.
Steven Bird and Edward Loper. 2004, NLTK: The natu- ral language toolkit. In Proceedings of the ACL In- teractive Poster and Demonstration Sesxions, pages 214-217, Barcelona, Spain. Association for Compu- tational Linguistics.
Acknowledgements
The authors would like to thank Lewis Tunstall for his help to submit AuT-Few's predictions to the RAFT leaderboard and Aditya Rawal for pointing us to relevant related work. We would also like to thank the anonymous reviewers for their time and effort giving us valuable feedback on our paper.
lñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020. Efficient intent detection with dual sentence encoders. In Pro- ceedings of the 2nd Workshop Natural Language Processing for Conversational Al, pages 38-45, On- line. Association for Computational Linguistics.
Jiaao Chen, Aston Zhang, Xingjian Shi, Mu Li, Alex Smola, and Diyi Yang. 2023. Parameter-efficient fine-tuning design spaces.
References
Neel Alex, Eli Lifland, Lewis Tunstall, Abhishek Thakur, Pegah Maham, C. Jess Riedel, Emmic Hine, Carolyn Ashurst, Paul Sedille, Alexis Car- lier, Michael Noetel, and Andreas Stuhlmüller. 2021. RAFT: A real-world few-shot text classification benchmark. In Thirty-fffth Conference on Neural Information Processing Systems Datasets and Bench- marks Track (Round 2).
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Debghani, Siddhartha Brahma, et al. 2022. Scaling cestucttionfinetune language models. arXiv preprint 1xXix:2100.11416
Alexis Conneau and Douwe Kiela. 2018. SentEval: An evaluation toolkit for universal sentence representa- tions. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (I.REC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Akari Asai, Mohammadreza Salehi, Matthew E. Peters, and Hannanch Hajishirzi. 2022. Parameter-effiieet multi-task tuning via attentional mixtures of soft prompts. In EMNLP, Abu Dhabi, United Arab Emi- rates.
Ganqu Cui, Shengding Hu, Ning Ding, Longtao Huang. and Zhiyuan Liu. 2022. Prototypical verbalizer for prompt- prompt-based few-shot tuning. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 7014-7024, Dublin, Ireland. Association for Compu- tational Linguistics.
Carolyn Ashurst, Emmie Hine, Paul Sedille, and Alexis Carlier. 2022. Ai ethics statements: analysis and lessons learnt from neurips broader impact state- ments. In 2022 ACM Conference on Fairness, Ac- countability, and Transparency, pages 2047-2056.
Stephen Bach, Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M. Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-david, Canwen Xu, Gun- jan Chhablani. Han Wang. Jason Fries. Maged Al- shaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang, Dragomir Radev, Mike Tian-jian Jiang, and Alexander Rush. 2022. Prompt- Source: An integrated development environment and repository for natural language prompts. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstra- tions, pages 93-104, Dublin, Ireland, Association for Computational Linguistics.
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment chal- lenge. In Proceedings of the PASCAL Challenges Workshop on Recognising Textual Entailment.
Aarie-Catherin de Marneffe, Mandy Simons, and Ju- dith Tonhauser. 2019. The commitmenthank: Inves- tigating projection naturally occurring discourse. Proceedings of Sinn und Bedeutung. 23(2):107-124.
Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4188 Minneapolis, Minnesota. Association for Computational Linguistics.
M Saiful Bari, Aston Zhang, Shuai Zheng, Xingjian Shi, Yi Zhu, Shafiq Joty, and Mu Li. 2022. Spt: Semi- parametric prompt tuning for multitask prompted learning. arXiv preprint 2rXiv:2212. 10929.

---

Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 38 16-3880, Online. Association for Computa- tional Linguistics.
Reasoning, KR'12, pages 552-561. AAAI Press, Rome, Italy.
Bill Yuchen Lin, Kangmin Tan, Chris Miller, Beiwen Tian, and Xiang Ren. 2022. Unsupervised cross- task generalization via retrieval augmentation. In NeurIPS 2022, New Orleans, LA, USA.
Marco Lippi, Przemystaw Palka, Giuseppe Contissa. Francesca Lagioia, Hans- WollsWoggg Micklitz, Gio- vanni Sartor, and Paolo Torroni. 2019, Claudette: automated detector of potentially unfair clauses in online terms of service. Artificial Intelligence and Law, 27(2): 17-139.
H. Gurulingappa, A. M. Rajput, A. Roberts, J. Fluck, M. Dofmann-Apitius and L. Toldo. 2012. Devel- opment of Benchmark Corpus to Support the Au- tomatic Extraction of Drug-related Adverse Effects from Medical Case Reports. Journal of Biomedical Informatics. Epub ahead of print.
Haokun Liu, Derek Tam, Mohammed Mugeeth. Jay Mo- hta, Tenghao Huang, Mohit Bansal, and Colin Raffel. 2022. Few-shot parameter- better and cheaper than in in-oontext learning. arXiv preprint arXiv:2205.05638.
Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. 2021. WARP: Word-level Adversarial ReProgramming. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pupers), pages 4921-4933 Online. Association for Computational Linguistics.
fine-tuning is
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang. and Jie Tang. 2021. Gpt understands, too. 1xXiv2200.10385..
Ruotian Ma. Xin Zhou, Tao Gui, Yiding Tan, Linyang Li, Qi Zhang, and Xuanjing Huang. 2022. Template- free prompt tuning for few-shot NER. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 5721-5732, Seattle, United States. Association for Computational Linguistics.
Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022a. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
Shengding Hu, Ning Ding, Huadong Wang, Zhiyuan Liu, Jingang Wang, Juanzi Li, Wei Wu, and Maosong Sun. 2022b. Knowledgeable prompt-tuning: Incor- porating knowledge into prompt verbalizer for text classification. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 2225-222222202222222222--22---22 2240, Dublin, Ireland. Association for Computational Lin- guistics.
Vangelis Metsis, Ion Androutsopoulos, and Georgios Paliouras. 2006. Spam filtering with naive bayes which naive bayes? In Proceedings of the 3rd Conference on Email and Anti-Spam (CEAS 2006).
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language under- standing. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 4885-4901, Online, Association for Computa- tional I. Linguistics.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint 2rXiv:2001.88361
James O' O'Nlill, Polina Rozenshtein, Ryuichi Kiryo, Mo- toko Kubota, and Danushka Bollegala. 2021. I wish would have loved this one, but didn't multilin- gual dataset for counterfactual detection in product review. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7092-7108, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Rabech Karimi Mahabadi, James Henderson, and Se- bastian Ruder. 2021. Compacter: Efficient low-rank hypercomplex adapter layers. Advances in Neural Information Processing Systems, 34:1022-1035.
Rabeeh Karimi Mahabadi, Luke Zettlemyer, James Henderson, Lambert Mathias, Marzieh Saeidi, Veselin Stoyanov, and Majid Yazdani. 2022. Prompt- free and efficient few-shot learning with language models. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3638-3652, Dublin, Ireland. Association for Computational Linguisties.
Mohammad Taher Pilehvar and Jose Camacho Colladoss 2019. WiC: the word-in-contex dataset for evalu- ating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 1267-1273, Minneapolis, Minnesota. Association for Computa- tional Linguistics.
Hector J. Levesque, Frnest Davis, and Leora Morgen- stem. 2012. The Winograd Schema Challenge. In Proceedings of the Thirteenth International Confer- ence on Principles of Knowledge Representation and

---

Daniel Preotiuc-Pietrs Mihaela Gaman, and Nikolaos Aletras. 2019. Automatically idemtifying complaints in social media, arXiv preprint arXiv: 038.0.
Timo Schick and Hinrich Schütze. 2022. True Few-Shot Learning with Prompts- A Real- World Perspective. Transactions of the Association for Computational Linguistics, 10:7 16771.
Guanghui Qin and Jason Eisner. 2021. Learning how to ask: Querying LMs with mixtures of soft prompts. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- fional Linguistics: Human Language Technologies, pages 5203-5212, Online. Association for Computa- tional Linguistics.
Taylor Shin, Yasaman Razeghi, Robert Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Elic- iting Knowledge from Language Models with Auto- matically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages Online. Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP),, pages 3982-3992, Hong Kong, China. Association for Com- putational Linguisties.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013, Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing, pages 1631 1-642, Seattle, Washington, USA. Association for Computational Linguistics.
Jess Riedel and Angelica Deibel. 2020. Tai safety bibli- ographic database. Online.
Lewis Tunstall, Nils Reimers, Unso Eun Seo Jo, Luke Bates, Daniel Korat, Moshe Wasserblat, and Oren Pereg. 2022. Efficient few-shot learning without prompts. arXiv preprint arXiv:2209. 11055.
Alexander K Saeri, Peter Slattery, Joannie Lee, Thomas Houlden, Neil Farr, Romy Gelber. Jake Stone, Lee Huuskes, Shane Timmons, Kai Windle, et al. 2022. What works increase charitable donations? meta- review with meta-meta-analyye national Journal of Volumiary and Nonprofit Organi- zations, pages 1-17.
Sowmya Vajjala and Ivana Lučié. 2018. On- eStopEnglish corpus: A new corpus for automatic readability assessment and text simplification. In Pro- ceedings of the Thirteenth Workshop Innovative Use of NLP for Building Educational Applications. pages 297-304, New Orleans, Louisiana. Association for Computational Linguistics.
VOLUNTAS: Inter-
Victor Sanh. Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M. Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang. Han Wang, Matte Manica, Sheng Shen, Zheng Xin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang, Tr- ishala Neeraj. Jos Rozen, Abheesht Sharma, An- drea Santilli. Thibault Fevry, Jason Alan Fries, Ryan Techan, Teven Le Seao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. 2022. Multi- task prompted training enables zero- shot task gener- alization. In International Conference on Learning Representations.
Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou', and Daniel Cer. 2022. SPoT: Better frozen model adaptation through soft prompt transfer. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pa- pers). pages 5039-5059, Dublin, Ireland. Association for Computational Linguistics.
Han Wang, Canwen Xu, and Julian McAuley. 2022a. Automatic multi-label prompting: Simple and inter- pretable few-shot classification. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 5483-5492, Seattle, United States. Association for Computational Linguistics.
Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen. 2018 CARER: Con- textualized affect representations for emotion recog- nition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3687-3697, Brussels, Belgium. Association for Computational Linguistics.
Yizhong Wang. Swaroop Mishra, Pegah Alipoor- molabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. 2022b. Benchmarking generalization via in-context instructions on 1, 1,600+ language tasks, arXiv preprint arXiv:2204.07705.
Timo Schick and Hinrich Schütze. 2021. It's not just size that matters: Small language models are also few- shot learners. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 2339-2352, Online. Association for Computational Linguistics.
A Abertt Webson and Ellie Pavlick. 2022. Do prompt- based models really understand the meaning of their prompts? In Proceedings of the 2022 Conference of the North American Chapter of the Association for

---

Computational Linguisties: Human Language Tech- nologies. pages 2300-2344, Seattle, United States. Association for Computational Linguistics.
A Datasets
We conduct experiments on a total of 12 text classi- fication datasets. The tasks we consider are 1) nat- ural language inference: RTE (Dagan et al., 2005) CB (de Marneffe et al., 2019), ANLI (Nie et al., 2020); 2) coreference resolution: WSC (Levesque et al., 2012); 3) word sense disambiguation: WiC (Pilchvar and Camacho-Collados, 2019); 4) coun- terfactual detection: Amazon-CHt (O'Neill et al., 2021); 5) sentiment classification: SST-5 (Socher et al., 2013), Customer reviews (CR) (Conneau and Kiela, 201 8); 6) emotion classification: emo- tion (Saravia et al., 2018); and 7) spam detection: Enron (Metsis et al., 2006). All datasets are in English. Enron contains personal identifiable in- formation, yet substantial efforts have been made to remove any integrity problems and samples of affected employees, see here for reference. Enron is an long-established dataset to use for classifica- tion problems and our use is in line with previous usages of it.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V Le. 2022a. Finetuned language models are zero-shot learners. In International Con- ference on Learning Representations.
Jason Wei. Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi. Tatsunori Hashimoto. Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022h. Emer- gent abilities of large language models. Transactions on Machine Learning Research. Survey Certifica- tion.
Seonghyeon Ye, Joel Jang, Doyoung Kim, Yongrae Jo, and Minjoon Seo. 2022. Retrieval of soft prompt en- hances zero-shot task generalization, arXiv preprint arXiv:2210.0302.
Aston Zhang, Yi Tay, Shuai Zhang, Alvin Chan, Anh Tuan Luu, Siu Hui, and Jie Fu. 2021. Beyond fully-connected layers with quaternions: Parameter- ization of hypercomplex multiplications with 1/n parameters. In International Conference on Learn- ing Representations.
B Implementation Details
Parameter-efficiet fine-tuning via low-rank adaptation and rescaling While exclusively rescaling weights of an upstream model via IA3 has shown to perform remarkably well, the ex- pressiveness of the fine-tuning process is restricted, due to Ah (the accumulated gradient update) be- ing always of the form Wo AWo J. with Wo being the weights of the upstream model and À being the rescaling vector. For tasks that require major adaptation capabilitics, this might pose a hindrance. In contrast, LoRA explicitly models via decomposition matrices the gradient update Ah BA, resulting in higher expressiveness (about 10x as many parameters as IA3), but has repeatably shown in our experiments have sub- stantially higher variability. We hence combine both PEFT strategies, by rescaling both the weights of the upstream model and the accumulated gradi- ent updates jointly: h = X(Wo + BAx).. After training, both À and BA can be applied to Wo. mak- ing the weight updates persistent without inquiring any additional computation during inference. Fol- lowing Liu et al. (2022), we pre-train the weights of the rescaling vectors in a similar fashion the upstream model. While the authors only train the vectors for 100K steps, we observed further im- provements when training them for longer (500K steps).
Lucia Zheng, Neel Guha, Brandon R. Anderson, Peter Henderson, and Daniel E. Ho. 2021. When does pre- training help? assessing self-supervised learning for law and the casehold dataset of 53,000+ legal hold- ings. In Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law, ICAIL '21, page 159-168, New York, NY, USA. Association for Computing Machinery.
Yongchao Zhou, Andrei loan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023. Large language models are human-level prompt engineers. In The Eleventh International Conference on Learning Representations.

---

Inference via Monte-Carlo Approximation Templates As outlined section 3, in its current version the expectation over template and choice space is approximated during inference by ran- domly drawing a template from a collection of handcrafted ones. Besides being non- non-deterministic, the selected template might be poor approxima- tion of the overall expectation. Instead, we run Monte-Carlo Approximation over the template space A. by computing a weighted average over all retrieved templates:
choices using NLTK (Bird and Loper, 2004). Our code and models will be made openly accessible under Apache License 2.0.
Baselines addition to the current state-of-the- art (Liu et al., 2022), we consider SetFit (Tun- stall et al., 2022), as well as standard fine- tuned LM. SetFit is of particular relevance to us since it is the state-of-the-art prompt-free few- shot method, shown perform competitively '-FFew in their experiments while being computa- tionally substantially more efficient. In their com- parison to T-Few, they use a very small variation of the sentence-transformmr MPNET, consisting only 1 110M, however, we observed substantially bet- ter performance with the larger ROBERTa sentence- transformer model (355M parameters). Hence, we report results the latter model The traditionally finetuned model is RoBERTaLARGE model, fully- finetuned with an additional linear head, based on the baseline (Tunstall et al., 2022).
ÿ argmax, Ex. w (yr x, 1 MF)
R
argmaxy Σ
po
x. &r, WA).
We determine the weights for each template by computing the log-likelihood each template on Diest and applying a softmax function on them, following the previously men- tioned motivation.
with El= Wy
Since our joint PEFT method converges substantially faster than IA by itself, we set the number of total steps to 600 (contrary 1000 used by T-Few). Further, for both T-Few and AuT-Few we use the following hyperparam- eters across all experiments: we use Adam, a learning rate of 1-3 cosine decay with a warmup ratio of 0.06, a learning rate decay of 1.0, and a batch size of 8. The contextual embeddings for template retrieval well the topic-specific choices generated using sentence- tranfforrtr all-MiniLM--66-2 encoder model. For all main experiments, we set the number of retrieved tem- plates to R = 5. The underlying prompt knowl- edge base used is PromptSource (Bach et al., 2022). For selecting the best answer choices, we split the training data using 3-fold cross-validation and train the upstream model with identical hyperparameters our final model for every choice option.
Hyperparameters
C Detailed Results
Detailed results of the experiment in Table 1 for different sample sizes are shown in Table 8.
C.1 Results Blind Test Set
Dataset
T-Few
AuT-Few
RTE
87.2
82.1
ANLI-RI
60.7
54.6
ANLI-R2
52.1
49.1
ANLI-R3
51.9
51.8
CB
93.6
96.0
Emotion
62.1
71.7
Enron
97.0
97.6
Amazon-CF
50.2
62.6
CR
93.7
92.8
System & Code All models (550M, 3B, 11B pa- rameters) are trained and run on a single A10G GPU with 23GB of memory by using gradient checkpointing, bfloat16 floating-point format, and in the case of the IIB model by offloading param- eters using DeepSpeed 6 We produce results for the SetFit and finetune baseline using the associ- ated repository?. We filter stopwords and punctua- tion from the vocabulary of copic-specing answer
SST-5
56.6
55.1
Avg
70.5
71.3
Table 5: Results on the held out test set. Excluding WiC and WSC as these wereeen TOpp's 's pre-training.
8 AAlbit some sentence-transfommer models are targeted for a certain domain, e.g. QA or NLI, in our experimental setup we aim to minimize human involvement, including model selection. Hence, the same pre-trained model used across all experiments.
t/gittt http://ttttuuuoooiiiiitttcott wtt//oi DDeeppee 'https: htwp:ttwww..co//ggggggggga////te

---

C.2 Results RAFT
RAFT consists of datasets from different do- mains. The individual datasets included in RAFT are ADE Corpus V2 (medical case reports) (Gu- rulingappa et al., 2012), Banking77 (Casanueva et al., 2020), NeurlPS impact statement risks (Ashurst et al., 2022), Onestop English (Vajjala and Lučié, 2018), Overruling (legal domain) (Zheng al., 2021), Systematic Review Inclusion (Saeri al., 2022), Tai safety research (Riedel and Deibel, 2020), Terms of Service (Lippi et al., 2019), Tweet Eval Hate (Basile et al.. 2019), and Twitter Com- plaints (Preotiuc-Pietro et al., 2019). All datasets are in English.
Since only small subset of the 77 classes appear in the training data of the Banking_77 77 dataset, directly use the dataset's class representations for the answer choices. Banking_77 77 is strictly speak- ing a zero-shot and few-shot evaluation dataset and previous work such as SetFit that does not use a verbalizer at all also had to make use of the given class representations for that dataset".
D Automated Choices
The generated and selected answer choices as used in AuT-Few with K 32 and TO as the upstream model on seed 0 are shown in Table
E Automated Templates
The retrieved templates as used in AuT-Few with K and TO as the upstream model on seed are shown Table 10.
"https::/ttwwrrdssatttsciie com/
sentence- frrannffrreer--i--tuuiiiiin
outperforms- tertperrfoomstttt--oo-f----- -hee-shtttttttttssss class)

---

Dataset
Handcrafted Dataset Choice
Automated Choice
Ade
ADE-relatednoot ADE-related
Banking
c.f. C.2
c.f. C.2
Neurips
I mention a harmful application/mentions
doesn't t mention harmful application/mmentions
harmful application
harmful application
One
deementaayy/mmeemecdatt/attt
Black/World/ciience
Stop
Overrul-
not overuulingoovrrrling
court/overrule
ing
Org
company/rescarch instifute/universityy
company/research institute/universityy
Types
Review
included/not included
included/not included agent/leaniin
TAI safety research not TAl safety research
Safety
ToS
not potentially unfair/potentially unfair
not potentially unfair/potentially unfair Sports/World
Eval
hate speech/not hate speech
Hate
Com-
complaint/no complaint
complaint/no complaint
plaints
Table 6: Generated answer choices, when using TO and 32 samples for seed O.
System
Ade
Banking
Neurips
One Stop
Overraling
Org Types
Review
Safery
ToS
Comploinzs
AuT-Few (H)
0.837
0.647
0.78
0.847
0.942
0.917
0.687
0.703
0.728
0.517
0.892
AnT-Few
0.846
0.898
0.77
0.801
0.742
0.738
0.350
0.901
yiwise
0,856
0.839
0.698
0.944
0,906
0.493
0.737
0.749
0.647
0.883
T-Few
0.804
0.695
0.833
0.676
0.95
0.915
0.508
0.736
0.75
0.586
0.879
SerFi
0.799
0.632
0.859
0.76
0.93
0.769
0.503
0.664
0.604
0.487
0.831
Table 7: Results on RAFT.

---

Majority
Zero-shot
Finetune
SetFit
T-Few
AuT-Few (II)
AuT-Few (wlo D)
AuT--Few (A)
RTE
52.7
65.61.2
60.326
52.74.0
81.01.5
81.83.
81.02.1
80.11.5
WSC
63.5
62.13.9
53.748
50.25.3
61.914
65.0s.7
50.54.8
48.96.2
WIC
50.0
51.30.6
53.341
57.0s9
54.429
60.62.
52.74.5
54.94.7
ANLI-RI
33.4
35.60.8
32.91.
32.31.
50.22.0
51.12.1
47.4..5
48.0g.7
ANLI-R2
33.4
33.6p.7
34.31.0
34.01.7
42.4p.7
40.8
41.30.6
41.11.9
ANLI-R3
33.5
34.20.a
33.2
32.30.9
43.01.5
42.81.
36.92.2
38.15.0
CB
50.0
57.50.8
64.36.2
81.44.7
85.72.
91.82.1
85.7a.1
87.57.7
Emotion
35.2
42.10.8
37.513
68.41.7
62.0g.2
72.63.3
61.5%11
66.01.
Fnron
50.9
53.30.4
90.733
94.51.
95.6p.9
96.11.
93.616
92.7g.
Amazon-CT
0.00
0.040.7
20.412.8
56.7.0
23,45,3
61.712.1
32.48.5
38.615.2
CR
64.2
88.90.4
74.7k1
91.01.1
93.42.
93.6n.4
92.71.3
92.61.7
SST-5
26.3
38.91.
37.5a.1
47.91.
51.72.3
52.10.9
51.6141
51.213
Average
41.1
47.31.0
48.54.6
58.126
62.12.3
64.4a.3
60.73.7
61.74.5
16 samples per class.
Majority
Zero-shot
Finetune
SetFit
T-Few
AuT-FFew (H)
AuT-Few (A w/o
AuT-Few (A)
65.61.2
$6.46.6
51.41.8
82.524
81.83.9
82.34.
81.42.4
WSC
63.5
62.13.
49.27.1
50.3
70.2a.
65.0s.r
50.8
59.21.5
WIC
50.0
51.30.8
53.95.1
55.0s.1
55.914
60.62.5
55.63.9
58.45.1
ANLI-RI
33.4
35.60.a
32.11.9
32.91.
52.920
51.12.1
50.13.8
49.13.7
ANLI-R2
33.4
33.6u.7
33.41.
34.01.7
42.514
40.81.9
42.71.8
42.01.
ANLI-33
33,5
34.20.8
31.51.
32.71.0
44.2
42.8
42.93.8
43.53.0
CB
50.0
57.50.8
86.16.6
84.35.0
91.43.2
91.82.0
93.92.0
93.916
Emotion
42.10a
3..6
71.93.2
65.42.
72.63.
70.52.2
72.625
Enron
50.9
53.30.4
92.224
95.11.2
96.50.4
96.11.1
95.51.2
95.50.5
Amazon-CF
0.00
0.04o.7
40.5a.
60.110
24.07.5
61.7121
53.2a.3
59.0s2
CR
88.96.4
84.813
90.71.7
93.70.
93.60.1
93.01.
92.51.1
SST-5
26.3
38.91.
42.1s.
49.2n.9
51.5
52.10.9
50.0s.2
48.62.5
Average
41.1
47.31.
55.044
59.02.
64.224
67.5..3
65.
66.32.6
32 samples per class.
Majority 52.7
Zero-shot
Finetune
SetFit
T-Few
AuT-Few (H)
AuT-Few
AuT-Few
RTE
65.61.
52.1s.1
52.31.1
86. lo.4
85.41.2
85.228
85.71.
WSC
63.5
62.1s9
48.824
48.95.3
71.72%
72.74.4
58.2s.1
65.15.9
51.30.
56.3s.
56.72.3
58.23.
60.73.3
56.81.1
58.73.1
ANLI-RI
33.4
35.60.8
34.31.
34.01.
55.021
52.42.
54.22.5
52.8s.7
ANLI-R2
33.4
33.60.7
36.43.a
33.32.2
43.50.9
44.42.2
45.1
45.1
ANLI-R3
33,5
34.20.8
33.42a
33.61.4
44.60.9
42.32.
45.12.1
44.5
CB
50.0
$7.50.8
84.2a.2
88.52.7
93.2a.
93.23.6
96.1
95.70.9
Emotion
35,2
42.1oa
72.22.4
76.92.4
69.0
80.12.0
75.2
80.1
Enron
50.9
53.30.4
95.12.
96.0g.s
97.1o.
97.20.9
97.10.1
97.80.1
Amazon-CF
0.00
0.040.7
55.74s
84.3
29.8
62.84.2
64.5a.7
66.6s.1
CR
64.2
88.904
89.31.
91.61.0
94.0u.7
94.3u8
92.621
92.61.8
SST-5
26.3
38.91.
46.11.1
50.81.3
$2.3
50.0s.1
49.43.2
45.65,2
Average 41.1
47.31a
58.62.
60.72.
66.21.7
69.62.
68.32.
69.226
64 samples per class.
Table 8: Results with TO upstream model. (H): Handcrafted, (A w/o D): Automated Prompts without dataset label candidates, (A): Automated Prompts.

---

Dataset
Answer Choice
Datasct
lemplate-tailored
Topic-Specific
Selected
RTE
entailment/not-enntee enttilmmet
/ No
scandal dictator
Yes Noo
WiC
No Yes
/ work
force sentence
No Yes
No
good / Yes
peter
No/ Yes
ANLI-
entailment/ neutral/
Yes / 
/ No
hound market presence
Yes / </>>>>>>>//// No
RI
contradiction
CB
entailment/ contradiction / neutral
Yes No/ no
pussage sentence funny
Yes No / no
Emo-
sad-
nega- tveeposs/good/bud/ppsiiii enrononline pricing fabric perfect
sadness / /joy love / anger fear /surprise enrononline pricing
tion
ressjoyylovelnnge//e//ssuprisss ham spam
Enron
Business 5 positive negative
Amazon
/ /
CF
counterfactual
counterfactual/coomrriicc
negative positive
negative/ positive No / negative / <unk> Yes positive
mp3player / ipod filmmaking genre / scene
mp3player / ipud
SST-5
very negative negative
No / negative / <unk>>/ Yes positive
neutral positive very positive
documentary cinematic
Table 9; Generated and selected answer choices, when using TO and 32 samples for seed 0.

---

Dataset
Rank
Template
[premise)] } Question: [[hypothesis] }
[[premise]] 'n a paraphrase the following sentence? (hypothesis] )
RTE
[prremise]] ]
that paraphrasing the following sentence? [hypothesis] )?
| [preemise] ) Question: [hypothesiis)
Sentence 1: [(premise)] ) Sentence ((hypothesis)
Question: Does Sentence paraphrase
Sentence 2?
Pick one category for the following text. 'hee options are [sentencee]]] ]] J. [[sentence]]] [word]
2
[sentencee] | [scntence2]] Given choice of categories | [word]] / }. the text refers to which onc?
WiC
This is a correct unswer to the following word about [sentencel] ). \n Answer: [[sentence2] In
Question: | [wordd]
Pick one category for the following text. The options are [[sentence]] ]]. [[entencee]]] ( [word]
2
([sentence1) } [[sentence2]] Given choice of categories This is a correct unswer to the following word about [sentencel] ). \n Answer: [[sentence2] In
[ [wword]] the text refers to which one?
WSC
3
Question: | [wordd]
[premise } Question: [[hypothesis]
[[premise)] ) la Is that paraphrase of the following sentence? lu (hypothesiss) [premisee] In Is that paraphrasing the following sentence? In (hypothesis)
ANLI
)
{ [premise } Question: [hypothesis]]
Sentence I: ((premise) Sentence 2: [hyppthhssss] In Question: Does Sentence parapprase
Sentence 2?
[premiise]] Question: [[hypothesis]]
[[premise) In a paraphrase the following sentence? (hypothesis] )
[premisse] ] that paraphrasing the following sentence? (hypothesis] )
( [prremise]] ) Question: | [hypothesis
Sentence I: ((premise)) Sentence 2: { (hyppothesis) Question: Does Sentence paraphrase
Sentence 2?
[ [text]] How does the viewer feel about the movie? { [text] } How does the reviewer feel about the movie?
2
Emotion
3
[ [teext] } Did regret it?
If you ask me whether like this place? The answer is [text]) (text)) ) Overall, the experience is [text)) If you ask me whether I will come again, answer is [text)) ) Will come here again? [text]] What is the sentiment expressed in this text? Based on that, my rating for this place is [text] [[text] } If you ask whether like this place? The answer is [[txx]] How does the reviewer fee) about the movic? ([text) Did the reviewer enjoy the movie? [ [text]] How does the viewer feel ahout the movic? Based on this review, would user recommend this product?
5
2
Enron
2
Amazon-CF
Review: [[eex)) } Answer: { [text]] What the sentiment expressed by the reviewer for the movic? Review: [ [text] \n Answer: { [text] How does the viewer feel about the movic?
Based on this review, would user recommend this product?
CR
{ [[ext]] How does the reviewer feel about the movie?
Review: [[ext]] In Overall rating:
[[text) } \n Overall, the experience is
| [txt]] How does the viewer feel ahout the movie?
2
{ [text]] How does the reviewer feel about the movie?
SST-5
3
[text]] || What sentiment does the writer express for the movie? The following movie review expresses what sentiment? ([text)) [textt) Did the reviewer enjoy the movie?
5
Table 10: Retrieved templates, when using TO and 32 samples for seed 0.


Reviews:
[{"review_id": "Y7ra1Tetub", "reviewer": "Reviewer_YfFv", "summary": "", "questions": "please refer to the reasons to reject section.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and comments on our paper! We address the reviewer's questions and the modifications we make to our paper based on the reviewer's suggestions below:\n\n**Q: [...] Given this, could you explain why there's a need for automatic prompt generation rather than using general prompts?**\n\nWe acknowledge that  “prompt robustness” in the abstract as a description for our findings in section 3 is too vague and potentially leaves the reader confused regarding our subsequent motivation for AuT-Few. We will ensure to adjust the abstract accordingly. \nInstruction fine-tuned language models in the context of PEFT exhibit surprisingly little variability (but not none!) on some dimensions of its design space (i.e. task-specific templates, template-tailored answer choices, multiple answer-choice configurations), however, they are susceptible to some others (i.e. answer choice configuration). Hence, an LM’s prompt remains a relevant hyperparameter worth optimizing, either by humans or automatically. For some datasets, the impact of optimizing the prompts is larger than for others (e.g. on Amazon Counterfactual, WSC, RTE, or ANLI prompts have a substantial impact on performance while on e.g. CR the performance is virtually unaffected by any modifications). \n\nAuT-Few is designed around the observations made in section 3: (i) distribute computational efforts for prompt optimization to reflect the components of the prompts that the instruction-tuned LM is more susceptibility to, i.e. spend more resources on answer choice automation than on template automation (template retrieval is extremely efficient but still provides performance improvements) (ii) optimize templates independently from answer choices for efficiency reasons (contrary to e.g. Gao et al., 2021 [1]), since template-tailored answer choices are not needed, (iii) select the single-best answer choice configuration instead of trying to find multiple distinct ones, as commonly done with handcrafted prompts.\n\n**It is noted that the inference time for this model is significantly longer than that of previous works. [...]**\n\nFor clarification: the inference time for AuT-Few is only longer when using Monte-Carlo approximation over all retrieved templates, however, without it (i.e. random sampling of retrieved templates) AuT-Few is *as efficient* as T-Few during inference while still outperforming it, see Table 4 in our ablation study. Notably, AuT-Few with the smaller BART0 backbone requires only 60% of SetFit’s computation during inference without Monte-Carlo, yet scores higher. Considering the efficiency of SetFit, we believe this is a notable achievement. We agree with the reviewer that the use of Monte-Carlo approximation during inference distracts from the overall efficiency of our model and will ensure to clarify this point by moving the scores with Monte-Carlo approximation to the appendix if accepted.\n\n**Q: The appendix includes examples of some strange automated choices [...] Could you provide an analysis explaining this?**\n\nOne surprising finding in section 3 is that on a couple of datasets (CR and Enron) completely uninformed and even destructive answer choices (i.e. inverted answer choices, see our comment to Reviewer 2 for a clear definition) have a negligible impact on the performance of instruction-tuned LMs in the context of PEFT. Both datasets are virtually unaffected by any modifications we made to both task templates and the answer choices (see Figure 3).  For CR, AuT-Few’s cross-validation strategy for selecting the best answer choice subsequently measures almost identical scores for all three choice configurations (90.1, 89.8, 90.4 for the dataset, template-tailored, and topic-specific choices, respectively), resulting in the seemingly erroneously selection of the topic-specific answer choices (mp3player / ipod). We will add a qualitative discussion of the selected answer choices and templates and analyse potential causes for our observations in section 3 regarding CR and Enron using the additional space provided in the camera-ready version if accepted.\n\n[1] https://aclanthology.org/2021.acl-long.295.pdf", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Autohrs conduct an analysis of diverse types of prompts, comparing their performance across a range of datasets. This comprehensive comparison can provide valuable insights that may greatly benefit future research in this field.\n\n2. Proposed method eliminates the need for manually hand-crafted prompts or any prerequisite background knowledge for training new tasks.\n\n", "reasons_to_reject": "1. You've mentioned in lines 009-012 that the instruction fine-tuned Language Model (LM) demonstrates significant prompt robustness. Given this, could you explain why there's a need for automatic prompt generation rather than using general prompts?\n\n2. It is noted that the inference time for this model is significantly longer than that of previous works. This aspect seems to detract from the model's overall effectiveness.\n\n3. The appendix includes examples of some strange automated choices that, despite their oddity, result in good performance. Could you provide an analysis explaining this?", "questions_to_authors": "please refer to the reasons to reject section.", "paper_topic_and_main_contributions": "The study presents an innovative method for generating prompts automatically, eliminating the need for manual crafting. This is achieved by retrieving appropriate task instructions from an instruction-tuning knowledge base and answer choices. Notably, this process can be executed with access only to training samples and categorical labels.\nTheir method, known as AuT-Few, is tested across 12 datasets and is shown to outperform the current state-of-the-art methods in few-shot learning. Further demonstrating its efficacy, the authors highlight the generalizability of AuT-Few by reporting successful results on tasks that were not seen during the training.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GeMlbgASP7", "reviewer": "Reviewer_rdNw", "summary": "", "questions": "Please refer to [reasons to reject]", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the time and comments on our paper! We address the reviewer's questions and modifications we make to our paper based on the reviewer's suggestions below:\n\n**Although the method outperforms T-Few on average, it only wins 4 out of 12 datasets (Table 1). Thus I think it is an overstatement that it outperforms T-Few.**\n\nWe acknowledge the reviewer’s point and will explicitly state the win rate of AuT-Few in the main results of the paper. However, we would like to add that while AuT-Few wins in 4 out of 12 datasets, the delta in performance for the cases where it wins is much larger than for the ones it loses (except to WSC), arguing in favour of AuT-Few’s robustness and overall performance. This observation is also confirmed on the RAFT benchmark, where AuT-Few performs more consistently than all other approaches, having the best average rank.\n\n**Q: What does inverted handcrafted answer choices mean (line 259)? Could you clarify?**\n\nWe will clarify the definition in section 3 of the paper. We recognize that *inverted* answer choices should rather be called *reversed* answer choices to avoid confusion and we will adjust the term in the paper. This prompt modification reverses the mapping from the labels to the answer choices, i.e. if $\\psi(0) = \\text{No}$, $\\psi(1) = \\text{Yes}$, then $\\text{reversed}(\\psi)(0) = \\text{Yes}$, and $\\text{reversed}(\\psi)(1) = \\text{No}$. For cases with more than two classes: e.g. $\\psi(0) = \\text{sadness}$, $\\psi(1) = \\text{joy}$, $\\psi(2) = \\text{love}$, $\\psi(3) = \\text{anger}$, $\\psi(4) = \\text{fear}$, $\\psi(5) = \\text{surprise}$, then $\\text{reversed}(\\psi)(0) = \\text{surprise}$, $\\text{reversed}(\\psi)(1) = \\text{fear}$,  $\\text{reversed}(\\psi)(2) = \\text{anger}$, $, \\cdots, $ $\\text{reversed}(\\psi)(5) = \\text{sadness}$. Hence, reversed answer choices are explicitly destructive prompts and they subsequently have the highest negative impact on performance (see Figure 3).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper proposes AuT-Few which could replace hand-designed task-specific prompts which can remove human intervention and expertise. \n\n2. Paper is well-written", "reasons_to_reject": "1. Although the method outperforms T-Few on average, it only wins 4 out of 12 datasets (Table 1). Thus I think it is an overstatement that it outperforms T-Few. \n\n2. What does inverted handcrafted answer choices mean (line 259)? Could you clarify?", "questions_to_authors": "Please refer to [reasons to reject]", "paper_topic_and_main_contributions": "The paper introduces AuT-Few, a method that eliminates the need for handcrafted prompts in few-shot learning. AuT-Few uses a retrieval module to fetch task instructions and generates two distinct class descriptions, selecting the best via cross-validation. Across 12 datasets and 8 tasks, AuT-Few outperforms leading few-shot methods and ranks top on the RAFT benchmark, all without task-specific prompts for new tasks.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pdBhYlpTbM", "reviewer": "Reviewer_YSqb", "summary": "", "questions": "Question A: Despite the conclusion that “neither template tailored answer choices nor multiple distinct answer choice configurations is needed in the empirical experiments” drawn in Chapter 3, would it be possible to combine the three answer choices, Dataset, template-tailored, and Topic-Specific, in a discrete space rather than choosing the one with the highest score? \n\nQuestion B: Why does Topic-Specific answer choice only consider single-word to semantically express the underlying content for each answer class, and was there any experiment on word number ablation?\n\nQuestion C: When comparing the FLOPS of training and reasoning with other models, could you explain in detail which improvements in your setting lead to the increase of FLOPS, and what are the marginal benefits they bring to the increase of FLOPS? Could you compare the training and inference time cost with other benchmark models in more detail based on the marginal benefits?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and comments on our paper! We address the reviewer's questions and the modifications we make to our paper based on the reviewer's suggestions below:\n\n**Question A: Combination of generated answer choice configurations**\n\nThis is a good question. We conducted experiments that combine the three answer choice configurations by: (i) optimizing directly over multiple answer choices $\\psi_j(y)$ and running a Monte-Carlo approximation during inference over both templates *and* answer choices, (ii) combining the textual descriptions for each class by string concatenation and/or autoregressive generation (e.g. for RTE, the entailment class would be described as “Yes, entailment or scandal”). However, for both approaches, we observe negative results, since some answer choice configurations are not informative for every dataset but instead can be considered misleading at times (e.g. the topic-specific answer choice “scandal” for RTE). Pre-filtering misleading answer choices to combine exclusively informative choices might lead to more positive results, however,  more experiments are difficult to conduct given the time constraint of the rebuttal but we will report additional findings in the paper.\n\n**Question B: Topic-specific answer choices with descriptions spanning multiple words**\n\nWe ran preliminary experiments concatenating the top-k ranked words for a single class together (for $k \\in \\\\{1, 2, 3\\\\}$), with $k=1$ producing the best scores. More generally, we observed a tendency for the LMs to prefer shorter answer choices across answer choice configurations. We will conduct a more detailed analysis of the quantitative characteristics of the selected answer choices and we will report our additional findings in the additional space if accepted.\n\n\n**Question C: Explanation and breakdown of increased computation**\n\nWe acknowledge that our analysis of the computational costs would benefit from a breakdown into how individual components impact both runtime and overall performance. We will address this in the camera-ready version of the paper if accepted. \n\nRuntime during inference: AuT-Few’s Monte-Carlo approximation over all retrieved templates is the sole cause for the computational increase, increasing compute by a factor of 5 (since 5 templates are retrieved). Running AuT-Few without Monte-Carlo approximation (i.e. random sampling of templates) makes AuT-Few *as efficient* as T-Few during inference while still outperforming it, see Table 4 in our ablation study. Moreover, AuT-Few with the smaller BART0 backbone requires only 60% of SetFit’s computation during inference without Monte-Carlo approximation, yet scores higher. As also noted to Reviewer 1, we acknowledge that the use of Monte-Carlo approximation during inference distracts from the overall efficiency of our approach and we will ensure to clarify this point by moving the scores with Monte-Carlo approximation to the appendix if accepted, making the efficiency analysis easier to follow.\n\nCosts for training: Training is almost exclusively impacted by running 3-fold cross-validation for each answer choice configuration to select the most appropriate one (the additional parameters added by LoRA are negligible in comparison). As seen in the ablation study (Table 4), the selection mechanism results in an average improvement of 0.8 points over the next base configuration (dataset label text) while also reducing variance. When not assuming the availability of an intelligible dataset label text (frequently the case with Hugginface datasets), the average improvement is 2.1 points (AuT-Few scores 65.4 versus 63.3 with template-tailored choices). Since training times in a few-shot scenario are generally very short (magnitudes shorter than for smaller models like BERT in a fully supervised setting), we do not consider these additions to the runtime problematic.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The article empirically investigates the robustness of two types of prompts, task template and answer choices, on multiple classification task datasets, and proves that template prompt is more robust than answer choices prompt, providing strong empirical and hence a simple retrieve method can be used for template prompts. Answer choices are more sensitive to prompts, so the authors propose two methods, template-tailored and topic-specific answer choices, based on retrieve. answer choices, and proved the rationality of these two methods in ablation experiments. \n\nThe authors also empirically verified the feasibility of fusing the two methods of IA3 and LoRA, which provides ideas for the fusion of current main different methods in PEFT. In addition, the authors conduct extensive experimental comparisons with the pre-SOTA T-Few on mainstream models and classification task datasets, and reach SOTA in RAFT benchmark to validate the effectiveness of the proposed methods.\n", "reasons_to_reject": "The paper’s main improvement which focuses on prompt setting of task template and answer choices in discrete space relies on the architecture of the pre-SOTA T-few, with fewer innovations in the PEFT modeling framework.\n\nIn addition, the reasons for the training time and increased computation are not sufficiently explained when comparing with the former SOTA. In terms of template prompt retrieval, there is a lack of analysis on the impact of the overall quality of collection prompts on template prompts, and subsequent researchers may have difficulties in finding suitable metrics to measure the quality of the prompt collection used to retrieve templates in the face of an ever-expanding prompt collection. ", "questions_to_authors": "Question A: Despite the conclusion that “neither template tailored answer choices nor multiple distinct answer choice configurations is needed in the empirical experiments” drawn in Chapter 3, would it be possible to combine the three answer choices, Dataset, template-tailored, and Topic-Specific, in a discrete space rather than choosing the one with the highest score? \n\nQuestion B: Why does Topic-Specific answer choice only consider single-word to semantically express the underlying content for each answer class, and was there any experiment on word number ablation?\n\nQuestion C: When comparing the FLOPS of training and reasoning with other models, could you explain in detail which improvements in your setting lead to the increase of FLOPS, and what are the marginal benefits they bring to the increase of FLOPS? Could you compare the training and inference time cost with other benchmark models in more detail based on the marginal benefits?\n", "paper_topic_and_main_contributions": "This paper aims to address the problem of hand-crafted prompt design on few-shot learning. The proposed AuT-few model can automatically generates appropriate prompt in discrete output space for downstream tasks. Extensive empirical studies on 12 datasets are conducted and results shows that the sensitivity of prompt design is different for template and answer choices, arguing the importance of rationally designing prompts in answer choices. Based on the experimental results, the article proposes a Retrieval approach for template design and two approaches, template-tailored and topic-specific, for answer choices. Extensive experiments demonstrate that AuT-Few outperforms the previous SOTA T-Few by 2 points on 8 tasks on 12 datasets, and has good robustness to different models and out-of-domain datasets. The training and inference overheads are also calculated, although it is inferior to SOTA or equivalent models in terms of training cost or inference speed, it is still in the comparable range, requiring only a few hours of training on an A10G single GPU, hence providing new automatic methodological prompt ideas to the few-shots community and easy to fuse on existing architectures.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "No", "Typos_Grammar_Style_and_Presentation_Improvements": "No"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper deals with automated prompt construction for few-shot (parameter-efficient fine-tuning, not in-context) learning for instruction-tuned LLMs. The authors first demonstrate that instruction-tuned LLMs are pretty robust to the selection of instruction prompts for classification task, which then informs an automated approach to prompt selection, based on prompt retrieval (over a catalog of instructions on which the backbone model was initially (pre)trained) and strategy for generating candidates for answer choices (i.e., class lexicalizations), based on template-conditioned probabilities of tokens for a class (on the training data). Evaluation of Aut-Few on a dozen short-text classification datasets renders it effective (more on reviewer concerns raised on Aut-Few performance below). 

There is some logical discrepancy between the initial study, which shows that instruction-tuned LLMs are robust to the choice of the prompt, and the rest of the paper that aims to automate prompt (task-description) selection/creation. The authors explain (in the rebuttal) that, despite the fact that instruction-tuned LLMs exhibit more robustness w.r.t. choice of task description than vanilla LLMs (i.e., not instruction-tuned), there is still variation, especially w.r.t. lexicalizations of classes. The finding of robustness of instruction-based LLMs is indeed somewhat at odds with the motivation for proposing Aut-Few, and the authors should address this in the structure of the paper.       

The main concern raised in the reviews is that of performance of Aut-Few in comparison to T-Few, which manually selects the prompts. The concern raised was that Aut-Few outperforms T-Few only on a minority of the evaluation datasets. The authors clarify the significance of their results in the rebuttal: on 3 datasets, Aut-Few is significantly better than T-Few, whereas on 1 dataset the opposite is true; on the remaining datasets the difference in performance between the two is not statistically significant. The crucial point here, that the reviewer may have missed, however, is that T-Few implies careful manual selection of the prompt, whereas Aut-Few removes the need for any manual effort.

Although one could argue that neither the retrieval step nor class label generation component are greatly innovative -- i.e., they can be seen as relatively straightforward extensions of existing body of work on prompting LLMs -- they are meaningful and convincing. Moreover, the practical benefit of this work -- removal of the need for manual effort in prompt design for instruction-tuned LLMs -- is substantial.
Decision: Accept-Findings