Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning
Authors: Haokun Liu
Derek Tam
Muqeeth Mohammed
Jay Mohta
Tenghao Huang
Mohit Bansal
Colin Raffel
Venue: NeurIPS 2022 Accept (2022)
Keywords: few-shot learning
in-context learning
parameter-efficient training

Abstract: Few-shot in-context learning (ICL) enables pre-trained language models to perform a previously-unseen task without any gradient-based training by feeding a small number of training examples as part of the input. ICL incurs substantial computational, memory, and storage costs because it involves processing all of the training examples every time a prediction is made. Parameter-efficient fine-tuning (PEFT) (e.g. adapter modules, prompt tuning, sparse update methods, etc.) offers an alternative paradigm where a small set of parameters are trained to enable a model to perform the new task. In this paper, we rigorously compare few-shot ICL and PEFT and demonstrate that the latter offers better accuracy as well as dramatically lower computational costs. Along the way, we introduce a new PEFT method called (IA)^3 that scales activations by learned vectors, attaining stronger performance while only introducing a relatively tiny amount of new parameters. We also propose a simple recipe based on the T0 model called T-Few that can be applied to new tasks without task-specific tuning or modifications. We validate the effectiveness of T-Few on completely unseen tasks by applying it to the RAFT benchmark, attaining super-human performance for the first time and outperforming the state-of-the-art by 6% absolute. All of the code used in our experiments will be publicly available.

Few-Shot Parameter-Efficient Fine- Tuning is Better and Cheaper than In-Context Learning
Haokun Liu"
Derek Tam"
Mohammed Mugeeth
Jay Mohta
Tenghao Huang
Mohit Bansal
Colin Raffel
Department of Computer Science
University of North Carolina at Chapel Hill
(hackunl, dtredsox, muqeeth, craffel)@cs. unc.edu
Abstract
Few-shot in-context learning (ICL) enables pre-trained language models to per- form a previously-unseen task without any gradient-based training by feeding a small number of training examples as part of the input. ICL incurs substantial computational, memory, and storage costs because it involves processing all of the training examples every time prediction is made. Parameter-efficien fine-tuning (PEFT) (e.g. adapter modules, prompt tuning. sparse update methods, etc.) offers alternative paradigm where a small set of parameters are trained to enable a model perform the new task. this paper, we rigorously compare few-shot ICI. and PEFT and demonstrate that the latter offers better accuracy as well as dramatically lower computational costs. Along the way, we introduce a new PEFT method called (IA) that scales activations by learned vectors, attaining stronger performance while only introducing a relatively tiny amount of new parameters. We also propose simple recipe based the TO model [1] called T-Few that can be applied to new tasks without specific tuning or modifications. We validate the effectiveness of T-Few on completely unseen tasks by applying the RAFT benchmark [2], attaining super-human performance for the first time and outperforming the state-of-the-arr by 6% absolute. All of the code used in experiments publicly available.'
Introduction
Pre-trained language models have become cornerstone of natural language processing, thanks to the fact that they can dramatically improve data efficiency on tasks of interest - i.e., using pre-trained language model for initialization often produces better results with less labeled data. A historically common approach has been gradient-baae bbased fine-tuning on downstream task of interest with pre-trained parameters the initialization. While fine-tuning has produced many state-of-the-an results [1], it results in a model specialized for a single task with an entirely new set of parameter values, which can become impractical when fine-tuning many downstream tasks.
alternative approach popularized by [3, 4] in-context learning (ICL), which induces model to perform downstream task by inputting prompted examples. Few-shot prompting converts small collection input-target pairs into (typically) uumnan-understaddable instructions and examples [3, 4], along with single unlabeled example for which prediction is desired. Notably, ICL requires no gradient-basee based training and therefore allows single model to immediately perform wide variety of tasks. Performing ICL therefore solely relies the capabilities that a model learned during pre-training These characteristics have led to a great deal of recent interest in ICL methods [5-10].
'Equal contribution. http://///taaabbccoa/tteeeeeeeee
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

(IA)3
Losses used in T-Few
Dense
LIM
loves grandaa' s bread, Surse called her grandra and asked her to some. Grandea lived very far anay. week passed grandea surprised by contig to visit. What possible continuation for the story?
softmax
1
TØ
Susso was happy
LIN
k
Noniinnarity
Suske was upset.
LUL
Dense
Figure I: Diagram of (IA)3 and the loss terms used in the T-Few recipe. Left: (IA)* introduces the learned vectors lis lv, and ler which respectively rescale (via element-wise multiplication, visualized as @) the keys and values in attention mechanisms and the inner activations in position- -iiie feed-forwe worwadd networks. Right: In addition to a standard cross-entropy loss LLM. we introduce an unlikelihood loss LuL that lowers the probability of incorrect outputs and length- nortth-nrmmllieed loss LL that applies standard softmax cross-entropy loss to length-none normalized pogprobbabslities of all output choices.
Despite the practical benefits of ICL, has several major drawbacks. First, processing all prompted input-target pairs every time the model makes prediction incurs significant compute costs. Second, ICL typically produces inferior performance compared to fine-tuning [4]. Finally, the exact formatting of the prompt (including the wording [11] and ordering of examples [12]) can have significant and unpredictable impact on the model's performance, far beyond inter-run variation of fine-tuning. Recent work has also demonstrated that ICL can perform well even when provided with incorrect labels, raising questions as to how much learning is taking place at all [9].
An additional paradigm for enabling model to perform new task with minimal updates is parameter- efficient fine-tuning (PEFT), where a pre-trained model fine-tuned by only updating a small number of added or selected parameters. Recent methods have matched the performance of fine-tuning the full model while only updating or adding a small fraction (e.g. 0.01% %) of the full model's ss parameters [13, 14). Furthermore, certain PEFT methods allow mixed-task batches where different examples in a batch are processed differently [14], making both PEFT and ICL viable for multitask models.
While the benefits of PEFT address some shortcomings of fine-tuning (when compared to ICL), there has been relatively little focus on whether PEFT methods work well when very little labeled data is available. Our primary goal in this paper is to close this gap by proposing a recipe- - i.e., a model, a PEFT method, and a fixed set of hyperparameters that attains strong performance on novel, unseen tasks while only updating a tiny fraction of the model's parameters. Specifically, we base our approach on the TO model [1]. variant of T5 [15] fine-tuned on a multitask mixture of prompted datasets. To improve performance on classification and multiple-choice tasks, we add unlikelihood [16, 17] and length normalization-haal -base [4] loss terms. In addition, we develop (IA)3, a PEFT method that multiplies intermediate activations by learned vectors. (IA) attains stronger performance than full-model fine-tuning while updating up to 10,000x fewer parameters. Finally, we demonstrate the benefits of pre-training the (IA) parameters before fine-tuning [18, 19]. Our overall recipe, which we dub "T-Few", performs significantly better than ICL (even against 16x larger models) and outperforms humans for the first time on the real-world few-shot learning benchmark RAFT [2] while requiring dramatically less compute and allowing for mixed-task batches during inference. To facilitate the use of T-Few new problems and future research on PEFT, we release our code.'
After providing background on ICL and PEFT in the following section, we discuss the design of T-Few in section 3. In section 4, we present experiments comparing T-Few to strong ICL baselines. Finally, we discuss related work in section and conclude in section
2 Background
In this section, we provide am verview of ICL and PEFT with a focus on characterizing the com- putation, memory, and on-disk storage costs of making prediction. Real-world costs depend on implementation and hardware, so we report costs in terms of FLOPSs for computation and bytes for memory and storage, respectively. Additional related work is discussed in section 5.

---

2.1 Few-shot in-context learning (ICL)
ICL [3, 4] aims to induce a model to perform a task by feeding in concatenated and prompted input-target examples (called "shots") along with unlabeled query example. Taking the cycled letter task from Brown et al. [4] as an example, a 4-shot input or context would be "Please unscramble the letters into word, and write that word: asinoc = casino, yfrogg = froggy, plesim = simple, iggestb biggest, astedro =", for which the desired output would be "roasted". ICI. induces an autoregressive language model perform this task by feeding in the context and sampling from the model. For classification tasks, each label is associated with string (e.g. "positive" and "negative" for sentiment analysis) and a label is assigned by choosing the label string that the model assigns the highest probability to. For multiple-choice tasks (e.g. choosing between N possible answers question), the model's prediction is similarly determined by determining which choice is assigned the highest probability.
The primary advantage of ICL is that it enables single model to perform many tasks immediately without fine-tuning. This also enables mixed task batches, where different examples in a batch of data correspond to different tasks by using different contexts in the input. ICL is also typically performed with only a limited number of labeled examples - called few-shot learning - making it data-efficient.
Despite these advantages, ICI. comes with significant practical drawbacks: First, making a prediction is dramatically more expensive because the model needs to process all of the in-context labeled examples. Specifically, ignoring the quadratic complexity of self-attention operations in Transformer language models (which are typically small compared to the costs of the rest of the model [20], processing the l training examples for k-shot ICL increases the computational cost by approximately k ++1 1 times compared to processing the unlabeled example alone. Memory costs similarly scale approximately linearly with k though during inference the memory costs typically dominated by storing the model's parameters. Separately, there is a small amount of on-disk storage required for storing the in-context examples for a given task. For example, storing 32 examples for a task where the prompted input and target for each example is 512 tokens long would require about 66 kilobytes of storage disk (32 examples 512 tokens x 32 bits).
Beyond the aforementioned costs, ICL also exhibits unintuitive behavior. Zhao et al. [12] showed that the ordering of examples in the context heavily influences the model's predictions. Min et al. [9] showed that ICL can still perform well even if the in-context example labels are swapped (i.e. made incorrect), which raises questions about whether ICL is really "learning" from the labeled examples.
Various approaches have been proposed to mitigate these issues. One way to decrease computational costs is to cache the key and value vectors for in-context examples. This is possible because decoder- only Transformer language models have causal masking pattern, so the model's is activations for the context do not do not depend on the unlabeled example. In an extreme case, 32-shot ICL. with 512 tokens per in-context example would result in over 144 gigabytes of cached key and value vectors for the GPT-3 model (32 examples x 512 tokens x 96 layers 12288 dlmodel x 32 bits each for the key and value vectors). Separately, Min et al. [21] proposed ensemble ICL, where instead of using the output probability from concatenating the A training examples, the output probabilities of the model on each training example (i.e. 1-shot ICL for each of the examples) are multiplied together. This lowers the non-parameter memory cost by factor of k/2 but increases the computational cost by a factor of 2. In terms of task performance, Min et al. [21] find that ensemble ICL outperforms the standard concatenative variant.
2.2 Parameter-efficient fine-tuning
While standard fine-tuning updates all parameters of the pre-trained model, it has been demonstrated that is possible instead update or add relatively small number of parameters. Early methods proposed adding adapters [22-24], which are small trainable feed- forward networks inserted between the layers in the fixed pre-trained model. Since then, various sophisticated PEFT methods have been proposed, including methods that choose sparse subset of parameters to train [2.5, 26], produce low-rank updates [13], perform optimization in a lower-dimensional subspace [27], add low-rank adapters using hypercomplex multiplication [28], and more. Relatedly, prompl funing [14] and prefix tuning [29] concatenate learned continuous embeddings to the model's 's input or activations to induce it to perform a task; this can be seen as PEFT method [30]. oftte-of-the-an PEFT methods can
3

---

match the performance of fine-tuning all of the model's 's parameters while updating only tiny fraction (e.g. 0.01%) of the model's parameters.
PEFT drastically reduces the memory and storage requirements for training and saving the model. In addition, certain PEFT methods straightforwardly allow mixed-task batches - for example, prompt tuning enables single model to perform many tasks simply by concatenating different prompt embeddings each example in the batch [14]. On the other hand, PEFT methods that re-parameterize the model (e.g. [27, 13I) are costly or onerous for mixed-task batches. Separately, different PEFT methods increase the computation and memory required to perform inference by different amounts. For example, adapters effectively add additional (small) layers to the model, resulting in small but non-negliiible increases in computational costs and memory. An additional cost incurred by PEFT is the cost of fine-tuning itself, which must be performed once and is then amortized as the model used for inference. However, we will show that PEFT can be dramatically more computationally efficient when considering both fine-tuning and inference while achieving better accuracy than ICI.
3 Designing the T-Few Recipe
Given that PEFT allows a model to be adapted to a new task with relatively small storage requirements and computational cost, we argue that PEFT presents a promising alternative to ICL, Our goal is therefore to develop a recipe that allows model to attain high accuracy new tasks with limited labeled examples while allowing mixed-task batches during inference and incurring minimal computational and storage costs. By recipe, we mean specific model and hyperparameter setting that provides strong performance on any new task without manual tuning or per-task adjustments. In this way, we can ensure that our approach is realistic option in few-shot settings where limited labeled data is available for evaluation [31, 32].
3.1 Model and Datasets
As first step, we must choose pre-trained model. Ideally, the model should attain high performance on new tasks after fine-tuning on a limited number of labeled examples. In preliminary experiments applying PEFT methods to different pre-trained models, we attained the best performance with TO [1]. TO is based on T5 [15], an encoder-decoder Transformer model [33] that was pre-trained via a masked language modeling objective [34] on a large corpus of unlabeled text data. TO was created by fine-tuning T5 on multitask mixture of datasets in order to enable zero-shot generalization, i.e. the ability to perform tasks without any additional gradient- based training. Examples in the datasets used to train TO were prompted by applying the prompt templates from the Public Pool of Prompts (P3 [35), which convert each example in each dataset to a prompted text-to-text format where each label corresponds to different string. For brevity, we omit detailed description of TO and T5; interested readers can refer to Sanh et al. [1] and Raffel et al. [15]. TO was released in three billion and eleven billion parameter variants, referred to "TO-3B" and simply "TO" respectively. In this section (where our goal is to design the T-Few recipe through extensive experimentation). we use TO-3B to reduce computational costs. For all models and experiments, we use Hugging Face Transformers [36].
While TO was designed for zero-shot generalization, we will demonstrate that it also attains strong performance after fine-tuning with only few labeled examples. To test TO's generalization, Sanh et al. [1] chose a set of tasks (and corresponding datasets) to hold out from the multitask training mixture specifically, sentence completion (COPA [37], H-SWAG [38], and Story Cloze [39] datasets), natural language inference (ANLI [40], CB [41], and RTE [42]), coreference resolution (WSC [43] and Winogrande [44]), and word sense disambiguation (WiC [45]). Evaluation of generalization capabilities can then be straightforwardly done by measuring performance on these held-out datasets. We also will later test T-Few's ' abilities in the RAFT benchmark [2] in section 4.3, collection of unseen "real-worke is licensed under Creative Commons License. Winogrande is licensed under an Apache license. COPA is under a BSD-2 Clause license. We could not find the license of RTE and CB but they are part of SuperGLUE which mentions the datasets are allowed for use in research context.
few-shot tasks with no validation set and a held-out test set. ANLI, WIC WSC
To ease comparison, we use the same number of few-shot training examples for each dataset as Brown et al. [4], which varies from 20 to 70. Unfortunately, the few-shot dataset subsets used by Brown et al. [4] have not been publicly disclosed. To allow for a more robust comparison, we therefore constructed five sho-shot datasets by sampling subsets with different seeds and report the median and

---

interquartile range. We prompt examples using uundoll--sampled prompt template from P3 Bach al. [35] for each example at each step. Unless otherwise stated, we train our model for 1K steps with batch size of 8 and report performance at the end of training.
For evaluation, we use "rank classification", where the model's log-probabilitis for all possible label strings are ranked and the model's prediction is considered correct if the highest-ranked choice is the correct answer. Rank classification evaluation is compatible with both classification and multiple- choice tasks, Since model performance can vary significantly depending on the prompt template used, we report the median accuracy across all prompt templates from P3 and across few-shot data subsets for each dataset. For all datasets, we report the accuracy on the test set or validation set when the test labels are not public (e.g. SuperGLUE datasets). In the main text, we report median accuracy across the nine datasets mentioned above. Detailed results on each dataset are provided in the appendices.
3.2 Unlikelihood Training and Length Normalization
Before investigating PEFT methods, we first explore two additional loss terms to improve the performance of few-shot fine-tuning of language models. Language models are normally trained with cross-entropy loss LIM =
+ = E. logr(y,|x, Vct)) where the model is trained to increase the probability of the correct target sequence y (V1, .2....., yr) given the input sequence x.
For evaluation, we use rank classification (described in section 3.1) which depends on both the probabilities that the model assigns to the correct choice as well as the incorrect choices. To account for this during training, we add an unlikelihood loss [16, 17]:
T()) Ei=1 log(1 p(U) X, (2))
LUL
ΣN T(m)
which discourages the model from predicting tokens from incorrect target sequences, where f(() (01. b2,..., (r is the n-th of N incorrect target sequences. We hypothesize that adding LUL will improve results rank classification because the model will be trained to assign lower probabilities to incorrect choices, thereby improving the chance that the correct choice is ranked highest.
The possible target sequences for given training example can have significantly different lengths, especially in multiple-choice tasks. Ranking each choice based on probability can therefore "favor" shorter choices because the model's assigned probability to each token is S 1. To rectify this, we consider using length normalization when performing rank classification, which divides the model's score each possible answer choice by the number of tokens in the choice (as used in GPT-3 [4]). When using length normalization during evaluation, we introduce an additional loss term during training that more closely reflects mangth-normalizer evaluation. First, compute the Length-normaliin log probability of a given output sequence 3(x. y) = + Σ | log p(y |x, yct)). Then, we maximize the norgth-eo log probability of the correct answer choice by minimizing the softmax cross-entropy loss:
exp(((xx,))
L log exp(3(xxy)) + En exp($(xx,5()))
When training a model with LLM- LUL, and LLN, we simply sum them. This avoids introducing any hyperparameters that would be problematic to tune in the few-shot setting (where realistically-sized validation sets are tiny by necessity [31, 32)).
We report the results of fine-tuning all of TO-3B's parameters with and without length normalization on all datasets in appendix B. We find that adding LLN improves the accuracy from 60.7% to 62.7 % and including both LUL and LLN provides a further improvement 63.3%. Since these loss terms improve performance without introducing any additional hyperparameters, we include them in our recipe and use them in all following experiments.
3.3 Parameter-efficient fine-tuning with (IA)3
In order to compare favorably to few-shot ICL, we need a PEFT method that has the following properties: First, it must add or update as few parameters as possible to avoid incurring storage and memory costs. Second, it should achieve strong accuracy after shot training on new tasks.
5

---

Finally, it must allow for mixed-tass batches, since that is a capability of ICL. In order to easily enable mixed-task batches, a PEFT method should ideally not modify the model itself. Otherwise, each example in batch would effectively need to be processed by different model or computational graph. A more convenient alternative is provided by methods that directly modify the activations of the model since this can be done independently and cheaply to each example in the batch according to which task the example corresponds to. Prompt tuning and prefix tuning methods [14, 29] work by concatenating learned vectors to activation embedding sequences and are therefore examples of activation- activation-modifying PEFT methods that allow for mixed-task batches. However, as we will discuss later, we were unable to attain reasonable accuracy with prompt tuning and found that the more performant PEFT methods did not allow for mixed-task batches. We therefore developed a new PEFT method that meets our desiderata.
As an alternative, we explored element- wise multiplication (i.e. rescaling) of the model's activations against learned vector. Specifically, we consider adaptation of the form / O a where E Rd is learned task-specific vector, O represents element- multiplication, and x E R is length-T sequence of activations. We use "broadcasting notation" [46] so that the j) entry of lf is (ن In preliminary experiments, we found it was not necessary to introduce a learned rescaling vector for each set of activations in the Transformer model. Instead, we found it was sufficient to introduce rescaling vectors on the keys and values in self-attention and and on the intermediate activation of the position- wise feed-forward networks. Specifically, using the notation from Vaswani et al. [33], we introduce three learned vectors L E R ly E R²,, and 4 E Rda, which are introduced into the attention mechanisms
attention mechanisms
softmax (Qlh O K") ViTe
(
(lu + VV)
and in the position-wis feed- forward networks as (lu O ((W2z)) where yy is the ford-forward network nonlinearity. We introduce separate set of 1x lv. and 4t vectors each Transformer layer block. This adds total of L(ds- d,, + d#) new parameters for L-layer-block Transformer encoder and L(2d; + 2d,,- + de) (with factors of accounting for the presence of both self-attention and encoder-decoder attention) for a L-layer-block decoder. 4 lv. and ly are all initialized with ones so that the overall function computed by the model does not change when they are added. We call our method (IA)3, which stands for "Infused Adapter by Inhibiting and Amplifying Inner Activations".
(IA)* makes mixed-task batches possible because each sequence of activations in the batch can be separately and cheaply multiplied by its associated learned task vector. We also note that, in the event that a model will only be used on single task, the modifications introduced by (IA) can also be applied to weight matrices permanently so that no elementwise multiplication is required and the model's 's architecture remains unchanged. This possible because element-wise multiplications performed in (IA) always co-occur with matrix multiplication, and @ Wx = (1 @ W)a. In this case, our method incurs no additional computational cost compared to the original model.
To validate (I we compare it to large variety of existing adaptation methods in our setting of fine-tuning TO-3B on few-shot datasets from held-out tasks. Specifically, we compare with 9 strong PEFT methods: BitFit [47] which updates only the bias parameters; Adapters [23] which introduce task-specific layers after the self-attention and position- wise feed-forward networks; Compacter and Compacter++ [28] which improve upon adapters by using low-rank matrices and hypercomplex mul- tiplication; prompt tuning [14] which learns task specifce prompt embeddings that are concatenated to the model's input; FISH Mask [26] which chooses subset of parameters to update based on their ap- proximate Fisher information; Intrinsic SAID [27] which performs optimization in a low-dimensional subspace; prefix-tuning [29] which learns task-specific vectors that are concatenated to the model's 's activations; and LoRA [13] which assigns low-rank updates to parameter matrices. Additionally, we include the baselines of full-model fine-tuning and updating only the layer normalization parameters. For certain methods that allow changing the parameter efficiency, we report results for different budgets: 0.2% and 0.02% sparsity for FISH Mask, 10 and 100 learned prompt vectors for prompt tuning, and 20,000- or
subspaces for Intrinsic SAID.
The results are shown in fig. 2, with detailed per-dataset results in appendix C. We find that (IA)3 the only method that attains higher accuracy than the full-model-fine-tnniny baseline. While other PEFT methods (e.g. Intrinsic SAID and prompt tuning) update or introduce fewer parameters, (IA)3 performs considerably better. Our results and setting differ with some past work on the PEFT methods we compare against. Mahabadi et al. [28] report that Compacter and Compacter++

---

outperform full- full-model fine-tuning, including in the few-shot setting. Lester et al. [14] found that prompt tuning could match full-model fine-tuning, and in subsequent work Wei al. [48] found that prompt tuning performed well when applied to multitask fine-tuned model in the few-shot setting. In both cases, we experimented with various hyperparameter choices to try to match past results. We hypothesize the disagreement comes from us using different model and different datasets. For prompt tuning specifically, we noticed that the validation set performance could fluctuate wildly over the course of training, hinting at possible optimization issues.
65
All parameters
70
60
Accuracy
Accuracy
65
80
50
55
50
0.001%
0.01%
0.1%
1012
1013
10
10
% of parameters updated
FLOPs per example
(IA)a
Prompt Tuning
T-Few
GPT-3 6.7B
LoRA
Prefix Tuning
TO
GPT-3 13B
BitFit
Adapter
LM+
GPT-3 175B
Layer Norm
FISH Mask
Compacter
Intrinsic SAID
Figure 3: Accuracy of different few-shot learning methods. T-Few uses (IA)3 for PEFT methods of TO, TO uses zero-shot learning, and T5+LM and the GPT-3 variants use few-shot ICL. The x-axis corresponds to inference costs; details provided in section 4.2.
Compacter++
Figure 2: Accuracy of PEFT methods with LUL and LLN when applied to TO-3B. Methods that with variable parameter budgets are represented with larger and smaller markers for more or less
parameters.
3.4 Pre-training 3IA)3
In recent work, Gu et [18], Vu et al. [19] showed that pre-training the prompt embeddings in prompt tuning can improve performance when fine-tuning on downstream few-shot tasks. For pre- training, Gu et al. [18] use a suite of self-supervised tasks applied to unlabeled text data, and Vu et al. [19] consider using embeddings from separate task multitask mixture. We follow Vu et al. [19] and simply pre-train the new parameters introduced by (IA)² on the same multitask mixture used to train TO. We pre-train for 100,000 steps with batch size of 16 before fine-tuning the (IA) parameters on each individual downstream dataset. A full comparison of accuracy with and without pre-training (IA) is detailed in appendix D. We find that pre-training improves fine-tuned accuracy from 64.6 to 65.8 and therefore add it to our recipe.
3.5 Combining the ingredients
In summary, the T-Few recipe is defined follows: We use the TO model backbone. We add (IA)* for downstream task adaptation and use parameters initialized from pre-training (IA) on the same multitask mixture for TO. As an objective, we use the sum of a standard language modeling loss LLM. an unlikelihood loss LUI for incorrect choices, and dength-normalized loss LLN- We train for 1,000 steps with a batch size of sequences using the Adafactor optimizer [49] with learning rate of 3c and a linear decay schedule with a 60-step step warmup. We apply prompt templates to downstream datasets during training and inference to convert each example into an instructive text-to-text format. Importantly, apply this recipe lo every downstream dataset in exuctly the same way without per-dataset hyperparameter tuning or modifications. This makes the recipe realistic option for sho- learning settings where validation sets are tiny by definition [31, 32].
7

---

4 Outperforming ICL with T-Few
Having designed and established the T-Few recipe on TO-3B, we now apply it to TO (with 11 billion parameters) and compare performance to strong few-shot ICL baselines. From this point onwards, we use exactly the same recipe and hyperparameters across all tasks.
4.1 Performance on TO tasks
First, we evaluate T-Few on the datasets that were held out from TO's training mixture. We compare against zero-shot learning with TO [1] (since we found few-shot ICL to performed worse than zero- shot for TO, see appendix E); few-shot ICL with T5+LM [14] (the
language model upon which TO is based); and few-shot ICL with the 6.7, 13, and 175 billion parameter variants of GPT-3. See appendix E for more details on these baselines. The accuracy on the held-out TO datasets (described in section 3.1) is shown in table 1 and fig. 3, with per-dataset results reported in appendix E. We find that T-Few outperforms all other methods by substantial margin. Notably, T-Few achieves a 6% higher accuracy than few-shot ICL with GPT-3 175B despite being about 16x smaller and outperforms the smaller GPT-3 variants by an even larger margin. T-Few also attains significantly higher accuracy than both zero-shot learning with TO and few-shot ICL with T5+LM.
Inference FLOPs
Training FLOPSs
Disk
Method
Acc.
Method
space
Acc.
T-Few
75.8%
T-Few
1.1e12
2.7e16
4.2 MB
72.4%
Human baseline [2]
73.5%
TO [1]
1.le12
0
O B
66.9%
PET [50]
69.6%
T5+LM [14]
4.5e13
0
16 kB
49.6%
SetFit [51]
66.9%
GPT-3 6.7B [4]
5.4e13
0
16 kB
57.2%
GPT-3 [4]
62.7%
GPT-3 13B [4]
1.0e14
0
16
60.3%
GPT-3 175B [4]
1.4e15
0
16 kB
66.6%
Table 2: Top-5 best methods on RAFT as of writing. T-Few is the first method to outperform the human baseline and achieves over 6% higher accuracy than the next- best method.
Table I: Accuracy on held-out TO tasks and computational costs for different few-shot learning methods and models. T-Few attains the highest accuracy with 1,000x lower computational cost than ICL with GPT-3 175B. Fine-tuning with T-Few costs about as much as ICL on 20 examples with GPT-3 175B.
4.2 Comparing computational costs
Having established that T-Few significantly outperforms ICL-based models, now compare the relative costs of each few-shot learning approach. For simplicity, we use the FLOPs-per-toker estimates for Transformer-based language models introduced by Kaplan et al. [20]. Specifically, we estimate that decoder-only Transformer (e.g. the GPT series) with N parameters uses 2N FLOPs per token for inference and 6N FLOPs per token for training. Fnnooderddecoder models like TO and T5 (where the encoder and decoder have the same number of layers and layer sizes) only process each token with either the encoder or decoder (each having roughly half the parameters of the full model), so the FLOPs per token estimates are halved to N and 3N FLOPs per token for inference and training. We note that FLOPs are not a direct measurement of real-world computational cost because latency, power usage, and other costs can vary significantly depending on hardware and other factors [52]. However, we focus on FLOPs because it is a hardwarr-indeepene real-world costs the hardware setup used for running the different methods we consider would likely vary significantly across methods. We summarize the costs in table and discuss them below. For all estimates, we use the median number of shots (41) across the datasets we consider. Rank evaluation and our unlikelihood loss both require processing every possible output choice to attain a prediction for unlabeled example. The median combined tokenized sequence length for the input and all possible targets is 103 for the datasets we consider. For in-context examples processed for few-shot ICL, only the correct target is required, producing median sequence length of 98. Assuming that key and value vectors are cached, processing a single example with ICL therefore involves processing 41 98 103 tokens. A summary of our cost estimates is provided in table 1.
metric that closely with
Inference cost. Beyond improved accuracy, the primary advantage of avoiding few-shot ICL is dramatically lower inference costs. Processing a single input and all target choices with T-Few
8

---

requires 11c9 x 103 = 1.11122FOP whereas few-shot ICL with GPT-3 175B requires 2 x 175e9 X (41 98 + 103) 1.4c15 FLOPs more than 3 orders of magnitude more. Inference costs with ICL using the smaller GPT-3 variants are also dramatically higher than the inference cost of T-Few. As discussed in section 2.1, caching the key and value vectors of the in-context examples can reduce the computational cost of ICI.. However, this would only result in an approximately 41 x reduction, which is not nearly enough to make any of the GPT-3 ICL costs as low as T-Few.
Training cost. Since T-Few is the only method that involves updating parameters, it is the only method that incurs a training cost. Training an eleven billion parameter encoder-decoder model for 1,000 steps with batch size of 8 length-103 sequences requires approximately x 1 11e9 x 000 x 8 x 103 = 2.7e16 FLOPs. While not insignificant, this is only about 20 times larger than the FLOPs required to process single example with few-shot ICL using GPT-3 175B. In other words, training T-Few costs as much as using GPT-3 175B to process 20 examples with few-shot ICL. We also found that fine-tuning TO with T-Few on single dataset only takes about half hour on a single NVIDIA A100 GPU. As of writing, this would cost about S2 USD using Microsoft Azure. ?
Storage cost. T-Few also incurs the largest storage cost. When stored as single-precision floats, the parameters added by (IA) take up 4.2 MB of space disk. In contrast, ICL methods only require storing the tokenized in-context examples (typically stored as 32-bit integers), resulting smaller 41 98 32 bits 16 kB disk space requirement. However, on-disk size of the model checkpoints themselves - storing the (IA) adaptation vectors for 10,000 tasks would take about as much space the TO checkpoint (41.5 GB).
note that 4.2 MB is dwarfed by the
Memory usage. During inference, the primary memory cost is incurred by the model's parameters. The only model smaller than TO (used by T-Few) GPT-3 6.7B; otherwise, T-Few will incur lower memory cost during inference. Additional memory costs are incurred when training T-Few due to the need cache intermediate activations for backpropagation and for the gradient accumulator variables in Adafactor. However, as mentioned above, a single 80GB A100 GPU is enough for T-Few.
4.3 Performance on Real-world Few-shot Tasks (RAFT)
So far, we have run evaluatation on a collection of datasets not explicitly designed for benchmarking few-shot learning. To better evaluate T-Few in the real world, we took our approach to the RAFT benchmark [2]. RAFT comprise "economically valuable" tasks that mirror real-world applications. Importantly, each RAFT datasets has only 50 training examples with no validation set and (larger) test set with no public labels, so is impossible to "cheat" by tuning on an murealistically-arg validation set by peeking at the test set [32, 31]. We apply T-Few to RAFT by using the standard prompts released alongside the dataset. The accuracy of the current top-5 methods is shown in table 2, with further details in appendix G. T-Few attains state-of-the-al accuracy of 75.8% and outperforms the human baseline 73.5% accuracy) for the first time. The next-best model (from Schick and Schütze [50]) achieves 6% lower accuracy and GPT-3 175B attains only 62.7%. These results validate that T-Few can be readily applied as-is to novel real-world tasks for strong performance.
4.4 Ablation experiments
Given that our T-Few design experiments were on TO-3B, we perform an ablation of some of the ingredients of T-Few on TO. Results are shown in appendix F. While the gains from adding each ingredient does not always significant increase the accuracy on each individual dataset, each ingredient consistently improves the average performance across datasets: Removing pre-training decreases accuracy by 1.6%, removing unlikelihood training and length normalization decreases accuracy by 4.1%, and removing both pre- pre-training and our additional loss terms reduces accuracy by 2.5%.
5 Related Work
Currently, prompt tuning is one of the most parameter-efficient methods for large language models [29, 14, 53]. Liu et al. [54] introduce several tricks to improve prompt tuning, An et al. [55] tune
hhttps://doo
microsoft. wwtpsofttccomeeeeeme een uan-u//an aus/azure/viic sazureevirrtul-l watpaaaaaaammmaaeenneeeeee wwwpieess/nmmaa00---10

---

prompts along with input embeddings for boost in performance, and Chen et al. [56] improve prompt embeddings through continued pre-training. Given optimization difficulties when training prompt embeddings, Diao et al. [57] recently used black-box optimization to train prompt embeddings without requiring gradients. Several works have analyzed prompt tuning from the perspective of
interpretability Khashabi et al. [58] and its similarity to other PEFT methods He et al. [30]. Prompt tuning has been applied to various applications for NLP including continual learning [59], model robustness [60, 61]. summarization [62], machine translation [63]. co-training [64], probing language
models [65, 65], inverse prompting [66] and transfer learning [67]. He et al. [68] recently proposed the use of a hyperneworr to predict prompts for new tasks (rather than training the prompt parameters with gradient descent). Prompt tuning and other PEFT methods have also been explored outside of the context of language models (e.g. vision [22, 69] and Usion-and-a language models [26]).
Separately, various studies have considered few-shot full-model fine-tuning with discrete prompts [70]. Recent work has analyzed training with discrete prompts, demonstrating boost in performance with prompting when training on various numbers of examples [71], finding that models perform similarly when trained on good and bad prompts [11], and exploring which prompts work well for few-shot and full-shot setting [72]. There have also been efforts to develop methods that find discrete prompts [73, 74] and training prompts using methods similar to prompt tuning [75].
There has also been a great deal of work on improving ICL. Chen et al. [5], Min et al. [6] use ICL for meta-learning to perform few-shot learning on new tasks. Lampinen et al. [7] show ICL can improve when explanations are provided and [8] use ICL with text retrieved from the web for open-domain drestion-answering Meanwhile, Min et al. [9] analyze how ICI. works and show that ICL. can still perform well when incorrect labels are provided for the in-context examples.
With the advent of billion-parameter language models, there has been a great deal of recent interest in PEFT methods and their compatibility in the few-shot setting. Mahabadi al. [28] found that PEFT outperforms standard fine-tuning in the low-resource setting. In concurrent work, Mahabadi et al. [76] find that PEFT compares favorably in few-shot fine-tuning against discrete prompts (e.g.
PET [70]. Also concurrently, Moosavi et al. [77] propose a framework for introducing adapters whose architecture and design vary from task to task and demonstrate improved results in few-shot settings. Gu et al. [18] and Vu et al. [19] both explored how pre- vre-training prompt tuning parameters can improve when limited labeled data is available. For few-shot learning, Triantafillou et al. [78] explore learning universal and dataset dependent parameters that can be blended for generalization. Requeima et al. [79] use conditional neural adaptive processes and Li et al. [80] leverage distillation from multiple feature extractors to learn new classes or domains in few-shot learning.
Conclusion
We introduced T-Few, a parameter-efficient few-shot learning recipe that attains higher accuracy than few-shot ICL at lower computational cost. T-Few uses (IA) new PEFT method that rescales inner activations with learned vectors. Using (IA)* produces better performance than fine-tuning the full model while introducing minimal additional parameters. T-Few also uses two additional loss terms that encourage the model to output lower probabilities for incorrect choices and account for the length of different answer choices. When applying T-Few as-is (with no task-specific hyperparameter tuning or other changes) to the RAFT benchmark, we attained super-human performance for the first time and outperformed prior submissions by a large margin. Through detailed characterization of computational costs, we found that T-Few uses over 1,000x fewer FLOPs during inference than few-shot ICL with GPT-3 and only requires 30 minutes to train on single NVIDIA A 100 GPU. Since all of our experiments were on classification tasks, we are interested in applying T-Few to generative tasks like as summarization and question answering in future work. We hope our results provide a new perspective on how best to perform few-shot learning with large language models.
Acknowledgments and Disclosure of Funding
We thank Brian Lester and Noah Constant for helpful discussion on debugging prompt tuning and Rabeeh Karimi Mahabadi for help with Compacter and Intrinsic SAID. We also thank Stella Biderman and the Google TPU Research Cloud who provided valuable computational resources to support this work. This work was supported by NSF-AI Engage Institute DRL-2112635,
10

---

References
[1] Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. arXiv preprint 2rXiv:2110002220 2021.
[2] Neel Alex, Eli Liftand, Lewis Tunstall, Abhishek Thakur, Pegah Maham, Jess Riedel, Emmic Hine, Carolyn Ashurst, Paul Sedille, Alexis Carlier, et al. RAFT: A real-world few-shot text classification benchmark. arXiv preprint vrXiv22109.10077 2021.
[3] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 2019.
[4] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint 21Xx::2005.14165, 2020.
[5] Yanda Chen, Ruiqi Zhong, Sheng Zha, George Karypis, and He He. Meta-learning via language model in-context tuning. arXiv preprint 10000:21 2021.
[6] Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. Metaicl: Learning to learn in context. arXiv preprint arXiv:21 (0. 15943, 2021.
[7] Andrew Kyle Lampinen, Ishita Dasgupta, Stephanie C. Y. Chan, Kory Matthewson, Michael Henry Tessler, Antonia Creswell, James L. McClelland, Jane X. Wang, and Felix Hill. Can language models learn from explanations in context? ArXiv, abs/2204.02333 2022.
[8] Angeliki Lazaridou, Elena Gribovskaya, Wojeiech Stokowiec, and Nikolai Grigorev. Internet- augmented language models through few-shot prompting for open-domain question answering. arXiv preprini arXiv:2203.05115. 2022.
[9] Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannanch Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12833 2022.
[10] Yizhong Wang. Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaci. Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasckaran, Atharva Naik, David Stap, et al. Benchmarking generalization via in-context instructions on 1,600+ language tasks. arXiv preprint arXiv:2204.07700 2022.
[11] Albert Webson and Ellie Pavlick. Do prompt- based models really understand the meaning of their prompts? arXiv preprint arXiv:2109, 01247, 2021.
[12] Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. arXiv preprint arXiv:2102.096600 2021.
[13] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. ArXiv, abs/2106.09685, 2021.
[14] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprins arXiv:2 /04. 08691, 2021.
[15] Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. ArXiv, abs/1910. 10683, 2020.
[16] Derek Tam, Rakesh R Menon, Mohit Bansal, Shashank Srivastava, and Colin Raffel. Improving and simplifying pattern exploiting training- arXiv preprint arXiv:2103. 1955, 2021.
[17] Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. arXiv preprint arXiv: 04319, 2019.
[18] Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang. PPT: Pre-trained prompt tuning for few-shot learning, arXiv preprint
2021.

---

[19] Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou, and Daniel Cer. SPoT: Better frozen model adaptation through soft prompt transfer. arXiv preprint arXiv:2110.07904, 2021.
[20] Jared Kaplan, Sam McCandlish, T. J. Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprini weXx:20001.083000 2020.
[21] Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Noisy channel language model prompting for few-shot text classification. arXiv preprint WXXi:2208.044066, 2021.
[22] Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. Advances in neural information processing systems, 30, 2017.
[23] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efficien transfer learning for NLP. arXiv preprint arXiv:1902.00755 2019.
[24] Ankur Bapna, Naveen Arivazhagan, and Orhan Firat. Simple, scalable adaptation for neural 2019.
machine translation. arXiv preprint
[25] Demi Guo, Alexander M. Rush, and Yoon Kim. Parameter-efficient transfer learning with diff' pruning. arXiv preprint arXiv:2012.07766. 2020.
[26] Yi-Lin Sung, Varun Nair, and Colin Raffel. Training neural networks with fixed sparse masks. arXiv preprint arXiv:21/1.09839, 2021.
[27] Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. arXiv preprint arXiv:20/2./3255, 2020.
[28] Rabech Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. arXiv preprint 0rXiv:2106.04647, 2021.
[29] Xiang Lisa Li and Percy Liang. Prefix-Tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv: 101.00900, 2021.
[30] Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Berg-Kirkpatrrik, and Graham Neubig. To- wards a unified view of parameter-efficient transfer learning. arXiv preprint arXiv:21 10.04366, 2021.
[31] Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few-shot learning with language models. arXiv preprint 11.X:::2005.114477. 2021.
[32] Avital Oliver, Augustus Odena, Colin Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. Advances in Neural Information Processing Systems, 2018
[33] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, Advances in Neural Information Processing Systems, 2017.
[34] Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 18100:18100.45500 2018.
[35] Stephen H. Bach, Victor Sanh, Zheng-X0 Yong, Albert Webson, Colin Raffel, Nihal Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Févry, et al. PromptSource: An integrated development environment and repository for natural language prompts. arXiv preprint arXiv:2202.012799 2022.
[36] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, et al. Transformers: State- of-the-art natural language processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2020.
12

---

[37] Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. 2011 AAAI Spring Symposium Series, 2011.
[38] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? arXiv preprint arXiv: 090.. 1995500830, 2019.
[39] Rishi Sharma, James Allen, Omid Bakhshandeh, and Nasrin Mostafazadeh. Tackling the story ending biases in the story cloze test. In Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 752-757, 2018.
[40] Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial NLI: A new benchmark for natural language understanding. arXiv preprint 1910:: 14599, 2019.
[41] Marie-Catherine de Marneffe, Mandy Simons, and Judith Tonhauser. The CommitmentBank: Investigating projection in naturally occurring discourse. Proceedings of Sinn und Bedeutung 23, 2019.
[42] Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop, pages 177-190. Springer, 2005.
[43] Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. Thir- teenth International Conference on the Principles of Knowledge Representation and Reasoning, 2012.
[44] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
[45] Mohammad Taher Pilehvar and Jose Camacho- Collados. WiC: the word-in-contest dataset for evaluating context-sensitive meaning representations. arXiv preprint 1808:0880.00112 2018.
[46] Stefan Van Der Walt, S. Chris Colbert, and Gael Varoquaux. The numpy array: a structure for efficient numerical computation. Computing in science & engineering, 13(2), 2011.
[47] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg, BitFit: Simple parameter-efficient fine-tuning for transformer-based masked language-moded models. arXiv preprint arXiv:2106.10199 2021.
[48] Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V. Le. Finetuned language models are zero-shot learners. arXiv preprint (oXiv:2/09.01600 2021.
[49] Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning. PMLR, 2018.
[50] Timo Schick and Hinrich Schüttee True few-shot learning with prompts-a weal-world perspective. arXiv preprini 1r 2021.
[51] Moshe Wasserblat. Sentence transformer fine-tuning (SetFit): Outperforming GPT-3 on few- shot text-classification while being 1600 times smaller, 2021.
[52] Mostafa Dehghani, Anurag Arnab, Lucas Beyer, Ashish Vaswani, and Yi Tay. The efficiency misnomer. arXiv preprint arXiv:2110.12288 2021.
[53] Guanghui Qin and Jason Eisner. Learning how to ask: Querying LMs with mixtures of soft prompts. arXiv preprint w.Xiv:2104.06500 2021.
[54] Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-Tuning v2: Prompt tuning can comparable to fine-tuning universally across scales and tasks. arXiv preprint 1100022110.07602, 2021.
13

---

[55] Shengnan An, Yifei Li. Zeqi Lin. Qian Liu, Bei Chen, Qiang Fu, Weizhu Chen, Nanning Zheng. and Guang Lou,
Adapting unfamiliar inputs to frozen pretrained models. arXiv preprini arXiv:2203.03131, 2022.
[56] Yulong Chen, Yang Liu, Li Dong, Shuohang Wang. Chenguang Zhu, Michael Zeng, and Yue 7hang. AdaPrompt: Adaptive model training for prompt-based NLP. arXiv preprint arXiv:2202.0482. 04824, 2022.
[57] Shizhe Diao, Xuechun Li, Yong Lin, Zhichao Huang, and Tong Zhang. Black-box prompt learning for pre-trained language models. arXiv preprint arXiv:2201.0835 1000000..831,000 2022.
[58] Daniel Khashabi, Shane Lyu, Sewon Min, Lianhui Qin. Kyle Richardson, Sameer Singh, Sean Welleck, Hannanch Hajishirzi, Tushar Khot, Ashish Sabharwal, et al. Prompt wayward- ness: The curious case of discretized interpretation of continuous prompts. arXiv preprint arXiv:21 2021.
[59] Zifeng Wang. Zizhao Zhang, Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. arXiv preprint arXiv: 12.08554 2021.
[60] Zonghan Yang and Yang Liu. On robust prefix-tuning for text classification. arXiv preprint arXiv:2203.10378, 2022.
[61] Yuting Yang, Pei Huang, Juan Cao, Jintao Li, Yun Lin, Jin Song Dong, Feifci Ma, and Jian Zhang. prompting- based approach for adversarial example generation and robustness enhancement. arXiv preprint arXiv: 10714, 2022.
[62] Xiaochen Liu, Yu Bai, Jiawei Li, Yinan Hu, and Yang Gao. PSP: Pre-trained soft prompts for few-shot abstractive summarization. arXiv preprint arXiv:2204.04413, 2022.
[63] Xavier Garcia and Orhan Firat. Using natural language prompts for machine translation. arXiv preprint arXiv: 11822, 2022.
[64] Hunter Lang, Monica Agrawal, Yoon Kim, and David Sontag. Co-training improves prompt- based learning for large language models. arXiv preprint arXiv:2202.00823. 2022.
[65] Boshi Wang, Xiang Deng, and Huan Sun. Shepherd pre-trained language models to develop a train of thought: An iterative prompting approach. arXiv preprint 20030:2003. 2022.
[66] Xu Zou, Da Yin, Qingyang Zhong, Hongxia Yang, Zhilin Yang, and Jie Tang. Controllable gener- ation from pre-trained language models via inverse prompting, arXiv preprint ar 103. 2021.
[67] Yusheng Su, Xiaozhi Wang. Yujia Qin, Chi-Min Chan, Yankai Lin, Zhiyuan Liu, Peng Li, Juanzi Li, Lei Hou, Maosong Sun, et al. On transferability of prompt tuning for natural language understanding. arXiv preprint arXiv:21 1.06719. 2021.
[68] Yun He, Huaixiu Steven Zheng, Yi Tay, Jai Gupta, Yu Du, Vamsi Aribandi, Zhe Zhao, YaGuang Li, Zhao Chen, Donald Metzler, al. HyperPrompt: Prompt-based task-conditioning of transformers. arXiv preprint arXiv:2203.00753 2022.
[69] Menglin Jia, Luming Tang, Bor- Chu-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. arXiv preprint 1rXiv:2203.12119 2022.
[70] Timo Schick and Hinrich Schütze. Exploiting cloze questions for few shot text classification and natural language inference. arXiv preprint MoXi::0001007670 2020.
[71] Teven Le Scao and Alexander M. Rush. How many data points prompt worth? arXiv preprint arXiv:2 2021.
[72] Sen Yang, Yunchen Zhang, Leyang Cui, and Yue Zhang. Do prompts solve NLP tasks using natural language? arXiv preprint arXiv:2203.1 20030002,,,,, 2022.
14

---

[73] Taylor Shin, Yasaman Razeghi, Robert L Logan IV. Eric Wallace, and Sameer Singh. Auto- Prompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprini arXiv:2010. 15980, 2020.
[74] Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. arXiv preprint (tx.xxv2212..52222. 2020.
[75] Ningyu Zhang, Luoqiu Li, Xiang Chen, Shumin Deng. Zhen Bi, Chuanqi Tan, Fei Huang, and Huajun Chen. Differentiable prompt makes pre-trained language models better few-shot learners. arXiv preprint arXiv: 13161, 2021.
[76] Rabeeh Karimi Mahabadi, Luke Zettlemoyer James Henderson, Marzieh Saeidi, Lambert Mathias, Veselin Stoyanov, and Majid Yazdani. PERFECT: Prompt-free and efficient few-shot learning with language models. arXiv preprint arXiv:2204.0117,, 2022.
[77] Nafise Sadat Moosavi, Quentin Delfosse, Kristian Kersting, and Iryna Gurevych. Adaptable adapters. arXiv preprint arXiv:2205.01549, 2022.
[78] Eleni Triantafiloou, Hugo Larochelle, Richard Zemel, and Vincent Dumoulin. Learning a universal template for few-shot dataset generalization. arXiv preprint arXiv:/2105.07022 2021.
[79] James Requeima, Jonathan Gordon, John Bronskill, Sebastian Nowozin, and Richard E. Turner. Fast and flexible multi-task classification using conditional neural adaptive processes. arXiv preprint arXiv; 1906. 07697, 2019.
[80] Wei-Hong Li, Xialei Liu, and Hakan Bilen. Universal representation learning from multiple domains for few-shot classification. Proceedings of the IEEE/CVF International Conference Computer Vision., 2021.
[81] Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, September 2021. URL httph::t///ee //doi. org/ 10. 5281 /122///eeodoo zenodo. 1371628.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] We compare different PEFT methods in Figure 2 and
demonstate PEFT outperforms ICL in section 4.
(b) Did you describe the limitations of your work? [Yes] Section 5
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or URL)? [Yes] Included code with README
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Section 3.1, Section 3.5, Appendix D,
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] Appendix C, D. E,
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] Appendix A
15

---

4. If you are using existing assets (e.g., code, data, models) or curating/releasnng new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Section 3.1, Section
4.3
(b) Did you mention the license of the assets? [Yes] Section 3.1
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'le rr using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "jrwvckediRi", "reviewer": "Reviewer_EGYW", "summary": "This paper introduces T-Few, a parameter-efficient few-shot learning protocol that achieves STOA few-shot performance on many tasks, and also outperforms full-model finetuning and in-context learning with huge LMs. They rescale inner activations of LMs with learned vectors, such that only those vectors are updated during training. Their design is slightly different from existing parameter-efficient methods to enable mix-task batch training. \nThis paper demonstrates good engineering practice of parameter-efficient learning but there are flaws in the evaluations.   ", "questions": ".", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The proposed method demonstrates strong few-shot performance on popular benchmarks and challenging RAFT benchmark. Those engineering practices will be a good contribution to the community and serve as bases for feature research on parameter-efficient tuning. \nThe presentation is clear and easy to follow, and the motivation is well-justified and supported. \n\nHowever, I have the following concern regarding evaluation.\nAlthough the proposed method clearly outperforms baselines, I'm worried that the comparison is not fair. For instance, IA^3 is powered by two new training objectives and mix-task batch training. All these have been proved effective in improving few-shot performance. However, in the evaluation, those tricks are not applied on baselines (especially new losses), resulting in an unfair comparison.  \n\nThere are also some mistakes in the description of the methodology, not sure if it's a typo or not. e.g., L212 we should minimize L_{LN} instead of maximizing it.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "YfzRP0Jhtfn", "reviewer": "Reviewer_jndu", "summary": "The paper introduces a parameter efficient fine-tuning method that works for few-shot learning. The main contributions of the paper include: Extending T0 to work for few-shot learning. A single recipe for few-shot fine-tuning and a new parameter-efficient approach that outperforms prior work like prompt tuning and adapters.", "questions": "Please see above.", "limitations": "There is no discussion about limitations. For example, it is still an open question what makes AI3 to work better than methods like Adapters or LoRA. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The paper is extremely well written. It was a joy to read.\n- The new parameter-efficient approach (IA3) achieves strong results compared with other methods.\n- The paper achieves strong few-shot results with moderate language model sizes (up to 11B), outperforming more expensive models like GPT3.\n- A single hyperparmeter and model configuration setting for fine-tuning on all tasks.\n\nWeaknesses:\n\n- Some of the ablations in the appendix do not show reliable improvements and standard deviation seems too high (assuming the subscript numbers in the the appendix table cells refer to stdev). \n- Figure 2 shows the benefits of IA3 compared with prior work when fine-tuning T0-3B. But does the entire “T-few recipe” fail if another parameter efficient model (like adapters) is used instead of IA3? I.e., how important is AI3 within the proposed framework?\n- I encourage the authors to move important ablations \n to the main text. If space is concern, related work could be shortened.\n- The only ICL baseline considered is GPT-3. There have been several follow up works on improving ICL. More recent baselines like Chinchilla [Hoffmann et al., 2022] could have been also considered.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Iyirer7BypB", "reviewer": "Reviewer_yoAz", "summary": "The paper focuses on few-shot learning in NLP and shows the limitation of few-shot in-context learning method (high computational cost and so on) and propose an adaptation strategy that adds light-weight learnable parameters (rescaling vectors) into a frozen pre-trained model and fine-tune only the learnable parameters on few labeled samples for few-shot learning. The experimental results shows the proposed strategy obtains much better performance than existing few-shot ICL methods and PEFT methods with lower inference computation cost while it requires largest training computation cost, disk cost and and higher memory cost during training T-Few.", "questions": "Some related literatures are missing [ref1,2,3].\n\n[ref1] Requeima, James, et al. \"Fast and flexible multi-task classification using conditional neural adaptive processes.\" Advances in Neural Information Processing Systems 32 (2019).\n\n[ref2] Triantafillou, Eleni, et al. \"Learning a universal template for few-shot dataset generalization.\" International Conference on Machine Learning. PMLR, 2021.\n\n[ref3] Li, Wei-Hong, Xialei Liu, and Hakan Bilen. \"Universal representation learning from multiple domains for few-shot classification.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.", "limitations": "The limitations of the method are discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. In overall, the paper is well-written.\n\n2. The idea of reusing the pre-trained network and only learn additional light-weight parameters to adapt the pre-trained model for downstream tasks with few samples is interesting and shown to be obtain promising performance on downstream tasks.\n\n3. The introduced rescaling vectors are light-weight.\n\nWeaknesses:\n\n1. One main weakness is the training cost of the T-Few. More specifically, when training the rescaling vectors on a new tasks, training T-Few will involves forward and backward passes through the whole model (though only rescaling vectors require gradients for updates) which will be computational costly for each tasks. Suppose there are a mass of tasks, the training cost is considerable.\n\n2. Also, applying the proposed method to larger model can also increase the cost, e.g. applying to GPPT-3 with 175B parameters.\n\nIn overall, I think the idea is interesting and promising in the context of few-shot learning. However, I am not an expert on NLP field and I can not  assess the novelty of the paper in NLP.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Hty0BCjtHa7", "reviewer": "Reviewer_Fdzk", "summary": "This paper presents a new parameter efficient fine-tuning (PEFT) method for pre-trained transformer models in NLP. The authors argue that, with a careful choice of parameters to be fine-tuned, this approach can be computationally much cheaper than in-context learning (ICL) and perform better than in-context learning approaches. The paper proposes a new method called IA3, where the trainable parameters are vectors that scale the activations in each layer. When applied to the T0 model, the authors show that the proposed fine-tuning method performs better than prior ICL and PEFT methods. In addition, a careful analysis is presented that shows the computation consumed by the proposed method and other approaches.", "questions": "* In Fig 2, how do you account for the difference in architecture, difference in pre-training data, etc.? Ideally, we would want to compare IA3 to other approaches for the same base model. ", "limitations": "Seems adequate", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* Clarity on experimental details, helps reproducibility. \n* Extensive experiments and strong performance.\n* Offers a practical recipe for few-shot learning. \n* Clearly presented motivations.\n* Computationally advantageous and better than in-context learning and other few-shot/parameter efficient learning methods.\n\nWeaknesses\n* The presentation can be reorganized so that the main contributions are highlighted better. For instance, IA3 could be introduced early on. The current presentation is dense and could use better structuring. \n* Although fig. 2 presents an extensive comparison, it is unclear whether the authors accounted for the difference in model architectures, pre-training data, etc.\n\nOverall this is an interesting paper with extensive evaluation of the proposed approach. I appreciate the careful analysis of computational requirements. The following issues have to be addressed by the authors.\n* How were hyperparameters and other design choices made? Did you stick to the ‘true few-shot learning setting’, or were there any validation sets involved? I would recommend making this clear to make sure that comparisons with baselines are fair. \n* Since the main method applies IA3 to T0, it is difficult to isolate the performance contribution of each. I would recommend presenting an analysis of IA3 applied to plain language models (no instruction tuning). \n* Did you also try baseline methods (adapters, prefix tuning, etc.) with/without unlikelihood training, length normalization, etc.? \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept