True Few-Shot Learning with Language Models
Authors: Ethan Perez
Douwe Kiela
Kyunghyun Cho
Venue: NeurIPS 2021 Poster (2021)
Keywords: Few-Shot Learning
Natural Language Processing

Abstract: Pretrained language models (LMs) perform well on many tasks even when learning from a few examples, but prior work uses many held-out examples to tune various aspects of learning, such as hyperparameters, training objectives, and natural language templates ("prompts"). Here, we evaluate the few-shot ability of LMs when such held-out examples are unavailable, a setting we call true few-shot learning. We test two model selection criteria, cross-validation and minimum description length, for choosing LM prompts and hyperparameters in the true few-shot setting. On average, both marginally outperform random selection and greatly underperform selection based on held-out examples. Moreover, selection criteria often prefer models that perform significantly worse than randomly-selected ones. We find similar results even when taking into account our uncertainty in a model's true performance during selection, as well as when varying the amount of computation and number of examples used for selection. Overall, our findings suggest that prior work significantly overestimated the true few-shot ability of LMs given the difficulty of few-shot model selection.

True Few-Shot Learning with Language Models
Ethan Perez', Douwe Kiela?, Kyunghyun Cho New York University, Faaceoook AI Research, CIFAR Fellow in Learning in Machines Brains perez@nyu.edu
Abstract
Pretrained language models (LMs) perform well on many tasks even when learning from a few examples, but prior work uses many held-out examples to tune various aspects of learning, such as hyperparameters, training objectives, and natural language templates ("prompts"). Here, we evaluate the few-shot ability of LMs when such held-out examples are unavailable, a setting we call true few-shot learning. We test two model selection criteria, validavan and minimum description length, for choosing LM prompts and hyperparameters in the true few-shot setting. On average, both marginally outperform random selection and greatly underperform selection based on held-out examples, Moreover, selection criteria often prefer models that perform significantly worse than seneeee ones. We find similar results even when taking into account our uncertainty in a model's true performance during selection, well as when varying the amount of computation and number of examples used for selection. Overall, our findings suggest that prior work significantly overestimated the true few-shot ability of LMs given the difficulty of few-shot model selection.
1 Introduction
Major progress language model (LM) pretraining has led to the idea that LMs can learn a new task using a small number of examples only, i.e., few-shot learning [133]. Few-shot learning overcomes many challenges with data-rich supervised learning: collecting labeled data expensive, often requires experts, and scales poorly with the number of tasks. However, the few-shot performance of LMs is very sensitive to the textual task description ["prompt"; 3-6 order of training examples (6)
decoding strategy 10] and other hyperparameters 5 9 12 as well as the learning algorithm itself 12 Thus, effective model selection is crucial for obtaining good few-shot performance.
There are issues with how recent work approaches model selection in few-shot learning, however. Prior work uses large train or held-out sets with many examples to choose prompts 1 [K] and hyperparameters [2]. Other work claims to use no validation set for hyperparameter selection B 14 but does not describe how they design other aspects of their learning algorithm (e.g.. training objectives). It is unlikely that no validation examples were used, given the sophisticated nature of the proposed algorithms. In this work, we examine prior few-shot learning methods still perform well when using only the provided examples for model selection, setting we term true few-shot learning.
We find that true few-shot model selection yields prompts that marginally outperform random selection and greatly underperform selection based on held-out examples. Our result shows that prior work may have greatly overestimated the few-shot ability of LMs. In other words, one reason that prompts are so effective ["worth many examples"; [5] is that they are often tuned using many examples. We evaluate two standard model selection criteria cross-validation (CV) and minimum description length (MDL) finding that both obtain only limited improvements over random selection and perform much worse than selection using held-out examples. For prompt selection, our observation holds for LMs ranging over orders of magnitude in size [6] on 3 classification tasks and 41
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

tasks in the LAMA benchmark [77] For choosing hyperparameters, true sho-shot selection causes performance to drop by 2-10% across tasks for ADAPET [2]. a ofate-of-the-art few-shot method. Furthermore, true few-shot model selection has high variance in performance; selected models often do much worse than randomly-chnn examples used, amount of computation, and conservativeness of our selection criterion. Altogether, our results suggest that model selection is a fundamental roadblock to true few-shot learning.
ones. We find similar results when varying the number of
2 Can We Do Model Selection in Few-Shot Learning?
Prior work uses the phrase "few-shot learning" in multiple senses, raising questions about what means to do few-shot learning. We categorize few-shot learning into three distinct settings, each of which assumes access to different data. Here, we formally disambiguate between these settings to help future work avoid inadvertently comparing few-shot methods that operate in different settings.
Consider the supervised learning scenario where we have a dataset of inputs æ I1.N and labels yr.N. sampled from distribution over datasets D. We aim to determine the learning algorithm A' E AI,...: AA with the smallest generalization loss L at predicting y given x on unseen validation examples vaal ~ D after learning on training examples D Duain ~ D. We say that an algorithm A(Duii, R) maps a training dataset Drain and various random factors R that influence training to a function that predicts given r. A specifies, e.g., the model architecture, hyperparameters. and prompt. R includes random factors that impact the results of a learning algorithm, such as parameter initialization and the order of training examples for online learning algorithms like stochastic gradient descent. We say that A obtains a generalization loss C(A((rrain R), Dval) on a given validation set D We aim to find the A that minimizes the expected loss across training and validation sets:
EL(A, R) ED Pral c (A((Drain: R); Deal)
In data-rich supervised learning. EL(AA, Rt) is usually evaluated with a single train-validation split (Durin v. Since large Durain and Dval are not always available, the traditional few-shot setting evaluates (A,, R) with many small (Danin Dval) drawn from many, distinct distributions D [see, e.g., work in meta-learning 18-21] Each distribution D is sampled from D a distribution over call this setting miulti-distribution few-shot learning.
distributions (e.g., of similar tasks),
Recent work does not assume access to data from other distributions, performing few-shot learning using only a few examples from single distribution to update a pretrained LM (2 12). These papers use large validation set Dval to tune the learning algorithm A, a setting we term luned few-shot learning. For example, Brown et al. [2] try prompts with different phrasings and numbers of training examples to improve the validation accuracy of GPT-3. Tam et al. [12 choose the early stopping iteration, prompt, and other model-specific hyperparameters based on validation performance. Tuned few-shot learning relies on many labeled examples, so we argue that tuned few-shot learning does not qualify as few-shot learning. If many validation examples are available, they could be incorporated into the training set and trained on using data-rich supervised learning. Tuned few-shot learning algorithms should be compared against data-rich supervised learning algorithms that use the same amount of total data Drrainl + Dval-
In this work, we evaluate the success of tuned few-shot learning methods when no large Dval is available, a setting we term true few-shot learning. Formally, we aim to choose a learning algorithm A with low expected loss (L, R) using only small training set Dinin drawn from single distribution. Here, we must choose A by approximating EL(A, R). e.g., using cross- validation. Several papers claim to circumvent the need to estimate EL(A, R) by choosing hyperparameters based on an educated guess 3 14]. However, the proposed learning algorithms themselves are quite sophisticated, and it is unclear how they were designed if not by using validation performance.
Other work chooses the learning algorithm and hyperparameters using one or multiple other datasets before evaluating on the target dataset [T]. Such approaches fall under multi- distriitittiitiion few-shot learning and cannot be directly compared to methods that attempt to perform true few-shot learning, even though prior work has made such comparisons [14).
In what follows, we describe two model selection criteria cross-validation and minimum description length which we use to evaluate tuned few-shot methods in the true seo-shot setting.

---

2.1 Cross-validation
Cross- Validation (CV) 22 is one the most widely used methods for estimating generalization loss [25]. CV has also been used in prior work on multi-distribution few-shot learning [26 CV randomly partitions Duain into K equally-sized folds F uaanni)
uainn) ) K and evaluates the average loss validation fold F F(auin) when training the remaining data F FDDuain)=k
(V,ARR,F) (k--U ( (AA(F(P)aa)) de R); F( F(Danan)t))
In this way, CV forms K train-validation splits out of the pool of labeled examples. CV with one example per fold (K = N folds) is commonly referred to as lcave-one-out CV (LOOCV).
2.2 Minimum description length
We may also form train-validation splits a different manner than CV, drawing inspiration from work on the Minimum Description Length (MDL) principle (8). MDL can be estimated by evaluating the average loss on a fold F(D); when training on the previous folds F(D)
( (A F ((Duan)s) (Duain)s)
MDL(A.R,F) Lm-
This procedure is referred to as "online coding" 29 30
as it evaluates the generalization loss of the
algorithm as learns "online" from more and more data There are other ways evaluate MDL [see 31 for overview]. We use online coding as it has been shown to be an effective way to estimate MDL, especially for deep learning methods [32]
MDL measures generalization because it evaluates how much a learning algorithm compresses the labels Y1.N given the inputs Z1:N. and because better compression implies better generalization [33]. Recent work has used MDL to determine which learning algorithms are most effective at explaining the given data [Rissanen Data Analysis: 10 34
2.3 Variance matters
We evaluate the generalization loss of the algorithm chosen by CV (likewise for MDL):
L(Acv (Drins R), Dval),
where Acv arg min (CV R,
The above loss should be low in expectation, across different datasets Duzin D, D D, and random factors R. F: E Dra...Da..RR FC ([t((((A(vv))) (Drin R), Dvai)]. The loss should also be low in variance: V Dam.. D.,. R, (LLL [C(Acv(Dirain R), Dva)]. Low variance implies that CV/MDL reliably choose an algorithm that generalizes to Dval when trained with a given Draain and random factors R, F. Reliability important for many practical commercial applications where worst-case performance is important, such as image recognition [35] 36), dialogue systems [37 38 and robotics (39 40).
We also experiment with explicitly taking into account algorithm's variance during model selection, choosing Acv to minimize a conservative estimate of CV, CV)(A), chosen such that the probability Pra.r (CV,,, R, F) (A)) is high:
CV) = ER.F [C((A.R,F) av / VR..(CV(AARREE
where is hyperparameter set based the desired probability. In particular, if CV R, F) follows a normal distribution. N when sampling R, F, then CV, R, F) S. CV)(A) with probability S (p 0, a = 1) for a given R, F. CV. resembles the Watanabe Akaike Information Criterion 4T, which estimates the generalization of a model trained with A using the expected loss from model trained with A plus the variance in training loss across models trained with A.
OOnine coding formally computes a sum over 2(.) rather than the expectation, which differs by constant factor. The two equivalent for our purposes (ranking A).

---

2.4 Other model selection criteria
Prior work has developed other model selection criteria such as the Akaike Information Criterion [AIC; 421. Watanabe-Akaike Information Criterion WAIC;A4,, and Mallows' Cp 433. These methods often rely on assumptions or quantities that are not available in the context of deep learning (AIC, Mallows' Cp) or are approximations of LOOCV (WAIC). Since state-of-the-ar few-shot learning methods tend to be based on deep learning, we focus on CV and MDL as our model selection criteria. In Appendix SB] we also test several other criteria that are applicable deep learning methods.
Selection criteria can be optimized automatically, e.g. with bayesian optimization 444/46),
evolutionary methods [45 47 48), reinforcement learning [49], gradient descent (50-53] Such methods aim to match the performance of exhaustive search, the optimal approach (used in our work).
3 True Few-Shot Prompt Selection
Recent work on LMs performs few-shot learning by providing training examples as input in the form
of natural language "prompt" (2 3 19. For example, for a question-answerring answering task, Brown et al. 2
prepend input examples with "READING COMPREHENSION ANSWER KEY" before providing them to GPT-3 (see Appendix Table for more examples). They then have the LM complete the remaining words the prompt, conditioning on earlier words (including various input examples), following the LM's pretraining objective (next word prediction). No parameter updates are involved. It is not obvious a priori which prompts will generalize well for given LM, and there is high variance in how well different prompts generalize 6 even between prompts with minor differences [e.g..
one comma; 5. Effective prompt selection is thus crucial for true few-shot learning.
3.1 Experimental setup
In what follows, we test on LAMA 17 a benchmark for retrieving facts with LMs, for which prior work has developed many strategies for designing prompts 4 56). LAMA evaluates the accuracy of LMs at choosing the correct target object for various (subject, relation, ob object) triples present in knowledge bases, such as (Dante, born-in, Florence). We use the "TREx" split (57], which consists of 41 relations (up to 1K examples each). Petroni et al. 17 design a prompt for each relation, which an LM completes to predict an answer (e.g., "The birthplace of Dante was _"). Some relations have multiple valid target entities, so LAMA evaluates how often one of the true answers matches the top- prepriideeed token (out of 20k candidates). We only use examples from the LAMA-UnHelpfulNAe
subset [LAMA-UHN] 58) which filters out easy-to- guess examples (e.g., "The Apple Watch was created by " with the answer Apple). We test the 5-shot accuracy of 9 popular LMs of various sizes: GPT-3 [175B, 13B, 6. 6.7B, 2.7B parameter models; 21, GPT-2 [1.5B, 782M. 345M, 117M models; I, and DistilGPT-2 [16), a distilled, 82M parameter version GPT-2 117M The supplementary material contains the code to reproduce all results and plots in our paper.
Prompts To form our of candidate prompts AI,...,AA AA. we rely on LAMA as well as the LM Prompt And Query Archive [. For each relation, we use the manually-written written prompt from LAMA, plus LPAQA prompts formed by (1) paraphrasing the manual prompt with back-translation (2) mining from Wikipedia, and (3) paraphrasing the top mined prompt. For each relation, we use up to 16 prompts with mean of 12 prompts (see Appendix I for details on the prompts we use).
Computing CV and MDL As the loss function C. we use the negative log-likelihood (NLL) of the label given the input over all held-out examples Σ (a.ww) - log p(y|xz). We use NLL as in prior work on MDL 32, 61 10), to retain MDL's property as measure of label compression. For CV, NLL avoids ties between prompts that arise from using accuracy in the context of such limited data (e.g.. examples). We use K = N folds (where N is the number of training examples) for both MDL and CV (here, LOOCV). ). Here, N-fold CV requires N forward passes to evaluate the loss on each of the N examples when conditioning on the N other examples. N-fold MDL can be computed using
We use OpenAl's API for GPT-3 (https: /bbaa. openai. com/) and HuggingFace Transformers (59 via Torch (60) for GPT-2/DistlGPT-2 OpenAl does not disclose API model sizes, so following 7 we assume that the API models are the 4 largest from (2]. We will update our paper should OpenAl release model sizes.

---

Test Acc. of Chosen Prompt (96)
60
100
Werst
50
Relative Test Acc. Chosen Prompt (%)
80
MDL
CV
40
30
40
20
20
0
-20
0.08
0.1
0.3
0.8
1.5
2.7
6.7
13
175
0.08
0.1
0.3
0.8
1.5
2.7
6.7
13 175
Model Parameters (B)
Model Parameters (B)
Figure 1: Left: LAMA-UHN accuracy of CV/MDL-chosen prompts vs. accuracy of the worst, average (randomly-selected) and best prompt (prior work). Right: The average accuracy gain from using CV/MDL-chosen prompts instead of randomly-chosen ones, relative to the gain from the best prompt. We plot mean/std. err. across runs with different training sets. CV/MDL-chosen prompts obtain only small improvements over randomly-chosen ones and do far worse than the best prompts.
single LM forward pass to compute the loss on each example conditioned on the previous examples. This feature makes MDL more computationally efficient than CV.
Marginalizing out example order The order of training examples impacts the generalization of LMs (7) so we treat order as random factor R that we marginalize over to evaluate the generalization of prompt A. We compute the exact ER, [CV (A, R, F)] and ER. MDLL(A, R. F)] by averaging over all !! training example orders. We use N examples to limit N!. We estimate the average test accuracy on N! 120 examples in LAMA, excluding the training examples, by evaluating on one test example per permutation of training examples. We compute CV, MDL, and test accuracy with Nọ 120 forward passes in total by appending a test example to each permutation of training examples, and we compute all selection criteria using the same set of N! 120 forward passes to maximize comparability across different methods. We show the test accuracy from CVMDL--chssen prompts, averaged over all relations. For comparison, (1) the best prompt, chosen using held-out accuracy as in prior work, (2) the worst prompt, as a lower bound, and (3) random prompts (we show the mean accuracy over all prompts).
show the test accuracy of always choosing
3.2 How well does prompt selection do in true few-shot learning?
Fig. (left) shows the results; prompt selection obtains marginal improvements over random selection across model sizes ranging over 3 orders of magnitude. Prompts chosen by CV and MDL alike underperform the best prompt (chosen using held-out performance) by 5-7% absolute on average. In fact, prompts chosen based on held-out performance often outperform larger models whose prompts are chosen in a true few-shot manner. CV and MDL do tend to choose better-than-aveer averrge prompts, but only close the gap between the average and best prompts by 20-40%, as shown in Fig. (right). We find similar results for several other tasks (.7) and selection criteria (Appendix ŞB).
Fig. (left) shows that CV/MDL struggle to choose the prompt with the highest test accuracy. Poor top-prompt selection is especially prevalent for larger models like GPT-3 175B that have spurred interest in prompt design (only % accuracy for CV vs. 9% for random chance). In Appendix &A we show that CVMDDL-chosen prompts differ from the best prompts not only in accuracy but also how effectively they transfer to models of different sizes. Altogether, our results show that optimal prompt selection difficult in the true few-shot setting, and that prior work overestimated the ability of LMs by using held-out examples for prompt selection.
3.3 How reliably does prompt selection improve over the average prompt?
If the average improvement from prompt selection is small, can we at least obtain an improvement with high probability for given task and train set? Fig. (left) shows that the worst prompts far worse than average, so it would be useful if prompt selection helped avoid the worst prompts. We
5

---

100
Prompt
CVa=
of "ime Cain Delow Thresirole
CVo=2
Chani
25-
CV
15-
at
Parameters
Threshold
Mean Prompt
Prompt
Figure 2: Left: CV/DLL have low accuracy at choosing the best prompt (mean/std. err. across runs with different train sets). Middle: The chance of various accuracy gains over the average prompt, when using prompts chosen by CV, and (Right) conservative estimates of CV that also minimize variance in CV; CV often chooses worse-than-average prompts, even with conservative selection.
examine the probability with which prompt selection obtains various accuracy gains over the average prompt and show results in Fig. 2 (middle) for CV (and similar results in Appendix $C for MDL),
CV/MDL-chosen prompts show high variance in test accuracy relative to the average prompt. For most model sizes (BB), the chance improving over the average, randomly-chosen prompt is only ~566 for CV and ~55% for MDL. The performance of prompt selection forms long-tailes tailed distribution; there is a ~27% chance that prompt selection causes an accuracy drop of ~ 13% for all model sizes and CV/MDL alike. Furthermore, the tails grow heavier model size increases. For the largest model (GPT-3 175B), CV/MDL-chosen prompts sometimes do far worse than average, e.g., 40% worse, 5% of the time. Our results suggest troubling trend: models grow bigger and generalize better, our ability to reliably choose good prompts degrades. One possible explanation that larger models have the capacity draw more complex decision boundaries, requiring more examples to estimate the true expected loss on unseen examples; we may need to scale validation sets along with model size. In $3.7 we show similar results for several other tasks. Overall, the limited average- case case gains from prompt selection cannot be expected with any reasonable confidence in the true few-shot setting, a problem that will only become worse with larger models.
3.4
Can we increase the likelihood of improved performance from prompt selection?
As we have shown, CV and MDL do not reliably choose better-than-average prompts. Here, we explore the extent to which we can reduce the variance in generalization by explicitly preferring prompts with low variance (2.3). For the largest model (GPT-3 175B), we choose prompts based on conservative estimate of generalization loss, CV, 2.3 We show the test accuracy for the prompt chosen with various levels confidence a {1,2,3) 3) and with CV (a = 0).
As shown in Fig. (right), all a lead to a similar distribution of performance gain as CV. For example, CV outperforms the average prompt 50% of the time vs. 51% for 2. These results suggest that it is non-trivial to choose prompts that reliably perform better than random selection, even when explicitly minimizing variance in generalization.
3.5 Does prompt selection improve with more labeled examples?
The poor performance of prompt selection methods may due to using such a small number of labeled examples. As the number of labeled examples increases, we expect prompt selection methods to improve. Thus, true few-shot prompt selection may be possible with a few dozen examples (though it is not always possible to use therefore examine the test accuracy of CV/MDL-chosen prompts as we use an increasing number of labeled examples N (5, 10, 15, 20, 30, 40). For N > 10, it is not feasible marginalize over all possible training example permutations, so we randomly sample 120 permutations (to match N = 5) each example
examples, due to limits input length for LMs like GPT). We
such that each example occurs the as the held-out CV fold the same number of times). We run the experiment for <..7B parameter models, since it is prohibitively costly
number of times in each position (i.e.,
with larger models via the OpenAl API.
As shown in Fig. there is no consistent trend in the performance of prompt selection, both in terms of task performance (left) and in terms of accuracy at choosing the highest accuracy prompt (right).

---

LOOCV
LOOCV
Relative Test Accuracy of
60
35.0
DGFT2 82M
Chasen Prompt (1%
Accuracy at Choesing
32.5
50
Best Promet (25
GPT2 345M
30.0
40
27.5
GPT2 1.58
GPT3 6.78
30
22.5
20.0
20
15
20
Number of Training Examples
Number of Training Examples
Figure 3: Increasing the number of examples up to 40 does not clearly improve CV in terms of (Left) accuracy gain over the average prompt (scaled 0), relative the best one (scaled to 100) or (Right) accuracy at choosing the best prompt. Mcan/std. err. on LAMA over runs (varying train sets).
Shot
Shot
30 Shot
100
Relaiiee lest Accurecy
Chosen Promitt (N3)
Nethot
NDL
LOOCV
Mean
20
Forward
Figure 4: For N E {5, 10,30} -shot learning, increasing the compute used to estimate CV/MDL does not notably improve the accuracy chosen prompts beyond a certain point forward pass for MDL, N forward passes for CV). Mean/std. err. across runs for GPT-3 6.7B.
Even in higher-data regimes (40 examples), CV/MDL struggle to choose effective prompts and do not consistently, across model sizes, perform better than choosing examples based on examples. Our findings surprising, because the true-few shot setting is where prompt design has been thought most promising, due to the scareity of training data [13]. However, the true few-shot setting is also one in which prompt selection is hardest, greatly undermining the potential value of prompts.
3.6 Does prompt selection improve with computation?
In the preceding sections, we computed E Eg,[[CV(A (CV R, F)] using fixed number of samples for R. Can improve prompt selection by using
samples, at the cost of increased computation? To answer this question, we vary the number samples R (and thus LM forward passes) used to compute the above expectation and choose prompts described in $2.3 To estimate CV with single forward pass, we sample single fold &e (here, a single example) and evaluate accuracy on fold k when conditioning the LM all others folds. Fig. 4 shows the results for N (5, 15, 30) training examples using the largest model from $3.5 (GPT-3 6.7B).
Computation is not the bottleneck in prompt selection, test accuracy roughly plateaus after one forward pass for MDL and N' forward passes for CV. This observation holds across N, as well as models with <6.7B parameters (omitted for space). Our results suggest that true few-shot prompt selection is fundamentally limited by the number of examples available.
3.7 Is Prompt Selection Challenging on Other Tasks?
We now examine the extent to which our results on LAMA tasks hold on other kinds of NLP tasks. We examine three classification tasks for which prior work has designed various prompts: Recognizing Textual Entailment (RTE), CommitmentBank (CB), and Word-in-Conter detecting whether one sentence entails or contradicts another, and WiC involves determining if polysemous word is used with the same sense in two sentences (e.g.. "Room and board" and "He nailed boards across the windows."), See Appendix $E.2 for further task details. We evaluate the accuracy of GPT models when using prompts chosen by CV, MDL, and test accuracy, as did
(WiC). RTE and CB involve
7

---

RTE
WIC
Selectio Herted
Test ACC, Chosen
53-
50
375
6.7 13 175
13 175
Model Paremeeters (B)
Model Parameters (B)
Model Parameters (B)
Figure 5: Accuracy of CV/MDL-chosen prompts vs. accuracy the worst, average (randomly- selected), and best prompt (prior work), on three classification tasks (mean/std. err. over runs). CV/MDL-chosen prompts generally perform several points worse than the best prompt and do not consistently improve over the average prompt across tasks and model sizes.
No Time Acc. Cain balow Threkkts
200
RTE
Wic
CB
Farame
0.08
175
Threshold for Acc. Gain over Mean Prompt Threshold for Acc. Gain over Mean Prompt Threshold for Acc. Gain over Mean Prompt
Figure 6: The chance of various accuracy gains the average prompt from CV RTE, WiC, and CB. CV often chooses prompts that are below average (RTE, WiC) or far below average (CB).
for LAMA. For each task, evaluate held-out accuracy using the full validation set when using 5 training examples randomly sampled from the task train set, while ensuring that we include least example per class. We evaluate the mean/std.
of prompts, we use the manually-written prompts from [2] and [9] 3 prompts for RTE/CB and prompts for WiC. Schick and Schütze [9] designed prompts for bidirectional LMs, when necessary, modify their prompts to be suitable for left-to-right LMs (see Appendix $E.2 for prompts). Fig. 5 shows the accuracy of the chosen prompts each task.
5 train
As
observe as similar trend as before, that across tasks and model sizes, the CV/MDL-chosen prompt almost always obtains lower average accuracy than choosing based on test accuracy. The trend holds even when choosing between fewer prompts (here, 3-4). CV/MDL-chosen prompts vary greatly in test accuracy across tasks and model sizes, often choosing worse-than--verrs averrge prompts (c.g.. CB).
We examine the variance in chosen prompt accuracy in more detail, by showing the chance that selection obtains various accuracy gains over the average prompt. Here, we choose prompts with CV using N forward passes (one evaluation per fold), as represents a good tradeoff between compute and accuracy that is likely be used in practice. As shown in Fig. 6 accuracy gains are again highly dispersed, often negative, and not consistently achieved. For CB, there a 20% change of a 15% accuracy drop for GPT-3 175B. Model sizes vary greatly in how often the CV-chosen prompt leads to improvement, c.g., from 38-82% for WiC and 1-83% for CB. Overall, our earlier findings carry over to other kinds of tasks, indicating that prompt selection is challenging in general.
4 True Few-Shot Hyperparameter Selection
Having shown that true few-shot prompt selection is difficult, we now study the effectiveness of model selection in the context of hyperparameter selection more generally. We examine the ADAPET model [12), as it is open-souree") and currently the top-performing few-shot model according to SuperGLUE [62] standard NLP benchmark. ADAPET finetunes the pretrained ALBERT xxlarge-12 LM 633 to (1) classify each label as correct/incorrect given the input and (2) to predict randomly

---

BoolQ
CB
COPA
RTE
WIC
WSC
MuhiRC
ReCoRD EM/FI
Avg
Ace
Acc/F1
Acc
Ace
Acc
Acc
EM/FI
Worst
79.5g.//67.37
76.822
63.24a
49.0r.a
77.21.8 8
38 //.50.
69.4
Mean
79.0,s
85.92.//4..511.
81.12.
70.82.5
51.51.
82,52.7
44.2a.//82.32.
78.31s/88..8o
73.912
MDL
7..5
85.75.o/74.81s.4
82.02.9
70.4a.5
52.2a.0
82.0g.1
39.7g. 1/80.6g..
78.9o./88.2o..
20.4
73.42.8
CV
78.9344
83.96.//99.22..
80.5a.a
68.77.0
51.11.6
83.12.
41.9/.2/11.4.1
78.71./88.122
73.02.1
Best
80.9.
89.8g.1/7..813
84.8..5
76.71.8
54.12.a
86.61.8
9/83.4.
80.41. 1/99.2.
77.26.9
ADAPET 12
80.3
89.3 86.8
89.0
76.5
54.4
81.7
39.2/ 80.1
85.4 92.1
77.3
iPET 9
80.6
92.9 /92.4
95.0
74.0
52.2
80.1
33.0/ 74.0
86.0 86.5
76.8
PET
9
79.4
85.1 59.44
95.0
69.8
52.4
80.1
$7.9/7773
86.0 86.5
74.1
GPT-3
77.5
82.1
92.0
72.9
55.3
75.0
74.8
89.0 90.1
73.2
Table 1: ADAPET results on SuperGLUE validation when choosing early stopping checkpoint and
masked LM rate using CV/MDL vs. the worst/mean/best hyperparameters chosen with validation (mean.u. dev. over four 32-shot train sets). On all tasks, CV/MDL-chosen hyperparameters perform similar to or worse than average, and several points below the best hyperparameters.
masked out imput tokens given the label and unmasked input tokens, similar to Masked LM 664 ADAPET was developed in the context of tuned few-shot learning, as ADAPET's hyperparameters were chosen using validation examples. We examine how ADAPET does in the true few-shot setting.
We choose two hyperparameters in a true few-shot manner: the early stopping checkpoint and fraction of words masked for the masked LM objective. ADAPET performs T 1000 gradient updates on batches of 16 examples and chooses the checkpoint at T E {250, 500, 750, 1000} with the highest validation accuracy. ADAPET also chooses the best masking fraction M E {0.075, 0.10, 0.105, 0.15]. Following ADAPET, we evaluate on SuperGLUE, a suite of 8 NLP tasks. SuperGLUE consists of four -nwweerin tasks (BoolQ. COPA, MultiRC, ReCoRD), coreference resolution task (WSC), two entailment detection tasks (RTE, CV), and a commonsense reasoning task (WiC); see Appendix I $ for task details. We use CV/MDL to choose T and M (out of 16 total combinations) and then train a model on the full dataset with the chosen T and M. We use FLwGLUE (9). the 32-example subset of SuperGLUEE used in prior work few-shot learning. We also use 3 other 32-example subsets that we randomly sample from SuperGLUE, to estimate variance in performance across training sets. ADAPET uses prompt during fine-tuning, choosing the prompt based on validation examples. To avoid using validation-tuned prompts, we use the first prompt for every task the authors do for ablation studies. Since training ADAPET is expensive, we evaluate We show results in Table
CV/MDL with K- = 8 folds [
Results Across all SuperGLUE tasks, CV/MDL hyperparameter selection performs similar to or worse than average (randomly-choo -hossn)) hyperparameters and several points worse than the best hyperparameters. In the true few-shot setting, the average SuperGLUE performance of ADAPET drops below that of earlier methods (PET and iPET), highlighting how the use of validation examples can give the false appearance of progress in few-shot learning. On MultiRC, CV/MDL choose hyperparameters that give similar performance to the worst hyperparameters, another indication that model selection methods do not consistently prevent worst-case behavior in the true few-shot setting. Preliminary analysis in Appendix suggesss that choosing better-than than-avera-aaeeee hyperparameters requires several thousand examples. Overall, our results indicate that it is not just prompt selection but model selection in general that is challenging in very low-data regimes.
5 Conclusion and Future Work
Our work shows that it is challenging to make even the most basic decisions about few-shot learning algorithms using only a few labeled examples. Instead, it may be more promising to make additional assumptions. The meta-learning setting assumes access to data from many other tasks in order to perform learning and model selection [20] 21 65 66). Transfer learning and multitask learning assume access data that is directly related to the task with limited data (67770). Data augmentation techniques assume there is a viable way to create more data from limited data [7TH74]. Other approaches assumee unlabeled data and develop unsupervised model selection techniques (75)77]. When labeled data is cheap, the simplest approach is to assume more examples for validation
'See Appendix $E.4 for details on how evaluate MDL different SuperGLUE tasks.

---

in which case we might be better off training on the additional examples. Unless we make such assumptions explicit, we cannot make meaningful comparisons between few-shot learning algorithms. We find the above avenues to be more promising future directions than true few-shot learning given the challenge of model selection.
Inspired by prior work [78-81) we offer recommendations for future work in true few-shot learning:
Report all hyperparameters (prompts) considered and the hyperparameter selection criteria.
Include validation examples in the number of examples used by a few-shot learning algorithm. Validation examples include all examples used to decide any aspect of learning: hyperparameters, prompts, training objectives, decoding strategies, model architecture, etc.
Once you have decided on the learning algorithm, submit your model for test evaluation directly, without first evaluating on validation. Report the total number of test evaluations conducted (ideally, just one). Use the validation set only after test evaluation for any ablations you report, to avoid making decisions about your algorithm with the validation set.
Do not rely on hyperparameters from prior work that were tuned using validation examples for the same benchmark (e.g., SuperGL.UE) avoid indirectly benefiting from validation examples. Instead, re-tune such hyperparameters using only the given few-shot examples.
The above protocols are striet but mimic how a true few-shot learning algorithm would be used in a real, low-data setting. To ensure researchers comply with such strict protocols, future benchmarks may need to keep large test sets private while releasing only a few labeled examples.
Given our negative results on true few-shot learning, a major question remains: is it possible to select models in a true zero-shot setting? Prior work uses LMs for zero-shot learning by choosing an arbitrary prompt [1 [2) which requires no data but is suboptimal [4]. Other efforts try multiple prompts and choose between them via trial and error alongside manual evaluation (.] effectively leveraging human supervision. CLIP [13] achieves high zero-shot accuracy on ImageNet after extensively tuning prompts and label names using ImageNet's training set (1 (1.28M examples), as we noticed from the open- open-source code [ The authors report 5% accuracy gain from tuning prompts alone, but the training examples used for tuning are not available in true zero-shot learning. Without any labeled data, the problem of model selection is even more challenging than in the true few-shot case. Overall, our work provides guidance for future work in few-shot learning by clarifying the assumptions made by the true few-shot setting and empirically demonstrates that model selection is major roadblock to true few-shot learning.
6 Limitations and Broader Impact
We facilitate fair comparisons between few-shot methods in future work by disambiguating between three few-shot settings: multi-distribution, tuned, and true few-shot learning. We highlight that one setting, tuned few-shot learning, gives up the practical advantage of few-shot learning by using many labeled examples. Furthermore, we show that several tuned few-shot learning algorithms work significantly worse in the true few-shot setting, without tuning on many examples. While we explored CV, MDL., and several other criteria (Appendix SB). it is possible that effective true few-shot model selection is possible using other criteria ($2.4) or heuristics not explored here. In this event, our work will have discouraged work on a few-shot learning setting with applications to low-data settings, e.g., that involve low-resource languages or expert annotation. Overall, however, we helieve our work will redirect future work to few-shot settings with more practical applications.
We show that it is hard to detect when a small input change hurts an LM's generalization, even when the change appears reasonable to human readers. We argue that practitioners will benefit from knowing such limitations, but they may also be discouraged from deploying LMs in many useful contexts, such as question-answwring hate speech detection, automatic translation, and commercial dialogue systems. Our findings may also encourage adversaries to target basd applications and highlight which models are most susceptible to attack (e.g., larger models). By shedding light on the shortcomings of (few-shot) LMs, we hope to spur future work to address these shortcomings.
httppp//ggihab...cooooem///////p httt::/thhhcaoeeee////o///ooooot ImageNet..pym
10

---

Acknowledgments
We are grateful to OpenAl for providing access and credits GPT-3 via the API Academic Access Program, and we thank Miles Brundage, David Schnurr, Felipe Such, Ryan Lowe, and Ilya Sutskever for help with the API. We thank GPT-3 authors Benjamin Mann and Gretchen Krueger for helpful feedback on our paper. We thank Rakesh Menon for assistance with the ADAPET codebase, Shenglong Wang for cluster support, Zhengbao Jiang for LPAQA prompts, and Tal Linzen, Patrick Lewis, Eric Wallace, Adam Fisch, Stephen Roller, Aravind Rajeswaran, Gretchen Krueger, Amanda Ngo, Udit Arora, Sébastian Jean, Jason Phang, and the NYU NLP group for feedback on our draft. KC partly supported by Samsung Advanced Institute of Technology (Next Generation Deep Learning: from pattern recognition to AI) and Samsung Research (Improving Deep Learning using Latent Structure). KC also thanks Naver, eBay, NVIDIA, and NSF Award 1922658 for support. EP is grateful to NSF and Open Philanthropy for fellowship support.
References
[1] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models unsupervised multitask learners, 2019.
[2] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
[3] Timo Schick and Hinrich Schütze. Exploiting cloze questions for few-shot text classification and natural language inference. Computing Research Repository, arXiv:2001.0761 07666,, 2020.
[4] Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. How we know what language models know? TACL, 11423-438, 2020.
[5] Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.1572) 2020.
[6] Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models, 2021.
[7] Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity, 2021.
[8] Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, L. Carin, and W. Chen. What makes good in-context examples for gpt-3? arXiv, abs/2101 01660044 2021.
[9] Timo Schick and Hinrich Schützze It's not just size that matters: Small language models are also few-shot learners. Computing Research Repository, arXiv:2009.07118, 2020.
[10] Ethan Perez, Douwe Kiela, and Kyunghyun Cho. Rissanen data analysis: Examining dataset characteristics via description length. In ICML, 2021.
[11] Timo Schick and H. Schutze. Few-shot text generation with pattern-exploiting training. arXiv, abs/2012.1 1926, 2020.
[12] Derek Tam, Rakesh R Menon, Mohit Bansal, Shashank Srivastava, and Colin Raffel. Improving and simplifying pattern exploiting training. arxiv preprint arXiv:2 /9555, 2021.
[13] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
[14] Sinong Wang, Han Fang, Madian Khabsa, Hanzi Mao, and Hao Ma. Entailment as few-shot learner, 2021.

---

[15] Teven Le Scao and Alexander M. Rush. How many data points is prompt worth?, 2021.
[16] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv, abs/1910.01 2019.
[17] Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In EMNLP, pages 2463-2473, Hong Kong, China, November 2019. ACL.
[18] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, koray kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, NeuRIPS, volume 29. Curran Associates, Inc., 2016.
[19] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In I. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, NeuRIPS, volume 30. Curran Associates, Inc., 2017.
[20] Ravi and Larochelle. Optimization as a model for few-shot learning, ICLR, 2017.
[21] Ke Li and Jitendra Malik. Learning to optimize. arXiv, abs/1666.01185, 2017.
[22] David M. Allen. The relationship between variable selection and data agumentation and method for prediction. Technometrics, 16(1):125-11) 1974.
[23] M. Stone. Cross-validatory choice and assessment of statistical predictions. Journal of the Royal Statistical Society. Series A (Methodological),
1974.
[24] Seymour Geisser. The predictive sample method with applications. Journal of the American Statistical Association, 70(350):3203322 1975.
[25] Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning. Springer Series in Statistics. Springer New York Inc., New York, NY, USA, 2001.
[26] Chelsea Finn, Pieter Abbecl, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, volume 70 of ICML'17, pages 1126-1135. JMLR.org. 2017.
[27] Aravind Rajeswaran, Chelsea Finn, Sham Kakade, and Sergey Levine. Meta-learning with implicit gradients. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, NeurIPS, volume 32. Curran Associates, Inc., 2019.
[28] J. Rissanen. Modeling by shortest data description. Automatica, 14(5):465 441,, 1978.
[29] Rissanen. Universal coding, information, prediction, and estimation. IEEE Transactions on Information Theory, 30(4):629-636, 1984.
[30] A. P. Dawid. Present position and potential developments: Some personal views: Statistical theory: The prequential approach. Journal of the Royal Statistical Society. Series (General), 147(2):278-292) 1984.
[31] Peter Grünwald. A tutorial introduction to the minimum description length principle. CoRR, math.ST/0400077, 06 2004.
[32] Léonard Blier and Yann Ollivier. The description length of deep learning models. S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi and R. Garnett, editors, NeuRIPS. volume 31, pages 2216-2226. Curran Associates, Inc., 2018.
[33] Alselm Blumer, Andrzej Ehrenfeucht, David Haussler, and Manfred K. Warmuth. Occam's razor. Inf. Process. Lett. 24(6):377-380, April 1987.
[34] Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. Masked language modeling and the distributional hypothesis: Order word matters pre-training for little. CoRR, abs/2104.06664 2021.
12

---

[35] P. J. Phillips, Hyeonjoon Moon, S. A. Rizvi, and P. Rauss. The feret evaluation methodology for face-recognition algorithms. TPAMI, 22(10):1090-1104 2000.
[36] Joy Buolamwini and Timnit Gebru. Gender shades:
Intersectional accuracy disparities in commercial gender classification. In Sorelle A. Friedler and Christo Wilson, editors. Fairness, Accountabiliity and Transparency, volume 81 of PMLR, pages 77-91, New York, NY, USA, 23-24 Feb 2018. PMLR.
[37] Peter Henderson, Koustuv Sinha, Nicolas Angelard- Nan Rosemary Ke, Genevieve Fried, Ryan Lowe, and Joelle Pineau. Ethical challenges data-driven dialogue systems. In AAAI/ACM Conference on AI, Ethics, and Society, AIES "18, page 123-129, New York, NY, USA, 2018. Association for Computing Machinery.
[38] Chandra Khatri, Behnam Hedayatmia, Anu Venkatesh, Jeff Nunn, Yi Pan, Qing Liu, Han Song, Anna Gottardi, Sanjeev Kwatra, Sanju Pancholi, Ming Cheng, Qinglang Chen, Lauren Stubel, Karthik Gopalakrishnan, Kate Bland, Raefer Gabriel, Arindam Mandal, Dilek Hakkani-Tite Gene Hwang, Nate Michel, Eric King. and Rohit Prasad. Advancing the state of the art in open domain dialog systems through the alexa prize. CoRR, abs/1812.10757, 2018.
[39] Javier García, Fern, and Fernández. comprehensive survey on safe reinforcement learning. JMLR, 16(42):1437-1480 2015.
[40] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul F. Christiano, John Schulman, and Dan Mané. Concrete problems in Al safety. CoRR, abs/1 600.06666..665, 2016.
[41] Sumio Watanabe. Asymptotic equivalence of bayes cross validation and widely applicable information criterion in singular learning theory. JMLR, 11(116)3571-3594 2010.
[42] H. Akaike. A new look at the statistical model identification. TACON, 19(6):166-72,, 1974.
[43] C. Mallows. Some comments on cp. Technometrics, 15(4))661-675, 1973.
[44] Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown. Sequential model-based optimization for general algorithm configuration. In Carlos A. Coello Coello, editor, Learning and Intelligent Optimization, pages 507-523, Berlin, Heidelberg, 2011. Springer Berlin Heidelberg.
[45] James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithmss for hyper- parameter optimization. In J. Shylor R. Zemel, P. Bartlett, F. Pereira, and K. Q. Weinberger. editors, NeuRIPS, volume 24. Curran Associates, Inc., 2011.
[46] Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. In F. Pereira, J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, NeuRIPS, volume 25. Curran Associates. Inc., 2012.
[47] Risto Mükkullinenn Jason Liang, Elliot Meyerson, Aditya Rawal, Daniel Fink, Olivier Francon, Bala Raju, Hormoz Shahrzad, Arshak Navruzyan, Nigel Duffy, and Babak Hodjat. Chapter 15 evolving deep neural networks. In Robert Kozma, Cesare Alippi, Yoonsuck Choe, and Francesco Carlo Morabito, editors, Artificial Intelligence in the Age of Neural Networks and Brain Computing, pages 293-312. Academic Press, 2019.
[48] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le. Regularized evolution for image classifier architecture search. AAAI, 43700))488044788 Jul. 2019.
[49] Barret Zoph and Quoe V. Le. Neural architecture search with reinforcement learning. In ICLR. OpenReview.net, 2017.
[50] J. Larsen, L.K. Hansen, C. Svarer, and M. Ohlsson. Design and regularization of neural networks: the optimal use of a validation set. In Neural Networks for Signal Processing VI. IEEE Signal Processing Society Workshop, pages 62-71, 1996.
[51] Yoshua Bengio. Gradient-Based Optimization of Hyperparameters. Neural Computation, 1899 08 2000.
[52] O. Chapelle, Vapnik, O. Bousquet, and S. Mukherjee, Choosing multiple parameters for support vector machines. Machine Learning, 16:131-159, 2004.
13

---

[53] Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In ICLR, 2019.
[54] Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. EMNLP, pages 4222-42355 Online, November 2020. ACL
[55] Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang. and Jie Tang. Gpt understands, too, 2021.
[56) Zexuan Zhong, Dan Friedman, and Danqi Chen. Factual probing is [MASK]] learning vs. learning to recall. CoRR, abs/2104.05240, 2021.
[57] Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. T-REx: A large scale alignment of natural language with knowledge base triples. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May 2018. European Language Resources Association (ELRA).
[58] Nina Poerner, Ulli Waltinger, and Hinrich Schütze. E-BERT: Efficient-yet-effeeeee entity embeddings for BERT. In Findings of EMNLP, pages 803-818, Online, November 2020. ACL.
[59] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jemnite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of- the-art natural language processing. In EMNLP: System Demonstrations pages 38 45, Online, October 2020. ACL.
[60] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf. Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett, editors, NeuRIPS, pages 8024-800 8035. Curran Associates, Inc.. 2019.
[61] Elena Voita and Ivan Titov. Information-theorrticc probing with minimum description length. In EMNLP. pages 183-196, Online, November 2020. ACL.
[62] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for gereral-purpose language understanding systems. In H. Wallach, Larochelle, A. Beygelzimer, d' d'Alché-Bun -ucc
E. Fox, and R. Garnett, editors, NeuRIPS, volume 32. Curran Associates, Inc., 2019.
[63] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for supe-supe learning of language representations. In ICLR, 2020.
[64] Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL, pages 4171-4186, Minneapolis, Minnesota, June 2019. ACL.
[65] Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evei, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle. Meta-dataset: A dataset of datasets for learning to learn from few examples. In ICLR, 2020.
[66] Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. Crossfit: A few-shot learning challenge for cross-task generalization NLP. CoRR, htss/104408885, 2021.
[67] Rich Caruana. Learning many related tasks at the same time with backpropagation. In
G. Tesauro, D. Touretzky, and T. Leen, editors, NeuRIPS, volume 7. MIT Press, 1995.
[68] Rich Caruana. Multitask learning, Machine Learning, 28(1):41-75, July 1997.
14

---

[69] Jason Phang, Thibault Févry, and Samuel R. Bowman. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. CoRR, abs/1811.01088, 2018.
[70] Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. In ACL, pages 4487-4496, Florence, Italy, July 2019, ACL.
[71] Vid Kocijan, Ana-Maria Cretu, Oana-Maria Camburu, Yordan Yordanov, and Thomas Lukasiewiez. A surprisingly robust trick for the Winograd schema challenge. In ACL, pages 4837-4842, Florence, Italy. July 2019. ACL.
[72] Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. In Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, NeuRIPS, volume 33, pages 6256-6268. Curran Associates, Inc., 2020.
[73] Jiaao Chen, Zichao Yang, and Diyi Yang. MixText: Linguisticalll--nformed interpolation of hidden space for semi-supervised text classification. In ACL, pages 2147-2157, Online, July 2020. ACL.
[74] Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ji- Ping Wang, Chandra Bhagavatula, Yejin Choi, and Doug Downey. Generative data augmentation for commonsense reasoning. In Findings of EMNLP, pages 1008-1025, Online, November 2020. ACL.
[75] Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. Unsupervised neural machine translation. In ICLR, 2018
[76] Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc' Aurelio Ranzato. Unsupervised machine translation using monolingual corpora only. In ICLR, 2018.
[77] Ethan Perez, Patrick Lewis, Wen-tau Yih, Kyunghyun Cho, and Douwe Kiela. Unsupervised question decomposition for question answering. In EMNLP, pages 8864-8880, Online, November 2020. ACL.
[78] Avital Oliver, Augustus Odena, Colin Raffel, Ekin D. Cubuk, and Ian J. Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. CoRR, abs/1804.09 804.09170, 2018.
[79] Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah Smith. Show your work: Improved reporting of experimental results. In EMNLP, pages 2185-2194, Hong Kong, China, November 2019. ACL.
[80] Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. A metric learning reality check, 2020.
[81] Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In International Conference on Learning Representations, 2021.
[82] Allyson Ettinger. What BERT is not: Lessons from new suite of psycholinguistic diagnostics for language models. TACL, 8:34 48, 2020.
[83] Aki Vehtari, Andrew Gelman, and Jonah Gabry. Practical bayesian model evaluation using leave- one-out cross-validation and waic. Statistics and Computing. 27(5):1413-1432 September 2017.
[84] Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah A. Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. CoRR, abs/2002.06305. 2020.
[85] Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In NAACI, pages 2924-2936, Minneapolis, Minnesota, June 2019. ACI.
[86] Marie Canre-Catherine de Marneffe, Mandy Simons, and Judith Tonhauser. The commitmentbank: Investigating projection in naturally occurring discourse. Sinn und Bedeutung, 1372)107 July 2019.
15

---

[87] Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. Joaquin Quiñionero- Candell,,, Ido Dagan, Bernardo Magnini, and Florence d' Alché Buc, editors, Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification and Recognising Tectual Entailment, pages 177-190, Berlin, Heidelberg, 2006. Springer Berlin Heidelberg.
[88] Roy Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second PASCAL recognising textual entailment challenge. In Second PASCAL Challenges Workshop Recognising Textual Entailment, 2006.
[89] Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third pascal recognizing textual entailment challenge. In ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, RTE '07, page 1-9, USA, 2007. ACI..
[90] Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. The fifth pascal recognizing textual entailment challenge. In TAC, 2009.
[91] Mohammad Taher Pilehvar and Jose Camacho-Collados, WiC: The word-in-context dataset for evaluating context-sensitive meaning representations. NAACL. ACL, 2019.
[92] Hector J. Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In KR, KR 12, page 552-561. AAAI Press, 2012.
[93] Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Looking beyond the surface: challenge set for reading comprehension over multiple sentences. In NAACL. ACL, 2018.
[94] Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. ReCoRD: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint 1810. 2885, 2018.
16

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you descrihe the limitations of your work? [Yes] See $
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See S
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] Included in the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We specify these details throughout our paper, since our work aims to investigate the importance of such details. We describe our data splitting procedure formally in $2 and dataset-specific details in (especialyy $3.1 and $ (c) Did you report error bars (e.g.. with respect to the random seed after running experiments multiple times)? [Yes] We show all plots and tables with error bars, except where it is non-trivial to plot error bars (the heatmaps in Fig. 7 or when we directly plot the distribution over errors (Fig. 2 middle/right).
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix $E.5
4. If you using existing assets (e.g., code, data, models) or curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See $3.1 3.7 and
Appendix $ citations for the dataset, prompts, and models we use for experiments
in $3 See $4 and Appendix SEI for citations for the dataset and model we use for experiments in $4
(b) Did you mention the license of the assets? [Yes] See Appendix
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes] We include the code reproduce all experiments as supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] We only use publicly-released datasets that allow for use in research context.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [Yes] See Appendix $E.33
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
17


Reviews:
[{"review_id": "ynN7I4jfHgD", "reviewer": "Reviewer_xPG6", "summary": "This paper studies few-shot learning within pre-trained language models. It notes that previous work has shown few-shot learning ability within these models but used many held-out examples to tune parts of learning, including determining what \"prompts\" to use to maximize few-shot performance. This paper considers \"true few-shot learning\", which involves only using the actual few-shot examples to tune these aspects of learning. The authors propose two ways to do this: (1) Cross-validation (CV) - randomly partitioning the training set into equally-size K folds and training on the k-1 sets and using the average loss on the kth set; (2) Minimum description length (MDL) - which is similar in spirit to (1) but defines different way of splitting the train and validation sets. The authors first conduct experiments on LAMA benchmark, which measure the ability of language models to retrieve facts. On this benchmark, it is shown that prompt selection via CV and MDL obtain marginal improvements over randomly picking the prompt. Another set of experiments is conducted on SuperGLUE benchmark where this time hyperparameter selection is done via CV and MDL. The pattern is similar to the first experiment and additionally it is noted that hyperparameter selection done this way drops the SuperGLUE performance below that of other work that used additional validation examples.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: interesting work that attempts to define what true few-shot learning with language models should actually look like and proposes methods that fit this definition and benchmarks them. As far as I'm aware, this type of work has not been pursued before.\n\nQuality: the problem and methods are well-defined. The experimental results seem to back up the hypothesis that true few-shot learning without using the information from held-out examples is still difficult for pre-trained language models.\n\nClarity: the paper is well-written. The problem is well-motivated and the methods are clearly defined. The experimental section is also very clear and the experiments measure interesting aspects of the discussed problem.\n\nSignificance: this paper could be impactful as it questions some of the assumptions of previous work showing few-shot learning in pre-trained language models and proposes best practices for future work for benchmarking few-shot learning in this setting.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "vXzBqFFLbYH", "reviewer": "Reviewer_5Pe4", "summary": "This paper evaluates prompt-based approaches with pretrained language models (LMs) for few-shot learning, where limited training examples are given and no validation set is provided. This setup is named “true few-shot learning” in the paper and is distinguished from multi-distribution and tuned few-shot learning, where data from other similar tasks and a large validation set is available, respectively. Under the true few-shot learning setup, model selection can only reply on the few training examples. \n\nExperimental results show that, when only relying on the few training examples, model selection using methods like cross validation (CV) and minimal description length (MDL) often fail to select the optimal prompts and hyperparameters. As a result, performance acquired by using CV and MDL is only slightly better than random model selection and is significantly worse than selection based on a held-out validation set. \n\nOverall, this work shows that the few-shot ability of pretrained LMs is overestimated for true few shot learning, where model selection is fundamentally difficult and needs to be addressed in future work. Moreover, this work provides suggestions for fair comparison on few-shot learning. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper criticizes recent advances made in prompt-based few-shot learning with pretrained LMs, arguing that the few-shot ability of pretrained LMs is overestimated for true few shot learning problems. The paper proves this by experimental results on various tasks showing the few-shot learning performance drops significantly when no validation set is available. \n\nThe paper is well-written and clear. The experiments are well-designed, and the results supports the paper’s argument well. The results and suggestions are informative and inspiring for future work. For example, future work should be explicit about the assumptions made for few-shot learning, i.e., what data is available besides the few-shot training examples. Also, future work should indicate clearly all how the model selection is performed.  \n\nHowever, the contribution of this work is limited by its scope. The results for true few-shot learning are interesting but are not widely applicable. Because most previous work, including those being criticized, focus on the multi-distribution or tuned few-shot learning settings and make assumptions like a validation set is available or optimal hyperparameters for other tasks could be re-used for the new few-shot tasks. Indeed, future works should be explicit about their model selection method to make fair comparison. But this has been a well-known issue for few-shot learning under additional assumptions. For example, researchers in computer vision have been working on datasets with standard splits for fair comparison [1]. The true few-shot learning setup might not be entirely impactful.\n\n[1] Triantafillou, Eleni, et al. \"Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples.\" International Conference on Learning Representations. 2019.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "(1) The true few-shot learning setting is good for showing the few-shot learning ability of pretrained LM without any additional information except the few training examples. This is true since the few-shot performance of LMs is much better than random guess, although being suboptimal compared to SOTA. \n\nAs discussed above, much of the work criticized by the paper is not designed for true few-shot learning, but for more practical settings like multi-dimension or tuned few-shot learning. We do need a better comparison framework for multi-distribution and tuned few-shot learning, e.g., specifying which external tasks can be used for training/validation and how many validation data can be used. For example, Tam et al. [1] assumes full access to development set while Schick and Schütze [2] didn’t make such assumption and thus the comparison between them is unfair. But this is a well-known issue for few shot learning or machine learning in general. \n\n\n(2)\nThe paper also investigates whether the results for LAMA also hold for other classification tasks.\nIn the LAMA benchmark, the LMs are asked to predict the answer out of 20k candidates given a sentence. But for regular classification tasks, the number of candidates is often much smaller, i.e., only 2 for binary review classification (negative/positive). Also, when talking about 5-shot classification tasks, people often mean 5 examples per class. But in the Appendix B, only 5 examples in total are given. It is thus unsurprising that CV and MDL will fail in this case. How about using K shots per class for classification tasks and making sure each fold for validation contains examples of all classes? Will the conclusion still hold?\n\n[1] Derek Tam, Rakesh R Menon, Mohit Bansal, Shashank Srivastava, and Colin Raffel. Improving and simplifying pattern exploiting training. arxiv preprint arXiv:2103.11955, 2021.\n[2] Timo Schick and Hinrich Schütze. Exploiting cloze questions for few-shot text classification and natural language inference. Computing Research Repository, arXiv:2001.07676, 2020.\n", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "gGwGyc4iVK", "reviewer": "Reviewer_Wsax", "summary": "- The paper studies few-shot learning on language models.\n- The paper first highlights the fact that many existing works on few-shot learning rely on a large validation set, which is unrealistic for a few-shot setting, and the authors propose a “true few-shot setting,” in which there are no held-out examples and there are a few points for training and validation combined.\n- They compare the two settings, specifically comparing two types of approaches for prompt selection and hyperparameter tuning: using a held-out validation set vs. using K-fold cross-validation (or maximum description length) for the true few-shot setting.\n- They show that K-fold CV doesn’t perform much better than random selection of prompts and hyperparameters, and underperforms the existing approach with a held-out validation set. \n- They argue that existing literature substantially overestimates the few-shot capabilities of language models.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- Overall, I think the paper highlights an important consideration for studying few-shot settings. The fact that existing few-shot methods rely on large validation sets is unrealistic as authors point out, but often overlooked.\n- The experiments show that LMs indeed perform better with the held-out validation set than in the true few-shot learning setting across multiple models, two datasets, etc. The experiments nicely demonstrate that it’s important to consider the proposed setting to appropriately measure LM performance when there are only a few labeled examples available.\n- One limitation of the experiment is the over-emphasis on the comparison with random selection of prompts and hyperparameters. This would be appropriate for the purposes of measuring the effectiveness of cross validation (or maximum description length), but not so much for the purposes of measuring the effect of not having a held-out validation set, which I thought was the main point of the paper. The authors include comparisons with tuning on a held-out validation set for most, but not all experiments. I think it’d be helpful to include those comparisons for all experiments and lean more on these comparisons over comparisons with random baselines. In particular, could you provide the results for Figure 3?\n- My main comment is about the high-level claim that the existing literature “greatly overestimates” the few-shot capabilities of LMs. While it’s subjective what ‘greatly overestimates’ means, I think the claim perhaps comes off as a bit too strong given the empirical results. In Figure 1, even though there are noticeable performance gaps between the true few-shot settings and the setting with a large held-out validation set, the models seem to perform reasonably well in the few-shot settings, regardless of prompt selection procedure. The presence of a held-out validation set doesn’t make or break the model performance, and model size seems to affect the performance more than the prompt selection procedure. My suggestion would be to update the abstract and introduction to tone things down. \n\nMinor comments\n- One limitation of the experiments is that the paper has one dataset for each of prompt selection (with a fixed pool of prompts) and hyperparameter tuning. I think showing consistent results across datasets would make the results stronger. \n- The paper is missing a discussion of related work. In particular, are there works in the few-shot literature that explores the “true few-shot” setting? I think the point raised in the paper are still good to highlight regardless, but it may be good to mention that some works already consider the “true few-shot” setting if that’s the case.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "gA20t1V0hXG", "reviewer": "Reviewer_V5jm", "summary": "The paper proposes to re-evaluate some recent LM-based few-shot learners like GPT-3 and PET without using any additional validation examples from the target task during evaluation. They term this as “true few-shot learning”. The paper proposes to use two approaches to evaluate without any validation examples: (1) leave-one-out CV (2) minimum description length. These methods are evaluated for few-shot prompt tuning as well as model selection and authors find that previous work often over-claimed their performance. For example, GPT-3 prompt tuning leads to much worse accuracy in LAMA than selecting the prompt based on full validation data (the approach used by GPT-3). Similarly, a recent variation over PET, called ADAPET, used full validation without which there is actually no improvement over the PET method which it claimed to improve over.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main finding of the paper is that both of these validation methods are only marginally better than random selection for both prompt tuning and model selection. This is also significantly worse than selection based on the full validation data (with thousands of examples), which was the strategy used in “popular” few-shot learning papers like GPT-3. The results are indeed what one would expect but I think it is really important to make this point in order to avoid future research continuing to make such poor validation practices for few-shot learning. The results in the paper cover many aspects of validation, albeit restricted to only these two simple choices, and yield analysis that can potentially inform experimentation for few-shot learning in the future. I believe the paper will have a stronger impact if they experimented with a few additional methods (see below) and also use some additional tasks for their prompt tuning experiments so that the analysis is more generally applicable.\n\nConcerns / Questions\n- The choice of LAMA for prompt tuning: While I understand that the authors need a large collection of prompts to be able to reliably experiment with prompt selection, I am not sure how much inference we can draw from the experiments on this task for other few-shot tasks like QA, NLI, classification. As I understand it, the LAMA task is more a test of how much factual knowledge a pre-trained model retains as opposed to being a test for its few-shot abilities. For instance, I don’t think there is a significant difference between zero-shot vs few-shot on LAMA. It will be useful to see results for prompt tuning on other classification or QA tasks, for example from SuperGLUE, where the tasks require quite a different reasoning than LAMA.\n\n- Is leave-one-out the most appropriate cross-validation strategy here in few-shot experiments? A related question is are the few-shot datasets balanced? This might not affect LAMA (also another reason why LAMA is so different than other few-shot tasks) but it will other classification tasks like SuperGLUE experiments. In meta-learning literature shots usually mean the number of examples per class. So leave-one-out will leave out one example of one class and make the few-shot support data imbalanced, which can, in turn, prefer hyper-parameters that favor the minority class. The more appropriate thing to do would be leave-m-out where m is the number of classes. It will actually be useful to have an ablation over k for leave-k-out and in particular, I feel this might have more of a bearing on the experiments in section 3.5. In case the few-shot data is not balanced, which is not standard, then that introduces a lot of additional variability from unbalanced datasets (or choice of the few-shot examples/classes) that also confound the results.\n\n- Another relevant approach to model selection is to use some additional, held-out, tasks to find model hyper-parameters and reuse them on all other tasks without any additional tuning. I think this setting is termed as multi-distribution few-shot learning in section 2 and is also a reasonable strategy to select hyper-parameters. This is a common approach in meta-learning, for instance where one has held out validation classes. This was also previously discussed by [1] that proposed to use validation languages for evaluation of low-resource language transfer and [2] used a similar approach for few-shot classification using validation tasks. While this may not always be possible for prompt-based methods, it is suitable for other methods like those in section 4. It will be useful to include this setting in the experiments in section 4.\n\nDiscussion of related works [1, 2] and their validation approach should be included. \n\n[1] Towards Realistic Practices In Low-Resource Natural Language Processing: The Development Set  \n[2] Self-Supervised Meta-Learning for Few-Shot Natural Language Classification Tasks\n\nOverall I really like the problem that the paper is bringing forward and I think this will be very useful in the future. My current score is mainly due to some of the above concerns which I hope the rebuttal will resolve.\nI understand doing all experiments can be difficult in the rebuttal timeframe but please respond to the best of your ability and I will be happy to revise my score.\n\n\nUpdate post rebuttal\n\nThe authors' response resolved many of my concerns and I have updated the score accordingly. I think it will be good to bring forth some of the additional experiments on SuperGlue in Appendix to the main paper and also include the results on multi-distribution validation provided in the rebuttal. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Adequately discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper focuses on an important shortcoming of the testing of and model selection in of language models for few-shot learning. The authors and reviewers engaged in a good discussion. Overall, the reviewers liked the paper giving above acceptance scores.

This is a timely contribution that can help improve the practice in few-shot language modelling. This paper will probably not benefit much from further revisions. 

Therefore, accept is recommended.
Decision: Accept (Poster)