True Few-Shot Learning with Language Models

Abstract: Pretrained language models (LMs) perform well on many tasks even when learning from a few examples, but prior work uses many held-out examples to tune various aspects of learning, such as hyperparameters, training objectives, and natural language templates ("prompts"). Here, we evaluate the few-shot ability of LMs when such held-out examples are unavailable, a setting we call true few-shot learning. We test two model selection criteria, cross-validation and minimum description length, for choosing LM prompts and hyperparameters in the true few-shot setting. On average, both marginally outperform random selection and greatly underperform selection based on held-out examples. Moreover, selection criteria often prefer models that perform significantly worse than randomly-selected ones. We find similar results even when taking into account our uncertainty in a model's true performance during selection, as well as when varying the amount of computation and number of examples used for selection. Overall, our findings suggest that prior work significantly overestimated the true few-shot ability of LMs given the difficulty of few-shot model selection.


Reviews and author responses:

 Thank you for the response. I have updated my score.

The paper proposes to re-evaluate some recent LM-based few-shot learners like GPT-3 and PET without using any additional validation examples from the target task during evaluation. They term this as “true few-shot learning”. The paper proposes to use two approaches to evaluate without any validation examples: (1) leave-one-out CV (2) minimum description length. These methods are evaluated for few-shot prompt tuning as well as model selection and authors find that previous work often over-claimed their performance. For example, GPT-3 prompt tuning leads to much worse accuracy in LAMA than selecting the prompt based on full validation data (the approach used by GPT-3). Similarly, a recent variation over PET, called ADAPET, used full validation without which there is actually no improvement over the PET method which it claimed to improve over.  The main finding of the paper is that both of these validation methods are only marginally better than random selection for both prompt tuning and model selection. This is also significantly worse than selection based on the full validation data (with thousands of examples), which was the strategy used in “popular” few-shot learning papers like GPT-3. The results are indeed what one would expect but I think it is really important to make this point in order to avoid future research continuing to make such poor validation practices for few-shot learning. The results in the paper cover many aspects of validation, albeit restricted to only these two simple choices, and yield analysis that can potentially inform experimentation for few-shot learning in the future. I believe the paper will have a stronger impact if they experimented with a few additional methods (see below) and also use some additional tasks for their prompt tuning experiments so that the analysis is more generally applicable.

Concerns / Questions
- The choice of LAMA for prompt tuning: While I understand that the authors need a large collection of prompts to be able to reliably experiment with prompt selection, I am not sure how much inference we can draw from the experiments on this task for other few-shot tasks like QA, NLI, classification. As I understand it, the LAMA task is more a test of how much factual knowledge a pre-trained model retains as opposed to being a test for its few-shot abilities. For instance, I don’t think there is a significant difference between zero-shot vs few-shot on LAMA. It will be useful to see results for prompt tuning on other classification or QA tasks, for example from SuperGLUE, where the tasks require quite a different reasoning than LAMA.

- Is leave-one-out the most appropriate cross-validation strategy here in few-shot experiments? A related question is are the few-shot datasets balanced? This might not affect LAMA (also another reason why LAMA is so different than other few-shot tasks) but it will other classification tasks like SuperGLUE experiments. In meta-learning literature shots usually mean the number of examples per class. So leave-one-out will leave out one example of one class and make the few-shot support data imbalanced, which can, in turn, prefer hyper-parameters that favor the minority class. The more appropriate thing to do would be leave-m-out where m is the number of classes. It will actually be useful to have an ablation over k for leave-k-out and in particular, I feel this might have more of a bearing on the experiments in section 3.5. In case the few-shot data is not balanced, which is not standard, then that introduces a lot of additional variability from unbalanced datasets (or choice of the few-shot examples/classes) that also confound the results.

- Another relevant approach to model selection is to use some additional, held-out, tasks to find model hyper-parameters and reuse them on all other tasks without any additional tuning. I think this setting is termed as multi-distribution few-shot learning in section 2 and is also a reasonable strategy to select hyper-parameters. This is a common approach in meta-learning, for instance where one has held out validation classes. This was also previously discussed by [1] that proposed to use validation languages for evaluation of low-resource language transfer and [2] used a similar approach for few-shot classification using validation tasks. While this may not always be possible for prompt-based methods, it is suitable for other methods like those in section 4. It will be useful to include this setting in the experiments in section 4.

Discussion of related works [1, 2] and their validation approach should be included. 

[1] Towards Realistic Practices In Low-Resource Natural Language Processing: The Development Set  
[2] Self-Supervised Meta-Learning for Few-Shot Natural Language Classification Tasks

Overall I really like the problem that the paper is bringing forward and I think this will be very useful in the future. My current score is mainly due to some of the above concerns which I hope the rebuttal will resolve.
I understand doing all experiments can be difficult in the rebuttal timeframe but please respond to the best of your ability and I will be happy to revise my score.


Update post rebuttal

The authors' response resolved many of my concerns and I have updated the score accordingly. I think it will be good to bring forth some of the additional experiments on SuperGlue in Appendix to the main paper and also include the results on multi-distribution validation provided in the rebuttal. 
 Adequately discussed.

 Hello, we just wanted to follow-up on our additional results, given that you mentioned that you would be happy to raise your score after seeing the results for the experiments presented above. We are happy to answer any questions you might have about these additional experiments as well.

Thanks for your time reviewing our work!

 I've updated my score. 

- The paper studies few-shot learning on language models.
- The paper first highlights the fact that many existing works on few-shot learning rely on a large validation set, which is unrealistic for a few-shot setting, and the authors propose a “true few-shot setting,” in which there are no held-out examples and there are a few points for training and validation combined.
- They compare the two settings, specifically comparing two types of approaches for prompt selection and hyperparameter tuning: using a held-out validation set vs. using K-fold cross-validation (or maximum description length) for the true few-shot setting.
- They show that K-fold CV doesn’t perform much better than random selection of prompts and hyperparameters, and underperforms the existing approach with a held-out validation set. 
- They argue that existing literature substantially overestimates the few-shot capabilities of language models.
  - Overall, I think the paper highlights an important consideration for studying few-shot settings. The fact that existing few-shot methods rely on large validation sets is unrealistic as authors point out, but often overlooked.
- The experiments show that LMs indeed perform better with the held-out validation set than in the true few-shot learning setting across multiple models, two datasets, etc. The experiments nicely demonstrate that it’s important to consider the proposed setting to appropriately measure LM performance when there are only a few labeled examples available.
- One limitation of the experiment is the over-emphasis on the comparison with random selection of prompts and hyperparameters. This would be appropriate for the purposes of measuring the effectiveness of cross validation (or maximum description length), but not so much for the purposes of measuring the effect of not having a held-out validation set, which I thought was the main point of the paper. The authors include comparisons with tuning on a held-out validation set for most, but not all experiments. I think it’d be helpful to include those comparisons for all experiments and lean more on these comparisons over comparisons with random baselines. In particular, could you provide the results for Figure 3?
- My main comment is about the high-level claim that the existing literature “greatly overestimates” the few-shot capabilities of LMs. While it’s subjective what ‘greatly overestimates’ means, I think the claim perhaps comes off as a bit too strong given the empirical results. In Figure 1, even though there are noticeable performance gaps between the true few-shot settings and the setting with a large held-out validation set, the models seem to perform reasonably well in the few-shot settings, regardless of prompt selection procedure. The presence of a held-out validation set doesn’t make or break the model performance, and model size seems to affect the performance more than the prompt selection procedure. My suggestion would be to update the abstract and introduction to tone things down. 

Minor comments
- One limitation of the experiments is that the paper has one dataset for each of prompt selection (with a fixed pool of prompts) and hyperparameter tuning. I think showing consistent results across datasets would make the results stronger. 
- The paper is missing a discussion of related work. In particular, are there works in the few-shot literature that explores the “true few-shot” setting? I think the point raised in the paper are still good to highlight regardless, but it may be good to mention that some works already consider the “true few-shot” setting if that’s the case. Yes

 Thank you for the additional detail and responses. 

 You AC

 Your AC

 We are glad that you liked our work overall and found our results to be really important in informing future work. Our rebuttal and Appendix sections B&C include all of the additional results you asked for, so we kindly ask that you consider revising your score after reading the results. We will bring additional results into the main paper with the extra page allowed for the camera-ready.

> “I believe the paper will have a stronger impact if they experimented with a few additional methods”

Please refer to Appendix C, where we show similar results across 6 model selection methods.


> “use some additional tasks for their prompt tuning experiments so that the analysis is more generally applicable.”

Please refer to Appendix B, where we show similar results on several SuperGLUE tasks.


> “As I understand it, the LAMA task is more a test of how much factual knowledge a pre-trained model retains as opposed to being a test for its few-shot abilities. For instance, I don’t think there is a significant difference between zero-shot vs few-shot on LAMA.”

GPT3 175B achieves 22% accuracy on LAMA in the zero-shot setting vs. 54% in the 5-shot setting, suggesting that language models do benefit from few-shot examples on LAMA. We will include this result in the camera-ready.


> “It will be useful to see results for prompt tuning on other classification or QA tasks, for example from SuperGLUE, where the tasks require quite a different reasoning than LAMA.”

We show similar results on three SuperGLUE tasks in Appendix B.


> “Discussion of related works [1, 2] and their validation approach should be included.”

Thanks for the references. We will add discussion to our paper.


> “are the few-shot datasets balanced?”

Our results hold across both balanced datasets (LAMA and CommitmentBank) and unbalanced ones (RTE and Winogrande-in-Context).


> “leave-one-out will leave out one example of one class and make the few-shot support data imbalanced... The more appropriate thing to do would be leave-m-out where m is the number of classes.”

In true few-shot learning, the true prior over class labels is not known, so we cannot use model selection criteria that rely on knowledge about the true prior (as in the leave-m-out selection scheme, which relies on knowing that the test distribution is balanced).


> “It will actually be useful to have an ablation over k for leave-k-out”

Thank you for the suggestion! We choose prompts using leave-k-out CV and show in [this plot](https://ibb.co/2633BPC) the accuracy gain over the average, randomly-chosen prompt (scaled to 0%) relative to the best one (scaled to 100%). All values of k perform similar to or worse than LOOCV (k=1), suggesting that our findings are not specific to LOOCV.

> “In case the few-shot data is not balanced, which is not standard, then that introduces a lot of additional variability from unbalanced datasets (or choice of the few-shot examples/classes) that also confound the results.”

Using imbalanced few-shot data is standard in prior work on few-shot learning with language models (GPT3 [[1](https://papers.nips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html)] and FewGLUE [[2](https://aclanthology.org/2021.naacl-main.185/)]), whose protocol we followed, where examples are sampled randomly regardless of class label. Indeed, most real-world datasets are not balanced in class distribution. Furthermore, there is no way to guarantee that the given set of few-shot examples has the same prior distribution over labels as the true distribution (despite artificially being the case in meta-learning research datasets).


> “Another relevant approach to model selection is to use some additional, held-out, tasks to find model hyper-parameters and reuse them on all other tasks without any additional tuning... It will be useful to include this setting in the experiments in section 4.”

Thank you for the suggestion! We ran the experiment you suggested, by choosing the hyperparameters for a target dataset in SuperGLUE using the average validation set performance on all 7 other SuperGLUE datasets (i.e., excluding the target dataset). The results of this “multi-distribution” hyperparameter selection are shown in the “Multi” row in [this table](https://ibb.co/VB1jjX0). As with CV/MDL, the performance is similar to that of random hyperparameter selection (“Mean”), suggesting that optimal hyperparameter selection for ADAPET requires many labeled validation examples for the specific target dataset.


[1] Language Models are Few-shot Learners. Brown et al. [NeurIPS 2020](https://papers.nips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html).

[2] Small Language Models Are Also Few-Shot Learners. Schick and Schütze. [NAACL 2021](https://aclanthology.org/2021.naacl-main.185/).


 We are glad that you found our experiments well-designed and our results inspiring for future work. We address your comments and questions below:

> “The results for true few-shot learning are interesting but are not widely applicable. Because most previous work, including those being criticized, focus on the multi-distribution or tuned few-shot learning settings”

The point of our work is to highlight that tuned “few-shot” learning is not actually few-shot learning at all (lines 65-75), and that, without a validation set, the performance of these learning algorithms drops significantly. Thus, we believe that our work has major implications for prior work on tuned few-shot learning - that the practical use of such methods has been overestimated.


> “We do need a better comparison framework for multi-distribution and tuned few-shot learning, e.g., specifying which external tasks can be used for training/validation and how many validation data can be used... But this is a well-known issue for few-shot learning or machine learning in general.”

We agree this issue is well-known (and indeed taught in introductory ML courses), but we do not believe it was thought of as important. Many of the papers we discuss are highly regarded in the ML community, such as GPT3 [[1](https://papers.nips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html); NeurIPS 2020 Best Paper], PET [[2](https://aclanthology.org/2021.naacl-main.185/); NAACL 2021 Best Short Paper], CLIP [[3](https://arxiv.org/abs/2103.00020)], and many more, with little discussion about this major methodological flaw (in reviews, in other papers, or publicly). We quantify the impact of this methodological flaw on the results of prior work to highlight its importance. Since the time of submission, our work has already been cited as guiding the experimental methodology of several recent papers and the development of a few-shot NLP benchmark whose authors did not release validation data (citations omitted to preserve anonymity).

> “when talking about 5-shot classification tasks, people often mean 5 examples per class. But in the Appendix B, only 5 examples in total are given.”

We followed the experimental protocol from prior work on few-shot learning with language models (GPT3 [[1](https://papers.nips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html)] and FewGLUE [[2](https://aclanthology.org/2021.naacl-main.185/)]), where examples are sampled randomly regardless of class label. We believe that this approach is realistic, because data annotation typically involves labeling IID examples, regardless of class label (which is not available before annotation). Prior datasets in multi-distribution few-shot learning often use N-examples per class during training in order to match the balanced nature of the test distribution. However, this dataset construction approach leverages knowledge about the prior distribution over test labels, which is not available in true few-shot learning.

[1] Language Models are Few-shot Learners. Brown et al. [NeurIPS 2020](https://papers.nips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html).

[2] Small Language Models Are Also Few-Shot Learners. Schick and Schütze. [NAACL 2021](https://aclanthology.org/2021.naacl-main.185/).

[3] Learning Transferable Visual Models From Natural Language Supervision. Radford et al. [arXiv 2021](https://arxiv.org/abs/2103.00020).


 We are glad that you found our work to nicely demonstrate an important and overlooked consideration. Below we clarify a few misunderstandings which we believe will help you view our work in a more positive light:

>“The authors include comparisons with tuning on a held-out validation set for most, but not all experiments.... could you provide the results for Figure 3?”

Several figures implicitly show the results for prompt selection based on held-out examples. In Figure 1 (Right) and Figure 3 (Left), we show the accuracy of CV-chosen prompts relative to held-out selection (which is scaled to have a value of 100%, outside the range pictured on the graph). Similarly, in Figure 2 (Left) and Figure 3 (Right), the accuracy for prompt selection based on held-out selection is also 100% (though not explicitly pictured on the graph). We will clarify this point in the camera-ready, to further emphasize the gap in performance against held-out selection as you suggested.


> “My suggestion would be to update the abstract and introduction to tone things down.”

Thank you for the feedback. We are happy to do so, and we will rephrase statements that came across as subjective.


> "The presence of a held-out validation set doesn’t make or break the model performance”

Figure 2 (middle) shows that the presence of a held-out set often does make or break model performance. In lines 205-210, we discuss how, for several tasks, prompt selection without a held-out set performs >40% worse than the best or even average prompt. Figure 1 (left) which you referenced shows the average performance across 41 different tasks, rather than the percentage of the time that the held-out set makes-or-breaks performance (as shown in Figure 2 middle).


> “the paper has one dataset for each of prompt selection (with a fixed pool of prompts) and hyperparameter tuning.”

For prompt selection, please refer to Appendix B for similar findings on three additional datasets, each with a distinct pool of prompts. We will bring these results into the main paper with the extra page allowed for the camera-ready. In the main paper, we showed results on a suite of 41 tasks, each with a distinct pool of prompts (5-16 per task); for examples of each task, please refer to Table 3 in the original LAMA paper ([Petroni et al. 2019](https://arxiv.org/abs/1909.01066)). For hyperparameter selection, we showed results on 8 distinct datasets. See Appendix F.2 for a description of how the datasets test diverse language capabilities.


> ‘The paper is missing a discussion of related work. In particular, are there works in the few-shot literature that explores the “true few-shot” setting? I think the point raised in the paper are still good to highlight regardless, but it may be good to mention that some works already consider the “true few-shot” setting if that’s the case.’

We discuss related work extensively throughout our paper, especially in Sections 1, 2, and 5, so we did not find it necessary to explicitly include a separate related work section. Lines 65-86 discuss prior attempts at true few-shot learning.


We hope you will consider revising your score given the above clarifications, which we believe concretely address your main reservations about our work.


 We are glad you found our work interesting, well-motivated, and impactful, and we are happy to answer any question you might have during the rebuttal period.

This paper studies few-shot learning within pre-trained language models. It notes that previous work has shown few-shot learning ability within these models but used many held-out examples to tune parts of learning, including determining what "prompts" to use to maximize few-shot performance. This paper considers "true few-shot learning", which involves only using the actual few-shot examples to tune these aspects of learning. The authors propose two ways to do this: (1) Cross-validation (CV) - randomly partitioning the training set into equally-size K folds and training on the k-1 sets and using the average loss on the kth set; (2) Minimum description length (MDL) - which is similar in spirit to (1) but defines different way of splitting the train and validation sets. The authors first conduct experiments on LAMA benchmark, which measure the ability of language models to retrieve facts. On this benchmark, it is shown that prompt selection via CV and MDL obtain marginal improvements over randomly picking the prompt. Another set of experiments is conducted on SuperGLUE benchmark where this time hyperparameter selection is done via CV and MDL. The pattern is similar to the first experiment and additionally it is noted that hyperparameter selection done this way drops the SuperGLUE performance below that of other work that used additional validation examples.  Originality: interesting work that attempts to define what true few-shot learning with language models should actually look like and proposes methods that fit this definition and benchmarks them. As far as I'm aware, this type of work has not been pursued before.

Quality: the problem and methods are well-defined. The experimental results seem to back up the hypothesis that true few-shot learning without using the information from held-out examples is still difficult for pre-trained language models.

Clarity: the paper is well-written. The problem is well-motivated and the methods are clearly defined. The experimental section is also very clear and the experiments measure interesting aspects of the discussed problem.

Significance: this paper could be impactful as it questions some of the assumptions of previous work showing few-shot learning in pre-trained language models and proposes best practices for future work for benchmarking few-shot learning in this setting. Yes.

This paper evaluates prompt-based approaches with pretrained language models (LMs) for few-shot learning, where limited training examples are given and no validation set is provided. This setup is named “true few-shot learning” in the paper and is distinguished from multi-distribution and tuned few-shot learning, where data from other similar tasks and a large validation set is available, respectively. Under the true few-shot learning setup, model selection can only reply on the few training examples. 

Experimental results show that, when only relying on the few training examples, model selection using methods like cross validation (CV) and minimal description length (MDL) often fail to select the optimal prompts and hyperparameters. As a result, performance acquired by using CV and MDL is only slightly better than random model selection and is significantly worse than selection based on a held-out validation set. 

Overall, this work shows that the few-shot ability of pretrained LMs is overestimated for true few shot learning, where model selection is fundamentally difficult and needs to be addressed in future work. Moreover, this work provides suggestions for fair comparison on few-shot learning. 
  This paper criticizes recent advances made in prompt-based few-shot learning with pretrained LMs, arguing that the few-shot ability of pretrained LMs is overestimated for true few shot learning problems. The paper proves this by experimental results on various tasks showing the few-shot learning performance drops significantly when no validation set is available. 

The paper is well-written and clear. The experiments are well-designed, and the results supports the paper’s argument well. The results and suggestions are informative and inspiring for future work. For example, future work should be explicit about the assumptions made for few-shot learning, i.e., what data is available besides the few-shot training examples. Also, future work should indicate clearly all how the model selection is performed.  

However, the contribution of this work is limited by its scope. The results for true few-shot learning are interesting but are not widely applicable. Because most previous work, including those being criticized, focus on the multi-distribution or tuned few-shot learning settings and make assumptions like a validation set is available or optimal hyperparameters for other tasks could be re-used for the new few-shot tasks. Indeed, future works should be explicit about their model selection method to make fair comparison. But this has been a well-known issue for few-shot learning under additional assumptions. For example, researchers in computer vision have been working on datasets with standard splits for fair comparison [1]. The true few-shot learning setup might not be entirely impactful.

[1] Triantafillou, Eleni, et al. "Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples." International Conference on Learning Representations. 2019.
 (1) The true few-shot learning setting is good for showing the few-shot learning ability of pretrained LM without any additional information except the few training examples. This is true since the few-shot performance of LMs is much better than random guess, although being suboptimal compared to SOTA. 

As discussed above, much of the work criticized by the paper is not designed for true few-shot learning, but for more practical settings like multi-dimension or tuned few-shot learning. We do need a better comparison framework for multi-distribution and tuned few-shot learning, e.g., specifying which external tasks can be used for training/validation and how many validation data can be used. For example, Tam et al. [1] assumes full access to development set while Schick and Schütze [2] didn’t make such assumption and thus the comparison between them is unfair. But this is a well-known issue for few shot learning or machine learning in general. 


(2)
The paper also investigates whether the results for LAMA also hold for other classification tasks.
In the LAMA benchmark, the LMs are asked to predict the answer out of 20k candidates given a sentence. But for regular classification tasks, the number of candidates is often much smaller, i.e., only 2 for binary review classification (negative/positive). Also, when talking about 5-shot classification tasks, people often mean 5 examples per class. But in the Appendix B, only 5 examples in total are given. It is thus unsurprising that CV and MDL will fail in this case. How about using K shots per class for classification tasks and making sure each fold for validation contains examples of all classes? Will the conclusion still hold?

[1] Derek Tam, Rakesh R Menon, Mohit Bansal, Shashank Srivastava, and Colin Raffel. Improving and simplifying pattern exploiting training. arxiv preprint arXiv:2103.11955, 2021.
[2] Timo Schick and Hinrich Schütze. Exploiting cloze questions for few-shot text classification and natural language inference. Computing Research Repository, arXiv:2001.07676, 2020.


Meta-review: This paper focuses on an important shortcoming of the testing of and model selection in of language models for few-shot learning. The authors and reviewers engaged in a good discussion. Overall, the reviewers liked the paper giving above acceptance scores.

This is a timely contribution that can help improve the practice in few-shot language modelling. This paper will probably not benefit much from further revisions. 

Therefore, accept is recommended.

Decision: accept