True Few-Shot Learning with Language Models

1 Introduction . Major progress in language model ( LM ) pretraining has led to the idea that LMs can learn a new task using a small number of examples only , i.e. , few-shot learning [ 1–3 ] . Few-shot learning overcomes many challenges with data-rich supervised learning : collecting labeled data is expensive , often requires experts , and scales poorly with the number of tasks . However , the few-shot performance of LMs is very sensitive to the textual task description [ “ prompt ” ; 3–6 ] , order of training examples [ 6–8 ] , decoding strategy [ 9 , 10 ] , and other hyperparameters [ 3 , 5 , 9 , 11 , 12 ] , as well as the learning algorithm itself [ 3 , 12 ] . Thus , effective model selection is crucial for obtaining good few-shot performance . There are issues with how recent work approaches model selection in few-shot learning , however . Prior work uses large train or held-out sets with many examples to choose prompts [ 2 , 12 , 13 ] and hyperparameters [ 12 ] . Other work claims to use no validation set for hyperparameter selection [ 3 , 11 , 14 ] but does not describe how they design other aspects of their learning algorithm ( e.g. , training objectives ) . It is unlikely that no validation examples were used , given the sophisticated nature of the proposed algorithms . In this work , we examine if prior few-shot learning methods still perform well when using only the provided examples for model selection , a setting we term true few-shot learning . We find that true few-shot model selection yields prompts that marginally outperform random selection and greatly underperform selection based on held-out examples . Our result shows that prior work may have greatly overestimated the few-shot ability of LMs . In other words , one reason that prompts are so effective [ “ worth many examples ” ; 15 ] is that they are often tuned using many examples . We evaluate two standard model selection criteria – cross-validation ( CV ) and minimum description length ( MDL ) – finding that both obtain only limited improvements over random selection and perform much worse than selection using held-out examples . For prompt selection , our observation holds for 9 LMs ranging over 3 orders of magnitude in size [ 1 , 2 , 16 ] on 3 classification tasks and 41 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . tasks in the LAMA benchmark [ 17 ] . For choosing hyperparameters , true few-shot selection causes performance to drop by 2-10 % across 8 tasks for ADAPET [ 12 ] , a state-of-the-art few-shot method . Furthermore , true few-shot model selection has high variance in performance ; selected models often do much worse than randomly-chosen ones . We find similar results when varying the number of examples used , amount of computation , and conservativeness of our selection criterion . Altogether , our results suggest that model selection is a fundamental roadblock to true few-shot learning . 2 Can We Do Model Selection in Few-Shot Learning ? Prior work uses the phrase “ few-shot learning ” in multiple senses , raising questions about what it means to do few-shot learning . We categorize few-shot learning into three distinct settings , each of which assumes access to different data . Here , we formally disambiguate between these settings to help future work avoid inadvertently comparing few-shot methods that operate in different settings . Consider the supervised learning scenario where we have a dataset of inputs x1 : N and labels y1 : N , sampled from a distribution over datasets D. We aim to determine the learning algorithm A∗ ∈ A1 , . . . , AA with the smallest generalization loss L at predicting y given x on unseen validation examples Dval ∼ D after learning on training examples Dtrain ∼ D. We say that an algorithm A ( Dtrain , R ) maps a training dataset Dtrain and various random factors R that influence training to a function that predicts y given x . A specifies , e.g. , the model architecture , hyperparameters , and prompt . R includes random factors that impact the results of a learning algorithm , such as parameter initialization and the order of training examples for online learning algorithms like stochastic gradient descent . We say that A obtains a generalization loss L ( A ( Dtrain , R ) , Dval ) on a given validation set Dval . We aim to find the A∗ that minimizes the expected loss across training and validation sets : EL ( A , R ) = EDtrain , Dval [ L ( A ( Dtrain , R ) ; Dval ) ] In data-rich supervised learning , EL ( A , R ) is usually evaluated with a single train-validation split ( Dtrain , Dval ) . Since large Dtrain and Dval are not always available , the traditional few-shot setting evaluates EL ( A , R ) with many small ( Dtrain , Dval ) drawn from many , distinct distributions D [ see , e.g. , work in meta-learning 18–21 ] . Each distribution D is sampled from D∗ , a distribution over distributions ( e.g. , of similar tasks ) , so we call this setting multi-distribution few-shot learning . Recent work does not assume access to data from other distributions , performing few-shot learning using only a few examples from a single distribution to update a pretrained LM [ 2 , 12 ] . These papers use a large validation set Dval to tune the learning algorithm A , a setting we term tuned few-shot learning . For example , Brown et al . [ 2 ] try prompts with different phrasings and numbers of training examples to improve the validation accuracy of GPT-3 . Tam et al . [ 12 ] choose the early stopping iteration , prompt , and other model-specific hyperparameters based on validation performance . Tuned few-shot learning relies on many labeled examples , so we argue that tuned few-shot learning does not qualify as few-shot learning . If many validation examples are available , they could be incorporated into the training set and trained on using data-rich supervised learning . Tuned few-shot learning algorithms should be compared against data-rich supervised learning algorithms that use the same amount of total data |Dtrain|+ |Dval| . In this work , we evaluate the success of tuned few-shot learning methods when no large Dval is available , a setting we term true few-shot learning . Formally , we aim to choose a learning algorithm A with low expected loss EL ( A , R ) using only a small training set Dtrain drawn from a single distribution . Here , we must choose A by approximating EL ( A , R ) , e.g. , using cross-validation . Several papers claim to circumvent the need to estimate EL ( A , R ) by choosing hyperparameters based on an educated guess [ 3 , 9 , 14 ] . However , the proposed learning algorithms themselves are quite sophisticated , and it is unclear how they were designed if not by using validation performance . Other work chooses the learning algorithm and hyperparameters using one or multiple other datasets before evaluating on the target dataset [ 5 , 11 ] . Such approaches fall under multi-distribution few-shot learning and can not be directly compared to methods that attempt to perform true few-shot learning , even though prior work has made such comparisons [ 14 ] . In what follows , we describe two model selection criteria – cross-validation and minimum description length – which we use to evaluate tuned few-shot methods in the true few-shot setting . 2.1 Cross-validation . Cross-Validation ( CV ) [ 22–24 ] is one of the most widely used methods for estimating generalization loss [ 25 ] . CV has also been used in prior work on multi-distribution few-shot learning [ 26 , 27 ] . CV randomly partitions Dtrain into K equally-sized folds F ( Dtrain ) 1 , . . . , F ( Dtrain ) K and evaluates the average loss on a validation fold F ( Dtrain ) k when training on the remaining data F ( Dtrain ) ¬k : CV ( A , R , F ) = Ek∼Unif ( 1 , K ) [ L ( A ( F ( Dtrain ) ¬k , R ) ; F ( Dtrain ) k ) ] In this way , CV forms K train-validation splits out of the pool of labeled examples . CV with one example per fold ( K = N folds ) is commonly referred to as leave-one-out CV ( LOOCV ) . 2.2 Minimum description length . We may also form train-validation splits in a different manner than CV , drawing inspiration from work on the Minimum Description Length ( MDL ) principle [ 28 ] . MDL can be estimated by evaluating the average loss on a fold F ( D ) k when training on the previous folds F ( D ) 1 : k−1 : MDL ( A , R , F ) = Ek∼Unif ( 1 , K ) [ L ( A ( F ( Dtrain ) 1 : k−1 , R ) ; F ( Dtrain ) k ) ] This procedure is referred to as “ online coding ” [ 29 , 30 ] , as it evaluates the generalization loss of the algorithm as it learns “ online ” from more and more data.1 There are other ways to evaluate MDL [ see 31 , for an overview ] . We use online coding as it has been shown to be an effective way to estimate MDL , especially for deep learning methods [ 32 ] . MDL measures generalization because it evaluates how much a learning algorithm compresses the labels y1 : N given the inputs x1 : N , and because better compression implies better generalization [ 33 ] . Recent work has used MDL to determine which learning algorithms are most effective at explaining the given data [ Rissanen Data Analysis ; 10 , 34 ] . 2.3 Variance matters . We evaluate the generalization loss of the algorithm chosen by CV ( likewise for MDL ) : L ( ACV ( Dtrain , R ) , Dval ) , where ACV = argmin A ER , F [ CV ( A , R , F ) ] . The above loss should be low in expectation , across different datasets Dtrain ∼ D , Dval ∼ D , and random factors R , F : EDtrain , Dval , R , F [ L ( ACV ( Dtrain , R ) , Dval ) ] . The loss should also be low in variance : VDtrain , Dval , R , F [ L ( ACV ( Dtrain , R ) , Dval ) ] . Low variance implies that CV/MDL reliably choose an algorithm that generalizes toDval when trained with a givenDtrain and random factorsR , F . Reliability is important for many practical or commercial applications where worst-case performance is important , such as image recognition [ 35 , 36 ] , dialogue systems [ 37 , 38 ] , and robotics [ 39 , 40 ] . We also experiment with explicitly taking into account an algorithm ’ s variance during model selection , choosing ACV to minimize a conservative estimate of CV , CVα ( A ) , chosen such that the probability PrR , F [ CV ( A , R , F ) < CVα ( A ) ] is high : CVα ( A ) = ER , F [ CV ( A , R , F ) ] + α √ VR , F [ CV ( A , R , F ) ] where α is a hyperparameter set based on the desired probability . In particular , if CV ( A , R , F ) follows a normal distributionN when samplingR , F , then CV ( A , R , F ) ≤ CVα ( A ) with probability∫ α −∞N ( µ = 0 , σ = 1 ) for a given R , F . CVα ( A ) resembles the Watanabe Akaike Information Criterion [ 41 ] , which estimates the generalization of a model trained with A using the expected loss from a model trained with A plus the variance in training loss across models trained with A . 1Online coding formally computes a sum over L ( . ) rather than the expectation , which differs by a constant factor . The two are equivalent for our purposes ( ranking A ) .

Summary: This paper evaluates prompt-based approaches with pretrained language models (LMs) for few-shot learning, where limited training examples are given and no validation set is provided. This setup is named “true few-shot learning” in the paper and is distinguished from multi-distribution and tuned few-shot learning, where data from other similar tasks and a large validation set is available, respectively. Under the true few-shot learning setup, model selection can only reply on the few training examples.   Experimental results show that, when only relying on the few training examples, model selection using methods like cross validation (CV) and minimal description length (MDL) often fail to select the optimal prompts and hyperparameters. As a result, performance acquired by using CV and MDL is only slightly better than random model selection and is significantly worse than selection based on a held-out validation set.   Overall, this work shows that the few-shot ability of pretrained LMs is overestimated for true few shot learning, where model selection is fundamentally difficult and needs to be addressed in future work. Moreover, this work provides suggestions for fair comparison on few-shot learning.