True Few-Shot Learning with Language Models

1 Introduction . Major progress in language model ( LM ) pretraining has led to the idea that LMs can learn a new task using a small number of examples only , i.e. , few-shot learning [ 1–3 ] . Few-shot learning overcomes many challenges with data-rich supervised learning : collecting labeled data is expensive , often requires experts , and scales poorly with the number of tasks . However , the few-shot performance of LMs is very sensitive to the textual task description [ “ prompt ” ; 3–6 ] , order of training examples [ 6–8 ] , decoding strategy [ 9 , 10 ] , and other hyperparameters [ 3 , 5 , 9 , 11 , 12 ] , as well as the learning algorithm itself [ 3 , 12 ] . Thus , effective model selection is crucial for obtaining good few-shot performance . There are issues with how recent work approaches model selection in few-shot learning , however . Prior work uses large train or held-out sets with many examples to choose prompts [ 2 , 12 , 13 ] and hyperparameters [ 12 ] . Other work claims to use no validation set for hyperparameter selection [ 3 , 11 , 14 ] but does not describe how they design other aspects of their learning algorithm ( e.g. , training objectives ) . It is unlikely that no validation examples were used , given the sophisticated nature of the proposed algorithms . In this work , we examine if prior few-shot learning methods still perform well when using only the provided examples for model selection , a setting we term true few-shot learning . We find that true few-shot model selection yields prompts that marginally outperform random selection and greatly underperform selection based on held-out examples . Our result shows that prior work may have greatly overestimated the few-shot ability of LMs . In other words , one reason that prompts are so effective [ “ worth many examples ” ; 15 ] is that they are often tuned using many examples . We evaluate two standard model selection criteria – cross-validation ( CV ) and minimum description length ( MDL ) – finding that both obtain only limited improvements over random selection and perform much worse than selection using held-out examples . For prompt selection , our observation holds for 9 LMs ranging over 3 orders of magnitude in size [ 1 , 2 , 16 ] on 3 classification tasks and 41 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . tasks in the LAMA benchmark [ 17 ] . For choosing hyperparameters , true few-shot selection causes performance to drop by 2-10 % across 8 tasks for ADAPET [ 12 ] , a state-of-the-art few-shot method . Furthermore , true few-shot model selection has high variance in performance ; selected models often do much worse than randomly-chosen ones . We find similar results when varying the number of examples used , amount of computation , and conservativeness of our selection criterion . Altogether , our results suggest that model selection is a fundamental roadblock to true few-shot learning . 2 Can We Do Model Selection in Few-Shot Learning ? Prior work uses the phrase “ few-shot learning ” in multiple senses , raising questions about what it means to do few-shot learning . We categorize few-shot learning into three distinct settings , each of which assumes access to different data . Here , we formally disambiguate between these settings to help future work avoid inadvertently comparing few-shot methods that operate in different settings . Consider the supervised learning scenario where we have a dataset of inputs x1 : N and labels y1 : N , sampled from a distribution over datasets D. We aim to determine the learning algorithm A∗ ∈ A1 , . . . , AA with the smallest generalization loss L at predicting y given x on unseen validation examples Dval ∼ D after learning on training examples Dtrain ∼ D. We say that an algorithm A ( Dtrain , R ) maps a training dataset Dtrain and various random factors R that influence training to a function that predicts y given x . A specifies , e.g. , the model architecture , hyperparameters , and prompt . R includes random factors that impact the results of a learning algorithm , such as parameter initialization and the order of training examples for online learning algorithms like stochastic gradient descent . We say that A obtains a generalization loss L ( A ( Dtrain , R ) , Dval ) on a given validation set Dval . We aim to find the A∗ that minimizes the expected loss across training and validation sets : EL ( A , R ) = EDtrain , Dval [ L ( A ( Dtrain , R ) ; Dval ) ] In data-rich supervised learning , EL ( A , R ) is usually evaluated with a single train-validation split ( Dtrain , Dval ) . Since large Dtrain and Dval are not always available , the traditional few-shot setting evaluates EL ( A , R ) with many small ( Dtrain , Dval ) drawn from many , distinct distributions D [ see , e.g. , work in meta-learning 18–21 ] . Each distribution D is sampled from D∗ , a distribution over distributions ( e.g. , of similar tasks ) , so we call this setting multi-distribution few-shot learning . Recent work does not assume access to data from other distributions , performing few-shot learning using only a few examples from a single distribution to update a pretrained LM [ 2 , 12 ] . These papers use a large validation set Dval to tune the learning algorithm A , a setting we term tuned few-shot learning . For example , Brown et al . [ 2 ] try prompts with different phrasings and numbers of training examples to improve the validation accuracy of GPT-3 . Tam et al . [ 12 ] choose the early stopping iteration , prompt , and other model-specific hyperparameters based on validation performance . Tuned few-shot learning relies on many labeled examples , so we argue that tuned few-shot learning does not qualify as few-shot learning . If many validation examples are available , they could be incorporated into the training set and trained on using data-rich supervised learning . Tuned few-shot learning algorithms should be compared against data-rich supervised learning algorithms that use the same amount of total data |Dtrain|+ |Dval| . In this work , we evaluate the success of tuned few-shot learning methods when no large Dval is available , a setting we term true few-shot learning . Formally , we aim to choose a learning algorithm A with low expected loss EL ( A , R ) using only a small training set Dtrain drawn from a single distribution . Here , we must choose A by approximating EL ( A , R ) , e.g. , using cross-validation . Several papers claim to circumvent the need to estimate EL ( A , R ) by choosing hyperparameters based on an educated guess [ 3 , 9 , 14 ] . However , the proposed learning algorithms themselves are quite sophisticated , and it is unclear how they were designed if not by using validation performance . Other work chooses the learning algorithm and hyperparameters using one or multiple other datasets before evaluating on the target dataset [ 5 , 11 ] . Such approaches fall under multi-distribution few-shot learning and can not be directly compared to methods that attempt to perform true few-shot learning , even though prior work has made such comparisons [ 14 ] . In what follows , we describe two model selection criteria – cross-validation and minimum description length – which we use to evaluate tuned few-shot methods in the true few-shot setting . 2.1 Cross-validation . Cross-Validation ( CV ) [ 22–24 ] is one of the most widely used methods for estimating generalization loss [ 25 ] . CV has also been used in prior work on multi-distribution few-shot learning [ 26 , 27 ] . CV randomly partitions Dtrain into K equally-sized folds F ( Dtrain ) 1 , . . . , F ( Dtrain ) K and evaluates the average loss on a validation fold F ( Dtrain ) k when training on the remaining data F ( Dtrain ) ¬k : CV ( A , R , F ) = Ek∼Unif ( 1 , K ) [ L ( A ( F ( Dtrain ) ¬k , R ) ; F ( Dtrain ) k ) ] In this way , CV forms K train-validation splits out of the pool of labeled examples . CV with one example per fold ( K = N folds ) is commonly referred to as leave-one-out CV ( LOOCV ) . 2.2 Minimum description length . We may also form train-validation splits in a different manner than CV , drawing inspiration from work on the Minimum Description Length ( MDL ) principle [ 28 ] . MDL can be estimated by evaluating the average loss on a fold F ( D ) k when training on the previous folds F ( D ) 1 : k−1 : MDL ( A , R , F ) = Ek∼Unif ( 1 , K ) [ L ( A ( F ( Dtrain ) 1 : k−1 , R ) ; F ( Dtrain ) k ) ] This procedure is referred to as “ online coding ” [ 29 , 30 ] , as it evaluates the generalization loss of the algorithm as it learns “ online ” from more and more data.1 There are other ways to evaluate MDL [ see 31 , for an overview ] . We use online coding as it has been shown to be an effective way to estimate MDL , especially for deep learning methods [ 32 ] . MDL measures generalization because it evaluates how much a learning algorithm compresses the labels y1 : N given the inputs x1 : N , and because better compression implies better generalization [ 33 ] . Recent work has used MDL to determine which learning algorithms are most effective at explaining the given data [ Rissanen Data Analysis ; 10 , 34 ] . 2.3 Variance matters . We evaluate the generalization loss of the algorithm chosen by CV ( likewise for MDL ) : L ( ACV ( Dtrain , R ) , Dval ) , where ACV = argmin A ER , F [ CV ( A , R , F ) ] . The above loss should be low in expectation , across different datasets Dtrain ∼ D , Dval ∼ D , and random factors R , F : EDtrain , Dval , R , F [ L ( ACV ( Dtrain , R ) , Dval ) ] . The loss should also be low in variance : VDtrain , Dval , R , F [ L ( ACV ( Dtrain , R ) , Dval ) ] . Low variance implies that CV/MDL reliably choose an algorithm that generalizes toDval when trained with a givenDtrain and random factorsR , F . Reliability is important for many practical or commercial applications where worst-case performance is important , such as image recognition [ 35 , 36 ] , dialogue systems [ 37 , 38 ] , and robotics [ 39 , 40 ] . We also experiment with explicitly taking into account an algorithm ’ s variance during model selection , choosing ACV to minimize a conservative estimate of CV , CVα ( A ) , chosen such that the probability PrR , F [ CV ( A , R , F ) < CVα ( A ) ] is high : CVα ( A ) = ER , F [ CV ( A , R , F ) ] + α √ VR , F [ CV ( A , R , F ) ] where α is a hyperparameter set based on the desired probability . In particular , if CV ( A , R , F ) follows a normal distributionN when samplingR , F , then CV ( A , R , F ) ≤ CVα ( A ) with probability∫ α −∞N ( µ = 0 , σ = 1 ) for a given R , F . CVα ( A ) resembles the Watanabe Akaike Information Criterion [ 41 ] , which estimates the generalization of a model trained with A using the expected loss from a model trained with A plus the variance in training loss across models trained with A . 1Online coding formally computes a sum over L ( . ) rather than the expectation , which differs by a constant factor . The two are equivalent for our purposes ( ranking A ) .

Summary: The paper proposes to re-evaluate some recent LM-based few-shot learners like GPT-3 and PET without using any additional validation examples from the target task during evaluation. They term this as “true few-shot learning”. The paper proposes to use two approaches to evaluate without any validation examples: (1) leave-one-out CV (2) minimum description length. These methods are evaluated for few-shot prompt tuning as well as model selection and authors find that previous work often over-claimed their performance. For example, GPT-3 prompt tuning leads to much worse accuracy in LAMA than selecting the prompt based on full validation data (the approach used by GPT-3). Similarly, a recent variation over PET, called ADAPET, used full validation without which there is actually no improvement over the PET method which it claimed to improve over.