FLEX: Unifying Evaluation for Few-Shot NLP

1 Introduction . Few-shot learning , the challenge of learning from a small number of examples , is critical for developing efficient , robust NLP techniques [ 71 , 76 ] . In recent years , separate threads of few-shot NLP research have pursued goals like generalization to new classes [ e.g. , 5 , 25 ] , adaptation to new domains and tasks [ e.g. , 3 , 4 , 21 ] , and direct application of pretrained language models ( LMs ) [ e.g. , 10 , 24 , 55 , 56 ] . Unfortunately , despite the shared goal of advancing few-shot NLP techniques , the community does not know which techniques work best or even if they perform better than simple baselines . Evaluation suites across these research threads are disjoint , lack challenging-yet-realistic testing setups ( e.g. , class imbalance , variable training set sizes , etc . ) , and do not employ careful experimental design to ensure accurate and precise evaluation estimates and minimal computational burden . Prior work in few-shot learning outside of NLP serves as a stark warning of the consequences of improper measurement : Dhillon et al . [ 19 ] showed that techniques from several years of prior work did not make clear progress due to large overlapping accuracy distributions and , moreover , do not outperform a simple , carefully-tuned baseline . Need for systematic benchmark design As such , a high-quality benchmark is urgently needed to enable rigorous comparison of techniques across disjoint , highly-active threads of few-shot NLP research . But what should such an evaluation suite look like ? Some best practices for evaluation of few-shot methods have been introduced in the computer vision ( CV ) literature [ 19 , 67 ] and should ∗Equal contribution 2Benchmark , leaderboard , and benchmark creation toolkit : https : //github.com/allenai/flex . Apache License 2.0 3Few-shot model : https : //github.com/allenai/unifew . Apache License 2.0 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . be applied to NLP . However , unifying few-shot NLP work introduces new challenges . For example , the benchmark needs to test all types of transfer studied in separate research threads to measure progress on new techniques that make gains in each of these important generalization settings ( §2 ) . Also , given the importance of zero-shot learning and learning from task descriptions [ 29 , 73 ] , the benchmark needs to include zero-shot episodes and textual labels to enable measuring progress for models that do not use conventional supervised training , including methods that leverage the latent knowledge in pretrained LMs [ 10 , 24 , 78 ] . Further , the benchmark must accommodate new , computationally-expensive approaches , without overly reducing the number of evaluation episodes at the expense of statistical accuracy [ 3 , 24 , 75 ] . Need for a robust few-shot model Recent prompt-based models [ 10 ] have shown strong results in few-shot learning . These models leverage the power of ( often large ) pretrained language models and adapt the format of downstream tasks to the underlying pretraining objective ( e.g. , Masked Language Modeling ) . This way , given the right natural language prompt ( and sometimes verbalizers [ 55 ] and additional demonstrative examples ) , the model can quickly fine-tune on the downstream task [ 24 , 43 , 44 , 55 , 56 ] . However , adapting task formats to the underlying ( masked ) language modeling objectives is not straightforward ; such models have been shown to be sensitive to varying choices of the prompt/demonstrations , training settings , hyperparameters , and learning algorithms [ 33 , 50 , 78 ] , often requiring large held out sets and/or complex methods to overcomes such challenges . Can models eschew complex prompt engineering by unifying pretraining and downstream task formats ? In this paper , we tackle these key issues by introducing FLEX—Few-shot Language Evaluation across ( X ) many transfer types—and contributing the following : • FLEX Principles ( §3 ) , a set of requirements and best practices for few-shot NLP evaluation that enables unified , rigorous , valid , and cost-sensitive measurements . – Sample Size Design : In support of valid , cost-sensitive measurement , we introduce a novel approach to few-shot sample size design ( §5 ) that optimizes for a benchmark ’ s statistical accuracy and precision while keeping computational costs accessible to a broad range of researchers . • FLEX benchmark ( §4 ) , an implementation of the FLEX Principles . It tests across four few-shot transfer settings,7 and includes a public leaderboard for few-shot NLP that covers 20 datasets across diverse NLP tasks ( e.g. , NLI , relation classification , entity typing ) . Table 1 summarizes key differences between FLEX and other few-shot NLP evaluation suites . 4The total number of training shots in each episode , not number of shots per class per episode . 5Most users use unlabeled examples , though recently , Tam et al . [ 65 ] do not . 6Average ( avg ) , confidence interval ( CI ) , standard deviation ( SD ) , individual episode metrics 7Prior work evaluated at most two settings . • UniFew ( §6 ) , a prompt-based model for few-shot learning in NLP . While most existing methods leverage pre-trained LMs for few-shot learning , LM pre-training tasks do not closely match natural downstream task formats , requiring complex methods ( e.g. , extensive prompt-engineering , use of verbalizers , episodic hyperparameter tuning , custom learning algorithms ) to make these models work in few-shot setting . Instead , the key idea of our model , UniFew , is to close the gap between pre-training and fine-tuning formats by posing tasks as multiple-choice QA and using an underlying model that is pre-trained on a similar natural QA task format . This eliminates the need for complexities of adapting downstream tasks to the LM objectives , while resulting in competitive performance with both recent few-shot and meta-learning methods . To aid similar efforts , our release of FLEX includes a toolkit for benchmark creation and few-shot NLP model development , which we used to create the FLEX benchmark and train UniFew . 2 Background and Related Work . We first provide background and notation for few-shot learning and evaluation , then discuss related work in NLP and outside NLP that motivated us to create the FLEX Principles and benchmark . Few-shot background and notation Broadly , modern approaches to few-shot learning are evaluated in a three-phase procedure [ 68 ] . In the first phase , a general-purpose pretrained model is obtained . In the subsequent “ meta-training ” phase,8 techniques aim to adapt the model to be well-suited for few-shot generalization . Finally , a “ meta-testing ” phase evaluates the adapted model in new few-shot prediction settings . Let D be a dataset of ( x , y ) examples with full label set YD . From it , we construct three sets of episodes , corresponding to meta-training , meta-validation , and meta-testing and denoted by Etrain , Eval , and Etest , respectively . Each episode in each of these sets is a few-shot problem with its own test Eset and other attributes . Formally , each episode E is a tuple ( DE ) , where YE is a train , DE test , YDval , DE D sampled subset of labels in YD and DE are disjoint sets of examples from D with labels in train|val|test EYD . 9 For each episode , the model ’ s objective is to correctly predict labels for examples DEtest . To accomplish this , models make use of labeled examples in DEtrain , which is typically configured such that each label i in YE has KE provided examples ; KE is known as the shot , and the setting when aD i i class has no examples in DE = 0 ) is called zero-shot.train ( i.e. , KEi Few-shot evaluation in NLP Research in few-shot NLP has proceeded in several parallel threads , each focused on a different type of transfer ability [ 76 ] . Each thread has separate evaluation practices , and the vast majority of few-shot NLP research has limited evaluation to a single transfer type ( see Table 1 ) . Here , we describe these types of transfer and their evaluation practices . Following the CV literature [ 67 , 68 ] , one thread of few-shot NLP focuses on class transfer , the problem of generalizing from a supervised set of classes at meta-train time to a different set of classes from the same dataset at meta-test time . Evaluation typically involves splitting classes YD into YDtrain , YDval and YD disjoint subsets . Class transfer has been studied on many text classification tasks [ 5 ] , test including relation classification [ 25 , 28 , 64 ] , intent classification [ 37 , 64 ] , inter alia . In contrast , domain transfer keeps the same classes between meta-training and meta-testing but changes the textual domain ( e.g. , generalizing from MNLI to science-focused SciTail [ 4 , 21 ] ) . Evaluation then requires identifying pairs of datasets with the same classes YD , where one dataset ’ s episodes are assigned to Etrain and the other ’ s to Etest . Domain transfer has also been studied on many tasks [ 3 , 4 ] , including dialogue intent detection & slot tagging [ 31 ] , sentiment classification [ 77 ] , NLI [ 21 ] , and machine translation [ 27 , 58 ] . Researchers have also begun to study task transfer , the problem of generalizing from a set of tasks at meta-train time to unseen tasks at meta-test time . Evaluation requires tasks ( e.g. , NLI ) appearing in Etest not to appear in Etrain or Eval . Prior work has used GLUE tasks [ 70 ] for meta-training before meta-testing on tasks such as entity typing [ 3 , 4 ] , while other work instead used GLUE for 8Meta-training may include a “ meta-validation '' component , for validating generalization . 9In the few-shot literature , DE test are also called the support and query sets , and |YDE | the way.train and DE meta-testing [ 21 ] . Very recent work has studied task transfer over a large set of datasets [ 75 , 80 ] . A limited amount of work evaluates both domain and task transfer [ 3 , 4 , 21 ] . An important emerging line of work ( not noted by Yin [ 76 ] ) is pretraining transfer , the problem of whether pretrained language models can perform well at meta-test time without any meta-training . Evaluation in this setting requires Etrain , Eval = ∅ . Prior work has shown that pretrained language models are capable of surprising performance on many few-shot tasks , even without fine-tuning [ 10 ] . More recent work , mainly focusing on text classification , has reported further gains with cloze-style formats [ 55 , 56 , 65 ] , prompt engineering [ 24 ] , or calibration [ 78 ] . FLEX is designed to exercise all four of these transfer types from previous work . Few-shot evaluation outside NLP The few-shot learning literature has largely focused on image classification , with the introduction of increasingly complex meta-learning algorithms [ e.g. , 23 , 39 , 54 , 61 , 68 ] . However , more recent work has shown that simple fine-tuning baselines are in fact competitive , and attribute this delayed discovery to problematic evaluation methodology [ 15 , 19 ] . FLEX adopts recommended methodology [ 19 , 67 ] , and we introduce an analogous baseline ( UniFew ) to provide a strong measurement foundation for few-shot NLP .

Summary: This paper proposes FLEET, a benchmark for few-shot learning in NLP. FLEET is comprised of different few-shot settings (new tasks, label sets, or domains at test time), different amount of training examples ("shots"),  and meta-learning strategies (w/ and w/o meta learning). The paper goes through the design decisions behind FLEET and detail its construction. Finally, they present a baseline based on UnifiedQA, wherein they reformulate downstream tasks as QA and then finetune the model on downstream tasks.