Few-shot Learning with Big Prototypes

1 INTRODUCTION . Learning from few examples , i.e. , few-shot learning , receives increasing attention in modern deep learning . On the one hand , constituting cognition of novel concepts with few instances is a crucial way for machines to imitate human intelligence . On the other hand , annotating large-scale supervised datasets is expensive and time-consuming ( Lu et al. , 2020 ) . Although traditional deep neural models have achieved tremendous success under sufficient supervision , it is still challenging to produce comparable performance when training examples are limited . Hence , a series of studies are proposed to generalize deep neural networks to low-data scenarios . One crucial branch of them is meta-learning with prototypes ( Reed , 1972 ; Nosofsky , 1986 ; Snell et al. , 2017 ) , where models are trained to quickly adapt to current tasks and carry out classification via metric-based comparisons between examples and newly introduced variables , prototypes of those classes . In a general way , prototypes are designed to represent abstract class-level information and calculated by taking the mean output of a few examples belonging to one same class . Thus , prototypes are represented as dense vectors with the same dimensions as the embeddings of training examples and can be regarded as center points of those classes in the embedding space . And for the queried examples that the model needs to predict , the basic idea is to calculate the distances between the queried examples and prototypes and conduct classification based on the distances . Originated from Prototypical Network ( Snell et al. , 2017 ) , sets of derivative prototype-based methods demonstrate the effectiveness in few-shot learning ( Ren et al. , 2018 ; Gao et al. , 2019a ; Allen et al. , 2019 ; Pan et al. , 2019 ; Ding et al. , 2021a ) . However , prototypes are estimated from a few sampled examples , which can not uncover the overall ground truth distribution ( Yang et al. , 2021 ) . Such biased distributions may generate biased prototypes and trigger sequent classification errors . In other words , this modeling approach of prototypes could lack the capability to express the universal class-level information . Hence , to enhance such expressibility of prototypes , we propose to use areas , i.e. , tensor fields , rather than points in the embedding space to represent the class-level information . In this paper , we propose big prototypes to use hyperspheres in the feature space to abstract the class-level information , and the feature points can be distribued inside or around the big prototypes . Such modeling is equipped with two obvious advantages : easy to model and easy to calculate the distances . On the one hand , even if we attempt to use areas but not points to represent class-level information , it is difficult to explicitly characterize manifolds with complex boundaries in deep learning . But via hyperspheres modeling , we can obtain a big prototype only through two sets of parameters : the center and the radius of hyperspheres . On the other hand , hyperspheres are suitable for calculating the distances in Euclidean space . We can simply calculate the distance from one feature point to the surface of the hypersphere to perform metric-based classification , which is also difficult for other manifolds . Moreover , it is easy to combine these two advantages in few-shot learning , the distances from one feature point to the surface of a big prototype can be formalized as the distance from the point to the center of the hypersphere minus the radius . Thus , both radius and hypersphere center can appear in the loss function and participate in the backward propagation during optimization . Intuitively , for the classes with sparse feature distribution , the corresponding radii of their prototypes are large , and the radii are small otherwise . We conduct extensive experiments in both natural language processing ( NLP ) and computer vision ( CV ) to evaluate the effectiveness of big prototypes . Specifically for NLP , we choose widely used benchmarks in the few-shot named entity ( Ding et al. , 2021b ) and relation extraction ( Han et al. , 2018 ; Gao et al. , 2019b ) . For CV , we use classical image classification datasets ( Vinyals et al. , 2016 ) in experiments . The results demonstrate that , with only a few additional parameters introduced , such modeling significantly outperforms the baseline . Surprisingly , big prototypes perform extremely well in cross-domain few-shot relation extraction , indicating the promising ability to domain adaptation . Given that such small changes can bring huge benefits , hopefully , big prototypes can inspire new ideas for the research community of representation learning . The source code and checkpoints of our models in experiments will be publicly available for reproducibility . 2 RELATED WORK . This work is related to studies of few-shot learning and meta-learning , whose primary goal is to quickly adapt deep neural models to new tasks with few training examples . To achieve that , two branches of studies are proposed : optimization-based methods and metric-based methods . The optimization-based studies ( Finn et al. , 2017 ; Franceschi et al. , 2018 ; Ravi & Beatson , 2018 ) regard few-shot learning as a bi-level optimization process , where a global optimization is conducted to learn a good initialization across various tasks , and a local optimization quickly adapts the initialization parameters to specific tasks with few training examples by few steps of gradient descent . Compared to the mentioned studies , our work is more related to the metric-based meta-learning approaches ( Vinyals et al. , 2016 ; Snell et al. , 2017 ; Satorras & Estrach , 2018 ; Sung et al. , 2018 ) , whose general idea is to learn a metric to measure the similarity between representations and find the closest labeled example ( or a derived prototype ) for an unlabeled example . Typically , these methods learn a metric function during episodic optimization . More specifically , we inherit the spirit that uses prototypes to abstractly represent class-level information , which could be tracked back to cognitive science ( Reed , 1972 ; Rosch et al. , 1976 ; Nosofsky , 1986 ) , statistical machine learning ( Graf et al. , 2009 ) and similar to the Nearest Mean Classifier ( Mensink et al. , 2013 ) . In the area of deep learning , Snell et al . ( 2017 ) propose the prototypical network to exploit the average of example embeddings as a prototype to perform metric-based classification in few-shot learning . In Prototypical Network , prototypes are estimated by the embeddings of instances , and it is hard to find a satisfying location of the prototypes of the entire dataset . Ren et al . ( 2018 ) adapt such prototype-based networks in the semi-supervised scenario where the dataset is partially annotated . A set of prototype-based networks are proposed concentrating on the improvements of prototype estimations and application to various downstream tasks ( Allen et al. , 2019 ; Gao et al. , 2019a ; Li et al. , 2019b ; Pan et al. , 2019 ; Seth et al. , 2019 ; Ding et al. , 2021a ; Li et al. , 2020c ) . We discuss big prototypes and some other prototype-enhanced methods in Appendix C. There has also been a series of works that embed prototypes into a non-Euclidean output space ( Mettes et al. , 2019 ; Keller-Ressel , 2020 ; Atigh et al. , 2021 ) . It should be noted that these studies regard hyperspheres or other non-Euclidean manifolds as the embedding space , and our proposed method use hyperspheres to represent big prototypes and conduct metric-based classification in the Euclidean space . Therefore , the focus of our proposed big prototypes is different from the nonEuclidean prototype-based works . We evaluate the effectiveness of big prototypes in three downstream tasks across NLP and CV , including few-shot named entity recognition , few-shot relation extraction , and few-shot image classification . Another technical route to achieve promising results in these downstream few-shot tasks is to use the ability of large-scale pre-trained models ( Brown et al. , 2020 ; Han et al. , 2021b ) , or build large-scale data sets to design specific pre-training tasks ( Huang et al. , 2020 ; Soares et al. , 2019 ) ( which may face the risk of information leakage ) . These techniques are also orthogonal to our contribution . Generally , the method of big prototypes is model-agnostic . 3 BIG PROTOTYPES . This section begins with the problem setup of few-shot learning . Then , we introduce the metrics , initialization , and learning of big prototypes . Generally , unlike previous prototype-based models that use estimated dense vectors as prototypes , our approach use hyperspheres to represent the conceptlevel information for classes . One big prototype is represented by two parameters : the center and the radius of the hypersphere , which are firstly initialized via estimation and then optimized by gradient descent in an end-to-end fashion . 3.1 PROBLEM SETUP . In this work , we consider the episodic N way K shot few-shot learning paradigm . In this setting , we have a large-scale annotated training setDtrain , and our goal is to learn a model that could predict for a set of new classes Dtest , where only a few examples are labeled . In this setting , the model will be trained in episodes constructed using Dtrain and tested on episodes constructed using Dtest . Each episode contains a support set for learning S = { xi , yi } N×Ki=1 with N classes and K examples for each class , and a query set for inference Q = { x∗j , y∗j } N×K′ j=1 of examples in the same N classes . Each input data is a vector xi ∈ RL with the dimension of L and yi is an index of class label . For each input xi , let hi = fϕ ( xi ) ∈ RD denote the D-dimensional output embedding of a neural network fϕ : RL → RD parameterized by ϕ . Beyond the conventional few-shot classification , we also carry out experiments in few-shot named entity recognition . This is a sequence labeling task where each token in a sequence is asked to be labeled as if it is a part of a named entity . But as the context is extremely important for this task , the examples are sampled in sequence-level . Thus the problem setup is slightly different from the traditional N way K shot classification . We follow the strategy in Ding et al . ( 2021b ) and sample sequences in a N way K ∼ 2K shot manner ( see Appendix B ) . 3.2 METRIC . We now introduce big prototypes , which are a set of hyperspheres in the feature space to abstractly represent the intrinsic features of classes . No matter what the dimension of embedding space is , one big prototype is represented by p = ( z , r ) , where z ∈ RD is the center of the hypersphere with the same dimension to h = fϕ ( x ) , and r ∈ R is a scalar denoting the radius of the hypersphere . The central idea is to learn a big prototype for each class with limited episodic supervision , and each example in the query set ( x∗j , y ∗ j ) is predicted by measuring the distance from the embedding h ∗ to the surface of the hyperspheres . The distance of two vectors is calculated by a metric function : d : RD × RD → [ 0 , +∞ ) . In the Euclidean space , the metric being d ( h , z ) = ∥h− z∥2 . ( 1 ) And the distance d̃ from an embedding to a big prototype p is the distance from the point to the center of the hypersphere minus the radius d̃ ( x , p ) = d ( fϕ ( x ) , z ) − r = ∥h− z∥2 − r. ( 2 ) Note that in this case , the value of d̃ ( · ) may be negative , that is geometrically speaking , the point is contained inside the hypersphere , and it does not affect the calculation of loss . Although more generally , the idea is to use areas but not points in the embedding space to model prototypes , hyperspheres naturally have two obvious advantages . As stated in § 1 , one big prototype could be uniquely modeled by the center and the radius p = { z , r } . While characterizing manifolds with complex boundaries in the embedding space is difficult.s Furthermore , it is easy to calculate metrics by Equation 2 . We use the distance from a point to the surface as the metric . In this way , the center and the radius are spontaneously in the loss function and optimized . In this geometric interpretation , sparse classes will have larger learned radii , while compact classes will have smaller learned radii .

Summary: The paper proposed to represent prototypes by hyperspheres with dynamic sizes.  A so-called big prototype is characterized by the center of the hypersphere and the radius of the sphere.  Empirical results are conducted on few-shot named entity recognition (NER), few-shot relation extraction (RE) and few-shot image classifica- tion.