Few-shot Learning with Big Prototypes

1 INTRODUCTION . Learning from few examples , i.e. , few-shot learning , receives increasing attention in modern deep learning . On the one hand , constituting cognition of novel concepts with few instances is a crucial way for machines to imitate human intelligence . On the other hand , annotating large-scale supervised datasets is expensive and time-consuming ( Lu et al. , 2020 ) . Although traditional deep neural models have achieved tremendous success under sufficient supervision , it is still challenging to produce comparable performance when training examples are limited . Hence , a series of studies are proposed to generalize deep neural networks to low-data scenarios . One crucial branch of them is meta-learning with prototypes ( Reed , 1972 ; Nosofsky , 1986 ; Snell et al. , 2017 ) , where models are trained to quickly adapt to current tasks and carry out classification via metric-based comparisons between examples and newly introduced variables , prototypes of those classes . In a general way , prototypes are designed to represent abstract class-level information and calculated by taking the mean output of a few examples belonging to one same class . Thus , prototypes are represented as dense vectors with the same dimensions as the embeddings of training examples and can be regarded as center points of those classes in the embedding space . And for the queried examples that the model needs to predict , the basic idea is to calculate the distances between the queried examples and prototypes and conduct classification based on the distances . Originated from Prototypical Network ( Snell et al. , 2017 ) , sets of derivative prototype-based methods demonstrate the effectiveness in few-shot learning ( Ren et al. , 2018 ; Gao et al. , 2019a ; Allen et al. , 2019 ; Pan et al. , 2019 ; Ding et al. , 2021a ) . However , prototypes are estimated from a few sampled examples , which can not uncover the overall ground truth distribution ( Yang et al. , 2021 ) . Such biased distributions may generate biased prototypes and trigger sequent classification errors . In other words , this modeling approach of prototypes could lack the capability to express the universal class-level information . Hence , to enhance such expressibility of prototypes , we propose to use areas , i.e. , tensor fields , rather than points in the embedding space to represent the class-level information . In this paper , we propose big prototypes to use hyperspheres in the feature space to abstract the class-level information , and the feature points can be distribued inside or around the big prototypes . Such modeling is equipped with two obvious advantages : easy to model and easy to calculate the distances . On the one hand , even if we attempt to use areas but not points to represent class-level information , it is difficult to explicitly characterize manifolds with complex boundaries in deep learning . But via hyperspheres modeling , we can obtain a big prototype only through two sets of parameters : the center and the radius of hyperspheres . On the other hand , hyperspheres are suitable for calculating the distances in Euclidean space . We can simply calculate the distance from one feature point to the surface of the hypersphere to perform metric-based classification , which is also difficult for other manifolds . Moreover , it is easy to combine these two advantages in few-shot learning , the distances from one feature point to the surface of a big prototype can be formalized as the distance from the point to the center of the hypersphere minus the radius . Thus , both radius and hypersphere center can appear in the loss function and participate in the backward propagation during optimization . Intuitively , for the classes with sparse feature distribution , the corresponding radii of their prototypes are large , and the radii are small otherwise . We conduct extensive experiments in both natural language processing ( NLP ) and computer vision ( CV ) to evaluate the effectiveness of big prototypes . Specifically for NLP , we choose widely used benchmarks in the few-shot named entity ( Ding et al. , 2021b ) and relation extraction ( Han et al. , 2018 ; Gao et al. , 2019b ) . For CV , we use classical image classification datasets ( Vinyals et al. , 2016 ) in experiments . The results demonstrate that , with only a few additional parameters introduced , such modeling significantly outperforms the baseline . Surprisingly , big prototypes perform extremely well in cross-domain few-shot relation extraction , indicating the promising ability to domain adaptation . Given that such small changes can bring huge benefits , hopefully , big prototypes can inspire new ideas for the research community of representation learning . The source code and checkpoints of our models in experiments will be publicly available for reproducibility . 2 RELATED WORK . This work is related to studies of few-shot learning and meta-learning , whose primary goal is to quickly adapt deep neural models to new tasks with few training examples . To achieve that , two branches of studies are proposed : optimization-based methods and metric-based methods . The optimization-based studies ( Finn et al. , 2017 ; Franceschi et al. , 2018 ; Ravi & Beatson , 2018 ) regard few-shot learning as a bi-level optimization process , where a global optimization is conducted to learn a good initialization across various tasks , and a local optimization quickly adapts the initialization parameters to specific tasks with few training examples by few steps of gradient descent . Compared to the mentioned studies , our work is more related to the metric-based meta-learning approaches ( Vinyals et al. , 2016 ; Snell et al. , 2017 ; Satorras & Estrach , 2018 ; Sung et al. , 2018 ) , whose general idea is to learn a metric to measure the similarity between representations and find the closest labeled example ( or a derived prototype ) for an unlabeled example . Typically , these methods learn a metric function during episodic optimization . More specifically , we inherit the spirit that uses prototypes to abstractly represent class-level information , which could be tracked back to cognitive science ( Reed , 1972 ; Rosch et al. , 1976 ; Nosofsky , 1986 ) , statistical machine learning ( Graf et al. , 2009 ) and similar to the Nearest Mean Classifier ( Mensink et al. , 2013 ) . In the area of deep learning , Snell et al . ( 2017 ) propose the prototypical network to exploit the average of example embeddings as a prototype to perform metric-based classification in few-shot learning . In Prototypical Network , prototypes are estimated by the embeddings of instances , and it is hard to find a satisfying location of the prototypes of the entire dataset . Ren et al . ( 2018 ) adapt such prototype-based networks in the semi-supervised scenario where the dataset is partially annotated . A set of prototype-based networks are proposed concentrating on the improvements of prototype estimations and application to various downstream tasks ( Allen et al. , 2019 ; Gao et al. , 2019a ; Li et al. , 2019b ; Pan et al. , 2019 ; Seth et al. , 2019 ; Ding et al. , 2021a ; Li et al. , 2020c ) . We discuss big prototypes and some other prototype-enhanced methods in Appendix C. There has also been a series of works that embed prototypes into a non-Euclidean output space ( Mettes et al. , 2019 ; Keller-Ressel , 2020 ; Atigh et al. , 2021 ) . It should be noted that these studies regard hyperspheres or other non-Euclidean manifolds as the embedding space , and our proposed method use hyperspheres to represent big prototypes and conduct metric-based classification in the Euclidean space . Therefore , the focus of our proposed big prototypes is different from the nonEuclidean prototype-based works . We evaluate the effectiveness of big prototypes in three downstream tasks across NLP and CV , including few-shot named entity recognition , few-shot relation extraction , and few-shot image classification . Another technical route to achieve promising results in these downstream few-shot tasks is to use the ability of large-scale pre-trained models ( Brown et al. , 2020 ; Han et al. , 2021b ) , or build large-scale data sets to design specific pre-training tasks ( Huang et al. , 2020 ; Soares et al. , 2019 ) ( which may face the risk of information leakage ) . These techniques are also orthogonal to our contribution . Generally , the method of big prototypes is model-agnostic . 3 BIG PROTOTYPES . This section begins with the problem setup of few-shot learning . Then , we introduce the metrics , initialization , and learning of big prototypes . Generally , unlike previous prototype-based models that use estimated dense vectors as prototypes , our approach use hyperspheres to represent the conceptlevel information for classes . One big prototype is represented by two parameters : the center and the radius of the hypersphere , which are firstly initialized via estimation and then optimized by gradient descent in an end-to-end fashion . 3.1 PROBLEM SETUP . In this work , we consider the episodic N way K shot few-shot learning paradigm . In this setting , we have a large-scale annotated training setDtrain , and our goal is to learn a model that could predict for a set of new classes Dtest , where only a few examples are labeled . In this setting , the model will be trained in episodes constructed using Dtrain and tested on episodes constructed using Dtest . Each episode contains a support set for learning S = { xi , yi } N×Ki=1 with N classes and K examples for each class , and a query set for inference Q = { x∗j , y∗j } N×K′ j=1 of examples in the same N classes . Each input data is a vector xi ∈ RL with the dimension of L and yi is an index of class label . For each input xi , let hi = fϕ ( xi ) ∈ RD denote the D-dimensional output embedding of a neural network fϕ : RL → RD parameterized by ϕ . Beyond the conventional few-shot classification , we also carry out experiments in few-shot named entity recognition . This is a sequence labeling task where each token in a sequence is asked to be labeled as if it is a part of a named entity . But as the context is extremely important for this task , the examples are sampled in sequence-level . Thus the problem setup is slightly different from the traditional N way K shot classification . We follow the strategy in Ding et al . ( 2021b ) and sample sequences in a N way K ∼ 2K shot manner ( see Appendix B ) . 3.2 METRIC . We now introduce big prototypes , which are a set of hyperspheres in the feature space to abstractly represent the intrinsic features of classes . No matter what the dimension of embedding space is , one big prototype is represented by p = ( z , r ) , where z ∈ RD is the center of the hypersphere with the same dimension to h = fϕ ( x ) , and r ∈ R is a scalar denoting the radius of the hypersphere . The central idea is to learn a big prototype for each class with limited episodic supervision , and each example in the query set ( x∗j , y ∗ j ) is predicted by measuring the distance from the embedding h ∗ to the surface of the hyperspheres . The distance of two vectors is calculated by a metric function : d : RD × RD → [ 0 , +∞ ) . In the Euclidean space , the metric being d ( h , z ) = ∥h− z∥2 . ( 1 ) And the distance d̃ from an embedding to a big prototype p is the distance from the point to the center of the hypersphere minus the radius d̃ ( x , p ) = d ( fϕ ( x ) , z ) − r = ∥h− z∥2 − r. ( 2 ) Note that in this case , the value of d̃ ( · ) may be negative , that is geometrically speaking , the point is contained inside the hypersphere , and it does not affect the calculation of loss . Although more generally , the idea is to use areas but not points in the embedding space to model prototypes , hyperspheres naturally have two obvious advantages . As stated in § 1 , one big prototype could be uniquely modeled by the center and the radius p = { z , r } . While characterizing manifolds with complex boundaries in the embedding space is difficult.s Furthermore , it is easy to calculate metrics by Equation 2 . We use the distance from a point to the surface as the metric . In this way , the center and the radius are spontaneously in the loss function and optimized . In this geometric interpretation , sparse classes will have larger learned radii , while compact classes will have smaller learned radii .

Summary: In few-shot learning, prototypes have been widely used to represent classes and then classification can be performed by computing distances to prototype representations of each class. This paper proposes to use hyperspheres to model prototypes in the feature space, instead of vector points, to enhance the expressivity of class-level information. The proposed hypersphere model only needs two sets of parameters (the center and the radius of hyperspheres), so it does not bring additional burden to the optimization calculation of the objective function. Extensive experiments in both NLP and CV are conducted to evaluate the effectiveness of the proposed model. The results shows that the proposed model significantly outperforms the baseline, and it performs well in cross-domain few-shot relation extraction.