Few-shot Learning with Big Prototypes
Authors: Ning Ding
Yulin Chen
Xiaobin Wang
Hai-Tao Zheng
Zhiyuan Liu
Pengjun Xie
Venue: ICLR 2022 Submitted (2022)
Keywords: Prototype
Few-shot Learning
Meta-learning

Abstract: Using dense vectors, i.e., prototypes, to represent abstract information of classes has become a common approach in low-data machine learning scenarios. Typically, prototypes are mean output embeddings over the instances for each class. In this case, prototypes have the same dimension of example embeddings, and such tensors could be regarded as ``points'' in the feature space from the geometrical perspective. But these points may lack the expressivity of the whole class-level information due to the biased sampling.
In this paper, we propose to use tensor fields (``areas'') to model prototypes to enhance the expressivity of class-level information. Specifically, we present \textit{big prototypes}, where prototypes are represented by hyperspheres with dynamic sizes. A big prototype could be effectively modeled by two sets of learnable parameters, one is the center of the hypersphere, which is an embedding with the same dimension of training examples. The other is the radius of the sphere, which is a constant. Compared with atactic manifolds with complex boundaries, representing hypersphere with parameters is immensely easier. Moreover, it is convenient to perform metric-based classification with big prototypes in few-shot learning, where we only need to calculate the distance from a data point to the surface of the hypersphere.
Extensive experiments on few-shot learning tasks across NLP and CV demonstrate the effectiveness of big prototypes.


Under review as a conference paper at ICLR 2022
FEW-SHOT LEARNING WITH BIG PROTOTYPES
Anonymous authors
Paper under double-blind review
ABSTRACT
Using dense vectors, i.e., prototypes, to represent abstract information of classes has become a common approach in low-data machine learning scenarios. Typi- cally, prototypes are mean output embeddings over the instances for each class. In this case, prototypes have the same dimension of example embeddings, and such tensors could be regarded as "points" in the feature space from the geometrical perspective. But these points may lack the expressivity of the whole class-level information due the biased sampling. In this paper, we propose to use ten- sor fields ("areas") to model prototypes to enhance the expressivity of class-level information. Specifically, we present big prototypes, where prototypes are repre- sented by hyperspheres with dynamic sizes. A big prototype could be effectively modeled by two sets of learnable parameters, one is the center of the hypersphere, which is embedding with the same dimension of training examples. The other is the radius of the sphere, which is a scalar. Compared with atactic manifolds with complex boundaries, representing hypersphere with parameters is immensely easier. Moreover, it is convenient to perform metric-based classification with big prototypes few-shot learning, where we only need calculate the distance from a data point to the surface of the hypersphere. Extensive experiments on few-shot learning tasks across NLP and CV demonstrate the effectiveness of big prototypes.
INTRODUCTION
Learning from few examples, i.e., few-shot learning, receives increasing attention in modern deep learning. On the one hand, constituting cogmition of novel concepts with few instances is a crucial way for machines to imitate human intelligence. On the other hand, annotating large-scale super- vised datasets is expensive and time-consuming (Lu et al., 2020). Although traditional deep neural models have achieved tremendous success under sufficient supervision, it is still challenging to pro- duce comparable performance when training examples are limited. Hence, a series of studies are proposed to generalize deep neural networks to low-data scenarios. One crucial branch of them is meta-learning with prototypes (Reed, 1972; Nosofsky, 1986; Snell et al., 2017), where models are trained to quickly adapt to current tasks and carry out classification via metric-based comparisons between examples and newly introduced variables, prototypes of those classes.
In a general way, prototypes are designed to represent abstract class-level information and caleu- lated by taking the mean output of a few examples belonging to one same class. Thus, prototypes are represented as dense vectors with the same dimensions as the embeddings of training examples and can be regarded as center points of those classes in the embedding space. And for the queried examples that the model needs to predict, the basic idea to calculate the distances between the queried examples and prototypes and conduct classification based on the distances. Originated from Prototypical Network (Snell al., 2017). sets of derivative prototype- based methods demonstrate the effectiveness in few-shot learning (Ren et al., 2018; 8; Gao et al., 20194 19;; Allen et al., 2019; Pan et al., 2019; Ding et al., 2021a). However, prototypes are estimated from few sampled examples, which cannot uncover the overall ground truth distribution (Yang al., 2021). Such biased distribu- tions may generate biased prototypes and trigger sequent classification errors. In other words, this modeling approach of prototypes could lack the capability express the universal class-level infor- mation. Hence, to enhance such expressibility of prototypes, we propose to use areas, i.e., tensor fields, rather than points in the embedding space to represent the class-level information.
In this paper, we propose big prototypes to use hyperspheres in the feature space to abstract the class-level information, and the feature points can be distribued inside around the big prototypes.

---

Under review as a conference paper at ICLR 2022
choter
Sunpled Data
Samppeed Data
matic
Ground Distribution
Ground Truth Distribution
Initialization
Learning
Figure 1: The illustration of our proposed big prototypes, where the data is sampled in 5-shot. The star symbol denotes the center of the hypershpere, the solid triangle denotes the sampled examples and the dotted triangle denotes other examples in the whole dataset. The green solid line denotes the distance from a data embedding to the surface of the hypersphere. The left part illustrates the initialization stage, where the center and radius are estimated by the sampled data, and the the right part illustrates the learning stage, where the center and radius are simultancously optimized. We can see that the location of the center and the value of the radius may change during the learning process.
Such modeling is equipped with two obvious advantages: easy to model and easy to caleulate the distances. On the one hand, even if we attempt to use areas but not points to represent class-level information, it is difficult to explicitly characterize manifolds with complex boundaries in deep learning. But via hyperspheres modeling, we can obtain a big prototype only through two sets of parameters: the center and the radius of hyperspheres. On the other hand, hyperspheres are suitable for calculating the distances in Euclidean space. We can simply calculate the distance from one feature point to the surface the hypersphere to perform metric-based classification, which is also difficult for other manifolds. Moreover, it is easy to combine these two advantages in few-shot learning, the distances from one feature point to the surface of big prototype can be formalized as the distance from the point to the center of the hypersphere minus the radius. Thus, both radius and hypersphere center can appear in the loss function and participate in the backward propagation during optimization. Intuitively, for the classes with sparse feature distribution, the corresponding radii their prototypes are large, and the radii are small otherwise.
We conduct extensive experiments in both natural language processing (NLP) and computer vision (CV) to evaluate the effectiveness of big prototypes. Specifically for NLP, we choose widely used benchmarks in the few-shot named entity (Ding et al., 2021b) and relation extraction (Han et al., 2018; Gao et al., 2019b). For CV, we use classical image classification datasets (Vinyals et al., 2016) in experiments. The results demonstrate that, with only few additional parameters intro- duced, such modeling significantly outperforms the baseline. Surprisingly, big prototypes perform extremely well in cross-domain few-shot relation extraction, indicating the promising ability to do- main adaptation. Given that such small changes can bring huge benefits, hopefully, big prototypes can inspire new ideas for the research community of representation learning. The source code and checkpoints of our models in experiments will be publicly available for reproducibility.
RELATED WORK
This work is related to studies of few-shot learning and meta-learning, whose primary goal is to quickly adapt deep neural models to new tasks with few training examples. To achieve that, two branches of studies are proposed: optimization-based methods and metric-based methods. The optimization-bn based studies (Finn al., 2017; Franceschi et al., 2018; Ravi & Beatson, 2018) regard few-shot learning a bi-level optimization process, where global optimization is conducted to learn good initialization across various tasks, and a local optimization quickly adapts the initial- ization parameters to specific tasks with few training examples by few steps of gradient descent.
Compared to the mentioned studies, our work is more related to the metric-based meta-learning approaches (Vinyals et al., 2016; Snell al., 2017; Satorras & Estrach, 2018; Sung et al., 2018), whose general idea to learn a metric to measure the similarity between representations and find the closest labeled example (or derived prototype) for unlabeled example. Typically, these
2

---

Under review as a conference paper at ICLR 2022
methods learn a metric function during episodic optimization. More specifically, we inherit the spirit that uses prototypes to abstractly represent class-level information, which could be tracked back to cognitive science (Reed, 1972; Rosch et al., 1976; Nosofsky, 1986), statistical machine learning (Graf et al., 2009) and similar to the Nearest Mean Classifier (Mensink et al., 2013). In the area of deep learning, Snell et al. (2017) propose the prototypical network to exploit the average of example embeddings as prototype to perform metric-based classification in few-shot learning. In Prototypical Network, prototypes are estimated by the embeddings of instances, and hard to find a satisfying location of the prototypes of the entire dataset. Ren et al. (2018) adapt such prototype-basee networks in the semi-supervised scenario where the dataset is partially annotated. A set of prototype-base networks are proposed concentrating on the improvements of prototype estimations and application to various downstream tasks (Allen et al., 2019; Gao et al., 2019a; Li et al., 2019b; Pan et al., 2019; Seth et al., 2019; Ding et al., 2021a; Li et al., 2020c). We discuss big prototypes and some other prototype-enhanced methods in Appendix C.
There has also been series of works that embed prototypes into non-Euclidean output space (Mettes et al., 2019; Keller-Ressel, 2020; Atigh et al., 2021). It should be noted that these studies regard hyperspheres or other oun-Euclidean manifolds as the embedding space, and our pro- posed method use hyperspheres to represent big prototypes and conduct metric- based classification in the Euclidean space. Therefore, the focus of our proposed big prototypes is different from the non- Euclidean prototype-based works. We evaluate the effectiveness of big prototypes in three down- stream tasks across NLP and CV, including few-shot named entity recognition, few-shot relation ex- traction, and few-shot image classification. Another technical route to achieve promising results in these downstream few-shot tasks is use the ability of large-scale pre-trained models (Brown al., 2020; Han et al., 2021b), or build large-scale data sets to design specific pre-training tasks (Huang al., 2020; Soares et al., 2019) (which may face the risk of information leakage). These techniques are also orthogonal to our contribution. Generally, the method of big prototypes is model-agnostic.
3 BIG PROTOTYPES
This section begins with the problem setup of few-shot learning. Then, we introduce the metrics, initialization, and learning of big prototypes. Generally, unlike previous prototype-bai models that use estimated dense vectors as prototypes, our approach use hyperspheres to represent the concept- level information for classes. One big prototype is represented by two parameters: the center and the radius of the hypersphere, which are firstly initialized via estimation and then optimized by gradient descent in an end-to-end fashion.
3.1 PROBLEM SETUP
In this work, we consider the episodic N way K shot few-shot learning paradigm. In this setting, we have a large-scale annotated training set Dirain and our goal is to learn a model that could predict for set of new classes Dies; where only a few examples are labeled.
In this setting, the model will be trained in episodes constructed using Dtrain and tested on episodes constructed using Drest- Each episode contains a support set for learning S = {x,, yi) ] N with
N classes and K examples for each class, and query set for inference Q = {x) }= N of
examples in the same N classes. Each input data is a vector x; E IR_ with the dimension of L and y: is an index of class label. For each input x,, let h; = fx(x) E TR denote the D-dimensional output embedding of a neural network fØ RL -> R parameterized by Φ.
Beyond the conventional few-shot classification, we also carry out experiments few-shot named entity recognition. This is a sequence labeling task where each token in a sequence is asked to be labeled as is a part of a named entity. But as the context extremely important for this task, the examples are sampled in seque Thus the problem setup is slightly different from the traditional N way K shot classification. We follow the strategy in Ding et al. (2021b) and sample sequences in N way K ~ 2K shot manner (see Appendix B).
3.2 METRIC
We now introduce big prototypes, which are a set of hyperspheres in the feature space to abstractly represent the intrinsic features of classes. No matter what the dimension of embedding space is, one
3

---

Under review as a conference paper at ICLR 2022
big prototype is represented by p (z, r), where z E IR same dimension to = (x)),, and r E IR is a scalar denoting the radius of the hypersphere.
P is the center of the hypersphere with the
The central idea is to learn big prototype for each class with limited episodic supervision, and each example in the query set v,) is predicted by measuring the distance from the embedding h" to the surface of the hyperspheres. The distance of two vectors is calculated by a metric function: d: IR D x R - [0, +.c). In the Euclidean space, the metric being
d(h, z) = || z]
(1)
And the distance d from an embedding to big prototype p is the distance from the point to the center of the hypersphere minus the radius
d(x, p) = =() -r
(2)
Note that in this case, the value of d(-) may be negative, that is geometrically speaking, the point contained inside the hypersphere, and it does not affect the calculation of loss. Although more generally, the idea is to use areas but not points in the embedding space to model prototypes, hy- perspheres naturally have two obvious advantages. As stated in $ 1, one big prototype could be uniquely modeled by the center and the radius p r.. While characterizing manifolds with complex boundaries in the embedding space is difficult.s Furthermore, is easy calculate metrics by Equation 2. We use the distance from point to the surface as the metric. In this way, the center and the radius are spontaneously in the loss function and optimized. In this geometric interpretation, sparse classes will have larger learned radii, while compact classes will have smaller learned radii.
3.3 INITIALIZATION
To construct big prototypes, the first step is the initialization, that is, initialize the center z and the radius of the hypersphere. We use subscripts denote indices of the classes and superscript to denote the indices of the support sets. As stated in $ 1, we cannot fathom the distribution of the whole data, thus, we initialize the parameters with the first support set of the n-th class S,
Zn
Σ
fa(x;), In
Σ d(Fo(xi),
SA
Sm
(K,,))S
(SS.DE))
where the center zn is initialized by the mean output of the embeddings of the current class like Snell et al. (2017), and rn is initialized by the average of all the metrics from the embeddings to the center. The initialization essentially approximates the center and the sparse degree of the representations, and is more secure than random initialization. In our experiments, to make it more comparable to previous works, we simply use the first single episode of each class for initialization. However, since the computational cost of this initialization strategy is small, an alternative way is to use the average of the initialized hyperspheres of multiple episodes as the final big prototype. In a K K-shot episodic training. for the first support set of the n-th class S. we have
1
T
Fn
m
where m is hyper- parameter. Equation 3 denotes the situation when m = 1. And if m is equal to the total number of episodes, establishing big prototypes degenerates to an estimation procedure.
3.4 LEARNING
Once initialized, big prototype will participate in the training process, where its center and radius are simultaneously optimized. A query point x is classified by softmax over the metrics to the big prototypes calculated by Equation 2
exp(F. h Zm (2)
ply n|x) Σn exp(Fnr - h Zn 12
(5)
And the parameters of f and big prototypes are optimized by minimizing the metric-based cross- entropy objective function:
L = log logr(y(x,p)
(6)
4

---

Under review as a conference paper at ICLR 2022
Equation 5 explains the combination of the two advantages of big prototypes, where d is calculated as r and z will participate in the optimization. The parameters the neural network are optimized along with the centers and radii of big prototypes through gradient descent. To sum up, in the initialization stage, 2 and r are estimated by the embeddings of examples. And in the learning stage, z and are optimized by an independent optimizer, because the final location and shape of the hyperspheres need to serve the performance of classification. Algorithm I expresses the initialization and learning stages of big prototypes in pseudocode.
The goal of few-shot learning is to learn new classes with a handful of examples. There are two strategies handle the unseen classes at the episodic evaluation stage. Since there still exist sup- port set and query set in evaluation, we could strictly follow the intialization-leariin paradigm. Alternatively, we could directly estimate the big prototype like vanilla prototypical network. Keep- ing aligned with the vanilla prototypes for fair comparison, we use the latter strategy in experiments.
Algorithm 1: Training process of big prototypes. fo is the feature encoder, T is the total number of samples in the training set, Noaal the total number of classes in the training set, N is the number of classes for support and query set, K is the number of examples per class in the support set, K' is the number of examples per class in the query set, and M is MANDOMMSMPPEL
N) denotes a set of N elements chosen uniformly at random from set S, without replacement.
parameer
Input: Training data Drain { (x1, y1), (xr, yr) where each yi E {1...., Nooia ) Dx denotes the subset of 'D containing all elements (x,, y4) such that y; = k
Output: The updated encoder f
// Ini iniiialf iaaioon phase
for n 1 to Nowl do
Sn f RANDOMS AMPLE (D, K)
Zn
Σ
(x:),
ES
Fn
B
Σ d(so(x:), kn);
(SS
// Learning phase
for i 1 to M do V <- RANDOMSAMPLEEC for (I...., N) do
N. L t-
On + RANDOMSAMPLLED,, K')
C+ I Σ (difo(x:); zn) - -nn + log Exp((Fr (xfo(x)),2.))
NK
(X.
UPDATE Ž. F, fo w.I. L
EXPERIMENTS
To evaluate the effectiveness of the proposed method and compare it with the baselines, we con- duct experiments on three common few-shot learning tasks both in NLP and CV, including few-shot named entity recognition (NER), few-shot relation extraction (RE) and few-shot image classifica- tion. This section describes and presents the results of each of the three tasks. Note that the goal of big prototypes is provide new insight for representation learning of concept-level information, not to lead in all the leaderboards. Since big prototypes are model-agnostic, the central idea of the design for the experiments is to make comparisons with the related baselines. We use original proto- types as our main baseline and give less attention to other orthogonal techniques like the structures of neural models or the task task-specific pre-training.
4.1 FEW-SHOT NAMED ENTITY RECOGNITION
We assess the effectiveness of big prototypes on NLP, specifically, the first task is few-shot named entity recognition (NER) and the dataset is FEW-NERD (Ding et al., 2021b). Different from typical instance-leval classification, few-shot NER is a sequence labeling task, where labels may share structural correlations. NER is the first step in automatic information extraction and the construction of large-scale knowledge graphs. Quickly detecting fine- fine-grained unseen entity types is of significant importance in NL.. To capture the latent correlation, many recent efforts in this field use large pre-
5

---

Under review as a conference paper at ICLR 2022
FEW-NERD (INTRA)
FEW-NERD (INTER)
Sctting
Eva.
NNShot
Prototypes
Big Prototypes
NNShot
Prototypes
Big Prototypes
28.95
18.58
40.18
50.400.60
38. 50.50
53.362.74
5 way 1~2 shot
R
33.40
31.83
26.962.07
58.84..13
52.60 6s
51.12494
F
31.011.21
23.45%
32.261.94
54.29%.40
44.580.26
52.092.49
32.872.45
35.87a.69
48.77a.79
45.80
53.731.77
62.26
5 way 5~100 shot
R
39. 19.177.17
50.50
53.262.60
56.452.93
64,992
69.32,
35.742.36 F
41.930.55
50.881.01
50.563.33
58.80142
65.590.50
20.38p.22
16.52..52
26.062.40
42.742.u
32,59p
45,380.49
10 way
R
23.63p.53
24.60u.72
22.32u.54
52.161.76
48,91294
43.22
1~2 shot
F
21.8 21.88p.23
19.76u,59
24.02
46.981.0 1696
39.096s7
44.260.5
P
25. 25.466.63
28.93g.82
38.943.39
45. 15..1
47.930.5
56.381.99
way
R
30.321.71
43.08
46.712.18
56.050.37
61.791.73
65.84 .61
5~~00 shot
F
27.671.06
34.61g.59
42.463.01
50.006.36
53.97038
60.73
Table I: Performance of state-of-the-art methods on FEW-NERD (INTRA) and FEW- NERD(INTER). In this table, is precision, R is recall and F refers to Fi-score.
trained language models (Han et al., 2021a) like BERT (Devlin et al., 2019) as backbone model and have achieved remarkable performance. The original prototypical network has also been applied to this task (Li et al., 2020b; Huang et al., 2020; de Lichy et al., 2021). We report the dataset, baselines, and experimental details in Appendix A.
Results Table I shows the performance of current state-of-art models on FEW-NERD. Overall, big prototypes have a considerable advantage over prototypes, with an increase of at least 5% in fl- score across all settings. The success on both datasets demonstrates that big prototypes can learn the general distribution pattern of entities across different entity types and thus can greatly improve the performance when little information is shared between training and test set. It also be observed that a large portion of the improvement comes from the increase in precision, showing that big prototypes can better distinguish entities from context. It is possibly because context words are much diverse and modeling them with a hypersphere as in big prototypes is more plausible than a single point as in prototypes. With respect to number of shots, big prototypes are more advantageous when larger shots are provided and becomes the new state-of-art in the 5~~10 shot setting. For the comparison with NNShot, big prototypes remains the superiority under the settings of high-shot (5~10), outperforming it by at least 10% of F1-scores. Interestingly, the performances of NNShot and big prototypes are comparable when comes to low-shot. This because in the sequece labeling task, it is more difficult to infer the class-level information from very limited tokens. And the use of big prototypes and tokens for metric-based classification play a similar role. And the mediocre performance on low shot settings indicates that the hypersphere parameters are learned more effectively when more shots are available. Also since in strict 1-shot setting the initialization of the radius would be zero, we believe a good initialization strategy is important for big prototypes.
4.2 FEW-SHOT RELATION EXTRACTION
The other common NLP task is relation extraction (RE), which aims at correctly classifying the relation between two given entities in sentence. This is fundamental task in information extrac- tion. RE is an important form of learning structured knowledge from unstructured text. We use FewRel Han al. (2018) and FewRel 2.0 Gao al. (2019b) as the datasets. In real-world datasets, many of the relations are long-tailed and thus cannot be identified accurately under the common supervised setting. Traditional methods often alleviate the problem with distant supervision, which would result in wrong labels. Recent approaches have applied few-shot learning models on the task to learn from a handful of samples, which yield promising results (Gao et al., 2019a). We report the datasets, baselines and experimental details in Appendix A.2.
Results Table presents the results on two FewRel tasks, we did not compare methods that use additional data conduct tpsk-specific encoder pre-training. Big prototypes generally perform better than prototypes across all settings. In terms of backbone models, when combining with pre- trained models like BERT, big prototypes can yield larger advantage against prototypes. It shows

---

Under review as a conference paper at ICLR 2022
Few 1.0
Model
way 1 shot
way shot
10 way 1 shot
10 way 5 shot
Meta Net (Munkhdalai & Yu, 2017)
64.46 0.54
80.57 + 0.48
53,96 + 0.56
69.23 0.52
SNAIL. (Mishra et al., 2017)
67.29 0.26
79.40 + 0.22
53.28 + 0.27
68.33 0.26
GNN CNN (Satorras & Estrach, 2018)
81.28
4
64.02 0.77
GNN BERT (Satorras & Estrach, 2018)
75.66 + 0.00
89.06 0.00
70.08 0.00
76.93+
Proto- MATT (Gao et al., 2019a)
76,30 a 0.06
64.13
83.05 : 0.05
MLMAN (Ye & Ling, 2019)
82.98 # 0.20
92.66 + 0.09
87.29 = 0.15
Proto CNN
69.20 0.20
84.79 0.16
56,44
75.55 0.19
Big Proto CNN (Ours)
66.05 4 3.44
87.31 + 0.93
56.74 1.06
77.87 = 2.60
Proto BERT Big Proto BERT (Ours)
80,68 = 0.28
89.60 + 0.09
7
82.89 = 0.11
84.34 + 1.23
93.42 # 0.50
77.24 + 6.05
88.71 0.31
FewRel 2.0 Domain Adaptation
Proto-ADV CNN (Wang et al., 2018)
42.21 + 0.09
58.71 + 0.06
28.91 0.10
44.35 = 0.09
Proto-ADV BERT (Gao al., 2019b)
41.90 0.44
54.74 0.22
27.36 0.50
37.40 0.36
BERT-pair (Gao al., 2019b)
56.25 + 0.40
67.44 + 0.54
53.17 # 0.09
Proto CNN
35,09 + 0.10
49.37 + 0.10
22,98 + 0.05
35,22 0.06
Big Proto CNN (Ours)
36.41 # 1.43
55.50 + 1.42
22.11 # 0.58
40.82 # 2.50
Proto BERT
40.12 + 0.19
51.50 + 0.29
26.45 + 0.10
36,93 = 0.01
Big Proto BERT (Ours)
59.03 + 3.68
74.85 + 4.59
45.88 + 7.43
61.61 + 4.69
Table 2: Accuracies of Prototypes and Big Prototypes FewRel 1.0 and FewRel 2.0 under 4 differ- ent settings. Separate blocks show the direct comparisons between prototypes and big prototypes.
that the distribution assumption of big prototypes may be closer to the real data distribution, which boosts the finetuning phase. Meanwhile it also sheds light the untapped potential ability of large pre-trained models and stresses that the assumptions made about data distribution may help us unlock the potential. It is also worth noting that big prototypes achieves outstanding performance on Domain Adaptation task, with an increase of about 20% in accuracies when using BERT. The success signals the great capability of big prototypes in transfer learning and also points to the importance of better abstraction of data distribution in transfer learning investigation. Although big prototypes yield surprisingly performance the domain adaptation task, the standard errors are also high. This observation suggests that when the domain shifts, the unstable encoding of examples by language model may lead to different results. Compared to other meta learning baselines, big prototypes also yield promising results on all the settings.
4.3 FEW-SHOT IMAGE CLASSIFICATION
Image classification is one of the most classical tasks in few-shot learning research. Seeking better solution for few-shot image classification is beneficial in two ways: (1) to alleviate the need for data augmentation, which is standard technique to enrich the labeled data by performing transforma- tions on a given image; (2) to facilitate the application where the labeled image is expensive. We use minilmageNet (Vinyals al., 2016) as the dataset in our experiment. The dataset, baselines, and experimental details are reported in Appendix A.3.
Results Table shows the result on minilmageNet few-shot classification under 2 settings. Big prototypes substantially outperform prototypes in most settings, displaying their ability in modeling class distribution of images. We observe that compared NLP, the results of image classification are more stable both for vanilla prototypes and big prototypes. This observation may indicate the differ- ence in encoding between the two technologies. Token representations in BERT are contextualized and changeable around different contexts, yet the image representation produced by deep CNNs aims to thoroughly capture the global and local features. Different from the NLP experiments, we use 5-way 1-shot and 5-way-5-shot settings in image classification. Under the 5-way 5-shot setting, the improvements of big prototypes are significant. Compared to the main baseline of prototypes, big prototypes could yield considerable improvements with all the three backbones. The effective- ness of our method is also demonstrated by the comparisons with other previous few-shot learning methods. With different backbones, big prototypes achieve better comparable results than the pre-
7

---

Under review as a conference paper at ICLR 2022
vious state-of- the-hhherrtt models. In partieular, with the WideResNet (Zagoruyko & Komodakis, 2016) backbone, big prototypes yield the best results of all the compared methods, suggesting that as the representational power of the model improves, does the expressive capability of big prototypes. Compared to the 5-shot setting, the big prototypes achieve less boost in the 1 1-shot setting of Convnet and ResNet-12 (He et al., 2015), the phenomenon consistent with the intuition that more examples would be more favourable to the learning of radius.
minilmageNet
Model
Backbone
way shot
5 way 5 shot
IMP (Allen et al., 2019)
ConvNet
33.30 + 0.71
65,88 0.71
Prototypes (Snell et al., 2017)
ConvNet
46.44 = 0.60
63.72 0.55
CovaMNet (I.i et al., 2019a)
ConvNet
51.83 + 0.64
65.65 0.77
Big Prototypes (Ours)
ConvNet
50.21 = 0.31
66.48 + 0.71
SNAIL (Mishra et al., 2017)
ResNet-12
55.71 = 0.99
68.88 + 0.92
Prototypes (Snell et al., 2017)
ResNet-12
53.81 0.23
75.68 0.11
Variational FSL (Zhang al., 2019)
ResNet-12
61.23 0.226
0.17
Prototypes TRAML (Li et al., 2020a)
ResNet-12
60.31
0.59
Meta-baseline (Chen et al., 2021)
ResNet-12
0.21
79.26 0.117
Big Prototypes (Ours)
ResNet-12
59.65
78.24 0.47
Prototypes (Snell et al., 2017)
WideRes
59.09 + 0.64
79.09 0.46
Activation to Parameter (Qiao et al., 2018)
59.60 2e
0.41
73.74 0.19
LEO (Rusu et al., 2018)
2ideResNett22-100
61.76 + 0.08
77.59 + 0.12
Big Prototypes (Ours)
28-
63.78 + 0.63
81.29 0.46
Table 3: Accuracies with 95% confidence interval of big prototypes and baselines on minilmageNet
4.4 ANALYSIS
Radius Dynamics In this subsection, the mechanism of big prototypes will be empirically analyzed. We demonstrate the mechanism of big prototypes by illustrating the change of radius for one spe- cific hypersphere. In the learning phase, the radius of a big prototype is changing according to the "density" of the sampled episode, which could be quantified by the mean distance of samples the corresponding prototype center. Practically, since the sampling is random, the value of the mean distance may oscillate at high frequency in this process, and the radius changes accordingly.
To better visualize the changing of radius along with the mean distance at each update, for each round of training we fix one specific class as the anchor class for mean distance and radius recording and apply a special sampling strategy at each episode. Specifically, we take FewRel training data and train on the way 5 shot setting with CNN encoder. While training, each episode contains the anchor class and 4 other randomly sampled classes. Training accuracy is logged every 50 steps. After a warmup training of 500 steps, we sample "good" or "bad" episodes for every 50 steps alternatively. A "good"" episode has higher accuracy on the anchor class than the previously logged accuracy, while conversely, a "bad" episode has an accuracy lower than before. The mean distance to the prototype center and radius each episode are logged every 50 steps after the warmup.
Figure shows the changing of mean distance and radius for 8 classes during 600~2000 training steps. Although the numeric values of distance and radius differ greatly and oscillate at different scales, they have similar changing patterns. Besides, it could be observed that there is often a small time lag in the change of radius, indicating that the change of radius is brought by the change in mean distance. This is in line with our expectations and perfectly demonstrates the learning mechanism of big prototypes. The experiment provides a promising idea, if we can control the sampling strategy through priori knowledge, we may find way to learn ideal big prototypes.
Visualization We also use t-SNE (van der Maaten & Hinton, 2008) to visualize the embedding before and after training. 5 classes are sampled from training set and test set of Few-NERD dataset, respectively, and for each class 500 samples are randomly chosen to be embedded by BERT trained on 5 way 5 shot NER task. Figure 2 is a graph showing the result of embeddings in 2-dimensional space, where each distinct color represents one class. Note that for NER task the representations obtained on the token level, where the interaction between the target token and its context may
8

---

Under review as a conference paper at ICLR 2022
Position held
Publisher
Owned by
Contains territorial entitt..
0500112
origin
part
Participating
19.9
5000000
12
LOc 100
2000 tratiing steps
120
training Mads
trwiiig
Eraitiin steps
Figure 2: The illustration depicts the radius change according to the degree of sparsity of the sampled episode. Each subligure represents a selected anchor class in the FewRel dataset.
result in after training, the representations of the same class in the training set become more compressed and casier to classify.
mixed-up distribution compared instance-level embedding. It could be seen that
also worth noting that the trained BERT encoder could even generate
a more clustered representation distribution for a novel class. The visualization demonstrates the effectiveness of big prototypes training learning discriminative features, especially the success on learning novel class representation that greatly boosts model performance under few-shot settings.
Soen Classes Before Training
Seen Classes After Training
Novel Classes Before Training
Novel Classes After Training
(a)
(b)
(c)
(d)
Figure 3: t-sne visualization feature distributions. (a) illustrates the representations of seen data (in training data) before training; (b) illustrates the representations of seen data (in training data) produced by our network; (c) illustrates novel data (in test data) before training; (d) illustrates novel data (in test data) produced by network after training. Note that even after training, the neural network has never seen the novel data and their classes.
CONCLUSION
This paper proposes novel metric-based few-shot learning method, big prototypes. Unlike previ- ous prototype-based methods that use dense vectors to represent the class-level semantics, we use hyperspheres to enhance the capabilities of prototypes to express the intrinsic information of the data. It is simple model hypersphere in the embedding space than other complex manifolds, we establish two variables, the center and the radius to represent a big prototype. Such modeling is easy to implement and also empirically effective, prototype on three tasks across NLP and CV. For potential future work, big prototypes could be extended to generalized representation learning like image, word, sentence representations.
observe significant improvements than vanilla

---

Under review as a conference paper at ICLR 2022
6 ETHICS STATEMENT
This section states the ethical considerations of our work. Generally, the method of big prototypes tool for few-shot learning, making it intrinsically unbiased. The impact, no matter positive or negative, is depend on the intentions of specific applications that uses our method. Nevertheless, since few-shot systems have already been deployed real-world scenarios, is crucial to carefully choose the training data avoid the potential bias. We strongly encourage the users of our method to careful about the data selection in practice. All the data used in our research are granted for research purpose, and it will not result in privacy issues. In terms of energy saving, we will make all the checkpoints our experiments publicly available to reduce the carbon emission.
7 REPRODUCIBILITY STATEMENT
We evaluate the effectiveness of big prototypes in three tasks across NLP and CV, which are few-shot named recognition, few-hot relation extraction, and few-shot image classification ($ 4). For each ex- periment, we provide an experimental setting part to introduce details for reproduction, including data processing, hyper-parameters, the choice of the neural encoder, and the steps for training, val- idating, and testing. Please refer $ 4.1, 4.2, 4.3 for the details of the three tasks, respectively. For the specific initialization and learning of big prototypes, we provide pseudocode in Algorithm 1. The source code and the checkpoints in our work will be released.
REFERENCES
Kelsey Allen, Evan Shelhamer, Hanul Shin, and Joshua Tenenbaum. Infinite mixture prototypes for few-shot learning. Proceedings of ICML. PMLR, 2019.
Mina Ghadimi Atigh, Martin Keller-Ressel, and Pascal Mettes. Hyperbolic busemann learning with ideal prototypes. arXiv preprint arXiv:2106. 14472, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Proceedings of NeurlPS, 2020.
Yinbo Chen, Zhuang Liu, Huijuan Xu, Trevor Darrell, and Xiaolong Wang. Meta-baseline; Explor- ing simple meta-learning for few-shot learning. In Proceedings of the IEEF/CVE International Conference on Computer Vision, pp. 9072-9071, 2021.
Debasmit Das and C. S. George Lee. A two-stage approach to few-shot learning for image recogni- tion. IEEE Transactions Image Processing, 29:3336-338 33500, 2020.
Cyprien de Lichy, Hadrien Glaude, and William Campbell. Meta-learning for few-shot named entity recognition. Proceedings of the 1st Workshop on Meta Learning of ACL, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of CVPR, 2009.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT:
Pre-training of deep
bidirectional transformers for language understanding. In Proceedings of NAACL, 2019.
Ning Ding, Xiaobin Wang, Yao Fu, Guangwei Xu, Rui Wang, Pengjun Xie, Ying Shen, Fei Huang. Hai-Tao Zheng, and Rui Zhang. Prototypical representation learning for relation extraction. In Proceedings of ICLR, 2021a.
Ning Ding, Guangwei Xu, Yulin Chen, Xiaobin Wang, Xu Han, Pengjun Xie, Haitao Zheng, and Zhiyuan Liu. Few-NERD: A few-shot named entity recognition dataset. Proceedings of ACL, b021b.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta- neta-learning for fast adaptation of deep networks. In Proceedings of ICMI., 2017.
10

---

Under review as a conference paper at ICLR 2022
Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel programming for hyperparameter optimization and meta-learning In Proceedings of ICML, 2018.
Tianyu Gao, Xu Han, Zhiyuan Liu, and Maosong Sun. Hybrid attention- based prototypical networks for noisy few-shot relation classification. In Proceedings of AAAI, 2019a.
Tianyu Gao, Xu Han, Hao Zhu, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. FewRel 2.0: Towards more challenging sew-shot relation classification. In Proceedings of EMNLP, 2019b.
Arnulf BA Graf, Olivier Bousquet, Gunnar Rätsch, and Bernhard Schölkopf. Prototype classifica- tion: Insights from machine learning. Neural computation, 2009.
Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun. FewRel: A large-scale supervised few-shot relation classification dataset with state-of-the-arr evaluation. In Proceedings of EMNLP, 2018.
Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Liang Zhang, Wentao Han, Minlie Huang, et al. Pre-trained models: Past, present and future. Al Open, 202
Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, and Maosong Sun. Ptr: Prompt tuning with rules for text classification. arXiv preprint arXiv:2105. / 2021b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of CVPR, 2015.
Jiaxin Huang, Chunyuan Li, Krishan Subudhi, Damien Jose, Shobana Balakrishnan, Weizhu Chen, Baolin Peng, Jianfeng Gao, and Jiawei Han. Few-shot named entity recognition: A comprehen- sive study. arXiv preprint arXiv:2012. 14978, 2020.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of ICML, 2015.
Martin Ressel. theory of hyperbolic prototype learning. arXiv preprint arXiv:2010.07744, 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of ICLR, 2017.
Aoxue Li, Weiran Huang, Xu Lan, Jiashi Feng, Zhenguo Li, and Liwei Wang. Boosting few-shot learning with adaptive margin loss. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 12576-12584, 2020a.
Jing Li, Billy Chiu, Shanshan Feng, and Hao Wang. Few-shot named entity recognition via meta- learning. IEEE Transactions on Knowledge and Data Engineering, 2020b.
Junnan Li, Pan Zhou, Caiming Xiong, and Steven Hoi. Prototypical contrastive learning of unsuper- vised representations. In Proceedings of ICLR, 2020c.
Wenbin Li, Jinglin Xu, Jing Huo, Lei Wang. Gao Yang, and Jiebo Luo. Distribution consistency based covariance metric networks for fhw-shot learning. In AAAI, 2019a.
Xiao Li, Min Fang, Dazheng Feng, Haikun Li, and Jinqiao Wu. Prototype adjustment for zero shot classification. Signal Processing: Image Communication, 2019b.
Jiang Lu, Pinghua Gong, Jieping Ye, and Changshui Zhang. Learning from very few samples: A survey. arXiv preprint urXiv:2009.02655 2020.
Puneet Mangla, Nupur Kumari, Abhishek Sinha, Mayank Singh, Balaji Krishnamurthy, and Vi- neeth N Balasubramanian. Charting the right manifold: Manifold mixup for few-shot learning. In Proceedings of WACV, pp. 2218-22111 1112-227,,22 2020.
Thomas Mensink, Jakob Verbeek, Florent Perronnin, and Gabriela Csurka. Distance- based image classification: Generalizing to new classes at near ysis and machine intelligence, 2013.
near-zero zero cost. IEEE transactions on pattern anal-
11

---

Under review as a conference paper at ICLR 2022
Pascal Mettes, Elise van der Pol, and Cees Snoek. Hyperspherical prototype networks. Proceedings of NewrlPS, 2019.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- learner. arXiv preprini vrxivv:707.03// 2017.
Tsendsuren Munkhdalai and Hong Yu. Meta networks. In International Conference on Machine Learning, pp. 2554-2563. PMLR, 2017.
Robert M Nosofsky. Attention, similarity, and the identifcatioon-categriiaaiiooi relatiosshpp............. Journal of experimental psychology: General, 1986.
Yingwei Pan, Ting Yao, Yehao Li, Yu Wang, Chong- Wah Ngo, and Tao Mei. Transferrable proto- typical networks for unsupervised domain adaptation. In Proceedings of CVPR, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. 2019.
Siyuan Qiao, Chenxi Liu, Wei Shen, and Alan L. Yuille. Few-shot image recognition by predicting parameters from activations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7229-7238, 2018
Sachin Ravi and Alex Beatson. Amortized bayesian meta-learning, Proceedings of ICLR, 2018.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In Proceedings of ICLR, 2017.
Stephen K Reed. Pattern recognition and categorization, Cognitive psychology, 3(3), 1972.
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenenbaum, Hugo Larochelle, and Richard S Zemel. Mcta-learning for semi-supervised few-shot classifica- tion. In Proceedings of ICLR, 2018.
Eleanor Rosch, Carolyn B Mervis, Wayne D Gray, David M Johnson, and Penny Boyes-Braem. Braem. Basic objects in natural categories. Cognitive psychology, 1976.
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osin- dero, and Raia Hadsell. Meta-learning with latent embedding optimization. arXiv preprint arXiv:1807.05960. 2018.
Victor Garcia Satorras and Joan Bruna Estrach. Few-shot learning with graph neural networks. In Proceedings of ICLR, 2018.
Harshita Seth, Pulkit Kumar, and Muktabh Mayank Srivastava. Prototypical metric transfer learning for continuous speech keyword spotting with limited training data. In Proceedings of SOCO. Springer, 2019.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Proceedings of NewrIPS, 2017.
Livio Baldini Soares, Nicholas FizzGerald, Jeffrey Ling, and Tom Kwiatkowski. Matching the blanks: Distributional similarity for relation learning. In Proceedings of ACL, 2019.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of CVPR, 2018
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 986))2559-22055 2008.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Proceedings of NeurIPS, 2016.
Xiaozhi Wang. Xu Han, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Adversarial multi- multi-lingual neural relation extraction. In Proceedings of COLING, pp. 166611666, 2018.
12

---

Under review as a conference paper at ICLR 2022
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's 's transformers: State-of- the-art natural language processing. In Proceedings of EMNLP, 2020.
Shuo Yang, Lu Liu, and Min Xu. Free lunch for few-shot learning: Distribution calibration. In Proceedings of ICLR, 2021.
Yi Yang and Arzoo Katiyar. Simple and effective few-shot named entity recognition with structured nearest neighbor learning. Proceedings of EMNLP. 2020.
Zhi-Xiu Ye and Zhen-Hut Ling. Multi-level matching and aggregation network for few-shot relation classification. In Proceedings of ACL. pp. 2872-2881 July 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.
Jian Zhang, Chenglong Zhao, Bingbing Ni, Minghao Xu, and Xiaokang Yang. Variational few-shot learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 1685-1694, 2019.
13

---

Under review as a conference paper at ICLR 2022
A EXPERIMENTAL DETAILS
This section reports the experimental details of all the three tasks in our evaluation. All the experi- ments are conducted on NVIDIA A100 and V100 GPUs with CUDA.
A.1 EXPERIMENTAL DETAILS FOR FEW-SIOTT NAMED ENTITY RECOGNITION
Dataset The experiment is run on FEW-NERD dataset (Ding et al., 2021b). It is a large-scale NER dataset containing over 400,000 entity mentions, across 8 coarse -rrained types and 66 fine- -eeggradedd types, which makes it an ideal dataset for few-shot learning. It has been shown that existing methods including prototypes are not effective enough on this dataset.
Baselines NNShot (Yang & Katiyar, 2020) is token-level metric- method that is specifically designed for few-shot labeling. Note that the main baseline here is the Proto method, which adapts the prototypical network on few-shot named entity recognition.
Implementation Details We run experiments under four settings on the two released benchmarks, FEW-NERD (INTRA) and FEW-NERD (INTER). Specifically, we use uncased BERT as the backbone encoder and lc-4 as the encoder learning rate. We manually tune the learning rate for the radius parameter and the best result is obtained with 10. AdamW is used as the BERT optimizer and Adam (Kingma & Ba, 2017) is used to optimize prototype radius. The batch size is set across all settings. All models are trained for 10000 steps and validated every 1000 steps. The re- sults are reported on 5000 steps of the test episode. For each setting, we run the experiment with different random seeds and report the average metrics including precision, recall, fl-score, and the standard error for each. We use PyTorch (Paszke et al., 2019) and huggingface transformers (Wolf et al., 2020) to implement the backbone encoder BERT base-
A.2 EXPERIMENTAL DETAILS FOR FEW-SHOT RELATION EXTRACTION
Dataset We adopt the FewRel dataset (Han al., 2018; Gao al., 2019b), relation extraction dataset specifically designed for few-shot learning. FewRel has 100 relations with 700 labeled instances each. The sentences are extracted from Wikipedia and the relational entities are obtained from Wikidata. FewRel 1.0 (Han et al., 2018) is released as a standard few-shot learning benchmark. FewRel 2.0 (Gao et al., 2019b) adds domain adaptation task and NOTA task on top of FewRel 1.0 with the newly released test dataset on PubMed corpus.
Baselines In addition to the main baseline, prototypical network Snell et al. (2017), we also choose the following few-shot learning methods as the baselines in relation extraction. (1) Proto-HATT Gao et al. (2019a) is neural model with hybrid prototypical attention. (2) MLMAN Ye & Ling (2019) is multi-level matching and aggregation network for few-shot relation classification. Note that Proto- HATT and MLMAN are not modcl-agnostic. (3) GNN Satorras & Estrach (2018) is a meta-learning model with graph neural network as the prediction head. (4) SNAIL Mishra et al. (2017) is meta- learning model with attention mechanisms. (5) Meta Net Munkhdalai & Yu (2017) is a classical meta-learning model with meta information. (6) Proto-ADV Gao et al. (2019b) is prototype-based method enhanced by adversarial learning. (7) BERT-pair Gao et al. (2019b) is a strong baseline that uses BERT for few-shot relation classification. We re-run all the baselines, except for MLMAN and report the corresponding performances.
Implementation Details The experiments are conducted on FewRel 1.0 and FewRel 2.0 domain adaptation tasks. For FewRel 1.0, we follow the official splits in Han et al. (2018). For FewRel2.0, we follow Gao al. (2019b), training the model on wiki data, validating on SemEval data, and test- ing on the PubMed data. We use the same CNN structure and BERT as encoders. The learning rate for big prototype radius is 0.1 and 0.01 for CNN and BERT encoder, respectively. Adam (Kingma & Ba, 2017) is used as radius optimizer. We train the model for 10000 steps, validate every 1000 steps, and test for 5000 steps. The other hyperparameters are the same as the original paper.
A.3 EXPERIMENTAL DETAILS FOR FEW-SHOT IMAGE CLASSIFICATION
Dataset miniImageNet (Vinyals et al., 2016) is used as a common benchmark for few-shot learning.
The dataset is extracted from the full ImageNet dataset (Deng al., 2009), and consists of 100
14

---

Under review as a conference paper at ICLR 2022
randomly chosen classes, with 600 instances each. Each image is of size 3x84x84. We follow the split in (Ravi & Larochelle, 2017) and use 64, 16, and 20 classes for training, validation, and testing.
Baselines The basclines we choose are as follows: (1) Prototypical network (Snell et al., 2017) is main baseline; (2) IMP (Allen al., 2019) is prototype-enhane
method that models infinite mixture of prototypes for few-shot learning; (3) CovaMNet (Li et al., 2019a) is a few-shot learn- ing method that uses covariance to model the distribution information enhance few-shot learning peformance. (4) SNAIL (Mishra et al., 2017) is an attention-based classical meta-learning method; (5) Variational FSL (Zhang et al., 2019) is a variational Beyasian framework for few-shot learning, which contains pre- pre-training stage; (6) Activation to Parameter (Qiao et al., 201 8) predicts parame- ters from activations in few-shot learning; (7) LEO (Rusu al., 2018) optimizes latent embeddings (8) TRAML (Li et al., 2020a) uses adaptive margin loss to boost few-shot learning, and Prototypes TRAML is strong baseline recent years.; (9) Meta-baseline Chen et al. (2021) is pre-training & tuning method that serves strong bascline in few-shot learning.
for few-shot learning.
Implementation Details The experiments are conducted on 5 way shot and way 5 shot settings. To ensure validity and fairness, we implement big prototypes with various backbone models includ- ing CNN, ResNet-12 and WideResNet (Zagoruyko & Komodakis, 2016) to make it comparable to all baseline results, and also re-run some of the baselines including prototypical network (Snell et al., 2017), infinite mixture prototypes (Allen et al., 2019), and CovaMNet (Li et al., 2019a) under settings based on their original code. Other baseline results are taken from the original pa- per. Each model is trained on 10,000 randomly sampled episodes for 30~40 epochs and tested on 1000 episodes. The result is reported with 95% confidence interval. Note that both ResNet-12 and WideResNet (Zagoruyko & Komodakis, 2016) are pretrained on the training data, where the pre- trained ResNet-12 is identical to Chen al. (2021) and the pretrained WideResNet follows Mangla et al. (2020). The CNN structure is the same as Snell et al. (2017), which is composed of 4 convolu- tional blocks each with a 64-filter x convolution, a batch normalization layer (loffe Szegedy, 2015), ReL.U nonlinearity, and 2 x 2 max-pooling layer. We use SGD optimizer for the encoder and Adam (Kingma & Ba, 2017) optimizer for the prototype radius. The learning rate for the back- bone model is 1e-3. The learning rate for radius is manually tuned and the reported result has learning rate of 10. At the training stage, the prototype center is re-initialized at each episode as the mean vector of the support embeddings.
B K~2K SAMPLING FOR FEW-NERD
In the sequence labeling task FEW-NERD, the sampling strategy is slightly different from other classification task. We follow the strategy of the original paper Ding et al. (2021b) and report it in Algorithm 2.
Algorithm 2: Greedy N- N-way K~2K -hot sampling algorithm for FEW-NERD
Input: Dataset X, Label set V, N, Output: output result
S Ø; // Init the support set // Init the count of entity types
for i to do
Count 0;
repeat
Randomly sample (x, y) E X: Compute [Count] and Count; after update: Coonnt > or [Coun[i]] > 2K then Continue:
else
S S=SU(z y)
Update Count;
until Count; 2 K for i to N;
15

---

Under review as a conference paper at ICLR 2022
C OTHER PROTOTYPE-ENHANCER METHODS
Here, we discuss the difference between big prototypes with four prototype-enhanced methods in few-shot learning: infinite mixture prototypes (Allen al., 2019), CovaMNet (Li et al., 2019a), Variational Few-Shot Learning (Zhang et al., 2019), and Two-Stage Approach (Das & Lee, 2020).
Infinite mixture prototypes (Allen et al., 2019) model each class indefinite number of clusters, and the prediction is obtained by computing and comparing the distance to the nearest cluster in each class. This method is an intermediate model between prototypes and nearest neighbor model, whereas big prototypes alleviate the seeegennraliiziionn problem of vanilla prototypes with a single additional parameter that turns single point modeling into hypersphere, The essential prototype- based feature of big prototypes allows faster computation and easier training.
CovaMNet (Li al., 2019a) calculates local variance for each class based on support samples and conduct metric-based learning via covariance metric, which basically evaluates the cosine similarity between query samples and the eigenvectors of local covariance matrix. To ensure the non- singularity of covariance matrix, the feature of each sample is represented with matrix, generated by a number of local descripters, with each extracting a feature vector. Compared to big prototypes, both methods attempt model more variance based vanilla prototypes, while the idea of big prototypes is more straightforward and requires fewer parameters. On the other hand, the multi- channel features adopted by CovaMNet are less natural for NLP tasks.
Variational Few-Shot Learning (Zhang et al., 2019) tackles few-shot learning problem under a bayesian framework. In order to improve single point based estimation, class-specific latent vari- able representing the class distribution is introduced and is assumed to be Gaussian. The method parameterizes the mean and variance of the latent variable distribution with neural networks that take the feature of single instance as input. The learning and inference process are both conducted on the latent variable level. The method adopts variational inference and is built on modeling distribu- tion latent variable, where the metric calculation highly relies on the Gaussian assumption. Big prototypes, on the other hand, models the distribution with a center vector and a radius parameter in the actual embedding space, which is more tangible and easier to calculate.
Two-Stage Approach first trains feature encoder and variance estimator on training data episodic manner with extracted absolute and relative features. Then in the second stage, training data split into "novel" class and base class, novel class prototypes are learned from both sample mean and base class features. The classification is carried out with the integrated prototypes. This method improves on vanilla prototypes by extracting more features and combining information from base class, but still follows kinle-point-baseee metric learning. Big prototypes extends single point to hypersphere in the embedding space, and therefore better captures within-class variance.
16


Reviews:
[{"review_id": "XFFF-edrY_d", "reviewer": "Reviewer_bJMv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper extends and mainly compares with prototypical network published in 2017. \nHowever, there are many works trying to improve prototypical network to model the prototypes more appropriately in literature. To name a few, see the following \n\n[1] Distribution Consistency Based Covariance Metric Networks for Few-Shot Learning, AAAI-19\n[2] Variational Few-Shot Learning, ICCV-19\n[3] A Two-Stage Approach to Few-Shot Learning for Image Recognition, TIP-19\n\nAs for ``Ren et al., 2018; Gao et al., 2019a; Allen et al., 2019; Pan et al., 2019; Ding et al., 2021a\" mentioned in introduction, can you compare with them empirically? Instead of just saying they \"cannot uncover the overall ground truth distribution\". \nWhy using hyperspheres  is better than using a distribution? Any theoretical analysis? Especially, please note that the hypersphere is still regular, while a distribution can be complicated. \nWithout fully discuss and empirically compare with the above-mentioned works, I cannot judge whether the performance gain is meaningful. \n", "summary_of_the_paper": "The paper proposed to represent prototypes by hyperspheres with dynamic sizes. \nA so-called big prototype is characterized by the center of the hypersphere and the radius of the sphere. \nEmpirical results are conducted on few-shot named entity recognition (NER), few-shot relation extraction (RE) and few-shot image classifica- tion. \n ", "summary_of_the_review": "The paper lacks  thorough literature review. To decide whether this solution indeed works, the authors should fully discuss and empirically compare with existing works. Just comparing with prototypical network is far from enough. \n\n==== After rebuttal\n\nThanks for the new empirical results. I raise the score because the empirical part is now more completed. \nHowever, it is still unknown why using hyperspheres can obtain better performance than others (especially those using distribution)  consistently. More insights and theoretical analysis are still required.  \n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "AXdLi1Aj5Kt", "reviewer": "Reviewer_7SK7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strength:\n- evaluation on a variety of tasks from different domains\n- simple, intuitive, well-justified idea\n\n# Weakness:\n- slim contribution:  addition of one parameter to an existing network\n- only tested out on a single prototype network from 2017, which hasn't been SOTA for a long time. No comparisons to more recent prototype baselines [1,2] or approaches such as meta-learning [3], or self-supervision [4]. This lack of comparison makes the claims much less substantial. Big prototypes improve on vanilla prototypes, but do they make Euclidean prototypes competitive wrt modern methods?\n- While the writing is generally clear, there are many mistakes and at times clumsy formulations getting in the way of clarity and rigor\n- The way the prototypes are initialized (3.3) is not very clear to me at all. What is the benefit of this approach instead of simply computing the average center and radius? And if it is a contribution, why isn't it evaluated?\n\n\n# Suggestions:\nThe authors could turn the extreme simplicity of their approach into a strength if they adapted their method to a variety of prototype-based algorithms, including more recent and better-performing ones. Spheres exist in hyperbolic space, for example.\n\n# Details:\n\n**abstract:**\n- dense vectors = prototype?\n- the radius of the sphere is not constant if I understand it properly\n- what is an atactic manifold?\n- the authors are constantly the world \"metric\" to mean \"distance in a metric space.\" This makes for some confusing sentences\nintroduction:\n- what is the meaning of derivative in  \"derivative prototype-based methods\"?\n- how are Big prototypes easier to \"model\"?\n\n**Fig1:**\n- the fact that the embeddings change as well makes the figures hard to understand.\n- are the center and radius really learned separately and not end-to-end? This is not my understanding from the rest of the paper\nRelated work\n- type of NMS\n-  \"find the true location of the prototypes of the entire dataset\" I don't think there is such a thing as a true embedding for a prototype.\n\n**3 Methods**\n-  \"One big prototype is represented by two terms\" -> parameters\n- why limit yourself to the case in which the data is a vector of dimension L? it could be an image, for example\n- \"We now introduce big prototypes, which are a series of hyperspheres \" -> a set of hyperspheres \n- 3.3 is quite confusing to read. For example, z^j_n is never defined. \n\nEquation 5: \\tilde{r} is indexed by n, and the sum at the denominator is not indexed. It makes it look like the \\tilde{r} could be canceled out, when it is obviously not the case\n\n**Algo1:**\n\nInit: And why not just average over Dn?\n\n**Table1**\n P,R,F never defined. \"spaticularly\"\n\n[1] Gao et al.  Hybrid attention-based prototypical networks for noisy few-shot relation classification. AAAI, 2019.\n\n[2] Mettes et al. Hyperspherical prototype networks. NeurIPS 2019\n\n[3] Rusu et al.  Meta-learning with latent embedding optimization. ICLR 2019\n\n[4] Gidaris et al. Dynamic few-shot visual learning without forgetting CVPR 2018\n\n\n\n", "summary_of_the_paper": "The authors propose to augment an existing prototype-based few-shot learning approach by augmenting each prototype with a radius. They detail varied numerical experiments showing the advantage provided by this addition.", "summary_of_the_review": "A good idea with an interesting multi-domain evaluation. However, the lack of comparison with modern approaches and the use of a single backbone from 4 years ago makes the method not as convincing as it could be. Ultimately, this is not enough for ICLR.\n\nI strongly encourage the authors to adapt their simple idea to a wider range of methods (including hyperbolic prototypes) and add more baselines (and also significantly polish the writing and rigor). There is a potentially very good paper to come from this.\n\nPost discussion: the authors have addressed my concerns and have added many experiments to better substantiate their claim. In the end, a simple yet interesting idea.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9PQ-DqsNT_", "reviewer": "Reviewer_6kXL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The proposed big prototype model is a simple and easy to understand extension of the prototype model, which brings more adaptability to the latter. The simplicity of the model makes it easy to generalize to tasks that require a prototype model.\n- The experimental section has demonstrated the effectiveness of big prototypes in three tasks across NLP and CV, which are few-shot named recognition, few-shot relation extraction, and few-shot image classification. \n- The additional analysis of the change of hypersphere radius with sample density is helpful to understand the mechanism of big prototype model more intuitively.\n- The writing is mostly clear and easy to follow.\n\n\nWeaknesses：\n- In the paper, the metric of two vectors is calculated by a L2 norm distance function. However, in some tasks using prototype models, cosine similarity metric considered more appropriate. So can the big prototype model be effectively applied in such a scenario? Is there anything that needs to be adjusted?\n- The feature representation of all samples of a class often has a specific manifold distribution, not necessarily Gaussian distribution. What is the effect on the manifold distribution of feature representation before and after optimization with big prototype model? Could relevant visualization results or analysis be provided to show this impact?", "summary_of_the_paper": "In few-shot learning, prototypes have been widely used to represent classes and then classification can be performed by computing distances to prototype representations of each class. This paper proposes to use hyperspheres to model prototypes in the feature space, instead of vector points, to enhance the expressivity of class-level information. The proposed hypersphere model only needs two sets of parameters (the center and the radius of hyperspheres), so it does not bring additional burden to the optimization calculation of the objective function. Extensive experiments in both NLP and CV are conducted to evaluate the effectiveness of the proposed model. The results shows that the proposed model significantly outperforms the baseline, and it performs well in cross-domain few-shot relation extraction.", "summary_of_the_review": "As the author stated in Section 4, the experimental goal is not to lead in all the leaderboards, but to verify the effectiveness of the model only with baseline. Therefore, from this point of view, the proposed model is indeed better than the baseline model for comparison.\n\nFor me, the method proposed in the article is simple and has aroused my thinking. At the same time, it also brings some more worthy questions. I expect the author’s answer. At present, I think the proposed model is effective and enlightening. Therefore, I tend to suggest acceptance. Of course, I am also happy to see different opinions from other reviewers, and further adjust the score according to the feedback of the author.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3KUlUPQHQwr", "reviewer": "Reviewer_4NsC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1) The motivation of the paper is interesting. Using two sets of learnable parameters to represent the class prototype can reduce the influence of biased data.\n2)The metrics of the proposed big prototypes are easy to calculate. Both the radius and hyper-spheres center participate in the backward propagation, which leads to the adjustable centers and radius corresponding to different categories.\n3) Extensive experiments in both NLP and CV evaluate the effectiveness of the proposed method, especially under the setting of cross-domian FSL learning.\n\n\nWeakness:\n1) More analysis on the experimental details are recommended. I am interested in the feature distribution in the training process. It seems that only the points outside the hyperspheres affect the loss function. That means the proposed method can reduce the number of outliers of classes, and the learned features gather more closer in each class compared with the original prototypes method. It is better to provide some visualization of feature distribution.\n\n2) The applicability of the method is somewhat limited. The result of 1-shot task on miniImageNet is missing. In 1-shot learning, the proposed model cannot estimate the radius of the class by a single image, so the model sill suffers from the point-biased limitation?\n\n", "summary_of_the_paper": "The paper proposes to use areas to model prototypes in FSL. The prototypes, named as big prototypes, are represented by hyper-spheres with dynamic sizes. Rather than point-based prototypes, the new area-based prototypes  in the embedding space can represent the class-level information with more expressivity.", "summary_of_the_review": "The paper proposes \"big prototypes\" for few shot learning, which represents the prototypes by hyper-spheres with dynamic sizes. The new method is easy to implement and can reduce the influence of biased data. Extensive experiments in both NLP and CV demonstrate the effectiveness of the proposed method. More details or analysis are needed to improve the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thanks for your submission to ICLR.

This paper presents an extension to prototypical networks based on using hyperspheres to represent the prototypes.  Strong empirical results are presented using this approach.

Overall, this is a very borderline paper and could go either way.  The idea itself it simple, though the results seem to be fairly strong.  I read through the paper myself and tend to think that it could use a bit more work before it's ready.  Some of the issues raised by the reviewers---particularly with respect to experiments and literature review---are worth nailing down.  Further, I think that the method could be explored in a more principled/theoretical way.  For instance, when reading this idea, the first thing that pops into my mind is that representing the prototype with a hypersphere is very similar to representing a distribution (e.g., a Gaussian) using a mean and covariance (in this case, a spherical covariance).  Indeed, if you take the KL divergence between two spherical Gaussians, you get something very similar to the expression used in the paper.  This is all to say that there may be other more general directions to take this idea, or other interpretations of what is going on.

Please do keep in mind the comments of the reviewers when preparing a future version of the manuscript.
Decision: Reject