Few-Shot Learning with Representative Global Prototype
Authors: Yukun Liu
Daming Shi
Hexiu Lin
Venue: Submitted to ICLR 2023 (2023)

Abstract: Few-shot learning is often challenged by low generalization performance due to the assumption that the data distribution of novel classes and base classes is similar while the model is trained only on the base classes. To mitigate the above issues, we propose a few-shot learning with representative global prototype method. Specifically, to enhance the generalization to novel classes, we propose a method to jointly train the base classes and the novel classes, using selected representative and non-representative samples to optimize representative global prototypes, respectively. Additionally, a method that organically combines the sample of base classes conditional on semantic embedding to generate new samples of novel classes with the original data is proposed to enhance the data of novel classes. Results show that this training method improves the model's ability to describe novel classes, improving the classification performance for a few shots. Intensive experiments have been conducted on two popular benchmark datasets, and the experimental results show that this method significantly improves the classification ability of few-shot learning tasks and achieves state-of-the-art performance.

Under review as a conference paper at ICLR 2023
FEW-SHOT LEARNING GLOBAL PROTOTYPE
WITH
REPRESENTATIVE
Anonymous authors Paper under double-blind review
ABSTRACT
Few-shot learning is often challenged by low generalization performance due to the assumption that the data distribution of novel classes and base classes is sim- ilar while the model is trained only on the base classes. To mitigate the above is- sues, we propose a few-shot learning with representative global prototype method. Specifically, to enhance the generalization novel classes, we propose method to jointly train the base classes and the novel classes, using selected representa- tive and mon-representative samples optimize representative global prototypes, respectively. Additionally, a method that organically combines the sample of base classes conditional semantic embedding to generate new samples of novel classes with the original data is proposed to enhance the data of novel classes. Results show that this training method improves the model's ability to describe novel classes, improving the classification performance for few shots. Intensive experiments have been conducted on two popular benchmark datasets, and the ex- perimental results show that this method significantly improves the classification ability of few-shot learning tasks and achieves state-of-the-art performance.
INTRODUCTION
The advantages of deep learning depend big data, and machines can learn effectively under the drive of big data. However, an insufficient amount of data leads to problems, such as model over- fitting, which makes the model not well on the dataset outside the training data, showing weak generalization ability Kang et al. (201)). Thus, few-shot learning for small datasets has become key technology to solve this type of problem. Few-shot learning (FSL.) refers to the use of a small amount of labeled data per class training to make the model show higher performance, in line with the law of human learning.
Meta-learning is the basis of most of the existing FSL methods. In meta-learning, training is modeled after a test, i.e., both base and novel classes learn a task consisting K-shot, after which knowledge is transferred from the base classes in the form of appropriate initial conditions FFnn et al. (2017): Fei et al. (2020):
few samples from N-way
et al. (2020). embeddings Vinyals et al. (2016); Chopra al. (2005); Hadsell al. (006)) Sung et al. (2018) optimization strategies Ravi & Larochelle (017). There is, however, a fundamental limitation to all of these approaches: the models assumed that the distribution of base class data and novel class data was similar, they mostly learned using only base class data, which does not guarantee that they are generalizable to novel class data.
A few-shot learning method uses the transferability of knowledge to learn the ability to characterize samples from novel classes with only a few markers. while traditional machine learning and deep learning feed all classes of samples to the model at the beginning of training. Once the data distri- bution has low consistency, transferred knowledge not applicable to generalize the novel class of data. Thus, learning the global representation of all class data together can alleviate the problem of model overfitting on the base classes like machine learning and deep learning. The novel class data is learned at the beginning, and the metric novel class data and the global representation strengthen the discriminative nature of the novel class samples Wang et al. 2018 Li et al. (2019). At the same time, the sample synthesis strategy of Wang al. (2018) used to enhance the samples of the novel classes to alleviate the sample imbalance. However, since the global prototype is optimized directly using all samples, it is representative. Moreover, the sample generation method actually requires

---

Under review as a conference paper at ICLR 2023
Repreentatvve globel prosaype of hee chas
Represeniavve globel procoope of ornevel class
Represeninive somple
Vmmrepoeentatie hav aboe
Regreseniatve sumple of novel cltes Simoepocetativ xampl: aovd chos
Genorate simples of nexel cles Enbesdiig
Figure I: An illustration of representative global prototype. Combining representative samples from base class, semantic embeddings, and novel class samples allows us to generate new samples, which are then used to train representative global prototype.
that the base classes data and the novel classes data are similarly distributed, and do not break the fundamental restrictions Wang et al. (018)). Finally, the generation process is not directly involved in sample training and cannot optimize the quality of the generated samples.
FSL method solve the problem of poor generalization ability for few- shot data and sample imbalance to better describe the data of the novel class. The novel class and the selected representative and non-representative data from the base class are fed together into the network to jointly learn the representational ubility of the data, and thus it is called a representative global prototype. our view, the FSL model does not have to meet the barrier of similar distribution of the novel and base class data, and is better suited to identify novel classes when it is trained with novel class data.
Therefore, propose
However, trying to learn the global prototype of the base and novel class requires overcoming the problem data imbalance with sparse data novel class. overcome the strong correlation between the base and novel class data distribution, we propose a new sample generation strategy. Specifically, as Xu Le (002), more representative samples are selected to generate more repre- sentative features for the novel class using the CVAE[Sohn al. (2015) generative model conditional on semantic embedding. In the above operation, we obtain enhanced data strongly correlated with the novel class, with the base class as auxiliary information.
Our main contributions are summarized as follows: (1) We propose a novel jointly training strategy for few-shot learning via representative and non-representative samples to break the assumption of meta-learning. (2) We propose a sample synthesis method to enhance novel classes of data. (3) Experiments show that method exhibits state-of-the-art performance both minilmageNet and tieredImageNet datasets.
2
RELATED WORK
2. FEW-SHOT LEARNING
Few-shot learning comes into play when we only have very limited training samples. Most of the recent deep learning approaches rely on meta-learning or learning-to-learn Hoccreiter al. (2001) strategy, which improves the performance of the novel task by the provided data set and the meta knowledge extracted across tasks by meta-learnr.. Specifically, the meta-learner transfers knowledge learned from many base class tasks to help few-shot learning to complete training tasks novel classes.
Presently, representative FSL via meta-learning can be divided into three categories: fine-tune based, metric based, and optimization based.
(1) The methods of the first category aim to learn suitable initialization parameters for training novel class samples in order to train novel classifier faster and better/Finn et al. al. (2017): Rusu et al. (219) Nichol Schulman (2018) Fei et al. (2020): [Zhag et al. (2020). The models of the second category learning good metric. the core of which is to learn effective discriminative features through kernel function. Specifically, metric learning calculates the similarity between two samples by learning embedding module and metric module. The
2

---

Under review as a conference paper at ICLR 2023
samples of all classes are embedded into the vector space through the embedding module, and then a similarity score is given based on the metric module Vinyals et al. (2016); Chopra et al. (2005); Hadsell et al. (2006); Sung et al. (2018) Snell et al. (017). The methods of the third category adapt optimizers for meta-learning scenarios by replacing general optimization methods Munkhdalai & Yu 2 Ravi & Larochelle (0017) However, the above meta-learning based methods all limited by one barrier: the samples of the novel class do not appear in the training phase, so the model is
prone to overfitting on the base class.
2.2 GLOBAL REPRESENTATION LEARNING
A model overfitting problem can be alleviated by learning the global representation of all classes data together Li et al. (2019; ; Xi et al. (2022). It aims to learn a global representation for the classification of novel classes samples. Nevertheless, not all labeled samples are equally important, and non-representative samples can influence the accuracy of global representations. Model training can involve different sampled instances depending the sampling strategy. Most mainstream work focuses on samples that affect decision boundaries, which is more common in active learning, in which training samples affecting classification are selected using different uncertainty metric [Bappy et al. (217);; Li & Guo ) Zhao et al. (202I). In contrast, the representativeness of the data distribution of less concern when sampling. In FSL, an approach that uses only representative samples to fit global representations is proposed by Xu & Lc (022). but it ignores the overall distribution of the data. A representative global representation is therefore proposed. To optimize the representativeness of the global representation, two loss functions are generated using representative samples and non-representativ. samples.
2.3 CONDITIONAL VARIATIONAL AUTOENCODER
In the past, conditional VAE has been used to model feature distributions for variety of computer vision tasks, including image classification Kim et al. (2019): [Schöneeld et al. (2019); Xing et al.]
(0199; Zhang al. (2019), image generation Esser & Sutter (2018); Liu et al. (2017). image re- covery Du et al. (2020). or video processing Pan et al. (2019). Essentially. VAE models are trained learn the data distribution of the training set. As reported in Xi et al. (002 : Wang et al. (2018);
Li et al. (2019) that VAE can be used to generate data for novel classes based on training fixed pa- rameters from base classes. However, the problem that the parameters learned using only the base class data, which are suitable for the base class to generate better data, but not necessarily for the novel class data. Therefore, the method of using CVAE Sohn et al. (2015) was proposed to combine the features of the extracted base class data conditioned on semantic embedding generate features or images of the novel class data Xu & Le (2022). In this way, the novel class data can be generated more accurately by using common semantic embeddings the base class. However, these methods are essentially still under the assumption of meta-learning. Therefore, we propose new sample generation strategy that makes full use of the novel and base class data to synthesize new data. This does not depend on the parameters of the base class data training but also facilitates the classification of the novel class.
bridge between the novel class and
METHOD
The key idea of our model is to learn representative global prototype with the base and novel class jointly using the representative and non-representative samples. Furthermore, we employ a new sample generation strategy for novel classes to overcome the sample imbalance problem. In this section, we will first discuss these two key components before reviewing the entire process.
3.1 PROBLEM DEFINITION
In this section, we assign data. There are a total of N classes of samples, denoted as Call {cI, c2,-.. , ). The class set Cau consists of two disjoint sets: set of base classes Ciase and a set of novel classes Cnovel. Please note, the classes of training set Derain are from Call but not Chase- And test set Diest with samples from classes in Crovel- The sets of class Dirain and Dreat are disjoint, i.e. D Dtrain n Drest = In Derain. the samples from the base class have enough labels but are very limited from the novel class, with only n f (n & 5) labeled samples. A total of n,
3

---

Under review as a conference paper at ICLR 2023
Ropresaitative support
Sample generation
Represamacive global representation and classification
Quary Non-sepresentatiee semples
class
Average dass
Represernnive gloaal proveepe
New ruprosctttiive global prosttype
CVAF
Merrl
Merr
Metric
Generation loss
For class
Global loss
loss
Figure 2: Overview. First, we propose a sample generation method to generate new samples of novel classes in the representative support set. Second, the representative global prototype are updated according to the similarity score of features of each class and representative global prototype, and the new representative global prototype and ron-representative samples are then used to classify query images. The classification loss of query images, global loss, and generation loss are used to jointly optimize the global representations and the generation module.
representative base classes samples and original small amount of novel classes samples are sampled from Dreai to form a support set S {(x, y;), 1, ... n, N). and the non-representative samples are formed a query set Q = {xx, yk.) 1, 1.... ny N).
3.2 REPRESENTATIVE GLOBAL PROTOTYPE LEARNING
As result of the lack of generalization ability for models trained only on base classes, a joint train- ing strategy proposed learn global prototype. Furthermore, enhance its representativeness, only the representative support set is used to compute the global loss, while the classification loss is computed using the query set consisting of non-representative samples to compensate for the global prototype's incompleteness.
Specifically, we use representativeness as criterion for selecting samples. (x, p(x; [4, Σ,) > c is representative set of training samples of the base class selected by the method Xu & Le 2022), where p(-) is the probability density of Gaussian distribution Xj, μi is the average of all vectors in class i, the covariance matrix E; is the distribution of class i. A artificial threshold of e is used to filter out samples with low probability. Identifies the sample easier if the sample is close to the mean vector u4 which is its probability p(-) is large. For training the global loss, we selected the support set that is closest to the center vector.
A global loss is calculated by metricating the visual features of samples and all global representations of each class G = (g(i). C Cau), where g(i) is the global prototypical representation the class The initialization of g(i) is the mean of embedding features of all the selected representative samples in S. For each visual feature 2y, the similarity vector V, = [o].e,,. by a metric kernel function, where the i-th element is the similarity score of class in the embedding space:
u ... -u is obtained
vf = |x, g(i)ll2
In this paper, in order to bring the global prototype closer to representative sample, i.e.: larger posterior probability and more identifiable sample, we define a loss for class the representative support set (called global loss).
C. CE(U, V,)
(2)
where CF(()) is cross entropy loss.
4

---

Under review as a conference paper at ICLR 2023
Immediately afterward, utilize the representative global prototype a class representative and identify non-representative samples of the query set. Due to the large number of reoeesereeettttive samples and the fact that consider the most uncertain samples to be more beneficial for the classification task (Settee 2009). This reduces the training cost and speeds up the training process by filtering some samples. Inspired by the largest margin uncertainty| Jooshi et al. (2009), we measure the maximum and minimum probability values if a sample belongs to a certain class. Obviously, if the difference between the two is significantly larger, then the classifier has a higher degree of confidence that the sample belongs to the class. Alternatiely, if the difference is small, the sample in the tail of a Gaussian distribution and its class is harder identify. Therefore, we implement the criterion of largest margin uncertainty to choose the samples that are most difficult to identify class affiliation as our query set.
Q
{x, max (p(us x²)} min {p(ys x,)) < (8)
Once the query set data has been determined, we measure representative global prototypes and non-representative samples. order to make the representative global prototype differentiable in backward propagation, softmax is used as the normalization function. At this point, the similarity vector V, is normalized to probability distribution P, [p]. _p,""
The global prototypical representation of each class is then updated according to:
gnew (i) GP, where G is consisting of the representative global prototype before being updated.
(4)
The similarity score We = [w], w
1 N1 between the new representative global prototype and the non-representative sample x in the query set Q defined as:
|x
gnew (i)l22
(5)
A loss for the non-representative sample X in the query dataset Q (called classification loss) is defined
Leta CE WE)
(6)
The representative global prototype is jointly optimized by joint training of samples from the base class and the novel class by Lg and Cela, taking into account both the sample distribution and the classification performance. In contrast methods using only representative samples, our approach considers the completeness of the data distribution. With a simple sample selection strategy, we reduce the amount of training data comparison to an all-sample approach. The next step to optimize a representative global prototype by jointly generating loss.
3.3 FEATURE GENERATION WITH CVAE
We propose a sample generation strategy to generate samples of the novel class without the assump- tion of meta-learning to solve the sample imbalance problem caused by limited data (few shots). Two steps are involved in the generation of features this paper: (1) The features of the novel class generated from original samples of the base class and semantic attributes.
(2) All the features obtained in the first step and the original features of the novel class are used to synthesize new features.
Specifically, we combine the base class data and semantic embedding to generate new features by CVAE model. The selected representative samples are fed into the CVAE model to generate more representative features conditioned on the semantic embedding, the generate loss function is defined
Chase
LOVAE
KT(g(z) x.a') p(z a²))
(7)
1=1 X
2,)'))
5

---

Under review as a conference paper at ICLR 2023
where a' is the semantic embedding of class i. And the semantic embeddings are natural language descriptions of classes that contain attribute vectors of class information. The first term is the regu- larization term, which aligns the variation term q(z the Kullback-Leibee
a) to the prior distribution p(z a) through divergence. The second term is the reconstruction loss, which aims to make the features from the encoder generated by the decoder p(x a) approximate the original input features. And it is stipulated that N (0, I) is the prior distribution.
Regarding the method of generating features x of novel classes, we input the semantic embedding a' and latent variable z of the class in each novel class into the decoder to obtain new features of the corresponding class.
Using the method on the original base class samples, we will obtain a total k samples for every novel class. After that, we further synthesize new features from k, samples of each class. To enhance the distinguish ability of the generated features x the similarity between the features x² and the
global representation g(i) is computed, denoted as: U, = [u], 2 u²]²
" x g(t)
(8)
Please note that only the score with the highest similarity to the novel classes are selected here. To obtain the final enhanced feature X3, normalize the similarity score Uy using softmax to p/] and add it as the weight of generating features xx, with the original
P Proeel = [p). PJ. pj...... global prototype representation.
&, g(i) + xjPP novel, C Cnovel
By the above operation, we obtain the enhanced data strongly related to the novel classes with the base classes as auxiliary information. Finally, the &, and xj in the novel class and the base class are fed to the model optimize the representative global prototype.
3.4 MODEL REVIEW
We divide the whole model into two parts as Figure 2: representative global prototype learning and sample generation. The base class, the novel class and the semantic embedding utilized to jointly generate the data of the novel class. After that, the novel and base class data are jointly trained to optimize the global representation. As additional step, the representative samples are trained in order to optimize the representative global prototype. The loss of the entire model is as follows:
Lrotal = Ly + Leta + LeV
(10)
During the test phase, the same procedure used to predict labels for unlabeled data in Cnovel- To increase the number of samples in the novel class. sample generation strategy is used. Our next step is to compute the Euclidean distance between the feature vectors the test samples and the representative global prototype in the novel class in order conduct the nearest neighbor search.
EXPERIMENTS
this section, we evaluate and compare our method with state-of-the-art approaches on two datasets, i.c., minilmageNet and tieredImageNet. As first step, explain the detail settings. Then, we compare the proposed method with several state-of the-art few-shot approaches to prove how effective is. The experimental results analyzed at the end of this section.
4.1 DATASETS
minilmageNet Dataset Vinyals et al. (2016) lightweight subset of the large-scale dataset Ima- geNet. contains a total 60,000 color images with an image size of 84 x 84. The set these images is labeled with size of 100, there are 600 classes in each class. The data is divided into

---

Under review as a conference paper at ICLR 2023
Table 1: Few-shot classification accuracies (%) with 5-way minilmaeeNet and mageee model
backbone
minilmageNet
tigredlmaeNNet
1-shot
5-shot
I-shot
5-shot
Matching Ne Vinyals et al. (2016
ResNet- 12
65.6410.20
78.72:0.15
68.50100.2
80.60:1.71
MAMI Finn et al. 2011
ResNet-18
64.06:10.18
80.58
SimpleSho Wang et al. 2019
ResNet-18
62.85:10.20
80.02:10114
69.0911.22
84.58:10.16
CoN et al. 2019
ResNet-12
63.85:10.48
79.4411.34
69.89:0.51
84.23=10.37
Mangla et 2202
ResNer-12 64.06:t0.18
80.58+0.12
TADAM Kreshkin TADAMOreshiinn et dl. 2018
ResNer-12 58.50:0.30
7.370-10.00
62.13:10.31
81.92.10.30
AMiXinn et al. 2019
ResNet-12 65.30:10.9
08.10.1.36
66.22::0.75
82.79:10448
Variational FST 7Shngh al.
2019
ResNer-12 61.23++0.6
77.69+0.17
MetaOptNetLLee et T9
ResNer-12 62.64+0.61
78.63+0.46
65.99++0.22
01,56++0.5
Robust20-distil Dvornik
al 2019
ResNer-18 63.06:+0.61
80.63+0.42
65.43:10.21
0..44-10.2
MELR Fci ct al. 2020
ResNet-12 67.40=+0.43
83.40=0.28
72.14+0.51
87.01=0.35
RFS Tian cl al. 2020
ResNet-12 62.02-+0.63
79.64+0.44
69.74+0.72
84,41=0.55
Neg-Cosinel] Iaii et al. 20020a
ResNet-12 63.85+00881
81.57+0.56
IEPTiZhang ct al. 2020
ResNet-12 67.05:10.44
82.90-0.30
72.244:0.0
86.73-0.34
Cetu-BaselmmCCiie hen cl 2021
ResNet-12 63.17=00.3
79.26=0017
68.62+40.7
83.29=00188
ProtoNe ct al. 2020
ResNet-12 62.39
80.53
68.23
84.03
E3BML al. 2020b
ResNet-12
64.09-1..77
80.2910.25
71.3410.41
85.82-10.29
R-SVALÄU & 2022
ResNet-12
74.8410.23
83.28=0:40
76.98:10.55
85.77=0.50
Gi(OOus))
ResNet-12
75.92:+0.08
84.03+0.53
79.85
87.03++0.10
RGP w/o (Ours)
ResNet-12
76.15::0.27
84.28=0.35
80.01+0.36
87.09=0.42
RGP (Ours)
ResNet-12
76.41=20244
84.52+0.31
80.33+0.17
07.18=00336
three parts by labels. The training set contains 64 classes, the validation set contains 16 classes, and the remaining 20 classes used the test set.
mieredl Dataset Ren et al. (2018) is also a subset of ImageNet like minilmageNet. Ima- geNet categories are sampled and grouped according to hierarchical structure separate the data into broader classes, and the image size is also 84 x 84. A total of 608 classes are contained in tieredImageNet, which has 34 categories containing between 10 and 30 classes. Among these, 20 categories / 351 classes make up the training set, 6 categories / 97 classes make up the validation set, and the remaining categories/ 160 classes make up the test set.
4.2 DETAIL SETTINGS
A feature extractor based on ResNet-12 with feature representations output from the final layer was trained using data from the base class before global training was conducted. A feature extractor is then used to average the visual features of all class samples to initialize the representative global prototype. Adam acts as an optimizer to train the entire network at 10 The dimensions of the latent space and semantic vector are both set to 512. Our semantic embeddings word2vec Mikolov et al. (2013) trained using a skip-gram model on the Wikipedia corpus. We use the same feature extractor on images in the support and query sets. During our experiments, is set to 0.9 and B3 is set to 0.1.
extracted from
4.3 RESULTS
We compare with state-of-the-art few-shot learning methods, among them, our method most re- lated to Xu & Le 2022). To be fair, all methods use feature extractors with a similar number of layers, namely: ResNet- 12 and ResNet-18. In the remainder of the paper, we denote the model of training a global prototype using all data as GP and the training model with the selected representa- tive and non-representative data as RGP, and denote the model only use representative data but no non-representative data RGP w/o.
We show Table the classification results for the 5-way k-shot (k= I or 5) on the dataset. The results show that our method outperforms state-of-the-art methods, especially compared to R-SVAE which combines CLIP and utilizes representative samples. Furthermore, not only did the use of selected representative samples and non-representative partial samples not degrade the classification performance of the model, but the RGP showed a slight improvement in performance compared to the GP using full training. For example, the RGP outperformed the GP by 0.5% in 1-shot classi-
7

---

Under review as a conference paper at ICLR 2023
Insgeles
85
Accuraey aey. i5 (5-way)
75-
70-
0.
0.4
0.6
0.
Different Levels of Simi lar ity Score
Figure 3: 5-way performance with different levels of similarity score for base and novel classes on NieredlmageNet.
Support Features
Query Features
Generated Features by CVAE
Generated Features by ours
(8)
(b)
(4)
Figure 4: Feature Visualization. We visualize the features of the original and generated data. The transparent points are the generated features and the non non-transparent points the original features. (a): the support set features. (b): the query set features. (c): the generated features by CVAE, (d): the generated features by ours.
fication minilmageNet. To verify the effect of non-representative data, we compared RGP w/o and RGP without non-representative data. Experimental results show that RGP slightly outperforms RGP w/o. Another interesting finding is that 5-shot not as good as 1-shot for classification. RGP outperforms previous state-of-the-us methods by 1.18 ~ 3.34% for 1-shot and 0.2 ~ 1.14% for 5-shot.
Since our method trains all classes together, we don't 't mind whether the base class distribution differs from the novel class distribution. However, in order verify that RGP has better performance on tasks of novel classes that differ greatly from the base class, we re- re-partition the dataset. According to the similarity size (between 0-1) of the prototype the dataset into 9 levels. As shown in Figure 3, the performance of RGP is better when there are large differences between the base class and the novel class than R-SVAE Le (2022 jadd GCR Liet al. (2019). Moreover, the similarity of the curve increases, it becomes more stable, indicating that RGP is less affected by the distribution data.
base class and the novel class, we divided
4.4 VISUALIZATION
During training, Figure 4 illustrates the process of generating data with different methods using tieredImageNet datasets. Each the four images in Figure depicts the features three novel classes, which mapped to 2-d space using t-SNE Van der Maaten & Hinton (2008). Figure 4 (d) illustrates the rationality and effectiveness of our synthesis strategy since the features generated by method are closer the real features than CVAE alone shown in Figure 4 (c).
8

---

Under review as a conference paper at ICLR 2023
Representative
Non- Representative
Figure 5: Some examples for representative samples (left) and non-representative samples (right) from minilmageNet.
Table 2: 1-shot classification accuracies (%) with different generation methods on minilmageNet and tieredImageNet.
model
minilmageNet
tieredImageNer
VAE
73.62++0441
75.64=0.37
CVAE
75.18++0.64
7..3
RGP(NSS)
74.13::0.23
77.24-10.16
RGP
76.42:10.35
80.35:10.47
According different selection criteria, Figure illustrates representative and non-representative samples. The representative samples are closer to the center the distribution and therefore have higher probability and contain the main clean features that make them easier to identify. Samples selected at the tail of the distribution have lower probability of being assigned and may contain features relating other categories and therefore misclassified or hard to identify.
4.5 ABLATION STUDY
Our main experiments involve synthesizing samples in four parts. We compare synthesis strategies that use only one step separately to our approach the ablation experiments: VAE generation with Wang et al. (2018), CVAE generation with semantic embedding condition Xu & Le (2022) and no synthesis strategies method RGP(NSS).
As shown in Table 2, the 1-shot performance in different sample generation strategies minilma- geNet and mageed datasets. We can see that the VAE-based approach fixes the parameters obtained by training the base class, which is less effective than using CVAE that span different dis- tributions in terms of semantic embeddings. The RGP(NSS) only use the generate features after CVAE but without data of novel classes. Finally, the classification performance of both is inferior to our proposed approach which does not rely on the assumption of similar distributions.
CONCLUSION
new few-shot learning with representative global prototype framework is proposed alleviate the problem of poor generalization ability. In order to obtain a representative global prototype with strong generalization ability on the performance of the novel class, joint training strategy is pro- posed, which trains both the base class and the novel class on representative and non-representative data. Second, to alleviate the problem of sample imbalance in joint training, sample synthesis strategy that combines base classes and novel classes conditioned semantic embeddings is pro- posed, which not restricted to the strong assumption meta-learning. As a final step, three loss functions combined to train the entire model. Experimental results show that method out- performs existing few-shot methods. In the future, we will explore the impact of different sample selection methods model classification performance.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Jawadul H. Bappy. Sujoy Paul, Ertem Tuncel, and Amit K. Roy-Chowdhury. The impact of typical- ity for informative representative selection. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 771-780, 2017.
Yinbo Chen, Zhuang Liu, Huijuan Xu, Trevor Darrell, and Xiaolong Wang. Meta-baseline: Explor- ing simple meta-learning for few-shot learning. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 9062-9071, 2021.
S. Chopra, R. Hadsell, and Y. LeCun. Learning similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), pp. 539-546, 2005.
Yingjun Du, Jun Xu, Xiantong Zhen, Ming-Ming Cheng, and Ling Shao. Conditional variational image deraining. IEEE Transactions on Image Processing, pp. 6288- 0008-6631,220 2020.
Nikita Dvornik, Cordelia Schmid, and Julien Mairal. Diversity with cooperation: Ensemble methods for few-shot classification. Proceedings of the IEEE/CVF international conference on computer vision, pp. 3723- 10313, 2019.
Patrick Esser and Ekaterina Sutter. A variational u-net for conditional appearance and shape gen- eration. In 2018 IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 8857- 8866, 2018
Nanyi Fei, Zhiwu Lu, Tao Xiang, and Songfang Huang. Meir: Meta-learning via modeling episode- level relationships for few-shot learning. In International Conference on Learning Representa- tions, 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta- meta-learning for fast adaptation of deep networks. In International conference machine learning, pp. 1126-1135, 2017.
R. Hadsell, S. Chopra, and Y. LeCun. Dimensionality reduction by learning invariant map- ping. In 2006 IEFE Computer Society Conference on Computer Vision and Puttern Recognition (CVPR'06), (0P), pp. 1735-1742, 2006.
Sepp Hochreiter, A. Steven Younger, and Peter R. Conwell. Learning to learn using gradient descent. In Artificial Neural Networks - ICANN 2001, pp. 87-94, 2001.
Ruibing Hou, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Cross attention network for few-shot classification. Advances in Neural Information Processing Systems, 32, 2019.
Ajay J. Joshi, Fatih Porikli, and Nikolaos Papanikolopoulos. Multi-class active learning for image classification. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2372- 2379, 2009.
Bingyi Kang, Zhuang Liu, Xin Wang, Fisher Yu, Jiashi Feng, and Trevor Darrell. Few-shot object detection via feature reweighting. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8419-8428, 2019.
Junsik Kim, Tae- Hyun Oh, Scokju Lee, Fei Pan, and In So Kweon. Variational prototyping-encone -encodrr. One-shot learning with prototypical images. In 2019 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1454-9463 2019.
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In 2019 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pp. 10649-10657,, 2019.
Aoxue Li, Tiange Luo, Tao Xiang, Weiran Huang, and Liwei Wang. Few-shot learning with global class representations. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9714-9723, 2019.
Xin Li and Yuhong Guo. Adaptive active learning for image classification. In 2013 IEEE Conference Computer Vision and Pattern Recognition, pp. 859-866, 2013.
10

---

Under review as a conference paper at ICLR 2023
Bin Liu, Yue Cao, Yutong Lin, Qi Li, Zheng Zhang, Mingsheng Long, and Han Hu. Negative margin matters: Understanding margin in few-shot classification. In European conference on computer vision, pp. 438 455, 2020a.
Ming-Y Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. Advances in neural information processing systems, 2017.
Yaoyao Liu, Bernt Schiele, and Qianru Sun. An ensemble of epoch-wise empirical bayes for few- shot learning. In European Conference on Computer Vision, pp. 404 421,,211110 2020b.
Puneet Mangla, Nupur Kumari, Abhishek Sinha, Mayank Singh, Balaji Krishnamurthy, and Vi- neeth Balasubramanian. Charting the right manifold: Manifold mixup for few-shot learning. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2218- 2227, 2020.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representa- tions of words and phrases and their compositionality. Advances in neural information processing systems, 26, 2013.
Tsendsuren Munkhdalai and Hong Yu. Meta networks. In International Conference on Machine Learning, pp. 2554-2563, 2017.
Alex Nichol and John Schulman. Reptile: scalable metalearning algorithm. arXiv preprint 103,99,8,,,0299,, pp. 4, 2018.
Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. Advances in neural information processing systems, pp- 721-731, 2018.
Junting Pan, Chengyu Wang, Xu Jia, Jing Shao, Lu Sheng, Junjie Yan, and Xiaogang Wang. Video generation from single semantic label map. In 2019 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 3728-3737, 2019.
Sachin Ravi and Hugo Larochelle. Optimization as model for few-shot learning. In International Conference on Learning Representations, 2017.
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua Tenenbaum, Hugo Larochelle, and Richard S Zemel. Meta-learning for semi-supervised few-shot classifica- tion. International Conference on Learning Representations, 2018.
Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In International Conference Learning Representations, 2019.
Edgar Schönfeld, Sayna Ebrahimi, Samarth Sinha, Trevor Darrell, and Zeynep Akata. Generalized zero- and few-shot learning via aligned variational autoencoders. In 20/9 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8239-8247, 2019.
Burr Settles, Active learning literature survey. 2009.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Ad- vances in neural information processing systems, pp. 4077-4087, 2017.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances neural information processing systems, 2015.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip H.S. Torr, and Timothy M. Hospedales. Learning compare: Relation network for few-shot learning. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1199-1208, 2018
Yonglong Tian, Yue Wang. Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: good embedding is all you need? European Conference on Computer Vision, pp. 266,282, 2020.
11

---

Under review as a conference paper at ICLR 2023
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 2008.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, pp. 3630-3638, 2016.
Yan Wang, Wei-Lun Chao, Kilian Weinberger, and Laurens van der Maaten. Simpleshot: Re- visiting nearest-neighhor classification for few-shot learning. arXiv preprint arXiv:1911.04623, 2019.
Yu-Xiong Wang, Ross Girshick, Martial Hebert, and Bharath Hariharan. Low-shot learning from imaginary data. In 2018 IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 7278-7286, 2018.
Bobo Xi. Jiaojiao Li, Yunsong Li, Rui Song, Danfeng Hong, and Jocelyn Chanussot. Few-shot learning with class-covariance metric for hyperspectral image classification. IEEE Transactions on Image Processing, pp. 5079-5092, 2022.
Chen Xing, Negar Rostamzadeh, Boris Oreshkin, and Pedro O0 Pinheiro. Adaptive cross-modal few-shot learning. Advances Neural Information Processing Systems, 2019.
Jingyi Xu and Hieu Le. Generating representative samples for few-shot classification. Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9003-9013, 2022.
Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha. Few-shot learning via embedding adaptation with set-to-sed functions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8808-88 181,888777 2020.
Jian Zhang, Chenglong Zhao, Bingbing Ni, Minghao Xu, and Xiaokang Yang. Variational few-shot learning. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1685- 1694, 2019.
Manli Zhang, Jianhong Zhang, Zhiwu Lu, Tao Xiang, Mingyu Ding, and Songfang Huang. lept: Instance-level and episode-level pretext tasks for few-shot learning. In International Conference on Learning Representations, 2020.
Guang Zhao, Edward Dougherty, Byung-Jun Yoon, Francis Alexander, and Xiaoning Qian. Uncertainty active learning for optimal bayesian classifier. In International Conference on Learning Representations (ICLR 2021), 2021.
12


Reviews:
[{"review_id": "S6XOTgblp0", "reviewer": "Reviewer_5Wqr", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a few-shot learning with representative global prototype method. It uses selected representative and non-representative samples to optimize the representative global prototypes. It also utilizes semantic embedding to generate new samples of novel classes in order to alleviate the imbalance problem. The experimental results show that the proposed method can improve few-shot learning tasks and achieves state-of-the-art performance. ", "summary_of_the_review": "The paper considers an interesting problem in few-shot learning, but more detailed disscussions experiments should be added.", "strength_and_weaknesses": "Strength:\n\n1.\tThe paper considers an interesting problem in the few-shot learning task: the data distribution of novel classes and base classes can be different. The different distributions lead to the performance degradation of few-shot learning methods. Correspondingly, the paper proposes to generate more samples for novel classes, which is reasonable.\n\n2.\tThe paper is well-written and easy to follow.\n\n3.\tThe proposed method achieves the state-of-the-art performance on miniImageNet and tieredImageNet datasets.   \n\nWeakness:\n\n1.\tThe paper claims that the data distribution of novel classes and base classes could be different, resulting in performance degradation. It seems that the proposed method is more suitable for cross-domain few-shot classification. On miniImageNet and tieredImageNet datasets, the base classes and novel classes follow a similar distribution, which can not show the effectiveness of the proposed method. The experiments on cross-domain few-shot classification tasks should be discussed and added.\n\n2.\tMore detailed discussions and experiments should be added. The paper should consider dividing novel tasks into multiple different levels based on the similarity with based classes, and verify whether the proposed method is more effective on the tasks that are more different from based classes.\n\n3.\tIn the experimental setting, the proposed method uses CLIP to extract the semantic embedding for generating new novel features. CLIP contains external information. So it is unfair to compare the proposed method  with other methods. For a fair comparison, it is better to replace CLIP with another way. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2qwIZ9y3FQ", "reviewer": "Reviewer_HxLp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a joint training strategy for few-shot learning via representative and non-representative samples to enhance the generalization to novel classes. \nThe representative samples are used for the global prototype, and the non-representative samples are further used for classification loss. \nA sample generation method is also used to generate samples of novel classes. \nThe proposed method exhibits state-of-the-art performance on the miniImageNet and tieredImageNet datasets.  \n", "summary_of_the_review": "Although the proposed method is reasonable and achieves state-of-the-art performance, the proposed method only combines existing components, and this paper does not have novel aspects for few-shot learning. The ablation study is insufficient, and the writing needs improvements.  \n", "strength_and_weaknesses": "Strengths \n- The proposed approach, which uses based and novel classes to train and generate the samples for novel classes, seems reasonable. \n- The proposed method shows better performance than the state-of-the-art methods. \n\nWeaknesses \n- Components of the proposed method are not novel (see Novelty below). \n- The ablation study is insufficient (see Quality below )\n- The writing of this paper needs improvement. (see Clarity below)  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0Ia04PS8J1S", "reviewer": "Reviewer_pA9r", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This submission introduces a new few-shot learning with representative global prototype framework. For one thing, they propose a novel jointly training strategy for few-shot learning via representative and non-representative samples. For another thing, they propose a sample generation strategy to generate samples of the novel class to solve the sample imbalance problem caused by limited data (few shots). \n\n", "summary_of_the_review": "Somewhat novel and reasonable if the method is fair when comparing with its baselines.", "strength_and_weaknesses": "strength: This work proposes a few-shot learning with representative global prototype method.\n\nweakness: I have some questions, described below. As stated in 3.1, there are a total of N classes of samples and the support set also has N classes. To build the support set, the authors stated that they select a representative set of training samples of the ''base class''. Do they mean base classs and novel classses? Besides, the authors select the sample using p, mean vector of class i and covariance of class i. Are the mean vector and covariance of class i being fixed for using all samples within this class i? I am confused about \"which is its covariance Σi is small if Σi  is fixed for a given class i\". To generate samples for novel classes, the authors combine the base class data and semantic embedding ai. How can we obtain the semantic embedding, such as one-hot vector? Could the authors provide more explaination for  semantic embedding and how the samples for novel classes are generated? More importantly, how many N way K shot tasks are used to evaluate the performance? Here, N means subset of all novel classes rather than total classes. The proposed RGP is used to train a standard N way K shot or train their mentioned support set. I think the latter includes more noval task information than former. Therefore, it may be unfair to compare standard few-shot learning methods if the authors using all novel tasks. Besides, how about the computational cost at training and testing stage? Lastly, the authors mentioned their proposed RGP can break the barrier: the sample distribution of the novel class and the base class are similar. It would be better if the authors can provide some experimental results to support this argument.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject