Feature hallucination in hypersphere space for few‐shot classification

IET Image ProcessingVolume 16, Issue 13 p. 3603-3616 ORIGINAL RESEARCHOpen Access Feature hallucination in hypersphere space for few-shot classification Sai Yang, Sai Yang orcid.org/0000-0002-3256-7306 School of Electrical Engineering, Nantong University, Nantong, Jiangsu, ChinaSearch for more papers by this authorFan Liu, Corresponding Author Fan Liu [email protected] orcid.org/0000-0001-8746-9845 College of Computer and Information, Hohai University, Nanjing, Jiangsu, China Correspondence Fan Liu, College of Computer and Information, Hohai University, Nanjing, Jiangsu 211100, China. Email: [email protected]Search for more papers by this authorZhiyu Chen, Zhiyu Chen College of Computer and Information, Hohai University, Nanjing, Jiangsu, ChinaSearch for more papers by this author Sai Yang, Sai Yang orcid.org/0000-0002-3256-7306 School of Electrical Engineering, Nantong University, Nantong, Jiangsu, ChinaSearch for more papers by this authorFan Liu, Corresponding Author Fan Liu [email protected] orcid.org/0000-0001-8746-9845 College of Computer and Information, Hohai University, Nanjing, Jiangsu, China Correspondence Fan Liu, College of Computer and Information, Hohai University, Nanjing, Jiangsu 211100, China. Email: [email protected]Search for more papers by this authorZhiyu Chen, Zhiyu Chen College of Computer and Information, Hohai University, Nanjing, Jiangsu, ChinaSearch for more papers by this author First published: 20 July 2022 https://doi.org/10.1049/ipr2.12579AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Few-shot classification (FSC) targeting at classifying unseen classes with few labelled samples is still a challenging task. Recent works show that transfer-learning based approaches are competitive with meta-learning ones, which usually pre-train a convolutional neural networks (CNN)-based network using cross-entropy (CE) loss and throw away the last layer to post-process the novel classes. Hereby, they still suffer the issue of getting a more transferable extractor and lacking enough labelled novel samples. Thus, the authors propose the algorithm of feature hallucination in hypersphere space (FHHS) for FSC. On the first stage, the authors pre-train a more transferable feature extractor using a hypersphere loss (HL), which supplies CE with supervised contrastive (SC) loss and self-supervised loss (SSL), in which SC can map the base and novel images onto the hypersphere space densely. On the second stage, the authors generate new samples for unseen classes using their novel algorithm of synthetic novel sampling with the base (SNSB), which linearly interpolate between each novel class prototype and its K nearest neighbour base class prototypes. Comprehensive experiments on multiple popular FSC demonstrate that HL loss can enhance the performance of backbone network and the authors' feature hallucination method is superior to the existing hallucination-based methods. 1 INTRODUCTION Image classification is a fundamental and important task in the field of computer vision, which plays a critical role in a range of advanced research fields, including object detection [1, 2], person re-identification [3], object tracking [4] etc. Accompanied by the rapid development of deep learning, this area has made remarkable breakthroughs in recent years. Deep neural network- based models have even reached human performance on benchmarks such as ImageNet [5]. However, such impressive progress depends critically on the large-scale manually annotated datasets for optimizing enormous representation parameters, which severely restricts the applicability of deep learning in many realistic scenarios. On the contrary, human beings are known to be able to recognize new objects just with very little supervision. Hence, this significant gap between human and deep learning has reawakened the research interest on few-shot classification (FSC) [6, 7]. Recently, many works [8-10] have shown that the classical classifier with a feature extractor which is pre-trained on all the meta-training datasets can achieve a very promising FSC performance. This kind of methods is referred to as transfer-based methods [11], which usually employ a ubiquitous cross-entropy (CE) loss to complete pretraining. However, CE loss only cares about the accuracy of the prediction probability of the correct label and ignores the differences of other incorrect labels, resulting in scattered features learned. To solve the above problem, we introduce supervised contrastive (SC) loss [12] into FSC. SC is built upon the popular unsupervised contrastive learning [13, 14] by accurately determining positive and negative samples through labels information. As it aims at pulling the samples of same class together and pushes apart the samples of different classes, it will mitigate the disadvantage of CE loss. Consider the example shown in Figure 1a, the samples belong to three different categories that fall apart with each other when CE loss is used for training. By comparison, as shown in Figure 1b, the samples belong to three different categories gather aground when SC loss is used for training. Otherwise, a recent study [8] revealed that the self-supervised task of rotation classification can further enhance the generalization performance of model. Therefore, we propose supplying the commonly used CE loss with SC learning and self-supervised learning. FIGURE 1Open in figure viewerPowerPoint Cross-entropy (CE) loss versus supervised contrastive (SC) loss. (a) When training with CE, the samples belong to three base categories of 'lion', 'dog', 'bowl' fall apart with each other. (b) When training with SC, the samples belong to three base categories of 'lion', 'dog', 'bowl' gather around Moreover, similar to unsupervised contrastive learning, SC also map the features onto the unit hypersphere, which leads to two properties of alignment and uniformity as investigated in [15]. As shown in Figure 2a, the former is capable of pulling similar samples closer and pulling apart diverse samples. As shown in Figure 2b, the latter induce the uniform distribution of samples on the unit hypersphere. Thus, samples of different classes uniformly distribute on the unit hypersphere with the samples of same class densely aggregating around a local region. Certainly, the novel class and the base class similar to it are also located together on a region of unit hypersphere. As shown in Figure 2c, there is an apparent positive linear correlation between the samples of same novel class and its similar base class. Over-sampling is a popular method to handle the long-tailed distributions problem, of which the famous SMOTE (synthetic minority oversampling technique) [16] and its extensions [17-19] usually linearly interpolate existing minority samples to generate new samples. Given above excellent properties of hypersphere space, it is valid to perform over-sampling to artificially synthesize data for novel class. FIGURE 2Open in figure viewerPowerPoint Illustration of the desirable properties of supervised contrastive loss. (a) The property of alignment makes that similar samples have similar features. (b) The property of uniformity makes those different kinds of samples uniformly distribute on the unit hypersphere. (c) The desirable property of supervised contrastive loss produces that samples of same novel class and its similar base class satisfy linear relationship To be concluded, we propose feature hallucination in hypersphere space (FHHS) to improve transfer-based methods in the FSC task. In specific, our method borrows SC learning to acquire more transferable backbone. In order to make the over-sampling strategy to deal with the under-represented novel class in the few-shot learning task, we propose a novel algorithm of synthetic novel sampling with the base (SNSB), which linearly interpolate between each novel class prototype and its K nearest-neighbour base class prototypes. The main contributions of this work are summarized as following: 1) We propose supplying the commonly used CE loss with SC and self-supervised loss (SSL) to obtain a hypersphere loss (HL) for pretraining, thereby acquiring a more transferable feature extractor for few-shot learning. 2) We propose a novel SNSB, which can effectively make the over-sampling strategies more suitable for generating data in few-shot learning. To the best of our knowledge, we are the first to introduce over-sampling strategies to realize the data augmentation for few-shot learning task. 2 RELATED WORK According to the training paradigm, current FSC approaches can be broadly grouped into meta-learning based and transfer-learning based. In addition, some methods following any above framework may introduce the effective data augmentation technique for settling FSC, which are referred to as hallucination-based methods. Thus, we subsequently discuss representative algorithms organized into three main categories: meta-learning-based, transfer-learning-based, and hallucination-based methods. Meta-learning-based methods Few-shot image classification is dominantly tackled by meta-learning, which can be divided into two groups: optimization-based methods and metric-based methods. The former methods target at learning an optimization process in a few steps to quickly adapt a new task. Classical methods including MAML [20] and Reptile [21] learn the optimal initialization parameters which enable the model to adapt new tasks within a few gradient steps. Meta-Learner LSTM [22] improve upon MAML by learning the update direction and learning rate. As this kind of methods suffers from the burden of computational and memory for higher-order derivatives, many works [23-26] have been proposed to settle this problem. For example, latent embedding optimization (LEO) [26] was proposed to perform optimization in a low-dimensional embedding space. Metric-based methods strive to learn a good feature embedding and metric for distinguishing query images from different categories. Most well-known models along this pipeline are Siamese networks [27], matching networks [28], prototypical networks [29], relational networks [30], which usually employ convolutional neural networks (CNNs) to represent images and some fixed metric or a learned metric to complete the classification. Recent works along this line are carried out from the following two respects. On the one hand, how to enhance the discriminative representations of feature extractor is a hot topic. Specifically, Wu et al. [31] and Wu et al. [32] respectively employed the deformable feature extractor and Capsule network to extract more efficient features to improve the CNN-based feature extractor. As attention mechanisms are effective on highlighting important local regions, several works introduced it into FSC to extract more discriminative features, which include cross attention network (CAN) [33], attentive weights generation via information maximization (AWGIM) [34], critic boosting attention network (CBANet) [35], category traversal module (CTM) [36]. Besides the above improvements on the feature embeddings, how to design a good similarity metric is also quite important for metric-based methods. Therefore, Nguyen et al. [37] proposed the squared root of the Euclidean distance and the Norm distance (SEN) to enhance the discriminative ability of Euclidean distance. Wertheimer et al. [38] proposed feature map reconstruction networks (FMPN) to reconstruct query features with respect to support features as the metric for non-parametric classifier. Moreover, some works like DN4 [39] and DeepEMD [40] obtained richer similarity by directly computing on local image descriptors. Transfer-learning-based methods Recently, some studies following transfer learning paradigm have achieved promising performance. The main idea is to pre-train a feature extractor on the whole base dataset, which is able to efficiently discriminate the novel classes with few examples. Early work in [41] pre-trained the backbone network with cosine classifier [42, 43] using a CE loss on base dataset, and then fine-tuned the classifier for novel class. As the pre-trained extractor is the key point for powerfully representing the features of novel images, several works concentrate on training a better backbone network. For example, Mangla et al. [8] applied the technique of self-supervised and Manifold Mixup during training backbone network to improve its generalization over novel task. Tian et al. [9] used sequential self-distillation to get a good embedding model. Also, Rizve et al. [44] improved the generalization of the feature extractor by designing a multi-task objective to simultaneously maintain equivariance and invariance. Liu et al. [45] learned more transferable representations by adopting the negative-margin Softmax loss to optimize the backbone network. Otherwise, as the base class-dependent classifier pre-trained on the first stage is always removed away when dealing with the novel class, it needs to leverage one of the traditional classifiers to complete the recognition on the second stage. Among them, the non-parametric approach of k-nearest-neighbours (KNN) is very popular due to its attractive simplicity and effectiveness. For instance, Wang et al. [10] found that the combination between the feature transformations on the feature of novel class and a nearest-neighbour classifier can obtain competitive performance. Wang et al [46] proposed calculating the similarity between the support samples, query samples, and base classes as the criterion to determine the neighbours for classification. Hallucination-based methods Hallucination-based methods generate more samples in the original image space or feature space to address the key issue of lacking enough training labelled data. One natural way to achieve this is the standard image augmentation techniques such as rotating, flipping, randomly cropping. However, this way cannot guarantee the diversity as the generated images are very similar to the original images. Thus, Chen et al. [47] synthesized new images by replacing some blocks of a support image by some parts of its similar unlabelled image. Zhang et al. [48] leveraged saliency detection to obtain foregrounds and backgrounds and mix them to hallucinate new images. Alternatively, several works unify meta-learning with a hallucinator added with noise [49] or a deformation sub-network [50] to model the image variance on base classes to act on the novel classes. Moreover, feature hallucination methods which perform data augmentation in the feature space seem more promising for modelling the variance. For instances, [51] and [52] respectively applied fully connected layers and Δ-encoders to model the intra-class variation of the base training classes and transfer it to the novel classes to generate additional samples in the same feature space. Aa generative adversarial networks (GANs) [53] provided an effective way of generating realistic samples, Gao et al. [54] introduced the powerful GAN to model the latent distribution of novel classes by preserving the covariance information. Li et al. [55] proposed adversarial feature hallucination networks (AFHN) to simultaneously ensure the generated features to be discriminable and diverse, which consists of a more stable Wasserstein GAN [56] and two novel regularizers. Additionally, explicitly establishing a probability distribution with few samples is another effective way to generate sufficient samples. For instance, Reed et al. [57] used PixelCNN combined with attention mechanism to perform few-shot density estimation and synthesis the samples in a sequential process. 3 OUR METHOD Overview of the framework Assume that we have a large labelled dataset D, which is divided into three subsets of training set D train $\mathop D\nolimits_{{\rm{train}}} $ , validation set D val $\mathop D\nolimits_{{\rm{val}}} $ , and test set D test $\mathop D\nolimits_{{\rm{test}}} $ . Following recent works [8-10], we do not employ an episodic training mechanism commonly used in meta-learning-based approaches. In this setting, the training set is also called the base dataset, which consists of Cb base classes with sufficient labelled samples in each class. Let the base dataset formulate as D base = D train = { S c } c = 1 c b $\mathop D\nolimits_{{\rm{base}}} = \mathop {\mathop D\nolimits_{{\rm{train}}} = \{ \mathop S\nolimits_c \} }\nolimits_{c = 1}^{\mathop c\nolimits_{\rm{b}} } $ , where S c = { ( x i , y i ) } i = 1 N b $\mathop S\nolimits_c = \mathop {\{ (\mathop x\nolimits_i ,\mathop y\nolimits_i )\} }\nolimits_{i = 1}^{\mathop N\nolimits_b } $ is the cth base class subset of images x i $\mathop x\nolimits_i $ and its corresponding labels y i $\mathop y\nolimits_i $ . Cb is the total number of base classes. Similar to the meta-learning-based approaches, D val $\mathop D\nolimits_{{\rm{val}}} $ is also used for hyperparameter optimization. During testing, it still needs to establish a series of N-way K-shot classification tasks to complete evaluation. In each task, the novel class dataset D novel $\mathop D\nolimits_{{\rm{novel}}} $ is composed of a support set D support = { S n } n = 1 N $\mathop D\nolimits_{{\rm{support}}} = \mathop {{\rm{\{ }}\mathop S\nolimits_n {\rm{\} }}}\nolimits_{n = 1}^N $ and a query set D query = { x q } q = 1 T × N $\mathop D\nolimits_{{\rm{query}}} = \mathop {\{ \mathop x\nolimits_q \} }\nolimits_{q = 1}^{T \times N} $ with unlabelled samples, where S n = { ( x s , y s ) } s = 1 K $\mathop {\mathop S\nolimits_n = \{ (\mathop x\nolimits_s ,\mathop y\nolimits_s )\} }\nolimits_{s = 1}^K $ is the nth novel class subset with K labelled samples in each class. The former randomly selects K samples from each of the N classes that are sampled from D test $\mathop D\nolimits_{{\rm{test}}} $ , while the latter contains the remainder T samples in these N classes. Under the pipeline of transfer-learning paradigm, the goal of few-shot learning is to learn a rich feature representation from D base $\mathop D\nolimits_{{\rm{base}}} $ , such that it can be used to classify the query samples into one of N unique novel classes based on K labelled support samples. We propose a method for FSC by hallucinating features in a hypersphere space for novel categories. The generated features are then used to train a few-shot classifier to predict the labels of query samples. The overview of our framework is illustrated in Figure 3. FIGURE 3Open in figure viewerPowerPoint Overview of the framework. Our method follows transfer learning paradigm, which consists of two stages. The first stage pretrains a good backbone network with HL, in which SC map the feature into the hypersphere space. The second stage hallucinate features with SNSB to address the issue of data scarcity. Note that the backbone network in HL shares the same structures and weights with the backbone network in SNSB. HL, hypersphere loss; SC, supervised contrastive; SNSB, synthetic novel sampling with the base. Hypersphere loss Given a batch of M images randomly sampled from D base $\mathop D\nolimits_{{\rm{base}}} $ , let any image x m $\mathop x\nolimits_m $ to be the anchor, which is sequentially through the data augmentation module and backbone network, resulting in 2M feature datapoints denoted as A ( m ) = { ( u m , y m ) } m = 1 2 M $\mathop {A(m) = \{ (\mathop u\nolimits_m ,\mathop y\nolimits_m )\} }\nolimits_{m = 1}^{2M} $ where u m = E α ( x m ) $\mathop {\mathop {\mathop u\nolimits_m = E}\nolimits_\alpha (x}\nolimits_m )$ , and A ( m ) $A(m)$ is a set including all the positive samples and negative samples related to the mth anchor. According to the label information in A ( m ) $A(m)$ , we get the positive samples set includes the original samples and augmented ones have the same label with the anchor. Let it be denoted as P ( m ) $P(m)$ and | P ( m ) | $| {P(m)} |$ is its cardinality. The remainders in A ( m ) $A(m)$ constitute of the negative samples set. Let it be denoted as N ( m ) $N(m)$ . Thus, the SC loss is formulated as L S C = ∑ m ∈ A ( m ) − log { 1 p ( m ) ∑ p ∈ p ( m ) exp ( τ ( S m p ) ) ∑ p ∈ p ( m ) exp ( τ ( S m p ) ) ) + ∑ a ∈ N ( m ) exp ( τ S m p ) } \begin{eqnarray} \mathop L\nolimits_{SC} &=& \sum_{m \in A(m)} - \log \Bigg\{ \frac{1}{{\left| {p(m)} \right|}}\sum_{p \in p(m)}\nonumber\\ && {\frac{{\exp (\tau (\mathop S\nolimits_{mp} ))}}{{\sum_{p \in p(m)} {\exp (\tau (\mathop S\nolimits_{mp} )))} + \sum_{a \in N(m)} {\exp (\tau \mathop S\nolimits_{mp} )} }} \Bigg\} } \end{eqnarray} (1)where τ ∈ i + $\tau \in \mathop {\rm{i}}\nolimits^ + $ is a scalar temperature parameter, u p $\mathop u\nolimits_p $ is the pth sample from P ( m ) $P(m)$ , u a $\mathop u\nolimits_a $ is the ath sample from N ( m ) $N(m)$ . s ( · , · ) $s( \cdot , \cdot )$ denotes the cosine similarity function between two normalized vectors, which is expressed as S m p = u m T u p u m T u p S m a = u m T u a u m T u a \begin{equation} \def\eqcellsep{&}\begin{array}{l} \mathop S\nolimits_{mp} = \dfrac{{\mathop {\mathop u\nolimits_m^T u}\nolimits_p }}{{\left\| {\mathop u\nolimits_m^T } \right\|{\kern 1pt} \;\left\| {\mathop u\nolimits_p } \right\|}}\\ \mathop S\nolimits_{ma} = \dfrac{{\mathop {\mathop u\nolimits_m^T u}\nolimits_a }}{{\left\| {\mathop u\nolimits_m^T } \right\|{\kern 1pt} \;\left\| {\mathop u\nolimits_a } \right\|}} \end{array} \end{equation} (2) A recent study has shown that the task of rotation classification can further enhance the performance of backbone network. In this setting, the image x i $\mathop x\nolimits_i $ is rotated by four angles, which are selected as {0°,90°,180°,270°}. In this setting, the backbone network is added with a four-way classifier C r ( g ) $\mathop C\nolimits_{\rm{r}} (g)$ and a base classifier C b ( g ) $\mathop C\nolimits_{\rm{b}} (g)$ . The former is described by matrix Wr = { w r } r = 1 4 $\mathop {\{ \mathop w\nolimits_r \} }\nolimits_{r = 1}^4 $ , the latter is described by matrix Wb = { w i } i = 1 c b $\mathop {\{ \mathop w\nolimits_i \} }\nolimits_{i = 1}^{\mathop c\nolimits_{\rm{b}} } $ . Now the d-dimensional feature of image x ̂ i $\mathop {\hat x}\nolimits_i $ from the backbone network is inputted into classifier C r ( g ) $\mathop C\nolimits_r (g)$ to implement the rotation prediction, the classification score can be formulated as S i r = C r ( z i ) = z i T W r = z i T [ w r 1 , w r 2 , w r 3 , w r 4 ] \begin{equation}\mathop {\mathop S\nolimits_i^r = C}\nolimits_{\rm{r}} (\mathop z\nolimits_i ) = \mathop z\nolimits_i^T \mathop W\nolimits_r = \mathop z\nolimits_i^T [\mathop w\nolimits_{r1} ,\mathop w\nolimits_{r2,} \mathop w\nolimits_{r3,} \mathop w\nolimits_{r4} ]\end{equation} (3) The SoftMax function is then applied over S i r $\mathop S\nolimits_i^r $ to get the prediction probability of the rth class as follows: p i r = e S i r ∑ j = 1 4 e S i r = e z i T w r ∑ r = 1 4 e z i T w r \begin{equation}\mathop p\nolimits_{ir} = \frac{{\mathop e\nolimits^{\mathop S\nolimits_i^r } }}{{\sum_{j = 1}^4 {\mathop e\nolimits^{\mathop S\nolimits_i^r } } }} = \frac{{\mathop e\nolimits^{\mathop {\mathop z\nolimits_i^T w}\nolimits_r } }}{{\sum_{r = 1}^4 {\mathop e\nolimits^{\mathop {\mathop z\nolimits_i^T w}\nolimits_r } } }}\end{equation} (4) Then SSL for rotation prediction can be formulated as L SSL = − ∑ i = 1 M ∑ r = 1 4 y i r log p i r = − ∑ i = 1 M log p i r = − ∑ i = 1 M log e z i T w r ∑ r = 1 4 e z j T w r \begin{equation} \mathop L\nolimits_{{\rm{SSL}}} = - \sum_{i = 1}^M {\sum_{r = 1}^4 {\mathop y\nolimits_{ir} } } \log \mathop p\nolimits_{ir} = - \sum_{i = 1}^M {\log } \mathop p\nolimits_{ir} = - \sum_{i = 1}^M {\log } \frac{{\mathop e\nolimits^{\mathop {\mathop z\nolimits_i^T w}\nolimits_r } }}{{\sum_{r = 1}^4 {\mathop e\nolimits^{\mathop {\mathop z\nolimits_j^T w}\nolimits_r } } }}\end{equation} (5) The features are also inputted into the base classifier to compute the base classification scores of S i $\mathop S\nolimits_i $ as S i = C w ( z i ) = z i T W = z i T [ w 1 , w 2 , … w c b ] = [ z i T w 1 , z i T w 2 , … z i T w c b ] \begin{equation}\mathop {\mathop S\nolimits_i = C}\nolimits_w (\mathop z\nolimits_i ) = \mathop z\nolimits_i^T W = \mathop z\nolimits_i^T [\mathop w\nolimits_1 ,\mathop w\nolimits_{2, \ldots } \mathop w\nolimits_{\mathop c\nolimits_{\rm{b}} } ] = [\mathop {\mathop z\nolimits_i^T w}\nolimits_1 ,\mathop z\nolimits_i^T \mathop w\nolimits_{2, \ldots } \mathop {\mathop z\nolimits_i^T w}\nolimits_{\mathop c\nolimits_{\rm{b}} } ]\end{equation} (6) The SoftMax function is then applied over S i $\mathop S\nolimits_i $ to get the prediction probability of the jth class as follows: p i j = e S i j ∑ j = 1 C b e S i j = e z i T w j ∑ j = 1 C b e z i T w j \begin{equation}\mathop p\nolimits_{ij} = \frac{{\mathop e\nolimits^{\mathop S\nolimits_{ij} } }}{{\sum_{j = 1}^{\mathop C\nolimits_b } {\mathop e\nolimits^{\mathop S\nolimits_{ij} } } }} = \frac{{\mathop e\nolimits^{\mathop {\mathop z\nolimits_i^T w}\nolimits_j } }}{{\sum_{j = 1}^{\mathop C\nolimits_b } {\mathop e\nolimits^{\mathop {\mathop z\nolimits_i^T w}\nolimits_j } } }}\end{equation} (7) Then the CE classification loss for the base classification can be formulated as L CE = − ∑ i = 1 M ∑ j = 1 C b y i j log p i j = − ∑ i = 1 M log p i j = − ∑ i = 1 M log e z i T w j ∑ j = 1 C b e z j T w j \begin{equation}\mathop L\nolimits_{{\rm{CE}}} = - \sum_{i = 1}^M {\sum_{j = 1}^{\mathop C\nolimits_{\rm{b}} } {\mathop y\nolimits_{ij} } } \log \mathop p\nolimits_{ij} = - \sum_{i = 1}^M {\log } \mathop p\nolimits_{ij} = - \sum_{i = 1}^M {\log } \frac{{\mathop e\nolimits^{\mathop {\mathop z\nolimits_i^T w}\nolimits_j } }}{{\sum_{j = 1}^{\mathop C\nolimits_{\rm{b}} } {\mathop e\nolimits^{\mathop {\mathop z\nolimits_j^T w}\nolimits_j } } }}\end{equation} (8) The SC loss can map the features onto the unit hypersphere, which will bring in some awesome properties. Thus, we will supply it to the CE to establish our HL, which is formulated as follows: L = α 1 L CE + α 2 L SC + α 3 L SSL \begin{equation}L = \mathop \alpha \nolimits_1 \mathop L\nolimits_{{\rm{CE}}} + \mathop \alpha \nolimits_2 \mathop L\nolimits_{{\rm{SC}}} + \mathop \alpha \nolimits_3 \mathop L\nolimits_{{\rm{SSL}}} \end{equation} (9) Synthetic novel sampling with the base Having trained backbone network with HL, we now discuss how to conduct FSC on the novel dataset D novel $\mathop D\nolimits_{{\rm{novel}}} $ , which faces the problem of lacking limited support samples and imbalance with D base $\mathop D\nolimits_{{\rm{base}}} $ .Therefore, we introduce the over-sampling strategy to simultaneously solve the above problems, which handles the class imbalance problem by synthetically generating new minority class samples. One of the dominant methods is SMOTE, which linearly interpolate between each sample and its K nearest neighbours within minority class. Numerous extensions such as Borderline, ADASYN have been developed to locate the region where synthetic oversampling should be performed. However, they still rely entirely on the samples and its neighbours inside the minority class to achieve over-sampling. This cannot be directly applied to the FSL task, because that the novel class is strongly underrepresented. Especially for the one-shot learning, it cannot find any neighbours inside the support set at all. Therefore, we propose a novel algorithm of SNSB, which leverage prototype of the base class similar to the novel class as the neighbours to realize over-sampling. On the second stage of evaluation, each N-way K-shot classification task consists of a support set D support = { S n } n = 1 N $\mathop D\nolimits_{{\rm{support}}} = \mathop {{\rm{\{ }}\mathop S\nolimits_n {\rm{\} }}}\nolimits_{n = 1}^N $ and a query set D query = { x q } q = 1 T × N $\mathop D\nolimits_{{\rm{query}}} = \mathop {\{ \mathop x\nolimits_q \} }\nolimits_{q = 1}^{T \times N} $ . Given an image x s $\mathop x\nolimits_s $ from the nth novel class subset S n $\mathop S\nolimits_n $ , the backbone network will output a d-dimensional feature u s = E α ( x s ) $\mathop {\mathop {\mathop u\nolimits_s = E}\nolimits_\alpha (x}\nolimits_s )$ . The prototype of each novel class is calculated as the mean of the support samples P n = 1 K ∑ s ∈ S n u s \begin{equation}\mathop P\nolimits_n = \frac{1}{K}\sum\nolimits_{s \in \mathop S\nolimits_n } {