Semantic-guided Diffusion Prototypical Network for Few-shot Classification

Authors: chuxin  zhang
jing  li
Venue: IEEE_org_ICIST_2024_Conference
Published: 2024-07-31 14:44:10

Abstract: Few-shot learning recognizes unlabeled samples from new classes using only a few of samples. Many recently proposed approaches have made progress based on meta-learning. However, current methods often overlook the category information within the query set and thus the obtained prototype is always unreliable. To tackle this issue, we design a Semantic-guided Diffusion Prototypical Network (SDPN) to generate representative prototypes for few-shot classification. Specifically, we leverage self-supervised learning to pre-train the feature extractor, thus obtaining accurately visual features. Furthermore, we introduce a semantic-guided diffusion process that aims to generate semantic features of the query set from random noise for a new task. Then, we introduce a visual-semantic fusion strategy that involves the alignment of semantic features with visual features to obtain representative prototypes that correspond to each image. We perform comprehensive experiments on miniImageNet and tieredImageNet datasets, and the results demonstrate that SDPN achieves enhancements in comparison to state-of-the-art methods.

Main paper: 
 i. introduction: 
 Deep learning techniques typically require substantial annotated data for training a model in order to achieve satisfactory performance for different computer vision tasks. Nevertheless, in practical applications, collecting such an extensive variety of labeled datasets is impractical and costly. Consequently, fewshot learning (FSL), which adapts to novel classes by training a model using a group of base classes with limited labeled data, has become an important research task. It is similar to the nature of human learning, where individuals can rapidly learn novel concepts based on limited samples [1] .Generally, FSL methods can be classified into three types: metric-based, meta-learning-based, and semantic-based methods. These methods all use the episode mechanism in the few-shot classification (FSC) task. The objective of FSC is to categorize unlabeled samples (the query set) using limited labeled samples (the support set). Metric-based methods [2] , [3] , [4] learn a task-agnostic embedding that can be used to calculate the distance between samples. Meta-learning methods [5] , [6] handle the FSL problem with a two-step This work was supported by National Natural Science Foundation of China under Grant 62373280.*Corresponding author learning process that involves meta-training and meta-testing. Specifically, the learning model will be trained with numerous independent supervised FSC tasks to learn how to adapt to novel tasks during the meta-training stage. Afterwards, the model is tested on a new unseen task during the meta-testing stage. Semantic-based methods [7] aim to extract meaningful features from the label information and combine visual and semantic features into a representative prototype with limited samples. To improve the accuracy of classification, some methods pre-train feature extractors on base classes to extract discriminative feature representations. Chen et al. [8] introduced an additional pre-training step to greatly enhance the accuracy of FSC tasks. This indicates that applying pretraining is critical for obtaining representative prototypes for FSC tasks. Substantial advancements have been achieved in FSC tasks by minimizing the cross-entropy loss of base class labels in the pre-training and meta-learning stages. However, the optimized model can only solve the FSC tasks of base class, but cannot generalize well to novel classes. Furthermore, the majority of current approaches rely on a single modality, which ignores the benefits of combining multiple modalities, such as visual data and semantic information, for enhancing the performance of FSC. Recently, some methods [9] , [10] have been proposed to generate more representative prototypes by combining the support set's visual and semantic features. Nevertheless, they did not consider the relationship between the semantic and visual modality of the query sample since its label is not available in the FSC tasks. In fact, the human perceptual system possesses a distinctive mechanism for visual perception, which can identify semantic information of new categories and promote the learning of new categories using prior knowledge. Therefore, it is necessary to design a new model that can effectively combine and utilize semantic and visual features of the query set to improve prototype representation.This paper introduces the Semantic-guided Diffusion Prototypical Network (SDPN) for few-shot classification. The SDPN employs a semantic-guided diffusion process to generate semantic features of the query set. Then, it combines these semantic features with visual features to create more accurate prototypes for few-shot classification. There are three stages that make up the proposed SDPN, which include pre-training, forward diffusion, and reverse diffusion. The feature extractor is trained on base classes via self-supervised learning during the pre-training stage, which can help extract more representative features. In the forward diffusion stage, for the support set, we utilize CLIP [11] to extract the features of class labels as semantic features, and obtain the images' visual features from the pre-trained feature extractor. We employ meta-learning on the support set to execute a generative process that smoothly transforms visual features into semantic features. In the reverse diffusion stage, we reconstruct the semantic features specific to the query set by iteratively denoising on a random noise. Finally, we fuse visual and semantic features of the support set and query set to generate accurate query prototypes and support prototypes based on the fusion strategy. We predict the label of the query sample by measuring the similarities between each query prototype and support prototype. The primary contributions of our paper are as follows:• We use self-supervised learning to train the feature extractor in the pre-training stage, with the goal of acquiring high-quality visual features of samples. • We propose a semantic-guided diffusion process that can generate semantic features of the query set by conditioning random noise on a limited sample of a given new task.• We design a fusion strategy that combines the semantic and visual features to obtain representative prototypes of samples.• We perform comprehensive experiments on two challenging benchmarks, miniImageNet [12] and tieredImageNet [13] , to verify the performance of our framework.Few-shot learning (FSL) aims to effectively adapt novel tasks using a few of labeled data. Representative FSL approaches include three types: metric-based, meta-learningbased and semantic-based methods. Specifically, Metric-based methods [2] , [4] , [12] learned a shared feature space, measuring the similarity between samples using a specific distance measure, such as Cosine similarity distance, Earth Mover distance and Euclidean distance. Snell et al. [2] introduced the prototypical network, a method that learns a metric space can be used to obtain a prototype of each class, computing the Euclidean distance between features of each category. The meta-learning-based methods [5] , [6] aim to learn an initial parameter of the model, in which the learner samples a series of training tasks from the base class for training and further adapted to novel FSC tasks. Finn et al. [5] designed a framework known as MAML, which learns an initialization parameter of the base-learner in order to enable quicker adaptation to novel tasks. Semantic-based methods [9] , [10] explore how to combine visual and semantic features for FSC tasks. They demonstrated that using semantic features, such as class names, can capture more accurate characteristics of the classes. Xing et al. [9] introduced a method that integrates visual and textual modalities by calculating an adjustable mixture coefficient.The Contrastive Language-Image Pre-training (CLIP) [11] aims to provide the models with better semantic understanding and generalization capabilities through contrastive learning between language and image modalities. It has achieved remarkable results in image classification, visual question answering and semantic segmentation. In FSC tasks, the CLIP extracts semantic features of the support set, which helps improve the performance of FSL models. Gao et al. [14] designed the CLIP-Adapter, which enhances the frozen CLIP model for downstream FSC tasks by incorporating a multilayer perceptron (MLP) and integrating language knowledge into the output. Zhang et al. [15] proposed Tip-Adapter, a nonparametric adaptive method of CLIP, which can extract knowledge from pre-trained CLIP with less data in FSC tasks. In our work, we utilize CLIP as the text encoder to extract semantic features from the samples without additional training parameters. Diffusion models [16] , [17] generate data by simulating a Gaussian diffusion process, which includes both the forward and reverse process. The overall diffusion process is illustrated in Fig. 1 . The forward process iteratively adds noise to the image x 0 ∼ q(x) through T times of accumulation to obtain x 0 , x 1 , . . . , x T . As the number of iterations approaches infinity, the generated sequence of samples tends towards a Gaussian distribution, and it can also be regarded as a Markov process:EQUATIONwhere {β t ∈ (0, 1)} T t=1 is a variance schedule. In this process, as t increases, x t becomes closer to the noise.By contrast, the reverse process is the denoising inference process of diffusion. We reconstruct the original image x 0 from the standard Gaussian distribution x T ∼ N (0, I). Since the reversed distribution q (x t-1 |x t ) is not computable, the diffusion model to predict such a reversed distribution, p θ (x t-1 |x t ) , as in (3) , where the parameter of the generation process is denoted as θ.EQUATIONwhereEQUATIONwhere α t = 1 -β t , and- α t = T i=1 α i ,N is a Gaussian distribution, µ is the mean and σ is the variance. The primary objective of the training process is to minimize the mean square error (MSE) between the predicted ε θ and the actual noise ε [18] , as shown in (6) : Then, we can sample from Gaussian noise x t and gradually denoise it to generate the image x 0 :EQUATIONEQUATIONChoi et al. [18] added conditions into the dependent variable of Denoising Diffusion Probabilistic Models (DDPM) [16] , resulting in the generation of high-quality and great-diversity images. Moreover, the diffusion model's sampling strategy has allowed it to be utilized in various downstream tasks, including translation from image to image and generation from text to image, achieving overwhelming performance. Therefore, inspired by using additional information to generate classspecific images, we propose a Semantic-guided Diffusion Prototypical Network (SDPN), which comprehensively analyzes image and label information in FSC tasks to generate classspecific prototypes.This section provides a brief overview of our proposed Semantic-guided Diffusion Prototypical Network, and then describe each stage in details.For few-shot classification, we typically partition the base dataset into two sets: the base classes, denoted as C b , and the novel classes, denoted as C n , where C b ∩ C n = ∅. For an Nway K-shot FSC task, there are two sets of samples used: i) the support-set, which has N classes and K samples per class, and ii) the query-set, which has Q unlabeled samples per class. The i-th N-way K-shot problem be denoted as T i = {S, Q}, which includes the support set S and the query set Q. Meanwhile, the objective of FSC is to accurately categorize the query set by utilizing limited samples. The proposed Semantic-guided Fig. 4 . The attention maps of CAN [4] , GLFA [19] and our proposed SDPN based on CAM [20] . We randomly select five images from the miniImageNet dataset [12] .Diffusion Prototypical Network (SDPN) includes three stages, which are pre-training, forward diffusion and reverse diffusion, as shown in Fig. 2 . Specifically, we train the feature extractor using a set of base classes to predict rotated angles of an image to improve feature representation during the pre-training stage. During the forward diffusion stage, the SDPN is trained with the support set, which generates diffused semantic features by using the ground-truth (GT) label. During the reverse diffusion stage, we add random noise into the reverse diffusion process to generate the query set's semantic features. Afterwards, we fuse the samples' semantic and visual features to generate more representative prototypes, enhancing the classification accuracy.We introduce an auxiliary rotation loss to train the features extractor f θ that encodes the visual information for FSC tasks. More precisely, we rotate the image using four angles, i.e., 0• , 90• , 180• and 270• , and f θ predicts the rotation angles of these rotated images via the auxiliary rotation loss, as shown in Fig. 3 . The auxiliary rotation loss [21] is defined as:EQUATIONwhere G represents the number of input samples and F = 4 denotes the four rotation angles. The x i, j indicates the i-th sample with the rotation angle j, and l (•) stands for the cross-entropy loss.Due to the particularity of the FSC task, we need to obtain the semantic features of the sample and improve the accuracy and reliability of the prototype by fusing visual and semantic features. However, the labels of the query set are not accessible, so we propose a semantic-guided diffusion process in forward diffusion and reverse diffusion stage. During forward diffusion, for the support set, we use the noise scheduler to add noise to semantic feature w S of to generate the noisy feature w * S , and compute the prototype v S via averaging the visual features. Additionally, we input the noisy feature w * S and the prototype v S into the semantic-guided diffusion module, which generates the diffused semantic vector w S . The goal is to generate the denoised semantic features of the support set by minimizing the simplified variational lower bound L D :EQUATION-EQUATIONEQUATIONHere, t θ (•, •) is achieved by the transformer model [24] , which is utilized to obtain the diffused semantic vector w S . In reverse diffusion, for a new FSC task, we input both v Q and random noise ε to the learned semantic-guided diffusion process, which generates the diffused semantic feature w T -1 Q of the query set. [12] AND TIEREDIMAGENET [13] Method Type Backbone MiniImageNet TiredImagenet After T iterations, we obtain the final semantic feature w Q = t θ (v Q , ε) of the query set, where ε is standard Gaussian variable given by ε ∼ N (0, I).First, we use pre-trained feature extractor to obtain visual features. Then, we compute visual prototypes v n by averaging the support set's visual features for each class.EQUATIONwhere S n represents a subset of S that includes samples from n different classes. Then, the semantic features w n are denoted as label embedding extracted from CLIP [11] . Finally, we construct the final prototypes by combining v n with w n based on the fusion strategy, which is denoted by:EQUATIONTo infer the labels of the query set, we measure the Euclidean distance between q i and p n .EQUATIONwhere q i denotes the i-th query prototype, v n represents visual prototype of the n-th support set and ŷi represents the predicted label of the i-th query sample.Accordingly, the overall loss function of the SDPN uses a weighted loss strategy:EQUATIONwhere Q represents the number of samples for each query set, β represents the hyperparameter of the feature extractor. We compared the SDPN to related state-of-the-art methods to evaluate its performance on two publicly available datasets, namely miniImagenet [12] and tieredImagenet [13] . Furthermore, the ablation studies are performed on miniImagenet [12] to assess the performance of each stage.The miniImagenet [12] is a part of the ImageNet dataset [25] . It includes 100 categories, and each category contains a total of 600 images. It was divided into 64, 16, 20 classes for training, validation and test.The tieredImagenet [13] is a part from the ImageNet dataset [25] , consisting of 608 categories, and each category contains a total of 1200 images. It was divided into 20, 6, 8 classes for training, validation and test.We utilize the ResNet12 as the feature extractor which is consistent with the majority of previous studies [3] , [4] , [6] . The features are obtained by taking an average of the outputs from the final residual block. Our semantic-guided diffusion network is based on the transformer [24] . For miniImageNet [12] , we perform training with 100 epochs, setting the batch size to 128. Moreover, the learning rate decays every 25 epochs. For tieredImageNet [13] , we used batch size of 128 and ran 100 epochs. Meanwhile, the learning rate decays at epoch 20 and 60. The studies were conducted using a PyTorch 3.9 framework on an NVIDIA GeForce RTX 3080 GPU. We perform numerous experiments on miniImageNet [12] and tieredImageNet [13] . Table I shows the results when comparing our proposed SDPN to the existing FSC approaches. Here, all methods employ the ResNet12 architecture as the backbone. Overall, the SDPN achieves competitive performance to existing methods. In the 5-way 1-shot and 5-way 5-shot settings, our SDPN surpasses GLFA [19] by 0.07% and 0.72% respectively on miniImageNet [12] . The SDPN performs better than the metric-based method SAPENet [23] by 4.12% and 0.85% respectively on tieredImageNet [13] . Furthermore, the SDPN performs better than the semantic-based method AM3-PNet [22] by 2.11% and 8.32% respectively on miniImageNet [12] . In contrast to AM3-PNet [22] , which only extracts word embeddings for category labels in the support set, our method utilizes the semantic information of the support set and query set. This enables a better understanding of the data and results in enhanced classification performance.To explore the performance of overall framework, we perform comprehensive experiments using the miniImageNet [12] dataset under the 5-way setting.a) The effects of different modules: We analyze the effects of various combinations of three modules. The results show that each of the three modules designed in SDPN are effective, as demonstrated in Table II .b) The effects of Semantic Embedding: In our experiments, we utilize CLIP [11] to extract semantic features. We train our model with Word2Vec [26] and CLIP as the Semantic Embedding, and the corresponding results are presented in Table III . Here, the CLIP model is trained using a dataset of 400 million pairs consisting of images and their corresponding text titles. In addition, Word2Vec is a model for efficient training of word vectors.c) The influence of the hyperparameters: Fig. 5 describes the results obtained by varying the hyperparameter λ, which represents the hyperparameter of the fusion strategy when combining the visual prototype and semantic prototype. The analysis indicates that λ = 0.4 can achieve best performance in the 1-shot and 5-shot settings for miniImageNet. Fig. 6 presents the results of various values on hyperparameter β, which represents the loss weight of the diffusion process. The analysis indicates that β = 0.45 can achieve best performance in the 1-shot and 5-shot settings for miniImageNet. The red line and blue line represent distinct settings of 5-shot and 1shot, respectively.To prove the performance of SDPN, we employ CAM [20] to generate attention maps for representative classes. By examining the attention maps generated by CAN [4] , GLFA [19] , and SDPN, as depicted in Fig. 4 . In particular, it indicates that SDPN can identify the regions that are more significant and relevant. For the example image from the malamute class, the SDPN can capture more meaningful information than both CAN [4] and GLFA [19] .In this work, we introduce a Semantic-guided Diffusion Prototypical Network for FSC tasks, which consists of three stages, i.e., pre-training, forward diffusion and reverse diffusion. The pre-training stage improves the reliability of the feature extractor via self-supervised learning by using auxiliary rotation loss. The forward diffusion stage learns to transform the visual features into semantic features of the support set. The reverse diffusion stage generates the query set's semantic features by conditioning random noise on a limited sample of a given new task, which generates representative support prototypes and query prototypes. Numerous experiments conducted on two benchmark datasets achieve competitive performance compared to related methods in FSC tasks.Although our method obtains satisfactory performance in few-shot learning, it faces a few limitations, such as the computational complexity of hyperparameter optimization. We need to optimize different hyperparameters for different datasets. Furthermore, noise sensitivity and higher inter-class similarity can limit performance in few-shot classification. In our future work, we will concentrate on executing adaptive hyperparameter optimization and improving the robustness to noise. In addition, we aim to improve our method to promote inter-class distinction and integrate prompt learning to enhance the effectiveness of our model. 
 
 Appendix:  



Reviews: review: title: title: Semantic-guided Diffusion Prototypical Network for Few-shot Classification
main_review: main_review: 1. The description of the proposed Semantic-guided Diffusion Prototypical Network is detailed and comprehensive. To further enhance accessibility, the authors could include additional figures and diagrams to visually illustrate the pre-training, forward diffusion, and reverse diffusion stages, as well as the prototype generation process. This would make the technical details and mechanisms behind SDPN more intuitive and easy to follow.

2.The introduction covers extensive background on few-shot learning. To enhance clarity, the authors could concisely summarize the key limitations of existing approaches and explicitly state how SDPN addresses these limitations, thereby highlighting its unique contribution and positioning.
title: title: Accept
main_review: main_review: 1.	It is suggested that the paper provide a more detailed comparison with existing methods that use semantic information, such as those leveraging CLIP or other multi-modal approaches.
2.	It would be useful to include more discussions on potential limitations or challenges in the proposed approach, such as computational complexity or scalability issues.
3.	The paper should include more comprehensive analyses, such as ablation studies to show the impact of each component (e.g. semantic-guided diffusion, visual-semantic fusion) on the overall performance.
title: title: Accept, some mistakes should be revised
main_review: main_review: 1. The overall structural description should be added to the Introduction.
2. The title of Section III is improper and should be modified.
3. Fig.4 and Fig.5 contain a lot of information with insufficient descriptions in the paper.