Invariance-Guided Feature Evolution  for  Few-Shot Learning

1 INTRODUCTION . Few-shot image classification aims to learn to predict the labels of query images from very small sets of support samples ( Finn et al. , 2017 ; Yu et al. , 2020 ) . More importantly , this learned representation needs to generalize well onto unseen classes during the test stage . Since the number of support images per class , also called shots , is very small , often in the range of 1 to 5 , the characteristics of support samples will vary significantly . Therefore , a successful few-shot learning ( FSL ) should be able to survive this large variation of image-specific characteristics in the support set ( Yu et al. , 2020 ; Bronskill et al. , 2020 ) . This requires that the FSL be able to capture the underlying or inherent visual relationship between the query and support images , instead of being distracted or over-fitted by the image- specific characteristics , especially those non-salient features from the background . In other words , this inherent visual relationship , once successfully learned , should remain relatively invariant across different sample configurations of the support sets . Otherwise , its classification performance with very few support samples and generalization capability onto novel classes will degrade significantly . Based on this observation , we put forward the following observation : the feature representation generated by the FSL , if successfully learned , should remain relatively invariant to sample configurations of the support sets , for example , the number of shots ( i.e , the number of samples per class ) and the selection of support samples for training . Driven by this observation , we propose to construct a feature evolution network with an ensemble of few-shot learners which evolve the feature embedding along dimensions of the support set configuration , as illustrated in Figure 1 . In this work , we choose to study two major parameters that control the support set configuration : the number of labeled samples per class ( called shots ) and the percentage of training samples ( called partition ) in the support set . This network aims to characterize and track the evolution and variation of learned features of the query samples across different shot-partition configuration of the support samples . We then introduce an invariance metric to measure the variation of the learned query features . This invariance metric serves as a loss function to be optimized during the FSL process . Our extensive experimental results demonstrate that the proposed IGFE method significantly improves the performance of FSL and outperforms existing state-of-the-art methods by large margins on various benchmark datasets . 2 RELATED WORK . There are three major categories of methods that have been developed for few-shot learning : meta learning , transfer learning , and metric learning . ( 1 ) In meta-learning based methods , episode training , which mimics the test tasks during the training stage , is explored to improve the knowledge transfer and generalization capability from metatraining to meta-testing ( Finn et al. , 2017 ; Rusu et al. , 2018 ) . In Zhang et al . ( 2020a ) ; Ye et al . ( 2020 ) ; Chen et al . ( 2020b ) , meta-learning is combined with pre-training to further improve the performance . Tian et al . ( 2020 ) ; Rizve et al . ( 2021 ) decoupled the learning procedure into the baseclass embedding pre-training and novel-class classifier learning and develop a multivariate logistic regression and knowledge distillation method to improve the meta-learning performance . ( 2 ) Transfer-learning methods ( Qiao et al. , 2018 ; Gidaris & Komodakis , 2018 ; Chen et al. , 2019 ) pre-train the model with a large set of base classes and then adapt the model to unseen classes . Transmatch ( Yu et al. , 2020 ) pre-trains a feature extractor on the base-class data , uses it as the feature extractor to initialize the classifier weights for unseen classes , and updates the model using semisupervised learning methods . Gidaris & Komodakis ( 2018 ) propose an attention module to predict the classifier weights for novel classes dynamically . Instead of separating the base classes into a set of few-shot tasks , methods in Qiao et al . ( 2018 ) ; Chen et al . ( 2019 ) use all base classes to pre-train the few-shot model , which is then adapted to novel-class recognition . ( 3 ) Metric-based methods ( Sung et al. , 2018 ; Zhang et al. , 2020b ) aim to learn an effective measurement space and identify query samples based on the feature distance between samples . MatchingNet ( Vinyals et al. , 2016 ) shows that neural networks can be successfully trained to measure the similarity between query and support samples . The prototype network method ( Snell et al. , 2017 ) introduces a prototype , which represents a class . The distance between prototypes and a query vector is used for the classification . Recent methods utilize task-specific prototypes . For example , FEAT ( Ye et al. , 2020 ) applies self-attention-based transformation on the prototypes . BSNet ( Li et al. , 2021 ) uses a dual-metric distance and achieves better performance than a single-metric distance . Recently , graph-based methods have been explored to model and predict the correlation between samples , aiming to propagate meta-knowledge and the labels from base classes to novel classes ( Rodrı́guez et al. , 2020 ; Satorras & Estrach , 2018 ) . Chen et al . ( 2020a ) develop a graph neural network-based few-shot learning method which iteratively aggregates features from neighbors and learn the correlation between samples . DPGN ( Yang et al. , 2020 ) has proposed a two-fold complete graph network to model instance-level and distribution-level relations with label propagation and transduction . The MCGN method ( Tang et al. , 2021 ) combines GNN and conditional random field ( CRF ) to achieve better affinity prediction . Knowledge distillation . This work is related to knowledge distillation , which learns a small student network using the supervision signals from both ground truth labels and a larger teacher network ( Gou et al. , 2021 ) . Three different types of knowledge or supervision signals have been explored in the literature . ( 1 ) Response-based knowledge distillation uses the neural response of the last output layer of the teacher network as the supervision information ( Hinton et al. , 2015 ; Zhang et al. , 2019 ; Meng et al. , 2019 ) . ( 2 ) Feature-based knowledge distillation uses the feature map of the middle layer as the supervision signal to train the student network ( Romero et al. , 2014 ; Wang et al. , 2020 ; Chen et al. , 2021 ) . ( 3 ) Relationship-based knowledge distillation further explores the relationship between different layers or data samples ( Yim et al. , 2017 ; Passalis et al. , 2020 ; Dong et al. , 2021 ) . For example , the second order statistics between different network layers are used in Yim et al . ( 2017 ) as the supervision signals . 3 METHOD . In this section , we present our proposed method of invariance-guided feature evolution ( IGFE ) for few-shot learning . 3.1 CLASS FEATURE VARIATIONS ACROSS DIFFERENT SUPPORT SET CONFIGURATIONS . In this section , we experimentally study the variation of query set features across different support set configurations , which motivates us to make the invariance observation discussed in Section 1 . In a typical setting , a K-way N -shot FSL has K classes with N support samples per class in its training batch . The training samples from one class will be divided into M batches . In total , we have K ×M samples per class . The performance of the FSL directly depends on the configuration of the support set . In this work , we study two important parameters that plays important roles in FSL : ( 1 ) N , the number of shots or the number of support samples per class , and ( 2 ) M , the number of sample batches or the selected partition of the support set used for training . For simplicity , we refer to these types of configuration as shot configuration and partition configuration . In the following experiments , we perform fewshot learning on the mini-ImageNet dataset . Figure 2 ( a ) shows the t-SNE plots for query sample features of 5 classes for different shots N . In each shot configuration , we choose different subset partitions of the support samples for training . We can see that , for FSL with larger N , the variation of query class features is becoming smaller . This implies that , in a better learned FSL model , the feature variation across different support set partition is becoming smaller . Figure 2 ( b ) shows the experimental results for different numbers of batches or subset partitions of the support set used for training . For each batch or support set partition , we perform the FSL with different N , and compute the average variation of query class features . We can see that , as more batches of support samples are used , the FSL is better learned , the query class feature variation across different shots is becoming smaller . These experimental results motivate us to make the following observation : in a better learned FSL model , the variation of query sample feature across different shot and partition configurations of the support will be smaller . Driven by this observation , we develop the invariance-guided feature evolution ( IGFE ) method for few-shot learning , which will be explained int the following section . 3.2 INVARIANCE-GUIDED FEATURE EVOLUTION . Figure 3 ( a ) provides an overview of the proposed invariance-guided feature evolution ( IGFE ) method . we use the one-shot learning as an example to explain our proposed IGFE method . For the partition variation , we use three different subsets of the support samples for training : ( 1 ) SW which is the whole support set ; ( 2 ) SA and SB which are two randomly chosen subsets of the support set . For example , in our experiments , each subset contains 60 % of the whole support samples . For the shot variation , besides the baseline one-shot learning module , we introduce two additional FSL modules with N = 2 ( two-shot ) and N = 3 ( three-shot ) . In this way , we have the following four FSL modules : Φ2W ( two-shot ) and Φ 3 W ( thee-shot ) trained on the whole support set , Φ 1 A ( one-shot ) and Φ2B ( one-shot ) trained on support subsets SA and SB , respectively . All of these four FSL modules are using the same graph neural network ( GNN ) design which will be explained in the following section . In this work , we propose to explore the invariance property of the learned FSL feature across different support set configurations to optimize the generalization capability of the FSL . As illustrated in Figure 3 ( a ) , let X be the input image . Φ1A ( X ) and Φ 1 B ( X ) be the features generated by the one-shot learning networks trained on support subsets SA and SB , respectively . Let Φ2W ( X ) and Φ3W ( X ) be the features generated by the two-shot and three-shot networks trained on the whole support set W , respectively . During our experiments , we observe that it is not efficient to directly measure and optimize the distance between the one-shot and two-shot features . Instead , we introduce two auto-encoder networks , Γ2→1 and Γ3→1 to transform the feature vectors Φ2W ( X ) and Φ3W ( X ) , respectively , so as to match the feature vectors generated by the one-shot networks . Then , we introduce the following two feature invariance loss functions for the one-shot network Φ1A LA1→2 = EX { ||Φ1A ( X ) − Γ2→1 ( Φ2W ( X ) ) ||2 } , ( 1 ) LA1→3 = EX { ||Φ1A ( X ) − Γ3→1 ( Φ3W ( X ) ) ||2 } . ( 2 ) Here , EX { · } represents the statistical average over the training batch . Similarly , we can define the feature variance loss LB1→2 and LB1→3 for the one-shot network Φ1B . Note that one-shot networks Φ1A ( X ) and Φ 1 B ( X ) are trained over two different subsets of the support samples . This unique design is motivated by the following two considerations . ( 1 ) It can be used to measure and minimize the variation across the partition configuration variation , i.e. , different partitions of the support set for training . ( 2 ) The fusion of the features generated by these network can result in more efficient few-shot classification with improved generalization capability . To measure and minimize the invariance between these two one-shot networks , we propose to analyze the their softmax outputs . Specifically , let ΘA ( X ) = [ θ1A , θ 2 A , · · · , θIA ] be the output at the softmax layer of network Φ1A . Then , the probability for sample X to be in class i is given by piA ( X ) = exp ( θiA ) ∑I i=1 exp ( θ i A ) . ( 3 ) Similarly , we can define the class probability piB ( X ) for network Φ 1 B . We use the Kullback-Leibler ( KL ) Divergence to measure the invariance between the outputs of these two one-shot networks LP = ∑ X I∑ i=1 piA ( X ) log [ piA ( X ) piB ( X ) ] . ( 4 ) In our experiments , we use the loss function LP + L1→2 + L1→3 to train both one-shot networks . It should be noted that the loss function in ( 4 ) is defined for the whole batch of samples . In other words , the training process aim to the minimize the overall variation of a set of query samples , instead of an individual one . From our experiments , we observe that , for each individual query sample , the features generated by these two one-shot networks will differ slightly . To produce the feature F1 ( X ) for each individual query sample , we propose to fuse these two features using the following average operation F1 ( X ) = Φ1A ( X ) + Φ 1 B ( X ) 2 . ( 5 ) It should be noted that the above shot-partition configuration variations are only performed on the support set during the training stage . During the testing stage , once successfully trained , only the baseline 1-shot network is used for testing . Figure 3 ( b ) shows the network structure in the test stage . The support samples ( with labels ) and the query samples ( without labels ) are passed to both trained networks Φ1A and Φ 1 B to produce the features for the each query sample . These two features are averaged according to ( 5 ) , and then passed to the classification head to predict the query labels . We have used the 5-way 1-shot learning as an example to explain the proposed IGFE method . It can be easily extended to other settings of few-shot learning . For example , for 5-way 5-shot , those two networks Φ2W ( 2-shot ) and Φ 3 W ( 3-shot ) introduced for shot variations will be replaced with 10- shot and 15-shot configurations . Note that both the shot and the partition configurations are simply different ways to use the support samples during the training stage .

Summary: This paper mainly targets few-shot learning. The authors propose to enhance the invariance of features among different configurations. They merge the regularization with GNN architecture to achieve good results on several datasets.