Invariance-Guided Feature Evolution  for  Few-Shot Learning

1 INTRODUCTION . Few-shot image classification aims to learn to predict the labels of query images from very small sets of support samples ( Finn et al. , 2017 ; Yu et al. , 2020 ) . More importantly , this learned representation needs to generalize well onto unseen classes during the test stage . Since the number of support images per class , also called shots , is very small , often in the range of 1 to 5 , the characteristics of support samples will vary significantly . Therefore , a successful few-shot learning ( FSL ) should be able to survive this large variation of image-specific characteristics in the support set ( Yu et al. , 2020 ; Bronskill et al. , 2020 ) . This requires that the FSL be able to capture the underlying or inherent visual relationship between the query and support images , instead of being distracted or over-fitted by the image- specific characteristics , especially those non-salient features from the background . In other words , this inherent visual relationship , once successfully learned , should remain relatively invariant across different sample configurations of the support sets . Otherwise , its classification performance with very few support samples and generalization capability onto novel classes will degrade significantly . Based on this observation , we put forward the following observation : the feature representation generated by the FSL , if successfully learned , should remain relatively invariant to sample configurations of the support sets , for example , the number of shots ( i.e , the number of samples per class ) and the selection of support samples for training . Driven by this observation , we propose to construct a feature evolution network with an ensemble of few-shot learners which evolve the feature embedding along dimensions of the support set configuration , as illustrated in Figure 1 . In this work , we choose to study two major parameters that control the support set configuration : the number of labeled samples per class ( called shots ) and the percentage of training samples ( called partition ) in the support set . This network aims to characterize and track the evolution and variation of learned features of the query samples across different shot-partition configuration of the support samples . We then introduce an invariance metric to measure the variation of the learned query features . This invariance metric serves as a loss function to be optimized during the FSL process . Our extensive experimental results demonstrate that the proposed IGFE method significantly improves the performance of FSL and outperforms existing state-of-the-art methods by large margins on various benchmark datasets . 2 RELATED WORK . There are three major categories of methods that have been developed for few-shot learning : meta learning , transfer learning , and metric learning . ( 1 ) In meta-learning based methods , episode training , which mimics the test tasks during the training stage , is explored to improve the knowledge transfer and generalization capability from metatraining to meta-testing ( Finn et al. , 2017 ; Rusu et al. , 2018 ) . In Zhang et al . ( 2020a ) ; Ye et al . ( 2020 ) ; Chen et al . ( 2020b ) , meta-learning is combined with pre-training to further improve the performance . Tian et al . ( 2020 ) ; Rizve et al . ( 2021 ) decoupled the learning procedure into the baseclass embedding pre-training and novel-class classifier learning and develop a multivariate logistic regression and knowledge distillation method to improve the meta-learning performance . ( 2 ) Transfer-learning methods ( Qiao et al. , 2018 ; Gidaris & Komodakis , 2018 ; Chen et al. , 2019 ) pre-train the model with a large set of base classes and then adapt the model to unseen classes . Transmatch ( Yu et al. , 2020 ) pre-trains a feature extractor on the base-class data , uses it as the feature extractor to initialize the classifier weights for unseen classes , and updates the model using semisupervised learning methods . Gidaris & Komodakis ( 2018 ) propose an attention module to predict the classifier weights for novel classes dynamically . Instead of separating the base classes into a set of few-shot tasks , methods in Qiao et al . ( 2018 ) ; Chen et al . ( 2019 ) use all base classes to pre-train the few-shot model , which is then adapted to novel-class recognition . ( 3 ) Metric-based methods ( Sung et al. , 2018 ; Zhang et al. , 2020b ) aim to learn an effective measurement space and identify query samples based on the feature distance between samples . MatchingNet ( Vinyals et al. , 2016 ) shows that neural networks can be successfully trained to measure the similarity between query and support samples . The prototype network method ( Snell et al. , 2017 ) introduces a prototype , which represents a class . The distance between prototypes and a query vector is used for the classification . Recent methods utilize task-specific prototypes . For example , FEAT ( Ye et al. , 2020 ) applies self-attention-based transformation on the prototypes . BSNet ( Li et al. , 2021 ) uses a dual-metric distance and achieves better performance than a single-metric distance . Recently , graph-based methods have been explored to model and predict the correlation between samples , aiming to propagate meta-knowledge and the labels from base classes to novel classes ( Rodrı́guez et al. , 2020 ; Satorras & Estrach , 2018 ) . Chen et al . ( 2020a ) develop a graph neural network-based few-shot learning method which iteratively aggregates features from neighbors and learn the correlation between samples . DPGN ( Yang et al. , 2020 ) has proposed a two-fold complete graph network to model instance-level and distribution-level relations with label propagation and transduction . The MCGN method ( Tang et al. , 2021 ) combines GNN and conditional random field ( CRF ) to achieve better affinity prediction . Knowledge distillation . This work is related to knowledge distillation , which learns a small student network using the supervision signals from both ground truth labels and a larger teacher network ( Gou et al. , 2021 ) . Three different types of knowledge or supervision signals have been explored in the literature . ( 1 ) Response-based knowledge distillation uses the neural response of the last output layer of the teacher network as the supervision information ( Hinton et al. , 2015 ; Zhang et al. , 2019 ; Meng et al. , 2019 ) . ( 2 ) Feature-based knowledge distillation uses the feature map of the middle layer as the supervision signal to train the student network ( Romero et al. , 2014 ; Wang et al. , 2020 ; Chen et al. , 2021 ) . ( 3 ) Relationship-based knowledge distillation further explores the relationship between different layers or data samples ( Yim et al. , 2017 ; Passalis et al. , 2020 ; Dong et al. , 2021 ) . For example , the second order statistics between different network layers are used in Yim et al . ( 2017 ) as the supervision signals . 3 METHOD . In this section , we present our proposed method of invariance-guided feature evolution ( IGFE ) for few-shot learning . 3.1 CLASS FEATURE VARIATIONS ACROSS DIFFERENT SUPPORT SET CONFIGURATIONS . In this section , we experimentally study the variation of query set features across different support set configurations , which motivates us to make the invariance observation discussed in Section 1 . In a typical setting , a K-way N -shot FSL has K classes with N support samples per class in its training batch . The training samples from one class will be divided into M batches . In total , we have K ×M samples per class . The performance of the FSL directly depends on the configuration of the support set . In this work , we study two important parameters that plays important roles in FSL : ( 1 ) N , the number of shots or the number of support samples per class , and ( 2 ) M , the number of sample batches or the selected partition of the support set used for training . For simplicity , we refer to these types of configuration as shot configuration and partition configuration . In the following experiments , we perform fewshot learning on the mini-ImageNet dataset . Figure 2 ( a ) shows the t-SNE plots for query sample features of 5 classes for different shots N . In each shot configuration , we choose different subset partitions of the support samples for training . We can see that , for FSL with larger N , the variation of query class features is becoming smaller . This implies that , in a better learned FSL model , the feature variation across different support set partition is becoming smaller . Figure 2 ( b ) shows the experimental results for different numbers of batches or subset partitions of the support set used for training . For each batch or support set partition , we perform the FSL with different N , and compute the average variation of query class features . We can see that , as more batches of support samples are used , the FSL is better learned , the query class feature variation across different shots is becoming smaller . These experimental results motivate us to make the following observation : in a better learned FSL model , the variation of query sample feature across different shot and partition configurations of the support will be smaller . Driven by this observation , we develop the invariance-guided feature evolution ( IGFE ) method for few-shot learning , which will be explained int the following section . 3.2 INVARIANCE-GUIDED FEATURE EVOLUTION . Figure 3 ( a ) provides an overview of the proposed invariance-guided feature evolution ( IGFE ) method . we use the one-shot learning as an example to explain our proposed IGFE method . For the partition variation , we use three different subsets of the support samples for training : ( 1 ) SW which is the whole support set ; ( 2 ) SA and SB which are two randomly chosen subsets of the support set . For example , in our experiments , each subset contains 60 % of the whole support samples . For the shot variation , besides the baseline one-shot learning module , we introduce two additional FSL modules with N = 2 ( two-shot ) and N = 3 ( three-shot ) . In this way , we have the following four FSL modules : Φ2W ( two-shot ) and Φ 3 W ( thee-shot ) trained on the whole support set , Φ 1 A ( one-shot ) and Φ2B ( one-shot ) trained on support subsets SA and SB , respectively . All of these four FSL modules are using the same graph neural network ( GNN ) design which will be explained in the following section . In this work , we propose to explore the invariance property of the learned FSL feature across different support set configurations to optimize the generalization capability of the FSL . As illustrated in Figure 3 ( a ) , let X be the input image . Φ1A ( X ) and Φ 1 B ( X ) be the features generated by the one-shot learning networks trained on support subsets SA and SB , respectively . Let Φ2W ( X ) and Φ3W ( X ) be the features generated by the two-shot and three-shot networks trained on the whole support set W , respectively . During our experiments , we observe that it is not efficient to directly measure and optimize the distance between the one-shot and two-shot features . Instead , we introduce two auto-encoder networks , Γ2→1 and Γ3→1 to transform the feature vectors Φ2W ( X ) and Φ3W ( X ) , respectively , so as to match the feature vectors generated by the one-shot networks . Then , we introduce the following two feature invariance loss functions for the one-shot network Φ1A LA1→2 = EX { ||Φ1A ( X ) − Γ2→1 ( Φ2W ( X ) ) ||2 } , ( 1 ) LA1→3 = EX { ||Φ1A ( X ) − Γ3→1 ( Φ3W ( X ) ) ||2 } . ( 2 ) Here , EX { · } represents the statistical average over the training batch . Similarly , we can define the feature variance loss LB1→2 and LB1→3 for the one-shot network Φ1B . Note that one-shot networks Φ1A ( X ) and Φ 1 B ( X ) are trained over two different subsets of the support samples . This unique design is motivated by the following two considerations . ( 1 ) It can be used to measure and minimize the variation across the partition configuration variation , i.e. , different partitions of the support set for training . ( 2 ) The fusion of the features generated by these network can result in more efficient few-shot classification with improved generalization capability . To measure and minimize the invariance between these two one-shot networks , we propose to analyze the their softmax outputs . Specifically , let ΘA ( X ) = [ θ1A , θ 2 A , · · · , θIA ] be the output at the softmax layer of network Φ1A . Then , the probability for sample X to be in class i is given by piA ( X ) = exp ( θiA ) ∑I i=1 exp ( θ i A ) . ( 3 ) Similarly , we can define the class probability piB ( X ) for network Φ 1 B . We use the Kullback-Leibler ( KL ) Divergence to measure the invariance between the outputs of these two one-shot networks LP = ∑ X I∑ i=1 piA ( X ) log [ piA ( X ) piB ( X ) ] . ( 4 ) In our experiments , we use the loss function LP + L1→2 + L1→3 to train both one-shot networks . It should be noted that the loss function in ( 4 ) is defined for the whole batch of samples . In other words , the training process aim to the minimize the overall variation of a set of query samples , instead of an individual one . From our experiments , we observe that , for each individual query sample , the features generated by these two one-shot networks will differ slightly . To produce the feature F1 ( X ) for each individual query sample , we propose to fuse these two features using the following average operation F1 ( X ) = Φ1A ( X ) + Φ 1 B ( X ) 2 . ( 5 ) It should be noted that the above shot-partition configuration variations are only performed on the support set during the training stage . During the testing stage , once successfully trained , only the baseline 1-shot network is used for testing . Figure 3 ( b ) shows the network structure in the test stage . The support samples ( with labels ) and the query samples ( without labels ) are passed to both trained networks Φ1A and Φ 1 B to produce the features for the each query sample . These two features are averaged according to ( 5 ) , and then passed to the classification head to predict the query labels . We have used the 5-way 1-shot learning as an example to explain the proposed IGFE method . It can be easily extended to other settings of few-shot learning . For example , for 5-way 5-shot , those two networks Φ2W ( 2-shot ) and Φ 3 W ( 3-shot ) introduced for shot variations will be replaced with 10- shot and 15-shot configurations . Note that both the shot and the partition configurations are simply different ways to use the support samples during the training stage .

Summary: This paper studies feature variance in a FSL model. It observes that support set configuration can greatly affect the feature distribution. To that end, this paper proposes modules based on GNN to learn features invariant to hyper-parameters. Specifically, this method subsamples training set into 4 subsets and each subset is encoded by the same backbone network + GNN. Then, feature invariance is enforced across subsets. By doing so, this method learns features that are invariant to both number of shots and the percentage of training samples, which are beneficial to few-shot learning evaluation.