A Study on Representation Transfer for Few-Shot Learning

1 INTRODUCTION . Few-shot learning ( FSL ) aims to train a prediction model with very few labeled examples . It is motivated both by our interest in the ability of humans to learn new concepts with very few examples ( Carey & Bartlett , 2011 ) , and a practical need to build machine learning models without the expensive process of collecting a large amount of labeled training data . In typical few-shot learning settings we assume we have access to a large amount of data ( labeled or unlabeled ) from a source domain as prior knowledge , and a small amount of labeled data for a target task in a different domain . Popular approaches for the few-shot learning problem include meta-learning and transfer learning . Meta-learning creates learning tasks that are similar to the target task from the source domain data , and tries to learn a model that can quickly adapt to a new task ( Finn et al. , 2017 ; Ren et al. , 2018 ; Wang et al. , 2019 ) . Transfer learning , on the other hand , transfers models learned from the source domain data to the target task through different adaptation techniques . In this work we focus on transfer learning through representations , which aims to learn a good feature representation from source data so that when learning the target task we can focus on training a much simpler task model ( e.g . a linear classifier ) instead of having to learn feature extraction at the same time . In computer vision and natural language processing , pre-training with large amount of data has become the dominant method for building state-of-art models ( Iglovikov & Shvets , 2018 ; Li et al. , 2018 ; Cui et al. , 2018 ; Ruder et al. , 2019 ) . For few-shot classification , recent works ( Tian et al. , 2020 ) show that even pre-training with simple classification task in the source domain gives a good feature extractor for training few-shot models . In this work we attempt to systematically study different representations for few-shot learning to identify the good ones , including representations from common supervised and self-supervised pretraining tasks ( Gidaris et al. , 2018 ; Doersch et al. , 2015 ; He et al. , 2020 ; Chen et al. , 2020a ) , and also representations from the meta-learning method MAML ( Finn et al. , 2017 ) by treating it as a feature extractor . Compared to typical evaluations of self-supervised representation learning which usually involve the same data distribution ( train and test on same input data but for different tasks ) , few-shot learning differ in two important aspects . The first one is data shift : the training and test data have different distribution and usually contain different object categories . The second one is extreme small sample size compared to typical evaluations of transfer learning . These differences make a proper study of these different representations worthwhile . In addition to learning good representations from source data , we believe there should also be emphasis on how to use them properly in few-shot settings . Representations learned from a large amount of pre-training data are usually high-dimensional , and for some few-shot learning task ( e.g. , 5-way 1-shot classification ) it is difficult to figure out which features are directly related to the target concept given only a handful of examples . In this work we also investigate ways to perform feature selection and noise reduction when adapting to the target task , given a good representation learned . In summary , our contributions in this work include : 1 ) We study systematically different representations for few-shot classification , to understand why some representations work better than others . 2 ) We investigate the use of multi-task feature representations in few-shot learning and show their utility . 3 ) We introduce feature selection via training with auxiliary classes , and a voting scheme that follows naturally from multi-task prediction to improve few-shot learning . 2 RELATED WORKS . Motivated by bridging the gap between AI and human learning ability , we have seen an ever increasing attention in the area of few-shot learning in machine learning community . Model Agnostic Meta-Learning ( MAML ) was proposed by Finn et al . ( 2017 ) ; it aims to search for a proper initialization of the neural network that can be rapidly adapted to variety of new tasks with a few optimization steps . Many improvements have been proposed based on MAML including first-order approximations ( Nichol et al. , 2018 ) or incorporating task-specific information ( Lee & Choi , 2018 ) , etc . Inspired by the concept of ” learning to learn ” , many metric-learning methods are also developed via meta-learning . By simulating the few-shot task setting while training the backbone model , these metric-learning models ( i.e Prototypical Networks , Relation Networks ) ( Snell et al. , 2017 ; Sung et al. , 2018 ) aim to learn the best feature representation which can be generalized well to unseen novel tasks . Our work is motivated by the recent studies Chen et al . ( 2019 ) and Tian et al . ( 2020 ) , which show that pre-training a good feature extractor ( e.g. , using supervised classification ) gives very good performance on few-shot classification . We continue along this line of study to investigate which pretraining tasks give the best feature extractor for few-shot classification . Our study of MAML as a feature extractor is motivated by the work in Raghu et al . ( 2019 ) , which show that MAML in few-shot learning benefits more from the learned features rather than fast model adaptation . Gidaris et al . ( 2019 ) and Su et al . ( 2020 ) study the use of self-supervised tasks including rotation ( Gidaris et al. , 2018 ) and jigsaw ( Noroozi & Favaro , 2016 ) to improve few-shot learning . They use the self-supervised tasks as feature regularizers on top of meta-learning algorithms such as ProtoNet ( Snell et al. , 2017 ) and MAML ( Finn et al. , 2017 ) , which is different from our direct joint representation learning followed by transfer . Our approach is also closely related to Doersch & Zisserman ( 2017 ) which consider jointly using multiple self-supervised tasks to learn visual representations for classification . Zhai et al . ( 2019 ) has done a large scale study of the performance of representation learning on a large set of downstream tasks , while we focus on few-shot classification in this work . 3 REPRESENTATION TRANSFER FOR FEW-SHOT LEARNING . In all the following experiments we adopt the following protocol to investigate transfer learning . We first train a feature representation using one or multiple tasks on the source domain data , and use the last layer of activations as our feature representation . During adaptation stage to the new fewshot domain , we train a linear logistic regression classifier on top of the fixed feature representation to adapt the model . This is a common protocol in many evaluations of representation learning methods . In this work we focus on two backbones : ConvNet4 and ResNet12 . ConvNet4 is a 4-layer convolutional neural network with hidden layer size of 64 in each layer , where each layer is a 3x3 convolution followed by ReLU , batchnorm , and max-pooling of stride 2 . ResNet12 is a residual network ( He et al. , 2016 ) with 4 residual blocks of width 64-160-320-640 . We also consider a wider version of ResNet12 which we denote as ResNetW12 by increasing the width to 128-320-640-640 , to better accommodate multi-task learning . The feature representation size is fixed at 640 . Our code is implemented in PyTorch ( Paszke et al. , 2019 ) . We focus on the two most commonly used datasets in studying few-shot learning , Mini-ImageNet ( Vinyals et al. , 2016 ) and Tierd-ImageNet ( Ren et al. , 2018 ) , in the work below . We found that there are two ways to construct the 84x84 inputs of these datasets from ImageNet ( see Appendix A.1 ) . We use the approach of directly taking random resized crops of 84x84 on the ImageNet images , as it allows us to experiment on more diverse set of representation learning methods . 3.1 REPRESENTATIONS FROM META-LEARNING . We begin our study of representation transfer for few-shot learning with feature representations produced by MAML . Previous studies Raghu et al . ( 2019 ) show that the benefits of MAML come mainly from the feature representation learned in the last layer of the neural network instead of the ability to do fast adaptation . In this section we want to go further and consider the question : from a representation transfer point of view , what source tasks are the best for a target distribution of few-shot learning tasks ? Is learning from a set of 5-way 1-shot source tasks always the best if we want to perform 5-way 1-shot classification in the target domain ? We use a modified version of MAML algorithm ( Algorithm 1 ) that only adapts the last layer of the neural network on the support set , as adapting the last layer only performs very close to adapting the whole network ( the almost-no-inner-loop ( ANIL ) algorithm in ( Raghu et al. , 2019 ) ) . We split the parameters of the neural network into weights for the linear layer φ and the rest of the earlier feature extraction layers θ , and write gφ ( fθ ( x ) ) for the output of network with input x . We use ( XSi , Y S i , X Q i , Y Q i ) denoting the support set inputs , support set labels , query set inputs , query set labels to represent the sampled few-shot classification task Ti . Note that only the linear layer parameters φ are updated in the inner loop using the support set , reflecting that we only update the linear layer during adaptation . Parameters for feature representation θ are only updated in the outer loop . Algorithm 1 Modified MAML for few-shot classification that adapts only the last layer ( ANIL ) Given : inner and outer step size hyperparameters α , β , task distribution p Randomly initialize θ , φ while not done do Sample batch of tasks Ti ∼ p ( T ) for all Ti = ( XSi , Y Si , X Q i , Y Q i ) do Update φ′i ← φ− α∇φL ( gφ ( fθ ( XSi ) ) , Y Si ) end for Update θ ← θ − β∇θ ∑ Ti L ( gφ′i ( fθ ( X Q i ) ) , Y Q i ) , φ← φ− β∇φ ∑ Ti L ( gφ′i ( fθ ( X Q i ) ) , Y Q i ) end while We train 5-way 1-shot ( 5w1s ) , 5-way 5-shot ( 5w5s ) , 10-way 1-shot ( 10w1s ) and 10-way 5- shot ( 10w5s ) tasks on Mini-ImageNet and Tiered-ImageNet , using both the ConvNet4 and ResNet12 architecture . To evaluate the representations learned , we extract the last layer features from these n-way k-shot networks , and train a logistic regression model on top of it during the adaptation stage on the support set . Each accuracy number reported in Tables 1 and 2 are the median of 5 trials , each of which is the average of 600 random draws of n-way k-shot tasks . The MAML algorithm is implemented using the higher package ( Grefenstette et al. , 2019 ) . We use an outer loop step size of 0.001 and inner step size of 0.01 for 5-way tasks and 0.05 for 10-way tasks . There are 5 inner loop steps during training and 10 inner loop steps during evaluations . We use a task batch size of 4 and run for 400 epochs . Due to the occassional overfitting of ResNet12 models , we select the model that performs best in terms of loss on the validation set provided in Mini-ImageNet and Tiered-Imagenet , and report the results on the test set . From the results on ConvNet4 in Table 1 , we can see that contrary to intuition , training and testing on the same type of n-way k-shot tasks do not result in the best performance . In general , representations learned from 10-way tasks perform better than representations learned from 5-way tasks , irrespective of the number of ways in the target tasks . And learning with more shots are also usually better . The results for ResNet12 are similar ( Table 2 ) , except for the 5w1s outlier in Mini-ImageNet . Snell et al . ( 2017 ) has previously noted the effect of models trained on larger number of ways performing better than models trained on number of ways matching the target task , while Cao et al . ( 2019 ) and Triantafillou et al . ( 2020 ) have studied the effect of number of shots in few-shot learning . From an adaptation point of view , 5-way models should transfer better to 5-way tasks , since they come from a more ‘ similar ’ task distribution . However , from a feature learning perspective , this is not surprising . More difficult pairing of classes like cat VS tiger are more common for higher way tasks in a random sampling setting , thus they force the models to learn more discriminative features that might transfer better to new domains . Note that our adaptation here is based on logistic regression , which is different from fast adaptation by performing a few gradient steps in MAML using the linear weights learned in the last layer . The logistic regression results sometimes can be 1-2 % lower than adaptation using MAML , since the feature representations are trained using MAML and the last layer linear weights also encode some prior information on what the adapted weights should be close to . However as we use logistic regression for all model adaptations the comparison should be fair , and the results indicate increasing the number of ways allow better representations to be learned for transfer .

Summary: The authors study impact of self-supervised methods on feature representation in few-shot learning task. They analyze the way features are transferred for MAML method and 3 self-supervised methods. They introduced a method combining self-supervised representation training methods with voting system in order to improve performance.