A Study on Representation Transfer for Few-Shot Learning

1 INTRODUCTION . Few-shot learning ( FSL ) aims to train a prediction model with very few labeled examples . It is motivated both by our interest in the ability of humans to learn new concepts with very few examples ( Carey & Bartlett , 2011 ) , and a practical need to build machine learning models without the expensive process of collecting a large amount of labeled training data . In typical few-shot learning settings we assume we have access to a large amount of data ( labeled or unlabeled ) from a source domain as prior knowledge , and a small amount of labeled data for a target task in a different domain . Popular approaches for the few-shot learning problem include meta-learning and transfer learning . Meta-learning creates learning tasks that are similar to the target task from the source domain data , and tries to learn a model that can quickly adapt to a new task ( Finn et al. , 2017 ; Ren et al. , 2018 ; Wang et al. , 2019 ) . Transfer learning , on the other hand , transfers models learned from the source domain data to the target task through different adaptation techniques . In this work we focus on transfer learning through representations , which aims to learn a good feature representation from source data so that when learning the target task we can focus on training a much simpler task model ( e.g . a linear classifier ) instead of having to learn feature extraction at the same time . In computer vision and natural language processing , pre-training with large amount of data has become the dominant method for building state-of-art models ( Iglovikov & Shvets , 2018 ; Li et al. , 2018 ; Cui et al. , 2018 ; Ruder et al. , 2019 ) . For few-shot classification , recent works ( Tian et al. , 2020 ) show that even pre-training with simple classification task in the source domain gives a good feature extractor for training few-shot models . In this work we attempt to systematically study different representations for few-shot learning to identify the good ones , including representations from common supervised and self-supervised pretraining tasks ( Gidaris et al. , 2018 ; Doersch et al. , 2015 ; He et al. , 2020 ; Chen et al. , 2020a ) , and also representations from the meta-learning method MAML ( Finn et al. , 2017 ) by treating it as a feature extractor . Compared to typical evaluations of self-supervised representation learning which usually involve the same data distribution ( train and test on same input data but for different tasks ) , few-shot learning differ in two important aspects . The first one is data shift : the training and test data have different distribution and usually contain different object categories . The second one is extreme small sample size compared to typical evaluations of transfer learning . These differences make a proper study of these different representations worthwhile . In addition to learning good representations from source data , we believe there should also be emphasis on how to use them properly in few-shot settings . Representations learned from a large amount of pre-training data are usually high-dimensional , and for some few-shot learning task ( e.g. , 5-way 1-shot classification ) it is difficult to figure out which features are directly related to the target concept given only a handful of examples . In this work we also investigate ways to perform feature selection and noise reduction when adapting to the target task , given a good representation learned . In summary , our contributions in this work include : 1 ) We study systematically different representations for few-shot classification , to understand why some representations work better than others . 2 ) We investigate the use of multi-task feature representations in few-shot learning and show their utility . 3 ) We introduce feature selection via training with auxiliary classes , and a voting scheme that follows naturally from multi-task prediction to improve few-shot learning . 2 RELATED WORKS . Motivated by bridging the gap between AI and human learning ability , we have seen an ever increasing attention in the area of few-shot learning in machine learning community . Model Agnostic Meta-Learning ( MAML ) was proposed by Finn et al . ( 2017 ) ; it aims to search for a proper initialization of the neural network that can be rapidly adapted to variety of new tasks with a few optimization steps . Many improvements have been proposed based on MAML including first-order approximations ( Nichol et al. , 2018 ) or incorporating task-specific information ( Lee & Choi , 2018 ) , etc . Inspired by the concept of ” learning to learn ” , many metric-learning methods are also developed via meta-learning . By simulating the few-shot task setting while training the backbone model , these metric-learning models ( i.e Prototypical Networks , Relation Networks ) ( Snell et al. , 2017 ; Sung et al. , 2018 ) aim to learn the best feature representation which can be generalized well to unseen novel tasks . Our work is motivated by the recent studies Chen et al . ( 2019 ) and Tian et al . ( 2020 ) , which show that pre-training a good feature extractor ( e.g. , using supervised classification ) gives very good performance on few-shot classification . We continue along this line of study to investigate which pretraining tasks give the best feature extractor for few-shot classification . Our study of MAML as a feature extractor is motivated by the work in Raghu et al . ( 2019 ) , which show that MAML in few-shot learning benefits more from the learned features rather than fast model adaptation . Gidaris et al . ( 2019 ) and Su et al . ( 2020 ) study the use of self-supervised tasks including rotation ( Gidaris et al. , 2018 ) and jigsaw ( Noroozi & Favaro , 2016 ) to improve few-shot learning . They use the self-supervised tasks as feature regularizers on top of meta-learning algorithms such as ProtoNet ( Snell et al. , 2017 ) and MAML ( Finn et al. , 2017 ) , which is different from our direct joint representation learning followed by transfer . Our approach is also closely related to Doersch & Zisserman ( 2017 ) which consider jointly using multiple self-supervised tasks to learn visual representations for classification . Zhai et al . ( 2019 ) has done a large scale study of the performance of representation learning on a large set of downstream tasks , while we focus on few-shot classification in this work . 3 REPRESENTATION TRANSFER FOR FEW-SHOT LEARNING . In all the following experiments we adopt the following protocol to investigate transfer learning . We first train a feature representation using one or multiple tasks on the source domain data , and use the last layer of activations as our feature representation . During adaptation stage to the new fewshot domain , we train a linear logistic regression classifier on top of the fixed feature representation to adapt the model . This is a common protocol in many evaluations of representation learning methods . In this work we focus on two backbones : ConvNet4 and ResNet12 . ConvNet4 is a 4-layer convolutional neural network with hidden layer size of 64 in each layer , where each layer is a 3x3 convolution followed by ReLU , batchnorm , and max-pooling of stride 2 . ResNet12 is a residual network ( He et al. , 2016 ) with 4 residual blocks of width 64-160-320-640 . We also consider a wider version of ResNet12 which we denote as ResNetW12 by increasing the width to 128-320-640-640 , to better accommodate multi-task learning . The feature representation size is fixed at 640 . Our code is implemented in PyTorch ( Paszke et al. , 2019 ) . We focus on the two most commonly used datasets in studying few-shot learning , Mini-ImageNet ( Vinyals et al. , 2016 ) and Tierd-ImageNet ( Ren et al. , 2018 ) , in the work below . We found that there are two ways to construct the 84x84 inputs of these datasets from ImageNet ( see Appendix A.1 ) . We use the approach of directly taking random resized crops of 84x84 on the ImageNet images , as it allows us to experiment on more diverse set of representation learning methods . 3.1 REPRESENTATIONS FROM META-LEARNING . We begin our study of representation transfer for few-shot learning with feature representations produced by MAML . Previous studies Raghu et al . ( 2019 ) show that the benefits of MAML come mainly from the feature representation learned in the last layer of the neural network instead of the ability to do fast adaptation . In this section we want to go further and consider the question : from a representation transfer point of view , what source tasks are the best for a target distribution of few-shot learning tasks ? Is learning from a set of 5-way 1-shot source tasks always the best if we want to perform 5-way 1-shot classification in the target domain ? We use a modified version of MAML algorithm ( Algorithm 1 ) that only adapts the last layer of the neural network on the support set , as adapting the last layer only performs very close to adapting the whole network ( the almost-no-inner-loop ( ANIL ) algorithm in ( Raghu et al. , 2019 ) ) . We split the parameters of the neural network into weights for the linear layer φ and the rest of the earlier feature extraction layers θ , and write gφ ( fθ ( x ) ) for the output of network with input x . We use ( XSi , Y S i , X Q i , Y Q i ) denoting the support set inputs , support set labels , query set inputs , query set labels to represent the sampled few-shot classification task Ti . Note that only the linear layer parameters φ are updated in the inner loop using the support set , reflecting that we only update the linear layer during adaptation . Parameters for feature representation θ are only updated in the outer loop . Algorithm 1 Modified MAML for few-shot classification that adapts only the last layer ( ANIL ) Given : inner and outer step size hyperparameters α , β , task distribution p Randomly initialize θ , φ while not done do Sample batch of tasks Ti ∼ p ( T ) for all Ti = ( XSi , Y Si , X Q i , Y Q i ) do Update φ′i ← φ− α∇φL ( gφ ( fθ ( XSi ) ) , Y Si ) end for Update θ ← θ − β∇θ ∑ Ti L ( gφ′i ( fθ ( X Q i ) ) , Y Q i ) , φ← φ− β∇φ ∑ Ti L ( gφ′i ( fθ ( X Q i ) ) , Y Q i ) end while We train 5-way 1-shot ( 5w1s ) , 5-way 5-shot ( 5w5s ) , 10-way 1-shot ( 10w1s ) and 10-way 5- shot ( 10w5s ) tasks on Mini-ImageNet and Tiered-ImageNet , using both the ConvNet4 and ResNet12 architecture . To evaluate the representations learned , we extract the last layer features from these n-way k-shot networks , and train a logistic regression model on top of it during the adaptation stage on the support set . Each accuracy number reported in Tables 1 and 2 are the median of 5 trials , each of which is the average of 600 random draws of n-way k-shot tasks . The MAML algorithm is implemented using the higher package ( Grefenstette et al. , 2019 ) . We use an outer loop step size of 0.001 and inner step size of 0.01 for 5-way tasks and 0.05 for 10-way tasks . There are 5 inner loop steps during training and 10 inner loop steps during evaluations . We use a task batch size of 4 and run for 400 epochs . Due to the occassional overfitting of ResNet12 models , we select the model that performs best in terms of loss on the validation set provided in Mini-ImageNet and Tiered-Imagenet , and report the results on the test set . From the results on ConvNet4 in Table 1 , we can see that contrary to intuition , training and testing on the same type of n-way k-shot tasks do not result in the best performance . In general , representations learned from 10-way tasks perform better than representations learned from 5-way tasks , irrespective of the number of ways in the target tasks . And learning with more shots are also usually better . The results for ResNet12 are similar ( Table 2 ) , except for the 5w1s outlier in Mini-ImageNet . Snell et al . ( 2017 ) has previously noted the effect of models trained on larger number of ways performing better than models trained on number of ways matching the target task , while Cao et al . ( 2019 ) and Triantafillou et al . ( 2020 ) have studied the effect of number of shots in few-shot learning . From an adaptation point of view , 5-way models should transfer better to 5-way tasks , since they come from a more ‘ similar ’ task distribution . However , from a feature learning perspective , this is not surprising . More difficult pairing of classes like cat VS tiger are more common for higher way tasks in a random sampling setting , thus they force the models to learn more discriminative features that might transfer better to new domains . Note that our adaptation here is based on logistic regression , which is different from fast adaptation by performing a few gradient steps in MAML using the linear weights learned in the last layer . The logistic regression results sometimes can be 1-2 % lower than adaptation using MAML , since the feature representations are trained using MAML and the last layer linear weights also encode some prior information on what the adapted weights should be close to . However as we use logistic regression for all model adaptations the comparison should be fair , and the results indicate increasing the number of ways allow better representations to be learned for transfer .

Summary: This paper conducts an empirical study on the usefulness of different representations for downstream few-shot learning tasks, using the transfer strategy of training a new linear layer on top of the frozen pre-trained network. They also propose to utilize auxiliary data during this downstream learning phase: instead of learning an N-way readout layer for an N-way classification task, they learn a (N+T)-way one, where T is the number of training classes, using a subset of the training dataset for this. To perform inference on the query set, though, they only consider the N classes appearing in the given task. They hypothesize that this approach prevents the linear classifier from paying attention to spurious features, and it sometimes yields a gain in performance. They also propose to combine different training objectives, including the supervised classification objective and different self-supervised ones in a multi-task manner, and propose to do so by keeping the input the same for all objectives involved, meaning that to train the supervised loss together with the rotation, the former will also be computed on rotated images. Finally, they propose a voting scheme, where the predictions for different augmentations of each input image are aggregated to form the final prediction. All experiments are conducted on mini-ImageNet and tiered-ImageNet, and gains are reported mostly on the former.