On the Role of Pre-training for Meta Few-Shot Learning

1 INTRODUCTION . In recent years , deep learning methods have outperformed most of the traditional methods in supervised learning , especially in image classification . However , deep learning methods generally require lots of labeled data to achieve decent performance . Some applications , however , do not have the luxury to obtain lots of labeled data . For instance , for bird classification , an ornithologist typically can only obtain a few pictures per bird species to update the classifier . Such needs of building classifiers from limited labeled data inspire some different research problems , including the few-shot learning problem ( Finn et al. , 2017 ; Snell et al. , 2017 ; Rajeswaran et al. , 2019 ; Oreshkin et al. , 2018 ; Vinyals et al. , 2016 ; Lee et al. , 2019 ) . In particular , few-shot learning starts with a training dataset that consists of data points for “ seen ” classes , and is required to classify “ unseen ” ones in the testing phase accurately based on limited labeled data points from unseen classes . Currently , there are two main frameworks , meta-learning ( Finn et al. , 2017 ; Snell et al. , 2017 ; Chen et al. , 2019 ) and transfer learning ( Dhillon et al. , 2020 ) , that deal with the few-shot learning problem . For transfer learning , the main idea is to train a traditional classifier on the meta-train dataset . In the testing phase , these methods finetune the model on the limited datapoints for the labeled novel classes . For meta-learning frameworks , their main concept is episodic training ( Vinyals et al. , 2016 ) . For the testing phase of few-shot learning , the learning method is given N novel classes , each containing K labeled data for fine-tuning and Q query data for evaluation . Unlike transfer learning algorithms , episodic training tries to simulate the testing literature in the training phase by sampling episodes in training dataset . In these two years , some transfer-learning methods ( Dhillon et al. , 2020 ) with sophisticated design in the finetuning part have a competitive performance to the meta-learning approaches . Moreover , researchers ( Lee et al. , 2019 ; Sun et al. , 2019 ; Chen et al. , 2019 ; Oreshkin et al. , 2018 ) have pointed out that combining both the global classifier ( pre-training part ) in the transfer learning framework and the episodic training concept for the meta-learning framework could lead to better performance . Yet , currently most of the attentions are on the episodic training part ( Vinyals et al. , 2016 ; Finn et al. , 2017 ; Snell et al. , 2017 ; Oreshkin et al. , 2018 ; Sun et al. , 2019 ; Lee et al. , 2019 ) and the role of pre-training is still vague . Meta-learning and pre-training has both improved a lot in the past few years . However , most of the works focus on the accuracy instead of the efficiency . For meta-learning , to make the progress more efficient , an intuitive way is to reduce the number of episodes . Currently , there are only limited researches ( Sun et al. , 2019 ) working on reducing the number of episodes . One of the methods ( Chen et al. , 2019 ; Lee et al. , 2019 ) is to apply a better weight initialization method , the one from pre-training , instead of the random initialization . Another method ( Sun et al. , 2019 ) is to mimic how people learn . For example , when we are learning dynamic programming , given a knapsack problem with simple constraint and the one with strong constraint , we will learn much more when we solve the problem with strong constraint . Sun et al . ( 2019 ) followed the latter idea and crafted the hard episode to decrease amount of necessary episodes . In this work , we study the role of pre-training in meta few-shot learning . We study the pre-training from the disentanglement of the representations . Disentanglement is the property that whether the datapoints within different classes has been mixed together . Frosst et al . ( 2019 ) pointed out that instead of the last layer of the model all representations after other layers were entangled . The last layer does the classifier and the rest captures some globally shared information . By analyzing the disentanglement property of episodic training , though the pre-training gives a better representation that benefits the episodic training , the representation becomes more disentangled after episodic training . That is to say , episodic training has spent some effort on making the representation more disentangled . Benefited from the understanding , we design a sophisticated pre-training method that is more disentangled and helps episodic training achieve competitive performance more efficiently . With our pre-training loss , the classical meta-learning algorithm , ProtoNet ( Snell et al. , 2017 ) , achieves competitive performance to other methods . Our study not only benefits the episodic training but also points out another direction to sharpen and speed-up episodic training . To sum up , there are three main contributions in this work : 1 . A brief study of the role of pre-training in episodic training . 2 . A simple regularization loss that sharpens the classical meta-learning algorithms . 3 . A new aspect for reducing the necessary episodic training episodes . 2 RELATED WORK . Few-shot learning tries to mimic the human ability to generalize to novel classes with limited datapoints . In the following , we briefly introduce the recent progress of the transfer-learning framework and two categories of the meta-learning framework . Afterward , we give a brief introduction of the not well studied episode efficiency problem . 2.1 TRANSFER-LEARNING FRAMEWORK . In the training phase , the transfer-learning framework trains a classifier on the general classification task across all base classes instead of utilizing episodic training . And for the testing phase , transferlearning methods finetune the model with the limited labeled data . There are several kinds of tricks . Qi et al . ( 2018 ) proposed a method to append the mean of the embedding with a given class as a final layer of the classifier . Qiao et al . ( 2018 ) used the parameter of the last activation output to predict the classifier for novel classes dynamically . Gidaris & Komodakis ( 2018 ) proposed a similar concept with Qiao et al . ( 2018 ) . They also embedded the weight of base classes during the novel class prediction . Moreover , they introduced an attention mechanism instead of directly averaging among the parameters of each shot . Besides embedding base classes weight to the final classifier , Dhillon et al . ( 2020 ) utilized label propagation by the uncertainty on a single prediction to prevent overfitting in the finetune stage , which is quite similar to the classical classification tasks . 2.2 META-LEARNING FRAMEWORK . For meta-learning like framework , the main concepts are learning to learn and episodic training ( Vinyals et al. , 2016 ) . Learning to learn refers to learn from a lot of tasks to benefit the new task learning . To prevent confusion , the original train and test phase are regarded as “ meta-train ” and “ meta-test ” . The term “ train ” and “ test ” would be referred to the one in each small task . Episodic training is the process of mimicking the task structure in meta-test during training . If the meta-test phase consists of K support examples and Q query examples from N classes , then we will sample lots of tasks that also have K support examples and Q query examples from N classes in the metatrain phase . Meta-learning algorithms have developed rapidly in recent years . We briefly categorize them into two categories , optimization-based methods , metric-based methods . Optimization-based Method Optimization-based methods try to get an embedding that could easily fit subtasks by adding some extra layers . Finn et al . ( 2017 ) proposed MAML ( Model-Agnostic Meta-Learning ) , which got a network that was the closest to all the best model in the low-way lowshot tasks . However , MAML might be quite inefficient due to the computation of Hessian matrix . To leverage the issue , iMAML ( Rajeswaran et al. , 2019 ) and foMAML ( Finn et al. , 2017 ) provides different approximation to avoid the heavy computation . However , MAML still suffers from the high dimensional overfitting issue . LEO model ( Rusu et al. , 2019 ) solves the overfitting issue by learning a low dimensional latent space . Instead of aiming to get an embedding that could benefit the latter fully connected layer , MetaOptNet ( Lee et al. , 2019 ) aims to get an embedding that could benefit the latter differentiable support vector machine . Metric-based Method Instead of learning an embedding that could benefit the latter additional layer , metric-based methods aim to get an embedding that could easily classify the classes by simple metrics . Matching Networks ( Vinyals et al. , 2016 ) conducts a cosine similarity metric with a Full Context Encoding module . Prototypical Networks ( Snell et al. , 2017 ) replaces the cosine similarity metric with the squared Euclidean distance and computes the mean of the embedding in the supportset as the prototype . Relation Network ( Sung et al. , 2018 ) embeds a relation module in the learning metric . Instead of using a consistent metric in the task , TADAM ( Oreshkin et al. , 2018 ) designs a task-dependent metric that could dynamically fit new class combinations . 2.3 MIXED FRAMEWORK . Some recent works have found that using a global pre-trained classifier as the initialization weight could lead to a better meta-learning result . Sun et al . ( 2019 ) used the pre-trained classifier weight as initialization weight and launched a simple gradient-based with restricting the learning process as shift and scale . Meta-Baseline ( Chen et al. , 2020 ) also follows the initialization literature and applies cosine similarity metric for the following learning process . Chen et al . ( 2019 ) changed the original pre-trained network structure into a weight imprinting taste and a simple gradient-based method for the episodic training part . Triantafillou et al . ( 2020 ) also utilized the pre-trained initialization and derived a combination between MAML Finn et al . ( 2017 ) and ProtoNet ( Snell et al. , 2017 ) . 2.4 EPISODE REDUCTION METHODS . Recent researchers have found that a pre-trained classifier leads to better meta-learning results . On the other hand , we could reduce the amount of episodes by using a pre-trained classifier . Besides utilizing the pre-training weight to reduce the number of episodes , Meta Transfer Learning ( Sun et al. , 2019 ) proposes the concept of hard episode . For each normal episode , MTL adds the class with the worst performance in a pool . After collecting for a while , MTL creates hard episodes by sampling from the pool . Instead of crafting hard episodes , our approach tries to utilize more in the pre-training phase . We propose a simple regularization that could reduce the difference between the embeddings of the pretrained classifier and the episodic training one . It has significantly reduced the number of episodes and achieves a similar ( even better ) performance for the original algorithms . Moreover , for shallow and deep backbones , it increases the final accuracy . 3 METHODOLOGY . No matter in the route of transferring the classifier or the route of episodic training , pre-training serves a crucial role . And in meta few-shot learning , pre-training provides an initialization weight for further episodic training . In recent episodic-based methods , the pre-training model is split into two parts , backbone and classifier . The last linear layer serves as the classifier and maps the embedding to logits . Others work as the backbone and transform the raw photo into embedding . After pretraining , the classifier part is directly dropped , since the target classes may be unseen or the order may have changed . Though the split is quite naive , the afterward episodic learning converges faster and better based on the initialization . Thus , previous works conclude that pre-training provides a better representation . However , what makes it better is not clear . What is the role of pre-training in meta few-shot learning ? More specifically , what is the character of backbone in pre-training and episodic training ? In ths section , we choose prototypical network ( Snell et al. , 2017 ) as the representative . Following the analysis of general learning literature by Frosst et al . ( 2019 ) , we utilize the similar loss to measure the disentanglement and entanglement property of the backbone in pre-training and episodic training . Benefited from our observation , we give an attempt to transfer the computation burden of episodic training by adding an sophisticated loss in pre-training .

Summary: This paper investigates the role of pre-training as an initialization for meta-learning for few-shot classification. In particular, they look at the extent to which the pre-trained representations are disentangled with respect to the class labels. They hypothesize that this disentanglement property of those representations is responsible for their utility as the starting point for meta-learning. Motivated by this, they design a regularizer to be used during the pre-training phase to encourage this disentanglement to be even more prominent with the hope that this pre-trained solution is now closer to the optimal one, thus requiring less additional episodic training which is time-consuming. They show experimentally that their modified pre-training phase sometimes leads to better results as an initialization for Prototypical Networks compared to the standard pre-trained solution, and sometimes converges faster.