MDFL: A UNIFIED FRAMEWORK WITH META-DROPOUT FOR FEW-SHOT LEARNING

1 INTRODUCTION . Deep Neural Networks ( DNNs ) have achieved great progress in many computer vision tasks Ren et al . ( 2016 ) ; Dai et al . ( 2016 ) ; Redmon & Farhadi ( 2017 ) ; Lin et al . ( 2017 ) . However , the impressive performance of these models largely relies on a large amount of data as well as expensive human annotation . When the annotated data are scarce , DNNs can not generalize well to testing data especially when the testing data belongs to different classes of the training data . In contrast , humans can learn to recognize or detect a novel object quickly with only a few labeled examples . Due to some object categories naturally have few samples or their annotations are extremely hard to obtain , the generalization ability of conventional neural networks is far from satisfactory . Few-shot learning , therefore , becomes an important research topic to achieve better generalization ability by learning from only a few examples . The mainstream few-shot learning approaches consists of episodic approaches Kang et al . ( 2019 ) ; Yan et al . ( 2019 ) ; Wang et al . ( 2019 ) ; Fan et al . ( 2020 ) and pretrain-finetune based approaches Dhillon et al . ( 2019 ) ; Chen et al . ( 2019 ) ; Wang et al . ( 2020 ) . Episodic meta-learning encapsulates the training samples into an episode Vinyals et al . ( 2016 ) to mimic the procedure of few-shot testing . Pre-train finetune-based methods are composed of the pre-training stage and fine-tuning stage , the former is responsible for obtaining a good initialization point from base classes , and the latter adapts the pre-trained model to a specific task , respectively . In order to transfer knowledge from base data to novel ones , both methods are trained with two stages , where the data-sufficient base classes and the data-scarce novel classes are used separately . However , there is no framework to unify the two very different streams , hindering exploring the common and eccentric problem of few-shot learning . In this paper , we incorporate episodic meta-learning-based which is denoted as episode-based for simplification , and pre-train finetune-based few-shot methods into one unified optimization framework based on the idea of meta-learning . The framework consists of a novel reformulated metatraining stage and a meta-testing stage . In the meta-training stage , our framework considers the common elements of few-shot learning , including meta-knowledge , task-knowledge , meta-loss , task-loss , and the distribution of the data and tasks . In the meta-testing stage , the final model for novel tasks is obtained based on the learned model . As shown in Figure.1 , a deep model can be divided into two components , called task-specific knowledge and transferable knowledge . The former represents the last fully-connected classifier for specific categories , the latter is the well-learned feature representation which is needed to be generalized to novel tasks . Therefore , it is crucial to improve the generalization power of transferable knowledge if we want to apply it from source tasks to novel tasks in few-shot learning . To achieve this goal , we propose a simple yet effective strategy , named meta-dropout . Because our unified framework can integrate two very different streams of few-shot learning and identify which part of a model is the transferable knowledge , the proposed meta-dropout can be easily applied to existing few-shot models . We select several different methods from the above two streams as the baselines to validate the correctness of our framework and the effectiveness of our meta-dropout . By utilizing meta-dropout , our model demonstrates great superiority towards the current excellent methods on few-shot object detection and few-shot image classification tasks . Our contributions can be summarized as threefold : • We utilize the idea of meta-learning to integrate two very different streams of few-shot learning , i.e. , the episodic meta-learning-based and pre-train finetune-based few-shot learning , and form a unified meta-learning framework . • We propose a simple yet effective strategy , named meta-dropout , to improve the generalization power of meta-knowledge in our framework . • Experiments of baselines from different streams evaluate the effectiveness of our approach on the few-shot object detection and image classification datasets , i.e. , Pascal VOC , MS COCO , CUB , and mini-ImageNet . 2 RELATED WORK . 2.1 EPISODE-BASED FEW-SHOT LEARNING . Few-shot learning is an important yet unsolved task in computer vision Hariharan & Girshick ( 2017 ) ; Koch et al . ( 2015 ) ; Vinyals et al . ( 2016 ) ; Tokmakov et al . ( 2019 ) . Nowadays , the meta-learning strategy which is called “ learning-to-learn ” has become an increasingly popular solution . The goal of meta-learning is to obtain task-level meta-knowledge that helps the model quickly generalize across all tasks Andrychowicz et al . ( 2016 ) ; Munkhdalai & Yu ( 2017 ) ; Santoro et al . ( 2016 ) ; Thrun ( 1998 ) . Recent methods for few-shot learning usually extract meta-knowledge from a set of auxiliary tasks via the episode-based strategy Vinyals et al . ( 2016 ) , where each episode containsC classes and K samples of each class , i.e. , C-way K-shot . In few-shot image classification , Vinyals et al . ( 2016 ) proposed Matching Networks to find the most similar class for the target image among a small set of labeled samples . Prototypical Networks ( PN ) Snell et al . ( 2017 ) extended Matching Networks by producing a linear classifier instead of the weighted nearest neighbor for each class . The cosine similarity-based classifier further enhanced the discriminative power of the trained model Chen et al . ( 2019 ) ; Gidaris & Komodakis ( 2018 ) . Relation Network ( RN ) Sung et al . ( 2018 ) used a neural network to learn a distance metric , in which the unlabeled images could be classified according to the relation scores between the target sample and a few labeled images . Graph Neural Network ( GNN ) Kim et al . ( 2019 ) ; Gidaris & Komodakis ( 2019 ) was also utilized to model relationships between different categories . In few-shot object detection , Kang et al . ( 2019 ) applied a feature re-weighting module to a single-stage object detector ( YOLOv2 ) with the support masks as inputs . Yan et al . ( 2019 ) introduced a Predictor-head Remodeling Network ( PRN ) that shared its backbone with Faster/Mask R-CNN . To disentangle the category-agnostic and category-specific components in a CNN model , Wang et al . ( 2019 ) proposed a weight prediction meta-model for predicting the parameters of category-specific components from few samples . Karlinsky et al . ( 2019 ) proposed a new module for distance metric learning ( DML ) that could be used as the classification head combined with the standard object detection model . Fan et al . ( 2020 ) introduced a few-shot object detection method , which consisted of an attention-rpn , a multi-relation detector , and a contrastive training strategy . 2.2 PRE-TRAIN FINETUNE-BASED FEW-SHOT LEARNING . Pre-train finetune-based approaches are basic yet ignored in few-shot learning due to the excellent performance of episode-based methods . However , some simple pre-train finetune-based methods turn out to be more favorable than many episode-based works Chen et al . ( 2019 ) ; Dhillon et al . ( 2019 ) ; Chen et al . ( 2021 ) . Chen et al . ( 2019 ) introduced a pre-train finetune baseline with a distancebased classifier , achieving a competitive performance with state-of-the-art episode-based classification approaches . Chen et al . ( 2021 ) explored a simple process : meta-learning over a whole classification pre-trained model . This simple method achieves competitive performance to state-of-the-art methods on standard benchmarks . By using the proposed regularization , the standard detectors , such as SSD Liu et al . ( 2016 ) and FRCNN Ren et al . ( 2016 ) , were fine-tuned for few-shot problems . Furthermore , Wang et al . ( 2020 ) demonstrated that only fine-tuning the last layer of existing models was crucial to the few-shot object detection task . Such a simple approach outperforms the episodebased approaches by 2 to 20 points and even doubles the accuracy of the prior works on current benchmarks . 3 METHOD . 3.1 THE SOLUTION OF FEW-SHOT PROBLEMS . Common supervised learning problems based on abundant training data can be solved by minizing the Equ ( 1 ) , in which Θ is the trainable parameters of neural networks , and x is input data sampled from p ( D ) . While few-shot tasks are limited in the number of samples , optimizing it directly is likely to results in the overfitting of Θ due to its high-dimensional property . min Θ E x∼p ( D ) L ( Θ ; x ) ( 1 ) Few-shot learning aims to solve learning problems with just a few training examples . To achieve this goal , the common solution is to reduce the learnable dimension of Θ , and thus the Equ ( 1 ) can be re-written as Equ ( 2 ) , in which Θ = [ θ , w ] . w represents the useful foundation for few-shot learning , such as a good initailization point or a well-learned feature representation , which is obtained from source tasks . θ are updated on target tasks based on the learned w. min θ E x∼p ( D ) L ( θ ; x|w ) ( 2 ) Given a labeled source dataset Dsource , there are Csource source classes with a large number of images in each class . Novel dataset Dtarget with novel classes Ctarget consists of a few samples in each class . Csource and Ctarget do not have overlapping categories . The learning goal is adapting the model from Dsource to Dtarget . Dsource and Dtarget are used to optimize w and θ respectively . The C-way K-shot few-shot setting that is used for evaluating the performance of a few-shot model means Dtarget has C novel categories and each novel category has K images . Existing few-shot learning methods consist of two different streams of categories , i.e. , the episodic meta-learning-based and pre-train finetune-based methods . However , both streams can be integrated by the above formulation as shown in Figure.2 . w in pre-train finetune-based methods , like TFA Wang et al . ( 2020 ) , is its backbone which is transferred to the second stage for initialization . θ in TFA means the last fully-connected layer that is optimized for novel tasks based on the frozen w. w in Meta R-CNN Yan et al . ( 2019 ) , which is a representative episodic meta-learning-based method , provides a backbone with good generalization ability . θ in Meta R-CNN is the trainable backbone and the last fully-connected regression and classification layers for novel tasks . 3.2 THE UNIFIED META-LEARNING FRAMEWORK . In order to explain the learning process of existing few-shot methods , we propose a unified metalearning framework to re-formulate Equ ( 2 ) as Equ ( 3 ) , in which the learning goal is to obtain a general transferable knowledge via optimizing the expectation of many source tasks and source datasets . Suppose p ( T ) presents the distribution of source tasks , in which T consists of infinite basic tasks , and p ( Dt ) is the distribution of the training dataset Dt of task t , in which the number of the training data is unlimited . w is used to specify meta-knowledge , which is the transferable parameters among different tasks . θ is task-specific knowledge , called task-knowledge . L is the loss function . The overall optimization goal is below : min w E t∼p ( T ) E x∼p ( Dt ) L ( w ; θt ; x ) ( 3 ) The goal is to let w get good performance for each task T= { D , L } . Our meta-learning framework consists of a meta-training phase and a meta-testing phase .

Summary: The paper presents a new framework to unify two streams of few-shot learning methods, the episodicc meta-learning and pre-train finetune-based few-shot learning. Besides, a new meta-dropout strategy is proposed to improve the generalization ability. Extensive results on several benchmark like PascalVOC, COCO, CUB and mini-ImageNet validate the proposed approach.