Meta-RCNN: Meta Learning for Few-Shot Object Detection

1 INTRODUCTION . Object detection is the task of identifying various objects in a given image , and localizing them with a bounding box . It is a widely studied problem in computer vision , and following the success deep convolutional neural networks ( DCNN ) in image classification ( Karpathy et al. , 2014 ; Krizhevsky et al. , 2012 ) , recent years have witnessed remarkable progress made in object detection based on deep learning . A series of detection algorithms based on DCNNs have been proposed which achieve state-of-the-art results on public detection benchmark datasets ( Gidaris & Komodakis , 2015 ; Girshick et al. , 2014 ; Ren et al. , 2015 ; Lin et al. , 2017a ; b ; Liu et al. , 2016 ; Redmon & Farhadi , 2016 ) . However , all these methods are data hungry , and require large amounts of annotated data to learn an immense number of parameters . For object detection , annotating the data is every expensive ( much more than image classification ) , as it requires not only identifying the categorical labels for every object in the image , but also providing accurate localization information through bounding box coordinates . Moreover , in some applications , such as medical research , it ’ s often impossible to even collect sufficient data to annotate . This warrants a need for effective detectors that can generalize well from small amounts of annotated data . We refer to the problem of learning detectors from limited labeled data as few-shot detection . For example , in one-shot detection , only one image is available with objects of interest annotated , and a detector needs to train on just this image and generalize . When presented with such small amounts of annotated data , traditional detectors tend to suffer from overfitting . Inspired by the fact that humans can learn a new concepts from little annotated data , we aim to develop a new few-shot detection algorithm . There have been several efforts exploring few-shot learning ( Vinyals et al. , 2016 ; Finn et al. , 2017 ; Snell et al. , 2017 ) . Many of them follow the principle of meta learning . In meta learning , a set of tasks in a few-shot setting is simulated from a large corpus of annotated data , and the model is optimized to perform well over these few shot tasks . This trains the model to learn how to solve few-shot tasks . However , most existing efforts of meta learning are mainly focused on classification . Adapting few-shot classification algorithms directly for few-shot detection ( e.g . by replacing the region classification branch of detector with a meta-learner ) is non-trivial because of two major concerns : i ) . Detection algorithms not only require classifying objects but also need to correctly localize objects in cluttered backgrounds by using a Region Proposal Network ( RPN ) and bounding box ( bbox ) regressors . It is thus also desirable that both RPN and bbox regressors should also be capable enough to adapt to few-shot settings . ii ) . For a given task with one ( or few ) annotated image ( s ) , the annotated image may contain objects from several classes . But only a few objects of interest are annotated . The goal of the few-shot detector is to detect only these objects of interest . Unfortunately , a naively trained meta-detector ’ s RPN would detect all objects ( even objects from classes not of interest ) and try to classify them as one of the classes of interest rather than background images ( See Figure 1 for an example ) . We aim to address these challenges by proposing a novel method for solving few-shot detection using the meta-learning paradigm . We develop Meta-RCNN , an end to end trainable meta object detector . The proposed Meta-RCNN follows the episodic learning paradigm of meta-learning ( Vinyals et al. , 2016 ) , where based on a give meta-train dataset , multiple few-shot tasks are simulated . For a given task , we first construct a class prototype for each of the annotated object categories in the support set . Using these prototypes , a class-specific feature map of the entire image is constructed , i.e. , we obtain a feature map of the entire image for each of the class prototypes . These feature maps are tailored to detect only objects of the class of the prototype , by giving higher attention to appropriate regions in the image containing that object . Finally , all feature maps are merged to produce a combined feature map , followed by an RPN , and then classification and bbox regression layers . Meta-RCNN learns few-shot detector where the whole framework can be trained via meta-learning in an end-to-end manner . In contrast to the naive adaptation of meta-learning for classification into an object detection framework , Meta-RCNN learns the few-shot classifier , the RPN , and the bbox regressor in the meta-learning setting , thus making all three components suitable for handling fewshot scenarios . Moreover , Meta-RCNN learns a class-specific feature map for a given class prototype enabling easier distinction between classes of interest and backgrounds ( where other objects in the image from classes not of interest are considered as backgrounds ) . We demonstrate the effectiveness of Meta-RCNN on two few-shot detection benchmarks : Pascal VOC and animal subset of ImageNet , and show that Meta-RCNN significantly improves the detection result in few shot settings . 2 RELATED WORK . Generic Object Detection . Object detection based on deep learning can be broadly divided into two families : two-stage detectors and one-stage detectors . Two-stage detectors such as RCNN ( Girshick et al. , 2014 ) , Fast RCNN ( Gidaris & Komodakis , 2015 ) and Faster RCNN ( Ren et al. , 2015 ) , first generate a sparse set of proposal candidates , and a fixed-length feature vector is extracted from each of these candidates , followed by a categorical classifier and a bounding box regressor . Twostage detection algorithms have achieved state-of-the-art results on many public benchmarks ( He et al. , 2016 ; Lin et al. , 2017a ) , but are relatively slower than one-stage detectors . One-stage detectors such as SSD ( Liu et al. , 2016 ) , Yolo ( Redmon et al. , 2016 ; Redmon & Farhadi , 2016 ) and RefineDet ( Zhang et al. , 2018 ) directly generate categorical proposals from the feature map and thus avoid cascaded region classifiers . One-stage detectors can achieve real-time inference speed but the detection accuracy is often inferior to two-stage detection algorithms . Both detection families assume access to a large set of annotated data , and are not suitable for scenarios where the model has access to small amounts of annotated training data . In contrast , our proposed Meta-RCNN method addresses detection problem of few-shot setting , and achieves promising results . Meta Learning for few-shot classification . Few-shot learning has been widely explored in image classification , and currently the most promising methods are mainly based on meta learning . Ravi & Larochelle ( 2016 ) optimized the base-model via an LSTM-based meta-learner which simulates traditional SGD optimization method . Finn et al . ( Finn et al. , 2017 ) proposed MAML which learns a good feature initialization which can adapt to a new task in only one gradient step udpate . Based on MAML , Li et al . ( 2017 ) proposed Meta-SGD which learns a set of learnable parameters to control gradient step of different tasks . Learning initialization is potentially a very general idea for few-shot learning however , the training process can be unstable ( Antoniou et al. , 2018 ) especially for complex problems such as detection . Vinyals et al . ( Vinyals et al. , 2016 ; Snell et al. , 2017 ) proposed a matching network which followed a non-parametric principle by learning a differentiable K-Nearest Neighbour model . Ren et al . ( Ren et al. , 2018 ) extended this idea to semi-supervised learning by self-learning from the unlabeled data . Sung et al . ( Sung et al. , 2018 ) proposed a relation network to automatically define the optimal distance metric . These metric-learning based methods are easy to train and effective in addressing few-shot classification . However , directly adapting these techniques for detection is very challenging as just replacing the object classification branch of a detector with a meta-learner is not sufficient , and training the RPN under a meta-learning paradigm is non-trivial . Few-shot Object Detection . Few-shot detection has received considerably less interest from the community . Dong et al . ( Dong et al. , 2018 ) addressed few-shot detection using large scale unlabeled data . Their model is based on a semi-supervised method which extracts knowledge from unlabeled dataset to enrich training dataset by self-paced learning and multi-modal learning . However , their method may be misled by the incorrect predictions from initial model and also requires re-training the model for every new task . Chen et al . ( Chen et al. , 2018 ) propose a Low-shot Transfer Detector ( LSTD ) using regularization to transfer the knowledge from source domain to target domain by minimizing the gap between these two domains . RepMet ( Schwartz et al. , 2019 ) is a few-shot detection algorithm based on meta learning . It replaces the fully connected classification layer of a standard detector with modified prototypical network . However , they suffer from the two limitations of the RPN and bbox regression not being able to handle few-shot settings , and difficulties in distinguishing object classes of interest from background ( including object classes not of interest ) . Our proposed method is also based on meta learning but can be optimized end-to-end and addresses these limitations to do effective few-shot detection . 3 PRELIMINARIES . 3.1 PROBLEM SETTING . In this section we present the formal problem setting of few-shot detection investigated in our paper . Assume we have two datasets L and S , where L is a large scale annotated dataset with Lc categories and S is a dataset with only a few annotated images with Sc categories . There is no category overlap between two datasets : Lc ∩ Sc = φ . Our goal is to learn a robust detector based on the annotated data in L and S to detect unlabeled objects of S. The proposed Meta-RCNN aims to learn a general detection framework which can be quickly adapted to different detection tasks which have only a few labeled samples . We follow the standard training scheme of meta learning , which splits the whole learning stage into two parts : meta-training and meta-testing , and the model is optimized over multiple few-shot tasks simulated from the metatraining data . Specifically , during meta-training , few-shot detection tasks are sampled from L , and each task contains a support set and a query set . For the i-th task , K ways ( or categories ) and N images per category are randomly selected from Lc to build support set : TL , si . Similarly , Q images per category are randomly selected to build query set TL , qi . Support set T L , s i and query set T L , q i construct a complete task extracted from L ( See Figure 1 ) : TLi = { TL , si , T L , q i } ( 1 ) where both the support set and query set are used to train the meta-model . The meta-model optimizes the base-model with respect to the support set and makes predictions on query set . Finally the loss suffered on the query set is used to update the model . In the meta-testing stage , similar to metatraining stage , a set of few-shot tasks are sampled from S : TSi = { TS , si , T S , q i } ( 2 ) where TS , si is support set and T S , q i is query set . The model makes predictions on the query set , and these results are averaged across several few-shot tasks to evaluate the expected performance of the few-shot detector over a variety of novel few-shot detection tasks .

Summary: The paper proposes a method for few-shot object detection (FSOD), a variant of few-shot learning (FSL) where using a support set of few training images for novel categories (usually 1 or 5) not only the correct category labels are predicted on the query images, but also the object instances from the novel categories are localized and their bounding boxes are predicted. The method proposes a network architecture where the sliding window features that enter the RPN are first attenuated using support classes prototypes discovered using (a different?) RPN and found as matching to the few provided box annotations on the support images. The attenuation is by channel wise multiplication of the feature map and concatenation of the resulting feature maps (one per support class). After the RPN, ROI-pooling is applied on the concatenated feature map that is reduced using 1x1 convolution and original feature map (before attenuation) being added to the result. Following this a two FC layer classifier is fine-tuned on the support data to form the final