Meta-RCNN: Meta Learning for Few-Shot Object Detection
Authors: Xiongwei Wu
Doyen Sahoo
Steven C. H. Hoi
Venue:  (2020)
Keywords: Few-shot detection
Meta-Learning
Object Detection

Abstract: Despite significant advances in object detection in recent years, training effective detectors in a small data regime remains an open challenge. Labelling training data for object detection is extremely expensive, and there is a need to develop techniques that can generalize well from small amounts of labelled data. We investigate this problem of few-shot object detection, where a detector has access to only limited amounts of annotated data. Based on the recently evolving meta-learning principle, we propose a novel meta-learning framework for object detection named ``Meta-RCNN", which learns the ability to perform few-shot detection via meta-learning. Specifically, Meta-RCNN learns an object detector in an episodic learning paradigm on the (meta) training data. This learning scheme helps acquire a prior which enables Meta-RCNN to do few-shot detection on novel tasks. Built on top of the Faster RCNN model, in Meta-RCNN, both the Region Proposal Network (RPN) and the object classification branch are meta-learned. The meta-trained RPN learns to provide class-specific proposals, while the object classifier learns to do few-shot classification. The novel loss objectives and learning strategy of Meta-RCNN can be trained in an end-to-end manner. We demonstrate the effectiveness of Meta-RCNN in addressing few-shot detection on Pascal VOC dataset and achieve promising results. 

Under review as a conference paper at ICLR 2020
META-RCNN: META LEARNING FOR FEW-SHOT OBJECT DETECTION
Anonymous authors Paper under double-blind review
ABSTRACT
Despite significant advances in object detection recent years, training effective detectors in a small data regime remains an open challenge. Labelling training data for object detection is extremely expensive, and there is a need develop techniques that can generalize well from small amounts of labelled data. We investigate this problem of few-shot object detection, where a detector has ac- cess to only limited amounts of annotated data. Based on the recently evolving meta-learning principle, we propose a novel meta-learning framework for object detection named MMeta-RCNN,, which learns the ability perform few-shot de- tection via meta-learning. Specifically, Meta-RCNN learns an object detector episodic learning paradigm the (meta) training data. This learning scheme helps acquire a prior which enables Meta-RCNN to do few-shot detection on novel tasks. Built top of the Faster RCNN model, in Meta-RCNN, both the Region Proposal Network (RPN) and the object classification branch are meta-leamed. The meta-trained RPN learns to provide speci-s proposals, while the ob- ject classifier learns to do few-shot classification. The novel loss objectives and learning strategy of Meta-RCNN can be trained in an end-to-end manner. We demonstrate the effectiveness of Meta-RCNN in addressing few-shot detection on Pascal VOC dataset and achieve promising results.
INTRODUCTION
Object detection is the task of identifying various objects in a given image, and localizing them with a bounding box. It is a widely studied problem in computer vision, and following the success deep convolutional neural networks (DCNN) in image classification (Karpathy et al., 2014 Krizhevsky et al. 2012). recent years have witnessed remarkable progress made in object detection based on deep learning. A series of detection algorithms based on DCNNs have been proposed which achieve state-of-the-art results public detection benchmark datasets GGidaris & Komodakis 2015 Gir- shick et al. 2014 Ren et al., 2015 Lin et al. 2017a|b) b. Liu et al.) 2016, Redmon
Farhadi (016)) However, all these methods data hungry, and require large amounts of annotated data to learn an immense number of parameters. For object detection, annotating the data is every expensive (much more than image classification), it requires not only identifying the categorical labels for every object in the image, but also providing accurate localization information through bounding box coordinates. Moreover, in some applications, such as medical research, it's often impossible to even collect sufficient data to annotate. This warrants a need for effective detectors that can gen- eralize well from small amounts of annotated data. We refer the problem of learning detectors from limited labeled data as few-shot detection. For example, in one-shot detection, only one image available with objects of interest annotated, and detector needs to train on just this image and generalize. When presented with such small amounts of annotated data, traditional detectors tend to suffer from overfitting. Inspired by the fact that humans can learn new concepts from little annotated data, aim to develop new few-shot detection algorithm.
There have been several efforts exploring few-shot learning (Vinyals et al. 2016 Finn et al. 2017 Snell et al. 2017). Many of them follow the principle of meta learning. In meta learning, a set of tasks in a few-shot setting is simulated from a large corpus of annotated data, and the model is optimized perform well over these few shot tasks. This trains the model learn how solve few-shot tasks. However, most existing efforts of meta learning are mainly focused on classification. Adapting few-shot classification algorithms directly for few-shot detection (e.g. by replacing the

---

Under review as a conference paper at ICLR 2020
Sample
Meta-Train
Dataset
Support set of 1-shot Task
Query set of 1-shot Task
Sample
Update
Meta- Test
Dataset
Support set of ofl-shot Task
Results of query image
Figure 1: Few-Shot object detection in the meta-learning setting. From the meta-train dataset, a K wayt-Nshot support set and a query set sampled to create task. The meta detector makes predictions on the query set by using the knowledge from the support set, and updates the detector based on the loss the query set. In the example above, there are many objects (person, dog and truck), but it is annotated with the goal of detecting only person. At test time, single annotated image from a novel class (bear) is available for the detector to learn a model that can generalize.
region classification branch of detector with meta-learner) is non-trivial because of two major concerns: i). Detection algorithms not only require classifying objects but also need to correctly localize objects in cluttered backgrounds by using a Region Proposal Network (RPN) and bounding box (bbox) regressors. It is thus also desirable that both RPN and bbox regressors should also be capable enough adapt to few-shot settings. ii). For given task with one (or few) annotated image(s), the annotated image may contain objects from several classes. But only few objects of interest are annotated. The goal of the few-shot detector is to detect only these objects of interest. Unfortunately, a naively trained meta meta-detector's 's RPN would detect all objects (even objects from classes not of interest) and try to classify them as one the classes of interest rather than background images (See Figure for example).
We aim to address these challenges by proposing a novel method for solving few-shot detection using the meta- meta-learnngg paradigm. We develop Meta-RCNN, end end trainable meta object de- tector. The proposed Mcta-RCNN follows the episodic learning paradigm of meta-learning (Vinyals) et al. 2016), where based a give meta-train dataset, multiple few-shot tasks are simulated. For a given task, we first construct a class prototype for each the annotated object categories in the support set. Using these prototypes, class-speeific feature map of the entire image is constructed, ie., we obtain a feature map of the entire image for each of the class prototypes. These feature maps tailored detect only objects of the class of the prototype, by giving higher attention to appro- priate regions in the image containing that object. Finally, all feature maps are merged to produce combined feature map, followed by RPN, and then classification and bbox regression layers.
Meta- Meta-RCNN learns few-shot detector where the whole framework can be trained via meta-learning an end-to-end manner. In contrast to the naive adaptation of meta-learning for classification into an object detection framework, Meta-RCNN learns the few-shot classifier, the RPN, and the bbox regressor in the meta-learning setting, thus making all three components suitable for handling few- shot scenarios. Moreover, Meta-RCNN learns class-specific feature map for given class prototype enabling easier distinction between classes of interest and backgrounds (where other objects in the image from classes not of interest are considered backgrounds). We demonstrate the effectiveness of Meta-RCNN RCNN on two few-shot detection benchmarks: Pascal VOC and animal subset of ImageNet, and show that Meta-RCNN significantly improves the detection result in few shot settings.
2

---

Under review as a conference paper at ICLR 2020
2 RELATED WORK
Generic Object Detection. Object detection based on deep learning can be broadly divided into Iwo families: two-stage detectors and one-stage detectors. Two-stage detectors such as RCNN (Gir- Shick et al. 2014). Fast RCNN (Gidaris & Komodakis, 2013) and Faster RCNN (Ren et al.) 2015). first generate a sparse set of proposal candidates, and a fixed-length feature vector is extracted from each of these candidates, followed by categorical classifier and a bounding box regressor. Two- stage detection algorithms have achieved state-of-the-art results on many public benchmarks (He) et al. 2016 Lin et al.] (017a)). but are relatively slower than one-stage detectors. One-stage de- tectors such as SSD (Liu et al. 2016). Yolo (Redmon et al., 2016 Redmon & Farhadi (016) and RefineDet (Zhang al. 2018) directly generate categorical proposals from the feature map and thus avoid cascaded region classifiers. One-stage detectors can achieve real-time inference speed but the detection accuracy is often inferior two-stage detection algorithms. Both detection families sume access to a large set of annotated data, and are not suitable for scenarios where the model has access to small amounts of annotated training data. In contrast, our proposed Meta-RCNN method addresses detection problem of few-shot setting, and achieves promising results.
Meta Learning for few-shot classification. Few-shot learning has been widely explored in image classification, and currently the most promising methods are mainly based on meta learning. Ravi & Larochelle| (2016) optimized the mase- via an LSTM-based meta-learner which simulates traditional SGD optimization method. Finn et al. (Finn et al. 2017) proposed MAMI. which learns good feature initialization which can adapt to a new task in only one gradient step udpate. Based on MAML. Li et al. (2017) proposed Meta-SGD which learns a set of learnable parameters to control gradient step of different tasks. Learning initialization is potentially a very general idea for few-shot learning however, the training process can be unstable (Antoniou et al. 20188) especially for complex problems such as detection. Vinyals et al. (Vinyals et al.] 2016; Snell et al. 2017) proposed a matching network which followed non conaaammetiic principle by learning differentiable K-Nearest Neighbour model. Ren et al. (Ren et al. 2018) extended this idea to semi-supervised learning by self-learning from the unlabeled data. Sung al. (Sung et al. (018)) proposed relation network to automatically define the optimal distance metric. These metric-learning based methods are easy to train and effective in addressing few-shot classification. However, directly adapting these techniques for detection is very challenging as just replacing the object classification branch of a detector with a meta-learner is not sufficient, and training the RPN under a meta- meta-learning paradigm is non- con-trivial.
Few-shot Object Detection. Few-shot detection has received considerably less interest from the community. Dong et al. (Dong al.] 2018) addressed few-shot detection using large scale unlabeled data. Their model is based on semi-supervised method which extracts knowledge from unlabeled dataset to enrich training dataset by self- self-paced learning and multi-modal learning. However, their method may be misled by the incorrect predictions from initial model and also requires re-training the model for every new task. Chen et al. (Chen et al. 2018) propose a Low-shot Transfer De- tector (LSTD) using regularization to transfer the knowledge from source domain to target domain by minimizing the gap between these two domains. RepMet (Schwartz et al., 2019) is few-shot detection algorithm based on meta learning. It replaces the fully connected classification layer of a standard detector with modified prototypical network. However, they suffer from the two limita- tions of the RPN and bbox regression not being able handle few-shot settings, and difficulties in distinguishing object classes of interest from background (including object classes not of interest). Our proposed method is also based on meta learning but can be optimized end-to-end and addresses these limitations to do effective few-shot detection.
3 PRELIMINARIES
3.1 PROBLEM SETTING
In this section we present the formal problem setting of few-shot detection investigated in our paper. Assume we have two datasets L and S, where L is a large scale annotated dataset with L. categories and S is dataset with only a few annotated images with S. categories. There is category overlap between two datasets: L, n S, = o. Our goal is to learn a robust detector based on the annotated data in To and S to detect unlabeled objects of S.
3

---

Under review as a conference paper at ICLR 2020
The proposed Meta-RCNN aims to learn general detection framework which can be quickly adapted to different detection tasks which have only a few labeled samples. We follow the standard training scheme of meta learning, which splits the whole learning stage into two parts: meta-training and meta-testing, and the model is optimized over multiple few-shot tasks simulated from the meta- training data. Specifically, during meta-training, few-shot detection tasks are sampled from L, and each task contains a support set and query set. For the i-th task, K ways (or categories) and N im- ages per category are randomly selected from Le to build support set: T Similarly, Q images per category are randomly selected to build query set T...". Support set Th-- and query set T/ construct a complete task extracted from L (See Figure TD:
T
(T.. T
where both the support set and query the badel with respect to the support set and makes predictions on query set. Finally the loss suffered the query set is used to update the model. In the meta-testing stage, similar to meta- training stage, a set of few-shot tasks are sampled from S:
used to train the meta-model. The meta-model optimizes
T
{ TT. T.
where T is support set and T is query set. The model makes predictions the query set, and these results are averaged across several few-shot tasks to evaluate the expected performance of the few-shot detector over variety of novel few-shot detection tasks.
3.2 OVERVIEW OF FASTER RCNN
Meta-RCNN is based on two-stage region based object detection algorithms. In this paper, we use the state-of-the-art detection algorithm Faster RCNN (Ren et al. 2015) as our base model, which is widely used in the computer vision community. Faster RCNN consists of two components, an RPN (Region Proposal Network) for proposal generation and Fast RCNN for region classification. RPN generates sparse set of proposals which are classified into different categories by the region classifiers. Specifically, RPN extracts a feature vector from each region by scanning the whole image using sliding windows. This is followed by a binary classifier (objects vs backgrounds) and a bounding box regressor, where easy negatives are filtered. For each proposal, a fixed-length feature vector is extracted by using ROI Pooling layers. This vector is then fed into a sequence of dense connected layers branching into two outputs. One output is responsible for representing softmax probability over K + classes( target classes and one background class), and the other one encodes four values for refining bounding box position. We denoted u and v as the category and bounding box label respectively, p as the predicted probability distribution over C classes, and l, as the predicted bounding box prediction of class and À as the trade-off parameter. Lets represents softmax loss and Toe represents Smoothl.1 loss function. The entire network can be optimized in an end-to-end manner by minimizing loss L(p, u. t" v):
[pp, u, tt.""" .u) = (ps u) + Xiu 2 (tL, u),
(3)
However, two-stage detectors require a lot of training samples obtain a good performance. In the next section, we present the proposed Meta-RCNN which builds over Faster RCNN and is specifi- cally designed to address few-shot detection.
META-RCNN
4.1 OVERVIEW
We now present our proposed method Meta-RCNN for few-shot detection (See Figure 2 for an overview). Meta-RCNN is trained with multiple few-shot tasks simulated from the meta-train dataset. For each episode, few object categories of interest are assumed to be annotated (Support set). During meta-training, a prototype is computed for each object category. For each of these cat- egory prototypes, class-specific feature map is generated by using class-attention module which combines the prototype information with the feature map of the entire image. This feature map only highlights the signals of the class of interest, and suppresses information from other classes. Finally,
4

---

Under review as a conference paper at ICLR 2020
prototypes
RPN
Objects DCNN
d5
ROI
Support Set
FC
DCNN
Attention
Query image: truck, person, dog
Figure 2: The Meta-RCNN workflow. set of prototypes of different categories are extracted from the support set. For each class, conditioned on these prototypes, a class-specific feature map from query set is generated by applying the class attention module to the feature map of the entire image. The new class-specti specific feature maps are concatenated together and finally, an RPN is applied followed with region classification layer and bounding box regressors. The whole network is optimized via meta learning and can be trained end-to-end,
feature map is tailored to detecting objects of that specific class. The class-
feature maps of all target categories combined, followed by RPN and RCNN branches to make predictions on the query set. Based on the loss on the query set, the model is updated.
Meta-RCNN is general paradigm train few-shot detector via by meta-learning. For each task, irrelevant categories and background can be filtered by attention module, and the final generated feature map learns general representation for the given few-shot detection task. Compared to other variants (Schwartz et al.) 2019) which directly replaces the FC classification branch with a meta- learning branch, MietaRRCNNFa more general and the whole framework can be optimized including RPN and bbox regressors, making all the components few-shot capable. Next, we present the details of the model.
4.2 TRAINING
During Meta-Training multiple Kway-Nshoo tasks are simulated from the annotated dataset To fit memory size, in Pett-Trainnin stage we train the model using 5way-Ishot tasks, and only query images (I query image per class). This results in a total of 10 images for one task. With this, implementing the meta-training not too difficult. For each task THE images of support set TH are fed into Faster RCNN to generate region features. For each of the object categories of interest. (those assumed to be annotated the support image), a prototype P. generated based on the corresponding region features:
N N
1
P
(4)
where P. denotes prototype of class C, and denotes i-th region features of all annotated objects from class c. Based on these generated prototypes, images of query set T....9 are fed into the same Faster RCNN model and we obtain the image feature map before RPN and ROI Pooling. For each category, class-specific feature map learned based on the input query image and its corresponding prototype. We use learnable class attention module here to highlight the signals of target class and suppress signals of other categories. The class attention module based on basic channel-win multiplication. The prototype P. is encoded by FC layer ó, which is later combined with feature map f by element-wise multiplication:
F =
(5)
5

---

Under review as a conference paper at ICLR 2020
For each category c, feature map F. is generated which aims highlight the objects of class c. Next, all these new feature maps are combined into one feature map F:
F concate[Fi, P2...., Pk)
(6)
F learns a general representation of K-classes, where each sub-channel contains information of different classes of interest. Based the new feature map F, 1xl conv layer is used to reduce computation cost, followed by RPN to produce region proposals. order to recover the information lost in attention module, we finally combine the new generated feature map with original feature map by element-wise summation, and crop region features based on the new generated map. Finally, a K+1 region classifier and a bbox regressors are optimized w.r.t the label info from query set T::
ITT T! 0) = LAve + Las + LRPN
where f represents the parameters Meta-RCNN.
4.3 META-TESTING
During meta-testing, we sample few-shot detection tasks from The annotations of support set are available and we make predictions on the query set to evaluate the performance of Meta-RCNN, For each task T,$, prototypes class-specific feature maps of images from query set T.5/4 In this stage, we need to finetune the model based on the labeled images of support set. The finetuning operation addresses the learning limitation of non-parametric method when more labeled images are provided. Finally, we evaluate the output from the query set as traditional detection problem:
generated from support set T: which are later used to generate new
p, u MetaRCNN(T.50TT"*
where p class probability vector and u location set of bounding boxes.
5 EXPERIMENTS
5.1
DATASETS AND IMPLEMENTATION DETAILS
DATASET
Train
#lmg
#cls
Test
#Img
#cls
VOC-FSOD
VOC2007trainval
4.9k
10
VOC2007test
2.2k
10
IMAGENET-FOO
ImageNet-LOC
~ 53k
100
ImageNet-LOCC
117k
214
Table 1: Two few-shot object detection benchmark testbeds for performance evaluation
We construct two benchmark testbeds to facilitate the performance evalua- tion for few-shot object detection in meta-learning settings. The first is on Pascal VOC2007, and the second is the animal subset of ImageNet-L.OC: dataset. Table details of these datasets. Pascal VOC2007 has 20 categories with 5k images trainval set and 5k images in test set. subset of 10 categories are randomly from selected for VOC2007 trainval set for Meta- Meea-Training and the remaining 10-category subset of VOC2007 test set is used for Meta-Testing. Images without tar- get object categories are removed. For ImageNet-FSOD benchmark, we use the subset of first 100 animal classes of ImageNet in Meta- Ceta-Training stage and the subset of remaining 2 animal species IMC in MeeaaTesting stage. The model used in VOC-FSOD benchmark is pre-traue trained on ImageNet, while in ImageNet-FSOD benchmark, the model is pre-trained on MSCOCO dataset with 1 11k images 80 categories.
Benchmark Datasets:
Task Generation:
For each benchmark, Meta-RCNN is evaluated on multiple tasks with different K way-N shot few-shot settings (N annotated images per category). For VOC-FSOD benchmark, we have 3 few-shot settings to evaluate Meta-RCNN: 5way-lshot, 5way-3shot and 5way-5shot. In detection, single image has more than one object, and proposal generation will automatically increase the number of training samples, the real number of training samples is about times larger than N. On ImageNet-FSOD benchmark, we mainly follow (Chen et al. 2018) and (Schwartz) al. 2019) with settings: 50way-1 shoo and 50way-5shot.

---

Under review as a conference paper at ICLR 2020
Meta-model Parameter Setting: In Meta- Training stage, totally 1000 distinct tasks and 5000 tasks are generated in VOC-FSOD benchmark and ImageNet-FSOD benchmark respectively. There are 10 images per class in query set to update the model weights for 10 epochs. The initial learning rate is set to le-3 and is reduced to le-4 every 600 tasks and 3500 tasks in VOC-FSOD benchmark and ImageNet-FSOOD benchmark. We set the batch size as 5 during query update.
Basic Detection Parameter Setting: The parameter settings in Meta-RCNN is identical to vanilla Faster RCNN. Proposal overlap with objects larger than 0.5 are considered positive and less than 0.3 are negative. During Meta-Training the top 128 confident proposals are selected for training and during evaluation, 300 proposals with largest confidence score are selected. we build our Meta- RCNN based on Faster RCNN with VGG16 (Simonyan & Zisserman, 2014) and ResNet50 (He ct al. 2016) model which is pretrained on ImageNet.
Model Evaluation: We evaluate Meta-RCNN based multiple tasks of few-shot settings, which follows the evaluation metric of standard meta learning. More specifically, in evaluation stage, 200 K Kshot-N shot tasks are sampled from dataset S and images in query set will be evaluated. Mean average (r)))))onmmAP) over selected K categories is used as evaluation metric.
5.2 RESULTS ON VOC-FSOD BENCHMARK
We validate the effectiveness of Meta-RCNN VOC-FSOD benchmark where subset of 10 VOC categories are selected for Meta- Veta-Trainnng and the other ten categories are used for Meta- Testing. For fair comparison, these two subsets are split as similar as possible. For example, we keep animal categories on both sides since they share similar semantic information (see appendix for details). Here we set up three baselines on VOC-FSOD benchmark to compete with proposed Meta-RCNN.
vanilla FRCN (Ren et al., 2015): the vanilla Faster RCNN which is the most popular object detection algorithm with competitive performance on many benchmarks. The vanilla FRCN is not designed for few-shot detection problem, but we try to include this baseline by fine-tuning the detector on the few-shot training data.
LSTD (Chen et al. 2018) is few-shot detection algorithm based on Faster RCNN, LSTD uses categorical regularization items which transfers knowledge of L dataset to S dataset.
FRCN-PN is modified version of Faster RCNN using meta-learning, which replaces final FC classification layer with non- parametrii prototypical network (PN), sharing the same principle of RepMet (Schwartz et al. 2019).
All three baselines as well as the proposed Mcta-RCNN are based on VGG16 (Simonyan Zisser- man (014) backbone. For Regular FRCN and LSTD, we first train global Faster RCNN during Meta-Training. Then the pretrained detector models are adapted to different tasks during Meta- Testing. During Meta-Testing, Meta-RCNN and vanilla FRCN are finetuned for 4 epochs while LSTD requires longer finetuning period (10 epochs). For FRCN-PN, prototypes of different cate- gories are extracted Mcta-RCNN, and metric distances are learned to assign correct labels to each proposal. We report the results on Table based on three different settings.
From Table 2 the performances of all four methods improve with training shot increasing. No- tably, FRCN-PN obtains much less improvement because the non-maamcttiic property of PN layer limits its learning capacity from increased training samples. Benefit from the finetuning operation as well as FC layer in final classification and regression, Meta-RCNN can still maintain consistent improvement when trained with more samples. Furthermore, it's interesting that Regular FRCN outperforms FRCN-PN even in very few-shot cases (Sway-Ishot), where non-parametric property does not help PN obtain better performance. We argue this because few-shot detection problem is more difficult than few-shot classification problem, as we discussed in introduction section. FRCN- PN cannot learn a representative prototype of background classes and the whole framework cannot be optimized by meta learning style (e.g., RPN and bbox regressors). The failure of FRCN-PN in- dicates naively attach components from few-shot classification framework cannot address few-shot detection problem. Finally, our Meta-RCNN achieves better results than all three baselines.
Performance of RPN: Here, we present the performance of RPN to validate our concerns of the negative impact of irrelevant categories. We use regular FRCN and FRCN-PN as our baseline. The models are optimized in the same manner as before but during SestaTesstin, we evaluate the recall on each task instead of mAP. From Table Regular FRCN baseline outperform the FRCN- PN significantly. This is because objects of irrelevant categories in the same image hurt the training process of RPN. And our proposed Meta-RCNN outperforms these two baseline significantly. Meta-
7

---

Under review as a conference paper at ICLR 2020
Method
5way-1shot
5way-3shot
5way-5shot
vanilla FRCN Ren al. 2015)
14.78% = 1.02%
20.34% = 1.26%
26.89%
1.23%
LSTD (Chen et al. 2018
17.66% = 1.65%
22.37% = 0.81%
29.00%
1.28%
FRCN-PN
12.71% % + 0.70%
13.91% + 0.70%
14.33%
0.61%
Meta-RCNN (ours)
19.22% % + 1.01%
24.45% % + 1.20%
31.11% + 0.88%
Table 2: mAP Performance Evaluation on the VOC-FSOD BENCHMARK
RCNN learns a general feature map for all Kway-Nshoo detection problem and optimize RPN by meta learning scheme, which proves more effective in few-shot settings. Notably, the results surprising since the recall of RPN in few-shot scenario is significantly lower (> 90% with enough training data on VOC dataset).
Model
Backbone
Sway-1shor
5way-3shot
5way-5shot
vanilla FRCN (Ren et al. 2015)
VGG16
24.9%
26.5%
28.4%
FRCN-PN
VGG16
24.7%
24.9%
26.1%
Meta-RCNN (ours)
VGG16
26.1%
27.9%
33.7%
Table 3: Recall evaluation of Meta-RCNN on VOC-FSOD BENCHMARK test set.
5.3 RESULTS ON IMAGENET-FSON BENCHMARK
On SongeNet--FSOD benchmark, adapt weights of detector pretrained on MSCOCO trainval set, and then optimize Meta-RCNN based on this starting point. The Meta-RCNN is evaluated on animal subset of ImageNet-LOO. Animal subset of ImageNet-LLOC only contains single animal category per image, so there are no irrelevant classes during training and it's simpler than the situation we dis- cussed. In addition to FRCN and LSTD, we also include another latest baseline RepMet (Schwartz) et al. 2019), which replaces FC classification layers in FRCN with more careful design of PN layers, as well as much more stronger backbone architecture (DCN (Dai et al.) 2017) and FPN (Lin al.] 2017a)). In this benchmark, we have 50 categories per task, so we attach 1x1 convolution layer before class-specific feature map generation reduce computation cost. We report the results in Tab. In 50way-1shor and 50way-5shot, Meta-RCNN is better than other methods.
Model
Backbone
50way-Ishot
50way-5shot
vanilla FRCN Ren et al.
2015)
VGG16
16.5%
34.3%
LSTD Chen et al. 2018
VGG16
19.2%
37.4%
RepMet Schwwrtz et al. 2019)
DCN+FPN
24.1%
39.6%
Meta-RCNN (ours)
VGG16
24.6%
40.1% 1%
Meta-RCNN (ours)
ResNet50
25.1%
40.3%
Table 4: mAP performance evaluation on IMAGENET-FSOD BENCHMARK.
5.4 DISCUSSIONS
Extension to other Meta-Learning Methods: Beyond prototypical networks, other meta-learning methods such as MAML (Finn et al. 2017) in principle also be applied, e.g., can apply MAML for vanilla FRCN framework, which updates the base model with the average gradient step of multiple tasks. However, in experiments, the training process of MAML unstable. This may be because few-shot detection is generally more difficult than few-shot classification, due multiple dependent loss objectives (FRCN relies RPN and regression loss etc.) and more complicated noisy contexts. In future, we plan to explore extensions to other meta-learnig methods.
CONCLUSION
Object detection has been widely explored but little attention has been given to learning detectors under few-shot regime. In this paper propose meta learning based detection algorithm Meta- RCNN, which robust to few-shot learning, and the proposed training strategies make more suit- able in detection scenario. Specifically it adapts the Faster RCNN method and enables meta-learning of the object classifier, the RPN and the bounding box regressor. The RPN is meta meta-traine trained through a novel class- attention module. We conduct several experiments and obtain promising results.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Antreas Antoniou, Harrison Edwards, and Amos Storkey. How to train your maml. arXiv preprint arXiv:1810.09500. 2018.
Hao Chen, Yali Wang, Guoyou Wang, and Yu Qiao. Lstd: A low-shot transfer detector for object detection. In AAAI, 2018.
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In ICCV, 2017.
Xuanyi Dong, Liang Zheng, Fan Ma, Yi Yang, and Deyu Meng. Few-example object detection with model communication. In TPAMI, 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, ICMI. 2017.
Spyros Gidaris and Nikos Komodakis. Object detection via multi-region and semantic segmentation-aware enn model. In ICCV, 2015.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accu- rate object detection and semantic segmentation. In CVPR, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei- Fei. Large-scale video classification with convolutional neural networks. In CVPR, 2014.
Alex Krizhevsky, Ila Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In NeurlPS, 2012.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few- shot learning. arXiv preprint arXiv: 1707.09857 2017.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017a.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017b.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander C Berg. SSD: Single shot multibox detector. In 2016.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. ICLR, 2016.
Joseph Redmon and Ali Farhadi. Yolo9000: Better, faster, stronger. In arXiv preprint arXiv:1612.08242,, 2016.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In CVPR, 2016.
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua Tenenbaum, Hugo Larochelle, and Richard S Zemel. Meta-learning for semi-supervised few-shot classifica- tion. arXiv preprint 32t3:/80030067. 2018.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurIPS, 2015.
Eli Schwartz, Leonid Karlinsky, Joseph Shtok, Sivan Harary, Mattias Marder, Sharathchandra Pankanti, Rogerio Feris, Abhishek Kumar, Raja Giries, and Alex M Bronstein. Repmet: Representative-basee metric learning for classification and one-shot object detection. In CVPR, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In arxiv preprint arXiv: 2014.

---

Under review as a conference paper at ICLR 2020
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, 2017.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In The IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR). 2018.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems, 2016.
Shifeng Zhang, Longyin Wen, Xiao Bian, Zhen Lei, and Stan Z Li. Single-shot refinement neural network for object detection. In CVPR, 2018.
A APPENDIX
A.1
CATEGORY SPLIT IN VOC-FSOD BENCHMARK AND 2
Here we describe the category splits (Le and S.) of VOC-FSOD benchmark and ImageNet-FSOD benchmark. These splits used in all our paper.
VOC-ESOD benchmark:
L
aeroplane, bicycle, 'bird, car, cat, chair, cow, person, pottedplant, tvmonitor
S.
bus, motorbike, train, dog, sheep, bottle, sofa, diningtable, horse, boat
ImageNet-FSSD benchmark:
L.
kit fox, English setter, Siberian husky, Australian terrier, English springer, grey whale, lesser panda, Egyptian cat, ibex, Persian cat, cougar, gazelle. porcupine, sea lion, malamute, badger, Great Dane, Walker hound, Welsh springer spaniel, whippet, Scottish deerhound, killer whale, mink, African ele- phant, Weimaraner, soft-coated wheaten terrier, Dandie Dinmont, red wolf, Old English sheepdog, jaguar, otterhound, bloodhound, Airedale, hyena, meerkat, giant schnauzer, titi, three-toed sloth, sor- rel, black-footed ferret, dalmatian, black-and-tan coonhound, papillon, skunk, polecat, Staffordshire bullterrier, Mexican hairless, Bouvier des Flandres, weasel, miniature poodle, malinois, bighorn, fox squirrel, colobus, tiger cat, Lhasa, impala, coyote, Yorkshire terrier, Newfoundland, brown bear, red fox. Norwegian elkhound, Rottweiler, hartebeest, Saluki, grey fox, schipperke, Pekinese, Brabancon griffon, West Highland white terrier, Sealyham terrier, guenon, mongoose, indri, tiger, Irish wolfhound, wild boar, EntleBucher, zebra, ram, French bulldog, orangutan, basenji, leopard, Bernese mountain dog, Maltese dog, Norfolk terrier toy terrier vizsla, cair,, squirrel monkey, groe- nendael, clumber, Siamese cat, chimpanzee, komondor, Afghan hound, Japanese spaniel, proboscis monkey, guinea pig
S.
Pomeranian, wombat, hare, snow leopard, Arctic fox, Sussex spaniel, lynx, wood rabbit, Saint Bernard, redbone, chow, collie, German shepherd, affenpinscher, dingo, golden retriever, Amer- ican Staffordshire terrier, briard, kelpic, Tibetan terrier, cocker spaniel, sloth bear, standard poo- dle, wire-haired fox terrier, Border terrier, American black bear, Bedlington terrier, banded gecko, wallaby, Tibetan mastiff, flat-coated retriever, koala, toy poodle, Border collie, Chesapeake Bay retriever, German short-haired pointer, great grey owl, Doberman, Lakeland terrier, miniature pin- scher, timber wolf, hog, marmot, Irish setter, bull mastiff, Irish terrier, Shetland sheepdog, keeshond, miniature schnauzer, llama, Pembroke, ice bear, standard schnauzer, white wolf, Boston bull, Gor- don setter, Great Pyrenees, Irish water spaniel, warthog, Scotch terrier, Chihuahua. Norwich terrier, Rhodesian ridgeback, borzoi, gibbon, Samoyed, tabby, Kerry blue terrier, Labrador retriever, thun- der snake, Ibizan hound, beagle, curly-coated retriever, African hunting dog, boxer, common newt,
10

---

Under review as a conference paper at ICLR 2020
giant panda, ringneck snake, Angora, beaver, lion, bluetick, basset, alligator lizard, armadillo, pug, Greater Swiss Mountain dog, hognose snake, dhole, echidna, sidewinder, Komodo dragon, silky terrier, Brittany spaniel, patas, European fire salamander, Madagascar cat, macaque, boa constrictor, gorilla, polecat, howler monkey, Appenzeller, Blenheim spaniel, Indian cobra, Shih-Tzu, baboon, kuvasz, horned viper, rhinoceros beetle, tailed frog, Eskimo dog, Gila monster, mud turtle, puchin, spider monkey, Leonberg, garter snake, African chameleon, barracouta, bullfrog, spotted salamander, leatherback turtle, rock python, marmoset, otter, Arabian camel, gar, tarantula, lan- gur, tench, platypus, Italian greyhound, box turtle. cheetah, hippopotamus, English foxhound, eft, admiral, night snake, whiptail, siamang, agama, bittern, terrapin, axoloti, African grey, African crocodile, frilled lizard, quail, water ouzel, sulphur-crested cockatoo, bison, bustard, bulbul, cock, prairie chicken, ruffed grouse, jay, partridge, tusker, spoonbill, green snake, junco, black grouse, crane, water buffalo, toucan, redshank, hornbill, ostrich, vine snake, humminghird, Indian elephant, magpic, albatross, king snake, little blue heron, bald eagle, peacock, limpkin, hamster, ruddy turn- stone, jacamar, green mamba, kite, indigo bunting, American egret, American coot, coucal, house finch, ptarmigan, black stork, robin, white stork, brambling, red- badbbcked sandpiper, king penguin, goldfinch, lorikeet, water snake, macaw, drake, vulture, bee eater, hen, dowitcher, red-breasted mer- ganser, ox, diamondback, oystercatcher, goose, pelican, black swan,


Reviews:
[{"review_id": "ryg6pPtQ5r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is about the task of object detection in the setting of few-shots dataset. The problem is addressed in the learning scheme of meta-learning paradigm: the proposed meta-rcnn trains the popular faster-rcnn on several tasks of few shots object detection while the RPN and the object classification networks are meta-learned among the tasks. Compared to previous work the paper introduces the meta learning framework and several changes to the faster rcnn detector. A prototype representation is derived from the standard RPN network and its proposed bounding box. An attention mechanism choose the object of interest and is used to train the final RPN and classification network. Experiments on the popular Pascal Voc 2007 and ImageNet-FSOD show that the proposed system have state of the art performance.\n\nThe paper is very well written, easy to read and of excellent presentation. The introduction of the meta learning paradigm and its use to learn the RPN and classification networks are incremental in novelty but interesting. The experiments are solid and show state of the art performance. As a result I recommend this paper to be accepted.\n\nMinor issues:\n- in caption of Fig1: avialable -> available\n- in 4.1: “Compared to other variants...” please add a reference to the specific methods you are comparing to.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HJeEHAb6tS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a method for few-shot object detection (FSOD), a variant of few-shot learning (FSL) where using a support set of few training images for novel categories (usually 1 or 5) not only the correct category labels are predicted on the query images, but also the object instances from the novel categories are localized and their bounding boxes are predicted. The method proposes a network architecture where the sliding window features that enter the RPN are first attenuated using support classes prototypes discovered using (a different?) RPN and found as matching to the few provided box annotations on the support images. The attenuation is by channel wise multiplication of the feature map and concatenation of the resulting feature maps (one per support class). After the RPN, ROI-pooling is applied on the concatenated feature map that is reduced using 1x1 convolution and original feature map (before attenuation) being added to the result. Following this a two FC layer classifier is fine-tuned on the support data to form the final \nRCNN head of the few-shot detector. The whole network is claimed to be meta-trained end to end following COCO or ImageNet (LOC? DET?) pre-training. The method is tested on a split of PASCAL VOC07 into two sets of 10 categories, one for meta-training and the other for meta-testing. In addition, experiments are carried out on ImageNet-LOC animals subset. In both cases, the result are compared to some baselines, and some prior work.\n\nAlthough FSOD is an important emerging problem, and advances on it are very important, I believe there are still certain gaps in the current paper that need to be fixed before it is accepted. Specifically:\n\n1. Some important details are missing from the description. For example, detectors are usually trained on high resolution images (e.g. 1000 x 1000) and hence are problematic to train with large batches, yet in the proposed approach it is claimed that the proposed model is meta-trained with batch size 5 on 5 way tasks with 10 queries each, so even in 1-shot case, does it mean that 5 x 15 = 75 high resolution images enter the GPU at each batch? I doubt that even in parallel mode with 5 GPUs and 15 high res image per GPU it is possible for claimed backbone architectures (ResNet-50 and VGG16).\nAs another example, the details of fine-tuning during meta-training seem to be left out, is the model optimized with an inner loop? Details of the RPN that is used to select the support categories prototypes are not specified, where it comes from and how is it trained (clearly as the \"main\" RPN relies on attenuated features, it cannot be it)? Some additional technical details are not very clear and hinder the reproducibility of the paper (no code seem to be promised?), in general I suggest the authors to improve the writing and clarity of the paper.\n\n2. In VOC07 experiment, FRCN-PN is very vaguely described and being claimed that it stands for RepMet (Karlinksy et al., CVPR 2019). It is not clear what it is and its training procedure on VOC07 is not clearly described.\nIt is also claimed in ImageNet experiment that the real RepMet is \"more carefully designed then FRCN-PN\" and has a better backbone, hence it is not clear why FRCN-PN should stand for it.\nI suggest the authors to either do a direct comparison or remove their claim of comparison.\n\n3. RepMet paper has proposed an additional benchmark on ImageNet-LOC with 5-way 1/5/10-shot episodes, and afaik it is reproducible as its code is released, so I am wondering as to why it was not used for \nevaluation given that the authors made the effort of reproducing another ImageNet-LOC test on the same categories? It should be evaluated for a fair comparison.\n\n4. Although they don't strictly have to compare to it, I am wondering if the authors would be willing to relate to a similar approach that was proposed for the upcoming ICCV 19: \n\"Meta R-CNN : Towards General Solver for Instance-level Low-shot Learning\", by Yan et al. Their approach is more similar to RepMet in a sense that the meta-learning is done in the classifier head,\nand better results are reported on VOC07 benchmark (and except for 1-shot, higher results are reported for the 3 and 5 shot FRCNN fine-tuning).", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BygTZ2FdYH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, authors propose a meta-learning based approach for low-shot object detection. Specifically, they use prototype in the support set as attention guidance, and learn the category-specific representation for each query image. Subsequently, they use the style of Faster RCNN for object detection.\n\nIt is an OK paper with good structure. The idea is somewhat novel, in terms of meta-learning based low-shot detection framework. My main concern is about experiment. First, the data setting is branch new. Why not use the data setting in the literature, e.g., COCO to VOC in LSTD (Chen et al., 2018)? As a result, how to make a fair comparison bothers me a little. Furthermore, LSTD is a non-episodic approach. How to make it in a meta-learning way? Please clarify the implementation details for all other related works in the comparison.  ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper develops a meta-learning approach for few-shot object detection. This paper is borderline and the reviewers are split. The problem is important, albeit somewhat specific to computer vision applications. The main concerns were that it was lacking a head-to-head comparison to RepMet and that it was missing important details (e.g. the image resolution was not clarified, nor was the paper updated to include the details). The authors suggested that the RepMet code was not available, but I was able to find the official code for RepMet via a simple Google search:
https://github.com/jshtok/RepMet
Reviewers also brought up concerns about an ICCV 2019 paper, though this should be considered as concurrent work, as it was not publicly available at the time of submission.
Overall, I think the paper is borderline. Given that many meta-learning papers compare on rather synthetic benchmarks, the study of a more realistic problem setting is refreshing. That said, it's unclear if the insights from this paper would transfer to other machine learning problem settings of interest to the ICLR community.
With all of this in mind, the paper is slightly below the bar for acceptance at ICLR.
Decision: Reject