Closing the Generalization Gap in One-Shot Object Detection

Search Task Generalization to Novel CategoriesA B 1 INTRODUCTION . It ’ s January 2021 and your long awaited household robot finally arrives . Equipped with the latest “ Deep Learning Technology ” , it can recognize over 21,000 objects . Your initial excitement quickly vanishes as you realize that your casserole is not one of them . When you contact customer service they ask you to send some pictures of the casserole so they can fix this . They tell you that the fix will be some time , though , as they need to collect about a thousand images of casseroles to retrain the neural network . While you are making the call your robot knocks over the olive oil because the steam coming from the pot of boiling water confused it . You start filling out the return form ... While not 100 % realistic , the above story highlights an important obstacle towards truly autonomous agents such as household robots : such systems should be able to detect novel , previously unseen objects and learn to recognize them based on ( ideally ) a single example . Solving this one-shot object detection problem can be decomposed into three subproblems : ( 1 ) designing a class-agnostic object proposal mechanism that detects both known and previously unseen objects ; ( 2 ) learning a suitably general visual representation ( metric ) that supports recognition of the detected objects ; ( 3 ) continuously updating the classifier to accommodate new object classes or training examples of existing classes . In this paper , we focus on the detection and representation learning part of the pipeline , and we ask : what does it take to learn a visual representation that allows detection and recognition of previously unseen object categories based on a single example ? We operationalize this question using an example-based visual search task ( Fig . 1 ) that has been investigated before using handwritten characters ( Omniglot ; Michaelis et al . ( 2018a ) ) and real-world image datasets ( Pascal VOC , COCO ; Michaelis et al . ( 2018b ) ; Hsieh et al . ( 2019 ) ; Zhang et al . ( 2019 ) ; Fan et al . ( 2020 ) ; Li et al . ( 2020 ) ) . Our central hypothesis is that scaling up the number of object categories used for training should improve the generalization capabilities of the learned representation . This hypothesis is motivated by the following observations . On ( cluttered ) Omniglot ( Michaelis et al. , 2018a ) , recognition of novel characters works almost as well as for characters seen during training . In this case , sampling enough categories during training relative to the visual complexity of the objects is sufficient to learn a metric that generalizes to novel categories . In contrast , models trained on visually more complex datasets like Pascal VOC and COCO exhibit a large generalization gap : novel categories are detected much less reliably than ones seen during training . This result suggests that on the natural image datasets , the number of categories is too small given the visual complexity of the objects and the models retreat to a shortcut ( Geirhos et al. , 2020 ) – memorizing the training categories . To test the hypothesis that wider datasets improve generalization , we increase the number of object categories during training by using datasets ( LVIS , Objects365 ) that have a larger number of categories annotated . Our experiments support this hypothesis and suggest the following conclusions : • The generalization gap between training and novel categories is a key problem in one-shot object detection . • This generalization gap can be almost closed by increasing the number of categories used for training : going from 80 classses in COCO to 1200 in LVIS improves relative performance from 45 % to 89 % . • The number of categories , not the amount of data , is the driving force behind this effect . • Closing the generalization gap allows us to use established methods from the object detec- tion community ( like e.g . stronger backbones ) to make further progress . • We use these insights to improve state-of-the-art performance on COCO by 5.4 % AP50 ( from 22 % AP50 to 27.5 % AP50 ) using annotations from LVIS . 2 RELATED WORK . Object detection Object detection has seen huge progress since the widespread adoption of DNNs ( Girshick et al. , 2014 ; Ren et al. , 2015 ; He et al. , 2017 ; Lin et al. , 2017a ; Chen et al. , 2019a ; Wu et al. , 2019b ; Carion et al. , 2020 ) . Similarly the number of datasets has grown steadily , fueled by the importance this task has for computer vision applications ( Everingham et al. , 2010 ; Russakovsky et al. , 2015 ; Lin et al. , 2014 ; Zhou et al. , 2017 ; Neuhold et al. , 2017 ; Krasin et al. , 2017 ; Gupta et al. , 2019 ; Shao et al. , 2019 ) . However most models and datasets focus on scenarios where abundant examples per category are available . Few-shot learning The two most common approaches to few-shot learning have been , broadly speaking , based on metric learning ( Koch et al. , 2015 ; Vinyals et al. , 2016 ; Snell et al. , 2017 ) and meta learning : Learn a good way to learn a new task ( Finn et al. , 2017 ; Rusu et al. , 2018 ) , or combinations thereof ( Sun et al. , 2019 ) . However , recent work has shown that much simpler approaches based on transfer learning achieve competitive performance ( Chen et al. , 2019b ; Nakamura & Harada , 2019 ; Dhillon et al. , 2019 ) . A particularly impressive example of this line of work is Big Transfer ( Kolesnikov et al. , 2019 ) , which uses transfer learning from a huge architecture trained on a huge dataset to perform one-shot ImageNet classification . Few-shot & one-shot object detection Recently , several groups have started to tackle few-shot learning for object detection . Two training and evaluation paradigms have emerged . The first is inspired by continual learning : incorporate a set of new categories with only a few labeled images per category into an existing classifier ( Kang et al. , 2018 ; Yan et al. , 2019 ; Wang et al. , 2019 ; 2020 ) . The second one phrases the problem as an example-based visual search : detect objects based on a single example image ( Fig . 1 left ; Michaelis et al. , 2018b ; Hsieh et al. , 2019 ; Zhang et al. , 2019 ; Fan et al. , 2020 ; Li et al. , 2020 ) . We refer to the former ( continual learning ) as few-shot object detection , since typically 10–30 images are used for experiments on COCO . In contrast , we refer to the latter ( visual search ) as one-shot object detection , since the focus is on the setting with a single example . In the present paper we work with this latter paradigm , since it focuses on the representation learning part of the problem and avoids the additional complexity of continual learning . Methods for one-shot object detection Existing methods for one-shot object detection employ a combination of a standard object detection architecture with a siamese backbone and various forms of feature attention and concatenation on the backbone output or in the heads ( Biswas & Milanfar , 2015 ; Michaelis et al. , 2018b ; Hsieh et al. , 2019 ; Zhang et al. , 2019 ; Fan et al. , 2020 ; Osokin et al. , 2020 ; Li et al. , 2020 ) . Spatially aware similarity measures ( Li et al. , 2020 ) or transformations ( Biswas & Milanfar , 2015 ; Osokin et al. , 2020 ) improve recognition in cases where the pose of the reference objects differs from that of the detected object . We here use one of the most straightforward models , Siamese Faster R-CNN ( Michaelis et al. , 2018b ) , to demonstrate that a change of the training data rather than the model architecture is sufficient to substantially reduce the generalization gap between known and novel categories . Related tasks A number of related pieces of work propose approaches to slightly different example-based search tasks . Examples include one-shot segmentation using handwritten characters ( Michaelis et al. , 2018a ) , natural textures ( Ustyuzhaninov et al. , 2018 ) and natural images ( Shaban et al. , 2017 ) . In addition , several groups have suggested one-shot and few-shot detection tasks with slightly different focus and protocols ( Dong et al. , 2018 ; Chen et al. , 2018 ; Schwartz et al. , 2019 ; Wu et al. , 2019a ) , including episodic evaluation ( Wu et al. , 2019a ) , transfer across datasets ( Chen et al. , 2018 ) and fine-grained detection ( Schwartz et al. , 2019 ) . Also closely related are instance retrieval ( Tolias et al. , 2016 ) and co-segmentation ( Rother et al. , 2006 ; Hsu et al. , 2019 ) . The key difference of our work is that we do not propose a new architecture , but instead investigate the relationship between the number of categories used during training and the generalization to novel categories . Number of categories in few-shot learning Most of the few-shot learning literature focuses on developing new methods for existing benchmarks . The influence of the training data was mostly observed indirectly , e.g . through better performance on datasets with more categories such as tieredImageNet vs. miniImageNet . Two concurrent studies report that more categories help few-shot object detection ( Fan et al. , 2020 ) and investigate the influence of data diversity , image complexity , intra- and inter-category diversity and other factors on few-shot classification ( Jiang et al. , 2020 ) . Both publications are consistent with out results that the number of categories is a key factor for improving few-shot performance . 3 EXPERIMENTS . Models We mainly use Siamese Faster R-CNN , a one-shot detection version of Faster R-CNN ( Ren et al. , 2015 ) similar to Siamese Mask R-CNN ( Michaelis et al. , 2018b ) . Briefly , it consists of a feature extractor , a matching step and a standard region proposal network and bounding box head ( Fig . 2 ) . The feature extractor ( called backbone in object detection ) is a standard ResNet-50 with feature pyramid networks ( He et al. , 2016 ; Lin et al. , 2017a ) which is applied to the image and reference with weight sharing . In the matching step the reference representation is compared to the image representation in a sliding window approach by computing a feature-wise L1 difference . The resulting similarity encoding representation is concatenated to the image representation and passed on to the region proposal network ( RPN ) . The RPN proposes a set of bounding boxes which potentially contain objects . These boxes are then classified as containing an object from the reference class or something else ( other object or background ) . Box coordinates are refined by bounding box regression and overlapping boxes are removed using non-maximum suppression . We additionally developed Siamese RetinaNet , a single-stage detector based on RetinaNet ( Lin et al. , 2017b ) . The feature extraction and matching steps are identical to Siamese Faster R-CNN , but it uses Siamese Faster R-CNN the unified RetinaHead to jointly propose and classify bounding boxes . To counter the effect of too many negative samples , the classifier is trained with focal loss Lin et al . ( 2017b ) . Training & Evaluation During training a reference category is randomly chosen for every image by picking a category with at least one instance in the image . A reference is retrieved by randomly selecting one instance from this category in another image and tightly cropping it . The labels for each bounding box are changed to 0 or 1 depending on whether the object is from the reference category or not . Annotations for objects from the held-out categories are removed from the dataset before training . At test time a similar procedure is chosen but instead of picking one category for each image , all categories with at least one object in the image are chosen Michaelis et al . ( 2018b ) and one ( 1-shot ) or five ( 5-shot ) reference images are provided . Predictions are assigned their corresponding category label and evaluation is performed using standard tools and metrics . Implementation We implemented Siamese Faster R-CNN and Siamese RetinaNet in mmdetection v1.0rc ( Chen et al. , 2019a ) , which improved performance by more than 30 % over the original paper ( Table 4 ; Michaelis et al. , 2018b ) . We keep all hyperparameters the same as in the standard Faster R-CNN implementation of mmdetection ( which achieves 36.4 % mAP/58.4 % AP50 on regular COCO ) . Due to resource constraints we reduce the number of samples per epoch to 120k for Objects365 . Datasets We use the four datasets shown in Table 1 : COCO ( Lin et al. , 2014 ) , Objects365 ( Shao et al. , 2019 ) , LVIS ( Gupta et al. , 2019 ) and Pascal VOC ( Everingham et al. , 2010 ) . We use standard splits and test on the validation sets except for Pascal VOC where we test on the 2007 test set . Due to resource constraints , we evaluate Objects365 on a fixed subset of 10k images from the validation set . Following common protocol ( Michaelis et al. , 2018b ; Shaban et al. , 2017 ) we split the categories in each dataset into four splits using every fourth category as hold-out set and the other 3/4 categories for training . So on Pascal VOC there are 15 categories for training in each split , on COCO there are 60 , on Objects365 274 and on LVIS 902 . We train and test four models ( one for each split ) and report the mean over those four models , so performance is always measured on all categories . Computing performance in this way across all categories is preferable to using a fixed subset as some categories may be harder than others . During evaluation , the reference images are chosen randomly . We therefore run the evaluation five times , reporting the average AP50 over splits . The 95 % confidence intervals for the average AP50 is below ±0.2 % AP50 for all experiments .

Summary: The paper suggests that a major factor for increasing few-shot performance in the few-shot object detection task is the number of categories in the base training set used to pre-train the few-shot model on a large set of data before it is adapted to novel categories using only a few (or even 1) examples. This effect is measured by the authors by trying out the existing Siamese few-shot detector on 4 datasets: PASCAL, COCO, Objects365, and LVIS showing that the gap in performance on the seen training and the unseen (novel) testing categories is reduced when the base dataset has more classes (e.g. on LVIS where there are more than 1K classes, this "generalization" gap is shown to be minimal). The authors also quantify empirically the effect of increasing the model size and of prolonging the training schedule on this gap. As well as testing on COCO classes while training on LVIS.