A Broad Dataset is All You Need for One-Shot Object Detection

1 INTRODUCTION . It ’ s January 2021 and your long awaited household robot finally arrives . Equipped with the latest “ Deep Learning Technology ” , it can recognize over 21,000 objects . Your initial excitement quickly vanishes as you realize that your casserole is not one of them . When you contact customer service they ask you to send some pictures of the casserole so they can fix this . They tell you that the fix will be some time , though , as they need to collect about a thousand images of casseroles to retrain the neural network . While you are making the call your robot knocks over the olive oil because the steam coming from the pot of boiling water confused it . You start filling out the return form ... While not 100 % realistic , the above story highlights an important obstacle towards truly autonomous agents : such systems should be able to detect novel , previously unseen objects and learn to recognize them based on ( ideally ) a single example . Solving this one-shot object detection problem can be decomposed into three subproblems : ( 1 ) designing a class-agnostic object proposal mechanism that detects both known and previously unseen objects ; ( 2 ) learning a suitably general visual representation ( metric ) that supports recognition of the detected objects ; ( 3 ) continuously updating the classifier to accommodate new object classes or training examples of existing classes . In this paper , we focus on the detection and representation learning part of the pipeline , and we ask : what does it take to learn a visual representation that allows detection and recognition of previously unseen object categories based on a single example ? We operationalize this question using an example-based visual search task ( figure 1 ) that has been investigated before using handwritten characters ( Omniglot ; Michaelis et al . ( 2018a ) ) and real-world image datasets ( Pascal VOC , COCO ; Michaelis et al . ( 2018b ) ; Hsieh et al . ( 2019 ) ; Zhang et al . ( 2019 ) ; Fan et al . ( 2020 ) ; Li et al . ( 2020 ) ) . Our central hypothesis is that scaling up the number of object categories used for training should improve the generalization capabilities of the learned representation . This hypothesis is motivated by the following observations . On ( cluttered ) Omniglot ( Michaelis et al. , 2018a ) , recognition of novel characters works almost as well as for characters seen during training . In this case , sampling enough categories during training relative to the visual complexity of the objects is sufficient to learn a metric that generalizes to novel categories . In contrast , models trained on visually more complex datasets like Pascal VOC and COCO exhibit a large generalization gap : novel categories are detected much less reliably than ones seen during training . This result suggests that on the natural image datasets , the number of categories is too small given the visual complexity of the objects and the models retreat to a shortcut ( Geirhos et al. , 2020 ) – memorizing the training categories . To test the hypothesis that wider datasets improve generalization , we increase the number of object categories during training by using datasets ( LVIS , Objects365 ) that have a larger number of categories annotated . Our experiments support this hypothesis and suggest the following conclusions : • The generalization gap between training and novel categories is a key problem in one-shot object detection . • This generalization gap can be almost closed by increasing the number of categories used for training : going from 80 classses in COCO to 1200 in LVIS improves relative performance from 45 % to 89 % . • A detailed analysis shows that number of categories , not the amount of data , is the driving force behind this effect . • Closing the generalization gap allows using established methods from the object detection community ( like e.g . stronger backbones ) to improve performance on known and novel categories alike . • We use these insights to improve state-of-the-art performance on COCO by 5.4 % AP50 ( from 22 % AP50 to 27.5 % AP50 ) using annotations from LVIS . 2 RELATED WORK . Object detection Object detection - the task of detecting objects in complex , cluttered scenes - has seen huge progress since the widespread adoption of DNNs Girshick et al . ( 2014 ) ; Ren et al . ( 2015 ) ; He et al . ( 2017 ) ; Lin et al . ( 2017a ) ; Chen et al . ( 2019a ) ; Wu et al . ( 2019 ) ; Carion et al . ( 2020 ) . Similarly the number of datasets has grown steadily , fueled by the importance this task has for computer vision applications Everingham et al . ( 2010 ) ; Russakovsky et al . ( 2015 ) ; Lin et al . ( 2014 ) ; Zhou et al . ( 2017 ) ; Neuhold et al . ( 2017 ) ; Krasin et al . ( 2017 ) ; Gupta et al . ( 2019 ) ; Shao et al . ( 2019 ) . However most models and datasets focus on scenarios where abundant examples per category are available . Few-shot learning Algorithms for few-shot learning - learning a model from only a few examples - can broadly be separated into two categories : Metric learningKoch et al . ( 2015 ) ; Vinyals et al . ( 2016 ) ; Snell et al . ( 2017 ) - learn a good representation and metric that generalizes to new data . And meta learning Finn et al . ( 2017 ) ; Rusu et al . ( 2018 ) - learn a good way to learn a new task . However , recent work has shown that complex algorithmic approaches can be rivaled by improving and scaling simple approaches like transfer learning Chen et al . ( 2019b ) ; Nakamura & Harada ( 2019 ) ; Dhillon et al . ( 2019 ) ; Kolesnikov et al . ( 2019 ) . Few-shot & one-shot object detection Recently , several groups have started to tackle few-shot learning for object detection . Two training and evaluation paradigms have emerged . The first is inspired by continual learning : incorporate a set of new categories with only a few labeled images per category into an existing classifier Kang et al . ( 2018 ) ; Yan et al . ( 2019 ) ; Wang et al . ( 2019 ; 2020 ) . The second one phrases the problem as an example-based visual search : detect objects based on a single example image ( Michaelis et al. , 2018b ; Hsieh et al. , 2019 ; Zhang et al. , 2019 ; Fan et al. , 2020 ; Li et al. , 2020 , figure 1 left ) . We refer to the former ( continual learning ) as few-shot object detection , since typically 10–30 images are used for experiments on COCO . In contrast , we refer to the latter ( visual search ) as one-shot object detection , since the focus is on the setting with a single example . In the present paper we work with this latter paradigm , since it focuses on the representation learning part of the problem and avoids the additional complexity of continual learning . Methods for one-shot object detection Existing methods for one-shot object detection usually combine a standard object detection architecture with metric or meta-learning methods Biswas & Milanfar ( 2015 ) ; Michaelis et al . ( 2018b ) ; Hsieh et al . ( 2019 ) ; Zhang et al . ( 2019 ) ; Fan et al . ( 2020 ) ; Osokin et al . ( 2020 ) ; Li et al . ( 2020 ) . To better handle complex scenes and pose changes methods such as spatial awareness Li et al . ( 2020 ) or pose transforms Biswas & Milanfar ( 2015 ) ; Osokin et al . ( 2020 ) have been proposed . A recent method uses a transformer to solve the matching problem Chen et al . ( 2021 ) . We here use one of the most straightforward models , Siamese Faster R-CNN Michaelis et al . ( 2018b ) , to demonstrate that a change of the training data rather than the model architecture is sufficient to substantially reduce the generalization gap between known and novel categories . Number of categories in few-shot learning Most of the few-shot learning literature focuses on developing algorithmic solutions to a set of existing small-scale benchmarks . In contrast a lot less attention has been payed to exploring new tasks or datasets . The influence of the training data was mostly observed indirectly , e.g . through better performance on datasets with more categories such as tieredImageNet vs. miniImageNet . We here flip the focus demonstrating that significant progress can be made by keeping the algorithm the same and only changing the training data . Concurrent studies confirm this finding that more categories help few-shot object detection Fan et al . ( 2020 ) and few-shot image classification Sbai et al . ( 2020 ) ; Jiang et al . ( 2020 ) . We add to this by not only looking at few-shot performance but comparing it with performance on known categories ( generalization gap ) . This allows us to uncover the functional relationship behind the effect ( closing a shortcut ) . 3 EXPERIMENTS . Models We mainly use Siamese Faster R-CNN , an example-based version of Faster R-CNN Ren et al . ( 2015 ) similar to Siamese Mask R-CNN Michaelis et al . ( 2018b ) . Briefly , it consists of a feature extractor , a matching step and a standard region proposal network and bounding box head ( figure 2 ) . The feature extractor ( called backbone in object detection ) is a standard ResNet-50 with feature pyramid networks He et al . ( 2016 ) ; Lin et al . ( 2017a ) which is applied to the image and reference with weight sharing . In the matching step the reference representation is compared to the image representation in a sliding window approach by computing a feature-wise L1 difference . The resulting similarity encoding representation is concatenated to the image representation and passed on to the region proposal network ( RPN ) . The RPN proposes a set of bounding boxes which potentially contain objects . These boxes are then classified as containing an object from the reference class or something else ( other object or background ) . Box coordinates are refined by bounding box regression and overlapping boxes are removed using non-maximum suppression . We additionally developed Siamese RetinaNet , a single-stage detector based on RetinaNet Lin et al . ( 2017b ) . The feature extraction and matching steps are identical to Siamese Faster R-CNN , but it uses the unified RetinaHead to jointly propose and classify bounding boxes . To counter the effect of too many negative samples , the classifier is trained with focal loss Lin et al . ( 2017b ) . Training & Evaluation The example-based training is slightly different from the traditional object detection training paradigm . For each image a reference category is randomly chosen by picking a category with at least one instance in the image . A reference is retrieved by randomly selecting one instance from this category in another image and tightly cropping it . The labels for each bounding box are changed to 0 or 1 depending on whether the object is from the reference category or not . Annotations for objects from the held-out categories are removed from the dataset before training . At test time a similar procedure is chosen but instead of picking one category for each image , all categories with at least one object in the image are chosen Michaelis et al . ( 2018b ) and one ( 1-shot ) or five ( 5-shot ) reference images are provided . Predictions are assigned their corresponding category label and evaluation is performed using standard tools and metrics . Implementation We implemented Siamese Faster R-CNN and Siamese RetinaNet in mmdetection v1.0rc Chen et al . ( 2019a ) , which improved performance by more than 30 % over the original Siamese Mask R-CNN ( Michaelis et al. , 2018b , Table 4 ) . We keep all hyperparameters the same as in the standard Faster R-CNN implementation of mmdetection . Due to resource constraints we reduce the number of samples per epoch to 120k for Objects365 . Hyperparameters Our model is derived from mmdetection v1.0rc Chen et al . ( 2019a ) and uses the same hyperparameters as used for Faster R-CNN and RetinaNet1 . Please note that the default settings for Pascal VOC differ slightly from those for COCO training . We use the COCO hyperparameters for experiments on COCO , LVIS and Objects365 and Pascal VOC settings for Pascal VOC . Datasets We use the four datasets shown in Table 1 : COCO Lin et al . ( 2014 ) , Objects365 Shao et al . ( 2019 ) , LVIS Gupta et al . ( 2019 ) and Pascal VOC Everingham et al . ( 2010 ) . We use standard splits and test on the validation sets except for Pascal VOC where we test on the 2007 test set . Due to resource constraints , we evaluate Objects365 on a fixed subset of 10k images from the validation set . Category splits Following common protocol for example-based detection Michaelis et al . ( 2018b ) ; Shaban et al . ( 2017 ) we split the categories in each dataset into four splits using every fourth category as hold-out set and the other 3/4 categories for training . So on Pascal VOC there are 15 categories for training in each split , on COCO there are 60 , on Objects365 274 and on LVIS 902 . We train and test four models ( one for each split ) and report the mean over those four models , so performance is always measured on all categories . Computing performance in this way across all categories is preferable to using a fixed subset as some categories may be harder than others . During evaluation , the reference images are chosen randomly . We therefore run the evaluation five times , reporting the average AP50 over splits . The 95 % confidence intervals for the average AP50 is below ±0.2 % AP50 for all experiments . 1All details can be found in the respective configs : https : //github.com/open-mmlab/ mmdetection/tree/5bf935e1b7621b234ddb34ef6c32b2b524243995/configs

Summary: This paper studies one-shot 2D object detections. The major conclusion is that when keeping the number of training images fixed, increasing the number of training categories can significantly increase the one-shot performance.  This paper demonstrated this conclusion empirically by doing controlled experiments on COCO, Objects365, and LIVS.   Inspired by this observation, this paper improves the state-of-the-art one-shot detection performance on COCO from 22.0 to 27.5 AP50 by training on LIVS.  Other two related conclusions in this paper: - PASCAL VOC is not suitable for evaluating one-shot object detection algorithms. The reason may be that the number of instances per image is low(2.9 ins/img on average). The algorithm only needs to recognize foreground objects rather than objects of target categories.  - Only when the training data is challenging enough, increasing the model size and training time can help improve one-shot performance.