A Broad Dataset is All You Need for One-Shot Object Detection
Authors: Claudio Michaelis
Matthias Bethge
Alexander S Ecker
Venue: ICLR 2022 Submitted (2022)
Keywords: One-Shot Learning
Object Detection
Generalization
Instance Segmentation

Abstract: Is it possible to detect arbitrary objects from a single example? A central problem of all existing attempts at one-shot object detection is the generalization gap: Object categories used during training are detected much more reliably than novel ones. We here show that this generalization gap can be nearly closed by increasing the number of object categories used during training. Doing so allows us to improve generalization from seen to unseen classes from 45% to 89% and improve the state-of-the-art on COCO by 5.4 AP50 (from 22.0 to 27.5). 
We verify that the effect is caused by the number of categories and not the number of training samples, and that it holds for different models, backbones and datasets. This result suggests that the key to strong few-shot detection models may not lie in sophisticated metric learning approaches, but instead simply in scaling the number of categories. We hope that our findings will help to better understand the challenges of few-shot learning and encourage future data annotation efforts to focus on wider datasets with a broader set of categories rather than gathering more samples per category.

Under review as a conference paper at ICLR 2022
A BROAD DATASET IS ALL YOU NEED FOR ONE-SHOT ОВJЕCT DETECTION
Anonymous authors Paper under double-blind review
ABSTRACT
Is it possible to detect arbitrary objects from a single example? A central prob- lem of all existing attempts one-shot object detection the generalization gap: Object categories used during training are detected much more reliably than novel ones. We here show that this generalization gap can be nearly closed by increasing the number of object categories used during training. Doing so allows us to im- prove generalization from seen unseen classes from 45% 89% and improve the tfiie-of-theaarr on COCO by 5.4 %APS5 (from 22.0 to 27.5). We verify that the effect is caused by the number of categories and the number training sam- ples, and that it holds for different models, backbones and datasets. This result suggests that the key to strong few-shot detection models may not lie in sophis- ticated metric learning approaches, but instead simply in scaling the number of categories. We hope that our findings will help to better understand the challenges of few-shot learning and encourage future data annotation efforts to focus wider datasets with a broader set of categories rather than gathering more samples per category.
Example based object detection
A broad training dataset
Successful Detection of Novel Objects!
Ro por somermaaaioo Pova///itn ol c84p025
76%
+
COCO
4856
Objecticnis
CO00
Osjoctus55
Categories
80
365
1203
Figure 1: Example based object detectors in theory detect any object based example im- age. However existing models trained on the datasets with few categories such as COCO perform significantly worse for novel than known objects. We here show that this generalization gap progres- sively shrinks when training the same models with more categories thus moving us closer to models which can actually detect any object.
INTRODUCTION
It's January 2021 and your long awaited household robot finally arrives. Equipped with the latest "Deep Learning Technology", it can recogmize over 21,000 objects. Your initial excitement quickly vanishes you realize that your casserole is not of them. When you contact customer service they ask you to send some pictures of the casserole so they can fix this. They tell you that the fix will be some time, though, as they need to collect about a thousand images of casseroles to retrain the neural network. While you are making the call your robot knocks over the olive oil because the steam coming from the pot of boiling water confused it. You start filling out the return form...
While not 100% realistic, the above story highlights important obstacle towards truly autonomous agents: such systems should be able to detect novel, previously unseen objects and learn to recog- nize them based on (ideally) single example. Solving this one-shot object detection problem can

---

Under review as a conference paper at ICLR 2022
be decomposed into three subproblems: (1) designing class- object proposal mechanism that detects both known and previously unseen objects; (2) learning a suitably general visual rep- resentation (metric) that supports recognition of the detected objects; (3) continuously updating the classifier to accommodate new object classes or training examples of existing classes. In this paper, focus the detection and representation learning part of the pipeline, and we ask: what does it take to learn a visual representation that allows detection and recognition of previously unseen object categories based on a single example?
We operationalize this question using an example-based visual search task (figure 1 that has been in- vestigated before using handwritten characters (Omniglot: Michaelis al. 2)) and real-world image datasets (Pascal VOC, COCO; Michaelis et al.] (2018b); Hsiih et al. (2019): Zhang et al.
2 Fan et al. (2020); Li et al. (2020). Our central hypothesis is that scaling up the number of object categories used for training should improve the generalization capabiliiiess of the learned representation. This hypothesis
motivated by the following observations. On (cluttered) Om- niglot (Michaelis et al. 2018a), recognition of novel characters works almost well as for char- acters seen during training. In this case, sampling enough categories during training relative to the visual complexity of the objects is sufficient learn metric that generalizes to novel categories. In contrast, models trained on visually more complex datasets like Pascal VOC and COCO exhibit large generalization gap: novel categories training. This result suggests that on the natural image datasets, the number of categories is too small given the visual complexity of the objects and the models retreat shortcut (Geirhos al., 2020 memorizing the training categories.
detected much less reliably than ones seen during
To test the hypothesis that wider datasets improve generalization, increase the number of object categories during training by using datasets (LVIS, Objects365) that have a larger number of cate- gories annotated. Our experiments support this hypothesis and suggest the following conclusions:
The generalization gap between training and novel categories is key problem in one-shot object detection.
This generalization gap can be almost closed by increasing the number of categories used for training: going from 80 classses in COCO to 1200 in LVIS improves relative perfor- mance from 45% to 89%.
A detailed analysis shows that number of categories, not the amount of data, is the driving force behind this effect.
Closing the generalization gap allows using established methods from the object detection community (like e.g. stronger backbones) to improve performance on known and novel categories alike.
We use these insights to improve state-of-the-art performance on COCO by 5.4 %AP50 (from 22 %AP50 27.5 %AP50) using annotations from LVIS.
2 RELATED WORK
Object detection Object detection the task of detecting objects in complex, cluttered scenes has seen huge progress since the widespread adoption of DNNs Girshick et al. (2014); Ren et al. 2015 : He et al. (2017): Lin et al. (2017a): Chen et al. (011a); Wu et al. (2019): Carion et al. 2020 Similarly the number of datasets has grown steadily, fueled by the importance this task has for computer vision applications Everingham et al. (2010); Russakovsky et al. 2015); Lin et al. (2014; Zhou et al. (2017): Neuhold et al. (2017): Krasin et al. (211);; Gupta et al. (2019): Shao et al. (2019). However most models and datasets focus on scenarios where abundant examples per category are available.
Few-shot learning Algorithms for few-shot learning learning model from only few examples can broadly be separated into two categories: Metric Koch et al. (015)); Vinyals et al. 2016); Snell et al. (2017)- learn a good representation and metric that generalizes to new data. And meta learning Finn et al. (): Rusu et al. (2018) learn a good way to learn a new task. However, recent work has shown that complex algorithmic approaches can be rivaled by improving and scaling simple approaches like transfer learning Chen et al. (2019b); Nakamura & Harada (2019): Dhillon et al. (2019); Kolesnikov al. (2019).
2

---

Under review as a conference paper at ICLR 2022
Few-shot & one-shot object detection Recently, several groups have started to tackle few-shot learning for object detection. Two training and evaluation paradigms have emerged. The first is inspired by continual learning: incorporate set of new categories with only few labeled images per category into an existing classifier Kang et al. 2018); Yan et al. 2019); [Wang et al. (2019) 2020).
The second one phrases the problem
example- sample-based visual search: detect objects based
single example image (Michaelis et al. 201 8b Hsich et al. 2019) Zhang et al. 2019 Fan et al. 2020)
Li al. 2020 figure llett.. We refer to the former (continual learning) as few-shot object detection, since typically 10-30 images are used for experiments on COCO. In contrast, we refer to the latter (visual search) as one-shot object detection, since the focus is on the setting with single example. In the present paper we work with this latter paradigm, since it focuses on the representation learning part of the problem and avoids the additional complexity of continual learning.
Methods for one-shot object detection Existing methods for one-shot object detection usually combine a standard object detection architecture with metric or meta-learning methods BBisws &
Milanfar (2015); Michaelis et al. (2018b); Hsieh et al. (2019); Zhang al. (2019) ; Fan et al. (2020);
Osokin et al. (2020); Li et al. (2020). To better handle complex scenes and pose changes methods such spatial awareness Li et al. (2020) or pose transforms Biswas & Milanfar (2015); Osokin et al. (2020) have been proposed. A recent method uses a transformer to solve the matching problem Chen al. 202T We here use one of the most straightforward models, Siamese Faster R-CNN Michaelss et al. (018b), to demonstrate that a change of the training data rather than the model architecture is sufficient substantially reduce the generalization gap between known and novel categories.
Number of categories in few-shot learning Most of the few-shot learning literature focuses on developing algorithmic solutions to a set of existing small-scale benchmarks. In contrast lot less attention has been payed to exploring new tasks or datasets. The influence of the training data was mostly observed indirectly, e.g. through better performance on datasets with more categories such as mieredl vs. minllmageNet. We here flip the focus demonstrating that significant progress can made by keeping the algorithm the same and only changing the training data. Concurrent studies confirm this finding that more categories help fhw-shot object detection Fan et al. (2020) and few-shot image classification Sbai et al. (2020); Jiang et al. (2020). We add to this by not only look- ing at few-shot performance but comparing it with performance on known categories (generalization gap). This allows us to uncover the functional relationship behind the effect (closing a shortcut).
3 EXPERIMENTS
Models We mainly use Siamese Faster R-CNN, an example-basee based version of Faster R-CNN Ren al. (2015) similar Siamese Mask R-CNN Michaelis et al. (2018b). Briefly, it consists of feature extractor, a matching step and a standard region proposal network and bounding box head (figure D The feature extractor (called backbone in object detection) is a standard ResNet-50 with feature pyramid networks He et al. (2016); Lin et al. (2017a) which is applied to the image and reference with weight sharing. In the matching step the reference representation is compared to the image representation in a sliding window approach by computing a feature-wisg LI difference. The resulting similarity encoding representation concatenated the image representation and passed on to the region proposal network (RPN), The RPN proposes set of bounding boxes which po- tentially contain objects. These boxes are then classified as containing an object from the reference
Siamese Faster R-CNN
Siamese Backbone
Matching
RPN
ResNet 50
ResNet 50
IF
BBOX
Heads
iragge L1 Peirovier Difernncc
Figure 2: Siamese Faster R-CNN
3

---

Under review as a conference paper at ICLR 2022
class or something else (other object or background). Box coordinates are refined by bounding box regression and overlapping boxes are removed using non -on-maximum suppression.
We additionally developed Siamese RetinaNet, single-stage detector based on RetinaNet Lin et al. (2017)). The feature extraction and matching steps are identical to Siamese Faster R-CNN, but uses the unified RetinaHead jointly propose and classify bounding boxes. To counter the effect of too many negative samples, the classifier is trained with focal loss Lin et al. (0017b).
Training & Evaluation The example- based training is slightly different from the traditional ob- ject detection training paradigm. For each image a reference category is randomly chosen by picking category with at least instance in the image. A reference is retrieved by randomly selecting one instance from this category in another image and tightly cropping it. The labels for each bounding depending on whether the object from the reference category or not. Annotations for objects from the held-out categories are removed from the dataset before training. At test time similar procedure is chosen but instead of picking one category for each image, all categories with at least one object in the image are chosen Michaelis et al. (2018b) and one (I-shot) or five (5-shot) reference images are provided. Predictions are assigned their corresponding category label and evaluation is performed using standard tools and metrics.
box are changed to
Implementation We implemented Siamese Faster R-CNN and Siamese RetinaNet in mmdetec- tion v1.0re Chen al. (2019a). which improved performance by more than 30% over the original Siamese Mask R-CNN (Michaclis et al. 2018b] Table 4) We keep all hyperparameters the same as in the standard Faster R-CNN implementation of mmdetection. Due resource constraints reduce the number of samples per epoch to 120k for Objects365.
Hyperparameters Our model is derived from mmdetection v1.0re Chen et al. (2019a) and uses the same hyperparameters as used for Faster R-CNN and RetinaNef] Please note that the default settings for Pascal VOC differ slightly from those for COCO training. We use the COCO hyper- parameters for experiments on COCÓ, LVIS and Objects365 and Pascal VOC settings for Pascal VOC.
Datasets We use the four datasets shown in Table COCO Lin et al. (0014) Objects365 Shao
al. (2010). We use standard splits and test on the validation sets except for Pascal VOC where we test on the 2007 test set. Due resource constraints, we evaluate Objects365 on a fixed subset of 10k images from the validation
al.) (209)), LVIS Gupta al. (2019) and Pascal VOC Everingham
set.
Category splits Following common protocol for example-based detection Michaelis et al. (2018)): (Shaban et al. (2017) we split the categories in each dataset into four splits using every fourth category as hold-out set and the other 3/4 categories for training. So on Pascal VOC there are 15 categories for training in each split, COCO there are 60, on Objects365 274 and on LVIS 902. We train and test four models (one for each split) and report the mean over those four models, so performance is always measured on all categories. Computing performance in this way actoss all categories is preferable to using fixed subset as some categories may be harder than others. During evaluation, the reference images chosen randomly. We therefore run the evaluation five times, reporting the average AP50 over splits. The 95% confidence intervals for the average AP50 is below +0.2%Ap5 for all experiments.
Dataset
Version
Classes
Images
Instances
Ins/Img
Cls/Img
Thr.
Pascal VOC
07+12
20
8k
23k
2.9
1.6
COCO
2017
80
118k
860k
7.3
2.9
LVIS
vl
1,203
100k
1.27M
12288*
'3,6'
Objects365
365
1.94M
28M
14.6
6.1
Table 1: Dataset comparison. Thr. Throughoutly annotated: every instance of every class is annotated in every image. "LVIS has potentially more objects and categories per image than are annotated due the non-exhaustive labeling.
!All details can he found in the respective configs: mmdettecti
https://gitthuu..omm/ppe.
4

---

Under review as a conference paper at ICLR 2022
Skateboard
Elephant
Person
Horee
Television Set
Goal
Shopping Bag
Lyn
Bottle
Fire Truck
Duck
Fork
Figure 3:
Example predictions on held-out categories (ResNet-50) backbone). The left three columns show success cases. The rightmost column shows failure cases in which objects are overlooked and/or wrongfully detected.
4 RESULTS
4.1
GENERALIZATION GAP ON COCO AND PASCAL VOC
We start by showing that objects of held-out categories are detected less reliably on COCO and Pascal VOC. On both datasets, Siamese Faster R-CNN shows strong signs of overfitting to the training categories (figure 4 Table 2): despite setting a new stat-of-the-art performance is much higher than for categories held-out during training (COCO: 49.7 22.8 %AP"0, Pascal VOC: 82.7 37.6 %AP*o). We refer to this drop in performance as the generalization gap. This result is consistent with the literature: Hsieh et al. (2019) the previous ofate--f-the-aee art report performance dropping 40.9 + 22.0 %APS0 on COCO (see Table 4 beloow. Some newer models reportedly close the gap Pascal VOC Zhang al. (2019); Hsieh al. (201)); Li al. (2020); will discuss Pascal VOC further in the next section. Example predictions show good localization (bounding boxes) even for unknown objects cluttered scenes while classification errors make up the majority of mistakes (figure 3
4.2 PASCAL VOC IS TOO EASY TO EVALUATE ONE-SHOT OBJECT DETECTION MODELS
Having identified this large generalization gap, we ask whether the models have learned a useful metric for one-shot detection contains, on average, only 1.6 categories and 2.9 instances per image. In this case, simply detecting all foreground objects may be viable strategy. To test how well such a trivial strategy would per- form, we provide the model with uninformative references (we use all-black images). Interestingly, this baseline performs very well, achieving 59.6 %AP30 on training and 33.2 %AP30 on held-out categories (Table 2b. For held-out categories, the difference to example-based search is marginal (33.2 + 37.6 %AP50). This result demonstrates that on Pascal VOC the model mostly follows shortcut and uses basic dataset statistics to solve the task.
whether they rely simple dataset statistics. Pascal VOC
In contrast, COCO represents a drastic increase in image complexity compared with Pascal VOC: contains, average, 2.9 categories and 7.3 instances per image. As expected, in this case the trivial baseline with uninformative references performs substantially worse than the example-base (training: 49.7 10.1 %a held-out: 22.8 4.4 %, Table 2). Thus, the added image complexity in COCO forces the model to use the reference image for classification but the small set of categories is not sufficient prevent memorizing the training categories.
search

---

Under review as a conference paper at ICLR 2022
COCO
Pascal VOC
Categories
Train
Held-Out
Train
Held-Out
Siam. Faster CNN
49.7
22.8
82.7
37.6
-empty Refs.
10.1
4.4
59.6
33.2
Table 2: On COCO and Pascal VOC there is clear performance gap (AP*)) between categories used during training (Train) and held-out categories (Held-Out). baseline getting a black image as reference which contains no information about the target category (- empty Refs.) performs surprisingly well on Pascal VOC but fails on COCO.
4.3 TRAINING ON MORE CATEGORIES REDUCES THE GENERALIZATION GAP
We now turn to our main hypothesis that increasing the number of cat- egories used during training could close the generalization gap iden- tified above. To this end we use Objects365 and LVIS, two fairly new datascts with 365 and 1203 categories, respectively (much more than the 20/80 in Pascal VOC/COCO). Indeed, training on these wider datasets improves the relative performance the held-out categories from 46% on COCO to 76% on 0bjects365 and up to 89% on LVIS (figure 1.. In absolute numbers this means going from 26.9 %Ap5u gap on COCO to a 4.6 %AP30 gap on Objects365 and a 3.5 %AP50 gap on LVIS (Table 3) in the one-shot setting. Increasing the number of references to five (5-shot) improves performance on all datasets but leaves relative performance unchanged (Table right columns).
Srewn Colegeticr Canspor
42
22.3
22.4
VOC
DECO
Oleeda225
LINS
Figure 4:
Performance on known and novel categories for different datasets.
This effect is not caused simply by differences between the datasets, as the following experiment shows: For both datasets (LVIS and Objects365), we train models on progressively more categories. When training on less than 100 categories (resembling training on COCO), a clear generalization gap is visible on both LVIS and Objects365 (figureß): leftmost data points). Increasing the number of training categories leads to better performance the held-out categories, while performance on the training categories stays the same (LVIS) or decreases (Objects365). The effect is the same in the 5-shot setting but with better baseline performance (figure A.1 in Appendix).
4.4 THE NUMBER OF CATEGORIES IS THE CRUCIAL FACTOR
The results so far show that increasing the number of categories used during training reduces the generalization gap and improves performance. However, this effect could also be caused by the fact that with more categories there is also more data available. Consider the situation where we train on 10% of the categories (90 the case of LVIS). As we sample these categories uniformly from the dataset, we use only approximately 10% of the total number of instances. To control for this confound, we created training sets that match the number of instances: in this case we use only 10% of the instances in the dataset but sample them uniformly from all 900 training categories.
Objects365
Objects365
30
30
30
25
30
25
training
subsample
15
held out cats.
10
15
all cats.
10
Categories during training
Categories during training
Fraction
Fraction
Figure 5: A. Experiment subsampling LVIS and Objects365 categories during training. When more categories are used during training performance on held-out categories (blue) improves while per- formance on the training categories (light blue) stays flat or decreases. B. Comparison of the per- formance on held-out categories if a fixed number of instances is chosen either from all categories (green) or from subset of categories (blue). Having more categories is more important than having more samples per category. (I-shot results, for 5-shot see figure A.I

---

Under review as a conference paper at ICLR 2022
COCO
1-shot Held-Out C.
5-shot Held-Out C.
Model
Backb.
Sched.
Train C.
Delta
Train C.
Delta
Siam. RetinaNet
R50
50.6
18.9
31.7
55.5
22.1
33.4
Siam. FRCNN
R50
1x
49.7
22.8
26.9
54.9
27.6
27.3
Siam. FRCNN
R50
3x
51.7
21.9
29.8
57.6
26.7
30.9
Siam. FRCNN
X101
56.4
23.5
32.9
61.9
28.6
33.3
IVIS
1-shot Held-Out C.
5-shot Held-Out
Model
Backh.
Sched.
Train C.
Delta
Train C.
Delta
Siam. RetinaNet
R50
1x
28.4
24.7
3.7
31.6
27.5
4.1
Siam. FRCNN
R50
31.5
28.0
3.5
37.0
33.0
4.0
Siam. FRCNN
R50
3x
32.7
28.7
4.0
38.2
33.5
4.7
Siam. FRCNN
X101
1x
35.4
31.3
4.1
41.4
36.3
5.1
Objects365
1-shot
5-shot Held-Out C.
Model
Backb.
Sched.
Train Cats.
Held-Out C.
Delta
Train C.
Delta
Siam. RetinaNet
R50
1x
19.7
14.5
5.2
23.4
17.2
6.2
Siam. FRCNN
R50
1x
19.4
14.8
4.6
25.7
19.9
5,8
Siam. FRCNN
R50
22.0
16.5
5.5
27.7
20.9
6.8
Siam. FRCNN
25.0
17.9
7.1
30.6
8.2
Table 3: Effect of a three times longer training schedule and larger backbone (ResNeXt-101 32x4d) model performance across datasets. While larger models and longer training times lead only minor improvements on held-out categories on COCO, they do have a larger effect on LVIS and Objects365.
The results can be seen in figure 5B. Our example from above with 10% of the data corresponds the leftmost datapoint in both plots. The model trained with more categories (green) clearly outperforms the model with more instances per category (blue). The same performance gap can be seen for any fraction of the data. Thus, for given budget of instances (labels) better to cover more categories than to collect as many samples per category as possible.
4.5 ONCE THE GENERALIZATION GAP CLOSED MORE POWERFUL MODELS BENEFIT NOVEL CATEGORIES
models indeed learn the distribution over categories, stronger models that can learn more powerful representations should perform better on known and novel categories alike. We test this hypothesis two ways: first, by replacing the standard ResNet-50 He al.] (2016) backbone with a more expressive ResNeXt-101 Xie et al. 2017); second, by using a three times longer training schedule.
The larger backbone does improve performance on the held-out categories on COCO (Table Instead the additional capacity is used to memorize the training categories, which is evident from performance on the training categories, but only small improvement (0.7 %AP50) on the held-out categories. In contrast, LVIS and Objects365 the gains the bigger backbone are not confined to the training categories but apply to the one-shot setting well. Only small difference remains Objects365 (3.0 %AP50 vs. 1.4 %AP50).
large improvement (6.7 %5p50)
Longer training schedules show the same pattern. For COCO, performance on the training categories improves while performance held-out categories even gets bit worse 3x schedule (Table B) In contrast, performance on LVIS and Objects365 improves for both training and held-out categories alike, suggesting that the models do overfit only the training categories.
4.6 RESULTS HOLD FOR DIFFERENT MODEL CONFIGURATION
To test if our findings apply single-stage detectors well, train and test Siamese RetinaNet COCO, LVIS and Objects365 (Table Results are very similar to Siamese Faster R-CNN. Siamese RetinaNet shows slightly larger generalization gap COCO (relative performance: Retina: 37%

---

Under review as a conference paper at ICLR 2022
Sports Ball
Dog
Dining Table
Elephant
Trained coco
ReeNet-S
Rey 101 32-4d Trained on LVIS
Figure 6: Predictions on COCO tend to be more accurate and cleaner when using a bigger backbone and training LVIS. Especially categories with more ambiguous references like sports ball or dining table the LVIS trained model is more precise. Additionally the ResNeXt backbone leads to "cleaner" results with less false positives.
I-shot
5-shot
Model
Backb,
Train Data
Train C.
Held Out C.
Train C.
Held-Our C.
Siam. Mask R-CNN"
R50
COCO
37.6
16.3
41.3
18.5
CoAE" *IT
R50
COCO
40.9
22.0
R50
COCO
47.5
24.3
Siam. RetinaNet
R50
COCO
50.6
18.9
55.5
22.1
Siam. Faster R-CNN
R50
COCO
49.7
22.8
54.9
27.6
Siam. Mask R-CNN
R50
COCO
51.9
22.9
57.9
27.8
Siam. Cascade R-CNN
R50
COCO
50.3
22.0
56.2
27.2
Siam. Faster R-CNN
X101 32x4d
COCO
56.4
23.5
61.9
28.6
Siam. Faster R-CNN
R50
LVTS
36.2
25.0
43.5
31.7
Siam. Faster R-CNN
X101 32x4d
LVIS
42.5
27.4
50.3
34.8
Table 4: Performance (APS0) COCO be improved by training LVIS. Siamese Mask R- CNN and Siamese Cascade R-CNN are identical to Siamese Faster R-CNN except for an additional mask head cascaded bbox heads. ("Michaelis al. (2018b), ** Hsiih et al. (2019). *** Chen et al. (2021)
vs. FRCNN: 46%) but results are very similar on LVIS (Retina: 87% FRCNN: 89%) and Objects365 (Retina: 74% vs. FRCNN: 76%).
Taken together we observe the same patterns for single- and two-stage detectors with different back- bones and learning rate schedules on two datasets (Objects365 and LVIS) for 1-shot and 5-shot evaluation. This suggests that our conclusions may extend to most object detection models and we can expect to significantly boost performance using the large toolboxes which exist for traditional object detection.
4.7 STATE-OF-THE-AR ON COCO USING LVIS
Using the insights from above, we now demonstrate state-of-the-art one-shot detection performance on COCO by training on large number of categories. We use LVIS and create four splits which leave out all categories that have a correspondence in the respective COCO split. As LVIS is re-annotation of COCO, this means that we expand the categories in the training set while training on the same set of images. Training with the more diverse LVIS annotations leads to notice- able performance improvement from 22.8 to 25.0 %AP50 which can be improved even further to 27.4 %0P50 by using the stronger 100 backbone, outperforming the previous best model by 5.4 %0 (Table 4). In relative terms that means going from 45% relative performance to 65% %, thus substantially outperforming the previous best method (55% relative performance Hsich et al. (2019) both in absolute and relative terms. Visual inspection the results (figure () shows cleaner predictions with less false positives especially for difficult reference images.
8

---

Under review as a conference paper at ICLR 2022
5 DISCUSSION
has long been assumed and recently shown (Sbai et al. 2020 Jiang et al. 2020; Fan et al. 2020) that training with more categories improves few-shot learning performance. However the question whether this is due to better overall model performance or better generalization has not been an- swered so far. Our results show that the underlying mechanism is an improvement in generalization from 45% relative performance on COCO to 89% on LVIS. The effect is consistent for different detectors, backbone architectures and training schedules which suggests that the effect will hold for almost any model. If this trend continues with more categories object detection that generalizes to any object within reach. This, however, does not mean that one-shot object detection "solved". There are at least three important steps to take:
First and foremost the performance of example-based object detectors has to improve significantly. Our experiments outline path forward, demonstrating that methods that profit general object de- tection transfer to novel categories when the generalization gap is closed. Secondly, we have to better understand the mechanisms that lead to the generalization gap. Our results indicate that one of the main reasons is a shortcut Geirhos et al. (2020) memorizing the training categories. That stronger models also perform better on novel categories with progressive closing of the gap is an indicator that the key issue was indeed overfitting. However more investigation will be required to determine which factors are important. Is it the sheer number of categories is it their diversity, granularity, frequency? Or is the main factor semantic relationship as results from [Saai et al. (2020) and Diang et al. (2020) suggest? Finally we have to find way to transfer this success smaller datasets with less categories. While we achieve new state-of-the-a on COCO the generalization gap there (69%) is still larger than on LVIS (89%).
5.1 FUTURE DATASETS SHOULD FOCUS ON THE DIVERSITY OF CATEGORIES.
Our findings have important implications for the design of future datasets. For the goal of gener- alization a broader range of categories is helpful at any dataset size (figure 5::: green curve above blue curve at any data fraction). while from certain point onward more examples per category lead to diminishing returns (figure 5B: green curve flattens out). At a time where few-shot and long-tail problems become more important in computer vision this suggest that future data collection and an- notation efforts should focus more on a broad set of categories and less on the number of instances for each of those categories.
An open question is, how broad datasets have to be. Despite being big step forward, training on LVIS still leaves a small generalization gap that widens when using stronger models. In other words: some amount of overfitting on the training categories remains. The good news is that we don't 't see a saturation (figure 5): dark blue curves still rise at the maximum number of categories) so further increasing the number of categories should reduce the remaining gap.
5.2 THE BIGGER PICTURE
Our insight that applying existing methods on larger and more diverse datasets can lead to unex- pected capabilities mirrored in other areas. This phenomenon has been observed time and again and was termed the "unreasonable effectiveness of data" [alevy et al. (2009); Sun et al. (2017) or the "bitter lesson" " [Sutton| (2019). It played a key role in the breakthrough of DNNs thanks to ImageNet Russakovsky et al. (2015); Krizhevsky et al. (2012) as well as recent results on game- play [Berner et al. (2019) or language modelling [Brown et al. (2020). Recently Kolesnikov et al.) (2099) and Radford et al. (202T achieve impressive results demonstrating strong performance at one-shot and zero-shot ImageNet classification. As in our study, simple methods (transfer learning in Kolesnikov et al. (2019) and unsupervised image captioning in Radford et al. (2021) large and diverse datasets led to results that are far better than what one would have expected: Achieving
ResNet performance with Kolesnikov al. (219) respectively zero Radford et al. (2021) anno- tated samples per class in their case; 89% relative performance on LVIS in our case. We hope that soon move from trying to solve few-shot learning towards using
by building this insight few-shot learning to solve other problems.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Christopher Berner. Greg Brockman, Brooke Chan, Vicki Cheung, Przemystaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, al. Dota 2 with large scale deep reinforcement learning. arXiv: /rXiv:/9/2.06666 2019.
Sujoy Kumar Biswas and Peyman Milanfar. One shot detection with laplacian object and fast matrix cosine similarity. TPAMI, 2015.
Tom B Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. (rXiv:2005.44165 2020.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and
Sergey 7agoruyko. End-to-end object detection with transformers. arXiv:2005. /2872, 2020.
Ding-Jie Jie Chen, He-Yen Hsieh, and Tyng-Luh Liu. Adaptive image transformer for one-shot object detection. CVPR, 2021.
Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy. and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv:1906.07155, 2019a.
Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. JCLR, 2019b.
Guneet S Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. A baseline for few-shot image classification. arXiv:1909.00 02729, 2019.
Mark Everingham, Luc Van Gool, Christopher K. Williams, John Winn, and Andrew Zisserman. The Pascal Visual Object Classes (VOC) Challenge. International Journal of Computer Vision, 2010.
Qi Fan, Wei Zhuo, Chi-Keung Tang, and Yu- Ju-Wing Tai. Few-shot object detection with attention-rpn and multi-relation detector. In CVPR, 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-Agnostic Meta-Learning for Fast Adapta- tion of Deep Networks. In /CML, 2017.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. arXiv:2004.07788. 2020.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation. In CVPR, 2014.
Agrim Gupta, Piotr Dollar, and Ross Girshick. LVIS: dataset for large vocabulary instance seg- mentation. In CVPR, 2019.
Alon Halevy, Peter Norvig, and Fernando Pereira. The unreasonable effectiveness of data. Intelligent Systems, 2009.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In ICCV, 2017. 2017.
Ting-I Hsieh, Yi-Chen Lo, Hwann-Tzong Chen, and Tyng-Luh Liu. One-shot object detection with co-attention and co-excitation. In NewrIPS, 2019.
Shuqiang Jiang, Yaohui Zhu, Chenlong Liu, Xinhang Song. Xiangyang Li, and Weiqing Min. Dataset bias in few-shot image recognition. woXiv:2008.07961 2020.
10

---

Under review as a conference paper at ICLR 2022
Bingyi Kang, Zhuang Liu, Xin Wang, Fisher Yu, Jiashi Feng, and Trevor Darrell. Few-shot object detection via feature reweighting. arXiv:1812.01866. 2018.
Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. Siamese Neural Networks for One-shot Image Recognition. JCML, 2015.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Large scale learning of general visual representations for transfer. arXiv: 2019.
Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Shahab Kamali, Matteo Malloci, Jordi Pont- Tuset, Andreas Veit, Serge Belongie, Victor Gomes, Abhinav Gupta, Chen Sun, Gal Chechik, David Cai, Zheyun Feng, Dhyanesh Narayanan, and Kevin Murphy. Openimages: A public dataset for large-scale multi-label and multi-class image classification. Dataset available from https://storage.
2017.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In NeurlPS, 2012.
Xiang Li, Lin Zhang, Yau Pun Chen, Yu- YunWing Tai, and Chi-Keung Tang. One-shot object detection without fine-tuning, arXiv:2005.0388 00009.,..99.... 2020.
Tsung-V Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common Objects in Context. In ECCV, 2014.
Tsung-Yr Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature Pyramid Networks for Object Detection. In CVPR, 2017a.
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017b.
Claudio Michaelis, Matthias Bethge, and Alexander S. Ecker. One-Shot segmentation in clutter. arXiv: 09597, 1903.0977
2018a.
Claudio Michaelis, Ivan Ustyuzhaninov, Matthias Bethge, and Alexander S. Beker. One-Shot in- stance segmentation. arXiv: 181.110 2018b.
Akihiro Nakamura and Tatsuya Harada.
Revisiting fine-tuning for few-shot learning.
19100000 16, 2019.
Gerhard Neuhold, Tobias Olimann, Samuel Rota Buld, and Peter Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In International Conference on Computer Vision (ICCV), 2017. URL http:://wwwwwpppwlp.m.m mapilaaayyyyla..aa..am.mmmmaaaaa wow./aaattt//tiitttttttt /visssssteestsssssssssssssssssss
Anton Osokin, Denis Sumin, and Vasily Lomakin. Os2d: One-stage one-shot object detection by matching anchor features. arXiv:2003.06880) 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. 1=XXi:2103.0000 2021.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NIPS, 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJCV, 2015.
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. arXiv:/807.05900 2018.
Othman Sbai, Camille Couprie, and Mathieu Aubry. Impact of base dataset design on few-shot image classification. In ECCV, 2020.
11

---

Under review as a conference paper at ICLR 2022
Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots. One-Shot Learning for Semantic Segmentation. BMVC, 2017.
Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In ICCV, 2019.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical Networks for Few-shot Learning. In NIPS, 2017.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable ef- fectiveness of data deep learning era. In ICCV, 2017.
Richard Sutton. The bitter lesson. Incomplete Ideas (blog), March, 2019.
Oriol Vinyals, Charles Blundell, Tim Lillicrap. Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In NIPS, 2016.
Xin Wang. Thomas Huang, Trevor Darrell, Joseph Gonzalez, and Fisher Yu. Frustratingly simple few-shot object detection. arXiv:2003.06957 2020.
Yu-Xior Xiong Wang, Deva Ramanan, and Martial Hebert. Meta-learning to detect rare objects. In ICCV, 2019.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. 2019.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In CVPR, 2017.
Xiaopeng Yan, Ziliang Chen, Anni Xu, Xiaoxi Wang, Xiaodan Liang, and Liang Lin. Meta r-cnn: Towards general solver for instance-level low-shot learning. In ICCV, 2019.
Tengfei Zhang, Yue Zhang, Xian Sun, Hao Sun, Menglong Yan, Xue Yang, and Kun Fu. Comparison network for one-shot conditional object detection, urXhv:1904.02317 2019.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017.
12

---

Under review as a conference paper at ICLR 2022
A
APPENDIX
A.1
ADDITIONAL FEW-SHOT RESULTS
We provide sive-shot results for the experiments in figure 5 figure A.1
Objects365
Objects366
40
40
AP50
30
osev
20
training
20
subsample
held-out cats.
cats.
10
10
Calegories duringg training
Categories during training
Figure A.I: Performing the experiments in figure 5 with five reference images (five-shot) leads to no qualitative difference.
13


Reviews:
[{"review_id": "km79LyTBuST", "reviewer": "Reviewer_wPts", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1.\tThe paper conducts many experiments to illustrate the impact of different factors on the performance of base classes and novel classes, including the number of training categories, the number of samples, the model capacity and the training time, etc. \n2.\tThe paper emphasizes that more categories are more beneficial to the task of FSOD, including shrinking the generalization gap, and improving the performance on novel classes. The authors suggest that the future datasets should focus on the diversity of categories.\n3.\tThe paper is well-written.\n\nWeaknesses:\n1.\tThe paper's major point of view has been observed in previous works, and it is very intuitive. For example, a similar point of view has already been pointed out in [1]. See section 5.4 “More Categories vs. More Samples?” and Table 7 in [1]. Moreover, a high-diversity dataset has been proposed and corresponding experiments have been carried to verify the importance of the number of categories. However, except for the related work section that mentions [1], the other parts of this paper do not mention [1], and do not compare with [1]. \n\n2.\tThe main point of the paper is not convincing: \na) The authors only conducted experiments on metric-based methods (called example-based visual search task in this paper) and ignored finetuning-based models. However, more and more finetuning-based models has been proposed, such as TFA[2]. The authors should validate their views in both types of methods, instead of just one of them.\nb) If there exists domain shift from training categories to testing categories, more training categories will not bring benefits. Many previous works performed COCO->VOC cross-domain experiments [3][4], where images from COCO are used for training and images from VOC are used for testing. Results show that the performance on VOC novel classes is worse than that when using base classes in PASCAL dataset, even COCO has more categories.\n\n3.\tThe contribution of the paper is limited. The paper does not propose a new dataset, nor does it propose a novel method, but uses existing datasets to draw a conclusion, and this conclusion was also drawn in the experiments of previous works. And this conclusion is only verified on the metric-based methods.\n\n\n[1] Fan, Qi, et al. \"Few-shot object detection with attention-RPN and multi-relation detector.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n[2] Wang, Xin, et al. \"Frustratingly Simple Few-Shot Object Detection.\" International Conference on Machine Learning. 2020.\n[3] Wang, Yu-Xiong, Deva Ramanan, and Martial Hebert. \"Meta-learning to detect rare objects.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.\n[4] Kang, Bingyi, et al. \"Few-shot object detection via feature reweighting.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.\n\nSome details:\n1.\tIn related work, the paper classifies [1] as one-shot object detection, which conflicts with the title of [1] and the previous classification method.\n2.\tIn the introduction, the authors claim that one-shot object detection problem can be decomposed into three subproblems. However, there are many methods that are not based on RPN[3][4], where class-agnostic object proposal is not needed.. Therefore, making such claim is inappropriate.\n", "summary_of_the_paper": "This paper shows that the key to reduce the generalization gap between base classes and novel classes is to increase the number of training categories, instead of training samples. The authors did many experiments on four existing datasets (PASCAL, COCO, Objects365, and LVIS) with Siamese Faster R-CNN to verify this point. Experiments show that with more categories in the training set, the generalization gap will be nearly closed. Finally, the author proposes that future data sets should focus on the diversity of categories.", "summary_of_the_review": "Considering that the contribution of this article does not meet the requirements of ICLR, I do not recommend to accept it.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cxDNzA3z93y", "reviewer": "Reviewer_b9Xw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths:\n- State-of-the-art performance is achieved on COCO one-shot detection benchmark by training the detector on LVIS.\n- This paper is well written. The presentation is easy to follow.\n\n# Weaknesses:\n- The study on the effect of category number in machine learning tasks has been conducted in previous works, e.g., Sun et al. (2017), which weakens the novelty of this paper.\n- A major concern is on the split of train categories and held-out categories. As the category number increases, the train categories have higher probability to contain more categories semantically related to the held-out categories. With more semantically related categories to train on, it is reasonable that the detector can have better performance on held-out. For example, adding more cat species in the train categories may better help improve the detection accuracy of a held-out “ragdoll” category than adding more vehicle categories. Therefore, the large number of categories may not be the root cause of reduced generalization gap, but instead semantic relation may be the key. Currently, it seems there is one single fixed train/held-out split in all the experiments. Have the authors tried with other split? For example, does adding more unrelated categories still improve the performance?\n", "summary_of_the_paper": "This paper studies the effect of category number in the one-shot object detection task. In the testing of one-shot detection, there exists a performance gap between the base (training) classes and held-out classes. It is claimed that this performance gap can be largely closed by increasing the number of categories used for training. And the number of categories is more crucial than the number of samples per category. Experiments are conducted on VOC, COCO, Object365, and LVIS using a Siamese-style detector to verify the claims.", "summary_of_the_review": "On the one hand, this paper is well written and produces sota performance. However, I am concerned about the novelty and the root cause for the improvements. I am leaning towards reject for now and would like to see the response from authors.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6lrlMcDdWsg", "reviewer": "Reviewer_djNN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- Strength\n    - The paper is generally well-written and well-motivated.\n    - I appreciate the authors for validating the fact that PASCAL is not suitable for one-shot detection, given the task simply becomes salient object detection.\n\n- Weakness:\n    - The contribution of the paper is limited.\n    - The paper simply raises the discovery that training more categories can be beneficial for generalisation on one-shot detection, which is probably not that surprising, given the architecture used is based on matching, and more importantly, while increasing the number of training categories, it becomes more likely that the training and held-out categories are subcategories of one root.\n\n    - As for architecture, the authors use Siamese Faster R-CNN, similar to tracking tasks, and class-agnostic countings.\n", "summary_of_the_paper": "- The paper considers the problem of one-shot object detection, meaning, the model is asked to detect unseen categories, based on only one provided a template.\n\n- The main discovery made by the authors is that, to generalise better, the model should be trained with data from as many categories as possible, given the same budget on number of samples for training.\n\n- Architecture-wise, a Siamese Faster R-CNN is adapted. ", "summary_of_the_review": "- Overall, I think the discovery is well-made, it shines light on one perspective, that is, using more diverse training data can be beneficial for generalisation,  however, it misses the other element on, how can we further boost the generalisation from an architecture or model design perspective.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3ThqMbQLunr", "reviewer": "Reviewer_fCN3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. The paper designs a smart experiment to demonstrate that PASCAL VOC is too easy/biased for evaluating one-shot object detection algorithms. The experiment shows that assigning black reference images does not degrade the performance on PASCAL VOC too much. \n2. The primary conclusion that increasing the number of training categories significantly influences one-shot performance is clearly stated. Experiments for the major conclusion are valid. (Section 4.3, 4.4, Figure 4,5). \n3. This paper improves stoa one-shot performance on COCO, though by training on LIVS. \n\n------------------------\nWeakness:\n\n1. By training on LIVS, this paper obtains state-of-the-art performance. But training on LIVS may not be fair compared with others. \n\n2. The major conclusion is not very non-trivial because previous work has shown that increasing the number of categories for pretraining can obtain better representations, like [1](figure 1) (It is not the exact same conclusion and scenarios, but it is similar.).  Meanwhile, similar conclusions like increasing the number of categories for recognitions increase the learning difficulty dramatically. (figure 2 in [1] ) \n\n3. There is some sort of overclaiming in some of this paper's sentences regarding the performance. Like in the introduction: 'This generalization gap can be almost closed by increasing the number of categories used for training.'  But the performance of one-shot detectors mentioned in table 4 is still below 30 AP50. \n\nOne thing I am curious about is that how training on Objects365 can affect the one-shot performance on COCO compared with current state-of-the-art results. \n\n[1] Mahajan D, Girshick R, Ramanathan V, et al. Exploring the limits of weakly supervised pretraining[C]//Proceedings of the European conference on computer vision (ECCV). 2018: 181-196.", "summary_of_the_paper": "This paper studies one-shot 2D object detections. The major conclusion is that when keeping the number of training images fixed, increasing the number of training categories can significantly increase the one-shot performance.  This paper demonstrated this conclusion empirically by doing controlled experiments on COCO, Objects365, and LIVS. \n\nInspired by this observation, this paper improves the state-of-the-art one-shot detection performance on COCO from 22.0 to 27.5 AP50 by training on LIVS. \nOther two related conclusions in this paper:\n- PASCAL VOC is not suitable for evaluating one-shot object detection algorithms. The reason may be that the number of instances per image is low(2.9 ins/img on average). The algorithm only needs to recognize foreground objects rather than objects of target categories. \n- Only when the training data is challenging enough, increasing the model size and training time can help improve one-shot performance.  \n", "summary_of_the_review": "I will not recommend accepting the paper. Mainly because of the limited contribution. I will vote for strong reject if I find similar references demonstrating similar conclusions in one-shot detection scenarios.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Summary: The given work studies one-shot object detection, and demonstrates an array of experiments that show that by increasing the number of categories in training data, the model can get better at one-shot detection. 

Pros:
- Well written
- Presents many experiments that are interesting
- Improves SOTA one-shot performance on COCO by using more data.

Cons:
- Authors did not test their claims on a variety of model architectures
- Similar conclusions have been made in prior art.
- Conclusions are intuitive. As more categories are added, the likelihood of reducing semantic dissimilarity to a new novel category is quite high.
- Overall contribution is currently limited.

Reviewers are unanimous in their decision. Authors did not alleviate concerns of reviewer. AC recommends authors take all feedback into consideration and submit to another venue or workshop.
Decision: Reject