Closing the Generalization Gap in One-Shot Object Detection
Authors: Claudio Michaelis
Matthias Bethge
Alexander S Ecker
Venue:  (2021)
Keywords: One-Shot Learning
Few-Shot Learning
Object Detection
One-Shot Object Detection
Generalization

Abstract: Despite substantial progress in object detection and few-shot learning, detecting objects based on a single example - one-shot object detection - remains a challenge. A central problem is the generalization gap: Object categories used during training are detected much more reliably than novel ones. We here show that this generalization gap can be nearly closed by increasing the number of object categories used during training. Doing so allows us to beat the state-of-the-art on COCO by 5.4 %AP50 (from 22.0 to 27.5) and improve generalization from seen to unseen classes from 45% to 89%. We verify that the effect is caused by the number of categories and not the amount of data and that it holds for different models, backbones and datasets. This result suggests that the key to strong few-shot detection models may not lie in sophisticated metric learning approaches, but instead simply in scaling the number of categories. We hope that our findings will help to better understand the challenges of few-shot learning and encourage future data annotation efforts to focus on wider datasets with a broader set of categories rather than gathering more samples per category.

Under review as a conference paper at ICLR 2021
CLOSING THE GENERALIZATION GAP IN ONE-SHOT OBJЕCT DETECTION
Anonymous authors
Paper under double-blind review
ABSTRACT
Despite substantial progress in object detection and few-shot learning, detecting objects based on a single example one-shot object detection - remains a chal- lenge. A central problem is the generalization gap: Object categories used during training are detected much more reliably than novel ones. We here show that this generalization gap can be nearly closed increasing the number of object cat- egories used during training. Doing so allows us to beat the state-of-the-art on COCO by 5.4 %AP (from 22.0 27.5) and improve generalization from seen to unseen classes from 45% to 89%. We verify that the effect is caused by the number of categories and not the amount of data and that it holds for different models, backbones and datasets. This result suggests that the key to strong few- shot detection models may not lie in sophisticated metric learning approaches, but instead simply in scaling the number of categories. We hope that our findings will help to better understand the challenges few-shot learning and encourage future data annotation efforts to focus wider datasets with broader set of categories rather than gathering more samples per category.
A
Search Task
B Generalization Novel Categories
82.7
Known Categories Novel Categories
60
AP50
40
37.6
31.5
22.8
20
19.4
Pascal VOC (20)
COCO (80)
Object:365 (365)
LVIS (1203)
Figure 1: A. One-shot object detection: Identify and localize all objects of a certain category within a scene based on (single) instructive example. B. Increasing the number of categories used during training reduces the generalization gap to novel categories presented at test time (in parenthesis: number categories each dataset).
INTRODUCTION
It's January 2021 and your long awaited household robot finally arrives. Equipped with the latest "Deep Learning Technology", it can recognize over 21,000 objects. Your initial excitement quickly vanishes as you realize that your casserole is not one of them. When you contact customer service they ask the casserole so they can fix this. They tell you that the fix will be some time, though, they need to collect about thousand images of casseroles to retrain the neural network. While you are making the call your robot knocks over the olive oil because the steam coming from the pot of boiling water confused it. You start filling out the return form...
to send some pictures of

---

Under review as a conference paper at ICLR 2021
While not 100% realistic, the above story highlights important obstacle towards truly autonomous agents such as household robots: such systems should be able to detect novel, previously unseen objects and learn to recognize them based on (ideally) single example. Solving this one-shot object detection problem can he decomposed into three subproblems: (1) designing a class-agnostic object proposal mechanism that detects both known and previously unseen objects; (2) learning a suitably general visual representation (metric) that supports recognition of the detected objects; (3) continuously updating the classifier accommodate new object classes or training examples of existing classes. In this paper, we focus on the detection and representation learning part of the pipeline, and we ask: what does take to learn a visual representation that allows detection and recognition of previously unseen object categories based on a single example?
We operationalize this question using an example-based visual search task (Fig. TD that has been in- vestigated before using handwritten characters (Omniglot: Michaelis et al. (201 8a)) and real-world image datasets (Pascal VOC, COCO: Michaelis et al. (2018b); HHiieh et al. (2019): Zhang et al. 2019 Fan et al. (2020); Li et al. (2020)) Our central hypothesis is that scaling up the number of object categories used for training should improve the generalization capabilities of the learned representation. This hypothesis is motivated by the following observations. On (cluttered) Om- niglot (Michaelis et al., 2018a), recognition of novel characters works almost as well as for char- acters seen during training. In this case, sampling enough categories during training relative the visual complexity of the objects is sufficient to learn a metric that generalizes to novel categories. In contrast, models trained on visually more complex datasets like Pascal VOC and COCO exhibit a large generalization gap: novel categories are detected much less reliably than ones seen during training. This result suggests that on the natural image datasets, the number of categories is too small given the visual complexity of the objects and the models retreat to shortcut (Geirhos et al.) 2020 memorizing the training categories.
To test the hypothesis that wider datasets improve generalization, we increase the number of object categories during training by using datasets (LVIS, Objects365) that have larger number of cate- gories annotated. Our experiments support this hypothesis and suggest the following conclusions:
The generalization gap between training and novel categories is key problem in one-shot object detection.
This generalization gap can be almost closed by increasing the number of categories used for training: going from 80 classses in COCO to 1200 in LVIS improves relative perfor- mance from 45% to 89%.
The number of categories, not the amount of data, is the driving force behind this effect.
Closing the generalization gap allows us to use established methods from the object detec- tion community (like e.g. stronger backbones) to make further progress.
We use these insights to improve state-of-the-art performance on COCO by 5.4 %AP50 (from 22 %AP50 to 27.5 %AP20) using annotations from LVTS.
2 RELATED WORK
Object detection Object detection has seen huge progress since the widespread adoption of DNNs
(Girshick et al., 2014 Ren al. 2015 He et al., 2017: Lin et al., [017a|] Chen et al. 2019a| Wu ct al. 2019b Carion et al. 2020). Similarly the number of datasets has grown steadily, fueled by the
importance this task has for computer vision applications (Everingham et al. 2010; Russakovsky
et al. 2015 Lin et al. 2014 Zhou et al. 2017 Neuhold et 2017 Krasin et al. 2017 Gupta et al.)
2019 Shao al.) 2019). However most models and datasets focus on scenarios where abundant examples per category are available.
Few-shot learning The two most common approaches to few-shot learning have been, broadly
speaking, based on metric learning (Koch et al. 2015 Vinyals et al. 2016
Snell et al. (017) and
meta learning: Learn a good way to learn a new task (Finn et al., combinations thereof (Sun et al. 2019). However, recent work has shown that much simpler ap- proaches based on transfer learning achieve competitive performance (Chen et al. 2019b; Nakamura & Harada, 2019; Dhillon al. 2019). A particularly impressive example of this line of work is Big Transfer (Kolesnikov et al. 2019), which uses transfer learning from a huge architecture trained on huge dataset to perform one-shot ImageNet classification.
2017: Rusu et al. 2018), or
2

---

Under review as a conference paper at ICLR 2021
Few-shot & one-shot object detection Recently, several groups have started to tackle few-shot learning for object detection. Two training and evaluation paradigms have emerged. The first is inspired by continual learning: incorporate set of new categories with only few labeled images per category into an existing classifier (Kang et al. 2018 Yan et al. 2019 Wang et al.
2019 2020).
The second one phrases the problem an example b-baded visual search: detect objects based
single example image (Fig. [left: Michaelis et al. 20188 Hsich et al. 2019, Zhang et al. 2019 Fan
ct al. 2020 Li et al. 2020). We refer to the former (continual learning) as few-shot object detection,
since typically 10-30 30 images are used for experiments on COCO. In contrast, we refer to the latter (visual search) as one-shot object detection, since the focus is on the setting with a single example. In the present paper we work with this latter paradigm, since it focuses on the representation learning part of the problem and avoids the additional complexity of continual learning.
Methods for one-shot object detection Existing methods for one-shot object detection employ a combination of a standard object detection architecture with siamese backbone and various forms of feature attention and concatenation on the backbone output or in the heads (Biswas & Milan-
[ar 2015] Michaelis et al.
2018b; Hsich et al.] 2019 Zhang et al. 2019 Fan et al.] 2020] [Oskin Spatially aware similarity measures (Li et al. 2020) or transformations
al. [2020]]u Li et al. 2020
Osokin et al.] (020) improve recognition in cases where the pose of the reference objects differs from that of the detected object. We here use one of the most straightfor- ward models, Siamese Faster R-CNN (Michaelis et al. 2018b), to demonstrate that a change of the training data rather than the model architecture is sufficient substantially reduce the generalization gap between known and novel categories.
(Biswas & Milanfar 2015
Related tasks A number of related pieces of work propose approaches slightly different example-based search tasks. Examples include one-shot segmentation using handwritten characters (Michaelis et al. 2018a), natural textures (Ustyuzhaninov et al. 2018) and natural images (Shaban et 2017). In addition, several groups have suggested one-shot and few-shot detection tasks with slightly different focus and protocols (Dong et al. 2018 Chen et al. 2018: Schwartz et al. 2019 Wu et al. 2019a), including episodic evaluation (Wu et al. 2019a), transfer across datasets (Chen et al., 2018 and fine-gres fine-grained detection (Schwartz et al. 2211)). Also closely related are instance retrieval (Tolias et al. 2016) and co-segmentation (Rother et al. 2006; Hsu et al. 2019). The key difference of our work is that we do not propose a new architecture, but instead investigate the relationship between the number of categories used during training and the generalization to novel categories.
Number of categories in few-shot learning Most of the few-shot learning literature focuses on developing new methods for existing benchmarks. The inffuence of the training data was mostly observed indirectly, e.g. through better performance on datasets with more categories such as fieredI vs. minilmageNet. Two concurrent studies report that more categories help few-shot object detection (Fan et al. (2020) and investigate the influence of data diversity, image complexity, intra- and inter-category diversity and other factors on few-shot classification (Jiang et al. 2020). Both publications are consistent with out results that the number of categories is a key factor for improving few-shot performance.
3 EXPERIMENTS
Models We mainly use Siamese Faster R-CNN, a one-shot detection version of Faster R-CNN (Ren et al. 2015) similar to Siamese Mask R-CNN (Michaclis et al. 2018b). Briefly, it consists of a feature extractor, matching step and a standard region proposal network and bounding box head (Fig. 2 The feature extractor (called backbone in object detection) is a standard 55 with feature pyramid networks (He al. 2016 Lin et al., 2017a) which is applied to the image and reference with weight sharing. In the matching step the reference representation is compared to the image representation a sliding window approach by computing a feature-wise I.1 difference. The resulting similarity encoding representation is concatenated to the image representation and passed on to the region proposal network (RPN). The RPN proposes a set of bounding boxes which po- tentially contain objects. These boxes are then classified containing class or something else (other object or background). Box coordinates are refined by bounding box regression and overlapping boxes are removed using non-maximum suppression.
object from the reference
We additionally developed Siamese RetinaNet, a single-stage detector based on RetinaNet (Lin et al.) 2017b). The feature extraction and matching steps are identical to Siamese Faster R-CNN, but it uses
3

---

Under review as a conference paper at ICLR 2021
Siamese Faster R-CNN
Siamese Backbone
Matching
RPN
ResNet 50
CLS BBOX
ResNet 50
IF
L1
Heads
IF= Features, Pointwise Difference,
Region Proposal Network, Classifter, Bounding Box Regressor
Figure Siamese Faster R-CNN
the unified RetinaHead to jointly propose and classify bounding boxes. To counter the effect of too many negative samples, the classifier is trained with focal loss Lin et al. (2017b).
Training Evaluation During training a reference category is randomly chosen for every image by picking category with at least instance in the image. A reference is retrieved by randomly selecting one instance from this category in another image and tightly cropping it. The labels for each bounding box are changed 0 depending whether the object is from the reference category or not. Annotations for objects from the held-out categories are removed from the dataset before training. At test time a similar procedure chosen but instead of picking one category for each image, all categories with at least one object in the image are chosen Michaelis et al. (2018b) and one (I-shot) or five (5-shot) reference images are provided. Predictions are assigned their corresponding category label and evaluation is performed using standard tools and metrics.
Implementation We implemented Siamese Faster R-CNN and Siamese RetinaNet in mmdetec- tion v1.0re (Chen et al.) 2019a),, which improved performance by more than 30% over the original paper (Table 4 Michaclis et al.) 2018b). We keep all hyperparameters the same in the standard Faster R-CNN implementation of mmdetection (which achieves 36.4% mAP/58.44 4% AP50 on reg- ular COCO). Due resource constraints reduce the number of samples per epoch to 120k for Objects365.
Datasets We use the four datasets shown in Table I COCO Lin et al. 2014). Objects365 (Shao al. 2019). LVIS (Gupta et al. 2019) and Pascal VOC Everingham al. 2010). We use standard splits and test on the validation sets except for Pascal VOC where we test on the 2007 test set. Due to resource constraints, we evaluate Objects365 on a fixed subset of 10k images from the validation set. Following common protocol (Michaelis et al. [0088]] Shaban et al. 2017) we split the categories in each dataset into four splits using every fourth category as hold-out set and the other 3/4 categories for training. So on Pascal VOC there are 15 categories for training in each split, on COCO there are 60, on Objects365 274 and on LVIS 902. We train and test four models (one for each split) and report the mean over those four models, performance is always measured on all categories. Computing performance in this way across all categories is preferable to using a fixed subset as some categories may be harder than others. During evaluation, the reference images chosen
Dataset
Version
Classes
Images
Instances
Ins/Img
Cls/Img
Thr.
Pascal VOC
07+12
20
8,000
23,000
2.9
1.6
/
COCO
2017
80
118,000
860,000
7.3
2.9
Y
LVIS
1,203
100,000
1,270,000
212.8*
23.6*
Objects365
v2
365
1,935,000
28,000,000
14.6
6.1
Table 1: Dataset comparison. Thr. Throughoutly annotated: every instance of every class is
annotated in every image. "LVIS has potentially more objects and categories per image than are annotated due the non-exhaustive labeling.
4

---

Under review as a conference paper at ICLR 2021
Skatsboard
Elephant
Parson
coco
Horse
Television Set
Goat
Shopping Bag
LIPtS
Bottle
Fire Truck
Duck
Fork
O365
Figure 3: Example predictions on held-out categories (ResNet-50 backbone). The left three columns show success cases. The rightmost column shows failure cases in which objects are overlooked and/or wrongfully detected.
COCO
Pascal VOC
Model
Train Cats.
Held- Cats.
Train Cats.
Held-Out Cats.
Siamese Faster R- R-CNN
49.7
22.8
82.7
37.6
empty Ref's.
10.1
4.4
59.6
33.2
On COCO and Pascal VOC there is clear performance gap (AP5)) between categories used during training (Train Cats.) and held-out categories (Held-Out Cats.). A baseline getting black image as reference which contains no information about the target category (- empty Refs.) performs surprisingly well on Pascal VOC but fails COCO.
Table 2:
randomly. We therefore the evaluation five times, reporting the average AP50 over splits. The 95% confidence intervals for the average 5 below +0.2%AP for all experiments.
RESULTS
4.1
GENERALIZATION GAP ON COCO AND PASCAL VOC
We start by showing that objects of held-out categories detected less reliably COCO and Pascal VOC. On both datasets, Siamese Faster R-CNN shows strong signs of overfitting to the train- ing categories (Table 2: performance is much higher than for categories held-out during training
(COCO: 49.7 + 22.8 %0 Pascal VOC: 82.7 + 37.6 %AP50). We refer to this drop in per- formance the generalization gap. This result consistent with the literature; Hsieh et al. (011)) the previous state-of-ther of-the-art art report performance dropping 40.9 - 22.0 %AP*0 on COCO (see Table 4 below). Some newer models reportedly close the gap on Pascal VOC (Zhang et al., 2019 Hsieh et al. 2019, Li et al. 2020); we will discuss Pascal VOC further in the next section. Example predictions show good localization (bounding boxes) even for unknown objects in cluttered scenes
while classification errors make up the majority of mistakes (Fig. 3)
4.2 PASCAL VOC IS TOO BASY TO EVALUATE ONE-SHOT OBJECT DETECTION MODELS
Having identified this large generalization gap, we ask whether the models have learned a useful metric for one-shot detection at all whether they rely simple dataset statistics. Pascal VOC
5

---

Under review as a conference paper at ICLR 2021
contains, average, only 1.6 categories and 2.9 instances per image. In this case, simply detecting all foreground objects may be a viable strategy. To test how well such a trivial strategy would per- form, provide the model with uninformative references (we all-black images). Interestingly, this baseline performs very well, achieving 59.6 %AP50 on training and 33.2 %AP30 on held-out categories (Table 2) For held-out categories, the difference to example-based search is marginal (33.2 + 37.6 )AP3)). This result demonstrates that on Pascal VOC the model mostly follows a shortcut and uses basic dataset statistics to solve the task.
In contrast, COCO represents a drastic increase in image complexity compared with Pascal VOC: it contains, on average, 2.9 categories and 7.3 instances per image. As expected, in this case the trivial baseline with uninformative references performs substantially worse than the example-based search (training: 49.7 - 10.1 %AP50, held-out: 22.8 + 4.4 %AP30; Table 2). Thus, the added image complexity forces the model to indeed rely on the learned metric for identifying matching objects, but this metric does not generalize well.
4.3 TRAINING ON MORE CATEGORIES REDUCES THE GENERALIZATION GAP
We now turn to our main hypothesis that increasing the number of categories used during training could close the generalization gap identified above. To this end we use Objects365 and LVIS, two fairly new datasets with 365 and 1203 categories, respectively (much more than the 20/80 in Pascal VOC/COCO). Indeed, training on these wider datasets improves the relative performance on the held-out categories from 46% on COCO to 76%
100
9556
Helld-Out/raining Performance
80
7651
40
20
Pascal coco (80)
(20)
(365)
Omniglot
on Objects365 and up to 89% on LVIS (Fig. 4 In ab-
solute numbers this means going from 26.9 %AP30 gap on COCO to a 4.6 %AP50 gap on Objects365 and 3.5 %AP50 gap on LVIS (Table in the one-shot set- ting. Increasing the number of references to five (5- shot) improves performance on all datasets but leaves relative performance unchanged (Table 3. right columns).
Figure 4: Relative performance grows with the number of categories but stays lower than that of Siamese- U-Net on Clut- tered Omniglot (Michaelis et al. 2018a).
This effect not caused simply by differences between the datasets, as the following experiment shows. For both datasets (LVIS and Objects365), we train models on progressively more categories. When training on less than 100 categories (resembling training on COCO), a clear generalization gap is visible on both LVIS and Objects365 (Fig. 5AA light blue vs. dark blue). Performance on the held-out categories increases with the number of training categories, while performance on the training categories stays the same (LVIS) decreases (Objects365). The same effect can be seen in the 5-shot setting but with better baseline performance (Fig. A.2 in Appendix).
Objects365
Objects365
30
30
30
25
30
25
training
subsample
15
held out cats.
10
15
all cats.
10
Categories during training
Categories during training
Fraction
Fraction
Figure 5: A. Experiment subsampling LVIS and Objects365 categories during training. When more categories are used during training performance on held-out categories (blue) improves while per- formance on the training categories (light blue) stays flat or decreases. B. Comparison of the per- formance on held-out categories if a fixed number of instances is chosen either from all categories (green) or from subset of categories (blue). Having more categories is more important than having more samples per category. (I-shot results, for 5-shot see Appendix Fig. A.2)

---

Under review as a conference paper at ICLR 2021
COCO
1-shot
5-shot Held-Out C.
Model
Backb.
Sched.
Train C.
Held-Oul C.
Delta
Train C.
Delta
S ReRiinaa
R50
1x
50.6
18.9
31.7
55.5
22.1
33.4
S-FRCNN
R50
1x
49.7
22.8
26.9
54.9
27.6
27.3
S-FRCNN
R550
3x
51.7
21.9
29.8
57.6
26.7
30.9
S-FRCNN
X101
1x
56.4
23.5
32.9
61.9
28.6
33.3
LVIS
1-shot
5-shot Held-Out C.
Model
Backb.
Sched.
Train C.
Held-Out C.
Delta
Train
Delta
S-Retina
R.50
1x
28.4
24.7
3.7
31.6
27.5
4.1
S-FRCNN
R50
31.5
28.0
3.5
37.0
33.0
4.0
S-FRCNN
R50
3x
32.7
28.7
4.0
38.2
33.5
4.7
S-FRCNN
X101
1x
35.4
31.3
4.1
41.4
36.3
5.1
Objects365 1-shot
5-shot Held-Out C.
Model
Backb.
Sched.
Train Cats.
Held-Out C.
Delta
Train C.
Delta
R-Retina
R50
1x
19.7
14.5
5.2
23.4
17.2
6.2
S-FRCNN
R50
1x
19.4
14.8
4.6
25.7
19.9
5.8
S-FRCNN
R50
22.0
16.5
5.5
27.7
20.9
6.8
S-FRCNN
X101
25.0
17.9
7.1
30.6
22.4
8.2
Table Effect a three times longer training schedule and larger backbone (ResNeXt-101 32x4d) on model performance across datasets. While larger models and longer training times lead to no or only minor improvements on held-out categories on COCO, they do have larger effect on LVIS and Objects365.
4.4 THE NUMBER CATEGORIES THE CRUCIAL FACTOR
The results far show that increasing the number of categories used during training reduces the generalization gap and improves performance. However, this effect could also be caused by the fact that with more categories there is also more data available. Consider the situation where we train on 10% of the categories (90 in the case of LVIS). As we sample these categories uniformly from the dataset, we use only approximately 10% of the total number of instances. To control for this confound, we created training sets that match the number of instances: in this case we use only 10% of the instances the datasct but sample them uniformly from all 900 training categories.
The results can be seen in Fig. 5B. Our example from above with 10% of the data corresponds the leftmost datapoint both plots. The model trained with more categories (green) clearly outperforms the model with more instances per category (blue). The same performance gap can be seen for any fraction the data. Thus, for a given budget of instances (labels) it better to cover more categories than to collect many samples per category possible. We will discuss the implications of this result in Section 5.1
4.5 ON LARGER DATASETS STANDARD TRICKS BENEFIT KNOWN AND NOVEL CATEGORIES (ALMOST) ALIKE
models indeed learn the distribution over categories, stronger models that can learn more powerful representations should perform better on known and novel categories alike. We test this hypothesis two ways: first, by replacing the standard ResNet-50 (He et al., (016) backbone with a more expressive ResNeXt-101 (Xic et al. 2017); second, by using three times longer training schedule.
The larger backbone does not improve performance on the held-out categories on COCO (Table Instead the additional capacity is used to memorize the training categories, which is evident from the large improvement (6.7 %% in performance on the training categories, but only a small improvement (0.7 %Ap50) the held-out categories. In contrast, LVIS and 0bjects365 the gains the bigger backbone are not confined to the training categories but applies to the one-shot setting to much larger extent than COCO.

---

Under review as a conference paper at ICLR 2021
Sports Ball
Dog
Dining Table
Elephant
Trained on COCO
ReaNet- 50
ReeNe 101 32x4d
Trained LiVS
Figure 6: Predictions on COCO tend to be more accurate and cleaner when using a bigger backbone and training LVIS. Especially categories with more ambiguous references like sports ball or dining table the LVIS trained model is more precise. Additionally the ResNeXt backbone leads to "cleaner" results with less false positives.
Longer training schedules show the same pattern of results the larger backbone. For COCO, performance on the training categories improves while performance on held-out categories even gets bit worse a 3x schedule (Table 3) In contrast, performance on LVIS and Objects365 improves for both training and held-out categories alike, suggesting that the models do not overfit only the training categories. Note that do observe a slightly larger improvement on the training categories than on the held-out ones; we will come back to this point in Section 5.1
4.6 RESULTS HOLD FOR DIFFERENT MODEL CONFIGURATIONS
To test if our findings apply to single-stage detectors as well, we train and test Siamese RetinaNet on COCO. LVIS and Objects365 (T Results very similar to Siamese Faster R-CNN. Siamese
RetinaNet shows a slightly larger generalization gap on COCO (relative performance: Retina: 37%
vs. FRCNN: 46%) but results very similar LVIS (Retina: 87% FRCNN: 89%) and
Objects365 (Retina: 74% vs. FRCNN: 76%).
Taken together results hold for single- and two-stage detectors with different backbones and learning rate schedules on two datasets (Objects365 and LVIS) for 1-shot and 5-shot evaluation. These results suggests that our conclusions may extend most object detection models and we can expect to significantly boost performance using the large toolboxes which exist for traditional object detection, A demonstration on COCO is described in the following section.
4.7 STARE-OF-THE-ART ONE-SHOT DETECTION COCO USING LVIS ANNOTATIONS
Using the insights from above, now demonstrate state-of-the-art one-shot detection performance on COCO by training a large number of categories. We use LVIS and create four splits which leave out all categories that have correspondence the respective COCO split. As LVIS a re- annotation of COCO, this means that we expand the categories in the training set while training on the same set of images.
Training with the more diverse LVIS annotations leads to a noticeable performance improvement from 22.8 to 25.0 %0P35 which can be improved even further to 27.4 %AP50 by using the stronger ResNe 101 backbone, outperforming the previous best model by 5.4 %AP50 (Table In relative terms that means going from 45% relative performance to 65% %, thus substantially outperforming the previous best method (55% relative performance Hsich al. (2019)) both in absolute and relative terms. Visual inspection of the results (Appendix Fig. (6) shows that the improved model generates cleaner predictions with less false positives especially for difficult reference images.
8

---

Under review as a conference paper at ICLR 2021
1-shot
5-shot
Model
Backb,
Train Data
Train C.
Held-Out C.
Train
Held-Out
Siam. Mask R-CNN'
R50
COCO
37.6
16.3
41.3
18.5
CoAE!
R50
COCO
40.9
22.0
Siam. RetinaNet
RS0
COCO
50.6
18.9
55.5
22.1
Siam. Faster R-CNN
COCO
49.7
22.8
54.9
27.6
Siam. Mask R-CNN
R550
COCO
51.9
22.9
57.9
27.8
Siam. Cascade R-CNN
R50
COCO
50.3
22.0
56.2
27.2
Siam. Faster R- R-CNN
X101 32x4d
COCO
56.4
23.5
61.9
28.6
Siam. Faster R-CNN
R50
LVIS
25.0
43.5
31.7
Siam. Faster R-CNN
X101 32x4d
LVIS
42.5
27.4
50.3
34.8
Table 4: Performance (APS)) on COCO can be improved by training on LVIS. Siamese Mask R- CNN and Siamese Cascade R-CNN are identical to Siamese Faster R-CNN except for an additional mask head or cascaded bbox heads. (MMichaelis et al. 2018b), i Hsieh et al. (2019))
5 DISCUSSION
5.1 FUTURE DATASETS SHOULD FOCUS ON THE DIVERSITY OF CATEGORIES.
Our findings have important implications for the design of future datasets. A broader range of cate- gories is helpful at any dataset size (Fig. [ more examples per category lead to diminishing returns in terms of generalization. This result is particularly evident in the experiment where subsample instances versus categories (Fig. [B):: using all the available categories, we need only z 60% the data on LVIS to achieve optimal performance. On Objects365 this effect is even performance is already saturated. We therefore suggest that future data collection and annotation efforts should focus categories does not have to be as large as e.g. in Objects365.
right). More importantly, from certain point onwards
extreme: with only 10% of the instances,
broader of categories while the number of instances for each of those
Despite being big step forward, models trained on LVIS still show generalization gap and this gap widens when using stronger models. In this case, increased capacity aids performance on known categories slightly more than on novel categories, suggesting that some amount of overfitting on the training categories occurs, However, the curves for subsampled categories (Fig, S5p are not saturated at the maximal number categories in each dataset, which leaves hope that the gap can be closed by further increasing the number of categories in future datasets.
5.2 OUTLOOK
Our insight that applying existing methods on larger and more diverse datasets can lead to unex- pected capabilities mirrored in other areas. This phenomenon has been observed time and again and was termed the "unreasonable effectiveness data" (Halevy et al. 2009 [Sun et al.] (017) or the "bitter lesson" (Sutton 2019). It played key role in the breakthrough of DNNs thanks ImageNet (Russakovsky et al.] 2015; Krizhevsky et al. 2012) as well as recent results on gampe-play (Berner al. (2199) or language modelling (Brown
2020). Recently Kolesnikov al. (2019) achieve impressive few-shot learning performance including the first approach at one-shot ImageNet. As in our study, using simple method (transfer learning) on large and diverse dataset led results that are far better than what one would have expected: AlexNet performance with 3 samples per class in their case: 89% relative performance on LVIS in our case. We hope that by building on this insight soon move from trying solve few-shot learning towards using few-shot learning solve other problems.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Christopher Berner. Greg Brockman, Brooke Chan, Vicki Cheung, Przemystaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, al. Dota 2 with large scale deep reinforcement learning. arXiv:/912.000
2019.
Sujoy Kumar Biswas and Peyman Milanfar. One shot detection with laplacian object and fast matrix cosine similarity. TPAMI, 2015.
Tom B Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. WrXiv:2005.44165 2020.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and
Sergey 7agoruyko. End-to-end object detection with transformers. arXiv:2005. /2872, 2020.
Hao Chen, Yali Wang, Guoyou Wang, and Yu Qiao. LSTD: A Low-Shot Transfer Detector for Object Detection. AAAI, 2018.
Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy. and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv:1906.07155, 2019a.
Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. JCLR, 2019b.
Guneet S Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. A baseline for few-shot image classification. 1909,02900002222 2019.
Xuanyi Dong, Liang Zheng, Fan Ma, Yi Yang, and Deyu Meng. Few-Example Object Detection with Model Communication. TPAMI, 2018.
Mark Everingham, Luc Van Gool, Christopher K. Williams, John Winn, and Andrew Zisserman. The Pascal Visual Object Classes (VOC) Challenge. International Journal of Computer Vision, 2010.
Qi Fan, Wei Zhuo, Chi-Keung Tang, and Yu- Kü-Wing Tai. Few-shot object detection with attention-rpn and multi-relation detector. In CVPR, 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-Agnostic Meta-Learning for Fast Adapta- tion of Deep Networks. In ICMI, 2017.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. arXiv:2004.07780, 2020.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation. In CVPR, 2014.
Agrim Gupta, Piotr Dollar, and Ross Girshick. LVIS: dataset for large vocabulary instance seg- mentation. In CVPR, 2019.
Alon Halevy, Peter Norvig, and Fernando Pereira. The enreasonaableffecciivessss of data. Intelligent Systems, 2009.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. ICCV, 2017.
Ting-I Hsich, Yi-Chen Lo, Hwann- Hwann-Tzong Chen, and Tyng-Luh Liu, One-shot object detection with co-attention and co-excitation. In NeurlPS, 2019.
10

---

Under review as a conference paper at ICLR 2021
Kuang-Jui Jui Hsu, Yen- Yu Lin, and Yung- Yu Chuang. Deepco*: Deep instance co-segmentation by co-peak search and co-saliency detection. In CVPR, 2019.
Shuqiang Jiang, Yaohui Zhu, Chenlong Liu, Xinhang Song, Xiangyang Li, and Weiqing Min. Dataset bias few-shot image recognition. arXiv:2008. 07960, 2020.
Bingyi Kang, Zhuang Liu, Xin Wang, Fisher Yu, Jiashi Feng, and Trevor Darrell. Few-shot object detection via feature reweighting. 1xXiv:8//2.0186 2018.
Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. Siamese Neural Networks for One-shot Image Recognition. ICMI,, 2015.
Alexander Kolesnikov. Lucas Beyer, Xiaohua Zhai, Joan Puigcerver. Jessica Yung, Sylvain Gelly, and Neil Houlsby. Large scale learning of general visual representations for transfer. arXiv:1912.11370, 2019.
Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Shahab Kamali, Matteo Malloci, Jordi Pont- Tuset, Andreas Veit, Serge Belongie, Victor Gomes, Abhinav Gupta, Chen Sun, Gal Chechik, David Cai, Zheyun Feng. Dhyanesh Narayanan, and Kevin Murphy. Openimages: A public dataset for large-scals multi-label and multi-class image classification. Dataset available from https://storage. httgl//apii.coooopinnnnee//////e
2017.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In NeurlPS, 2012.
Xiang Li, Lin Zhang, Yau Pun Chen, Yu- Wing Tai, and Chi-Keung Tang. One-shot object detection without fine-tuning. arXiv:2005.03877 2020.
Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common Objects in Context. In ECCV, 2014.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature Pyramid Networks for Object Detection. In CVPR, 2017a.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017b.
Claudio Michaelis, Matthias Bethge, and Alexander S. Ecker. One-Shot segmentation in clutter. 1999097 2018a.
Claudio Michaelis, Ivan Ustyuzhaninov, Matthias Bethge, and Alexander S. Beker. One-Shot in- stance segmentation. arXiv:1811.1150/ 2018b.
Akihiro Nakamura and Tatsuya Harada.
Revisiting fine-tuning for few-shot learning.
arXiv:1910.00216, 2019.
Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulô, and Peter Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In International Conference on Computer Vision (ICCV). 2017. URL by com/ http:/daaatttiiiieeee /vittass/vsstss
Anton Osokin, Denis Sumin, and Vasily Lomakin. Os2d: One-stage one-shot object detection by matching anchor features. arXiv:2003.06880 2020.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NIPS, 2015.
Carsten Rother, Tom Minka, Andrew Blake, and Vladimir Kolmogorov. Cosegmentation of image global constraint into mrfs. In CVPR, 2006.
pairs by histogram
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJCV, 2015.
11

---

Under review as a conference paper at ICLR 2021
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. arXiv:1807.05960, 2018.
Eli Schwartz, Leonid Karlinsky, Joseph Shtok, Sivan Harary, Mattias Marder, Sharathchandra Pankanti, Rogerio Feris, Abhishek Kumar, Raja Giryes, and Alex M. Bronstein. RepMet: Representative-based metric learning for classification and one-shot object detection. In CVPR, 2019.
Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots. One-Shot Learning for Semantic Segmentation. BMVC, 2017.
Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. ICCV, 2019.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical Networks for Few-shot Learning. In NIPS. 2017.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable fectiveness of data in deep learning era. In ICCV, 2017.
Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. Meta-transfer learning for few-shot learning. CVPR, 2019.
Richard Sutton. The bitter lesson. Incomplete Ideas (blog). March, 2019.
Giorgos Tolias, Ronan Sicre, and Hervé Jégou. Particular object retrieval with integral max- max-pooling of CNN activations. ICLR, 2016.
Ivan Ustyuzhaninov, Claudio Michaelis, Wieland Brendel, and Matthias Bethge. One-shot Texture Segmentation. 1rxiv:1807.02654 2018.
Oriol Vinyals, Charles Blundell, Tim Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In NIPS, 2016.
Xin Wang, Thomas E Huang, Trevor Darrell, Joseph E Gonzalez, and Fisher Yu. Frustratingly simple few-shot object detection. arXiv:2003.06957 2020.
Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Meta-learning to detect rare objects. In ICCV, 2019.
Xiongwei Wu, Doyen Sahoo, and Steven CH Hoi. Meta-renn: Meta learning for few-shot object detection. 2019a.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. 2019b.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In CVPR, 2017.
Xiaopeng Yan. Ziliang Chen, Anni Xu, Xiaoxi Wang, Xiaodan Liang, and Liang Lin. Meta r-cnn: Towards general solver for instance-level low-shot learning. In ICCV, 2019.
Tengfei Zhang, Yue Zhang, Xian Sun, Hao Sun, Menglong Yan, Xue Yang, and Kun Fu. Comparison network for one-shot conditional object detection, WrXvv:194.02311 2019.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017.
12

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 HYPERPARAMETER SETTINGS
Our model is derived from mmdetection v1. v1.0re Chen et al. (019a) and uses the same hyperparam- eters as used for Faster R-CNN and RetinaNet. All details can be found in the config files of the re- spective mmdetection checkpoint: https:
com open mmlab/ /tm///mmtecciooon. tree/ Eree/5bf w5wf933ee1b6666 (16162224444 2d22344d33333 (eleleecccdoefffo222222222333999 h0l4443999//oo//ii/ configs Please note that the default settings for Pascal VOC differ slightly from those for COCO training. We use the COCO hyperparameters for experiments COCO, LVIS and Objects365 and Pascal VOC settings for Pascal VOC and GroZi-3.2k.
A.2 GROZI-3.2K
GroZi-3.2k is a dataset of in-store products developed by Osokin al.) (2020). In contrast to the datasets we study in the main paper, the depicted products cover only very limited range of objects and categories are very fine- fine-grained (one specific product makes one category). Smıdying such a different dataset will give better idea to which scenarios
expect findings generalize.
Experiment details: We train Siamese Faster R-CNN with the same hyperparameters as on Pascal VOC but increase the number of epochs 80 without pre-training and 20 with pre-training. The learning rate step is scheduled after 3/4 of the epochs (60 and 15 epochs respectively). We train on the train split defined in Osokin et al. (2020) and test on val-ole cl and val wal-new-cl.
Different to our approach on the other datasets, GroZi-3.2U has a fixed set of reference images which we use for training and evaluation. We therefore cannot average over multiple evaluation runs to determine how much the selection of reference images influences the result. As the validation sets are very small (60 images val-oldd ol-cl and 84 val-new-cl) we can assume that the error is rather large compared to the 0.2 % A we observe on the other datasets.
Note that our results are not directly comparable to those presented in Osokin al. (2020) as our evaluation procedure is markedly different. Most importantly we don't ignore 'difficult" examples which explains much of the drop in performance we observe.
Results: Without pre-training find sig-
GroZi scratch
GroZi
nificant generalization gap (Table [5] first row). As for LVIS and Objects365 performance on held-out categories depends more on the num-
80
70
ber of categories than the amount of data alone (Fig. A.).
40
categories suscample Instances
0.5 Fraction data
However, small datasets are usually not used to train models from scratch but most approaches will involve some form of transfer learning. We therefore train and tested models that were pre-trained on COCO and LVIS. These models don't show generalization gap (Table 5 Pre-Train COCO & LVIS) and performance depends more or less only on the amount of data available (Fig. A.13).
Fraction
Figure A.I: Performance on GroZi3.2 (Osokin
et al. 2020 held-out categories.
Conclusion: In contrast to the other datasets we study GroZi-3.2k has very fine-grained set of categories from narrow range of objects (in store products). So it is pretty remarkable that we find very similar effects when training from scratch. In contrast the results with pre-training may
Model
Pre-Train
val-old-cl
val-new-cl
S-FRCNN
81.7
60.6
S-FRCIN
COCO LVIS
74.9
73.6
S-FRCNN
75.8
69.1
Table 5: Performance (AP50) on GroZi-3.2k
13

---

Under review as a conference paper at ICLR 2021
at first seem contradict previous findings. They however make sense when compared to the situation on COCO: There training on the more diverse LVIS dataset leads to better results and smaller generalization gap. We here have very similar scenario where pre-train much larger and more diverse datasets and consequently find better generalization capabilities. This is also very similar the findings Kolesnikoo et al. (2019) who achieve good results for few-shot learning ImageNet by training on significantly larger dataset (Section 5.2).
A.3 ADDITIONAL FEW-SHOT RESULTS
We provide sioe results for the experiments in Fig- 5
in Fig.
A.2
LVIS
Objects365
LVIS
Objects365
40
40
AP50
AP50
20
training
20
subsample all cats.
held-out cats.
Categories during training
Categories during training
data
Figure A.2: Five-shot results for the subsampling experiments in Fig.
14


Reviews:
[{"review_id": "ua05ngEUoR", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper suggests that a major factor for increasing few-shot performance in the few-shot object detection task is the number of categories in the base training set used to pre-train the few-shot model on a large set of data before it is adapted to novel categories using only a few (or even 1) examples. This effect is measured by the authors by trying out the existing Siamese few-shot detector on 4 datasets: PASCAL, COCO, Objects365, and LVIS showing that the gap in performance on the seen training and the unseen (novel) testing categories is reduced when the base dataset has more classes (e.g. on LVIS where there are more than 1K classes, this \"generalization\" gap is shown to be minimal). The authors also quantify empirically the effect of increasing the model size and of prolonging the training schedule on this gap. As well as testing on COCO classes while training on LVIS.\n\nPros:\n- number of base classes is indeed an important factor in few-shot methods performance (not just in detection)\n- the paper is easy to follow and generally well written, the message conveyed is clear and the experiments are useful\n\nCons:\n- the positive effect of increasing the number of base classes on few-shot performance is long since known, and numerous works even in the few-shot classification literature have noted this fact, so nothing seems to be new here\n- the effect of increasing backbone size and prolonging the train schedule does not seem to indicate a strong correlation to the number of train classes, the original gap is maintained, while the gains of the tested modifications seem to be relatively similar up to some noise\n- I might be wrong, but it seems the authors mostly target few-shot localization, assuming the target object (given by the reference image example) is always present in the image. This is opposed to what I understand by few-shot detection, wherein a test episode there are several target objects, each accompanied by its support example and query images can have an arbitrary mix of these target objects or none at all.\n\nTo summarize: I like the paper, yet I fear it does not meet the plank of what I would consider a paper fitting ICLR standards in terms of novelty. There is nothing wrong in not proposing a new algorithm and instead - uncovering an important fact that was so far overlooked, but as I noted above this is not the case, the paper highlights a well known fact.... I would be happy to monitor other reviewers responses and authors comments on this issue of novelty and would be happy to be convinced otherwise.", "extra_scores": {}, "extra_text": {}}, {"review_id": "uUPMDaNplpb", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides a variety of studies to understand the generalization gap between known and novel classes in one-shot object detection. The studies are carried out by using siamese Faster R-CNN framework on four benchmark datasets. The most notable observation was that it was more important to increase the number of object category than to increase the number of instances per each category in order to reduce the generalization gap. This observation is very useful to anyone planning to build a dataset for this task or implement the appropriate method. Figure 5 is very important and well presented to support the main claim.\n\nHowever, there are some factors that need further studies to fully trust this observation in terms of generalization capabilities:\n1. Is it possible to get the same observations from other one-shot object detection methods other than the siamease Faster R-CNN?\n2. Can this observation be presented from using a variety of backbone methods which can be either a shallow method or a deeper CNN model?\n3. Can this claim be applied to any kind of category? (e.g., detection of person category having a more diverse appearances can be more affected by the number of instances used in training.)\n\nIn addition, why did increasing the number of categories reduce the detection accuracy of known objects? (Figure 5A) Did the method suffer from training as the number of categories increased? In other words, do we need to improve the accuracy of novel object classes at the expense of reduced detection accuracy of known classes?", "extra_scores": {}, "extra_text": {}}, {"review_id": "ppwj8hWq7ZV", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros.:\n1. The hypothesis is interesting and novel\n2. The paper is clearly presented and well-organized.\n3. The authors conduct many ablation studies to validate the proposed hypothesis.\n\nCons:\n1. Claimed observation:\n\nIn your claimed observation, increasing the number of classes could improve performance. However, increasing the number of classes also increasing the number of training samples in your experiments, and using more training data could have higher performance. Therefore, if the total number of training images is fixed, what are the results when more classes are included?\n\n2. About the usage of Siamese Faster R-CNN\n\nThe siamese faster R-CNN is a suitable network to validate the hypothesis, but it is somewhat old even with a more powerful backbone network. If more powerful methods are used, what is the performance gap? If more powerful methods could reduce the performance gap more, I think the performance issue is not related to the number of classes. The authors should adopt the latest method, such as [Ref.1,2,3], to repeat the experiments.\n\n[Ref. 1] Hsieh et al., One-Shot Object Detection with Co-Attention and Co-Excitation, NIPS'19\n\n[Ref. 2] Osokin et al., OS2D: One-Stage One-Shot Object Detection by Matching Anchor Features, ECCV'20\n\n[Ref. 3] Li et al., One-Shot Object Detection without Fine-Tuning, arXiv' 20\n\n3. About the shot number\n\nThe authors focus on only the one-shot setting, but the proposed hypothesis could be kept for the N-shot setting where N > 2.\nIf more shots are used, what is the performance gap? Besides, with more shots, could the proposed hypothesis be still effective? The authors should conduct the same experiments under a different number of shots.\n\nOverall, I appreciate the proposed hypothesis, but my primary concerns are the experiments. Please refer to my comments above.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Jx8nC_8UViN", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- The paper is well written and it’s easy to follow the story. \n- It provides a comprehensive review on related papers on object detection especially one-shot detection and their limitations. \n- The idea is simple and clear. Although the paper focuses on the data used for training, it gave great insights for understanding the generalization of object detector and provides practical guidelines for future large scale data collection.\n- Finally the extensive experimentation and analysis on results are convincing.  \n\nWeaknesses:\n- The paper studies the importance of number of object categories in training dataset and claims that the gap in one-shot detection can be closed by increasing the number of categories. However, it’s not the case that the more categories the better, there should also be enough diversity in data distribution and granularity in label definition. This is also an important guideline for future data collection. It would be interesting to see some analysis on data diversity and label granularity. \n\nMinor comments: \n- Fig 4 was not referred anywhere in the paragraphs.\n- In Table 3, the results will be complete with one more experiment using X101 and 3x are together\n- Typo in Figure 5’s caption: “ether” -> either\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers have ranked this paper as borderline accept. On the negative side, the main claim of the paper (the more categories for training a one-shot detector, the better) has already been observed in several works and very intuitive. However, the paper has done significant experimental work to support this claim. The paper is very well written, it carefully explores the existing setups for one-shot detection and highlights their weaknesses. The paper also gives advice on how to construct better datasets for one-shot detection (the conclusion "add more diverse categories" is somewhat obvious but the paper demonstrates how important that is).
Decision: Reject