Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper suggests that a major factor for increasing few-shot performance in the few-shot object detection task is the number of categories in the base training set used to pre-train the few-shot model on a large set of data before it is adapted to novel categories using only a few (or even 1) examples. This effect is measured by the authors by trying out the existing Siamese few-shot detector on 4 datasets: PASCAL, COCO, Objects365, and LVIS showing that the gap in performance on the seen training and the unseen (novel) testing categories is reduced when the base dataset has more classes (e.g. on LVIS where there are more than 1K classes, this "generalization" gap is shown to be minimal). The authors also quantify empirically the effect of increasing the model size and of prolonging the training schedule on this gap. As well as testing on COCO classes while training on LVIS.

Pros:
- number of base classes is indeed an important factor in few-shot methods performance (not just in detection)
- the paper is easy to follow and generally well written, the message conveyed is clear and the experiments are useful

Cons:
- the positive effect of increasing the number of base classes on few-shot performance is long since known, and numerous works even in the few-shot classification literature have noted this fact, so nothing seems to be new here
- the effect of increasing backbone size and prolonging the train schedule does not seem to indicate a strong correlation to the number of train classes, the original gap is maintained, while the gains of the tested modifications seem to be relatively similar up to some noise
- I might be wrong, but it seems the authors mostly target few-shot localization, assuming the target object (given by the reference image example) is always present in the image. This is opposed to what I understand by few-shot detection, wherein a test episode there are several target objects, each accompanied by its support example and query images can have an arbitrary mix of these target objects or none at all.

To summarize: I like the paper, yet I fear it does not meet the plank of what I would consider a paper fitting ICLR standards in terms of novelty. There is nothing wrong in not proposing a new algorithm and instead - uncovering an important fact that was so far overlooked, but as I noted above this is not the case, the paper highlights a well known fact.... I would be happy to monitor other reviewers responses and authors comments on this issue of novelty and would be happy to be convinced otherwise.
review_title: Official Blind Review #3

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper provides a variety of studies to understand the generalization gap between known and novel classes in one-shot object detection. The studies are carried out by using siamese Faster R-CNN framework on four benchmark datasets. The most notable observation was that it was more important to increase the number of object category than to increase the number of instances per each category in order to reduce the generalization gap. This observation is very useful to anyone planning to build a dataset for this task or implement the appropriate method. Figure 5 is very important and well presented to support the main claim.

However, there are some factors that need further studies to fully trust this observation in terms of generalization capabilities:
1. Is it possible to get the same observations from other one-shot object detection methods other than the siamease Faster R-CNN?
2. Can this observation be presented from using a variety of backbone methods which can be either a shallow method or a deeper CNN model?
3. Can this claim be applied to any kind of category? (e.g., detection of person category having a more diverse appearances can be more affected by the number of instances used in training.)

In addition, why did increasing the number of categories reduce the detection accuracy of known objects? (Figure 5A) Did the method suffer from training as the number of categories increased? In other words, do we need to improve the accuracy of novel object classes at the expense of reduced detection accuracy of known classes?
review_title: Further study is needed to verify the generalization ability of the main claims.

final_score: rating: 6: Marginally above acceptance threshold
review_content: Pros.:
1. The hypothesis is interesting and novel
2. The paper is clearly presented and well-organized.
3. The authors conduct many ablation studies to validate the proposed hypothesis.

Cons:
1. Claimed observation:

In your claimed observation, increasing the number of classes could improve performance. However, increasing the number of classes also increasing the number of training samples in your experiments, and using more training data could have higher performance. Therefore, if the total number of training images is fixed, what are the results when more classes are included?

2. About the usage of Siamese Faster R-CNN

The siamese faster R-CNN is a suitable network to validate the hypothesis, but it is somewhat old even with a more powerful backbone network. If more powerful methods are used, what is the performance gap? If more powerful methods could reduce the performance gap more, I think the performance issue is not related to the number of classes. The authors should adopt the latest method, such as [Ref.1,2,3], to repeat the experiments.

[Ref. 1] Hsieh et al., One-Shot Object Detection with Co-Attention and Co-Excitation, NIPS'19

[Ref. 2] Osokin et al., OS2D: One-Stage One-Shot Object Detection by Matching Anchor Features, ECCV'20

[Ref. 3] Li et al., One-Shot Object Detection without Fine-Tuning, arXiv' 20

3. About the shot number

The authors focus on only the one-shot setting, but the proposed hypothesis could be kept for the N-shot setting where N > 2.
If more shots are used, what is the performance gap? Besides, with more shots, could the proposed hypothesis be still effective? The authors should conduct the same experiments under a different number of shots.

Overall, I appreciate the proposed hypothesis, but my primary concerns are the experiments. Please refer to my comments above.

review_title: I appreciate the proposed hypothesis, but my major concerns are the experiments. Please refer to my comments.

final_score: rating: 7: Good paper, accept
review_content: Strengths:
- The paper is well written and it’s easy to follow the story. 
- It provides a comprehensive review on related papers on object detection especially one-shot detection and their limitations. 
- The idea is simple and clear. Although the paper focuses on the data used for training, it gave great insights for understanding the generalization of object detector and provides practical guidelines for future large scale data collection.
- Finally the extensive experimentation and analysis on results are convincing.  

Weaknesses:
- The paper studies the importance of number of object categories in training dataset and claims that the gap in one-shot detection can be closed by increasing the number of categories. However, it’s not the case that the more categories the better, there should also be enough diversity in data distribution and granularity in label definition. This is also an important guideline for future data collection. It would be interesting to see some analysis on data diversity and label granularity. 

Minor comments: 
- Fig 4 was not referred anywhere in the paragraphs.
- In Table 3, the results will be complete with one more experiment using X101 and 3x are together
- Typo in Figure 5’s caption: “ether” -> either

review_title: In this paper, authors propose to close the generalization gap in one-shot object detection by increasing the number of object categories used in training. The paper also showed that standard methods to improve object detection models like stronger backbones or longer training schedules also benefit novel categories, which was not the case for smaller datasets. The conclusion provides guidelines for future data collection.

Meta-review: The reviewers have ranked this paper as borderline accept. On the negative side, the main claim of the paper (the more categories for training a one-shot detector, the better) has already been observed in several works and very intuitive. However, the paper has done significant experimental work to support this claim. The paper is very well written, it carefully explores the existing setups for one-shot detection and highlights their weaknesses. The paper also gives advice on how to construct better datasets for one-shot detection (the conclusion "add more diverse categories" is somewhat obvious but the paper demonstrates how important that is).
Decision: Reject