Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: This paper claims that in few-shot detection tasks, performance can be degraded due to the lack of RPN ability to find object proposals for new categories with aspect ratios significantly different from those of the base categories. To overcome the limitations of RPN in few-shot detection, this paper uses multiple RPNs that have been trained in the direction of enhancing diversity among RPNs and cooperating with each other. To adequately validate this benefit of the proposed method, base/novel categories split of datasets are modified so that the (few-shot) images of the novel categories have very different aspect ratios from images of the base categories, which are referred ARShift. Object detectors trained using the proposed method successfully increases the accuracy for the novel categories, avoiding the performance degradation caused by the catastrophic forgetting issue for the base categories in the few-shot detection task on the three datasets, PASCAL VOC, COCO, and LVIS datasets.
strength_and_weaknesses: * Strengths
1. For the test environment using the ARShift benchmark, it was effective to significantly increase the few-shot detection accuracy using the proposed method on the three detection datasets. It also prevented performance degradation for the base categories, which is often caused by the catastrophic forgetting problem that is common in a few-shot detections.


* Weaknesses
1. The problem in a specific environment of the few-shot detection problem, which has a large difference in the aspect ratio distribution of the basic category and the novel category, as claimed in this paper, is of interest to only a few researchers, and the applicability/generality of the problem is very Limited. Thus, the environment claimed in this paper did not appear in the general dataset, but only in the dataset that was modified to emphasize the environment.

2. In addition, the proposed method using multiple RPNs is not designed to address a claimed problem. Experimentally, several RPNs were specialized in generating region proposals of specific aspect ratios (Tab 5), but this function was not considered at all in the CoRPN design. 

3. Paper presentation is immature yet.
- Figure 1 is not mentioned anywhere in the manuscript's content.
- The same explanation about using Faster R-CNN are given in the last paragraph of Section 3.1 and the second paragraph of Section 3.3.
- In Figure 2, it's hard to figure out what 'density' is as the sum of the densities over the entire aspect ratio range seems to be greater than 1.
- In the top row of Figure 3, there are no explanations about the difference betwee the gray and blue box. In addition, non-maximum suppression is applied after performing bouding box regression so the illustrations in the bottom row of Figure 3 may bring incorrect information. Most of all, this figure does not deliver important information and nor is hard to understand as it has to occupy the large space of the manuscript.
- In eq 3, Sigma may be Sigma_jk
- Important details of experimental setup is missing such as the number of RPNs. According to Tab. 5, the number of RPNs may be larger than five.

4. Some questionable impact of the proposed method based on the experimental results
- In Tab. 5, only two or three RPNs appear to be activated. Does it mean the proposed method uses multiple RPNs inefficiently.
- What is the version of CoRPN's with the best accuracy? Is it the model trained with the Cooperation loss with phi=0.5 and the diversity loss? If so, the CoRPN seems to be very sensitive to the phi.

5. Similar problems and similar method designs have already been claimed in [a]. [a] is applied to general object detection and more experts than RPN are used for the classifier, but since it seems very simple to apply it to RPN, comparison with [a] will be necessary.

[a] H. Lee, Multi-expert R-CNN for Object Detection, In IEEE TIP.
clarity,_quality,_novelty_and_reproducibility: - As mentioned in the weaknesses, the manuscript must be revised thoroughly to improve its clarity and presentation quality.
- As for novelty, it doesn't seem to meet the ICLR acceptance criteria.
- Since the important implementation details (i.e., the number of RPNs) is missing, it is difficult to be reproduced.
summary_of_the_review: My rating was based on the points presented in Weaknesses. My biggest concern is that there is no consistency between the problem to be claimed and the method design to solve the problem.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper addresses the problem of the aspect-ratio shift between the base and novel classes in few-shot object detection resulting in the poor box proposals of the RPN in a two-stage detector. They propose using multiple classifiers instead of one as in the current RPN. Their experiments show significant improvements with the new versions of the COCO and LVIS datasets but marginal improvements with the original datasets. The new splits of the COCO and LVIS datasets are created by re-select the subset of base and novel classes whose aspect ratios are significantly different. 
strength_and_weaknesses: *Strengths*
+ The finding is quite interesting, i.e., shifting in aspect ratio distribution of object bounding boxes between the base and novel classes.
+ The structure of this paper is well-written.

*Weaknesses*   
+ Incorrect average results in Tab. 3. For DeFRCN, the AP for split 1, 2, and 3 are 9.6, 8.0, and 9.0, respectively. But they report the overall AP is 15.0. The same observation applies to the DeFRCN Ensemble of Experts. Thus, the conclusions drawn are wrong for this table. DeFRCN Ensemble of Experts does help a lot.
+ This method aims to address the poor proposal generation problem of RPN. It would be much better to include the AR results to prove their recall is better than that of the baseline. 
+ The authors should explain the intuition behind the formulation of diversity loss. It is not trivial to interpret the proposed loss to enhance the diversity of the classifiers.
+ Do the authors claim the new split of COCO and LVIS datasets as a contribution? The improvement in the original splits seems very marginal. If the base classes contain high-variation in the bounding box aspect ratios, e.g. human, they can arguably cover the aspect ratio of many novel classes. 
+ Since the paper mainly focuses on the improvement of RPN, they should move the qualitative results to the main paper, not the supplementary material.  
+ Fig. 1 does not reflect the motivation of this paper. The proposals on the right are still good which can be further regressed in the second stage of Faster-RCNN.

clarity,_quality,_novelty_and_reproducibility: I cannot see the attached code so I cannot verify the reproducibility of the paper. The paper seems not easy to implement. 
summary_of_the_review: The paper proposed a method for alleviating the problems of aspect ratio shift in the few-shot object detection by introducing multiple classifiers of the RPN instead of one. It is arguable to see the practicality of the new splits of the COCO and LVIS datasets since the proposed methods have marginal improvements in the common setting. Also, the paper also has the incorrect average results reported thus drawing an incorrect observation of the ensemble RPN.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper aims to solve the aspect ratio gap between base and novel classes in few-shot object detection. The authors present a very simple CoRPN method, which only uses multiple classifiers in RPN to generate more diverse anchors. To evaluate the effects of the aspect ratio gap, the authors propose ARShift benchmarks, which enlarge the aspect ratio differences between the training and testing sets in standard detection benchmarks. Experimental results show that the proposed CoRPN achieves remarkable improvements on ARShift benchmarks.
strength_and_weaknesses: Strength
1. The motivation is interesting, which aims to solve the aspect ratio gap problem.
2. New ARShift benchmarks are useful for future research on aspect ratio.
3. The proposed CoRPN is simple yet effective.

Weaknesses
1. Is CoRPN only designed for few-shot object detection? If so, it is better to add 'few-shot' to the title. If not, can you provide more results on other detection settings?
2. Can your show the aspect ratio distributions of RPN and CoRPN?  It is important to prove that your CoRPN generates more diverse anchors than RPN.
3. ARShift benchmarks artificially enlarge aspect ratio gaps between the training and test sets, which is somewhat tricky. Can you show the results on standard detection benchmarks?
clarity,_quality,_novelty_and_reproducibility: The clarity, quality, novelty and reproducibility are good. 
summary_of_the_review: Overall, it's a good work, simple yet effective. My main concern is experiments. The authors only test on self-made benchmarks.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This submission is about few-shot object detection with a distribution shift of the bounding box aspect ratios between base and novel categories. The paper first elaborates that there can be splits of base and novel categories that do show a clear distribution shift of the bounding box aspect ratios and that existing few-shot object detectors perform poorly in this scenario. Then, a novel region proposal mechanism for a two-stage object detector is proposed that consists of an ensemble of foreground/background classifiers and a new training strategy. Specifically, the ensemble is trained without explicitly assigning certain aspect ratios to certain RPNs, but the RPNs "select" the boxes based on their predictive confidence. Additionally, a diversity loss and a cooperation loss are used for training. Experiments are conducted on specifically selected splits of base and novel categories for the Pascal VOC, COCO and LVIS datasets.
strength_and_weaknesses: Strengths:
- Two different few-shot object detectors are used (DeFRCN and TFA) to showcase the benefits of the novel RPN
- The overall research direction is important I think, because it studies the practical situation of distribution shifts for novel categories in a few-shot setting.
- The results on Pascal VOC show a big gain over existing baselines.
- The analysis of the aspect ratio distributions is interesting in general, even beyond this specific few-shot learning setting.
- The ensemble baseline (and the experiment with larger RPNs in the appendix) are a fair comparison point.
- The additional experiments in the appendix are good in general, I think

Weaknesses:
- The paper's scope is limited to two-stage detectors with RPNs, and RPNs trained on the base categories for the few-shot setting. Other detector designs (like RetinaNet or DETR) are only discussed in related work, but the scope of the actual problem addressed in this paper (distribution shift of aspect ratios) should also include some form of (quantitative) experiments for these detector designs. The statement "... but likely occurs in one-stage detection too" at the end of page 2 is insufficient in my opinion.
- The experimental setting needs more justification and analysis. I understand that distribution shifts can obviously happen between base and novel categories. But I'm wondering if the aspect ratio distribution shift here is only an artifact of the small number of base and novel categories in this specific experimental setting. As far as I understood, the number of base categories is only 15, 20 and 20 for the datasets Pascal VOC, COCO, and LVIS. Why is that, when datasets COCO and LVIS have 80 and over 1200 categories, respectively? It would be good to study the distribution shift under varying number of base categories. What would happen if only five more base categories are added?
- The topic of the paper is rather narrow for a much broader problem, i.e., distribution shift between base and novel categories. The paper only addresses aspect ratio distributions for two-stage detectors. While authors comment on object location and scale, another distribution shift is due to semantic and appearance differences. Related work on open-vocabulary detection [A,B] can be discussed, where RPNs are evaluated on novel categories in a zero-shot experiment.
- The improvements for COCO and LVIS (Tables 2 & 3) are not as obvious as for the Pascal VOC dataset (Table 1). Compared to the plain ensemble, the proposed RPNs do not show a clear edge. Why is that? Is it related to the number of base and novel categories in the datasets? For VOC, there are only 5 novel categories. How do the per-category results look like?

References:
- [A] Open-Vocabulary Object Detection via Vision and Language Knowledge Distillation. Gu et al. ICLR'22
- [B] Exploiting Unlabeled Data with Vision and Language Models for Object Detection. Zhao et al. ECCV'22
clarity,_quality,_novelty_and_reproducibility: - Reproducibility: The authors provide details about the novel splits between base and novel categories, as well as various hyper-parameters. I still think, code will be needed to fully reproduce the training strategy and the loss functions.
- Overall, the quality of the paper is good: well-written, reasonable baselines; the experimental setting can be improved
- Figures 1 and 2, and Table 1 are only referred to in the text in Section 3.2 on page 5. Figure 3 is referred to earlier. The order of figures/tables or the text should be adjusted.
- Is there a reference for the "OR" strategy? Does it stand for something specific or just literally the word "or"?
- Is "unfairness" the right word to use here? I guess bias is better, no?
- Caption of Figure 2: "different classes might have quite different bounding box distributions" ... even within one category, the aspect ratio can change significantly based on pose, consider a bicycle.
summary_of_the_review: I'm leaning toward rejection because of the narrow problem setting, solution and experimental evaluation. Distribution shifts in few-shot learning go beyond aspect ratios, and a more thorough experiments and analyses are needed (impact of number of base categories, evaluation on larger sets of novel categories, per-category results, ...) to convince me of the importance of this problem.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper claims that in few-shot detection tasks, performance can be degraded due to the lack of RPN ability to find object proposals for new categories with aspect ratios significantly different from those of the base categories. To overcome the limitations of RPN in few-shot detection, this paper uses multiple RPNs that have been trained in the direction of enhancing diversity among RPNs and cooperating with each other. To adequately validate this benefit of the proposed method, base/novel categories split of datasets are modified so that the (few-shot) images of the novel categories have very different aspect ratios from images of the base categories, which are referred ARShift. Object detectors trained using the proposed method successfully increases the accuracy for the novel categories, avoiding the performance degradation caused by the catastrophic forgetting issue for the base categories in the few-shot detection task on the three datasets, PASCAL VOC, COCO, and LVIS datasets.

The proposed method improves the few-shot accuracy in three detection datasets. 

One of the main problems is the difference in aspect ratios. The dataset is not realistic. The paper writing is immature. Missing motivation. The paper's scope is limited to two-stage detectors with RPNs. The experimental setting needs more justification and analysis. The topic of the paper is rather narrow for a much broader problem. The improvements for COCO and LVIS (Tables 2 & 3) are not as obvious as for the Pascal VOC dataset (Table 1).

The authors addressed some of the comments. However, didn't manage to convince the reviewers. And the paper is not mature enough to be published.

justification_for_why_not_higher_score: Immature paper
Decision: Reject