Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: Post-review comments:

After reading the reply I decided I will keep my low score though it hurts to do so for a paper into which the authors definitely invested a lot of energy. Here is the reason why:

I think there are usually two ways in which a paper can make an important contribution: Through a new insight or through a new piece of modelling that will be widely used afterwards. I think in it's current form the paper presents neither.

The potential insight I see is proposal neglect. However even with the added experiments in Table 3 I find the results neither sufficient to prove the effect exists. The GT boxes are already added so overlooking objects while fine-tuning should not be a big issue. If it was it should have a bigger impact (improvements in Table 3 and on general performance are minimal).

So what about a widely usable piece of modelling? Despite the anecdotal motivation the presented method improves performance. So the question becomes: Will this be widely used? My prediction is that it won't. The improvement is rather incremental but the effort to use it is high. While the model is simple it comes with three additional hyperparameters which the authors tune individually for each experiment (Section 4, Hyperarameters VOC & COCO). This is the biggest issue I have with the method: It requires tuning a bunch of hyperparameters to achieve a marginal improvement.

Compare this to TFA [1], the method the paper builds upon. TFA is built upon a very simple insight (fine-tuning the heads works better than comparing representations) and because the insight and model are easy to use they will be the basis for a number of follow-up works (e.g. this paper). I cannot see the same happen with the presented method as long as hyperparameters have to be tuned for each dataset and split. I am sorry but in my eyes this is bad practice!.

To me this means the method will likely be of no lasting value. While it is SotA in the one-shot case for now I think the insights and methods used for achieving this performance cannot be used by other groups to improve performance even more. 

So what can I recommend the authors to do with the paper? I think there are two ways to increase the papers contribution: 
1. Study different effects and problems of RPNs in few-shot object detection. A better understanding will for sure help moving the problem forward. Even if the end result is: The RPN works surprisingly well. That would be a great insight as well in my eyes as it would free resources to address the other problems.
2. Improve the method so it provides a significant gain without any additional hyperparameter tuning. At only 3% more computing time using it would probably be a no-brainer if it was not for the hyperparameter tuning.

[1] Wanget. al ICML 2020, "Frustratingly simple few-shot object detection"


-------------------------------
1. Summary

The paper addresses the region proposal network (RPN) as a key component in few-shot object detection. Their analysis is centered around the concept of "proposal neglect": objects in the meta training set are ignored because no proposals are generated for them. They propose CoRPN, an assemble of RPN classifiers, to tackle the problem and present results on the original few-shot COCO and Pascal VOC benchmarks as well as ablation results.


2. Strengths 
+ I really like the general motivation and the RPN is for sure a key component for good few-shot object detection.
+ The proposed model and especially the diversity loss seem well chosen to address "proposal neglect".
+ The method brings a small but consistent performance improvement over TFA.

3. Weaknesses
- I think the main flaw of the paper is that there is no quantitative analysis of "proposal neglect", only some qualitative examples in the appendix. I don't want to say that it does not exist it makes only sense that such an effect messes up meta training. But I am missing experiments demonstrating the effect and it's impact on performance. 
- If proposal neglect is such a big issue for meta-training, why not simply add the ground truth boxes as proposals? Many object detection frameworks have that option built in. Comparison to this baseline seems crucial to evaluate the value of the proposed method.
- The new hyperparameters (N-rpn, phi, lambda_c, lambda_d) are apparently not fixed but chosen in a per-experiment basis as only ranges are provided in the appendix, not exact values. This makes it hard to judge the actual value of the method as hyperparameters have to be chosen in advance in real-world applications.
- I find the ablations lacking for a number of reasons which is critical as this should be the section that allows assessing the impact of "proposal neglect" and the proposed method. Specifically I have three issues: 
-- I am not sure the proposed experiments are suited to grasp the effect of the original problem and the effect of the method. How does the method impact RPN recall? What about classification accuracy, false negatives etc.? Table 6 does attempt to provide this information but I find it hard to understand what exactly is reported. Probably what I am interested in is already there it's just not that clear yet?
-- Ablations have only been performed on Pascal VOC and not on the more interesting and more challenging COCO dataset. Especially Table 6 would be interesting for COCO as well.
-- Ablation results appear to be extremely noisy. Compare the differences between configurations in Figure 4, adding or removing a classifier should not lead to a 5% performance change especially if this effect is not consistent across splits and changes again when another classifier is added or removed. To be clear this is not the authors fault but a problem of the setup introduced in Kang et. al. 2018 using a fixed set of images for meta-training. This approach makes overfitting to the specific chosen examples a big concern. Now I think it is good the authors follow the established evaluation protocol but this design makes it rally hard to examine if results are valid. In the case of the presented ablation studies this effect definitely casts a large shadow of doubt on the significance of these ablations. 

4. Recommendation

I think in it's current form this paper has to be rejected. At first I was excited to see someone addressing proposals in few-shot object detection but the paper sadly only addresses "proposal neglect" without any quantitative analysis or consideration of other potential problems. Furthermore due to the potentially large variability of results due to the fixed meta-training set (see variability in Figure 4) and potential hyperparameter variations between experiments it is hard to judge if the reported performance gains are robust and could be transferred to real-world applications. I do feel an investigation of the RPNs role in few-shot object detection is very worthwhile and the proposed method may in fact be valuable, however to make either of these two contributions a deeper analysis is needed.

5. Questions/Suggestions
- Provide quantitative analysis of the "proposal neglect" problem. Even better: Do a throughout assessment of the RPNs role and problems in few-shot object detection.
- Compare to a simple baseline: Adding the gt bounding boxes to the proposals during meta-training
- Please clarify if hyperparameters were the same for all experiments and report the chosen values in the main paper, not the appendix.
- Run ablation experiments on COCO as well
- Better explain Table 6

6. Additional feedback

None, the paper is overall well written, figures are clear and understandable. 
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
The paper argues that in the case of few shot object detection, the quality of region proposals is important, as if the region proposal network misses any of the very scarce positive box, the performance is severely impacted. It proposes to use multiple cooperating RPNs to alleviate the problem and improve few shot detection.


Positives
- The paper is easy to follow and proposes a simple idea of improving the RPN classifiers so that any proposal with high IOU with the object is not pruned out early due to the RPN. At the same time the RPN maintains pruning performance and does not send too many false positives.
- The method proposed is clear
- The evaluations are done on standard challenging benchmarks for the problem and good performances are reported

Negatives
- The thesis is simple and acceptable, i.e. for few shot detection training, missing any high IOU proposal can be very bad for performance, so improve the RPN to avoid that happening at that stage, however the proposed method is the best way to do it is not convincing. Why does what is being proposed to solve this problem of proposal neglect the right way to do so?
- In particular there can be many other ways of making the RPN classifier better, e.g. by simply making the sub network bigger, which would be fair as in the proposed method also more parameters are added
- There seems to be no empirical proof with ablation experiments, that the diversity and cooperation losses are useful for that task? What happens when we train the same without using, either or both of the losses (with acceptable settings of parameters)
- In fig 4 all results with number of RPNs from 1 to 10 should be reported. The sudden jump from 5 to 10 seems a bit convenient for the claim that large number of RPNs lead to large number of FPs leaking reducing the performance. The performances of naive RPN ensembles can also be added to the bar plot (instead of selected numbers in tab. 4)
- The choice of number of RPNs should be properly cross-validated/justified. Right now it feels a bit ad-hoc; in supplementary it is mentioned "We mostly use five RPN’s, except for PASCAL VOC novel split 3, where two RPN’s lead to better performance." -- has this been tuned based on the performance on the test sets?
- The hypothesis that large number of FPs degrade the performance for large number of RPNs, can be evaluated empirically also by computing the actual statistics for region proposals accepted or rejected.
- Own baseline (same implementation details/strategy as CoRPN) with #RPN=1 should be reported for all tables. Right now the point of reference seems to be TFA, which is implemented elsewhere; the implementation here might already have slight benefits due to the parameter settings etc.

Overall, the thesis is acceptable but the method is still not very convincing. The performances reported are state of the art in COCO (more for very few shots 1,2 etc. and marginal for larger), marginal for VOC low shots and not for higher shots. I am a bit lukewarm about the paper in general.

review_title: Using multiple RPNs to improve few-shot object detection

final_score: rating: 7: Good paper, accept
review_content: The authors propose a few-show object detection architecture, which improves the 1st stage of two-stage detectors (R-CNN in particular). In a few shot setting, the existing approaches the region proposal generator may ignore some novel out-of-distribution classes as they have not been included at training time. The proposed method attempts to correct this by using many RPN's, and training them such that the gradient is only passed to one of them at a time; thus forcing them to learn mutually different kinds of regions.

The topic is relevant, and not too widely studied. Few-shot learning is well presented in the literature in the classification setup, but less so in detection. The idea is novel, although quite straightforward and somewhat arbitrary. I encourage the authors to clarify the insight as to why this particular combination of losses was chosen (I could think of alternative formulations). Also it would be important to see some discussion about the computational cost: for object detection, people often prefer simple and fast approachs (SSD, Yolo) that may be less accurate but are fast to execute. How much does the proposed method with N classifiers increase computation (something less than N-fold, but how much)?

The results reach state-of-the-art on some cases; especially with very-few-shot domain. 
review_title: Few shot object detection using multiple region proposal networks that specialize in their own domains, reaching some SotA results

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this paper, the authors present a interesting, novel idea of promoting the diversity and cooperation among multiple RPNs for the problem of few-shot detection. They first identify a critical problem in few-shot detection, which is that existing RPNs can miss objects of novel classes and that their proposed boxes make very similar errors. In order to resolve this issue, the authors propose to utilize multiple RPNs in the same detection pipeline and include a diversity loss when training the RPNs, such that they provide diverse scores to the anchor boxes. In this way, the chance of all the RPNs missing an object becomes small. The authors further include a cooperation score to enforce the RPNs to provide more meaningful scores (rather than sparse scores just to promote diversity). The proposed approach is evaluated on COCO and Pascal, and the results demonstrate that the proposed approach does improve the detection performance, as compared to the state of the art, especially in very low-shot cases.

Overall, this is a very well-written paper. The authors have identified the existing problem and motivated their work well. The proposed method is novel and well described, and the evaluation is thorough. 

There is one thing that I am curious about and would hope to get the authors' insights. In the evaluation, the authors compare their proposed method with a naive ensemble approach, which combines the results of RPNs with different random initializations.  I am curious about the performance of a slightly less naive, yet standard ensemble method, i.e., bootstrapping the training set. This could potentially allow the RPNs to learn from slightly different data and then provide diverse outputs as well. 

I don't have any other major comments. Here are a few minor ones.

- For the diversity loss, did the authors try only penalizing the case when the determinant is close to zero, as compared to making it large? It seems that the sign of the determinant shouldn't matter. The current choice is reasonable and results in a good detector, but I am just curious what the authors think.

- How does one determine a good number of RPNs in general? The authors mention that this can be problem/dataset-dependent. In practice, should one rely on a validation set to choose this number? Or is there any exploratory analysis that one could perform to find a good (mostly not optimal) number?

- Please increase the gap between Tables 4 and 5.


----------- updates after reading author response -----------

After looking at the authors' response, the revised paper, and the other reviews, I'd like to update my rating from "Accept" to "Marginally below acceptance threshold". While I still like the overall idea of this paper and I believe this is an interesting direction of research, the additional results in the revised paper actually raise more questions and there are issues that need to be addressed in this current study.

1. Properly quantifying "proposal neglect"

The improvements shown in Table 3 are very small, which do not support the claim that the proposed method produce "more and better boxes". Table 3 might actually not be the correct way to assess the "proposal neglect" (or resolving proposal neglect). I would expect to see a more in-depth result analysis. For instance, when comparing with the SOTA TFA approach, for each image on average, how many more correct objects are detected due to the proposed approach being able to find boxes that TFA cannot. Something like this would clearly indicate the benefit of the proposed approach.

2. Sensitivity to hyperparameters

In Fig. 4 (left), there is a huge sensitivity to the hyperparameter selection. For instance, the performance of the proposed approach would drop drastically by just changing the number of RPNs from 2 to 3. This creates uncertainty for people to use this approach. Furthermore, for the COCO results, when the hyperparameters are not selected based on this set, the improvements are pretty small as compared to TFA. I believe a more robust hyperparameter selection method would be needed for the proposed approach to thrive.
review_title: Interesting idea, clearly-written paper, but there are some analyses missing (review updated)

Meta-review: The reviewers have not supported the acceptance of this paper where the key weakness is that the study of the proposal neglect effect is not sufficient (see the reviews for the details). I agree with the assessment of the reviewers and recommend rejecting the paper in its current form.
Decision: Reject