Cooperating RPN's Improve Few-Shot Object Detection
Authors: Weilin Zhang
Yu-Xiong Wang
David Forsyth
Venue:  (2021)
Keywords: Few-shot learning
Object detection

Abstract: Learning to detect an object in an image from very few training examples - few-shot object detection - is challenging, because the classifier that sees proposal boxes has very little training data.  A particularly challenging training regime occurs when there are one or two training examples. In this case, if the region proposal network (RPN) misses even one high intersection-over-union (IOU) training box, the classifier's model of how object appearance varies can be severely impacted. We use multiple distinct yet cooperating RPN's.  Our RPN's are trained to be different, but not too different; doing so yields significant performance improvements over state of the art for COCO and PASCAL VOC in the very few-shot setting. This effect appears to be independent of the choice of classifier or dataset.  


Under review as a conference paper at ICLR 2021
COOPERATING RPN's IMPROVE FEW-SHOT OBJECT DETECTION
Anonymous authors Paper under double-blind review
ABSTRACT
Learning to detect object in image from very few training examples few- shot object detection is challenging, because the classifier that sees proposal boxes has very little training data. particularly challenging training regime oc- curs when there are one or two training examples. In this case, if the region pro- posal network (RPN) misses even one high intersection-over-uiii ing box, the classifier's model of how object appearance varies can be severely impacted. We use multiple distinct yet cooperating RPN's. Our RPN's are trained to be different, but not too different; doing so yields significant performance im- provements over state of the art for COCO and PASCAL VOC in the very few-shot setting. This effeet appears to be independent of the choice of classifier or dataset.
(IOU) train-
INTRODUCTION
Achieving accurate few-shot object detection is difficult, because one must rely on classifier build- ing useful model of variation in appearance with very few examples. This paper identifies an important effect that causes existing detectors to have weaker than necessary performance in the few-shot regime. By remediating this difficulty, we obtain substantial improvements in performance with current architectures.
The effect is most easily explained by looking at the "script" that modern object detectors mostly follow. As one would expect, there are variations detector structure, but these do not mitigate the effect. A modern object detector will first find promising image locations; these are usually, but not always, boxes. We describe the effect terms of boxes reported by a region proposal network (RPN) (Ren et al., 2015), but expect that it applies to other representations of location, too. The detector then passes the promising locations through a classifier to determine what, if any, object is present. Finally, it performs various cleanup operations mon-mma suppression, bounding box regression, etc.), aimed at avoiding multiple predictions the same location and improving localization. The evaluation procedure for reported labeled boxes uses intersection-over-unioo (IOU) test as part of determining whether box is relevant.
A detector that is trained for few-shot detection is trained two types of categories. Base categories have many training examples, and are used to train an RPN and the classifier. Novel categories have one (or three, or five, etc.) examples per category. The existing RPN used find boxes, and the classifier is fine-tuned to handle novel categories.
Now assume that the detector must learn to detect category from single example. The RPN is already trained on other examples. It produces a collection of relevant boxes, which are used to train the classifier. The only way that the classifier can build a model of the categories variation in appearance is by having multiple high IOU boxes reported by the RPN. In turn, this means that an RPN that behaves well on base categories may create serious problems for novel categories. Imagine that the RPN reports only few of the available high IOU boxes in training data. For base categories, this is not a problem; many high IOU boxes will pass to the classifier because there is a lot of training data, and so the classifier will be able to build model of the categories variation in appearance. This variation will be caused by effects like aspect, in-class variation, and the particular RPN's choice of boxes. But for novel categories, an RPN must report many high IOU boxes as possible, because otherwise the classifier will have too weak model of appearance variation - for example, it might think that the object must be centered in the box. This will significantly depress

---

Under review as a conference paper at ICLR 2021
(a) TFA
(b) Ours
Figure 1: Few-shot detectors are subject to the proposal neglect effect. On the left: an image (from the PASCAL VOC split 3 novel classes test set) showing the top 10 proposals from a sfate-of th-the-art few-shot detector TFA (Wang et al., 2020). On the right: the top 10 proposals from cooperating RPN's. Cat is in the novel classes (boat, cat, mbike, sheep, sofa} (i.e.. both models are cat detectors). Note that the classifier in TFA will not see a cat box, and so it cannot detect the cat. This is a disaster when this image is used training. For our approach, the cat box is found and is high in the top 10 proposals.
accuracy. As Figure 1 and our results illustrate, this effect (which we call proposal neglect) is present in the state-of-the-art few-shot detectors.
One cannot escape this effect by simply reporting lots of boxes, because doing so will require the classifier to be very good at rejecting false positives. Instead, one wants the box proposal process to not miss high IOU boxes, without wild speculation. We offer relatively straightforward strategy. We train multiple RPN's to be somewhat redundant (so that if one RPN misses a high IOU box, another will get it). without overpredicting. In what follows, we demonstrate how to do so and show how to balance redundancy against overprediction.
Our contributions are three-fold: (1) We identify an important effect in few-shot object detection that causes existing detectors to have weaker than necessary performance in the few-shot regime. (2) We propose to overcome the proposal neglect effect by utilizing RPN redundancy. (3) We design an RPN ensemble mechanism that trains multiple RPN's simultaneously while enforcing diversity and cooperation among RPN's. We achieve state-of-the-art performance on COCO and PASCAL VOC in very few-shot settings.
2 BACKGROUND
Object Detection with Abundant Data. The best-performing modern detectors are based on con- volutional neural networks. There are two families of architecture, both relying the remarkable fact that one can quite reliably tell whether an image region contains an object independent of cat- egory (Endres & Hoiem, 2010; van de Sande et al., 2011). In serial detection, a proposal process (RPN/s in what follows) offers the classifier a selection of locations likely to contain objects, and the classifier labels them, with the advantage that the classifier "knows" the likely support of the object fairly accurately. This family includes R-CNN and its variants (for R-CNN (Girshick et al., 2014); Fast R-CNN (Girshick, 2015); Faster R-CNN (Ren et al., 2015); Mask R-CNN (He et al., 2017): SPP-Net (He et al., 2015); FPN (Lin et al., 2017); and DCN (Dai et al., 2017)). In parallel, the proposal process and classification process are independent; these methods can be faster, but the classifier "knows" very little about the likely support of the object, which can affect accuracy. This family includes YOLO and its variants (for YOLO versions (Redmon et al., 2016; Redmon & Farhadi, 2017; 2018; Bochkovskiy et al., 2020); SSD (Liu et al., 2016); Comernet (Law & Deng, 2018); and ExtremeNet (Zhou et al., 2019)). This paper identifies an issue with the proposal process that can impede strong performance when there is very little training data (the few-shot case). The effect is described in the context of serial detection, but likely occurs in parallel detection too.
Few-Shot Object Detection. Few-shot object detection involves detecting objects for which there are very few training examples. There is a rich few-shot classification literature (roots in (Thrun, 1998; Fei-Fei et al., 2006)). Dvorik et al. (2019) uses ensemble procedures for few-shot classifi-
2

---

Under review as a conference paper at ICLR 2021
cation. As to detection, Chen et al. (2018) proposes a regularized fine-tuning approach to transfer knowledge from a pre-trained detector to a few-shot detector. Schwartz et al. (2019) exploits metric- learning for modeling
distributions for each class. ofteeof-thhearr few-shot detectors are usually serial (Wang et al., 2019; Yan et al., 2019; Wang et al., 2020; Fan et al., 2020; Wu et al., 2020; Xiao & Marlet, 2020). The existing literature can be seen as variations framework, where one splits data into two sets of categories: base classes C.. (which have many training examples) and novel classes Cn (which have few). The RPN and classifier are then trained with instances from the base classes, producing a detector for [C&] categories. The final layer of the resulting classifier is expanded to classify into [C&] |Ca classes by inserting random weights con- necting the final feature layer to the [Cn] novel categories. Now the model is fine-tuned using either only the novel class instances or balanced dataset containing training instances of both base and novel classes. Wang et al. (2020) shows that simple two-stage fine-tuning approach outperforms other complex methods. Much work seeks improvements by applying few-shot classification tech- niques. Kang et al. (2019) designs meta-model that learns to reweight pre-trained features given few-shot data. Wang et al. (2019) and Yan et al. (2019) further explore the meta-learning direction by attaching meta-learned classifiers to Faster R-CNN. Wu et al. (2020) improves few-shot detection by positive sample refinement.
standard
Relatively little work adjusts the proposal process, which is usually seen as robust to few-shot issues because there are many base examples. One possibility is to introduce attention mechanisms and feed category-aware features instead of plain image features into the proposal process (Hsich al., 2019; Fan et al., 2020; Xiao & Marlet, 2020; Osokin et al., 2020), as well as re-ranking proposals based on similarity with query images (Hsieh et al., 2019; Fan al., 2020). Making the RPN category-awas aware improves the quality of novel class proposals, but at inference time the model suffers from catastrophic forgetting of base categories current category- awwre features cannot summarize the very large number of base class examples efficiently or accurately. An RPN that is generally well-behaved can still create serious trouble in the few-shot case by missing high IOU proposals for the novel classes during fine-tuning- the proposal neglect effect. We show that this problem is severe in the few-shot regime, and can be fixed by a carcfully constructed ensemble of RPNs without substantial loss of performance for the base classes.
Evaluating Few-Shot Detectors. The standard procedure is to compute average precision (AP) separately for novel and base categories for detector that is engineered to detect |Ca] + C.,) classes, typically using standard test/train splits and standard novel/base splits (Wang et al., 2020). This evaluation procedure the same in incremental few-shot detection (Yang al., 2020; Pérez- Rúa et al., 2020). This procedure makes sense, because in most applications an incoming test image could contain instances from both base and novel classes. Furthermore, the standard procedure exposes any catastrophic forgetting that occurs. However, other evaluation methodologies occur, and some detectors are evaluated using variant procedures, making the comparison of AP's s diffiuult. In one variant, the detector detects only the |C,, novel classes or only one novel class. In this paper, all reported results are for the standard procedure; when relevant, we re-implement and re-evaluate using the standard procedure.
3 OUR APPROACH
We believe that the proposal neglect effect is generic, and it applies to any detector that uses structure like the standard structure. For this reason, we focus on finding and fixing the effect within a standard state-of-ther -frt few-shot object detection framework, as described below.
Few-Shot Object Detection Framework. We use the few-shot detection setting introduced in Kang et al. (2019). We split the dataset into two sets of categories: base classes C. and novel classes On- As shown in Figure 2, the training process is two-phase: (1) base classes training, and (2) fine- tuning with novel classes. In phase 1, the model is trained with base class instances which results in a |Ca/-way detector. After base classes training, weights for novel classes are randomly initialized, making the classifier a ([C&) + (C,,)) way classifier. In phase 2, the model is fine-tuned using either set of few novel class instances or a balanced dataset containing both novel and base classes. After the fine-tuning phase. we evaluate our model by average precision (AP) on novel and base categories. Although the focus of few-shot detection is the novel classes, since most test images
3

---

Under review as a conference paper at ICLR 2021
treining
Box classifier
Backbone feature extractor
Region Proposal Network
ROI pooling
Many desses
Box regressor
Phase 2: Novel classes turintuinn
Box classifier
Backbone festure extractor
Region
Proposal Network
ROI pooling
Few-shot novel clesses
regressor
Figure 2: Illustration of two-phase training in Faster R-CNN. During phase all blocks trained. During phase only the top-layer classifier and bounding box regressor are fine- tne-tuned.
Single
Cooperating
Bounding regressor
Bounding
regressor
RPN feature extractor
RPN feature
classifier
Foreground/ background classifier
maximum suppression
classiffer
Non- maximum suppression
dlssiiierr
is the original structure for the RPN Faster R-CNN; right is our CoRPNs, consisting of cooperating box classifiers.
Figure 3:
contain instances from both base and novel classes, it is essential to maintain good performance on base classes.
We adopt the widely- used Faster R-CNN (Ren et al., 2015) as our base model. As shown in Figure 2, Faster R-CNN is a serial detector, which consists of backbone image feature extractor, regional proposal network (RPN), followed by the region of interest (ROI) pooling layer, and bounding box classifier and bounding box regressor on top of the model. The RPN determines if box is foreground or a background box. Following the RPN is non-maximum suppression (NMS) which ranks and selects top proposal boxes. After passing the ROI pooling layer, the predictor's head classifies and localizes each box. In phase the whole model is trained many-shot base class instances. Phase 2 fine-tunes the head classifier and the bounding box regressor with novel class instances; everything before and including the proposal generator frozen.
Learning Cooperating RPN's (CoRPNs). We wish to avoid high IOU boxes for novel classes being dropped by RPN when novel classes trained. Our strategy to train multiple redundant RPN's. The RPN's should be distinct, but cooperate - if one misses high IOU box, we want another find it. However, they should not be distinct that the classifier is flooded with false positives.
As Figure 3 shows, Faster R-CNN's RPN consists of feature extractor, binary classifier (which decides whether a box is foreground or background), and bounding box regressor (which is not relevant our current purpose). There no reason for our RPN's to use distinct of features. and we do not want to create problems with variance, so we construct redundant classifiers while keeping both the feature extractor and the bounding box regressor shared between all RPN's. In what follows, a reference to an RPN is actually reference to the RPN classifier, unless otherwise noted. An RPN with a single classifier is trained with a cross-entropy loss Lets LeE and produces a single prediction. In case, we train N different binary classifiers simultaneously, and must determine (1) what prediction is made at test time and (2) what gradient goes to what classifier at training time. At test time, given box gets the score from the most certain RPN. If the highest foreground probability is closer to one than the highest background probability, the box foreground; otherwise, is background.

---

Under review as a conference paper at ICLR 2021
Training time is more interesting. Merely taking the gradient of the best RPN score is not good enough, because we may find that one RPN scores all boxes, and the others do nothing interesting. For any foreground box, we want at least one RPN to have very strong foreground score and all others to have good foreground scores too (so that no foreground box is missed).
We use the following strategy. For a specific anchor box i, each RPN j outputs a raw score r/, indicating if the box is a foreground box or not: r; = [r] r?....,,,;" After applying a sigmoid, the jth RPN produces the foreground probability f = (r?) for anchor box i. We choose the score from the j'th RPN such that j argmin, min(f!,
}. namely the most certain RPN which produces probability closest to the edge of the [0, 1] interval. At training time, only the chosen j th RPN gets the gradient from anchor box i. The RPN selection procedure is per-box, even adjacent boxes can pass through different RPN's.
Other than the standard cross-entropy loss, we use two additional loss terms: a diversity loss Loit encourages RPN's to be distinct, and a cooperation loss Looop encourages cooperation and sup- presses foreground false negatives. The final loss Ldse == L' + Aa + XeCcoop, where Xa and A, are trade-off hyperparameters.
Enforcing Diversity. We do not want our RPN's to be too similar. For each positive anchor box, RPN responses should be different because want ensure that novel class training time, if one
RPN misses a high IOU anchor box, another will find it. To this end, we propose a methodology to enforce diversity among RPN's. Given a set of N A anchor boxes, the N RPN's produce an N The covariance matrix [(F) is E(F
by NA matrix of probabilities F = f f fNj
E[ E (f E[)* We define the diversity loss Ldiv by the log determinant loss Ldse
log (IF)()
By the diversity loss, we encourage the probability matrix have rank N, so each RPN is reacting differently on the collection of NA boxes. This procedure ensures each RPN to be the most certain RPN for some boxes, so that every RPN being selected and trained. Omitting this loss can cause an RPN to receive no or little training.
Learning to Cooperate. We also want the RPN's to cooperate so that they all agree to a certain extent for foreground boxes. We propose a cooperation loss to prevent any RPN from firmly reject- ing any foreground box. For foreground box i, with the jth RPN, we define the cooperation loss Cidop ;= max(0, Ø - }. where é is a constant parameter (usually less than 0.5), acting as a lower bound for each RPN's probability assigning to foreground box. If RPN's response is below Φ, that RPN is going to be penalized. The final cooperation loss is an average of cooperation losses over all foreground boxes and all RPN's.
4 EXPERIMENTS
Datasets and Implementation Details. We evaluate our approach on two widely-used few-shot de- tection benchmarks: MS-COCO (Lin et al., 2014) and PASCAL VOC (07 + 12) (Everingham et al., 2010). For a fair comparison, et al. (2019); Wang et al. (2020) to train and evaluate all models. On COCO, we report base/novel classes AP, AP50, and AP75 under shots 1, 2, 3. 5, 10, and 30. On PASCAL VOC, we report AP50 for three different base/novel classes splits under shots 1, 2, 3, 5, and 10. Consistent with recent works (Yan al., 2019; Wang et al., 2020), we mainly use an ImageNet pre-trained (Russakovsky et al,,, 2015) ResNet- 101 architecture (He et al., 2016) as the backbone, unless otherwise noted. Fol- lowing Wang et al. (2020), for all TFA and CoRPNs results, ground-truth boxes are appended as training samples. As reported in Wang et al. (2020), adding the ground-truth boxes leads to a 0.5% AP gain on COCO. More implementation details can be found in the Appendix.
the same train/test splits and novel class instances as in Kang
Hyperparameters on PASCAL VOC. For ease of comparison, we use the same values of all shared training and fine-tuning hyperparameters (batch size, learning rate, momentum, weight decay, etc.) as Wang et al. (2020). CoRPNs have the following additional hyperparameters: the number of RPN's, the cooperation loss threshold Φ, the diversity loss trade-off Xd, and the cooperation loss trade-off Ag. The number of RPN's does not affect the results much as long other hyperparameters are suitably selected (Figure 4). For results described here, we use the following hyperparameters sets: VOC split 1, RPNs, & 0.2, X = 2, da= 0.025; VOC split 2, RPNs, 0.1, X 1, dd
5

---

Under review as a conference paper at ICLR 2021
0.025; VOC split 3, 2 RPNs, á o= 0.5, de- 1, A 0.3 (more information good ranges in the Appendix). We selected hyperparameters using base classes and the following criteria on PASCAL VOC: (1) The average number of RPNs responding positively to foreground box (a larger average means that the RPN's are cooperating well with each other by responding positively to foreground boxes); (2) The cumulative variance in the RPN's response matrix on foreground boxes (details in Appendix); (3) The average number of positive samples (larger means that the CoRPNs produce more high IOU boxes). Because the hyperparameters are selected on base classes, we can use different set per split without concerns about generalization.
Hyperparameters on COCO. For COCO, we selected hyperparameters that worked well on all three novel splits in PASCAL VOC and used those. This means that improvements in COCO are not caused by hyperparameter variance. Results reported were obtained with 5 RPNs, & 10:0..m 0.3, de 1, Aa = 0.025. We report results for two other hyperparameter sets in the Appendix.
Baselines and Evaluation Procedure. To investigate the proposal neglect effect and for a fair comparison, we mainly focus on comparing against the state-of-the-art baseline TFA (Wang et al., 2020). Our approach introduces CoRPNs into TFA, while keeping other model components and design choices unchanged. In addition, we thoroughly compare with a variety of recent few-shot detectors, including CoAE (Hsich et al., 2019), Meta R-CNN (Yan et al., 2019), FSOD (Fan et al., 2020), MPSR (Wu et al., 2020), FSDetView (Xiao & Marlet, 2020), and ONCE (Pérez-Rúa et al., 2020). These baselines address other aspects of few-shot detection which are different from us (Section 2), and their modifications evaluation follows the standard procedure in Wang et al. (2020). As explained in Section 2, this standard procedure computes AP separately for novel and base categories for detector that is engineered to detect |Col + [C,l classes. For a fair comparison, we re-evaluate CoAE (Hsich et al., 2019) and FSOD (Fan et al., 2020) using the standard procedure.
thus largely orthogonal to our effort here. Note that our
Main Results. Tables 1 and 2 summarize the results for novel classes on COCO and PASCAL VOC, respectively, and Table summarizes the results for base classes on COCO.
Comparisons with the Main Baseline TFA on Novel Classes. From Tables 1 and 2, we have the following important observations. (I) Our approach produces substantial improvement in AP over TFA (Wang et al., 2020) on novel classes in the very low-shot regime (1, 2, and 3 shots), and marginal improvement or sometimes slight degradation in the higher-shot regime. These improvements are manifest for both existing benchmarks. Interestingly, on the more challenging COCO dataset, our improvements over TFA are consistent across different shots (except slight degradation in 30 shot under AP75 with fully-connected classifier). (II) We investigate two types of classifiers: either fully- connected (denoted as 'fc' in Tables and 2) or cosine (denoted as 'cos' in Tables 1 and 2). Note that our approach obtains improvements regardless of classifier choice. This is because CoRPNs is a strategy to control variance in the estimate of classifier parameters that applies independently of the classifier. Any high IOU box missing from the RPN output in the training phase must cause variance for the few-shot regime. Because there are very few such boxes, the effect of not losing boxes is pronounced. We provide visualization comparisons of detection results in the Appendix.
Is Proposal Neglect Occurring? Of course, CoRPNs might outperform TFA for some obscure reason. But there is good evidence that proposal neglect occurs, and that CoRPNs help. Table records the average number of foreground boxes at three IoU's produced by CoRPNs and by TFA at the fine-tuning stage for novel classes. This table shows that CoRPNs produce more (and better) foreground boxes than TFA. We expect these small improvements in the box proposal pool to have a large effect, because such boxes improve the classifier's model of appearance variation for novel classes; small changes in this model can result in significant movements of the class boundaries.
Comparisons with Other State-nf-the-Art Approaches. With our simple modification on RPN, we also outperform other sophisticated approaches on both benchmarks in the very low-shot regime, and achieve comparable performance in the higher-shot regime. In particular, we significantly outper- form those baselines that introduce attention mechanisms for adjusting proposal generation (Hsieh et al., 2019; Fan et al., 2020). For other approaches that improve few-shot detection from different perspectives, such as exploiting better multi-scale representation (Wu et al., 2020), our approach can be potentially combined with them for further improvements.
Comparisons Base Classes. While improving detection novel classes through fine-tuning. we maintain strong performance on base classes without suffering from catastrophic forgetting as

---

Under review as a conference paper at ICLR 2021
shown in Table 4. By contrast, the performance of the oftte-of-the-arr baselines dramatically drops,
demonstrating that they cannot simultaneously deal with both novel and base classes.
I-shot
2-sbot
3-shot
Method
Backhone
AP
AP50
AP75
AP
AP50
AP75
APSD
AP75
CaRPNs w/ fc
ResNer-101 101
3.4
6.7
3.0
5.4
10.4
5.1
7.1
13.7
6.8
Ours
CoRPNs w/ cos
ResNet-101
4.1
7.2
4.4
5.4
9.6
5.6
7.1
13.2
7.2
CoRPNs w/
55
3.7
6.8
3.8
4.7
8.8
4.4
6.3
12.0
6.0
Main baselines
TFA w/ To (Wang ot al. 2020)
ResNor-101
2.9
5.7
2.8
4.3
8.5
4.1
6.7
12.6
6.6
TEA w/ cos (Wang et 2020)
ResNer.100 101
3.8
4.6
8,3
6.5
Meta R-CNN (Yan et ol., 2019)
ResNer- 101
FRCN+St-fall (Wang et al., 2020)
ResNet-101
Other buselines
MPSR (Wu et al,, 2020)
ResNet-101
2.3
4.1
2.3
3.5
6.3
9.5
5.1
DeDetView (Xiao & Marlet, 2020)
ResNet-101
2.9
8,3
1.2
3.7
10.3
1.6
4.7
12.9
2.0
ONCE Prrre-RRaa et at., 2020)
Re
0.7
FSOD* ot 2020)
ResNot-50
4.8
2.0
2.9
5.9
3.3
5-shol
10-sbot
30-shot
Method
Backbome
AP
AP50
AP75
AP
AP50
AP75
AP
AP50
AP75
CaRPN: fc
ResNer. 101
8.6
10.5
20.2
13.5
25.0
12.9
Ours
CoRPNs w/ cos
ResNet-101
8.8
16.4
8.7
10.6
19.9
10.1
13.9
25.1
13.9
CaRPNs w/ cos
ResNet-50
7.8
14.4
7.6
9.0
17.6
8.3
13.4
24.6
13.3
w/ (Wang et 2020)
ResNet-101
8.4
10.0
19.2
13.2
Main baselines
TFA w/ cos (Wang et al. 2020)
ResNet-101
8.3
15.3
8.0
10.0
19.1
9.3
13.7
24.9
13.4
Meta R R-CNN (Yan et ol., 2019)
ResNet.101
8.7
6.6
12.4
10.8
FRCN+B-ful (Wang et al., 2020)
ResNet-101
9.2
12.0
Other haselines
MPSR (Wu et al., 2020)
101
6.7
12.6
6.4
9.7
18.0
9.4
13.7
25.0
13.4
VeDerView (Xiao & Marlet, 2020)
ResNet-101
5.8
15.6
2.9
6.7
17.3
3.7
9.6
22.1
6.6
ONCE (Pérez. Rúa et al., 2020)
R0
1.0
1.2
FSOD* (Fan et al., 2020)
ResNet-50
4.2
8.2
4.0
4.3
8.7
3.8
5.4
10.4
5.0
Few-shot detection performance on COCO novel classes. The upper row shows the 1, 2. 3-shot results, and the lower shows the 5, 10, 30-shot results. Results in red are the best, and results in blue the second best. All approaches are evaluated following the standard procedure Wang et al. (2020). *Model re-evaluated using the standard procedure for a fair comparison. the corresponding paper. Note that the publicly released models of ONCE and FSOD are based on ResNet-50; we include our CoRPNs based on ResNet-50 as well for a fair comparison. CoRPNs consistently outperform state of the in almost all settings, with substantial improvements especially in the very few-shot regime. Our strategy is also effective regardless of classifier choice.
Table I:
denotes that numbers are not reported in
Novel Set
Nuvel Sa: 2
Nood Sat 3
Mathed
shrest
Ours
CoRPNs wi (Oun))
CURPNA w/eee (Oun)
40,5
44.4
44.5
38.5
45.7
46.4
55.1
$4.1
54.8
55.7
25.7
2044
29,2
29.5
36.3
37.3
36.2
36.5
41.5
41.3
29.4
35.8
41LA
41.8
44.7
44.5
51.6
51.7
49.6
49.9
Main beaclines
TEA wi cos (baseline) (Wang et 2020)
(baseline) (Wang et 2020)
39.5
36.8
36.1
29.1
43.6
44.7
55.7
55.7
57.0
56.0
23.5
18.2
29.0
26.9
38.1
35.1
39.0
39.1
23.7
30,8
34.8
41.5
41.8
43.7
49.5
493
FRCN+fi-full (Wang 2009)
15.2
20.3
79.0
40.1
45.5
134
20.5
31.4
38.8
19.6
208
26.7
41.3
42.1
Mera
2019)
19.9
35.0
45.7
51.5
10.4
19.4
14.3
1822
23.5
41.1
Other boselines
CaAE (Hoich et al. 2019)
12.7
14.6
148
18.2
21.7
4.4
11.3
20.5
15.0
19.0
6.3
76
9.5
15.0
39.0
MPSR (Wu et al., 2020)
41.7
43.1
51.
55.2
61.
24.4
29.5
39.2
39.9
47.5
35.6
40.6
42.3
48.0
49.7
F-DDaViox (Xiuo Marks, 2000)
24.2
35.3
42.2
49.1
574
37.2
43.8
Table 2: Few-shot detection performance (AP50) on PASCAL VOC novel classes under three base/novel splits. All models are based on Faster R-CNN with a ResNet- 101 backbone. Results in red are the best, and results in blue are the second best. We follow the standard evaluation procedure in Wang et al. (2020). *Model re- evaluated under the standard procedure. CoRPNs outperform all the basclines in the very low shots, and achieve comparable performance in the higher shots.
VOC Split
VOC Split 2
Method
Avg 4FG (0.5)
Avg AFG (0.6)
Avg MFG (0.7)
Avg AFG (0.5)
Avg MFG (0.6)
Avg AFG (0.7)
CoRPNs
18.67
7.85
2.54
14.14
6.27
TFA (Wang et al., 2020)
17.61
7.41
2.37
13.61
5.86
2.03
VOC Split 3
COCO
Avg 4FG (0.5)
Avg #PG (0.6)
Avg #PG (0.7)
Avg #FG (0.5)
Avg #FG (0.6)
Avg #FG (0.7)
CoRPNs
16.02
6.65
2.30
11.97
5.41
1.84
TFA (Wang et al., 2020)
16.76
6.75
2.13
11.55
5.32
1.79
Table 3: CoRPNs produce more and better boxes at the fine-tuning stage. This table shows the average number of foreground boxes TFA and CoRPNs produce for novel classes in the 1-shot experiments. 'Avg FG' is cal- culated from proposals after non- maximum suppression. We exclude the ground truth box, which the classifier always sees, in calculating 'Avg # FG'. At each of three loU thresholds, for two of three splits in PASCAL VOC and for COCO, CoRPNs reliably produce more boxes. This table uses the same hyperparameters in Ta- bles and 2. The relative improvement is generally large; we expect small improvements to have large effects, because the classifier must use the very few boxes it receives to build a model of variation in object appearance.
7

---

Under review as a conference paper at ICLR 2021
1-shot finetuned
2-shot finetuned
3-shot linetuned
Method
Backhone
AP
AP50
AP75
AP
AP50
AP7S
AP
AP50
AP7S
CoRPNs w/ cos
RexNet-101 101
34.1
55.1
36.5
34.7
55.3
37.3
34,8
55.2
37.6
Ours
CoRPNs w/ cos
ResNet-50
32.1
52.9
34.4
32.7
52.9
35,5
32.6
52.4
35.4
Main buseline
TFA w/ cos (Wang et al., 2020)
ResNet-101
34.1
54.7
36.4
34.7
55.1
37.6
34.7
54,8
37,9
MPSR (Wu et al., 2020)
ResNet- 101
12.1
17.1
14.2
14.4
20.7
16.9
15.8
23.3
18.3
Other havelines
ViDw (Xiao & Marlet, 2020)
ResNet-101
1.9
5.7
0.8
2.7
8.2
0.9
3.9
10.8
2.0
FSOD*
2020)
RosNet-50
11.9
20.3
12.5
24.4
17.2
17.4
19.0
5-shot finetumed
10-shot linctuned
30-shot finetumed
Method
Backhone
AP
AP50
AP75
AP
AP50
AP75
AP
AP50
AP75
CoRPNs w/
ResNet-101
34.7
54.8
37.5
34,6
54.5
38.2
35.8
55.4
39.4
Ours
CoRPNs w/ cos
ResNet-50
32.3
51.6
34.9
32.7
51.9
36.0
33.5
52.7
37.0
Main baseline
TEA w/ cos (Wang et al., 2020)
RexNet- 101
34.7
54.4
37.6
35.0
55.0
38.3
35.8
55.5
39.4
MPSR (Wu et al., 2020)
RexNet- 101
17.4
25.9
19.8
19.5
29.5
21.9
21.0
32.4
23.4
Other baselines
FsDetView (Xinn & Marlet, 2020)
ResNet-101
5.3
14.2
2.8
6.4
15.9
4.1
9.0
20.6
6.7
FSOD* (Fan et st., 2020)
ResNet-50
16.7
26.2
18.3
18.9
29.3
20.7
18.8
29.4
20.1
Table 4:
Detection performance on COCO base classes after fine-tuning with k-shot novel classes instances.
*Model re-evaluated under the standard procedure (Wang et al., 2020). Our CoRPNs and TFA (Wang et al., 2020) maintain good performance base classes, whereas MPSR (Wu et al., 2020) and FsDetView (Xiao & Marlet, 2020) suffer from severe catastrophic forgetting.
Method
AP50
Method
AP50
TFA (Wang et al., 2020)
30.8
2 RPN's, Dvornik et al. (2019)
32.4
2 RPN's CoRPNs
35.8
2 RPN's, Bootstrapping
31.8
2 RPN's Diversity Loss Only
28.4
2 RPN's, CoRPNs (Ours)
35.8
2 RPN's Cooperation Loss Only 27.2
Table 6: Our diversity term - the log-determinant loss offers improvements over the pairwise cosine sim- ilarity based diversity loss in Dvornik et al. (2019), and bootstrapping baseline. The table shows novel class AP50 of all models under PASCAL VOC novel split 3, shot 1.
Table 5: Our diversity loss and cooperation loss are both required for CoRPNs to obtain an improvement. The table presents novel class AP50 of CoRPNs with diversity or cooperation loss only, under PASCAL VOC split 3, shot 1.
Ablation Studies. We conduct a series of ablations to evaluate the contribution of each component and different design choices. Specifically, we show that (1) with our novel loss terms, CoRPNs outperform naive RPN ensembles; (2) CoRPNs outperform bootstrapping bascline; (3) CoRPNs outperform baselines with larger RPN sub-networks; (4) CoRPNs outperform an existing cosine loss based diversity method; (5) our cooperation loss is helpful; (6) the number of RPN's does not matter very much; (7) CoRPNs need both the diversity loss and the cooperation loss.
CoRPNs vs. Naive RPN Ensembles. We compare CoRPNs with an ensemble of the same number of RPN's trained separately, with different initializations. At the fine-tuning phase, we apply the same RPN selection mechanism both approaches. Figure 4 shows that CoRPNs (middle, with selected hyperparameters) outperforms naive ensembles (right) across different numbers. This suggests that pure redundancy does not lead much improvement, confirming that our loss terms enforce diver- sity and cooperation. Additionally, since each model in naive ensemble is trained separately, the number of parameters and training time of our CoRPNs are N times less than naive ensembles.
CoRPNs vs. Bootstrapping. We compare CoRPNs with a bootstrapping baseline in Table 6, with implementation details include Appendix. CoRPNs outperform the bootstrapping baseline, sug- gesting that RPN variance obtained by training data variation does not lead to much improvement.
CoRPNs vs. Larger RPN. We compare CoRPNs with baselines having larger RPN sub-networks in Table 7. We find that using larger RPN sub-networks does not improve performance, suggest- ing that the advantage of CoRPNs is not simply the result of using more parameters. Please find implementation details in the Appendix.
CoRPNs vs. Cosine Loss Diversity. Table confirms that in this context, our diversity loss-the the log- determinant loss- outperforms another diversity loss introduced in Dvornik et al. (2019). Dvomik et al. (2019) encourage diversity among classifiers by cnlarging the cosine distance in classifier outputs. We replace our diversity loss term by pairwise cosine similarity loss among RPN's (for N RPN's, the pairwise cosine similarity loss is an average over N(N - 1)/2 pairs).
Cooperation loss on Avoiding False Negatives. Our cooperation loss Lcoop pushes all RPN's to agree on a certain degree for all foreground boxes by having a threshold hyperparameter ø. Table shows that Leoop reduces the number of false-negatives, and felse-negatives decreases as ø increases.
8

---

Under review as a conference paper at ICLR 2021
CoRPNs, same hyperparameters
CoRPNS, selected hyperparamettrs
selected
Nalve Ensemblee
30-
36
36-
35
Nover AP50
34-
34-
34
33
12
32
32
31
31
31
29
29
29
(TFA22
(TFA22
Number of RPNs
Numbbr of RPNS
Number RPNS
Figure 4: Effect of the number of RPN's on novel AP50, under PASCAL VOC novel split 3, shot 1. The left shows results using same hyperparameters (ò. Au and X.). There is considerable variance across the number RPN's. The middle shows results of selected hyperparameters (with selection criteria discussed earlier this section) for different numbers of RPN's. CoR with any number RPN's outperform TFA (number of RPN = 1), and the variance is much smaller than on the left. The right shows naive ensemble results. Naive ensemble improves the performance but not good as CoRPNs with selected hyperparameters (middle).
Method
AP50
Avg # FN ())
Avg # FG (t)
Method
#param added
AP50
TEA, original
30.8
TEA (Wang et al., 2020)
28.9
3.1
17.6
TFA, Large RPN
28*256
25.9
= 0.1
29.5
2,4
21.3
TFA, Larger RPN
268+256
27.8
31.5
18.5
CoRPNs, RPN's
3*256
35.8
CoRPNs w/ 6= 0.5
32.2
2.5
18.3
CoRPNs, RPN's
12*256
34.8
0.7
26.8
1.3
20.3
CoRPNs, 10 RPN's
27*256
35.7
0.9
31.7
19.0
Table 8: Our threshold &i controls the average number of false- negative foreground boxes and the number of foreground sam- ples. 'Avg # FN' is the average number of novel class train- ing foreground boxes misclassified by the RPN classifier (cal- culated before non-maximum suppression). 'Avg # FG' is the same as 'Avg FG (0.5)' Table 3, which is the aver- age number of foreground boxes after non-maximum suppres- sion. When ó gets larger, all RPN's produce higher scores for all foreground boxes, CoRPNs get foreground boxes wrong. so 'Avg # FN de- creases. CoRPNs with all é values also have larger 'Avg # FG' compared with TFA, All results under PASCAL VOC novel split 1, shot 1. Different from other experiments, here we fine-tune with novel classes only (detector is C. (w-waay.
Table 7: This table shows novel class AP50
TF' (Wang et al., 2020) with a large RPN and an even larger RPN, and compares with CoRPNs with different numbers of RPN's. All results
with PASCAL VOC novel split 3, shot The second column presents how many additional parameters are intro- duced to the original TFA model. Us- ing larger RPN sub-networks does not im- prove the performance, while CoRPNs sig- nificantly improve with fewer added param- eters.
more and more unlikely that
The Number of RPN's. We present results with different numbers of RPN's in Figure 4. We find that the number of RPN's doesn't 't affect the result much with selected hyperparameters.
CoRPNs need both diversity and cooperation losses. Table 5 shows that using either loss alone does not improve the performance.
CONCLUSION
A substantial improvement in few-shot performance be obtained by engineering system of RPN's to ensure that high IOU boxes for few-shot training images almost always pass the system of RPN's. Our method achieves new state of the art on widely-used benchmarks and outperforms the current state of the art by a large margin in the very few-shot regime. This is because, as ablation experiments indicate, proposal neglect is real effect. We engineered our ensemble of RPN's by using diversity loss and cooperation loss, which gave us some control of the false negative rate. We showed that controlling proposal neglect is an essential part of building a strong few- shot object detector by showing that the state-of-the-art few-shot detector could be improved in this way. We do not claim that our method the best way to control proposal neglect and we plan to investigate methods that offer more delicate control. Furthermore, is very likely that proposal neglect applies to one-stage detectors like YOLO (Redmon & Farhadi, 2018), and future work will investigate this.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Alexey Bochkovskiy, Chien- Yao Wang, and Hong-Yuas Yuan Mark Liao. YOLOv4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004. 10934, 2020. 2
Hao Chen, Yali Wang, Guoyou Wang, and Yu Qiao. LSTD: A low-shot transfer detector for object detection. In AAAI, 2018. 3
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In ICCV, 2017.
Nikita Dvornik, Cordelia Schmid, and Julien Mairal. Diversity with cooperation: Ensemble methods for few-shot classification. In ICCV, 2019. 2,
Ian Endres and Derek Hoiem. Category independent object proposals. In ECCV, 2010.
Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman. The PASCAL visual object classes (VOC) challenge. IJV,, 88(2):303-338, 2010. 5
Qi Fan, Wei Zhuo, Chi-Keung Tang, and Yu- Yu-Wing Tai. Few-shot object detection with attention-rpn and multi-relation detector. In CVPR, 2020. 6, 7, 12
Li Fei-Fei, Rob Fergus, and Pictro Perona. One-shot learning of object categories. IEEE TPAMI, 28 (4):594-611, 2006. 2
Ross Girshick. Fast R-CNN. In ICCV, 2015. 2
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accu- rate object detection and semantic segmentation. CVPR, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Spatial pyramid pooling in deep convo- lutional networks for visual recognition. IEEE TPAMI, 190))1904-1996, 2015. 2
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In ICCV, 2017. 2
Ting-I Hsieh, Yi-Chen Lo, Hwann-Tzong Chen, and Tyng-Luh Liu. One-shot object detection with co-attention and co-excitation. In NeurIPS, 2019. 3, 6, 12
Bingyi Kang, Zhuang Liu, Xin Wang, Fisher Yu, Jiashi Feng, and Trevor Darrell. Few-shot object detection via feature reweighting. In ICCV, 2019. 3,5
Hei Law and Jia Deng. Comernet: Detecting objects paired keypoints. In ECCV, 2018. 2
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects context. In ECCV, 2014.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017. 2, 12
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander C. Berg. SSD: Single shot multibox detector. ECCV, 2016.
Anton Osokin, Denis Sumin, and Vasily Lomakin. OS2D: One-stage one-shot object detection by matching anchor features. In ECCV, 2020.
Juan-Manuel Pérez-Rúa, Xiatian Zhu, Timothy Hospedales, and Tao Xiang. Incremental few-shot object detection. CVPR, 2020. 3, 6,
Joseph Redmon and Ali Farhadi. YOL09000: better, faster, stronger. In CVPR, 2017. 2
Joseph Redmon and Ali Farhadi. 1004v:804..0211 02767, 2018. 2,9
YOLOv3: An incremental improvement. arXiv preprint
10

---

Under review as a conference paper at ICLR 2021
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: real-time object detection. In CVPR, 2016. 2
Unified,
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NeurIPS, 2015. 1,2,4
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJV,, 115(3):211225 1-22, 2015. 5, 12
Eli Schwartz, Leonid Karlinsky, Joseph Shtok, Sivan Harary, Mattias Marder, Sharathchandra Pankanti, Rogerio Feris, Abhishek Kumar, Raja Giries, and Alex M. Bronstein. RepMet: Representative- metric learning for classification and one-shot object detection. In CVPR, 2019. 3
Sebastian Thrun. Lifelong learning algorithms. Learning to learn, 8:181-209, 1998.
Koen van de Sande, Jasper Uijlings, Theo Gevers, and Arnold Smeulders. Segmentation as selective search for object recognition. In ICCV, 2011. 2
Xin Wang, Thomas E. Huang, Trevor Darrell, Joseph E Gonzalez, and Fisher Yu. Frustratingly simple seo- object detection. In ICML, 2020. 2, 3, 5, 6, 7, 8, 9, 12, 13, 14, 15
Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Meta-learning to detect rare objects. In ICCV, 2019. 3
Jiaxi Wu, Songtao Liu, Di Huang, and Yunhong Wang. Multi-scale positive sample refinement for few-shot object detection. In ECCV, 2020. 3, 7, 12
Yang Xiao and Renaud Marlet. Few-shot object detection and viewpoint estimation for objects in the wild. In ECCV, 2020. 3, 6, 7, 8, 12
Xiaopeng Yan, Ziliang Chen, Anni Xu, Xiaoxi Wang, Xiaodan Liang, and Liang Lin. Meta R-CNN: Towards general solver for instance-level low-shot learning. In ICCV, 2019. 6,
Ze Yang. Yali Wang. Xianyu Chen, Jianzhuang Liu, and Yu Qiao. Context-transfommmr: Tackling object confusion for few-shot detection. In AAAI, 2020. 3
Xingyi Zhou, Jiacheng Zhuo, and Philipp Krahenbuhl. Bottom-up object detection by grouping extreme and center points. In CVPR, 2019. 2

---

Under review as a conference paper at ICLR 2021
A IMPLEMENTATION DETAILS
Following Wang et al. (2020), we use Faster R-CNN as our base model and use an ImageNet pre- trained (Russakovsky et al., 2015) ResNet-101 with Feature Pyramid Network (Lin et al., 2017) as the backbone.
As discussed in the main paper, the training process of CoRPNs is two-phase. Our training and fine-tuning procedures are consistent with Wang et al. (2020). On PASCAL VOC, at phase I base classes training, each model is trained on the union set of VOC 07+ trainval data. Evaluation is on the VOC 07 test set. At the fine-tuning phase, each model is fine-tuned with balanced few-shot dataset sampled from VOC 07+12 that contains both base classes and novel classes On COCO, the fine-tuning phase is two-stage: at stage 1, we fine-tune the model on novel classes; at stage 2, we then fine-tune the model with a balanced few-shot dataset containing both base and novel classes.
We showed the effect of using different levels of o in ablation studies of the main paper. As discussed in the main paper, the number of RPN's doesn't 't affect the results much with selected hyperparam- eters. For 5 RPN's, we find that & E [0.1, 0.3] works best, and results are relatively stable for X E [0.5,2] and Xa E 0.1].
Evaluation. As mentioned in the main paper, we use the standard evaluation procedure for all the compared models for a fair comparison. We also compare against other approaches with the same novel classes instances and test images. On COCO, We fine-tuned and re-evaluated the publicly released models of MPSR (Wu et al., 2020) and FSDetView (Xiao & Marlet, 2020) with the same novel classes instances in Wang et al. (2020). In the standard evaluation procedure, when a test im- age comes in, the model has no assumption on what categories the image contains. We re-evaluated FSOD (Fan et al., 2020) and CoAE (Hsieh et al., 2019) using the standard evaluation. In Wang et al. (2020) and CoRPNs, the classifier is a (ICa) + |Cn, 1-way classifier. FSOD replaces it with class-agnostic 2-way classifier, which only determines if an object is foreground or background. At inference time, FSOD produces balanced number of proposals per novel category and con- catenates these proposals before NMS. In the standard evaluation, we produce balanced number of proposals for every category, including base categories. In CoAE, for test image containing a certain category, CoAE samples support image(s) from this category and collects boxes based on the support image(s). We modify the inference process by providing each test image support image(s) from each category and performing a forward pass for all categories. We then collect boxes from all categories and evaluate them. For a fair comparison, for the results reported in the main paper, we also fine-tuned FSOD and CoAE with the same novel category instance(s) as in Wang et al. (2020) and CoRPNs.
Training Time and Memory, The only architectural difference between CoRPNs and TFA (Wang et al., 2020) is that CoRPNs have multiple RPN classifiers. In TFA, the RPN's classifier is 1*1 convolutional layer, with input channels as the number of feature channels and output channels as the number of cell anchors. In TFA, the input channels are 256, and the output channels are 3. CoRPNs with 5 RPN's add four additional RPN classifiers to the model, 256$1 12 additional parameters in total. In our experiments, we find CoRPNs with 5 RPN's increase the training time by 3%, with roughly the same memory compared with TFA.
We mentioned the hyperparameter selection criteria in the experiment section. The second criteria is the cumulative variance in the RPN's response matrix to foreground boxes. Given set of M anchor boxes, the N RPN's produce an N by M matrix of probabilities F = f2......... NİT. We run a PCA on the F with N components and compare the cumulative percentage of variance explained by each of the selected components. We would like the variance to be distributed across components.
Selection Procedure- - Cumulative Variance.
Baseline - Bootstrapping. We compare CoRPNs with a bootstrapping baseline in Table 6. We construct the bootstrapping bascline by using multiple RPN classifiers, same CoRPNs. Instead of selecting the most certain RPN to get gradients, we randomly select an RPN to get gradients during training. In the fine-tuning stage and the inference time, we use the same selection procedure as CoRPNs (select the most certain RPN).
Baseline Larger RPN. We compare CoRPNs with two baselines with larger RPN sub-networks in Table 7. In CoRPNs, all RPN classifiers share the same RPN feature extractor, as shown in Figure 3. To have larger RPN sub-network, we need to enlarge the feature dimensions. We tried
12

---

Under review as a conference paper at ICLR 2021
1-shot
2,tbo
3-shot
Method
AP
AP50
AP73
AP50
AP73
AP
AP50
AP75
CoRENs w/ param (Table reported)
7.20
4.40
9.58
5.22
7.06
13.19
7.24
CoRPNs w/ oos, parm set
4,47
7.27
5.10
5.14
9.28
535
7.21
13.12
7.32
CoRPNs cos, param
6.97
5.53
12.89
CoRPNs w/ cos, Standard Devistion
0.20
0.13
0.36
0.12
0.25
0.11
0.13
0.15
0.26
CoRPNs w/ cos, 95% confidence intervals
4,2110222
7.15_0.15
4.5910.41
5,2310.14
9,27.1029
5.50.0.13
7.0610.14
13.06.1..15
7,09:0229
(Wang ol., 2020)
4.78
6,48
Table 9: We present COCO 1, 2, 3-shot results under three of hyperparameters. All hyperparameters selected from PASCAL VOC by the selection criteria described in the experiment section. This table also shows the standard deviation and 95% confidence intervals across three sets of hyperparameters. All results from CoRPNs outperform TFA, Results with 95% confidence intervals also show that the improvements are significant.
two options: large RPN where the RPN feature extractor's output channels increase from 256 to 272, and a larger RPN where the output channels double to 512. We also change the RPN classifier and bounding box regressor take in larger features for both options.
B PERFORMANCE VARIANCE ACROSS HYPERPARAMETERS
We provide the results from two other sets of hyperparameters on COCO and computed the mean, standard deviation, and 95% confidence intervals from the three sets of results. All three hyperpa- rameter sets are selected from PASCAL VOC, by the criteria described in the experiments section. As shown in Table 9, the improvement by CoRPNs is significant.
C RESULTS ON BASE CLASSES
CoRPNs use two-phase approach like TFA (Wang et al., 2020). Here we provide the base classes AP after phase 1 base classes training. As shown in Tables 10 and 11, our performance classes is comparable with TFA (Wang et al., 2020). Note that the setting of Tables 10 and 11 is different from that of Table 3 (COCO base classes) in the main paper, where the results are reported after phase 2 shot novel classes fine-tuning.
base
Method
Split
Split 2 Split
Method
AP
AP50
AP75
TFA w/ cos (Wang et al., 2020)
80.8
81.9
82.0
cos (Wang et al., 2020)
59.3
42.8
CoRPNs w/ (Ours)
81.2
81.8
82.4
CoRPNs w/ cos (Ours)
39.2
59.4
42.5
Table 10:
Base classes AP50 on PASCAL VOC after phase base class training. The same parameter set- ting applics to both models. Notice that ours is com- parable to the results of TFA.
Base classes AP, AP50, and AP75 on COCO after phase base class training. The same parameter setting applics to both models. Ours is comparable to the reported numbers of TFA.
Table 11:
D BASELINE: FINETUNE RPN
Table 12 provides results with RPNs fine-tuned in stage 2. In addition to fine-tuning the bounding box regressor and the classifier, we also fine-tuned the RPN and the ROI feature extractor.
Method
Novel Set 1, 1-shot Novel Set 2, 1-shot Novel Set 3, 1-shot
TFA (Wang et al., 2020) TFA, RPN fine-tuned
39.8
23.5
30.8
36.9
20.7
34.1
CoRPNs
44.4
25.7
35.8
Table 12: This table presents results from the RPN finetuned baseline, under PASCAL VOC three novel splits, all in 1-shot. Finetuning RPN improves the performance in split but degrades the performance in other splits. CoRPNs are the best in all cases.
with cosine classifiers PASCAL three splits, all on 1-shot. We also need to fine-tune the box head (the feature extractor in the ROI head) after the RPN. We found the results are not comparable to the TFA baselines;
13

---

Under review as a conference paper at ICLR 2021
E ERROR ANALYSIS
We provide some proposal and final detection visualizations of CoRPNs and TFA (Wang al., 2020) on PASCAL VOC. Specifically, Figure 5 presents the proposals and the detection results for cat test images from 1-shot fine-tuned models under base/novel split 3. Cat a novel category for this split. Compare to TFA, CoRPNs generates more cat boxes. However, since the few-shot classifier is weak on novel classes, and the classifier has engaged with strong priors with base classes, cats are often misclassified as other base categories, leading to performance drop. This suggests that if we have a better classifier, we can further improve the performance.
Figure 5: Top 10 proposals and final detection results of TFA (Wang et al., 2020) (proposal row 1; detection row 3) and CoRPNs (proposal row 2; detection row 4). All images here are cat (cat is a novel category) test images. Due to the proposal neglect effect, state-of-the-art TFA misses all cat boxes. By contrast, CoRPNs catch many more cat objects by eliminating the effect, thus significantly outperforming TFA. However, we also notice that CoRPNs still have difficulty of correctly classifying the proposals as cats, suggesting that improving the classifier will further boost the detection performance, which we leave as future work.
30
Figure 6: Detection results of TFA (Wang et al., 2020) (row 1 & 3) and CoRPNs (row 2 & 4) under PASCAL. VOC split shot (row 1 & 2), and split 3 shot 1 (row & 4). Novel classes shown here (hird, bus, cow, boat, cat, motorbike, sheep, sofa). TFA fails to detect many novel objects due to proposal neglect effect, while our CoRPNs catch many more novel objects by eliminating the effect.
14

---

Under review as a conference paper at ICLR 2021
BROWN
HESPO
BROWN
BROWN
BROWN
Figure 7: Detection results of TFA (Wang al., 2020) (row 1 & & and CoRPNs (row & 4 under COCO test sets (val2014). Novel classes shown are (airplane, bird, boat, bottle, bus, cat, cow, horse, motorcycle, person, train). TFA fails to detect many novel objects due to the proposal neglect effect, while our CoRPNs catch many more novel objects by eliminating the effect.
F PASCAL VOC VISUALIZATIONS
We provide novel object detection result visualizations of CoRPNs and TFA (Wang et al., 2020) PASCAL VOC in Figure 6. TFA reflects the proposal neglect effect and misses lots of novel objects, while CoRPNs detect many more novel objects.
G COCO VISUALIZATIONS
We provide additional novel object detection result visualizations of CoRPNs and TFA (Wang al., 2020) on COCO in Figure 7. Again, TFA fails to detect many novel objects due to the proposal neglect effect, whereas CoRPNs catches many more novel objects by eliminating the effect.
15


Reviews:
[{"review_id": "wjaaFt0igpF", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Post-review comments:\n\nAfter reading the reply I decided I will keep my low score though it hurts to do so for a paper into which the authors definitely invested a lot of energy. Here is the reason why:\n\nI think there are usually two ways in which a paper can make an important contribution: Through a new insight or through a new piece of modelling that will be widely used afterwards. I think in it's current form the paper presents neither.\n\nThe potential insight I see is proposal neglect. However even with the added experiments in Table 3 I find the results neither sufficient to prove the effect exists. The GT boxes are already added so overlooking objects while fine-tuning should not be a big issue. If it was it should have a bigger impact (improvements in Table 3 and on general performance are minimal).\n\nSo what about a widely usable piece of modelling? Despite the anecdotal motivation the presented method improves performance. So the question becomes: Will this be widely used? My prediction is that it won't. The improvement is rather incremental but the effort to use it is high. While the model is simple it comes with three additional hyperparameters which the authors tune individually for each experiment (Section 4, Hyperarameters VOC & COCO). This is the biggest issue I have with the method: It requires tuning a bunch of hyperparameters to achieve a marginal improvement.\n\nCompare this to TFA [1], the method the paper builds upon. TFA is built upon a very simple insight (fine-tuning the heads works better than comparing representations) and because the insight and model are easy to use they will be the basis for a number of follow-up works (e.g. this paper). I cannot see the same happen with the presented method as long as hyperparameters have to be tuned for each dataset and split. I am sorry but in my eyes this is bad practice!.\n\nTo me this means the method will likely be of no lasting value. While it is SotA in the one-shot case for now I think the insights and methods used for achieving this performance cannot be used by other groups to improve performance even more. \n\nSo what can I recommend the authors to do with the paper? I think there are two ways to increase the papers contribution: \n1. Study different effects and problems of RPNs in few-shot object detection. A better understanding will for sure help moving the problem forward. Even if the end result is: The RPN works surprisingly well. That would be a great insight as well in my eyes as it would free resources to address the other problems.\n2. Improve the method so it provides a significant gain without any additional hyperparameter tuning. At only 3% more computing time using it would probably be a no-brainer if it was not for the hyperparameter tuning.\n\n[1] Wanget. al ICML 2020, \"Frustratingly simple few-shot object detection\"\n\n\n-------------------------------\n1. Summary\n\nThe paper addresses the region proposal network (RPN) as a key component in few-shot object detection. Their analysis is centered around the concept of \"proposal neglect\": objects in the meta training set are ignored because no proposals are generated for them. They propose CoRPN, an assemble of RPN classifiers, to tackle the problem and present results on the original few-shot COCO and Pascal VOC benchmarks as well as ablation results.\n\n\n2. Strengths \n+ I really like the general motivation and the RPN is for sure a key component for good few-shot object detection.\n+ The proposed model and especially the diversity loss seem well chosen to address \"proposal neglect\".\n+ The method brings a small but consistent performance improvement over TFA.\n\n3. Weaknesses\n- I think the main flaw of the paper is that there is no quantitative analysis of \"proposal neglect\", only some qualitative examples in the appendix. I don't want to say that it does not exist it makes only sense that such an effect messes up meta training. But I am missing experiments demonstrating the effect and it's impact on performance. \n- If proposal neglect is such a big issue for meta-training, why not simply add the ground truth boxes as proposals? Many object detection frameworks have that option built in. Comparison to this baseline seems crucial to evaluate the value of the proposed method.\n- The new hyperparameters (N-rpn, phi, lambda_c, lambda_d) are apparently not fixed but chosen in a per-experiment basis as only ranges are provided in the appendix, not exact values. This makes it hard to judge the actual value of the method as hyperparameters have to be chosen in advance in real-world applications.\n- I find the ablations lacking for a number of reasons which is critical as this should be the section that allows assessing the impact of \"proposal neglect\" and the proposed method. Specifically I have three issues: \n-- I am not sure the proposed experiments are suited to grasp the effect of the original problem and the effect of the method. How does the method impact RPN recall? What about classification accuracy, false negatives etc.? Table 6 does attempt to provide this information but I find it hard to understand what exactly is reported. Probably what I am interested in is already there it's just not that clear yet?\n-- Ablations have only been performed on Pascal VOC and not on the more interesting and more challenging COCO dataset. Especially Table 6 would be interesting for COCO as well.\n-- Ablation results appear to be extremely noisy. Compare the differences between configurations in Figure 4, adding or removing a classifier should not lead to a 5% performance change especially if this effect is not consistent across splits and changes again when another classifier is added or removed. To be clear this is not the authors fault but a problem of the setup introduced in Kang et. al. 2018 using a fixed set of images for meta-training. This approach makes overfitting to the specific chosen examples a big concern. Now I think it is good the authors follow the established evaluation protocol but this design makes it rally hard to examine if results are valid. In the case of the presented ablation studies this effect definitely casts a large shadow of doubt on the significance of these ablations. \n\n4. Recommendation\n\nI think in it's current form this paper has to be rejected. At first I was excited to see someone addressing proposals in few-shot object detection but the paper sadly only addresses \"proposal neglect\" without any quantitative analysis or consideration of other potential problems. Furthermore due to the potentially large variability of results due to the fixed meta-training set (see variability in Figure 4) and potential hyperparameter variations between experiments it is hard to judge if the reported performance gains are robust and could be transferred to real-world applications. I do feel an investigation of the RPNs role in few-shot object detection is very worthwhile and the proposed method may in fact be valuable, however to make either of these two contributions a deeper analysis is needed.\n\n5. Questions/Suggestions\n- Provide quantitative analysis of the \"proposal neglect\" problem. Even better: Do a throughout assessment of the RPNs role and problems in few-shot object detection.\n- Compare to a simple baseline: Adding the gt bounding boxes to the proposals during meta-training\n- Please clarify if hyperparameters were the same for all experiments and report the chosen values in the main paper, not the appendix.\n- Run ablation experiments on COCO as well\n- Better explain Table 6\n\n6. Additional feedback\n\nNone, the paper is overall well written, figures are clear and understandable. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "wQEzPV6jaet", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors present a interesting, novel idea of promoting the diversity and cooperation among multiple RPNs for the problem of few-shot detection. They first identify a critical problem in few-shot detection, which is that existing RPNs can miss objects of novel classes and that their proposed boxes make very similar errors. In order to resolve this issue, the authors propose to utilize multiple RPNs in the same detection pipeline and include a diversity loss when training the RPNs, such that they provide diverse scores to the anchor boxes. In this way, the chance of all the RPNs missing an object becomes small. The authors further include a cooperation score to enforce the RPNs to provide more meaningful scores (rather than sparse scores just to promote diversity). The proposed approach is evaluated on COCO and Pascal, and the results demonstrate that the proposed approach does improve the detection performance, as compared to the state of the art, especially in very low-shot cases.\n\nOverall, this is a very well-written paper. The authors have identified the existing problem and motivated their work well. The proposed method is novel and well described, and the evaluation is thorough. \n\nThere is one thing that I am curious about and would hope to get the authors' insights. In the evaluation, the authors compare their proposed method with a naive ensemble approach, which combines the results of RPNs with different random initializations.  I am curious about the performance of a slightly less naive, yet standard ensemble method, i.e., bootstrapping the training set. This could potentially allow the RPNs to learn from slightly different data and then provide diverse outputs as well. \n\nI don't have any other major comments. Here are a few minor ones.\n\n- For the diversity loss, did the authors try only penalizing the case when the determinant is close to zero, as compared to making it large? It seems that the sign of the determinant shouldn't matter. The current choice is reasonable and results in a good detector, but I am just curious what the authors think.\n\n- How does one determine a good number of RPNs in general? The authors mention that this can be problem/dataset-dependent. In practice, should one rely on a validation set to choose this number? Or is there any exploratory analysis that one could perform to find a good (mostly not optimal) number?\n\n- Please increase the gap between Tables 4 and 5.\n\n\n----------- updates after reading author response -----------\n\nAfter looking at the authors' response, the revised paper, and the other reviews, I'd like to update my rating from \"Accept\" to \"Marginally below acceptance threshold\". While I still like the overall idea of this paper and I believe this is an interesting direction of research, the additional results in the revised paper actually raise more questions and there are issues that need to be addressed in this current study.\n\n1. Properly quantifying \"proposal neglect\"\n\nThe improvements shown in Table 3 are very small, which do not support the claim that the proposed method produce \"more and better boxes\". Table 3 might actually not be the correct way to assess the \"proposal neglect\" (or resolving proposal neglect). I would expect to see a more in-depth result analysis. For instance, when comparing with the SOTA TFA approach, for each image on average, how many more correct objects are detected due to the proposed approach being able to find boxes that TFA cannot. Something like this would clearly indicate the benefit of the proposed approach.\n\n2. Sensitivity to hyperparameters\n\nIn Fig. 4 (left), there is a huge sensitivity to the hyperparameter selection. For instance, the performance of the proposed approach would drop drastically by just changing the number of RPNs from 2 to 3. This creates uncertainty for people to use this approach. Furthermore, for the COCO results, when the hyperparameters are not selected based on this set, the improvements are pretty small as compared to TFA. I believe a more robust hyperparameter selection method would be needed for the proposed approach to thrive.", "extra_scores": {}, "extra_text": {}}, {"review_id": "P0PCz1qiJHm", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a few-show object detection architecture, which improves the 1st stage of two-stage detectors (R-CNN in particular). In a few shot setting, the existing approaches the region proposal generator may ignore some novel out-of-distribution classes as they have not been included at training time. The proposed method attempts to correct this by using many RPN's, and training them such that the gradient is only passed to one of them at a time; thus forcing them to learn mutually different kinds of regions.\n\nThe topic is relevant, and not too widely studied. Few-shot learning is well presented in the literature in the classification setup, but less so in detection. The idea is novel, although quite straightforward and somewhat arbitrary. I encourage the authors to clarify the insight as to why this particular combination of losses was chosen (I could think of alternative formulations). Also it would be important to see some discussion about the computational cost: for object detection, people often prefer simple and fast approachs (SSD, Yolo) that may be less accurate but are fast to execute. How much does the proposed method with N classifiers increase computation (something less than N-fold, but how much)?\n\nThe results reach state-of-the-art on some cases; especially with very-few-shot domain. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "9gmCP_Hvivd", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper argues that in the case of few shot object detection, the quality of region proposals is important, as if the region proposal network misses any of the very scarce positive box, the performance is severely impacted. It proposes to use multiple cooperating RPNs to alleviate the problem and improve few shot detection.\n\n\nPositives\n- The paper is easy to follow and proposes a simple idea of improving the RPN classifiers so that any proposal with high IOU with the object is not pruned out early due to the RPN. At the same time the RPN maintains pruning performance and does not send too many false positives.\n- The method proposed is clear\n- The evaluations are done on standard challenging benchmarks for the problem and good performances are reported\n\nNegatives\n- The thesis is simple and acceptable, i.e. for few shot detection training, missing any high IOU proposal can be very bad for performance, so improve the RPN to avoid that happening at that stage, however the proposed method is the best way to do it is not convincing. Why does what is being proposed to solve this problem of proposal neglect the right way to do so?\n- In particular there can be many other ways of making the RPN classifier better, e.g. by simply making the sub network bigger, which would be fair as in the proposed method also more parameters are added\n- There seems to be no empirical proof with ablation experiments, that the diversity and cooperation losses are useful for that task? What happens when we train the same without using, either or both of the losses (with acceptable settings of parameters)\n- In fig 4 all results with number of RPNs from 1 to 10 should be reported. The sudden jump from 5 to 10 seems a bit convenient for the claim that large number of RPNs lead to large number of FPs leaking reducing the performance. The performances of naive RPN ensembles can also be added to the bar plot (instead of selected numbers in tab. 4)\n- The choice of number of RPNs should be properly cross-validated/justified. Right now it feels a bit ad-hoc; in supplementary it is mentioned \"We mostly use five RPN’s, except for PASCAL VOC novel split 3, where two RPN’s lead to better performance.\" -- has this been tuned based on the performance on the test sets?\n- The hypothesis that large number of FPs degrade the performance for large number of RPNs, can be evaluated empirically also by computing the actual statistics for region proposals accepted or rejected.\n- Own baseline (same implementation details/strategy as CoRPN) with #RPN=1 should be reported for all tables. Right now the point of reference seems to be TFA, which is implemented elsewhere; the implementation here might already have slight benefits due to the parameter settings etc.\n\nOverall, the thesis is acceptable but the method is still not very convincing. The performances reported are state of the art in COCO (more for very few shots 1,2 etc. and marginal for larger), marginal for VOC low shots and not for higher shots. I am a bit lukewarm about the paper in general.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers have not supported the acceptance of this paper where the key weakness is that the study of the proposal neglect effect is not sufficient (see the reviews for the details). I agree with the assessment of the reviewers and recommend rejecting the paper in its current form.
Decision: Reject