Improving Aspect Ratio Distribution Fairness in Detector Pretraining via Cooperating RPN’s
Authors: Weilin Zhang
Xiang Li
Yu-Xiong Wang
David Forsyth
Venue: Submitted to ICLR 2023 (2023)
Keywords: Few-Shot Learning
Object Detection
Distribution Shift

Abstract: Region proposal networks (RPN) are a key component of modern object detectors. An RPN identifies image boxes likely to contain objects, and so worth further investigation.  An RPN false negative is unrecoverable, so the performance of an object detector can be significantly affected by RPN behavior, particularly in low-data regimes. The RPN for a few shot detector is trained on base classes.  Our experiments demonstrate that, if the distribution of box aspect ratios for base classes is different from that for novel classes, errors caused by RPN failure to propose a good box become significant.  This is predictable: for example, an RPN trained on base classes that are mostly square will tend to miss short wide boxes.  It has not been noticed to date because the (relatively few) standard base/novel class splits on current datasets do not display this effect. But changing the base/novel split highlights the problem. We describe datasets where the distribution shift is severe using PASCAL VOC, COCO, and LVIS datasets.
We show that the effect can be mitigated by training multiple distinct but cooperating specialized RPNs.  Each specializes in a different aspect ratio, but cooperation constraints reduce the extent to which the RPNs are tuned. This means that if a box is missed by one RPN, it has a good chance of being picked up by another.  Experimental evaluation confirms this approach results in substantial improvements in performance on the ARShift benchmarks, while remaining comparable to SOTA on conventional splits.  Our approach applies to any few-shot detector and consistently improves performance of detectors.

Under review as a conference paper at ICLR 2023
IMPROVING ASPECT RATIO DISTRIBUTION FAIRNESS IN FEW-SHOT DETECTOR PRETRAINING VIA COOPER- ATING RPN'S
Anonymous authors Paper under double-bling review
ABSTRACT
Region proposal networks (RPN) are key component of modern object detectors. An RPN identifies image boxes likely to contain objects, and so worth further investigation. An RPN false negative unrecoverable, so the performance of an object detector can be significantly affected by RPN behavior, particularly in low-data regimes. The RPN for few shot detector trained on base classes. Our experiments demonstrate that, if the distribution of box aspect ratios for base classes is different from that for novel classes, errors caused by RPN failure to propose good box become significant. This is predictable: for example, an RPN trained on base classes that are mostly square will tend to miss short wide boxes. It has not been noticed to date because the (relatively few) standard base/novel class splits on current datasets do not display this effect. But changing the base/novel split highlights the problem. We describe datasets where the distribution shift is severe using PASCAL VOC, COCO, and LVIS datasets.
We show that the effect can be mitigated by training multiple distinct but cooperat- ing specialized RPNs. Each specializes a different aspect ratio, but cooperation constraints reduce the extent which the RPNs are tuned. This means that if box is missed by one RPN, it has a good chance of being picked up by another. Experimental evaluation confirms this approach results in substantial improve- ments in performance on the ARShift benchmarks, while remaining comparable SOTA conventional splits. Our approach applies to any few-shot detector and consistently improves performance of detectors.
INTRODUCTION
Most state-of-ther of-th---t- object detectors follow a two-stage detection paradigm. A region proposal network (RPN) finds promising locations, and these passed through classifter to determine what, if any, object is present. In this architecture, if an RPN makes no proposal around an object, the object will not be detected. For few-shot detector, splits the classes into base and novel, then trains the RPN and classifier on base classes, fixes the RPN, and finally fine-tunes the classifier on novel classes using the RPN's predictions.
Objects in large-scale object detection datasets (e.g. COCO (Lin et al., 2014); LVIS (Gupta et al., 2019)) have typical aspect ratio that varies somewhat from instance to instance, and often differs sharply from category to category. As a result, the few-shot training procedure has a built-in problem with distribution shift. This phenomenon illustrated in Figure roughly square, and all novel classes are either short and wide, or tall and narrow. The RPN trained on the base classes should miss some novel class boxes. These boxes will have two effects: the training data the classifier sees will be biased against the correct box shape; and, at run time, the detector may miss objects because RPN failures. We refer to this problem as the bias (the RPN does not deal fairly with different aspect ratios). The bias occurs because the RPN sees few no examples of the novel classes during training (Kang et al., 2019; Wang et al., 2020; Yan et al., 2019). 19).
Imagine all base classes are
To date, this bias has not been remarked This is accident of dataset construction: the standard base/novel splits in standard datasets do not result in a distribution shift. But other base/novel splits do result in distribution shift large enough to have notable effects, and Section 3 shows evidence

---

Under review as a conference paper at ICLR 2023
Figure 1: RPN is severely affected by the distribution shift of object aspect ratios from base to novel classes, leading to degenerated few-shot detection performance. After training on the base classes which are mostly boxy objects (bi ke, chai r, table, e, tv, inima 1 s, etc.), (Left) state-of-the-an few-shot detector DeFRCN (Qiao et al., 2021) built on the conventional RPN misses the elongated novel class (train) object at the proposal stage and generates no proposal with 1oUgt > 0.7 this is a disaster (the classifier of DeFRCN will not see a t train box proposal, and so it cannot detect the cra in). By contrast, (Right) our CoRPN's remedy this issue and thus improve few-shot detection for DeFRCN. Red boxes are the groundtruth box of the novel class t train object; green boxes are box proposals output by the model. We plot positive proposals with IoUgt > 0.7 following Qiao et al. (2021).
that this effect occurs in practice. In particular, we describe ARShift benchmarks that simulate the real-world scenario where the aspect ratio distribution shift is severe. RPNs in state-of-the-ar few- shot detectors are heavily biased towards familiar aspect ratio distributions, and so have weaker than necessary performance on non-standard splits because their RPNs are unfair. Evaluation practice should focus on performance under hard splits.
In few-shot detection applications, a more robust RPN will be more reliable, because applications typically offer no guarantees about the aspect ratio of novel classes. We show how to build a more robust RPN by training multiple RPN classifiers to be specialized but cooperative. Our CoRPN's can specialize (and degree of specialization emerges naturally), but our cooperation constraints discourage individual RPN classifiers from overspecializing and so face generalization problems. CoRPN's s are competitive with SOTA widely used conventional benchmarks of few-shot detec- tion, using conventional splits. But on our ARShift benchmarks with hard splits based on PASCAL VOC, MS-COCO, and LVIS (Everingham et al., 2010; Lin et al., 2014; Kang al., 2019; Wang et al., 2020), they beat SOTA, because they are more robust to shifts in aspect ratio distribution.
Our contributions: (1) We show the bias has severe effects detector performance, and describe ARShift benchmarks that evaluate these effects. (2) We describe a general approach to improving RPN robustness to distribution shifts. Our CoRPN construction works with many types of few-shot detectors. (3) We show that performance improvements resulting from CoRPN's results from im- proved fairness. (4) Our CoRPN's are competitive with SOTA on widely used conventional bench- marks. But on the hard splits in ARShift, they beat SOTA, because they are fair.
RELATED WORK
Object Detection with Abundant Data. There are two families of detector architecture, both rely- ing on the fact that one can quite reliably tell whether image region contains an object independent of category (Endres & Hoiem, 2010; van de Sande et al., 201 1). In serial detection, a proposal pro- cess (RPN in what follows) offers the classifier a selection of locations likely to contain objects, and the classifier labels them. This family includes R-CNN and its variants (Girshick, 2015; Girshick et al., 2014; He et al., 2017: Ren et al., 2015) In parallel detection, there is no explicit proposal step; these methods can be faster but the accuracy may be lower. This family includes YOLO and its variants (Bochkovskiy et al., 2020; Redmon & Farhadi, 2017: Redmon et al., 2016; Redmon & Farhadi, 2018), SSD (Liu et al., 2016), point-based detectors such CornerNet (Law & Deng, 2018) and ExtremeNet (Zhou et al., 2019), and emerging transformer reasedeeerr--eeedd methods exemplified by DETR (Carion et al., 2020). This paper identifies issue with the proposal process that can impede strong performance when there is very little training data (the few-shot case). The effect is described in the context of two-stage detection, but likely occurs in one-stage detection too.
2

---

Under review as a conference paper at ICLR 2023
Few-Shot Object Detection. Few-shot detection involves detecting objects for which there are very few training examples (Chen et al., 2018; Kang et al., 2019; Schwartz et al., 2019), and sfate-of-the- art methods are usually serial (Wang et al., 2019; Yan et al., 2019; Wang et al., 2020; Fan et al., 2020; Wu et al., 2020; Xiao & Marlet, 2020; Yang et al., 2020; Li et al., 2021a; Hu et al., 2021; Zhang et al., 2021; Li et al., 2021b; Zhu et al., 2021). There is rich few-shot classification literature (roots in Thrun (1998); Fei-Fei et al. (2006)). Dvornik et al. (2019) uses ensemble procedures for few-shot classification. As to detection, TFA (Wang et al., 2020) shows that a simple two- stage fine-tuning approach outperforms other complex methods. Much work seeks improvements by applying different techniques, such as meta-learning (Wang al., 2019; Yan al., 2019; Hu al., 2021; Zhang et al., 2021), metric learning (Han et al., 2021; Wu et al., 2021; Yang et al., 2020), refinement (Wu et al., 2020; Li et al., 2021b), feature reweighting (Kang et al., 2019), semantic relations (Zhu et al., 2021), augmentation (Li & Li, 2021; Zhang & Wang. 2021), and margin loss (Li ct al., 2021a). Other work (Fan et al., 2021) alleviates forgetting of base classes. In particular. Qiao et al. (2021) achieves state-of-the-art performance by decoupling the gradient of the backbone and other components of the detector, as well as adding a prototypical calibration module. Here we focus on the two most representative methods- the state-of-the-ar DeFRCN (Qiao et al., 2021) and the widely used TEA (Wang et al., 2020) as our main baselines.
Few-Shot Detection Benchmarks. The existing literature can be seen as variations a standard detection framework, where one splits data into two sets of categories: base classes C. (which have many training examples) and novel classes C₂ (which have few). The RPN and classifier are trained with instances from the base classes, and then fine-tuned with the few-shot novel class data. While the choice of the split can be important in revealing different aspects of few-shot detection, existing benchmarks (Kang et al., 2019; Wang et al., 2020) have only focused on few fixed, rather arbitrary splits. However, we explore the scenario where there exists notable distribution shift between base and novel classes, and investigate the behavior of RPNs accordingly.
Proposal Process Few-Shot Detection. Relatively little work adjusts the proposal process, which is usually seen robust to few-shot issues because there are many base examples. (Sun et al., 2021) introduces contrastive-awar object proposal encodings to facilitate classification. Attention mechanisms are also introduced that feed category- into the proposal process (Hsieh et al., 2019; Fan et al., 2020; Xiao & Marlet, 2020; Osokin et al., 2020), well re-ranking proposals based similarity with query images (Hsieh et al., 2019; Fan et al., 2020). Making the RPN category- improves the quality of novel class proposals, but at inference time the model suffers from catastrophic forgetting of base categories current category-aware features cannot summarize the very large number of base class examples efficiently or accurately. An RPN that is generally well-behaver can still create serious trouble in the few-shot case by missing important proposals for the novel classes during fine-tuning. We show that the proposal process can be improved by carefully constructed cooperating RPN's without substantial loss of performance for the base classes.
features instead of plain image features
3 OUR APPROACH
We believe that improving the population of RPN boxes seen by the classifier in training will always tend to improve a detector, and so focus on finding and fixing the effect within a standard few-shot object detection framework. Our proposed strategy is general and can work with different types of few-shot detectors. Here we consider the two most representative methods: the state-of-the-ant DeFRCN (Qiao et al., 2021) and the widely used TFA (Wang et al., 2020). We first observe the box aspect ratio distribution shift problem through our pilot study, and show that naïve ensemble of RPN experts does not sufficiently solve this problem on our hard ARShift splits. Then we introduce our CoRPN's 's that effectively tackles the aspect ratio distribution shift problem via the cooperation and diversity losses.
3.1 BACKGROUND
We use the few-shot detection setting introduced in Kang et al. (2019). We split the dataset into two sets of categories: base classes C. and novel classes Cn. The training process is two-phase: (1) base classes training, and (2) fine-tuning with novel classes. In phase 1, the model is trained with base class instances which results in |Col-way detector. After base classes training, weights
3

---

Under review as a conference paper at ICLR 2023
Fander Spit
65
03
(a)
(h)
The choice of base and novel class split can have strong effect the distribution of box aspect ratios. This histogram on COCO (a) and LVIS (b) shows: (blue) the distribution of aspect ratios for base classes; (pink) randomly selected set of novel classes note the bimodal structure, suggesting that different classes might have quite different bounding box distributions; and (yellow) for each of our selected hard set of novel classes in COCO-ARShift and LVIS-ARShift Note that these classes are "rare" classes in LVIS so the histogram appears sparse. Experimental evidence (Tables 1, 2 and 3) suggests that standard RPN trained on the base set performs poorly on our novel classes, while our CoRPN's s address the issue and substantially improve the performance. We believe that our hard splits in ARShift are relatively easy to generate and examples are provided in the appendix.
Figure 2:
Method
shot=1
2
3
10
TFA w/ cos + CoRPN's
28.1
31.7
36.0
38.6
44.7
DeFRCN CoRPN's
37.0
49.3
55.1
56.7
59.0
TFA w/ cos (Wang et al., 2020)
19.1
27.2
28.0
34.4
42.1
DeFRCN (Qiao et al., 2021)
31.0
44.6
47.5
55.2
57.2
DeFRCN Ensemble of Experts
29.7
42.7
50.9
55.3
56.0
Table 1: Our proposed split VOC-ARShift causes serious problems for current state-of-the-art few- shot detectors, which are resolved by using CoRPN's. CoRPN's bring significant improvements (AP50) to all baseline models under distribution shift of base/novel class box aspect ratios. Note that naive ensemble of RPN experts sometimes leads even worse results, potentially due to the overspecialization of its individual RPNs. Results in bold are the better result between ours and the baselines. Results in red are the best.
for novel classes are randomly initialized, making the classifier a (|Col + [Cal]-way classifier. In phase 2, the model is fine-tuned using either a set of few novel class instances balanced dataset containing both novel and base classes. The classifier sees the ground truth box and RPN boxes; typically trained to regard RPN boxes with IoU>0.7 as positive, and with IoU<<0.3 negative. After the fine-tuning phase, we evaluate our model by average precision (AP) on novel and base categories. Although the focus of few-shot detection the novel classes, since most test images contain instances from both base and novel classes, it is essential to maintain good performance on base classes.
We adopt the widely-used Faster R-CNN (Ren et al., 2015) as our base model. Faster R-CNN is two-stage detector, which consists of a backbone image feature extractor, an RPN, followed by the region of interest (ROI) pooling layer, and a bounding box classifier and a bounding box regressor on top of the model. The RPN determines if a box a foreground or a background box. Following the RPN is non- manimaxmmmm suppression (NMS) which ranks and selects top proposal boxes. In phase 1, the whole model trained on many-sho base class instances. Phase 2 fine-tume tune-tunes part of the model on novel class instances with other parts frozen. Specifically. for TFA (Wang et al., 2020), only the top layer of the bounding box classifier and regressor are fine-tuned. For DeFRCN (Qiao et al., 2021), 1), the whole model is fine-tuned except for the convolutions in the bounding box classifier and regressor.
3.2 PIL.TT STUDY: BOx DISTRIBUTION SHIFT
In the real world, the bounding box distribution these novel categories often differs from the base categories, resulting in unfair few-shot performances. Namely, the difference in the distribution of
4

---

Under review as a conference paper at ICLR 2023
I-shot
2-shot
3-shot
Method
AP
AP50
AP75
AP
AP50
AP75
AP50
AP75
Ours
DeFRCN + CoRPN's
8.7
15.3
8.7
12.4
20.5
13.4
14.7
24.4
15,9
DeFRCN (Qiao et ul., 2021)
7.1
13.0
7.0
11.5
20.3
12.5
14.3
24.4
15.1
Baselines
DeFRCN Fasemble of Experts
8.0
14.3
7.9
12.3
20.6
13.0
14.2
23,5
15.0
Table 2: Our proposed split COCO-ARShift causes serious problems for current state-of-the-art few-shot detectors, which are resolved by using CoRPN's. CoRPN's consistently outperform the DeFRCN (Qiao et al., 2021) baseline in all cases (novel split in 1, 2 and 3 shots) and are superior to the ensemble of experts method overall. Results in red are the best.
Splir
Splic
Split
Entire test
Method
AP
AP50
AP75
AP
AP50
AP7S
AP
AP50
AP75
AP
AP50
AP75
Ours
DeFRCN CaRPN's
12.3
31.0
9.4
16.0
27.5
18.8
13.7
20.4
16.8
15.3
26.4
15.6
Baseline
DeHRCN (Qiao et 2021)
7.0
8.0
19.1
18.8
6,3
14.2
DeFRCN Emsemble of Esperts
11.5
26.0
8.4
14.8
24.3
12.1
12.4
20.1
14.7
14.8
25.0
13.9
3: Our proposed three splits on LVIS (LViS-ARShift) cause serious problems for current state-
of-the-art few-shot detectors, which are resolved by using CoRPN's. As reference, we also provide the results on all rare classes in the entire LVIS test set. Our CoRPN's outperform the baseline by large margins on all the splits with aspect ratio distribution shift, especially in AP75, suggesting that CoRPN's improve the quality of detection under aspect ratio shift. CoRPN's also marginally outperform the baseline the entire test set in mAp and AP75. Results in red are the best.
box scale, aspect ratio, and center location all inhibit a successful transfer. Previous work tries to alleviate the scale issues with multiscale features and the location issues with translation invariance of convolution. However, these approaches fail to solve all of these problems, especially when the distribution between base and novel class box aspect ratios has significant shift. We simulate this scenario by proposing new splits: our ARShift benchmark, on the PASCAL VOC, COCO and LVIS datasets that emphasize this distribution shift and the fairness to different aspect ratio distributions.
We manually select a set of classes that will likely have a different box distribution to the base cate- gories. Figure 2 shows that split has more significant shift in the box aspect ratio distribution. As an naive approach to alleviating this issue, we modified the RPN classifier to be an ensemble of expert RPN classifiers. Instead of using one RPN classifier for all proposals, we use 3 RPN classi- fiers for the anchors of 3 different aspect ratios (0.5, 1, and 2.0 respectively). The 3 RPN classifiers independently output their prediction for their respective anchors, which are combined as the final prediction. Tables 1, 2 and 3 show that compared to the baseline DeFRCN (Qiao et al., 2021). this approach represented by 'DEFRCN Ensemble of Experts' performs comparably and thus cannot fully address the issue. Intuitively, in this ensemble of experts method, individual RPN classifiers might be overspecializing and so facing generalization problems. Instead, we propose a method where we do not explicitly enforce each RPN classifier to specialize in an aspect ratio, but let the specialization emerge in the learning process. This method, named CoRPN's, ss, shows a large improvement over the baselines in Tables 1, and 3.
3.3 LEARNING COOPERATING RPN'S (CORPN'S)
We would like to alter the RPN to improve the population of boxes reported, especially on our ARShift benchmark where novel classes has a large box distribution shift. We expect that doing so affects mainly the few-shot case. As illustrated in Figure 3, we use multiple redundant RPN classifiers, but our goals imply that these RPN classifiers need be trained cooperate (i.e., they should not be a pure ensemble of experts). In what follows, we use the term RPN and RPN classifier interchangeably unless otherwise noted, In particular, we train and evaluate our RPN classifiers using an OR strategy gets the gradient during training. This has two effects. First, the RPN's can specialize though we do not allow the RPN's to drift too far apart. In turn, if one RPN misses a positive box, the other might find it. Second, the training strategy may improve the variation of proposals, which is especially essential when dealing with a different proposal box distribution in the few-shot case. Both effects may bring improvements to the model's 's faimess with respect to aspect ratio distribution.
box is classified with the label reported by the most confident RPN, which
degree,
Faster R-CNN's RPN consists of feature extractor, a binary classifier (which decides whether box is foreground or background), and a bounding box regressor (which is not relevant to our current purpose). We do not intend for our RPN's 's to use distinct sets of features, since it would introduce large number of additional parameters, so we construct redundant classifiers while keeping both the feature extractor and the bounding box regressor shared between all RPN's.
5

---

Under review as a conference paper at ICLR 2023
Single RPN
Cooperating RPN's
Bounding regressor
Boundiig regressor
RPN feature extractor
RPN feature extractor
Clessifeer
Foreground/ background clsssfee
Non maximum suppression
Non maximum suppression
dassifier
Figure 3: Illustration for our CoRPN's network. Left is the original structure for the RPN in Faster R-CNN; right is our CoRPN's, consisting of cooperating bounding box classifiers. Note that we only add multiple classifier heads that share features, so the model complexity CoRPN's 8 does not significantly exceeds regular RPN. For convenience, we use the term RPN and RPN classifier interchangeably.
An RPN with a single classifier is trained with cross-entropy loss Leis = LCE and produces a single prediction. In our case, we train different binary classifiers simultaneously, and must determine (I) what prediction is made at test time and (2) what gradient goes to what classifier at training time. At test time, a given box gets the score from the most confident RPN. If the highest foreground probability is closer to than the highest background probabilityy the box is predicted to be foreground; otherwise, is predicted to be the background. In training time, merely taking the gradient of the best RPN score is not good enough, because the model may collapse solution where one RPN scores all boxes, and the others do nothing interesting. For any foreground box, we want least one RPN to have very confident foreground prediction and all others have good foreground scores too (so that no foreground box is missed).
the trivial
We the following strategy. For specific anchor box i, each RPN (of the N RPN's) outputs raw score r/. indicating the box is foreground box or not: r; |r] r? "....., r:M After applying sigmoid, the jthh RPN produces the foreground probability f = (r²)) ) for anchor box We choose the score from the j'th RPN such that
f" = argmin, min }!},
(1)
namely the most certain RPN which produces probability closest the edge of the [0. interval. At training time, only the chosen j'th RPN gets the gradient from anchor box i. The RPN selection procedure is per-box, and even adjacent boxes can pass through different RPN's.
Other than the standard cross-entropy loss, we use two additional loss terms: a diversity loss Ldiv en- courages RPN's to be distinct, and cooperation loss Teoop encourages cooperation and suppresses foreground false negatives. The final loss is
Cela == diB + AdLaiv + AeCcoop
where Ad and de are trade-off hyperparameters.
3.4 ENFORCING DIVERSITY
We do not want RPN's to be too similar and prefer their specialization. For each positive anchor box, RPN responses should be different because we want different RPN's to cover different types propose loss function enforce diversity among RPN's.
of proposal boxes. To this end, Given a set of N anchor boxes, the N RPN's produce an N by NA matrix of probabilities F [f], f2 ..., NN The covariance matrix
[5k(F) E[ ( - "
(3)
We define the diversity loss Laiv by the log determinant loss
Laiv := log(det(2(F))).
(4)
This log determinant loss has been widely used in previous work (Boyd & Vandenberghe, 2008; Dhillon, 2008) to encourage diversity. By this diversity loss, we encourage the probability matrix to have rank N, each RPN is reacting differently ensures each RPN to be the most certain RPN for some boxes, so that every RPN is being selected and trained. Omitting this loss cause some RPN classifier to receive little training.
the collection of NA boxes. This procedure

---

Under review as a conference paper at ICLR 2023
Clas
Baie
L-sket
I-bbet
2-lact
3-skck
Method
AP50
AP
AP.
AP75
TRA one CaREN's
7.1
39.1
34.7
35.2
37.5
DeFRIN ColIN's
16.1
10.9
30,3
31.2
31.0
35.1
THA ass o 3000)
58
0.6
6.
98.1
56.7
54.3
37.6
14.7
54.8
319
Deftic e 2021)
9.5
163
10.9
90.3
65.7
51.2
36.5
47.9
15.1
MPSRT (Wo al.
4.1
23
35
63
34
32
51
121
173
142
14.
207
16.9
153
2333
18.3
RiDerVee (N0o Marke, 20200
8.9
1.4
133
23
6.7
18.6
28
3.4
7.0
100
4.4
11.
23
4.9
136
22
RSOD7 (tan.rrat, 50000
30
50
11.9
15.6
17.3
17.4
25.3
19.0
Table 4: CoRPN's not only improve performance on our hard split, but also perform comparably with state of the art on more traditional splits without forgetting the base classes, hence improving model fairness for aspect ratio distribution. CoRPN's mostly beat strong baselines in few-shot de- tection performance on the (extremely difficult) COCO novel classes task for 1, 2, and 3-shot cases. Results in bold are the better result between ours and the baselines. All approaches are evaluated following the standard procedure in Xiao & Marlet (2020). *Model re-evaluated using the standard procedure (with base and novel classes joint space) for fair comparison. * **Model evaluated using public code and pre-trained base classes detector.
Ratio
Novel Set
Novel Set 2
Novel Set 3
0.5
RPN2 99.68%
RPN5 99.72%
RPN2 98.64%
1.0
RPN3 99.68%
RPN2 99.67%
RPN2 99.83%
2.0
RPN1 99.67%
RPNI 99.59%
RPN1 10.75%. RPN2 89.25%
Table 5: Aspect ratio coverage of different RPN classifiers on VOC standard splits. For example, RPN/ refers to the first RPN classifier in the CoRPN's. This provides evidance that CoRPN's indeed learn to specialize in box aspect ratios, despite no direct supervision.
3.5 LEARNING TO COOPERATE
We also want the RPN's to cooperate so that they all agree to a certain extent for foreground boxes. We propose cooperation loss to prevent any RPN from firmly rejecting any foreground box. For foreground box i, with the jth RPN, we define the cooperation loss
Ceoap == max(0: 6 -
(5)
where é constant parameter (usually less than 0.5), acting as a lower bound for each RPN's probability assigning to foreground box. If an RPN's response is below o that RPN is going to be penalized. The final cooperation loss is an average of cooperation losses over all foreground boxes and all RPN's.
EXPERIMENTS
Benchmarks. We propose a new base/novel split on both the PASCAL VOC (Everingham et al., 2010), MS-COCO (Lin et al., 2014) and LVIS (Gupta et al., 2019) datasets to simulate the real- world scenario, where the novel class box distribution deviates from the base class counterpart. Our proposed split VOC-ARShift is similar to the conventional few-shot detection VOC split (Kang et al., 2019; Wang et al., 2020). We use the images and annotations of VOC (07 + 12) and select 15 classes as base classes, and leave the rest novel classes. In proposed COCO-ARShiff split, we use the images and annotation of COCO 2014, select the 20 VOC classes as training, and select 10 out of the other 60 classes as novel classes. We select these classes to explicitly produce a distribution shift in the box aspect ratios, as shown in Figure 2. In our LVIS-ARShift benchmark, we use LVIS v0.5 and 10 shots following Wang et al. (2020). In each setting, the base classes are the 20 VOC classes in COCO, while the novel classes are 10 rare classes manually picked that have an aspect ratio distribution shift from the base classes. The detailed classes of our proposed splits are in Section H in the appendix.
Apart from our proposed ARShift setting, we also evaluate on two widely-used few-shot detection benchmarks (Kang et al., 2019; Wang et al., 2020) based on PASCAL VOC and COCO. For fair comparison, we use the same train/test splits and novel class instances as Kang al. (2019); Wang et al. (2020) to train and evaluate all models. On COCO, report basc/novel classes AP, AP50, and AP75 under shots 1, 2, 3, 5, 10, and 30. On PASCAL VOC, we report AP50 for three different base/novel class splits under shots 1, 2, 3, 5, and 10. Following Wang et al. (2020) and Qiao et al. (2021), we use Faster R-CNN as our base model and use an ImageNet pre-trained (Russakovsky et al., 2015) Res as the backbone, unless otherwise noted.
Training Procedure. Our training and fine-tuning procedures are consistent with previous work TFA (Wang al., 2020) and DeFRCN (Qiao et al., 2021). On PASCAL VOC, at phase base
7

---

Under review as a conference paper at ICLR 2023
Baseline
Ours
Baseline
Ours
Baseline
Ours
Baseline
Ours
Figure 4: Qualitative results of our CoRPN's + DeFRCN on COCO-ARShit on 5-shots, compared with the DeFRCN baseline. The results illustrate that our approach discovers novel objects missed by the baseline, and also has less misclassification.
classes training, each model is trained on the union set of VOC 07+12 trainval data. Evaluation is on the VOC 07 test set. At the fine-tuning phase, each model is fine-tuned with balanced few-shot dataset sampled from VOC 07+12 that contains both base classes and novel classes. On COCO, following Wang et al. (2020), the fine-tuning phase is two-stage: at stage on novel classes; at stage 2, we then fine-tune the model with a balanced few-shot dataset containing both base and novel classes. Please refer to Section of the appendix for our implementation details and hyperparameters.
fine-tune the model
Baselines and Evaluation Procedure. We mainly focus on comparing against the state-of-the-as baseline DeFRCN (Qiao et al., 1021), and widely-used previous work TFA (Wang et al., 2020). Our approach incorporates CoRPN's into the baseline models, while keeping other model compo- nents and design choices unchanged. In addition, we thoroughly compare with variety of recent few-shot detectors, including FSOD (Fan et al., 2020), MPSR (Wu et al., 2020), FSDetView (Xiao & Marlet, 2020). These baselines address other aspects of few-shot detection which are different from us (Section 2). and their modifications are thus largely orthogonal to our effort here. Note that our evaluation follows the standard procedure in Wang et al. (2020). This standard procedure computes AP separately for novel and base categories for a detector that is engineered to detect both novel and base classes ((CCs) |C..D-way) We focus the novel class performance, and also report the base class performance in Table 4. For work (Fan et al., 2020) with different evaluation procedures, re-evaluate their methods with the standard procedure, so that results in Tables 4 be different from the original reported results.
4.1
MAIN RESULTS
Our evaluation mainly focuses on our proposed ARShift splits, but also includes the conventional splits. Also, we focus on the extremely few-shot regime, which is the most challenging scenario for few-shot detection. Tables 1, 2 and 3 show the detection performance on our proposed ARShift splits where bounding box aspect ratio distribution shift is present between base and novel classes. Table 4 summarize the detection results for base and novel classes in low shots on the conventional COCO benchmark, respectively. For completeness, the results for base and novel classes in higher shots on PASCAL. VOC and COCO are summarized in Section B of the appendix, where our model also performs comparably. We present our qualitative results Figures 4 and 6.
CoRPN's consistently outperform the baselines. Tables 1, and 3 show that CoRPN's greatly outperform the baseline TEA (Wang et al., 2020), DeFRCN (Qiao et al., 2021) and an aspect ratio ensemble modified from the DeFRCN model, especially in very low shots. Especially, we provide the results on three additional base and novel class splits on the LVIS dataset (Gupta et al., 2019) in Table 3. LVIS is much more challenging long- tail recognition dataset, containing large amount of rare classes, hence the fairness for aspect ratio is more crucial. We also provide the results on all rare classes in the LVIS test set reference. Our CoRPN's outperform the mate-of-the-ar
8

---

Under review as a conference paper at ICLR 2023
Method
AP
AP50
AP75
DeFRCN Qiao et al. (2021)
13.6
31.0
9.6
CoRPN's
18.8
37.0
17.2
No Cooperation Loss
14.7
30.9
11.7
No Diversity Loss
13.8
30.2
10.0
= 0.1
14.2
31.0
11.1
Threshold @ = 0.4
16.8
35.2
13.5
@ = 0.7
14.7
31.5
11.6
0.9
14.8
33.6
10.1
Table 6: Our diversity loss and cooperation loss are both required for CoRPN's to obtain the largest improvement. sub-optimal threshold in the cooperation loss also has an adverse effect on perfor- mance. The table shows I-shot novel class performance of all models under our proposed VOC- ARShift novel split.
DeFRCN baseline by large margins all the splits with aspect ratio distribution shift, especially in AP75, suggesting that CoRPN's improve the quality of detection in such scenarios. CoRPN's also marginally outperform the DeFRCN baseline on the entire set of rare classes in mAP and AP75 on LVIS. On the conventional splits, as shown in Tables 4, CoRPN's consistently improve over TFA for all shots, and also marginally outperform DeFRCN on the challenging COCO dataset. The combination of both results shows significant improvement in aspect ratio distribution faimess in our model.
CoRPN's beat other state of the art. With our simple modification RPN, we also outperform other sophisticated approaches in the very low-shot regime on the more challenging COCO dataset. In particular, significantly outperform baselines that introduce attention mechanisms for adjust- ing proposal generation (Hsich al., 2019; Fan et al., 2020) under the standard evaluation procedure. We believe CoRPN's could be combined with other approaches with improvements from different perspectives, such as exploiting better multi-scale representation (Wu et al., 2020), incorporating metric learning (Yang al., 2020), adding feature aggregation module (Xiao & Marlet, 2020) for further improvements.
CoRPN's don't forget base classes. While improving detection on novel classes through fine- tuning, we maintain strong performance on base classes - there is no catastrophic forgetting (Ta- ble 4). By contrast, the base class performance of some state-of-the-art baselines dramatically drops, demonstrating that they cannot fairly detect novel and base classes.
4.2 ABLATION STUDY
We investigate how the proposals of CoRPN's change and conduct a series of ablations that evaluate the contribution of each loss component and different design choices. Specifically, find that: (1) CoRPN's 's specialize in different aspect ratios without explicit supervision; (2) Our cooperation loss and diversity loss both necessary for CoRPN's improve faimess; (3) (in Section F of the appendix) CoRPN's outperform other baselines such as with larger RPN sub-networks, an existing cosine loss based diversity, and bootstrapping.
Specialization for aspect ratios emerges in CoRPN's training. Table shows that the boxes of three different aspect ratios are handled by different RPN classifiers in CoRPN's. Instead of explicitly training different RPN classifiers to handle different aspect ratios, CoRPN's learn a more flexible specialization, improving the performance.
CoRPN's need both diversity and cooperation losses. Table 6 shows that after removing either loss, CoRPN's does improve the performance over the baseline. Also, when the threshold hyper- parameter is suboptimal, the performance surpasses the baseline but still substantially underperforms the CoRPN's with the optimal hyperparameter.
5 CONCLUSION
We identify the bias of few-shot detectors towards familiar aspect ratio distribution. As illustrated in our ARShift benchmark, substantial improvement on under-represented aspect ratio distribution can be obtained by our proposed CoRPN's which produce more informative proposals. Our method achieves
state of the art on both our proposed settings with hard base and novel splits and widely-used benchmarks in the very few-shot regime. This is achieved by training CoRPN's ss with diversity and cooperation losses.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Alexey Bochkovskiy, Chien-Ya Yao Wang, and Hong-Yuae Yuan Mark Liao. YOLOv4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004. 10934, 2020. 2
Stephen Boyd and Lieven Vandenberghe. Convex optimization interior-point methods. 2008.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and
Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020. 2
Hao Chen, Yali Wang, Guoyou Wang, and Yu Qiao. LSTD: A low-shot transfer detector for object detection. In AAAI, 2018.
Inderjit S. Dhillon. The log- dog-deteemmnant divergence and its applications. 2008. 6
Nikita Dvomikk Cordelia Schmid, and Julien Mairal. Diversity with cooperation: Ensemble methods for few-shot classification. In ICCV, 2019. 3, 17, 18
Ian Endres and Derek Hoiem. Category independent object proposals. In ECCV, 2010.
Mark Everingham, Luc Van Gool, Christopher K. Williams, John Winn, and Andrew Zisserman. The PASCAL visual object classes (VOC) challenge. IJC,, 88(2):303-338 2010. 2,
Qi Fan, Wei Zhuo, Chi-Keung Tang, and Yu- Wing Tai. Few-shot object detection with attention-rpn and multi-relation detector. CVPR, 2020. https:/ com/ hing5//Few/ 3, 7,
8, 9, 14
Zhibo Fan, Yuchen Ma, Zeming Li, and Jian Sun. Generalized few-shot object detection without forgetting. CVPR, 2021. 3
Li Fei-Fei, Rob Fergus, and Pictro Perona. One-shot learning of object categories. IEEE TPAMI, 28 (4):594-611 2006.
Ross Girshick. Fast R-CNN. In ICCV, 2015. 2
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accu- rate object detection and semantic segmentation. In CVPR, 2014. 2
Agrim Gupta, Piotr Dollár, and Ross Girshick. LVIS: A dataset for large vocabulary instance seg- mentation. In CVPR, 2019. 1,7,8
Guangxing Han, Yicheng He, Shiyuan Huang, Jiawei Ma, and Shih-Fu Chang. Query adaptive few-shot object detection with heterogeneous graph convolutional networks. In ICCV, 2021. 3
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In ICCV, 2017. 2
Ting-I I Hsieh, Yi-Chen Lo, Hwann-Tzong Chen, and Tyng-Luh Liu. One-shot object detection with co-attention and co-excitation. In NeurIPS, 2019.
One-Shot-Ob jttpt-obttett--teetiiiiie
3, 9, 14
Hanzhe Hu, Shuai Bai, Aoxue Li, Jinshi Cui, and Liwei Wang. Dense relation distillation with context-aware aggregation for few-shot object detection. In CVPR, 2021. 3
Bingyi Kang, Zhuang Liu, Xin Wang, Fisher Yu, Jiashi Feng, and Trevor Darrell. Few-shot object detection via feature reweighting. In ICCV, 2019. 2, 3,7
Hei Law and Jia Deng. Cornernet Detecting objects as paired keypoints. In ECCV, 2018.
Aoxue Li and Zhenguo Li. Transformation invariant few-shot object detection. CVPR, 2021.
Bohao Li, Boyu Yang, Chang Liu, Feng Liu, Rongrong Ji, and Qixiang Ye. Beyond max max-margin: Class margin equilibrium for few-shot object detection. In CVPR, 202 3, 14. 17
Yiting Li. Haiyue Zhu, Yu Cheng, Wenxin Wang, Chek Sing Teo, Cheng Xiang, Prahlad Vadakkepat, and Tong Heng Lee. Few-shot object detection via classification refinement and distractor retreat- ment. In CVPR, 2021b.
10

---

Under review as a conference paper at ICLR 2023
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In E 2014. 1,2,7
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yare Fu, and Alexander C. Berg. SSD: Single shot multibox detector. In ECCV, 2016.
Anton Osokin, Denis Sumin, and Vasily Lomakin. OS2D: One-stage one-shot object detection by matching anchor features. ECCV, 2020.
Juan-Manuel Pérez-Rúa, Xiatian Zhu, Timothy Hospedales, and Tao Xiang. Incremental few-shot object detection. In CVPR, 2020. 14
Limeng Qiao, Yuxuan Zhao, Zhiyuan Li, Xi Qiu, Jianan Wu, and Chi Zhang. Defren: Decoupled faster r-cnn for few-shot object detection. In ICCV, 2021. 2, 3, 4, 5, 7, 8, 9, 13, 14, 15, 16,
Joseph Redmon and Ali Farhadi. YOLO9000: better, faster, stronger. In CVPR, 2017. 2
Joseph Redmon and Ali Farhadi. YOLOv3: An incremental improvement. arXiv preprint 1:X1::880..02760 2018. 2
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In CVPR, 2016. 2
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NeurlPS, 2015. 2, 4
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJV,, (1533:211-2552 2015.
Eli Schwartz, Leonid Karlinsky, Joseph Shtok, Sivan Harary, Mattias Marder, Sharathchandra Pankanti, Rogerio Feris, Abhishek Kumar, Raja Giries, and Alex M. Bronstein. RepMet: Representative-bats based metric learning for classification and one-shot object detection. In CVPR, 2019. 3
Bo Sun, Banghuai Li, Shengeai Cai, Ye Yuan, and Chi Zhang, Fsce: Few-shot object detection via contrastive proposal encoding. In CVPR, 2021. 3, 13, 14, 16, 17
Sebastian Thrun. Lifelong learning algorithms. Learning to learn, 8:181-209, 1998. 3
Koen van de Sande, Jasper Uijlings, Theo Gevers, and Arnold Smeulders. Segmentation as selective search for object recognition. In ICCV, 2011.
Xin Wang, Thomas E. Huang, Trevor Darrell, Joseph E Gonzalez, and Fisher Yu. Frustratingly simple few-shot object detection. In ICML, 2020. 1, 2, 3, 4, 7, 8, 13, 14, 15, 16, 17, 18
Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Meta-learning to detect rare objects. In ICCV, 2019. 3
Aming Wu, Yahong Han, Linchao Zhu, and Yi Yang. Universal-prototype enhancing for few-shot object detection. In ICCV, 2021. 3
Jiaxi Wu, Songtao Liu, Di Huang, and Yunhong Wang. Multi-scale positive sample refinement for few-shot object detection. In ECCV, 2020. https://qithuu: com/ jiaxi-iiiiiiiiii wuu-d SR. 3, 7, 9, 14, 17
Yang Xiao and Renaud Marlet. Few-shot object detection and viewpoint estimation for objects in the wild. In ECCV, 2020. 3, 7, 9, 17
Xiaopeng Yan, Ziliang Chen, Anni Xu, Xiaoxi Wang, Xiaodan Liang, and Liang Lin. Meta R-CNN: Towards general solver for instance-level low-shot learning. In ICCV, 2019. 1,3
Yukuan Yang, Fangyun Wei, Miaojing Shi, and Guoqi Li. Restoring negative information in few- shot object detection. In NewrIPS, 2020. 3, 14, 17

---

Under review as a conference paper at ICLR 2023
Lu Zhang, Shuigeng Zhou, Jihong Guan, and Ji Zhang. Accurate few-shot object detection with support-query mutual guidance and hybrid loss. CVPR, 2021.
Weilin Zhang and Yu-Xiong Wang. Hallucination improves few-shot object detection. In CVPR, 2021. 3
Xingyi Zhou, Jiacheng Zhuo, and Philipp Krahenbuhl. Bottom-up object detection by grouping extreme and center points. In CVPR, 2019. 2
Chenchen Zhu, Fangyi Chen, Uzair Ahmed, and Marios Savvides. Semantic relation reasoning for shot-stable few-shot object detection. In CVPR, 2021. 3, 14, 17
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
This appendix provides additional experimental results and details that support the approach in the main paper and are not included there due to limited space. The seven sections include (1) additional results for higher-shols on the COCO-ARSHin comparisons with additional baselines for ARShift, and results on base classes before and after finetuning; (2) explainations for additional evaluation and experimental details; (3) discussion the training and inference time and memory; (4) results on conventional splits; (5) analysis with additional ablation study; (6) addtional qualitative results; and (7) details of the proposed base and novel class splits ARShift for PASCAL vOC, COCO, and LVIS.
B ADDITIONAL RESULTS
Higher-Shot Results on COCO-ARShift
In Table 2 of the main paper, we show that CoRPN's outperform the DeFRCN (Qiao et al., 2021) baseline consistently on I, 2, and shots. Table 10 shows that this performance improvement persists in 5, 10, and 30 shots. In this scenario where more support instances available, CoRPN's consistently outperform the DeFRCN (Qiao et al., 2021) baseline.
Additional Baseline on ARShift
In Table of the main paper, we mainly compare our method against the state-of-the-art method VOC-ARShift. In Table 11 also evaluate additional baseline FSCE (Sun et al., 2021). Note that other recently published methods represented by FSCE underperform our DeFRCN baseline by significant margins our ARShift splits well, do not include these other methods in our main paper.
DeFRCN (Qiao et al., 2021)
Results on Base classes
Here we provide the detection results on base classes after base classes training (stage 1) and after fine-tuning (stage 2). As shown in Table 9, the performance of our CoRPN's DeFRCN on base classes is comparable with DeFRCN (Qiao et al., 2021), while we achieve large improvements on novel test classes shown in Table of the main paper. After fine-tuning, CoRPN's also do not forget base classes. Table 9 shows that the performance of our CoRPN's + DeFRCN after fine-tuning still comparable with the baseline.
Average Recall
We present the average recall (AR) result the three splits in LVIS-ARShift benchmark in Table 7. Here we show AR1000 by the convention of COCO. Our CoRPN's also improve few- shot detection in the AR, meaning that our CoRPN's miss fewer novel objects under aspect ratio distribution shift.
Per-Category Result
In Table 8 we present the 1-shot AP50 for each novel class on our VOC-ARShift benchmark. Our CoRPN's improve upon the vanilla DeFRCN bascline by large amounts in 4 of the categories, while the naive ensemble of experts only achieves marginal improvements.
Analysis of Proposal Aspect Ratio
Figure shows the aspect ratio distribution of proposals from the baseline RPN and our CoRPN's. Our CoRPN's produce more diverse proposals that are robust the aspect ratio distribution shift.
C IMPLEMENTATION AND EVALUATION DETAILS
Implementation Details and Hyperparameters. For ease of comparison, we use the same values for all shared training and fine-tuning hyperparameters (batch size, learning rate, momentum, weight decay, etc.) as the baselines Wang et al. (2020) and Qiao et al. (2021). CoRPN's have the following additional hyperparameters: the number RPN's, the cooperation loss threshold ø, the diversity loss trade-off >a, and the cooperation loss trade-off Aa- For COCO and LVIS, directly used
13

---

Under review as a conference paper at ICLR 2023
Method
Split
Split
Split
Ours
DeFRCN + CoRPN's
18.4
24.8
28.3
DeFRCN (Qiao et al., 2021)
17.6
18.5
13.9
Baselines
DeFRCN Ensemble of Experts
18.1
23.4
25.0
Table 7: Average recall (ARI000) on our LVIS-ARShift benchmark. We achieve performance im-
provements over both baselines in recall.
Method
Train
Bottle
Aeroplane
Horse
Bus
Ours
DeFRCN CoRPN's
46.2
4.5
43.4
10.5
80.4
DeFRCN (Qiao et al., 2021)
27.5
1.8
23.2
50.6
51.7
Basclines
DeFRCN Ensemble of Experts
27.6
3.1
24.3
54.7
38.6
Table 8: Per-Cateree Per-Category 1-shot AP50 result on our VOC-ARShift benchmark. Our method achieves large improvement in 4 of the 5 classes.
hyperparameter sets that worked well on PASCAL VOC. In Table 4, we report CoRPN's detection results. We find that hyperparameters selected from PASCAL VOC are generalizable the more challenging COCO and LVIS benchmark. Results reported in Table 4 are obtained with RPNs, ø 0.3, Aa- Xa = 0.025 for TFA(Wang al., 2020), and X, 2. A 0.05 for DeFRCN (Qiao et al., 2021).
Selection Procedure Cumulative Variance. We summarized the hyperparameter selection cri- teria for PASCAL VOC in the experiment section in the main paper. The second criterion is the cumulative variance the RPN's response matrix foreground boxes. Specifically, given set of M anchor boxes, the N RPN's produce an N by M matrix of probabilities F f ..... We run a principal component analysis (PCA) on with N components and compare the cumulative percentage of variance explained by each component. We would like the variance to be distributed across components.
ff....,,NNN fN.T.
Evaluation. As mentioned in the main paper, for a fair comparison, we use the standard evaluation procedure for all the compared models. We also compare against other approaches with the same novel class instances and test images. Authors of methods compared in this submission have helped us ensure the performance we report is a proper reflection of their methods, and we will acknowledge properly in any final version. In the standard evaluation procedure, when test image comes in, the model has no assumption on what category the image contains (Wang et al., 2020). The detector's classifier is [[C%|+[Ca ()-wy,y detecting objects from a joint space of both base and novel categories. In the main paper, we marked results with * if they were re-evaluated under the standard procedure, and with ** if the original work used the standard procedure, but the results were not reported and so they were evaluated by us.
Specifically, below include the details on how we obtained the results (with special marks) in the main paper; other results (without special marks) (Wang et al., 2020; Pérez- Rúa et al., 2020; Yang al., 2020; Sun al., 2021: Li al., 2021a; Zhu al., 2021) including the concurrent work (Sun et al., 2021; Li et al., 2021a; Zhu et al., 2021) are from the original papers.
We fine-tuned and evaluated MPSR using the publicly released code and the pre-trained detection model for base classes (Wu et al., 2020).
We re-evaluated FSOD (Fan et al., 2020) and CoAE (Hsieh et al., 2019) under the standard procedure, using the publicly released code and the pre-trained model (Fan et al., 2020; Hsich et al., 2019). FSOD uses a class-agnostic 2-way classifier that determines if an object is foreground or background. At inference time, FSOD produces a balanced number proposals per novel category and collects these proposals for NMS. We adapt FSOD under the standard procedure such that there are balanced number of proposals for all base and novel categories. For a fair comparison, we also fine-tuned FSOD with the same novel category instance(s) in TFA (Wang et al., 2020).
We re-evaluated CoAE under the standard evaluation, using the publicly released code and a pre-trained model (Hsich et al., 2019). For each test image containing a certain category, CoAE samples support image(s) from this category and collects boxes based
14

---

Under review as a conference paper at ICLR 2023
RPNs CoRPNs
Density
0.4
0.3
0.2
0.0
0.14
0.37
1
2.72
7.39
Aspect Ratio
Figure 5: The aspect ratio distribution of proposals from the baseline RPN and our CoRPN's. Our CoRPN's produce more diverse proposals robust to the aspect ratio distribution shift.
After Phase I w/o fine-tuning
After Phase 2
Method DeFRCN (Qiao al., 2021)
1-shot 76.5
2-shot
3-shot
5-shot
10-shot
81.4
77.0
76.9
77.0
76.7
DeFRCN CoRPN's (Ours)
82.8
75.6
75.8
75.7
75.5
75.4
Table 9: Base classes AP50 on VOC-ARShift after base class training (phase 1) and after fine-tuning
(phase 2). The same hyperparameter setting applies to both models. Notice that ours is higher after phase and slightly lower but comparable with the results DeFRCN after phase 2, but our method achieves significantly better performance test classes as shown in Table of the main paper (e.g., 7.3AP on I-shot and 6.6 AP on 2-shot). Neither our method nor the baseline forgets base classes significantly.
the support image(s). At inference time, instead of providing each test image with support feature(s) from this category, provide each test image support feature(s) from all base and novel categories. We then collect boxes from all categories and evaluate them. For a fair comparison, also fine-tuned CoAE with the same novel category instance(s) in TFA (Wang et al., 2020).
D TRAINING & INFERENCE TIME AND MEMORY
The only architectural difference between CoRPN's and the baseline RPNs of TFA (Wang et al., 2020) and DeFRCN(Qiao et al., 2021) is that CoRPN's have multiple RPN classifiers. Note that we only duplicate the RPN classifiers, but not the entire RPN. In the basclines, the RPN's classifier is 1xl convolutional layer, with input channels channels as the number of cell anchors (3). Compared with the baselines, CoRPN's with 5 RPN's have four additional RPN classifiers and thus consist of 256x 12 additional parameters. In our experiments, we find that CoRPN's with 5 RPN's increase the training time by only 3%, with roughly the inference time and same memory footprint, compared with the baselines.
the number of feature channels (256) and output
E RESULTS ON CONVENTIONAL SPLITS
In Table 4 of the main paper, we show that CoRPN's perform comparably even better conven- tional splits on 1, 2, and shots on COCO. In this section we present more conventional split results. Table 12 shows PASCAL VOC conventional split results where our CoRPN's perform comparably with the baselines. Table 13 shows the results on COCO 5, 10, and 30 shots, where our method also obtains similar performance the baseline.
15

---

Under review as a conference paper at ICLR 2023
5-shot
10-shot
30-shot
Method
AP50
AP75
AP
APSU
AP75
AP50
AP75
DcFRCN (Qiao et al., 2021)
16.9
27.4
18.0
18.9
30.8
20.2
21.2
34.5
22.2
DeFRCN CoRPN's (Ours)
17.2
27.0
18.3
19.3
31.0
21.0
21.4
34,9
22.0
Table When there are many shots (our proposed COCO-ARShift in 5, and 30 shots), while the impact of using CoRPN's is reduced, CoRPN's still outperform the DeFRCN (Qiao et al., 2021) baseline in most cases. The reduced improvement is likely because missing objects in the RPN stage becomes less important effect. Results in red are the best.
Method
shot=1
5
10
TFA w/ cos CoRPN'S
28.1
31.7
36.0
38.6
44.7
DeFRCN CoRPN's
37.0
49.3
55.1
56.7
59.0
TFA w/ cos (Wang et al., 2020)
19.1
27.2
28.0
34.4
42.1
FSCE (Sun et al., 2021)
23.7
33,5
35.8
41.5
48.1
DeFRCN (Qiao al., 2021)
31.0
44.6
47.5
55.2
57.2
DeFRCN Ensemble
29.7
42.7
50.9
55.3
56.0
Table 11: In Table 1 of the main paper, we mainly compare our method against the state-of-the- VOC-ARShift. Here we evaluate the result of
method DeFRCN (Qiao al., 2021) additional baseline FSCE (Sun et al., 2021) (marked in green). Note that other recently published methods represented by FSCE underperform our DeFRCN baseline by significant margins on our ARShift splits as well, so we do not include these methods in our main paper.
Baseline
Ours
Baseline
Ours
Baseline
Ours
Baseline
Ours
Baseline
Ours
Baseline
Qurs
Figure 6: Additional qualitative results of our CoRPN's + DeFRCN on COCO-ARShift on 5-shots,
compared with the DeFRCN baseline.
F ADDITIONAL ANALYSIS
We present some additional ablation studies for CoRPN's. Note that these ablations per- formed on the conventional split. Even on conventional splits, our CoRPN's outperform these naive strategies. Our CoRPN's will likely achieve more performance improvement our ARShift splits.
16

---

Under review as a conference paper at ICLR 2023
Novel Set
Novel Set 2
Novel Set 3
Method
shat-1
shot11
shee=1
Ours
TEA wifc CoRPN's
40.8
64.8
45.7
55.1
54.8
20.4
29.2
36.3
36.5
41.5
29.4
40.4
44.7
$1.7
49,9
TFA C-RPN'S
44.4
46.4
54.1
25.7
29.5
41.8
51.6
DeFRCN CoRPN's
44.1
55.8
62.2
06.0
65.5
31.8
41.2
46.1
49.9
53.3
38.9
51.3
54.6
59.4
61.5
Msin bonelines
TEA w/lc (Wang
2020)
36.8
29.1
43.5
55.7
18.2
29.0
33.4
35.5
39,0
27.7
53.6
48.7
59.2
TEA wi cos (Wang al., 2020)
39.8
36.1
44.7
55.7
56.0
23.5
26.9
34.1
35.1
39.1
30.8
34.8
42.8
49.5
49.8
DeFRCN (Qinn et al, 2021)
43.8
57.5
61.4
65.3
67.0
31.5
40.9
45.6
50,1
$2.9
38.2
90.9
54.1
59.2
61.9
FRCN4R-full (Wang 65 al. 20001)
15.2
20.3
29.3
40.1
13.4
20.6
32.4
35.8
19.6
20.8
42.2
42.1
MPSR (Wu et al., 2020)
41.7
42.5
51.4
55.2
61.8
24.4
29.3
39.2
39.9
47.8
35.6
41.8
42.3
48.0
49.7
Other huscines
Repblet (Ying al..22200
37.8
41.6
430
47.4
CME (Li al., 2021a)
41.5
47.5
50.4
58.2
60.9
27.2
30.2
41.4
42.5
46.8
34,3
39.6
45.1
48.3
51.5
SRR ISD (2hu 2022)
47.8
50.5
55.
32.5
35.3
40.8
43.8
41.5
44.3
46.9
46.4
Table 12: CoRPN's not only improve performance on our ARShift split, but also perform compa- rably with state of the art on more traditional splits: Few-shot detection (AP50) on PASCAL VOC novel classes under three base/novel splits in the generalized few-shot learning setting. CoRPN's outperform the main baselines TFA and DeFRCN mostly in the very low shots, with comparable per- formance in the higher shots, regardless of classifier choice. Note that these other baselines address different aspects of few-shot detection, and could be combined with them for further improvements. All models are based on Faster R-CNN with a ResNet-101 backbone, and follow the evaluation procedure in Wang et al. (2020). Results in bold are the better result between ours and the main baselines.
5-chot
10-shor
30-shot
Mathod
Backbone
AP
AP50
AP75
AP
AP50
AP75
AP
AP50
AP75
TFA w/ CoRPN's
ResNat-101 101
16.9
8.6
10.5
20.2
9.8
13.5
25.0
12.9
Ours
TEA w/ cos CoRPN's
ResNet-101
8.8
16.4
8.7
10.6
19.9
10.1
13.9
25.1
13.9
DeFRCN CoRPN's
BesNet-
25.2
29.7
20.8
TFA w/ To (Wing et al. 2020)
BesNer-101
8.4
16.0
8.4
10.0
19.2
9.2
13.4
24.7
13.2
Main baselines
w/ (Wang
2020)
R0t
15.3
80
19.1
13.7
13.4
DeFRCN (Qian et al. 2021)
ResNer-101
13.5
24.7
13.0
16.7
29.6
16.7
21.0
36.7
21.4
DeDei (Xiso & Marlet, 2020)
ResNet- 101
20.1
4.4
10.7
25.6
6.5
15.9
31.7
15.1
Other baselines
FSCE (Sun et al., 2021)
ResNet-101
11.9
10.5
16.4
16.2
CME (Sun al., 202.)
ResNet- 101
15.1
24.6
16.4
16.9
28.0
17.8
SRR-PSD
al., 2021)
ResNet-101
23.0
14.7
Table 13: When there are many shots, the impact of using CoRPN's on the conventional split is reduced, likely because missing objects in the RPN stage becomes a less important effect. CoRPN's perform acceptably for few-shot detection performance on the difficult COCO novel classes task for 5-shot, 10-shot, and 30-shot. *Model re-evaluated using the standard procedure (with base and novel classes joint space) for a fair comparison. denotes that numbers are not reported in the corresponding paper. CoRPN's consistently outperform the main baseline DeFRCN except for the 30-shot case.
Bootstrapping. Table in the main paper shows that our CoRPN's outperform naive ensembles. Here we further compare CoRPN's with a bootstrapping baseline in Table 14. We construct the boot- strapping baseline by using multiple RPN classifiers, the number of which is the same as CoRPN's. Instead of selecting the most confident RPN get gradients,
randomly select an RPN to get gradients during training. There are no additional loss terms in training these RPN classifiers. In the fine-tuning stage and the inference time, we use the same selection procedure as CoRPN's (i... using the most certain RPN).
Diversity loss in Dvornik al. (2019). In Table 14, we also compare our CoRPN's with (Dvornik et al., 2019), which utilizes a pairwise cosine similarity based diversity loss. Our CoRPN's also outperform this baseline with a considerable margin.
Larger RPN. We compare CoRPN's with a baseline with larger RPN sub-networks in Table 15. We find that using larger RPN sub-networks does not improve performance, suggesting that the advantage of CoRPN's 's is not simply the result of using more parameters. In CoRPN's, all RPN classifiers share the same RPN feature extractor, as shown in Figure (main paper). We enlarge the feature dimension in the RPN. We implemented two options: a large RPN where the RPN feature extractor's 's output channels increase from 256 to 272, and a larger RPN where the output channels double to 512. We also modified the RPN classifier and bounding box regressor to take in larger features for both options. Table 15 shows that simply enlarging the model capacity and the anchor density of the original RPN cannot improve the few-shot detection performance.
17

---

Under review as a conference paper at ICLR 2023
Method
AP50
2 RPN's, Bootstrapping
31.8
2 RPN's, Dvornik et al. (2019)
32.4
2 RPN's, CoRPN's (Ours)
35.8
Table 14: Our diversity term - the log-determinant loss - offers improvements over a bootstrapping baseline and pairwise cosine similarity based diversity loss in Dvornik al. (2019). The table shows novel class AP50 under PASCAL VOC novel split 3, shot 1.
Method
# param added
AP50
TFA, original
0
30,8
TFA, Large RPN
28x
25,9
TFA, Larger RPN
268x25
27.8
TFA + CoRPN's, s, RPN's
3x256
35.8
TFA + CoRPN's, 5 RPN's 27x
12x256
34.8
TFA + CoRPN's, 10 RPN's
35.7
Table 15: Performance improvements CoRPN's s are not due to the increased number of parameters in CoRPN's. This table shows the novel class AP50 of TFA (Wang et al., 2020) with large RPN and even larger RPN, and compares with CoRPN's with different numbers of RPN's. All results are with PASCAL VOC novel split shot 1. The second column presents how many additional parameters are introduced to the original TFA model. Using larger RPN sub-networks does not improve the performance, while CoRPN's significantly improve with fewer added parameters.
G ADDITIONAL QUALITATIVE RESULTS
In Figure 6 we show some additional qualitative result of our CoRPN's model on COCO. Notably, model could successfully discover novel objects omitted by the baseline, and also successfully classify some novel objects that cause confusion for the baseline method.
H DATA SPLITS
PASCAL VOC Split. For our proposed PASCAL VOC-ARShift split, the base classes include "bicycle", "boat", "car", "cal", "chair", "diningtable", "dog", "person", "sheep", "tymoni- tor", "bird", "potledplani", "cow", "motorbike", and "sofa"; the novel classes include "train" " "bottle", "aeroplane" "horse", and "bus".
COCO Split. For our proposed COCO-ARShift split, the base classes include the 20 VOC classes in COCO; the novel classes include "hot dog", "tennis racket", "fire hydrant", "laptop", "suitcase", "frisbee", " "teddy bear", "bowl", "kite", and "elephant".
LVIS Splits. For our proposed LVIS- splits, the base classes include the 20 VOC classes in LVIS: the novel classes of the first split include "Loafer (type of shoe)". "batter (food)", "cabin car". "cylinder". "egg roll", "liquor". "nailfile". "plow (farm equipment)", "vinegar", and "yoke (animal equipment)"; the novel classes of the second split include "broach". "burrito", "cargo ship". "crayon", "incense", "peeler (tool for fruit and vegetables)", "pin (non jewelry)", "roller skate", "tinsel", and "vodka"; the novel classes of the third split include "ax", "beaker", "ferry", "fish (food)", "funnel", "incense", "needle", "space shuttle", ", "stepladder", and "vulture".
18


Reviews:
[{"review_id": "UNMpRNWwlIa", "reviewer": "Reviewer_NdSa", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the problem of the aspect-ratio shift between the base and novel classes in few-shot object detection resulting in the poor box proposals of the RPN in a two-stage detector. They propose using multiple classifiers instead of one as in the current RPN. Their experiments show significant improvements with the new versions of the COCO and LVIS datasets but marginal improvements with the original datasets. The new splits of the COCO and LVIS datasets are created by re-select the subset of base and novel classes whose aspect ratios are significantly different. ", "summary_of_the_review": "The paper proposed a method for alleviating the problems of aspect ratio shift in the few-shot object detection by introducing multiple classifiers of the RPN instead of one. It is arguable to see the practicality of the new splits of the COCO and LVIS datasets since the proposed methods have marginal improvements in the common setting. Also, the paper also has the incorrect average results reported thus drawing an incorrect observation of the ensemble RPN.", "strength_and_weaknesses": "*Strengths*\n+ The finding is quite interesting, i.e., shifting in aspect ratio distribution of object bounding boxes between the base and novel classes.\n+ The structure of this paper is well-written.\n\n*Weaknesses*   \n+ Incorrect average results in Tab. 3. For DeFRCN, the AP for split 1, 2, and 3 are 9.6, 8.0, and 9.0, respectively. But they report the overall AP is 15.0. The same observation applies to the DeFRCN Ensemble of Experts. Thus, the conclusions drawn are wrong for this table. DeFRCN Ensemble of Experts does help a lot.\n+ This method aims to address the poor proposal generation problem of RPN. It would be much better to include the AR results to prove their recall is better than that of the baseline. \n+ The authors should explain the intuition behind the formulation of diversity loss. It is not trivial to interpret the proposed loss to enhance the diversity of the classifiers.\n+ Do the authors claim the new split of COCO and LVIS datasets as a contribution? The improvement in the original splits seems very marginal. If the base classes contain high-variation in the bounding box aspect ratios, e.g. human, they can arguably cover the aspect ratio of many novel classes. \n+ Since the paper mainly focuses on the improvement of RPN, they should move the qualitative results to the main paper, not the supplementary material.  \n+ Fig. 1 does not reflect the motivation of this paper. The proposals on the right are still good which can be further regressed in the second stage of Faster-RCNN.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "L8wLTsMze_D", "reviewer": "Reviewer_haU4", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper claims that in few-shot detection tasks, performance can be degraded due to the lack of RPN ability to find object proposals for new categories with aspect ratios significantly different from those of the base categories. To overcome the limitations of RPN in few-shot detection, this paper uses multiple RPNs that have been trained in the direction of enhancing diversity among RPNs and cooperating with each other. To adequately validate this benefit of the proposed method, base/novel categories split of datasets are modified so that the (few-shot) images of the novel categories have very different aspect ratios from images of the base categories, which are referred ARShift. Object detectors trained using the proposed method successfully increases the accuracy for the novel categories, avoiding the performance degradation caused by the catastrophic forgetting issue for the base categories in the few-shot detection task on the three datasets, PASCAL VOC, COCO, and LVIS datasets.", "summary_of_the_review": "My rating was based on the points presented in Weaknesses. My biggest concern is that there is no consistency between the problem to be claimed and the method design to solve the problem.", "strength_and_weaknesses": "* Strengths\n1. For the test environment using the ARShift benchmark, it was effective to significantly increase the few-shot detection accuracy using the proposed method on the three detection datasets. It also prevented performance degradation for the base categories, which is often caused by the catastrophic forgetting problem that is common in a few-shot detections.\n\n\n* Weaknesses\n1. The problem in a specific environment of the few-shot detection problem, which has a large difference in the aspect ratio distribution of the basic category and the novel category, as claimed in this paper, is of interest to only a few researchers, and the applicability/generality of the problem is very Limited. Thus, the environment claimed in this paper did not appear in the general dataset, but only in the dataset that was modified to emphasize the environment.\n\n2. In addition, the proposed method using multiple RPNs is not designed to address a claimed problem. Experimentally, several RPNs were specialized in generating region proposals of specific aspect ratios (Tab 5), but this function was not considered at all in the CoRPN design. \n\n3. Paper presentation is immature yet.\n- Figure 1 is not mentioned anywhere in the manuscript's content.\n- The same explanation about using Faster R-CNN are given in the last paragraph of Section 3.1 and the second paragraph of Section 3.3.\n- In Figure 2, it's hard to figure out what 'density' is as the sum of the densities over the entire aspect ratio range seems to be greater than 1.\n- In the top row of Figure 3, there are no explanations about the difference betwee the gray and blue box. In addition, non-maximum suppression is applied after performing bouding box regression so the illustrations in the bottom row of Figure 3 may bring incorrect information. Most of all, this figure does not deliver important information and nor is hard to understand as it has to occupy the large space of the manuscript.\n- In eq 3, Sigma may be Sigma_jk\n- Important details of experimental setup is missing such as the number of RPNs. According to Tab. 5, the number of RPNs may be larger than five.\n\n4. Some questionable impact of the proposed method based on the experimental results\n- In Tab. 5, only two or three RPNs appear to be activated. Does it mean the proposed method uses multiple RPNs inefficiently.\n- What is the version of CoRPN's with the best accuracy? Is it the model trained with the Cooperation loss with phi=0.5 and the diversity loss? If so, the CoRPN seems to be very sensitive to the phi.\n\n5. Similar problems and similar method designs have already been claimed in [a]. [a] is applied to general object detection and more experts than RPN are used for the classifier, but since it seems very simple to apply it to RPN, comparison with [a] will be necessary.\n\n[a] H. Lee, Multi-expert R-CNN for Object Detection, In IEEE TIP.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "E-QzlJ9_Gew", "reviewer": "Reviewer_ZdBo", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This submission is about few-shot object detection with a distribution shift of the bounding box aspect ratios between base and novel categories. The paper first elaborates that there can be splits of base and novel categories that do show a clear distribution shift of the bounding box aspect ratios and that existing few-shot object detectors perform poorly in this scenario. Then, a novel region proposal mechanism for a two-stage object detector is proposed that consists of an ensemble of foreground/background classifiers and a new training strategy. Specifically, the ensemble is trained without explicitly assigning certain aspect ratios to certain RPNs, but the RPNs \"select\" the boxes based on their predictive confidence. Additionally, a diversity loss and a cooperation loss are used for training. Experiments are conducted on specifically selected splits of base and novel categories for the Pascal VOC, COCO and LVIS datasets.", "summary_of_the_review": "I'm leaning toward rejection because of the narrow problem setting, solution and experimental evaluation. Distribution shifts in few-shot learning go beyond aspect ratios, and a more thorough experiments and analyses are needed (impact of number of base categories, evaluation on larger sets of novel categories, per-category results, ...) to convince me of the importance of this problem.", "strength_and_weaknesses": "Strengths:\n- Two different few-shot object detectors are used (DeFRCN and TFA) to showcase the benefits of the novel RPN\n- The overall research direction is important I think, because it studies the practical situation of distribution shifts for novel categories in a few-shot setting.\n- The results on Pascal VOC show a big gain over existing baselines.\n- The analysis of the aspect ratio distributions is interesting in general, even beyond this specific few-shot learning setting.\n- The ensemble baseline (and the experiment with larger RPNs in the appendix) are a fair comparison point.\n- The additional experiments in the appendix are good in general, I think\n\nWeaknesses:\n- The paper's scope is limited to two-stage detectors with RPNs, and RPNs trained on the base categories for the few-shot setting. Other detector designs (like RetinaNet or DETR) are only discussed in related work, but the scope of the actual problem addressed in this paper (distribution shift of aspect ratios) should also include some form of (quantitative) experiments for these detector designs. The statement \"... but likely occurs in one-stage detection too\" at the end of page 2 is insufficient in my opinion.\n- The experimental setting needs more justification and analysis. I understand that distribution shifts can obviously happen between base and novel categories. But I'm wondering if the aspect ratio distribution shift here is only an artifact of the small number of base and novel categories in this specific experimental setting. As far as I understood, the number of base categories is only 15, 20 and 20 for the datasets Pascal VOC, COCO, and LVIS. Why is that, when datasets COCO and LVIS have 80 and over 1200 categories, respectively? It would be good to study the distribution shift under varying number of base categories. What would happen if only five more base categories are added?\n- The topic of the paper is rather narrow for a much broader problem, i.e., distribution shift between base and novel categories. The paper only addresses aspect ratio distributions for two-stage detectors. While authors comment on object location and scale, another distribution shift is due to semantic and appearance differences. Related work on open-vocabulary detection [A,B] can be discussed, where RPNs are evaluated on novel categories in a zero-shot experiment.\n- The improvements for COCO and LVIS (Tables 2 & 3) are not as obvious as for the Pascal VOC dataset (Table 1). Compared to the plain ensemble, the proposed RPNs do not show a clear edge. Why is that? Is it related to the number of base and novel categories in the datasets? For VOC, there are only 5 novel categories. How do the per-category results look like?\n\nReferences:\n- [A] Open-Vocabulary Object Detection via Vision and Language Knowledge Distillation. Gu et al. ICLR'22\n- [B] Exploiting Unlabeled Data with Vision and Language Models for Object Detection. Zhao et al. ECCV'22", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DQ5JJv2uIL", "reviewer": "Reviewer_UxEa", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to solve the aspect ratio gap between base and novel classes in few-shot object detection. The authors present a very simple CoRPN method, which only uses multiple classifiers in RPN to generate more diverse anchors. To evaluate the effects of the aspect ratio gap, the authors propose ARShift benchmarks, which enlarge the aspect ratio differences between the training and testing sets in standard detection benchmarks. Experimental results show that the proposed CoRPN achieves remarkable improvements on ARShift benchmarks.", "summary_of_the_review": "Overall, it's a good work, simple yet effective. My main concern is experiments. The authors only test on self-made benchmarks.", "strength_and_weaknesses": "Strength\n1. The motivation is interesting, which aims to solve the aspect ratio gap problem.\n2. New ARShift benchmarks are useful for future research on aspect ratio.\n3. The proposed CoRPN is simple yet effective.\n\nWeaknesses\n1. Is CoRPN only designed for few-shot object detection? If so, it is better to add 'few-shot' to the title. If not, can you provide more results on other detection settings?\n2. Can your show the aspect ratio distributions of RPN and CoRPN?  It is important to prove that your CoRPN generates more diverse anchors than RPN.\n3. ARShift benchmarks artificially enlarge aspect ratio gaps between the training and test sets, which is somewhat tricky. Can you show the results on standard detection benchmarks?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject