Cooperating RPN's Improve Few-Shot Object Detection

1 INTRODUCTION . Achieving accurate few-shot object detection is difficult , because one must rely on a classifier building a useful model of variation in appearance with very few examples . This paper identifies an important effect that causes existing detectors to have weaker than necessary performance in the few-shot regime . By remediating this difficulty , we obtain substantial improvements in performance with current architectures . The effect is most easily explained by looking at the “ script ” that modern object detectors mostly follow . As one would expect , there are variations in detector structure , but these do not mitigate the effect . A modern object detector will first find promising image locations ; these are usually , but not always , boxes . We describe the effect in terms of boxes reported by a region proposal network ( RPN ) ( Ren et al. , 2015 ) , but expect that it applies to other representations of location , too . The detector then passes the promising locations through a classifier to determine what , if any , object is present . Finally , it performs various cleanup operations ( non-maximum suppression , bounding box regression , etc . ) , aimed at avoiding multiple predictions in the same location and improving localization . The evaluation procedure for reported labeled boxes uses an intersection-over-union ( IOU ) test as part of determining whether a box is relevant . A detector that is trained for few-shot detection is trained on two types of categories . Base categories have many training examples , and are used to train an RPN and the classifier . Novel categories have one ( or three , or five , etc . ) examples per category . The existing RPN is used to find boxes , and the classifier is fine-tuned to handle novel categories . Now assume that the detector must learn to detect a category from a single example . The RPN is already trained on other examples . It produces a collection of relevant boxes , which are used to train the classifier . The only way that the classifier can build a model of the categories variation in appearance is by having multiple high IOU boxes reported by the RPN . In turn , this means that an RPN that behaves well on base categories may create serious problems for novel categories . Imagine that the RPN reports only a few of the available high IOU boxes in training data . For base categories , this is not a problem ; many high IOU boxes will pass to the classifier because there is a lot of training data , and so the classifier will be able to build a model of the categories variation in appearance . This variation will be caused by effects like aspect , in-class variation , and the particular RPN ’ s choice of boxes . But for novel categories , an RPN must report as many high IOU boxes as possible , because otherwise the classifier will have too weak a model of appearance variation – for example , it might think that the object must be centered in the box . This will significantly depress accuracy . As Figure 1 and our results illustrate , this effect ( which we call proposal neglect ) is present in the state-of-the-art few-shot detectors . One can not escape this effect by simply reporting lots of boxes , because doing so will require the classifier to be very good at rejecting false positives . Instead , one wants the box proposal process to not miss high IOU boxes , without wild speculation . We offer a relatively straightforward strategy . We train multiple RPN ’ s to be somewhat redundant ( so that if one RPN misses a high IOU box , another will get it ) , without overpredicting . In what follows , we demonstrate how to do so and show how to balance redundancy against overprediction . Our contributions are three-fold : ( 1 ) We identify an important effect in few-shot object detection that causes existing detectors to have weaker than necessary performance in the few-shot regime . ( 2 ) We propose to overcome the proposal neglect effect by utilizing RPN redundancy . ( 3 ) We design an RPN ensemble mechanism that trains multiple RPN ’ s simultaneously while enforcing diversity and cooperation among RPN ’ s . We achieve state-of-the-art performance on COCO and PASCAL VOC in very few-shot settings . 2 BACKGROUND . Object Detection with Abundant Data . The best-performing modern detectors are based on convolutional neural networks . There are two families of architecture , both relying on the remarkable fact that one can quite reliably tell whether an image region contains an object independent of category ( Endres & Hoiem , 2010 ; van de Sande et al. , 2011 ) . In serial detection , a proposal process ( RPN/s in what follows ) offers the classifier a selection of locations likely to contain objects , and the classifier labels them , with the advantage that the classifier “ knows ” the likely support of the object fairly accurately . This family includes R-CNN and its variants ( for R-CNN ( Girshick et al. , 2014 ) ; Fast R-CNN ( Girshick , 2015 ) ; Faster R-CNN ( Ren et al. , 2015 ) ; Mask R-CNN ( He et al. , 2017 ) ; SPP-Net ( He et al. , 2015 ) ; FPN ( Lin et al. , 2017 ) ; and DCN ( Dai et al. , 2017 ) ) . In parallel , the proposal process and classification process are independent ; these methods can be faster , but the classifier “ knows ” very little about the likely support of the object , which can affect accuracy . This family includes YOLO and its variants ( for YOLO versions ( Redmon et al. , 2016 ; Redmon & Farhadi , 2017 ; 2018 ; Bochkovskiy et al. , 2020 ) ; SSD ( Liu et al. , 2016 ) ; Cornernet ( Law & Deng , 2018 ) ; and ExtremeNet ( Zhou et al. , 2019 ) ) . This paper identifies an issue with the proposal process that can impede strong performance when there is very little training data ( the few-shot case ) . The effect is described in the context of serial detection , but likely occurs in parallel detection too . Few-Shot Object Detection . Few-shot object detection involves detecting objects for which there are very few training examples . There is a rich few-shot classification literature ( roots in ( Thrun , 1998 ; Fei-Fei et al. , 2006 ) ) . Dvornik et al . ( 2019 ) uses ensemble procedures for few-shot classifi- cation . As to detection , Chen et al . ( 2018 ) proposes a regularized fine-tuning approach to transfer knowledge from a pre-trained detector to a few-shot detector . Schwartz et al . ( 2019 ) exploits metriclearning for modeling multi-modal distributions for each class . State-of-the-art few-shot detectors are usually serial ( Wang et al. , 2019 ; Yan et al. , 2019 ; Wang et al. , 2020 ; Fan et al. , 2020 ; Wu et al. , 2020 ; Xiao & Marlet , 2020 ) . The existing literature can be seen as variations on a standard framework , where one splits data into two sets of categories : base classes Cb ( which have many training examples ) and novel classes Cn ( which have few ) . The RPN and classifier are then trained with instances from the base classes , producing a detector for |Cb| categories . The final layer of the resulting classifier is expanded to classify into |Cb|+ |Cn| classes by inserting random weights connecting the final feature layer to the |Cn| novel categories . Now the model is fine-tuned using either only the novel class instances or a balanced dataset containing training instances of both base and novel classes . Wang et al . ( 2020 ) shows that a simple two-stage fine-tuning approach outperforms other complex methods . Much work seeks improvements by applying few-shot classification techniques . Kang et al . ( 2019 ) designs a meta-model that learns to reweight pre-trained features given few-shot data . Wang et al . ( 2019 ) and Yan et al . ( 2019 ) further explore the meta-learning direction by attaching meta-learned classifiers to Faster R-CNN . Wu et al . ( 2020 ) improves few-shot detection by positive sample refinement . Relatively little work adjusts the proposal process , which is usually seen as robust to few-shot issues because there are many base examples . One possibility is to introduce attention mechanisms and feed category-aware features instead of plain image features into the proposal process ( Hsieh et al. , 2019 ; Fan et al. , 2020 ; Xiao & Marlet , 2020 ; Osokin et al. , 2020 ) , as well as re-ranking proposals based on similarity with query images ( Hsieh et al. , 2019 ; Fan et al. , 2020 ) . Making the RPN category-aware improves the quality of novel class proposals , but at inference time the model suffers from catastrophic forgetting of base categories – current category-aware features can not summarize the very large number of base class examples efficiently or accurately . An RPN that is generally well-behaved can still create serious trouble in the few-shot case by missing high IOU proposals for the novel classes during fine-tuning – the proposal neglect effect . We show that this problem is severe in the few-shot regime , and can be fixed by a carefully constructed ensemble of RPNs without substantial loss of performance for the base classes . Evaluating Few-Shot Detectors . The standard procedure is to compute average precision ( AP ) separately for novel and base categories for a detector that is engineered to detect |Cb| + |Cn| classes , typically using standard test/train splits and standard novel/base splits ( Wang et al. , 2020 ) . This evaluation procedure is the same as in incremental few-shot detection ( Yang et al. , 2020 ; PérezRúa et al. , 2020 ) . This procedure makes sense , because in most applications an incoming test image could contain instances from both base and novel classes . Furthermore , the standard procedure exposes any catastrophic forgetting that occurs . However , other evaluation methodologies occur , and some detectors are evaluated using variant procedures , making the comparison of AP ’ s difficult . In one variant , the detector detects only the |Cn| novel classes or only one novel class . In this paper , all reported results are for the standard procedure ; when relevant , we re-implement and re-evaluate using the standard procedure . 3 OUR APPROACH . We believe that the proposal neglect effect is generic , and it applies to any detector that uses a structure like the standard structure . For this reason , we focus on finding and fixing the effect within a standard state-of-the-art few-shot object detection framework , as described below . Few-Shot Object Detection Framework . We use the few-shot detection setting introduced in Kang et al . ( 2019 ) . We split the dataset into two sets of categories : base classes Cb and novel classes Cn . As shown in Figure 2 , the training process is two-phase : ( 1 ) base classes training , and ( 2 ) finetuning with novel classes . In phase 1 , the model is trained with base class instances which results in a |Cb|-way detector . After base classes training , weights for novel classes are randomly initialized , making the classifier a ( |Cb|+ |Cn| ) -way classifier . In phase 2 , the model is fine-tuned using either a set of few novel class instances or a balanced dataset containing both novel and base classes . After the fine-tuning phase , we evaluate our model by average precision ( AP ) on novel and base categories . Although the focus of few-shot detection is the novel classes , since most test images contain instances from both base and novel classes , it is essential to maintain good performance on base classes . We adopt the widely-used Faster R-CNN ( Ren et al. , 2015 ) as our base model . As shown in Figure 2 , Faster R-CNN is a serial detector , which consists of a backbone image feature extractor , a regional proposal network ( RPN ) , followed by the region of interest ( ROI ) pooling layer , and a bounding box classifier and a bounding box regressor on top of the model . The RPN determines if a box is a foreground or a background box . Following the RPN is non-maximum suppression ( NMS ) which ranks and selects top proposal boxes . After passing the ROI pooling layer , the predictor ’ s head classifies and localizes each box . In phase 1 , the whole model is trained on many-shot base class instances . Phase 2 fine-tunes the head classifier and the bounding box regressor with novel class instances ; everything before and including the proposal generator is frozen . Learning Cooperating RPN ’ s ( CoRPNs ) . We wish to avoid high IOU boxes for novel classes being dropped by RPN when novel classes are trained . Our strategy is to train multiple redundant RPN ’ s . The RPN ’ s should be distinct , but cooperate – if one misses a high IOU box , we want another to find it . However , they should not be so distinct that the classifier is flooded with false positives . As Figure 3 shows , Faster R-CNN ’ s RPN consists of a feature extractor , a binary classifier ( which decides whether a box is foreground or background ) , and a bounding box regressor ( which is not relevant to our current purpose ) . There is no reason for our RPN ’ s to use distinct sets of features , and we do not want to create problems with variance , so we construct redundant classifiers while keeping both the feature extractor and the bounding box regressor shared between all RPN ’ s . In what follows , a reference to an RPN is actually a reference to the RPN classifier , unless otherwise noted . An RPN with a single classifier is trained with a cross-entropy loss Lcls = LCE and produces a single prediction . In our case , we train N different binary classifiers simultaneously , and must determine ( 1 ) what prediction is made at test time and ( 2 ) what gradient goes to what classifier at training time . At test time , a given box gets the score from the most certain RPN . If the highest foreground probability is closer to one than the highest background probability , the box is foreground ; otherwise , it is background . Training time is more interesting . Merely taking the gradient of the best RPN score is not good enough , because we may find that one RPN scores all boxes , and the others do nothing interesting . For any foreground box , we want at least one RPN to have a very strong foreground score and all others to have good foreground scores too ( so that no foreground box is missed ) . We use the following strategy . For a specific anchor box i , each RPN j outputs a raw score rji , indicating if the box is a foreground box or not : ri = [ r1i , r 2 i , . . . , r N i ] . After applying a sigmoid , the jth RPN produces the foreground probability f ji = σ ( r j i ) for anchor box i . We choose the score from the j∗th RPN such that j∗ = argminj min { f j i , 1 − f j i } , namely the most certain RPN which produces probability closest to the edge of the [ 0 , 1 ] interval . At training time , only the chosen j∗th RPN gets the gradient from anchor box i . The RPN selection procedure is per-box , even adjacent boxes can pass through different RPN ’ s . Other than the standard cross-entropy loss , we use two additional loss terms : a diversity loss Ldiv encourages RPN ’ s to be distinct , and a cooperation loss Lcoop encourages cooperation and suppresses foreground false negatives . The final loss Lcls : = Lj ∗ CE + λdLdiv + λcLcoop , where λd and λc are trade-off hyperparameters . Enforcing Diversity . We do not want our RPN ’ s to be too similar . For each positive anchor box , RPN responses should be different because we want ensure that at novel class training time , if one RPN misses a high IOU anchor box , another will find it . To this end , we propose a methodology to enforce diversity among RPN ’ s . Given a set of NA anchor boxes , the N RPN ’ s produce an N by NA matrix of probabilities F = [ f1 , f2 , . . . , fN ] T . The covariance matrix Σ ( F ) is Σ ( F ) = E [ ( f j −E [ f j ] ) ( fk −E [ fk ] ) ] . We define the diversity loss Ldiv by the log determinant loss Ldiv : = − log ( det ( Σ ( F ) ) ) . By the diversity loss , we encourage the probability matrix to have rank N , so each RPN is reacting differently on the collection of NA boxes . This procedure ensures each RPN to be the most certain RPN for some boxes , so that every RPN is being selected and trained . Omitting this loss can cause an RPN to receive no or little training . Learning to Cooperate . We also want the RPN ’ s to cooperate so that they all agree to a certain extent for foreground boxes . We propose a cooperation loss to prevent any RPN from firmly rejecting any foreground box . For foreground box i , with the jth RPN , we define the cooperation loss Li , jcoop : = max { 0 , φ− f j i } , where φ is a constant parameter ( usually less than 0.5 ) , acting as a lower bound for each RPN ’ s probability assigning to a foreground box . If a RPN ’ s response is below φ , that RPN is going to be penalized . The final cooperation loss is an average of cooperation losses over all foreground boxes and all RPN ’ s .

Summary: In this paper, the authors present a interesting, novel idea of promoting the diversity and cooperation among multiple RPNs for the problem of few-shot detection. They first identify a critical problem in few-shot detection, which is that existing RPNs can miss objects of novel classes and that their proposed boxes make very similar errors. In order to resolve this issue, the authors propose to utilize multiple RPNs in the same detection pipeline and include a diversity loss when training the RPNs, such that they provide diverse scores to the anchor boxes. In this way, the chance of all the RPNs missing an object becomes small. The authors further include a cooperation score to enforce the RPNs to provide more meaningful scores (rather than sparse scores just to promote diversity). The proposed approach is evaluated on COCO and Pascal, and the results demonstrate that the proposed approach does improve the detection performance, as compared to the state of the art, especially in very low-shot cases.