Decoupling Classifier for Boosting Few-shot Object Detection and Instance Segmentation
Authors: Bin-Bin Gao
Xiaochen Chen
Zhongyi Huang
Congchong Nie
Jun Liu
Jinxiang Lai
GUANNAN JIANG
Xi Wang
Chengjie Wang
Venue: NeurIPS 2022 Accept (2022)
Keywords: few-shot object detection
few-shot instance segmentaton
instance-level few-shot
missing label

Abstract: This paper focus on few-shot object detection~(FSOD) and instance segmentation~(FSIS), which requires a model to quickly adapt to novel classes with a few labeled instances. The existing methods severely suffer from bias classification because of the missing label issue which naturally exists in an instance-level few-shot scenario and is first formally proposed by us. Our analysis suggests that the standard classification head of most FSOD or FSIS models needs to be decoupled to mitigate the bias classification. Therefore, we propose an embarrassingly simple but effective method that decouples the standard classifier into two heads. Then, these two individual heads are capable of independently addressing clear positive samples and noisy negative samples which are caused by the missing label. In this way, the model can effectively learn novel classes while mitigating the effects of noisy negative samples. Without bells and whistles, our model without any additional computation cost and parameters consistently outperforms its baseline and state-of-the-art by a large margin on PASCAL VOC and MS-COCO benchmarks for FSOD and FSIS tasks.\footnote{\url{https://csgaobb.github.io/Projects/DCFS}.}

Decoupling Classifier for Boosting Few-shot Object Detection and Instance Segmentation
Bin-Bin Gao
Xiaochen Chen'
Zhongyi Huang'
Congchong Nie Chengjie Wang!
Jun Liu
Jinxiang Lai"
Guannan Jiang"
Xi Wang"
Tencent YouTu Lab
2CATL
Abstract
This paper focus on few-shot object detection (FSOD) and instance segmenta- tion (FSIS), which requires model to quickly adapt to novel classes with few labeled instances. The existing methods severely suffer from bias classification because of the missing label issue which naturally exists in instance-level few- shot scenario and is first formally proposed by us. Our analysis suggests that the standard classification head of most FSOD or FSIS models needs be decoupled to mitigate the bias classification. Therefore, we propose an embarrassingly simple but effective method that decouples the standard classifier into two heads. Then, these two individual heads are capable of independently addressing clear positive samples and noisy negative samples which are caused by the missing label. In this way, the model can effectively learn novel classes while mitigating the effects of noisy negative samples. Without bells and whistles, our model without any addi- tional computation cost and parameters consistently outperforms its baseline and state-of-the-arr by a large margin on PASCAL VOC and MS-COCO benchmarks for FSOD and FSIS tasks
Introduction
Fully supervised deep convolutional neural network have achieved remarkable progress on various computer vision tasks, such as image classification [5], object detection [12] 3. 29]] semantic seg- mentation 22 2 and instance segmentation [4] in recent years. However, the superior performance heavily depends on large amount of annotated images. In contrast, humans can quickly learn novel concepts from few training examples. To this end, few-shot learning paradigm and its goal aims to adapt novel classes when only providing few labeled examples (instances). Unfortunately, existing few-shot models are still far behind humans, especially for few-shot object detection (FSOD) and few-shot instance segmentation (FSIS).
presented,
Various methods have been proposed tackle the problem of the FSOD and FSIS. The earlier works +Z 39
mainly follow meta-learning paradigm M to acquire task-level knowledge on base classes and generalize better to novel classes. However, these methods usually suffer from com- plicated training process (cpisodic training) and data organization (support query pair). The recent transier- (fine-tuning) methods [33 35
2 significantly outperforms the earlier meth- meta-learning ones. Furthermore, it is more simple and efficient. These transfer-leare
ods mainly follow fully supervised object detection or instance segmentation framework, e.g.. Faster-RCNN 600 or MCsk-RCNN [14]. Therefore, it may be suboptimal for few-shot scenario.
The PASCAL VOC and MS-COCO 2 are widely used evaluate the performance of object detection or instance segmentation. Under fully supervised setting, the model can be well-trained
Conresponding author: B.-B, Gao (csgaobb@ and C. Wang (jasoncjwange tencent.com). https://csgaobb. github. httpb//o//ooooe////////o
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

on these two datasets because all interest objects are almost completely labeled. Under an instance- level feo-shot scenario, however, we find that there is a large number of instances that are missing annotations as shown in Fig. T (detailed discussion in Sec. 3.2 The reason is that the community considers an instance as a shot for controlling the number of labeled instances when building instance- level benchmarks. This is different from image-level few-shot image classification [2]] because there are generally multiple instances in an image for instance-level few-shot learning. In fact, missing (partial or incomplete) label learning is more difficult and challenging, especially instance-level few-shot scenarios. It requires that learning algorithms deal with training images each associated with multiple instances, among which only partial instances are labeled; while the common supervised learning typically assumes that all interest instances are fully labeled. In some real- world applications, such open- -ocabbulyy object detection 40, it is almost impossible to label all instances, and thus there still may exist some instances left to be missing labeled. In addition, it is more friendly and convenient for users label partial instances than all ones even in few-shot setrings.
Most methods 2 have been developed address missing label (partial label or incomplete label) learning but mainly focus on image-level multi-class multi-label classification. To address the instance-level missing label issue, some recent works have attempted to regard the missing (unlabeled) instances as hard negative samples and re- re-weight K77 re-calibrate [I]] their losses. However, these works still only focus on general object detection. For instance- few-shot recognition, may result in biased classification and thus limit the generalization ability of novel classes using the model trained on these mislabeled datasets if we don't 't take any action.
Recently, one work closely related to ours is the state-of-the-art DeFRCN Paster-RCNN to alleviate the brgground-backgrooun
23 which decouples
confusion between base pre-training and novel fine-tuning in FSOD. It also can be interpreted from a missing label perspective. Here, we could view fine-tuning few-shot learning paradigm as a domain adaption procedure from base to novel. In this procedure, a few-shot detector may suffer from one background proposal (negative class may be potential novel class) in the base learning stage will become foreground (positive class) in the novel fine-tuning phase. To mitigate the label conflict between the two domains, DeFRCN decouples RCNN and RPN by stopping gradient backpropagation of RPN in Faster-RCNN, RCNN. Different from the missing label of cross-domain in DeFRCN, we focus on the missing label issue only in the novel (or balanced base-novel) fine-tuning stage. Another recent work, Pseudo- Lseudo-Labelling Labelling 126 |R mines the missing labeled instances for increasing the number of positive training samples and reducing the biased classification. Unfortunately, this method may lead to mhicken-and-ege problem-we need a good fhw-shot detector to generate good pseudo labels, but we need good few-shot annotations to train good few-shot detector. Unlike this work, our method completely avoids using any pseudo-label information.
confusion because
In this paper, we propose a simple decoupling method to mitigate the biased classification issue. Specifically, we firstly decouple the standard classifier into two parallel heads, positive and negative ones. Then, these two heads independently process clear positive and noisy negative samples with different strategics. Our contributions are summarized follows:
We rethink FSOD and FSIS from the perspective of label completeness and discover that existing transfer-learning few-shot methods severely suffer from bias classification because the missing label issue naturally exists in instance-level few-shot scenarios. To be best of our knowledge, this is the first to propose missing label issue in FSOD and FSIS.
To mitigate the bias classification, we propose simple but effective method that decouples the standard classifier into two parallel heads to independently process clear positive samples and noisy negative ones. Without bells and whistles, the proposed decoupling classifier can be taken as an alternative to the standard classifier in state-of-the-ar FSOD or FSIS models.
Comprehensive experimental results on PASCAL VOC and MS-COCO show that our approach without any additional parameters and computation cost outperforms ofte-of-the-ars art both on FSOD and FSIS tasks.
2
Related Work
FSOD aims to recognize novel objects and localize them with bounding boxes when only providing a few training instances on each novel class. Existing works can be roughly grouped into two families, meta-lcarning and transfer-leaming, according to training paradigm. The meta-learning
2

---

6:36
8.20
Rote 1h1
10
1:
2.290 Sohet
>-16 10
10hel 2.00 3-AM 3.1 20-mn 30
(c) shot labeled image Figure I: The proportion of missing instances in the training set for FSOD and gFSOD (a) PASCAL-VOC and (b) AS-COCO datasets. can be observed that there a high missing rate in each shot, especially the gFSOD. In (c), two "person" instances present in the one-shot labeled image, but they are mislabeled.
(a) PASCAL VOC
(b) AS-COCC
episodic training to acquire task-level knowledge on base classes and generalize better to novel classes. The transfer-learning methods [6 [55 3l
methods
K
4 28 generally utilize two-stage training strategy first pre-training base classes and then fine-tuning on novel classes, which have significantly outperformed many earlier meta-learning approaches. Recent FSCE BI shows that the degradation of detection performance mainly comes from misclassifying novel instances as confusable classes, and they propose a contrastive proposal encoding loss to ease the issue. Similar the FSCE, FADI [1 explicitly associates each novel class with a semantically similar base class to learn a compact intra-class distribution. DeFRCN [28 alleviate the
decouples Faster-RCNN and fine-tuning stage.
30
confusion between pre-training
FSIS needs not only recognize novel objects and their location but also perform pixel-level semantic segmentation for each detected instance. Many FSIS approaches typically use the FSOD framework, e.g., Mask-RCNN [14], it is built the Faster-RCNN by adding a mask segmentation head. Therefore, most FSIS methods generally follow the FSOD learning paradigm, ie.. meta-
learning 23 39 24 transfer-learinng [10] Siamese Mask-RCN
23 and Meta-RCNN (39] commonly compute embeddings of support images and combine them with those features of query images produced by backbone network. Their difference is the combination strategy, e.g., subtraction in [23] and concatenation 39). These works only focus on the performance of novel classes and ignore that of base classes. In real-world applications, we expect that one few-shot model not only recognizes novel classes but also remembers base classes. The recent iMTFA [00] introduces incremental learning into FSIS and propose incremental FSIS task.
3 Methodology
3.1 Few-shot Object Detection and Instance Segmentation Setting
Given an image dataset D (X,, Y.J where X; denotes the i-th image and Y; is its corre- sponding annotation. For object detection, {== cs Je= where by and represent bounding box coordinates and category of the k-th instance presented image X,, respectively. For instance segmentation, Y, includes pixel-level mask my annotation beyond category and bounding box ones, i.., (y= m&, Cb c k=1- Under few-shot learning setting, these annotations can be grouped into two sets, base and novel classes, denoted Chase and Cnowet- Note that the base and novel classes are non-overlapping (i.e., TonsenCnoode
FSOD or FSIS aims to detect/segment novel class instances through training model based on plenty of labeled instances on a set of base classes and few instances (c.g. 1, 2, 3 and 5) on each novel class. Note, FSOD FSIS only focuses recognizing novel class instances but ignores base class ones. It is impractical from the perspective of many real-world applications because people always expect that a few-shot model is capable of not only recognizing novel classes but also remembering base classes. To this end, generalized FSOD and FSIS (abbreviated as gFSOD and gFSIS) [I] 27 6 10
stresses that good few-shot learning system should adapt to tasks (novel classes) rapidly while maintaining the performance on previous knowledge (base classes) without forgetting.
As mentioned above, transfer-learning FSOD and FSIS methods mainly consist of two stages, i.e., pre-training on base classes and fine-tuning on novel classes. The former is trained on plenty of labeled instances in Chase while the latter only uses few labeled instances in Crovel. For generalized

---

few-shot learning, the main difference is novel stage training which uses few labeled instances in both base and Crovet Note that the labeled instances are abundant at the base stage, but there are only a few labeled instances at novel stage both FSOD/FSIS and gFSOD/gFSIS. Therefore, most instances are unlabeled (missing labeled) as only K instances are provided at the fine-tuning stage. This is, in fact, reasonable from the perspective of data privacy (incremental learning).
3.2 Revisiting Object Detection and Instance Segmentation
We focus on transfer-lerning FSOD and FSIS methods because they are more simple and effective compared to meta-learning ones. As we know, Rester-RCNN and Mask-RCNN are very popular and powerful solutions as two-stage stacking architecture for fully supervised object detection and instance segmentation. We firstly describe the two-stage detector and segmenter. In general, the first stage is designed to generate class-agnostic region proposals which can be formulated as:
Fsi (Osii)) FROT FRPN FEF())
where the 0=1 all the parameters of the first stage. Specifically, an input image X₃ is firstly fed into backbone network to Extract high-level Features (FEF). Then, Region Proposal Network (RPN) is adopted to generate candidate regions based on these extracted features (FRPN). Finally, all these region proposals are pooled into fixed size feature maps using Region of Interest (ROI) pooling module (FRoi) for the following stage. The structure of the second stage varies depending on the specific task. For object detection, the ROI features of sampled region proposals will be parallelly fed into two heads for performing box classification and regression, that is
C CCLs(Oets: -) + LEDG (freg:
(2)
where fats and freg the parameters of classification and regression head, respectively. Instance segmentation method (e.g.. BCck) follows the second stage structure of object detection and applies it to three heads, i.e., box classification, box regression and mask segmentation, that is
Is 2 ()) (CLs(fass:)) + (org(Oregi)) + CSEG(Osegi>
where seg is parameters of the mask segmentation head. Given an input image X, and its corre- ) sponding annotation (,= C, k=1 or (,- myg Cx Eqs. 2 and can be jointly optimized
end-to-end by minimizing C22(() (.) and C which follows a multi-tase learning paradigm. For simplicity, omit the RPN learning in Eqs. and
Missing Label Issue. We can obtain a powerful model by minimizing Eqs. 2 and when all interesting instances are completely labeled in large-scale image dataset D. The completeness of annotation at label space is, fact, a general precondition for most fully supervised learning algorithms. However, it is not satisfactory for a few-shot learning scenario. The reason is that only few instances are manually labeled given training images, and a lot of potential instances may be presented but unlabeled. From the perspective of practical application, it is natural and unavoidable to miss annotations when facing a sew-shot setting. As shown in Fig. (c), we can see that there at least three instances including two "person" and one "dog" in this image, but only the "dog" instance carrying annotation (box, mask, category), and other two "person" instances are unlabeled (i.e., missing labels).
In order to quantitatively measure the proportion of missing labeled instances, we compute the average missing rate for each shot on PASCAL VOC and MS-COCO benchmark datasets as shown in Fig. M Firstly, it can be seen that there is a high missing rate on each shot. For example, 74.3% novel class instances potentially present on 3-shot MS-COCO training images, but they are not given any annotations. Secondly, the missing rate is further increased under a generalized few-shot setting. For example, the missing rate increases nearly two times of few-shot setting the PASCAL VOC. Despite some efforts try mining these missing labeled instances in a cemi-supervised manner for boosting few-shot performance, the fully supervised loss with Eqs. 2 and may result in a suboptimal solution when some instances are unlabeled. To be best of our knowledge, this is the first to propose the missing label issue in few-shot object detection and instance segmentation.
Biased Classification Issue. The Eqs. 2 BandB] caan be optimized using sampled ROI features and their labels as mentioned above. The sampling operation performs box matching between "ROI features" and "annotation", and assigns training labels (positive or negative) to the corresponding ROIS. Here, the positive (foreground) ROIs are sampled from object proposals that have loU overlap with the

---

tround-truth bounding box at a threshold (e.g., 0.5), while negative (background) ROIs are sampled from the remaining proposals. The classification head is trained based on these sampled positive and negative ROIs. Different from the classification head, the box regression, and mask segmentation head learning are only associated with positive ROIs.
The annotations of positive ROIs are accurate, c.g., the annotations of the "dog" instances in Fig. (c). However, the sampled negative ROIs may be noisy because of the missing label issue under the few-shot setting. For example, those two "people" instances will be assigned to negative labels (i.e., background) if they are sampled in Fig. | (c), according to the above label assignment strategy. This will make the standard classification head confused with positive and noisy negative samples. On one hand, the model is correctly guided recognize positive objects because all positive samples are accurate. On the other hand, the model may be misguided by noisy negative samples and thus incorrectly recognize positive objects as background. Therefore, the bias classification may happen when meeting the missing label issue, especially in a few-shot scenario. Furthermore, it potentially limits the generalization ability to adapt to the novel class quickly and efficiently.
3.3 Decoupling Classifier to Mitigate the Bias Classification
Standard Classifier. We assume that a E RC+1 is the predicted logit of a sampled ROI feature obtained from Eq. [ and its corresponding class label vector is y RC+1, where there are C foreground categories and one background class, and yi is if the corresponding proposal belongs to the i-th category, otherwise. Then, we use softmax function to transform into probability distribution, that is
exp(xa)
(4)
E, exp(x:)
The cross-entropy loss is used as the measurement of the similarity between the ground-truth y and predicted distribution p. that is
C
LELS = Σ (log((.))
(5)
1=0
Note that the standard classifier (Eqs. 4 and 5) may confuse with clear positive and noisy negative samples in few-shot scenario.
Decoupling Classifier. In order to process positive and negative samples differentially, we decouple the standard classifier into heads, i.e., positive (foreground) head and negative (background) head, which are formulated as
bg CCLS chu + CLss
(6)
Here, the positive and negative heads are responsible for positive and negative samples, respectively. Considering that the labels of positive samples (foreground) are accurate, we can use cross-entropy loss (i.e.. Eq. 5 for all positive instances. The labels of those negative examples may be noisy because of the missing label issue. Therefore, it is not reasonable to employ normal cross-entropy loss for training the negative head. Note that these negative examples are generally sampled from those object proposals that have a maximum IoU overlap with the ground truth bounding box at interval [0, 0.5), and thus we can infer that they may not belong to the ground truth class, although we don't tt know their true category. We expect that the bias classification would be mitigated if the negative head performs learning only between few-shot labeled categories and the background class. To this end, we first obtain image-level mubti-label with instance-level few-shot annotation of a training image, and denote it as m=[m² .m! ..... m C-1 CIT where m' is a binary indicator, and m² is 1 if the image is labeled with the i-th category, 0 otherwise. Note that m indicates that each image at least contains a background class. Then, we can obtain a constrained logit ă conditioned on the m, that is
F; m.3.-
Substituting Eq. into the softmax function Eq. yields:
exp(m,a..)
pa Σ, exp(maar)
(8)
We compute cross-entropy loss between p and the corresponding ground truth y²x that is
C
cbg =
Σ
M log(p:),
(9)
1=0
5

---

where yC y and D -0.
bg
Optimization with Decoupling Classifier. Considering the joint optimization goal (Eq. 2 and 3 of object detection and instance segmentation, the optimal parameters @ is determined by minimizing Eq. 2 and 3 where = {6.1, Bets, rreg| } in object detection, and e = f0,1, 8cts, Oreg sseg | in
instance segmentation. For simplicity, only consider the optimization for the classification head and omit the box regression and mask segmentation head in the following analysis. Odt is updated by gradient descent step, that is
Oas +- Octs &LCLS
(10)
00ats
where À is the learning rate. Note that we have decoupled the standard classifier into positive and negative learning in Eq. 6 We firstly analyze the Oets optimization for positive head. According to the chain rule in Eq and we have
ac dr
- D nd
(11)
Then, the derivative of fg CLS
with respect to Oets is
act
dx Oets
(f- - y
(12)
a0
Combining the negative head in Eq. with respect to Oat is
and Eqs. 8 and
we can similarly obtain derivative of T CLS
act
dx
m(p b8)
(13)
80.s
Octs
where y is the ground truth label vector of a negative sample. Comparing Eqs. 12 with 13
we can
that the parameters Oot of the classification head will be updated with different ways for positive and negative examples. For the positive head, the gradient is updated in each dimension of the class space. But for the negative head, the gradient is limited in some special dimensions because of the introduced m and thus the bias classification may be alleviated.
In order further understand Eqs. and 13 we give a visualization example for decoupling classifier illustrated in Fig. 2 Note that we use Gaussian normal distribution for prediction and ground truth distribution for intuition. Here, we take the one-shot labeled image in Fig. (c) for example, where only the "dog" instance is labeled. We assume that one "person" instance sampled negative sample, and it will be mis- taken as background class (ground truth). Due to the proposed decoupling classifier, the opti- mization of the "person" instance is constrained between the "dog" and "background" as shown in Fig. [ and doesn't t affect the predictions of other categories such as the "person" class.
22 person of
5bg 06
hg
29
perennce
dog
07
bg
(a) Positive head
(b) Negative head
Figure 2: Illustration of the gradient of decoupling classifier, where the blue arrow represents the gradient dircction. (a) illustrates the gradient propagation the positive head, and (b) reveals that the gradient propaga- tion constrained between few-shot labeled class (e.g.. dog) and the background and thus the bias classification is mitigated. Best viewed in color and zoom in.
Experiments
In this section, we empirically evaluate the proposed method for FSOD/gFSOD and FSIS//FSIS tasks and demonstrate its effectiveness by comparison with state-of-the-ar methods.
4.1 Experimental Setup
Datasets, We follow the previous works and evaluate our method on PASCAL VOC and MS- COCO 2ll datasets. For fair comparison,
the same data splits given in 63 28
PASCAL VOC covers 20 categories, which are randomly split into 15 base classes and novel classes. There are three such splits in total. In each class, there are K (1, 2, 3, 5, 10) objects for

---

Table FSIS performance for Novel classes on COCC. The superseript indicates that the results are our re-implementation. The results are averaged over 10 seeds and the hest ones are in bold, the same below.
Methods
Tasks
5
10
30
AP
AP50
AP
AP50
AP50
AP
AP50
AP
AP50
AP
AP50
R-CNN 33
ICCV
9.9
MTTA [0]
CVPR 21
2.47
4.85
6.61
12.32
8.52
15.53
IMTFA 1T
CVPR 21
Der
3.28
6.01
6.22
11.28
7.14
12.91
Mask-DeFRCCT
13
ICCV
7.54
11.01
20.20
23.28
27.29
32.80
Ours
8.09
15.85
11.90
22.39
14.04
25.74
16.39
29.96
19.33
34.78
22.73
40.24
Meta R-CNN
13
ICCV 19
2.8
6.9
4.4
10.6
MTEA 10
CVPR 21
2.66
4.56
6.62
11.58
8.39
14.64
IMTFA 10)
CVPR
Seg
2,83
4.75
5.24
8.73
5.94
9.96
Misk-DeFRCNI
X8
2CCV211
6.69
13:24
9:51
18.58
11.01
21.27
12.66
24.58
15.39
29.71
18.28
35.20
7.18
14.33
20.43
23.24
13:48
26.67
31.33
18.34
35.99
few-shot training. And the PASCAL VOC 2007 testing set is used for evaluation. The dataset is only used to evaluate FSOD task. We report the Average Precision (IoU=0.5) for novel classes (AP50).
MS-COCO contains 80 classes. The 20 categories presented in the PASCAL VOC are used as novel classes and the remaining 60 categories are used as base classes. We train a few-shot model based on K (1, 2, 3, 5, 10, 30) instances for each class and evaluate on the MS-COCO validation set. This dataset has been widely used to evaluate the performance of FSOD and FSIS. We report Average Precision (loU=0.5:0.95) Average Precision (loU=0.5) on novel classes for FSOD and FSIS settings. In addition, we also report AP and AP50 for overall classes, base classes, and novel classes under gFSOD and gFSIS settings, respectively.
Experimental Details. The experiments are conducted with Detectron2 36 on NVIDIA GPU V100 on CUDA 11.0. We use standard Faster-RCNN with ResNet- 101 backbone extracted features from the final convolutional layer of the 4-th stage for few-shot object detection, which is the same as DeFRCN 128]. For instance segmentation, we add a mask prediction head at the ROI of the FaNter-RCNN. For model training, we employ a two-stage transfer- learning approach: first training the network on the base classes with pre-trate pre-trained by ImageNet and then fine-tuning on KCshots for every class. The SGD used to optimize our network end-to-end with a mini-batch size of 16, momentum 0.9, and weight decay 5e on 8 GPUs. The learning rate is set to 0.02 during base training and 0.01 during few-shot fine-tuning. Following the previous work [33, all experimental
results are averaged over 10 seeds. For a fair comparison with DeFRCN 28 we also report the average results over 10 times repeated runs on seed0.
Strong Baseline. For FSOD and gFSOD, we take the state-of-the-art DeFRCN 2S as strong baseline of our method. For FSIS and gFSIS, we extend DeFRCN similarly to how Mask-RCNN extends Faster-RCNN, i.e., adding a mask prediction head at the ROI of the DeFRCN and keeping others as same as DeFRCN, and thus we call Mask-DeFRCN Our method only replaces the standard classifier in DeFRCN and Deek- with our decoupling classifier. Therefore, DeFRCN and Mask-DeFRCN can be taken as our strong baseline for FSOD/gFSOD and FSIS//FSIS tasks.
4.2 Comparison with the State-of-the-Art
Few-shot Instance Segmentation on the MS-COCO. Our method (simple decoupling classifier) outperforms the state-of-the-art on the MS-COCO in both FSIS and gFSIS settings. The main results on MS-COCO are reported in Table and for FSIS and gFSIS, respectively. Based on the experiment results, we have the following observations: 1) The strong baseline (i.e., Mask-D)FRCN) outperforms all the state-of-the-art methods for FSIS and gFSIS; 2) Our method consistently outperforms the baseline Mask-DeFRCN for FSIS; Compared to FSIS, our method has significant improvements for gFSIS. This is not surprised because the missing rate of gFSIS always is higher than that of FSIS as shown in Fig. (b). This indicates our method capable of addressing the missing label issue under few-shot setting; 3) Our method has a better advantage, especially in low-shot (I-, 2-, 3-shot), and thus is very suitable for a few-shot scenario.
On one hand, the missing rate of low-shot is generally higher than that of high-shot so that it leaves more improvement space for our method. On the other hand, common few-shot models may be weak against noisy negative samples when the number of positive training samples is very small under few-shot conditions. In contrast, our method is designed to deal with noisy negative samples issue, and thus it is more effective. In short, the proposed decoupling classifier is a promising approach to cope with the missing labels issue for FSIS//FSIS.
7

---

Table 2: eFSIS performance for Overall, Base and Novel classes on AS-COCO.
Object Detection
Instance Segmentation
Shots
Methods
Overall
Base
Novel
Overall
Base
Novel
AP
AP50
AP
AP50
AP
AP50
AP
AP50
AP
AP50
AP
AP50
Base-Ouly
39.86
59.25
32.58
55.12
iMTHA 10]
21.67
31.55
27.81
40.11
3.23
5.89
20.13
30.64
25,90
39.28
2.81
4.72
1
Mask-DeFRCN
23.82
35.70
30.11
44,42
4.95
9.55
19.58
33.38
24.63
41.57
4.45
8.81
27.35
42.55
34.35
52.46
6.34
12.79
22.45
39.33
28.03
48.60
5.72
11.53
Blask-De--RNN
A8
25.42
38.31
31.06
45.82
8.52
15.79
21.09
35.92
25.61
43,03
7.54
14.59
2
Ours
28.63
44,74
34.67
52.82
10.52
20.49
23.73
41.49
28.52
49.12
9.38
18.62
3
Mast-DeFRCN
26.54
31.77
46.83
10.87
19:55
22.04
37.48
26.
43,95
9.48
18.06
Ours
29.59
46.21
35.07
$3.30
13.15
24.95
24.55
42.81
28.91
49.61
11.46
22.43
IMTFA [10
19.62
28.06
24.13
33.69
6.U7
11.15
18.22
27.10
22.56
33.25
5.19
8.65
5
Mask-DeFRCN
27.82
42.12
32.54
48.03
13.69
24.41
23.03
39.37
26.84
45.04
11.60
22.36
Ours
30.48
47.75
35.30
$3.65
16.02
30.05
25.20
44.12
29.10
49.87
13,50
26.86
IMTFA 10)
19.26
32.41
12.72
17.87
26.46
21.87
32.01
5.88
9.81
10
Mask-DeFRCN
43
29.88
45.25
34.17
50.48
17.02
29.58
24.75
42.32
28.23
47.33
14.32
27.29
Ours
31.77
49.77
36.14
$4.85
18.67
34.55
26.36
46.13
29.91
51.11
15.71
31.19
Mask DeFRCN Ours
28
31,66
35.10
52.01
21.33
26.23
44.97
29.12
48.55
17.57
33.42
30
32.92
51.37
36.45
$5.05
22.30
40.31
27.31
47.61
30.32
51.41
18.29
36.22
Table FSOD and gFSOD performance (APso) for Novel classes on PASCAL VOC. The term w/g
indicates whether we use the gFSOD setting [3)) The superscript * indicates that the results are averaged over 10 times repeated runs on seed0, the same below.
Methods / Shots
w//
2
Novel Set
3
2
Novel Set
3
2
Novel Set
3
5
FRCN-N 309
ICCV 19
X
13.8
19.6
32.8
41.5
45.6
7.9
15.3
26.2
31.6
39.1
9.8
11.3
19.1
35.0
45.1
FSRW [T]]
ICCV
X
14.8
15.5
26.7
33.9
47.2
15.7
15.2
22.7
30.1
40.5
21.3
25.6
28.4
42.8
45.9
MetaDet X3
ICCV 19
X
18.9
20.6
30.2
36.8
49.6
21.8
23.1
27.8
31.7
43.0
20.6
23.9
29.4
43.9
44.1
MetaRCNN
ICCV
X 19.9
25.5
35.0
45.7
10.4
19,4
29.6
14.3
18.2
27.5
41,2
48.1
TEA 55
ICML 20
X 39.8
36.1
44.7
55.7
56.0
23.5
26.9
34.1
35.1
39.1
30.8
34.8
42.8
49.5
49.8
MPSR K5
ECCV 20
X 41.7
51,4
55.2
61.8
24.4
19.2
39,9
47.8
35.6
42.3
48.0
49.7
27.7
36.5
43.3
22.7
30.1
33.8
40.9
30.6
38.1
44.5
DCNet [G
CVPR 21
X 33.9
37.4
43.7
$1.1
59.6
23.2
24.8
30.6
36.7
46.6
32.3
34.9
39.7
42.6
50.7
CMF XX]
CVPR 21 X 41.5
47.5
50.4
58.2
60.9
27.2
30.2
41
42.5
46.8
34,3
39.5
45.1
48.3
$1.5
FSCE 111
CVPR
K
44.2
43.8
51A4
61.9
63.4
27.3
29.5
43.5
44.
37.2
41.9
47.5
54.6
58.5
SKR-
CVPR 21
X
47
50.5
$1.3
55.2
56.8
32.5
35,3
39.1
40.8
43.8
40.1
41.5
44.3
46,9
46.4
FADI [|
NewrlPS 21
X
50.3
51.8
51.2
59.3
63.2
30.6
35.0
10.3
42.8
48.0
45.7
49.7
19.1
55.0
59.6
FCT [R]
CVPR 22
X
38.5
49.6
53.5
59.8
64.3
25.9
34.2
40.1
44.9
47.4
34.7
43.9
49.3
53.1
56.3
DeFRCN
2CCV
X 46.2
56.4
59.3
62.4
63.7
32.6
39.9
44.5
48.3
51.8
39.8
49.9
52.6
56.1
59.7
Ours
X 46.2
57.4
59.9
62.9
64.5
32.6
39.9
43.4
47.9
51.3
40.3
50.5
53.88
56,9
60.7
DeFRCN
X8
2C
X
53.6
57.5
61.5
64.1
60.8
30.1
38.1
47.0
53.3
47.9
48.4
50.9
52.3
54,9
57.4
Ours
56.6
59.6
62.9
65.6
62.5
29.7
38.7
46.2
48.9
48.1
47.9
51.9
53.3
56.1
59.4
FRCN-N
ICCV
9.9
15.6
21.6
28.0
$2.0
9:4
13.8
17.4
21.9
39.7
8.1
13.9
19.0
23.9
44.6
FSRW 17
ICCV.19
14.2
23.6
29.8
36.5
35.6
12.3
19.6
25.1
31.4
29.8
12.5
21.3
26.8
33.8
31.0
TFA
133
JCML 20
25.3
36.4
42.1
47.9
52.8
18.3
27.5
30,9
34.1
39.5
17.9
27.2
34.3
40.8
45.5
ViDe
2C
24.2
35.3
42.2
49.1
21.6
24.6
31.9
37.0
45.7
21.2
30.0
37.2
43.8
49.6
DeHRON
28
10CV
/
40.2
53.6
58.2
63.6
66.5
29.5
39.7
43.4
48.
52.8
35.0
38.3
52.9
57.7
60.88
Ours
45.8
59.1
62.1
66.8
68.0
31.8
41.7
46.6
50.3
53.7
39.6
52.1
56.3
60.3
63..
Few-shot Object Detection the PASCAL VOC and MS-COCO. Our method significantly outperforms the state-of-the-ee art few-shot object detection methods by a large margin both on the PASCAL VOC and MS-COCO datasets der gFSOD setting again. For the FSOD set- ting, our method is also betler than the state- of-the-art under mosl cases. The results on the PASCAL. VOC and MS-COCO reported Ta- bles and respectively. Some interesting observations are summarized as follows: 1) Our method significantly and consistently exceeds the current otate-of-the-art DeFRCN under the gFSOD setting both on the PASCAL VOC and MS-COCO, which is similar that of the gF- SIS; 2) Our method is better than the strong DeFRCN in all shots on the MS-COCO and in most cases on the PASCAL VOC under the FSOD setting (averaging the results on 10 seeds). It is worth noting that our method's performance on the MS-COCO is close (maybe slightly worse, about 0.5%) to that of DeFRCN if we compare the results based on 10 times repeated runs the seed0. We recheck the missing rate of the seed0, and find
Table 5: FSOD performance (AP) for Novel classes on MS-COCO. The superscripts x indicate that the results are reported in DeFRCN 28)
Methods Shots
2
3
5
10
30
FRCN-R
39
ICCV
1.8
28
4,0
6.5
FSRW
ICCV
5.6
9.1
MetaDer
14
ICCV 19
7.1
11.3
MetaRCNN 80
ICCV 19
8.7
12.4
TFA
[3
ICML 20
5.4
6.0
7.7
10.0
13.7
MPSR
BCCV
6.7
7.1
8.7
9.8
FSDeVView
ICCV 20
4.5
66
7.3
10.7
12.5
14.7
TIP [
CVPR
16.3
18.3
CVPR 21
12.8
18.6
CME RM
21
PSCE
CVP8 21
SRR.PSD 53
CVPE
11.3
14.7
FAD]
11
NewriPS 21
5.7
7.0
3.6
10.1
12.2
16.1
FCT [k]
CVPR 22
7.2
9.8
12.0
15.3
20.2
DeFRCN
ICCV
22.5
Ours
8.1
12.1
14.4
16.6
19.5
22.7
DeFRCN
X
2CCV
9.3
12.9
14.8
16.1
18.5
22.6
Ours
10.0
13.6
15.7
22.2
8

---

Table 4: 2FSOD performance (AP) for Overall, Base and Novel classes on MS COCO.
Method/
10
30
0
FRCN-D 90
16.2
21.0
1.7
15.8
20.0
3.1
15.0
188
3.7
14.4
17.6
4.6
134
16.1
5.5
13.5
15.6
7.4
TFA
24.9
25.9
7.
32.4
12.1
View
3.2
4.9
6.7
8.1
10.7
15.9
DeFRCN
P3
24.4
30.4
4.8
25.7
314
8.5
26.6
32
10.7
27.8
32.6
13.6
29.7
34.0
16.8
31.4
34,8
21.2
Ours
27.4
34,4
6.2
28.6
34.7
10.4
29.4
34,9
12.9
30.2
35,0
15.7
31,
35.7
18.3
32.3
35.8
21.9
Table 6: The effects of DC and PCB for gFSIS performance on MS AI-CCOCO GFLOPs are averaged over all 5000 MS-COCO validation images.
Complexity
Delection
Segmentation
Shots
M-Rate
DC
PCB
#aramss.
GFLOPs
Base
Novel
Base
Novel
AP
AP50
AP
AP50
AP
APS0
AP
AP50
54,9M
334.54
30,09
44.45
3.89
7.43
24.62
41.58
3.52
6.88
83.3%
54.9M
334.54
34.35
52.46
5.04
10.03
28.03
48.60
4.59
9.12
X
99.4M
377.88
30.11
44.42
4.95
9.55
24.63
41.57
4.45
8.81
99 AM
377.88
34.35
52.46
12.79
28.03
48.60
5.72
11.53
54,9M
334.54
32.54
48.03
11.94
21.16
26.84
45.04
10,10
19.37
54.9M
334.54
35,30
53.65
14.01
26.17
29.10
49.87
11.80
23,38
80.3%
99 99
377.88 88
32.54
48.03
13.69
24.41
26.84
45.04
11.60
22.36
99.4M
377.88
35.30
53.65
16.02
30.05
29.10
49.87
13,50
26.86
54.9M
334.54
34.05
50.21
14,96
25.70
28.12
47.10
12.60
23.81
76.7%
54,9M
334.54
36.13
54.81
16,66
30.79
29.90
51.07
13,98
27.72
99.4M
377.88
34.17
50.48
17.02
29.58
28.23
47.33
14.32
27.29
99.4M
377.88
36.14
54.85
18.67
34.55
29,91
51.11
15.71
31.19
that the corresponding missing rate is significantly lowered (even zero) than that of the other 9 seeds. This also further indicates that our method is robust when the missing rate small even zero.
4.3 Ablation Study and Analysis
We conduct the ablation study to analyze the component of our method. Models in this section are based on the gFSIS setting (I-, 5-, 10-shot) using MS-COCO. Note that the DeFRCN uses Prototypical Calibration Block (PCB) to refine the classification score which is effective for improving the FSOD performance, but this brings additional computation cost. Therefore, we consider these two factors including decoupling classifier (DC) head and PCB in the following analysis.
FG (Novel Classes) necall (%)
Xathul
50
Ctarheal CE-Racal
60
FG (All Classes) Recall
40-
50
30
40-
20
30
20
10
1-shot 2-shot 3-shot 5-shot 10-shot 30-shot
1-shot
2-shot 3-shot 5-shot 10-shot 30-shot
(a) FSIS
(b) gFSIS
Figure 3: Comparison mRecall and Recall of the proposed decoupling classifier (DC) and standard classification head (CE) under FSIS and gFSIS settings. The mean and standard deviation results are computed on all 10 seeds for each shot. Best viewed in color and zoom in.
We only simply replace the standard classifier with the proposed decoupling classifier in DeFRCN, which results in significant improvements, especially for the higher missing rate (e.g.. low shot on MS-COCO). What's 's more, our decoupling classifier is effective not only on novel classes but also on base classes, while the PCB seems only effective novel classes. In addition, our decoupling classifier without the PCB significantly outperforms the counterpart with the PCB on base classes and is also comparable novel classes.
Effectiveness.
Efficiency. Firstly, our decoupling classifier does not introduce any additional parameters or com- putation cost. Secondly, our method obtains better detection and segmentation performance when using the same complexity as Mask-DeFRCN whether the PCB used or not. Last but not least, we only need almost half of the parameters and fewer GFLOPs when removing the PCB block, and still achieve significant improvements on base classes and comparable performance on novel classes compared to Mask- Besk- using the PCB.

---

Mack DeFRC
Mack- DeFR-
Ours
Figure 4: Visualization results of our method and the strong baseline Baskk-DeFRCN) on MS-COCO validation images. Best viewed in color and zoom in.
Why DC works? We have given some analysis from the perspective of gradient optimization in Sec. Here, we try to discuss from the generalization ability of decoupling classifier and compare with the baseline. We want to explore whether the decoupling classifier mitigates the bias classification. To this end, we employ Recall metric to evaluate the classification head for all ground- foreground objects. Note that the classification head outputs multi-class probability distribution p E RC+1, The predicted class is determined by argmax,f; pú.- We define that an object is recalled its prediction not background, i.e., any foreground category. Considering that the number of each foreground category varies considerably, we also compare mltecall (mean Recalls of all classes). The comparison results shown in Fig. We outperforms the standard one on each shot both FSIS and gFSIS. This indicates that our decoupling classifier is helpful to mitigate the bias classification thus boosting the performance of FSIS and gFSIS.
that the mReecal and Recall of the decoupling classifier significantly
Qualitative Evaluation In Fig. visualize the results of our method and the strong baseline (Mask-DeFle Desk- on MS-COCO validation images with 10-shot setting for gFSIS task. In the top rows, show success cases with method but partly failures with the baseline. These failures are mainly caused by the missing detection because the baseline method may tend to incorrectly recognize positive objects background (i.e., bias classification). In addition, our method may also produce some failure predictions as shown in the bottom row from left to right, including the missing detection of small occultation objects, coarse boundary segmentation, and the misclassification of similar appearance objects.
Conclusion
In this paper, we firstly find that the missing label widely exists few-shot scenario. Furthermore, we analyze that the missing label issue may result in biased classification and thus limit the generalization ability on novel classes. Therefore, we propose a simple but effective method that decouples the standard classifier into two parallel heads to independently process positive and negative examples. Comprehensive experiments on the few-shot object detection and instance segmentation benchmark datasets show that approach can effectively and efficiently boost FSOD/gISSO and FSIS/gISSS performance without any additional parameters and computation cost. We hope this study attract more interest in designing simple method for FSOD FSIS in the future. A limitation of our method is that it may not be suitable when the missing label rate is small. However, our method is still comparable to its counterpart even if the missing label rate is zero, which indicates its robustness.
10

---

References
[1] Yuhang Cao, Jiaqi Wang, Ying Jin, Tong Wu, Kai Chen, Ziwei Liu, and Dahua Lin. Few-shot object detection via association and discrimination. In NewrIPS, 2021.
[2] Liang-Chieh Chen, George Papandreou, Jasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. TPAMI, 40(4))834-844 2017.
[3] Jifeng Dai, Haozhi Qi. Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In ICCV, pages 764-773, 2017.
[4] Mark Everingham, Luc Van Gool, Christopher Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJCV, 88(2):303-338, 2010.
[5] Qi Fan, Wei Zhuo, Chi-Keung Tang. and Yu- Wu-Wing Tai. Few-shot object detection with attention- rpn and multi-relation detector. In CVPR, pages 4013-4022, 2020.
[6] Zhibo Fan, Yuchen Ma, Zeming Li, and Jian Sun. Generalized few-shot object detection without forgetting. In CVPR, pages 4527-4536, 2021.
[7] Zhibo Fan, Jin-Gang Yu, Zhihao Liang, Jiarong Ou, Changxin Gao, Gui-Song Xia, and Yuanqing Li. Fgn: Fully guided network for few-shot instance segmentation. In CVPR, pages 9172-9181, 2020.
[8] Michael Fink. Object classification from single example utilizing class relevance metrics. In NIPS, 2004.
[9] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adapta- tion of deep networks. In ICML, pages 1126-1135, 2017.
[10] Dan Andrei Ganea, Bas Boom, and Ronald Poppe. Incremental few-shot instance segmentation. In CVPR, pages 1 185--1994, 2021.
[11] Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In CVPR, pages 4367-4375, 2018.
[12] Ross Girshick. Fast R-CNN. In ICCV, pages 1440-1448, 2015.
[13] Guangxing Han, Jiawei Ma, Shiyuan Huang, Long Chen, and Shih-Fu Chang. Few-shot object detection with fully cross-transformer. In CVPR, pages 5321-5330, 2022.
[14] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick, Mask R-CNN. In ICCV, pages 2961-2969, 2017.
[15) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, pages 770-778, 2016.
[16] Hanzhe Hu, Shuai Bai, Aoxue Li, Jinshi Cui, and Liwei Wang. Dense relation distillation with context-aware aggregation for few-shot object detection. In CVPR, pages 10185-10194, 2021.
[17] Bingyi Kang. Zhuang Liu, Xin Wang, Fisher Yu, Jiashi Feng, and Trevor Darrell. Few-shot object detection via feature reweighting. In ICCV, pages 8420-8429, 2019.
[18] Prannay Kaul, Weidi Xie, and Andrew Zisserman. Label, verify, correct: simple few shot object detection method. In CVPR, pages 14237-14247, 2022.
[19] Aoxue Li and Zhenguo Li. Transformation invariant few-shot object detection. In CVPR, pages 3094-3102, 2021.
[20] Bohao Li, Boyu Yang, Chang Liu, Feng Liu, Rongrong Ji, and Qixiang Ye. Beyond max-mamm max-mangin: Class margin equilibrium for few-shot object detection. In CVPR, pages 7363-7372, 2021.
[21] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. ECCV, pages 740-755, 2014.
11

---

[22] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR. pages 3431-3440, 2015.
[23] Claudio Michaelis, Ivan Ustyuzhaninov, Matthias Bethge, and Alexander S Ecker. One-shot instance segmentation. arXiv preprint arXiv:1811.115000 2018.
[24] Khoi Nguyen and Sinisa Todorovic. Fapis: A few-shot anchor-free part-based instance seg- menter. In CVPR, pages 11099-11108, 2021.
[25] Tam Nguyen and Raviv Raich. Incomplete label multiple instance multiple label learning. TPAMI, 2020.
[26] Yusuke Nittani, Takuya Akiba, Tommi Kerola, Toru Ogawa, Shotaro Sano, and Shuji Suzuki. Sampling techniques for large-scale object detection from sparsely annotated objects. In CVPR, pages 6510-651111 2019.
[27] Juan-Manuel Perez- Rua, Xiatian Zhu, Timothy M Hospedales, and Tao Xiang, Incremental few-shot object detection. In CVPR, pages 13846-13855, 2020.
[28] Limeng Qiao, Yuxuan Zhao, Zhiyuan Li, Xi Qiu, Jianan Wu, and Chi Zhang. DeFRCN: Decoupled faster r-cnn for few-shot object detection. In ICCV, pages 1681-8690, 2021.
[29] Joseph Redmon and Ali Farhadi. Yolo9000: better, faster, stronger. In CVPR, pages 7263-7271, 2017.
[30] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NIPS, pages 91-99, 2015.
[31] Bo Sun, Banghuai Li, Shengeai Cai, Ye Yuan, and Chi Zhang. FSCE: Few-shot object detection via contrastive proposal encoding. In CVPR, pages 7352-7362 2021.
[32] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. NIPS, 2016.
[33] Xin Wang, Thomas E Huang, Trevor Darrell, Joseph E Gonzalez, and Fisher Yu. Frustratingly simple few-shot object detection. In ICML, pages 9919-9928, 2020.
[34] Yu-Xiong Wang. Deva Ramanan, and Martial Hebert. Meta-learning to detect rare objects. In ICCV, pages 9925-9934, 2019.
[35] Jiaxi Wu, Songtao Liu, Di Huang, and Yunhong Wang. Multi-scale positive sample refinement for few-shot object detection. In ECCV, pages 456-472, 2020.
[36] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. wttp://ecceeoooeeeeerrraaaaaoaaa
2019.
[37] Zhe Wu, Navaneeth Bodla, Bharat Singh, Mahyar Najibi, Rama Chellappa, and Larry S Davis. Soft sampling for robust object detection. In BMVC, 2019.
[38] Yang Xiao and Renaud Marlet. Few-shot object detection and viewpoint estimation for objects in the wild. In ECCV, pages 192-210, 2020.
[39] Xiaopeng Yan, Ziliang Chen, Anni Xu, Xiaoxi Wang, Xiaodan Liang, and Liang Lin. Meta R- CNN: Towards general solver for instance-level low-shot learning. In ICCV, pages 9577-9586. 2019.
[40] Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. In CVPR, pages 14393-14444 2021.
[41] Han Zhang, Fangyi Chen, Zhiqiang Shen, Qiqi Hao, Chenchen Zhu, and Marios Savvides. Solving annottainnooootionn object detection with background recalibration loss. In ICASSP. pages 1888-1892, 2020.
[42] Min-Ling Zhang and Jun-Peng Fang. Partial multi-label learning via credible label elicitation. TPAMI, 43(10):3587-3599 2020.
[43] Chenchen Zhu, Fangyi Chen, Uzair Ahmed, Zhiqiang Shen, and Marios Savvides. Semantic relation reasoning for shot-stable few-shot object detection. In CVPR, pages 8782-8791, 2021.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section and S
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code. data, and instructions needed to reproduce the main experi- mental results (cither in the supplemental material or as a URL)? [Yes] See Section and htttps://csgaos
gi github. io PProjeect///DSS
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.1
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Fig. E and Tables R and Min in Appendix C
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Section 41
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] PASCAL VOC
MS-COCO 2 and Detectron2 [36]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [Yes] All datasets used publicly available.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "uh1_KGH7ZkQ", "reviewer": "Reviewer_1weo", "summary": "This paper addresses the missing label issue in few-shot object detection (FSOD) and few-shot instance segmentation (FSIS). Some foreground examples are not labeled in the few-shot scenario, which causes classification bias for the conventional classification head used in object detectors. This paper proposes a simple yet effective algorithm that treats the foreground and background proposals differently in the classification loss. The proposed method is evaluated on benchmark datasets and shows competitive performance against SOTA methods for both FSOD and FSIS.", "questions": "1. If the novel class training samples are provided in a way that every example is labeled in each image but only a small group of images are labeled (basically the strategy I mentioned in Cons), do you think the proposed method works better or worse? Also, will other FSOD/FSIS methods work better or worse in this case? Do you think using the proposed method with the current labeling strategy (i.e. randomly labeling a single class and ignoring others in one image) is still more beneficial as it sees more data?", "limitations": "The authors discussed the limitations", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n\n1. The proposed method is very simple yet effective. It decouples the loss function for foreground and background objects to remedy the effect on the classification head from the missing labels for potential foreground objects. The evaluations on benchmark datasets demonstrate its effectiveness in both FSOD and FSIS.\n2. The authors show that the proposed loss can be incorporated into other FSOD methods to achieve better performance. It also has no negative impact on base classes per the evaluation results. It might become a plug-in for many FSOD methods.\n3. The paper is generally well written and easy to follow.\n\nCons:\n\n1. While I acknowledge the issue of missing labels in the FSOD, it is more like a bad design in the current FSOD evaluation protocol rather than a real-world problem. It is weird when you have a lot of images, but you only label a single class and ignore others, especially when you know it will have a negative impact. If you have limited resources (as in a few-shot setting), shouldn't you focus on labeling a small group of images to ensure every class is labeled so that the total # labels are roughly similar? I understand the protocol is designed to control the number of shots, but it just seems unrealistic. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Ya4tNviElV", "reviewer": "Reviewer_DeAx", "summary": "The paper formally proposes the missing label issue which naturally happens in few-shot scenarios including few-shot object detection (FSOD) and few-shot instance segmentation (FSIS). The missing label issue may result in biased classification (mistakenly recognizing novel class objects as background, i.e., missing detection) and thus reduce the generalization performance of many current FSOD/FSIS models.  From the missing label perspective, the paper presents a simple but effective method that decouples the standard classifier into two parallel heads which are capable of addressing clear positive samples and noisy negative samples, respectively. Using the proposed decoupling classifier, the model can effectively learn novel classes because the effects of noisy negative samples are well mitigated. The effectiveness of the proposed method is evaluated on standard few-shot benchmarks PASCAL VOC and MS COCO for both FSOD/FSIS and generalized FSOD/FSIS tasks.", "questions": "- This missing label issue should be comprehensively analyzed from a broad perspective, e.g., between base and novel classes stages, the novel fine-tuning stage itself.\n\n- The inference pipeline (using the standard classifier or the decoupling classifier) should be more clarified.\n", "limitations": "Yes. The authors have discussed the limitations of the proposed methods in Sec.4 (Experiments) and Sec.5 (Conclusion). As stated in the paper, It may not be suitable when the missing label rate is small. However, the proposed method is still comparable to its counterpart even if the missing label rate is zero, which indicates the robustness of the proposed method. Considering that it is unavoidable to meet the missing label issue in few-shot FSOD and FSIS, especially under multiple categories scenarios and so it is ok for me.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ The paper observes the fact that the missing label issue naturally exists in few-shot scenarios such as FSOD and FSIS.  Furthermore, the authors analyze the missing label issue may result in biased classification and reduce the generalization performance of FSOD and FSIS. This new perspective might be inspiring to others in the field. \n\n+ In order to mitigate the effects of noisy negative samples and reduce the biased classification in FSOD and FSIS, the paper proposes a novel classifier decoupling idea that is quite interesting and strongly motivated. It can individually address clear positive samples and noisy negative samples and thus mitigate the biased classification issue. The proposed method is really simple and easy to understand and follow.\n\n+ The performance of the proposed method outperforms its baseline and state-of-the-art by a large margin on PASCAL VOC and MS-COCO benchmarks for both FSOD/FSIS and generalized FSOD/FSIS tasks. Meanwhile, the proposed method doesn’t introduce any additional computation costs and hyper-parameters because it only simply decouples the standard classifier into two parallel classifiers. Therefore, it seems like a practical algorithm that can be put immediately into practice.\n\n+ The proposed method is well analyzed and interpreted from the gradient optimization perspective. On the other hand, the authors evaluate the performance of the learned classifier using the Recall metric for all ground-truth foreground objects of all testing images and demonstrate that the proposed decoupling classifier indeed mitigates the biased classification. \n\n+ Ablation studies cover all the crucial components of the proposed approach. The experiment results are quite convincing.\n\n+ The manuscript is well written and the organization is very clarified.\n\nWeaknesses:\n- Relationship with some existing works~(e.g., DeFRCN). The missing label issue also, in fact, exists between base and novel classes. For example, the novel class objects may potentially present in base images at the base learning stage but these potentially novel objects are viewed as background and it is opposite to the base learning stage at the novel fine-tuning stage. Therefore, the missing label issue also leads to foreground-background confusion between base and novel learning stages because of this two-stage fine-tuning mechanism. I understand that this paper focuses on the missing label issue at the novel fine-tuning stage. However, it is more helpful for the readers to understand the proposed method if these two types of missing label issues can be comprehensively analyzed.\n\n- It is unclear how to use the decoupling classifier at the inference stage because the image-level few-shot label ($m_i$ in Eq. 7) is agnostic for any testing images. I guess that the standard classifier is used at the inference stage and the proposed decoupling classifier only is employed during training time. The authors should clarify it.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "SC2oWgFdn_i", "reviewer": "Reviewer_mATA", "summary": "This paper proposes a decoupled classifier to deal with positive and negative proposals separately when learning general few-shot object detection (FSOD) and few-shot instance segmentation (FSIS) networks. Specifically, the paper argues that for FSOD/FSIS problem, missing labels are a common issue. For an image, some classes could be ignored during annotation. The missing annotation problem would lead to a biased classifier, especially when learning with a few images. To solve this problem, the paper proposes to deal with positive proposals with normal cross-entropy loss and deal with negative proposals by only considering the positive class and background of the images.\n\nThe proposed method is implemented based on the architecture of DeFRCN [21] and evaluated on benchmark datasets VOC 15/5 and COCO 60/20. The method is compared with several SOTA methods, including DeFRCN, and appears to be better. \n\nOverall, the idea of the paper should be easily reproducible. The benchmark, evaluation, and comparison are solid. However, I think there are several major issues with the premise, and the proposed method could lead to more bias in certain situations. Details can be found in the  Strengths And Weaknesses section.", "questions": "Please refer to the weakness section for questions.", "limitations": "N/A", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strengths of the paper are:\n\n1. The paper is relatively easy to follow. The proposed method is described in detail and should be easily reproducible.\n2. The comparisons and evaluation are adequate, and the results are positive on the benchmark.\n\nThe main weaknesses of the paper are:\n\n1. My major concern with the paper is the premise. I think the paper proposes a solution to deal with the bias of existing benchmark datasets rather than the problem (FSOD/FSIS) themselves. In particular, in Fig 1(c), the paper mentions that for a one-shot image, 'dog' could be annotated and the 'person' could be ignored.  This situation is mainly due to how existing benchmarks (VOC/COCO) are proposed. Prior works [14, 26] proposed the benchmarks that consider an object instance (bounding boxes) as a 'shot'. In this case, the training images could just be sampled with one instance, even though there are multiple instances in the same image. I agree with the paper that this is an issue with these benchmarks and needs to be addressed. However, I do not think this is an inherent issue with FSOD/FSIS problems.  \n2. Even if the problem is inherent with FSOD, the proposed solution could create more bias rather than solving the problem. With the same example in Fig 1(c). If it is a one-shot scenario for the class 'person', then the other person might not be annotated. In fact, this is a very common problem for object detection, even for large-scale datasets. There are plenty of examples of 'group' bounding boxes with a single example instance in the OpenImages dataset (https://storage.googleapis.com/openimages/web/index.html).  These missing annotations would be considered negative ('background') and the proposed method restricted the learning to between ['person', 'background'], and backpropagate wrong losses. I think in many situations, the proposed method can lead to more bias and confusion.\n3. Some of the paper's claims are not well-grounded. For example, Line 94-95, the paper claims that \" It is impractical from the perspective of many real-world applications because ones always expect that few shot model is capable of not only recognizing novel classes but also remembering base classes\". However, for many few-shot learning papers (MAML [a] and Prototypical Network [b] for example), base-class performance is not a concern. Could the author motivate more on why we 'always expect the few-shot model to remember base classes‘？\n\n[a] Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks, Finn et al, ICML 2017\n[b] Prototypical Networks for Few-shot Learning,  Snell at al, NeurIPS 2017", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HOSUAdkyu6J", "reviewer": "Reviewer_LjLJ", "summary": "The paper proposes a method of correcting the missing label issue in the context of few-shot instance segmentation and few-shot object detection problems.", "questions": "- The notation in equations 11-13 is confusing: equation 11 is the derivative of $L_{cls}^{fg}$, equation 12 is the derivative of $L_{cls}^{fg}$ and equation 13 is the derivative of $L_{cls}^{bg}$, while on the LHS there is always $L_{cls}$\n- How easy it is to extend other existing backbones with the proposed technique? What does it take in terms of actual code changes? Can you provide a clean code example?", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The paper is very well focused on solving one specific issue\n- reasonably clearly written\n- empirical framework is solid\n- SOTA results\n- Section \"Why DC works?\" seems to provide solid quantitative evidence to support the main claim of the paper.\n\nWeaknesses:\n\n- Contribution is rather shallow and incremental, the paper almost reads as a workshop paper in some respects. It is hard to evaluate the significance of the work, because problem being solved is not well articulated (see continuation below)\n- problem being solved is not well articulated\n  - In Fig 1 it is unclear why one of the person instances is mislabeled\n  - it is unclear if the label noise is due to the dataset deficiency or it is due to the incorrect sampling scheme used to train few-shot methods. Is it possible to rectify the sampling scheme for few shot to avoid label noise? How does it happen that the objects that are not supposed to used as supervision signals end up being used as such?\n  - It is unclear why this problem is solved architecturally by introducing additional heads while it seems like it is really a problem with the data. I would much prefer if the problem could be solved in the source, while in this case it looks like the data problem remains and instead of solving it directly, a heuristic is added at the model level to account for it\n- Missing qualitative examples showing when the proposed technique succeeds in solving the stated problem and when it fails\n- Since the proposed technique is claimed to solve a high level issue, applying it to a few base methods and showing that it works seems very important. However, this paper only shows results based on Mask-DeFRCN. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept