CROSS-SUPERVISED OBJECT DETECTION
Authors: Zitian Chen
Zhiqiang Shen
Jiahui Yu
Erik Learned-Miller
Venue:  (2021)
Keywords: Object detection
weakly supervised
transfer leaning

Abstract: After learning a new object category from image-level annotations (with no object bounding boxes), humans are remarkably good at precisely localizing those objects. However, building good object localizers (i.e., detectors) currently requires expensive instance-level annotations. While some work has been done on learning detectors from weakly labeled samples (with only class labels), these detectors do poorly at localization. In this work, we show how to build better object detectors from weakly labeled images of new categories by leveraging knowledge learned from fully labeled base categories. We call this learning paradigm cross-supervised object detection. While earlier works investigated this paradigm, they did not apply it to realistic complex images (e.g., COCO), and their performance was poor. We propose a unified framework that combines a detection head trained from instance-level annotations and a recognition head learned from image-level annotations, together with a spatial correlation module that bridges the gap between detection and recognition. These contributions enable us to better detect novel objects with image-level annotations in complex multi-object scenes such as the COCO dataset.

Under review as a conference paper at ICLR 2021
CROSS-SUPERVISED OBJECT DETECTION
Anonymous authors
Paper under double-blind review
ABSTRACT
After learning new object category from image-level annotations (with no object bounding boxes), humans are remarkably good at precisely localizing those objects. However, building good object localizers (i.e., detectors) currently requires expensive instance-level annotations. While some work has been done on learning detectors from weakly labeled samples (with only class labels), these detectors do poorly at localization. In this work, we show how to build better object detectors from weakly labeled images of new categories by leveraging knowledge learned from fully labeled base categories. We call this learning paradigm cross-supervised object detection. While earlier works investigated this paradigm, they did not apply it to realistic complex images (e.g.. COCO), and their performance was poor. We propose a unified framework that combines detection head trained from instance-level annotations and recognition head learned from image-level annotations, together with a spatial correlation module that bridges the gap between detection and recognition. These contributions enable us to better detect novel objects with image-level annotations in complex multi-object scenes such as the COCO dataset.
INTRODUCTION
Deep architectures have achieved great success in many computer vision tasks including object recognition and the closely related problem of object detection. Modern detectors, such as the Faster RCNN (Ren et al. 2015). YOLO (Redmon et al.) 20166,, and RetinaNet (Lin et al. 2017). use the same network backbone as popular recognition models. However, even with the same backbone architectures, detection and recognition models require different types of supervision. good detec- tor relies heavily on precise bounding boxes and labels for each instance (we shall refer to these as instance-lever annotations), whereas a recognition model needs only image-level labels. Needless to say, it is more time consuming and expensive to obtain high quality bounding box annotations than class labels. As a result, current detectors are limited to a small set of categories relative to their object recognition counterparts. To address this limitation, it is natural ask, "Is it possible to learn detectors with only class labels?" This problem is commonly referred to as weakly supervised object detection (WSOD).
Early WSOD work (Hoffman et al. 2014) showed fair performance by directly applying recogni- tion networks object detection. More recently, researchers have used multiple instance learning methods (Dietterich et al. 1997) to recast WSOD as multi-label classification problem (Bilen & Vedaldi 2016). However, these weakly supervised detectors perform poorly at localization. Most WSOD experiments have been conducted on the ILSVRC (Russakovsky et al.) 2015) data set, in which images have only a single object. or on the PASCAL VOC (Everingham et al., 2010) data set, which has only 20 categories. The simplicity of these data sets limits the number and types of distractors in an image, making localization substantially easier. Learning from only class labels, it is challenging detect objects at different scales in an image that contains many distractors. In particular, as shown in our experiments, weakly supervised object detectors do not work well in complex multi-object scenes, such the COCO dataset (Lin et al. 2014).
To address this challenge, we focus on a form learning in which the localization of classes with only object labels (weakly labeled classes) can benefit from other classes that have ground truth bounding boxes (fully labeled classes). We refer to this interesting learning paradigm as cross- supervised object delection (CSOD). While several works (Hoffman et al. 2014 Tang al. 2016

---

Under review as a conference paper at ICLR 2021
Yang et al. 2019a Redmon & Farhadi 2017) have explored this problem before, they still have the same limitation as the WSOD work we mentioned above. Those cross-supervised object detectors work under simplified scenarios (e.g.. ILSVRC data set) where images contain single objects and are object-centered They struggle to learn under more complex and realistic scenarios, where there multiple objects from potentially very different classes, and objects could be small and appear anywhere in the images. In this work, we show that by doing multi-task learning on both weakly- supervised base classes and suply-suee novel classes, our model is able to learn good detector under the CSOD setting.
More formally, we define CSOD as follows. At training time, we are given 1) images contain objects from both base and novel classes, 2) both class labels and ground truth bounding boxes for base objects, and 3) only class labels for novel objects. Our goal to detect novel objects. In CSOD, base classes and novel classes are disjoint. Thus, it can be seen as performing fully- supervised detection on the base classes and weakly supervised detection on the novel classes. has similarities to both transfer learning and semi-supervised learning, since it transfer knowledge from base class to novel class and have more information about some instances than other instances. However, CSOD represents distinct and novel paradigm for learning.
The current As shown in Fig. weakly supervised object detector tends to detect only the most discriminating part of novel objects instead of the whole object. Notice how only the head of the person, and not the whole body, is detected. Another issue is that the localizer for one object (e.g., the horse) may be confused by the occurrence of another object, such as the person on the horse. This example illustrates the gap between detection and recognition: without ground truth bounding boxes, the detector acts like a standard recognition model - focusing on discriminating rather than detecting.
method has several drawbacks to learn from a multi objects image.
In this paper, we explore two major mechanisms for improving on this. Our first mechanism is unifying detection and recognition. Using the same network backbone architecture, recognition and detection can be seen as image-level classification and region-level classification respectively, sug- gesting strong relation between them. In particular, it suggests shared training framework in which the same backbone used with different heads for detection and recognition. Thus, we com- bine detection head learned from ground truth bounding boxes, and a recognition head learned in weakly supervised fashion from class labels. Unlike a traditional recognition head, our recognition head produces class score for multiple proposals and is capable of detecting objects. The second mechanism is learning a spatial correlation module to reduce the gap between detection and recog- nition. It takes several high-confidence bounding boxes produced by the recognition head as input, and learns to regress ground truth bounding boxes. By combining these mechanisms together, our model outperforms all previous models when all novel objects weakly labeled.
In summary, our contributions are three-fold. First, we define a new task-cross wwwss--spprrriseeeeee object detection, which enables us leverage knowledge from fully labeled base categories help learn a robust detector from novel object class labels only. Second, we propose a unified framework in which two heads are learned from class labels and detection labels respectively, along with spatial correlation module bridging the gap between recognition and detection. Third, we significantly outperform existing methods (Zhang et al.) (20T8a): Tang et al. (2017, 2018)) on PASCAL VOC and COCO, suggesting that CSOD could be promising approach for expanding object detection to much larger number of categories.
RELATED WORK
Weakly supervised object detection. WSOD Kosugi al. (2019) 9): Zeng et al. (2019))
Yang et al.
(201b): Wan et al. (2019) Arun et al. (2019): : Wan et al. (2018): Zhang et al. 2018b):
Ren et al. 2020) Zhang et al. (2018c) Li et al. (2019); Gao et al. (2019b): Kosugi et al. (019)) attempts to
learn detector with only image category labels. Most of these methods adopt the idea of Multiple
Instance Learning (Dietterich et al. (197)) to recast WSOD as a multi-label classification task. Bilen
& Vedaldi (2016) propose an end-to-end network by modifying classifier to operate at the level of
image regions, serving as a region selector and a classifier simultancously. [Tag et al. 2017) and
Tang et al. (2118) find that several iterations of online refinement based on the outputs of previous
iterations boosts performance. |Wei et al. (2018) and Diia et al. (2017) use semantic segmentation
based on class activation maps (Zhou et al. (2016)) help generate tight bounding boxes. However,
2

---

Under review as a conference paper at ICLR 2021
Weakly supervised object detector
Cross-supervised object detector
Figure 1: A comparison between weakly supervised object detector and our detector. Weakly supervised object detector only detects the most discriminating part of an object, e.g., focus on head of person when detecting person; or being distracted by co-occurring instances, e.g., distracted by the person on the horse when detecting horse. Our detector can address these issues.
WSOD methods tend to focus the most discriminating part of an object and prone distrac- tions from co-occurring objects. Detecting a part of the object or distractors represents convergence local optimum. Thus, their performance depends heavily proposed cross-supervised object detector alleviates the issue of getting trapped in a local optimum by leveraging knowledge learned from fully labeled base categories.
initialization. In comparison,
Cross-supervised object detection. There are several previous works using both image-level and instance-level annotations. Kuen al. (2019) learned a parameter transferring function between classifier and a detector, enabling an image- image-based classification network to be adapted to region- based classification network. [Hoftman al. (2014) and [Tang al. (2016) propose methods of adaptation for knowledge transfer from classification features to detection features. Uijlings et al. (2018) use a proposal generator trained on base classes to transfer knowledge by leveraging a MII. framework, organized in semantic hierarchy. HHffman et al. (2015) design threc-step framework to learn feature representation from weakly supervised classes and strongly supervised classes jointly. However, these methods can only perform object localization in single object scenes such as ILSVRC, whereas our method can perform object detection in complex multi-object scenes as well, e.g. COCO. Also, it is worth noting that are doing multi-tast learning, which means that we jointly learn from base and novel classes. In comparison, some works (Uijlings et al.] (018)) doing transfer learning. They first learn a model on base classes and then transfer and fine-tune the model on novel classes. Gao et al. (2019a) use a few instance-level labels and large scale of image-level labels for each category
training-mining framework, which is referred as semi- supervised detection. IZhang et al. (0118) 8a) propose a framework named MSD that learn objectness on base categories and use to reject distractors when learning novel objects. In comparison, our spatial correlation module not only learns objectness, but also refines coarse bounding boxes. Further, our model learns from both base and novel classes instead of only novel classes.
CROSS-SUPERVISS OBJECT DETECTION
CSOD requires to learn from instance-level annotations (detection labels) and image-level anno- tations (recognition labels). In this section, we explain the unification of detection and recognition and introduce our framework. In the next section, we describe our novel spatial correlation module.
3.1 UNIFYING DETECTION AND RECOGNITION
How to learn a detector from both instance-level and image-level annotations? Since detection and recognition can be seen as region-level and image-level classification respectively, natural choice is to design a unified framework that combines a detection head and recognition head that can learn from image-level and instance-level annotations respectively. Here we exploit several baselines to unify the detection and recognition head. (1) Finetune. We first learn through the detection head on base classes with fully labeled samples. Then, we finetune
model using the (2) Two Head. We simultaneously learn the detection and recognition head base and novel classes, respectively. The weights of the
recognition head on novel classes with only class labels.
3

---

Under review as a conference paper at ICLR 2021
Recognition Head
Propaul
Ioge
Novel
clova
Sumowar
Person
Proposals
prepou
Cony Layers
Fe Layers
Sultra
Supervision
clases
Top-scoring
Person
Sapevision
Pooling
Bost
Regrovke
Base
Detection Head
Figure 2: Our Detection-Recogniin Network (DRN) without the spatial correlation module. In this illustration, Person belongs to novel classes and Boat belongs to base classes. The recogni- tion head learns from the class label Person and outputs the top-scoring bounding box to help the detection head learn to detect the person. The spatial correlation module, discussed in $ can be added to further refine the top-scoring bounding boxes.
backbones are updated using the loss backpropagated from both heads jointly. (3) Two head + Instead of learning only novel classes, we learn the recognition head from class labels of both base and novel classes whereas the recognition head remain the same. (4) Two Branch. Instead of having two shared fully-connected layers after Rol pooling layer (see Fig. 2 make these two fully-connected layers seperated, allowing the detection and recognition head to have separate unshared pair of fully-connected layers each. Everything else is the same the Two Head baseline. Experiments are conducted to compare these baselines in $ [5. and 5.2
Our proposed model is
based on Two Head. We will discuss the details 3.2
The connection between the recognition and detection head. The baselines mentioned above only use the recognition head to detect novel objects, ignoring the fact that detection head can play the same role even better. A majority of WSOD methods (Tang et al. (2017); y): Wan et al. (2019); Wei al. (2018)) find that re-train new detector taking the top-scoring bounding boxes from weakly supervised object detector as ground truth marginally improve the performance. Even with coarse and noisy pseudo bounding boxes, a standard object detector produces better detection results than a weakly supervised object detector. Keeping this hypothesis in mind, we introduce a guidance from the recognition head to the detection head. For each of the novel categories existing a training sample, the recognition head outputs the top-scoring bounding box, which are then used by the detection head as supervision in that sample.
3.2 DETECTIIN-RECCONNTION RECOGNITION NETWORK
The structure of our Detection- Recitiinionoon Network (DRN) is shown in Fig. Given an image, we first generate 2000 object proposals by Selective Search (Uijlings al. (013
) RPN (Ren et al.
(2013)) trained on base classes. The image and proposals are led into several convolutional (conv) layers followed by region-of-interes. (Rol) pooling layer (Girshick (015 to output fixed-size feature maps. Then, these feature maps are fed into two fully connected (fc) layers to produce a collection of proposal features, which are further branched into the recognition and detection head.
Recognition Head. We followed previous WSOD methods to design our recognition head, Since OICR (Tang al. (2077) simple, neat, and commonly being used, make our recognition head the same as OICR, but with fewer refinement branches to reduce the computation cost. However, our recognition head can be replaced by any WSOD structure as shown in 5.3
Within the recognition head as shown in Fig. 2 the proposal features are branched into three streams where C is the number of novel classes and R is the number of proposals. Then the two matrices x" and x passed through softmax function
producing three matrices x".x". x E C T
4

---

Under review as a conference paper at ICLR 2021
Nigh-confidence bbox
Class-agnostic heatmap
Final pseudo bbox
Heatmap
detection
Person
Figure 3: Our spatial correlation module (SCM). Our SCM learns to capture spatial correlation among high-confidence bounding boxes, generating class- class-agnostic heatmap for the whole image. A heatmap detector is then trained to learn ground truth bounding boxes.
over classes and proposals respectively: oxx)) and (xx). A proposal score ft indicating the score of th novel class for th proposal, corresponds the respective element of the matrix x" a (x (xx), where @ refers to an element-wise product. Finally, we obtain the image score of cd class de by summing over proposals: de
R zerr Then we calculate a standard multi-class cross-entropy loss as shown in the first term of Eg Another matrix x" is passed through softmax function over classes, the result of which is expresses
weighted multi-class cross entropy loss as shown in the second term of [q We set the pseudo label for each proposal based on its IoU (or overlap) with the top-scoring proposal of th class, yer if IoU > 0.5 and yer otherwise. The weight w,- for each proposal is its IoU with the top-scoring proposal. The total loss for the recognition head is
C Σ
C+1 M
Lree 1-
Velogge + -2--04 yo))oo Φc)1 +
1
Σ
w Woyer
R
Supervision from our recognition head. We use the matrix x" to propose pseudos bounding boxes to guide the detection head. Specifically, we select one top-scoring proposal for each object category that appears the image as pseudo bounding box, as done in OICR. We introduce the spatial correlation module in & 4 to further refine this pseudo ground truth.
Detection Head. Now that we have pseudo bounding boxes for novel objects and ground truth bounding boxes for base objects, we train our detection head like standard detector. For simplicity and efficiency, our detection head use the same structure of Faster R-CNN (Ren et al. (2015)). At inference time, the detection head produces detection results for both base categories and novel categories.
LEARNING TO MODEL SPATIAL CORRELATION
Our intuition is that there exists spatial correlation among high-confidence bounding boxes, and such spatial correlation can be captured to predict ground truth bounding boxes. By representing the spatial correlation in a class-agnostic heatmap, we can easily learn a mapping from recognition- based bounding boxes to ground truth bounding boxes for base categories, and then transfer this mapping to novel categories.
Thus, we propose spatial correlation module (SCM). SCM is used as guidance refinement technique, taking sets of high-confidence bounding boxes from the recognition head, and corre- spondingly returning pseudo ground truth bounding boxes the detection head. These pseudo ground truth boxes act as supervision while training on novel categories. The framework of SCM showed in Fig. 3 Within this module, we first generate a class agnostic heatmap based on the high-con cogh-confidence bounding boxes predicted by recognition head, and then perform detection on top of the heatmap.
Heatmap synthesis. We want to capture the information about how the high-confidence bounding boxes interact amongst themselves. Here, we introduce a simple way of achieving this using a class-agnostic heatmap. For each category existing in the image ye 1,c C. we first threshold and select high-confidence bounding boxes of class c. Then we synthesize a corresponding class- agnostic heatmap, which is essentially two-channel feature map of the same size as the original
5

---

Under review as a conference paper at ICLR 2021
Base
Novel
Method
mean
table
dog
horse
mbike
person
plant
sheep
sofa
train
tv
mean
OICR
42.1
33.4
29.3
56.3
64.6
8.0
23.5
47.2
47.2
48.3
61.7
42.0
PCL
49.2
51.5
37.3
63.3
63.9
15.8
23.6
48.8
55.3
61.2
62.1
48.3
MSD-VGG16
50.6
14.3
69.3
65.4
69.6
2.4
20.5
54.6
34.3
58.3
54.6
44.3
MSD-Ens
53.4
18.3
70.6
66.7
69.8
3.7
24.7
55.0
37.4
58.3
57.3
46.1
MSDDEns+FRCN
53.9
15.3
72.0
74.4
15.4
25.1
53.6
54.4
45.6
61.4
48.2
Weight Transfer
68.4
10.4
61.0
58.0
65.1
19.8
19.5
58.0
50.8
58.6
52.7
45.4
Finetune
71.8
17.8
22.9
15.2
71.2
10.2
15.1
61.7
36.6
21.9
61.3
33.4
Two Head"
72.9
60.6
33.2
47.7
70.2
3.9
25.5
52.6
58.4
54.7
64.4
47.1
Two Head
72.4
44.5
29.5
52.4
68.4
22.6
53.0
55.5
58.6
64.8
45.4
Two Branch"
72.7
57.3
30.2
44.2
68.1
3.0
21.4
52.2
53.5
51.2
59.7
44.1
Ours w/ SCM
71.6
62.3
41.9
38.2
73.0
11.3
26.0
60.6
63.8
70.5
65.3
51.3
Ours
72.9
61.0
57.1
63.5
72.0
19.5
24.2
60.9
58.6
68.5
65.5
55.1
Ours" w/o SCM
72.7
66.8
50.4
57.0
71.5
12.1
27.6
57.1
62.7
54.2
64.2
52.4
Ours"
72.7
60.9
59.4
70.5
71.0
17.5
24.1
62.0
60.5
62.4
69.1
55.7+3.3
Table 1: Object Detection performance (mAP %) on PASCAL VOC 2007 test set. 'indicates using the structure of OICR in the recognition head. "MSD-Ens" is the ensemble of AlexNet and VGG16. "MSD-Ens+FRCNN" indicates using an ensemble model to predict pseudo ground truths and then learn Fast-RCNN (Girshick (2015) using VGG-16.
image. The value each pixel is the sum and the maximum confidence over all selected bounding boxes covering that pixel.
Heatmap detection. We consider each class-agnostic heatmap as two-channel image, and perform detection on it. Specifically, we learn class-agnostic detector on base classes, that we further use to produce pseudo ground truth bounding boxes for novel objects.
For this task, we use lightweight one-stage detector, consisting of only five convolutional lay- ers. We follow the same network architecture and loss as FCOS (Tian et al. (2019), replacing the backbone and feature pyramid network with five max pooling layers. In our experiments, also compare this tiny detector to a baseline: using three fully- fully-connected layers to regress the ground- truth location taking the coordinates of high-confidence bounding boxes input.
Loss of DRN. After introducing our SCM, we can formulate the full loss function for DRN. We use Lrec. Ldet, and Leem to indicate the losses from our recognition head, detection head, and spatial correlation module respectively. Arecs Adet. and Asem are the regularization hyperparameters used to balance the three separate loss functions. We train our DRN using the following loss:
L = Aveehrec + Adet + Aacm
5 EXPERIMENTS
5.1 PASCAL VOC
Setup. PASCAL VOC 2007 and 2012 datasets contain 962 and 22, 531 images respectively for
20 object classes. They are divided into train, val, and test sets. Here we follow previous work
(Tang et al. (2017) choose the trainval set (5, 011 images from 2007 and 11, 540 images from 2012). We divide the first 10 classes into base classes and the other 10 classes into novel classes. To evaluate our methods, we calculate mean of Average Precision (mAP) based on the PASCAL. criteria, i.e., 1OU>005 between predicted boxes and ground truths.
Implementation details. All our baselines, competitors and our framework are based on VGG16 Simonyynn & Zisserman (2015) followed most of weakly supervised object detection methods. We set Arec = 1, Adet = 10, and Asem = 10. We train the whole framework for 20 epochs using SGD with momentum of 0.9, weight decay of 0.0005 and learning rate of 0.001, which is reduced by a factor of 10 at th epoch. For a stable learning process, we don't 't provide supervision from recognition head to detection head in the first 9 epochs.

---

selective search and RPN Under review as a conference paper at ICLR 2021
B
sixty
twenty:
test
{twenty)
method
AP
AP
AP
APs
A
APL
AP
APso
AP75
APs
APM
APr
Rec. Head
4.0
13.4
0.9
1.2
5.7
5.8
4.7
16.4
1.3
1.7
8.0
6.9
OICR
4.2
15.7
1.0
1.3
5.5
5,9
4.5
16.6
1.4
2.0
8.2
7.1
PCL
9.2
19.6
9.2
19.6
Weight
9.3
26.4
5.7
5.8
11.7
12.4
8.7
25.5
5.5
5.4
11.5
11.7
Finetune
2.3
7.4
0.3
0.7
3.1
3.3
2.4
7.7
0.2
0.5
2.8
3.0
Head
11.0
30.2
15.4
15.4
11.3
29.5
5.8
14.8
15.0
Two Head*
9.1
26.7
5.4
5.5
12.1
12.3
9.0
27.1
5.4
5.7
11.7
11.6
Two Branch
9.4
26.6
5.6
5.7
12.3
12.4
8.5
24.4
4.5
4.3
11.9
11.9
Ours w/o SCM
12.5
33.6
6.6
7.3
19.2
16.4
12.6
32.3
7.8
7.0
19.4
17.4
Ours
13.9
36.2
7.7
6,9
18.8
19.9
14.0
34.5
8.9
7.1
19.2
20.6
The results on COCO. We compare our method with several strong baselines and competitors. Our method significantly outperforms these approaches, showing that cross- supervised object detector is capable of detecting novel objects complex multi-object scenes.
Table 2:
3.1
Baselines and competitors. We compare against several baselines as mentioned in $ 3.1 two WSOD methods: OICR (Tang al. (2017) and PCL (Tang et al. (2018)), and two cross-supervised object detector: MSD (Zhagg et al. 22188)), weight transfer (Kuen et al. (2019)).
Results. As shown in Table D our method outperforms all other approaches by a large margin (over 7% relative increase in mAP on novel classes). The results are consistent with our discussion 3.1 We note that (1) sharing backbone for the recognition and detection head learns a more discriminative embedding for novel objects. In Table [ Two Head" boosts the performance by points as compared to only using the recognition head (OICR). (2) A supervision from recognition head to detection head exploits the full potential of detection model. By adding the supervision (Ours* w/o SCM) ), the result is improved by points as compared to Two Head. (3) Our spatial correlation module successfully captures the spatial correlation between high-confidence proposals. further boosts the performance by points.
method non-voc-+We
sixty sixty->twenty
non-voc->vOL
sixty tweety
APso on B
AP50 B
method
AP50 on B
APo0 on B
35.5
33.8
layer
31.0
28.7
sum
36.0
34.0
Fc layer
layer
30.8
28.3
num
31.5
29.5
layer
30.5
28.5
max+
36.2
34.5
R-50-FPN
36.4
34.8
35.7
34.1
FCOS
conv
35.8
33.8
sum+num
35.9
34.2
conv
36.2
34.5
max+sum+num 36.1
34.2
w/o SCM
33.6
32.3
(a) Ablation Heatmap synthesis. The re- sult suggests using two-channel heatmap consists of maximum confidence and sum of confidence over proposals covering that position.
(b) Ablation on the structure of SCM. FCOS with conv layers has nearly the best performance and very few parameters compared to ResNet-50 back- bone.
non-voc- +voc sixty->twent
base nooel
method
APso on
APso
dataset
method
APs0 on A APso on
WSDDN
35.7
33.8
RPN
76.2
46.1
PASCAL VOC
OICR
36.6
34.7
SS
72.7
55.7
Ours
36.4
34.5
RPN
46.3
36.2
non-voc- VVOC
SS
42.5
34.5
(c) Ablation on the structure of the recognition head. OICR has more refinement branches so it be- haves little better than recognition head but takes double the computation time.
(d) Ablation on the proposal generator. On PAS-
CAL VOC, there are not enough categories to learn good RPN. So, to generate proposals for PASCAL VOC and COCO respectively.
Table 3: Ablation study of our method.

---

Under review as a conference paper at ICLR 2021
Recognition
Head
SCM
Detection Head
Figure 4: Detection results on novel objects. The results are from our proposed model but with different heads. The first row shows the results of the recognition head. The second row lists the results from SCM. The third row displays the results from the detection head.
5.2 COCO
Setup. We train on the COCO train2017 split and test on val2017 split. We simulate the cross- supervised object detection scenario COCO by splitting the 80 classes into base and novel classes. We use a 20/60 split same as Hu et al. 2018). dividing the COCO categories into all the 20 classes contained in PASCAL VOC and the 60 that not. We refer to these as the 'voc' and 'non-voc' category sets. 'vo--vnon-voc' indicates that we take 'voc" as our base classes and 'non-voc' as our novel classes. Similarly, we split the first 20 classes into 'twenty' and the last 60 classes into 'sixty'.
Implementation details. The implementation details are the same as 5.. default. We train the whole framework for 13 epochs. There no supervision from recognition head detection head in the first 5 epochs. The learning rate is reduced by factor of 10 at 8th, and 12th epochs.
Baselines and competitors. Most baselines and competitors are the same represents only using our recognition head structure weakly supervised object detector.
5.1 'Rec. Head'
Results. The results on COCO still support our discussion $ 5.1 Even complex multi objects scenes, our DRN outperforms all baselines and competitors by large margin.
5.3 ABLATION EXPERIMENTS
Heatmap synthesis. Table 3al we compare the different methods synthesize the heatmaps in the spatial correlation module. For each position in the heatmap, we consider three kinds of values: the ma ximum of confidence, the sum of confidence, and the number of proposals covering the position. This result informs us use max and sum to create two-channel heatmap.
Structure of SCM. Table 3b) we compare different implementations SCM. compare FCOS (Tian et al. (2019) with convolutional layers and the standard FCOS with ResNet-50 (He ct al. (2016)) backbone. We also compare to the regression bascline mentioned in Considering the computation cost, choose FCOS with 5 convolutional layer our heatmap detector.
Structure of the Recognition head. Table 3d we compare different structures for the recognition head. WSDDN (Bilen & Vedaldi 22166) and OICR compared to structure. The results support that our model can benefit from a stronger recognition head.
Different proposal generation methods. Table 3d shows the ablation of different ways generate proposals. In PASCAL VOC with only 10 base classes, RPN performs worse than selective search. In COCO with 60 base classes, RPN performs better than selective search.
8

---

Visualization. Fig. shows detection results novel objects. Images in the first row, the second row, and the third row are detected by our model from the recognition head, the SCM, and the detection head respectively. The images in the first tend to focus on the discriminating parts of the objects, e.g. the first and the second images contain only a part of the person. It also tends to detect co-occurring objects, e.g. the fourth image not only detects horse but also large part of the person. Our SCM alleviates these problems, It tends to focus on the whole object, c.g. the first and the third samples detect the whole person instead of only the head. Also, it can correct unsatisfactory bounding boxes distracted by co-occurring objects, e.g. SCM correctly localizes the horse instead Under review as a conference paper at ICLR 2021
of localizing both the person and the horse in the fourth example. Obviously, bounding boxes in the third row are the best, indicating the efficacy of our framework.
CONCLUSION
In this paper, have focused on cross-supervised object detection in realistic settings with complex imagery. We explore two major ways to build a good cross-supervised object detector: sharing net- work backbone between recognition head and detection head, and learning spatial correlation module to bridge the gap between recognition and detection. Significant improvement on PASCAL VOC and COCO suggests a novel and promising approach for expanding object detection to a much larger number of categories.
REFERENCES
Aditya Arun, C.V. Jawahar, and M. Pawan Kumar. Dissimilarity coefficient based weakly super- vised object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Hakan Bilen and Andrea Vedaldi. Weakly supervised deep detection networks. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition (CVPR), 2016.
Ali Diba, Vivek Sharma, Ali Pazandeh, Hamed Pirsiavash, and Luc Van Gool. Weakly supervised cascaded convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Thomas G Dietterich, Richard H Lathrop, and Tomás Lozano-Pérez. Solving the multiple instance problem with axis axis-parallel rectangles. Artificial intelligence, 1997.
Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision (IJCV), 2010.
Jiyang Gao, Jiang Wang, Shengyang Dai, Li-Jia Li, and Ram Nevatia. Note-renn: Noise tolerant ensemble renn for semi-supervised object detection. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2019a.
Yan Gao, Boxiao Liu, Nan Guo, Xiaochun Ye, Fang Wan, Haihang You, and Dongrui Fan. C- midn: Coupled multiple instance detection network with segmentation guidance for weakly pervised object detection. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2019b.
Ross Girshick. Fast r-cnn. In Proceedings the IEEE International Conference on Computer Vision (ICCV), 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Judy Hoffman, Sergio Guadarrama, Eric S Tzeng, Ronghang Hu, Jeff Donahue, Ross Girshick, Trevor Darrell, and Kate Saenko. Lsda: Large scale detection through adaptation. In Advances in Neural Information Processing Systems (NeurIPS), 2014.

---

Under review as a conference paper at ICLR 2021
Judy Hoffman, Deepak Pathak, Trevor Darrell, and Kate Saenko. Detector discovery in the wild: Joint multiple instance and representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
Ronghang Hu, Piotr Dollár, Kaiming He, Trevor Darrell, and Ross Girshick. Learning segment every thing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Satoshi Kosugi, Toshihiko Yamasaki, and Kiyoharu Aizawa. Object-aware instance labeling for weakly supervised object detection. Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2019.
Jason Kuen, Federico Perazzi, Zhe Lin, Jianming Zhang, and Yap-Peng Tan. Scaling object detection by transferring classification weights. Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2019.
Xiaoyan Li, Meina Kan, Shiguang Shan, and Xilin Chen. Weakly supervised object detection with segmentation collaboration. Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2019.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Proceedings of the European Conference on Compuler Vision (ECCV), 2014.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017.
Joseph Redmon and Ali Farhadi. Yolo9000: better, faster, stronger. In Proceedings of the IEEE International Conference on Computer Vision (CVPR), 2017.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE Conference on Computer Vision and Pat- tern Recognition (CVPR), 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems (NeurIPS), 2015.
Zhongzheng Ren, Zhiding Yu, Xiaodong Yang. Ming- Yu Liu, Yong Jae Lee, Alexander G Schwing, and Jan Kautz. Instance-aware, context-focused, and memory-efficient weakly supervised object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 2015.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Proceedings of the International Conference on Learning Representations (ICLR), 2015.
Peng Tang, Xinggang Wang, Xiang Bai, and Wenyu Liu. Multiple instance detection network with online instance classifier refinement, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Peng Tang, Xinggang Wang. Song Bai, Wei Shen, Xiang Bai, Wenyu Liu, and Alan Yuille. Pel: Proposal cluster learning for weakly supervised object detection. IEEE transactions on pattern analysis and machine intelligence, 2018.
10

---

Under review as a conference paper at ICLR 2021
Yuxing Tang, Josiah Wang, Boyang Gao, Emmanuel Dellandréa, Robert Gaizauskas, and Liming Chen. Large scale semi superviie object detection using visual and semantic knowledge transfer. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2016.
Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Feos: Fully convolutional one-stage object detection. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2019.
Jasper Uijlings, Stefan Popov, and Vittorio Ferrari. Revisiting knowledge transfer for training ob- ject class detectors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Jasper RR Uijlings, Koen EA Van De Sande, Theo Gevers, and Amold WM Smeulders. Selective search for object recognition. International journal of computer vision (IJCV), 2013.
Fang Wan, Pengxu Wei, Jianbin Jiao, Zhenjun Han, and Qixiang Ye. Min-entropy latent model for weakly supervised object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Fang Wan, Chang Liu, Wei Ke, Xiangyang Ji, Jianbin Jiao, and Qixiang Yc. C-mil: Continuation multiple instance learning for weakly supervised object detection. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2019.
Yunchao Wei, Zhiqiang Shen, Bowen Cheng, Honghui Shi, Jinjun Xiong, Jiashi Feng, and Thomas Huang. Ts2c: Tight box mining with surrounding segmentation context for weakly supervised object detection. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
Hao Yang, Hao Wu, and Hao Chen. Detecting 11k classes: Large scale object detection without fine-grained bounding boxes. In Proceedings of the IEEE International Conference on Computer Vision (CVPR), 2019a.
Ke Yang. Dongsheng Li, and Yong Dou. Towards precise end-to-end weakly supervised object detection network. In Proceedings of the IEEE International Conference Computer Vision (ICCV), 2019b.
Zhaoyang Zeng, Bei Liu, Jianlong Fu, Hongyang Chao, and Lei Zhang. Wsod2: Learning bottom- up and top-down objectness distillation for weakly-supervised object detection. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2019.
Junge Zhang, Kaiqi Huang, Jianguo Zhang, et al. Mixed supervised object detection with robust objectness transfer. IEEE transactions on pattern analysis and machine intelligence, 2018a.
Xiaopeng Zhang, Jiashi Feng, Hongkai Xiong, and Qi Tian. Zigzag learning for weakly super- vised object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018b.
Yongqiang Zhang, Yancheng Bai, Mingli Ding, Yongqiang Li, and Bernard Ghanem. W2f: A fully-supervised framework for object detection. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2018c.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralb. Learning deep features for discriminative localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
11


Reviews:
[{"review_id": "y3sAjjDew_3", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper summary:\nThe paper proposes a new task cross-supervised object detection, which trains object detectors on the combination of base class images with instance-level annotations and novel class image with only image-level annotations. A network with a recognition head which is trained by image-level annotations and a detection head which is trained by instance-level annotations is proposed for the task. To generate instance-level annotations for novel class images with only image-level annotations, the paper proposes a spatial correlation module to generate pseudo gt boxes from high-confidence boxes. Results on PASCAL VOC and COCO show that the proposed method obtains very promising object detection results for novel classes.\n\n\nStrengths:\n\n+ The paper is well written.\n\n+ Promising experimental results are obtained by the proposed method.\n\n+ The proposed spatial correlation module is interesting.\n\n\nWeaknesses:\n\n- Duplicate task settings.\nThe proposed new task, cross-supervised object detection, is almost the same as the task defined in (Hoffman et al. 2014, Tang et al. 2016, Uijlings et al. 2018). Both of these previous works study the task of training object detectors on the combination of base class images with instance-level annotations and novel class image with only image-level annotations. The work (Uijlings et al. 2018) also conducts experiments on COCO which contains multi-objects in images.\nIn addition, the work  (Khandelwal et al. 2020) unifies the setting of training object detectors on the combination of fully-labeled data and weakly-labeled data, and conducts experiments on multi-object datasets PASCAL VOC and COCO. The task proposed by this paper could be treated as a special case of the task studied in (Khandelwal et al. 2020).\nWe should avoid duplicate task settings.\n\n- Limited novelty.\nThe novelty of the proposed method is limited.\nCombining recognition head and detection head is not new in weakly supervised object detection. The weakly supervised object detection networks (Yang et al. 2019, Zeng et al. 2019) also generate pseudo instance-level annotations from recognition head to train detection head (i.e., head with bounding box classification and regression) for weakly-labeled data.\n\n\nReview summary:\nIn summary, I would like to give a rejection to this paper due to the duplicate task settings and limited novelty.\n\nKhandelwal et al., Weakly-supervised Any-shot Object Detection, 2020\n\n---------- Post rebuttal ----------\n\nAfter discussions with authors and reading other reviews, I acknowledge the contribution that this paper advances the performance of cross-supervised object detection.\n\nHowever, I would like to keep my original reject score. The reasons are as follows.\n\nExtending datasets from PASCAL VOC to COCO is not a significant change comparing to previous tasks. The general object detection papers also evaluated on PASCAL VOC only about five years ago and now evaluate mainly on COCO. With the development of computer vision techniques, it is natural to try more challenging datasets. So although this paper claims that this paper focuses on more challenging datasets, there is no significant difference between the tasks studied in previous works like [a] and this paper.\n\nIn addition, apart from ImageNet, the work [b] also evaluates their method on the Open Images dataset which is even larger and more challenging than COCO. The difference between the tasks studied in [b] and this paper is only that, [b] adds a constraint that weakly-labeled classes have semantic correlations with fully-labeled classes and this paper doesn't. This difference is also minor.\n\nTherefore, the task itself cannot be one of the main contributions of this paper (especially the most important contribution of this paper). I would like to suggest the authors change their title / introduction / main paper by 1) giving lower wights to the task parts 2) giving higher weights to intuitions of why previous works fail on challenging datasets like COCO and motivations of the proposed method.\n\n[a] YOLO9000: Better, Faster, Stronger, In CVPR, 2017\n\n[b] Detecting 11K Classes: Large Scale Object Detection without Fine-Grained Bounding Boxes, In ICCV, 2019", "extra_scores": {}, "extra_text": {}}, {"review_id": "sfdV4h2xpzl", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper defines cross-supervised object detection which learns a detector from both image-level and instance-level annotations. It proposes a unified framework along with a spatial correlation module for the task. The spatial correlation module is used for transfer mapping information from base categories to novel categories. It conducts experiments on the PASCAL VOC dataset and COCO dataset, demonstrating the effectiveness.\n\nPros:\n(1) The proposed spatial correlation module is a novel and effective transfer module.\n(2) The ablation studies are relatively complete. \n\nCons:\n(1) The structure of the proposed spatial correlation module should be described in more detail. What is the meaning of “replacing the backbone and feature pyramid network with five max-pooling layers.” in the heatmap detection part?\n(2) In Table 1, are the experimental settings of those competitors such as MSD-VGG16, MSD-Ens, and Weight Transfer et al. exactly the same as those used in this paper?\n(3) In Table 2, it seems like the method taking the non-VOC as the base classes while Hu et al. (2018) use the non-VOC as those classes without mask annotation. Can you tell me the reasons for this choice?\n(4) Some similar problem settings are defined in [a] and [b]. The paper fails to compare the problem settings and justify the usefulness of the proposed problem setting in real application scenarios.\n\nOverall evaluation: The major contribution of this paper comes from the spatial correlation module. However, I still have some doubts about the structure of this module. Since this task is first presented, I want to make sure that the comparison is as fair as possible. \n\n[a] Weakly- and Semi-Supervised Fast Region-Based CNN for Object Detection. Journal of Computer Science and Technology (JCST) 34(6): 1269–1278 Nov. 2019.\n[b] LSTD: A Low-Shot Transfer Detector for Object Detection, AAAI 2018", "extra_scores": {}, "extra_text": {}}, {"review_id": "joZJrrjMxV2", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary and contributions**:\nThe paper presents a new task formulation for transferring knowledge for object detection from fully labeled classes to weakly labeled ones, with better results for complex scenes. So the authors propose to learn object detectors for novel classes (which were not seen at training time but specified apriori), based only on that object class label and the bounding boxes for other objects in the image (from the base classes). Compared with other works, the solution claims to be better on the localization aspect, focusing on the object as a whole, not only on its discriminative parts. Nevertheless, there are very few competitors taken into account. The model combines three components: a detection head and a recognition head, based on the same, unified backbone architecture, and a spatial correlation module that aligns the two heads. The authors test their solution, Cross-Supervised Object Detection, on PASCAL VOC and COCO (multi-object scenes).\n\n**Strengths**:\n- Proposing a model that succeeds in learning both novel and base classes at the same time, on two heads, extracting information from the weak labels (recognition head) and using it as supervision for the second head (detection head).\n- The qualitative results look significantly better compared with single heads, containing more of the entire object, not only the important/discriminatory parts as the authors pointed out.\n- FCOS with only 5 conv layers reaches a good enough performance.\n\n\n**Weaknesses**:\n* How does this work compare with other WSOD recent methods, e.g. C-MIL, [A, B, C] from the experimental results point of view, and the amount of supervision?\n* The text/figure should be adjusted to better map one each other: Sec. 3.2: “The image and proposals are fed into several convolutional layers”, but in the figure, the proposals skip those layers.\n* Section 3.2 is rather hard to read and understand (it has several inaccuracies in the formulas).\n\n**Quality**:\nThe paper is technically sound.\n\n**Clarity**: \nThe paper is mainly clearly written (except for Section 3.2 Recognition Head).\n\n**Novelty**:\nThe components are not novel, but the proposed learning paradigm is novel.\n\n**Significance of this work**:\nThe impact of the work is significant, enabling a new way (CSOD) of transferring knowledge between classes, using weak labels.\n\n**Typos**:\n- “corresponds to the respective element of the matrix...” the following formula is wrong (d should be replaced with r) \n- what does the upper index R mean and why is it useful?\n- matrices should be capital letters in bold\n- “which is expresses”\n\n\nWhile I have some doubts regarding the comparison with other works, I hope they will be clarified during the rebuttal.\n\n\n[A] High-Quality Proposals for Weakly Supervised Object Detection, Cheng, G., Yang, J., Gao, D., Guo, L., & Han, J..Transactions on Image Processing 2020 \n\n[B] Instance-aware, Context-focused, and Memory-efficient Weakly Supervised Object Detection. Zhongzheng Ren, Zhiding Yu, Xiaodong Yang, Ming-Yu Liu, Yong Jae Lee, Alexander G. Schwing, Jan Kautz. CVPR 2020\n\n[C] Mixed Supervised Object Detection with Robust Objectness Transfer, Yan Li, Junge Zhang, Kaiqi Huang, Jianguo Zhang. PAMI 2019", "extra_scores": {}, "extra_text": {}}, {"review_id": "6aIxHHNHztI", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a new method for training an object detector on a dataset that consists of some object categories with instance-level bounding box annotations, as well as some other object categories with only image-level labels. The topic is interesting, important, and potentially very useful for real applications. The authors propose an idea to transfer knowledge from a weakly supervised (WS) detection head into a fully supervised (FS) detection head, by producing pseudo-ground-truth bounding boxes for classes with image-level labels. The idea is straightforward and interesting. Experiments show significant and consistent gains in various scenarios. The paper is well-written.\n\nThe main drawback of this paper is the lack of substantial novelty. The authors claim to propose a \"novel paradigm\" named cross-supervised object detection, while this task has already been studied as reviewed in Section 2. Particularly, the idea of jointly training WS and FS heads on WS and FS labels with a shared backbone has been explored before (e.g. in [1], which should have been discussed in the related work). The idea of using a WS model to create pseudo-ground-truth bounding boxes for training another model has also been studied before (e.g. in [2]). The only novel idea of this paper seems to be the Spatial Correlation Module (SCM), which is discussed more below.\n\nSCM is used to transform proposal boxes selected by the WS detector head into more accurate bounding boxes to serve as pseudo-ground-truth for WS classes. To this end, the authors train a class-agnostic bounding box refinement module on FS classes, and apply it on the proposals of WS classes. However, a similar result could have been (probably) achieved by simply replacing the bounding box regression head of the Faster R-CNN with a class-agnostic head, and training it on base classes, while using it to refine the proposals of both base and novel classes during test. The authors did not explore such simple alternatives to SCM.\n\nAnother issue is that although the authors cited several existing methods for cross-supervised object detection in Section 2, they did not discuss why the proposed method is superior, and they did not include them in performance tables, claiming \"these methods can only perform object localization in single object scenes.\" I cannot verify the correctness of this claim, as any proposal-based model can detect multiple objects per image and per class. \n\nNevertheless, the paper has a clear motivation and idea, a scientifically sound analysis, and significant results and insights that can be helpful for future work. Therefore, I recommend acceptance. If the authors can convince me that the paper also has substantial novelty and advantages over all existing works, I am willing to raise my rating.\n\nMinor comment: In equation (1), the two terms of the binary cross-entropy should probably be placed in parentheses, so the sum is applied on both terms. Also in the paragraph above eq (1), that loss term should be defined as a multi-label cross-entropy, rather than multi-class.\n\n[1] Yang, Hao, Hao Wu, and Hao Chen. \"Detecting 11k classes: Large scale object detection without fine-grained bounding boxes.\" Proceedings of the IEEE International Conference on Computer Vision. 2019.\n\n[2] Tang, Peng, et al. \"Multiple instance detection network with online instance classifier refinement.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017.\n\n######## Post-Rebuttal Updates:\n\nAfter reading the authors' response and other reviewers' opinions (especially R3), I would like to downgrade my rating slightly from 7 to 6. I still think the paper makes valuable contributions, but I also think the contributions are overstated and not precisely justified. Particularly:\n\n1. I agree with R3 that the limitation of novelty should be considered from the two perspectives of \"task\" and \"method\". The task is certainly not new, which should be made clear in the paper. The newest version of the paper still claims \"we define a new task—cross-supervised object detection\"\n\n2. The method is indeed somewhat new, due to the use of multi-task learning and SCM, but its novelty should be clarified, and compared to all similar methods, not only some of them which are weaker. For instance, the authors did not adequately justify whether/why their method is superior to YOLO 9000, or Yang et al ([1] above). They did mention some differences in response to R3's comments and mine, but I am not convinced. Moreover, the authors did not explicitly discuss and compare those distinctions in the paper, neither quantitatively nor intuitively.\n\n3. In response to R3, The authors claim they \"are the first to address this problem in situations of realistic complexity\", which is not accurate. Particularly, the paper reads \"While several works [...] have explored this problem before, [...] They struggle to learn under more complex and realistic scenarios, where there are multiple objects from potentially very different classes\" This is not entirely true, as Yang et al. successfully evaluate on Open Images (and also VOC and COCO in their supplementary materials). The authors do not provide a convincing reason or evidence of existing methods \"struggling\" in realistic settings. YOLO 9000 is open-source, and could have been compared to the proposed method to confirm that claim.\n\nAccordingly, I strongly encourage the authors to refine their claims and lay more emphasis on the actually novel aspects of their method, by thoroughly comparing those novelties with all similar methods. I still hope to see this paper accepted, but cannot endorse it due to insisting on inaccurate claims.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: After the rebuttal phase, all scores are borderline (6) or negative (4). Among the most confident reviewers (confidence 5), one gives 6 and one gives 4. The reviewer with confidence 4 gives overall score 6 but states they cannot support the paper. There were several concerns about the novelty of the task and method, the challenge of the experimental settings, missing comparisons to recent prior work in the original paper, etc. While the reviewers see merit, the paper can benefit from another revision before being accepted, including to better position the novelty of its method and perhaps reduce claims of novelty of the task. 
Decision: Reject