Mixed Supervised Object Detection by Transferring Mask Prior and Semantic Similarity
Authors: Yan Liu
Zhijie Zhang
Li Niu
Junjie Chen
Liqing Zhang
Venue: NeurIPS 2021 Poster (2021)
Keywords: object detection
weak-shot learning
transfer learning

Abstract: Object detection has achieved promising success, but requires large-scale fully-annotated data, which is time-consuming and labor-extensive. Therefore, we consider object detection with mixed supervision, which learns novel object categories using weak annotations with the help of full annotations of existing base object categories. Previous works using mixed supervision mainly learn the class-agnostic objectness from fully-annotated categories, which can be transferred to upgrade the weak annotations to pseudo full annotations for novel categories. In this paper, we further transfer mask prior and semantic similarity to bridge the gap between novel categories and base categories. Specifically, the ability of using mask prior to help detect objects is learned from base categories and transferred to novel categories. Moreover, the semantic similarity between objects learned from base categories is transferred to denoise the pseudo full annotations for novel categories. Experimental results on three benchmark datasets demonstrate the effectiveness of our method over existing methods. Codes are available at https://github.com/bcmi/TraMaS-Weak-Shot-Object-Detection.


Mixed Supervised Object Detection by Transferring Mask Prior and Semantic Similarity
Yan Liu," Zhijie Zhang, Li Niu, Junjie Chen, Liqing Zhang
MoE Key Lab of Artificial Intelligence
Department of Computer Science and Engineering
Shanghai Jiao Tong University (loseover, zzj506506, ustcnewly, chen. chen.bys) bys js edu. cn zhang- -lgcs sjtu, edu. cn
Abstract
Object detection has achieved promising success, but requires large-scale fully- annotated data, which is time-consuming and labor-extensive. Therefore, we consider object detection with mixed supervision, which learns novel object cat- egories using weak annotations with the help of full annotations of existing base object categories. Previous works using mixed supervision mainly learn the class- agnostic objectness from fully-annotated categories, which can be transferred to upgrade the weak annotations to pseudo full annotations for novel categories. In this paper, we further transfer mask prior and semantic similarity to bridge the gap between novel categories and base categories. Specifically, the ability of using mask prior to help detect objects is learned from base categories and transferred novel categories. Moreover, the semantic similarity between objects learned from base categories is transferred to denoise the pseudo full annotations for novel categories. Experimental results three benchmark datasets demonstrate the effectiveness of our method over existing methods. Codes are available at http://ggguubb.oom//mmm//////iee
Seeak-Shot-Objee-DDeee
Introduction
With the ubiquitous application of convolutional neural networks and the release of large-scale fully-annotated detection benchmark datasets (e.g., MS COCO (27). ILSVRC Detection (32). and PASCAL VOC (10),, object detection has achieved significant advances in recent years [14]
However, Fully Supervised Object Detection (FSOD) requires massive training images with precise bounding boxes and box-level category labels. Acquiring such full annotations is time-consuming and intensint which limits the application of FSOD in real-world scenarios. In contrast, weak annotations like image-level category labels are much easier to acquire (e.g.. crawl from public websites). Thus. plenty of Weakly Supervised Object Detection (WSOD) methods [2] 30 [5]
have been proposed to bridge the gap between full annotation and weak annotation. Due to the lack of box-level annotation, WSOD methods rely on unsupervised proposal generation strategy (e.g., Selective Search 43). Edge Boxes (48) to extract region proposals. Based on the proposals, 3,38 399 formulated WSOD task as Multiple Instance Learning (MIL) problem, which treats each image a bag multiple bags of proposals. With known image-level labels and unknown box-level labels, we can infer the box-level label of each proposal with an MIL classifier. However, WSOD methods usually only localize the most discriminative region and confuse objects with co-occurring distractors, so the performance gap between FSOD and WSOD is still very large.
Equual contribution Corresponding author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

It is worth noting that WSOD ignores the existence of fully-annotaten object detection datasets. By taking this into consideration, another learning paradigm using mixed supervision, i.e., full annotations for set of categories and weak annotations for another set of categories, has been explored by [18 45 5 26]. This learning paradigm has inconsistent names like mixed/cross- supervised object detection in previous literature [). However, "mixed/cross-superviiee" does not emphasize cross-category transfer and could be easily confused with "semm/ommi-supprriieed"""" Inspired by weak-shot learning 44) which transfers knowledge from fully-annotated base categories to weakly-annotated novel categories, we refer to this learning paradigm as Weak-SHot Object base
Detection (WSHOD). Assuming the existence of an off-the-shelff dataset with fully-annotate categories (precise bounding boxes and box-level labels), with the goal to detect the objects of novel categories. we can collect weakly-annotttee data (only image-level labels) for novel categories. Fully-annotated base categories and weakly-annotated novel categories have no overlap. We refer to the dataset with hase (resp., novel) categories as source (resp., target) dataset, since our goal is utilizing base categories to help improve the detection performance on novel categories.
Under this paradigm, the key problem is what and how transfer from fully-annotated base categories to weakly-annotated novel categories. Existing works 126] 45) still follow the typical framework of WSOD, that is, feeding proposals into an MIL classifier. Due to the availability of box-level annotations of base categories, they train a CNN backbone with a region proposal network (RPN) to generate proposals. Since the objects of different categories share many common characteristics (e.g., closed contour), objectness is supposed to be transferrable across different categories. Therefore, the trained network could also generate proposals for novel categories, which will be sent to the MIL classifier.
In this paper, beyond class-agnostic objectness, we consider another two targets to be transferred: mask prior and semantic similarity. Since both base categories and novel categories have imaee-level labels, we can obtain the coarse semantic masks (e.g., CAM (46)) for each image. We conjecture that the coarse semantic mask could provide strong guidance for detecting objects, so we attempt to integrate coarse semantic mask into our CNN backbone. In particular, we employ the image-level labels to train a classifier to derive CAM (46), and append the derived CAM to the feature map in CNN backbone. Because CAM provides the rough probability that each pixel belongs to a certain category, such mask prior information combined with the feature map could greatly enhance the ability to locate and identify candidate bounding boxes. By saying "transfer mask prior", mean that the ability to detect objects based on mask prior and feature map can be transferred from base categories novel categories, which will help detect the objects of novel categories.
Besides, following the iterative training strategy in 455, we progressively mine the pseudo bounding boxes of novel categories, during which the noisy bounding boxes are filtered out. The pseudo bounding boxes with correct (resp., incorrect) pseudo labels assigned by the MIL classifier are referred to as inliers (resp., outliers). We argue that semantic similarity, whether two bounding boxes belong to the same category or not, is class-agnostic. Then, we attempt to transfer semantic similarity from base categories to help identify the outliers of novel categories. Specifically, we train a similarity network based on the ground-truth bounding boxes of base categories to verify whether two bounding boxes belong to the same category. We divide the pseudo bounding boxes of novel categories into batches and the pseudo bounding boxes in each batch have the same pseudo label. Within each batch, we apply the trained similarity network to all pairs of pseudo bounding boxes to compute their semantic similarities, and then average the similarities for each one. If one batch is dominated by inliers, an inlier should be close most other pseudo bounding boxes and its average similarity should be high. Thus, we can use average similarity to distinguish outliers and inliers.
In summary, the transferred mask prior could help obtain better candidate bounding boxes, while the transferred semantic similarity could help discard the noisy pseudo bounding boxes. We conduct extensive experiments on three datasets. The results demonstrate that our proposed approach with transferred mask prior and semantic similarity can significantly boost the performance of weak-shot object detection. Considering that the key of our method is Transferring Mask prior and Similarity, we dub our method as TraMaS. Our main contributions are as follows:
We propose novel approach named TraMaS for weak-shot object detection (WSHOD). Besides alass-agnostic objectness, we also propose to transfer mask prior and semantic similarity.

---

We integrate CAM derivation into our network, which can help generate better candidate bounding boxes with mask prior.
We employ a similarity network to learn semantic similarity, which can be used to filter out noisy pseudo bounding boxes.
Our TraMaS method outperforms all state-of-the-ar WSOD and WSHOD methods on three benchmark datasets.
2 Related Work
2.1 Weakly Supervised Object Detection
Weakly supervised object detection (WSOD) proposes to learn a detector with only image-level category labels. Owing to the scareity of bounding boxes, WSOD tends to generate candidate proposals using unsupervised proposal generation strategies such Edge Boxes [48] and Selective Search 43. The final object results are screened from these candidate proposals. Essentially, WSOD image classification problem with multi-instance learning (MIL) classifier. B
can be cast proposed an end-to-end weakly supervised deep detection network which employs two data streams to select proposals and perform region-level classification simultaneously. Following 3 previous WSOD methods can be categorized into two groups according to whether using iterative refinement.
On the one hand, 8 introduced weak cascaded convolutional networks by utilizing Class Activation Map (CAM) and weakly supervised segmentation to improve the quality of generated proposals. 2 minimized the difference between annotation-aware conditional distribution and annotation- agnostic prediction distribution, with a dissimilarity coefficient based WSOD framework. [30] proposed a unified instance aware and context- foccuse weakly supervised learning architecture to tackle the challenges of WSOD. On the other hand, some methods adopted iterative refinement. (38) unified MIL. and multi-stage instance classifiers into a single deep network with online instance classifier refinement. The subsequent work [39] introduced proposal clusters to [38 (21 explored attention regularization for WSOD training and proposes comprehensive attention self-distillation. Despite the fact that these methods have achieved promising results, because of the scarcity of ground-truth bounding boxes, the localization of objects is still far from satisfactory. In this paper, we follow the second group of methods and adopt an iterative training strategy.
2.2 Weak-shot Object Detection
Due the lack of box-level annotations, WSOD heavily relies the quality of candidate proposals, and the learned detector can not refine the proposals. Thanks to the off-the-shelf fully-annotated object detection datasets, weak-shot object detection (WSHOD) proposes to leverage the fully-annotated dataset as source dataset to improve the performance of WSOD on weakly-annotater data. Existing WSHOD methods [42] 16, 36] mainly focus on mining various types of knowledge in the base categories and transferring them to novel categories. Many methods B1 35 9
learn an appearance model from base categories and transfer to novel categories. Previous results have proved that inasainnvaiaatt knowledge can effectively alleviate the problem that WSOD cannot accurately locate objects. For example, 18
and [24] transferred the difference between classifier and detector from base category to novel category. (26) learned class-invariant objectness from base categories
with an adversarial domain classifier. With the help of learned objectness knowledge, this method can distinguish objects and distractors, and thus improves the ability to reject distractors in novel categorics. In [45]. Zhong et al. employed one-class universal detector learned from base category to provide proposals for the MIL classifier, which are both used to mine pseudo ground-truth for iterative refinement. The method in (5) exploited the spatial correlation between high-confidence bounding boxes output by the MIL classifier. In this paper, besides objectness and regression ability, we additionally explore two transfer targets: mask prior and semantic similarity.
Besides weak-shot object detection, weak-shot learning paradigm (transferring knowledge from fully-annotated base categories to weakly annotated novel categories) has also been studied in other computer vision tasks like classification [4) semantic segmentation [7], and instance segmentation
19 25 Although transferring similarity has been investigated in transferring mask prior
remains unexplored due to the uniqueness of object detection task.
3

---

Mask Generator
MIL.
Similarit
image lobel
Classifler
Network
euth bbox
alWP
Rastume
sottunas
puirsise frotures
Lyin
Maps
psoudo
stgrood
background corsotnnt
oiflnes
Mask Generator
MIL. classlieer
poeudo
Similurity
Network
D,
weighted biuss
ebjeciness
Backbone
regressor
Fron
D.
candidane bbox
(sheop, person)
Figure I: Our network is built upon an object detection network (backbone, RPN, and detection head) and MIL classifier. We unify mask generator with the object detection network provide mask prior information. The similarity network is trained to predict semantic similarity to denoise pseudo bounding boxes. Both the source dataset D, and target dataset D, are involved in the whole network training. During testing, we use the object detection network, mask generator, and MIL classifier. Note that the blue (resp., orange) arrow line indicates the data stream of proposals (resp., candidate bounding boxes).
3
Our Method
In this paper, our target is to improve the performance of object detection eeakly-annotated novel categories with the help of fully-annotated base categories. Under this paradigm, we have access to source dataset D, and target dataset Dr. D2 has full annotations for base categories (i.e., precise bounding boxes and box-level category labels) while D, only has weak annotations (image-level category labels) for novel categories. Base categories and novel categories have overlap. Our method is trained with a mixture D, and D,, and evaluated on the test set novel categories.
3.1 Object Detection Network and MIL Classifier
Our basic framework is illustrated in Figure T The architecture composed of object detection network and an MIL classifier. The object detection network is built upon Faster-RCNN [29 and comprised of backbone, Region Proposal Network (RPN), and detection head. Specifically, we employ ResNet50 [17] as backbone and RPN to generate proposals. The region features of generated proposals are sent to the detection head, which consists of objectness predictor and box regressor. Both objectness predictor and box regressor are fully connected layer. The objectness predictor is used predict the objectness ground-truth bounding box is above the preset threshold and ( otherwise. The box regressor is used to regress the proposal to candidate bounding box. The loss functions of objectness predictor and box regressor are formulated follow,
proposal. The objectness label of proposal
if its loU with
Lobj = Σ à,logo; + (1 6,) oz), Lreg Σ
Lz
(1)
where L, indicates smooth L, loss 13
Og (resp., 6;) and b; (resp., b.) mean the predicted (resp.. ground-truth) objectness label and regression value the i-th proposal. Following [45), we filter the proposals with low objectness using the threshold 0.05 and refer to the remaining proposals as candidate bounding boxes. At the beginning training process, we supervise the object detection network with the ground-truth bounding boxes in the source dataset. Since objectness has proved to be class-agnostic 226 45). the object detection network can also generate reasonable candidate bounding boxes for novel categories.

---

The region features of candidate bounding boxes are fed into an MIL classifier from WSDDN [3. Note that the MIL classifier only utilizes the target dataset and focuses on novel categories. As shown in Figure 1 the MII. classifier has two branches (i.e., classification branch and detection branch), which are almost the same except the regularization operation. Each branch has two fully connected layers and one activation layer. Given the region features of N candidate bounding boxes, the classification (resp., detection) branch outputs the score matrix S E IR XN (resp., S E IgCxXM, where Cn is the number of novel categories), Softmax is applied along different dimensions of S and S to obtain category-awand score matrix S" and proposal-aware -aware score matrix $4, Intuitively, S is class score matrix for each proposal while Sd indicates the contribution of each proposal to classes. The final score matrix $ is the multiplication of S and Sd. Then, S are summed along the proposal dimension to obtain image-level classification score. By denoting each entry in S (resp., S) as Sc,i (resp., Se,)) the above procedure can be formulated as
N
ΣN En 8c,s = 5
S og De
mi
Σ Sa,s -mil=Baa (v.tti Bee,,
where Timit is the binary cross-entropy loss, which uses image-level category labels Vc of novel categories supervise the final classification scores. Based the final prediction scores S, we filter out the candidate bounding boxes with the threshold 0.8. Then, we assign pseudo category labels to the remaining pseudo bounding boxes based on category-aware prediction scores S*.
After training the MIL. classifier, we can mine the pseudo bounding boxes of novel categories from both source and target datasets. These pseudo bounding boxes can be added to supervise the object detection network to detect the objects of novel categories better. Next, we explore two targets to be transferred from base categories to novel categories: mask prior in Section 3.2 and semantic similarity in Section 3
3.2 Incorporating Mask Prior into Object Detection Network
Provided with image-level labels, we can obtain the coarse semantic masks (e.g., CAM) of images. These coarse masks provide the rough probability that each pixel belongs to certain category, which may enhance the ability of the object detection network to locate and identify objects. Previously,
some WSOD methods 66 37 34 11 12] utilize coarse masks (e.g., CAM) to assist in detecting
objects. Besides, many weakly supervised semantic segmentation methods (20 23 take the coarse mask as a baseline and propose different refinement strategies to obtain more accurate semantic masks. this paper, we unify semantic mask generation and object detection into one network.
Based on the object detection network, we introduce an additional mask generator which is fed with the output feature from the penultimate layer of backbone, as shown in Figure [ Inspired by M. the mask generator is composed of three convolutional layers and a normalised Global Weighted Pooling (nGWP). We assume the pixel-wiig lixee-wise output of convolutional layers in channel (channel c corresponds to category c) is Pe,i.. and then add a background channel with constant value to Pe.i.j- Further we compute softmax to obtain the semantic mask Mc.i.j- The predicted image-level score ye and the multi-label soft-margin loss Lmask (28) of mask generator are defined as follow,
Li.j Ca..P... ye = c+ e+E,, Ens Meas'"
Lmaos
1 dolog(
Σ
1
+ (1 - Nogl 1 Ite ve )
(3)
C
I -evvo
c=
where is the total number of categories and e is set as 1.0. de = if the image contains the category and de = 0 otherwise.
In contrast to WSOD methods 8 which directly select positive proposals based on the mask we append the coarse mask to the feature map in the backbone to acquire a mask-enhanced feature map, which can provide prior information for objectness prediction and box regression. Based on the mask-enhanced feature map, we can train a well werl-perrorming detection network with fully-annotated source dataset to generate faithful proposals for base categories. We conjecture that the ability of using mask prior to facilitate object detection can be transferred from base categories to novel categories. Thus, the enhanced detection network is expected to produce better candidate bounding boxes for both base and novel categories.
5

---

3.3 Semantic Similarity Transfer
Recall that based on the output scores of the MIL classifier, we can mine pseudo bounding boxes of novel categories. However, the labels of pseudo bounding boxes could still be inaccurate. We refer to the bounding boxes with accurate (resp., inaccurate) labels as inliers (resp.. outliers). To mitigate the adverse effect of outliers, we attempt to use semantic similarity to recognize the outliers, in which semantic similarity indicates whether two bounding boxes belong to the same category. Specifically, we train a Similarity Network (SimNet) on base categories to predict the semantic similarity between two bounding boxes. Our SimNet consists of two fully connected layers followed by a sigmoid activation layer. We first construct batch of the region features F E Bxd from the ground-truth bounding boxes of base categories, in which is the batch size and dl is the feature dimension. we reorganize to generate a batch of pairwise features F E TR. x2d The pairwise features are fed into SimNet and the predicted similarity scores A are supervised by the ground-truth similarities A. Specifically, if a pair of bounding boxes belong to the same category, the corresponding entry in A is 1. Otherwise, the corresponding entry in Å is O. We adopt binary cross-entropy loss for SimNet:
sim LBC
(4)
Considering unbalanced similar and dissimilar pairs, the standard batch sampling strategy (i.e., sampling from the datasets randomly) will cause severe imbalance issues. To this end, we adopt sampling strategy adapted to our setting. At first, we select K categories from all categories at random. Then we sample M bounding boxes from each selected category randomly to construct a batch, leading relatively balanced batch the input for SimNet. In our experiments, we set both K and M to 8, so
x M 64.
After training SimNet on the ground-truth bounding boxes of base categories, we apply the trained SimNet to the pseudo bounding boxes of novel categories. At each time, we randomly sample M pseudo bounding boxes from the same novel category to construct a batch. Then, we predict the similarity between pairs of bounding boxes. The similarity score between the i-th and j-th bounding boxes is a. For the i-th bounding box. calculate the average of similarity scores between this bounding box and the other bounding boxes:
M
1 M
Oi.j
+ aj.
(5)
2
For the bounding boxes with the same pseudo label, we conjecture that inliers should be dominant and thus the average similarities of inliers should be higher than those of outliers. Therefore, we use the average similarity w, as the assigned weight to each bounding box, leading to the following weighted losses:
Lbj = - Σ (o(oooog(o)) + (1- 1oooo - o,)), o.))),
L = Σ w.Li(be,bu bb).
(6)
which are basically the same as Eqn. except the weights wy, By assigning higher weights to inliers, the inliers would contribute more to learning a robust classifier.
3.4 Iterative Training Strategy
Similar to 45 train the whole framework iteratively. In each iteration, execute the following three steps. The whole algorithm is summarized in Algorithm
In the first step, we train the object detection network with mask generator based on the fround-truth bounding boxes of base categories in the source dataset. The loss function of the first step as follows,
Litept L%, + L + almask,
where a and y are saper--arraeters set as 0.1 and 1.0 via vrads-aali,ttion, respectively. the first iteration, weights w; in Eqn. (7) are all set as
In the second step, we use the object detection network trained in the first step to generate candidate bounding boxes for the target dataset. We train the MIL classifier based on the candidate bounding

---

novel Algorithm/ Iterative Training Strategy
Input: source dataset Ds, target dataset D., number of refinement iterations T
Output: Object Detection Network (ODN), MIL classifier, mask generator
1: Train ODN with mask generator based on D,
2: Utilize ODN to generate candidate bboxes for D,. Train MIL classifier together with backbone and mask generator based Dr;
3: for i=1, 2, ... ,T do
4: Mine pseudo bboxes of novel categories D, and Dr with MII. classifier. Freeze backbone and mask generator, train SimNet based on D. and utilize SimNet to assign weights to pseudo
bboxes;
5: Refine ODN with mask generator based on Dj and Di:
6: Utilize ODN to generate candidate bboxes for Dr. Refine MIL classifier together with backbone and mask generator based on Dz.
7: end for
boxes in the target dataset. Note that when training the MII. classifier, the backbone and mask generator are also updated to learn better region features of candidate bounding boxes. The loss function of the second step is as follows,
Listep2 = Lemil + Lomaks
(8)
where 3 is hyper-parameter set 0.1 via cross-validation.
In the third step, we use the MII. classifier trained the second step to mine the pseudo bounding boxes of novel categories on both source and target dataset. Note that for the source dataset, we remove the pseudo bounding boxes whose IoUs with ground-truth bounding boxes of base categories are larger than 0.1. We train the SimNet using the region features of ground-trutt toound-tuuth bounding boxes of base categories, with the following loss: Latep3 Liiim
Then, apply the trained SimNet the pseudo bounding boxes of novel categories obtain their weights as described in Section[.33 Finally, we merge the weighted pseudo bounding boxes of categories with the ground-truth bounding boxes of base categories the supervision of the first step for the next iteration.
4 Experiments
4.1 Datasets
In our experiments, we introduce our used source dataset and target dataset separately, The source (resp., target) dataset is fully-annotated (resp., weakly- -
Source Dataset: Following [45), we investigate COCO 2017 detection dataset 27 as our source dataset. COCO 2017 is composed of official train/validation split with 118287 and 5000 images with 80 categories covering the 20 categories in VOC. To reduce the impact of novel categories in the source dataset, we follow [45] and remove all the images that have novel categories, i.e., 20 categories in VOC. After that, we denote the resulting COCO 2017 as COCO-60 which consists of 21987 training images and 921 validation images. Both the training and validation set merged together as the source dataset.
Besides, choose ILSVRC 2013 detection dataset 32 another source dataset. The 200-category dataset, including the 20 categories in VOC, is split into training (395909 images) and validation (20121 images) sets. Similar to COCO-60, we remove the images which contain novel categories and obtain 143905/6229 train/val images. We denote the total 150134 34 images as ILSVRC-179 (water bottle and wine bottle in ILSVRC while bottle in VOC). COCO-60 and ILSVRC-179 are used the source dataset separately in different experiments.
Target Dataset: We take Pascal VOC 2007 [10 as the target dataset, which is split into trainval (5011 images) and test (4952 images) sets. Following 45), we leverage the trainval set as our training set and evaluate method on the test set. All 20 categories in VOC are treated as novel categories.
7

---

Table 1: mAP comparison with state-of-the-arts on VOC-20 test set. We re-train Faster RCNN with mined boxes and denote it as "+distill". Unless marked, the backbones these methods are VGG16. indicates that the backbone is ResNet50 instead. (single-scale) means single-scale testing.
Metlid
8010
dog
ztilke
pess.
plut
sheep
WSOD:
WSDIDN
46.4
58.3
35.5
25.9
14.0
66.7
53.0
39.2
8,9
41.5
26,6
38,6
44.7
59,0
10.3
17.3
40.7
48.6
56.9
50.5
39.3
WOON
49.5
54.
38.5
20-2
16.2
70.8
58.0
42.5
10.9
46.1
20-9
43.3
47.8
64.1
13.8
23.3
45.0
56.1
60.5
54.5
42.6
OILR 1X
58.0
62.4
31.1
19.4
13:0
65.1
62.3
24.4
243
44.7
306
25.3
5788
65.5
15.7
28.1
41.7
46.9
68.3
62.6
41.3
69.0
19.2
15.7
64-4
63.3
06.4
58.9
71.5
66.2
32.8
47.4
71.7
54.8
CASD 21
56.4
92.2
36.5
33.1
27.2
76.0
54.3
09.4
33.5
59.9
57.4
52.3
68.4
73,1
26.7
28.1
68.3
55.2
13.4
71.2
56.6
CASD 11
-dsill
06.0
61.5
58.4
31.9
51.6
75.7
55.3
68.3
36.8
59.5
61.00
53.9
66-4
73.0
26.1
29-4
65.7
54.2
76.5
70.7
57.1
56.3
38.8
75.0
76.0
Zhann
64.8
50.7
65.5
45.3
75.7
14.0
80.1
31.3
77.0
26.2
79.3
14.8
66.5
57.9
11.5
68.2
59.0
14.7
65.5
59.7
7hongaa a
X
edieil
02.6
16.1
64.5
99.9
68.5
74.6
75.6
90.9
10.6
75-4
25.5
99.9
73.4
71.0
59.1
16.7
64.1
99.5
73.4
68.0
59A
Zbong ct
S
55.5
57.7
65.1
41.3
43.0
73.6
75.7
90.4
33.4
72.2
33.8
81.3
79.6
63.0
59.4
10.9
65.1
64.2
72.7
67.2
60.2
55.6
53.7
67.4
76.3
76.6
33.0
20.3
66.2
61.1
66.4
53.7
08.0
76.2
76.3
54.3
29.0
78.
64.0
66.7
Oun
06.5
16.7
08:1
67.7
67.0 9
76.5
78.0
81.
13.9
77.6
594
90.1
73.0
06.3
630
15.1
69
60.7
76.
68.1
62.1
Oun--haill
57.8
59.9
67.9
43.9
47.5
75.4
75.2
79.3
33.1
75.4
32.1
79.8
77.4
68.3
63.1
18.4
70.0
59.9
76.2
60.3
62.4
Ounedielll
68.6
69.5
76.3
71.8
8038
71.0
99.9
78.
6400
62.3
Foster RUNN
75
630
74.4
60.8
56.5
7900
83.8
K3A
54M
66.
77.4
676
Table 2: CorLoc comparison with state-of-the-are on VOC-20 trainval set. We re-train a Faster RCNN with mined boxes and denote it as "+distill". Unless marked, the backbones of these methods are VGG16. indicates that the backbone is ResNet50 instead. (single-scale) means single-scale testing.
Method
0000
Nike
Nrld
beot
boodle
Nos
Car
COR
char
COW
rable
dig
horse
mhbie
pars.
platt
soft.
IC
Coe
WSDDN
68.9
68.7
42.5
40,5
7366
75.2
53.7
33.5
56.1
27.5
76.0
WCEN
83.9
72.8
64.5
44.1
40.1
65.2
82.5
58.0
53.3
12.5
25.6
$3.7
67.4
37.4
26.8
49.1
27.9
64.5
55.2
56.7
OIUR
15
$1.7
58.4
48.7
49.5
32.8
81.7
85.4
43.1
806
39.5
35.7
33.7
60.5
58.8
21.5
57.9
76.3
563
75.3
51.4
00.6
KL
50
79.6
85.5
52.2
47.9
37.0
$3.8
$3.4
43.0
38.3
40.1
336
30.9
$7.3
90.8
27.0
58.2
75.3
68.5
75.7
78.9
62.7
96.7
12.4
16.4
72.1
63.8
35.7
24.3
$4.3
CASD 01
70.2
42.5
51.0
$7.9
CASO 11
81.4
79.7
46.9
43.7
16.3
12.8
93.6
75.4
31.3
37.0
449
23.5
46.3
70.8
60.6
WSHODD:
69.9
41.3
88.1
91.9
53.8
96.1
24.9
45.5
40.3
57.0
Zhosg
$7.4
$6.3
88.8
85.1
56.3
Thmng
wiiiil
23.9
06.7
47.7
61.6
70.3
85.8
$9.9
80.2
47.9
$4.5
90.4
01.5
67.6
73.3
71.8
67
61.3
15.7
2boag
3 disill
25.8
67.5
57.1
68.5
68.3
$5.8
90.4
85.7
43.5
95.2
31.5
90.9
04.2
58.8
72.4
23.5
98.7
06.1
99.7
76.7
75.2
Oure
(isgle-scale
Oan"
29.1
67.9
49%
68.0
20.4
88.6
90.6
91.6
90.1
$6.6
11.8
91.5
91.3
65.2
73.6
25.3
67.00
64.4
17.
disil
99.7
09.4
90.9
68.5
71.1
56.9
91.5
91.0
50.1
96.4
33.2
92.4
92.7
90.1
75.3
24.5
95.3
09.3
90.6
53.1
17.5
Oumedreill"
90.6
67.4
59.7
70.5
73.8
66.6
91.7
99.8
$1.00
46.1
34.0
93.7
04.3
90.
73.0
26.5
98.2
66.2
69.8
63.1
77.7
ESOD:
Considering that VOC is fully- anlly-annotated, we ignore the bounding boxes and only keep the image-level category labels as the supervision for WSHOD and name the processed VOC as VOC-20.
4.2
Implementation Details
Our proposed framework is based Faster RCNN 29 with ResNet-50 17 pretrained ImageNet 32 as the backbone. Following [45), we train our model using the SGD optimizer. The learning rate is initialized to 10-3 and reduced 8 10 The weight decay and momentum are set to 1 x 10 and 0.9, respectively. The random seed is set to 222. All experiments are conducted on two 24GB TITAN RTX, with batch size of 16 images. Recall that we train our whole framework in an iterative manner, we conduct four iterations (T 4) following (5). The threshold used to obtain candidate bounding boxes is 0.05. The threshold used obtain pseudo bounding boxes of novel categories is 0.8. Following [45, we adopt two evaluation metrics, i.e., mean average precision (mAP) and Correct Localization (CorLoc) 77. mAP is calculated on VOC-20 test set and CorLoc is calculated on VOC-20 trainval set. Following multi-scale training/testing strategy in previous works 26 38 we use five image scales (480, 576, 688, 864, 1200) for both training and testing unless otherwise specified.
8

---

Table 3: Ablation studies on different modules on VOC- 20 test set. B-1 (resp., B-2) indicates that mask is ap- pended to the feature map of the last (resp., penultimate) layer of the backbone. All methods adopt single-scale testing.
Table 4: Result comparison with WSHOD methods when taking ILSVRC- 179 as source dataset. Unless marked, the backbones of these methods are VGGI6. indicates that the backbone is ResNet50 instead.
Mask
Similarity
Methods
mAP
CorLoc
B-1
B_2
Cosine
SimNet
mAP
MSD [26]
47.5
65.3
1
58.2
Chen et al. [
55.1
69.3
2
V
59.7
Zhong et al. 45
56.5
3
V
59.1
Ours
57.8
74.1
4
58.8
Ours
58.3
74.8
5
59.5
6
v
60.9
4.3 Experiments on COCO-60
In this section, use COCO-60 as the source dataset and VOC-20 the target dataset. We compare our TraMaS method with the state-of-the-ar approaches on VOC-20, including Weakly-Supervised
Object Detection (WSOD) methods WSDDN (3. WCCN [8, OICR (38] PCL (39] Ren et al.
30). and CASD 121, as well as Weak-SHot Object Detection (WSHOD) methods 22 All baselines take VGG16 as the backbone except 45 which also uses ResNet50. Following (45]
we apply distillation strategy (i.e., retrain Faster RCNN with pseudo labels) to two competitive WSOD baselines (CASD 21 and Ren al. 30), the competitive WSHOD baseline [45) and our method. Specifically, distillation means training a full supervised object detector with obtained pseudo bounding boxes of novel categories
COCO-60 and VOC-20. The results of all baselines are copied from 21 45 except [26] 5 which are reproduced based on our implementation. We use ResNet50 the backbone by default and also report the results using VGGI6. We observe that the results using VGG16 are only slightly worse than those using ResNet50, which is consistent with the observation in [3] One possible explanation is that the MIL classifier may back-propagate uncertain
and inaccurate gradients to backbones while skip connection in ResNet50 can not alleviate this issue. We report mAP and CorLoc in Table [ and Table 2 respectively. can be seen that the strongest baseline is [45]. No matter using ResNet50 or VGGI6+disttll, our TraMaS method significantly outperforms
4.4 Experiments on ILSVRC-179
We also conduct experiments by using ILSVRC-179 as the source dataset and VOC-20 as the target dataset. The results are reported in Table [ Note that since the target dataset is the same, the results of WSOD baselines are the same as in Table Tan Table 2 The baseline results are copied from 26 45 except (5) which is reproduced based on our implementation. The results Table again demonstrate the effectiveness of our method.
4.5 Ablation Studies
By taking COCO-60 as the source dataset, we conduct ablation studies to investigate the effectiveness of mask prior and semantic similarity, the results are shown in Table 3 We first evaluate the plain framework without mask prior and similarity weight (row 1). Based on the plain framework in row 1, append the coarse semantic mask to the last layer in the backbone (row 2). We also try to append the coarse mask to the penultimate layer (row 3). By comparing row 2, 3 and row 1, we can see that incorporating mask prior into the backbone can dramatically increase the performance, i.e., 59, 59.7% and 59. % v.S. 58.2%. The comparison between row 2 and row 3 shows that B_, achieves better performance,
append the mask prior to the last layer by default.
Based on the plain framework in row 1, we use similarity weights to suppress the outliers (pseudo bounding boxes with inaccurate labels) mined by the MII. Classifier, leading to the results in row 5. By comparing row and row 5, we can see that transferred similarities are effective in identifying the noisy pscudo bounding boxes. Considering that similarity can be calculated in casier way, also

---

remove SimNet and simply calculate cosine similarities based on the region features between two bounding boxes. The results are shown row which shows that it is useful to train SimNet to predict similarity.
The last is full-fledged method, in which the combination of mask prior and similarity weight achieves further improvement.
4.6 Hyper- papper-prrameters Analyses
Our method introduces three hyper-parameters, i.e., O, 3, and in Eqn. and (8) Training SimNet also involves two Supplementary due to space limitation. For all the other syper-parametent we follow [45 without further tuning.
K and M. The analyses of these hyper- hyper-paramedee parameters are left to
4.7 Size of Source Data
In order explore the influence of the amount of fully supervised data on weak-shot object detection. We study different scales of datasets as source data and conduct experiments with our method. Specifically, we randomly sample 20% and 50% of COCO-60 the source dataset and use VOC-20 as target dataset. Based on the setting of "Ours* (sinle scale)', we report the results on VOC-20 test set. The final mAP on 20% source data 59. 99.4% and mAP on 50% source data is 59.8%. It can be seen that as the size of source dataset increases, the performance is improved as well.
4.8 Overlap in Base and Novel Categories
In some real-world cases, few categories may have both fully-annotated data and weakly-annotate data. Therefore, we conduct the experiments with our proposed method under this setting. In our original COCO-60 dataset, we removed all the images which contains the objects of novel categories following [45. Now among the 20 novel categories, the alphabetical order as mixed categories, and treat the remaining 10 categories as novel categories. Then, we only remove the images which contains the objects of 10 novel categories from COCO and name it COCO-70 instead of COCO-60 dataset, so that 10 mixed categories have both fully-annotated data in the source dataset and weakly-annotater data in the target dataset. We conduct experiments with 'Ours+distill*', The final mAP 69.9% (mixed categories) v.s. 62.4% (novel categories) show that the performance mixed categories much higher than novel categories thanks to the extra supervision bounding boxes of mixed categories.
choose the first 10 novel categories based on
4.9 Visualization Results
To better explore the superiority of method, visualize some test images, with bounding boxes predicted by different methods on VOC-20. The visualization results can be found in Supplementary. It can be seen that previous WSOD and WSHOD methods (e.g., [45] tend focus the most discriminative regions, while our method is encouraged to detect the whole object and thus locates the objects with higher precision.
Conclusion
In this work, we have investigated weak-shot object detection by transferring the knowledge learned from the fully-annotated source dataset. Besides the class-invariant objectness explored in previous works, have also transferred mask prior and semantic similarity. Extensive experiments have proved the superiority of our proposed method.
Acknowledgements
The work was supported by the National Key R&D Program of China (2018AAA0100704), National Natural Science Foundation of China (Grant No. 61902247), the Shanghai Municipal Science and Technology Major Project (Grant No. SHZDZXX00)))) and the Shanghai Municipal Science and Technology Key Project (Grant No. 205 1 11000000).
10

---

References
[1] Araslanov and Roth. Large-scale knowledge transfer for object localization in imagenet. In CVPR, 2020.
[2] Aditya Arun, CV Jawahar, and M Pawan Kumar. Dissimilarity coefficient based weakly supervised object detection. In CVPR, 2019.
[3] Hakan Bilen and Andrea Vedaldi. Weakly supervised deep detection networks. In CVPR, 2016.
[4] Junjie Chen, Li Niu, Liu Liu, and Liqing Zhang. Weak-shot fine-grained classification via similarity transfer. NeurIPS, 2021.
[5] Zitian Chen, Zhiqiang Shen, Jiahui Yu, and Erik Leamed-Miller. Cross-supervised object detection. arXiv preprint (tti::2006.15555: 2020.
[6] J. Choe, S. Lee, and H. Shim. Attention-based dropout layer for weakly supervised single object localization and semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 11 2020.
[7] Thomas Deselaers, Bogdan Alexe, and Vittorio Ferrari. Weakly supervised localization and learning with generic knowledge. International Journal of Computer Vision, 100(3):275-293, 2012.
[8] Ali Diba. Vivek Sharma, Ali Pazandeh, Hamed Pirsiavash, and Lue Van Gool. Weakly super- vised cascaded convolutional networks. In CVPR, 2017.
[9] L. Dong, J. B. Huang, Y. Li, S. Wang, and M. H. Yang. Weakly supervised object localization with progressive domain adaptation. In CVPR, 2016.
[10) Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88 (2):303-338, 2010.
[11] Yan Gao, Boxiao Liu, Nan Guo, Xiaochun Ye, Fang Wan, Haihang You, and Dongrui Fan. C-midn: Coupled multiple instance detection network with segmentation guidance for weakly supervised object detection. In ICCV, 2019.
[12] Weifeng Ge, Sibei Yang, and Yizhou Yu. Multi-evidence filtering and fusion for multi-labe classification, object detection and semantic segmentation based on weakly supervised learning. In CVPR, 2018.
[13] Ross Girshick. Fast r-cnn. In 2015.
[14] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014.
[15] Nicolas Gonthier, Said Ladjal, and Yann Gousseau. Multiple instance learning on deep fea- tures for weakly supervised object detection with extreme domain shifts. arXiv preprint arXiv:2008.01 178, 2020.
[16] M. Guillaumin and V. Ferrari. Large-scale knowledge transfer for object localization in imagenet. In CVPR, 2012.
[17] K. He, X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. In CVPR, 2016.
[18] Judy Hoffman, Sergio Guadarrama, Eric Tzeng, Ronghang Hu, Jeff Donahue, Ross Girshick, Trevor Darrell, and Kate Saenko. Lsda: Large scale detection through adaptation. In NeurlPS, 2014.
[19] Ronghang Hu, Piotr Dollár, Kaiming He, Trevor Darrell, and Ross Girshick. Learning to segment every thing. CVPR, 2018.
[20] Z. Huang, Wang, J. Wang, Liu, and Wang. Weakly-supervised semantic segmentation network with deep seeded region growing. In CVPR, 2018.
11

---

[21] Zeyi Huang, Yang Zou, BVK Kumar, and Dong Huang. Comprehensive attention self eeff-istillation for seakkly-uperrviee object detection. Advances Neural Information Processing Systems, 33, 2020.
[22] Naoto Inoue, Ryosuke Furuta, Toshihiko Yamasaki, and Kiyoharu Aizawa. Cross-domain weakly-supervised object detection through progressive domain adaptation. In CVPR, 2018.
[23] Sanghyun Jo and In-Jae Yu. Puzzle-cam: Improved localization via matching partial and full features. arXiv preprint wxxiv:2201.11255 2021.
[24] J Kuen, F. Perazzi, Z. Lin, J Zhang, and Y. P. Tan. Scaling object detection by transferring classification weights. In ICCV, 2020.
[25] Weicheng Kuo, Anelia Angelova, Jitendra Malik, and Tsung- Yi Lin. Shapemask: Learning to segment novel objects by refining shape priors. ICCV, 2019.
[26] Yan Li, Junge Zhang, Kaiqi Huang, and Jianguo Zhang. Mixed supervised object detection with robust objectness transfer. IEEE transactions on pattern analysis and machine intelligence, 41 (3):639-655 653, 2018.
[27] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
[28] A. Paszke, S. Gross, F. Massa, A. Lerer, and S. Chintala. Pytorch: An imperative style, gegh-performance deep learning library. In NeurlPS, 2019.
[29] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NeurIPS, 2015.
[30] Zhongzheng Ren, Zhiding Yu, Xiaodong Yang. Ming- Yu Liu, Yong Jae Lee, Alexander G Schwing, and Jan Kautz. Instance-award context-focused, and memory-efficient weakly supervised object detection. In CVPR, 2020.
[31] M. Rochan and W. Yang. Weakly supervised localization of novel objects using appearance transfer. In CVPR, 2015.
[32] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 115(3):211-255 1-222, 2015.
[33] Y. Shen, R. Ji, Y. Wang, Z. Chen, and Y. Wu. Enabling deep residual networks for weakly supervised object detection. In ECCV, 2020.
[34] Yunhang Shen, Rongrong Ji, Yan Wang, Yongjian Wu, and Liujuan Cao. Cyclic guidance for weakly supervised joint detection and segmentation. In CVPR, 2019.
[35] M. Shi, H. Caesar, and V. Ferrari. Weakly supervised object localization using things and stuff transfer. Computer Society, pages 3401-3410, 2017.
[36] Z. Shi, P. Siva. and T. Xiang. Transfer learning by ranking for weakly supervised object annotation. In BMVC, 2012.
[37] K. K. Singh and L.. Yong. Hide-and-seek: Foreing a network be meticulous for weakly- supervised object and action localization. In ICCV, 2017.
[38] Peng Tang, Xinggang Wang, Xiang Bai, and Wenyu Liu. Multiple instance detection network with online instance classifier refinement. In CVPR, 2017.
[39] Peng Tang, Xinggang Wang, Song Bai, Wei Shen, Xiang Bai, Wenyu Liu, and Alan Yuille. Pel: Proposal cluster learning for weakly supervised object detection. IEEE transactions on pattern analysis and machine intelligence, 12(1):176-191, 2018.
12

---

[40] Y. Tang, J. Wang, B. Gao, E Dellandréa, R. Gaizauskas, and L. Chen. Large scale semi- supervised object detection using visual and semantic knowledge transfer. In CVPR, 2016.
[41] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Fcos: Fully convolutional one-stage object detection. In /CCV, 2019.
[42] J. Uijlings, S. Popov, and V. Ferrari. Revisiting knowledge transfer for training object class detectors. In CVPR, 2018.
[43] Jasper RR Uijlings, Koen EA Van De Sande, Theo Gevers, and Arnold WM Smeulders. Selective search for object recognition. International Journal of Computer Vision, 104(2)114-1771, 2013.
[44] X. Zhang, Y. Wei, Feng, Y. Yi, and T. Huang. Adversarial complementary learning for weakly supervised object localization. In CVPR, 2018.
[45] Yuanyi Zhong, Jianfeng Wang, Jian Peng, and Lei Zhang. Boosting weakly supervised object detection with progressive knowledge transfer. In ECCV, 2020.
[46] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In CVPR, 2016.
[47] Siyuan Zhou, Li Niu, Jianlou Si, Chen Qian, and Liqing Zhang. Weak-shot semantic seg- mentation by transferring semantic affinity and boundary. arXiv preprint arXi:2111000119, 2021.
[48] Lawrence Zitnick and Piotr Dollár. Edge boxes: Locating object proposals from edges, In ECCV, 2014.
13


Reviews:
[{"review_id": "omy9r4JNpMC", "reviewer": "Reviewer_3LCY", "summary": "This paper studies mixed supervised object detection where some object categories are fully supervised and the rest are only labeled with class tags. It introduces a (1) weakly-supervised segmentation module and (2) a semantic similarity learning module to the existing framework to improve the performance. Experimental results on two benchmarks validated the effectiveness of the proposed framework.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- The problem setting is interesting. Compared to WSOD setting, the mixed-supervised task is more realistic and common in practice. However, this task is less explored than WSOD in the community.\n- The experimental results are promising and both novel components help.\n- Writing is mostly clear and easy to follow.\n\nWeakness:\n- To unifying the weakly-supervised segmentation and detection isn't a new idea. For example, multiple prior works [1-3] have studied and explored different architectures for this purpose. Some of these methods can be swapped into the proposed framework seamlessly and should be ablated and studied.\n \n\n[1] Cyclic Guidance for Weakly Supervised Joint Detection and Segmentation. Shen et al. CVPR'19\n\n[2] C-MIDN: Coupled Multiple Instance Detection Network With Segmentation Guidance for Weakly Supervised Object Detection. Gao et al., ICCV'19\n\n[3] Multi-Evidence Filtering and Fusion for Multi-Label Classification, Object Detection and Semantic Segmentation Based on Weakly Supervised Learning. Ge et al., CVPR'18\n\n- Sec.3.4 Iterative training seems to be complicated and hard to generalize. How important is this design choices? Will it become significantly worse if end-2-end training is adopted? I don't find experimental evidence to support this design choices.\n\n- The learned mask is concatenated with the feature maps as input features. In fact, a more straight-forward solution to me is to multiply the feature map. I'm wondering whether this has been tried and how well does it perform.\n\n- This paper follows the experimental setting from prior work where the datasets is split into non-overlap base and target categories. However, in practice the real problems are often more messy. The same class can have both annotation often the case. There are also prior work [4] studying this setting, which is missed in this paper. It would be interesting to see how well the proposed method performs to deal with mix-supervised for the same classes.\n\n[4]. UFO2: A Unified Framework towards Omni-supervised Object Detection. Ren et al., ECCV'20\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. Limitations is discussed in the Appendix Sec.7. I think this section is pretty detailed and in-depth.  \n2. No societal impacts are discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {"ethical_concerns": "I have read the Ethics Guidelines, and I don't find any concerns from this paper.\n1. Potential negative societal impacts: It doesn't involve any social experiments, and doesn't use human-derived data.\n2. General ethical conduct: the dataset used in this work are public ones."}}, {"review_id": "gvnJrr41sUu", "reviewer": "Reviewer_gbMi", "summary": "The paper proposes a method to do `mixed supervision` object detection, i.e. some part of the dataset is annotated fully while some part only has weak image level annotations.\n\nThe first contribution is to use semantic mask prior in object detection network. Prior works have already used semantic masks for weakly supervised detection, but the networks were separate. The paper proposes to integrate the mask prediction networks with the object detection network and use the masks to augment features for prediction candidate bounding boxes. The hope is that this would also generalize and produce better candidate boxes for target/novel categories.\n\nThe second contribution is to transfer semantic similarity. The paper incorporates a similarity (metric learning) network in the architectures and learns it using the bounding boxes of the base categories. Then the network is used to estimate the average similarities of pseudo bounding boxes for the novel categories. With the intuition that the average distance of a correct example wrt the rest of the class examples would be lower than that for an incorrect example, they use a weighted loss for object class prediction for the bounding boxes.\n\nThe training is done in an iterative manner and results are provided on public benchmarks COCO and ILSVRC datasets as source, and Pascal VOC 2007 as the target dataset.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "It is specified that base and novel categories have no overlap, but is it also assumed that the weakly supervised images for novel categories would not have any of the base category objects? It seems to be so from the experimental settings too. This would be a strong assumption for deployment, and should be mentioned/discussed explicitly.\n\nThe contributions are novel but, the use of semantic maps has been done already before in weakly supervised settings with separate networks while the present work integrates the networks. This might be good or bad as the integrated network is trained only on the source dataset but a separate network might be trained on other data as well. Similar case for metric learning algorithm. However, showing that an integrated network works for the task and performs better than existing method is a good contribution.\n\nFor, the results table it seems the row Zhong et al.[39]+distill uses VGG16 as the backbone as there is no * on its side. Could you please clarify, what is the backbone? If VGG is used as the backbone for this setting, then for fair comparison the results with ResNet50 should also be reported as it is the best existing method. \n\nA critical aspect in a mixed supervised detection setting is the amount of fully supervised data. An experiment showing the performance with varying amounts of supervised data would further provide an evaluation for the trade-off between supervised data and performance.\n\nOverall the paper is well written and provides two contributions to the architecture as well as a tailored training method. The experiments reported are convincing.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "n/a", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "Z-UJKj55SjS", "reviewer": "Reviewer_itic", "summary": "This paper introduces a mixed supervised object detection method with several improvements (mask generator/prior, semantic similarity transfer) proposed to improve the detection performance in the mixed supervision setting. The ability of detect novel categories is enhanced by the output of the mask generator that is supervised by only image-level labels. A MIL classifier is used to generate pseudo labels (detections) for the novel categories and the quality (average similarity scores) of the pseudo labels are determined by the semantic similarity network trained on only the base categories. Then, the pseudo labels are used in the detector's subsequent training and their losses are weighted by the average similarity scores obtained from the semantic similarity network. This encourages \"good\" pseudo labels to contribute more to the training and vice versa.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n+ The ideas of using mask prior to enhance the detection performance on novel categories and semantic similarity transfer to weight the pseudo labels are quite interesting and novel in the context of mixed/weakly supervised object detection.\n+ Strong experimental results in the standard setting. The proposed method outperforms state-of-the-art mixed- and weakly-supervised methods from very recent published papers.\n+ Paper is generally well written, easy to follow, and has sufficient details for most parts of it.\n\nWeaknesses\n+ Lack of good explanation and evidence as to why the (metric-based) SimNet transfers well to novel categories. At the very least, the authors should show that the average similarity scores on base categories and \"novel\" categories are comparable, when that the scores are computed from the ground-truth boxes.\n+ Table 3 does not specify clearly how the pseudo label weights are when no similarity function is used.\n+ Sec 4.5 fails to mention which table it is referring to in the text.\n+ Most of the weakly- and mixed-supervision prior works were focused on the small-scale dataset and splits. It would be very useful and more convincing if this paper could show its effectiveness in larger settings and datasets like Open Images and Objects365.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "They are almost adequately addressed. It would be good for the authors to relate the limitations of their method to some negative societal impact they may have.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "MH3L6BP9JE", "reviewer": "Reviewer_ExAZ", "summary": "This paper presents a Mixed Supervised Object Detection (MSOD) framework that is built on the Fast R-CNN framework and employs Multi-Instance Learning (MIL) to deal with weakly-supervised images. It adds two novel components to the underlying framework: Mask prior and Semantic similarity. Mask prior which is a semantic activation mask (e.g., CAM) works to enhance the ability to generate robust region proposals. Semantic similarity is used to remove inprecise proposals by checking its semantic similarity with other proposals. The proposed framework yields the better accuracy than other WSOD and MSOD methods on VOC 07 dataset.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper presents novel contributions in the methodology sufficient for consideration by NeurIPS community. The manuscript is very easy to follow.\n\nMy only concern is in experiments. Comparing with other baseline methods, most baselines used VGG16, but not the proposed method. This paper only provides the accuracy with VGG16 for the distrillation version. Therefore, I can not agree with that the proposed method has a benefit in the performance. I also wonder if other baselines also adopt multi-scale testing.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I did not find any limitation or potential adverse societal impact.", "review_text": "", "extra_scores": {"ethical_concerns": "I did not find any ethical concern from this work.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers have discussed the paper and are generally positive. In particular, some of the concerns raised in the reviewer were lifted, which resulting in improving the scores. I would like to encourage the authors to incorporate the suggestions into the final version and recommend accepting the paper.
Decision: Accept (Poster)