Mixed Supervised Object Detection by Transferring Mask Prior and Semantic Similarity

1 Introduction . With the ubiquitous application of convolutional neural networks and the release of large-scale fully-annotated detection benchmark datasets ( e.g. , MS COCO [ 27 ] , ILSVRC Detection [ 32 ] , and PASCAL VOC [ 10 ] ) , object detection has achieved significant advances in recent years [ 14 , 29 , 41 ] . However , Fully Supervised Object Detection ( FSOD ) requires massive training images with precise bounding boxes and box-level category labels . Acquiring such full annotations is time-consuming and labor-intensive , which limits the application of FSOD in real-world scenarios . In contrast , weak annotations like image-level category labels are much easier to acquire ( e.g. , crawl from public websites ) . Thus , plenty of Weakly Supervised Object Detection ( WSOD ) methods [ 22 , 30 , 15 ] have been proposed to bridge the gap between full annotation and weak annotation . Due to the lack of box-level annotation , WSOD methods rely on unsupervised proposal generation strategy ( e.g. , Selective Search [ 43 ] , Edge Boxes [ 48 ] ) to extract region proposals . Based on the proposals , [ 3 , 38 , 39 ] formulated WSOD task as Multiple Instance Learning ( MIL ) problem , which treats each image as a bag or multiple bags of proposals . With known image-level labels and unknown box-level labels , we can infer the box-level label of each proposal with an MIL classifier . However , WSOD methods usually only localize the most discriminative region and confuse objects with co-occurring distractors , so the performance gap between FSOD and WSOD is still very large . ∗Equal contribution †Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . It is worth noting that WSOD ignores the existence of fully-annotated object detection datasets . By taking this into consideration , another learning paradigm using mixed supervision , i.e. , full annotations for a set of categories and weak annotations for another set of categories , has been explored by [ 18 , 45 , 5 , 26 ] . This learning paradigm has inconsistent names like mixed/crosssupervised object detection in previous literature [ 26 , 5 ] . However , “ mixed/cross-supervised '' does not emphasize cross-category transfer and could be easily confused with “ semi/omni-supervised '' . Inspired by weak-shot learning [ 4 ] , which transfers knowledge from fully-annotated base categories to weakly-annotated novel categories , we refer to this learning paradigm as Weak-SHot Object Detection ( WSHOD ) . Assuming the existence of an off-the-shelf dataset with fully-annotated base categories ( precise bounding boxes and box-level labels ) , with the goal to detect the objects of novel categories , we can collect weakly-annotated data ( only image-level labels ) for novel categories . Fully-annotated base categories and weakly-annotated novel categories have no overlap . We refer to the dataset with base ( resp. , novel ) categories as source ( resp. , target ) dataset , since our goal is utilizing base categories to help improve the detection performance on novel categories . Under this paradigm , the key problem is what and how to transfer from fully-annotated base categories to weakly-annotated novel categories . Existing works [ 26 , 45 ] still follow the typical framework of WSOD , that is , feeding proposals into an MIL classifier . Due to the availability of box-level annotations of base categories , they train a CNN backbone with a region proposal network ( RPN ) to generate proposals . Since the objects of different categories share many common characteristics ( e.g. , closed contour ) , objectness is supposed to be transferrable across different categories . Therefore , the trained network could also generate proposals for novel categories , which will be sent to the MIL classifier . In this paper , beyond class-agnostic objectness , we consider another two targets to be transferred : mask prior and semantic similarity . Since both base categories and novel categories have image-level labels , we can obtain the coarse semantic masks ( e.g. , CAM [ 46 ] ) for each image . We conjecture that the coarse semantic mask could provide strong guidance for detecting objects , so we attempt to integrate coarse semantic mask into our CNN backbone . In particular , we employ the image-level labels to train a classifier to derive CAM [ 46 ] , and append the derived CAM to the feature map in CNN backbone . Because CAM provides the rough probability that each pixel belongs to a certain category , such mask prior information combined with the feature map could greatly enhance the ability to locate and identify candidate bounding boxes . By saying “ transfer mask prior '' , we mean that the ability to detect objects based on mask prior and feature map can be transferred from base categories to novel categories , which will help detect the objects of novel categories . Besides , following the iterative training strategy in [ 45 ] , we progressively mine the pseudo bounding boxes of novel categories , during which the noisy bounding boxes are filtered out . The pseudo bounding boxes with correct ( resp. , incorrect ) pseudo labels assigned by the MIL classifier are referred to as inliers ( resp. , outliers ) . We argue that semantic similarity , whether two bounding boxes belong to the same category or not , is class-agnostic . Then , we attempt to transfer semantic similarity from base categories to help identify the outliers of novel categories . Specifically , we train a similarity network based on the ground-truth bounding boxes of base categories to verify whether two bounding boxes belong to the same category . We divide the pseudo bounding boxes of novel categories into batches and the pseudo bounding boxes in each batch have the same pseudo label . Within each batch , we apply the trained similarity network to all pairs of pseudo bounding boxes to compute their semantic similarities , and then average the similarities for each one . If one batch is dominated by inliers , an inlier should be close to most other pseudo bounding boxes and its average similarity should be high . Thus , we can use average similarity to distinguish outliers and inliers . In summary , the transferred mask prior could help obtain better candidate bounding boxes , while the transferred semantic similarity could help discard the noisy pseudo bounding boxes . We conduct extensive experiments on three datasets . The results demonstrate that our proposed approach with transferred mask prior and semantic similarity can significantly boost the performance of weak-shot object detection . Considering that the key of our method is Transferring Mask prior and Similarity , we dub our method as TraMaS . Our main contributions are as follows : • We propose a novel approach named TraMaS for weak-shot object detection ( WSHOD ) . Besides class-agnostic objectness , we also propose to transfer mask prior and semantic similarity . • We integrate CAM derivation into our network , which can help generate better candidate bounding boxes with mask prior . • We employ a similarity network to learn semantic similarity , which can be used to filter out noisy pseudo bounding boxes . • Our TraMaS method outperforms all state-of-the-art WSOD and WSHOD methods on three benchmark datasets . 2 Related Work . 2.1 Weakly Supervised Object Detection . Weakly supervised object detection ( WSOD ) proposes to learn a detector with only image-level category labels . Owing to the scarcity of bounding boxes , WSOD tends to generate candidate proposals using unsupervised proposal generation strategies such as Edge Boxes [ 48 ] and Selective Search [ 43 ] . The final object results are screened from these candidate proposals . Essentially , WSOD can be cast as an image classification problem with multi-instance learning ( MIL ) classifier . [ 3 ] proposed an end-to-end weakly supervised deep detection network which employs two data streams to select proposals and perform region-level classification simultaneously . Following [ 3 ] , previous WSOD methods can be categorized into two groups according to whether using iterative refinement . On the one hand , [ 8 ] introduced weak cascaded convolutional networks by utilizing Class Activation Map ( CAM ) and weakly supervised segmentation to improve the quality of generated proposals . [ 2 ] minimized the difference between an annotation-aware conditional distribution and an annotationagnostic prediction distribution , with a dissimilarity coefficient based WSOD framework . [ 30 ] proposed a unified instance-aware and context-focused weakly supervised learning architecture to tackle the challenges of WSOD . On the other hand , some methods adopted iterative refinement . [ 38 ] unified MIL and multi-stage instance classifiers into a single deep network with online instance classifier refinement . The subsequent work [ 39 ] introduced proposal clusters to [ 38 ] . [ 21 ] explored attention regularization for WSOD training and proposes comprehensive attention self-distillation . Despite the fact that these methods have achieved promising results , because of the scarcity of ground-truth bounding boxes , the localization of objects is still far from satisfactory . In this paper , we follow the second group of methods and adopt an iterative training strategy . 2.2 Weak-shot Object Detection . Due to the lack of box-level annotations , WSOD heavily relies on the quality of candidate proposals , and the learned detector can not refine the proposals . Thanks to the off-the-shelf fully-annotated object detection datasets , weak-shot object detection ( WSHOD ) proposes to leverage the fully-annotated dataset as source dataset to improve the performance of WSOD on weakly-annotated data . Existing WSHOD methods [ 42 , 16 , 36 ] mainly focus on mining various types of knowledge in the base categories and transferring them to novel categories . Many methods [ 31 , 35 , 9 ] learn an appearance model from base categories and transfer it to novel categories . Previous results have proved that class-invariant knowledge can effectively alleviate the problem that WSOD can not accurately locate objects . For example , [ 18 ] , [ 40 ] and [ 24 ] transferred the difference between classifier and detector from base category to novel category . [ 26 ] learned class-invariant objectness from base categories with an adversarial domain classifier . With the help of learned objectness knowledge , this method can distinguish objects and distractors , and thus improves the ability to reject distractors in novel categories . In [ 45 ] , Zhong et al . employed one-class universal detector learned from base category to provide proposals for the MIL classifier , which are both used to mine pseudo ground-truth for iterative refinement . The method in [ 5 ] exploited the spatial correlation between high-confidence bounding boxes output by the MIL classifier . In this paper , besides objectness and regression ability , we additionally explore two transfer targets : mask prior and semantic similarity . Besides weak-shot object detection , weak-shot learning paradigm ( transferring knowledge from fully-annotated base categories to weakly annotated novel categories ) has also been studied in other computer vision tasks like classification [ 4 ] , semantic segmentation [ 47 ] , and instance segmentation [ 19 , 25 ] . Although transferring similarity has been investigated in [ 4 , 47 ] , transferring mask prior remains unexplored due to the uniqueness of object detection task . 3 Our Method . In this paper , our target is to improve the performance of object detection on weakly-annotated novel categories with the help of fully-annotated base categories . Under this paradigm , we have access to source dataset Ds and target dataset Dt . Ds has full annotations for base categories ( i.e. , precise bounding boxes and box-level category labels ) while Dt only has weak annotations ( image-level category labels ) for novel categories . Base categories and novel categories have no overlap . Our method is trained with a mixture of Ds and Dt , and evaluated on the test set of novel categories .

Summary: The paper proposes a method to do `mixed supervision` object detection, i.e. some part of the dataset is annotated fully while some part only has weak image level annotations.  The first contribution is to use semantic mask prior in object detection network. Prior works have already used semantic masks for weakly supervised detection, but the networks were separate. The paper proposes to integrate the mask prediction networks with the object detection network and use the masks to augment features for prediction candidate bounding boxes. The hope is that this would also generalize and produce better candidate boxes for target/novel categories.  The second contribution is to transfer semantic similarity. The paper incorporates a similarity (metric learning) network in the architectures and learns it using the bounding boxes of the base categories. Then the network is used to estimate the average similarities of pseudo bounding boxes for the novel categories. With the intuition that the average distance of a correct example wrt the rest of the class examples would be lower than that for an incorrect example, they use a weighted loss for object class prediction for the bounding boxes.  The training is done in an iterative manner and results are provided on public benchmarks COCO and ILSVRC datasets as source, and Pascal VOC 2007 as the target dataset.