Mixed Supervised Object Detection by Transferring Mask Prior and Semantic Similarity

1 Introduction . With the ubiquitous application of convolutional neural networks and the release of large-scale fully-annotated detection benchmark datasets ( e.g. , MS COCO [ 27 ] , ILSVRC Detection [ 32 ] , and PASCAL VOC [ 10 ] ) , object detection has achieved significant advances in recent years [ 14 , 29 , 41 ] . However , Fully Supervised Object Detection ( FSOD ) requires massive training images with precise bounding boxes and box-level category labels . Acquiring such full annotations is time-consuming and labor-intensive , which limits the application of FSOD in real-world scenarios . In contrast , weak annotations like image-level category labels are much easier to acquire ( e.g. , crawl from public websites ) . Thus , plenty of Weakly Supervised Object Detection ( WSOD ) methods [ 22 , 30 , 15 ] have been proposed to bridge the gap between full annotation and weak annotation . Due to the lack of box-level annotation , WSOD methods rely on unsupervised proposal generation strategy ( e.g. , Selective Search [ 43 ] , Edge Boxes [ 48 ] ) to extract region proposals . Based on the proposals , [ 3 , 38 , 39 ] formulated WSOD task as Multiple Instance Learning ( MIL ) problem , which treats each image as a bag or multiple bags of proposals . With known image-level labels and unknown box-level labels , we can infer the box-level label of each proposal with an MIL classifier . However , WSOD methods usually only localize the most discriminative region and confuse objects with co-occurring distractors , so the performance gap between FSOD and WSOD is still very large . ∗Equal contribution †Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . It is worth noting that WSOD ignores the existence of fully-annotated object detection datasets . By taking this into consideration , another learning paradigm using mixed supervision , i.e. , full annotations for a set of categories and weak annotations for another set of categories , has been explored by [ 18 , 45 , 5 , 26 ] . This learning paradigm has inconsistent names like mixed/crosssupervised object detection in previous literature [ 26 , 5 ] . However , “ mixed/cross-supervised '' does not emphasize cross-category transfer and could be easily confused with “ semi/omni-supervised '' . Inspired by weak-shot learning [ 4 ] , which transfers knowledge from fully-annotated base categories to weakly-annotated novel categories , we refer to this learning paradigm as Weak-SHot Object Detection ( WSHOD ) . Assuming the existence of an off-the-shelf dataset with fully-annotated base categories ( precise bounding boxes and box-level labels ) , with the goal to detect the objects of novel categories , we can collect weakly-annotated data ( only image-level labels ) for novel categories . Fully-annotated base categories and weakly-annotated novel categories have no overlap . We refer to the dataset with base ( resp. , novel ) categories as source ( resp. , target ) dataset , since our goal is utilizing base categories to help improve the detection performance on novel categories . Under this paradigm , the key problem is what and how to transfer from fully-annotated base categories to weakly-annotated novel categories . Existing works [ 26 , 45 ] still follow the typical framework of WSOD , that is , feeding proposals into an MIL classifier . Due to the availability of box-level annotations of base categories , they train a CNN backbone with a region proposal network ( RPN ) to generate proposals . Since the objects of different categories share many common characteristics ( e.g. , closed contour ) , objectness is supposed to be transferrable across different categories . Therefore , the trained network could also generate proposals for novel categories , which will be sent to the MIL classifier . In this paper , beyond class-agnostic objectness , we consider another two targets to be transferred : mask prior and semantic similarity . Since both base categories and novel categories have image-level labels , we can obtain the coarse semantic masks ( e.g. , CAM [ 46 ] ) for each image . We conjecture that the coarse semantic mask could provide strong guidance for detecting objects , so we attempt to integrate coarse semantic mask into our CNN backbone . In particular , we employ the image-level labels to train a classifier to derive CAM [ 46 ] , and append the derived CAM to the feature map in CNN backbone . Because CAM provides the rough probability that each pixel belongs to a certain category , such mask prior information combined with the feature map could greatly enhance the ability to locate and identify candidate bounding boxes . By saying “ transfer mask prior '' , we mean that the ability to detect objects based on mask prior and feature map can be transferred from base categories to novel categories , which will help detect the objects of novel categories . Besides , following the iterative training strategy in [ 45 ] , we progressively mine the pseudo bounding boxes of novel categories , during which the noisy bounding boxes are filtered out . The pseudo bounding boxes with correct ( resp. , incorrect ) pseudo labels assigned by the MIL classifier are referred to as inliers ( resp. , outliers ) . We argue that semantic similarity , whether two bounding boxes belong to the same category or not , is class-agnostic . Then , we attempt to transfer semantic similarity from base categories to help identify the outliers of novel categories . Specifically , we train a similarity network based on the ground-truth bounding boxes of base categories to verify whether two bounding boxes belong to the same category . We divide the pseudo bounding boxes of novel categories into batches and the pseudo bounding boxes in each batch have the same pseudo label . Within each batch , we apply the trained similarity network to all pairs of pseudo bounding boxes to compute their semantic similarities , and then average the similarities for each one . If one batch is dominated by inliers , an inlier should be close to most other pseudo bounding boxes and its average similarity should be high . Thus , we can use average similarity to distinguish outliers and inliers . In summary , the transferred mask prior could help obtain better candidate bounding boxes , while the transferred semantic similarity could help discard the noisy pseudo bounding boxes . We conduct extensive experiments on three datasets . The results demonstrate that our proposed approach with transferred mask prior and semantic similarity can significantly boost the performance of weak-shot object detection . Considering that the key of our method is Transferring Mask prior and Similarity , we dub our method as TraMaS . Our main contributions are as follows : • We propose a novel approach named TraMaS for weak-shot object detection ( WSHOD ) . Besides class-agnostic objectness , we also propose to transfer mask prior and semantic similarity . • We integrate CAM derivation into our network , which can help generate better candidate bounding boxes with mask prior . • We employ a similarity network to learn semantic similarity , which can be used to filter out noisy pseudo bounding boxes . • Our TraMaS method outperforms all state-of-the-art WSOD and WSHOD methods on three benchmark datasets . 2 Related Work . 2.1 Weakly Supervised Object Detection . Weakly supervised object detection ( WSOD ) proposes to learn a detector with only image-level category labels . Owing to the scarcity of bounding boxes , WSOD tends to generate candidate proposals using unsupervised proposal generation strategies such as Edge Boxes [ 48 ] and Selective Search [ 43 ] . The final object results are screened from these candidate proposals . Essentially , WSOD can be cast as an image classification problem with multi-instance learning ( MIL ) classifier . [ 3 ] proposed an end-to-end weakly supervised deep detection network which employs two data streams to select proposals and perform region-level classification simultaneously . Following [ 3 ] , previous WSOD methods can be categorized into two groups according to whether using iterative refinement . On the one hand , [ 8 ] introduced weak cascaded convolutional networks by utilizing Class Activation Map ( CAM ) and weakly supervised segmentation to improve the quality of generated proposals . [ 2 ] minimized the difference between an annotation-aware conditional distribution and an annotationagnostic prediction distribution , with a dissimilarity coefficient based WSOD framework . [ 30 ] proposed a unified instance-aware and context-focused weakly supervised learning architecture to tackle the challenges of WSOD . On the other hand , some methods adopted iterative refinement . [ 38 ] unified MIL and multi-stage instance classifiers into a single deep network with online instance classifier refinement . The subsequent work [ 39 ] introduced proposal clusters to [ 38 ] . [ 21 ] explored attention regularization for WSOD training and proposes comprehensive attention self-distillation . Despite the fact that these methods have achieved promising results , because of the scarcity of ground-truth bounding boxes , the localization of objects is still far from satisfactory . In this paper , we follow the second group of methods and adopt an iterative training strategy . 2.2 Weak-shot Object Detection . Due to the lack of box-level annotations , WSOD heavily relies on the quality of candidate proposals , and the learned detector can not refine the proposals . Thanks to the off-the-shelf fully-annotated object detection datasets , weak-shot object detection ( WSHOD ) proposes to leverage the fully-annotated dataset as source dataset to improve the performance of WSOD on weakly-annotated data . Existing WSHOD methods [ 42 , 16 , 36 ] mainly focus on mining various types of knowledge in the base categories and transferring them to novel categories . Many methods [ 31 , 35 , 9 ] learn an appearance model from base categories and transfer it to novel categories . Previous results have proved that class-invariant knowledge can effectively alleviate the problem that WSOD can not accurately locate objects . For example , [ 18 ] , [ 40 ] and [ 24 ] transferred the difference between classifier and detector from base category to novel category . [ 26 ] learned class-invariant objectness from base categories with an adversarial domain classifier . With the help of learned objectness knowledge , this method can distinguish objects and distractors , and thus improves the ability to reject distractors in novel categories . In [ 45 ] , Zhong et al . employed one-class universal detector learned from base category to provide proposals for the MIL classifier , which are both used to mine pseudo ground-truth for iterative refinement . The method in [ 5 ] exploited the spatial correlation between high-confidence bounding boxes output by the MIL classifier . In this paper , besides objectness and regression ability , we additionally explore two transfer targets : mask prior and semantic similarity . Besides weak-shot object detection , weak-shot learning paradigm ( transferring knowledge from fully-annotated base categories to weakly annotated novel categories ) has also been studied in other computer vision tasks like classification [ 4 ] , semantic segmentation [ 47 ] , and instance segmentation [ 19 , 25 ] . Although transferring similarity has been investigated in [ 4 , 47 ] , transferring mask prior remains unexplored due to the uniqueness of object detection task . 3 Our Method . In this paper , our target is to improve the performance of object detection on weakly-annotated novel categories with the help of fully-annotated base categories . Under this paradigm , we have access to source dataset Ds and target dataset Dt . Ds has full annotations for base categories ( i.e. , precise bounding boxes and box-level category labels ) while Dt only has weak annotations ( image-level category labels ) for novel categories . Base categories and novel categories have no overlap . Our method is trained with a mixture of Ds and Dt , and evaluated on the test set of novel categories .

Summary: This paper studies mixed supervised object detection where some object categories are fully supervised and the rest are only labeled with class tags. It introduces a (1) weakly-supervised segmentation module and (2) a semantic similarity learning module to the existing framework to improve the performance. Experimental results on two benchmarks validated the effectiveness of the proposed framework.