CROSS-SUPERVISED OBJECT DETECTION

1 INTRODUCTION . Deep architectures have achieved great success in many computer vision tasks including object recognition and the closely related problem of object detection . Modern detectors , such as the Faster RCNN ( Ren et al. , 2015 ) , YOLO ( Redmon et al. , 2016 ) , and RetinaNet ( Lin et al. , 2017 ) , use the same network backbone as popular recognition models . However , even with the same backbone architectures , detection and recognition models require different types of supervision . A good detector relies heavily on precise bounding boxes and labels for each instance ( we shall refer to these as instance-level annotations ) , whereas a recognition model needs only image-level labels . Needless to say , it is more time consuming and expensive to obtain high quality bounding box annotations than class labels . As a result , current detectors are limited to a small set of categories relative to their object recognition counterparts . To address this limitation , it is natural to ask , “ Is it possible to learn detectors with only class labels ? ” This problem is commonly referred to as weakly supervised object detection ( WSOD ) . Early WSOD work ( Hoffman et al. , 2014 ) showed fair performance by directly applying recognition networks to object detection . More recently , researchers have used multiple instance learning methods ( Dietterich et al. , 1997 ) to recast WSOD as a multi-label classification problem ( Bilen & Vedaldi , 2016 ) . However , these weakly supervised detectors perform poorly at localization . Most WSOD experiments have been conducted on the ILSVRC ( Russakovsky et al. , 2015 ) data set , in which images have only a single object , or on the PASCAL VOC ( Everingham et al. , 2010 ) data set , which has only 20 categories . The simplicity of these data sets limits the number and types of distractors in an image , making localization substantially easier . Learning from only class labels , it is challenging to detect objects at different scales in an image that contains many distractors . In particular , as shown in our experiments , weakly supervised object detectors do not work well in complex multi-object scenes , such as the COCO dataset ( Lin et al. , 2014 ) . To address this challenge , we focus on a form of learning in which the localization of classes with only object labels ( weakly labeled classes ) can benefit from other classes that have ground truth bounding boxes ( fully labeled classes ) . We refer to this interesting learning paradigm as crosssupervised object detection ( CSOD ) . While several works ( Hoffman et al. , 2014 ; Tang et al. , 2016 ; Yang et al. , 2019a ; Redmon & Farhadi , 2017 ) have explored this problem before , they still have the same limitation as the WSOD work we mentioned above . Those cross-supervised object detectors work under simplified scenarios ( e.g. , ILSVRC data set ) where images contain single objects and are object-centered . They struggle to learn under more complex and realistic scenarios , where there are multiple objects from potentially very different classes , and objects could be small and appear anywhere in the images . In this work , we show that by doing multi-task learning on both weaklysupervised base classes and fully-supervised novel classes , our model is able to learn a good detector under the CSOD setting . More formally , we define CSOD as follows . At training time , we are given 1 ) images contain objects from both base and novel classes , 2 ) both class labels and ground truth bounding boxes for base objects , and 3 ) only class labels for novel objects . Our goal is to detect novel objects . In CSOD , base classes and novel classes are disjoint . Thus , it can be seen as performing fullysupervised detection on the base classes and weakly supervised detection on the novel classes . It has similarities to both transfer learning and semi-supervised learning , since it transfer knowledge from base class to novel class and have more information about some instances than other instances . However , CSOD represents a distinct and novel paradigm for learning . The current weakly-supervised method has several drawbacks to learn from a multi objects image . As shown in Fig . 1 , a weakly supervised object detector tends to detect only the most discriminating part of novel objects instead of the whole object . Notice how only the head of the person , and not the whole body , is detected . Another issue is that the localizer for one object ( e.g. , the horse ) may be confused by the occurrence of another object , such as the person on the horse . This example illustrates the gap between detection and recognition : without ground truth bounding boxes , the detector acts like a standard recognition model – focusing on discriminating rather than detecting . In this paper , we explore two major mechanisms for improving on this . Our first mechanism is unifying detection and recognition . Using the same network backbone architecture , recognition and detection can be seen as image-level classification and region-level classification respectively , suggesting a strong relation between them . In particular , it suggests a shared training framework in which the same backbone is used with different heads for detection and recognition . Thus , we combine a detection head learned from ground truth bounding boxes , and a recognition head learned in a weakly supervised fashion from class labels . Unlike a traditional recognition head , our recognition head produces a class score for multiple proposals and is capable of detecting objects . The second mechanism is learning a spatial correlation module to reduce the gap between detection and recognition . It takes several high-confidence bounding boxes produced by the recognition head as input , and learns to regress ground truth bounding boxes . By combining these mechanisms together , our model outperforms all previous models when all novel objects are weakly labeled . In summary , our contributions are three-fold . First , we define a new task—cross-supervised object detection , which enables us to leverage knowledge from fully labeled base categories to help learn a robust detector from novel object class labels only . Second , we propose a unified framework in which two heads are learned from class labels and detection labels respectively , along with a spatial correlation module bridging the gap between recognition and detection . Third , we significantly outperform existing methods ( Zhang et al . ( 2018a ) ; Tang et al . ( 2017 ; 2018 ) ) on PASCAL VOC and COCO , suggesting that CSOD could be a promising approach for expanding object detection to a much larger number of categories . 2 RELATED WORK . Weakly supervised object detection . WSOD ( Kosugi et al . ( 2019 ) ; Zeng et al . ( 2019 ) ; Yang et al . ( 2019b ) ; Wan et al . ( 2019 ) ; Arun et al . ( 2019 ) ; Wan et al . ( 2018 ) ; Zhang et al . ( 2018b ) ; Ren et al . ( 2020 ) ; Zhang et al . ( 2018c ) ; Li et al . ( 2019 ) ; Gao et al . ( 2019b ) ; Kosugi et al . ( 2019 ) ) attempts to learn a detector with only image category labels . Most of these methods adopt the idea of Multiple Instance Learning ( Dietterich et al . ( 1997 ) ) to recast WSOD as a multi-label classification task . Bilen & Vedaldi ( 2016 ) propose an end-to-end network by modifying a classifier to operate at the level of image regions , serving as a region selector and a classifier simultaneously . Tang et al . ( 2017 ) and Tang et al . ( 2018 ) find that several iterations of online refinement based on the outputs of previous iterations boosts performance . Wei et al . ( 2018 ) and Diba et al . ( 2017 ) use semantic segmentation based on class activation maps ( Zhou et al . ( 2016 ) ) to help generate tight bounding boxes . However , WSOD methods tend to focus on the most discriminating part of an object and are prone to distractions from co-occurring objects . Detecting a part of the object or distractors represents convergence to a local optimum . Thus , their performance depends heavily on initialization . In comparison , our proposed cross-supervised object detector alleviates the issue of getting trapped in a local optimum by leveraging knowledge learned from fully labeled base categories . Cross-supervised object detection . There are several previous works using both image-level and instance-level annotations . Kuen et al . ( 2019 ) learned a parameter transferring function between a classifier and a detector , enabling an image-based classification network to be adapted to a regionbased classification network . Hoffman et al . ( 2014 ) and Tang et al . ( 2016 ) propose methods of adaptation for knowledge transfer from classification features to detection features . Uijlings et al . ( 2018 ) use a proposal generator trained on base classes to transfer knowledge by leveraging a MIL framework , organized in a semantic hierarchy . Hoffman et al . ( 2015 ) design a three-step framework to learn a feature representation from weakly supervised classes and strongly supervised classes jointly . However , these methods can only perform object localization in single object scenes such as ILSVRC , whereas our method can perform object detection in complex multi-object scenes as well , e.g . COCO . Also , it is worth noting that we are doing multi-task learning , which means that we jointly learn from base and novel classes . In comparison , some works ( Uijlings et al. , 2018 ) are doing transfer learning . They first learn a model on base classes and then transfer and fine-tune the model on novel classes . Gao et al . ( 2019a ) use a few instance-level labels and a large scale of image-level labels for each category in a training-mining framework , which is referred to as semisupervised detection . Zhang et al . ( 2018a ) propose a framework named MSD that learn objectness on base categories and use it to reject distractors when learning novel objects . In comparison , our spatial correlation module not only learns objectness , but also refines coarse bounding boxes . Further , our model learns from both base and novel classes instead of only novel classes . 3 CROSS-SUPERVISED OBJECT DETECTION . CSOD requires us to learn from instance-level annotations ( detection labels ) and image-level annotations ( recognition labels ) . In this section , we explain the unification of detection and recognition and introduce our framework . In the next section , we describe our novel spatial correlation module . 3.1 UNIFYING DETECTION AND RECOGNITION . How to learn a detector from both instance-level and image-level annotations ? Since detection and recognition can be seen as region-level and image-level classification respectively , a natural choice is to design a unified framework that combines a detection head and a recognition head that can learn from image-level and instance-level annotations respectively . Here we exploit several baselines to unify the detection and recognition head . ( 1 ) Finetune . We first learn through the detection head on base classes with fully labeled samples . Then , we finetune our model using the recognition head on novel classes with only class labels . ( 2 ) Two Head . We simultaneously learn the detection and recognition head on base and novel classes , respectively . The weights of the backbones are updated using the loss backpropagated from both heads jointly . ( 3 ) Two head + . Instead of learning only on novel classes , we learn the recognition head from class labels of both base and novel classes whereas the recognition head remain the same . ( 4 ) Two Branch . Instead of having two shared fully-connected layers after RoI pooling layer ( see Fig . 2 ) , we make these two fully-connected layers seperated , allowing the detection and recognition head to have separate unshared pair of fully-connected layers each . Everything else is the same as the Two Head baseline . Experiments are conducted to compare these baselines in § 5.1 and § 5.2 . Our proposed model is based on Two Head . We will discuss the details in § 3.2 . The connection between the recognition and detection head . The baselines mentioned above only use the recognition head to detect novel objects , ignoring the fact that a detection head can play the same role even better . A majority of WSOD methods ( Tang et al . ( 2017 ) ; Wan et al . ( 2019 ) ; Wei et al . ( 2018 ) ) find that re-train a new detector taking the top-scoring bounding boxes from a weakly supervised object detector as ground truth marginally improve the performance . Even with coarse and noisy pseudo bounding boxes , a standard object detector produces better detection results than a weakly supervised object detector . Keeping this hypothesis in mind , we introduce a guidance from the recognition head to the detection head . For each of the novel categories existing in a training sample , the recognition head outputs the top-scoring bounding box , which are then used by the detection head as supervision in that sample .

Summary: This paper defines cross-supervised object detection which learns a detector from both image-level and instance-level annotations. It proposes a unified framework along with a spatial correlation module for the task. The spatial correlation module is used for transfer mapping information from base categories to novel categories. It conducts experiments on the PASCAL VOC dataset and COCO dataset, demonstrating the effectiveness.