Feature Intertwiner for Object Detection

1 INTRODUCTION . Classifying complex data in the high-dimensional feature space is the core of most machine learning problems , especially with the emergence of deep learning for better feature embedding ( Krizhevsky et al. , 2012 ; He et al. , 2016 ; Li et al. , 2018 ) . Previous methods address the feature representation problem by the conventional cross-entropy loss , l1 / l2 loss , or a regularization constraint on the loss term to ensure small intra-class variation and large inter-class distance ( Janocha & Czarneck , 2017 ; Liu et al. , 2017b ; Wen et al. , 2016 ; Liu et al. , 2017a ) . The goal of these works is to learn more compact representation for each class in the feature space . In this paper , we also aim for such a goal and propose a new perspective to address the problem . Our observation is that samples can be grouped into two sets in the feature space . One set is more reliable , while the other is less reliable . For example , visual samples may be less reliable due to low resolution , occlusion , adverse lighting , noise , blur , etc . The learned features for samples from the reliable set are easier to classify than those from the less reliable one . Our hypothesis is that the Meta_105_quick_7.1 vis_result_ep_0013_iter_001238 Ep190 meta_105_quick_1_roipool vis_result_ep_0013_iter_000619 set1_bs_1024_debug5 Ep149 ( a ) ( b ) small large small large ( overlapped ) person chair car book bottle cup bowl dining table donut traffic light Figure 1 : ( Zoom in for better view ) Visualization of the features in object detection using t-SNE ( van der Maaten & Hinton , 2008 ) ( a ) without and ( b ) with feature intertwiner on COCO . Each point is a sample mapped onto the low-dim manifold . reliable set can guide the feature learning of the less reliable set , in the spirit of a teacher supervising the student . We refer to this mutual learning process as a feature intertwiner . In this paper , a plug-and-play module , namely , feature intertwiner , is applied for object detection , which is the task of classifying and localizing objects in the wild . An object of lower resolution will inevitably lose detailed information during the forward pass in the network . Therefore , it is well-known that the detection accuracy drops significantly as resolutions of objects decrease . We can treat samples with high resolution ( often corresponds to large objects or region proposals ) as the reliable set and samples with low resolution ( small instances ) as the less reliable set1 . Equipped with these two ‘ prototypical ’ sets , we can apply the feature intertwiner where the reliable set is leveraged to help the feature learning of the less reliable set . Fig . 1 on the left visualizes the learned detection features before classifier2 . Without intertwiner in ( a ) , samples are more scattered and separated from each other . Note there are several samples that are far from its own class and close to the samples in other categories ( e.g. , class person in blue ) , indicating a potential mistake in classification . With the aid of feature intertwiner in ( b ) , there is barely outlier sample outside each cluster . the features in the lower resolution set approach closer to the features in the higher resolution set - achieving the goal of compact centroids in the feature space . Empirically , these two settings correspond to the baseline and intertwiner experiments ( marked in gray ) in Table 2a . The overall mAP metric increases from 32.8 % to 35.2 % , with an evident improvement of 2.6 % for small instances and a satisfying increase of 0.8 % for large counterparts . This suggests the proposed feature intertwiner could benefit both sets . Two important modifications are incorporated based on the preliminary intertwiner framework . The first is the use of class-dependent historical representative stored in a buffer . Since there might be no large sample for the same category in one mini-batch during training , the record of all previous features of a given category for large instances is recorded by a representative , of which value gets updated dynamically as training evolves . The second is an inclusion of the optimal transport ( OT ) divergence as a deluxe regularization in the feature intertwiner . OT metric maps the comparison of two distributions on high-dimensional feature space onto a lower dimension space so that it is more sensible to measure the similarity between two distributions . For the feature intertwiner , OT is capable of enforcing the less reliable set to be better aligned with the reliable set . We name the detection system equipped with the feature intertwiner as InterNet . Full code suite is available at https : //github.com/hli2020/feature intertwiner . For brevity , we put the descriptions of dividing two sets in the detection task , related work ( partial ) , background knowledge on OT theory and additional experiments in the appendix . 1We use the term ‘ large object/ ( more ) reliable/high resolution set ’ interchangeably in the following to refer to the same meaning ; likewise for the term ‘ small set/less reliable set/low-resolution set ’ . 2Only top ten categories with the most number of instances in prediction is visualized . For each category , the high-resolution objects ( reliable set ) are shown in solid color while the low-resolution instances ( less reliable set ) are shown in transparent color with dashed boundary . 2 RELATED WORK . Object detection ( Dai et al. , 2016 ; He et al. , 2017 ; Girshick , 2015 ; Ren et al. , 2015 ; Lin et al. , 2017a ; Redmon & Farhadi , 2016 ) is one of the most fundamental computer vision tasks and serves as a precursor step for other high-level problems . It is challenging due to the complexity of features in high-dimensional space ( Krizhevsky et al. , 2012 ) , the large intra-class variation and inter-class similarity across categories in benchmarks ( Deng et al. , 2009 ; Tsung-Yi Lin , 2015 ) . Thanks to the development of deep networks structure ( Simonyan & Zisserman , 2015 ; He et al. , 2016 ) and modern GPU hardware acceleration , this community has witnessed a great bloom in both performance and efficiency . The detection of small objects is addressed in concurrent literature mainly through two manners . The first is by looking at the surrounding context ( Li et al. , 2016 ; Mottaghi et al. , 2014 ) since a larger receptive filed in the surrounding region could well compensate for the information loss on a tiny instance during down-sampling in the network . The second is to adopt a multiscale strategy ( Li et al . ; Lin et al. , 2017a ; Liu et al. , 2015 ; Shrivastava et al. , 2016 ) to handle the scale problem . This is probably the most effective manner to identify objects in various sizes and can be seen in ( almost ) all detectors . Such a practice is a “ sliding-window ” version of warping features across different stages in the network , aiming for normalizing the sizes of features for objects of different resolutions . The proposed feature intertwiner is perpendicular to these two solutions . We provide a new perspective of addressing the detection of small objects - leveraging the feature guidance from high-resolution reliable samples . Designing loss functions for learning better features . The standard cross-entropy loss does not have the constraint on narrowing down the intra-class variation . Several works thereafter have focused on adding new constraints to the intra-class regularization . Liu et al . ( Liu et al. , 2017a ) proposed the angular softmax loss to learn angularly discriminative features . The new loss is expected to have smaller maximal intra-class distance than minimal inter-class distance . The center loss ( Wen et al. , 2016 ) approach specifically learns a centroid for each class and penalizes the distances between samples within the category and the center . Our feature intertwiner shares some spirit with this work in that , the proposed buffer is also in charge of collecting feature representatives for each class . A simple modification ( Liu et al. , 2017b ) to the inner product between the normalized feature input and the class centroid for the softmax loss also decreases the inner-class variation and improves the classification accuracy . Our work is from a new perspective in using the reliable set for guiding the less reliable set . 3 FEATURE INTERTWINERS FOR OBJECT DETECTION . In this paper , we adopt the Faster RCNN pipeline for object detection ( He et al. , 2016 ; 2017 ; Girshick , 2015 ) . In Faster RCNN , the input image is first fed into a backbone network to extract features ; a region proposal network ( Ren et al. , 2015 ) is built on top of it to generate potential region proposals , which are several candidate rectangular boxes that might contain objects . These region proposals vary in size . Then the features inside the region are extracted and warped into the same spatial size ( by RoI-pooling ) . Finally , the warped features are used by the subsequent CNN layers for classifying whether an object exists in the region . 3.1 FEATURE INTERTWINER OVERVIEW . We now explicitly depict how the idea of feature intertwiner could be adapted into the object detection framework . Fig . 2 describes the overall pipeline of the proposed InterNet . A network is divided into several levels based on the spatial size of feature maps . For each level l , we split the set of region proposals into two categories : one is the large-region set whose size is larger than the output size of RoI-pooling layer and another the small-region set whose size is smaller . These two sets corresponds to the reliable and less reliable sets , respectively . For details on the generation of these two sets in object detection , refer to Sec . 6.2 in the appendix . Feature map Pl at level l is fed into the RoI layer and then passed onto a make-up layer . This layer is designed to fuel back the lost information during RoI and compensate necessary details for instances of small resolution . The refined high-level semantics after this layer is robust to factors ( such as pose , lighting , appearance , etc . ) despite sample variations . It consists of one convolutional layer without ICLR version rebuttal altering the spatial size . The make-up unit is learned and optimized via the intertwiner unit , with aid of features from the large object set , which is shown in the upstream ( green ) of Fig . 2 . The feature intertwiner is essentially a data distribution measurement to evaluate divergence between two sets . For the reliable set , the input is directly the outcome of the RoI layer of the large-object feature maps Pm|l , which correspond to samples of higher level/resolution . For the less reliable set , the input is the output of the make-up layer . Both inputs are fed into a critic module to extract further representation of these two sets and provide evidence for intertwiner . The critic consists of two convolutions that transfer features to a larger channel size and reduce spatial size to one , leaving out of consideration the spatial information . A simple l2 loss can be used for comparing difference between two sets . The final loss is a combination of the standard detection losses ( Girshick , 2015 ) and the intertwiner loss across all levels . The detailed network structure of the make-up and critic module in the feature intertwiner is shown in the appendix ( Sec . 6.6 ) . There are two problems when applying the aforementioned pipeline into application . The first is that the two sets for the same category often do not occur simultaneously in one mini-batch ; the second is how to choose the input source for the reliable set , i.e. , feature map Pm|l for the large object set . We address these two points in the following sections .

Summary: The authors tackle the problem of detecting small/low resolution objects in an image. Their key idea is that detecting bigger objects is an easier task and can be used to guide the detection of smaller objects. This is done using the "Feature Intertwiner"  which consists of two branches, one for the larger objects (more reliable set that is also easier to detect) and one for the smaller objects (less reliable set). The second branch contains a make-up layer learned during training (which acts as the guidance from the more reliable set) that helps compensate details needed for detection. The authors define a class buffer that contains representative elements of object features from the reliable set for every category & scale and an intertwiner loss that computes the L2 loss between the features from the less reliable set & the class buffer. They also use an Optimal Transport procedure with a Sinkhorn divergence loss between object features from both sets. The overall loss of the system is now a sum of the detection loss, the intertwiner loss and the optimal transport loss. They evaluate their model on the COCO Object detection challenge showing state-of-the-art performance. They also provide thorough ablation analysis of various design choices. The qualitative result in Fig.1 showing well clustered features for both high & low resolution objects via t-SNE is a nice touch.