Conscious Inference for Object Detection

Current Convolutional Neural Network ( CNN ) -based object detection models adopt strictly feedforward inference to predict the final detection results . However , the widely used one-way inference is agnostic to the global image context and the interplay between input image and task semantics . In this work , we present a general technique to improve off-the-shelf CNN-based object detection models in the inference stage without re-training , architecture modification or ground-truth requirements . We propose an iterative , bottom-up and top-down inference mechanism , which is named conscious inference , as it is inspired by prevalent models for human consciousness with top-down guidance and temporal persistence . While the downstream pass accumulates category-specific evidence over time , it subsequently affects the proposal calculation and the final detection . Feature activations are updated in line with no additional memory cost . Our approach advances the state of the art using popular detection models ( Faster-RCNN , YOLOv2 , YOLOv3 ) on 2D object detection and 6D object pose estimation . 1 INTRODUCTION . The goal of object detection is to localize every instance from a set of predetermined categories within any given image . In recent years a large number of works have advanced visual object detection ( Girshick et al . ( 2014 ) ; Girshick ( 2015 ) ; Ren et al . ( 2015 ) ; Redmon et al . ( 2016 ) ; Liu et al . ( 2016 ) ) , while building on the success of Convolutional Neural Networks ( CNNs ) as rich feature extractors . However , despite the impressive performance of the existing detectors in the datasets that they are trained on , their generalization power typically suffers when they are deployed on unseen data . The largely shifted data distribution caused by various external factors , such as camera distance , lighting conditions , background variation etc , tremendously degrades the performance . Besides , even for objects of the same class , the intra-class variability is a hard problem to handle . Such shifted data distribution problem does not only exist in object detection , but it also appears in many other computer vision tasks including segmentation Sankaranarayanan et al . ( 2017 ) and visual question answering Wang et al . ( 2018 ) . Recent approaches tackle this issue by predicting saliency ( Stone et al . ( 2017 ) ; Sankaranarayanan et al . ( 2017 ) ; Wang et al . ( 2018 ) ) . Stone et al . ( 2017 ) require extra mask supervision during training in order to learn to mask out the irrelevant to the class of interest regions at test time . Wang et al . ( 2018 ) rely on ground-truth information either for a subset of classes or for an auxiliary task ( visual question answering ) . Unlike these works , our method requires no additional prior information . Instead , it utilizes the models category-specific posterior confidence to highlight the pixels of positive evidence for each detected object and suppress the background . In line with our strategy , Sankaranarayanan et al . ( 2017 ) use structural perturbations from pixel-wise predictions in order to utilize context and improve semantic segmentation . To the best of our knowledge , this work is the first online inference algorithm to address the shifted testing data distribution problem in object detection . Unlike classification tasks which only rely on the class-specific feedback , an equally important factor in object detection is the bounding box ( bbox ) estimation for each prediction . Our self-correcting algorithm is designed to guide inference via category-specific , generic object and localization evidence using corresponding losses . Other recent methods ( Jiang et al . ( 2018 ) ; Pirinen & Sminchisescu ( 2018 ) ; Rao et al . ( 2018 ) ; Xie et al . ( 2018 ) ) improve baseline two-stage object detectors by introducing an assistant network , typically instantiated as a reinforcement learning agent , which demands parameter learning during training . Our approach is directly applicable to CNN-based detectors without extra layer training , and can be used for both one-stage and two-stage frameworks . In this work , we design a general online algorithm that improves off-the-shelf performance of pre-trained CNN-based detectors at inference time without re-training , architecture modification or any ground-truth requirements . An iterative detection algorithm is proposed to utilize the positive evidence provided by the top-down feedback flow for input refinement . The updated feature activation is then fed to the same network again to start a new round of inference . While the downstream pass accumulates category-specific evidence over time , it subsequently affects the proposal calculation and the final detection . Since feature activations are updated in line and the network parameters are fixed , there is no additional memory cost . Our method achieves significant improvement for different state-of-the-art object detectors ( Faster-RCNN ( Ren et al . ( 2015 ) ) , YOLOv2 ( Tekin et al . ( 2018 ) ) , YOLOv3 ( Redmon & Farhadi ( 2018 ) ) ) in two different computer vision tasks , that is 2D object detection and 6D object pose estimation . Our proposed algorithm is inspired by well-founded theories for human brain consciousness of which two main aspects are captured by our algorithm , the temporal persistence in human perception and top-down feedback signal . As illustrated by the Global Neuronal Workspace theory proposed by Dehaene ( 2014 ) , the consciousness is the global sharing of information within the human brain . To achieve this state of global ignition , both temporal and top-down signals are critical drives . The concept of Guided Search as an attention mechanism which is guided by the output of earlier processes has been well-established in pre-CNN literature ( Wolfe et al . ( 1989 ) ; Tsotsos et al . ( 1995 ) ) . Our proposed guided , iterative inference algorithm is designed to model both temporal persistence and top-down guidance and it is therefore termed as conscious inference . 2 RELATED WORK . Our work is closely related to CNN-based object detection methods , refinement techniques for existing object detectors , self-corrective CNN techniques and other areas . 2.1 CNN-BASED OBJECT DETECTOR . Recently , CNN-based object detectors have achieved overwhelming success to dramatically improve the state of the art in detection . A series of region proposal-based detectors ( R-CNN Girshick et al . ( 2014 ) , Fast R-CNN Girshick ( 2015 ) , Faster R-CNN Ren et al . ( 2015 ) , etc ) are designed to develop and accelerate detection by sharing CNN features and combining CNN-based Region Proposal Network ( RPN ) , respectively . Since these detectors exhibit a two-stage propose-refine pipeline , the detection accuracy is promising but they suffer from high computation burden . Therefore , several single-shot one-stage detectors Lin et al . ( 2018 ) ; Liu et al . ( 2016 ) ; Redmon & Farhadi ( 2018 ) have been proposed which aim to achieve real-time detection by utilizing an anchor-refine pipeline . For both families of detectors , the final detection results are obtained by a bottom-up one-way inference process . However , such one-way inference is agnostic to the global image context . 2.2 OBJECT DETECTOR REFINEMENT . Besides designing novel object detection network architectures , several works focus on how to further improve the performance of the existing detectors with no or minor architecture modification . By modifying the standard cross-entropy gradient , Rao et al . ( 2018 ) proposed a simple yet effective method to learn globally optimized detector for object detection based on scores and bounding boxes . In that case no modification in the network architecture is needed . Pirinen & Sminchisescu ( 2018 ) proposed a deep reinforcement learning-based RPN which replaces the greedy RoI selection process with a sequential attention mechanism trained via reinforcement learning . Jiang et al . ( 2018 ) designed a side network , called IoU-Net , to predict the IoU between each detected bounding box and the matched ground-truth . The non-maximum suppression ( NMS ) procedure is improved by preserving accurately localized bounding boxes according to the obtained localization confidence . Cai & Vasconcelos ( 2018 ) unwrap Faster R-CNN to a sequence of detectors trained with progressively increasing IoU thresholds . The detectors are trained stage by stage , which adds parameters and training overhead linearly to the number of stages . The aforementioned detector refinement methods are not only limited by one-way inference but they also add extra training cost . 2.3 SELF-CORRECTIVE BEHAVIOR IN CNN . In recent years , works exploiting the capability of CNNs to improve their off-the-shelf performance without re-training or additional data have attracted more and more attention . Sankaranarayanan et al . ( 2017 ) proposed a self-corrective mechanism for semantic segmentation . The structural perturbation generated by computing the error between output prediction and pseudo ground-truth is injected to the input image , expecting the updated prediction from the perturbed input to be improved due to the context . Carreira et al . ( 2016 ) designed an error feedback layer to iteratively refine the human pose estimation . Such self-correction algorithm is different from ours since our proposed top-down conscious feedback is totally parameter-free without any prior training needed or any model modification . Wang et al . ( 2018 ) proposed an inference procedure to iteratively update the feature maps , targeting to improve the predictions for unknown classes when partial evidence is available . However , it uses ground-truth for a subset of known classes or for the predictions of an auxiliary task , which is a very restrictive requirement in practice during inference . Instead , our inference scheme needs no ground-truth information . Additionally , these self-correction methods only utilize the category-specific feedback signal for error correction , which is not sufficient for object detection task . For our inference method , different kinds of perturbations are generated specifically for improving both recognition and localization accuracy . 3 CONSCIOUS INFERENCE FOR OBJECT DETECTION . 3.1 ONE-WAY INFERENCE FOR OBJECT DETECTION . We start by reviewing the general one-way inference process of traditional object detectors ( Fast/Faster R-CNN , YOLO , SSD , etc ) briefly . Let I ∈ RW∗H∗C represent the input image for detection , B ∈ RD∗k is the k ground-truth object bboxes . In order to train a baseline detector network with parameter Θ ( . ) , a loss function L ( Θ ( . ) , I , B ) is usually optimized . CNN training is out of the scope of this work so we assume the network Θ ( . ) has already been well-trained and remains unaltered at inference stage . During testing , the one-way inference gives them detection predictions as Bpred = Θ ( I ) ∈ RD∗m , that each prediction Bi ∈ RD contains the bbox localization , objectness score and class score information . Finally the non-maximum suppression ( NMS ) is performed to obtain the final detection results Bnmspred ⊆ Bpred ∈ RD∗n , n ≤ m. The one-way inference result Bnmspred is compared with the ground-truth B to quantitatively evaluate the detection performance . 3.2 CONSCIOUS INFERENCE ( CI ) FOR OBJECT DETECTION . To facilitate the understanding , in the this section , we present our proposed conscious inference algorithm using a recent end-to-end one-stage object detector , YOLOv3 Redmon & Farhadi ( 2018 ) , as a baseline . This model is a new version of the classic one-shot object detector Redmon et al . ( 2016 ) . Assuming the updated input after the ( r − 1 ) th round of conscious inference is Ir−1 , then the detection prediction of input Ir−1 is Brpred . Each dimension of Brpred is represented by bbox Bi = [ bih , b i w , h i , wi , oi , ci1 , c i 2 , ... , c i l ] ∈ RD , where { bih , biw , hi , wi } are 4 bounding box offsets , oi is the objectness score and { ci1 , ci2 , ... , cil } are l class prediction scores . Since the final detection results after NMS Bnms , rpred ⊆ Brpred are the most reliable detection predictions , they are further used to filter the ROIs 1 so that the selected ROI candidates Brroi ⊆ Brpred have the following properties : ( 1 ) high IoU score over a threshold λiou ( they are highly overlapped with Bnms , rpred ) ; ( 2 ) high objectness score over a threshold λobj ( with high probability , the selected ROI contains objects ) . Therefore , for one ROI Biroi in Brroi extracted by B∗ = [ b∗h , b∗w , h∗ , w∗ , o∗ , c∗1 , c∗2 , ... , c∗l ] ∈ B nms , r pred , extending Sankaranarayanan et al . ( 2017 ) to form a three-fold guided signal , three pseudo ground-truths are generated respectively as : 1There is no concept of “ ROI ” in one-stage detectors without RPN , but we use the term “ ROI ” here to represent the detection predictions before NMS , which in practice are based on network stride and anchors . Br , ibox = [ b ∗ h , b ∗ w , h ∗ , w∗︸ ︷︷ ︸ bbox , 0︸︷︷︸ objectness , 0 , 0 , ... , 0︸ ︷︷ ︸ class ] Br , iobj = [ 0 , 0 , 0 , 0︸ ︷︷ ︸ bbox , 1︸︷︷︸ objectness , 0 , 0 , ... , 0︸ ︷︷ ︸ class ] Br , icls = [ 0 , 0 , 0 , 0︸ ︷︷ ︸ bbox , 0︸︷︷︸ objectness , 0 , 0 , ... , 1︸︷︷︸ i-th cls , .. , 0︸ ︷︷ ︸ class ] ( 1 ) Next , the rth conscious feedback pass will generate three kinds of perturbations respectively as : P rbox = · step ( 5Ir−1L ( Θ ( . ) , Ir−1 , Brbox ) ) P robj = · step ( 5Ir−1L ( Θ ( . ) , Ir−1 , Brobj ) ) P rcls = · step ( 5Ir−1L ( Θ ( . ) , Ir−1 , Brcls ) ) ( 2 ) where is the weighting parameter , step ( . ) is the step activation function , 5Ir−1L ( ) is the gradient map w.r.t input layer , P rbox , P r obj and P r cls are the bounding box , objectness and class-specific perturbations , respectively . Based on our experimental observation , step ( . ) performs better than the signum activation function used in Sankaranarayanan et al . ( 2017 ) . By updating Ir−1 as Ir = Ir−1 + ( P rboc + P r obj + P r cls ) , the ( r + 1 ) th conscious inference round is performed : Br+1pred = Θ ( I r ) ( 3 ) The overall conscious inference process is illustrated in Fig . 1 . As we claimed , our proposed conscious inference algorithm is a general scheme that is applicable to any existing CNN-based object detectors . There is no requirement for global average pooling layer as in Zhou et al . ( 2016 ) . We next briefly describe how to implement our method to other baseline detection-based networks . CI For Two-Stage Detector : The conscious inference algorithm is also implemented on top of a classic two-stage object detector , Faster R-CNN Ren et al . ( 2015 ) . By design of the RPN , perturbations P robj and P r box are generated from the RPNcls and RPNbbox layers and added to the RPNconv1 . As for the P rcls , the RCNNcls layer predicts the class score of each B i from RCNNbbox . Thus , following the same way used for YOLOv3 , a perturbation P rcls can be obtained and added to the base feature map ( the output of the backbone network ) . Compared with the implementation on YOLOv3 , our Faster R-CNN conscious inference version is more efficient since the conscious feedback pass goes less deep . CI For Pose Estimator : A state of the art object pose estimator Tekin et al . ( 2018 ) is integrated to benchmark conscious inference in a scenario that naturally extends the 2D detection paradigm to a richer 6D representation . A YOLOv2-like end-to-end network with a modified regression output layer and a PnP algorithm to fit the pose estimation requirement is proposed in Tekin et al . ( 2018 ) to jointly detect the object as well as estimate its pose . Similar as in Eqn . 2 , P rbox , P r obj and P r cls can be readily generated following the same manner .

Summary: The paper proposes a method called cautious inference to improve inference accuracy for object detection models. The main idea is inspired by the previous work of Guided Perturbations, which is applied to fully convolutional networks to improve the segmentation/accuracy accuracy purely during inference time.  The original idea is to use the predicted labels of the network as pseudo ground truths (after making the predictions to be a one-hot vector), and then back propagate the error signals to the network input to get the gradients. And finally the gradients are added back to the original inputs to perform another round of prediction. Here the inputs can be either the original image, or some intermediate feature maps. Experiments are shown for both 2D and 6D object detections.