Year: 2018

Abstract:
Current Convolutional Neural Network (CNN)-based object detection models adopt strictly feedforward inference to predict the final detection results. However, the widely used one-way inference is agnostic to the global image context and the interplay between input image and task semantics. In this work, we present a general technique to improve off-the-shelf CNN-based object detection models in the inference stage without re-training, architecture modification or ground-truth requirements. We propose an iterative, bottom-up and top-down inference mechanism, which is named conscious inference, as it is inspired by prevalent models for human consciousness with top-down guidance and temporal persistence. While the downstream pass accumulates category-specific evidence over time, it subsequently affects the proposal calculation and the final detection. Feature activations are updated in line with no additional memory cost. Our approach advances the state of the art using popular detection models (Faster-RCNN, YOLOv2, YOLOv3) on 2D object detection and 6D object pose estimation.

text: Current Convolutional Neural Network (CNN)-based object detection models adopt strictly feedforward inference to predict the final detection results. However, the widely used one-way inference is agnostic to the global image context and the interplay between input image and task semantics. In this work, we present a general technique to improve off-the-shelf CNN-based object detection models in the inference stage without re-training, architecture modification or ground-truth requirements. We propose an iterative, bottom-up and top-down inference mechanism, which is named conscious inference, as it is inspired by prevalent models for human consciousness with top-down guidance and temporal persistence. While the downstream pass accumulates category-specific evidence over time, it subsequently affects the proposal calculation and the final detection. Feature activations are updated in line with no additional memory cost. Our approach advances the state of the art using popular detection models (Faster-RCNN, YOLOv2, YOLOv3) on 2D object detection and 6D object pose estimation.

heading: 1 INTRODUCTION
text: The goal of object detection is to localize every instance from a set of predetermined categories within any given image. In recent years a large number of works have advanced visual object detection (Girshick et al. (2014); Girshick (2015); Ren et al. (2015); Redmon et al. (2016); Liu et al. (2016)), while building on the success of Convolutional Neural Networks (CNNs) as rich feature extractors. However, despite the impressive performance of the existing detectors in the datasets that they are trained on, their generalization power typically suffers when they are deployed on unseen data. The largely shifted data distribution caused by various external factors, such as camera distance, lighting conditions, background variation etc, tremendously degrades the performance. Besides, even for objects of the same class, the intra-class variability is a hard problem to handle.
Such shifted data distribution problem does not only exist in object detection, but it also appears in many other computer vision tasks including segmentation Sankaranarayanan et al. (2017) and visual question answering Wang et al. (2018). Recent approaches tackle this issue by predicting saliency (Stone et al. (2017); Sankaranarayanan et al. (2017); Wang et al. (2018)). Stone et al. (2017) require extra mask supervision during training in order to learn to mask out the irrelevant to the class of interest regions at test time. Wang et al. (2018) rely on ground-truth information either for a subset of classes or for an auxiliary task (visual question answering). Unlike these works, our method requires no additional prior information. Instead, it utilizes the models category-specific posterior confidence to highlight the pixels of positive evidence for each detected object and suppress the background. In line with our strategy, Sankaranarayanan et al. (2017) use structural perturbations from pixel-wise predictions in order to utilize context and improve semantic segmentation. To the best of our knowledge, this work is the first online inference algorithm to address the shifted testing data distribution problem in object detection.
Unlike classification tasks which only rely on the class-specific feedback, an equally important factor in object detection is the bounding box (bbox) estimation for each prediction. Our self-correcting algorithm is designed to guide inference via category-specific, generic object and localization evidence using corresponding losses. Other recent methods (Jiang et al. (2018); Pirinen & Sminchisescu (2018); Rao et al. (2018); Xie et al. (2018)) improve baseline two-stage object detectors by introducing an assistant network, typically instantiated as a reinforcement learning agent, which demands parameter learning during training. Our approach is directly applicable to CNN-based
detectors without extra layer training, and can be used for both one-stage and two-stage frameworks.
In this work, we design a general online algorithm that improves off-the-shelf performance of pre-trained CNN-based detectors at inference time without re-training, architecture modification or any ground-truth requirements. An iterative detection algorithm is proposed to utilize the positive evidence provided by the top-down feedback flow for input refinement. The updated feature activation is then fed to the same network again to start a new round of inference. While the downstream pass accumulates category-specific evidence over time, it subsequently affects the proposal calculation and the final detection. Since feature activations are updated in line and the network parameters are fixed, there is no additional memory cost. Our method achieves significant improvement for different state-of-the-art object detectors (Faster-RCNN(Ren et al. (2015)), YOLOv2(Tekin et al. (2018)), YOLOv3(Redmon & Farhadi (2018))) in two different computer vision tasks, that is 2D object detection and 6D object pose estimation.
Our proposed algorithm is inspired by well-founded theories for human brain consciousness of which two main aspects are captured by our algorithm, the temporal persistence in human perception and top-down feedback signal. As illustrated by the Global Neuronal Workspace theory proposed by Dehaene (2014), the consciousness is the global sharing of information within the human brain. To achieve this state of global ignition, both temporal and top-down signals are critical drives. The concept of Guided Search as an attention mechanism which is guided by the output of earlier processes has been well-established in pre-CNN literature (Wolfe et al. (1989); Tsotsos et al. (1995)). Our proposed guided, iterative inference algorithm is designed to model both temporal persistence and top-down guidance and it is therefore termed as conscious inference.

heading: 2 RELATED WORK
text: Our work is closely related to CNN-based object detection methods, refinement techniques for existing object detectors, self-corrective CNN techniques and other areas.

heading: 2.1 CNN-BASED OBJECT DETECTOR
text: Recently, CNN-based object detectors have achieved overwhelming success to dramatically improve the state of the art in detection. A series of region proposal-based detectors (R-CNN Girshick et al. (2014), Fast R-CNN Girshick (2015), Faster R-CNN Ren et al. (2015), etc) are designed to develop and accelerate detection by sharing CNN features and combining CNN-based Region Proposal Network (RPN), respectively. Since these detectors exhibit a two-stage propose-refine pipeline, the detection accuracy is promising but they suffer from high computation burden. Therefore, several single-shot one-stage detectors Lin et al. (2018); Liu et al. (2016); Redmon & Farhadi (2018) have been proposed which aim to achieve real-time detection by utilizing an anchor-refine pipeline. For both families of detectors, the final detection results are obtained by a bottom-up one-way inference process. However, such one-way inference is agnostic to the global image context.

heading: 2.2 OBJECT DETECTOR REFINEMENT
text: Besides designing novel object detection network architectures, several works focus on how to further improve the performance of the existing detectors with no or minor architecture modification. By modifying the standard cross-entropy gradient, Rao et al. (2018) proposed a simple yet effective method to learn globally optimized detector for object detection based on scores and bounding boxes. In that case no modification in the network architecture is needed. Pirinen & Sminchisescu (2018) proposed a deep reinforcement learning-based RPN which replaces the greedy RoI selection process with a sequential attention mechanism trained via reinforcement learning. Jiang et al. (2018) designed a side network, called IoU-Net, to predict the IoU between each detected bounding box and the matched ground-truth. The non-maximum suppression (NMS) procedure is improved by preserving accurately localized bounding boxes according to the obtained localization confidence. Cai & Vasconcelos (2018) unwrap Faster R-CNN to a sequence of detectors trained with progressively increasing IoU thresholds. The detectors are trained stage by stage, which adds parameters and training overhead linearly to the number of stages. The aforementioned detector refinement methods are not only limited by one-way inference but they also add extra training cost.

heading: 2.3 SELF-CORRECTIVE BEHAVIOR IN CNN
text: In recent years, works exploiting the capability of CNNs to improve their off-the-shelf performance without re-training or additional data have attracted more and more attention. Sankaranarayanan et al. (2017) proposed a self-corrective mechanism for semantic segmentation. The structural perturbation generated by computing the error between output prediction and pseudo ground-truth is injected to the input image, expecting the updated prediction from the perturbed input to be improved due to the context. Carreira et al. (2016) designed an error feedback layer to iteratively refine the human pose estimation. Such self-correction algorithm is different from ours since our proposed top-down conscious feedback is totally parameter-free without any prior training needed or any model modification. Wang et al. (2018) proposed an inference procedure to iteratively update the feature maps, targeting to improve the predictions for unknown classes when partial evidence is available. However, it uses ground-truth for a subset of known classes or for the predictions of an auxiliary task, which is a very restrictive requirement in practice during inference. Instead, our inference scheme needs no ground-truth information. Additionally, these self-correction methods only utilize the category-specific feedback signal for error correction, which is not sufficient for object detection task. For our inference method, different kinds of perturbations are generated specifically for improving both recognition and localization accuracy.

heading: 3 CONSCIOUS INFERENCE FOR OBJECT DETECTION

heading: 3.1 ONE-WAY INFERENCE FOR OBJECT DETECTION
text: We start by reviewing the general one-way inference process of traditional object detectors (Fast/Faster R-CNN, YOLO, SSD, etc) briefly. Let I ∈ RW∗H∗C represent the input image for detection, B ∈ RD∗k is the k ground-truth object bboxes. In order to train a baseline detector network with parameter Θ(.), a loss function L(Θ(.), I,B) is usually optimized. CNN training is out of the scope of this work so we assume the network Θ(.) has already been well-trained and remains unaltered at inference stage. During testing, the one-way inference gives them detection predictions as Bpred = Θ(I) ∈ RD∗m, that each prediction Bi ∈ RD contains the bbox localization, objectness score and class score information. Finally the non-maximum suppression (NMS) is performed to obtain the final detection results Bnmspred ⊆ Bpred ∈ RD∗n, n ≤ m. The one-way inference result Bnmspred is compared with the ground-truth B to quantitatively evaluate the detection performance.

heading: 3.2 CONSCIOUS INFERENCE (CI) FOR OBJECT DETECTION
text: To facilitate the understanding, in the this section, we present our proposed conscious inference algorithm using a recent end-to-end one-stage object detector, YOLOv3 Redmon & Farhadi (2018), as a baseline. This model is a new version of the classic one-shot object detector Redmon et al. (2016). Assuming the updated input after the (r − 1)th round of conscious inference is Ir−1, then the detection prediction of input Ir−1 is Brpred. Each dimension of Brpred is represented by bbox Bi = [bih, b i w, h i, wi, oi, ci1, c i 2, ..., c i l] ∈ RD, where {bih, biw, hi, wi} are 4 bounding box offsets, oi is the objectness score and {ci1, ci2, ..., cil} are l class prediction scores. Since the final detection results after NMS Bnms,rpred ⊆ Brpred are the most reliable detection predictions, they are further used to filter the ROIs 1 so that the selected ROI candidates Brroi ⊆ Brpred have the following properties: (1) high IoU score over a threshold λiou (they are highly overlapped with Bnms,rpred ); (2) high objectness score over a threshold λobj (with high probability, the selected ROI contains objects). Therefore, for one ROI Biroi in Brroi extracted by B∗ = [b∗h, b∗w, h∗, w∗, o∗, c∗1, c∗2, ..., c∗l ] ∈ B nms,r pred , extending Sankaranarayanan et al. (2017) to form a three-fold guided signal, three pseudo ground-truths are generated respectively as:
1There is no concept of “ROI” in one-stage detectors without RPN, but we use the term “ROI” here to represent the detection predictions before NMS, which in practice are based on network stride and anchors.
Br,ibox = [b ∗ h, b ∗ w, h ∗, w∗︸ ︷︷ ︸ bbox , 0︸︷︷︸ objectness , 0, 0, ..., 0︸ ︷︷ ︸ class ] Br,iobj = [0, 0, 0, 0︸ ︷︷ ︸ bbox , 1︸︷︷︸ objectness , 0, 0, ..., 0︸ ︷︷ ︸ class ] Br,icls = [0, 0, 0, 0︸ ︷︷ ︸ bbox , 0︸︷︷︸ objectness , 0, 0, ..., 1︸︷︷︸ i-th cls
, .., 0︸ ︷︷ ︸ class ]
(1)
Next, the rth conscious feedback pass will generate three kinds of perturbations respectively as: P rbox = · step ( 5Ir−1L(Θ(.), Ir−1,Brbox) ) P robj = · step ( 5Ir−1L(Θ(.), Ir−1,Brobj)
) P rcls = · step ( 5Ir−1L(Θ(.), Ir−1,Brcls)
) (2) where is the weighting parameter, step(.) is the step activation function, 5Ir−1L() is the gradient map w.r.t input layer, P rbox, P r obj and P r cls are the bounding box, objectness and class-specific perturbations, respectively. Based on our experimental observation, step(.) performs better than the signum activation function used in Sankaranarayanan et al. (2017).
By updating Ir−1 as Ir = Ir−1 + (P rboc + P r obj + P r cls), the (r + 1)th conscious inference round is performed: Br+1pred = Θ(I r) (3)
The overall conscious inference process is illustrated in Fig. 1. As we claimed, our proposed conscious inference algorithm is a general scheme that is applicable to any existing CNN-based object detectors. There is no requirement for global average pooling layer as in Zhou et al. (2016). We next briefly describe how to implement our method to other baseline detection-based networks.
CI For Two-Stage Detector: The conscious inference algorithm is also implemented on top of a classic two-stage object detector, Faster R-CNN Ren et al. (2015). By design of the RPN, perturbations P robj and P r box are generated from the RPNcls and RPNbbox layers and added to the RPNconv1.
As for the P rcls, the RCNNcls layer predicts the class score of each B i from RCNNbbox. Thus, following the same way used for YOLOv3, a perturbation P rcls can be obtained and added to the base feature map (the output of the backbone network). Compared with the implementation on YOLOv3, our Faster R-CNN conscious inference version is more efficient since the conscious feedback pass goes less deep.
CI For Pose Estimator: A state of the art object pose estimator Tekin et al. (2018) is integrated to benchmark conscious inference in a scenario that naturally extends the 2D detection paradigm to a richer 6D representation. A YOLOv2-like end-to-end network with a modified regression output layer and a PnP algorithm to fit the pose estimation requirement is proposed in Tekin et al. (2018) to jointly detect the object as well as estimate its pose. Similar as in Eqn. 2, P rbox, P r obj and P r cls can be readily generated following the same manner.

heading: 4 EXPERIMENTS
text: We validate the effectiveness and efficiency of conscious inference on different object detection baselines and tasks: Sec. 4.1 shows the results of the one-stage detector YOLOv3, Sec. 4.2 presents the results of the two-stage detector Faster R-CNN and Sec. 4.3 demonstrates the performance on a 6D pose estimator. All the experiments are conducted on a single NVIDIA TITAN Xp GPU.

heading: 4.1 EXPERIMENTS ON YOLOV3 FOR 2D OBJECT DETECTION
text: Dataset and Evaluation. The experiment is conducted on MS-COCO Lin et al. (2014), where the 5k subset of validation (minival) images are tested. For evaluation, the standard COCO-style Average Precision (AP) across IoU thresholds from 0.5 to 0.95 with an interval of 0.05 is adopted.
Experimental Setting. The pre-trained YOLOv3 model from Redmon & Farhadi (2018) and the exact same algorithm parameter setting are adopted in our experiments without any modification. For our conscious inference algorithm, we set = 0.004, λobj = 0.5 and λiou = 0.9. Unless otherwise stated, the result from the first conscious iteration is reported due to most favorable performance and time efficiency trade-off. Results from more iteration rounds are also reported in Table 1 and Fig. 4.
Experimental Results and Ablation Study. The results on COCO minival are reported in Table 1. Largest improvement is achieved after one iteration, while more inference rounds have diminishing returns. Several ablation studies are also conducted. (1) The influence from different perturbations can be compared in Table 2. As it can be seen, Pbox performs well for refining the bbox of originally detected object (improvement on high IoU); Pobj is able to discover more missing objects (large improvement on AP);Pcls focuses more on correcting the wrong classification of existing detections. (2) Some parameter search experiments are conducted and shown in Fig. 3. With the increase of , the performance keeps raising at first then turn to decrease since a large will over-modify the original input. Larger λiou is able to filter more reliable and confident ROI candidates so better performance is achieved. If we keep raising λobj , some reliable ROI candidates will be mistakenly eliminated causing a worse result. (3) Different feedback perturbation strategies are compared in Fig. 4. Besides the quantitative results, some visualization results are presented in Fig. 2 and Fig. 5.
Table 3 shows the trade-off between efficiency and effectiveness of our algorithm by truncating the back-propagation pass and injecting the generated perturbations into different layers. The deeper the feedback signal goes, the better the improvement is while the computation increases. The time overhead is linear to the number of layers that the algorithm traverses during sequential top-down and bottom-up passes. Next we show an efficient implementation of our method for Faster R-CNN.

heading: 4.2 EXPERIMENTS ON FASTER R-CNN FOR 2D OBJECT DETECTION
text: Dataset and Evaluation. Besides the MS-COCO minival evaluated in Sec. 4.1, PASCAL VOC 2007 Everingham et al. (2010) dataset is also tested. There are 5k trainval images and 5k test images over 20 object categories. Compared with MS-COCO which often contains multiple small objects in one image, PASCAL VOC 2007 focuses more on large objects, so the accuracy of predicted bounding box is even more crucial here. Like COCO-style evaluation metric, the mean Average Precision (mAP) over different IoU thresholds is tested.
Experimental Setting. We use a pre-trained Faster R-CNN model (Res101 backbone) from a pytorch implementation 2, which achieves comparable performance against (Ren et al., 2015). For our
inference algorithm, we set = 0.020, λobj = 0.9 and λiou = 0.4. Unless otherwise stated, the result from the first iteration is reported.
Experimental Results. Tables 4 and 5 and Fig. 6 report the detection results on COCO and PASCAL VOC 2007 datasets respectively. Conscious inference improves the performance under various evaluation criteria, especially for high IoU and large objects. However, the gain is not as large as with YOLOv3, which can be partially attributed to the shallower conscious feedback pass.

heading: 4.3 EXPERIMENTS ON YOLOV2-6D FOR 6D OBJECT POSE PREDICTION
text: Dataset and Evaluation. The extremely challenging multi-object detection and pose estimation dataset, OCCLUSION Brachmann et al. (2014), is evaluated in our experiment. As its name suggests, several objects in the images are severely occluded due to scene clutter, which makes pose
estimation extremely challenging. The widely-used 2D reprojection error Brachmann et al. (2016) under various thresholds is adopted to evaluate 6D pose estimation accuracy.
Experimental Setting. The pre-trained YOLOv2-6D pose estimation model from Tekin et al. (2018) is used. For all the 13 categories, we follow the same experimental setting as in Tekin et al. (2018), where 7 of them (ape, can, cat, driller, duck, glue, holepuncher) are used for testing.
Experimental Results. The comparison results are shown in Table 6. For each grid cell in the table, the first number is the baseline result and the second number is ours. As can be seen, for the most strict evaluation criterion, Acc5px, our conscious inference is able to improve the baseline performance by a large margin. The overall performance of all categories is improved. Some visualization improvement results are shown in Fig. 7.

heading: 5 CONCLUSION
text: In this paper, we propose a guided, iterative inference algorithm, which can be applied on general CNN-based object detectors at inference stage. The proposed approach does not involve any model modification, re-training or ground-truth requirements. The term “conscious” is inspired by Dehaene (2014) since our method models two important traits identified in human cognition: top-down feedback and temporal persistence. Experiments based on different state-of-the-art object detectors show consistent improvement in diverse detection tasks. Our algorithm has no memory overhead (as opposed to strong detection refinement frameworks Cai & Vasconcelos (2018)), while the added computation is linear to the number of traversed layers. The trade-off between extra computation and performance gain is an important factor to consider based on the application scenario. Empirical results on this trade-off using our straight-forward implementation are presented in Table 3.

Review summaries:
The paper proposes a method called cautious inference to improve inference accuracy for object detection models. The main idea is inspired by the previous work of Guided Perturbations, which is applied to fully convolutional networks to improve the segmentation/accuracy accuracy purely during inference time.  The original idea is to use the predicted labels of the network as pseudo ground truths (after making the predictions to be a one-hot vector), and then back propagate the error signals to the network input to get the gradients. And finally the gradients are added back to the original inputs to perform another round of prediction. Here the inputs can be either the original image, or some intermediate feature maps. Experiments are shown for both 2D and 6D object detections.