Title: CONSCIOUS INFERENCE FOR OBJECT DETECTION Anonymous

Abstract: Current Convolutional Neural Network (CNN)-based object detection models adopt strictly feedforward inference to predict the final detection results. However, the widely used one-way inference is agnostic to the global image context and the interplay between input image and task semantics. In this work, we present a general technique to improve off-the-shelf CNN-based object detection models in the inference stage without re-training, architecture modification or ground-truth requirements. We propose an iterative, bottom-up and top-down inference mechanism, which is named conscious inference, as it is inspired by prevalent models for human consciousness with top-down guidance and temporal persistence. While the downstream pass accumulates category-specific evidence over time, it subsequently affects the proposal calculation and the final detection. Feature activations are updated in line with no additional memory cost. Our approach advances the state of the art using popular detection models (Faster-RCNN, YOLOv2, YOLOv3) on 2D object detection and 6D object pose estimation.

1. INTRODUCTION:

The goal of object detection is to localize every instance from a set of predetermined categories within any given image. In recent years a large number of works have advanced visual object detection (Girshick et al. (2014) ; Girshick (2015) ; Ren et al. (2015) ; Redmon et al. (2016) ; Liu et al. (2016) ), while building on the success of Convolutional Neural Networks (CNNs) as rich feature extractors. However, despite the impressive performance of the existing detectors in the datasets that they are trained on, their generalization power typically suffers when they are deployed on unseen data. The largely shifted data distribution caused by various external factors, such as camera distance, lighting conditions, background variation etc, tremendously degrades the performance. Besides, even for objects of the same class, the intra-class variability is a hard problem to handle. Such shifted data distribution problem does not only exist in object detection, but it also appears in many other computer vision tasks including segmentation Sankaranarayanan et al. (2017) and visual question answering Wang et al. (2018) . Recent approaches tackle this issue by predicting saliency (Stone et al. (2017) ; Sankaranarayanan et al. (2017) ; Wang et al. (2018) ). Stone et al. (2017) require extra mask supervision during training in order to learn to mask out the irrelevant to the class of interest regions at test time. Wang et al. (2018) rely on ground-truth information either for a subset of classes or for an auxiliary task (visual question answering). Unlike these works, our method requires no additional prior information. Instead, it utilizes the models category-specific posterior confidence to highlight the pixels of positive evidence for each detected object and suppress the background. In line with our strategy, Sankaranarayanan et al. (2017) use structural perturbations from pixel-wise predictions in order to utilize context and improve semantic segmentation. To the best of our knowledge, this work is the first online inference algorithm to address the shifted testing data distribution problem in object detection.

Unlike classification tasks which only rely on the class-specific feedback, an equally important factor in object detection is the bounding box (bbox) estimation for each prediction. Our self-correcting algorithm is designed to guide inference via category-specific, generic object and localization evidence using corresponding losses. Other recent methods (Jiang et al. (2018) ; Pirinen & Sminchisescu (2018) ; Rao et al. (2018) ; Xie et al. (2018) ) improve baseline two-stage object detectors by introducing an assistant network, typically instantiated as a reinforcement learning agent, which demands parameter learning during training. Our approach is directly applicable to CNN-based detectors without extra layer training, and can be used for both one-stage and two-stage frameworks.

In this work, we design a general online algorithm that improves off-the-shelf performance of pre-trained CNN-based detectors at inference time without re-training, architecture modification or any ground-truth requirements. An iterative detection algorithm is proposed to utilize the positive evidence provided by the top-down feedback flow for input refinement. The updated feature activation is then fed to the same network again to start a new round of inference. While the downstream pass accumulates category-specific evidence over time, it subsequently affects the proposal calculation and the final detection. Since feature activations are updated in line and the network parameters are fixed, there is no additional memory cost. Our method achieves significant improvement for different state-of-the-art object detectors (Faster-RCNN(Ren et al. (2015) ), YOLOv2(Tekin et al. ( 2018)), YOLOv3 (Redmon & Farhadi (2018) )) in two different computer vision tasks, that is 2D object detection and 6D object pose estimation.

Our proposed algorithm is inspired by well-founded theories for human brain consciousness of which two main aspects are captured by our algorithm, the temporal persistence in human perception and top-down feedback signal. As illustrated by the Global Neuronal Workspace theory proposed by Dehaene (2014) , the consciousness is the global sharing of information within the human brain. To achieve this state of global ignition, both temporal and top-down signals are critical drives. The concept of Guided Search as an attention mechanism which is guided by the output of earlier processes has been well-established in pre-CNN literature (Wolfe et al. (1989); Tsotsos et al. (1995) ). Our proposed guided, iterative inference algorithm is designed to model both temporal persistence and top-down guidance and it is therefore termed as conscious inference.

2. RELATED WORK:

Our work is closely related to CNN-based object detection methods, refinement techniques for existing object detectors, self-corrective CNN techniques and other areas.

2.1. CNN-BASED OBJECT DETECTOR:

Recently, CNN-based object detectors have achieved overwhelming success to dramatically improve the state of the art in detection. A series of region proposal-based detectors (R- CNN Girshick et al. (2014) , Fast R-CNN Girshick (2015) , Faster R-CNN Ren et al. (2015) , etc) are designed to develop and accelerate detection by sharing CNN features and combining CNN-based Region Proposal Network (RPN), respectively. Since these detectors exhibit a two-stage propose-refine pipeline, the detection accuracy is promising but they suffer from high computation burden. Therefore, several single-shot one-stage detectors Lin et al. (2018) ; Liu et al. (2016) ; Redmon & Farhadi (2018) have been proposed which aim to achieve real-time detection by utilizing an anchor-refine pipeline. For both families of detectors, the final detection results are obtained by a bottom-up one-way inference process. However, such one-way inference is agnostic to the global image context.

2.2. OBJECT DETECTOR REFINEMENT:

Besides designing novel object detection network architectures, several works focus on how to further improve the performance of the existing detectors with no or minor architecture modification. By modifying the standard cross-entropy gradient, Rao et al. (2018) proposed a simple yet effective method to learn globally optimized detector for object detection based on scores and bounding boxes. In that case no modification in the network architecture is needed. Pirinen & Sminchisescu (2018) proposed a deep reinforcement learning-based RPN which replaces the greedy RoI selection process with a sequential attention mechanism trained via reinforcement learning. Jiang et al. (2018) designed a side network, called IoU-Net, to predict the IoU between each detected bounding box and the matched ground-truth. The non-maximum suppression (NMS) procedure is improved by preserving accurately localized bounding boxes according to the obtained localization confidence. Cai & Vasconcelos (2018) unwrap Faster R-CNN to a sequence of detectors trained with progressively increasing IoU thresholds. The detectors are trained stage by stage, which adds parameters and training overhead linearly to the number of stages. The aforementioned detector refinement methods are not only limited by one-way inference but they also add extra training cost.

2.3. SELF-CORRECTIVE BEHAVIOR IN CNN:

In recent years, works exploiting the capability of CNNs to improve their off-the-shelf performance without re-training or additional data have attracted more and more attention. Sankaranarayanan et al. ( 2017) proposed a self-corrective mechanism for semantic segmentation. The structural perturbation generated by computing the error between output prediction and pseudo ground-truth is injected to the input image, expecting the updated prediction from the perturbed input to be improved due to the context. Carreira et al. (2016) designed an error feedback layer to iteratively refine the human pose estimation. Such self-correction algorithm is different from ours since our proposed top-down conscious feedback is totally parameter-free without any prior training needed or any model modification. Wang et al. (2018) proposed an inference procedure to iteratively update the feature maps, targeting to improve the predictions for unknown classes when partial evidence is available. However, it uses ground-truth for a subset of known classes or for the predictions of an auxiliary task, which is a very restrictive requirement in practice during inference. Instead, our inference scheme needs no ground-truth information. Additionally, these self-correction methods only utilize the category-specific feedback signal for error correction, which is not sufficient for object detection task. For our inference method, different kinds of perturbations are generated specifically for improving both recognition and localization accuracy. pred is compared with the ground-truth B to quantitatively evaluate the detection performance.

3. CONSCIOUS INFERENCE FOR OBJECT DETECTION:



3.2. CONSCIOUS INFERENCE (CI) FOR OBJECT DETECTION:

To facilitate the understanding, in the this section, we present our proposed conscious inference algorithm using a recent end-to-end one-stage object detector, YOLOv3 Redmon & Farhadi (2018) , as a baseline. This model is a new version of the classic one-shot object detector Redmon et al. (2016) . Assuming the updated input after the (r -1) th round of conscious inference is I r-1 , then the detection prediction of input 

I r-1 is B r pred . Each dimension of B r pred is represented by bbox B i = [b i h , b i w , h i , w i , o i , c i 1 , c i 2 , ..., c i l ] ∈ R D ,

Next, the r th conscious feedback pass will generate three kinds of perturbations respectively as:

EQUATION

where is the weighting parameter, step(.) is the step activation function, I r-1 L() is the gradient map w.r.t input layer, P r box , P r obj and P r cls are the bounding box, objectness and class-specific perturbations, respectively. Based on our experimental observation, step(.) performs better than the signum activation function used in Sankaranarayanan et al. (2017) .

By updating I r-1 as I r = I r-1 + (P r boc + P r obj + P r cls ), the (r + 1) th conscious inference round is performed:

EQUATION

The overall conscious inference process is illustrated in Fig. 1 . As we claimed, our proposed conscious inference algorithm is a general scheme that is applicable to any existing CNN-based object detectors. There is no requirement for global average pooling layer as in Zhou et al. (2016) . We next briefly describe how to implement our method to other baseline detection-based networks.

CI For Two-Stage Detector::

The conscious inference algorithm is also implemented on top of a classic two-stage object detector, Faster R-CNN Ren et al. (2015) . By design of the RPN, perturbations P r obj and P r box are generated from the RPN cls and RPN bbox layers and added to the RPN conv1 .

As for the P r cls , the RCNN cls layer predicts the class score of each B i from RCNN bbox . Thus, following the same way used for YOLOv3, a perturbation P r cls can be obtained and added to the base feature map (the output of the backbone network). Compared with the implementation on YOLOv3, our Faster R-CNN conscious inference version is more efficient since the conscious feedback pass goes less deep.

CI For Pose Estimator: A state of the art object pose estimator Tekin et al. ( 2018) is integrated to benchmark conscious inference in a scenario that naturally extends the 2D detection paradigm to a richer 6D representation. A YOLOv2-like end-to-end network with a modified regression output layer and a PnP algorithm to fit the pose estimation requirement is proposed in Tekin et al. (2018) to jointly detect the object as well as estimate its pose. Similar as in Eqn. 2, P r box , P r obj and P r cls can be readily generated following the same manner.

4. EXPERIMENTS:

We validate the effectiveness and efficiency of conscious inference on different object detection baselines and tasks: Sec. 4.1 shows the results of the one-stage detector YOLOv3, Sec. 4.2 presents the results of the two-stage detector Faster R-CNN and Sec. 4.3 demonstrates the performance on a 6D pose estimator. All the experiments are conducted on a single NVIDIA TITAN Xp GPU.

4.1. EXPERIMENTS ON YOLOV3 FOR 2D OBJECT DETECTION:

Dataset and Evaluation. The experiment is conducted on MS- COCO Lin et al. (2014) , where the 5k subset of validation (minival) images are tested. For evaluation, the standard COCO-style Average Precision (AP) across IoU thresholds from 0.5 to 0.95 with an interval of 0.05 is adopted.

Experimental Setting. The pre-trained YOLOv3 model from Redmon & Farhadi (2018) and the exact same algorithm parameter setting are adopted in our experiments without any modification. For our conscious inference algorithm, we set = 0.004, λ obj = 0.5 and λ iou = 0.9. Unless otherwise stated, the result from the first conscious iteration is reported due to most favorable performance and time efficiency trade-off. Results from more iteration rounds are also reported in Table 1 and Fig. 4 .

Experimental Results and Ablation Study. The results on COCO minival are reported in Table 1 . Largest improvement is achieved after one iteration, while more inference rounds have diminishing returns. Several ablation studies are also conducted. (1) The influence from different perturbations can be compared in Table 2 . As it can be seen, P box performs well for refining the bbox of originally detected object (improvement on high IoU); P obj is able to discover more missing objects (large improvement on AP); P cls focuses more on correcting the wrong classification of existing detections.

(2) Some parameter search experiments are conducted and shown in Fig. 3 . With the increase of , the performance keeps raising at first then turn to decrease since a large will over-modify the original input. Larger λ iou is able to filter more reliable and confident ROI candidates so better performance is achieved. If we keep raising λ obj , some reliable ROI candidates will be mistakenly eliminated causing a worse result. (3) Different feedback perturbation strategies are compared in Fig. 4 . Besides the quantitative results, some visualization results are presented in Fig. 2 Table 3 : The trade-off between efficiency and effectiveness of conscious inference on YOLOv3.

Table 3 shows the trade-off between efficiency and effectiveness of our algorithm by truncating the back-propagation pass and injecting the generated perturbations into different layers. The deeper the feedback signal goes, the better the improvement is while the computation increases. The time overhead is linear to the number of layers that the algorithm traverses during sequential top-down and bottom-up passes. Next we show an efficient implementation of our method for Faster R-CNN. Experimental Setting. We use a pre-trained Faster R-CNN model (Res101 backbone) from a pytorch implementation 2 , which achieves comparable performance against (Ren et al., 2015) . For our inference algorithm, we set = 0.020, λ obj = 0.9 and λ iou = 0.4. Unless otherwise stated, the result from the first iteration is reported.

Experimental Results. Tables 4 and 5 and Fig. 6 report the detection results on COCO and PAS-CAL VOC 2007 datasets respectively. Conscious inference improves the performance under various evaluation criteria, especially for high IoU and large objects. However, the gain is not as large as with YOLOv3, which can be partially attributed to the shallower conscious feedback pass. Experimental Results. The comparison results are shown in Table 6 . For each grid cell in the table, the first number is the baseline result and the second number is ours. As can be seen, for the most strict evaluation criterion, Acc 5px , our conscious inference is able to improve the baseline performance by a large margin. The overall performance of all categories is improved. Some visualization improvement results are shown in Fig. 7 .



5. CONCLUSION:

In this paper, we propose a guided, iterative inference algorithm, which can be applied on general CNN-based object detectors at inference stage. The proposed approach does not involve any model modification, re-training or ground-truth requirements. The term "conscious" is inspired by Dehaene (2014) since our method models two important traits identified in human cognition: top-down feedback and temporal persistence. Experiments based on different state-of-the-art object detectors show consistent improvement in diverse detection tasks. Our algorithm has no memory overhead (as opposed to strong detection refinement frameworks Cai & Vasconcelos (2018) ), while the added computation is linear to the number of traversed layers. The trade-off between extra computation and performance gain is an important factor to consider based on the application scenario. Empirical results on this trade-off using our straight-forward implementation are presented in Table 3 .

:

There is no concept of "ROI" in one-stage detectors without RPN, but we use the term "ROI" here to represent the detection predictions before NMS, which in practice are based on network stride and anchors.

Figures and Tables:

3.1 ONE-WAY INFERENCE FOR OBJECT DETECTIONWe start by reviewing the general one-way inference process of traditional object detectors (Fast/Faster R-CNN, YOLO, SSD, etc) briefly. Let I ∈ R W * H * C represent the input image for detection, B ∈ R D * k is the k ground-truth object bboxes. In order to train a baseline detector network with parameter Θ(.), a loss function L(Θ(.), I, B) is usually optimized. CNN training is out of the scope of this work so we assume the network Θ(.) has already been well-trained and remains unaltered at inference stage. During testing, the one-way inference gives the m detection predictions as B pred = Θ(I) ∈ R D * m , that each prediction B i ∈ R D contains the bbox localization, objectness score and class score information. Finally the non-maximum suppression (NMS) is performed to obtain the final detection results B nms pred ⊆ B pred ∈ R D * n , n ≤ m. The one-way inference result B nms

Figure 1: An illustration of the pipeline for our proposed conscious inference on YOLOv3.

Figure 2: The visualization of consistent improvement of detection bbox in a sample image. Red bbox is the ground-truth, green bbox is the baseline result and blue bbox is our conscious inference result.

Figure 3: Ablation studies for various factors. (a): The influence of . (b): The influence of λ iou . (c): The influence of λ obj . (d): The influence of injecting perturbation into different layers.

Figure 5: The visualization of detection improvement on COCO minival based on YOLOv3 detector. The red bbox is the ground-truth, green bbox is the detection result from baseline (one-way inference) and blue bbox is the detection result from our conscious inference (iter-1).

Figure 6: The visualization of detection improvement on (a) COCO minival and (b) VOC 2007 datasets based on Faster R-CNN detector. The red bbox is the ground-truth, green bbox is the detection result from baseline (one-way inference) and blue bbox is the detection result from our conscious inference (iter-1).

Figure 7: The visualization of pose refinement on OCCLUSION dataset based on Tekin et al.(2018). The red bbox is the ground-truth, green bbox is the detection result from baseline (one-way inference) and blue bbox is the detection result from our conscious inference (iter-1).

where {b i h , b i w , h i , w i } are 4 bounding box offsets, o i is the objectness score and {c i 1 , c i 2 , ..., c i l } are l class prediction scores. Since the final detection results after NMS B nms,r pred ⊆ B r pred are the most reliable detection predictions, they are further used to filter the ROIs 1 so that the selected ROI candidates B r roi ⊆ B r pred have the following properties: (1) high IoU score over a threshold λ iou (they are highly overlapped with B nms,r pred ); (2) high objectness score over a threshold λ obj (with high probability, the selected ROI contains objects). Therefore, for one ROI B i roi in B r roi extracted by B * = [b * h , b * w , h * , w * , o * , c * 1 , c * 2 , ..., c * l ] ∈ B nms,r pred , extending Sankaranarayanan et al. (2017) to form a three-fold guided signal, three pseudo ground-truths are generated respectively as:

Table content: <table/>

and Fig. 5. The influence of different perturbations. BP=P box , OP=P obj and CP=P cls . Iter-1 result is reported.

Table content: <table><tr><td>Models YOLOv3 39.67 58.28 45.00 AP AP 50 AP 75 Our-iter1 40.11 58.93 45.45 Our-iter2 40.11 59.14 45.43 Our-iter3 40.13 59.23 45.53 Our-iter4 40.14 59.30 45.53 Our-iter5 40.13 59.36 45.46</td><td>Models YOLOv3 YOLOv3+OP YOLOv3+BP YOLOv3+CP YOLOv3+OP+BP YOLOv3+OP+CP YOLOv3+CP+BP</td><td>AP 39.67 58.28 45.00 1.60 AP 50 AP 75 AP 95 40.01 58.87 45.36 1.53 39.65 58.26 44.96 1.63 39.87 58.65 45.21 1.58 40.02 58.87 45.39 1.56 40.08 58.93 45.41 1.59 39.85 58.62 45.21 1.60</td></tr><tr><td>Table 1: Comparison between con-</td><td colspan="2">YOLOv3+OP+CP+BP 40.11 58.93 45.45 1.61</td></tr><tr><td>scious inference against standard in-</td><td/><td/></tr><tr><td>ference on YOLOv3 across iterations.</td><td/><td/></tr></table>

2 https://github.com/jwyang/faster-rcnn.pytorchModelAP AP 50 AP 75 AP S AP M AP L AR S AR M AR L The influence of different perturbations on COCO. Iter-1 result is reported.

Table content: <table><tr><td>Faster (Res101)</td><td>34.5 54.9</td><td>36.9 14.4 39.2 52.4 22.8</td><td>52.1</td><td>66.4</td></tr><tr><td>Faster+OP</td><td>34.6 55.0</td><td>36.8 14.4 39.3 52.4 22.8</td><td>52.3</td><td>66.5</td></tr><tr><td>Faster+BP</td><td>34.5 54.9</td><td>37.0 14.5 39.3 52.5 22.9</td><td>52.3</td><td>66.5</td></tr><tr><td>Faster+CP</td><td>34.5 54.9</td><td>36.9 14.5 39.2 52.5 22.9</td><td>52.1</td><td>66.4</td></tr><tr><td>Faster+OP+BP</td><td>34.7 55.0</td><td>37.1 14.6 39.5 52.8 23.0</td><td>52.5</td><td>67.1</td></tr><tr><td colspan="2">Faster+OP+BP+CP 34.7 55.0</td><td>37.1 14.5 39.4 52.8 22.9</td><td>52.4</td><td>66.9</td></tr></table>

AP 60 AP 70 AP 80 AP 90 The results on PASCAL VOC 2007. Iter-1 result is reported.

Table content: <table><tr><td/><td/><td>Perturbation Generation Results</td><td/><td/><td colspan="2">Structure Search Results</td><td>Iterative Conscious Inference Results</td></tr><tr><td/><td>58</td><td/><td/><td>58</td><td/><td/><td>58</td></tr><tr><td/><td>56</td><td/><td/><td>56</td><td/><td/><td>56</td></tr><tr><td>mAP</td><td>54</td><td/><td>mAP</td><td>54</td><td>YOLOv3</td><td/><td>mAP</td><td>54</td></tr><tr><td/><td>50 52</td><td>Original Our_full Our_w/Half Mask Our_w/Whole Mask Our_w/Signum Our_w/o Dilation</td><td/><td>52 50</td><td>YOLOv3+OP YOLOv3+BP YOLOv3+CP YOLOv3+OP+BP YOLOv3+OP+CP YOLOv3+CP+BP YOLOv3+OP+CP+BP</td><td/><td>52 50</td><td>Original Iter=1 Iter=2 Iter=3 Iter=4 Iter=5</td></tr><tr><td/><td/><td>0.500 0.525 0.550 0.575 0.600 0.625 0.650 0.675 0.700 IoU</td><td/><td/><td colspan="3">0.500 0.525 0.550 0.575 0.600 0.625 0.650 0.675 0.700 IoU</td><td>0.500 0.525 0.550 0.575 0.600 0.625 0.650 0.675 0.700 IoU</td></tr><tr><td colspan="3">(a) Generate perturbation by differ-</td><td colspan="5">(b) Generate perturbation by differ-</td><td>(c) More iterations for conscious in-</td></tr><tr><td colspan="3">ent components</td><td colspan="3">ent structure designs</td><td/><td>ference</td></tr><tr><td colspan="8">Figure 4: (a): Different strategies to generate combined perturbation P r boc + P r obj + P r cls . (b):</td></tr><tr><td colspan="8">Ablation study on using different components. (c): Detection performance across inference rounds.</td></tr><tr><td/><td/><td colspan="4">Model AP AP 50 Faster (Res101) 46.1 74.1</td><td>66.2</td><td>52.5</td><td>30.8</td><td>6.9</td></tr><tr><td/><td/><td colspan="4">Faster+OP+BP+CP 46.8 74.3</td><td>66.4</td><td>53.3</td><td>31.4</td><td>8.4</td></tr></table>

ON YOLOV2-6D FOR 6D OBJECT POSE PREDICTION Dataset and Evaluation. The extremely challenging multi-object detection and pose estimation dataset,OCCLUSION Brachmann et al. (2014), is evaluated in our experiment. As its name suggests, several objects in the images are severely occluded due to scene clutter, which makes pose Acc mean 62.75\65.22 78.52\78.47 47.18\47.25 67.30\67.48 43.73\44.33 65.19\65.36 Comparison of our conscious inference against standard inference using the 6D pose prediction baselineTekin et al. (2018). As customary, 2D reprojection error is the evaluation metric. Results are shown in format baseline\ours, where blue means improvement and red means decline.

Table content: <table><tr><td>Category</td><td>Ape</td><td>Can</td><td>Cat</td><td>Driller</td><td>Glue</td><td>Holepuncher</td></tr><tr><td>Acc 5px</td><td>6.07\7.01</td><td>10.11\11.35</td><td>3.45\3.45</td><td>1.07\1.24</td><td>5.20\5.32</td><td>8.10\9.50</td></tr><tr><td>Acc 10px</td><td colspan="6">39.32\43.25 58.16\58.99 21.74\21.74 16.97\16.97 25.69\25.91 38.84\38.93</td></tr><tr><td>Acc 15px</td><td colspan="6">59.83\63.08 79.70\79.29 38.42\38.50 40.44\40.77 39.20\39.09 52.73\52.81</td></tr><tr><td>Acc 20px</td><td colspan="6">68.29\71.11 86.00\85.75 49.20\49.20 62.11\62.27 46.73\46.84 62.40\62.23</td></tr><tr><td>Acc 25px</td><td colspan="6">72.74\74.87 88.48\88.24 54.42\54.59 80.15\80.40 50.06\51.16 68.84\68.84</td></tr><tr><td>Acc 30px</td><td colspan="6">74.96\77.26 90.89\90.47 58.05\58.13 89.95\89.99 52.49\53.27 73.97\73.97</td></tr><tr><td>Acc 35px</td><td colspan="6">75.31\78.29 91.88\91.71 59.98\60.07 93.90\94.15 53.71\54.60 80.17\80.25</td></tr><tr><td>Acc 40px</td><td colspan="6">76.32\78.72 92.79\92.46 60.99\61.08 95.47\95.47 54.15\55.26 85.45\85.62</td></tr><tr><td>Acc 45px</td><td colspan="6">76.67\79.15 93.45\93.04 62.34\62.43 96.29\96.62 54.82\55.81 89.50\89.50</td></tr><tr><td>Acc 50px</td><td colspan="6">78.03\79.49 93.79\93.37 63.27\63.27 96.71\96.87 55.26\56.04 91.90\91.90</td></tr></table>

Peer review (iclr2019):
 The paper proposes a iterative approach at inference time to improve object detections. The work relies on updating the feature activations and perform new feed forward passes to obtain improved results. Pros: (+) The idea of iterative inference is potentially effective (+) The paper is well written and clear (+) The authors show results on compelling benchmarks Cons: (-) Reported improvements are very small (-) Important baselines are missing First, while the authors state correctly that their updates have no memory cost and no new parameters are added, they do require more FLOPs at test time. For N-stages, the approach requires xN more operations for forward passes and xN for backward passes. This is a serious shortcoming as it adds compute time per image for the inference stage and cannot be parallelized. The authors show small improvements for AP on COCO. From their analysis, it seems that the biggest gains come from N=1 stages, while the improvement added for N>1 are miniscule (Table 1). Note that the authors show results on COCO minival (5k images) and from my experience there, it*s expected to see a +/- 0.2% AP between different trained models of the same architecture. The authors report a +0.46% gain. In addition, the authors do not provide results for other baseline approaches that have similar FLOPs at test time, such as iterative bounding box regression and input scale augmentation. Note that both these approaches do not add any parameters and require no additional memory, but add to the FLOPs at test time. From my personal experience, test time augmentations can add +1.5% to the final performance. Concretely, look at Mask R-CNN arXiv Table 8 last two rows. Test time augmentations add 1.5% on top of an already enhanced model. Empirically, the better the model the harder it is to get gains from inference tricks! And still test time augmentations boost performance significantly. Given the small gains and the lack of competing baselines, it is hard to make a case for accepting the paper.