Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes a CRF-based context module for CNN-based object detectors. In particular for the two-stage region-based detector, like Faster RCNN, the context module is added right before the output layer of the classification head. Every box proposed by the RPN is a node in the CRF, and its label is the classification label. Message passing is unrolled as neural network layers. Potentials are defined based on object detector outputs, box overlap, and co-occurrence of class labels. Experiments are performed on the MS COCO object detection task. 

The proposed method is similar to the CRF-based method for FCN segmentation networks except that the nodes are boxes instead of pixels. And simpler ideas had been explored in the pre-DL era. In this sense, the main idea of this work is incrementally novel. 

The presentation of the paper is fine in general. But the “global message passing” paragraph in Sec 4.2 can be improved. In particular, how p(l) is used in message passing? It is better to provide more specific descriptions so that the paper is self-contained.

The proposed model outperformed the baseline with a reasonably significant margin. However, its average accuracy on all 80 COCO categories is no better than “the Faster RCNN + Relation”. Since the proposed model is also not significantly simpler than the Relation Network, the experimental results do not establish a new state-of-the-art. 

The experimental results are also a bit thin. More ablative studies can be helpful, e.g., global message parsing/local message passing.

Given the moderate novelty (which is good but not good enough as a standalone reason to accept this paper) and the not-strong-enough experimental results, I feel more work is needed for the paper to be readily publishable. 

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The paper proposes a contextual reasoning module following the approach proposed by the NIPS 2011 paper for object detection. Specifically, the algorithm proposed by NIPS 2011 are first converted to end-to-end modules step-by-step, and then added to the detection framework Faster R-CNN. The comparison is only to one other baseline (Relation Module), and some improvements are show especially for small objects.

+ Context reasoning is an important but unsolved problem in vision, it is a good attempt to apply CRF to object detection

- The subset results are not convincing. For example, AP_L has decreased dramatically from 51.2 to 44.9. This is shocking. Large objects should maintain a similar performance if the contextual module really works. This result is actually very negative to me.
- I am not sure the idea of applying conCNN to 6 categories of COCO is a great idea to begin with: context reasoning requires a lot of categories, especially those with few examples to benchmark with. If the categories are supplied with a lot of training examples, it is very hard for context to really help there. I would at least stick to full set of COCO (I am fine with reducing the number of training images), or switch to the recently proposed LVIS dataset:
https://www.lvisdataset.org/
- I think the work did a less comprehensive literature survey for object detection with context reasoning. For example: 
Qi, Lu, et al. "Sequential context encoding for duplicate removal." Advances in Neural Information Processing Systems. 2018.
Chen, Xinlei, and Abhinav Gupta. "Spatial memory for context reasoning in object detection." Proceedings of the IEEE International Conference on Computer Vision. 2017.
There are actually many other attempts from different groups that try to nail down this problem, please at least review them in the paper.

Overall this is a paper that's not ready. There might be technical contributions (still with the presence of Zheng et al ICCV 2015, it is limited), but the paper needs to be refined in many ways to get accepted.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper introduces a context-aware neural network (conCNN) that integrates context semantics into account for object detection. The proposed approach achieves this by embedding a context-aware module into the Faster R-CNN detection framework. The context-aware module simulates the learning process of Conditional Random Fields (CRF) model using a stack of common CNN operations. Specifically, this paper employs the mean-field approach of [1]. Experiments are performed on COCO dataset.

The  paper reads well. The idea of combining the strengths of both CNNs and CRF-based graphical models in a detection framework is interesting. Though the main idea is borrowed from the mean-field approach of [1], it has been successfully reformulated as a stack of common CNN layers. However, my main concern is that the experimental results (Page 8 Table 2) does not support the merits of the proposed approach. When comparing with the existing Faster R-CNN + Relation work of [2], the improvement provided by the proposed approach is negligible. In fact the proposed approach achieves inferior results on large objects (APL), compared to [2]. [1] uses attention to model the geometric relationships among objects in the same image. In addition to geometric relationships, the proposed approach leverages the inherent co-occurence relationships between object classes. However, this additional information does not seem to provide much help by looking at the results on COCO dataset. The paper does present an experiment to justify this additional information in Table 1. However, that small experiment is only on 6 object categories and does not seem to generalize when going towards full COCO dataset with 80 categories (Table 2). Therefore, the reviewer would recommend to thoroughly evaluate the merits of the proposed approach in depth on COCO and on additional datasets (i.e., Visual Genome). 

[1] Philipp Krähenbühl, Vladlen Koltun: Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials. NIPS 2011.
[2] Han Hu, Jiayuan Gu, Zheng Zhang, Jifeng Dai, Yichen Wei: Relation Networks for Object Detection. CVPR 2018. 

review_title: Official Blind Review #3

Meta-review: The paper proposes a contextual reasoning module following the approach proposed by the NIPS 2011 paper for object detection. Although the reviewers find the proposed approach reasonable, the experimental results are weak and noisy. Multiple reviewers believe that the paper will benefit from another review cycle, pointing out that the authors response confirmed that multiple additional (or redoing of) experiments are needed. 

Decision: Reject