Context-Aware Object Detection With Convolutional Neural Networks
Authors: Yizhou Yan
Lei Cao
Samuel Madden
Elke Rundensteiner
Venue:  (2020)
Keywords: Object Detection
CNN
Context
CRF

Abstract: Although the state-of-the-art object detection methods are successful in detecting and classifying objects by leveraging deep convolutional neural networks (CNNs), these methods overlook the semantic context which implies the probabilities that different classes of objects occur jointly. In this work, we propose a context-aware CNN (or conCNN for short) that for the first time effectively enforces the semantics context constraints in the CNN-based object detector by leveraging the popular conditional random field (CRF) model in CNN. In particular, conCNN features a context-aware module that naturally models the mean-field inference method for CRF using a stack of common CNN operations. It can be seamlessly plugged into any existing region-based object detection paradigm. Our experiments using COCO datasets showcase that conCNN improves the average precision (AP) of object detection by 2 percentage points, while only introducing negligible extra training overheads.

Under review as a conference paper at ICLR 2020
CONTEXT-AWARE OBJECT DETECTION WITH CON- VOLUTIONAL NEURAL NETWORKS
Anonymous authors Paper under double-blind review
A ABSTRACT
Although the state-of-the-art object detection methods are successful in detecting and classifying objects by leveraging deep convolutional neural networks (CNNs), these methods overlook the semantic context which implies the probabilities that different classes of objects occur jointly. In this work, propose context-aware CNN (or conCNN for short) that for the first time effectively enforces the semantics context constraints in the CNN-basod object detector by leveraging the popular conditional random field (CRF) model in CNN. In particular, conCNN features a context-aware module that naturally models the mean-field inference method for CRF using a stack of common CNN operations. It can be seamlessly plugged into any existing region-based object detection paradigm. Our experiments using COCO datasets showcase that con CoNCNN improves the average precision (AP) of object detection by 2 percentage points, while only introducing negligible extra training overheads.
1 INTRODUCTION
In recent years, deep convolutional neural networks (CNN) Goodfellow et al. 2016) have been used with great success in object detection tasks. However, some well known principles in computer vision that have been shown to be effective in object detection are largely overlooked. In particular, the vision community has shown that the semantic context, namely the correlations among the objects, helps object detection Rabinovich et al. (2007). That is, when performing the task of object detection, objects' class labels should inferred with respect to other objects in the scene. For example, when detecting objects in scene of baseball match in the COCO dataset, which contains four objects: "Baseball", "Baseball bat", "human", and "baseball glove", the CNN-based object detection model tends to recognize the "bascball bat" as 'tooth brush". Yet it can casily be seen that this object of "tooth brush" does not fit into the context with other labels. This is because the state-of-the-arr object detection ion methods (He et al.] 2015 Girshick 2015 Ren el al. 2015 Dai el al. 2016 Lin
et. al. 2017 He et al. 2017) adopt the region-based paradigın since it was introduced the seminal C-CNN work Girshick et al. 2014). Given a set of region proposals, these methods
perform object classification and bounding box regression on each proposal individually,
without taking the semantic context into consideration.
In this work, we now propose a context-award object detection strategy called conCNN to address the above shortcoming. By seamlessly plugging a contert- aware module into the existing CNN- object detection paradigm, conCNN automatically learns and effectively enforces the semantics context constraints, yet requiring minimal modification of the existing deep object detection architecture.
ConCNN is inspired by our observation that probabilistic graphical models, in particular, Conditional Random Fields (CRFs) have been successful in enhancing the accuracy low level computer vision tasks (Ladicky et al. 2009 Zheng et al. 2015 Rabinovich et al. 2007 Krähenbühl & Koltun, 2011) such as image segmentation or object detection. More specifically, CRF models the label assignment problem as probabilistic inference problem such that constraints can be incorporated, for example, the label agreement between similar pixels in image segmentation.

---

Under review as a conference paper at ICLR 2020
We thus design context-aware module that for the first time leverages CRF in deep object detection architecture to maximize the contextual agreement among the objects in the same scene, yet only slightly increasing the training time of the object detection model. The overall context-aware deep object detection network, corresponds to a deep architecture with traditional CNN-based module concatenated by a context-aware module, can be trained end-to-end following the common practice of prop-propagation in deep neural networks. Therefore, we successfully combine the strengths of both CNN and CRF-based graphical models in one unified framework.
More specifically, the context- aware module reformulates the iterative mean-field infer- ence (Krähenbühl & Koltun 2011) popular approximate inference method for CRF as stack of common CNN layers. This way, learns a compatibility matrix as normal parameters of deep neural network that represents the probabilities of label co-occurrences, as the compatibility function CRF.
Our experimental evaluation on the COCO datasets confirıss that conCNN improves the AP of object detection by percentage points.
2 NETWORK ARCHITECTURE OF CONCNN
In this work, proposal context-aware neural network (conCNN) for object detection. Essentially, we achieve this by plugging a context-aware module into a CNN network built upon the popular region-basod object detection paradigm as shown in Fig.
Reseon-based Object Detection. The region-based object detection paradigm GGrshick et al. 2014) first processes the whole image with several convolutional and max pooling layers (hhe CNN backbone) to produce conv feature map. Taking the feature map as input, the Region Proposal Network (RPN in short) generates a set of rectangular object proposals that most likely contain objects. Then for each object proposal, a region of interest (Rol) pooling layer extracts a fixed-length feature vector from the feature map.
Each feature vector is fed into a sequence of fully connected layers (FC layers) that finally branch into two sibling output layers corresponding to the classification task and the bounding box regression task. The classification task produces softmax probability estimates over L object classes. The bounding box regression task out puts four real-valued numbers for each of the I. object classes. Each set of 4 values encodes refined bounding box positions for one of the I classes.
Context-aware Module. Our context-aware module is plugged into the classification task. Specifically, takes the prediction from the last FC layer in the classification task as well as the location for each bounding box in the regression task as input and produces modified prediction that incorporates the context semantics. The modified prediction distribution is then fed into the softmax layer to generate the final probability estimates.
Context-aware Module
Softmax
Prediction location per BBox
Refined
Region Proposal Network
ROI
ROI Pooling
Feature Vector
Classification Task
prediction
Backbone
Layers
Feature Map
FC
Bbox Regressor
Bounding Box Regression Task
Figure 1: Overall Network Architecture of conCNN
3 CONDITIONAL RANDOM FIELD IN OBJECT DETECTION
In this section briefly overview Conditional Random Fields (CRF) for object detection. In object detection, a CRF models object labels (classes) as random variables that form a Markov Random Field (MRF) when conditioned upon the input image.
2

---

Under review as a conference paper at ICLR 2020
Conditional Random Field. Let X; be the random variable w.r.t. object i. It represents the label assigned to the object and can take any value from a pre-defined set of labels L (=..,.......111 4.).
Let X be the vector formed by the random variables X1, X2, XN. where N denotes the number of objects in the image 7. Then the probability of some label assignment conditional on the global observation given by the CRF model:
P(X = x|1)
2(1) ezpl -E(())
(1)
Here E(x) denotes the energy of the configuration x E LN,, while (()) denotes the partition function Lafferty et al., 2001
3.1 MEAN-FIELD APPROXIMATION
Minimizing the above CRF energy F(x) yields the most probable label assignment æ (maxi- mizing P(x)) for the given image. Since this exact minimization is intractable, in the fully connected pairwise ORF model, mean-fiele approximation to the CRF distribution (Krähen bühl & Knltun, 2011) is used for approximate maximum posterior marginal inference. It approximates the CRF distribution P(X) using a simpler distribution Q(X), which can be expressed as the product of independent marginal distributions, Q = II; Q,(X.), where (X;) corresponds to one object image Leveraging Krähenbühh & Koltun (2011) for object detection, the following update equation to iteratively compute valid Q+
K
Qi(X, = Z(1) eap
vu 5x
Σ m(l.t) PEC
Σ m=1
w Σ KTh (f,,))) it
=x,
Unary Component
Pairwise Component
(2)
In Equation 2 the unary component (X = (denoted as (!)) measures the probability that the label / is assigned to the object (X, (). As shown in Fig. [ in our conCNN it is obtained from the classification task, which assigns label to object i without considering the context semanties.
1') reflects the pair- wise influence of the label assignment (X, = =') of objects on the label assignment of object i (X; l). Generally speaking, The pairwise component measures the cost of assigning labels 1 to objects simultaneously. It enforces the context semantics constraints.
The pairwise component Eree Σ wm km.,))
Among this pairwise component, function p(l, captures the compatibility (or the possibility of co-occurrence) between pair of labels I and l'. It corresponds to the new parameters that have to learned in the training of conCNN to enforce the context semantics constraints.
Each function (," for m = 1, ..., M, represents a function applied on feature vectors (Ji, f3) of objects and j. The feature vector fa of object i corresponds to some characteristics of object derived from a CNN. In scenarios, fi corresponds to the location of the bounding box or the probability estimation of object taking label E L. Essentially, km (fr.f) reflects how strongly object is related to other objects hence called relation function.
Note in Krähenbühl Koltun 2011 and Zheng et al. 2015 which apply CRF image segmentation, (""(),))) corresponds to the Gaussian kernel. However, in object detection Gaussian kernel does not necessarily work well (Rabinovich et al. 2007). In Sec. 4.2 we will introduce in more details the relation functions used in work.
3

---

Under review as a conference paper at ICLR 2020
Algorithm 1 Mean-field for dense pairwise CRFs
1: Qi(1) ziT expl exppp (U:());
Initialization
2: while not converged do
3:
(1) t Ejsr kTT (1)
> Message passing from all X, to all X; Weighting b Compatibillty transform Local Update Normalization
Qi(1) Em wm (I);
4:
5:
Qi(1) + Evee (htec(l1""))(("""
6:
Qi(1) Ui(l) Qi(1);
7: Q:(1) z(n) exp( Qi))));
8: end while
3.2 MEAN-FIELD ALGORITHM
The update equation (Equation 2 leads to the mean-field algorithm summarized in Algorithm
1
performs a message passing step, a compatibility transform, a local update, and a normalization.
[Kähhenühh & Koltun 2011 Each iteration of Algorithm 1
Next, we show how to reformulate all these steps in Algorithm 1 common CNN layers and hence effectively enforees the context semantics constraints in CNN-basod object detection, using the classical practice of back propagation in deep neural networks.
CRF INFERENCE As COMMON CNN LAYERS
To reformulate the mcan-field inference as common CNN layers, it is essential to be able to calculate error differentials in each layer w.r.t. its inputs in order to back-propagate the error dilferent lo previous layers during training. This way, CRF parameters such the label compatibility function pul,," l') can be learned automatically during the training the full work. Therefore, in this section we also discuss how to calculate error differentials with respect to the parameters in each layer.
Prediction
L
Compute loU [N*N)
Prediction (N
Local Prediction IN L)
Weighted Prediction (N w
(N UU
N 4
Message
Normalization
Compatible Matrix (L L)
Prediction
per bbox
Max spoling
Class Prediction [1
Prediction per bbox (N*L)
Gloabl Prediction L)
Influence IN Local Update
Compatibilty Transformation
Global Message Passing
Weighting
Figure Context-aware aware Module, L stands for the number object classes and stands for the number of objects in image 1.
4.1 INITIALIZATION
In this step (Line 1, Algorithm 1. the initialization operation Q,(1) 1
2(1) eapp (?(()) is
performed. Since Z()) (Lafferty et al. 2001 corresponds Eiec ezp( (1))),, we get:
eI)(U.(())
2,01 = Z(T)
exp(U.()) Eiec eap((U,())
(3)
Note that the form of Q:(1) in Equation 3 is equivalent to the softmax function (Goodfellow ct al. 2016), widcly used in CNN architectures. Naturally supports back-propagation. That is, during back-propagation its error differentials received at the output of this step is passed back to the input of the context-an aware module (the prediction produced by bounding box classification) after performing backward pass calculations of the softmax function.
4

---

Under review as a conference paper at ICLR 2020
4.2
MESSAGE PASSING
The message passing step updates each single variable X; w.r.t. object by aggregating information from all other variables X, w.r.t. other objects j. As shown in Fig. 2 conCNN supports two types of information aggregation, called local message passing and global message passing.
Local Message Passing. Local message passing models the geometry relationship among the objects in the same image Similar to Hu et. al. (2018), it is based on the intuition that given an object i, the objects j tend to have strong influence on i its bounding box overlaps with the bounding box of i. Therefore, the local message passing takes the location information into consideration when aggregating the influence from other objects.
More specifically, we first compute the Intersect of Union (IoU) between the bounding boxes of a pair of objects. The IoUs w.r.t. all object pairs correspond to a x N matrix, where N need as input the
denoles the number of objects in image I. To compute the loU matrix, bounding box information of all N objects which can be considered as a N x 4 matrix. To efficiently compute the IoU matrix, in this work we formulate its computation as a matrix transformation operation such that this process can be performed on GPU in parallel. The diagonals of the IoU matrix are then reset to zeros because we only need to aggregate the predictions from other objects.
Once the IoU matrix ready, the local message passing process can be mapped to matrix product operation between the N x N IoU matrix and N x L prediction matrix which correspond to the original predictions w.r.t. the N objects and the L classes. This results in the final aggregated predictions (a N x L malrix), weighting the predictions from the overlapping bounding boxes more.
Since this step is equivalent to set of linear weighted-sum transformations and does not involve any parameters, the back-propagation is trivially supported.
Global Message Passing. Inherent co-occurrence relations also exist among the objects in the same scene, not tied to the geometry locations these objects. For instance, although keyboard and mouse do not overlap, the two object classes together define computer-related context and can help differentiate mouse and soap which look similar in appearance.
In this work, this o-oceurrence relationship is captured by the global message passing. It uses the probability that each object class appears in an image to model the context of this image, denoted as p()). Here define p(l) as the maximal probability estimation of class I among all objects in the image, that is:
p(l) mo)bpe()))object i I}
(4)
where p!()) denotes the probability that the classification task assigns object i to class L.
The p(l) w.r.t. all / E can be computed by applying global max pooling function (Good fellow et al.] 2016) on the N x L prediction matrix. This produces a 1 x L class probability vector, which the global context of this image, influences all objects equally.
Since max pooling is common operation in CNN, the back-propagation can be naturally supported in the typical way.
4.3 WRIGHTING STEP
The weighting step takes a weighted sum of the M outputs the message passing step, corresponding to the M relation functions. This step can be mapped to 1 x convolution filter with M input channels and 1 output channel. Here each input channel corresponds to N x L matrix. The 1 x 1 M x parameters of this filter correspond to the weights w' of the M relation functions. As a common convolution operation, error derivative can be computed in the usual manner to pass the error derivatives back to the previous step.
5

---

Under review as a conference paper at ICLR 2020
4.4 COMPATIBILITY TRANSFORM
The outputs (!) on different labels I E & influence each other to a variant extent, depending on the compatibility between these labels. In conCNN, A two labels and l' L, the compatibility is parameterized by the compatibility function pll, I"), corresponding to a L compatibility matrix C.
Organizing (1) with respect to all N objects in image I and all labels I E as N L matrix Q the compatibility transform step can be implemented as a matrix product: x C. The output is also N L matrix.
In CNN, this matrix product can be viewed as a common convolution filter, where the spatial receptive field of the filter is 1 1, and the number of imput and output channels are both L. The 1 x L x L weights of this filter correspond to the compatibility matrix C. other words, learning the label compatibility function μ is equivalent to learning the weights of this filter. Since this step is a common convolution operation, transferring error differentials from the output its input can be done the usual way.
4.5 LOCAL UPDATE
In this step (Line 6, Algorithm Dh. the output Q+(1) from the compatibility transform step is subtracted from the unary input U,).). Since this step does not have any parameters, transferring error differentials can be done trivially by copying the diffcrentials at the output of this step to both inputs (1) and Ut()).
4.6 NORMALIZATION
Similar to the initialization step, the normalization step (Line 7, Algorithm corresponds to softmax operation without parameters.
EXPERIMENTAL EVALUATION
OVERVIEW OF EXPERIMENTAL SETTING
Datasets. We demonstrate the effectiveness of our context-aware module using the bench- mark COCO dataset and a subset of it.
The original COCO dataset contains 80 categories. As in previous work (He et al. 2017 Lin el al. 2017), we train using the union of 80k train images and a 35k subset of validation images (tramvaa33kk), and evaluate on the remaining 5k validation images (minival).
The COCO subset dataset contains 6 categories of the original COCO including baseball bat, baseball glove, sink, toilet, mouse and keyboard. These categories represent different scenes (outdoor, bathroom, indoor) with 2 classes from each scene. There are 13k training images and 531 validation images in this COCO subset.
Methodology. We evaluate: (1) Faster R-CNN (Lin et al.] 2017 as baseline; (2) Relation Network: Faster R-CNN with Relation Module (Hu al. 2018); ; (3) Our method conCNN: Faster R-CNN with our contexi-aware module (Sec. 2 We ran experiments on 4 P100 GPU instances on Google cloud. The results show that our conCNN outperforms Relation Network in improving the AP of object recognition.
Evaluation Metrics. We report the standard COCO metrics including AP (averaged over loU thresholds from 0.5 to 0.95), AP50 (IoU 0.5), AP75 (IoU 0.75) and AP at different scales including APs (Small), APM (Medium), APt (Large).
5.2 IMPLEMENTATION DETAILS
All approaches use ResNet-101- as backbone. We implemented our conCNN by plugging the context-aware module into the penultimate layer of the classification task. We set

---

Under review as a conference paper at ICLR 2020
hyper- paperrpprramterr following the existing Faster R-CNN work (Ln et al. 2017 Ren et al.
2015 The evaluation of the baseline (Lin et al. 2017 and Relation Network Hu et al. 2018 is based on the models published by the authors of Relation Network.
During training, we adopt image-centric training (Girshick] 2013). Images are resized such that their scale (shorter edge) is 800 pixels as done in (Lin et al. 2017). An RoI is considered positive has IoU with a ground-truth box of at least 0.5 and negative otherwise. Each mini-batch has 4 images per GPU and each image has 512 sampled Rols, with ratio of 1:3 of positives to negatives. For the COCO subset with 6 categories, we train on 4 GPUs for 12k iterations, with a learning rate of 0.02 which is decreased by 10 at the 9kth iteration. For the COCO dataset with 80 categories, we train on 4 GPUs for 90k iterations, with a learning rate of 0.02 which is decreased by 10 the 60kth and 80kth iteration. At test time, the proposal number is 1000 (as in Lin et al. (2017)).
5.3 EXPERIMENTS ON COCO SUBSET WITH 6 CATEGORIES
We first test our ConCNN on the COCO subset with categories. This subset contains images from 3 different scenes including outdoor, bathroom, and indoor. We use this subset
to validate that our conCNN can effectively capture the co-occurrence relationship between
the objects in the same scene.
Table 1: Results COCO (6 categories).
Methods
Backbone
AP
AP
AP
APs
APM
APL
Faster R-CNN
Resnet-101-FP1
41.6
67.7
44.5
21.1
47.1
51.2
Faster RCNN + Relation
Resnet-101- 101-FFP
40.6
66.8
44.1
23.1
45.4
48.9
conCNN
Resnet-101-FPN
44.07
70.57
48.34
24.54
48.47
44.94
As shown in Table [ in almost all cases, conCNN outpertorms Faster R-CNN and Relation Network. The performance gain results from the context-aware module in our conCNN, which leverages both the geometry and co-occurrence relationships among the objects. In particular, conCNN is good at detecting small and medium objects. Although small and medium objects are difficult to recognize purely by appearance, the relationships among the objects can help emphasize the correlated objects in the same image and thus improve the APs and APM Relation network does not perform well on this dataset because the Relation Network only considers the geometry relationship among the objects overlapping with each other, while objects hardly overlap in this dataset.
In addition, our conCNN introduces negligible training overhead. Specifically, the training takes 4.12 hours and 4.24 hours for Faster R-CNN and conCNN respectively. Relation Network takes 5.02 hours for training, because it uses a more complex relation module.
baseball bat
-0.14
-0.01
0.09
0.09
0.2
baseball glove
-0.06
0.1
toilet
0.28
0.25
-0.13
0.25
0.12
40.55
mouse
0.09
0.19
0.04
keyboard
0.23
0.15
0.24
-0.1
0.25
0.23
-.
0.16
0.09
0.14
-02
baseball glove
tallet
mouse
keyboard
baseball
Figure 3: Compatibility Matrix for COCO with 6 Categories.
Fig. 3 shows the compatibility matrix learnt by conCNN. By Eq. 2 correlated classes should have negative values in this matrix. The smaller the value is, the stronger the correlation between the pair of classes. As expected, the learnt compatibility matrix shows that baseball glove is strongly correlated to baseball bat, while toilet/mouse mouse is highly correlated to sink/keybon keyboard. This confirms the effectiveness of conCNN in capturing context semantics.
7

---

Under review as a conference paper at ICLR 2020
5.4 EXPERIMENTS ON COCO WITH 80 CATEGORIES
We also test our conCNN on the whole COCO dataset with 80 categories. As shown in Table 2 our conCNN still outperforms Faster R-CNN and Relation Network. Relation network performs well when detecting large objects. That is as expected because the Relation Network only considers the geometry relationship among the objects overlapping with each other, while large objects tend to overlap with more objects than small objects. The training time of conCNN is 27.25 hours only 0.75 hours longer than the Faster R-CNN which takes 26.5 hours. On the other hand, the Relation Network takes 32.25 hours to train.
Table 2: Results on COCO (80 categories).
Methods
Backbone
AP
AP
AP7.
APs
APM AP2
Faster R-CNN Faster RCNN + Relation
Resnet- 10nmet-101-FP)
36.6
59.3
39.3
20.3
40.5
49.4
Resnet-101-FF1
38.6
59.9
43
22.1
42.3
52.8
conCNN
Resnet-101-FPN
39.48
61.05
43.20
22.67
42.58
52
RELATED WORK
CRF in Computer Vision. Zheng et al. 2015 combines CRF and CNN to perform pixel-level semantic image segmentation. It tormulates the CRF as an RNN. The RNN is then plugged in as part of CNN which can be trained end-lo-end using back-propagation. Our work instead focuses on leveraging CRF in object detection. Instead of modeling the training process of CRF a RNN, we factorize the mean-field approximate inference of CRF as common CNN layers and use the common CNN operations such as softmax and convolution to naturally simulate CRF in enforcing the context semantics constraints. Further, unlike Zheng et al. (2015) which uses Gaussian kernels to encourage label agrecment between similar pixels, design customized functions for object detection task. These functions elfectively reflect the co-occurrence relationship among the objects in the same scene. CRF was also used post-processing lo refine the pixel-level label predictions
object-level class assignment (Rabinovich et al. 2007 Ladicky et al., 2009 Krähenbühh & Koltun 2011] Since CRF not incorporated into the training process segmentation or object detection model, these methods are shown to be not as effective as Zheng et al. (2015).
Object Relation in Deep Learning. Hu al. (2018) proposed relation network that adapts the concept of attention in natural language processing (NLP) into the classical CNN- object detection framework. The key idea to use attention to model the geomelry relationships among the objects in the same image, such as plate on top of lable or a looth brush in a cup. These dependencies are represented as relation features which are used together with other features to classify objects. Our work instead combines the strengths of CNN and CRF. It not only models the geometry relationships among the objects in one image, but also leverages the inherent co-occurrence relationships between object classes, such as mouse and keyboard, fridge and microwave, or toilct and sink.
Object Relation in Classical Computation Vision. Similar to the use of CRF, most object detection works before deep learning arises use object relations as a post-processing step DDvvala et al. 2009 Galleguillos et, al. 2008 Mottaghi al.] 2014 Torralba et al. 2003
Tu & Bail 2010 Rabinovich et al.] 2007 Felzenszwalb et al. 2010 The detected objects are re-scored by considering object relationship such as the co- co-ocuureence of different object classes (Rabinovich et al. 2007 Felzenszwalb et al. 2010). However, unlike our work, in this sctup the object dotector is unaware of the object relations during the training phase.
Therefore, it cannot fully employ the object relations in object detection.
7 CONCLUSION
In this work, we proposed a context-aware neural network conCNN that takes the context semantics into consideration in object detection. The key idea is to embed a context-aware module into the CNN-based object detection network that effectively simulates the learning process of Conditional Random Field (CRF) model using stack of common CNN operations. Combining the strengths of both CNN and CRF, conCNN effectively improve the AP of object detection confirmed in our experiments on COCO datasets.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Jifeng Dai, Yi Li, Kaiming He, and Jian Sun. R-FCN: object detection via region-based fully convolutional networks. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5- 5-10, 2016, Borcelona, Spoin, pp. 379 387, 2016.
Santosh Kumar Divvala, Derek Hoiem, James Hays, Alexei A. Efros, and Martial Hebert. An empirical study of context in object detection. In CVPR, 20-25 June 2009, Miami, Florida, USA, pp. 1271 1271-1278, 2009.
Pedro F. Felzenswwalb, Ross Girshick, David McAllester, and Deva Ramanan. Object detection with discriminatively trained part-based models. IEEE Trans. Pallern Anal. Mach. Intell., 32(9):1627 1645, 2010.
Carolina Galleguillos, Andrew Rabinovich, and Serge J. Belongie. Object categorization using co-occurrence, location and appearance. In CVPR, 24-26 June 2008, Anchorage, Alaska, USA, 2008.
Ross Girshick. Fast r-cmn. In Proceedings of the IEEE international conference on computer vision, pp. 1440 1448, 2015.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 580-587, 2014.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Trans. Pattern Anal. Mach. Intell., 1799)-1904-1916, 2015.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cmn. In Proceedings of the IEEE international conference on compuler vision, pp.
2017.
Han Hu, Jiayuan Gu, Zheng Zhang, Jifeng Dai, and Yichen Wei. Relation networks for object detection. In CVPR, Salt Lake Cily, UT, USA, June 18-22, 2018, pp. 3588 3597, 2018.
Philipp Krähenbühl and Vladlen Koltun. Efficient inference in fully connected erfs with gaussian edge potentials. In NIPS, 12-14 December 2011, Granada, Spain., pp. 109-117, 2011.
Lubor Ladicky, Chris Russell, Pushmeet Kohli, and Philip H. S. Torr, Associative hierarchical crfs for object class image segmentation. In ICCV, Kyoto, Japan, September 27 October 4, 2009, pp. 739-746, 2009.
John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In ICML, Williams College, Williamstown, MA, USA, June 28 July 1, 2001, pp. 282 289, 2001.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2117-2125, 2017.
Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Wie Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan L. Yuille. The role of context for object detection and semantic segmentation in the wild. In CVPR, Columbus, OH, USA, June 23-28, 2014, pp. 891-895 2014.
Andrew Rabinovich, Andrea Vedaldi, Carolina Galleguillos, Eric Wiewiora, and Serge J. Belongie. Objects in context. In ICCV, Rio de Janeiro, Brazil, October 14-20, 2007, pp. 1-8, 2007.
9

---

Under review as a conference paper at ICLR 2020
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real- time object detection with region proposal networks. In Advances in neural information processing systems, pp. 91-99, 2015.
Antonio
Torralba, Kevin P. Murphy, William T. Freeman, and Mark A. Rubin. Context-based vision system for place and object recognition. In ), 14-17 October 2003, Nice, France, pp. 273-280, 2003.
Zhuowen Tu and Xiang Bai. Auto-context and its application to high-level vision tasks and 3d brain image segmentation. IEEE Trans. Pottern Anal. Mach. Intell., 32(10):1744 1757, 2010.
Shuai Zheng, Sadeep Jayasumana, Bernardino Romera-Paredes, Vibhav Vineet, Zhizhong Su, Dalong Du, Chang Huang, and Philip H. S. Torr. Conditional random fields recurrent neural networks. In ICCV, Santiago, Chile, December 7-13, 2015, pp. 1529 1537, 2015.
10


Reviews:
[{"review_id": "rkgZwtqI9r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a contextual reasoning module following the approach proposed by the NIPS 2011 paper for object detection. Specifically, the algorithm proposed by NIPS 2011 are first converted to end-to-end modules step-by-step, and then added to the detection framework Faster R-CNN. The comparison is only to one other baseline (Relation Module), and some improvements are show especially for small objects.\n\n+ Context reasoning is an important but unsolved problem in vision, it is a good attempt to apply CRF to object detection\n\n- The subset results are not convincing. For example, AP_L has decreased dramatically from 51.2 to 44.9. This is shocking. Large objects should maintain a similar performance if the contextual module really works. This result is actually very negative to me.\n- I am not sure the idea of applying conCNN to 6 categories of COCO is a great idea to begin with: context reasoning requires a lot of categories, especially those with few examples to benchmark with. If the categories are supplied with a lot of training examples, it is very hard for context to really help there. I would at least stick to full set of COCO (I am fine with reducing the number of training images), or switch to the recently proposed LVIS dataset:\nhttps://www.lvisdataset.org/\n- I think the work did a less comprehensive literature survey for object detection with context reasoning. For example: \nQi, Lu, et al. \"Sequential context encoding for duplicate removal.\" Advances in Neural Information Processing Systems. 2018.\nChen, Xinlei, and Abhinav Gupta. \"Spatial memory for context reasoning in object detection.\" Proceedings of the IEEE International Conference on Computer Vision. 2017.\nThere are actually many other attempts from different groups that try to nail down this problem, please at least review them in the paper.\n\nOverall this is a paper that's not ready. There might be technical contributions (still with the presence of Zheng et al ICCV 2015, it is limited), but the paper needs to be refined in many ways to get accepted.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "Sylu28cFYB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a context-aware neural network (conCNN) that integrates context semantics into account for object detection. The proposed approach achieves this by embedding a context-aware module into the Faster R-CNN detection framework. The context-aware module simulates the learning process of Conditional Random Fields (CRF) model using a stack of common CNN operations. Specifically, this paper employs the mean-field approach of [1]. Experiments are performed on COCO dataset.\n\nThe  paper reads well. The idea of combining the strengths of both CNNs and CRF-based graphical models in a detection framework is interesting. Though the main idea is borrowed from the mean-field approach of [1], it has been successfully reformulated as a stack of common CNN layers. However, my main concern is that the experimental results (Page 8 Table 2) does not support the merits of the proposed approach. When comparing with the existing Faster R-CNN + Relation work of [2], the improvement provided by the proposed approach is negligible. In fact the proposed approach achieves inferior results on large objects (APL), compared to [2]. [1] uses attention to model the geometric relationships among objects in the same image. In addition to geometric relationships, the proposed approach leverages the inherent co-occurence relationships between object classes. However, this additional information does not seem to provide much help by looking at the results on COCO dataset. The paper does present an experiment to justify this additional information in Table 1. However, that small experiment is only on 6 object categories and does not seem to generalize when going towards full COCO dataset with 80 categories (Table 2). Therefore, the reviewer would recommend to thoroughly evaluate the merits of the proposed approach in depth on COCO and on additional datasets (i.e., Visual Genome). \n\n[1] Philipp Krähenbühl, Vladlen Koltun: Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials. NIPS 2011.\n[2] Han Hu, Jiayuan Gu, Zheng Zhang, Jifeng Dai, Yichen Wei: Relation Networks for Object Detection. CVPR 2018. \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BJguUtsIcS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a CRF-based context module for CNN-based object detectors. In particular for the two-stage region-based detector, like Faster RCNN, the context module is added right before the output layer of the classification head. Every box proposed by the RPN is a node in the CRF, and its label is the classification label. Message passing is unrolled as neural network layers. Potentials are defined based on object detector outputs, box overlap, and co-occurrence of class labels. Experiments are performed on the MS COCO object detection task. \n\nThe proposed method is similar to the CRF-based method for FCN segmentation networks except that the nodes are boxes instead of pixels. And simpler ideas had been explored in the pre-DL era. In this sense, the main idea of this work is incrementally novel. \n\nThe presentation of the paper is fine in general. But the “global message passing” paragraph in Sec 4.2 can be improved. In particular, how p(l) is used in message passing? It is better to provide more specific descriptions so that the paper is self-contained.\n\nThe proposed model outperformed the baseline with a reasonably significant margin. However, its average accuracy on all 80 COCO categories is no better than “the Faster RCNN + Relation”. Since the proposed model is also not significantly simpler than the Relation Network, the experimental results do not establish a new state-of-the-art. \n\nThe experimental results are also a bit thin. More ablative studies can be helpful, e.g., global message parsing/local message passing.\n\nGiven the moderate novelty (which is good but not good enough as a standalone reason to accept this paper) and the not-strong-enough experimental results, I feel more work is needed for the paper to be readily publishable. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a contextual reasoning module following the approach proposed by the NIPS 2011 paper for object detection. Although the reviewers find the proposed approach reasonable, the experimental results are weak and noisy. Multiple reviewers believe that the paper will benefit from another review cycle, pointing out that the authors response confirmed that multiple additional (or redoing of) experiments are needed. 

Decision: Reject