Context-Aware Object Detection With Convolutional Neural Networks

1 Introduction . In recent years , deep convolutional neural networks ( CNN ) ( Goodfellow et al. , 2016 ) have been used with great success in object detection tasks . However , some well known principles in computer vision that have been shown to be effective in object detection are largely overlooked . In particular , the vision community has shown that the semantic context , namely the correlations among the objects , helps object detection Rabinovich et al . ( 2007 ) . That is , when performing the task of object detection , objects ’ class labels should be inferred with respect to other objects in the scene . For example , when detecting objects in a scene of a baseball match in the COCO dataset , which contains four objects : “ Baseball ” , “ Baseball bat ” , “ human ” , and “ baseball glove ” , the CNN-based object detection model tends to recognize the “ baseball bat ” as “ tooth brush ” . Yet it can easily be seen that this object of “ tooth brush ” does not fit into the context with other labels . This is because the state-of-the-art object detection methods ( He et al. , 2015 ; Girshick , 2015 ; Ren et al. , 2015 ; Dai et al. , 2016 ; Lin et al. , 2017 ; He et al. , 2017 ) adopt the region-based paradigm since it was introduced in the seminal R-CNN work ( Girshick et al. , 2014 ) . Given a set of region proposals , these methods perform object classification and bounding box regression on each proposal individually , without taking the semantic context into consideration . In this work , we now propose a context-aware object detection strategy called conCNN to address the above shortcoming . By seamlessly plugging a context-aware module into the existing CNN-based object detection paradigm , conCNN automatically learns and effectively enforces the semantics context constraints , yet requiring minimal modification of the existing deep object detection architecture . conCNN is inspired by our observation that probabilistic graphical models , in particular , Conditional Random Fields ( CRFs ) have been successful in enhancing the accuracy of low level computer vision tasks ( Ladicky et al. , 2009 ; Zheng et al. , 2015 ; Rabinovich et al. , 2007 ; Krähenbühl & Koltun , 2011 ) such as image segmentation or object detection . More specifically , CRF models the label assignment problem as a probabilistic inference problem such that constraints can be incorporated , for example , the label agreement between similar pixels in image segmentation . We thus design a context-aware module that for the first time leverages CRF in deep object detection architecture to maximize the contextual agreement among the objects in the same scene , yet only slightly increasing the training time of the object detection model . The overall context-aware deep object detection network , corresponds to a deep architecture with a traditional CNN-based module concatenated by a context-aware module , can be trained end-to-end following the common practice of back-propagation in deep neural networks . Therefore , we successfully combine the strengths of both CNN and CRF-based graphical models in one unified framework . More specifically , the context-aware module reformulates the iterative mean-field inference ( Krähenbühl & Koltun , 2011 ) – a popular approximate inference method for CRF as a stack of common CNN layers . This way , it learns a compatibility matrix as normal parameters of deep neural network that represents the probabilities of label co-occurrences , as the compatibility function in CRF . Our experimental evaluation on the COCO datasets confirms that conCNN improves the AP of object detection by 2 percentage points . 2 Network Architecture of conCNN . In this work , we proposal a context-aware neural network ( conCNN ) for object detection . Essentially , we achieve this by plugging a context-aware module into a CNN network built upon the popular region-based object detection paradigm as shown in Fig . 1 . Region-based Object Detection . The region-based object detection paradigm ( Girshick et al. , 2014 ) first processes the whole image with several convolutional and max pooling layers ( the CNN backbone ) to produce a conv feature map . Taking the feature map as input , the Region Proposal Network ( RPN in short ) generates a set of rectangular object proposals that most likely contain objects . Then for each object proposal , a region of interest ( RoI ) pooling layer extracts a fixed-length feature vector from the feature map . Each feature vector is fed into a sequence of fully connected layers ( FC layers ) that finally branch into two sibling output layers corresponding to the classification task and the bounding box regression task . The classification task produces softmax probability estimates over L object classes . The bounding box regression task outputs four real-valued numbers for each of the L object classes . Each set of 4 values encodes refined bounding box positions for one of the L classes . Context-aware Module . Our context-aware module is plugged into the classification task . Specifically , it takes the prediction from the last FC layer in the classification task as well as the location for each bounding box in the regression task as input and produces a modified prediction that incorporates the context semantics . The modified prediction distribution is then fed into the softmax layer to generate the final probability estimates . 3 Conditional Random Field in Object Detection . In this section we briefly overview Conditional Random Fields ( CRF ) for object detection . In object detection , a CRF models object labels ( classes ) as random variables that form a Markov Random Field ( MRF ) when conditioned upon the input image . Conditional Random Field . Let Xi be the random variable w.r.t . object i . It represents the label assigned to the object i and can take any value from a pre-defined set of labels L = { l1 , l2 , ... , lL } . Let X be the vector formed by the random variables X1 , X2 , ... , XN , where N denotes the number of objects in the image I . Then the probability of some label assignment x conditional on the global observation I is given by the CRF model : P ( X = x|I ) = 1 Z ( I ) exp ( −E ( x ) ) ( 1 ) Here E ( x ) denotes the energy of the configuration x ∈ LN , while Z ( x ) denotes the partition function ( Lafferty et al. , 2001 ) . 3.1 Mean-field Approximation . Minimizing the above CRF energy E ( x ) yields the most probable label assignment x ( maximizing P ( x ) ) for the given image . Since this exact minimization is intractable , in the fully connected pairwise CRF model , a mean-field approximation to the CRF distribution ( Krähenbühl & Koltun , 2011 ) is used for approximate maximum posterior marginal inference . It approximates the CRF distribution P ( X ) using a simpler distribution Q ( X ) , which can be expressed as the product of independent marginal distributions , Q ( X ) = ∏N i Qi ( Xi ) , where Qi ( Xi ) corresponds to one object i in image I. Leveraging Krähenbühl & Koltun ( 2011 ) for object detection , we use the following update equation to iteratively compute a valid Qi ( Xi ) : Qi ( Xi = l ) = 1 Z ( I ) exp −ψu ( Xi = l ) ︸ ︷︷ ︸ Unary Component − ∑ l′∈L µ ( l , l′ ) K∑ m=1 wm ∑ i6=j km ( fi , fj ) Qj ( Xj = l ′ ) ︸ ︷︷ ︸ Pairwise Component ( 2 ) In Equation 2 , the unary component −ψu ( Xi = l ) ( denoted as Ui ( l ) ) measures the probability that the label l is assigned to the object i ( Xi = l ) . As shown in Fig . 1 , in our conCNN it is obtained from the classification task , which assigns label to object i without considering the context semantics . The pairwise component ∑ l′∈L µ ( l , l ′ ) ∑K m=1 w m ∑ i 6=j k m ( fi , fj ) Qj ( xj = l ′ ) reflects the pairwise influence of the label assignment ( Xj = l′ ) of objects j on the label assignment of object i ( Xi = l ) . Generally speaking , The pairwise component measures the cost of assigning labels l , l′ to objects i , j simultaneously . It enforces the context semantics constraints . Among this pairwise component , function µ ( l , l′ ) captures the compatibility ( or the possibility of co-occurrence ) between a pair of labels l and l′ . It corresponds to the new parameters that have to be learned in the training of conCNN to enforce the context semantics constraints . Each function km ( fi , fj ) for m = 1 , ... , M , represents a function applied on feature vectors ( fi , fj ) of objects i and j . The feature vector fi of object i corresponds to some characteristics of object i derived from a CNN . In our scenarios , fi corresponds to the location of the bounding box or the probability estimation of object i taking label l ∈ L. Essentially , km ( fi , fj ) reflects how strongly an object i is related to other objects j , hence called relation function . Note in Krähenbühl & Koltun ( 2011 ) and Zheng et al . ( 2015 ) which apply CRF in image segmentation , km ( fi , fj ) corresponds to the Gaussian kernel . However , in object detection Gaussian kernel does not necessarily work well ( Rabinovich et al. , 2007 ) . In Sec . 4.2 , we will introduce in more details the relation functions used in our work . Algorithm 1 Mean-field for dense pairwise CRFs 1 : Qi ( l ) ← 1Z ( I ) exp ( Ui ( l ) ) ; . Initialization 2 : while not converged do 3 : Q̂mi ( l ) ← ∑ j 6=i k m ( fi , fj ) Qj ( l ) ; . Message passing from all Xj to all Xi 4 : Q̂i ( l ) ← ∑ m w mQ̂mi ( l ) ; . Weighting 5 : Q̂i ( l ) ← ∑ l′∈L µ ( l , l ′ ) Q̂i ( l ′ ) ; . Compatibility transform 6 : Q̂i ( l ) ← Ui ( l ) - Q̂i ( l ) ; . Local Update 7 : Qi ( l ) ← 1Z ( I ) exp ( Q̂i ( l ) ) ; . Normalization 8 : end while 3.2 Mean-field Algorithm . The update equation ( Equation 2 ) leads to the mean-field algorithm summarized in Algorithm 1 ( Krähenbühl & Koltun , 2011 ) . Each iteration of Algorithm 1 performs a message passing step , a compatibility transform , a local update , and a normalization . Next , we show how to reformulate all these steps in Algorithm 1 as common CNN layers and hence effectively enforces the context semantics constraints in CNN-based object detection , using the classical practice of back propagation in deep neural networks . 4 CRF Inference As Common CNN Layers . To reformulate the mean-field inference as common CNN layers , it is essential to be able to calculate error differentials in each layer w.r.t . its inputs in order to back-propagate the error differentials to previous layers during training . This way , CRF parameters such as the label compatibility function µ ( l , l ′ ) can be learned automatically during the training of the full network . Therefore , in this section we also discuss how to calculate error differentials with respect to the parameters in each layer .

Summary: This paper introduces a context-aware neural network (conCNN) that integrates context semantics into account for object detection. The proposed approach achieves this by embedding a context-aware module into the Faster R-CNN detection framework. The context-aware module simulates the learning process of Conditional Random Fields (CRF) model using a stack of common CNN operations. Specifically, this paper employs the mean-field approach of [1]. Experiments are performed on COCO dataset.