Context-Aware Object Detection With Convolutional Neural Networks

1 Introduction . In recent years , deep convolutional neural networks ( CNN ) ( Goodfellow et al. , 2016 ) have been used with great success in object detection tasks . However , some well known principles in computer vision that have been shown to be effective in object detection are largely overlooked . In particular , the vision community has shown that the semantic context , namely the correlations among the objects , helps object detection Rabinovich et al . ( 2007 ) . That is , when performing the task of object detection , objects ’ class labels should be inferred with respect to other objects in the scene . For example , when detecting objects in a scene of a baseball match in the COCO dataset , which contains four objects : “ Baseball ” , “ Baseball bat ” , “ human ” , and “ baseball glove ” , the CNN-based object detection model tends to recognize the “ baseball bat ” as “ tooth brush ” . Yet it can easily be seen that this object of “ tooth brush ” does not fit into the context with other labels . This is because the state-of-the-art object detection methods ( He et al. , 2015 ; Girshick , 2015 ; Ren et al. , 2015 ; Dai et al. , 2016 ; Lin et al. , 2017 ; He et al. , 2017 ) adopt the region-based paradigm since it was introduced in the seminal R-CNN work ( Girshick et al. , 2014 ) . Given a set of region proposals , these methods perform object classification and bounding box regression on each proposal individually , without taking the semantic context into consideration . In this work , we now propose a context-aware object detection strategy called conCNN to address the above shortcoming . By seamlessly plugging a context-aware module into the existing CNN-based object detection paradigm , conCNN automatically learns and effectively enforces the semantics context constraints , yet requiring minimal modification of the existing deep object detection architecture . conCNN is inspired by our observation that probabilistic graphical models , in particular , Conditional Random Fields ( CRFs ) have been successful in enhancing the accuracy of low level computer vision tasks ( Ladicky et al. , 2009 ; Zheng et al. , 2015 ; Rabinovich et al. , 2007 ; Krähenbühl & Koltun , 2011 ) such as image segmentation or object detection . More specifically , CRF models the label assignment problem as a probabilistic inference problem such that constraints can be incorporated , for example , the label agreement between similar pixels in image segmentation . We thus design a context-aware module that for the first time leverages CRF in deep object detection architecture to maximize the contextual agreement among the objects in the same scene , yet only slightly increasing the training time of the object detection model . The overall context-aware deep object detection network , corresponds to a deep architecture with a traditional CNN-based module concatenated by a context-aware module , can be trained end-to-end following the common practice of back-propagation in deep neural networks . Therefore , we successfully combine the strengths of both CNN and CRF-based graphical models in one unified framework . More specifically , the context-aware module reformulates the iterative mean-field inference ( Krähenbühl & Koltun , 2011 ) – a popular approximate inference method for CRF as a stack of common CNN layers . This way , it learns a compatibility matrix as normal parameters of deep neural network that represents the probabilities of label co-occurrences , as the compatibility function in CRF . Our experimental evaluation on the COCO datasets confirms that conCNN improves the AP of object detection by 2 percentage points . 2 Network Architecture of conCNN . In this work , we proposal a context-aware neural network ( conCNN ) for object detection . Essentially , we achieve this by plugging a context-aware module into a CNN network built upon the popular region-based object detection paradigm as shown in Fig . 1 . Region-based Object Detection . The region-based object detection paradigm ( Girshick et al. , 2014 ) first processes the whole image with several convolutional and max pooling layers ( the CNN backbone ) to produce a conv feature map . Taking the feature map as input , the Region Proposal Network ( RPN in short ) generates a set of rectangular object proposals that most likely contain objects . Then for each object proposal , a region of interest ( RoI ) pooling layer extracts a fixed-length feature vector from the feature map . Each feature vector is fed into a sequence of fully connected layers ( FC layers ) that finally branch into two sibling output layers corresponding to the classification task and the bounding box regression task . The classification task produces softmax probability estimates over L object classes . The bounding box regression task outputs four real-valued numbers for each of the L object classes . Each set of 4 values encodes refined bounding box positions for one of the L classes . Context-aware Module . Our context-aware module is plugged into the classification task . Specifically , it takes the prediction from the last FC layer in the classification task as well as the location for each bounding box in the regression task as input and produces a modified prediction that incorporates the context semantics . The modified prediction distribution is then fed into the softmax layer to generate the final probability estimates . 3 Conditional Random Field in Object Detection . In this section we briefly overview Conditional Random Fields ( CRF ) for object detection . In object detection , a CRF models object labels ( classes ) as random variables that form a Markov Random Field ( MRF ) when conditioned upon the input image . Conditional Random Field . Let Xi be the random variable w.r.t . object i . It represents the label assigned to the object i and can take any value from a pre-defined set of labels L = { l1 , l2 , ... , lL } . Let X be the vector formed by the random variables X1 , X2 , ... , XN , where N denotes the number of objects in the image I . Then the probability of some label assignment x conditional on the global observation I is given by the CRF model : P ( X = x|I ) = 1 Z ( I ) exp ( −E ( x ) ) ( 1 ) Here E ( x ) denotes the energy of the configuration x ∈ LN , while Z ( x ) denotes the partition function ( Lafferty et al. , 2001 ) . 3.1 Mean-field Approximation . Minimizing the above CRF energy E ( x ) yields the most probable label assignment x ( maximizing P ( x ) ) for the given image . Since this exact minimization is intractable , in the fully connected pairwise CRF model , a mean-field approximation to the CRF distribution ( Krähenbühl & Koltun , 2011 ) is used for approximate maximum posterior marginal inference . It approximates the CRF distribution P ( X ) using a simpler distribution Q ( X ) , which can be expressed as the product of independent marginal distributions , Q ( X ) = ∏N i Qi ( Xi ) , where Qi ( Xi ) corresponds to one object i in image I. Leveraging Krähenbühl & Koltun ( 2011 ) for object detection , we use the following update equation to iteratively compute a valid Qi ( Xi ) : Qi ( Xi = l ) = 1 Z ( I ) exp −ψu ( Xi = l ) ︸ ︷︷ ︸ Unary Component − ∑ l′∈L µ ( l , l′ ) K∑ m=1 wm ∑ i6=j km ( fi , fj ) Qj ( Xj = l ′ ) ︸ ︷︷ ︸ Pairwise Component ( 2 ) In Equation 2 , the unary component −ψu ( Xi = l ) ( denoted as Ui ( l ) ) measures the probability that the label l is assigned to the object i ( Xi = l ) . As shown in Fig . 1 , in our conCNN it is obtained from the classification task , which assigns label to object i without considering the context semantics . The pairwise component ∑ l′∈L µ ( l , l ′ ) ∑K m=1 w m ∑ i 6=j k m ( fi , fj ) Qj ( xj = l ′ ) reflects the pairwise influence of the label assignment ( Xj = l′ ) of objects j on the label assignment of object i ( Xi = l ) . Generally speaking , The pairwise component measures the cost of assigning labels l , l′ to objects i , j simultaneously . It enforces the context semantics constraints . Among this pairwise component , function µ ( l , l′ ) captures the compatibility ( or the possibility of co-occurrence ) between a pair of labels l and l′ . It corresponds to the new parameters that have to be learned in the training of conCNN to enforce the context semantics constraints . Each function km ( fi , fj ) for m = 1 , ... , M , represents a function applied on feature vectors ( fi , fj ) of objects i and j . The feature vector fi of object i corresponds to some characteristics of object i derived from a CNN . In our scenarios , fi corresponds to the location of the bounding box or the probability estimation of object i taking label l ∈ L. Essentially , km ( fi , fj ) reflects how strongly an object i is related to other objects j , hence called relation function . Note in Krähenbühl & Koltun ( 2011 ) and Zheng et al . ( 2015 ) which apply CRF in image segmentation , km ( fi , fj ) corresponds to the Gaussian kernel . However , in object detection Gaussian kernel does not necessarily work well ( Rabinovich et al. , 2007 ) . In Sec . 4.2 , we will introduce in more details the relation functions used in our work . Algorithm 1 Mean-field for dense pairwise CRFs 1 : Qi ( l ) ← 1Z ( I ) exp ( Ui ( l ) ) ; . Initialization 2 : while not converged do 3 : Q̂mi ( l ) ← ∑ j 6=i k m ( fi , fj ) Qj ( l ) ; . Message passing from all Xj to all Xi 4 : Q̂i ( l ) ← ∑ m w mQ̂mi ( l ) ; . Weighting 5 : Q̂i ( l ) ← ∑ l′∈L µ ( l , l ′ ) Q̂i ( l ′ ) ; . Compatibility transform 6 : Q̂i ( l ) ← Ui ( l ) - Q̂i ( l ) ; . Local Update 7 : Qi ( l ) ← 1Z ( I ) exp ( Q̂i ( l ) ) ; . Normalization 8 : end while 3.2 Mean-field Algorithm . The update equation ( Equation 2 ) leads to the mean-field algorithm summarized in Algorithm 1 ( Krähenbühl & Koltun , 2011 ) . Each iteration of Algorithm 1 performs a message passing step , a compatibility transform , a local update , and a normalization . Next , we show how to reformulate all these steps in Algorithm 1 as common CNN layers and hence effectively enforces the context semantics constraints in CNN-based object detection , using the classical practice of back propagation in deep neural networks . 4 CRF Inference As Common CNN Layers . To reformulate the mean-field inference as common CNN layers , it is essential to be able to calculate error differentials in each layer w.r.t . its inputs in order to back-propagate the error differentials to previous layers during training . This way , CRF parameters such as the label compatibility function µ ( l , l ′ ) can be learned automatically during the training of the full network . Therefore , in this section we also discuss how to calculate error differentials with respect to the parameters in each layer .

Summary: This paper proposes a CRF-based context module for CNN-based object detectors. In particular for the two-stage region-based detector, like Faster RCNN, the context module is added right before the output layer of the classification head. Every box proposed by the RPN is a node in the CRF, and its label is the classification label. Message passing is unrolled as neural network layers. Potentials are defined based on object detector outputs, box overlap, and co-occurrence of class labels. Experiments are performed on the MS COCO object detection task.