FAST OBJECT LOCALIZATION VIA SENSITIVITY ANALYSIS

1 INTRODUCTION . Deep Convolutional Neural Networks ( CNNs ) have been shown to be effective at image classification , accurately performing object recognition even with thousands of object classes when trained on a sufficiently rich data set of labeled images Krizhevsky et al . ( 2012 ) . One advantage of CNNs is their ability to learn complete functional mappings from image pixels to object categories , without any need for the extraction of hand-engineered image features Sermanet et al . ( 2013 ) . To facilitate learning through stochastic gradient descent , CNNs are ( at least approximately ) differentiable with regard to connection weight parameters . Image classification , however , is only one of the problems of computer vision . In the task of image classification , each image has a single label , associated with the class identity of the main object in the image , and the goal is to assign correct labels in a manner that generalizes to novel images . This can be accomplished by training a machine learning classifier , such as a CNN , on a large data set of labeled images Deng et al . ( 2009 ) . In the object localization task , in comparison , the output for a given image is not a class label but the locations of a specified number of objects in the image , usually encoded as bounding boxes . Evaluation of an object localization system generally requires ground truth bounding boxes to compare to the system ’ s output . The detection task is more difficult than the localization task , as the number of objects are not predetermined Sermanet et al . ( 2013 ) . In this paper , we focus on object localization , identifying the position in the image of a recognized object . As is common in the localization literature , position information is output in the form of a bounding box . Previously developed techniques for accomplishing this task generally involve searching the image for the object , considering many candidate bounding boxes with different sizes and locations , sometimes guided by an auxilliary algorithm for heuristically identifying regions of interest Sermanet et al . ( 2013 ) ; Girshick ( 2015 ) ; He et al . ( 2017 ) . For each candidate location , the sub-image captured by the bounding box is classified for object category , with the final output bounding box either being the specific candidate region classified as the target object with the highest level of certainty or some heuristic combination of neighboring or overlapping candidate regions with high classification certainty . These approaches tend to be time consuming , often requiring deep CNN classification calculations of many candidate regions at multiple scales . Efforts to speed these methods mostly focus on reducing the number of regions considered , typically by using some adjunct heuristic region proposal algorithm Girshick ( 2015 ) ; Ren et al . ( 2015 ) ; He et al . ( 2017 ) . Still , the number of considered regions is often reported to be roughly 2,000 per image . While these approaches can be fairly accurate , their slowness limits their usefulness , particularly for online applications . A noteworthy alternative approach is to directly train a deep CNN to produce outputs that match ground truth localization bounding boxes , using a large image data set that provides both category and localization information for each image . It appears as if some form of this method was used with AlexNet Krizhevsky et al . ( 2012 ) , though details concerning localization , rather than image classification , are difficult to discern from the published literature . A natural approach would be to cast the learning of bounding boxes as a simple regression problem , with targets being the four coordinates that specify a bounding box ( e.g. , coordinates of upper-left and lower-right corners , or region center coordinates along with region width and height ) . It is reasonable to consider sharing early layers of a deep CNN , such as those performing convolution and max pooling , between both an image classification network and an object localization network . Indeed , taking such a multitask learning approach Caruana ( 1997 ) can allow for both object category and object location training data to shape connection weights throughout the network . Thus , the deep CNN would have ” two heads ” , one for image classification , using a classification cross-entropy loss function , and one for object localization , reducing the ` 2 norm between ground truth and predicted bounding box coordinates Krizhevsky et al . ( 2012 ) . While this approach can produce a network that quickly outputs location information , extensive training on large data sets containing ground truth bounding box information is necessary to produce good generalization . In this paper , we introduce an approach to object localization that is both very fast and robust in the face of limited ground truth bounding box training data . This approach is rooted in the assertion that any deep CNN for image classification must contain , implicit in its connection weights , knowledge about the location of recognized objects Selvaraju et al . ( 2016 ) . The goal , then , is to interpret the flow of activation in an object recognition network when it is performing image classification so as to extract information about object location . Furthermore , the goal is to do this quickly . Thus , this approach aims to leverage location knowledge that is already latent in extensively trained and tuned image classification networks , without requiring a separate learning process for localization . Our method makes use of the notion of a sensitivity analysis Sobol ( 1993 ) . We propose estimating the sensitivity of the category outputs , or activation patterns at internal network layers , of an image classification CNN to variance in each input pixel , given a specific input image . The result is a numeric value for each pixel in the input image that captures the degree to which small changes in that pixel ( locally , around its current value ) give rise to large changes in the output category . Together , these numeric values form a sensitivity map of the image , encoding image regions that are important for the current classification . Our proposed measure of sensitivity is the partial derivative of activity with regard to each pixel value , evaluated for the current image . For a deep CNN that formally embodies a differentiable mapping ( at least approximately ) from image pixels to output categories , this partial derivative can be quickly calculated . While many tools currently exist for efficiently calculating such derivatives , we provide a simple algorithm that computes these values through a single backward pass through the image classification network , similar to that used to calculate unit error ( delta ) values in the backpropagation of error learning algorithm Rumelhart et al . ( 1986 ) . Thus , we can generate a sensitivity map for an image in about the same amount of time as it takes the employed image classification network to produce an output . Some example sensitivity maps are shown in Figure 1 . The idea of using sensitivity information , like that in our sensitivity maps , for a variety of tasks , including localization , has previously appeared in the literature Simonyan et al . ( 2013 ) ; Zhou et al . ( 2016 ) ; Selvaraju et al . ( 2016 ) . Indeed , some of these past efforts have used more sophisticated measures of sensitivity . In this paper , we show that even our very simple sensitivity measure can produce strong localization performance , and it can do so quickly , without any modifications to the classification network , and even for object categories on which the classification network was not trained . The relationship of the results reported here to previously reported work is discussed further in Section 4 . As previously mentioned , object localization methods typically encode object location as a bounding box . Since our sensitivity maps encode location differently , in terms of pixels , we propose learning a simple linear mapping from sensitivity maps to bounding box coordinates , allowing our method to output a bounding box for each classified image . We suggest that this linear mapping can be robustly learned from a relatively small training set of images with ground truth bounding boxes , since the sensitivity maps form a much more simple input than the original images . The primary contributions of this paper may be summarized as follows : • We propose a new general approach to performing object localization , interpreting a previously trained image classification network by performing a sensitivity analysis , identifying pixels to which the category output , or a more general internal representation , is particularly sensitive . • We demonstrate how a linear function from the resulting sensitivity maps to object location bounding box coordinates may be learned from training images containing ground truth location information . • We provide a preliminary assessment of our approach , measuring object localization performance on the ImageNet and PASCAL VOC data sets using the VGG16 image classification CNN , showing strong accuracy while maintaining short computation times . 2 METHOD . 2.1 CALCULATING PIXEL SENSITIVITIES IN A TRAINED CNN . Calculating derivatives of a function of network output with regard to network parameters , such as connection weights , is a standard part of CNN training . It is common for learning in a deep CNN to involve stochastic gradient decent , which involves such derivatives . In that case , the derivatives are of an objective function with regard to connection weight values . In image classification networks , the objective function is designed to have optima where training images are correctly classified . In the case of object localization , a similar objective function could be designed to minimize differences between output bounding box coordinates and provided ground truth bounding box coordinates , for all images in an appropriately labeled training set . For example , given N training images , stored in the matrix X , with the ground truth 4-dimensional bounding box vector for image xi being yi , and G ( xi ; w ) being the CNN output vector for image xi given connection weights w , an appropriate loss function would be : ` ( X , w ) = 1 N N∑ i=1 ‖yi −G ( xi ; w ) ‖22 ( 1 ) The CNN will produce good estimates of the training image bounding boxes when this loss function is minimized with regard to w. Network weight parameters that minimize this loss , w∗ , may be sought through stochastic gradient decent , incrementally updating w according to the gradient of ` ( X , w ) with regard to w. A primary drawback of this approach is that it requires a large and representative sample of images with ground truth bounding box information . Consider that , once weights are found , the gradient of ` ( X , w∗ ) with regard to X would provide information about the sensitivity of the bounding box loss function with regard to the pixels in the images . This gradient can be calculated as efficiently as the gradient of the loss with regard to the weights , with both depending on the gradient of G ( xi ; w ) with regard to a subset of its arguments . This means that the gradient of G ( xi ; w∗ ) with regard to xi can be efficiently computed , and that gradient would capture the sensitivity of bounding box coordinates with regard to the specific pixels in image xi . Note that this gradient can be calculated for images beyond those in the training set . Knowing which pixels in a novel image play an important role in determining the bounding box provides useful information for object localization . Using this calculation to address the object localization task makes little sense , however , as G ( xi ; w∗ ) provides an estimate of object location without a need to consider pixel sensitivity . Rather than training a deep CNN to output bounding boxes , requiring extensive labeled data , we propose calculating the same gradient for a different network – one successfully trained to perform image classification . If we now seeG ( xi ; w∗ ) as the output of such an image classification network , its gradient with regard to xi would provide information about the sensitivity of the assigned category to individual pixels . Pixels with the largest absolute values of this derivative will , around the input xi , produce the largest changes in the classification decision of the CNN . This can be seen as one measure of how important pixels are for classifying the object in the image . Consider that the object class output is not immediately affected by changes to pixels with a derivative of zero . The calculation of this gradient can be performed as efficiently as a single ” backward pass ” through the classification network . This is well illustrated by considering the case of a simple layered backpropagation network Rumelhart et al . ( 1986 ) in which the ” net input ” of unit i , ηi , is a weighted sum of the activations of units in the previous layer , and the activation of unit i is g ( ηi ) , where g ( · ) is the unit activation function . In this case , we can define a sensitivity value for each unit , si , as the derivative of the network output with regard to ηi . Using the chain rule of calculus , it is easy to show that the sensitivity of an output unit is g′ ( ηi ) , and , for units in earlier layers the gradients are computed as follows : si = g ′ ( ηi ) ∑ k wki sk ( 2 ) where k iterates over all units in the immediately downstream layer from unit i and wki is the connection weight from unit i to unit k. This calculation may be performed , layer by layer , from outputs to inputs , until si values for each pixel input unit are available . This demonstrates how efficiently pixel sensitivity values can be calculated for a given classified image . Of course , there are currently a variety of software packages that include tools for calculating gradients . In the evaluation of our approach in Section 3 , we report results using the tools provided by TensorFlow Abadi et al . ( 2015 ) .

Summary: The paper focuses on improving object localization, though the title highlights "interpreting deep neural network" which is another area. It analyzes the classifier weights for image classification, and compute the derivative of the feature maps from the network for a sensitivity map of the image. Then it learns a simple linear mapping over the sensitivity map for bounding box regression. Experiments report competitive performance.