Count-guided Weakly Supervised Localization Based on Density Map

1 INTRODUCTION . Deep convolutional neural networks ( CNN ) have significantly pushed the frontier of image processing . Trained with enormous amount of data , CNNs have surpassed human performance in many object detecting tasks . However , humans are still ahead in many aspects . One of them is the segmentation based on abstract knowledge : once a human can tell what is in an image , he/she can easily crop the objects out . Researches have been exploring similar potentials of CNNs . Related techniques are referred to as Weakly Supervised Localization ( WSL ) , in which models are trained with only image-level labels but expected to give the positions of objects at once . Most of existing WSL methods are guided by class labels and realized in two different ways . The first is the Multiple Instance Mining ( MIM ) . MIM is based on multiple instance learning ( MIL ) , which trains a model to identify multiple instances from incomplete labels ( Babenko , 2008 ; Siva et al. , 2012 ) . For object detection , the regions around the target have the highest influence on the label decision . By evaluating the contributions of different regions , the location of the target can be determined . A straightforward approach to find the relevant regions is to compute the derivative of the decision with respect to the input image ( Simonyan et al. , 2013 ) . But the found regions can also be the ones that introduce the most uncertainty to the result . A more effective approach is to use an attention mechanism , in which one detector proposes potential regions according to a classifier ’ s evaluation of contribution ( Gokberk Cinbis et al. , 2014 ; Song et al. , 2014a ; Bilen et al. , 2014 ; Ren et al. , 2016 ; Cinbis et al. , 2017 ) . The second way to WSL is to train the locations as latent variables . Zhou et al . reported that a deep CNN can maintain spatial information through layers ( Zhou et al. , 2016 ) . Based on this observation they proposed the class activation map ( CAM ) which averages every channel of a CNN ’ s final layer and feeds the averages to a soft-max classifier . This structure is analogous to the density map used for object counting . The density map method annotates each object by one pixel with a value of one and computes the count by summing up the map ( Lempitsky & Zisserman , 2010 ) . Inspired by the similarities between the CAM and the density map , we generalize WSL to counting models based on density map . Our method trains the density map as latent variables of an end- to-end CNN by giving only the final count of objects . It is observed that the trained density map can roughly capture the positions of objects , but the accuracy is degraded by the convolution and pooling operations . This paper will discuss the impacts of these two operations on an end-to-end WSL network and propose techniques to improve accuracy . The density map counts objects by detecting discriminative features . To improve interpretability of the found features , we introduce a Gini impurity penalty to regularize the density map . Furthermore , the similarities and differences between this regularization and the variational term of the β-variational autoencoder ( VAE ) ( Higgins et al. , 2017 ) will be discussed . Throughout this paper , we will use a simple simulated bubble dataset to demonstrate basic ideas and a widely used crowd-counting dataset the Mall Loy et al . ( 2013 ) ; Chen et al . ( 2013 ) ; Change Loy et al . ( 2013 ) ; Chen et al . ( 2012 ) to show the methods ’ effectiveness in practice . 2 RELATED WORK . 2.1 MULTIPLE INSTANCE MINING . Coarsely labelled images are labelled only by the class of the main object . However , due to the usage of softmax function as output and the cross entropy for training , a classifier has the potential to identify multiple instances ( Babenko , 2008 ) . A MIM WSL method trains a detector to propose either a bounding box ( Bilen & Vedaldi , 2016 ; Tang et al. , 2017 ) or a segmentation ( Kolesnikov & Lampert , 2016 ; Wei et al. , 2017 ) that highlights the object in the image . A few candidates for bounding boxes or segmentations are initialized at the beginning , and the detector is trained by trial and error . This always results in a non-convex optimization problem that easily converges to local optimum . Solutions for this problem focus on both the initialization ( Deselaers et al. , 2012 ; Siva et al. , 2013 ; Shi et al. , 2013 ; Song et al. , 2014b ) and the training strategy ( Crandall & Huttenlocher , 2006 ; Cinbis et al. , 2017 ; Li et al. , 2016 ) . This attention mechanism can also be realized in an adversarial style ( Wei et al. , 2017 ; Shen et al. , 2018 ; Bartz et al. , 2018 ) . Among all MIM methods , the min-entropy latent model ( MELM ) and the count-guided weakly supervised localization ( C-WSL ) are mostly related to this work . The MELM ( Wan et al. , 2018 ) uses the min-entropy metric to measure and reduce the randomness of localization proposals . Likewise , our method leverages the Gini impurity to decrease the complexity of the latent density map . The C-WSL ( Gao et al. , 2018 ) takes advantage of per-class object count labels to reduce label cost and improve performance . Even though it is also guided by count , it applies a MIM paradigm in contrast to our end-to-end density-map-based approach . 2.2 DENSITY MAP COUNTING . Density map is a counting technique based on detection ( Lempitsky & Zisserman , 2010 ) . Combined with CNNs , it has achieved state-of-the-art performance in tasks like cell ( Xie et al. , 2018 ) , crowd ( Zhang et al. , 2016 ) , and wild animal ( Arteta et al. , 2016 ) counting . Commonly , counting CNNs have the structure of a Fully Convolutional Neural Network ( FCNN ) ( Xie et al. , 2018 ) , because images for counting usually contain similar objects spreading all over the frame , and a FCNN can process images of arbitrary sizes . 2.3 CLASS ACTIVATION MAPPING . The CAM localizes the positions of discriminative features by applying the global average pooling ( GAP ) to the final layer of a CNN . Because it is trained in a MIL way , the CAM can identify multiple features in a image ( Zhou et al. , 2016 ) . A trained CAM can also be used for proposing candidates in a MIM method ( Chen et al. , 2018 ) . To some extent , CAM is analogous to density map counting , as averaging the channels is equivalent to integrating the density map . On the other hand , CAM turns a CNN with fully connected layers to a FCNN . 2.4 VARIATIONAL AUTOENCODER . VAE ( Kingma & Welling , 2013 ) or more generally β-VAE ( Higgins et al. , 2017 ) is an unsupervised data exploring method . The hidden features of the dataset are learned by training a network to compress and reconstruct its input . Experiments and theories have shown that the variational term of the VAE forces an auto-encoder to learn more disentangled latent features ( Higgins et al. , 2017 ; Alemi et al. , 2016 ; Pu et al. , 2016 ; Kim & Mnih , 2018 ; Burgess et al. , 2018 ) . In the following work , we will use a similar technique to make the density map yield interpretable results . 3 COUNT-GUIDED WSL BASED ON DENSITY MAP . In this section , we investigate the count-guided weakly supervised localization based on density map . We first describe the basic counting structure that takes the density map as latent variables and then discuss the impact of pooling and convolution on this end-to-end WSL network . Finally , we propose the Gini impurity penalty to control the representation of a density map and show its relation with the β-VAE . 3.1 BASIC COUNTING STRUCTURE . The feature extractor is a FCNN that outputs a map fΘ ( I ) of the same size as the input image I , where Θ are the parameters of the network . The map is activated by the rectified linear unit ( ReLU ) and then goes through a hyperbolic tangent function to make sure all its elements are either zero or less than one ; i.e . D = tanh ( f ) . Finally , the estimated count ĉ is the integral of the density map : ĉ = ∑ DΘ ( I ) ( 1 ) At the training stage , we only provide the final count c of objects as ground truth . Θ̂ = arg min Θ Edata [ ( c− ĉ ) 2 ] ( 2 ) The goal is to train the latent density map for localization : Θ̂ = arg max Θ EqΘ ( D|I ) [ log ( p ( c|D ) ) ] ( 3 ) In terms of training , equation 2 and 3 are identical . Following the notation of VAE ( Burgess et al. , 2018 ) , we expose the hidden density map in 3 for further analysis . 3.2 SPATIAL INFORMATION LOSS . Conventionally , a CNN applies pooling layers to reduce computations and extract abstract features . However , as a down-sampling operation , pooling does harm to the spatial information going through a deep network . Here we use a simple bubble counting task to study the impact of pooling on this end-to-end WSL network . 3.2.1 SETUP OF BUBBLE COUNTING . The simulated data is a set of 64×64 binary images with a randomly selected number of 1 to 15 bubbles in the frame , as shown in Figure 1a . The bubble centers are located within a square as shown in Figure 1b and Figure 2d , and their radii are 3 to 5 pixels . Slight overlaps are allowed . The goal is to localize the bubble centers by providing only the counts for training . Table 1 shows the structure of the FCNN for this test . It uses max-pooling following the second , third , and fourth convolutional layers to down-sample features . After reaching the bottleneck , it uses transposed convolutions ( deconvolution ) for staged up-sampling to finally match the input size at the output . 3.2.2 SUPERVISED LOCALIZATION . Firstly , we make sure this FCNNs capacity is sufficient for this localizing task . Both the ground-truth density maps D and the final counts c are provided for training : Θ̂ = arg min Θ Edata [ ( c− ĉ ) 2 + ∑ ( D− D̂ ) 2 ] ( 4 ) We generate a mini-batch of 10 samples for every training step and a set of 500 examples for testing . After 3 × 105 evaluations , the average pixel-by-pixel error of the density map converges to a sufficiently small value of 0.001 . An example of the estimated centers is shown in Figure 1c . This demonstrates that the proposed network can count as well as localize bubbles . 3.2.3 WEAKLY SUPERVISED LOCALIZATION . Now we use the same training strategy to train the network again but provide it only with the final counts ( equation 2 ) . After 3 × 105 epochs , the counting error converges to 0.02 . But , as shown in Figure 2c , the localization is not as good as in the case of supervised localization and shows two artifacts : Firstly , the centers are relocated to match a low resolution grid . Figure 2e compiles the statistics of 500 test samples to draw this grid clearly . This spatial information loss is caused by the pooling layers , and it can not be recovered by up-samplings . In contrast , this loss of detail does not happen in the case of supervised localization ( Figure 1 ) , because the presence of the true centers in the labels forces the network to retain position information even through down-sampling processes . Due to this drawback , we will not use any pooling layers in the following WSL models . Secondly , the estimated centers as a whole drift to the right bottom corner of the density map . We believe this is because CNNs rely on edge detection , and our training data is homogeneous . The output of the first convolutional layer ( sum of all channels ) in Figure 3 shows that the trained kernels highlight the right bottom edges more than other sides . Going through all the layers , this bias will finally show in the density map . Zhou et al . ( 2016 ) discussed that the GAP , unlike the max-pooling , encourages the network to identify the full extent of the object instead of certain edges . However , in this count-guided case , the bias happens before any pooling layers ( Figure 3 ) , and the averaging ( sum ) can not recover the correct positions of the centers ( Figure 2 ) . Thus , we resort to a direct solution to tackle this problem .

Summary: The main contribution of the paper is the extension of techniques for weakly supervised localization, i.e. given ground truth counts of objects in a given image, one can do training to generate hidden layer density maps that allow for feature detection and localization of objects.   The main contribution of the paper seems to be the regularization of the density map by incorporation of a Gini impurity penalty and the contrasting of the regularizer against beta-variational autoencoder formulation. The experiments show the utility of the method in a toy example and an example involving pedestrians in video surveillance.