Title: COUNT-GUIDED WEAKLY SUPERVISED LOCALIZA-TION BASED ON DENSITY MAP Anonymous

Abstract: Weakly supervised localization (WSL) aims at training a model to find the positions of objects by providing it with only abstract labels. For most of the existing WSL methods, the labels are the class of the main object in an image. In this paper, we generalize WSL to counting machines that apply convolutional neural networks (CNN) and density maps for counting. We show that given only ground-truth count numbers, the density map as a hidden layer can be trained for localizing objects and detecting features. Convolution and pooling are the two major building blocks of CNNs. This paper discusses their impacts on an end-toend WSL network. The learned features in a density map present in the form of dots. In order to make these features interpretable for human beings, this paper proposes a Gini impurity penalty to regularize the density map. Furthermore, it will be shown that this regularization is similar to the variational term of the βvariational autoencoder. The details of this algorithm are demonstrated through a simple bubble counting task. Finally, the proposed methods are applied to the widely used crowd counting dataset the Mall to learn discriminative features of human figures.

1. INTRODUCTION:

Deep convolutional neural networks (CNN) have significantly pushed the frontier of image processing. Trained with enormous amount of data, CNNs have surpassed human performance in many object detecting tasks. However, humans are still ahead in many aspects. One of them is the segmentation based on abstract knowledge: once a human can tell what is in an image, he/she can easily crop the objects out. Researches have been exploring similar potentials of CNNs. Related techniques are referred to as Weakly Supervised Localization (WSL), in which models are trained with only image-level labels but expected to give the positions of objects at once. Most of existing WSL methods are guided by class labels and realized in two different ways.

The first is the Multiple Instance Mining (MIM). MIM is based on multiple instance learning (MIL) , which trains a model to identify multiple instances from incomplete labels (Babenko, 2008; Siva et al., 2012) . For object detection, the regions around the target have the highest influence on the label decision. By evaluating the contributions of different regions, the location of the target can be determined. A straightforward approach to find the relevant regions is to compute the derivative of the decision with respect to the input image (Simonyan et al., 2013) . But the found regions can also be the ones that introduce the most uncertainty to the result. A more effective approach is to use an attention mechanism, in which one detector proposes potential regions according to a classifier's evaluation of contribution (Gokberk Cinbis et al., 2014; Song et al., 2014a; Bilen et al., 2014; Ren et al., 2016; Cinbis et al., 2017) .

The second way to WSL is to train the locations as latent variables. Zhou et al. reported that a deep CNN can maintain spatial information through layers (Zhou et al., 2016) . Based on this observation they proposed the class activation map (CAM) which averages every channel of a CNN's final layer and feeds the averages to a soft-max classifier. This structure is analogous to the density map used for object counting. The density map method annotates each object by one pixel with a value of one and computes the count by summing up the map (Lempitsky & Zisserman, 2010) .

Inspired by the similarities between the CAM and the density map, we generalize WSL to counting models based on density map. Our method trains the density map as latent variables of an end- 

2. RELATED WORK:



2.1. MULTIPLE INSTANCE MINING:

Coarsely labelled images are labelled only by the class of the main object. However, due to the usage of softmax function as output and the cross entropy for training, a classifier has the potential to identify multiple instances (Babenko, 2008) . A MIM WSL method trains a detector to propose either a bounding box (Bilen & Vedaldi, 2016; Tang et al., 2017) or a segmentation (Kolesnikov & Lampert, 2016; Wei et al., 2017) that highlights the object in the image. A few candidates for bounding boxes or segmentations are initialized at the beginning, and the detector is trained by trial and error. This always results in a non-convex optimization problem that easily converges to local optimum. Solutions for this problem focus on both the initialization (Deselaers et al., 2012; Siva et al., 2013; Shi et al., 2013; Song et al., 2014b) and the training strategy (Crandall & Huttenlocher, 2006; Cinbis et al., 2017; Li et al., 2016) . This attention mechanism can also be realized in an adversarial style (Wei et al., 2017; Shen et al., 2018; Bartz et al., 2018) .

Among all MIM methods, the min-entropy latent model (MELM) and the count-guided weakly supervised localization (C-WSL) are mostly related to this work. The MELM (Wan et al., 2018) uses the min-entropy metric to measure and reduce the randomness of localization proposals. Likewise, our method leverages the Gini impurity to decrease the complexity of the latent density map. The C-WSL (Gao et al., 2018) takes advantage of per-class object count labels to reduce label cost and improve performance. Even though it is also guided by count, it applies a MIM paradigm in contrast to our end-to-end density-map-based approach.

2.2. DENSITY MAP COUNTING:

Density map is a counting technique based on detection (Lempitsky & Zisserman, 2010) . Combined with CNNs, it has achieved state-of-the-art performance in tasks like cell (Xie et al., 2018) , crowd (Zhang et al., 2016) , and wild animal (Arteta et al., 2016) The CAM localizes the positions of discriminative features by applying the global average pooling (GAP) to the final layer of a CNN. Because it is trained in a MIL way, the CAM can identify multiple features in a image (Zhou et al., 2016) . A trained CAM can also be used for proposing candidates in a MIM method (Chen et al., 2018) . To some extent, CAM is analogous to density map counting, as averaging the channels is equivalent to integrating the density map. On the other hand, CAM turns a CNN with fully connected layers to a FCNN.

2.4. VARIATIONAL AUTOENCODER:

VAE (Kingma & Welling, 2013) or more generally β-VAE (Higgins et al., 2017) is an unsupervised data exploring method. The hidden features of the dataset are learned by training a network to compress and reconstruct its input. Experiments and theories have shown that the variational term of the VAE forces an auto-encoder to learn more disentangled latent features (Higgins et al., 2017; Alemi et al., 2016; Pu et al., 2016; Kim & Mnih, 2018; Burgess et al., 2018) . In the following work, we will use a similar technique to make the density map yield interpretable results.

3. COUNT-GUIDED WSL BASED ON DENSITY MAP:

In this section, we investigate the count-guided weakly supervised localization based on density map. We first describe the basic counting structure that takes the density map as latent variables and then discuss the impact of pooling and convolution on this end-to-end WSL network. Finally, we propose the Gini impurity penalty to control the representation of a density map and show its relation with the β-VAE.

3.1. BASIC COUNTING STRUCTURE:

The feature extractor is a FCNN that outputs a map f Θ (I) of the same size as the input image I, where Θ are the parameters of the network. The map is activated by the rectified linear unit (ReLU) and then goes through a hyperbolic tangent function to make sure all its elements are either zero or less than one; i.e. D = tanh(f ). Finally, the estimated count ĉ is the integral of the density map:

EQUATION

At the training stage, we only provide the final count c of objects as ground truth. The goal is to train the latent density map for localization:

Θ = arg min Θ E data [(c -ĉ) 2 ] (2)

EQUATION

In terms of training, equation 2 and 3 are identical. Following the notation of VAE (Burgess et al., 2018) , we expose the hidden density map in 3 for further analysis.

3.2. SPATIAL INFORMATION LOSS:

Conventionally, a CNN applies pooling layers to reduce computations and extract abstract features. However, as a down-sampling operation, pooling does harm to the spatial information going through a deep network. Here we use a simple bubble counting task to study the impact of pooling on this end-to-end WSL network.

3.2.1. SETUP OF BUBBLE COUNTING:

The simulated data is a set of 64×64 binary images with a randomly selected number of 1 to 15 bubbles in the frame, as shown in Figure 1a . The bubble centers are located within a square as shown in Figure 1b and Figure 2d , and their radii are 3 to 5 pixels. Slight overlaps are allowed.

The goal is to localize the bubble centers by providing only the counts for training. Table 1 shows the structure of the FCNN for this test. It uses max-pooling following the second, third, and fourth convolutional layers to down-sample features. After reaching the bottleneck, it uses transposed convolutions (deconvolution) for staged up-sampling to finally match the input size at the output.

3.2.2. SUPERVISED LOCALIZATION:

Firstly, we make sure this FCNNs capacity is sufficient for this localizing task. Both the ground-truth density maps D and the final counts c are provided for training:

Θ = arg min Θ E data [(c -ĉ) 2 + (D -D) 2 ] (4)

We generate a mini-batch of 10 samples for every training step and a set of 500 examples for testing. After 3 × 10 5 evaluations, the average pixel-by-pixel error of the density map converges to a sufficiently small value of 0.001. An example of the estimated centers is shown in Figure 1c . This demonstrates that the proposed network can count as well as localize bubbles.

3.2.3. WEAKLY SUPERVISED LOCALIZATION:

Now we use the same training strategy to train the network again but provide it only with the final counts (equation 2). After 3 × 10 5 epochs, the counting error converges to 0.02. But, as shown in Figure 2c , the localization is not as good as in the case of supervised localization and shows two artifacts: Firstly, the centers are relocated to match a low resolution grid. Figure 2e compiles the statistics of 500 test samples to draw this grid clearly. This spatial information loss is caused by the pooling layers, and it cannot be recovered by up-samplings. In contrast, this loss of detail does not happen in the case of supervised localization (Figure 1 ), because the presence of the true centers in the labels forces the network to retain position information even through down-sampling processes. Due to this drawback, we will not use any pooling layers in the following WSL models.

Secondly, the estimated centers as a whole drift to the right bottom corner of the density map. We believe this is because CNNs rely on edge detection, and our training data is homogeneous. The output of the first convolutional layer (sum of all channels) in Figure 3 shows that the trained kernels highlight the right bottom edges more than other sides. Going through all the layers, this bias will finally show in the density map. Zhou et al. (2016) discussed that the GAP, unlike the max-pooling, encourages the network to identify the full extent of the object instead of certain edges. However, in this count-guided case, the bias happens before any pooling layers (Figure 3 ), and the averaging (sum) cannot recover the correct positions of the centers (Figure 2 ). Thus, we resort to a direct solution to tackle this problem.

3.3. ROTATION ROBUSTNESS:

In order to concentrate the learned features at the center of an object, we encourage the network to be robust to rotations. To accomplish this, the input image is rotated by 90 • , 180 • , and 270 • , and the differences between rotations are to be reduced. Denote the rotated image as I degree , its density map as D degree , and its count as c degree . The loss function comes to

E data [ k (c -ĉk ) 2 + i,j ( Di -Dj ) 2 ] i, j, k = 0 • , 90 • , 180 • , 270 • (5)

We use the FCNN structure in Table 1 for feature extraction, but the down and up sampling operations are discarded for accurate localization. The results are shown in Figure 4 . Compared with Figure 2 , the density map is no longer limited by the lower resolution grid because of the absence of pooling layers. However, one bubble is represented by four dots because we turned the training images four times to make the network immune to rotations. To further centralize the dots, we propose a Gini impurity penalty in the following section.

3.4. GINI IMPURITY PENALTY:

Intuitively, using more dots in a density map increases the maps impurity. One of the widely used metrics of impurity is the Gini impurity (Fayyad & Irani, 1992) . It is defined as the probability of one state changing to its opposite. For a two-dimensional density map, it is 

EQUATION

p(D) = D D + (7)

We add a small number to the denominator in case the integral is zero in practice.

Because all the elements of the density map are less than one, if two density maps have the same integral, the one with more non-zero elements will have higher Gini impurity. For example, in Figure 4 , the sums of the true and estimated density map are very close (4.00 and 4.08), but their Gini impurities are 0.75 and 0.93, respectively. We can reduce the amount of non-zero pixels by decreasing the Gini impurity of a density map. The loss function becomes

E data [ k (c -ĉk ) 2 + i,j ( Di -Dj ) 2 ] + β G( Dk ) i, j, k = 0 • , 90 • , 180 • , 270 • (8)

where β is a positive real hyper-parameter that controls this regularization term.

Like before, the FCNN in Table 1 without pooling layers are used for feature extraction. β = 10.0 has been applied to regularize the density map. The outcomes are shown in Figure 5 . Compared with previous results (Figure 2 and Figure 4 ), the detected centers are no longer limited by a shifted low-resolution grid, and the amount of non-zero dots matches the exact count to a great extent.

Equation 8 indicates that the Gini impurity penalty is related to the β-VAE. Here we study the case without the rotations. Given the count of objects, the Gini impurity of the true density map is a fixed number:

EQUATION

After training, we assume that D = c. Because all elements of the density map are between zero and one, there is

EQUATION

) Thus, minimizing the Gini impurity can be achieved by reducing the information gain

EQUATION

which is equivalent to minimizing the Kullback-Leibler (KL) divergence between the prior distributions p(D) and the posterior distribution q Θ ( D|I). Then the training maximizes the log-likelihood of counting while minimizing the KL divergence:

EQUATION

Equation 12 has the same form as the β-VAE (Higgins et al., 2017) with four major differences. Firstly, this formula computes the count of objects instead of regenerating the input image. Secondly, no random variables are involved during training (equation 8), so we do not need to use the reparameterization trick (Alemi et al., 2016) . Thirdly, the latent variable of a VAE is usually a 1-D vector in contrast to a 2-D density map in this problem. Finally, the prior in a VAE is often set to be an isotropic Gaussian distribution (Burgess et al., 2018) , whereas we do not know the exact distribution of the true density map; we only know its Gini impurity.

4. MALL DATA WSL:

Similar to the variational term of the β-VAE, the Gini impurity penalty degrades the reconstruction (counting) performance, but it also encourages the density map to extract interpretable features.

Here we take the Mall dataset as an example. The Mall set is a 2000-sample video clip collected through a surveillance camera above the hall of a shopping mall (Figure 6a ). Although the heads of people are annotated (Figure 6b ) by hand for counting and localization Loy et al. (2013) ; Chen et al. (2013) ; Change Loy et al. (2013) ; Chen et al. (2012) , we aim to find the them from only the count (WSL).

Because human figures and the environment of the shopping mall are more complicated, we build up another FCNN for this task. It is a concatenation of 7 Residual Blocks (Res-block) He et al. (2016) as shown in Figure 7 . The first convolutional layer of the Res-block increases or decreases the depth of its input tensor. Whereas the second layer maintains its inputs shape, so that the outputs of these two layers can be added together. To accelerate training, we apply batch-normalization Ioffe & Szegedy (2015) to the output of each Res-block. As before, no pooling is used here. Details of this FCNN are shown in Table 2 . We use the first 1900 samples of the Mall set for training and the last 100 for validation. The original resolution of the Mall data is 480×640, but at the training stage we randomly crop a 300×300 patch of it to reduce computations. This is viable because a trained FCNN can be generalized to inputs of arbitrary sizes. We trained the network according to equation 2, where the ground-truth count within a 300×300 patch is computed by adding up the head annotations. The counting error converges to around 0.85 after 3×10 5 epochs. An example of the hidden density map is shown in Figure 6c . It shows that the CNN pays more attention to the human figures than the irrelevant backgrounds. But the network does make some mistakes by counting parts of the booth, the tree, reflections, and the models as human beings, and for most of the bodies the detected features are along the their left sides. It can also be noticed that for an individual person, the highlighted points concentrate on the upper body. This could be because we used the appearances of heads for counting.

However, it is not clear by what discriminative features the network detects a human figure. Therefore, we decrease the Gini impurity (β=10.0) to reduce the amount of dots in the density map. After 3×10 5 epochs, the counting error converges to around 1.4, which is higher than before because of the regularization. The hidden density map is shown in Figure 6d . Compared with the previous result, the detected features concentrate over the heads of people. Herein, we can conclude that the regularized network counts human figures by detecting their heads. However, it is also noticed that the network struggles at detecting those who are remote or sitting on the bench. We believe this is because crowds standing far away have lower resolutions, and those on the bench sat still for a long time. Besides, they both have more overlaps than walking individuals.

Compared to class-guided weakly supervised localization, our method shows how specific features of a certain type of object are used for detecting. For human detection, the features could be the heads, hands, feet, etc., but some of them may not be used for the final counting. With the Gini impurity penalty, we are able to manipulate these features for better interpretation.

5. CONCLUSION:

Because a deep CNN can maintain spatial information throughout very deep layers, a CNN trained with image-level labels has the potential of localizing detected targets. This paper uses a densitymap-based counting machine to study this weakly supervised localization problem. First of all, the spatial information loss caused by pooling operations has been shown. This does not mean pooling cannot be used in any WSL networks, but for a plain end-to-end structure the lost information can hardly be recovered. Secondly, a simple technique has been applied to enhance the models robustness to rotation. By doing this, the detected features are concentrated within the object instead of along certain edges. Thirdly, a Gini impurity penalty has been introduced to control the amount of discriminative features in the hidden density map. This regularization acts in a similar way to the variational term of a β-VAE. Unlike a traditional β-VAE, which controls the capacity of a 1-D hidden space, this Gini impurity penalty restricts the capacity of a 2-D layer. Its potential in unsupervised learnings could be further explored in the future.

:



Figures and Tables:

Figure 1: Bubble image and supervised localization

Figure 2: Weakly supervised Localization with max-pooling

Figure 4: Weakly supervised Localization robust to rotation

Figure 5: Weakly supervised localization with rotation robustness and Gini impurity penalty

Figure 6: Mall data weakly supervised localization (images are publicly accessible for research purposes)

FCNN for bubble counting

Table content: <table><tr><td>operations</td><td>kernel</td><td>size</td><td>channels</td><td/></tr><tr><td>conv</td><td>3×3</td><td>64×64</td><td>32</td><td/></tr><tr><td>conv</td><td>3×3</td><td>64×64</td><td>64</td><td/></tr><tr><td>max-pooling</td><td>3×3</td><td>32×32</td><td>64</td><td/></tr><tr><td>conv</td><td>3×3</td><td>32×32</td><td>128</td><td/></tr><tr><td>max-pooling</td><td>3×3</td><td>16×16</td><td>128</td><td/></tr><tr><td>conv</td><td>3×3</td><td>16×16</td><td>512</td><td/></tr><tr><td>max-pooling</td><td>3×3</td><td>8×8</td><td>512</td><td/></tr><tr><td>deconv</td><td>3×3</td><td>16×16</td><td>128</td><td/></tr><tr><td>deconv</td><td>3×3</td><td>32×32</td><td>64</td><td/></tr><tr><td>deconv</td><td>3×3</td><td>64×64</td><td>32</td><td>Figure 3: First layer output</td></tr><tr><td>conv (f (I))</td><td>3×3</td><td>64×64</td><td>1</td><td>(yellow for higher values)</td></tr></table>

FCNN for crowd counting

Table content: <table><tr><td>operation</td><td>kernel</td><td>size</td><td>channels</td><td/></tr><tr><td>Res-block</td><td>3×3</td><td>300×300</td><td>32</td><td/></tr><tr><td>Res-block</td><td>3×3</td><td>300×300</td><td>64</td><td/></tr><tr><td>Res-block</td><td>3×3</td><td>300×300</td><td>128</td><td/></tr><tr><td>Res-block</td><td>3×3</td><td>300×300</td><td>128</td><td/></tr><tr><td>Res-block</td><td>3×3</td><td>300×300</td><td>128</td><td/></tr><tr><td>Res-block</td><td>3×3</td><td>300×300</td><td>64</td><td/></tr><tr><td>Res-block</td><td>3×3</td><td>300×300</td><td>32</td><td/></tr><tr><td>conv (f (I))</td><td>3×3</td><td>16×16</td><td>1</td><td>Figure 7: Residual Block</td></tr></table>

Peer review (iclr2020):
This article proposes a method for object counting which can be trained with weak supervision. Object counting methods are often trained with point annotations, i.e., one click-point per object. In this article, a weaker way of annotation is used: count-based annotation, i.e., the number of objects of each class present in the image are given as annotation but no precise location of the objects. This article is an extension of density-based object counting methods for weakly supervision. This article analyzes the effect of pooling layers on WSL. The findings indicate that the pooling layer constrained the objects* locations to a predefined grid and accordingly they remove it. When removing this effect is alleviated. Moreover, the authors realized that a sift on the predictions also happen. Accordingly, they learn rotated versions of the model for centering the predictions. As a countereffect, several points per object are detected, and then a gini impurity regularization is proposed to reduce the number of detected objects. Then, the authors connect this formulation with VAE. The experiments are conducted on a toy example, i.e., circle finding, and on the well established Mall dataset. Main concerns: The experimental section should make experiments on standard datasets (i.e., USCD, Trancos, Mall, PkLot, Shangai, Penguins) using standard evaluation protocols (MAE, GAME). The current evaluation only shows the qualitative results on one frame of the Mall dataset. The results of the proposed method should be compared with baselines. For instance, Glance method, which is only trained with count-level annotation, and with *C-WSL: Count-guided Weakly Supervised Localization*. It should be also compared with WSL methods based on image-level labels such as *Object Counting and Instance Segmentation with Image-level Supervision* or *Where are the Masks: Instance Segmentation with Image-level Supervision* and then with methods that use point supervision as an upper bound such as density method or Where are the blobs. Moreover, some ablation studies about the effect of each component of the model would be needed. Regarding the model I have some concerns: - Removing the pooling layers does not seem a very good idea. This will remove part of the ability to detect objects at multiple scales and moreover will increase its computational power. Maybe for easy cases as circles and heads of mall people where the resolution changes are small could work. But not for more challenging datasets. - The rotation of the model could be ok for circles or heads. But making the model to learn rotation-invariant features could be challenging in more difficult datasets. - Eq.5 and 8 which are used in the model seem to use point supervision and not just the count based. Which makes the method not a weakly supervised one. Are these equations the ones used or others?