Count-guided Weakly Supervised Localization Based on Density Map
Authors: Ming Ma
Stephan Chalup
Fayeem Aziz
Yang Liu
Defu Cheng
Zhijian Zhou
Venue:  (2020)
Keywords: Semi-supervised Learning
Weakly Supervised Localization
Variational Autoencoder
Density Map
Counting

Abstract: Weakly supervised localization (WSL) aims at training a model to find the positions of objects by providing it with only abstract labels. For most of the existing WSL methods, the labels are the class of the main object in an image. In this paper, we generalize WSL to counting machines that apply convolutional neural networks (CNN) and density maps for counting. We show that given only ground-truth count numbers, the density map as a hidden layer can be trained for localizing objects and detecting features. Convolution and pooling are the two major building blocks of CNNs. This paper discusses their impacts on an end-to-end WSL network. The learned features in a density map present in the form of dots. In order to make these features interpretable for human beings, this paper proposes a Gini impurity penalty to regularize the density map. Furthermore, it will be shown that this regularization is similar to the variational term of the $\beta$-variational autoencoder. The details of this algorithm are demonstrated through a simple bubble counting task. Finally, the proposed methods are applied to the widely used crowd counting dataset the Mall to learn discriminative features of human figures.

Under review as a conference paper at ICLR 2020
COUNT-GUIDED WEAKLY SUPERVISED LOCALIZA- TION BASED ON DENSITY MAP
Anonymous authors Paper under double-blind review
ABSTRACT
Weakly supervised localization (WSL) aims training model find the posi- tions of objects by providing it with only abstract labels. For most of the existing WSL methods, the labels are the class of the main object in an image. In this paper, we generalize WSL to counting machines that apply convolutional neu- ral networks (CNN) and density maps for counting. We show that given only ground-truth count numbers, the density map hidden layer can be trained for localizing objects and detecting features. Convolution and pooling are the two major building blocks of CNNs. This paper discusses their impacts on an end-to- end WSL network. The learned features in a density map present in the form dots. In order to make these features interpretable for human beings, this paper proposes a Gini impurity penalty to regularize the density map. Furthermore, it will be shown that this regularization is similar to the variational term of the 3- variational autoencoder. The details of this algorithm are demonstrated through simple bubble counting task. Finally, the proposed methods are applied the widely used crowd counting dataset the Mall to learn discriminative features of human figures.
INTRODUCTION
Deep convolutional neural networks (CNN) have significantly pushed the frontier of image process- ing. Trained with enormous amount of data, CNNs have surpassed human performance in many object detecting tasks. However, humans are still ahead in many aspects. One of them is the seg- mentation based on abstract knowledge: once a human can tell what in an image, he/she can easily crop the objects out. Researches have been exploring similar potentials of CNNs. Related techniques are referred to as Weakly Supervised Localization (WSL), in which models are trained with only image-level labels but expected give the positions of objects at once. Most of existing WSL methods are guided by class labels and realized in two different ways.
The first is the Multiple Instance Mining (MIM). MIM is based multiple instance learning (MIL), which trains a model to identify multiple instances from incomplete labels (Babenko) 2008 Siva al. 2012). For object detection, the regions around the target have the highest influence label decision. By evaluating the contributions of different regions, the location of the target can be determined. A straightforward approach to find the relevant regions is to compute the derivative of the decision with respect to the input image (Simonyan et al. 2013). But the found regions can also be the ones that introduce the most uncertainty the result. more effective approach to use an attention mechanism, in which one detector proposes potential regions according to classifier's evaluation of contribution (Gokberk Cinbis et al. 2014: Song et al., 2014a Bilen et al. 2014 Ren et al. 2016 Cinbis et al. 2017).
the
The second way to WSL is to train the locations as latent variables. Zhou et al. reported that a deep CNN can maintain spatial information through layers (Zhou al. 2016). Based on this observation they proposed the class activation map (CAM) which averages every channel a CNN's final layer and feeds the averages to soft-max classifier. This structure is analogous to the density map used for object counting. The density map method annotates each object by one pixel with a value of one and computes the count by summing up the map (Lempitsky & Zisserman 2010).
Inspired by the similarities between the CAM and the density map. we generalize WSL to counting models based on density map. Our method trains the density map latent variables of an end-

---

Under review as a conference paper at ICLR 2020
(a) imput image
(b) ground-trutt centers
(c) estimated centers
Figure 1: Bubble image and supervised localization
to-end CNN by giving only the final count of objects. It is observed that the trained density map can roughly capture the positions of objects. but the accuracy is degraded by the convolution and pooling operations. This paper will discuss the impacts of these two operations on an end-to-end
WSL network and propose techniques improve accuracy. The density map counts objects by detecting discriminative features. To improve interpretability of the found features, we introduce Gini impurity penalty to regularize the density map. Furthermore, the similarities and differences
between this regularization and the variational term of the 3-variational autoencoder (VAE) (Higgins) ctt al., 2017) will be discussed. Throughout this paper, we will use a simple simulated bubble dataset to demonstrate basic ideas and widely used crowd-counting dataset the Mall Loy al. (2013);
Chen et al. (2013): Change Loy et al. (2013): Chen et al. (2012) to show the methods' effectiveness practice.
2 RELATED WORK
2.1 MULTIPLE INSTANCE MINING
Coarsely labelled images are labelled only by the class of the main object. However, due to the usage of softmax function as output and the cross entropy for training, a classifier has the potential to identify multiple instances (Babenko 2008). A MIM WSL method trains detector to propose either a bounding box (Bilen & Vedaldi 2016 Tang et al, 2017) or a segmentation (Kolesnikov & Lampert 2016 Wei 2017) that highlights the object in the image. A few candidates for bounding boxes or segmentations are initialized at the beginning, and the detector trained by trial and error. This always results in non-convex optimization problem that easily converges to local optimum. Solutions for this problem focus on both the initialization (Deselaers et al.) 2012 Siva al. 2013 Shi al. [2013] Song et 2014b) and the training strategy (Crandall & Huttenlocher 2006 Cinbis et al. 2017: Li et al. 2016). This attention mechanism can also be realized in an adversarial style (Wei
2017 Shen al. [0188] Bartz al. 2018).
Among all MIM methods, the min-entropy latent model (MELM) and the count-guided weakly supervised localization (C-WSL) are mostly related to this work. The MELM (Wan al. 2018) uses the min-entropy metric to measure and reduce the randomness of localization proposals. Likewise, our method leverages the Gini impurity to decrease the complexity of the latent density map. The C-WSL (Gao et al.) 2018) takes advantage of per-class object count labels to reduce label cost and improve performance. Even though is also guided by count, it applies a MIM paradigm in contrast to our end-to-end density- ceanstt--map-bar maseb approach.
2.2 DENSITY MAP COUNTING
Density map a counting technique based on detection (Lempitsky & Zisserman 2010). Combined with CNNs, it has achieved state-of-the-art performance in tasks like cell (Xie et al. 2018). crowd
Zhang et al., 2016), and wild animal (Arteta et al., 2016) counting. Commonly, counting CNNs have the structure of Fully Convolutional Neural Network (FCNN) (Xie et al.) 2018). because images for counting usually contain similar objects spreading all over the frame, and a FCNN can process images of arbitrary sizes.
2

---

Under review as a conference paper at ICLR 2020
(a) imput image
(b) true density map
(c) estimated density map
(d) statisties of true centers
(e) statistics of estimated centers
Figure 2: Weakly supervised Localization with max-pooling
2.3 CLASS ACTIVATION MAPPING
The CAM localizes the positions of discriminative features by applying the global average pooling (GAP) to the final layer a CNN. Because it trained in a MIL way, the CAM can identify multiple features in image (Zhou et al. 2016). A trained CAM can also be used for proposing candidates in a MIM method (Chen et al. 2018).. To some extent, CAM is analogous to density map counting, as averaging the channels is equivalent integrating the density map. On the other hand, CAM turns a CNN with fully connected layers to a FCNN.
2.4
VARIATIONAL AUTOENCODER
VAE (Kingma & Welling 2013) or more generally 3-VAE (Higgins et al. 2017) is an unsupervised data exploring method. The hidden features of the dataset are learned by training a network to compress and reconstruct its input. Experiments and theories have shown that the variational term of the VAE forces an auto-encoder to learn more disentangled latent features (Higgins et al. 2017 Alemi et al. 2016 Pu et al. 2016 Kim & Mnih 2018 Burgess et al. 2018). In the following work, we will use a similar technique to make the density map yield interpretable results.
3 COUNT-GUIDED WSL BASED ON DENSITY MAP
In this section, we investigate the count-guided weakly supervised localization based density map. We first describe the basic counting structure that takes the density map as latent variables and then discuss the impact of pooling and convolution this end-to-end WSL network. Finally, we propose the Gini impurity penalty to control the representation of a density map and show its relation with the 3-VAE.
3.1 BASIC COUNTING STRUCTURE
The feature extractor is FCNN that outputs a map fI) of the same size as the input image I, where & are the parameters of the network. The map is activated by the rectified linear unit (ReLU) and then goes through hyperbolic tangent function to make sure all its elements are either zero or less than one; i.e. D = tfn Finally, the estimated count è is the integral of the density map:
è
De(!)
At the training stage, we only provide the final count e of objects as ground truth.
θ arg min Estata [[c 2)
(2)

---

Under review as a conference paper at ICLR 2020
Table I: FCNN for bubble counting
operations
kernel
size
channels
conv
3x3
64x64
32
conv
3x3
64
64
max-pooling
3x3
32x32
3x3
32x32
128
max-pooling
3x3
16x
128
3x3
16x16
512
max-pooling
3x3
8x8
512
deconv
3x3
16x16
128
deconv
3x3
32x32
64
deconv
3x3
64x64
32
Figure 3: First layer output
conv (f())
3x3
64x64
(yellow for higher values)
The goal is to train the latent density map for localization:
= arg max Ege Frgg
(3)
In terms of training, equation and 3 are identical. Following the notation of VAE (Burgess et al.
2018
we expose the hidden density map for further analysis.
3.2 SPATIAL INFORMATION LOSS
Conventionally, a CNN applies pooling layers to reduce computations and extract abstract features. However, dlwn-ssmmping operation, pooling does harm to the spatial information going through a deep network. Here we use a simple bubble counting task to study the impact of pooling on this end-to-end WSL network.
3.2. 1 SETUP BUBBLE COUNTING
The simulated data set of 64x64 binary images with randomly selected number of 15 bubbles in the frame, as shown in Figure Ta The bubble centers are located within square as shown in Figure Tb and Figure 2d and their radii are The goal is to localize the bubble centers by providing only the counts for training. Table shows the structure of the FCNN for this test. It uses max-pooling following the second, third, and fourth convolutional layers down-sample features. After reaching the bottleneck, it transposed convolutions (deconvolution) for staged up-sampling to finally match the input size at the output.
pixels. Slight overlaps are allowed.
3.2.2 SUPERVISED LOCALIZATION
Firstly, we make sure this FCNNs capacity is sufficient for this localizing task. Both the ground-truth density maps D and the final counts e are provided for training:
θ arg min Edtra - 2)2 Σ -
We generate a mini-batch of 10 samples for every training step and a set of 500 examples for test- ing. After x 10 evaluations, the average pixel-by-pixel error of the density map converges sufficiently small value of 0.001. An example of the estimated centers is shown in Figure Ic This demonstrates that the proposed network can count as well as localize bubbles.
3.2.3 WEAKLY SUPERVISED LOCALIZATION
Now we use the same training strategy to train the network again but provide it only with the final counts (equation 2) After 3 x 10a epochs, the counting error converges to 0.02. But, shown in Figure 2c the localization is not as good as in the case of supervised localization and shows two artifacts:
4

---

Under review as a conference paper at ICLR 2020
(a) imput image
(b) true density map
(c) estimated density map
(d) statisties of true centers (e) statistics of estimated centers
Figure 4: Weakly supervised Localization robust rotation
Firstly, the centers are relocated to match a low resolution grid. Figure 2c compiles the statistics of 500 test samples to draw this grid clearly. This spatial information loss is caused by the pooling layers, and it cannot be recovered by up-samplings. In contrast, this loss of detail does not happen in the case of supervised localization (Figure D. because the presence of the true centers in the labels forces the network to retain position information even through down-sampling processes. Due to this drawback, we will not use any pooling layers in the following WSL models.
Secondly, the estimated centers as a whole drift to the right bottom corner of the density map. We believe this is because CNNs rely edge detection, and our training data is homogeneous. The output of the first convolutional layer (sum of all channels) in Figure Jiswws that the trained kernels highlight the right bottom edges finally show in the density map. Zhou et al. (2016) discussed that the GAP, unlike the max max-pooling, encourages the network identify the full extent of the object instead of certain edges. However, in this count-guided case, the bias happens before any pooling layers (Figure 3 and the averaging (sum) cannot recover the correct positions of the centers (Figure Thus, we resort to direct solution to tackle this problem.
than other sides. Going through all the layers, this bias will
3.3 ROTATION ROBUSTNESS
In order to concentrate the learned features at the center of an object, we encourage the network to be robust to rotations. To accomplish this, the input image is rotated by 90", 180%, and 270", and the differences between rotations are to reduced. Denote the rotated image as Idegrees its density map as Deegree and its count as Cdegree: The loss function comes
Edatal E(e - C2) ED. - D,)31 i,j, k 0%, 90" 180", 270
(5)
We use the FCNN structure in Table [ for feature extraction, but the down and up sampling oper- ations are discarded for accurate localization. The results are shown Figure 4 Compared with Figure 2] the density map is no longer limited by the lower resolution grid because of the absence of pooling layers. However, one bubble is represented by four dots because we turned the training im- ages four times to make the network immune to rotations. To further centralize the dots, we propose Gini impurity penalty the following section.
3.4 GINI IMPURITY PENALTY
Intuitively, using more dots in density map increases the maps impurity. One of the widely used metrics of impurity the Gini impurity (Fayyad & Irani] 1992). defined as the probability of one state changing its opposite. For two-dimensional density map, is
G(D) P(D) p(D) = >2² (D)
(6)
5

---

Under review as a conference paper at ICLR 2020
input image
(h) density map
(c) estimated density map
(d) statistics of true centers
(e) statistics of estimated centers
Figure Weakly supervised localization with rotation robustness and Gini impurity penalty
where p(D) a 2-D distribution obtained by normalizing the density map:
D
p(D)
E
We add small number to the denominator in case integral is zero in practice.
Because all the elements of the density map are less than one, if two density maps have the same integral, the one with more non-zero elements will have higher Gini impurity. For example, in Figure 4 the sums of the true and estimated density map very close (4.00 and 4.08), but their Gini impurities are 0.75 and 0.93, respectively. We can reduce the amount non-zero pixels by decreasing the Gini impurity of density map. The loss function becomes
Edota E (c)=2)) (D.. D,) EG(DA i,j, = 09, 90" 180" 270
(8)
where 3 is a positive real hyper-parameter that controls this regularization term.
Like before, the FCNN in Table T without pooling layers used for feature extraction. / 10.0 has been applied to regularize the density map. The outcomes are shown in Figure (5) Compared with previous results (Figure 22 and Figure the detected centers are no longer limited by shifted low-resolution grid, and the amount of non-zero dots matches the exact count to a great extent.
Equation indicates that the Gini impurity penalty is related to the 3-VAE. Here study the without the rotations. Given the count of objects, the Gini impurity of the true density map is fixed number:
-
1 2
G(D)
After training, we assume that Σ ò c. Because all elements of the density map are between zero and one, there is
G(Ò) 2 G(D)
(10)
Thus, minimizing the Gini impurity can be achieved by reducing the information gain
AH D) G(D) - G(D)
(11)
which is equivalent to minimizing the Kullback-Leeiler (KI.) divergence between the prior distribu- tions p(D) and the posterior distribution qe (D)I). Then the training maximizes the log- likeliilldood
of counting while minimizing the KL divergence:
arg max E (DD) (lggllllopp))))))) 3Dx (D01
(12)
Equation 12 has the same form the 3-VAE (Higgins al.) 2017) with four major differences. Firstly, this formula computes the count of objects instead of regenerating the input image. Sec- ondly, no random variables involved during training (equation 8), do not need to the

---

Under review as a conference paper at ICLR 2020
200
200
100
200
200
2010
+000
200
500
(a) a frame of surveillance video
(b) head annotations
0.8
0.4
0.3
0.4
0.2
0.1
02
100
200
400
100
200
300
400
500
900
(c) WSL without regularization
(d) WSL with Gini impurity penalty
Figure 6: Mall data weakly supervised localization (images publicly accessible for research purposes)
reparameterization trick (Alemi et al.) 2016). Thirdly, the latent variable of a VAE is usually a 1-D vector in contrast 2-D density map in this problem. Finally, the prior in VAE is often to be an isotropic Gaussian distribution (Burgess et al.) 2018). whereas we do not know the exact distribution of the true density map; we only know its Gini impurity.
MALL DATA WSL
Similar to the variational term the 3-VAE, the Gini impurity penalty degrades the reconstruction (counting) performance, but it also encourages the density map extract interpretable features. Here we take the Mall dataset as an example. The Mall set 2000-sample video clip collected through a surveillance camera above the hall of a shopping mall (Figure|6a) 6a). Although the heads of people are annotated (Figure 6b) by hand for counting and localization Loy et al. (2013); Chen et al. 2013 Change Loy et (2013) Chen al. (2012), aim find the them from only the count WSL).
Because human figures and the environment of the shopping mall are more complicated, we build up another FCNN for this task. It is a concatenation of 7 Residual Blocks (Res-block) He et al. (2016) as shown in Figure 7 The first convolutional layer of the Res-block increases or decreases the depth of its input tensor. Whereas the second layer maintains its inputs shape, so that the outputs of these two layers can be added together. To accelerate training, we apply nallh-normalization loffe & Szegedy (2015) to the output of each Res-block. As before,
pooling is used here. Details of this FCNN are shown in Table 2 We use the first 1900 samples of the Mall set for training and the last 100 for validation. The original resolution of the Mall data is 480x 640, but at the training stage we randomly crop a 300 300 patch of it to reduce computations. This is viable because a trained FCNN can be generalized to inputs of arbitrary sizes.

---

Under review as a conference paper at ICLR 2020
Table 2: FCNN for crowd counting
Conv layer (ReLU)
operation
kernel
size
channels
Res-block
3x3
300x300
32
Conv layer (Identity)
Res-block
3x3
300x300
64
Res-block
3x3
300x300
128
ReLU
Res-block
3x3
300x300
128
Res-block
3x3
300x300
128
Batch-normalization
Res-block
3x3
300x300
64
Res-block
3x3
300x300
32
conv (J(I)
3x3
16x 16
1
Figure 7: Residual Block
We trained the network according to equation 2 where the ground count within 300x300 patch is computed by adding up the head annotations. The counting error converges to around 0.85 after 3 x 10 epochs. An example of the hidden density map is shown in Figure 6c shows that the CNN pays more attention to the human figures than the irrelevant backgrounds. But the network does make some mistakes by counting parts of the booth, the tree, reflections, and the models as human beings, and for most of the bodies the detected features are along the their left sides. It can also be noticed that for an individual person, the highlighted points concentrate on the upper body. This could be because we used the appearances of heads for counting.
However, it is not clear by what discriminative features the network detects a human figure. There- fore, we decrease the Gini impurity (8=10.0) to reduce the amount of dots the density map. After 3x10 10 epochs, the counting error converges to around 1 1.4, which is higher than before because of the regularization. The hidden density map is shown in Figure 6d Compared with the previous result, the detected features concentrate over the heads of people. Herein, we can conclude that the regularized network counts human figures by detecting their heads. However, is also noticed that the network struggles detecting those who remote sitting the bench. We believe this is because crowds standing far away have lower resolutions, and those on the bench sat still for a long time. Besides, they both have more overlaps than walking individuals.
Compared to class-guided weakly supervised localization, our method shows how specific features of certain type of object used for detecting. For human detection, the features could be the heads, hands, feet, etc., but some of them may not be used for the final counting. With the Gini impurity penalty, we are able to manipulate these features for better interpretation.
CONCLUSION
Because deep CNN can maintain spatial information throughout very deep layers, CNN trained with image-level labels has the potential localizing detected targets. This paper uses a density- map-based counting machine study this weakly supervised localization problem. First of all, the spatial information loss caused by pooling operations has been shown. This does not mean pooling cannot be used in any WSL networks, but for plain end-to-end structure the lost information can hardly be recovered. Secondly, a simple technique has been applied to enhance the models robustness to rotation. By doing this, the detected features concentrated within the object instead of along certain edges. Thirdly, a Gini impurity penalty has been introduced to control the amount of discriminative features in the hidden density map. This regularization acts in similar way to the variational term of B-VAE Unlike a traditional VAE,, which controls the capacity of a 1- hidden space, this Gini impurity penalty restricts the capacity of 2-D layer. Its potential in unsupervised learnings could be further explored in the future.
REFERENCES
Alexander A Alemi, Ian Fischer, Joshua Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv: 1612.00410, 2016.
8

---

Under review as a conference paper at ICLR 2020
Carlos Arteta, Victor Lempitsky, and Andrew Zisserman. Counting in the wild. In Proceedings of the European conference on computer vision, pp. 483-498, 498, 2016.
Boris Babenko. Multiple instance learning: algorithms and applications. View Article PubMed/NCBI Google Scholar, pp. 1-19, 2008.
Christian Bartz, Haojin Yang. Joseph Bethge, and Christoph Meinel. Loans: Weakly supervised object detection with localizer assessor networks. arXiv preprint arXiv:1811.0577., 2018.
Hakan Bilen and Andrea Vedaldi. Weakly supervised deep detection networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2846-2854, 2016.
Hakan Bilen, Marco Pedersoli, and Tinne Tuytelaars. Weakly supervised object detection with posterior regularization. In British machine vision conference, volume 3, 2014.
Christopher P Burgess, Irina Higgins, Arka Pal, Loie Matthey, Nick Watters, Guillaume Des- jardins, and Alexander Lerchner. Understanding disentangling in 3-VAE. arXiv preprint arXiv: 03599, 2018.
Chen Change Loy, Shaogang Gong, and Tao Xiang, From semi-supervised to transfer counting of crowds. In Pruceedings of the IEEE international conference on computer vision, pp. 2256-2263, 2013.
Ke Chen, Chen Change Loy, Shaogang Gong, and Tony Xiang. Feature mining for localised crowd counting. In British machine vision conference, volume pp. 3, 2012.
Ke Chen, Shaogang Gong, Tao Xiang, and Chen Change Loy. Cumulative attribute space for age and crowd density estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2467-2474, 2013.
Yilun Chen, Zhicheng Wang, Yuxiang Peng. Zhiqiang Zhang, Gang Yu, and Jian Sun. Cascaded pyramid network for multi-person pose estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition, June 2018.
Ramazan Gokberk Cinbis, Jakob Verbeek, and Cordelia Schmid. Weakly supervised object local- ization with multi-fold multiple instance learning. IEEE transactions pattern analysis and machine intelligence, 39(1):189-200 2017.
David Crandall and Daniel Huttenlocher. Weakly supervised learning of part-based spatial models for visual object recognition. In Proceedings of the European conference on computer vision, pp. 16 16,29, 2006.
Thomas Deselaers, Bogdan Alexe, and Vittorio Ferrari. Weakly supervised localization and learning with generic knowledge. International journal of computer vision. 100(3):275 293, Dec 2012. ISSN 1573-1405. doi: 10. 10.10007/111233-02-00333 8
URL httpp::///ecc.c.... /ct.//ci.ii.. http:///....0/111101001000
Usama M Fayyad and Keki B Irani. The attribute selection problem in decision tree generation. In Proceedings of the tenth national conference on Artificial intelligence, pp. 104-110, 1992.
Mingfei Gao, Ang Li, Ruichi Yu, Vlad Morariu, and Larry S. Davis. C-wsl: Count-guided weakly supervised localization. In Proceedings of the European conference on computer vision, Septem- ber 2018.
Ramazan Gokberk Cinbis, Jakob Verbeek, and Cordelia Schmid. Multi-foll mil training for weakly supervised object localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2409-2416. 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.

---

Under review as a conference paper at ICLR 2020
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerehner. 3-VAE: Learning basic visual concepts with a con- strained variational framework. In International conference learning representations, vol- ume 3, 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint drXiv:/502.03167,22 2015.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. arXiv preprint arXiv:1802.05956 2018.
Diederik P Kingma and Max Welling. arXiv: 2013.
Auto-encoding variational bayes.
arXiv preprint
Alexander Kolesnikov and Christoph Lampert. Seed, expand and constrain: Three principles for weakly-supervised image segmentation. In Proceedinggs of the European conference on computer vision, pp. 695-711. Springer, 2016.
Victor Lempitsky and Andrew Zisserman. Learning to count objects in images. In Advances in neural information processing systems, pp. 1324-1332, 2010.
Dong Li, Jia-Bin Huang, Yali Li, Shengjin Wang, and Ming-Hsuan Yang. Weakly supervised ob- ject localization with progressive domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3512-3520 2016.
Chen Change Loy, Ke Chen, Shaogang Gong, and Tao Xiang. Crowd counting and profiling: Methodology and evaluation. In Modeling, simulation and visual analysis of crowds, pp. 347- 382. Springer, 2013.
Yunchen Pu, Zhe Gan, Ricardo Henao, Xin Yuan, Chunyuan Li, Andrew Stevens, and Lawrence Carin. Variational autoencoder for deep learning of images, labels and captions. In Advances in neural information processing systems, pp. 2352-23660 2016.
Weiqiang Ren, Kaiqi Huang. Dacheng Tao, and Tieniu Tan. Weakly supervised large scale object lo- calization with multiple instance learning and bag splitting. IEFE transactions pattern analysis and machine intelligence, 38(2):405 2016.
Yunhan Shen, Rongrong Ji, Shengchuan Zhang, Wangmeng Zuo, and Yan Wang. Generative adver- sarial learning towards fast weakly supervised detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5764-5773, 2018.
Zhiyuan Shi, Timothy M. Hospedales, and Tao Xiang. Bayesian joint topic modelling for weakly supervised object localisation. In Proceedings of the IEEE international conference computer vision, December 2013.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Vi- sualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034 2013.
Parthipan Siva, Chris Russell, and Tao Xiang. In defence negative mining for annotating weakly labelled data. In Proceedings of the European conference on computer vision, pp. 594-608. Springer, 2012.
Parthipan Siva, Chris Russell, Tao Xiang, and Lourdes Agapito. Looking beyond the image: Un- supervised learning for object saliency and detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, June 2013.
Hyun Oh Song, Ross Girshick, Stefanie Jegelka, Julien Mairal, Zaid Harchaoui, and Trevor Darrell. On learning to localize objects with minimal supervision. arXiv preprini urXiv: 1ooxxv::4400 1403. 1024, 2014a.
Hyun Oh Song, Yong Jae Lee, Stefanie Jegelka, and Trevor Darrell. Weakly-supervised discovery of visual pattern configurations. In Advances in neural information processing systems, pp. 1637-1645. Curran Associates, Inc., 2014b. URL http::/papprs...../// www.. h
pdf
10

---

Under review as a conference paper at ICLR 2020
Peng Tang, Xinggang Wang, Xiang Bai, and Wenyu Liu. Multiple instance detection network with online instance classifier refinement. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2-441-851,.2 2017.
Fang Wan, Pengxu Wei, Jianbin Jiao, Zhenjun Han, and Qixiang Ye. Min-entropy latent model for weakly supervised object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, June 2018.
Yunchao Wei, Jiashi Feng, Xiaodan Liang, Ming-Ming Cheng, Yao Zhao, and Shuicheng Yan. OB- ject region mining with adversarial erasing: A simple classification to semantic segmentation approach. Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2017.
Weidi Xie, J. Alison Noble, and Andrew Zisserman. Microscopy cell counting and detection with fully convolutional regression networks. Computer methods in biomechanics and biomedical en- gineering: imaging visualization, 6(3):283-299) 292, 2018. doi: 10. 10.100022688 16111163.11000 149104. URL http:ttttp...oo.oo/o/ooooo/ wttp:///.o...........o /ttp:///........./////1111...... 201 111.:1111..211.11111111111 1149104
Yingying Zhang, Desen Zhou, Siqin Chen, Shenghua Gao, and Yi Ma. Single-image crowd count- ing via multi-column convolutional neural network. Proceedings of the IEEE conference on computer vision and pattern recognition, June 2016.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2921-2929, 2016.


Reviews:
[{"review_id": "rylrszTH9H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The main contribution of the paper is the extension of techniques for weakly supervised localization, i.e. given ground truth counts of objects in a given image, one can do training to generate hidden layer density maps that allow for feature detection and localization of objects.   The main contribution of the paper seems to be the regularization of the density map by incorporation of a Gini impurity penalty and the contrasting of the regularizer against beta-variational autoencoder formulation. The experiments show the utility of the method in a toy example and an example involving pedestrians in video surveillance.  \n\nWhile the extensions are interesting, I find that the paper contributions are incremental and do not offer sufficient experimental results. The experiments are qualitative (just illustrations of results on a few images).  While I understand the context of the contribution from authors point of view and can appreciate the study of weakly-supervised learning, I find it important to ask how the present framework compares with a traditional method that uses background subtraction (thus gaining appearance invariance to an extent) followed by geometric prior driven indexing of counts and locations (see for example:  Lan Dong et al (ICCV 2007 - Fast Crowd Segmentation via Shape Indexing). The inductive bias and interpretability of design is extremely transparent and obvious.  Can you please elaborate how your work compares in performance to such a technique and what you mean by feature interpretability in your context? \n\nMy rating is based on the fact that the results are preliminary and further quantitative experimentation is needed.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "HJxfMpScdB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This article proposes a method for object counting which can be trained with weak supervision. Object counting methods are often trained with point annotations, i.e., one click-point per object. In this article, a weaker way of annotation is used: count-based annotation, i.e., the number of objects of each class present in the image are given as annotation but no precise location of the objects. This article is an extension of density-based object counting methods for weakly supervision.\n\nThis article analyzes the effect of pooling layers on WSL. The findings indicate that the pooling layer constrained the objects' locations to a predefined grid and accordingly they remove it. When removing this effect is alleviated. Moreover, the authors realized that a sift on the predictions also happen. Accordingly, they learn rotated versions of the model for centering the predictions. As a countereffect, several points per object are detected, and then a gini impurity regularization is proposed to reduce the number of detected objects. Then, the authors connect this formulation with VAE. \n\nThe experiments are conducted on a toy example, i.e., circle finding, and on the well established Mall dataset.\n\nMain concerns:\nThe experimental section should make experiments on standard datasets (i.e., USCD, Trancos, Mall, PkLot, Shangai, Penguins) using standard evaluation protocols (MAE, GAME). The current evaluation only shows the qualitative results on one frame of the Mall dataset.\n\nThe results of the proposed method should be compared with baselines. For instance, Glance method, which is only trained with count-level annotation, and with \"C-WSL: Count-guided Weakly Supervised Localization\". It should be also compared with WSL methods based on image-level labels such as \"Object Counting and Instance Segmentation with Image-level Supervision\" or \"Where are the Masks: Instance Segmentation with Image-level Supervision\" and then with methods that use point supervision as an upper bound such as density method or Where are the blobs.\n\nMoreover, some ablation studies about the effect of each component of the model would be needed.\n\nRegarding the model I have some concerns:\n - Removing the pooling layers does not seem a very good idea. This will remove part of the ability to detect objects at multiple scales and moreover will increase its computational power. Maybe for easy cases as circles and heads of mall people where the resolution changes are small could work. But not for more challenging datasets.\n - The rotation of the model could be ok for circles or heads. But making the model to learn rotation-invariant features could be challenging in more difficult datasets.\n - Eq.5 and 8 which are used in the model seem to use point supervision and not just the count based. Which makes the method not a weakly supervised one. Are these equations the ones used or others?\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BygI0NATYB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method to train a network for counting/localizing objects in a weakly supervised framework.\nThe network is optimized based on weak supervision about target object counts; training images have only the ground truth of the total number of object counts without their positions in images.\nThrough analyzing the feature maps of the network trained on toy bubble images, the authors propose two regularization techniques for rotation robustness and sparseness of the map in order to improve  performance of object localization in the feature map.\nThe experimental results on object counting tasks using the Mall dataset show that the proposed method produces favorable performance.\n\nThis paper is leaning toward rejection due to the following two reasons.\n(1) The presented techniques are limited to the specific task of object counting.\n(2) They are derived in an ad-hoc way based on less theoretical background and thus lack novelty.\n\nThe detailed comments are as follows.\n\n* The technique to embed rotation invariance by Eq.(4) is presented in an ad-hoc way. This seems to be limited to the tasks on toy data such as bubble images shown in the paper. In the real-world tasks/images to count the more complicated objects, the spatial correlation among object parts is an important clue to provide discriminative features for detecting objects such as torso below the head in human figures. Such spatial dependency could be missed by imposing the rotation invariance on the network.\n\n* The regularization by Gini impurity is a well-known technique to induce sparsity, lacking novelty. There is a large body of researches for enhancing sparsity, and thus to validate the regularization, the authors should compare the Gini impurity with the other sparsity-inducing regularization such as entropy. For example, it could be possible to argue the regularizations in terms of their derivatives (gradients) to be used for back-propagation. This paper lacks detailed analysis and discussion about the regularization in this counting framework.\n\n* It is unclear how the perspective from the (beta-)VAE contributes to the analysis of the proposed method. There seems to be less connection between Eq.(8) and the object counting task/framework. The authors just show the similarity between the presented method and VAE in terms of formulation. Since the proposed method belongs to just a simple optimization with sparsity-inducing regularization, the reviewer cannot find any convincing reason to discuss the connection to VAE.\n\n* The method is not fully validated in the experiments. The authors provide only one experimental result on the Mall dataset which is insufficient to validate the effectiveness of the proposed method. Considering that the method is limited to the specific task of object counting, it is necessary to qualitatively and thoroughly evaluate the performance on various datasets that exhibit various environmental conditions regarding such as lighting and occlusion.\n\nMinor comments:\n- Show the index for summation in Eqs.(1,4,6,7,8).\n- It is confusing to show q_\\Theta(D|I) in Eq.(3), even though the authors aim to discuss the connection to VAE. Show the mathematical definition of q_\\Theta(D|I) before that.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes a new regularization method for weakly supervised localization based on counting.
Reviewers agree that this is an interesting topic but the experimental validation is weak (qualitative, lack of baselines), and the contribution too incremental.
Therefore, we recommend rejection.
Decision: Reject