Objects Counting by Estimating a Density Map With Convolutional Neural Networks | by Tomasz Bonus | Towards Data Science

Written by Tomasz Bonus and Tomasz Golan.
Counting objects in images is one of the fundamental tasks in computer vision. It has a lot of applications, in particular in:
microbiology (e.g. counting bacterial colonies in a Petri dish);
surveillance (e.g. counting people);
agriculture (e.g. counting fruits or vegetables);
medicine (e.g. counting tumor cells in histopathological images);
wildlife conservation (e.g. counting animals).
The task of counting objects is relatively easy for us, people, but it can be challenging for a computer vision algorithm, especially when different instances of an object vary significantly in terms of shape, color, texture or size. If a problem is complex from an algorithmic point of view but simple from a human point of view, machine learning methods can be an answer.
Currently, deep learning (DL) methods provide the state-of-the-art performance in digital image processing. However, they require collecting a lot of annotated data, which is usually time consuming and prone to labelling errors.
The common way of count objects using DL is to first detect them using convolutional neural networks, like e.g. GCNet [1], and then count all found instances. It is effective but requires bounding box annotations, like presented in Fig. 1 (left), which are hard to obtain. To overcome this issue, alternative approaches leverage point-like annotations of objects positions (see Fig. 1 right), which are much cheaper to collect.
In this article we describe our studies on counting objects in images with fully convolutional networks (FCN), trained on data with point-like annotations. In the next sections details on a model we used are presented together with the implementation, considered datasets, and results we obtained.
We started with the approach described in [7]. The main idea is to count objects indirectly by estimating a density map. The first step is to prepare training samples, so that for every image there is a corresponding density map. Let’s consider an example shown in Fig. 2.
The image presented in Fig. 2 (left) is annotated with points in the positions of pedestrians heads (Fig. 2 right). A density map is obtained by applying a convolution with a Gaussian kernel (and normalized so that integrating it gives the number of objects). The density map for the example above is presented in Fig. 3.
Now, the goal is to train a fully convolutional network to map an image to a density map, which can be later integrated to get the number of objects. So far, we have considered two FCN architectures: U-Net [8] and Fully Convolutional Regression Network (FCRN) [7].
U-Net is a widely used FCN for image segmentation, very often applied to biomedical data. It has autoencoder-like structure (see Fig. 4). An input image is processed by a block of convolutional layers, followed by a pooling layer (downsampling). This procedure is repeated several times on subsequent blocks outputs, which is demonstrated on the left side of Fig. 4. This way the network encodes (and compresses) the key features of an input image. The second part of U-Net is symmetric, but pooling layers are replaced with upsampling, so that an output dimensions match the size of an input image. The information from higher resolution layers in the downsampling part is passed to corresponding layers in the upsampling part, which allows to reuse learned higher level features to decode contracted layers more precisely.
Fully Convolutional Regression Network (FCRN) was proposed in [7]. The architecture is very similar to U-Net. The main difference is that the information from higher resolution layers from downsampling part is not passed directly to the corresponding layers in upsampling part. In the paper two networks are proposed: FCRN-A and FCRN-B, which differ in downsampling intensity. While FCRN-A perform pooling every convolutional layer, FCRN-B does that every second layer.
Our implementation can be found here. It is based on the code from Weidi Xi’s GitHub, but PyTorch is used instead of Keras.
Currently, U-Net and FCRN-A are implemented. They both use three downsampling and three upsampling convolutional blocks with fixed filter size 3×3. By default there are two convolutional layers in each block for U-Net, and one for FCRN-A. For U-Net we keep constant number of filters for all convolutional layers, and for FCRN-A we increase this number every subsequent layer to compensate for the loss of higher resolution information caused by pooling (which is not passed directly as in the case of U-Net).
The basic piece to build both U-Net and FCRN is a convolutional block, consisting of a convolutional layer, batch normalization, and activation function:
The conv_block function creates N convolutional layers with OUT number of filters with ReLU activation function and batch normalization applied in each layer.
The FCRN-A architecture is obtained by stacking multiple such blocks followed by either downsampling (max pooling) or upsampling layers:
U-Net also requires to concatenate the output from the downsampling path with the input to the corresponding layer in the upsampling part, which is performed by ConvCat class:
The U-Net downsampling part is built in the same way as in the case of FCRN:
However, as mentioned above, upsampling exploits ConvCat class:
We considered three datasets in our study. They all are annotated with point-like objects positions, so we could use them directly to generate density maps for all images and test the method described above.
We provide the get_data.py script to preprocess all considered datasets to a common format stored in HDF5 files. Each entry consists of an image and a corresponding label (a density map) generated with generate_labelfunction:
Note, that we set a value of 100 for each object position before applying Gaussian filter, so the results must be properly normalized when integrating a density map to get the number of objects present on an image.
For visualization purpose we used higher standard deviation value for density maps presented on images comparing to the one we actually apply for training dataset.
Fluorescent cells (FC) dataset is generated by Visual Geometry Group (VGG) with a computational framework from [9]. It can be downloaded from VGG website. An example image along with generated density map is presented in Fig. 5.
UCSD dataset [10] contains videos of pedestrians recorded on walkways in the University of California San Diego campus. It is widely used for various problems, such as counting, motion segmentation, and analysis of pedestrians behaviour. It can be downloaded from Statistical Visual Computing Lab website. An example image along with generated density map is presented in Fig. 6.
Mall dataset [3–6] was created for crowd counting and profiling. It contains a video recorded by a publicly available webcam. Every frame is annotated with head positions of every pedestrian. It can be downloaded from here. An example image along with generated density map is presented in Fig. 3.
The method of counting objects in images by integrating an estimated density map has been already applied to both fluorescent cells and UCSD datasets [11]. We chose mall dataset to be our test dataset for the method.
As stated above, two models were tested, namely U-Net and FCRN. Using U-Net we were able to achieve more accurate results so below we present our findings obtained with this architecture.
In the table below the summary of our results is presented for each dataset with minimum and maximum numbers of objects in validation sets and the mean absolute error (MAE) we obtained.
We used standard definition for MAE:
where ti is true and pi is predicted number of objects for i-th sample.
       Dataset      | Min. #objects | Max. #objects | MAE   -------------------|---------------|---------------|------   Fluorescent cells |            74 |           317 | 1.89   UCSD              |            20 |            47 | 2.27   Mall              |            20 |            48 | 2.86
The scatter plots for true (ti) vs predicted (pi) number of objects for each validation sample are presented in Figs. 7–9. As expected, the model handles well relatively simple fluorescent cells dataset despite high number of objects in single image. However, there is much higher deviation when counting pedestrians. This could be happening due to the fact that it is difficult even for human labeler to decide whether person standing behind a plant or just barely visible from behind corner should be counted.
This is just the beginning of our research on object counting. We are looking forward to conduct more experiments including trying out different architectures and methods.
[1] Cao, Y., Xu, J., Lin, S., Wei, F., & Hu, H. (2019). GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyond. arXiv preprint arXiv:1904.11492.
[2] Lin, T. Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., ... & Zitnick, C. L. (2014, September). Microsoft coco: Common objects in context. In European conference on computer vision (pp. 740–755). Springer, Cham.
[3] Change Loy, C., Gong, S., & Xiang, T. (2013). From semi-supervised to transfer counting of crowds. In Proceedings of the IEEE International Conference on Computer Vision (pp. 2256–2263).
[4] Chen, K., Gong, S., Xiang, T., & Change Loy, C. (2013). Cumulative attribute space for age and crowd density estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2467–2474).
[5] Loy, C. C., Chen, K., Gong, S., & Xiang, T. (2013). Crowd counting and profiling: Methodology and evaluation. In Modeling, simulation and visual analysis of crowds (pp. 347–382). Springer, New York, NY.
[6] Chen, K., Loy, C. C., Gong, S., & Xiang, T. (2012, September). Feature mining for localised crowd counting. In BMVC (Vol. 1, No2, p. 3).
[7] Weidi, X., Noble, J. A., & Zisserman, A. (2015). Microscopy cell counting with fully convolutional regression networks. In 1st Deep Learning Workshop, Medical Image Computing and Computer-Assisted Intervention (MICCAI).
[8] Ronneberger, O., Fischer, P., & Brox, T. (2015, October). U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention (pp. 234–241). Springer, Cham.
[9] Lehmussola, A., Ruusuvuori, P., Selinummi, J., Huttunen, H., & Yli-Harja, O. (2007). Computational framework for simulating fluorescence microscope images with cell populations. IEEE transactions on medical imaging, 26(7), 1010–1016.
[10] Chan, A. B., Liang, Z. S. J., & Vasconcelos, N. (2008, June). Privacy preserving crowd monitoring: Counting people without people models or tracking. In 2008 IEEE Conference on Computer Vision and Pattern Recognition (pp. 1–7). IEEE.
[11] Lempitsky, V., & Zisserman, A. (2010). Learning to count objects in images. In Advances in neural information processing systems (pp. 1324–1332).