Deep Feature Aggregation with Heat Diffusion for Image Retrieval

Image-level feature descriptors obtained from convolutional neural networks have shown powerful representation capabilities for image retrieval. In this paper, we present an unsupervised method to aggregate deep convolutional features into compact yet discriminative image vectors by simulating the dynamics of heat diffusion. A distinctive problem in image retrieval is that repetitive or bursty features tend to dominate feature representations, leading to less than ideal matches. We show that by considering each deep feature as a heat source, our unsupervised aggregation method is able to avoiding over-representation of bursty features. We additionally provide a practical solution for the proposed aggregation method, and further show the efficiency of our method in experimental evaluation. Finally, we extensively evaluate the proposed approach with pre-trained and fine-tuned deep networks on common public benchmarks, and show superior performance compared to previous work. Image retrieval has always been an attractive research topic in the field of computer vision. By allowing users to search similar images from a large database of digital images, it provides a natural and flexible interface for image archiving and browsing. Convolutional Neural Networks (CNNs) have shown remarkable accuracy in tasks such as image classification, and object detection. Recent research has also shown positive results of using CNNs on image retrieval (Babenko and Lempitsky 2015; Kalantidis, Mellina, and Osindero 2016; Hoang et al. 2017). However, unlike image classification approaches which often use global feature vectors produced by fully connected layers, these methods extract local features depicting image patches from the outputs of convolutional layers and aggregate these features into compact (a few hundred dimensions) image-level descriptors. Once meaningful and representative image-level descriptors are defined, visually similar images are retrieved by computing similarities between pre-computed database feature representations and query representations. In this paper we devise a method to avoid overrepresenting bursty features. Inspired by an observation of similar phenomena in textual data, Jegou et al. (Jégou, Douze, and Schmid 2009) identified burstiness as the phenomenon by which overly repetitive features within an instance tend to dominate the instance feature representation. In order to alleviate this issue, we propose a feature aggregation approach that emulates the dynamics of heat diffusion. The idea is to model feature maps as a heat system where we weight highly the features leading to low system temperatures. This is because that these features are less connected to other features, and therefore they are more distinctive. The dynamics of the temperature in such system can be estimated using the partial differential equation induced by the heat equation. Heat diffusion, and more specifically anisotropic diffusion, has been used successfully in various image processing and computer vision tasks. Ranging from the classical work of Perona and Malik (Perona and Malik 1990) to further applications in image smoothing, image regularization, image co-segmentation, and optical flow estimation (Zhang, Zheng, and Cai 2010; Tschumperle and Deriche 2005; Kim et al. 2011; Bruhn, Weickert, and Schnörr 2005). However, to our knowledge, it has not been applied to weight features from the outputs of a deep convolutional neural network. We show that by combining this classical image processing technique with a deep learning model, we are able to obtain significant gains against previous work. Our contributions can be summarized as follows: • By greedily considering each deep feature as a heat source and enforcing the temperature of the system be a constant within each heat source, we propose a novel efficient feature weighting approach to reduce the undesirable influence of bursty features. • We provide a practical solution to computing weights for our feature weighting method. Additionally, we conduct extensive quantitative evaluations on commonly used image retrieval benchmarks, and demonstrate substantial performance improvement over existing unsupervised methods for feature aggregation.