Feedback-based Dynamically Weighted BoF for Image Retrieval

Bag of Features (BoF) has been successfully exploited in content-based image retrieval for several years. Due to its performance and popularity, several extensions have been proposed that involve feature description, dictionary building, feature encoding and post-query process, etc. This paper proposes a dynamically weighting scheme for BoF-based image retrieval based on feedback. It involves two contributions: (i) analyzing the statistical distribution characteristic of similar BoF representations and (ii) computing weights dynamically based on the feedback obtained from different initial query results. We quantitatively evaluate the proposed method on two different databases. Experiments confirm that the proposed weighting scheme has better performance than the baseline of BoF-based image retrieval systems. Meanwhile, the results demonstrate the effectiveness of the weighting scheme in terms of the precision of top-N returned images. Introduction Content-Based Image Retrieval (CBIR) [1,2,3] is facing difficulties with the ever increasing size of databases. Bag of Features (BoF) [4], originated from the textual information retrieval literature, has been playing an important role in the development of CBIR. Many state-of-art image retrieval systems were built upon the BoF representation of images, some shown to work well on large scale databases [4,5]. BoF representation of images involves two main aspects: dictionary building and feature quantization. First proposed by Sivic and Zisserman [6], their BoF-based system returns query result based on a similarity score between fixed length vectors of query and target images. Since then, many researchers have extended on this fundamental approach to include (1) feature detection [7,8]and description improvements [9,10], (2) clustering [4,5] and quantization methods [11,12,13] and (3) result reranking by post-query rank adjustment [14,15]. More recent researches tend to focus on the last two aspects. In this paper, we focus on the post-query processing of BoF-based retrieval system, and propose a dynamically weighted BoF (DWBoF) by utilizing the information of positive instances labeled in the initial query result. Afterwards, the similarity score is re-computed to rerank the query result. This paper is organized as follows. The related works are summarized in Section 2. Section 3 provides the details of the proposed method, and the experiment results are given in Section 4 followed by the conclusion in Section 5. Related Works The concept of using BoF as visual words to build dictionaries has been intensively studied in visual search. The local patches should be obtained either by random sampling [7]. or key-point detection [8], often described by SIFT [8] or other descriptors. In Ref [10], Spatial Pyramid Matching were proposed using SIFT descriptors to describe dense grids with the spacing of 8 pixels. Another key aspect of BoF representation for images is clustering and quantization. For clustering the features, k-means is the most often used method, and a number of variations have been proposed. For example, Phibin et.al. [4] used approximate k-means to build the dictionary. Nister and Stewenius 3rd International Conference on Materials Engineering, Manufacturing Technology and Control (ICMEMTC 2016) © 2016. The authors Published by Atlantis Press 1539 [5] proposed a hierarchical k-means to represent the vocabulary tree. After obtaining the dictionary, feature descriptors should be encoded to obtain a vector with fixed length, according to the visual words in the dictionary. Many encoding techniques have also been proposed such as histogram encoding, kernel code-book encoding [11], Fisher encoding [12], locality-constrained linear encoding (LLC) [13], etc. Kernel codebook encoding [11] is a variant in which descriptors are assigned to the visual words in a soft manner, rather than histogram encoding. In essence, Fisher encoding [12] captures the average first and second order differences between the image descriptors and the center of a Gaussian Mixture Model (GMM), where GMM is exploited for clustering in the process of building the dictionary. LLC [13] projects each descriptor down to the local linear subspace spanned by M (M<<dictionary size) visual words closest to the current descriptor. There also have been several proposals in the literature on improving the performance of BoF-based image retrieval process by post-query processing. Jegou et.al. [14] exploited rank aggregation to improve retrieval results, which performed retrieval for several times using separate vocabularies for each query. Chum et.al. [15] proposed query expansion, which re-submitted the top-ranked results from the initial query as the additional queries in an attempt to increase the recall at a given precision. Broadly, our work aims to improve retrieval performance based on the initial query. It is motivated by the analyzing the statistical distribution characteristics of the similarities in BoF representations. Section 3 gives the details of our method. Dynamically Weighting Scheme For BoF Codebook Generation and Local Feature Encoding. The general procedure of generating a BoF model for representing images is shown in Fig.1 and can be summarized as follows: (1) Building dictionary: Extract local features for all the images in the dataset using some low feature descriptor, then cluster which to build the dictionary (also called visual codebook) with the cluster center as the term (code) of the dictionary. (2) Assign terms to generate a BoF representation for a given image. Training set Feature sets Dictionary (Codebook ) Query image Feature vector Bag of visual words Term vector Extract features Cluster features Assign terms Extract features Fig.1 Process of building BoF representation for an image. Level 0, weight=1/4 Level 1, weight=1/2 Level 2, weight=1/4 Fig.2 Spatial Pyramid with three levels. In this paper, Hessian-affine detector[16] is used to obtain local elliptical regions and SIFT descriptor is exploited to describe the extracted regions. Histogram Intersection Kernel (HIK) k-means[17] is then used to generate the codebooks. In order to take spatial information of images into account, spatial pyramid[10] is employed when encoding local features. Fig.2 shows spatial pyramid with three levels. The quantization is based on hard assignment which projects a local patch onto its nearest word according to the distances between its feature descriptor and words. After assigning each local patch, the algorithm proceeds to count the visual words that fall in each spatial bin, and define their appearance frequency as the histogram descriptor for each bin with weights shown in Fig.2. Finally, the SPBoF representation for an image is