Hypergraph Propagation and Community Selection for Objects Retrieval

1 Introduction . Image search is a fundamental problem in computer vision with numerous applications such as content-based image browsing [ 36 , 19 , 1 ] , visual localization [ 2 , 30 ] , and 3D reconstruction [ 15 , 31 ] . To retrieve the database images that contain the same object of a query , a search engine represents images using global or local features [ 5 , 21 , 33 , 26 , 3 ] that describe their similarities , and then performs a nearest neighbor search for the query image . An object is depicted in various conditions in the real world [ 25 ] . Global features of images depicting the same object form a manifold in the feature space [ 39 ] . Euclidean-distance-based nearest neighbor search [ 20 , 4 ] does not work well on this manifold . To improve the retrieval performance , query expansion [ 9 , 13 , 27 , 3 , 14 ] issues a new query by aggregating useful information of nearest neighbors of the prior query . Unfortunately , this approach only explores the neighborhood of very similar images , resulting in a low recall [ 16 ] . Diffusion [ 16 , 39 ] explores more images on a neighborhood graph of the dataset . It , however , adopts false positive items in practice , leading to a low precision [ 25 ] . Spatial verification [ 23 ] uses spatial information to detect the true positive images . However , due to its heavy computational overhead , it is impractical to apply spatial verification to all the images during diffusion . This paper aims to pre-compute the spatial matching information of database images offline and quickly propagate them for a given query online . To achieve this goal , we propose a hypergraph propagation model . Our model efficiently builds a hypergraph for a given query and propagates the spatial relationship with the query through hyperedges in a sequence of images . When the query image is not in the database , diffusion represents it using its top-ranked images in the initial search [ 16 , 37 ] . However , not all the top-ranked images contain the object of the query [ 16 , 25 ] . Some works [ 25 , 6 ] show that applying spatial verification before diffusion improves the retrieval 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . performance , but it requires heavy computational overheads [ 25 , 9 ] . The second contribution of this paper is the proposal of an initial search technique , community selection , which reduces the overhead of spatial verification while maintaining high accuracy . Experimental result shows that our hypergraph propagation significantly outperforms the existing query expansion and diffusion methods . On the hard protocol of the ROxford dataset , our hypergraphpropagation-based approach achieves the impressive mAPs of 73.0 and 60.5 with and without R1M distractors , respectively . In conclusion , our contributions are : 1 ) we propose a novel hypergraph model to efficiently propagate the spatial information in a sequence of images to improve the search performance ; 2 ) we propose community selection to predict the accuracy of the initial search and to provide correct starting points for hypergraph diffusion ; 3 ) we achieve state-of-the-art performance on the ROxford dataset with and without R1M distractors . 2 Background . 2.1 Query expansion and diffusion : relation based search . Query expansion ( QE ) [ 9 ] aggregates the top-ranked initial candidates into an expanded query , which is then used to search more images in the database . The critical factor here is how to aggregate only helpful information and cull out unrelated information . Average query expansion ( AQE ) [ 9 ] mean-aggregates the top k retrieved images as the expanded query . Average query expansion with decay ( AQEwD ) [ 13 ] gives the top k images the monotonically decaying weights over their rank . Alpha query expansion ( αQE ) [ 27 ] uses the power-normalized similarity between the query and the top-ranked images as the aggregation weights . Discriminative query expansion ( DQE ) [ 3 ] also uses the weighted average , where the weight is the dual-form solution of an SVM , which classifies the positive ( top-ranked ) and negative ( low-ranked ) images of a query . Learnable attention-based query expansion ( LAttQE ) [ 14 ] uses self-attention [ 34 ] to share information between the query and the top-ranked items to compute better aggregation weights . Unfortunately , these QE methods only explore the neighborhood of very similar images of the query . The recursive methods [ 9 ] improve the results by recursively generating new queries from all spatially verified results returned so far . However , this is at the cost of increased query time . Diffusion [ 39 , 16 ] ameliorates the problem mentioned above by propagating similarities through a pairwise affinity matrix . The works of Chang et al . [ 6 ] and Liu et al . [ 17 ] are two novel variants of this line . The critical issue in diffusion is how to calculate the affinity matrix . Some works [ 24 , 16 ] use the reciprocal neighborhood relations to refine the search results , and other [ 23 , 6 ] use the inliers number of SP to re-weight the similarity . However , all of these methods compress the relations between two images as a single scala . As a result , these prior methods lost the spatial matching detail . Although the propagation methods are elegant , they have difficulties recovering good relations to guide the diffusion direction . We propose a novel and straightforward hypergraph model to utilize the relations among database images better . It connects the corresponding local features among images using hyperedges ; in a sequence of similar images , it detects the image from which the following images are no longer relevant to the query . 2.2 Local descriptors and spatial verification . Local descriptors are the representations of an image ’ s important patches . Some popular local descriptors are RootSIFT [ 3 ] , deep local feature ( DELF ) [ 21 ] and deep local and global feature ( DELG ) [ 5 ] . Local descriptors ’ location information is usually used in the spatial verification ( SP ) [ 22 ] stage . A standard SP pipeline is first using a KD-tree [ 22 ] to find the nearest neighbors between local descriptors in two images . Then it uses the RANSAC algorithm [ 12 ] to estimate the homography matrix [ 32 ] and the number of inlier correspondences between two images . Existing image search engines commonly apply SP on the shortlist , say the top 100 images ranked by a global descriptor . SP is known to be crucial for verifying true positive images [ 25 ] . Performing SP before diffusion improves the final retrieval performance [ 25 , 6 ] . However , it is also the slowest process [ 3 , 23 ] in image search . In this paper , we propose community selection to reduce the computing overhead of SP . Community selection frames the diffusion initialization task as selecting a community for the given query rather than verifying the top-ranked images one by one . Experiment shows that community selection reduces much SP overhead without hurting the accuracy . 3 Overview . We first discuss issues of query expansions and diffusion process of ordinary graphs in Sec.3.1 and our novel hypergraph based propagation ( Sec . 3.2 ) and initialization scheme utilizing the concept of community ( Sec . 3.3 ) for achieving accurate diffusion process . 3.1 Propagating in ordinary graph . We can use a graph G = ( X , F ) to represent the database images . X : = { x1 , ... , xm } is the node set , where item xi could be either an image or an image region depending on a chosen retrieval method ; F is the set of edges whose scalar weight represents the similarity between two nodes connected by each edge . Utilizing information in this ordinary graph is an important strategy to improve retrieval results , and , thus , widely used in various methods , such as query expansion [ 9 , 8 , 27 , 14 ] and diffusion [ 16 , 39 ] . Both diffusion and query expansion can be represented by the following function : f∗ = I ( f0 ) , ( 1 ) where I represents the post-processing process , fu ∈ R is the ranking score for xu , f0 is the initial ranking vector with f0u = 1 if xu is a query and f0u = 0 otherwise , and f ∗ is the ranking vector after query expansion or diffusion . Diffusion methods diffuse the label of the query node to its neighbors along the graph edges until reaching a stationary state . The stationary state is found by repeating the iteration function : ft+1 = αOft + ( 1−α ) f0 , where O is the normalized affinity matrix that describes the weights of graph edges , α is a jumping probability . Query expansion gathers useful information in the query node ’ s nearest neighbors to update the ranking list . Diffusion and query expansion have different advantages and disadvantages . Diffusion can efficiently propagate information to the whole graph through a compact formulation . However , the propagation is conducted through the affinity matrix O [ 39 , 16 ] , whose scalar value can not represent accurate spatial information , resulting in low precision . Take Figure 1 as an example , where images of x1 and x5 are incorrectly connected through image x2 in the ordinary graph with scalar edge weights ; note that they do not share common regions . This leads to false positives in the diffusion process . We denote this as the ambiguity problem of propagation . In contrast , query expansion uses sophisticated approaches , such as transformer [ 14 ] and spatial verification [ 9 , 3 ] , to guarantee that only shared information between query and its nearest neighbors are used to update the ranking list . This approach , however , only explores the neighborhood of very similar images due to the computational overhead [ 25 , 3 , 9 ] , resulting in a low recall . This paper proposes novel hypergraph propagation and community selection , settling the ambiguity problem separately in the propagation and initialization stages . We describe these two techniques next . 3.2 Extension to hypergraph . One meaningful direction to overcome the drawbacks of diffusion and query expansion is to handle the ambiguity problem of graph propagation by explicitly resolving the propagation in a more informative space . The local features contain informative spatial information that can help to distinguish the ambiguity through geometric verification [ 22 ] . However , considering the large scale of the image database , it is impractical to do the geometric verification on all the pairs . In practice , image search engines usually build the kNN graph instead of the complete graph , which means every image only connects with its top k similar images [ 16 ] . To utilize the spatial information on the kNN graph , we propose a novel hypergraph propagation model to efficiently propagate spatial information through hyperedges , connecting an arbitrary number of nodes . This technique is appropriate for the real-world situation where the same object in different images has a different number of local features . In the example of Figure 1 , image x1 can connect image x3 by matching the local features with those of image x2 , but will not connect image x5 . When a query is given , we define a hypergraph H = ( Y , E ) on top of the ordinary image graph , where its node set Y = { y1 , ... , yn } contains all the local features extracted , and the edge set E consists of hyperedges , which are used to connect the local features related to the query object in different images . Our model then propagates the label of local features of the query to their neighbors and uses an aggregation function to find the final ranking list of images , as follows : l∗ = Ih ( l0 ) , f∗ = Agg ( l∗ ) , ( 2 ) where l0 and l∗ are the initial and final ranking lists of local features respectively , f∗ is the ranking list of images , Ih is the propagation function in a hypergraph , and Agg is the aggregation function that converts ranking scores of local features to ranking scores of images . We will discuss the construction of hypergraph in Section 4.1 and the propagation process in Section 4.2 .

Summary: This paper proposes a new query expansion method for image object retrieval. Firstly, they propose a hypergraph among local features of base images. To expand new queries, it is time-consuming to check whether the query is related to initial search results. They propose a community selection method to speed up the procedure. In experiments, the proposed method outperforms traditional methods.