Hypergraph Propagation and Community Selection for Objects Retrieval

1 Introduction . Image search is a fundamental problem in computer vision with numerous applications such as content-based image browsing [ 36 , 19 , 1 ] , visual localization [ 2 , 30 ] , and 3D reconstruction [ 15 , 31 ] . To retrieve the database images that contain the same object of a query , a search engine represents images using global or local features [ 5 , 21 , 33 , 26 , 3 ] that describe their similarities , and then performs a nearest neighbor search for the query image . An object is depicted in various conditions in the real world [ 25 ] . Global features of images depicting the same object form a manifold in the feature space [ 39 ] . Euclidean-distance-based nearest neighbor search [ 20 , 4 ] does not work well on this manifold . To improve the retrieval performance , query expansion [ 9 , 13 , 27 , 3 , 14 ] issues a new query by aggregating useful information of nearest neighbors of the prior query . Unfortunately , this approach only explores the neighborhood of very similar images , resulting in a low recall [ 16 ] . Diffusion [ 16 , 39 ] explores more images on a neighborhood graph of the dataset . It , however , adopts false positive items in practice , leading to a low precision [ 25 ] . Spatial verification [ 23 ] uses spatial information to detect the true positive images . However , due to its heavy computational overhead , it is impractical to apply spatial verification to all the images during diffusion . This paper aims to pre-compute the spatial matching information of database images offline and quickly propagate them for a given query online . To achieve this goal , we propose a hypergraph propagation model . Our model efficiently builds a hypergraph for a given query and propagates the spatial relationship with the query through hyperedges in a sequence of images . When the query image is not in the database , diffusion represents it using its top-ranked images in the initial search [ 16 , 37 ] . However , not all the top-ranked images contain the object of the query [ 16 , 25 ] . Some works [ 25 , 6 ] show that applying spatial verification before diffusion improves the retrieval 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . performance , but it requires heavy computational overheads [ 25 , 9 ] . The second contribution of this paper is the proposal of an initial search technique , community selection , which reduces the overhead of spatial verification while maintaining high accuracy . Experimental result shows that our hypergraph propagation significantly outperforms the existing query expansion and diffusion methods . On the hard protocol of the ROxford dataset , our hypergraphpropagation-based approach achieves the impressive mAPs of 73.0 and 60.5 with and without R1M distractors , respectively . In conclusion , our contributions are : 1 ) we propose a novel hypergraph model to efficiently propagate the spatial information in a sequence of images to improve the search performance ; 2 ) we propose community selection to predict the accuracy of the initial search and to provide correct starting points for hypergraph diffusion ; 3 ) we achieve state-of-the-art performance on the ROxford dataset with and without R1M distractors . 2 Background . 2.1 Query expansion and diffusion : relation based search . Query expansion ( QE ) [ 9 ] aggregates the top-ranked initial candidates into an expanded query , which is then used to search more images in the database . The critical factor here is how to aggregate only helpful information and cull out unrelated information . Average query expansion ( AQE ) [ 9 ] mean-aggregates the top k retrieved images as the expanded query . Average query expansion with decay ( AQEwD ) [ 13 ] gives the top k images the monotonically decaying weights over their rank . Alpha query expansion ( αQE ) [ 27 ] uses the power-normalized similarity between the query and the top-ranked images as the aggregation weights . Discriminative query expansion ( DQE ) [ 3 ] also uses the weighted average , where the weight is the dual-form solution of an SVM , which classifies the positive ( top-ranked ) and negative ( low-ranked ) images of a query . Learnable attention-based query expansion ( LAttQE ) [ 14 ] uses self-attention [ 34 ] to share information between the query and the top-ranked items to compute better aggregation weights . Unfortunately , these QE methods only explore the neighborhood of very similar images of the query . The recursive methods [ 9 ] improve the results by recursively generating new queries from all spatially verified results returned so far . However , this is at the cost of increased query time . Diffusion [ 39 , 16 ] ameliorates the problem mentioned above by propagating similarities through a pairwise affinity matrix . The works of Chang et al . [ 6 ] and Liu et al . [ 17 ] are two novel variants of this line . The critical issue in diffusion is how to calculate the affinity matrix . Some works [ 24 , 16 ] use the reciprocal neighborhood relations to refine the search results , and other [ 23 , 6 ] use the inliers number of SP to re-weight the similarity . However , all of these methods compress the relations between two images as a single scala . As a result , these prior methods lost the spatial matching detail . Although the propagation methods are elegant , they have difficulties recovering good relations to guide the diffusion direction . We propose a novel and straightforward hypergraph model to utilize the relations among database images better . It connects the corresponding local features among images using hyperedges ; in a sequence of similar images , it detects the image from which the following images are no longer relevant to the query . 2.2 Local descriptors and spatial verification . Local descriptors are the representations of an image ’ s important patches . Some popular local descriptors are RootSIFT [ 3 ] , deep local feature ( DELF ) [ 21 ] and deep local and global feature ( DELG ) [ 5 ] . Local descriptors ’ location information is usually used in the spatial verification ( SP ) [ 22 ] stage . A standard SP pipeline is first using a KD-tree [ 22 ] to find the nearest neighbors between local descriptors in two images . Then it uses the RANSAC algorithm [ 12 ] to estimate the homography matrix [ 32 ] and the number of inlier correspondences between two images . Existing image search engines commonly apply SP on the shortlist , say the top 100 images ranked by a global descriptor . SP is known to be crucial for verifying true positive images [ 25 ] . Performing SP before diffusion improves the final retrieval performance [ 25 , 6 ] . However , it is also the slowest process [ 3 , 23 ] in image search . In this paper , we propose community selection to reduce the computing overhead of SP . Community selection frames the diffusion initialization task as selecting a community for the given query rather than verifying the top-ranked images one by one . Experiment shows that community selection reduces much SP overhead without hurting the accuracy . 3 Overview . We first discuss issues of query expansions and diffusion process of ordinary graphs in Sec.3.1 and our novel hypergraph based propagation ( Sec . 3.2 ) and initialization scheme utilizing the concept of community ( Sec . 3.3 ) for achieving accurate diffusion process . 3.1 Propagating in ordinary graph . We can use a graph G = ( X , F ) to represent the database images . X : = { x1 , ... , xm } is the node set , where item xi could be either an image or an image region depending on a chosen retrieval method ; F is the set of edges whose scalar weight represents the similarity between two nodes connected by each edge . Utilizing information in this ordinary graph is an important strategy to improve retrieval results , and , thus , widely used in various methods , such as query expansion [ 9 , 8 , 27 , 14 ] and diffusion [ 16 , 39 ] . Both diffusion and query expansion can be represented by the following function : f∗ = I ( f0 ) , ( 1 ) where I represents the post-processing process , fu ∈ R is the ranking score for xu , f0 is the initial ranking vector with f0u = 1 if xu is a query and f0u = 0 otherwise , and f ∗ is the ranking vector after query expansion or diffusion . Diffusion methods diffuse the label of the query node to its neighbors along the graph edges until reaching a stationary state . The stationary state is found by repeating the iteration function : ft+1 = αOft + ( 1−α ) f0 , where O is the normalized affinity matrix that describes the weights of graph edges , α is a jumping probability . Query expansion gathers useful information in the query node ’ s nearest neighbors to update the ranking list . Diffusion and query expansion have different advantages and disadvantages . Diffusion can efficiently propagate information to the whole graph through a compact formulation . However , the propagation is conducted through the affinity matrix O [ 39 , 16 ] , whose scalar value can not represent accurate spatial information , resulting in low precision . Take Figure 1 as an example , where images of x1 and x5 are incorrectly connected through image x2 in the ordinary graph with scalar edge weights ; note that they do not share common regions . This leads to false positives in the diffusion process . We denote this as the ambiguity problem of propagation . In contrast , query expansion uses sophisticated approaches , such as transformer [ 14 ] and spatial verification [ 9 , 3 ] , to guarantee that only shared information between query and its nearest neighbors are used to update the ranking list . This approach , however , only explores the neighborhood of very similar images due to the computational overhead [ 25 , 3 , 9 ] , resulting in a low recall . This paper proposes novel hypergraph propagation and community selection , settling the ambiguity problem separately in the propagation and initialization stages . We describe these two techniques next . 3.2 Extension to hypergraph . One meaningful direction to overcome the drawbacks of diffusion and query expansion is to handle the ambiguity problem of graph propagation by explicitly resolving the propagation in a more informative space . The local features contain informative spatial information that can help to distinguish the ambiguity through geometric verification [ 22 ] . However , considering the large scale of the image database , it is impractical to do the geometric verification on all the pairs . In practice , image search engines usually build the kNN graph instead of the complete graph , which means every image only connects with its top k similar images [ 16 ] . To utilize the spatial information on the kNN graph , we propose a novel hypergraph propagation model to efficiently propagate spatial information through hyperedges , connecting an arbitrary number of nodes . This technique is appropriate for the real-world situation where the same object in different images has a different number of local features . In the example of Figure 1 , image x1 can connect image x3 by matching the local features with those of image x2 , but will not connect image x5 . When a query is given , we define a hypergraph H = ( Y , E ) on top of the ordinary image graph , where its node set Y = { y1 , ... , yn } contains all the local features extracted , and the edge set E consists of hyperedges , which are used to connect the local features related to the query object in different images . Our model then propagates the label of local features of the query to their neighbors and uses an aggregation function to find the final ranking list of images , as follows : l∗ = Ih ( l0 ) , f∗ = Agg ( l∗ ) , ( 2 ) where l0 and l∗ are the initial and final ranking lists of local features respectively , f∗ is the ranking list of images , Ih is the propagation function in a hypergraph , and Agg is the aggregation function that converts ranking scores of local features to ranking scores of images . We will discuss the construction of hypergraph in Section 4.1 and the propagation process in Section 4.2 .

Summary: This work tackles the problem of image retrieval, specifically in the domain of re-ranking with ambiguous objects. It proposes two core ideas to improve upon existing query expansion and diffusion techniques, namely  1) Introduction of the hypergraph which unifies both global and local relations in the graph propagation process, hence resolving potential ambiguities. 2) Community selection, which resolves the ambiguities in initializing the graphs.  The proposed method shows strong performance on the standard ROxf-RPar (and the 1M distractor set) datasets over existing image retrieval methods with the re-ranking process. Qualitative results are also provided to help understand the way it increases the precision of top-ranked results.