Learning to Anneal and Prune Proximity Graphs for Similarity Search

This paper studies similarity search , which is a crucial enabler of many feature vector–based applications . The problem of similarity search has been extensively studied in the machine learning community . Recent advances of proximity graphs have achieved outstanding performance through exploiting the navigability of the underlying graph structure . In this work , we introduce the annealable proximity graph ( APG ) method to learn and reshape proximity graphs for efficiency and effective similarity search . APG makes proximity graph edges annealable , which can be effectively trained with a stochastic optimization algorithm . APG identifies important edges that best preserve graph navigability and prune inferior edges without drastically changing graph properties . Experimental results show that APG achieves state-of-the-art results not only by producing proximity graphs with less number of edges but also speeding up the search time by 20–40 % across different datasets with almost no loss of accuracy . 1 INTRODUCTION . Similarity search ( nearest neighbor search ) is an integral and indispensable task in many machine learning applications , such as non-parametric classification/regression , computer vision , information retrieval , and language modeling . Recently , it has been demonstrated that it is possible to build a vector search engine to support semantic search ( Chen et al. , 2018 ; Sullivan , 2018 ; Wang et al. , 2018a ; Johnson et al. , 2017 ) , which leverages high-quality neural ranking models ( Nogueira & Cho , 2019 ; Xiong et al. , 2017 ; Zamani et al. , 2018a ) to encode both natural language query and documents into dense continuous feature vectors and performs similarity search to retrieve relevant documents with vast data volumes ( e.g. , based on Euclidean distance ) . This approach has demonstrated significant relevance gains in a wide range of applications and outperforms existing term matching baselines , such as web search ( Huang et al. , 2013 ; Zamani et al. , 2018b ) , question and answering ( Yu et al. , 2014 ) , ad-hoc retrieval ( Mitra et al. , 2017 ; Dehghani et al. , 2017 ; Guo et al. , 2016 ) , mobile search ( Aliannejadi et al. , 2018 ) , and product search ( Van Gysel et al. , 2016 ) . The efficiency and effectiveness of the similarity search approaches have become a problem of great interest , due to the widespread commercial value and the exciting prospect . Recent advance of proximity graphs has demonstrated great potential for fast and accurate nearest neighbor retrieval ( Malkov & Yashunin , 2016 ; Fu et al. , 2019 ) , and the empirical performance of proximity graphs outperforms existing tree–based ( Bentley , 1975 ; Beckmann et al. , 1990 ; Yianilos , 1993 ; Muja & Lowe , 2014 ) , locality sensitive hashing–based ( Gionis et al. , 1999 ) , and product quantization– based methods ( Jegou et al. , 2011 ; Ge et al. , 2013 ; Norouzi & Fleet , 2013 ; Lempitsky , 2012 ; Kalantidis & Avrithis , 2014 ) by a large margin . Proximity graphs exploit the navigability of graph structures , which the search process relies on to converge and achieve good efficiency . In practice , that often results in dense connectivity and large memory consumption because they need to have sufficient edges to maintain specific graph properties , which is a major limitation of this class of approaches . We wish to improve the efficiency of similarity search . In this paper , we address the following research question : can we learn to prune edges of a proximity graph while still being accurate to find nearest neighbors ? Specifically , the pruned proximity graph should be more efficient than the state-of-the-art proximity graphs with comparable accuracy . Before providing a definite answer to the question , we briefly review the findings in percolation theory that motivates our research . Percolation describes the phase transition of a physical system when one or more of its properties change abruptly after a slight change in controlling variables ( e.g. , temperature , pressure , or others ) ( Broadbent & Hammersley , 1957 ) . Prototypical percolation processes include water turning into ice or steam , and the spontaneous emergence of magnetization and superconductivity in metals . Percolation theory mathematically models these physical systems as complex networks and phase transition as a dramatic change of the properties of network connections . We believe that if we can model edge importance as the robustness of proximity graphs to the removal of the edges between vertices , we can produce a proximity graph with less number of edges without dramatically changing the navigability of the graph . We present Annealable Proximity Graph ( APG ) , for simplifying proximity graphs . In particular , we make the following contributions : • We introduce the annealable proximity graph and summarize its key characteristics . • To learn edge importance , we present a percolation inspired method for identifying impor- tant edges and introduce a domain-specific loss derived from search distance errors . • Our formulation makes it possible to leverage a stochastic optimization algorithm to opti- mize the objective and prune edges with low importance . • We prove the convergence of our optimization process and a theoretical guarantee of the search quality of the pruned graph . This approach is unique compared with previous proximity graph algorithms , where most of them only exploit the structure of the underlying index instead of learning from query distribution to reshape proximity graphs . We provide a detailed empirical analysis of our approach . Experimental results show that our approach reduces the number of edges of state-of-the-art proximity graphs significantly by 50 % while also speeds up the search time by 20–40 % across different datasets with almost no loss of accuracy . 2 RELATED WORK . In this section , we review the main ideas from the existing work that is relevant to our approach . Approximate nearest neighbor search ( ANN ) . The problem of similarity search has been extensively studied in the literature of ANN algorithms , which trade the guarantee of exactness against high-efficiency improvement . Some representative methods include tree structure–based ( Bentley , 1975 ; Beckmann et al. , 1990 ; Yianilos , 1993 ; Muja & Lowe , 2014 ) , locality sensitive hashing ( LSH ) –based ( Gionis et al. , 1999 ) , product quantization ( PQ ) –based ( Jegou et al. , 2011 ; Ge et al. , 2013 ; Norouzi & Fleet , 2013 ; Lempitsky , 2012 ; Kalantidis & Avrithis , 2014 ) , and nearest neighbor graph–based ( Hajebi et al. , 2011 ; Fu & Cai , 2016 ) approaches . Although some of these methods , such as LSH , have strong theoretical performance guarantee even in the worst case ( Indyk & Motwani , 1998 ) , recent advances of the proximity graphs have demonstrated logarithmic search complexity and outperformed prior approaches by a large margin ( Malkov & Yashunin , 2016 ; Douze et al. , 2018 ; Fu et al. , 2019 ; Li et al. , 2019 ) . Proximity graphs . A proximity graph exploits the closeness relationship among feature vectors to support similarity search . In particular , let V = { vi ∈ RD|i = 1 , ... , N } be a database of vectors , a proximity graph G ( V , E ) is a directed graph , where each vertex corresponds to one of the vectors v and the whole graph achieves great local connectivity ( as in a lattice graph ) combined with a small graph diameter ( as in a random graph ) ( Malkov et al. , 2014 ; Malkov & Yashunin , 2016 ; Fu et al. , 2019 ) . Such a graph exhibits strong navigability and enables quick search with an N -greedy bestfirst search algorithm . During the search , a candidate queue of size L is used to determine the tradeoff between the searching time and accuracy . Recent studies look into optimizing proximity graphs with product quantization ( Douze et al. , 2018 ; Baranchuk et al. , 2018 ) . However , these approaches often suffer from a considerable amount of recall loss on large datasets because quantization errors tend to be large on dense continuous feature vectors ( e.g. , generated by neural networks ) . Learning to prune . Pruning is a common method to derive sparse neural networks and reduce their heavy inference cost ( Han et al. , 2015a ; b ) . These methods annihilate the non-important weights through the introduction of an L0 or L1 regularizer to the loss function . Many of them remove weights and keep important weights to best preserve the accuracy . Neural network pruning can also be viewed as an architecture search technique , where the network is viewed as a computational graph where the vertices denote the computation nodes and the edges represent the flow of tensors ( Wang et al. , 2018b ; Liu et al. , 2019 ) . To the best of our knowledge , learning to prune has not yet been applied to the task of proximity graphs for similarity search . However , it appears to be a natural fit for restructuring the proximity graphs to improve similarity search . 3 CHALLENGES . The navigability of proximity graphs comes as a result of approximating monotonicity graphs , e.g. , Delaunay graphs ( Lee & Schachter , 1980 ) . According to the graph monotonicity theory ( Dearholt et al. , 1988 ) , a monotonicity graph has a strong guarantee to find the exact nearest neighbor by following a monotonic path with 1-greedy search ( Fu et al. , 2019 ) . However , monotonicity graphs in high dimensional space quickly become almost fully connected , and search in fully connected graphs would be infeasible , due to the out-degree explosion problem ( Hajebi et al. , 2011 ) . To address the problem , proximity graphs limit each node to connect to only a number of R neighbors , aiming to minimize the loss of graph monotonicity while still letting greedy search be effective . However , several challenges remain . The correct choice of R is not so obvious . R can not be too small , because then the graph tends to lose too much monotonicity and search can frequently get stuck at non-global local minima , hurting accuracy . A sufficiently large R is often required to reach high accuracy , but it also increases the number of edges significantly and decreases efficiency : ( 1 ) It ubiquitously raises the connections of both ” hubs ” ( i.e. , nodes in dense areas ) and nodes in sparse areas ; and ( 2 ) it makes the graph more densely connected with many vertices sharing a lot of common neighbors , increasing unnecessary distance computations . Ideally , each vertex should have a different R that best preserves graph monotonicity . The problem is beyond selecting a good value for R and seems to require more fundamental changes to existing proximity graphs . 4 METHODS . In this section , we propose the annealable proximity graph ( APG ) , which supports the exploration of edge heterogeneity in proximity graphs and learning to prune such graphs with an eye towards getting to the nearest neighbor as quickly as possible with minimal loss of accuracy . 4.1 OVERVIEW . APG starts with augmenting a pre-built proximity graph with a learnable weight associated with each edge , representing its importance to preserve graph monotonicity , and a keep probability ( §§ 4.2 ) . The weight is updated in an iterative manner according to a learning rule . APG employs a multi-step learning process , as illustrated in Fig . 1 . At initialization of APG , all edges have equal weights . Since no edge heterogeneity has been learned so far , applying pruning at this stage could lead to premature decisions . Therefore , we start with performing a “ warm-up ” iteration over the edge weights only , without optimizations , i.e. , as the grace-period . Once this warm-up ends , we introduce a systematic and principled approach to optimize the edge keep probability distribution of the APG ( §§ 4.4 ) . In particular , APG models edge importance as the robustness of graph monotonicity to edge removal and defines an objective function that reflects the destruction of graph monotonicity , based on relative search distance errors ( §§ 4.3 ) . It then generates a sequence of randomized subgraphs through a sampling policy to learn edge importance and uses a predefined annealing schedule to optimize the objective function . The process ends once we meet a stopping criterion . After this step , APG marks low weight edges as less important and perform a hard pruning to remove inferior edges , as shown in Fig . 2 . Hops Checks Entry Query Ground Truth APG Figure 2 : Example of before and after pruning proximity graph .

Summary: This paper studies the problem of improving proximity graph for nearest neighbor search. It formulates the task of pruning the graph as a problem of learning annealable proximity graph. A hard pruning processes is used after the learning process, and the results shows that the proposed method can reduce 50% of the edges and speed up the search time by 16-41%.