Fast Binary Functional Search on Graph

1 INTRODUCTION . Approximate Nearest Neighbor Search ( ANNS ) , or Similarity Search , has been a classical data science task for decades . In the ANNS paradigm , each given query is compared with a subset instead of the whole dataset , which reduces time complexity significantly and show powerful performance in Search Engines and Recommender Systems . To obtain the subset with a high recall rate , specific index structures are proposed for dense continuous vectors ( Charikar , 2002 ; Jegou et al. , 2008 ) or high dimensional sparse data ( Broder , 1997 ; Li et al. , 2012 ) . For the dense case , which is our focus in this paper , hashing based algorithms ( Gionis et al. , 1999 ; Charikar , 2002 ; Weiss et al. , 2009 ) , quantization-based methods ( Jegou et al. , 2008 ; 2011 ; Ge et al. , 2013 ; Wu et al. , 2017 ) and graph based approaches ( Hajebi et al. , 2011 ; Malkov et al. , 2014 ; Wu et al. , 2014 ; Malkov & Yashunin , 2016 ; Fu et al. , 2017 ) are applied . Particularly , graph based methods show promise in improving ANNS performance ( Aumüller et al. , 2017 ) . Besides , graph based index structures have the flexibility that user can define any matching function as similarity measure in graph construction and searching . However , most existing graph based ANNS methods concentrate on searching by metric distances , such as ` 2 distance or angular distance . Efficient searching beyond metric distances is considered hard for ANNS . Only a few simple non-metric measures were studied in literature , such as Maximum Inner Product Search ( MIPS ) ( Shrivastava & Li , 2014 ) , Max-Kernel Search ( Curtin et al. , 2013 ) and searching by Bregman divergence ( Cayton , 2008 ) . For more complicated matching measures , efficient searching is still an open question . Development of machine learning techniques aggravates this problem by increasing matching measures ’ complexity . Recently , learning based ranking/matching models were continuously proposed to capture semantic level relevance . Among various learning paradigms , Matching Function Learning becomes popular these years ( He et al. , 2017 ; Xiong et al. , 2017 ; Tay et al. , 2018 ; Dai et al. , 2018 ; Xu et al. , 2018 ) . It assumes a mapping exists between query and document ( or user and item ) . Therefore , the goal of Matching Function Learning is to learn the mapping by tunable models , such as Neural Networks . Matching Function Learning is widely adopted in academic works for its significant performance . However , as the learned matching functions are usually too complicated than metric distances , efficient searching and on-line model prediction remains scarce . Thus , the application of Matching Function Learning approaches in real searching or recommendation systems is still constraining . In this paper , we introduce a general task by formulating a Optimal Binary Functional Search ( OBFS ) problem . ANNS and MIPS problems can be considered as its special cases . Under the definition of OBFS , performing Search on Graph methods can be summarized as a common methodology , Search on Binary Functional Graph ( SBFG ) : constructing the ( approximate ) Delaunay graph with respect to binary functional f for all indexing data and then greedy searching most relevant vertices on the graph for each query q . We validate SBFG ’ s outperforming efficiency theoretically on metric distances . However , constructing or approximately constructing Delaunay graphs for complicated binary functionals is extremely hard . It is difficult to extend SBFG to more general OBFS problems , such as OBFS with Neural Network as binary functionals . To overcome the limitations of SBFG , we propose a new graph based searching methodology , Search on L2 Graph ( SL2G ) . Different from SBFG , SL2G constructs the index graph by ` 2 distance among searching data , no matter which binary functional f we work on . In the searching phase , SL2G works similarly with SBFG : greedy search by the focus binary functional f . Paradigms of SL2G breaks the symmetric limitation of SBFG . As analyzed in the theoretical part , SL2G approximates the gradient descent in Euclidean space . Even if f is non-convex , SL2G can reach an approximate local optimal , based on some affordable assumptions . In experiments , we test the proposed methodology , SL2G , by two Neural Network based binary functions . As will be seen , as SL2G is designed for general OBFS problem , it works consistently well on different matching measures ( i.e. , binary functionals ) . But the traditional methodology , SBFG , works badly on complicated matching measures , especially for asymmetric measures . In summary , the contributions of this paper are as below : • Extend ANNS to a more general task , Optimal Binary Functional Search ( OBFS ) . Under OBFS , more advanced matching measures can be considered in large-scale search . • Theoretically explain why existing Search on Graph methods work well on metric similarity measures ( e.g. , ` 2 distance ) and specify their limitations when extending to complicated searching measures . • Propose a solution , SL2G , to tackle the OBFS problem . SL2G breaks the limitations of existing Search on Graph methods for OBFS and is applicable for any complex searching measures ( e.g. , Neural Network models ) . Theoretical analysis for SL2G is provided . 2 BACKGROUND OF SEARCH ON GRAPHS . Graph structures provide a natural way to partition a high dimensional continuous space into discrete regions . Theoretically , formulation of Voronoi Diagram and its dual graph , Delaunay Graph ( Fortune , 1995 ; Aurenhammer , 1991 ) provide the foundation for Nearest Neighbor Search ( NNS ) on graph . If the space is partitioned via Voronoi Diagram , nearest neighbors of given queries can be returned by Delaunay Graph ( or Delaunay triangulation ) with exactness . Generally , in order to preserve the preciseness in NNS , the network must contain the Delaunay Graph as its subgraph . However , as Delaunay Graphs quickly become fully connected at high dimensionality , efficiency of NNS on Delaunay Graphs reduces dramatically ( Harwood & Drummond , 2016 ) . Moreover , for applications of NNS , finding exact nearest neighbors can be substituted by Approximate NNS . Therefore , support for whole and exact Delaunay Graph is not required in ANNS . Instead , alternative graphs are proposed to approximate their performance with affordable time complexity . k Nearest Neighbor ( kNN ) graph is claimed to be an approximation of Delaunay Graph ( Fu et al. , 2017 ; Hajebi et al. , 2011 ) . For each vertex , kNN graph connects vertices within top k-th smallest distances to it . Then greedy search is performed on kNN graph to return nearest neighbors . In experiments , kNN graphs show effectiveness in ANNS on high dimensional real world spaces . Hajebi et al . ( 2011 ) , Fu & Cai ( 2016 ) and Jin et al . ( 2014 ) use hashing and randomized KD-trees to provide better starting positions of greedy search on graph and show promising performance in ANNS . Inspired by the small world phenomenon , Navigable Small World Network ( NSWN ) is introduced ( Kleinberg , 2000 ) and shown their potentials in ANNS . Malkov et al . ( 2014 ) propose a Navigable Small World ( NSW ) graph that approximates both NSWN and Delaunay Graphs simultaneously . NSW consists short-range links and long-range links as its edges . Short-range links are approximations to Delaunay Graphs , which ensure that approximate nearest neighbors will be returned via greedy search . Long-range links are regarded as shortcuts , which are responsible for navigation between small world clusters . The existence of long-range links help logarithmic scaling of greedy search on graphs . Thus NSW demonstrates better performance than kNN graph at high dimensionality . However , NSW ’ s efficiency is constraining as degree of the graph is too high and connectivity problems may exist in graph . To tackle this issue , Malkov and Yashunin ’ s following work ( Malkov & Yashunin , 2016 ) propose Hierarchical NSW ( HNSW ) , an extended version of NSW . HNSW adopts edge selection strategy from Relative Neighborhood Graphs ( RNG ) to reduce degree of the NSW graph . Meanwhile , a hierarchical structure is used in HNSW to perform multi-scale hopping . This “ zoom-out ” and “ zoom-in ” effect offers significantly better performance . Although there are few theoretical literature on proving whether approximately constructed graphs contain Delaunay Graph as subgraph , both kNN and NSW graphs perform Delaunay Graph ’ s proprieties empirically . Actually , graph based ANNS algorithms significantly outperforms hashing based ( Gionis et al. , 1999 ; Lv et al. , 2007 ) and quantization based ANNS methods ( Jegou et al. , 2008 ; 2011 ) in well-known benchmarks ( Aumüller et al. , 2017 ) . 3 BINARY FUNCTIONAL SEARCH ON GRAPH . In this section , we extend ANNS problem to a broader setting . Before introducing the proposed method for the new defined problem , we will first analyze theoretical foundations of existing Search on Graph methods and their limitations . 3.1 DEFINITIONS AND NOTATIONS . Although Search on Graph methods claimed that there are no constraints on measures defined in graph constructing and searching ( Hajebi et al. , 2011 ; Malkov & Yashunin , 2016 ) , most existing ANNS works mainly focus on searching by some metric distances . It was shown that graph based methods are outperforming on a few non-metric measures in empirical experiments ( Malkov & Yashunin , 2016 ) , but more potential searching measures remain to be discussed . Meanwhile , systemic theoretical analysis has few been conducted on the measure variation . In this paper , we formulate the searching task in a more general way , Optimal Binary Functional Search ( OBFS ) . It will be shown that ANNS and MIPS are special cases of OBFS . Definition 1 . ( OBFS ) Let X and Y be subsets of Euclidean spaces ( possibly with different dimensions ) , given a data set S = { x1 , . . . , xn } ⊂ X and a continuous binary functional , f : X×Y → R , OBFS aims to find arg max xi∈S f ( xi , q ) , for q ∈ Y . ( 1 ) Let us consider X ⊂ Rd and Y ⊂ Rk . Note that we consider subsets of Euclidean space because some functions ’ domain is not the whole space , for example , cosine is undefined at origin . Take the recommendation problem as an example . We consider items as searching data X and the user set as queries Y . Matching function is specified or learned by Binary functional f , which takes a pair of item and user as input . The output of f is the ranking score for recommendation . ` 2 distance , angular distance and inner product are special cases of binary functionals when d = k. Since Search on Graph methods are based on Delaunay graphs , we also introduce related definitions here . Definition 2 . The Voronoi cell Ri with respect to f and xi is the set Ri : = { q ∈ Y : f ( xi , q ) ≥ f ( xj , q ) for j 6= i } . ( 2 ) Definition 3 . The Delaunay graph G with respect to f and S is an undirected graph with vertices S satisfies { xi , xj } ∈ G if and only if Ri ∩Rj 6= ∅ . That is two data points in the Delaunay graph will be connected if the Voronoi cell is adjacent to each other . Here , adjacency means their boundary has nonempty intersection . It is worth noting that , the Voronoi diagram is built on Y , while the Delaunay graph is constructed with vertices S on X . For an undirected graph G , we say { xi , xj } ∈ G if there is an edge between xi and xj . We refer Delaunay graphs w.r.t . any f on some data set S as Binary Functional Graph . Example 1 . Let f : Rd × Rd → R be negative ` 2 distance , i , e. , f ( x , y ) = −‖x − y‖ , then the Delaunay graph defined in Definition 3 coincides with classical definition , for instance , the one in Lee & Schachter ( 1980 ) . Notations used in this paper are summarized in Table 1 .

Summary: This work extends the approximate nearest neighbor search (ANNS) algorithm to a more general setting. Instead of search with a "separable" similarity measure, the authors propose Optimal Binary Functional Search (OBFS), where the scoring function f() is in general non-separable. The exact construction of the Binary Function Graph wrt f() and X is computationally expensive. The specific approximate algorithm of OBFS proposed in the paper is to: