Graph-based Nearest Neighbor Search in Hyperbolic Spaces

1 INTRODUCTION . Given a dataset D of n elements and an arbitrary query q , the goal of the nearest neighbor search ( NNS ) is to quickly retrieve one or several elements fromD closest to q. NNS is extensively used in a wide range of machine learning systems : for non-parametric classification and regression , language modeling , information retrieval , recommendations , and many others ( Bishop , 2006 ; Chen et al. , 2018 ; May & Ozerov , 2015 ; Shakhnarovich et al. , 2006 ) . In large-scale applications , searching for the nearest neighbors can be a bottleneck , so this operation ’ s efficiency is of the essence . Many efficient NNS algorithms exist for Euclidean and , generally , ` p spaces . Among them are methods based on recursive partitions of the space , e.g. , k-d trees and random projection trees ( Bentley , 1975 ; Dasgupta & Freund , 2008 ; Dasgupta & Sinha , 2015 ; Keivani & Sinha , 2018 ) , well studied Locality Sensitive Hashing ( LSH ) ( Indyk & Motwani , 1998 ) , and methods based on similarity graphs ( Fu et al. , 2019 ; Iwasaki & Miyazaki , 2018 ; Malkov et al. , 2014 ; Malkov & Yashunin , 2018 ) . However , hyperbolic geometry is often found to be more suitable for data representation in various domains , including natural language processing , computer vision , graph analysis , and recommendations ( Chamberlain et al. , 2017 ; Chami et al. , 2020 ; Khrulkov et al. , 2020 ; Mirvakhabova et al. , 2020 ; Nickel & Kiela , 2017 ; 2018 ; Tifrea et al. , 2018 ) . Intuitively , the hyperbolic space has advantages over standard representations when data has a latent hierarchical structure which is often the case . Hence , a natural question is : how to perform efficient NNS in hyperbolic spaces ? This problem has started to gain attention recently and was addressed in Wu & Charikar ( 2020 ) . In a nutshell , Wu & Charikar ( 2020 ) show how any given Euclidean NNS algorithm can be applied to the hyperbolic space . To get convergence guarantees , one has to call the chosen NNS method several times , which leads to the increased NNS complexity compared to the original Euclidean algorithm . In the current paper , we choose an alternative approach and apply a leading NNS technique directly to the hyperbolic space . We demonstrate that under some assumptions , the proposed method does not have increased complexity and may even reduce the query time compared to its Euclidean counterpart . We focus on graph-based NNS methods that are based on a best-first routing over similarity graphs . Namely , at the pre-processing step , a similarity graph is constructed on the elements of the dataset . Then , given a query q , we start from some element ofD and measure distances between its neighbors and q . The neighbors are added to a priority queue . Then , we pick the closest to q candidate from this queue and continue the process ( Fu et al. , 2019 ; Iwasaki & Miyazaki , 2018 ; Malkov et al. , 2014 ; Malkov & Yashunin , 2018 ) . It was found empirically that graph-based NNS outperforms other approaches in many large-scale applications ( Aumüller et al. , 2019 ) . In contrast to other NNS methods , graph-based approaches can be easily applied to non-Euclidean spaces . Indeed , for most algorithms , both graph construction and graph routing can be performed without appealing to the geometry of the underlying space — only the relative ordering of neighbors is used . Thus , one can replace Euclidean distance with the hyperbolic one in a favorite graph-based approach . However , it is a priori not known how graph-based approaches would work in such a non-standard scenario . In this paper , we show that graph-based approaches are well suited for NNS in hyperbolic spaces . From the theoretical perspective , we analyze the time and space complexity of graph-based NNS , assuming that the data elements are distributed uniformly within a ball of radius R in the hyperbolic space of curvature −1 . In this regard , we compare the obtained results with Prokhorenkova & Shekhovtsov ( 2020 ) who studied graph-based algorithms assuming the uniform distribution over a d-dimensional sphere . In the dense setting ( assuming that d log n and R does not grow too fast ) , we derive the time and space complexity using the fact that locally hyperbolic graph-based NNS behaves similarly to the Euclidean case . However , globally , graph-based search in hyperbolic spaces requires fewer search steps . Thus , hyperbolic NNS can be even more efficient compared to its Euclidean counterpart . In the sparse setting ( d log n ) , we show that hyperbolic NNS has similar complexity to the Euclidean case . From a practical perspective , we illustrate our theoretical findings via synthetic experiments on uniform data and also analyze graph-based hyperbolic NNS in practical applications where our theoretical results can not be directly applied . For real datasets , we consider Poincaré GloVe word embeddings ( Tifrea et al. , 2018 ) and hyperbolic representations of queries and items for a recommendation task ( Shevkunov & Prokhorenkova , 2021 ) . We compare graph-based NNS with existing baselines ( Wu & Charikar , 2020 ) and show that graph-based approaches have significantly better performance . We also compare the performance of graph-based NNS for Euclidean and hyperbolic representations of the same sets of elements and observe that hyperbolic NNS can be more efficient . 2 PRELIMINARIES . Graph-based nearest neighbor search Graph-based algorithms for NNS are known to outperform other approaches in many large-scale applications ( Aumüller et al. , 2019 ) . The main idea is to construct a proximity graph at the pre-processing stage and traverse this graph towards the query q at query time . The best-first search is usually used for traversal : we maintain a priority queue , and at each step , we pick the closest to q candidate from this queue , measure distances between its neighbors and q , and add the neighbors to the queue . Many algorithms are based on this idea ( Baranchuk et al. , 2019 ; Fu et al. , 2019 ; Iwasaki & Miyazaki , 2018 ; Malkov & Yashunin , 2018 ) . Unfortunately , despite their success in practice , graph-based approaches do not have solid theoretical guarantees . This problem was first approached by Laarhoven ( 2018 ) who provided time–space trade-offs for approximate nearest neighbor ( ANN ) search in sparse datasets ( d log n ) uniformly distributed on a d-dimensional Euclidean sphere . This work considered nearest neighbor graphs , where each point is connected to its neighborhood lying within a ball of a fixed radius , and assumed greedy graph traversal . Prokhorenkova & Shekhovtsov ( 2020 ) extended this result : for the uniform distribution , the authors consider both sparse ( d log n ) and dense ( d log n ) regimes , and they analyze the effect of two improvements commonly used in practice : replacing greedy search by best-first ( or beam ) search and adding long edges to speed up graph traversal at early stages . In this paper , we aim at understanding graph-based NNS in hyperbolic spaces . For this , we adopt a similar setup to Prokhorenkova & Shekhovtsov ( 2020 ) , but instead of a uniform distribution over a sphere , we assume a uniform distribution within a ball of a certain radius in the hyperbolic space . The obtained time and space complexity heavily relies on this radius : for small values , the obtained results are similar to the spherical case , while for larger values , the exponential growth of hyperbolic volumes allows us to obtain even faster convergence in the dense regime . Hyperbolic geometry In this section , we discuss the basics of hyperbolic geometry . Let Rd,1 be the ( d + 1 ) -dimensional Minkowski space , i.e. , the real vector space Rd+1 equipped with the inner product of signature ( d , 1 ) : 〈x , y〉h = −x0y0 + x1y1 + . . .+ xdyd . ( 1 ) The vector model of the d-dimensional hyperbolic Lobachevsky space Hd is the connected component of the standard two-sheeted hyperboloid Hd = { x ∈ Rd,1 | 〈x , x〉h = −1 } contained in the future light cone : Hd = { x ∈ Rd,1 | 〈x , x〉h = −1 , x0 > 0 } . The hyperbolic metric is given by cosh ρ ( x , y ) = −〈x , y〉h . From the definition of the hyperbolic space , we see that x0 = √ 1 + x21 + . . .+ x 2 d. The volume element in Hd is ( see Ratcliffe ( 2019 ) ) dx1 . . . dxd√ 1 + x21 + . . .+ x 2 d . Hyperbolic space is known to be useful for data representation in various domains . It is especially convenient for embedding hierarchical data ( Nickel & Kiela , 2017 ) . The reason is that the volume of a ball in the hyperbolic space grows exponentially with its radius , in contrast to the Euclidean space , where the growth is polynomial . Formally , let B ( x , r ) be a closed hyperbolic ball centered at a point x ∈ Hd of radius r > 0 . Then , VolHd ( B ( x , r ) ) = Vol ( Sd−1 ) ∫ r 0 sinhd−1 ( t ) dt , ( 2 ) where Sd−1 is the unit ( d−1 ) -sphere and Vol ( M ) denotes the full volume of a Riemannian manifold M. That is , Vol ( Sd−1 ) = 2πd/2/Γ ( d/2 ) . For r log d , we have VolHd ( B ( x , r ) ) ∼ Vol ( Sd−1 ) er ( d−1 ) Θ ( d−12−d ) . ( 3 ) The main term here is er ( d−1 ) , so the ball volume VolHd ( B ( x , r ) ) grows exponentially when r is large . However , for small hyperbolic balls ( as r → 0 ) , we have sinh ( r ) ∼ r. Hence , the volume of a ball ( 2 ) behaves similarly to the Euclidean case . In Section 4.1 , we provide a formal general result that shows that not only the above argument is correct , but also the volume of a measurable subset of a hyperbolic ball behaves asymptotically exactly like its Euclidean analog . Nearest neighbor search in hyperbolic space While there are many NNS algorithms for ` p spaces , little is known about efficient hyperbolic analogs . Krauthgamer & Lee ( 2006 ) proved the existence of an approximate near neighbor algorithm for general geodesic spaces of hyperbolicity δ . From a theoretical perspective , the bound is given in terms of the doubling dimension . For uniform datasets , we may conclude that the obtained time complexity is 2O ( d ) log2 n , while the constant in the exponent is not given . In contrast , we specify this term as we focus on d → ∞ . Another difference is that the algorithm of Krauthgamer & Lee ( 2006 ) requires O ( n2 ) storage that is usually infeasible in practice . In contrast , graph-based algorithms require nMd for some constant M if d log n. Finally , while the theoretical analysis is given in a very general setup , there is no implementation of the proposed algorithm , and extracting an efficient implementation from the proof is challenging . In a recent paper , Wu & Charikar ( 2020 ) reduce the problem of hyperbolic NNS to Euclidean NNS . Namely , assume that we have an efficient Euclidean algorithm and a dataset is represented in the Poincaré ball model . Then , we can apply the algorithm to the Euclidean representation . To guarantee convergence , the algorithm should be applied several times . There are several alternatives proposed . In one of them , we apply the algorithm to the whole dataset , but after each run , appropriately changes the query point to get better hyperbolic neighbors at subsequent iterations . The time complexity of this algorithm is k + 1 times larger than the Euclidean search if the Euclidean nearest neighbor for the query is the k-th nearest hyperbolic neighbor . In another algorithm , one splits the ball into thin annuli and performs the approximate Euclidean NNS for each of them separately . From a theoretical perspective , it is hard to compare our results since Wu & Charikar ( 2020 ) do not consider any particular algorithm . We can not apply this general result to graph-based algorithms since known guarantees for the Euclidean space assume a uniform distribution . However , if we consider the uniform distribution in the Poincaré model of the hyperbolic space , the corresponding Euclidean distribution is far from being uniform , i.e. , the guarantees can not be transferred . From a practical perspective , we compare graph-based algorithms with the method of Wu & Charikar ( 2020 ) and show that graph-based NNS is more efficient .

Summary: The paper approaches the problem of nearest neighbor search in hyperbolic spaces with focus on graph methods. The authors claim to obtain an acceleration in hyperbolic spaces over Euclidean ones using graph based NNS method (e.g. Prokhorenkova 2020). The provide bounds on the dimension and Radius of search in the dense  NN domain (ie d<<logn). Experimental validation is presented which demonstrate the advantage of graph methods and also tradeoffs on efficiency in different parameters regimes to validate the theoretical analysis.