Graph-based Nearest Neighbor Search in Hyperbolic Spaces

1 INTRODUCTION . Given a dataset D of n elements and an arbitrary query q , the goal of the nearest neighbor search ( NNS ) is to quickly retrieve one or several elements fromD closest to q. NNS is extensively used in a wide range of machine learning systems : for non-parametric classification and regression , language modeling , information retrieval , recommendations , and many others ( Bishop , 2006 ; Chen et al. , 2018 ; May & Ozerov , 2015 ; Shakhnarovich et al. , 2006 ) . In large-scale applications , searching for the nearest neighbors can be a bottleneck , so this operation ’ s efficiency is of the essence . Many efficient NNS algorithms exist for Euclidean and , generally , ` p spaces . Among them are methods based on recursive partitions of the space , e.g. , k-d trees and random projection trees ( Bentley , 1975 ; Dasgupta & Freund , 2008 ; Dasgupta & Sinha , 2015 ; Keivani & Sinha , 2018 ) , well studied Locality Sensitive Hashing ( LSH ) ( Indyk & Motwani , 1998 ) , and methods based on similarity graphs ( Fu et al. , 2019 ; Iwasaki & Miyazaki , 2018 ; Malkov et al. , 2014 ; Malkov & Yashunin , 2018 ) . However , hyperbolic geometry is often found to be more suitable for data representation in various domains , including natural language processing , computer vision , graph analysis , and recommendations ( Chamberlain et al. , 2017 ; Chami et al. , 2020 ; Khrulkov et al. , 2020 ; Mirvakhabova et al. , 2020 ; Nickel & Kiela , 2017 ; 2018 ; Tifrea et al. , 2018 ) . Intuitively , the hyperbolic space has advantages over standard representations when data has a latent hierarchical structure which is often the case . Hence , a natural question is : how to perform efficient NNS in hyperbolic spaces ? This problem has started to gain attention recently and was addressed in Wu & Charikar ( 2020 ) . In a nutshell , Wu & Charikar ( 2020 ) show how any given Euclidean NNS algorithm can be applied to the hyperbolic space . To get convergence guarantees , one has to call the chosen NNS method several times , which leads to the increased NNS complexity compared to the original Euclidean algorithm . In the current paper , we choose an alternative approach and apply a leading NNS technique directly to the hyperbolic space . We demonstrate that under some assumptions , the proposed method does not have increased complexity and may even reduce the query time compared to its Euclidean counterpart . We focus on graph-based NNS methods that are based on a best-first routing over similarity graphs . Namely , at the pre-processing step , a similarity graph is constructed on the elements of the dataset . Then , given a query q , we start from some element ofD and measure distances between its neighbors and q . The neighbors are added to a priority queue . Then , we pick the closest to q candidate from this queue and continue the process ( Fu et al. , 2019 ; Iwasaki & Miyazaki , 2018 ; Malkov et al. , 2014 ; Malkov & Yashunin , 2018 ) . It was found empirically that graph-based NNS outperforms other approaches in many large-scale applications ( Aumüller et al. , 2019 ) . In contrast to other NNS methods , graph-based approaches can be easily applied to non-Euclidean spaces . Indeed , for most algorithms , both graph construction and graph routing can be performed without appealing to the geometry of the underlying space — only the relative ordering of neighbors is used . Thus , one can replace Euclidean distance with the hyperbolic one in a favorite graph-based approach . However , it is a priori not known how graph-based approaches would work in such a non-standard scenario . In this paper , we show that graph-based approaches are well suited for NNS in hyperbolic spaces . From the theoretical perspective , we analyze the time and space complexity of graph-based NNS , assuming that the data elements are distributed uniformly within a ball of radius R in the hyperbolic space of curvature −1 . In this regard , we compare the obtained results with Prokhorenkova & Shekhovtsov ( 2020 ) who studied graph-based algorithms assuming the uniform distribution over a d-dimensional sphere . In the dense setting ( assuming that d log n and R does not grow too fast ) , we derive the time and space complexity using the fact that locally hyperbolic graph-based NNS behaves similarly to the Euclidean case . However , globally , graph-based search in hyperbolic spaces requires fewer search steps . Thus , hyperbolic NNS can be even more efficient compared to its Euclidean counterpart . In the sparse setting ( d log n ) , we show that hyperbolic NNS has similar complexity to the Euclidean case . From a practical perspective , we illustrate our theoretical findings via synthetic experiments on uniform data and also analyze graph-based hyperbolic NNS in practical applications where our theoretical results can not be directly applied . For real datasets , we consider Poincaré GloVe word embeddings ( Tifrea et al. , 2018 ) and hyperbolic representations of queries and items for a recommendation task ( Shevkunov & Prokhorenkova , 2021 ) . We compare graph-based NNS with existing baselines ( Wu & Charikar , 2020 ) and show that graph-based approaches have significantly better performance . We also compare the performance of graph-based NNS for Euclidean and hyperbolic representations of the same sets of elements and observe that hyperbolic NNS can be more efficient . 2 PRELIMINARIES . Graph-based nearest neighbor search Graph-based algorithms for NNS are known to outperform other approaches in many large-scale applications ( Aumüller et al. , 2019 ) . The main idea is to construct a proximity graph at the pre-processing stage and traverse this graph towards the query q at query time . The best-first search is usually used for traversal : we maintain a priority queue , and at each step , we pick the closest to q candidate from this queue , measure distances between its neighbors and q , and add the neighbors to the queue . Many algorithms are based on this idea ( Baranchuk et al. , 2019 ; Fu et al. , 2019 ; Iwasaki & Miyazaki , 2018 ; Malkov & Yashunin , 2018 ) . Unfortunately , despite their success in practice , graph-based approaches do not have solid theoretical guarantees . This problem was first approached by Laarhoven ( 2018 ) who provided time–space trade-offs for approximate nearest neighbor ( ANN ) search in sparse datasets ( d log n ) uniformly distributed on a d-dimensional Euclidean sphere . This work considered nearest neighbor graphs , where each point is connected to its neighborhood lying within a ball of a fixed radius , and assumed greedy graph traversal . Prokhorenkova & Shekhovtsov ( 2020 ) extended this result : for the uniform distribution , the authors consider both sparse ( d log n ) and dense ( d log n ) regimes , and they analyze the effect of two improvements commonly used in practice : replacing greedy search by best-first ( or beam ) search and adding long edges to speed up graph traversal at early stages . In this paper , we aim at understanding graph-based NNS in hyperbolic spaces . For this , we adopt a similar setup to Prokhorenkova & Shekhovtsov ( 2020 ) , but instead of a uniform distribution over a sphere , we assume a uniform distribution within a ball of a certain radius in the hyperbolic space . The obtained time and space complexity heavily relies on this radius : for small values , the obtained results are similar to the spherical case , while for larger values , the exponential growth of hyperbolic volumes allows us to obtain even faster convergence in the dense regime . Hyperbolic geometry In this section , we discuss the basics of hyperbolic geometry . Let Rd,1 be the ( d + 1 ) -dimensional Minkowski space , i.e. , the real vector space Rd+1 equipped with the inner product of signature ( d , 1 ) : 〈x , y〉h = −x0y0 + x1y1 + . . .+ xdyd . ( 1 ) The vector model of the d-dimensional hyperbolic Lobachevsky space Hd is the connected component of the standard two-sheeted hyperboloid Hd = { x ∈ Rd,1 | 〈x , x〉h = −1 } contained in the future light cone : Hd = { x ∈ Rd,1 | 〈x , x〉h = −1 , x0 > 0 } . The hyperbolic metric is given by cosh ρ ( x , y ) = −〈x , y〉h . From the definition of the hyperbolic space , we see that x0 = √ 1 + x21 + . . .+ x 2 d. The volume element in Hd is ( see Ratcliffe ( 2019 ) ) dx1 . . . dxd√ 1 + x21 + . . .+ x 2 d . Hyperbolic space is known to be useful for data representation in various domains . It is especially convenient for embedding hierarchical data ( Nickel & Kiela , 2017 ) . The reason is that the volume of a ball in the hyperbolic space grows exponentially with its radius , in contrast to the Euclidean space , where the growth is polynomial . Formally , let B ( x , r ) be a closed hyperbolic ball centered at a point x ∈ Hd of radius r > 0 . Then , VolHd ( B ( x , r ) ) = Vol ( Sd−1 ) ∫ r 0 sinhd−1 ( t ) dt , ( 2 ) where Sd−1 is the unit ( d−1 ) -sphere and Vol ( M ) denotes the full volume of a Riemannian manifold M. That is , Vol ( Sd−1 ) = 2πd/2/Γ ( d/2 ) . For r log d , we have VolHd ( B ( x , r ) ) ∼ Vol ( Sd−1 ) er ( d−1 ) Θ ( d−12−d ) . ( 3 ) The main term here is er ( d−1 ) , so the ball volume VolHd ( B ( x , r ) ) grows exponentially when r is large . However , for small hyperbolic balls ( as r → 0 ) , we have sinh ( r ) ∼ r. Hence , the volume of a ball ( 2 ) behaves similarly to the Euclidean case . In Section 4.1 , we provide a formal general result that shows that not only the above argument is correct , but also the volume of a measurable subset of a hyperbolic ball behaves asymptotically exactly like its Euclidean analog . Nearest neighbor search in hyperbolic space While there are many NNS algorithms for ` p spaces , little is known about efficient hyperbolic analogs . Krauthgamer & Lee ( 2006 ) proved the existence of an approximate near neighbor algorithm for general geodesic spaces of hyperbolicity δ . From a theoretical perspective , the bound is given in terms of the doubling dimension . For uniform datasets , we may conclude that the obtained time complexity is 2O ( d ) log2 n , while the constant in the exponent is not given . In contrast , we specify this term as we focus on d → ∞ . Another difference is that the algorithm of Krauthgamer & Lee ( 2006 ) requires O ( n2 ) storage that is usually infeasible in practice . In contrast , graph-based algorithms require nMd for some constant M if d log n. Finally , while the theoretical analysis is given in a very general setup , there is no implementation of the proposed algorithm , and extracting an efficient implementation from the proof is challenging . In a recent paper , Wu & Charikar ( 2020 ) reduce the problem of hyperbolic NNS to Euclidean NNS . Namely , assume that we have an efficient Euclidean algorithm and a dataset is represented in the Poincaré ball model . Then , we can apply the algorithm to the Euclidean representation . To guarantee convergence , the algorithm should be applied several times . There are several alternatives proposed . In one of them , we apply the algorithm to the whole dataset , but after each run , appropriately changes the query point to get better hyperbolic neighbors at subsequent iterations . The time complexity of this algorithm is k + 1 times larger than the Euclidean search if the Euclidean nearest neighbor for the query is the k-th nearest hyperbolic neighbor . In another algorithm , one splits the ball into thin annuli and performs the approximate Euclidean NNS for each of them separately . From a theoretical perspective , it is hard to compare our results since Wu & Charikar ( 2020 ) do not consider any particular algorithm . We can not apply this general result to graph-based algorithms since known guarantees for the Euclidean space assume a uniform distribution . However , if we consider the uniform distribution in the Poincaré model of the hyperbolic space , the corresponding Euclidean distribution is far from being uniform , i.e. , the guarantees can not be transferred . From a practical perspective , we compare graph-based algorithms with the method of Wu & Charikar ( 2020 ) and show that graph-based NNS is more efficient .

Summary: This paper looks into graph-based algorithms for nearest neighbor search (NNS) problem in the hyperbolic space.  Theoretically, the authors derived the time and space complexity of graph-based NNS under some assumptions (data uniformed distributed in the ball, low dimensional dense), and in some cases, graph-based NNS has lower time complexity in the hyperbolic space.   Practically, the authors apply the methodology to Poincar{/'e} Glove word embeddings, where it outperforms some existing baselines.