Title: Worst-case Performance of Popular Approximate Nearest Neighbor Search Implementations: Guarantees and Limitations
Venue: NeurIPS 2023
Accepted: yes
Keywords: Nearest neighbor search; graph-based algorithms; worst-case analysis
Abstract: Graph-based approaches to nearest neighbor search are popular and powerful tools for handling large datasets in practice, but they have limited theoretical guarantees. 
We study the worst-case performance of recent graph-based approximate nearest neighbor search algorithms, such as HNSW, NSG and DiskANN. For DiskANN, we show that its "slow preprocessing'' version provably supports approximate nearest neighbor search query with constant approximation ratio and poly-logarithmic query time, on data sets with bounded "intrinsic'' dimension. 
For the other data structure variants studied, including DiskANN with "fast preprocessing'',  HNSW and NSG,  we present a family of instances on which the empirical query time required to achieve a "reasonable'' accuracy is linear in instance size. For example, for DiskANN, we show that the query procedure can take at least  $0.1 n$ steps on instances of size $n$ before it encounters any of the $5$ nearest neighbors of the query.

Worst-case Performance of Popular Approximate Nearest Neighbor Search Implementations: Guarantees and Limitations

Abstract

Graph-based approaches to nearest neighbor search are popular and powerful tools for handling large datasets in practice, but they have limited theoretical guarantees. We study the worst-case performance of recent graph-based approximate nearest neighbor search algorithms, such as HNSW, NSG and DiskANN. For DiskANN, we show that its 'slow preprocessing' version provably supports approximate nearest neighbor search query with constant approximation ratio and poly-logarithmic query time, on data sets with bounded 'intrinsic' dimension. For the other data structure variants studied, including DiskANN with 'fast preprocessing', HNSW and NSG, we present a family of instances on which the empirical query time required to achieve a 'reasonable' accuracy is linear in instance size. For example, for DiskANN, we show that the query procedure can take at least 0 . 1 n steps on instances of size n before it encounters any of the 5 nearest neighbors of the query.

1 Introduction

The nearest neighbor search (NN) problem is defined as follows: given a set of n points P in a metric space p X,D q , build a data structure that, given any query point q P X , returns p P P closest to q . More generally, given a parameter k , the data structure should report k points in P that are closest to q . Often, though not always, the metric space is a d -dimensional vector space with the distance function induced by the Euclidean norm. Since its introduction in the influential book by Minsky and Papert in the 1960s [31], the problem has found a tremendous number of applications in machine learning and computer vision [34]. In most of those applications, the underlying metric is induced by a set of points in a high-dimensional space. Since in this setting worst-case efficient nearest neighbor algorithms are unlikely to exist (see e.g., [2]), various approximate formulations of the problem have been studied extensively. One popular formulation allows the algorithm to return any point p 1 P P whose distance to the query q is at most c times the distance between q and its true nearest neighbor in P ; such point p 1 is called a c -approximate nearest neighbor . In practice, the accuracy of an approximate data structure is often evaluated by estimating the recall, i.e., the average fraction of the true k nearest neighbors returned by the data structure.

Over the last few decades, many nearest neighbor data structures have been proposed, both exact and approximate. The most popular classes of data structures are tree algorithms (e.g., kd-trees [4]); locality sensitive hashing [1]; learning-to-hash and product quantization [35, 36] ; and metric data structures. The last class includes algorithms that work for point-sets in arbitrary metrics, not just those in d -dimensional vectors normed spaces. This category can be further subdivided into algorithms based on metric trees [6, 32], divide and conquer algorithms [24, 5, 10], and methods based on greedy search in proximity graphs [3, 28, 15, 21]. See the surveys [8, 27] for further overview of these classes of algorithms.

Several algorithms in the latter class, such as HNSW [28], NSG [15] and DiskANN [21], are widely used in practice. 1 They have been recently shown empirically to provide excellent tradeoffs between accuracy and query speed [27]. Their design is based on approximating theoretical concepts such as Delaunay or Relative Neighborhood Graphs. However, their worst-case performance is not well understood, especially when the dimension is high. For example, the authors of [28] point out that 'Further analytical evidence is required to confirm whether the resilience of Delaunay graph approximations generalizes to higher dimensional spaces.' This stands in contrast to the older algorithms, e.g., those based on the divide and conquer approach, which come with worst-case performance guarantees. Those algorithms are typically analyzed under the assumption that the input point set P has low doubling dimension (a measure of the intrinsic dimensionality of the pointset) 2 and provide near-linear space and logarithmic query time bounds, with the big-Oh constants depending on the doubling dimension. This raises the question whether similar bounds can be obtained for the newer algorithms. On the practical side, investigating the worst-case behavior of data structures is important to understand their benefits and limitations.

Our results In this paper we initiate the study of the worst-case performance of the recent metric data structures based on proximity graphs. As in [21], we mostly focus on three popular algorithms and their implementations: HNSW, NSG and DiskANN. (In addition, we present similar results for other graph-based algorithms in the supplementary material section.) We present both upper and lower bounds on their worst-case search times. Our specific contributions are as follows:

Upper bounds : For one of the data structures studied, namely DiskANN version with 'slow preprocessing', we are able to show a provable worst-case upper bound on its performance. Specifically, we show that (a) the greedy search procedure returns an ´ α ` 1 α ´ 1 ` ϵ ¯ -approximate neighbor in O ´ log α ∆ p α ´ 1 q ϵ ¯ steps and (b) each step takes at most O pp 4 α q d log ∆ q time. This implies that the overall running time is poly-logarithmic in ∆ when d is constant. Here α ą 1 denotes a parameter of the DiskANN algorithm (described in Preliminaries, typically set to 2 ), d denotes the doubling dimension, while ∆ denotes the aspect ratio of the input set P , i.e., the ratio between the diameter and the distance of the closest pair 3 . We also show that our approximation bound is tight, and that the logarithmic dependence of the query time bound on ∆ cannot be removed.

Lower bounds : For the other data structure variants studied (NSG, HNSW and DiskANN with 'fast preprocessing') we present a family of point sets of size n for all n large enough, such that for each n , the empirical query time required to achieve 'reasonable' accuracy is linear in n . For example, for DiskANN, we show that the query procedure can take at least 0 . 1 n steps before it encounters any of the 5 nearest neighbors of the query. Remarkably, the point sets are relatively simple: they live in a 2 -dimensional Euclidean space, and therefore have a constant doubling dimension (see Preliminaries). We use implementations provided by the authors, publicly available on GitHub [16, 29, 22]. Our hard instance examples are available on GitHub at [18].

To the best of our knowledge, these are the first worst-case upper bounds and lower bounds for these data structures.

We emphasize that our results do not contradict the empirical evaluations given in the original papers, or in summary studies such as [27]. Indeed, these algorithms have been demonstrated to be very useful and can be highly effective in practice. Nevertheless, we believe that our results provide important information about the behavior of these algorithms. For example, they demonstrate the importance of validating the quality of answers reported by the algorithms when applying them to new data sets. They also shed light on the types of data sets which result in suboptimal performance of the algorithms.

Related work Approximate nearest neighbor search has been studied extensively; the references in the introduction provide some of the main milestones of that rich body of research. In the context of this paper, we also mention [25, 33], which studied theoretical properties of modern graph-based nearest neighbor data structure. However, their focus is on the average-case performance, i.e., under the assumption that the data is generated according to some well-defined distribution, like uniform over the sphere. In contrast, this work is focused on the worst-case behavior of those algorithms, and show that their empirical running time can be high even for relatively simple 2-dimensional data sets.

2 Preliminaries

We denote the underlying metric space by p X,D q . For any point p P X and radius r ą 0 , we use B p p, r q to denote a ball of radius r centered at p , i.e., B p p, r q ' t q P X : D p p, q q ď r u .

Consider a set of points P . We say that P has the doubling constant C if for any ball B p p, 2 r q centered at some p P P , the set P X B p p, 2 r q can be covered using at most C balls of radius r , and C is the smallest number with this property. Doubling constant is a popular measure of 'intrinsic dimensionality' of high dimensional point sets, see e.g., [17, 24, 5]. The value log 2 C is called the doubling dimension of P . Doubling dimension is often used as a measure of the 'intrinsic dimensionality' of a data set. It generalizes the 'standard' (topological) dimension: for any data set P Ă R d equipped with a metric D p p 1 , p 2 q ' } p 1 ´ p 2 } p , the doubling dimension of P is at most O p d q . However, the doubling dimension of P Ă R D could be much lower than D , e.g., if points in P lie on a low-dimensional manifold. The doubling dimension can be viewed as a finite version of the fractal Hausdorff dimension. Empirical studies (e.g., [11]) showed that the fractal dimension of real data sets is often smaller than their ambient dimension D .

The following fact is standard and follows from the definition.

Lemma 2.1. Consider a set of points P with doubling dimension d . For any ball B p p, r q centered at some p P P and a constant k , the set B p p, r q X P can be covered by a set of m ď O p k d q balls with diameter smaller than r { k , i.e. B p p, r q X P Ď Ť m i ' 1 B p p i , r { k q .

We use d to denote the doubling dimension of the point set P , ∆ ' D max D min to denote the aspect ratio of the input set P , where D max ( D min ) is the maximal (minimal) distance between any pair of vertices in the point set. For two points x u , x v in P ' t x 1 , ...x n u , we use D p x u , x v q to denote the distance between them, or sometimes D p u, v q for simplicity.

3 Analysis of DiskANN

In this section we show bounds on the performance of DiskANN with slow preprocessing.

3.1 DiskANN recap

In this section we give an overview of the DiskANN procedures. For the full description the reader is referred to the original paper [21], or section A (supplementary material).

The DiskANN data structure is based on a directed graph G over the set P , i.e., the set of vertices V of G are associated with the set of points P . After the graph is constructed, to answer a given query q , the algorithm performs search starting from some vertex s . In what follows we describe the search and insertion procedure in more detail.

The search procedure, GreedySearch p s, q, L q , has the following parameters: the start vertex s , the query point q , and the queue size L . It performs a best-first-search using a queue of with a bounded length L , until the L vertices v with the smallest value of D p v, q q seen so far are all scanned. Upon completion, it returns a list of vertices in an increasing distance from q where the first vertex (or the first k vertices) are answers for the query. Note that as long as the graph is connected, the procedure runs for at least L steps. The total running time of the procedure is bounded by the number of steps times the out-degree bound of the graph G .

The construction of the graph G ' p V, E q is done by a repeated invocation of a procedure called RobustPruning . For any vertex v , a set of vertices U (specified later), and parameters α ą 1 and R , RobustPruning p v, U, α, R q proceeds as follows. First, the set U is sorted in the increasing order

of the distance to v . The algorithm traverses this sequence in order. After encountering a new vertex u , the algorithm deletes all other vertices w from U such that D p u, w q ¨ α ă D p v, w q . Finally, the node v is connected to all vertices in U that have not been pruned.

The starting point of the DiskANN data structure construction algorithm 4 is the following simple procedure: for each vertex v , execute RobustPruning p v, U, α, R q with U ' V and R ' n . That is, robust pruning is applied to all vertices in the graph. We refer to this procedure as slow preprocessing , as it can be seen that a naive implementation of this method takes time O p n 3 q . Although the construction time is slow, we show that this construction method provably constructs a graph whose degree depends only logarithmically on the aspect ratio of the graph (assuming constant doubling dimension), and guarantees that the greedy search procedure has polylogarithmic running time. We note that this result is inspired by an observation in [21] about convergence of greedy search in a logarithmic number of steps, though to obtain our result we also need to bound the degree of the search graph and analyze the approximation ratio.

Since the slow-preprocessing-algorithm is too slow in practice, the authors of [21] propose a faster heuristic method to construct the graph G , which we call fast preprocessing method. At the beginning, the graph G is initialized to be a random R -regular graph. Then the construction of the graph G ' p V, E q is done incrementally. The construction algorithms make two passes of the point set in random order. For each vertex v met, the algorithm computes a set of vertices U ' GreedySearch p s, x v , L q (for some starting vertex s ) and then calls the pruning procedure on U , not V . That is, it executes RobustPruning p v, U, α, R q . After pruning is performed, the insertion procedure adds both edges p v, u q and p u, v q for all vertices u P U output by the prunning procedure. Finally, if the degree of any of u P U exceeds a threshold R , then the set of neighbors of u is pruned via RobustPruning p u, N out p u q , α, R q as well. This construction method is implemented and evaluated in the paper.

3.2 Analysis: preprocessing

For the sake of simplicity, we let RobustPruning p p, V, α q be the no-degree-limit version of edge selection, i.e., where R ' n . We first analyze the property of graph constructed by the no-degree-limit pruning, and then show that setting R ' O pp 4 α q d log ∆ q yields equivalent results.

Definition 3.1 ( α -shortcut reachability) . Let α ě 1 . We say a graph G ' p V, E q is α -shortcut reachable from a vertex p if for any other vertex q , either p p, q q P E , or there exists p 1 s.t. p p, p 1 q P E and D p p 1 , q q ¨ α ď D p p, q q . We say a graph G is α -shortcut reachable if G is α -shortcut reachable from any vertex v P V .

First, we show that the slow preprocessing algorithm constructs a graph that is α -reachable.

Lemma 3.2 ( α -shortcut reachable) . For each vertex p , if we connect p to the output of RobustPruning p p, V, α q , then the graph formed is α -shortcut reachable.

Proof. By Definition 3.1, we only need to prove that the constructed graph is α -shortcut reachable from each vertex. Suppose that, for some vertex p , vertex q is not connected to p . Then, in RobustPruning p p, V, α q , there must have existed a vertex p 1 P V connected to p s.t. D p p 1 , q q ď D p p, q q{ α .

Next, we show that the graph produced by no degree limit RobustPruning is actually sparse.

Lemma 3.3 (sparsity) . For any vertex p , let U ' RobustPruning p p, V, α q , then | U | ď O pp 4 α q d log ∆ q where d is the doubling dimension of the point set P .

Proof. Weuse Ring p p, r 1 , r 2 q to denote the set of vertices that lie in B p p, r 2 q but not in B p p, r 1 q . We use D max ( D min ) to denote the maximal (minimal) distances between a pair of vertices in the point set P and by definition ∆ ' D max D min . For each i P r log 2 ∆ s , we consider Ring p p, D max { 2 i , D max { 2 i ´ 1 q separately. We cover Ring p p, D max { 2 i , D max { 2 i ´ 1 q using balls with radius D max { α 2 i ` 1 by Lemma 2.1. The number of balls required is bounded by O pp 4 α q d q . Because every two points in the same ball have distance at most D max { α 2 i from each other, and this value is α

times smaller than their distance lower bound to p , at most one of them will remain after performing RobustPruning p p, V, α q . Therefore, the number of vertices remain after performing RobustPruning p p, V, α q is upper bounded by O pp 4 α q d log ∆ q .

3.3 Analysis: query procedure

We now show that if the graph is constructed using the slow indexing algorithm analyzed in the previous section, then GreedySearch p s, q, 1 q starting from any vertex s returns an p α ` 1 α ´ 1 q -approximate nearest neighbor from query q in a logarithmic number of steps.

Theorem 3.4. Let G ' p V, E q be an α -shortcut reachable graph constructed using the slow preprocessing method. Consider GreedySearch p s, q, L q starting with any vertex s P V and L ' 1 (i.e., the algorithm performs no back-tracking) answering query q . The algorithm finds an ´ α ` 1 α ´ 1 ` ϵ ¯ -approximate nearest neighbor in O ´ log α ∆ p α ´ 1 q ϵ ¯ steps.

Proof. Let a be the nearest neighbor of q , v i be the i -th scanned vertex, d i ' D p v i , q q , with approximate ratio c i ' d i D p a,q q . We use ∆ ' D max D min to denote the aspect ratio of the vertex set.

We know that the distance between v i and a is no more than d i ` D p a, q q by triangle inequality, and because G is α -shortcut reachable, each v i is either connected to a , or v i is connected to a vertex v 1 whose distance to a is shorter than d i ` D p a,q q α . In best-first-search, the next scanned vertex v i ` 1 should have distance from q no farther than v 1 , which is at most d i ` D p a,q q α ` D p a, q q . By induction:

Consider the following three cases, depending on the value of D p a, q q :

Case (1): D p s, q q ą 2 D max . In this case we have D p a, q q ą D p s, q q´ D p a, s q ą D p s, q q´ D max ą D p s, q q{ 2 . Plugging this into inequality 1 gives us c i ' d i D p a,q q ď D p s,q q α i D p a,q q ` α ` 1 α ´ 1 ď 2 α i ` α ` 1 α ´ 1 .

´

¯

Therefore, for any ϵ ą 0 , we get a α ` 1 α ´ 1 ` ϵ -approximate nearest neighbor in log α 2 ϵ steps.

Case (2): D p s, q q ď 2 D max and D p a, q q ě α ´ 1 4 p α ` 1 q D min . By inequality 1, the algorithm reaches an p α ` 1 α ´ 1 ` ϵ q -approximate nearest neighbor when D p s,q q α i ă ϵD p a, q q . Substituting the value of D p s, q q and D p a, q q with corresponding upper and lower bound, we obtain that the number of steps needed is log α 8 p α ` 1 q ∆ p α ´ 1 q ϵ ď O ´ log α ∆ p α ´ 1 q ϵ ¯

Case (3): D p s, q q ď 2 D max and D p a, q q ă α ´ 1 4 p α ` 1 q D min . Suppose at step i , d i ą d p a, q q is not the nearest neighbor. Here we obtain a new lower bound for d i . We know d p v i , a q ą D min , d p v i , q q ą d p a, q q , and d p a, q q ă D min . By triangle inequality, we have d i ' d p v i , q q ą d p v i , a q{ 2 ą D min { 2 . Combing this with inequality 1, we obtain that if v i is not the exact nearest neighbor, it must satisfy D min 2 ď d i ď D p s,q q α i ` α ` 1 α ´ 1 D p a, q q ď 2 D max α i ` D min 4 . This can only happen when i ď log α 8∆ . Therefore, the algorithm reaches the exact nearest neighbor in O p log α ∆ q steps.

We note that similar neighbor selection strategies in the greedy search procedure are also used in HNSW [28] and NSG [15]. However, we cannot prove query performance guarantees similar to the ones above, as those algorithms use α ' 1 .

3.4 A tight convergence rate lower bound for DiskANN

In this section we show that the logarithmic dependence of the query time bound on ∆ is unavoidable, i.e., we cannot replace log ∆ with log n . The proof is deferred to appendix B.

Theorem 3.5. For any choice of α ą 1 , there exists a set of n ' 2 k ´ 1 points on a one dimensional line with aspect ratio ∆ ' O p α n q , and a query q , such that after the slow preprocessing version of DiskANN, greedy search must scan at least Ω p log ∆ q or Ω p n q vertices before reaching an O p 1 q -approximate nearest neighbor of q .

3.5 A tight approximation lower bound for DiskANN

In Theorem 3.4, we know that the slow preprocessing version of DiskANN algorithm can asymptotically get to an α ` 1 α ´ 1 -approximate nearest neighbor. Here, we provide a simple instance showing that this approximation ratio is tight.

Figure 1: α ` 1 α ´ 1 -approximation ratio lower bound instance for slow preprocessing version of DiskANN. The whole instance can be embedded in a two-dimensional grid using l 1 distance, and therefore has a constant doubling dimension. Black dots (solid or hollow) are points in the database, the red dot is the query point. Grids are only used to show the layout structure. Please refer to Section 3.5 for detailed instance description.

2

Q-1

1+6

E

）

E

2

po

Q-1

Q

:0.5e

:√n

We draw the instance in Figure 1. The whole instance can be embedded in a two dimensional grid using l 1 distance. Note that in the figure, grids are drawn to label distances and highlight the layout structure. The vertex set V consists of a point set P and two single points p 1 and a . The point set P further consists of a ? n ˆ ? n (we assume that n is a perfect square) square grid with grid length 0 . 5 ϵ ? n where each grid point is associated with a point. We denote the upper-rightmost point in P by p 0 .

Some important distances are D p p 0 , p 1 q ' 2 , D p p 1 , a q ' 2 α ´ 1 , D p p 0 , a q ' 2 α α ´ 1 , D p a, q q ' 1 ` ϵ , D p p 0 , q q ' 2 α ´ 1 ` 1 ´ ϵ , where ϵ ă 0 . 01 .

Lemma 3.7. Some properties regarding the graph G ' p V, E q built on the instance in Figure 1 using the slow preprocessing version of DiskANN:

For any p P P , p p, p 1 q P E and p p, a q R E

The subgraph of G induced by point set P Ď V is strongly connected.

The starting vertex s is in P .

Proof. (1): According to the l 1 distance, we can see that for any point p P P , D p p, p 1 q ą D p p 0 , p 1 q ' 2 and D p p, a q ą D p p 0 , a q ' 2 ` 2 α ´ 1 ' 2 α α ´ 1 , and D p p 1 , a q ' 2 α ´ 1 . Therefore, in the procedure RobustPruning p p, V, α q , the edge p p, p 1 q remains and edge p p, a q is pruned.

(2): Because the point set P is a uniform grid, we know that for each vertex p , the distances between p and its four adjacent vertices in the grid are smaller than to all other vertices and must remain after RobustPruning p p, V, α q . Therefore the subgraph induced by the vertex set P in the final graph G is strongly connected.

(3): In the implementation of DiskANN, the starting point is the closest vertex to the centroid of the whole vertex set, which lies in P .

Proof of Theorem 3.6. Now we analyze the behavior of running GreedySearch p s, q, L q on the instance above on the graph constructed by DiskANN's slow version. We will show that the first n vertices entering the queue are all vertices in the set P . Additionally, the only neighbor of set P , p 1 , is further from q than any vertex in the set P . If L ď n , GreedySearch p s, q, L q terminates before getting to p 1 or a . Consequently, the nearest vertex returned by GreedySearch p s, q, L q has distance at least 2 α ´ 1 ` 1 ´ ϵ from q , which is not a α ` 1 α ´ 1 -approximate nearest neighbor when ϵ approaches 0 .

The first vertex in the queue is s P P . According to property (1) in Lemma 3.7, no vertex in P is connected to a , any vertex p P P has distance D p p, q q ă D p p 1 , q q thus has higher priority to be scanned than p 1 , and the subset P is strongly connected, so in the first n steps (recall that | P | ' n ), GreedySearch p s, q, L q will always scan vertices in P .

4 Experiments

In our experiments, we test the performance of DiskANN, NSG, and HNSW, on two of our constructed hard instances. We run each algorithm on 20 different data sizes n P t 10 5 , 2 ¨ 10 5 , . . . , 2 ¨ 10 6 u . Each data set consists of n points in the two-dimensional plane. We plot Recall@5 rates (Figures 3 and 5) and approximation ratios (Figure 6) for answering the query with queue length L equal to pn where the percentage p is enumerated from the set 1% , 2% , . . . , 12% , 15% , 18% , 20% , 30% , 40% , 50% . Note that for all graph-based nearest neighbor search algorithms, the value L lower bounds the number of vertices scanned (and therefore the running time) of the algorithm.

Our results show that, unlike on standard benchmarks, the algorithms scan at least 10% of the points in our instances before finding quality nearest neighbors. All experiments were ran on Google Cloud. Since our experiments only use combinatorial measures of algorithm complexity (i.e., the number of vertices searched), the results do not depend on the exact details of the computer architecture.

4.1 Hard instance for DiskANN

Though we provide p α ` 1 α ´ 1 q -approximate ratio upper bound for slow preprocessing version of DiskANN algorithm, in the experiments, we show that there exists a family of instances where the DiskANN algorithm with fast preprocessing cannot reach any top 5 nearest neighbor before scanning 10% of the vertices. We test our constructed instance using the authors' DiskANN code (fast preprocessing) on GitHub [22] . For a comparison, we also test our constructed instance on our implementation of the slow preprocessing DiskANN algorithm.

Q:

:m=(-12000,12000)

p'=(0,;104)

α =(0:103)

104,0)

q = (-:4000,0):

Figure 2: Our constructed hard instance for DiskANN for n ' 10 6 . The instance lives in a two-dimensional Euclidean space, and therefore has a constant doubling dimension. Black dots (solid or hollow) are points in the database, the red dot is the query point. Grids are only used to show the layout structure. See section 4.1 for detailed description and supplementary materials for its implementation.

In the left plot of Figure 3, we can see that in most cases, DiskANN (fast preprocessing version) cannot achieve non-zero Recall@5 unless scanning at least 10% of the vertex set. In the right plot of Figure 3, we can see that DiskANN (slow preprocessing version) generates very sparse graph data structure given our hard instance. Furthermore, after preprocessing, greedy search is very efficient: it finds the exact nearest neighbor in only 2 steps! Overall, it can be seen that, on the hard instances proposed in this paper, the greedy search procedure for query answering is highly efficient after slow preprocessing, and slow (linear time) after fast preprocessing.

DiskANN

50%

40%

DiskANN(slowpreprocessing)

30%

20%

39

L(as a percentage of n)

18%

15%

12%

0.8

1.0

38

37

36

10%

11%

Recall@5

Maximum Degree

9%

8%

0.4

7%

0.2

6%

5%

0.0

30

4%

29

3%

28

2%

27

1%

26

4567891011121314151617181920

n (in units of 105points)

1

2

3

n (in unit of 104points)

4

5

6

7

8

9

10

Description of instance in Figure 2 The instance lives in a 2-dimensional plane under the l 2 distance, so in what follows, we give the coordinates for each point, which defines the distances between all points. In Figure 2, n ' 10 6 , the vertex set V consists of three sets of points M,P and P 1 , of sizes 0 . 8 n, 0 . 1 n, 0 . 1 n respectively, and a single answer point a . Let l ' 0 . 01 ˚ n ' 10 4 . M is a a | M | ˆ a | M | square grid with grid side length 1 whose bottom-right corner is m ' p´ 1 . 2 l, 1 . 2 l q . P is a a | P | ˆ a | P | square grid with grid side length 1 whose upper-right corner is p ' p´ l, 0 q . P 1 is a a | P 1 | ˆ a | P 1 | square grid with grid side length 1 and whose bottom-left corner is p ' p 0 , l q . (We assume that | M | , | P | , | P 1 | are perfect squares.) The query point is q ' p´ 0 . 4 l, 0 q , whose nearest neighbor is a ' p 0 , 0 . 1 l q . Since our experiments measure Recall @5 , we add another 4 points very close to a , which is not shown in the figure for simplicity.

Intuition First, the start point s should lie in the point set M . The three key properties we want to maintain in the construction of the graph are that (1) s is always connected to at least one vertex in the point sets P and P 1 . (2) Except for the points randomly connected to a at initialization, only the points in the set P 1 can still have edges to a at the end of the construction. (3) In the final graph, at least L vertices in set P are reachable from start point s without passing through any vertex not in P . If these three properties hold, GreedySearch p s, q, L q scans only the vertices in P (except for the start point s ) until it finds a vertex p i P P which is randomly connected to a at initialization. In expectation, this requires scanning Ω p n { R q vertices, and DiskANN won't reach the nearest neighbor a before that. The actual running time of the code is even slower.

4.2 Hard instance for NSG and HNSW

A variant of the hard instance from the previous section works for NSG [15] and HNSW [28] algorithms as well. In Figure 4, we draw this instance for n ' 10 6 . It is easy to mimic this instance for other data sizes.

The parameters of NSG are as follows. We set K ' 400 , L ' 400 , iter ' 12 , S ' 15 , R ' 100 for EFANNA [12] to construct the KNN graph. We set L ' 60 , R ' 70 , C ' 500 for constructing NSG from KNN graph. These parameters were used by the authors when testing on GIST1M dataset, whose size is similar to ours. The parameters of HNSW are as follows. ef construction ' 200 (as used in their example code), M ' 64 (maximal degree limit in their suggested range), num _ threads ' 1 . We test our constructed instance using the authors' code available at [29] (for HNSW) and at [16]

m=（÷12000,12000)

:p' = (0:104)

α = (0,103)

4000,0):

NSG

HNSW

50%

F%0S

40%

40%

%00

%0E

20%

20%

18%

1.0

18%

1.0

L (as a percentage of n)

15%

L(as a percentage of n)

15%

0.8

12%

0.8

0.6

Recall@5

11%

10%

0.6

Recall@5

0.4

9%

0.4

8%

7%

0.2

0.2

6%

5%

0.0

5%

0.0

4%

4%

3%

3%

2%

2%

1%

1%

1 2 3 4 5 6 7 8 9 1011121314151617181920

1 2 3 4 5 6 7 8 9 1011121314151617181920

n (in units of 105 points)

n (in units of 105 points)

(for NSG). We note that both implementations are randomized, but their random seeds are hardwired into the code. To obtain more informative results, we generate and use different random seeds for each algorithm execution, and plot the average recall.

In Figure 5, we can see that, for most values of n , both NSG and HNSW algorithms cannot achieve good Recall@5 unless scanning at least 10% of the vertex set.

Description of instance in Figure 4 This instance is based on the last instance in Figure 2. The main difference is that we use a few chains of points to connect the three subsets M , P , and P 1 . Specifically, on the dotted line, we add points uniformly spaced out, separated by a distance of 5 in the horizontal and/or vertical directions. For n ' 10 6 and l ' 0 . 01 n ' 10 4 , the instance consists of 400 p 0 . 04% q points on the diagonal (from m ' p´ 1 . 2 l, ´ 1 . 2 l q to p´ l, ´ l q ), 2000 p 0 . 2% q on the horizontal line (from p´ l, l q to p 1 ' p 0 , l q ) and 2000 p 0 . 2% q points on the vertical line (from p´ l, l q to p ' p´ l, 0 q ). In total, there are 4400 new added points on the chain compared with the previous instance, occupying 0 . 44% of the vertex set.

Intuition for the instance in Figure 4 The new added vertex chains are there to make KNN graph connected for those nearest neighbor algorithms using KNN as part of their constructions. Thanks to these chains, most of the vertices in subset P and P 1 are reachable from the start point. The construction algorithm will only add edges from the vertices in the set P 1 to a , but not from P . Then,

GreedySearch on query q will first traverse the whole subset P before going to P 1 . Therefore, GreedySearch cannot get to the nearest neighbor if the queue length limit L is smaller than | P | .

4.3 Cross-comparisons

We also evaluated DiskANN on hard examples for NSG and HNSW, and vice versa. For n ' 10 6 the results are similar to the ones reported in earlier sections: none of the algorithms can achieve non-zero recall unless L exceeds 10% of the data set size. We did not experiment with other values of n , as two different families of instances are anyway helpful for other algorithms, as outlined in the next section and described in detail in supplementary material.

4.4 Evaluating approximation ratio of the algorithms

In addition to Recall @5 , we also measure the average approximation ratio of DiskANN (fast preprocessing), NSG and HNSW on slightly modified hard instances from Figure 2 and 4. The modifications only involve scaling of the entire instance and bringing points a and q closer, to amplify the approximate ratio for other points. As per Figure 6, the average ratio is very high unless L ě 0 . 1 n . Note that we do not include the plot for DiskANN (slow preprocessing) here because, as shown in Figure 3, that algorithm (with L ' 1 ) can find the exact nearest neighbor in two steps.

DiskANN

103

NSG

HNSW

50%

50%

50%

L(as a percentage of n)

40%

(asapercentage of n)

30%

40%

102

L(as a percentage of n)

40%

102

102

ApproxRatio

ApproxRatio

ApproxRatio

%0

的

10%

101

101

10

100

100

100

123 456 7 891011121314151617181920

123456 7 891011121314151617181920

1234567 891011121314151617181920

n (in units of 105points)

n (in units of 105 points)

n (in units of 105 points)

4.5 Experiments on other popular approximate nearest neighbor search algorithms

We also tested other popular approximate nearest neighbor search algorithms covered in the survey [37]: NGT [20], SSG [14], KGraph [37], DPG [27], NSW [30], SPTAG-KDT [7] and EFANNA [12]. We ran them on our hard instances, with n ranging over 10 5 , 2 ¨ 10 5 . . . 2 ¨ 10 6 and the queue length L ' 0 . 1 n . The table depicts average Recall@5 rates over all values of n and 5 or 10 repetitions. One can see that all algorithms achieve sub-optimal recall. See the appendix for more details.

5 Conclusions

In this paper we study the worst-case performance of popular graph-based nearest neighbor search algorithms. We demonstrate empirically that almost all of them suffer from linear query times on carefully constructed instances, despite being fast on benchmark data sets [37]. The exception is DiskANN with slow-preprocessing, for which we bound the approximation ratio and the running time. However, its super-linear preprocessing time makes it difficult to use for large data sets.

An important question raised by our work is whether there is a fast preprocessing algorithm, and a query answering procedure, that are empirically fast (e.g., as fast as for DiskANN) while having worst case query time and approximation guarantees. Another interesting direction is to investigate whether it is possible to replicate our findings for real data sets, using adversarially selected queries.

Acknowledgement: This work was supported by the Jacobs Presidential Fellowship, the NSF TRIPODS program (award DMS-2022448), Simons Investigator Award and MIT-IBM Watson AI Lab.

References

Alexandr Andoni and Piotr Indyk. Near-optimal hashing algorithms for approximate nearest neighbor in high dimensions. Communications of the ACM , 51(1):117-122, 2008.

Alexandr Andoni, Piotr Indyk, and Ilya Razenshteyn. Approximate nearest neighbor search in high dimensions. In Proceedings of the International Congress of Mathematicians: Rio de Janeiro 2018 , pages 3287-3318. World Scientific, 2018.

Sunil Arya and David M Mount. Approximate nearest neighbor queries in fixed dimensions. In SODA , volume 93, pages 271-280, 1993.

Sunil Arya, David M Mount, Nathan S Netanyahu, Ruth Silverman, and Angela Y Wu. An optimal algorithm for approximate nearest neighbor searching fixed dimensions. Journal of the ACM (JACM) , 45(6):891-923, 1998.

Alina Beygelzimer, Sham Kakade, and John Langford. Cover trees for nearest neighbor. In Proceedings of the 23rd international conference on Machine learning , pages 97-104, 2006.

Edgar Chávez, Gonzalo Navarro, Ricardo Baeza-Yates, and José Luis Marroquín. Searching in metric spaces. ACM computing surveys (CSUR) , 33(3):273-321, 2001.

Qi Chen, Bing Zhao, Haidong Wang, Mingqin Li, Chuanjie Liu, Zengzhong Li, Mao Yang, and Jingdong Wang. SPANN: highly-efficient billion-scale approximate nearest neighbor search. CoRR , abs/2111.08566, 2021.

Kenneth L Clarkson et al. Nearest-neighbor searching and metric space dimensions. Nearestneighbor methods for learning and vision: theory and practice , pages 15-59, 2006.

Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine , 29(6):141-142, 2012.

Yury Elkin and Vitaliy Kurlin. A new near-linear time algorithm for k-nearest neighbor search using a compressed cover tree. In International Conference on Machine Learning , pages 9267-9311. PMLR, 2023.

Christos Faloutsos and Ibrahim Kamel. Beyond uniformity and independence: Analysis of r-trees using the concept of fractal dimension. In Proceedings of the thirteenth ACM SIGACTSIGMOD-SIGART symposium on Principles of database systems , pages 4-13, 1994.

Cong Fu and Deng Cai. EFANNA : An extremely fast approximate nearest neighbor search algorithm based on knn graph. CoRR , abs/1609.07228, 2016.

Cong Fu, Changxu Wang, and Deng Cai. Ssg : Satellite system graph for approximate nearest neighbor search. https://github.com/ZJULearning/SSG , 2020.

Cong Fu, Changxu Wang, and Deng Cai. High dimensional similarity search with satellite system graph: Efficiency, scalability, and unindexed query compatibility. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(8):4139-4150, 2022.

Cong Fu, Chao Xiang, Changxu Wang, and Deng Cai. Fast approximate nearest neighbor search with the navigating spreading-out graph. Proceedings of the VLDB Endowment , 12(5):461-474, 2019.

Cong Fu, Chao Xiang, Changxu Wang, and Deng Cai. Nsg : Navigating spread-out graph for approximate nearest neighbor search. https://github.com/ZJULearning/nsg , 2019.

Anupam Gupta, Robert Krauthgamer, and James R Lee. Bounded geometries, fractals, and low-distortion embeddings. In 44th Annual IEEE Symposium on Foundations of Computer Science, 2003. Proceedings. , pages 534-543. IEEE, 2003.

Piotr Indyk and Haike Xu. Hard-instances-for-popular-approximatenearest-neighbor-search-implementations. https://github.com/xuhaike/ Hard-Instances-for-Popular-Approximate-Nearest-Neighbor-Search-Implementations , 2023.

Masajiro Iwasaki. Neighborhood graph and tree for indexing high-dimensional data. https: //github.com/yahoojapan/NGT , 2023.

Masajiro Iwasaki and Daisuke Miyazaki. Optimization of indexing based on k-nearest neighbor graph for proximity search in high-dimensional data. arXiv preprint arXiv:1810.07355 , 2018.

Suhas Jayaram Subramanya, Fnu Devvrit, Harsha Vardhan Simhadri, Ravishankar Krishnawamy, and Rohan Kadekodi. Diskann: Fast accurate billion-point nearest neighbor search on a single node. Advances in Neural Information Processing Systems , 32, 2019.

Suhas Jayaram Subramanya, Fnu Devvrit, Harsha Vardhan Simhadri, Ravishankar Krishnawamy, and Rohan Kadekodi. Diskann. https://github.com/microsoft/DiskANN , 2023.

Hervé Jégou, Matthijs Douze, and Cordelia Schmid. Product Quantization for Nearest Neighbor Search. IEEE Transactions on Pattern Analysis and Machine Intelligence , 33(1):117-128, January 2011.

Robert Krauthgamer and James R Lee. Navigating nets: Simple algorithms for proximity search. In Proceedings of the fifteenth annual ACM-SIAM symposium on Discrete algorithms , pages 798-807. Citeseer, 2004.

Thijs Laarhoven. Graph-based time-space trade-offs for approximate near neighbors. In 34th International Symposium on Computational Geometry (SoCG 2018) . Schloss Dagstuhl-LeibnizZentrum fuer Informatik, 2018.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Wen Li, Ying Zhang, Yifang Sun, Wei Wang, Mingjie Li, Wenjie Zhang, and Xuemin Lin. Approximate nearest neighbor search on high dimensional data-experiments, analyses, and improvement. IEEE Transactions on Knowledge and Data Engineering , 32(8):1475-1488, 2019.

Yu A Malkov and Dmitry A Yashunin. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE transactions on pattern analysis and machine intelligence , 42(4):824-836, 2018.

Yu A Malkov and Dmitry A Yashunin. Hnswlib - fast approximate nearest neighbor search. https://github.com/nmslib/hnswlib , 2022.

Yury Malkov, Alexander Ponomarenko, Andrey Logvinov, and Vladimir Krylov. Approximate nearest neighbor algorithm based on navigable small world graphs. Information Systems , 45:61-68, 2014.

Marvin Minsky and Seymour A Papert. Perceptrons, Reissue of the 1988 Expanded Edition with a new foreword by Léon Bottou: An Introduction to Computational Geometry . MIT press, 2017.

Gonzalo Navarro. Searching in metric spaces by spatial approximation. The VLDB Journal , 11(1):28-46, 2002.

Liudmila Prokhorenkova and Aleksandr Shekhovtsov. Graph-based nearest neighbor search: From practice to theory. In International Conference on Machine Learning , pages 7803-7813. PMLR, 2020.

Gregory Shakhnarovich, Trevor Darrell, and Piotr Indyk. Nearest-neighbor methods in learning and vision . MIT press, 2006.

Jingdong Wang, Ting Zhang, Nicu Sebe, Heng Tao Shen, et al. A survey on learning to hash. IEEE transactions on pattern analysis and machine intelligence , 40(4):769-790, 2017.

Jun Wang, Wei Liu, Sanjiv Kumar, and Shih-Fu Chang. Learning to hash for indexing big data-a survey. Proceedings of the IEEE , 104(1):34-57, 2015.

Mengzhao Wang, Xiaoliang Xu, Qiang Yue, and Yuxiang Wang. A comprehensive survey and experimental comparison of graph-based approximate nearest neighbor search. arXiv preprint arXiv:2101.12631 , 2021.

Mengzhao Wang, Xiaoliang Xu, Qiang Yue, and Yuxiang Wang. A comprehensive survey and experimental comparison of graph-based approximate nearest neighbor search. https: //github.com/Lsyhprum/WEAVESS , 2021.

A DiskANN Algorithm

Algorithm 1 RobustPruning p i, U, α, R q

1: Input Vertex i , candidate neighbor set U , pruning parameter α , degree limit R (default R is n if not given)

2: Result Update N out p i q , the set of outneighbors of i

3: U Ð U Y N out p i q

4: N out p i q Ð ∅

5: while U ‰ ∅ and | N out p i q| ă R do

6: v Ð argmin v P U D p x v , x i q

7: N out p i q Ð N out p i q Y v

8: U Ð U z v

9: U Ð t v 1 P U : D p x v , x v 1 q ¨ α ą D p x i , x v 1 qu

10: end while

Algorithm 2 GreedySearch p s, q, L q 1: Input Graph G ' p V, E q , seed s , query point q , queue length limit L 2: Output visited vertex list U 3: A Ðt s u 4: U Ð ∅ 5: while A z U ‰ ∅ do 6: v Ð argmin v P A z U D p x v , q q 7: A Ð A Y N out p v q 8: U Ð U Y v 9: if | A | ą L then 10: A Ð top L closest vertices to q in A 11: end if 12: end while 13: sort U in increasing distance from q 14: return U

Algorithm 3 DiskANN indexing algorithm (with fast preprocessing)

2: Output A proximity graph G ' p V, E q where V ' t 1 ..n u are associated with point sets P .

1: Input Point set P ' t x 1 ...x n u , degree limit R , queue length L 3: G Ð randomly sample a R -regular graph on vertex set V ' t 1 ..n u 4: s Ð vertex for the point closest to the centroid of P 5: for k ' 1 to 2 do 6: σ Ð a random permutation of r 1 ...n s 7: for i ' 1 to n do 8: U Ð GreedySearch p s, x σ p i q , L q 9: RobustPruning p σ p i q , U, α, R q 10: for vertex j in N out p σ p i qq do 11: N out p j q Ð N out p j q Y σ p i q 12: if | N out p j q| ą R then 13: RobustPruning p j, N out p j q , α, R q 14: end if 15: end for 16: end for 17: end for

Algorithm 4 DiskANN indexing algorithm (with slow preprocessing)

1: Input Vertex set P ' t x 1 ...x n u , parameters: degree limit R

2: Output A proximity graph G ' p V, E q where V ' t 1 ..n u are associated with point sets P .

3: s Ð vertex for the point closest to the centroid of P

4: for i ' 1 to n do

5: N out p i q Ð RobustPruning p i, V, α, R q

6: end for

B Dependence on aspect ratio ∆

Consider the following 1-dimensional data set with n ' 2 k points located at t x i u n i ' 1 , where

and β ' max p 1 α ´ 1 , α ´ 1 q . This is a symmetric line starting from 0 to p 2 ` β q α k . Each point's distance toward the closer endpoint is α times larger than that of the previous point.

Lemma B.1. The graph G ' p V, E q built on the above instance using the slow preprocessing version of DiskANN satisfies the following properties:

For any i P r k ` 1 , n s , p i, k q P E and p i, j q R E for any j ă k

For any j ă i ď k , p i, j q P E if and only if j ' i ´ 1

Since the x i 's are symmetric, the same properties also hold in the other direction.

Proof. (1): For any i P r k ` 1 , n s , we can check that no vertex j such that k ă j ă i can delete k from i 's neighborhood, because x j ´ x k x i ´ x k ą α k β α k β ` α k ě 1 α . Thus, we have p i, k q P E . Similarly, we have that k will delete any vertex j ă k from i 's neighborhood because x k ´ x j x i ´ x j ď α k α k ` α k β ď 1 α . These two inequalities use that β ' max p 1 α ´ 1 , α ´ 1 q

Proof of Theorem 3.5. Based on the graph properties in Lemma B.1, let us determine the length of the shortest path from a starting point s (selected arbitrarily by the DiskANN algorithm) to a constant approximate nearest neighbor of a given query q . We select our query q to be either 0 or 2 α k ` α k β , i.e., one of the two endpoints of the data set, whichever is farther from x s . To find an O p 1 q -approximate nearest neighbor of the query q within l steps of GreedySearch, there should be at least one path with less than l hops from s to q 's approximate nearest neighbor. WLOG, let's assume q ' 0 and s ą k . By Property (1) of Lemma B.1, among t 1 ...k u , the vertex k is the only neighbor of any vertex on the right of k . By Property (2) of Lemma B.1, for any vertex i P r 1 , k s , its only neighbor on its left is i ´ 1 . Therefore, it takes at least l ě Ω p n q and l ě Ω p log ∆ q steps for slow preprocessing DiskANN with GreedySearch to reach any O p 1 q - approximate nearest neighbor in this constructed instance.

C More experimental results

C.1 Hard instance for KD-Tree based nearest neighbor search algorithm

Some nearest neighbor search algorithms use KD-tree to find the entry point for greedy search. In this case, we design a hard instance where KD-tree cannot get good entry point close to the nearest neighbor. See Figure 7. We draw our constructed instance for n ' 10 6 . It is easy to mimic this instance for other data sizes.

Description of instance in Figure 7 Our instance has 6 dimensions, where the first two dimensions of the instance are similar to Figure 2 but with different choices of parameters. Again, the vertex set V consists of three sets of points M,P,P 1 (with size 0 . 1 n, 0 . 1 n, 0 . 8 n respectively) and a single answer point a : M is a a | M | ˆ a | M | square grid with unit length 1 and with bottom right corner m ' p´ 10 9 , 10 9 q . P is a vertical chain of points consisting of unit intervals from p l ' p´ 10 9 , ´ 0 . 05 n q to p h ' p´ 10 9 , 0 . 05 n q . P 1 is a a | P 1 | ˆ a | P 1 | square grid with unit length 1 whose bottom left corner is p 1 ' p 0 , 10 9 q . The answer point is a ' p 0 , 3 ˚ 10 8 q . The query point is q ' p´ 3 ˚ 10 8 , 0 q . For a vertex v P M Y P 1 Yt a u , each of v 's other 4 coordinates are sampled from a uniform distribution supported on r 5 ˚ 10 7 , 6 ˚ 10 7 s . For a vertex v P P , its other 4 coordinates are sampled from a uniform distribution supported on r 10 7 , 2 ˚ 10 7 s . And for point q , its other 4 coordinates are all 0 . Note that though such choices of parameters are tailored to the implementation of KD-trees used in SPTAG and EFANNA in [37], some general ideas are reusable for constructing hard instances for other implementations.

Intuition for instance in Figure 7 The main reason why we construct a new example here is to handle the use of KD-tree to search for a good starting point. Implementation of KD-tree provided in [37] always randomly picks one of the top 5 dimensions with the largest variance as the splitting

D

m=(-10°,109)

O

O-

O-

p

(0", 109)

α=(0,3* 108)

:ph = (-10°,5* 104)

:q= (-3* 108,0)

:p(-10%,-5*104)

dimension, and divides the vertices into two halves at their mean. In Figure 7, because of the separation on the other 4 dimensions, our construction can make sure that KD-tree quickly gets to a subtree with vertices only in P . Then KD-tree will only horizontally split the chain from p h to p l or split via the other 4 dimensions. In the vertical axis, the coordinates for vertices in P and a are quite close, so KD-tree will assign them a low distance estimation based on only a horizontal split, resulting in KD-tree scanning all vertices on the chain before scanning other vertices outside of the set P . As long as we make the KD-tree select a vertex in P as the starting point, we can ensure (as in Figure 2) that GreedySearch will scan all vertices in P before going to M or P 1 .

C.2 More experiments on other popular nearest neighbor search algorithms

We further test the other 7 popular nearest neighbor search algorithms studied in the survey [37]. We use the same setting as in Section 4. We run each algorithm for 20 different data sizes n P t 10 5 , 2 ¨ 10 5 , . . . , 2 ¨ 10 6 u using hard instances in Figure 4, Figure 2, or Figure 7 (introduced in Appendix C.1). We plot the Recall@5 rate for answering the query with queue length L equal to pn where the percentage p is enumerated from the set 1% , 2% , . . . , 12% , 15% , 18% , 20% , 30% , 40% , 50% .

NGT [20] We use NGT's implementation from the authors' GitHub repository [19]. We run NGT on the hard instance in Figure 2, using all default parameters as stated in GitHub's readme, except that we use the command '-i g' to generate only the graph index, because of our focus on graph-based nearest neighbor search algorithms. We use command '-p 1' to set the number of threads to 1. We run this experiment 10 times and report the average recall.

SSG [14] We use SSG implementation from the authors GitHub repository [13]. We run SSG on the hard instance in Figure 4, using parameters K ' 200 , L ' 200 , iter ' 12 , S ' 10 , R ' 100 (for building KNN graph) L ' 100 , R ' 50 , Angle ' 60 (constructing SSG). The parameters chosen here are copied from the author's selected parameters for data set SIFT1M [23], whose data size is close to ours. We run this experiment 10 times using different random seeds and report the average recall.

KGraph We use KGraph implementation due to [37], from the GitHub repository [38]. We run KGraph on the hard instance in Figure 4 using parameter K ' 100 , L ' 130 , iter ' 12 , S ' 20 , R ' 50 . Parameters here are copied from the authors' selected parameters used for their synthetic data set named ' n _ 1000000' , whose size is close to ours. We run this experiment 5 times using different random seeds and report the average recall.

NGT

50%

40%

30%

20%

18%

1.0

L (as a percentage of n)

15%

12%

0.8

11%

10%

0.6

Recall@5

9%

0.4

8%

7%

6%

0.2

5%

0.0

4%

3%

2%

1%

3

45 6 7 8 9 1011121314151617181920

n (in units of 105 points)

DPG [27] We use DPG implementation due to [37], from the GitHub repository [38]. We run DPG on the hard instance in Figure 4 using parameter K ' 100 , L ' 100 , iter ' 12 , S ' 20 , R ' 300 . Parameters here are copied from the authors' selected parameters for their synthetic dataset named ' n _ 1000000' , whose size is close to ours. We run this experiment 10 times using different random seeds and report the average recall.

NSW [30] We use NSW's implementation due to [37], from the GitHub repository [38]. We run NSW on the hard instance in Figure 4 (with a different vertex permutation) using parameter max _ m 0 ' 100 , ef _ construction ' 400 . Parameters here are copied from the author's selected parameters for their synthetic dataset named ' n _ 1000000' , whose data size is close to ours. We run this experiment 5 times using different random seeds and report the average recall.

SPTATG-KDT [7] We use SPTATG-KDT implementation due to [37], from the GitHub repository [38]. We run SPTATG-KDT on the hard instance in Figure 7 using parameters KDT _ number ' 1 , T P T _ number ' 16 , T P T _ leaf _ size ' 1500 , scale ' 2 , CEF ' 1500 . Parameters here are copied from the authors' selected parameters for their synthetic dataset named ' n _ 1000000' , whose size is close to ours. We run this experiment 5 times and report the average recall.

EFANNA [12] We use EFANNA's implementation due to [37], from the GitHub repository [38]. We run EFANNA on the hard instance in Figure 7 using parameter nTrees ' 4 , mLevel ' 8 , K ' 80 , L ' 140 , iter ' 7 , S ' 25 , R ' 150 . Parameters here are copied from the authors' selected parameters for their synthetic dataset named ' n _ 1000000' , whose size is close to ours. We run this experiment 10 times and report the average recall.

Experimental results for these 7 algorithms are plotted in Figure 8, Figure 9, and Figure 10. We can see that all algorithms achieve suboptimal Recall@5 rates until the queue length L is greater than 10% of the data size.

SSG

KGraph

50%

50%

40%

40%

%08

%0E

20%

20%

L(as a percentage of n)

1.0

L(as a percentage of n)

18%

1.0

0.8

0.8

0.6

Recall@5

Recall@5

0.4

%6

8%

0.2

6%

0.2

0.0

5%

0.0

4%

3%

T

2%

1%

1%

1 2 3 4 5 6 7 8 9 1011121314151617181920

1 2 3 4 5 6 7 8 91011121314151617181920

n (in units of 105 points)

n (in units of 105 points)

DPG

NSW

50%

F

50%

T

40%

40%

%08

%08

20%

20%

L(as a percentage of n)

1.0

L(as a percentage of n)

18%

15%

0.8

%

0.8

0.6

Recall@5

10%

0.6

Recall@5

0.4

%6

0.4

8%

0.2

7%

0.2

6%

0.0

5%

0.0

4%

3%

2%

1%

1%

1 2 3 4 5 6 7 8 91011121314151617181920

1 2 34567891011121314151617181920

n (in units of 105 points)

n (in units of 105 points)

SPTAG-KDT

EFANNA

50%

50%

40%

40%

30%

30%

20%

20%

L(as a percentage of n)

18%

1.0

L(as a percentage of n)

18%

1.0

15%

15%

12%

0.8

12%

0.8

10%

11%

0.6

Recall@5

11%

10%

0.6

Recall@5

8%

9%

0.4

%6

8%

0.4

7%

0.2

7%

0.2

6%

6%

5%

0.0

5%

0.0

4%

4%

3%

3%

2%

2%

1%

十

1%

4

5

67 8 9 10 11 12 13 141516 1718 1920

3

5 6 7 8 9 10 1112 1314151617181920

n (in units of 105 points)

n (in units of 105 points)

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper theoretically analyzes the worst-case performance of DiskANN. It shows that DiskANN (with slow preprocessing) can provably solve the approximate nearest neighbor problem with a constant approximation ratio. It also provides empirical results for other algorithms, such as DiskANN with fast preprocessing, HNSW, and NSG, and shows that there are hard instances that require linear query time.\n\nSTRENGTHS:\n1) The paper theoretically analyzes graph-based nearest neighbor search in quite a general context. While previous works made some strict assumptions about the distribution of elements (uniform), here the analysis is performed in terms of doubling dimension, which is a significant step forward.\n\n2) For one algorithm, DiskANN with slow preprocessing, the constant approximation ratio can be guaranteed. Moreover, it is proven that the theoretically obtained approximation is tight. The proofs are mostly clear and easy to follow.\n\n3) To the best of my knowledge, this is the first paper that shows the effect of neighbor pruning on the performance of graph-based NNS - previous works only considered nearest-neighbor graphs.\n\nWEAKNESSES:\nMost of my concerns are about the experimental part:\n- Theoretical analysis only shows the constant approximation factor, while the experiments mainly focus on Recall@5 – note that for Recall@5, there are no theoretical guarantees for DiskANN with slow preprocessing.\n- It would be informative if all the algorithms were tested on all the “hard” examples and both Recall@5 and approximation ratio were shown. These hard examples may also include the example for DiskANN from Section 3.4. Otherwise, it is hard to conclude that DiskANN is empirically better.\n- On the figures illustrating the performance, Recall@5 is shown as a function of L. The figures would be more informative if they showed the fraction of considered nodes (distance computations) instead, as is usually done.\n- For a better comparison of the two versions of DiskANN, it would be better to plot Recall@5 as a function of the number of distance computations for both of them.\n- Figure 6 analyzes the approximation ratio for three algorithms but not for DiskANN with slow preprocessing.\n\nIn summary, currently, it can be hard to conclude the advantage of DiskANN with slow preprocessing over other algorithms since all the algorithms are evaluated in different setups.\n\nSome comments on the presentation:\n- Description of DiskANN is important for understanding the theoretical part of the paper. Thus, I suggest moving the key steps of DiskANN to the main text — for instance, the definition of RobustPruning.\n- A short description of NSG and HNSW would also help follow their empirical analysis.\n- To make a connection with previous works, in Lemma 3.3, it can be useful to comment that for uniformly distributed data, $|U|$ is logarithmic in $n$ and exponential in dimension.\n- After Theorem 3.4, it would be helpful to write how the claimed statement follows from (1) and (2).\n- Figure 2: $0.5 \\epsilon$ should probably be $0.5 \\epsilon / \\sqrt{n}$ here.\n\nMinor comments:\n- I suggest defining the aspect ratio in Section 2 (preliminaries).\n- $q$ denotes the query but also some other points in $X$ (e.g., in Sections 2 and 3).\n- Proof of Lemma 3.2 is very simple and can be omitted – it can be written that the lemma directly follows from the RobustPruning definition.\n- In l136, a reference to Lemma 2.1 can be helpful.\n- In Section 4.1, it is better to move the description of the instance to the beginning. Otherwise, it is hard to follow the beginning of the section and Figure 3.\n- Parameter R (degree limit) is not discussed in Appendix A (except for the pseudocode).\n\nSome typos:\n- l14: “in a some”\n- l37, l63: footnotes are properly typeset after punctuation marks\n- l70: “they live a 2-dimensional” – missing “in”\n- l110: “run” -> “runs”\n- l146: “the algorithms performs”\n- l153: redundant “for”\n- l234: “!.”\n- l406: “of with”\n\nQUESTIONS:\n1. In l134, should it be $diam/2^i$ instead of $\\Delta/2^i$? Since we want the radius to vary from the minimum distance to diameter.\n\n2. l182 – here a constant bound is given for $\\epsilon$, but we also need $\\epsilon < 1/(\\alpha-1)$ for $a$ to be the nearest neighbor.\n\n3. How specific is Theorem 3.6 to $l_1$ distance? Can similar examples be constructed for $l_2$?\n\n4. Can the analysis be extended to beam search instead of greedy search?\n\n5. Is it possible to guarantee finding the exact nearest neighbor under some conditions? I assume this should be possible if the distance between $q$ and its nearest neighbor is sufficiently small.\n\n6. Could you give more details on how a logarithmic number of steps follows from Theorem 3.4?\n\n7. Can the intuition in the last paragraph of Section 4.1 be potentially transformed to a formal result?\n\n8. Is $d$ assumed to be constant in the paper? Or do all the results hold for $d = d(n)$?\n\n9. I think that it is good to mention that the standard neighbor pruning (used, e.g., in HNSW) uses $\\alpha = 1$. Thus, a constant approximation cannot be guaranteed.\n\nLIMITATIONS:\nLimitations are not discussed, the negative societal impact is not relevant for this work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies a specific class of graph-based similarity search algorithms, and establishes approximation upper bounds, and runtime lower bounds for this algorithm. In the process of establishing lower bounds, the paper also presents various point configurations that would be hard for these graph-based search algorithms, and then evaluates multiple graph-based search algorithms on these hard instances, highlighting that these examples are hard for most search algorithms, even though these graph-based search algorithms are known to perform really well in practice.\n\nSTRENGTHS:\n**Important class of algorithm analysed.**\nGraph-based approximate nearest-neighbour search algorithms can be extremely efficient in practice, and the goal of this paper to analyse one such algorithm, DiskANN, is well motivated. This paper establishes upper bounds on the approximation in the nearest-neighbour search result at any given point of the greedy search algorithm. Then, the paper goes on to present explicit examples with interesting geometrical configurations, embedded into a 2 dimensional plane, which are challenging instances of nearest-neighbour search for DiskANN. These examples make explicit one scenario where the graph built on the data forces the greedy search to visit almost all nodes.\n\n\n**Wide coverage of graph-based search algorithms.**\nBoth in the empirical evaluation, and the development of \"hard instances\", this paper covers multiple graph-based search algorithms such as DiskANN, HSNW, NSG, and SPTAG-KDT. Each of these examples exploit the specific graph construction scheme for these methods. The empirical evaluation of these multiple algorithm highlights how these hard instances are hard (to different extents) for all such graph-based algorithms.\n\nWEAKNESSES:\n**Interpreting the worst-case upper-bounds.**\nOne weakness of this paper is that I am unable to get an intuition of what the bound in Theorem 3.4 is telling us. One interpretation is that, as $i \\to \\infty$ (in the asymptotic range), we get a solution that is $\\left(\\frac{\\alpha+1}{\\alpha-1}\\right)$-approximate (as stated in line 170-172). But the runtime for $i$ iterations is $O(i (4\\alpha)^d \\log \\Delta)$ (as per Lemma 3.3). But in the nearest-neighbor search, the runtime for the asymptotic $i \\to \\infty$ scenario is bounded by $O(n)$. So it seems that we are saying that if we wish to achieve constant approximation with DiskANN, we have to do $O(n)$ work, which seems a somewhat vacuous result -- if we are ready to do $O(n)$ work, usually we are able to get the exact solution (not even an approximation). And Theorem 3.5 seems to be saying the same -- we always need to do $O(n)$ work. Does that mean that we are unable to get anything better than $O(n)$ guarantees for DiskANN (even with the slow preprocessing)?\n\n**Interpreting the motivation behind the hard synthetic examples.**\nThe authors do a great job at creating examples such as the one in Figure 1 & 2, where the slow preprocessing and the fast preprocessing versions of DiskANN respectively are unable to find the constant approximation nearest-neighbor in time less than $O(n)$. Figure 4 is another great synthetic example that is hard for NSG and HNSW. However, it is not directly clear to me why these examples are of interest, or something we should evaluate graph-based algorithms on. These examples are very structured, with very specific careful placements of the query $q$ and its nearest-neighbor $a$. First, it is not clear whether these examples are unique and canonical in some sense, or are there other (similarly constructed) point configurations which would similarly be a hard instance for the graph based algorithms. If they are not unique, why should these examples be studied and not others? Or if we are able to do well on these examples, can we say anything about other \"hard\" or \"easy\" instances? Secondly, even if these examples are canonical in some way, it is not clear if these resemble real datasets in any form or if these examples are somehow practical viable. Without such motivation and justification, it is not clear why we should care about these examples.\n\n\n\nMinor:\n- Without the algorithms being analyzed in the main paper, the analysis is hard to follow. In my opinion, it would useful to have the algorithm being analysed in the main paper.\n\nQUESTIONS:\n- Aspect ratios are somewhat misleading because they sometimes provide counter-intuitive results. For example, searching for near-duplicates (that is, the distance to the nearest-neighbor is close to zero) are arguably the easiest scenario for most nearest-neighbor search methods -- For branch & bound algorithms, the greedy branching usually finds the near-duplicate very quickly, and obtains the tightest possible bound and is able to prune most aggressively. For hashing based methods, near-duplicates almost always collide, hence the search will always find the nearest-neighbor even with a very small number of hash tables. For graph-based algorithms, as long as the graph is well-connected, the greedy search on the graph will get to the near-duplicate node quite quickly and prune everything else after that in the queue. However, bounds based on aspect ratios appear to tell a different story -- near-duplicates cause the aspect ratio to be arbitrarily high, leading to larger upper-bounds on the runtime complexity or the approximation ratio, making it appear that the search problem is harder with near-duplicates. This appears counter-intuitive. How do the results in this paper handle this issue?\n\n- The aspect ratio $\\Delta$ used in Lemma 3.3 is based on only the set of points $X$, while the aspect ratio $\\Delta$ used in Theorem 3.4 is based on the set of points and the query $q$. So in that case, is it fair to assume that the aspect ratio in Theorem 3.4 **can be** significantly greater than the aspect ratio in Lemma 3.3, especially if (say) $d_{min} \\approx 0$.\n\n- In Lemma 3.3, why is it that the index $i$ of the  Rings are restricted to $i \\in [\\log_2 \\Delta]$?\n\n- In Theorem 3.4, what is the range of interest for $i$? As mentioned in lines 170-171, we can \"asymptotically get to an $\\left(\\frac{\\alpha+1}{\\alpha-1}\\right)$-approximate nearest-neighbour\". However, it is easy to see that, for different values of $i$, different terms will be the dominating term in $\\left( \\frac{\\Delta}{\\alpha^i} + \\frac{\\alpha+1}{\\alpha-1} \\right)$, and it would good to understand which values of $i$ we are interested in, and what the guarantees are for those values.\n\n- Can you please provide a motivation for Theorem 3.5? It is quite well known that $O(\\log n)$ bounds are hard to get, and usually they come with exponential dependence in dimensionality (in the best case, exponential in the intrinsic dimensionality). Even the celebrated LSH guarantees sublinear bounds in $n$ with polynomial dependence on the dimensionality. So it is generally hard to expect that we can just replace the aspect ratio with the number of points.\n\n- For the experiment in Section 4.4, Table 1, are the results averaged over different \"hard instances\"? As in, are multiple problem configurations are selected for each of the Figure 1, 2, 4 and 7, and then the average recall@5 is reported across all such problem instances is reported? This point is not clear even in the discussion in Appendix C.2. Furthermore, it would have been good to understand the performances grouped by the hard instances (at least in the appendix). For example, with algorithms such as NSG, DPG, KGraph, where the average is significantly above 0, it would be good to see whether they performed at this mediocre level for all hard instances, or if they performed really well on some hard instances (for example, ones from Figure 2) while struggling on others (for example, figure 7). This information seems important.\n\nLIMITATIONS:\nI did not find any explicit discussion of limitations by the authors. However, I do not anticipate any potential negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nNearest neighbour (NN) queries select for a given data point the closest point in a set and can be answered exactly in linear time by scanning the whole set. c-approximate Nearest Neighbour (ANN) queries ask for an arbitrary data point that has at most a c times larger distance than the NN. A popular approach in the literature is DiskANN that builds as an index structure a proximity graph that links close-by points together. In this manuscript two variants of DiskANN are analysed.\n\nSTRENGTHS:\nS1. The work seems to fill a gap left by prior works: works proposing the studied methods seem to lack a more extensive theoretical analysis as featured in this work.\n\nS2. Constructed problem instances used in experiments provided in supplementary material (as well as links to code of studied ANN approaches) and details of how various methods have been implemented are extensively documented in the supplementary material.\n\nS3. Combination of empirical and theoretical results to furthen the understanding of the worst-case / average-case performance of ANN methods.\n\nWEAKNESSES:\nW1. Some parts of the paper are a bit confusing.\n\nIt is not clear which point is meant to be the (exact) nearest neighbour (NN) in Figure 1. If \"a\" is the NN, then it would seem that \"p0\" would be an approximate nearest neighbour given the values.\n\nQUESTIONS:\nQ1. What is meant to be the (exact) nearest neighbour (NN) of \"q\" in Figure 1? If the NN of \"q\" is meant to be \"a\", why would \"p0\" not be a c-approximate nearest neighbour with c = (alpha+1)/(alpha-1)?\n\nQ2. If a point that coincides with the query point Q is in the data set, do all c-approximate nearest neighbours (for c > 1) then coincide with the (exact) nearest neighbours? What are the implications of this edge case on the claims in the paper?\n\nLIMITATIONS:\nRelevant limitations seem to be discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}