SlowGuess/NeurIPS_2023_OCR huggingface dataset:
indyk@mit.edu
Haike Xu
haikexu@mit.edu
Graph-based approaches to nearest neighbor search are popular and powerful tools for handling large datasets in practice, but they have limited theoretical guarantees. We study the worst-case performance of recent graph-based approximate nearest neighbor search algorithms, such as HNSW, NSG and DiskANN. For DiskANN, we show that its "slow preprocessing" version provably supports approximate nearest neighbor search query with constant approximation ratio and poly-logarithmic query time, on data sets with bounded "intrinsic" dimension. For the other data structure variants studied, including DiskANN with "fast preprocessing", HNSW and NSG, we present a family of instances on which the empirical query time required to achieve a "reasonable" accuracy is linear in instance size. For example, for DiskANN, we show that the query procedure can take at least  $0.1n$  steps on instances of size  $n$  before it encounters any of the 5 nearest neighbors of the query.
The nearest neighbor search (NN) problem is defined as follows: given a set of  $n$  points  $P$  in a metric space  $(X, D)$ , build a data structure that, given any query point  $q \in X$ , returns  $p \in P$  closest to  $q$ . More generally, given a parameter  $k$ , the data structure should report  $k$  points in  $P$  that are closest to  $q$ . Often, though not always, the metric space is a  $d$ -dimensional vector space with the distance function induced by the Euclidean norm. Since its introduction in the influential book by Minsky and Papert in the 1960s [31], the problem has found a tremendous number of applications in machine learning and computer vision [34]. In most of those applications, the underlying metric is induced by a set of points in a high-dimensional space. Since in this setting worst-case efficient nearest neighbor algorithms are unlikely to exist (see e.g., [2]), various approximate formulations of the problem have been studied extensively. One popular formulation allows the algorithm to return any point  $p' \in P$  whose distance to the query  $q$  is at most  $c$  times the distance between  $q$  and its true nearest neighbor in  $P$ ; such point  $p'$  is called a  $c$ -approximate nearest neighbor. In practice, the accuracy of an approximate data structure is often evaluated by estimating the recall, i.e., the average fraction of the true  $k$  nearest neighbors returned by the data structure.
Over the last few decades, many nearest neighbor data structures have been proposed, both exact and approximate. The most popular classes of data structures are tree algorithms (e.g., kd-trees [4]); locality sensitive hashing [1]; learning-to-cache and product quantization [35, 36]; and metric data structures. The last class includes algorithms that work for point-sets in arbitrary metrics, not just those in  $d$ -dimensional vectors normed spaces. This category can be further subdivided into algorithms based on metric trees [6, 32], divide and conquer algorithms [24, 5, 10], and methods based on greedy search in proximity graphs [3, 28, 15, 21]. See the surveys [8, 27] for further overview of these classes of algorithms.
Several algorithms in the latter class, such as HNSW [28], NSG [15] and DiskANN [21], are widely used in practice. They have been recently shown empirically to provide excellent tradeoffs between accuracy and query speed [27]. Their design is based on approximating theoretical concepts such as Delaunay or Relative Neighborhood Graphs. However, their worst-case performance is not well understood, especially when the dimension is high. For example, the authors of [28] point out that "Further analytical evidence is required to confirm whether the resilience of Delaunay graph approximations generalizes to higher dimensional spaces." This stands in contrast to the older algorithms, e.g., those based on the divide and conquer approach, which come with worst-case performance guarantees. Those algorithms are typically analyzed under the assumption that the input point set  $P$  has low doubling dimension (a measure of the intrinsic dimensionality of the point-set)<sup>2</sup> and provide near-linear space and logarithmic query time bounds, with the big-Oh constants depending on the doubling dimension. This raises the question whether similar bounds can be obtained for the newer algorithms. On the practical side, investigating the worst-case behavior of data structures is important to understand their benefits and limitations.
Our results In this paper we initiate the study of the worst-case performance of the recent metric data structures based on proximity graphs. As in [21], we mostly focus on three popular algorithms and their implementations: HNSW, NSG and DiskANN. (In addition, we present similar results for other graph-based algorithms in the supplementary material section.) We present both upper and lower bounds on their worst-case search times. Our specific contributions are as follows:
- Upper bounds: For one of the data structures studied, namely DiskANN version with "slow preprocessing", we are able to show a provable worst-case upper bound on its performance. Specifically, we show that (a) the greedy search procedure returns an  $\left(\frac{\alpha + 1}{\alpha - 1} + \epsilon\right)$ -approximate neighbor in  $O\left(\log_{\alpha} \frac{\Delta}{(\alpha - 1)\epsilon}\right)$  steps and (b) each step takes at most  $O((4\alpha)^d \log \Delta)$  time. This implies that the overall running time is poly-logarithmic in  $\Delta$  when  $d$  is constant. Here  $\alpha > 1$  denotes a parameter of the DiskANN algorithm (described in Preliminaries, typically set to 2),  $d$  denotes the doubling dimension, while  $\Delta$  denotes the aspect ratio of the input set  $P$ , i.e., the ratio between the diameter and the distance of the closest pair<sup>3</sup>. We also show that our approximation bound is tight, and that the logarithmic dependence of the query time bound on  $\Delta$  cannot be removed.
- Lower bounds: For the other data structure variants studied (NSG, HNSW and DiskANN with "fast preprocessing") we present a family of point sets of size  $n$  for all  $n$  large enough, such that for each  $n$ , the empirical query time required to achieve "reasonable" accuracy is linear in  $n$ . For example, for DiskANN, we show that the query procedure can take at least  $0.1n$  steps before it encounters any of the 5 nearest neighbors of the query. Remarkably, the point sets are relatively simple: they live in a 2-dimensional Euclidean space, and therefore have a constant doubling dimension (see Preliminaries). We use implementations provided by the authors, publicly available on GitHub [16, 29, 22]. Our hard instance examples are available on GitHub at [18].
To the best of our knowledge, these are the first worst-case upper bounds and lower bounds for these data structures.
We emphasize that our results do not contradict the empirical evaluations given in the original papers, or in summary studies such as [27]. Indeed, these algorithms have been demonstrated to be very useful and can be highly effective in practice. Nevertheless, we believe that our results provide important information about the behavior of these algorithms. For example, they demonstrate the importance of validating the quality of answers reported by the algorithms when applying them to new data sets. They also shed light on the types of data sets which result in suboptimal performance of the algorithms.
$^{1}$ E.g., DiskANN has been developed and used at Microsoft and NSG at Alibaba.
2Informally, this means that any subset of points that falls into a ball of radius  $2r$  can be covered using a constant number of balls of radius  $r$ . See Preliminaries for the formal definition and discussion.
3We note running time bounds that depend on  $\log (\Delta)$  versus those that depend on  $\log (n)$  are incomparable. Although  $\Delta$  could be much larger than  $n$ , we observed that its value is typically quite low. For example, the MNIST data set [26, 9] contains  $n = 60,000$  points, while its aspect ratio  $\Delta$  is around 20 when the distances are measured according to the Euclidean norm.
Related work Approximate nearest neighbor search has been studied extensively; the references in the introduction provide some of the main milestones of that rich body of research. In the context of this paper, we also mention [25, 33], which studied theoretical properties of modern graph-based nearest neighbor data structure. However, their focus is on the average-case performance, i.e., under the assumption that the data is generated according to some well-defined distribution, like uniform over the sphere. In contrast, this work is focused on the worst-case behavior of those algorithms, and show that their empirical running time can be high even for relatively simple 2-dimensional data sets.
We denote the underlying metric space by  $(X, D)$ . For any point  $p \in X$  and radius  $r > 0$ , we use  $B(p, r)$  to denote a ball of radius  $r$  centered at  $p$ , i.e.,  $B(p, r) = \{q \in X : D(p, q) \leqslant r\}$ .
Consider a set of points  $P$ . We say that  $P$  has the doubling constant  $C$  if for any ball  $B(p, 2r)$  centered at some  $p \in P$ , the set  $P \cap B(p, 2r)$  can be covered using at most  $C$  balls of radius  $r$ , and  $C$  is the smallest number with this property. Doubling constant is a popular measure of "intrinsic dimensionality" of high dimensional point sets, see e.g., [17, 24, 5]. The value  $\log_2 C$  is called the doubling dimension of  $P$ . Doubling dimension is often used as a measure of the "intrinsic dimensionality" of a data set. It generalizes the "standard" (topological) dimension: for any data set  $P \subset \mathbb{R}^d$  equipped with a metric  $D(p_1, p_2) = \| p_1 - p_2 \|_p$ , the doubling dimension of  $P$  is at most  $O(d)$ . However, the doubling dimension of  $P \subset \mathbb{R}^D$  could be much lower than  $D$ , e.g., if points in  $P$  lie on a low-dimensional manifold. The doubling dimension can be viewed as a finite version of the fractal Hausdorff dimension. Empirical studies (e.g., [11]) showed that the fractal dimension of real data sets is often smaller than their ambient dimension  $D$ .
The following fact is standard and follows from the definition.
Lemma 2.1. Consider a set of points  $P$  with doubling dimension  $d$ . For any ball  $B(p, r)$  centered at some  $p \in P$  and a constant  $k$ , the set  $B(p, r) \cap P$  can be covered by a set of  $m \leqslant O(k^d)$  balls with diameter smaller than  $r / k$ , i.e.  $B(p, r) \cap P \subseteq \bigcup_{i=1}^{m} B(p_i, r / k)$ .
We use  $d$  to denote the doubling dimension of the point set  $P$ ,  $\Delta = \frac{D_{max}}{D_{min}}$  to denote the aspect ratio of the input set  $P$ , where  $D_{max}(D_{min})$  is the maximal (minimal) distance between any pair of vertices in the point set. For two points  $x_u, x_v$  in  $P = \{x_1, \ldots, x_n\}$ , we use  $D(x_u, x_v)$  to denote the distance between them, or sometimes  $D(u, v)$  for simplicity.
3 Analysis of DiskANN
In this section we show bounds on the performance of DiskANN with slow preprocessing.
3.1 DiskANN recap
In this section we give an overview of the DiskANN procedures. For the full description the reader is referred to the original paper [21], or section A (supplementary material).
The DiskANN data structure is based on a directed graph  $G$  over the set  $P$ , i.e., the set of vertices  $V$  of  $G$  are associated with the set of points  $P$ . After the graph is constructed, to answer a given query  $q$ , the algorithm performs search starting from some vertex  $s$ . In what follows we describe the search and insertion procedure in more detail.
The search procedure, GreedySearch  $(s,q,L)$ , has the following parameters: the start vertex  $s$ , the query point  $q$ , and the queue size  $L$ . It performs a best-first-search using a queue of with a bounded length  $L$ , until the  $L$  vertices  $v$  with the smallest value of  $D(v,q)$  seen so far are all scanned. Upon completion, it returns a list of vertices in an increasing distance from  $q$  where the first vertex (or the first  $k$  vertices) are answers for the query. Note that as long as the graph is connected, the procedure runs for at least  $L$  steps. The total running time of the procedure is bounded by the number of steps times the out-degree bound of the graph  $G$ .
The construction of the graph  $G = (V, E)$  is done by a repeated invocation of a procedure called RobustPruning. For any vertex  $v$ , a set of vertices  $U$  (specified later), and parameters  $\alpha > 1$  and  $R$ , RobustPruning  $(v, U, \alpha, R)$  proceeds as follows. First, the set  $U$  is sorted in the increasing order
of the distance to  $v$ . The algorithm traverses this sequence in order. After encountering a new vertex  $u$ , the algorithm deletes all other vertices  $w$  from  $U$  such that  $D(u, w) \cdot \alpha < D(v, w)$ . Finally, the node  $v$  is connected to all vertices in  $U$  that have not been pruned.
The starting point of the DiskANN data structure construction algorithm<sup>4</sup> is the following simple procedure: for each vertex  $v$ , execute RobustPruning(v, U, α, R) with  $U = V$  and  $R = n$ . That is, robust pruning is applied to all vertices in the graph. We refer to this procedure as slow preprocessing, as it can be seen that a naive implementation of this method takes time  $O(n^3)$ . Although the construction time is slow, we show that this construction method provably constructs a graph whose degree depends only logarithmically on the aspect ratio of the graph (assuming constant doubling dimension), and guarantees that the greedy search procedure has polylogarithmic running time. We note that this result is inspired by an observation in [21] about convergence of greedy search in a logarithmic number of steps, though to obtain our result we also need to bound the degree of the search graph and analyze the approximation ratio.
Since the slow-preprocessing-algorithm is too slow in practice, the authors of [21] propose a faster heuristic method to construct the graph  $G$ , which we call fast preprocessing method. At the beginning, the graph  $G$  is initialized to be a random  $R$ -regular graph. Then the construction of the graph  $G = (V, E)$  is done incrementally. The construction algorithms make two passes of the point set in random order. For each vertex  $v$  met, the algorithm computes a set of vertices  $U = GreedySearch(s, x_v, L)$  (for some starting vertex  $s$ ) and then calls the pruning procedure on  $U$ , not  $V$ . That is, it executes RobustPruning  $(v, U, \alpha, R)$ . After pruning is performed, the insertion procedure adds both edges  $(v, u)$  and  $(u, v)$  for all vertices  $u \in U$  output by the pruning procedure. Finally, if the degree of any of  $u \in U$  exceeds a threshold  $R$ , then the set of neighbors of  $u$  is pruned via RobustPruning  $(u, N_{out}(u), \alpha, R)$  as well. This construction method is implemented and evaluated in the paper.
3.2 Analysis: preprocessing
For the sake of simplicity, we let RobustPruning  $(p,V,\alpha)$  be the no-degree-limit version of edge selection, i.e., where  $R = n$ . We first analyze the property of graph constructed by the no-degree-limit pruning, and then show that setting  $R = O((4\alpha)^{d}\log \Delta)$  yields equivalent results.
Definition 3.1 ( $\alpha$ -shortcut reachability). Let  $\alpha \geqslant 1$ . We say a graph  $G = (V, E)$  is  $\alpha$ -shortcut reachable from a vertex  $p$  if for any other vertex  $q$ , either  $(p, q) \in E$ , or there exists  $p'$  s.t.  $(p, p') \in E$  and  $D(p', q) \cdot \alpha \leqslant D(p, q)$ . We say a graph  $G$  is  $\alpha$ -shortcut reachable if  $G$  is  $\alpha$ -shortcut reachable from any vertex  $v \in V$ .
First, we show that the slow preprocessing algorithm constructs a graph that is  $\alpha$ -reachable.
Lemma 3.2 ( $\alpha$ -shortcut reachable). For each vertex  $p$ , if we connect  $p$  to the output of RobustPruning  $(p, V, \alpha)$ , then the graph formed is  $\alpha$ -shortcut reachable.
Proof. By Definition 3.1, we only need to prove that the constructed graph is  $\alpha$ -shortcut reachable from each vertex. Suppose that, for some vertex  $p$ , vertex  $q$  is not connected to  $p$ . Then, in RobustPruning  $(p, V, \alpha)$ , there must have existed a vertex  $p' \in V$  connected to  $p$  s.t.  $D(p', q) \leqslant D(p, q) / \alpha$ .
Next, we show that the graph produced by no degree limit RobustPruning is actually sparse.
Lemma 3.3 (sparsity). For any vertex  $p$ , let  $U = \text{RobustPruning}(p, V, \alpha)$ , then  $|U| \leqslant O((4\alpha)^d \log \Delta)$  where  $d$  is the doubling dimension of the point set  $P$ .
Proof. We use  $Ring(p, r_1, r_2)$  to denote the set of vertices that lie in  $B(p, r_2)$  but not in  $B(p, r_1)$ . We use  $D_{\text{max}}(D_{\text{min}})$  to denote the maximal (minimal) distances between a pair of vertices in the point set  $P$  and by definition  $\Delta = \frac{D_{\text{max}}}{D_{\text{min}}}$ . For each  $i \in [\log_2 \Delta]$ , we consider  $Ring(p, D_{\text{max}} / 2^i, D_{\text{max}} / 2^{i-1})$  separately. We cover  $Ring(p, D_{\text{max}} / 2^i, D_{\text{max}} / 2^{i-1})$  using balls with radius  $D_{\text{max}} / \alpha 2^{i+1}$  by Lemma 2.1. The number of balls required is bounded by  $O((4\alpha)^d)$ . Because every two points in the same ball have distance at most  $D_{\text{max}} / \alpha 2^i$  from each other, and this value is  $\alpha$
4See the discussion in Section 2.2 of [21].
times smaller than their distance lower bound to  $p$ , at most one of them will remain after performing RobustPruning  $(p, V, \alpha)$ . Therefore, the number of vertices remain after performing RobustPruning  $(p, V, \alpha)$  is upper bounded by  $O((4\alpha)^d \log \Delta)$ .
3.3 Analysis: query procedure
We now show that if the graph is constructed using the slow indexing algorithm analyzed in the previous section, then GreedySearch  $(s,q,1)$  starting from any vertex  $s$  returns an  $\left(\frac{\alpha + 1}{\alpha - 1}\right)$ -approximate nearest neighbor from query  $q$  in a logarithmic number of steps.
Theorem 3.4. Let  $G = (V, E)$  be an  $\alpha$ -shortcut reachable graph constructed using the slow preprocessing method. Consider GreedySearch(s, q, L) starting with any vertex  $s \in V$  and  $L = 1$  (i.e., the algorithm performs no back-tracking) answering query  $q$ . The algorithm finds an  $\left(\frac{\alpha + 1}{\alpha - 1} + \epsilon\right)$ -approximate nearest neighbor in  $O\left(\log_{\alpha} \frac{\Delta}{(\alpha - 1)\epsilon}\right)$  steps.
Proof. Let  $a$  be the nearest neighbor of  $q$ ,  $v_{i}$  be the  $i$ -th scanned vertex,  $d_{i} = D(v_{i},q)$ , with approximate ratio  $c_{i} = \frac{d_{i}}{D(a,q)}$ . We use  $\Delta = \frac{D_{max}}{D_{min}}$  to denote the aspect ratio of the vertex set.
We know that the distance between  $v_{i}$  and  $a$  is no more than  $d_{i} + D(a,q)$  by triangle inequality, and because  $G$  is  $\alpha$ -shortcut reachable, each  $v_{i}$  is either connected to  $a$ , or  $v_{i}$  is connected to a vertex  $v'$  whose distance to  $a$  is shorter than  $\frac{d_{i} + D(a,q)}{\alpha}$ . In best-first-search, the next scanned vertex  $v_{i+1}$  should have distance from  $q$  no farther than  $v'$ , which is at most  $\frac{d_{i} + D(a,q)}{\alpha} + D(a,q)$ . By induction:
$$
d _ {i} \leqslant \frac {D (s , q)}{\alpha^ {i}} + \frac {\alpha + 1}{\alpha - 1} D (a, q) \tag {1}
$$
Consider the following three cases, depending on the value of  $D(a,q)$ :
Case (1):  $D(s,q) > 2D_{max}$ . In this case we have  $D(a,q) > D(s,q) - D(a,s) > D(s,q) - D_{max} > D(s,q)/2$ . Plugging this into inequality 1 gives us  $c_i = \frac{d_i}{D(a,q)} \leqslant \frac{D(s,q)}{\alpha^i D(a,q)} + \frac{\alpha + 1}{\alpha - 1} \leqslant \frac{2}{\alpha^i} + \frac{\alpha + 1}{\alpha - 1}$ . Therefore, for any  $\epsilon > 0$ , we get a  $\left(\frac{\alpha + 1}{\alpha - 1} + \epsilon\right)$ -approximate nearest neighbor in  $\log_{\alpha} \frac{2}{\epsilon}$  steps.
Case (2):  $D(s,q) \leqslant 2D_{max}$  and  $D(a,q) \geqslant \frac{\alpha - 1}{4(\alpha + 1)} D_{min}$ . By inequality 1, the algorithm reaches an  $(\frac{\alpha + 1}{\alpha - 1} + \epsilon)$ -approximate nearest neighbor when  $\frac{D(s,q)}{\alpha^i} < \epsilon D(a,q)$ . Substituting the value of  $D(s,q)$  and  $D(a,q)$  with corresponding upper and lower bound, we obtain that the number of steps needed is  $\log_{\alpha} \frac{8(\alpha + 1)\Delta}{(\alpha - 1)\epsilon} \leqslant O\left(\log_{\alpha} \frac{\Delta}{(\alpha - 1)\epsilon}\right)$
Case (3):  $D(s,q) \leqslant 2D_{max}$  and  $D(a,q) < \frac{\alpha - 1}{4(\alpha + 1)} D_{min}$ . Suppose at step  $i$ ,  $d_i > d(a,q)$  is not the nearest neighbor. Here we obtain a new lower bound for  $d_i$ . We know  $d(v_i,a) > D_{min}$ ,  $d(v_i,q) > d(a,q)$ , and  $d(a,q) < D_{min}$ . By triangle inequality, we have  $d_i = d(v_i,q) > d(v_i,a) / 2 > D_{min} / 2$ . Combining this with inequality 1, we obtain that if  $v_i$  is not the exact nearest neighbor, it must satisfy  $\frac{D_{min}}{2} \leqslant d_i \leqslant \frac{D(s,q)}{\alpha^i} + \frac{\alpha + 1}{\alpha - 1} D(a,q) \leqslant \frac{2D_{max}}{\alpha^i} + \frac{D_{min}}{4}$ . This can only happen when  $i \leqslant \log_\alpha 8\Delta$ . Therefore, the algorithm reaches the exact nearest neighbor in  $O(\log_\alpha \Delta)$  steps.
We note that similar neighbor selection strategies in the greedy search procedure are also used in HNSW [28] and NSG [15]. However, we cannot prove query performance guarantees similar to the ones above, as those algorithms use  $\alpha = 1$ .
3.4 A tight convergence rate lower bound for DiskANN
In this section we show that the logarithmic dependence of the query time bound on  $\Delta$  is unavoidable, i.e., we cannot replace  $\log \Delta$  with  $\log n$ . The proof is deferred to appendix B.
Theorem 3.5. For any choice of  $\alpha > 1$ , there exists a set of  $n = 2k - 1$  points on a one-dimensional line with aspect ratio  $\Delta = O(\alpha^n)$ , and a query  $q$ , such that after the slow preprocessing version of DiskANN, greedy search must scan at least  $\Omega (\log \Delta)$  or  $\Omega (n)$  vertices before reaching an  $O(1)$ -approximate nearest neighbor of  $q$ .
3.5 A tight approximation lower bound for DiskANN
In Theorem 3.4, we know that the slow preprocessing version of DiskANN algorithm can asymptotically get to an  $\frac{\alpha + 1}{\alpha - 1}$ -approximate nearest neighbor. Here, we provide a simple instance showing that this approximation ratio is tight.
Figure 1:  $\frac{\alpha + 1}{\alpha - 1}$ -approximation ratio lower bound instance for slow preprocessing version of DiskANN. The whole instance can be embedded in a two-dimensional grid using  $l_{1}$  distance, and therefore has a constant doubling dimension. Black dots (solid or hollow) are points in the database, the red dot is the query point. Grids are only used to show the layout structure. Please refer to Section 3.5 for detailed instance description.
Theorem 3.6. For any  $\alpha > 1$ , there exists a set of  $n + 2$  points in the two-dimensional plane under  $l_{1}$  distance where the slow preprocessing version of DiskANN will scan at least  $n$  vertices before getting to a vertex with an approximation ratio smaller than  $\frac{\alpha + 1}{\alpha - 1}$ -approximate nearest neighbor.