SPANN: Highly-efficient Billion-scale Approximate Nearest Neighborhood Search

billion-scale , vector search , inverted index solution 1 Introduction . Vector nearest neighbor search has played an important role in information retrieval area , such as multimedia search and web search , which provides relevant results by searching vectors with minimum distance to the query vector . Exact solutions for K-nearest neighbor search [ 49 , 40 ] are not applicable in big data scenario due to substantial computation cost and high query latency . Therefore , researchers have proposed many kinds of approximate nearest neighbor search ( ANNS ) algorithms in the literature [ 11 , 18 , 38 , 10 , 14 , 31 , 34 , 13 , 29 , 21 , 16 , 26 , 43 , 42 , 33 , 44 , 37 , 32 , 19 , 27 , 9 , 12 , 39 , 50 , 20 , 36 ] . However , most of the algorithms mainly focus on how to do low latency and high recall search all in memory with offline pre-built indexes . When targeting to the super large scale vector search scenarios , such as web search , the memory cost will become extremely expensive . There is an increasing request for the hybrid ANNS solutions that use small memory and inexpensive disk to serve the large scale datasets . ∗Corresponding author . †Work done while at Microsoft . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . There are only a few approaches working on the hybrid ANNS solutions , including DiskANN [ 39 ] and HM-ANN [ 36 ] . Both of them are graph based solutions . DiskANN uses Product Quantization ( PQ ) [ 25 ] to compress the vectors stored in the memory while putting the navigating spread-out graph along with the full-precision vectors on the disk . When a query comes , it traverses the graph according to the distance of quantized vectors and then reranks the candidates according to distance of the full-precision vectors . HM-ANN leverages the heterogeneous memory by placing pivot points in the fast memory and navigable small world graph in the slow memory without data compression . However , it consumes more than 1.5 times larger fast memory than DiskANN . Moreover , the slow memory is still much expensive than disk . Therefore , due to the cheap serving cost , high recall and low latency advantages of DiskANN , it has become the start-of-the-art for indexing billion-scale datasets . In this paper , we argue that the simple inverted index approach can also achieve state-of-the-art performance for large scale datasets in terms of recall , latency and memory cost . We propose SPANN , a simple but surprising efficient memory-disk hybrid vector indexing and search system , that follows the inverted index methodology . SPANN only stores the centroid points of the posting lists in the memory while putting the large posting lists in the disk . We guarantee both low latency and high recall by greatly reducing the number of disk accesses and improving the quality of posting lists . In the index-building stage , we use a hierarchical balanced clustering method to balance the length of posting lists and expand the posting list by adding the points in the closure of the corresponding clusters . In the search stage , we use a query-aware scheme to dynamically prune the access of unnecessary posting lists . Experiment results demonstrate that SPANN is more than two times faster than the state-of-the-art disk-based ANNS algorithm DiskANN to reach the same recall quality 90 % with same memory cost in three billion-scale datasets . It can reach 90 % recall @ 1 and recall @ 10 in just around one millisecond with only about 10 % of original memory cost . SPANN has already been deployed into Microsoft Bing to support hundreds of billions scale vector search . 2 Background and Related Work . Given a set of data vectors X ∈ Rn×m ( the data set contains n vectors with m-dimensional features ) and a query vector q ∈ Rm , the goal of vector search is to find a vector p∗ from X , called nearest neighbor , such that p∗ = argminp∈X Dist ( p , q ) . Similarly , we can defineK-nearest neighbors . Due to the substantial computation cost and high query latency of the exhaustive search , ANNS algorithms are designed to speedup the search for the approximiate K-nearest neighbors in a large dataset in an acceptable amount of time . Most of the ANNS algorithms in the literature mainly focus on the fast high-recall search in the memory , including hash based methods [ 14 , 24 , 47 , 48 , 45 , 46 , 51 ] , tree based methods [ 11 , 31 , 44 , 33 ] , graph based methods [ 21 , 16 , 43 , 32 ] , and hybrid methods [ 42 , 12 , 23 , 22 ] . However , with the explosive growth of the vector scale , the memory has become the bottleneck to support large scale vector search . There are only a few approaches working on the ANNS solutions for billon-scale datasets to minimize the memory cost . They can be divided into two categories : inverted index based and graph based methods . The inverted index based methods , such as IVFADC [ 26 ] , FAISS [ 27 ] and IVFOADC+G+P [ 9 ] , split the vector space into K Voronoi regions by KMeans clustering and only do search in a few regions that are closed to the query . To reduce the memory cost , they use vector quantization , e.g . Product Quantization ( PQ ) [ 25 ] , to compress the vectors and store them in the memory . The inverted multi-index ( IMI ) [ 7 ] also uses PQ to compress vectors . It splits the feature space into multiple orthogonal subspaces and constructs a separate codebook for each subspace . The full feature space is produced as a Cartesian product of the corresponding subspaces . Multi-LOPQ [ 28 ] uses locally optimized PQ codebook to encode the displacements in the IMI structure . GNO-IMI [ 8 ] optimizes the IMI by using non-orthogonal codebooks to produce the centroids . Although they can cut down the memory usage to less than 64GB for one billion 128 dimensional vectors , the recall @ 1 is very low ( only around 60 % ) due to lossy data compression . Although they can achieve better recall by returning 10 to 100 times more candidates for further reranking , it is often not acceptable in many scenarios . The graph based methods include DiskANN [ 39 ] and HM-ANN [ 36 ] . Both of them adopt the hybrid solution . DiskANN also stores the PQ compressed vectors in the memory while storing the navigating spread-out graph along with the full-precision vectors on the disk . When a query comes , it traverses the graph using best-first manner according to the distance of quantized vectors and then reranks the candidates according to distance of the full-precision vectors . Similarly , it uses the lossy data compression which will influence the recall quality even though full-precision vector reranking can help retrieve some missing candidates back . The high-cost random disk accesses limit the number of graph traverse and candidate reranking . HM-ANN leverages the heterogeneous memory by placing pivot points promoted by the bottom-up phase in the fast memory and navigable small world graph in the slow memory without data compression . However , it will lead to more than 1.5 times larger fast memory consumption . Moreover , the slow memory is still much expensive than disk and may be not available in some platforms . The theoretical analysis of the limits and the benefits of the graph based methods are given in [ 35 ] . 3 SPANN . In this paper , we propose SPANN , a simple but efficient vector indexing and search system , that follows the inverted index methodology . Different from previous inverted index based methods that leverage the lossy data compression to reduce the memory cost , SPANN adopts a simple memory-disk hybrid solution . Index structure : The data vectors X are divided into N posting lists { X1 , X2 , · · · , XN } , X1 ∪ X2 ∪ ... ∪XN = X3 . The centroids of these posting lists , c1 , c2 , · · · , cN , are stored in the memory as the fast coarse-grained index that point to the location of the corresponding posting lists in the disk . Partial search : When a query q comes , we find the K closest centroids , { ci1 , ci2 , . . . , ciK } , K N , and load the vectors in the posting lists Xi1 , Xi2 , · · · , XiK that correspond to the closest K centroids into memory for further fine-grained search . 3.1 Challenges . Posting length limitation : Since all the posting lists are stored in the disk , in order to reduce the disk accesses , we need to bound the length of each posting list so that it can be loaded into memory in only a few disk reads . This requires us to not only partition the data into a large number of posting lists but also balance the length of posting lists . This is very difficult due to the substantial high clustering cost and the balance partition problem itself . The imbalanced posting lists will lead to high variance of query latency especially when posting lists are stored in the disk . Boundary issue : The nearest neighbor vectors of a query q may locate in the boundary of multiple posting lists . Since we only search a small number of relevant posting lists , some true neighbors of q that located in other posting lists will be missing ( Illustrated in Figure 1 ) . If red points are only represented by the centroid of blue posting list , they will be missing in the nearest neighbor search of yellow point . 3For convenience , we use X to denote both the matrix and the vector set . Diverse search difficulty : We find that different queries may have different search difficulty . Some queries only need to be searched in one or two posting lists while some queries require to be searched in a large number of posting lists ( Illustrated in Figure 2 ) . If we search the same number of posting lists for all queries , it will result in either low recall or long latency . All of the above challenges are the reasons why all of previous inverted index approaches adopt lossy data compression solution that stores all the compressed vectors and the posting lists in the memory . 3.2 Key techniques to address the challenges . In this paper , we introduce three key techniques that solve the above challenges to enable the memorydisk hybrid solution . In the index-building stage , we firstly limit the length of the posting lists to effectively reduce the number of disk accesses for each posting list in the online search . Then we improves the quality of the posting list by expanding the points in the closure of the corresponding posting lists . This increases the recall probability of the vectors located on the boundary of the posting lists . In the search stage , we propose a query-aware scheme to dynamically prune the access of unnecessary posting lists to ensure both high recall and low latency . The detail design of each technique will be introduced in the following sections . 3.2.1 Posting length limitation . Limiting the length of posting lists means we need to partition the data vectors X into a large number of posting lists X1 , X2 , · · · , XN . Balancing the length of posting lists means we need to minimize the variance of the posting length ∑N i=1 ( |Xi| − |X|/N ) 2 . To address the posting length balance problem , we can leverage the multi-constraint balanced clustering algorithm [ 30 ] to partition the vectors evenly into multiple posting lists : min H , C ||X−HC||2F + λ N∑ i=1 ( |X|∑ l=1 hli − |X|/N ) 2 , s.t . N∑ i=1 hli = 1 . ( 1 ) where C ∈ RN×m is the centroids , H ∈ { 0 , 1 } |X|×N represents the cluster assignment , ∑|X| l=1 hli is the number of vectors assigned to the i-th cluster ( i.e . |Xi| ) and λ is a trade-off hyper parameter between clustering and balance constraints . However , we find that when the vector number |X| and the partition number N are very large , directly using multi-constraint balanced clustering algorithm can not work due to the difficulty of large N -partition balanced clustering problem and the extremely high clustering cost . Therefore , we introduce a hierarchical multi-constraint balanced clustering technique ( Figure 3 ) to not only reduce the clustering time complexity from O ( |X| ∗m ∗ N ) to O ( |X| ∗m ∗ k ∗ logk ( N ) ) ( k is a small constant ) but also balance the length of posting lists . We cluster the vectors into a small number ( i.e . k ) of clusters iteratively until each posting list contains limit number of vectors . By using this technique , we can greatly reduce not only the length of each posting list ( disk accesses ) but also the index build cost . Moreover , since the number of centroids is very large , finding the nearest posting lists for a query needs to consume large computation cost . In order to make the navigating computation more meaningful , we replace the centroid with the vector that is closest to the centroid to represent each posting list . Then the wasted navigating computation is transformed to the distance computation for a subset of real candidates . What ’ s more , in order to quickly find a small number of nearest posting lists for a query , we create a memory SPTAG [ 12 ] ( MIT license ) index for all the vectors that represent the centorids of the posting lists . SPTAG constructs space partition trees and a relative neighborhood graph as the vector index which can speedup the nearest centroids search to sub-millisecond response time .

Summary: SPANN is a hybrid disk/RAM vector indexing method.   SPANN uses a hierarchical clustering (with some tricks) to split the dataset into buckets that are stored uncompressed on disk. At search time, the buckets corresponding to the nearest centroids to the query (again with some tricks) are visited to retrieve the nearest neighbors.   The indexing-time tricks are intended to balance the bucket sizes and reduce assignment ambiguity with some level of multiple storage.   At search time, the number of buckets to visit is tuned based on the distance of the query to the centroids.   There is also a section about a distributed version of the method.