Title: SOAR: Improved Indexing for Approximate Nearest Neighbor Search
Venue: NeurIPS 2023
Accepted: yes
Keywords: ann

quantization

mips

nearest neighbor search

retrieval
Abstract: This paper introduces SOAR: **S**pilling with **O**rthogonality-**A**mplified **R**esiduals, a novel data indexing technique for approximate nearest neighbor (ANN) search. SOAR extends upon previous approaches to ANN search, such as spill trees, that utilize multiple redundant representations while partitioning the data to reduce the probability of missing a nearest neighbor during search. Rather than training and computing these redundant representations independently, however, SOAR uses an *orthogonality-amplified residual* loss, which optimizes each representation to compensate for cases where other representations perform poorly. This drastically improves the overall index quality, resulting in state-of-the-art ANN benchmark performance while maintaining fast indexing times and low memory consumption.

SOAR: Improved Indexing for Approximate Nearest Neighbor Search

Abstract

This paper introduces SOAR: S pilling with O rthogonalityA mplified R esiduals, a novel data indexing technique for approximate nearest neighbor (ANN) search. SOAR extends upon previous approaches to ANN search, such as spill trees, that utilize multiple redundant representations while partitioning the data to reduce the probability of missing a nearest neighbor during search. Rather than training and computing these redundant representations independently, however, SOAR uses an orthogonality-amplified residual loss, which optimizes each representation to compensate for cases where other representations perform poorly. This drastically improves the overall index quality, resulting in state-of-the-art ANN benchmark performance while maintaining fast indexing times and low memory consumption.

1 Introduction

The k -nearest neighbor search problem is defined as follows: we are given an n -item dataset X ∈ R n × d composed of d -dimensional vectors, and for a query q ∈ R d , we would like to return the k vectors in X closest to q . This problem naturally arises from a number of scenarios that require fast, online retrieval from vector databases; such applications include recommender systems [7], image search [14], and question answering [10], among many others.

While nearest neighbor search may be easily implemented with a linear scan over the elements of X , many applications of nearest neighbor search utilize large datasets for which a brute-force approach is computationally intractable. The rapid development of deep learning and embedding techniques has been an especially strong driver for larger k -nearest neighbor datasets. For instance, multiple recent large language model (LLM) works incorporate external information by using k -nearest neighbors to retrieve from longer contexts [17] or large text corpora [5]. While datasets from typical applications a decade ago (e.g. Netflix) had sizes of around 1 million [15], the standard evaluation datasets from big-ann-benchmarks [16] all have 1 billion vectors with hundreds of dimensions.

These large datasets, in conjunction with the curse of dimensionality , a phenomenon which often makes it impossible to find the exact nearest neighbors without resorting to a linear scan, has led to a focus on approximate nearest neighbor (ANN) search, which can trade off a small search accuracy loss for a significant increase in search throughput.

A number of indexing schemes proposed for ANN search, including spill trees [12], assign datapoints to multiple portions of the index, such that the overall index provides a replicated, non-disjoint ('spilled") view of the dataset. These algorithms have used partitioning schemes amenable to random analysis, so that the replication factor can be provably shown to exponentially reduce the probability of missing a nearest neighbor. However, partitioning high-dimensional data is difficult, even without the constraint of doing so with a method amenable to randomized analysis; such constrained partitioning techniques have led to inferior quality indices that have not yielded good compute-accuracy tradeoffs.

On the other hand, another family of approaches, those leveraging vector quantization (VQ), have demonstrated excellent empirical ANN performance, but little research has been done exploring the use of multiple randomized initializations of VQ indices to further increase ANN search efficiency. This is partly due to the fact that multiple VQ indices initialized through different random seeds do not have strong statistical guarantees and cannot be proven independent in their failure probabilities. This paper bridges the gap between these two approaches to ANN:

We demonstrate the weaknesses of current VQ-based ANN indices and illustrate how multiple VQ indices may be used in conjunction to improve ANN search efficiency.

Weshowthat the naive training of multiple VQ indices for a single dataset leads to correlation in failure probabilities among the indices, limiting ANN search efficiency uplift, and present spilling with orthogonality-amplified residuals (SOAR) to ameliorate this correlation.

We benchmark SOAR and achieve state-of-the-art performance, outperforming standard VQ indices, spilled VQ indices trained without SOAR, and all other approaches to ANN search that were submitted to the benchmark.

2 Preliminaries and Notation

2.1 Maximum inner product search (MIPS)

SOAR applies to the subclass of nearest neighbor search known as maximum inner product search, defined as follows for a query q and a dataset X :

Many nearest neighbor problems arise in the MIPS space naturally [6], and a number of conversions exist [4] from other commonly used ANN search metrics, such as Euclidean and cosine distance, to MIPS, and vice versa. We measure MIPS search accuracy using recall@ k , defined as follows: if our algorithm returns the set S of k candidate nearest neighbors, its recall@ k equals | MIPS k ( q, X ) ∩ S | /k . Additionally, we introduce the following notation to assist with MIPS analysis:

The max inner product's RANK is 1; assuming no ties, RANK ( q, v, X ) ∈ [1 , k ] for v ∈ MIPS k ( q, X ) .

2.2 Vector quantization (VQ)

Vector quantization can be leveraged to construct data structures that effectively prune the ANN search space; these data structures are commonly known as inverted file indices (IVF) or k-means trees. Vector-quantizing a dataset X produces two outputs:

C ∈ R c × d , the codebook containing the c partition centers.

π ( v ) : R d ↦→{ 1 , . . . , c } , the partition assignments that map each vector in X to one of the partition centers in C . Oftentimes, π ( v ) is defined as arg min i ∈{ 1 ,...,c } ‖ v -C i ‖ 2 , although other assignment functions may also be used.

We can then construct an inverted index over π ; for each partition i , we store the set of datapoint indices belonging to that partition: { j | π ( X j ) = i } . Then, instead of computing MIPS k ( q, X ) directly, we may first compute MIPS k ′ ( q, C ) , which is much faster because |C| glyph[lessmuch] |X| . We can then use our inverted index data structure to further evaluate the datapoints within the top k ′ partitions.

2.2.1 The k-means recall (KMR) curve

The effectiveness of the VQ-based pruning approach depends on the rank of the partitions that MIPS k ( q, X ) are in. If these partitions rank very well, we can set k ′ very low, and search through few partitions while still achieving great MIPS recall. If these partitions rank poorly, however, the algorithm will have to spend lots of compute searching through many partitions in order to find the nearest neighbors. We may quantify this effectiveness using the k-means recall (KMR) curve (named

because VQ indices are commonly trained through k-means, although KMR can be computed for any VQ index, not just those trained via k-means), defined as follows:

The KMR curve for a given query sample Q , dataset X , and VQ index ( C , π ) quantifies the proportion of MIPS nearest neighbors present in the top t VQ partitions, over varying t . The KMR curve is nondecreasing, with KMR k (0) = 0 and KMR k ( |C| ) = 1 . A KMR curve that more quickly approaches 1 indicates superior index quality.

3 Method

The key insight of our work is the use of a novel loss function to assign a datapoint x to multiple VQ partitions, such that these additional partitions effectively recover x in the pathological cases when x is a nearest neighbor for a query that x 's original VQ partition handles poorly. These partitions that include x work together synergistically to provide greater ANN search efficiency than any single partition could alone. Below, we describe the motivations behind our new loss and this multiple assignment. The supplementary materials contains the source code to generate this section's plots.

3.1 Search difficulty and quantized score error

For a datapoint x , define r as the partitioning residual , equal to x -C π ( x ) . The partition center C π ( x ) is the quantized form of x , so the difference between the exact and the quantized inner product scores is

which we denote the quantized score error . Consider the case when x is a nearest neighbor for the query: x ∈ MIPS k ( q, X ) . In this scenario, we would like RANK ( q, C π ( x ) , C ) to be low, so that the algorithm may search just the top few partitions and find the nearest neighbor x . A high RANK would lead to increased search difficulty, because the algorithm would have to search more partitions and therefore do more work to find x .

Given that 〈 q, C π ( x ) 〉 = 〈 q, x 〉 - 〈 q, r 〉 and that, by definition, 〈 q, x 〉 is large when x ∈ MIPS k ( q, X ) , we know that 〈 q, C π ( x ) 〉 will be small (leading to greater search dif-

0

500

1000

1500

2000

Rank(

q

,

C

(

x

)

,

C

)

0.0

0.2

0.4

0.6

Mean

q

,

r

ficulty) when 〈 q, r 〉 is highly positive. Indeed, this can be confirmed empirically; in Figure 1 we plot the mean 〈 q, r 〉 as a function of RANK ( q, C π ( x ) , C ) for all query-neighbor pairs ( q, x ) in the Glove-1M dataset. The more difficult-to-find pairs have, on average, notably higher 〈 q, r 〉 .

SOAR increases search efficiency in these situations when x ∈ MIPS k ( q, X ) and 〈 q, r 〉 is high.

3.2 Quantized score error decomposition

By the definition of the inner product,

where θ is the angle formed between the query and the partitioning residual. Without loss of generality, we may assume that ‖ q ‖ = 1 without any effect on the ranking of MIPS nearest neighbors. The two contributors to a highly positive 〈 q, r 〉 are therefore a large ‖ r ‖ and cos θ being near unity; reducing 〈 q, r 〉 requires targeting either, or both, of these contributors.

SOAR targets cos θ , because that term is both easier to reduce and has greater impact on 〈 q, r 〉 :

The VQ training loss already aims to minimize E [ ∥ ∥ x -C π ( x ) ∥ ∥ 2 ] = E [ ‖ r ‖ 2 ] , so further reductions in ‖ r ‖ are difficult. In contrast, cos θ is not directly optimized for, making it more amenable to reduction.

cos θ is typically much more strongly correlated with 〈 q, r 〉 than ‖ r ‖ is, so its reduction has greater impact on search difficulty. Qualitatively, this stronger correlation can be explained by the far greater range of values cos θ may take on, compared to ‖ r ‖ . The latter's values are concentrated between 0 and ‖ x ‖ , with not a very large ratio between the largest and smallest value. In comparison, cos θ can take on any value in the range [ -1 , 1] ; only this large multiplicative dynamic range can explain the variance in 〈 q, r 〉 . Empirically, this can be seen demonstrated on Glove-1M in Figure 2 below.

0.6

Corr=0.986

F9:0

Corr=0.457

6

Error

Error

0.4

0.4

0.2

0.2

S

S

e

e

0.0

Z

0.0

nti

t

n

-0.2

-0.2

-0.4

-0.2

0.0

0.2

0.4

0.6

0.2

0.4

0.6

0.8

1.0

1.2

Query-Residual Anglecosθ

Partitioning Residual Norm Ill

3.3 Spilled VQ assignment

We may attempt to mitigate the increase in search difficulty presented by high cos θ by assigning each datapoint x to a second VQ partition π ′ ( x ) , resulting in a second partitioning residual r ′ that forms an angle θ ′ with the query. We denote this second assignment a spilled assignment .

This gives our ANN algorithm a 'second chance"; assuming θ and θ ′ are independently distributed, the situations where cos θ is high, thereby leading to high 〈 q, r 〉 and greater search difficulty, are unlikely to also have a high cos θ ′ . Alow cos θ ′ should lead to a low 〈 q, r ′ 〉 and should allow the ANN algorithm to find x fairly efficiently. This idea can be extended to further (>2) assignments as well.

This approach leads to some implementation intricacies, as discussed in Section 3.5, but for now we focus on problems concerning theory when the spilled assignment π ′ ( x ) is chosen so as to minimize || r ′ || 2 .

This proposed approach fails when there is correlation between cos θ and cos θ ′ , shown to the extreme in Figure 3. In this figure, if π ( x ) = 1 , we may pick π ′ ( x ) = 2 because C 2 is the second-closest centroid to x . However, C 2 is

x

C

1

C

2

C

3

glyph[epsilon1]r

r

glyph[negationslash]

collinear with C 1 and x , resulting in θ = θ ′ . This leads to C 2 acting as a strictly worse version of C 1 for quantizing x : 〈 q, r ′ 〉 = (1 + glyph[epsilon1] ) 〈 q, r 〉 . The choice π ′ ( x ) = 3 would've been more effective, despite ‖ x -C 3 ‖ > ‖ x -C 2 ‖ , due to C 3 's position giving a θ ′ = θ .

Experiments show that more than a theoretical concern, this is also a real-world issue. On the Glove-1M dataset, if we choose π ( x ) and π ′ ( x ) to be the closest and second-closest centroids by Euclidean distance, respectively, to x , we find a noticeable correlation between cos θ and cos θ ′ , shown in Figure 4a. An even stronger correlation occurs between two VQ indices trained separately with different random seeds (Figure 4b). This correlation diminishes the benefit of spilled assignment, and reducing this correlation is critical to improving ANN search performance.

0.6-

0.4

S

0.2

0.0

-0.2

Corr=0.485

-0.4

-0.2

0.0

0.2

0.4

0.6

COSθ

0.6

0.4

S

2

0.2

0.0

-0.2

Corr=0.786

-0.4

-0.4

-0.2

0.0

0.2

0.4

0.6

COSθ1

3.4 Spilling with orthogonality-amplified residuals

We now derive spilling with orthogonality-amplified residuals (SOAR), a novel VQ assignment loss that directly confronts the problem of correlated query-residual angles. To derive this loss, assume that the VQ centroids C and assignments π are fixed. The goal of further spilled assignments π ′ is to achieve low quantized score error 〈 q, r ′ 〉 specifically when the original spilled assignment has high cos θ , leading to high 〈 q, r 〉 . In cases where cos θ is low, the original assignments already achieve low quantized score error, so in such cases, it is less important to also have low 〈 q, r ′ 〉 .

We now modify our assignment loss function to reflect this emphasis; rather than selecting c ′ ∈ C to minimize E q ∈Q [( 〈 q, x 〉 - 〈 q, c ′ 〉 ) 2 ] = E q ∈Q [ 〈 q, r ′ 〉 2 ] , we now also add a weighting term for cos θ :

where the weight function w ( · ) : R ↦→ R ≥ 0 should be chosen to give greater emphasis to higher cos θ . To utilize this loss function in VQ assignment, we must evaluate the expectation, which leads to the following result:

Theorem 3.1. For the weight function w ( t ) = | t | λ and a query distribution Q that is uniformly distributed over the unit hypersphere,

Proof.

By leveraging Theorem 3.1, we may efficiently perform spilled VQ assignment using our new loss by computing the squared glyph[lscript] 2 distance to each centroid, adding a penalty term for parallelism between the original residual r and the candidate residual r ′ , and taking the arg min among all centroids. We present the following statements to help develop intuition about the SOAR loss:

Corollary 3.1.1. For the uniform weight function w ( t ) = 1 = | t | 0 , the SOAR loss is equivalent to standard Euclidean assignment: L ( r ′ , r, Q ) ∝ ‖ r ′ ‖ 2 .

The parameter λ controls how much the spilled assignment π ′ should prioritize quantization performance when the original assignment performs poorly, relative to general quantization performance. As λ is increased, the former is further emphasized; at λ = 0 , only the latter is considered, leading to standard Euclidean assignment. See Figure 9 in Experiments for a visualization of this tradeoff.

Corollary 3.1.2. For a fixed ‖ r ′ ‖ , L is minimized when r and r ′ are orthogonal, in which case λ ‖ proj r r ′ ‖ 2 = 0 and L ( r ′ , r, Q ) ∝ ‖ r ′ ‖ 2 .

Our technique's name originates from the observation that residuals are encouraged to be orthogonal.

Lemma 3.2. ‖ proj r r ′ ‖ = ‖ r ′ ‖ · ρ 〈 q,r 〉 , 〈 q,r ′ 〉 , where ρ is the Pearson correlation coefficient, and the correlation is computed over q uniformly distributed over the hypersphere.

The above lemma shows that in addition to the weighted quantization error derivation in Equation 2, we may also interpret the SOAR loss as adding a penalization term for correlation in quantized score error, scaled by the magnitude of the quantization error.

Proof.

and by symmetry over the hypersphere, E [ 〈 q, r 〉 ] = E [ 〈 q, r ′ 〉 ] = 0 so our expression simplifies to

It's well known that for uniformly distributed, unit-norm q ∈ R d that E [ qq T ] = I d /d , where I d is the d -dimensional identity matrix, and that σ 〈 q,v 〉 = ‖ v ‖ / √ d for any fixed v . We leverage this to get

3.5 Implementation considerations

SOAR comes with some memory overhead relative to a non-spilled VQ MIPS index, due to its multiple assignments causing some duplication of data. The overhead is quite negligible because only the product-quantized (PQ) [9] datapoint representation, not the datapoint's highest-bitrate representation (usually int8 or float32), is duplicated. Since PQ is much more compressed than int8 or float32, duplicating the PQ data for each of a datapoint's spilled assignments only marginally increases memory footprint, as illustrated in Figure 5. This can be quantified analytically:

SOAR increases memory consumption by 4 + d 2 s bytes/datapoint, assuming 16 centers per subspace (usually chosen for amenability to SIMD) and s dimensions per PQ subspace.

The datapoint's original assignment already contained a PQ representation occupying 4+ d 2 s bytes, and its highest-bitrate representation requires d bytes (int8) or 4 d bytes (float32).

When the ANN index's highest-bitrate datapoint representation is int8 or float32, the relative index size growth is therefore 4 + d 2 s d +4+ d 2 s ≈ d 2 s d + d 2 s = 1 2 s +1 or 4 + d 2 s 4 d +4+ d 2 s ≈ d 2 s 4 d + d 2 s = 1 8 s +1 , respectively. Table 1 empirically corroborates these estimates and shows the memory cost is indeed quite low (5%-20% relative increase over a non-spilled VQ index).

Legend

Rescoring data

:

int8 or float32 representation;

d

or 4

d

bytes per datapoint

PQ representation

:

d

/2

s

bytes per datapoint

Index into rescoring data

: 4

bytes per datapoint

= data for original

datapoint

= data for spilled

datapoint

original PQ to

= rescoring

mapping

spilled PQ to

= rescoring

mapping

VQ Partition 1

VQ Partition 3

Height = memory footprint per datapoint

VQ Partition 2

SOAR also results in a small amount of query-time CPU overhead over a non-spilled VQ index, due to SOAR's need for deduplication: a single datapoint may now appear in multiple partitions that are all searched. This overhead is low, though, because VQ inverted index performance is primarily bottlenecked by memory accesses to the VQ partition data, and this access pattern remains the same.

Finally, SOAR requires a minor tweak to the indexing pipeline. Creating a SOAR-enabled index first requires training a standard, non-spilled VQ index as usual. This gets used to populate the primary assignments, but is also used to calculate the residual r , which is then used in the SOAR loss to select further, spilled assignments. Other parts of the ANN indexing pipeline remain unchanged.

Overall, SOAR's search efficiency gains greatly outweigh its memory, CPU, and indexing overheads.

3.5.1 Spilling to further centroids

The analysis conducted in Section 3.4 focused specifically on the case of assigning to two partitions, π and π ′ . This can be generalized to an arbitrary number of further assignments, where each subsequent assignment is done with a loss considering the distribution of quantized score errors from all prior assignments. We forgo this generalization in experiments and focus on the two-assignment setup because further assignments have diminishing returns; the first spilled assignment is generally sufficient to handle the cases where the primary assignment has high quantized score error, so further assignments are only needed in rare, doubly pathological situations. Meanwhile, the additional memory and indexing cost increases linearly with further assignments; two assignments strikes a good balance between these costs and KMR benefits while also keeping implementation simple.

4 Related Works

4.1 Spill trees

Spill trees (or sp-trees) were originally introduced in [12], and also assign datapoints to multiple vertices at the same level of the tree, just as SOAR may assign a datapoint to multiple VQ partitions. However, there are significant differences; spill trees were searched using defeatist search , a greedy, no-backtracking strategy that took the most promising root-to-leaf path. SOAR indices utilize backtracking in their search procedure; multiple partitions, not just the top one, are evaluated further.

Spill trees also differ in how they perform assignment; their multiple assignments occur at each level of the tree, such that a single datapoint could appear in exponentially many leaves, with respect to the tree depth (which was significant, as spill trees are binary and therefore rather deep). This led to a large (sometimes >100x [11]) storage overhead, which was very costly. In contrast, SOAR performs the multiple assignments at individual levels of the tree, leading to a storage overhead that is constant with respect to the tree depth, and much lower (typically 10%-20%; see Table 1). Finally, SOAR leverages its custom, spilling-aware assignment loss to decide which among the many VQ partitions to spill to, while the spill tree, as a consequence of its binary structure, only had to make the binary decision of whether to spill at all.

4.2 Graph-based algorithms

Graph-based approaches to ANN search, such as [13], have received significant amounts of recent research attention, and a number of such algorithms are featured in comparisons in Section 5.4. If one considers datapoints and partition centers to be vertices, and datapoint-to-partition assignments to be edges, a traditional VQ index would form a tree structure, while spilled VQ assignment (including SOAR) would transform the tree into a cycle-containing general graph data structure. In this sense, SOAR makes VQ indices more closely resemble graph-based ANN indices.

However, graph-based ANN algorithms (as defined in the traditional sense) have no concept of partition centers, and only add edges between datapoints themselves, leading to a lack of hierarchy and linearizability. SOAR inherits the linearizability of tree-based approaches, leading to predictable and sequential memory access patterns that are necessary for achieving maximal performance on modern hardware. SOAR also inherits the small index sizes more easily achievable with tree-based methods. When viewed as a graph, a SOAR index only uses two edges (two partition assignments) per datapoint, while traditional graph-based indices typically use dozens of edges per datapoint, which incurs significant storage overhead that SOAR avoids.

5 Experiments

5.1 KMRcurve comparison

The KMR curve is a deterministic and easily computable metric for VQ-style ANN indices that's highly predictive of real-world performance. Below, we plot KMR k ( t ) as defined in Equation 1, but rather than plotting with respect to t , we plot with respect to the sum of the sizes of the t topranked partitions. This partition size weighting is necessary because spilled VQ data structures have more points per partition due to their multiple assignments, making each individual partition more expensive to search. The results are shown in Figure 6; further details are given in Appendix A.2.

0

100000

200000

300000

Datapoints Searched

85

90

95

100

Recall@100 (Percent)

No spilling

Spilling, no SOAR

SOAR

(a) Glove-1M

0

20000

40000

60000

80000

Datapoints Searched

75

80

85

90

95

100

Recall@100 (Percent)

No spilling

Spilling, no SOAR

SOAR

(b) Microsoft SPACEV

0

50000

100000

150000

Datapoints Searched

75

80

85

90

95

100

Recall@100 (Percent)

No spilling

Spilling, no SOAR

SOAR

(c) Microsoft Turing-ANNS

5.2 Correlation analysis

Here, we look at a number of statistics relating to the quantized score error that shed light on how SOARimproves KMR and overall VQ index quality. These statistics were computed on the Glove-1M dataset with the SOAR λ set to 1; see source code in supplementary materials for more details.

Figure 7 looks at the query-residual angle, where the residuals are taken from the nearest neighbor results for each query. The scatterplot looks at cos θ from the primary VQ assignment and cos θ ′ from the spilled VQ done with SOAR. In comparison with the equivalent scatterplots in Figure 4 done without SOAR, we can see that SOAR significantly reduces the correlation between the cosines of the two angles, thus increasing the efficacy of multiple VQ assignments.

0.6-

0.4

S

0.2

0.0

-0.2

Corr=0.197

-0.4

-0.2

0.0

0.2

0.4

0.6

0

500

1000

1500

2000

Rank(

q

,

C

(

x

)

,

C

)

0

200

400

600

800

1000

1200

1400

Rank(

q

,

C

′

(

x

)

,

C

)

No SOAR

SOAR

The end result of this reduced correlation can be seen in Figure 8, which plots the mean rank of a nearest neighbor's spilled centroid against the rank of that neighbor's primary centroid. We can

see that without SOAR, when a nearest neighbor's primary partition π ( x ) ranks poorly, the nearest neighbor's spilled partition π ′ ( x ) tends to also rank poorly, meaning the spilled assignment is doing little to help ANN performance; the ANN algorithm will still have to search through many partitions to find x . With SOAR, RANK ( q, C π ′ ( x ) , C ) remains low, allowing the ANN algorithm to search significantly fewer partitions to find the nearest neighbors.

Next, in Figure 9, we investigate the effect of SOAR's λ parameter on the resulting VQ index. We can see that as λ is increased, the resulting partitioning π ′ from SOAR will have quantized score error 〈 q, r ′ 〉 less and less correlated with the quantized score error 〈 q, r 〉 from π . This is beneficial for ANN search efficiency, but on the other hand, E [ ∥ ∥ x -C π ′ ( x ) ∥ ∥ 2 ] = E [ ‖ r ′ ‖ 2 ] increases with respect to λ . The diminishing relative importance of ‖ r ′ ‖ 2 as λ increases is a natural consequence of Theorem 3.1, but nonetheless results in a general increase in magnitude of 〈 q, r ′ 〉 and is detrimental to ANN search accuracy; setting the optimal λ requires balancing these effects.







/

/

/

}√√˜⌈∐√]}{

/

/

/

〈]√√}√√]}{

⋃⊗∫(

5.3 Effects of dataset size and recall target

In Figure 10, we take various size samples from the ann-benchmarks.com DEEP dataset and compare the ratio of how many datapoints a SOAR VQ index must access compared to a standard VQ index to achieve the same recall. This ratio is the dependent variable in Figure 10; a higher ratio indicates a greater improvement from SOAR. We can see that higher recall targets, and larger samples, lead to greater ratios.

We maintain a ratio of 400 datapoints per partition across all sample sizes (for example, the 1M datapoint sample was trained with 2500 partitions). This sampling-based approach was chosen over using various datasets of differing sizes, to eliminate effects due to varying search difficulties among datasets.

5.4 End-to-end recall-speed benchmarks

ANNsearch algorithms are ultimately compared by their ability to trade off ANN search accuracy (measured by recall), and speed, measured by queries-per-second throughput. Here we perform this comparison between SOAR and stateof-the-art algorithms in two standardized benchmarking frameworks.

First, we modify ScaNN [8] to utilize SOAR and benchmark its performance on a standardized testing environment with an Intel Xeon W-2135 processor and the test setup from annbenchmarks.com [3]. The algorithms for comparison were installed, configured, and tuned according to their respective benchmark submissions, and run on the same environment; see Appendix A.3 for more details. As shown in Figure 11, SOAR improves upon ScaNN's already state-of-the-art performance.

/⌋

/⌋

/⌋

/

/

/

/

/

⊙⊕∫(〉˜˜]̂]˜{̂glyph[arrowbt](⊔∐]{

⊙

⊙

⊕

⊕

/⊕

∫˜̂∐⌈⌈(⋂∐√˜˜√({∫}

0

2000

4000

6000

8000

10000

12000

0.86  0.88

0.9

0.92  0.94  0.96  0.98

1

Speed (Queries per Second)

annoy

faiss-ivfpqfs

hnsw(faiss)

hnswlib

mrpt

glass

NGT-onng

hnsw(nmslib)

ScaNN

ScaNN-SOAR

Next, we benchmark an ANN search implementation using SOAR on a multi-level k-means clustering against submissions to Track 3 of big-ann-benchmarks.com [16]. These benchmarks allow the use of custom hardware, enabling greater flexibility in approaches to ANN search, but also complicating direct comparison due to the difficulty in accounting for varying hardware costs. In an effort to present existing benchmark submissions in a fair light, we compare SOAR against these submissions in two ways:

We compare the ratio of search throughput to hardware price. The ratios for competing algorithms were taken directly from the benchmark leaderboard, which was straightforward, but SOAR's results were from virtualized cloud compute, so the capital expenditure of buying a server with equivalent power had to be estimated. Hardware prices now may be slightly lower than prices used by other submissions in 2022, their time of submission; on the other hand, SOAR likely would've performed better if run on dedicated hardware, where no other jobs are competing for the processor's various shared resources.

We compare the ratio of search throughput to estimated monthly cloud billing cost for each submission's hardware. One benefit of this method is that hardware costs are completely equalized; in contrast, the alternate method of comparison rewarded contestants for finding especially discounted offerings for the same memory or compute. Additionally, the popularity of cloud infrastructure over self-hosting implies this ratio is what truly matters for many deployers of ANN search algorithms. However, we can't compare against submissions that use proprietary or deprecated hardware, and benchmark contestants never directly optimized for this ratio, because cloud cost was never used in any original benchmark leaderboard.

Although neither comparison alone is perfect, the two in aggregate provide a solid characterization of SOAR's performance. Further experimental details are provided in Appendix A.4, and results are shown in Figure 12 below; our index leads under both cost metrics, and SOAR is critical for our index's performance, roughly doubling throughput over a traditional, non-spilled VQ index.

QPS at 90% R@10 / (Hardware Cost)

0

5

10

15

20

MS-SPACEV

MS-Turing

FAISS Baseline

Gemini

DiskANN

CuANNS-IVFPQ

CuANNS-Multi

Optane

Ours

Ours (no SOAR)

(a) Throughput-to-capex ratio (higher is better)

QPS at 90% R@10 / (Monthly Cloud Cost)

0

10

20

30

40

MS-SPACEV

MS-Turing

FAISS Baseline

DiskANN

CuANNS-IVFPQ

CuANNS-Multi

Ours

Ours (no SOAR)

(b) Throughput-to-cloud bill ratio (higher is better)

Finally, we present memory consumption figures for the ANN indices in these benchmarks in Table 1. The minor increase in memory usage is discussed and analyzed in Section 3.5, and we find that analysis corroborates well with the empirical measurements below. For all three datasets, the index size increase is very minor, and the throughput gains from SOAR could allow the same query traffic to be served by fewer server replicas, in fact potentially reducing net memory consumption.

References

Compute engine pricing. https://cloud.google.com/compute/all-pricing#general_purpose . Accessed: 2023-03-16.

Intel R © optane TM business update: What does this mean for warranty and support. https://www.intel. com/content/www/us/en/support/articles/000091826/memory-and-storage.html . Accessed: 2023-05-15.

Martin Aumüller, Erik Bernhardsson, and Alexander Faithfull. Ann-benchmarks: A benchmarking tool for approximate nearest neighbor algorithms. Information Systems , 87:101374, 2020.

Yoram Bachrach, Yehuda Finkelstein, Ran Gilad-Bachrach, Liran Katzir, Noam Koenigstein, Nir Nice, and Ulrich Paquet. Speeding up the xbox recommender system using a euclidean transformation for inner-product spaces. In Proceedings of the 8th ACM Conference on Recommender Systems , RecSys '14, page 257-264, New York, NY, USA, 2014. Association for Computing Machinery.

Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack Rae, Erich Elsen, and Laurent Sifre. Improving language models by retrieving from trillions of tokens. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 2206-2240. PMLR, 17-23 Jul 2022.

Thomas Dean, Mark A. Ruzon, Mark Segal, Jonathon Shlens, Sudheendra Vijayanarasimhan, and Jay Yagnik. Fast, accurate detection of 100,000 object classes on a single machine. In 2013 IEEE Conference on Computer Vision and Pattern Recognition , pages 1814-1821, 2013.

Miao Fan, Jiacheng Guo, Shuai Zhu, Shuo Miao, Mingming Sun, and Ping Li. Mobius: Towards the next generation of query-ad matching in baidu's sponsored search. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , KDD '19, page 2509-2517, New York, NY, USA, 2019. Association for Computing Machinery.

Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, and Sanjiv Kumar. Accelerating large-scale inference with anisotropic vector quantization. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 3887-3896. PMLR, 13-18 Jul 2020.

Herve Jégou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. IEEE Transactions on Pattern Analysis and Machine Intelligence , 33(1):117-128, 2011.

Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 6769-6781, Online, November 2020. Association for Computational Linguistics.

Herwig Lejsek, Björn Þór Jónsson, and Laurent Amsaleg. Nv-tree: Nearest neighbors at the billion scale. In Proceedings of the 1st ACM International Conference on Multimedia Retrieval , ICMR '11, New York, NY, USA, 2011. Association for Computing Machinery.

Ting Liu, Andrew Moore, Ke Yang, and Alexander Gray. An investigation of practical approximate nearest neighbor algorithms. In L. Saul, Y. Weiss, and L. Bottou, editors, Advances in Neural Information Processing Systems , volume 17. MIT Press, 2004.

Yu A. Malkov and D. A. Yashunin. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE Trans. Pattern Anal. Mach. Intell. , 42(4):824-836, apr 2020.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 8748-8763. PMLR, 18-24 Jul 2021.

Anshumali Shrivastava and Ping Li. Asymmetric lsh (alsh) for sublinear time maximum inner product search (mips). In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 27. Curran Associates, Inc., 2014.

Harsha Vardhan Simhadri, George Williams, Martin Aumüller, Matthijs Douze, Artem Babenko, Dmitry Baranchuk, Qi Chen, Lucas Hosseini, Ravishankar Krishnaswamny, Gopal Srinivasa, Suhas Jayaram Subramanya, and Jingdong Wang. Results of the neurips'21 challenge on billion-scale approximate nearest neighbor search. In Douwe Kiela, Marco Ciccone, and Barbara Caputo, editors, Proceedings of the NeurIPS 2021 Competitions and Demonstrations Track , volume 176 of Proceedings of Machine Learning Research , pages 177-189. PMLR, 06-14 Dec 2022.

Yuhuai Wu, Markus N. Rabe, DeLesley Hutchins, and Christian Szegedy. Memorizing transformers, 2022.

A Appendix

A.1 Proof of Theorem 3.1

We may expand our expectation as follows:

To evaluate this integral, first decompose q and r ′ into components parallel and orthogonal to r ; let us call these components q ‖ , q ⊥ , r ′ ‖ , and r ′ ⊥ , respectively. This allows us to simplify the inner expectation:

Meanwhile, dP [ 〈 q, r 〉 ‖ r ‖ · ‖ q ‖ < cos θ ] is proportional to the surface area of a ( d -1) -dimensional hypersphere of radius sin θ , which we may express as A sin d -2 θ for some constant A . Our integral then becomes

Now define I d = A ∫ π 0 w (cos θ ) sin d θ ; note that L ( r ′ , r, Q ) = ( I d -2 -I d ) ∥ ∥ ∥ r ′ ‖ ∥ ∥ ∥ 2 + I d ‖ r ′ ⊥ ‖ 2 / ( d -1) . For the weight function w ( t ) = | t | λ , we find that I d has a recursive definition if we utilize integration by parts with v = cos θ and u = -cos λ θ sin d -1 θ :

Combining terms, we have ( d + λ ) I d = ( d -1) I d -2 ; this implies for our loss that

Note that r ′ ‖ = proj r r ′ , so this is our desired result. This is very similar to the analysis behind Theorem 3.3 of [8].

A.2 KMRcurve comparison: further details

The Glove-1M dataset came from ann-benchmarks.com [3], while the Microsoft SPACEV and Microsoft Turing-ANNS datasets came from big-ann-benchmarks.com.

Glove-1M was trained on an anisotropic loss [8] with 2000 partitions, and SOAR was run with λ = 1 .

The two billion-datapoint datasets were trained on an anisotropic loss with approximately 7.2 million partitions, and SOAR was run with λ = 1 . 5 .

Table 2 below presents the approximate number of datapoints needed to search in order to achieve various recall targets on these datasets, as well as SOAR's KMR gain over non-spilled VQ indices.

A.3 ann-benchmarks.com Glove-1M benchmark details

The ann-benchmarks.com benchmark results came from a VQ-PQ index; the VQ index had 2000 partitions and used SOAR λ = 1 . Both VQ and PQ were trained with anisotropic loss. The PQ quantization was configured with 16 subspaces and s = 2 dimensions per subspace, and the highestbitrate representation of the datapoints was encoded in 32-bit floats. By the analysis from Section 3.5, we would therefore expect SOAR to increase index size by 1 / 17 ≈ 5 . 9% , which is quite close to the empirical measurement in Table 1.

A.4 End-to-end recall-speed benchmarks: further details

A.4.1 General index setup

The big-ann-benchmarks.com results used a multilayer VQ index; the lower VQ index had approximately 7.2 million partitions, and these partition centers were vector-quantized again to 40000 partitions. PQ-quantized forms of the VQ residuals were used as intermediate scoring stages. The highest-bitrate form of the dataset stored by the index was an INT8-quantized representation. All VQ, PQ, and INT8 quantizations were trained with anisotropic loss.

A.4.2 Estimated cost of SOAR benchmark hardware

The SOAR benchmark results came from running on a server using 32 vCPU (16 physical cores) on an Intel Cascade Lake generation processor with 150GB of memory. The Supermicro SYS-510P-M configured with:

1 x Intel R © Xeon R © Silver 4314 Processor 16-Core 2.40 GHz 24MB Cache (135W)

6 x 32GB DDR4 3200MHz ECC RDIMM Server Memory (2Rx8 - 16Gb)

1 x 1TB 3.5" MG04ACA 7200 RPM SATA3 6Gb/s 128M Cache 512N Hard Drive

should provide an upper bound for the cost of the SOAR benchmark setup, because the Supermicro server as configured has a newer generation and higher-clocked processor, and more memory, than the actual virtualized hardware used in benchmarking SOAR. At the time of writing, such a Supermicro server costs $2740.60. Combining this with the big-ann-benchmarks.com results gives us the following table, used to produce the plots in Figure 12a:

A.4.3 Cloud cost details

The monthly cloud infrastructure costs were derived from Google Compute Engine's on-demand pricing in the us-central1 region. At the time of writing, the costs were [1]:

Using this information, we computed the monthly cloud spend required for each of the entries to Track 3 of the big-ann-benchmarks competition:

Two submissions from the original big-ann-benchmarks competition could not be included in this comparison: Intel's OptANNe GraphANN submission, and GSI Technology's Gemini submission. The former relied on Intel Optane storage technology, which has been discontinued [2] and therefore isn't available on any mainstream cloud compute provider, and can't be priced. The latter leverages proprietary hardware not available to cloud providers. However, both of these entries are present in the throughput-per-capex ranking featured earlier.

Our own results came from a 32 vCPU, 150GB machine, which would cost $1293.09 per month. We achieved 32608 QPS on Microsoft Turing-ANNS and 46712 QPS on Microsoft SPACEV, leading to the numbers presented in Figure 12. These results are also reproduced in tabular form below.

Peer reviews:
comments: SUMMARY:
The paper proposes a neat variant of spill trees for approximate nearest neighbor search (ANN). The idea is to reduce correlation between the set of points retrieved from each of the partition elements in a spill tree (where the partitions elements overlap or "spill over") by adding a penalty term in the assignment function that involves the expectation of the angle between the query $q$ and the quantization residual $r$ (see Eq. (2)).

The idea is logical and the empirical results indicate state-of-the-art performance in large-scale benchmarks.

STRENGTHS:
+ important problem
+ clear and sound idea
+ rigorous theory
+ outstanding empirical results
+ a pleasure to read

WEAKNESSES:
- the theoretical assumption underlying the derivation of the practical assignment formula (Thm. 3.1) assumes that the query distribution is uniform over a d-dimensional unit sphere, which is an unrealistic assumption -- however, the resulting method seems to perform well enough, so apparently the performance isn't sensitive to this

QUESTIONS:
detailed comments:

p. 2: "top k' partition": the use of the term partition to refer to the parts that make up the partitioned set is a bit problematic. I would prefer to use the term 'partition element' (or perhaps 'part' or 'cell') to distinguish the parts from the partition which includes them.

p. 4: is the axis label "quantized score error $\langle q,r \rangle$" (also mentioned in the heading of Sec. 3.2) a definition of a new term? if yes, it should be given in the main text

LIMITATIONS:
Yes, the authors have adequately addressed limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an innovative technique for constructing a vector index to perform approximate nearest neighbor search, building upon the foundations of vector quantization and redundancy through spill trees. The authors propose a novel loss function aimed at identifying k-alternate centroids in the vector quantization process, thereby minimizing the failure correlation between assignments to the original and alternate centroids. This reduction in correlation effectively decreases the likelihood of missing a nearest neighbor during the search phase. Notably, the authors substantiate their approach by showcasing improved recall@k scores on benchmark datasets, all achieved with minimal computational overhead.\n\nSTRENGTHS:\n* The ideas introduced in the paper are about spilling and redudancy in cluster assignments which have been previous explored in different capacities in the past. But the paper gives meaningful improvements on this task of approximate nearest neighbor(ANN) search which is a critical real world application. \n* The authors' approach is characterized by its simplicity and ease of implementation, supported by a solid theoretical foundation. The motivation behind the proposed loss function is well articulated, emphasizing the necessity of reducing the correlation between <q,r> and <q,r'> while promoting orthogonality among the residuals.\n* The paper is well-written and effectively conveys its ideas, providing adequate background information for readers to comprehend the research context.\n* The recall@100 results achieved by the proposed method are notably strong, particularly on larger-scale datasets, underscoring its efficacy in handling real-world scenarios.\n* The benchmarks conducted demonstrate satisfactory throughput results, indicating that the approach imposes minimal memory and computation overhead. This attribute enhances the practical viability of the method for deployment in real-world applications.\n* The effectiveness of the proposed method scales well with increasing dataset sizes, as evidenced by the more pronounced improvements observed when dealing with larger datasets where the probability of failure in cluster assignments tends to rise.\n* The authors further enhance the credibility of their findings by providing a compelling correlation analysis, elucidating the reasons behind the success of their proposed method (referred to as SOAR).\n\nOverall, the paper presents a valuable contribution to the field, offering practical enhancements to approximate nearest neighbor search while maintaining simplicity in implementation. The results obtained through rigorous experimentation and analysis demonstrate the effectiveness and scalability of the proposed method.\n\nWEAKNESSES:\n* Section 3.5 lacks a lot of details and paper could heavily benefit from providing a more structured and detailed explanation of the final algorithm. For eg. clarifying whether the spilling is done hierarchically and addressing corresponding memory considerations.\n* The improvement over previous SOTA ScaNN is marginal, and it would be helpful to if authors could elaborate more on the trade-offs and benefits over ScaNN\n* The paper would also benefit from including more details in Section 4.1 about the comparison with spill trees and referencing relevant literature, making it more comprehensive and easier to read.\n* A formal theoretical analysis of the time and space complexity would be valuable, providing a deeper understanding of the computational requirements of the proposed algorithm.\n\nQUESTIONS:\n* It would be useful to know if only one alternate centroid is used for each data point or if multiple alternate centroids are employed. If multiple centroids are utilized, it would be informative to see the effect of increasing the number of alternate centroids on the recall performance.\n* In Figure 8, it would be beneficial if the authors provided a proper definition of \"distortion\" and explained why it matters. Additionally, offering more details on how to effectively choose the parameter λ would enhance the comprehension of the research findings.\n* Suggestion: Authors can highlight the fact that decreased correlation between r and r’ means probability of misses decreases exponentially as you add more and more redundancy, through some formal analysis?\n* Suggestion: The community would benefit from incorporating soar into common open-source tools like FAISS and ScaNN and increase it’s adoption\n\nLIMITATIONS:\nYes the authors have adequately addressed limitations", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper discusses how a data index is built for Approximate Nearest Neighbor Search (ANNS). However, existing methods train and compute redundant representations independently, leading to inefficiency and inaccuracy. The paper proposes a novel data indexing technique called SOAR for approximate nearest neighbor (ANN) search, which utilizes multiple redundant representations and an orthogonality-amplified residual loss, drastically improving the overall index quality, resulting in state-of-the-art ANN benchmark performance.\n\nSTRENGTHS:\n1. The paper studies a very important and a very fundamental problem that finds a lot of applications in search tasks.\n2. This paper has a clear motivation, i.e., increasing search efficiency when <q,r> is high. This idea is technically sound. \n3. Many analyses and experiments have been conducted.\n\nWEAKNESSES:\n1. Both Preliminary and Method are based on the MIPS problem. However, the billion datasets of the experiment are not focused on the inner product completely (Microsoft SPACEV and Microsoft Turing-ANNS are L2 datasets).\n2. Results of recall-QPS curves on more datasets (such as Yandex Text-to-Image, whose metric is the inner product) are needed.\n3. Lack of discussion on the complexity of storage under spilled VQ assignment. Statistics in Tab. 1 are not detailed enough.\n\nQUESTIONS:\n1. Why use the L2 datasets to study the efficiency of MIPS methods? maybe more details on datasets are needed (How to use datasets? Which metric are datasets based on? )\n2. Include more results on different datasets which are based on inner product metrics (such as Yandex Text-to-Image) or explain the reasonability of existing datasets.\n3. Discuss the space cost of spilled VQ assignment with relative notations. \n4. Please add more details about the loss function. How to generalize the loss function if the spilled assignments are above 2 ? (mentioned on line 126: This idea can be extended to further (>2) assignments as well.)\n\nLIMITATIONS:\nPlease refer to Paper Weakness.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper revists the \"vector quantization\" technique for nearest neighbor search. In the VQ technique, points are represented by their closest centroid in a partition schema, typically based on k-means. Answering a query consists of finding the nearest centroids to the query and enumerating points represented by each centroid in the nearest centroid list.\n\nGiven that the nearest neighbor of a query need not be in the centroid closest to the query, one typicall has to explore a few centroids to find the right answer. In this paper, the authors study alternate schemes to reduce the number of centroids to explore.\n\nTheir contention is that centroids should be designed to minimize <q,r> where r is the residual x-C(x), of x, the nearest neighbor of q and its centroid C(x). They further focus on cos \\theta, where \\theta is the angle between q and r. Apart from the first eucliden residual minimizing partition scheme, the authors design a second cos \\theta minimizing partition scheme.\n\ncollectively the intention is to reduce the depth of exploration.\n\nThe results are constrasted with existing techniques FAISS and DiskANN and on benchmark datasets.\n\nSTRENGTHS:\nA clear explanation of the techniques involved and visualization of why alternate spill over schemes dont work.\n\nEmpirical measurements indicated the rank of target vectors is improved by new design (Fig. 5.1)\n\nExperiments comparing to ann-benchmarks and also cost-normalized metric on big-ann-benchmark. Strong results overall.\n\nWEAKNESSES:\nThe index design  5.4 is unclear. line 276-278 need a bit more explanation -- what is the exact index and query path design.\n\nThe results in Fig 10b needs more explanation and information to be verifiable. There is no plot of QPS alone. Nor a report of the parameters used for baselines for others to judge if they are the best config. For DiskANN, I suspect the IO/s of a GCP local SSD (680K for 3.2TB ssd: https://cloud.google.com/compute/docs/disks/local-ssd) is not enough for 72 cores. Performance might saturate below what 72 cores issuing queries might need (in which case DiskANN must be priced in table A.3 with fewer cores).\n\nQUESTIONS:\n1. Can you please provide more details of the index and query algorithms\n\n2. Can you please provide more details about how your arrived at plot 10b and what care was taken to prepare baselines.\n\n3. How does this idea compare this idea with ScaNN which optimizes for angular metrics too, on large datasets.\n\n4. What is the performance on other (non-Glove)datasets on ann-benchmarks.com\n\n5. Can your ideas also be applied to product quantization (in each chunk) and improve PQ's error (normalized for total bytes)?\n\nLIMITATIONS:\nna", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-08"}