SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The analysis conducted in Section 3.4 focused specifically on the case of assigning to two partitions,  $\pi$  and  $\pi'$ . This can be generalized to an arbitrary number of further assignments, where each subsequent assignment is done with a loss considering the distribution of quantized score errors from all prior assignments. We forgo this generalization in experiments and focus on the two-assignment setup because further assignments have diminishing returns; the first spilled assignment is generally sufficient to handle the cases where the primary assignment has high quantized score error, so further assignments are only needed in rare, doubly pathological situations. Meanwhile, the additional memory and indexing cost increases linearly with further assignments; two assignments strike a good balance between these costs and KMR benefits while also keeping implementation simple.
4.1 Spill trees
Spill trees (or sp-trees) were originally introduced in [12], and also assign datapoints to multiple vertices at the same level of the tree, just as SOAR may assign a datapoint to multiple VQ partitions. However, there are significant differences; spill trees were searched using defeatist search, a greedy, no-backtracking strategy that took the most promising root-to-leaf path. SOAR indices utilize backtracking in their search procedure; multiple partitions, not just the top one, are evaluated further.
Spill trees also differ in how they perform assignment; their multiple assignments occur at each level of the tree, such that a single datapoint could appear in exponentially many leaves, with respect to the tree depth (which was significant, as spill trees are binary and therefore rather deep). This led to a large (sometimes  $>100x$  [11]) storage overhead, which was very costly. In contrast, SOAR performs the multiple assignments at individual levels of the tree, leading to a storage overhead that is constant with respect to the tree depth, and much lower (typically  $10\% -20\%$ ; see Table 1). Finally, SOAR leverages its custom, spilling-aware assignment loss to decide which among the many VQ partitions to spill to, while the spill tree, as a consequence of its binary structure, only had to make the binary decision of whether to spill at all.
4.2 Graph-based algorithms
Graph-based approaches to ANN search, such as [13], have received significant amounts of recent research attention, and a number of such algorithms are featured in comparisons in Section 5.4. If one considers datapoints and partition centers to be vertices, and datapoint-to-partition assignments to be edges, a traditional VQ index would form a tree structure, while spilled VQ assignment (including SOAR) would transform the tree into a cycle-containing general graph data structure. In this sense, SOAR makes VQ indices more closely resemble graph-based ANN indices.
However, graph-based ANN algorithms (as defined in the traditional sense) have no concept of partition centers, and only add edges between datapoints themselves, leading to a lack of hierarchy and linearizability. SOAR inherits the linearizability of tree-based approaches, leading to predictable and sequential memory access patterns that are necessary for achieving maximal performance on modern hardware. SOAR also inherits the small index sizes more easily achievable with tree-based methods. When viewed as a graph, a SOAR index only uses two edges (two partition assignments) per datapoint, while traditional graph-based indices typically use dozens of edges per datapoint, which incurs significant storage overhead that SOAR avoids.
5.1 KMR curve comparison
The KMR curve is a deterministic and easily computable metric for VQ-style ANN indices that's highly predictive of real-world performance. Below, we plot  $\mathrm{KMR}_k(t)$  as defined in Equation 1, but rather than plotting with respect to  $t$ , we plot with respect to the sum of the sizes of the  $t$  top-ranked partitions. This partition size weighting is necessary because spilled VQ data structures have more points per partition due to their multiple assignments, making each individual partition more expensive to search. The results are shown in Figure 6; further details are given in Appendix A.2.
(a) Glove-1M
(b) Microsoft SPACEV
Figure 6: SOAR allows a VQ index to achieve a given recall target while reading fewer total datapoints and therefore utilizing less memory bandwidth, increasing ANN search performance. SOAR's improvements are especially large on Microsoft SPACEV and Microsoft Turing-ANNS, the billion-scale datasets in this experiment; see Section 5.3 for further analysis of dataset size.
(c) Microsoft Turing-ANNS
5.2 Correlation analysis
Here, we look at a number of statistics relating to the quantized score error that shed light on how SOAR improves KMR and overall VQ index quality. These statistics were computed on the Glove-1M dataset with the SOAR  $\lambda$  set to 1; see source code in supplementary materials for more details.
Figure 7 looks at the query-residual angle, where the residuals are taken from the nearest neighbor results for each query. The scatterplot looks at  $\cos \theta$  from the primary VQ assignment and  $\cos \theta^{\prime}$  from the spilled VQ done with SOAR. In comparison with the equivalent scatterplots in Figure 4 done without SOAR, we can see that SOAR significantly reduces the correlation between the cosines of the two angles, thus increasing the efficacy of multiple VQ assignments.
Figure 7: Scatter plot, analogous to Figure 4a, but with spilled assignments performed using SOAR loss; angular correlation is much lower.
Figure 8: The nearest neighbors most difficult to find under partitioning  $\pi$ , as quantified by  $\mathrm{RANK}(q, \mathcal{C}_{\pi(x)}, \mathcal{C})$ , also tend to be difficult to find under  $\pi'$  when SOAR isn't used.
The end result of this reduced correlation can be seen in Figure 8, which plots the mean rank of a nearest neighbor's spilled centroid against the rank of that neighbor's primary centroid. We can
see that without SOAR, when a nearest neighbor's primary partition  $\pi(x)$  ranks poorly, the nearest neighbor's spilled partition  $\pi'(x)$  tends to also rank poorly, meaning the spilled assignment is doing little to help ANN performance; the ANN algorithm will still have to search through many partitions to find  $x$ . With SOAR,  $\mathrm{RANK}(q, \mathcal{C}_{\pi'(x)}, \mathcal{C})$  remains low, allowing the ANN algorithm to search significantly fewer partitions to find the nearest neighbors.
Next, in Figure 9, we investigate the effect of SOAR's  $\lambda$  parameter on the resulting VQ index. We can see that as  $\lambda$  is increased, the resulting partitioning  $\pi'$  from SOAR will have quantized score error  $\langle q, r' \rangle$  less and less correlated with the quantized score error  $\langle q, r \rangle$  from  $\pi$ . This is beneficial for ANN search efficiency, but on the other hand,  $\mathbb{E}\left[\|x - \mathcal{C}_{\pi'(x)}\|^2\right] = \mathbb{E}\left[\|r'\|^2\right]$  increases with respect to  $\lambda$ . The diminishing relative importance of  $\|r'\|^2$  as  $\lambda$  increases is a natural consequence of Theorem 3.1, but nonetheless results in a general increase in magnitude of  $\langle q, r' \rangle$  and is detrimental to ANN search accuracy; setting the optimal  $\lambda$  requires balancing these effects.
Figure 9: Raising SOAR  $\lambda$  increases VQ distortion  $\mathbb{E}[\| r^{\prime}\|^{2}]$ , but lowers score correlation  $\rho_{\langle q,r\rangle ,\langle q,r^{\prime}\rangle}$ .
5.3 Effects of dataset size and recall target
In Figure 10, we take various size samples from the ann-benchmarks.com DEEP dataset and compare the ratio of how many datapoints a SOAR VQ index must access compared to a standard VQ index to achieve the same recall. This ratio is the dependent variable in Figure 10; a higher ratio indicates a greater improvement from SOAR. We can see that higher recall targets, and larger samples, lead to greater ratios.
We maintain a ratio of 400 datapoints per partition across all sample sizes (for example, the 1M datapoint sample was trained with 2500 partitions). This sampling-based approach was chosen over using various datasets of differing sizes, to eliminate effects due to varying search difficulties among datasets.
Figure 10: SOAR's ANN search efficiency benefit grows as SOAR is applied to larger datasets, and as the ANN recall target rises.
5.4 End-to-end recall-speed benchmarks
ANN search algorithms are ultimately compared by their ability to trade off ANN search accuracy (measured by recall), and speed, measured by queries-per-second throughput. Here we perform this comparison between SOAR and state-of-the-art algorithms in two standardized benchmarking frameworks.
First, we modify ScaNN [8] to utilize SOAR and benchmark its performance on a standardized testing environment with an Intel Xeon W-2135 processor and the test setup from annbenchmarks.com [3]. The algorithms for comparison were installed, configured, and tuned according to their respective benchmark submissions, and run on the same environment; see Appendix A.3 for more details. As shown in Figure 11, SOAR improves upon ScaNN's already state-of-the-art performance.
Figure 11: SOAR + ScaNN on Glove-1M.
Next, we benchmark an ANN search implementation using SOAR on a multi-level k-means clustering against submissions to Track 3 of big-ann-benchmarks.com [16]. These benchmarks allow the use of custom hardware, enabling greater flexibility in approaches to ANN search, but also complicating direct comparison due to the difficulty in accounting for varying hardware costs. In an effort to present existing benchmark submissions in a fair light, we compare SOAR against these submissions in two ways:
1. We compare the ratio of search throughput to hardware price. The ratios for competing algorithms were taken directly from the benchmark leaderboard, which was straightforward, but SOAR's results were from virtualized cloud compute, so the capital expenditure of buying a server with equivalent power had to be estimated. Hardware prices now may be slightly lower than prices used by other submissions in 2022, their time of submission; on the other hand, SOAR likely would've performed better if run on dedicated hardware, where no other jobs are competing for the processor's various shared resources.
2. We compare the ratio of search throughput to estimated monthly cloud billing cost for each submission's hardware. One benefit of this method is that hardware costs are completely equalized; in contrast, the alternate method of comparison rewarded contestants for finding especially discounted offerings for the same memory or compute. Additionally, the popularity of cloud infrastructure over self-hosting implies this ratio is what truly matters for many deployers of ANN search algorithms. However, we can't compare against submissions that use proprietary or deprecated hardware, and benchmark contestants never directly optimized for this ratio, because cloud cost was never used in any original benchmark leaderboard.
Although neither comparison alone is perfect, the two in aggregate provide a solid characterization of SOAR's performance. Further experimental details are provided in Appendix A.4, and results are shown in Figure 12 below; our index leads under both cost metrics, and SOAR is critical for our index's performance, roughly doubling throughput over a traditional, non-spilled VQ index.
(a) Throughput-to-capex ratio (higher is better)
(b) Throughput-to-cloud bill ratio (higher is better)
Figure 12: SOAR leads big-ann-benchmarks performance under both methods of comparison.
Finally, we present memory consumption figures for the ANN indices in these benchmarks in Table 1. The minor increase in memory usage is discussed and analyzed in Section 3.5, and we find that analysis corroborates well with the empirical measurements below. For all three datasets, the index size increase is very minor, and the throughput gains from SOAR could allow the same query traffic to be served by fewer server replicas, in fact potentially reducing net memory consumption.
Dataset	Memory Usage, No SOAR	Memory Usage with SOAR
Glove-1M	453.5 MB	488.4 MB (+7.7%)
Microsoft Turing-ANNS	120.03 GB	140.23 GB (+16.8%)
Microsoft SPACEV	120.85 GB	141.80 GB (+17.3%)
Table 1: ANN index memory consumption before/after SOAR.
[16] Harsha Vardhan Simhadri, George Williams, Martin Aumüller, Matthijs Douze, Artem Babenko, Dmitry Baranchuk, Qi Chen, Lucas Hosseini, Ravishankar Krishnaswamny, Gopal Srinivasa, Suhas Jayaram Subramanya, and Jingdong Wang. Results of the neurips'21 challenge on billion-scale approximate nearest neighbor search. In Douwe Kiela, Marco Ciccone, and Barbara Caputo, editors, Proceedings of the NeurIPS 2021 Competitions and Demonstrations Track, volume 176 of Proceedings of Machine Learning Research, pages 177–189. PMLR, 06–14 Dec 2022.
[17] Yuhuai Wu, Markus N. Rabe, DeLesley Hutchins, and Christian Szegedy. Memorizing transformers, 2022.
We may expand our expectation as follows:
$$
\begin{array}{l} \mathcal {L} \left(r ^ {\prime}, r, \mathcal {Q}\right) = \mathbb {E} _ {q \in \mathcal {Q}} \left[ w \left(\frac {\langle q , r \rangle}{\| r \| \cdot \| q \|}\right) \langle q, r ^ {\prime} \rangle^ {2} \right] \\ = \int_ {0} ^ {\pi} w (\cos \theta) \mathbb {E} _ {q \in \mathcal {Q}} \left[ \langle q, r ^ {\prime} \rangle^ {2} \mid \frac {\langle q , r \rangle}{\| r \| \cdot \| q \|} = \cos \theta \right] d P \left[ \frac {\langle q , r \rangle}{\| r \| \cdot \| q \|} <   \cos \theta \right]. \\ \end{array}
$$
To evaluate this integral, first decompose  $q$  and  $r'$  into components parallel and orthogonal to  $r$ ; let us call these components  $q_{\parallel}$ ,  $q_{\perp}$ ,  $r'_\parallel$ , and  $r'_\perp$ , respectively. This allows us to simplify the inner expectation:
$$
\begin{array}{l} \mathbb {E} _ {q \in \mathcal {Q}} \left[ \langle q, r ^ {\prime} \rangle^ {2} \mid \frac {\langle q , r \rangle}{\| r \| \cdot \| q \|} = \cos \theta \right] = \mathbb {E} _ {q \in \mathcal {Q}} [ \langle q, r ^ {\prime} \rangle^ {2} \mid \| q _ {\parallel} \| = \cos \theta ] \\ = \mathbb {E} _ {q \in \mathcal {Q}} \left[ \left\langle q _ {\parallel} + q _ {\perp}, r _ {\parallel} ^ {\prime} + r _ {\perp} ^ {\prime} \right\rangle^ {2} \mid \| q _ {\parallel} \| = \cos \theta \right] \\ = \mathbb {E} _ {q \in \mathcal {Q}} \left[ \left(\left\langle q _ {\parallel}, r _ {\parallel} ^ {\prime} \right\rangle + \left\langle q _ {\perp}, r _ {\perp} ^ {\prime} \right\rangle\right) ^ {2} | \| q _ {\parallel} \| = \cos \theta \right] \\ = \mathbb {E} _ {q \in \mathcal {Q}} \left[ \left(\cos \theta \left\| r _ {\parallel} ^ {\prime} \right\| + \langle q _ {\perp}, r _ {\perp} ^ {\prime} \rangle\right) ^ {2} \mid \| q _ {\parallel} \| = \cos \theta \right] \\ = \cos^ {2} \theta \| r _ {\|} ^ {\prime} \| ^ {2} + 2 \cos \theta \| r _ {\|} ^ {\prime} \| \left\langle \mathbb {E} [ q _ {\perp} \mid \| q _ {\|} \| = \cos \theta ], r _ {\perp} ^ {\prime} \right\rangle \\ + \mathbb {E} [ \left\langle q _ {\perp}, r _ {\perp} ^ {\prime} \right\rangle) ^ {2} | \| q _ {\parallel} \| = \cos \theta ] \\ = \cos^ {2} \theta \left\| r _ {\parallel} ^ {\prime} \right\| ^ {2} + \sin^ {2} \theta \| r _ {\perp} ^ {\prime} \| ^ {2} / (d - 1). \\ \end{array}
$$
Meanwhile,  $dP\left[\frac{\langle q,r\rangle}{\|r\|\cdot\|q\|} < \cos \theta \right]$  is proportional to the surface area of a  $(d - 1)$ -dimensional hypersphere of radius  $\sin \theta$ , which we may express as  $A\sin^{d - 2}\theta$  for some constant  $A$ . Our integral then becomes
$$
\begin{array}{l} \mathcal {L} \left(r ^ {\prime}, r, \mathcal {Q}\right) = \int_ {0} ^ {\pi} w (\cos \theta) \left[ \cos^ {2} \theta \left\| r _ {| |} ^ {\prime} \right\| ^ {2} + \sin^ {2} \theta \| r _ {\perp} ^ {\prime} \| ^ {2} / (d - 1) \right] A \sin^ {d - 2} \theta d \theta \\ = \left(A \int_ {0} ^ {\pi} w (\cos \theta) \sin^ {d - 2} \theta \cos^ {2} \theta\right) \left\| r _ {| |} ^ {\prime} \right\| ^ {2} + \\ \left(A \int_ {0} ^ {\pi} w (\cos \theta) \sin^ {d} \theta\right) \| r _ {\perp} ^ {\prime} \| ^ {2} / (d - 1). \\ \end{array}
$$
Now define  $I_{d} = A\int_{0}^{\pi}w(\cos \theta)\sin^{d}\theta$ ; note that  $\mathcal{L}(r',r,\mathcal{Q}) = (I_{d - 2} - I_d)\left\| r_\parallel '\right\|^2 +I_d\| r_\perp '\| ^2 /(d - 1)$ . For the weight function  $w(t) = |t|^{\lambda}$ , we find that  $I_{d}$  has a recursive definition if we utilize integration by parts with  $v = \cos \theta$  and  $u = -\cos^{\lambda}\theta \sin^{d - 1}\theta$ :
$$
\begin{array}{l} I _ {d} = A \int_ {0} ^ {\pi} | \cos \theta | ^ {\lambda} \sin^ {d} \theta d \theta \\ = 2 A \int_ {0} ^ {\pi / 2} \cos^ {\lambda} \theta \sin^ {d} \theta d \theta \\ = - 2 A \cos^ {\lambda + 1} \theta \sin^ {d - 1} \theta \Big | _ {0} ^ {\pi / 2} \\ - 2 A \int_ {0} ^ {\pi / 2} \cos \theta \left(- (d - 1) \cos^ {\lambda + 1} \theta \sin^ {d - 2} \theta - \lambda \sin^ {d} \theta \cos^ {\lambda - 1} \theta\right) d \theta \\ = 2 A (d - 1) \int_ {0} ^ {\pi / 2} \cos^ {\lambda + 2} \theta \sin^ {d - 2} \theta d \theta - 2 \lambda A \int_ {0} ^ {\pi / 2} \cos^ {\lambda} \theta \sin^ {d} \theta d \theta \\ = 2 A (d - 1) \int_ {0} ^ {\pi / 2} \cos^ {\lambda} (1 - \sin^ {2} \theta) \theta \sin^ {d - 2} \theta d \theta - \lambda I _ {d} \\ = (d - 1) I _ {d - 2} - (d - 1) I _ {d} - \lambda I _ {d}. \\ \end{array}
$$
Combining terms, we have  $(d + \lambda)I_{d} = (d - 1)I_{d - 2}$ ; this implies for our loss that
$$
\begin{array}{l} \mathcal {L} \left(r ^ {\prime}, r, \mathcal {Q}\right) = \left(I _ {d - 2} - I _ {d}\right) \left\| r _ {\parallel} ^ {\prime} \right\| ^ {2} + I _ {d} \left\| r _ {\perp} ^ {\prime} \right\| ^ {2} / (d - 1) \\ = \left(\frac {d + \lambda}{d - 1} - 1\right) I _ {d} \left\| r _ {\parallel} ^ {\prime} \right\| ^ {2} + I _ {d} \left\| r _ {\perp} ^ {\prime} \right\| ^ {2} / (d - 1) \\ = \frac {I _ {d}}{d - 1} \cdot \left(\left(\lambda + 1\right) \left\| r _ {\parallel} ^ {\prime} \right\| ^ {2} + \left\| r _ {\perp} ^ {\prime} \right\| ^ {2}\right) \\ = \frac {I _ {d}}{d - 1} \cdot \left(\lambda \left\| r _ {\parallel} ^ {\prime} \right\| ^ {2} + \| r ^ {\prime} \| ^ {2}\right) \\ \propto \lambda \left\| r _ {\parallel} ^ {\prime} \right\| ^ {2} + \| r ^ {\prime} \| ^ {2}. \\ \end{array}
$$
Note that  $r_{\parallel}^{\prime} = \mathrm{proj}_{r} r^{\prime}$ , so this is our desired result. This is very similar to the analysis behind Theorem 3.3 of [8].
A.2 KMR curve comparison: further details
The Glove-1M dataset came from ann-benchmarks.com [3], while the Microsoft SPACEV and Microsoft Turing-ANNS datasets came from big-ann-benchmarks.com.
- Glove-1M was trained on an anisotropic loss [8] with 2000 partitions, and SOAR was run with  $\lambda = 1$ .
- The two billion-datapoint datasets were trained on an anisotropic loss with approximately 7.2 million partitions, and SOAR was run with  $\lambda = 1.5$ .
Table 2 below presents the approximate number of datapoints needed to search in order to achieve various recall targets on these datasets, as well as SOAR's KMR gain over non-spilled VQ indices.
Dataset	Recall Target (R@100)	No Spilling	Spilling, No SOAR	SOAR	KMR gain, SOAR over No Spilling
Glove-1M	80%	19983	21126	18352	1.09x
85%	32639	34109	29369	1.11x
90%	59063	61392	52292	1.13x
95%	127819	135612	112488	1.14x
Microsoft SPACEV	80%	2850	2700	2450	1.16x
85%	5550	4650	3700	1.50x
90%	14350	10850	6950	2.06x
95%	69050	47050	20550	3.36x
Microsoft Turing-ANNS	80%	4500	3950	3400	1.32x
85%	9350	7250	5600	1.67x
90%	26700	17800	11150	2.39x
95%	145900	82100	33800	4.32x
Table 2: KMR curve results in tabulated form; see Figure 6 for graphs of this data.
A.3 ann-benchmarks.com Glove-1M benchmark details
The ann-benchmarks.com benchmark results came from a VQ-PQ index; the VQ index had 2000 partitions and used SOAR  $\lambda = 1$ . Both VQ and PQ were trained with anisotropic loss. The PQ quantization was configured with 16 subspaces and  $s = 2$  dimensions per subspace, and the highest-bitrate representation of the datapoints was encoded in 32-bit floats. By the analysis from Section 3.5, we would therefore expect SOAR to increase index size by  $1/17 \approx 5.9\%$ , which is quite close to the empirical measurement in Table 1.
A.4 End-to-end recall-speed benchmarks: further details
A.4.1 General index setup
The big-ann-benchmarks.com results used a multilayer VQ index; the lower VQ index had approximately 7.2 million partitions, and these partition centers were vector-quantized again to 40000 partitions. PQ-quantized forms of the VQ residuals were used as intermediate scoring stages. The highest-bitrate form of the dataset stored by the index was an INT8-quantized representation. All VQ, PQ, and INT8 quantizations were trained with anisotropic loss.
A.4.2 Estimated cost of SOAR benchmark hardware
The SOAR benchmark results came from running on a server using 32 vCPU (16 physical cores) on an Intel Cascade Lake generation processor with 150GB of memory. The Supermicro SYS-510P-M configured with:
- 1 x Intel® Xeon® Silver 4314 Processor 16-Core 2.40 GHz 24MB Cache (135W)
- 6 x 32GB DDR4 3200MHz ECC RDIMM Server Memory (2Rx8 - 16Gb)
- 1 x 1TB 3.5" MG04ACA 7200 RPM SATA3 6Gb/s 128M Cache 512N Hard Drive
should provide an upper bound for the cost of the SOAR benchmark setup, because the Supermicro server as configured has a newer generation and higher-clocked processor, and more memory, than the actual virtualized hardware used in benchmarking SOAR. At the time of writing, such a Supermicro server costs $2740.60. Combining this with the big-ann-benchmarks.com results gives us the following table, used to produce the plots in Figure 12a:
Algorithm	Hardware Cost	MS-SPACEV Queries/Sec (90% R@10, from here)	MS-Turing Queries/Sec (90% R@10, from here)
FAISS Baseline	$22021.90	3265	2845
DiskANN	$11742	6503	17201
Gemini	$55726.66	16422	21780
CuANNS-IVFPQ	$150000	108302	109745
CuANNS-Multi	$150000	839749	584293
OptANNe GraphANN	$14664.20	157828	161463
Ours	$2740.60	46712	32608
A.4.3 Cloud cost details
The monthly cloud infrastructure costs were derived from Google Compute Engine's on-demand pricing in the us-central1 region. At the time of writing, the costs were [1]:
Item	Monthly Cost (USD)
1 vCPU	$24.81
1GB Memory	$3.33
1GB Local SSD	$0.08
A100 80GB	$2868.90
V100 16GB	$1267.28
Using this information, we computed the monthly cloud spend required for each of the entries to Track 3 of the big-ann-benchmarks competition:
Algorithm	vCPU	RAM (GB)	Other	Total Monthly Cost (USD)
FAISS Baseline	32	768	1x V100 16GB	$4617.57
DiskANN	72	64	3276.8GB SSD	$2261.18
CuANNS-IVFPQ	256	2048	1x A100 80GB¹	$16036.46
CuANNS-Multi	256	2048	8x A100 80GB	$36118.76
Two submissions from the original big-ann-benchmarks competition could not be included in this comparison: Intel's OptANNe GraphANN submission, and GSI Technology's Gemini submission. The former relied on Intel Optane storage technology, which has been discontinued [2] and therefore isn't available on any mainstream cloud compute provider, and can't be priced. The latter leverages proprietary hardware not available to cloud providers. However, both of these entries are present in the throughput-per-capex ranking featured earlier.
Our own results came from a 32 vCPU, 150GB machine, which would cost $1293.09 per month. We achieved 32608 QPS on Microsoft Turing-ANNS and 46712 QPS on Microsoft SPACEV, leading to the numbers presented in Figure 12. These results are also reproduced in tabular form below.
Algorithm	MS-SPACEV Queries/Sec (90% R@10, from here)	MS-SPACEV Throughput / Cost	MS-Turing Queries/Sec (90% R@10, from here)	MS-Turing Throughput / Cost
FAISS Baseline	3265	0.707	2845	0.616
DiskANN	6503	2.876	17201	7.607
CuANNS-IVFPQ	108302	6.753	109745	6.843
CuANNS-Multi	839749	23.25	584293	16.18
Ours	46712	36.12	32608	25.22
<sup>1</sup>The benchmarked machine had eight GPUs, but only one was used, so we only account for one GPU's cost.