SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Philip Sun, David Simcha, Dave Dopson, Ruiqi Guo, and Sanjiv Kumar
{sunphil,dsimcha,ddopson,guorq,sanjivk}@google.com
This paper introduces SOAR: Spilling with Orthogonality-Amplified Residuals, a novel data indexing technique for approximate nearest neighbor (ANN) search. SOAR extends upon previous approaches to ANN search, such as spill trees, that utilize multiple redundant representations while partitioning the data to reduce the probability of missing a nearest neighbor during search. Rather than training and computing these redundant representations independently, however, SOAR uses an orthogonality-amplified residual loss, which optimizes each representation to compensate for cases where other representations perform poorly. This drastically improves the overall index quality, resulting in state-of-the-art ANN benchmark performance while maintaining fast indexing times and low memory consumption.
The  $k$ -nearest neighbor search problem is defined as follows: we are given an  $n$ -item dataset  $\mathcal{X} \in \mathbb{R}^{n \times d}$  composed of  $d$ -dimensional vectors, and for a query  $q \in \mathbb{R}^d$ , we would like to return the  $k$  vectors in  $\mathcal{X}$  closest to  $q$ . This problem naturally arises from a number of scenarios that require fast, online retrieval from vector databases; such applications include recommender systems [7], image search [14], and question answering [10], among many others.
While nearest neighbor search may be easily implemented with a linear scan over the elements of  $\mathcal{X}$ , many applications of nearest neighbor search utilize large datasets for which a brute-force approach is computationally intractable. The rapid development of deep learning and embedding techniques has been an especially strong driver for larger  $k$ -nearest neighbor datasets. For instance, multiple recent large language model (LLM) works incorporate external information by using  $k$ -nearest neighbors to retrieve from longer contexts [17] or large text corpora [5]. While datasets from typical applications a decade ago (e.g. Netflix) had sizes of around 1 million [15], the standard evaluation datasets from big-ann-benchmarks [16] all have 1 billion vectors with hundreds of dimensions.
These large datasets, in conjunction with the curse of dimensionality, a phenomenon which often makes it impossible to find the exact nearest neighbors without resorting to a linear scan, has led to a focus on approximate nearest neighbor (ANN) search, which can trade off a small search accuracy loss for a significant increase in search throughput.
A number of indexing schemes proposed for ANN search, including spill trees [12], assign datapoints to multiple portions of the index, such that the overall index provides a replicated, non-disjoint ("spilled") view of the dataset. These algorithms have used partitioning schemes amenable to random analysis, so that the replication factor can be provably shown to exponentially reduce the probability of missing a nearest neighbor. However, partitioning high-dimensional data is difficult, even without the constraint of doing so with a method amenable to randomized analysis; such constrained partitioning techniques have led to inferior quality indices that have not yielded good compute-accuracy tradeoffs.
On the other hand, another family of approaches, those leveraging vector quantization (VQ), have demonstrated excellent empirical ANN performance, but little research has been done exploring the use of multiple randomized initializations of VQ indices to further increase ANN search efficiency. This is partly due to the fact that multiple VQ indices initialized through different random seeds do not have strong statistical guarantees and cannot be proven independent in their failure probabilities. This paper bridges the gap between these two approaches to ANN:
- We demonstrate the weaknesses of current VQ-based ANN indices and illustrate how multiple VQ indices may be used in conjunction to improve ANN search efficiency.
- We show that the naive training of multiple VQ indices for a single dataset leads to correlation in failure probabilities among the indices, limiting ANN search efficiency uplift, and present spilling with orthogonality-amplified residuals (SOAR) to ameliorate this correlation.
- We benchmark SOAR and achieve state-of-the-art performance, outperforming standard VQ indices, spilled VQ indices trained without SOAR, and all other approaches to ANN search that were submitted to the benchmark.
2.1 Maximum inner product search (MIPS)
SOAR applies to the subclass of nearest neighbor search known as maximum inner product search, defined as follows for a query  $q$  and a dataset  $\mathcal{X}$ :
$$
\operatorname{MIPS}_{k}(q,\mathcal{X}) = k\text{-}\operatorname *{arg  max}_{x\in \mathcal{X}}\left\langle q,x\right\rangle   .
$$
Many nearest neighbor problems arise in the MIPS space naturally [6], and a number of conversions exist [4] from other commonly used ANN search metrics, such as Euclidean and cosine distance, to MIPS, and vice versa. We measure MIPS search accuracy using recall@  $k$ , defined as follows: if our algorithm returns the set  $S$  of  $k$  candidate nearest neighbors, its recall@  $k$  equals  $|\mathrm{MIPS}_k(q,\mathcal{X})\cap S| / k$ . Additionally, we introduce the following notation to assist with MIPS analysis:
$$
\operatorname {R A N K} (q, v, \mathcal {X}) = \sum_ {x \in \mathcal {X}} \mathbb {1} _ {\langle q, v \rangle \leq \langle q, x \rangle}.
$$
The max inner product's RANK is 1; assuming no ties,  $\mathrm{RANK}(q,v,\mathcal{X})\in [1,k]$  for  $v\in \mathrm{MIPS}_k(q,\mathcal{X})$
2.2 Vector quantization (VQ)
Vector quantization can be leveraged to construct data structures that effectively prune the ANN search space; these data structures are commonly known as inverted file indices (IVF) or k-means trees. Vector-quantizing a dataset  $\mathcal{X}$  produces two outputs:
-  $\mathcal{C} \in \mathbb{R}^{c \times d}$ , the codebook containing the  $c$  partition centers.
-  $\pi(v): \mathbb{R}^d \mapsto \{1, \ldots, c\}$ , the partition assignments that map each vector in  $\mathcal{X}$  to one of the partition centers in  $\mathcal{C}$ . Oftentimes,  $\pi(v)$  is defined as  $\arg \min_{i \in \{1, \ldots, c\}} \|v - \mathcal{C}_i\|^2$ , although other assignment functions may also be used.
We can then construct an inverted index over  $\pi$ ; for each partition  $i$ , we store the set of datapoint indices belonging to that partition:  $\{j|\pi (\mathcal{X}_j) = i\}$ . Then, instead of computing  $\mathrm{MIPS}_k(q,\mathcal{X})$  directly, we may first compute  $\mathrm{MIPS}_{k'}(q,\mathcal{C})$ , which is much faster because  $|\mathcal{C}|\ll |\mathcal{X}|$ . We can then use our inverted index data structure to further evaluate the datapoints within the top  $k^{\prime}$  partitions.
2.2.1 The k-means recall (KMR) curve
The effectiveness of the VQ-based pruning approach depends on the rank of the partitions that  $\mathrm{MIPS}_k(q,\mathcal{X})$  are in. If these partitions rank very well, we can set  $k^{\prime}$  very low, and search through few partitions while still achieving great MIPS recall. If these partitions rank poorly, however, the algorithm will have to spend lots of compute searching through many partitions in order to find the nearest neighbors. We may quantify this effectiveness using the  $k$ -means recall (KMR) curve (named
because VQ indices are commonly trained through k-means, although KMR can be computed for any VQ index, not just those trained via k-means), defined as follows:
$$
\operatorname {K M R} _ {k} (t; \mathcal {Q}, \mathcal {X}, \mathcal {C}, \pi) = \frac {1}{k \cdot | \mathcal {Q} |} \sum_ {q \in \mathcal {Q}} \sum_ {v \in \operatorname {M I P S} _ {k} (q, \mathcal {X})} \mathbb {1} _ {\operatorname {R A N K} (q, \mathcal {C} _ {\pi (v)}, \mathcal {C}) \leq t} \tag {1}
$$
The KMR curve for a given query sample  $\mathcal{Q}$ , dataset  $\mathcal{X}$ , and VQ index  $(\mathcal{C},\pi)$  quantifies the proportion of MIPS nearest neighbors present in the top  $t$  VQ partitions, over varying  $t$ . The KMR curve is non-decreasing, with  $\mathrm{KMR}_k(0) = 0$  and  $\mathrm{KMR}_k(|\mathcal{C}|) = 1$ . A KMR curve that more quickly approaches 1 indicates superior index quality.
The key insight of our work is the use of a novel loss function to assign a datapoint  $x$  to multiple VQ partitions, such that these additional partitions effectively recover  $x$  in the pathological cases when  $x$  is a nearest neighbor for a query that  $x$ 's original VQ partition handles poorly. These partitions that include  $x$  work together synergistically to provide greater ANN search efficiency than any single partition could alone. Below, we describe the motivations behind our new loss and this multiple assignment. The supplementary materials contains the source code to generate this section's plots.
3.1 Search difficulty and quantized score error
For a datapoint  $x$ , define  $r$  as the partitioning residual, equal to  $x - \mathcal{C}_{\pi(x)}$ . The partition center  $\mathcal{C}_{\pi(x)}$  is the quantized form of  $x$ , so the difference between the exact and the quantized inner product scores is
$$
\langle q, x \rangle - \left\langle q, \mathcal {C} _ {\pi (x)} \right\rangle = \left\langle q, x - \mathcal {C} _ {\pi (x)} \right\rangle = \left\langle q, r \right\rangle ,
$$
which we denote the quantized score error. Consider the case when  $x$  is a nearest neighbor for the query:  $x \in \mathrm{MIPS}_k(q, \mathcal{X})$ . In this scenario, we would like  $\mathrm{RANK}(q, \mathcal{C}_{\pi(x)}, \mathcal{C})$  to be low, so that the algorithm may search just the top few partitions and find the nearest neighbor  $x$ . A high RANK would lead to increased search difficulty, because the algorithm would have to search more partitions and therefore do more work to find  $x$ .
Given that  $\langle q,\mathcal{C}_{\pi (x)}\rangle = \langle q,x\rangle -\langle q,r\rangle$  and that, by definition,  $\langle q,x\rangle$  is large when  $x\in \mathrm{MIPS}_k(q,\mathcal{X})$  , we know that  $\langle q,\mathcal{C}_{\pi (x)}\rangle$  will be small (leading to greater search dif
ficulty) when  $\langle q,r\rangle$  is highly positive. Indeed, this can be confirmed empirically; in Figure 1 we plot the mean  $\langle q,r\rangle$  as a function of  $\mathrm{RANK}(q,\mathcal{C}_{\pi (x)},\mathcal{C})$  for all query-neighbor pairs  $(q,x)$  in the Glove-1M dataset. The more difficult-to-find pairs have, on average, notably higher  $\langle q,r\rangle$ .
SOAR increases search efficiency in these situations when  $x \in \mathrm{MIPS}_k(q, \mathcal{X})$  and  $\langle q, r \rangle$  is high.
Figure 1: Greater search difficulty, as quantified by a higher  $\mathrm{RANK}(q,\mathcal{C}_{\pi (x)},\mathcal{C})$  , is associated with highly positive  $\langle q,r\rangle$
3.2 Quantized score error decomposition
By the definition of the inner product,
$$
\langle q, r \rangle = \| q \| \cdot \| r \| \cdot \cos \theta
$$
where  $\theta$  is the angle formed between the query and the partitioning residual. Without loss of generality, we may assume that  $\| q\| = 1$  without any effect on the ranking of MIPS nearest neighbors. The two contributors to a highly positive  $\langle q,r\rangle$  are therefore a large  $\| r\|$  and  $\cos \theta$  being near unity; reducing  $\langle q,r\rangle$  requires targeting either, or both, of these contributors.
SOAR targets  $\cos \theta$ , because that term is both easier to reduce and has greater impact on  $\langle q, r \rangle$ :
- The VQ training loss already aims to minimize  $\mathbb{E}[\| x - \mathcal{C}_{\pi (x)}\|^2 ] = \mathbb{E}[\| r\|^2 ]$ , so further reductions in  $\| r\|$  are difficult. In contrast,  $\cos \theta$  is not directly optimized for, making it more amenable to reduction.
-  $\cos \theta$  is typically much more strongly correlated with  $\langle q, r \rangle$  than  $\|r\|$  is, so its reduction has greater impact on search difficulty. Qualitatively, this stronger correlation can be explained by the far greater range of values  $\cos \theta$  may take on, compared to  $\|r\|$ . The latter's values are concentrated between 0 and  $\|x\|$ , with not a very large ratio between the largest and smallest value. In comparison,  $\cos \theta$  can take on any value in the range  $[-1, 1]$ ; only this large multiplicative dynamic range can explain the variance in  $\langle q, r \rangle$ . Empirically, this can be seen demonstrated on Glove-1M in Figure 2 below.
Figure 2: The cosine of the query-residual angle,  $\cos \theta$  (left), is far more correlated with  $\langle q,r\rangle$  than the residual norm  $\| r\|$  (right), making the former a more promising target for reducing  $\langle q,r\rangle$ .
3.3 Spilled VQ assignment
We may attempt to mitigate the increase in search difficulty presented by high  $\cos \theta$  by assigning each datapoint  $x$  to a second VQ partition  $\pi'(x)$ , resulting in a second partitioning residual  $r'$  that forms an angle  $\theta'$  with the query. We denote this second assignment a spilled assignment.
This gives our ANN algorithm a "second chance"; assuming  $\theta$  and  $\theta^{\prime}$  are independently distributed, the situations where  $\cos \theta$  is high, thereby leading to high  $\langle q,r\rangle$  and greater search difficulty, are unlikely to also have a high  $\cos \theta^{\prime}$ . A low  $\cos \theta^{\prime}$  should lead to a low  $\langle q,r^{\prime}\rangle$  and should allow the ANN algorithm to find  $x$  fairly efficiently. This idea can be extended to further  $(>2)$  assignments as well.
This approach leads to some implementation intricacies, as discussed in Section 3.5, but for now we focus on problems concerning theory when the spilled assignment  $\pi'(x)$  is chosen so as to minimize  $||r'||^2$ .
This proposed approach fails when there is correlation between  $\cos \theta$  and  $\cos \theta^{\prime}$ , shown to the extreme in Figure 3. In this figure, if  $\pi (x) = 1$ , we may pick  $\pi^{\prime}(x) = 2$  because  $\mathcal{C}_2$  is the second-closest centroid to  $x$ . However,  $\mathcal{C}_2$  is
collinear with  $\mathcal{C}_1$  and  $x$ , resulting in  $\theta = \theta'$ . This leads to  $\mathcal{C}_2$  acting as a strictly worse version of  $\mathcal{C}_1$  for quantizing  $x$ :  $\langle q, r' \rangle = (1 + \epsilon) \langle q, r \rangle$ . The choice  $\pi'(x) = 3$  would've been more effective, despite  $\| x - \mathcal{C}_3 \| > \| x - \mathcal{C}_2 \|$ , due to  $\mathcal{C}_3$ 's position giving a  $\theta' \neq \theta$ .
Figure 3: Naive spilled VQ assignment may be ineffective; selecting the two closest centroids  $\mathcal{C}_1$  and  $\mathcal{C}_2$  provides no benefit over using just  $\mathcal{C}_1$ .
Experiments show that more than a theoretical concern, this is also a real-world issue. On the Glove-1M dataset, if we choose  $\pi(x)$  and  $\pi'(x)$  to be the closest and second-closest centroids by Euclidean distance, respectively, to  $x$ , we find a noticeable correlation between  $\cos \theta$  and  $\cos \theta'$ , shown in Figure 4a. An even stronger correlation occurs between two VQ indices trained separately with different random seeds (Figure 4b). This correlation diminishes the benefit of spilled assignment, and reducing this correlation is critical to improving ANN search performance.
Figure 4: On Glove-1M, both a naive top-2 spilled assignment and two separately trained VQ indices exhibit noticeable correlation in query-residual angles; this reduces spilled assignment efficacy.
3.4 Spilling with orthogonality-amplified residuals
We now derive spilling with orthogonality-amplified residuals (SOAR), a novel VQ assignment loss that directly confronts the problem of correlated query-residual angles. To derive this loss, assume that the VQ centroids  $\mathcal{C}$  and assignments  $\pi$  are fixed. The goal of further spilled assignments  $\pi'$  is to achieve low quantized score error  $\langle q, r' \rangle$  specifically when the original spilled assignment has high  $\cos \theta$ , leading to high  $\langle q, r \rangle$ . In cases where  $\cos \theta$  is low, the original assignments already achieve low quantized score error, so in such cases, it is less important to also have low  $\langle q, r' \rangle$ .
We now modify our assignment loss function to reflect this emphasis; rather than selecting  $c' \in \mathcal{C}$  to minimize  $\mathbb{E}_{q \in \mathcal{Q}}[(\langle q, x \rangle - \langle q, c' \rangle)^2] = \mathbb{E}_{q \in \mathcal{Q}}[\langle q, r' \rangle^2]$ , we now also add a weighting term for  $\cos \theta$ :
$$
\mathcal {L} \left(r ^ {\prime}, r, \mathcal {Q}\right) = \mathbb {E} _ {q \in \mathcal {Q}} \left[ w \left(\frac {\langle q , r \rangle}{\| r \| \cdot \| q \|}\right) \left\langle q, r ^ {\prime} \right\rangle^ {2} \right] = \mathbb {E} _ {q \in \mathcal {Q}} \left[ w (\cos \theta) \left\langle q, r ^ {\prime} \right\rangle^ {2} \right], \tag {2}
$$
where the weight function  $w(\cdot):\mathbb{R}\mapsto \mathbb{R}^{\geq 0}$  should be chosen to give greater emphasis to higher  $\cos \theta$ . To utilize this loss function in VQ assignment, we must evaluate the expectation, which leads to the following result:
Theorem 3.1. For the weight function  $w(t) = |t|^{\lambda}$  and a query distribution  $\mathcal{Q}$  that is uniformly distributed over the unit hypersphere,
$$
\mathcal {L} \left(r ^ {\prime}, r, \mathcal {Q}\right) \propto \left\| r ^ {\prime} \right\| ^ {2} + \lambda \left\| \operatorname {p r o j} _ {r} r ^ {\prime} \right\| ^ {2}.
$$
By leveraging Theorem 3.1, we may efficiently perform spilled VQ assignment using our new loss by computing the squared  $\ell_2$  distance to each centroid, adding a penalty term for parallelism between the original residual  $r$  and the candidate residual  $r'$ , and taking the arg min among all centroids. We present the following statements to help develop intuition about the SOAR loss:
Corollary 3.1.1. For the uniform weight function  $w(t) = 1 = |t|^0$ , the SOAR loss is equivalent to standard Euclidean assignment:  $\mathcal{L}(r', r, \mathcal{Q}) \propto \| r' \|^2$ .
The parameter  $\lambda$  controls how much the spilled assignment  $\pi^{\prime}$  should prioritize quantization performance when the original assignment performs poorly, relative to general quantization performance. As  $\lambda$  is increased, the former is further emphasized; at  $\lambda = 0$ , only the latter is considered, leading to standard Euclidean assignment. See Figure 9 in Experiments for a visualization of this tradeoff.
Corollary 3.1.2. For a fixed  $\| r^{\prime}\|$ ,  $\mathcal{L}$  is minimized when  $r$  and  $r^\prime$  are orthogonal, in which case  $\lambda \| \mathrm{proj}_r r^\prime \| ^2 = 0$  and  $\mathcal{L}(r',r,\mathcal{Q})\propto \| r'\|^2$ .
Our technique's name originates from the observation that residuals are encouraged to be orthogonal.
Lemma 3.2.  $\| \mathrm{proj}_r r' \| = \| r' \| \cdot \rho_{\langle q, r \rangle, \langle q, r' \rangle}$ , where  $\rho$  is the Pearson correlation coefficient, and the correlation is computed over  $q$  uniformly distributed over the hypersphere.
The above lemma shows that in addition to the weighted quantization error derivation in Equation 2, we may also interpret the SOAR loss as adding a penalization term for correlation in quantized score error, scaled by the magnitude of the quantization error.
$$
\rho_ {\langle q, r \rangle , \langle q, r ^ {\prime} \rangle} = \frac {\operatorname {C o v} [ \langle q , r \rangle , \langle q , r ^ {\prime} \rangle ]}{\sigma_ {\langle q , r \rangle} \sigma_ {\langle q , r ^ {\prime} \rangle}} = \frac {\mathbb {E} [ \langle q , r \rangle \langle q , r ^ {\prime} \rangle ] - \mathbb {E} [ \langle q , r \rangle ] \mathbb {E} [ \langle q , r ^ {\prime} \rangle ]}{\sigma_ {\langle q , r \rangle} \sigma_ {\langle q , r ^ {\prime} \rangle}},
$$
and by symmetry over the hypersphere,  $\mathbb{E}[\langle q,r\rangle ] = \mathbb{E}[\langle q,r^{\prime}\rangle ] = 0$  so our expression simplifies to
$$
\rho_ {\langle q, r \rangle , \langle q, r ^ {\prime} \rangle} = \frac {\mathbb {E} [ \langle q , r \rangle \langle q , r ^ {\prime} \rangle ]}{\sigma_ {\langle q , r \rangle} \sigma_ {\langle q , r ^ {\prime} \rangle}} = \frac {\mathbb {E} [ r ^ {T} q q ^ {T} r ^ {\prime} ]}{\sigma_ {\langle q , r \rangle} \sigma_ {\langle q , r ^ {\prime} \rangle}} = \frac {r ^ {T} \mathbb {E} [ q q ^ {T} ] r ^ {\prime}}{\sigma_ {\langle q , r \rangle} \sigma_ {\langle q , r ^ {\prime} \rangle}}.
$$
It's well known that for uniformly distributed, unit-norm  $q \in \mathbb{R}^d$  that  $\mathbb{E}[qq^T] = I_d / d$ , where  $I_d$  is the  $d$ -dimensional identity matrix, and that  $\sigma_{\langle q,v\rangle} = \| v\| /\sqrt{d}$  for any fixed  $v$ . We leverage this to get
$$
\rho_ {\langle q, r \rangle , \langle q, r ^ {\prime} \rangle} = \frac {r ^ {T} (I _ {d} / d) r ^ {\prime}}{(\| r \| / \sqrt {d}) (\| r ^ {\prime} \| / \sqrt {d})} = \frac {r ^ {T} r ^ {\prime}}{\| r \| \cdot \| r ^ {\prime} \|} = \left\langle \frac {r}{\| r \|}, \frac {r ^ {\prime}}{\| r ^ {\prime} \|} \right\rangle = \frac {1}{\| r ^ {\prime} \|} \| \mathrm {p r o j} _ {r} r ^ {\prime} \|.
$$
3.5 Implementation considerations
SOAR comes with some memory overhead relative to a non-spilled VQ MIPS index, due to its multiple assignments causing some duplication of data. The overhead is quite negligible because only the product-quantized (PQ) [9] datapoint representation, not the datapoint's highest-bitrate representation (usually int8 or float32), is duplicated. Since PQ is much more compressed than int8 or float32, duplicating the PQ data for each of a datapoint's spilled assignments only marginally increases memory footprint, as illustrated in Figure 5. This can be quantified analytically:
- SOAR increases memory consumption by  $4 + \frac{d}{2s}$  bytes/datapoint, assuming 16 centers per subspace (usually chosen for amenability to SIMD) and  $s$  dimensions per PQ subspace.
- The datapoint's original assignment already contained a PQ representation occupying  $4 + \frac{d}{2s}$  bytes, and its highest-bitrate representation requires  $d$  bytes (int8) or  $4d$  bytes (float32).
When the ANN index's highest-bitrate datapoint representation is int8 or float32, the relative index size growth is therefore  $\frac{4 + \frac{d}{2s}}{d + 4 + \frac{d}{2s}} \approx \frac{\frac{d}{2s}}{d + \frac{d}{2s}} = \frac{1}{2s + 1}$  or  $\frac{4 + \frac{d}{2s}}{4d + 4 + \frac{d}{2s}} \approx \frac{\frac{d}{2s}}{4d + \frac{d}{2s}} = \frac{1}{8s + 1}$ , respectively. Table 1 empirically corroborates these estimates and shows the memory cost is indeed quite low (5%-20% relative increase over a non-spilled VQ index).
Figure 5: Memory layout changes for a SOAR-enabled ANN index. Memory footprint is proportional to area of colored cell. VQ centroid data (not shown) remains unchanged. We can see that the additional memory occupied by SOAR (dark blue) is low relative to the total memory consumption.
SOAR also results in a small amount of query-time CPU overhead over a non-spilled VQ index, due to SOAR's need for dedduplication: a single datapoint may now appear in multiple partitions that are all searched. This overhead is low, though, because VQ inverted index performance is primarily bottlenecked by memory accesses to the VQ partition data, and this access pattern remains the same.
Finally, SOAR requires a minor tweak to the indexing pipeline. Creating a SOAR-enabled index first requires training a standard, non-spilled VQ index as usual. This gets used to populate the primary assignments, but is also used to calculate the residual  $r$ , which is then used in the SOAR loss to select further, spilled assignments. Other parts of the ANN indexing pipeline remain unchanged.
Overall, SOAR's search efficiency gains greatly outweigh its memory, CPU, and indexing overheads.
3.5.1 Spilling to further centroids