SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where correct_pred@k is the number of k-NN with correctly predicted labels for a given query. As noted in Section 3, k-Recall@N is the overlap between  $k$  exact search nearest neighbors (considered as ground truth) and the top N retrieved documents. As Faisss [24] supports a maximum of 2048-NN while searching the indexed database, we report 40-Recall@2048 in Figure 13. Also note that for ImageNet-1K, which constitutes a bulk of the experimentation in this work,  $|Q| = 50000$ ,  $|N_{D}| = 1281167$  and num_classeses  $= 1000$ . For ImageNetv2 [44],  $|Q| = 10000$  and num_classeses  $= 1000$ , and for ImageNet-4K [31],  $|Q| = 210100$ ,  $|N_{D}| = 4202000$  and num_classeses  $= 4202$ . For NQ [32],  $|Q| = 3610$  and  $|N_{D}| = 21015324$ . As NQ consists of question-answer pairs (instance-level), num_classeses  $= 3610$  for the test set.
(a) Exact Search  $^+$  OPQ on ImageNet-1K
(b) IVF + OPQ on ImageNet-1K
(c) DiskANN + OPQ on ImageNet-1K
(d) HNSW + OPQ on ImageNet-1K
Figure 6: Top-1 Accuracy of AdANNS composite indices with OPQ distance computation compared to MR and Rigid baselines models on ImageNet-1K and Natural Questions.
D AdANNS-OPQ
In this section, we take a deeper dive into the quantization characteristics of MR. In this work, we restrict our focus to optimized product quantization (OPQ) [13], which adds a learned space rotation and dimensionality permutation to PQ's sub-vector quantization to learn more optimal PQ codes. We compare OPQ to vanilla PQ on ImageNet in Table 4, and observe large gains at larger embedding dimensionalities, which agrees with the findings of Jayaram Subramanya et al. [22].
We perform a study of composite OPQ  $m \times b$  indices on ImageNet-1K across compression compute budgets  $m$  (where  $b = 8$ , i.e. 1 Byte), i.e. Exact Search with OPQ, IVF+OPQ, HNSW+OPQ, and DiskANN+OPQ, as seen in Figure 6. It is evident from these results:
1. Learning OPQ codebooks with AdANNS (Figure 6a) provides a  $1 - 5\%$  gain in top-1 accuracy over rigid representations at low compute budgets ( $\leq 32$  Bytes). AdANNS-OPQ saturates to Rigid-OPQ performance at low compression ( $\geq 64$  Bytes).
2. For IVF, learning clusters with MRs instead of RRs (Figure 6b) provides substantial gains (1-4%). In contrast to Exact-OPQ, using AdANNS for learning OPQ codebooks does not provide substantial top-1 accuracy gains over MR with  $d = 2048$  (highest), though it is still slightly better or equal to MR-2048 at all compute budgets. This further supports that IVF performance generally scales with embedding dimensionality, which is consistent with our findings on ImageNet across robustness variants and encoders (See Figures 9 and 15 respectively).
3. Note that in contrast to Exact, IVF, and HNSW coarse quantizers, DiskANN inherently re-ranks the retrieved shortlist with high-precision embeddings ( $d = 2048$ ), which is reflected in its high top-1 accuracy. We find that AdANNS with 8-byte OPQ (Figure 6c) matches the top-1 accuracy of rigid representations using 32-byte OPQ, for a  $4 \times$  cost reduction for the same accuracy. Also note that using AdANNS provides large gains over using MR-2048 at high compression ( $1.5\%$ ), highlighting the necessity of AdANNS's flexibility for high-precision retrieval at low compute budgets.
4. Our findings on the HNSW-OPQ composite index (Figure 6d) are consistent with all other indices, i.e. HNSW graphs constructed with AdANNS OPQ codebooks provide significant gains over RR and MR, especially at high compression ( $\leq$  32 Bytes).
OPQ on NQ dataset
(a) Exact Search + OPQ on Natural Questions
(b) IVF + OPQ on Natural Questions
Figure 7: Top-1 Accuracy of AdANNS composite indices with OPQ distance computation compared to MR and Rigid baselines models on Natural Questions.
Our observations on ImageNet with ResNet-50 MR across search structures also extend to the Natural Questions dataset with Dense Passage Retriever (DPR with BERT-Base MR embeddings). We note that AdANNS provides gains over RR-768 embeddings for both Exact Search and IVF with OPQ (Figure 7a and 7b). We find that similar to ImageNet (Figure 15) IVF performance on Natural Questions generally scales with dimensionality. AdANNS thus reduces to MR-768 performance for  $M \geq 16$ . See Appendix G for a more in-depth discussion of AdANNS with DPR on Natural Questions.
Table 4: Comparison of PQ-MR with OPQ-MR for exact search on ImageNet-1K across embedding dimensionality  $d \in \{8,16,\ldots,2048\}$  quantized to  $m \in \{8,16,32,64\}$  bytes. OPQ shows large gains over vanilla PQ at larger embedding dimensionalities  $d \geq 128$ . Entries with the highest top-1 accuracy for a given  $(d,m)$  tuple are bolded.
Config	PQ	OPQ
d	m	Top-1	mAP@10	P@100	Top-1	mAP@10	P@100
8	8	62.18	56.71	61.23	62.22	56.70	61.23
16	8	67.91	62.85	67.21	67.88	62.96	67.21
16	67.85	62.95	67.21	67.96	62.94	67.21
32	8	68.80	63.62	67.86	68.91	63.63	67.86
16	69.57	64.22	68.12	69.47	64.20	68.12
32	69.44	64.20	68.12	69.47	64.23	68.12
64	8	68.39	63.40	67.47	68.38	63.42	67.60
16	69.77	64.43	68.25	69.95	64.55	68.38
32	70.13	64.67	68.38	70.05	64.65	68.38
64	70.12	64.69	68.42	70.18	64.70	68.38
128	8	67.27	61.99	65.78	68.40	63.11	67.34
16	69.51	64.32	68.12	69.78	64.56	68.38
32	70.27	64.72	68.51	70.60	64.97	68.51
64	70.61	64.93	68.49	70.65	64.98	68.51
256	8	66.06	60.44	64.09	67.90	62.69	66.95
16	68.56	63.33	66.95	69.92	64.71	68.51
32	70.08	64.83	68.38	70.59	65.15	68.64
64	70.48	64.98	68.55	70.69	65.09	68.64
512	8	65.09	59.03	62.53	67.51	62.12	66.56
16	67.68	62.11	65.39	69.67	64.53	68.38
32	69.51	64.01	67.34	70.44	65.11	68.64
64	70.53	65.02	68.52	70.72	65.17	68.64
1024	8	64.58	58.26	61.75	67.26	62.07	66.56
16	66.84	61.07	64.09	69.34	64.23	68.12
32	68.71	62.92	66.04	70.43	65.03	68.64
64	69.88	64.35	67.68	70.81	65.19	68.64
2048	8	62.19	56.11	59.80	66.89	61.69	66.30
16	65.99	60.27	63.18	69.25	64.09	67.99
32	67.99	62.04	64.74	70.39	64.97	68.51
64	69.20	63.46	66.40	70.57	65.15	68.51
E AdANNs-IVF
Inverted file index (IVF) [48] is a simple yet powerful ANNS data structure used in web-scale search systems [16]. IVF construction involves clustering (coarse quantization often through k-means) [36] on  $d$ -dimensional representation that results in an inverted file list [53] of all the data points in each cluster. During search, the  $d$ -dimensional query representation is first assigned to the closest clusters (# probes, typically set to 1) and then an exhaustive linear scan happens within each cluster to obtain the nearest neighbors. As seen in Figure 9, IVF top-1 accuracy scales logarithmically with increasing representation dimensionality  $d$  on ImageNet-1K/V2/4K. The learned low- $d$  representations thus provide better accuracy-compute trade-offs compared to high- $d$  representations, thus furthering the case for usage of AdANNS with IVF.
Our proposed adaptive variant of IVF, AdANNS-IVF, decouples the clustering, with  $d_{c}$  dimensions, and the linear scan within each cluster, with  $d_{s}$  dimensions - setting  $d_{c} = d_{s}$  results in non-adaptive vanilla IVF. This helps in the smooth search of design space for the optimal accuracy-compute trade-off. A naive instantiation yet strong baseline would be to use explicitly trained  $d_{c}$  and  $d_{s}$  dimensional rigid representations (called MG-IVF-RR, for multi-granular IVF with
Figure 8: Top-1 accuracy vs. compute cost per query of AdANNS-IVF compared to IVF-MR, IVF-RR and MG-IVF-RR baselines on ImageNet-1K.
rigid representations). We also examine the setting of adaptively choosing low-dimensional MR to linear scan the shortlisted clusters built with high-dimensional MR, i.e. AdANNS-IVF-D, as seen in Table 5. As seen in Figure 8, AdANNS-IVF provides pareto-optimal accuracy-compute tradeoff across inference compute. This figure is a more exhaustive indication of AdANNS-IVF behavior compared to baselines than Figures 1a and 2. AdANNS-IVF is evaluated for all possible tuples of  $d_c, d_s, k = |C| \in \{8, 16, \dots, 2048\}$ . AdANNS-IVF-D is evaluated for a pre-built IVF index with  $d_c = 2048$  and  $d_s \in \{8, \dots, 2048\}$ . MG-IVF-RR configurations are evaluated for  $d_c \in \{8, \dots, d_s\}$ ,  $d_s \in \{32, \dots, 2048\}$  and  $k = 1024$  clusters. A study over additional  $k$  values is omitted due to high compute cost. Finally, IVF-MR and IVF-RR configurations are evaluated for  $d_c = d_s \in \{8, 16, \dots, 2048\}$  and  $k \in \{256, \dots, 8192\}$ . Note that for a fair comparison, we use  $n_p = 1$  across all configurations. We discuss the inference compute for these settings in Appendix B.1.
E.1 Robustness
(a) ImageNet-1K
(b) ImageNetV2
Figure 9: Top-1 Accuracy variation of IVF-MR of ImageNet 1K, ImageNetV2 and ImageNet-4K. RR baselines are omitted on ImageNet-4K due to high compute cost.
(c) ImageNet-4K
As shown in Figure 9, we examined the clustering capabilities of MRs on both in-distribution (ID) queries via ImageNet-1K and out-of-distribution (OOD) queries via ImageNetV2 [44], as well as on larger-scale ImageNet-4K [31]. For ID queries on ImageNet-1K (Figure 9a), IVF-MR is at least as accurate as Exact-RR for  $d \leq 256$  with a single search probe, demonstrating the quality of in-distribution low-d clustering with MR. On OOD queries (Figure 9b), we observe that IVF-MR is on average  $2\%$  more robust than IVF-RR across all cluster construction and linear scan dimensionalities  $d$ . It is also notable that clustering with MRs followed by linear scan with # probes  $= 1$  is more robust than exact search with RR embeddings across all  $d \leq 2048$ , indicating the adaptability of MRs to distribution shifts during inference. As seen in Table 5, on ImageNetV2 AdANNS-IVF-D is the best configuration for  $d \leq 16$ , and is similarly accurate to IVF-MR at all other  $d$ . AdANNS-IVF-D with
Table 5: Top-1 Accuracy of AdANNS-IVF-D on out-of-distribution queries from ImageNetV2 compared to both IVF and Exact Search with MR and RR embeddings. Note that for AdANNS-IVF-D, the dimensionality used to build clusters  $d_{c} = 2048$ .
d	AdANNS-IVF-D	IVF-MR	Exact-MR	IVF-RR	Exact-RR
8	53.51	50.44	50.41	49.03	48.79
16	57.32	56.35	56.64	55.04	55.08
32	57.32	57.64	57.96	56.06	56.69
64	57.85	58.01	58.94	56.84	57.37
128	58.02	58.09	59.13	56.14	57.17
256	58.01	58.33	59.18	55.60	57.09
512	58.03	57.84	59.40	55.46	57.12
1024	57.66	57.58	59.11	54.80	57.53
2048	58.04	58.04	59.63	56.17	57.84
$d = 128$  is able to match its own accuracy with  $d = 2048$ , a  $16 \times$  compute gain during inference. This demonstrates the potential of AdANNS to adaptively search pre-indexed clustering structures.
On 4-million scale ImageNet-4K (Figure 9c), we observe similar accuracy trends of IVF-MR compared to Exact-MR as in ImageNet-1K (Figure 9a) and ImageNetV2 (Figure 9b). We omit baseline IVF-RR and Exact-RR experiments due to high compute cost at larger scale.
E.2 IVF-MR Ablations
(a) 4K Search Probes  $(n_p)$
(b) Centroid Recall
Figure 10: Ablations on IVF-MR Clustering: a) Analysis of accuracy-compute tradeoff with increasing IVF-MR search probes  $n_p$  on ImageNet-4K compared to Exact-MR and b) k-Recall@N on ImageNet-1K cluster centroids across representation sizes  $d$ . Cluster centroids retrieved with highest embedding dim  $d = 2048$  were considered ground-truth centroids.
As seen in Figure 10a, IVF-MR can match the accuracy of Exact Search on ImageNet-4K with  $\sim 100\times$  less compute. We also explored the capability of MRs at retrieving cluster centroids with low-d compared to a ground truth of 2048-d with k-Recall@N, as seen in Figure 10b. MRs were able to saturate to near-perfect 1-Recall@N for  $d\geq 32$  and  $N\geq 4$  , indicating the potential of AdANNS at matching exact search performance with less than 10 search probes  $n_p$
E.3 Clustering Distribution
We examined the distribution of learnt clusters across embedding dimensionalities  $d$  for both MR and RR models, as seen in Figure 11. We observe IVF-MR to have less variance than IVF-RR at  $d \in \{8, 16\}$ , and slightly higher variance for  $d \geq 32$ , while IVF-MR outperforms IVF-RR in top-1 across all  $d$  (Figure 9a). This indicates that although MR learns clusters that are less uniformly distributed than RR at high  $d$ , the quality of learnt clustering is superior to RR across all  $d$ . Note that a uniform distribution is  $N / k$  data points per cluster, i.e.  $\sim 1250$  for ImageNet-1K with  $k = 1024$ . We quantitatively evaluate the proximity of the MR and RR clustering distributions with Total Variation
Figure 11: Clustering distributions for IVF-MR and IVF-RR across embedding dimensionality  $d$  on ImageNet-1K. An IVF-MR and IVF-RR clustered with  $d = 16$  embeddings is denoted by MR-16 and RR-16 respectively.
Distance [33], which is defined over two discrete probability distributions  $p, q$  over  $[n]$  as follows:
$$
d _ {T V} (p, q) = \frac {1}{2} \sum_ {i \in [ n ]} | p _ {i} - q _ {i} |
$$
We also compute  $d_{TV,2048}(\mathrm{MR - d}) = d_{TV}(\mathrm{MR - d},\mathrm{RR - 2048})$ , which evaluates the total variation distance of a given low-d MR from high-d RR-2048. We observe a monotonically decreasing  $d_{TV,2048}$  with increasing  $d$ , which demonstrates that MR clustering distributions get closer to RR-2048 as we increase the embedding dimensionality  $d$ . We observe in Figure 11 that  $d_{TV}(\mathrm{MR - d},\mathrm{RR - d})\sim 7e - 4$  for  $d\in \{8,256,\dots ,2048\}$  and  $\sim 3e - 4$  for  $d\in \{16,32,64\}$ . These findings agree with the top-1 improvement of MR over RR as shown in Figure 9a, where there are smaller improvements for  $d\in \{16,32,64\}$  (smaller  $d_{TV}$ ) and larger improvements for  $d\in \{8,256,\dots ,2048\}$  (larger  $d_{TV}$ ). These results demonstrate a correlation between top-1 performance of IVF-MR and the quality of clusters learnt with MR.
F AdANNS-DiskANN
DiskANN is a state-of-the-art graph-based ANNS index capable of serving queries from both RAM and SSD. DiskANN builds a greedy best-first graph with OPQ distance computation, with compressed vectors stored in memory. The index and full-precision vectors are stored on the SSD. During search,
Table 6: Wall clock search latency  $(\mu s)$  of AdANNS-DiskANN across graph construction dimensionality  $d\in \{8,16,\dots ,2048\}$  and compute budget in terms of OPQ budget  $M\in \{8,16,32,48,64\}$ . Search latency is fairly consistent across fixed embedding dimensionality  $D$
d	M=8	M=16	M=32	M=48	M=64
8	495	-	-	-	-
16	555	571	-	-	-
32	669	655	653	-	-
64	864	855	843	844	848
128	1182	1311	1156	1161	2011
256	1923	1779	1744	2849	1818
512	2802	3272	3423	2780	3171
1024	5127	5456	5724	4683	5087
2048	9907	9833	10205	10183	9329
when a query's neighbor shortlist is fetched from the SSD, its full-precision vector is also fetched in a single disk read. This enables efficient and fast distance computation with PQ on a large initial shortlist of candidate nearest neighbors in RAM followed by a high-precision re-ranking with full-precision vectors fetched from the SSD on a much smaller shortlist. The experiments carried out in this work primarily utilize a DiskANN graph index built in-memory<sup>5</sup> with OPQ distance computation.
As with IVF, DiskANN is also well suited to the flexibility provided by AdANNS as we demonstrate on both ImageNet and NQ that the optimal PQ codebook for a given compute budget is learnt with a smaller embedding dimensionality  $d$  (see Figures 6c and 7a). We demonstrate the capability of AdANNS-DiskANN with a compute budget of  $m \in \{32,64\}$  in Table 1. We tabulate the search time latency of AdANNS-DiskANN in microseconds  $(\mu s)$  in Table 6, which grows linearly with graph construction dimensionality  $d$ . We also examine DiskANN-MR with SSD graph indices on ImageNet-1K across OPQ budgets for distance computation  $m_{dc} \in \{32,48,64\}$ , as seen in Figure 12. With SSD indices, we store PQ-compressed vectors on disk with  $m_{disk} = m_{dc}$ , which essentially disables DiskANN's implicit high-precision re-ranking. We ob
serve similar trends to other composite ANNS indices on ImageNet, where the optimal dim for fixed OPQ budget is not the highest dim ( $d = 1024$  with fp32 embeddings is current highest dim supported by DiskANN which stores vectors in 4KB sectors on disk). This provides further motivation for AdANNS-DiskANN, which leverages MRs to provide flexible access to the optimal dim for quantization and thus enables similar Top-1 accuracy to Rigid DiskANN for up to  $1/4$  the cost (Figure 6c).
Figure 12: DiskANN-MR with SSD indices for ImageNet-1K retrieval, with compute budgets  $m_{disk} = m_{dc} \in \{32,48,64\}$  across graph and OPQ codebook construction. dimensionalities  $d \in \{32,\dots ,1024\}$ . Note that this does not use any re-ranking after obtaining OPQ based shortlist.
G AdANNS on Natural Questions
In addition to image retrieval on ImageNet, we also experiment with dense passage retrieval (DPR) on Natural Questions. As shown in Figure 6, MR representations are  $1 - 10\%$  more accurate than their
5https://github.com/microsoft/DiskANN
RR counterparts across PQ compute budgets with Exact Search + OPQ on NQ. We also demonstrate that IVF-MR is  $1 - 2.5\%$  better than IVF-RR for Precision@k,  $k \in \{1, 5, 20, 100, 200\}$ . Note that on NQ, IVF loses  $\sim 10\%$  accuracy compared to exact search, even with the RR-768 baseline. We hypothesize the weak performance of IVF owing to poor clusterability of the BERT-Base embeddings fine-tuned on the NQ dataset. A more thorough exploration of AdANNS-IVF on NQ is an immediate future work and is in progress.
H Ablations
H.1 Recall Score Analysis
Figure 13: k-Recall@N of  $d$ -dimensional MR for IVF and HNSW with increasing search probes  $n_p$  on ImageNet-1K and ImageNet-4K. On ImageNet-4K, we restrict our study to IVF-MR with  $d \in \{8,64,256,2048\}$ . Other embedding dimensionalities, HNSW-MR and RR baselines are omitted due to high compute cost. We observe that trends from ImageNet-1K with increasing  $d$  and  $n_p$  extend to ImageNet-4K, which is  $4 \times$  larger.
Figure 14: Relative contrast of varying capacity MRs and RRs on ImageNet-1K corroborating the findings of He et al. [18].
In this section we also examine the variation of k-Recall@N with by probing a larger search space with IVF and HNSW indices. For IVF, search probes represent the number of clusters shortlisted for linear scan during inference. For HNSW, search quality is controlled by the efSearch parameter [38], which represents the closest neighbors to query  $q$  at level  $l_{c}$  of the graph and is analogous to number of search probes in IVF. As seen in Figure 13, general trends show a) an intuitive increase in recall with increasing search probes  $n_p$  for fixed search probes, b) a decrease in recall with increasing search dimensionality  $d$  c) similar trends in ImageNet-1K and  $4\times$  larger ImageNet-4K.
H.2 Relative Contrast
We utilize Relative Contrast [18] to capture the difficulty of nearest neighbors search with IVF-MR compared to IVF-RR. For a given database  $X = \{x_{i} \in \mathbb{R}^{d}, i = 1, \dots, N_{D}\}$ , a query  $q \in \mathbb{R}^{d}$ , and a distance metric  $D(.,.)$  we compute relative contrast  $C_{r}$  as a measure of the difficulty in finding the 1-nearest neighbor (1-NN) for a query  $q$  in database  $X$  as follows:
1. Compute  $D_{\min}^{q} = \min_{i=1\dots n}D(q,x_{i})$ , i.e. the distance of query  $q$  to its nearest neighbor  $x_{nn}^{q} \in X$ .
2. Compute  $D_{mean}^{q} = E_{x}[D(q,x)]$  as the average distance of query  $q$  from all database points  $x \in X$
3. Relative Contrast of a given query  $C_r^q = \frac{D_{mean}^q}{D_{min}^q}$ , which is a measure of how separable the query's nearest neighbor  $x_{nn}^q$  is from an average point in the database  $x$
4. Compute an expectation over all queries for Relative Contrast over the entire database as
$$
C _ {r} = \frac {E _ {q} [ D _ {m e a n} ^ {q} ]}{E _ {q} [ D _ {m i n} ^ {q} ]}
$$
It is evident that  $C_r$  captures the difficulty of Nearest Neighbor Search in database  $X$ , as a  $C_r \sim 1$  indicates that for an average query, its nearest neighbor is almost equidistant from a random point in the database. As demonstrated in Figure 14, MRs have higher  $R_c$  than RR Embeddings for an Exact Search on ImageNet-1K for all  $d \geq 16$ . This result implies that a portion of MR's improvement over RR for 1-NN retrieval across all embedding dimensionalities  $d$  [31] is due to a higher average separability of the MR 1-NN from a random database point.
H.3 Generality across Encoders
We perform an ablation over the representation function  $\phi : X \to \mathbb{R}^d$  learnt via a backbone neural network (primarily ResNet50 in this work), as detailed in Section 3. We also train MRL models [31]  $\phi^{MR(d)}$  on ResNet18/34/101 [19] that are as accurate as their independently trained RR baseline models  $\phi^{RR(d)}$ , where  $d$  is the default max representation size of each architecture. We also train
MRL with a ConvNeXt-Tiny backbone with  $[d] = \{48,96,192,384,786\}$ . MR-768 has a top-1 accuracy of  $79.45\%$  compared to independently trained publicly available RR-768 baseline with top-1 accuracy  $82.1\%$  (Code and RR model available on the official repo<sup>6</sup>). We note that this training had no hyperparameter tuning whatsoever, and this gap can be closed with additional model training effort. We then compare clustering the MRs via IVF-MR with  $k = 2048$ ,  $n_p = 1$  on ImageNet-1K to Exact-MR, which is shown in Figure 15. IVF-MR shows similar trends across backbones compared to Exact-MR, i.e. a maximum top-1 accuracy drop of  $\sim 1.6\%$  for a single search probe. This suggests the clustering capabilities of MR extend beyond an inductive bias of  $\phi^{MR(d)} \in \mathrm{ResNet}50$ , though we leave a detailed exploration for future work.
Figure 15: Top-1 Accuracy variation of IVF-MR on ImageNet-1K with different embedding representation function  $\phi^{MR(d)}$  (see Section 3), where  $\phi \in \{\mathrm{ResNet}18/34/101, \mathrm{ConvNeXt-Tiny}\}$ . We observe similar trends between IVF-MR and Exact-MR on ResNet18/34/101 when compared to ResNet50 (Figure 9a) which is the default in all experiments in this work.
$^{6}$ https://github.com/facebookresearch/ConvNeXt