SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our experiments on regular IVF with MRs and RRs (IVF-MR & IVF-RR) of varying dimensionalities and IVF configurations (# clusters, # probes) show that (Figure 2) matryoshka representations result in a significantly better accuracy-compute trade-off. We further studied and found that learned lower-dimensional representations offer better accuracy-compute trade-offs for IVF than higher-dimensional embeddings (see Appendix E for more results).
AdANNS utilizes  $d$ -dimensional matryoshka representation to get accurate  $d_{c}$  and  $d_{s}$  dimensional vectors at no extra compute cost. The resulting AdANNS-IVF provides a much better accuracy-compute trade-off (Figure 2) on ImageNet-1K retrieval compared to IVF-MR, IVF-RR, and MG-IVF-RR - multi-granular IVF with rigid representations (akin to AdANNS without MR) - a strong baseline that uses  $d_{c}$  and  $d_{s}$  dimensional RRs. Finally, we exhaustively search the design space of IVF by varying  $d_{c}, d_{s} \in [8, 16, \dots, 2048]$  and the number of clusters  $k \in [8, 16, \dots, 2048]$ . Please see Appendix E for more details. For IVF experiments on the NQ dataset, please refer to Appendix G.
Empirical results. Figure 2 shows that AdANNS-IVF outperforms the baselines across all accuracy-compute settings for ImageNet-1K retrieval. AdANNS-IVF results in  $10 \times$  lower compute for the best accuracy of the extremely expensive MG-IVF-RR and non-adaptive IVF-MR. Specifically, as shown in Figure 1a, AdANNS-IVF is up to  $1.5\%$  more accurate for the same compute and has up to  $100 \times$  lesser FLOPS/query ( $90 \times$  real-world speed-up!) than the status quo ANNS on rigid representations (IVF-RR). We filter out points for the sake of presentation and encourage the reader to check out Figure 8 in Appendix E for an expansive plot of all the configurations searched.
The advantage of AdANNS for construction of search structures is evident from the improvements in IVF (AdANNS-IVF) and can be easily extended to other ANNS structures like ScaNN [16] and
HNSW [38]. For example, HNSW consists of multiple layers with graphs of NSW graphs [37] of increasing complexity. AdANNS can be adopted to HNSW, where the construction of each level can be powered by appropriate dimensionalities for an optimal accuracy-compute trade-off. In general, AdANNS provides fine-grained control over compute overhead (storage, working memory, inference, and construction cost) during construction and inference while providing the best possible accuracy.
4.2 AdANNS-OPQ
Standard Product Quantization (PQ) essentially performs block-wise vector quantization via clustering. For example, suppose we need 32-byte PQ compressed vectors from the given 2048 dimensional representations. Then, we can chunk the representations in  $m = 32$  equal blocks/sub-vectors of 64-d each, and each sub-vector space is clustered into  $2^{8} = 256$  partitions. That is, the representation of each point is essentially cluster-id for each block. Optimized PQ (OPQ) [13] further refines this idea, by first rotating the representations using a learned orthogonal matrix, and then applying PQ on top of the rotated representations. In ANNS, OPQ is used extensively to compress vectors and improves approximate distance computation primarily due to significantly lower memory overhead than storing full-precision data points IVF.
AdANNS-OPQ utilizes MR representations to apply OPQ on lower-dimensional representations. That is, for a given quantization budget, AdANNS allows using top  $d_{s} \ll d$  dimensions from MR and then computing clusters with  $d_{s} / m$ -dimensional blocks where  $m$  is the number of blocks. Depending on  $d_{s}$  and  $m$ , we have further flexibility of trading-off dimensionality/capacity for increasing the number of clusters to meet the given quantization budget. AdANNS-OPQ tries multiple  $d_{s}$ ,  $m$ , and number of clusters for a fixed quantization budget to obtain the best performing configuration.
We experimented with 8 - 128 byte OPQ budgets for both ImageNet and Natural Questions retrieval with an exhaustive search on the quantized vectors. We compare AdANNS-OPQ which uses MRs of varying granularities to the baseline OPQ built on the highest dimensional RRs. We also evaluate OPQ vectors obtained projection using SVD [14] on top of the highest-dimensional RRs.
Empirical results. Figures 3 and 1b show that AdANNS-OPQ significantly outperforms – up to  $4\%$  accuracy gain – the baselines (OPQ on RRs) across compute budgets on both ImageNet and NQ. In particular, AdANNS-OPQ tends to match the accuracy of a 64-byte (a typical choice in ANNS) OPQ baseline with only a 32-byte budget. This results in a  $2\times$  reduction in both storage and compute FLOPS which translates to significant gains in real-world web-scale deployment (see Appendix D).
We only report the best AdANNS-OPQ for each budget typically obtained through a much lower-dimensional MR (128 & 192; much faster to build as well) than the highest-dimensional MR (2048 & 768) for ImageNet and NQ respectively (see Appendix G for more details). At the same time, we
Figure 3: AdANNS-OPQ matches the accuracy of 64-byte OPQ on RR using only 32-bytes for ImageNet retrieval. AdANNS provides large gains at lower compute budgets and saturates to baseline performance for larger budgets.
Figure 4: Combining the gains of AdANNS for IVF and OPQ leads to better IVFOPQ composite indices. On ImageNet retrieval, AdANNS-IVFOPQ is  $8 \times$  cheaper for the same accuracy and provides  $1 - 4\%$  gains over IVFOPQ on RRs.
note that building compressed OPQ vectors on projected RRs using SVD to the smaller dimensions (or using low-dimensional RRs, see Appendix D) as the optimal AdANNS-OPQ does not help in improving the accuracy. The significant gains we observe in AdANNS-OPQ are purely due to better information packing in MRs – we hypothesize that packing the most important information in the initial coordinates results in a better PQ quantization than RRs where the information is uniformly distributed across all the dimensions [31, 49]. See Appendix D for more details and experiments.
4.3 AdANNS for Composite Indices
We now extend AdANNS to composite indices [24] which put together two main ANNS building blocks - search structures and quantization - together to obtain efficient web-scale ANNS indices used in practice. A simple instantiation of a composite index would be the combination of IVF and OPQ - IVFOPQ - where the clustering in IVF happens with full-precision real vectors but the linear scan within each cluster is approximated using OPQ-compressed variants of the representation - since often the full-precision vectors of the database cannot fit in RAM. Contemporary ANNS indices like DiskANN [22] make this a default choice where they build the search graph with a full-precision vector and approximate the distance computations during search with an OPQ-compressed vector to obtain a very small shortlist of retrieved datapoints. In DiskANN, the shortlist of data points is then re-ranked to form the final list using their full-precision vectors fetched from the disk. AdANNS is naturally suited to this shortlist-erank framework: we use a low- $d$  MR for forming index, where we could tune AdANNS parameters according to the accuracy-compute trade-off of the graph and OPQ vectors. We then use a high- $d$  MR for re-ranking.
Empirical results. Figure 4 shows that AdANNS-IVFOPQ is  $1 - 4\%$  better than the baseline at all the PQ compute budgets. Furthermore, AdANNS-IVFOPQ has the same accuracy as the baselines at  $8\times$  lower overhead. With DiskANN, AdANNS accelerates shortlist generation by using low-dimensional representations and recoups the accuracy by reranking with the highest-dimensional
Table 1: AdANNS-DiskANN using a  $16 - d$  MR + re-ranking with the 2048-  $d$  MR outperforms DiskANN built on 2048-  $d$  RR at half the compute cost on ImageNet retrieval.
RR-2048	AdANNS
PQ Budget (Bytes)	32	16
Top-1 Accuracy (%)	70.37	70.56
mAP@10 (%)	62.46	64.70
Precision@40 (%)	65.65	68.25
MR at negligible cost. Table 1 shows that AdANNS-DiskANN is more accurate than the baseline for both 1-NN and ranking performance at only half the cost. Using low-dimensional representations further speeds up inference in AdANNS-DiskANN (see Appendix F).
These results show the generality of AdANNS and its broad applicability across a variety of ANNS indices built on top of the base building blocks. Currently, AdANNS piggybacks on typical ANNS pipelines for their inherent accounting of the real-world system constraints [16, 22, 25]. However, we believe that AdANNS's flexibility and significantly better accuracy-compute trade-off can be further informed by real-world deployment constraints. We leave this high-potential line of work that requires extensive study to future research.
5 Further Analysis and Discussion
5.1 Compute-aware Elastic Search During Inference
AdANNS search structures cater to many specific large-scale use scenarios that need to satisfy precise resource constraints during construction as well as inference. However, in many cases, construction and storage of the indices are not the bottlenecks or the user is unable to search the design space. In these settings, AdANNS-D enables adaptive inference through accurate yet cheaper distance computation using the low-dimensional prefix of matryoshka representation. Akin to composite indices (Section 4.3) that use PQ vectors for cheaper distance computation, we can use the low-dimensional MR for faster distance computation on ANNS structure built non-adaptively with a high-dimensional MR without any modifications to the existing index.
Empirical results. Figure 2 shows that for a given compute budget using IVF on ImageNet-1K retrieval, AdANNS-IVF is better than AdANNS-IVF-D due to the explicit control during the building
of the ANNS structure which is expected. However, the interesting observation is that AdANNS-D matches or outperforms the IVF indices built with MRs of varying capacities for ImageNet retrieval.
However, these methods are applicable in specific scenarios of deployment. Obtaining optimal AdANNS search structure (highly accurate) or even the best IVF-MR index relies on a relatively expensive design search but delivers indices that fit the storage, memory, compute, and accuracy constraints all at once. On the other hand AdANNS-D does not require a precisely built ANNS index but can enable compute-aware search during inference. AdANNS-D is a great choice for setups that can afford only one single database/index but need to cater to varying deployment constraints, e.g., one task requires  $70\%$  accuracy while another task has a compute budget of 1 MFLOPS/query.
5.2 Why MRs over RRs?
Quite a few of the gains from AdANNS are owing to the quality and capabilities of matryoshka representations. So, we conducted extensive analysis to understand why matryoshka representations seem to be more aligned for semantic search than the status-quo rigid representations.
Difficulty of NN search. Relative contrast  $(C_r)$  [18] is inversely proportional to the difficulty of nearest neighbor search on a given database. On ImageNet-1K, Figure 14 shows that MRs have better  $C_r$  than RRs across dimensionalities, further supporting that matryoshka representations are more aligned (easier) for NN search than existing rigid representations for the same accuracy. More details and analysis about this experiment can be found in Appendix H.2.
Clustering distributions. We also investigate the potential deviation in clustering distributions for MRs across dimensionalities compared to RRs. Unlike the RRs where the information is uniformly diffused across dimensions [49], MRs have hierarchical information packing. Figure 11 in Appendix E.3 shows that matryoshka representations result in clusters similar (measured by total variation distance [33]) to that of rigid representations and do not result in any unusual artifacts.
Robustness. Figure 9 in Appendix E shows that MRs continue to be better than RRs even for out-of-distribution (OOD) image queries (ImageNetV2 [44]) using ANNS. It also shows that the highest data dimensionality need not always be the most robust which is further supported by the higher recall using lower dimensions. Further details about this experiment can be found in Appendix E.1.
Generality across encoders. IVF-MR consistently has higher accuracy than IVF-RR across dimensionalities despite having similar accuracies with exact NN search (for ResNet50 on ImageNet and BERT-Base on NQ). We find that our observations on better alignment of MRs for NN search hold across neural network architectures, ResNet18/34/101 [19] and ConvNeXt-Tiny [35]. Appendix H.3 delves deep into the experimentation done using various neural architectures on ImageNet-1K.
Recall score analysis. Analysis of recall score (see Appendix C) in Appendix H.1 shows that for a similar top-1 accuracy, lower-dimensional representations have better 1-Recall@1 across search complexities for IVF and HNSW on ImageNet-1K. Across the board, MRs have higher recall scores and top-1 accuracy pointing to easier "searchability" and thus suitability of matryoshka representations for ANNS. Larger-scale experiments and further analysis can be found in Appendix H.
Through these analyses, we argue that matryoshka representations are better suited for semantic search than rigid representations, thus making them an ideal choice for AdANNS.
5.3 Search for AdANNS Hyperparameters
Choosing the optimal hyperparameters for AdANNS, such as  $d_c$ ,  $d_s$ ,  $m$ , # clusters, # probes, is an interesting and open problem that requires more rigorous examination. As the ANNS index is formed once and used for potentially billions of queries with massive implications for cost, latency and queries-per-second, a hyperparameter search for the best index is generally an acceptable industry practice [22, 38]. The Faiss library [24] provides guidelines<sup>2</sup> to choose the appropriate index for a specific problem, including memory constraints, database size, and the need for exact results. There have been efforts at automating the search for optimal indexing parameters, such as Autofaiss<sup>3</sup>, which maximizes recall given compute constraints.
$^{2}$ https://github.com/facebookresearch/faiss/wiki/Guidelines-to-choose-an-index
<sup>3</sup>https://github.com/criteo/autofaiss
In case of AdANNS, we suggest starting at the best configurations of MRs followed by a local design space search to lead to near-optimal AdANNS configurations (e.g. use IVF-MR to bootstrap AdANNS-IVF). We also share some observations during the course of our experiments:
1. AdANNS-IVF: Top-1 accuracy generally improves (with diminishing returns after a point) with increasing dimensionality of clustering  $(d_c)$  and search  $(d_s)$ , as we show on ImageNet variants and with multiple encoders in the Appendix (Figures 9 and 15). Clustering with low- $d$  MRs matches the performance of high- $d$  MRs as they likely contain similar amounts of useful information, making the increased compute cost not worth the marginal gains. Increasing # probes naturally boosts performance (Appendix, Figure 10a). Lastly, it is generally accepted that a good starting point for the # clusters  $k$  is  $\sqrt{N_D / 2}$ , where  $N_D$  is the number of indexable items [39].  $k = \sqrt{N_D}$  is the optimal choice of  $k$  from a FLOPS computation perspective as can be seen in Appendix B.1.
2. AdANNS-OPQ: we observe that for a fixed compute budget in bytes  $(m)$ , the top-1 accuracy reaches a peak at  $d < d_{max}$  (Appendix, Table 4). We hypothesize that the better performance of AdANNS-OPQ at  $d < d_{max}$  is due to the curse of dimensionality, i.e. it is easier to learn PQ codebooks on smaller embeddings with similar amounts of information. We find that using an MR with  $d = 4 \times m$  is a good starting point on ImageNet and NQ. We also suggest using an 8-bit (256-length) codebook for OPQ as the default for each of the sub-block quantizer.
3. AdANNS-DiskANN: Our observations with DiskANN are consistent with other indexing structures, i.e. the optimal graph construction dimensionality  $d < d_{max}$  (Appendix, Figure 12). A careful study of DiskANN on different datasets is required for more general guidelines to choose graph construction and OPQ dimensionality  $d$ .
5.4 Limitations
AdANNS's core focus is to improve the design of the existing ANNS pipelines. To use AdANNS on a corpus, we need to back-fill [43] the MRs of the data - a significant yet a one-time overhead. We also notice that high-dimensional MRs start to degrade in performance when optimizing also for an extremely low-dimensional granularity (e.g.,  $< 24$ -d for NQ) - otherwise is it quite easy to have comparable accuracies with both RRs and MRs. Lastly, the existing dense representations can only in theory be converted to MRs with an auto-encoder-style non-linear transformation. We believe most of these limitations form excellent future work to improve AdANNS further.
We proposed a novel framework, AdANNS, that leverages adaptive representations for different phases of ANNS pipelines to improve the accuracy-compute tradeoff. AdANNS utilizes the inherent flexibility of matryoshka representations [31] to design better ANNS building blocks than the standard ones which use the rigid representation in each phase. AdANNS achieves SOTA accuracy-compute trade-off for the two main ANNS building blocks: search data structures (AdANNS-IVF) and quantization (AdANNS-OPQ). The combination of AdANNS-based building blocks leads to the construction of better real-world composite ANNS indices – with as much as  $8 \times$  reduction in cost at the same accuracy as strong baselines – while also enabling compute-aware elastic search. Finally, we note that combining AdANNS with elastic encoders [11] enables truly adaptive large-scale retrieval.
We are grateful to Kaifeng Chen, Venkata Sailesh Sanampudi, Sanjiv Kumar, Harsha Vardhan Simhadri, Gantavya Bhatt, Matthijs Douze and Matthew Wallingford for helpful discussions and feedback. Aditya Kusupati also thanks Tom Duerig and Rahul Sukthankar for their support. Part of the paper's large-scale experimentation is supported through a research GCP credit award from Google Cloud and Google Research. Sham Kakade acknowledges funding from the ONR award N00014-22-1-2377 and NSF award CCF-2212841. Ali Farhadi acknowledges funding from the NSF awards IIS 1652052, IIS 17303166, DARPA N66001-19-2-4031, DARPA W911NF-15-1-0543, and gifts from Allen Institute for Artificial Intelligence and Google.
Algorithm 1 AdANNS-IVF Psuedocode
Index database to construct clusters and build inverted file system   
def adannsConstruction(database, d_cluster, num_clusters): # Slice database with cluster construction dim (d_cluster) xb = database[:d_cluster] cluster_centroids = constructClusters(xb, num_clusters) return cluster_centroids   
def adannsInference queries, centroids, d_shortlist, d_search, num_probes, k): # Slice queries and centroids with cluster shortlist dim (d_shortlist) xq = queries[:d_shortlist] xc = centroids[:d_shortlist] for q in queries: # compute distance of query from each cluster centroid candidate_distances  $=$  computeDistances(q,xc) # sort cluster candidates by distance and choose small number to probe clusterCandidates  $\equiv$  sortAscending(candidate_distance)[:num_probes] database_candidateates  $\equiv$  getClusterMembers(cluster_candidateates) # Linear Scan all shortlisted clusters with search dim (d_search) knearest_neighbors[q]  $=$  linearScan(q,database_candidateates,d_search, k) return knearest_neighbors
Figure 5: The schematic of inverted file index (IVF) outlaying the construction and inference phases. Adaptive representations can be utilized effectively in the decoupled components of clustering and searching for a better accuracy-compute trade-off (AdANNS-IVF).
A AdANNS Framework
Table 2: Mathematical formulae of the retrieval phase across various methods built on IVF. See Section 3 for notations.
Method	Retrieval Formula during Inference
IVF-RR	arg minj∈Ch(q) ||φRR(d)(q) - φRR(d)(xj)||, s.t. h(q) = arg minh||φRR(d)(q) - μhRR(d)||
IVF-MR	arg minj∈Ch(q) ||φMR(d)(q) - φMR(d)(xj)||, s.t. h(q) = arg minh||φMR(d)(q) - μhMR(d)||
AdANNS-IVF	arg minj∈Ch(q) ||φMR(ds)(q) - φMR(ds)(xj)||, s.t. h(q) = arg minh||φMR(dc)(q) - μhMR(dc)||
MG-IVF-RR	arg minj∈Ch(q) ||φRR(ds)(q) - φRR(dc)(xj)||, s.t. h(q) = arg minh||φRR(dc)(q) - μhRR(dc)||
AdANNS-IVF-D	arg minj∈Ch(q) ||φMR(d)(q)[1:d] - φMR(d)(xj)[1:d]|, s.t. h(q) = arg minh||φMR(d)(q)[1:d] - μhMR(d)[1:d]|
IVFOPQ	arg minj∈Ch(q) ||φPQ(m,b)(q) - φPQ(m,b)(xj)||, s.t. h(q) = arg minh||φ(q) - μh||
B Training and Compute Costs
A bulk of our ANNS experimentation was written with Faiss [24], a library for efficient similarity search and clustering. AdANNS was implemented from scratch (Algorithm 1) due to difficulty in decoupling clustering and linear scan with Faiss, with code available at https://github.com/RAIVNLab/AdANNS. We also provide a version of AdANNS with Faiss optimizations with the restriction that  $D_{c} \geq D_{s}$  as a limitation of the current implementation, which can be further optimized. All ANNS experiments (AdANNS-IVF, MG-IVF-RR, IVF-MR, IVF-RR, HNSW, HNSWOPQ, IVFOPQ) were run on an Intel Xeon 2.20GHz CPU with 12 cores. Exact Search (Flat L2, PQ, OPQ) and DiskANN experiments were run with CUDA 11.0 on a A100-SXM4 NVIDIA GPU with 40G RAM. The wall-clock inference times quoted in Figure 1a and Table 3 are reported on CPU with Faiss optimizations, and are averaged over three inference runs for ImageNet-1K retrieval.
Table 3: Comparison of AdANNS-IVF and Rigid-IVF wall-clock inference times for ImageNet-1K retrieval. AdANNS-IVF has up to  $\sim 1.5\%$  gain over Rigid-IVF for a fixed search latency per query.
AdANNS-IVF	Rigid-IVF
Top-1	Search Latency/Query (ms)	Top-1	Search Latency/Query (ms)
70.02	0.03	68.51	0.02
70.08	0.06	68.54	0.05
70.19	0.06	68.74	0.08
70.36	0.88	69.20	0.86
70.60	5.57	70.13	5.67
DPR [27] on NQ [32]. We follow the setup on the DPR repo<sup>4</sup>: the Wikipedia corpus has 21 million passages and Natural Questions dataset for open-domain QA settings. The training set contains 79,168 question and answer pairs, the dev set has 8,757 pairs and the test set has 3,610 pairs.
B.1 Inference Compute Cost
We evaluate inference compute costs for IVF in MegaFLOPS per query (MFLOPS/query) as shown in Figures 2, 10a, and 8 as follows:
$$
C = d _ {s} k + \frac {n _ {p} d _ {s} N _ {D}}{k}
$$
where  $d_{c}$  is the cluster construction embedding dimensionality,  $d_{s}$  is the embedding dim used for linear scan within each probed cluster, which is controlled by # of search probes  $n_p$ . Finally,  $k$  is the number of clusters  $|C_i|$  indexed over database of size  $N_{D}$ . The default setting in this work, unless otherwise stated, is  $n_p = 1$ ,  $k = 1024$ ,  $N_{D} = 1281167$  (ImageNet-1K trainset). Vanilla IVF supports only  $d_{c} = d_{s}$ , while AdANNS-IVF provides flexibility via decoupling clustering and search (Section 4). AdANNS-IVF-D is a special case of AdANNS-IVF with the flexibility restricted to inference, i.e.,  $d_{c}$  is a fixed high-dimensional MR.
4https://github.com/facebookresearch/DPR
C Evaluation Metrics
In this work, we primarily use top-1 accuracy (i.e. 1-Nearest Neighbor), recall@k, corrected mean average precision  $(\mathrm{mAP}@\mathrm{k})$  [30] and k-Recall@N (recall score), which are defined over all queries  $Q$  over indexed database of size  $N_{D}$  as:
$$
\text {t o p - 1} = \frac {\sum_ {Q} \text {c o r r e c t} _ {-} \text {p r e d} @ 1}{| Q |}
$$
$$
\operatorname {R e c a l l} @ \mathrm {k} = \frac {\sum_ {Q} \operatorname {c o r r e c t} _ {\text {p r e d}} @ k}{| Q |} * \frac {\operatorname {n u m} _ {\text {c l a s s e s}}}{| N _ {D} |}
$$