Title: AdANNS: A Framework for Adaptive Semantic Search
Venue: NeurIPS 2023
Accepted: yes
Keywords: Semantic Search

Approximate Nearest Neighbor Search

Large-scale search

Representation Learning
Abstract: Web-scale search systems learn an encoder to embed a given query which is then hooked into an approximate nearest neighbor search (ANNS) pipeline to retrieve similar data points. To accurately capture tail queries and data points, learned representations typically are _rigid, high-dimensional_ vectors that are generally used as-is in the entire ANNS pipeline and can lead to computationally expensive retrieval. In this paper, we argue that instead of rigid representations, different stages of ANNS can leverage _adaptive representations_ of varying capacities to achieve significantly better accuracy-compute trade-offs, i.e., stages of ANNS that can get away with more approximate computation should use a lower-capacity representation of the same data point. To this end, we introduce AdANNS, a novel ANNS design framework that explicitly leverages the flexibility of Matryoshka Representations. We demonstrate state-of-the-art accuracy-compute trade-offs using novel AdANNS-based key ANNS building blocks like search data structures (AdANNS-IVF) and quantization (AdANNS-OPQ). For example on ImageNet retrieval, AdANNS-IVF is up to $\mathbf{1.5}$% more accurate than the rigid representations-based IVF at the same compute budget; and matches accuracy while being up to $\mathbf{90}\times$ faster in _wall-clock time_. For Natural Questions, $32$-byte AdANNS-OPQ matches the accuracy of the $64$-byte OPQ baseline constructed using rigid representations -- _same accuracy at half the cost!_ We further show that the gains from AdANNS translate to modern-day composite ANNS indices that combine search structures and quantization. Finally, we demonstrate that AdANNS can enable inference-time adaptivity for compute-aware search on ANNS indices built non-adaptively on matryoshka representations. Code is open-sourced at https://github.com/RAIVNLab/AdANNS.

AdANNS : A Framework for Adaptive Semantic Search

Abstract

Web-scale search systems learn an encoder to embed a given query which is then hooked into an approximate nearest neighbor search (ANNS) pipeline to retrieve similar data points. To accurately capture tail queries and data points, learned representations typically are rigid, high-dimensional vectors that are generally used as-is in the entire ANNS pipeline and can lead to computationally expensive retrieval. In this paper, we argue that instead of rigid representations, different stages of ANNS can leverage adaptive representations of varying capacities to achieve significantly better accuracy-compute trade-offs, i.e., stages of ANNS that can get away with more approximate computation should use a lower-capacity representation of the same data point. To this end, we introduce AdANNS , a novel ANNS design framework that explicitly leverages the flexibility of Matryoshka Representations [31]. We demonstrate state-of-the-art accuracy-compute trade-offs using novel AdANNS -based key ANNS building blocks like search data structures ( AdANNS -IVF) and quantization ( AdANNS -OPQ). For example on ImageNet retrieval, AdANNS -IVF is up to 1 . 5 % more accurate than the rigid representations-based IVF [48] at the same compute budget; and matches accuracy while being up to 90 × faster in wall-clock time . For Natural Questions, 32 -byte AdANNS -OPQ matches the accuracy of the 64 -byte OPQ baseline [13] constructed using rigid representations same accuracy at half the cost! We further show that the gains from AdANNS translate to modern-day composite ANNS indices that combine search structures and quantization. Finally, we demonstrate that AdANNS can enable inference-time adaptivity for compute-aware search on ANNS indices built non-adaptively on matryoshka representations. Code is open-sourced at https://github.com/RAIVNLab/AdANNS .

1 Introduction

Semantic search [24] on learned representations [40, 41, 50] is a major component in retrieval pipelines [4, 9]. In its simplest form, semantic search methods learn a neural network to embed queries as well as a large number ( N ) of data points in a d -dimensional vector space. For a given query, the nearest (in embedding space) point is retrieved using either an exact search or using approximate nearest neighbor search (ANNS) [21] which is now indispensable for real-time large-scale retrieval.

Existing semantic search methods learn fixed or rigid representations ( RRs ) which are used as is in all the stages of ANNS (data structures for data pruning and quantization for cheaper distance computation; see Section 2). That is, while ANNS indices allow a variety of parameters for searching the design space to optimize the accuracy-compute trade-off, the provided data dimensionality is typically assumed to be an immutable parameter. To make it concrete, let us consider inverted file index (IVF) [48], a popular web-scale ANNS technique [16]. IVF has two stages (Section 3) during inference: (a) cluster mapping : mapping the query to a cluster of data points [36], and (b) linear

























×







(a) Image retrieval on ImageNet-1K.

























×







(b) Passage retrieval on Natural Questions.

scan : distance computation w.r.t all points in the retrieved cluster to find the nearest neighbor (NN). Standard IVF utilizes the same high-dimensional RR for both phases, which can be sub-optimal.

Why the sub-optimality? Imagine one needs to partition a dataset into k clusters for IVF and the dimensionality of the data is d - IVF uses full d representation to partition into k clusters. However, suppose we have an alternate approach that somehow projects the data in d/ 2 dimensions and learns 2 k clusters. Note that the storage and computation to find the nearest cluster remains the same in both cases, i.e., when we have k clusters of d dimensions or 2 k clusters of d/ 2 dimensions. 2 k clusters can provide significantly more refined partitioning, but the distances computed between queries and clusters could be significantly more inaccurate after projection to d/ 2 dimensions.

So, if we can find a mechanism to obtain a d/ 2 -dimensional representation of points that can accurately approximate the topology/distances of d -dimensional representation, then we can potentially build significantly better ANNS structure that utilizes different capacity representations for the cluster mapping and linear scan phases of IVF. But how do we find such adaptive representations ? These desired adaptive representations should be cheap to obtain and still ensure distance preservation across dimensionality. Post-hoc dimensionality reduction techniques like SVD [14] and random projections [25] on high-dimensional RRs are potential candidates, but our experiments indicate that in practice they are highly inaccurate and do not preserve distances well enough (Figure 2).

Instead, we identify that the recently proposed Matryoshka Representations ( MRs ) [31] satisfy the specifications for adaptive representations. Matryoshka representations pack information in a hierarchical nested manner, i.e., the first m -dimensions of the d -dimensional MR form an accurate low-dimensional representation while being aware of the information in the higher dimensions. This allows us to deploy MRs in two major and novel ways as part of ANNS: (a) low-dimensional representations for accuracy-compute optimal clustering and quantization, and (b) high-dimensional representations for precise re-ranking when feasible.

To this effort, we introduce AdANNS , a novel design framework for semantic search that uses matryoshka representation-based adaptive representations across different stages of ANNS to ensure significantly better accuracy-compute trade-off than the state-of-the-art baselines.

Typical ANNS systems have two key components: (a) search data structure to store datapoints, (b) distance computation to map a given query to points in the data structure. Through AdANNS , we address both these components and significantly improve their performance. In particular, we first propose AdANNS -IVF (Section 4.1) which tackles the first component of ANNS systems. AdANNS -IVF uses standard full-precision computations but uses adaptive representations for different IVF stages. On ImageNet 1-NN image retrieval (Figure 1a), AdANNS -IVF is up to 1 . 5% more accurate for the compute budget and 90 × cheaper in deployment for the same accuracy as IVF.

We then propose AdANNS -OPQ (Section 4.2) which addresses the second component by using AdANNS -based quantization (OPQ [13]) - here we use exhaustive search overall points. AdANNS -OPQ is as accurate as the baseline OPQ on RRs while being at least 2 × faster on Natural Questions [32] 1-NN passage retrieval (Figure 1b). Finally, we combine the two techniques to obtain AdANNS -IVFOPQ (Section 4.3) which is more accurate while being much cheaper - up to 8 × - than the traditional IVFOPQ [24] index. To demonstrate generality of our technique, we adapt AdANNS to DiskANN [22] which provides interesting accuracy-compute tradeoff; see Table 1.

While MR already has multi-granular representations, careful integration with ANNS building blocks is critical to obtain a practical method and is our main contribution . In fact, Kusupati et al. [31] proposed a simple adaptive retrieval setup that uses smaller-dimensional MR for shortlisting in retrieval followed by precise re-ranking with a higher-dimensional MR . Such techniques, unfortunately, cannot be scaled to industrial systems as they require forming a new index for every shortlisting provided by low-dimensional MR . Ensuring that the method aligns well with the modern-day ANNS pipelines is important as they already have mechanisms to handle real-world constraints like load-balancing [16] and random access from disk [22]. So, AdANNS is a step towards making the abstraction of adaptive search and retrieval feasible at the web-scale.

Through extensive experimentation, we also show that AdANNS generalizes across search data structures, distance approximations, modalities (text & image), and encoders (CNNs & Transformers) while still translating the theoretical gains to latency reductions in deployment. While we have mainly focused on IVF and OPQ-based ANNS in this work, AdANNS also blends well with other ANNS pipelines. We also show that AdANNS can enable compute-aware elastic search on prebuilt indices without making any modifications (Section 5.1); note that this is in contrast to AdANNS -IVF that builds the index explicitly utilizing 'adaptivity' in representations. Finally, we provide an extensive analysis on the alignment of matryoshka representation for better semantic search (Section 5.2).

We make the following key contributions:

We introduce AdANNS , a novel framework for semantic search that leverages matryoshka representations for designing ANNS systems with better accuracy-compute trade-offs.

AdANNS powered search data structure ( AdANNS -IVF) and quantization ( AdANNS -OPQ) show a significant improvement in accuracy-compute tradeoff compared to existing solutions.

AdANNS generalizes to modern-day composite ANNS indices and can also enable compute-aware elastic search during inference with no modifications.

2 Related Work

Approximate nearest neighbour search (ANNS) is a paradigm to come as close as possible [7] to retrieving the 'true' nearest neighbor (NN) without the exorbitant search costs associated with exhaustive search [21, 52]. The 'approximate' nature comes from data pruning as well as the cheaper distance computation that enable real-time web-scale search. In its naive form, NN-search has a complexity of O ( dN ) ; d is the data dimensionality used for distance computation and N is the size of the database. ANNS employs each of these approximations to reduce the linear dependence on the dimensionality (cheaper distance computation) and data points visited during search (data pruning).

Cheaper distance computation. From a bird's eye view, cheaper distance computation is always obtained through dimensionality reduction (quantization included). PCA and SVD [14, 26] can reduce dimensionality and preserve distances only to a limited extent without sacrificing accuracy. On the other hand, quantization-based techniques [6, 15] like (optimized) product quantization ((O)PQ) [13, 23] have proved extremely crucial for relatively accurate yet cheap distance computation and simultaneously reduce the memory overhead significantly. Another naive solution is to independently train the representation function with varying low-dimensional information bottlenecks [31] which is rarely used due to the costs of maintaining multiple models and databases.

Data pruning. Enabled by various data structures, data pruning reduces the number of data points visited as part of the search. This is often achieved through hashing [8, 46], trees [3, 12, 16, 48] and graphs [22, 38]. More recently there have been efforts towards end-to-end learning of the search data structures [17, 29, 30]. However, web-scale ANNS indices are often constructed on rigid d -dimensional real vectors using the aforementioned data structures that assist with the real-time search. For a more comprehensive review of ANNS structures please refer to [5, 34, 51].

Composite indices. ANNSpipelines often benefit from the complementary nature of various building blocks [24, 42]. In practice, often the data structures (coarse-quantizer) like IVF [48] and HNSW [37] are combined with cheaper distance alternatives like PQ [23] (fine-quantizer) for massive speed-ups in web-scale search. While the data structures are built on d -dimensional real vectors, past works consistently show that PQ can be safely used for distance computation during search time. As evident in modern web-scale ANNS systems like DiskANN [22], the data structures are built on d -dimensional real vectors but work with PQ vectors ( 32 -64 -byte) for fast distance computations.

ANNS benchmark datasets. Despite the Herculean advances in representation learning [19, 42], ANNS progress is often only benchmarked on fixed representation vectors provided for about a dozen million to billion scale datasets [1, 47] with limited access to the raw data. This resulted in the improvement of algorithmic design for rigid representations ( RRs ) that are often not specifically designed for search. All the existing ANNS methods work with the assumption of using the provided d -dimensional representation which might not be Pareto-optimal for the accuracy-compute tradeoff in the first place. Note that the lack of raw-image and text-based benchmarks led us to using ImageNet-1K [45] (1.3M images, 50K queries) and Natural Questions [32] (21M passages, 3.6K queries) for experimentation. While not billion-scale, the results observed on ImageNet often translate to real-world progress [28], and Natural Questions is one of the largest question answering datasets benchmarked for dense passage retrieval [27], making our results generalizable and widely applicable.

In this paper, we investigate the utility of adaptive representations - embeddings of different dimensionalities having similar semantic information - in improving the design of ANNS algorithms. This helps in transitioning out of restricted construction and inference on rigid representations for ANNS. To this end, we extensively use Matryoshka Representations ( MRs ) [31] which have desired adaptive properties in-built. To the best of our knowledge, this is the first work that improves accuracy-compute trade-off in ANNS by leveraging adaptive representations on different phases of construction and inference for ANNS data structures.

3 Problem Setup, Notation, and Preliminaries

The problem setup of approximate nearest neighbor search (ANNS) [21] consists of a database of N data points, [ x 1 , x 2 , . . . , x N ] , and a query, q , where the goal is to 'approximately' retrieve the nearest data point to the query. Both the database and query are embedded to R d using a representation function φ : X → R d , often a neural network that can be learned through various representation learning paradigms [2, 19, 20, 40, 42].

Matryoshka Representations ( MRs ). The d -dimensional representations from φ can have a nested structure like Matryoshka Representations ( MRs ) [31] in-built φ MR ( d ) . Matryoshka Representation Learning (MRL) learns these nested representations with a simple strategy of optimizing the same training objective at varying dimensionalities. These granularities are ordered such that the lowest representation size forms a prefix for the higher-dimensional representations. So, high-dimensional MR inherently contains low-dimensional representations of varying granularities that can be accessed for free - first m -dimensions ( m ∈ [ d ] ) ie., φ MR ( d ) [1 : m ] from the d -dimensional MR form an m -dimensional representation which is as accurate as its independently trained rigid representation ( RR ) counterpart φ RR ( m ) . Training an encoder with MRL does not involve any overhead or hyperparameter tuning and works seamlessly across modalities, training objectives, and architectures.

Inverted File Index (IVF). IVF [48] is an ANNS data structure used in web-scale search systems [16] owing to its simplicity, minimal compute overhead, and high accuracy. IVF construction involves clustering (coarse quantization through k-means) [36] on d -dimensional representation that results in an inverted file list [53] of all the data points in each cluster. During search, d -dimensional query representation is assigned to the most relevant cluster ( C i ; i ∈ [ k ] ) by finding the closest centroid ( µ i ) using an appropriate distance metric ( L 2 or cosine). This is followed by an exhaustive linear search across all data points in the cluster which gives the closest NN (see Figure 5 in Appendix A for IVF overview). Lastly, IVF can scale to web-scale by utilizing a hierarchical IVF structure within each cluster [16]. Table 2 in Appendix A describes the retrieval formula for multiple variants of IVF.

Optimized Product Quantization (OPQ). Product Quantization (PQ) [23] works by splitting a d -dimensional real vector into m sub-vectors and quantizing each sub-vector with an independent 2 b

length codebook across the database. After PQ, each d -dimensional vector can be represented by a compact m × b bit vector; we make each vector m bytes long by fixing b = 8 . During search time, distance computation between the query vector and PQ database is extremely efficient with only m codebook lookups. The generality of PQ encompasses scalar/vector quantization [15, 36] as special cases. However, PQ can be further improved by rotating the d -dimensional space appropriately to maximize distance preservation after PQ. Optimized Product Quantization (OPQ) [13] achieves this by learning an orthonormal projection matrix R that rotates the d -dimensional space to be more amenable to PQ. OPQ shows consistent gains over PQ across a variety of ANNS tasks and has become the default choice in standard composite indices [22, 24].

Datasets. We evaluate the ANNS algorithms while changing the representations used for the search thus making it impossible to evaluate on the usual benchmarks [1]. Hence we experiment with two public datasets: (a) ImageNet-1K [45] dataset on the task of image retrieval - where the goal is to retrieve images from a database (1.3M image train set) belonging to the same class as the query image (50K image validation set) and (b) Natural Questions (NQ) [32] dataset on the task of question answering through dense passage retrieval - where the goal is to retrieve the relevant passage from a database (21M Wikipedia passages) for a query (3.6K questions).

Metrics Performance of ANNS is often measured using recall score [22], k -recall@ N - recall of the exact NN across search complexities which denotes the recall of k 'true' NN when N data points are retrieved. However, the presence of labels allows us to compute 1-NN (top-1) accuracy. Top-1 accuracy is a harder and more fine-grained metric that correlates well with typical retrieval metrics like recall and mean average precision (mAP@ k ). Even though we report top-1 accuracy by default during experimentation, we discuss other metrics in Appendix C. Finally, we measure the compute overhead of ANNS using MFLOPS/query and also provide wall-clock times (see Appendix B.1).

Encoders. For ImageNet, we encode both the database and query set using a ResNet50 ( φ I ) [19] trained on ImageNet-1K. For NQ, we encode both the passages in the database and the questions in the query set using a BERT-Base ( φ N ) [10] model fine-tuned on NQ for dense passage retrieval [27].

We use the trained ResNet50 models with varying representation sizes ( d = [8 , 16 , . . . , 2048] ; default being 2048 ) as suggested by Kusupati et al. [31] alongside the MRL-ResNet50 models trained with MRL for the same dimensionalities. The RR and MR models are trained to ensure the supervised one-vs-all classification accuracy across all data dimensionalities is nearly the same - 1-NN accuracy of 2048 -d RR and MR models are 71 . 19% and 70 . 97% respectively on ImageNet-1K. Independently trained models, φ RR ( d ) I , output d = [8 , 16 . . . , 2048] dimensional RRs while a single MRL-ResNet50 model, φ MR ( d ) I , outputs a d = 2048 -dimensional MR that contains all the 9 granularities.

We also train BERT-Base models in a similar vein as the aforementioned ResNet50 models. The key difference is that we take a pre-trained BERT-Base model and fine-tune on NQ as suggested by Karpukhin et al. [27] with varying (5) representation sizes (bottlenecks) ( d = [48 , 96 , . . . , 768] ; default being 768 ) to obtain φ RR ( d ) N that creates RRs for the NQ dataset. To get the MRL-BERTBase model, we fine-tune a pre-trained BERT-Base encoder on the NQ train dataset using the MRL objective with the same granularities as RRs to obtain φ MR ( d ) N which contains all five granularities. Akin to ResNet50 models, the RR and MR BERT-Base models on NQ are built to have similar 1-NN accuracy for 768 -d of 52 . 2% and 51 . 5% respectively. More implementation details can be found in Appendix B and additional experiment-specific information is provided at the appropriate places.

4 AdANNS - Adaptive ANNS

In this section, we present our proposed AdANNS framework that exploits the inherent flexibility of matryoshka representations to improve the accuracy-compute trade-off for semantic search components. Standard ANNS pipeline can be split into two key components: (a) search data structure that indexes and stores data points, (b) query-point computation method that outputs (approximate) distance between a given query and data point. For example, standard IVFOPQ [24] method uses an IVF structure to index points on full-precision vectors and then relies on OPQ for more efficient distance computation between the query and the data points during the linear scan.

Below, we show that AdANNS can be applied to both the above-mentioned ANNS components and provides significant gains on the computation-accuracy tradeoff curve. In particular, we present AdANNS -IVF which is AdANNS version of the standard IVF index structure [48], and the closely related ScaNN structure [16]. We also present AdANNS -OPQ which introduces representation adaptivity in the OPQ, an industry-default quantization. Then, in Section 4.3 we further demonstrate the combination of the two techniques to get AdANNS -IVFOPQ - an AdANNS version of IVFOPQ [24] - and AdANNS -DiskANN, a similar variant of DiskANN [22]. Overall, our experiments show that AdANNS -IVF is significantly more accuracy-compute optimal compared to the IVF indices built on RRs and AdANNS -OPQ is as accurate as the OPQ on RRs while being significantly cheaper.

4.1 AdANNS -IVF

Recall from Section 1 that IVF has a clustering and a linear scan phase, where both phase use same dimensional rigid representation. Now, AdANNS -IVF allows the clustering phase to use the first d c dimensions of the given matryoshka representation ( MR ). Similarly, the linear scan within each cluster uses d s dimensions, where again d s represents top d s coordinates from MR . Note that setting d c = d s results in nonadaptive regular IVF. Intuitively, we would set d c glyph[lessmuch] d s , so that instead of clustering with a high-dimensional representation, we can approximate it accurately with a low-dimensional embedding of size d c followed by a linear scan with a higher d s -dimensional representation. Intuitively, this helps in the smooth search of design space for state-of-the-art accuracy-compute

































trade-off. Furthermore, this can provide a precise operating point on accuracy-compute tradeoff curve which is critical in several practical settings.

Our experiments on regular IVF with MRs and RRs (IVFMR &IVFRR ) of varying dimensionalities and IVF configurations (# clusters, # probes) show that (Figure 2) matryoshka representations result in a significantly better accuracy-compute trade-off. We further studied and found that learned lower-dimensional representations offer better accuracy-compute trade-offs for IVF than higherdimensional embeddings (see Appendix E for more results).

AdANNS utilizes d -dimensional matryoshka representation to get accurate d c and d s dimensional vectors at no extra compute cost. The resulting AdANNS -IVF provides a much better accuracycompute trade-off (Figure 2) on ImageNet-1K retrieval compared to IVFMR , IVFRR , and MGIVFRR - multi-granular IVF with rigid representations (akin to AdANNS without MR ) - a strong baseline that uses d c and d s dimensional RRs . Finally, we exhaustively search the design space of IVF by varying d c , d s ∈ [8 , 16 , . . . , 2048] and the number of clusters k ∈ [8 , 16 , . . . , 2048] . Please see Appendix E for more details. For IVF experiments on the NQ dataset, please refer to Appendix G.

Empirical results. Figure 2 shows that AdANNS -IVF outperforms the baselines across all accuracy-compute settings for ImageNet-1K retrieval. AdANNS -IVF results in 10 × lower compute for the best accuracy of the extremely expensive MG-IVFRR and non-adaptive IVFMR . Specifically, as shown in Figure 1a, AdANNS -IVF is up to 1 . 5% more accurate for the same compute and has up to 100 × lesser FLOPS/query ( 90 × real-world speed-up!) than the status quo ANNS on rigid representations (IVFRR ). We filter out points for the sake of presentation and encourage the reader to check out Figure 8 in Appendix E for an expansive plot of all the configurations searched.

The advantage of AdANNS for construction of search structures is evident from the improvements in IVF ( AdANNS -IVF) and can be easily extended to other ANNS structures like ScaNN [16] and

HNSW [38]. For example, HNSW consists of multiple layers with graphs of NSW graphs [37] of increasing complexity. AdANNS can be adopted to HNSW, where the construction of each level can be powered by appropriate dimensionalities for an optimal accuracy-compute trade-off. In general, AdANNS provides fine-grained control over compute overhead (storage, working memory, inference, and construction cost) during construction and inference while providing the best possible accuracy.

4.2 AdANNS -OPQ

Standard Product Quantization (PQ) essentially performs block-wise vector quantization via clustering. For example, suppose we need 32 -byte PQ compressed vectors from the given 2048 dimensional representations. Then, we can chunk the representations in m = 32 equal blocks/sub-vectors of 64 -d each, and each sub-vector space is clustered into 2 8 = 256 partitions. That is, the representation of each point is essentially cluster-id for each block. Optimized PQ (OPQ) [13] further refines this idea, by first rotating the representations using a learned orthogonal matrix, and then applying PQ on top of the rotated representations. In ANNS, OPQ is used extensively to compress vectors and improves approximate distance computation primarily due to significantly lower memory overhead than storing full-precision data points IVF.

AdANNS -OPQ utilizes MR representations to apply OPQ on lower-dimensional representations. That is, for a given quantization budget, AdANNS allows using top d s glyph[lessmuch] d dimensions from MR and then computing clusters with d s /m -dimensional blocks where m is the number of blocks. Depending on d s and m , we have further flexibility of trading-off dimensionality/capacity for increasing the number of clusters to meet the given quantization budget. AdANNS -OPQ tries multiple d s , m , and number of clusters for a fixed quantization budget to obtain the best performing configuration.

We experimented with 8 -128 byte OPQ budgets for both ImageNet and Natural Questions retrieval with an exhaustive search on the quantized vectors. We compare AdANNS -OPQ which uses MRs of varying granularities to the baseline OPQ built on the highest dimensional RRs . We also evaluate OPQ vectors obtained projection using SVD [14] on top of the highest-dimensional RRs .

Empirical results. Figures 3 and 1b show that AdANNS -OPQ significantly outperforms - up to 4% accuracy gain - the baselines (OPQ on RRs ) across compute budgets on both ImageNet and NQ. In particular, AdANNS -OPQ tends to match the accuracy of a 64 -byte (a typical choice in ANNS) OPQ baseline with only a 32 -byte budget. This results in a 2 × reduction in both storage and compute FLOPS which translates to significant gains in real-world web-scale deployment (see Appendix D).

We only report the best AdANNS -OPQ for each budget typically obtained through a much lowerdimensional MR ( 128 & 192 ; much faster to build as well) than the highest-dimensional MR ( 2048 & 768 ) for ImageNet and NQ respectively (see Appendix G for more details). At the same time, we























































note that building compressed OPQ vectors on projected RRs using SVD to the smaller dimensions (or using low-dimensional RRs , see Appendix D) as the optimal AdANNS -OPQ does not help in improving the accuracy. The significant gains we observe in AdANNS -OPQ are purely due to better information packing in MRs - we hypothesize that packing the most important information in the initial coordinates results in a better PQ quantization than RRs where the information is uniformly distributed across all the dimensions [31, 49]. See Appendix D for more details and experiments.

4.3 AdANNS for Composite Indices

We now extend AdANNS to composite indices [24] which put together two main ANNS building blocks - search structures and quantization - together to obtain efficient web-scale ANNS indices used in practice. A simple instantiation of a composite index would be the combination of IVF and OPQ - IVFOPQ - where the clustering in IVF happens with full-precision real vectors but the linear scan within each cluster is approximated using OPQ-compressed variants of the representation since often the full-precision vectors of the database cannot fit in RAM. Contemporary ANNS indices like DiskANN [22] make this a default choice where they build the search graph with a full-precision vector and approximate the distance computations during search with an OPQ-compressed vector to obtain a very small shortlist of retrieved datapoints. In DiskANN, the shortlist of data points is then re-ranked to form the final list using their full-precision vectors fetched from the disk. AdANNS is naturally suited to this shortlist-rerank framework: we use a lowd MR for forming index, where we could tune AdANNS parameters according to the accuracy-compute trade-off of the graph and OPQ vectors. We then use a highd MR for re-ranking.

Empirical results. Figure 4 shows that AdANNS -IVFOPQ is 1 -4% better than the baseline at all the PQ compute budgets. Furthermore, AdANNS -IVFOPQ has the same accuracy as the baselines at 8 × lower overhead. With DiskANN, AdANNS accelerates shortlist generation by using low-dimensional representations and recoups the accuracy by reranking with the highest-dimensional

Table 1: AdANNS -DiskANN using a 16 -d MR+ re-ranking with the 2048 -d MR outperforms DiskANN built on 2048 -d RR at half the compute cost on ImageNet retrieval.

MR at negligible cost. Table 1 shows that AdANNS -DiskANN is more accurate than the baseline for both 1-NN and ranking performance at only half the cost. Using low-dimensional representations further speeds up inference in AdANNS -DiskANN (see Appendix F).

These results show the generality of AdANNS and its broad applicability across a variety of ANNS indices built on top of the base building blocks. Currently, AdANNS piggybacks on typical ANNS pipelines for their inherent accounting of the real-world system constraints [16, 22, 25]. However, we believe that AdANNS 's flexibility and significantly better accuracy-compute trade-off can be further informed by real-world deployment constraints. We leave this high-potential line of work that requires extensive study to future research.

5 Further Analysis and Discussion

5.1 Compute-aware Elastic Search During Inference

AdANNS search structures cater to many specific large-scale use scenarios that need to satisfy precise resource constraints during construction as well as inference. However, in many cases, construction and storage of the indices are not the bottlenecks or the user is unable to search the design space. In these settings, AdANNS -D enables adaptive inference through accurate yet cheaper distance computation using the low-dimensional prefix of matryoshka representation. Akin to composite indices (Section 4.3) that use PQ vectors for cheaper distance computation, we can use the lowdimensional MR for faster distance computation on ANNS structure built non-adaptively with a high-dimensional MR without any modifications to the existing index.

Empirical results. Figure 2 shows that for a given compute budget using IVF on ImageNet-1K retrieval, AdANNS -IVF is better than AdANNS -IVF-D due to the explicit control during the building

of the ANNS structure which is expected. However, the interesting observation is that AdANNS -D matches or outperforms the IVF indices built with MRs of varying capacities for ImageNet retrieval.

However, these methods are applicable in specific scenarios of deployment. Obtaining optimal AdANNS search structure (highly accurate) or even the best IVF-MR index relies on a relatively expensive design search but delivers indices that fit the storage, memory, compute, and accuracy constraints all at once. On the other hand AdANNS -D does not require a precisely built ANNS index but can enable compute-aware search during inference. AdANNS -D is a great choice for setups that can afford only one single database/index but need to cater to varying deployment constraints, e.g., one task requires 70% accuracy while another task has a compute budget of 1 MFLOPS/query.

5.2 Why MRs over RRs ?

Quite a few of the gains from AdANNS are owing to the quality and capabilities of matryoshka representations. So, we conducted extensive analysis to understand why matryoshka representations seem to be more aligned for semantic search than the status-quo rigid representations.

Difficulty of NN search. Relative contrast ( C r ) [18] is inversely proportional to the difficulty of nearest neighbor search on a given database. On ImageNet-1K, Figure 14 shows that MRs have better C r than RRs across dimensionalities, further supporting that matryoshka representations are more aligned (easier) for NN search than existing rigid representations for the same accuracy. More details and analysis about this experiment can be found in Appendix H.2.

Clustering distributions. We also investigate the potential deviation in clustering distributions for MRs across dimensionalities compared to RRs . Unlike the RRs where the information is uniformly diffused across dimensions [49], MRs have hierarchical information packing. Figure 11 in Appendix E.3 shows that matryoshka representations result in clusters similar (measured by total variation distance [33]) to that of rigid representations and do not result in any unusual artifacts.

Robustness. Figure 9 in Appendix E shows that MRs continue to be better than RRs even for outof-distribution (OOD) image queries (ImageNetV2 [44]) using ANNS. It also shows that the highest data dimensionality need not always be the most robust which is further supported by the higher recall using lower dimensions. Further details about this experiment can be found in Appendix E.1.

Generality across encoders. IVFMR consistently has higher accuracy than IVFRR across dimensionalities despite having similar accuracies with exact NN search (for ResNet50 on ImageNet and BERT-Base on NQ). We find that our observations on better alignment of MRs for NN search hold across neural network architectures, ResNet18/34/101 [19] and ConvNeXt-Tiny [35]. Appendix H.3 delves deep into the experimentation done using various neural architectures on ImageNet-1K.

Recall score analysis. Analysis of recall score (see Appendix C) in Appendix H.1 shows that for a similar top-1 accuracy, lower-dimensional representations have better 1-Recall@1 across search complexities for IVF and HNSW on ImageNet-1K. Across the board, MRs have higher recall scores and top-1 accuracy pointing to easier 'searchability' and thus suitability of matryoshka representations for ANNS. Larger-scale experiments and further analysis can be found in Appendix H.

Through these analyses, we argue that matryoshka representations are better suited for semantic search than rigid representations, thus making them an ideal choice for AdANNS .

5.3 Search for AdANNS Hyperparameters

Choosing the optimal hyperparameters for AdANNS, such as d c , d s , m , # clusters, # probes, is an interesting and open problem that requires more rigorous examination. As the ANNS index is formed once and used for potentially billions of queries with massive implications for cost, latency and queries-per-second, a hyperparameter search for the best index is generally an acceptable industry practice [22, 38]. The Faiss library [24] provides guidelines 2 to choose the appropriate index for a specific problem, including memory constraints, database size, and the need for exact results. There have been efforts at automating the search for optimal indexing parameters, such as Autofaiss 3 , which maximizes recall given compute constraints.

In case of AdANNS , we suggest starting at the best configurations of MRs followed by a local design space search to lead to near-optimal AdANNS configurations (e.g. use IVF-MR to bootstrap AdANNS -IVF). We also share some observations during the course of our experiments:

AdANNS -IVF: Top-1 accuracy generally improves (with diminishing returns after a point) with increasing dimensionality of clustering ( d c ) and search ( d s ), as we show on ImageNet variants and with multiple encoders in the Appendix (Figures 9 and 15). Clustering with lowd MRs matches the performance of highd MRs as they likely contain similar amounts of useful information, making the increased compute cost not worth the marginal gains. Increasing # probes naturally boosts performance (Appendix, Figure 10a). Lastly, it is generally accepted that a good starting point for the # clusters k is √ N D / 2 , where N D is the number of indexable items [39]. k = √ N D is the optimal choice of k from a FLOPS computation perspective as can be seen in Appendix B.1.

AdANNS -OPQ: we observe that for a fixed compute budget in bytes ( m ), the top-1 accuracy reaches a peak at d < d max (Appendix, Table 4). We hypothesize that the better performance of AdANNS -OPQ at d < d max is due to the curse of dimensionality, i.e. it is easier to learn PQ codebooks on smaller embeddings with similar amounts of information. We find that using an MR with d = 4 × m is a good starting point on ImageNet and NQ. We also suggest using an 8-bit (256-length) codebook for OPQ as the default for each of the sub-block quantizer.

AdANNS -DiskANN: Our observations with DiskANN are consistent with other indexing structures, i.e. the optimal graph construction dimensionality d < d max (Appendix, Figure 12). A careful study of DiskANN on different datasets is required for more general guidelines to choose graph construction and OPQ dimensionality d .

5.4 Limitations

AdANNS 's core focus is to improve the design of the existing ANNS pipelines. To use AdANNS on a corpus, we need to back-fill [43] the MRs of the data - a significant yet a one-time overhead. We also notice that high-dimensional MRs start to degrade in performance when optimizing also for an extremely low-dimensional granularity (e.g., < 24 -d for NQ) - otherwise is it quite easy to have comparable accuracies with both RRs and MRs . Lastly, the existing dense representations can only in theory be converted to MRs with an auto-encoder-style non-linear transformation. We believe most of these limitations form excellent future work to improve AdANNS further.

6 Conclusions

We proposed a novel framework, AdANNS , that leverages adaptive representations for different phases of ANNS pipelines to improve the accuracy-compute tradeoff. AdANNS utilizes the inherent flexibility of matryoshka representations [31] to design better ANNS building blocks than the standard ones which use the rigid representation in each phase. AdANNS achieves SOTA accuracy-compute trade-off for the two main ANNS building blocks: search data structures ( AdANNS -IVF) and quantization ( AdANNS -OPQ). The combination of AdANNS -based building blocks leads to the construction of better real-world composite ANNS indices - with as much as 8 × reduction in cost at the same accuracy as strong baselines - while also enabling compute-aware elastic search. Finally, we note that combining AdANNS with elastic encoders [11] enables truly adaptive large-scale retrieval.

Acknowledgments

We are grateful to Kaifeng Chen, Venkata Sailesh Sanampudi, Sanjiv Kumar, Harsha Vardhan Simhadri, Gantavya Bhatt, Matthijs Douze and Matthew Wallingford for helpful discussions and feedback. Aditya Kusupati also thanks Tom Duerig and Rahul Sukthankar for their support. Part of the paper's large-scale experimentation is supported through a research GCP credit award from Google Cloud and Google Research. Sham Kakade acknowledges funding from the ONR award N00014-22-1-2377 and NSF award CCF-2212841. Ali Farhadi acknowledges funding from the NSF awards IIS 1652052, IIS 17303166, DARPA N66001-19-2-4031, DARPA W911NF-15-1-0543, and gifts from Allen Institute for Artificial Intelligence and Google.

References

M. Aumüller, E. Bernhardsson, and A. Faithfull. Ann-benchmarks: A benchmarking tool for approximate nearest neighbor algorithms. Information Systems , 87:101374, 2020.

Y. Bengio. Deep learning of representations for unsupervised and transfer learning. In Proceedings of ICML workshop on unsupervised and transfer learning , pages 17-36. JMLR Workshop and Conference Proceedings, 2012.

E. Bernhardsson. Annoy: Approximate Nearest Neighbors in C++/Python , 2018. URL https://pypi.org/project/annoy/ . Python package version 1.13.0.

S. Brin and L. Page. The anatomy of a large-scale hypertextual web search engine. Computer networks and ISDN systems , 30(1-7):107-117, 1998.

D. Cai. A revisit of hashing algorithms for approximate nearest neighbor search. IEEE Transactions on Knowledge and Data Engineering , 33(6):2337-2348, 2021. doi: 10.1109/ TKDE.2019.2953897.

T. Chen, L. Li, and Y. Sun. Differentiable product quantization for end-to-end embedding compression. In International Conference on Machine Learning , pages 1617-1626. PMLR, 2020.

K. L. Clarkson. An algorithm for approximate closest-point queries. In Proceedings of the tenth annual symposium on Computational geometry , pages 160-164, 1994.

M. Datar, N. Immorlica, P. Indyk, and V. S. Mirrokni. Locality-sensitive hashing scheme based on p-stable distributions. In Proceedings of the twentieth annual symposium on Computational geometry , pages 253-262, 2004.

J. Dean. Challenges in building large-scale information retrieval systems. In Keynote of the 2nd ACM International Conference on Web Search and Data Mining (WSDM) , volume 10, 2009.

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Devvrit, S. Kudugunta, A. Kusupati, T. Dettmers, K. Chen, I. Dhillon, Y. Tsvetkov, H. Hannaneh, S. Kakade, A. Farhadi, and P. Jain. Matformer: Nested transformer for elastic inference. arXiv preprint arxiv:2310.07707 , 2023.

J. H. Friedman, J. L. Bentley, and R. A. Finkel. An algorithm for finding best matches in logarithmic expected time. ACM Transactions on Mathematical Software (TOMS) , 3(3):209226, 1977.

T. Ge, K. He, Q. Ke, and J. Sun. Optimized product quantization for approximate nearest neighbor search. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2946-2953, 2013.

G. Golub and W. Kahan. Calculating the singular values and pseudo-inverse of a matrix. Journal of the Society for Industrial and Applied Mathematics, Series B: Numerical Analysis , 2(2): 205-224, 1965.

R. Gray. Vector quantization. IEEE Assp Magazine , 1(2):4-29, 1984.

R. Guo, P. Sun, E. Lindgren, Q. Geng, D. Simcha, F. Chern, and S. Kumar. Accelerating large-scale inference with anisotropic vector quantization. In International Conference on Machine Learning , pages 3887-3896. PMLR, 2020.

N. Gupta, P. H. Chen, H.-F. Yu, C.-J. Hsieh, and I. S. Dhillon. End-to-end learning to index and search in large output spaces. arXiv preprint arXiv:2210.08410 , 2022.

J. He, S. Kumar, and S.-F. Chang. On the difficulty of nearest neighbor search. In International Conference on Machine Learning (ICML) , 2012.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770778, 2016.

K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9729-9738, 2020.

P. Indyk and R. Motwani. Approximate nearest neighbors: towards removing the curse of dimensionality. In Proceedings of the thirtieth annual ACM symposium on Theory of computing , pages 604-613, 1998.

S. Jayaram Subramanya, F. Devvrit, H. V. Simhadri, R. Krishnawamy, and R. Kadekodi. Diskann: Fast accurate billion-point nearest neighbor search on a single node. Advances in Neural Information Processing Systems , 32, 2019.

H. Jegou, M. Douze, and C. Schmid. Product quantization for nearest neighbor search. IEEE transactions on pattern analysis and machine intelligence , 33(1):117-128, 2010.

J. Johnson, M. Douze, and H. Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data , 7(3):535-547, 2019.

W. B. Johnson. Extensions of lipschitz mappings into a hilbert space. Contemp. Math. , 26: 189-206, 1984.

I. T. Jolliffe and J. Cadima. Principal component analysis: a review and recent developments. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences , 374(2065):20150202, 2016.

V. Karpukhin, B. O˘ guz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W.-t. Yih. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906 , 2020.

S. Kornblith, J. Shlens, and Q. V . Le. Do better imagenet models transfer better? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2661-2671, 2019.

T. Kraska, A. Beutel, E. H. Chi, J. Dean, and N. Polyzotis. The case for learned index structures. In Proceedings of the 2018 international conference on management of data , pages 489-504, 2018.

A. Kusupati, M. Wallingford, V. Ramanujan, R. Somani, J. S. Park, K. Pillutla, P. Jain, S. Kakade, and A. Farhadi. Llc: Accurate, multi-purpose learnt low-dimensional binary codes. Advances in Neural Information Processing Systems , 34:23900-23913, 2021.

A. Kusupati, G. Bhatt, A. Rege, M. Wallingford, A. Sinha, V. Ramanujan, W. Howard-Snyder, K. Chen, S. Kakade, P. Jain, and A. Farhadi. Matryoshka representation learning. In Advances in Neural Information Processing Systems , December 2022.

T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics , 7:453-466, 2019.

D. A. Levin and Y. Peres. Markov chains and mixing times , volume 107. American Mathematical Soc., 2017.

W. Li, Y. Zhang, Y. Sun, W. Wang, W. Zhang, and X. Lin. Approximate nearest neighbor search on high dimensional data-experiments, analyses, and improvement. IEEE Transactions on Knowledge and Data Engineering , 2020.

Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11976-11986, 2022.

S. Lloyd. Least squares quantization in pcm. IEEE transactions on information theory , 28(2): 129-137, 1982.

Y. Malkov, A. Ponomarenko, A. Logvinov, and V. Krylov. Approximate nearest neighbor algorithm based on navigable small world graphs. Information Systems , 45:61-68, 2014.

Y. A. Malkov and D. Yashunin. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE Transactions on Pattern Analysis & Machine Intelligence , 42(04):824-836, 2020.

K. Mardia, J. Kent, and J. Bibby. Multivariate analysis. Probability and Mathematical Statistics , 1979.

P. Nayak. Understanding searches better than ever before. Google AI Blog , 2019. URL https: //blog.google/products/search/search-language-understanding-bert/ .

A. Neelakantan, T. Xu, R. Puri, A. Radford, J. M. Han, J. Tworek, Q. Yuan, N. Tezak, J. W. Kim, C. Hallacy, et al. Text and code embeddings by contrastive pre-training. arXiv preprint arXiv:2201.10005 , 2022.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , pages 8748-8763. PMLR, 2021.

V. Ramanujan, P. K. A. Vasu, A. Farhadi, O. Tuzel, and H. Pouransari. Forward compatible training for representation learning. arXiv preprint arXiv:2112.02805 , 2021.

B. Recht, R. Roelofs, L. Schmidt, and V. Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning , pages 5389-5400. PMLR, 2019.

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision , 115(3):211-252, 2015.

R. Salakhutdinov and G. Hinton. Semantic hashing. International Journal of Approximate Reasoning , 50(7):969-978, 2009.

H. V. Simhadri, G. Williams, M. Aumüller, M. Douze, A. Babenko, D. Baranchuk, Q. Chen, L. Hosseini, R. Krishnaswamy, G. Srinivasa, et al. Results of the neurips'21 challenge on billion-scale approximate nearest neighbor search. arXiv preprint arXiv:2205.03763 , 2022.

J. Sivic and A. Zisserman. Video google: A text retrieval approach to object matching in videos. In Computer Vision, IEEE International Conference on , volume 3, pages 1470-1470. IEEE Computer Society, 2003.

D. Soudry, E. Hoffer, M. S. Nacson, S. Gunasekar, and N. Srebro. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research , 19(1):2822-2878, 2018.

C. Waldburger. As search needs evolve, microsoft makes ai tools for better search available to researchers and developers. Microsoft AI Blog , 2019. URL https://blogs.microsoft. com/ai/bing-vector-search/ .

M. Wang, X. Xu, Q. Yue, and Y. Wang. A comprehensive survey and experimental comparison of graph-based approximate nearest neighbor search. Proceedings of the VLDB Endowment , 14 (11):1964-1978, 2021.

R. Weber, H.-J. Schek, and S. Blott. A quantitative analysis and performance study for similaritysearch methods in high-dimensional spaces. In VLDB , volume 98, pages 194-205, 1998.

I. H. Witten, I. H. Witten, A. Moffat, T. C. Bell, T. C. Bell, E. Fox, and T. C. Bell. Managing gigabytes: compressing and indexing documents and images . Morgan Kaufmann, 1999.

A AdANNS Framework

Algorithm 1 AdANNS-IVF Psuedocode

return k_nearest_neighbors

Query

µ

1

𝑋 𝜖 C

1

µ

i

𝑋 𝜖 C

i

µ

k

𝑋 𝜖 C

k

Linear

Scan	with

ℝ

d

s

Top	k

Relevant

Data	Points

Select	Closest	Cluster

Database

Construct	Clusters	with	ℝ

d

c

𝜙: 𝑋 → ℝ

!

Construction

Inference

B Training and Compute Costs

A bulk of our ANNS experimentation was written with Faiss [24], a library for efficient similarity search and clustering. AdANNS was implemented from scratch (Algorithm 1) due to difficulty in decoupling clustering and linear scan with Faiss, with code available at https://github.com/ RAIVNLab/AdANNS . We also provide a version of AdANNS with Faiss optimizations with the restriction that D c ≥ D s as a limitation of the current implementation, which can be further optimized. All ANNS experiments ( AdANNS -IVF, MG-IVFRR , IVFMR , IVFRR , HNSW, HNSWOPQ, IVFOPQ) were run on an Intel Xeon 2.20GHz CPU with 12 cores. Exact Search (Flat L2, PQ, OPQ) and DiskANN experiments were run with CUDA 11.0 on a A100-SXM4 NVIDIA GPU with 40G RAM. The wall-clock inference times quoted in Figure 1a and Table 3 are reported on CPU with Faiss optimizations, and are averaged over three inference runs for ImageNet-1K retrieval.

DPR [27] on NQ [32]. We follow the setup on the DPR repo 4 : the Wikipedia corpus has 21 million passages and Natural Questions dataset for open-domain QA settings. The training set contains 79,168 question and answer pairs, the dev set has 8,757 pairs and the test set has 3,610 pairs.

B.1 Inference Compute Cost

We evaluate inference compute costs for IVF in MegaFLOPS per query (MFLOPS/query) as shown in Figures 2, 10a, and 8 as follows:

where d c is the cluster construction embedding dimensionality, d s is the embedding dim used for linear scan within each probed cluster, which is controlled by # of search probes n p . Finally, k is the number of clusters | C i | indexed over database of size N D . The default setting in this work, unless otherwise stated, is n p = 1 , k = 1024 , N D = 1281167 (ImageNet-1K trainset). Vanilla IVF supports only d c = d s , while AdANNS -IVF provides flexibility via decoupling clustering and search (Section 4). AdANNS -IVF-D is a special case of AdANNS -IVF with the flexibility restricted to inference, i.e., d c is a fixed high-dimensional MR .

C Evaluation Metrics

In this work, we primarily use top-1 accuracy (i.e. 1-Nearest Neighbor), recall@k, corrected mean average precision (mAP@k) [30] and k-Recall@N (recall score), which are defined over all queries Q over indexed database of size N D as:

where correct_pred @ k is the number of k-NN with correctly predicted labels for a given query. As noted in Section 3, k-Recall@N is the overlap between k exact search nearest neighbors (considered as ground truth) and the top N retrieved documents. As Faiss [24] supports a maximum of 2048NN while searching the indexed database, we report 40-Recall@2048 in Figure 13. Also note that for ImageNet-1K, which constitutes a bulk of the experimentation in this work, | Q | = 50000 , | N D | = 1281167 and num_classes = 1000 . For ImageNetv2 [44], | Q | = 10000 and num_classes = 1000 , and for ImageNet-4K [31], | Q | = 210100 , | N D | = 4202000 and num_classes = 4202 . For NQ [32], | Q | = 3610 and | N D | = 21015324 . As NQ consists of question-answer pairs (instancelevel), num_classes = 3610 for the test set.    ⋂}√∖(̂̂√√∐̂glyph[arrowbt]({⌋} ⋂}√∖(̂̂√√∐̂glyph[arrowbt]({⌋}

̂⊗⊗⋃∖∮∧⊔〉⊗∑∏

∮∧⊔〉⊗∑∏∖⊕∫∖

}⌉√√√˜(√̂˜˜√({glyph[arrowbt]√˜√}

̂⊗⊗⋃∖∮⊗⋃∨〉⊗∑∏

∮⊗⋃∨〉⊗∑∏∖⊕∫∖

}⌉√√√˜(√̂˜˜√({glyph[arrowbt]√˜√}









}⌉√√√˜(√̂˜˜√({glyph[arrowbt]√˜√}



̂⊗⊗⋃∖⊗∑∏

⊗∑∏∖⊕∫∖

∫]˜]̂∖⊗∑∏

(a) Exact Search + OPQ on ImageNet-1K

















∫]˜]̂∖∮∧⊔〉⊗∑∏

(b) IVF + OPQ on ImageNet-1K









}⌉√√√˜(√̂˜˜√({glyph[arrowbt]√˜√}

/

/

/

⋂}√∖(̂̂√√∐̂glyph[arrowbt]({⌋}

̂⊗⊗⋃∖〈]√⌋⊗⊗〉⊗∑∏

〈]√⌋⊗⊗〉⊗∑∏∖⊕∫∖

∫]˜]̂∖〈]√⌋⊗⊗〉⊗∑∏

(c) DiskANN + OPQ on ImageNet-1K

















⋂}√∖(̂̂√√∐̂glyph[arrowbt]({⌋}

∫]˜]̂∖∮⊗⋃∨〉⊗∑∏

(d) HNSW + OPQ on ImageNet-1K

D AdANNS -OPQ

In this section, we take a deeper dive into the quantization characteristics of MR . In this work, we restrict our focus to optimized product quantization (OPQ) [13], which adds a learned space rotation and dimensionality permutation to PQ's sub-vector quantization to learn more optimal PQ codes. We compare OPQ to vanilla PQ on ImageNet in Table 4, and observe large gains at larger embedding dimensionalities, which agrees with the findings of Jayaram Subramanya et al. [22].

We perform a study of composite OPQ m × b indices on ImageNet-1K across compression compute budgets m (where b = 8 , i.e. 1 Byte), i.e. Exact Search with OPQ, IVF+OPQ, HNSW+OPQ, and DiskANN+OPQ, as seen in Figure 6. It is evident from these results:

Learning OPQ codebooks with AdANNS (Figure 6a) provides a 1-5% gain in top-1 accuracy over rigid representations at low compute budgets ( ≤ 32 Bytes). AdANNS -OPQ saturates to Rigid-OPQ performance at low compression ( ≥ 64 Bytes).

For IVF, learning clusters with MRs instead of RRs (Figure 6b) provides substantial gains (14%). In contrast to Exact-OPQ, using AdANNS for learning OPQ codebooks does not provide substantial top-1 accuracy gains over MR with d = 2048 (highest), though it is still slightly better or equal to MR -2048 at all compute budgets. This further supports that IVF performance generally scales with embedding dimensionality, which is consistent with our findings on ImageNet across robustness variants and encoders (See Figures 9 and 15 respectively).

Note that in contrast to Exact, IVF, and HNSW coarse quantizers, DiskANN inherently re-ranks the retrieved shortlist with high-precision embeddings ( d = 2048 ), which is reflected in its high top-1 accuracy. We find that AdANNS with 8 -byte OPQ (Figure 6c) matches the top-1 accuracy of rigid representations using 32 -byte OPQ, for a 4 × cost reduction for the same accuracy. Also note that using AdANNS provides large gains over using MR -2048 at high compression (1.5%), highlighting the necessity of AdANNS 's flexibility for high-precision retrieval at low compute budgets.

Our findings on the HNSW-OPQ composite index (Figure 6d) are consistent with all other indices, i.e. HNSW graphs constructed with AdANNS OPQ codebooks provide significant gains over RR and MR , especially at high compression ( ≤ 32 Bytes).

OPQ on NQ dataset





































































(a) Exact Search + OPQ on Natural Questions

(b) IVF + OPQ on Natural Questions

Our observations on ImageNet with ResNet-50 MR across search structures also extend to the Natural Questions dataset with Dense Passage Retriever (DPR with BERT-Base MR embeddings). We note that AdANNS provides gains over RR -768 embeddings for both Exact Search and IVF with OPQ (Figure 7a and 7b). We find that similar to ImageNet (Figure 15) IVF performance on Natural Questions generally scales with dimensionality. AdANNS thus reduces to MR -768 performance for M ≥ 16 . See Appendix G for a more in-depth discussion of AdANNS with DPR on Natural Questions.

E AdANNS -IVF

Inverted file index (IVF) [48] is a simple yet powerful ANNS data structure used in web-scale search systems [16]. IVF construction involves clustering (coarse quantization often through k-means) [36] on d -dimensional representation that results in an inverted file list [53] of all the data points in each cluster. During search, the d -dimensional query representation is first assigned to the closest clusters (# probes, typically set to 1) and then an exhaustive linear scan happens within each cluster to obtain the nearest neighbors. As seen in Figure 9, IVF top-1 accuracy scales logarithmically with increasing representation dimensionality d on ImageNet-1K/V2/4K. The learned lowd representations thus provide better accuracy-compute trade-offs compared to highd representations, thus furthering the case for usage of AdANNS with IVF.

Our proposed adaptive variant of IVF, AdANNS -IVF, decouples the clustering, with d c dimensions, and the linear scan within each cluster, with d s dimensions - setting d c = d s results in nonadaptive vanilla IVF. This helps in the smooth search of design space for the optimal accuracycompute trade-off. A naive instantiation yet strong baseline would be to use explicitly trained d c and d s dimensional rigid representations (called MG-IVFRR , for multi-granular IVF with









































rigid representations). We also examine the setting of adaptively choosing low-dimensional MR to linear scan the shortlisted clusters built with high-dimensional MR , i.e. AdANNS -IVF-D, as seen in Table 5. As seen in Figure 8, AdANNS -IVF provides pareto-optimal accuracy-compute tradeoff across inference compute. This figure is a more exhaustive indication of AdANNS -IVF behavior compared to baselines than Figures 1a and 2. AdANNS -IVF is evaluated for all possible tuples of d c , d s , k = | C | ∈ { 8 , 16 , . . . , 2048 } . AdANNS -IVF-D is evaluated for a pre-built IVF index with d c = 2048 and d s ∈ { 8 , . . . , 2048 } . MG-IVFRR configurations are evaluated for d c ∈ { 8 , . . . , d s } , d s ∈ { 32 , . . . , 2048 } and k = 1024 clusters. A study over additional k values is omitted due to high compute cost. Finally, IVFMR and IVFRR configurations are evaluated for d c = d s ∈ { 8 , 16 , . . . , 2048 } and k ∈ { 256 , . . . , 8192 } . Note that for a fair comparison, we use n p = 1 across all configurations. We discuss the inference compute for these settings in Appendix B.1.

E.1 Robustness











































(a) ImageNet-1K











































(b) ImageNetV2





































(c) ImageNet-4K

As shown in Figure 9, we examined the clustering capabilities of MRs on both in-distribution (ID) queries via ImageNet-1K and out-of-distribution (OOD) queries via ImageNetV2 [44], as well as on larger-scale ImageNet-4K [31]. For ID queries on ImageNet-1K (Figure 9a), IVFMR is at least as accurate as ExactRR for d ≤ 256 with a single search probe, demonstrating the quality of indistribution low-d clustering with MR . On OOD queries (Figure 9b), we observe that IVFMR is on average 2% more robust than IVFRR across all cluster construction and linear scan dimensionalities d . It is also notable that clustering with MRs followed by linear scan with # probes = 1 is more robust than exact search with RR embeddings across all d ≤ 2048 , indicating the adaptability of MRs to distribution shifts during inference. As seen in Table 5, on ImageNetV2 AdANNS -IVF-D is the best configuration for d ≤ 16 , and is similarly accurate to IVFMR at all other d . AdANNS -IVF-D with

d = 128 is able to match its own accuracy with d = 2048 , a 16 × compute gain during inference. This demonstrates the potential of AdANNS to adaptively search pre-indexed clustering structures.

On 4-million scale ImageNet-4K (Figure 9c), we observe similar accuracy trends of IVFMR compared to ExactMR as in ImageNet-1K (Figure 9a) and ImageNetV2 (Figure 9b). We omit baseline IVFRR and ExactRR experiments due to high compute cost at larger scale.

E.2 IVFMR Ablations



























×













(a) 4K Search Probes (

n

p

)



















































(b) Centroid Recall

As seen in Figure 10a, IVFMR can match the accuracy of Exact Search on ImageNet-4K with ∼ 100 × less compute. We also explored the capability of MRs at retrieving cluster centroids with low-d compared to a ground truth of 2048-d with k-Recall@N, as seen in Figure 10b. MRs were able to saturate to near-perfect 1-Recall@N for d ≥ 32 and N ≥ 4 , indicating the potential of AdANNS at matching exact search performance with less than 10 search probes n p .

E.3 Clustering Distribution

We examined the distribution of learnt clusters across embedding dimensionalities d for both MR and RR models, as seen in Figure 11. We observe IVFMR to have less variance than IVFRR at d ∈ { 8 , 16 } , and slightly higher variance for d ≥ 32 , while IVFMR outperforms IVFRR in top-1 across all d (Figure 9a). This indicates that although MR learns clusters that are less uniformly distributed than RR at high d , the quality of learnt clustering is superior to RR across all d . Note that a uniform distribution is N/k data points per cluster, i.e. ∼ 1250 for ImageNet-1K with k = 1024 . We quantitatively evaluate the proximity of the MR and RR clustering distributions with Total Variation

0

500

1000

1500

2000

2500

Number of data points per cell

0

50

100

150

200

250

300

Number of cells

dTV

=0.00078

dTV

, 2048=0.00128

MR-8

RR-8

0

500

1000

1500

2000

2500

Number of data points per cell

0

50

100

150

200

250

300

Number of cells

dTV

=0.00027

dTV

, 2048=0.00121

RR-16

0

500

1000

1500

2000

2500

Number of data points per cell

0

50

100

150

200

250

300

350

Number of cells

dTV

=0.00016

dTV

, 2048=0.00095

MR-32

RR-32

0

500

1000

1500

2000

2500

Number of data points per cell

0

50

100

150

200

250

300

350

Number of cells

dTV

=0.00037

dTV

, 2048=0.00078

MR-64

RR-64

0

500

1000

1500

2000

2500

Number of data points per cell

0

50

100

150

200

250

300

350

Number of cells

dTV

=0.00061

dTV

, 2048=0.00069

MR-128

RR-128

0

500

1000

1500

2000

2500

Number of data points per cell

0

100

200

300

400

Number of cells

dTV

=0.00068

dTV

, 2048=0.00064

MR-256

RR-256

0

500

1000

1500

2000

2500

Number of data points per cell

0

50

100

150

200

250

300

350

Number of cells

dTV

=0.00076

dTV

, 2048=0.00065

MR-512

RR-512

0

500

1000

1500

2000

2500

Number of data points per cell

0

100

200

300

400

Number of cells

dTV

=0.00075

dTV

, 2048=0.00059

MR-1024

RR-1024

0

500

1000

1500

2000

2500

Number of data points per cell

0

100

200

300

400

Number of cells

dTV

=0.00054

dTV

, 2048=0.00054

MR-2048

Distance [33], which is defined over two discrete probability distributions p, q over [ n ] as follows:

We also compute d TV, 2048 (MR -d ) = d TV (MR -d , RR -2048 ) , which evaluates the total variation distance of a given low-d MR from high-d RR -2048. We observe a monotonically decreasing d TV, 2048 with increasing d , which demonstrates that MR clustering distributions get closer to RR -2048 as we increase the embedding dimensionality d . We observe in Figure 11 that d TV (MR -d , RR -d ) ∼ 7 e -4 for d ∈ { 8 , 256 , . . . , 2048 } and ∼ 3 e -4 for d ∈ { 16 , 32 , 64 } . These findings agree with the top-1 improvement of MR over RR as shown in Figure 9a, where there are smaller improvements for d ∈ { 16 , 32 , 64 } (smaller d TV ) and larger improvements for d ∈ { 8 , 256 , . . . , 2048 } (larger d TV ). These results demonstrate a correlation between top-1 performance of IVFMR and the quality of clusters learnt with MR .

F AdANNS -DiskANN

DiskANN is a state-of-the-art graph-based ANNS index capable of serving queries from both RAM and SSD. DiskANN builds a greedy best-first graph with OPQ distance computation, with compressed vectors stored in memory. The index and full-precision vectors are stored on the SSD. During search,

when a query's neighbor shortlist is fetched from the SSD, its full-precision vector is also fetched in a single disk read. This enables efficient and fast distance computation with PQ on a large initial shortlist of candidate nearest neighbors in RAM followed by a high-precision re-ranking with fullprecision vectors fetched from the SSD on a much smaller shortlist. The experiments carried out in this work primarily utilize a DiskANN graph index built in-memory 5 with OPQ distance computation.

As with IVF, DiskANN is also well suited to the flexibility provided by AdANNS as we demonstrate on both ImageNet and NQ that the optimal PQ codebook for a given compute budget is learnt with a smaller embedding dimensionality d (see Figures 6c and 7a). We demonstrate the capability of AdANNS -DiskANN with a compute budget of m ∈ { 32 , 64 } in Table 1. We tabulate the search time latency of AdANNS -DiskANN in microseconds ( µs ) in Table 6, which grows linearly with graph construction dimensionality d . We also examine DiskANNMR with SSD graph indices on ImageNet-1K across OPQ budgets for distance computation m dc ∈ { 32 , 48 , 64 } , as seen in Figure 12. With SSD indices, we store PQ-compressed vectors on disk with m disk = m dc , which essentially disables DiskANN's implicit high-precision re-ranking. We ob-





































serve similar trends to other composite ANNS indices on ImageNet, where the optimal dim for fixed OPQ budget is not the highest dim ( d = 1024 with fp32 embeddings is current highest dim supported by DiskANN which stores vectors in 4KB sectors on disk). This provides further motivation for AdANNS -DiskANN, which leverages MRs to provide flexible access to the optimal dim for quantization and thus enables similar Top-1 accuracy to Rigid DiskANN for up to 1 / 4 the cost (Figure 6c).

G AdANNS on Natural Questions

In addition to image retrieval on ImageNet, we also experiment with dense passage retrieval (DPR) on Natural Questions. As shown in Figure 6, MR representations are 1 -10% more accurate than their

RR counterparts across PQ compute budgets with Exact Search + OPQ on NQ. We also demonstrate that IVFMR is 1 -2 . 5% better than IVFRR for Precision@ k , k ∈ { 1 , 5 , 20 , 100 , 200 } . Note that on NQ, IVF loses ∼ 10% accuracy compared to exact search, even with the RR -768 baseline. We hypothesize the weak performance of IVF owing to poor clusterability of the BERT-Base embeddings fine-tuned on the NQ dataset. A more thorough exploration of AdANNS -IVF on NQ is an immediate future work and is in progress.

H Ablations

H.1 Recall Score Analysis











np



















































np



















































































































































np







































np







































































In this section we also examine the variation of k-Recall@N with by probing a larger search space with IVF and HNSW indices. For IVF, search probes represent the number of clusters shortlisted for linear scan during inference. For HNSW, search quality is controlled by the efSearch parameter [38], which represents the closest neighbors to query q at level l c of the graph and is analogous to number of search probes in IVF. As seen in Figure 13, general trends show a) an intuitive increase in recall with increasing search probes n p ) for fixed search probes, b) a decrease in recall with increasing search dimensionality d c) similar trends in ImageNet-1K and 4 × larger ImageNet-4K.

H.2 Relative Contrast

We utilize Relative Contrast [18] to capture the difficulty of nearest neighbors search with IVFMR compared to IVFRR . For a given database X = { x i ∈ R d , i = 1 , . . . , N D } , a query q ∈ R d , and a distance metric D ( ., . ) we compute relative contrast C r as a measure of the difficulty in finding the 1-nearest neighbor (1-NN) for a query q in database X as follows:

Compute D q min = min i =1 ...n D ( q, x i ) , i.e. the distance of query q to its nearest neighbor x q nn ∈ X

Compute D q mean = E x [ D ( q, x )] as the average distance of query q from all database points x ∈ X

q

Relative Contrast of a given query C q r = D mean D q min , which is a measure of how separable the query's nearest neighbor x q nn is from an average point in the database x

Compute an expectation over all queries for Relative Contrast over the entire database as

It is evident that C r captures the difficulty of Nearest Neighbor Search in database X , as a C r ∼ 1 indicates that for an average query, its nearest neighbor is almost equidistant from a random point in the database. As demonstrated in Figure 14, MRs have higher R c than RR Embeddings for an Exact Search on ImageNet-1K for all d ≥ 16 . This result implies that a portion of MR 's improvement over RR for 1-NN retrieval across all embedding dimensionalities d [31] is due to a higher average separability of the MR 1-NN from a random database point.

H.3 Generality across Encoders

We perform an ablation over the representation function φ : X → R d learnt via a backbone neural network (primarily ResNet50 in this work), as detailed in Section 3. We also train MRL models [31] φ MR ( d ) on ResNet18/34/101 [19] that are as accurate as their independently trained RR baseline models φ RR ( d ) , where d is the default max representation size of each architecture. We also train

MRL with a ConvNeXt-Tiny backbone with [ d ] = { 48 , 96 , 192 , 384 , 786 } . MR -768 has a top-1 accuracy of 79 . 45% compared to independently trained publicly available RR -768 baseline with top-1 accuracy 82 . 1% (Code and RR model available on the official repo 6 ). We note that this training had no hyperparameter tuning whatsoever, and this gap can be closed with additional model training effort. We then compare clustering the MRs via IVFMR with k = 2048 , n p = 1 on ImageNet-1K to ExactMR , which is shown in Figure 15. IVFMR shows similar trends across backbones compared to ExactMR , i.e. a maximum top-1 accuracy drop of ∼ 1 . 6% for a single search probe. This suggests the clustering capabilities of MR extend beyond an inductive bias of φ MR ( d ) ∈ ResNet50, though we leave a detailed exploration for future work.

















































Peer reviews:
comments: SUMMARY:
This paper proposes to use the Matryoshka Representations for approximate nearest neighbor search (ANNS). Matryoshka Representations provide the flexibility to adjust the budget for index search, index storage, and distance computation by changing the dimension of the used embedding. The paper instantiates the idea using IVF and OPQ, two popular techniques for ANNS, and shows impressive experiment results.

STRENGTHS:
I like that this paper focuses on the end-to-end performance of ANNS, i.e., the top-1 accuracy of ANNS applications. This differs from the widely used recall, which only considers distance in the embedding space, and could be inspiring for the field. 
1. The property of Matryoshka Representations is interesting and using it to make ANNS flexible makes sense. However, the authors could introduce more about Matryoshka Representations, e.g., how are they trained and what is the training cost compared with standard embedding.
2. Two examples of using the Matryoshka Representations are provided.
3. The experiment results are comprehensive and impressive. 
4. The gains of Matryoshka Representations and limitations are discussed in Section 5.

WEAKNESSES:
The labels of the figures need to be enhanced to make them self-content.

QUESTIONS:
NA

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMatryoshka Representation representations have the advantage that the first m-bits of the d-dim vector can as-is serve as a good m-dim representation of the original d-dim vector. This paper demonstrates how Matryoshka Representations (MR) can be used together with approximate nearest neighbor search indices to speed up test-time inference as well as optimize for indexing cost and storage budgets for semantic search applications. The authors run extensive experiments on two datasets with up to 21 million items, carefully exploring the design space.\n\nUpdate: I have updated my score to 7 after reading clarifications provided by authors.\n\nSTRENGTHS:\n- The paper is very well-written and easy to follow (although it requires quite a few jumps to and from the appendix to really understand the results :) ).\n- The papers presents a clever use of properties of Matryoshka Representations for semantic search. Unlike standard dimensionality reduction methods like SVD or random projection, use of MR representations allows use of lower-dim representations when needed without additional computation to get low-dim representations.\n- Extensive experiments and analysis of the proposed approach.\n\nWEAKNESSES:\n- Too many hyper-parameters to optimize — This might limit wide scale adoption of proposed methods.\n    - A major chunk of performance boost observed comes from merely using Matryoshka Representations (MR) instead of traditional dense representations (RR). For instance, in Fig. 2, IVF built with MR gives significant improvement over IVF built with RR. While proposed adaptive IVF outperforms fixed-IVF with MR, it may require an non-trivial exhaustive search of hyper-parameters. And, as show in Fig. 10, only a small fraction of these hyper-parameter settings for adaptive-IVF-MR outperform fixed-IVF with MR representations.\n    - It would be helpful to also include a practical guide of how to choose these hyper-parameters in practice. In my opinion, this can significantly help wide-scale adoption of ideas in this work.\n- (Minor) The proposed adaptive search methods only work with Matryoshka Representations and can not be applied directly to any dense vector representations. This might limit the impact of the paper as the user may not have the budget or infrastructure to train MR representations especially when such embeddings are obtained form some large pre-trained vision/language models.\n\nQUESTIONS:\n- Is ground-truth 1-NN (as per the dense representations) used or ground-truth label (as per labelled data for downstream task) used for eval?\n- Can authors elaborate on why using 2k clusters with d/2 dim vectors is just as costly as using k clusters with d-dim vectors? What cost is being referred in this statement? Is it test-time inference cost or is it indexing cost?\n- In Fig.2, why does AdANNS-IVF perform better than AdANNS-IVF-D? Why does using a smaller dim MR embedding give better results than using full 2048-dim embedding for clustering?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduced AdANNS, a framework that effectively harnesses the flexibility of Matryoshka Representations. This approach is applied to two fundamental components of typical ANNS systems: (a) the search data structure that stores datapoints, and (b) the distance computation that maps a given query to points within the data structure. To incorporate matryoshka representations with each of these ANNS building blocks, the authors proposed AdANNS-IVF and AdANNS-OPQ.\n\nExtensive experiments demonstrated that AdANNS achieves a state-of-the-art accuracy-compute trade-off for the two primary ANNS building blocks. Moreover, when combining AdANNS-based building blocks, superior real-world composite ANNS indices can be constructed.\n\nDespite these findings, it should be noted that the authors conducted their research on small-scale open-source datasets only. Nevertheless, they assert that the primary contribution of this research is the proposal of AdANNS, which they claim is applicable to web-scale industrial systems.\n\nSTRENGTHS:\n(1) Although Matryoshka Representations (MR) already contain multi-granular representations, meticulous integration with ANNS building blocks is paramount to develop a functional method, representing the key contribution of this work.\n(2) Through extensive experiments, it has been demonstrated that AdANNS achieves a state-of-the-art accuracy-compute trade-off for the two primary ANNS building blocks. Furthermore, the combination of AdANNS-based building blocks results in the creation of superior real-world composite ANNS indices.\n\nWEAKNESSES:\n(1) It may be beneficial for the authors to restructure the introduction. For instance, in the second paragraph, the authors delve into the details of IVF, explaining its two phases and why using the same high-dimensional Rigid Representation (RR) for cluster mapping and linear scanning could be sub-optimal. While I understand the authors' intent to introduce adaptive representations followed by matryoshka representations, I believe it would be more logical to first describe each component of ANNS and explain how adaptive representations could serve as a replacement for rigid representations.\n\n(2) It's important to acknowledge that the authors conducted their research exclusively on small-scale open-source datasets. Despite this limitation, they maintain that the main contribution of their study is the introduction of AdANNS, which they assert can be applied to web-scale industrial systems.\n\nQUESTIONS:\nWhy didn't you include a larger dataset for the experiments? Was it due to data availability or limitations in your implementation?\n\nLIMITATIONS:\nThe authors have discussed the limitations of this work in section 5.3: To use AdANNS on a corpus, one need to back-fill the MRs of the data – a significant yet a one-time overhead.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe author introduced an adaptive method for searching near-neighbors called AdANNS, which employs different representations of the same item at various stages of the engine. Rather than relying on traditional fixed vector representations, the authors utilized Matryoshka representations, creating a nested representation with varying dimensions for each item.\n\nSTRENGTHS:\n1) AdANNS beats the traditional IVF and OPQ on accuracy and compute-resource tradeoff by proposing two main ANNS building blocks: search data structures (AdANNS-IVF) and quantization (AdANNS-OPQ).\n2) The comparisons are made against the standard NN approaches IVF and OPQ.\n\nWEAKNESSES:\n1) Limited datasets: The authors have only used ImagNet-1K and NQ datasets for vector-based retrieval. Many standard open-source datasets are available for this purpose (ANN benchmark and BigANN benchmark). A simple MLP/autoencoder can be trained for varying embedding sizes, or dimensionality reduction (like PCA, random projection) methods can be used.\n2) The paper is weak in novelty. It uses the existing method Matryyoshika Representations of each vector for the near neighbor retrieval. I do acknowledge that the modern composite ANNS index framework is being modified to support the Matryyoshika Representations. This provides a gain in top-1 accuracy for a given compute budget. However, I see this as a trivial change.\n3) There is no comprehensive theoretical analysis that compares the AdANNS-IVF, AdANNSOPQ, and AdANNS-DiskANN over their baseline counterparts. \n4) Baselines: The baselines comparison are limited. As AdANNS can be adapted to HNSW and Scann as well, it will be good to analyze the performance improvement there. \n5) Marginal gains: (Table 3 Appendix) It seems like AdANNS-IVF provides marginal accuracy gains over IVF with similar query times. This is good but not very convincing in favor of Matryoshka representations against single rigid representations.\n\nQUESTIONS:\n1)  How are the vector sizes $d_c$ and $d_s$ decided for a particular dataset? A grid search is not practical. Similarly how the parameter $k$ is set? Is there any rule?\n\nLIMITATIONS:\nThe authors have discussed the limitations in section 5.3. There are no potential negative societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}