Title: MUVERA: Multi-Vector Retrieval via Fixed Dimensional Encoding
Venue: NeurIPS 2024
Accepted: yes
Keywords: Retrieval

Late Interaction

ColBERT

Multi-Vector
Abstract: Neural embedding models have become a fundamental component of modern information retrieval (IR) pipelines. These models produce a single embedding $x \in \mathbb{R}^d$ per data-point, allowing for fast retrieval via highly optimized maximum inner product search (MIPS) algorithms. Recently, beginning with the landmark ColBERT paper, multi-vector models, which produce a set of embedding per data point, have achieved markedly superior performance for IR tasks. Unfortunately, using these models for IR is computationally expensive due to the increased complexity of multi-vector retrieval and scoring. 

In this paper, we introduce MUVERA (MUlti-VEctor Retrieval Algorithm), a retrieval mechanism which reduces multi-vector similarity search to single-vector similarity search. This enables the usage of off-the-shelf MIPS solvers for multi-vector retrieval. 
MUVERA asymmetrically generates Fixed Dimensional Encodings (FDEs) of queries and documents, which are vectors whose inner product approximates multi-vector similarity. We prove that FDEs give high-quality $\epsilon$-approximations, thus providing the first single-vector proxy for multi-vector similarity with theoretical guarantees. Empirically, we find that FDEs achieve the same recall as prior state-of-the-art heuristics while retrieving 2-5$\times$ fewer candidates. Compared to prior state of the art implementations, MUVERA achieves consistently good end-to-end recall and latency across a diverse set of the BEIR retrieval datasets, achieving an average of 10% improved recall with 90% lower latency.

MUVERA: Multi-Vector Retrieval via Fixed Dimensional Encodings

Abstract

Neural embedding models have become a fundamental component of modern information retrieval (IR) pipelines. These models produce a single embedding x ∈ ❘ d per data-point, allowing for fast retrieval via highly optimized maximum inner product search (MIPS) algorithms. Recently, beginning with the landmark ColBERT paper, multi-vector models , which produce a set of embedding per datapoint, have achieved markedly superior performance for IR tasks. Unfortunately, using these models for IR is computationally expensive due to the increased complexity of multi-vector retrieval and scoring.

In this paper, we introduce MUVERA ( Mu ltiVe ctor R etrieval A lgorithm), a retrieval mechanism which reduces multi-vector similarity search to single-vector similarity search. This enables the usage of off-the-shelf MIPS solvers for multivector retrieval. MUVERA asymmetrically generates Fixed Dimensional Encodings (FDEs) of queries and documents, which are vectors whose inner product approximates multi-vector similarity. We prove that FDEs give high-quality ε -approximations, thus providing the first single-vector proxy for multi-vector similarity with theoretical guarantees. Empirically, we find that FDEs achieve the same recall as prior state-of-the-art heuristics while retrieving 2-5 × fewer candidates. Compared to prior state of the art implementations, MUVERA achieves consistently good end-to-end recall and latency across a diverse set of the BEIR retrieval datasets, achieving an average of 10 % improved recall with 90% lower latency.

1 Introduction

Over the past decade, the use of neural embeddings for representing data has become a central tool for information retrieval (IR) [56], among many other tasks such as clustering and classification [39]. Recently, multi-vector (MV) representations, introduced by the late-interaction framework in ColBERT [29], have been shown to deliver significantly improved performance on popular IR benchmarks. ColBERT and its variants [17, 21, 32, 35, 42, 44, 49, 54] produce multiple embeddings per query or document by generating one embedding per token. The query-document similarity is then scored via the Chamfer Similarity (§1.1), also known as the MaxSim operation, between the two sets of vectors. These multi-vector representations have many advantages over single-vector (SV) representations, such as better interpretability [15, 50] and generalization [16, 36, 51, 55].

Despite these advantages, multi-vector retrieval is inherently more expensive than single-vector retrieval. Firstly, producing one embedding per token increases the number of embeddings in a dataset by orders of magnitude. Moreover, due to the non-linear Chamfer similarity scoring, there is a lack of optimized systems for multi-vector retrieval. Specifically, single-vector retrieval is

MUVERA

PLAID

Embeddings

Query

Top(K)

FDEs

Centroids

Embeddings

Query

Prune<

Centroid

Centroids

Residuals

Centroid

Approx

Embeddings

Query

QueryFDE

Reranking

Chamfer

Scores

MaxSim

MaxSim

Relevance

TopK. (np

TopK (ndecs)

MaxSim

TopK (k)

MIPSIndex

Top(K)

(DiskANN)

Documents

PIDs

PIDs

PIDs

Stage 1: Construct FDEs

and Query MIPS

Stage 2: Chamfer

Reranking

Stage 1: Initial

Candidate

Stage 2: Centroid

Interaction with

Stage 3: Centroid

Interaction w/out

Stage 4: Final

Ranking with

Generation

Pruning

Pruning

Decompression

generally accomplished via Maximum Inner Product Search (MIPS) algorithms, which have been highly-optimized over the past few decades [18]. However, SV MIPS alone cannot be used for MV retrieval. This is because the MV similarity is the sum of the SV similarities of each embedding in a query to the nearest embedding in a document. Thus, a document containing a token with high similarity to a single query token may not be very similar to the query overall. Thus, in an effort to close the gap between SV and MV retrieval, there has been considerable work in recent years to design custom MV retrieval algorithms with improved efficiency [12, 21, 42, 43].

The most prominent approach to MV retrieval is to employ a multi-stage pipeline beginning with single-vector MIPS. The basic version of this approach is as follows: in the initial stage, the most similar document tokens are found for each of the query tokens using SV MIPS. Then the corresponding documents containing these tokens are gathered together and rescored with the original Chamfer similarity. We refer to this method as the single-vector heuristic . ColBERTv2 [44] and its optimized retrieval engine PLAID [43] are based on this approach, with the addition of several intermediate stages of pruning. In particular, PLAID employs a complex four -stage retrieval and pruning process to gradually reduce the number of final candidates to be scored (Figure 1). Unfortunately, as described above, employing SV MIPS on individual query embeddings can fail to find the true MV nearest neighbors. Additionally, this process is expensive, since it requires querying a significantly larger MIPS index for every query embedding (larger because there are multiple embeddings per document). Finally, these multi-stage pipelines are complex and highly sensitive to parameter setting, as recently demonstrated in a reproducibility study [37], making them difficult to tune. To address these challenges and bridge the gap between single and multi-vector retrieval, in this paper we seek to design faster and simplified MV retrieval algorithms.

Contributions. We propose MUVERA: a multi-vector retrieval mechanism based on a light-weight and provably correct reduction to single-vector MIPS. MUVERA employs a fast, data-oblivious transformation from a set of vectors to a single vector, allowing for retrieval via highly-optimized MIPS solvers before a single stage of re-ranking. Specifically, MUVERA transforms query and document MV sets Q,P ⊂ ❘ d into single fixed-dimensional vectors glyph[vector] q, glyph[vector] p , called Fixed Dimensional Encodings (FDEs), such that the the dot product glyph[vector] q · glyph[vector] p approximates the multi-vector similarity between Q,P (§2). Empirically, we show that retrieving with respect to the FDE dot product significantly outperforms the single vector heuristic at recovering the Chamfer nearest neighbors (§3.1). For instance, on MS MARCO, our FDEs Recall @ N surpasses the Recall @ 2-5N achieved by the SV heuristic while scanning a similar total number of floats in the search.

We prove in (§2.1) that our FDEs have strong approximation guarantees; specifically, the FDE dot product gives an ε -approximation to the true MV similarity. This gives the first algorithm with provable guarantees for Chamfer similarity search with strictly faster than brute-force runtime (Theorem 2.2). Thus, MUVERA provides the first principled method for MV retrieval via a SV proxy.

We compare the end-to-end retrieval performance of MUVERA to PLAID on several of the BEIR IR datasets, including the well-studied MS MARCO dataset. We find MUVERA to be a robust and efficient retrieval mechanism; across the datasets we evaluated, MUVERA obtains an average of 10% higher recall, while requiring 90% lower latency on average compared with PLAID. Additionally, MUVERA crucially incorporates a vector compression technique called product quantization that enables us to compress the FDEs by 32 × (i.e., storing 10240 dimensional FDEs using 1280 bytes) while incurring negligible quality loss, resulting in a significantly smaller memory footprint.

1.1 Chamfer Similarity and the Multi-Vector Retrieval Problem

Given two sets of vectors Q,P ⊂ ❘ d , the Chamfer Similarity is given by

where 〈· , ·〉 is the standard vector inner product. Chamfer similarity is the default method of MV similarity used in the late-interaction architecture of ColBERT, which includes systems like ColBERTv2 [44], Baleen [28], Hindsight [41], DrDecr [34], and XTR [32], among many others. These models encode queries and documents as sets Q,P ⊂ ❘ d (respectively), where the query-document similarity is given by CHAMFER ( Q,P ) . We note that Chamfer Similarity (and its distance variant) itself has a long history of study in the computer vision (e.g., [4, 6, 14, 27, 45]) and graphics [33] communities, and had been previously used in the ML literature to compare sets of embeddings [3, 5, 30, 48]. In these works, Chamfer is also referred to as MaxSim or the relaxed earth mover distance ; we choose the terminology Chamfer due to its historical precedence [6].

In this paper, we study the problem of Nearest Neighbor Search (NNS) with respect to the Chamfer Similarity. Specifically, we are given a dataset D = { P 1 , . . . , P n } where each P i ⊂ ❘ d is a set of vectors. Given a query subset Q ⊂ ❘ d , the goal is to quickly recover the nearest neighbor P ∗ ∈ D , namely:

For the retrieval system to be scalable, this must be achieved in time significantly faster than bruteforce scoring each of the n similarities CHAMFER ( Q,P i ) within D .

1.2 Our Approach: Reducing Multi-Vector Search to Single-Vector MIPS

MUVERA is a streamlined procedure that directly reduces the Chamfer Similarity Search to MIPS. For a pre-specified target dimension d FDE , MUVERA produces randomized mappings F q : 2 ❘ d → ❘ d FDE (for queries) and F doc : 2 ❘ d → ❘ d FDE (for documents) such that, for all query and document multi-vector representations Q,P ⊂ ❘ d , we have:

We refer to the vectors F q ( Q ) , F doc ( P ) as Fixed Dimensional Encodings (FDEs). MUVERA first applies F doc to each document representation P ∈ D , and indexes the set { F doc ( P ) } P ∈ D into a MIPS solver. Given a query Q ⊂ ❘ d , MUVERA quickly computes F q ( Q ) and feeds it to the MIPS solver to recover the topk most similar document FDE's F doc ( P ) . Finally, we re-rank these candidates by the original Chamfer similarity. See Figure 1 for an overview. We remark that one important advantage of the FDEs is that the functions F q , F doc are data-oblivious , making them robust to distribution shifts, and easily usable in streaming settings.

1.3 Related Work on Multi-Vector Retrieval

The early multi-vector retrieval systems, such as ColBERT [29], all implement optimizations of the previously described SV heuristic, where the initial set of candidates is found by querying a MIPS index for every query token q ∈ Q . In ColBERTv2 [44], the document token embeddings are first clustered via k-means, and the first round of scoring uses cluster centroids instead of the original token. This technique was further optimized in PLAID [43] by employing a four-stage pipeline to progressively prune candidates before a final re-ranking (Figure 1).

An alternative approach with proposed in DESSERT [12], whose authors also pointed out the limitations of the SV heuristic, and proposed an algorithm based on Locality Sensitive Hashing (LSH) [20]. They prove that their algorithm recovers ε -approximate nearest neighbors in time ˜ O ( n | Q | T ) , where T is roughly the maximum number of document tokens p ∈ P i that are similar to any query token q ∈ Q , which can be as large as max i | P i | . Thus, in the worst case, their algorithm runs no faster than brute-force. Conversely, our algorithm recovers ε -approximate nearest neighbors and always runs in time ˜ O ( n | Q | ) . Experimentally, DESSERT is 2-5 × faster than PLAID, but attains worse recall (e.g. 2-2.5 % R @ 1000 on MS MARCO). Conversely, we match and sometimes strongly exceed PLAID's recall with up to 5.7 × lower latency. Additionally, DESSERT still employs an initial filtering stage based on k -means clustering of individual query token embeddings (in the manner of ColBERTv2), thus they do not truly avoid the aforementioned limitations of the SV heuristic.

2 Fixed Dimensional Encodings

We now describe our process for generating FDEs. Our transformation is reminiscent of the technique of probabilistic tree embeddings [1, 7, 10, 13], which can be used to transform a set of vectors into a single vector. For instance, they have been used to embed the Earth Mover's Distance into the glyph[lscript] 1 metric [1, 10, 22, 24], and to embed the weight of a Euclidean MST of a set of vectors into the Hamming metric [9, 22, 23]. However, since we are working with inner products, which are not metrics, instead of glyph[lscript] p distances, an alternative approach for our transformation will be needed.

The intuition behind our transformation is as follows. Hypothetically, for two MV representations Q,P ⊂ ❘ d , if we knew the optimal mapping π : Q → P in which to match them, then we could create vectors glyph[vector] q, glyph[vector] p by concatenating all the vectors in Q and their corresponding images in P together, so that 〈 glyph[vector] q, glyph[vector] p 〉 = ∑ q ∈ Q 〈 q, π ( q ) 〉 = CHAMFER ( Q,P ) . However, since we do not know π in advance, and since different query-document pairs have different optimal mappings, this simple concatenation clearly will not work. Instead, our goal is to find a randomized ordering over all the points in ❘ d so that, after clustering close points together, the dot product of any query-document pair Q,P ⊂ ❘ d concatenated into a single vector under this ordering will approximate the Chamfer similarity.

The first step is to partition the latent space ❘ d into B clusters so that vectors that are closer are more likely to land in the same cluster. Let ϕ : ❘ d → [ B ] be such a partition (for an integer N glyph[greaterorequalslant] 1 we use the notation [ N ] = { 1 , 2 , . . . , N } ); ϕ can be implemented via Locality Sensitive Hashing (LSH) [20], k -means, or other methods; we discuss choices for ϕ later in this section. After partitioning via ϕ , the hope is that for each q ∈ Q , the closest p ∈ P lands in the same cluster (i.e. ϕ ( q ) = ϕ ( p ) ). Hypothetically, if this were to occur, then:

If p is the only point in P that collides with q , then (1) can be realized as a dot product between two vectors glyph[vector] q, glyph[vector] p by creating one block of d coordinates in glyph[vector] q, glyph[vector] p for each cluster k ∈ [ B ] (call these blocks glyph[vector] q ( k ) , glyph[vector] p ( k ) ∈ ❘ d ), and setting glyph[vector] q ( k ) , glyph[vector] p ( k ) to be the sum of all q ∈ Q (resp. p ∈ P ) that land in the k -th cluster under ϕ . However, if multiple p ′ ∈ P collide with q , then 〈 glyph[vector] q, glyph[vector] p 〉 will differ from (1), since every p ′ with ϕ ( p ′ ) = ϕ ( q ) will contribute at least 〈 q, p ′ 〉 to 〈 glyph[vector] q, glyph[vector] p 〉 . To resolve this, we set glyph[vector] p ( k ) to be the centroid of the p ∈ P 's with ϕ ( p ) = ϕ ( q ) . Formally, for k = 1 , . . . , B , we define

Setting glyph[vector] q = ( glyph[vector] q (1) , . . . , glyph[vector] q ( B ) ) and glyph[vector] p = ( glyph[vector] p (1) , . . . , glyph[vector] p ( B ) ) , then we have

Note that the resulting dimension of the vectors glyph[vector] q, glyph[vector] p is dB . To reduce the dependency on d , we can apply a random linear projection ψ : ❘ d → ❘ d proj to each block glyph[vector] q ( k ) , glyph[vector] p ( k ) , where d proj < d . Specifically, we define ψ ( x ) = (1 / √ d proj ) Sx , where S ∈ ❘ d proj × d is a random matrix with uniformly distributed ± 1 entries. We can then define glyph[vector] q ( k ) , ψ = ψ ( glyph[vector] q ( k ) ) and glyph[vector] p ( k ) , ψ = ψ ( glyph[vector] p ( k ) ) , and define the FDE's with inner projection as glyph[vector] q ψ = ( glyph[vector] q (1) , ψ , . . . , glyph[vector] q ( B ) , ψ ) and glyph[vector] p ψ = ( glyph[vector] p (1) , ψ , . . . , glyph[vector] p ( B ) , ψ ) . When d = d proj , we simply define ψ to be the identity mapping, in which case glyph[vector] q ψ , glyph[vector] p ψ are identical to glyph[vector] q, glyph[vector] p . To increase accuracy of (3) in approximating (1), we repeat the above process R reps glyph[greaterorequalslant] 1 times independently, using different randomized partitions ϕ 1 , . . . , ϕ R reps and projections ψ 1 , . . . , ψ R reps . We denote the vectors resulting from i -th repetition by glyph[vector] q i, ψ , glyph[vector] p i, ψ . Finally, we concatenate these R reps vectors together, so that our final FDEs are defined as F q ( Q ) = ( glyph[vector] q 1 , ψ , . . . , glyph[vector] q R reps , ψ ) and F doc ( P ) = ( glyph[vector] p 1 , ψ , . . . , glyph[vector] p R reps , ψ ) . Observe that a complete FDE mapping is specified by the three parameters ( B,d proj , R reps ) , resulting in a final dimension of d FDE = B · d proj · R reps .

Choice of Space Partition. When choosing the partition function ϕ , the desired property is that points are more likely to collide (i.e. ϕ ( x ) = ϕ ( y ) ) the closer they are to each other. Such functions

F,(Q)

Fdoc(P)

B

X

0

C

B

0

0

DocEmbeddings

A

W

C

y+z

W

:=Query Embeddings

t

D

0

四

V

D

E

0

0

F

F

0

V

E

d

d

with this property exist, and are known as locality-sensitive hash functions (LSH) (see [20]). When the vectors are normalized, as they are for those produced by ColBERT-style models, SimHash [8] is the standard choice of LSH. Specifically, for any k sim glyph[greaterorequalslant] 1 , we sample random Gaussian vectors g 1 , . . . , g k sim ∈ ❘ d , and set ϕ ( x ) = ( 1 ( 〈 g 1 , x 〉 > 0) , . . . , 1 ( 〈 g k sim , x 〉 > 0)) , where 1 ( · ) ∈ { 0 , 1 } is the indicator function. Converting the bit-string to decimal, ϕ ( x ) gives a mapping from ❘ d to [ B ] where B = 2 k sim . In other words, SimHash partitions ❘ d by drawing k sim random half-spaces, and each of the 2 k sim clusters is formed by the k sim -wise intersection of each of these halfspaces or their complement. Another natural approach is to choose k CENTER glyph[greaterorequalslant] 1 centers from the collection of all token embeddings ∪ n i =1 P i , either randomly or via k -means, and set ϕ ( x ) ∈ [ k CENTER ] to be the index of the center nearest to x . We compare this method to SimHash in (§3.1).

glyph[negationslash]

Filling Empty Clusters. Akey source of error in the FDE's approximation is when the nearest vector p ∈ P to a given query embedding q ∈ Q maps to a different cluster, namely ϕ ( p ) = ϕ ( q ) = k . This can be made less likely by decreasing B , at the cost of making it more likely for other p ′ ∈ P to also map to the same cluster, moving the centroid glyph[vector] p ( k ) farther from p . If we increase B too much, it is possible that no p ∈ P collides with ϕ ( q ) . To avoid this trade-off, we directly ensure that if no p ∈ P maps to a cluster k , then instead of setting glyph[vector] p ( k ) = 0 we set glyph[vector] p ( k ) to the point p that is closest to cluster k . As a result, increasing B will result in a more accurate estimator, as this results in smaller clusters. Formally, for any cluster k with P ∩ ϕ -1 ( k ) = ∅ , if fill_empty_clusters is enabled, we set glyph[vector] p ( k ) = p where p ∈ P is the point for which ϕ ( p ) has the fewest number of disagreeing bits with k (both thought of as binary strings), with ties broken arbitrarily. We do not enable this for query FDEs, as doing so would result in a given q ∈ Q contributing to the dot product multiple times.

Final Projections. A natural approach to reducing the dimensionality is to apply a final projection ψ ′ : ❘ d FDE → ❘ d final (also implemented via multiplication by a random ± 1 matrix) to the FDE's, reducing the final dimensionality to any d final < d FDE . Experimentally, we find that final projections can provide small but non-trivial 1 -2% recall boosts for a fixed dimension (see §C.2).

2.1 Theoretical Guarantees for FDEs

We now state our theoretical guarantees for our FDE construction. For clarity, we state our results in terms of normalized Chamfer similarity NCHAMFER ( Q,P ) = 1 | Q | CHAMFER ( Q,P ) . This ensures NCHAMFER ( Q,P ) ∈ [ -1 , 1] whenever the vectors in Q,P are normalized. Note that this factor of 1 / | Q | does not affect the relative scoring of documents for a fixed query. In what follows, we assume that all token embeddings are normalized (i.e. ‖ q ‖ 2 = ‖ p ‖ 2 = 1 for all q ∈ Q,p ∈ P ) . Note that ColBERT-style late interaction MV models indeed produce normalized token embeddings. We will always use the fill_empty_clusters method for document FDEs, but never for queries.

Our main result is that FDEs give ε -additive approximations of the Chamfer similarity. The proof uses the properties of LSH (SimHash) to show that for each query point q ∈ Q , the point q gets mapped to a cluster ϕ ( q ) that only contains points p ∈ P that are close to q (within ε of the closest point to q ); the fact that at least one point collides with q uses the fill_empty_partitions method.

Theorem 2.1 (FDE Approximation) . Fix any ε, δ > 0 , and sets Q,P ⊂ ❘ d of unit vectors, and let m = | Q | + | P | . Then setting k sim = O ( log( mδ -1 ) ε ) , d proj = O ( 1 ε 2 log( m εδ ) ) , R reps = 1 , so that

d FDE = ( m/δ ) O (1 /ε ) , then in expectation and with probability at least 1 -δ we have

Finally, we show that our FDE's give an ε -approximate solution to Chamfer similarity search, using FDE dimension that depends only logarithmically on the size of the dataset n . Using the fact that our query FDEs are sparse (Lemma A.1), one can run exact MIPS over the FDEs in time ˜ O ( | Q | · n ) , improving on the brute-force runtime of O ( | Q | max i | P i | n ) for Chamfer similarity search.

Theorem 2.2. Fix any ε > 0 , query Q , and dataset P = { P 1 , . . . , P n } , where Q ⊂ ❘ d and each P i ⊂ ❘ d is a set of unit vectors. Let m = | Q | + max i ∈ [ n ] | P i | . Let k sim = O ( log m ε ) , d proj = O ( 1 ε 2 log( m/ε )) and R reps = O ( 1 ε 2 log n ) so that d FDE = m O (1 /ε ) · log n . Then if i ∗ = arg max i ∈ [ n ] 〈 F q ( Q ) , F doc ( P i ) 〉 , with high probability (i.e. 1 -1 / poly( n ) ) we have:

Given the query Q , the document P ∗ can be recovered in time O ( | Q | max { d, n } 1 ε 4 log( m ε ) log n ) .

3 Evaluation

In this section, we evaluate our FDEs as a method for MV retrieval. First, we evaluate the FDEs themselves (offline) as a proxy for Chamfer similarity (§3.1). In (§3.2), we discuss the implementation of MUVERA, as well as several optimizations made in the search. Then we evaluate the latency of MUVERA compared to PLAID, and study the effects of the aforementioned optimizations.

Datasets. Our evaluation includes results from six of the well-studied BEIR [46] information retrieval datasets: MS MARCO [40] (CC BY-SA 4.0), HotpotQA (CC BY-SA 4.0) [53], NQ (Apache-2.0) [31], Quora (Apache-2.0) [46], SciDocs (CC BY 4.0) [11], and ArguAna (Apache-2.0) [47]. These datasets were selected for varying corpus size (8K-8.8M) and average number of document tokens (18-165); see (§B) for further dataset statistics. Following [43], we use the development set for our experiments on MS MARCO, and use the test set on the other datasets.

MV Model, MV Embedding Sizes, and FDE Dimensionality. We compute our FDEs on the MVembeddings produced by the ColBERTv2 model [44] (MIT License), which have a dimension of d = 128 and a fixed number | Q | = 32 of embeddings per query. The number of document embeddings is variable, ranging from an average of 18 . 3 on Quora to 165 on Scidocs. This results in 2,300-21,000 floats per document on average (e.g. 10,087 for MS MARCO). Thus, when constructing our FDEs we consider a comparable range of dimensions d FDE between 1,000-20,000. Furthermore, using product quantization, we show in (§3.2) that the FDEs can be significantly compressed by 32 × with minimal quality loss, further increasing the practicality of FDEs.

3.1 Offline Evaluation of FDE Quality

We evaluate the quality of our FDEs as a proxy for the Chamfer similarity, without any re-ranking and using exact (offline) search. We first demonstrate that FDE recall quality improves dependably as the dimension d FDE increases, making our method relatively easy to tune. We then show that FDEs are a more effective method of retrieval than the SV heuristic. Specifically, the FDE method achieves Recall @ N exceeding the Recall @ 2-4N of the SV heuristic, while in principle scanning a similar number of floats in the search. This suggests that the success of the SV heuristic is largely due to the significant effort put towards optimizing it (as supported by [37]), and similar effort for FDEs may result in even bigger efficiency gains. Additional plots can be found in (§C). All recall curves use a single FDE instantiation, since in (§C.1) we show the variance of FDE recall is negligible.

FDE Quality vs. Dimensionality. We study how the retrieval quality of FDE's improves as a function of the dimension d FDE . We perform a grid search over FDE parameters R reps ∈ { 1 , 5 , 10 , 15 , 20 } , k sim ∈ { 2 , 3 , 4 , 5 , 6 } , d proj ∈ { 8 , 16 , 32 , 64 } , and compute recall on MS MARCO (Figure 3). We find that Pareto optimal parameters are generally achieved by larger R reps , with k sim , d proj playing a lesser role in improving quality. Specifically, ( R reps , k sim , d proj ) ∈ { (20 , 3 , 8) , (20 , 4 , 8)(20 , 5 , 8) , (20 , 5 , 16) } were all Pareto optimal for their respective dimensions (namely R reps · 2 k sim · d proj ). While there are small variations depending on the parameter choice, the FDE quality is tightly linked to dimensionality; increase in dimensionality will generally result in

1.0

1.0

0.9

0.8

0.9

0.9

0.7

0.8

0.8

Recall

0.7

Recall

0.5

0.7

0.3

0.4

FDE SimHash Partitioning R@100

FDE Nearest Center Partitioning R@100

Exact Chamfer R@10

Exact ChamferR@25

0.5

0.6

FDE SimHash Partitioning R@1000

ExactChamferR@100

FDE Nearest Center Partitioning R@1000

ExactChamferR@250

0.6

X

FDE SimHash Partitioning R@10000

FDE Nearest Center Partitioning R@10000

ExactChamferR@250

0.2

0

X

5000

10000

15000

20000

0.4

0

5000

10000

15000

20000

0.5

0

5000

10000

15000

20000

Dimension

Dimension

Dimension

MSMarco

HotpotQA

Quora

NQ

1.0

1.0

1.00

1.0

0.9

0.9

0.9

0.95

Recall

0.8

0.8

0.8

FDE dimension

0.7

0.7

0.7

1280

0.90

2560

0.6

0.6

0.6

5120

10240

0.5

0.5

0.85

0.5

5

100

200

300

400

500

5

100

200

300

400

500

5

100

200

300

400

500

5

100

200

300

400

500

quality gains. We also evaluate using k -means as a method of partitioning instead of SimHash. Specifically, we cluster the document embeddings with k -means and set ϕ ( x ) to be the index of the nearest centroid to x . We perform a grid search over the same parameters (but with k ∈ { 4 , 8 , 16 , 32 , 64 } to match B = 2 k sim ). We find that k -means partitioning offers no quality gains on the Pareto Frontier over SimHash, and is often worse. Moreover, FDE construction with k -means is no longer data oblivious. Thus, SimHash is chosen as the preferred method for partitioning for the remainder of our experiments.

In Figure 4, we evaluate the FDE retrieval quality with respect to the Chamfer similarity (instead of labelled ground truth data). We compute 1 Recall@ N , which is the fraction of queries for which the Chamfer 1 -nearest neighbor is among the topN most similar in FDE dot product. We choose FDE parameters which are Pareto optimal for the dimension from the above grid search. We find that FDE's with fewer dimensions that the original MV representations achieve significantly good recall across multiple BEIR retrieval datasets. For instance, on MS MARCO (where d · m avg ≈ 10 K ) we achieve 95% recall while retrieving only 75 candidates using d FDE = 5120 .

Single Vector Heuristic vs. FDE retrieval. We compare the quality of FDEs as a proxy for retrieval against the previously described SV heuristic, which is the method underpinning PLAID. Recall that in this method, for each of the i = 1 , . . . , 32 query vectors q i we compute the k nearest neighbors p 1 ,i , . . . , p k,i from the set ∪ i P i of all document token embeddings. To compute Recall @ N , we create an ordered list glyph[lscript] 1 , 1 , . . . , glyph[lscript] 1 , 32 , glyph[lscript] 2 , 1 , . . . , where glyph[lscript] i,j is the document ID containing p i,j , consisting of the 1 -nearest neighbors of the queries, then the 2 -nearest neighbors, and so on. When reranking, one first removes duplicate document IDs from this list. Since duplicates cannot be detected while performing the initial 32 SV MIPS queries, the SV heuristic needs to over-retrieve to reach a desired number of unique candidates. Thus, we note that the true recall curve of implementations of the SV heuristic (e.g. PLAID) is somewhere between the case of no deduplication and full deduplication; we compare to both in Figure 5.

To compare the cost of the SV heuristic to running MIPS over the FDEs, we consider the total number of floats scanned by both using a brute force search. The FDE method must scan n · d FDE floats to compute the k -nearest neighbors. For the SV heuristic, one runs 32 brute force scans over n · m avg vectors in 128 dimensions, where m avg is the average number of embeddings per document (see §B for values of m avg ). For MS MARCO, where m avg = 78 . 8 , the SV heuristic searches through 32 · 128 · 78 . 8 · n floats. This allows for an FDE dimension of d FDE = 322,764 to have comparable cost! We can extend this comparison to fast approximate search - suppose that approximate MIPS

1.0

MSMarco

0.9

HotpotQA

1.00

Quora

1.0

NQ

0.9

0.8

0.95

0.9

Recall

0.8

0.7

0.7

0.90

0.8

0.7

FDE 20480

SV

SV w/ Dedup

0.6

0.6

0.85

0.6

FDE 4096

FDE 10240

FDE 2048

0.5

0.5

0.80

0.5

100

250

500

750

1000

100

250

500

750

1000

100

250

500

750

1000

100

250

500

750

1000

over n vectors can be accomplished in sublinear n ε time, for some ε ∈ (0 , 1) . Then even in the unrealistic case of ε = 0 , we can still afford an FDE dimension of d FDE = 32 · 128 = 4096 .

The results can be found in Figure 5. We build FDEs once for each dimension, using R reps = 40 , k sim = 6 , d proj = d = 128 , and then applying a final projection to reduce to the target dimension (see C.2 for experiments on the impact of final projections). On MS MARCO, even the 4096 -dimensional FDEs match the recall of the (deduplicated) SV heuristic while retrieving 1.75-3.75 × fewer candidates (our Recall @ N matches the Recall @ 1.75-3.75 N of the SV heuristic), and 10.5-15 × fewer than to the non-deduplicated SV heuristic. For our 10240 -dimension FDEs, these numbers are 2.6-5 × and 20-22.5 × fewer, respectively. For instance, we achieve 80% recall with 60 candidates when d FDE = 10240 and 80 candidates when d FDE = 4096 , but the SV heuristic requires 300 and 1200 candidates (for dedup and non-dedup respectively). See Table 1 for further comparisons.

Variance. Note that although the FDE generation is a randomized process, we show in (§C.1) that the variance of the FDE Recall is essentially negligible; for instance, the standard deviation Recall @ 1000 is at most 0 . 08 -0 . 16% for FDEs with 2 -10 k dimensions.

3.2 Online Implementation and End-to-End Evaluation

We implemented MUVERA, an FDE generation and end-to-end retrieval engine in C++. We discussed FDE generation and various optimizations and their tradeoffs in (§3.1). Next, we discuss how we perform retrieval over the FDEs, and additional optimizations.

Single-Vector MIPS Retrieval using DiskANN. Our single-vector retrieval engine uses a scalable implementation [38] of DiskANN [25] (MIT License), a state-of-the-art graph-based ANNS algorithm. We build DiskANN indices by using the uncompressed document FDEs with a maximum degree of 200 and a build beam-width of 600. Our retrieval works by querying the DiskANN index using beam search with beam-width W , and subsequently reranking the retrieved candidates with Chamfer similarity. The only tuning knob in our system is W ; increasing W increases the number of candidates retrieved by MUVERA, which improves the recall.

Product Quantization (PQ). To further improve the memory usage of MUVERA, we use a textbook vector compression technique called product quantization (PQ) with asymmetric querying [19, 26] on the FDEs. We refer to product quantization with C centers per group of G dimensions as PQ -C -G . For example, PQ -256 -8 , which we find to provide the best tradeoff between quality and compression in our experiments, compresses every consecutive set of 8 dimensions to one of 256 centers. Thus PQ -256 -8 provides 32 × compression over storing each dimension using a single float, since each block of 8 floats is represented by a single byte. See (§C.4) for further experiments and details on PQ.

Experimental Setup. We run our online experiments on an Intel Sapphire Rapids machine on Google Cloud (c3-standard-176). The machine supports up to 176 hyper-threads. We run latency experiments using a single thread, and run our QPS experiments on all 176 threads.

Ball Carving. To improve re-ranking speed, we reduce the number of query embeddings by clustering them via a ball carving method and replacing the embeddings in each cluster with their sum. This speeds up reranking without decreasing recall. Specifically, we group the queries Q into clusters C 1 , . . . , C k , setting c i = ∑ q ∈ C i q and Q C = { c 1 , . . . , c k } . Then, after retrieving a set of candidate documents with the FDEs, instead of rescoring via CHAMFER ( Q,P ) for each candidate P , we rescore via CHAMFER ( Q C , P ) , which runs in time O ( | Q C | · | P | ) , offering speed-ups when the number of clusters is small. Instead of fixing k , we perform greedy ball-carving to allow k to adapt to Q . Specifically, given a threshold τ , we select an arbitrary point q ∈ Q , cluster it with all other points q ′ ∈ Q with 〈 q, q ′ 〉 glyph[greaterorequalslant] τ , remove the clustered points and repeat until all points are clustered.

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

80

0

.

85

0

.

90

0

.

95

k

NQ

k

100.0

1000.0

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

75

0

.

80

0

.

85

0

.

90

0

.

95

k

MSMarco

k

100.0

1000.0

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

92

0

.

94

0

.

96

0

.

98

1

.

00

k

Quora

k

100.0

1000.0

0

.

95

0

.

96

0

.

97

0

.

98

0

.

99

Recall

@100

10

1

10

2

10

3

10

4

QPS

Quora

Uncompressed

PQ-256-2

PQ-256-4

PQ-256-5

PQ-256-8

PQ-256-16

0

.

775

0

.

800

0

.

825

0

.

850

0

.

875

0

.

900

0

.

925

Recall

@100

10

1

10

2

10

3

QPS

NQ

Uncompressed

PQ-256-2

PQ-256-4

PQ-256-5

PQ-256-8

PQ-256-16

0

.

78

0

.

80

0

.

82

0

.

84

0

.

86

0

.

88

0

.

90

Recall

@100

10

1

10

2

10

3

QPS

MS MARCO

PQ-256-2

PQ-256-4

PQ-256-5

PQ-256-8

PQ-256-16

In Figure 6, we show the the trade-off between end-to-end Recall @ k of MUVERA and the ball carving threshold used. Notice that for both k = 100 and k = 1000 , the recall curves flatten after a threshold of τ = 0 . 6 , and for all datasets they are essentially flat after τ glyph[greaterorequalslant] 0 . 7 . Thus, for such thresholds we incur essentially no quality loss by performing ball carving. Based on these empirical results, we choose the value of τ = 0 . 7 in our end-to-end experiments.

In (§C.3), we show the impact on end-to-end QPS of ball carving on the MS MARCO dataset. For sequential re-ranking, we find that ball carving at a τ = 0 . 7 threshold provides a 25% QPS improvement, and when re-ranking is being done in parallel (over all cores simultaneously) it yields a 20 % QPS improvement. Moreover, with a threshold of τ = 0 . 7 , there were an average of 5 . 9 clusters created per query on MS MARCO. This reduces the number of query embeddings by 5.4 × , down from the initial fixed setting of | Q | = 32 . This finding shows that pre-clustering the queries before re-ranking gives non-trivial runtime improvements with negligible quality loss. It also suggests that a fixed setting of | Q | = 32 query embeddings used by existing approaches is likely excessive for MV similarity quality, and that fewer queries could achieve a similar performance.

QPS vs. Recall. A useful metric for retrieval is the number of queries per second (QPS) a system can serve at a given recall; evaluating the QPS of a system tries to fully utilize the system resources (e.g., the bandwidth of multiple memory channels and caches), and deployments where machines serve many queries simultaneously. Figure 7 shows the QPS vs. Recall@100 for MUVERA on a subset of the BEIR datasets, using different PQ schemes over the FDEs. We show results for additional datasets, as well as Recall@1000, in the Appendix. Using PQ -256 -8 not only reduces the space usage of the FDEs by 32 × , but also improves the QPS at the same query beamwidth by up to 20 × , while incurring a minimal loss in end-to-end recall. Our method has a relatively small dependence on the dataset size, which is consistent with prior studies on graph-based ANNS data structures, since the number of distance comparisons made during beam search grows roughly logarithmically with increasing dataset size [25, 38]. We tried to include QPS numbers for PLAID [43], but unfortunately their implementation does not support running multiple queries in parallel, and is optimized for measuring latency.

Latency and Recall Results vs. PLAID [43] We evaluated MUVERA and PLAID [43] on the 6 datasets from the BEIR benchmark described earlier in (§3); Figure 8 shows that MUVERA achieves essentially equivalent Recall@ k as PLAID (within 0.4%) on MS MARCO, while obtaining up to 1.56 × higher recall on other datasets (e.g. HotpotQA). We ran PLAID using the recommended settings for their system, which reproduced their recall results for MS MARCO. Compared with PLAID, on

NQ-100

NQ-1000

HotpotQA-100

HotpotQA-1000

MS MARCO-100

MS MARCO-1000

Quora-100

Quora-1000

SCIDOCS-100

ArguAna-100

ArguAna-1000

0.00

0.20

0.40

0.60

Latency in seconds

0.152

0.292

0.14

0.25

0.31

0.444

0.0437

0.0814

0.0796

0.285

0.0236

0.0822

0.187

0.335

0.273

0.4

0.221

0.318

0.108

0.12

0.14

0.286

0.136

0.253

The value on top of each bar is the latency in seconds (lower is better).

Muvera

PLAID

NQ-100

NQ-1000

HotpotQA-100

HotpotQA-1000

MS MARCO-100

MS MARCO-1000

Quora-100

Quora-1000

SCIDOCS-100

ArguAna-100

ArguAna-1000

0

.

0

0

.

5

1

.

0

Recall@k

0.904

0.951

0.668

0.751

0.902

0.971

0.997

0.998

0.347

0.572

0.89

0.966

0.831

0.914

0.462

0.481

0.903

0.975

0.984

0.998

0.344

0.571

0.887

0.961

The value on top of each bar is the Recall@k (higher is better).

Muvera

PLAID

average over all 6 datasets and k ∈ { 100 , 1000 } , MUVERA achieves 10% higher Recall@ k (up to 56% higher), and 90% lower latency (up to 5.7 × lower).

Importantly, MUVERA has consistently high recall and low latency across all of the datasets that we measure, and our method does not require costly parameter tuning to achieve this-all of our results use the same 10240-dimensional FDEs that are compressed using PQ with PQ -256 -8 ; the only tuning in our system was to pick the first query beam-width over the k that we rerank to that obtained recall matching that of PLAID. As Figure 8 shows, in cases like NQ and HotpotQA, MUVERA obtains much higher recall while obtaining lower latency. Given these results, we believe a distinguishing feature of MUVERA compared to prior multi-vector retrieval systems is that it achieves consistently high recall and low latency across a wide variety of datasets with minimal tuning effort.

4 Conclusion

In this paper, we presented MUVERA: a principled and practical MV retrieval algorithm which reduces MV similarity to SV similarity by constructing Fixed Dimensional Encoding (FDEs) of a MVrepresentation. We prove that FDE dot products give high-quality approximations to Chamfer similarity (§2.1). Experimentally, we show that FDEs are a much more effective proxy for MV similarity, since they require retrieving 2-4 × fewer candidates to achieve the same recall as the SV Heuristic (§3.1). We complement these results with an end-to-end evaluation of MUVERA, showing that it achieves an average of 10% improved recall with 90% lower latency compared with PLAID. Moreover, despite the extensive optimizations made by PLAID to the SV Heuristic, we still achieve significantly better latency on 5 out of 6 BEIR datasets we consider (§3). Given their retrieval efficiency compared to the SV heuristic, we believe that there are still significant gains to be obtained by optimizing the FDE method, and leave further exploration of this to future work.

Broader Impacts and Limitations: While retrieval is an important component of LLMs, which themselves have broader societal impacts, these impacts are unlikely to result from our retrieval algorithm. Our contribution simply improves the efficiency of retrieval, without enabling any fundamentally new capabilities. As for limitations, while we outperformed PLAID, sometimes significantly, on 5 out of the 6 datasets we studied, we did not outperform PLAID on MS MARCO, possibly due to their system having been carefully tuned for MS MARCO given its prevalence. Additionally, we did not study the effect that the average number of embeddings m avg per document has on retrieval quality of FDEs; this is an interesting direction for future work.

References

Alexandr Andoni, Piotr Indyk, and Robert Krauthgamer. Earth mover distance over highdimensional spaces. In Proceedings of the 19th ACM-SIAM Symposium on Discrete Algorithms (SODA '2008) , pages 343-352, 2008.

Rosa I Arriaga and Santosh Vempala. An algorithmic theory of learning: Robust concepts and random projection. Machine learning , 63:161-182, 2006.

Kubilay Atasu and Thomas Mittelholzer. Linear-complexity data-parallel earth mover's distance approximations. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 364-373. PMLR, 09-15 Jun 2019.

Vassilis Athitsos and Stan Sclaroff. Estimating 3d hand pose from a cluttered image. In 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings. , volume 2, pages II-432. IEEE, 2003.

Ainesh Bakshi, Piotr Indyk, Rajesh Jayaram, Sandeep Silwal, and Erik Waingarten. Near-linear time algorithm for the chamfer distance. Advances in Neural Information Processing Systems , 36, 2024.

Harry G Barrow, Jay M Tenenbaum, Robert C Bolles, and Helen C Wolf. Parametric correspondence and chamfer matching: Two new techniques for image matching. In Proceedings: Image Understanding Workshop , pages 21-27. Science Applications, Inc, 1977.

Yair Bartal. Probabilistic approximation of metric spaces and its algorithmic applications. In Proceedings of the 37th Annual IEEE Symposium on Foundations of Computer Science (FOCS '1996) , 1996.

Moses S Charikar. Similarity estimation techniques from rounding algorithms. In Proceedings of the thiry-fourth annual ACM symposium on Theory of computing , pages 380-388, 2002.

Xi Chen, Vincent Cohen-Addad, Rajesh Jayaram, Amit Levi, and Erik Waingarten. Streaming euclidean mst to a constant factor. In Proceedings of the 55th Annual ACM Symposium on Theory of Computing , STOC 2023, page 156-169, New York, NY, USA, 2023. Association for Computing Machinery.

Xi Chen, Rajesh Jayaram, Amit Levi, and Erik Waingarten. New streaming algorithms for high dimensional emd and mst. In Proceedings of the 54th Annual ACM SIGACT Symposium on Theory of Computing , pages 222-233, 2022.

Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S Weld. Specter: Document-level representation learning using citation-informed transformers. arXiv preprint arXiv:2004.07180 , 2020.

Joshua Engels, Benjamin Coleman, Vihan Lakshman, and Anshumali Shrivastava. Dessert: An efficient algorithm for vector set search with vector set queries. Advances in Neural Information Processing Systems , 36, 2024.

Jittat Fakcharoenphol, Satish Rao, and Kunal Talwar. A tight bound on approximating arbitrary metrics by tree metrics. Journal of Computer and System Sciences , 69(3):485-497, 2004.

Haoqiang Fan, Hao Su, and Leonidas J Guibas. A point set generation network for 3d object reconstruction from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 605-613, 2017.

Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. A white box analysis of colbert. In Advances in Information Retrieval: 43rd European Conference on IR Research, ECIR 2021, Virtual Event, March 28-April 1, 2021, Proceedings, Part II 43 , pages 257-263. Springer, 2021.

Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. Match your words! a study of lexical matching in neural information retrieval. In European Conference on Information Retrieval , pages 120-127. Springer, 2022.

Luyu Gao, Zhuyun Dai, and Jamie Callan. Coil: Revisit exact lexical match in information retrieval with contextualized inverted list. arXiv preprint arXiv:2104.07186 , 2021.

Ruiqi Guo, Sanjiv Kumar, Krzysztof Choromanski, and David Simcha. Quantization based fast inner product search. In Artificial intelligence and statistics , pages 482-490. PMLR, 2016.

Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, and Sanjiv Kumar. Accelerating large-scale inference with anisotropic vector quantization. In International Conference on Machine Learning , pages 3887-3896. PMLR, 2020.

Sariel Har-Peled, Piotr Indyk, and Rajeev Motwani. Approximate nearest neighbor: Towards removing the curse of dimensionality. Theory of Computing , 8(1):321-350, 2012.

Sebastian Hofstätter, Omar Khattab, Sophia Althammer, Mete Sertkan, and Allan Hanbury. Introducing neural bag of whole-words with colberter: Contextualized late interactions using enhanced reduction. In Proceedings of the 31st ACM International Conference on Information &Knowledge Management , pages 737-747, 2022.

Piotr Indyk. Algorithms for dynamic geometric problems over data streams. In Proceedings of the 36th ACM Symposium on the Theory of Computing (STOC '2004) , pages 373-380, 2004.

Rajesh Jayaram, Vahab Mirrokni, Shyam Narayanan, and Peilin Zhong. Massively parallel algorithms for high-dimensional euclidean minimum spanning tree. In Proceedings of the 2024 Annual ACM-SIAM Symposium on Discrete Algorithms (SODA) , pages 3960-3996. SIAM, 2024.

Rajesh Jayaram, Erik Waingarten, and Tian Zhang. Data-dependent lsh for the earth mover's distance. In Proceedings of the 56th Annual ACM Symposium on Theory of Computing , 2024.

Suhas Jayaram Subramanya, Fnu Devvrit, Harsha Vardhan Simhadri, Ravishankar Krishnawamy, and Rohan Kadekodi. Diskann: Fast accurate billion-point nearest neighbor search on a single node. Advances in Neural Information Processing Systems , 32, 2019.

Herve Jegou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. IEEE transactions on pattern analysis and machine intelligence , 33(1):117-128, 2010.

Li Jiang, Shaoshuai Shi, Xiaojuan Qi, and Jiaya Jia. Gal: Geometric adversarial loss for single-view 3d-object reconstruction. In Proceedings of the European conference on computer vision (ECCV) , pages 802-816, 2018.

Omar Khattab, Christopher Potts, and Matei Zaharia. Baleen: Robust multi-hop reasoning at scale via condensed retrieval. Advances in Neural Information Processing Systems , 34:2767027682, 2021.

Omar Khattab and Matei Zaharia. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval , pages 39-48, 2020.

Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. From word embeddings to document distances. In International conference on machine learning , pages 957-966. PMLR, 2015.

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, et al. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics , 2019.

Jinhyuk Lee, Zhuyun Dai, Sai Meher Karthik Duddu, Tao Lei, Iftekhar Naim, Ming-Wei Chang, and Vincent Zhao. Rethinking the role of token retrieval in multi-vector retrieval. Advances in Neural Information Processing Systems , 36, 2024.

Chun-Liang Li, Tomas Simon, Jason Saragih, Barnabás Póczos, and Yaser Sheikh. Lbs autoencoder: Self-supervised fitting of articulated meshes to point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11967-11976, 2019.

Yulong Li, Martin Franz, Md Arafat Sultan, Bhavani Iyer, Young-Suk Lee, and Avirup Sil. Learning cross-lingual ir from an english retriever. arXiv preprint arXiv:2112.08185 , 2021.

Weizhe Lin, Jinghong Chen, Jingbiao Mei, Alexandru Coca, and Bill Byrne. Fine-grained lateinteraction multi-modal retrieval for retrieval augmented visual question answering. Advances in Neural Information Processing Systems , 36, 2024.

Simon Lupart, Thibault Formal, and Stéphane Clinchant. Ms-shift: An analysis of ms marco distribution shifts on neural retrieval. In European Conference on Information Retrieval , pages 636-652. Springer, 2023.

Sean MacAvaney and Nicola Tonellotto. A reproducibility study of plaid. arXiv preprint arXiv:2404.14989 , 2024.

Magdalen Dobson Manohar, Zheqi Shen, Guy Blelloch, Laxman Dhulipala, Yan Gu, Harsha Vardhan Simhadri, and Yihan Sun. Parlayann: Scalable and deterministic parallel graphbased approximate nearest neighbor search algorithms. In Proceedings of the 29th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming , pages 270-285, 2024.

Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. Mteb: Massive text embedding benchmark. arXiv preprint arXiv:2210.07316 , 2022.

Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. Ms marco: A human-generated machine reading comprehension dataset. 2016.

Ashwin Paranjape, Omar Khattab, Christopher Potts, Matei Zaharia, and Christopher D Manning. Hindsight: Posterior-guided training of retrievers for improved open-ended generation. arXiv preprint arXiv:2110.07752 , 2021.

Yujie Qian, Jinhyuk Lee, Sai Meher Karthik Duddu, Zhuyun Dai, Siddhartha Brahma, Iftekhar Naim, Tao Lei, and Vincent Y Zhao. Multi-vector retrieval as sparse alignment. arXiv preprint arXiv:2211.01267 , 2022.

Keshav Santhanam, Omar Khattab, Christopher Potts, and Matei Zaharia. Plaid: an efficient engine for late interaction retrieval. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management , pages 1747-1756, 2022.

Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. Colbertv2: Effective and efficient retrieval via lightweight late interaction. arXiv preprint arXiv:2112.01488 , 2021.

Erik B Sudderth, Michael I Mandel, William T Freeman, and Alan S Willsky. Visual hand tracking using nonparametric belief propagation. In 2004 Conference on Computer Vision and Pattern Recognition Workshop , pages 189-189. IEEE, 2004.

Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models. arXiv preprint arXiv:2104.08663 , 2021.

Henning Wachsmuth, Shahbaz Syed, and Benno Stein. Retrieval of the best counterargument without prior topic knowledge. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 241-251, 2018.

Ziyu Wan, Dongdong Chen, Yan Li, Xingguang Yan, Junge Zhang, Yizhou Yu, and Jing Liao. Transductive zero-shot learning with visual structure constraint. Advances in neural information processing systems , 32, 2019.

Xiao Wang, Craig Macdonald, Nicola Tonellotto, and Iadh Ounis. Pseudo-relevance feedback for multiple representation dense retrieval. In Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval , pages 297-306, 2021.

Xiao Wang, Craig Macdonald, Nicola Tonellotto, and Iadh Ounis. Reproducibility, replicability, and insights into dense multi-representation retrieval models: from colbert to col. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 2552-2561, 2023.

Orion Weller, Dawn Lawrie, and Benjamin Van Durme. Nevir: Negation in neural information retrieval. arXiv preprint arXiv:2305.07614 , 2023.

David P Woodruff et al. Sketching as a tool for numerical linear algebra. Foundations and Trends R © in Theoretical Computer Science , 10(1-2):1-157, 2014.

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600 , 2018.

Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-image pre-training. arXiv preprint arXiv:2111.07783 , 2021.

Jingtao Zhan, Xiaohui Xie, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma. Evaluating interpolation and extrapolation performance of neural retrieval models. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management , pages 2486-2496, 2022.

Ye Zhang, Md Mustafizur Rahman, Alex Braylan, Brandon Dang, Heng-Lu Chang, Henna Kim, Quinten McNamara, Aaron Angert, Edward Banner, Vivek Khetan, et al. Neural information retrieval: A literature review. arXiv preprint arXiv:1611.06792 , 2016.

A Missing Proofs from Section 2.1

In this section, we provide the missing proofs in Section 2.1. For convenience, we also reproduce theorem statements as they appear in the main text before the proofs. We begin by analyzing the runtime to compute query and document FDEs, as well as the sparsity of the queries.

Lemma A.1. For any FDE parameters k sim , d proj , R reps glyph[greaterorequalslant] and sets Q,P ⊂ ❘ d , we can compute F q ( Q ) in time T q := O ( R reps | Q | d ( d proj + k sim )) , and F q ( P ) in time O ( T q + R reps | P | 2 k sim k sim ) . Moreover, F q ( Q ) has at most O ( | Q | d proj R reps ) non-zero entries.

Proof. We first consider the queries. To generate the queries, we must first project each of the | Q | queries via the inner random linear productions ψ i : ❘ d → ❘ d proj , which requires O ( | Q | dd proj R reps ) time to perform the matrix-query products for all repetitions. Next, we must compute ϕ i ( q ) for each q ∈ Q and repetition i ∈ [ R reps ] , Each such value can be compute in d · k sim time to multiply the q ∈ ❘ d by the k sim Gaussian vectors. Thus the total running time for this step is O ( R reps | Q | dk sim ) . Finally, summing the relevant values into the FDE once ϕ i ( q ) , ψ i ( q ) are computed can be done in O ( | Q | d proj ) time. For sparsity, note that only the coordinate blocks in the FDE corresponding to clusters k in a repetition i with at least one q ∈ | Q | with ϕ i ( q ) = k are non-zero, and there can be at most O ( R reps | Q | ) of these blocks, each of which has O ( d proj ) coordinates.

The document runtime is similar, except with the additional complexity required to carry out the fill_empty_clusters option. For each repetition, the runtime required to find the closest p ∈ P to a give cluster k is O ( | P | · k sim ) , since we need to run over all | p | values of ϕ ( p ) and check how many bits disagree with k . Thus, the total runtime is O ( R reps | P | Bk sim ) = O ( R reps | P | 2 k sim k sim ) .

In what follows, we will need the following standard fact that random projections approximately preserve dot products. The proof is relatively standard, and can be found in [2], or see results on approximate matrix product [52] for more general bounds.

Fact A.2 ([2]) . Fix ε, δ > 0 . For any d glyph[greaterorequalslant] 1 and x, y ∈ ❘ d , let S ∈ ❘ t × d by a matrix of independent entries distributed uniformly over { 1 , -1 } , where t = O (1 /ε 2 · log δ -1 ) . Then we have E [ 〈 Sx,Sy 〉 ] = 〈 x, y 〉 , and moreover with probability at least 1 -δ we have

To anaylze the approximations of our FDEs, we begin by proving an upper bound on the value of the FDE dot product. In fact, we prove a stronger result: we show that our FDEs have the desirable property of being one-sided estimators - namely, they never overestimate the true Chamfer similarity. This is summarized in the following Lemma.

LemmaA.3 (One-Sided Error Estimator) . Fix any sets Q,P ⊂ ❘ d of unit vectors with | Q | + | P | = m . Then if d = d proj , we always have

Furthermore, for any δ > 0 , if we set d proj = O ( 1 ε 2 log( m/δ )) , then we have 1 | Q | 〈 F q ( Q ) , F doc ( P ) 〉 glyph[lessorequalslant] NCHAMFER ( Q,P ) + ε in expectation and with probability at least 1 -δ .

Proof. First claim simply follows from the fact that the average of a subset of a set of numbers can't be bigger than the maximum number in that set. More formally, we have:

Which completes the first part of the lemma. For the second part, to analyze the case of d proj < d , when inner random projections are used, by applying Fact A.2, firstly we have E [ 〈 ψ ( p ) , ψ ( q )] = 〈 q, p 〉 for any q ∈ Q,p ∈ P, , and secondly, after a union bound we over | P | · | Q | glyph[lessorequalslant] m 2 pairs, we have 〈 q, p 〉 = 〈 ψ ( p ) , ψ ( q ) 〉 ± ε simultaneously for all q ∈ Q,p ∈ P, with probability 1 -δ , for any constant C > 1 . The second part of the Lemma then follows similarly as above.

We are now ready to give the proof of our main FDE approximation theorem.

Theorem 2.1 (FDE Approximation). Fix any ε, δ > 0 , and sets Q,P ⊂ ❘ d of unit vectors, and let m = | Q | + | P | . Then setting k sim = O ( log( mδ -1 ) ε ) , d proj = O ( 1 ε 2 log( m εδ ) ) , R reps = 1 , so that d FDE = ( m/δ ) O (1 /ε ) , we have

in expectation, and with probability at least 1 -δ .

Proof of Theorem 2.1. The upper bound follows from Lemma A.3, so it will suffice to prove the lower bound. We first prove the result in the case when there are no random projections ψ , and remove this assumption at the end of the proof. Note that, by construction, F q is a linear mapping so that F q ( Q ) = ∑ q ∈ Q F ( q ) , thus

So it will suffice to prove that

for all q ∈ Q , since then, by a union bound 5 will hold for all over all q ∈ Q with probability at least 1 -εδ , in which case we will have

which will complete the theorem.

In what follows, for any x, y ∈ ❘ d let θ ( x, y ) ∈ [0 , π ] be the angle between x, y . Now fix any q ∈ Q , and let p ∗ = arg max p ∈ P 〈 q, p 〉 , and let θ ∗ = θ ( q, p ∗ ) . By construction, there always exists some set of points S ⊂ P such that

Moreover, the RHS of the above equation is always bounded by 1 in magnitude, since it is an average of dot products of normalized vectors q, p ∈ S d -1 . In particular, there are two cases. In case (A) S is the set of points p with ϕ ( p ) = ϕ ( q ) , and in case (B) S is the single point arg min p ∈ P ‖ ϕ ( p ) -ϕ ( q ) ‖ 0 , where ‖ x -y ‖ 0 denotes the hamming distance between any two bitstrings x, y ∈ { 0 , 1 } k sim , and we are interpreting ϕ ( p ) , ϕ ( q ) ∈ { 0 , 1 } k sim as such bit-strings. Also let g 1 , . . . , g k sim ∈ ❘ d be the random Gaussian vectors that were drawn to define the partition function ϕ . To analyze S , we first prove the following:

Claim A.4. For any q ∈ Q and p ∈ P , we have

glyph[negationslash]

Proof. Fix any such p , and for i ∈ [ k sim ] let Z i be an indicator random variable that indicates the event that 1 ( 〈 g i , p 〉 > 0) = 1 ( 〈 g i , q 〉 > 0) . First then note that ‖ ϕ ( p ) -ϕ ( q ) ‖ 0 = ∑ k sim i =1 Z i . Now by rotational invariance of Gaussians, for a Gaussian vector g ∈ ❘ d we have Pr [ 1 ( 〈 g, x 〉 > 0) = 1 ( 〈 g, y 〉 > 0)] = θ ( x,y ) π for any two vectors x, y ∈ ❘ d . It follows that Z i is a Bernoulli random variable with E [ Z i ] = θ ( x,y ) π . By a simple application of Hoeffding's inequality, we have glyph[negationslash]

where we took k sim glyph[greaterorequalslant] 1 / 2 · log( m 2 ) /ε

εδ , which completes the proof.

We now condition on the event in Claim A.4 occurring for all p ∈ P , which holds with probability at least 1 -| P | · ( εδ m 2 ) > 1 -( εδ m ) by a union bound. Call this event E , and condition on it in what follows.

Now first suppose that we are in case (B) , and the set S of points which map to the cluster ϕ ( q ) is given by S = { p ′ } where p ′ = arg min p ∈ P ‖ ϕ ( p ) -ϕ ( q ) ‖ 0 . Firstly, if p ′ = p ∗ , then we are done as 〈 F q ( q ) , F doc ( P ) 〉 = 〈 q, p ∗ 〉 , and 5 follows. Otherwise, by Claim A.4 we must have had | θ ( q, p ′ ) -θ ( q, p ∗ ) | glyph[lessorequalslant] π · √ ε . Using that the Taylor expansion of cosine is cos( x ) = 1 -x 2 / 2+ O ( x 4 ) , we have

which proves the desired statement 5 after a constant factor rescaling of ε .

Next, suppose we are in case (A) where S = { p ∈ P ′ | ϕ ( p ) = ϕ ( q ) } is non-empty. In this case, S consists of the set of points p with ‖ ϕ ( p ) -ϕ ( q ) ‖ 0 = 0 . From this, it follows again by Claim A.4

Thus

that θ ( q, p ) glyph[lessorequalslant] √ επ for any p ∈ S . Thus, by the same reasoning as above, we have

which again proves the desired statement 5 in case (A) , thereby completing the full proof in the case where there are no random projections.

To analyze the expectation, note that using the fact that |〈 F q ( q ) , F doc ( P ) 〉| glyph[lessorequalslant] 1 deterministically, the small O ( εδ ) probability of failure (i.e. the event that E does not hold) above can introduce at most a O ( εδ ) glyph[lessorequalslant] ε additive error into the expectation, which is acceptable after a constant factor rescaling of ε .

Finally, to incorporate projections, by standard consequences of the Johnson Lindenstrauss Lemma (Fact A.2) setting d proj = O ( 1 ε 2 log m ε ) and projecting via a random Gaussian or ± 1 matrix from ψ : ❘ d → ❘ d proj , for any set S ⊂ P we have that E [ 〈 ψ ( q ) , ψ ( 1 | S | ∑ p ∈ S p ) 〉 ] = 〈 q, 1 | S | ∑ p ∈ S p 〉 , and moreover that 〈 q, 1 | S | ∑ p ∈ S p 〉 = 〈 ψ ( q ) , ψ ( 1 | S | ∑ p ∈ S p ) 〉‖ q ‖ 2 ‖ 1 | S | ∑ p ∈ S p ‖ 2 ± ε for all q ∈ Q,p ∈ P with probability at least 1 -εδ . Note that ‖ q ‖ 2 = 1 , and by triangle inequality ‖ 1 | S | ∑ p ∈ S p ‖ 2 glyph[lessorequalslant] 1 | S | ∑ p ∈ S ‖ p ‖ 2 = 1 . Thus, letting F q ( Q ) , F doc ( P ) be the FDE values without the inner projection ψ and F ψ q ( Q ) , F ψ doc ( P ) be the FDE values with the inner projection ψ , conditioned on the above it follows that

Finally, to analyze the expectation, note that since

as before conditioning on this small probability event changes the expectation of 5 by at most a ε additive factor, which completes the proof of the Theorem after a constant factor rescaling of ε .

Equipped with Theorem 2.1, as well as the sparsity bounds from Lemma A.1, we are now prepared to prove our main theorem on approximate nearest neighbor search under the Chamfer Similarity.

Theorem 2.2 . Fix any ε > 0 , query Q , and dataset P = { P 1 , . . . , P n } , where Q ⊂ ❘ d and each P i ⊂ ❘ d is a set of unit vectors. Let m = | Q | + max i ∈ [ n ] | P i | . Then setting k sim = O ( log m ε ) , d proj = O ( 1 ε 2 log( m/ε )) and R reps = O ( 1 ε 2 log n ) so that d FDE = m O (1 /ε ) · log n . Then setting i ∗ = arg max i ∈ [ n ] 〈 F q ( Q ) , F doc ( P i ) 〉 , with high probability (i.e. 1 -1 / poly( n ) ) we have:

Given the query Q , the document P ∗ can be recovered in time O ( | Q | max { d, n } 1 ε 4 log( m ε ) log n ) .

Proof of Theorem 2.2. First note, for a single repetition, for any subset P j ∈ D , by Theorem 2.1 we have

Moreover, as demonsrated in the proof of Theorem 2.1, setting δ = 1 / 10 , we have

It follows that for each repetition i ∈ [ R reps ] , letting F q ( Q ) i , F doc ( P j ) i be the coordinates in the final FDE vectors corresponding to that repetition, the random variable X i = 1 | Q | 〈 F i q ( Q ) , F i doc ( P j ) 〉 is bounded in [ -1 , 1] and has expectation NCHAMFER ( Q,P j ) ± ε . By Chernoff bounds, averaging over R reps = O ( 1 ε 2 log( n )) repetitions, we have

with probability 1 -1 /n C for any arbitrarily large constant C > 1 . Note also that ∑ R reps i =1 1 R reps | Q | 〈 F i q ( Q ) , F i doc ( P j ) 〉 = 1 R reps | Q | 〈 F q ( Q ) , F doc ( P j ) 〉 , where F q ( Q ) , F doc ( P j ) are the final FDEs. We can then condition on (11) holding for all documents j ∈ [ n ] , which holds with probability with probability 1 -1 /n C -1 by a union bound. Conditioned on this, we have

which completes the proof of the approximation after a constant factor scaling of ε . The runtime bound follows from the runtime required to compute F q ( Q ) , which is O ( | Q | R reps d ( d proj + k sim )) = O ( | Q | log n ε 2 d ( 1 ε 2 log( m/ε ) + 1 ε log m ) , plus the runtime required to brute force search for the nearest dot product. Specifically, note that each of the n FDE dot products can be computed in time proportional to the sparsity of F q ( Q ) , which is at most O ( | Q | d proj R reps ) = O ( | Q | 1 ε 4 log( m/ε ) log n ) . Adding these two bounds together yields the desired runtime.

B Additional Dataset Information

In Table 9 we provide further dataset-specific information on the BEIR retrieval datasets used in this paper. Specifically, we state the sizes of the query and corpuses used, as well as the average number of embeddings produced by the ColBERTv2 model per document. Specifically, we consider the six BEIR retrieval datasets MS MARCO [40], NQ [31], HotpotQA [53], ArguAna [47], SciDocs [11], and Quora [46], Note that the MV corpus (after generating MV embeddings on all documents in a corpus) will have a total of # Corpus × (Avg # Embeddings per Doc) token embeddings. For even further details, see the BEIR paper [46].

C Additional Experiments and Plots

In this Section, we provide additional plots to support the experimental results from Section 3. We provide plots for all six of the datasets and additional ranges of the x -axis for our experiments in Section (§3.1), as well as additional experimental results, such as an evaluation of variance, and of the quality of final projections in the FDEs.

FDE vs. SV Heuristic Experiments. In Figures 10 and 11, we show further datasets and an expanded recall range for the comparison of the SV Heuristic to retrieval via FDEs. We find that our 4k+ dimensional FDE methods outperform even the deduplciated SV heuristic (whose cost is somewhat unrealistic, since the SV heuristic must over-retrieve to handle duplicates) on most datasets, especially in lower recall regimes. In Table 1, we compare how many candidates must be retrieved by the SV heuristic, both with and without the deduplication step, as well as by our FDE methods, in order to exceed a given recall threshold.

Retrieval quality with respect to exact Chamfer. In Figure 12, we display the full plots for FDE Recall with respects to recovering the 1 -nearest neighbor under Chamfer Similarity for all six BEIR datasets that we consider, including the two omitted from the main text (namely, SciDocs and ArguAna).

C.1 Variance of FDEs.

Since the FDE generation is a randomized process, one natural concern is whether there is large variance in the recall quality across different random seeds. Fortunately, we show that this is not the case, and the variance of the recall of FDE is essentially negligible, and can be easily accounted for via minor extra retrieval. To evaluate this, we chose four sets of FDE parameters ( R reps , k sim , d proj ) which were Pareto optimal for their respective dimensionalities, generated 10 independent copies of the query and document FDEs for the entire MS MARCO dataset, and computed the average recall @100 and 1000 and standard deviation of these recalls. The results are shown in Table 2, where for all of the experiments the standard deviation was between 0 . 08 -0 . 3% of a recall point, compared to the 80 -95% range of recall values. Note that Recall @1000 had roughly twice as small standard deviation as Recall @ 100.

MSMarco

HotpotQA

1.0

0.9

1.00

0.9

0.8

0.95

Recall

0.8

0.7

0.90

0.7

0.6

0.85

0.6

SV

0.5

1000 2000300040005000

0.5

1000 2000 3000 4000 5000

0.80

100020003000 40005000

SVw/Dedup

FDE 20480

NQ

ArguAna

FDE 10240

1.0

0.8

1.0

FDE 4096

FDE 2048

0.7

0.9

0.9

0.6

Recall

0.8

0.8

0.5

0.7

0.4

0.7

0.6

0.3

0.6

0.2

0.5

10002000300040005000

10002000300040005000

10002000300040005000

MSMarco

HotpotQA

Quora

1.0

0.9

1.00

0.9

0.8

0.95

0.8

0.90

Recall

0.7

0.7

0.85

0.6

0.6

0.80

0.5

0.4

0.5

0.75

SV

0.3

0.4

0.70

SVw/Dedup

100

200

300

400

500

100

200

300

400

500

100

200

300

400

500

FDE 20480

NQ

ArguAna

FDE 10240

1.0

0.5

1.0

FDE 4096

0.9

FDE 2048

0.9

0.4

0.8

Recall

0.8

0.3

0.7

0.7

0.2

0.6

0.5

0.6

0.1

0.4

0.5

0.0

0.3

100

200300

400

500

100

200300

400

500

100

200300

400

MSMarco

HotpotQA

1.0

1.0

1.0

0.9

0.9

0.9

Recall

0.8

0.8

0.8

0.7

0.7

0.7

0.6

0.6

0.6

0.5

0.5

0.5

FDE dimension

5

100

200

300

400

500

5

100

200

300

400

500

5

100

200

300

400

500

1280

NQ

ArguAna

2560

1.00

1.0

1.00

5120

10240

0.9

0.99

0.95

Recall

0.8

0.98

0.7

0.97

0.90

0.6

0.96

0.85

0.5

0.95

5

100

200

300

400

500

5

100

200

300

400

500

5

100

200

300

400

500

C.2 Comparison to Final Projections.

We now show the effect of employing final projections to reduce the target dimensionality of the FDE's. For all experiments, the final projection ψ ′ is implemented in the same way as inner projections are: namely, via multiplication by a random ± 1 matrix. We choose four target dimensions, d FDE ∈ { 2460 , 5120 , 10240 , 20480 } , and choose the Pareto optimal parameters ( R reps , k sim , d proj ) from the grid search without final projections in Section 3.1, which are (20 , 4 , 8) , (20 , 5 , 8) , (20 , 5 , 16) , (20 , 5 , 32) . We then build a large dimensional FDE with the parameters ( R reps , k sim , d proj ) = (40 , 6 , 128) . Here, since d = d proj , we do not use any inner projections when constructing the FDE. We then use a single random final projection to reduce the dimensionality of this FDE from R reps · 2 k sim · d proj = 327680 down to each of the above target dimensions d FDE . The results are show in Tables 3 and 4. Notice that incorporating final projections can have a non-trivial impact on recall, especially for Recall @ 100, where it can increase by around 3% . In particular, FDEs with the final projections are often better than FDEs with twice the dimensionality without final projections. The one exception is the 2460 -dimensional FDE, where the Recall @ 100 only improved by 1 . 1% , and the Recall @ 1000 was actually lower bound 0 . 3% .

C.3 Ball Carving

Continuing our discussion from Section 3.2, we show that ball-carving at this threshold of 0 . 7 gives non-trivial efficiency gains. Specifically, in Figure 13, we plot the per-core queries-per-second of re-ranking (i.e. computing CHAMFER ( Q C , P ) ) against varying ball carving thresholds for the MS MARCO dataset. Please see the discussion in Section 3.2 for analysis of the figure.

C.4 Product Quantization

PQ Details We implemented our product quantizers using a simple 'textbook' k -means based quantizer. Recall that AH -C -G means that each consecutive group of G dimensions is represented by C centers. We train the quantizer by: (1) taking for each group of dimensions the coordinates

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

15000

20000

25000

30000

35000

40000

45000

50000

55000

Per-Core Chamfer Computations Per Second

Setting

Sequential

Parallel

0

.

88

0

.

89

0

.

90

0

.

91

Recall

@100

10

3

QPS

ArguAna

Uncompressed

PQ-256-2

PQ-256-4

PQ-256-5

PQ-256-8

PQ-256-16

0

.

31

0

.

32

0

.

33

0

.

34

0

.

35

Recall

@100

10

2

10

3

QPS

SCIDOCS

Uncompressed

PQ-256-2

PQ-256-4

PQ-256-5

PQ-256-8

PQ-256-16

0

.

95

0

.

96

0

.

97

0

.

98

0

.

99

Recall

@100

10

1

10

2

10

3

10

4

QPS

Quora

Uncompressed

PQ-256-2

PQ-256-4

PQ-256-5

PQ-256-8

PQ-256-16

0

.

775

0

.

800

0

.

825

0

.

850

0

.

875

0

.

900

0

.

925

Recall

@100

10

1

10

2

10

3

QPS

NQ

Uncompressed

PQ-256-2

PQ-256-4

PQ-256-5

PQ-256-8

PQ-256-16

0

.

45

0

.

50

0

.

55

0

.

60

0

.

65

0

.

70

0

.

75

Recall

@100

10

1

10

2

10

3

QPS

HotpotQA

Uncompressed

PQ-256-2

PQ-256-4

PQ-256-5

PQ-256-8

PQ-256-16

0

.

78

0

.

80

0

.

82

0

.

84

0

.

86

0

.

88

0

.

90

Recall

@100

10

1

10

2

10

3

QPS

MS MARCO

PQ-256-2

PQ-256-4

PQ-256-5

PQ-256-8

PQ-256-16

of a sample of at most 100,000 vectors from the dataset, and (2) running k -means on this sample using k = C = 256 centers until convergence. Given a vector x ∈ ❘ d , we can split x into d/G blocks of coordinates x (1) , . . . , x ( d/G ) ∈ ❘ G each of size G . The block x ( i ) can be compressed by representing x ( i ) by the index of the centroid from the i -th group that is nearest to x ( i ) . Since there are 256 centroids per group, each block x ( i ) can then be represented by a single byte.

Results In Figures 14 and 15 we show the full set of results for our QPS experiments from Section 3.2 on all of the BEIR datasets that we evaluated in this paper. We include results for both Recall@ 100 (Figure 14) and Recall@ 1000 (Figure 15).

We find that PQ -256 -8 is consistently the best performing PQ codec across all of the datasets that we tested. Not using PQ at all results in significantly worse results (worse by at least 5 × compared to using PQ) at the same beam width for the beam; however, the recall loss due to using PQ -256 -8 is minimal, and usually only a fraction of a percent. Since our retrieval engine works by over-retrieving with respect to the FDEs and then reranking using Chamfer similarity, the loss due to approximating the FDEs using PQ can be handled by simply over-retrieving slightly more candidates.

We also observe that the difference between different PQ codecs is much more pronounced in the lower-recall regime when searching for the top 1000 candidates for a query. For example, most of the plots in Figure 15 show significant stratification in the QPS achieved in lower recall regimes,

0

.

90

0

.

92

0

.

94

0

.

96

0

.

98

Recall

10

3

10

4

QPS

ArguAna

Uncompressed

PQ-256-2

PQ-256-4

PQ-256-5

PQ-256-8

PQ-256-16

0

.

35

0

.

40

0

.

45

0

.

50

0

.

55

Recall

10

2

10

3

10

4

QPS

SCIDOCS

Uncompressed

PQ-256-2

PQ-256-4

PQ-256-5

PQ-256-8

PQ-256-16

0

.

95

0

.

96

0

.

97

0

.

98

0

.

99

1

.

00

Recall

10

2

10

3

10

4

QPS

Quora

Uncompressed

PQ-256-2

PQ-256-4

PQ-256-5

PQ-256-8

PQ-256-16

0

.

80

0

.

85

0

.

90

0

.

95

Recall

10

1

10

2

10

3

10

4

QPS

NQ

Uncompressed

PQ-256-2

PQ-256-4

PQ-256-5

PQ-256-8

PQ-256-16

0

.

5

0

.

6

0

.

7

0

.

8

Recall

10

1

10

2

10

3

10

4

QPS

HotpotQA

Uncompressed

PQ-256-2

PQ 256-4

PQ-256-5

PQ-256-8

PQ-256-16

0

.

80

0

.

85

0

.

90

0

.

95

Recall

10

1

10

2

10

3

10

4

QPS

MS MARCO

PQ-256-2

PQ-256-4

PQ-256-5

PQ-256-8

PQ-256-16

with PQ -256 -16 (the most compressed and memory-efficient format) usually outperforming all others; however, for achieving higher recall, PQ -256 -16 actually does much worse than slightly less compressed formats like PQ -256 -8 and PQ -256 -4 .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: All claims made in the abstract and introduction are thoroughly discussed and evaluated in the remaining sections of the paper. The theoretical claims are justified in Section 2.1, and the experimental claims are justified in Section 3.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: A limitations section is provided at the end of the paper in Section 4, along with the broader impacts. We discuss there the primary limitations of the paper. The computational efficiency of the algorithms are discussed in the latency experiments Section 3.2 of the paper, as well as how the MIPS index scales with dataset size.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All theoretical results are formally stated in the main paper, along with all relevant assumptions in Section 2.1. All definitions are formally given in the main paper, with full proofs deferred to the appendix in Section A, with ideas of the proof given in the main paper in Section 2.1.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The algorithm for generating FDEs is thoroughly and exactly described in Section 2, and is easy to reproduce from the description. As mentioned in Section 3.2, we will publish a standalone open-source shared-memory implementation of the FDE generation step upon publication. Further note that the DiskANN library used for retrieving via MIPS is publicly available, and we explicitly state the parameters we ran it with.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case

of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: Our end-to-end retrieval engine is implemented in C++ in a proprietary codebase, preventing us from directly releasing it. As described in Section 3.2, we plan to publish a standalone open-source implementation of the FDE generation step upon publication, along with the product quantization code (which is a textbook method) and the ball-carving code. The only other component of the algorithm is DiskANN, which is publicly available and properly cited.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: No models were trained for this paper, however details about all datasets and how we evaluated our algorithms on them were precisely defined. All parameters used in our algorithms were explicitly stated for every experiment in their corresponding section.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We run detailed variance experiments in the supplementary in Appendix C.1, and describe the main conclusions of these results in Section 3.1, demonstrating that our FDE recall is extremely stable and varies very little depending on the random seed.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: For all our online (latency) experiments, we state the exact hardware and compute resources we used in Section 3.2. The results of offline experiments (Section 3.1) do not depend on hardware implementation, as they are measuring fixed recall properties of an algorithm (without any runtime benchmarking), and therefore can be reproduced on any hardware and infastructure.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have read the code of ethics and verified that research conducted in the paper fully conforms to the code.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the broader impacts in a standalone section at the end of the paper in Section 4. In essence, since our work is solely about improving the efficiency of multi-vector retrieval, it is unlikely to have any direct path to negative applications, and we discuss this fact in the Broader Impacts section.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out

that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper proposes a new algorithm that purely improves the efficiency of multi-vector retrieval; we do not release any data or new models. Therefore our paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The authors and sources of all six BEIR datasets that were used in this paper are cited, as well as the ColBERTv2 model that was used to generate the multi-vector embeddings, and the DiskANN algorithm we used for single vector retrieval. For each of these aforementioned assets, the license of the asset was explicitly stated in the text: we stated the license for the BEIR datasets in Section 3, and the license for DiskANN in Section 3.2. The terms of use of all these licenses were properly respected in this paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: No new assets were introduced in this paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA] .

Justification: No crowdsourcing nor research with human subjects with used in this paper.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA] .

Justification: No crowdsourcing nor research with human subjects with used in this paper.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies how to optimize multi-vector retrieval under the MaxSim similarity function (commonly used in NLP literatures like ColBERT). This is done by mapping queries and item embeddings asymmetrically to fixed dimensional embeddings (FDEs), such that in the mapped embedding space, the inner product similarity is a good approximation of the original MaxSim similarity. This approach then enables classical MIPS techniques to be applied to optimize for MaxSim similarity search. The authors compare FDEs with PLAID / SV heuristics, and found it to outperform those optimized similarity search baselines.\n\nSTRENGTHS:\n**S1**: I really like the MIPS approximation construction for MaxSim in Section 2. While the techniques used are not that novel (e.g., Section 2 heavily reminds me of the classical AND- and OR- constructions on top of LSH etc.), being able to reduce MaxSim to MIPS is likely of significant interest to many researchers in NLP and vector search community.\n\n**S2**: Reducing MaxSim to MIPS enables many classical optimizations for MIPS to be applied incl PQ, RP (line 177-180) etc, likely significantly improving the usability and applicability of ColBERT-style approaches. \n\n**S3**: MaxSim similarity (and related setting under late-interaction settings) is an important topic in NLP, as inner products have been proven to be suboptimal for similarity search, especially when we are interested in Recall@K for small Ks (e.g., 10s-50s).\n\nWEAKNESSES:\n**W1**: With FDE, the MaxSim approach (used in ColBERT and other work)'s effectiveness is degraded dramatically, to the point that it underperforms dense retrieval baselines.  E.g., Figure 3 (LHS) suggest that Recall@100 of FDE is quite bad vs Exact MaxSim in ColBERT (up to ~.84, vs ~.92). The main issue here is that, gains of non-MIPS similarity approaches (like MaxSim and more recently DSI/NCI etc.) are primarily on smaller @Ks (e.g., ColBERT v2's quality gains are primarily demonstrated on Metric@10 in their original paper). The current paper only reported Recall@K={100, 1K, 10K}, which are not aligned with intended use cases for ColBERT (as it didn't provide that much gains over dense baselines for these Ks).\n- Even at 100, the 1-.84/.92 = 0.087 degradation in recall will likely make MaxSim+FDE underperform classical dense retrieval baselines. E.g., on MS MACRO, DPR achieves 82.2 R@50 whereas ColBERT achieves 82.9 R@50 per [1] (and gap @100 will be smaller). So adjusting for FDE's quality degradation, we get 82.2 for dual encoder baselines but just 82.9 * .84/.92 = 75.69 for MaxSim, and people will ask - why not use classical dense encoders like DPRs to begin with?\n- Additionally, to achieve the ~.84 recall, FDE encodings already used 20K floats vs 10,087 in MS MACRO multi-vector baselines per Fig. 3...\n\n**W2**: Random projection techniques (incl. their multi-hash variants, or $R_{reps}$ in this paper), and production quantization techniques used in FDEs are commonly used in MIPS systems, and generally universally reduce memory access costs and improve throughput/latency, etc. of nearest neighbor methods (after hparam tuning). Have we evaluated applying random projection and PQ techniques directly to baselines like ColBERTv2, PLAID, etc.? How much of the throughput gains presented in the paper are due to RPs/PQs?\n\n**W3**: Writing could be improved.\n- It would be good to add a table of notations, at least to appendix to help with references. Even though I work on vector search (incl multi-vector search and variants), I ended up spending quite some time figuring out what the authors meant by various notations in Section 2. \n- Line 141-142: \"To resolve this, we set ~p(k) to be the centroid of the p ∈ P’s with ϕ(p) = ϕ(q)\". - This sentence is confusing as the FDE mapping for query (p) should not be item (q) aware. Please consider rewriting it to reflect the actual mapping done in Equation (3). \n- Section 3: consider removing codebase/dataset licenses (CC BY-SA MIT etc.) as most readers may not find them useful.\n- Line 311 - \"We run latency experiments using a single thread, and run our QPS experiments on all 176 threads.\"  Could you explain why latency experiments are run with a single thread? As using additional threads should further improve latency.\n- Typos: Line 132-133 are more are more likely to land in -> are more likely to land in\n\t\n\n**References**:\n- [1] Ren et al. PAIR: Leveraging Passage-Centric Similarity Relation for Improving Dense Passage Retrieval. ACL 2021\n\nQUESTIONS:\n**Q1**: Would it be possible to report effectiveness of FDE + MaxSim vs corresponding baselines (incl dense retrieval and ColBERTv2 etc.), esp. for common Ks (10, 50, 100)?\n\n**Q2**: Can we equally apply techniques like PQ and random projection to baselines (again ideally dense retrieval with a comparable embedding dimensionality to normalize I/O costs), and evaluate how much throughput gains are due to PQs/RPs vs the particular construction of FDEs proposed in this paper? \n\n**Q3**: FDE's approximation quality seems to heavily rely on how the B clusters are defined. If B is large: a) the overall dimensionality is large, requiring significantly more memory accesses, and b) Max(P, Q) may not map to anything.  OTOH if B is small, many false positives collisions may occur, resulting in (potentially significant) overestimation of the MaxSim similarity function. Is there a reliable way of grid searching B (other than in 2^k as done in Sec 3.1)? Could we discuss this somewhere in the paper?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nEfficient vector retrieval to maximise inner product similarity is well studied, and this paper explores the issue of multi-vector retrieval to support late interaction models like Colbert. The core idea is to use SimHash to generate clusters of multiple representation of documents, represent each document's vectors in a cluster using a centroid, and use that with the centroid of query vectors that land in the same cluster. Authors rely on the SimHash's approximation theory to show that this approach approximates the maxSim measure used by Colbert. Experimental results indicate Muvera outperforms PLAID in some of the datasets from BEIR in terms of query latency and Recall@k.\n\nSTRENGTHS:\ns1. Simple yet effective idea that seems to work on a variety of datasets. \ns2. Well written, and experiments are well conducted.\n\nWEAKNESSES:\nw1. the theoretical analysis of the presented approach is rather limited. SimHash theory focuses on EMD minimisation while the focus here is on Chamfer similarity maximisation. So, it would be good detail the theory further. \n\nw2. despite multiple stages listed for PLAID, the performance of MUVERA is not particularly strong -- in terms of latency, PLAID outperforms MUVERA on largest dataset (MS-MARCO). It leads one to question if the observed latency gains are primarily due to small(er) dataset sizes? Similarly in terms of recall@k, MUVERA outperforms PLAID only in a couple of datasets. \n\nw3. there is no mention of how far from ideal (i.e, ColbertV2 performance) is the proposed model?\n\nQUESTIONS:\nq1. I would have liked to see the connection with the EMD approximation theory of SimHash with the Chamfer distance approximation presented here. In general, it seems to me --authors can clarify if I am mistaken-- the key innovation is the use of centroids in each bucket, instead of the standard O(n^2) similarity computation done even with SimHash in applications such as shingling. It would also be useful to show the sensitivity to the value of B = number of partitions. \n\nq2. Improvements over PLAID are not significant in my opinion -- in terms of both latency and recall @ k. There are only few datasets (NQ-{100,1000}, HotPotQA-{100, 1000} the recall is better, and latency is worse in ms-marco (a commonly used Colbert benchmark). Further, it would be also worthwhile comparing with the baseline of ColbertV2 (without using any of these efficient indexing techniques).\n\nLIMITATIONS:\nGiven that the work is aimed at improving the performance of an existing retrieval framework, I do not expect to see any specific negative societal impact from the work. Authors also have given similar remarks.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-03"}

comments: SUMMARY:
The paper proposes a method of speeding up text retrieval approximating ColBERT multi-vector ranker with very high dimensional single vector using projections. 

Authors demonstrate that the proposed approach is better compared to a colbert-based single vector heuristic and is comparable to PLAID in terms of accuracy and speed, but is easier to tune.

STRENGTHS:
* The paper is relatively easy to follow (though using  many acronyms like SV, MV, FDE makes it harder)

* The approach is more efficient at retrieval compared to the SV heuristic baseline

* Authors show that the method does not require much parameter tuning to achieve results comparable to PLAID.

WEAKNESSES:
1) The heuristic approach is reminiscent of [Morozov, Stanislav, and Artem Babenko https://arxiv.org/pdf/1908.06887 ] where a more general problem is stated with a more general solution (using the ranker as the metric during the graph search). This approach seems extremely relevant, but not discussed/compared in the paper. 

2) Using a single vector embedding search to approximate a more complex interaction sounds like distillation of a ranker to a retrieval, which is a well-known baseline (e.g. googling gives https://arxiv.org/pdf/2210.11708 or https://arxiv.org/pdf/2403.20327 ). As both ranker and retrieval are trained, it is not clear why not directly train the retrieval model to directly optimize the ranking objective. There might scenarios where distilling is not feasible for some reason, but for a research paper I think it is important to show how much of the performance is lost.

QUESTIONS:
I'd like the authors to address both points of weaknesses.

LIMITATIONS:
The approach is limited to multi-vector ranking as the function.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMulti-vector representation can greatly help retrieval systems work efficiently and accurately, transforming these sets of multiple vectors into a single vector representation that can still encapsulate the information from the multiple vectors, allowing efficient search using traditional single-vector search techniques while preserving the benefits of multi-vector search for rich document representation.\n\nSTRENGTHS:\nNovelty and generalizability: by introducing Fixed Dimensional Encodings (FDEs) in MUVERA, which compress multi-vector data into a single vector. This concept is innovative as it allows for applications of single-vector retrieval techniques, such as MIPS, to multi-vector problems. And I can foresee how this work is generalizable to other tasks and other datasets.\n\nComprehensive experiment: this experiment fully compared the state-of-the-art techniques with careful experiment settings, as well as good methodology.\n\nWEAKNESSES:\nComputational overhead: as introduced by the FDE creation and query processing, the computational overhead may be high as the dataset grows. The MUVERA may have problems in preprocessing and dynamic updates.\n\nThe robustness of MUVERA across different types of retrieval tasks and its effectiveness in different domains (e.g., legal documents versus scientific articles) have not been extensively validated, for those domain-specific tasks, it could be beneficial to run experiments on different datasets from diverse domains.\n\nQUESTIONS:\nWhat are the main trade-offs in this work? Maybe dimension reduction and accuracy are one of them: it is advantageous for processing speed and memory usage, but it can also lead to information loss.\n\nWhat are the specific challenges encountered when implementing FDEs in real-world systems, particularly concerning maintaining the balance between efficiency and retrieval accuracy?\n\nTuning parameter W in this work: \"The only tuning knob in our system is W; increasing W increases the number of candidates\n298 retrieved by MUVERA, which improves the recall\" Can this be pre-determined or you can use a specific algorithm to decide it?\n\nLIMITATIONS:\nIf the computation overhead is a barrier problem for MUVERA running on a very large dataset, it may face limitations on scalability of the FDE approach, especially in terms of how well it can be adapted to very large datasets or highly dynamic environments.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a method of converting multi-vector query-document retrieval problem to a single query-document vector-based retrieval problem. The basic idea is to project the multi-vector queries and documents into fixed-dimensional embedding through random projections. Further efficiency in storage of the random projections is achieved through vector quantization (product quantization) methods. The method is shown to result in improved recall as well as improvements in time performance. The projection into the fixed-dimensional space is obtained by latent semantic hashing (LSH). This concept could have been explained in a much simpler manner than presented in the paper.\n\nSTRENGTHS:\nThe formulation of the problem and the theoretical proofs are the strength of the work. The experimentation has been done on the obligatory BEIR benchmark datasets. Comparison has been made to one other approach (PLAID).\n\nWEAKNESSES:\nIt seems to me in the current age of LLMs, this technique appears to be a bit dated. There wasn't any mention of SpladeV2 and its variants either that are actively being used in commercial vector databases along with vector quantization in billion-vector size databases. Addressing this would be good for related work.  A 0.6 sec time performance for a single search is too slow for commercial uses, since typically multiple searches are rolled into an overall one. \n\nIt is difficult to reproduce this work from the level of detail provided. A block diagram illustrating the overall steps would be useful.\n\nThere are typos in the draft so a good spell check is needed.\n\nQUESTIONS:\nThe fact that LSH preserves nearness is known. How does adding product quantization alter this ability? Does this technique allow for vector search using the compressed representation or the vectors need to be decompressed back before similarity search?\n\nLIMITATIONS:\nThe limitations of the algorithm have not been adequately addressed. For example, for what sized datasets is this technique suitable or will be an overkill?", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to improve the search efficiency of multi-vector retrieval models, such as ColBERT. Specifically, the authors propose MUVERA framework, which reduces the multi-vector (MV) similarity of a query/document pair to the single-vector (SV) similarity by constructing Fixed Dimensional Encoding (FDE) of MV representation. Such reduction allows re-using many highly-optimized MIPS solvers for multi-vector retrieval. The authors also provide theoretical guarantee on the approximation error of FDEs. Empirically, compared to previous SOTA heuristic for multi-vector retrieval, MUBERA can achieve a similar Recall while significantly reduce the latency.\n\nSTRENGTHS:\n- The proposal of Fixed Dimensional Encodings (FDEs) such that its inner product approximate the multi-vector (MV) similarity is novel and interesting\n- The authors also provide strong theoretical guarantee under $\\epsilon$-approximation to the true MV similarity\n- Promising empirical results that significantly reduce the latency on most BEIR datasets compared to PLAID\n\nWEAKNESSES:\nHere are the summarized version. See detailed comments/questions in the next Section.\n- The index size of MUVERA framework can be quite large\n- MUVERA has sub-optimal performance on the MS-Marco dataset\n- no released code\n\nQUESTIONS:\n### Q1: The index size of MUVERA\nAs illustrated in Figure 1, MUVERA consider a two-stage search procedure: (1) first using MIPS to obtain a candidate set of documents; and (2) re-rank the candidates with exact Chamfer similarity.\n- (a) For the first step, using PQ codes on the FDE of documents and build ANN index indeed save the memory. For the second step, however, computing the exact Chamfer similarity still require storing all the original token embeddings per document?\n- (b) What's the memory usage and index size for stage 1 and stage 2, respectively?\n\n### Q2: Why do MUVERA has sub-optimal performance, compared to PLAID, on the MS-Marco dataset? \n- (a) Is it because of using data-agnostic partitioning functions (SimHash), making the approximation error of FDE larger?\n- (b) If the token embedding distributions are skew and concentrating on certain sub-space (maybe this is the case in MS-Marco?) , is it still a good idea to use SimHash in FDEs? Or k-means would do a better work?\n\n### Other minor comments\n- (a) In Figure 1 caption, there is a typo: \"comapred\" => \"compared\"\n- (b) At Line 133: \"vectors that are closer are more are more likely to...\" => there seems to be some repeatedly wording\n- (c) In Figure 2, the legend on the left-hand-side seems wrong? The orange square should be Doc Embeddings and the blue circle should be Query Embedding, according to the right-hand-side plot?\n\nLIMITATIONS:\nTo my knowledge, there's no potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}