Title: DESSERT: An Efficient Algorithm for Vector Set Search with Vector Set Queries
Venue: NeurIPS 2023
Accepted: yes
Keywords: Embedding Based Retrieval

Passage Ranking

Locality Sensitive Hashing

Randomized Algorithms
Abstract: We study the problem of $\text{\emph{vector set search}}$ with $\text{\emph{vector set queries}}$. This task is analogous to traditional near-neighbor search, with the exception that both the query and each element in the collection are $\text{\textit{sets}}$ of vectors. We identify this problem as a core subroutine for semantic search applications and find that existing solutions are unacceptably slow. Towards this end, we present a new approximate search algorithm, DESSERT ($\text{\bf D}$ESSERT $\text{\bf E}$ffeciently $\text{\bf S}$earches $\text{\bf S}$ets of $\text{\bf E}$mbeddings via $\text{\bf R}$etrieval $\text{\bf T}$ables). DESSERT is a general tool with strong theoretical guarantees and excellent empirical performance. When we integrate DESSERT into ColBERT, a state-of-the-art semantic search model, we find a 2-5x speedup on the MS MARCO and LoTTE retrieval benchmarks with minimal loss in recall, underscoring the effectiveness and practical applicability of our proposal.

DESSERT: An Efficient Algorithm for Vector Set Search with Vector Set Queries

Abstract

We study the problem of vector set search with vector set queries . This task is analogous to traditional near-neighbor search, with the exception that both the query and each element in the collection are sets of vectors. We identify this problem as a core subroutine for semantic search applications and find that existing solutions are unacceptably slow. Towards this end, we present a new approximate search algorithm, DESSERT ( D ESSERT E ffeciently S earches S ets of E mbeddings via R etrieval T ables). DESSERT is a general tool with strong theoretical guarantees and excellent empirical performance. When we integrate DESSERT into ColBERT, a state-of-the-art semantic search model, we find a 2-5x speedup on the MS MARCO and LoTTE retrieval benchmarks with minimal loss in recall, underscoring the effectiveness and practical applicability of our proposal.

1 Introduction

Similarity search is a fundamental driver of performance for many high-profile machine learning applications. Examples include web search [16], product recommendation [33], image search [21], de-duplication of web indexes [29] and friend recommendation for social media networks [39]. In this paper, we study a variation on the traditional vector search problem where the dataset D consists of a collection of vector sets D = { S 1 , ...S N } and the query Q is also a vector set. We call this problem vector set search with vector set queries because both the collection elements and the query are sets of vectors. Unlike traditional vector search, this problem currently lacks a satisfactory solution.

Furthermore, efficiently solving the vector set search problem has immediate practical implications. Most notably, the popular ColBERT model, a state-of-the-art neural architecture for semantic search over documents [23], achieves breakthrough performance on retrieval tasks by representing each query and document as a set of BERT token embeddings. ColBERT's current implementation of vector set search over these document sets, while superior to brute force, is prohibitively slow for real-time inference applications like e-commerce that enforce strict search latencies under 20-30 milliseconds [34, 5]. Thus, a more efficient algorithm for searching over sets of vectors would have significant implications in making state-of-the-art semantic search methods feasible to deploy in large-scale production settings, particularly on cost-effective CPU hardware.

Given ColBERT's success in using vector sets to represent documents more accurately, and the prevailing focus on traditional single-vector near-neighbor search in the literature [1, 41, 19, 12, 14, 28, 18], we believe that the potential for searching over sets of representations remains largely untapped. An efficient algorithmic solution to this problem could enable new applications in domains

where multi-vector representations are more suitable. To that end, we propose DESSERT, a novel randomized algorithm for efficient set vector search with vector set queries. We also provide a general theoretical framework for analyzing DESSERT and evaluate its performance on standard passage ranking benchmarks, achieving a 2-5x speedup over an optimized ColBERT implementation on several passage retrieval tasks.

1.1 Problem Statement

More formally, we consider the following problem statement.

Definition 1.1. Given a collection of N vector sets D = { S 1 , ...S N } , a query set Q , a failure probability δ ≥ 0 , and a set-to-set relevance score function F ( Q,S ) , the Vector Set Search Problem is the task of returning S ∗ with probability at least 1 -δ :

Here, each set S i = { x 1 , ...x m i } contains m i vectors with each x j ∈ R d , and similarly Q = { q 1 , ...q m q } contains m q vectors with each q j ∈ R d .

We further restrict our consideration to structured forms of F ( Q,S ) , where the relevance score consists of two 'set aggregation' or 'variadic' functions. The inner aggregation σ operates on the pairwise similarities between a single vector from the query set and each vector from the target set. Because there are | S | elements in S over which to perform the aggregation, σ takes | S | arguments. The outer aggregation A operates over the | Q | scores obtained by applying A to each query vector q ∈ Q . Thus, we have that

Here, sim is a vector similarity function. Because the inner aggregation is often a maximum or other non-linearity, we use σ ( · ) to denote it, and similarly since the outer aggregation is often a linear function we denote it with A ( · ) . These structured forms for F = A ◦ σ are a good measure of set similarity when they are monotonically non-decreasing with respect to the similarity between any pair of vectors from Q and S .

1.2 Why is near-neighbor search insufficient?

It may at first seem that we could solve the Vector Set Search Problem by placing all of the individual vectors into a near-neighbor index, along with metadata indicating the set to which they belonged. One could then then identify high-scoring sets by finding near neighbors to each q ∈ Q and returning their corresponding sets.

There are two problems with this approach. The first problem is that a single high-similarity interaction between q ∈ Q and x ∈ S does not imply that F ( Q,S ) will be large. For a concrete example, suppose that we are dealing with sets of word embeddings and that Q is a phrase where one of the items is 'keyword.' With a standard near-neighbor index, Q will match (with 100% similarity) any set S that also contains 'keyword,' regardless of whether the other words in S bear any relevance to the other words in Q . The second problem is that the search must be conducted over all individual vectors, leading to a search problem that is potentially very large. For example, if our sets are documents consisting of roughly a thousand words and we wish to search over a million documents, we now have to solve a billion-scale similarity search problem.

Contributions: In this work, we formulate and carefully study the set of vector search problem with the goal of developing a more scalable algorithm capable of tackling large-scale semantic retrieval problems involving sets of embeddings. Specifically, our research contributions can be summarized as follows:

We develop the first non-trivial algorithm, DESSERT, for the vector set search problem that scales to large collections ( n > 10 6 ) of sets with m> 3 items.

We formalize the vector set search problem in a rigorous theoretical framework, and we provide strong guarantees for a common (and difficult) instantiation of the problem.

We provide an open-source C++ implementation of our proposed algorithm that has been deployed in a real-world production setting 1 . Our implementation scales to hundreds of millions of vectors and is 3-5x faster than existing approximate set of vector search techniques. We also describe the implementation details and tricks we discovered to achieve these speedups and provide empirical latency and recall results on passage retrieval tasks.

2 Related Work

Near-Neighbor Search: Near-neighbor search has received heightened interest in recent years with the advent of vector-based representation learning. In particular, considerable research has gone into developing more efficient approximate near-neighbor (ANN) search methods that trade off an exact solution for sublinear query times. A number of ANN algorithms have been proposed, including those based on locality-sensitive hashing [1, 41], quantization and space partition methods [19, 12, 14], and graph-based methods [28, 18]. Among these classes of techniques, our proposed DESSERT framework aligns most closely with the locality-sensitive hashing paradigm. However, nearly all of the well-known and effective ANN methods focus on searching over individual vectors; our work studies the search problem for sets of entities. This modification changes the nature of the problem considerably, particularly with regards to the choice of similarity metrics between entities.

Vector Set Search: The general problem of vector set search has been relatively understudied in the literature. A recent work on database lineage tracking [25] addresses this precise problem, but with severe limitations. The proposed approximate algorithm designs a concatenation scheme for the vectors in a given set, and then performs approximate search over these concatenated vectors. The biggest drawback to this method is scalability, as the size of the concatenated vectors scales quadratically with the size of the vector set. This leads to increased query latency as well as substantial memory overhead; in fact, we are unable to apply the method to the datasets in this paper without terabytes of RAM. In this work, we demonstrate that DESSERT can scale to thousands of items per set with a linear increase (and a slight logarithmic overhead) in query time, which, to our knowledge, has not been previously demonstrated in the literature.

Document Retrieval: In the problem of document retrieval, we receive queries and must return the relevant documents from a preindexed corpus. Early document retrieval methods treated each documents as bags of words and had at their core an inverted index [30]. More recent methods embed each document into a single representative vector, embed the query into the same space, and performed ANN search on those vectors. These semantic methods achieve far greater accuracies than their lexical predecessors, but require similarity search instead of inverted index lookups [15, 33, 26].

ColBERT and PLAID: ColBERT [23] is a recent state of the art algorithm for document retrieval that takes a subtly different approach. Instead of generating a single vector per document, ColBERT generates a set of vectors for each document, approximately one vector per word. To rank a query, ColBERT also embeds the query into a set of vectors, filters the indexed sets, and then performs a brute force sum of max similarities operation between the query set and each of the document sets. ColBERT's passage ranking system is an instantiation of our framework, where sim( q, x ) is the cosine similarity between vectors, σ is the max operation, and A is the sum operation.

In a similar spirit to our work, PLAID [37] is a recently optimized form of ColBERT that includes more efficient filtering techniques and faster quantization based set similarity kernels. However, we note that these techniques are heuristics that do not come with theoretical guarantees and do not immediately generalize to other notions of vector similarity, which is a key property of the theoretical framework behind DESSERT.

3 Algorithm

At a high level, a DESSERT index D compresses the collection of target sets into a form that makes set to set similarity operations efficient to calculate. This is done by replacing each set S i with a sketch D [ i ] that contains the LSH values of each x j ∈ S i . At query time, we compare the corresponding

Indexing

Hash Tables

Querying

Index

QuerySet

..N

L

Count

Collisions

Index

SetRelevance

Target Sets

...N

.

Scores

0.8

0.3

0.7

...N

0.3

LSH values of the query set Q with the hashes in each D [ i ] to approximate the pairwise similarity matrix between Q and S (Figure 1). This matrix is used as the input for the aggregation functions A and σ to rank the target sets and return an estimate of S ∗ .

We assume the existence of a locality-sensitive hashing (LSH) family H ⊂ ( R d → Z ) such that for all LSH functions h ∈ H , p ( h ( x ) = h ( y )) = sim( x, y ) . LSH functions with this property exist for cosine similarity (signed random projections) [8], Euclidean similarity ( p -stable projections) [11], and Jaccard similarity (minhash or simhash) [6]. LSH is a well-developed theoretical framework with a wide variety of results and extensions [4, 3, 20, 40]. See Appendix C for a deeper overview.

Algorithm 1 describes how to construct a DESSERT index D . We first take L LSH functions f t for t ∈ [1 , L ] , f t ∈ H . We next loop over each S i to construct D [ i ] . For a given S i , we arbitrarily assign an identifier j to each vector x ∈ S i , j ∈ [1 , | S i | ] . We next partition the set [1 , m i ] using each hash function h t , such that for a partition p t , indices j 1 and j 2 are in the same set in the partition iff h ( S j 1 ) = h ( S j 2 ) . We represent the results of these partitions in a universal hash table indexed by hash function id and hash function value, such that D [ i ] t,h = { j | x j ∈ S i ∧ f t ( x j ) = h } .

Algorithm 2 describes how to query a DESSERT index D . At a high level, we query each sketch D i to get an estimate of F ( Q,S i ) , score i , and then take the argmax over the estimates to get an estimate of argmax i ∈{ 1 ,...N } F ( Q,S i ) . To get these estimates, we first compute the hashes h t,q for each query q and LSH function f t . Then, to get an estimate score i for a set S i , we loop over the hashes h t,q for each query vector q and count how often each index j appears in D [ i ] t,h t,q . After we finish this step, we have a count for each j that represents how many times h t ( q ) = h t ( x j ) . Equivalently, since p ( h ( x ) = h ( y )) = sim( x, y ) , if we divide by L we have an estimate for sim( x j , q ) . We then apply σ to these estimates and save the result in a variable agg q to build up the inputs to A , and then apply A to get our final estimate for F ( Q,S i ) , which we store in score i .

4 Theory

In this section, we analyze DESSERT's query runtime and provide probabilistic bounds on the correctness of its search results. We begin by finding the hyperparameter values and conditions that are necessary for DESSERT to return the top-ranked set with high probability. Then, we use these results to prove bounds on the query time. In the interest of space, we defer proofs to the Appendix.

Notation: For the sake of simplicity of presentation, we suppose that all target sets have the same number of elements m , i.e. | S i | = m . If this is not the case, one may replace m i with m max in our analysis. We will use the boldface vector s ( q, S i ) ∈ R | S i | to refer to the set of pairwise similarity calculations { sim( q, x 1 ) , . . . , sim( q, x m i ) } between a query vector and the elements of S i , and we will drop the subscript ( q, S i ) when the context is clear. See Table 1 for a complete notation reference.

Algorithm 1 Building a DESSERT Index

4.1 Inner Aggregation

We begin by introducing a condition on the σ component of the relevance score that allows us to prove useful statements about the retrieval process.

Definition 4.1. A function σ ( x ) : R m → R is ( α, β ) -maximal on U ⊂ R m if for 0 < β ≤ 1 ≤ α , ∀ x ∈ U :

The function σ ( x ) = max x is a trivial example of an ( α, β ) -maximal function on R m , with β = α = 1 . However, we can show that other functions also satisfy this definition:

Lemma 4.1.1. If φ ( x ) : R → R is ( α, β ) -maximal on an interval I , then the following function σ ( x ) : R m → R is ( α, β m ) -maximal on U = I m :

Algorithm 2 Querying a DESSERT Index

Note that in R , the ( α, β ) -maximal condition is equivalent to lower and upper bounds by linear functions βx and αx respectively, so many natural functions satisfy Lemma 4.1.1. We are particularly interested in the case I = [0 , 1] , and note that possible such φ include φ ( x ) = x with β = α = 1 , the exponential function φ ( x ) = e x -1 with β = 1 , α = e -1 , and the debiased sigmoid function φ ( x ) = 1 1+ e -x -1 2 with β ≈ 0 . 23 , α = 0 . 25 . Our analysis of DESSERT holds when the σ component of the relevance score is an ( α, β ) maximal function.

In line 12 of Algorithm 2, we estimate σ ( s ) by applying σ to a vector of normalized counts ˆ s . In Lemma 4.1.2, we bound the probability that a low-similarity set (one for which σ ( s ) is low) scores well enough to outrank a high-similarity set. In Lemma 4.1.3, we bound the probability that a high-similarity set scores poorly enough to be outranked by other sets. Note that the failure rate in both lemmas decays exponentially with the number of hash tables L .

Lemma 4.1.2. Assume σ is ( α, β ) -maximal. Let 0 < s max < 1 be the maximum similarity between a query vector and the vectors in the target set and let ˆ s be the set of estimated similarity scores. Given a threshold αs max < τ < α , we write ∆ = τ -αs max , and we have

for γ = ( s max ( α -τ ) τ (1 -s max ) ) τ α ( α (1 -s max ) α -τ ) ∈ ( s max , 1) . Furthermore, this expression for γ is increasing in s max and decreasing in τ , and γ has one sided limits lim τ ↘ αs max γ = 1 and lim τ ↗ α γ = s max .

Lemma 4.1.3. With the same assumptions as Lemma 4.1.2 and given ∆ > 0 , we have:

4.2 Outer Aggregation

Our goal in this section is to use the bounds established previously to prove that our algorithm correctly ranks sets according to F ( Q,S ) . To do this, we must find conditions under which the algorithm successfully identifies S ⋆ based on the approximate F ( Q,S ) scores.

Recall that F ( Q,S ) consists of two aggregations: the inner aggregation σ (analyzed in Section 4.1) and the outer aggregation A . We consider normalized linear functions for A , where we are given a set of weights 0 ≤ w ≤ 1 and we rank the target set according to a weighted linear combination of σ scores.

With this instantiation of the vector set search problem, we will proceed in Theorem 4.2 to identify a choice of the number of hash tables L that allows us to provide a probabilistic guarantee that the algorithm's query operation succeeds. We will then use this parameter selection to bound the runtime of the query operation in Theorem 4.3.

Theorem 4.2. Let S ⋆ be the set with the maximum F ( Q,S ) and let S i be any other set. Let B ⋆ and B i be the following sums (which are lower and upper bounds for F ( Q,S ⋆ ) and F ( Q,S i ) , respectively)

̸

Here, s max ( q, S ) is the maximum similarity between a query vector q and any element of the target set S . Let B ′ be the maximum value of B i over any set S i = S . Let ∆ be the following value (proportional to the difference between the lower and upper bounds)

If ∆ > 0 , a DESSERT structure with the following value 2 of L solves the search problem from Definition 1.1 with probability 1 -δ .

4.3 Runtime Analysis

Theorem 4.3. Suppose that each hash function call runs in time O ( d ) and that |D [ i ] t,h | < T ∀ i, t, h for some positive threshold T , which we treat as a data-dependent constant in our analysis. Then, using the assumptions and value of L from Theorem 4.2, Algorithm 2 solves the Vector Set Search Problem in query time

This bound is an improvement over a brute force search of O ( m q mNd ) when m or d is large. The above theorem relies upon the choice of L that we derived in Theorem 4.2.

5 Implementation Details

Filtering: We find that for large N it is useful to have an initial lossy filtering step that can cheaply reduce the total number of sets we consider with a low false-negative rate. We use an inverted index on the documents for this filtering step.

To build the inverted index, we first perform k -means clustering on a representative sample of individual item vectors at the start of indexing. The inverted index we will build is a map from centroid ids to document ids. As we add each set S i to D in Algorithm 1, we also add it into the inverted index: we find the closest centroid to each vector x ∈ S i , and then we add the document id i to all of the buckets in the inverted index corresponding to those centroids.

This method is similar to PLAID, the recent optimized ColBERT implementation [37], but our query process is much simpler. During querying, we query the inverted index buckets corresponding to the closest filter_probe centroids to each query vector. We aggregate the buckets to get a count for each document id, and then only rank the filter_k documents with DESSERT that have the highest count.

Space Optimized Sketches: DESSERT has two features that constrain the underlying hash table implementation: (1) every document is represented by a hash table, so the tables must be low memory, and (2) each query performs many table lookups, so the lookup operation must be fast. If (1) is not met, then we cannot fit the index into memory. If (2) is not met, then the similarity approximation for the inner aggregation step will be far too slow. Initially, we tried a naive implementation of the table, backed by a std::vector, std::map, or std::unordered_map. In each case, the resulting structure did not meet our criteria, so we developed TinyTable, a compact hash table that optimizes memory usage while preserving fast access times. TinyTables sacrifice O (1) update-access (which DESSERT does not require) for a considerable improvement to (1) and (2).

A TinyTable replaces the universal hash table in Algorithm 1, so it must provide a way to map pairs of (hash value, hash table id) to lists of vector ids. At a high level, a TinyTable is composed of L inverted indices from LSH values to vector ids. Bucket b of table i consists of vectors x j such that h i ( x j ) = b . During a query, we simply need to go to the L buckets that correspond to the query vector's L lsh values to find the ids of S i 's colliding vectors. This design solves (1), the fast lookup requirement, because we can immediately go to the relevant bucket once we have a query's hash value. However, there is a large overhead in storing a resizable vector in every bucket. Even an empty bucket will use 3 ∗ 8 = 24 bytes. This adds up: let r be the hash range of the LSH functions (the number of buckets in the inverted index for each of the L tables). If N = 1 M , L = 64 , and r = 128 , we will use N · L · r · 24 = 196 gigabytes even when all of the buckets are empty .

Thus, a TinyTable has more optimizations that make it space efficient. Each of the L hash table repetitions in a TinyTable are conceptually split into two parts: a list of offsets and a list of vector ids. The vector ids are the concatenated contents of the buckets of the table with no space in between (thus, they are always some permutation of 0 through m - 1). The offset list describes where one bucket ends and the next begins: the i th entry in the offset list is the (inclusive) index of the start of the i th hash bucket within the vector id list, and the i +1 th entry is the (exclusive) end of the ith hash bucket (if a bucket is empty, indices [ i ] = indices [ i +1] ). To save more bytes, we can further concatenate the L offset lists together and the L vector id lists together, since their lengths are always r and m i respectively. Finally, we note that if m ≤ 256 , we can store all of the the offsets and ids can be safely be stored as single byte integers. Using the same hypothetical numbers as before, a filled TinyTable with m = 100 will take up just N (24 + L ( m + r +1)) = 14 . 7 GB.

The Concatenation Trick: In our theory, we assumed LSH functions such that p ( h ( x ) = h ( y )) = sim( x, y ) . However, for practical problems such functions lead to overfull buckets; for example, GLOVE has an average vector cosine similarity of around 0 . 3 , which would mean each bucket in the LSH table would contain a third of the set. The standard trick to get around this problem is to concatenate C hashes for each of the L tables together such that p ( h ( x ) = h ( y )) = sim( x, y ) C . Rewriting, we have that

During a query, we count the number of collisions across the L tables and divide by L to get ˆ p ( h ( x ) = h ( y )) on line 11 of Algorithm 2. We now additionally pass count/L into Equation 1 to get an accurate similarity estimate to pass into σ on line 12 . Furthermore, evaluating Equation 1 for every collision probability estimate is slow in practice. There are only L +1 possible values for the count/L , so we precompute the mapping in a lookup table.

6 Experiments

Datasets: We tested DESSERT on both synthetic data and real-world problems. We first examined a series of synthetic datasets to measure DESSERT's speedup over a reasonable CPU brute force algorithm (using the PyTorch library [35] for matrix multiplications). For this experiment, we leave out the prefiltering optimization described in Section 5 to better show how DESSERT performs on its own. Following the authors of [25], our synthetic dataset consists of random groups of Glove [36] vectors; we vary the set size m and keep the total number of sets N = 1000 .

We next experimented with the MS MARCO passage ranking dataset (Creative Commons License) [32], N ≈ 8 . 8 M . The task for MS MARCO is to retrieve passages from the corpus relevant to a query. We used ColBERT to map the words from each passage and query to sets of embedding vectors suitable for DESSERT [37]. Following [37], we use the development set for our experiments, which contains 6980 queries.

Finally, we computed the full resource-accuracy tradeoff for ten of the LoTTE out-of-domain benchmark datasets, introduced by ColBERTv2 [38]. We excluded the pooled dataset, which is simply the individual datasets merged.

Experiment Setup: We ran our experiments on an Intel(R) Xeon(R) CPU E5-2680 v3 machine with 252 GB of RAM. We restricted all experiments to 4 cores (8 threads). We ran each experiment with the chosen hyperparameters and reported overall average recall and average query latency. For all experiments we used the average of max similarities scoring function.

6.1 Synthetic Data

The goal of our synthetic data experiment was to examine DESSERT's speedup over brute force vector set scoring. Thus, we generated synthetic data where both DESSERT and the brute force implementation achieved perfect recall so we could compare the two methods solely on query time.

The two optimized brute force implementations we tried both used PyTorch, and differed only in whether they computed the score between the query set and each document set individually ("Individual") or between the query set and all document sets at once using PyTorch's highly performant reduce and reshape operations ("Combined").

In each synthetic experiment, we inserted 1000 documents of size m for m ∈ [2 , 4 , 8 , 16 , ..., 1024] into DESSERT and the

Query Time v.SetSizeon SyntheticGlove Data

DESSERT

Pytorch Brute Force Individual

100

Pytorch Brute Force Combined

Query time (s)

10-1

10-2

101

102

103

m = # vectors per set

brute force index. The queries in each experiment were simply the documents with added noise. The DESSERT hyperparameters we chose were L = 8 and C = log 2 ( m ) + 1 . The results of our experiment, which show the relative speedup of using DESSERT at different values of m , are in Figure 2. We observe that DESSERT achieves a consistent 10-50x speedup over the optimized Pytorch brute force method and that the speedup increases with larger m (we could not run experiments with even larger m because the PyTorch runs did not finish within the time allotted).

6.2 Passage Retrieval

Passage retrieval refers to the task of identifying and returning the most relevant passages from a large corpus of documents in response to a search query. In these experiments, we compared DESSERT to PLAID, ColBERT's heavily-optimzed state-of-the-art late interaction search algorithm, on the MS MARCO and LoTTE passage retrieval tasks.

We found that the best ColBERT hyperparameters were the same as reported in the PLAID paper, and we successfully replicated their results. Although PLAID offers a way to trade off time for accuracy, this tradeoff only increases accuracy at the sake of time, and even then only by a fraction of a percent. Thus, our results represent points on the recall vs time Pareto frontier that PLAID cannot reach.

MS MARCO Results For MS MARCO, we performed a grid search over DESSERT parameters C = { 4 , 5 , 6 , 7 } , L = { 16 , 32 , 64 } , filter _ probe = { 1 , 2 , 4 , 8 } , and filter _ k = { 1000 , 2048 , 4096 , 8192 , 16384 } . We reran the best configurations to obtain the results in Table 2. We report two types of results: methods tuned to return k = 10 results and methods tuned to return k = 1000 results. For each, we report DESSERT results from a low latency and a high latency part of the Pareto frontier. For k = 1000 we use the standard R @1000 metric, the average recall of the top 1 passage in the first 1000 returned passages. This metric is meaningful because retrieval pipelines frequently rerank candidates after an initial retrieval stage. For k = 10 we use the standard MRR @10 metric, the average mean reciprocal rank of the top 1 passage in the first 10 returned passages. Overall, DESSERT is 2-5x faster than PLAID with only a few percent loss in recall.

LoTTE Results For LoTTE, we performed a grid search over C = { 4 , 6 , 8 } , L = { 32 , 64 , 128 } , filter _ probe = { 1 , 2 , 4 } , and filter _ k = { 1000 , 2048 , 4096 , 8192 } . In Figure 3, we plot the full Pareto tradeoff for DESSERT on the 10 LoTTE datasets (each of the 5 categories has a "forum" and "search" split) over these hyperparameters, as well as the single lowest-latency point achievable by PLAID. For all test datasets, DESSERT provides a Pareto frontier that allows a tradeoff between recall and latency. For both Lifestyle test splits, both Technology test splits, and the Recreation and Science test-search splits, DESSERT achieves a 2-5x speedup with minimal loss in accuracy. On Technology, DESSERT even exceeds the accuracy of PLAID at half of PLAID's latency.

7 Discussion

We observe a substantial speedup when we integrate DESSERT into ColBERT, even when compared against the highly-optimized PLAID implementation. While the use of our algorithm incurs a slight recall penalty - as is the case with most algorithms that use randomization to achieve acceleration Table 2 and Figure 3 shows that we are Pareto-optimal when compared with baseline approaches.

We are not aware of any algorithm other than DESSERT that is capable of latencies in this range for set-to-set similarity search. While systems such as PLAID are tunable, we were unable to get them to operate in this range. For this reason, DESSERT is likely the only set-to-set similarity search algorithm that can be run in real-time production environments with strict latency constraints.

writing

recreation

82

86

☆

Succeess@10

84

80

78

82

76

80

15

74

78

☆

72

70

76

70

74

2.5.50

7.5

10.0

12.5

5

10

15

5

10

15

technology

science

75

80

7055605505

70

PLAID test-forum

DESSERT test-forum

09

PLAID test-search

DESSERT test-search

PLAID dev-forum

DESSERT dev-forum

50

PLAID dev-search

DESSERT dev-search

40

40

5

10

15

10

15

We also ran a single-vector search baseline using ScaNN, the leading approximate kNN index [14]. ScaNN yielded 0.77 Recall@1000, substantially below the state of the art. This result reinforces our discussion in Section 1.2 on why single-vector search is insufficient.

Broader Impacts and Limitations: Ranking and retrieval are important steps in language modeling applications, some of which have recently come under increased scrutiny. However, our algorithm is unlikely to have negative broader effects, as it mainly enables faster, more cost-effective search over larger vector collections and does not contribute to the problematic capabilities of the aforementioned language models. Due to computational limitations, we conduct our experiments on a relatively small set of benchmarks; a larger-scale evaluation would strengthen our argument. Finally, we assume sufficiently high relevance scores and large gaps in our theoretical analysis to identify the correct results. These hardness assumptions are standard for LSH.

8 Conclusion

In this paper, we consider the problem of vector set search with vector set queries, a task understudied in the existing literature. We present a formal definition of the problem and provide a motivating application in semantic search, where a more efficient algorithm would provide considerable immediate impact in accelerating late interaction search methods. To address the large latencies inherent in existing vector search methods, we propose a novel randomized algorithm called DESSERT that achieves significant speedups over baseline techniques. We also analyze DESSERT theoretically and, under natural assumptions, prove rigorous guarantees on the algorithm's failure probability and runtime. Finally, we provide an open-source and highly performant C++ implementation of our proposed DESSERT algorithm that achieves 2-5x speedup over ColBERT-PLAID on the MS MARCO and LoTTE retrieval benchmarks. We also note that a general-purpose algorithmic framework for vector set search with vector set queries could have impact in a number of other applications, such as image similarity search [42], market basket analysis [22], and graph neural networks [43], where it might be more natural to model entities via sets of vectors as opposed to restricting representations to a single embedding. We believe that DESSERT could provide a viable algorithmic engine for enabling such applications and we hope to study these potential use cases in the future.

9 Acknowledgments

This work was completed while the authors were working at ThirdAI. We do not have any external funding sources to acknowledge.

References

Alexandr Andoni and Piotr Indyk. Near-optimal hashing algorithms for approximate nearest neighbor in high dimensions. Communications of the ACM , 51(1):117-122, 2008.

Alexandr Andoni, Piotr Indyk, Thijs Laarhoven, Ilya Razenshteyn, and Ludwig Schmidt. Practical and optimal lsh for angular distance. Advances in neural information processing systems , 28, 2015.

Alexandr Andoni, Piotr Indyk, Huy L Nguyen, and Ilya Razenshteyn. Beyond locality-sensitive hashing. In Proceedings of the twenty-fifth annual ACM-SIAM symposium on Discrete algorithms , pages 1018-1028. SIAM, 2014.

Alexandr Andoni and Ilya Razenshteyn. Optimal data-dependent hashing for approximate near neighbors. In Proceedings of the forty-seventh annual ACM symposium on Theory of computing , pages 793-801, 2015.

Ioannis Arapakis, Souneil Park, and Martin Pielot. Impact of response latency on user behaviour in mobile web search. In Proceedings of the 2021 Conference on Human Information Interaction and Retrieval , pages 279-283, 2021.

A. Broder. On the resemblance and containment of documents. In Proceedings of the Compression and Complexity of Sequences 1997 , SEQUENCES '97, page 21, USA, 1997. IEEE Computer Society.

J Lawrence Carter and Mark N Wegman. Universal classes of hash functions. In Proceedings of the ninth annual ACM symposium on Theory of computing , pages 106-112, 1977.

Moses S. Charikar. Similarity estimation techniques from rounding algorithms. In Proceedings of the Thiry-Fourth Annual ACM Symposium on Theory of Computing , STOC '02, page 380-388, New York, NY, USA, 2002. Association for Computing Machinery.

Benjamin Coleman, Richard Baraniuk, and Anshumali Shrivastava. Sub-linear memory sketches for near neighbor search on streaming data. In International Conference on Machine Learning , pages 2089-2099. PMLR, 2020.

Benjamin Coleman and Anshumali Shrivastava. Sub-linear race sketches for approximate kernel density estimation on streaming data. In Proceedings of The Web Conference 2020 , pages 1739-1749, 2020.

Mayur Datar, Nicole Immorlica, Piotr Indyk, and Vahab S. Mirrokni. Locality-sensitive hashing scheme based on p-stable distributions. In Proceedings of the Twentieth Annual Symposium on Computational Geometry , SCG '04, page 253-262, New York, NY, USA, 2004. Association for Computing Machinery.

Yihe Dong, Piotr Indyk, Ilya Razenshteyn, and Tal Wagner. Learning space partitions for nearest neighbor search. arXiv preprint arXiv:1901.08544 , 2019.

Joshua Engels, Benjamin Coleman, and Anshumali Shrivastava. Practical near neighbor search via group testing. Advances in Neural Information Processing Systems , 34:9950-9962, 2021.

Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, and Sanjiv Kumar. Accelerating large-scale inference with anisotropic vector quantization. In International Conference on Machine Learning , pages 3887-3896. PMLR, 2020.

Jui-Ting Huang, Ashish Sharma, Shuying Sun, Li Xia, David Zhang, Philip Pronin, Janani Padmanabhan, Giuseppe Ottaviano, and Linjun Yang. Embedding-based retrieval in facebook search. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 2553-2561, 2020.

Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of the 22nd ACM international conference on Information & Knowledge Management , pages 2333-2338, 2013.

Piotr Indyk and Rajeev Motwani. Approximate nearest neighbors: Towards removing the curse of dimensionality. In Proceedings of the Thirtieth Annual ACM Symposium on Theory of Computing , STOC '98, page 604-613, New York, NY, USA, 1998. Association for Computing Machinery.

Masajiro Iwasaki and Daisuke Miyazaki. Optimization of indexing based on k-nearest neighbor graph for proximity search in high-dimensional data. arXiv preprint arXiv:1810.07355 , 2018.

Herve Jegou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. IEEE transactions on pattern analysis and machine intelligence , 33(1):117-128, 2010.

Jianqiu Ji, Jianmin Li, Shuicheng Yan, Bo Zhang, and Qi Tian. Super-bit locality-sensitive hashing. Advances in neural information processing systems , 25, 2012.

Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Transactions on Big Data , 7(3):535-547, 2019.

Manpreet Kaur and Shivani Kang. Market basket analysis: Identify the changing trends of market data using association rule mining. Procedia computer science , 85:78-85, 2016.

Omar Khattab and Matei Zaharia. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval , pages 39-48, 2020.

Runze Lei, Pinghui Wang, Rundong Li, Peng Jia, Junzhou Zhao, Xiaohong Guan, and Chao Deng. Fast rotation kernel density estimation over data streams. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 892-902, 2021.

Michael Leybovich and Oded Shmueli. Efficient approximate search for sets of vectors. arXiv preprint arXiv:2107.06817 , 2021.

Sen Li, Fuyu Lv, Taiwei Jin, Guli Lin, Keping Yang, Xiaoyi Zeng, Xiao-Ming Wu, and Qianli Ma. Embedding-based product retrieval in taobao search. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 3181-3189, 2021.

Chen Luo and Anshumali Shrivastava. Arrays of (locality-sensitive) count estimators (ace) anomaly detection on the edge. In Proceedings of the 2018 World Wide Web Conference , pages 1439-1448, 2018.

Yu A Malkov and Dmitry A Yashunin. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE transactions on pattern analysis and machine intelligence , 42(4):824-836, 2018.

Gurmeet Singh Manku, Arvind Jain, and Anish Das Sarma. Detecting near-duplicates for web crawling. In Proceedings of the 16th international conference on World Wide Web , pages 141-150, 2007.

Christopher D Manning, Prabhakar Raghavan, and Hinrich Schütze. Introduction to information retrieval . Cambridge university press, 2008.

Nicholas Meisburger and Anshumali Shrivastava. Distributed tera-scale similarity search with mpi: Provably efficient similarity search over billions without a single distance computation. arXiv preprint arXiv:2008.03260 , 2020.

Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. Ms marco: A human generated machine reading comprehension dataset. In Workshop on Cognitive Computing at NIPS , 2016.

Priyanka Nigam, Yiwei Song, Vijai Mohan, Vihan Lakshman, Weitian Ding, Ankit Shingavi, Choon Hui Teo, Hao Gu, and Bing Yin. Semantic product search. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 2876-2885, 2019.

Steve Olenski. Why brands are fighting over milliseconds, Nov 2016.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP) , pages 1532-1543, 2014.

Keshav Santhanam, Omar Khattab, Christopher Potts, and Matei Zaharia. Plaid: An efficient engine for late interaction retrieval. arXiv preprint arXiv:2205.09707 , 2022.

Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. Colbertv2: Effective and efficient retrieval via lightweight late interaction. arXiv preprint arXiv:2112.01488 , 2021.

Aneesh Sharma, C Seshadhri, and Ashish Goel. When hashes met wedges: A distributed algorithm for finding high similarity vectors. In Proceedings of the 26th International Conference on World Wide Web , pages 431-440, 2017.

Anshumali Shrivastava and Ping Li. Improved asymmetric locality sensitive hashing (alsh) for maximum inner product search (mips). In Proceedings of the Thirty-First Conference on Uncertainty in Artificial Intelligence , pages 812-821, 2015.

Yiqiu Wang, Anshumali Shrivastava, Jonathan Wang, and Junghee Ryu. Flash: Randomized algorithms accelerated over cpu-gpu for ultra-high dimensional similarity search. arXiv preprint arXiv:1709.01190 , 2017.

Jun Yang, Yu-Gang Jiang, Alexander G Hauptmann, and Chong-Wah Ngo. Evaluating bagof-visual-words representations in scene classification. In Proceedings of the international workshop on Workshop on multimedia information retrieval , pages 197-206, 2007.

Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L. Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. CoRR , abs/1806.01973, 2018.

A Proofs of Main Results

Lemma 4.1.1. If φ ( x ) : R → R is ( α, β ) -maximal on an interval I , then the following function σ ( x ) : R m → R is ( α, β m ) -maximal on U = I m :

Proof. Take some x ∈ D (so each x i ∈ I ). Since in R , max( x ) = x , we have from the definition of ( α, β ) -maximal that

For the upper bound, we have

where the second inequality follows by the properties of the max function.

For the lower bound, we have that

where the the second inequality again follows by the properties of the max function.

Lemma 4.1.2. Assume σ is ( α, β ) -maximal. Let 0 < s max < 1 be the maximum similarity between a query vector and the vectors in the target set and let ˆ s be the set of estimated similarity scores. Given a threshold αs max < τ < α , we write ∆ = τ -αs max , and we have

for γ = ( s max ( α -τ ) τ (1 -s max ) ) τ α ( α (1 -s max ) α -τ ) ∈ ( s max , 1) . Furthermore, this expression for γ is increasing in s max and decreasing in τ , and γ has one sided limits lim τ ↘ αs max γ = 1 and lim τ ↗ α γ = s max .

Proof. We first apply a generic Chernoff bound to σ (ˆ s ) , which gives us the following bounds for any t > 0 :

We now proceed by continuing to bound the numerator. Because σ is ( α, β ) -maximal, we can bound σ (ˆ s ) with α maxˆ s . We can further bound maxˆ s by bounding the maximum with the sum and the sum with m times the maximal element. We are now left with the formula for the moment generating function for ˆ s max . ˆ s max ∼ scaled binomial L -1 B ( s max , L ) , so we can directly substitute the binomial moment generating function into the expression:

Combining these two equations yields the following bound:

We wish to select a value of t to minimize the upper bound. By setting the derivative of the upper bound to zero, and imposing 0 < τ < α , α ≥ 1 , and 0 < s max < 1 , we find that

This is greater than zero when the numerator inside the ln is greater than the denominator, or equivalently when τ > s max α . Thus the valid range for τ is ( s max α, α ) (and similarly the valid range for s max is (0 , τ /α ) ). These bounds have a natural interpretation: to be meaningful, the threshold must be between the expected value and the maximum value for α times a p = s max binomial. Substituting t = t ⋆ into our upper bound, we obtain:

Thus we have that

We will now prove our claims about γ viewed as a function of s max ∈ (0 , τ /α ) and τ ∈ ( s max α, α ) . We will first examine the limits of γ with respect to τ at the ends of its range. Since γ is continuous, we can find one of the limits by direct substitution:

The second limit is harder; we merge γ into one exponent and then simplify:

where we use the fact that lim x ↘ 0 x x = e lim x ↘ 0 x ln( x ) = 1 (we can see that lim x → 0 x ln( x ) = lim x → 0 ln( x ) / (1 /x ) = 0 with L'Hopital's rule). We next find the partial derivatives of γ :

We are interested in the signs of these partial derivatives. First examining δγ δs max , τ > αs max = ⇒ τ -αs max > 0 . Similarly, α > τ = ⇒ α -τ > 0 and s max ( α -τ ) = s max α -s max τ > 0 . Finally, s max < 1 = ⇒ τ (1 -s max ) = τ -τs max > 0 . Thus every term is positive and the entire fraction is positive. Next examining δγ δτ , by similar logic α -τ > 0 and τ -s max τ > 0 and αs max -s max τ > 0 . For the ln , since τ > αs max , τ -s max τ > αs max -s max τ , so the numerator is greater than the denominator and the ln is positive. Finally, since s max < 1 , s max -1 < 0 , and thus the entire fraction has a single negative term in the product, so it is negative.

This completes our lemma: γ is a strictly decreasing function of τ and a strictly increasing function of s max . Since τ is decreasing and has a leftward limit of 1 and a rightward limit of s max , all values for γ are in ( s max , 1) .

First, we will make a substitution. We note that γ is a strictly decreasing function on this interval of τ with range ( s max , 1) . To see this, we will first make the following change of variabls:

for k ∈ (0 , ∞ ) . This parameterizes τ ∈ ( s max α, α ) as a weighted sum of s max α and α . Plugging in and simplifying, we have that

This is a continous function over k ∈ (0 , ∞ ) and s max ∈ ()

Lemma 4.1.3. With the same assumptions as Lemma 4.1.2 and given ∆ > 0 , we have:

Proof. We will prove this lemma with a chain of inequalities, starting with Pr[ σ (ˆ s ) ≤ βs max -∆] :

The explanations for each step are as follows:

Because σ (ˆ s ) ≥ β max s , we can replace σ (ˆ s ) with β max s and the probability will be strictly larger.

By the definition of the max operator, each individual ˆ s i ≤ maxˆ s , and in particular this is true for ˆ s max (the estimated similarity for the ground-truth maximum similarity vector). Thus, we have β ˆ s max ≤ β maxˆ s , so we can again apply a replacement to get a further upper bound.

Rearranging.

Because Pr [ | a -b | ≥ c ] = Pr [ a -b ≥ c ] + Pr [ b -a ≥ c ]

ˆ s max is the sum of L Bernoulli trials with success probability s max and scaled by β/L . Thus, we can directly apply the Hoeffding ineuqliaty with L trials with success probability βs max L .

Theorem 4.2. Let S ⋆ be the set with the maximum F ( Q,S ) and let S i be any other set. Let B ⋆ and B i be the following sums (which are lower and upper bounds for F ( Q,S ⋆ ) and F ( Q,S i ) , respectively)

̸

Here, s max ( q, S ) is the maximum similarity between a query vector q and any element of the target set S . Let B ′ be the maximum value of B i over any set S i = S . Let ∆ be the following value (proportional to the difference between the lower and upper bounds)

If ∆ > 0 , a DESSERT structure with the following value 3 of L solves the search problem from Definition 1.1 with probability 1 -δ .

Proof. For set S ⋆ to have the highest estimated score ˆ F ( Q,S ⋆ ) , we need all other sets to have lower scores. Our overall proof strategy will find a minimum L that upper bounds the probability that each inner aggregation of a set S = S ∗ is greater than ∆+ αs ′ j, max and a minimum L that lower bounds the probability that the inner aggregation of S ∗ is less βs ∗ j, max -∆ . Finally, we will show that an L that is a maximum of these two values solves the search problem.

̸

̸

Upper Bound : We start with the upper bound on S i = S ∗ : we have from Lemma 4.1.2 that

with

and γ i ∈ (0 , 1) . To make our analysis simpler, we are interested in the maximum γ max of all these γ i as a function of ∆ , since then all of these bounds will hold with the same γ , making it easy to solve for L . Since lim s max ↘ 0 γ = 0 and lim s max ↗ 1 -∆ = 1 -∆ /α , there must be some γ max ∈ (1 -∆ /α, 1) that maximizes this expression over any s max . This exact maximum is hard to find analytically, but we are guaranteed that it is less than 1 by Lemma 4.1.2. We will use the term γ max in our analysis, since it is data dependent and guaranteed to be in the range (0 , 1) . We also numerically plot some values of γ max here with α = 1 to give some intuition for what the function looks like over different ∆ ; we note that it is decreasing in ∆ and approximates a linear function for ∆ >> 0 .

Numerical Solutions for Ymax with α = 1

1.0

0.8

0.6

Ymax

0.4

0.2

0.0

0.0

0.2

0.4

0.6

0.8

1.0

△

To hold with the union bound over all N -1 target sets and all m q query vectors with probability δ 2 , we want the probability that our bound holds on a single set and query vector to be less than δ 2( N -1) m q . We find that this is true with L ≥ log 2( N -1) m q m δ log( 1 γ max ) -1 for any q j and S i :

Lower Bound We next examine the lower bound on S ∗ : we have from Lemma 4.1.3 that

To hold with the union bound over all m q query vectors with probability δ 2 , we want the probability that our bound holds on a single set and query vector to be less than δ 2 m q . We find that this is true

Putting it Together

Let

̸

Then the upper and lower bounds we derived in the last two sections both apply. Let ✶ be the random variable that is 1 when the m ∗ m q ∗ ( N -1) upper bounds and the m q lower bounds hold and that is 0 otherwise. Consider all sets S i = S ∗ . Then the probability we solve the Vector Set Search Problem from Definition 1.1 is equal to the probability that all ∀ i , ( ˆ F ( Q,S ∗ ) -ˆ F ( Q,S i ) > 0) . We now lower bound this probability:

and thus DESSERT solves the Vector Set Search Problem with this choice of L . Finally, we can now examine the expression for L to determine its asymptotic behavior. Dropping the positive data dependent constants 1 γ max , 1 2∆ 2 , and β 2 , the left term in the max for L is O (log( Nm q m δ )) and the right term in the max is O (log( m q δ )) , and thus L = O ( log( Nm q m δ ) ) .

Theorem 4.3. Suppose that each hash function call runs in time O ( d ) and that |D [ i ] t,h | < T ∀ i, t, h for some positive threshold T , which we treat as a data-dependent constant in our analysis. Then, using the assumptions and value of L from Theorem 4.2, Algorithm 2 solves the Vector Set Search Problem in query time

Proof. If we suppose that each call to the hash function f t is O ( d ) , the runtime of the algorithm is

To bound this quantity, we use the sparsity assumption we made in the theorem: no set S i contains too many elements that are very similar to a single query vector q j . Formally, we require that

for some positive threshold T . With this assumption, the runtime of Algorithm 2 is

Plugging in the L we found in our previous theorem, and treating T as data dependent constant, we have that the runtime of Algorithm 2 is

which completes the proof.

B Hyperparameter Settings

Settings for DESSERT corresponding to the first row in the left part of Table 2, where DESSERT was optimized for returning 10 documents in a low latency part of the Pareto frontier:

Settings for DESSERT corresponding to the second row in the left part of Table 2, where DESSERT was optimized for returning 10 documents in a high latency part of the Pareto frontier:

Settings for DESSERT corresponding to the first row in the right part of Table 2, where DESSERT was optimized for returning 1000 documents in a low latency part of the Pareto frontier:

Settings for DESSERT corresponding to the second row in the right part of Table 2, where DESSERT was optimized for returning 1000 documents in a high latency part of the Pareto frontier:

Intuitively, these parameter settings make sense: increase the initial filtering size and the number of total hashes for higher accuracy, and increase the initial filtering size for returning more documents (1000 vs. 10).

C Background on Locality-Sensitive Hashing and Inverted Indices for Similarity Search

Here, we offer a refresher on using locality-sensitive hashing for similarity search with a basic inverted index structure.

Consider a set of distinct vectors X = { x 1 , . . . , x N } where each x i ∈ R d . A hash function h with a range m maps each x i to an integer in the range [1 , m ] . Two vectors x i and x j are said to "collide" when h ( x i ) = h ( x j ) .

̸

As a warmup, we will first consider the case of a hash function h drawn from a set of universal hash functions H . Under such a function, if i = j , p ( h ( x ) = h ( y )) = 1 m ; such families exist in practice [7]. We can build an inverted index using this hash function by mapping each hash value in [1 , m ] to the set of vectors X v that have this hash value. Then, given a new vector y , we can query the inverted index with v = h ( y ) . We can see that y ∈ X iff y ∈ X v . Such an index is in a sense solving a search problem, if we only care about finding exact duplicates of our search query. Additionally, we can solve the nearest neighbor problem with this index in time O ( N ) , by going to every bucket and checking the distance of a query against every vector in the bucket.

Now, in a similar way as in the universal case, let h to be drawn from a family of locality-sensitive hash functions H . At a high level, instead of mapping vectors uniformly to [1 , m ] , h maps vectors that are close together to the same hash value more often. Formally, if we define a "close" threshold r 1 , a "far" threshold r 2 , a "close" probability p 1 , and a "far" probability p 2 , with p 1 > p 2 and r 1 < r 2 , then we say H is ( r 1 , r 2 , p 1 , p 2 ) -sensitive if

where d is a distance metric. See [17] for the origin of locality-sensitive hashing and this definition. Intuitively, if we build an inverted index using h in the same way as before, it now seems we have a strategy to solve the (approximate) nearest neighbor problem more efficiently: given a query q , only search for nearest neighbors in the bucket h ( q ) , since each of these points x likely has d ( q, x ) < r 1 . However, this strategy has a problem: with our definition, even a close neighbor might not be a collision with probability (1 -p 1 ) . Thus, we can repeat our inverted index L times with different h i drawn independently from H , such that our probability of not finding a close neighbor in any bucket is (1 -p 1 ) L . FALCONN [2] is an LSH inverted index algorithm that uses this basic idea, along with concatenation and probing tricks, to achieve an asymptotically optimal (and data-dependent sub-linear) runtime; see the paper and associated code repository for more details.

One final note is that in practice, most LSH families satisfy a much stronger condition than the above. Consider a similarity function sim ∈ [0 , 1] , where s ( x, y ) = 1 = ⇒ x = y . As x and y get more dissimilar (e.g. their distance increases according to some distance metric), s ( x, y ) decreases. For most LSH families, there exists an explicit similarity function that their collision probability satisfies, such that p ( h ( x ) = h ( y )) = sim( x, y ) . Such LSH families exist for most common similarity

functions, including cosine similarity (signed random projections) [8], Euclidean similarity ( p -stable projections) [11], and Jaccard similarity (minhash or simhash) [6]. Following [13, 9, 10, 27, 24, 31], in our work, we use LSH families with this explicit similarity description to provide tight analyses and strong guarantees for similarity-search algorithms.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study the general case of multi-vector retrieval, i.e., ColBERT and beyond. They propose and analyze a new algorithm for this \"vector set\" search task, with theoretical guarantees. When integrated into ColBERT, the proposed DESSERT method is 2-5x faster at some, relatively small loss in quality.\n\nSTRENGTHS:\n1. The authors discuss a general class of \"vector set\" retrieval scoring functions (Sec 1.1), generalizing ColBERT.\n\n2. The authors formalize the \"vector set\" search problem, perhaps for the first time (although I'd be interested in a comparison with Luan et al. 2021 \"Sparse, Dense, and Attentional Representations for Text Retrieval\" for completeness). They propose a new algorithm, unlike ColBERT's. The algorithm provides theoretical guarantees.\n\n3. The proposed method is shown to be ~2pt worse than a recent state-of-the-art ColBERT method (PLAID) on MRR quality, while being 3x faster, applying search in just 15.5 milliseconds per query on CPU. This is on the Pareto frontier with respect to the ColBERT retrieval quality/latency curve.\n\nWEAKNESSES:\nDESSERT is primarily tested on the MS MARCO query set (besides a small synthetic experiment). The authors justify much of this by citing PLAID [35], but that other paper like many IR papers in the last 1.5-2 years tests on several datasets, including especially out-of-domain and larger datasets. It's not inherently clear how much quality loss DESSERT would incur out-of-domain, say, on the LoTTE or BEIR (or even a subset of one of them, if computational resources do not permit more tests). This is the main weakness in my opinion.\n\nQUESTIONS:\n- How much space does the full method consume? I see 15GB are consumed by the hash table. Is that all?\n- How long does preparing the search data structures (indexing) take?\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors studied a new problem of vector set search with vector set queries. They have formalized this problem and proposed a novel, provable hashing scheme, DESSERT, to efficiently deal with this problem. Moreover, they have provided theoretical analysis for DESSERT and conducted experiments to validate its efficiency and accuracy.\n\nSTRENGTHS:\n- The motivation to study this new problem is clear, which is a vital sub-problem in semantic search, and the authors formalize a general definition of this problem.\n\n- They design an efficient hashing to solve this challenging problem and provide a theoretical bound and query time complexity.\n\n- The presentation is clear and easy to follow.\n\nWEAKNESSES:\n***Novelty and Contributions***\n\n- I appreciate the authors have provided the theoretical bound and query time complexity for their proposed hashing scheme. One of my concerns is that the improvement of the query time complexity is only marginal compared to the brute force method, which is still O(N) in general. Moreover, the impact of L is less discussed either in the theory part or in the experiments, making the efficiency improvement less promising.\n\n***Experiments***\n\n- Another primary concern is the experiments. The experimental results are too few. They only validate the performance of DESSERT on a single real-world dataset, making their conclusion of DESSERT compared to PLAID less convincing. Moreover, no ablation study and parameter study is performed, making users hard to know the effectiveness of implementation tricks and how to set the hyperparameters of DESSERT. Please see Q1-Q4.\n\n***Presentation***\n\n- The paper still needs some careful proofreading. There are some minor issues in Algorithm 2. Please refer to Q5 for more details.\n\nQUESTIONS:\n***Experiments***\n\nQ1: For the passage retrieval experiments, it is less convincing to me to use a single dataset for validation. Can they conduct and report results on at least one more dataset (for example, TREC Complex Answer Retrieval dataset [1] used in ColBERT)? \n\nQ2: The metric R@1000 is too weak as an accuracy measure. Can they report recall together with R@1000? \n\nQ3: What are the best hyperparameter settings of C, L, probe, and k for the results they show in Table 1? Moreover, can they conduct the parameter study for some vital parameters (e.g., L and C) of DESSERT?\n\nQ4: In Section 5, the authors developed some implementation tricks for DESSERT. Can they conduct an ablation study to validate the effectiveness of the three tricks they proposed (i.e., filtering by k-means clustering, space optimized sketches, and the concatenation trick)?\n\n***Presentations***\n\nQ5: There exist some typos in Algorithm 2:\n- In line 2: $G\\circ H(G,S_i)$ -> $A_1 \\circ A_2(Q,S_i)$;\n- In line 4: $f_k(q)$ -> $f_L(q)$;\n- In line 11: $count_{m_q}/L$ -> ${count}_{m_i}/L$;\n- In line 13: $0,\\cdots,N-1$ -> $1,\\cdots,N$.\n\n***Reference***\n\n[1] Laura Dietz, Manisha Verma, Filip Radlinski, and Nick Craswell. 2017. TREC\nComplex Answer Retrieval Overview. In TREC.\n\nLIMITATIONS:\nThis work does not appear to have any negative social impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the vector set search problem, which is an extension of the canonical near-neighbor search problem and finds an application in the semantic search task. The paper claims that existing methods for vector set search are unacceptably slow, and thus, proposes an approximate search algorithm called DESSERT. The paper presents both theoretical analysis and empirical evaluation to demonstrate the effectiveness of the DESSERT algorithm.\n\nSTRENGTHS:\nS1. The paper proposes a new algorithm, DESSERT, to solve the vector set search problem efficiently. \n\nS2. The paper presents both theoretical analysis and empirical evaluation to testify the effectiveness of the DESSERT algorithm. \n\nS3. The DESSERT algorithm can be applied to the semantic search task.\n\nWEAKNESSES:\nW1. [Motivations & Contributions] The paper is the first to formally formulate the vector set search problem (if not, then citations are required to be provided to the first work), yet only providing one concrete application of the vector set search problem (i.e., semnatic search). The reasons why we need to study the vector set search problem require further clarified. It seems to me that the DESSERT algorithm is actually an incremental work, with a specific focus for improving the running time of the ColBERT model. This severely limits the paper's contributions.\n\nW2. [Experiments] According to the paper, the proposed algorithm, DESSERT, is implemented in C++. However, the baseline method, the brute force algorithm, is implemented in Python using the PyTorch library. The reasonability and fairness of implementing two methods in different programming languages need further explanations. \n\nW3. [Presentations] The paper's presentation needs improvement. There exist several typos and undefined notations in the paper. Some concrete examples are given as follows. \n\n- Line 139: I believe the notation H refers to the hash set, which requires further clarification. \n\n- Line 30: citations are required to the literature on traditional single-vector near-neighbor search. \n\n- Algorithm 2: the output given in the pseudo code doesn't concur with the desciptions given in the main text (though they may refer to the same variable).  \n\n\n\n---\nAfter the rebuttal process, most of my concerns mentioned above have been sufficiently addressed. Thus, I raise my rating from 4 to 5.\n\nQUESTIONS:\nPlease refer to the Weaknesses given above, and provide further explanations, particularly for W1 and W2.\n\nLIMITATIONS:\nI do not see any potential negative societal impact in this paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the problem of set-to-set similarity calculation and retrieval, which is a problem with any downstream applications. While previous approaches inevitably perform a brute force similarity calculation over |Q| query vectors and |S| target vectors for each set comparison F(Q, S), this paper leverages random hashing functions to estimate similarity between query and target vectors to avoid this brute force search. Combined with highly space-optimized hash tables, and building on top of previous centroid-based candidate set filtering mechanisms, the proposed algorithm achieves notable speed improvements over existing methods at only a small cost to recall. Theoretical support is provided for the algorithm, along with experiments on synthetic, solvable data and the standard MS MARCO benchmark dataset.\n\nSTRENGTHS:\n- Very important: The proposed method cleverly utilizes randomness via an LSH based approach to estimate similarity and avoid brute force similarity calculation between sets of vectors, leading to notable improvements in speed over previous methods, at reasonable memory requirements for large scale settings, and with very limited negative impact on recall.\n- Important: The experiments are simple and clear.\n- Important: The choice of baselines seems appropriate. As the authors note, their method reaches points on the Pareto curve, w.r.t. latency, that are not reachable by baselines even though the baselines are nominally customizable for trading off latency vs. recall. \n- Important: The paper is clearly written.\n- Of some importance: Theoretical analysis is provided for the proposed method, although, as the authors note, they “assume sufficiently high relevance scores and large gaps in our theoretical analysis to identify the correct results.”\n\nWEAKNESSES:\n- Important: As noted by the authors, this work draws heavily upon LSH-based NN vector search methods that already use LSH methods to approximate exact similarity calculation. In my opinion, the extension to the set-to-set comparison setting is fairly straightforward. This is a drawback of the methodological contribution of the paper, in terms of novelty, though it may be entirely outweighed by the positive empirical performance of the approach over strong baselines.\n- Of some importance: The intro of the paper is written as if the paper proposes a generic framework for cross-query-vector aggregation (A2) and cross-target-vector aggregation steps (A1), but to my understanding, the only setting experimentally considered in the paper is that of A2 = weighted sum and A1 = max. This seems to be the main practical setting of interest in settings like passage retrieval, but I would not say this paper really broadens how people have been thinking about set-to-set comparison.\n\nQUESTIONS:\n- The paper argues that vector-level NN retrieval is clearly limited, but I think this assumes that one is really looking for a comparison using something like A2=weighted sum and A1=max. I think there are settings where one would be interested in A2=max and A1=max, as well as other variants. Are there problems besides passage retrieval that are better evaluated in terms of other combinations of A2 and A1 besides those considered in this paper?\n- The set-to-set comparison problem might also be viewed more generally as comparison of two distributions over R^d. When |Q| and |S| are large relative to d, these distributions might be accurately estimated and then distributional divergence measures deployed. What do you think the trade-offs are with a vector similarity-based approach? (And could the perspectives be unified based on specifying A2/A1 and sim?)\n- I think I never saw which specific LSH functions you used, besides using 16/32/64 of them. Are they all signed random projection to approximate cos-sim?\n- What does perfect recall mean on synthetic data? That you already know what the correct top-1 avg-max-sim “passage” is?\n- Can you increase the latency of your approach to reach the performance of PLAID?\n- In the next version of the appendix, could you include the results of your hyperparameter grid search?\n- Since you make a claim about Pareto improvement, it would be nice to include graphs visualizing the Pareto curves of the methods.\n- More of a comment, really: You might add “passage retrieval” for language modeling pretraining and finetuning to the list of potentially valuable applications for this kind of approach. Current methods use simple heuristics like embedding averaging across tokens to reduce the problem to vector-level NN search: https://arxiv.org/pdf/2112.04426.pdf.\n\nLIMITATIONS:\nYes, the limitation section was satisfactory.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a study focused on the problem of vector set search with vector set queries, which is a crucial subroutine for various web applications. The authors highlight the insufficiency of existing solutions in terms of speed. To address this, they propose a new approximate search algorithm called DESSERT. DESSERT is a versatile tool that offers robust theoretical guarantees and demonstrates excellent empirical performance. The authors integrate DESSERT into ColBERT, a highly optimized state-of-the-art semantic search method, and report a notable 2-5x speed improvement on the MSMarco passage ranking task.\n\nSTRENGTHS:\n1. The problem formulation is truly innovative. Vector set search is undeniably an essential and challenging task, with practical applicability in serving LLMs. \n\n2. The proposed methodologies are lucid and firmly grounded, ensuring the reproducibility of the research.  DESSERT uses hash tables for element-wise similarity search to build a data structure for vector set similarity search. Moreover, DESSERT taking advantage of hash tables in terms of query speed and performs an efficient query for similar sets.\n\n3. DESSERT has a detailed theoretical analysis in terms of inner and outer aggregation, which justify its choices of aggregation functions in the algorithm.\n\nWEAKNESSES:\nIt would be better to have a section that unifies the notation for algorithms and theory. Maybe a table would be better. \n\nOverall, I have a positive impression on the paper. If authors are willing to further polish the theory part of the paper, I'm willing to raise the score.\n\nQUESTIONS:\n1. How can DESSERT be applied to different similarity measures?\n\n2. How to intuitively explain the running time and memory? How do the terms $L$  and $T$ relate to the search quality?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-17"}

comments: SUMMARY:
This paper considers a nearest neighbor search problem where each point is a set of vectors, and the distance function is drawn from a general class of aggregation functions over the vectors. 

The approach presented here is based on the LSH algorithms, but since we're dealing with multiple vectors, the bounds behind LSH need to be re-derived. Assumptions used in order to allow or speed up the search include that the distance function obeys a certain Lipschitz smoothness property, and tools include inverted indices, sampling, and compressed tables. These are all fairly standard in the field.

STRENGTHS:
The query model introduced is interesting, and the authors make a good argument as to its utility. The techniques utilized are non-trivial, and constitute a contribution to the field of nearest neighbor search. The fact that the distance function is very general is also a plus.

WEAKNESSES:
The presentation can be somewhat improved. The authors actually put in significant work into this direction, but the presentation can be improved by adding a high-level overview of the approach before introducing the pseudo-code. This is a more effective approach than referring to code to explain an approach.

The authors should also add background on LSH and inverted indices, so that the paper can be more accessible to non-experts.

The explanation of the motivation behind TinyTable is garbled, and a clearer exposition is necessary. 

---------------------------------

In response to the author rebuttal, and in particular the improvement in presentation, I raised my score.

QUESTIONS:
None.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-18