SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We are interested in the signs of these partial derivatives. First examining  $\frac{\delta\gamma}{\delta s_{\mathrm{max}}}$ ,  $\tau > \alpha s_{\mathrm{max}} \Rightarrow \tau - \alpha s_{\mathrm{max}} > 0$ . Similarly,  $\alpha > \tau \Rightarrow \alpha - \tau > 0$  and  $s_{\mathrm{max}}(\alpha - \tau) = s_{\mathrm{max}}\alpha - s_{\mathrm{max}}\tau > 0$ . Finally,  $s_{\mathrm{max}} < 1 \Rightarrow \tau (1 - s_{\mathrm{max}}) = \tau - \tau s_{\mathrm{max}} > 0$ . Thus every term is positive and the entire fraction is positive. Next examining  $\frac{\delta\gamma}{\delta\tau}$ , by similar logic  $\alpha - \tau > 0$  and  $\tau - s_{\mathrm{max}}\tau > 0$  and  $\alpha s_{\mathrm{max}} - s_{\mathrm{max}}\tau > 0$ . For the ln, since  $\tau > \alpha s_{\mathrm{max}}$ ,  $\tau - s_{\mathrm{max}}\tau > \alpha s_{\mathrm{max}} - s_{\mathrm{max}}\tau$ , so the numerator is greater than the denominator and the ln is positive. Finally, since  $s_{\mathrm{max}} < 1$ ,  $s_{\mathrm{max}} - 1 < 0$ , and thus the entire fraction has a single negative term in the product, so it is negative.
This completes our lemma:  $\gamma$  is a strictly decreasing function of  $\tau$  and a strictly increasing function of  $s_{\mathrm{max}}$ . Since  $\tau$  is decreasing and has a leftward limit of 1 and a rightward limit of  $s_{\mathrm{max}}$ , all values for  $\gamma$  are in  $(s_{\mathrm{max}}, 1)$ .
First, we will make a substitution. We note that  $\gamma$  is a strictly decreasing function on this interval of  $\tau$  with range  $(s_{\max}, 1)$ . To see this, we will first make the following change of variables:
$$
\tau = \frac {\alpha (k + s)}{k + 1}
$$
for  $k \in (0, \infty)$ . This parameterizes  $\tau \in (s_{\max} \alpha, \alpha)$  as a weighted sum of  $s_{\max} \alpha$  and  $\alpha$ . Plugging in and simplifying, we have that
$$
\gamma = \left(\frac {s _ {\operatorname* {m a x}}}{k + s _ {\operatorname* {m a x}}}\right) ^ {\frac {k + s _ {\operatorname* {m a x}}}{k + 1}} (k + 1)
$$
This is a continuous function over  $k\in (0,\infty)$  and  $s_{\mathrm{max}}\in ()$
$$
P r [ \sigma (\hat {\mathbf {s}}) \leq \beta s _ {\max} - \Delta ] \leq 2 e ^ {- 2 L \Delta^ {2} / \beta^ {2}}
$$
Proof. We will prove this lemma with a chain of inequalities, starting with  $\operatorname{Pr}[\sigma(\hat{\mathbf{s}}) \leq \beta s_{\max} - \Delta]$ :
$$
\begin{array}{l} \Pr [ \sigma (\hat {\mathbf {s}}) \leq \beta s _ {\max } - \Delta ] \leq \Pr [ \beta \max  \hat {\mathbf {s}} \leq \beta s _ {\max } - \Delta ] \\ \leq \Pr [ \beta s _ {\max } ^ {\hat {}} \leq \beta s _ {\max } - \Delta ] \\ = \Pr \left[ \beta s _ {\max } - \beta s _ {\hat {\max }} ^ {\prime} \geq \Delta \right] \\ \leq \Pr \left[ \left| \beta s _ {\max } - \beta s _ {\hat {\max }} ^ {\prime} \right| \geq \Delta \right] = \Pr \left[ \left| \beta s _ {\hat {\max }} ^ {\prime} - \beta s _ {\max } \right| \geq \Delta \right] \\ \leq 2 e ^ {- 2 L \Delta^ {2} / \beta^ {2}} \\ \end{array}
$$
The explanations for each step are as follows:
1. Because  $\sigma(\hat{\mathbf{s}}) \geq \beta \max \mathbf{s}$ , we can replace  $\sigma(\hat{\mathbf{s}})$  with  $\beta \max \mathbf{s}$  and the probability will be strictly larger.
2. By the definition of the max operator, each individual  $\hat{s}_i \leq \max \hat{\mathbf{s}}$ , and in particular this is true for  $s_{\max}^*$  (the estimated similarity for the ground-truth maximum similarity vector). Thus, we have  $\beta s_{\max}^* \leq \beta \max \hat{\mathbf{s}}$ , so we can again apply a replacement to get a further upper bound.
3. Rearranging.
4. Because  $Pr[|a - b| \geq c] = Pr[a - b \geq c] + Pr[b - a \geq c]$
5.  $\hat{s}_{\mathrm{max}}$  is the sum of  $L$  Bernoulli trials with success probability  $s_{\mathrm{max}}$  and scaled by  $\beta / L$ . Thus, we can directly apply the Hoeffding ineuqliaty with  $L$  trials with success probability  $\frac{\beta s_{\mathrm{max}}}{L}$ .
Proof. For set  $S^{\star}$  to have the highest estimated score  $\hat{F}(Q, S^{\star})$ , we need all other sets to have lower scores. Our overall proof strategy will find a minimum  $L$  that upper bounds the probability that each inner aggregation of a set  $S \neq S^{*}$  is greater than  $\Delta + \alpha s_{j,\max}'$  and a minimum  $L$  that lower bounds the probability that the inner aggregation of  $S^{*}$  is less  $\beta s_{j,\max}' - \Delta$ . Finally, we will show that an  $L$  that is a maximum of these two values solves the search problem.
Upper Bound: We start with the upper bound on  $S_{i} \neq S^{*}$ : we have from Lemma 4.1.2 that
$$
\operatorname * {P r} [ \sigma (\hat {\mathbf {s}} _ {\mathbf {i}}, q _ {j}) \geq \alpha s _ {i, \max} + \Delta ] \leq m \gamma_ {i} ^ {L}
$$
$$
\gamma_ {i} = \left(\frac {\left(\Delta + \alpha s _ {i , \max}\right) \left(1 - s _ {i , \max}\right)}{s _ {i , \max} (\alpha - \left(\Delta + \alpha s _ {i , \max}\right))}\right) ^ {- \frac {\Delta + \alpha s _ {i , \max}}{\alpha}} \left(\frac {\alpha \left(1 - s _ {i , \max}\right)}{\alpha - \left(\Delta + \alpha s _ {i , \max}\right)}\right)
$$
and  $\gamma_{i}\in (0,1)$ . To make our analysis simpler, we are interested in the maximum  $\gamma_{\mathrm{max}}$  of all these  $\gamma_{i}$  as a function of  $\Delta$ , since then all of these bounds will hold with the same  $\gamma$ , making it easy to solve for  $L$ . Since  $\lim_{s_{\mathrm{max}}\searrow 0}\gamma = 0$  and  $\lim_{s_{\mathrm{max}}\nearrow 1 - \Delta} = 1 - \Delta /\alpha$ , there must be some  $\gamma_{\mathrm{max}}\in (1 - \Delta /\alpha ,1)$  that maximizes this expression over any  $s_{\mathrm{max}}$ . This exact maximum is hard to find analytically, but we are guaranteed that it is less than 1 by Lemma 4.1.2. We will use the term  $\gamma_{\mathrm{max}}$  in our analysis, since it is data dependent and guaranteed to be in the range  $(0,1)$ . We also numerically plot some values of  $\gamma_{\mathrm{max}}$  here with  $\alpha = 1$  to give some intuition for what the function looks like over different  $\Delta$ ; we note that it is decreasing in  $\Delta$  and approximates a linear function for  $\Delta >> 0$ .
To hold with the union bound over all  $N - 1$  target sets and all  $m_q$  query vectors with probability  $\frac{\delta}{2}$ , we want the probability that our bound holds on a single set and query vector to be less than  $\frac{\delta}{2(N - 1)m_q}$ . We find that this is true with  $L \geq \log \frac{2(N - 1)m_q m}{\delta} \log \left( \frac{1}{\gamma_{\max}} \right)^{-1}$  for any  $q_j$  and  $S_i$ :
$$
\operatorname * {P r} [ \sigma (\hat {\mathbf {s _ {i}}}, q _ {j}) \geq \alpha s _ {i, \max } + \Delta ] \leq m \gamma_ {i} ^ {L} \leq m \gamma_ {\max } ^ {L} \leq m (\gamma_ {\max }) ^ {\log \frac {2 (N - 1) m _ {q} m}{\delta} \log (\frac {1}{\gamma_ {\max }}) ^ {- 1}} = \frac {\delta}{2 (N - 1) m _ {q}}
$$
${}^{3}L$  additionally depends on the data-dependent parameter  $\Delta$  ,which we elide in the asymptotic bound; see the proof in the appendix for the full expression for  $L$  .
Lower Bound We next examine the lower bound on  $S^*$ : we have from Lemma 4.1.3 that
$$
\operatorname * {P r} [ \sigma (\hat {\mathbf {s}} ^ {*}, q _ {j}) \leq \beta s _ {*, \max} - \Delta ] \leq 2 e ^ {- 2 L \Delta^ {2} / \beta^ {2}}
$$
To hold with the union bound over all  $m_q$  query vectors with probability  $\frac{\delta}{2}$ , we want the probability that our bound holds on a single set and query vector to be less than  $\frac{\delta}{2m_q}$ . We find that this is true with  $L \geq \frac{\log\left(\frac{4m_q}{\delta}\right)\beta^2}{2\Delta^2}$  for any  $q_j$ :
$$
\operatorname * {P r} [ \sigma (\hat {\mathbf {s}} ^ {*}, q _ {j}) \leq \beta s _ {*, \max} - \Delta ] \leq 2 e ^ {- 2 L \Delta^ {2} / \beta^ {2}} \leq 2 e ^ {- 2 \frac {\log (\frac {4 m _ {q}}{\delta}) \beta^ {2}}{2 \Delta^ {2}}} \Delta^ {2} / \beta^ {2} = \frac {\delta}{2 m _ {q}}
$$
Putting it Together
$$
L = \max  \left(\frac {\log \frac {2 (N - 1) m _ {q} m}{\delta}}{\log (\frac {1}{\gamma_ {\operatorname* {m a x}}})}, \frac {\log (\frac {4 m _ {q}}{\delta}) \beta^ {2}}{2 \Delta^ {2}}\right)
$$
Then the upper and lower bounds we derived in the last two sections both apply. Let  $\mathbb{1}$  be the random variable that is 1 when the  $m*m_q*(N-1)$  upper bounds and the  $m_q$  lower bounds hold and that is 0 otherwise. Consider all sets  $S_i \neq S^*$ . Then the probability we solve the Vector Set Search Problem from Definition 1.1 is equal to the probability that all  $\forall_i$ ,  $(\hat{F}(Q, S^*) - \hat{F}(Q, S_i) > 0)$ . We now lower bound this probability:
$$
\begin{array}{l} \operatorname * {P r} \left(\forall_ {i} (\hat {F} (Q, S ^ {*}) - \hat {F} (Q, S _ {i}) > 0)\right) \\ = \Pr \left(\forall_ {i} \left(\frac {1}{m _ {q}} \sum_ {j = 1} ^ {m _ {q}} w _ {j} \sigma \left(\hat {\mathbf {s}} ^ {*}, q _ {j}\right) - \frac {1}{m _ {q}} \sum_ {j = 1} ^ {m _ {q}} w _ {j} \sigma \left(\hat {\mathbf {s}} _ {i}, q _ {j}\right) > 0\right)\right) \\ = \Pr \left(\forall_ {i} \left(\sum_ {j = 1} ^ {m _ {q}} w _ {j} \left(\sigma \left(\hat {\mathbf {s}} ^ {*}, q _ {j}\right) - \sigma \left(\hat {\mathbf {s}} _ {i}, q _ {j}\right)\right) > 0\right)\right) \\ = \Pr \left(\forall_ {i} \left(\sum_ {j = 1} ^ {m _ {q}} w _ {j} \left(\sigma \left(\hat {\mathbf {s}} ^ {*}, q _ {j}\right) - \sigma \left(\hat {\mathbf {s}} _ {i}, q _ {j}\right)\right) > 0 \mid \mathbb {1} = 1\right)\right) \Pr (\mathbb {1} = 1) \quad \Pr (A) \geq \Pr (A \wedge B) \\ \geq \Pr \left(\forall_ {i} \left(\sum_ {j = 1} ^ {m _ {q}} w _ {j} \left(\beta s _ {*, \max } - \Delta - \left(\alpha s _ {i, \max } + \Delta\right)\right) > 0\right)\right) \Pr (\mathbb {1} = 1) \quad \text {B o u n d s h o l d o n} \mathbb {1} = 1 \\ = \Pr \left(\forall_ {i} \left(\sum_ {j = 1} ^ {m _ {q}} w _ {j} \left(\beta s _ {*, \max } - \alpha s _ {i, \max }\right) > 2 \Delta \sum_ {j = 1} ^ {m _ {q}} w _ {j}\right)\right) \Pr (\mathbb {1} = 1) \\ = \Pr \left(\forall_ {i} \left(m _ {q} \left(B ^ {*} - B _ {i}\right) > 2 \Delta \sum_ {j = 1} ^ {m _ {q}} w _ {j}\right)\right) \Pr (\mathbb {1} = 1) \quad \text {D e f i n i t i o n} B ^ {*}, B _ {i} \\ \geq \Pr \left(\forall_ {i} \left(m _ {q} \left(B ^ {*} - B ^ {\prime}\right) > 2 \Delta \sum_ {j = 1} ^ {m _ {q}} w _ {j}\right)\right) \Pr (\mathbb {1} = 1) \quad \text {D e f i n i t i o n} B ^ {\prime} \\ \geq \Pr \left(\forall_ {i} \left(3 m _ {q} \Delta > 2 \Delta \sum_ {j = 1} ^ {m _ {q}} w _ {j}\right)\right) \Pr (\mathbb {1} = 1) \quad \text {D e f i n i t i o n} \Delta \\ \geq \Pr \left(\forall_ {i} \left(3 m _ {q} \Delta > 2 m _ {q} \Delta\right)\right) \Pr (\mathbb {1} = 1) \quad w _ {j} \leq 1 \\ = 1 * (\mathbb {1} = 1) \quad \Delta > 0 \\ \end{array}
$$
$$
\begin{array}{l} = 1 - (\mathbb {1} = 0) \\ \geq 1 - \left(m * m _ {q} * (N - 1) * \frac {\delta}{2 (N - 1) m _ {q}} + \frac {\delta}{2 m _ {q}} * m _ {q}\right) = 1 - \delta \quad \text {U n i o n b o u n d} \\ \end{array}
$$
and thus DESSERT solves the Vector Set Search Problem with this choice of  $L$ . Finally, we can now examine the expression for  $L$  to determine its asymptotic behavior. Dropping the positive data dependent constants  $\frac{1}{\gamma_{\mathrm{max}}}, \frac{1}{2\Delta^2}$ , and  $\beta^2$ , the left term in the max for  $L$  is  $O(\log (\frac{Nm_qm}{\delta}))$  and the right term in the max is  $O(\log (\frac{m_q}{\delta}))$ , and thus  $L = O\left(\log (\frac{Nm_qm}{\delta})\right)$ .
$$
O \left(m _ {q} \log \left(N m _ {q} m / \delta\right) d + m _ {q} N \log \left(N m _ {q} m / \delta\right)\right)
$$
Proof. If we suppose that each call to the hash function  $f_{t}$  is  $O(d)$ , the runtime of the algorithm is
$$
O \left(n L d + \sum_ {i = 0} ^ {n - 1} \sum_ {k = 0} ^ {N - 1} \sum_ {t = 0} ^ {L - 1} \left| M _ {k, t, f _ {t} \left(q _ {j}\right)} \right|\right)
$$
To bound this quantity, we use the sparsity assumption we made in the theorem: no set  $S_{i}$  contains too many elements that are very similar to a single query vector  $q_{j}$ . Formally, we require that
$$
\left| \mathcal {D} [ i ] _ {t, h} \right| <   T \quad \forall i, t, h
$$
for some positive threshold  $T$ . With this assumption, the runtime of Algorithm 2 is
$$
O \left(m _ {q} L d + m _ {q} N L T\right)
$$
Plugging in the  $L$  we found in our previous theorem, and treating  $T$  as data dependent constant, we have that the runtime of Algorithm 2 is
B Hyperparameter Settings
Settings for DESSERT corresponding to the first row in the left part of Table 2, where DESSERT was optimized for returning 10 documents in a low latency part of the Pareto frontier:
hashes_per_table (C) = 7  
num_tables (L) = 32  
filter_k = 4096  
filterprobe = 1
Settings for DESSERT corresponding to the second row in the left part of Table 2, where DESSERT was optimized for returning 10 documents in a high latency part of the Pareto frontier:
hashes_per_table (C) = 7  
num_tables (L) = 64  
filter_k = 4096  
filterprobe = 2
Settings for DESSERT corresponding to the first row in the right part of Table 2, where DESSERT was optimized for returning 1000 documents in a low latency part of the Pareto frontier:
hashes_per_table  $(\mathbb{C}) = 6$
num_tables (L) = 32
filter_k = 8192
filter Probe = 4
Settings for DESSERT corresponding to the second row in the right part of Table 2, where DESSERT was optimized for returning 1000 documents in a high latency part of the Pareto frontier:
hashes_per_table (C) = 7
filter_k = 16384
filter Probe  $= 4$
Intuitively, these parameter settings make sense: increase the initial filtering size and the number of total hashes for higher accuracy, and increase the initial filtering size for returning more documents (1000 vs. 10).
C Background on Locality-Sensitive Hashing and Inverted Indices for Similarity Search
Here, we offer a refresher on using locality-sensitive hashing for similarity search with a basic inverted index structure.
Consider a set of distinct vectors  $X = \{x_{1},\ldots ,x_{N}\}$  where each  $x_{i}\in \mathbb{R}^{d}$ . A hash function  $h$  with a range  $m$  maps each  $x_{i}$  to an integer in the range  $[1,m]$ . Two vectors  $x_{i}$  and  $x_{j}$  are said to "collide" when  $h(x_{i}) = h(x_{j})$ .
As a warmup, we will first consider the case of a hash function  $h$  drawn from a set of universal hash functions  $H$ . Under such a function, if  $i \neq j$ ,  $p(h(x) = h(y)) = \frac{1}{m}$ ; such families exist in practice [7]. We can build an inverted index using this hash function by mapping each hash value in  $[1, m]$  to the set of vectors  $X_v$  that have this hash value. Then, given a new vector  $y$ , we can query the inverted index with  $v = h(y)$ . We can see that  $y \in X$  iff  $y \in X_v$ . Such an index is in a sense solving a search problem, if we only care about finding exact duplicates of our search query. Additionally, we can solve the nearest neighbor problem with this index in time  $O(N)$ , by going to every bucket and checking the distance of a query against every vector in the bucket.
Now, in a similar way as in the universal case, let  $h$  to be drawn from a family of locality-sensitive hash functions  $H$ . At a high level, instead of mapping vectors uniformly to  $[1,m]$ ,  $h$  maps vectors that are close together to the same hash value more often. Formally, if we define a "close" threshold  $r_1$ , a "far" threshold  $r_2$ , a "close" probability  $p_1$ , and a "far" probability  $p_2$ , with  $p_1 > p_2$  and  $r_1 < r_2$ , then we say  $H$  is  $(r_1,r_2,p_1,p_2)$ -sensitive if
$$
d (x, y) <   r _ {1} \Longrightarrow P r (h (x) = h (y)) > p _ {1}
$$
$$
d (x, y) > r _ {2} \Longrightarrow P r (h (x) = h (y)) <   p _ {2}
$$
where  $d$  is a distance metric. See [17] for the origin of locality-sensitive hashing and this definition. Intuitively, if we build an inverted index using  $h$  in the same way as before, it now seems we have a strategy to solve the (approximate) nearest neighbor problem more efficiently: given a query  $q$ , only search for nearest neighbors in the bucket  $h(q)$ , since each of these points  $x$  likely has  $d(q,x) < r_1$ . However, this strategy has a problem: with our definition, even a close neighbor might not be a collision with probability  $(1 - p_1)$ . Thus, we can repeat our inverted index  $L$  times with different  $h_i$  drawn independently from  $H$ , such that our probability of not finding a close neighbor in any bucket is  $(1 - p_1)^L$ . FALCONN [2] is an LSH inverted index algorithm that uses this basic idea, along with concatenation and probing tricks, to achieve an asymptotically optimal (and data-dependent sub-linear) runtime; see the paper and associated code repository for more details.
One final note is that in practice, most LSH families satisfy a much stronger condition than the above. Consider a similarity function  $\mathrm{sim} \in [0,1]$ , where  $s(x,y) = 1 \Rightarrow x = y$ . As  $x$  and  $y$  get more dissimilar (e.g., their distance increases according to some distance metric),  $s(x,y)$  decreases. For most LSH families, there exists an explicit similarity function that their collision probability satisfies, such that  $p(h(x) = h(y)) = \mathrm{sim}(x,y)$ . Such LSH families exist for most common similarity
functions, including cosine similarity (signed random projections) [8], Euclidean similarity ( $p$ -stable projections) [11], and Jaccard similarity (minhash or simhash) [6]. Following [13, 9, 10, 27, 24, 31], in our work, we use LSH families with this explicit similarity description to provide tight analyses and strong guarantees for similarity-search algorithms.