SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Suppose a query  $q$  is associated with  $N_{q\oplus}$  relevant corpus items (as judged by humans). Suppose the system provides a ranking over all  $N$  corpus items, and the relevant items occur at ranks  $r_1,\dots ,r_{N_{q\oplus}}$ . Then AP for query  $q$  is defined as  $(1 / N_{q\oplus})\sum_{i = 1}^{N_{q\oplus}}(i / r_i)$ . This is because, up to position  $r_i$ , we have seen  $i$  relevant items, which means we can shorthand  $i / r_i$  as  $\mathrm{prec}@i$  (precision at  $i$ ). We can rewrite the sum as  $\frac{1}{N_{q\oplus}}\sum_{r = 1}^{N}\mathrm{prec}@\boldsymbol {r}\times \mathrm{rel}@\boldsymbol {r}$ , where  $N$  is the size of the whole corpus, and  $\mathrm{rel}@\boldsymbol {r}$  is the (0/1) relevance of the item at rank  $r$ . In case the retrieval algorithm does not assess all  $N$  corpus items, but stops with the best  $L$  hash buckets, which contain, say,  $N_q'$  items, we should use the following formula for AP:  $\frac{1}{N_{q\oplus}}\sum_{r = 1}^{N_q'}\mathrm{prec}@\boldsymbol {r}\times \mathrm{rel}@\boldsymbol {r}$ . Note that we should still divide by  $N_{q\oplus}$ , otherwise an algorithm that maps the query to a densely relevant but small bin, which fails to retrieve most relevant items, might be rewarded in an unfair manner.
E.7 Top-10 score measurement
In Appendix F, we provide additional experiments where we compare FOURIERHASHNET with all the baselines not only in terms of MAP, but also in terms of the Top-10 score. We use the sum of Top-10 scores normalized in [0, 1] via the sigmoid transformation used in Eq. (15):  $\mathrm{Top - }10(q) = \sum_{x\in \mathrm{Top - }10(N_q^{\prime})}\sigma (-d(q,x))$ . Any hashing protocol is expected to retrieve the corpus items, which have the highest similarity scores with respect to any given query. The Top-10 score evaluates it independently of how the retrieved items match with true relevant items. Therefore, the Top-10 scores provide an evaluation mechanism that is independent of the gold relevance labels and solely relies on the scores dictated by the trained embeddings. This offers a valuable means of assessing performance without being influenced by subjective human judgments.
E.8 Licenses
We utilize a publicly available pre-trained sentence transformer model [41], which is licensed under the Apache License 2.0. Additionally, we employ the DrHash toolkit [58] for various implementations of the baseline Weighted Minhash (WMH) algorithms. The DrHash toolkit is publicly available under the MIT License. We duly acknowledge the original authors of the baseline methods in our citations.
F.1 Applying baseline LSHs on hinge distance guided embeddings
In continuation of the results reported in Figure 3, in Figure 8, we present the complete view as well as the zoomed versions of the trade-off curves for all datasets.
Figure 8: Trade-off between average query time and accuracy (MAP and Top-10 scores) for MSWEB and MSNBC datasets (First two rows: complete view across full time axis, last two rows: Zoomed version of first two rows until the average query time there is  $\geq 10\mathrm{X}$  speedup compared to exhaustive search). We apply different LSH methods on hinge distance guided embeddings similar to Figure 3, then use the hinge distance to finally rank the retrieved items.
Beyond the observations made in Figure 3, we make the following additional observations.
(1) The complete view for both Top-10 score and MAP score, clearly demonstrates FLORA's high sensitivity to hyperparameter tuning. FLORA is seen to have the highest variance in scores for any given time budget across all the baselines. In terms of Top-10 score, while FLORA is marginally ahead of FOURIERHASHNET in a few instances in the MSWEB datasets, it is significantly outperformed by FOURIERHASHNET in the MSNBC datasets. This may be due to the significantly higher average query selectivity in the MSNBC datasets.
(2) In terms of Top-10 score, FOURIERHASHNET achieves the maximum possible value  $4 \times$  faster than the nearest competitor RH+Hinge, in MsWEB-1 and MsWEB-2. In terms of MAP score,
RH+Hinge and WMH+Hinge achieve a maximum MAP of 0.5 in MSWEB-1 and MSWEB-2, and 0.62 in MSNBC-1 and MSNBC-2. However, FOURIERHASHNET achieves the same MAP values  $1.33 \times$  faster in MSNBC-1 and MSNBC-2, and  $2 \times$  faster in MSNBC-1 and MSNBC-2.
(3) Interestingly, the gap between FOURIERHASHNET and RH+Hinge seems to widen for MsWEB-2, when compared to MsWEB-1. This is possibly due to the presence of several queries in MsWEB-1, which have  $\geq 300$  relevant corpus items. This affords RH+Hinge a greater opportunity to fetch high scoring items, which is not the case in MsWEB-2.
F.2 Ablation study on hashcode training
In continuation of the results reported in Figure 5, in Figure 9, we present the complete view as well as the zoomed versions of the trade-off curves for all datasets.
(b) MSNBC-1
(c) MSWEB-2
Figure 9: Trade-off between average query time and accuracy (MAP and Top-10 scores) for MSWEB and MSNBC datasets (First two rows: complete view across full time axis, last two rows: Zoomed version of first two rows until the average query time there is  $\geq 10\mathrm{X}$  speedup compared to exhaustive search). We compare FHash (UNTRAINED) against the untrained versions of RH+Hinge and DP-RH+Hinge, as well as against FOURIERHASHNET.
Beyond the observations made in Figure 5, we make the following additional observations.
(1) In terms of both Top-10 score and MAP score, FHash (UNTRAINED) clearly outperforms both  $\mathrm{RH + Hinge}$  and DP-RH+Hinge, across all four datasets. This strongly highlights the advantage of our
Fourier feature generation method.
(2) In every setup, FOURIERHASHNET enables a wider range of options for accuracy score vs retrieval time trade-off.
F.3 Ablation study on collision minimizer (14)
(c) MSWEB-2
(d) MSNBC-2
Figure 10: Trade-off between mean query time and accuracy (MAP and Top-10 scores) for MSWEB and MSNBC datasets (First two rows: complete view across full time axis, last two rows: Zoomed version of first two rows until the mean query time there is  $\geq 10\mathrm{X}$  speedup compared to exhaustive search). We compare our hashcode training loss  $\mathrm{loss}(Q,X|\mathbf{W})$  (14), against a variant which replaces the collision minimizer component  $\Delta_{1}$  with a decorrelation loss  $\sum_{x\neq y}|\tanh (\pmb {W}\pmb {z}_x)^\top \tanh (\pmb {W}\pmb {z}_y)|$
In continuation of the results reported in Figure 6, we present the complete view as well as the zoomed versions of the trade-off curves for all datasets. Beyond the observations made in Figure 6, we make the following additional observations.
(1) In MSWEB-2, the alternative variant shows a sudden plunge in MAP performance trade-off at around 0.1 seconds. This type of discontinuous drop is not observed in any of our cases.
(2) In MSWEB-1, there is a variation of 0.1 MAP at around 0.05 seconds. Such high variability is not observed for any of our trade-off curves.
F.4 Identifying best performing Weighted Minhash algorithm for our datasets
For implementation of baseline Weighted Minhash (WMH) algorithm, we use the best performing WMH implementation available in the DrHash toolkit [58] to obtain the hashcodes. We compare
(c) MSWEB-2
Figure 11: We compare performance of Weighted Minhash variations, in terms of trade-off between mean query time and accuracy (MAP and Top-10 scores) for MSWEB and MSNBC datasets, until the query time there is  $\geq 10\mathrm{X}$  speedup compared to exhaustive search.
across the 8 available baselines in the toolkit: minhash [7], chum [12], icws [20], pcws [56], licws [28], ccws [55], i2cws [57] and gollapudi2 [16].
We make the following observations.
(1) Across all four of our datasets, for both MAP and Top-10 score, the top 3 performers are minhash, gollapudi2 and chum. The remaining algorithms are often significantly worse in performance, as can be seen for MAP in all 4 datasets and for Top-10 in MSNBC-2.
(2) Among the top 3 performers, chum is seen to be the best perform in terms for both MAP and Top-10 score, in MSWEB-2, MSNBC-1 and MSNBC-2. In MSWEB-1, chum is tied with minhash and gollapudi2 for the top position.
Driven by these observations, we choose chum as the representative baseline for WMH in our experiments.
F.5 Effect of  $M$ , number of samples of  $\omega$  on FOURIERHASHNET performance
(b) MSWEB-1
Figure 12: Effect of  $M$ , number of  $\omega$  samples, on the trade-off between mean query time and accuracy (MAP and Top-10 scores) for MsWEB datasets.
Here we check the impact of varying the number of samples  $(M)$  for  $\omega$ . We consider three different values of  $M$ , i.e., 10, 50 and 100, for generating the fourier features  $\mathbf{F}_q(\iota \omega^{1..M})$  and  $\mathbf{F}_x(\iota \omega^{1..M})$  which are then fed into the neural networks  $\phi_q$  and  $\phi_x$  for learning the transformed Fourier representations  $\mathbf{z}_q$  and  $\mathbf{z}_x$ , using the BCE loss specified in Eq. (12). Finally, we train the random hyperplanes  $\mathbf{W}$  and check the MAP and Top-10 score performances for the three variations - FOURIERHASHNET(10), FOURIERHASHNET(50) and FOURIERHASHNET(100). We observe that the final performance trade-off of both MAP and Top-10 scores, remains roughly the same across all three variants. This shows that trainable Fourier transformation is able to compensate for the quality of Monte Carlo approximations affected by the number of  $\omega$  samples  $M$ .
Next, we investigate how well the MC estimates of the Fourier features approximate the dominance similarity function  $\sin (q,x)$ . Here, we set the dimension of  $\pmb{q}$  and  $\pmb{x}$  as  $K = 1$ . We set  $T = 20$  and we sample  $\pmb{q},\pmb{x} \sim \mathrm{Unif}[-20,20]$ . Finally, we compute  $\widehat{\operatorname{sim}}_M(q,x) = \frac{||\pmb{F}_\bullet(\iota\omega^{1..M})||^2}{M}\cos(\pmb{F}_q(\iota\omega^{1..M}),\pmb{F}_x(\iota\omega^{1..M}))$  and measure the variation of  $\epsilon_{\mathrm{sim}} = \mathbb{E}_{\pmb{q},\pmb{x} \sim \mathrm{Unif}[-20,20]}[||\sin (q,x) - \widehat{\operatorname{sim}}_M(q,x)||]$  with  $M$ , the number of samples of  $\omega$ . Figure 13 summarizes the results, which shows as  $M$  increases,  $\epsilon_{\mathrm{sim}}$  decreases.
Figure 13: Variation of  $\epsilon_{\mathrm{sim}} = \mathbb{E}_{\boldsymbol{q},\boldsymbol{x}\sim \mathrm{Unif}[-20,20]}\left[\| \mathrm{sim}(q,x) - \widehat{\mathrm{sim}}_M(q,x)\|\right]$  with  $M$ , the number of  $\omega$  samples.
F.6 Applying baseline LSHs on hinge distance guided embeddings with noisy labels
In certain applications, the accuracy of ground truth labels can be compromised by noise or subjective human judgments of relevance. We evaluate the performance of FOURIERHASHNET and the baselines in a noisy label setup to test its robustness.
Starting with the hinge distance guided embeddings, we initially rank the corpus items based on their dominance similarity scores. Subsequently, we intentionally flip the labels of the bottom-ranking  $10\%$  of positive labels to negative, and an equal number of highest ranked negatively labeled items to positive. This simulation reflects a plausible scenario since the lowest ranked positive items and the highest ranked negative items are particularly susceptible to misclassification in real-world settings. Furthermore, this approach ensures that the average query selectivity for each dataset remains unchanged.
As before, we apply different the LSH methods on hinge distance guided embeddings, viz, RH+Hinge, DP-RH+Hinge, WMH+Hinge, FLORA and FOURIERHASHNET; and, then use the hinge distance to finally rank the retrieved items.
Figure 14: Trade-off between average query time and accuracy (MAP and Top-10 scores) for MSWeb and MSNBC datasets (First two rows: complete view across full time axis, last two rows: Zoomed version of first two rows until the average query time there is  $\geq 10\mathrm{X}$  speedup compared to exhaustive search). We apply different LSH methods on hinge distance guided embeddings similar to Figure 8, and then use hinge distance to rank retrieved items. Evaluations are conducted using noisy labels.
(1) In terms of MAP score, FOURIERHASHNET continues to outperform all other baselines across all four datasets. Furthermore, when comparing the performance in the presence of noise, as depicted in Figure 14, to the corresponding results obtained in the noiseless setting illustrated in Figure 8, we observe that FOURIERHASHNET outperforms all its competitors by a significantly higher margin in the presence of noise.
(2) In terms of Top-10 score, we note that the results in Figure 14 for the noisy setup are identical to the results presented in the noiseless setting shown in Figure 8. This observation indicates that the evaluation based on Top-10 score is unaffected by label noise. This supports the argument made in Appendix E.7 that Top-10 score evaluation enables a more subjective assessment of performance, focusing on the quality of the embeddings themselves.
F.7 Comparison with graph based indexing methods
Here, we compare the performance of FOURIERHASHNET with a representative graph based indexing method. We use the Hierarchical Navigable Small Worlds (HNSW) implementation of Hnswlib [33]. We extend the SpaceInterface class of nmslib/Hnswlib to implement HNSW for hinge distance. In order to track the number of distance computations performed by HNSW during retrieval, we used a counter inside fstdistfunc_. We count the number of distance computations as a surrogate for real time, to avoid non-determinism in measurements and low-level implementation differences. We search across different values of M, ef and ef_construction, and track the number of distance computations against corresponding MAP values.
#calls to fstdistfunc_	MAP	Method
839	0.17	HNSW
1162	0.53	FOURIERHASHNET
1549	0.43	HNSW
1668	0.51	HNSW
1846	0.58	FOURIERHASHNET
2578	0.59	HNSW
3529	0.71	HNSW
3926	0.72	FOURIERHASHNET
4773	0.77	HNSW
Table 15: Number of distance computations and the corresponding MAP values on MSWEB-1 for HNSW and FOURIERHASHNET.
Table presents our study on MSWEB-1 dataset with 10734 corpus items. We observe that FOURIER-HASHNET LSH has an edge over HNSW in the regime of fewer distance computations, with a MAP of 0.53 using 1162 distance computations. However, when allowed more distance computations, HNSW outperforms FOURIERHASHNET with a MAP of 0.71 in 3528 computations, and a MAP of 0.77 in 4773 computations.