SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Small Scale These experiments are motivated from common use cases of Chamfer distance in the computer vision and NLP domains. In our small scale experiments, we use two different datasets: (a) the ShapeNet dataset, a collection of point clouds of objects in three dimensions [10]. ShapeNet is a common benchmark dataset frequently used in computer graphics, computer vision, robotics and Chamfer distance is a widely used measure of similarity between different ShapeNet point clouds [10]. (b) We create point clouds of words from text documents from [16]. Each point represents a word embedding obtained from the word-to-vec model of [19] in  $\mathbb{R}^{300}$  applied to the Federalist Papers corpus. As mentioned earlier, a popular relaxation of the common Earth Mover Distance is exactly the (weighted) version of the Chamfer distance [16, 7].
Since ShapenNet is in three dimensions, we implement nearest neighbor queries using KD-trees to accelerate the brute force baseline as KD-trees can perform exact nearest neighbor search quickly in small dimensions. However, they have runtime exponential in dimension meaning they cannot be used for the text embedding dataset, for which we use a standard naive brute force computation. For both these datasets, we implement our algorithms using Python 3.9.7 on an M1 MacbookPro with 32GB of RAM. We also use an efficient implementation of KD trees in Python and use Numpy and Numba whenever relevant. Since the point clouds in the dataset have approximately the same  $n$  value, we compute the symmetric version  $\mathrm{CH}(A,B) + \mathrm{CH}(B,A)$ . For these experiments, we use the  $\ell_1$  distance function.
Outliers This experiment is meant to showcase the robustness of our algorithm. We consider two point clouds,  $A$  and  $B$ , each sampled from Gaussian points in  $\mathbb{R}^{100}$  with identity covariance. Furthermore, we add an "outlier" point to  $A$  equal to  $0.5n \cdot \mathbf{1}$ , where  $\mathbf{1}$  is the all ones vector.
This example models scenarios where the distances from points in  $A$  to their nearest neighbors in  $B$  vary significantly, and thus uniform sampling might not accurately account for all distances, missing a small fraction of large ones.
Large Scale The purpose of these experiments is to demonstrate that our method scales to datasets with billions of points in hundreds of dimensions. We use two challenging approximate nearest neighbor search datasets: DEEP1B [9] and Microsoft Turing-ANNS [21]. For these datasets, the set  $A$  is the query data associated with the datasets. Due to the asymmetric sizes, we compute  $\mathrm{CH}(A,B)$ . These datasets are normalized to have unit norm and we consider the  $\ell_2$  distance function.
These datasets are too large to handle using the prior configurations. Thus, we use a proprietary in-memory parallel implementation of the SimHash algorithm, which is an  $\ell_2$  LSH family for normalized vectors according to Definition 2.2 [11], on a shared virtual compute cluster with 2x64 core AMD Epyc 7763 CPUs (Zen3) with 2.45Ghz - 3.5GHz clock frequency, 2TB DDR4 RAM and 256 MB L3 cache. We also utilize parallization on the same compute cluster for naive brute force search.
(a) ShapeNet
(b) Federalist Papers
(c) Gaussian Points
(d) DEEP
(e) Turing
(a) ShapeNet
Figure 4: Runtime experiments. We set the number of samples for uniform and importance sampling such that the relative errors of their respective approximations are similar.
Figure 3: Sample complexity vs relative error curves.
(b) Federalist Papers
Small Scale First we discuss configuring parameters. Recall that in our theoretical results, we use  $\mathcal{O}(\log n)$  different scales of the LSH family in CrudeNN. CrudeNN then computes (over) estimates of the nearest neighbor distance from points in  $A$  to  $B$  (in near linear time) which is then used
(a) DEEP
(b) Turing
Figure 5: The figures show sample complexity vs relative error curves as we vary the number of LSH data structures and window sizes. Each curve maps  $k \times W$  where  $k$  is the number of LSH data structures we use to repeatedly hash points in  $B$  and  $W$  is the window size, the number of points retrieved from  $B$  that hash closest to any given  $a$  at the smallest possible distance scales.
for importance sampling by Chamfer-Estimate. Concretely for the  $\ell_1$  case, this the LSH family corresponds to imposing  $\mathcal{O}(\log n)$  grids with progressively smaller side lengths. In our experiments, we treat the number of levels of grids to use as a tuneable parameter in our implementation and find that a very small number suffices for high quality results in the importance sampling phase.
Figure 6 (b) shows that only using 3 grid levels is sufficient for the crude estimates  $D_{a}$  to be within a factor of 2 away from the true nearest neighbor values for the ShapeNet dataset, averaged across different point clouds in the dataset. Thus for the rest of the Small Scale experiments, we fix the number of grid levels to be 3.
Figure 3 (a) shows the sample complexity vs accuracy trade offs of our algorithm, which uses importance sampling, compared to uniform sampling. Accuracy is measured by the relative error to the true value. We see that our algorithm possesses a better trade off as we obtain the same relative error using only 10 samples as uniform sampling does using  $50+$  samples, resulting in at least a  $5\mathbf{x}$  improvement in sample complexity. For the text embedding dataset, the performance gap between our importance sampling algorithm and uniform sampling grows even wider, as demonstrated by Figure 3 (b), leading to  $>10\mathbf{x}$  improvement in sample complexity.
In terms of runtimes, we expect the brute force search to be much slower than either importance sampling and uniform sampling. Furthermore, our algorithm has the overhead of first estimating the values  $D_{a}$  for  $a \in A$  using an LSH family, which uniform sampling does not. However, this is compensated by the fact that our algorithm requires much fewer samples to get accurate estimates.
Indeed, Figure 4 (a) shows the average time of 100 Chamfer distance computations between randomly chosen pairs of point clouds in the ShapeNet dataset. We set the number of samples for uniform sampling and importance sampling (our algorithm) such that they both output estimates with (close to)  $2\%$  relative error. Note that our runtime includes the time to build our LSH data structures. This means we used 100 samples for importance sampling and 500 for uniform. The brute force KD Tree algorithm (which reports exact answers) is approximately  $5\mathrm{x}$  slower than our algorithm. At the same time, our algorithm is  $50\%$  faster than uniform sampling. For the Federalist Papers dataset (Figure 4 (b)), our algorithm only required 20 samples to get a  $2\%$  relative error approximation, whereas uniform sampling required at least 450 samples. As a result, our algorithm achieved  $2\mathrm{x}$  speedup compared to uniform sampling.
Outliers We performed similar experiments as above. Figure 3 (c) shows the sample complexity vs accuracy trade off curves of our algorithm and uniform sampling. Uniform sampling has a very large error compared to our algorithm, as expected. While the relative error of our algorithm decreases smoothly as the sample size grows, uniform sampling has the same high relative error. In fact, the relative error will stay high until the outlier is sampled, which typically requires  $\Omega(n)$  samples.
Large Scale We consider two modifications to our algorithm to optimize the performance of CrudeNN on the two challenging datasets that we are using; namely, note that both datasets are standard for benchmarking billion-scale nearest neighbor search. First, in the CrudeNN algorithm, when computing  $D_{a}$  for  $a \in A$ , we search through the hash buckets  $h_1(a), h_2(a), \ldots$  containing
$a$  in increasing order of  $i$  (i.e., smallest scale first), and retrieve the first  $W$  (window size) distinct points in  $B$  from these buckets. Then, the whole process is repeated  $k$  times, with  $k$  independent LSH data structures, and  $\pmb{D}_a$  is set to be the distance from  $a$  to the closest among all  $Wk$  retrieved points.
Note that previously, for our smaller datasets, we set  $D_{a}$  to be the distance to the first point in  $B$  colliding with  $a$ , and repeated the LSH data structure once, corresponding to  $W = k = 1$ . In our figures, we refer to these parameter choices as  $k \times W$  and test our algorithm across several choices.
For the DEEP and Turing datasets, Figures 3 (d) and 3 (e) show the sample complexity vs relative error trade-offs for the best parameter choice (both  $64 \times 10^{6}$ ) compared to uniform sampling. Qualitatively, we observe the same behavior as before: importance sampling requires fewer samples to obtain the same accuracy as uniform sampling. Regarding the other parameter choices, we see that, as expected, if we decrease  $k$  (the number of LSH data structures), or if we decrease  $W$  (the window size), the quality of the approximations  $\{D_{a}\}_{a \in A}$  decreases and importance sampling has worse sample complexity trade-offs. Nevertheless, for all parameter choices, we see that we obtain superior sample complexity trade-offs compared to uniform sampling, as shown in Figure 5. A difference between these parameter choices are the runtimes required to construct the approximations  $\{D_{a}\}_{a \in A}$ . For example for the DEEP dataset, the naive brute force approach (which is also optimized using parallelization) took approximately  $1.3 \cdot 10^{4}$  seconds, whereas the most expensive parameter choice of  $64 \times 10^{6}$  took approximately half the time at  $6.4 \times 10^{3}$  and the cheapest parameter choice of  $8 \times 10^{5}$  took 225 seconds, leading to a  $2\mathbf{x} - 50\mathbf{x}$  factor speedup. The runtime differences between brute force and our algorithm were qualitative similar for the Turing dataset.
Similar to the small scale dataset, our method also outperforms uniform sampling in terms of runtime if we require they both output high quality approximations. If we measure the runtime to get a  $1\%$  relative error, the  $16 \times 2 \cdot 10^{5}$  version of our algorithm for the DEEP dataset requires approximately 980 samples with total runtime approximately 1785 seconds, whereas uniform sampling requires  $>1750$  samples and runtime  $>2200$  seconds, which is  $>23\%$  slower. The gap in runtime increases if we desire approximations with even smaller relative error, as the overhead of obtaining the approximations  $\{D_{a}\}_{a \in A}$  becomes increasingly overwhelmed by the time needed to compute the exact answer for our samples.
Additional Experimental Results We perform additional experiments to show the utility of our approximation algorithm for the Chamfer distance for downstream tasks. For the ShapeNet dataset, we show we can efficiently recover the true exact nearest neighbor of a fixed point cloud  $A$  in Chamfer distance among a large collect of different point clouds. In other words, it is beneficial for finding the 'nearest neighboring point cloud'. Recall the ShapeNet dataset, contains approximately  $5 \cdot 10^4$  different point clouds. We consider the following simple (and standard) two step pipeline: (1) use our algorithm to compute an approximation of the Chamfer distance from  $A$  to every other point cloud  $B$  in our dataset. More specifically, compute an approximation to  $\mathrm{CH}(A,B) + \mathrm{CH}(B,A)$  for all  $B$  using 50 samples and the same parameter configurations as the small scale experiments. Then filter the dataset of points clouds and prune down to the top  $k$  closest point cloud candidates according to our approximate distances. (2) Find the closest point cloud in the top  $k$  candidates via exact computation.
We measure the accuracy of this via the standard recall @k measure, which computes the fraction of times the exact nearest neighbor  $B$  of  $A$ , averaged over multiple  $A$ 's, is within the top  $k$  choices. Figure 6 (a) shows that the true exact nearest neighbor of  $A$ , that is the point cloud  $B$  which minimizes  $\mathsf{CH}(A,B) + \mathsf{CH}(B,A)$  among our collection of multiple point clouds, is within the top 30 candidates  $>98\%$ , time (averaged over multiple different choices of  $A$ ). This represents a more than 1000x reduction in the number of point clouds we do exact computation over compared to the naive brute force method, demonstrating the utility of our algorithm for downstream tasks.
4 Lower Bound for Reporting the Alignment
We presented an algorithm that, in time  $\mathcal{O}\bigl (nd\log (n) / \varepsilon^2\bigr)$ , produces a  $(1 + \varepsilon)$ -approximation to  $\mathrm{CH}(A,B)$ . It is natural to ask whether it is also possible to report a mapping  $g:A\to B$  whose cost  $\sum_{a\in A}\| a - g(a)\| _1$  is within a factor of  $1 + \varepsilon$  from  $\mathrm{CH}(A,B)$ . (Our algorithm uses on random sampling and thus does not give such a mapping). This section shows that, under a popular complexity-theoretic conjecture called the Hitting Set Conjecture [24], such an algorithm does not exist. For simplicity, we focus on the case when the underlying metric  $d_X$  is induced by the Manhattan distance,
(a) ShapeNet NNS pipeline experiments
Figure 6: Additional figures for the ShapeNet dataset.
(b) Quality of approximations  $D_{a}$  vs the number of levels of LSH data structure
i.e.,  $d_X(a, b) = \| a - b \|_1$ . The argument is similar for the Euclidean distance, Euclidean distance squared, etc. To state our result formally, we first define the Hitting Set (HS) problem.
Definition 4.1 (Hitting Set (HS) problem). The input to the problem consists of two sets of vectors  $A, B \subseteq \{0,1\}^d$ , and the goal is to determine whether there exists some  $a \in A$  such that  $a \cdot b \neq 0$  for every  $b \in B$ . If such an  $a \in A$  exists, we say that  $a$  hits  $B$ .
It is easy to see that the Hitting Set problem can be solved in time  $\mathcal{O}(n^2 d)$ . The Hitting Set Conjecture [24] postulates that this running time is close to the optimal. Specifically:
Conjecture 4.2. Suppose  $d = \Theta(\log^2 n)$ . Then for every constant  $\delta > 0$ , no randomized algorithm can solve the Hitting Set problem in  $\mathcal{O}(n^{2 - \delta})$  time.
Our result can be now phrased as follows.
Theorem 4.3 (Hardness for reporting a mapping). Let  $T(N, D, \varepsilon)$  be the running time of an algorithm ALG that, given sets of  $A^{\prime \prime}$ ,  $B^{\prime \prime} \subset \{0,1\}^{D}$  of sizes at most  $N$ , reports a mapping  $g: A^{\prime \prime} \to B^{\prime \prime}$  with cost  $(1 + \varepsilon)\mathsf{CH}(A^{\prime \prime}, B^{\prime \prime})$ , for  $D = \Theta (\log^2 N)$  and  $\varepsilon = \frac{\Theta(1)}{D}$ . Assuming the Hitting Set Conjecture, we have that  $T(N, D, \varepsilon)$  is at least  $\Omega (N^{2 - \delta})$  for any constant  $\delta > 0$ .
We present an efficient approximation algorithm for estimating the Chamfer distance up to a  $1 + \varepsilon$  factor in time  $\mathcal{O}\bigl (nd\log (n) / \varepsilon^2\bigr)$ . The result is complemented with a conditional lower bound which shows that reporting a Chamfer distance mapping of similar quality requires nearly quadratic time. Our algorithm is easy to implement in practice and compares favorably to brute force computation and uniform sampling. We envision our main tools of obtaining fast estimates of coarse nearest neighbor distances combined with importance sampling can have additional applications in the analysis of high-dimensional, large scale data.
AB was supported by Ankur Moitra's ONR grant and the NSF TRIPODS program (award DMS-2022448). PI was supported by the NSF TRIPODS program (award DMS-2022448), Simons Investigator Award and GIST-MIT Research Collaboration grant.
[1] Pdal: Chamfer. https://pdal.io/en/2.4.3/apps/chamfer.html, 2023. Accessed: 2023-05-12.
[2] Pytorch3d: Loss functions. https://torch3d.readthedocs.io/en/latest/modules/loss.html, 2023. Accessed: 2023-05-12.
[18] Jiri Matousek. Lectures on discrete geometry, volume 212. Springer Science & Business Media, 2013.
[19] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26, 2013.
[20] Dhruv Rohatgi. Conditional hardness of earth mover distance. In Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques (APPROX/RANDOM 2019). Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik, 2019.
[21] Harsha Vardhan Simhadri, George Williams, Martin Aumüller, Matthijs Douze, Artem Babenko, Dmitry Baranchuk, Qi Chen, Lucas Hosseini, Ravishankar Krishnaswamny, Gopal Srinivasa, et al. Results of the neurips'21 challenge on billion-scale approximate nearest neighbor search. In NeurIPS 2021 Competitions and Demonstrations Track, pages 177-189. PMLR, 2022.
[22] Erik B Sudderth, Michael I Mandel, William T Freeman, and Alan S Willsky. Visual hand tracking using nonparametric belief propagation. In 2004 Conference on Computer Vision and Pattern Recognition Workshop, pages 189-189. IEEE, 2004.
[23] Ziyu Wan, Dongdong Chen, Yan Li, Xingguang Yan, Junge Zhang, Yizhou Yu, and Jing Liao. Transductive zero-shot learning with visual structure constraint. Advances in neural information processing systems, 32, 2019.
[24] Virginia Vassilevska Williams. On some fine-grained questions in algorithms and complexity. In Proceedings of the international congress of mathematicians: Rio de janeiro 2018, pages 3447-3487. World Scientific, 2018.