Title: Near-Linear Time Algorithm for the Chamfer Distance
Venue: NeurIPS 2023
Accepted: yes
Keywords: chamfer distance

earth mover distance

high dimensional data analysis

nearest neighbor search

high dimensional data

high-dimensional geometry

sublinear algorithms

point clouds

theory
Abstract: For any two point sets $A,B \subset \mathbb{R}^d$ of size up to $n$, the Chamfer distance from $A$ to $B$ is defined as $\texttt{CH}(A,B)=\sum_{a \in A} \min_{b \in B} d_X(a,b)$, where $d_X$ is the underlying distance measure (e.g., the Euclidean or Manhattan distance). The Chamfer distance is a popular measure of dissimilarity between point clouds, used in many machine learning, computer vision, and graphics applications, and admits a straightforward $O(d n^2)$-time brute force algorithm. Further, Chamfer distance is often used as a  proxy for the more computationally demanding Earth-Mover (Optimal Transport) Distance. However, the \emph{quadratic} dependence on $n$ in the running time  makes the naive approach intractable for large datasets.

We overcome this bottleneck and present the first $(1+\epsilon)$-approximate algorithm for estimating Chamfer distance with a near-linear running time. Specifically, our algorithm runs in time $O(nd \log (n)/\epsilon^2)$ and is implementable. Our experiments demonstrate that it is both accurate and fast on large high-dimensional datasets. We believe that our algorithm will open new avenues for analyzing large high-dimensional point clouds. We also give evidence that if the goal is to report a $(1+\epsilon)$-approximate mapping from $A$ to $B$ (as opposed to just its value), then any sub-quadratic  time algorithm is unlikely to exist.

A Near-Linear Time Algorithm for the Chamfer Distance

Abstract

For any two point sets A,B ⊂ d of size up to n , the Chamfer distance from A to B is defined as CH ( A,B ) = ∑ a ∈ A min b ∈ B d X ( a, b ) , where d X is the underlying distance measure (e.g., the Euclidean or Manhattan distance). The Chamfer distance is a popular measure of dissimilarity between point clouds, used in many machine learning, computer vision, and graphics applications, and admits a straightforward O ( dn 2 ) -time brute force algorithm. Further, the Chamfer distance is often used as a proxy for the more computationally demanding Earth-Mover (Optimal Transport) Distance. However, the quadratic dependence on n in the running time makes the naive approach intractable for large datasets.

We overcome this bottleneck and present the first (1+ ε ) -approximate algorithm for estimating the Chamfer distance with a near-linear running time. Specifically, our algorithm runs in time O ( nd log( n ) / ε 2 ) and is implementable. Our experiments demonstrate that it is both accurate and fast on large high-dimensional datasets. We believe that our algorithm will open new avenues for analyzing large highdimensional point clouds. We also give evidence that if the goal is to report a (1 + ε ) -approximate mapping from A to B (as opposed to just its value), then any sub-quadratic time algorithm is unlikely to exist.

1 Introduction

For any two point sets A,B ⊂ d of sizes up to n , the Chamfer distance 1 from A to B is defined as

where d X is the underlying distance measure, such as the Euclidean or Manhattan distance. The Chamfer distance, and its weighted generalization called Relaxed Earth Mover Distance [16, 7], are popular measures of dissimilarity between point clouds. They are widely used in machine learning (e.g., [16, 23]), computer vision (e.g., [8, 22, 12, 15]) and computer graphics [17]. Subroutines for computing Chamfer distances are available in popular libraries, such as Tensorflow [3], Pytorch [2] and PDAL [1]. In many of those applications (e.g., [16]) Chamfer distance is used as a faster proxy for the more computationally demanding Earth-Mover (Optimal Transport) Distance.

Despite the popularity of Chamfer distance, the naïve algorithm for computing it has quadratic O ( n 2 ) running time, which makes it difficult to use for large datasets. Faster approximate algorithms can be obtained by performing n exact or approximate nearest neighbor queries, one for each point in A . By utilizing the state of the art approximate nearest neighbor algorithms, this leads to (1 + ε ) -approximate estimators with running times of O ( n (1 / ε ) O ( d ) log n ) in low dimensions [6] or roughly O ( dn 1+ 1 2(1+ ε ) 2 -1 ) in high dimensions [5]. Alas, the first bound suffers from exponential dependence on the dimension, while the second bound is significantly subquadratic only for relatively large approximation factors.

1.1 Our Results

In this paper we overcome this bottleneck and present the first (1 + ε ) -approximate algorithm for estimating Chamfer distance that has a near-linear running time, both in theory and in practice. Concretely, our contributions are as follows:

When the underlying metric d X is defined by the /lscript 1 or /lscript 2 norm, we give an algorithm that runs in time O ( nd log( n ) / ε 2 ) and estimates the Chamfer distance up to 1 ± ε with 99% probability (see Theorem 2.1). In general, our algorithm works for any metric d X supported by Locality-Sensitive Hash functions (see Definition 2.2), with the algorithm running time depending on the parameters of those functions. Importantly, the algorithm is quite easy to implement, see Figures 1 and 2.

For the more general problem of reporting a mapping g : A → B whose cost ∑ a ∈ A d X ( a, g ( a )) is within a factor of 1 + ε from CH ( A,B ) , we show that, under a popular complexity-theoretic conjecture, an algorithm with a running time analogous to that of our estimation algorithm does not exist, even when d X ( a, b ) = ‖ a -b ‖ 1 . Specifically, under a Hitting Set Conjecture [24], any such algorithm must run in time Ω ( n 2 -δ ) for any constant δ > 0 , even when the dimension d = Θ (log 2 n ) and ε = Θ (1) d . (In contrast, our estimation algorithm runs in near-linear time for such parameters). This demonstrates that, for the Chamfer distance, estimation is significantly easier than reporting.

We experimentally evaluate our algorithm on real and synthetic data sets. Our experiments demonstrate the effectiveness of our algorithm for both low and high dimensional datasets and across different dataset scales. Overall, it is much faster ( >5x ) than brute force (even accelerated with KD-trees) and both faster and more sample efficient ( 5-10x ) than simple uniform sampling. We demonstrate the scalability of our method by running it on billion-scale Big-ANN-Benchmarks datasets [21], where it runs up to 50x faster than optimized brute force. In addition, our method is robust to different datasets: while uniform sampling performs reasonably well for some datasets in our experiments, it performs poorly on datasets where the distances from points in A to their neighbors in B vary significantly. In such cases, our algorithm is able to adapt its importance sampling probabilities appropriately and obtain significant improvements over uniform sampling.

2 Algorithm and Analysis

In this section, we establish our main result for estimating Chamfer distance:

Theorem 2.1 (Estimating Chamfer Distance in Nearly Linear Time) . Given as input two datasets A,B ⊂ R d such that | A | , | B | /lessorequalslant n , and an accuracy parameter 0 < ε < 1 , Chamfer-Estimate runs in time O ( nd log( n ) / ε 2 ) and outputs an estimator η such that with probability at least 99 / 100 , when the underlying metric is Euclidean ( /lscript 2 ) or Manhattan ( /lscript 1 ) distance.

For ease of exposition, we make the simplifying assumption that the underlying metric is Manhattan distance, i.e. d X ( a, b ) = ‖ a -b ‖ 1 . Our algorithm still succeeds whenever the underlying metric admits a locality-sensitive hash function (the corresponding analogue of Definition 2.2).

Definition 2.2 (Hashing at every scale) . There exists a fixed constant c 1 > 0 and a parameterized family H ( r ) of functions from X to some universe U such that for all r > 0 , and for every x, y ∈ X

Close points collide frequently:

/negationslash

Input:

Two subsets A,B ⊂ d of size at most n , and a parameter T ∈ .

Output:

Execute the algorithm CrudeNN ( A,B ) , and let the output be a set of positive real numbers { D a } a ∈ A which always satisfy D a /greaterorequalslant min b ∈ B ‖ a -b ‖ 1 . Let D := ∑ a A D a .

A number η ∈ /greaterorequalslant 0 .

∈ 2. Construct the probability distribution D , supported on the set A , which satisfies that for every a ∈ A ,

For /lscript ∈ [ T ] , sample x /lscript ∼ D and spend O ( | B | d ) time to compute

Far points collide infrequently:

Uniform vs Importance Sampling. A natural algorithm for estimating CH ( A,B ) proceeds by uniform sampling : sample an a ∈ A uniformly at random and explicitly compute min b ∈ B ‖ a -b ‖ 1 . In general, we can compute the estimator ˆ z for CH ( A,B ) by averaging over s uniformly chosen samples, resulting in runtime O ( nds ) . It is easy to see that the resulting estimator is un-biased, i.e. E [ˆ z ] = CH ( A,B ) . However, if a small constant fraction of elements in A contribute significantly to CH ( A,B ) , then s = Ω ( n ) samples could be necessary to obtain, say, a 1% relative error estimate with constant probability. Since each sample requires a linear scan to find the nearest neighbor, this would result in a quadratic runtime.

While such an approach has good empirical performance for well-behaved datasets, it does not work for data sets where the distribution of the distances from points in A to their nearest neighbors in B is skewed. Further, it is computationally prohibitive to verify the quality of the approximation given by uniform sampling. Towards proving Theorem 2.1, it is paramount to obtain an algorithm that works regardless of the structure of the input dataset.

Amore nuanced approach is to perform importance sampling where we sample a ∈ A with probability proportional to its contribution to CH ( A,B ) . In particular, if we had access to a distribution, D a , over elements a ∈ A such that, min b ∈ B ‖ a -b ‖ 1 /lessorequalslant D a /lessorequalslant λ min b ∈ B ‖ a -b ‖ 1 , for some parameter λ > 1 , then sampling O ( λ ) samples results in an estimator ˆ z that is within 1% relative error to the true answer with probability at least 99% . Formally, we consider the estimator defined in Algorithm 1, where we assume access to CrudeNN ( A,B ) , a sub-routine which receives as input A and B and outputs estimates D a ∈ /greaterorequalslant 0 for each a ∈ A which is guaranteed to be an upper bound for min b ∈ B ‖ a -b ‖ 1 . Based on the values { D a } a ∈ A we construct an importance sampling distribution D supported on A . As a result, we obtain the following lemma:

Lemma 2.3 (Variance Bounds for Chamfer Estimate) . Let n, d ∈ and suppose A,B are two subsets of d of size at most n . For any T ∈ , the output η of Chamfer-Estimate ( A,B,T ) satisfies

Output

for D from Line 1 in Figure 1. The expectations and variance are over the randomness in the samples of Line 3 of Chamfer-Estimate ( A,B,T ) . In particular,

The proof follows from a standard analysis of importance sampling and is deferred to Appendix A. Observe, if D /lessorequalslant λ CH ( A,B ) , it suffices to sample T = O ( λ / ε 2 ) points in A , leading to a running time of O ( nd λ / ε 2 ) .

Subroutine CrudeNN ( A,B )

Input: Two subsets A,B of a metric space ( X, ‖ · ‖ 1 ) of size at most n such that all non-zero distances between any point in A and any point in B is between 1 and poly( n/ ε ) . We assume access to a locality-sensitive hash family at every scale H ( r ) for any r /greaterorequalslant 0 satisfying conditions of Definition 2.2. (We show in Appendix A that, for /lscript 1 and /lscript 2 , the desired hash families exist, and that distances between 1 and poly( n/ ε ) is without loss of generality).

We instantiate L = O (log( n/ ε )) and for i ∈ { 0 , . . . , L } , we let r i = 2 i .

Output: A list of numbers { D a } a ∈ A where D a /greaterorequalslant min b ∈ B ‖ a -b ‖ 1 .

For each i ∈ { 0 , . . . , L } sample a hash function h i : X → U from h i ∼ H ( r i ) .

The above may be done by first hashing each point b ∈ B and i ∈ { 0 , . . . , L } according to h i ( b ) . Then, for each a ∈ A , we iterate through i ∈ { 0 , . . . , L } while hashing a according to h i ( a ) until the first b ∈ B with h i ( a ) = h i ( b ) is found.

For each a ∈ A , find the smallest i ∈ { 0 , . . . , L } for which there exists a point b ∈ B with h i ( a ) = h i ( b ) , and set D a = ‖ a -b ‖ 1 .

Obtaining importance sampling probabilities. It remains to show how to implement the CrudeNN ( A,B ) subroutine to obtain the distribution over elements in A which is a reasonable over-estimator of the true probabilities. A natural first step is to consider performing an O (log n ) -approximate nearest neighbor search (NNS): for every a ′ ∈ A , find b ′ ∈ B satisfying ‖ a ′ -b ′ ‖ 1 / min b ∈ B ‖ a ′ -b ‖ 1 = O (log n ) . This leads to the desired guarantees on { D a } a ∈ A . Unfortunately, the state of the art algorithms for O (log n ) -approximate NNS, even under the /lscript 1 norm, posses extraneous poly (log n ) factors in the runtime, resulting in a significantly higher running time. These factors are even higher for the /lscript 2 norm. Therefore, instead of performing a direct reduction to approximate NNS, we open up the approximate NNS black-box and give a simple algorithm which directly satisfies our desired guarantees on { D a } a ∈ A .

To begin with, we assume that the aspect ratio of all pair-wise distances is bounded by a fixed polynomial, poly( n/ ε ) (we defer the reduction from an arbitrary input to one with polynomially bounded aspect ratio to Lemma A.3). We proceed via computing O (log( n/ ε )) different (randomized) partitions of the dataset A ∪ B . The i -th partition, for 1 /lessorequalslant i /lessorequalslant O (log( n/ ε )) , can be written as A ∪ B = ∪ j P i j and approximately satisfies the property that points in A ∪ B that are at distance at most 2 i will be in the same partition P i j with sufficiently large probability. To obtain these components, we use a family of locality-sensitive hash functions , whose formal properties are given in Definition 2.2. Intuitively, these hash functions guarantee that:

For each a ′ ∈ A , its true nearest neighbor b ′ ∈ B falls into the same component as a ′ in the i 0 -th partition, where 2 i 0 = Θ ( ‖ a ′ -b ′ ‖ 1 ) 2 , and

/negationslash

Every other extraneous b = b ′ is not in the same component as a ′ for each i < i 0 .

It is easy to check that any hash function that satisfies the aforementioned guarantees yields a valid set of distances { D a } a ∈ A as follows: for every a ′ ∈ A , find the smallest i 0 for which there exists a b ′ ∈ B in the same component as a ′ in the i 0 -th partition. Then set D a ′ = ‖ a ′ -b ′ ‖ 1 . Intuitively, the b ′ we find for any fixed a ′ in this procedure will have distance that is at least the closest neighbor in B and with good probability, it won't be too much larger. A caveat here is that we cannot show the

above guarantee holds for 2 i 0 = Θ ( ‖ a ′ -b ′ ‖ 1 ) . Instead, we obtain the slightly weaker guarantee that, in the expectation , the partition b ′ lands in is a O (log n ) -approximation to the minimum distance, i.e. 2 i 0 = Θ (log n · ‖ a ′ -b ′ ‖ 1 ) . Therefore, after running CrudeNN ( A,B ) , setting λ = log n suffices for our O ( nd log( n ) / ε 2 ) time algorithm. We formalize this argument in the following lemma:

Lemma 2.4 (Oversampling with bounded Aspect Ratio) . Let ( X,d X ) be a metric space with a locality-sensitive hash family at every scale (see Definition 2.2). Consider two subsets A,B ⊂ X of size at most n and any ε ∈ (0 , 1) satisfying

/negationslash

Algorithm 2, CrudeNN ( A,B ) , outputs a list of (random) positive numbers { D a } a ∈ A which satisfy the following two guarantees:

With probability 1 , every a ∈ A satisfies D a /greaterorequalslant min b ∈ B d X ( a, b ) .

For every a ∈ A , E [ D a ] /lessorequalslant O (log n ) · min b ∈ B d X ( a, b ) .

Further, Algorithm 2, runs in time O ( dn log( n/ ε )) time, assuming that each function used in the algorithm can be evaluated in O ( d ) time.

Proof Sketch for Theorem 2.1. Given the lemmas above, it is straight-forward to complete the proof of Theorem 2.1. First, we reduce to the setting where the aspect ratio is poly( n/ ε ) (see Lemma A.3 for a formal reduction). We then invoke Lemma 2.4 and apply Markov's inequality to obtain a set of distances D a such that with probability at least 99 / 100 , for each a ∈ A , min b ∈ B ‖ a -b ‖ 1 /lessorequalslant D a and ∑ a ∈ A D a /lessorequalslant O (log( n )) CH ( A,B ) . We then invoke Lemma 2.3 and set the number of samples, T = O ( log( n ) / ε 2 ) . The running time of our algorithm is then given by the time of CrudeNN ( A,B ) , which is O ( nd log( n/ ε )) , and the time needed to evaluate the estimator in Lemma 2.3, requiring O ( nd log( n ) / ε 2 ) time. Refer to Section A for the full proof.

Other Related Works We note that importance sampling is a popular technique used for speeding up geometric algorithms. For example, [14] uses it to obtain a fast c -approximate algorithm for computing Earth Mover Distance (EMD) in two (or any constant) dimensions, for some constant c > 2 . However, the application and implementation of importance sampling in that paper is quite different from ours. In [14], the space containing all input points is subdivided into regions, and the total EMD value is represented as a sum of EMDs restricted to point-sets in each region (plus an additional representing the 'global' EMD). The EMD cost in each region is then approximated quickly by embedding EMD into /lscript 1 using a randomly shifted quadtree with logarithmic distortion; these estimations define the sampling probabilities.

In contrast, in our paper, the value of the Chamfer distance is exactly equal to the sum of distances from each point to its nearest neighbor, so there is no decomposition involved. Instead, we approximate each distance to nearest neighbor using a randomized hierarchical decomposition; for the case of the /lscript 1 norm, each level of the decomposition partitions the space into rectangular boxes, as in quadtrees. Crucially, however, to ensure that the running time of our algorithm is within the stated bounds, we cannot use a standard randomly shifted quadtree where each level is shifted by the same random vector (as in [14]). This is because shifting all levels by the same amount only ensures that the expected distortion between a fixed pair of points is logarithmic; to ensure that the distance to the nearest neighbor is distorted by O (log n ) , we would need to use O (log n ) independent quadtrees and apply the union bound. Instead, we use independent random partitions at each level, and show (Lemma 2.4) that this suffices to bound the expected distortion of the distance to the nearest neighbor, without incurring any additional factors. This makes it possible to obtain the running time as stated.

There are other works on quickly computing EMD and related distances. For example, the algorithm of [4] runs in time that is linear in the number of distances, i.e. it runs in Ω ( n 2 ) time and gets a 1 + ε approximation to EMD. This means that their approach requires a runtime quadratic in the size of the dataset n . In contrast, Chamfer distance admits a trivial time O ( n 2 ) algorithm and our main contribution is to provide a nearly linear O ( n log( n ) / ε 2 ) algorithm to get (1 + ε ) -approximation to Chamfer distance.

3 Experiments

We perform an empirical evaluation of our Chamfer distance estimation algorithm.

Summary of Results Our experiments demonstrate the effectiveness of our algorithm for both low and high dimensional datasets and across different dataset sizes. Overall, it is much faster than brute force (even accelerated with KD-trees). Further, our algorithm is both faster and more sample-efficient than uniform sampling. It is also robust to different datasets: while uniform sampling performs well for most datasets in our experiments, it performs poorly on datasets where the distances from points in A to their neighbors in B vary significantly. In such cases, our algorithm is able to adapt its importance sampling probabilities appropriately and obtain significant improvements over uniform sampling.

Table 1: Summary of our datasets. For ShapeNet, the value of | A | and | B | is averaged across different point clouds in the dataset.

3.1 Experimental Setup

We use three different experimental setups, small scale, outlier, and large scale. They are designed to 'stress test' our algorithm, and relevant baselines, under vastly different parameter regimes. The datasets we use are summarized in Table 1. For all experiments, we introduce uniform sampling as a competitive baseline for estimating the Chamfer distance, as well as (accelerated) brute force computation. All results are averaged across 20+ trials and 1 standard deviation error bars are shown when relevant.

Small Scale These experiments are motivated from common use cases of Chamfer distance in the computer vision and NLP domains. In our small scale experiments, we use two different datasets: (a) the ShapeNet dataset, a collection of point clouds of objects in three dimensions [10]. ShapeNet is a common benchmark dataset frequently used in computer graphics, computer vision, robotics and Chamfer distance is a widely used measure of similarity between different ShapeNet point clouds [10]. (b) We create point clouds of words from text documents from [16]. Each point represents a word embedding obtained from the word-to-vec model of [19] in 300 applied to the Federalist Papers corpus. As mentioned earlier, a popular relaxation of the common Earth Mover Distance is exactly the (weighted) version of the Chamfer distance [16, 7].

Since ShapenNet is in three dimensions, we implement nearest neighbor queries using KD-trees to accelerate the brute force baseline as KD-trees can perform exact nearest neighbor search quickly in small dimensions. However, they have runtime exponential in dimension meaning they cannot be used for the text embedding dataset, for which we use a standard naive brute force computation. For both these datasets, we implement our algorithms using Python 3.9.7 on an M1 MacbookPro with 32GB of RAM. We also use an efficient implementation of KD trees in Python and use Numpy and Numba whenever relevant. Since the point clouds in the dataset have approximately the same n value, we compute the symmetric version CH ( A,B ) + CH ( B,A ) . For these experiments, we use the /lscript 1 distance function.

Outliers This experiment is meant to showcase the robustness of our algorithm. We consider two point clouds, A and B , each sampled from Gaussian points in 100 with identity covariance. Furthermore, we add an "outlier" point to A equal to 0 . 5 n · 1 , where 1 is the all ones vector.

This example models scenarios where the distances from points in A to their nearest neighbors in B vary significantly, and thus uniform sampling might not accurately account for all distances, missing a small fraction of large ones.

Large Scale The purpose of these experiments is to demonstrate that our method scales to datasets with billions of points in hundreds of dimensions. We use two challenging approximate nearest neighbor search datasets: DEEP1B [9] and Microsoft Turing-ANNS [21]. For these datasets, the set A is the query data associated with the datasets. Due to the asymmetric sizes, we compute CH ( A,B ) . These datasets are normalized to have unit norm and we consider the /lscript 2 distance function.

These datasets are too large to handle using the prior configurations. Thus, we use a proprietary inmemory parallel implementation of the SimHash algorithm, which is an /lscript 2 LSH family for normalized vectors according to Definition 2.2 [11], on a shared virtual compute cluster with 2x64 core AMD Epyc 7763 CPUs (Zen3) with 2.45Ghz - 3.5GHz clock frequency, 2TB DDR4 RAM and 256 MB L3 cache. We also utilize parallization on the same compute cluster for naive brute force search.

3.2 Results

(a) ShapeNet

(b) Federalist Papers

(d) DEEP

(e) Turing

(a) ShapeNet

(b) Federalist Papers

Small Scale First we discuss configuring parameters. Recall that in our theoretical results, we use O (log n ) different scales of the LSH family in CrudeNN . CrudeNN then computes (over) estimates of the nearest neighbor distance from points in A to B (in near linear time) which is then used

(a) DEEP

(b) Turing

for importance sampling by Chamfer-Estimate . Concretely for the /lscript 1 case, this the LSH family corresponds to imposing O (log n ) grids with progressively smaller side lengths. In our experiments, we treat the number of levels of grids to use as a tuneable parameter in our implementation and find that a very small number suffices for high quality results in the importance sampling phase.

Figure 6 (b) shows that only using 3 grid levels is sufficient for the crude estimates D a to be within a factor of 2 away from the true nearest neighbor values for the ShapeNet dataset, averaged across different point clouds in the dataset. Thus for the rest of the Small Scale experiments, we fix the number of grid levels to be 3 .

Figure 3 (a) shows the sample complexity vs accuracy trade offs of our algorithm, which uses importance sampling, compared to uniform sampling. Accuracy is measured by the relative error to the true value. We see that our algorithm possesses a better trade off as we obtain the same relative error using only 10 samples as uniform sampling does using 50+ samples, resulting in at least a 5x improvement in sample complexity. For the text embedding dataset, the performance gap between our importance sampling algorithm and uniform sampling grows even wider, as demonstrated by Figure 3 (b), leading to > 10x improvement in sample complexity.

In terms of runtimes, we expect the brute force search to be much slower than either importance sampling and uniform sampling. Furthermore, our algorithm has the overhead of first estimating the values D a for a ∈ A using an LSH family, which uniform sampling does not. However, this is compensated by the fact that our algorithm requires much fewer samples to get accurate estimates.

Indeed, Figure 4 (a) shows the average time of 100 Chamfer distance computations between randomly chosen pairs of point clouds in the ShapeNet dataset. We set the number of samples for uniform sampling and importance sampling (our algorithm) such that they both output estimates with (close to) 2% relative error. Note that our runtime includes the time to build our LSH data structures. This means we used 100 samples for importance sampling and 500 for uniform. The brute force KD Tree algorithm (which reports exact answers) is approximately 5x slower than our algorithm. At the same time, our algorithm is 50% faster than uniform sampling. For the Federalist Papers dataset (Figure 4 (b)), our algorithm only required 20 samples to get a 2% relative error approximation, whereas uniform sampling required at least 450 samples. As a result, our algorithm achieved 2x speedup compared to uniform sampling.

Outliers We performed similar experiments as above. Figure 3 (c) shows the sample complexity vs accuracy trade off curves of our algorithm and uniform sampling. Uniform sampling has a very large error compared to our algorithm, as expected. While the relative error of our algorithm decreases smoothly as the sample size grows, uniform sampling has the same high relative error. In fact, the relative error will stay high until the outlier is sampled, which typically requires Ω ( n ) samples.

Large Scale We consider two modifications to our algorithm to optimize the performance of CrudeNN on the two challenging datasets that we are using; namely, note that both datasets are standard for benchmarking billion-scale nearest neighbor search. First, in the CrudeNN algorithm, when computing D a for a ∈ A , we search through the hash buckets h 1 ( a ) , h 2 ( a ) , . . . containing

a in increasing order of i (i.e., smallest scale first), and retrieve the first W (window size) distinct points in B from these buckets. Then, the whole process is repeated k times, with k independent LSH data structures, and D a is set to be the distance from a to the closest among all Wk retrieved points.

Note that previously, for our smaller datasets, we set D a to be the distance to the first point in B colliding with a , and repeated the LSH data structure once, corresponding to W = k = 1 . In our figures, we refer to these parameter choices as k × W and test our algorithm across several choices.

For the DEEP and Turing datasets, Figures 3 (d) and 3 (e) show the sample complexity vs relative error trade-offs for the best parameter choice (both 64 × 10 6 ) compared to uniform sampling. Qualitatively, we observe the same behavior as before: importance sampling requires fewer samples to obtain the same accuracy as uniform sampling. Regarding the other parameter choices, we see that, as expected, if we decrease k (the number of LSH data structures), or if we decrease W (the window size), the quality of the approximations { D a } a ∈ A decreases and importance sampling has worse sample complexity trade-offs. Nevertheless, for all parameter choices, we see that we obtain superior sample complexity trade-offs compared to uniform sampling, as shown in Figure 5. A difference between these parameter choices are the runtimes required to construct the approximations { D a } a ∈ A . For example for the DEEP dataset, the naive brute force approach (which is also optimized using parallelization) took approximately 1 . 3 · 10 4 seconds, whereas the most expensive parameter choice of 64 × 10 6 took approximately half the time at 6 . 4 × 10 3 and the cheapest parameter choice of 8 × 10 5 took 225 seconds, leading to a 2x-50x factor speedup. The runtime differences between brute force and our algorithm were qualitative similar for the Turing dataset.

Similar to the small scale dataset, our method also outperforms uniform sampling in terms of runtime if we require they both output high quality approximations. If we measure the runtime to get a 1% relative error, the 16 × 2 · 10 5 version of our algorithm for the DEEP dataset requires approximately 980 samples with total runtime approximately 1785 seconds, whereas uniform sampling requires > 1750 samples and runtime > 2200 seconds, which is > 23% slower. The gap in runtime increases if we desire approximations with even smaller relative error, as the overhead of obtaining the approximations { D a } a ∈ A becomes increasingly overwhelmed by the time needed to compute the exact answer for our samples.

Additional Experimental Results We perform additional experiments to show the utility of our approximation algorithm for the Chamfer distance for downstream tasks. For the ShapeNet dataset, we show we can efficiently recover the true exact nearest neighbor of a fixed point cloud A in Chamfer distance among a large collect of different point clouds. In other words, it is beneficial for finding the 'nearest neighboring point cloud'. Recall the ShapeNet dataset, contains approximately 5 · 10 4 different point clouds. We consider the following simple (and standard) two step pipeline: (1) use our algorithm to compute an approximation of the Chamfer distance from A to every other point cloud B in our dataset. More specifically, compute an approximation to CH ( A,B ) + CH ( B,A ) for all B using 50 samples and the same parameter configurations as the small scale experiments. Then filter the dataset of points clouds and prune down to the top k closest point cloud candidates according to our approximate distances. (2) Find the closest point cloud in the top k candidates via exact computation.

We measure the accuracy of this via the standard recall @ k measure, which computes the fraction of times the exact nearest neighbor B of A , averaged over multiple A 's, is within the top k choices. Figure 6 (a) shows that the true exact nearest neighbor of A , that is the point cloud B which minimizes CH ( A,B )+ CH ( B,A ) among our collection of multiple point clouds, is within the top 30 candidates > 98% , time (averaged over multiple different choices of A ). This represents a more than 1000x reduction in the number of point clouds we do exact computation over compared to the naive brute force method, demonstrating the utility of our algorithm for downstream tasks.

4 Lower Bound for Reporting the Alignment

We presented an algorithm that, in time O ( nd log( n ) / ε 2 ) , produces a (1 + ε ) -approximation to CH ( A,B ) . It is natural to ask whether it is also possible to report a mapping g : A → B whose cost ∑ a ∈ A ‖ a -g ( a ) ‖ 1 is within a factor of 1 + ε from CH ( A,B ) . (Our algorithm uses on random sampling and thusdoes not give such a mapping). This section shows that, under a popular complexitytheoretic conjecture called the Hitting Set Conjecture [24], such an algorithm does not exists. For simplicity, we focus on the case when the underlying metric d X is induced by the Manhattan distance,

(a) ShapeNet NNS pipeline experiments

(b) Quality of approximations D a vs the number of levels of LSH data structure

i.e., d X ( a, b ) = ‖ a -b ‖ 1 . The argument is similar for the Euclidean distance, Euclidean distance squared, etc. To state our result formally, we first define the Hitting Set (HS) problem.

/negationslash

It is easy to see that the Hitting Set problem can be solved in time O ( n 2 d ) . The Hitting Set Conjecture [24] postulates that this running time is close to the optimal. Specifically:

Definition 4.1 (Hitting Set (HS) problem) . The input to the problem consists of two sets of vectors A,B ⊆ { 0 , 1 } d , and the goal is to determine whether there exists some a ∈ A such that a · b = 0 for every b ∈ B . If such an a ∈ A exists, we say that a hits B .

Conjecture 4.2. Suppose d = Θ (log 2 n ) . Then for every constant δ > 0 , no randomized algorithm can solve the Hitting Set problem in O ( n 2 -δ ) time. Our result can be now phrased as follows.

Theorem 4.3 (Hardness for reporting a mapping) . Let T ( N,D, ε ) be the running time of an algorithm ALG that, given sets of A ' , B ' ⊂ { 0 , 1 } D of sizes at most N , reports a mapping g : A ' → B ' with cost (1 + ε ) CH ( A ' , B ') , for D = Θ (log 2 N ) and ε = Θ (1) D . Assuming the Hitting Set Conjecture, we have that T ( N,D, ε ) is at least Ω ( N 2 -δ ) for any constant δ > 0 .

5 Conclusion

We present an efficient approximation algorithm for estimating the Chamfer distance up to a 1 + ε factor in time O ( nd log( n ) / ε 2 ) . The result is complemented with a conditional lower bound which shows that reporting a Chamfer distance mapping of similar quality requires nearly quadratic time. Our algorithm is easy to implement in practice and compares favorably to brute force computation and uniform sampling. We envision our main tools of obtaining fast estimates of coarse nearest neighbor distances combined with importance sampling can have additional applications in the analysis of high-dimensional, large scale data.

References

Pdal: Chamfer. https://pdal.io/en/2.4.3/apps/chamfer.html , 2023. Accessed: 2023-0512.

Pytorch3d: Loss functions. https://pytorch3d.readthedocs.io/en/latest/modules/loss. html , 2023. Accessed: 2023-05-12.

Tensorflow graphics: Chamfer distance. https://www.tensorflow.org/graphics/api_ docs/python/tfg/nn/loss/chamfer_distance/evaluate , 2023. Accessed: 2023-05-12.

Jason M. Altschuler, Jonathan Weed, and Philippe Rigollet. Near-linear time approximation algorithms for optimal transport via sinkhorn iteration. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 1964-1974, 2017.

Alexandr Andoni and Ilya Razenshteyn. Optimal data-dependent hashing for approximate near neighbors. In Proceedings of the forty-seventh annual ACM symposium on Theory of computing , pages 793-801, 2015.

Sunil Arya, David M Mount, Nathan S Netanyahu, Ruth Silverman, and Angela Y Wu. An optimal algorithm for approximate nearest neighbor searching fixed dimensions. Journal of the ACM (JACM) , 45(6):891-923, 1998.

Kubilay Atasu and Thomas Mittelholzer. Linear-complexity data-parallel earth mover's distance approximations. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 364-373. PMLR, 09-15 Jun 2019.

Vassilis Athitsos and Stan Sclaroff. Estimating 3d hand pose from a cluttered image. In 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings. , volume 2, pages II-432. IEEE, 2003.

Artem Babenko and Victor Lempitsky. Efficient indexing of billion-scale datasets of deep descriptors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2055-2063, 2016.

Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012 , 2015.

Moses S Charikar. Similarity estimation techniques from rounding algorithms. In Proceedings of the thiry-fourth annual ACM symposium on Theory of computing , pages 380-388, 2002.

Haoqiang Fan, Hao Su, and Leonidas J Guibas. A point set generation network for 3d object reconstruction from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 605-613, 2017.

Piotr Indyk. Stable distributions, pseudorandom generators, embeddings, and data stream computation. Journal of the ACM (JACM) , 53(3):307-323, 2006.

Piotr Indyk. A near linear time constant factor approximation for euclidean bichromatic matching (cost). In Nikhil Bansal, Kirk Pruhs, and Clifford Stein, editors, Proceedings of the Eighteenth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA 2007, New Orleans, Louisiana, USA, January 7-9, 2007 , pages 39-42. SIAM, 2007.

Li Jiang, Shaoshuai Shi, Xiaojuan Qi, and Jiaya Jia. Gal: Geometric adversarial loss for single-view 3d-object reconstruction. In Proceedings of the European conference on computer vision (ECCV) , pages 802-816, 2018.

Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. From word embeddings to document distances. In International conference on machine learning , pages 957-966. PMLR, 2015.

Chun-Liang Li, Tomas Simon, Jason Saragih, Barnabás Póczos, and Yaser Sheikh. Lbs autoencoder: Self-supervised fitting of articulated meshes to point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11967-11976, 2019.

Jiri Matousek. Lectures on discrete geometry , volume 212. Springer Science & Business Media, 2013.

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems , 26, 2013.

Dhruv Rohatgi. Conditional hardness of earth mover distance. In Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques (APPROX/RANDOM 2019) . Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik, 2019.

Harsha Vardhan Simhadri, George Williams, Martin Aumüller, Matthijs Douze, Artem Babenko, Dmitry Baranchuk, Qi Chen, Lucas Hosseini, Ravishankar Krishnaswamny, Gopal Srinivasa, et al. Results of the neurips'21 challenge on billion-scale approximate nearest neighbor search. In NeurIPS 2021 Competitions and Demonstrations Track , pages 177-189. PMLR, 2022.

Erik B Sudderth, Michael I Mandel, William T Freeman, and Alan S Willsky. Visual hand tracking using nonparametric belief propagation. In 2004 Conference on Computer Vision and Pattern Recognition Workshop , pages 189-189. IEEE, 2004.

Ziyu Wan, Dongdong Chen, Yan Li, Xingguang Yan, Junge Zhang, Yizhou Yu, and Jing Liao. Transductive zero-shot learning with visual structure constraint. Advances in neural information processing systems , 32, 2019.

Virginia Vassilevska Williams. On some fine-grained questions in algorithms and complexity. In Proceedings of the international congress of mathematicians: Rio de janeiro 2018 , pages 3447-3487. World Scientific, 2018.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a fast estimation of Chamfer distance which is a famous distance between two sets. The key idea is to use random sampling on the first set and then find the closest point on the other set. To enhance further the approximation performance and quality of samples, importance sampling is utilized. The paper proposes to use a proposal $D_a$ such that for any $a \\in A$, we have $ min_{b \\in B} |a-b| \\leq D_a \\leq \\lambda  min_{b\\in B} |a-b| $ for some $\\lambda >1$.  From that, the paper can obtain an $\\epsilon$-approximation computational complexity of the estimation. To obtain the proposal distribution, a fast sub-routine (CrudeNN) is proposed. The experiments are designed to show that the proposed importance sampling has better relative error than the uniform sampling and is faster than the  KD-tree implementation.\n\nSTRENGTHS:\n* Stochastic approximation of Chamfer distance is new.\n* The design of the importance sampling proposal is interesting.\n* An approximation error can be obtained.\n* The final approximation is faster than the conventional Chamfer distance.\n\nWEAKNESSES:\n* The proposed algorithm can only work with some restricted set of ground metrics that satisfies  (Hashing at every scale).\n* The main weakness of the paper is lacking real applications comparison. For example, the paper can compare the proposed approximation with the conventional Chamfer distance on a point-cloud reconstruction/completion task. \n* In deep learning applications, the gradient of Chamfer distance with respect to an input set is required. Therefore, some investigation about the approximation error of the gradient should be considered.\n\nQUESTIONS:\n* As the paper mentions Chamfer is a fast alternative distance to EMD, the stochastic approximation of EMD should be discussed [1].\n\n[1] Improving Mini-batch Optimal Transport via Partial Transportation\n\n* For near-linear computational complexity, the sliced Wasserstein distance has the same complexity. Therefore, it is a direct competitor to the proposed approximated Chamfer and should be compared in a benchmark point-cloud task in the paper.\n\nLIMITATIONS:\nNot applicable.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides the first near-linear time approximation scheme to compute the Chamfer distance $\\texttt{CH}(A,B) = \\sum_{a \\in A} \\min_{b \\in B} d_X(a,b)$ for any two point sets $A,B \\in \\mathbb{R}^d$ and $d_X$ is the distance measure under consideration. In particular, for $|A|, |B| \\leq n$, and for any $\\epsilon > 0$, they provide an algorithm which outputs a $(1 + \\epsilon)$-approximate solution with probability greater than 99% in time $O(nd \\log(n)/\\epsilon^2)$. They also provide experimental evaluation of their algorithm and also a matching lower bound for the harder task of not just computing the value of the Chamfer distance but also providing a valid mapping from $A \\mapsto B$ which achieves that Chamfer distance.\n\nSTRENGTHS:\nOriginality: The paper is the first to provide a near-linear time approximation scheme for computing Chamfer distances.\n\nQuality and Clarity: Paper is overall pretty well-written.\n\nSignificance: I have personally not used Chamfer distances in any of my prior work before so cannot directly attest the importance of Chamfer distances but the authors have done a reasonably good job of motivating the problem in the introduction of the paper. It looks like Chamfer distances have already been implemented in several popular ML libraries like TensorFlow and PyTorch. They have also been used in prior papers which have appeared in NeurIPS and other related conferences.\n\nWEAKNESSES:\nNot any that I can think of right now.\n\nQUESTIONS:\nNot a major question, but more of a suggestion. As a person who has not worked with Chamfer distances earlier, I don't know if they are useful other than as a proxy for Earth-mover distance (at least that's what I got from your introduction). A quick google search on approximating Earth-mover distances led me to this following paper: Near-linear time approximation algorithms for optimal transport via Sinkhorn iteration, by Altschuler et.al, which appeared in NeurIPS 2017. The paper seems to be pretty well-known by now, with over 500 citations. Given that your paper is about approximating Chamfer distances, which seem to only be a proxy for Earth-mover distance, I think it is very natural that you at least refer this paper, or even better give a more detailed explanation of why your paper is relevant even though there is an approximation scheme for Earth-mover distances already.\n\nLIMITATIONS:\nMainly theoretical paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper gives an algorithm to approximate Chamfer distance between two sets within $(1 \\pm \\epsilon) $ of the original value. The brute force algorithm to calculate the distance exactly between two sets of size $n$ and with points in $d$-dimension is $O(n^2d)$ which is prohibitive for large values of $n$. The authors use a crude version of the locality sensitive hashing technique, to design an importance sampling algorithm that samples $O(\\log{n})$ points from the first set and approximates Chamfer distance only using these weighted and sampled points, hence giving an approximation algorithm with time complexity $O(nd\\log{n})$. They validate the efficiency of their algorithm for a variety of datasets. They also show negative result where they show that it is not possible to give the actual mapping between the two sets which gives the $(1 + \\epsilon) $approximate Chamfer distance with a faster algorithm under the assumption that the hitting set conjecture is true.\n\nSTRENGTHS:\n1) The paper is written very nicely and is coherent and easy to follow\n2) Experiments are performed under a variety of settings and code is provided. Authors should make it public if the paper gets accepted.\n3) Proofs appear sound to the best of my understanding.\n\nOverall, the paper does a good job of efficiently approximation a quantity which finds application in a variety of problems in machine learning and will be of interest to the community.\n\nWEAKNESSES:\n1) I think it would be good if the authors can at least give the definition of Locality Sensitivity Hashing in the main body of the paper. It would increase the readability.\n2) The only other arguable weakness I find is that the ideas used in the paper and the proof techniques are not novel and as such may not be so interesting from theoretical perspective.\n\nQUESTIONS:\nPlease refer to weaknesses section\n\nLIMITATIONS:\nPlease refer to weaknesses section", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the approximation algorithm for computing the Chamfer distance between two datasets. The Chamfer distance is a very useful distance metric in machine learning, and the naive approach for computing it takes $O(n^2d)$-time for $n$ points in $d$ dimensions. This paper proposes an approximation algorithm that runs in $O(nd\\log(n)/\\epsilon^2)$-time with $(1+\\epsilon)$-relative error. This algorithm is the first to achieve linear dependence in $nd$ and polynomial dependence in $1/\\epsilon$. Moreover, it is implementable on real datasets and empirically improves the previous methods. In addition, it shows a fine-grained lower bound that no randomized algorithm can find a $(1+ \\epsilon)$-approximate mapping for the Chamfer distance in $n^{2-o(1)}$-time for $d\\sim \\log^2 n$ and $\\epsilon\\sim 1/d$, assuming the Hitting Set Conjecture. Technically, their algorithm is based on the importance sampling and Locality-Sensitive Hashing.\n\nSTRENGTHS:\nThe problem studied in this paper is significant in theory and practice. The algorithm in this paper runs in nearly-linear time in $n$ and works for large dimensions and high accuracy. For comparison, previous approximation algorithms either incur an exponential dependence in $d$ or only work for the low-accuracy regime. This paper provides an essential theoretical improvement. Another advantage is that the algorithm is quite clean that can be implemented efficiently, which increases the practical value of this paper.  This paper thoroughly explains the experiments.  And most theorems and lemmas in this paper are mathematically sound to me.\n\nWEAKNESSES:\nSome notations could be more precise in the main text. For example, the subroutine CrudeNN uses $H(r)$ without defining it. And more intuition should be given for this subroutine. Also, the algorithm in this paper works for any distance metric with LSH functions. Some examples other than $\\ell_1$ or $\\ell_2$ can be provided. For the hardness result, the Hitting Set Conjecture is not very standard in the field of fine-grained complexity.\n\nQUESTIONS:\n1. In Section 2, it discusses the uniform sampling approach. And the runtime is $O(nds)$ if we sample $s$ points. What if we use any computational geometry data structure to pre-process the set $B$ such that for each sample, the computational cost can be reduced to $o(nd)$? \n\n2. Line 27: $O(n^2)$ should be $O(n^2d)$ to avoid confusion. \n\n3. Line 85: ``distribution, $D_a$”. However, $D_a$ itself is not a probability distribution but an estimator. \n\n4. Proof of Lemma A.2: the event $E_1(\\gamma)$ is missing.\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of computing the Chamfer distance between two point sets. Informally, the Chamfer distance is simply the total cost of mapping points of A to their closest point in B. Practically, Chamfer distance seems to be used as a proxy for the earth mover’s distance.\n\nOne can always compute Chamfer distance is quadratic time – for every point in A, find the cheapest edge by simply scanning all the edges. Of course, in low dimensional spaces, one can speed this up by the use of exact nearest neighbor data structures. To approximate the Chamfer distance, one can use the approximate nearest neighbor data structure leading to a near-linear time algorithm in fixed dimension and a super-linear time algorithm in higher dimensions. \n\nThe main contribution of this paper is to show that Chamfer distance can be estimated in near-linear time even in higher dimensions. As opposed to this, the authors show that computing the maps themselves is hard to approximate in near-linear time (the running time has to be only slightly sub-quadratic under the hitting set conjecture). \n\nThe main technique used is the idea of importance sampling. Their algorithm first obtains a crude approximation of the contribution of each point and then uses these values to obtain a small sized weighted sample which can then be used to estimate the Chamfer distance in $O(nd\\log n/\\varepsilon^2)$ time. The idea of using Importance sampling to estimate distances efficiently is not new. It has been used, for instance, to estimate EMD in low dimensions by Indyk in his paper titled “A near linear time constant factor approximation for Euclidean bichromatic matching (cost).”\nNonetheless, the use of importance sampling for approximate Chamfer distance is novel and the contribution towards computing high dimensional Chamfer distance is significant. For this reason, I recommend accepting this paper.\n\nSTRENGTHS:\nThe algorithm is simple and estimates Chamfer distance for high dimensional data with solid theoretical guarantees and strong practical results.\n\nWEAKNESSES:\nI’m not convinced that Chamfer distance has strong theoretical properties. For instance, it doesn’t appear to be symmetric, i.e., Chamfer(A,B) is not equal to Chamfer(B,A). Is there a relationship between Chamfer distance and EMD even under some stochastic assumptions for B and A? \n\nSimilar techniques seem to have been used for EMD approximations before (Indyk, SODA 2007). Could you add a comparison of your algorithm to the techniques used in other papers?\n\nQUESTIONS:\nPlease answer the questions in weaknesses.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-26"}