Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper compares Chamfer Distance (CD) and Earth Mover’s Distance (EMD) and then points out their limitations separately. To address the limitations, the authors propose a Balanced Chamfer Distance (BCD) which considers disparity of density distributions, locally detailed structures and has a bounded value stabilizing the training process. These issues are not considered in either CD or EMD. Furthermore, the paper proposes a point discriminator and up/down sampling to enhance the performance of PCN. The experimental results on point cloud completion demonstrate that BCD can overcome shortcomings in CD and EMD.
main_review: This paper is well-motivated as it pinpoints the existing issues with respect to CD and EMD. Figure 1 supports the authors' claim. The Taylor Expansion based modification and the density based “query” estimation (at least) is novel to me. And I believe BCD would have a broader application in 3D vision.  

A key problem is how to decide n_y and n_x, ie, the nearest neighbor assignment. Does it depend on an R^3 sphere whose radii is a hyper-parameter? If so, an ablative study is needed to evaluate the effect of n_y and n_x. Another problem is whether the choice of alpha in Eq.(4) will affect the effectiveness of BCD? Ie, is it possible for some values of alpha that BCD performs better than CD and EMD, but for some other values that BCD performs worse? If the performance of BCD heavily depends on the choice of alpha, it will hurt the merits of BCD. From Eq.(3) and Eq. (4), it is not obvious that BCD considers the global density distribution of point clouds as EMD. It still emphasizes the local structure. Despite the above issues, BCD is still an interesting idea for measuring similarity between point clouds.

The main concern is the point discriminator and up/down sampling presented in Sec. 4.2 are heavily coupled with PCN. as PCN adopts a two-stage coarse-to-fine completion process, which limits its application to other methods. One problem w.r.t this part is whether using point sampling privilege will cause significant computational overhead during inference. 

The experiments are mainly conducted on point cloud completion and use CD, EMD and BCD as evaluation metrics. Though BCD achieves lower values in most (not all) cases (Table 2), I would like to point out that such evaluation is not efficient. As the loss function for completion will always push each metric to a lower value. To strongly support that BCD is more powerful than CD and EMD in 3D vision tasks (in a general sense), experiments on other tasks (ie, rigid registration) are preferred. As CD is usually used as the loss function for rigid registration. 

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper starts with a discussion on the currently used point cloud similarity metrics (Chamfer distance and earth mover’s distance) discussing their drawbacks and proposes a novel similarity measure. It is based on the commonly used Chamfer distance and proposes two principal modifications to it:
1) the distance is scaled exponentially to be robust to outliers and is bound to be in [0, 1];
2) it is weighted by the number of neighboring matched points to account for the local density of the ground truth.
New metric is used for evaluation in the point cloud completion task for which the authors also propose a model based on prior work, but additionally apply outlier filtering by a trainable component and furthest point sampling to additional refine the output.

Experiments show that the proposed modifications achieve state-of-the-art performance with respect to the proposed metric. The qualitative evaluations argue, that the proposed metric bridges the gap between the ability of CD to capture finer details and the ability of EMD to better account for the density distribution of the output.
main_review: Pros:
* The paper is well written and properly justifies the need for improved point cloud similarity measures. The proposed measure is introduced gradually with clear argumentation why each change is required.
* Evaluations on the point completion task bring insightful information about the proposed measure.
* The proposed trainable probabilistic filtering technique produce visually improved results.

Cons:
* The main drawback is discussed in the supplementary and is the inability to use the proposed measure as a loss function because it is not differentiable. In that sense, it is not clear how to optimize existing models for this measure.
* There are some minor corrections and clarifications required:
1) $g(x)$ in equation 5 is a very specific function, which only takes some discrete values depending on the number of neighbors. Could you elaborate why it is used exactly this form of this function and why do you need to regress its values, rather than just classify, how many neighbors the point have?
2) How exactly the probability in equation 7 is used? Do you sample according to it or just take a threshold?
3) It is not stated clearly in the text to what exactly the abbreviations Sample and Filter in Table 3 are referring to. Filter is probabilistic trainable filtering and Sample is FPS?
4) L238 number of epochs
5) L272 more efficient

Overall, I tend to acceptance. However, I would like to see my questions addressed in the rebuttal.
limitations_and_societal_impact: The authors provided a discussion on the importance of reliable similarity measures.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Post rebuttal comment: in my original review I have copied an incorrect paper summary from my notes (from another paper from my review batch). All remaining sections of this review were copied correctly. My deepest apologies about this, I have only noticed this mistake after the rebuttal period. Below I am copying the correct summary: 

---
This paper argues both commonly used evaluation measures for point cloud completion, Chamfer Distance (CD) and Earth Movers Distance (EMD) both suffer from certain limitations. Firstly, both are unbounded (certainly not a desirable property of evaluation measure) and sensitive to outliers. Furthermore, CD is insensitive to point cloud density and thus it doesn’t align well with human perception, while EMD tends to be insensitive to details, requires inputs of equal size, and is computationally expensive. 
To alleviate this, the paper argues for an extension of CD, that transforms the Euclidean distance to a similarity measure, which is bounded and robust to outliers. Second, by normalizing the contribution of each point, the proposed variant of CD becomes sensitive to the local point cloud density, which is argued to be a desired property for the evaluation point cloud completion quality. 
Overall, the paper presents good arguments for the proposed metric being a good substitute for standard CD. The second part of the paper presents an extension to a two-stage point cloud completion network (PCN) that tackles the point imbalance issue wrt. the point discriminator. The experiments show that the proposed method obtains betters results in terms of BCD compared to baselines. 
main_review: 

**Pros**

* This is certainly an interesting paper, pointing out significant drawbacks of existing evaluation measures for assessing point cloud completion. 
* The theoretical discussion of the drawbacks of existing metrics and the proposed extension of CD make perfect sense to me and looks like a nice contribution! 
* Looking at the metric comparison discussion, the proposed metric is computationally effective (which presents an advantage over EMD) and bounded (in contrast to both). It is difficult to interpret results when evaluation measures are unbounded. 


**Cons**


* Given the focus of this paper (on the novel distance metric) and the claims on the “metric being superior in terms of visual quality”, I would have expected s very thorough analysis on this part, discussing the difference between CD, EMD, the proposed method as well as the method by [Nguyen et al, Arxiv:2102.04014], that is (similarly to the proposed method) computationally efficient and has similar properties as EMD. This is only mentioned in the related work. Such as study should also contain a visual human assessment; without it, I don’t see how the claims on this measure being “superior in terms of visual quality” can be justified.
* At some point, it becomes unclear what the focus of this paper is: on a new evaluation measure for point cloud completion, or addressing an inherent imbalance problem of existing point cloud completion methods (Sec. 4). Related to that, It is difficult to draw good conclusions from Sec. 5.1; the proposed method obtains the best results overall in terms of the proposed BCD metrics, while results in terms of CD and EMD are mixed. On one hand, this is indeed what the criticism of existing metrics is (neither of them is comprehensive and weakly correlated), but this experiment somewhat mixes-up evaluation of new metrics and method, that is based on the same observations. Again, I find it very difficult to draw conclusions when the focus is not clear (new metrics vs. methods). However I can observe this: when looking at the class average score, neither CD or EMD reaches a consensus with the proposed BCD. 
* Why is the metric proposed by [Nguyen et al, Arxiv:2102.04014] excluded from the metric comparison and discussion (5.2)?
* This doesn't impact my final rating, but still want to point out: this paper would certainly benefit from investing more time in improving the grammar, sentence structure, and presentation flow. At the bare minimum, I would recommend using grammar recommendation tools. For example, please do not start sentences with “And it would be helpful …” (L196). I would also suggest minimizing the usage of passive voice. 


**Justification**

I do not think this paper is ready for publication at the moment. In principle, the proposed metric seems sound and makes perfect sense. However, I would have expected that the paper will have a strong focus on justifying the proposed evaluation measure; instead, the evaluation mixes up a new variant of point cloud completion network and the metric, making it difficult to draw conclusions. 
In terms of experimental justification of the metric, I would have expected a human quality assessment (to support the claims on being superior when it comes to visual quality evaluation), a discussion and comparison to [Nguyen et al, Arxiv:2102.04014], and analysis across different datasets for point cloud completion (e.g., SemanticKITTI [Behley et al., ICCV’19]) to demonstrate that the proposed metric is versatile (i.e., conclusions are consistent across different scenarios, objects vs. scenes, and different sensors). 

This being said, I am not an expert in this field and I will certainly keep an open mind. 

**Post rebuttal** 
The authors provided a very good rebuttal. Let me quickly summarize:

*Human assessment study* Authors point out that this study was conducted and reported in the supplementary. 
In the future, please make that clear in the paper. Supplementary material is not meant to be read "standalone." I refer to it when I want to see additional results or studies that didn't make it to the main paper (as most reviewers do). Apologies in case this study was referenced, and I've missed it. 

*Confusion in sec. 5.1.* I will trust the authors to revise this section.

*Comparison to Nguyen et al.* Authors point out that this is only a pre-print, therefore I cannot really ask for this comparison. It would still be potentially interesting for the camera-ready.

Overall, my main doubts were addressed well in the rebuttal, and I am ready to raise my ratings. As reviewer gwGK points out, this paper tackles a relevant problem with existing point cloud similarity measures, and I appreciate the core contribution.

I would still like to clarify that after reading Sec. 3, I was expecting a thorough evaluation of the merit of the proposed similarity measure across different datasets and tasks. I think this paper would be stronger if it had a laser focus on the metric.

limitations_and_societal_impact: The authors did not discuss the limitations. I don't see any potential negative social impact of this work. 

**Post rebuttal** The authors did discuss limitations in the supplementary, my bad.
review_title: null

Meta-review: The reviewers appreciated the contribution of the paper, and thought it is a timely introduction of a new measure of comparison of point clouds. There were some doubts whether the BCD measure should be evaluated on point cloud completion task in particular. The authors are encouraged to evaluate the measure on other tasks.
Decision: Accept (Poster)