Unsupervised Point Cloud Completion through Unbalanced Optimal Transport

Authors: Taekyung Lee, Jaemoo Choi, Jaewoong Choi, Myungjoo Kang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Point Cloud Completion, Unpaired Point Cloud Completion, Unbalanced optimal transport, Optimal transport

Abstract:
Unpaired point cloud completion explores methods for learning a completion map from unpaired incomplete and complete point cloud data. In this paper, we propose a novel approach for unpaired point cloud completion using the unbalanced optimal transport map, called Unbalanced Optimal Transport Map for Unpaired Point Cloud Completion (UOT-UPC). We demonstrate that the unpaired point cloud completion can be naturally interpreted as the Optimal Transport (OT) problem and introduce the Unbalanced Optimal Transport (UOT) approach to address the class imbalance problem, which is prevalent in unpaired point cloud completion datasets. 
Moreover, we analyze the appropriate cost function for unpaired completion tasks. This analysis shows that the InfoCD cost function is particularly well-suited for this task.
Our model is the first attempt to leverage UOT for unpaired point cloud completion, achieving competitive or superior results on both single-category and multi-category datasets. In particular, our model is especially effective in scenarios with class imbalance, where the proportions of categories are different between the incomplete and complete point cloud datasets.

Decision: reject
Reviewer scores: 5, 3, 3

Reviews and discussion:
<thread>
creation date:
1730202485139

thread_type:
official_review

<thread_object>
creator: reviewer_ulat

<content>
summary:
This paper propose an unbalanced optimal transport for unpaired point cloud completion. They treat the completion problem as mapping incomplete set to complete set and utilize InfoCD for cost function. They show results for single-category and multiple-category unpaired point cloud completion and conducted many ablation study.

strengths:
1. They propose to deal with the unbalance problem in point cloud completion.
2. They present better results than the competitors.

weaknesses:
1. Optimal Transport Map is utilized to map the incomplete set to complete set. However, this assume the set of incomplete point cloud and the set of complete set should be complete set. For instance, this work utilized the complete point cloud from ShapeNet and partial point cloud from ScanNet. The original dataset only align the ShapeNet model with the real scans, however, the assumption in Optimal Transport is hard to be satisfied.
2. Experiments are only conducted in data from Scan2CAD. More experiments should be conducted on MatterPort3D or KITTI like "ACL-SPC: Adaptive Closed-Loop system for Self-Supervised Point Cloud Completion" and ModelNet or 3D-FUTURE like "CloudMix: Dual Mixup Consistency for Unpaired Point Cloud Completion".
3. More related works like "ACL-SPC: Adaptive Closed-Loop system for Self-Supervised Point Cloud Completion" and "CloudMix: Dual Mixup Consistency for Unpaired Point Cloud Completion" should be compared.

questions:
Whether the assumption in optimal transport map is satisfied by the given data should be discussed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730480346976

thread_type:
official_review

<thread_object>
creator: reviewer_d7gl

<content>
summary:
This paper introduces UOT-UPC, an unpaired point cloud completion model based on the unbalanced optimal transport (UOT) map. The key idea is to train a model that learns the UOT map between the distribution of incomplete point clouds and the distribution of complete point clouds. This approach leverages the UOT framework's ability to address the class imbalance problem commonly found in unpaired point cloud completion datasets. The paper also identifies the InfoCD cost function as particularly well-suited for unpaired point cloud completion tasks. Experiments show that using InfoCD leads to better performance compared to other cost functions like l2, L2-Chamfer distance, and one-directional L2-Chamfer distance.

strengths:
1. Novelty: UOT-UPC is the first model to apply the unbalanced optimal transport map to unpaired point cloud completion.

2. Robustness to Class Imbalance: The UOT framework allows UOT-UPC to effectively handle class imbalance. Experiments demonstrated that UOT-UPC maintained consistent performance across various class imbalance ratios, better than other models like USSPA and OT-UPC.

3. Better Performance: UOT-UPC achieves better performance on both single-category and multi-category settings on the dataset proposed by Ma et al. 2023.

weaknesses:
1. Limited Cost Function Exploration: The paper's claim of identifying the "optimal" cost function can be challenged. While the authors compare four different cost functions (l2, L2-Chamfer distance (cdl2), one-directional L2-Chamfer distance (cdl2fwd), and InfoCD), this is a relatively small selection of potential options. Other cost functions might exist that could yield even better performance. The study primarily relies on the ShapeNet dataset for evaluating these cost functions. 

2. Dataset Dependence: The experiments primarily focus on a single dataset proposed in Ma et al. 2023, despite the availability of other datasets in the field. This raises concerns about the generalizability of UOT-UPC's performance to other datasets. Evaluating the model on a wider variety of datasets would strengthen the paper's conclusions and provide a more comprehensive understanding of UOT-UPC's capabilities and limitations.

questions:
According to the weakness part, please clarify the choice of cost function and report more results on other datasets. Besides, the source mixture trick seems interesting but not well-explained. Can you give more explanation and/or more insights about this trick?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730717481476

thread_type:
official_review

<thread_object>
creator: reviewer_duov

<content>
summary:
This paper an unpaired point cloud completion approach based on the unbalanced optimal transport map. The key idea is to formulate the unpaired point cloud completion task as the optimal transport problem and investigate the optimal cost function for this task, and introduce an unbalanced optimal transport framework for addressing the class imbalance problem. Experimental results show the proposed method achieves state-of-the-art performance in unpaired point cloud completion.

strengths:
1.	I like the 3D shape completion topic, and the pipeline is carefully designed.
2.	The proposed method is evaluated on the dataset proposed in USSPA and performs better than SOTAs, although some important SOTAs are missing.
3.	The paper is clear and easy to follow.

weaknesses:
1.	The authors should test more categories to explore the effects of class imbalance.
2.	The authors should conduct more datasets for demonstrating the effectiveness of the proposed mehtods, such as PCN datasets.
3.	Some important SOTA methods for 3D shape completion are missing. The authors should compare and discuss them with the proposed method.
[1] ASFM-Net: Asymmetrical Siamese Feature Matching Network for Point Completion
[2] 3D Shape Generation and Completion Through Point-Voxel Diffusion
4. The computational cost should be analyzed and compared with the other methods.
5. The authors are encouraged to provide code for reimplementation.

questions:
1.	Compared with the diffusion based mehtods, what are the advantages of the proposed mehtod?
2.	Compared with USSPA, the proposed method has comparable (lower) performance in some categories. What is the reason?
3.	Is it possiable to complete unseen categories?
4.	Do the authors think the CD or F_score are the best metrics for evaluating 3D shape completion?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732515949610

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response

comment:
Dear reviewers,

We thank the reviewers for their thoughtful comments and suggestions. We are pleased that the reviewers recognize the novelty of our framework as the first model to utilize the unbalanced optimal transport map [D7gL] and acknowledge its superior performance compared to previous models [dUov, D7gL, uLAt].

We consider that the main concern is that the evaluation of our model primarily relies on the USSPA benchmark (dUov, D7gL, uLAt), although we adopted both the single-category and multi-category scenarios. To address this concern, we have conducted the following additional experiments:

- (1) We evaluated our model's point cloud completion performance on the PCN dataset (**Table 9**) [dUov, D7gL, uLAt]. 
- (2) We evaluated the generalizability of our model on real scan data (KITTI dataset) (**Fig 11**) [uLAt].
- (3) We evaluated the robustness of our model to the class imbalance problem by testing two different class combinations across diverse class imbalance ratios (**Table 8**) [dUov].
- (4) We demonstrated that our cost function evaluation for the OT framework generalizes to another dataset by conducting an additional cost ablation study on the PCN dataset (**Table 10**) [D7gL].

We hope our responses to be helpful in addressing the reviewer's concerns. We highlighted the corresponding revisions in the manuscript in Blue [dUov], Red [D7gL], and Brown [uLAt].
</content>
</thread_object>

</thread>


Metareview:
metareview:
After rebuttal and multiple rounds of discussion, all three reviewers unanimously agreed to reject this submission, especially the problem formulation and inferior performance. The specific comments provided by reviewers are presented as follows.


Reviewer dUov：The paper lacks experiments on diverse datasets (e.g., PCN) and does not explore the effects of class imbalance thoroughly. Additionally, comparisons with key state-of-the-art methods (e.g., ASFM-Net and Point-Voxel Diffusion) and analysis of computational costs are missing.

Reviewer D7gL: The exploration of cost functions is limited, and the claim of identifying the "optimal" cost function is overstated. Experiments rely heavily on a single dataset (USSPA), raising concerns about generalizability, and additional datasets like PCN or KITTI should be used to validate the method further.

Reviewer uLAt: The assumption of distribution alignment between incomplete and complete point clouds is unrealistic in many cases, as datasets like ShapeNet and ScanNet come from different sources. Experiments are limited to Scan2CAD, and comparisons with more relevant methods (e.g., ACL-SPC and CloudMix) are lacking.


Two reviewers recommended rejection (scores of 3), and one provided a marginally below acceptance threshold score (5). So I recommend “reject”.