Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposed a 3D object segmentation method that can be trained in an unsupervised fashion. The core underlying assumption is the rigid motion dynamics and shape invariance between point cloud sequences. By carefully designing geometry consistency losses, supervision signals can be effectively extracted from a pair of point clouds. The paper has demonstrated promising results on multiple different datasets and scenarios.
strengths_and_weaknesses: 1) This paper carefully designed multiple loss functions, which powers the unsupervised learning of object segmentation. Removing the need for large scale human annotation is a critical step towards making such method applicable in different new fields and settings.
2) This paper demonstrated great results on multiple datasets.
questions: 1) Motion rigidity is an excellent clue for learning object segmentation. However, in certain datasets, or for certain sensor modality, the actual shape invariance and rigid dynamic might not hold in the sensory data. For example, rolling shutter effect is a very well known effect that distorts high speed moving objects captured by a rolling shutter camera. For point clouds captured by mechanical lidar, motion distortion caused by scene flow, especially high speed moving objects, such as vehicles coming from the opposite direction, is usually very obvious. This paper has made great progress in leveraging object rigidity, but it would be interesting to see, how different degrees of motions can affect the proposed method. Especially, for cases like mechanical lidar point cloud, if motion distortion can affect the proposed method.
limitations: 1) The proposed method relies on the object rigidity for object segmentation. But there are objects, such as articulated buses, semi-truck with trailers, that changes shapes during their movement. The proposed method might not be able to handle such cases well.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies the problem of unsupervised 3D object segmentation from a point cloud. In particular, the proposed approach learns the scene flow in a self-supervised manner which is used as supervision signal and enforces the object geometric consistency for segmenting the static objects. The proposed unsupervised approach is evaluated on various synthetic and real dataset to demonstrate the performance.
strengths_and_weaknesses: # Strengths
1)	Unsupervised 3D object segmentation from point cloud is overlooked in the computer vision field. The paper proposed a solution to solve the problem.
2)	Point cloud motion is applied as supervision signal for 3D object point segmentation. The motion consistency indeed provides strong information for 3D object point segmentation.
3)	The geometry invariance loss is applied to encourage the segmentation of static objects in the scene.
4)  The method is evaluated on both synthetic and real datasets.

# Weaknesses
1)	The training loss quality will be influenced by the self-supervised scene flow estimation approach.
2)	The proposed method may fail if two objects are close with similar motion. 
3)	There are still over-segmentation issues in the results.
4)   The proposed method can only handle objects from a finite dataset which hinders its generalisation ability.

questions: 1)	How is the MaskFormer [8] trained? Is a pre-trained model applied to obtain the coarse object mask?
2)	Please analyse Influence of the smoothness parameter H for over segmentation.
3)	Could the method handle incomplete point cloud? 
4)	It would be great to show the proposed method can handle static objects with qualitative results.

5)	Based on the reviewer’s understanding, the proposed approach mainly handle foreground objects without road. A large number of points belongs to floor or road which can be easily removed by plane fitting. The baseline clustering results should become much better. 

limitations: The paper has discussed limitations of the proposed method.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a pipeline of 3D object segmentation from point clouds. The key idea is to add several regularization losses to enforce geometry consistency on object motion. The proposed method is evaluated on datasets for object part instance segmentation, general object segmentation, and scene flow estimation, where it achieves competitive performance.
strengths_and_weaknesses: **Originality**: The proposed pipeline is a novel combination of existing techniques. Only limited work has been proposed to address the problem of unsupervised 3D object segmentation. This paper leverages self-supervised scene flow estimation models to obtain initial motion estimation from a pair of point clouds and a neural network to obtain initial object masks, which are then used to supervise the learning of the object segmentation backbone, via introducing regularization techniques to enforce rigid transformation on objects, local smoothness, and invariance on semantic predictions. Each subcomponent by itself is not new, but the reviewer gives the credit to the combination.

**Quality**: The submission is not technically sound. Segmenting 3D objects from the dynamics of point clouds is not a new concept, which has been explored in the scene flow estimation community as the motion segmentation task, e.g., experiments in FlowNet3D [1]. While in this submission, the authors explicitly add an object mask model and combine its prediction with the scene flow prediction to obtain auxiliary supervision signals.  The authors identify the creation of auxiliary supervision signals using motion dynamics as the key challenge and push it as the key contribution. However, the resulting three losses, i.e., the multi-object dynamic rigid consistency loss,  the object shape smoothness prior loss, and the object shape invariance loss are well-known techniques in scene flow estimation and augmentation-based self-supervised learning. 
- The multi-object dynamic rigid consistency loss.  The concept has been explored in Rigid3DSceneFlow[2], where they add rigid-motion constraints for both ego-motion and object motion.
- The object shape smoothness prior loss. It shares the same spirit of local smoothness losses that are widely used in the scene flow community, where the key idea is to have a similar motion prediction in nearby points. Here the prior loss is a minor modification by changing the motion prediction to the object mask prediction.
- The object shape invariance loss is highly relevant to the unsupervised semantic segmentation pipeline. The authors resolve the binding problem, e.g., point-to-label, using the Hungarian algorithm between predictions from augmented versions of point clouds and then check the consistency between these predictions. The high-level concept comes from augmentation-based self-supervised learning, still, the proposed loss is somewhat novel.

Overall, the reviewer considers the paper has limited novelty due to the above discussions.


[1] FlowNet3D: Learning Scene Flow in 3D Point Clouds. CVPR 2019

[2] Weakly Supervised Learning of Rigid 3D Scene Flow. CVPR 2021
 
 
**Clarity:** The submission has some issues.

It is not clear to the reviewer on the object segmentation network. Is it a supervised model? How do the authors obtain the initial multiple object masks without any human annotations? The authors mentioned the transformer for mask prediction. But it remains unclear to the reviewer how the transformer is trained. 
 
Equation (3) is a wrong notation to ignore the assignment. It didn’t make sense to directly associate two sets of object masks without a `permutation’.



questions: The reviewer was concerned that the authors made a wild claim “we introduce the first unsupervised multi-object segmentation pipeline on 3D point clouds, without needing any human annotations in training” as this has been previously explored in the scene flow community.
limitations: The evaluation of the SAPIEN and OGC-DR for this paper seems to be limited as these toy-like datasets have zero background and well-separated objects. The reviewer would expect more exploration of more complicated LIDAR datasets while the paper is currently limited on a small KITTI-SF dataset.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduced a self-supervised approach to do instance segmentation on point clouds. 
The key contribution is a set of auxiliary losses to ensure segmentation consistency under per-object rigid transformation and view-variance.
The paper showed large improvement over other unsupervised approaches, which are based on clustering and motion segmentation, and in some cases, the proposed method performs competitively compared to supervised baselines.


---

*I thank the authors for the answer and in particular I appreciate the added experiments. Final recommendation: While I think the proposed method has large potential for improvement, it's relevant to the community as it is the first paper for unsupervised detection. **I maintain my original score.** However the response of the authors provided renewed insights, shedding light to limitations the actual usability due to high training time and the limited generalisability of the learned features. I think it's only fair if the authors mention these in the main paper and link the experiment results to the appendix.*
strengths_and_weaknesses: **Strength**
- the first unsupervised method to perform instance segmentation on point clouds (although I'm not a domain expert and couldn't really verify this claim) 
- thorough experiments and ablation study on various datasets
- a novel iterative optimization procedure to improve the segmentation mask via improving scene flow

**Weakness**
- The method seems only to be able to detect rigid moving objects, although L155-156 suggests that the geometry invariance loss can also help detect static objects. In general, the object segmentation network doesn't seem to have any fundamental issues in predicting masks for static objects. The dynamic loss also does not penalize static objects (e.g. $M^t = 0, T_k = I$). It would be good if the authors can further clarify the limitation of the proposed method in segmenting static objects.
- I'm not sure the geometry smooth term is well designed. For one, it could lead to blurry prediction boundaries? To me, it makes sense to weigh the smoothness term based on the predicted $T_k$. 
- The bold font in a mixed table of supervised and unsupervised comparison is a bit confusing, especially in Table 4, all other methods seem to be better than OGC but OGC is marked bold. 
- Also regarding table formatting, it would be good to indicate whether the metric is larger the better or smaller the better, e.g. with $\uparrow$ and $\downarrow$
- One interesting benefit of unsupervised training is that it can be used as a pertaining procedure to boost performance when the data annotation is sparse or for transferring to another domain. The others have shown domain transfer to KITTI-Det, but the baselines are all supervised methods. What's missing is to demonstrate the advantage of the OGC(unsup) as a pertaining step for semi-supervised learning with partial training data, preferably doing so while comparing other unsupervised methods.
questions: - Are $T_k$ and $O_k^t$ jointly optimized? 
- What is the training time required to perform the iterative optimization? How does it compare to the supervised version? 
- It seems like neither MaskFormer and SceneFlow nets are pretrained. Why? For each dataset, do you need to train these nets from scratch? Do the trained nets to transfer well to another dataset?
- Is the dynamic loss able to recover small objects misclassified as part of another larger object or background? I'm concerned that the estimation $T_k$ will be prone to fit the larger object and ignore the smaller one. This may likely make the segmentation mask fit tighter to the larger object. But I don't see any incentive to segment the smaller object as a new object, say in $o_{n,k'}$
limitations: Yes
review_title: null

Meta-review: metareview: This is an interesting paper that proposes a novel unsupervised approach for object segmentation from point clouds, for rigid objects. The strong results demonstrated can be impactful both for 3d as well as potentially 2d vision. After rebuttal, all 4 expert reviewers are convinced that the paper should be accepted, so the decision to accept the paper was easy.
confidence: Certain
Decision: Accept