Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper tackles the problem of object discovery. They propose the 4d unsupervised object discovery from 4D data - 3D point clouds and 2D RGB images with temporal information together with a new model, ClusterNet that is jointly iterative and optimized with a 2D localization network. Finally, results are reported on the Waymo Open Dataset.
strengths_and_weaknesses: The paper tackles an important problem. It proposes the task of object discovery from 3D point clouds together with 2D images along with the temporal information which can be of interest for the community. I find the joint optimization to be interesting.

My main concern related to this paper is the comparison with state of the art which I find to be quite limited. I would have wished to see more comparisons with other methods for 2D unsupervised object detection. Since the comparison is quite limited, it is hard to estimate the gain brought by the proposed approach. So, the only estimate that remains is the comparison with the supervised case. However, for this setup I feel like there are little details about the used architecture.
questions: 1. How is "n" chosen for the number of instances?

2. How hard is to collect data with cloud points?

3. Can you give more details about the fully supervised baseline? Does it use a previously published method?

4. In the implementation details (lines 218-232) it is stated that for example for 2D localization network, Faster R-CNN is utilized. Does this mean that you also initialize some weights with a pre-trained Faster R-CNN model or that you rely on the architecture? This aspect needs to be clearly stated.
limitations: The authors discuss some of the limitations and some potential societal impact problems.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper solves the problem of unsupervised object discovery by utilizing the 4D information (cloud point, and time), they designed an iterative refinement frameworks which do the 2D information refinement and 4D information refinement one-by-one (usually 4~5 iters). 

The results achieves the best result under the new setting. A few ablations about the importance of cues, training strategies, etc are provided to help us to better understand the task and the methods. 
strengths_and_weaknesses: Pro:

- Interesting and sound setting to explore the 4d unsupervised object discovery.
- The writing is easy to follow from intuition, to method, and finally experimental parts.


Con:
- Figure 1 is not clear about the dataflow. I am totally missed.
- One detail I am curious is that whether your network is initialized from scratch, or from another pretrained network. That counts about whether you can name it as 'unsupervised'.
- The contribution claim of clusternet, can the author highlight the difference of it with VoteNet?
questions: Listed as above.
limitations: Yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, the authors have proposed a novel task named *4D Unsupervised Object Discovery*, aiming at discovery the objects from 4D sequential data (3D point clouds + 2D RGB images), without any human annotation.

For this purpose, they proposed a method called *ClusterNet* to automatically discovery the objects in 2D images (bounding boxes) and the objects in 3D point clouds (point-wise segments), which can be jointly optimized on the 4D sequential data by taking the temporal information into account. 

The generated labels are evaluated in 2D object detection and 3D instance segmentation, showing on par results with supervised methods and superior performances compared to other object discovery methods. Furthermore, the 2D object boxes can be converted to 2D object segments by projecting the 3D instance segments to 2D images, which allows for joint segmentation on 4D data.
strengths_and_weaknesses: Strengths:

* This paper proposes a novel task and a novel method with clear presentations, and thorough ablation studies supporting the validation of each component of the proposed method.

* While existing object discovery works mainly focus at extracting discriminative information from pure 2D or 3D data, this work chooses to combine the best of both modes by leveraging the 4D sequential data, and to consider the temporal information as well as the multi-modality constraints. This removes the dependence of a strong self-supervised learning models, and compensate the weaknesses of each side.

* The superior experimental results shows that the advantages of object discovery on 4D data compared to pure 2D or 3D.

Weaknesses:

* No experimental result was given to demonstrate the quality of discovered objects by training existing models with the generated labels instead of the human annotations.


questions: 1. Since one important objective of object discovery is to reduce human effort by automatically generating object labels, retraining existing models with the generated object labels should be added to show the true efficiency of the proposed object discovery method. The performance gap between models trained with generated labels and human labels can serve as a key indicator for the final application in real world. 

2. As this method works on 4D data, the authors propose to initial object labels on 3D, then project to 2D and iteratively optimize them, I am wondering is it possible to initial object labels on both 2D and 3D by clustering moving objects with optical flows and scene flows. This is not necessary, but we might get better initial labels by looking at both modes.

3. The current version of Sec 3.3 is a bit too abstractive, with many details hiding in several parts of the paper.
For example, it is not trivial to train the 2D localization network to discover static objects with initial proposals projected from 3D point clouds, a summarization of the training strategy (L276-290) could be added in Sec 3.3 to help the readers understand quickly how this was done.
limitations: Yes, the limitations are discussed in the last section of the paper.
review_title: null

Meta-review: metareview: This paper focuses on expanding the problem of unsupervised object discovery (detection) to a new setup, where a 3D point cloud is available as well as an RGB sequence. The paper received three detailed reviews from expert reviewers, all of which had their major concerns about the paper resolved through the author rebuttal and author-reviewer discussion period. With the extra analyses and experiments presented in the discussion period, the paper has reached the level of impact and contribution expected by NeurIPS papers. The authors are recommended to add these extra items to the final version of the paper.
confidence: Certain
Decision: Accept