GrabS: Generative Embodied Agent for 3D Object Segmentation without Scene Supervision

Authors: Zihui Zhang, Yafei YANG, Hongtao Wen, Bo Yang
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: 3D scene object segmentation, unsupervised learning

Abstract:
We study the hard problem of 3D object segmentation in complex point clouds
without requiring human labels of 3D scenes for supervision. By relying on the
similarity of pretrained 2D features or external signals such as motion to group 3D
points as objects, existing unsupervised methods are usually limited to identifying
simple objects like cars or their segmented objects are often inferior due to the
lack of objectness in pretrained features. In this paper, we propose a new two-
stage pipeline called GrabS. The core concept of our method is to learn generative
and discriminative object-centric priors as a foundation from object datasets in the
first stage, and then design an embodied agent to learn to discover multiple ob-
jects by querying against the pretrained generative priors in the second stage. We
extensively evaluate our method on two real-world datasets and a newly created
synthetic dataset, demonstrating remarkable segmentation performance, clearly
surpassing all existing unsupervised methods.

Decision: accept (spotlight)
Reviewer scores: 8, 8, 6

Reviews and discussion:
<thread>
creation date:
1730637855708

thread_type:
official_review

<thread_object>
creator: reviewer_bja3

<content>
summary:
To overcome the limitations of prior works, which require labor-intensive large-scale annotations, the authors explore the challenging problem of 3D instance segmentation in complex point clouds without relying on human labels.

They introduce a two-stage unsupervised 3D instance segmentation framework, GOPS: (1) in the first stage, an object-centric network is trained to learn generative object-centric priors, and (2) in the second stage, a multi-object estimation network then identifies similar 3D objects by querying against the learned priors and receiving objectiveness rewards through a reinforcement learning strategy.

Extensive experiments on two real-world datasets, ScanNet and S3DIS, and a newly created synthetic dataset demonstrate the effectiveness of GOPS with superior 3D instance segmentation performance.

strengths:
- The paper is well-organized, with clear explanations in both text and diagrams for each section.

- Based on a thorough analysis of prior methods and their limitations, the authors clearly articulate the motivation underlying their proposed approach.

- Each module in the two-stage pipeline is technically sound to improve the unsupervised 3D instance segmentation without relying on large-scale human labels.

weaknesses:
- The experiments for each module are somewhat lacking.
In addition to the points mentioned below, it would be helpful to provide experiments that validate the detailed performance of each module.
1) If the agent is indeed well-trained, it would be better to visualize the regions discovered by the agent or trajectories of the agent during exploration.
2) While the ultimate goal of GOPS is instance segmentation, the performance of the object-centric network seems to be of significant importance.
Therefore, providing relevant experimental results for the object-centric network would further solidify the effectiveness of the network.
For example, providing visualizations of the input point cloud of the trained object-centric network along with the corresponding recovered full shape would be beneficial. 

- The authors conducted training and evaluation solely on the chair class of real-world datasets (ScanNet and S3DIS).
While they evaluate performance on the synthetic dataset with six classes, the model's effectiveness in real-world scenarios, including various instance categories, remains unclear.
Is it possible to train and evaluate GOPS for six class objects from real-world datasets (ScanNet and S3DIS) using the object-centric network trained for the six class objects used in the synthetic dataset experiments?
Or train object-centric networks for six class objects in real-world datasets again?

- While GOPS does not require labor-intensive human annotations, training the two-stage GOPS frameworks seems to demand heavy resources. 
It would be helpful to clarify their implementation details, including the memory and time costs for the training process.

questions:
- Can the proposed object-centric network learn knowledge about multiple objects simultaneously?
In the experiments on the synthetic dataset, did the authors train separate object-centric networks for each of the six objects or utilize a single object-centric network that learns from all six objects? 

- Could the authors explain the rationale behind adding a self-attention block to the encoder in the object-centric network?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730652248606

thread_type:
official_review

<thread_object>
creator: reviewer_xek4

<content>
summary:
The authors propose a pipeline with multiple components to identify instances in 3D scenes without human annotations. First, they train an object orientation module, followed by training a generative prior network that is tasked with recovering objects with different kinds of noises and obstructions. This network acts as a filter in a reinforcement based learning setting, where a cylinder is used to search the entire 3D point cloud for instances that match the patterns learned by the generative object prior network. Overall, the method works well and the authors conduct ablations on the methods components.

strengths:
- The ideas presented in the paper are intuitive and make sense
- The writing is easy to follow and describes the contributions well
- The authors examine different contemporary learning mechanisms for their generative prior learning module, not just VAE but also diffusion 
- First teaching a network what an instance should look like, the iteratively searching the 3D space with this sort of filter to identify instances, makes total sense, is quite intriguing and well executed
- The authors have conducted a good amount of ablations to observe different aspects of their method

weaknesses:
- The paper investigates the sensibility of the threshold δc on 1 dataset, which is fine, but it would be interesting to know if this threshold is general or needs to be tuned individually for each dataset
- It would be interesting to know how a successful discovery of a mask influences the next iteration of the policy network 
- In Figure 5, it says the scene in cropped and then encoded, while in section 3.3, the authors seem to argue against random cropping. In the beginning, the container-based cropping should also be close to random right? Is the idea here that the cropping will become more targeted as the container is better navigated by the policy network? Please clear up the confusion
- What puzzles me is how the qualitative results with diffusion prior look better in some cases that the VAE based ones (Fig 6 row 1, Fig 7 row 3), but this is not reflected in the quantitative evaluation. Could you provide an intuition for this is the case? Can you also show failure cases?
- It would be nice if acronyms like SDF would be introduced. Even though this is an established method, its still also done for acronyms like ViT=Vision Transformer.

questions:
Overall, I think the idea of the paper is quite neat! The writing is well executed, the results well presented and the method well ablated. However, to make it a good submission, I think it would be important to learn about the following aspects:
- How sensible is the method to the threshold δc across datasets? 
- How does a successful discovery of a mask influence the next iteration of the policy network?
- It would be great to also have failure cases of the method

I think the outlined points are important to be addressed before acceptance, but I like the idea and it works well. Therefore, I will give a weak accept.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730706690136

thread_type:
official_review

<thread_object>
creator: reviewer_mm1f

<content>
summary:
The paper presents an unsupervised method to perform object detection on 3D scans based on reinforcement learning with an object prior model trained to generate objects of a specific category. The model performs a search on the 3D scan based on a policy network trained with reinforcement learning using as a reward the reconstruction quality obtained from a pre-trained generative model. The paper presents several experiments on real and synthetic datasets.

strengths:
- The paper addresses an important problem, unsupervised object detection, since in most real-world scenarios labels are not available for training.
- The paper is well-written and easy to follow.
- The use of reinforcement learning is novel and an interesting idea that might be a useful tool to solve other problems on 3D scene understanding.

weaknesses:
Although I think some of the ideas presented in the paper might have value for the community, I believe the framing of the paper and the evaluation is not adequate, and important baselines are missing. In the following paragraphs, I list my main concerns in detail:

- The method is presented as an unsupervised method. However, it relies on annotated data to train the generative model. Therefore, the method is not unsupervised, but weakly supervised, and has a greater advantage over other methods such as Unscene3D. In the paper is stated that those methods have an advantage since only the annotations of the correct class are kept, but those methods are designed to detect any object in the scene while the proposed method is trained specifically to detect a single type of object.

- The reinforcement learning search of objects in the scene will stop when an object is found. To find all objects in the scene it will require several starting positions for different searches. In the paper is indicated that several searches in parallel are used during training, however, this hyperparameter is not evaluated in the paper. An ablation study of this parameter and how many initialization are need to find all objects in the scene will help the reader understand the behavior of the method better.

- The reinforcement search will not be able to find all the objects in the scene in many cases. This is solved by using the objects found as pseudo-labels to train an instance segmentation model. However, this step I believe is not used for EFEM which also suffers from missing objects in the scene. This combination should be tested to show the effectiveness of the reinforcement learning algorithm. If not, we could train an instance segmentation model on the output of EFEM.

- The proposed method trains a Mask3D model on the pseudo labels generated by the reinforcement learning algorithm. However, Mask3D relies on superpoints to perform the instance segmentation prediction. The same superpoints are the ones used to annotate the labels in ScanNet, which gives Mask3D an unfair advantage over other methods since Mask3D then uses the perfect boundaries of the objects. Since the proposed method is based on Mask3D, it also has the same unfair advantage, which might explain the big improvement on ScanNet.

- The synthetic dataset is only evaluated against EFEM and not Unscene3D or Part2Object. These baselines should be included.

- The paper fails to cite in the related work a relevant work that also used a search on the scene to perform object detection based on an object pre-trained network:

Finding your (3D) center: 3D object detection using a learned loss
D Griffiths, J Boehm, T Ritschel
European Conference on Computer Vision

questions:
I would like to hear the opinion of the authors on the concerns I raised in the weaknesses section and clarify possible misunderstandings in my evaluation.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a pipeline with several modules to find objects in 3D scenes without any human labels, which is important because many real-world situations lack labeled data for training. A policy network trained with reinforcement learning is used as a reward for the reconstruction quality obtained from a pre-trained generative model to search the entire 3D point cloud for objects. Reviewers all agree that the paper is well-motivated and clearly presented. Detailed ablations have been conducted to analyze different aspects of the pipeline. All reviews are positive, with ratings of 8, 8, and 6. The area chair concurs with the reviewers' decisions and, therefore, recommends accepting the paper.