Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper addresses the task of unsupervised object segmentation from RGB-D
video data.  In particular, the authors introduce an object-centric
representation that is used to decompose each RGB-D image into a set of NeRFs
per scene object. Note the camera poses from which the RGB-D video data were
captured are considered to be known. The RGB-D input, is first converted into a
pointcloud which is then passed into a KPConv backbone, that extracts per-point
features. These features are subsequently used to compute the background
embedding and the soft attention masks per object, which are produced by
clustering the per-point features. Given these masks, they then predict the
pose of each object. The location of each object is simply the center of mass
of the points within each cluster, and the rotation is computed following the
minimum volume principle. Finally, conditioned on the object poses, they
extract features that are passed to a per-object NeRF. The proposed model is
evaluated on the task of unsupervised scene segmentation on the CLEVR dataset
and on the YCB dataset using both RGB-D videos and multi-view static scenes.
The authors consider two baselines: The ObSurf and slot attention and
demonstrate that their method outperforms prior works in terms of various
metrics such as the mean Intersection over Union (mIOU), the Adjusted Rand
Index (ARI) and the Mean Segmentation Covering (MSC).

strength_and_weaknesses: ## Strengths:
------------

1. The idea of disentangling the concept of "where to look" from "what to look"
is very interesting and is applicable to various tasks beyond oibject segmentation.

2. I also liked the author's solution for finding th object's pose using the
minimum volume principle.

3. I liked the additional results on scene editing in Section 4.3. However,
this section seems a bit unconnected with the rest of the text. So it might be
good to revise the text to better prepare the reader for this experiment.

4. I appreciate that for the CLEVR experiment, the authors also visualize the
corresponding depth maps, since their model can also predict them.

5. Overall the proposed idea is simple, novel and achieves very good results.

## Weaknesses:
-------------
1. The experimental evaluation is a bit weak. First of all, I think that the
datasets that the authors use for evaluating their model are rather simple,
namely the YCB dataset consists only of two scenes and the CLEVR has very
simply object arrangements. However, since also ObSuRF considers such a simple
experimental setup I am eager to overlook this. Nevertheless the experiments
could be improved. For example,  is there a reason why there is no comparison
with slot attention on the CLEVR dataset (see Table 1.)? Furthermore, can the
authors provide some additional qualitative comparison with ObSuRF and Slot
Attention also on the CLEVR dataset.

2. Looking at the original results from the ObSuRF paper, their segmentations
look quite plausible. Hence, I find it a bit weird that for the YCB dataset it
fails completely. Therefore, I think it is important to show a similar
comparison as in Figure 3 also for the CLEVR dataset.

3. A less important weakness of this paper in comparison to slot attention is
that it assumes that the number of objects in the scene is known. While I
believe that this constraint does not impose such a strong limitation it is
still a limitation that is useful to be addressed.

4. Since the authors only consider very simple datasets in their evaluation, I
am wondering how generalizable this model is to more complex scenes.

clarity,_quality,_novelty_and_reproducibility: I think the paper is overall nicely written. Some section can be improved but
in general it is easy to follow. To the best of my knoweledge the proposed
model is novel and I think reproducing the presented results won't be an issue.
summary_of_the_review: Overall, I like the idea of this paper but I am a bit skeptical regarding the experimental evaluation. I think it is essential to provide the additional results I mentioned above on the CLEVR dataset in order to ensure a stronger submission. Below are some more comments/questions and suggestions:

1. Symbol $\zeta_f$ does not appear in Figure 2. I recommend to add in Figure 2
to avoid confusion. From the description in the last sentences of Section 2.1
it is not 100% clear how $\zeta_f$ and $\zeta_c$ are computed from $\zeta$.

2. Can the authors provide some more details regarding the computation of
$\mathbf{b}z^{\textbf{bg}}$ from $\zeta$? I think that providing additinal details on
this would significantly improve clarity.

3. If I understand correctly the Instance Colouring
Stick-Breaking Process, described in Section 2.1.1 is simply a clustering
mechanism. I believe that it would significantly improve the paper's clarity if the authors
added one/two sentences in the beginning of this section pointing out this
fact. Starting with all the specific details make understanding quite hard. In
particular, regarding section 2.1.1, I believe that the authors should revisit
the text and try to provide some intuition regarding the things they describe.
For example, why is the stochastic ordering of the mask important? I think that
giving intuitive examples always facilitates understanding.

4. From Section 2.1 it is not clear that the feature embedding $\zeta_f$ refers
to point features at a particular time step t (see Section 2.1.2). This is
quite confusion and should be clarified in Section 2.1

5. The reconstruction results in Figure 7 in the supplementary seem to have
some artifacts. I am wondering to the authors use also importance sampling in
their NeRF, or do they only rely on a coarse NeRF model?

6. How sensitive is the IC-SBP module to initialization?
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes an object-centric 3D inference method that is trained in a self-supervised way using a variational auto-encoder framework and neural radience fields. While a large number of methods in the literature address object-centric self-supervised segmentation, very few also learn to infer 3D pose and shape while doing so. The authors propose using the existing video-specific instance colouring stick-breaking process to encode the input imagines into segmented attention masks, and propose a novel mechanism to refine and estimate each detected object's pose. Given disentangled latents pose (where) and content (what), novel images can be rendered using a NeRF MLP for any viewpoint. Once trained ObPose obtains near-perfect segmentations on a number of datasets and is shown to outpeform ObSurf as well as a number of other baselines.  
strength_and_weaknesses: **Strengths**
- The use of strong inductive biases allows this self-supervised model to obtain representations suitable for domains in which detailed scene understanding is required. These biases are: object-centric segmentations, inference of disentangled 3D pose and shape, and the ability to represent objects in explicit 3D voxels.
- The authors propose a novel method for shape representation via voxelization using the "minimum volume principle", an efficient approach that allows end-to-end training.
- Results highlight the model's ability to obtain precise 3D (and 2D) segmentations from just one or a few images, and to consistently outperform closely related work in this task on CLEVR-3D and YCB datasets.

**Weaknesses**
1. While the minimal bounding-box volume selection is an interesting heuristic for pose estimation, it's unclear from the paper if the method proposed learns consistent canonical poses across instances of the same category. Related work [1, 2] in self-supervised pose estimation methods ultimately evaluate their method against some form of GT category-level poses. This work would be strengthened if similar evidence is provided (in the form of additional evaluations against GT poses) or if at least extensive examples of estimated poses are shown. Note that if the above estimated poses are not consistent, the fast shape evaluation may also fail to produce consistent shapes across instances of the same category.
2. Experimental results suggest that using the inferred rotations do not seem to help, which weakens the significance of one of the paper's novelties (i.e. the benefits of the where->what pipeline). That said, shape estimation is used to refine the object location prediction via $L_{where}$. Thus, for the whole approach to be an interesting contribution, the authors should show that location refinement (delta T) is important to the overall model performance. 
3. Finally, I am concerned regarding the fairness of the comparison with ObSurf. In the most challenging YCB datasets, the authors only report ObSurf w/o overlap loss. In ObSurf's paper it is shown to be an important loss. In my eyes, for this comparison to be fair, the authors should perform a reasonable hyper-parameter search and not just use ObSurf's "default settings for 3D". Usually optimal settings may vary for each dataset and indeed ObPose's settings also change between CLEVR-3D and YCB as per Table 3 in the appendix.

**Questions / suggestions**

- What happens if $N_k$ is small in the first time step?
- Requiring depth in general is a stronger requirement than many methods in the literature, including ObSurf which only uses depths for training signal. Due to ObPose's point-cloud approach, it cannot be trained without depths, but for the sake of fairness the authors can compare with ObSurf that additionally conditions the encoder with depths. 
- What are failure modes of the method? It would be enlightening to see more challenging examples i.e. clutter and occlusion from YCB.
- How do $L_{att}$ and $L_{scope}$ (which don’t seem essential) affect performance?
- Also, since $L_{att}$ is different for some datasets, this should be stated in the main text.
- Is the IC-SBP baseline exactly ObPose but without the use of pose-based transformation of features?
- The use of tanh activation to compute the global density means densities can be negative. Are these the densities the ones used for volumetric rendering? If so breaks the non-negativity assumption of volumetric rendering. 

[1] Li et al. Leveraging SE(3) Equivariance for Self-supervised Category-Level Object Pose Estimation from Point Clouds. NeurIPS 2021

[2] Zero-Shot Category-Level Object Pose Estimation. ECCV 2022
clarity,_quality,_novelty_and_reproducibility: The paper is well written and method is clearly described. Related work also covers the most relevant works in the literature to my knowledge. Perhaps the authors should be more precise about what type of 6D pose their method is claimed to estimate, as there are a number of different perspectives in the literature.
The combination of IC-SBP, estimation of where and what latents, variational inference, and object-centric NeRF decoding is certainly novel (although none of these methods are new). Their main technical novelty is the specific bound-box volume method for pose estimation and the way they use it to efficiently voxelize a NeRF object and use it to refine the pose. This can be a useful contribution to the literature but as per the weaknesses above, further experiments would help clarify the significance of this novelty wrt the model's performance.
Finally, I believe the method is presented in sufficient detail to be reproducible, and the datasets used are well known.
summary_of_the_review: The direction of this work is important for robotics and this method can be a useful way to learn detailed representations from images without requiring large datasets of segmented objects. The combination of techniques is novel and the authors propose a new heuristic to estimate and refine the pose of the segmented objects. Further clarification on the method's ability to detect pose is important, as well as additional experiments wrt. ablations of some losses and a fairer comparison with ObSurf. I believe these points are necessary to establish the significance of this work. As it is now, I am leaning towards rejecting it.

---
**Post rebuttal update:**

I thank the authors for addressing my questions and concerns, and for updating the manuscript with improvements in a wide range of aspects. Most importantly, ensuring a fairer comparison with ObSurf as well as the addition of another dataset. I believe the paper is in a better state now, but I still think it's insufficient to raise my score due to the fundamental limitations below.

As reviewer aWJo also shares, two of my main concerns not been fully resolved after the rebuttal.

First, it is still unclear to me what the benefits are of the pose refinment novelty via the minimum volume principle. As acknowledged by the authors, conditioning the encoding on rotation doesn't result in clear benefits (while conditioning on location does); and neither have the authors provided evidence that the location refinement (delta T) is important to the overall model performance. It is also still unclear if the poses reflect consistently the real poses of the objects which would justify the hypothesis the authors refer to regarding lower variance of appearance representation. In other words, if poses estimated are arbitrary for different instances of similar classes (e.g. chairs of different shape but same appearance) why should we expect the appearance representations of them to be invariant?

Second, while the comparison with ObSurf is now fairer, it is still difficult to assess this work in the wider 3D scene understanding literature. 
This method requires RGB-D whereas most unsupervised segmentation methods do not. Methods such as ObSurf and Object Scene Representation Transformers (Sajjadi et al. 2022) work without depth and also are able to produce great reconstructions (especially the latter). ObPose's renders are fairly poor (Fig. 7) which casts doubt about how well it can work (i.e. that relies on reconstructions) on the increasingly richer and more realistic datasets such as MultiShapenet-Hard, e.g. scenes with heavy clutter and complex textures. While this may be improved with better latent representations of z_what and z_where and a stronger NeRF decode, all in all it would involve substantial changes to the method and paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a method for the unsupervised discovery of object-segmented 3D representations with explicit pose parameters, given an RGB-D input image. Building on the ObSuRF model, the proposed ObPose system consists of an encoder inferring pose and shape parameters for a set of objects, and a NeRF decoder which processes these into a set of composable 3D volumes. It is demonstrated that ObPose yields improved segmentation results on new synthetic datasets made up of YCB objects. Additionally, scene editing capabilities are demonstrated.
strength_and_weaknesses: Strengths:
 - The paper investigates the interesting question of discovering disentangled 3D representations from RGB-D input.
 - While the possibility of introducing explicit pose parameters was discussed in the ObSuRF paper, to my knowledge, this work is the first to do so. Explicit pose parameters have the potential to simplify object manipulation for downstream tasks.
 - The method shows improved segmentation results on a new synthetic benchmark.

Weaknesses:
 - The provided quantitative results only cover image segmentation results on synthetic benchmarks. Performance on CLEVR3D is similar to ObSuRF. While the model outperforms its baselines on the new YCB images, it remains unclear why new data was generated instead of using the Multishapenet dataset from ObSuRF, given that the two appear to be of similar complexity. In any case, it cannot be claimed that ObPose improves state of the art in 3D segmentation, as the considered datasets are significantly simpler than MSN-hard as used by OSRT [2].
 - The model's most important novel capability, its ability to infer explicit pose parameters, is not directly evaluated. This could be done by comparing to ground truth poses, or computing 3D IoU with respect to ground truth bounding boxes. As a result, it remains unclear if the model's pose parameters function as claimed. Indicating the inferred poses in Figures 3/4 would also help.
 - The softmax formulation in Eq. 8 suppresses the rendering of overlapping objects. However, it does not seem to prevent multiple slots from *representing* the same object with high $\sigma_i$. If these are then rendered separately, e.g. in a scene editing context, the previously suppressed overlaps would emerge, potentially hurting compositionality.
 - The way the paper uses voxels is unclear to me: The term seems to indicate that space is discretized into cubes, but no details are provided on this, and the equations appear to be consistent with standard, continuous NeRF rendering. 
 - In general, the paper appears somewhat unfocused, and its presentation could be significantly improved (see below).
 - Contrary to what is stated in this paper, ObSuRF does not use depth information at test time, only as a training signal. ObSuRF's encoder only receives RGB input. Was this changed to allow for a fair comparison?
 - Explicit pose parameters are not strictly necessary for scene editing, as suggested in section 4.3. Adding, deleting and moving objects is also possible with ObSuRF, with the only qualitative difference being that pose parameters allow stating positions in absolute terms, instead of just relative movement. It might be the case that doing this with ObSuRF leads to more artifacts due to imperfect segmentations, however.
clarity,_quality,_novelty_and_reproducibility: **Clarity**: The presentation is quite dense, with the majority of the paper being dedicated to a detailed description of the method. Even so, important aspects, such as the overall probabilistic setup, and the role of voxels, appear along the way without the necessary introduction. Clarity could be improved by focusing on a smaller number of contributions (e.g. cutting the video aspect), abstracting away details into the appendix (such as the specifics of the *where*-module), and adding an overview outlining the general setup. Some of the formulas could use some editing, e.g. using large parenthesis in Eq. 5, and fixing the indices in Eq. 8. Referring to the new synthetic video dataset as "YCB-Video" is misleading, as this term is frequently used for the real world video dataset introduced here: https://rse-lab.cs.washington.edu/projects/posecnn/

**Novelty**: Constructing a 3D model with explicit pose parameters is novel. The paper however seems to miss other recent unsupervised 3D scene understanding methods, namely uORF [1] and OSRT [2] Given the subtle differences between their problem settings, I think it is acceptable to only compare to one baseline, however.

**Reproducibility**: Despite the open questions about the method, the availability of code, and the details in the appendix help reproducibility.

[1] https://arxiv.org/abs/2107.07905

[2] https://arxiv.org/abs/2206.06922
summary_of_the_review: Overall, while the paper proposes an interesting variant of ObSuRF, there are currently too many issues with the experimental evaluation to recommend acceptance. If the authors can show that the pose parameters work as intended, and that their improvements extend to existing hard benchmarks, I am willing to increase my evaluation.

----

**Post rebuttal update:**
I thank the authors for their response and the updates to the manuscript. I think especially the MSN-easy experiment is an important (and impressive) addition. However, some important issues remain.

**Unfair experimental setup**: The fact that ObPose leverages depths at test time, while ObSuRF doesn't, renders the main experimental results of the paper unfair. The fact that this was swept under the rug in the manuscript is a significant red flag. The added "ObSuRF with depth" baseline cannot fully address this issue:
  * No details were provided on how it was constructed.
  * ObPose's encoder is directly tailored to RGB-D input in many ways, e.g. by using KPConv layers, and by explicitly making use of input point clouds to identify object locations. ObSuRF's encoder is designed for RGB input, so the comparison is still not particularly fair.
  * The more meaningful "ObSuRF with depth" baseline is only provided in Table 2, for the YCB experiment, not in Table 1, for CLEVR3D and MSN.
  * The manuscript still incorrectly states that ObSuRF operates on RGB-D images, and does not make this issue transparent.

I recognize that 3D scene understanding is a developing paradigm in which many problem formulations are possible, and for many of them, there are no appropriate baselines. But these issues need to be treated transparently and put into context. If the authors are looking to argue that their explicit treatment of object location is the key driver of performance for ObPose, an appropriate experiment would be to compare it with an ablation which still leverages RGB-D input, but doesn't contain the pose module. In a setting with RGB-D input, ObSuRF can only provide a lower bound on the performance we may expect.

**Pose Evaluation**: The authors have clarified that the goal of their model is not pose estimation. I agree that there is no need to compare to pose estimation state of the art, or anything like that. However, if the story of the paper is that explicit pose parameters lead to improved segmentation performance, it is still crucial to convince the reader that these pose parameters actually work as advertised. An experiment to that effect is still missing from the main paper.

**Clarity**: No major changes have been made to the structure of the text, as far as I can tell. As a result, my clarity concerns remain.

I believe these issues are too significant to recommend acceptance at this time. I am therefore leaving my score unchanged.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a model OBPose for unsupervised 3D object segmentation from RGB-D images/videos. OBPose first represents each object as disentangled location and appearance information, then re-renders the scene with a NERF decoder. Experimental results prove the effectiveness of the proposed object representation.
strength_and_weaknesses: Strengths:

1. Compared to the slot-attention-based methods that represent each object as a single latent code, this work explicitly disentangles object location and appearance information, which gives more accurate scene-decomposing results.
2. Ablation study in Table2 justifies its superiority against the slot-attention-based method and IC-SBP-only method.

Weakness:
1. The novelty is limited. (1) The key components such as IC-SBP and canonical pose representation are mostly borrowed from previous methods [7][19]. (2) The idea of representing objects as disentangled location and appearance information is also common, e.g., BlockGAN[R1].   
2. The loss function is complicated and the contribution of each component is unclear from the results of the experiments.   
3. It would be more convincing if the authors could provide some quantitative comparisons on the unsupervised 6D pose estimation task.   

[R1] Nguyen-Phuoc, Thu H., et al. "Blockgan: Learning 3d object-aware scene representations from unlabelled images." Advances in Neural Information Processing Systems 33 (2020): 6767-6778.



clarity,_quality,_novelty_and_reproducibility: This paper is well-written and the overall quality is good. I believe it is reproducible. The novelty is somewhat incremental.
summary_of_the_review: I have reviewed this paper in NeurIPS, and the quality of this submission has been improved compared with its previous version. It is nice to see the experiment results and analysis on CLEVR-3D dataset. Therefore, I think it is marginally above the borderline.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper aims to learn a factorized scene representation, separately encoding object location (where) and appearance (what). The reviewers agree that this is a hard and important problem to solve and appreciate the initial results on the simple dataset. However, there are a few important issues that remain after the rebuttal and the discussion. These issues include the fairness of the comparison (e.g., whether depth is needed as input), the quality of the results, the technical novelty, and the presentation. Eventually, reviewers and the AC agree that this paper is promising but below the bar.  The authors are encouraged to revise the paper accordingly for the next venue.
justification_for_why_not_higher_score: There are multiple concerns including the fairness of the comparison (e.g., whether depth is needed as input), the quality of the results, the technical novelty, and the presentation.
Decision: Reject