ObPose: Leveraging Pose for Object-Centric Scene Inference and Generation in 3D
Authors: Yizhe Wu
Oiwi Parker Jones
Ingmar Posner
Venue: Submitted to ICLR 2023 (2023)
Keywords: Object-centric scene inference
Object-centric representation learning
Scene generation
3D

Abstract: We present ObPose, an unsupervised object-centric inference and generation model which learns 3D-structured latent representations from RGB-D scenes. Inspired by prior art in 2D representation learning, ObPose considers a factorised latent space, separately encoding object location (where) and appearance (what). ObPose further leverages an object's pose (i.e. location and orientation), defined via a minimum volume principle, as a novel inductive bias for learning the where component. To achieve this, we propose an efficient, voxelised approximation approach to recover the object shape directly from a neural radiance field (NeRF). As a consequence, ObPose models each scene as a composition of NeRFs, richly representing individual objects. To evaluate the quality of the learned representations, ObPose is evaluated quantitatively on the YCB and CLEVR datatasets for unsupervised scene segmentation, outperforming the current state-of-the-art in 3D scene inference (ObSuRF) by a significant margin. Generative results provide qualitative demonstration that the same ObPose model can both generate novel scenes and flexibly edit the objects in them. These capacities again reflect the quality of the learned latents and the benefits of disentangling the where and what components of a scene. Key design choices made in the ObPose encoder are validated with ablations. 

Under review as a conference paper at ICIR 2023
OBPOSE: LEVERAGING POSE FOR OBIECT-CENTRIE SCENE INFERENCE AND GENERATION IN 3D
Anonymous authors Paper under double-blind review
ABSTRACT
We present OBPOSE, an unsupervised object-centric inference and generation model which learns 3D-structured latent representations from RGB-D scenes. spired by prior art 2D representation learning, OBPOSE considers a factorised latent space, separately encoding object location (where) and appearance (what). OBPOSE further leverages an object's 's pose (i.c. location and orientation), defined via a minimum volume principle, as a novel inductive bias for learning the where component. To achieve this, we propose an efficient, voxelised approximation ap- proach to recover the object shape directly from a neural radiance field (NeRF). As consequence, OвPОSE models each scene as composition of NeRFs, richly rep- resenting individual objects. To evaluate the quality of the learned representations, OBPOSE is evaluated quantitatively on the YCB. MultiShapeNet,, and CLEVR datatasets for unsupervised scene segmentation. outperforming the current state-of- the-art 3D scene inference (ObSuRF) by significant margin. Generative results provide qualitative demonstration that the same ORPOSE model both generate novel scenes and flexibly edit the objects in them. These capacities again reflect the quality of the learned latents and the benefits of disentangling the where and what components of a scene. Key design choices made in the POSE encoder are validated with ablations.
INTRODUCTION
recent years, object-centric representations have emerged as paradigm shift in machine perception. Intuitively, inference or prediction tasks in down-stream applications are significantly simplified by reducing the dimensionality of the hypothesis space from raw perceptual inputs, such as pixels or point-clouds, to something more akin to a traditional state-space representation. While reasoming over objects rather than pixels has long been the aspiration of machine vision research, it is the ability to learn such a representation in an unsupervised. generative way that unlocks the use of large-scale, unlabelled data for this purpose. As consequence, research into object-centric generative models (OCGMs) is rapidly gathering pace.
Central to the success of an OCGM are the inductive biases used to encourage the decomposition of scene into its constituent components. With the field still largely in its infancy, much of the work to date has confined itself to 2D scene observations to achieve both scene inference (e.g. Eslami al.. 2016; Burgess et al., 2019; Greff et al., 2016; 2017: Locatello et al., 2020; Engelcke al., 2019: 2021) and, in some cases, generation (e.g. Engeleke et al., 2019; 2021). In contrast, unsupervised methods for object-centric scene decomposition operating directly 3D inputs remain comparatively unexplored (Elich et al., 2022: Stelzner et al., 2021) despite the benefits due the added information contained in the input. As a case point, Stelzner et al. (2021) recently established that access to 3D information significantly speeds up learning. Another benefit is that, for the parts an object visible to 3D sensor, object shape is readily accessible and does not have to be inferred, either from a single view (e.g. Liu el al., 2019; Kato et al., 2018) or from multiple views (e.g. Yu et al., 2021; Xie et al., 2019). We conjecture that object shape can serve as a highly informative inductive hias for object-centric learning, As we elaborate below, we reason that the asymmetry of a shape can be used to discover object's pose, and pose can help to identify and locate an object in space.
Here we present OBPOSE, an unsupervised OCGM that takes RGB-D images (or video) as input and learns to segment the underlying scene into its constituent 3D objects, as well as into an explicit background representation. As we will show. OвPOSE can also be used for scene generation and editing. Inspired by prior art in 2D settings (Fislami et al., 2016; Crawford & Pineau, 2019; Lin et al., 2020; Kosiorek et al., 2018; Jiang et al., 2019; Wu et al., 2021), OBPOSE factorises its

---

Under review as a conference paper at ICTR 2023
latent embedding into a component capturing an object's location and appearance (where and what components, respectively). This factorisation provides a strong inductive bias, helping the model disentangle its input into meaningful concepts for downstream use.
key contribution of OвPOSE the introduction of pose (i.e. location and orientation) as novel
inductive bias.
OBPOSE is not a pose-estimation model. Rather. OBPOSE infers pose information from an object's shape to reduce apparent variance to simplify the learning (see fig. 11 the appendix) - ultimately
model's what component in 3D use in downstream tasks like segmentation and scene editing. In an OGGM, the intuition that the variance we observe an object's pose should be made invariant in the latent space. OBPOSE infers pose information without supervision, using a minimum volume principle defined using the tightest bounding box that constrains the object. Effectively, the tightest bounding box will reveal asymmetries in an object's shape, if there are any. which can be used to constrain the object's orientation. We further propose voxelised approximation approach that recovers object's shape in computationally tractable way from neural radiance field (NeRF) (Mildenhall al.. 2020). Although the recovery of object's 's shape from a NeRF can be prohibitively expensive, approach allows this be integrated efficiently into the OBPOSE training loop.
In series of experiments, ОвРОSЕ oulperforms the current state-of-the-ar in 3D scene inference, ObSuRF (Stelzner al., 2021). by significant margins. Evaluations performed on the CLEVR dataset (Johnson et al., 2017). the MultiShapeNNt dataset (Stelzner et al., 2021: Chang al., 2015), and the YCB dataset for unsupervised scene segmentation (Calli al., 2015), in the latter case using both RGB-D moving-objes ОвРOSE encoder serves to validate the design decisions distinguish its use of attention from alternative attention mechanisms represented by Slot Attention (Locatello et al., 2020) and GENESIS-v2 (Engeleke et al., 2021). summary, key contributions of this paper are: (1) new state-of-the-ar unsupervised scene segmentation model for 3D, OBPOSE, together with insights into its design decisions; (2) novel inductive bias for 3D OCGMs, pose, together with its motivation; and (3) general method for fast shape evaluation from NcRFs
(video) and multi-view static scenes. ablation study on
pose,
OBPOSE
Object-NeRF
Time
RGB-D
shhat
wheer
Clpect coord harts Wherld wwwod hare
Origs
Coodioie Orgin
Condiir Orign
Figure 1: Overview. Front-view observations can be fed into OBPOSE (e.g. video of objects on table, top left). OpPOSE then reconstructs the sparse voxelised point cloud and normalised pase of each object (two objects shown, top right). Each point within slot is coloured to represent specific object. Note that while object may move in the world frame (middle row), appears stationary when viewed from the perspective of its normalised frame (bottom row).
2

---

Under review as a conference paper at ICLR 2023
METHODS
OBPOSE takes RGB-D videos (or images) as input and learns to segment scenes into a set of foreground objects, with single background component. The location and orientation of objects can be estimated from the respective shapes which we reconstruct using NeRFs (Mildenhall et al., 2020). In contrast to previous works (Stelzner et al., 2021) where object NeRFs have to model object appearance and location jointly, each object NeRF in OвPOSE only models the 3D geometry and the texture of the objects by conditioning on the predicted object locations and orientations. To perform location and orientation conditioned inference, ОвРОSE clusters pointwise embeddings that are encoded from a standard (KPConv-based) backbone (Thomas et al., 2019) into a soft attention mask for each object using instance colouring stick-breaking process (IC-SBP). Then where-inference step predicts the object locations and orientations given the object-wiss attention masks from the IC-SBP. Finally, what-inference step encodes the appearance and shape information (conditioned on the object locations and orientations). This information is then decoded into NeRFs and all of the NeRFs are composed with the background component reconstruct the original scene. A schematic overview of the OвPOSE architecture provided Figure 2.
Tig Cla
IC-SBP
x
Se
m
#1
athore
2 tskuct
0 C1
KPConv
:
mng
mlerr #R
mbbet K
Tgg CK
NeRF Decoder
Figure 2: Model architecture. Given an input RGB-D image x, the KPConv backbone extracts point embeddings &e and a background component gbz The point embeddings are clustered by the IC-SBP into soft attention masks, mi ... mm, for each object slot. Given these object masks, a where module infers the location of cach object encoded by z. wherr Conditioned on object poses, which are recovered via the minimum volume principle, the what latents, z are encoded and then decoded into object NeRFs. Object NeRFs are finally composed with the background component reconstruct the observed scene.
2.1 ENCODER
The imput observation x, is RGB-D image of height H and width W for each time-step / E {1..... (1...7) 7}.
RGB-D depth images are converted into the point clouds using the known camera parameters (extrinsic and intrinsic). The point clouds and the RGB channels of x, are then concatenated and encoded into a point embedding z <HxXWixD) This achieved using a U-Net-like backbone module (Ronneberger ct al., 2015) consisting of several KPCony layers (Thomas et al., 2019). A KPConv layer is an extension of 2D convolutional neural network (CNN) layer for point clouds, which preserves the translation invariance properties of CNNs. For computational efficiency. output embeddings are upsampled to the resolution of the first downsampled embedding using trilinear interpolation. An aggregation layer is required summarise the global information for encoding the background information into the latent representation 23 For classification tasks, average pooling was adopted for aggregation (Thomas et al., 2019). However, the pooling is not suitable for capturing the location information of the input features. Instead, we use a OPCoov-aggregate layer to aggregate the output features, initialising a convolutional kernel at the coordinate centre. In this way, global spatial information is preserved and encoded in the relative distances between the output features and the kernel points. Following prior work (Engelcke et al., 2021), we additionally build two heads (point- wise MLPs) upon the output embedding 6, with one predicting the colour embedding HDXWixDy for other encoding tasks.
xDD, for the IC-SBP, and the other the feature embedding f Sea
3

---

Under review as a conference paper at ICIR 2023
2 INSTANCE COLOURING
PROCESS FOR VIDEO
process (IC-SBP) is a clustering algorithm that takes point embeddings Sc,t <HiWWiDD as inputs and outputs K predicted soft attention masks my e [o, 1] j The stick-breaking process (Burgess et al., 2019) guarantees that the masks normalised:
An instance colouring
m dr
my 8k-1 (k,
mg
Sg.
where the scope 83 10, 1] Mixw tracks which pixels have not yet been explained. Sy is initialised and updated as follows:
5o Mi S St-1 9 (1 (1-ox)
(2)
The alpha mask ar is computed the distance between cluster seed Si,; and all individual point embeddings according kemnel dr. Readers are referred to (Engeleke al., 2021) for details on kernel selection; our implementation uses Gaussian kernels. The IC-SBP prevents the model from learning a fixed order of decomposition. achieves this by selecting the cluster seed the spatial location (i. f) according to the argmax of score mask 5 = Sg @ u with u being a score map sampled from uniform distribution U(0, 1) MixW,, The score map ensures the stochastic ordering of the mask my-
Although the IC-SBP is by default agnostic with respect to the slots, there are nonetheless special components in scene segmentation such as the background We model slots like these that contain special components by passing the pixel embeddings Seat (MLP) o to compute a pre-scope s":
XWxxD through a multilayer perceptron
s softmax(p(() E R HRWWWWR
(3)
Here K' -1 11 is the number of special slots. By convention, we take the last channel of s" to be the scope So in the IC-SBP. In the present study, we set K' = 2 for only modelling the background. A lermination mechanism is implemented by sampling from the remaining scope šę hefore computing the alpha mask or at cach step k. Each point in the scope 5ę is treated an independent Bernoulli distribution with the probabilities being the scope value and perform point- wise sampling. The current slot is set the idle state when all sampled values are zero, which indicates that the whole observation has been explained by previous slots.
For the video input, extend the original IC-SBP include additional propagation step. The cluster seed &.,j can he used as ID for each slot throughout the video. The cluster seed sampled f is stored and used compute the alpha masks for frames of r 2. The idle slots detected the masks my, the
[ will remain in idle states during propagation. To ensure the normalisation SBP operation takes the remaining scope 8g as the last component mask my. This does not have an associated seed for propagation. We run one more step of the SBP for the last component and flag the remaining scope Sk as unused. Concretely, we add an additional penalty loss to encourage the final remaining scope Sg to be zero everywhere, thereby motivating the model to explain the whole observation using only the previous slots.
2.1.2 OBJECT LOCATION AND ORIENTATION CONDITIONED ENCODING
To facilitate disentangled encoding of what and where, we firstly introduce shape-based object location and orientation estimation. We note that a point cloud Pr, = {pyli E {l.... N2 object k at time step can be viewed as discrete sampling from the object surface and thus preserves the shape information of the objects. Intuitively, the location of the object T,, can initialised at the centre of mass Pr.::
1
Σ P/
T
(4)
N jE((.....NN.
with Nj being the number of points of object k. For the orientation the object, which can be represented by a rotation matrix R E SO(3), we propose the following minimum volume principle to define a unique thape-based object orientation: given a set of points, we set the orientation of the object represented by those points to the orientation of the tightest bounding box that contains those points. Concretely, we find this bounding hox by first transforming the points under several selection rotation matrices Rs SO(3) and then computing the volume of the axis-aligned bounding boxes (AABBs) that contain these points. Selection rotation matrices are generated equivolumetric grids on the SO(3) manifold (Murphy et al., 2021; Yershova et al., 2010). This method is based the
4

---

Under review as a conference paper at ICIR 2023
HEALPix method of generating equal area grids on the 2-sphere (Gorski et al., 2005). Due to the symmetry property of the bounding boxes, the smallest bounding box can have multiple solutions, e.g. swapping the a-axis and the y-axis of the coordinates of the bounding box will result in another bounding box that has the same volume. Therefore, we select all the bounding boxes whose volumes are in the range [Vmin (1 + (3) Vmin to account for this issue, where Vmin is the minimum volume of all the AABBs and B is small tolerance factor. We pick the AABB whose orientation is closest the world coordinates and whose orientation is represented by the identity matrix I R for the first time step, and otherwise to the orientation in the previous time step. The distance dsor3y measured by the geodesic distance on the SO(3) manifold:
tr ir(R,, R
dso(3) = arccos
(5)
The points P, E Pr, are then transformed to their object coordinates given a pair of pose parameters (T. R as follows:
k (p. T) 2 (R) (P)
(p)
(6)
The bounding box size y R is initialised to a sufficiently large number for all objects. All points that are out of the bounding box are discarded so that the spatial coordinate of each point is in the range of 1].
For each object slot at time step 1, we find its associated point cloud P., by computing the argmax over the attention masks m, x predicted from the IC-SBP. The point features < at time step are also masked by the attention masks, i.e. L1,A = fr. c no grad x.). The object location Ty, estimated from the observed points Pr. is, however, biased toward object surface as the object is only partially observed. We thus use a where module that takes P,, and 6. as input and predicts a A to correct the bias. To learn a relative translation we first transform P,, using the pose parameters (T I) as in eq. (6). Concretely, the where module is an encoder that consists of several КPConv layers and a aCCon-agggeeatee layer as the final output layer. The encoded embedding is then fed to recurrent neural network (RNN) with its hidden states later being decoded into the mean and where standard deviation of the posterior distribution qzz),, (xst) parameterised as a Gaussian. The L.k is decoded to the &T,, through an MLP fuhere such that:
AT, = Tmaxtanh( pwhore ( (2))Arr))
(7)
with T Tiaxx B. being the maximum delta translation. Given the updated object location T., = Tr, + AT,,,, we encode the shape and the appearance of the object from the observations transformed in the object pose (Tr,,, R. Similar to the where module, the posterior distribution (z (( is also parameterised by KPConv encoder appended by an RNN. If the point number N₂ is small, we instead initialise the object pose Tr, R with the object location from the last time step, i.e. (f.L.x. I) during the propagation.
We use two RNN networks to model the prior distributions p(z (aaber)) and p(z) (zuhar) whose hidden states are decoded by an MLP to the mean and standard deviation. The prior distribution at 1 0 are Gaussian distributions of zero mean for both the virer and the z zwiat.
2.2 DECODER
To explicitly model the 3D geometry of scenes, we represent each object and the background as Neural Radiance Field (NeRF) (Mildenhall et al., 2020). Each NeRF is function parameterised by an MLP that maps the world coordinates p and the viewing direction a color value e and a density value or One limitation is that given NeRF can only represent single scene and cannot condition on the latent encoding. The approach has therefore been extended (Schwarz et al., 2020; Niemeyer & Geiger. 2021) to learn the mapping f (p. d. z) - (c. This introduces a mapping from the latent encoding z to modifications to the hidden layer outputs of the NeRF MLP. To compose the individual NeRPs into a single scene function, previous approaches (Stelzner et al., 2021; Niemeyer Geiger, 2021) combine the density values by summation operation. i.e. o Σx_ oxe This way of composing NeRFs is equal to superposition of independent Poisson point processes (Stelzner et al., 2021). The scene colour can then be computed as weighted mean ΣX Gg We know that voxel should not be occupied by multiple objects. To account for this property, (Stelzner al..
5

---

Under review as a conference paper at ICLR 2023
2021) proposes an auxiliary loss to penalize the difference between the sum of the density values and the maximum density values of all components, as simply composing NeRFs with a summation does not suffice. This auxiliary loss, however, requires a warm-up training strategy which introduces additional pperrppaameters for different datasets. We propose to model the scene function as:
Σ softplus (172) . orsoftmax (crz)
TTaa manhh
(8)
In our case, dn can be interpreted as the logits the probability, indicating whether this voxel occupied or not, and 0% is the normalised density with range of [0, dmax] For the colours, we compute the weighted mean: d'uas and model the probabiliy of whether the voxel has been occupied through tanh. The softplus function ensures that no component can contribute negative occupancy. We use the softmax function to account for the fact that the objects should not overlap, without needing to introduce extra hyper- parameters. To estimate the shape of the objects reconstructed from the NeRFs, we would like fast approximation approach, the full evaluation of the volumetric rendering computationally expensive. Inspired by prior work (Liu al., 2020), we divide each object bounding box into S sparse voxels along cach dimension. The occupancy at these voxel centres Pv can then be evaluated by or tanh (softplus (f Ze ))). We denote voxels as occupied or > Oyy, with the oyy being threshold. Given the set of the occupied voxels and their centre positions Pv. we can recover the object centre Tellape and the object orientation R Rehupe as discussed in 2.1.2.
_ Σt ogC We upper bound the density by max E R.
TRAINING
Training a NeRF with known depth requires relatively few evaluations. For example, Stelzner al. (2021) propose to use only two evaluations of the NeRF for each training iteration, i.c. one evaluation the surface and one evaluation at points between the camera
surface. To avoid learning an extremely thin surface, the points sampled at the surface are sampled between the depth range d dsurface 6]. with the ó denoting the surface thickness. The observation loss can be divided into two terms, i.c. a texture loss term:
Lootour log
N(xlc2, o'da) o @-aarface /Ormax
(9)
with asa denoting fixed standard deviation, and depth loss term (Stelzner al., 2021):
Laepth lpg(or(p"utaee)) (rp"")/p"
(10)
where -ar is a probability density of the point par being sampled. The latent embedkling is regularized using the KI. term:
Lx o xxt) (=)))lpo(z()zz))
(11)
An L2 loss is used to supervise the where module:
Lwttene
Ext. T
(12)
For the attention mask of IC-SBP, the learning the attention masks is supervised via:
Lan
log
@ (N((,, (ssu) O a s-sarface (crama)) log
my
surface
/eimax
(13)
Optionally, we found that using a I.2 loss for the Lat can he beneficial for complex 3D scenes, please refer to appendix A.2 for further details. In the end, we penalise the remaining scope sy by introducing remaining scope loss:
Wi Lecope ΣΣ SKiJj
(14)
Taken together, these losses contribute straightforwardly to the overall loss:
f Leukour Ldeph + LKL Lwbere + Lau Lycope
(15)
In training, we use the Adam optimizer with fixed learning rate of 4e without any learning rate warm-up or decay strategy.
6

---

Under review as a conference paper at ICTR 2023
EXPERIMENTS
To evaluate OBPOSE's performance unsupervised object-centric inference and generation of 3D conduct experiments the CLEVR-3D dataset (Johnson et al., 2017) and the Multi- YCB object datasets (Calli al.,
ShapeNet dataset used by OhSuRF (Stelzner al., 2021), and 2015). One the YCB object datasets is moving-objes (video) dataset of RGB-D images captured from fixed front view. The other YCB dataset contains static images captured from three different points of view, where the three views are obtained by rotating the camera by 120" /240 around the z-axis of the world frame. All images are shuftled so that ground truth pairing information is not used for training. In each YCB object dataset, two to four randomly selected objects are spawned on the table. The metrics used for evaluation are descrihed below. Performance on all datasets compared against the recent bascline of ObSuRF (Stelzner et al., 2021), which, to the best of our knowledge, is the only unsupervised scene inference and generation model that operates on RGB-D images of 3D scenes. To validate our model design decisions, alternative attention mechanisms in the encoder, one from slot attention (Locatello al., 2020) and another from
compare performance using two
IC-SBP that does explicitly model object location and orientation (Engelcke al., 2021). ). We further compare ablations of the OBPOSE encoder: conditioned only the locations of the objects, and conditioned the full 6D poses.
4.1 METRICS
evaluations, we quantify segmentation quality using the Adjusted Rand Index (ARI) (Rand, 1971; Hubert Arabie, 1985) and Mean Segmentation Covering (MSC) as metrics. ARI measures clustering similarity between invariant fashion. This is appropriate for unsupervised segmentation
predicted segmentation masks the ground-truth segmen-
tation masks in a permutation cormutationiivaaiian approaches where there are no fixed associations between slots and objects. We evaluate segmentation accuracy on foreground objects using the foreground-only ARI and MSC (denoted ARI-FG and MSC-FG, respectively), and on the background using mean Intersection over Union (mloU). All metrics normalised between 0 and where score of indicates perfect segmentation.
UNSUPERVISED 3D OBIECT SEGMENTATION
Scond Image Trons. Conditiont (Duns) Pose Cond (0uns)
Skit Attontionn
3-3BP
O6GURF wb Ovetap
Figure 3: Visualisation of the segmentation results using the YCB moving-object dataset. Our model (OBPOSE) learns better segmentations the demonstrated video. Each slot associated fixed colour throughout the video. The colours are chosen to visually distinct from one another.
7

---

Under review as a conference paper at ICLR 2023
Table 1:
The segmentation results on the CLEVR-3D dataset and MultiShapeNet. The results are rounded to two decimal places.
CLEVR-3D
MultiShapeNet ARI-FGT
ARI-GT
ARI
ART
OBSURF
0.96
0.95
0.81
0.64
OBSURF W/O OVERLAP
0.86
0.06
0.94
0.16
SLOT ATT. OBPOSE (ours)
0.46
0.01
0.52
0.08
0.99
0.84
0.99
0.81
Table 2: Mean and standard deviation of the segmentation metrics on the YCB moving-object dataset
and the YCB static dataset from three random seeds. The results are rounded to two decimal places. OBPOSE outperforms the baseline and the ablations as it explicitly estimates the locations and the orientations of the objects, disentangling object location and appearance.
YCB Movang-O6joc
YCB Statie ARL-FG
mloU-BGT
ARI-FG
MSC-FG
mloU-BGf
MSC-FGI
OBSURF w/o OVERLAP WITH DEPTH
0.89 2 0.09
0.18
0.21 0
0.92
0.04
0.31 0.115
0.37 a 0.07
OESURF wo OVERLAP
0.97 0.02
0.28
0.98 0.00
0.22
0,36 = 0.05
0,98
0.03
0.09
0.00
0.00
0.84 0.00
IC-SBP
0,96 1 0.05
0.87
0.02
0.97 0.03
0.00
0.30 0.15
OBPOSE NO ROT. (ours)
1.00 0.00
0.96 0.00
0.97= 0.00
0.99 0.00
0.89+0000
0.87 0.03
OBPOSE (ours)
1.00 = 0.00
0.96 =: 0.00
0.97 = 0.00
0.98 a 0.00
0.88 = 0.01
0.84 = 0.03
"The SLOT ATT resulls are computed with one failed random seed being excluded For the YCB static dataset.
Quantitative results for the CLEVR dataset are shown in Table I with qualitative segmentation results in the appendix (Figure 8). We first observe that OвPOSE achieves better foreground segmentation performance (ARI-FG) compared to ObSuRF, which indicates that OвРоSE can perform better scene inference hy conditioning on the location and the orientation. The lower full ART score is caused by the fact that OB POSE learns to include shadows caused by the existence of the objects in each object slot (see Figure 8), whereas object shadows are labelled as background in the ground-truth masks. Qualitative results on the YCB moving-objee object dataset are depicted in Figure 3. Quantitative segmentation results are additionally summarised in Table 2. Here we only report ObSuRF results without using the overlap loss we first observe that the ObSuRF baseline fails to segment the scenes properly using the default setting for 3D data from the open-soureed code. This might be attributed the weight of overlap loss used ObSuRF. to discourage the objects from overlapping. In Stelzner et al. (2021) the same failure mode reported. In OBPOSE, we instead account for the overlap using a sepper-prammter--eee expensive hyper-parameter searching process. Interestingly, using the full 6D pose including the orientation and the location of the objects does not strongly affect the segmentation performance companed to using only the object positions as a way to condition the encoding. This suggests that for object segmentation tasks, the object location itself already provides a strong inductive bias for successtul decomposition. Similar results are observed elsewhere in the literature (Kipf et al., 2021), where simple ground-truth position information for objects is used in the first video frame, allowing the model to perform scene segmentation of 2D video data in weakly supervised fashion. In our approach, we explicitly leverage the 3D reconstruction of the objects whose shape is estimated by the proposed voxelised shape approximation approach. This allows the model to infer the locations and the orientations of objects in computationally efficient way without using any ground-truth labels. OBPOSE also achieves lower variation on metrics for both datasets, suggesting more stable training compared to the original IC-SBP.
function (softmax) Equation (8). This alleviates the computationally
4.3 SCENE GENERATION AND SCENE EDITING
I Leveraging the disentangled where and what object-centric representation, OBPOSI can perform more flexible scene generation and scene editing than previous works(Stelzzne) et al., 2021; Engeleke et al., 2021). We demonstrate this benefit with the CLEVR-3D dataset. In Figure 4(a), we show scenes that are generated from OвPOSE by first sampling from the learned object and the background latent space and then rendering from the composition of object and background NcRFs. The object locations and orientations can be arbitrarily set to user-defined values. Figure 4(b-d) demonstrates that OBPOSE can further be used for flexible scene editing (i.e. adding, removing, or manipulating objects in generated or inferred scene). The object-level scene manipulation Figure 4(d) an
8

---

Under review as a conference paper at ICTR 2023
important function OCGMs has raised people's attention in generative models (Niemeyer Geiger, 2021), OoPOSE thus for the first time implements this in an inference model. Additional view synthesis and segmentation results can be found in the appendix.
2
g
8
Figure 4: We demonstrate the rendered RGB and depth results of the scenes generated by sampling from the learned latent space (a). We also show the scene editing functions of object addition (b), object removal (c) and object-level scene manipulation (d).
5 RELATED WORK
OвРOSE builds upon prior OCGM work on unsupervised segmentation in both 2D and 3D. Most OCGMs for 2D scene segmentation are formulated as variational autoencoders (VABs) (Kingma & Welling, 2013; Rezende et al., 2014), where different likelihood models serve to explain observations. One set of VAB-OCGMs use bounding boxes, derived from spatial transformer networks (STNs) represent (glimpse) individual objects (Eslami al., 2016; Xu et al., 2019; Crawford & Pineau, 2019; Lin et al., 2020; Kosiorek al., 2018; Jiang al., 2019). Another represents objects via unsupervised instance segmentation, using pixel-wise mixture models (Burgess al., 2019; Engelcke al., 2019; 2020; Greff et al., 2016; 2017: Van Steenkiste et 2018; Greff et al., 2019; Veerapaneni al., 2020; Locatello et al., 2020). This latter set relaxes the spatial- consistency requirements imposed by bounding boxes (Jaderberg et al., 2015), permitting more flexible modelling of objects with complex shapes and textures. However, relaxing spatial consistency has the side-effect that performance can sometimes be biased by features such as the colour of the object (Weis et al., 2020), which has motivated the search for additional inductive biases. A promising candidate is temporal information. To this end, some works (Vcerapanemi et al., 2020; Ehrhardt et al., 2020; Kipf et al., 2021) operate on video data and model the correlations between objects explicitly using graph neural networks (GNNs) (Scarselli et al., 2009; Sanchez-Lengehee et al., 2021) or Transformers (Vaswani al., 2017).
The idea of a reference pose for object, has precedent in the context of 6D pose estimation, which aims to find the translation and rotation of an object with respect some frame of reference. In the supervised setting, labels are defined with respect al., 2020a; Li al., 2020; Wang et al., 2020; Chen et al., 2020b: Tian et al., 2020). Recently, it has been shown that pose between views of an object, or objects from common category. can be inferred without labels, Such relative poses have been found for point clouds et al., 2021) and RGB-D images (Goodwin al., 2022). To the best our knowledge, we are the first to propose a minimum volume approach for discovering pose without supervision and include pose information, to reduce its variance the latent code, as an inductive bias.
given reference frame (Wang al., 2019; Chen
CONCLUSION
We present OnPOSE, an object-centric inference and generation model that learns 3D-structured latent representations. The model extends IC-SBP for video input, and is noteworthy for introducing poses as an inductive bias for scene inference. The model's I'ss ability to infer object pose is facilitated by several recent innovations, including the use of NeRFs and the fast voxelised shape approximation proposed in this paper. Our experimental results are validated on the CLEVR dataset, the MultiShapeNet dataset, and two synthetic YCB objects datasets. Given its empirical success, outperforming the prior state-of-the-an art for static 3D scenes (Stelzner et al., 2021) and establishing a baseline for video, we plan apply OBPOSE as vision backbone for robot applications.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Christopher Burgess. Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. MONet: Unsupervised scene decomposition and representation. arXiv preprint orXiv:/901. 1390, 2019.
Berk Calli, Arjun Singh, Aaron Walsman, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M Dollar. The YCB object and model set: Towards common benchmarks for manipulation research. In 2015 International Conference on Advanced Robotics (ICAR), pp. 510-517. IEEE, 2015.
Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An information-rich 3d model repository. arXiv preprint arXiv: 2015.
Dengsheng Chen, Jun Li, Zheng Wang, and Kai Xu. Learning canonical shape space for category- level 6d object pose and size estimation. Computer Vision and Pattern Recognition (CVPR), pp.
11973-11982 11922, 2020a.
Xu Chen, Zijian Dong. Jie Song. Andreas Geiger, and Otmar Hilliges. Category level object pose estimation via neural analysis-by-syee by-syntheisss.. European Conference on Computer Vision, pp. 139-156, 2020b.
Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convolutional neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp.
3412-3420, 2019.
Sébastien Ehrhardt, Oliver Groth, Aron Monszpart, Martin Engelcke. Ingmar Posner, Niloy Mitra, and Andrea Vedaldi. RELATE: Physically plausible multi-object scene synthesis using structured latent spaces. Advances in Neural Information Processing Systems (NeurIPS), 33: 13:11202-11213 2020.
Cathrin Elich, Martin R. Oswald, Mare Pollefeys, and Joerg Stueckler. Weakly supervised learning multi-object..... 3D scene decompositions using deep shape priors. Computer Vision and Image Understanding, 220: 103440, 2022.
Martin Engelcke, Adam R Kosiorek, Oiwi Parker Jones, and Ingmar Posner. GENESIS: Gener- ative scene inference and sampling with object-centric latent representations. arXiv preprint arXiv:1907.1 2019.
Martin Engelcke, Oiwi Paker Jones, and Ingmar Posner. Reconstruction bottlenecks in object-centric generative models. arXiv preprint arXiv:2007.06245. 2020.
Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner. GENESIS-v2: Inferring unordered object representations without iterative refinement, Advances Neural Information Processing Systems (NeurIPS), 34, 2021.
SM Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Geoffrey E Hinton, et al. Attend, infer, repeat: Fast scene understanding with generative models. Advances in Neural Information Processing Syxtems (NeurlPS), 29, 2016.
Walter Goodwin. Sagar Vaze, Ioannis Havoutis, and Ingmar Posner. Zero-shot category-level object pose estimation. arXiv preprint arXiv:2204.00 036635...333.... 2022.
Krzysztof M Gorski, Eric Hivon, Anthony J Banday, Benjamin D Wandelt, Frode K Hansen, Mstvos Reinecke, and Matthia Bartelmann. HÉALPix: A framework for high-resolution discretization and fast analysis of data distributed on the sphere. The Astrophysical Journal, 622(2):759, 2005.
Klaus Greff, Antti Rasmus, Mathias Berglund, Tele Hao, Harri Valpola, and Jürgen Schmidhuber. Tagger: Deep unsupervised perceptual grouping. Advances in Neural Information Processing Syxtems (NeurIPS), 29, 2016.
Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. Advances in Neural Information Processing Systems (NewrIPS), 30, 2017.
Klaus Greff, Rephaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In International Conference Machine Learning, pp. 2424-2433. PMLR, 2019.
10

---

Under review as a conference paper at ICTR 2023
Lawrence Hubert and Phipps Arabie. Comparing partitions. Journal of Classification, 2(1):193-118 1985.
Max Jaderberg. Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. Advances Neural Information Processing Systems (NeurIPS), 28, 2015.
Jindong Jiang, Sepehr Janghorbani, Gerard De Melo, and Sungjin Ahn. SCALOR: Generative world models with scalable object representations. arXiv preprint arXiv: 1910.02384 2019.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2901-2910, 2017.
Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3D mesh renderer. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3907-3916, 2018.
Diederik Kingma and Max Welling.
Auto-encoding variational bayes.
arXiv preprint
arXiv:1312.6114. 2013.
Thomas Kipf. Gamaleldin F Elsayed, Aravindh Mahendran, Austin Stone. Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional object-centric learning from video. arXiv preprint arXi::2111.12594 2021.
Adam Kosiorek. Hyunjik Kim, Yee Whye Teh, and Ingmar Posner. Sequential attend, infer, repeat: Generative modelling of moving objects. Advances in Neural Information Processing Systems (NeurIPS), 31, 2018.
Xiaolong Li, He Wang, Li Yi, Leonidas J. Guibas, A. Lynn Abbolt, and Shuran Song. Category level articulated object pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3706-3715, 2020.
Xiaolong Li, Yijia Weng, Li Yi, Lconidas Guibas, A Lynn Abbott, Shuran Song, and He Wang. Leveraging SE(3) equivariance for sulf-supe category-level object pose estimation. Advances in Neural Information Processing Systems (NeurIPS), 2021.
Zhixuan Lin, Yi-Fu Wu. Skand Vishwanath Peri, Weihao Sun. Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. SPACE: Unsupervised object-oriented scene representation via spatial attention and decomposition. arXiv preprint XiX::200000024000 2020.
Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel 2020.
fields. Advances Neural Information Processing Systems (NeurlPS),
Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. Soft rasterizer: A differentiable renderer for image- based 3D reasoning. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 7708-7717, 2019.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, A Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances in Neural Information Processing Systems (NewrIPS), 13:115525-11 11388, 2020.
Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision, pp. 405-421. Springer, 2020.
Kieran Murphy, Carlos Esteves, Varun Jampani. Srikumar Ramalingam, and Ameesh Makadia. Implicit-pdf: Non- pana-paraetric representation of probability distributions on the rotation manifold. arXiv preprint 15X.5:2100.05965, 2021.
Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes compositional generative neural feature fields. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. (1453-11464, 2021.
William M Rand. Objective criteria for the evaluation of clustering methods. Journal of the American Statistical Association, 666336))84--550 1971.
11

---

Under review as a conference paper at ICLR 2023
Danilo Jimenez. Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International Conference on Machine Learning. pp. 1278-1286. PMLR, 2014.
Olaf Ronneberger. Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015.
Benjamin Sanchez-Lengeling, Emily Reif, Adam Pearce, and Alexander B. Wiltschko, A gentle introduction to graph neural networks. Distill, 2021. doi: 10.23915/distill000033 http::/ intro.
Franco Scarselli, Marco Gori, Chung Tsoi, Markus Hagenbuchner. and Gabriele Monfardini, The graph neural network model. IEEE Trunsactions on Neural Networks, 20(1):61-80, 2009.
Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. GRAF: Generative radiance fields for 3D-aware image synthesis. Advances in Neural Information Processing Systems (NeurIPS), 33: 20154-20166, 2020.
Karl Stelzner, Kristian Kersting, and Adam R Kosiorek. Decomposing 3D scenes into objects via unsupervised volume segmentation. arXiv preprint arXiv:2104.01 2021.
Hugues Thomas, Charles R Qi, Ammanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas Guibas. Kpconv: Flexible and deformable convolution for point clouds. In Proceedings of the IEFF/CVP international conference on computer vision, pp. 6411-6420, 2019.
Meng Tian, Marcelo H. Ang, and Gim Hee Lee. Shape prior deformation for categorical 6d object pose and size estimation. European Conference on Computer Vision, pp. 530-546, 2020.
Sjoerd Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational neural expectation maximization; Unsupervised discovery objects and their interactions. arXiv preprint 1802 10353, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems (NewrIPS), 30, 2017.
Rishi Veerapaneni, John D Co-Reyes, Michael Chang, Michael Janner, Chelsea Finn, Jiajun Wu, Joshua Tenenbaum, and Sergey Levine. Entity abstraction in visual model-based reinforcement learning. In Conference on Robot Learning, pp. 1439-1456. PMLR, 2020.
Chen Wang, Roberto Martin-Martin, Danfei Xu, Jun Lv, Cewu Lu, Li Fei-Fei, Silvio Savarese, and Yuke Zhu. 6-pack: Category-level 6d pose tracker with ansed keypoints. International Conference on Robotics and Automation (ICRA), pp. 1009610066,, 2020.
He Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, and Leonidas J. Guibas. Normalized object coordinate space for category-level 6d object pose and size estimation. arXiv preprint arXiv: 1901. 02270, 2019.
Marissa Weis, Kashyap Chitta, Yash Sharma, Wieland Brendel, Mathhis Bethge, Andreas Geiger, and Alexander S Eoker. Unmasking the inductive biases unsupervised object representations for video sequences. arXiv preprint arXiv:2006.0703: 2, 2020.
Yizhe Wu, Oiwi Parker Jones, Martin Engelcke, and Ingmar Posner. APEX: unsupervised, object- centric scene segmentation and tracking for robot manipulation. International Conference on Intelligent Robots and Systems (IROS), 2021.
Yuxin Wu and Kaiming He. Group normalization. In Proceedingy of the European conference on computer vision (ECCV), pp. 3-19, 2018
Haozhe Xie, Hongxun Yao, Xiaoshuai Sun, Shangehen Zhou, and Shengping Zhang. Pix2vox: Context-aware 3D reconstruction from single and multi-view images. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 2690-2698, 2019.
Taufik Xu, Chongxuan Li, Jun Zhu, and Bo Zhang. Multi-objects generation with amortized structural regularization, Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
12

---

Under review as a conference paper at ICLR 2023
Anna Yershova, Swati Jain, Steven M Lavalle, and Julie C Mitchell. Generating uniform incremental grids on (3(3) using the Hopf fibration. The International Journal of Robotics Research, 29(7): 801-812, 2010.
Hong- Xing Yu, Leonidas J Guibas, and Jiajun Wu. Unsupervised discovery of object radiance fields. arXiv preprint arXiv:200..07005, 2021.
13


Reviews:
[{"review_id": "XAoJ50seMK", "reviewer": "Reviewer_N5Sr", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a model OBPose for unsupervised 3D object segmentation from RGB-D images/videos. OBPose first represents each object as disentangled location and appearance information, then re-renders the scene with a NERF decoder. Experimental results prove the effectiveness of the proposed object representation.", "summary_of_the_review": "I have reviewed this paper in NeurIPS, and the quality of this submission has been improved compared with its previous version. It is nice to see the experiment results and analysis on CLEVR-3D dataset. Therefore, I think it is marginally above the borderline.", "strength_and_weaknesses": "Strengths:\n\n1. Compared to the slot-attention-based methods that represent each object as a single latent code, this work explicitly disentangles object location and appearance information, which gives more accurate scene-decomposing results.\n2. Ablation study in Table2 justifies its superiority against the slot-attention-based method and IC-SBP-only method.\n\nWeakness:\n1. The novelty is limited. (1) The key components such as IC-SBP and canonical pose representation are mostly borrowed from previous methods [7][19]. (2) The idea of representing objects as disentangled location and appearance information is also common, e.g., BlockGAN[R1].   \n2. The loss function is complicated and the contribution of each component is unclear from the results of the experiments.   \n3. It would be more convincing if the authors could provide some quantitative comparisons on the unsupervised 6D pose estimation task.   \n\n[R1] Nguyen-Phuoc, Thu H., et al. \"Blockgan: Learning 3d object-aware scene representations from unlabelled images.\" Advances in Neural Information Processing Systems 33 (2020): 6767-6778.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WbKECU-cO4W", "reviewer": "Reviewer_opyj", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the task of unsupervised object segmentation from RGB-D\nvideo data.  In particular, the authors introduce an object-centric\nrepresentation that is used to decompose each RGB-D image into a set of NeRFs\nper scene object. Note the camera poses from which the RGB-D video data were\ncaptured are considered to be known. The RGB-D input, is first converted into a\npointcloud which is then passed into a KPConv backbone, that extracts per-point\nfeatures. These features are subsequently used to compute the background\nembedding and the soft attention masks per object, which are produced by\nclustering the per-point features. Given these masks, they then predict the\npose of each object. The location of each object is simply the center of mass\nof the points within each cluster, and the rotation is computed following the\nminimum volume principle. Finally, conditioned on the object poses, they\nextract features that are passed to a per-object NeRF. The proposed model is\nevaluated on the task of unsupervised scene segmentation on the CLEVR dataset\nand on the YCB dataset using both RGB-D videos and multi-view static scenes.\nThe authors consider two baselines: The ObSurf and slot attention and\ndemonstrate that their method outperforms prior works in terms of various\nmetrics such as the mean Intersection over Union (mIOU), the Adjusted Rand\nIndex (ARI) and the Mean Segmentation Covering (MSC).\n", "summary_of_the_review": "Overall, I like the idea of this paper but I am a bit skeptical regarding the experimental evaluation. I think it is essential to provide the additional results I mentioned above on the CLEVR dataset in order to ensure a stronger submission. Below are some more comments/questions and suggestions:\n\n1. Symbol $\\zeta_f$ does not appear in Figure 2. I recommend to add in Figure 2\nto avoid confusion. From the description in the last sentences of Section 2.1\nit is not 100% clear how $\\zeta_f$ and $\\zeta_c$ are computed from $\\zeta$.\n\n2. Can the authors provide some more details regarding the computation of\n$\\mathbf{b}z^{\\textbf{bg}}$ from $\\zeta$? I think that providing additinal details on\nthis would significantly improve clarity.\n\n3. If I understand correctly the Instance Colouring\nStick-Breaking Process, described in Section 2.1.1 is simply a clustering\nmechanism. I believe that it would significantly improve the paper's clarity if the authors\nadded one/two sentences in the beginning of this section pointing out this\nfact. Starting with all the specific details make understanding quite hard. In\nparticular, regarding section 2.1.1, I believe that the authors should revisit\nthe text and try to provide some intuition regarding the things they describe.\nFor example, why is the stochastic ordering of the mask important? I think that\ngiving intuitive examples always facilitates understanding.\n\n4. From Section 2.1 it is not clear that the feature embedding $\\zeta_f$ refers\nto point features at a particular time step t (see Section 2.1.2). This is\nquite confusion and should be clarified in Section 2.1\n\n5. The reconstruction results in Figure 7 in the supplementary seem to have\nsome artifacts. I am wondering to the authors use also importance sampling in\ntheir NeRF, or do they only rely on a coarse NeRF model?\n\n6. How sensitive is the IC-SBP module to initialization?\n", "strength_and_weaknesses": "## Strengths:\n------------\n\n1. The idea of disentangling the concept of \"where to look\" from \"what to look\"\nis very interesting and is applicable to various tasks beyond oibject segmentation.\n\n2. I also liked the author's solution for finding th object's pose using the\nminimum volume principle.\n\n3. I liked the additional results on scene editing in Section 4.3. However,\nthis section seems a bit unconnected with the rest of the text. So it might be\ngood to revise the text to better prepare the reader for this experiment.\n\n4. I appreciate that for the CLEVR experiment, the authors also visualize the\ncorresponding depth maps, since their model can also predict them.\n\n5. Overall the proposed idea is simple, novel and achieves very good results.\n\n## Weaknesses:\n-------------\n1. The experimental evaluation is a bit weak. First of all, I think that the\ndatasets that the authors use for evaluating their model are rather simple,\nnamely the YCB dataset consists only of two scenes and the CLEVR has very\nsimply object arrangements. However, since also ObSuRF considers such a simple\nexperimental setup I am eager to overlook this. Nevertheless the experiments\ncould be improved. For example,  is there a reason why there is no comparison\nwith slot attention on the CLEVR dataset (see Table 1.)? Furthermore, can the\nauthors provide some additional qualitative comparison with ObSuRF and Slot\nAttention also on the CLEVR dataset.\n\n2. Looking at the original results from the ObSuRF paper, their segmentations\nlook quite plausible. Hence, I find it a bit weird that for the YCB dataset it\nfails completely. Therefore, I think it is important to show a similar\ncomparison as in Figure 3 also for the CLEVR dataset.\n\n3. A less important weakness of this paper in comparison to slot attention is\nthat it assumes that the number of objects in the scene is known. While I\nbelieve that this constraint does not impose such a strong limitation it is\nstill a limitation that is useful to be addressed.\n\n4. Since the authors only consider very simple datasets in their evaluation, I\nam wondering how generalizable this model is to more complex scenes.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "S5WNXaccui", "reviewer": "Reviewer_uLft", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes an object-centric 3D inference method that is trained in a self-supervised way using a variational auto-encoder framework and neural radience fields. While a large number of methods in the literature address object-centric self-supervised segmentation, very few also learn to infer 3D pose and shape while doing so. The authors propose using the existing video-specific instance colouring stick-breaking process to encode the input imagines into segmented attention masks, and propose a novel mechanism to refine and estimate each detected object's pose. Given disentangled latents pose (where) and content (what), novel images can be rendered using a NeRF MLP for any viewpoint. Once trained ObPose obtains near-perfect segmentations on a number of datasets and is shown to outpeform ObSurf as well as a number of other baselines.  ", "summary_of_the_review": "The direction of this work is important for robotics and this method can be a useful way to learn detailed representations from images without requiring large datasets of segmented objects. The combination of techniques is novel and the authors propose a new heuristic to estimate and refine the pose of the segmented objects. Further clarification on the method's ability to detect pose is important, as well as additional experiments wrt. ablations of some losses and a fairer comparison with ObSurf. I believe these points are necessary to establish the significance of this work. As it is now, I am leaning towards rejecting it.\n\n---\n**Post rebuttal update:**\n\nI thank the authors for addressing my questions and concerns, and for updating the manuscript with improvements in a wide range of aspects. Most importantly, ensuring a fairer comparison with ObSurf as well as the addition of another dataset. I believe the paper is in a better state now, but I still think it's insufficient to raise my score due to the fundamental limitations below.\n\nAs reviewer aWJo also shares, two of my main concerns not been fully resolved after the rebuttal.\n\nFirst, it is still unclear to me what the benefits are of the pose refinment novelty via the minimum volume principle. As acknowledged by the authors, conditioning the encoding on rotation doesn't result in clear benefits (while conditioning on location does); and neither have the authors provided evidence that the location refinement (delta T) is important to the overall model performance. It is also still unclear if the poses reflect consistently the real poses of the objects which would justify the hypothesis the authors refer to regarding lower variance of appearance representation. In other words, if poses estimated are arbitrary for different instances of similar classes (e.g. chairs of different shape but same appearance) why should we expect the appearance representations of them to be invariant?\n\nSecond, while the comparison with ObSurf is now fairer, it is still difficult to assess this work in the wider 3D scene understanding literature. \nThis method requires RGB-D whereas most unsupervised segmentation methods do not. Methods such as ObSurf and Object Scene Representation Transformers (Sajjadi et al. 2022) work without depth and also are able to produce great reconstructions (especially the latter). ObPose's renders are fairly poor (Fig. 7) which casts doubt about how well it can work (i.e. that relies on reconstructions) on the increasingly richer and more realistic datasets such as MultiShapenet-Hard, e.g. scenes with heavy clutter and complex textures. While this may be improved with better latent representations of z_what and z_where and a stronger NeRF decode, all in all it would involve substantial changes to the method and paper.", "strength_and_weaknesses": "**Strengths**\n- The use of strong inductive biases allows this self-supervised model to obtain representations suitable for domains in which detailed scene understanding is required. These biases are: object-centric segmentations, inference of disentangled 3D pose and shape, and the ability to represent objects in explicit 3D voxels.\n- The authors propose a novel method for shape representation via voxelization using the \"minimum volume principle\", an efficient approach that allows end-to-end training.\n- Results highlight the model's ability to obtain precise 3D (and 2D) segmentations from just one or a few images, and to consistently outperform closely related work in this task on CLEVR-3D and YCB datasets.\n\n**Weaknesses**\n1. While the minimal bounding-box volume selection is an interesting heuristic for pose estimation, it's unclear from the paper if the method proposed learns consistent canonical poses across instances of the same category. Related work [1, 2] in self-supervised pose estimation methods ultimately evaluate their method against some form of GT category-level poses. This work would be strengthened if similar evidence is provided (in the form of additional evaluations against GT poses) or if at least extensive examples of estimated poses are shown. Note that if the above estimated poses are not consistent, the fast shape evaluation may also fail to produce consistent shapes across instances of the same category.\n2. Experimental results suggest that using the inferred rotations do not seem to help, which weakens the significance of one of the paper's novelties (i.e. the benefits of the where->what pipeline). That said, shape estimation is used to refine the object location prediction via $L_{where}$. Thus, for the whole approach to be an interesting contribution, the authors should show that location refinement (delta T) is important to the overall model performance. \n3. Finally, I am concerned regarding the fairness of the comparison with ObSurf. In the most challenging YCB datasets, the authors only report ObSurf w/o overlap loss. In ObSurf's paper it is shown to be an important loss. In my eyes, for this comparison to be fair, the authors should perform a reasonable hyper-parameter search and not just use ObSurf's \"default settings for 3D\". Usually optimal settings may vary for each dataset and indeed ObPose's settings also change between CLEVR-3D and YCB as per Table 3 in the appendix.\n\n**Questions / suggestions**\n\n- What happens if $N_k$ is small in the first time step?\n- Requiring depth in general is a stronger requirement than many methods in the literature, including ObSurf which only uses depths for training signal. Due to ObPose's point-cloud approach, it cannot be trained without depths, but for the sake of fairness the authors can compare with ObSurf that additionally conditions the encoder with depths. \n- What are failure modes of the method? It would be enlightening to see more challenging examples i.e. clutter and occlusion from YCB.\n- How do $L_{att}$ and $L_{scope}$ (which don’t seem essential) affect performance?\n- Also, since $L_{att}$ is different for some datasets, this should be stated in the main text.\n- Is the IC-SBP baseline exactly ObPose but without the use of pose-based transformation of features?\n- The use of tanh activation to compute the global density means densities can be negative. Are these the densities the ones used for volumetric rendering? If so breaks the non-negativity assumption of volumetric rendering. \n\n[1] Li et al. Leveraging SE(3) Equivariance for Self-supervised Category-Level Object Pose Estimation from Point Clouds. NeurIPS 2021\n\n[2] Zero-Shot Category-Level Object Pose Estimation. ECCV 2022", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DixPvqx49NA", "reviewer": "Reviewer_aWJo", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method for the unsupervised discovery of object-segmented 3D representations with explicit pose parameters, given an RGB-D input image. Building on the ObSuRF model, the proposed ObPose system consists of an encoder inferring pose and shape parameters for a set of objects, and a NeRF decoder which processes these into a set of composable 3D volumes. It is demonstrated that ObPose yields improved segmentation results on new synthetic datasets made up of YCB objects. Additionally, scene editing capabilities are demonstrated.", "summary_of_the_review": "Overall, while the paper proposes an interesting variant of ObSuRF, there are currently too many issues with the experimental evaluation to recommend acceptance. If the authors can show that the pose parameters work as intended, and that their improvements extend to existing hard benchmarks, I am willing to increase my evaluation.\n\n----\n\n**Post rebuttal update:**\nI thank the authors for their response and the updates to the manuscript. I think especially the MSN-easy experiment is an important (and impressive) addition. However, some important issues remain.\n\n**Unfair experimental setup**: The fact that ObPose leverages depths at test time, while ObSuRF doesn't, renders the main experimental results of the paper unfair. The fact that this was swept under the rug in the manuscript is a significant red flag. The added \"ObSuRF with depth\" baseline cannot fully address this issue:\n  * No details were provided on how it was constructed.\n  * ObPose's encoder is directly tailored to RGB-D input in many ways, e.g. by using KPConv layers, and by explicitly making use of input point clouds to identify object locations. ObSuRF's encoder is designed for RGB input, so the comparison is still not particularly fair.\n  * The more meaningful \"ObSuRF with depth\" baseline is only provided in Table 2, for the YCB experiment, not in Table 1, for CLEVR3D and MSN.\n  * The manuscript still incorrectly states that ObSuRF operates on RGB-D images, and does not make this issue transparent.\n\nI recognize that 3D scene understanding is a developing paradigm in which many problem formulations are possible, and for many of them, there are no appropriate baselines. But these issues need to be treated transparently and put into context. If the authors are looking to argue that their explicit treatment of object location is the key driver of performance for ObPose, an appropriate experiment would be to compare it with an ablation which still leverages RGB-D input, but doesn't contain the pose module. In a setting with RGB-D input, ObSuRF can only provide a lower bound on the performance we may expect.\n\n**Pose Evaluation**: The authors have clarified that the goal of their model is not pose estimation. I agree that there is no need to compare to pose estimation state of the art, or anything like that. However, if the story of the paper is that explicit pose parameters lead to improved segmentation performance, it is still crucial to convince the reader that these pose parameters actually work as advertised. An experiment to that effect is still missing from the main paper.\n\n**Clarity**: No major changes have been made to the structure of the text, as far as I can tell. As a result, my clarity concerns remain.\n\nI believe these issues are too significant to recommend acceptance at this time. I am therefore leaving my score unchanged.", "strength_and_weaknesses": "Strengths:\n - The paper investigates the interesting question of discovering disentangled 3D representations from RGB-D input.\n - While the possibility of introducing explicit pose parameters was discussed in the ObSuRF paper, to my knowledge, this work is the first to do so. Explicit pose parameters have the potential to simplify object manipulation for downstream tasks.\n - The method shows improved segmentation results on a new synthetic benchmark.\n\nWeaknesses:\n - The provided quantitative results only cover image segmentation results on synthetic benchmarks. Performance on CLEVR3D is similar to ObSuRF. While the model outperforms its baselines on the new YCB images, it remains unclear why new data was generated instead of using the Multishapenet dataset from ObSuRF, given that the two appear to be of similar complexity. In any case, it cannot be claimed that ObPose improves state of the art in 3D segmentation, as the considered datasets are significantly simpler than MSN-hard as used by OSRT [2].\n - The model's most important novel capability, its ability to infer explicit pose parameters, is not directly evaluated. This could be done by comparing to ground truth poses, or computing 3D IoU with respect to ground truth bounding boxes. As a result, it remains unclear if the model's pose parameters function as claimed. Indicating the inferred poses in Figures 3/4 would also help.\n - The softmax formulation in Eq. 8 suppresses the rendering of overlapping objects. However, it does not seem to prevent multiple slots from *representing* the same object with high $\\sigma_i$. If these are then rendered separately, e.g. in a scene editing context, the previously suppressed overlaps would emerge, potentially hurting compositionality.\n - The way the paper uses voxels is unclear to me: The term seems to indicate that space is discretized into cubes, but no details are provided on this, and the equations appear to be consistent with standard, continuous NeRF rendering. \n - In general, the paper appears somewhat unfocused, and its presentation could be significantly improved (see below).\n - Contrary to what is stated in this paper, ObSuRF does not use depth information at test time, only as a training signal. ObSuRF's encoder only receives RGB input. Was this changed to allow for a fair comparison?\n - Explicit pose parameters are not strictly necessary for scene editing, as suggested in section 4.3. Adding, deleting and moving objects is also possible with ObSuRF, with the only qualitative difference being that pose parameters allow stating positions in absolute terms, instead of just relative movement. It might be the case that doing this with ObSuRF leads to more artifacts due to imperfect segmentations, however.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject