Spatially Invariant Unsupervised 3D Object-Centric Learning and Scene Decomposition
Authors: Tianyu Wang
miaomiao Liu
Kee Siong Ng
Venue: ICLR 2022 Submitted (2022)
Keywords: generatvie model
variational autoencoder
mixture model
unsupervised object centric learning

Abstract: We tackle the problem of deep object-centric learning from a point cloud which is crucial for high-level relational reasoning and scalable machine intelligence. 
In particular, we introduce a framework, SPAIR3D, to factorize a 3D point cloud into a spatial mixture model where each component corresponds to one object. 
To model the spatial mixture model on point clouds, we derive the Chamfer Mixture Loss, which fits naturally into our variational training pipeline. Moreover, we adopt an object-specification scheme that describes each object’s location relative to its local voxel grid cell. 
Such a scheme allows SPAIR3D to model scenes with an arbitrary number of objects. 
We evaluate our method on the task of unsupervised scene decomposition.
Experimental results demonstrate that SPAIR3D has strong scalability and is capable of detecting and segmenting an unknown number of objects from a point cloud in an unsupervised manner.

Under review as a conference paper at ICLR 2022
SPATIALLY INVARIANT UNSUPERVISED 3D OBJECT- CENTRIC LEARNING AND SCENE DECOMPOSITION
Anonymous authors Paper under double-blind review
ABSTRACT
We tackle the problem of deep object-centric learning from point cloud which is crucial for high-level relational reasoning and scalable machine intelligence. particular, introduce framework, SPAIR3D, factorize a 3D point cloud into a spatial mixture model where each component corresponds to one object. To model the spatial mixture model point clouds, derive the Chamfer Mixture Loss, which fits naturally into our variational training pipeline. Moreover, we adopt an sejfect-peeiiiication scheme that describes each object's location relative its local voxel grid cell. Such a scheme allows SPAIR3D to model scenes with an arbitrary number of objects. We evaluate our method on the task of unsupervised scene decomposition. Experimental results demonstrate that SPAIR3D has strong scalability and capable of detecting and segmenting an unknown number of objects from point cloud in unsupervised manner.
INTRODUCTION
Motivated in part by cognitive psychology studies (Kahneman et al. 1992) that suggest human brains organize observations an object level, recent advances in physical prediction (Chang al. 2017) and reinforcement learning (Diuk et al. 2008 Kansky et al.) (017) have demonstrated superior robustness with environments that are modeled in an object-orienter manner. To exploit these techniques for 3D scenes, which can exhibit complex and combinatorially large observation spaces even when there are only a few basic elements, robust algorithms are needed to identity objects from 3D observations. We tackle in this paper the problem of deep object-centric learning from a point cloud, which is crucial for high-level relational reasoning and scalable machine intelligence.
There is good body of existing literature unsupervised object-centric generative models for images and videos. In particular, generative models based on Variational Autoencoders (VAE) (Kingma & Welling, 2014) have been employed model the pixel intensities of an image with spatial Gaussian mixture models (Greff et al. 2019: [Burgess et al. 2019 Lin et al. 2020 Crawford & Pineau] 2019 Eslami et al.) 2016). SPAIR (Crawford & Pineau| 2019) makes of an object-specification scheme which allows the method to scale well to scenes with a large number of objects.
The aforementioned works define objectness
region with strong appearance correlations, and all of them employ VAE (Kingma & Welling (2014: Higgins et al. (017)) in their structure. To be more precise, the encoder-decoder structure VAE effectively creates an information bottleneck (Tishyy et al. 1999 Burgess et al., 2018) 8) limiting the amount of information passing through. To reconstruct the observation under limited information budget, highly correlated information must be exploited. Thus, VAE plays critical role in exploiting such correlations. The above mentioned works mainly exploit appearance correlations on objects colored uniformly. In this work, we aim to prove that this paradigm is also applicable to structural correlations conveyed by point clouds without appearance information.
The irregularity of point cloud (detailed in section 3) renders direct transfer of the above method infeasible. Inspired by SPAIR and the fact that 3D point cloud object generation requires centred- points (Chang et al. 2015). Attend, Infer, Repeat in 3D (SPAIR3D), a model that generates spatial mixture distributions on point clouds to discover 3D objects in static scenes. Here we summarize the key contributions of this paper:
propose in this paper a VAE-basee model called Spatially Invariant

---

Under review as a conference paper at ICLR 2022
We propose, to the best of our knowledge, the first unsupervised point cloud object-centric learning pipeline, named SPAIR3D.
We also propose new Chamfer Mixture Loss function tailored for learning mixture models over point cloud data with a novel graph neural network that can be used to model and generate variable number of 3D points.
We provide qualitative and quantitative results to show that SPAIR3D learns meaningful object-centric representation and decompose point clouds scene with an arbitrary number of objects in an object-oriented manner.
2 RELATED WORK
Unsupervised Object-centric Learning. Unsupervised object-centric learning has attracted increas- ing attention recently. A major focus of these methods is on joint object appresentationnleannii
and scene decomposition from images or videos via generative models (Burgess et al.
2019 Greff
ctt al. 2019 Engeleke al., 2020 Lin et al. 20200] Crawford & Pineau 2019: Li et al. 2020 Chen
cct al. 2021. In particular, spatial Gaussian mixture models are commonly adopted to model pixel
colors. The object-centric representation learning problem is then framed as generative latent- variable modelling task. For example, IODINE (Greff et al., (019)) employs amortized inference that iteratively updates the latent representation of each object. GENESIS (Engelcke et al.] (2020) and MONET (Burgess et al.) 2019) sequentially decode each object. Slot attention (Locatello et al. (0020)
and Neural Expectation Maximization (NEM) (Greff al. 2017 also adopt the same formulation.
Instead of treating each component of the mixture model as a full-scale observation, Attend, Infer, Repeat (AIR) (Eslami et al. 2016) confines the extent of each object to local region. To improve the scalability of AIR, SPAIR (Crawford & Pineau (201)) employs a grid spatial attention mechanism to propose objects locally, which has proven effective in object-tracking tasks (Crawford & Pineau 2020j. To achieve a complete scene segmentation, SPACE (Lin et al.) (2020) includes MONET in its framework for background modeling. Spatial attention models are also employed to reconstruct 3D scenes in the form of meshes or voxel in an object-centric fashion from sequence of RGB frames (Henderson & Lampert 2020). These methods similarly rely heavily on appearance and motion cues.
Graph Neural Network for Point Cloud Generation. Generative models such as VAEs (Gadelha ct al. 2018) and generative adversarial networks (Achlioptas et al. 2018) have been successfully used for point-cloud generation but with a pre-defined number of points per object. Luo & Hu (2021) models the point cloud generation process as a latent variable conditioned Markov chain. Yang et al. (2019) proposed VAE and normalizing flow-based approach that models object shapes as continuous distributions. While the proposed approach allows the generation of a variable number of points, it could not be naturally integrated into our framework because of the need for an ODE solver.
SPAIR3D
While the literature on deep unsupervised object-centric learning is rich, none of the methods listed above can be applied directly to 3D point cloud data. The reconstructions of 2D images, 31) voxel space, and neural radiance field in these methods are all coordinate-dependent (Watters al. 2019 Stelzner et al., 2022I. To be more precise, given coordinate, an occupancy value (mask or density) and feature vector (RGB color) are generated for each mixture component for that coordinate form a Gaussian mixture model. For image data, the coordinate dependency can be implicitly embedded in the network structure since the input and output are of fixed sizes (Islam * et al., (2200). The coordinate thus provides the correspondence between input and reconstruction, inducing likelihood function.
However, point cloud data are irregular and take the form of an unordered set. Each point cloud may have varying number of points. Most importantly, the point coordinates carry all the structural information and form the reconstruction target, making coordinate-dependent reconstruction not an option. Due to the irregularity of point-cloud data, there is also usually natural correspondence between the input and the reconstruction. While Chamfer Distance commonly serves as a loss function for point cloud reconstruction, it does not support mixture model formulation directly. Such data irregularity makes defining a mixture model over point cloud a non-trivial task. We, therefore, introduce SPAIR3D, VAE-based generative model, to achieve 3D object-centric learning well as
2

---

Under review as a conference paper at ICLR 2022
3D scene decomposition via object-centric point-cloud generation. It takes point cloud input and generates a structured latent representation for foreground objects and scene layout.
In the following, first describe latent representation learning. We then leverage variational inference to jointly learn the generative model (33.2) and inference model ($3.4). We further discuss the particular challenges arising for generative models in handling a varying number of points with novel Chamfer Mixture Loss (3.3 and Point Graph Decoder ($3.4).
Recamntruu acttion
Clobal Encoder
Giobei Decudder
gloee
Olimpoe Ports
Build Viaii Grld
geronnttriztion
Fointii Mix
Prac Flw
Maxic Dccode
pe-sommetratioo
0.99
Viset Grld Enooder
Glimpse Procosal
Glimpse Encoder Gäimps Decoder
becer strrs
FE wRo Mact
Cimpps 2
(a) Structure of SPAIR3D
(b) Structure of Glimpse VAE
Figure I: (a) Structure of SPAIR3D. For better illustration, we adopt 2D abstraction and use colors to highlight important correspondence. (b) Structure of Glimpse VAE. Glimpse encoder encodes foreground glimpses and produce z. maak and 2, for each glimpse. Point Graph Decoder takes zhahat and reconstructs input points (left branch). Mask Decoder takes z and generates masks for each point (middle branch). The dashed line represents the dependency the coordinates of the intermediate points the hierarchy and G. Multi-layer PointGNN networks enable message passing between (c,, f.) and produces z jares (right branch).
3.1 LOACL OBJECT PROPOSAL
As shown in Fig. La SPAIR3D first divides a 3D scene into a spatial attention voxel grid. There can be empty voxel cells covering points. We discard empty cells and associate bounding box with each non-empty voxel cell. The set of input points captured by bounding box is termed an object glimpse. Besides object glimpses, SPAIR3D also defines input scene. Later, we show that we encode and decode (reconstruct) points in each glimpse and generate a mixing weight each point form a probability mixture model.
glimpse covering all points in
3.2 GENERATIVE MODEL
Similar to SPAIR, each grid cell generates posterior distributions over a set of latent variables defined z5ell (2. (2,chere, 2, apothemy where z. whhheee I& encodes the relative position of the center of the ith bounding box to the center of the 7th cell, z; apothem g encodes the apothem of the bounding box. Thus, each zęel induces one object glimpse associated with the 7th cell. Each object glimpse is then associated with posterior distributions over latent variables specified as abject (z,chat, ..mask zfres,, where zhhha E A encodes the structure information of the corresponding object glimpse, 2, xmask E B encodes the mask for each point in the glimpse, z E =10.1 is binary variable indicating whether the proposed object should exist (z, (2)rrs =) not (z. (z. = 0).
The scene glimpse is associated with only latent variable =cene (zghhtt). We assume 2ires follows a Bernoulli distribution. The posteriors and priors of other latent variables are set to isotropic Gaussian distributions (see the appendix Sec. A for details).
Given latent representations of objects and the scene, the complete likelihood for a point cloud X' is formulated as p(x') = ((z))((A'lz)d) where z (U, (U)z U (U, z, 2 U z scene As maximising the objective p(((' is intractable, we resort to variational inference method to maximize its evidence lower bound (ELBO).
3.3 CHAMFER MIXTURE Loss
Unlike generative model-based unsupervised 2D segmentation methods that reconstruct the pixel-wise appearance conditioning its spatial coordinate, the reconstruction of a point cloud lost its point-wise
3

---

Under review as a conference paper at ICLR 2022
correspondence to the original point cloud. Chamfer distance is commonly adopted to measure the discrepaney between the generated point cloud (2t and the imput point cloud (2).. Formally, Chamfer distance is defined by de (X, X) Enex min ae, || refer to the first and the second term on the r.h.s as the forward loss and the backward loss, respectively.
_ Lee minlge || =|| We
Unfortunately, the Chamfer distance does not fit into the variational-inference framework. To get around that, we propose a Chamfer Mixture Loss (CML) tailored for training probability mixture models defined on point clouds. The Chamfer Mixture Loss is composed of a forward likelihood and a backward regularization corresponding to the forward and backward loss, respectively.
Denote the tth glimpse as G., i E {o............... {O.....n} and its reconstruction as G., i {O..... n). Specifically,
we treat the scene glimpse as the 0th glimpse that contains all input points, that is, 9o = X. Note that one input point can be a member of multiple glimpses. Below we use N ((,, o (2x to denote the probability density value of point x evaluated at Gaussian distribution of mean μ and variance o. For each input point in the 7th glimpse, the glimpse-wise forward likelihood of that point is defined as Cf (x) = u. max (())), where U, = frex maxxEE (V, (x)dd is the normalizer and o. is a hyperparameter. For each glimpse 9;, E (o..... n). of E [0, defines a mixing weight for point x in the glimpse and Ei= of 1. In particular, af TT. where π. is the predicted mask value and .द 0 x & G.. The mixing Di=1 of for x E Go-
1] .n... determined by
2; z weight for the scene layout points completes the distribution through og = Ei oz C. (x:). The total forward Thus, the final mixture model for input point x: is C (=(x)= =
af =,"""" n.
likelihood of X' is then defined as L (x) IIrex L (x)).
The forward likelihood alone leads to a trivial aub-optimal solution with Le distributed densely and uniformly in the space. To enforce high-quality reconstruction, define backward reg- ularization term. For each predicted point é, the point-wise backward regularization is defined as L (.) max Nxx, (.), where i(.) returns the glimpse index of å. We denote (00) = arg maxay (1x, (+) and X = U. G. The backward regularization is then defined as L (X) IT= ILaed ( The exponential weighting, i... a; =()) [0, 1], is erucial. As
each predicted point å E A' belongs to one and only one glimpse, it is difficult to impose a mixture model interpretation on the backward regularization. The exponential weighting encourages the generated points in object glimpse to be close to input points with high probability belonging to 9,. Combining the forward likelihood and the backward regularization together, we define Chamfer Mixture Loss as LCD (X, , x) = C (2)) (²)). During inference, the segmentation label for each point is naturally obtained by arg max; a².
The overall loss function is L KL divergence between the prior and posterior of the latent variables (appendix Sec. A for details). In general, the normalizers in Chamfer Mixture Loss don't have closed-form solutions. However, the experiments below show that we can safely ignore the normalization constants during optimization.
log LeD(X, &) L obbject sceee)) where Laz is the
3.4 MODEL. STRUCTURE
We next introduce the encoder and decoder network structure for SPAIR3D. The building blocks are based on graph neural networks and point convolution operations (See appendix Sec. C for details).
Encoder network. We design an encoder network (2(xx) to obtain the latent representations {2pel) J==1 and (z.
objectt.n [ from point cloud. To achieve the spatially invariant property, we group
one PointConv (Wu et al.) 2019) layer and one PointGNN (Shi & Rajkumar] (020)) layer into pairs for message passing and information aggregation among points and between cells.
(a) Voxel Grid Encoding. The voxel-grid encoder takes a point cloud as input and generates for each spatial attention voxel cell C, two latent variables 2.there E IR3 and z, opothem glimpse G, potentially occupied by an object.
R to propose a
To better capture the point cloud information in C, we build a voxel pyramid within each cell C, with the bottom level corresponding the finest voxel grid. We aggregate information hierarchically using CaanCooo-PPinGNNN pairs from bottom to top through each level of the pyramid. For each layer of the pyramid, we aggregate the features of all points and assign it to the point spawned the
4

---

Under review as a conference paper at ICLR 2022
center of mass of the voxel cell. Then PointGNN is employed to perform message passing on the radius graph built on all spawned points. The output of the final aggregation block produces z; and 2
zhehere
apothem via the re- pa-aametrrizaionn trick (Kingma & Welling 2014).
We obtain the offset distance of glimpse center from its corresponding grid cell center using Ag: = tanh(2
L, where L is the maximum offset distance. The apothems of the glimpse in the 2; y, = direction is given by Ag ( (rmoz pmin) + pmin where T(-) is the sigmoid function and [rmin pmaas] defines the range of apothem.
(b) Glimpse Encoding. Given the predicted glimpse center offset and the apothems, we can associate one glimpse with each spatial attention voxel cell. We adopt the same encoder structure to encode each glimpse G into one point a; = (c,, f.), where C; is the glimpse center coordinate and f, is the glimpse feature vector. We then generate zi and z z.nask from a, via the re- pe-prammterizationn trick.
The generation of 2. z determines the glimpse rejection process and is crucial to the final decom- position quality. Unlike previous work (Crawford & Pineau 2019 Lin et al., (020), SPAIR3D generates zPres from glimpse features instead of cell features based on our observation that message passing across glimpses provides more benefits in the glimpse-rejection process. To this end, radius graph is first built on the point set ((c,, } to connect nearby glimpse centers, which is followed by multiple PointGNN layers with decreasing output channels to perform local message passing, The z!"re* of each glimpse is then obtained via the re- ce-paramcterization trick. Information exchange between nearby glimpses can help avoid over-segmentation that would otherwise occur because of the high dimensionality of point cloud data.
(c) Global Encoding. The global encoding module adopts the same encoder as the object glimpse encoder to encode scene glimpse Go. The learned latent representation is 20 wichot with =fres 1.
Decoder network. We now introduce the decoders used for point-cloud and mask generation.
(a) Point Graph Decoder (PGD). Given the x: pbject of each glimpse, the decoder is used for point- cloud reconstruction as well as segmentation-mask generation. In reconstruction, the number of generated points has a direct effect on the magnitudes of the forward and backward terms in the Chamfer Mixture Loss. An unbalanced number of reconstruction points can lead to under- or over- segmentation. To balance the forward likelihood and the backward regularization, the number of predictions for each glimpse should be approximately the same the number of input points. We propose a graph network based point decoder to allow setting the size of &' in run time.
Similar to (Luo & Hu 2021), we treat the point cloud reconstruction as a point diffusion process. The input to the PGD is set of 3D points with coordinates sampled from a zero-centered Gaussian distribution, with the population determined by the number of points in the current glimpse. Features of the input points are set uniformly to the latent variable z. z.chat PGD is composed of several PointGNN layers, each of which is preceded by a radius graph operation. The output of each PointGNN layer is of dimension f+3, with the first f dimensions interpreted as the updated features and the last 3 dimensions interpreted as the updated 3D coordinates for estimated points. Since we only focus on point coordinates prediction, we set f for the last PoinIGNN layer.
(b) Mask Decoder.
The Mask Decoder decodes (c,, 2, x.mass) to the mask value, n/ E [0, 1). of each point within a glimpse G.. The decoding process follows the exact inverse pyramid structure of the Glimpse Encoder. To be more precise, the mask decoder can access the spatial coordinates of the intermediate aggregation points of the Glimpse Encoder as well as the point coordinates of G, During decoding, PointConv is used as deconvolution operation.
Glimpse VAE and Global VAE. The complete Glimpse VAE structure is presented in Fig. [b] The Glimpse VAE is composed of Glimpse Encoder, Point Graph Decoder, Mask Decoder and multi-layer PointGNN network. The Glimpse Encoder takes all glimpses as input and encodes each glimpse G, individually and in parallel into feature points (c,, f.). Via the re- pe-arammterrizaiion trick, 2. ziaatt and z.uas are then obtained from f, From there, we use the Point Graph Decoder to decode z, whaat to reconstruct the input points, and we use the Mask Decoder to decode zmmas to assign a mask value for each input point within Gi Finally, z,res is generated using message passing among neighbour glimpses. The processing of all glimpses happens in parallel. The Global VAE consisting of the Global Encoder and a PGD outputs the reconstructed scene layout.
5

---

Under review as a conference paper at ICLR 2022
3.5 SOFT BOUNDARY
The prior of zopohhem set to encourage apothem to shrink so that the size of the glimpses will not be overly large. However, if points are excluded from glimpse, the gradient from the likelihood of the excluded points will not influence the size and location of the glimpse anymore, and this can lead to over-segmentttion. To solve this problem, we introduce soft boundary weight oy E [0, 1] which decreases when a point x E 9i moves away from the bounding box of Gr. Taking by into
2 #6 " the computation of a, we obtain an updated mixing weight o 2 25 " % 2 T. By employing such boundary loss, the gradual exclusion of points from glimpses will be reflected in gradients to counter over-segmentation. Details can be found in appendix Sec. B
4 EXPERIMENTS
4.1 DATASETS
While many benchmark datasets are established (Johnson et al.) 2017: Kabra et al. 2019) for unsupervised object-centric learning, they don't provide camera poses and depth images, which are necessary for point cloud generation. Thus, introduce two new point-clout datasets Unity Object Room and Unity Object Table built on the Unity platform (Juliani et al., 2020). The Unity Object Room (UOR) dataset is built approximate Object Room (Kabra et al. 2019) dataset but with increased scope and complexity. In each scene, objects sampled from a list of 8 regular geometries are randomly placed on a square floor. The Unity Object Table (UOT) dataset approximates Robotic Object Grasping scenario where multiple objects are placed on a round table. Instead of using objects of simple geometries, we create each scene of the UOT dataset with more challenging objects selected from pool of 9 irregular objects. For both datasets, the number of objects placed in each scene varies from 2 with equal probabilities. During the scene generation, the size and orientation of the objects are varied randomly within pre-defined range.
We capture the depth, RGB, normal frames, and pixel-wise semantics as well as instance labels for each scene from 10 different viewpoints. This setup aims to approximate the scenario where a robot cquipped with depth and RGB sensors navigates around target objects and captures data. The point cloud data for each scene is then constructed by merging these 10 depth maps. For each dataset, we collect 50K training scenes, 10K validation scenes and 5K testing scenes. In-depth dataset specification and analysis be found in supplementary D.
4.2 UNSUPERVISED SEGMENTATION
Baseline. Due to the sparse unsupervised 3D point cloud object-centric learning literature, we could not find a generative baseline to compare with. Thus, we compare SPAIR3D with PointGroup (PG) (Jiang et al. (020). a recent supervised 3D point cloud segmentation model. PointGroup performs semantic prediction and instance predictions from a point cloud and RGB data using a model trained with ground-truth -ruth semantic labels and instance labels. To ensure fair comparison, we assign each point the same color (white) so appearance information doesn't 't play a role. The PointGroup network fine-tuned on the validation to achieve the best performance.
Performance Metric. We use the Adjust Rand Index (ARI) (Hubert & Arabie 1985) to measure the segmentation performance against the ground truth instance labels. We also employ foreground Segmentation Covering (SC) and foreground unweighted mean Segmentation Covering (mSC) (Engelcke al. 2020) for performance measurements ARI does not penalize models for object
over-segmentation EEggllcee et al., 2020).
Evaluation. Table shows that SPAIR3D achieves comparable performance to the supervised baseline on both UOT and UOR datasets. As demonstrated in Fig. [2, each foreground object is proposed by one and only one glimpse. The scene layout is separated from objects and accurately modelled by the global VAE. It is worth noting that the segmentation errors mainly happen at the bottom of objects. Without appearance information, points at the bottom of objects are also correlated to the ground. In Fig. 4 we sort the test data based on their performance in an ascending order and plot the performance distributions. As expected, the supervised baseline (Orange) performs better but SPAIR3D manages to achieve high-qualit segmentation (SC score > 0.8) around 80% of the scenes without supervision. See appendix Sec. E for more segmentation results including failure cases.

---

Under review as a conference paper at ICLR 2022
(a) UOR imput
(b) our reconstruction
(c) UOT input
(d) our reconstruction
(c) UOR instance label
(l) our segmentation
(g) UOT instance label
(h) our segmentation
(i) UOR close-up glimpses visualization, fore- ground alpha and scene layout reconstruction.
① UOT close-up glimpses visualization, fore- ground alpha and scene layout reconstruction.
Figure 2:
Visualization of segmentation results UOR and UOT dataset.
Table 3D point cloud segmentation results on UOR dataset (blue) and UOT dataset (red).
UOR
ARI1
SC1
mSCt
UOT
0.976
0.907
0.900
PG
0.923
0.917
0.907
0.915 + 0.03
0.832 + 0.04
0.836 + 0.04
Ours
0.901 + 0.02
0.835 + 0.03
0.831 + 0.03
voxel size 0.757
0.932
0.853
0.850
voxel size 1.251
0.922
0.857
0.861
0.912
0.846
0.842
6 - 12 objects
0.892
0.843
0.834
0.872
0.856
0.861
object matrix
0.879
0.877
0.886
worth noting that UOR and UOT datasets include around 25% objects (for scenes of to objects) and 60% objects (for scenes of 12 objects) that close its neighbors with touching or almost touching surfaces. The reported quantitative (Table I and qualitative results (Fig. (dd) show that our method achieves stable performance for those challenging scenes.
D6x
cap
Figure 3: t-SNE visualization of 2 what on UOT
7

---

Under review as a conference paper at ICLR 2022
04
20
800
05-
PG AB
USA NC
03:
LGR m1
UON Adlition A1J
04
JOL RoMiec SC
UOB An MCC
2000
4020
3000
4000
3005
5000
Seroed
1.0
0.3
0.5
0.5
6.6
LOT S74R10 A3
BCT 00T22002 SC
LET SENITED RGC
Serted Indoo
Strted Indee
Serted Indox
Figure 4: Test set performance distributions UOR (first row) and UOT (second row).
(a) UOR instance label
(b) Our segmentation
(c) UOT instance label
(d) Our segmentation
(c) UOR instance lahel
(f) Our segmentation
(g) UOT instance label
(h) Our segmentation
Figure Segmentation on scenes with 12 objects (a d) and on object matrix (e-h)
4.3 OBJECT CENTRIC REPRESENTATION
To show that our model learns meaningful representations, for each object type in UOT dataset collect the 2 what of 200 instances and visualize them with t-SNE algorithm (van der Maaten & Hinton 2008). Fig 3 clearly shows the whot different object types cluster at different regions. Not surprisingly. the embeddings of pot and box instances occupy the same area since they have almost identical spatial structure. See appendix Sec. D for object specification.
4.4 VOXEL SIZE ROBUSTNESS AND SCALABILITY
In the literature (Lin et al., 2020 Crawford & Pineau 2019). the cell voxel size, an important hyperparameter, chosen to match the object size in the scene. To evaluate the robustness of our method w.r.t voxel size, we train our model on the UOR dataset with voxel size set to 0.75/ and 1.25/ with being the average size of the objects. Results in Table show that our method achieves stable performance w.r.t the voxel size.
To demonstrate scalability, we evaluate our pre-trained model on 1000 scenes containing 12 randomly selected objects and report performance in Table Due the spatial invariance property, SPAIR3D suffers no performance drop on 6 12 object scenes that were never used for training.
We also evaluated our approach on scenes termed Object Matrix, which consists of 16 objects placed in a matrix form. We fixed the position of all 16 objects but set their size and rotation randomly. For each dataset, SPAIR3D is evaluated Table Note that our model is trained on scenes with to 5 objects, which is less than one-third of the number of objects in Object Matrix scenes. Fig 5 (h) is illustrative of the results.
100 Object Matrix scenes. The results are reported in
8

---

Under review as a conference paper at ICLR 2022
With PointGNNs
(b) PointGNNs
Figure 6:
The comparison between models with (left) and without (right) multi-layer PointGNNs. It shows that objects are over-segmented severely without multi-layer PointGNNs.
4.5 ABLATION STUDY OF MULTI-LAYER POINTGNN
To evaluate the importance of multi-layer PointGNN in z 2: generation (right branch in Fig. lbp. we remove the multi-layer PointGNN and generate z pres' directly from f.. The ablated model on the UOR dataset achieves ARI:0. 8411 SC:0.610, and mSC:0. 627, which significantly worse than the full SPAIR3D model. The performance distribution of ablated SPAIR3D (Fig 4 first row) indicates that removing the multi-layer PointGNN has negative influence on the entire dataset. Fig. O shows that the multi-layer PointGNN is crucial to preventing sereseegmntttion.
4.6 EMPIRICAL EVALUATION OF PGD
objects of the same category can be modeled by a varying number of points. The generation quality of the point cloud largely depends on the robustness of our model against the number of points representing each object. To demonstrate that PGD can reconstruct each object with a dynamic number of points, train the global VAE on the ShapeNet dataset (Chang et al. 2015), where each object composed of roughly 2000 points, and reconstruct the object with a varying number of points. For reference input point clouds of size N, force PGD reconstruct point cloud of size 1.5,, 1.25N. N, 0.75N, and 0.5.V, respectively. As shown in Fig- 7 while with less details compared to the input, the reconstructions capture the overall object structure in all 5 settings.
(a)
(b)
(c)
(d)
(e)
Figure 7: PGD trained on ShapeNet. (a) Input point cloud with points. Reconstruction with (b)
1.5N, (c) 1.25N, (d) N, (e) 0.75N and (f) 0.5NN points.
LIMITATIONS
SPAIR3D extends the SPAIR framework to point cloud data and inherits its strengths and limitations. Similar to SPAIR, the key achieving high scalability in SPAIR3D is the independence assumption among objects, which is reflected in the local attention and reconstruction mechanisms. The price we pay for that scalability that the local object proposal mechanism can only reliably detect objects whose size fall within a bounded range. By design, each voxel cell can only propose one object. Thus, is difficult detect multiple objects that exist in the same voxel cell. If one object is much larger than the size of the voxel cells, no voxel cells can accurately infer complete object information from its local perceptive field. Given these limitations are inherent to the SPAIR framework, they can really only be resolved at meta level. In our intended use case, intelligent agent that employs SPAIR3D as a subroutine can use voxel cells of vastly different sizes to segment out objects of different sizes and use spatial knowledge representation and reasoning formalism [Cohn & Renz (2008) to resolve inconsistencies and get to a full scene representation.
CONCLUSIIN AND FUTURE WORK
We propose SPAIR3D, to the best of our knowledge, the first generative unsupervised object- centric learning model on point cloud with applications to 3D object segmentation task. Based on experimental results on UOR and UOT datasets, we demonstrate that SPAIR3D can generalise well to previously unseen scenes with a large number of objects without performance degeneration. The spatial mixture interpretation of SPAIR3D allows the integration of memory mechanism (Bornschein jet al., 2017) or iterative refinement GGreff et al. 2019). Our model may tend to over-segment objects of significant different scale from those in the training dataset, which is left our future work.

---

Under review as a conference paper at ICLR 2022
7 ETHICAL STATEMENTS
Similar most data-driven approaches, the proposed method also potentially brings the risk of learning biases. While our object-centric representation learning has shown good generalisation ability new scenes, the application of our method should be done with careful consideration of the consequences from any potential underlying biases in the data-collection process.
REPRODUCIBILITY STATEMENT
We described in details our model structure, the choice of hyperparameter appendix Sec. B and
Dataset construction details are presented in appendix Sec D We believe our framework can be reproduced based on these information.
REFERENCES
Panos Achlioptas, Olga Diamanti, loannis Mitliagkas, and Leonidas Guibas. Learning representations and generative models for 3d point clouds. In ICMI,, pp. 40-49. PMLR, 2018.
J. Bornschein, A. Mnih, Daniel Zoran, and Danilo Jimenez Rezende. Variational memory addressing in generative models. In NIPS, 2017.
Christopher Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. ArXiv, abs/1901.11390 01 2019. URL http:shtts///
901 1 11390
Christopher P. Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in beta-vae. ArXiv,
2018.
Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song. Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An Information-Rich 3D Model Repository. Technical Report 1rXiv:1512.0301 [cs.GR], Stanford University Princeton University Toyota Technological Institute Chicago, 2015.
B. Michael Chang, Tomer Ullman, Antonio Torralba, and B. Joshua Tenenbaum. A compositional object-based approach learning physical dynamics. ICLR, 2017.
Chang Chen, Fei Deng, and Sungjin Ahn. Roots: Object-centric representation and rendering of 3d scenes, 2021.
Anthony G. Cohn and Jochen Renz. Qualitative spatial representation and reasoning. In Frank van Harmelen, Vladimir Lifschitz, and Bruce Porter (eds.), Handbook of Knowledge Representation, volume 3 of Foundations of Artificial Intelligence, pp. 551-596. Elsevier, 2008.
Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convolutional neural networks. AAAI, 33:3412-3420, 07 2019. doi:
Eric Crawford and Joelle Pineau. Exploiting spatial invariance for scalable unsupervised object tracking. AAAI, 34:3684-3692, 04 2020. doi: 10.
Carlos Diuk, Andre Cohen, and Michael L. Littman. An object-oriented representation for efficient
reinforcement learning. In /CML, ICML '08, pp. 240-247. New York, NY, USA, 2008. Association for Computing Machinery. ISBN 6781605582054. doi: 10.1145/1390155.19901 https:://8oi.org/100 1tt.:/....1111199111 httt::/:///ooooo///0////////0100 1390187
URL
Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Generative scene inference and sampling with object-centric latent representations. In ICLR, 2020. URL
https:://ppenevviee net hww.
10

---

Under review as a conference paper at ICLR 2022
S. M. Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Koray Kavukcuoglu, and Geoffrey E. Hinton. Attend, infer. repeat: Fast scene understanding with generative models. In Proceedings of the 30th International Conference Neural Information Processing Systems, NIPS' 16, pp. 3233-3241, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819.
S. M. Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S. Morcos, Marta Garnelo, Avraham Ruderman, Andrei A. Rusu, Ivo Danihelka, Karol Gregor, David P. Reichert, Lars Buesing, Theophane Weber, Oriol Vinyals, Dan Rosenbaum, Neil Rabinowitz, Helen King, Chloe Hillier, Matt Botvinick, Daan Wierstra, Koray Kavukcuoglu, and Demis Hassabis. Neural scene representation and rendering. Science. (2046 10.1126/science aar661 70. eeee sccccccccm........ccccceeee.. wwwp://www....oooo..o..o.. 6394/1204]
2018. ISSN 0036-8075. doi:
Matheus Gadelha, Rui Wang, and Subhransu Maji. Multiresolution tree networks for 3d point cloud processing. In ECCV, pp. 103-118, 2018.
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. In NeurlPS, NIPS' 17, pp. 6694-6700 6704, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860999
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loie Matthey, Matthew M Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In ICML, 2019.
Paul Henderson and Christoph H. Lampert. Unsupervised object-centric video generation and decomposition 3D. In NeurIPS, 2020.
Irina Higgins, Loie Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew M Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In ICLR, 2017.
L. Hubert and Arabie. Comparing partitions. Journal of Classification, 2:193-218, 1985.
Md Amirul Islam*, Sen Jia* and Neil D. Bruce. How much position information do convolutional neural networks encode? In International Conference on Learning Represeniations, 2020. URL
reB3dddree6KKk httpps://geeneeveew..ett net htteee//foommmmrre333333333
Li Jiang, Hengshuang Zhao, Shaoshuai Shi, Shu Liu, Chi- Chi-Wing Fu, and Jiaya Jia. Pointgroup: Dual-set point grouping for 3d instance segmentation. CVPR, 2020.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Zitnick, and Ross Girshick. Clevr: diagnostic dataset for compositional language and elementary visual reasoning. pp. 1988-1997, 07 2017. doi: 10.1109/CVPR.2017..25
A. Juliani, V. Berges, E. Teng, A. Cohen, J. Harper, C. Blion, C. Goy. Y. Gao, H. Henry, M. Mattar, and D. Lange Unity: general platform for intelligent agents. ArXiv, abs/ 00211889.022 026.7,227 2020.
Rishabh Kabra, Chris Burgess, Loie Matthey, Raphael Lopez. Kaufman, Klaus Greff, Malcolm Reynolds, and Alexander Lerchner. Multi-object datasets. https///ghhbb.co//eemmm//////// object-datasets/, 2019.
Daniel Kahneman, Anne Treisman, and Brian J Gibbs. The reviewing object files: Object-specific integration of information. Cognitive Psychology. 24(2): 219, 1992. ISSN 0010-0285. doi: http:://6oaogg////0/o1ooo0000022 100.0006600000025599900000000000 /ccience/arrriceee//////00222222
TtLLhtt::/ www. a www.saciencedirec..t com/
Ken Kansky, Tom Silver, David A. Mély, Mohamed Eldawy, Miguel Lázaro-Gredilla, Xinghua Lou, Nimrod Dorfman, Szymon Sidor, D. Scott Phoenix, and Dileep George. Schema networks: Zero-shot transfer with generative causal model of intuitive physics. ArXiv, abs/1706.00317. 2017.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
11

---

Under review as a conference paper at ICLR 2022
Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
Eric Kolve, Roozbeh Mottaghi. Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. A12-THOR: An Interactive 3D Environment for Visual AI. arXiv, 2017.
Nanbo Li, Cian Eastwood, and Robert Fisher. Learning object-centric representations of multi- object scenes from multiple views. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 5656- 5666. Curran Associates, Inc., 2020. URL /tpocppss//roocedigggg www.://inn...uuii..co.o.oo.o www..ip.....ppp......eeeeeeeeeee http:///e....a222222222222222222
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial
attention and decomposition. In ICLR, 2020. URL htttttts://pppeeeieeeeee bttpenneeveww.ee/forumm
Id=rk103ySYDH
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In NeurlPS, 2020.
Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021.
W. Shi and R. Rajkumar. Point-gnn: Graph neural network for 3d object detection in a point cloud. In CVPR, pp. 1708-1716, 2020.
Karl Stelzner, Kristian Kersting, and Adam R. Kosiorek. Decomposing 3d scenes into objects via unsupervised volume segmentation, 2021.
Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method. In Proc. of the 37-th Annual Allerton Conference on Communication, Control and Computing, pp. 368-377, 1999. / wwwoor/g//aoooo..oooo /ttpab/sssyhpeeaa http//hyssscssc /000
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Ma-
chine Learning Research, 9:2579-2600 2008. URLhttp: http://www jmlr. org/papprs/vv/
Wandermaaten ht.
Nicholas Watters, Loic Matthey, Christopher Burgess, and Alexander Lerchner. Spatial broad- cast decoder: A simple architecture for learning disentangled representations in vaes. ArXiv, abs/1901.07017,, 08 2019. URL arxiv. org/ abs /1t...91.... 07017
Wenxuan Wu, Zhongang Qi, and Fuxin Li. Pointcony: Deep convolutional networks on 3d point clouds. In CVPR, pp. 961 9613-9622, 06 2019. doi: 10. D0.1109VVR.220190000
Guandao Yang, Xun Huang, Zekun Hao, Ming- Yu Liu, Serge Belongie, and Bharath Hariharan. Pointflow: 3d point cloud generation with continuous normalizing flows. pp. 4549, 10 2019. doi:
19.044444
12

---

Under review as a conference paper at ICLR 2022
A HYPERPARAMETERS AND PRIOR DISTRIBUTIONS
In this section, we present hyperparameters we used in our model as well as definitions of the prior distributions of latent variables. We use tuples in the form of (n, m, p. to denote annealing of hyperparameter values from n to m, starting from iteration to iteration q. Glimpse related Priors are specified in Table Other hyperparameters
hyperparameters are shown in Table 2 specified in Table
Table 2: Glimpse related hyperparameters.
Term
Value
spatial attention grid cell size
glimpse max apothem glimpse min apothem glimpse max center offset
1
0.25
0.75
Table 3: Prior distributions.
Term
Value
1 uyothem
(2, 1, 10000, 20000) (0.01, 0.0001, 15000)
3rres
x whheee prior
(.,,
gapothem prior N(aapohhem 0.5)
pres prior Bernoulli( (Srees)
2 what prior N 1) ((),
2 mask prior
Table 4: Other hyperparameters.
Term
Value
CML O.
(0.1, 0.05, 10000, 15000) NO0, oz)
CML dist.
relaxed Bernoulli temp
(2.5, 0.5, 0, 10000)
PGD initial distribution
(.0,0.3)
Mixing Weight. Recall that the mixing weight defined in Eq. (6) in our main paper for the forward Chamfer Likelihood implies the segmentation of the point cloud. To encourage the mixing weight to approach ( or 1, we include temperature factor 10 into the computation of of, and the mixing weight is implemented as
(2f 1
O
Weight for Regularisation with Priors. Here we introduce the detailed formulation of LKL, the second term of the evidence lower bound, in the following Eq. 2
Czl 2 object , zcene) = wD wD (DDD)p(((2p (lg(()re)) (z)) )- + grees
Dx (pzzpw what laa what (z))
D (p(z mask
119 mask lo:))+
(2)
Pri(p(e where
11g(2 where [x:))+
D (p(2 spothem
|| apothem [x))]]
Note that the weight for KL Divergence of prres is to encourage glimpse rejection. More specifi- cally, w is set to 10 for UOR dataset and is annealed from 10 to 20 in the first 15000 steps for UOT dataset in experiments. Following Crawford & Pineau (2019), we also set the weight for KL divergence of the rest latent variables with pres so that the rejected glimpses won't 't produce penalties encouraging glimpse rejection.
13

---

Under review as a conference paper at ICLR 2022
Training. We use Adam (Kingma & Ba 2014) optimizer with learning rate to 0.0001 during training process. Training takes roughly days on one RTX 3090 for both datasets.
B SOFT BOUNDARY
glimpse boundary
1.0
glimpse
0.6
gradlent
-1.0
W
0.2
-15
0.0
-2.0
0.0
0.4
0.5
0.0
2.0
d/w
(c) Illustration of glimpse boundary structure.
(a) Boundary weight function.
(b) Gradient.
Figure 8: Visualization of glimpse boundary structure, glimpse boundary weights and the correspond- ing gradients. Fig. [8 iSllststtss the glimpse structure where is the glimpse center and is an point living in the glimpse boundary. The linear decay function (orange) and parabola decay function (blue) are plotted in Fig. Sa with the corresponding gradients shown Fig. 8b
As shown in Fig 8c divide glimpse into glimpse interior of apothem and glimpse boundary of width w. For an input point r in glimpse Gj with center location C, its boundary weight by e (0, is defined as
x E glimpse interior; (f(d w),
by
x E glimpse boundary,
where / is continuous function and d = |0 C inf
One example of is linear decay function (d, w) = d. Another example is parabola decay function f(d, w) = ($) 2( 4 1,, which generates a larger gradient compared with the linear one when d S 2. and smaller gradient, otherwise. For the case where the distance between two objects smaller than w but larger than 2. parabola decay function preferred to preventing overly large glimpse. In this work, w. = 0.75r.
w0
parabola decay function and set the width of the glimpse boundary as
From Fig. Sa and Fig. 88b observe that both types of boundary decay function / provide negative gradient when points are being excluded from a glimpse which has increasing value.
C MODEL STRUCTURE
In this section, we introduce the detailed model structure. We first introduce the notation that we used in our network structure. One PointConv layer is specified with (Cmid: Cout) (Wu al,, 2019) (See Fig. for structure illustration). To achieve spatially invariant operation in PointConv, we divide feature value of each point by the total number of points involved for each point convolution operation instead of the pre-computed point density as in (Wu et al. 2019).
One PointGNN layer contains three two-layer MIPs which are M M.P,, MLP, and MLP, respec- tively (Shi & Rajkumar 2020) and the structure can be summarized by
out
g (max JEN.
h(s)"
where 8!"h and sgut are the features of the point before and after PointGNN layer, N; defines the points connected to i, z; indicates the 3D coordinate of the point i. The max operation is performed over all points that are connected to the point i. For all PointGNN layers, we consistently set hhidden = 32 and hout = 3. Thus, we define the structure of a PointGNN layer with parameters (Shidden; Jouts Ghidden I We also list the parameters for other operations such the radius
14

---

Under review as a conference paper at ICLR 2022
for radius graph operation, the voxel cell size for voxel pooling operation, and the output size for linear layers in following tables. Table shows the structure of voxel grid encoder. The structure of glimpse VAE including the glimpse encoder, the mask decoder, the glimpse Point Graph Flow, and the Multi-layer PoinGGNN is presented T Table Table 8 and Table 11. The structure of global VAE including the Global encoder and the Global Point Graph Flow is detailed in Table TO
and Table 7
Input point coordinates are reduced by factor of 16.
Table Voxel grid encoder.
Table 6: Glimpse encoder.
Layer/Operation
Parameter
Layer'Operation
Parameter
Radius Graph
0.0625
Radius Graph
0.25
PointConv
(8,8)
PointGNN
88,8,8,8)
Celu
LayerNorm
PointConv
(16, 16)
Voxel Pool
0.25
Celu
PointConv
(16, 32)
PointConv
(32,32)
Celu
Celu
Radius Graph
0.5
Voxel Pool
0.03125
PointGNN
(32,32, 32,
PointConv
(32, 64)
LayerNorm
Celu
Voxel Pool
0.25
Radius Graph
0.03125
PointConv
(64, 128)
PointGNN
(64, 64, 64, 64)
Celu
LayerNorm
Radius Graph
1.0
Voxel Pool
0.0625
PointGNN
(128, 128, 128, 128)
PointConv
(64, 128)
LayerNorm
Celu
PointConv
(128, 256)
Radius Graph
0.125
Celu
PointGNN
(128, 128, 128, 128)
Linear
256
LayerNorm
Voxel Pool
0.125
PointConv
(128, 256)
Celu
Radius Graph
0.25
PointGNN
(256, 256, 256, 256)
Table 7: Global Point Graph Flow.
PointGNN
(256, 256, 256, 256)
PointGNN
(256, 256, 256, 256)
Layer/Operation
Parameter
LayerNorm
PointConv
226)
Random Sampling
Radius Graph
0.2
Linear
12
PointGNN
(128, 128, 128, 64 3)
Radius Graph
0.1
PointGNN
(64, 64, 62,32 +
Radius Graph
0.05
PointGNN
(16, 16, 16,3)
Table 8: Glimpse Point Graph Flow.
Layer/Operation
Parameter
Random Sampling
Radius Graph
0.2
PointGNN
(128, 128, 128, 64 + 3)
Radius Graph
0.1
PointGNN
(64, 64, 64, + 3)
Radius Graph
0.05
PointGNN
(16, 16, 16,3)
15

---

Under review as a conference paper at ICLR 2022
Table Mask decoder.
x Cout
Layer/Operation
Parameter
MLP
PointConv
(64, 32)
x (Cin Cmiia
Celu
Matrix Multiplication
PointConv
(16, 16)
Celu
PointConv
(8, 8)
N Cynid
Input Feature N x Cin
Celu
MLP
Linear
MLP
Input Pos N
Figure 9; Structure of PointConv.
Table 10: Global encoder.
Layer/Operation
Parameter
Radius Graph
0.25
PointGNN
(8, 88,
LayerNorm
Voxel Pool
0.25
PointConv
(16, 32)
Cclu
Radius Graph
0.5
PointGNN
(32,32,32, 32)
LayerNorm
Voxel Pool
0.25
PointConv
(64, 128)
Celu
Radius Graph
1.0
PointGNN
(128, 128, 128, 128)
LayerNorm
PointConv
(128, 256)
Celu
PointConv
(256, 512)
Table 11: Multi-layer PointGNN.
Layer/Operation
Parameter
Random Sampling
Radius Graph
1.0 (128, 64, 64, 64) (32, 32, 32, 32) (16, 16, 16
PointGNN
PointGNN
PointGNN
Linear
16

---

Under review as a conference paper at ICLR 2022
D DATASET SPEC
We provide more details about the UOR and UOT dataset in this section. For both datasets, in each scene, 2-5 objects are uniformly randomly selected (with replacement) from the candidate set and placed at random locations in the scene with the constraint that they cannot largely overlap with each other (slight overlapping and touching are permitted). All objects are randomly rotated along y-axis.
For the point cloud construction, we convert the 10 depth frames for each scene into 10 partial point clouds. To down sample the point clouds, we apply voxel grid pooling on the 10 partial point clouds. More specifically, a voxel grid with cell size 0.15 is placed across the space where each partial point cloud exists. For all points in one cell, the coordinates are aggregated by average pooling operation. After we merge the 10 partial point clouds into one, we perform voxel grid pooling again to get the final input point cloud. We divide the coordinates of all points by 8 to scale the complete point clouds.
The intrinsic parameters of cameras are shown in Table which is crucial to accurate point cloud construction. The object pool for UOR and UOT are presented Table 13 and Table 14 respectively, where we also specify the detailed dimension range of each object.
Table 12: Camera intrinsic parameters
Term
Value
focal length
10 mm
sensor size x
16 mm
sensor size y
16 mm
clipping plane
20 m
Data captured by each camera in UOR dataset (top) and UOT dataset (bottom). From left to right are RGB, depth, normal, instance label, semantic label and constructed point cloud. Point clouds are obtained by merging multi-view depth images. Instance labels and semantic labels used to train PointGroup (Jiang et al. 2020) baseline. RGB images and normal maps are not used in this work.
Figure 10:
To show that dataset is indeed challenging for models capturing spatial structure correlations, plot the empirical closest neighbor distance distribution of our data generation process. To obtain the distribution, for each object in each scene, note down the distance between the center of this object and the center of its closest neighbor (the surface to surface distance is hard to compute). Thus, distance means complete overlapping between two objects, which is not physically plausible. In our UOR and UOT dataset the minimum distance is set to one (not applicable to Object Matrix layout). With object dimensions specified in Table 13 and Table [4 we consider distance below to be extremely close. Thus, reading from Fig 11 for scenes containing 2-5 objects, 25 percent of objects are spawned close to at least one other object. For scenes containing 6- objects, the number goes up to more than 60 percent.
17

---

Under review as a conference paper at ICLR 2022
1.0
0.8
Probability
0.4
0.2
8
Distance
0.8
Probabliity
0.6
0.4
0.2
0.0
4
7
Distance
Figure 11: Closest neighbor distance distribution of our data generation process for 2-5 objects (above), and for 6-12 objects (below)
18

---

Under review as a conference paper at ICLR 2022
Table 13: UOR object pool.
min length/max length
min width/max width
min height/max height
geometry
figure
Capsule (standing)
0.75/1.25
0.75/1.25
1.5/2.5
Capsule (flat)
1.5/2.5
1.5/2.5
0.75/1.25
Cube
0.75/1.25
0.75/1.25
0.75/1.25
Cylinder (standing)
0.75/1.25
0.75/1.25
1.5/2.5
Cylinder (flat)
1.5/2.5
1.5/2.5
0.75/1.25
Hexagonal Prism
1/1.33
1/1.33
0.5/0.83
Sphere
0.75/1.25
0.75/1.25
0.75/1.25
Rhombicosiddddecaheddrn
0.75/1.25
0.75/1.25
0.75/1.25
Square Antiprism
1/1.5
1/1.5
0.67/1
Triangular Prism (standing)
0.75/1.5
0.75/1.5
0.75/1.5
Triangular Prism (flat)
0.75/1.5
0.75/1.5
0.75/1.5
19

---

Under review as a conference paper at ICLR 2022
Table 14: UOT Object pool. Some object meshes are obtained from ai2thor (Kolve et environment.
2017
min length/max length
min width/max width
min height/max height
object
figure
Chess piece
0.69/0.86
0.69/0.86
1.43/1.80
Bear
1.1/1.46
0.71/0.95
1.06/1.41
Box
0.98/1.23
0.98/1.23
0.67/0.84
Car
1.17/1.96
0.74/1.24
0.70/0.78
Cup
0.87/1.10
0.70/0.87
0.92/1.16
Kettle
0.85/1.13
0.69/0.91
1.01/1.35
Pot
1.06/1.42
0.88/1.17
0.80/1.07
Tissue
1.05/1.31
0.95/1.18
1.16/1.44
Toaster
0.58/0.78
1.01/1.36
0.61/0.81
20

---

Under review as a conference paper at ICLR 2022
E MORE RESULTS
In this section, we show more segmentation results our model. To demonstrate that our model can handle non-trivial scene layout, we apply SPAIR3D classic Object Room dataset (Eslami al. 2018) and show quantitative results in Fig. 12
(a)
(b)
(d)
Figure 12: Scene layout in Object Room dataset has four walls. SPAIR3D groups four walls together with the floor in each scene as the scene layout component. Column (a) and column (c) are instance labels. Column (b) and column (d) are the corresponding SPAIR3D segmentation.
We show more test cases where SPAIR3D achieves above 0.8 SC scores on UOR in Fig. 13 and UOT
in Fig. 14 To provide more comprehensive inspection, we further display segmentation results on scenes where SPAIR3D achieves below 0.7 SC scores on UOR in Fig. [5] and UOT in Fig. T6 respectively. Segmentation results demonstrated below, when examined together with performance distribution shown in Fig. 4 confirm that our model achieves high quality segmentation majority scenes.
More segmentation results scenes with 6 12 objects shown in Fig. and Fig. 18
In Fig 19
we show more examples on Object Matrix scenes to further demonstrate the scalability of our model.
21

---

Under review as a conference paper at ICLR 2022
(b)
Figure 13: Typical UOR test cases that achieves above 0.8 SC. Column (a) and column (c) are instance labels. Column (b) and column (d) are the corresponding SPAIR3D segmentation.
22

---

Under review as a conference paper at ICLR 2022
(a)
(b)
(d)
Figure 14: Typical UOT test cases that achieve above 0.8 SC. Column (a) and column (c) are instance labels. Column (b) and column (d) are the corresponding SPAIR3D segmentation results.
23

---

Under review as a conference paper at ICLR 2022
(d)
Figure 15: UOR test cases that achieves below 0.7 SC. Column (a) and column (c) are instance labels. Column (b) and column (d) are the corresponding SPAIR3D segmentation. Failed cases reflect that (1) objects clustered together are more vulnerable to mis-segmentation due to complicated local spatial structure layout (2) and objects with extreme dimensions, e.g. cylinder, are vulnerable to seeeetegmentttions.. Note that currently all derivatives of SPAIR (Crawford & Pineau 2019) sensitive to object size.
(a)
(b)
(d)
Figure 16: UOT test cases that achieve below 0.7 SC. Column (a) and column (c) are instance labels.
Column (b) and column (d) are the corresponding SPAIR3D segmentation results.
24

---

Under review as a conference paper at ICLR 2022
(a)
(b)
(d)
Figure 17: More results on UOR scenes with 6-12 objects. Column (a) and column (c) are instance labels. Column (b) and column (d) are the corresponding SPAIR3D segmentation.
(a)
(b)
(d)
Figure 18: More results on UOT scenes with 6-12 objects. Column (a) and column (c) are instance labels. Column (b) and column (d) are the corresponding SPAIR3D segmentation.
25

---

Under review as a conference paper at ICLR 2022
(a)
(b)
(c)
(d)
Figure 19: More results on UOR and UOT Object Matrix scenes. Column (a) and column (c) are instance labels. Column (b) and column (d) are the corresponding SPAIR3D segmentation results.
26


Reviews:
[{"review_id": "o_K9lRhqM2Q", "reviewer": "Reviewer_tBSM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes an unsupervised object-level segmentation framework from point cloud input. The framework is unsupervised, driven by a Chamfer Mixture Loss. Two training datasets, UOR and UOT are rendered by Unity to train and evaluate the proposed method. In experiments, the method is compared with a supervised method PointGroup, where PointGroup outperforms SPAIR3D, but the performances are comparable. \nThe unsupervised method and the new loss are good and novel, but I have a few confusions below:\n1. Firstly the problem setting is a little confusing, since the framework is unsupervised, I thought that the training data would be captured point clouds or those from other various datasets, since GT is not needed. However, the authors still make a lot of efforts to render UOR and UOT. If so, the GT is easy to get, why do we need an unsupervised framework for these synthetic datasets? On the other hand, the synthesized datasets look relatively simple and objects are placed sparsely and at similar sizes. I would like to see results on more challenging data, and I would like to hear the reason why the training is not performed on real point clouds or existing datasets. \n2. In a real scene, objects are often stacked. Would SPAIR3D work and how is it work? \n3. Typos: title of 3.1 \"Local\". \n", "summary_of_the_paper": "This paper introduces a framework, SPAIR3D, to decompose a 3D point cloud into several objects (point cloud segmentation). The framework is unsupervised, driven by a Chamfer Mixture Loss. Two training datasets, UOR and UOT are rendered by Unity to train and evaluate the proposed method. In experiments, the method is compared with a supervised method PointGroup, where PointGroup outperforms SPAIR3D, but the performances are comparable. ", "summary_of_the_review": "The idea of unsupervised point cloud segmentation is cool since point clouds are hard to annotate, and they are easy to capture or collect. However, the dataset and evaluations are not quite reasonable to me, as described above. Thus I give a borderline reject for now. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "UUp6Ocogvnn", "reviewer": "Reviewer_yyA6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrengths\n[novelty]  \n1. According to my knowledge, this is the first work that explores variational generative methods for scene-level 3D point cloud understanding; the learnt representation could be used for unsupervised 3D  instance segmentation, or scene decomposition, while existing methods like PointGroup[Jiang et al.] or 3D-BoNet[Yang et al.] requires ground truth labels to cluster neighborhood points or refine from the detected bounding boxes; \n\n2. The proposed mixture chamfer loss is novel, which treats each point as a Gaussian distribution and turns the original Chamfer distance better for variational inference. It also utilizes a weighting mechanism when deciding each point’s corresponding glimpse, this formulation is critical for closed loop mask prediction;\n\n[effectiveness] \n1. The detection-inspired local voxel attention mechanism makes it possible to handle arbitrary number of objects in the 3D scene, handling well with 2-12 objects from their experiments;\n\n2. The proposed method achieves performance that is on a par with supervised SOTA method on two synthetic scene point cloud datasets, including challenging cases like almost touch surfaces;\n\n[completeness]\nThe author provides both quantitative and qualitative results, with ablation studies on reconstruction point number, voxel size, etc. They also mentioned its drawbacks like handling large objects that are much bigger than the voxel size, mixed objects inside one voxel. \n\n\nWeakness\nThough this work is sound overall, after going through the details carefully in the paper, here I’d like to point out several concerns/issues below. I have examined the key claims over the technical contributions, together with the experimental settings.\n\n1. The chamfer mixture distance loss seems to be critical for the variational model training,  but the additional per-point probability modeling and bidirectional closest point finding, plus multiple glimpse proposals seem to make the loss computation very expensive, does the author have rough estimation of the increased computation compared to the original Chamfer distance loss? One quick related question, is it possible for the reconstruction to come in the form of regular voxels? In this way, regular one-to-one correspondence could be established naturally, without the expensive argmax operation, or searching for the closest pairs;\n\n2. Current experiments are performed on synthetic complete scene point clouds, which is still quite a toy setting. On these datasets, there is strong bias that a flat and clean desk/room background is presented, would it be possible that the model simply learns to discard the flat plane when segmenting the object instead of learning good object-centric representations? A simple way to verify that is trying to tilt the point cloud data, so the desk plane would not be flat. Also I guess the proposed method might be limited when handling partial 3D point clouds, especially for the point cloud reconstruction supervision, is that expected?\n\n3. Following the previous question, is it possible for the proposed method to test on S3DIS dataset as well? This is a common benchmark for 3D instance segmentation and doesn’t have quite challenging partial data. Even though there might be challenging cases due to various object scales, the author should be able to find a reasonable voxel size for glimpse generation. \n\n4. The author is expected to provide more training details, like each glimpse may contain a different number of points, in the glimpse encoder and decoder, are different glimpses processed in batch or sequentially? Is there any difference for loss weights when training global VAE and per-glimpse VAE?\n\n5. The author needs to be aware of some recent works in this direction, like this ICCV 2021 paper: Unsupervised Point Cloud Object Co-Segmentation by Co-Contrastive Learning and Mutual Attention Sampling[Yang et al.]\n\n6. Could the author leverage more intuition about why this method could learn to develop a concept of visual entities in an unsupervised way? It is still amazing the proposed pipeline achieves such a good performance even on the toy datasets. But if the per-glimpse reconstruction generates points containing background points, the loss would not change, but then there is no way that  model could tell the object mask. And why do we need this global VAE for background when each glimpse module already handles the target  foreground objects?\n\nMinor comments about weakness:\nWhat is the foreground alpha in figure 2 (i) and (j). Besides the glimpse alpha and scene layout reconstruction, Could the author also show more reconstruction visualization in individual glimpses?\nIn the title, there is a word ‘Invariant', which I found quite interesting. What does this ‘invariant’ mean exactly?\nPage 2. Section 3, 2nd paragraph should be ‘However, point cloud data is ...’\n\n", "summary_of_the_paper": "This paper is a pioneering work that extends the success of unsupervised object-centric learning in 2D images to 3D scene point clouds. The framework could detect and segment multiple 3D objects from a 3D scene point cloud through a decompose-by-reconstruct strategy in an unsupervised manner. \n\nTheir variational training pipeline utilizes a VAE-based generative model that jointly considers the global 3D background and local individual objects, and a novel chamfer mixture loss is proposed for irregular 3D point clouds.  The method is validated on two synthetic datasets, and the performance is demonstrated to be close to a supervised SOTA method.\n", "summary_of_the_review": "This paper is a valuable pioneering work on unsupervised 3D scene segmentation, with a full variational training pipeline built up for irregular 3D point cloud data. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8t8MNYYEM", "reviewer": "Reviewer_PtZ1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n#### S1 - Important problem\n- Unsupervised 3D object learning is an important important problem. This paper studies a specific case of it, which is detecting 3D objects in point clouds.\n\n#### S2 - Carefully designed framework\n- The proposed method is carefully designed for handling the irregular structure of point clouds in a VAE framework. Specifically, it adopts a likelihood-based extension of Chamfer distance as a reconstruction loss for point clouds, and incorporates several recent architectures, including pointCNN and pointGNN.\n\n### Weaknesses\n#### W1 - Over-complicated method on simplistic datasets\n- A major concern with this paper is the simplicity of the evaluation experiments. The datasets used in this paper appear extremely simple, with almost perfect shapes, uncluttered scenes and clean, planar background, which makes task of foreground object detection seem rather trivial, and hence the learning of 3D object representations. Moreover, compared to the previous studies in 2D images, the 3D scenes here intrinsically alleviate challenges like occlusion and illumination effects. In contrast to this seemingly trivial task, the proposed method appears overly complicated.\n- Also, the failure examples shown in appendix (Fig. 15 and Fig. 16) suggests the model is still unable to handle scenes where objects are closer.\n- I would suggest increasing the diversity of the 3D scenes to verify the effectiveness of the proposed method, eg, using more complicated geometries for the background, varying the sizes and point densities of the objects, introducing noise to the point clouds etc.\n", "summary_of_the_paper": "This paper studies the problem of unsupervised object-centric learning in the context of point clouds. Specifically, the goal is to learn a generative model of 3D objects from a collection of untextured point clouds of multi-object scenes, and this is achieved via a VAE-based generative framework.\n\nThis task extends the setup of prior work on image-based 2D object-centric learning to 3D. To be able to handle irregular structures of point clouds, a novel Chamfer Mixture Loss is proposed as reconstruction loss, which essentially extends the Chamfer distance with probability mixture modeling. The resulting model seems to work well on simple scenes with small isolated objects on a plane.", "summary_of_the_review": "Overall, I think the problem is interesting, and the proposed framework is technical sound, but the results presented on the simplistic datasets cannot sufficiently verify the effectiveness of the method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8JTvpyUnIF", "reviewer": "Reviewer_orM7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths\n- The overall writing is clear.\n- The paper is the first to extend 2D object-centric representation learning to 3D point clouds. The Chamfer Mixture Loss is a reasonable adaptation for variational training.\n\n# Weaknesses/Questions\n1. Lack of baselines. Although there are no direct 3D object-centric representation learning baselines, 3D object discovery is not an unexplored field. Apart from \"Unsupervised Discovery of Repetitive Objects\", a common baseline in 3D object detection is to cluster/group 3D points according to their spatial relationship. Concretely, the authors can compare with a baseline, where the background/scene layout (planes in the paper's datasets) is first detected by a RANSAC and the rest objects are separated by DBSCAN/MeanShift (hyperparameters need to be tuned/searched). Although such a baseline is not scalable to more complicated scenes, SPAIR3D is not shown to have stronger scalability currently. Besides, the authors can introduce some 3D over-segementation baselines (e.g., [1], [2])\n\n2. How important is VAE in the object-centric representation learning framework? It might be a general question (also for 2D counterparts). From some experience, I suspect that the prior of model structure, like sliding window in 2D/3D grid and relative center prediction (techniques studied in 2D/3D detection), is the most important. VAE, especially the probabilistic model, mainly provides a sampling procedure during training, which might be considered as a search process (Monto-Carlo sampling). An easy way to check is to remove KL divergence terms between the prior and posterior probability of latent codes, and only maximize posterior probabilities of observations.\n\n3. I am a little confused about the claim \"Unfortunately, the Chamfer distance does not fit into the variational-inference framework\". There is no detailed explanation about it. From my understanding, the Chamfer Mixture Loss just introduces additional variance term and changes the distance to probability, since the variance term is a hyperparameter rather than per-point prediction and the likelihood is only based on the best/maximum matched point.\n\n4. As mentioned in Sec 4.1, the point cloud is fused from 10 different views. Can SPAIR3D work on partial 3D scenes? Theoretically, current SPAIR3D may have difficulty since the Chamfer Mixture Loss ignores visibility and it becomes much harder to recognize repetitive objects.\n\n# References\n1. Toward better boundary preserved supervoxel segmentation for 3D point clouds\n2. Superpoint Network for Point Cloud Oversegmentation", "summary_of_the_paper": "The paper tackles unsupervised 3D scene decomposition/object discovery from point clouds. It proposes SPAIR3D, inspired by the 2D counterpart SPAIR, to factorize a 3D point cloud into a spatial object-centric mixture model. It derives the Chamfer Mixture Loss to fit 3D reconstruction into the variational training pipeline. It introduces two customized point-cloud datasets, and compares with a supervised baseline PointGroup.", "summary_of_the_review": "To the best of my knowledge, the paper is the first to tackle 3D object-centric scene decomposition (unsupervised object discovery). It is well written and reasonably extends SPAIR to 3D scenes with new challenges only in 3D cases. However, the baselines are not carefully chosen, and the scalability of the proposed SPAIR3D is questionable. Overall, I think the paper is slightly above the acceptance threshold.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5AzUyD7H7S", "reviewer": "Reviewer_oa19", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths：\n1. This paper proposes a novel Chamfer Mixture loss formulated in a probabilistic framework. \n2. The proposed method achieves astonishing performance comparable to a supervised baseline on two simple datasets.\n\nWeakness：\n1.\tThis paper is built on the SPAIR framework and focuses on point cloud data, which is somehow incremental.\n2.\tThere is no ablation study to validate the effectiveness of the proposed components and the loss. \n3.\tIt is hard to follow Sec. 3.2. The author may improve it and give more illustrations and examples.\n4.\tIt is unclear how the method can work and decompose a scene into different objects. I did not see how Chamfer Mixture loss can achieve this goal. More explanation should go here. \n", "summary_of_the_paper": "This paper proposes a VAE-based unsupervised generative model, to achieve 3D object-centric learning as well as 3D scene decomposition. The method divides a 3D scene into voxel grids and generates latent features to reconstruct an original point cloud. A chamfer Mixture loss is proposed to train the model. ", "summary_of_the_review": "The idea to decompose a scene in an unsupervised way is novel. However, the proposed method is mainly based on SPAIR and the only technical contribution seems to be Chamfer Mixture loss. It is unclear what is the advantage of this loss design over traditional chamfer distance and how it can help scene decomposition. Moreover, I did not see further analysis on different proposed components of the method and inspiring conclusion for the community. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a VAE-based generative model of 3D point-clouds inspired by SPAIR that can do unsupervised segmentation, named SPAIR3D. The model uses both global and local latent variables to encode global scene structure as well as individual objects.

The proposed model is relatively complex, but the presentation is overall clear. 

Experimental results on simple synthetic datasets look promising. However, one might argue that for these simple tasks a direct application of a simpler mixture of VAEs (such as IODINE) might be sufficient, so it would be informative to make a direct comparison between these methods and/or show results on a problem clearly out of the scope of these simpler methods (e.g. with high imbalance in the point clouds).
Decision: Reject