Semi-Supervised Learning of Multi-Object 3D Scene Representations
Authors: Cathrin Elich
Martin R. Oswald
Marc Pollefeys
Joerg Stueckler
Venue:  (2021)
Keywords: scene understanding
representation learning
multi-object scene decomposition
pose estimation
shape and appearance estimation

Abstract: Representing scenes at the granularity of objects is a prerequisite for scene understanding and decision making. We propose a novel approach for learning multi-object 3D scene representations from images. A recurrent encoder regresses a latent representation of 3D shapes, poses and texture of each object from an input RGB image. The 3D shapes are represented continuously in function-space as signed distance functions (SDF) which we efficiently pre-train from example shapes. By differentiable rendering, we train our model to decompose scenes self-supervised from RGB-D images. Our approach learns to decompose images into the constituent objects of the scene and to infer their shape, pose and texture properties from a single view. In experiments, we evaluate the accuracy of our model in inferring the 3D scene layout and demonstrate the capabilities of the generative 3D scene model.

Under review as a conference paper at ICLR 2021
SEMI-SUPERVISES LEARNING OF MULTI-OBJECT 3D SCENE REPRESENTATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Representing scenes at the granularity of objects is prerequisite for scene under- standing and decision making. We propose a novel approach for learning multi- object 3D scene representations from images. A recurrent encoder regresses a latent representation of 3D shapes, poses and texture of each object from an in- put RGB image. The 3D shapes are represented continuously in function-space as signed distance functions (SDF) which we efficiently pre-train from example shapes in supervised way. By differentiable rendering then train model to decompose scenes self-suuprvised from RGB-D images. Our approach learns decompose images into the constituent objects of the scene and to infer their shape, pose and texture from a single view. We evaluate the accuracy of our model in inferring the 3D scene layout and demonstrate its generative capabilities.
INTRODUCTION
Humans have the remarkable capability to decompose scenes into its constituent objects and to infer object properties such as 3D shape and texture from just single view. Providing intelligent systems with similar capabilities is long-standing goal in artificial intelligence. Such representations would facilitate object-level description, abstract reasoning and high-level decision making. Moreover, object-level scene representations could improve generalization for learning in downstream tasks such as robust object recognition or action planning.
Previous work on learning-basee based scene representations focused on single-object scenes (Sitzmann et al. 2019) or neglected to model the 3D geometry of the scene and the objects explicitly (Burgess al. 2019 Greff et al. 2019 Eslami et al. 2016). In our work, we propose a multi-object scene representation network which learns to decompose scenes into objects and represents the 3D shape and texture of the objects explicitly. Shape, pose and texture are embedded in a latent representation which our model decodes into textured 3D geometry using differentiable rendering. This allows for training our scene representation network semi-supervised way. Our approach jointly learns the tasks of object detection, instance segmentation, object pose estimation and inference of 3D shape and texture in single RGB images. Inspired by (Park et al.] 2019 Occhsle et al.) 2019: Sitzmann represent 3D object shape and texture continuously in function-space
al., 2019), distance and color values at continuous 3D locations. The scene representation network infers the object poses and its shape and texture encodings from the input RGB image. We propose novel differentiable renderer which efficiently generates color and depth images as well as instance masks from the object-wise scene representation. By this, our model facilitates to generate new scenes by altering an interpretable latent representation (see Fig. Tb. Our network is trained in two stages: In first stage, we train an nuto-decoder subnetwork of our full pipeline embed collection of meshes in continuous SDF shape embeddings as in DeepSDF (Park et al. 2019). With this pre-trained shape space, we train the remaining parts of our full multi-object network to decompose and describe the scene by multiple objects in self-supervised way from RGB-D images. No ground truth of object pose, shape, texture, or instance segmentation is required for the training on multi-object scenes. We denote our learning approach semi-supervised due to the supervised pre-training of the shape embedding and the self-supervised learning of the scene decomposition.
signed
We evaluate approach synthetic scene datasets with images composed of multiple objects to show its capabilities with shapes such as geometric primitives and vehicles and demonstrate the properties of our geometric and semi-supervised learning approach for scene representation. In sum-

---

Under review as a conference paper at ICLR 2021
Switch Position
Sample Position
Remove Object
Shage
Change
Tnput
Reconctructon
Teput
Recunstruction
Tnput
Recomstmction
Input
Reconsimuciin
Inpul
Revorstruction
ShapeNer Clour
Figure I: Example scenes with object manipulation. For each example, we input the left images and compute the middle one standard reconstruction. After the manipulation in the latent space, we obtain the respective right image. Plausible new scene configurations are shown on the Clevr dataset (Johnson al. 2017) (top) and on composed ShapeNet models (Chang et al. 2015 (bottom).
mary, we make the following contributions: (1) We propose a novel model learn representations of scenes composed of multiple objects. Our model describes the scene by explicitly encoding ob- ject poses, 3D shapes and texture. To the best of our knowledge, our approach is the first to jointly learn the tasks of object instance detection, instance segmentation, object localization, and inference of 3D shape and texture in a single RGB image through self-supervised scene decomposition. (2) Our model is trained by using differentiable rendering for decoding the latent representation into images. For this, we propose a novel differentiable renderer using sampling- based raycasting for deep SDF shape embeddings which renders color and depth images as well instance segmenta- tion masks. (3) By representing 3D geometry explicitly, our approach naturally respects occlusions and collisions between objects and facilitates manipulation of the scene within the latent space. We demonstrate properties of our geometric model for scene representation and augmentation, and dis- cuss advantages over multi-object scene representation methods which model geometry implicitly. We plan to make source code and datasets of our approach publicly available upon paper acceptance.
2 RELATED WORK
Deep learning of single object geometry. Several recent 3D learning approaches represent single object geometry by implicit surfaces of occupancy or signed distance functions which are discretized
in 3D voxel grids (Kar et al. 2017: Tulsiani et al. 2017 Wu et al.) 2016 Gadelha et al., 2017 Qi et al. 2016 Jimenez Rezende et al. [216]] [Choy et al. 2016 Shin et al. 2018; Xie et al. 2019). Voxel grid representations typically waste significant memory and computation resources in scene parts which are far away from the surface. This limits their resolution and capabilities to represent fine details. Other methods represent shapes with point clouds (Qi al., 2017 Achlioptas et al., 2018), meshes (Groueix et al.) 2018), deformations of shape primitives (Henderson & Ferrari 2019) or multiple views (Tatarchenko et al.) 2016). In continuous function-space representations, deep neural networks are trained to directly predict signed distance (Park et al. 2019 Xu et al. 2019
Sitzmann et al. 2019). occupancy (Mescheder et al. 2019 Chen & Zhang 2019). or texture Oech- sle et al. 2019) at continuous query points. We use such representations for individual objects.
Deep learning of multi-object scene representations. Self-supervised learning of multi-object scene representations from images recently gained significant attention in the machine learning com- munity. MONet (Burgess et al. 2019) presents a multi-object network which decomposes the scene using a recurrent attention network and an object-wise autoencoder. It embeds images into object- wise latent representations and overlays them into images with a neural decoder. Yang et al. (2020) improve upon this work. Greff et al. (219) use iterative variational inference to optimize object- wise latent representations using a recurrent neural network. SPAIR (Crawford & Pineau 2019)
and SPACE (Lin al.) 2020) extend the attend- -ifeed------reeeatt approach (Eslami et al. 2016) by lay- ing grid over the image and estimating the presence, relative position, and latent representation of objects in each cell. In GENESIS (Engeleke et al., 2020), the image is recurrently encoded into
latent codes per object in variational framework. In contrast to method, the above methods do not represent the 3D geometry of the scene explicitly. Recently, Liao et al. (2020) introduced 3D controllable image synthesis to generate novel scenes instead of explaining input views like we do.
Supervised learning for object instance segmentation, pose and shape estimation. Loosely related our approach are supervised deep learning methods that segment object instances (Hou) ct al. 2019 Prabhudesai et al. 22220). estimate their poses (Xiang et al.) (2017) or recover their 3D
shape (Gkioxari al. 2019: Kniaz al. 2020). In Mesh R-CNN (Gkioxari et al. 2019). objects
2

---

Under review as a conference paper at ICLR 2021
9bg
F Flzgg 21....
bb
i DI.N MI.N
I
A 1:0
9o
Aha-1
M1.0
21
7
M
2;
F(z1)
Flzi,...,
F FZz1..... 2i)
encoder
latent representation
scene rendering
Figure 2: Multi-object 3D scene representation network. The image sequentially encoded into object representations using encoder network go- The object encoders additionally receive image and mask compositions (AT, M) generated from the previous object encodings. A differentiable renderer based decoder F composes images and masks from the encodings of previous steps. The background is encoded from the image in parallel and used in the final scene reconstruction.
object eneoder
objoct-wisc render
A
(zz, sh:
raycast
O
M
(z,
ein
te
colorize
T
I
M1-1-1
Z;
Figure 3: Object-wise encoding and rendering. We feed the input image and scene composi- tion images and masks from the previously found objects an object encoder network go which regresses the encoding of the next object Z;. The object encoding decomposes into shape Zi,sh, ex- trinsics ext and texture latents Zi,er- The shape latent parametrizes an SDF function network + which we use in combination with the pose and scale of the object encoded in Zi,ent for raycasting object depth and mask using our differentiable renderer f. Finally, the color of pixels is found with texture function network A parametrized by the texture latent.
are detected in bounding boxes and a 3D mesh predicted for each object. The method trained supervised images with annotated object shape ground truth.
Neural and differentiable rendering. Eslami et al. (2018) encode images into latent representa- tions which can be aggregated from multiple view points. Scene rendering is deferred to neural network which needs to be trained to decode the latents into images from examples. Several dif- ferentiable rendering approaches have been proposed using voxel occupancy grids (Tulsiani al.
2017
Gadelha et al.) 2017 Jimenez Rezende et al.) 2016]] Yan et al.) 20166] Gwak et al. 2017 Zhu
al.
2018|
al. [2017] Nguyen-Phuoe al. 2018)), meshes (Kato al. 20188 Loper & Black
2014
Chen et al. [019]] Delaunoy & Prados. 2011 Ramamoorthi & Hanrahan [200 Meka et al.
2018
Athalye et al. [2018] Richardson al. 2016 Liu al. 2019 Henderson & Ferrari 2019)
signed distance functions (Sitzmann et al.) 2019). or point clouds (Lin et al. [2018] Yifan et al.)
2019).
Recent literature overviews can found in (Tewari et al. 2020 Kato al. (020). our
approach, we find depth and mask values through equidistant sampling along the ray.
METHOD
We propose autoencoder architecture which embeds images into object-wise scene representa- tions (see Fig. 2 for an overview). Each object is explicitly described by its 3D pose and latent embeddings for both its shape and textural appearance. Given the object-wise scene description, decoder composes the images back from the latent representation through differentiable rendering. We train autoencoder-like network in self-supervised way from RGB-D images.

---

Under review as a conference paper at ICLR 2021
Scene Encoding. The network infers latent z (Z1..... ................................ 2N, Zby) which decomposes the scene into object latents z, E R., PE E {L..... N and a background component Zby E Rd where d, dog are the dimensionality of the object and background encodings and N is the number of objects.
Object are sequentially encoded by a deep neural network z; = go Ahs-1, M1a-1) (see Fig. ( We share the same object encoder network and weights between all objects. To guide the encoder to regress the latent representation of one object after the other, we for- ward additional information about already reconstructed objects. Specifically, we decode the previous object latents into object composition images, depth images and occlusion masks D1s-1, Mi.--) == F F(Zbg: 21,-... Z(-1). They generated by F using differentiable rendering which we detail in the subsequent paragraph. We concatenate the input image / with the difference image A/ ;= / T and occlusion masks M and input this to the encoder for inferring the representation of object i.
The object encoding = (x 2 1eze ?T decomposes into encodings for shape Zi,sh, tex- tural appearance Zi,tes. and 3D extrinsics Ziee (see Fig. 3 The shape encoding Zi,sh E R Pa parametrizes the 3D shape represented by a DeepSDF autodecoder (Park et al. 2019). Similarly, the texture is encoded in latent vector Zi,tes E Die which is used by the decoder to obtain color values for each pixel that observes the object. Object position Pi = (Ii, Vi, 2:) orientation 0; and scale are regressed with the extrinsics encoding ei, = (P, Acos,i; Zain,i, 8:) The object
S,R, R.P. is parametrized in world coordinate frame with known transformation T from the camera frame.
pose To (Zt,at))
We assume the objects are placed upright and model rotations around the vertical axis with an- gle 0g = arrctan(ssin.t Zcos.s) and corresponding rotation matrix R, We use a two parameter repre- sentation for the angle as suggested in (Zhou et al. 2019). We scale the object shape by the factor 8, E [smin |max] which we limit an appropriate range using a sigmoid squashing function. The background encoder 9bg := Zbg Rd regresses the uniform color of the background plane, i.. dbg- = 3. We assume the plane extrinsics and hence its depth image is known in our experiments.
Scene Decoding. Given our object-wise scene representation, we use differentiable rendering to generate individual images of objects based on their geometry and appearance and compose them into scene images. An object-wise renderer (Ie, D., M,) == f(z.) determines color im- age Ir depth image δ; and occlusion mask Mj from each object encoding independently (see Fig. 3h. The renderer determines the depth at each pixel u E R² (in normalized image coordi- nates) through raycasting in the SDF shape representation. Inspired by (Wang al., 2020), we trace the SDF zero-crossing along the ray by sampling points x, := [d, dz) in equal inter- vals dj := do + j E {0, .Q....,N: N - 1} with start depth dg- The points are transformed to the object coordinate system by (Zi,et) == Te (Zi, T Subsequently, the signed distance o, to the shape at these transformed points obtained by evaluating the SDF function network & (hi (Z )x,) Note that the SDF network is also parametrized by the inferred shape latent of the object. The algorithm finds the zero-crossing at the first pair of samples with a sign change of the SDF $ The uub-discretization accurate location x(u) of the surface is found through linear interpolation of the depth regarding the corresponding SDF values of these points. The depth at a pixel (u) is given by the z coordinate of the raycasted point x(u) on the object surface in camera coordinates. If no zero crossing is found, the depth is set to a large constant. The binary occlusion mask M, is set to if zero-crossing is found the pixel and O otherwise. The pixel color (u) is determined using a decoder network & which receives the texture latent Zi,tez of the object and the raycasted 3D point x(u) in object coordinates as inputs, i.e. L.(u) We speed up the raycasting process by only considering pixels that lie within the projected 3D bounding box of the object shape representation. This bounding box is known since the SDF func- tion network is trained with meshes that are normalized to fit into unit cube with constant padding. Note that this rendering procedure can be implemented using differentiable operations which makes it fully differentiable for the shape, color and extrinsics encodings of the object.
(Zi,teay (z ext X u)).
The scene images, depth images and occlusion masks (7 Di,, MI... = F( Zi.... Zn) are composed from the individual objects 1, ,,..., n with n < N and the decoded background through z- buffering. We initialize them with the background color, depth image of the empty plane and empty mask. Recall that the background color is regressed by the encoder network. For each pixel u, we
4

---

Under review as a conference paper at ICLR 2021
search the occluding object i with the smallest depth at the pixel. If such an object exists, set the pixel's values in I1,N, DI.N, MI,N the corresponding values in the object images and masks.
Training. We train our network architecture in two stages. In a first stage, we learn the SDF function network from collection of meshes. The second stage uses the pre-trained SDF models to learn the remaining components for the object-wise scene decomposition and rendering network. We train the SDF networks according to (Park et al. 2019) from a collection of meshes and sample points in volume around the object and on the object surface. We normalize the size of the input meshes to fit into the unit cube with constant padding e 0.1.
Our multi-object network architecture is trained supe-sup from RGB-D images containing example scenes composed of multiple objects. To this end, we minimize the loss function
Litell = ر + ADLD + > Lgr + Lites which is a weighted sum of multiple sub-loss functions defined by
(I)
Lr
Σ G (TwN) (u) - (() 2 LD = Σ uER
1
a
(DiN)) (u) (v(
[
UER
L max(0, -2+) + -o.
Lih =
Σ ||?
(2)
In particular, L, is the mean squared error on the image reconstruction with Q being the set of image pixels and Tgz the ground-truth color image. The depth reconstruction loss LD penalizes deviations from the ground-truth depth D We apply Gaussian smoothing G(-) for which we decrease the standard deviation over time. Lah regularizes the shape encoding stay within the training regime of the SDF network. Lastly, grr favors objects to reside above the ground plane with 2; being the coordinate of the object in the world frame, 2/ the corresponding projection onto the ground plane, and ox == 4 (Zi.ah: (mn( The shape regularization loss is scheduled with time-dependent weighting. This prevents the network from learning to generate unreasonable extrapolated shapes in the initial phases of the training. but lets the network refine them over time.
We use a CNN for both the object and the background encoder. Both consist of a number of convo- lutional layers with kernel size (3. 3) and strides (1, 1) each followed by ReLU activation and (2, 2) max-pooling. The subsequent fully connected layers yield the encodings for objects and back- ground. Similar to (Park et al.) p.19, shape decoder T and texture decoder L. Further details are provided in the supplementary material.
multi-layer fully-comnected neural networks for the
4 EXPERIMENTS
We evaluate our approach on synthetic scenes based on the Clevr dataset (Johnson et al., 2017) and scenes generated with ShapeNet models (Chang et al. (015). The Clevr-based scenes contain images with a varying number of colored shape primitives (spheres, cylinders, cubes) on planar single-colored background. We modify the data generation of Clevr in number of aspects: (1) We remove shadows and additional light sources and only use the Lambertian rubber material for the objects' surfaces. (2) To further increase shape variety, apply random scaling along the principal axes of the primitives. (3) An object might be completely hidden behind another one. Hence, the network needs to learn to hide single objects. We generate several multi-object datasets. Each dataset contains scenes with a specific number of objects which we choose from two to five. Each dataset consists of 12.5K images with size of 64x64 pixels. Objects are randomly rotated and placed in a range of [-1.5, 1.5)² on the ground plane while ensuring that any two objects do not intersect. Additionally to the RGB images, we also generate depth maps for training as well as instance masks for evaluation. The images are split into 9K training, 1K validation, and 2.5K testing examples. For the pre-training of the DeepSDF (Park et al. 2019) network, we generate a small set of nine shapes per category with different scaling along the axes for which we generate ground truth SDF samples. Different to (Park et al. 2019), we sample a higher ratio of points randomly in the unit cube instead of close to the surface. We also evaluate on scenes depicting either cars or armchairs well as mixed set consisting of mugs, bottles and cans (tabletop) from the ShapeNet model set. Specifically, select 25 models per setting which use both for pre- mre-training the DeepSDF well as for the generation of the multi-object datasets. We increase the size of the dataset to (18K (SS//KK/50 (5)). The evaluation is performed on two different test sets: (1) with known shapes and (2) with new objects.
5

---

Under review as a conference paper at ICLR 2021
Inpur
Prodicted
Regresentation Objectt
Depti
Prediction
Objjctt
Object
Objjctt
Objjctt
All Instamens
Prediction
Prediction
Ohbeets Ofjects
Figure 4: Qualitative results on the Clevr dataset (Johnson et al., 2017) with three and five objects. Our object-wise scene representation decouples all objects from the background.
Network Parameters. For the Clevr ShapeNet datasets, the object encoding dimension set to Dah 8/16, and Diao = 7/15. The shape decoder is pre-trained for 10K epochs. We decrease during the first 500K itcrations. The remaining weights fixed A 1.0, Adepth 0.1/0.05, Agr 0.01. We add Gaussian noise to the input RGB images. Depth images are clipped at a distance of 12. The renderer evaluates at 12 steps along each ray. Gaussian smoothing is applied with kernel size 16 and decreasing sigma from 16 to 2 in 250K steps. We use the ADAM optimizer (Kingma & Ba 2014) with learning rate 0.0001 and batch size 8 to train for dataset-specific number of epochs (see supplementary material for more details).
the loss weight Ash from 0.025/0. /0.1 to 0.0025 0.0025/0.01 /0,01
1
Evaluations Metrics. We evaluate the task of learning object-level 3D scene representations using measures for instance segmentation, image reconstruction, and pose estimation. To evaluate the capability of our model to recognize objects that best explain the input image, we consider established instance segmentation metries. An object is considered to be correctly segmented if the intersection- pnnrrcctioonovv-uuiiooo over-union (IoU) score between ground truth and predicted mask is higher than some threshold 7. To account for occlusions, only objects that occupy at least 25 pixels are taken into account. We report average precision (APo.s). average recall (ARg.s), Fla.5- for fixed r = 0.5 as well as the mean AP over thresholds range [0.5, 0.95) with stepsize 0.05 (Everingham et al. 2010). Furthermore, we list the ratio of scenes were all visible objects were found w.r.t. (allObj). Next, we evaluate the quality of both the RGB and depth reconstruction obtained from the generated objects. To assess the image reconstruction, we report Root Mean Squared Error (RMSE), Structural SIMilarity Index (SSIM) and Peak Signal-to-Noise Ratio (PSNR) scores. For the object geometry, compute similar to (Eigen et al.] 2014) the Absolute Relative Difference (AbsRD), Squared Relative Difference (SqRD), as well as the RMSE for the predicted depth. Furthermore, report the error the estimated objects' position (mean) and rotation (median, sym.: up to symmetries) for objects with a valid match w.r.t. r = 0.5. More details on the metrics are provided in the supplementary material. We show results over five runs per configuration and report the mean.
0.5
4.1 CLEVR DATASET
In Fig. we show reconstructed images, depth and normal maps on the Clevr (Johnson et al.) 2017) scenes. Our model provides a complete reconstruction of the individual objects although they might be partially hidden in the image. The network can infer the color of the objects correctly and gets basic idea about shading (c.g. that spheres are darker on the lower half) and coarse texture. The shape characteristics such extent, edges curved surfaces all object slots. We sometimes observed that it fantasizes and hides additional objects behind others. Some reconstruction artifacts at object boundaries are due rendering hard transitions between objects and background. More results and typical failure cases are shown in the supplementary material. Our 3D scene model naturally facilitates generation and manipulation of scenes by altering the latent representation. In Fig. [ we show example operations like switching the positions of two objects, changing their shape, or removing an entire object. The explicit knowledge about 3D shape also allows us to reason about object penetrations when generating new scenes. Specifically, we evaluate an object intersection loss Tins on the newly sampled scenes filter out those that turn out to be unrealistic due an intersection between objects (sec supplementary material for details).
well recognized. Our model needs fill
Ablation Study. We evaluate various components of our model on the Clevr dataset with three objects. In Table we evaluate training settings where we left out each of the loss functions and also demonstrate the benefit of Gaussian smoothing (denoted by G) on the image reconstructions. At the beginning of training, the shape regularization loss is crucial to keep the shape encoder close to the pretrained DeepSDF shape space and to prevent it from diverging due to the inaccurate pose estimates of the objects. Applying and decaying Gaussian blur distributes gradient information in the

---

Under review as a conference paper at ICLR 2021
Table 1: Results on Clevr dataset (Johnson et al. 2017]. The combination of our proposed loss with Gaussian blur is essential to guide the learning of scene decomposition and object-wise repre- sentations. We highlight best (bold) and second best (underlined) result for each measure. Using different maximum numbers of objects in our network, we further train our model on scenes with 2, 4, or 5 objects. Despite the increased difficulty for larger number of objects, our model recognizes most objects in scenes with two to five objects. Models trained with fewer objects can successfully explain scenes with a larger number of objects (# obj=0main/
Instance Reconstraction
Image Reconstruction
Depth Reconstruction
Pose Fist.
mAP1 1 APos
ARas 1 Flas 1 allObi
RMSE
1 PSNR
SSIM
RMSE 4. AbsRD SyRD
Empos
obj=3/3, we Lr
0.686
0,941
0.879
0,899
0.709
0.199
14.176
0.713
0.595
0.023
0.073
0.159
obj=3/3, wro Lp
0.023
D.086
0.076
0.078
0.008
0.085
22.142
0.837
2.745
0.231
1.061
1.341
o33-3/3, we Lub
0.01
0.032
0.027
0.028
0.001
0.13
17.907
0.763
1,455
0.147
0.556
0.676
obj=3/3, w/o Lge
0.09
0.195
0.205
0.198
0.008
0.09
21.163
0.799
1.139
0.087
0.32
0.81
obj=M3, w/o
0.164
0.296
0.161
0.199
0.001
0.114
19.065
0.792
1.331
0.112
0.441
0.182
obj=3/3. full
0.712
0,949
0,942
0.943
0.85
0.049
26.466
0.914
0.554
0.019
0.061
0.155
#obj-2/2
0.782
0.977
0.963
0.967
0.928
0.039
28.
0.941
0.432
0.012
0.04
0.138
#bbj=4/4
0.688
0,941
0.919
0.926
0.746
0.054
25.632
0.899
0.584
0.022
0.064
0,151
objes5/5
0.604
0.895
0.861
0.872
0.539
0.061
24.568
0.876
0.593
0.025
0.067
0.149
obj=3//2
0.756
0.974
0.969
0.942
0.041
22.011
0.937
0.452
0.013
0.044
d.14
obj=3/4
0.613
0.883
0.853
0.863
0.512
0.06
24.669
0.88
0.665
0.028
0.083
0.179
obj-3/5
0.478
0.775
0.71
0.735
0.212
0.072
23.093
0.841
0.69
0.033
0.086
0.201
images beyond the object masks and allows the model to be trained in coarse-to-fing manner. This helps the model to localize the various objects in the scene. Moreover, the depth loss is essential for learning the scene decomposition. Without this loss, the network can simply describe several objects using a single object with more complex texture. The usage of the ground loss prevents the model from fitting objects into the ground plane. The image reconstruction loss plays only a minor part for the scene decomposition task but is merely responsible for learning the texture of the objects. Using all proposed loss functions yields best results over all metrics. Remarkably, our model is able to find objects at high recall rates (0.942 AR at 50% IoU).
Object Count. We also report results when varying the maximum number of objects in our model in Tab. [ We train the models with the corresponding number of objects in the dataset. Obviously, it is average easier for our model to find and describe the objects in less crowded scenes, while it still performs with high accuracy for five objects.
Due to the sequential architecture of our model, it can even be extended for scenes with more objects than that it has been trained for. As we use a shared encoder for all objects, we can simply reset the number of encoding rollouts to the number of objects in the test data. Again, we assume the number of objects to be known. Although our model would be able to hide redundant objects behind already reconstructed ones without this explicit change, it could not reconstruct additional objects. In these experiments, it performs less well than the trained models for the respective object counts. The achieved average recall and allObj measures still indicate that the model is able to detect the objects at good rates. For instance, for # obj=3/5, we find all objects in about 21 % cases but overall 71% of the objects according to ARo.s. Extended quantitative evaluation as well as qualitative results can be viewed in the supplementary material.
4.2 SHAPENET DATASET
Our composed multi-object variant of ShapeNet (Chang et al. 2015) models is more difficult in shape and texture variation than Clevr (Johnson et al. 2017). For some object categories such as cups or armchairs, training can converge to local minima. We report mean and best results over five training runs in Tab. 2 where the best run is chosen according to Fl score the validation set. Evaluation is performed on two different testsets: scenes containing (I) object instances with shapes and textures used for training and (2) unseen object instances. We show several scene reconstructions in Fig. 5) Further qualitative results are provided in the supplementary material.
For the cars, our model yields consistent performance in all runs with comparable decomposition results to our Clevr experiments. However, we found that cars exhibit a pseudo- (ecuudo-10-deepee 180-degree shape symmetry which was difficult for our model to differentiate. Especially for small objects in the
7

---

Under review as a conference paper at ICLR 2021
Table 2: Evaluation on scenes with ShapeNet objects (Chang et al., 2015). Results for scenes
containing objects from different categories. We differentiate between scenes that consist of shapes that were seen during training and novel objects. We report mean and best outcome over five runs.
Instance Reconstruction
Inage Reconstruction
Depih Reconstruction
Pose Estimation
mAPt
APos
ARgs 1 Flass 1 aallOhj
RMSE.
J. PSNR 1 SSIM
RMSE J. LhhhRD J J.SRRD
Firpo
Fr [sym.] 0
seon
best
0.750
0.991
0,991
0.991
0.979
0.064
24.092
0.898
0.158
0.006
0.004
0.144
23.67"
[3.29"]
mean
0.738
0.990
0.990
0.990
0.064
23.979
0.160
0.006
22.09'
[3.07"]
best
0.639
0,980
0.980
0.980
0.955
0,077
22.442
0.843
0,210
0.010
0.008
0.183
24.24 [4.53%
mean
0.632
0.977
0.977
0.977
0.944
0.077
22.454
0.842
0.208
0.010
0.008
0.184
24.25
[4.41%]
Jeen
best
0.432
0.897
0,871
0.881
0.640
0.086
21.576
0,803
0.829
0.040
0.117
0,308
43.64"
[9.13%]
chairs
mean
0.638
0.640
20.137
1,021
0.196
55.12" [7.25")
best
0.377
0.852
0,821
0.833
0.534
0.092
20.994
0.778
0.890
0.052
0.137
0.395
58.79'
[10.66]]
mean
0.278
0.613
0.607
0.609
0.158
0.106
19.740
0.746
1,068
0.069
0.213
0.372
68.29
[9.28"]
tabletop
seen
best
0.628
0.936
0,870
0.895
0.659
0.057
25.242
0.908
0.786
0.026
0.132
0.182
89.14
mean
0.394
0.537
0.546
0.251
0.078
22.871
1.022
0.050
0.231
88.53"
unseer best
0.435
0.839
0,816
0.823
0.569
0.083
21.807
0.840
1.034
0.044
0.224
0.275
89.25
mean
0.285
0.521
0.523
0.102
20.160
1.172
0.061
89.99 89.99
Inpul View
Normals Prediction
Input Prediction
Normals Prodiction
Input View Prodiction
Normalss
Prediction
GT
GT
Prodiction
unreen
Figure 5: Qualitative results on ShapeNet (Chang et al., 2015). Our model obtains a good scene understanding if confronted with more difficult objects (cars, armchairs) and even handles objects from different categories (tabletop scenes with mugs, bottles and cans). It is able to estimate plausi- ble pose and shape of individual objects and learns to decode more complex textures.
background, it favors to adapt the texture over rotating the object. For the armchair shapes, our model finds local minima in pseudo-90-degree symmetries. The median rotation error indicates better than chance prediction for the correct orientation. Rotation error histograms can be found in the supplementary material. For approximately correct rotation predictions, we found that our model was able differentiate between basic shape types but often neglected finer details like thin armrests which are difficult to differentiate in the images. Our tabletop dataset provides another type of challenge: the network needs to distinguish different object categories with larger shape and scale variation. For this setting, we added further auxiliary losses to penalize object positions outside of the image view as well as object intersections (see supplementary material for details). Our model is able to predict the different shape types with coarse textures. On scenes with instances that were not seen during training, our model often approximates the shapes with similar training instances.
Due to the learned 3D structure, model is able render novel views from scene given single image (see Fig. (). Although our model never saw multiple views of the same scene during training and is not tuned for this task, obtain reasonable results for both scene geometry and appearance. We observe a lower reconstruction accuracy for invisible scene parts, especially for the texture.
Imput
.90
.2.5
U
+2250
+
+90
1800
Figure 6: Novel view renderings. Our model is able to generate new scene renderings for largely rotated camera views from just single input RGB image. While we noticed a reduced texture accuracy for unseen object parts compared to visible parts, the normal maps are generally good and demonstrate that our model obtains a good 3D structural understanding of the scene.
8

---

Under review as a conference paper at ICLR 2021
Figure 7: Evaluation on real images. We show preliminary results on real images by our model that was trained on synthetic data. We notice that our model able capture the coarse scene layout and shape properties of the objects. However, challenges arise due to domain, lighting, camera intrinsics and view point changes indicating interesting directions for future research.
Imput
Imput
Input
Prediction
Inpui
Input
Prediction
Figure 8: Limitations. Input and output pairs for typical failure cases and limitations of our method due to ambiguities for self-supervised learning. See text for details.
We further evaluated our model on real images of toy cars and building blocks (see Fig. for which we adjusted brightness and contrast to visually match the background color of the synthetic data. Note while the scene perspective, camera and image properties are different, our model able to decompose the scene in these examples into the individual objects and obtain a coarse understanding about their shape and appearance without any further fine-tuning on the new data domain.
Limitations. We show typical failure cases of our approach in Fig. 8 Self-supervised learning without regularizing assumptions leads typically to ill-conditioned problems. We use pre-trained 3D shape space to confine the possible shapes, impose multi-object decomposition of the scene, and use a differentiable renderer of the latent representation. In our self-supervised approach, am- biguities arise due to the decoupling of shape and texture. For instance, the network choose to occlude the background partially with the shape but fix the image reconstruction by predicting background color in these areas. Rotations can only be learned up pseudo-symmetry by self- supervision when object shapes are rotationally similar and the subtle differences in shape or texture difficult differentiate in the image. In such cases, the network can favor to adapt texture over rotating the shape. Depending on the complexity of the scenes and the complex combination of loss terms, training can run into local minima in which objects are moved outside the image or fit the ground plane. Currently, the network is trained for maximum number of objects. If all objects in the scene are explained, hides further objects which could be alleviated by learning stop criterion.
CONCLUSION
We propose a novel deep learning approach for multi-object scene representation learning and pars- ing. Our approach infers the 3D structure of scene in RGB images by recursively parsing the image for shape, texture and poses of the objects. A differentiable renderer allows images to generated from the latent scene representation and the network to be trained semi-supervised from RGB-D images. We represent object shapes by signed distance functions. To confine the search space of possible shapes, employ pre-trate pre-trained shape spaces in our network. The shape space is represented by a deep neural network using continuous function representation. Our experiments demonstrate that model achieves scene parsing for variety of object counts and shapes. We provide an ablation study to motivate design choices and discuss assumptions and limitations of our approach. We further demonstrate the advantages of our model to reason about the underlying 3D space of seen scene by performing explicit manipulation on the individual objects or rendering novel views. To the best of our knowledge, our approach is the first to jointly learn the tasks of object instance detection, instance segmentation, object pose estimation, and inference of 3D shape and texture in a single RGB image semi-supervised way. We believe our approach provides an important step towards self-supervise learning of object-level 3D scene parsing and generative modeling of com- plex scenes from real images. Our work is currently limited to simple scenes with few objects on a uniformly colored background. The usage of such synthetic data allows us evaluate the individual design choices our model in a controlled setup. future work, we plan to address challenges of more complex scenes with more diverse background and objects and real imagery.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning represen- tations and generative models for 3D point clouds. In Proceedings of the 35th International Conference on Machine Learning (ICML), pp. 40-49, 2018.
Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adversarial examples. In Proceedings of the International Conference on Machine Learning (ICML), pp. 284 293, 2018.
Christopher P. Burgess, Loic Matthe, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. MONet: Unsupervised scene decomposition and representa- tion, 2019.
Angel X. Chang, Thomas A. Funkhouser, Leonidas J. Guibas, Pat Hanrahan, Qi-Xing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. Shapenet: An information-rice 3d model repository. ArXiv, abs/1512.03012, 2015.
Wenzheng Chen, Jun Gao, Huan Ling, Edward Smith, Jaakko Lehtinen, Alec Jacobson, and Sanja Fidler. Learning predict 3D objects with interpolation-based differentiable renderer. In Advances In Neural Information Processing Systems (NeurIPS), 2019.
Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5939-5948, 2019. doi: 10.1109/
Christopher Choy, Danfei Xu, Jun Young Gwak, Kevin Chen, and Silvio Savarese. 3d-r2n2: A uni- fied approach for single and multi-view 3D object reconstruction. In Proceedings of the European Conference on Computer Vision (ECCV), 2016.
Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convolu- tional neural networks. In Proceedings of the AAAI Conference Artificial Intelligence, 2019.
Amael Delaunoy and Emmanuel Prados. Gradient flows for optimizing triangular mesh-based sur- faces: Applications to 3D reconstruction problems dealing with visibility. International Journal of Computer Vision (IJC)), 95: 10:1001, 11 2011. doi: 10.1007/s 1263-010-0408-9,
David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep network. In Proceedings of the 27th International Conference on Neural Infor- mation Processing Systems Volume 2, NIPS' 14, pp. 2366-2374, Cambridge, MA, USA, 2014. MIT Press.
Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. GENESIS: Gener- ative scene inference and sampling with object-centric latent representations. In Accepted for International Conference on Learning Representations (ICLR), 2020.
M. Ali Eslami, Nicolas Heess, Theophane Weher, Yuval Tassa, David Szepesvari, Koray Kavukcuoglu, and Geoffrey E. Hinton. Attend, infer, repeat: Fast scene understanding with generative models. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, pp. 3233-3241, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819
S. M. Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S. Morcos, Marta Garnelo, Avraham Ruderman, Andrei A. Rusu, Ivo Danihelka, Karol Gregor, David Reichert, Lars Buesing, Theophane Weber, Oriol Vinyals, Dan Rosenbaum, Neil Rabinowitz, Helen King, Chloe Hillier. Matt Botvinick, Daan Wierstra. Koray Kavukcuoglu, and Demis Hassabis. Neural scene representation and rendering. Science, 12006394)110041221 2018. doi: 10. 126/scc/c... aar6170.
Mark Everingham, Lue Gool, Christopher K. Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. Int. J. Comput. Vision, 88(2)3003-338 2010. ISSN 0920-5691. doi: 1007/s -09-266300007-5--4 URL hallaattss///et //ccsi://///oo/oo/ org/10. 611263-009-0275-
10

---

Under review as a conference paper at ICLR 2021
M. Gadelha, S. Maji, and R. Wang. 3d shape induction from 2d views of multiple objects. In 2017 International Conference on 3D Vision (3DV), pp. 402-411, Oct 2017. doi: 10.1109/3D.2017. 00053.
Georgia Gkioxari, Jitendra Malik, and Justin Johnson. Mesh R-CNN. In Proc. of International Conference on Computer Vision (ICCV), 2019.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-ohject representation learning with iterative variational inference. In Proceedings of the 36th International Conference on Machine Learning (ICML), 2019.
Thibault Groueix, Matthew Fisher, Vladimir G. Kim, Bryan C. Russell, and Mathieu Aubry. At- lasNet: A papier-mâché approach to learning 3d surface generation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 216 224. IEEE Computer Society, 2018.
doi: 10.11099 8000030.
Jun Young Gwak, Christopher B Choy, Manmohan Chandraker, Animesh Garg, and Silvio Savarese. Weakly supervised 3d reconstruction with adversarial constraint. In Int. Conf. on 3D Vision (3DV), 2017.
Paul Henderson and Vittorio Ferrari. Learning Lingle-iaag 3d reconstruction by generative mod- elling of shape, pose and shading. International Journal of Computer Vision, 2019, doi: 10. si 11126-011--12199.. URLpttps:///dat wrg.//r.gg.../.0 100700111/10111111101
Hou, Angela Dai, and Matthias Niessner. 3D-SIS: 3D semantic instance segmentation of RGB-D scans. In Proc. of Computer Vision and Pattern Recognition (CVPR), 2019.
Danilo Jimenez Rezende, S. M. Ali Eslami, Shakir Mohamed, Peter Battaglia, Max Jaderberg, and Nicolas Heess. Unsupervised learning of 3D structure from images. In Advances in Neural Information Processing Systems 29, pp. 4996-5004, 2016.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, 2017.
A. Kar, C. Häne, and J. Malik. Learning a multi-view stereo machine. In Proc. of Advances in Neural Information Processing (NeurIPS), 2017.
Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3d mesh renderer. In IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR), 2018
Hiroharu Kato, Deniz Beker, Mihai Morariu, Takahiro Ando, Toru Matsuoka, Wadim Kehl, and Adrien Gaidon. Differentiable rendering: A survey, 2020.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference Learning Representations, 12 2014.
Vladimir A. Kniaz, Vladimir V. Knyaz, Fabio Remondino, Artem Bordodymov, and Petr Moshkant- sev. Image-to-voxel model translation for 3d scene reconstruction and segmentation. In Proceed- ings of the European Conference on Computer Vision (ECCV). Springer International Publishing, 2020.
Yiyi Liao, Katja Schwarz, Lars Mescheder, and Andreas Geiger. Towards unsupervised learning of generative models for 3d controllable image synthesis. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020.
Chen-Hsure Lin, Chen Kong, and Simon Lucey. Learning efficient point cloud generation for dense 3d object reconstruction. Sheila A. Mcllraith and Kilian Q. Weinberger (eds.), Proceedings of the Thirty-Secome AAAI Conference Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the Sth AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans. Louisiana, USA, February 2-7, 2018, pp. 1114-71214 AAAI Press, 2018 URL https: www. aaai. or ccs /w/iinnxxx./// php/
AAAT AAAAAAAATAAAAA 8 www.AA///papp////....
6530

---

Under review as a conference paper at ICLR 2021
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang. and Sungjin Ahn. SPACE: Unsupervised object-oriented scene representation via spatial attention and decomposition. In Accepted for International Conference on Learning Representa- tions (ICLR), 2020.
Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. Soft rasterizer: A differentiable renderer for image-based 3d reasoning. In Proc. of the IEEE International Conference on Computer Vision (ICCV), 2019.
Matthew M. Loper and Michael J. Black. OpenDR: An approximate differentiable renderer. In European Conference on Computer Vision (ECCV), 2014.
Abhimitra Meka, Maxim Maximov, Michael Zollhoefer, Avishek Chatterjee, Hans-Peter Seidel, Christian Richardt, and Christian Theobalt. LIME: Live intrinsic material estimation. In Pro- ceedings of Computer Vision and Pattern Recognition (CVPR), 2018.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2019.
Thu Nguyen-Phuoc, Chuan Li, Stephen Balaban, and Yong-Liang Yang. RenderNet: A deep convo- lutional network for differentiable rendering from 3D shapes. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Michael Oechsle, Lars Mescheder, Michael Niemeyer, Thilo Strauss, and Andreas Geiger. Texture fields: Learning texture representations in function space. In Proceedings IEEE International Conf. on Computer Vision (ICCV). 2019.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Mihir Prabhudesai, Shamit Lal, Hsiao-Yo Yu Fish Tung, Adam W. Harley, Shubhankar Potdar, and Katerina Fragkiadaki. 3dq-nets: Visual concepts emerge in pose equivariant 3d quantized neural scene representations. Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1577-1570. IEEE, 2020. doi:
URLhttps.
7/doi org 10. 1119///PR/504099990 2020. 00202
Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proc. of IEEE Int. Conf. on Computer Vision and Pattern Recognition (CVPR), 2017.
Charles Ruizhongtai Qi, Hao Su, Matthias Nießner, Angela Dai, Mengyuan Yan, and Leonidas Guibas. Volumetric and multi-view cnns for object classification on 3d data. In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 2016.
Ravi Ramamoorthi and Pat Hanrahan. A signal-processing framework for inverse rendering. In SIGGRAPH, pp. 117-128, 2001.
Elad Richardson, Matan Sela, Roy Or-El, and Ron Kimmel. Learning detailed face reconstruction from a single image. In Proceedings of the Int. Conf. on Computer Vision and Pattern Recognition (CVPR), 2016.
Dacyun Shin, Charless Fowlkes, and Derek Hoiem. Pixels, voxels, and views: A study of shape representations for single view 3d object shape prediction. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Con- tinuous 3d-structure-awann neural scene representations. In Advances in Neural Information Pro- cessing Systems (NeurIPS), 2019.
Maxim Tatarchenko, Alexey Dosovitskiy, and Thomas Brox. Multi-view 3d models from single images with a convolutional network. In Proc. of the European Conference on Computer Vision (ECCV), pp. 322-337, Cham, 2016. Springer International Publishing.
12

---

Under review as a conference paper at ICLR 2021
A. Tewari, O. Fried, J. Thies, V. Sitzmann, S. Lombardi, K. Sunkavalli, R. Martin-Brualla, T. Simon, J. Saragih, M. Nicßner. R. Pandey, S. Fanello, G. Wetzstein, J.-Y. Zhu, C. Theobalt, M. Agrawala, E. Shechtman, D. B Goldman, and M. Zollhöfer. State of the art neural rendering. Computer Graphics Forum (EG STAR 2020), 2020.
Shubham Tulsiani, Tinghui Zhou, Alexei Efros, and Jitendra Malik. Multi-view supervision for single-view reconstruction via differentiable ray consistency. Proc. of the IEEE Int. Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 209-217, 07 2017. 2017.30.
doi: 10. 10.1109/CVPR.
R. Wang, N. Yang. J. Stueckler, and D. Cremers, Directshape: Photometric alignment of shape priors for visual vehicle pose and shape estimation. In Proc. of the IEEE International Conference Robotics and Automation (ICRA), 2020.
Jiajun Wu, Chengkai Zhang, Tianfan Xue, William Freeman, and Joshua B Tenenbaum. Learning probabilistic latent space of object shapes via 3d generative-aaversarial modeling. In Advances in Neural Information Processing Systems (NeurIPS), pp. 82-90, 2016.
Jiajun Wu, Yifan Wang. Tianfan Xue, Xingyuan Sun, William T Freeman, and Joshua B Tenen- baum. MarrNet: 3D shape reconstruction via 2.5D sketches. Advances In Neural Information Processing Systems (NeurIPS), 2017.
Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox. PoseCNN: A convolutional neural network for 6D object pose estimation in cluttered scenes. In Robotics: Science and Systems (RSS), 2017.
Haozhe Xie, Hongxun Yao, Xiaoshuai Sun, Shangchen Zhou, and Shengping Zhang. Pix2vox: Context-awarre 3d reconstruction from single and multi-view images. In International Conference on Computer Vision (ICCV), pp. 2690-2698, 2698. IEEE, 2019, doi: 10.10099 URL
httpps://dot org/ 10. 10.00.1000 110111199/10 ICC/ 201 9. 00278
Qiangeng Xu, Weiyue Wang, Duygu Ceylan, Radomír Mech, and Ulrich Neumann. DISN: deep implicit surface network for high-quality single-view 3d reconstruction. In Advances in Neural Information Processing Systems (NeurIPS), pp. 490-500, 2019.
Xinchen Yan. Jimei Yang. Ersin Yumer, Yijie Guo, and Honglak Lee. Perspective transformer nets: Learning single-view 3d object reconstruction without 3d supervision. In Advances in Neural Information Processing Systems (NeurIPS). 2016.
Yanchao Yang, Yutong Chen, and Stefano Soatto. Learning to manipulate individual objects in an image. In 2020 JEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 6557-6566 IEEE, 2020, doi: 10. 109/CVP22222 2020.00659.
2020 00659
Wang Yifan, Felice Serena, Shihao Wu, Cengiz Öztireli, and Olga Sorkine- Hornung. Differentiable surface splatting for point-based geometry processing. ACM Transactions on Graphics (proceed- ings of ACM SIGGRAPH ASIA), 38(6), 2019.
Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Jun Yan Zhu, Zhoutong Zhang, Chengkai Zhang, Jiajun Wu, Antonio Torralba, Josh Tenenbaum, and Bill Freeman. Visual object networks: Image generation with disentangled 3d representations. In A Advances in Neural Information Processing Systems (NeurIPS). 2018.
13


Reviews:
[{"review_id": "tOIJ-Q0-CD", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Weaknesses\n\n- All experiments are synthetic. Although synthetic experiments are common in the field, I see this as a weakness because the method heavily relies on image reconstruction loss on rendered objects. Real scene are very different due to both high level (more clutter, occlusion, number of objects) and low level (shading, less predicable texture) differences from the examples used in the synthetic experiments and will likely require different constraints to solve. The model has access to dense ground truth depth maps too, which is difficult to obtain. The motivation for self-supervised learning is to take advantage of unlabeled in-the-wild images, so considering all of those things, I think the synthetic RGB-D assumption hurts this paper more than an average paper.\n\n- There's much to learn from the object count experiment (Table 1 and Page 6), but I found the results confusing. According to Page 5 which says \"each dataset contains scenes with a specific number of objects which we choose from two to five\", the network was trained to model a specific number of objects separately in each experiment. If you evaluate a three-object trained network on images containing two objects (\"obj=3/2\" in Table 1, 10th row), one would expect it to suffer from false positives, but the quantitative results do not show this. The results are almost the same as \"obj=3/3\". Why is this? Did it learn to solve this problem, or were they somehow not included in the evaluation? I think there's some resemblance to EM, and it would matter a lot if you knew the number of true clusters beforehand, and how over or under-estimations are handled, in understanding the model's behavior.\n\n- Lack of basic baselines. Currently there's no way to know how difficult this dataset is, and how well a fully supervised model would perform.\n\n-  Potentially missing details, additional questions, and suggestions:\n     - In RGB reconstruction loss and evaluation, do you also count the background, or is this on ground truth object region only?\n     - Page 2 says \"By representing 3D geometry explicitly, our approach naturally respects occlusions and collisions between objects and facilitates manipulation of the scene within the latent space.\" This part was confusing to me. How does it avoid collision of objects in 3D? My understanding is that this is not part of equation 1.\n     - Figure 2 is confusing. I think a few captions in the image would have helped a lot.\n     - In Table 2, would you attribute the gap between seen and unseen instances more to the shape, or the texture of the object?\n\n- One suggestion is, would you be able to render the scene from a novel viewpoint, since you know the depth, shape and pose of each object? And then evaluate against ground truth rendering from that viewpoint? Being able to do such an experiment could make the synthetic setting more useful.  \n\n### Strengths\n\n- Ablation study in Table 1 shows that all of the loss terms are important. This justifies the design and I would consider it an insightful discovery.  For example, the ground truth depth already globally provides sharp object boundaries, but in order to reproduce them, both shape and ground plane loss terms were needed. So the model provides the right constraints in which all of those factors are considered.\n\n- \"How do you turn single-object 3D reconstruction models into priors, for scene-level inference?\" is still a very open question, and more research is needed in this area. The fact that this methods requires such a prior could be a weakness to someone who prefers to see less assumptions made in favor of generalizability. But I believe objects have good enough scientific foundation to be beneficial in the long run.\n\n### Justification of rating\n\nDespite the weaknesses I listed, I feel that this paper is above threshold, albeit marginal. Other researchers can still learn from the design choices made even if it turns out that those constraints don't work well in real scenes. I understand that this is a difficult task, and having a synthetic model as a reference point would ultimately help, assuming the code for reproducing the experiments will be fully released, to serve as a baseline.", "extra_scores": {}, "extra_text": {}}, {"review_id": "sSLepV5uMIG", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper introduces a new model for obtaining object-level 3D scene representations from images. The proposed method models 3D scenes by parsing objects one-by-one into structured representations of shape, texture, and poses. A differentiable renderer incorporated in this pipeline enables self-supervised training as well as test-time image generation.\n\nThe method is interesting and novel, though general ideas are borrowed from Park et al. 2019, Sitzmann et al. 2019 and Wang et al. 2020. The paper is well-written and easy to follow for the most part. Also, the experiments performed have supported the claims from the authors. \n\nPros: \n- This is the first work to jointly learn the tasks of object instance detection, instance segmentation, object localization, and inference of 3D shape and texture in a single RGB image through self-supervised scene decomposition.\n- The idea of incorporating a renderer into the parsing pipeline is interesting and meaningful. In this way, the method is able to perform self-supervised decomposing of the scene.\n- The authors perform extensive experiments and ablation studies on the clevr and ShapeNet dataset, which have proved the effectiveness of the proposed pipeline on these datasets and provided useful information on designing such kind of pipeline.\n\nCons:\n- Although the authors provide preliminary results on real-world images, the extension of this pipeline to real-world scenes is doubtful:\n  - The number of objects is fixed/given for the framework; the increasing number of objects in complex scenes leads to more ill-conditioned problems, which will make the self-supervised decomposition extremely hard.\n  - Scene conditions (e.g. light) are not modeled or overly simplified (e.g. background) within this pipeline, and the differentiable renderer used (or even the sota differentiable renderers) are not able to sufficiently model complex real scene conditions.\n  - It would be appreciated if you could include more complex shapes in real-world images. For example chairs/toy cars as in the ShapeNet.\n\nMinor Comments & Questions:\n- The `Change Shape` example for ShapeNet in Figure 1 does not seem to be obvious. \n- Figure 2 is not clear enough at first glance. You can add some texts in Figure 2 to make it more self-contained and understandable, e.g. \"g_O: object encoder\", \"g_{bg}: background encoder\", etc. Also, the `Top` and `Bottom` is clearly separated. You should consider to split and reorganize these two figures.\n- The method section is slightly verbose and can be better organized.\n- Is it possible to render new views given the parsed scene representation? This seems to be possible but not mentioned in the paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "K8C1LQugs6u", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors proposed a method for 3D scene inference, which jointly does object instance detection, instance segmentation, object localization, and 3D shape and texture inference. The authors designed an autoencoder-like network such that it can be trained in a self-supervised way from RGBD images. Some limitations:\n- The authors claimed they are the first to do these tasks jointly, and they did not compare their method to any existing work, but only tested on the Clever and ShapeNet dataset. However, there are existing depth/normal estimation algorithms which can serve as baselines.\n- It is not justified the superiority of the design of the method. Did it model the relationships between objects? Why not using an object detector combined with a 3D shape/texture reconstruction neural network?\n- The authors did not test their method on real data. Only tested on the Clever and ShapeNet dataset.\n- The pipeline figure (Figure 2) is very difficult to understand. Might be better with more text illustrations in the figure, not only notations.\n\n---Post-Rebuttal---\n\nThank the authors for their response. I now agree with the authors and other reviewers that the authors' approach has its novelty (self-supervised, rendering, etc.), and the ablation study in Table 1 is reliable to prove each component is useful. Therefore, I increase my rating by 1.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes learning of 3D object representation from images. The pretraining used assumes it can generate implicit 3D models for the objects, and then objects are detected in multi-object scenes without further supervision. Reviewers raised concerns regarding experiments being conducted only on synthetic data. Authors are encouraged to try out their approach on real data, to demonstrate the benefits of their solution.
Decision: Reject