Neural Groundplans: Persistent Neural Scene Representations from a Single Image
Authors: Prafull Sharma
Ayush Tewari
Yilun Du
Sergey Zakharov
Rares Andrei Ambrus
Adrien Gaidon
William T. Freeman
Fredo Durand
Joshua B. Tenenbaum
Vincent Sitzmann
Venue: ICLR 2023 poster (2023)
Keywords: Neural scene representations
3D
nerf
scene understanding
neural rendering
object-centric representations

Abstract: We present a method to map 2D image observations of a scene to a persistent 3D scene representation, enabling novel view synthesis and disentangled representation of the movable and immovable components of the scene. Motivated by the bird’s-eye-view (BEV) representation commonly used in vision and robotics, we propose conditional neural groundplans, ground-aligned 2D feature grids, as persistent and memory-efficient scene representations. Our method is trained self-supervised from unlabeled multi-view observations using differentiable rendering, and learns to complete geometry and appearance of occluded regions. In addition, we show that we can leverage multi-view videos at training time to learn to separately reconstruct static and movable components of the scene from a single image at test time. The ability to separately reconstruct movable objects enables a variety of downstream tasks using simple heuristics, such as extraction of object-centric 3D representations, novel view synthesis, instance-level segmentation, 3D bounding box prediction, and scene editing. This highlights the value of neural groundplans as a backbone for efficient 3D scene understanding models.

Published as a conference paper at ICLR 2023
NEURAL GROUNDPLANS: PERSISTENT NEURAL SCENE REPRESENTATIONS FROM A SINGLE IMAGE
Prafull Sharma'
Ayush Tewari!
Yilun Du
Sergey Zakharov
Rares Ambrus
Adrien Gaidon
William T. Freeman
Frédo Durand
Joshua B. Tenenbaum
Vincent Sitzmann!
MITT CSAIL MIT BCS 'MIT CBMM
*Toyota Research Institute
Thee NSF AI Institute for Artificial Intelligence and Fundamental Interactions
prafullsharme www.eeee/eeeeeee neurall groundplans
ABSTRACT
We present method map 2D image observations of scene a persistent 3D scene representation, enabling novel view synthesis and disentangled representation of the movable and immovable components of the scene. Motivated by the bird's- eye-view (BEV) representation commonly used in vision and robotics, we propose conditional neural groundplans, ground-aligned 2D feature grids, as persistent and memory-efficient scene representations. Our method is trained self-supervised from unlabeled multi-view observations using differentiable rendering, and learns complete geometry and appearance of occluded regions. In addition, that we can leverage multi-view videos at training time to learn to separately reconstruct static and movable components of the scene from a single image test time. The ability to separately reconstruct movable objects enables variety of downstream tasks using simple heuristics, such as extraction of object-centric 3D representations, novel view synthesis, instance-level segmentation, 3D bounding box prediction, and scene editing. This highlights the value of neural groundplans as backbone for efficient 3D scene understanding models.
show
INTRODUCTION
We study the problem of inferring a persistent 3D scene representation given a few image observations, while disentangling static scene components from movable objects (referred to as dynamic). Recent works in differentiable rendering have made significant progress in the long-standing problem of
3D reconstruction from small sets image observations (Yu et al., 2020) [Sitzmann et al. [219bb]
Sajjadi et al. 2021). Approaches based on pixel-aligned features Yu et al.) 2020; Trevithick & Yang 2021 Henzler et al. (02T) have achieved plausible novel view synthesis of scenes composed of independent objects from single images. However, these methods do not produce persistent
3D scene representations that can be directly processed in 3D, for instance, via 3D convolutions. Instead, all processing has be performed in image space. contrast, some methods infer 3D voxel grids, enabling processing such as geometry and appearance completion via shift-equivariant 3D convolutions (Lal al. 2021] Guo al. [2022), which is however expensive both in terms of computation and memory. Meanwhile, bird view view (BEV) representations, 2D grids aligned with the ground plane of a scene, have been fruitfully deployed as state representations for navigation, layout generation, and future frame prediction (Saha et al.) 2022) Philion & Fidler 2020 Roddick ct al. 2019 Jeong et al.) 2022 Mani et al.) 2020). While they compress the height axis and are thus not a full 3D representation, 2D convolutions top of BEVs retain hhitt-equivariance in the ground plane and are, in contrast to image-space convolutions, free of perspective camera distortions.
Inspired by BEV representations, propose conditional neural groundplans, 2D grids of learned features aligned with the ground plane of a 3D scene, as a persistent 3D scene representation
"Email: prafull @ mit.edu, sermann@@mit.dd

---

Published as a conference paper at ICLR 2023
Instance level Bounding box Bird'seye sopmentation
Inpor
Reconstraction
Statie
Dynamic
Depth
segmentation
view
views
Figure I: Given a single image, our model infers separate 3D representations for static and dynamic scene elements, enabling high-quality novel view synthesis with plausible completion, unsupervised instance- segmentation, 3D bounding box prediction, 3D scene editing, and extraction of object- centric 3D representations. Our model is trained self-supervised using unlabeled multi-view videos.
reconstructed in feed-forward manner. Neural groundplans neural scene representation (Chan et al. 2022 Peng et al. [2200] Philion & Fidler 2020; Roddick al. 2019 Mani et al., 2020) and enable 3D queries by projecting 3D point onto the groundplan, retrieving the respective feature. and decoding it via an MLP into full 3D scene. This enables self-supervised training via differentiable volume rendering. By compactifying 3D space with a nonlinear mapping, neural groundplans can encode unbounded 3D scenes in bounded region. We further propose to reconstruct separate neural groundplans for 3D regions and 3D regions of scene that are static given single input image. This requires that objects are moving in the training data, enabling us to learn a prior to predict which parts of a scene are movable and static from single image at test time. We achieve this additional factorization by training multi-view videos, such as those available from cameras at traffic intersections or sports game footage. Our model is trained sule-supe via neural rendering without sveudo-groung truth, bounding boxes, or any instance labels. We demonstrate that separate reconstruction of movable objects enables instance-level segmentation, recovery of 3D object-centric representations, and 3D bounding box prediction via a simple heuristic leveraging that connected regions of 3D space that move together belong to the same object. This further enables intuitive 3D editing of the scene.
hybrid discrete-continen 31>
scene that are movable
Since neural groundplans are 2D grids of features without perspective camera distortion, shift- equivariant processing using inexpensive 2D CNNs effectively completes occluded regions. Our model thus outperforms prior pixel-aligned approaches in the synthesis of novel views that observe 3D regions that are occluded in the input view. We further show that by leveraging motion cues at training time, our method outperforms prior work on the self-supervised discovery of 3D objects.
In summary, our contributions are:
We introduce self-supervised training of conditional neural groundplans, a hybrid discrete- continuous 3D neural scene representation that can be reconstructed from a single image, enabling efficient processing of scene appearance and geometry directly in 3D.
We leverage object motion as cue for disentangling static background and movable foreground objects given only a single input image.
Using the 3D geometry encoded in the dynamic groundplan, demonstrate single-image 3D instance segmentation and 3D bounding box prediction, as well as 3D scene editing.
2 RELATED WORK
Neural Scene Representation and Rendering. Several works have explored learning neural scene representations for downstream tasks in 3D. Emerging neural scene representations enable recon- struction of geometry and appearance from images as well as high-quality novel view synthesis via differentiable rendering. large part of recent work focuses on the case of reconstructing a single 3D scene given dense observations (Cheng et al. 2018 Tung et al.] 2019) Sitzmann et al. 201 9a, Lombardi et al. 2019 Mildenhall et al.) 2020 Yariv et al. 22200] Tewari et al. 2021). Alternatively, differentiable rendering may be used to supervise encoders to reconstruct scenes from single or few images in a feedforward manner. Pixel-aligned conditioning enables reconstruction of compositional
scenes (Yu et al. 2020; Trevithick & Yang| 2021). but does not infer compact 3D representation. Methods with a single latent code per scene do, but do not generalize to compositional scenes (Sitz)
mann et al. 2019c Jang & Agapito, 2021
Niemeyer et al. 2020 Sitzmann al. 2021 Kosiorek
2

---

Published as a conference paper at ICLR 2023
2021 Voxel grid based approaches offer both benefits, but are computationally costly Lal et al.
et al. 2021; Sajjadi et al. 2021 Dupont et al. 2020). Hybrid discrete- contrte- neural scene representations offer compromise by factorizing a dense 3D field into several lower-dimensional representations that are used to condition an MI.P (Chan et al. 2022 Chen et al. 2022a). In particular, neural groundplans and axis-aligned 2D grids enable high-quality unconditional generation of 31) scenes (DeVries et al.
2021 Chan et al. (022) as well as reconstruction of 3D geometry from pointclouds (Peng et al.
2020). We similarly use axis-aligned 2D grids of features for self-supervieed scene representation via neural rendering, but reconstruct them directly from few or a single 2D image observations.
Bird's-Eye View Representations. Bird's-eye view has been explored as 31) representation in vision and robotics, particularly for autonomous driving applications. Prior work uses ground-phan -lann 2D grids as representations for object detection and segmentation (Saha et al. 2022] Harley et al. 2022, Philion & Fidler, 2020; Reiher et al. 2020; Roddick et al. 2219). layout generation and completion (Cao & de Charette| [022 2022 Jeong et al. 2022; Mani et al., 2020 Yang al., 2021b). and next-frame prediction (Hu et al. [021] Zieba et al. (022)). The bird's s-eye view is generated either directly without 3D inductive biases (Mani et al., 2020). or similar to our proposed approach, by using 3D geometry-driven inductive biases such as unprojection into a volume (Harley et al.
2022 Chen et al. 2022b; Roddick et al.) 2019), or by generating 3D point cloud (Philion & Fidler supervised, using ground truth bounding boxes or semantic segmentation as supervision. In contrast, we present a self-supervised conditional groundplan representation, learned only from images via neural rendering. While show that our self-supervised representation can be used for rich inference tasks using simple heuristics, our method may be extended for more challenging tasks using the techniques developed in prior work.
2020 Hu et al.) 2021). However, prior approaches
Dynamic-Static Disentanglement. Our work is related to prior work on learning to disentangle dynamic objects and static background. Some prior work leverages object motion across video frames to learn separate representations for movable foreground and static background in 2D (Kasten 2022 Bao et al. 2022), while other recent work can also learn 3D representa-
et al. 2021 Ye et al.
tions Yuan et al. 2021 l'schemezki al. 1021T. Our approach is similar in using object motion as cue for disentanglente lement and multi-view as cue for 3D reconstruction, but uses it as supervision to train an encoder-based approach that enables reconstruction from single image instead of scene-specific disentanglement from multiple video frames.
Object-centric Scene Representations. Prior work has aimed to infer object-centric representations directly from images, with objects either represented as localized object-centric patches (Lin et al.
2020 Eslami et al. 2016 Crawford & Pineau 2019 Kosiorek et al. 2018]] Jiang et al., 2019) or
scene mixture components (Engeleke al. 2020: Burgess al.] 2019 Greff al. 2019 2016
2017 Du et al. 2021a)) with the slot attention module (Locatello et al. 2020) increasingly driving
object-centric inference. Resulting object representations may be decoded into object-centric 3D representations and composed for novel view synthesis (Yu et al. 2022 Smith et al. [2022] Elich et al.
2022 Chen et al. 202T Bear et al. 2020 Zakharov al. 2020 202 Beker et al. 2020 Du et al.
2021b BlockGAN and GIRAFFE [Nguyen-Phuoe et al. 2020 Niemeyer & Geiger (021) build unconditional generative models for compositions of 3D-structured representations, but are restricted to only generation. Some methods rely annotations such as bounding boxes, object classes, 3D object models, or instance segmentation to recover object-centric neural radiance fields (Ost eet al. 2021; Yang et al./ 22021a] Guo et al. 2020 Yang al. (022)). Several scene reconstruction methods (Zakharov et al. 2020 2021 Beker et al. 2020 Nie et al. 2020) use direct supervision to train object representation and detector to infer an editable 3D scene from a single frame observation. KKipf et al.) (202T) leverage motion as a cue for self-supervised object disentanglement, but do not reconstruct 3D and require additional conditioning in the form of bounding boxes. In this work, we demonstrate that a representation factooized into static and movable 3D regions can serve as powerful backbone for object discovery. While not explored in this work, slot attention and related object-centric algorithms could be run on our already sparse groundplan of movable 3D regions, faced with dramatically easier task than when run on images directly.
3

---

Published as a conference paper at ICLR 2023
3D Scome
Piller Aggregation
Statie- Matti-Dyuaamic Diseutanglenentt
eofmax
MLP
Eaa
Static ground plan
2D CNN
Iuput
CNN
Eutsugled Neural Ground Plan
Camera
Features
Dynamie ground plan
Figure 2: Groundplan inference. Given a context image, we first extract set of CNN features. We unproject the features into 3D and re-sample them at "pillars" on top of the location of groundplan vertices. Pillars are aggregated into groundplan features using a softmax-weight 2D grid of features decomposed into separate dynamic and static groundplans by a 2D CNN. The nooddinateeenodingg
sum. The resulting
coordinate encoding MLP is not visualized in this figure. Please refer to Sec. 3 for details.
3 CONDITIONAL NEURAL GROUNDPLANS
In this section, we describe the process of inferring neural ground plan given one or several image observations in feed-forward manner, as well as subsequent novel view synthesis. The method will be trained on dataset of multi-view videos with calibrated cameras with wide baselines. Please see Fig. 2 for an overview.
Compactified neural groundplans for unbounded scene representations. neural groundplan is 2D grid of features aligned with the ground plane of the 3D scene, which we define to be the xz zz-plane. A 3D point decoded by projecting it onto the groundplan and retrieving the corresponding feature vector using bilinear interpolation. This feature is then concatenated with the vertical y-coordinate of the query point and decoded into radiance and density values via fully connected network, enabling novel view synthesis using volume rendering (Mildenhall et al. (2000). In this definition, however, it only possible to decode 3D points that lie within the boundaries the neural groundplan, which precludes reconstruction and representation of unbounded scenes. We thus compactify I by implementing a non-linear coordinate re- mapping as proposed by Barron et al. (202T). Points x within a radius Tinner around the groundplan origin remain unaffected, but points outside this radius are contracted. For any 3D point x, the contracted 3D coordinate can be computed as x' C(x) ((1 k) k/ //||u[|) (u/Ilu] [[]]]))mmmers where = x/Timmers and & is hyperparameter which controls the size of the contracted region. Note that C' is invertible, such that x = C-² (x') is a function that takes a 3D point in contracted space x' to the original 3D point x in linear space.
Reconstructing neural groundplans from images. Inferring neural groundplan from one or several images proceeds in three steps: (1) feature extraction, (2) feature unprojection, (3) pillar aggregation. Given a single image I, we first extract per-pixel features via CNN encoder to yield a feature tensor F. We define the camera as the world origin and center the neural groundplan accordingly, approximately aligned with the ground level. The image features are unprojected to 3D feature volume v in contracted world space using the inverse of the contract function defined earlier. We extract the feature contracted 3D point x' in
v(x') F F[ (C (x')), where C='(x') first maps the contracted point to linear world space and <(-) projects it onto the image plane of the context view using camera extrinsics and intrinsics. At any vertex of the groundplan, the discretized y-coordinates of the volume form a "pillar". Next, we aggregate each pillar into a point to create the 2D groundplan. We first use a coordinate-encodin MLP D(-) to transform the volume as f(x') D(v(x (0v(x'), Xc, d), where X. denotes the 3D point in linear camera coordinates of the context camera, and denotes the ray direction from the camera center to that point. Since all features along camera ray identical in v, coordinate encoding is used to add the depth information to the features. In the case of multi-vies view input images, the volumes corresponding to each input view are mean pooled. Associated to each 2D vertex of the groundplan is now set of features {f, /==1> where N is the number of samples along the y-dimension. We use a softmax scores a; = P(f,, x.). where P(-) denotes the MLP and x; is the linear coordinate of the i-th point on the pillar. Finally, the features are aggregated by computing the weighted sum of the features, g Σ; a.f.
N
MIP to compute
Differentiable Rendering. We can render images from novel camera views via differentiable volume rendering (De Vries et al. 202T] Lombardi et al.) 2019; Mildenhall et al. 2020). To resolve points closer to the camera more finely, we adopt logarithmic sampling of points along the ray with more
4

---

Published as a conference paper at ICLR 2023
Context Frames
timestep Groud Plan Inference
Compusiting & Rendering
Pool
laa-- 120 1
g.t.
2
g.t.
Statie Ground Plan
t= 0
Statio
MLP
Composite
Reconstruction Losses
Dynamic
11oa1
1=1
= 0
1=1 Dynamic Ground Plans
1= Dynamic Sparsity Loss
Figure 3: Learning Static-Dynamic Disentanglement. Given multiple frames of video, we extract per-frame compactified, static and dynamic groundplans according to Fig. 2 Static groundplans are pooled into a time-invariant groundplan. We then composite per-frame dynamic and static time-invariant groundplans via differentiable volume rendering. Our model is supervised only via re-rendering loss on video frames. We encourage the model to explain as much of the scene density possible with the static groundplan via sparsity loss per-fran per-frame dynamic volume rendering densities. The surface loss is not visualized here.
samples close to the camera (Neff et al. (021). For each sampled point x on the camera ray, we need compute its density and color for volume rendering. This accomplished using rendering MLP, as (Cx, xx) = R(gx; Vx), where R(-) denotes the MLP, gx are the groundplan features for the point x computed by projecting the query 3D coordinates onto the groundplane and bilinearly interpolating the nearest grid points, and yx is the value of the sampled point
4 LEARNING STATIC-DYNAMIL DISENTANGLEMER
We now describe training on multi-view video to learn to disentangle static and dynamic compo- nents of the scene. Furthermore, describe method for performing self-supervised 3D object discovery and 3D bounding box prediction using the geometry encoded in the dynamic groundplan representation. Please see Fig. for an overview of the multi-frame training for static-dynamic disentanglement.
Disentangling static and dynamic neural groundplans. We leverage the fact that objects move in the given multi-view videos the training signal. We pick two random frames of video. For each frame, we infer an entangled neural groundplan as described in the previous section. Features in this entangled neural groundplan parameterize both static and dynamic features of the scene, for instance, a car as well as the road below it. We feed this groundplan into a fully convolutional 2D network, which disentangles into two separate groundplans containing static and dynamic features. The per-frame static groundplans are mean-pooled to obtain single, time-inneene inme-independent static groundplan.
Compositing groundplans. To render scene using the disentangled static and dynamic groundplans, first decode query points using both groundplans, yielding swo sets of (density, color) values for each point. We use the compositing operation proposed by Yuan et al. (202T) to compose the contribution from static and dynamic components along the ray. Given the color and density for static (c o and dynamic (c" ob) parts, the density of the combined scene is calculated as の +o The color at the sampled point computed as weighted linear combination w S w DD where w (1 exp( (xp(-5o=)) ))(() exp( S + oP)), wD = (1 exp(- -50 50o) /)/((1 - exp( (xp(- + +o))) and the distance between adjacent samples on the camera ray.
Losses and Training. We train our model multi-view video, where multi-view information is used to leamn the 3D structure, while motion is used to disentangle the static and dynamic components in the scene. During training, we sample two time-steps per video. For each time-step, we sample multiple images from different camera views; some of the views are used as input to the method while others are used compute the loss function. We use the input images infer static and dynamic groundplans, and use them to render out per-frame query views. Our per- framframe loss consists of an image reconstruction term, hard surface constraint, and a sparsity term.
L | 11 + ALPIPS LUPIPS (R,
Asurface Σ log(P(w:)) + Asparso Elopl-
Cing
Cartes
5

---

Published as a conference paper at ICLR 2023
CLEVR (1 view)
CoSY (1 view)
CoSY (5 views)
PSNR f SSIM LPIPS
PSNR1
SSIMT
L
PSNR
SSIM LPIPS
Ours 34.5
0.956
0.15
15.71
0.43
0.53
18.29
0.57
0.43
PixeINeRF
33.98
0.945
0.200
14.61
0.34
0.64
17.31
0.49
0.50
uORF
29.35
0.898
0.151
Table 1: Quantitative baseline comparison of novel view synthesis results. We outperform Pixel- NeRF YYu et al.] 2020) and uORF (Yu et al.. 2022) in terms of PSNR, SSIM, and LPIPS on both CLEVR and CoSY datasets.
Ling measures the difference between the rendered and ground truth images, R and I respectively, using a combination of εg and patch-based LPIPS perceptual loss. uurfae encourages both static and dynamic weight values (the weight for each sample in the rendering equation) w, for all samples along the rendered rays to be either or 1, encouraging hard surfaces (Rebain et al.] 2022). Here, P(w;) (up(-|w,,) + exp(-l1 wyl). The sparsity term Edynspaasity takes input densities decoded from the dynamic groundplan for all the rendered rays, and encourages the values to be sparse. This forces the model to explain most of the non-empty 3D structure as possible via the static groundplan and only expressing the moving objects using the dynamic groundplan, leading to reliable static-dynamic disentanglement. Without this loss, the model could explain the entire scene with just the dynamic component. The loss functions are weighed using the hyperparameters ALPIPS. Asrriace and Asparee. While describe the loss functions for a single sample of ground-truth and rendered image, in practice, we construct matihe by randomly choosing multiple views of scene at different time steps, and evaluate the loss function on each sample.
Unsupervised object detection and extracting object-centric 3D representations. Our formulation yields a model that maps a single image to two radiance fields, parameterizing static and dynamic 3D regions respectively. Please see Fig. for example. We now perform a search for connected components in the dynamic neural groundplan to perform 3D instance-level segmentation, monocular 3D bounding box prediction, and the extraction of object-centric 3D representations. Specifically, given dynamic groundplan, we first sample points in a 3D grid around the groundplan origin and decode their densities. We now perform conventional connected-componet groundplan space using accumulated density values, identifying the disconnected dynamic objects. We perform 2D instance-level segmentation for queried viewpoint using volume rendering based on the densities expressed by the dynamic groundplan and assigning a color to the points corresponding to each of the identified objects, see Fig. for an example. Furthermore, we compute the smallest box that contains the connected component to get 3D bounding box for each identified object. Finally, we crop tiles of the dynamic groundplan that belong to a given object instance to obtain object-centric 3D representations, enabling editing of 3D scenes such deletion, insertion, and rigid-body transformation of objects. This approach is not limited to a fixed number of objects during training or test time. As we will show, this simple method is at par with the state of the art on self-supervised learning of object-centric 3D representations, uORF Yu et al. 2022)) Note that our approach is compatible with prior work leveraging slot attention (Locatello et al.) 2020 Kipf et al. 2021) and other inference modules, which can be run on the disentangled dynamic groundplan which, in contrast to image space, enables shift--quivariant processing free from perspective distortion and encodes 3D structure. For implementation details, refer to Appendix A
labeling in the
RESULTS
We demonstrate that our method infers a 3D scene representation from single image while disentan- gling static and dynamic components of the scene into static and dynamic groundplans respectively. We then show that connected components analysis suffices to leverage the densities in the dynamic groundplan for instance-level segmentation, bounding box prediction, and scene editing.
Datasets. Our method is trained on multi-view observations of dynamic scenes. We present results on the moving CLEVR dataset (Yu et al. 2022), commonly used for self-supervised object discovery benchmarks Yu et al. 2022). and the procedurally generated autonomous driving dataset CoSY (Bhandari 2018). CoSY enables generation of a high-quality, path-traces dataset of multi-view videos with large camera baselines. We rendered multi-view observations 9000 scenes with moving cars, sampled using 15 background city models and 95 models. We train 8000 scenes, and

---

Published as a conference paper at ICLR 2023
CLEVR
CoSY
Input
GT
pixeINeRF
uORF
Ours
Input
GT
PixeINeRF
Ours
Figure 4: Qualitative comparisons. Comparison for novel-view synthesis given single context view with PixeINeRF Yu et al. 2020) and uORF Yu et al. 2022).
Input
Reconstruction
Static
Object
Depth
Novel views
Figure 5: Single-image reconstruction, disentanglement of static and dynamic objects, and novel view synthesis. Given a single input image, our method can disentangle the observed scene into static and object components based on what the model observed as not-moving and moving in the training data. In these examples, the cars are isolated in the object component as the model was training on video data of cars moving on the road.
evenly split the rest into validation and test sets. Further details about dataset generation are presented in the Appendix A.6 Datasets and code will be made publicly available.
Novel View Synthesis and Scene Completion. We present novel views rendered from groundplans inferred from a single image from CLEVR and CoSY. For single-shot 3D reconstruction and novel view synthesis, compare against PixeINeRF Yu et al.) 2020), saate-of-the-e single-image 3D reconstruction method, and uORF (Yu et al. 2022). state-of-the-art unsupervised object-centric 3D reconstruction method. We train PiX models code. We finetune the uORF model pretrained on CLEVR on our CLEVR renderings, and train from scratch on CoSY using publicly available code. Fig. 4 provides qualitative comparison to PixeINeRF and uORF in terms of single-image 3D novel view synthesis on both CoSY and CLEVR. Note that our model produces novel views with plausible completions of parts of the scene that are unobserved in the context image such as the back-side of objects. As expected from a non-generative method, regions that are entirely unconstrained such as occluded parts the background (such as buildings) are blurry. While PixeINeRF succeeds in novel view synthesis CLEVR, renderings on the complex CoSY dataset show significant artifacts, possibly caused by the linear sampling employed by PixeINeRF. uORF does not synthesize realistic images when trained CoSY. Please refer to the supplemental webpage for results uORF on the CoSY (Appendix (.p5) On CLEVR, uORF generally produces high-quality renderings, but lacks high-frequency detail. In contrast
datasets using publicly available

---

Published as a conference paper at ICLR 2023
Birds-eye view localization
Instance level segmentation
3D Bounding-bon box segmentation
Input
Reconstruction
Birds-eye view
Figure Object Localization. Given a single input image, we can use our inferred 3D representation of the scene to compute (a) bird's s-ye view rendering, (b) object localization in the bird's-eye view space, (c) instance-level segmentation, and (d) 3D bounding box prediction.
Input
GT
PORF
Ours
Input
GT
uORF
Ours
ARIT
NV-ARI
RGB
ROB
Ours 0.84
0.84
uORF
0.83
0.82
Maskss
Masks
Quantitative segmen- tation accuracy evaluation on CLEVR using ARI and NV- ARI metries.
Table 2:
Figure 7: Qualitative Comparison for instance-level segmenta- tion. Qualitative comparison of instance-level segmentation result with uORF Yu et 2022).
these methods, our method reliably synthesizes novel views with high- detail for both datasets. Quantitatively, we outperform both methods novel-view synthesis in terms of PSNR, SSIM, and LPIPS metrics on both datasets (refer to Table D. Note that qualitatively, the performance gap to bascline methods is significantly larger than quantitative results would suggest. This due to the fact that much of the pixels used to compute PSNR are observing scene regions far outside the frustum the input view. Here, methods fail to reconstruct the true 3D appearance and geometry, as it is completely uncertain given the context view, resulting in low PSNR numbers for all methods (refer to Appendix B.I.. As can be seen in the qualitative results, our method achieves significantly better reconstruction quality in parts of the 3D scene that lie in the frustum of the input camera, these areas are occluded in the input view. Our method further succeeds at fusing information across multiple context views, increasing the quality of the renderings with increasing number of context views from varied viewpoints (refer to Appendix B.2).
Static-Dynamie Disentanglement. Given only a single image, our method computes separate static and dynamic groundplans that can be used to individually render the static and movable parts of the scene respectively. Fig. 5 shows results on single-image reconstruction of static and movable scene reliably encoded by the dynamic groundplan, and method inpaints
elements. Note that regions occluded in the input view.
Instance-level Segmentation and Bounding Box prediction. The separate reconstruction of movable scene components in the dynamic groundplan enables object detection via instance-level segmentation and bounding box prediction. Fig. 6 presents the instance-level segmentation and 3D bounding box prediction results of the proposed 3D object discovery via connected component discovery using the density inferred using the dynamic groundplan from the bird's-eye s-eye view. Fig. provides qualitative comparison of object discovery with uORF CLEVR dataset. While uORF succeeds at segmenting CLEVR scenes with fidelity comparable to ours, fails to provide reconstruc- tion and instance-level segmentation for diverse and visually complex street-scale CoSY dataset.
8

---

Published as a conference paper at ICLR 2023
Input
Reconstruction
Individual Objects
Deletion
Addition
Rearrangement
3
Figure 8: Object-centric representations and scene editing. The proposed static-dynamie neural groundplans enables object discovery using connected components labeling on the density of objects encoded in the dynamic groundplan (left). This enables straight- forward scene editing such as deletion, addition, or rigid-body transformation via directly editing the neural groundplan (right).
Our method reliably segments separate car instances and predicts the 3D bounding boxes, including for cars that are only partially observed. Table quantitiiat compares the computed segmentation maps on CLEVR uORF. We use the Adjusted Rand Index (ARI) metries following uORF. We evaluate this metric in the input view (ARI), as well as in novel view (NV-ARI), We perform at par with uORF on both of these metrics, demonstrating that our 3D ground plan representation reaches state of the art results with simple heuristics. Please refer to the supplemental webpage for video results (Appendix B. In addition, as mentioned before, we achieve higher-quality novel-view synthesis results, and also achieve significantly better results the challenging CoSY dataset. Since uORFin is based on slot attention, it can only attend to finite number of objects, whereas groundplans support any number of objects and require single forward pass render all objects.
Scene Editing. Instance-level segmentation, dynamic-static disentanglement, and 3D bounding boxes enable straight-forward 3D editing, such as translation, rotation, deletion, and insertion of individual objects in the scene. Objects can be rotated by arbitrary angles by simple bilinear interpolation of the groundplan features (refer to Appendix B.3). As the dynamic groundplan does not encode static scene regions such as the street below cars, can easily be moved from scene to another. Fig. provides scene editing results of our method. Note that such editing is difficult with methods that lack persistent 3D representation, such as PixeINeRF.
DISCUSSION
Limitations and Future Work. Although our method achieves high-quality novel view synthesis from single image, generated views are not photorealistic, and unobserved scene parts are blurry commensurate with the amount of uncertainty. Future work may explore plausible hallucinations of unobserved scene parts. Future work may further explore the use of more sophisticated downstream processing of the groundplan to enable, for instance, brsedd inference of object-centric repre- sentations (Locatello et al. 2020). Finally, we plan to investigate the combination of the proposed approach with bloed dynamics reasoning, which would negate the need for multi-view video.
Conclusion. Our paper demonstrates self-superviset learning of 3D scene representations that are disentangled into movable and immovable scene elements. By leveraging multi-view video at training time, we can reconstruct disentangled 3D representations from a single image observation at test time. We show the potential of neural ground plans as representation that may enable data-efficient solutions to downstream processing of the 3D scene, such as completion, instance-level segmentation, 3D bounding box prediction, and 3D scene editing. We hope that our paper will inspire future work on the of self-supervise neural scene representations for general scene understanding tasks.

---

Published as a conference paper at ICLR 2023
7 ACKNOWLEDGEMENTS AND DISCLOSURE OF FUNDING
This work is in part supported by DARPA under CW303 (Transfer, Augmentation and Automatic Learning with Less Labels) and the Machine Common Sense program, Singapore DSTA under DST000ECI20300823 (New Representations for Vision), NSF CCRI # 2120095, NSF RI #221 1259 the National Science Foundation under Cooperative Agreement 2019766 (The NSF AI Institute for Artificial Intelligence and Fundamental Interactions, http::/iaaiiiioooo.... http///aiiiiro/)...oo// Stanford Institute for Human- Centered Artificial Intelligence (HAI), Stanford Center for Integrated Facility Engineering (CIFE), Qualcomm Innovation Fellowship (QIF), Samsung, Ford, Amazon, and Meta. The Toyota Research Institute also partially supported this work. This article solely reflects the opinions and conclusions of its authors and not other entity.
REFERENCES
Zhipeng Bao, Pavel Tokmakov, Allan Jabri, Yu-Xiong Wang, Adrien Gaidon, and Martial Hebert. Discovering objects that can move. In Proc. CVPR, pp. 11789-11798, 2022.
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. Proc. CVPR, 2021.
Daniel Bear, Chaofei Fan, Damian Mrowca, Yunzhu Li, Seth Alter, Aran Nayebi, Jeremy Schwartz, Li F Fei-Fei, Jiajun Wu, Josh Tenenbaum, al. Learning physical graph representations from visual scenes. Proc. NewrIPS, 33, 2020.
Deniz Beker, Hiroharu Kato, Mihai Adrian Morariu, Takahiro Ando, Toru Matsuoka, Wadim Kehl, and Adrien Gaidon. Monocular differentiable rendering for self-supervised 3d object detection. In Proc. ECCV, pp. 514 529. Springer, 2020.
Nishchal Bhandari. Procedural synthetic data for self-driving cars using 3D graphics. PhD thesis, Massachusetts Institute of Technology, 2018.
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. arXiv preprint 19.iv:190..11333 2019.
Anh-Quan Cao and Raoul de Charette. Monoscene: Monocular 3d semantic completion. In Proc. CVPR, pp. 3991 1001, 2022.
Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry-awand aaware 3d generative adversarial networks. In Proc. CVPR, pp. 20123-16133, 2022.
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. arXiv preprint arXiv:2203.000 29030 2022a.
Chang Chen, Fei Deng. and Sungjin Ahn. Roots: Object-centric representation and rendering of 3d scenes. JMLR, 22:259-1, 2021.
Shaoyu Chen, Tianheng Cheng, Xinggang Wang, Wenming Meng, Qian Zhang, and Wenyu Liu. Efficient and robust 2e-to-bev representation learning via geometry -guided kernel transformer. arXiv preprini arXiv:2206.04550 2022h.
Ricson Cheng, Ziyan Wang, and Katerina Fragkiadaki. Geometry recurrent neural networks for active visual recognition. Proc. NewrIPS, 31, 2018.
Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convolutional neural networks. In Proc. AAAI, 2019.
Terrance Vries, Miguel Angel Bautista, Nitish Srivastava, Graham W. Taylor, and Joshua M. Susskind. Unconstrained scene generation with locally conditioned radiance fields. Proc. ICCV, 2021.
10

---

Published as a conference paper at ICLR 2023
Yilun Du, Shuang Li, Yash Sharma, B. Joshua Tenenbaum, and Igor Mordatch. Unsupervised learning of compositional energy concepts. In Proc. NeurlPS, 2021a.
Yilun Du, Kevin Smith, Tomer Ulman, Joshua B Tenenbaum, and Jiajum Wu. Unsupervised discovery of 3d physical objects from video. In Proc. ICLR, 2021b.
Emilien Dupont, Miguel Bautista Martin, Alex Colburn, Aditya Sankar, Josh Susskind, and Qi Shan. Equivariant neural rendering. In Proc. ICML, pp. 2761-2770 PMLR, 2020.
Cathrin Elich, Martin R. Oswald, Mare Pollefeys. and Joerg Stueckler. Weakly supervised learn- ing of multi-object 3d scene decompositions using deep shape priors. Computer Vision and Image Understanding, 220: 2022. ISSN 1077-3142. doi: http:///bi.orr//0.1006....iii 2022.10344 URL https: / www. sci encedi rect. com/ sci /nce///rtiicee/pi/
S107731 4222000583
Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Generative scene inference and sampling with object-centric latent representations. In Proc. ICLR, 2020. URL
SM Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari. Geoffrey E Hinton, et al. Attend, infer, repeat: Fast scene understanding with generative models. Proc. NeurIPS, 29, 2016.
SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Gar- nelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al. Neural scene representation and rendering. Science, (2006394):1204-110
2018.
Klaus Greff, Antti Rasmus, Mathias Berglund, Tele Hao, Harri Valpola, and Jürgen Schmidhuber. Tagger: Deep unsupervised perceptual grouping. Proc. NewrIPS, 29, 2016.
Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. Proc. NeurlPS, 30, 2017.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. International Conference on Machine Learning. 2019.
Michelle Guo, Alireza Fathi, Jiajun Wu, and Thomas Funkhouser. Object-centric neural scene rendering. arXiv preprint arXiv:2012.085033 2020.
Pengsheng Guo, Miguel Angel Bautista, Alex Colburn, Liang Yang, Daniel Ulbricht, Joshua M. Susskind, and Qi Shan. Fast and explicit neural view synthesis. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 3791-3800, January 2022.
Adam W Harley, Zhaoyuan Fang, Jie Li, Rares Ambrus, and Katerina Fragkiadaki. A simple baseline for bev perception without lidar. arXiv preprint wxXiv22006007950 2022.
Philipp Henzler, Jeremy Reizenstein, Patrick Labatut, Roman Shapovalov, Tobias Ritschel, Andrea Vedaldi, and David Novotny. Unsupervised learning of 3d object categories from videos in the wild. In Proc. CVPR, pp. 4700 2021.
Anthony Hu, Zak Murez, Nikhil Mohan, Sofía Dudas, Jeffrey Hawke, Vijay Badrinarayanan, Roberto Cipolla, and Alex Kendall. Fiery: Future instance prediction in bird's s-yee-ee view from surround monocular cameras. In Proc. CVPR, pp. 15273-15282, 2021.
Zhewei Huang, Tianyuan Zhang, Wen Heng, Boxin Shi, and Shuchang Zhou. Rife: Real-time intermediate flow estimation for video frame interpolation. arXiv preprint arXiv.2011.06294, 2020.
Wonbong Jang and Lourdes Agapito. Codenerf: Disentangled neural radiance fields for object categories. In Proc. ICCV, pp. -2958,, October 2021.
11

---

Published as a conference paper at ICLR 2023
Jacbong Jeong, Janghun Jo. Sunghyun Cho, and Jaesik Park. 3d scene painting via semantic image synthesis. In Proc. CVPR, pp. 2262-222227111122 June 2022.
Jindong Jiang, Sepehr Janghorbani, Gerard de Melo, and Sungjin Ahn. Scalable object-oriented sequential generative models. Unknown Journal, 2019.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: diagnostic dataset for compositional language and elementary visual reasoning, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2901-2910, 2017.
Yoni Kasten, Dolev Ofri, Oliver Wang. and Tali Dekel. Layered neural atlases for consistent video editing. Proc. TOG, 10 2021.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 1412.6900 2014.
Thomas Kipf, Gamaleldin F Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional object-centric learning from video. arXiv preprint arXiv:2111. 2021.
Adam Kosiorek. Hyunjik Kim, Yee Whye Tch, and Ingmar Posner. Sequential attend, infer, repeat: Generative modelling of moving objects. Proc. NewrIPS, 2018.
Adam R Kosiorek, Heiko Strathmann, Daniel Zoran, Pol Moreno, Rosalia Schneider, Soña Mokrá, and Danilo J Rezende. Nerf-vae: A geometry aware 3d scene generative model. Proc. ICMI., 2021.
Shamit Lal, Mihir Prabhudesai, Ishita Mediratta, Adam W Harley, and Katerina Fragkiadaki. Co- conets: Continuous contrastive 3d scene representations. In Proc. CVPR, pp. 12487-12496, 2021.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. In Proc. ICLR, 2020. URL /ttttss:/oppereeeeee net.eeewweeeeeeee.e.eeeeeeeeeeee www.://wewtefouumeeeem forummidumm rk 03ySYDH
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Proc. NeurlPS, 33:1 1525-11538, 2020.
Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural volumes: Learning dynamic renderable volumes from images. Proc. SIGGRAPH, 2019.
Kaustubh Mani, Swapnil Daga, Shubhika Garg, Sai Shankar Narasimhan, Madhava Krishna, and Krishna Murthy Jatavallabhula. Monolayout: Amodal scene layout from a single image. In Proc. CVPR, pp. 1689-1697, 2020.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In Proc. ECCV, 2020.
Thomas Neff, Pascal Stadlbauer, Mathias Parger, Andreas Kurz, Joerg H Mueller, Chakravarty R Alla Chaitanya, Anton Kaplanyan, and Markus Steinberger. Donerf: Towards real-time rendering of compact neural radiance fields using depth oracle networks. Computer Graphics Forum. Wiley Online Library, 2021.
Thu H Nguyen- Phuoc, Christian Richardt, Long Mai, Yongliang Yang, and Niloy Mitra. Blockgan: Learning 3d object-aware scene representations from unlabelled images. Proc. NeurIPS, 33: 6767-6778, 2020.
Yinyu Nie, Xiaoguang Han, Shihui Guo, Yujian Zheng, Jian Chang, and Jian Jun Zhang, To- tulBdunderstanding: Joint layout, object pose and mesh reconstruction for indoor scenes from a single image. In Proc. CVPR, pp. 55-64, 2020.
12

---

Published as a conference paper at ICLR 2023
Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes compositional generative neural feature fields. In Proc. CVPR, pp. 1 11453-1144, 2021.
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In Proc. CVPR, pp. 3504-3515, 2020.
Julian Ost, Fahim Mannan, Nils Thuerey, Julian Knodt, and Felix Heide. Neural scene graphs for dynamic scenes. In Proc. CVPR, 2021.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathicu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825-2830, 2011.
Songyou Peng, Michael Niemeyer, Lars Mescheder, Mare Pollefeys, and Andreas Geiger. Convolu- tional occupancy networks. In Proc. ECCV, 2020.
Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In Proc. ECCV, pp. 194-210. Springer. 2020.
Daniel Rebain, Mark Matthews, Kwang Moo Yi, Dmitry Lagun, and Andrea Tagliasacchi. Lolnerf: Learn from one look. Proc. CVPR, pp. 1558- 1567, 2022.
Lennart Reiher, Bastian Lampe, and Lutz Eckstein. A sim2real deep learning approach for the transformation of images from multiple vehicle-mounted cameras semantically segmented image in bird's eye view. In 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), pp. 1-7. IEEE, 2020.
Thomas Roddick, Alex Kendall, and Roberto Cipolla. Orthographic feature transform for monocular 3d object detection. In BMVC, 2019.
Avishkar Saha, Oscar Mendez, Chris Russell, and Richard Bowden. Translating images into maps. In Proc. ICRA, 2022. Utt.https:: /tmp::://www /www. zon. se www.inneepubliicaaaeea.eeeoe/eea trans peanslation aatiinggmmmsssss. intors---nttompppps
Mehdi SM Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lucic, Daniel Duckworth, Alexey Dosovitskiy, et al. Scene representation transformer: Geometry freee novel view synthesis through set-latent scene representations. Proc. CVPR, 1021.
Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhöfer. Deepvoxels: Learning persistent 3d feature embeddings. In Proc. CVPR, 2019a.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Con- tinuous -drstrrcture-aware neural scene representations. In Advances in Neural Information Processing Systems, pp. 1121-1132, 2019b.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Contin- uous 3d-structure-aware neural scene representations. In Proc. NeurIPS 2019, 2019c.
Vincent Sitzmann, Semon Rezchikov, William T. Freeman, Joshua B. Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-evaluation rendering. In Proc. NeurlPS, 2021.
Cameron Smith. Hong Xing Yu, Sergey Zakharov, Fredo Durand, Joshua B Tenenbaum, Jiajun Wu, and Vincent Sitzmann. Unsupervised discovery and composition of object light fields. arXiv preprint arXiv:2205. 03923, 2022.
Ayush Tewari, Justus Thies, Ben Mildenhall, Pratul Srinivasan, Edgar Tretschk, Yifan Wang, Christoph Lassner, Vincent Sitzmann, Ricardo Martin-Brualla, Stephen Lombardi, et al. Ad- vances in neural rendering. Proc. Eurographics STAR, 2021.
Alex Trevithick and Bo Yang. Grf: Learning general radiance field for 3d representation and rendering. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 15182-15192, October 2021.
13

---

Published as a conference paper at ICLR 2023
Vadim Tschernezki, Diane Larlus, and Andrea Vedaldi. Neuraldiff: Segmenting 3d objects that move in egocentric videos. In Proc. 3DV, pp. 910-919. IEEE, 2021.
Hsiao- Yu Fish Tung, Rieson Cheng, and Katerina Fragkiadaki. Learning spatial common sense with geometry-awarr recurrent networks. In Proc. CVPR, pp. 2595-2603, 2019.
Bangbang Yang, Yinda Zhang, Yinghao Xu, Yijin Li, Han Zhou, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui. Learning copeecccompoitioonll neural radiance field for editable scene rendering. In Proc. ICCV, pp. 13779-13788, October 2021a.
Bangbang Yang, Yinda Zhang, Yijin Li, Zhaopeng Cui, Sean Fanello, Hujun Bao, and Guofeng Zhang. Neural rendering in a room: Amodal 3d understanding and free-viewpoint rendering for the closed scene composed of pre-captured objects. arXiv preprint urXiv:2205.02714. 2022.
Weixiang Yang, Qi Li, Wenxi Liu, Yuanlong Yu, Yuexin Ma, Shengfeng He, and Jia Pan. Projecting your view attentively: Monocular road scene layout estimation via cross-view transformation. In Proc. CVPR, pp. 15536-15545, 2021b.
Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. Proc. NeurIPS, 2020.
Vickie Ye, Zhengqi Li, Richard Tucker, Angjoo Kanazawa, and Noah Snavely. Deformable sprites for unsupervised video decomposition. In Proc. CVPR, pp. 2657-2666, 2022.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one few images. Proc. CVPR, 2020.
Hong-Xing Yu, Leonidas J Guibas, and Jiajun Wu. Unsupervised discovery of object radiance fields. Proc. ICLR, 2022.
Wentao Yuan, Zhaoyang Lv, Tanner Schmidt, and Steven Lovegrove. Star: Self-supervised tracking and reconstruction of rigid objects in motion with neural rendering. In Proc. CVPR, pp. 13144- 13152, 2021.
Sergey Zakharov, Wadim Kehl, Arjun Bhargava, and Adrien Gaidon. Autolabeling 3d objects with differentiable rendering of sdf shape priors. In Proc. CVPR, 2020.
Sergey Zakharov, Rares Andrei Ambrus, Vitor Campagnolo Guizilini, Dennis Park, Wadim Kehl, Fredo Durand, Joshua B Tenenbaum, Vincent Sitzmann, Jiajun Wu, and Adrien Gaidon. Single-shot scene reconstruction. In Proc. CORL, 2021.
Maciej Zięba, Marcin Przewiezlikowski, Marek Šmieja, Jacek Tabor, Tomasz Trzcinski, and Prze- myslaw Spurek. Regflow: Probabilistic flow-based regression for future prediction. arXiv preprint arXiv:2011.146220 2020.
A IMPLEMENTATION DETAILS
In this section, we provide training details of our method to enable reproducibility.
A.1 NETWORK ARCHITECTURES
CNN Encoder. Following PixeINeRF, we use the first 4 convolutional blocks of the ResNet34 architecture to create feature grid. For a 128 x 128 resolution image, this results in feature grid of 64 x 64 resolution with 512 features per point. We further use a shallow CNN to reduce the number of feature channels, and to aggregate information along the y-axis. This step helps us reduce the memory footprint for further processing in 3D. This feature aggregation CNN has two convolutional hidden layers with 256 hidden units and stride of 2 along the height of the feature tensor. The output is a 16 64 resolution feature grid with 128 features per point. We use ReLU as the activation function for all layers in the encoder.
14

---

Published as a conference paper at ICLR 2023
Unprojection. We populate coarse 3D feature volume of shape 64 x 16 x 64 with 128 28-dimensional latents for points in the camera frustum by bilinearly sampling the aggregated encoder feature tensor. This volume is processed as explained in the main paper: coordinate encoding MLP transforms each feature into a new 128 dimensional feature vector. This MIP has two hidden layers with 128 hidden units each and ReLU activation after each hidden layer.
Multi-view input. In case of multi-view inputs, we average the unprojected latents for all 3D points in the canonical 3D coordinate system for all views. This results, like in the monocular case, in a 3D volume of 128 x 64 x 16 x 64 (CH x X x x x Z).
Aggregation along the height. To construct an entangled groundplan, we aggregate the 3D feature volume into a 2D groundplan using pillar aggregation MLP. Consider pillar of features orthogonal to the groundplane for particular point (z, z), in our case that is 128 x 16. The MLP takes in each latent with its corresponding 3D coordinate in the world coordinate system. It outputs the softmax scores for each of these latents which can be used to sum these latents along the pillar into single 228-dimensional latent, as explained in the main paper.
Disentanglement CNN. We use shallow CNN with 4 hidden convolutional layers to disentangle the entangled groundplan of shape 128 x 64 x 64. The first two convolutional layers have 128 hidden units with kernel size of 3, stride of 1, and reflection padding of
The last two convolutional layers comprise of 256 hidden units with the same configuration for other parameters. These are alsso followed by 2x bilinear upsampling layers. This shallow CNN outputs groundplan of shape 256 x 256 x 256, in which the first 128 channels of the feature tensor are attributed as the static groundplan and the rest of the 128 channels are used as the dynamic groundplan for that timestep.
Projections. Our method performs project of 3D point on the image plane to get the corresponding feature tensor. This is performed using the intrinsic matrix of the camera. For a given 3D point in camera coordinates X, and intrinsic matrix K, the resulting 2D point on the image plane is computed as Kx. For projecting a 3D query point x on the groundplan, we use grid sample using (zz, --oordiiisss of the 3D query points
Neural renderer. Similar to PixeINeRF, we use a MLP as a renderer with 4 hidden layers that have 128 hidden units. Our renderer and the input latent are significantly smaller than the ones used in PixeINeRF, making our rendering cheaper.
We use two rendering MLPs, for coarse and fine sampling with 256 samples for the coarse MLP and 128 samples for the fine rendering along with 32 samples at the predicted depth based on the coarse renderings.
Initialization. We use the kaiming normal initialization for initializing all the weights. All biases were initialized with uniform distribution in [-le 3 le 1).
A.2 HYPERPARAMETERS
We use Adam Kingma& & Ba 2014) with a learning rate of 3e 4 train our pipeline with the image reconstruction loss (L2), hard surfaces loss, and the alpha sparsity loss for 200 epochs. The losses were weighted by Ximg = AnsI = 0, and Asgurse = 0.01. The model was then further finetuned by adding the LPIPS loss weighted by Appip 0.5. We found that using LPIPS loss from the beginning of the training process made the training unstable. We sampled 1e rays to compute the loss for each training sample in the input batch in the initial phase. During the first phase of training, the rays are sampled randomly and in the second phase when LPIPS loss is applied to the training, we sample rays to render image patches of 16 X 16. LPIPS loss using VGG is applied to these patches by normalizing the range of the output RGB values be between [-1, 1]. The model trained on a single 32G V100 GPU with a batch of input samples with 2 timesteps for N views (N=5).
A.3 CURRICULUM TRAINING
Our model supports multi- well monocular reconstruction. We start our training only using multi-view reconstruction with 5 input views for the first 200 epochs, and then switch to a variable mode where the model is given varying number of input views ranging between 1-5 views. This
15

---

Published as a conference paper at ICLR 2023
curriculum approach helps the network to first learn the relevant scene priors, before learning to complete the 3D structure of the scene.
A.4 HEURISTICS FOR LOCALIZATION
For localization of objects, we consider the dynamic groundplan and render it from an orthographic bird's-eye view. Along with the bird s-eye view RGB image, we also generate the occupancy map (per-pixel accumulated alpha) as shown in the manuscript in Fig. 5. Since the groundplans have spatial extent of 256 x 256, the pphic occupancy map is rendered at the same resolution. The hard-surfaces loss encourages densities to be close to either 0 or 1. Thus, we threshold the density map with a threshold of 0.9. We find the regions of the map with connected components in the thresholded density map using label and regi onsprop functions from the sklearr. Pedregosa al.) (201T). To remove any remaining artifacts, we only keep the regions which have an area larger than 6 in the pixel space of the groundplan. This is a hyperparameter based on the size of the objects in the scene. Given the localization in the orthographic bird's s-eye view, we can find the height of the objects by computing the depth within each region. This information gives us 3D bounding box around each of the localized object. The instance level segmentation produced via volume rendering, by overriding the RGB values within the detected region to chosen color for the object. These predictions can be made more accurate, for example, by (1) further tuning these hyperparameters, (2) sampling the orthographic density map at higher resolution, (3) increasing the resolution of the groundplans at training time, and (4) training with more samples per camera ray.
A.5 SCENE EDITING
Scene editing is performed by editing the dynamic groundplan. Once the different objects have been localized in the dynamic groundplan, we edit the dynamic groundplan carry out object deletion, insertion, and rearrangement. Localizing an object gives us its features in the spatial region of the groundplan used for rendering. To delete the object, we replace the features in this regions with features from the dynamic groundplan that encode zero density. For inserting an object at given location, we find the corresponding (z, 2) location in the dynamic groundplan and set the features at that location to the features corresponding to the object. Rearrangement of objects can be seen as a combination of deletion and insertion where we first delete the object from the existing location, followed by inserting the object at the new location in possibly new orientation. To perform rotation of an object, we rotate the patch of features that correspond to the object to be rotated.
A.6 DATASETS
CLEVR, We generate scenes using the default configuration from (Johnson et al. 2017) (BSD License ). using the rubber material and default object sizes. We render the scene with different cameras, all at the same fixed distance from the origin as the camera used in CLEVR. but with azimuth angles increasing at 60 degree increments. Objects in the CLEVR dataset are captured at timesteps, and are simulated to move distance of 0.25 to 0.75 meters between the two timesteps. Images were rendered across 6 different cameras, with resolution of 128 x 128 using CYCLES renderer with 512 samples per pixel. Our dataset consists of 1500 samples, divided into 1000 train and 500 test samples.
CoSY. We develop this dataset using the city generation code provided by Bhandari (018). We generate 15 different configurations of the city using CityEngine, with variations in building shapes, heights, and materials. This city layout is further processed in Blender using Python to add cameras, trees, bus stops, and moving cars. Cameras are sampled on hemispheres of radii in range [4-6m] maximum height of 4m viewing the center of the circular base of the hemisphere located at randomly chosen points. We sample 15 cameras for each of the randomly chosen centers, with the varying radius of the hemisphere for each camera. The field-of-view for all sampled cameras is 60 degrees, with symmetric sensor size of 32mm, resulting in a focal length of 1 110.85 in pixel space. Note that we never sample any bird's-eye view as the maximum height of the camera is clipped at 4m. We choose 50-100 cars to be spawned in different locations of the generated city. We environment map with diffused white light. In addition to the rendering capabilities of CoS Y, we add
fixed
http:///tttheooooeoooooeeeeeeeee ICENSE
16

---

Published as a conference paper at ICLR 2023
Context images
Views around the objects
GT
Pred
GT
GT
Pred
GT
Pred
GT
Pred
Figure 9: Visual quality of views for a given input image.
the ability to move cars over timesteps. Given the location and direction of the movement (direction where the is facing), change the location of the over 10 timesteps translation from range 2-4 m. For each sampled camera, we render the 10 frames with a resolution of 128 128 using CYCLES renderer with 512 samples per pixel. Our dataset has 9000 such samples, which are further divided into 8000 train, 500 validation, and 400 test samples. Thus each sample has images, poses, focal length, and principal point for 15 cameras. The dataset will be publicly released for further research in this direction.
sampled total
B ADDITIONAL RESULTS
B.1 VISUALIZATION OF VIEWS OUTSIDE THE OBSERVED VIEW
Fig. 9 presents the output renderings for various target camera viewpoints for the given input image. We observe that as the target view shifts away from the context view, our method renders the geometry with appropriate texture of the car in the view, but outputs blurry background as expected from non-generative model.
B.2 UNCERTAINTY WITH RESPECT TO THE NUMBER OF INPUT VIEWS
Fig. I0 provides results of street-scale scenes reconstructed using an increasing number of input images from different camera viewpoints. Our method successfully integrates information across observations into single, multi-view consistent representation. The background reconstructions improve with more input views, result of less uncertainty for the occluded regions.
17

---

Published as a conference paper at ICLR 2023
Input context views
Reconstructions 12233
Groud truth
1,2
1,2.3.4
12334
Input context views
Groud truth
1,2
Reconstructions 1 1,,3
1,2,344
1,2,3,455
Figure 10: Resolving uncertainty with increasing number of context views Our model can take variable number of viewpoints scene as input. We find that the model gets good prior on the object from a single view but struggles to reconstruct high-qualis details in the unobserved regions of the background. This uncertainty gets resolved with more input views.
18

---

Published as a conference paper at ICLR 2023
Figure 11: Scene editing with rotated objects. We present images of scenes with edited groundplans where cars are rotated at various angles in the ground plane.
Apart from effect of uncertainty in the unobserved regions the visual quality, there are two factors that contribute to blurriness in the results. First, our method is in the regime of prior-based reconstruction from a few images. This is in contrast the regime of single-scene overfitting (e.g. NeRF) in which prior work has demonstrated photo-realistic results. In our task setup of prior based reconstruction, a certain degree of uncertainty exists. For instance, there uncertainty about the exact depth and geometry of the 3D scene given the input images. In these cases, the model will learn to blur proportionate to the amount of uncertainty. We note that this is not limitation of our method specifically all prior-based 3D reconstruction methods share this property. We outperform pixeINeRI, a strong baseline this regime of 3D reconstruction from few images, both quantitatively and qualitatively. Second, rendering quality is limited by computational cost. In contrast to single-scene methods, our method needs to fit not only the differentiable rendering in GPU memory, but also the whole inference pipeline CNNs, 3D lifting, groundplans, etc. This limits the resolution of the groundplans (a car is expressed by ~ 6 latents) as well as the number of volume rendering samples. Increasing computational budget would lead to better renderings.
B.3 ROTATING OBJECTS
Fig. 11 provides rendered images from edited groundplans where the cars are rotated at different angles. Note that the representation only allows for rotations in the zz-planc.
B.4 MOTION INTERPOLATION
The dynamic groundplan be further used perform motion interpolation in 3D, using optical flow for groundplan interpolation prediction using off-the-shelf, state-of-the-art optical flow and frame interpolation methods. To demonstrate the efficacy of the groundplans for frame interpolation. we trained RIFE Huang et al. (2020) on our model trained on the GQN-rooms Eslami et al. (2018) dataset. We generated simple linear motion trajectories for objects over 10 frames. We added tall static cylinders as pillars in the room which generate occlusions. The scene was rendered from 15 different camera views. We first trained our model on the GQN dataset and used the output dynamic groundplans to train the frame interpolation method. The training was done in 2 steps. Firstly, we extracted dynamic groundplan for the samples over different timesteps by passing the multi-view observation through our groundplan generation pipeline. Two dynamic groundplans at different timesteps were given as input to RIFE, and an intermediate timestep was queried. An L2 loss the output dynamic plan against the output of pipeline for that timestep was sufficient to obtain a good initialization. In the training stage, we combined the RIFE model with our method for higher-quality results. All losses discussed the main paper were applied on the rendered novel views generated using the output dynamic floorplans on the intermediate timesteps. In Fig. 12, we show the rendered output of our model for the intermediate timesteps given the leftmost and rightmost frames as input. The proposed method succeeds at inferring the correct object motion, and enables novel view synthesis through space and time.
B.5 SUPPLEMENTAL WEBPAGE
We strongly encourage the readers to refer to our supplemental webpage for more novel-view synthesis, static-dynamic disentanglement, localization, and scene editing results, as well as video comparisons with the state of the art.
19

---

Published as a conference paper at ICLR 2023
Input 1=U
Predicted Intermediaaee frames
Input t=1
Figure 12: Temporal 3D scene interpolation for novel view synthesis in space and time.
20


Reviews:
[{"review_id": "r8Ceyd3COXu", "reviewer": "Reviewer_KVRy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "An approach is presented in this paper for mapping 2D image observations to persistent 3D scene representations in order to create a novel view synthesis and disentangle the movable and immovable components of the scene. A ground-aligned 2D feature grid based on conditional neural groundplans can be employed as a memory-efficient representation of scene data. This notion is inspired by the bird's-eye-view (BEV) representation commonly used in vision and robotics. Self-supervised learning is conducted using differentiable rendering from unlabeled multi-view observations, and the method learns to complete the geometry and appearance of occluded regions.\n", "summary_of_the_review": "The method is good and the idea is interesting. But only synthetic data is used for evaluation, and the access of real data should not be a problem.", "strength_and_weaknesses": "Strength：\n1. The conditional neural groundplans proposed are intriguing. This allows efficient processing of scene appearance and geometry directly in 3D, through the self-supervised learning of conditional neural groundplans, a hybrid discrete continuous representation of 3D neural scenes.\n2. The proposed disentanglement of static background and movable foreground objects is novel. This approach uses object motion for training an encoder-based system that can reconstruct scenes from a single image instead of decomposing video frames based on scene specifics.\n3. The proposed method is promising for understanding 3D from a single image. The dynamic groundplan encodes 3D geometry, allowing us to segment and animate objects in 3D using single images and 3D bounding boxes.\n\nWeaknesses：\n1. The biggest weakness is that only results on simple synthetic data are presented. Currently, there are a number of well-annotated 3D driving scenes available. An example is KITTI 360 (https://www.cvlibs.net/datasets/kitti-360/). Could the authors explain why results on the real datasets are not presented? Is it because of some specific annotation (such as BEV correspondence or the number of views) needed by the method? Otherwise, the generalizability on natural scenes and the practical value of the method are questioned.\n2. I think there are some ambiguity behind the proposed Static-Dynamic Disentanglement. A definition of Static and Dynamic should be made clear in the proposed framework. A car remaining still in the sequence will be considered as Static. Considering a situation when two moving and static cars looks similar or exactly the same, how can we distinguish from a single image if a car is static or dynamic? The method should mention the above cases as an assumption, or making a clear definition of being Static and Dynamic.\n3. Groundplans implicitly assume that objects are aligned on a plane. However, in real life, it is common that scenes/objects are not placed on a flat ground plane (e.g., a curved plane). The ability of persistent 3D scene representation is a little overclaimed, if we consider the implicit assumption behind groundplans.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "W4gqrRhkSD", "reviewer": "Reviewer_5tSi", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed a method that maps a single 2D image of a scene to a persistent 3D scene representation, enabling novel view synthesis and disentangled representation of the movable and immovable components of the scene. The enabler is a new representation called conditional neural groudplans, motivated by the Birds’-eye-view representation. Because of the ability to separate the movable and immovable components of the scene, the representation enables a variety of downstream tasks, such as extraction of object-centric 3D representations, novel view synthesis, instance-level segmentation, 3D bounding box prediction, and scene editing. The authors conduct experiments on CLEVR and CoSY. The proposed method achieves state-of-the-art novel view synthesis compared with PixelNeRF and uORF on CLEVR. In addition, the proposed method obtain favorable instance segmentation performance compared with uORF on the CLEVR dataset.", "summary_of_the_review": "The work presents a novel 3D persistent scene representation framework and demonstrates impressive novel view synthesis performance and scene understanding tasks such as instance segmentation and localization. The reviewer found the work novel and has the potential to stimulate the community to study the direction collectively. However, there are several concerns mentioned in the Weaknesses section. The reviewer would like to get feedbacks from the authors.", "strength_and_weaknesses": "Strength:\n\n1. The authors tackle a challenging yet valuable problem, disentanglement of the movable and immovable components of the scene from a single image. They propose a neural scene representation called conditional neural groundplans and a self-supervised framework to disentangle static background and movable foreground objects given only a single image. The framework enables a variety of downstream tasks, such as extraction of object-centric 3D representations, novel view synthesis, instance-level segmentation, 3D bounding box prediction, and scene editing. The direction would interest a wide range of communities, including 3D scene understanding, 3D neural representation, disentangled representation learning, and novel view synthesis.\n2. The proposed 3D persistent scene representation learning framework elegantly leverages ideas from BEV representation, neural representation learning, volume rendering, and compositing operation (Yuan et al., 2021). Moreover, the proposed method utilizes multi-view videos (in a self-supervised manner) at training time to learn to separately reconstruct static and movable components of the scene from a single image at test time. The authors present a self-supervised neural scene representation framework for general scene understanding tasks. The reviewer found the work valuable to the community.\n3. The proposed 3D scene representations enable novel view synthesis and disentangle movable and immovable components of the scene via diverse qualitative evaluation on two datasets, CLEVR and CoSY. In particular, the qualitative results shown on CoSY are impressive, as shown in the manuscript and supplementary website.\n\nWeaknesses:\n\n1. Persistent scene representation: By sufficient qualitative analysis, the proposed 3D scene representations enable novel view synthesis and disentangle movable and immovable elements for different scene understanding tasks. One failure case shown on the top row of p.17 is that the framework cannot effectively synthesize the viewing perspective behind the box with a green door. The framework seems unable to capture the background correctly, yielding incorrect synthesis. Moreover, the box's location seems incorrect from the images on the left. The reviewer wonder if the performance can be improved with additional observations.\n2. The authors propose to have two separate groundplans neural representations for static and dynamic, respectively. As we know from the literature (e.g., Yuan et al., 2021), it is possible to keep separate 3D neural representations for novel view synthesis. It seems reasonable to tackle the goal of this paper, i.e., achieve decent performance on a variety of tasks from a single image, with separate 3D neural representations based on the proposed framework. It would be great for the authors to elaborate on the technical challenge of this direction. That could motivate the community to tackle the direction collectively.\n3. Differentiable rendering based on neural groundplans feature: the reviewer found it less intuitive to understand why the differentiable rendering would work using the groundplan features g_x. Specifically, g_x is obtained based on the weighted sum of the features along the y-axis. The groundplan feature g_x contains information in a particular pillar. Why is it a better feature for volume rendering? Empirically, the design choice renders better quality than STOA methods (i.e., PixelNeRF and uORF). Could the authors please elaborate more on this design choice?\n4. The design choice of BEV representation: The authors utilize a particular operation to form BEV representations, i.e., the weighted sum of the features. However, there are other approaches, e.g., Roddick and Cipolla, 2020, Philion and Fidler, 2020 and Saha et al., 2022. It would be valuable for the audience to learn the difference between these operations for representation learning.\n5. The reviewer is interested in the qualitative performance of novel view synthesis without using the contracted coordinate. With the analysis, readers would be able to visualize the difference.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "TvtqcBx7Pn", "reviewer": "Reviewer_U5HL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper describes a 3D scene representation built from 2D observations. \n\nThe representation allows the decomposition of static and movable objects in the scene, and can support tasks including novel view synthesis, 3D instance segmentation, and object-level scene editing. The authors propose a self-supervised training method for learning such a representation. \n\nEvaluations are conducted on two synthetic datasets. ", "summary_of_the_review": "The major highlights of the paper are the use of multi-view videos as self-supervision and, consequently, the ability to decompose static/movable objects to support segmentation and editing. The results are promising, although they appear to be somewhat preliminary. \n\nMy primary concerns are around generalization and the lack of evidence on how the approach performs on real, non-synthetic images. There are also some details requiring clarifications that should be easy fixes. ", "strength_and_weaknesses": "Strengths include: \n* The self-supervised training removes the need for GT annotation, e.g., object bounding boxes, for the supported downstream tasks. \n* The decomposition of static and movable object feature spaces looks like quite an elegant solution to utilize the inductive bias in multi-view videos while enabling capabilities such as segmentation and object-level scene editing. \n* The results overall look promising. Single-view/few-view novel view synthesis is a very challenging task, and the comparisons with PixelNeRF and uORF clearly demonstrated the advantages in synthesis quality. The analysis of adding more views (Fig.10) is strong evidence that the learned 3D representation can figure out how 2D views are related to, and hence help improve, 3D reconstruction. \n\nWeaknesses include:\n* There is no study of the generalization capability of the model. It seems likely that the learned models can overfit the limited appearances in the dataset (a few types of shapes/objects with mostly uniform colors). \n* Related to the last point, the evaluation is completely done on synthesized datasets, and it's not clear how well the model can learn real-world data with much more diverse appearances. Since the method does not require GT annotations, some evaluation on real-world data does not seem too difficult. \n* The are some quality issues with the presented view synthesis results. For example, while unobserved regions are understandably blurry, why are observed regions also not sharp? We can see from Fig.10 that adding more views does not help the quality of observed regions any further. \n* Training requires multi-view videos. I can't find clarification on whether these need to be from static cameras. Nothing in the model appears to have such a requirement, but if so, that's a strong limitation. Moreover, why can't the model be trained on frames from a single moving camera? \n* It's not clearly described how scene editing is done. It says that's enabled by \"directly editing the neural groundplan\". Does that mean editing the (movable object) feature space? The feature space is not sparse (i.e. locations without objects have non-zero features), so how does cropping in such space work, and why can that make sense? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GRCll88e8gz", "reviewer": "Reviewer_eeHh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a bird’s eye-view (BEV) representation for a 3D scene. The representation is disentangled into the static part and the dynamic part by a 2D CNN. The authors train their representation network on multiple dynamic videos with multiple views. When given a new image, the network can generate the implicit 3D representation and decompose the scene automatically. Differential rendering is used to generate novel view images. The authors conducted experiments on two datasets and showed impressive results.", "summary_of_the_review": "Overall, the paper proposed a new representation for the neural rendering, and is novel to me. However, the clarity of the paper needs further improvement. The reviewer looks forward to the author feedback and will consider increase my rating if the discussion can address my questions.\n\n=====\nThe authors addressed most of my concerns during the discussion and I would increase my rating.", "strength_and_weaknesses": "====> Strength\n1. The authors introduced the bird’s eye-view (BEV) representation as a scene representation, which is novel in the area of neural rendering.\n2. The disentanglement of the static and moving component of the scene adds more contribution to the proposed method. Although it is not new to automatically separate these components in the neural rendering community, using a vanilla 2D CNN to separate the feature representation is new to the reviewer.\n3. The authors show that using a simple heuristic method, they can generate instance segmentation and bounding boxes for the dynamic components in the scene.\n4. Good qualitative results are given.\n\n====> Weakness\n1. The details of the neural groundplans is confusing: \n\n    i. In page 4, there are multiple $x$ such as $x'$, $x$, $x_c$, $x_i$. Their shape should be clarified. Whether they belong to 2D or 3D point, and also their coordinate system (i.e., the image frame coordinate or the world coordinate, and how to transform between these coordinates using the camera intrinsic)\n\n    ii. For equation like $v(x ′ ) = F(π(C^{-1}(x ′ )))$, $f(x ′ ) = ...$, the parentheses here denote indexing the feature volume $v$ and feature tensor $F$. However, it is easy to confuse them with the functions like $\\pi()$ and $D()$. The authors should consider using other formulation.\n\n    iii.The $\\pi(\\cdot)$ is not clear. What is the meaning of it and how to learn $\\pi(\\cdot)$, is it learned per image I?\n\n    iv. ``$g_x$ are the groundplan features for the point $x$ computed by projecting the coordinates onto the groundplane...'' How is the projection conducted? Is it projected by the camera extrinsic？\n\n    v. In page 5, the authors say ``The per-frame static groundplans are pooled to obtain a single, time-independent static groundplan.'' How is the pooling process performed? The authors need to detail it.\n\n2. The task setting is not clear. The authors should consider formulate the problem setting in a separated subsection (e.g., train one model for multiple scenes, all images are calibrated, the view density per scene, etc.)\n\n3. It seems the compared baselines like pixelNerf and uORF are not proposed for the topic of separating dynamic and static representation. It would be fairer to compare with ``Neuraldiff: Segmenting 3d objects that move in egocentric videos.'' that used different components for the moving and static objects.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster