SGAM: Building a Virtual 3D World through Simultaneous Generation and Mapping
Authors: Yuan Shen
Wei-Chiu Ma
Shenlong Wang
Venue: NeurIPS 2022 Accept (2022)
Keywords: 3D generation
mapping
view synthesis

Abstract: We present simultaneous generation and mapping (SGAM), a novel 3D scene generation algorithm. Our goal is to produce a realistic, globally consistent 3D world on a large scale. Achieving this goal is challenging and goes beyond the capacities of existing 3D generation or video generation approaches, which fail to scale up to create large, globally consistent 3D scene structures. Towards tackling the challenges, we take a hybrid approach that integrates generative sensor model- ing with 3D reconstruction. Our proposed approach is an autoregressive generative framework that simultaneously generates sensor data at novel viewpoints and builds a 3D map at each timestamp. Given an arbitrary camera trajectory, our method repeatedly applies this generation-and-mapping process for thousands of steps, allowing us to create a gigantic virtual world. Our model can be trained from RGB-D sequences without having access to the complete 3D scene structure. The generated scenes are readily compatible with various interactive environments and rendering engines. Experiments on CLEVER and GoogleEarth datasets demon- strates ours can generate consistent, realistic, and geometrically-plausible scenes that compare favorably to existing view synthesis methods. Our project page is available at https://yshen47.github.io/sgam.

SGAM: Building a Virtual 3D World through Simultaneous Generation and Mapping
Yuan Shen'
Wei-Chiu Ma² Shenlong Wang'
University of Illinois at Urbana-Champaige "Massachusetts Institute of Technology {yshen47, chenlong)@i111 @illinois edu weichium@mit.c edu
Abstract
We present simultaneous generation and mapping (SGAM), a novel 3D scene generation algorithm. Our goal is to produce realistie, globally consistent 3D world on a large scale. Achieving this goal is challenging and goes beyond the capacities of existing 3D generation or video generation approaches, which fail scale up to create large, globally consistent 3D scene structures. Towards tackling the challenges, we take hybrid approach that integrates generative sensor model- ing with 3D reconstruction. Our proposed approach is an autoregressive generative framework that simultancously generates sensor data at novel viewpoints and builds a 3D map at each timestamp. Given an arbitrary camera trajectory, our method repeatedly applies this generation-and-mappe allowing us to create a gigantic virtual world. Our model can be trained from RGB-D sequences without having access to the complete 3D scene structure. The generated scenes are readily compatible with various interactive environments and rendering engines. Experiments on CLEVER and GoogleEarth datasets demon- strates ours can generate consistent, realistic, and geometrically-plasii scenes that compare favorably to existing view synthesis methods. Our project page available https://ysheen/..gitbbboo//////
process for thousands of steps,
1 Introduction
Human perception goes way beyond simple recognition and reconstruction. Our extraordinary abilities not only allow us to make sense of what we see, but also enable us to imagine what we do not (e.g., reason what the scene looks like outside the image). With simple glance, we can effortlessly re-build a mental world, that may not exactly like the original, but is perceived by our brain to be the same. In fact, it is such profound ability drawing apart from existing AI systems.
The goal of this paper is to equip computational machines with similar capabilities. We aim to develop program that can create an arbitrarily large, realistic, and consistent 3D world from single snapshot of the scene. The task is of paramount interest many applications in computer vision (60) 40 74), computer graphics (65) (65), geography (5), and robotics (13) 44), since it unlocks
numerous potentials. For instance, may allow us to build an interactive virtual environment without any costly and laborious 3D modeling pipeline (12).
Indeed, there has been consistent pursuit within the computer vision and graphics community in the 35
past few decades (17 1o
65
D
68 (72), where people attempt to design algorithms and models
that are capable of extrapolating and hallucinating the world from a single input image. Unfortunately, a large body of efforts have been focusing on modeling in the 2D image space (7
[
85 I8 37 9).
or require external 3D assets and manually specified rules for restricted 3D generation 10)
Recently, with the advent of deep generative models 76, 40 61 60). researchers have made great strides in unconstrained 3D syntheses. However, due to the highly complex structure of the task, these approaches mostly focus on object-centric scenarios, or generating small-scale environments.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Figure 1: A large-scale virtual world created after 2000 SGAM steps. SGAM is an auto-regressive generative framework that simultaneously generates sensor data at novel views and builds a 3D map.
With these problems in mind, present Simultaneous Generation and Mapping (SGAM), novel 3D scene generation algorithm. SGAM builds upon insights from state-space representation in well as recent generative models (73 I8 66.. At its core lies two key modules: (i) a generative sensor module that takes a scene representation and a novel query viewpoint as input and renders a photo-realistic RGB-D image of a 3D scene; and (ii) mapping module that exploits the newly generated RGB-D observation at given camera pose and updates the scene representation. Given an arbitrary camera trajectory, our method iteratively applies this procedure construct the 3D scene. By grounding scene generation with "a map", we can produce a realistic and spatially consistent 3D world at a significantly larger scale than any previous 3D generation effort without drifting or collapsing. Additionally, our generated scenes can be represented explicitly as textured 3D models, making them readily compatible with various interactive environments and rendering engines. We note that we are not the first to tackle the task of scene synthesis under large viewpoint differences (56 60% 40.. Several recent efforts have demonstrated promising results in this direction, from which our method is inspired. Nonetheless, these works do not produce consistent 3D scene representation of the infinite world and potentially suffer from mode collapse (60) 40).
SLAM 48 23)
We validate the efficacy of SGAM on two large-scale 3I) scene datasets, Clevr-infinite and GoogleEarth. Both benchmarks produce substantially larger 3D scenes than existing datasets. This allows us to benchmark large-scale 3D scene generation. We evaluate the standard metrics for perceptual image quality, such as PSNR, SSIM (75), LPIPS benchmark the realism of the generated 3D scenes in terms of Lensen-Shannor divergence and maximum- mean mean discrepancy (22). Experimental results suggest that 1) our method produces more realistic results than existing aangl-image novel view synthesis methods; 2) our approach generates a more meaningful 3D world than the prior perpetual view generation algorithms.
IS (63). and FID (7). We further
2 Related Work
Image generation: How to synthesize an image realistically has been a long-standing problem in computer vision. The task dates back to 60s (2 4 where researchers attempted to generate textures by matching statistics (42 26 Through parametric sampling (85) or -arramamtt matching (17 (T.). they were able to synthesize an infinite amount of high-fidelity texture images. Unfortunately, these approaches fall short when applied to natural images, since the images have much
higher complexity. Recently, with the help of deep generative models 21 35 have demonstrated promising results on generating photo-realistic images 30 33 34). With proper
55|
28
66), researchers
design and inductive biases 773 18 [T] they are even able to scale the output to mega-pixel level
top of VQ-GAN (8) However, instead of treating scene generation as a pure 2D task as in the past, we explicitly consider the relationship between appearance and geometry both at the input level and the output level. By encoding both information into the quantized codebook, we are able to "grow" the 3D scene from an initial seed image and generate boundless world, which is not exactly like the original, but will be perceived by humans to be the same region.
77 (.. In this paper, we build our generation module
3D Generation: 3D modeling and synthesis have been an active yet challenging research problem for decades @ l 68). Recently, with the development of image generation techniques (2 335 57).
the field has experienced a rapid growth 77. 49 59). Drawing inspiration from its corresponding
2D analogue, researchers have been able to generate high quality point clouds (38 79 7. voxels
119 77), meshes (24 20), etc. Unfortunately, since 3D solution space much more intricate than that of 2D, these approaches are typically object-centric; also, they mostly focus generating common

---

Graphics compatible
Input observation
SGAM: Simultaneous Generation and Mapping
3D scene
Generative Sensing p(x,)/Me1::5)
X
X
Mi
mnn M &.
Mapping M.
Figure 2: Overview of our approach. Our method consists two steps. 1) a generative sensing step (g(x,MM)11 which renders realistic and complete sensor observation given an (incomplete) scene representation M, and a query viewpoint Er: 2) a mapping module pe (MMMMMMML Xt, Et). which integrates the new sensory observation to update the scene. Given an initial snapshot and camera trajectory, we repeatedly run this generation generation-mapping process to create the virtual 3D world.
objects in our daily lives
15). To enable scene-level synthesis, researchers have sought to incorporate more structures into the generation pipeline 447
54
51
53), such reducing the output
space from full 3D to predefined, compact representation (52) 46 67. While these strategies greatly alleviate the issue, the generated scene scale is still rather limited (e.g., indoor environment). In this work, we push the boundary of 3D generation systems and present model that is capable of generating coherent 3D scene structure over thousands of meters. The produced high-quality 3D world not only allows us to build large-scale map, but also provides geometric grounding for visual appearance synthesis.
Generative sensor models: Generative sensor models serve critical roles in graphics/robotics simulation systems (44 IC.. Their goal is to faithfully simulate the sensor measurement in real world. Depending the sensor modalities (e.g., rgb/depth cameras, LiDAR scanner), the output can either photometric or geometric quantities. These methods usually assume an underlying (proxy) 3D geometry is given (80, (59) and then perform conditional generation (45). The geometry can take various forms (e.g. point clouds (2), surfels (80) 44), meshes (58), etc), as long as it can effectively ground the system. Our full model is combination of generative sensor module and mapping module. The former takes as input RGBD images and simulates both the appearances and the geometry of the scene.
Single image novel view synthesis (NVS): Our work is also related to single image NVS ap- proaches
76 29), whose goal to simulate novel views of a scene using only visual cues from one input image. These methods typically rely on neural networks to learn statistical priors from data. Together with carefully designed scene representations (e.g., radiance fields (8I 744) multi-plane images 7T. layered depth images 772 64). etc), they are able to synthesize scene appear- ance with mild viewpoint change. To push the frontier even further, more recently, researchers have proposed to go beyond local view synthesis, and instead simulate the
under larger viewpoint change (60) 36 61 56). For instance, generate one room from the other. The task extremely challenging it requires the model to learn not only the underlying scene representationss, but also the long-ram range dependencies. Without proper state/map representations, the generated output will easily drift away and the constructed world will longer be coherent 440 61). To address this issue, in this work, we develop a model that simultaneously generates novel views and builds a map. By integrating synthesized results into the map, re-visit a spot and then decode, allowing us to produce realistic and consistent video footage.
able query relevant information whenever
3 SGAM: Simultaneous Generation and Mapping
In this paper, we seek to devise method that can generate an arbitrarily large, realistic, and consistent 3D world from single snapshot of the scene. Based on the observation that current 3D scene generation approaches prone to drifting and often produce inconsistent 3D worlds, we present
3

---

GT
GFVS-Explici (61
GFVS-Implicit 66
Ours
Figure 3: Randomly sampled scenes from CleverInfinite dataset.
Frame
Frame 2
Frame 450
Frame 900
Frame
Frame
Frame 450
Frame 900
GFVS Explicit
Intiatt Nature
Figure 4: Unrolling results on CLVER-Infinite.
a novel framework, dubbed as SGAM, that simultancously generates novel scenes and constructs world maps. By grounding the generation and the mapping process in tightly coupled fashion, are able to produce 3D world at a significantly larger scale than any previous 3D generation effort.
We unfold this section by formulating how each module can be implemented with a neural network and how we conduct inference to construct the virtual 3D world. Finally, discuss the learning procedure, our design choices, and the relationships to existing methods.
task in probabilistic framework. Then describe
3.1 Problem Formulation
Given camera trajectory (&, SB(3)) over discrete time steps {1} and initial RGB-D sensor observation Xo E x SGAM aims to compute an estimate of the sensor observation e g and perceived map of the environment M, as set of 3D surface points and associated colors {p, e R E IR3) J Due to the inherent randomness, we consider all the quantities to be random variables, the objective is to model:
p(M1... XI...)
Using Bayes rule and Markov property, SGAM factors the joint probability into a product of conditional probability over all steps and concerns the following auto-regressive model each step
p(M, X,,,x,114 M
E,) -
At each step SGAM needs to estimate two conditional probabilities: 1) a generative sensor module po(x,|M,-1,5 rendering a realistic and complete sensor observation x, given an (incomplete) scene M, and query viewpoint & 2) mapping module Po(M,)MM-., x,, &t), which integrates the new observation to update the scene. Next will describe each module in detail.

---

Renderer
PSNRT
SSIMI
LPIPS.
ref-FID4
FIDL
Runtime 1
GFVS-implicit 611
23.06
0.872
0.138
10.47
22.85
#1.66
33.65
GFVS-cxplicit 61
18.53
0.812
0.256
15.66
23.90
#1.02
10.86
InfiniteNature 40
20.48
0.876
0.247
21.09
33.05 1.81
0.15
Ours
24.65
0.901
0.111
7.80
21.91
1.28
0.26
Table Comparison study ours and baseline methods on the CLEVR-Infiiit validation set. The unit for runtime is average second per image. FID is computed over 2500 samples for each method.
3.2 Generative Sensing
We first describe our generative sensing module p(x+|M:-1. i,))),)),, which samples a new RGBD observation x, given query viewpoint & and the current scene M Three desired properties 1) consistency the generated sensor is consistent with existing 3D scene; 2) realism: the resulting images look realistic, regardless of how many auto-regressive steps we run; 3) completeness: the sensing module should produce complete observation over both explored and unexplored areas, allowing our scene expand as camera moves. Towards these goals, we propose two-step process including neural rendering and conditional generative refinement as shown in Figure
Scene rendering: We firstly create guidance image at the target pose by rendering the current scene x; π 1, where и is an image-based rendering procedure. Given triangular mesh extracted from the scene representation Mr-1, the image-based rendering pipeline first produces a dense depth map at the target pose &, through rasterization. The rendered depth map is then used for inverse warping, producing an RGB image by transferring color appearance from existing sensor observations to the target image. A geometry consistency check will be conducted between each pixel at the rendered target depth and source depth, ensuring the color comes from the same point in 3D. incomplete RGBD observation, where the missing pixels
The image-basee based rendering step gives are due to the incompleteness of the current scene representation. Nevertheless, this step promotes the consistency between the generated sensory data and our current knowledge about the scene, built upon the past generated sensory data.
Next, we generate the final complete virtual sensor observation x; based on the incomplete rendering x/. First, the incomplete image x, is fed into an encoder network, fene which outputs continuous-valued latent feature map 2. Then, we quantize z with pre-trained codebook D, by sampling the closest quantized feature Znp-
Generative refinement:
arrgmin|22 zoll] E Rxxwx", where 2 = feno (x,) 3, q(2, D)
E,D
where, h and is the height and width of latent feature map, and n. is the embedding dimension.
Alternatively for q(2, D), to encourage diversity, we could also sample each quantized embedding from the following distribution of its corresponding latent code 2ij by measuring the soft-max simi- larity to each code word embedding 2₁ in the pre-trained codebook D. We provide qualitative results of temperature sampling CLEVR-Infinite dataset in supplementary material. Mathematically,
-sp(-d./'') ds Iz 2,y112
g(k= r(x) Σ, Exxpp_dd,/T)
where ql = r(xx) is the probability of r-th latent code following a K-dimensional categorical distribution; K is the size of the codebook; 7' is temperature parameter; and dą is the quantization error between the feature vector and the codebook element.
Finally, send Z, to decoder network faee to get complete observation x,. Our generative sensing model efficient due to our one-shot denoising procedure during the sampling stage, as opposed to the cmmpuuationalll-expessiii sequential decoding in other VQGAN-based methods (8 61.
3.3 Mapping
Representation Maintaining an efficient and flexible scene representation is crucial for high fidelity 3D world generation at large scale. Inspired by its success in sopth-basce SLAM (48) and 3D
5

---

Image metries
Scene metries
Model
FID J.
IS 4
JSD (10 9)
MMD (10 9
MinDist +
I-NNA
Cov 1
GFVS impliiii (1)
16.14
1.56 +0.05
0.775
4.510
0.205
0.675
GFVS explicit 61
82.82
3.11 +0.18
10.870
211,500
0.215
0.825
0.122
26.60
1.55
+0.08
0.656
4.441
0.221
0.585
0.454
Table 2: Scene-level comparison study on unrolling 30x30 scenes with prior arts CLEVR-Infinite.
For scene-level metrics, we use 3D hist on 15 resolution with Cosine distance.
Figure 5: Generated 3D map GoogleEarth-Inie Infinite dataset. Each sequence unrolls for 70 steps with the same relative movements between frames. The top right is the initial RGB image.
reconstruction (83), we leverage hashing based volumetric representation (50) for M, Each voxel P the space maintains a signed distance value M,)p) R$ - RR where the magnitude encodes the distance to the surface boundary, and the sign represents whether it is interior or exterior. Such representation is flexible, allowing us to update online each step when new sensory data is generated. Additionally, voxel-hashing enables sparse storage, allowing the map scale to huge scenes.
Integration At each time given a new observation Xt, we update our map M, following KinectFusion 4% Specifically, weighted sum update is conducted for points along the camera ray:
Me(P) W/- (P,
O(x,,) p), (p) = wp-1(p) +
(p) +1
where O(x,, p) is the observed signed distance function at p. calculated by the new sensor observation X& We is counter that summarizes how many steps that p has been updated. After each integration step, we will also conduct marching cube (4I) to extract triangular mesh, allowing us to render images from new viewpoint during the generative sensing process.
3.4 Learning
SGAM adopts a two-stage training pipeline. In the first stage, inspired by VQGAN 18) first pre-trained discrete-latent encoder, decoder, and codebook on the clean RGB-D image dataset an auto-encoding fashion. During training, we perform online re-initialization of the codebook embedding by performing k-means clustering over a past feature cache on the unquantized feature embedding space, whenever the active codeword hit rate within fixed time interval drops below pre-defined threshold. Following the practice in GFVS 61.. we use L,1 reconstruction loss, perceptual discriminator loss 30 and commitment loss (8)
loss (22),
Our second stage aims to train the encoder so that learns to complete the latent code space. Specifically, fine-tune the encoder with incomplete/complete image pairs. The incomplete RGB-D images are simulated through inverse warping from neighboring viewpoints. We use perceptual loss (k2) and patch-gan-based discriminator loss (30) at this stage. In this stage, freeze the codebook and decoder parameters. We find this mechanism helps maintain the output images realistic even after many unrolling steps during inference.
We use Adam to train our entire network. Learning through a con-differentiable quantization step is hard. To stabilize training, first train the encoder-decoder network without quantization for the first 30k iterations and then add it back and train the entire network for another 60k iterations using the thriigh gradient estimator trick (3) to back- eokkaapp gradient.
4 Experiments
We first evaluate the effectiveness of SGAM on large-scale synthetic dataset. Next, we comprehen- sively study the characteristics of our method. Finally we showcase how SGAM can be applied in real-world scenarios to generate high-quality 3D worlds.

---

Ground-truth
InfiniteNature 440
GFVS-Explici (61)
GFVS-Implicit (611
Ours
Figure 6: Qualitative result of one-step predictions on CLEVR-Infinite. Shaded areas are visible regions in source views based on forward warping.
ZigZag
Row Major
Column Major
Greedy
Spiral
Ring
Figure 7:
Diversity samples of the 3D scene using different virtual camera trajectories.
4.1
Setup
Existing static, large-scale 3D datasets, such as ACID 400 and RealEstate (44) do not provide accurate 3D world for evaluation. We thus exploit Blender and the assets from CLEVR (31 to render an extremely large-scale synthetic benchmark, which we called CLEVR-Infinie CLEVR-Infintt contains 72 training scenes and 8 test scenes. For each scene, we sampled fixed number of objects from the asset bank and randomly placed them on the ground plane. The asset bank consists of four primitive shapes with random scales, materials and colors. The cameras are distributed uniformly among the 50 x 50 grid plane right above and parallel to the ground. We set the pitch (i.e., the angle between the look-at vector and the plane) of the cameras to be 80%. In total, the dataset contains 200k RGB-D images at the resolution of 256 x 256.
Dataset:
Metrics: We evaluate the generated 3D world both locally at the image level and globally at the scene level. The former measures to what degree the synthetic scene reproduces the local details, while the latter quantifies how well the virtual world matches the statistics of the original counterpart.
Following previous work 45 Similarity (SSIM), and LPIPS
first Peak Signal-to-Noise Ratio (PSNR), Structural to measure the image level similarity with respect to the GT. However, since model is generative, we might synthesize scenes that plausible yet different from the original one. We thus further employ Fréchet inception distance (FID) (27) to measure the generative quality. Specifically, two FID score are reported: (i) ref-FID, which we compare against the GT; and (ii) FID, which we compare against separate set of images from test set.
KT
As for scene-level evaluation, overall geometry quality with a diverse set of metrics: mean discrepancy (MMD), coverage (COV), minimum matching distance (MinDist), and 1-nearest neighbor accuracy (1-NNA). Specifically, we first divide the whole scene into non-overlapping regions. Each region is then discretized into 15 voxels. Finally, compute the 3D histogram within each local grid and employ cosine distance to compute the metrics. To evaluate the realism, we also compute the Inception Score (IS) and FID against random sampled scenes from the test set.
follow prior art in 3D generative models 11 79) to gauge the
Shanmon divergence (JSD), maximum
We report the runtime of a single prediction step, benchmarked on Nvidia Quadro 8000.
Baselines:
We compare SGAM against sfate-of-the-an art approaches in long range scene generation: GFVS (61) and InfiniteNature 40). For GFVS, we consider both the explicit-imaee and implicit- depth variants with the following modifications: first, we use single codebook to encode RGB-D rather than two, which better combines texture and geometry during scene expansion; second, to support multi-image input, we conduct average pooling over the features of source images before passing it to MLP to infer the target code. For InfiniteNature, since the authors did not release
7

---

Frame
Frame 2
Frame
Frame 20
Frame
Frame
Frame
Frame 20
GFVS Exglaci
GEY
Intinie Naure
Figure 8: Unrolling results on Google-Earth.
Frame Frame 20
Frame 40
Frame 60
Frame 80
Frame
Frame 20 Frame 40 Frame 60 Frame 80
Figure 9: Diversity samples of the 3D scene using different step size.
their training scripts, we re-implement their method in Pytorch. We remove the geometric grounding step since find it harmful when unroll the scene in CLEVR-Infinite.
During full scene evaluation, generate the scene in zig-zag fashion on a 30 x 30 grid. We start from coordinate (0, O) and unroll for 900 steps. The above procedure repeated for 10 times with different seed images. For fair comparison, GFVS-implicit, GFVS- explicit and SGAM share the same codebook embedding and the same weights for decoder. We set the vocabulary size to 16384. We implement SGAM in PyTorch, and train with batch size of on 2 Nvidia A40 GPU until convergence. For other baselines, we maximize batch size to fit GPU memory, and train until convergence. During the first-stage codebook training, we follow the exact training config as GFVS first-stage training in the official codebase, GFVS -FVS-implicit, GFVS-explicit and SGAM share the same learning rate at 0.0625 for the second-stage -tage trainig.
Implementation details:
4.2 Experimental results
As shown in Tab. M SGAM achieves superior or comparable performance across all image level metrics. Comparing to the second best approach, GFVS-Implicit, we are much more efficient. We less CPU and GPU memory and our runtime is 150 times faster. Fig. shows few one step prediction results. SGAM is able to faithfully reconstruct the color and the shape of the assets within the visible area, and generate the invisible region in a coherent, harmonic fashion. When unrolling multiple steps (see. Fig. and Tab. 2 baselines such as InfiniteNature and GFVS-explicit suffer from error accumulation, and collapse catastrophically. GFVS-implicit generates objects that resemble the GT data, yet it sacrifices the speed and fails to enforce consistency across frames (see Fig. 6 Our approach, in contrast, is able to produce consistent and realistic results in an efficient manner.
4.3 Analysis
Importance of two-stage training: Two-stage training critical. Removing either stage degrades our performance. Without the first-stage codebook training, the generative sensing module will have learn the codebook from warped inputs, which could inevitably encode missing holes into the codebook. On the other hand, by finetuning the encoder from converged first-stage checkpoint, we consistently improve the performance of our model across all image level metrics. We refer readers to the supp. materials for detailed numbers.
Real-world scene: We apply SGAM and the baselines to real-world images that we scraped from Google Earth (see Fig. B.. with global mapping visualized in Fig. 5 While our results produce less realistic results than InfiniteNature in one-step prediction (Tab. InfiniteNatrre cannot estimate the geometry well invisible regions, and thus compounding error occurs during unrolling as shown in Tab. 3 GFVS-explicit suffers from severe domain shift. Since the images contain more diverse and complicated scene structures, GFVS-implichs fails to capture the viewpoint shift precisely during unrolling. In contrast, SGAM can produce consistent and realistic results.

---

Frame
Frame
Frame 30
Frame
Frame 100
Frame 200
Frame 900
Figure 10: Ablation Study on the effect of RGBD-integration to SGAM on GoogleEarth- infinite
dataset. Top: RGBD-integration. Bottom: with RGiDD-integration.
Renderer
FIDL
InfiniteNature 40
182.6
GFVS-implich 61
160.40
GFVS-explicit 6
133.12
Ours
79.26
Table 3: Comparison study on scene generation between ours and baseline methods on the GoogleEarth-Inne Infinite validation set. FID is computed based on the generated sequences unrolled for 60 steps against randomly sampled frames from the GoogleEarth-Infinic validation set.
Renderer
PSNR1
SSIM1
LPIPS
Las (vis)
Laup (invis)
Las
ref-FIDD
FID
InfiniteNature
400
24.78
0,880
0.172
0.017
0.041
0,020
12.61
37.92 =0.20
GFVS-implicit d
19.77
0.473
0.398
23.61
38.70 +: 0.28
GFVS-explicit 61
20.09
0.494
0.398
35.88
44.51 + 0.26
Ours
23.07
0.609
0.304
0.017
0.022
0.018
22.88
42.29 +
Table 4: Comparison study on one-step prediction between ours and baseline methods on the wrogleearth-Ininniee validation set. Lasp.. (vis) denotes the disparity L1 loss in the visible region at target pose. Ldisp. (invis) denotes the disparity Li loss in the invisible region at the target pose. The disparity is scaled between 0 and 1. We highlight best, and second best scores.
Diversity: SGAM allows us create diverse 3D world by varying camera trajectories and by varying the step size. As shown in Fig. 7 and Fig. 2 both of which are very effective. Greedy samples the next target pose based on the I.I distance of the visible region of the warped views.
Global mapping: The explicit 3D mapping ensures global consistency during generation steps. Without global mapping, direct depth- based image warping (40) hurdles inference quality at the generative refinement stage. Importantly, it cannot ensure global consistency when having a loop as shown in Fig. 4 We demonstrate Fig. That that RGBD-integration denoised mapping can achieve much more diverse scenarios than the one without, which scene degrades to large empty airport ground around 200 steps of unrolling becomes it unable fit latent code to noisy features.
Societal impact: Realistic 3D generation technology has many applications, such visual content creation and simulation. Unfortunately, it may also be used to spread misinformation.
Conclusion
In this paper, we present novel 3D scene generation algorithm, SGAM, which create consistent, realistic, and large-scale 3D virtual world through simultaneous generation and mapping. Our result demonstrates we can achieve similar (in synthetic data) or even more realistie (in real-world data) sensor observation than methods with Transformer-basee latent code sampler during unrolling.
The authors thank Derek Hoiem and David Forsyth for their early feedback. We thank Vlas Zyrianov for proofreading the paper. The project is partially funded by the Amazon Research Award and Illinois Smart Transportation Initiative ST1I-21-07. We also thank Nvidia for the Academic Hardware Grant.
Acknowledgements:

---

References
[1] Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning representations and generative models for 3d point clouds. In ICML, 2018. [2] Kara-Ali Aliev, Artem Sevastopolsky, Maria Kolos, Dmitry Ulyanov, and Victor Lempitsky. Neural point-based graphics. 2020.
[3] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 1.08.3432 2013.
[4] James R Bergen and Edward H. Adelson. Early vision and texture perception. Nature, 1988.
[5] Filip Biljecki, Jantien Stoter, Hugo Ledoux, Sisi 7latanova, and Arzu Çöltekin. Applications of 3d city models: State of the art review. ISPRS International Journal of Geo-Information, 4(4):2842- 2015.
[6] Volker Blanz and Thomas Vetter. A morphable model for the synthesis of 3d faces. In SIGGRAPH, 1999,
[7] Lucas Caccia, Herke Van Hoof, Aaron Courville, and Joelle Pineau. Deep generative modeling of lidar data. In IROS, 2019.
[8] Wayne E Carlson. An algorithm and data structure for 3d object synthesis using surface patch intersections.
In SIGGRAPH, 1982.
[9] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image
transformer. arXiv, 2022.
[10] Yun Chen, Frieda Rong. Shivam Duggal, Shenlong Wang, Xinchen Yan, Sivabalan Manivasagam, Shangjie Xue, Ersin Yumer, and Raquel Urtasun. Geosim: Realistic video simulation via geometry aware composi- tion for self-driving.
CVPR, 2021.
[11] Yen-Chi Cheng, Chich Hubert Lin, Hsin- Ying Lee, Jian Ren, Sergey Tulyakov, and Ming-Hsuan Yang. In&out: Diverse image outpainting via gan inversion. arXiv, 2021.
[12] Blender Online Community. Blender a 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018.
[13] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. In Proceedings of the 1st Annual Conference on Robot Learning, pages
1- 2017.
[14] Shivam Duggal and Deepak Pathak. Topologically-wwan deformation fields for single-view 3d reconstruc-
tion. CVPR, 2022.
[15] Shivam Duggal, Zihao Wang. Wei-Chiu Ma. Sivabalan Manivasagam, Justin Liang, Shenlong Wang, and Raquel Urtasun, Mending neural implicit modeling for 3d vehiele reconstruction in the wild. In WACV, 2022.
[16] Alexei A Efros and William Freeman. Image quilting for texture synthesis and transfer. In Proceedings
of the 28th annual conference on Computer graphics and interactive techniques. 2001.
[17]
Alexci A Efros and Thomas K Leung. Texture synthesis by non- parametric sampling. In ICCV, 1999, Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis.
[18]
In CVPR, 2021.
[19] Rohit Girdhar, David Fouhey, Mikel Rodriguez, and Abhinav Gupta. Learning a predictable and generative vector representation for objects. In ECCV, 2016.
[20] Georgia Gkioxari, Jitendra Malik, and Justin Johnson. Mesh r-can. In ICCV, 2019.
[21] lan J. Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014.
[22]
Arthur Gretton, Karsten Borgwardt, Malte J Rasch, Bernhard Scholkopf, and Alexander J Smola. A kernel
method for the two-sample problem. arXiv preprint arXiv:0805.2364 2368, 2008.
[23] Giorgio Grisetti, Rainer Kümmerle, Cyrill Stachniss, and Wolfram Burgard. A tutorial on graph-based
slam. IEEE Intelligent Transportation Systems Magazine, 2(4):31-43, 2010.
[24] Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu Aubry. A papier-máché
approach to learning 3d surface generation. In CVPR, 2018
[25] Jiayuan Gu, Wei-Chiu Ma, Sivabalan Manivasagam, Wenyuan Zeng, Zihao Wang. Yuwen Xiong, Hao Su, and Raquel Urtasun. Weakly-supervised 3d shape completion in the wild. In ECCV, 2020.
[26] David Heeger and James R Bergen. Pyramid-bass based texture analysis/synthesis. In Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, 1995.
[27] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans
trained by a two time-scale update rule converge to local nash equilibrium. NeurlPS, 2017.
[28]
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurlPS, 2020.
[29]
Ronghang Hu, Nikhila Ravi, Alex Berg, and Deepak Pathak. Worldsheet: Wrapping the world in a 3d
sheet for view synthesis from a single image. ICCV, 2021.
[30] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros, Image-to-image translation with conditional
adversarial networks. In CVPR, 2017.
[31] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR. 2017.
[32] B. Julesz. Visual pattern discrimination. IRE Transactions on Information Theory, 1962.
[33]
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved
quality, stability, and variation. arXiv, 2017.
[34] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
10

---

[35]
Diederik P Kingma and Max Welling. Auto-encoding variational bayes, arXiv, 2013.
[36]
Jing Yu Koh, Honglak Lec, Yinfei Yang, Jason Baldridge, and Peter Anderson. Pathdreamer: A world
model for indoor navigation. 2021.
[37]
Chich Hubert Lin, Yen-Chi Cheng, Hsin-Ying Lee, Sergey Tulyakov, and Ming-Hsuan Yang. InfinityGAN:
Towards infinite-pixel image synthesis. In ICLR, 2022.
[38]
Chen-Hsuan Lin, Chen Kong, and Simon Luccy. Learning efficient point cloud generation for dense 3d
object reconstruction. In AAAI, 2018.
[39] Chen-Hsuan Lin, Chaoyang Wang, and Simon Lucey. Sdf-sm: Learning signed distance 3d object
reconstruction from static images. NeurlPS, 2020.
[40] Andrew Liu, Richard Tucker, Varun Jampani, Ameesh Makadia, Noah Snavely, and Angjoo Kanazawa. Infinite nature: Perpetual view generation of natural scenes from a single image. In ICCV, 2021.
[41] William E. Lorensen and Harvey E. Cline. Marching cubes: A high resolution surface construction al- gorithm. In Proceedings of the 14th Annual Conference on Computer Gruphics and Interactive Techniquex, SIGGRAPH '87, page 163-169, New York, NY, USA, 1987. Association for Computing Machinery.
[42] Jitendra Malik and Pietro Perona. Preattentive texture discrimination with early vision mechanisms. (5),
1990.
[43] Arun Mallya, Ting-Chie Chun Wang, Karan Sapra, and Ming- Yu Liu. World-consistent video-to-vider synthesis.
In ECCV, 2020.
[44]
Sivabalan Manivasagam, Shenlong Wang, Kelvin Wong, Wenyuan Zeng, Mikita Sazanovich, Shuhan Tan, Bin Yang. Wei-Chiu Ma, and Raquel Urtasun. Lidarsim: Realistic lidar simulation by leveraging the real
world. CVPR, 2020.
[45]
Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[46]
Nelson Nauata, Kai-Hung Chang. Chin-Yi Yi Cheng, Greg Mori. and Yasutaka Furukawa. House-gan: Relational generative adversarial networks for craph-constrained house layout generation. In ECCV, 2020.
[47] Nelson Nauata, Sepidehsadat Hosseimi, Sai-Hung Chang. Hang Chu, Chin-Yi Cheng, and Yasutaka Furukawa. House-gan++4: Generative adversarial layout refinement networks. arXiv, 2021.
[48] Richard A. Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J. Davison. Pushmeet Kohi, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon. Kinectfusion: Real-time dense surface mapping and tracking. In 2011 10th IEEE International Symposium Mixed and Augmented Reality, pages 127-136, 2011.
[49] Thu Nguyen-Phuoce Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang. Hologan: Unsuper- visdd learning of representations from natural images. In ICCV, 2019.
[50] Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Mare Stamminger. Real- time 3d reconstruction at scale using voxel hashing, ACM Transactions on Graphics (ToG), 32(6):1111, 2013.
[51]
Despoina Paschalidou, Amlan Kar, Maria Shugrina, Karsten Kreis. Andreas Geiger, and Sanja Fidler. Atiss: Autoregressive transformers for indoor scene synthesis. In NewrIPS, 2021.
[52] Chi-Han Peng, Yong-Liang Yang, and Peter Wonka. Computing layouts with deformable templates. TOG, 2014.
[53] Shengyi Qian, Alexander Kirillov, Nikhila Ravi, Devendra Singh Chaplot, Justin Johnson, David F Fouhey, and Georgia Gkioxari. Recognizing scenes from novel viewpoints. arXiv, 2021.
[54] Viming Qian, Hao Zhang, and Yasutaka Furukawa. Roof-gan: learning to generate roof geometry and
relations for residential houses. In CVPR, 2021.
[55] Alec Radford, Luke Metz. and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv, 2015.
[56] Xuanchi Ren and Xiaolong Wang. Look outside the room: Synthesizing consistent long-term 3d scene
video from a single image. In CVPR, 2022.
[57]
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In ICML, 2015.
[58] Gernot Riegler and Vladien Koltun. Free view synthesis. In European Conference on Computer Vision,
2020.
[59]
Gernot Riegler and Vladlen Koltun. Stable view synthesis. In CVPR, 2021,
[60] Chris Rockwell, David F. Fouhey, and Justin Johnson. Pixelsynth: Generating 3d-consistent experience
from a single image. In ICCV, 2021.
[61]
Robin Rombach, Patrick Esser, and Bjöm Ommer. Geometry-free view synthesis: Transformers and 3d priors. /CCV, 2021.
[62] Bryan C. Russell and Antonio Torralba. Building database of 3d scenes from user annotations. In CVPR, 2009.
[63] Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung. Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems. 29, 2016. Meng-Li Shih, Shih- Yang Su, Johannes Kopf, and Jia-Bin Huang. 3d photography using context-aware layered depth inpainting. In CVPR, 2020.
[64]
[65]
Ruben M Smelik, Tim Tutenel, Rafacl Bidarr,, and Bedrich Benes, A survey on procedural modelling for virtual worlds. In Computer Graphics Forum, volume 33, pages 31-50. Wiley Online Library, 2014.
[66] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishck Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stnchastic differential equations. arXiv, 2020.
[67] Shuhan Tan, Kelvin Wong, Shenlong Wang, Sivabalan Manivasagam, Mengye Ren, and Raquel Urtasun. Scenegen: Learning to generate realistic traffic scenes. In CVPR, 2021.
11

---

[68]
Johan WH Tangelder and Remco C Veltkamp. A survey of content based 3d shape retrieval methods. Multimedia tools and applications. 2008.
[69]
Maxim Tatarchenko, Stephan R Richter, René Ranftl, Zhuwen Li, Vladlen Koltun, and Thomas Brox.
What do single-view 3d reconstruction networks learn? In CVPR, 2019.
[70] Piotr Teterwak, Aaron Sarna, Dilip Krishnan, Aaron Maschinot, David Belanger, Ce Liu, and William T Freeman. Boundless: Generative adversarial networks for image extension. ICCV. 2019.
[71] Richard Tucker and Noah Snavely. Single-view view synthesis with multiplane images. In CVPR, 2020. Shubham Tulsiani, Richard Tucker, and Noah Snavely. Layer-structured 3d scene inference via view
[72]
synthesis. ECCV, 2018.
[73]
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. NeurIPS, 2017.
[74]
Qianqian Wang. Zhicheng Wang. Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo Martin-Brualla Noah Snavely, and Thomas Funkhouser. Ibmnt:: Learning multi-view image-based
rendering. In CVPE, 2021.
[75] Zhou Wang. Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. JEEE transactions on image processing, 13(4):600-612, 2004.
[76] Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson. Synsin: End- to end view synthesis
from a single image. In CVPR, 2020.
(77] Jiajun Wu, Yifan Wang, Tianfan Xue, Xingyuan Sun, Bill Freeman, and Josh Tenenbaum. Marmet: 3d
shape reconstruction via 2.5 d sketches. NewrlPS, 2017.
[78] Jiajun Wu, Chengkai Zhang. Tianfan Xue, Bill Freeman, and Josh Tenenbaum. Learning probabilistic
latent space of object shapes via 3d generative-adversariaa modeling. NewrIPS, 2016.
[79] Guandao Yang, Xun Huang, Zekun Hao, Ming- Yu Liu, Serge Belongie, and Bharath Haribaran. Pointflow:
3d point cloud generation with continuous normalizing flows. In ICCV, 2019.
[80] Zhenpei Yang. Yuning Chai, Dragomir Anguelov, Yin Zhou, Pei Sun, Dumitru Erhan, Scan Rafferty, and Henrik Kretzschmar. Surfelgan: Synthesizing realistic sensor data for autonomous driving. CVPR, 2020.
[81] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa, pixeINeRF: Neural radiance fields from one or few images. In CVPR, 2021.
[82] Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
[83] Qian-Yi Yi Zhou, Stephen Miller, and Vladlen Koltun. Elastic fragments for dense scene reconstruction. In Proceedings of the IEEE International Conference Computer Vision. pages 473-480, 2013. [84] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification:
Learning view synthesis using multiplane images. arXiv, 2018.
[85] Song Chun Zhu, Yingnian Wu, and David Mumford. Filters, random fields and maximum entropy (frame):
Towards unified theory for texture modeling. International Journal of Computer Vision, 1998.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Please see supp. material for the discussion.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [No] We plan to release the code after acceptance.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Ys]] Our SGAM is built on top of VQGAN codebase. We re-use the hyperparameters among all baselines.
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? We randomly generate multiple scene with different seed images, i.c., initiall sensor observation to capture overall model performance. [N/A]
12

---

(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or a URL? [No] We plan to release our dataset upon acceptance.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're
using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "zAC5VWhLkNr", "reviewer": "Reviewer_SFD9", "summary": "This paper presents a large scale scene generation method which promotes global 3D scene consistency. The method works by taking as input a random view of a scene, and then moving through the scene continually using the current understanding/representation to create incomplete observations from new perspectives, using a generative model to in-paint the incomplete regions, and then using the newly generated information to update the understanding/representation of the scene. With this setup they demonstrate impressive scene generations on a dataset made up of simples shapes scattered around a white room, and on google street view images. ", "questions": "How difficult is this process to train? Have you observed consistent convergence to attractive scenes over multiple seeds? Does this green color bias effect all seeds similarly? \n\nIf I understand correctly the camera angle is fixed throughout the trajectory? Have you explored at all with random trajectories through the space with the camera orientation changing as you move? I would be interested how the method performed with less structured passes through the search space.", "limitations": "Yes ", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper is interesting and the results are impressive. The method is strong, and integrates lots of different techniques from different areas to produce a novel and successful solution to a hard problem. \n\nWith respect to the method I am slightly skeptical how well it will scale to more complex scenes. The underlying 3D representation is a voxel grid, and while hashing is performed to make it more efficient, I am not sure if this will support complex detail that natural scenes poses. It might be good to speak to this in the limitations section. \n\nConnected to this I comment I would have also liked to have seen applications of this method to complex indoor scenes, as shown in the GFVS paper. I realize you require depth which is not available in general for indoor image datasets, however there exists some simulated indoor datasets for which you could again capture depth in blender. This would really help to understand how well your model scales, as at the moment you only demonstrate performance on practically 2D maps. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "y-JwMIVf4YT", "reviewer": "Reviewer_QFY4", "summary": "The work tackles the problem of 3D scene generation, learning from sequences of RGB-D  images and its poses. The main contribution of the work comes from proposing an algorithm that uses a generative sensing module and a mapping module. The proposed method was evaluated on CLEVR and GoogleEarth dataset and achieves SOTA results on the former. The method is also more efficient compared to previous results.\n", "questions": "- In the video there seems to be flickers in GoogleEarth dataset around 1:54, where the scene changes abruptly? Could the authors explain why this happens? Or is this just another video and not a continuous scene change.\n", "limitations": "Yes, the authors addressed the limitations of the work. But I recommend adding the comments regarding the method requires exact depth information.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\n- The proposed method SGAM uses generative sensing module and mapping module, where the former is a VQGAN and the latter is mostly KinectFusion using volumetric representations. The formulation is a sound integration of the neural network with previous methods. \n\n- The paper is well written and easy to understand\n\n- The proposed method achieves SOTA results on CLEVR dataset on image based metrics and also on 3D generative metrics (MMD, JSD, 1-NNA…). \n\n- I enjoyed watching the results in video. I appreciate the authors for the hard work. \n\n\nWeaknesses: \n\n- My main concern lies in the experimental section, especially on real-world datasets. The main experiment was mainly on the CLEVR dataset. As the authors mentioned, the essence of scene generation comes from generating diverse but realistic results. Although the CLEVR dataset is a good way to show the distributional similarity between the generated set and the real distribution, the main limitation is that this is a synthetic dataset. The appendix in Table 4 shows that the method shows worse results on real dataset. I would like to hear from the authors regarding the discrepancy on the evaluation metrics on real and synthetic datasets.\n\n- Compared to other methods [1, 2], the method requires accurate depth. What happens if the method is trained on ACID dataset, where the accurate depth information is unavailable? I regard this as an important problem, since many of the real-world applications do not provide the exact depth information. \n\n\nSummary:\nThe proposed method that leverages generative sensing module and a mapping module makes sense. However, my main concern lies in the experimental section, where the method achieves best results only on synthetic dataset and requires accurate depth. I would like to hear the response from authors regarding the experimental section and listen to other reviewers before making the final decision.\n\n===============================================================================\n\nAfter rebuttal, I'm convinced that the proposed method has the strength of generating long-term, globally consistent scenes. Therefore I'm willing to increase the score. However, I strongly encourage the authors to compare the method against previous methods such as ACID and RealEstate10k datasets to show the strength/weakness in real/standardized datasets. I do not think achieving SOTA results on these datasets is crucial since the method has clear strength, but the method should at least be competitive to verify that the method works on real-world datasets.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1SvEP7yVVYz", "reviewer": "Reviewer_GmLE", "summary": "The paper proposes a method to generate a large-scale RGBD image in a near NADIR view. It uses a standard image generative model VQGAN while using the already-generated scenes as an input constraint. It qualitatively and quantitatively evaluate on the Clever dataset. For Google earth data, only qualitative evaluations are given.", "questions": "I want answers to my criticism (weakness analysis) above.", "limitations": "The limitation description is OK. I cannot think of a better one.", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "To me, this paper is over-selling. The paper is just a 4 channel image generation by repeatedly applying a standard VQGAN. Simultaneous generation and mapping does not make much sense.\n\nThe core of the method is an application of an existing technique and the technical contribution is weak. I also do not like the way authors present their method to handle a general camera pose, while in reality they just use near nadir views.\n\nAlso, CLEVER scene is a bit too simple and through evaluations should be given on real data (Google Earth).\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept