GAUDI: A Neural Architect for Immersive 3D Scene Generation
Authors: Miguel Ángel Bautista
Pengsheng Guo
Samira Abnar
Walter Talbott
Alexander T Toshev
Zhuoyuan Chen
Laurent Dinh
Shuangfei Zhai
Hanlin Goh
Daniel Ulbricht
Afshin Dehghan
Joshua M. Susskind
Venue: NeurIPS 2022 Accept (2022)
Keywords: generative modeling
3D
radiance fields

Abstract: We introduce GAUDI, a generative model capable of capturing the distribution of complex and realistic 3D scenes that can be rendered immersively from a moving camera. We tackle this challenging problem with a scalable yet powerful approach, where we first optimize a latent representation that disentangles radiance fields and camera poses. This latent representation is then used to learn a generative model that enables both unconditional and conditional generation of 3D scenes. Our model generalizes previous works that focus on single objects by removing the assumption that the camera pose distribution can be shared across samples. We show that GAUDI obtains state-of-the-art performance in the unconditional generative setting across multiple datasets and allows for conditional generation of 3D scenes given conditioning variables like sparse image observations or text that describes the scene.

GAUDI: A Neural Architect for Immersive 3D Scene Generation
Miguel Angel Bautista" *
Pengsheng Guo"
Samira Abnar
Walter Talbott
Alexander Toshev
Zhuoyuan Chen
Laurent Dinh
Shuangfei Zhai
Hanlin Goh
Daniel Ulbricht
Afshin Dehghan
Josh Susskind
Apple https:///itthub.comt wwtp:///..coooapppemmm//aee-e
Abstract
We introduce GAUDI, generative model capable of capturing the distribution of complex and realistic 3D scenes that can be rendered immersively from moving camera. We tackle this challenging problem with scalable yet powerful approach, where we first optimize a latent representation that disentangles radiance fields and camera poses. This latent representation is then used to learn generative model that enables both unconditional and conditional generation of 3D scenes. Our model generalizes previous works that focus on single objects by removing the assumption that the camera pose distribution can be shared across samples. We show that GAUDI obtains state-of-the-art performance the unconditional generative setting across multiple datasets and allows for conditional generation of 3D scenes given conditioning variables like sparse image observations or text that describes the scene.
Introduction
In order for learning systems to be able understand and create 3D spaces, progress in generative models for 3D is sorely needed. The quote "The creation continues incessantly through the media of man." is often attributed to Antoni Gaudí, who we pay homage to with our method's name. In this paper ask the question: are interested in generative models that can capture the distribution of 3D scenes and then render views from scenes sampled from the learned distribution. Extensions of such generative models to conditional inference problems could have tremendous impact wide range of tasks in machine learning and computer vision. For example, consistent with an image observation, or text description (see Fig. for 3D scenes sampled from GAUDI). In addition, such models would be of great practical learning and planning [4], SLAM
creation continue through the media of learning machines? We
could sample plausible scene completions that are
model-basee reinforcement
4% or 3D content creation.
Recent works generative modeling for 3D objects scenes Adversarial Network (GAN) where the generator explicitly encodes radiance fields - a parametric function that takes as input the coordinates of point in 3D space and camera pose, and outputs density scalar and RGB value for that 3D point. Images can be rendered from radiance field generated by the model by passing the queried 3D points through the volume rendering equation
60 6
employ Generative
denotes equal contribution. Corresponding email: maattistamartcinapple.coo
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Unconditinnll sample scene and poses p(c)
Conditional
image
conditioned on image
text
category
conditioned on I'o through hallway")
Figure I: GAUDI allows to model both conditional and unconditional distributions over complex 3D scenes. Sampled scenes and poses from (left) unconditional distribution, and (right) distribution conditioned on an image observation or a text prompt.
to project onto any 2D camera view. While compelling on small or simple 3D datasets (e.g. single objects small number of indoor scenes), GANs suffer from training pathologies including mode collapse [88 (55 and are difficult to train on data for which canonical coordinate system does not exist, as is the case for 3D scenes 61.. In addition, one key difference between modeling distributions of 3D objects vs. scenes that when modeling objects, the distribution of valid camera poses does not depend each object and is defined per dataset (i.e. typically SO(3)), which is not true for scenes. See Fig. B(b) and note how the two scene layouts define different areas of navigable space (different dark grey shaded areas) where valid camera poses be placed, revealing strong dependency between scenes and their camera pose distributions. GAUDI captures this dependency by modeling the joint distribution of scenes and camera poses.
In GAUDI, we map each trajectory (i.e. a sequence of posed images from 3D scene) into latent representation that encodes radiance field (e.g. the 3D scene) and camera path in completely disentangled way. We find these latent representations by interpreting them as free parameters and formulating optimization problem where the latent representation for each trajectory is optimized via a reconstruction objective. This simple training process scalable to thousands trajectories. Interpreting the latent representation of each trajectory a free parameter also makes it simple to handle large and variable number of views for each trajectory rather than requiring a sophisticated encoder architecture pool across large number of views. After optimizing latent representations for an observed empirical distribution of trajectories, we learn a generative model over the set of latent representations. In the unconditional case, the model can sample radiance fields entirely from the prior distribution learned by the model, allowing it to synthesize scenes by interpolating within the latent space, In the conditional case, conditional variables available the model training time (e.g. images, text prompts, etc.) can be used to generate radiance fields consistent with those variables. Our contributions can be summarized
We scale 3D scene generation to thousands of indoor scenes containing hundreds of thousands of images, without suffering from mode collapse canonical orientation issues during training.
We introduce a novel denoising optimization objective to find latent representations that jointly model a radiance field and the camera poses in a disentangled manner.
Our approach obtains state-of-the-ar generation performance across multiple datasets.
Our approach allows for various generative setups: unconditional generation as well as conditional on images or text.
2 Related Work
In recent years the field has witnessed outstanding progress in generative modeling for the 2D image domain, with most approaches focusing either on adversarial [21] 22 or auto-regressive models
669 46 I1. More recently, score matching based approaches [X 62 have gained popularity. In
particular, Denoising Diffusion Probabilistie Models (DDPMs) 17
G8 have emerged as
strong contenders to both adversarial and ruto-regressive approaches. In DDPMs, the goal is to learn

---

a step-by-ster inversion of a fixed diffusion Markov Chain that gradually transforms an empirical data distribution to a fixed posterior. which typically takes the form of an isotropic Gaussian distribution. In parallel, the last couple of years have seen revolution in how 3D data is represented within neural networks. By representing a 31) scene as a radiance field, NeRF [
introduces an approach to optimize the weights of MIP to represent the radiance of 3D points that fall inside the field-of-vien of a given set of posed RGB images. Given the radiance for a set of 3D points that lie on ray shot from given camera pose, NeRF 33 uses volumetric rendering to compute the color for the corresponding pixel and optimizes the MLP weights via a reconstruction loss in image space.
A few attempts have also been made at incorporating a radiance field representation within generative models. Most approaches have focused on the problem of single objects with known canonical orientations like faces or Shapenet objects with shared camera pose distributions across samples in a Extending these approaches from single objects to completely unconstrained 3ID scenes is an unsolved problem. One paper worth mentioning in this space is GSN R which breaks the radiance field into a grid of local radiance fields that collectively represent a scene. While this decomposition of radiance fields endows the model with high representational capacity, GSN still suffers from the standard training pathologies of GANs, like mode collapse (65), which are exacerbated by the fact that unconstrained 3D scenes do not have canonical orientation. As we show in our experiments (cf. Sect. D. these issues become prominent as the training set size increases, impacting the capacity of the generative model to capture complex distributions. Separately, a line of recent approaches have also studied the problem of learning generative models of scenes without employing radiance fields 40 77, 5T. These works assume that the model has access to room layouts and a database of object CAD models during training, simplifying the problem of scene generation to selection of objects from the database and pose predictions for each object.
dataset 600 38.0 25 12
77
Finally, approaches that learn to predict a target view given a single (or multiple) source view and relative pose transformation have been recently proposed 27 74 57
9 13 The pure reconstruction objective employed by these approaches forces them to learn a deterministic conditional function that maps source image and relative camera transformation to a target image. The first is that this scene completion problem is ill-posed (e.g. given a single source view of a scene there are multiple target completions that are equally likely). Attempts at modeling the problem in probabilistic manner have been proposed [53] 49 However, these approaches suffer from inconsistency in predicted scenes because they do not explicitly model 3D consistent representation like radiance field.
3 GAUDI
Our goal is to learn generative model given an empirical distribution of trajectories over 3D scenes. Let X = {aie (o.....)) } denote a collection of examples defining empirical distribution, where each example m; is a trajectory. Every trajectory x, is defined as a variable length sequence of corresponding RGB, depth images and 6DOF camera poses (see Fig. 3.
We decompose the task of learning a generative model in two stages. First, we obtain a latent representation z = (Zsceee pppss for each example x E X that represents the scene radiance field and pose in separate disentangled vectors. Second, given a set of latents Z (Z..(o.....))) we learn the distribution pZZ).
3.1 Optimizing latent representations for radiance fields and camera poses
We now turn to the task of finding a latent representation z E Z for each example x E X (i.e. for each trajectory in the empirical distribution). To obtain this latent representation we take an encoder-less view and interpret z is as free parameters to be found via an optimization problem 2 35 To map latents to trajectories we design network architecture (i.e. decoder) that disentangles camera poses and radiance field parameterization. Our decoder architecture is composed of networks (shown in Fig. 2
The camera pose decoder network e (parameterized by 0.), is responsible for predicting camera poses T. SE(3) at the normalized temporal position 8 E [-1, 1] in the trajectory, conditioned on 2psse which represents the camera poses for the whole trajectory. To ensure that the output of c is a valid camera pose (e.g. an element of SF))),, we output a 3D vector representing a normalized quaternion qN for the orientation and 3D translation vector t..
3

---

T, (q. [c,]
11
Poss latents
ppose
w
07
Vekure
Scone Intents
Way
Wagz
d
Zecenne
Wags W Wedl
Figure 2: Architecture of the decoder model that disentangles camera poses from 3D geometry and appearance of the scene. Our decoder is composed by submodules. A decoder d that takes as imput a latent code representing the scene Zacene and produces a factorized representation of 3D space via tri-plane latent encoding W. A radiance field network f that takes as input points p R and is conditioned on W to predict a density and signal a to be rendered via volumetric rendering (Eq. [ Finally, we decode the camera poses through a network c that takes as input a normalized temporal position
[-1, 1] and is conditioned on Zpose which represents camera poses for the whole trajectory x: to predict the camera pose T, (3)))..
The scene decoder network d (parameterized by 0a). is responsible for predicting conditioning variable for the radiance field network f. This network takes as input latent code that represents the scene Zecene and predicts an axis-aligned tri-plane representation 41 5 W E p3x5xx5xF Which correspond to 3 feature maps [Wa, zy. Wz., zz, Waz] of spatial dimension S x S and channels, one for each axis aligned plane: 2y, x2 and yz.
The radiance field decoder network f (parameterized by 8y), ), is tasked with reconstructing image level targets using the volumetric rendering equation in Eq. [ The input to is p IR3 and the tri-plane representation W = =[Way ay: Waz: Wys] (we do not condition on the camera orientation to improve consistency [12). ). Given 3D point p [ k] for which radiance is to be predicted,
orthogonally project into each plane in W and perform bi-linear sampling. We concatenate the
bi-linearly sampled vectors into Wayz [Way(t, j), W.. k), W W= k)] R3F, which is used to condition the radiance field function f. We implement ff as a MLP that outputs a density value o and signal To predict the value v of pixel, the volumetric rendering equation is used (cf. Eq. [ where a 3D point is expressed as ray direction r (corresponding with the pixel location) at particular depth w.
1 u Tr(uja (r(u), Wayz) (r(u), Wayz) du
v(r, W)
Tr(u)= exp (-/" (r(()), Ways)
We formulate a reconstruction objective to jointly optimize for 6a. lz, Oy and (z):;=o......))) shown in Eq. 2 Note that while latents z are optimized for each example x independently, the parameters of the networks fa,, 0,.. fr are amortized across all examples x' E X. As opposed to previous auto-decoding approaches 2 [39. each latent z is perturbed during training with additive noise that is proportional
to the empirical standard deviation across all latents, z = z + 3N (0. std (Z)). inducing a contractive representation [50) In this setting, / controls the trade-off between the entropy of the distribution of latents z E Z and the reconstruction quality. With = the distribution of z's becomes a set of indicator functions (i.e. similar as would get from vanilla auto-encoder). For large 3 > 0 the structure in distribution of z's is destroyed, as latents are perturbed with large magnitudes of noise. We use small / value to enforce latent space in which interpolated samples (or samples that contain small deviations from the empirical distribution, as the ones that one might get from sampling subsequent generative model) are included in the support of the decoder function, sacrificing small cost in reconstruction fidelity.
Ea-x [frceme (x Zeeene, T + XC (T Zpase, s)] 02.2
min
(2)
We optimize parameters 0s, 0,, Og and latents z E Z with two different losses. The first loss function Lecene measures the reconstruction between the radiance field encoded in Zacene and the images in

---

the trajectory mm (where denotes the normalized temporal position of the frame in the trajectory). given ground-truth camera poses T, required for rendering. We use an l2 loss for RGB and for depth The second loss function Loose measures the camera pose reconstruction error between the poses T, encoded in Epose and the ground-truth poses. We employ an l2 loss on translation and I loss for the normalized quaternion part of the camera pose. Although theoretically normalized quaternions are not necessarily unique (e.g- q and -q) we do not observe any issues empirically during training.
3.2 Prior Learning
Given set of latents Z resulting from minimizing the objective in Eq. our goal is learn generative model p(Z) that captures their distribution (i.e. after minimizing the objective in Eq. 2
interpret z E Z examples from an empirical distribution in latent space). In order to model we employ Denoising Diffusion Probabilistic Model (DDPM) 177. recent score-matching 18 based model that learns reverse diffusion Markov Chain with large but finite number of timesteps. In DDPMs [17] it is shown that this reverse process is equivalent to learning a sequence of
z(Z)
denoising auto-encoders with tied weights. The supervised denoising objective DDPMs makes learning p(2) simple and scalable. This allows us to learn powerful generative model that enables both unconditional and conditional generation of 3D scenes. For training our prior pe, P2) we take the objective function in [ defined in Eq. In Eq. B t denotes the timestep, (0, I) is the noise and ãe is a noise magnitude parameter with a fixed scheduling. Finally, c0, denotes the denoising model.
min E, Z, [[|e E0, (Vazz VI a,, 0g
At inference time, we sample z Pe, (Z) by following the inference process in DDPMs. We start by sampling žy (O,0 I) and iteratively apply Eg, to gradually denoise 27, thus reversing the diffusion Markov Chain to obtain Zg. We then feed Zn as input to the decoder architecture (cf. Fig. 2 and reconstruct radiance field and camera path.
If the goal learn a conditional distribution of the latents P(Z given paired data {z E Z. Y}, the denoising model is augmented with a conditioning variable y. resulting in cØ, (z, t, y). implementation details about how the conditioning variable is used in the denoising architecture can be found in the appendix
4 Experiments
In this section we show the applicability of GAUDI to multiple problems. First, we evaluate reconstruction quality and performance of the reconstruction stage. Then, we evaluate the performance of our model in generative tasks including unconditional and conditional inference, in which radiance fields are generated from conditioning variables corresponding to images or text prompts. Full experimental settings and details can be found in the appendix
4.1 Data
We report results on 4 datasets: Vizdoom 23 Replica 64 VLN-CE 26 and ARKit Scenes 1.
which vary in number of scenes and complexity (see Fig. and Tab.
Vizdoom 2
Vizdoom is synthetic simulated environment with simple texture and geometry. We the data provided by g to train our model. It the simplest dataset in terms of number of scenes and trajectories, as well as texture, serving as a test bed to examine GAUDI in the simplest setting.
Replica is dataset comprised of 18 realistic scenes from which trajectories rendered via Habitat [59] We used the data provided by [ to train our model.
Replica [64]:
VLN-CE 26 VLN-CE is dataset originally designed for vision and language navigation in continuous environments. This dataset composed of 3.6K trajectories of an agent navigating between two points in a 3D scene from the Matterport 3D dataset f We render observations via
We obtain depth predictions by aggregating densities across ray in 33
5

---

(a)
(b)
Figure 3: (a) Examples of the 4 datasets we use in this paper (from left to right): Vizdoom 23
Replica 6d VLN-CE [26 ARKitScenes M. (b) Layouts for scenes in VLN-CE (26 where
navigable areas are shaded in dark gray. Blue and red dots represent start-end positions and the camera path is highlighted blue.
Habitat 59
Notably, this dataset contains also textual descriptions of the trajectories taken by an we train GAUDI in conditional manner generate 3D scenes given description.
agent. In Sect. 4.5
ARKitScenes I ARKitScenes is dataset of scans of indoor spaces. This dataset contains more than 5K scans of about 1.6K different indoor spaces. As opposed to the previous datasets where RGB, depth and camera poses are obtained via rendering in simulation (i.e. either Vizdoom [23] or Habitat 159 ARKitScenes provides RGB and depth of the scans and camera poses estimated using ARKit SLAM. In addition, whereas trajectories from the previous datasets are point-to-poing typically done in navigation, the camera trajectories for ARKitScenes resembles natural of full indoor space. In our experiments we use subset of 1K scans from ARKitScenes to train our models.
4.2 Reconstruction
We first validate the hypothesis that the optimization problem described in Eq. 2 can find latent codes 2 that are able reconstruct the trajectories in the empirical distribution in satisfactory way. In Tab. T we report reconstruction performance of our model across all datasets. Fig. shows reconstructions of random trajectories for each dataset. For all our experiments set the dimension of Zacene and pooee to 2048 and B = 0.1 unless otherwise stated. During training, we normalize camera poses for each trajectory so that the middle frame in trajectory becomes the origin of the coordinate system. See appendix for ablation experiments.
Figure 4: Qualitative reconstruction results of random trajectories different datasets (one for each column): Vizdoom [23], Replica [4. VLN-CE 26] ARKitScenes For each pair of images the left is ground-truth and right is reconstruction.
#sc Hir tim
PSNR
SSIM 7
Rot Em:
Trans. Er
Vizdoum
1-32-1k
0.004
$6.12
0.98
0.01
1.26
Replica
61
18- 18-100-14
0.066
38.86
0.99
0.03
0.01
VI-N-CE
ARKiSSemes
1k-600k
0.009
24.51
0.76
0.16
Table 1: Reconstruction results of the optimization process described in Eq. The first column shows the number of scenes (#sc), trajectories (#tr) and images (#im) per dataset. Due to the large number of images on VIN-CE [26] and ARKitScenes [ datasets we sample 10 random images per trajectory to compute the reconstruction metrics.
4.3 Interpolation
In addition, evaluate the structure of the latent representation obtained from minimizing the optimization problem in Eq. 2 we show interpolation results between pairs of latents (z,, 2,) in

---

To render images while interpolating the scene we place a fixed camera at the origin of the coordinate system. We observe a smooth transition of scenes in both geometry (walls, ceilings) and texture (stairs, carpets). More visualizations are included in the appendix H
Fig.
Zi
Zj
Figure 5: Interpolation of 3D scenes in latent space (e.g. interpolating the encoded radiance field) for the VLN-CE dataset 126 Each row corresponds to a different interpolation path.
4.4 Unconditional generative modeling
Given latent representations E Z that reconstruct samples æ X with high accuracy as shown in Sect. 4.2 we now evaluate the capacity of the prior po, (Z) to capture the empirical distribution x E X by learning the distribution of latents 2; E Z. To do so we sample z ~ po, (Z) by following the inference process in DDPMs, and then feed through the decoder network, which results in trajectories of RGB images that are then used for evaluation. We compare our approach with the following baselines: GRAF (60), T-GAN (6] and GSN [8), where all models have access to ground-truth depth information during training. We sample 5k images from predicted and target distributions for each model and dataset and report both FID [6] and SwAV-FID 55 scores.
We report quantitative results in Tab. D where we can see that GAUDI obtains state-of of-h----rt--tt performance across all datasets and metrics. We attribute this performance improvement the fact that GAUDI Iearns disentangled yet corresponding latents for radiance fields and camera poses, which key when modeling scenes (see ablations in the appendix E). We note that to obtain these great empirical results GAUDI needs to simultaneously find latents with high reconstruction fidelity while also efficiently learning their distribution.
VizDoom
Replica
ARKitScenes
FID
SIA
Se
FID
SwAW
FID J
SwAV FID
GRAF 70
47.50 + 2.13
5.44 + 0.43
65.37 + 1.64
5.76
90.43
4.83
8.65 + 0.27
87.06 + 9.99
13.44 + 0.26
16.26
168.55
13.17
0.20
151.26
4.19674
14.07
0.56
184.80
10.60
15.58
1.17
0.19
41.75
4.14
43.32
79.54
0.15
88.70
1.27
8.24
1.76 0.05
0.11
8.68 0.65 87.85 0.38
Table 2: Generative performance of state-of-the-art approaches for generative modelling of radiance
fields scene datasets: Vizdoom 23 Replica 64]
VIN-CE 26 and ARKitScenes 11, according
to FID [6 and SwAV-FID 35 metrics.
In Fig. we show samples from the unconditional distribution learnt by GAUDI for different datasets. We observe that GAUDI is able to generate diverse and realistic 3D scenes from the empirical distribution which can rendered from the sampled camera poses.
4.5 Conditional Generative Modeling
GAUDI can also tackle conditional generative problems of the form p(Z)Y,, where conditioning variable y E Y given to condition p(Z). For given conditional inference problem we assume
existence of paired data 46
[1 45 As example, for training text-conditional model we assume
the existence of pairs (z, Vi) where z; a latent scene representation and y is corresponding
text prompt. In this section we show both quantitative and qualitative results for conditional inference
7

---

Figure 6: Different scenes sampled from unconditional GAUDI (one sample per row) and rendered from their corresponding sampled camera poses (one dataset per column): Vizdoom 231, Replica 64 VLN-CE [26] ARKitScenes [I]. The resolutions 64 x 64, 64 64, 128 128 and 64 x 64 respectively.
problems. The first conditioning variable we consider are textual descriptions of trajectories. Second, we consider model where randomly sampled RGB images in trajectory act conditioning. Finally, we use categorical variable that indicates the 3D environment (i.e. the particular indoor space) from which each trajectory was obtained. Tab. S shows quantitative results for the different conditional inference problems. Details on the implementation of the conditional DDPM are given in Sect.
Text Conditioning
Image Conditioning
Categorical Conditioning
Avg. PerEmiirommett
SWAV-FID
FID
SwAV-FFD
SwAV-FID
SwAV-FID
18.50
3.75
19.51
18.74
3.61
-4.10
Table 3: Quantitative results of Conditional Generative Modeling on VLN-CE 26 dataset. GAUDI is able to produce high-quality scene renderings with low FID and SwAV-FID scores. In the right table we show the difference in average per-environment FID score between the conditional and unconditional models.
4.5.1 Text Conditioning
We tackle the challenging task of training text conditional model for 3D scene generation. We use the navigation text descriptions provided in VLN-CE [26] to condition our model. These text descriptions contain high level information about the scene as well the navigation path (i.e. "Walk out of the bedroom and into the living room". "Exit the room through the swinging doors and then enter the bedroom"). We employ pre-trained RoBERTa-base (2] text encoder and use its intermediate representation to condition the diffusion model. Fig. 7 shows qualitative results of GAUDI for this task. To the best of our knowledge, this the first model that allows for conditional 3D scene generation from text in an amortized manner (i.e. without distilling CLIP through costly optimization problem
Go through the hallway
Go the stairs
Walk into the kitchen
Figure 7: Text conditional 3D scene generation using GAUDI (one sample per row). Our model is able to capture the conditional distributions of scenes by generating multiple plausible scenes and camera paths that match the given text prompts.

---

4.5.2 Image Conditioning
We now analyze whether GAUDI is able to pick up information from the RGB images to predict distribution over Z. this experiment we randomly pick images in a trajectory x: E and use it as conditioning variable For this experiment each training iteration we sample a random image for each trajectory a and use as conditioning variable. We employ pre-trained ResNet-18 [5]] as an image encoder. During inference, the resulting conditional GAUDI model is able to sample radiance fields where the given image is observed from a stochastic viewpoint. In Fig. g we show samples from the model conditioned different RGB images.
use trajectories in the VLN-CE dataset [26]. During
Figure 8: Image conditional 3D scene generation using GAUDI (one sample per row). Given conditioned image (top row), our model is able to sample scenes where the same or contextually similar view is observed from stochastic viewpoint.
Environment
Eavironnent ID:
Enviromment ID:
Figure 9: Samples from the GAUDI model conditioned on categorical variable denoting the indoor scene (one sample per row).
4.5.3 Categorical Conditioning
Finally, analyze how GAUDI performs when conditioned a categorical variable that indi- cates the underlying 3D indoor environment in which each trajectory was recorded. We perform experiments in the VLN-CE 26 dataset, where employ trainable embedding layer learn rep- resentation for categorical variables indicating each environment. We compare the per-enviromment FID score of conditional model with its unconditional counterpart. This per-enviroment FID score is computed only on real images of the same indoor enviromment that the model is conditioned on. Our hypothesis is that if the model efficiently captures the information in the conditioning variable it should capture the environment specific distribution better than its unconditional counterpart trained on the same data. In Tab. 3 the last column shows difference (e.g. the A) on the average envirrnrromnn FID score between the conditional and unconditional model on VLN-CE dataset. We observe that the conditional model consistently obtains a better FID score than the unconditional model across all indoor environments, resulting in sharp reduction of average FID and Sw SWAFFIID scores. In addition, in Fig. g we show samples from the model conditioned on given categorical variable.

---

5 Conclusion
We have introduced GAUDI, a generative model that captures distributions of complex and realistic 3D scenes, GAUDI uses a scalable two-stage approach which first involves learning latent repre- sentation that disentangles radiance fields and camera poses. The distribution of disentangled latent representations is then modeled with a powerful prior. Our model obtains state-of-the -art performance when compared with recent baselines across multiple 3D datasets and metries. GAUDI can be used both for conditional and unconditional problems, and enabling new tasks like generating 3D scenes from text descriptions.
References
[1] Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Yuri Feigin, Peter Fu, Thomas Gebauer, Daniel Kurz, Tal Dimry, Brandon Joffe, Arik Schwartz, et al. Arkitscenes: A diverse real-world dataset for 3d indoor scene understanding using mobile rgh-d data. In Phirty-fifth Conference on Neural Information Processing Systems Datasets and Benckmarks Track (Round 1), 2021.
[2] Piotr Bojanowski, Armand Joulin, David Lopez-Paz. and Arthur Szlam. Optimizing the latent space of generative networks. arXiv preprint 1rXiv: 2017. 2017.
[3] Marcus Carter and Ben Egliston. Ethical implications of emerging mixed reality technologies. 2020.
[4] Lluis Castrejon, Nicolas Ballas, and Aaron Courville. Improved conditional vrnns for video prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages (608-7617, 2019.
[5] Eric R Chan, Connor Z Lin, Matthew Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry-aware 3d generative adversarial networks. arXiv preprint arXiv: 2tt::2112000 07945,, 2021.
[6] Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. pi-GAN: Periodic implicit generative adversarial networks for 3d-aware image synthesis. arXiv preprint arXiv:2012.00926 2020.
[7] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. arXiv preprint arXiv: 1709006188 2017.
[8] Terrance Deiries, Miguel Angel Bautista, Nitish Srivastava, Graham W Taylor, and Joshua M Susskind. Unconstrained scene generation with locally conditioned radiance fields. ICCV, 2021.
[9] Emilien Dupont, Miguel Angel Bautista, Alex Colburn, Aditya Sankar, Carlos Guestrin, Josh Susskind, and Qi Shan. Equivariant neural rendering. International Conference on Machine Learning, pages 2761-2770. PMLR, 2020.
[10] Emilien Dupont, Hyunjik Kim, SM Ali Eslami, Danilo Jimenez Rezende, and Dan Rosenbaum. From data to functa: Your data point is function and you can treat it like one. In International Conference on Machine Learning, pages 5694-5725. PMLR, 2022.
[11] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high righ-recolution image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12873-12883, 2021.
[12] Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Thcobalt. Stylenerf: A style-based 3d-aware generator for high-resolution image synthesis. arXiv preprint arXiv:2110.08985) 2021.
[13] Pengsheng Guo, Miguel Angel Bautista, Alex Colburn, Liang Yang, Daniel Ulbricht, Joshua M Susskind, and Qi Shan. Fast and explicit neural view synthesis. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pages 3791 2022.
[14] David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv: 10122, 2,222,,0000 2018
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pallern recognition, pages 770-778, 2016.
[16] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. arXiv preprint arXiv:1706.0 08500, 2017.
10

---

[17] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[18] Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
[19] Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole. Zero-shot text-guided object generation with dream fields. arXiv preprint arXiv:21 2021.
[20] Niharika Jain, Alberto Olmo, Sailik Sengupta, Lydia Manikonda, and Subbarao Kambhampati. Imperfect imaganation: Implications of gans exacerbating hiases on facial data augmentation and snapchat selfie lenses. arXiv preprint 2000:2000, 2020.
[21] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4401 04400, 2019.
[22] Tero Karras, Samuli I Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 81 10-8119, 2020.
[23] Michal Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toezek, and Wojeiech Jaškowski. Vizdoom:
A boom-based ai research platform for visual reinforcement learning. In 2016 IEEE Conference on Computational Intelligence and Games (CIG), pages 1-8. IEFE, 2016.
[24] Seung Wook Kim, Jonah Philion, Antonio Torralba, and Sanja Fidler. Drivegan: Towards a controllable high-quality neural simulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 5820-5829, 2021.
[25] Adam R Kosiorek, Heiko Strathmann, Daniel Zoran, Pol Moreno, Rosalia Schneider, Sona Mokrá, and Danilo Jimenez Rezende. Nerf-vae: A geometry aware 3d scene generative model. In International Conference on Machine Learning, pages 5742-5752. PMIR, 2021.
[26] Jacob Krantz, Erik Wijmans, Arjun Majundar. Dhruv Batra, and Stefan Lee. Beyond the nav-graph: Vision and language navigation in continuous environments. In European Conference on Computer Vision (ECCV), 2020.
(27] Zihang Lai, Sifei Liu, Alexei A Efros, and Xiaolong Wang. Video autoencoder: self-supervised disentan- glement of static 3d structure and motion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9730-9740, 2021.
[28] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. arXiv preprint arXiv:2007. / //571, 2020.
[29] Yinhan Liu, Myle Ott, Naman Goyal. Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, arXiv preprint arv: 107 1692, 2019.
[30] Willi Menapace, Stéphane I Lathuilière, Sergey Tulyakov, Aliaksandr Siarohin, and Elisa Ricci. Playable video generation. In Proceddings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10061-10070, 2021.
[31] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction function space. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4460 4470. 2019.
[32] Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka. Text2mesh: Text-driven neural stylization for meshes. arXiv preprint arXiv:2112.03221. 2021.
[33] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. arXiv preprint arXiv:2003.088: 0893.4 2020.
[34] Yisroel Mirsky and Wenke Lee. The creation and detection of deepfakes: A survey. ACM Computing Surveys (CSUR), 54(1):1-41, 2021.
[35] Stanislav Morozov, Andrey Voynov, and Artem Babenko. On self-supervised image representations for GAN evaluation. In International Conference on Learning Representations, 2021.
11

---

[36] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, llya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112. 107.4 2021.
[37] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162-8171. PMI.R, 2021.
[38] Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes compositional generative neural feature fields. arXiv preprint 1xXtv:2011.12100 2020.
[39] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 165-174, 2019.
[40] Despoina Paschalidou, Amlan Kar, Maria Shugrina, Karsten Kreis, Andreas Geiger, and Sanja Fidler. Atiss: Autoregressive transformers for indoor scene synthesis, Advances in Neural Information Processing Systems, 34, 2021.
[41] Songyou Peng, Michael Niemeyer, Lars Mescheder, Mare Pollefeys, and Andreas Geiger. Convolutional occupancy networks. arXiv preprint arXiv:2003. 2, 2020.
[42] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reason- ing with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[43] Benjamin Planche, Xuejian Rong. Ziyan Wu, Srikrishna Karanam. Harald Kosch, YingLi Tian, Jan Ernst. and Andreas Hutter. Incremental scene synthesis. In Advances in Neural Information Processing Syxtems, pages 1668-16788 2019.
[44] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry. Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMIR, 2021.
[45] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831, PMLR, 2021.
[46] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
[47] Daniel Rebain, Mark Matthews, Kwang Moo Yi, Dmitry Lagun, and Andrea Tagliasacchi. LoInerf: Learn from one look. arXiv preprint arXiv:2111.09990 2021.
[48] Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14335-14345, 2021.
[49] Xuanchi Ren and Xiaolong Wang. Look outside the room: Synthesizing consistent long-term 3d scene video from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[50] Salah Rifai. Grégoire Mesnil, Pascal Vincent. Xavier Muller, Yoshua Bengio, Yann Dauphin, and Xavier Glorot. Higher order contractive auto-encoder. In Joint European conference on machine learning and knowledge discovery in databases, pages 645-660. Springer, 2011.
[51] Daniel Ritchie, Kai Wang, and Yu-an Lin. Fast and flexible indoor scene synthesis via deep convolu- tional generative models. In Proceedings the Recognition, pages 6182-6190, 2019.
the IEEF//VF Conference on Computer Vision and Pattern
[52] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. arXiv preprint arXiv:21 2021.
[53] Robin Rombach, Patrick Esser, and Bjöm Ommer. Geometry-free view synthesis: Transformers and no 3d priors, 2021.
[54] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and compster-assisted intervention, pages 234-241 241. Springer, 2015.
12

---

[55] Negar Rostamzadeh, Emily Denton, and Linda Petrini. Ethics and creativity in computer vision. arXiv preprini arXiv:2112.33111, 2021.
[56] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Flect, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding, 2022.
[57] Mchdi S. M. Sajjadi, Henning Meyer, Etienne Pot, Uns Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lucic. Daniel Duckworth, Alexey Dosovitskiy, Jakob Uszkoreit, Thomas Funkhouser, and Andrea Tagliasacchi. Scene Representation Transformer: Geometry- Novel View Synthesis Through Set-Latent Scene Representations. CVPR, 2022.
[58] Tim Salimans, Ian Goodfellow, Wajciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. NeurlPS' 16, page 2234-2242, Red Hook, NY, USA, 2016. Curran Associates Inc.
[59] Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied research. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9339-9347, 2019.
[60] Katja Schwarz, Viyi Liao, Michael Niemeyer, and Andreas Geiger. GRAF: Generative radiance fields for 3d-aware image synthesis. arXiv preprint arXiv:2007.02442 2020.
[61] Edward J Smith and David Meger. adveoeadrverssarri adversarial systems for 3d object generation and reconstruction. In Conference on Robot Learning, pages 87-96. PMLR, 2017.
[62] Jascha Sohl- Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256-2265. PMLR, 2015.
[63] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
[64] Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, et al. The replica dataset: digital replica of indoor spaces. arXiv preprint wxXiv:190.0057999 2019.
[65] Hoang Thanh- Tung and Truyen Tran. Catastrophic forgetting and mode collapse in gans. In 2020 International Juint Conference on Neural Networks (IJCNN) pages 1-10. IEEE, 2020.
[66] Patrick Tinsley, Adam Czajka, and Patrick Flynn. This face does not exist... but it might be yours! identity leakage in generative models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1320-1328, 2021.
[67] Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metrie & challenges. arXiv preprint arXiv: 1812.01717, 2018.
[68] Arash Vahdat, Karsten Kreis, and Jan Kautz. Score- based generative modeling in latent space. Advances in Neural Information Processing Systems, 34:1 11302, 2021.
[69] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[70] Xinpeng Wang, Chandan Yeshwanth, and Matthias Nießner. Sceneformer: Indoor scene generation with transformers. In 2021 International Conference on 3D Vision (3DV), pages 106-115. IEEE, 2021.
[71] Daniel Watson, William Chan, Jonathan Ho, and Mohammad Norouzi. Learning fast samplers for diffusion models by diffcrentiating through sample quality. In International Conference on Learning Representations, 2021.
[72] Daniel Watson, Jonathan Ho, Mohammad Norouzi, and William Chan. Learning to efficiently sample from diffusion probabilistic models. arXiv preprint
2021.
[73] Alex Yu, Sara Fridovich- Keil, Matthew Tancik. Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. arXiv preprint 112.5.211..05133. 2021.
13

---

[74] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa, pixelnerf: Neural radiance fields from one few images. arXiv preprint urXiv:2012.02190. 2020.
[75] Peng Zhou, Lingxi Xie, Bingbing Ni, and Qi Tian. Cips-3d: A 3d-aware generator of gans based on conditionall--nddeeedeen pixel synthesis. arXiv preprint arXiv:21 2021.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A] You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section ??.
Did you include the license to the code and datasets? [No] The code and the data are proprictary.
Did you include the license the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/amswers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] These will be discussed in appendix A
(c) Did you discuss any potential negative societal impacts of your work? [Yes] These will be discussed in appendix A
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] We will release code publicly upon acceptance.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] These details will clarified
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] These details will be clarified in
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Sect.
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you's 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
14

---

(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "wS5C2Ee7Q13", "reviewer": "Reviewer_TNJD", "summary": "This paper generalizes previous NeRF-based generative models which focus on single objects to 3D scenes modeling where the canonical coordinate system may not exist. Instead of assuming the camera pose distribution to be shared across different scenes, the authors study the case that camera moves in the scene and thus forms a trajectory. They disentangle the modeling of camera pose and scene representation into different latent variables (and decoders) and learned these variables in an encoder-less method. After the latent variables are learned, the authors further learn a diffusion-based prior on them to enable sampling. Experimental results demonstrate the model's ability for reconstruction, interpolation, generative modeling for both unconditional and conditional cases. ", "questions": "Please check the Strengths and Weaknesses part.\n\nA minor question I woild like the authors to clearify here is for line 133, what does the \"denoising recontrution objective\" mean here. As I understand, the authors just use l2 reconstruction loss for image and l1 reconstruction loss for pose. Then what does \"denoise\" mean or where does the nose come from? Does it just mean you perturbed z with additive noise during training? Given the authors also use denoising diffusion probablistic model for prior learning in the next part, I think it might be better to make this more clear.   ", "limitations": "I think the authors adequately addressed the limitations and potential negative societal impact of their work.\nBut one potential limitation here might be whether the model can be scaled up to larger scenes like a trajectory accross multiple different rooms or an outdoor scene. The current model uses a single tri-plane representation for the whole scene, which means that the movements of the agent are limited to a predefined area. Then what if the agent moves outside the boundary？Simply enlarging the size of the predefined area size for larger areas or simply enlarging the size of the tri-plane representation seems not to be very efficient.   ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "For Strengths:\n\nI think this paper study an important problem. Recently, the NeRF-based generative models show great potential on modeling single objects or simple scenes. On the other hand, modeling real-word, immersive 3D scenes where agent moves freely in the enviroment, is an important topic for many applications in robotics, VR/AR, but they are relatively under explored for NeRF-based generators. This paper proposes to solve this problem by disentangling the latent representations for camera pose and 3D enviroment. They also introduce the diffusion-based prior to enable sampling. Their model show good performance in both reconstruction and generation tasks. The paper is overall well-written and easy to follow. \n\nQuestions and potential weaknesses:\nI have several concerns/questions regarding the experiments.\n1. Novel view synthesis performance:  In 4.2 the authors show the reconstruction results. I assume these results are got from the trajectories/frames used to train the model. But how well does the model do for novel view synthesis? If during training we use a certain trajectory to get the scene representation $z_{scene}$ for a scene, then can this scene representation be generalized to model other trajectories in the same scene? (Or what will the PSNR/SSIM be if I use the learned scene representation to reconstruct a novel trajctory in the same room?) I think this is an important metrics to see whether the model just overfits the observed trajectories or it really builds a valid 3D representation.\n2. Test on novel objects: Similar to 1, how well can the model do to reconstruct a novel scene that has not be seen during training?\n3. For unconditional generation, the authors mainly demonstrate their model's ability through FID score, which is based on the quality of a single image. However, the model proposed here actually generates a trajectory in the scene. Would it be possible to compare the sample quality on the trajectory level (which, besides the quality of each frame, also consider the consistency among each frames)? A reference metrics here might be the FVD [1] score.\n4. For conditional generation, why there are no baselines?\n\n[1] Unterthiner, Thomas, et al. Towards Accurate Generative Models of Video: A New Metric & Challenges             \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "jSHGkHSBoxs", "reviewer": "Reviewer_gUke", "summary": "This paper proposes a generative model for both indoor scenes and camera pose trajectories. The method can be trained as an uncondintional sampler or as a onditional one given a reference image/ a text prompt. At the core of the method, is a Denoising diffusion model based latent sampler, together with a scene generator, a trajactory generator, and a NeRF-based rendering pipeline. Compared with previous method, the proposed method scales well for indoor scenes, and generates reasonable results. ", "questions": "1. As mentioned above, how robust is the method to camera pose registration errors? it seems the ARkit results is worse than the results from Replica.\n\n2. How generalizable it is to use a DDPM to sample the latents? Would it still apply to other generative tasks where simple priors fail?\n\n3. How robust is the method to succesfully train? \n\n4. It seems that the generated results lack temporal coherency. Though that's quite common for NeRF based generators, is it possible to visualize the generated geometry/depth maps as well? It seems the NeRF model is view-dependent, would a view independent model be better at consistency?\n\n", "limitations": "The authors do not provide limitation discussions; it would be really helpful to have them.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality: \nThe authors proposed a DDPM-based prior sampler together with trajectory and scene generation, which is a hard problem to tackel. The use of DDPM is well justified, as simple priors are limited in modeling different variations of camera poses and scene contents.\n\nQuality:\nThe image generation quality is limited, but is understandable since the task is hard. The authors provided enough results to faithfully represent the quality of the method, which is good. \n\nSignificance:\nI think this paper is of singinifcance especially to people interested in 3D aware video generation. But it should be noted that the method requires GT camera poses to train, and it seems the ARkit results are worse than Replica, potentially because the inaccuracies in camera calibration.\n\nClarity:\nThe paper is well written and easy to follow.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "K3qL-Gj05QT", "reviewer": "Reviewer_KBe3", "summary": "The paper proposed a method for the generative modeling of indoor scenes. The model contains a tri-plane based radiance field branch, and a camera trajectory branch. They are trained with indoor scene datasets that include RGBD images and ground truth camera trajectories. \n\nThe model employs an encoder-free architecture similar to DeepSDF [29]. Specifically, each individual scene in the dataset is associated with a learnable latent variable, which is optimized together with the model parameters during training. \n\nThe model is trained with RGB reconstruction loss, depth loss and loss between ground truth and predicted camera trajectories. After the auto-decoder model is trained, a diffusion model can be trained on top of the learned latents to enable sampling. The latent generative model can optionally be conditioned on text embeddings to achieve language-driven generation, provided that the dataset also contains language navigation.", "questions": "* There appears to be strong view inconsistencies in the interpolation video (in the Supp.) as the camera moves within the same scene (geometry or texture changes with camera pose). I wonder why is this happening?\n* I wonder how does the proposed method (auto-decoder + latent DDPM) compare with GAN based methods in terms of synthesizing novel scenes? Does it have the tendency of memorizing the existing training scenes instead of generating novel ones?", "limitations": "Covered in the supplemental material.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n* Language-driven neural scene synthesis is a challenging problem which can lead to interesting applications.\n* The paper shows that the encoder-free (auto-decoder) training objective used in DeepSDF [29] is also applicable to neural radiance fields. This significantly simplifies the training of a generalizable neural radiance field.\n* The paper demonstrates that when the access to camera poses and depth maps is available, a 3D generative model producing higher quality images can be obtained with the use of stronger supervision signals.\n\n### Weaknesses\n* The comparison with previous methods, shown in Table 2 is not a fair comparison. The proposed model is trained with extra information from the dataset, such as ground truth camera poses, depths, and which images belong to the same scenes, while the baseline methods are trained with unposed images only. This important difference is not addressed in the paper. Instead, the paper choose to attribute the improvement to the learning of better latents (L224-228).\n* Compared to previous methods, the proposed method puts scene-dependent constrains on the camera pose, limiting its use in arbitrary viewpoint synthesis. In fact, it appears that the model performed poorly on novel view/trajectories, as evident by Table 5 \"GAUDI w. Random Pose\" in the supplement material.\n* The benefit of adding perturbation $\\beta$ to the latent variables (L138) is not justified by the experiment. Supp. Table 4 shows that the reconstruction quality strictly decreases with larger $\\beta$. What's worse, Supp. Table 5 also suggests that large $\\beta$ affects generation performance while setting $\\beta = 0$ leads to one of the best performing models.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Hq8i3-o_gtw", "reviewer": "Reviewer_si6w", "summary": "This paper proposes a framework for 3D scene generation. The proposed framework enables the modeling of the scene distributions and scene-dependent camera distributions by learning priors from disentangled latent representations for radiance field and camera poses, which are obtained via optimizing a reconstruction objective over the training trajectories. Experimental results show that the proposed framework can be used for both unconditional and conditional generations. Extensive results are provided for ablation study to study the influence of key parameters and modules in the framework.", "questions": "1)\tSince the results are demonstrated via short rendered videos under sampled camera path, it’s not that friendly to observe the diversity of the generated scenes. For example, the provided results for Replica in the supplementary video seem very similar. Are there any chances that authors can provide some visualizations of the underlying meshes or global layouts of the generated scenes?\n\n2)\tIs the proposed model able to generate longer camera paths (e.g, camera paths that are longer than the provided ones?)\n\n3)\tCan the proposed model guarantee the continuity and validness of the sampled camera poses where a new latent representation is sampled from the Gaussian distribution? If so, how does the proposed model achieve this? Is it achieved by the noise perturbations in latent optimization and the camera pose normalization?\n\n4)\tHow long does it exactly take to generate a new scene?\n\n5)\tFor text-based conditional generation, there are many artifacts observed within the provided results, what is the possible reason for this phenomenon? \n", "limitations": "The authors clearly state the limitations and potential negative impact of the proposed framework.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The authors tackle the problem of 3D complex scene generation. For learning priors from a large amount of indoor scene observation trajectories, the authors propose to first learn a disentangled representation and then learn a generative model over the latent representations. The proposed method is technically sound. The writing and organization of this paper are good, and the authors provide sufficient experimental results in the manuscript and supplementary materials.\nThe technical part is not very easy to follow, especially for the conditional generation part. It would be better if more details can be given for better clarity.\n\nOverall, the problem setting and the technical route of this work make sense to me. But there are still converns over the work in its current form. See the issue of the technical part above and more in the following.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept