Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposed a method for the generative modeling of indoor scenes. The model contains a tri-plane based radiance field branch, and a camera trajectory branch. They are trained with indoor scene datasets that include RGBD images and ground truth camera trajectories. 

The model employs an encoder-free architecture similar to DeepSDF [29]. Specifically, each individual scene in the dataset is associated with a learnable latent variable, which is optimized together with the model parameters during training. 

The model is trained with RGB reconstruction loss, depth loss and loss between ground truth and predicted camera trajectories. After the auto-decoder model is trained, a diffusion model can be trained on top of the learned latents to enable sampling. The latent generative model can optionally be conditioned on text embeddings to achieve language-driven generation, provided that the dataset also contains language navigation.
strengths_and_weaknesses: ### Strengths
* Language-driven neural scene synthesis is a challenging problem which can lead to interesting applications.
* The paper shows that the encoder-free (auto-decoder) training objective used in DeepSDF [29] is also applicable to neural radiance fields. This significantly simplifies the training of a generalizable neural radiance field.
* The paper demonstrates that when the access to camera poses and depth maps is available, a 3D generative model producing higher quality images can be obtained with the use of stronger supervision signals.

### Weaknesses
* The comparison with previous methods, shown in Table 2 is not a fair comparison. The proposed model is trained with extra information from the dataset, such as ground truth camera poses, depths, and which images belong to the same scenes, while the baseline methods are trained with unposed images only. This important difference is not addressed in the paper. Instead, the paper choose to attribute the improvement to the learning of better latents (L224-228).
* Compared to previous methods, the proposed method puts scene-dependent constrains on the camera pose, limiting its use in arbitrary viewpoint synthesis. In fact, it appears that the model performed poorly on novel view/trajectories, as evident by Table 5 "GAUDI w. Random Pose" in the supplement material.
* The benefit of adding perturbation $\beta$ to the latent variables (L138) is not justified by the experiment. Supp. Table 4 shows that the reconstruction quality strictly decreases with larger $\beta$. What's worse, Supp. Table 5 also suggests that large $\beta$ affects generation performance while setting $\beta = 0$ leads to one of the best performing models.

questions: * There appears to be strong view inconsistencies in the interpolation video (in the Supp.) as the camera moves within the same scene (geometry or texture changes with camera pose). I wonder why is this happening?
* I wonder how does the proposed method (auto-decoder + latent DDPM) compare with GAN based methods in terms of synthesizing novel scenes? Does it have the tendency of memorizing the existing training scenes instead of generating novel ones?
limitations: Covered in the supplemental material.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a generative model for both indoor scenes and camera pose trajectories. The method can be trained as an uncondintional sampler or as a onditional one given a reference image/ a text prompt. At the core of the method, is a Denoising diffusion model based latent sampler, together with a scene generator, a trajactory generator, and a NeRF-based rendering pipeline. Compared with previous method, the proposed method scales well for indoor scenes, and generates reasonable results. 
strengths_and_weaknesses: Originality: 
The authors proposed a DDPM-based prior sampler together with trajectory and scene generation, which is a hard problem to tackel. The use of DDPM is well justified, as simple priors are limited in modeling different variations of camera poses and scene contents.

Quality:
The image generation quality is limited, but is understandable since the task is hard. The authors provided enough results to faithfully represent the quality of the method, which is good. 

Significance:
I think this paper is of singinifcance especially to people interested in 3D aware video generation. But it should be noted that the method requires GT camera poses to train, and it seems the ARkit results are worse than Replica, potentially because the inaccuracies in camera calibration.

Clarity:
The paper is well written and easy to follow.

questions: 1. As mentioned above, how robust is the method to camera pose registration errors? it seems the ARkit results is worse than the results from Replica.

2. How generalizable it is to use a DDPM to sample the latents? Would it still apply to other generative tasks where simple priors fail?

3. How robust is the method to succesfully train? 

4. It seems that the generated results lack temporal coherency. Though that's quite common for NeRF based generators, is it possible to visualize the generated geometry/depth maps as well? It seems the NeRF model is view-dependent, would a view independent model be better at consistency?


limitations: The authors do not provide limitation discussions; it would be really helpful to have them.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a framework for 3D scene generation. The proposed framework enables the modeling of the scene distributions and scene-dependent camera distributions by learning priors from disentangled latent representations for radiance field and camera poses, which are obtained via optimizing a reconstruction objective over the training trajectories. Experimental results show that the proposed framework can be used for both unconditional and conditional generations. Extensive results are provided for ablation study to study the influence of key parameters and modules in the framework.
strengths_and_weaknesses: The authors tackle the problem of 3D complex scene generation. For learning priors from a large amount of indoor scene observation trajectories, the authors propose to first learn a disentangled representation and then learn a generative model over the latent representations. The proposed method is technically sound. The writing and organization of this paper are good, and the authors provide sufficient experimental results in the manuscript and supplementary materials.
The technical part is not very easy to follow, especially for the conditional generation part. It would be better if more details can be given for better clarity.

Overall, the problem setting and the technical route of this work make sense to me. But there are still converns over the work in its current form. See the issue of the technical part above and more in the following.
questions: 1)	Since the results are demonstrated via short rendered videos under sampled camera path, it’s not that friendly to observe the diversity of the generated scenes. For example, the provided results for Replica in the supplementary video seem very similar. Are there any chances that authors can provide some visualizations of the underlying meshes or global layouts of the generated scenes?

2)	Is the proposed model able to generate longer camera paths (e.g, camera paths that are longer than the provided ones?)

3)	Can the proposed model guarantee the continuity and validness of the sampled camera poses where a new latent representation is sampled from the Gaussian distribution? If so, how does the proposed model achieve this? Is it achieved by the noise perturbations in latent optimization and the camera pose normalization?

4)	How long does it exactly take to generate a new scene?

5)	For text-based conditional generation, there are many artifacts observed within the provided results, what is the possible reason for this phenomenon? 

limitations: The authors clearly state the limitations and potential negative impact of the proposed framework.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper generalizes previous NeRF-based generative models which focus on single objects to 3D scenes modeling where the canonical coordinate system may not exist. Instead of assuming the camera pose distribution to be shared across different scenes, the authors study the case that camera moves in the scene and thus forms a trajectory. They disentangle the modeling of camera pose and scene representation into different latent variables (and decoders) and learned these variables in an encoder-less method. After the latent variables are learned, the authors further learn a diffusion-based prior on them to enable sampling. Experimental results demonstrate the model's ability for reconstruction, interpolation, generative modeling for both unconditional and conditional cases. 
strengths_and_weaknesses: For Strengths:

I think this paper study an important problem. Recently, the NeRF-based generative models show great potential on modeling single objects or simple scenes. On the other hand, modeling real-word, immersive 3D scenes where agent moves freely in the enviroment, is an important topic for many applications in robotics, VR/AR, but they are relatively under explored for NeRF-based generators. This paper proposes to solve this problem by disentangling the latent representations for camera pose and 3D enviroment. They also introduce the diffusion-based prior to enable sampling. Their model show good performance in both reconstruction and generation tasks. The paper is overall well-written and easy to follow. 

Questions and potential weaknesses:
I have several concerns/questions regarding the experiments.
1. Novel view synthesis performance:  In 4.2 the authors show the reconstruction results. I assume these results are got from the trajectories/frames used to train the model. But how well does the model do for novel view synthesis? If during training we use a certain trajectory to get the scene representation $z_{scene}$ for a scene, then can this scene representation be generalized to model other trajectories in the same scene? (Or what will the PSNR/SSIM be if I use the learned scene representation to reconstruct a novel trajctory in the same room?) I think this is an important metrics to see whether the model just overfits the observed trajectories or it really builds a valid 3D representation.
2. Test on novel objects: Similar to 1, how well can the model do to reconstruct a novel scene that has not be seen during training?
3. For unconditional generation, the authors mainly demonstrate their model's ability through FID score, which is based on the quality of a single image. However, the model proposed here actually generates a trajectory in the scene. Would it be possible to compare the sample quality on the trajectory level (which, besides the quality of each frame, also consider the consistency among each frames)? A reference metrics here might be the FVD [1] score.
4. For conditional generation, why there are no baselines?

[1] Unterthiner, Thomas, et al. Towards Accurate Generative Models of Video: A New Metric & Challenges             

questions: Please check the Strengths and Weaknesses part.

A minor question I woild like the authors to clearify here is for line 133, what does the "denoising recontrution objective" mean here. As I understand, the authors just use l2 reconstruction loss for image and l1 reconstruction loss for pose. Then what does "denoise" mean or where does the nose come from? Does it just mean you perturbed z with additive noise during training? Given the authors also use denoising diffusion probablistic model for prior learning in the next part, I think it might be better to make this more clear.   
limitations: I think the authors adequately addressed the limitations and potential negative societal impact of their work.
But one potential limitation here might be whether the model can be scaled up to larger scenes like a trajectory accross multiple different rooms or an outdoor scene. The current model uses a single tri-plane representation for the whole scene, which means that the movements of the agent are limited to a predefined area. Then what if the agent moves outside the boundary？Simply enlarging the size of the predefined area size for larger areas or simply enlarging the size of the tri-plane representation seems not to be very efficient.
review_title: null

Meta-review: metareview: This paper proposes a framework to learn disentangled latent representation of radiance field and camera pose from trajectories of 3D scenes. The denoising diffusion probabilistic model can be further trained on the extracted latent representation for a conditioned or unconditioned generation. Experiments are conducted to validate the performance of the proposed method. The paper receives a total of 4 reviews. All reviewers lean to (borderline/weakly) accept the paper because of the novelty of the tasks, even though most of them raised concern about the view-consistency problem. AC recommends accepting the paper because the task of generating egocentric video, with an option of text-prompt input, is a novel and interesting direction, and this paper's merit outweighs its flaws.  AC urges the authors to improve their paper by taking into account all the feedback from the reviewers.
confidence: Certain
Decision: Accept