Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The work tackles the problem of 3D scene generation, learning from sequences of RGB-D  images and its poses. The main contribution of the work comes from proposing an algorithm that uses a generative sensing module and a mapping module. The proposed method was evaluated on CLEVR and GoogleEarth dataset and achieves SOTA results on the former. The method is also more efficient compared to previous results.

strengths_and_weaknesses: Strengths: 

- The proposed method SGAM uses generative sensing module and mapping module, where the former is a VQGAN and the latter is mostly KinectFusion using volumetric representations. The formulation is a sound integration of the neural network with previous methods. 

- The paper is well written and easy to understand

- The proposed method achieves SOTA results on CLEVR dataset on image based metrics and also on 3D generative metrics (MMD, JSD, 1-NNA…). 

- I enjoyed watching the results in video. I appreciate the authors for the hard work. 


Weaknesses: 

- My main concern lies in the experimental section, especially on real-world datasets. The main experiment was mainly on the CLEVR dataset. As the authors mentioned, the essence of scene generation comes from generating diverse but realistic results. Although the CLEVR dataset is a good way to show the distributional similarity between the generated set and the real distribution, the main limitation is that this is a synthetic dataset. The appendix in Table 4 shows that the method shows worse results on real dataset. I would like to hear from the authors regarding the discrepancy on the evaluation metrics on real and synthetic datasets.

- Compared to other methods [1, 2], the method requires accurate depth. What happens if the method is trained on ACID dataset, where the accurate depth information is unavailable? I regard this as an important problem, since many of the real-world applications do not provide the exact depth information. 


Summary:
The proposed method that leverages generative sensing module and a mapping module makes sense. However, my main concern lies in the experimental section, where the method achieves best results only on synthetic dataset and requires accurate depth. I would like to hear the response from authors regarding the experimental section and listen to other reviewers before making the final decision.

===============================================================================

After rebuttal, I'm convinced that the proposed method has the strength of generating long-term, globally consistent scenes. Therefore I'm willing to increase the score. However, I strongly encourage the authors to compare the method against previous methods such as ACID and RealEstate10k datasets to show the strength/weakness in real/standardized datasets. I do not think achieving SOTA results on these datasets is crucial since the method has clear strength, but the method should at least be competitive to verify that the method works on real-world datasets.
questions: - In the video there seems to be flickers in GoogleEarth dataset around 1:54, where the scene changes abruptly? Could the authors explain why this happens? Or is this just another video and not a continuous scene change.

limitations: Yes, the authors addressed the limitations of the work. But I recommend adding the comments regarding the method requires exact depth information.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a large scale scene generation method which promotes global 3D scene consistency. The method works by taking as input a random view of a scene, and then moving through the scene continually using the current understanding/representation to create incomplete observations from new perspectives, using a generative model to in-paint the incomplete regions, and then using the newly generated information to update the understanding/representation of the scene. With this setup they demonstrate impressive scene generations on a dataset made up of simples shapes scattered around a white room, and on google street view images. 
strengths_and_weaknesses: This paper is interesting and the results are impressive. The method is strong, and integrates lots of different techniques from different areas to produce a novel and successful solution to a hard problem. 

With respect to the method I am slightly skeptical how well it will scale to more complex scenes. The underlying 3D representation is a voxel grid, and while hashing is performed to make it more efficient, I am not sure if this will support complex detail that natural scenes poses. It might be good to speak to this in the limitations section. 

Connected to this I comment I would have also liked to have seen applications of this method to complex indoor scenes, as shown in the GFVS paper. I realize you require depth which is not available in general for indoor image datasets, however there exists some simulated indoor datasets for which you could again capture depth in blender. This would really help to understand how well your model scales, as at the moment you only demonstrate performance on practically 2D maps. 
questions: How difficult is this process to train? Have you observed consistent convergence to attractive scenes over multiple seeds? Does this green color bias effect all seeds similarly? 

If I understand correctly the camera angle is fixed throughout the trajectory? Have you explored at all with random trajectories through the space with the camera orientation changing as you move? I would be interested how the method performed with less structured passes through the search space.
limitations: Yes
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The paper proposes a method to generate a large-scale RGBD image in a near NADIR view. It uses a standard image generative model VQGAN while using the already-generated scenes as an input constraint. It qualitatively and quantitatively evaluate on the Clever dataset. For Google earth data, only qualitative evaluations are given.
strengths_and_weaknesses: To me, this paper is over-selling. The paper is just a 4 channel image generation by repeatedly applying a standard VQGAN. Simultaneous generation and mapping does not make much sense.

The core of the method is an application of an existing technique and the technical contribution is weak. I also do not like the way authors present their method to handle a general camera pose, while in reality they just use near nadir views.

Also, CLEVER scene is a bit too simple and through evaluations should be given on real data (Google Earth).


questions: I want answers to my criticism (weakness analysis) above.
limitations: The limitation description is OK. I cannot think of a better one.
review_title: null

Meta-review: metareview: The paper explores generation of a volumetric (voxel map via hashing, a la KinectFusion et seq) from a sequence of 2D images.  This is achieved by synthesizing sensor images, and feeding them into a mapping module (like KinectFusion).  As the reviewers note, this is an interesting goal, and the approach is reasonable, and no novelty of the overall system was questioned.

However, the reviewers concur that the synthetic dataset is not sufficient to give confidence that the system is effective in practice.  The rather more limited Google Earth (GE) examples, do, however, provide evidence that this strategy is effective.

As a reader, the most important figure for me is Fig 3 in the supmat - a qualitative view of the rerendered GE scene.  It is quite clear that the rerendered scenes have learned the characteristic 3D structures of the GE datasets, indicating that the scene-specific training data is a strong contributor to the results.  

The paper would have been stronger if such qualitative results had been shown for ACID scenes, either trained on Google maps (even if the training needed to be e.g. coastline specific), or trained on depth-from-stereo as indicated in the rebuttal. 

confidence: Certain
Decision: Accept