Title: SCube: Instant Large-Scale Scene Reconstruction using VoxSplats
Venue: NeurIPS 2024
Accepted: yes
Keywords: Large-scale Scene Reconstruction

Sparse-view Reconstruction

Sparse Voxels

Gaussian Splatting

Diffusion Models
Abstract: We present SCube, a novel method for reconstructing large-scale 3D scenes (geometry, appearance, and semantics) from a sparse set of posed images. Our method encodes reconstructed scenes using a novel representation VoxSplat, which is a set of 3D Gaussians supported on a high-resolution sparse-voxel scaffold. To reconstruct a VoxSplat from images, we employ a hierarchical voxel latent diffusion model conditioned on the input images followed by a feedforward appearance prediction model. The diffusion model generates high-resolution grids progressively in a coarse-to-fine manner, and the appearance network predicts a set of Gaussians within each voxel. From as few as 3 non-overlapping input images, SCube can generate millions of Gaussians with a 10243 voxel grid spanning hundreds of meters in 20 seconds. Past works tackling scene reconstruction from images either rely on per-scene optimization and fail to reconstruct the scene away from input views (thus requiring dense view coverage as input) or leverage geometric priors based on low-resolution models, which produce blurry results. In contrast, SCube leverages high-resolution sparse networks and produces sharp outputs from few views. We show the superiority of SCube compared to prior art using the Waymo self-driving dataset on 3D reconstruction and demonstrate its applications, such as LiDAR simulation and text-to-scene generation.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a method to reconstruct large-scale 3D scenes from a sparse set of posed images. The method has two stages: the first stage is for voxel grid reconstruction, which is based on the XCube [38], a 3D generative model, and the authors introduce the image condition to the model.  The second stage is for appearance reconstruction, which is a feedforward appearance prediction model to predict a certain number of Gaussians for each voxel. The authors also propose a GAN-based post-processing step to boost the appearance of the reconstructed scene. The proposed method is evaluated on the Waymo self-driving dataset, showing the superiority of the proposed method compared to prior art on 3D reconstruction tasks. Besides, the authors also demonstrate the applications of the proposed method, such as LiDAR simulation and text-to-scene generation.\n\nSTRENGTHS:\n- The proposed method addresses an important problem in 3D vision, i.e., reconstructing large-scale 3D scenes from a sparse set of posed images. The authors cleverly take advantage of the pre-trained 3D diffusion generative model to relieve the ill-posedness of the problem.\n- Although technically the proposed method is not entirely new, the paper presents a framework to combine the voxel grid reconstruction and appearance prediction, and the presented result looks promising to me.\n- The paper is well-written and easy to follow. The authors provide a clear explanation of the proposed method and the experiments.\n\nWEAKNESSES:\n- The proposed representation's abbreviation \"VoxSplats\" is somewhat confusing to me, since the term \"splatting\" in original 3DGS refers to the rendering technique that splats the 3D Gaussians onto the image plane, rather than the 3D representation. The proposed representation is more like a hybrid of voxel grids and Gaussian representation. The authors may consider using a more accurate term to describe the representation.\n- It's unclear how the foreground and background are separated in the proposed method. Using a panorama to represent the sky makes sense as the sky is usually far away from the scene, but some buildings are treated as the background and seem to be represented by the panorama as well in the supplementary video. A more detailed explanation and discussion of the foreground-background separation is needed.\n- There are some lighting/exposure/white-balance inconsistencies in the reconstructed scenes in the supplementary video. The authors should discuss the limitations of the proposed method in handling the lighting/exposure/white-balance variations in the input images.\n- For SCube+, how to guarantee consistency among the rendered views? Also, rendering with the proposed GAN-based post-processing step is time-consuming, which would destroy the valuable real-time rendering performance of the Gaussian Splatting. The authors should discuss these issues in the paper.\n- Missing the evaluation of the geometric accuracy of the reconstructed scenes. As the rendering quality of SCube is not so good, geometric accuracy is crucial to the applications of the reconstructed scenes. The authors should provide a detailed evaluation of the geometric accuracy of the reconstructed scenes.\n\nQUESTIONS:\nThe most important questions that I would like the authors to address are:\n- Geometric accuracy evaluation of the reconstructed scenes.\n\nLIMITATIONS:\nYes, the authors have discussed the limitations of the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
- The paper proposes a new method for sparse-view 3D reconstruction using 3DGS.
- The framework uses two stages:
    1) it learns a latent voxel grid (based on XCube) to represent the geometry.
    2) it trains an appearance model to decode the latent voxel grid into a set of Gaussians
- The method further uses a background model to handle the sky.
- The authors propose an (optional) GAN postprocessing which improves the visual quality at the cost of longer (20m) per-scene optimization.
- The method was evaluated on the Waymo Open Dataset, and it outperforms compared methods.

STRENGTHS:
- The paper is very well written and easy to follow.
- The method is novel and the results look good.
- I really like the idea of having probabilistic latent space representation (VAE).
- The paper contains an ablation study illustrating the tradeoff between model complexity and performance as various hyperparameters are changed.

WEAKNESSES:
- Section Method "Training and Inference" is not very clear. It would be preferable to expand the diffusion loss from Appendix A, and clearly describe the input/targets.
- I would also like to have more details on the Sky Panorama model.
- The method was evaluated on a single dataset. It would be interesting to see how the method performs on other datasets.

QUESTIONS:
- How does the method compare to other methods like MVSplatting, MVSGaussians? How does it differ? How do you think it would compare in terms of performance?

LIMITATIONS:
- Limitations are properly discussed in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

comments: SUMMARY:
The paper introduces a pipeline for street scene reconstruction given a sparse set of images as input. The reconstruction process follows a feed-forward manner. The method builds upon the existing XCube work. First, it generates sparse voxels of the scene to represent the geometries, then each voxel feature is decoded into Gaussian primitives for appearance rendering. Experimental results, both quantitative and qualitative, show that SCube can reconstruct the scene with high quality and provides benefits for downstream applications such as autonomous driving simulation.

STRENGTHS:
(1) The results are impressive. Given the sparse images with little overlap, the reconstructed Gaussians are of high quality and show good generalization ability in novel views.

(2) Overall, the method sounds reasonable. Technical details are provided, and I believe the results are reproducible.

WEAKNESSES:
(1)	I am curious if we really need a diffusion model for this task. Since there are several images as conditions, the uncertainty (or randomness) of the output should be very small. Why not just train a regressor for the sparse voxel reconstruction? Is there any specific motivation for using a diffusion model?

QUESTIONS:
(1)	Why is the GAN loss used independently for each scene in the inference stage, instead of using it in the training stage?

LIMITATIONS:
Currently, the results are presented in the sparse view settings of a small region of the street scene. How can we reconstruct a very large scene using a long sequence of images in a feed-forward manner? This is an interesting topic for future work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new method to reconstruct 3D outdoor scenes from a sparse set of posed images. The key idea is to utilize a new hybrid 3D representation, which assigns 3D Gaussians to each sparse voxel. Given input images, the paper first adapts XCube to condition on sparse view images. Along with the generated fine-level voxels, the paper then designs a feed-forward model to reconstruct the appearance. This proposed pipeline can also be used in text-to-scene generation. Experiments show that the proposed method outperforms existing approaches.\n\nSTRENGTHS:\nOriginality:\n\nThe paper is trying to solve a challenging problem: efficient large-scale scene reconstruction from sparse posed images. The motivation is to utilize the data prior to complement the sparse images. The proposed representation is a straightforward combination of XCube with 3D Gaussians (simply assigning 3D Gaussians into each voxel), which has also been widely applied (e.g., Scaffold-GS). While combining these two representations is acceptable, the concern is that the overall pipeline design seems more like a simple combination of two representations: first generating XCube, then generating 3D Gaussians with geometry & images. However, the problem to be solved is important, and this paper might inspire more researchers to work on this problem.\n\nQuality:\n\nThe submission seems technically sound.\n\nClarity:\n\nThe paper is mostly clear and easy to read.\n\nSignificance:\n\nThough there are some concerns with the methods and results, the paper is aiming at solving a very good problem.\n\nWEAKNESSES:\n- One concern is that the novelties of this paper are not very clear. As stated above (Originality), the paper seems to simply use two stages to combine the two representations in a simple way: one for XCube, and one for 3D Gaussians. It would be better if the author can clearly state the contributions.\n\n- As a two-stage method, it would be better to discuss more about: (1) First stage: how to evaluate whether the reconstructed voxels align with the input images? (2) Second stage: how robust is the model when the input images and the voxels are not aligned?\n\n- L187: Considering the temporal inputs, can the methods give consistent results temporally? Adding results might be better.\n\n- L244-245: In Table 1, SCube seems to have very similar results to PixelSplat for future prediction. Why? An obviously better reconstruction result alone seems not able to show the effectiveness of the proposed method. The appearance reconstruction uses generated geometry (fine-level voxels) as input, and adds some tricks such as sky modeling. It is unknown how they affect the results.\n\nQUESTIONS:\nSee Weaknesses\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}