SCube: Instant Large-Scale Scene Reconstruction using VoxSplats

Authors: xuanchi  ren
yifan  lu
hanxue  liang
zhangjie  wu
huan  ling
mike  chen
sanja  fidler
francis  williams
jiahui  huang
  nvidia
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-04 06:37:00

Abstract: We present SCube, a novel method for reconstructing large-scale 3D scenes (geometry, appearance, and semantics) from a sparse set of posed images. Our method encodes reconstructed scenes using a novel representation VoxSplat, which is a set of 3D Gaussians supported on a high-resolution sparse-voxel scaffold. To reconstruct a VoxSplat from images, we employ a hierarchical voxel latent diffusion model conditioned on the input images followed by a feedforward appearance prediction model. The diffusion model generates high-resolution grids progressively in a coarse-to-fine manner, and the appearance network predicts a set of Gaussians within each voxel. From as few as 3 non-overlapping input images, SCube can generate millions of Gaussians with a 10243 voxel grid spanning hundreds of meters in 20 seconds. Past works tackling scene reconstruction from images either rely on per-scene optimization and fail to reconstruct the scene away from input views (thus requiring dense view coverage as input) or leverage geometric priors based on low-resolution models, which produce blurry results. In contrast, SCube leverages high-resolution sparse networks and produces sharp outputs from few views. We show the superiority of SCube compared to prior art using the Waymo self-driving dataset on 3D reconstruction and demonstrate its applications, such as LiDAR simulation and text-to-scene generation.

Main paper: 
 introduction: 
 Recovering 3D geometry and appearance from images is a fundamental problem in computer vision and graphics which has been studied for decades. This task lies at the core of many practical applications spanning robotics, autonomous driving, and augmented reality; just to name a few. Early algorithms tackling this problem use stereo matching and structure from motion (SfM) to recover 3D signals from image data (e.g. [44] ). More recently, a line of work starting from Neural Radiance Fields [32] (NeRFs) has augmented traditional SfM pipelines by fitting a volumetric field to a set of images, which can be rendered from novel views. NeRFs augment traditional reconstruction pipelines by encoding dense geometry, and view-dependent lighting effects. While radiance-field methods present a drastic step forward in our ability to recover 3D information from images, they require a time-consuming per-scene optimization scheme. Furthermore, since each scene is recovered in isolation, radiance fields do not make use of data priors, and cannot extrapolate reconstructions away from the input views. Thus, radiance-field methods require dense view coverage in order to produce high-quality 3D reconstructions. Another recent line of work applies deep learning to predict 3D from images. These methods either meta-learn an initialization to the radiance-field optimization problem [7, 30, 49] , or directly predict 3D from images using a feed-forward network [17, 57, 73] . While learning-based approaches can produce reconstructions from sparse views, they have only been used successfully for the case of single objects at low resolutions. Furthermore, these methods often suffer from 3D inconsistencies (e.g. the multi-layer surface or the Janus problem). In order to solve the general 3D reconstruction from images problem, we need methods that can (1) generalize reconstruction to general scenes over the pure object case, (2) produce accurate and high-quality reconstructions in the presence of dense views, leveraging data priors to produce plausible reconstructions in the sparse-view regime, and(3) run quickly and efficiently (in terms of runtime and memory) on large-scale and high-resolution inputs. These demands are difficult to satisfy in practice since high-quality ground-truth 3D data is not widely available for scenes, 3D representations for deep learning that scale to large and diverse inputs are under-explored in the literature, and corresponding scalable and easy-to-train model designs need to be developed alongside any new 3D representation.Nevertheless, we remark that some of these issues have been resolved in isolation: Gaussian Splatting [23] enables fast, differentiable rendering and high reconstruction quality (but is not being used with data priors), and sparse voxel hierarchies [40] have been successfully used to build generative models of large-scale 3D scenes with attributes such as semantics and colors, and have been trained on partial data such as LiDAR scans from autonomous vehicle captures.In light of the above observations, we introduce SCube, a feed-forward method for large 3D scene reconstruction from images. Our method encodes 3D scenes as a hybrid of Gaussian splats (which enable fast rendering), supported on a sparse-voxel-hierarchy (which enables efficient generative modeling of large 3D scenes with semantics). We call this hybrid representation VoxSplats and predict a VoxSplat from images using a feed-forward process consisting of two steps: (1) A generative geometry network that predicts a sparse voxel hierarchy conditioned on input images, and (2) an appearance network that predicts the Gaussian attributes within the voxels as well as a skybox texture to represent the background. The networks are implemented using highly efficient sparse convolution [14, 40] designed for 3D data which enables us to reconstruct a full scene from images in under 20 seconds. We evaluate our performance on the Waymo Open Dataset [53] on the challenging task of reconstructing a scene from sparse images with low overlap. We show that SCube significantly outperforms existing methods on this task. Furthermore, we demonstrate that SCube enables downstream applications such as LiDAR simulation and text-to-scene generation.2 Related Work 3D Scene Representation. Scenes in the wild are often large in scale and contain complicated internal structures which cause representations such as tri-planes [12] , dense voxel grids [36] , or meshes [19, 46] to fail due to capacity or memory limitations. Optimization-based reconstruction methods [15, 32] use high-resolution hash grids [1, 33] , but these are non-trivial to infer using a neural (2) We predict the appearance of the scene represented as VoxSplats and a sky panorama using a feedforward network. Our method allows us to synthesize novel views in a fast and accurate manner, along with many other applications.network [30] . In contrast, sparse voxel grids are effective for learning scene-reconstruction [40, 75] thanks to efficient sparse neural operators [8, 54] . Recently, Gaussian splatting [23] has enabled real-time neural rendering and has been applied to overfitting large scenes [66, 76] . [31, 39] use a hybrid of the above two representations, but the voxel grid or octree is only used to regularize the Gaussian positions without any data priors learned. This is in contrast to our VoxSplat that allows reconstruction in a direct inference pass thanks to the efficiency of sparse grids and the high representation power of Gaussian splats. We support operating only on sparse-view images, significantly lifting the input requirements by learning from large datasets.Sparse-view 3D Reconstruction. Sparse-view images often contain insufficient correspondences required by traditional reconstruction methods [44] . One line of work uses learned image-space priors such as depth [9] , normal maps [70] , and appearance from GANs [43] or diffusion models [63] to augment an optimization process such as NeRF. To speed up inference, another line of work uses a feed-forward model to predict renderable features [4, 6, 22, 29, 57, 73] . Alternatively, some papers perform learning directly in 3D space, which yields better consistency and less distortion [5, 13, 17, 68] . Our setting is similar to [13] where input images come from the same rig, but ours is more challenging since we do not use temporally-sequenced inputs with high overlap. We remark that semantic scene completion works [21, 26, 51, 61] also reconstruct voxels but at much lower resolutions and without appearance.Generative Models for 3D. 3D reconstruction can also be formulated as a conditional generative problem (i.e.modeling the distribution of scenes given partial observations). Text and single-image to-3D generation has been explored for objects [17, 28, 38, 47, 55, 56, 60, 69] . Extending this task to large-scenes is comparatively unexplored, and object-based methods often fail due to scaling limitations or assumptions on the data. [48, 72] recursively apply an image generative model to inpaint missing regions in 3D, but produces blurry reconstructions at a limited scale. XCube [40] is among the first to directly learn high-resolution 3D scene priors. Here, we extend this model with multiview image conditioning and enable it to predict appearance on top of geometry. 
method: 
 Our method reconstructs a high-resolution 3D scene from N sparse images I = {I i } N i=1 in two stages: (1) We reconstruct the scene geometry represented as a sparse voxel grid G with semantic features ( § 3.1). (2) We predict the appearance A of the scene that allows for high-quality novel view synthesis ( § 3.2) using VoxSplats and a sky panorama. We can express our pipeline as taking samples from the distribution p(G, A|I) = p(A|G, I)p(G|I). In order to improve the final view quality of the output, we apply an optional post-processing step discussed in § 3.3.Background: 3D Generation with XCube. XCube [40] is a 3D generative model that produces high-quality samples for both objects and large outdoor scenes. XCube uses a hierarchical latent diffusion model to generate sparse voxel hierarchies, i.e., a hierarchy of sparse voxel grids where each fine voxel is contained within a coarse voxel. XCube learns a distribution over latent X encoded by a sparse structure Variational Autoencoder (VAE). Both the VAE and the diffusion model are instantiated with sparse convolutional neural networks [14] , and can generate geometry at up to 1024 3 resolution. We use XCube as the backbone for our geometry reconstruction module. We remark that while the original paper only focused on unconditional or text-conditioned generation, we introduce a novel image-based conditioning C.Image Conditioned Geometry Generation. To condition XCube on posed input images, we lift DINO-v2 [34] features computed on the input images to 3D space as follows. First, we use the pre-trained DINOv2 model to extract robust visual features for input images, and process the DINO feature using several trainable 2D conv layers to reduce the feature channel to C + D. We then split the channel C + D into two parts for each pixel j and input image index i: one part is a regular C-dimensional feature F i j and the other will be a D-dimensional Softmax-normalized vector θ i j ∈ R D . Here θ i j can be viewed as a distribution over the depth of the corresponding pixel, and we follow a strategy similar to LSS [37] to unproject the images into a dense 3D voxel grid Ω where v denotes the index of a voxel and d ∈ [1, D] indexes the depth buckets:EQUATIONNote that we quantize the depth into D bins dividing the range from a predefined z near to z far . Unlike image-conditioning techniques used in object-level or indoor-level datasets where the camera frusta have significant overlap, our large-scale outdoor setting only takes sparse low-overlapping views captured from an ego-centric camera. Hence previous methods [28, 50, 52] that broadcast the same features to all the voxels along the rays corresponding to the pixel are not suitable here to precisely locate the geometries such as vehicles. The use of the weight θ allows us to handle occlusions effectively and produce a more accurate conditioning signal. After building C, we directly concatenate it with the latent X and feed it into the XCube diffusion network as conditioning.Training and Inference. Our training pipeline is similar to [40] , where we first train a VAE to learn a latent space over sparse voxel hierarchies. We add semantic logit prediction as in [40] to the grid and empirically find that it helps the model to learn better geometry. Then we train the diffusion model conditioned on C using the following loss:EQUATIONwhere L Diffusion is the loss for diffusion model training (see Appendix A for details). Focal(•) is the multi-class focal loss [27] . This additional depth loss is an explicit supervision to properly weigh the image features and encourage correct placement into the corresponding voxels. Due to the generative nature of XCube, we could learn the data prior to generate complete geometry even if some of the ground-truth 3D data is incomplete.The VoxSplat Representation. In the second stage, we fix the voxel grid G generated from the geometry stage and predict a set of Gaussian splats in each voxel to model the scene appearance.Gaussian splatting [23] is a powerful 3D representation technique that models a scene's appearance volumetrically as sum of Gaussians:EQUATIONwhere α ∈ [0, 1] is the opacity, µ ∈ R 3 is the center of each Gaussian, and Σ = RSS ⊤ R ⊤ ∈ R 3×3 is its covariance. The covariance matrix is factorized into a rotation matrix R parameterized by a quaternion q and a scale diagonal matrix S = diag(s). Each Gaussian additionally stores a color value RGB. Note that the original paper uses a set of SH coefficients for view-dependent colors, but we only use the 0 th -order SH in our model (i.e., without view-dependency) which we found to be sufficient for sparse-view reconstruction.While the original Gaussian Splatting paper and its follow-ups [23, 67, 71] propose many heuristics to optimize the positions of Gaussians for a given scene, we instead choose to predict M Gaussians pervoxel using a feed-forward model. We limit the positions of the Gaussians within a neighborhood of their supporting voxels, thus preserving the geometric structure of the supporting grid. By grounding the splats on a voxel scaffold, our reconstructions achieve better geometric quality without resorting to heuristics. Fittingly, we dub our voxel-supported Gaussian splats VoxSplats.The output of our network is {[ μv , ᾱv , sv , qv , RGB v ] ∈ R 14 } M for each voxel v. To compute the per-Gaussian parameters used for rendering we apply the following activations:EQUATIONwhere Center v is the centroid of the voxel v, and r is a hyperparameter that controls the range of a Gaussian within its supporting voxel. Here, we set r to three times the voxel size. We can efficiently render the Gaussians predicted by our model using rasterization [23] or raytracing [11] .Sky Panorama for Background. To capture appearance away from the predicted geometry, our model builds a sky feature panorama L ∈ R Hp×Wp×Cp from all input images, which can be considered as an expanded unit sphere with an inverse equirectangular projection. For each pixel in the panorama L, we get its cartesian coordinate P = (x, y, z) on the unit sphere and project P to the image plane to retrieve the image feature; since only the view direction decides the sky color, we zero the translation part of the camera pose in the projection step. We also apply a sky mask to ensure the panorama only focuses on the sky region.To render a novel viewpoint with its extrinsics and intrinsics, we recover the background appearance by sampling the sky panorama and decoding it into RGB values. For each camera ray from the novel view, we calculate its pixel coordinate on the 2D sky panorama L with equirectangular projection and get the sky features via trilinear interpolation, resulting in a 2D feature map for the novel view. We finally decode the 2D feature map with a CNN network to get the background image I bg , which will be alpha-composited with the foreground image from Gaussian rasterization:EQUATIONwhere I GS (u, v) is the rendered image of Gaussians, (u, v) indicates the pixel coordinate, and T(u, v) is the accumulated transmittance map of the Gaussians (see [23] for details).Architecture Details. We predict the (M × 14)-dimensional vector {[ μv , ᾱv , sv , qv , RGB v ]} M for each voxel via a 3D sparse convolutional U-Net which takes as input the sparse voxel grid Ω outputted by the geometry stage, where each voxel contains a feature sampled from the input images as follows: We process each input image I i using a CNN to get the image feature, and then cast a ray from each image pixel into Ω, accumulating the feature in the first voxel intersected by that ray. Voxels that are not intersected by any rays receive a zero feature vector.For the sky panorama model, we use the same image feature as above. In the training stage, we set smaller H p and W p for faster training and lower memory usage; in the inference stage, we increase H p and W p to get a sharper and more detailed sky appearance.Given a set of training images {I i gt } i and sky masks {M i } i distinct from the inputs, we supervise the appearance model using the loss:EQUATIONwhere the training views I i gt are sampled from nearby 10 views of the input images; the predicted views I i pred and transmittance masks T i are rendered using Eq (5); and L LPIPS /L SSIM are perceptual and structural metrics defined in [74] and [59] . We add COLMAP [44] dense reconstruction points to the accumulated LiDAR points and compensate for dynamic objects using their bounding boxes. This provides us with a more complete geometry for training.Optional GAN Postprocessing. The novel views directly rendered from our appearance model sometimes suffer from voxelization artifacts or noise. We resolve this with an optional lightweight conditional Generative Adversarial Network (GAN) that takes the rendered images as input and outputs a refined version. The discriminator of this GAN takes 256×256 image patches sampled from the input sparse view images, as well as the generated images conditioned on the rendered images. Drawing inspiration from [41, 43, 45] , we fit the GAN independently for each scene at inference time, which takes ∼20min to train. Due to the excessive time cost, we apply this step optionally only when higher-quality images are needed (which we call SCube+). Fig. 8 shows the results with and without this step, and we further present a general postprocessing without per-scene optimization in Appendix C.Application: Consistent LiDAR Simulation. LiDAR simulation [77] aims at reproducing the point cloud output given novel locations of the sensor and is an important application for training and verification of autonomous driving systems. The generated LiDAR point clouds should accurately reflect the underlying 3D geometry and a sequence of LiDAR scans should be temporally consistent. Our method enables converting sparse-view images directly into LiDAR point clouds, i.e., a sensorto-sensor conversion scheme. To achieve this, we leverage the output high-resolution Gaussians from our model and ray-trace the LiDAR rays to obtain the corresponding distances. Thanks to our clean voxel scaffold, the reconstructed scene is free of floaters and we set the opacity α to 1 for all the Gaussians to ensure a hard intersection that aligns better with the geometry.Application: Text-to-Scene Generation. Our method can be easily extended to generate 3D scenes from text prompts. Similar to MVDream [47] , we train a multi-view diffusion model with the architecture of VideoLDM [2] that generates images from text prompts. The original spatial self-attention layer is inflated along the view dimension to achieve content consistency [25, 65] . For training, we use CogVLM [58] to annotate the images automatically on a large scale. After the model is trained, we directly feed the output of the multi-view model to SCube to lift the observations into 3D space for novel view synthesis.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] Justification: NA Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experiments: 
 In this section, we validate the effectiveness of SCube. First, we present our new data curation pipeline that produces ground-truth voxel grids ( § 4.1). Next, we demonstrate SCube's capabilities in scene reconstruction ( § 4.2), and further highlight its usefulness in assisting the state-of-the-art Gaussian splatting pipeline ( § 4.3). Finally, we showcase other applications of our method ( § 4.4) and perform ablation studies to justify our design choices ( § 4.5).Accurate 3D data is essential for our method to learn useful geometry and appearance priors. Fortunately, many autonomous driving datasets [3, 53] are equipped with 3D LiDAR data, and one can simply accumulate the point clouds to obtain the 3D scene geometry [20, 40] . However, the LiDAR We hence build a data processing pipeline based on Waymo Open Dataset [53] as shown in Fig. 3 , consisting of three steps: Step 1, we accumulate the LiDAR points in the world space, removing the points within the bounding boxes of dynamic objects such as cars and pedestrians. We additionally obtain the semantics of each accumulated LiDAR point, where non-annotated points are assigned the semantics of their nearest annotated neighbors.Step 2, we use the multi-view stereo (MVS) algorithm available in COLMAP [44] to reconstruct the dense 3D point cloud from the images, and the semantic labels of the points are obtained by Segformer [64] .Step 3, we add point samples for the dynamic objects according to their bounding boxes at a given target frame. This gives us static and accumulated ground truths available for training. For each data sample, we crop the point cloud into a local chunk of 102.4m × 102.4m centered around a randomly sampled ego-vehicle pose. Since there are no rear-view cameras in the Waymo dataset, we allocate more space for the chunk in the forward direction, as shown in the inset figure. See Appendix A for additional details.Evaluation and Baselines. To assess our method's power for 3D scene reconstruction, we follow the common protocol to evaluate the task of novel view synthesis [4, 42, 68] . Given input multi-view images (details about choosing views are in Appendix A) at frame T , we render novel views at future timestamps T + 5 and T + 10, and compare them to the corresponding ground-truth frames by calculating Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS) [74] . We exclude the regions of moving objects for T + 5 and T + 10 evaluation, and only use three front views when computing the metrics.We use PixelNeRF [68] , PixelSplat [4] , DUSt3R [57] , MVSplat [6] , and MVSGaussian [29] as our baselines for comparison. [4, 6, 29, 68] take images and their corresponding camera parameters as input and reconstruct NeRFs or 3D Gaussian representations. DUSt3R [57] directly estimates per-pixel point clouds from the images. We append additional heads to the its decoder which predicts other 3D Gaussian attributes along with the mean positions and finetune it with a rendering loss. For all other baselines, we take the official code and re-train them on our dataset. We tried to add the We show the comparison between our method and Metric3Dv2 [18] . The semantics of Metric3Dv2 are obtained from Segformer [64] .state-of-the-art depth estimator Metric3Dv2 [18] for depth supervision but empirically found that the performance degraded.Results and Analysis. We show our results quantitatively in Tab. 1 and visually in Fig. 4 . Our method outperforms all the baselines for both the current frame (reconstruction) and future frames (prediction) by a considerable margin on all metrics. PixelNeRF is limited by the representation power of the network, and simply fails to capture high-frequency details in the scene. PixelSplat highly relies on overlapping regions in the input views and cannot adapt well to our sparse view setting. It fails to model the true 3D geometry as shown in the top-down view, and simply collapses the images into constant depths. The multi-view-stereo-based methods [6, 29] cannot enable extreme novel view synthesis such as the bird-eye view, and could not recover highly-occluded regions. Thanks to the effective pre-training of DUSt3R, it is able to learn plausible displacements in the image domain, but the method still suffers from missing regions, misalignments, or inaccurate depth boundaries. In contrast, our method can reconstruct complete scene geometry even for far-away regions. It is both accurate and consistent while producing high-quality novel view renderings.To better demonstrate the power of learning priors in 3D, we build another baseline using the state-ofthe-art metric depth estimator Metric3Dv2 [18] to unproject the images into point clouds using 2D learned priors. As shown in Fig. 5 , our method can reconstruct more complete, uniform, and accurate scenes, justifying the power of representing and learning geometry directly in the true 3D space.Our method creates scene-level 3D Gaussians with accurate geometry and appearance, which can be used to initialize large-scale 3D Gaussian splatting [23] training. This is particularly useful in outdoor driving scenarios where structure-from-motion (SfM) may fail due to the sparsity of viewpoints.To demonstrate this, we consider and compare three initialization methods: Random initialization is where points are uniformly sampled within the range of (-20m, 20m) 3 around each camera.Metric3Dv2 initialization is where we use the unprojected cloud from Metric3Dv2 [18] 's monocular R = 10 R = 20 R = 40We demonstrate the applications of our method as described in § 3.3. Fig. 6 shows the consistent LiDAR simulation results, where the simulated sequences could effectively cover a long range away from the input camera positions, while resolving intricate geometric structures such as buildings, trees, or poles. Fig. 7 exemplifies the text-to-scene generation capability enabled by our method. The 3D geometry and appearance respect the input text prompt and the corresponding images. Readers are referred to Appendix D.3 for more generative results.Image-Conditioning Strategy. We replace the image conditioning strategy described in Eq (1) in the voxel grid reconstruction stage with a vanilla scheme that broadcasts the same feature to all the voxels along the pixel's ray. The final IoU of the fine-level voxel grid drops from 34.31% to 30.33%, and the mIoU that considers the accuracy of the voxel's semantic prediction drops from 20.00% to 16.61%. This shows the effectiveness of our conditioning strategy being able to disambiguate voxels at different depths.Two-stage Reconstruction. We disentangle the voxel grid and appearance reconstruction stages to make the best use of different types of models. Using a single-stage model 2 that simultaneously … a palm tree, a road with traffic, lined with trees and buildings, under a blue sky with scattered white clouds, … Appearance Reconstruction. We validate the effect of voxel grid resolution and the number of Gaussians per voxel M in the appearance reconstruction stage. Results are shown in Tab. 3. We find that higher-resolution voxel grids are crucial for capturing detailed geometry, and using a larger number of Gaussians only slightly increases the performance. Thus, we set M = 4 as a moderate value for the final results. Compared in Fig. 8 , the GAN-based postprocessing, despite the time cost, is beneficial for producing high-quality images by sharpening the renderings. See Appendix D.2 for more visual comparisons.Reconstruct ReconstructQuestion: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We describe our method in great detail and provide enough information that a researcher familiar with diffusion models and neural rendering should be able to reproduce the pipeline. We describe our experiments in detail as well and run them on Waymo Open which is a public dataset accessible to all. The appendix includes detailed training and network architecture information. Together, these should make our results fully reproducible.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
discussion: 
 Conclusion. In this work, we have introduced SCube, a feed-forward method for large 3D scene reconstruction from images. Given sparse view non-overlapping images, our method is able to predict a high-resolution 3D scene representation consisting of voxel-supported Gaussian splats (VoxSplat) and a light-weight sky panorama in a single forward pass within tens of seconds. We have demonstrated the effectiveness of our method on the Waymo Open Dataset, and have shown that our method outperforms the state-of-the-art methods in terms of reconstruction quality.Limitations. Our method does suffer from some limitations. First, the current method is not able to handle complicated scenarios such as dynamic scenes under extreme lighting or weather conditions. Second, the quality of appearance in occluded regions still carries uncertainty. Third, the method itself still requires ground-truth 3D training data which is not always available for generic outdoor scenes. In future work, we plan to address these limitations by incorporating more advanced neural rendering techniques and by exploring more effective ways to generate training data.-Appendix - Input and Evaluation Details. Waymo dataset provides 5 views for each camera frame, namely front, front-left, front-right, side-left and side-right. However, not all of the baseline methods we compared with in § 4.2 can handle the unconventional camera intrinsic in the side-left and side-right views. We hence only use the first three views (with a resolution of 1920 × 1280) in § 4.2 for both the input and the evaluation metrics. However, in § 4.3 we opt to use all 5 views for the input to both our method and the baseline due to compatibility and maximized performance.For the baselines, the original PixelSplat [4] method does not have depth supervision. To make the comparison fair, we attempt to add a depth supervision loss to it. However, the experimental result shows that the additional loss hurts the performance as shown in Tab. 4. We thus report the results of vanilla PixelSplat in the main paper. Training Details. The diffusion loss in Eq (2) is defined similar to [16, 40] with a v-parametrization as:EQUATIONwhere v(•) is the diffusion network, t is the randomly sampled diffusion timestamp, and ᾱt is the scheduling factor for the diffusion process, whose details are referred to in [16] .We train all of our models using the Adam [24] optimizer with β 1 = 0.9 and β 1 = 0.999. We use PyTorch Lightning [10] for building our distributed training framework. For the voxel grid reconstruction stage, we train both coarse-level and fine-level voxel latent diffusion models with 64× NVIDIA Tesla A100s for 2 days. For the appearance reconstruction model, we train it using 8× NVIDIA Tesla A100s for 2 days. Empirically, we use λ = 1.0 for L Depth in Eq (2) . Additionally, we use λ 1 = 0.9, λ 2 = 1.0, λ SSIM = 0.1 and λ LPIPS = 0.6 in Eq (6) . For image condition, we set the feature channel C = 32, the number of depth bins D = 64, z near = 0.1 and z far = 90.0. We linearly increase the interval of depth bins.Voxel Grid Reconstruction. We follow [40] to implement the Sparse Structure VAE and the Diffusion UNet using the sparse 3D deep learning framework fVDB [62] . Hyperparameters for training them are listed in Tab. 5 and Tab. 6. We pass the images to distilled DINO-v2 [34] ViT-B/14. We use four 2D convolutional layers (channel dims: [768, 256, 256, 32, 32] , kernel size: 3, stride: 1) to further process the DINO-v2 output to predict the image feature and the depth distribution.Appearance Reconstruction. We process the original input images with three 2D convolutional layers (channel dims: [3, 16, 32, 32] , kernel size: In § 3.3 we introduce a GAN postprocessing module to refine the rendered images, which is finetuned on each scene. To further improve the efficiency of our method, we hereby present a postprocessing module that is jointly trained on the full dataset, without the need of per-scene finetuning. Specifically, we replace the original GAN with a pix2pix-turbo model [35] (which we denote as SCube+*) and train it with image pairs inferred from our model and the ground truths. The results are shown in Fig. 9 . This improved model not only reduces the voxel block artifacts, but also resolve the ISP inconsistencies within the image. After enabling this module, the FPS drops from 138 to 20 but can still be visualized interactively. In this section, we provide more qualitative results on all datasets. We additionally provide a supplementary video in the accompanying files to better illustrate our results.We note that the uncertainty of the scene geometry given our input images is large, and the problem that the model tackles is indeed non-trivial and sometimes even ill-posed. To demonstrate this, we compute the percentage of occluded voxels (that are invisible from the input images) w.r.t. all the ground-truth voxels, and the number is around 80%. To quantitatively evaluate the geometry quality, we compute an additional metric called 'voxel Chamfer distance' that measures the L2-Chamfer distance between the predicted voxels and ground-truth voxels (that are pixel-aligned), divided by the voxel size. This metric reflects the geometric accuracy of our prediction by measuring on average how many voxels is the prediction apart from the ground truth. The results on Waymo Open Dataset are shown in Tab. 9.Quantile 0.5 (median) 0.6 0.7 0.8 0.9 Ours 0.26 0.28 0.32 0.37 0.51 Table 9 : Geometry Quality Comparison. We show the voxel Chamfer distance comparison between our two-stage model and a single-stage non-diffusion model.Tab. 9 indicates that on 90% of the test samples, the predicted voxel grid is only half of a voxel off from the ground truth. We note that during our data curation process, there could be errors in the ground-truth voxels (e.g., due to COLMAP failures), accounting for the outliers in the above metric.In the meantime, we visualize the sample with the worst voxel Chamfer distance in Fig. 10 . The predicted results are decent even though the ground truth is corrupted due to the lack of motion in the ego car. This demonstrates the robustness of our method.In addition to the quantitative ablation study in Tab. 3, we present a qualitative demonstration in Fig. 11 . For the single-stage model, we test the upper bound of it by feeding the ground-truth 1024 3 voxel grids because otherwise the fully-dense high-resolution condition will lead to out-of-memory. The qualitative results match the numbers, showing the importance of using higher-resolution voxel grids and the two-stage model.Predicted Voxels Dataset Ground-truth Voxels Reconstructed 3D Gaussians Scene We provide additional text-2-scene generation results in Fig. 12 and Fig. 13 .A residential neighborhood features houses with well-maintained gardens, autumn-colored trees, lawns with scattered leaves, parked cars, driveways, and clear blue skies.A residential area features multiple houses, some with specific decorations and vehicles parked outside, including a white pickup truck and a gray car, along with various greenery and utility elements. A suburban neighborhood features two-story houses with reddish-brown roofs and beige walls, marked roads, various parked vehicles, stop signs, and a mixture of gravel, rocks, and trees providing shade on a sunny day.A suburban neighborhood features a park with green trees, residential houses with red-tiled roofs, streets with bike lane signs and white markings, well-maintained lawns, and sidewalks. We provide additional LiDAR Simulation results in Fig. 15 . We also show the result on a long sequence input in Fig. 16 .Input Front View Simulated LiDAR Sequence • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: Due to institutional constraints, we are not able to release the code until the paper is fully accepted. Upon acceptance, we will release all code and data required to reproduce this work.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] .Justification: See descrition in Section 4 and appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. Justification: We report these explicitly in the experiments section Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes]Justification: We have reviewed the code of ethics and believe our paper fully conforms to it.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [No] Justification: We feel there are no direct negative societal impacts of this work as it is a foundational method for reconstruction. While such a foundational method could eventually be used in pipelines with potential negative consequences (e.g. offensive applications in military settings), we do not believe our method directly enables this. Furthermore, by training on an open driving dataset, we focus simply on reconstructing the 3D world from car footage, which alone cannot be used to negatively impact society.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We don't believe our model poses an immediate risk of misuse. We believe it will help advance research in 3D reconstruction from images.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.In practice, we test the upper bound of the single-stage model by feeding in ground-truth 1024voxel grids, because otherwise the fully-dense high-resolution condition will lead to out-of-memory. 
 
 Appendix:  


Decision: This paper was reviewed by four experts in the field and received mixed reviews (1) Borderline Reject, (1) Weak Accept, (1) Accept, (1) Borderline Accept. Reviewers appreciate the idea of combining xcube diffusion model and gaussian splatting for sparse view-based large scale scene reconstruction. While one reviewer is concerned about the novelty, other reviewers find the idea is effective especially in reconstructing large scale scene from sparse views. The paper is well-written and justified with good results.  Reviewers have raised concerns regarding the details of  sky panorama model, evaluation of the results for both stages, and whether it is necessary to use diffusion for the task. Authors have provided responses to address reviewers’ concerns.  AC agrees with reviewers on the contribution of the idea which is effective and efficient in the reconstruction especially in handling the sparse view based reconstruction. AC thus recommends Accept.

Reviews: review: paper_summary: paper_summary: The paper proposes a new method to reconstruct 3D outdoor scenes from a sparse set of posed images. The key idea is to utilize a new hybrid 3D representation, which assigns 3D Gaussians to each sparse voxel. Given input images, the paper first adapts XCube to condition on sparse view images. Along with the generated fine-level voxels, the paper then designs a feed-forward model to reconstruct the appearance. This proposed pipeline can also be used in text-to-scene generation. Experiments show that the proposed method outperforms existing approaches.
strength_weakness: strength_weakness: - One concern is that the novelties of this paper are not very clear. As stated above (Originality), the paper seems to simply use two stages to combine the two representations in a simple way: one for XCube, and one for 3D Gaussians. It would be better if the author can clearly state the contributions.

- As a two-stage method, it would be better to discuss more about: (1) First stage: how to evaluate whether the reconstructed voxels align with the input images? (2) Second stage: how robust is the model when the input images and the voxels are not aligned?

- L187: Considering the temporal inputs, can the methods give consistent results temporally? Adding results might be better.

- L244-245: In Table 1, SCube seems to have very similar results to PixelSplat for future prediction. Why? An obviously better reconstruction result alone seems not able to show the effectiveness of the proposed method. The appearance reconstruction uses generated geometry (fine-level voxels) as input, and adds some tricks such as sky modeling. It is unknown how they affect the results.
questions: questions: See Weaknesses
limitations: limitations: N/A
paper_summary: paper_summary: The paper introduces a pipeline for street scene reconstruction given a sparse set of images as input. The reconstruction process follows a feed-forward manner. The method builds upon the existing XCube work. First, it generates sparse voxels of the scene to represent the geometries, then each voxel feature is decoded into Gaussian primitives for appearance rendering. Experimental results, both quantitative and qualitative, show that SCube can reconstruct the scene with high quality and provides benefits for downstream applications such as autonomous driving simulation.
strength_weakness: strength_weakness: (1)	I am curious if we really need a diffusion model for this task. Since there are several images as conditions, the uncertainty (or randomness) of the output should be very small. Why not just train a regressor for the sparse voxel reconstruction? Is there any specific motivation for using a diffusion model?
questions: questions: (1)	Why is the GAN loss used independently for each scene in the inference stage, instead of using it in the training stage?
limitations: limitations: Currently, the results are presented in the sparse view settings of a small region of the street scene. How can we reconstruct a very large scene using a long sequence of images in a feed-forward manner? This is an interesting topic for future work.
paper_summary: paper_summary: - The paper proposes a new method for sparse-view 3D reconstruction using 3DGS.
- The framework uses two stages:
    1) it learns a latent voxel grid (based on XCube) to represent the geometry.
    2) it trains an appearance model to decode the latent voxel grid into a set of Gaussians
- The method further uses a background model to handle the sky.
- The authors propose an (optional) GAN postprocessing which improves the visual quality at the cost of longer (20m) per-scene optimization.
- The method was evaluated on the Waymo Open Dataset, and it outperforms compared methods.
strength_weakness: strength_weakness: - Section Method "Training and Inference" is not very clear. It would be preferable to expand the diffusion loss from Appendix A, and clearly describe the input/targets.
- I would also like to have more details on the Sky Panorama model.
- The method was evaluated on a single dataset. It would be interesting to see how the method performs on other datasets.
questions: questions: - How does the method compare to other methods like MVSplatting, MVSGaussians? How does it differ? How do you think it would compare in terms of performance?
limitations: limitations: - Limitations are properly discussed in the paper.
paper_summary: paper_summary: This paper presents a method to reconstruct large-scale 3D scenes from a sparse set of posed images. The method has two stages: the first stage is for voxel grid reconstruction, which is based on the XCube [38], a 3D generative model, and the authors introduce the image condition to the model.  The second stage is for appearance reconstruction, which is a feedforward appearance prediction model to predict a certain number of Gaussians for each voxel. The authors also propose a GAN-based post-processing step to boost the appearance of the reconstructed scene. The proposed method is evaluated on the Waymo self-driving dataset, showing the superiority of the proposed method compared to prior art on 3D reconstruction tasks. Besides, the authors also demonstrate the applications of the proposed method, such as LiDAR simulation and text-to-scene generation.
strength_weakness: strength_weakness: - The proposed representation's abbreviation "VoxSplats" is somewhat confusing to me, since the term "splatting" in original 3DGS refers to the rendering technique that splats the 3D Gaussians onto the image plane, rather than the 3D representation. The proposed representation is more like a hybrid of voxel grids and Gaussian representation. The authors may consider using a more accurate term to describe the representation.
- It's unclear how the foreground and background are separated in the proposed method. Using a panorama to represent the sky makes sense as the sky is usually far away from the scene, but some buildings are treated as the background and seem to be represented by the panorama as well in the supplementary video. A more detailed explanation and discussion of the foreground-background separation is needed.
- There are some lighting/exposure/white-balance inconsistencies in the reconstructed scenes in the supplementary video. The authors should discuss the limitations of the proposed method in handling the lighting/exposure/white-balance variations in the input images.
- For SCube+, how to guarantee consistency among the rendered views? Also, rendering with the proposed GAN-based post-processing step is time-consuming, which would destroy the valuable real-time rendering performance of the Gaussian Splatting. The authors should discuss these issues in the paper.
- Missing the evaluation of the geometric accuracy of the reconstructed scenes. As the rendering quality of SCube is not so good, geometric accuracy is crucial to the applications of the reconstructed scenes. The authors should provide a detailed evaluation of the geometric accuracy of the reconstructed scenes.
questions: questions: The most important questions that I would like the authors to address are:
- Geometric accuracy evaluation of the reconstructed scenes.
limitations: limitations: Yes, the authors have discussed the limitations of the proposed method.

Comments: title: Comment
comment: Thank you very much for the experiment! I really appreciate it! I've raised my score back.
Dear reviewer, 

we sincerely hope the supplementary experiments comparing to MVSGaussian and MVSplat have addressed your concerns; and we really appreciate it if you could reconsider our method and raise the score back.
Thank you for taking the time to review our rebuttal!
Thank you for taking the time to review our rebuttal. We are pleased that most of your concerns have been addressed. 

While our approach incorporates various state-of-the-art techniques such as Gaussian splatting and diffusion models, we recognize that many well-justified and published works also leverage one or more of these building stones to formulate their reconstruction models. Our primary contribution lies in the novel way of synergizing these foundational techniques to create a fully integrated pipeline that achieves state-of-the-art results for large-scale outdoor scene reconstruction encompassing true 3D geometric priors, while also introducing significant technical contributions necessary to make the method effective (as noted in the rebuttal above). 

Importantly, our method introduces the first possible way to tackle the challenging task of reconstructing large-scale 3D scenes from low-overlapping outward-facing input images. We are able to reach a much better and more robust performance on this task than the baselines (e.g. PixelSplat, DUSt3R, MVSGaussian, MVSplat, etc.) where a faithful reconstruction of the underlying 3D geometry is missing.

We would greatly appreciate it if SCube's contribution could be reconsidered during the discussion phase.
Dear reviewer PK9U, thank you for your comment. To further resolve your raised question, we provide the quantitative comparison with MVSplat [a] and MVSGaussians [b] as below:

- PSNR (↑ The higher the better)

|                 | Reconstruction (T) | Prediction (T + 5) | Prediction (T + 10) |
| --------------- | ------------------ | -------------------| ------------------- | 
| **MVSplat**     | 21.84              | 20.14              | 18.78               | 
| **MVSGaussian** | 21.25              | 16.49              | 16.42               | 
| **SCube (Ours)**| 25.90              | 19.90              | 18.78               | 
| **SCube+ (Ours)**| **28.01**         | **22.32**          | **21.09**           |

- SSIM  (↑ The higher the better)

|                 | Reconstruction (T) | Prediction (T + 5) | Prediction (T + 10) |
| --------------- | ------------------ | -------------------| ------------------- | 
| **MVSplat**     | 0.71               | 0.71               | 0.69                |
| **MVSGaussian** | 0.80               | 0.70               | 0.60                |
| **SCube (Ours)**| 0.77               | 0.72               | 0.70                | 
| **SCube+ (Ours)**| **0.81**          | **0.74**           | **0.72**            |

- LPIPS (↓ The lower the better)

|                 | Reconstruction (T) | Prediction (T + 5) | Prediction (T + 10) |
| --------------- | ------------------ | -------------------| ------------------- | 
| **MVSplat**     | 0.46               | 0.48               | 0.52                | 
| **MVSGaussian**   | 0.51               | 0.60               | 0.59                | 
| **SCube (Ours)**| 0.45               | 0.47               | 0.49                | 
| **SCube+ (Ours)**| **0.25**          | **0.34**           | **0.38**            |

For both methods, we use their respective official codebases and retrain the model using the same dataset split as our method with two 8-GPU nodes. We follow the recommended training hyperparameter and strategies to train the models. Please note that these methods are originally demonstrated only on indoor datasets with a smaller scale and without sky modeling.

The above quantitative results echo our reasoning as posted before: While both methods show good reconstruction quality, they fail to model the true 3D geometry and the occluded parts of the scene and hence have unsatisfactory generalization capability to large-scale outdoor scenes. 

Geometry-wise, the geometry of MVSplat degenerates to multiple planes, and the geometry of MVSGaussian contains too many outliers on the rays of the corresponding pixels. To measure the geometric accuracy we compute the L2 Chamfer distance between predicted voxels and ground-truth voxel centers (note that the metric is in `meter`). The results are as follows:

| Quantile               | 0.5 (median) | 0.6      | 0.7      | 0.8      | 0.9      |
| ---------------------- | ------------ | -------- | -------- | -------- | -------- |
| **MVSplat** |   43.61     |   45.54  |  47.47   |   49.67  |  53.77   |
| **Ours**    | **0.10**     | **0.11** | **0.13** | **0.15** | **0.20** |

*(Note that we do not include the results of MVSGaussian due to the amount of outliers reconstructed.)*

Due to the restriction of NeurIPS, we have sent an anonymous link to the AC containing **qualitative** results of these baselines. Please ask the AC for the link if necessary. We will include the new comparisons to the final version of our paper and add corresponding citations.

**We sincerely hope the supplementary experiments comparing to MVSGaussian and MVSplat have addressed your concerns, and we really appreciate it if you could reconsider our method and raise the score back.**

***References***:

[a] Chen, Yuedong, et al. "Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images." ECCV 2024.

[b] Liu, Tianqi, et al. "Fast Generalizable Gaussian Splatting Reconstruction from Multi-View Stereo." arXiv preprint arXiv:2405.12218 (2024).
Thanks for the rebuttal. Most of my concerns are addressed but I am still concerned about the novelty.  I don't have follow-up questions within this rebuttal, and will have more discussion with other reviewers for the final decision.
Thank you for the rebuttal and the nice explanation of the diffusion loss and the panorama model. I feel like including them in the paper will improve the quality of the method section. As for my main question, I don't feel like it has been properly addressed. I would have liked a quantitative comparison (even limited) with the suggested baselines. While I still like the paper, I've decided to drop my rating weak accept.
Thank you for the answers! They addressed my questions.