SCube: Instant Large-Scale Scene Reconstruction using VoxSplats
Authors: Xuanchi Ren
Yifan Lu
hanxue liang
Jay Zhangjie Wu
Huan Ling
Mike Chen
Sanja Fidler
Francis Williams
Jiahui Huang
Venue: NeurIPS 2024 poster (2024)
Keywords: Large-scale Scene Reconstruction
Sparse-view Reconstruction
Sparse Voxels
Gaussian Splatting
Diffusion Models
Primary area: generative_models
TL;DR: SCube reconstructs large-scale 3D scenes represented with voxel-supported Gasussian splats from non-overlapping sparse views in 20 seconds.

Abstract: We present SCube, a novel method for reconstructing large-scale 3D scenes (geometry, appearance, and semantics) from a sparse set of posed images. Our method encodes reconstructed scenes using a novel representation VoxSplat, which is a set of 3D Gaussians supported on a high-resolution sparse-voxel scaffold. To reconstruct a VoxSplat from images, we employ a hierarchical voxel latent diffusion model conditioned on the input images followed by a feedforward appearance prediction model. The diffusion model generates high-resolution grids progressively in a coarse-to-fine manner, and the appearance network predicts a set of Gaussians within each voxel. From as few as 3 non-overlapping input images, SCube can generate millions of Gaussians with a 10243 voxel grid spanning hundreds of meters in 20 seconds. Past works tackling scene reconstruction from images either rely on per-scene optimization and fail to reconstruct the scene away from input views (thus requiring dense view coverage as input) or leverage geometric priors based on low-resolution models, which produce blurry results. In contrast, SCube leverages high-resolution sparse networks and produces sharp outputs from few views. We show the superiority of SCube compared to prior art using the Waymo self-driving dataset on 3D reconstruction and demonstrate its applications, such as LiDAR simulation and text-to-scene generation.

SCube: Instant Large-Scale Scene Reconstruction using VoxSplats
Xuanchi Ren 1,,,+ Yifan Lu 1: Hanxue Liang 4.5, Zhangjie Wul.0, Huan Ling! 123,3 Mike Chen', Sanja F2d Francis Williams', Jiahui Huang*
IVIIIA, University of Toronto, Vector Institute, 'Shanghai Jiao Tong University University of Cambridge, National University of Singapore https://research: nvidia. http//////tooottaaassssss
Abstract
We present SCube, novel method for reconstructing large-scale 3D scenes (geom- etry, appearance, and semantics) from a sparse posed images. Our method encodes reconstructed scenes using novel representation VoxSplat, which is a set of 3D Gaussians supported on high-resolution sparse-voxed scaffold. To re- construct VoxSplat from images, we employ hierarchical voxel latent diffusion model conditioned on the input images followed by feedforward appearance pre- diction model. The diffusion model generates high-resolution grids progressively in a coarse-to-fine manner, and the appearance network predicts set of Gaussians within each voxel. From as few as 3 non-overlapping inpul images, SCube can generate millions of Gaussians with a 10243 voxel grid spanning hundreds of meters in 20 seconds. Past works tackling scene reconstruction from images either rely on pep-peeseeeee ver-scene optimization and fail to reconstruct the scene away from input views (thus requiring dense view coverage input) leverage geometric priors based on low-resolution models, which produce blurry results. In contrast, SCube leverages high-resolution sparse networks and produces sharp outputs from few views. We show the superiority of SCube compared to prior art using the Waymo self-driving dataset 3D reconstruction and demonstrate its applications, such LiDAR simulation and text-to-scene generation.
Introduction
Recovering 3D geometry and appearance from images is fundamental problem in computer vision and graphics which has been studied for decades. This task lies at the core of many practical applications spanning roboties, autonomous driving, and augmented reality; just to name a few. Early algorithms tackling this problem use stereo matching and structure from motion (SfM) to recover 3D signals from image data (e.g.[44]). More recently, a line of work starting from Neural Radiance Fields [32] (NeRFs) has augmented traditional SfM pipelines by fitting volumetric field to set of images, which can be rendered from novel views. NeRFs augment traditional reconstruction pipelines by encoding dense geometry, and view-dependent lighting effects. While radiance-field methods present drastic step forward in our ability to recover 3D information from images, they require time-consuming per-scene optimization scheme. Furthermore, since each scene is recovered in isolation, radiance fields do not make use of data priors, and cannot extrapolate reconstructions away from the input views. Thus, radiance-fiel methods require dense view coverage in order to produce high-quality 3D reconstructions.
* Equal contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Real-world Capture
Text2Image
"I mad the fongrouud, sevend vetacier motion, lamppoos sidk.
Render
Render
Reconstruct
Reconstruct
Input Images
Novel Views
Input Images
Novel Views
Figure 1: SCube. Given sparse input images with little overlap, model reconstructs high-resolution and large-scale scene in 3D represented with VoxSplats, ready to be used for novel view synthesis LiDAR simulation.
Another recent line of work applies deep learning to predict 3D from images. These methods either meta-learn an initialization to the radiance- field optimization problem [7, 30, 49), or directly predict 3D from images using feed-forward network [17, 57, 73J. While ceariingbaseed approaches can produce reconstructions from sparse views, they have only been used successfully for the case of single objects at low resolutions. Furthermore, these methods often suffer from 3D inconsistencies (e.g. the multi-layer surface or the Janus problem). In order to solve the general 3D reconstruction from images problem, need methods that can (I) generalize reconstruction general scenes over the pure object case, (2) produce accurate and high-quality reconstructions in the presence of dense views, leveraging data priors produce plausible reconstructions in the sparse- view regime, and (3) run quickly and efficiently (in terms of runtime and memory) on large-scale and inputs. These demands are difficult satisfy in practice since high-quality ground-truth 3D data is not widely available for scenes, 3D representations for deep learning that scale to large and diverse inputs are under-explored in the literature, and corresponding scalable and easy-to-train model designs need to be developed alongside any new 3D representation.
Nevertheless, we remark that some of these issues have been resolved in isolation: Gaussian Splatting [23] enables fast, differentiable rendering and high reconstruction quality (but is not being used with data priors), and sparse voxel hierarchies [40] have been successfully used to build generative models of large-scale 3D scenes with attributes such as semantics and colors, and have been trained on partial data such LiDAR scans from autonomous vehicle captures.
In light of the above observations, we introduce SCube, ferd-forward method for large 3D scene reconstruction from images. Our method encodes 3D scenes as hybrid of Gaussian splats (which enable fast rendering), supported on a sparse-voxel-hierarde modeling of large 3D scenes with semantics). We call this hybrid representation VoxSplats and predict a VoxSplat from images using a feed-forward process consisting two steps: (1) generative geometry network that predicts sparse voxel hierarchy conditioned input images, and (2) an appearance network that predicts the Gaussian attributes within the voxels as well as a skybox texture to represent the background. The networks implemented using highly efficient sparse convolution [14, 40] designed for 3D data which enables us to reconstruct a full scene from images in under 20 seconds. We evaluate performance
(which enables efficient generative
the Waymo Open Dataset [53] the challenging task of reconstructing a scene from sparse images with low overlap. We show that SCube significantly outperforms existing methods on this task. Furthermore, demonstrate that SCube enables downstream applications such as LiDAR simulation and text-to-scene generation.
2 Related Work
3D Scene Representation. Scenes the wild are often large in scale and contain complicated internal structures which cause representations such tri-planes [12], dense voxel grids [36), or meshes [19, 46] to fail due to capacity or memory limitations. Optimization-ban
reconstruction methods [15, 32] use high-resolution hash grids [1, 33], but these non-tri trivial to infer using neural

---

Voxel Grid Reconstruction
Condition
Condi son
Input Feature Cube
1)
Voxel Latent, Diftusion
Coarse level Voxels (256")
Voxe Latent
Diffusion
Fine-level Voxels (1024")
Appearance Reconstruction
Pano. Encoding
Sangle
Sky Feat. Panorama
20 Feature Map
2D ecoder
Background Img
Prevuuse
Render
Ray Casting
Ganssars
Voxel + Image Feature Sparse 3D Lieet
3D Gausslans Scene
Foreground Img
Figure 2: Framework. SCube consists of two stages: (1) We reconstruct a sparse voxel grid with
semantic logit conditioned on the input images using conditional latent diffusion model based on XCube [40]. (2) We predict the appearance of the scene represented as VoxSplats and sky panorama using a feedforward network. Our method allows us to synthesize novel views in fast and accurate manner, along with many other applications.
network [30]. In contrast, sparse voxel grids are effective for learning scene-reconstruction [40, 75] thanks to efficient sparse neural operators [8. 54]. Recently, Gaussian splatting [23] has enabled real-time neural rendering and has been applied to overfitting large scenes [66, 76]. [31, 39] use hybrid of the above two representations, but the voxel grid octree is only used to regularize the Gaussian positions without any data priors learned. This is in contrast to our VoxSplat that allows reconstruction in direct inference pass thanks to the efficiency of sparse grids and the high representation power of Gaussian splats. We support operating only on sparse-view images, significantly lifting the input requirements by learning from large datasets.
Siar 3D Reconstruction. Sparse-view images often contain insufficient correspondences required by traditional reconstruction methods [44]. One line of work uses learned image-space priors such depth [9], normal maps [70], and appearance from GANs [43] or diffusion models [63] to augment an optimization process such as NeRF. To speed up inference, another line work uses feed-forward model to predict renderable features [4, 6, 22, 29, 57, 73]. Alternatively, some papers perform learning directly in 3D space, which yields better consistency and less distortion [5, 13, 17, 68]. Our setting is similar [13] where input images come from the same rig, but is more challenging since we do not use that semantic scene completion works [21, 26, 51, 61] also reconstruct voxels but at much lower resolutions and without appearance.
inputs with high overlap. We remark
Generative Models for 3D. 3D reconstruction can also be formulated as a conditional generative problem (i.c.modeling the distribution of scenes given partial observations). Text and single-image to-3D generation has been explored for objects [17, 28, 38, 47, 55, 56, 60, 69]. Extending this task to large-scenes is comparatively unexplored, and object- based methods often fail due to scaling limitations or assumptions on the data. [48, 72] recursively apply an image generative model to inpaint missing regions in 3D, but produces blurry reconstructions at limited scale. XCube [40] is among the first to directly learn high-resolution 3D scene priors. Here, we extend this model with multiview image conditioning and enable to predict appearance on top of geometry.
3 Method
Our method reconstructs a high-resolution 3D scene from N sparse images I N in two stages: (1) We reconstruct the scene geometry represented sparse voxel grid G with semantic

---

features ($ 3.1). (2) We predict the appearance A of the scene that allows for high-quality novel view synthesis ($ 3.2) using VoxSplats and a sky panorama. We can express our pipeline as taking samples from the distribution p(G, A(z) = p(Alg ())( In order to improve the final view quality of the output, we apply an optional post- prst-proesssing step discussed in $ 3.3.
3.1 Voxel Grid Reconstruction
Background: 3D Generation with XCube. XCube [40] is a 3D generative model that produces high-quality samples for both objects and large outdoor scenes. XCube uses a hierarchical latent diffusion model generate sparse voxel hierarchies, i.e., hierarchy of sparse voxel grids where each fine voxel is contained within a coarse voxel. XCube learns distribution over latent X encoded by sparse structure Variational Autoencoder (VAE). Both the VAE and the diffusion model instantiated with sparse convolutional neural networks [14], and can generate geometry at up to 10243 resolution. We use XCube as the backbone for our geometry reconstruction module. We remark that while the original paper only focused on unconditional or text-conditioned generation, we introduce a novel image-based conditioning C.
Image Conditioned Geometry Generation. To condition XCube on posed input images, we lift DINO-v2 [34] features computed on the input images to 3D space as follows. First, we use the pre-trained DINOv2 model to extract robust visual features for input images, and process the DINO feature using several trainable 2D conv layers to reduce the feature channel to C'+ ++1 D. We then split the channel C + D into two parts for each pixel and input image index é: one part is regular C-dimensional feature F'J and the other will be a D-dimensional Softmax-normalize normalized vector 0; E IR.. Here f can be viewed as distribution over the depth of the corresponding pixel, and we follow a strategy similar to LSS [37] to unproject the images into dense 3D voxel grid f where v denotes the index of a voxel and d E [1, DJ indexes the depth buckets:
F = 0ja- Ej, C = Σ F ERC.
(1.j.d)
Note that we quantize the depth into D bins dividing the range from predefined Anear to Zfar- Unlike image-condiiionin techniques used in object-level or indoor-level datasets where the camera frusta have significant overlap. our large-scale outdoor setting only takes sparse low-overlapping views captured from an ego-centric camera. Hence previous methods [28, 50, 52] that broadcast the same features to all the voxels along the rays corresponding to the pixel are not suitable here to precisely locate the geometries such as vehicles. The use of the weight 0 allows us to handle occlusions effectively and produce more accurate conditioning signal. After building C, we directly concatenate with the latent X and feed into the XCube diffusion network as conditioning.
Training and Inference. Our training pipeline is similar [40], where we first train VAE to learn a latent space over sparse voxel hierarchies. We add semantic logit prediction as in [40] to the grid and empirically find that it helps the model to learn better geometry. Then we train the diffusion model conditioned on C using the following loss:
C LDifiusion + DCDDpph
DDppth Ex,i, 10)ler),
where LpiItssion is the loss for diffusion model training (see Appendix A for details). Focal(.) is the multi-class focal loss [27]. This additional depth loss is an explicit supervision to properly weigh the image features and encourage correct placement into the corresponding voxels. Due the generative nature of XCube, we could learn the data prior to generate complete geometry even if some of the ground-truth 3D data incomplete.
3.2 Appearance Reconstruction
The VoxSplat Representation. In the second stage, we fix the voxel grid G generated from the geometry stage and predict a set of Gaussian splats in each voxel to model the scene appearance. Gaussian splatting [23] is powerful 3D representation technique that models scene's appearance volumetrically as sum of Gaussians:
G(æ) RGB a-e - (a (a- 11)
(3)

---

where a E [0, 1] is the opacity, P E R is the center of each Gaussian, and Σ = RSS E R its covariance. The covariance matrix is factorized into rotation matrix R parameterized by a quaternion q and scale diagonal matrix S = diag(s) s). Each Gaussian additionally stores a color value RGB. Note that the original paper uses set of SH coefficients for view-dependent colors, but we only use the -rder SH in our model (i.e., without view-dependency) which we found to be sufficient for sparse-view reconstruction.
While the original Gaussian Splatting paper and its follow-ups [23, 67, 71J propose many heuristics to optimize the positions of Gaussians for given scene, we instead choose to predict M Gaussians per- voxel using feed-forwe herd-fo model. We limit the positions of the Gaussians within neighborhood of their supporting voxels, thus preserving the geometric structure of the supporting grid. By grounding the splats on a voxel scaffold, our reconstructions achieve better geometric quality without resorting to heuristics. Fittingly, we dub our voxel-supported Gaussian splats VoxSplats.
The output of our network is { [i,, u. 8,; qu, RGB,] E RR4) as for each voxel v. To compute the per-Gaussian parameters used for rendering we apply the following activations:
ny r- tanh f. Center,, a2 sigmoid(ā.). 8, exp S., R qua(2ro(qq)),
(4)
where Center, is the centroid of the voxel v, and is hyperparameter that controls the range of Gaussian within its supporting voxel. Here, we set to three times the voxel size. We can efficiently render the Gaussians predicted by our model using rasterization [23] or raytracing [11].
Sky Panorama for Background. To capture appearance away from the predicted geometry. model builds sky feature panorama L E R g2,,W,XC; from all input images, which can be considered as an expanded unit sphere with an inverse equirectangular projection. For each pixel in the panorama L, get its cartesian coordinate P (z, y, the unit sphere and project P to the image plane to retrieve the image feature; since only the view direction decides the sky color, zero the translation part of the camera pose in the projection step. We also apply sky mask to ensure the panorama only focuses on the sky region.
To render a novel viewpoint with its extrinsics and intrinsics, we recover the background appearance by sampling the sky panorama and decoding it into RGB values. For each camera ray from the novel view, we calculate its pixel coordinate on the 2D sky panorama L with equirectangular projection and get the sky features via trilinear interpolation, resulting in a 2D feature map for the novel view. We finally decode the 2D feature map with a CNN network to get the background image Ibg. which will be composied with the foreground image from Gaussian rasterization:
Ireed (u, v) (u, v) + (1 - T(u, u)) Ing (u, v)
(5)
where Ias (u, v) is the rendered image of Gaussians, (u, v) indicates the pixel coordinate, and T(u, v) is the accumulated transmittance map of the Gaussians (see [23] for details).
Architecture Details. We predict the (M x 14) dimeensiial vector {( c. Su, qu; RGB..] ar for each voxel via a 3D sparse convolutional U-Net which takes as input the sparse voxel grid Ω outputted by the geometry stage, where each voxel contains a feature sampled from the input images as follows: We process each input image I' using a CNN to get the image feature, and then cast ray from each image pixel into Pt accumulating the feature in the first voxel intersected by that ray. Voxels that are not intersected by any rays receive a zero feature vector.
For the sky panorama model, we use the same image feature as above. In the training stage, we set smaller Hy and Wp for faster training and lower memory usage; in the inference stage, we increase Hp and W,, to get a sharper and more detailed sky appearance.
Given set of training images {Tg J. and sky masks {M'), J. distinct from the inputs, we supervise the appearance model using the loss:
L C In)) A2C1(T', M') + ASSIM (Fpe Ig.) + ALPIPS (T pg))
(6)
where the training views I are sampled from nearby 10 views of the input images; the predicted views Fpeed and transmittance masks T' are rendered using Eq (5); and EspIesLSsmM are perceptual and structural metrics defined in [74] and [59].
5

---

COLMAP
Object
Points
BBox
Curb Lanc
Pedestrian
Vchicles
Road
Sidewalk
Vegetation
Building
Sign Pole
Figure 3: Data Processing Pipeline. We add COLMAP [44] dense reconstruction points to the accumulated LiDAR points and compensate for dynamic objects using their bounding boxes. This provides with more complete geometry for training.
3.3 Postprocessing and Applications
Optional GAN Postprocessing. The novel views directly rendered from our appearance model sometimes suffer from voxelization artifacts or noise. We resolve this with an optional lightweight conditional Generative Adversarial Network (GAN) that takes the rendered images input and outputs a refined version. The discriminator this GAN takes 256 256 image patches sampled from the input sparse view images, as well as the generated images conditioned on the rendered images. Drawing inspiration from [41, 43, 45], we fit the GAN independently for each scene at inference time, which takes ~0min to train. Due to the excessive time cost, apply this step optionally only when qughet-quality images are needed (which we call SCube+). Fig. shows the results with and without this step, and further present general postprocessing without per-scene optimization in Appendix C.
Application: Consistent LiDAR Simulation. LiDAR simulation [77] aims at reproducing the point cloud ouiput given novel locations of the sensor and is an important application for training and verification autonomous driving systems. The generated LiDAR point clouds should accurately reflect the underlying 3D geometry and a sequence of LiDAR scans should be temporally consistent. Our method enables converting sparse-view images directly into LiDAR point clouds, i.e., a sensor- to-sensor conversion scheme. To achieve this, we leverage the output high-resolution Gaussians from our model and ray-trace the LiDAR rays obtain the corresponding distances. Thanks our clean voxel scaffold, the reconstructed scene is free of floaters and we set the opacity to for all the Gaussians to ensure hard intersection that aligns better with the geometry.
Application: Text-to-Scene Generation. Our method can be easily extended to generate 3D scenes from text prompts. Similar to MVDream [47], we train a multi-view diffusion model with the architecture of VideoLDM [2] that generates images from text prompts. The original spatial self-attention layer is inflated along the view dimension to achieve content consistency [25, 65]. For training,
CogVLM [58] annotate the images automatically on large scale. After the model is trained, we directly feed the output of the multi-view model to SCube to lift the observations into 3D space for novel view synthesis.
4 Experiments
In this section, we validate the effectiveness of SCube. First, we present our new data curation pipeline that produces ground-truth voxel grids ($ 4.1). Next, we demonstrate SCube's capabilities in scene reconstruction ($ 4.2), and further highlight its usefulness in assisting the state-of state-of-the-ar Gaussian splatting pipeline ($ 4.3). Finally, we showcase other applications of our method ($ 4.4) and perform ablation studies to justify our design choices ($ 4.5).
4.1 Dataset Processing
Accurate 3D data is essential for our method to learn useful geometry and appearance priors. Fortu- nately, many autonomous driving datasets [3, 53] are equipped with 3D LiDAR data, and one can simply accumulate the point clouds to obtain the 3D scene geometry [20, 40]. However, the LiDAR

---

Input Views
Input Views
Timestamp T'
Novel View Rendering
Novel View Rendering
PixelSppat
Timestapp T+10 DUSER
Soiber (Ours)
Front-l Left
Front
Front-Right
Front-Lefft
Front
Front-R Front-Right
Figure 4: Novel View Synthesis. We show the synthesized novel views of SCube+ compared to baselines approaches. The inset of each subfigure shows a top-down visualization (an extreme novel view) of the reconstructed scene geometry.
points usually do not cover high-up regions such as tall buildings and contain dynamic (non-rigid) objects that are non-trivial to accumulate.
We hence build a data processing pipeline based on Waymo Open Dataset [53] as shown in Fig. 3, consisting of three steps: Step 1, we accumulate the LiDAR points in the world space, removing the points within the bounding boxes of dynamic objects such as cars and pedestri- ans. We additionally obtain the semanties of each accumulated LiDAR point, where con-annotated points are assigned the semantics of their nearest annotated neighbors. Step 2, we use the multi-view stereo (MVS) algorithm available in COLMAP [44] to reconstruct the dense 3D point cloud from the images, and the semantic labels of the points are obtained by Segformer [64]. Step
102.4m
102.4m
90%
50%
50%
10%
add point samples for the dynamic objects according to their bounding boxes at a given target frame. This gives us static and accumulated ground truths available for training. For each data sample, we crop the point cloud into a local chunk of 102.4m x 102.4m centered around a randomly sampled ego-vehicle pose. rear-view cameras in the Waymo dataset, allocate more space for the chunk in
Since there the forward direction, as shown in the inset figure. See Appendix A for additional details.
4.2 Large-scale Scene Reconstruction
Evaluation and Baselines. To assess our method's power for 3D scene reconstruction, we follow common protocol to evaluate the task of novel view synthesis [4, 42, 68). Given input multi-view images (details about choosing views are in Appendix A) at frame T, we render novel views at future timestamps T + and + 10, and compare them to the corresponding ground-truth frames by calculating Peak Signal-to-Noise Ratio (PSNR). Structural Similarity Index Measure (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS) [74]. We exclude the regions of moving objects for T' + 5 and T + 10 evaluation, and only use three front views when computing the metrics.
We use PixelNeRF [68], PixelSplat [4], DUSt3R [57], MVSplat [6], and MVSGaussian [29] as our baselines for comparison. [4. 6, 29, 68] take images and their corresponding camera parameters as input and reconstruct NeRFs or 3D Gaussian representations. DUSt3R [57] directly estimates per-pixel point clouds from the images. We append additional heads to the its decoder which predicts other 3D Gaussian attributes along with the mean positions and finetune it with a rendering loss. For all other baselines, we take the official code and re-train them our dataset. We tried to add the
7

---

Reconstruction
Prediction (T'
Prediction (T'
PSNRT
SSIMT
LPIPS
PSNR1
SSIMT
LPIPS
PSNR
SSIM
LPIPS
PixeINeRF [68]
15.26
0.51
0.66
15.21
0.52
0.64
14.61
0.49
0.66
PixelSplat [4]
22.15
0.71
0.61
20.11
0.70
0.60
18.77
0.66
0.62
DUS:3 [57]
17.17
0.60
0.58
17.08
0.62
0.56
16.08
0.58
0.60
MVSplat [6]
21.84
0.71
0.46
20.14
0.71
0.48
18.78
0.69
0.52
MVSGaussian [29]
21.25
0.80
0.51
16.49
0.70
0.60
16.42
0.60
0.59
SCube (Ours)
25.90
0.77
0.45
19.90
0.72
0.47
18.78
0.70
0.49
SCubc+ (Ours)
28.01
0.81
0.25
22.32
0.74
0.34
21.09
0.72
0.38
Table 1:
Quantitative Comparisons on 3D Reconstruction. The metrics are computed both at the input frame T and future frames. t: higher better, 4: lower is better.
Metric3Dv2
Ours
Metric3Dv2
Ours
Figure 5: Geometry Reconstruction from Sparse Views. We show the comparison between our
method and Metric3Dv2 [18]. The semantics of Metric3Dv2 are obtained from Segformer [64].
state-of-the-are depth estimator Metric3Dv2 [18] for depth supervision but empirically found that the performance degraded.
Results and Analysis. We show our results quantitatively in Tab. and visually in Fig. 4. Our method outperforms all the baselines for both the current frame (reconstruction) and future frames (prediction) by considerable margin on all metrics. PixeINeRF is limited by the representation power of the network, and simply fails to capture high-frequency details in the scene. PixelSplat highly relies on overlapping regions in the input views and cannot adapt well to our sparse view setting. fails to model the true 3D geometry shown in the top-down view, and simply collapses the images into constant depths. The multi wwww-iiteew-wbbweeeeeee methods [6, 29] cannot enable extreme novel view synthesis such the bird-eye view, and could recover highly-oceluded regions. Thanks the effective pre-training of DUSt3R, it is able to learn plausible displacements in the image domain, but the method still suffers from missing regions, misalignments, or inaccurate depth boundaries. In contrast, our method can reconstruct complete scene geometry even for far-away regions. It is both accurate and consistent while producing high-quality novel view renderings.
To better demonstrate the power of learning priors 3D, we build another baseline using the state-of- the-art metric depth estimator Metric3lvv2 [18] to unproject the images into point clouds using 2D learned priors. As shown in Fig. 5, our method can reconstruct more complete, uniform, and accurate scenes, justifying the power representing and learning geometry directly in the true 3D space.
4.3 Assisting Gaussian Splatting Initialization
Our method creates scene-level 3D Gaussians with accurate geometry and appearance, which can be used to initialize large-scale 3D Gaussian splatting [23] training. This is particularly useful in outdoor driving scenarios where structure- ffrore-ffoo---- motion (SfM) may fail due to the sparsity of viewpoints.
froom-mtiion
To demonstrate this, consider and compare three initialization methods: Random initialization where points are uniformly sampled within the range of (-20m, 20m)² around each camera. Metric3Dv2 initialization is where the unprojected cloud from Metric3Dv2 [18]'s monocular

---

R 20
R 40
PSNRT
SSIM
LPIPS]
PSNR
SSIM
LPIPS.
PSNR
SSIM
LPIPS,
Random
21.66
0.72
0.38
24.27
0.78
0.34
24.93
0.80
0.35
Metric3Dv2 [18]
23.30
0.75
0.33
25.21
0.80
0.32
25.58
0.80
0.34
SCube (Ours)
24.10
0.77
0.32
25.94
0.81
0.30
26.07
0.82
0.32
Table 2: Initializations for Gaussian Splatting training. We train 3D Gaussians with different initialization for frames. We report the test-set metrics. f: higher is better, 4: lower is better.
Input Front View
Simulated LiDAR Soquenee
Figure 6: LiDAR Simulation. We demonstrate qualitative results of image-to-oonsisee--LLDAA transfer. The LiDAR sequences are simulated by moving the camera forward by 60 meters.
depth and align its scale to metric-scale LiDAR. SCube (ours) initialization directly adopts the positions and colors of the Gaussians from our pipeline. For input to these methods, we choose the views from the first frame T and control the number of initial points to 200k. then incorporate subsequent frames into the full training, with every 3 frames skipped to be used in the test set. The number of training iterations is fixed to 30k and the initial positional learning rate is set to 1.6e-5. 5. We select 15 static scenes for experiments and report their average metrics, which shown in Tab. 2. The results consistently demonstrate SCube's effectiveness as an initialization strategy that provides accurate 3D grounding and alleviates overfitting the training views.
4.4 Other Applications
We demonstrate the applications of our method as described in $ 3.3. Fig. 6 shows the consistent LiDAR simulation results, where the simulated sequences could effectively cover a long range away from the input camera positions, while resolving intricate geometric structures such buildings, trees, or poles. Fig. exemplifies the text-to-scene generation capability enabled by our method. The 3D geometry and appearance respect the input text prompt and the corresponding images. Readers are referred to Appendix D.3 for more generative results.
4.5 Ablation Study
Strategy. We replace the image conditioning strategy described in Eq (1) in the voxel grid reconstruction stage with a vanilla scheme that broadcasts the same feature to all the voxels along the pixel's ray. The final loU of the fine-level voxel grid drops from 34.31% % to 30. 33%,, and the mloU that considers the accuracy of the voxel's 's semantic prediction drops from 20.00% to 16.61%. This shows the effectiveness of conditioning strategy being able disambiguate voxels at different depths.
Two-stage Reconstruction. We disentangle the voxel grid and appearance reconstruction stages to make the best use of different types of models, Using a single-stage model that simultancously
II practice, we test the upper bound of the single-stage model by feeding in ground-truth 1024 voxel grids, because otherwise the dense high-resolution condition will lead out-of-memory

---

a paln tree, a road with traffic, lined with troes and buildings, under a blue sky with seattered white clouds,
Reconstruct
Reconstruct
Figure 7: Text-2-Scene Generation. Given a text prompt, we could generate various multi-view images and lift them to 3D scenes with SCube. See Appendix D.3 for more text-2-scene results.
Resolution
M
PSNR1
LPIPS
256
4
18.58
0.62
1024
19.34
0.52
1024
19.34
0.48
Figure Effects GAN Postprocessing. Left: SCube+; Right: SCube.
Table 3: Ablation Study for Appearance Re- construction.
predicts the sparse voxels and the appearance from images, we only achieve PSNR/LPIPS of 17.88/0.57, in comparison to 19.34/0.48 when using the two-stage model. Here the values are the average of T' 5 and T 10 frames. In terms of geometry quality, the single-stage model is also significantly worse (up to 100x) in Chamfer distance than the two-stage model. Please refer to more details about the analysis of geometry quality in Appendix D.I.
Appearance Reconstruction. We validate the effect of voxel grid resolution and the number of Gaussians per voxel M the appearance reconstruction stage. Results are shown Tab. We find that higher-resolution voxel grids are crucial for capturing detailed geometry, and using larger number of Gaussians only slightly increases the performance. Thus, we set M 4 as moderate value for the final results. Compared in Fig. 8, the GAN-based postprocessing, despite the time cost, beneficial for producing high-quality images by sharpening the renderings. See Appendix D.2 for more visual comparisons.
5 Discussion
Conclusion. In this work, have introduced SCube, feed-forward method for large 3D scene reconstruction from images. Given sparse view non-overlapping images, our method is able to predict high-resolution 3D scene representation consisting of voxel-supported Gaussian splats (VoxSplat) and a light- sky panorama in a single forward pass within tens of seconds. We have demonstrated the effectiveness of our method on the Waymo Open Dataset, and have shown that our method outperforms the state-of-the-art methods in terms of reconstruction quality.
Limitations. Our method does suffer from some limitations. First, the current method is not able to handle complicated scenarios such as dynamic scenes under extreme lighting or weather conditions. Second, the quality of appearance in occluded regions still carries uncertainty. Third, the method itself still requires ground-truth 3D training data which is not always available for generic outdoor scenes. In future work, we plan to address these limitations by incorporating more advanced neural rendering techniques and by exploring more effective ways to generate training data.
10

---

References
[1] J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman. Zip-nerf: Anti-aliased grid-based neural radiance fields. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19697-19705, 2023.
[2] A. Blattmann, R. Rombach, H. Ling. T. Dockhorn, S. W. Kim, S. Fidler, and K. Kreis. Align your latents: regeuresoution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 22563-22575 2023.
[3] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom. nuscenes: multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621-11631, 2020.
[4] D. Charatan, Li, A. Tagliasacchi, and V. Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. arXiv preprint arXiv:2312. 12337, 2023.
[5] Y. Chen, J. Wang, Z. Yang. S. Manivasagam, and R. Urtasun. G3r: Gradient guided generalizable reconstruction. In ECCV 2024 Workshop on Wild 3D: 3D Modeling, Reconstruction, and Generation in the Wild, 2024.
[6] Y. Chen, H. Xu, C. Zheng, B. Zhuang. M. Pollefeys, A. Geiger, T.-J. Cham, and J. Cai. Mvsplat: Efficient gaussian splatting from sparse multi-view images, arXiv preprint arXiv:2403. 14627, 2024.
[7] G. Chou, Chugunov, and F. Heide. Gensdf: Two-stage learning of generalizable signed distance functions. Advances in Neural Information Processing Systems, 35:24905-244 22919,, 2022.
[8] T. Dao. Flashattention 2: Faster attention with better parallelism and work partitioning. arXiv preprint axXiv:2307.08691, 2023.
[9] K. Deng, A. Liu, J.-Y. Zhu, and D. Ramanan. Depth-supervised nerf: Fewer views and faster training for free. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 12882-12891, 2022.
[10] W. Falcon and The PyTorch Lightning team. PyTorch Lightning, Mar. 2019.
[11] J. Gao, C. Gu, Y. Lin, H. Zhu, X. Cao. L. Zhang, and Y. Yao. Relightable 3d gaussian: Real-time point cloud relighting with briff decomposition and ray tracing. arXiv preprint arXiv:2311.1604. 2023.
[12] Gao, Shen, Z. Wang. W. Chen, Yin, Li, O. Litany, Z. Gojeie, and S. Fidler. Get3d: A generative model of high quality 3d textured shapes learned from images. Advances In Neural Information Processing Systems, 35:31841- 1131144--18544 2022.
[13] T. Gierue, M. Kästingschäfer. S. Bernhard, and M. Salzmann. 6img-to-3d: Few-image large-scale outdoor driving seene reconstruction. arXiv preprint arXiv:2404. 12378, 2024.
[14] B. Graham and L. Van der Maaten. Submanifold sparse convolutional networks. arXiv preprint arXiv:1706.0 3130 2017.
[15] J. Hasselgren, N. Hofmann, and J. Munkberg. Shape, light, and material decomposition from images using monte carlo rendering and denoising. Advances in Neural Information Processing Systems, 35:22856- 22869, 2022.
[16] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[17] Y. Hong, K. Zhang, J. Gu. S. Bi, Y. Zhou, D. Liu, F. Liu, K. Sunkavalli, T. Bui, and H. Tan. Lrm: Large reconstruction model for single image to 3d. arXiv preprint S.Ii::2311.04000 2023.
[18] M. Hu, W. Yin, C. Zhang, Z. Cai, X. Long, H. Chen, K. Wang, G. Yu, C. Shen, and S. Shen. Metric3d v2: A versatile monocular geometric foundation model for zero-shot metric depth and surface normal estimation. arXiv preprint arXiv:2404.15006 2024.
[19] S.-M. Hu, Z.NN Liu, M.-H. Guo, J.-X. Cai, J. Huang, T.-J. Mu, and R. R. Martin. Subdivision-based mesh convolution networks. ACM Transactions on Graphics (TOG), 41(3):1-16, 2022.
[20] S. Huang. Z. Gojcic, J. Huang, A. Wieser, and K. Schindler. Dynamic 3d scene analysis by point cloud accumulation. In European Conference on Computer Vision, pages 674-690 Springer, 2022.

---

[21] Y. Huang. W. Zheng. Y. Zhang, J. Zhou, and J. Lu. Tri-perspective view for vision-basel 3d semantic occupancy prediction. In Proceedings of the IEEE/CVE conference computer vision and puttern recogmition, pages 9223-9232, 2023.
[22] Z. Inshad, S. Zakharov, K. Liu, V. Guizilini, T. Kollar, A. Gaidon, Kira, and R. Ambrus. Neo 360: Neural fields for sparse view synthesis of outdoor scenes. In ICCV, 2023.
[23] B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions Graphics, 42(4):1-14, 2023.
[24] D. Kingma and J. Ba. Adam: method for stochastic optimization. In ICLR, 2015.
[25] X. Li, Y. Zhang, and X. Ye. Drivingdiffusion: Layout-guided multi-view driving scene video generation with latent diffusion model. CoRR, abx/2310.07771, 2023.
[26] Y. Li, Z. Yu, C. Choy, C. Xiao, J. M. Alvarez, S. Fidler, C. Feng. and A. Anandkumar. Voxformer: Sparse voxel transformer for camera-base 3d semantic scene complction. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 9087-9098, 2023.
[27] T.-Y. Lin, Goyal, R. Girshick, K. He, and P. Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980 2988, 2017.
[28] M. Liu, R. Shi, L. Chen, Z. Zhang, C. Xu, X. Wei, H. Chen, C. Zeng, J. Gu, and H. Su. One-2-3-45++: Fast single image to 3d objects with consistent multi-view generation and 3d diffusion. arXiv preprint arXiv:2311.00 1000223,..0788,,, 2023.
[29] T. Liu, G. Wang. S. Hu, L. Shen, X. Ye, Y. Zang. Z. Cao, W. Li, and Z. Liu. Mysgaussian: Fast generalizable gaussian splatting reconstruction from multi view stereo. arXiv preprint arXiv:2405.12218, 2, 2024.
[30] Lorraine, K. Xie. X. Zeng, C.-H. Lin, Takikawa, N. Sharp. T.-Y. Lin, M.-Y. Liu, S. Fidler, and Lucas. Att3d: Amortized text-to-3d object synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17946-15 1-66--7799 17956, 2023.
[31] T. Lu, M. Yu, L. Xu, Y. Xiangli, L. Wang, D. Lin, and B. Dai. Scaffold-gs: Structured 3d gaussians for view-adaptive rendering. arXiv preprint arXiv:2312.00100, 2023.
[32] B. Mildenhall, P. Srinivasan, M. Tancik, T. Barron, R. Ramamoorth, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-166,, 2021.
[33] T. Müller, A. Evans, C. Schied, and A. Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM transactions graphics (TOG), 1155) 2022.
[34] M. Oquah, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby,, al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.071999 2023.
[35] G. Parmar, T. Park, S. Narasimhan, and J.-Y. Zhu. One-step image translation with text-to-image models. arXiv preprint arXiv:2403. /2036, 2024.
[36] S. Peng, M. Niemeyer. L. Mescheder, M. Pollefeys, and A. Geiger. Convolutional occupancy networks. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part III 16. pages 923-540. Springer, 2020.
[37] J. Philion and Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16, pages 194-210. Springer. 2020.
[38] B. Poole, A. Jain, J. T. Barron, and B. Mildenhall, Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 14988, 2022.
[39] K. Ren, Jiang. T. Lu, M. Yu, L. Xu, Z. Ni, and B. Dai. Octree-gs: Towards consistent real-time rendering with lod-structured 3d gaussians. arXiv preprint arXiv:2403.1788/ 2024.
[40] X. Ren, J. Huang, X. Zeng, K. Museth, S. Fidler, and F. Williams. Xcube: Large-scale 3d generative modeling using sparse voxel hierarchies. In CYPR, 2024.
[41] X. Ren, Z. Qian, and Q Chen. Video deblurring by fitting to test data. CoRR, abs/2012.05228,, 2020.
12

---

[42] X. Ren and X. Wang. Look outside the room: Synthesizing A consistent long-term 3d scene video from A single image. In IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 3553-3563, IEEE, 2022.
[43] B. Roessle, N. Müller, L. Porzi, S. R. Buld, P. Kontschieder, and M. NieBner. Ganerf: Leveraging discriminators to optimize neural radiance fields. ACM Transactions on Graphics (TOG), 42(6):1-14, 2023.
[44] Schonberger and J.-M. Frahm. Structure-from from-motion revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4104 4113, 2016.
[45] T. R. Shaham, T. Dekel, and T. Michaeli. Singam: Learning generative model from a single natural image. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4570-4580, 2019.
[46] T. Shen, J. Munkberg, Hasselgren, K. Yin. Z. Wang. W. Chen, Z. Gojeie, Fidler. N. Sharp, and Gao. Flexible isosurface extraction for gradient-based mesh optimization. ACM Transactions on Graphics (TOG), 42,4):1-11, 2023.
[47] Y. Shi, Wang. J. Ye, M. Long, K. Li, and X. Yang. Mydream: Multi-view diffusion for generation. arXiv preprint arXiv:2308. 16512, 2023.
[48] Shriram, A. Trevithick, L. Liu, and R. Ramamoorthi. Realmdreamer: Text-driven 3d seene generation with inpainting and depth diffusion. arXiv preprint arXiv:2404.071999 2024.
[49] V. Sitzmann, E. Chan, R. Tucker, N. Snavely, and G. Wetzstein. Metasdf: Meta-learning signed distance functions. Advances in Neural Information Processing Systems, 13:10136-10147, 2020.
[50] V. Sitzmann, J. Thies, Heide, M. NieBner, G. Weizstein, and M. Zollhofer. Deepvoxels: Learning persistent 3d feature embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2437-2446, 2019.
[51] S. Song, F. Yu, A. Zeng, A. X. Chang, M. Savva, and T. Funkhouser. Semantic scene completion from a single depth image. In Proceedings of the IEEE conference computer vision and puttern recognition, pages 1746-17544 2017.
[52] J. Sun, Y. Xie, L. Chen, X. Zhou, and H. Bao. Neuralrecon: Real-time coherent 3d reconstruction from monocular video. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 15598-156077 2021.
[53] P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Y. Chai, B. Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEFE/CVF conference on computer vision and pattern recognition, pages 2446-2454, 2020.
[54] H. Tang, Z. Liu, X. Li, Y. Lin, and S. Han. Torchsparse: Efficient point cloud inference engine. Proceedings of Machine Learning and Systems, 1:302-315, 2022.
[55] D. Tochilkin, D. Pankratz, 7. Liu, 7. Huang, A. Letts, Y. Li, D. Liang, C. Laforte, V. Jampani, and Y.-P. Cao. Triposr: Fast object reconstruction from a single image, arXiv preprint arXiv:2403.02/55/ 2024.
[56] V. Voleti, C.-H. Yao, M. Boss, A. Letts, Pankratz, D. Tochilkin, Laforte, R. Rombach, and V. Jampani. Sv3d: Novel multi-view synthesis and 3d generation from a single image using latent video diffusion. arXiv preprint arXiv:2403. /2008, 2024.
[57] S. Wang, V. Leroy, Cabon, B. Chidlovskii, and J. Revaud. Dust3r: Geometric vision made easy. arXiv preprint 1...:2212..11122..
[58] Wang, Q. Lvv, Yu, W. Hong, J. Qi, Wang, J. Ji, Yang. I.. Zhao, X. Song, et al. Cogylm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03072 2023.
[59] Z. Wang, C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process., 2004.
[60] Z. Wang. C. Lu, Y. Wang, F. Bao, C. Li, H. Su, and J. Zhu. Prolificdreamer: High-fidell and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems, 36. 2024.
[61] Y. Wei, L. Zhao, W. Zheng, Z. Zhu, J. Zhou, and J. Lu. Surroundoce: Multi-camera 3d occupancy prediction for autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21729-211111111 2023.
13

---

[62] F. Williams. J. Huang. J. Swartz, G. Klar, V. Thakkar, M. Cong. X. Ren, R. Li, C. Fuji-Tsang, S. Fidler. E. Sifakis, and K. Museth. fvdb: A deep-learning framework for sparse, large-scale, and high-performance spatial intelligence. ACM Transactions on Gruphics (TOG), 43(4):133:1-133:155 2024.
[63] R. Wu, B. Mildenhall, P. Henzler, K. Park, R. Gao, D. Watson, P. Srinivasan, D. Verbin, J. T. Barron, B. Poole, al. Reconfusion: reconstruction with diffuusion priors, arXiv preprint arXiv:2312.02981. 2023.
[64] E. Xie, W. Wang, Z. Yu, A. Anandkumar, M. Alvarez, and Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. In Neural Information Processing Systems (NeurIPS), 2021.
[65] Y. Xu, H. Tan, F. Luan, S. Bi, P. Wang, J. Li, Z. Shi, K. Sunkavalli, G. Weizstein, Z. Xu, et al. Dmv3d: Denoising multi-view diffusion using 3d large reconstruction model. arXiv preprint arXiv:2311.09211 2023.
[66] Y. Yan. H. Lin. C. Zhou, W. Wang. H. Sun, K. Zhan. X. Lang, X. Zhou, and S. Peng. Street gaussians for modeling dynamic urban scenes. arXiv preprint 2rXiv:2401.0..3 2024.
[67] Z. Ye, W. Li, S. Liu, P. Qiao, and Y. Dou. Absgs: Recovering fine details for 3d gaussian splatting, arXiv preprint X4X: 10484, 2024.
[68] A. Yu, V. Ye. M. Tancik, and A. Kanazawa, pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4578 45887, 2021.
[69] X. Yu, Y.-. Guo, Y. D. Liang, S.H.. Zhang, and X. Qi. Text-to-3d with classifier score distillation. arXiv preprint arXiv:2310.99415, 2023.
[70] Z. Yu, S. Peng, M. Niemeyer, T. Sattler, and A. Geiger. Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction. Advances neural information processing systems, 35:25018 25032, 2022.
[71] Z. Yu. T. Sattler. and A. Geiger. Gaussian opacity fields: Efficient and compact surface reconstruction unbounded scenes. arXiv preprint arXiv:2404. 10772, 2024,
[72] F. Zhang, Zhang, Q Zheng, R. Ma. Hua, Bao, Xu, Zou. sd-scenedreamer: Text-driven 3d consistent scene generation. arXiv preprint arXiv:2403.0943 2024.
[73] Zhang, S. Bi, H. Tan, Y. Xiangli, N. Zhao, K. Sunkavalli, and Z. Xu. Gs-Imm: Large reconstruction model for 3d gaussian splatting. arXiv preprint arXiv:2404.1977 2024.
[74] R. Zhang, Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
[75] X.-Y. Zheng. H. Pan, P.-. Wang, X. Tong, Y. Liu, and H.-Y. Shum. Locally attentional diffusion for controllable 3d shape generation. ACM Transactions on Graphics (TOG), 42(4):1-13, 2023.
[76] X. Zhou, Z. Lin, X. Shan, Wang. D. Sun, and M.-H. Yang, Drivinggaussian: Composite gaussian splatting for surrounding dynamic autonomous driving scenes. arXiv preprint arXiv:2312.07920 2023.
[77] V. Zyrianov, H. Che, Liu, and Wang. Lidardm; Generative lidar simulation in generated world. arXiv preprint arXiv:2404.0200 2024.
14

---

Appendix
-
Implementation Details
Additional Data Processing Details. For each data sample, we crop the point cloud obtained from S 4.1 into local chunk of 102.4m x 102.4m. The point cloud is then voxelized into the fine-level and coarse-level grids used 3.1 with 1024 and 256 resolutions respectively (with voxel sizes of 0.Im and 0.4m). Our dataset contains 20243 chunks for training and 5380 chunks for evaluation, out of the 798 training and 202 validation sequences.
Input and Evaluation Details. Waymo dataset provides 5 views for each camera frame, namely front, front-left, front-right, side-left and side-right. However, not all of the baseline methods we compared with in 4.2 can handle the unconventional camera intrinsic in the side-lett and side-right views. We hence only use the first three views (with a resolution of 1920 x 1280) in 4.2 for both the input and the evaluation metrics. However, in $ 4.3 we opt to use all 5 views for the input to both our method and the baseline due to compatibility and maximized performance.
For the baselines, the original PixelSplat [4] method does not have depth supervision. To make the comparison fair, we attempt to add a depth supervision loss to it. However, the experimental result shows that the additional loss hurts the performance shown in Tab. 4. We thus report the results of vanilla PixelSplat in the main paper.
T 5
T+ 10
PSNR1
SSIM
LPIPS|
PSNR
SSIM
LPIPS,
PixelSplat [4] PixelSplat [4] w/ Depth Supervision
20.11
0.70
0.60
18.77
0.66
0.62
19.91
0.58
0.66
18,87
0.56
0.67
Table 4: Comparison of PixelSplat and PixelSplat with Depth Supervision.
Training Details. The diffusion loss in Eq (2) is defined similar to [16, 40] with a u-parametrization as:
Diffusioo -.....
[ + VI- -,e, - -V-
(7)
where v--) is the diffusion network, / is the randomly sampled diffusion timestamp, and di is the scheduling factor for the diffusion process, whose details are referred to in [16].
We train all of our models using the Adam [24] optimizer with 81 = 0.9 and 81 0.999. We use PyTorch Lightnnig [10] for building our distributed training framework. For the voxel grid reconstruction stage, we train both coarse-level and fine-level voxel latent diffusion models with 64x NVIDIA Tesla A 100ss for 2 days. For the appearance reconstruction model, we train using 8x NVIDIA Tesla A 100s for days. Empirically, we use X = 1.0 for Depph in Eq (2). Additionally, we use A 0.9, A2 = 1.0, ÀSSIM = 0.1 and ALPIPS 0.6 in Eq (6). For image condition, we set the feature channel C = 32, the number of depth bins D 64, Anear 0.1 and žfr = 90.0. We linearly increase the interval depth bins.
B Network Architecture
Voxel Grid Reconstruction. We follow [40] to implement the Sparse Structure VAE and the Diffusion UNet using the sparse 3D deep learning framework fVDB [62]. Hyperparameters for training them are listed in Tab. 5 and Tab. 6. We pass the images to distilled DINO-v2 [34] ViT-B/14. We use four 2D convolutional layers (channel dims: [768, 256, 256, 32, 32], kernel size: 3. stride: 1) to further process the DINO-v2 output to predict the image feature and the depth distribution.
Appearance Reconstruction. We process the original input images with three 2D convolutional layers (channel dims: [3, 16, 32, 32), kernel size: 3, stride: [1, 1, 2)). For the last two convolutional
15

---

Waymo
Waymo
64
256
256' 10243
Model Size
14.9M
3.8M
Base Channels
64
32
Channels Multiple
1,2,4
1,2,4
Latent Dim
Batch Size
32
32
Epochs
50
50
Learning Rate
le-4
Table 5: Hyperparameters for VAE.
Waymo 64
Waymo 256
Diffusion Steps
1000
Noise Schedule
linear
Model Size
728M
83.0M
Base Channels
192
64
Depth
2
Channels Multiple
1,2,4,4
1,2,2,4
Heads
8
Attention Resolution
16
32
Dropout
0.0
0.0
Batch Size
512
256
Iterations
40K
20K
Learning Rate
Se-5
Table 6: Hyperparameters for voxel latent diffusion models.
layers, we set the residual connections. We additionally positionally encode each voxel and then concatenate the positional encoding [32] of each voxel with the corresponding voxel feature after ray casting. We then apply a 3D sparse UNet to output per-Gaussian parameters. We use GT voxels in appearance reconstruction training. Hyperparameters of this 3D sparse UNet are listed Tab. 7.
Model Size
Base Channels
Channels Multiple
Batch Size
Epochs
Learning Rate
4.3M
32
2, 4]
32
15
le-4
Table 7:
Hyperparameters for 3D sparse UNet in appearance reconstruction stage.
Sky Panorama for Background. For the sky panorama model, we set H, 768, W, 1536 in the training stage and increase H, 1024, W, 2048 in the inference time. To decode sampled sky features into the RGB image. we utilize 2D CNN network reducing the channel from 32 to 16 to with stride 1, keeping the spatial resolution unchanged.
C SCubet without Per-scene Training
In 3.3 we introduce GAN postprocessing module refine the rendered images, which is finetuned on each scene. To further improve the efficiency of our method, we hereby present postprocessing module that is jointly trained on the full dataset, without the need of per-scene finetuning. Specifically, we replace the original GAN with pix2pix- train with image pairs inferred from our model and the ground truths. The results are shown in Fig. 9. This improved model not only reduces the voxel block artifacts, but also resolve the ISP inconsistencies within the image. After enabling this module, the FPS drops from 138 to 20 but can still be visualized interactively.
model [35] (which we denote as SCube+*) and
16

---

Reconstruction (7)
Prediction (T +)
Prediction (7 + 10)
PSNR1
SSIM1
I.IIS
PSNRT
SSIM
LPIPS
PSNR1
SSIM
LPIPS.
SCube
25.90
0.77
0.45
19.90
0.72
0.47
18.78
0.70
0.49
SCubc+
28.01
0.81
0.25
22.32
0.74
0.34
21.09
0.72
0.38
SCube+* 22.59
0.68
0.38
20.37
0.66
0.41
19.65
0.65
0.42
Table Quantitative Comparisons on 3D Reconstruction, The metrics are computed both the input frame T and future frames. t: higher is better, J: lower is better.
SCube
SCube+4
SCube
SCube+*
SCube
SCube+*
Figure 9: SCube+*, Results from the postprocessing network without per-scene optimization. White- balance inconsistencies from different views (marked in red box) can fixed.
D
Additional Results
In this section, we provide more qualitative results all datasets. We additionally provide supplementary video in the accompanying files to better illustrate our results.
D.1 Geometry Quality
We note that the uncertainty of the scene geometry given input images is large, and the problem that the model tackles is indeed non-trivial and sometimes even iol-posed. To demonstrate this, we compute the percentage of occluded voxels (that are invisible from the input images) w.r.t. all the ground-truth voxels, and the number is around 80%. To quantitatively evaluate the geometry quality, compute an additional metric called 'voxel Chamfer distance" that measures the L.2 2-Cahmferr distance between the predicted voxels and ground-truth voxels (that are pixel-aligned), divided by the voxel size. This metric reflects the geometric accuracy of prediction by measuring on average how many voxels is the prediction apart from the ground truth. The results on Waymo Open Dataset shown Tab, 9.
0.5 (median)
Quantile
0.6
0.7
0.8
0.9
Ours
0.26
0.28
0.32
0.37
0.51
Table 9: Geometry Quality Comparison. We show the voxel Chamfer distance comparison between our two-stage model and single-stage non-diffusion model.
Tab. indicates that on 90% of the test samples, the predicted voxel grid only half of a voxel off from the ground truth. We note that during data curation process, there could be errors in the ground-truth voxels (e.g., due to COLMAP failures), accounting for the outliers in the above metric. In the meantime, predicted results are decent even though the ground truth is corrupted due to the lack of motion in the ego car. This demonstrates the robustness of method.
visualize the sample with the worst voxel Chamfer distance in Fig. 10. The
D.2 Visual Ablation Study
In addition to the quantitative ablation study Tab. 3, we present a qualitative demonstration in Fig. 11. For the single-stage model, test the upper bound of it by feeding the ground-truth 10243 voxel grids because otherwise the fully-dense high-resolution condition will lead to out-of of-ommmoor..... The qualitative results match the numbers, showing the importance of using higher- resolution voxel grids and the two-stage model.
17

---

Dataset Ground-truth Voxels
Predicted Voxels
Recorstructed 3D Gaussian Soene
Fimal Rendering
Figure 10: Result on the data sample with the worst voxel Chamfer distance. We show geometry reconstruction and the image renderings.
(a)
(b)
(c)
(d)
Figure 11: Visual Ablation Study. (a) SCube+ (b) SCube (c) SCube with 256 resolution input grid (d) Single-stage model. Zoom in for a better view.
18

---

D.3 Additional Results on Text-2-Scene Generation
We provide additional text-2-scene generation results in Fig. 12 and Fig. 13.
A residential neighborhood features houses with well-maintained gardens, autumn-colored trees, lawns with scattered leaves, parked cars, driveways, and clear blue skies.
Reconstruct
A residential area features multiple houses, some with specific decorations and vehicles parked outside, including white pickup truck and gray car, along with various greenery and utility elements.
Reconstruct
Figure 12: More Text-2-Scent Generation. The generated multi-view images may contain flaws, while SCube is still able to reconstruct the 3D scenes.
19

---

featuree A suburban neighborhood features two-story houses with reddish-browe roofs and beige walls, marked roads, various parked vehicles, stop signs, and mixture of gravel, rocks, and trees providing shade on sunny day.
Reconstruct
A suburban neighborhood features park with green trees, residential houses with red-tiled roofs. streets with bike lane signs and white markings, well-maintained lawns, and sidewalks.
Reconstruct
Figure 13: More Text-2-Scene Generation.
20

---

D.4 Additional Results on Large-scale Scene Reconstruction
We provide additional results on large-scale scene reconstruction from real-world captures in Fig. 14.
Novel Views
Render
Reconstruct
Input Images
Novel Views
Render
Reconstruct
Input Images
Figure 14: More Novel View Synthesis. Our method is able to synthesis extreme novel views.
21

---

D.5 Additional Results on LiDAR Simulation
We provide additional LiDAR Simulation results in Fig. 15. We also show the result on long sequence input in Fig. 16.
Input Front View
Simulated LiDAR Sequence
Figure 15: More LiDAR Simulation results.
Figure 16: SCube with Long Sequence Input. Up: reconstructed scene with appearance. Down:
LiDAR simulation result. We chunk the long sequence into clips and apply out method iteratively.
22

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes].
Justification: We make three claims: (1) We are the first reconstruct GSplats grounded in a predicted 3D space. To the best of our knowledge, aware of other work in the literature doing this. (2) We obtain full 3D reconstruction from which we can render novel views single forward pass. We show many instances of this in the experiments section (both novel views, semantic reconstruction, and LiDAR resimulation). (3) We are SOTA when compared to baseline methods which perform reconstruction from sparse views. We compare against a wide variety of baselines and demonstrate qualitative and quantitative improvements over these, (4) Our inference pipeline takes under 20 seconds. In the meantime, simply report this fact in the paper.
model to directly
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize to other settings. It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes].
Justification: Section 5 includes an explicit limitations section. We remark on difficulties with dynamic scenes and oui-i--distribution weather and lighting conditions, challenges reconstructing high-quality appearance in highly occluded regions, and training data limita- tions. See this section for more details.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-ter system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they seale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
23

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We describe our method in great detail and provide enough information that researcher familiar with diffusion models and neural rendering should be able reproduce the pipeline. We describe our experiments in detail as well and run them on Waymo Open which is a public dataset accessible to all. The appendix includes detailed training and network architecture information. Together, these should make our results fully reproducible.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
24

---

(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Due to institutional constraints, we are not able to release the code until the paper fully accepted. Upon acceptance, we will release all code and data required to reproduce this work.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
public/ public/guides. gwwggguiid/seeee Salaas//ooeSuubisic wwriiisssoooiicyyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ //nips. wwtpcccccppppppptceccccc gwwces/////////eeeceooocccc.ss /ood//oooeeesssseeeeeeee cettuummisssoonii wsissnoliccyy. for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes].
Justification:
See descrition Section 4 and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
25

---

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: It is not clear what assumptions should be made emror distributions. Un- fortunately, in this particular literature, error bars and confidence intervals are not typically reported.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report these explicitly in the experiments section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: / ht////eurippp cc/publiccccccccccccccccccc ctuiii/ ines?
Answer: [Yes]
Justification:
We have reviewed the code ethics and believe our paper fully conforms to
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
26

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: We feel there are no direct negative societal impacts of this work as it is foundational method for reconstruction. While such foundational method could eventually be used in pipelines with potential negative consequences (c.g. offensive applications in military settings),
do not believe our method directly enables this. Furthermore, by training on an open driving dataset, we focus simply on reconstructing the 3D world from car footage, which alone cannot be used negatively impact society.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: We don't believe our model poses an immediate risk of misuse. We believe will help advance research in 3D reconstruction from images.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
27

---

12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite relevant related works and datasets. All image and video assets in the paper and supplementary material are our own.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paporswithcode. thcode com datasets
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do release new assets alongside the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We did no crowdsourcing or human subject experiments.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
28

---

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
We did not study on human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "IZDNbXk5XG", "reviewer": "Reviewer_CDsX", "summary": "The paper proposes a new method to reconstruct 3D outdoor scenes from a sparse set of posed images. The key idea is to utilize a new hybrid 3D representation, which assigns 3D Gaussians to each sparse voxel. Given input images, the paper first adapts XCube to condition on sparse view images. Along with the generated fine-level voxels, the paper then designs a feed-forward model to reconstruct the appearance. This proposed pipeline can also be used in text-to-scene generation. Experiments show that the proposed method outperforms existing approaches.", "questions": "See Weaknesses", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Originality:\n\nThe paper is trying to solve a challenging problem: efficient large-scale scene reconstruction from sparse posed images. The motivation is to utilize the data prior to complement the sparse images. The proposed representation is a straightforward combination of XCube with 3D Gaussians (simply assigning 3D Gaussians into each voxel), which has also been widely applied (e.g., Scaffold-GS). While combining these two representations is acceptable, the concern is that the overall pipeline design seems more like a simple combination of two representations: first generating XCube, then generating 3D Gaussians with geometry & images. However, the problem to be solved is important, and this paper might inspire more researchers to work on this problem.\n\nQuality:\n\nThe submission seems technically sound.\n\nClarity:\n\nThe paper is mostly clear and easy to read.\n\nSignificance:\n\nThough there are some concerns with the methods and results, the paper is aiming at solving a very good problem.", "weaknesses": "- One concern is that the novelties of this paper are not very clear. As stated above (Originality), the paper seems to simply use two stages to combine the two representations in a simple way: one for XCube, and one for 3D Gaussians. It would be better if the author can clearly state the contributions.\n\n- As a two-stage method, it would be better to discuss more about: (1) First stage: how to evaluate whether the reconstructed voxels align with the input images? (2) Second stage: how robust is the model when the input images and the voxels are not aligned?\n\n- L187: Considering the temporal inputs, can the methods give consistent results temporally? Adding results might be better.\n\n- L244-245: In Table 1, SCube seems to have very similar results to PixelSplat for future prediction. Why? An obviously better reconstruction result alone seems not able to show the effectiveness of the proposed method. The appearance reconstruction uses generated geometry (fine-level voxels) as input, and adds some tricks such as sky modeling. It is unknown how they affect the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback, which helps us improve SCube. We are happy that you agree that the problem we solve is challenging and important, and the techniques are sound and well-written. In the following, we will quote your original comments and try to resolve them.\n\n> **Concerns about novelty.** The paper seems to simply use two stages to combine the two representations in a simple way: one for XCube, and one for 3D Gaussians. It would be better if the author can clearly state the contributions.\n\nOur main contribution is the novel framework that enables 3D priors for large-scale fast scene reconstruction that no prior work achieves. We humbly believe that our work adds new insights and knowledge into the literature to achieve ‘promising results’, as echoed by reviewer NKmV. \nAlthough at a high level, our method could be decomposed into two parts, we have made non-trivial technical contributions to synergize and tame them for the sparse-image-based 3D reconstruction task. This includes the image-based conditioning techniques to handle occlusions and 2D-3D alignments in the sparse diffusion architecture ($\\S$ 3.1), the hybrid scene representation that facilitates both learning and fast rendering ($\\S$ 3.2), along with effective training procedures ($\\S$ 4.1). We remark that our framework significantly outperforms baseline approaches on the same tasks, enabling reconstructing scenes with $102.4\\text{m}$ in scale with nice geometry under seconds. We also demonstrate a couple of useful applications available only with our proposed method ($\\S$ 3.3).\nHence, a re-evaluation of our novelty would be greatly appreciated.\n\n> **Alignments of voxels w.r.t. input images.** As a two-stage method, it would be better to discuss more about: (1) First stage: how to evaluate whether the reconstructed voxels align with the input images? (2) Second stage: how robust is the model when the input images and the voxels are not aligned?\n\nTo quantitatively evaluate the pixel-voxel alignments (question (1)), we compute an additional metric called 'voxel Chamfer distance' that measures the L2 Chamfer distance between predicted voxels and ground-truth voxels (that are pixel-aligned), divided by the voxel size. This metric reflects the geometric accuracy of our prediction by measuring on average how many voxels is the prediction apart from the ground truth. The results on Waymo Open Dataset are as follows:\n\n\n| Quantile               | 0.5 (median) | 0.6  | 0.7  | 0.8  | 0.9  |\n| ---------------------- | ------------ | ---- | ---- | ---- | ---- |\n| Voxel Chamfer distance | **0.26**     | 0.28 | 0.32 | 0.37 | 0.51 |\n\nThe table indicates that on 90% of the test samples, the predicted voxel grid is only half of a voxel off from the ground truth. We note that during our data curation process, there could be errors in the ground-truth voxels (due to, e.g., COLMAP failures), accounting for the outliers in the above metric. \nTo answer question (2), we visualize the sample with the worst voxel Chamfer distance in `Fig. D` of the rebuttal PDF: we show that the predicted results are decent even though the ground truth is corrupted due to the lack of motion in the ego car. This also indicates that the voxels and the images are rarely misaligned, demonstrating the robustness of our method.\n\n> **Temporal inputs.** Considering the temporal inputs, can the methods give consistent results temporally? Adding results might be better.\n\nThank you for your suggestion. We apply our full pipeline to a set of temporal inputs comprising of consecutive frames ($T$, $T+5$, $T+10$, $T+15$, $T+20$), and show the reconstruction and LiDAR simulation results in `Fig. A` in the rebuttal PDF. Our method is able to produce consistent 3D along the driving trajectory though the frames are fed independently.\n\n> **Future prediction results vs. PixelSplat.** In Table 1, SCube seems to have very similar results to PixelSplat for future prediction. Why? The appearance reconstruction uses generated geometry (fine-level voxels) as input and adds some tricks such as sky modeling. It is unknown how they affect the results.\n\nWe kindly refer to Fig. 4 (2nd row) in the paper and the supplement video (at 00:43) for a 3D visualization of the reconstruction results for PixelSplat. The predicted geometry of PixelSplat is degenerated to planar patches, and the future rendering has a lot of black hole artifacts. In this particular scenario, PSNR is less sensitive to these artifacts and is admittedly not an ideal indicator of the quality, as echoed in [a, b], while LPIPS can capture such artifacts and faithfully reflect the results (Ours = 0.47 vs PixelSplat = 0.60 at $T+5$, where the lower the better).\nThe appearance reconstruction module does require fine-level voxels as input otherwise it cannot determine the correct positions to generate the Gaussians. As our geometry of interest is limited to a square region around the ego vehicle, the VoxSplat representation cannot cover faraway regions such as the sky. Without sky modeling we cannot render the sky regions, and we show qualitative results in the supplementary video at, e.g., 00:09.\n\n***Reference:***\n\n[a] Rockwell et al. Pixelsynth: Generating a 3d-consistent experience from a single image. ICCV 2021.\n\n[b] Ren et al. Look outside the room: Synthesizing a consistent long-term 3d scene video from a single image. CVPR 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KTUZCrn99F", "reviewer": "Reviewer_6vwn", "summary": "The paper introduces a pipeline for street scene reconstruction given a sparse set of images as input. The reconstruction process follows a feed-forward manner. The method builds upon the existing XCube work. First, it generates sparse voxels of the scene to represent the geometries, then each voxel feature is decoded into Gaussian primitives for appearance rendering. Experimental results, both quantitative and qualitative, show that SCube can reconstruct the scene with high quality and provides benefits for downstream applications such as autonomous driving simulation.", "questions": "(1)\tWhy is the GAN loss used independently for each scene in the inference stage, instead of using it in the training stage?", "limitations": "Currently, the results are presented in the sparse view settings of a small region of the street scene. How can we reconstruct a very large scene using a long sequence of images in a feed-forward manner? This is an interesting topic for future work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The results are impressive. Given the sparse images with little overlap, the reconstructed Gaussians are of high quality and show good generalization ability in novel views.\n\n(2) Overall, the method sounds reasonable. Technical details are provided, and I believe the results are reproducible.", "weaknesses": "(1)\tI am curious if we really need a diffusion model for this task. Since there are several images as conditions, the uncertainty (or randomness) of the output should be very small. Why not just train a regressor for the sparse voxel reconstruction? Is there any specific motivation for using a diffusion model?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the constructive feedback and your positive comments on the results and technical contribution. We appreciate your effort in this process. In what follows we will quote your questions and try to resolve them.\n\n> **Necessity of a Diffusion Model.** I am curious if we really need a diffusion model for this task. Since there are several images as conditions, the uncertainty (or randomness) of the output should be very small. Why not just train a regressor for the sparse voxel reconstruction?\n\nWe note that the uncertainty of the scene geometry given our input images is still large, and the problem that the model tackles is indeed non-trivial and sometimes even ill-posed. To demonstrate this, we compute the percentage of occluded voxels (that are invisible from the input images) w.r.t. all the ground-truth voxels, and the number is around **80%**. In addition to the generative modeling power, compared to a simple regressor, multi-step denoising diffusion models are empirically verified to have more data-fitting power [a]. \n\nAs a comparison specific to our task, we trained a single-step model that is conditioned on the input images and directly regresses the desired sparse voxel grid. To measure the geometric accuracy, we use a metric called 'voxel Chamfer distance' that measures the L2 Chamfer distance between predicted voxels and ground-truth voxels, divided by the voxel size. The results on Waymo Open Dataset are as follows:\n\n\n| Quantile               | 0.5 (median) | 0.6      | 0.7      | 0.8      | 0.9      |\n| ---------------------- | ------------ | -------- | -------- | -------- | -------- |\n| Simple regressor model | 15.46        | 19.12    | 22.61    | 34.81    | 52.57    |\n| Our diffusion model    | **0.26**     | **0.28** | **0.32** | **0.37** | **0.51** |\n\nAs demonstrated in the above table, the diffusion model significantly outperforms a simple regressor.\n\n> **Sharing postprocessing GANs across datasets.** Why is the GAN loss used independently for each scene in the inference stage, instead of using it in the training stage?\n\nThank you for the constructive suggestion -- we hereby present a postprocessing module that is jointly trained on the full dataset, without the need of per-scene finetuning. Specifically, we replace the original GAN with a more modern Img2img-Turbo model [b] and show the results in `Fig. B` in the PDF file. Note that while the essence idea of applying neural postprocessing on the rendered images stays the same, the new module removes the need for per-scene training for high-quality rendering. We will adopt this module in our revised version, and thanks for your suggestion again!\n\n> **Extending to longer input sequences.** Currently, the results are presented in the sparse view settings of a small region of the street scene. How can we reconstruct a very large scene using a long sequence of images in a feed-forward manner? This is an interesting topic for future work.\n\nWe agree that this is an interesting and meaningful direction of future work and we present some preliminary results in `Fig. A` in the rebuttal PDF. We feed multiple frames of input images independently into our model and stitch the inference results from multiple timesteps together, without any finetuning. Results show that the stitched 3D scenes are temporally consistent despite minor artifacts, and one can similarly apply a longer LiDAR simulation session over the reconstruction. We also note that extending to even longer sequences could be implemented by recursively conditioning on past (latent) reconstructions, and we are actively exploring its feasibility.\n\n\n***Reference:***\n\n[a] Yang et al. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys 2023.\n\n[b] Parmar et al. One-step image translation with text-to-image models. arXiv:2403.12036.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9HphIpPU6B", "reviewer": "Reviewer_PK9U", "summary": "- The paper proposes a new method for sparse-view 3D reconstruction using 3DGS.\n- The framework uses two stages:\n    1) it learns a latent voxel grid (based on XCube) to represent the geometry.\n    2) it trains an appearance model to decode the latent voxel grid into a set of Gaussians\n- The method further uses a background model to handle the sky.\n- The authors propose an (optional) GAN postprocessing which improves the visual quality at the cost of longer (20m) per-scene optimization.\n- The method was evaluated on the Waymo Open Dataset, and it outperforms compared methods.", "questions": "- How does the method compare to other methods like MVSplatting, MVSGaussians? How does it differ? How do you think it would compare in terms of performance?", "limitations": "- Limitations are properly discussed in the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is very well written and easy to follow.\n- The method is novel and the results look good.\n- I really like the idea of having probabilistic latent space representation (VAE).\n- The paper contains an ablation study illustrating the tradeoff between model complexity and performance as various hyperparameters are changed.", "weaknesses": "- Section Method \"Training and Inference\" is not very clear. It would be preferable to expand the diffusion loss from Appendix A, and clearly describe the input/targets.\n- I would also like to have more details on the Sky Panorama model.\n- The method was evaluated on a single dataset. It would be interesting to see how the method performs on other datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your encouraging feedback for SCube, we are glad to understand that our method is easy to follow and you enjoy the idea and the results of our paper. In the following text, we will try to address your concerns.\n\n> **Details about the diffusion loss.** Section \"Training and Inference\" is not very clear. It would be preferable to expand the diffusion loss from Appendix A, and clearly describe the input/targets.\n\nThank you for your suggestion, and we will expand the descriptions and clarify the input and targets in the revised paper.\nThe training loss $\\mathcal{L}_\\text{diffusion}$ (in Eq. 7) is derived by maximizing the log-likelihood of the data $\\mathbf{X}$ distribution. It can also be viewed as a score-matching process whose loss encourages the alignment of the predicted stochastic score and the ground truth. In each training iteration, we sample a random variable $\\mathbf{X}$ (in our case this is the sparse voxel hierarchy) and pass its noised version to the sparse U-Net, which aims to predict the added Gaussian noise ${\\epsilon}$. Practically we re-parametrize the network with the v-parametrization technique as introduced in [a].\n\n> **Details about sky panorama model.** I would also like to have more details on the Sky Panorama model.\n\nModeling the sky independently plays an important role in outdoor scene reconstruction [b], which helps the model to disentangle the foreground and the sky at infinite depth. We adopt a 2D panorama $\\mathbf{L} \\in R^{H\\times W \\times C}$ as the sky representation, and the sky modeling consists of the following two steps:\n\n1) *Encode the 2D sky panorama*: Since the 2D panorama is an expanded (hemi-)sphere with inverse equirectangular projection, we can get a cartesian coordinate $P=(x,y,z)$ on a unit sphere for each pixel $(u, v)$ in the panorama $\\mathbf{L}$. We then project $P$ to the image plane to retrieve the image feature (note that we zero the translation part of the camera pose since only the view direction decides the sky color). We also leverage the sky mask which is computed as the inverse of the rendered sparse voxel grid mask to make sure only sky areas are retrieved. After this step, sky features are stored in $\\mathbf{L}$.\n2) *Sample and decode features from the panorama*: Given a camera with image resolution $(H', W')$, intrinsics $K$ and camera pose $\\xi$, we generate camera rays and calculate their hit point on the unit sphere. With equirectangular projection, we obtain their pixel positions on the 2D sky panorama and query its features via trilinear interpolation, resulting in a 2D feature map $\\mathbf{F} \\in {R}^{H'\\times W' \\times C}$. We finally decode it with a 2D CNN network to get the RGB sky image of $\\mathbf{I} \\in {R}^{H'\\times W' \\times 3}$, which is alpha-composited with the foreground.\n\nWe use $H=1024, W=2048, C=32$ for the 2D sky panorama in our implementation. In the sampling and decoding step, the 2D CNN network includes several Convolution--BatchNorm--ReLU layers, reducing the channel number from 32 to 16 to 3 with stride 1. We will make the descriptions utmost clear in our revision.\n\n> **Results on other datasets.** The method was evaluated on a single dataset. It would be interesting to see how the method performs on other datasets.\n\nOur method is able to generalize well to novel inputs, as already demonstrated in the Text-to-Scene generation experiment (Fig. 7), where the images are taken from a multi-view diffusion generative model. Furthermore, we demonstrate the general applicability of our method on a new larger-scale internal dataset consisting of 1.1k clips. Each clip contains about 200 frames and has dense point clouds for the GT voxel generation. We show a reconstruction and novel view synthesis sample on this dataset in `Fig. C` in the PDF file. The result shows that our model is capable of learning over a larger dataset and obtaining better reconstructions. Notably, in this example, we demonstrate the flexibility of our model that also supports back-view inputs (REAR-LEFT and REAR-RIGHT) for a 360-degree novel view rendering.\n\n\n> **Comparison to MVSplat and MVSGaussians.** How does the method compare to other methods like MVSplatting, MVSGaussians? How does it differ? How do you think it would compare in terms of performance?\n\nThe main difference of our model compared to multi-view stereo-based methods such as MVSplat or MVSGaussians is the usage of true 3D priors. Specifically, MVSplat uses cross-attention to learn image-space correspondences and infer pixel-aligned Gaussians, and MVSGaussians lift from the images to a cost volume that is strongly correlated to the target view for rendering. While these methods achieve good rendering quality, they cannot learn the distribution of the 3D geometry. Comparably:\n- Quality-wise, MVS-based method cannot enable *extreme* novel view synthesis such as the bird-eye view of the scene as shown in Fig. 1 and the supplementary video. They also cannot recover *highly-occluded regions* as shown in Fig. 5.\n- Performance-wise, with the support of the fast sparse convolution infrastructure and the Gaussian renderer, our reconstruction is built within seconds and rendered in real-time. This is comparable to the baselines.\n\nThinking ahead, the availability of the 3D prior and latent spaces could allow more explicit 3D editing or control capability (e.g. for the traffic) in future works.\n\n\n***Reference:***\n\n[a] Salimans et al. Progressive Distillation for Fast Sampling of Diffusion Models. ICLR 2022.\n\n[b] Wang et al. Neural Fields meet Explicit Geometric Representations for Inverse Rendering of Urban Scenes. CVPR 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VDS8xIJCvv", "reviewer": "Reviewer_NKmV", "summary": "This paper presents a method to reconstruct large-scale 3D scenes from a sparse set of posed images. The method has two stages: the first stage is for voxel grid reconstruction, which is based on the XCube [38], a 3D generative model, and the authors introduce the image condition to the model.  The second stage is for appearance reconstruction, which is a feedforward appearance prediction model to predict a certain number of Gaussians for each voxel. The authors also propose a GAN-based post-processing step to boost the appearance of the reconstructed scene. The proposed method is evaluated on the Waymo self-driving dataset, showing the superiority of the proposed method compared to prior art on 3D reconstruction tasks. Besides, the authors also demonstrate the applications of the proposed method, such as LiDAR simulation and text-to-scene generation.", "questions": "The most important questions that I would like the authors to address are:\n- Geometric accuracy evaluation of the reconstructed scenes.", "limitations": "Yes, the authors have discussed the limitations of the proposed method.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed method addresses an important problem in 3D vision, i.e., reconstructing large-scale 3D scenes from a sparse set of posed images. The authors cleverly take advantage of the pre-trained 3D diffusion generative model to relieve the ill-posedness of the problem.\n- Although technically the proposed method is not entirely new, the paper presents a framework to combine the voxel grid reconstruction and appearance prediction, and the presented result looks promising to me.\n- The paper is well-written and easy to follow. The authors provide a clear explanation of the proposed method and the experiments.", "weaknesses": "- The proposed representation's abbreviation \"VoxSplats\" is somewhat confusing to me, since the term \"splatting\" in original 3DGS refers to the rendering technique that splats the 3D Gaussians onto the image plane, rather than the 3D representation. The proposed representation is more like a hybrid of voxel grids and Gaussian representation. The authors may consider using a more accurate term to describe the representation.\n- It's unclear how the foreground and background are separated in the proposed method. Using a panorama to represent the sky makes sense as the sky is usually far away from the scene, but some buildings are treated as the background and seem to be represented by the panorama as well in the supplementary video. A more detailed explanation and discussion of the foreground-background separation is needed.\n- There are some lighting/exposure/white-balance inconsistencies in the reconstructed scenes in the supplementary video. The authors should discuss the limitations of the proposed method in handling the lighting/exposure/white-balance variations in the input images.\n- For SCube+, how to guarantee consistency among the rendered views? Also, rendering with the proposed GAN-based post-processing step is time-consuming, which would destroy the valuable real-time rendering performance of the Gaussian Splatting. The authors should discuss these issues in the paper.\n- Missing the evaluation of the geometric accuracy of the reconstructed scenes. As the rendering quality of SCube is not so good, geometric accuracy is crucial to the applications of the reconstructed scenes. The authors should provide a detailed evaluation of the geometric accuracy of the reconstructed scenes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your detailed review. We are glad to understand that you have found the results promising and the paper well-written. We hereby quote and answer the raised questions as below:\n\n> **Naming of VoxSplat.** The authors may consider using a more accurate term to describe the representation. \n\n Thank you for the constructive feedback, and we are actively considering renaming our representation to, e.g., `VoxGaussian` for accuracy. We would still use the term `VoxSplat` in this rebuttal though to avoid confusion, but we will change the name in the revised version.\n\n> **Foreground-background separation.** It's unclear how the foreground and background are separated. Using a panorama to represent the sky makes sense as the sky is usually far away from the scene, but some buildings are treated as the background and seem to be represented by the panorama.\n\nConceptually the scene to be reconstructed could be decomposed into three parts:\n\n1. *Foreground*: This refers to the sparse voxel grids within the 102.4m region enclosing the camera (as visualized in Appendix A). This is the main part we reconstruct using our VoxSplat representation.\n2. *Background*: This is the sky panorama that we explained in L160-165, which is assumed to be at infinity. \n3. *Midground*: This is the region between *Foreground* and *Background* which, as the reviewer has pointed out, we do not explicitly tackle in our framework.\n\nFor the novel view synthesis task where we render the sky panorama, buildings in the *Midground* are baked into the *Background*. We humbly argue that this is a sufficient approximation for this task at hand, since the novel viewpoint to be rendered is not drastically far away from the input views, so the buildings far away can be approximated in the panorama. For geometry reconstruction or bird-eye-view visualization purposes, the *Midground* as well as the *Background* are both cropped. \nNotably, since our ultimate goal is to build *true 3D* geometry, developing ad-hoc ways for modeling the midground is not a principled solution for us. To solve this in a principled manner, we choose to apply our framework to a longer sequence input (as shown in `Fig.A` in the rebuttal PDF), in which case the *Midground* for the beginning frames is just *Foreground* in the subsequent frames and could be modeled explicitly by the VoxSplat representation.\n\n> **Handling white-balance inconsistencies.** There are some lighting/exposure/white-balance inconsistencies in the reconstructed scenes in the supplementary video.\n\nWe agree that this is not explicitly handled currently and will add more discussions to the limitations. In fact, we note that our image-space postprocessing module (L178-186, i.e., **SCube+**) can already mitigate this issue. Notably, we have improved this part of the pipeline to remove the need for per-scene training by introducing a new postprocessing renderer that jointly learns on *all* the dataset samples. Similarly, we take the rendered image (with potential artifacts) and the ground-truth images as the training pairs, and find that this simple module could already resolve the ISP inconsistencies within the image. Please refer to `Fig.B` in the rebuttal PDF for a visualization.\n\n> **Rendering in SCube+.** For SCube+, how to guarantee consistency among the rendered views? Also, rendering with the proposed GAN-based post-processing step is time-consuming.\n\nThe GAN post-processing module in SCube+ only aims to remove the quantization artifacts and refine the details (ref. Fig. 8), but will leave the general scene structure untouched. This means that our renderings are still *grounded/supported* by the underlying VoxSplat representation that is guaranteed to be consistent. Empirically we observe the recovered fine details in SCube+ are also decently consistent as shown in the supplementary video (e.g. 00:27 lower-right). We agree that enforcing consistency in a more principled way is meaningful future work and note that one may apply a video model instead to enforce temporal consistency.\n\nAfter enabling this module, the FPS drops from 138 to 20 but can still be visualized interactively. We will discuss this in our limitation and aim to improve the timing in future work. Please note that this step is only needed if higher-quality rendering is required, and one can alternatively choose to finetune the 3DGS representation with optimization as we have demonstrated in $\\S$ 4.3.\n\n> **Evaluation of geometric accuracy.** geometric accuracy is crucial to the applications of the reconstructed scenes. The authors should provide a detailed evaluation of the geometric accuracy of the reconstructed scenes.\n\nThank you for the suggestion. We already have a light measurement of voxel IoU at L278-283 in terms of semantic voxel accuracy and we will include a more detailed evaluation of the geometric accuracy in the revision.\nSpecifically, we compute an additional metric called 'voxel Chamfer distance' that measures the L2 Chamfer distance between predicted voxels and ground-truth voxels, divided by the voxel size. This metric reflects the geometric accuracy of our prediction by measuring on average how many voxels is the prediction apart from the ground truth. The results on Waymo Open Dataset are as follows:\n\n\n| Quantile               | 0.5 (median) | 0.6  | 0.7  | 0.8  | 0.9  |\n| ---------------------- | ------------ | ---- | ---- | ---- | ---- |\n| Voxel Chamfer distance | **0.26**     | 0.28 | 0.32 | 0.37 | 0.51 |\n\nThis reflects that the geometry is very accurate in comparison to the ground truth, on average being smaller than half of a voxel for more than 90% of the data samples.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate the insightful comments provided by the reviewers who all agree that SCube is technically sound and easy to understand, the results are impressive, and the problem solved is challenging. We post responses to reviewers individually in the corresponding section, while referenced figures are jointly included in the PDF file.

Meta-review: This paper was reviewed by four experts in the field and received mixed reviews (1) Borderline Reject, (1) Weak Accept, (1) Accept, (1) Borderline Accept. Reviewers appreciate the idea of combining xcube diffusion model and gaussian splatting for sparse view-based large scale scene reconstruction. While one reviewer is concerned about the novelty, other reviewers find the idea is effective especially in reconstructing large scale scene from sparse views. The paper is well-written and justified with good results.  Reviewers have raised concerns regarding the details of  sky panorama model, evaluation of the results for both stages, and whether it is necessary to use diffusion for the task. Authors have provided responses to address reviewers’ concerns.  AC agrees with reviewers on the contribution of the idea which is effective and efficient in the reconstruction especially in handling the sparse view based reconstruction. AC thus recommends Accept.
Decision: Accept (poster)