Title: VCR-GauS: View Consistent Depth-Normal Regularizer for Gaussian Surface Reconstruction
Venue: NeurIPS 2024
Accepted: yes
Keywords: Gaussian Splatting

3D Reconstruction

Surface Reconstruction
Abstract: Although 3D Gaussian Splatting has been widely studied because of its realistic and efficient novel-view synthesis, it is still challenging to extract a high-quality surface from the point-based representation. Previous works improve the surface by incorporating geometric priors from the off-the-shelf normal estimator. However, there are two main limitations: 1) Supervising normal rendered from 3D Gaussians updates only the rotation parameter while neglecting other geometric parameters; 2) The inconsistency of predicted normal maps across multiple views may lead to severe reconstruction artifacts. In this paper, we propose a Depth-Normal regularizer that directly couples normal with other geometric parameters, leading to full updates of the geometric parameters from normal regularization. We further propose a confidence term to mitigate inconsistencies of normal predictions across multiple views. Moreover, we also introduce a densification and splitting strategy to regularize the size and distribution of 3D Gaussians for more accurate surface modeling. Compared with Gaussian-based baselines, experiments show that our approach obtains better reconstruction quality and maintains competitive appearance quality at faster training speed and 100+ FPS rendering. Our code will be made open-source upon paper acceptance.

VCR-GauS: View Consistent Depth-Normal Regularizer for Gaussian Surface Reconstruction

Abstract

Although 3D Gaussian Splatting has been widely studied because of its realistic and efficient novel-view synthesis, it is still challenging to extract a high-quality surface from the point-based representation. Previous works improve the surface by incorporating geometric priors from the off-the-shelf normal estimator. However, there are two main limitations: 1) Supervising normals rendered from 3D Gaussians effectively updates the rotation parameter but is less effective for other geometric parameters; 2) The inconsistency of predicted normal maps across multiple views may lead to severe reconstruction artifacts. In this paper, we propose a DepthNormal regularizer that directly couples normal with other geometric parameters, leading to full updates of the geometric parameters from normal regularization. We further propose a confidence term to mitigate inconsistencies of normal predictions across multiple views. Moreover, we also introduce a densification and splitting strategy to regularize the size and distribution of 3D Gaussians for more accurate surface modeling. Compared with Gaussian-based baselines, experiments show that our approach obtains better reconstruction quality and maintains competitive appearance quality at faster training speed and 100+ FPS rendering.

1 Introduction

Normal

GT Surface

Ray

P

1

P

2

(a)

L

n

GT Surface

Ray

P

1

P

2

(b)

L

n

(c)

L

dn

GT Surface

P

1

P

2

𝐍 "

d

𝐍

Multi-view stereo (MVS) is a long-standing problem that aims to create 3D surfaces of an object or scene captured from multiple viewpoints [9; 5; 25; 41]. This technique has applications in robotics, graphics, virtual reality, etc . Recently, rendering methods [49; 59; 26; 16] have enhanced the quality of reconstructions. These approaches which are often based on implicit neural representations require extensive training time. For instance, Neuralangelo [26] uses hash encoding [33] for creating highfidelity surfaces but requires 128 GPU hours for a single scene. On the other hand, the novel 3D Gaussian Spatting method [22] employs 3D Gaussians to render complex scenes photorealistically in real-time, offering a more efficient alternative. Consequently, many recent works attempted the utilization of Gaussian Splatting for surface reconstruction [8; 15; 45; 19]. Although they achieve success in object-level reconstruction, it is still challenging to extract a high-quality surface for large scenes. Previous works [59] improve the surface for scene-level reconstruction by incorporating geometric priors from the off-the-shelf normal estimator. However, there are two main limitations for Gaussian-based reconstruction: 1) Supervising normals rendered from 3D Gaussians effectively updates the rotation parameter but is less effective for other geometric parameters; 2) The predicted normal maps are inconsistent across multiple views, which may lead to severe reconstruction artifacts.

In this paper, we introduce a novel view-consistent Depth-Normal (D-Normal) regularizer to alleviate the above-mentioned limitations. As illustrated in Fig. 2, we notice that the supervision of the Gaussian normals can effectively update its rotations but is less effective for affecting its positions. Consequently, the supervision of Gaussian normals is not as effective as NeuS-based methods [49; 59; 26] whose normal is the gradient of the signed distance function (SDF) that is directly related to the position in 3D space. To solve this issue, we are inspired by the depth and normal estimation [1; 56] to introduce a D-Normal formulation, where the normal is derived from the gradient of rendered depth instead of directly blended from 3D Gaussians. Unlike existing works that obtain depth from the center position of 3D Gaussians, we compute the depth as the intersection of the ray and the compressed Gaussians. Specifically, we first make the Gaussians suitable for 3D reconstruction by applying a scale regularization similar to NeuSG [8] to compress the 3D Gaussian ellipsoids into a plane. Subsequently, the computation of the depth can be simplified to the intersection between a ray and a plane. As a result, our novel parametrization of the depth allows effective full supervision of the Gaussian geometric parameters by any data-driven monocular normal estimator.

To mitigate the inconsistent normal predictions across views, we further propose an uncertaintyaware normal regularizer as shown in Fig. 1. Particularly, we introduce a confidence term for each normal prediction. A high confidence means low uncertainty leading to enhancement of

the normal regularization, and vice-versa. Typically, the predicted normal maps from different views are combined to assess the uncertainty of a specific view. However, it is challenging to find correspondence across different views. We circumvent this issue by using the rendered normal learned from multi-view normal priors since we notice that it represents an average of normal priors across views. Furthermore, the confidence term is computed as the cosine distance between the rendered and predicted normals. Although the normal supervision has made the normals more accurate, there is still a minor error leading to depth error arising from the remnant large Gaussians. We thus devise a new densification that splits large Gaussians into smaller ones to represent the surface better. Finally, we incorporate a new splitting strategy to alleviate the surface bumps caused by densification. Experiments show that our approach outperforms Gaussian-based baselines in terms of both reconstruction quality and rendering speed.

Our main contributions are summarized below:

We formulate a novel multi-view D-Normal regularizer that enables full optimization of the Gaussian geometric parameters to achieve better surface reconstruction.

We further design a confidence term to weigh our D-Normal regularizer to mitigate inconsistencies of normal predictions across multiple views.

We introduce a new densification and splitting strategy to alleviate depth error towards more accurate surface modeling.

Our method outperforms prior work in terms of reconstruction accuracy and running efficiency on the benchmarking Tank and Temples, Replica, MipNeRF360, and DTU datasets.

2 Related Work

Novel View Synthesis. The pursuit of novel view synthesis began with Soft3D [34], which integrated deep learning and volumetric ray-marching to form a continuous, differentiable density field for geometry representation [18; 42]. While effective, this approach was computationally expensive. Neural Radiance Fields (NeRF) [32] improved render quality with importance sampling and positional encoding, but the deep neural networks slowed down processing. Subsequent methods aimed to optimize both quality and speed. Techniques like position encoding and band-limited coordinate networks are combined with neural radiance fields for pre-filtered scene representation [2; 3; 28]. Innovations to speed up rendering included leveraging spatial data structures and adjusting MLP size [6; 11; 14; 17; 35; 44]. Notable examples are InstantNGP [33], which uses a hash grid and a reduced MLP for faster computation, and Plenoxels [11], which employs a sparse voxel grid to eliminate neural networks entirely. Both use Spherical Harmonics to enhance rendering. Despite these advancements, challenges remain in representing empty space and maintaining image quality with structured grids and extensive sampling. Recently, 3D Gaussian Splatting (3DGS) [22] has addressed these issues with unstructured and GPU-optimized splatting, achieving faster and higher-quality rendering without neural components. In this work, we utilize the advantage of Gaussian Splatting to perform surface reconstruction and incorporate normal priors to guide the reconstruction, especially for large indoor and outdoor scenes.

Multi-View Surface Reconstruction. Surface reconstruction is key in 3D vision. Traditional MVS methods [4; 9; 5; 25; 38; 41; 40] use feature matching for depth [4; 38] or voxel-based shapes [9; 5; 25; 41; 46]. Depth-based methods combine depth maps into point clouds, while volumetric methods estimate occupancy and color in voxel grids [9; 5; 29]. However, the finite resolution of voxel grids limits precision. Learning-based MVS modifies traditional steps such as feature matching [31; 48; 60], depth integration [37], or depth inference from images [20; 51; 61; 52; 58]. Further advancements [49; 53] integrated implicit surfaces with volume rendering, achieving detailed surface reconstructions from RGB images. These methods have been extended to largescale reconstructions via additional regularization [59; 26]. Despite these impressive developments, efficient large-scale scene reconstruction remains a challenge. For example, Neuralangelo [26] requires 128 GPU hours for reconstructing a single scene from the Tanks and Temples Dataset [24]. To accelerate the reconstruction process, some works [15; 19] introduce the 3D Gaussian splitting technique. However, these works still fail in large-scale reconstructions. In this work, we focus on introducing normal regularization for large-scale reconstructions.

Regularization

Rendered Normal

D-Normal

∇

Pseudo Normal

Confidence

3D Gaussians

Intersection Depth

Selected Gaussian (1st intersection)

𝑦

𝐩 = (𝑝

!

, 𝑝

"

, 𝑝

#

)

𝐧

𝑑(𝐧, 𝐩)

𝑝

#

𝑧

𝑥

Traditional

𝑝

!

𝐝

∇

𝐑

𝑓(𝐩)

∇𝑓(𝐩)

𝐩

SDF

NeuS

Normal

vs.

D-Normal

∇

Ours

𝐩

n

𝑑(𝐧, 𝐩)

Rendered Depth

Densification and Splitting

Unselected Gaussian

Densify

Render

ℒ

!"

ℒ

"

𝐸𝑞. (12)

Rendered Normal

D-Normal

3D Gaussian Splatting. Since 3DGS [22] was introduced, it has been rapidly extended to surface reconstruction. We highlight the distinctions between our method and concurrent works SuGaR [15], 2DGS [19], NeuSG [8], and DN-Splatter [47]. In contrast to SuGaR and 2DGS with unsatisfactory performance on large-scale scenes, our method focuses on introducing normal regularization to improve large-scale reconstructions. 2DGS obtaining 2D Gaussian primitives by setting the last entry of scaling factors to zero which is hard to optimize by original Gaussian Splatting technique as noted in [65; 19], while our method utilizes scale regularization to flatten 3D Gaussians which are easier to optimize. NeuSG utilizes both 3D Gaussian splitting and neural implicit rendering jointly and extracts the surface from an SDF network, while our approach is faster and conceptually simpler by leveraging only Gaussian splatting for surface approximation. Although normal prior is also used for indoor scenes, DN-Splatter may show severe reconstruction artifacts due to their normal supervision can only update the rotation parameters and normal maps inconsistencies across multiple views. Moreover, we do not use the ground truth depth for supervision utilized by DN-Splatter. In comparison, our work is designed to solve both limitations.

3 Our Method

Our proposed view-consistent D-Normal regularizer efficiently reconstructs complete and detailed surfaces of scenes from multi-view images. Sec. 3.1 provides an overview of 3D Gaussian Splatting [22]. Our normal and depth formulation of 3D Gaussians is detailed in Sec. 3.2. Sec. 3.3 introduces our proposed regularizations. The densification and splitting of the Gaussian is described in Sec. 3.4. Fig. 3 depicts our whole framework.

3.1 Preliminaries: 3D Gaussian Splatting

3D Gaussian Splatting [22] is an explicit 3D scene representation with 3D Gaussians. Each Gaussian is defined by a covariance matrix Σ and a center point p ∈ R 3 which is the mean of the Gaussian. The 3D Gaussian distribution can be represented as:

To maintain positive semi-definiteness during optimization, the covariance matrix Σ is expressed as the product of a scaling matrix S and a rotation matrix R :

where S is a diagonal matrix, stored by a scaling factor s ∈ R 3 , and the rotation matrix R is represented by a quaternion r ∈ R 4 .

For novel view rendering, the splatting technique [55] is applied to the Gaussians on the camera planes. Using the viewing transform matrix W and the Jacobian of the affine approximation of the projective transformation J [64], the transformed covariance matrix Σ ′ can be determined as:

A 3D Gaussian is defined by its position p , quaternion r , scaling factor s , opacity o ∈ R , and color represented with spherical harmonics coefficients H ∈ R k . For a given pixel, the combined color and opacity from multiple Gaussians are weighted by Eq. 1. The color blending for overlapping points is:

where c i and α i = o i G ( x i ) denote the color and density of a point, respectively.

3.2 Geometric Properties

To reconstruct the 3D surface, we introduce two geometric properties: normal and depth of a Gaussian, which are used to render the corresponding normal map and depth map for regularization.

Normal Vector. Following NeuSG [8], the normal of the Gaussian can be represented as the direction of the minimized scaling factor. The normal in the world coordinate system is defined as:

The normal n and position p are transformed into the camera coordination system with the camera extrinsic matrix, which we subsequently take as the default unless otherwise stated.

Intersection Depth. The existing work [45] obtains the depth from the center position p = ( p x , p y , p z ) of each Gaussian in the camera coordinate system. However, this formulation is inaccurate and results in the depth from each Gaussian center being unrelated to its normal n during optimization. A more reasonable depth calculation is to compute the depth of the intersection between the Gaussian and the ray emitted from the camera center. To simplify the computation of intersection and represent the surface, we incorporate a scale regularization loss L scale from NeuSG [8] to squeeze the 3D Gaussian ellipsoids into highly flat shapes. This loss constrains the minimum component of the scaling factor s = ( s 1 , s 2 , s 3 ) ⊤ ∈ R 3 for each Gaussian towards zero:

This process effectively flattens the 3D Gaussian towards a planar shape which we represent by ( p , n ) . As a result, any point o p on the plane follows the incidence equation given by: n · ( o p -p ) = 0 . We further denote any point o l on a ray that passes through the origin in 3D space as o l = r t , where t ∈ R is the distance from the point to the origin along the ray. We set o l = o p at the intersection of the ray with the plane, which we can then solve for the depth of the intersection along the z -axis as:

where r z is the z-value of the ray direction r . From the equation, we can see the intersection depth is related to both the position p and the normal n of the Gaussian. This not only offers more accurate depth calculation but also enables the D-Normal regularization to backpropagate its loss to all different Gaussian parameters.

3.3 View-Consistent D-Normal Regularization

We first introduce our D-Normal formulation to allow the full optimization of the Gaussian geometric parameters. We further propose a confidence term to relieve the constraint from uncertain predictions and strengthen it from certain ones to avoid the wrong guidance from the inconsistent normal priors from a pretrained monocular model across multiple views.

D-Normal Regularizer. To improve the reconstruction quality, we utilize a normal prior N predicted from a pretrained monocular deep neural network [1] to supervise the rendered normal map ˆ N with

L1 and cosine losses:

However, normal regularization alone is insufficient for surface reconstruction as compared with NeuS-based methods. There are two main reasons for this: 1) Updating the position of a Gaussian using G ( x ) only moves it closer to or farther from the intersecting ray, as shown in Fig. 2 (a) (the mathematical proof is provided in A.2 of the supplemental material); 2) NeuS-based methods calculate normals as gradients of the SDF function from the input position p and therefore normal regularization effectively influences position updates. However, since the normal is only related to the rotation of the Gaussian in 3DGS, supervising the rendered normals does not efficiently update positions as shown in Fig. 3. To solve this problem and inspired by normal and depth estimation [1; 56], we propose a new depth-normal formulation. First, we render the depth map by weighted summing the depths:

where d i is the intersection depth from Eq. 7. Subsequently, we convert the rendered depth ˆ D to a D-Normal ¯ N d and use the predicted normal N by a pretrained model to supervise the depth via the D-Normal ¯ N d . In particular, the D-Normal is computed by back-projecting the depth map into point clouds { d k ( n , p ) } with the camera intrinsic matrix. The D-Normal ¯ N d is then computed by the cross-product with the horizontal and vertical finite differences from the neighboring points:

From this equation, we can see the D-Normal is a function of both the normal n and the position p of Gaussians. This allows the regularization on the D-Normal to optimize both normal n and position p . The D-Normal regularization is formulated as:

Confidence. Although the D-Normal regularizer resolves the issue with the Gaussian position in the supervision of its normal, the normal maps predicted by a pretrained model are not always accurate. This is especially problematic when inconsistencies arise across multiple views. We thus introduce a confidence term w to emphasize the regularization for high certainty areas while reducing on low certainty areas. Typically, the normals from different views are combined to assess the certainty of a specific view. However, it is challenging to find correspondence between different views. We circumvent the challenge by using the rendered normal learned from multi-view pseudo normals, which represents an average of the pseudo normals across the views. As a result, we can use the rendered normal to gauge the uncertainty of the predicted normal in the current view. Specifically, the confidence term w is computed as the cosine distance between the rendered and predicted normals, i.e. :

where γ is a hyper-parameter. Consequently, the view-consistent D-Normal regularizer is defined as:

The overall loss function combining these elements is:

with λ 1 , λ 2 and λ 3 balancing the individual components. L RGB includes L1 and D-SSIM losses.

3.4 Densification and Splitting

We observe that the original densification and splitting in Gaussian Splatting causes depth error as well as surface bumps and protrusions to appear. To address this issue, we further propose a new densification and splitting strategy as depicted in Fig. 3 (Bottom Left).

Densification. Although normal supervision has made the normals more accurate, there is still a minor error θ leading to depth error arising from the remnant large Gaussians since Gaussian size is not the consideration in the original 'large position gradient" selection criteria for Gaussians to be densified. As illustrated in Fig. 4a, a very small normal error at the edges can result in a significant depth error sin θ · r away from the center for larger Gaussians (top of figure). Comparatively, the depth error is small for smaller Gaussians since r ′ becomes relatively smaller (bottom of figure). Consequently, we subdivide the larger Gaussians into smaller Gaussians to keep the depth error small. To achieve this, we first randomly sample camera views from a cuboid that encompasses the entire scene for object-centric outdoor scenes and from the training views for indoor scenes. Since we aim to densify only the surface Gaussians, we only keep the first intersected Gaussian and discard the rest for each ray emitted from the camera. Subsequently, we densify only those with a scale above a threshold β among the collected Gaussians.

𝑠𝑖𝑛𝜃 ⋅ 𝑟

𝜃

𝑟

𝑠𝑖𝑛𝜃 ⋅ 𝑟′

𝜃

𝑟′

𝜃

𝑟′

𝜃

𝑟′

Surface

(a) Large

vs.

Small Gaussians

Original

Ours

(b) Splitting Strategy

Splitting. We notice that the Gaussians tend to protrude the ground truth surface after densification due to the clustering of many Gaussians. As also observed in Mip-Splatting [57], Gaussians splitted from the same parents tend to remain clustered with relatively stable positions due to the sampling from the same Gaussian distribution. To avoid clustering, we split the old Gaussian into two new Gaussian along the axis with the largest scale instead of using the Gaussian sampling with the position of the Gaussian as mean and the 3D scale of the Gaussian as variance. The positions of the new Gaussians evenly divide the maximum scale of the old Gaussian. Other parameters of new Gaussians are obtained following the original 3DGS. This process is shown in Fig. 4b.

4 Experiments

We first evaluate our method on 3D surface reconstruction in Sec. 4.1. We also report the rendering results in Sec. 4.1. Additionally, we validate the effectiveness of the proposed techniques in Sec. 4.2.

2DGS

Truck

Caterpillar

Ours

SuGar

NeuS

GT

Dataset. We evaluate the performance of our method on various datasets. For surface reconstruction, we evaluate on Tanks and Temples (TNT) [24]. To further validate the effectiveness of our method, we compare with other methods on Replica [43]. Although we focus on the large-scale reconstruction, we also report our results on DTU [21], which can be seen in the supplementary. Furthermore, we evaluate the rendering results on Mip-NeRF360 [3]. For all the datasets, we use COLMAP [39] to generate a sparse point cloud for each scene as initialization.

Implementation Details. Our method is built upon the open-source 3DGS code base [22] and the intersection depth calculation is implemented with custom CUDA kernels. λ 1 , λ 2 , and λ 3 are set to 1 , 0 . 01 , and 0 . 015 , respectively. The densification threshold β is set to 0 . 002 . The hyperparameter γ is set to 0 . 005 . We use pretrained DSINE [1] to predict normal maps for outdoor scenes and pretrained GeoWizard [13] for indoor scenes. We also employ a semantic surface trimming approach to avoid unwanted background elements like the sky in the reconstructions for outdoor scenes, which is introduced in the supplementary material. Similar to 3DGS, we stop densification at 15k iterations and optimize all of our model parameters for 30k iterations. For mesh extraction, we adapt truncated signed distance fusion (TSDF) to fuse the rendered depth maps using Open3D [63].

4.1 Comparison

Surface Reconstruction. As shown in Tab. 1, our method outperforms SDF models ( i.e. , NeuS [49], MonoSDF [59], and Geo-NeuS [12]) on the TNT dataset, and reconstructs significantly better surfaces than explicit reconstruction methods ( i.e. , 3DGS [22], SuGaR [15], and 2DGS [19]). Notably, our model demonstrates exceptional efficiency, offering a reconstruction speed that is approximately 20 times faster compared to NeuS-based reconstruction methods. Compared with the concurrent work 2DGS[19], although our method is a little slower than it (about

20 minutes), it works much better (0.3 vs. 0.4). As shown in Fig. 5, our approach better recovers planar surfaces ( e.g. , roof in Barn and ground in Truck ) as well as finer geometry details. In addition, 2DGS renders much slower than ours (68 FPS vs. 145 FPS). On the Replica dataset shown in Tab. 3, our method is much faster than MonoSDF (10+ hours vs. 50 minutes) although showing comparable performance. Compared with explicit reconstruction methods, including 3DGS, SuGaR, and 2DGS, our method achieves significantly higher F1-score for reconstruction. Although we focus on large-scale reconstruction, we also report the results on object-level reconstruction DTU [21] ( cf. supplementary).

Novel View Synthesis. Our method can reconstruct 3D surfaces and provide high-quality novel view synthesis. As shown in Tab. 2, we compare our novel view rendering results against baseline approaches on the Mip-NeRF360 dataset in this section. Remarkably, our method consistently achieves competitive novel view synthesis results compared to state-of-the-art techniques ( e.g. , MipNeRF360, 3DGS, etc .) while providing geometrically accurate surface reconstruction. Furthermore, our method renders a few times faster than the concurrent work 2DGS (128 FPS vs. 27 FPS). The reason is that our method utilizes the original and efficient Gaussian Splatting technique, while 2DGS applies a time-consuming ray-splat technique.

w/o D-Normal

w/ D-Normal

4.2 Ablation Studies

We verify the effectiveness of different design choices on reconstruction quality, including regularization terms, intersection depth, and densification on the TNT dataset [24] and report the F1-score. We first examine the effect of our view-consistent DNormal regularization. Our full model (Tab. 4 E) provides the best perfor-

mance (0.40 F1-score). The performance drops 0.10 F1-score from 0.4 to 0.3 without the D-Normal regularizer (Tab. 4 A) while keeping rendered normal regularization. It proves that it is insufficent to supervise only the normal maps rendered from Gaussian Splatting. The visualization in Fig. 6 demonstrates that our d-normal regularization can effectively push the 3D Gaussians towards the surface. Furthermore, the result drops by 0.04 F1-score without the confidence (Tab. 4 B) and with the D-Normal regularizer. It demonstrates that confidence can mitigate the problem of inconsistency of the predicted normal maps. From Fig. 7, we can observe that disabling the confidence leads to an unsmooth surface. Both of these validate the effectiveness of the view-consistent D-Normal regularization. Additionally, the absence of intersection depth (Tab. 4 C) results in poor performance. Lastly, the performance increases from 0.33 F1-score to 0.40 with our densification and split (Tab. 4 D), proving small Gaussians represent surfaces better than large Gaussians.

Confidence

Inconsistent Pseudo Normal

w/ Confidence

w/o Confidence

View 2

View 3

View 1

M

5 Conclusion

In this work, we have introduced a view-consistent D-Normal regularizer for efficient, high-quality, and compact surface reconstruction. We formulate the D-Normal regularizer that directly couples normal with the other geometric parameters. This allows for the full update of all geometric parameters during normal regularization. We also propose a confidence term that weighs our DNormal regularizer to mitigate inconsistencies of normal predictions across multiple views. Finally, we introduce a densification and splitting strategy to regularize the scales and distribution of 3D Gaussians for more precise surface modeling. Our evaluations on diverse datasets demonstrate that our method outperforms existing works in surface reconstruction.

Acknowledgement. This research / project is supported by the National Research Foundation (NRF) Singapore, under its NRF-Investigatorship Programme (Award ID. NRF-NRFI09-0008).

References

Gwangbin Bae and Andrew J. Davison. Rethinking inductive biases for surface normal estimation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024.

Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5855-5864, 2021.

Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mipnerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5470-5479, 2022.

Michael Bleyer, Christoph Rhemann, and Carsten Rother. Patchmatch stereo-stereo matching with slanted support windows. In Bmvc , volume 11, pages 1-11, 2011.

Adrian Broadhurst, Tom W Drummond, and Roberto Cipolla. A probabilistic framework for space carving. In Proceedings eighth IEEE international conference on computer vision. ICCV 2001 , volume 1, pages 388-393. IEEE, 2001.

Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In European Conference on Computer Vision , pages 333-350. Springer, 2022.

Hanlin Chen, Chen Li, Mengqi Guo, Zhiwen Yan, and Gim Hee Lee. Gnesf: Generalizable neural semantic fields. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Hanlin Chen, Chen Li, and Gim Hee Lee. Neusg: Neural implicit surface reconstruction with 3d gaussian splatting guidance. arXiv preprint arXiv:2312.00846 , 2023.

Jeremy S De Bonet and Paul Viola. Poxels: Probabilistic voxelized volume reconstruction. In Proceedings of International Conference on Computer Vision (ICCV) , volume 2, page 2. Citeseer, 1999.

Zhiwen Fan, Kevin Wang, Kairun Wen, Zehao Zhu, Dejia Xu, and Zhangyang Wang. Lightgaussian: Unbounded 3d gaussian compression with 15x reduction and 200+ fps, 2023.

Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5501-5510, 2022.

Qiancheng Fu, Qingshan Xu, Yew Soon Ong, and Wenbing Tao. Geo-neus: Geometry-consistent neural implicit surfaces learning for multi-view reconstruction. Advances in Neural Information Processing Systems , 35:3403-3416, 2022.

Xiao Fu, Wei Yin, Mu Hu, Kaixuan Wang, Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. Geowizard: Unleashing the diffusion priors for 3d geometry estimation from a single image. arXiv preprint arXiv:2403.12013 , 2024.

Stephan J Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien Valentin. Fastnerf: High-fidelity neural rendering at 200fps. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 14346-14355, 2021.

Antoine Guédon and Vincent Lepetit. Sugar: Surface-aligned gaussian splatting for efficient 3d mesh reconstruction and high-quality mesh rendering. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024.

Mengqi Guo, Chen Li, and Gim Hee Lee. Incremental learning for neural radiance field with uncertainty-filtered knowledge distillation. ECCV , 2024.

Peter Hedman, Pratul P Srinivasan, Ben Mildenhall, Jonathan T Barron, and Paul Debevec. Baking neural radiance fields for real-time view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5875-5884, 2021.

Philipp Henzler, Niloy J Mitra, and Tobias Ritschel. Escaping plato's cave: 3d shape from adversarial rendering. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9984-9993, 2019.

Binbin Huang, Zehao Yu, Anpei Chen, Andreas Geiger, and Shenghua Gao. 2d gaussian splatting for geometrically accurate radiance fields. SIGGRAPH , 2024.

Po-Han Huang, Kevin Matzen, Johannes Kopf, Narendra Ahuja, and Jia-Bin Huang. Deepmvs: Learning multi-view stereopsis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2821-2830, 2018.

Rasmus Jensen, Anders Dahl, George Vogiatzis, Engil Tola, and Henrik Aanæs. Large scale multi-view stereopsis evaluation. In 2014 IEEE Conference on Computer Vision and Pattern Recognition , pages 406-413. IEEE, 2014.

Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics , 42(4), July 2023.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv:2304.02643 , 2023.

Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun. Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Transactions on Graphics (ToG) , 36(4):1-13, 2017.

Kiriakos N Kutulakos and Steven M Seitz. A theory of shape by space carving. International journal of computer vision , 38:199-218, 2000.

Zhaoshuo Li, Thomas Müller, Alex Evans, Russell H Taylor, Mathias Unberath, Ming-Yu Liu, and Chen-Hsuan Lin. Neuralangelo: High-fidelity neural surface reconstruction. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Jiaqi Lin, Zhihao Li, Xiao Tang, Jianzhuang Liu, Shiyong Liu, Jiayue Liu, Yangdi Lu, Xiaofei Wu, Songcen Xu, Youliang Yan, and Wenming Yang. Vastgaussian: Vast 3d gaussians for large scene reconstruction. In CVPR , 2024.

David B Lindell, Dave Van Veen, Jeong Joon Park, and Gordon Wetzstein. Bacon: Band-limited coordinate networks for multiscale scene representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16252-16262, 2022.

Shaohui Liu, Yinda Zhang, Songyou Peng, Boxin Shi, Marc Pollefeys, and Zhaopeng Cui. Dist: Rendering deep implicit signed distance function with differentiable sphere tracing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2019-2028, 2020.

Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 , 2023.

Wenjie Luo, Alexander G Schwing, and Raquel Urtasun. Efficient deep learning for stereo matching. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5695-5703, 2016.

Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision , pages 405-421. Springer, 2020.

Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACMTransactions on Graphics (ToG) , 41(4):115, 2022.

Eric Penner and Li Zhang. Soft 3d reconstruction for view synthesis. ACM Transactions on Graphics (TOG) , 36(6):1-11, 2017.

Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 14335-14345, 2021.

Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li, Qing Jiang, and Lei Zhang. Grounded sam: Assembling open-world models for diverse visual tasks, 2024.

Gernot Riegler, Ali Osman Ulusoy, Horst Bischof, and Andreas Geiger. Octnetfusion: Learning depth fusion from data. In 2017 International Conference on 3D Vision (3DV) , pages 57-66. IEEE, 2017.

Johannes L Schönberger, Enliang Zheng, Jan-Michael Frahm, and Marc Pollefeys. Pixelwise view selection for unstructured multi-view stereo. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14 , pages 501-518. Springer, 2016.

Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-from-motion revisited. In Conference on Computer Vision and Pattern Recognition (CVPR) , 2016.

Steven M Seitz, Brian Curless, James Diebel, Daniel Scharstein, and Richard Szeliski. A comparison and evaluation of multi-view stereo reconstruction algorithms. In 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR'06) , volume 1, pages 519-528. IEEE, 2006.

Steven M Seitz and Charles R Dyer. Photorealistic scene reconstruction by voxel coloring. International Journal of Computer Vision , 35:151-173, 1999.

Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhofer. Deepvoxels: Learning persistent 3d feature embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2437-2446, 2019.

Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, et al. The replica dataset: A digital replica of indoor spaces. arXiv preprint arXiv:1906.05797 , 2019.

Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Neural geometric level of detail: Real-time rendering with implicit 3d shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11358-11367, 2021.

Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309.16653 , 2023.

Shubham Tulsiani, Tinghui Zhou, Alexei A Efros, and Jitendra Malik. Multi-view supervision for single-view reconstruction via differentiable ray consistency. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2626-2634, 2017.

Matias Turkulainen, Xuqian Ren, Iaroslav Melekhov, Otto Seiskari, Esa Rahtu, and Juho Kannala. Dn-splatter: Depth and normal priors for gaussian splatting and meshing, 2024.

Benjamin Ummenhofer, Huizhong Zhou, Jonas Uhrig, Nikolaus Mayer, Eddy Ilg, Alexey Dosovitskiy, and Thomas Brox. Demon: Depth and motion network for learning monocular stereo. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5038-5047, 2017.

Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106.10689 , 2021.

Yunsong Wang, Hanlin Chen, and Gim Hee Lee. Gov-nesf: Generalizable open-vocabulary neural semantic fields. arXiv preprint arXiv:2404.00931 , 2024.

Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. Mvsnet: Depth inference for unstructured multi-view stereo. In Proceedings of the European conference on computer vision (ECCV) , pages 767-783, 2018.

Yao Yao, Zixin Luo, Shiwei Li, Tianwei Shen, Tian Fang, and Long Quan. Recurrent mvsnet for high-resolution multi-view stereo depth inference. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 5525-5534, 2019.

Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems , 34:4805-4815, 2021.

Mingqiao Ye, Martin Danelljan, Fisher Yu, and Lei Ke. Gaussian grouping: Segment and edit anything in 3d scenes. arXiv preprint arXiv:2312.00732 , 2023.

Wang Yifan, Felice Serena, Shihao Wu, Cengiz Öztireli, and Olga Sorkine-Hornung. Differentiable surface splatting for point-based geometry processing. ACM Transactions on Graphics (TOG) , 38(6):1-14, 2019.

Wei Yin, Yifan Liu, Chunhua Shen, and Youliang Yan. Enforcing geometric constraints of virtual normal for depth prediction. In Proceedings of the IEEE/CVF international conference on computer vision , pages 5684-5693, 2019.

Zehao Yu, Anpei Chen, Binbin Huang, Torsten Sattler, and Andreas Geiger. Mip-splatting: Alias-free 3d gaussian splatting. Conference on Computer Vision and Pattern Recognition (CVPR) , 2024.

Zehao Yu and Shenghua Gao. Fast-mvsnet: Sparse-to-dense multi-view stereo with learned propagation and gauss-newton refinement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1949-1958, 2020.

Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger. Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction. Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Sergey Zagoruyko and Nikos Komodakis. Learning to compare image patches via convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4353-4361, 2015.

Jingyang Zhang, Yao Yao, Shiwei Li, Zixin Luo, and Tian Fang. Visibility-aware multi-view stereo network. British Machine Vision Conference (BMVC) , 2020.

Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew Davison. In-place scene labelling and understanding with implicit scene representation. In Proceedings of the International Conference on Computer Vision (ICCV) , 2021.

Qian-Yi Zhou, Jaesik Park, and Vladlen Koltun. Open3D: A modern library for 3D data processing. arXiv:1801.09847 , 2018.

Matthias Zwicker, Hanspeter Pfister, Jeroen Van Baar, and Markus Gross. Surface splatting. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques , pages 371-378, 2001.

Matthias Zwicker, Jussi Rasanen, Mario Botsch, Carsten Dachsbacher, and Mark Pauly. Perspective accurate splatting. In Proceedings-Graphics Interface , pages 247-254, 2004.

A Supplemental Material

A.1 Semantic Surface Trimming

w/ Semantic Trimming

w/o Semantic Trimming

To avoid unwanted background elements like the sky in reconstructions for outdoor scenes, we employ a semantic surface trimming approach by learning a semantic field [54; 7; 50; 62] that leverages a pretrained semantic model [23; 30; 36]. Specifically, we assign each Gaussian with a learnable semantic feature. Subsequently, similar to color blending, alpha blending is applied on these features to get the pixel-level semantics. We use the predicted semantic map from Grounded-SAM [36] with cross-entropy loss to train the learnable features. After the optimization, we render the semantic map for each view and then use the semantics to mask out the background. Although we can use the predicted semantic maps from Grounded-SAM to prune the background directly, the predicted semantic maps are not always accurate and consistent across views. Our proposed method can get a more accurate semantics with noisy pseudo labels to a certain extent, which is also observed in [62].

A.2 Proof on Our D-Normal Regularizer

Fig. 2 is to illustrate the optimization of positions of Gaussians under normal and d-normal supervisions. As we mentioned in Sec. 1, in contrast to supervision on rendered normal maps which only updates Gaussian rotations, our D-Normal regularizer can also effectively update the positions of the Gaussians. We show the mathematical proof below.

Proposition 1.1: Supervision on rendered normal cannot effectively affect the positions of Gaussians.

Proposition 1.2: Supervision on our D-Normal regularizer can effectively affect the positions of Gaussians.

Proof: Without a loss of generality, we omit the summation over multiple views in our following derivations for brevity. Based on the loss L n on rendered normal ( cf. Eq. 8), the gradient of L n with

respect to position p :

Putting Eq. 17 into Eq. 16, we get:

Based on the D-Normal regularization L dn (cf. Eq. 12), the gradient of L dn with respect to position p :

We can deduce the following from Eq. 16 and Eq. 19:

Case 1: From Eq. 16, we can see that the gradient-update ∂ L n ∂ p i of position is independent of the normal n . Consequently, the supervision on rendered normal cannot effectively affect the Gaussian position p .

Case 2: From Eq. 19, there is an additional term with n n · p , where the denominator n · r is a scalar term. This effectively makes the change in the position ∂ ˆ D ∂ p i to move along the direction of the normal n . Consequently, the supervision on D-Normal directly affects the Gaussian position p .

We can further deduce that the gradient-update on the Gaussian position pulls the position along the normal towards the surface, which achieves better reconstruction.

In view of the above proof, we conclude that it is better to do supervision on the D-Normal regularizer. In addition to the mathematical proof, we also visualize the positions of Gaussian centers from the optimized 3D scenes both with and without the D-Normal regularizer in Fig. 6, thereby providing experimental validation of the conclusion.

A.3 Implementation Details

We use PyTorch 2.0.1 and CUDA 11.8. for most experiments. All experiments are conducted on an NVIDIA 3090/4090/A5000/A6000 GPU. We set most hyperparameters to the same as that used in

Gaussian Splatting [22]. For outdoor scenes in the TNT dataset, we also utilize decoupled appearance modeling [27] to alleviate the exposure issue. Moreover, to remove some outlier Gaussians, we adopt a pruning technique from LightGaussian [10]. We also use a cuboid bounding box to contain the scene we need to reconstruct and we only regulate and add the new densification to Gaussians inside the box. We use the same train and test data with 2DGS on TNT, Mip-NeRF360, and DTU datasets.

A.4 Additional Ablation Studies

We further verify the effectiveness of additional design choices on reconstruction quality. We conduct experiments on the TNT dataset [24] and report the F1-score. The quantitative result is reported in Tab. 5. We first verify the proposed semantic trimming strategy (Tab. 5 F). From the table, we can see the strategy can increase the F1-score by 0.02. We can also observe the qualitative result from Fig. 8 that the trimming strategy can prune the sky correctly. Additionally, we confirm the effectiveness of scale regularization (Tab. 5 G), which leads to an improvement of 0.04. The current work [19] sets the last item of the scale factor to zero to flatten the 3D Gaussians for surface reconstruction. Additionally, we also ablate the setting zero (Tab. 5 H) and the scale regularization (Tab. 5 I). From the table, we can see that using the scale regularization instead of setting zero can obtain a 0.03 F1-score improvement.

A.5 Additional Qualitative Results

Fig. 9 shows the rendering (top) and reconstruction (down) results on the Mip-NeRF360 [2] dataset. The rendering results on the TNT and Replica datasets are shown in Fig. 11. We also compare the qualitative results of VCR-GauS with SuGar and 2DGS, shown in Fig. 10. From the figure, we can see that our method can reconstruct both complete and high-detailed surfaces. In addition to the visualization in the form of pictures, we have also recorded a video in the supplementary material that can be downloaded and watched.

A.6 Additional Dataset

In this section, we report the result of object-level reconstruction. Although we focus on large-scale reconstruction, we still outperform implicit ( i.e. , NeRF, VolSDF, NeuS, and MonoSDF) and most explicit methods ( i.e. , 3DGS and SuGaR) on DTU [21]. While our method is comparable with current work 2DGS on object-level reconstruction, our method is much better than it on large-scale reconstruction, as shown in Tab. 1 and Tab. 3. The qualitative results on DTU are shown in Fig. 12.

Render

Mesh

B Limitations

Although our method can alleviate the problem of inaccurate normal prediction from a pretrained normal estimator, especially the inconsistent normal predictions across views, it fails under the extreme case when almost all predicted normal across views are wrong. In addition, our method cannot reconstruct beyond the observed scene. Furthermore, our method cannot capture the surface of the semi-transparent object as shown in Fig. 13.

2DGS

Ours

SuGar

GT

Render

Mesh

日4啡

AT

650)851-3600

cleont&Son

LC8V78

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have discussed them on page 2.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have discussed about that in the supplementary material.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: We do not have theoretical contributions in this work, where our contributions are validated with experiments.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All the hyper-parameters and network organizations are provided in the main paper and the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: Our codes have been released.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Training and test details are described in the main paper and the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We follow existing related works for the setting of error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have provided that in the supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed that and claim we conform that Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our method focuses on reconstructing 3D surfaces using Gaussian Splatting technique, which is a component of 3D reconstruction. It does not have further societal impacts than existing 3D reconstruction works.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper does not have such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited them in the references.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We use and cite existing datasets in this work. Other assets including code/model will be released after submitting.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We do not include such experiments.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We do not include such experiments.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a method for improving geometry reconstruction for 3D Gaussian Splatting. It adopts a relatively flattened 3D Gaussian and incorporates normal regularization from monocular priors. Specifically, the paper proposes supervising the geometry by minimizing the differences between the normals derived from the depth map and the monocular normals. The method further refines the depth map by calculating the ray-Gaussian intersection and introduces a densification technique that splits large Gaussians on the surface into smaller ones, effectively reducing depth errors. To address multi-view inconsistencies arising from monocular priors, the paper proposes a weighting scheme to boost performance. Extensive results show that the method achieves competitive results on common benchmarks, including TNT, MipNeRF360, DTU, and Replica.\n\nSTRENGTHS:\n1. The paper presents a confidence map to resolve the multi-view inconsistencies arising from monocular priors estimated from pre-trained models.\n2. The paper identifies that large Gaussians cause significant depth errors and proposes a reasonable splitting scheme to address this issue.\n3. The paper is well-written, easy to follow, and the evaluation is comprehensive.\n\nWEAKNESSES:\n1. The contributions seem limited. The core component of the paper is the combination of depth normals and rendered normals for supervision with monocular priors. However, the depth normals and rendered normals appear similar to those in 2DGS, despite this paper using monocular priors for supervision between them. The intersected depth also shares a similar motivation to 2DGS and Gaussian Surfels [1]. However, the paper lacks acknowledge and reference to these works. The paper should mention these or provide sufficient discussion.\n\n2. While the introduced monocular normals improve reconstruction quality, they also cause oversmoothing, as shown in the DTU results. Additionally, it remains unclear how the monocular priors work for more general cases. The paper should discuss these as limitation.\n\n3. The idea of using monocular normal prior has been adopted in [1], the paper should have a citation or comparison.\n\n[1] High-quality Surface Reconstruction using Gaussian Surfels. In SIGGRAPH'24\n\nQUESTIONS:\n1. The paper states that using the rendered normal for supervision can only update the rotation of the Gaussians. Why? Given that the rendering equation involves all the Gaussians and they are all differentiable, if we use a multi-view normal map and train the Gaussians with the rendered normal map, is it really true that these Gaussians will not be optimized into a shape that fits the given normal maps?\n\n2. Why setting the last scale to zero poses difficulties in optimization (L109)? Many papers have verified that setting the Gaussians to be 2-dimensional does not hurt performance [1][2]. I believe more justification is needed.\n\n3. Why 2DGS is faster than the proposed method in the training but has a lower FPS. The paper states the reason is that 2DGS applies a time-consuming ray-splat technique. However, according to the 2DGS paper, the ray-splat intersection does not seem as expensive, at least not more expensive than in this paper, since they also use intersected depth. Some analysis would be beneficial.\n\n\n[1] High-quality Surface Reconstruction using Gaussian Surfels. In SIGGRAPH'24\n\n[2] 3D-HGS: 3D Half-Gaussian Splatting. arXiv:2406.02720\n\n**Minors**:\n\nL175: \"splitting\" should be \"splatting.\"\n\nL198: I am also concerned about the use of \"view consistent\". In my opinion, the monocular normal priors are still multi-view inconsistent, although the proposed weighting scheme can reduce some negative effects.\n\nLIMITATIONS:\nThe weakness has been outlined previously, and I have a few questions that need clarification. I may raise my score if these questions are addressed properly.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a confidence-aided Depth-Normal regularizer that directly couples the normal with other geometric parameters, thus enabling the optimization of all geometric parameters from monocular normal priors. The paper also introduces a densification and splitting strategy to regulate the size and distribution of the 3D Gaussian distributions for more precise surface modeling. Experimental results demonstrate that the method achieves better reconstruction quality with faster training speeds and rendering compared to 3DGS and 2DGS.\n\nSTRENGTHS:\n1. The main contribution of the paper is clear. The proposed Depth-Normal regularizer provides a simple but effective way to utilize the monocular normal priors in 3DGS-based reconstruction.\n\n2. Experimental results strongly support the proposed strategy.\n\nWEAKNESSES:\n1. The conversion between depth and normals is not a new concept and has been explored in both the field of depth estimation and normal estimation. I suggest referencing some previous works on depth-to-normal conversion, such as VNL[1]. I also acknowledge the contribution of the authors in applying this technique to 3DGS, which allows for better optimization of geometric attributes.\n\n[1] Yin W, Liu Y, Shen C, et al. Enforcing geometric constraints of virtual normal for depth prediction[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2019: 5684-5693.\n\n2. 2DGS also proposes a normal consistency loss, which connects the depth and normal. It would be better to clarify the difference and conduct the ablation study comparing the two conversion strategies. \n\n3. Building upon question 2, I'm curious to know if the gain in this paper is primarily due to the introduction of monocular normal priors. For instance, if we maintain the strategy of normal computation in 2DGS and incorporate the monocular normal prior in this paper. Since the major contribution of this paper lies in establishing the correlation between depth and normals, it would be better to provide an explanation or comparison to illustrate the superiority of the used conversion.\n\n4. As shown in Fig.7 and Fig.10, the details of the reconstruction results could still benefit from further improvement. This is also understandable since the reconstruction is based on the monocular normal prior, whose details are limited.\n\nQUESTIONS:\n1. I am curious why training time is longer than 2DGS but inference time is shorter.\n\n2. The reconstruction results of 2DGS in TNT as shown in Fig.4 seems different from 2DGS's original paper in Fig.10, such as the ground of 'truck scene' is not incomplete in 2DGS's paper, the head of 'Caterpillar' does not contain the noisy mesh.\n\nLIMITATIONS:\nThe authors kindly point out their limitations in the paper. I am curious about the semi-transparent object. Does the monocular normal estimation fails in semi-transparent object?", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to reconstruct surface from 3D Gaussians with a view-consistent depth-normal regularization. By introducing normal prior (DSINE/GeoWizard) to regularize the distribution of 3DGS, this approach is able to render smooth and view-consistent depth, facilitating reconstruction. This paper also tries to mitigate the inconsistencies from single-view normal prediction by introducing a confidence map. However, the effect seems to be limited.\n\nSTRENGTHS:\n1) The performance is good. The proposed design can assist 3DGS for better depth rendering.\n2) Using the gradient of depth to supervise normal makes sense, which makes the position instead of rotation of each GS effectively optimized and leads to better depth results for reconstruction.\n2) The paper is easy to understand and follow.\n\nWEAKNESSES:\n1) Actually, using prior from monocular normal estimation has been introduced in several works, e.g. DN-Splatter; Would it help if the d-normal (calculated from simple GS depth / ray-GS intersection depth) is applied? \n2) The calculation of confidence is basically the same as D-Norm regularization, according to equation 12 and 13. Typically, in Bayesian learning, we introduce an uncertainty term. It is predicted separately and added in log form as a regularization term to the final loss function. I'm not sure if this approach has been tried before, as the current confidence constraints seem to have limited effectiveness.\n3) Some related work should be compared and cited: \n- Neuralangelo: High-Fidelity Neural Surface Reconstruction\n- Relightable 3D Gaussian: Real-time Point Cloud Relighting with BRDF Decomposition and Ray Tracing\n- 3DGSR: Implicit Surface Reconstruction with 3D Gaussian Splatting\n\nQUESTIONS:\n1) I'm not sure why manual normalization is needed in the equation (9), because according to equation (4), the denominator should already be a normalized coefficient.\n\nLIMITATIONS:\nSimilar to many other reconstruction methods, this approach struggles to handle the reconstruction of specular or semi-transparent objects well.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

comments: SUMMARY:
The paper introduces a novel view-consistent Depth-Normal (D-Normal) regularizer and an uncertainty-aware normal regularizer followed by a new densification and splitting strategy to address the limitations of existing Gaussian Splatting methods in surface reconstruction tasks, such as the supervision of rendered normal updates affecting only the rotation parameter and the inconsistency of predicted normal maps. This approach enables the full optimization of Gaussian geometric parameters and decreases inaccuracy, thereby improving the quality of 3D surface reconstruction.

STRENGTHS:
The proposed method introduces novel insights, such as supervising normals rendered from 3D Gaussians, which only updates the rotation parameter. This is quite interesting. The overall model is supported by extensive experiments across multiple datasets, demonstrating significant improvements in both reconstruction quality and rendering speed compared to previous methods. The simplicity and effectiveness of this method are particularly commendable. Additionally, the paper is well-written and presents its concepts in a clear and accessible manner.

WEAKNESSES:
There are not many significant weaknesses in the method. However, a more in-depth analysis of scenarios where the method may fail, particularly with highly inconsistent normal predictions, would provide a clearer understanding of the method's boundaries.

QUESTIONS:
Please see the weakness section.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13