VCR-GauS: View Consistent Depth-Normal Regularizer for Gaussian Surface Reconstruction
Authors: Hanlin Chen
Fangyin Wei
Chen Li
Tianxin Huang
Yunsong Wang
Gim Hee Lee
Venue: NeurIPS 2024 poster (2024)
Keywords: Gaussian Splatting
3D Reconstruction
Surface Reconstruction
Primary area: machine_vision

Abstract: Although 3D Gaussian Splatting has been widely studied because of its realistic and efficient novel-view synthesis, it is still challenging to extract a high-quality surface from the point-based representation. Previous works improve the surface by incorporating geometric priors from the off-the-shelf normal estimator. However, there are two main limitations: 1) Supervising normal rendered from 3D Gaussians updates only the rotation parameter while neglecting other geometric parameters; 2) The inconsistency of predicted normal maps across multiple views may lead to severe reconstruction artifacts. In this paper, we propose a Depth-Normal regularizer that directly couples normal with other geometric parameters, leading to full updates of the geometric parameters from normal regularization. We further propose a confidence term to mitigate inconsistencies of normal predictions across multiple views. Moreover, we also introduce a densification and splitting strategy to regularize the size and distribution of 3D Gaussians for more accurate surface modeling. Compared with Gaussian-based baselines, experiments show that our approach obtains better reconstruction quality and maintains competitive appearance quality at faster training speed and 100+ FPS rendering. Our code will be made open-source upon paper acceptance.

VCR-GauS: View Consistent Depth-Normal Regularizer for Gaussian Surface Reconstruction
Hanlin Chen'
Fangyin Wei
Yunsong Wang' School of Computing, National University of Singapore
Chen Li Tianxin Huang'
Gim Hee Lee
2 Princeton University
hanlin. chendu. edu gimhee leednus. edu. ag
https://blinchen wwwnchhnngggthue. github..h..... www.:gggttbhhhpoee. htt.:iiooojjcss///ee... VVCss
High Confidence
Fiew
Low Confidence
Fiev
Reconstructed Scene Geomerry
Flew 3
Inconsistent Pseudo Normal Predicted Confidence
Reconstructed Scene Texture
Figure View-Consistent D-Normal Regularizer. Pseudo normals predicted from pretrained monocular normal estimators tend to be inconsistent across different views (left). Our method calculates confidence map indicating the confidence of the pseudo normals (middle). The confidence is used to weigh the loss imposed on our proposed D-Normals. Our method achieves new state-of- the-art surface reconstruction results and rendering quality comparable with prior work.
Abstract
Although 3D Gaussian Splatting has been widely studied because of its realistic and efficient novel-view synthesis, it is still challenging extract high-quality surface from the point-based representation. Previous works improve the surface by incorporating geometrie priors from the off-the-shelf normal estimator. However, there are two main limitations: 1) Supervising normals rendered from 3D Gaussians effectively updates the rotation parameter but less effective for other geometric parameters; 2) The inconsistency of predicted normal maps across multiple views may lead severe reconstruction artifacts. In this paper, we propose Depth- Normal regularizer that directly couples normal with other geometric parameters, leading to full updates of the geometric parameters from normal regularization. We further propose confidence term to mitigate inconsistencies of normal predictions across multiple views. Moreover, we also introduce densification and splitting strategy to regularize the size and distribution of 3D Gaussians for more accurate surface modeling. Compared with Gaussian-based basclines, experiments show that our approach obtains better reconstruction quality and maintains competitive appearance quality at faster training speed and 100+ FPS rendering.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
Moving Direction
Gaussian
Normal
Mean Depth Point
GT Surface
GT Surface
GT Surface
2
Pz
N
P
N
Ray
Ray
P
Pi
P
(a) Lm
(b) L
(c) Lau
Figure 2: Illustration of rendered normal supervision and the D-Normal regularizer. (a) As a result of the back- propaaappagttion through alpha-blending via Eq. [ rendered normal supervision fn moves Gaussians closer to (P,) or away from (P2) the intersecting ray. When the normal of Gaussian is closer to the GT surface normal, the supervision pushes this Gaussian (P 1) towards the ray to increase its weight in the rendering equation, and vice-versa (P2). (b) Such movement of Gaussians stops when the rendered normal loss &, is equal to zero. either case ((a) or (b)), the rendered normal loss cannot move Gaussian towards the surface. In contrast, (c) the D-Normal regularizer Lan can move Gaussians towards or away from GT surface. P1 and P2 are the 3D positions corresponding to the mean depth of two neighboring pixels (rays) via Eq. 10 The D-Normal N₂ is derived from P1 and P2 in Eq. 11 Lan encourages Na to align with the ground truth normal N, resulting in Gaussians moving towards away from the surface.
Multi-view stereo (MVS) is a long-standing problem that aims to create 3D surfaces of an object or scene captured from multiple viewpoints 9 5 25 41 This technique has applications in robotics,
59 26 16 have enhanced the quality of reconstructions. These approaches which are often based on implicit neural representations require extensive training time. For instance, Neuralangelo 266 uses hash encoding [33] for creating high- fidelity surfaces but requires 128 GPU hours for a single scene. On the other hand, the novel 3D Gaussian Spatting method 22 employs 3D Gaussians render complex scenes photorealistically in real-time, offering more efficient alternative. Consequently, many recent works attempted the utilization of Gaussian Splatting for surface reconstruction 8
graphics, virtual reality, elc. Recently, rendering methods 49
45 19] Although they achieve success in object-level reconstruction, it is still challenging to extract a high-quality surface for large scenes. Previous works (59) improve the surface for scene-level reconstruction by incorporating geometric priors from the off-the-shelf normal estimator. However, there are two main limitations for Gaussian-based reconstruction: 1) Supervising normals rendered from 3D Gaussians effectively updates the rotation parameter but is less effective for other geometric parameters: 2) The predicted normal maps are inconsistent across multiple views, which may lead to severe reconstruction artifacts.
15|
In this paper, we introduce novel view-consistent Depth-Normal (D-Normal) regularizer to alleviate the above-mentionel limitations. As illustrated Fig. we notice that the supervision of the Gaussian normals can effectively update its rotations but is less effective for affecting its positions. Consequently, the supervision of Gaussian normals not as effective as NeuS-based methods [49 (59
26 whose normal is the gradient of the signed distance function (SDF) that is directly related to the position in 3D space. To solve this issue, we are inspired by the depth and normal estimation [ 56 to introduce a D-Normal formulation, where the normal is derived from the gradient of rendered depth instead of directly blended from 3D Gaussians. Unlike cxisting works that obtain depth from the center position of 3D Gaussians, compressed Gaussians. Specifically, we first make the Gaussians suitable for 3D reconstruction by applying scale regularization similar to NeuSG [8] to compress the 3D Gaussian ellipsoids into plane. Subsequently. the computation of the depth can be simplified to the intersection between a ray and plane. As result, our novel parametrization of the depth allows effective full supervision of the Gaussian geometrie parameters by any data-driven monocular normal estimator.
compute the depth the intersection of the ray and the
To mitigate the inconsistent normal predictions across views, we further propose uncertainty- aware normal regularizer as shown in Fig. []] Particularly, we introduce a confidence term for each normal prediction. A high confidence means low uncertainty leading to enhancement of

---

the normal regularization, and vice-versa. Typically, the predicted normal maps from different views are combined assess the uncertainty specific view. However, it is challenging to find correspondence across different views. We circumvent this issue by using the rendered normal learned from multi-view normal priors since we notice that represents an average of normal priors across views. Furthermore, the confidence term is computed as the cosine distance between the rendered and predicted normals. Although the normal supervision has made the normals more accurate, there is still minor error leading to depth error arising from the remnant large Gaussians. We thus devise a new densification that splits large Gaussians into smaller ones to represent the surface better. Finally, we incorporate new splitting strategy to alleviate the surface bumps caused by densification. Experiments show that our approach outperforms Gaussian-basee baselines in terms of both reconstruction quality and rendering speed.
Our main contributions are summarized below:
We formulate a novel vuet D-Normal regularizer that enables full optimization of the Gaussian geometric parameters achieve better surface reconstruction.
We further design confidence term weigh our D-Normal regularizer to mitigate inconsis- tencies of normal predictions across multiple views.
We introduce a new densification and splitting strategy to alleviate depth error towards more accurate surface modeling.
Our method outperforms prior work in terms of reconstruction accuracy and running effi ciency on the benchmarking Tank and Temples, Replica, MipNeRF360, and DTU datasets.
2
Related Work
which integrated deep learning and volumetric ray-marching to form continuous, differentiable density field for geometry representation [8 42). While effective, this approach was computationally expensive.
Novel View Synthesis. The pursuit of novel view synthesis began with Soft3D 34.
Neural Radiance Fields (NeRF) (32 improved render quality with importance sampling and positional encoding. but the deep neural networks slowed down processing. Subsequent methods aimed to optimize both quality and speed. Techniques like position encoding and band-limited coordinate networks are combined with neural radiance fields for pre-filtered scene representation [2] 3. 28). Innovations to speed up rendering included leveraging spatial data structures and adjusting MLP size (6 I 14 35 44]. Notable examples are InstantNGP 333 which uses a hash grid and reduced MLP for faster computation, and Plenoxels (L]) which employs a sparse voxel grid to eliminate neural networks entirely. Both use Spherical Harmonics to enhance rendering. Despite these advancements, challenges remain in representing empty space and maintaining image quality with structured grids and extensive sampling. Recently, 3D Gaussian Splatting (3DGS) [22) has addressed these issues with unstructured and GPU-optimized splatting, achieving faster and higher-quality rendering without neural components. In this work, we utilize the advantage of Gaussian Splatting to perform surface reconstruction and incorporate normal priors to guide the reconstruction, especially for large indoor and outdoor scenes.
Multi-View Surface Reconstruction. Surface reconstruction is key in 3D vision. Traditional 38
MVS methods 4
9
5
25
use feature matching for depth [4] 38 or voxel-based Depth-based methods combine depth maps into point clouds, while
41
40
shapes 5
46
volumetric methods estimate occupancy and color in voxel grids 9 5 29). However, the finite resolution of voxel grids limits precision. Learning- based MVS modifies traditional steps such as feature matching [31 [48) 60), depth integration [37]. or depth inference from images [20] (1] 61 (5)
58]. Further advancements 49) 53) integrated implicit surfaces with volume rendering, achieving detailed surface reconstructions from RGB images. These methods have been extended to large- scale reconstructions via additional regularization [59] Despite these impressive developments, efficient large-scale scene reconstruction remains a challenge. For example, Neuralangelo [26]
requires 128 GPU hours for reconstructing a single scene from the Tanks and Temples Dataset [24] To accelerate the reconstruction process, some works [15 19 introduce the 3D Gaussian splitting
technique. However, these works still fail large-scale reconstructions. In this work, we focus on introducing normal regularization for large-scale reconstructions.
3

---

Gaussians
Regularization
Normal vs. D-Normal
P
R
(r(p)
Render
SDF
d(n,
[((ee
I
NeuS
Gauzsian Traditonal
Ours
Densify
Rendered Deptk
Psendo Normal
Rondered Normal
D-Normol
Densification and Spliting
dn
V
Intersection Depth
PycPa)
* Selected Gaussian (ist intersection)
Uluselected Gausian SSplitting and Densified Gaussian
d(n.p)
D-Normal
Confidance
Figure 3: Overview of our VCR-GauS. During densification and splitting, our method only keeps the Gaussians at the first intersections and splits large Gaussians into smaller ones along the major principle axis. The rendered normals supervised with pseudo normals predicted from a pretrained monocular normal estimator in Ln- We further calculate an uncertainty map based on the discrepancies between the rendered and pseudo normals (ef. Eq. 13) weigh the loss Lan between pseudo normals and D-Normals derived from the rendered depth maps. We compare different approaches for normal calculation (Top Right) and show our intersection depth (Bottom Right).
3D Gaussian Splatting, Since 3DGS 22 was introduced, it has been rapidly extended to surface reconstruction. We highlight the distinctions between our method and concurrent works SuGaR (15), 2DGS [19]. NeuSG [8) and DN-Splatter [4).. In contrast to SuGaR and 2DGS with unsatisfactory performance on large-scale scenes, our method focuses on introducing normal regularization to improve large-scale reconstructions. 2DGS obtaining 2D Gaussian primitives by setting the last entry of scaling factors to zero which is hard optimize by original Gaussian Splatting technique as noted in [65] [9) while our method utilizes scale regularization to flatten 3D) Gaussians which are easier to optimize. NeuSG utilizes both 3D Gaussian splitting and neural implicit rendering jointly and extracts the surface from SDF network, while our approach is faster and conceptually simpler by leveraging only Gaussian splatting for surface approximation. Although normal prior is also used for indoor scenes, DN-Splatter may show severe reconstruction artifacts due to their normal supervision can only update the rotation parameters and normal maps inconsistencies across multiple views. Moreover, do not use the ground truth depth for supervision utilized by DN- In comparison, our work is designed to solve both limitations.
Our Method
Our proposed view-consistent D-Normal regularizer efficiently reconstructs complete and detailed surfaces scenes from multi-view images. Sec. 3.1 provides an overview of 3D Gaussian Splat- ting (22 Our normal and depth formulation of 3D Gaussians is detailed in Sec. [3.2] Sec. 3.3
introduces our proposed regularizations. The densification and splitting of the Gaussian is described Sec. 3.4 Fig. depicts our whole framework.
3.1
Preliminaries: 3D Gaussian Splatting
3D Gaussian Splatting 22 is an explicit 3D scene representation with 3D Gaussians. Each Gaussian is defined by covariance matrix Σ and center point R which is the mean of the Gaussian. The 3D Gaussian distribution can be represented as:
G(x) = exp (-, p) 1x (x-p)).
To maintain positive semi-definiteness during optimization, the covariance matrix Σ expressed as the product of scaling matrix S and rotation matrix R:
Σ RSS R
(2)

---

where is a diagonal matrix, stored by scaling factor s E R., and the rotation matrix R. is represented by a quaternion R.
For novel view rendering. the splatting technique (55 is applied to the Gaussians on the camera planes. Using the viewing transform matrix W and the Jacobian of the affine approximation of the projective transformation J [64], the transformed covariance matrix y can be determined as:
E' JWEW JJ..
(3)
Gaussian is defined by its position p. quaternion r, scaling factor s, opacity o IR, and color represented with spherical harmonies coefficients H E IR* For given pixel, the combined color and opacity from multiple Gaussians are weighted by Eq. The color blending for overlapping points is:
i- I (1-a)),
Č
Σ c iEM
j=1
where and (x,) denote the color and density of a point, respectively.
3.2 Geometric Properties
To reconstruct the 3D surface, we introduce two geometric properties: normal and depth a Gaussian, which are used to render the corresponding normal map and depth map for regularization.
Normal Vector. Following NeuSG (8) the normal of the Gaussian can be represented the direction of the minimized scaling factor. The normal in the world coordinate system defined as:
R(k: R argmin ([s), 82, 8a)),
The normal n and position are transformed into the camera coordination system with the camera extrinsic matrix, which we subsequently take as the default unless otherwise stated.
Intersection Depth. The existing work [45] obtains the depth from the center position p Py. of each Gaussian in the camera coordinate system. However, this formulation is in- accurate and results in the depth from each Gaussian center being unrelated to its normal n during optimization. more reasonable depth calculation is to compute the depth of the intersection between the Gaussian and the emitted from the camera center. To simplify the computation of intersection squeeze
and represent the surface, we incorporate scale regularization loss Licale from NeuSG the 3D Gaussian ellipsoids into highly flat shapes. This loss constrains the minimum component of the scaling factor s (81, 82, $3) R for each Gaussian towards zero:
C. min( 82, sy) 1|1-
(6)
This process effectively flattens the 3D Gaussian towards planar shape which we represent by (p. n).
As result, any point Op on the plane follows the incidence equation given by: n (op. - p) 0. We further denote any point on ray that passes through the origin in 3D space as o, = rt, where t IR is the distance from the point to the origin along the ray. We set o; = O, at the intersection of the ray with the plane, which we can then solve for the depth of the intersection along the 2-axis as:
d(n,p) p) r. * (n p)/(n.r)
where r, is the z-luuue of the ray direction From the equation, related to both the position p and the normal n of the Gaussian. This not only offers more accurate depth calculation but also enables the D-Normal regularization backpropagate its loss to all different Gaussian parameters.
the intersection depth is
3.3
View-Consistent D-Normal Regularization
We first introduce our D-Normal formulation to allow the full optimization of the Gaussian geometric parameters. We further propose a confidence term to relieve the constraint from uncertain predictions and strengthen it from certain ones to avoid the wrong guidance from the inconsistent normal priors from a pretrained monocular model across multiple views.
D-Normal Regularizer. To improve the reconstruction quality, we utilize normal prior N predicted from pretrained monocular deep neural network [ to supervise the rendered normal map Ñ with
5

---

LI and cosine losses:
Ca = INN -N11 (1- (1-N-NN (1-Ñ-N)
(8)
1-1
where Ñ
Σ n2Os iEM
II (1-aa)) 1=1
Σ a. II - iEM
0g).
j=1
However, normal regularization alone is insufficient for surface reconstruction compared with NeuS-based methods. There are two main reasons for this: 1) Updating the position of a Gaussian using (xx) only moves it closer or farther from the intersecting ray, as shown in Fig. 2 (a) (the mathematical proof is provided in of the supplemental material); 2) NeuS-based methods calculate normals as gradients of the SDF function from the input position p and therefore normal regularization effectively influences position updates. However, since the normal is only related to the rotation of the Gaussian in 3DGS, supervising the rendered normals does not efficiently update positions as shown in Fig. 3 To solve this problem and inspired by normal and depth estimation [6]], we propose new depth-norma formulation. First, we render the depth map by weighted summing the depths:
D= E d, d.o EM
II
a5)/
Σ a; II az), iEM
(10)
where de is the intersection depth from Eq. 7 Subsequently, we convert the rendered depth to a D-Normal Na and use the predicted normal N by a pretrained model to supervise the depth via the D-Normal Na. In particular, the D-Normal is computed by back-projecting the depth map into point clouds {d& (n, p}} with the camera intrinsic matrix. The D-Normal N₂ is then computed by the cross-product with the horizontal and vertical finite differences from the neighboring points:
v,d(n. p) V,d(n.p)
(a p) V,d p) x Vxd(n, p)]
(11)
From this equation, we can see the D-Normal is function of both the normal and the position p of Gaussians. This allows regularization on the D-Normal to optimize both normal n and position p. The D-Normal regularization is formulated as:
Lu
N N + (1 -N.
(12)
Confidence, Although the D-Normal regularizer resolves the issue with the Gaussian position in the supervision of its normal, the normal maps predicted by a pretrained model are not always accurate. This is especially problematic when inconsistencies arise across multiple views. We thus introduce a confidence
to emphasize the regularization for high certainty areas while reducing on low certainty areas. Typically, the normals from different views are combined to assess the certainty of specific view. However, is challenging find correspondence between different views. We circumvent the challenge by using the rendered normal learned from multi-view pseudo normals, which represents an average of the pseudo normals across the views. As a result, we can use the rendered normal to gauge the uncertainty of the predicted normal in the current view. Specifically, the confidence term w computed as the cosine distance between the rendered and predicted normals, i.e.:
w cxp 1-1)/7,
(13)
where y is hyper-parameter. Consequently, the view-consistent D-Normal regularizer is defined as: Can = w [[ N||1 + (1 -)).
(14)
The overall loss function combining these elements is:
Listal LRGB + X1 Xzha dzhan=
(15)
with A1, A2 and Ag balancing the individual components. LRGB includes L1 and D-SSIM losses.
3.4 Densification and Splitting
We observe that the original densification and splitting in Gaussian Splatting causes depth error as well as surface bumps and protrusions to appear. To address this issue, we further propose new densification and splitting strategy as depicted in Fig. 3 (Bottom Left).

---

Densification. Although normal supervision has made the normals more accurate, there is still minor eπтоr 0 leading to depth error arising from the remnant large Gaussians since Gaussian size is not the consideration in the original "large position gradient" selection criteria for Gaussians to be densified. As illustrated Fig. 44] a very small normal error at the edges can result significant depth error sin 0 away from the center for larger Gaussians (top of figure). Comparatively, the depth error is small for smaller Gaussians since r' becomes relatively smaller (bottom of figure). Consequently, we subdivide the larger Gaussians into smaller Gaussians keep the depth error small. To achieve this, we first randomly sample camera views from cuboid that encompasses the entire scene for object-centric outdoor scenes and from the training views for indoor scenes. Since aim to densify only the surface Gaussians, we only keep the first intersected Gaussian and discard the rest for each ray emitted from the camera. Subsequently, densify only those with scale above threshold 3 among the collected Gaussians.
sine
sinO
Original Gaussian before splitting
Ours
Gaussian
Surface
Gaussian after splitting
(a) Large Small Gaussians
Splitting Strategy
Figure 4: Illustration of the rationals behind the densification and splitting strategies. (a) Comparison between large and small Gaussians of depth errors caused by a small normal error (in side view). (b) Comparison of the original and the proposed splitting strategies (in bird-eye view).
Splitting. We notice that the Gaussians tend to protrude the ground truth surface after densification due to the clustering of many Gaussians. As also observed Mip-Splatting [57], Gaussians splitted from the same parents tend to remain clustered with relatively stable positions due to the sampling from the same Gaussian distribution. To avoid clustering, we split the old Gaussian into two new Gaussian along the axis with the largest scale instead of using the Gaussian sampling with the position of the Gaussian as mean and the 3D scale the Gaussian as variance. The positions of the new Gaussians evenly divide the maximum scale of the old Gaussian. Other parameters of new Gaussians are obtained following the original 3DGS. This process shown in Fig, 4b
Experiments
We first evaluate our method on 3D surface reconstruction Sec. 4.1
We also report the rendering results in Sec. 4.1 Additionally, we validate the effectiveness of the proposed techniques in Sec. 4.2
Reconstructions Table 1: Quantitative results on the Tanks and Temples Dataset 24
evaluated with the official evaluation scripts and we report Fl Fl-score, average optimization time and FPS. Ours outperforms 3DGS-based surface reconstruction methods by a large margin and performs better than neural implicit methods by minor margin while optimizing significantly faster.
NeuS-basee MonoSDF
Gaussian-bars based
NeuS
Geo-Neus
SuGaR
3DGS
2DGS
Ours
Barn
0.29
0.49
0.33
0.14
0.13
0.36
0.62
Caterpillar
0.29
0.31
0.26
0.16
0.08
0.23
0.26
Courthouse
0.17
0.12
0.12
0.08
0.09
0.13
0.19
Ignatius
0.83
0.78
0.72
0.33
0.04
0.44
0.61
Meetingroom
0.24
0.23
0.20
0.15
0.01
0.16
0.19
Truck
0.45
0.42
0.45
0.26
0.19
0.26
0.52
Mean
0.38
0.39
0.35
0.19
0.09
0.30
0.40
Time
>24h
>24h
>24h
>1h
14.3 m
34.2
53 m
FPS
<10
159
68
145

---

Barn
Truck
Caterpillar
Ours
Sar
2DCS
NeuS
A
GT
Figure 5: Qualitative comparison TNT dataset. From top bottom, show the reconstructed meshes from our method, SuGar, 2DGS, and NeuS, as well as the ground truth colored point cloud. Our method reconstructs complete surfaces featuring smoother planar regions and finer details.
Dataset. We evaluate the performance of our method various datasets. For surface reconstruction, we evaluate on Tanks and Temples (TNT) (24]. To further validate the effectiveness of our method,
we compare with other methods on Replica 45 Although we focus on the large-scale reconstruction,
we also report our results on DTU (21. which can seen in the supplementary. Furthermore, we evaluate the rendering results on Mip-NeRF360 3. For all the datasets, we use COLMAP 39 to generate a sparse point cloud each scene as initialization.
Table Quantitative results on Mip-NeRF 360 [3]. Our method achieves NVS rendering quality and speed comparable with other Gaussian- based methods.
Outdoor Scene
Indoor Scene
FPS
PSNR
SSIM
LPIPS 1
PSNR
SSIM t
LIPPS 1
NeRF
21.46
0.458
0.515
26.84
0.790
0.370
Deep Blending
21.54
0.524
0.364
26.40
0.844
0.261
Instant NGP
22.90
0.566
0.371
29.15
0.880
0.216
<10
MERF
23.19
0.616
0.343
27.80
0.855
0.271
Mi6NeRF330
24.47
0.691
0.283
31.72
0.917
0.180
MoRile-NeRF
21.95
0.470
0.470
<100
BakedSDF
22.47
0.585
0.349
27.06
0.836
0.258
3DGS
24.64
0.731
0.234
30.41
0.920
0.189
134
SuGaR
22.93
0.629
0.356
29.43
0.906
0.225
2DGS
24.21
0.709
0.276
30.10
0.913
0.211
27
Ours
24.31
0.707
0.280
30.53
0.921
0.184
128

---

Implementation Details. Our method is built upon the open- source 3DGS code base [22 and the intersection depth calculation is implemented with custom CUDA kemnlls. A1, Ag, and A are set to 1, 0.01, and 0.015, respectively. The densification threshold 3 is set to 0.002. The hyperparameter y set to 0.005. We use pretrained DSINE [1] to predict normal maps for outdoor scenes and pretrained WioWizard [13] for indoor scenes. We also employ a semantic surface trimming approach to avoid unwanted background elements like the sky in the reconstructions for outdoor scenes, which is introduced in the supplementary material. Similar to 3DGS, we stop densification at 15k iterations and optimize all of our model parameters for 30k iterations. For mesh extraction, we adapt truncated signed distance fusion (TSDF) to fuse the rendered depth maps using Open3D [63]
4.1 Comparison
Surface Reconstruction. As shown in Tab. our method outperforms SDF models (i.e. NeuS [49]. MonoSDF 15 and Geo-NeuS [2]) on the TNT dataset, and reconstructs significantly better surfaces than explicit reconstruction methods (i.e. 3DGS 22]. SuGaR 15] and 2DGS [I9). Notably, our model demonstrates exceptional efficiency, offering re- construction speed that is approximately 20 times faster compared to NeuS-based reconstruction meth- ods. Compared with the concurrent work IDGS[191, although our method is a little slower than (about 20 minutes), it works much better (0.3 0.4). As shown in Fig. 5 our approach better recovers planar surfaces (e.g., roof in Barn and ground in Truck) as well as finer geometry details. addition, 2DGS renders much slower than ours (68 FPS vs. 145 FPS). On the Replica dataset shown in Tab. B] our method is much faster than MonoSDF (10+ hours vs. 50 minutes) although showing comparable performance. Compared with explicit reconstruction methods, including 3DGS, SuGaR, and 2DGS, our method achieves significantly higher F1-score for reconstruction. Although we focus large-scale reconstruction, we also report the results object-level reconstruction DTU 21 (cf. supplementary).
Table 3: Quantitative assessment Replica
43 Bold indicates the best.
Method
F1-score
Time
NeuS
65.12
Implicit
10h
MonoSDF
81.64
3DGS
50.79
SuGar
63.20
Explicit
<Th
2DGS
64.36
Ours
78.17
Novel View Synthesis. Our method can reconstruct 3D surfaces and provide high-quality novel view synthesis. As shown in Tab. 2 we compare our novel view rendering results against baseline method consistently
approaches on the Mip-NeRF360 dataset in this section. Remarkably, achieves competitive novel view synthesis results compared to state-of saae-of-thheart techniques (e.g.. Mip- NeRF360, 3DGS, etc.) while providing geometrically accurate surface reconstruction. Furthermore, our method renders few times faster than the concurrent work 2DGS (128 FPS vs. 27 FPS). The reason is that our method utilizes the original and efficient Gaussian Splatting technique, while 2DGS applies a time-consuming ray-splat technique.
w/o D-Normal
w/ D-Normal
Figure 6: Qualitative ablation for the D-Normal regularizer. We plot the positions of Gaussian centers from the optimized 3D scenes. The left disables the D-Normal regularizer with only rendered normal supervision, and the right enables both D-Normal and rendered normal supervision. Compared to w/o D-Normal that produces many noisy Gaussians floating off the surface, our proposed D-Normal regularizer effectively pushes the 3D Gaussians towards the surface and thus providing much cleaner reconstruction.

---

4.2
Ablation Studies
We verify the effectiveness of dif- ferent design choices on reconstruc- tion quality, including regularization terms, intersection depth, and densi- fication on the TNT dataset (24 and report the Fl-score. We first examine the effect of our view-consistent D- Normal regularization. Our full model (Tab. 4 E) provides the best perfor- mance (0.40 Fl-score). The performance drops 0. Fl-score from 0.4 to 0.3 without the D-Normal regularizer (Tab, A) while keeping rendered normal regularization. It proves that it is insufficent to supervise only the normal maps rendered from Gaussian Splatting. The visualization in Fig. 6 demonstrates that our d-normal regularization can effectively push the 3D Gaussians towards the surface. Furthermore, the result drops by 0.04 F1-score without the confidence (Tab. B) and with the D-Normal regularizer. demonstrates that confidence can mitigate the problem of inconsistency of the predicted normal maps. From Fig. 7 can observe that disabling the confidence leads to an unsmooth surface. Both of these validate the effectiveness of the view-consistent D-Normal regularization. Additionally, the absence of intersection depth (Tab, (C) results in poor performance. Lastly, the performance increases from 0.33 Fl-score to 0,40 with our densification and split (Tab, D). proving small Gaussians represent surfaces better than large Gaussians.
Table 4: Ablation on TNT 224 Bold indicates best result.
Ablation Item
Precision
Recall f
F-score f
A. w/o D-Normal
0.27
0.34
0.30
B. w/o confidence
0.36
0.37
0.36
C. w/o intersection depth
0.35
0.37
0.35
D. w/o densify and split
0.32
0.35
0.33
E. Full
0.39
0.42
0.40
View
View 2
w/o Confidence
View
Inconsistent Pseudo Normal
Confidence
w/ Confidence
Figure 7: Qualitative ablation for the confidence. Without the confidence weight, the reconstructed surface shows protrusions caused by the inconsistent pseudo normal maps across different views.
5 Conclusion
In this work, we have introduced a view-consistent D-Normal regularizer for efficient, high-quality, and compact surface reconstruction. We formulate the D-Normal regularizer that directly couples normal with the other geometric parameters. This allows for the full update of all geometric parameters during normal regularization. We also propose a confidence term that weighs D- Normal regularizer to mitigate inconsistencies of normal predictions across multiple views. Finally, introduce densification and splitting strategy to regularize the scales and distribution of 3D Gaussians for more precise surface modeling. Our evaluations on diverse datasets demonstrate that method outperforms existing works in surface reconstruction.
Acknowledgement. This research project is supported by the National Research Foundation (NRF) Singapore, under its NRF-Investigatorship Programme (Award ID. NRF-NRFI09-00008
10

---

References
[1] Gwangbin Bae and Andrew Davison. Rethinking inductive biases for surface normal es- timation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[2] Jonathan Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3855-5864, 2021.
[3] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip- nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 5470-5479, 2022.
[4] Michael Bleyer, Christoph Rhemann, and Carsten Rother. Patchmatch stereo-stereo matching with slanted support windows. In Bmve, volume 11, pages 1-11, 2011.
[5] Adrian Broadhurst, Tom W Drummond, and Roberto Cipolla. probabilistic framework for space carving. In Proceedings eighth IEEE international conference on computer vision. ICCV 2001, volume 1, pages 388-393. IEEE, 2001.
[6] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In European Conference on Computer Vision, pages 333-350. Springer, 2022.
[7] Hanlin Chen, Chen Li, Mengqi Guo, Zhiwen Yan, and Gim Hee Lee. Gnesf: Generalizable neural semantic fields. In sevent- Conference on Neural Information Processing Systems, 2023.
[8] Hanlin Chen, Chen Li, and Gim Hee Lee. Neusg: Neural implicit surface reconstruction with 3d gaussian splatting guidance. arXiv preprint arXiv:2312.00846 2023.
[9] Jeremy S De Bonet and Paul Viola. Poxels: Probabilistic voxelized volume reconstruction. In Proceedings of International Conference on Computer Vision (ICCV), volume 2, page 2. Citeseer, 1999.
[10] Zhiwen Fan, Kevin Wang, Kairun Wen, Zehao Zhu, Dejia Xu, and Zhangyang Wang. Light- gaussian: Unbounded 3d gaussian compression with 15x reduction and 200+ fps, 2023.
[11] Sara Fridovich-Keil. Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5501-5510, 2022.
[12] Qiancheng Fu, Qingshan Xu, Yew Soon Ong, and Wenbing Tao. Geo-neus: Geometry-consisee neural implicit surfaces learning for niew reconstruction. Advances in Neural Information Processing Systems, 15:3403-3414 2022.
[13] Xiao Fu, Wei Yin, Mu Hu, Kaixuan Wang. Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. Geowizard: Unleashing the diffusion priors for 3d geometry estimation from single image. arXiv preprint arXiv:2403. /2013, 2024.
[14] Stephan J Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien Valentin. Fast- nerf: High-fidelin neural rendering 200fps. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14346-14355, 2021.
[15] Antoine Guédon and Vincent Lepetit. Sugar: Surface-aligned gaussian splatting for efficient 3d mesh reconstruction and high-quality mesh rendering. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[16] Menggi Guo, Chen Li, and Gim Hee Lee. Incremental learning for neural radiance field with uncertainty-filteee knowledge distillation. ECCV, 2024.
[17] Peter Hedman, Pratul P Srinivasan, Ben Mildenhall, Jonathan T Barron, and Paul Debevec. Baking neural radiance fields for real-time view synthesis. Proceedings of the IEEF/CVE International Conference Computer Vision, pages 5875-5884, 2021.
11

---

[18] Philipp Henzler, Niloy J Mitra, and Tobias Ritschel. Escaping plato's cave: 3d shape from adversarial rendering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9984-9993, 2019.
[19] Binbin Huang, Zehao Yu, Anpei Chen, Andreas Geiger, and Shenghua Gao. 2d gaussian splatting for geometrically accurate radiance fields. SIGGRAPH, 2024.
[20] Po-Han Huang, Kevin Matzen, Johannes Kopf, Narendra Ahuja, and Jia-Bin Huang. Deepmvs: Learning multi-view stereopsis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2821-28341 2018.
[21] Rasmus Jensen, Anders Dahl, George Vogiatzis, Engil Tola, and Henrik Aanæs. Large scale multi-view stereopsis evaluation. In 2014 IEEE Conference on Computer Vision and Puttern Recognition, pages 406-413. IEFF, 2014.
[22] Bernhard Kerbl, Georgios Kopanas, Thomas Leimküühle, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), July 2023.
[23] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan- Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv:2304.02643; 2023.
[24] Arno Knapitsch, Jaesik Park, Qian- Yi Zhou, and Vladlen Koltun. Tanks and temples: Bench- marking large-scale scene reconstruction. ACM Transactions on Graphics (ToG), 36(4):1-13, 2017.
[25] Kiriakos N Kutulakos and Steven M Seitz. A theory of shape by space carving. International journal of computer vision, 38:1999-11 2000.
[26] Zhaoshuo Li, Thomas Müller, Alex Evans, Russell H Taylor, Mathias Unberath, Ming- Yu Liu, and Shen-Hsuan Lin. Neuralangelo: High-fidelity neural surface reconstruction. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[27] Jiaqi Lin, Zhihao Li, Xiao Tang, Jianzhuang Liu, Shiyong Liu, Jiayue Liu, Yangdi Lu, Xiaofei Wu, Songeen Xu, Youliang Yan, and Wenming Yang. Vastgaussian: Vast 3d gaussians for large scene reconstruction. In CVPR, 2024.
[28] David B Lindell, Dave Van Veen, Jeong Joon Park, and Gordon Wetzstein. Bacon: Band-limited coordinate networks for multiscale scene representation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 16252-16262, 2022.
[29] Shaohui Liu, Yinda Zhang, Songyou Peng. Boxin Shi, Mare Pollefeys, and Zhaopeng Cui. Dist: Rendering deep implicit signed distance function with differentiable sphere tracing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2019-2028, 2020.
[30] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.cm 054999 2023.
[31] Wenjie Luo, Alexander G Schwing, and Raquel Urtasun. Efficient deep learning for stereo matching. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5695-5703, 2016.
[32] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision, pages 405-421. Springer, 2020.
[33] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG),41(4); 41 15, 2022.
12

---

[34] Eric Penner and Li Zhang. Soft 3d reconstruction for view synthesis. ACM Transactions on Graphics (TOG), 36(6):1-11, 2017.
[35] Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14335-14345, 2021.
[36] Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang. Hongyang Li, Qing Jiang, and Lei Zhang. Grounded sam: Assembling open-world models for diverse visual tasks, 2024.
[37] Gernot Riegler, Ali Osman Ulusoy, Horst Bischof, and Andreas Geiger. Octnetfusion: Learning depth fusion from data. In 2017 International Conference on 3D Vision (3DV), pages 57-66. IEEE, 2017.
[38] Johannes L Schönberger, Enliang Zheng, Jan-Michael Frahm, and Mare Pollefeys. Pixelwise view selection for unstructured multi-view stereo. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 1 11-14, 2016, Proceedings, Part III 14, pages 501-518. Springer, 2016.
[39] Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-from-mmotie revisited. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[40] Steven M Seitz, Brian Curless, James Diebel, Daniel Scharstein, and Richard Szeliski. A comparison and evaluation of multi-view stereo reconstruction algorithms. In 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR'06), volume 1, pages 519 128. IEEE, 2006.
[41] Steven M Seitz and Charles Dyer. Photorealistic scene reconstruction by voxel coloring. International Journal of Computer Vision, 35:151-173, 1999.
[42] Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießnnr, Gordon Wetzstein, and Michael Zollhofer. Deepvoxels: Learning persistent 3d feature embeddings. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2437 2476, 2019.
[43] Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, et al. The replica dataset: A digital replica of indoor spaces. arXiv preprint arXiv: 1906.05977 2019.
[44] Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Neural geometric level of detail: Real-time rendering with implicit 3d shapes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11358 2021.
[45] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309./6653 2023.
[46] Shubham Tulsiani, Tinghui Zhou, Alexei A Efros, and Jitendra Malik. Multi-view supervision for single-view reconstruction via differentiable ray consistency. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2626-2634 2017.
[47] Matias Turkulainen, Xuqian Ren, Iaroslav Melekhov, Otto Seiskari, Esa Rahtu, and Juho Kannala. Dn-splatter: Depth and normal priors for gaussian splatting and meshing, 2024.
[48] Benjamin Ummenhofer, Huizhong Zhou, Jonas Uhrig, Nikolaus Mayer, Eddy Ilg, Alexey Dosovitskiy, and Thomas Brox. Demon: Depth and motion network for learning monocular stereo. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5038-5047, 2017.
[49] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106.1068 2021.
13

---

[50] Yunsong Wang, Hanlin Chen, and Gim Hee Lee. Gov-nesf: Generalizable open vocabuaay neural semantic fields. arXiv preprint arXiv:2404.00931,, 2024.
[51] Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. Mvsnet: Depth inference for unstructured multi-view stereo. In Proceedings of the European conference computer vision (ECCV), pages 767-783, 2018.
[52] Yao Yao, Zixin Luo, Shiwei Li, Tianei Shen, Tian Fang. and Long Quan. Recurrent mvsnet for high-resolution multi-view stereo depth inference. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5525-5534, 2019.
[53] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems, 34:4805-4815, 2021.
[54] Mingqiao Ye, Martin Danelljan, Fisher Yu, and Lei Ke. Gaussian grouping: Segment and edit anything in 3d scenes. arXiv preprint arXiv:2312.00732, 2023.
[55] Wang Yifan, Felice Serena, Shihao Wu, Cengiz Öztireli, and Olga Sorkine- Hornung. Differen- tiable surface splatting for point-base bant-based& geometry processing, ACM Transactions on Graphics (TOG), 38(6):1-14, 2019.
[56] Wei Yin, Yifan Liu, Chunhua Shen, and Youliang Yan. Enforcing geometric constraints of virtual normal for depth prediction. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5684-5693, 2019.
[57] Zehao Yu, Anpei Chen, Binbin Huang, Torsten Sattler, and Andreas Geiger. Mip-splatting: Alias-free fres-freee 3d gaussian splatting. Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[58] Zehao Yu and Shenghua Gao. Fast-mvsnet: Sparse-to-dense multi-view stereo with learned propagation and gauss-newton refinement. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1949-1958, 2020.
[59] Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger. Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction. Advances in Neural Information Processing Systems (NeurIPS), 2022.
[60] Sergey Zagoruyko and Nikos Komodakis. Learning to compare image patches via convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4353-4361, 2015.
[61] Jingyang Zhang, Yao Yao, Shiwei Li, Zixin Luo, and Tian Fang. Visibility-aware multi-view stereo network. British Machine Vision Conference (BMVC), 2020.
[62] Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew Davison. In-place scene la- belling and understanding with implicit scene representation. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
[63] Qian- Yi Zhou, Jaesik Park, and Vladlen Koltun. Open3D: A modern library for 3D data processing. arXiv:1801 09847, 2018
[64] Matthias Zwicker, Hanspeter Plister, Jeroen Van Baar, and Markus Gross. Surface splatting. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pages 371-378, 2001.
[65] Matthias Zwicker, Jussi Rasanen, Mario Botsch, Carsten Dachsbacher, and Mark Pauly. Per- spective accurate splatting. In Proceedings-Graplee Interface, pages 247-254, 2004.
14

---

A Supplemental Material
A.1 Semantic Surface Trimming
I!!
w/o Semantic Trimming
Semantic Trimming
Figure Qualitative ablation for the semantic trimming. The left is disabling the semantic trimming and the right is enabling the strategy. We can see that the proposed semantic trimming can prune unwanted regions, e.g. the sky the left figure.
To avoid unwanted background elements like the sky in reconstructions for outdoor scenes, employ a semantic surface trimming approach by learning a semantic field [54 7
that leverages pretrained semantic model [23] 30 [6.. Specifically, we assign each Gaussian with learnable semantic feature. Subsequently, similar to color blending, alpha blending is applied on these features to get the pixel-level semantics. We use the predicted semantic map from Grounded-SAM [36] with cross-entropy loss to train the learnable features. After the optimization, we render the semantic map for each view and then use the semantics to mask out the background. Although we can use the predicted semantic maps from Grounded-SAl -AMM to prune the background directly, the predicted semantic maps not always accurate and consistent across views. Our proposed method can get more accurate semantics with noisy pscudo labels to a certain extent, which is also observed in [62
A.2 Proof Our D-Normal Regularizer
Fig. is to illustrate the optimization of positions of Gaussians under normal and d-normal supervi- sions. As we mentioned in Sec. [ in contrast to supervision on rendered normal maps which only updates Gaussian rotations, D-Normal regularizer also effectively update the positions of the Gaussians. We show the mathematical proof below.
Proposition 1.1: Supervision on rendered normal cannot effectively affect the positions of Gaussians.
Proposition 1.2: Supervision on our D-Normal regularizer can effectively affect the positions of Gaussians.
Proof: Without loss of generality, we omit the summation over multiple views in our following derivations for brevity. Based on the loss Cn rendered normal (cf. Eq.8) 8) the gradient of Cn with
15

---

respect to position p:
Lm
UN
(16)
0p:
ON
0p:
OÑ
ON 8a;
On.
Üp,
do:
Öp:
on;
Öp:
NN
do:
(G(x)
da:
(G())
Op:
ON da:
[-G(x)
R RT-- .x p.)]
da:
(G(x)
oÑ da:
[-G(x) (x-P.)l
(17)
da;
dG(x)
Putting Eq. into Eq. 16 we get:
Ln
da: [-G(xx) (x-)
Öp;
oN
da:
(G(x)
duy
3
Gxxx (x (x-pi)) P:)]
(G
fn oΝ
x (x - pi), where 3 ON da,
is a scalar.
(18)
Based on the D-Normal regularization Lan (cf. Eq.
12 the gradient of Lan with respect to position p:
0Lm 0Lan oNa a
0p: ON. aD 0p,
a aD dos aD ad:
Öp;
da: dp: Od, Ōp:
ad 0os
(G(() aD
da; OG(x)
dp.
Dd,
r.
n-r
(19)
We can deduce the following from Eq. and Eq.
19
0Ca of position independent of the we can see that the gradient-update dp. normal n. Consequently, the supervision on rendered normal cannot effectively affect the Gaussian position p.
Case From Eq.
16
Case 2: From Eq. 19 there is additional term with n where the denominator r is scalar n
aD move along the direction of the normal term. This effectively makes the change in the position Sp. n. Consequently, the supervision on D-Normal directly affects the Gaussian position p.
We can further deduce that the gradient-update on the Gaussian position pulls the position along the normal towards the surface, which achieves better reconstruction.
In view of the above proof, we conclude that it better to do supervision on the D-Normal regularizer. In addition to the mathematical proof, also visualize the positions of Gaussian centers from the optimized 3D scenes both with and without the D-Normal regularizer in Fig, 6 thereby providing experimental validation of the conclusion.
A.3 Implementation Details
We use PyTorch 2.0.1 and CUDA 11.8. for most experiments. All experiments are conducted on an NVIDIA 3090/4090/A5000/A6600 GPU. We set most hyperparameters the same as that used in
16

---

Gaussian Splatting 22 For outdoor scenes in the TNT dataset, we also utilize decoupled appearance modeling [27 alleviate the exposure issue. Moreover, to remove some outlier Gaussians, we adopt pruning technique from LightGaussian [TQ]. We also use a cuboid bounding box to contain the scene we need to reconstruct and we only regulate and add the new densification to Gaussians inside the box. We use the same train and test data with 2DGS on TNT, Mip-NeRF360, and DTU datasets.
Table 5: Additional ablation on TNT Dataset.
Precision
Recall f
F-score t
F. w/o semantic trimming
0.37
0.42
0.38
G. w/o scale regularization
0.35
0.38
0.36
H. Set scale $3 zero
0.36
0,40
0.37
I. Scale Regularization
0.39
0.42
0.40
A.4 Additional Ablation Studies
We further verify the effectiveness of additional design choices on reconstruction quality. We conduct experiments on the TNT dataset [24] and report the Fl-score. The quantitative result is reported in [ We first verify the proposed semantic trimming strategy (Tab. F). From the table, the strategy can increase the F1-score by 0.02. We can also observe the qualitative result from Fig. g that the trimming strategy can prune the sky correctly. Additionally, we confirm the effectiveness of scale regularization (Tab.F5)G) G)), which leads to an improvement of 0.04. The current work [19] sets the last item of the scale factor to zero to flatten the 3D Gaussians for surface reconstruction. Additionally, we also ablate the setting zero (Tab, (T and the scale regularization (Tab, [].. From
the table, can see that using the scale regularization instead of setting zero can obtain a 0.03 Fl-score improvement.
A.5
Additional Qualitative Results
Fig. 9 shows the rendering (top) and reconstruction (down) results on the Mip-NeRF360 dataset. The rendering results on the TNT and Replica datasets are shown in Fig. 11 We also compare the qualitative results of VCR-Gaus with SuGar and 2DGS, shown in Fig. 10 From the figure, we can see that our method can reconstruct both complete and high-detailed surfaces. In addition to the visualization in the form of pictures, we have also recorded video the supplementary material that can be downloaded and watched.
A.6 Additional Dataset
Table 6: Quantitative comparison on the DTU Dataset
21 We show the Chamfer distance and
average optimization time.
Method
24
37
40
55
63
65
69
83
97
105
106
110
114
118
122
Mean
Time
NeRF 32
1.90 1.60
1,85
0.58
2.28
1.27
1.47
1.67
2.05
1.07
0.88
2.53 1.06
1.15 0.96
1.49 >1hh
Implicit
VolSDF S3
1.14
1.26
0.81
0.49
1.25
0.70
0.72
1.29
1.18
0.70
0.66 1.08 0.42
0.61 0.55
0.86
>1hh
NeuS
1.00 1.37 0.93 0.43
1.10
0.65
1.48
0.83
0.52 1.20
0.35
0.49
0.54
0.84
>12h
MonoSDF
59
0.83 1.61 0.65
0.47
0.92
0.87
0.87
1.30
1.25
0.68
0.65
0.96
0.41
0.62
0.58
0.84 >12h
3DGS 22
2.14 1.53 2.08 1.68
3.49
2.21
1.43
2.07
2.22
1.75 1.79 2.55 1,53 1.52 1.50
1.96
< 1h
Explicit
SuGaR 15
1.47 1.33 1.13 0.61 2.25 1.71 1.15
1.63
1.62
1.07 0.79
2.45 0.98
0.88 0.79
1.33
< lh
2DGS
19
0.48 0.91 0.39
0.39
1.01
0.83
0.81
1.36
1.27
0.76
0.70
1 40
0.40
0.76
0.52
0.80
1h
Ours
0.55 0.91 0.40 0.43 0.97 0.95
0.84
1.39 1.30
0.90
0.76
0.92
0.44
0.75
0.54
0.80
< 1h
In this section, we report the result of object-level reconstruction, Although we focus on large-scale reconstruction, we still outperform implicit (i.e., NeRF, VoISDE, NeuS, and MonoSDIF) and most explicit methods (i.e., 3DGS and SuGaR) on DTU (1 While our method is comparable with current work 2DGS on object-level reconstruction, our method much better than on large-scale reconstruction, shown in Tab. I and Tab. 3 The qualitative results on DTU are shown in Fig. 12
17

---

Render
Mesh
Figure 9: Qualitative results the Mip-NeRF360 dataset. Our method reconstructs surfaces with fine geometry details and produces high-fidelity renderings on Mip-NeRF360 dataset.
B Limitations
Although our method can alleviate the problem of inaccurate normal prediction from pretrained normal estimator, especially the inconsistent normal predictions across views, it fails under the extreme case when almost all predicted normal across views are wrong. In addition, our method cannot reconstruct beyond the observed scene. Furthermore, our method cannot capture the surface of the semi trmi-transpprent object as shown in Fig. B
18

---

Ours
SuGar
20GS
Gr
Figure 10: Qualitative results on the Replica dataset.
Figure 11: Qualitative rendering results on the TNT and Replica dataset.
19

---

Render
Mesh
Figure 12: Qualitative results the DTU dataset.
AAT
Figure 13: Illustration of limitations: Our VCR- GauS encounters difficulties in accurately recon- structing semi-tre -ami-transparent surfaces, such as the window depicted in Caterpillar.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have discussed them on page 2.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed about that in the supplementary material.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
21

---

Justification: We do not have theoretical contributions in this work, where our contributions are validated with experiments.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All the syper-parameten and network organizations are provided in the main paper and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
22

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
Our codes have been released.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ "pipps.c/ppulli//guu / ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Training and test details are described in the main paper and the appendix. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
We follow existing related works for the setting of error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
23

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have provided that in the supplementary material.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httppp/////aarpp.ccc/o////////aa
Answer: [Yes]
Justification:
We have reviewed that and claim we conform that Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our method focuses on reconstructing 3D surfaces using Gaussian Splatting technique, which is component of 3D) reconstruction. It does not have further societal impacts than existing 3D reconstruction works.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not have such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited them in the references.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
25

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We and cite existing datasets in this work. Other assets including code/model will be released after submitting.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We do not include such experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not include such experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "4e1rBSDwYr", "reviewer": "Reviewer_kSsq", "summary": "The paper introduces a novel view-consistent Depth-Normal (D-Normal) regularizer and an uncertainty-aware normal regularizer followed by a new densification and splitting strategy to address the limitations of existing Gaussian Splatting methods in surface reconstruction tasks, such as the supervision of rendered normal updates affecting only the rotation parameter and the inconsistency of predicted normal maps. This approach enables the full optimization of Gaussian geometric parameters and decreases inaccuracy, thereby improving the quality of 3D surface reconstruction.", "questions": "Please see the weakness section.", "limitations": "Yes.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed method introduces novel insights, such as supervising normals rendered from 3D Gaussians, which only updates the rotation parameter. This is quite interesting. The overall model is supported by extensive experiments across multiple datasets, demonstrating significant improvements in both reconstruction quality and rendering speed compared to previous methods. The simplicity and effectiveness of this method are particularly commendable. Additionally, the paper is well-written and presents its concepts in a clear and accessible manner.", "weaknesses": "There are not many significant weaknesses in the method. However, a more in-depth analysis of scenarios where the method may fail, particularly with highly inconsistent normal predictions, would provide a clearer understanding of the method's boundaries.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer kSsq (R\\#1)\n\n**Q1**: A more in-depth analysis of scenarios where the method may fail, particularly with highly inconsistent normal predictions, would provide a clearer understanding of the method's boundaries.  \n**A:** Our method is likely to fail for ***semi-transparent objects*** (shown in Fig. 11 of paper) since the monocular normal estimator (*cf.* Line 168 of paper) we use to supervise the normal map (*cf.* Eq. 8 and Eq. 11\\) cannot accurately estimate the normal prior of semi-transparent objects. The failure of the monocular normal estimator on semi-transparent objects is due to the inability of depth cameras to capture ground truth depth maps accurately, which are converted to ground truth normal maps, of semi-transparent objects for training.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qR4rV6VIE5", "reviewer": "Reviewer_6weG", "summary": "This paper proposes to reconstruct surface from 3D Gaussians with a view-consistent depth-normal regularization. By introducing normal prior (DSINE/GeoWizard) to regularize the distribution of 3DGS, this approach is able to render smooth and view-consistent depth, facilitating reconstruction. This paper also tries to mitigate the inconsistencies from single-view normal prediction by introducing a confidence map. However, the effect seems to be limited.", "questions": "1) I'm not sure why manual normalization is needed in the equation (9), because according to equation (4), the denominator should already be a normalized coefficient.", "limitations": "Similar to many other reconstruction methods, this approach struggles to handle the reconstruction of specular or semi-transparent objects well.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) The performance is good. The proposed design can assist 3DGS for better depth rendering.\n2) Using the gradient of depth to supervise normal makes sense, which makes the position instead of rotation of each GS effectively optimized and leads to better depth results for reconstruction.\n2) The paper is easy to understand and follow.", "weaknesses": "1) Actually, using prior from monocular normal estimation has been introduced in several works, e.g. DN-Splatter; Would it help if the d-normal (calculated from simple GS depth / ray-GS intersection depth) is applied? \n2) The calculation of confidence is basically the same as D-Norm regularization, according to equation 12 and 13. Typically, in Bayesian learning, we introduce an uncertainty term. It is predicted separately and added in log form as a regularization term to the final loss function. I'm not sure if this approach has been tried before, as the current confidence constraints seem to have limited effectiveness.\n3) Some related work should be compared and cited: \n- Neuralangelo: High-Fidelity Neural Surface Reconstruction\n- Relightable 3D Gaussian: Real-time Point Cloud Relighting with BRDF Decomposition and Ray Tracing\n- 3DGSR: Implicit Surface Reconstruction with 3D Gaussian Splatting", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer 6weG (R\\#2)\n\n**Q1**: Actually, using prior from monocular normal estimation has been introduced in several works, e.g. DN-Splatter; Would it help if the d-normal (calculated from simple GS depth / ray-GS intersection depth) is applied?  \n**A**: As mentioned in L31 and L113-115 of our paper, the normal supervision in DN-Splatter can only effectively update the rotation (and scale) parameters without effectively affecting their positions for surface reconstruction. Refer to Eq. (6) of DN-Splatter for the formulation of normal prediction using only the rotation matrix and scaling coefficient of the Gaussian. We mitigate this limitation by deriving our D-Normal formulation in Eq. (10) of our paper, which is a function of both the normal (thus rotation and scale, *cf.* Eq (5) of our paper) and the position of the Gaussians. Consequently, our D-Normal would improve the performance when applied to DN-Splatter. We do experiments based on the DN-Splatter official code on the TNT dataset to verify the effectiveness of D-Normal, shown in the table below. We will update this comparison with DN-Splatter in the final version.\n\n| Ablation Item | Precision ↑ | Recall ↑ | F1-score ↑ |\n| :---- | :---: | :---: | :---: |\n| A. DN-Splatter | 0.13 | 0.14 | 0.13 |\n| B. DN-Splatter \\+ D-Normal | 0.29 | 0.34 | 0.31 |\n\n**Q2**: The calculation of confidence is basically the same as D-Norm regularization, according to equations 12 and 13\\. Typically, in Bayesian learning, we introduce an uncertainty term. It is predicted separately and added in log form as a regularization term to the final loss function. I'm not sure if this approach has been tried before, as the current confidence constraints seem to have limited effectiveness.  \n**A:** The Bayesian learning in \\[e\\] plays the same role as the uncertainty term we proposed in the paper, i.e. to downweigh error terms with high uncertainty in the cost function. Following the reviewer’s suggestion, we incorporate an uncertainty term using Bayesian learning following \\[e\\]. Specifically, we set each Gaussian with a learnable uncertainty term and then render the uncertainty map by alpha-blending uncertainty terms of Gaussians. Finally, we replace our proposed confidence map with the uncertainty map. We can see from the table below our proposed confidence is better than the learnable uncertainty on the TNT dataset.   \nBayesian learning in \\[e\\] is useful only when we have no prior information on the uncertainty term. However, in our case, we know the prior information geometrically as the cosine distance between the rendered and predicted normals. Consequently, our geometric-aware confidence term is more effective in suppressing excessive errors with high inconsistency across multiple views from dominating the overall cost.   \n\n| Ablation Item | Precision ↑ | Recall ↑ | F1-score ↑ | Time (min) |\n| :---- | :---: | :---: | :---: | :---: |\n| A. Uncertainty (Bayesian) | 0.35 | 0.36 | 0.37 | 65 |\n| B. Ours (confidence) | **0.39** | **0.42** | **0.40** | **53** |\n\n**Q3**: Some related work should be compared and cited:  \n\\[f\\] Neuralangelo: High-Fidelity Neural Surface Reconstruction  \n\\[g\\] Relightable 3D Gaussian: Real-time Point Cloud Relighting with BRDF Decomposition and Ray Tracing  \n\\[h\\] 3DGSR: Implicit Surface Reconstruction with 3D Gaussian Splatting  \n**A**: Thank you for your suggestion. Neuralangelo has been cited in our paper, and we will cite the two other papers. Since \\[g\\] is not for surface reconstruction, we only compare our method with Neuralangelo \\[f\\] and 3DGSR \\[h\\] on the TNT and DTU datasets as shown in the table below. Since 3DGSR does not show results on the TNT dataset and the code has not been released, we cannot compare our method with it on the TNT dataset. The results of Neuralangelo and 3DGSR are from the original papers. From the table, we can see that compared with Gaussian-based methods, including 2DGS and 3DGSR, our method achieves a significantly higher F1-score for large-scale reconstruction (TNT dataset) and is comparable with current work 2DGS and 3DGSR on object-level reconstruction (DTU dataset). Similar to other Gaussian-based methods such as 2DGS, our method shows worse performance than Neuralangelo. Nonetheless, our method is significantly more efficient than Neuralangelo with a reconstruction speed that is approximately 100x faster and a rendering speed of 145FPS (ours) vs. 10FPS (Neuralangelo).\n\n|   | NeuS-based |  | Gaussian-based |  |  |\n| :---: | :---: | :---: | :---: | :---: | :---: |\n| Dataset | NeuS | NeuralAngelo | 3DGSR | 2DGS | Ours |\n| DTU (CD ↓) | 0.84 | 0.61 | 0.81 | 0.8 | 0.8 |\n| TNT (F1-Score ↑) | 0.38 | 0.5 | \\- | 0.3 | 0.4 |\n| Training Time (h) | \\> 24 | 20 / 128 | \\< 1 | \\< 1 | \\< 1 |\n| Rendering speed (FPS) | \\< 10 | \\< 10 | \\- | 68 | 145 |\n\n**Q4**: I'm not sure why manual normalization is needed in equation (9), because according to equation (4), the denominator should already be a normalized coefficient.  \n**A**: Instead of accumulating depth like color in Eq. (4), we should take the average (normalized) depth of the flattened Gaussians since depth is a geometry entity (vs. appearance of color). We find in the official code of 2DGS that mean depth is also used for large scene reconstruction. We perform an ablation on normalized depth (i.e. mean depth) on the TNT dataset. The table below shows that using mean depth results in better reconstruction than unnormalized depth. \n\n| Ablation Item | Precision ↑ | Recall ↑ | F1-score ↑ |\n| :---- | :---: | :---: | :---: |\n| A. Unnormalized | 0.35 | 0.37 | 0.35 |\n| B. Normalized | **0.39** | **0.42** | **0.40** |\n\n\\[e\\] NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F3SbntDMCx", "reviewer": "Reviewer_n7KL", "summary": "This paper presents a confidence-aided Depth-Normal regularizer that directly couples the normal with other geometric parameters, thus enabling the optimization of all geometric parameters from monocular normal priors. The paper also introduces a densification and splitting strategy to regulate the size and distribution of the 3D Gaussian distributions for more precise surface modeling. Experimental results demonstrate that the method achieves better reconstruction quality with faster training speeds and rendering compared to 3DGS and 2DGS.", "questions": "1. I am curious why training time is longer than 2DGS but inference time is shorter.\n\n2. The reconstruction results of 2DGS in TNT as shown in Fig.4 seems different from 2DGS's original paper in Fig.10, such as the ground of 'truck scene' is not incomplete in 2DGS's paper, the head of 'Caterpillar' does not contain the noisy mesh.", "limitations": "The authors kindly point out their limitations in the paper. I am curious about the semi-transparent object. Does the monocular normal estimation fails in semi-transparent object?", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The main contribution of the paper is clear. The proposed Depth-Normal regularizer provides a simple but effective way to utilize the monocular normal priors in 3DGS-based reconstruction.\n\n2. Experimental results strongly support the proposed strategy.", "weaknesses": "1. The conversion between depth and normals is not a new concept and has been explored in both the field of depth estimation and normal estimation. I suggest referencing some previous works on depth-to-normal conversion, such as VNL[1]. I also acknowledge the contribution of the authors in applying this technique to 3DGS, which allows for better optimization of geometric attributes.\n\n[1] Yin W, Liu Y, Shen C, et al. Enforcing geometric constraints of virtual normal for depth prediction[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2019: 5684-5693.\n\n2. 2DGS also proposes a normal consistency loss, which connects the depth and normal. It would be better to clarify the difference and conduct the ablation study comparing the two conversion strategies. \n\n3. Building upon question 2, I'm curious to know if the gain in this paper is primarily due to the introduction of monocular normal priors. For instance, if we maintain the strategy of normal computation in 2DGS and incorporate the monocular normal prior in this paper. Since the major contribution of this paper lies in establishing the correlation between depth and normals, it would be better to provide an explanation or comparison to illustrate the superiority of the used conversion.\n\n4. As shown in Fig.7 and Fig.10, the details of the reconstruction results could still benefit from further improvement. This is also understandable since the reconstruction is based on the monocular normal prior, whose details are limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer n7KL (R\\#3)\n\n**Q1**: The conversion between depth and normal is from VNL.  \n**A**: We have cited VNL in our paper as \\[55\\], where we mentioned in L39 and L177 that our depth-normal formulation is inspired by them. Nonetheless, we are different from VNL as follows: the depth-normal formulation in VNL is proposed for the depth map prediction task. We adapted it to effectively update all geometric properties of the Gaussians for 3D reconstruction. Additionally, we show the state-of-the-art performance of this task with this formulation.\n\n**Q2**: 2DGS proposes a normal consistency. Clarify the difference and conduct the ablation comparing the two conversion strategies.  \n**A**: We differ from 2DGS as follows: \n\n1) We are first to show the insightful and important analysis on how our d-normal formulation can lead to a effective update of all the geometric parameters (rotation, scale, and position) of the Gaussian (*cf.* L184-185 of our paper) in comparison with the naive formulation of normal which effectively updates only the rotation and scale (*cf.* Eq. 5 of our paper). Since this important analysis is missing in 2DGS, they did not realize that it is better to supervise d-normal with monocular normal map predictions. As a result, they propose the weaker normal consistency loss (*cf.* Eq. 14 of 2DGS paper), which updates both the splat’s normal and depth normal and thus can lead to conflicting updates that hurt performance. \n\n    To verify our d-normal is better than the normal consistency used in 2DGS, we replace our d-normal with the normal consistency in our framework as shown in Column B in the table below. This leads to a performance degradation from 0.4 to 0.37.  \n   \n2) We propose a geometrically meaningful confidence term (*cf.* L187-198 of our paper) based on the cosine distance between the rendered and predicted normals to downweigh inconsistent normal priors across multiple views. Consequently, our confidence term suppresses excessive errors with high inconsistency from dominating the overall cost.   \n\n3) Although the normal supervision has made the normals more accurate, there is still a minor error leading to depth error arising from the remnant large Gaussians. We further devise a new densification strategy that splits large Gaussians into smaller ones to represent the surface better. \n\n|   | A. Ours | B. Ours \\+ normal consistency | C. 2DGS | D. 2DGS \\+ d-normal |\n| :---: | :---: | :---: | :---: | :---: |\n| F1-score ↑ | 0.4 | 0.37 | 0.3 | 0.34 |\n\n**Q3**: If the gain is due to the normal priors. Provide a comparison to illustrate the superiority of the used conversion.  \n**A**: Tab. 4 of our paper shows the contribution of each component of our framework in improving the performance. In addition to the monocular prior, our densification and splitting strategy also contributes significantly to improving the performance on the F1-score from 0.33 to 0.40. The confidence term and the intersection depth are also playing an important role in increasing the F1-score by 0.04 and 0.05, respectively. Following the suggestion of the reviewer, we conduct an experiment that adds our view-consistent d-normal regularizer to 2DGS. As shown in column D of the table above, our proposed d-normal regularizer added to 2DGS improves the performance on the F1-score from 0.3 to 0.34. Nonetheless, it is still worse than our method.\n\n**Q4**: As shown in Fig.7 and Fig.10, the details of the reconstruction need further improvement.  \n**A**: We respectfully disagree with the reviewer that Fig. 7 lacks details. Nonetheless, we agree that Fig. 10 is overly smooth and it is caused by the monocular prior whose details are limited. Although the reconstruction in Fig. 10 is too smooth, we still achieved superior performance on the DTU dataset as shown in Tab. 6\\. The over smoothness on the DTU dataset is due to the monocular normal estimator DSINE that we used for the prior is mainly trained on the scene-level dataset with only 160K images, and therefore cannot produce details on the object-level DTU dataset. To obtain more details of normal prior, we can train a larger DSINE with more parameters on a larger dataset, e.g. Omnidata \\[a\\] with 12M images, Objaverse \\[b\\] with over 10M objects, and MVImageNt \\[c\\] with 6.5M frames.\n\n**Q5**: Why training time is longer than 2DGS but inference time is shorter?  \n**A**: Our long training time is mainly incurred by the computation of the intersection depth, the rendering of semantic maps, and the optimization of more Gaussians generated by our proposed densification strategy. However, we do not need to compute the intersection depth for novel view rendering since we utilize the original rendering method from Gaussian Splatting. Also, the rendering of semantic maps can be given up when novel view rendering since it is only used to trim meshes. Consequently, our inference time (i.e. rendering) is almost as fast as Gaussian Splatting. However, the rendering method in 2DGS differs significantly from Gaussian Splatting in its need to compute intersection depth. Specifically, 2DGS needs to compute the intersection point first and then use it to get the Gaussian value for rendering. The inference (rendering) speed of 2DGS is therefore slower than ours. Furthermore, the training speed of 2DGS is also slower than the original 3DGS due to its slower rendering time. For example, on the TNT dataset, 2DGS needs 34.2 minutes while 3DGS only needs 14.3 minutes.\n\n**Q6**: The reconstruction of 2DGS in TNT seem different from 2DGS's paper.  \n**A**: The pretrained models or meshes on the TNT dataset are not released by 2DGS before the deadline for NeurIPS submission. We thus train 2DGS on the TNT dataset with the official codes of 2DGS and the meshes are extracted by the provided script. This would lead to slight deviations from the original paper.\n\n**Q7**: Does the normal estimator fail in semi-transparent objects?  \n**A**: Yes. Details can be found in R\\#1Q1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LAVb2EW8ZH", "reviewer": "Reviewer_5p3w", "summary": "The paper presents a method for improving geometry reconstruction for 3D Gaussian Splatting. It adopts a relatively flattened 3D Gaussian and incorporates normal regularization from monocular priors. Specifically, the paper proposes supervising the geometry by minimizing the differences between the normals derived from the depth map and the monocular normals. The method further refines the depth map by calculating the ray-Gaussian intersection and introduces a densification technique that splits large Gaussians on the surface into smaller ones, effectively reducing depth errors. To address multi-view inconsistencies arising from monocular priors, the paper proposes a weighting scheme to boost performance. Extensive results show that the method achieves competitive results on common benchmarks, including TNT, MipNeRF360, DTU, and Replica.", "questions": "1. The paper states that using the rendered normal for supervision can only update the rotation of the Gaussians. Why? Given that the rendering equation involves all the Gaussians and they are all differentiable, if we use a multi-view normal map and train the Gaussians with the rendered normal map, is it really true that these Gaussians will not be optimized into a shape that fits the given normal maps?\n\n2. Why setting the last scale to zero poses difficulties in optimization (L109)? Many papers have verified that setting the Gaussians to be 2-dimensional does not hurt performance [1][2]. I believe more justification is needed.\n\n3. Why 2DGS is faster than the proposed method in the training but has a lower FPS. The paper states the reason is that 2DGS applies a time-consuming ray-splat technique. However, according to the 2DGS paper, the ray-splat intersection does not seem as expensive, at least not more expensive than in this paper, since they also use intersected depth. Some analysis would be beneficial.\n\n\n[1] High-quality Surface Reconstruction using Gaussian Surfels. In SIGGRAPH'24\n\n[2] 3D-HGS: 3D Half-Gaussian Splatting. arXiv:2406.02720\n\n**Minors**:\n\nL175: \"splitting\" should be \"splatting.\"\n\nL198: I am also concerned about the use of \"view consistent\". In my opinion, the monocular normal priors are still multi-view inconsistent, although the proposed weighting scheme can reduce some negative effects.", "limitations": "The weakness has been outlined previously, and I have a few questions that need clarification. I may raise my score if these questions are addressed properly.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents a confidence map to resolve the multi-view inconsistencies arising from monocular priors estimated from pre-trained models.\n2. The paper identifies that large Gaussians cause significant depth errors and proposes a reasonable splitting scheme to address this issue.\n3. The paper is well-written, easy to follow, and the evaluation is comprehensive.", "weaknesses": "1. The contributions seem limited. The core component of the paper is the combination of depth normals and rendered normals for supervision with monocular priors. However, the depth normals and rendered normals appear similar to those in 2DGS, despite this paper using monocular priors for supervision between them. The intersected depth also shares a similar motivation to 2DGS and Gaussian Surfels [1]. However, the paper lacks acknowledge and reference to these works. The paper should mention these or provide sufficient discussion.\n\n2. While the introduced monocular normals improve reconstruction quality, they also cause oversmoothing, as shown in the DTU results. Additionally, it remains unclear how the monocular priors work for more general cases. The paper should discuss these as limitation.\n\n3. The idea of using monocular normal prior has been adopted in [1], the paper should have a citation or comparison.\n\n[1] High-quality Surface Reconstruction using Gaussian Surfels. In SIGGRAPH'24", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer 5p3w (R\\#4)\n\n**Q1**: Normal priors cause oversmoothing, as shown in the DTU results. Additionally, it remains unclear how the normal priors work for more general cases.  \n**A**: We agree that the monocular priors cause oversmoothing on the DTU results in Fig. 10 of our paper. The over smoothness on the DTU dataset is due to the monocular normal estimator DSINE we used that is mainly trained on scene-level dataset with only 160K images, and therefore cannot produce details on the object-level DTU dataset. To obtain more details of normal prior, we can train a larger DSINE with more parameters on a larger dataset, e.g. Omnidata \\[a\\] with 12M images, Objaverse \\[b\\] with over 10M objects, and MVImageNt \\[c\\] with 6.5M frames.\n\nWe are not sure of the meaning of “more general cases” mentioned in the reviews. We reckon that this refers to the “generalization ability across different scenes”, and we provide our clarification based on this understanding. Actually, we have performed experiments on four datasets: TNT, DTU, ScanNet++, and Mip-NeRF360 which cover large outdoor scenes, small objects, and large indoor scenes. We think these different types of scenes are sufficient to verify the effectiveness of our method.\n\n**Q2**: The paper states that using the rendered normal for supervision can only update the rotation of the Gaussians. Why?  \n**A**: We want to clarify why the supervision of the rendered normal maps can only update their rotations without directly affecting their positions. From the rendered normal equation shown below, we can see that 1\\) Regularization on normal maps can affect or supervise the normal of Gaussians in 3D space. In this case, since the normal is only determined by the rotation and scale of a Gaussian, the position cannot be updated. 2\\) you are right that the positions of Gaussians are also updated (thanks for catching this\\! We will update the paper to clarify this is what we meant). A more accurate expression is that the supervision of the rendered normal maps can only effectively update their rotations without **effectively** affecting their positions **for surface reconstruction**. In fact, the positions are only updated through the alpha weights by affecting G(x) in the alpha-blending equation. As illustrated in Fig. 2 (a) and (b) of the PDF file in our rebuttal, this can only move the 3D Gaussian towards or further away from the intersecting ray through the blended alpha. In other words, **the 3D Gaussians cannot move along the camera ray to be closer to the ground truth surface**. In contrast, as illustrated in Fig. 2 (c) of the PDF file in our rebuttal, **our D-Normal regularizer can move the Gaussian in the direction parallel to the pseudo normal of the surface**. To verify that the D-Normal regularizer actually behaves like this, we also visualize the optimized Gaussian centers of a scene in Fig.1 of the PDF file in our rebuttal. We can see that D-Normal regularizer significantly eliminates off-surface Gaussians.\n\n$$  \n\\\\hat{\\\\textbf{N}} \\= \\\\frac{\\\\sum\\_{i \\\\in M} {n\\_i} \\\\alpha\\_i \\\\prod\\_{j=1}^{i-1} (1 \\- \\\\alpha\\_j)}{\\\\sum\\_{i \\\\in M} \\\\alpha\\_i \\\\prod\\_{j=1}^{i-1} (1 \\- \\\\alpha\\_j)}, \\\\alpha\\_i \\= o\\_i G(\\\\textbf{x}\\_i)  \n$$\n\n**Q3**: Why does setting the last scale to zero pose difficulties in optimization (L109)?  \n**A**: The reasons are: 1\\) Setting the last scale to zero is a poor initialization for the normals of the Gaussians since all normals of the Gaussians in Gaussian Surfels \\[d\\] are initialized to \\[0,0,1\\]. Also, setting the last scale to zero means that the Gaussians can only be rotated to achieve alignment with the surface during optimization. In contrast, our scale regularization is less constrained by offering the possibility of selecting the minimal scaling factor as the normal in addition to rotating the Gaussians. 2\\) At the beginning, nothing can be observed at the view that is vertical to the Gaussians that are initialized as surfels. As shown in Fig. 3 of the PDF file in our rebuttal, this leads to a local minimum with no optimization. However, this situation does not happen in our method because we gradually flatten the 3D Gaussians using scale regularization. We perform ablation in Tab. 5 H to show that an F1-score improvement can be achieved with our scale regularization instead of setting the last scale to zero.\n\n**Q4**: Why 2DGS is faster than the proposed method in the training but has a lower FPS.  \n**A**: The answer can be found in R\\#3Q5.\n\n**Q5**: L175: \"splitting\" should be \"splatting.\"  \n**A**: Thanks for pointing it out, we'll correct it in the final version.\n\n**Q6**: L198: Concerned about the use of \"view consistent\".  \n**A**: We clarify that the “view-consistent” in L198 does not refer to the monocular normal priors used to supervise the D-Normals. The monocular normal priors are not changed and they are still inconsistent across multi-views. We mean that the addition of the confidence term in Eq. (12) downweighs the terms that have high multiview inconsistency, and therefore the remaining terms in the loss are “view-consistent”. The result drops by 0.04 F1-score without the confidence (Tab. 4 B) and with the D-Normal regularizer. It demonstrates that confidence can mitigate the problem of inconsistency of the predicted normal maps. In our title, “view consistent” is used to describe the proposed confidence term which places more emphasis on the view of consistent terms in the loss in Eq (13).\n\n**Q7**: The contributions seem limited. The depth normals and rendered normals appear similar to those in 2DGS. The intersected depth shares a similar motivation to 2DGS and Gaussian Surfels. Using monocular normal prior has been adopted in \\[d\\].  \n**A**: The answer can be found in the part of ‘To all Reviewers’ at the beginning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## To all reviewers:

We thank the reviewers for their constructive feedback. As summarized by our reviewers, this paper introduces “novel insights” (R\#1) with “clear contributions” (R\#3), and the proposed method is “simple and effective” (R\#1, 3). Through “comprehensive evaluation” (R\#4), our method achieves “good performance” (R\#2) with “significant improvements” (R\#1). 

Here, we explain the difference between our method with our concurrent works: 2DGS and Gaussian Surfels \[d\] to answer R\#3Q2 and R\#4Q7.

**A**: We cited 2DGS and even showed experimental comparisons with 2DGS in our paper. However, Gaussian Surfels first appeared on arXiv at the beginning of May 2024, which was too close to the NeurIPS submission deadline. We did not notice the paper until after our NeurIPS submission. It should also be noted that both 2DGS and Gaussian Surfels are SIGGRAPH 2024 papers which took place less than 2 weeks ago, well after the NeurIPS submission deadline in May 2024\. Although we consider Gaussian Surfels as our concurrent work, we will cite and compare with it in our final paper.

We differ from 2DGS and Gaussian Surfels as follows: 

1) We show an insightful and important analysis on how our D-Normal formulation can lead to a complete and effective update of all the geometric parameters (rotation, scale, and position) of the Gaussian (*cf.* L184-185 of our paper) in comparison with the naive formulation of normal which updates effectively only the rotation and scale (*cf.* Eq. 5 of our paper). We also mention in our paper that our derivation of D-Normal is inspired by VLN (*cf.* reference \[55\] in our paper) which was originally derived for depth map prediction.   

    Since this important analysis is missing in 2DGS and Gaussian Surfels, they did not realize that it is better to supervise D-Normal with only monocular normal map predictions. As a result, both 2DGS and Gaussian Surfels propose the weaker normal consistency loss (*cf.* Eq. 14 of 2DGS paper), which updates both the splat’s normal and depth normal and thus can lead to conflicting updates that hurt performance. Although Gaussian Surfels also supervised the depth normal with monocular normal, their addition of the weaker normal consistency loss that also updates the splat’s normal can cause the performance to drop (*cf.* column A vs B of the table in the response to R\#3).

2) To address the inconsistency across multiple views of the normal prior from a pretrained monocular model, we propose a geometrically meaningful confidence term (*cf.* L187-198 of our paper). This is based on the cosine distance between the rendered and predicted normals to downweigh inconsistent normal priors across multiple views. Consequently, our confidence term suppresses excessive errors with high inconsistency from dominating the overall cost. This important confidence term is not present in both 2DGS and Gaussian Surfels.  
     
3) Although the normal supervision has improved the normal accuracy, we notice that there is still a minor error leading to depth error arising from the remnant large Gaussians. We thus devise a new densification that splits large Gaussians into smaller ones to represent the surface better. In contrast, both 2DGS and Gaussian Surfels stopped at regularizing the normals.   
     
4) Different from 2DGS which has to compute the intersection depth/point first and then render a novel view based on the point because of the different splatting method with original Gaussian Splatting, our method only utilizes the intersection depth for training and surface reconstruction and we can discard it for faster rendering during inference.   
     
5) Compared with Gaussian Surfels, the formulations of the intersection depth are different: ours simplifies the problem to the intersection between a ray and a plane which is exact, while Gaussian Surfels uses a local Taylor expansion to approximate the intersection. 

Finally, we compare our method with Gaussian Surfels on the TNT and DTU datasets, shown in the table below. The result of Gaussian Surfels on DTU is directly from the paper. Since Gaussian Surfels did not conduct the experiment on the TNT dataset, we ran their official code on the TNT dataset with the same normal priors as ours. From the table, we can see our method outperforms Gaussian Surfels \[d\].

| Dataset | Gaussian Surfels | Ours |
| :---: | :---: | :---: |
| DTU (CD ↓) | 1.19 | 0.8 |
| TNT (F-Score ↑) | 0.17 | 0.4 |

Here are some references used in the rebuttal:  
\[a\] Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets From 3D Scans. Eftekhar, Ainaz and Sax, Alexander and Malik, Jitendra and Zamir, Amir. ICCV'21.  
\[b\] Objaverse: A universe of annotated 3d objects. Deitke Matt, etc. CVPR'23.  
\[c\] MVImgNet: A Large-scale Dataset of Multi-view Images. Yu Xianggan, etc. CVPR'23.  
\[d\] High-quality Surface Reconstruction using Gaussian Surfels. Pinxuan Dai, etc. SIGGRAPH'24.  
\[e\] NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections. CVPR'21.

Meta-review: After the rebuttal and significant discussion between the reviewers and the authors, this paper received consistent positive scores.

Reviewers find that it is well-motivated, well executed with interesting methodological insights on the nature of the relationship of gaussian position, their depth, and their normals, as well as an extensive evaluation.

After significant discussion with SACs and other ACs, the AC concludes that this paper is of significant interest to the part of the community working on 3D gaussian splatting and novel view synthesis, but not necessarily relevant to the broader NeurIPS community. The AC thus proposes acceptance as a poster.
Decision: Accept (poster)