GSDF: 3DGS Meets SDF for Improved Neural Rendering and Reconstruction
Authors: Mulin Yu
Tao Lu
Linning Xu
Lihan Jiang
Yuanbo Xiangli
Bo Dai
Venue: NeurIPS 2024 poster (2024)
Keywords: Neural Rendering; 3D Reconstruction;3D Gaussian Splatting; Signed Distance Field
Primary area: machine_vision
TL;DR: We propose GSDF, a dual-branch system that enhances rendering and reconstruction at the same time, leveraging the mutual geometry regularization and guidance between Gaussain primitives and neural surface.

Abstract: Representing 3D scenes from multiview images remains a core challenge in computer vision and graphics, requiring both reliable rendering and reconstruction, which often conflicts due to the mismatched prioritization of image quality over precise underlying scene geometry. Although both neural implicit surfaces and explicit Gaussian primitives have advanced with neural rendering techniques, current methods impose strict constraints on density fields or primitive shapes, which enhances the affinity for geometric reconstruction at the sacrifice of rendering quality. To address this dilemma, we introduce GSDF, a dual-branch architecture combining 3D Gaussian Splatting (3DGS) and neural Signed Distance Fields (SDF). Our approach leverages mutual guidance and joint supervision during the training process to mutually enhance reconstruction and rendering. Specifically, our method guides the Gaussian primitives to locate near potential surfaces and accelerates the SDF convergence. This implicit mutual guidance ensures robustness and accuracy in both synthetic and real-world scenarios. Experimental results demonstrate that our method boosts the SDF optimization process to reconstruct more detailed geometry, while reducing floaters and blurry edge artifacts in rendering by aligning Gaussian primitives with the underlying geometry.

GSDF: 3DGS Meets SDF for Improved Neural Rendering and Reconstruction
Mulin Yu
Tao Lu Linning Xu² Lihan Jiang Yuanbo Xianglist Bo Dai
Shanghai Artificial Intelligence Laboratory, "The Chinese University of Hong Kong, &Cornell University, 4 University of Science and Technology of China, 5 The University of Hong Kong
3DGS
Explicit Regularization (3D 2D)
Ours
(GS-branch)
(SDF-branch)
N0.12
+0.11
4.12
Representation
1015
8.1
(compramised modeling capactt)
(structured with implicit regular (zation))
Rendering
Reconstruction
(looslly structured)
loveraging GS geometry praxy)
(anchoring accurate eurface
Figure 1: Conceptual Illustration of GSDE. Rendering and reconstruction tasks have traditionally involved trade-offs in neural representation methods. While 3D-GS achieves high-fidelity view- dependent rendering, it often compromises on geometric accuracy. Recent approaches [3. use explicit regularization align Gaussian primitives near surfaces, but this can reduce model capacity for high-fidelity visuals. Our GSDF introduces a dual-branch framework with specialized GS- and SDF-branches for rendering and geometry tasks. We propose three mutual guidances (detailed in to enhance the quality of both tasks.
Sec. 3.2
Abstract
Representing 3D scenes from multiview images remains a core challenge com- puter vision and graphics, requiring both reliable rendering and reconstruction, which often conflicts due to the mismatched prioritization of image quality over pre- cise underlying scene geometry. Although both neural implicit surfaces and explicit Gaussian primitives have advanced with neural rendering techniques, current meth- ods impose strict constraints density fields or primitive shapes, which enhances the affinity for geometric reconstruction at the sacrifice of rendering quality. To address this dilemma, we introduce GSDF, a dual-branch architecture combining 3D Gaussian Splatting (3DGS) and neural Signed Distance Fields (SDF). Our ap- proach leverages mutual guidance and joint supervision during the training process to mutually enhance reconstruction and rendering. Specifically, our method guides the Gaussian primitives locate near potential surfaces and accelerates the SDF convergence. This implicit mutual guidance ensures robustness and accuracy in both synthetic and real-world scenarios. Experimental results demonstrate that our method boosts the SDF optimization process to reconstruct more detailed geometry, while reducing floaters and blurry edge artifacts in rendering by aligning Gaussian primitives with the underlying geometry.
'Equal contribution Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
Recent advancements in neural scene representations have showcased superior rendering capabilities
23
24
17. these advancements have sparked significant interest in neural surface reconstruction [39
31
19
32
10 [3]. They seek to develop unified representations, which simultaneously support high-
fidelity rendering and accurate geometric reconstruction, to better support downstream applications
such as roboties [16 (27), physical simulations [35] g and XR applications [36 15].
Although both neural implicit surfaces and explicit Gaussian primitives have advanced with neural rendering techniques. current methods often suffer from a mismatched prioritization between appear- ance and geometry. Imposing regularization or constraints on the unified representation may boost one task, while unavoidably deteriorating the performance of the other. Recent approaches [TO] [5]. have explored using flat Gaussian primitives for surface modeling. Enforeing binary opacity [T]] and jointly learned NeuS models [5] to regularize attributes have led to degraded rendering quality due to primitive constraints. However, works such as Adaptive Shell [4], Binary Occupancy Field [26], and Scaffold-GS (1 have demonstrated that incorporating geometry guidance significantly enhances rendering quality by producing well-regularized spatial structures with hybrid representations.
Building on these insights, we propose synchronously optimized dual-brunch system, addressing rendering and reconstruction with hybrid representations, to bypass the confficts and further achieve mutual enhancements, as illustrated in Fig. I Our method leverages mutual guidance and joint supervision to balance rendering and reconstruction without compromising their intrinsic advantages. Specifically, our system features a 3D Gaussian Splatting (3DGS) branch for rendering and a Signed Distance Field (SDF) branch for surface reconstruction. The key innovations of our approach include: (1) Utilizing rasterized depth from the cS-branch to guide ray sampling in the SDF-branch, enhancing volume rendering efficiency and avoiding local minima. (2) Applying SDF-guidane
for density control in 3DGS, directing the growth of 3D Gaussians near-surface regions and pruning elsewhere. (3) Aligning geometry properties (depth and normal) estimated from both branches. This unified system overcomes the limitations inherent in each method due to differences in rendering techniques (rasterization vs. dense ray sampling) and scene representation (discrete primitives vs. continuous fields). Morcover, our framework is designed to accommodate future advancements in each branch.
Extensive experiments demonstrate that our dual-branch design allows: 1) The GS-branch to generate structured primitives closely aligned with the surface, reducing floaters and improving detail and edge quality in view synthesis. 2) Accelerated convergence in the SDF-branch, resulting in superior geometric accuracy and enhanced surface details. Our results confirm that an integrated blend of both reconstruction and rendering is achievable, enhancing overall robustness and performance.
2 Related work
Advanced Neural Rendering Techniques. Neural Radiance Fields (NeRFs) 23 have achieved remarkable photorealistic rendering with view-dependent effects. They use Multi-Layer Perceptrons (MLPs) to map 3D spatial locations to color and density, which are then aggregated into pixel colors through neural volumetric rendering. This approach excels in novel view synthesis but is slow due to the need for extensive point sampling along each ray and the global MLP architecture's scalability limitations. Recent research [24] 37 4
9 has shifted the learning burden to locally optimized spatial features, enhancing scalability. Alternative methods like rasterizing geometric primitives (e.g., point clouds) 2 41 offer efficiency and flexibility but struggle with discontinuities and outliers. Augmenting points with neural features and incorporating volumetric rendering (38) improves quality but adds computational overhead. Recently, 3D Gaussian Splatting (3DGS) [7] revolutionized neural rendering by using anisotropic 3D Gaussians as primitives, sorted by depth and rasterized onto a 2D screen with a-blending. This method achieves high-quality, detailed results at real-time frame rates. Scaffold-GS [2I] enhanced 3DGS by introducing a hierarchical structure that aligns anchors with scene geometry, improving rendering quality and memory efficiency. However, these methods prioritize view synthesis over accurate scene geometry, often resulting in fuzzy volumetric density fields that hinder the extraction of high-quality surfaces
Neural Surface Reconstruction. The success of neural rendering has sparked significant interest in 19 32 40). These methods typically use coordinate-basse
neural surface reconstruction (25 39
networks to encode scene geometry through occupancy fields Signed Distance Field (SDF) values.

---

Anchor - Neural GS
ROB
[b) Grow Surface Anchors
Rasterize
GS-branch
Prune Outlier Anchors
SDF C-level
Mutual Geometry
Geometry-aware Density Control
Supervision (wr foreground mask)
Depth-guided Ray Sampling
SDF O-level
kt|s|
SDF-branch
depth
Namnal
Distance to
Volume Rendering
RGB
Sampling
range
Figure 2: Overview Dual-branch Guidance. Our dual-branch framework includes a GSSbranch for rendering and an SDF-branch for learning neural surfaces. This design preserves the efficiency and fidelity of Gaussian primitive for rendering [7 21 while accurately approximating scene surfaces from an SDF field adapted from NeuS [31] Specifically: (I) The GS-branch renders depth maps to guide SDF-branch ray sampling, querying absolute SDF values [s] and sampling points within 2k|s] (e.g., k = 4). (2) Predicted SDF values guide GS-branch density control, growing Gaussians near surfaces and pruning deviated ones. (3) Mutual geometry consistency is enforced by comparing depth and normal maps from both branches, ensuring coherent alignment between Gaussians and surfaces.
While MLPs with volume rendering produce smooth and complete surfaces, they often lack high- fidelity details and are slow to optimize. Recent works 19 26 have leveraged multi-resolution hashed feature grids from iNGP (24] to enhance representation power, achieving state-of-the-ard results. Hybrid approaches combining surface and volume rendering 30 34 26 have also emerged to maintain rendering speed and quality. More recently, methods have explored integrating 3D Gaussian Splatting for surface learning. For example, SuGaR (10) aligns 3D Gaussians with potential surfaces by approximating them to 2D planar primitives with binary opacity. Similarly, 2D Gaussian Splatting [13 replaces 3D Gaussians with 2D Gaussians for more accurate ray-splat intersection and regularizes depth maps to enforce more condensed Gaussian primitive distribution. Another approach 7 optimizes Gaussian surfels constrained by normal prior from pre-trained monocular estimator, enhancing surface representation. The concurrent work NeuSG [5] jointly optimizes 3D Gaussian Splatting with NeuS (31] by encouraging flat 3D Gaussians as well, with normals aligned with NeuS predictions. 3DGSR 22 also aligns the geometry from 3DGS with a SDF field. Despite advances in neural surface reconstruction, a fidelity gap persists due to the explicit regularization of Gaussian primitives, which hampers rendering quality compared to 3DGS. Our method optimizes primitives aligned with surfaces using geometric clues, enhancing structural integrity without losing representational power.
Method
We present a dual-branch framework with GS- and SDF-branches, jointly optimized to provide mutual guidance, as shown in Fig. 2 Our approach aligns Gaussian primitives with surfaces using geometric clues rather than constraining their shapes, enhancing both structure and representational power. These improved primitives yield more accurate and detailed scene surfaces.
In Sec 3.1 we briefly review neural rendering methods 17 21 and neural implicit surface representa-
tions 31 19]. See 3.2 details our framework and the three proposed mutual guidance techniques.
Sec 3.3 outlines our training strategy and loss design.

---

3.1 Preliminary
3D Gaussian Splatting. 3D Gaussian Splatting (3DGS) represents scenes using 3D Gaussian primitives, achieving state-of-the the rendering quality and speed with a blsed rasterizer. Each Gaussian is defined by its mean μ R and covariance Σ R with G(x)) e (<(e-/=="<<<>00> where z is 3D position the scene. The rasterizer efficiently sorts and a-blends Gaussians onto the 2D image plane [43]. The adaptive control of Gaussians based on gradients is also critical for improving scene representation accuracy, reducing redundancy, and refining Gaussians to better match the underlying geometry and appearance.
Scaffold-GS (1 improves 3DGS by enhancing scene structure fidelity and robustness to view- dependent effects. It uses hierarchical 3D Gaussian representation, with anchor points encoding local scene information and generating local neural Gaussians. Each anchor, optimized with a feature vector, predicts the color, center, variance, and opacity of the neural Gaussians.
Neural Implicit SDFs. NeuS [3 integrates signed distance functions (SDFs) with NeRF's volu- metric rendering, converting SDF values into volume densities to maintain geometric accuracy. SDF
values are converted to opacities using a logistic function a; = max ((x))))))= (f(,.))))) .0)
4 (x(f(x)))
The color of ray is also calculated by accumulating weighted colors of the sample points:
C(r) = E C, and T; exp (-E }50z) where 6j is the interval between sampled
points. Inspired by [24]. [ introduces multi-resolution hash grid to enhance representation power and accelerate rendering and training.
3.2 GSDF: Dual-branch for Rendering and Reconstruction
To bridge the gap between rendering and geometric accuracy, we propose novel approach that combines the strengths of Gaussian-basee and SDF-based methods. By leveraging the high-quality rendering capabilities of 3DGS and the precise geometric representation of SDFs, our method aims to achieve superior results in both tasks.
As illustrated in Fig. our bual-branch design integrates GS-branch and an SDF-branch. We use Scaffold-GS (2 and NeuS (3 with adapted hash-encoding implementation [1] as the backbones, chosen for their effectiveness and simplicity. Importantly, framework is versatile and can be easily adapted to incorporate future advanced methods for each branch.
3.2.1
GS + SDF: Depth Guided Ray Sampling
To address the computational expenses of ray- ray-sampling, techniques such as hierarchical sampling [23] occupancy grids 20 24), early stopping [24]. and proposal networks [3] 26 are widely used with CUDA accelerations. When depth maps are available, either from sensors or monocular estimation, samples can be strategically placed around surface regions, which is crucial for effective optimization of the Signed Distance Field (SDF). Unlike neural implicit SDF- based methods that rely on their own predicted SDF values for ray sampling [31] 19 28),
we employ the GS- ES-branch to provide surface proximity, avoiding the chicken-and-egg dilemma. Inspired by (37), where more efficient branch provides coarse geometry guidance, we leverage depth maps from the GS-branch to refine the ray sampling range for the SDF-branch. While Gaussian primitives may be less precise, they efficient and flexible, offering sufficient geometric clues without significant overhead.
Concretely, for ray emitted from camera center ő in the direction ग. the depth value D) from the GS-branch is:
D= Σ di. (1-0,,). iEN
1=1
where is the number of 3D Gaussians encountered, σ; is the opacity, and d, is the distance from the i-th Gaussian the camera. For branch optimization, points are sampled around ø D ū. The sampling range adapts based on predicted SDF values s at various depths:
s = Fo D 7)
(2)
where F, is two-layer MLP predicting the SDF value at given position. The sampling range
defined 1 (D- k|ss) 5, (D + k|ss) T.. As shown in Fig, 2 inspired from NeRF's

---

we use coarse and fine ranges with it = 3 and is = 1, respectively, we uniformly sample M points along the ray within each range.
hierarchical sampling strategy [23]
3.2.2 SDF GS: Geometry-aware Gaussian Density Control
have attempted to learn scene surfaces from 3DGS by flattening 3D Gaussians along the normal direction and enforcing nearly binary opacity, treating them like surface primitives similar to mesh triangles. However, this approach often leads to degraded rendering quality and incomplete surfaces. Our approach, instead of putting additional regularization on the 3D Gaussian primitives, enhances the distribution of Gaussian primitives using geometry-aware density control strategy. Building the original gradient-based density control, we leverage the zero-level set of the SDF-branch to determine the proximity of Gaussian primitives able
Many previous and concurrent methods [33 10 7
the surface. By querying the SDF-branch with the positions of Gaussian primitives, identify Hose-to-surface Gaussian primitives (i.e. with smaller absolute SDF values), allowing for more precise control over the placement and density of Gaussians.
Growing Operator. For each Gaussian primitive at position e, we obtain its SDF value s = F.(c) from the SDF-branch. The growth criteria for Gaussians then defined as:
Eg Vg + Wgp(s),
where Vg is the averaged gradient of Gaussian primitives accumulated over training iterations, as in Scaffold-GS (2T). yus) exp (20}(
Gaussian function converting the SDF value to positive factor, decreasing monotonically with distance from the zero level set. Wg controls the influence of geometrie guidance. New Gaussian primitives are added eg are greater than predefined Tg.
Pruning Operator. Beyond the original opacity based criteria, we prune Gaussian primitives far from the surface, indicated by large SDF values. The pruning criteria are:
Ep oa uip (())),
(4)
where a, is the aggregated opacity over K iterations. The weight wp balances the contributions of transparency and SDE, Primitives with Ep less than a predefined are pruned.
3.2.3 GS + SDF: Mutual Geometry Supervision
To enhance both rendering and reconstruction outcomes, our framework incorporates mutual supervi- sion between the two branches, using depth and normal maps as key geometric features to facilitate this interconnection.
For the SDF-branch, a per-view depth map D, is rendered using volumetric rendering principles, and a normal map N, is derived by rendering the gradients of the SDF volumetrically. For the GS-branch, compute per-view depth map D, following Eq. [ The normal of each 3D Gaussian To render the normal map for each camera view, we accumulate the normals of the 3D Gaussians using a-blending
is determined by the direction of its smallest scaling factor, as in [10] 5 6]
Ng EiEN noo; | - -5)) where n, the estimated normal of the i-th 3D Gaussian.
This mutual geometry supervision ensures that the depth and normal maps from both branches consistently aligned. This consistent alignment is crucial for maintaining coherence between the rendering and reconstruction processes, helps to prevent discrepancies that could lead artifacts and inaccuracies when performing mutual guidance. This consistency is especially important for achieving high-quality results in complex scenes, where precise geometric alignment can significantly enhance the visual and structural integrity of the output.
3.3 Training Strategy and Loss Design
The GS- branch is supervised by rendering losses Ci and LSSIM. which measure the difference between the rendered RGB images and ground truth images, and volume regularization term Lool as in 21]. The complete loss function for GS-branch is:
Cg X1L1 + (1- CIS + Aoothoots
(5)
5

---

where A1 and Avo are weighting coefficients. Similarly. the SDF-branch is supervised by the L rendering loss, supplemented with Eikonal and curvature penalties to ensure accurate geometry reconstruction. The loss function for the SDF-branch is:
C = L1 Ceik Aeury
(6)
where Leik represents the Eikonal loss, ensuring that the gradients of the predicted SDF field normalized, and Laury denotes the curvature loss, promoting surface smoothness as described in [19
23
The coefficients Xeik and Aqure balance the influence of these loss terms.
The mutual geometry supervision includes depth and normal consistency losses applied to both branches. For unbounded scenes with distant backgrounds, we only apply the mutual geometric supervision to foreground regions, as these geometries are more reliable and of primary interest The mutual loss is formulated as:
Lmmtual = XaLa + AnC >all Dall An (1 1g .ol
| |N
where La and Cn represent the depth and normal discrepancies between the two branches, and Ad and An balance their importance. Finally, the total loss for joint learning is defined as:
C Lg + La Lumtual-
(8)
The param--e settings are detailed in the supplementary material.
Experiment
4.1 Experimental Setup
Datasets. We evaluated results using 26 real-world and synthetic scenes from various datasets: 7 from Mip-NeRF360 [3] 2 from DeepBlending [12). 2 from Tanks&Temples [18), and 15 from DTU [14. featuring a wide range of indoor, outdoor, and object-centric scenarios.
Implementation Details. We implemented our dual-branch model based on 1) Scaffold-GS 21 and 2) enhanced version of NeuS [31] with hash-grid variant [11), following the practice of [9 The hash grid resolution spans from 2 to 2 with 16 levels, each entry having a feature dimension of and maximum of 2 entries per level. The coarsest 4 layers were activated initially for the DTU M. and 8 layers for other datasets, with finer levels added every 2k iterations. We trained the GS-branch for 15k iterations, followed by joint training of both branches for 30k iterations. The SDF-branch was warmed up for 2k iterations on the DTU and 5k on other datasets without depth-guided ray sampling. Note that our system adaptable with various existing or futuure rendering and reconstruction models.
Evaluations. We evaluated our method against state-of-the- art rendering and reconstruction ap- proaches. For rendering comparison, we compared with Scaffold-GS 21, 3D-GS [17), NeuS [I], 2D-GS [3), and SuGaR [10] using PSNR, SSIM 333, and LPIPS [22 for quantitative comparisons. We trained 3D-GS and Scaffold-GS GS for 45% iterations to align with our configurations. For reconstruc- tion, we compared with NeuS B1, Instant-NSR [ (our representative
branch,,, SuGaR [10] and 2D-GS (13.. We use Chamfer distance for quantitative evaluation. For all datasets, used 1/8 of the images as test sets and the other 7/8 as training sets. All experiments were conducted on single NVIDIA A A10 GPU with 80G memory.
4.2 Results Analysis
4.2.1 Rendering Comparisons
Our GSDF retained high rendering quality compared to state-of-the-an art 3D-GS and SDF-based methods, as shown in Table Quantitative metrics against 3D-GS [7], 2D-GS [13], and Scaffold- GS (2I show that GSDF consistently outperformed these methods across all four datasets, showcasing improvements in all metrics. Notably, the prominent improvements in the LPIPS metric indicate that our method effectively captures high-frequency scene details and renders perceptually superior results compared to baselines.

---

Table 1: Rendering and reconstruction comparisons against baselines over four benchmark
scenes. 3D-GS 17], Scaffold-GS (11. 2D-GS 13 SuGaR 10 and GSDF initialized the Gaussian
primitives with COLMAP (29] sparse points.
TaspooTeunpla
Mip-NeRF16c
Deep Blending
Matied Mesics
LPIPS
LPPPS
PSNK
LPTPS
SSIM
LPIPS
SDI Based
InstinNSi
Neu5 H]
BJ825
1.391
26.34
31.93
0.912
0.866
0.330
0.165
19.96
0.641
01405
21.87
0.612
0.909
24.00
0.205
0.462
30-05
4:1144
0.433
26.75
0.89%
0.210
0.357
0.211
29.44
0.899
ScatfolL.co
Hyhnd
GSDF
03002
01988
0.875
0388
Table 2: Rendering comparisons with random initialization against baselines over three bench- mark scenes. Scaffold-GS (rand) [2I]. 2D-GS (rand) and GSDF (rand) initialized the Gaussian primitives with random points.
Dataset
Mip-NeRF330
Tanks&
18
Deep Blending [12
Method Metrics
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
2D-GS (rand)
26.94
0.793
0.300
0.817
0.275
28.76
0.892
0.274
Scaffold-GS (rand)
27.84
0.817
0.259
26.37
0.838
0.230
29.12
0.895
0.260
GSDF (rand)
28.05
0.830
0.229
26.55
0.852
0.197
29.57
0.899
0.244
20-65
30-65
Scaffelé-GS
660F (Our)
GT
UA VECLETS
Figure 3: Qualitative comparisons of GSDF against popular Gaussian-basse baselines 17 21 13
across diverse 3D scene datasets [I8 3 12. As highlighted, GSDF excels modeling delicate geometries (Ist & 2nd rows) and handling texture-less and sparsely observed regions (3rd & 4th rows), which are commonly presented larger scenes where baseline approaches struggle address.
Enchanced Detail and Fidelity. Notably, method excelled in achieving high rendering quality in texture-less areas, as showcased Fig which aligned with our design motivation for geometry- aware density control. Specifically, in texture-less areas where vanilla 3D Gaussians struggled due to small accumulated gradients, our method overcame this limitation by growing anchors in surface regions which brings enhanced accuracy and scene details.
Robustness with Noisy Gaussians. To test robustness, we experimented with randomly initialized Gaussian primitives on 2D-GS, Scaffold-GS. and our method. Quantitative results in Table hhighlght the advantages of our geometric guidance, demonstrating superior performance and stability even with random input. Further visual results are provided in the supplementary material.
7

---

Instant-NSR
SuGaR
GSDF
FRRE
FEEE
RELBB
BEERE
RUEBR
Figure 4: Reconstruction Comparison. We visualize the reconstructed meshes from Instant- NSR (our SDF-branch), SuGaR [10), 2D-GS (13), and Ours.
4.2.2 Reconstruction Comparisons
We quantitatively evaluated the reconstruction ability the DTU dataset, where method achieved the best Chamfer distance as shown in Table
Enhanced Geometry Accuracy and Completeness. As illustrated in Fig our method reconstructed more complete and detailed meshes compared to baseline methods. Our approach effectively bypassed local minima, preventing holes in the meshes. Notably, method outperformed Instant-NSR, delivering smooth, continuous meshes with weel-preserved high- high-frequency details. In contrast, meshes extracted from SuGaR [00] were non-manifold with broken topological relationships. 2D- GS 13 extracts mesh from fused depth, which leads to over-smoote geometry.
Accerelated Convergence and Rendering. Benefited from the GS-branch, our method optimized SDF field significantly faster in terms training iterations than previous methods with accurate sample placements, which often required slow and exhaustive training and rendering. For instance, NeuS [31] required about hours of optimization on a single GPU for the DTU dataset, whereas our method achieved comparable or better results within just 2 hours on single GPU,

---

Table 3: Quantitative Results on Ablation Studies. We separately listed the rendering metrics for
each ablation described in Sec.
Datasel
Mip NeRF360
Tanks& Tanks&Temmles [8]
Deep Blending 12]
Metrics
PSNR 1
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR 1
SSIM
LPIPS
GSDF (Pull)
29.38
0.865
0.185
27.37
0.875
0.156
30.38
0.909
0.223
w/o geometric supervision
29.27
0.862
0.196
27.18
0.865
0.191
30.21
0.907
0.234
depth guided sampling
0.196
0.159
0,908
0.228
w/o geometry- aware densification
29.29
0.863
0.196
27.29
0.870
0.179
30.25
0.908
0.236
GSDF (ull)
depth-guidde
we density control
mutual superssiins
Surface
Rerseeiing
Figure 5: Ablation results. Visualizations of reconstructed meshes and rendered images from 1) our full method, 2) ours w/o depth-guided ray sampling, 3) ours w/o geometry-aware density control, and 4) ours w/o geometric supervision. We highlight the degradation of quality using numbered patches.
4.3 Ablation Studies
In this section, we examine the effectiveness of each individual module. Quantitative metrics and qualitative visualizations are provided in Table 3 and Fig.
Depth-Guided Ray Sampling. To evaluate the effectiveness of our depth-guided ray sampling
detailed in See 3.2 we conducted an ablation on depth-guided ray sampling using the original stratified ray sampling approach (33 The results show that this ablated setting produced overly smoothed surfaces, failing to capture finer geometry details such as the heater and doors (Fig. 5 column 2, patches & 3). The resulting less accurate SDF also impacted rendering quality, making details like the sticker on the wall appear more blurred compared to our full model.
Geometry-awee aware Gaussian Density Control. We replaced geometry-awarr Gaussian density control with the pruning and growing strategy from Scaffold-GS [21] to evaluate its efficacy. As shown in Fig. 5 (3rd column), this change led to missed details like the sticker on the wall (patch 4) due to small accumulated gradients on the texture-less surface. The reconstruction results also showed the absence of thin objects such table legs and the chandelier (patch 1), highlighting the limitations of the vanilla strategy.
Mutual Geometric Supervision. Lastly, ablated the proposed mutual geometric supervision by setting both A and A, to (Eq.7). This resulted a significant decay surface quality and omission of details rendering. These findings indicate that neural rendering is more tolerant of deviations than neural surface reconstruction. Without explicitly aligning Gaussians with SDF-derived geometry during optimization, the branches diverge, leading to sub-optimal results for both.
4.4 Limitations and Future Works
Currently, our method is not tailored to handle challenging scenes with reflections and intense lighting variations, such as those found in indoor environments. However, we have observed that employing more structured and surface-aligned Gaussian primitives holds promise in capturing such view-dependent appearance changes with improved scene geometry. Our framework requires more memory usage because it simultaneously considers two representations. Furthermore, the performance our SDF-branch significantly lags behind the GS-branch, leading extended training durations compared to Scaffold-GS [21 only. Hence, improving the efficiency of the MLP-base

---

SDF-branch is a crucial direction for future research. Still, training time is not the highest priority compared to inference, where primitives are stored and can be accessed efficiently.
Conclusion
In this work, we introduced dual-branch framework that leverages the strengths of both 3D-GS and SDF, showcasing its potential to achieve both enhanced rendering and reconstruction quality. The inherent differences in two representations, rendering approaches, and supervision loss pose challenge to the seamless integration. We therefore integrate a bidirectional mutual guidance approach during the training to circumvent these restrictions. Three types of guidance have been introduced and validated in our framework, namely: 1) depth guided sampling (GS 2) geometry- aware Gaussian density control (SDF++S); and 3) mutual geometry supervision (GS+SDFF) Our extensive results demonstrate the efficiency and joint performance improvement on both tasks. As the two branches maintain their original architectures, we keep their efficiency during inference, allowing room for potential enhancements by substituting each branch with more advanced models in the future. We envision our model benefiting applications demanding high-quality rendering and geometry, including embodied environments, physical simulation, and immersive VR experiences.
6 Acknowledgements
This work is funded in part by the National Key R&D Program of China (2022ZD0160201), and Shanghai Artificial Intelligence Laboratory.
References
[1] Henrik Aanæs, Rasmus Ramsbøl Jensen, George Vogiatzis, Engin Tola, and Anders Bjorholm Dahl. Large-scall data for multiple-veew stereopsis. International Journal of Computer Vision, pages 1-16, 2016.
[2] Kara-Ali Aliev, Artem Sevastopolsky. Maria Kolos, Dmitry Ulyanov, and Victor Lempitsky. Neural point-based graphics. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXII 16, pages 696-712. Springer, 2020.
[3] Jonathan Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip- nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5470-5479, 2022.
[4] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. TensoRF: Tensorial radiance fields. 2022.
[5] Hanlin Chen, Chen Li, and Gim Hee Lee. Neusg: Neural implicit surface reconstruction with 3d gaussian splatting guidance. arXiv preprint arXiv:2312.00846 2023.
[6] Kai Cheng, Xiaoxiao Long, Kaizhi Yang, Yao Yao, Wei Yin, Yuexin Ma, Wenping Wang, and Xuejin Chen. Gaussianpro: 3d gaussian splatting with progressive propagation. arXiv preprint arXiv:, 2024.
[7] Pinxuan Dai, Jiamin Xu, Wenxiang Xie, Xinguo Liu, Huamin Wang, and Weiwei Xu. High- quality surface reconstruction using gaussian surfels. 2024.
[8] Yutao Feng. Xiang Feng. Yintong Shang, Ying Jiang, Chang Yu, Zeshun Zong, Tianjia Shao, Hongzhi Wu, Kun Zhou, Chenfanfu Jiang, et al. Gaussian splashing: Dynamic fluid synthesis with gaussian splatting, arXiv preprint arXiv:2401.15318 2024.
[9] Sara Fridovich-Ken Keil, Alex Yu. Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Pienoxels: Radiance fields without neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5501-5510. 10, 2022.
[10] Antoine Guédon and Vincent Lepetit. Sugar: Surface-aligned gaussian splatting for efficient 3d mesh reconstruction and high-quality mesh rendering, arXiv preprint arXiv:2311.12775. 2023.
10

---

[11] Yuan-Chen
Guo.
Instant
neural
surface
reconstruction,
2022.
[12] Peter Hedman, Julien Philip, True Price, Jan- Machhel Frahm, George Drettakis, and Gabriel Brostow. Deep blending for vieepoin image-based rendering. 37(6):257:1-257777 2018.
[13] Binbin Huang, Zehao Yu, Anpei Chen, Andreas Geiger, and Shenghua Gao. 2d gaussian splatting for geometrically accurate radiance fields. ArXiv, abs/2403. 17888, 2024.
[14) Rasmus Jensen, Anders Dahl, George Vogiatzis, Engin Tola, and Henrik Aanæs. Large scale vulti-view stereopsis evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 406-413, 2014.
[15] Ying Jiang, Chang Yu, Tianyi Xie, Xuan Li, Yutao Feng, Huamin Wang, Minchen Li, Henry Lau, Feng Gao, Yin Yang, et al. Vr-gs: A physical dynamics- interactive gaussian splatting system in virtual reality. arXiv preprint 0rXiv:2401.16663 2024.
[16] Nikhil Keetha, Jay Karhade, Krishna Murthy Jatavallabhula, Gengshan Yang, Sebastian Scherer, Deva Ramanan, and Jonathon Luiten. Splatam: Splat, track & map 3d gaussians for dense rgb-d slam. arXiv preprint 23Xiv:2312.02111 2023.
[17] Bernhard Kerbl, Georgios Kopanas, Thomas Leimküüher, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions Graphics, 42(4), 2023.
[18] Arno Knapitsch, Jacsik Park, Qian- Yi Zhou, and Vladlen Koltun. Tanks and temples: Bench- marking large-scale scene reconstruction. ACM Transactions Graphics, 36(4), 2017.
[19] Zhaoshuo Li, Thomas Müller, Alex Evans, Russell H Taylor, Mathias Unberath, Ming- Yu Liu, and Chen-Hsuan Lin. Neuralangelo: High-fidelity neural surface reconstruction. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 8456-8465, 2023.
[20] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. Advances Neural Information Processing Systems, 33:15651-15665 2020.
[21] Tao Lu, Mulin Yu, Linning Xu, Yuanbo Xiangli, Limin Wang, Dahua Lin, and Bo Dai. Scaffold- gs: Structured 3d gaussians for view-adaptiv rendering. 2023.
[22] Xiaoyang Lyu, Yang- Tian Sun, Yi-Hua Huang, Xiuzhe Wu, Ziyi Yang, Yilun Chen, Jiangmiao Pang, and Xiaojuan Qi. 3dgsr: Implicit surface reconstruction with 3d gaussian splatting. arXiv preprint arXiv:2404.00409, 2024.
[23] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[24] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. ACM Transactions on Graphics (ToG), 41(4):1- 15, 2022.
[25] Michael Oechsle, Songyou Peng, and Andreas Geiger. Unisurf:
Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages $589-5599, 2021.
[26] Christian Reiser, Stephan Garbin, Pratul P Srinivasan, Dor Verbin, Richard Szeliski, Ben Mildenhall, Jonathan Barron, Peter Hedman, and Andreas Geiger. Binary opacity grids: Capturing fine geometric detail for mesh-based view synthesis. arXiv preprint 223. 2024.
[27] Antoni Rosinol, John J Leonard, and Luca Carlone. Nerf-slam: Real-time dense monocular slam with neural radiance fields. In 2023 IFFF/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3437-3444. IEEE, 2023.

---

[28] Radu Alexandru Rosu and Sven Behnke. Permutosdf: Fast multi-view reconstruction with implicit surfaces using permutohedral lattices. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8466-8475 2023.
[29] Johannes L Schonberger and Jan-Michael Frahm. cooiuue-frommmotiiinn revisited. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 4104 4113, 2016.
[30] Haithem Turki, Vasu Agrawal, Samuel Rota Bulè, Lorenzo Porzi, Peter Kontschieder, Deva Ramanan, Michael Zollhöfer, and Christian Richardt. Hybridnerf: Efficient neural rendering via adaptive volumetric surfaces. arXiv preprint arXiv:2312.03100 2023.
[31] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2 10..210000.089 2021.
[32] Yiming Wang. Qin Han, Mare Habermann, Kostas Daniilidis, Christian Theobalt, and Lingjie Liu. Neus2: Fast learning of neural implicit surfaces for multi-view reconstruction. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 3272-3283, 2022.
[33] Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions Image Processing, 13(4):600-612 2004.
[34] Zian Wang, Tianchang Shen, Merlin Nimier-David, Nicholas Sharp, Jun Gao, Alexander Keller, Sanja Fidler, Thomas Müller, and Zan Gojeic. Adaptive shells for efficient neural radiance field rendering. arXiv preprint vrXiv2231..1009) 2023.
[35] Tianyi Xie, Zeshun Zong, Yuxin Qiu, Xuan Li, Yutao Feng, Yin Yang, and Chenfanfu Jiang. Physgaussian: Physics-integrater 3d gaussians for generative dynamics. arXiv preprint 1rX1:23311./2198, 2023.
[36] Linning Xu, Vasu Agrawal, William Laney, Tony Garcia, Aayush Bansal, Changil Kim, Samuel Rota Buld, Lorenzo Porzi, Peter Kontschieder, Aljaž Božič, et al. Vr-nerf: High-fidelity virtualized walkable spaces. In SIGGRAPH Asia 2023 Conference Papers, pages 1-12, 2023.
[37] Linning Xu, Yuanbo Xiangli, Sida Peng, Xingang Pan, Nanxuan Zhao, Christian Theobalt, Bo Dai, and Dahua Lin. Grid-guided neural radiance fields for large urban scenes. 2023.
[38] Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann. Point-nerf: Point-based neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5438 0448,448445 2022.
[39] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems, 34:4805-453 -8155 2021.
[40] Lior Yariv, Peter Hedman, Christian Reiser, Dor Verbin, Pratul P Srinivasan, Richard Szeliski, Jonathan T Barron, and Ben Mildenhall. Bakedsdf: Meshing neural sdfs for real-time view synthesis. arXiv preprint ar0iv:2302. /4859, 2023.
[41] Wang Yifan, Felice Serena, Shihao Wu, Cengiz Öztireli, and Olga Sorkine- Differen- tiable surface splatting for point-based geometry processing. ACM Transactions on Graphics (TOG), 38(6):1-14, 2019.
[42] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2018.
[43] Matthias Zwicker, Hanspeter Pfister, Jeroen Van Baar, and Markus Gross. Ewa volume splatting. In Proceedings Visualization, 2001. VIS'O1., pages 29-538. IEEE, 2001.
12

---

A Supplementary Material
The following sections are organized follows: 1) The first section elaborates on the implementa- tion details of GSDF, covering hyper-parameters and curvature loss. 2) We then show additional experimental results. 3) Lastly, we also delve into limitations and future directions. More results and video demos can be found in supplementary webpage.
A.1
Implementation details
A.1.1
Contigurations.
Below, we enumerate the hyper-parameters used in our experiments:
The variance x² for the Gaussian function in Eq. and 4 is set to 0.005.
For the rendering loss discussed in Sec. 3.3 we set X = 0.2 and Awot = 0.01, in consistency with the configurations specified in Scaffold-GS [21].
For the SDF-branch, we set λesk 0.1 and implement an adaptive scheme for Xeare- Specifically, Acury increases linearly from to (0.5 on DTU) over the first 2000 iterations, after which it remains at 0.05 (0.01 DTU) for subsequent iterations. This strategy is based on our observation that increasing the weight of the curvature loss significantly encourages the convergence of the SDF to geometric outline.
For the mutual geometry loss, we typically assign Xa 0.5 and An 0.01.
A.1.2 Curvature Loss.
While Instant-NSR [ derives the curvature loss from a discrete Laplacian, we follow a more robust and explicit method as described in PermutoSDF [28]. For any given point, we randomly perturb it within the tangent plane orthogonal to normal. The curvature loss is then measured by the cosine similarity of normals between the original point and its perturbed counterpart. Note that, we applied the same curvature loss for both Instant-NSR and GSDE.
A.2
More experiments
A.2.1 Reconstruction with 500k iterations.
As discussed in Sec. 4 optimizing the SDF is a time-intensive process. While our GS- 6S-branch assists in the convergence of the SDF-branch, we conjecture that further improvements could be made through additional iterations. Hence, we train our method for 500k: iterations. As shown in Fig 6 our method achieves superior reconstruction quality compared Instant-NSR (our SDF-branch) with the same training iterations.
A.2.2 Rendering comparison with random initialization.
In Sec. 3.2 we discussed how the SDF-branch can improve the rendering quality of the GS-branch. This enhancement is particularly noticeable when Gaussian primitives are randomly initialized, as shown in Fig. and Fig.
A.2.3 Per-scene Results.
We list the per-scene quality metrics (PSNR, SSIM, LPIPS, and Chamfer Distance) used in rendering and reconstruction evaluation Sec. 4 for all considered methods, as shown from Tab, to
Tab. 6
13

---

Table 4: Quantitative results for DTU scenes M. Note that we tried our best, but NeuS 31 failed to reconstruct geometries in scenes 69 and 105 with ] training images.
CD
24
37
55
6.3
63
69
83
977
105
106
110
114
118
122
NeuS [3]
0.828
0.939
1.120
0.421
1.061
0.623
1.474
1,180
0.555
1.218
0.340
0.462
0.480
NSR 11
1.296
6.582
1,459
2.361
1.535
0.897
0.736
2.580
2,803
3D-GS [7
4.877
5.720
4.738
3.133
7.119
3,713
3.486
3,830
4.294
4.323
3,218
3.418
3.648
2.445
2,550
Scaffold-GS 21
7:234
6.234
7,436
8.173
4.266
5.779
5.451
6.565
6.358
5.049
5.953
6:317
5.615
2.899
2D-GS 03
0.662
1.108
0.707
0.430
1.307
1,095
0.901
1,385
1.257
0.823
0.754
1.419
0.484
0.974
0.606
SuGaR
10
1.166
1.075
0.958
0.471
2. 187
1.638
1.022
1.812
1.505
0.930
0.672
2.686
0.701
0.943
0.819
GSDF
0,588
0.966
0.460
1,300
1.589
0.588
0.518
0.515
PSNR1
24
37
40
55
63
65
69
83
97
105
106
110
114
118
122
NeuS [1]
26.96
26.08
26.86
89
33.76
32.90
1.
28. 94
33.92
52.09
29.62
35.77
37.19
Instant-NSR
11
21.64
21.24
23.27
19.10
30.40
27.25
25.07
32.87
24.32
29.71
28.90
27.39
24.96
29.57
29.44
3D-GS [7]
27.33
75
31.78
34.66
36,83
37.78
Scaffold-GS GS 21
30.59
27.85
29.65
32.33
29.27
33.22
29.43
38.96
32.04
36.60
36.38
34.05
31.07
36.67
37.73
2D-GS 1
30.31
27.90
29.02
32.79
35.52
33.12
29.58
37.01
32.32
36,50
36.25
33.47
30.87
57.30
38.76
SuGaR 10
26.94
28.10
34.72
31.01
36.21
30.31
37.12
38.
GSDF
30.16
27.65
28.92
90
35.46
33.73
29:46
39:32
32.28
36.81
36.47
34.00
30.99
37.57
39:17
SSIM f
24
55
6.3
83
97
105
106
110
114
118
122
NeuS 33
0.780
0.816
0.749
0.884
0.950
0.962
0.972
0.922
0.924
0.939
0.909
0.953
0.964
Instant- NSR 1I
0.689
0.754
0.670
0.725
0.938
0.928
0.861
0.957
0.857
0.898
0.871
0.912
0.833
0.897
0.913
3D-GS 17
0.915
0.897
0.887
0.951
0.964
0.959
0.933
0.976
0.943
0.957
0.952
0.952
0.934
0.962
0.971
Scaffold-GS
21
0.914
0,906
0.902
0.956
0.48
0.962
0.936
0.976
0.945
0.957
0.957
0.958
0.338
0.965
0.972
0.920
0.911
0.958
0.961
0.938
0.970
0.957
0.964
SuGaR
0.901
0.903
0.889
0.955
0.956
0.955
0.932
0.974
0.939
0.952
0.958
0.947
0.935
0.965
0,973
GSDF
0.916
0,905
0.902
0.959
0.965
0,963
0,938
0.978
0.947
0.957
0.959
0.960
0.939
0.967
0.975
LPIPS 4
24
37
40
55
63
65
69
83
97
105
106
110
114
118
122
Neu8 [3]
0.780
0.816
0.749
0.884
0.950
0.962
0.972
0.922
0.924
0.939
0.909
0.953
0.964
Instant-NSBR 11
0.689
0.754
0.670
0.725
0.938
0.928
0.861
0.957
0.857
0.898
0.871
0.912
0.833
0.897
0.913
3D-GS [7]
0.915
0.897
0.887
0.951
0.964
0,959
0.935
0.976
0.943
0.957
0.952
0.952
0,934
0.962
0.971
Scaffold
21
0.914
0.956
0.948
0.936
0.976
0.957
0.957
0.965
0.972
2D-GS
11
0.920
0,911
0.903
0.958
0.964
0.962
0.938
0.970
0.947
0.954
0.957
0.954
0.999
0.964
0.973
SuGaR
10
0.901
0.903
0.889
0.955
0.956
0,955
0.932
0.974
0.939
0.952
0.958
0.947
0.955
0.965
0.973
GSDF
0.916
0.905
0.965
0.938
0.978
0.957
0.959
0.967
0.975
Table 5: Quantitative results for MiR-NeRF 360 scenes
Outdoor
bicycle SSIM f
garden SSIM f
stump SSIM
PSNR 1
LPIPS |
PSNR
LPIPS
PSNR
LPIPS
Instant-NSt -NRR
11
19.68
0.449
0.615
19.74
0.422
0.627
18.77
0.486
0.592
3D-GS 17
24.46
0.707
0.313
26.63
0.819
0.175
26.31
0.758
0.309
Scaffold-SS
21
24.63
0.721
0,289
26.59
0.814
0.182
26.58
0.765
0.302
Scaffold-GS(ranc))
23.48
0.633
0.397
26.09
0.776
0.239
22.88
0.652
0.428
2D-GS [33
23.96
0.683
0.358
26.07
0.803
0.211
25.69
0.746
0.349
2raGd(rand)
22.65
0.578
0.455
24.98
0.734
0.298
22.51
0.637
0.471
SuGaR 110
23,26
0.630
0.356
25.44
0.763
0.239
25.12
0.705
0.324
GSDF
24.61
0.724
0.261
26.91
0.824
0.158
26.04
0.757
0.290
GSDF (rand)
24.09
0.682
0.328
26.25
0.798
0.196
21.92
0.625
0.416
Iudoor
PSNR
SSIM 1
I.PIPS
PSNR
SSIM 1
LPIPS
PSNR
kitchen SSIM
LPIPS J
SSIM
bonsai
Instant-NSR
23.18
0.769
0.439
23:00
0.723
0.459
23.87
0.685
0.400
24.84
0.753
0.430
3D-GS [7
31.68
0.925
0.194
29.25
0.914
0.180
31.69
0.933
0.113
32.19
0.915
0.176
ScaTold-SS 21
32.45
0.933
31.99
0.934
0.112
33.24
0.951
0.165
ScalTol--GSSamml
31.84
0.920
0.205
29.00
0.888
0.206
30.50
0.914
0.144
31.08
0.928
B.196
2D-GS [3]
30.77
0.913
0.219
28.19
0.898
0.212
30.50
0.922
0.133
31.39
0.934
0.201
2D-GS(ramd)
30.01
0.895
0.256
27.78
0.874
0.248
30.10
0.914
0.147
30.56
0.922
0.222
SatiaR [o
0.179
0.219
GSDF
32:46
0.937
0.165
30.11
0.923
0.159
31.93
0.936
0.106
33.60
0.954
0.155
GSDF (rand)
32.01
0.929
0.181
29.17
0.909
0.183
31.15
0.929
0.118
31.43
0.936
0,182
14

---

Table 6: Quantitative results for Tanks& Temples 18 and Deep Blending
Scenes.
Dutiset
Tamks&Taample
Deep Blending
Method & Metrics
Barn SSIM
Truck SSIM
Dr Johnson
Playroom SSIM
PSNR
LPIPS
PSNR
LPIPS
PSNR
SSIM
LPIPS
PSNR
LPIPS
303-38 17
28.21
0,835
0.223
25.24
0.861
0.197
29.04
0.897
0.248
29.84
0.900
0.248
Scaffold-GS GS 21
28.77
0.869
0.192
25.80
0.869
0.172
29.86
0.908
0.236
30.89
0.908
0.239
Scaffeld-GS GS (rand)
27.82
0.832
0.243
24.93
0.345
0.217
28.58
0.892
0.262
29.66
0.898
0.258
2D-G5 (rand)
0.810
0.288
23.81
0.824
0.262
28.13
0.839
0.276
29.10
0.895
0.273
SuGaR [T]
26.67
0.837
0.233
22.69
0.817
0.227
28.77
0.889
0.271
30.30
0,900
0.258
GSDP (rand)
28.93
0.877
0.176
25.81
0.873
0.135
29.87
0,909
0.225
30.89
0,909
0.222
Instant-NSR
GSDF
500k
500k
500k
Figure 6: Reconstruction comparison with 500k training iterations.
15

---

GT
Scaffold-Ge GS (rand)
GSDF (rand)
Figure 7: Rendering Comparison with random initialization (Part 1). We compare rendering results between Scaffol--GS (1) and our method. The highlighted patches indicate that our method superior in expressing finer details in both geometry and appearance.
16

---

GT
Scaffold-G5 (rand)
GSDF (rand)
K H
K H
KX
KH
Figure 8: Rendering Comparison with random initialization (Part 2)- Cont.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We introduce GSDF, dual-branch structure combining the strengths 3D-GS and SDF, for improving both rendering and reconstruction. Please see abstract and introduction for further details.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Our framework still struggles with the dilemma like most SDF-based recon-
struction methods. Please see the limitation section (Sec. 4.4) for further details.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
18

---

Answer: [Yes]
Justification: In the method and experiment sections, we provide a detailed discussion and substantial evidence to demonstrate each result.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversell, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: GSDF is robust framework that comprises GS-branch dedicated rendering and an SDF-branch focusing on learning neural surfaces. By leveraging the mutual guidance between these two branches, our method achieves state-of-the- performance on both rendering and reconstruction tasks. Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
19

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification:
We will release the code after acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /nips.cc/public. wwwpppppc/gguuaaeecceeeoeoooooee /ttpgdddddd/ooeSuumiiieee Ctt///oooeemisiiieeeaee www.bbmiiiiioooccccccc for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In the Experimental Setup section (Sec. 4.1 and Implementation details section (Sec. I. we provide comprehensive details, including the data splits and hyperparameters used in experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The experimental result for redering and reconstruction tasks from dense views are very stable, the standard deviation is negligible, so we follow the previous work's 's practice and report the error bar.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
20

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Experiment section (Sec. 4.. We provide sufficient information on the computer resources, including the type of compute workers GPU and training time.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We have carefully reviewed the NeurIPS Code Ethics and are committed to strictly adhering to its guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Neural rendering and surface reconstruction are crucial tasks with important downstream applications in areas like robotics, physical simulations, and augmented reality, We believe our GSDF system has the potential to advance progress this domain and support the development of these applications.
21

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
There is no such risks in the rendering and reconstruction tasks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: In the Experimental Setup section (Sec. 4.1 and public datasets.
we cite each used code package
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
22

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, papprsaipphaeeeooccaaapppaapppse thcode. com/ com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification:
We plan to release the code upon acceptance of the paper.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve any crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve any crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
23

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "ETnQlVWuBZ", "reviewer": "Reviewer_96yp", "summary": "This paper introduces GSDF that utilizes a joint optimization of the GS-branch and SDF-branch to constrain the inherent geometric issues of the 3DGS. Furthermore, it proposes three mutual guidances to ensure satisfactory outcomes in both rendering and reconstruction. The extensive experiments on datasets such as DTU, Tanks and Temples (T&T), and Mip-NeRF 360 demonstrate that the method can achieve high rendering quality while obtaining better geometric results.", "questions": "please refer to weaknesses section.", "limitations": "The authors didn't explicitly address the limitation of their approach.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The article is clearly written, allowing one to easily understand the motivation in designing the two branches.\n2. The author conducted extensive experiments across various datasets to illustrate the high quality of rendering and geometric results achieved by GSDF.\n3. Utilizing the SDF-branch to supervise the inaccurate geometry or depth of 3DGS seems to be reasonable.", "weaknesses": "1. Instead of sampling relying on predicted SDF, GSDF samples near the depth from GS branch. However, I guess the imprecise depth of 3DGS could infulence the SDF-branch. I am curious whether there is a difference between the results of the SDF-branch and the baseline's SDF-branch results? \n2. Which branch outputs the final result of GSDF method? If the GS branch outputs both rendering and gemetry results, how about the rendering and geometry outputs of another branch? \n3. It seems that the time-consuming aspect is missing in Table 1 or Table 2, and the quantitative result of NeuS is missing in Figure 4. Additionally, as reported on lines 248-249, the time consumption of GSDF (2 hours) is significantly higher compared to 2DGS (5-10 minutes) and the original 3DGS. This substantial gap may be attributed to the volumetric rendering of the SDF branch; thus, the improved geometry results might be benefiting from this branch. It may be necessary to compare it with some neural implicit reconstruction methods (like Neuralangelo) to better evaluate its performance given the additional computational cost.\n4. In Table 3 of the ablation study, if depth-guided sampling is removed (w/o depth-guided sampling), one would need to consider whether the SDF-branch is now completely identical to the baseline NeuS. At this point, it would be important to assess if the results of the SDF-branch are close to baseline of SDF-branch. Furthermore, as mentioned in Lines 248-249, one should investigate whether the time consumption has significantly increased due to the absence of depth-guided sampling, given that this might affect the efficiency of the volumetric rendering performed by the SDF-branch.\n5. The result in table1, the CD results of NeuS and 2DGS is not as good as in their papers. And there might be some ambiguity in the symbolic expressions presented in the text. e.g. is line154 Fsdf the same with Line139 Fs?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your efforts and valuable comments. Below we address concerns for each question. Common concerns are detailedly responded in the global rebuttal. Additional Figures and Tables are provided in the attached PDF, in which the index is denoted as *Figure A-D* and *Table A-B*.\n\n**Q1. Imprecise depth; SDF-branch vs. Baseline's SDF-branch.**\n\n**A1**: As we described in the paper (L124-133), the GS-branch is effective in locating the sampling area while accurate depth is not a strict requirement. The depth guided sampling uses the position of depth point's SDF value as the interval of sampling, considering both the depth from the GS-branch and the predicted SDF from the SDF-branch. Additionally, to enhance robustness, we warm up the GS-branch to provide a coarse depth. Extensive experiments demonstrate that GSDF remains robust even if the depth from the GS branch is not precise.\n\nThere are noticeable differences between the results of SDF-branch and the baseline's SDF-branch. In Figure 4 of the main paper, the first column is the results of the baseline's SDF-branch and the last column shows the results of our SDF-branch.\n\n**Q2. Each branch's output.**\n\n**A2**: Note that, our framework features a two-branch design, where each branch excels in its respective task while benefiting from mutual supervision: the SDF-branch specializes in accurate geometry reconstruction, while the GS-branch focuses on high-quality image rendering. This mechanism enables our approach to achieve superior results in both individual methods (L24-27).\n\n**Q3. Time consuming, comparison with NeuS and Neuralangelo.**\n\n**A3**: We did not include NeuS results in Figure 4 of the main paper because NeuS struggles with reconstructing complex scenes. We present several object-level cases in *Figure C*, showing GSDF consistently outperforming NeuS.\nWe included the time consumption for GSDF and single-branch methods in *Table A*. Although GSDF is slower per iteration, it achieves faster SDF convergence compared to the SDF branch alone in terms of both iteration and training time (see *Figure A*). Importantly, GSDF yields significantly better quality results.\n\nFollowing the reviewer's suggestion, we compared GSDF with Neuralangelo. Neuralangelo requires over 12 hours on 2 GPUs and produces inferior results compared to GSDF, as shown in *Figure D*. Additionally, the actual time for 2DGS in real scenes is longer than 5-10 minutes, and its rendering quality is degraded, as shown in *Tables A* and *B*.\n\n**Q4. Ablation of the sampling process.**\n\n**A4**: In Table 3 of the main paper, when depth-guided sampling is removed, we switch to SDF-guided sampling, as used in most NeuS-based reconstruction methods. We compared the SDF convergence speed between GSDF and our SDF branch alone. As shown in Figure A, GSDF achieves faster SDF convergence in both training iterations and time. Specifically, following [1], we use kernel size as an indicator of reconstruction quality, where a smaller kernel size indicates better geometry. Our method consistently achieves better results with faster convergence.\n\n*Reference*: [1] Wang Z, Shen T, Nimier-David M, et al. Adaptive shells for efficient neural radiance field rendering. ACM Trans. Graph., 42, 2023.\n\n**Q5. The configuration of the reported report and notation.**\n\n**A5**: Since our method is not purely a reconstruction method, we split the dataset into training and test sets, with 1 test view for every 8 views. This differs from the default settings in NeuS and 2DGS, which used all images for training. Therefore, we used their officially released code to train the models with our settings.\n\nRegarding the notation, F_sdf and F_s are indeed the same MLP. We will clarify this in the revised version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4s0BH4oz7f", "reviewer": "Reviewer_yTkf", "summary": "This paper tackles the challenge of representing 3D scenes from multiview images by introducing a novel dual-branch architecture named GSDF, which combines 3D Gaussian Splatting and neural Signed Distance Fields. This architecture enhances both rendering and reconstruction through mutual guidance and joint supervision. By aligning Gaussian primitives with potential surfaces and speeding up SDF convergence, the method achieves finer geometric reconstructions and minimizes rendering artifacts. Demonstrating robustness and accuracy, the approach is effective in both synthetic and real-world scenarios.", "questions": "1. The rendering and reconstruction results mentioned in the paper are derived from which branch? Are both rendering and reconstruction results obtained from the 3DGS branch, or is rendering performed by the 3DGS branch while mesh is obtained from the Instant-NSR branch?\n2. Can using pseudo-depth and normals (obtained from other estimation algorithms) directly enhance Instant-NSR or Gaussian, potentially achieving good reconstruction and rendering results without the need for bidirectional optimization?\n3. Which branch's rendering results are displayed in the ablation study? If it is the Instant-NSR branch, then different strategies for adding and removing points should not affect its rendering results. Similarly, if it is the 3DGS branch, should depth-guided sampling also not affect its rendering outcomes?", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper employs a dual-branch approach for simultaneous scene rendering and mesh reconstruction. It utilizes an SDF (Signed Distance Field) branch to guide the geometric optimization of the Gaussian branch, including operations such as adding and removing points. By leveraging bidirectional optimization, the method simultaneously enhances the reconstruction and rendering quality of both branches. This approach ensures high rendering quality while effectively reconstructing the scene's mesh, achieving smooth reconstruction results on both object-level and scene-level datasets. Additionally, the method has also demonstrated commendable performance in novel view synthesis tasks.", "weaknesses": "It appears that in the reconstruction and rendering branches, the authors have merely pieced together appropriate solutions. Although bidirectional optimization has been applied to both branches, the unity of the method seems lacking. The interrelation between the methods is weak, resembling more of an incremental improvement rather than a cohesive, integrated advancement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your efforts and valuable comments. Below we address concerns for each question. Common concerns are detailedly responded in the global rebuttal. Additional Figures and Tables are provided in the attached PDF, in which the index is denoted as *Figure A-D* and *Table A-B*.\n\n**Q1. Output of each branch.**\n\n**A1**: In our framework, we use the SDF-branch to reconstruct accurate geometry and the GS-branch to render the images.  Please refer to the common responses for a detailed explanation.\n\n**Q2. Pseudo-depth and normals from other estimation algorithms.**\n\n**A2**: Using pseudo-depth and normals from other estimation algorithms is possible. However, it would make the method highly dependent on the quality of these algorithms. In contrast, our framework allows the two branches to mutually promote each other, resulting in a more robust approach.\n\n\n**Q3. Ablations on sampling process.**\n\n**A3**: The rendering results in the ablation study are from the GS branch, and the reconstruction results are from the SDF branch. 'Depth-guided sampling' and other operations affect both branches due to the 'Mutual Geometry Supervision' mechanism.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vAwBFi04cu", "reviewer": "Reviewer_LpsC", "summary": "This paper proposes to jointly optimize 3D Gaussian Splatting (3DGS) and SDF (like NeuS). During GS optimization, the Gaussians are aligned to the zero-level set (and normals) of SDF. During the NeuS-like optimization, Gaussians are used to limit the range of ray sampling, resulting in efficient optimization. Experiments show that the proposed method achieves better reconstruction and rendering compared with SDF-based methods (e.g., NeuS) and GS-based methods, including recent SuGaR and 2D-GS, which explicitly align Gaussians to surfaces.", "questions": "- Although I may miss some information, an interesting potential merit of the proposed method compared to SDF-based methods is the efficient ray sampling, while the overall optimization should take longer than vanilla 3DGS. I would like to see the discussions on training time compared to those methods.", "limitations": "I did not find notable negative social impacts.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ This paper proposes a nice combination of 3DGS and SDF (but not the first to do, which I'll detail in the weakness section). Both Gaussians and SDF representations provide merit to each other, resulting in accurate reconstruction and rendering.\n+ Experiments are well done. Very recent methods, like 2D Gaussian splatting and SuGaR are adequately evaluated, showing the merit of doing joint optimization of Gaussians and SDF, not just by aligning Gaussians on the surfaces.", "weaknesses": "## Related work\nThis method is not the first one combining the 3DGS and SDF, as mentioned in related work (NeuSG [5]). Regarding CfP of NeurIPS, \"papers that appeared online within two months of a submission will generally be considered \"contemporaneous\" in the sense that the submission will not be rejected on the basis of the comparison to contemporaneous work.\"\n\nIn other words, NeuSG, which was submitted on Dec 1, 2023, in arXiv, has to be regarded as an \"official\" existing work. Thus, the paper should discuss more the technical differences from the NeuSG work. I basically agree with the argument in L81-86 that mentions the subjective merit of the proposed method compared to NeuSG; the proposed method does significantly more for tighter integration of 3DGS and SDF.\n\nMeanwhile, the readers should wonder if the proposed method achieves better than NeuSG or not. Unfortunately, the code of NeuSG does not seem available, and an official comparison may be difficult. I did not come across the NeuSG paper deeply and may mention something wrong, but I'm wondering if the authors may show a quick ablation study using a baseline that somewhat mimics a simplified version of NeuSG by only using normal supervision by SDF and force the Gaussians flat.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your efforts and valuable comments. Below we address concerns for each question. Common concerns are detailedly responded in the global rebuttal. Additional Figures and Tables are provided in the attached PDF, in which the index is denoted as *Figure A-D* and *Table A-B*.\n\n**Q1. Comparison to NeuSG.**\n\n**A1**: Please refer to the common responses for a detailed explanation. \n\nFollowing the reviewer's recommendation (\"by only using normal supervision by SDF and force the Gaussians flat\"), we implemented the NeuSG. *Figure D* shows that GSDF achieves better reconstruction than NeuSG. Additionally, NeuSG's rendering quality decreased from 28.77 to 28.63 PSNR, while GSDF increased PSNR to 28.93\n\n**Q2. Time consuming.**\n\n**A2**: Please refer to the common responses for a detailed explanation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3WswDodu2R", "reviewer": "Reviewer_H3YA", "summary": "The paper introduces GSDF, a new approach for both novel view synthesis and surface reconstruction that relies on a dual-branch architecture combining 3D Gaussian Splatting (3DGS) with neural Signed Distance Functions (SDF).\n\n**Details:**\n\nThe paper aims to improve the quality of both the rendering compared to vanilla 3DGS and the reconstructed surface compared to neural SDF approaches like NeuS. To this end, it uses two branches:\n1. **3DGS Branch**: Inspired by Scaffold-GS, this branch outputs RGB images with splatting-based, rasterized rendering.\n2. **Neural SDF Branch**: Inspired by Instant-NSR, a custom implementation of NeuS using hash grids (similar to Instant-NGP) to accelerate optimization. This branch outputs RGB images using volumetric rendering.\n\nThe method establishes mutual guidance and joint supervision between the two representations, relying on three main ideas:\n1. **Efficient Sampling**: Using depth maps rasterized with Gaussian splats to guide the volumetric rendering in the SDF branch, allowing for much more efficient sampling along rays. In practice, the sampling range for a ray $r$ going through a pixel $p$ is computed using the SDF value of the point $x$ resulting from the backprojection of the 3DGS depth map at pixel $p$.\n2. **Guided Densification and Pruning**: Using the SDF branch to guide the densification and pruning processes in the 3DGS representation. The underlying idea is straightforward: Densification should be stronger near the surfaces, and Gaussians far from the surfaces should be pruned. In practice, the paper proposes to compute a criterion depending on both the gradient of Gaussian primitives (similarly to Scaffold-GS) and the SDF value of the center of a Gaussian. The higher the gradient and the closer the Gaussian is to the surface, the more likely it is to be densified. In the meantime, Gaussians located far from the surface are pruned.\n3. **Geometric Alignment**: Aligning geometric properties (depth and normal maps) of both branches to ensure the consistency of the two branches as well as better geometric accuracy of the 3DGS representation, as Gaussians generally tend to cheat on the geometry to allow for better rendering.\n\nThe proposed approach limits floaters and ensures better alignment of 3D Gaussian primitives with the underlying surface. The overall geometric quality and details of the 3DGS representation are improved, reaching higher performance than concurrent state-of-the-art methods. Moreover, the paper explains that the 3DGS branch allows for accelerated convergence (and better performance) of the SDF branch thanks to much more efficient sampling along rays.", "questions": "1. Why rely on Scaffold-GS? I understand that the underlying voxel+anchor-based structure of Scaffold-GS helps enforce a better structure of 3D Gaussians as well as quickly identifying which Gaussians could be pruned or densified. However, it would be really interesting to have an ablation using vanilla 3DGS rather than Scaffold-GS for the 3DGS branch, to know if the underlying regularization coming from the Scaffold-GS structure is essential to the overall approach, or if it can generalize to unstructured 3D Gaussians.\n\n2. What resources (time and memory) are needed for optimizing the model? The paper explains that the approach requires 2 hours on DTU compared to 8 hours with NeuS. However, the paper uses Instant-NSR’s implementation, which makes NeuS converge much faster (10 minutes for scenes from the Blender dataset, for example). Is the speedup really due to the proposed method, or is it due to the hash-grid implementation from Instant-NSR?\n\n3. What is the optimization time (and memory requirement) for unbounded scenes, like Mip-NeRF 360 or Tanks&Temples?\n\n4. What surface extraction method is used? Is it possible to use the Gaussians for extracting the surface? Or both branches?\n\n5. (*Bonus question!*) Looking at Figure 4, it is undeniable that the proposed method achieves sharper results than the concurrent 2DGS approach. However, in the particular case of the Truck scene from Tanks&Temples (2nd row of Figure 4), it seems that 2DGS better reconstructs fine holes in the topology, looking at the back and the top of the truck. I’m curious to know if the authors have an idea of what component in their approach causes such a limitation in this particular example.", "limitations": "The authors addressed most of the limitations of their work. However, they do not discuss the memory requirements of the approach (relying on two different models at the same time), which might be much higher than concurrent approaches such as 2DGS, for example.\n\nMoreover, the limitations are only discussed in the supplementary material, which is a problem in my opinion. I encourage the authors to try to move the limitations to the main paper in the final version.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is clear and well-written. The details are easy to follow.\n\n2. The proposed approach shows excellent performance on quantitative benchmarks, effectively outperforming concurrent state-of-the-art works.\n\n3. The qualitative results are impressive and clearly demonstrate the improvement in geometric accuracy and surface details brought by the approach.\n\n4. As the paper explains, *“[The] framework is versatile and can be easily adapted to incorporate future advanced methods for each branch”* (lines 121-122). In other words, from a high-level perspective, I believe the paper offers a simple plug-in strategy that may allow jointly optimizing any Gaussian-based approach with any Neural SDF approach. I would be very curious to know if the authors tried their strategy with a regular, vanilla 3DGS representation rather than Scaffold-GS; I have no doubt that the results would be worse, but for generalization purposes, it would be interesting to know if the Scaffold-GS structural properties are essential to the approach, or if it can easily extend to 3DGS-based representations that do not rely on anchor points and MLP-decoders for the parameters.", "weaknesses": "1. **Concerning Speedup Contribution**: It is undeniable that the proposed approach allows for better reconstruction quality. However, it is unclear how the dual-branch system helps to make the SDF optimization faster, as claimed in the paper (see subsection 4.2.2, for example, line 246: “*our method optimized the SDF field significantly faster than previous methods*”). Although the paper claims a speedup over NeuS (which requires 8 hours for DTU scenes), the proposed approach uses a custom implementation of NeuS, called Instant-NSR, that is supposed to be able to train NeuS models in 10 minutes on some scenes. Is the speedup over NeuS really due to the dual-branch system (and the benefits each approach brings to the other), or is it due to the Hash-grid architecture? In the end, it appears that the proposed approach actually slows down the optimization compared to a single branch relying on Instant-NSR (or a single branch relying on Gaussians, as the optimization time is longer than the concurrent 2DGS approach). Theoretically, the speedup claimed in the paper would make sense; however, in practice, it is not clear how using two branches allows for establishing a virtuous circle and speeding up optimization. On the contrary, it might slow down optimization compared to using a method relying on a single branch.\n\n2. **Surface extraction method**: I did not see any discussion about the mesh extraction method for evaluating the surface reconstruction. Since the approach relies on two different representations that are supposed to become consistent with each other, I assume it would be possible to leverage one representation or the other (or both) to extract a surface mesh. I suppose the authors use a Marching Cubes algorithm on the neural SDF, but I might be wrong. If Gaussians are indeed consistent with the SDF and well-aligned with the surface, would it be possible to extract a mesh using Poisson reconstruction, similar to SuGaR, or TSDF, similar to 2DGS (although it would not scale well to background regions)? Would it be worse than applying a marching algorithm on the SDF? An ablation regarding the surface extraction method would have been very interesting, especially in a setup where two representations are available at the same time.\n\n3. **Memory Footprint**: 3DGS has a pretty high memory footprint in itself. Scaffold-GS might help reduce this footprint, but what about the proposed dual-branch approach? The approach sounds very heavy memory-wise, as it simultaneously optimizes two NVS models. The paper explains that a single NVIDIA A100 GPU with 80GB of VRAM (which is a lot) was used, but does not give details about the memory consumption of the approach. What is the minimum memory requirement to optimize the proposed model? Memory (and time) consumption are very important criteria for graphics-based applications.\n\n4. **Novelty of the Approach**: The approach heavily relies on two existing works (Scaffold-GS and Instant-NSR), and might look like a combination of these works with limited novelty. However, I think the combination of both works is not trivial, and the paper proposes a satisfying strategy to make each model benefit from the other. The proposed work is also quite reminiscent of NeuSG, a CVPR 2024 paper that proposes to jointly optimize a set of 3D Gaussians and a neural SDF. Even though the novelty of the proposed work might not be so great, this is no point for rejection in my opinion, as the work is technically solid and the results are of high quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your efforts and valuable comments. Below we address concerns for each question. Common concerns are detailedly responded in the global rebuttal. Additional Figures and Tables are provided in the attached PDF, in which the index is denoted as *Figure A-D* and *Table A-B*.\n\n**Q1. Why using Scaffold-GS; Generalization of the framework.**\n\n**A1**: We used Scaffold-GS because it manages 3D Gaussians more efficiently, resulting in less memory consumption, more accurate depth predictions, and better rendering quality.\n\nFollowing the reviewer's suggestion, we tested our framework's generalization by switching the GS branch to vanilla 3DGS. As shown in *Figure D*, GSDF_3DGS results in better reconstruction quality than using only the SDF branch (Instant-NSR). Additionally, the rendering quality improved from 28.21 to 28.31, as shown in *Table B*.\n\n**Q2. Time consuming and memory usage.**\n\n**A2**: Thanks for pointing out. Indeed the speedup partially comes from the hash-grid implementation. We additionally compared the reconstruction quality of GSDF and only SDF-branch (I.e. instant NSR) with regard to training time, as illustrated in *Figure A*. It shows that GSDF can achieve higher reconstruction quality than the baseline when trained for the same amount of time and iterations. Please also refer to the global response.\n\nRegarding resources, we recorded the memory usage of our method and other methods in *Table A*. GSDF indeed uses more memory and we will clarify the limitations and move the limitations from supplementary material to the main paper.\n\n\n**Q3. The surface extraction method and the geometry of GS-branch.**\n\n**A3**: We extract the mesh using Marching Cubes on the SDF branch. As described in the paper (L26-30), we identified that enforcing Gaussian primitives to align with scene surfaces by restricting their shape and position can lead to compromise in rendering. Therefore, we keep the diversity of 3D Gaussians in pursuit of a better rendering quality.  In addition, the placement of the Gaussian primitives of our GS-branch is truly closer to the potential surfaces compared to using only our GS branch, as illustrated in *Figure B*.\n\n**Bonus Question: fine holes in 2DGS.**\n\n**Answer**: 2DGS optimizes discrete 2D Gaussians, while SDF optimizes a continuous field. Compared to global representation, the discrete primitives are more flexible to represent holes, while global representation is better at capturing continuous surface. Additionally, we introduced a curvature term in the loss function when optimizing SDF to avoid incorrect holes. However, this term can lead to over-smoothing, which may also lead to the missing fine holes in the Truck scene.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their valuable feedback. 
We are encouraged that reviewers find 
- our two-branch design is novel and effective in boosting reconstruction and rendering quality simultaneously; 
- our analysis and experiments are useful and comprehensive. 

We will release our code for reproduction and future research. Here we address some common concerns.

**1. Function of each branch**

Note that, our framework features a two-branch design, where each branch excels in its respective task while benefiting from mutual supervision: the SDF-branch specializes in accurate geometry reconstruction, while the GS-branch focuses on high-quality image rendering.

We emphasize that achieving good qualities in both rendering and reconstruction is extremely hard and no recent method can attain this goal to the best of our knowledge. As we described in the paper (line 24-26), a representation alone struggles to achieve good quality in both reconstruction and rendering. Through thorough analysis, we noticed that a naive integration (like guidance merely by loss) hardly balances the learning priority during training, thus, it can only boost the quality of either rendering or reconstruction at the cost of sacrificing the other.

Instead, we dug deeper into the architecture characteristics of the two branches and propose a tight guidance from model architecture, e.g., using the predicted depth from GS-branch to guide the ray sampling of SDF-branch (significantly guide the convergence of SDF branch) and use the SDF field to guide the densification of the GS-branch (geometry-based density control rather than heuristic strategy). The effectiveness is confirmed through comprehensive evaluations, showing improved reconstruction and rendering quality.

**2. Time consumption**

*Inference time* remains unaffected as each branch can be used individually.

Regarding *training time*:
- In each iteration, the training overhead of our two-branch design is higher than single-branch designs (Tab A). However, GSDF achieves better reconstruction quality comparing to the SDF-branch alone when trained for the same amount of time/iterations, as illustrated in Fig A. 
- The improvement in both the rendering and reconstruction quality are nonnegligible. The effectiveness of our proposed framework comes from the combination of local and global optimization. By introducing mutual optimization, the training time will inevitably increase. However, without much overhead on training efficiency, we can achieve improvement in both rendering and reconstruction, whilst our design does not have an impact on the inference stage, guaranteeing efficiency in downstream applications.
- The core contribution is our design of the two-branch (geometry + rendering) framework, where each branch can be updated to a more efficient version in the future. We demonstrated the generalizability of our framework by switching the GS-branch from Scaffold-GS to 3DGS, which still exhibited superior rendering and reconstruction quality (Fig D, Tab B)

**3. Comparison with Concurrent work**

The concurrent work NeuSG aims for an improved reconstruction, which augments the SDF branch by a SDF loss to encourage the SDF value of Gaussian points and MVS points to be 0, and a normal loss to encourage normal consistency between SDF and Gaussian. Extra regularization loss (including flattening the gaussian shape) is introduced to make gaussian more friendly to geometry, regardless of the potential sacrifice in rendering quality.

GSDF has critical differences comparing to NeuSG:
- Instead of only focusing on reconstruction, GSDF aims to improve both geometry and rendering, whose effectiveness has been verified by extensive experiments.
- Beyond loss-based guidance, we investigate the combination inside the model architecture and propose a tightly-coupled two-branch design including depth-guided sampling and SDF-guided densification.
- Moreover, GSDF does not require an extra MVS process to provide accurate geometry, making ours a more versatile method requiring only coarse initialization.

Meta-review: The ratings for this paper include two accepts (LpsC and H3YA), and two borderline accepts (96yp and yTkf). All reviewers acknowledge the effectiveness of the method presented in this paper. The combination of 3DGS and SDF mutually enhances each other, improving both reconstruction and rendering quality. Reviewers H3YA and LpsC have concerns about the paper's contribution and originality. All reviewers raised some technical concerns. The rebuttal addressed most of these concerns, so the AC recommends acceptance.
Decision: Accept (poster)