Title: GSDF: 3DGS Meets SDF for Improved Neural Rendering and Reconstruction
Venue: NeurIPS 2024
Accepted: yes
Keywords: Neural Rendering; 3D Reconstruction;3D Gaussian Splatting; Signed Distance Field
Abstract: Representing 3D scenes from multiview images remains a core challenge in computer vision and graphics, requiring both reliable rendering and reconstruction, which often conflicts due to the mismatched prioritization of image quality over precise underlying scene geometry. Although both neural implicit surfaces and explicit Gaussian primitives have advanced with neural rendering techniques, current methods impose strict constraints on density fields or primitive shapes, which enhances the affinity for geometric reconstruction at the sacrifice of rendering quality. To address this dilemma, we introduce GSDF, a dual-branch architecture combining 3D Gaussian Splatting (3DGS) and neural Signed Distance Fields (SDF). Our approach leverages mutual guidance and joint supervision during the training process to mutually enhance reconstruction and rendering. Specifically, our method guides the Gaussian primitives to locate near potential surfaces and accelerates the SDF convergence. This implicit mutual guidance ensures robustness and accuracy in both synthetic and real-world scenarios. Experimental results demonstrate that our method boosts the SDF optimization process to reconstruct more detailed geometry, while reducing floaters and blurry edge artifacts in rendering by aligning Gaussian primitives with the underlying geometry.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces GSDF, a new approach for both novel view synthesis and surface reconstruction that relies on a dual-branch architecture combining 3D Gaussian Splatting (3DGS) with neural Signed Distance Functions (SDF).\n\n**Details:**\n\nThe paper aims to improve the quality of both the rendering compared to vanilla 3DGS and the reconstructed surface compared to neural SDF approaches like NeuS. To this end, it uses two branches:\n1. **3DGS Branch**: Inspired by Scaffold-GS, this branch outputs RGB images with splatting-based, rasterized rendering.\n2. **Neural SDF Branch**: Inspired by Instant-NSR, a custom implementation of NeuS using hash grids (similar to Instant-NGP) to accelerate optimization. This branch outputs RGB images using volumetric rendering.\n\nThe method establishes mutual guidance and joint supervision between the two representations, relying on three main ideas:\n1. **Efficient Sampling**: Using depth maps rasterized with Gaussian splats to guide the volumetric rendering in the SDF branch, allowing for much more efficient sampling along rays. In practice, the sampling range for a ray $r$ going through a pixel $p$ is computed using the SDF value of the point $x$ resulting from the backprojection of the 3DGS depth map at pixel $p$.\n2. **Guided Densification and Pruning**: Using the SDF branch to guide the densification and pruning processes in the 3DGS representation. The underlying idea is straightforward: Densification should be stronger near the surfaces, and Gaussians far from the surfaces should be pruned. In practice, the paper proposes to compute a criterion depending on both the gradient of Gaussian primitives (similarly to Scaffold-GS) and the SDF value of the center of a Gaussian. The higher the gradient and the closer the Gaussian is to the surface, the more likely it is to be densified. In the meantime, Gaussians located far from the surface are pruned.\n3. **Geometric Alignment**: Aligning geometric properties (depth and normal maps) of both branches to ensure the consistency of the two branches as well as better geometric accuracy of the 3DGS representation, as Gaussians generally tend to cheat on the geometry to allow for better rendering.\n\nThe proposed approach limits floaters and ensures better alignment of 3D Gaussian primitives with the underlying surface. The overall geometric quality and details of the 3DGS representation are improved, reaching higher performance than concurrent state-of-the-art methods. Moreover, the paper explains that the 3DGS branch allows for accelerated convergence (and better performance) of the SDF branch thanks to much more efficient sampling along rays.\n\nSTRENGTHS:\n1. The paper is clear and well-written. The details are easy to follow.\n\n2. The proposed approach shows excellent performance on quantitative benchmarks, effectively outperforming concurrent state-of-the-art works.\n\n3. The qualitative results are impressive and clearly demonstrate the improvement in geometric accuracy and surface details brought by the approach.\n\n4. As the paper explains, *“[The] framework is versatile and can be easily adapted to incorporate future advanced methods for each branch”* (lines 121-122). In other words, from a high-level perspective, I believe the paper offers a simple plug-in strategy that may allow jointly optimizing any Gaussian-based approach with any Neural SDF approach. I would be very curious to know if the authors tried their strategy with a regular, vanilla 3DGS representation rather than Scaffold-GS; I have no doubt that the results would be worse, but for generalization purposes, it would be interesting to know if the Scaffold-GS structural properties are essential to the approach, or if it can easily extend to 3DGS-based representations that do not rely on anchor points and MLP-decoders for the parameters.\n\nWEAKNESSES:\n1. **Concerning Speedup Contribution**: It is undeniable that the proposed approach allows for better reconstruction quality. However, it is unclear how the dual-branch system helps to make the SDF optimization faster, as claimed in the paper (see subsection 4.2.2, for example, line 246: “*our method optimized the SDF field significantly faster than previous methods*”). Although the paper claims a speedup over NeuS (which requires 8 hours for DTU scenes), the proposed approach uses a custom implementation of NeuS, called Instant-NSR, that is supposed to be able to train NeuS models in 10 minutes on some scenes. Is the speedup over NeuS really due to the dual-branch system (and the benefits each approach brings to the other), or is it due to the Hash-grid architecture? In the end, it appears that the proposed approach actually slows down the optimization compared to a single branch relying on Instant-NSR (or a single branch relying on Gaussians, as the optimization time is longer than the concurrent 2DGS approach). Theoretically, the speedup claimed in the paper would make sense; however, in practice, it is not clear how using two branches allows for establishing a virtuous circle and speeding up optimization. On the contrary, it might slow down optimization compared to using a method relying on a single branch.\n\n2. **Surface extraction method**: I did not see any discussion about the mesh extraction method for evaluating the surface reconstruction. Since the approach relies on two different representations that are supposed to become consistent with each other, I assume it would be possible to leverage one representation or the other (or both) to extract a surface mesh. I suppose the authors use a Marching Cubes algorithm on the neural SDF, but I might be wrong. If Gaussians are indeed consistent with the SDF and well-aligned with the surface, would it be possible to extract a mesh using Poisson reconstruction, similar to SuGaR, or TSDF, similar to 2DGS (although it would not scale well to background regions)? Would it be worse than applying a marching algorithm on the SDF? An ablation regarding the surface extraction method would have been very interesting, especially in a setup where two representations are available at the same time.\n\n3. **Memory Footprint**: 3DGS has a pretty high memory footprint in itself. Scaffold-GS might help reduce this footprint, but what about the proposed dual-branch approach? The approach sounds very heavy memory-wise, as it simultaneously optimizes two NVS models. The paper explains that a single NVIDIA A100 GPU with 80GB of VRAM (which is a lot) was used, but does not give details about the memory consumption of the approach. What is the minimum memory requirement to optimize the proposed model? Memory (and time) consumption are very important criteria for graphics-based applications.\n\n4. **Novelty of the Approach**: The approach heavily relies on two existing works (Scaffold-GS and Instant-NSR), and might look like a combination of these works with limited novelty. However, I think the combination of both works is not trivial, and the paper proposes a satisfying strategy to make each model benefit from the other. The proposed work is also quite reminiscent of NeuSG, a CVPR 2024 paper that proposes to jointly optimize a set of 3D Gaussians and a neural SDF. Even though the novelty of the proposed work might not be so great, this is no point for rejection in my opinion, as the work is technically solid and the results are of high quality.\n\nQUESTIONS:\n1. Why rely on Scaffold-GS? I understand that the underlying voxel+anchor-based structure of Scaffold-GS helps enforce a better structure of 3D Gaussians as well as quickly identifying which Gaussians could be pruned or densified. However, it would be really interesting to have an ablation using vanilla 3DGS rather than Scaffold-GS for the 3DGS branch, to know if the underlying regularization coming from the Scaffold-GS structure is essential to the overall approach, or if it can generalize to unstructured 3D Gaussians.\n\n2. What resources (time and memory) are needed for optimizing the model? The paper explains that the approach requires 2 hours on DTU compared to 8 hours with NeuS. However, the paper uses Instant-NSR’s implementation, which makes NeuS converge much faster (10 minutes for scenes from the Blender dataset, for example). Is the speedup really due to the proposed method, or is it due to the hash-grid implementation from Instant-NSR?\n\n3. What is the optimization time (and memory requirement) for unbounded scenes, like Mip-NeRF 360 or Tanks&Temples?\n\n4. What surface extraction method is used? Is it possible to use the Gaussians for extracting the surface? Or both branches?\n\n5. (*Bonus question!*) Looking at Figure 4, it is undeniable that the proposed method achieves sharper results than the concurrent 2DGS approach. However, in the particular case of the Truck scene from Tanks&Temples (2nd row of Figure 4), it seems that 2DGS better reconstructs fine holes in the topology, looking at the back and the top of the truck. I’m curious to know if the authors have an idea of what component in their approach causes such a limitation in this particular example.\n\nLIMITATIONS:\nThe authors addressed most of the limitations of their work. However, they do not discuss the memory requirements of the approach (relying on two different models at the same time), which might be much higher than concurrent approaches such as 2DGS, for example.\n\nMoreover, the limitations are only discussed in the supplementary material, which is a problem in my opinion. I encourage the authors to try to move the limitations to the main paper in the final version.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to jointly optimize 3D Gaussian Splatting (3DGS) and SDF (like NeuS). During GS optimization, the Gaussians are aligned to the zero-level set (and normals) of SDF. During the NeuS-like optimization, Gaussians are used to limit the range of ray sampling, resulting in efficient optimization. Experiments show that the proposed method achieves better reconstruction and rendering compared with SDF-based methods (e.g., NeuS) and GS-based methods, including recent SuGaR and 2D-GS, which explicitly align Gaussians to surfaces.\n\nSTRENGTHS:\n+ This paper proposes a nice combination of 3DGS and SDF (but not the first to do, which I'll detail in the weakness section). Both Gaussians and SDF representations provide merit to each other, resulting in accurate reconstruction and rendering.\n+ Experiments are well done. Very recent methods, like 2D Gaussian splatting and SuGaR are adequately evaluated, showing the merit of doing joint optimization of Gaussians and SDF, not just by aligning Gaussians on the surfaces.\n\nWEAKNESSES:\n## Related work\nThis method is not the first one combining the 3DGS and SDF, as mentioned in related work (NeuSG [5]). Regarding CfP of NeurIPS, \"papers that appeared online within two months of a submission will generally be considered \"contemporaneous\" in the sense that the submission will not be rejected on the basis of the comparison to contemporaneous work.\"\n\nIn other words, NeuSG, which was submitted on Dec 1, 2023, in arXiv, has to be regarded as an \"official\" existing work. Thus, the paper should discuss more the technical differences from the NeuSG work. I basically agree with the argument in L81-86 that mentions the subjective merit of the proposed method compared to NeuSG; the proposed method does significantly more for tighter integration of 3DGS and SDF.\n\nMeanwhile, the readers should wonder if the proposed method achieves better than NeuSG or not. Unfortunately, the code of NeuSG does not seem available, and an official comparison may be difficult. I did not come across the NeuSG paper deeply and may mention something wrong, but I'm wondering if the authors may show a quick ablation study using a baseline that somewhat mimics a simplified version of NeuSG by only using normal supervision by SDF and force the Gaussians flat.\n\nQUESTIONS:\n- Although I may miss some information, an interesting potential merit of the proposed method compared to SDF-based methods is the efficient ray sampling, while the overall optimization should take longer than vanilla 3DGS. I would like to see the discussions on training time compared to those methods.\n\nLIMITATIONS:\nI did not find notable negative social impacts.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the challenge of representing 3D scenes from multiview images by introducing a novel dual-branch architecture named GSDF, which combines 3D Gaussian Splatting and neural Signed Distance Fields. This architecture enhances both rendering and reconstruction through mutual guidance and joint supervision. By aligning Gaussian primitives with potential surfaces and speeding up SDF convergence, the method achieves finer geometric reconstructions and minimizes rendering artifacts. Demonstrating robustness and accuracy, the approach is effective in both synthetic and real-world scenarios.\n\nSTRENGTHS:\nThe paper employs a dual-branch approach for simultaneous scene rendering and mesh reconstruction. It utilizes an SDF (Signed Distance Field) branch to guide the geometric optimization of the Gaussian branch, including operations such as adding and removing points. By leveraging bidirectional optimization, the method simultaneously enhances the reconstruction and rendering quality of both branches. This approach ensures high rendering quality while effectively reconstructing the scene's mesh, achieving smooth reconstruction results on both object-level and scene-level datasets. Additionally, the method has also demonstrated commendable performance in novel view synthesis tasks.\n\nWEAKNESSES:\nIt appears that in the reconstruction and rendering branches, the authors have merely pieced together appropriate solutions. Although bidirectional optimization has been applied to both branches, the unity of the method seems lacking. The interrelation between the methods is weak, resembling more of an incremental improvement rather than a cohesive, integrated advancement.\n\nQUESTIONS:\n1. The rendering and reconstruction results mentioned in the paper are derived from which branch? Are both rendering and reconstruction results obtained from the 3DGS branch, or is rendering performed by the 3DGS branch while mesh is obtained from the Instant-NSR branch?\n2. Can using pseudo-depth and normals (obtained from other estimation algorithms) directly enhance Instant-NSR or Gaussian, potentially achieving good reconstruction and rendering results without the need for bidirectional optimization?\n3. Which branch's rendering results are displayed in the ablation study? If it is the Instant-NSR branch, then different strategies for adding and removing points should not affect its rendering results. Similarly, if it is the 3DGS branch, should depth-guided sampling also not affect its rendering outcomes?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces GSDF that utilizes a joint optimization of the GS-branch and SDF-branch to constrain the inherent geometric issues of the 3DGS. Furthermore, it proposes three mutual guidances to ensure satisfactory outcomes in both rendering and reconstruction. The extensive experiments on datasets such as DTU, Tanks and Temples (T&T), and Mip-NeRF 360 demonstrate that the method can achieve high rendering quality while obtaining better geometric results.\n\nSTRENGTHS:\n1. The article is clearly written, allowing one to easily understand the motivation in designing the two branches.\n2. The author conducted extensive experiments across various datasets to illustrate the high quality of rendering and geometric results achieved by GSDF.\n3. Utilizing the SDF-branch to supervise the inaccurate geometry or depth of 3DGS seems to be reasonable.\n\nWEAKNESSES:\n1. Instead of sampling relying on predicted SDF, GSDF samples near the depth from GS branch. However, I guess the imprecise depth of 3DGS could infulence the SDF-branch. I am curious whether there is a difference between the results of the SDF-branch and the baseline's SDF-branch results? \n2. Which branch outputs the final result of GSDF method? If the GS branch outputs both rendering and gemetry results, how about the rendering and geometry outputs of another branch? \n3. It seems that the time-consuming aspect is missing in Table 1 or Table 2, and the quantitative result of NeuS is missing in Figure 4. Additionally, as reported on lines 248-249, the time consumption of GSDF (2 hours) is significantly higher compared to 2DGS (5-10 minutes) and the original 3DGS. This substantial gap may be attributed to the volumetric rendering of the SDF branch; thus, the improved geometry results might be benefiting from this branch. It may be necessary to compare it with some neural implicit reconstruction methods (like Neuralangelo) to better evaluate its performance given the additional computational cost.\n4. In Table 3 of the ablation study, if depth-guided sampling is removed (w/o depth-guided sampling), one would need to consider whether the SDF-branch is now completely identical to the baseline NeuS. At this point, it would be important to assess if the results of the SDF-branch are close to baseline of SDF-branch. Furthermore, as mentioned in Lines 248-249, one should investigate whether the time consumption has significantly increased due to the absence of depth-guided sampling, given that this might affect the efficiency of the volumetric rendering performed by the SDF-branch.\n5. The result in table1, the CD results of NeuS and 2DGS is not as good as in their papers. And there might be some ambiguity in the symbolic expressions presented in the text. e.g. is line154 Fsdf the same with Line139 Fs?\n\nQUESTIONS:\nplease refer to weaknesses section.\n\nLIMITATIONS:\nThe authors didn't explicitly address the limitation of their approach.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-15"}