Injecting Inductive Bias to 3D Gaussian Splatting for Geometrically Accurate Radiance Fields

Authors: Sungwon Hwang, Soohyun Ryu, Jaeseong Yun, Jaeseong Lee, Junha Hyung, Min-Jung Kim, Taewoong Kang, Joohyun Park, Jaegul Choo, Suyong Yeon
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: 3D Gaussian Splatting, Surface Reconstruction

Abstract:
3D Gaussian Splatting (3DGS) has significantly advanced high-fidelity, real-time novel view synthesis. However, its discrete nature limits the accurate reconstruction of geometry. To address this issue, recent methods have introduced rendering and regularization of depth and normal maps from 3D Gaussians, leading to plausible results. In this paper, we argue that computing normals from independently trainable Gaussian covariances contradicts the strict definition of normals, which should instead be derived from the distribution of neighboring densities. To address this, we introduce an inductive bias into 3DGS by explicitly parameterizing covariances of Gaussians using principal axes and variances of distribution computed from neighboring Gaussians. These axes and variances are then regularized to ensure local surface smoothness. Our approach achieves competitive performance on multiple datasets.

Decision: reject
Reviewer scores: 8, 6, 6, 3

Reviews and discussion:
<thread>
creation date:
1729780102107

thread_type:
official_review

<thread_object>
creator: reviewer_2feu

<content>
summary:
This paper aims to overcome the limitations of 3DGS in geometry reconstruction, with a focus on regularization techniques. Previous methods independently define each Gaussian novel vector and only used screen space regularization. However, the paper argues that the normal vector of a Gaussian should depend on its local neighbors. To incorporate structural priors (also referred to as inductive bias in the paper), the paper proposes to parameterize the covariance of each primitive based on its neighborhood. Several regularizations based on the parameterized covariance are then integrated. Experiments suggest that this approach achieves state-of-the-art geometry reconstruction results across multiple datasets.

strengths:
1. The paper is well-written and easy to follow. The results seem reproducible given sufficient details.

2. Using local neighbors to compute the orientation seems original and reasonable for achieving better results. The visualization also consistently suggests that the proposed regularizations alleviate some holes presented in previous methods.

weaknesses:
1. The significance of the proposed method appears to be limited as it is built upon RaDeGS with additional regularization. However, it only results in a 0.02 gain in the DTU dataset while being five times slower than RaDeGS.

2. I understand that the normal vector should ideally be derived from the density distribution of the superposition of Gaussian functions.  However, However, I believe that the current method doesn't address the core issue. From a geometric view, the normal vector should not depend on the viewing angle. In Section 3.1, the normal is calculated by determining a plane formed by the intersection point and the center of the Gaussian, which is view-dependent. Consequently, the definition of the normal may contradict the argument based on the density distribution and the orientation defined by the covariance obtained through local neighbors.

3. Lastly, constructing local covariance using KNN may pose challenges. Each Gaussian represents a three-dimensional signal, not a zero-dimensional point, so we should consider its anisotropic nature when taking neighbor information into account. For example, a 3D Gaussian with a density that encompasses a point but with its mean located away from that point.

questions:
1. What is the cause of lower training efficiency? Is it because of the regularization? What is the number of primitives? 

2. How can you address the problem mentioned in the weakness or if the impact is negligible in practice?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730377807050

thread_type:
official_review

<thread_object>
creator: reviewer_amx1

<content>
summary:
This paper focuses on constructing high-quality geometric surfaces in 3D Gaussian Splatting (3DGS) representation, with depth and normal learning being key aspects. Unlike existing methods that learn normals for each Gaussian individually, this paper proposes deriving normals using the distribution of neighboring Gaussians. Validated across multiple datasets, this approach enables the paper to achieve smoother local surfaces.

strengths:
● This paper is a comprehensive work that starts by addressing issues arising from existing methods that optimize each Gaussian individually. It proposes a normal expression based on local regions and designs several regularization methods based on this normal computation. 

● The starting point of the paper is indeed one of the key differences between the existing 3DGS framework and traditional surface reconstruction, making it quite insightful. Traditional surface reconstruction places greater emphasis on the distribution of point clouds in the neighborhood, while existing 3DGS frameworks rely more on 2D rendering loss, often overlooking 3D continuity.

● The experimental validation in this paper is thorough, including both quantitative and qualitative comparisons with existing methods on mainstream datasets, as well as ablation studies of the proposed regularization losses.

● The paper’s presentation is very clear and easy to read.

weaknesses:
● With the use of the proposed covariance design and additional regularization losses, the improvement over the existing optimal solution is not significant; on the DTU dataset, the Chamfer Distance (CD) only improves by 0.02 compared to RaDe-GS, while the training time is 75 minutes longer (more than five times that of RaDe-GS).

questions:
● If the PCA-based regularization on local regions proposed in the paper is directly introduced into the existing 3DGS framework, can it also achieve similar smoothing effects as described in the paper?

● As far as I know, in traditional point cloud-to-surface reconstruction, the size of the local region has a significant impact on the final reconstruction results. Does the value of k in the kNN used in the paper also have a substantial influence? Should it be individually adjusted for scenes with varying levels of surface complexity?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730642284448

thread_type:
official_review

<thread_object>
creator: reviewer_ymcd

<content>
summary:
The paper proposes to improve the geometry quality of GS reconstruction by incorperating an inductive bias in GS covariance parameterization. The author claim that the proposed method reaches the state-of-art.

strengths:
1) The presentation and writing is clear and easy to understand.
2) The proposed method regard every splat is not independent in cov computing process, which is intuitively reasonable. 
3) The sparsity-aware density control strategy is interesting and useful.
4) The geometries of the demonstrated samples are impressive.

weaknesses:
1) As shown in Table 1, although the proposed method maintains the faster training speed compared with the implicit reconstruction method, it takes longer time to train compared all listed explicit methods. However, the fast training  is one of the main advantages of GS. The proposed method need over 6 times of training time compared with the original GS.
2) The proposed method uses adjacent Gaussians for covariance parameterization. In different scenes with different scales, how to select K neighbours as "adjacent Gaussians"  is a question.
3) No failure cases are discussed.

questions:
1) Have you investigated that which part of the proposed GS variant cost too much time in training? 
2) Any ablations on the K neighbours of Gaussian for cov parameterization?
3) Any failure cases?
4) Since the GS pointclouds are sparse, with shape of "Splat". The surface reconstruction quality may lies in any part of the whole chain of reconstruction. For example, how do you integrate depth to TSDF volume? Do the listed methods for comparison use same method for mesh extraction?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730885434569

thread_type:
official_review

<thread_object>
creator: reviewer_xitp

<content>
summary:
This paper introduces a method called IBGS that "injects Inductive Bias to 3D Gaussian Splatting". The key idea is computing normals from the distribution of neighboring densities instead of from independently trainable Gaussian covariances. This paper also proposes geometry regularization methods to help form smooth local surface. Experiences on multiple datasets show that the proposed method achieves state of the art in surface reconstruction tasks among 3DGS-based methods while maintaining faster training time compared to implicit neural representation-based methods.

strengths:
1. The key idea is well-motivated and novel. 3DGS-based methods have been a promising direction for faster surface reconstruction algorithms. A few papers have been trying to improve the normal calculation from 3D Gaussians. This paper brings a new perspective and proposes a novel normal calculation approach, aiming to predict more coherent surface. The new calculation also comes with novel regularization techniques for the normals.

2. Quantitative and qualitative results on multiple datasets demonstrate the effectiveness of the proposed method -- it achieves new state of the art in surface reconstruction among 3DGS-based methods while maintaining faster training time compared to implicit neural representation-based methods.

3. The paper is overall well-written and easy to follow. Implementation details sufficiently discussed.

weaknesses:
1. The rationale behind calculating normals form neighboring Gaussians is well explained. Some of the regularization / splitting techniques used in this paper look very similar to the techniques applied in methods that optimize individual Gaussians. Could the authors help explain the difference and share some ablation studies (e.g., if they happen to have done these comparisons) for readers to better understand the effects of these techniques?

(1) This paper minimizes the smallest Eigen value lambda_3 (Sec. 3.3. 1, Eq. 8), this looks similar to (Sec. 3.2.1, Eq. 7) of NeuSG [A] that minimizes the smallest component of each Gaussian's three scaling factors.

(2) The sparsity-aware adaptive density control of this paper (Sec. 3.4) seems very similar to VCR-GauS [B] Sec. 3.4 that tries to split large Gaussians in the textureless areas by placing new Gaussians that evenly divide the maximum scale of the old Gaussian.

[A] Chen et al. Neusg: Neural implicit surface reconstruction with 3d gaussian splatting guidance. (already in the references)
[B] Chen et al. VCR-GauS: View Consistent Depth-Normal Regularizer for Gaussian Surface Reconstruction. NeurIPS 2024.

questions:
The improved surface quality is well illustrated in the results. I was wondering if there are some remaining failure cases / limitations of the proposed approach. For example, the training speed seems a bit slower than other 3DGS-based methods, is it caused by the overhead of computing properties among K nearest neighbors?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731978295678

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We thank the reviewers for their insightful and valuable feedback.

**Especially, we appreciate all four reviewers for recognizing:**
  * Novelty and sanity of our approach
  * Clear demonstration in qualitative improvements on multiple datasets
  * Clear presentation

Below, we summarize some common concerns from the reviewers.

>
> ## **(1) On seemingly negligible improvement in quantitative measures.**
>

We totally understand that 0.02 gain in Chamfer Distance (CD) may seem negligible if viewed by numbers only. However, the improvement becomes more obvious when the equivalent scene is observed qualitatively. For instance, CD of the scene (scan ID `65` of DTU dataset) in Figure 1 of our paper trained with RaDe-GS and our method is `0.76` and `0.73`, respectively, yielding an improvement of `0.03`. Although this number may also seem negligible, the following visual difference is significant. 

Also, please zoom in for detailed comparisons on parts that are not highlighted as well, from which smoother surfaces can be observed from our method over RaDe-GS. Such sophisticated improvements make reasonable impacts on users, but can easily be underestimated when perceived in quantitative measures. 

>
> ## **(2) Failure case & limitations**
>

One limitation of our method is that it does not work on scenes where objects of interest are located far away from the cameras or unbounded scenes. It is due to the nature of perspective projection-based inverse rendering on descrete representations, where Gaussians gets sparser when they get farther away from camera. For instance, consider two rays from two adjacent pixels, respectively. Also assume that there are two Gaussians, each of which intersect one of the rays and cover the following pixel. Then, the farther away these two Gaussians are from the camera, the larger the distance would be between the two Gaussians. Since kNN will cover larger areas on sparser Gaussians, it is hard to assume the locality of the plane formed by these Gaussians. 

A possible solution to this limitation is to find neighbors using radius-based ball query **[A]**, so that neighbors that are too far away from the query can be rejcted. However, we did not experiment on ball query as there exists no implementation that is tensor-friendly (because ball query returns variable number of neighbors) and efficient compared to GPU-based library for kNN such as Faiss **[B]**. We will make sure to add these limitations in our paper. 


>
> ## **(3) Analysis and discussion on training time**
> 

* ***Training time analysis***

Compared to RaDe-GS, extra forward and backward process introduced by SVD and kNN were two key factors of increased training time. We made experimental analysis on how each of them contribute to the training time. We experimented on DTU Dataset with a single RTX 3090 GPU, and computed average training time for all scenes.  

First, applying SVD only increase training time from `16.2` min to `43.8` min, from which we can do a simple math to conclude that SVD contributed by `27.6` min. Since our full implementation took `91.2` min, kNN can be estimated to contribute by $91.2 - 43.8 = $ `47.4` min. By percentage, SVD and kNN contributes `30.2%` and `52.0%` of the training time, respectively. Note that kNN is conducted for every $100$ iteration throughout the whole training steps of $30\text{k}$ iterations.  

* ***Plans on efficient implementation for faster training***

One possible way to reduce training time to conduct SVD on points in view-frustum only for every iteration. For kNN, we can keep track of the change for every kNN graph update, and stop updating when few of the neighbors change. We believe that these simple engineering tricks can noticably reduce the training time, and planning to make the implementation upon the release of the code.


However, we would like to point out that even without these tricks, training time is already similar to the recent 3DGS-based methods such as SuGaR and GoF. We also hope that future research can investigate on other fundamentally novel approaches to improve training time while keeping the inductive bias.

>
> ## (4) **Ablation on $K$**
> 

We share the ablation results on $K$ on DTU Dataset. From the results, we learned that $K=5$ yields the best results in Chamfer Distance (CD). We have edited the experimental result in the appendix section of our manuscript.

  |K|2|3|4|5|6|7|8|9|10|
  |-|-|-|-|-|-|-|-|-|-|
  |CD| 0.701 | 0.709 | 0.684 | ***0.674*** | 0.678 | 0.686 | 0.684  | 0.688 | 0.692 |

> 
> ***Reference***
> 

  **[A]** Qi, Charles Ruizhongtai, et al. "Pointnet++: Deep hierarchical feature learning on point sets in a metric space." NeurIPS, 2017.

  **[B]** Johnson, et al. "Billion-scale similarity search with GPUs." IEEE Transactions on Big Data, 2019.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper describes a method for improving the geometry quality of Gaussian splatting by incorporating the inductive bias in reconstruction. Unlike previous methods that learn a surface normal for each Gaussian, the paper proposes to learn normals using the distribution of neighboring Gaussians. The strength of the work is the introduction of the new idea in the surface normal (or Gaussian covariance) computation scheme, and the quality of the result. On the other hand, the chief weakness was that the proposed surface normal computation is not adequately grounded. There was an extensive discussion about this respect during the reviewer-author discussion phase. Unfortunately, the concern was not clarified during the discussion. As a result, there were mixed opinions about the paper. The AC agreed with the opinion that the justification for the new surface normal computation needed to be fully explained because it was the key motivation for the work. Due to the lack of clear justification, we reached this recommendation.