Flow Distillation Sampling: Regularizing 3D Gaussians with Pre-trained Matching Priors

Authors: Lin-Zhuo Chen, Kangjie Liu, Youtian Lin, Zhihao Li, Siyu Zhu, Xun Cao, Yao Yao
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: 3D Vision, Differentiable Rendering, 3D Gaussian Splatting

Abstract:
3D Gaussian Splatting (3DGS) has achieved excellent rendering quality with fast training and rendering speed. However, its optimization process lacks explicit geometric constraints, leading to suboptimal geometric reconstruction in regions with sparse or no observational input views. In this work, we try to mitigate the issue by incorporating a pre-trained matching prior to the 3DGS optimization process. We introduce Flow Distillation Sampling (FDS), a technique that leverages pre-trained geometric knowledge to bolster the accuracy of the Gaussian radiance field. Our method employs a strategic sampling technique to target unobserved views adjacent to the input views, utilizing the optical flow calculated from the matching model (Prior Flow) to guide the flow analytically calculated from the 3DGS geometry (Radiance Flow). Comprehensive experiments in depth rendering, mesh reconstruction, and novel view synthesis showcase the significant advantages of FDS over state-of-the-art methods. Additionally, our interpretive experiments and analysis aim to shed light on the effects of FDS on geometric accuracy and rendering quality, potentially providing readers with insights into its performance.

Decision: accept (poster)
Reviewer scores: 5, 6, 8, 8

Reviews and discussion:
<thread>
creation date:
1729474971888

thread_type:
official_review

<thread_object>
creator: reviewer_ltym

<content>
summary:
In this paper, Flow Distillation Sampling (FDS) is proposed to improve the geometric accuracy and rendering quality of  3D Gaussians Splatting.
FDS first adopts a camera sampling scheme to sample unobserved views near the training views, and then uses the flow predicted by the pre-trained model to guide the flow calculated from the 3DGS geometry.

strengths:
1. The ideas are intuitive, the paper is well written and easy to understand.
2. FDS leverages the matching prior mitigate the overfitting problem and enhance the geometry.

weaknesses:
1. More ablation studies are needed.
- The depth-adaptive radius is calculated by Eq. (8). How to determine the value of the hyperparameter $\sigma$? Is FDS robust to different $\sigma$?
- Is  FDS must require the normal consistency loss $\mathcal{L}_{n}$? In Table 3, how is the performance of “2D-GS+FDS”?
- How to determine the weight for FDS $\lambda_{fds}$?
- How to determine the start iteration (e.g., 15,000) of applying FDS?

2 Lack of visual comparison on the ScanNet dataset.

questions:
See `Weakness`.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730134973922

thread_type:
official_review

<thread_object>
creator: reviewer_w1zd

<content>
summary:
This paper aims to improve the 3D Gaussian Splatting reconstruction quality in regions with sparse or no observational input views, by integrating a pre-trained matching prior into the 3DGS optimization process. The matching prior is incorporated through optical flow from a pre-trained model, which supervises the Radiance Flow calculated using 3DGS-rendered depth. Additionally, the authors introduce a Flow Distillation Sampling scheme to efficiently sample unobserved camera views around the input views. The proposed Flow Distillation Loss effectively avoids the scale ambiguity existing in monocular priors. The authors present clear ablation studies and quanlitative improvements to support their claims.

strengths:
- The idea is simple but makes intuitive sense. Matching priors can provide absolute scale information in constrast to monocular priors. This paper offers a promising direction of using pair-wise information prior to advance sparse view reconstruction.
- This ablation study clearly demonstrates the improvement benefited from using this pair-wise matching prior without scale ambiguity. The quanlitative results shown in Table 3. validate the advatange compared to monocular depth prior, even to multi-view depth prior. A clear visualization of the mutual refinement of two flows is also provided.

weaknesses:
- This paper lacks evaluation on widely used geometry reconstruction and novel view synthesis benchmarks such as DTU, Tanks and Temples, and MipNeRF 360. The advantages of the proposed method would be more convincing if the authors could present results on one or more of these benchmarks.
- As mentioned in line 252, both the Prior Flow and Radiance Flow suffer from inaccuracies, raising concerns about the stability of the benefits provided by the proposed Flow Distillation Loss. It is possible that this loss could introduce artifacts or incorrect guidance due to bias. While the metrics in Table 3 appear strong, it’s unclear why the loss significantly outperforms multi-view depth supervision, which does not suffer from inaccurate prior flow. More explanation and analysis are needed to clarify this point.

questions:
- Given that RAFT is computed at every time step, how does the training time for 2DGS + FDS compare to 2DGS?
- Why do both monodepth and multi-view depth seem to only worsen the results, as shown in Table 3.?
- it's said in line 316 that, the normal prior is introduced for evaluation on the ScanNet dataset. But the metrics in Table 2. and Table 3. seem inconsistenct regarding the results of 2DGS + FDS. Does this mean the ScanNet result in Table 2. doesn't use the normal prior?
- Can 2DGS + FDS outperform 2DGS + Normal in Table 3.?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730666737354

thread_type:
official_review

<thread_object>
creator: reviewer_148k

<content>
summary:
The paper proposes a optical flow based regularization for 3D and 2D Gaussian Splatting. It compares the optical flow output between an input view and a sampled unobserved view to a so-called radiance flow determined from the camera motion and the reconstructed scene. The authors claim that a loss on the difference between the radiance flow an the optical flow results in improved geometry and view synthesis quality. Experiments show improved performance of 3DGS and 2DGS with flow distillation on the SceneNet, Mushroom and Replica dataset.

strengths:
- The paper describes an interesting idea on incorporating a flow prior for 3D reconstruction with Gaussian Splatting.
- The related works contains all relevant geometry reconstruction-based 3DGS works and set them in context to the proposed method.
- The approach is simple and easy to understand, as Figure 1 and 2 are well done and intuitive.

weaknesses:
- The experimental evaluation only considered indoor room datasets, MuSHRoom, ScanNet and Replica. Baseline methods usually use more diverse datasets such as DTU [Jensen et al. 2014], Tanks and Tem-
ples [Knapitsch et al . 2017] and Mip-NeRF360 [Barron et al. 2022].
- The authors found that the depth distortion loss in 2DGS degrades the results. However they do not provide evidence or explanations for that and it is unclear how this influenced the quantitative comparison to 2DGS.
- The paper solely focus on 3DGS-based methods in the related work and also in the experimental evaluation. Comparisons to neural field based methods such as Geo-NeUS or NeuralAngelo would significantly strengthen the claim of state-of-the-art performance.
- The overall quality and the presentation should be improved, e.g. the conclusion is unspecific and contains general claims,  inconsistent capitalization of 'Gaussian', no explanation of \hat{\alpha} in equation 2.

questions:
- The Radiance Flow maps pixels from the source view  to the target view. Considering pixels in the target view containing splatted Gaussians that are occlude in the source view, how is the Radiance Flow computed for these regions?
- In line 267, how does detaching the optical flow influence the overall performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730688766880

thread_type:
official_review

<thread_object>
creator: reviewer_gpcw

<content>
summary:
the geometric under-constrain problem of 3D Gaussian Splatting (3DGS) in sparse view setups. By incorporating pre-trained matching priors into the optimization process of 3DGS, this method significantly improves both the geometric accuracy and rendering quality of 3DGS.

strengths:
The method proposed in this paper integrates matching priors derived from a pretrained optical flow model to guide the optimization of 3DGS. This approach effectively enhances both the reconstruction quality and rendering quality of existing 3DGS-based methods.

The methodology is well-structured, providing clear and detailed explanations of the proposed FDS technique, the adaptive camera sampling scheme, and the associated loss functions.

Comprehensive experimental evaluations are conducted across multiple datasets, demonstrating the method's effectiveness and robustness.

Additionally, the paper includes interpretive experiments to illustrate the mutual refinement process of the flows, thereby enhancing the understanding of the method's capabilities.

weaknesses:
The results of the proposed method are constrained by the initial quality of the prior flow, however, the reliability of the prior flow cannot be assured under certain sparse viewpoint configurations.

As noted in the limitations section, the method's reliance on the performance of a pretrained optical flow model restricts its generalizability.

While the authors have conducted experiments across multiple datasets, it is important to point out that these datasets are primarily limited to indoor scenes. It is recommended that the authors evaluate their method on a more diverse range of datasets to assess its applicability in various scenarios.

The paper lacks a discussion on the computational complexity of the method. It is recommended that the authors include a detailed report on the training and inference times of the model in the experimental section, along with comparative metrics against other existing methods.

questions:
In the related work section, the review of existing prior art aimed at improving 3DGS performance should be more comprehensive and clearer, particularly concerning the relevant work on optical flow priors. Additionally, in the subsection on Prior Regulation for Rendering, there are sentences with grammatical errors that require careful review and correction.

In Algorithm 1, there are notation errors that need careful checking and correction.

The comparison of depth reconstruction experiments needs to be supplemented with results from other methods to validate the superiority of the proposed approach in geometric reconstruction.

In the dataset section, the paper mentions that the authors have evaluated their method on the Replica dataset, but the experimental results are not presented.

The authors provide limited comparisons with existing methods; it is recommended that they include more baseline methods for a more robust evaluation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732347936951

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal by Authors

comment:
We sincerely thank all the reviewers for their constructive and insightful advice. 
We are encouraged that all reviewers consider the proposed FDS intuitive and interesting.
- Reviewer w1zd: The idea is simple but makes intuitive sense. This paper offers a promising direction of using pair-wise information prior to advance sparse view reconstruction.
- Reviewer LTym: The ideas are intuitive, the paper is well written and easy to understand.
- Reviewer 148K: The paper describes an interesting idea on incorporating a flow prior for 3D reconstruction with Gaussian Splatting.
- Reviewer GPCw: This approach effectively enhances both the reconstruction quality and rendering quality of existing 3DGS-based methods.

Meanwhile, we acknowledge the reviewers' primary concern that
this paper lacks evaluation on some widely used geometry reconstruction dataset, such
as DTU.
Since DTU is an object-level dense observation dataset, our FDS is
primarily designed to mitigate the issue of insufficient sampling in observation regions. 
Therefore,
we did not prioritize testing on the DTU dataset initially. 
We have now included results of our FDS on the DTU dataset in our supplementary material, as shown below.
While FDS performs better under sparse observations, it still achieves notable improvements on the DTU dataset with dense observations.

| Method        | 24    | 37    | 40    | 55    | 63    | 65    | 69    | 83    |
|---------------|-------|-------|-------|-------|-------|-------|-------|-------|
| 2DGS          | **0.48** | 0.86  | 0.36  | 0.43  | 0.90  | **0.94** | 0.80  | 1.27  |
| 2DGS+FDS      | 0.51  | **0.85** | 0.36  | 0.43  | **0.79** | 1.00  | **0.77** | **1.23** |

| Method        | 97    | 105   | 106   | 110   | 114   | 118   | 122   | mean  |
|---------------|-------|-------|-------|-------|-------|-------|-------|-------|
| 2DGS          | 1.30  | **0.72** | 0.70  | 1.24  | 0.47  | 0.70  | 0.58  | 0.78  |
| 2DGS+FDS      | **1.06** | 0.73  | **0.65** | **1.14** | **0.44** | **0.58** | **0.53** | **0.73** |

We have updated the submitted paper based on each reviewer's suggestions. The main updates include:
- We replaced the original RAFT model with a more advanced optical flow model, SEA-RAFT, enabling FDS to achieve a significant improvement in its performance. The experimental results presented in the submitted paper have all been updated accordingly.
- We have updated Table 3 to include more comprehensive ablation experiments on prior supervision.
- We have added detailed derivations to the paper to clarify the physical meaning of the hyperparameter $\sigma$, providing valuable insights for its configuration.

We have carefully addressed the additional concerns 
raised by each reviewer in the corresponding official comments. 
Finally, we extend our heartfelt gratitude to all reviewers 
and remain open to further 
suggestions to improve all aspects of our work.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents a method that integrates pretrained matching priors to guide the optimization of 3D Gaussian Splatting (3DGS). This approach leverages pretrained geometric knowledge to effectively enhance both the reconstruction and rendering quality of 3DGS methods. Incorporating a flow prior to 3D reconstruction with Gaussian Splatting is interesting and intuitive. The primary concern in the original submission was the lack of experiments on popular datasets such as DTU and MipNeRF. The rebuttal addressed this by including additional experiments on the DTU dataset, strengthening the evaluation of the proposed method.