HiSplat: Hierarchical 3D Gaussian Splatting for Generalizable Sparse-View Reconstruction

Authors: Shengji Tang, Weicai Ye, Peng Ye, Weihao Lin, Yang Zhou, Tao Chen, Wanli Ouyang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: 3D reconstruction, Gaussian Splatting, Generalizable Multi-View Reconstruction

Abstract:
Reconstructing 3D scenes from multiple viewpoints is a fundamental task in stereo vision. Recently, advances in generalizable 3D Gaussian Splatting have enabled high-quality novel view synthesis for unseen scenes from sparse input views by feed-forward predicting per-pixel Gaussian parameters without extra optimization. However, existing methods typically generate single-scale 3D Gaussians, which lack representation of both large-scale structure and texture details, resulting in mislocation and artefacts. In this paper, we propose a novel framework, HiSplat, which introduces a hierarchical manner in generalizable 3D Gaussian Splatting to construct hierarchical 3D Gaussians via a coarse-to-fine strategy. Specifically, HiSplat generates large coarse-grained Gaussians to capture large-scale structures, followed by fine-grained Gaussians to enhance delicate texture details. To promote inter-scale interactions, we propose an Error Aware Module for Gaussian compensation and a Modulating Fusion Module for Gaussian repair. Our method achieves joint optimization of hierarchical representations, allowing for novel view synthesis using only two-view reference images. Comprehensive experiments on various datasets demonstrate that HiSplat significantly enhances reconstruction quality and cross-dataset generalization compared to prior single-scale methods. The corresponding ablation study and analysis of different-scale 3D Gaussians reveal the mechanism behind the effectiveness. Code is at https://github.com/Open3DVLab/HiSplat.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730142835579

thread_type:
official_review

<thread_object>
creator: reviewer_psl3

<content>
summary:
This paper proposes HiSplat to estimate the 3D Gaussian Splatting structure of two given images with known camera poses in a feedforward process. It aims to use multi-scale information to improve the quality and for which two novel modules are proposed. Extensive experiments on various datasets can show its SOTA performance and good generalizability, and validate the effects of each proposed component.

strengths:
- The proposed Error Aware Module and Modulating Fusion Module are interesting, which can effectively filter the errors from photometric loss in a simple way.

- Experiments are extensive. Both qualitative and quantitative results show that HiSplat achieves SOTA performance compared to the compared baselines. The results are convincing.

- The paper is easy to follow.

weaknesses:
- Some related methods are missed in the discussion, e.g., FreeSplat [1] and Splatt3R [2]. Especially, FreeSplat also uses a cost-volume-based structure to estimate the depth and then Gaussians with multi-scale strategy. A step further, the architecture of this work is also somehow similar to CasMVS [3] that uses a cascade structure to solve the MVS depth estimation task. The main differences lie in how the multi-scale features are used. Need more deeper discussions about these related works to show the technical differences and more valuable analysis.

- The setting of this paper seems to be fixed at using only two images. Would like to see some discussion about if it's possible to extend the method to more input views, just like what FreeSplat does. Even if it is a part of limitation, it would still be much valuable.



[1] Wang, Yunsong, et al. "FreeSplat: Generalizable 3D Gaussian Splatting Towards Free-View Synthesis of Indoor Scenes." arXiv preprint arXiv:2405.17958 (2024).

[2] Smart, Brandon, et al. "Splatt3r: Zero-shot gaussian splatting from uncalibrated image pairs." arXiv preprint arXiv:2408.13912 (2024).

[3] Gu, Xiaodong, et al. "Cascade cost volume for high-resolution multi-view stereo and stereo matching." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730517154836

thread_type:
official_review

<thread_object>
creator: reviewer_r9ea

<content>
summary:
This work presents a new 3DGS-based generalizable sparse-view scene reconstruction method, HiSplat, which effectively integrates multi-scale hierarchical MVS features with proposed EAM and MFM modules to enhance the quality of feed-forward 3DGS models, especially in fine-grained details. Experimental results demonstrate the superior reconstruction quality compared to prior SoTA methods.

strengths:
1. The paper is well-motivated from the prior works on multi-scale hierarchical visual knowledge. It is well-written.
2. The way the authors integrate the multi-scale features is clever and interesting. I love the Error Aware part, HiSplat wisely incorporates the reconstruction error refinement, usually through gradient optimizations, into this feed-forward reconstruction model.  The error maps are used to guide the refinement on the higher levels. I believe this idea can also be beneficial for other related tasks.
3. The analysis in 4.4 is very informative and useful. The figures effectively demonstrate how multi-scale features can help improve the reconstruction quality.

weaknesses:
1. The current model is pretty complex. It contains a lot of different neural models (MVSformer++, DINOv2, and a bundle of UNets and MLPs). I am not very clear which modules are trainable and which are not. The authors should clarify and summarize this in 3.6. Also, it would be even better, this is not required for rebuttal, if HiSplat could be a relatively simpler unified foundation model (e.g., like LRM).  
2. Although the writing is generally good, the citation format of this paper is a bit disturbing. Please properly use `\cite` and `\citep` provided in the ICLR template.
3. Minor typos:
    * MVSpalt → MVSplat
    * L298: fellow → follow

questions:
1. Since it is unlikely we can always get accurate camera poses from sparse views in real life, can this work be extended to handle the pose-free sparse-view reconstruction for the captures from real life? If it is easy to implement, it would be great to see some examples.
2. The depth coefficient $\eta$ is a bit tricky. I assume it is a fixed value for each stage after the training, is this correct? If so, how do you choose the proper $\eta$ values?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730533211742

thread_type:
official_review

<thread_object>
creator: reviewer_gqrm

<content>
summary:
This paper claims that previous generalizable 3D Gaussian Splatting methods utilize uniform 3D Gaussians, making it challenging to simultaneously capture large-scale structures and intricate texture details. The authors propose a hierarchical approach to generalizable 3D Gaussian Splatting, constructing hierarchical 3D Gaussians through a coarse-to-fine strategy. Additionally, they design an Error Aware Module and a Modulating Fusion Module to manage interactions among different hierarchical Gaussians.

strengths:
1. The idea of using different hierarchical Gaussians to simultaneously capture large-scale structures and delicate textures in scene generation sounds reasonable.
2. The ablative study seems complete. 
3. The motivation is clear since not all points in a single image are equally important.

weaknesses:
1. The paper introduces many modules, which slows down the process compared to MVSplat, which is also revealed in Tab.4, after adding all the stages together. 
2. The improvement in metrics compared with baseline method does not seem very significant from the reviewer's perspective. But the generalization capability to unseen datasets like DTU seems interesting. The authors can show more cases that demonstrates generalization capability. 
3. Some of the visualized cases does not seem to possess a very significant improvement compared with MVSplat. For example, the last line of Fig.6; the left red bounding box in the first example in Fig. 1.

questions:
Apart from the weakness part, I also have the following questions.
1. Will the number of Gaussian primitives change significantly after applying this hierarchical structure? A comparison with baseline methods on the primitive numbers will be nice.
2. It seems that the first example in Fig.6. shows that HiSplat can generate less artifacts in the occluded area, as highlighted by the red bounding box. Does this just happen in coincidence or is it a prevalent phenomenon?

I still have the aforementioned concerns and I would be glad to raise my rating if they can be addressed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730571203905

thread_type:
official_review

<thread_object>
creator: reviewer_lyvm

<content>
summary:
HiSplat introduces a coarse-to-fine strategy to construct hierarchical 3D Gaussians for generalizable 3D Gaussian Splatting. Additionally, HiSplat proposes an Error-Aware Module and a Modulating Fusion Module to enhance inter-scale interactions. Comprehensive experiments demonstrate that HiSplat achieves better results than single-scale methods.

strengths:
1. HiSplat first introduces the hierarchical 3D Gaussian representation in the task of generalizable 3D Gaussian Splatting.
2. The experiment effectively demonstrated the effectiveness of its hierarchical representation through Fig 4, which shows the primitives of Gaussians obtained from different stages.
3. This paper show the effective ablation study, comparing the impact of different modules on the performance of Generalizable Sparse-View Reconstruction.

weaknesses:
1. Several spelling errors have been identified in the document, specifically on lines 73, 106, and 508.
2. The hierarchical representations appear to be memory-intensive and computationally inefficient. It is recommended that the paper discusses the implications for memory, FLOPs, and inference time.
3. The rationale for incorporating the DINO feature seems underwhelming, and the provided experiments do not convincingly demonstrate the necessity of including the DINO feature.
4. I hope to see additional experiments conducted to explore the impact of increasing the number of perspectives on the quality of reconstruction. It appears that the majority of the experiments presented in the article are limited to just two perspectives, which may not fully capture the potential benefits of a multi-perspective approach.

questions:
1. Discuss more about the importance of incorporating the DINO feature.
2. Discuss more about the complexity analysis of the method.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732269178024

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
The response to all reviewers

comment:
We appreciate the reviewers (R1:Lyvm, R2:gQRm, R3:R9EA, R4:PsL3) for their constructive suggestions and approving our contributions:
(1) interesting, well-motivated and novel approach (R1, R2, R3, R4); (2) comprehensive and complete experiments (R1, R2, R4); (3) well-writen and easy to follow (R3, R4). (4) informative and useful analysis (R1, R3).

We have updated the revision to follow the reviewers' suggestions, including correcting some errors, adding some experimental results, supplying discussion, and declaring confusion. To indicate the corresponding changes suggested by reviewers, we use a unique text color for each reviewer(R1:red, R2:blue, R3:green, R4:cyan) in the updated revision.  We will gladly incorporate all the feedback in the final version under the ICLR policy. We provide unique responses to each reviewer’s specific questions individually. Here, we respond to some common questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732269472906

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Common Question 1(R1Q2, R2Q1): HiSplat introduces many modules slowing down the process. Discuss the computational and memory costs of the method in detail.

Answer: We have discussed the cost of memory and computation in Appendix A.2. Besides, we also provide more details and analysis as follows.
1. Sources of additional inference overhead.  The extra overhead is from ① more Gaussian primitives: Compared with previous pixel-wise Gaussian methods such as MVSplat and PixelSplat, HiSplat introduces three-stage hierarchical Gaussian representation, resulting in approximately 1.31x Gaussian primitives. However, thanks to the highly efficient parallelizable rasterization library of Gaussian Splatting, the inference time during rendering only marginally increases (approximately 1.1x). ②more computational modules: we incorporate a pre-trained DINOv2 feature extractor based on a ViT-B/14 architecture. Moreover, we propose Error Aware Module and Modulating Fusion Module, mainly composed of lightweight U-Nets and MLPs. These introduced modules increase the computational demands of inference, which account for the majority of the inference time. Nevertheless, as shown in Table 3 in the manuscript, they play an important role in processing hierarchical Gaussian representation, delivering higher novel view synthesis performance and improved generalization. Thus, we believe the additional inference overhead is valuable. It is hard to have a free lunch. Besides, the total computational cost of our full-version approach is higher than MVSplat (about 2x) but lower than PixelSplat, indicating that the efficiency remains within an acceptable range.
2. Potential techniques for cost reduction. The current HiSplat reported in the manuscript is an experimental version, focusing on evaluating performance and feasibility rather than efficiency. There are several potential strategies to reduce inference costs in practical deployment: ①Pipeline parallelism. Because HiSplat can be divided into three stages after extracting features using the U-net backbone, pipeline parallelism can be employed for multiple input batches easily. Utilizing existing pipeline parallelism libraries can further enhance HiSplat's efficiency. ② Dynamic network manner. Hisplat's unique hierarchical design supports dynamic networks for flexible implementation. When efficiency is required, as shown in Table 4 in the manuscript, it is possible to use only the Gaussians in the early stages to reconstruct the scene, which sacrifices the performance but reduces inference time and memory requirements. HiSplat offers more deployment flexibility compared to methods with fixed-resolution Gaussians. ③Transformer acceleration techniques. A significant portion of computational and memory overhead in HiSplat arises from transformer-based modules like the corresponding modules of DINOv2 and UniMatch. Techniques like flash attention[1] and other transformer-specific optimizations can substantially reduce memory and computation requirements without sacrificing performance.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732269513870

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Common Question 2(R1Q4, R4Q2): Can HiSplat be extended to more input views? More experiments are needed.

Answer: Although HiSplat focuses on two-view reconstruction, the main components of HiSplat, such as the hierarchical Gaussian representation, shared feature extractors, Error Aware Module, and Modulating Fusion Module, are view-invariant, which means that HiSplat can be easily extended to handle reconstruction from an arbitrary number of views. To perform an initial verification, we directly use the weights trained by inputting two views on RealEstate10K to conduct cross-dataset testing with three views on DTU and Replica. The results are shown in Table t1. Compared with the two-view cases, the performance slightly improves with inputting three views, and HiSplat still outperforms MVSplat, demonstrating the feasibility of using HiSplat for multi-view reconstruction directly. However, since HiSplat is not specifically trained for three-view tasks and lacks specific structural adjustments for more views in Table t1, the gain is marginal. Considering the limited experimental time during the rebuttal phase, we will provide additional results on training and testing under more multi-view settings in the final version. Additionally, the core contribution of HiSplat, i.e., the hierarchical Gaussian representation, has the potential to be integrated with recent multi-view reconstruction methods, such as FreeSplat, to enhance their ability of novel view synthesis.

**Table t1**: Cross-dataset evaluation with more reference views. We train the models on RealEstate10K with two reference views and test them on DTU and Replica with three reference views.
| Method          | RealEstate10K→DTU | RealEstate10K→Replica |
|-----------------|-------------------|-----------------------|
| MVSplat+2 views | 13.94             | 23.79                 |
| MVSplat+3 views | 14.29             | 24.75                 |
| HiSplat+2 views | 16.05             | 27.17                 |
| HiSplat+3 views | 16.35             | 27.47                 |


[1] Flashattention-2: Faster attention with better parallelism and work partitioning.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a coarse-to-fine strategy to construct hierarchical 3D Gaussians for generalizable sparse-view 3D Gaussian Splatting. In addition, this paper proposes an Error-Aware Module and a Modulating Fusion Module to promote inter-scale interactions. Comprehensive experiments demonstrate that the proposed method achieves better results than single-scale methods.

Main strengths of this paper are as below: 
- The paper is clearly written and easy to follow. 
- The paper is well-motivated from the prior works on multi-scale hierarchical visual knowledge. 
- The paper introduces a coarse-to-fine strategy into sparse-view Gaussian Splatting. And it proposes an Error-Aware Module and a Modulating Fusion Module to promote inter-scale interactions, which are innovative and inspiring.
- The experiments demonstrate improvements over previous methods. And the ablation study shows the effectiveness of Error Aware Module and Modulating Fusion Module.

Needing more experiments and analyses were raised by the reviewers. Please revise the paper according to the discussions before submitting the final version.