SplatFormer: Point Transformer for Robust 3D Gaussian Splatting

Authors: Yutong Chen, Marko Mihajlovic, Xiyi Chen, Yiming Wang, Sergey Prokudin, Siyu Tang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: novel view synthesis, gaussian splatting, point cloud modeling

Abstract:
3D Gaussian Splatting (3DGS) has recently transformed photorealistic reconstruction, achieving high visual fidelity and real-time performance. However, rendering quality significantly deteriorates when test views deviate from the camera angles used during training, posing a major challenge for applications in immersive free-viewpoint rendering and navigation. In this work, we conduct a comprehensive evaluation of 3DGS and related novel view synthesis methods under out-of-distribution (OOD) test camera scenarios. By creating diverse test cases with synthetic and real-world datasets, we demonstrate that most existing methods, including those incorporating various regularization techniques and data-driven priors, struggle to generalize effectively to OOD views. To address this limitation, we introduce SplatFormer, the first point transformer model specifically designed to operate on Gaussian splats. SplatFormer takes as input an initial 3DGS set optimized under limited training views and refines it in a single forward pass, effectively removing potential artifacts in OOD test views. To our knowledge, this is the first successful application of point transformers directly on 3DGS sets, surpassing the limitations of previous multi-scene training methods, which could handle only a restricted number of input views during inference. Our model significantly improves rendering quality under extreme novel views, achieving state-of-the-art performance in these challenging scenarios and outperforming various 3DGS regularization techniques, multi-scene models tailored for sparse view synthesis, and diffusion-based frameworks. The project url is https://sergeyprokudin.github.io/splatformer.

Decision: accept (spotlight)
Reviewer scores: 6, 8, 8, 8

Reviews and discussion:
<thread>
creation date:
1729337797523

thread_type:
official_review

<thread_object>
creator: reviewer_avem

<content>
summary:
This paper proposes a method towards enhancing the 3DGS performance on out-of-distribution views. This paper leverages a transformer-based point cloud backbone to encode and refine per-scene optimized 3DGS, and supervise on both interpolated views and out-of-distribution views. The empirical results show that this method leads to signficantly improved quality on OOD views.

strengths:
1. The task of enhancing OOD views quality is important and with good motivation. 
2. The idea of training a point cloud backbone to learn priors of OOD views for 3DGS refinement is novel and promising.
3. The experiments are extensive and the results are attractive.
4. The paper is well-written.

weaknesses:
I did not find obvious weaknesses of this paper.

questions:
1. It seems that this method is focused on object-centric scenes with specific camera trajectories (mainly difference in elevations). In both the training datasets and the testing datasets, the input views and OOD views are captured similarly. I'm curious about the results when the trainig views and OOD views are not captured similarly to the training data? For example, if the training views are high-elevation and testing views are low-elevation, or if the training and testing views are with similar elevation but are distant.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730038474068

thread_type:
official_review

<thread_object>
creator: reviewer_mwlj

<content>
summary:
This paper presents SplatFormer, a novel zero-shot model for 3DGS refinement trained on large datasets, in order to enhance the synthesized appearance robustness observed from OOD views. The presented problem OOD-NVS it aims to solve is valuable. Extensive experiments show it achieves SOTA performance on various object-centric datasets.

strengths:
- The presented new problem OOD-NVS is of great value.
- Experiments are extensive, which can well validate the performance of the proposed method.
-  SplatFormer achieves SOTA performance on various object-centric datasets in OOD-NVS task compared to current related methods.

weaknesses:
- Although some experiments using real-world datasets are conducted, all involved datasets are still mainly object-centric. It is still a problem that if this learning-based method can be applied to real-world and non-object-centric scenes with more complex foreground and background. The corresponding data are much more difficult to collect than the object-centric data, and also more difficult to process and use in training.
- Lack of reporting geometry results. Although there are many comparisons in appearance, it's another important problem that how much can the refinement benefit the reconstructed geometry. However, there are no results like depth and surface normal are shown.

questions:
- Would like to see some discussion and exploration for non-object-centric scenes.
- Would like to see more comparisons on geometry, like surface normal and depth.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730277049287

thread_type:
official_review

<thread_object>
creator: reviewer_ma5d

<content>
summary:
Although existing 3D representations such as 3DGS or Nerf can achieve novel view synthesis, their rendering performances on OOD views with relatively large elevations are relatively limited. This may come from the large differences between training and evaluation OOD views. In this work, the authors propose a framework, named SplatFormer, using transformer to refine the optimized 3DGS for better performances under OOD views. Benefited from the training under both normal and OOD views, SplatFormer can indeed improve the rendering under OOD views.

strengths:
1. The core contribution of this work to refine 3D Gaussians with a genelizable transformer is meaningful;
2. The authors construct training and evaluation sets for the claimed OOD problem, from ShapeNet and Objaverse dataset.
3. Extensive experiments with different baselines on multiple datasets confirm that the proposed method can obviously improve the rendering performances under poses with large elevations.

weaknesses:
My major concerns lie on that some comparisons between the proposed method and baselines  may be not so fair. For example, the optimization of the proposed method use 32 low-elevation views, while the results of some methods, e.g., LaRa, take only 4 views for input. The lack of training views may naturally affect its performances. Can we apply the proposed framework to the 3D Gaussian primitives generated by LaRa directly? In this way, the performances of proposed refinement might be evaluated more fairly.

questions:
Except the mentioned problem in the weakness section, I have some other problems.
 1. As the method is mainly proposed to address the problems of rendering under relatively large elevations, the limitation of performances may come from the lack of corresponding training views. Could we just use some novel view synthesis baselines, such as Zero123, SV3D to generate pseudo images from such poses with large elevations, and then optimize 2DGS, 3DGS, etc. for reconstruction? Would this also improve the performances under poses with large elevations?
 2. What is the specific settings for the training of the Gaussian primitive transformer? Would it select input views and OOD views randomly? Does different selection strategies have influences on the final performances?
 3. How is the efficiency of the transformer? As the density of Gaussian primitives might be quite high after optimization, wouldn't it take great time and memory cost to incorporate such a transformer framework?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730687272451

thread_type:
official_review

<thread_object>
creator: reviewer_k5mf

<content>
summary:
The paper introduces SplatFormer, a point transformer model for refining 3D Gaussian Splatting (3DGS) representations under out-of-distribution (OOD) view conditions (with initialized Gaussian Splats). This is motivated by 3DGS struggles with quality degradation when test views differ significantly from training views. SplatFormer addresses this by learning to refine Gaussian splats, leveraging attention mechanisms to maintain consistency across viewpoints and removing artifacts in OOD scenariosl, with the collected large-scale object-centric data. The approach outperforms prior methods in robustness on several test datasets.

strengths:
- The paper introduces a novel and important direction for rendering at unseen, highly relevant test views, addressing a significant gap in current 3D rendering research.
- By employing point transformers for aggregating Gaussian splats, the method offers a sound and efficient approach to achieve improved detail and visual fidelity.

weaknesses:
- The paper does not explore the potential of utilizing generative priors for OOD-NVS, particularly by introducing diffusion models to assist in hallucinating unseen views, which could enhance performance in novel view synthesis in a more reasonable way.
- The study is primarily focused on object-centric cases, despite the availability of scene-level 3D datasets (scannet, scannet++, blendedmvs, megascene, megadepth, mvsimgnet). Expanding the scope to scene-wise data could provide a broader basis for extrapolation and robustness in more complex environments.
- For object-centric cases, single-image-to-3D methods may suffice for preserving geometric consistency and hallucinating texture details. It is unclear why some introduced baselines, including generalizable GS and sparse-view GS, underperform in these scenarios relative to expectations.

questions:
Please refer to the questions in the weaknesses section concerning the problem-solving approach and dataset scope. The reviewer strongly suggests that the authors include a video comparison, as novel view synthesis is highly dependent on visual assessment.



--------------------
Thank you for the detailed explanation and for addressing my concerns. After reviewing the comments from other reviewers and considering your explanation regarding the broader scope of your work, I agree that the merit of addressing OOD challenges in neural rendering using a large-scale model is valuable.

I appreciate the clarification and the balance you’ve struck in presenting the scope of your contributions, the promised clarification in the future abstract and introduction. As a result, I will raise my score to support the acceptance of your submission.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732549102708

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of the authors' responses

comment:
We sincerely thank all reviewers for their valuable comments and constructive suggestions. To facilitate discussion among the reviewers and the area chair, we have summarized the reviewers' feedback in the table below.

|Strengths|R-K5mf|R-MA5d|R-MWLJ| R-aVEM|
|----------|---|---|---|---|
|Addresses a significant research gap | &#10004;  | &#10004; | &#10004;  | &#10004; |
|Proposes a novel and sound method | &#10004;  |  |  | &#10004; |
|Construct novel OOD datasets | | &#10004;   |  | |
|Provides extensive experiments |  | &#10004; | &#10004;   | &#10004; |
|Achieves superior performance| &#10004;  | &#10004; | &#10004;  | &#10004; |
|The paper is well-written|  |  | | &#10004; |
||||||
|**Weaknesses** | | | |  |
|No diffusion priors | ✘|  |  |  |
|Unknown potential in unbounded scenes |✘ |  |  ✘| |
|Insufficient analysis on sparse-view baselines | ✘ | ✘ |  |  |
|Lack of geometry results|  |  | ✘ | |
||||||
|**Rating**| **5**|**6**|**6**|**8**|


We appreciate the reviewers' acknowledgment that our work addresses a significant research gap and their recognition of several strengths, including a novel dataset, extensive experiments, and a technically sound, high-performance method.

To address the reviewers' comments on limitations, we provided additional results and clarifications as follows:
1. **Comparison with Diffusion-based and Sparse-view Methods** (R-K5mf, R-MA5d): In addition to the existing comparisons with SyncDreamer [1], SSDNeRF [12], DiffBIR [13], and LaRa [4] in our original manuscript, we have included more numerical and visual comparisons with state-of-the-art open-source diffusion-based methods [1, 2, 3] and the sparse-view baseline LaRa [4]. We have also analyzed their limitations in the OOD-NVS setup. 
2. **Potential in Unbounded Scenes** (R-K5mf, R-MWLJ): Building on the discussion in the original manuscript, supported by experimental results on MVImgNet [5] (Appendix F), we have expanded it with additional details, considerations for future improvements, and qualitative comparisons.
3. **Geometry Evaluation** (R-MWLJ): We have included qualitative and quantitative evaluations of depth and normal errors.

Furthermore, we addressed additional questions raised by the reviewers:
1. **Video Comparison** (R-K5mf): We directed the reviewer to the video included in our originally submitted supplementary material (_compare_with_major-baselines.mp4_), which provides visual comparison between our method and representative baselines. Additionally, we included a new video (compare_with_diffusion-sparse-baselines.mp4) comparing our method to [1, 2, 3, 4], highlighting the limitations of diffusion-based and sparse-view baselines in terms of hallucination errors and 3D inconsistencies.
2. **Computational Efficiency** (R-MA5d): We evaluated the inference time and memory usage of our model to demonstrate its efficiency. 
3. **Training Setup** (R-MA5d):  We clarified the ratio of OOD views used during SplatFormer's training and included an ablation study on its effect. 
4. **OOD Views Different from Training** (R-aVEM): We demonstrated that our method consistently improves novel test views across diverse viewing angles and distances and discussed its performance under varying input-view trajectories.  

### **Additional Updates**:
* Additional Image Comparisons:  These have been added to Appendix G in the updated manuscript, including:
	* **Figure G.1**: Comparisons with diffusion-based and sparse-view baselines.
	* **Figure G.2**: Additional results in unbounded scenes.
	* **Figure G.3**: Geometry comparisons.
* Video Comparisons: Two videos are now available in the supplementary material:
	* **_compare_with_major-baselines.mp4_**: Originally submitted as part of our supplementary material, this video demonstrates comparisons between our method and major baselines.
	* **_compare_with_diffusion-sparse-baselines.mp4_**: Newly added to address the questions raised by R1-K5mf and R2-MA5d, this video compares our method with diffusion-based and sparse-view baselines.

We sincerely thank all reviewers and the area chair for their time, patience, and thoughtful feedback.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732752907855

thread_type:
public_comment

<thread_object>
creator: ~yiping_ji1

<content>
title:
Question about speed

comment:
Hi Authors, 
Thanks for this great work and I am quite interested in it. I was just wondering if there is extra cost in inference speed using splatformer compared with other methods.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper receives unanimous positive ratings of 6,8,8,8. The AC follows the recommendations of the reviewers to accept the paper. The reviewers comment that the method introduced by the paper is novel and the task is an important direction for rendering unseen test views which addressed a significant gap in the current research.