GSFixer: Improving 3D Gaussian Splatting with Reference-Guided Video
  Diffusion Priors
Authors: name: Xingyilang Yin, name: Qi Zhang, name: Jiahao Chang, name: Ying Feng, name: Qingnan Fan, name: Xi Yang, name: Chi-Man Pun, name: Huaqi Zhang, name: Xiaodong Cun

Reconstructing 3D scenes using 3D Gaussian Splatting (3DGS) from sparse views
is an ill-posed problem due to insufficient information, often resulting in
noticeable artifacts. While recent approaches have sought to leverage
generative priors to complete information for under-constrained regions, they
struggle to generate content that remains consistent with input observations.
To address this challenge, we propose GSFixer, a novel framework designed to
improve the quality of 3DGS representations reconstructed from sparse inputs.
The core of our approach is the reference-guided video restoration model, built
upon a DiT-based video diffusion model trained on paired artifact 3DGS renders
and clean frames with additional reference-based conditions. Considering the
input sparse views as references, our model integrates both 2D semantic
features and 3D geometric features of reference views extracted from the visual
geometry foundation model, enhancing the semantic coherence and 3D consistency
when fixing artifact novel views. Furthermore, considering the lack of suitable
benchmarks for 3DGS artifact restoration evaluation, we present DL3DV-Res which
contains artifact frames rendered using low-quality 3DGS. Extensive experiments
demonstrate our GSFixer outperforms current state-of-the-art methods in 3DGS
artifact restoration and sparse-view 3D reconstruction. Project page:
https://github.com/GVCLab/GSFixer.