Visibility-Uncertainty-guided 3D Gaussian Inpainting via Scene Conceptional Learning

Authors: mingxuan cui, Qing Guo, Yuyi Wang, Hongkai Yu, Di Lin, Qin Zou, Ming-Ming Cheng, Xi Li
Primary area: generative models
Keywords: 3D reconstruction; 3D Inpainting;  Conceptional Diffusion

Abstract:
3D Gaussian Splatting (3DGS) has emerged as a powerful and efficient 3D representation for novel view synthesis. This paper extends 3DGS capabilities to inpainting, where masked objects in a scene are replaced with new contents that blend seamlessly with the surroundings. Unlike 2D image inpainting, 3D Gaussian inpainting (3DGI) faces the challenge of effectively leveraging complementary visual and semantic cues from multiple input views, as occluded areas in one view may be visible in others. To address this, we propose a method that measures the visibility uncertainties of 3D points across different input views and uses them to guide 3DGI in utilizing complementary visual cues. We also employ the uncertainties to learn a semantic concept of the scene without the masked object and use a diffusion model to fill masked objects in the input images based on the learned concept. Finally, we build a novel 3DGI framework VISTA by integrating VISibility-uncerTainty-guided 3DGI with scene conceptuAl learning. VISTA generates high-quality 3DGS models capable of synthesizing artifact-free and naturally inpainted novel views. Furthermore, our approach extends to handling dynamic distractors arising from temporal object changes, enhancing its versatility in diverse scene reconstruction scenarios. We demonstrate the superior performance of our method over state-of-the-art techniques using two challenging datasets: the SPIn-NeRF dataset, featuring 10 diverse static 3D inpainting scenes, and an underwater 3D inpainting dataset derived from UTB180, which includes fast-moving fish as inpainting targets.

Decision: reject
Reviewer scores: 6, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1730216688052

thread_type:
official_review

<thread_object>
creator: reviewer_vfku

<content>
summary:
This paper introduces a framework for 3D scene inpainting using 3D Gaussian Splatting that leverages multi-view information. The framework consists of two key components: VISTA-GI, which guides inpainting by measuring the visibility uncertainty of 3D points across multiple views, and VISTA-CL, which learns scene concepts without masked objects using a diffusion model. These components work iteratively to refine the 3D representation using geometric and semantic information progressively. The method extends to handling dynamic distractors in scenes, making it effective for challenging scenarios like underwater scenes with moving objects. Evaluated on the SPIn-NeRF dataset and a new underwater dataset derived from UTB180, VISTA demonstrates superior performance over state-of-the-art methods in quantitative metrics and visual quality, particularly in maintaining consistency across viewpoints.

strengths:
- Novel use of visibility uncertainty to guide 3D inpainting
- Integration of 3D Gaussian Splatting with diffusion-based scene learning
- Iterative refinement combining geometric and semantic cues
- Strong quantitative results across metrics (PSNR, SSIM, LPIPS, etc.)

weaknesses:
- Missing analysis of how the visibility uncertainty threshold affects results
- The iterative process requires three iterations, potentially making it computationally expensive for real-time applications. And no clear ablation showing the impact of different numbers of iterations beyond Figure 5
- No discussion of failure cases or limitations with large occlusions
- Only tested on ten forward-facing scenes from the SPIn-NeRF dataset - a relatively small evaluation set and scenario.
- Missing key hyperparameters like the size of the adjacent view window for uncertainty calculation
- No evaluation of common real-world scenarios like removing people from tourist photos. Dynamic object types are limited (mostly fish)

questions:
- What thresholds were used for the visibility uncertainty measure in Equation (2)? How sensitive is the method to these thresholds? Could you provide ablation studies showing how different thresholds affect the final results?
- Why exactly are three iterations optimal? The plot in Figure 5 shows diminishing returns, but could more iterations help in challenging cases? What is the runtime for each iteration? Is real-time performance possible with architectural optimizations?
- Could you provide examples of failure cases and discuss their causes?
- Could you evaluate more diverse scenarios, especially tourist photos with people?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730477994309

thread_type:
official_review

<thread_object>
creator: reviewer_qvjd

<content>
summary:
The paper introduces a general 3D Gaussian Splats (3DGS) inpainting framework called VISTA, which achieves 3D inpainting for both static and dynamic objects within 3DGS reconstruction, resulting in enhanced novel view synthesis performance.
By leveraging multiview information, the method extracts visibility uncertainty maps to guide the 3D reconstruction of inpainted scenes. Additionally, masked multiviews are employed to learn the scene concept through textual inversion techniques, which improves the accuracy of image inpainting priors. VISTA iteratively inpaints scenes and learns scene concepts from inter-view visibility information, enhancing scene consistency and preserving scene identity during the 3D inpainting process.

strengths:
1. The work proposed a general 3D Gaussian inpainting framework applicable to both static or dynamic scenes.
2. The proposed VISTA leverages multiview visibility information for more detailed texture consistency, facilitating the inpaiting of unseen areas through conceptual learning.
3. The whole paper is well-written and easy to follow.

weaknesses:
1. Although the comparison experiments are sufficient and the discussions are insightful, there is a lack of certain experiments, such as an ablation study on the effect of conceptual learning. Please see details in Question 1.
2. The paper does not discuss the multi-view inpainting consistency of this method, nor does it compare the results with or discuss differences from those obtained using video diffusion models. See details in Question 2.

questions:
1. As a complex scene (e.g. with objects, backgrounds), the scene feature is not as distinct as ojects, is the concetual learning robust in scenes? How the conceptual learning improve inpainting performance?

2. It would be benefitial to discuss multi-view consitency during inpainting, as well as review and discuss video inpainting works which concerns about multi-view consitency.

3. Minor issues:

   3.1. Fig.1 InFusion's upper results with red boxes appears to be incorrectly cropped in misaligned areas?

    3.2. Fig.2 refers to wrong section numbers.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730676891512

thread_type:
official_review

<thread_object>
creator: reviewer_ema4

<content>
summary:
This paper introduces VISTA (Visibility-Uncertainty-Guided 3D Gaussian Inpainting via Scene Conceptual Learning), a framework specifically designed for 3D scene inpainting. VISTA combines visibility uncertainty maps with scene conceptual learning to improve inpainting quality across multiple viewpoints. The main contributions include VISTA-GI, which guides 3D Gaussian inpainting across views, and VISTA-CL, which uses scene conceptual learning to fill in areas lacking semantic information. Experimental evaluations demonstrate that VISTA outperforms existing methods on the SPIn-NeRF dataset for static scenes and an underwater dataset with dynamic objects.

strengths:
1. VISTA introduces a novel multi-view 3D inpainting framework by combining visibility uncertainty maps with a diffusion model. This approach effectively uses complementary information across multiple views and includes scene conceptual learning to handle semantic filling, showing a certain level of originality.

2. The methodology clearly separates VISTA-GI and VISTA-CL, using an alternating iterative optimization to ensure consistency across views. This design handles both static and dynamic objects, broadening the framework’s flexibility for different application scenarios.

weaknesses:
1. The main contribution of this paper is the introduction of visibility uncertainty to guide inpainting, but the uncertainty map considers only the visibility of 3D points across views without explicitly addressing texture consistency. This may lead to color or texture inconsistencies in the inpainted regions. Further analysis on how the model maintains texture consistency would strengthen this aspect.

2. VISTA-GI is designed to maintain consistency between neighboring views. However, as noted in the appendix, it struggles with scenes that have large viewpoint differences, which haven’t been quantitatively evaluated. Testing on larger viewpoint change would provide a clearer picture of the model's performance in maintaining consistency across wide angles.

3. The framework involves multiple components (visibility uncertainty maps, a diffusion model, and scene conceptual learning), which could increase computational demands. This may impact feasibility in large-scale or real-time applications. Providing more details on computational costs, such as training time, would support its applicability.

4. The paper does not include ablation studies for VISTA-GI and VISTA-CL, making it difficult to assess the individual contribution of each component. Adding these ablation studies would clarify each module’s role in the framework’s overall performance.

5. The paper includes few visual results on the SPIn-NeRF dataset, limiting readers’ understanding of the framework’s effectiveness in static scenes. Providing more visual results across different scenarios, as well as showing the impact of the visibility uncertainty map on static scenes, would help illustrate the method's inpainting capabilities.

6. There is a lack of visualizations of depth maps to verify the geometric accuracy of the inpainting results.

7. The paper does not compare VISTA with GScream[1], which is another 3D inpainting method operating in a similar setting. Including a comparison with GScream would help establish VISTA’s advantages.

[1] Wang, Yuxin, et al. "GScream: Learning 3D Geometry and Feature Consistent Gaussian Splatting for Object Removal" ECCV. 2024.

questions:
1. I still have concerns about texture consistency. Could you explain how the model ensures texture consistency in the inpainting results and provides visualizations of texture across different viewpoints in static scenes?

2. I am curious about the performance drop with large viewpoint differences. Could you quantify this to assess VISTA's robustness to viewpoint variations?

3. Could you conduct ablation studies on ***VISTA-GI*** and ***Scene Conceptual Learning*** to show their specific roles within the overall framework?

4. Could you visualize depth maps to demonstrate the geometric accuracy of the inpainting results?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730688620170

thread_type:
official_review

<thread_object>
creator: reviewer_m4f5

<content>
summary:
This paper presents VISTA, a novel framework for 3D Gaussian inpainting that leverages visibility uncertainty and scene conceptual learning to address the challenges of inpainting in both static and dynamic scenes. The authors propose a method that integrates visibility uncertainty maps to guide the inpainting process, allowing for more coherent and contextually appropriate fills for masked regions. The framework is evaluated on two challenging datasets, demonstrating superior performance compared to existing state-of-the-art methods.

strengths:
•	Originality: The integration of visibility uncertainty with scene conceptual learning is a fresh approach that addresses existing limitations in 3D inpainting. Although the idea is intuitively sample, but the performance is good under the complementary visual cues.

•	Quality: The experiments are robust, showcasing the effectiveness of the proposed method across diverse datasets. The paper also shows some ablation study in discussion and appendix section to support the results.

•	Clarity: The main ideas are presented clearly, and the paper provides a comprehensive overview of related work and the proposed contributions.

•	Significance: The results indicate potential applications in augmented and virtual reality, highlighting the practical implications of the research.

weaknesses:
•	While the paper presents a robust methodology, certain experimental comparisons could be expanded. For instance, a more detailed analysis of the impact of diﬀerent hyperparameters on performance would provide deeper insights, particularly regarding the parameter v in equation (3) and the level of decrease initial noise during inference.

•	There are some writing issues to address: In section 4.1, you state that Figure 1 shows the initial representation’s failure to exclude dynamic objects; however, Figure 1 actually displays the results among diﬀerent approaches rather than the initial representation itself

questions:
1.	Since the iterative process may slow down the pipeline, how does the computational eﬃciency of VISTA compare to other state‐of‐the‐art methods, particularly in real‐time applications? 

2.	In equation (3), Ui represents the visibility uncertainty map for a single frame, and the uncertainty for each pixel is calculated from equation (2). I would like to know if any normalization is applied to Ui between equations (2) and (3).

3.	Regarding the experimental setup illustrated in the appendix, you mention that the initial noise strength during inference decreases by 0.2 with each iteration. Was any ablation study conducted? I agree that reducing the initial noise over iterations is reasonable, as the rendered image quality becomes more conﬁdent with each iteration, but does this hyperparameter aﬀect the ﬁnal result?

4.	The results in Figure 7 show the reconstruction without a mask, which retains the fish in the top-left corner of the image while removing the other fish. I believe the visibility uncertainty used in Equation (3) to update the mask has favorable properties for removing dynamic distractors. However, why does only the top-left fish remain? Could you also provide some statistical evaluation for this ablation study?

5.	I would also like to inquire about the image resolution used for training and inference with stable‐diﬀusion‐v1.5, as the resolution may inﬂuence the inpainting quality in SD‐1.5
</content>
</thread_object>

</thread>
<thread>
creation date:
1732243980613

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to All Reviewers

comment:
# To all reviewers:

Thank you very much to all the reviewers for their constructive and responsible feedback. In this paper, we present an innovative method for removing dynamic and static interference in 3DGS reconstruction. This approach combines geometric information from three-dimensional reconstruction with semantic information from concept learning, facilitating better scene reconstruction of real-world environments. We are delighted that Reviewer **M4f5** emphasized the "Originality" of our method and its "Significance" in the field of virtual reality. Reviewer **eMA4** described our framework as "novel," highlighting its "flexibility" across different scenarios. Reviewer **qvjD** noted that our framework is "general" and considered our experiments to be "sufficient." Reviewer **VfkU** not only highlighted the "novel" use of uncertainty in our method but also pointed out that it achieved "Strong quantitative results." While some reviewers expressed concerns about the details of our method and its performance in more scenarios, we believe our rebuttal can address these concerns. Below are some key responses, and we will provide more detailed replies in our responses to each reviewer.

**List of Changes**

According to the comments of all four reviewers, we have made the following major revisions to our paper.

- We have added the ablation study on VISTA-GI and VISTA-CL in Sec 5.2 to demonstrate the roles of these two components in our algorithm in response to Reviewer **eMA4** and Reviewer **qvjD**.

- We have added a discussion in Sec 5.3 to compare the computing cost with other SOTA methods in response to Reviewer **M4f5**, Reviewer **eMA4**, and Reviewer **VfkU**.

- We have add a discussion in Sec A.1 to clarify the impact of mask on inpainting in response to Reviewer **M4f5**.

- We have presented more dimensional and various types of visualization results for our method in Sec A.2 in response to Reviewer **M4f5** and Reviewer **VfkU**.

- We have added two experiments and some discussions in Sec A.3 to clarify the hyparameters that are related to iteration in response to Reviewer **M4f5** and Reviewer **VfkU**.

- We have added a subsection to discuss the impacts of different resolutions during the training and inference process of diffusion in Sec A.4 in response to Reviewer **M4f5**.

- We have added ablation studies and comparisons on the cases with large viewpoint differences in Sec A.5 in response to Reviewer **eMA4**.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces VISTA, a framework for 3D Gaussian inpainting that integrates visibility uncertainty to enhance inpainting. While the method demonstrates technical novelty and competitive performance, several key concerns from reviewers remain unresolved. These include limited real-world evaluation, insufficient analysis of key parameters, lack of comprehensive ablation studies, and missing valid comparisons with relevant baselines such as GScream. I’d recommend the authors to consider all these and further improve the work.