In-N-Out: Lifting 2D Diffusion Prior for 3D Object Removal via Tuning-Free Latents Alignment
Authors: Dongting Hu
Huan Fu
Jiaxian Guo
Liuhua Peng
Tingjin Chu
Feng Liu
Tongliang Liu
Mingming Gong
Venue: NeurIPS 2024 poster (2024)
Keywords: Neural Radiance Fields
3D Inpainting
Neural Representation Editing
Diffusion Models
Object Removal
Primary area: machine_vision

Abstract: Neural representations for 3D scenes have made substantial advancements recently, yet object removal remains a challenging yet practical issue, due to the absence of multi-view supervision over occluded areas.  Diffusion Models (DMs), trained on extensive 2D images, show diverse and high-fidelity generative capabilities in the 2D domain. However, due to not being specifically trained on 3D data, their application to multi-view data often exacerbates inconsistency, hence impacting the overall quality of the 3D output. To address these issues, we introduce "In-N-Out", a novel approach that begins by inpainting a prior, i.e., the occluded area from a single view using DMs, followed by outstretching it to create multi-view inpaintings via latents alignments. Our analysis identifies that the variability in DMs' outputs mainly arises from initially sampled latents and intermediate latents predicted in the denoising process. We explicitly align of initial latents using a Neural Radiance Field (NeRF) to establish a consistent foundational structure in the inpainted area,  complemented by an implicit alignment of intermediate latents through cross-view attention during the denoising phases, enhancing appearance consistency across views. To further enhance rendering results, we apply a patch-based hybrid loss to optimize NeRF. We demonstrate that our techniques effectively mitigate the challenges posed by inconsistencies in DMs and substantially improve the fidelity and coherence of inpainted 3D representations.

In-N-Out: Lifting 2D Diffusion Prior for 3D Object Removal via Tuning-Free Latents Alignment
Dongting Hu"
Huan Fu"
Jiaxian Guo Tongliang Liu 2,5
Liuhua Peng
Tingjin Chu
Feng Liu"
Mingming Gong 1,5
The University of Melbourne 2 The University of Sydney 3 'Alibaba
Google Research Mohamedd bin Zayed University of Artificial Intelligence
Project Page: https://timmy wtw.github.o//ddr.giiiioi
Abstract
Neural representations for 3D scenes have made substantial advancements recently, yet object removal remains challenging yet practical issue, due to the absence multi-view supervision over occluded areas. Diffusion Models (DMs), trained extensive 2D images, show diverse and high-fidelity generative capabilities in the 2D domain. However, due to not being specifically trained on 3D data, their application to multi-view data often exacerbates inconsistency, hence impacting the overall quality of the 3D output. To address these issues, we introduce "In-N- Out", novel approach that begins by inpainting prior, i.e., the occluded area from a single view using DMs, followed by outstretching it to create multi-view inpaintings via latents alignments. Our analysis identifies that the variability in DMs' outputs mainly arises from initially sampled latents and intermediate latents predicted in the denoising process. We explicitly align of initial latents using a Neural Radiance Field (NeRF) to establish consistent foundational structure in the inpainted area, complemented by an implicit alignment intermediate latents through cross-view view attention during the denoising phases, enhancing appearance consistency across views. To further enhance rendering results, we apply patch- based hybrid loss to optimize NeRF. We demonstrate that our techniques effectively mitigate the challenges posed by inconsistencies in DMs and substantially improve the fidelity and coherence of inpainted 3D representations.
Introduction
have effectively revolutionized 3D scene reconstruction from multi-view images. These models offer high-fidelity novel-view
Neural Radiance Fields (NeRFs) 50
2 23
58 81
37
13 10 3 92
synthesis, proving beneficial across a variety of domains (32 87 88 43 107 the impressive ability reconstruct highly detailed scenes, these learning- bassdd methods depend on the availability of consistent multi-view training data. This reliance limits their generalizability. particularly in editing 3D representations for tasks like object removal and inpainting occluded areas.
6
63 72
60 Despite
Recently, diffusion models (DMs) [30 18 71 80 have gained significant attention in the field of generative modelling for 2D images. These models are well-known for their robustness generative priors, capable of producing diverse and high-fidelity results in 2D inpainting tasks. However, adapting these 2D priors for 3D object removal is not straightforward. While the inherent diversity of DMs benefits the generation of varied outputs, it also poses a significant challenge: high variance in the inpainted results (Fig I middle column). Consequently, these models frequently produce outputs that, while visually appealing in isolation, may appear misaligned when incorporated into 3D
domain [53] 94 93 19, 97 84 This misalignment often results in the loss of high-frequency
details, crucial for realistic and coherent scene rendering.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

View
View
A2
(a) Original Image
(b) Inpainted by Stable Diffusion
(c) Inpainted by Ours
Figure I: Inpainting outcomes of multi-view images from original Stable Diffuion 771 (middle) with those achieved by our approach (right). The inpainted areas are highlighted in red and green boxes.
Previous studies addressing such 3D inconsistencies can be broadly categorized into approaches: multi-view and single-view priors. The former tackled inconsistencies across multi-view inpainted images by optimizing NeRFs with modified objectives (53 94 93 65 91. While these methods have shown promise in refining inconsistent inputs, they sometimes suffer from loss of detail fidelity during the training process, illustrated in Fig. Conversely, other studies have attempted to overcome the multi-view inconsistency bottleneck by anchoring the inpainting process to a single reference image that serves the entire scene 442 [1 [09]. This approach, however, places significant reliance on the selection of an appropriate reference image and the accuracy of depth estimates, which could lead to geometric artifacts during testing, as shown in Fig.
To address these challenges, we aim to overcome 3D inconsistencies by guiding 2D DMs to achieve multi-view consistent inpainting results (Fig. right column). Our analysis reveals that the variance in model outputs primarily comes from the random noise as the initial latent sample, and intermediate latents inferred by the denoising network. Each frame's initial latents are independently sampled, while intermediate latents are individually predicted, highlighting how view-dependent data impacts the generation process. Therefore, our approach focuses on aligning these two critical elements across multiple inputs. We introduce "In-N-Out", a canditional-smmllin like approach that inpaints sampled view and outstretches multiple views. Our method contains three key components:
1. Conditional Inpainting Pipeline: We propose pipeline that first samples inpainting outcome from random view as an inpainting prior. This prior then serves as condition to guide the inpainting process of multiple views, ensuring consistent inpainting foundation.
2. Explicit Latents Alignment: Leveraging the geometry derived from pre-trained NeRF and the inpainting prior, we sample multi-view initial latents conditional on the geometry dictated by the inpainting prior. This ensures that the primary components within the inpainted areas are structurally consistent and align with the underlying 3D geometry.
3. Implicit Latents Alignment: We employ a cross-view attention mechanism during the denoising steps to align predicted intermediate latents concerning the inpainting prior. This enhances the appearance consistency across the inpainted images.
To further enhance our method's performance in the 3D domain, we have implemented a patch-based optimization strategy using hybrid loss on our inpainted multi-view images. This strategy employs perceptual loss rectify spatial mismatches, and adversarial loss to preserve high-frequency details. By addressing these key challenges, our framework effectively handles multi-view inconsistencies and enhances fidelity and coherence of 3D representations. The effectiveness our approach demonstrated through both qualitative and quantitative evaluations of challenging object removal dataset. Our results indicate comprehensive improvement compared to existing methods, highlighting model's 's ability to achieve greater fidelity and consistency inpainted scenes.

---

2 Related Works
2D Editing with Diffusion Models Diffusion models 30
33
105
59
79
80
have revolutionized
image generation with their capacity create highly realistic images. These models facilitate customizable generation via textual prompts [8]
29 69 74). predominantly using pre-trained Stable
Diffusion 771). Several editing methods 122 57 56 611 allow users to adjust images by moving anchor points to new locations. Editing typically begins by inverting the latent representation of the image be edited back to its initial noise [80] with modifications made during the denoising phase.
Prompt-to-Pronee Prompt (P2P) 226 edits images by adjusting the gross-attention between the image and
text. Null-text inversion [55] addresses artifacts in DDIM inversion [80] when using classifier-free
guidance [29]. Delta Denoising Score (DDS) (27] optimizes the latent image representation by aligning the predicted noises of the original and modified texts. Additionally, several studies 7 26]
have identified a relationship between the appearance of images generated by diffusion models and the key-value pairs. While these advancements represent significant progress preserve some content from the original image in 2D image editing, they do not account for multi-view consistency, thus can not be lifted to 3D editing directly.
Lifting 2D diffusion models for 3D editing Recent advancements in 3D editing and generation have effectively utilized 2D DMs to enhance these processes, as demonstrated in various studies [52
776 96 62 97 99
[03]. Pioneering works have used images inferred by DMs for direct supervision. NeeF (IN2N) (25) approached the editing task by transforming
41
95
68
49
3D model editing into a 2D image editing task, utilizing Instruct Pix2Pix (IP2P) (5] to iteratively update 3D scenes. Similarly, ViCA-NeRF [19] addressed editing challenges by modifying reference
images and integrating these changes into the scene. DreamEditor [10]] opted for different strategy by converting NeRF into a mesh for direct optimization. GaussianEditor [1) applies semantic
tracing to identify and modify editing targets within 3D Gaussian Splatting (3DGS) 37 Similarly, Gaussian Grouping [100] implements Identity Encoding for each Gaussian to create masks for editing. Conversely, Score Distillation Sampling (SDS) 64 provides an alternative way to guide 3D representations by backpropagating gradients from diffusion model's denoiser [6 73 71 into
the underlying scene representation. This technique has been effectively applied to generate realistic 3D and 4D scenes using NeRFss
39 08 110
and 3DGS 770 101
15
2D and 3D Inpainting 2D inpainting methods reconstruct images by filling missing content in 73
1022 Early techniques, exemplified by (21]. relied on copying textures from known to unknown regions. LaMa [82] excels in restoring large missing areas using fast Fourier convolutions, extensive receptive fields, and large training masks. Although highly effective at generating plausible background textures within specified masks, LaMa limits the fidelity of its outputs. In contrast, probabilistic diffusion models [30] have shown impressive results in image generation and offer a wide range of inpainted outputs. DMs can be adapted for inpainting without specific training, and modify known regions during each denoising step to fit the task [47]. Similarly, Stable Diffusion [7]] excels at inpainting by operating within latent space, allowing for efficient and effective image generation. In this work, we adopt as our 2D inpainter.
areas defined by mask 20 77 104
86
46
3D scene inpainting aims to fill missing areas within a space, such as removing objects and generating coherent geometry and textures to complete the scene. Although 3D generative models have garnered they are often limited by the scarcity of 3D training data, hence result in poor generalization, particularly in scene inpainting tasks. Therefore, most current 3D inpainting models (53) 51
large interest [4] [34] 38,
89 78
31
85
45
14| 12
109 91) enhance their effectiveness by adopting priors from 2D models. SPIn-NeRF (53) reduces multi-view inconsistencies by first inpainting views and then optimizing NeRF using perceptual loss, NeRFiller [93] tackles multiple frames simultaneously by tiling images for DMs. GaussianEditor 12) edit targets within 3DGS (37), guided by inpainted multi-view images from DMs While these methods show promise, they can sometimes compromise detail fidelity during training. Alternatively, some studies circumvent multi-view inconsistencies by using single reference image for the entire scene [42] [1] 109].
42
65
93
Infusion [009 stands out the inpainting of 3DGS, leveraging a pre-trained depth completion network to infer point clouds from a single inpainted view, though this method depends heavily on precise depth estimates. Concurrent works 655 (54) address these challenges using SDS objective (64
to better align 2D model priors with 3D scene consistency.
3

---

3 Preliminaries
3.1 Neural Radiance Fields
Neural Radiance Fields (NcRFs) 50) represents a breakthrough in 3D rendering by employing a multilayer perceptron (MLP), denoted é to represent scene. This MLP serves continuous volumetric function to capture and reconstruct scene in unprecedented detail. Specifically, NeRFs take as input the view direction d and 3D coordinate r(r) sampled from a camera ray defined by r(7) o + rd. At each position along this ray r(T), the network predicts the volume density and view-dependent color, represented as (a, c). To render camera pixel, NeRFs perform aggregation of the predicted densities and color emissions (T;),, c(Ti) along the camera ray. This process is
mathematically formulated an approximation of a volume rendering integral [48), which used to compute the final color of the pixel:
exp C(r) = -
i-1 with r, exp (x
(I)
where 8(T.) Ti+1 is the distance between adjacent samples along the ray. During the training phase, rays are uniformly sampled from the training images, and the volumetric field optimized using mean square error (MSE) to enhance the accuracy and realism of the rendered scenes.
3.2 Diffusion Models
Diffusion models [30] consist of two processes: forward process that gradually introduces noise to a data sample 20 ~ Pdnm (z), and a learned reverse process that iteratively denoises purely Gaussian
noise sample 2T NO. back into clean image 20. The reverse process is parameterized by conditional noise prediction network ep. trained to predict the noise using the simplified objective:
(2 0:T (c) p(zx") II pol2=-110 lktt,),,, po po(="-1/z',e) = W((--111 N (,))",,,,)))))
o2, (2)
where is the time step in the diffusion process, 2" is an intermediate noisy sample, and e represents a condition (c.g., images, masks, or text). Utilizing deterministic sampler like DDIM (80), the sample zt11 can be obtained by =t~1 = 2 c0 c); note that scaling is omitted for simplicity. In practice, as we use Stable Diffusion 71 latent diffusion as the inpainting backbone, is latent and the generated image is obtained with a decoder 12(2)). Hence, the variability of the generated
image 2 depends solely on initial sampled latent z" and intermediate inferred latents
{2'-1)²
1 1==1"
4 Method
Given set of multi-vier training images {I,);= from the scene with corresponding masks (M,) [= indicate the unwanted object in each frame, our approach seeks to generate consistently inpainted training set { and use them to supervise NeRF. Our approach is structured into three key stages:
Stage 1: Pretrain NeRF & using (I and (MM along with sampled inpainted
prior Ip rough hallucination of the inpaint feature. (Sec. 4.1)
Stage 2: Leverage á to inpaint additional views {ī, | 1,.., conditioned on
the inpainting prior Σ, via explicit and implicit latents alignment. (Sec. 4.2
Stage 3: Using the inpainted image set {ī, L we optimize & with a patch- based hybrid
loss to distill multi-view supervision. (Sec. 4.3
An overview of our method is shown in Fig. 2
4.1 Stage 1: Pre-train NeRF
The initial stage involves training the NeRF on the unmasked region, we follow the original work [00 where simple MSE loss is applied:
Σ |(((()) C(r)
(r)
Rumawked

---

patch
I4
Lre
X
M
pricr
I
Ip
CVA
ELA
ILA
(a) Pretraining NeRF with inpainting prior
(b) Inpainting with latents alignment
Figure 2: Overview of our method. Our approach begins with (a) pre-training the NeRF with sampled inpainting prior Ip from Stable Diffusion 8, detailed in Sec. 4.1 then progresses (b) latent-aligned impainting Z, for multi-view images through Explicit Latents Alignment (ELA) and
Implicit Latents Alignment (II.A), deseribed in Sec. 4.2 Finally, the NeRF is optimized using patch-based hybrid loss strategy outlined in Sec. 4.3 Throughout the training process, we fix Stable Diffusion 0 and update the scene-specific NeRF parameters Φ only.
where Ranmasked represent the unmasked pixels across all the training images. Then we sample prior view Ip with its mask Mp and regularly inpaint it using Stable Diffusion 0. For illustration, we replace of condition in Eq.2 with two components used by Stable Inpainting Diffusion (7]) as for the input prompt, and I for the masked image that fed into the diffusion models. Hence the inpainting process can be formulated as:
f-1 -o I, e), for 1- T,,......!! with 2 N(0, 1)-
The inpainted image then can be obtained by Σ, = ((22). We then use monocular depth estimator on Ip to get depth map Dp. We regress the scale and offset parameters align D with the depth estimated from field & on the unmasked pixels. Hence, we can introduce the geometry and appearance supervision of the inpainting prior Σ, into the NeRF's optimization through:
(c-m)-co-11 (1DD((rr)) Dp(r)
Cprioe (Φ)
(5)
P Rezalook))
where Rmaskedlp) denoted the masked (inpainted) pixels of Ż,. and Do is the depth estimated by NeRF This stage is depicted in Fig. a).
4.2 Stage Latents Alignment
In this section, we introduce our key approach to condition the additional inpainted frames to have an inpainting feature based the prior Σp- As discussed before, in deterministic sampling of the diffusion inpainting model 0, the generation structure and layout highly depend on (I) initial sampled noise* 27 and (2) intermediate predicted latents {2/-1 - 1 Hence if we can align the latents from different views with the prior one, the model is likely to generate multi-view consistent latent 2.
hence image I, In this section, discuss how to align two terms respectively.
5

---

I
T
2p
T
Self Alm
Kps Vp
I
(a) Explicit Initial Latent Alignment (ELA)
(b) Implicit Intermediate Latents Alignment (ILA)
Figure Illustration of two types of Latent Alignment. This figure depicts the Explicit Latents Alignment (ELA) and Implicit Latents Alignment (ILA) processes, as detailed in Sec. 4.2
Explicit Initial Latent Alignment (ELA) Given the sampled latent of prior view 27 used in Stage I (Secc.4.1) we could leverage geometric information to explicitly align the initial latent 3D space. Given that estimated depth Dp. one possible solution is to warp the "p into other views using camera matrices. However, D, is not guaranteed to be accurate hence such hard projection could yield significant errors. Alternatively, we propose to leverage the pre-trained NeRF Φ, as it's a naturally 3D-consistent representation. Specifically, to sample resolution- -rgiinn initial latent 2 (r), we utilize the original formulation of volume rendering (1 but with the substitution of color e by latent S We query the density from Φ, and acquire z" by reprojecting the sampled point to the image plane of the prior latent view z/=
--))= (1-exp---pp (-))))(())))))))
with =T())) = (na((f
(6)
where fp. denote camera perspective projection according to p and i camera matrices. Such soft projection could avoid error accumulation in the inpainting process, and reduce the precision burden on the depth estimator. We illustrate this process in Fig. Ba] There are two key reasons why we propose fine-tuning the NeRF and using NeRF, the geometric is represented by NeRF as a sharp (low variance) unimodal distribution on the ray. Consequently, the aggregated feature remains sharp, preserving the variations in the initial latents. (b) We empirically found the depth prior inferred by the monocular depth estimator is not perfectly aligned with the NeRF. Fine-tuning the NeRF can also benefit this depth prior. Since NeRF Iearns relatively certain geometry in the known (unmasked) areas, this geometry constraint can improve the geometry of neighboring inpainted (masked) areas due their geometric proximity. We compromise the view-dependent effect in NeRF within the ELA module. Due to the heuristic nature of diffusion models, incorporating such view-dependent effects into diffusion models' output remains elusive.
geometric prior for ELA: (a) After finetuning the
Implicit Intermediate Latents Alignment (ILA) While the initial latent could be aligned using the explicit method, intermediate latents are predicted by denoising network eg which is hard to control. We address this issue by exploring the conditioning mechanism of the denoising network in Stable Diffusion (71). Recall that Eq. 4 denoising network еı relies on the input prompt e and masked image Z to predict the noise occurrence in the current step. While we can use the unified prompt for all views to align the text condition in cross-attention of eg. the masked images (Z,) ;=1 inherently different due multi-view nature. Note that I/ condition is introduced based on spatial self-attention (SA) 900 in the U-Net:
QiKY
SA(Q K,, V.) Softmax
V.,
Vd
where Q: obtained from each spatial resolution of the latent, K,, V are derived from corresponding latent encoded from masked image I/. We can impose the coherence of the denoising step by introducing cross- cross-view view attention (CVA) of the prior view (Fig, 3bb.
CVA(Q,, Kp, Vo)) = Softmax (((Kg)))
Vp,
(8)
Vd

---

where Kp: Vp are from masked base image T. We can then implicitly align the denoising step by replacing the original SA with a weighted sum from SA and CVA, i.. d * SA(Q,, K,, V.) + +11 Aa) * CV, kp, Vp)). Through this technique we ensure that the intermediate latents {2ī fxf-11 } are also conditioned on the prior 1==, while retain its distinctiveness due to the individual viewpoint. The rationale of replacing KV with "prior" p. but not Q is that the appearance information (V) of the prior image should be considered when inpainting the other views, with the amount of information propagation is weighted by its attention key value (K). The attention query value comes from the current inpainting view i, Q; representing the information the current inpainting for view i searching for. Together with Kp. decides how much attention the view inpainter should place on the prior view, and finally incorporates the information of the prior view Vp into view i.
4.3 Stage 3: Joint Optimization
As the original intention of this work, we seek distill the inpainted views into NeRF Φ in way such that the high-fidelity is preserved as much as possible as the unmasked region. While some priors work in 3D editing 25 96 97 931 propose to update the training set iteratively until converge, we empirically find it not suitable for inpainting task since the loss of fidelity is significant and could fall into local optima. Hence we propose to inpaint a subset of training images once and regard them as supplementary guidance using a patch- patch-based hybrid loss:
Σ 110000 - I1will, ((())) + Cipipa( (o((), + Lan (fa,( PEPub
pucch (o)
(9)
where p is a patch sample from the masked area of subset views Puub. (p) is NeRF predicted patch, and Lipips, Lady are perceptual distance LPIPS [106] and adversarial loss (24). Here LPIPS is utilized to address geometry mismatches, while adversarial loss is employed to preserve high-frequency details. As shown in Fig. 2 the final optimization objective is:
[()) = (6) + ( + (pcnchCuchhh)).
(10)
The patches are uniformly sampled within the bounding box of the mask with size of 256x256. Therefore, only the inpainted area is being optimized by the patch loss.
5 Experiment
5.1 Evaluation Setting
Aligning with methodologies employed in prior works [53 51 109,, our experiments utilize the SPIn-NeRF dataset [53), selected for its comprehensive ground truth availability. This dataset is specifically designed for object removal evaluations and comprises 10 scenes. Each scene includes 60 images featuring an unwanted object (training views) and 40 images from which the object has been removed (test views). For both the training and test views, human-annotated masks indicating the object region are available. We further collected 9 forward-facing scenes with manually annotated masks to evaluate the effectiveness of our method. This dataset includes 4 indoor scenes and 5 outdoor scenes. In the training set, the masked region contains the unwanted object, while the test set contains the ground truth background in the masked region.
Dataset:
Baselines: In our study, we benchmark our method against a variety of established 3D inpainting approaches to ascertain its relative performance. These include the perceptual- based SPIn-NeRF (53). tiling-based NeRFiller [93] (both multi-view guidance) and InFusion [109] (single-view guidance). To ensure a fair comparison, we employ the same inpainting diffusion models across all methods and maintain consistency in the number of denoising steps and used prompts. We utilized the source code provided by the authors and ran all the methods using one NVIDIA A 100 (80G) GPU.
Metrics: To quantitatively evaluate the effectiveness of our approach, we employ two similarity metrics: LPIPS [106] and FID (28). Additionally, we use MUSIQ (36), sharpness metric that quantifies the clarity and detail retention in the edited images. Following established protocols from previous studies 53 all metrics are calculated specifically within the bounding boxes defined by the
masks, focusing the evaluation precisely on the regions most affected by the object removal task.
7

---

Training Inputs
SPIn-NeRF
NeRFiller
InFusion
Ours
Figure 4: Qualitative results on the SPIn-NeRF the coll-collected dataset.

---

5.2 Main Results
We first present the quantitative results in Tab. I where our method outperforms all baselines in terms of similarity metrics and sharpness. Our approach also excels in qualitative assessments, important to note that in simpler scenes with low variability in inpainting results, where the inconsistency issue is less pronounced (first row), most methods perform adequately. In other cases, high-frequency loss observed multi- methods (SPIn-NeRF and NeRFiller). NeRFiller [93], through its use of multiple joint denoising steps, ensures consistency but often produces overly smooth outputs that lack fine details. It is noteworthy that the single- view-based method, InFusion [109], relies one view and its depth to represent the entire scene. performs well when geometry estimation accurate. However, its performance deteriorates in scenarios where depth accuracy is compromised, leading to geometry artifacts (sixth and seventh rows). This underscores the critical role of multi-view supervision addressing such challenges. By incorporating consistent multi-vinn view supervision, geometry is inaccurate, achieving robust and promising results. This explains why our method shows little difference from InFusion when the geometry is accurate (first and third row), but excels when the depth is inaccurate. Additionally, the exclusive reliance on perceptual loss by SPIn-NeRF [53] fails to fully address the multi-view inconsistencies introduced by inpainting diffusion models, often resulting in a blurred effect, particularly visible in the third and fourth rows. To further validate our findings, conducted a user study based on the SPIn-NeRI dataset, focusing on the coherence of the background within the inpainted area, the fidelity of detail preservation in the inpainted region, and overall preference. The results of this study are summarized in Tab. 2 This evaluation clearly demonstrates superior performance across all assessed criteria.
as demonstrated in Fig. 4
method remains effective even when depth or
Table 1: Quantitative Results
Table 2: User Study
Method
LPIPS
FID
MUSIQ f
Method
Coherence
Fidelity
Overall
SPIn-NeRF 53
0.54
185.63
38.69
SPIn-NeRF (53
22.72%
20.45%
21.82%
NeRFiller 93
0.71
315.83
32.60
NeR Filler
93
2.73%
4.33%
2.50%
InFusion 109
0.62
153.77
39.29
InFusion
100
27.50%
24.77%
25.00%
Ours
0.49
130.92
50.97
Ours
47.05 %
50.45%
50.68 %
5.3 Ablation Studies
We initially demonstrate the efficacy of our latents alignment approach with an example in Fig. 5 The first column displays the inpainting prior (sampled view), and the subsequent columns show the same training image being inpainted under different conditions. Notably, the variant without ELA (w/o ELA) retains colors similar the prior but fails to preserve the texture structure. Conversely, the version without ILA (w/o ILA) maintains structural integrity but lacks appearance consistency with the prior. Our method effectively merges the strengths of both mechanisms, resulting in inpaintings that are highly consistent and cohesive across all evaluated aspects.
Inpainting prior
Ours w/o ELA
Ous wio ILA
Ours
Figure 5: Ablation study on latent aligned onpainting. 2D Inpainting results when key components of our proposed method are omitted. Naive inpainting using Stable Diffusion can refer to Fig.
We conducted further ablation studies to underscore the importance of our key design elements in object removal tasks. The quantitative and qualitative results, showcased in Tab. Tab. and Fig. 6 clearly indicate the impact of each component. Notably, removing ELA leads to geometry mismatches in the NeRF outputs (w/o ELA), while deactivating ILA results in blurry coloration (w/o ILA)) This observation confirms our initial findings: the initial latents primarily influence the inpainting's structural pattern, whereas the intermediate denoising steps largely affect its appearance, including

---

colour nuances. Additionally, our patch-based loss plays a crucial role in the optimization process (wlo pacch). Specifically, the Lipps loss helps alleviate geometry mismatches (w/o Lp ). and the frnd serves as a Sreeil-preserving supervisor (w/o ). These results highlight the effectiveness of our design choices enhancing the overall quality and coherence of the inpainted outputs.
Table 3: Quantitative Results of Ablation Study.
Method
LPIPS
FID 4
MUSIQ 1
Ours w/o ELA
0.52
133.09
48.90
Ours w/o ILA
0.50
141.78
49.70
Ours w/o Lyacch
0.73
293.32
33.76
Ours Lipips
0.55
223.31
46.07
Ours w/o Lady
0.51
134.70
49.88
Ours full model
0.49
130.92
50,97
Ours full model
Ours ELA
Ours wio ILA
Ours w/o Cparch.
Ours wlo 4igip
Ours Cade
Figure 6: Ablation study on design choices based on rendering quality. This figure displays rendering results from NeRF when key components are individually removed from our full model.
Conclusion
In this work, we demonstrate the significant improvement achieved through our novel latents align- ment approach in 3D object removal. By integrating both explicit and implicit latent alignment mechanisms, we have successfully addressed key challenges associated with geometry mismatches and color inconsistencies that are prevalent in the baselines, enhancing the fidelity and detail of the inpainted 3D scenes. The improvements achieved through our work offer significant societal benefits, such enhanced editability of radiance fields. However, it also poses risks, including the potential perpetuation of biases and discrimination. If the data used to train diffusion models is biased, our approach could inadvertently reinforce these biases.
Despite notable advancements, our method has limitations: (1) struggles with full 3D consistency, especially on high-frequency details, due to the constraints of applying 2D diffusion models to multi-view data. Future work could address this by integrating multi-view training into 2D inpainting diffusion models or leveraging true 3D generative models. (2) It is tailored for forward- scenes, limiting its applicability to diverse 360 views. Further exploration of latent relationships for broader view coverage is needed. (3) Predefined masks are currently required. Integrating advanced 3D perception methods [66] 677 could enhance accuracy and flexibility, enabling precise language-dring driven interactions and creating more automated, user-friendly framework for neural 3D scene editing.
7
Acknowledgements
This research was mainly undertaken using the LIEF HPC-GPGPU Facility hosted at the University of Melbourne. This Facility established with the assistance of LIEF Grant LE170100200. This research was also partially supported by the Research Computing Services NCI Access scheme at the University of Melbourne. DH supported by the Melbourne Research Scholarship from the University of Melbourne. FL is supported by the Australian Research Council (ARC) with grant numbers DP230101540 and DE240101089, and the NSF&CSIRO Responsible AI program with grant number 2303037.
10

---

References
[1] Y. Balaji, S. Nah, X. Huang, A. Vahdat, J. Song, Q. Zhang, K. Kreis, M. Aittala, T. Aila, S. Laine, al. ediff-i: Text-to-image diffusion models with ensemble of expert denoisers. arXiv preprint arXiv:2211.011 0.3224, 2022.
[2] J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pages 5470-5479, 2022.
[3] J. T. Barron, B. Mildenhall, D. Verbin, P. Srinivasan, and P. Hedman. Zip-nerf: Anti-aliased grid-based neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19697-19705, 2023.
[4] M. A. Bautista, P. Guo, S. Abnar, W. Talbott, A. Toshev, Z. Chen, L. Dinh, S. Zhaai, H. Goh, Ulbricht, al. Gaudi: neural architect for immersive 3d scene generation. Advances in Neural Information Processing Systems, 55:25102-25111 2022.
[5] T. Brooks, A. Holynski, and A. A. Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392-18402, 2023.
[6] A. Cao and J. Johnson. Hexplane: A fast representation for dynamic scenes. CVPR, 2023.
[7] M. Cao, X. Wang, Z. Qi. Y. Shan, X. Qie, and Y. Zheng. Masactrl: Tuning-free free mutual self- attention control for consistent image synthesis and editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 22560-225700 October 2023.
[8] J. Cen, Z. Zhou, Fang, Yang. W. Shen, Xie, Zhang, and Q. Tian. Segment anything in 3d with nerfs. In NeurIPS, 2023.
[9] L. Cerkezi, Davtyan, Sameni, and P. Favaro. Multi-view unsupervised image generation with cross attention guidance. arXiv preprint arXiv:2312.04337, 2023.
[10] A. Chen, 7. Xu, A. Geiger, J. Yu, and H. Su. Tensorf: Tensorial radiance fields. In European Conference on Computer Vision (ECCV), 2022.
[11] R. Chen, Y. Chen, N. Jiao, and K. Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22246-2222 22256, 2023.
[12] Chen, Z. Chen, C. Zhang, Wang, X. Yang. Y. Wang, Z. Cai, L. Yang. H. Liu, and G. Lin. Gaussianeditor: Swift and controllable 3d editing with gaussian splatting, 2023.
[13] Z. Chen, T. Funkhouser, Hedman, and A. Tagliasacchi. Mobilenerf: Exploiting the polygon rasterization pipeline for efficient neural field rendering on mobile architectures. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 16569- 16578, 2023.
[14] Y.-. Cheng, H.-Y. Lee, S. Tulyakov, A. G. Schwing, and L.-Y. Gui. Sdfusion: Multimodal 3d shape completion, reconstruction, and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4456-4465, 2023.
[15] J. Chung, S. Lee, H. Nam, J. Lee, and K. M. Lee. Luciddreamer: Domain-free generation of 3d gaussian splatting scenes. arXiv preprint arXiv:231 2. 133384, 2023.
[16] X. Dai, Hou, C.-Y. Ma, S. Tsai, J. Wang. R. Wang, P. Zhang, S. Vandenhende, X. Wang, A. Dubey, et al. Emu: Enhancing image generation models using photogenic needles in haystack. arXiv preprint arXiv:2309.1550 15807, 2023.
[17] K. Deng, A. Liu, J.-Y. Zhu, and D. Ramanan. Depth-superrised NeRF: Fewer views and faster training for free. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition (CVPR), June 2022.
11

---

[18] P. Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-877 2021.
[19] J. Dong and Y.-X. Wang. Vica-nerf: View-consisteney-awaal 3d editing of neural radiance fields. Advances in Neural Information Processing Systems, 36, 2024.
[20] Q. Dong, C. Cao, and Y. Fu. Incremental transformer structure enhanced image inpainting with masking positional encoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11358-11368, 2022.
[21] A. A. Efros and T. K. Leung. Texture synthesis by panaaarmettic sampling. In Proceedings of the seventh IEEE international conference on computer vision, volume 2, pages 1033-1038. IEEE, 1999.
[22] Epstein, A. Jabri, B. Poole, A. Efros, and A. Holynski. Diffusion self-guidance for controllable image generation. Advances in Neural Information Processing Systems, 36: 16222-16239, 2023.
[23] S. Fridovich- Keil, A. Yu, M. Tancik, Q. Chen, B. Recht, and A. Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 5501-5510, 2022.
[24] I. Goodfellow, J. Pouget-Abadiee M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[25] A. Haque, M. Tancik, A. A. Efros, A. Holynski, and A. Kanazawa. Instruct-nerfZnerft Editing 3d scenes with instructions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19740-19750, 2023.
[26] A. Hertz, R. Mokady, J. Tenenbaum, K. Aberman, Y. Pritch, and D. Cohen-On Prompt-to- prompt image editing with cross attention control. arXiv preprint arXiv:2208.011: 2022.
[27] A. Hertz, K. Aberman, and D. Cohen-Or. Delta denoising score. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2328-2337, 2023.
[28] M. Heusel, Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[29] J. Ho and Salimans. Classifier- free diffusion guidance. arXiv preprint arXiv:220711259: 2022.
[30] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[31] Y. Hong, K. Zhang, J. Gu, S. Bi, Y. Zhou, D. Liu, F. Liu, K. Sunkavalli, T. Bui, and H. Tan. Lrm: Large reconstruction model for single image to 3d. arXiv preprint (tt::2311.04400 2023.
[32] B. Hu, J. Huang, Y. Liu, Y.-. Tai, and C.-K. Tang. Nerf-rpn: A general framework for object detection in nerfs. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, June 2023. doi: 10.1 11009ccp552229.2223.222222 //dx. doi. www.ddo..oggo.oooooo org/ 10. 1119 CVVRS2/P222299..... 2023. 02253
URL http
[33] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2 10600965, 2021.
[34] N. Kalischek. T. Peters, D. Wegner, and K. Schindler. Tetrahedral diffusion models for 3d shape generation. arXiv preprint arXiv:2211.13222 2022.
[35] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 81 11108119 2020.
12

---

[36] J. Ke, Q. Wang. Y. Wang. P. Milanfar, and F. Yang. Musiq: Multi-scale image quality transformer. In Proceedings the IEEE/CVF international conference on computer vision, pages 5148-5157, 2021.
[37] Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis. 3d gaussian splatting for real- time radiance field rendering. ACM Transactions on Graphics, 42(4), July 2023. URL https://repo:
inria.frr www.ff/rrff/////a/ee -
[38] S. W. Kim, B. Brown, K. Yin, K. Kreis, K. Schwarz, D. Li, R. Rombach, A. Torralba, and S. Fidler. Neuralfield-ldm: Scene generation with hierarchical latent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 8496-8506, 2023.
[39] Y. Liang, X. Yang, J. Lin, H. Li. X. Xu, and Y. Chen. Luciddreamer: Towards high-fidelity text-to-3d generation via interval score matching. arXiv preprint arXiv:2311.11244 2023.
[40] C.-H. Lin, J. Gao, L. Tang, T. Takikawa, X. Zeng, X. Huang, K. Kreis, S. Fidler, M.-Y. Liu, and T.-Y. Lin. Magic3d: High-resolution text-to-3d content creation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 300-309, 2023.
[41] Y. Lin, H. Han, C. Gong, Z. Xu, Y. Zhang, and X. Li. Consistent123: One image to highly consistent 3d asset using case-aware diffusion priors. arXiv preprint arXiv:2309. 2023.
[42] H.-K. Liu, I. Shen, B.-Y. Chen, et al. Nerf-in: Free-form nerf inpainting with rgb-d priors.
arXiv preprint 200.:2206.44901, 2022.
[43] K. Liu, F. Zhan, Y. Chen, J. Zhang, Y. Yu, A. E. Saddik, S. Lu, and E. Xing. Stylerf: Zero-shot 3d style transfer of neural radiance fields. 2023.
[44] Z. Liu, Y. Feng, M. J. Black, D. Nowrouzezahrai, L. Paull, and W. Liu. Meshdiffusion: Score-based generative 3d mesh modeling. arXiv preprint arX1v:2303.08133, 2023..
[45] X. Long, Y.-C. Guo, C. Lin, Y. Liu, Z. Dou, L. Liu, Y. Ma, S.-H. Zhang, M. Habermann, C. Theobalt, al. Wonder3d: Single image to 3d using cross doms-n diffusion. arXiv preprint arXiv:2310. NDtv:2310.10000 15008, 2023.
[46] Z. Lu, J. Jiang. J. Huang, G. Wu, and X. Liu. Glama: Joint spatial and frequency loss for general image inpainting. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1301-1310, 2022.
[47] A. Lugmayr, M. Danelljan, A. Romero, F. Yu, R. Timofte, and L. Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11461-11471, 2022.
[48] N. Max. Optical models for direct volume rendering. IEEE Transactions on Visualization and Computer Graphics, 1(2):99-100 108, 1995.
[49] Metzer, E. Richardson, O. Patashnik, Giryes, and D. Cohen-Or. Latent-nerf for shape- guided generation of 3d shapes and textures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12663-162 12663-12673, 2023.
[50] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
[51] Mirzaei, nis, and I. Gilitschenski. Reference-guided controllable inpainting of neural radiance fields. In ICCV, 2023.
Soveentad-AAmstrorn M. A. Brubaker, Kelly, A. Levinshtein, K. G. Derpa-
[52] A. Mirzaei, T. Aumentado-AAmstrongg M. A. Brubaker, J. Kelly, A. Levinshtein, K. G. Derpa- nis, and Gilitschenski. Watch your steps: Local image and scene editing by text instructions. arXiv preprint arXiv:2308.0880 2023.
13

---

[53] A. Mirzaei, T. ski, and A. Levinshtein. SPIn-NeRF: Multiview segmentation and perceptual inpainting with neural radiance fields. In CVPR, 2023.
K. G. Derpanis, J. Kelly, M. A. Brubaker, I. Gilitschen-
[54] A. Mirzaei, R. De Lutio, S. W. Kim, D. Acuna, J. Kelly, S. Fidler, I. Gilitschenski, and Z. Gojeic. Reffusion: Reference adapted diffusion models for 3d seene inpainting. arXiv preprint arXiv:2404.10760 2024.
[55] R. Mokady, A. Hertz, K. Aberman, Y. Pritch, and D. Cohen-Or, Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6038-6047, 2023.
[56] C. Mou, X. Wang, J. Song, Y. Shan, and J. Zhang. Dragondiffusion: Enabling drag-style manipulation on diffusion models. arXiv preprint arXiv:2307.02421, 2023.
[57] C. Mou, X. Wang, J. Song. Y. Shan, and J. Zhang. Diffeditor: Boosting accuracy and flexibility on diffusion- based image editing. arXiv preprint arXiv:2402.02588 2022.
[58] T. Müller, A. Evans, C. Schied, and A. Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM transactions on graphics (TOG), 41()):1-15 2022.
[59] A. Q. Nichol and P. Dhariwal. Improved denoising diffusion probabilistic models. In Interna- tional conference machine learning, pages 8162-8171. PMLR, 2021.
[60] M. Niemeyer and A. Geiger. Giraffe: Representing scenes as compositional generative neural feature fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11153-11 11464, 2021.
[61] X. Pan, A. Tewari, T. Leimkühler, L. Liu, A. Meka, and C. Theobalt. Drag your gan: Interactive point-based manipulation on the generative image manifold. In ACM SIGGRAPII 2023 Conference Proceedings, pages 1-11, 2023.
[62] J. Park, G. Kwon, and J. C. Ye. Ed-nerf: Efficient text-guided editing of 3d scene using latent space nerf. arXiv preprint arXiv:2310.02712, 2023.
[63] K. Park, U. Sinha, J. T. Barron, S. Bouaziz, D. B. Goldman, S. M. Seitz, and R. Martin-Brualia Nerfies: Deformable neural radiance fields. Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 5865-5874, 2021.
[64] B. Poole, A. Jain, J. T. Barron, and B. Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.144: 2022.
[65] K. Prabhu, J. Wu, L. Tsai, P. Hedman, D. B. Goldman, B. Poole, and M. Broxton. Inpaint3d: 3d scene content generation using 2d inpainting diffusion. arXiv preprint arXiv:23 2023.
[66] M. Qin, W. Li. J. Zhou, H. Wang, and H. Pfister. Langsplat: 3d language gaussian splatting. arXiv preprint arXiv:2312.16088 2023.
[67] Y. Qu, S. Dai, X. Li. J. Lin, L. Cao. S. Zhang, and Ji. Goi: Find 3d gaussians of interest with an optimizable open-vocabulary semantic-space hyperplane. arXiv preprint 2xXx:2245.1/550 2024.
[68] A. Raj. S. Kaza, B. Poole, M. Niemeyer, N. Ruiz, B. Mildenhall, S. Zada, K. Aberman, M. Rubinstein, J. Barron, al. Dreambooth3d: Subject-driven text-to-3d generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2349- 2359, 2023.
[69] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06/22, 1(2):3 2022.
[70] J. Ren, I., Pan, J. Tang, Zhang, A. Cao, 7eng, and Liu. Dreamgaussian4d: Generative 2023.
4d gaussian splatting, arXiv preprint
14

---

[71] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognifion, pages 10684-10695, 2022.
[72] V. Rudnev, M. Elgharib, W. Smith, L. Liu, V. Golyanik, and C. Theobalt. Nerf for outdoor scene relighting. In European Conference Computer Vision (ECCV), 2022.
[73] Saharia, W. Chan, H. Chang, C. Lee, Ho, T. Salimans, D. Fleet, and M. Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 conference proceedings, pages 1-10, 2022.
[74] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gon- tijo Lopes, B. Karagol Ayan, T. Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35: 2022.
[75] A. Sargsyan, S. Navasardyan, X. Xu, and H. Shi. Mi-gan: A simple baseline for image inpainting on mobile devices. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7335-7345, 2023.
[76] E. Sella, G. Fiebelman, P. Hedman, and Averbuch-Flor, Vox-e: Text-guided voxel editing of 3d objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 430 440, 2023.
[77] P. Shamsolmoali, M. Zareapoor, and E. Granger. Transinpaint: Transformer-basse image inpainting with context adaptation, Proceedings of the IEEE/CVF International Conference Computer Vision, pages 849-858, 2023.
[78] Siddiqui, A. Alliegro, Artemov, Tommasi, D. Sirigatti, V. Rosov, A. Dai, and M. Nießner. Meshgpt: Generating triangle meshes with decoder-only transformers. arXiv preprint arXiv:2311.15475 2023.
[79] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2265.2265. PMLR, 2015.
[80] J. Song, C. Meng, and Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020.
[81] Sun, M. Sun, and H.-T. Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5459-5469, 2022.
[82] R. Suvorov, E. Logacheva, A. Mashikhin, A. Remizova, A. Ashukha, A. Silvestrov, N. Kong, H. Goka, Park, and V. Lempitsky. Resolution-robust large mask inpainting with fourier convolutions. In Proceedings of the IEEE/CVF winter conference applications of computer vision, pages 2149-2159, 2022.
[83] M. Tancik, E. Weber, E. Ng. R. Li, B. Yi, J. Kerr, T. Wang, A. Kristoffersen, J. Austin, K. Salahi, A. Ahuja, McAllister, and A. Kanazawa. Nerfstudio: A modular framework for neural radiance field development. In ACM SIGGRAPH 2023 Conference Proceedings, SIGGRAPH '23, 2023.
[84] J. Tang, J. Ren, H. Zhou, Z. Liu, and G. Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv.2309.1663: 2023.
[85] J. Tang, Z. Chen, X. Chen, T. Wang, G. Zeng, and Z. Liu. Igm: Large multi- gaussian model for high-resolution 3d content creation. arXiv preprint arXiv:2402.00 05054, 2024.
[86] L. Tang, N. Ruiz, Q Chu, Y. Li, A. Holynski, D. E. Jacobs, B. Hariharan, Y. Pritch, N. Wadhwa, K. Aberman, al. Realfill: Reference-driven generation for authentic image completion. arXiv preprint arXiv:2309.1666 2023.
15

---

[87] M. A. Uy, R. Martin-Brualla Brualla, L. Guibas, and K. Li. Scade: Nerfs from space carving with ambiguity-awer depth estimates. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[88] M. A. Uy, R. Martin-Brualla, L. Guibas, and K. Li. Scade: Nerfs from space carving with ambiguity-awan depth estimates. In Proceedings of the Vision and Pattern Recognition. pages 16518-16522 2023.
the IEEE/CVF Conference on Computer
[89] A. Vahdat, F. Williams, Z. Gojcic, O. Litany, S. Fidler, K. Kreis, et al. Lion: Latent point diffusion models for 3d shape generation. Advances in Neural Information Processing Systems, 35:10021-1003, 2022.
[90] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[91] D. Wang. T. Zhang, A. Abboud, and S. Süsstrunk. Inpaintnerf360:: Text-guided 3d inpainting on unbounded neural radiance fields. arXiv preprint M.0Xiv::335.500000 2023.
[92] Q. Wang, Z. Wang, K. Genova, P. P. Srinivasan, H. Zhou, J. T. Barron, R. Martin-Brualla N. Snavely, and T. Funkhouser. Ibrnet: Learning multi-view image- based rendering. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4690-4699, 2021.
[93] E. Weber, A. Holynski, V. Jampani, S. Saxena, N. Snavely, A. Kar, and A. Kanazawa. Nerfiller: Completing scenes via generative 3d inpainting. CVPR, 2024.
[94] S. Weder, G. Garcia-Hernan Hernando, Á. Monszpart, M. Pollefeys, G. Brostow, M. Firman, and S. Vicente. Removing objects from neural radiance fields. In CVPR, 2023.
[95] H. Weng. T. Yang, J. Wang, Y. Li, T. Zhang, C. Chen, and L. Zhang. Consistent123: Improve consistency for one image to 3d object synthesis. arXiv preprint arXiv:2310.08092, 2023.
[96] J. J.WW. Bian, X. Li, G. Wang, I. Reid, P. Torr, and V. A. Prisacariu. Gaussetrl: Multi-view consistent text-driven 3d gaussian splatting editing. arXiv preprint arXiv:2403.08733 2024.
[97] R. Wu, B. Mildenhall, P. Henzler, K. Park, R. Gao, D. Watson, P. P. Srinivasan, D. Verbin, T. Barron, B. Poole, al. Reconfusion: 3d reconstruction with diffusion priors. arXiv preprint arXiv:2312.02981, 2023.
[98] L. Yang. B. Kang, Z. Huang, X. Xu, J. Feng, and H. Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In CVPR, 2024.
[99] J. Ye, P. Wang, K. Li, Y. Shi, and H. Wang. Consistent-I-to-3. Consistent image to 3d view synthesis via geometry-aware diffusion models. arXiv preprint arXiv:2310.03020, 2023.
[100] M. Ye. M. Danclljan, F. Yu, and L. Ke. Gaussian grouping: Segment and edit anything in 3d scenes. arXiv preprint 23/.::23/2207332 2023.
[101] T. J. J. Fang, J. Wang, G. Wu, L. Xie, X. Zhang, W. Liu, Q. Tian, and X. Wang. Gaussian- dreamer: Fast generation from text to 3d gaussians by bridging 2d and 3d diffusion models. In CVPR, 2024.
[102] T. Yu, R. Feng, R. Feng. J. Liu, X. Jin, W. Zeng, and Z. Chen. Inpaint anything: Segment anything meets image inpainting. arXiv preprint EXXx:2304.06790 2023.
[103] J. Zhang, Z. Tang, Y. Pang, X. Cheng, P. Jin, Y. Wei, W. Yu, M. Ning, and L. Yuan. Repaint123: Fast and high-quality image 3d generation with progressive controllable 2d repainting. arXiv preprint arXiv:2312. 13271, 2023.
[104] L. Zhang, Y. Zhou, C. Barnes, S. Amirghodsi, Z. Lin, E. Shechtman, and J. Shi. Perceptual artifacts localization for inpainting. European Conference on Computer Vision, pages 146-164. Springer, 2022.
16

---

[105] L. Zhang, A. Rao, and M. Agrawala. Adding conditional control to text-to- text-to-image diffusion models, 2023.
[106] Zhang, Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features perceptual metric. In CVPR, 2018.
[107] M. Zheng, Z. Haiyu, H. Yang. and D. Huang. Neuface: Realistic 3d neural face rendering from multi-view images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[108] Y. Zheng, X. Li, K. Nagano, S. Liu, O. Hilliges, and S. De Mello. unified approach for text-and image-guided 4d scene generation. arXiv preprint arXiv:2311. 1.6224411,.,4, 2023.
[109] Q. W. K. L. C. J. X. K. N. X. Y. L. Y. S. Y. C. Zhiheng Liu, Hao Ouyang. Infusion: Inpainting 3d gaussians via learning depth completion from diffusion prior. arXiv preprint arXiv:2404. / 1613, 2024.
[110] J. Zhuang, C. Wang, L. Lin, L. Liu, and G. Li. Dreameditor: Text-driven 3d scene editing with neural fields. In SIGGRAPH Asia 2023 Conference Papers, pages 1-10, 2023.
17

---

A Supplemental Material for "In-N-Out: Lifting 2D Diffusion Prior for 3D Object Removal via Tuning-Free Latents Alignment"
A.1 Implementation Detail
For the inpainting network, we employ the taablediiffusion---nnpaa- masked image into the same dimensional latent space and integrates conditions via concatenation. We set the denoising steps for inpainting at 20. To achieve better generalization, we propose sampling the base frame according to the gcometrical centroid of the training camera poses, meaning the camera that sits most centrally among the training views. However, occasionally inpaints strange artifacts in the masked region. To mitigate this, we propose sampling n candidate views around the geometrical centroid and selecting the one with the highest similarity votes. This approach automatically avoids such occasion artifacts without human intervention. In our implementation, we used five candidate views, and the similarity was calculated using perceptual hashing. In the reprojection procedure of ELA, we adjust the camera intrinsics to match the latent dimensionality. Furthermore, to refine the ILA mechanism, we incorporate Cross- View Attention (CVA) into every self aelf-attention layer of the inpainting model. Each step in this modified approach is controlled with D set 0.2.
model 77],, which encodes
found that Stable Diffusion
For our 3D representation (NeRF) implementation, we utilize the "nerfacto" framework proposed by NerfStudio [83] To ensure stable training, deactivated the view-dependent effect. We pre-train the NeRF using 10000 iterations in stage and jointly optimize using 5000 iterations in stage 3. Our monocular depth estimation adopts DeythAny (98), complemented by the depth loss outlined in DS-NeRF 17 Moreover, we employ StyleGAN2 discriminator [35] to implement adversarial loss.
A.2 Sensitivity Analysis
We conducted several sensitivity analyses regarding the base view selection, Xa in ILA, and the subset selection. Due to the computational burden, we conduct the sensitivity analysis on six out of ten scenes with higher inpainting variability from the SPIn-NeRF dataset.
(a) Base View Selection:
To achieve better generalization, we propose sampling the base frame according to the geometrical centroid of the training camera poses, meaning the camera that sits most centrally among the training views. However, we found that Stable Diffusion occasionally inpaints strange artifacts in the masked region. To mitigate this, we propose sampling n candidate views around the geometrical centroid and selecting the one with the highest similarity votes. This approach automatically avoids such occasion artifacts without human intervention. In our implementation, we used five candidate views, and the similarity was calculated using perceptual hashing.
We tested our results under different settings (candidate numbers): 3, 5, 7, and 9. The base frame selection algorithm proved to be robust, with our algorithm typically yielding the same base frame. However, another factor influencing this step is the random seed. Setting different seeds causes the 2D inpainting model to produce different results, leading to different base frames being selected. We tested our methods under five different seeds, and the final scores are reported in Table 4 While different seeds cause the final NeRF to differ in the appearance of the masked region, the consistency of the multi-view inpainting results remains robust, resulting in minimal variance in the evaluation scores.
(b) >a in ILA:
To effectively examine the effect of the hyyer-paramene X in ILA, we evaluated our method's rendering quality with different A values of 0.2, 0.4, 0.6, and 0.8. The metrics are reported in T Quantitatively, the results are consistent across different Aa values, indicating that the effect of this hyper-paramete is relatively small. This conclusion also supported by qualitative results. Larger A values tend to produce slight variations in some small regions, but the global structure and semantics are preserved. This stability is attributed to the significant role of the initial latent alignment in ELA, which effectively aligns the underlying inpainting structure, thereby maintaining low variability in appearance. Additionally, the self-attention layer, where cross-view attention introduced, does not dominate the entire Stable Diffusion Unet. It is balanced by the presence of
18

---

Table 4: Sensitivity analysis on the prior inpainting results and prior view selection. Results are evaluated on the SPIn-NeRF dataset with different random seeds.
Seed
LPIPS +
MUSIQ
FID
I
0,46
46.61
264.91
2
0.44
48.04
255.29
3
0,44
46.47
262.09
4
0.44
45.72
261.04
5
0.46
48.65
258.50
Avg
0.45
47.10
260.37
Std
0.01
1.21
3.657
other (residual and linear) layers, ensuring cross-view attention does not override the signal during the denoising process. Hence we simply set A, as 0.2 in our implementation.
Table 5: Sensitivity analysis on A used in ILA.
Aa
LPIPS
MUSIQ t
FID
0.2
0.44
47.11
261.62
0.4
0.44
46.76
264.91
0.6
0.44
46.47
264.37
0.8
0.45
46.33
265.10
Avg
0.44
46.67
264.00
Std
0.01
0.35
1.62
(c) Subset Selection:
We found that for reconstruction tasks, more views enhance quality; however, for generation tasks, using the entire set of images can introduce unnecessary inconsistencies. Therefore, we propose selecting the subset according to the distribution of camera viewpoints.
We evenly split the viewpoints into 12 groups based on the base view's 's camera space (evenly 2 on the and y axes and 3 on the z-axis) and select 50 percent within each group according perceptual hashing similarity to the base view. This approach avoids redundant views introducing supervision conflicts while covering different viewpoints for effective supervision.
We also evaluated our method based on different percentages, as reported in Table The quantitative scores are quite close, indicating that for most scenes, the difference isn' significant. For one complex scene with extremely high frequencies, setting the percentage too low (0.2) yields artifacts in the test view due to insufficient viewpoint coverage. Conversely, setting the percentage too high (0.8) introduces appearance conflicts due the high variability of the inpainted results.
Overall, for most scenes, the subset selection algorithm is robust due to the consideration of viewpoints distribution. For extreme cases, careful selection of the percentage might be necessary. However, values between 0.5 and 0.7 remain reliable choice.
Table 6: Sensitivity analysis on proportion of images selected for the subset.
Percentage
LPIPS $
MUSIQ
FID
0.2
0.46
45.98
265.48
0.4
0.44
46.32
264.91
0.6
0.44
47.11
261.62
0.8
0.45
46.47
263.20
(d) Apateh patch loss:
To assess the sensitivity of the patch loss multiplier Xpateh, we evaluated the method's performance using various values of A 0.001, 0.005, 0.01, 0.05, and 0. 1. The results are reported Table 7
Analysis of the table indicates that varying Apatch leads to similar performance across different
19

---

settings, with low standard deviation of the metrics. However, there is an observable trend where setting Apatch too low or too high adversely affects performance. The multiplier Xootok is critical as it determines the extent of influence multi-view images have on the NeRF. Insufficient multi-view supervision can lead to inadequate training, whereas excessive supervision may result conflicting inputs. Consequently, we have set patch at 0.01 in our implementation for optimal balance.
Table 7: Sensitivity analysis on Apotok used for patch loss.
Xpatch
LPIPS +
MUSIQ f
FID 4
0.001
0.46
46.078
263.32
0.005
0.45
47.08
262.43
0.010
0,44
47.11
261.62
0.050
0.47
44.93
265.31
0.100
0,49
44.05
277.36
Avg
0.46
45.85
266.01
Std
0.02
1.35
6.49
A.3
More Qualitative Results
This section presents extended qualitative results from experiments the SPIn-NeRF Dataset. Fig. 7 and Fig. g showcase a series of multi-view comparative inpaintings.
A.4 Details on User Study and Impact
To comprehensively evaluate our method using human subjects, we conducted user study focusing on three aspeets: (1) Background Coherence assessing whether the inpainted area blends scamlessly with the remaining background, (2) Detail Preservation determining if the inpainted area retains figh-fidelity details, and (3) Overall Quality - gauging participants preference rates for the inpainted results. For each method, we presented users with multi-view test images from each instructed them to choose the method that best met the criteria for each aspect. Clear instructions were provided to ensure participants understood the rating process. An example screenshot of the study interface is shown in Fig. ?
and
The user study we conducted focused solely collecting participants' preferences regarding different inpainting results, involving no sensitive or personal data collection beyond their aesthetic judgments. The study's 'ss design inherently low-risk as it required participants to simply view and evaluate digital images based on their visual appeal and perceived quality. Furthermore, the participation was entirely voluntary, with clear instructions provided, allowing participants withdraw at any time without any consequence. Given these factors, the potential for harm or discomfort to participants negligible, ensuring the study maintained minimal risk profile.
20

---

Traning Input
SPIn-NeRF
NeRFiller
LnFusion
Ours
Figure 7: Additional Qualitative Results on the SPIn-NeRF Dataset.
21

---

Traning input
SPin-NeRF
NeRFiller
InFusion
Ours
Figure 8: Multi-view Qualitative Results on the SPIn-NeRI Dataset.
22

---

3D Inpolnting User Study
25
iraniing Prose
Leds vor oper senu backgrundd ronnect
thee othonet dicr bos, top bectaar
Please
Dedigcas
Figure 9: Example of User Study.
23

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and introduction, we clearly state that our method contributes to the field of object removal in neural radiance fields, offering novel and effective solution to improve multi-view consistency.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: the conclusion (Sec. 6 we discuss the limitations of our approach, outlining the current bottlenecks.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
24

---

Answer:
[NA]
Justification:
Our paper does not present theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We detail our implementation in Sec. A.
The evaluation settings are clearly described in Sec. 5.I ensuring fair comparison by standardizing the input across all
baselines.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
25

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Due to the extensive work required for this task, we will need clean up the code before its release. However, to ensure reproducibility, we provide detailed instructions in Sec. 4 and Sec. A.T The code will be made available upon completion of the cleanup
process,
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
/tw.geeee///eeeubisssss/////// sotiessuubbissioooooooooy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https / ctttlps.ccpubbicc/// guiieg SonndeSSbmissiionPooe
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We clearly stated the evaluation setting and implementation details in Sec. 5.1 and Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Previous 3D object removal methods did not report error bars, and the compu- tational resources required for these experiments were relatively high, making impractical to replicate the experiments multiple times.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
26

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provided details the computational hardware used in experiment (Sec. 5.1 and Sec. A.I.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We have reviewed the NeurIPS Code of Ethics and ensured that research adheres to its guidelines.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: the conclusion (Sec. 6 we discuss the positive socictal impacts of our research and state the potential harms of technology misuse.
Guidelines:
27

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
No new dataset or model is proposed in this work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All existing assets used in this work are properly credited and are the license and terms of use explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
28

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssippsssii thcode cowc datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: We have provided clear instructions for human evaluation in our user study and included sample screenshot in the supplementary materials, presented in Sec. Slad and
Sec. A.4
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: We have explicitly informed all participants in the user study that the collected data will be used solely for research purposes.
Guidelines:
29

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "uuT71gzYN4", "reviewer": "Reviewer_S5oa", "summary": "This paper proposes to solve multi-view or 3D inpainting problem. Given multiple posed views and the masks in each view specifying an object to be removed. It first trains a NeRF on the unmasked regions. To in-paint the masked regions, a seed view is selected and inpainted with a pre-trained diffusion inpainting pipeline. Depth is also predicted for this seed view, together with the RGB value, fused into the NeRF representation. To propagate the features from this view to other views for NeRF supervision, it proposes explicit and implicit feature alignment methods. For the explicit one, it propagates the initial latent noise to another view via a depth/density-based aggregation method.  For the implicit one, it uses a method similar to reference-only control, i.e., extracting intermediate key/value pairs from the seed view and injecting them into other inpainting views via cross-attention.", "questions": "See weakness.", "limitations": "Limitations addressed, however, the author has not mentioned that the proposed method is only tested in a limited scene case.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1) Experiments show that properly propagating anchor-view to other views with methods shows good performance for multi-view consistency. More notably, propagating the initial noise to other views is novel and interesting.", "weaknesses": "(1) The proposed method seems much simpler(or even over-simplified) than prior works such as NeRFiller and InFusion. The former uses a synchronized multi-view inpainting technique which is similar to the ILA of this paper. The latter one introduces new diffusion-based depth completion. Both prior methods have used a depth prediction model, and InFusion has shown that using a pre-trained depth model and aligning it with  NeRF/Gaussian depth is sub-optimal.  \nFor qualitative comparisons, there is little or only marginal difference between this paper and InFusion, while InFusion/NeRFillter has shown much more results than only the scenes included in this paper. \nI suspect that the result difference is due to the choice of base components, (i.e., inpainting pipeline, depth estimation model, etc.) which can not be considered as the novelty of this paper.\n\n(2) As I understand, the three stages run sequentially. i.e., Train unmasked NeRF(Stage-1) -> Update the NeRF with seed view and depth(Stage-1) -> Update the NeRF with novel inpainted views (guided by the seed view)(Stage2 then 3). Although Equation. (10) shows a total loss function. \n- This way, the proposed ELA approach seems over-complicated. After stage 1, there is no NeRF information in the masked region, and there is only one view with depth provided to supervise the masked area. There seems no need to use the NeRF density to propagate the feature from the reference inpainted view to other views. An alternative can be using the depth prior and the epipolar line to aggregate features. \n\n(3) Prior works show both object insertion and removal tasks, while the proposed method seems can only be applied to object removal tasks, as it heavily relies on the initial inpainting and depth estimation result. Moreover, the examples shown in this paper don't include complex structures and only show simple background, inconsistency caused by possible occlusion or partial observation has not been tested. Overall, this paper compares the results with the previous generation's scene completion method in a very limited scenario, which I believe is unfair. \n\n(4) The artifacts are obvious in the result videos, which can be caused by the noise/artifacts in the depth estimation model. More evidence is needed to make this paper stronger.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for the constructive feedback.  Below we provide the answers to reviewer's concern:\n- **1. Method over-simplified, and marginal difference compared to priors works.**\n    \nWe would like to argue that while our motivation may appear straightforward, the solution we propose is both non-trivial and novel. \n    \nThe multi-view inpainter NeRFiller achieves consistent inpainting results by averaging noise predictions derived from multiple iterations. This approach tends to produce smooth results, as demonstrated in Figure 4 of the main paper and Figure 1 of the rebuttal PDF. In contrast, our method explicitly and implicitly aligns latents without sacrificing high-frequency details.\n    \nInFusion relies on a single reference view and its depth to represent the entire scene. This reliance becomes problematic when the assumption of accurate depth is violated. As shown in Figure 4 of the main paper, misaligned geometry would lead to disconnected primitives (second and fourth rows). Similar evidence can also be seen in Figure 1 of the rebuttal PDF. By incorporating consistent multi-view supervision, our method remains effective even when depth or geometry is inaccurate, achieving robust and promising results. **This explains why our method shows little difference from InFusion when geometry is accurate** (first and third rows of Figure 4 in the main paper), but excels when the depth is inaccurate.\n    \nWe further collected nine scenes with manually annotated masks to evaluate the effectiveness of our method. This dataset includes four indoor scenes and five outdoor scenes. We conducted the experiment and evaluation under the same protocol as in the main paper, using the same inpainting pipeline for all methods as mentioned in the paper. The quantitative results are reported in table below and the qualitative results are shown in Figure 1 of the rebuttal PDF. We will include this evaluation with more details in the revised manuscript.\n\n>| Method    | LPIPS ↓ | MUSIQ ↑ | FID ↓  |\n>|-----------|---------|---------|--------|\n>| NeRFiller | 0.68    | 19.43   | 399.19 |\n>| InFusion  | 0.47    | 31.35   | 319.59 |\n>| Ours      | **0.35**  | **37.22** | **250.63** |\n\n_Table1: Quantitative Result on new collected dataset._\n    \nIn summary, although our motivation is simple, i.e.,using consistently inpainted multi-view images to optimize the radiance field, our approach to achieve consistent results is both novel and effective without relying on any geometric assumptions.\n \n\n- **2. ELA approach seems over-complicated.**\n    \nThank you to the reviewer for providing valuable comments. First, the reviewer's understanding of the three stages is totally correct. However, there are two key reasons why we propose fine-tuning the NeRF and using it as a geometric prior for ELA: \n    \n  >(a) After finetuning the NeRF, the geometric is represented by NeRF as a sharp (low variance) unimodal distribution on the ray. Consequently, the aggregated feature remains sharp, preserving the variations in the initial latents. \n    \n  >(b) As we discussed in the paper, we empirically found the depth prior inferred by the monocular depth estimator is not perfectly aligned for the NeRF. Fine-tuning the NeRF can also benefit this depth prior. Since NeRF learns relatively certain geometry in the known (unmasked) areas, this geometry constraint can improve the geometry of neighboring inpainted (masked) areas due to their geometric proximity.\n\nWe will explain these points in more detail in the revised version of the paper.\n    \n- **3. Limited scenario.**\n\nWhile prior works have been proposed for more general cases, our motivation focuses on a relatively smaller scope, specifically \"object removal.\" As outlined in the paper, we have found that this task is extremely challenging due to the requirement for scene-level generation, which demands consistent multi-view supervision of a scene rather than just an object. Current approaches still exhibit some limitations in this regard.\n    \nIn contrast, there are many works focusing on 3D object-level generation using multi-view diffusion models. Therefore, we believe that tasks such as object insertion and completion should follow this more promising direction, which we have decided to leave as future work.\n    \nHowever, in response to the reviewer's comments, we have further tested our method on three 3D object completion scenes, which are already included in the nine scenes mentioned in our first response (marginal difference compared to prior works). The results, shown in the rebuttal PDF file Figure 1 (first two rows), indicate that our method still performs promisingly on this complex task. Despite this, we continue to believe that inpainting tasks for objects might be more effectively addressed using multi-view diffusion or video diffusion techniques. We will elaborate on these points in the revised version of the paper.\n    \n- **4. Artifacts are obvious in the result video.**\n    \nWe have carefully examined our implementation of the method and identified that the artifacts are caused by the depth loss used. Initially, we borrowed the loss function proposed in \"Depth-supervised NeRF: Fewer Views and Faster Training for Free (CVPR 2022)\", which applies KL divergence to the rays' density distribution for sparse view reconstruction. We found that this loss is relatively sensitive and tends to produce more noise in under-constrained areas (with no multi-view supervision). To address this issue, we have replaced our depth loss with a more stable L2 loss, which has largely eliminated the artifacts. The revised implementation now produces significantly cleaner results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XSCOktRTo0", "reviewer": "Reviewer_uTYn", "summary": "This paper presents how to tackle the challenge of 3D object removal by involving 2D diffusion prior. The approach involves pretraining NeRF with inpainting prior and then jointly optimizing it with latent alignment to align feature priors.", "questions": "1. In Equ. 5, does the first term indicate the use of ground truth images (with the object removed) for supervising the mask region? If so, are there any concerns regarding potential information leakage? \n2. Regarding efficiency, does the multi-stage training approach in this paper have a comparison with baseline methods in terms of computational efficiency?", "limitations": "The author provides a comprehensive discussion of the broader impacts in the paper.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Introducing a 2D diffusion prior as a solution for the 3D inpainting task is an innovative and interesting approach.\n2. The incorporation of latent alignment to align feature priors is a promising design.", "weaknesses": "1. Although the generated results outperform the listed baseline methods, the paper lacks a comparison with other baselines, such as \"Reference-guided controllable inpainting of neural radiance fields\" (ICCV 2023)\", which achieves good results of novel view synthesis. \n2. The presentation of the paper could be improved, for example, it is uncertain how to get the patches in the patch-based loss (Equ. 9).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for providing these valuable comments.\n\n- **1. Lacks a comparison with other baselines.**\n\nThank you to the reviewer for pointing this out. Unfortunately, the method \"Reference-guided Controllable Inpainting of Neural Radiance Fields\" (ICCV 2023) is not open-sourced. However, this method, as stated in the introduction of our paper, operates similarly to the baseline method (InFusion), i.e., using a single reference view to serve the entire scene. These approaches assume that the geometry inferred by the radiance field (NeRF or Gaussian Splatting) is almost accurate, thus relying heavily on the geometric prior.\n    \nFrom the results of InFusion in Figure 4 of the main paper, we can observe that the accuracy of the geometry cannot be guaranteed from scene to scene. For instance, the InFusion excels in the first and third rows but exhibit some disconnection of the primitives in the second and fourth rows. Similarly evidence can be found in our new collected datasets as well, as in rebuttal PDF Figure 1.\n    \nThese findings highlight the importance of multi-view supervision for 3D generation and inpainting tasks. Consistent multi-view supervision can mitigate the reliance on geometric priors and improve the robustness and accuracy of the inpainting results. By incorporating multi-view supervision, our method still works in the circumstances when such depth/geometry is not accurate, achieving promising results.\n\n- **2. How to get the patches.**\n\nThank you to the reviewer for the constructive feedback. We will revise the paper to include more complete details in the revision. The patches are uniformly sampled within the bounding box of the mask, and the patch size we used is 256×256. Therefore, only the content within the bounding box (mostly the inpainted area) is being optimized by the patch loss.\n\n- **3. Eq.5 ground truth supervision.**\n\nThank you to the reviewer for the comment. There is no ground truth supervision for the masked region. We used the official train-test split provided by SPIn-NeRF. In the training set, the masked region contains the unwanted object, while the test set contains the ground truth background in the masked region. We only optimize our radiance field on the training set, ensuring no information leakage. We will explicitly clarify this point in the revision.\n\n- **4. Computational efficiency.**\n\nThank you to the reviewers for the valuable feedback. We would like to provide a comparison of training time in below table.  InFusion ranks first in efficiency due to the high rendering efficiency of Gaussian splatting and the simplicity of single-view optimization. The three NeRF-based multi-view methods (SPIn-NeRF, NeRFiller, and ours) exhibit similar optimization efficiency. Our method has a slight advantage in efficiency due to the sampled prior image and the relatively consistent multi-view images, which facilitate faster convergence.\n\n>| Method     | Supervision | Training Time (min) |\n>|------------|-------------|---------------------|\n>| SPIn-NeRF  | Multi-view  | 71                  |\n>| NeRFiller  | Multi-view  | 65                  |\n>| InFusion   | Single-view | **17**              |\n>| Ours       | Multi-view  | *49*                |\n\n_Table 1: A comparison of Training Time._", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6M5ZEBB7la", "reviewer": "Reviewer_f6e3", "summary": "The paper deals with 3D inpainting problem in the NeRF setup with 2D diffusion models. To achieve multiview consistency, they took a \"inpaint-outstretch\" strategy. They first inpaint one key frame, and conditioned on which, generate a view-consistent inpainted image set. Finally, they use the multiview images to train the final NeRF model, together with the reconstruction losses. The method is compared to a few recent works (Spin-NeRF, NeRFiller, InFusion) and achieved better results both qualitatively and quantitatively.", "questions": "- Is Eq (10) applied to Stage 3? If so, what is the purpose of the prior loss L_prior? Once the inpainted base image is used to generate a multiview training set, I would expect this information to be unnecessary? Can this term be merged into L_patch? \n- In L216, what is the subset? Why not inpaint the whole set of images and how much does it affect the results?", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The method is interesting and seems effective. They improve the consistency of multiview generation in two ways. The first is explicit latent alignment (ELA) that initializes the latent from the inpainted base frame. The second is ILA that post-hoc modifies the attention layer of stable diffusion models by replacing the key and value with the ones from the inpainted base frame. ILA does not need finetuning.\n\n- The method is evaluated on the SPIn-NeRF dataset and the performance gain over existing methods is quite large, both quantitatively and via user study. \n\n- There is also an ablation study that shows the importance of ELA, ILA, and the patch-based losses applied on the generated image sets.\n\n- The method is well written and easy to follow.", "weaknesses": "- My biggest confusion is on the explanation of ILA. \n  - I don't fully understand why replacing the KV with the base frame KV would encourage consistency. I would expect the style to be consistent but not the local geometry, where the latter is more important for 3d reconstruction.\n  - What is the rationale of replacing KV with \"prior\" p, but not Q?\n- The algorithm relies on a pre-defined base frame p. How is the base frame chosen, and how sensitive is the algorithm to the chosen frame?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- **1. Explanation of ILA**\n\n(a) The reviewer's intuition of ILA is entirely correct. ILA primarily contributes to appearance (color) consistency. Geometry consistency is achieved by ELA (the alignment of initial noise) since this element serves as a foundational structure (semantic) for the inpainting. This is verified in our ablation study in Figure 6 of the main paper. Removing ELA results in geometry mismatch (blurry boundaries), while dropping ILA achieves geometry alignment but causes color mismatch issues (blurry leaves). This is why we propose ELA and ILA together to ensure both geometry and appearance consistency.\n    \n(b) The rationale of replacing $KV$ with \"prior\" $p$, but not $Q$ is that the appearance information (**$V$**) of the prior image should be considered when inpainting the other views, with amount of information propagation is weighted by its attention key value (**$K$**). The attention query value comes from the current inpainting view $i$, $Q_i$ , representing the information the current inpainting for view $i$ is searching for. Together with $K_p$ , it decides how much attention the view $i$ inpainter should place on the prior view, and finally incorporates the prior view information $V_p$ into view $i$. We will elaborate on these points in the revised version of the paper.\n\n- **2. Discussion of selecting base frame.**\n\nTo achieve better generalization, we propose sampling the base frame based on the geometrical centroid of the training camera poses. However, Stable Diffusion occasionally inpaints strange artifacts in the masked region. To mitigate this, we sample $n$ candidate views around the centroid of the camera viewpoints and select the one with the highest similarity votes, automatically avoiding artifacts without human intervention. In our implementation, we used five candidate views, with similarity calculated using perceptual hashing.\n    \nWe tested our results under different settings (candidate numbers): 3, 5, 7, and 9. The base frame selection algorithm proved to be robust, with our algorithm typically yielding the same base frame. However, another factor influencing this step is the random seed. Setting different seeds causes the 2D inpainting model to produce different results, leading to different base frames being selected. We tested our method under five different seeds, with final scores reported in the table below and additional qualitative results in Figure 2 of the rebuttal PDF. While different seeds affect the appearance of the masked region in the final NeRF, the consistency of multi-view inpainting remains robust, resulting in minimal variance in evaluation scores. We will explicitly clarify this point in the revision.\n\n>| Seed | LPIPS ↓ | MUSIQ ↑ | FID ↓ |\n>|------|---------|---------|-------|\n>| 1    | 0.46    | 46.61   | 264.91|\n>| 2    | 0.44    | 48.04   | 255.29|\n>| 3    | 0.44    | 46.47   | 262.09|\n>| 4    | 0.44    | 45.72   | 261.04|\n>| 5    | 0.46    | 48.65   | 258.50|\n>| *Avg* | *0.45* | *47.10* | *260.37* |\n>| *Std* | *0.01* | *1.21* | *3.657* |\n\n_Table 1: Sensitivity analysis on the prior inpainting results and prior view selection. Results are evaluated on SPIn-NeRF dataset with different random seeds._\n\n- **3. Eq (10) applied to Stage 3.**\n\nYes, Eq. 10 is the final loss for Stage 3, designed to alleviate noise and inconsistency. Thank you for the reviewer's insight, and I fully understand the confusion. The prior loss $L_\\text{prior}$ is an L2 loss first proposed in Stage 2. We propose keeping this term in Stage 3 because we believe the base view does not require patch-based optimization. As discussed in the limitations section, achieving perfect 3D consistency is challenging, so the remaining views may still have subtle inconsistencies. To preserve high-frequency details, we use patch loss instead of the exact-match L2 loss. We regard the base view as a baseline that does not exhibit inconsistency; thus, we do not apply the patch loss to it.\n\n- **4. Discussion of Subset.**\n\nWe found that for reconstruction tasks, more views can enhance quality; however, for generation tasks, using the entire set of images introduces unnecessary inconsistencies. Therefore, we propose selecting a subset based on the distribution of camera viewpoints.\n    \nWe evenly split the viewpoints into 12 groups based on the base view's camera space (evenly 2 on the x and y axes and 3 on the z axis) and select 50 percent frames within each group according to perceptual hashing similarity to the base view. This approach avoids redundant views introducing supervision conflicts, while covering different viewpoints effectively.\n    \nWe also evaluated our method with different selection percentages, as reported in the table below. The quantitative scores are quite close, indicating minimal differences for most scenes. For one complex scene with high frequencies, setting the percentage too low (0.2) yields artifacts due to insufficient viewpoint coverage, while setting it too high (0.8) introduces appearance conflicts due to variability in inpainted results. These results are visualized in Figure 4 of the rebuttal PDF.\n\n>| Percentage | LPIPS ↓ | MUSIQ ↑ | FID ↓   |\n>|------------|---------|---------|---------|\n>| 0.2        | 0.46    | 45.98   | 265.48  |\n>| 0.4        | *0.44*  | 46.32   | 264.91  |  <!-- Used italics to simulate underlining -->\n>| 0.6        | **0.44**| **47.11** | **261.62** | <!-- Bold for emphasis -->\n>| 0.8        | 0.45    | *46.47* | *263.20* | <!-- Used italics to simulate underlining -->\n\n*Table 2 : Sensitivity analysis on proportion of images selected for the subset.*\n    \nOverall, for most scenes, the subset selection algorithm is robust due to the consideration of viewpoints distribution. For extreme cases, careful selection of the percentage might be necessary. However, values between 0.5 and 0.7 remain a reliable choice. We will carefully revise the manuscript to include these details.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gAPFKyNQiE", "reviewer": "Reviewer_P7tn", "summary": "The paper introduces a novel approach termed \"In-N-Out\" for enhancing the performance of 3D object removal tasks by leveraging tuning-free latents alignment. The authors have addressed the challenges of geometric mismatches and color inconsistencies prevalent in existing methods.", "questions": "1. In ELA module, using latent z^T to replace color c for volume rendering. Will this compromise the view-dependent in NeRF? In NeRF，the color  is dependent on the position x and direction d, whereas in ELA, the z^T is merely reprojected to the image plane. \n\n2. How do you choose lambda_a in ILA.", "limitations": "The method involves several hyperparamete. The paper does not discuss the sensitivity analysis or the robustness of the method against variations in hyperparameters.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "The authors have conducted an extensive review of related work and clearly delineated the distinctions and connections between their work and existing literature.The explicit and implicit latents alignment proposed by the authors is an intriguing concept that could inspire work in related domains.", "weaknesses": "The paper exhibits signs of being finished in a rush, which affects the overall quality of academic writing, and is hard to follow.\nFor example，\n1. Line172，Omega(·) is not defined\n2. Formula (5), D_hat is note defined\n3. Line 208， by replace -> by replacing\nThe paper might has the potential to contribute to the field but requires significant revisions to improve the academic writing quality.\n\nThe method involves several hyperparamete. The paper does not discuss the sensitivity analysis or the robustness of the method against variations in hyperparameters.  Especially，the lambda_a, which plays an essential role in ILA.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- **1. Quality of academic writing.**\n\nThanks for pointing out these typos. In line 172, Omega(·) is a decoder which is introduced in line 151, and  $\\hat{D}_{\\phi}$ is the depth estimated by NeRF. We hope these typos do not affect the clarity of the ideas presented in this paper. \n\n- **2. Compromise of view-dependent effect in ELA.**\n\nYes, we compromise the view-dependent effect in NeRF within the ELA module, using NeRF only as a geometry prior to propagate and aggregate initial latents. Due to the heuristic nature of diffusion models, incorporating view-dependent effects into their output remains elusive.\n\n- **3. Lack of sensitivity analysis.**\n\nThanks for the constructive feedback. We've conducted 4 sensitivity analysis regarding the base view selection, $\\lambda_a$ in ILA, subset selection and $\\lambda_\\text{patch}$ for patch loss. Due to the computational burden, we conduct the sensitivity analysis on six out of ten scenes with higher inpainting variability from SPIn-NeRF dataset.\n\n### (a) Base View Selection:\nTo achieve better generalization, we propose to sample $n$ candidate views around the geometry centroid of the camera viewpoints and select the one with the highest similarity votes, automatically avoiding artifacts without human intervention. We used 5 candidate views, with similarity calculated using perceptual hashing.\n    \nWe tested our results under different settings (candidate numbers): 3, 5, 7, and 9. The selection algorithm proved to be robust, with our algorithm typically yielding the same base frame. However, another factor influencing this step is the random seed. Setting different seeds causes the 2D inpainting model to produce different results, leading to different base frames being selected. We tested our method under five different seeds, with final scores reported in the table below and additional qualitative results in Figure 2 of the rebuttal PDF. While different seeds affect the appearance of the masked region in the final NeRF, the consistency of multi-view inpainting remains robust, resulting in minimal variance in evaluation scores. We will explicitly clarify this point in the revision.\n\n>|Seed|LPIPS↓|MUSIQ↑|FID↓|\n>|------|---------|---------|-------|\n>| 1|0.46|46.61|264.91|\n>| 2|0.44|48.04|255.29|\n>| 3|0.44|46.47|262.09|\n>| 4|0.44|45.72|261.04|\n>| 5|0.46|48.65|258.50|\n>|*Avg*|*0.45*|*47.10*|*260.37*|\n>|*Std*|*0.01*|*1.21*|*3.657*|\n\n### (b) $\\lambda_a$ in ILA:\n\nTo examine the effect of the hyper-parameter $\\lambda_a$ in ILA, we evaluated our method's rendering quality with varies $\\lambda_a$ value. The metrics are reported in below table. The results are consistent across different $\\lambda_a$ values, indicating a relatively small effect. This conclusion is supported by qualitative results in Figure 3 of the rebuttal PDF, where larger $\\lambda_a$ values produce slight variations in small regions, but the global structure and semantics are preserved.\n\nThis stability is attributed to the significant role of the initial latent alignment in ELA, which effectively aligns the underlying inpainting structure, thereby maintaining low variability in appearance.  Additionally, the self-attention layer, where cross-view attention is introduced, does not dominate the entire Stable Diffusion Unet. It is balanced by the presence of other (residual and linear) layers, ensuring cross-view attention does not override the signal during the denoising process. Hence we simply set $\\lambda_a$ in our implementation. We will explicitly clarify this point in the revision.\n\n>|$\\lambda_a$|LPIPS↓|MUSIQ↑|FID↓|\n>|-------------|---------|---------|---------|\n>|0.2|0.44|**47.11**|**261.62**|\n>|0.4|**0.44**|46.76|264.91|\n>|0.6|0.44|46.47|264.37|\n>|0.8|0.45|46.33|265.10|\n>|*Avg*|*0.44*|*46.67*|*264.00*|\n>|*Std* |*0.01*|*0.35 *|*1.62*|\n\n### (c) Subset Selection:\n\nWe propose selecting a subset for stage 3 training based on the distribution of camera viewpoints. We evenly split the viewpoints into 12 groups based on the base view's camera space (evenly 2 on the x and y axes and 3 on the z axis) and select 50% views within each group according to perceptual hashing similarity to the base view. This approach avoids redundant views introducing supervision conflicts, while covering different viewpoints effectively.\n    \nWe evaluated our method with varies selection percentages, as reported in the table below. The scores are close, indicating minimal differences for most scenes. For one complex scene with high frequencies, setting the percentage too low (0.2) yields artifacts due to insufficient viewpoint coverage, while setting it too high (0.8) introduces appearance conflicts due to variability in inpainted results. These results are visualized in Figure 4 of the rebuttal PDF.\n\n>|Percentage|LPIPS↓|MUSIQ↑|FID↓|\n>|------------|---------|---------|---------|\n>|0.2|0.46|45.98|265.48|\n>|0.4|*0.44*|46.32|264.9| \n>|0.6|**0.44**|**47.11** |**261.62**|\n>|0.8|0.45|*46.47*|*263.20*|\n>|*Avg*|0.44|46.67|264.00|\n>|*Std*|0.01|0.35|1.62|\n\n### (d) $\\lambda_{patch}$ in patch loss:\n\nTo assess the sensitivity of the patch loss multiplier $\\lambda_{patch}$ in Eq. 10 of the main paper, we evaluated the method's performance using varies $\\lambda_{patch}$ values. The results in below table show similar performance across different settings, with low standard deviation. However, setting $\\lambda_{patch}$ too low or too high adversely affects performance. $\\lambda_{patch}$ is critical as it influences the extent of multi-view supervision on NeRF; insufficient supervision leads to inadequate training, while excessive supervision causes conflicts. We set $\\lambda_{patch}$ at 0.01 in our implementation for optimal balance.\n\n>|$\\lambda_{patch}$|LPIPS↓|MUSIQ↑|FID↓|\n>|-------------------|---------|---------|---------|\n>|0.001|0.46|46.07|263.32|\n>|0.005|0.45|47.08|262.43|\n>|0.01|**0.44**|**47.11**|**261.62**|\n>|0.05|0.47| 4.93|265.31|\n>|0.1|0.49|44.05|277.36|\n>|*Avg*|*0.46*|*45.85*|*266.01*|\n>|*Std*|*0.02*|*1.35*|*6.49*|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all the reviewers for their thoughtful and constructive feedback on our manuscript.

Below, we summarize the modifications made to the manuscript based on your comments.

- To Reviewer S5oa
  - We have clarified the motivation and contribution compared to the priors work.
  - We have explained the rationale behind our ELA approach.
  - We added experiments to further demonstrate the effectiveness of our method, showing promising results for complex tasks like object completion, as shown in Figure 1 of the attachment.
  - The artifacts in the videos have been discussed and resolved.

- To Reviewer uTYn
  - We have clarify the lack of comparison with other baselines.
  - We have detailed the technical process of obtaining patches.
  - We have addressed the concern of potential information leakage.
  - We have provided a comparison of computational efficiency.

- To Reviewer f6e3
  - We have elaborated on the intuition behind ILA with more details.
  - We have discussed our algorithm for selecting the base frame, including a sensitivity analysis and following discussion.
  - We have explained the motivation for retaining the loss of the prior image in Eq. 10.
  - We have discussed our motivation and methodology for selecting subsets, including a sensitivity analysis and following discussion.

- To Reviewer P7tn
  - We have clarify the notation confusion in the paper.
  - We have clarify the compromise of view-dependent effect in ELA.
  - We have conducted four sensitivity analyses on different aspects of our method, providing deeper insights into our approach.

We appreciate the reviewers' insights, which have been invaluable in refining our work.

Meta-review: This paper is about 3D object inpainting using 2D diffusion prior. The paper received mixed scores of two borderline reject, one weak accept, and one accept. The negative reviewers (S5oa, uTYn) have concerns about the details and motivation of the proposed ELA approach, missing comparisons with baselines, and the lack of computational efficiency comparisons. In the rebuttal, the authors provided more detailed descriptions of the proposed method, showed preliminary results on object completion, explained why reference-guided inpainting (ICCV 2023) is not compared (as it is not open-sourced), and provided computational efficiency results. Unfortunately, the reviewers have not responded to these new results. The other two reviewers appreciate the technical novelty of the method and believe that the evaluation is convincing. 

Based on the reviews and the author's rebuttal, the AC agrees with the reviewers that the authors' rebuttal largely addresses the initial concerns. The AC believes that this paper has sufficient merits and thus recommends its acceptance.
Decision: Accept (poster)