Title: In-N-Out: Lifting 2D Diffusion Prior for 3D Object Removal via Tuning-Free Latents Alignment
Venue: NeurIPS 2024
Accepted: yes
Keywords: Neural Radiance Fields

3D Inpainting

Neural Representation Editing

Diffusion Models

Object Removal
Abstract: Neural representations for 3D scenes have made substantial advancements recently, yet object removal remains a challenging yet practical issue, due to the absence of multi-view supervision over occluded areas.  Diffusion Models (DMs), trained on extensive 2D images, show diverse and high-fidelity generative capabilities in the 2D domain. However, due to not being specifically trained on 3D data, their application to multi-view data often exacerbates inconsistency, hence impacting the overall quality of the 3D output. To address these issues, we introduce "In-N-Out", a novel approach that begins by inpainting a prior, i.e., the occluded area from a single view using DMs, followed by outstretching it to create multi-view inpaintings via latents alignments. Our analysis identifies that the variability in DMs' outputs mainly arises from initially sampled latents and intermediate latents predicted in the denoising process. We explicitly align of initial latents using a Neural Radiance Field (NeRF) to establish a consistent foundational structure in the inpainted area,  complemented by an implicit alignment of intermediate latents through cross-view attention during the denoising phases, enhancing appearance consistency across views. To further enhance rendering results, we apply a patch-based hybrid loss to optimize NeRF. We demonstrate that our techniques effectively mitigate the challenges posed by inconsistencies in DMs and substantially improve the fidelity and coherence of inpainted 3D representations.

In-N-Out: Lifting 2D Diffusion Prior for 3D Object Removal via Tuning-Free Latents Alignment

Abstract

Neural representations for 3D scenes have made substantial advancements recently, yet object removal remains a challenging yet practical issue, due to the absence of multi-view supervision over occluded areas. Diffusion Models (DMs), trained on extensive 2D images, show diverse and high-fidelity generative capabilities in the 2D domain. However, due to not being specifically trained on 3D data, their application to multi-view data often exacerbates inconsistency, hence impacting the overall quality of the 3D output. To address these issues, we introduce 'In-NOut', a novel approach that begins by inpainting a prior, i.e., the occluded area from a single view using DMs, followed by outstretching it to create multi-view inpaintings via latents alignments. Our analysis identifies that the variability in DMs' outputs mainly arises from initially sampled latents and intermediate latents predicted in the denoising process. We explicitly align of initial latents using a Neural Radiance Field (NeRF) to establish a consistent foundational structure in the inpainted area, complemented by an implicit alignment of intermediate latents through cross-view attention during the denoising phases, enhancing appearance consistency across views. To further enhance rendering results, we apply a patchbased hybrid loss to optimize NeRF. We demonstrate that our techniques effectively mitigate the challenges posed by inconsistencies in DMs and substantially improve the fidelity and coherence of inpainted 3D representations.

1 Introduction

Neural Radiance Fields (NeRFs) [50, 2, 23, 58, 81, 37, 13, 10, 3, 92] have effectively revolutionized 3D scene reconstruction from multi-view images. These models offer high-fidelity novel-view synthesis, proving beneficial across a variety of domains [32, 87, 88, 43, 107, 6, 63, 72, 8, 60]. Despite the impressive ability to reconstruct highly detailed scenes, these learning-based methods depend on the availability of consistent multi-view training data. This reliance limits their generalizability, particularly in editing 3D representations for tasks like object removal and inpainting occluded areas.

Recently, diffusion models (DMs) [30, 18, 71, 80] have gained significant attention in the field of generative modelling for 2D images. These models are well-known for their robustness as generative priors, capable of producing diverse and high-fidelity results in 2D inpainting tasks. However, adapting these 2D priors for 3D object removal is not straightforward. While the inherent diversity of DMs benefits the generation of varied outputs, it also poses a significant challenge: high variance in the inpainted results (Fig.1 middle column). Consequently, these models frequently produce outputs that, while visually appealing in isolation, may appear misaligned when incorporated into 3D domain [53, 94, 93, 25, 19, 97, 84]. This misalignment often results in the loss of high-frequency details, crucial for realistic and coherent scene rendering.

View

#2

(a) Original Image

(b) Inpainted by Stable Diffusion

(c) Inpainted by Ours

View

#1

Previous studies addressing such 3D inconsistencies can be broadly categorized into two approaches: multi-view and single-view priors. The former tackled inconsistencies across multi-view inpainted images by optimizing NeRFs with modified objectives [53, 94, 93, 65, 91]. While these methods have shown promise in refining inconsistent inputs, they sometimes suffer from a loss of detail fidelity during the training process, as illustrated in Fig. 4. Conversely, other studies have attempted to overcome the multi-view inconsistency bottleneck by anchoring the inpainting process to a single reference image that serves the entire scene [42, 51, 109]. This approach, however, places significant reliance on the selection of an appropriate reference image and the accuracy of depth estimates, which could lead to geometric artifacts during testing, as shown in Fig. 4.

To address these challenges, we aim to overcome 3D inconsistencies by guiding 2D DMs to achieve multi-view consistent inpainting results (Fig. 1 right column). Our analysis reveals that the variance in model outputs primarily comes from the random noise as the initial latent sample, and intermediate latents inferred by the denoising network. Each frame's initial latents are independently sampled, while intermediate latents are individually predicted, highlighting how view-dependent data impacts the generation process. Therefore, our approach focuses on aligning these two critical elements across multiple inputs. We introduce 'In-N-Out', a conditional-sampling-like approach that inpaints a sampled view and outstretches it to multiple views. Our method contains three key components:

Conditional Inpainting Pipeline: We propose a pipeline that first samples an inpainting outcome from a random view as an inpainting prior. This prior then serves as a condition to guide the inpainting process of multiple views, ensuring a consistent inpainting foundation.

Explicit Latents Alignment: Leveraging the geometry derived from a pre-trained NeRF and the inpainting prior, we sample multi-view initial latents conditional on the geometry dictated by the inpainting prior. This ensures that the primary components within the inpainted areas are structurally consistent and align with the underlying 3D geometry.

Implicit Latents Alignment: We employ a cross-view attention mechanism during the denoising steps to align predicted intermediate latents concerning the inpainting prior. This enhances the appearance consistency across the inpainted images.

To further enhance our method's performance in the 3D domain, we have implemented a patch-based optimization strategy using a hybrid loss on our inpainted multi-view images. This strategy employs perceptual loss to rectify spatial mismatches, and adversarial loss to preserve high-frequency details. By addressing these key challenges, our framework effectively handles multi-view inconsistencies and enhances the fidelity and coherence of 3D representations. The effectiveness of our approach is demonstrated through both qualitative and quantitative evaluations of a challenging object removal dataset. Our results indicate comprehensive improvement compared to existing methods, highlighting our model's ability to achieve greater fidelity and consistency in inpainted scenes.

2 Related Works

2D Editing with Diffusion Models Diffusion models [30, 33, 105, 59, 79, 80], have revolutionized image generation with their capacity to create highly realistic images. These models facilitate customizable generation via textual prompts [18, 29, 69, 74], predominantly using pre-trained Stable Diffusion [71]. Several editing methods [22, 57, 56, 61] allow users to adjust images by moving anchor points to new locations. Editing typically begins by inverting the latent representation of the image to be edited back to its initial noise [80], with modifications made during the denoising phase. Prompt-to-Prompt (P2P) [26] edits images by adjusting the cross-attention between the image and text. Null-text inversion [55] addresses artifacts in DDIM inversion [80] when using classifier-free guidance [29]. Delta Denoising Score (DDS) [27] optimizes the latent image representation by aligning the predicted noises of the original and modified texts. Additionally, several studies [7, 9, 26] have identified a relationship between the appearance of images generated by diffusion models and the key-value pairs. While these advancements represent significant progress preserve some content from the original image in 2D image editing, they do not account for multi-view consistency, thus can not be lifted to 3D editing directly.

Lifting 2D diffusion models for 3D editing Recent advancements in 3D editing and generation have effectively utilized 2D DMs to enhance these processes, as demonstrated in various studies [52, 76, 96, 62, 97, 99, 41, 95, 68, 49, 103]. Pioneering works have used images inferred by DMs for direct supervision. Instruct-NeRF2NeRF (IN2N) [25] approached the editing task by transforming 3D model editing into a 2D image editing task, utilizing Instruct Pix2Pix (IP2P) [5] to iteratively update 3D scenes. Similarly, ViCA-NeRF [19] addressed editing challenges by modifying reference images and integrating these changes into the scene. DreamEditor [110] opted for a different strategy by converting NeRF into a mesh for direct optimization. GaussianEditor [12] applies semantic tracing to identify and modify editing targets within 3D Gaussian Splatting (3DGS) [37]. Similarly, Gaussian Grouping [100] implements Identity Encoding for each Gaussian to create masks for editing. Conversely, Score Distillation Sampling (SDS) [64] provides an alternative way to guide 3D representations by backpropagating gradients from a diffusion model's denoiser [1, 16, 73, 71] into the underlying scene representation. This technique has been effectively applied to generate realistic 3D and 4D scenes using NeRFs [11, 40, 39, 108, 110] and 3DGS [70, 101, 84, 15].

2D and 3D Inpainting 2D inpainting methods reconstruct images by filling missing content in areas defined by a mask [20, 77, 104, 86, 46, 75, 102]. Early techniques, exemplified by [21], relied on copying textures from known to unknown regions. LaMa [82] excels in restoring large missing areas using fast Fourier convolutions, extensive receptive fields, and large training masks. Although highly effective at generating plausible background textures within specified masks, LaMa limits the fidelity of its outputs. In contrast, probabilistic diffusion models [30] have shown impressive results in image generation and offer a wide range of inpainted outputs. DMs can be adapted for inpainting without specific training, and modify known regions during each denoising step to fit the task [47]. Similarly, Stable Diffusion [71] excels at inpainting by operating within latent space, allowing for efficient and effective image generation. In this work, we adopt it as our 2D inpainter.

3D scene inpainting aims to fill missing areas within a space, such as removing objects and generating coherent geometry and textures to complete the scene. Although 3D generative models have garnered large interest [4, 34, 38, 89, 78, 44, 31, 85, 45, 14, 12], they are often limited by the scarcity of 3D training data, hence result in poor generalization, particularly in scene inpainting tasks. Therefore, most current 3D inpainting models [53, 51, 42, 65, 93, 94, 109, 91] enhance their effectiveness by adopting priors from 2D models. SPIn-NeRF [53] reduces multi-view inconsistencies by first inpainting views and then optimizing NeRF using perceptual loss. NeRFiller [93] tackles multiple frames simultaneously by tiling images for DMs. GaussianEditor [12] edit targets within 3DGS [37], guided by inpainted multi-view images from DMs. While these methods show promise, they can sometimes compromise detail fidelity during training. Alternatively, some studies circumvent multi-view inconsistencies by using a single reference image for the entire scene [42, 51, 109]. Infusion [109] stands out in the inpainting of 3DGS, leveraging a pre-trained depth completion network to infer point clouds from a single inpainted view, though this method depends heavily on precise depth estimates. Concurrent works [65, 54] address these challenges using SDS objective [64] to better align 2D model priors with 3D scene consistency.

3 Preliminaries

3.1 Neural Radiance Fields

Neural Radiance Fields (NeRFs) [50] represents a breakthrough in 3D rendering by employing a multilayer perceptron (MLP), denoted as ϕ to represent a scene. This MLP serves as a continuous volumetric function to capture and reconstruct a scene in unprecedented detail. Specifically, NeRFs take as input the view direction d and a 3D coordinate r ( τ ) sampled from a camera ray defined by r ( τ ) = o + τd . At each position along this ray r ( τ ) , the network predicts the volume density and view-dependent color, represented as ( σ, c ) . To render a camera pixel, NeRFs perform an aggregation of the predicted densities and color emissions σ ( τ i ) , c ( τ i ) along the camera ray. This process is mathematically formulated as an approximation of a volume rendering integral [48], which is used to compute the final color of the pixel:

where δ ( τ i ) = τ i +1 -τ i is the distance between adjacent samples along the ray. During the training phase, rays are uniformly sampled from the training images, and the volumetric field is optimized using mean square error (MSE) to enhance the accuracy and realism of the rendered scenes.

3.2 Diffusion Models

Diffusion models [30] consist of two processes: a forward process that gradually introduces noise to a data sample z 0 ∼ p data ( z ) , and a learned reverse process that iteratively denoises a purely Gaussian noise sample z T ∼ N (0 , 1) back into a clean image z 0 . The reverse process is parameterized by a conditional noise prediction network ϵ θ , trained to predict the noise using the simplified objective:

where t is the time step in the diffusion process, z t is an intermediate noisy sample, and c represents a condition (e.g., images, masks, or text). Utilizing a deterministic sampler like DDIM [80], the sample z t -1 can be obtained by z t -1 = z t -ϵ θ ( z t , t, c ) ; note that scaling is omitted for simplicity. In practice, as we use Stable Diffusion [71], a latent diffusion as the inpainting backbone, z is latent and the generated image is obtained with a decoder Ω( z 0 ) . Hence, the variability of the generated image z 0 depends solely on initial sampled latent z T and intermediate inferred latents { z t -1 } T t =1 .

4 Method

Given a set of multi-view training images {I i } N i =1 from the scene with corresponding masks {M i } N i =1 indicate the unwanted object in each frame, our approach seeks to generate consistently inpainted training set { ˜ I i } N i =1 and use them to supervise NeRF. Our approach is structured into three key stages:

Stage 1: Pretrain a NeRF ϕ using {I i } N i =1 and {M i } N i =1 , along with a sampled inpainted prior ˜ I p as a rough hallucination of the inpaint feature. (Sec. 4.1).

̸

Stage 3: Using the inpainted image set { ˜ I i } N i =1 , we optimize ϕ with a patch-based hybrid loss to distill multi-view supervision. (Sec. 4.3)

Stage 2: Leverage ϕ to inpaint additional views { ˜ I i | i = p, i = 1 , . . . , N } conditioned on the inpainting prior ˜ I p via explicit and implicit latents alignment. (Sec. 4.2)

An overview of our method is shown in Fig. 2.

4.1 Stage 1: Pre-train NeRF

The initial stage involves training the NeRF on the unmasked region, we follow the original work [50] where simple MSE loss is applied:

<latexi sh

1_b

64="

2

y/

w+

EgoSRJd

VvB

uQ

>A

7X

c

DLT

M

Nz

O

jF

k

CWI

G

8

9

p

rUf

Y

m

0

n

P

3Z

H

K5

q

<latexi sh

1_b

64="u

A0

5jzfk

8pBcy g9oSKVU

>

/n

DL

M

F

3

rP

q7

Iv

2XR

d

GTZ

QJ

O

C

Nm

Y

W

w

E

+

H

<latexi sh

1_b

64="PRDp

82Kv

7

Ey

UI

Q

>A

B9

c

V

LSgM

T

W+

d

r

j

u

n

O5Z

m

k

J

/

f

z

0

C

w

N

Y3

o

G

H

<latexi sh

1_b

64="DF

Jp0

o

q5Z3

2dA

VSK+

7

>

B9X

c

L

gM

T

W

Iv

y

r

Rj

Y8mk

Q

/

ONC

u

z

U

f

G

w

n

EH

P

<latexi sh

1_b

64="

2+F9CoEAR

p8J

gPrn

Zk

Ic

>

B7

V

NS

3U

q/

L

0

K

D

wm

G

vH

T

u

5

Q

M

f

dW

j

y

O

X

z

Y

<latexi sh

1_b

64="rpq

L

oU0+cIQwyA2n

D

/

N

>

C

X

V

S

FJ3

v

B

T

k

Z

5

P

EKY

f

M

z

G

u

g

O

d

7

8

W

H

9

j

m

R

<latexi sh

1_b

64="W

GqcPO2

BuD3

9

Q

MA

7

Y

>

C

V

LS

N

FJ

U

v

d

I

g

T

0

8

f

o

n

z

+

/

X

H

E

mw

j

y

r

k

K

p

Z

5

R

<latexi sh

1_b

64="CSyX

W

Uz

LAdk wM8Z5m

g

>

B

3

c

V

N

EJ

r

q/

9

0n

F

7

o

2

p

Q

+

D

I

GT

u

j

Y

O

f

v

R

K

P

H

<latexi sh

1_b

64="Tq9rmL

Q

80GoW

I

3

n

>A

C

c

VD

S

N

FJ

U

2v dB

g

fu

O

z

Y

+

/

PX

M

H

E

w

j

y

R

p

k

7

5

Z

K

<latexi sh

1_b

64="90Op

+Dm8Ak

7Tr2y d/F

>

B

3

c

V

LS

N

U

vq

GzW

RX

fC

n

o

Q

M

u

I

K

H

g

P

Z

j

w

Y

<latexi sh

1_b

64="PycTj kNC

SIzn

B

F/KH

Y

>A

EX

ZDL

9V

r

qu3

wW

J

M

p

QdO

m

7

+

o

v

0

f

8

2

G

U

g

5

R

<latexi sh

1_b

64="Od

IBR

mzHcy

g

wf7GKrVJ

8

>A

X

DLT

M

EN

jFok

CW

9S

p

2

U

+

Y

/

u

0

n

P

3Z

Q

v

5

q

<latexi sh

1_b

64="PKDS

dm2WAo

EF3

p

UV

j

>

B/n

c

L

N

J

r

u

vgq

T

0

7

9

kz

G

O

Y

w

X

Z

C

y

+

Q

M

8

I

f

H

5

R

<latexi sh

1_b

64="

2+F9CoEAR

p8J

gPrn

Zk

Ic

>

B7

V

NS

3U

q/

L

0

K

D

wm

G

vH

T

u

5

Q

M

f

dW

j

y

O

X

z

Y

<latexi sh

1_b

64="

Q

jFp

LG

8

qRPuJfB

/Km

>A

3

c

VD

S

N

U

2v

O

zW

X

E

C

n

o

9

M

y

T

I

d

0H

g7

k

r

5

+

Z

w

Y

CVA

ELA

ILA

(a) Pretraining NeRF with inpainting prior

(b) Inpainting with latents alignment

φ

θ

θ

I

i

M

i

I

′

p

˜

I

p

˜

I

i

I

′

i

z

T

p

z

T

i

L

rec

L

prior

L

patch

/a0

∇

σ

米

米

where R unmasked represent the unmasked pixels across all the training images. Then we sample a prior view I p with its mask M p and regularly inpaint it using Stable Diffusion θ . For illustration, we replace of condition in Eq. 2 with two components used by Stable Inpainting Diffusion [71] as e for the input prompt, and I ′ p for the masked image that fed into the diffusion models. Hence the inpainting process can be formulated as:

The inpainted image then can be obtained by ˜ I p = Ω( z 0 p ) . We then use a monocular depth estimator on ˜ I p to get a depth map ˜ D p . We regress the scale and offset parameters to align ˜ D p with the depth estimated from field ϕ on the unmasked pixels. Hence, we can introduce the geometry and appearance supervision of the inpainting prior ˜ I p into the NeRF's optimization through:

where R masked(p) denoted the masked (inpainted) pixels of ˜ I p , and ˆ D ϕ is the depth estimated by NeRF. This stage is depicted in Fig. 2(a).

4.2 Stage 2: Latents Alignment

In this section, we introduce our key approach to condition the additional inpainted frames to have an inpainting feature based on the prior ˜ I p . As discussed before, in deterministic sampling of the diffusion inpainting model θ , the generation structure and layout highly depend on (1) initial sampled noise z T i and (2) intermediate predicted latents { z t -1 i } T t =1 . Hence if we can align the latents from different views with the prior one, the model is likely to generate multi-view consistent latent z 0 i , hence image ˜ I i . In this section, we discuss how to align two terms respectively.

<latexi sh

1_b

64="CSyX

W

Uz

LAdk wM8Z5m

g

>

B

3

c

V

N

EJ

r

q/

9

0n

F

7

o

2

p

Q

+

D

I

GT

u

j

Y

O

f

v

R

K

P

H

<latexi sh

1_b

64="

2qk

+3HrGu

NQLP08T

p

M

>A

B

n

c

VDJSg

EK

W

OX

Cj

d

R

y

U

o

v

5

/

F

w

f

9

Y

I

z

Z

m

7

<latexi sh

1_b

64="yS

P

fO

T

o

r7G

NK

508

>A

B

X

c

VDJ

g

E

2

W

Cj

d

R

L

M

k

pZ

u

Q

/

v

3

F

w

+

qn

m

9Y

H

z

I

U

φ

σ

z

T

θ

θ

I

′

p

z

T

p

I

′

i

z

T

i

K

p

, V

p

: Self Attn

(a) Explicit Initial Latent Alignment (ELA)

(b) Implicit Intermediate Latents Alignment (ILA)

Explicit Initial Latent Alignment (ELA) Given the sampled latent of prior view z T p used in Stage 1 (Sec. 4.1), we could leverage geometric information to explicitly align the initial latent in 3D space. Given that estimated depth ˜ D p , one possible solution is to warp the z T p into other views using camera matrices. However, ˜ D p is not guaranteed to be accurate hence such hard projection could yield significant errors. Alternatively, we propose to leverage the pre-trained NeRF ϕ , as it's a naturally 3D-consistent representation. Specifically, to sample a resolution-grain initial latent z T ( r ) , we utilize the original formulation of volume rendering (Eq.1) but with the substitution of color c by latent z T . We query the density σ from ϕ , and acquire z T by reprojecting the sampled point to the image plane of the prior latent view z T p :

where f p,i denote camera perspective projection according to p and i camera matrices. Such soft projection could avoid error accumulation in the inpainting process, and reduce the precision burden on the depth estimator. We illustrate this process in Fig. 3a. There are two key reasons why we propose fine-tuning the NeRF and using it as a geometric prior for ELA: (a) After finetuning the NeRF, the geometric is represented by NeRF as a sharp (low variance) unimodal distribution on the ray. Consequently, the aggregated feature remains sharp, preserving the variations in the initial latents. (b) We empirically found the depth prior inferred by the monocular depth estimator is not perfectly aligned with the NeRF. Fine-tuning the NeRF can also benefit this depth prior. Since NeRF learns relatively certain geometry in the known (unmasked) areas, this geometry constraint can improve the geometry of neighboring inpainted (masked) areas due to their geometric proximity. We compromise the view-dependent effect in NeRF within the ELA module. Due to the heuristic nature of diffusion models, incorporating such view-dependent effects into diffusion models' output remains elusive.

Implicit Intermediate Latents Alignment (ILA) While the initial latent could be aligned using the explicit method, intermediate latents are predicted by denoising network ϵ θ which is hard to control. We address this issue by exploring the conditioning mechanism of the denoising network in Stable Diffusion [71]. Recall that in Eq. 4, denoising network ϵ θ relies on the input prompt e and masked image I ′ i to predict the noise occurrence in the current step. While we can use the unified prompt for all views to align the text condition in cross-attention of ϵ θ , the masked images {I ′ i } N i =1 are inherently different due to multi-view nature. Note that I ′ i condition is introduced based on spatial self-attention (SA) [90] in the U-Net:

where Q i obtained from each spatial resolution of the latent, K i , V i are derived from corresponding latent encoded from masked image I ′ i . We can impose the coherence of the denoising step by introducing cross-view attention (CVA) of the prior view (Fig. 3b):

<latexi sh

1_b

64="

2

y/

w+

EgoSRJd

VvB

uQ

>A

7X

c

DLT

M

Nz

O

jF

k

CWI

G

8

9

p

rUf

Y

m

0

n

P

3Z

H

K5

q

<latexi sh

1_b

64="PKDS

dm2WAo

EF3

p

UV

j

>

B/n

c

L

N

J

r

u

vgq

T

0

7

9

kz

G

O

Y

w

X

Z

C

y

+

Q

M

8

I

f

H

5

R

<latexi sh

1_b

64="u

A0

5jzfk

8pBcy g9oSKVU

>

/n

DL

M

F

3

rP

q7

Iv

2XR

d

GTZ

QJ

O

C

Nm

Y

W

w

E

+

H

<latexi sh

1_b

64="Od

IBR

mzHcy

g

wf7GKrVJ

8

>A

X

DLT

M

EN

jFok

CW

9S

p

2

U

+

Y

/

u

0

n

P

3Z

Q

v

5

q

<latexi sh

1_b

64="

2+F9CoEAR

p8J

gPrn

Zk

Ic

>

B7

V

NS

3U

q/

L

0

K

D

wm

G

vH

T

u

5

Q

M

f

dW

j

y

O

X

z

Y

<latexi sh

1_b

64="

2+F9CoEAR

p8J

gPrn

Zk

Ic

>

B7

V

NS

3U

q/

L

0

K

D

wm

G

vH

T

u

5

Q

M

f

dW

j

y

O

X

z

Y

<latexi sh

1_b

64="T

r

m0nWoyJMY

u9FZ

cO

>A

B73

VDLSgN

E

fU

P

Q

K

+

Gv

Ip

H

5

z

j

8

/

qC

dX

k

2

w

R

where K p , V p are from masked base image I ′ p . We can then implicitly align the denoising step by replacing the original SA with a weighted sum from SA and CVA, i.e. λ a ∗ SA ( Q i , K i , V i ) + (1 -λ a ) ∗ CVA ( Q i , K p , V p ) . Through this technique we ensure that the intermediate latents { z t -1 i } T t =1 are also conditioned on the prior { z t -1 p } T t =1 , while retain its distinctiveness due to the individual viewpoint. The rationale of replacing KV with "prior" p , but not Q is that the appearance information ( V ) of the prior image should be considered when inpainting the other views, with the amount of information propagation is weighted by its attention key value ( K ). The attention query value comes from the current inpainting view i , Q i , representing the information the current inpainting for view i is searching for. Together with K p , it decides how much attention the view i inpainter should place on the prior view, and finally incorporates the information of the prior view V p into view i .

4.3 Stage 3: Joint Optimization

As the original intention of this work, we seek to distill the inpainted views into NeRF ϕ in a way such that the high-fidelity is preserved as much as possible as the unmasked region. While some priors work in 3D editing [25, 96, 97, 93] propose to update the training set iteratively until converge, we empirically find it not suitable for inpainting task since the loss of fidelity is significant and could fall into local optima. Hence we propose to inpaint a subset of training images at once and regard them as supplementary guidance using a patch-based hybrid loss:

where ρ is a patch sample from the masked area of subset views P sub , ˆ I ϕ ( ρ ) is NeRF predicted patch, and L lpips , L adv are perceptual distance LPIPS [106] and adversarial loss [24]. Here LPIPS is utilized to address geometry mismatches, while adversarial loss is employed to preserve high-frequency details. As shown in Fig. 2, the final optimization objective is:

The patches are uniformly sampled within the bounding box of the mask, with a size of 256×256. Therefore, only the inpainted area is being optimized by the patch loss.

5 Experiment

5.1 Evaluation Setting

Dataset: Aligning with methodologies employed in prior works [53, 51, 109], our experiments utilize the SPIn-NeRF dataset [53], selected for its comprehensive ground truth availability. This dataset is specifically designed for object removal evaluations and comprises 10 scenes. Each scene includes 60 images featuring an unwanted object (training views) and 40 images from which the object has been removed (test views). For both the training and test views, human-annotated masks indicating the object region are available. We further collected 9 forward-facing scenes with manually annotated masks to evaluate the effectiveness of our method. This dataset includes 4 indoor scenes and 5 outdoor scenes. In the training set, the masked region contains the unwanted object, while the test set contains the ground truth background in the masked region.

Baselines: In our study, we benchmark our method against a variety of established 3D inpainting approaches to ascertain its relative performance. These include the perceptual-based SPIn-NeRF [53], tiling-based NeRFiller [93] (both multi-view guidance) and InFusion [109] (single-view guidance). To ensure a fair comparison, we employ the same inpainting diffusion models across all methods and maintain consistency in the number of denoising steps and used prompts. We utilized the source code provided by the authors and ran all the methods using one NVIDIA A100 (80G) GPU.

Metrics: To quantitatively evaluate the effectiveness of our approach, we employ two similarity metrics: LPIPS [106] and FID [28]. Additionally, we use MUSIQ [36], a sharpness metric that quantifies the clarity and detail retention in the edited images. Following established protocols from previous studies [53], all metrics are calculated specifically within the bounding boxes defined by the masks, focusing the evaluation precisely on the regions most affected by the object removal task.

SPIn-NeRF

NeRFiller

Ours

InFusion

5.2 Main Results

We first present the quantitative results in Tab. 1, where our method outperforms all baselines in terms of similarity metrics and sharpness. Our approach also excels in qualitative assessments, as demonstrated in Fig. 4. It is important to note that in simpler scenes with low variability in inpainting results, where the inconsistency issue is less pronounced (first row), most methods perform adequately. In other cases, high-frequency loss is observed in multi-view-based methods (SPIn-NeRF and NeRFiller). NeRFiller [93], through its use of multiple joint denoising steps, ensures consistency but often produces overly smooth outputs that lack fine details. It is noteworthy that the singleview-based method, InFusion [109], relies on one view and its depth to represent the entire scene. It performs well when geometry estimation is accurate. However, its performance deteriorates in scenarios where depth accuracy is compromised, leading to geometry artifacts (sixth and seventh rows). This underscores the critical role of multi-view supervision in addressing such challenges. By incorporating consistent multi-view supervision, our method remains effective even when depth or geometry is inaccurate, achieving robust and promising results. This explains why our method shows little difference from InFusion when the geometry is accurate (first and third row), but excels when the depth is inaccurate. Additionally, the exclusive reliance on perceptual loss by SPIn-NeRF [53] fails to fully address the multi-view inconsistencies introduced by inpainting diffusion models, often resulting in a blurred effect, particularly visible in the third and fourth rows. To further validate our findings, we conducted a user study based on the SPIn-NeRF dataset, focusing on the coherence of the background within the inpainted area, the fidelity of detail preservation in the inpainted region, and overall preference. The results of this study are summarized in Tab. 2. This evaluation clearly demonstrates superior performance across all assessed criteria.

5.3 Ablation Studies

We initially demonstrate the efficacy of our latents alignment approach with an example in Fig. 5. The first column displays the inpainting prior (sampled view), and the subsequent columns show the same training image being inpainted under different conditions. Notably, the variant without ELA (w/o ELA) retains colors similar to the prior but fails to preserve the texture structure. Conversely, the version without ILA (w/o ILA) maintains structural integrity but lacks appearance consistency with the prior. Our method effectively merges the strengths of both mechanisms, resulting in inpaintings that are highly consistent and cohesive across all evaluated aspects.

Inpainting prior

Ours w/o ELA

Ours w/o ILA

Ours

We conducted further ablation studies to underscore the importance of our key design elements in object removal tasks. The quantitative and qualitative results, showcased in Tab. 3 and Fig. 6, clearly indicate the impact of each component. Notably, removing ELA leads to geometry mismatches in the NeRF outputs (w/o ELA), while deactivating ILA results in blurry coloration (w/o ILA). This observation confirms our initial findings: the initial latents primarily influence the inpainting's structural pattern, whereas the intermediate denoising steps largely affect its appearance, including

Table 2: User Study

colour nuances. Additionally, our patch-based loss plays a crucial role in the optimization process (w/o L patch). Specifically, the L lpips loss helps to alleviate geometry mismatches (w/o L lpips), and the L adv serves as a detail-preserving supervisor (w/o L adv). These results highlight the effectiveness of our design choices in enhancing the overall quality and coherence of the inpainted outputs.

Ours full model

Ours w/o ELA

Ours w/o ILA

Ours w/o patch

Ours w/o vgg

Ours w/o adv

L

patch

L

lpips

L

adv

6 Conclusion

In this work, we demonstrate the significant improvement achieved through our novel latents alignment approach in 3D object removal. By integrating both explicit and implicit latent alignment mechanisms, we have successfully addressed key challenges associated with geometry mismatches and color inconsistencies that are prevalent in the baselines, enhancing the fidelity and detail of the inpainted 3D scenes. The improvements achieved through our work offer significant societal benefits, such as enhanced editability of radiance fields. However, it also poses risks, including the potential perpetuation of biases and discrimination. If the data used to train diffusion models is biased, our approach could inadvertently reinforce these biases.

Despite notable advancements, our method has limitations: (1) It struggles with full 3D consistency, especially on high-frequency details, due to the constraints of applying 2D diffusion models to multi-view data. Future work could address this by integrating multi-view training into 2D inpainting diffusion models or leveraging true 3D generative models. (2) It is tailored for forward-facing scenes, limiting its applicability to diverse 360° views. Further exploration of latent relationships for broader view coverage is needed. (3) Predefined masks are currently required. Integrating advanced 3D perception methods [66, 67] could enhance accuracy and flexibility, enabling precise language-driven interactions and creating a more automated, user-friendly framework for neural 3D scene editing.

References

Y. Balaji, S. Nah, X. Huang, A. Vahdat, J. Song, Q. Zhang, K. Kreis, M. Aittala, T. Aila, S. Laine, et al. ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01324 , 2022.

J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5470-5479, 2022.

J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman. Zip-nerf: Anti-aliased grid-based neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 19697-19705, 2023.

M. A. Bautista, P. Guo, S. Abnar, W. Talbott, A. Toshev, Z. Chen, L. Dinh, S. Zhai, H. Goh, D. Ulbricht, et al. Gaudi: A neural architect for immersive 3d scene generation. Advances in Neural Information Processing Systems , 35:25102-25116, 2022.

T. Brooks, A. Holynski, and A. A. Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18392-18402, 2023.

A. Cao and J. Johnson. Hexplane: A fast representation for dynamic scenes. CVPR , 2023.

M. Cao, X. Wang, Z. Qi, Y. Shan, X. Qie, and Y. Zheng. Masactrl: Tuning-free mutual selfattention control for consistent image synthesis and editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 22560-22570, October 2023.

J. Cen, Z. Zhou, J. Fang, C. Yang, W. Shen, L. Xie, X. Zhang, and Q. Tian. Segment anything in 3d with nerfs. In NeurIPS , 2023.

L. Cerkezi, A. Davtyan, S. Sameni, and P. Favaro. Multi-view unsupervised image generation with cross attention guidance. arXiv preprint arXiv:2312.04337 , 2023.

A. Chen, Z. Xu, A. Geiger, J. Yu, and H. Su. Tensorf: Tensorial radiance fields. In European Conference on Computer Vision (ECCV) , 2022.

R. Chen, Y. Chen, N. Jiao, and K. Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 22246-22256, 2023.

Y. Chen, Z. Chen, C. Zhang, F. Wang, X. Yang, Y. Wang, Z. Cai, L. Yang, H. Liu, and G. Lin. Gaussianeditor: Swift and controllable 3d editing with gaussian splatting, 2023.

Z. Chen, T. Funkhouser, P. Hedman, and A. Tagliasacchi. Mobilenerf: Exploiting the polygon rasterization pipeline for efficient neural field rendering on mobile architectures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1656916578, 2023.

Y.-C. Cheng, H.-Y. Lee, S. Tulyakov, A. G. Schwing, and L.-Y. Gui. Sdfusion: Multimodal 3d shape completion, reconstruction, and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4456-4465, 2023.

J. Chung, S. Lee, H. Nam, J. Lee, and K. M. Lee. Luciddreamer: Domain-free generation of 3d gaussian splatting scenes. arXiv preprint arXiv:2311.13384 , 2023.

X. Dai, J. Hou, C.-Y. Ma, S. Tsai, J. Wang, R. Wang, P. Zhang, S. Vandenhende, X. Wang, A. Dubey, et al. Emu: Enhancing image generation models using photogenic needles in a haystack. arXiv preprint arXiv:2309.15807 , 2023.

K. Deng, A. Liu, J.-Y. Zhu, and D. Ramanan. Depth-supervised NeRF: Fewer views and faster training for free. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2022.

P. Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems , 34:8780-8794, 2021.

J. Dong and Y.-X. Wang. Vica-nerf: View-consistency-aware 3d editing of neural radiance fields. Advances in Neural Information Processing Systems , 36, 2024.

Q. Dong, C. Cao, and Y. Fu. Incremental transformer structure enhanced image inpainting with masking positional encoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11358-11368, 2022.

A. A. Efros and T. K. Leung. Texture synthesis by non-parametric sampling. In Proceedings of the seventh IEEE international conference on computer vision , volume 2, pages 1033-1038. IEEE, 1999.

D. Epstein, A. Jabri, B. Poole, A. Efros, and A. Holynski. Diffusion self-guidance for controllable image generation. Advances in Neural Information Processing Systems , 36: 16222-16239, 2023.

S. Fridovich-Keil, A. Yu, M. Tancik, Q. Chen, B. Recht, and A. Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5501-5510, 2022.

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. Advances in neural information processing systems , 27, 2014.

A. Haque, M. Tancik, A. A. Efros, A. Holynski, and A. Kanazawa. Instruct-nerf2nerf: Editing 3d scenes with instructions. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 19740-19750, 2023.

A. Hertz, R. Mokady, J. Tenenbaum, K. Aberman, Y. Pritch, and D. Cohen-Or. Prompt-toprompt image editing with cross attention control. arXiv preprint arXiv:2208.01626 , 2022.

A. Hertz, K. Aberman, and D. Cohen-Or. Delta denoising score. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2328-2337, 2023.

M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems , 30, 2017.

J. Ho and T. Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 , 2022.

J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems , 33:6840-6851, 2020.

Y. Hong, K. Zhang, J. Gu, S. Bi, Y. Zhou, D. Liu, F. Liu, K. Sunkavalli, T. Bui, and H. Tan. Lrm: Large reconstruction model for single image to 3d. arXiv preprint arXiv:2311.04400 , 2023.

B. Hu, J. Huang, Y. Liu, Y.-W. Tai, and C.-K. Tang. Nerf-rpn: A general framework for object detection in nerfs. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . IEEE, June 2023. doi: 10.1109/cvpr52729.2023.02253. URL http: //dx.doi.org/10.1109/CVPR52729.2023.02253 .

E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 , 2021.

N. Kalischek, T. Peters, J. D. Wegner, and K. Schindler. Tetrahedral diffusion models for 3d shape generation. arXiv preprint arXiv:2211.13220 , 2022.

T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8110-8119, 2020.

J. Ke, Q. Wang, Y. Wang, P. Milanfar, and F. Yang. Musiq: Multi-scale image quality transformer. In Proceedings of the IEEE/CVF international conference on computer vision , pages 5148-5157, 2021.

B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis. 3d gaussian splatting for realtime radiance field rendering. ACM Transactions on Graphics , 42(4), July 2023. URL https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/ .

S. W. Kim, B. Brown, K. Yin, K. Kreis, K. Schwarz, D. Li, R. Rombach, A. Torralba, and S. Fidler. Neuralfield-ldm: Scene generation with hierarchical latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8496-8506, 2023.

Y. Liang, X. Yang, J. Lin, H. Li, X. Xu, and Y. Chen. Luciddreamer: Towards high-fidelity text-to-3d generation via interval score matching. arXiv preprint arXiv:2311.11284 , 2023.

C.-H. Lin, J. Gao, L. Tang, T. Takikawa, X. Zeng, X. Huang, K. Kreis, S. Fidler, M.-Y. Liu, and T.-Y. Lin. Magic3d: High-resolution text-to-3d content creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 300-309, 2023.

Y. Lin, H. Han, C. Gong, Z. Xu, Y. Zhang, and X. Li. Consistent123: One image to highly consistent 3d asset using case-aware diffusion priors. arXiv preprint arXiv:2309.17261 , 2023.

H.-K. Liu, I. Shen, B.-Y. Chen, et al. Nerf-in: Free-form nerf inpainting with rgb-d priors. arXiv preprint arXiv:2206.04901 , 2022.

K. Liu, F. Zhan, Y. Chen, J. Zhang, Y. Yu, A. E. Saddik, S. Lu, and E. Xing. Stylerf: Zero-shot 3d style transfer of neural radiance fields. 2023.

Z. Liu, Y. Feng, M. J. Black, D. Nowrouzezahrai, L. Paull, and W. Liu. Meshdiffusion: Score-based generative 3d mesh modeling. arXiv preprint arXiv:2303.08133 , 2023.

X. Long, Y.-C. Guo, C. Lin, Y. Liu, Z. Dou, L. Liu, Y. Ma, S.-H. Zhang, M. Habermann, C. Theobalt, et al. Wonder3d: Single image to 3d using cross-domain diffusion. arXiv preprint arXiv:2310.15008 , 2023.

Z. Lu, J. Jiang, J. Huang, G. Wu, and X. Liu. Glama: Joint spatial and frequency loss for general image inpainting. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 1301-1310, 2022.

A. Lugmayr, M. Danelljan, A. Romero, F. Yu, R. Timofte, and L. Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11461-11471, 2022.

N. Max. Optical models for direct volume rendering. IEEE Transactions on Visualization and Computer Graphics , 1(2):99-108, 1995.

G. Metzer, E. Richardson, O. Patashnik, R. Giryes, and D. Cohen-Or. Latent-nerf for shapeguided generation of 3d shapes and textures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12663-12673, 2023.

B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM , 65(1):99-106, 2021.

A. Mirzaei, T. Aumentado-Armstrong, M. A. Brubaker, J. Kelly, A. Levinshtein, K. G. Derpanis, and I. Gilitschenski. Reference-guided controllable inpainting of neural radiance fields. In ICCV , 2023.

A. Mirzaei, T. Aumentado-Armstrong, M. A. Brubaker, J. Kelly, A. Levinshtein, K. G. Derpanis, and I. Gilitschenski. Watch your steps: Local image and scene editing by text instructions. arXiv preprint arXiv:2308.08947 , 2023.

A. Mirzaei, T. Aumentado-Armstrong, K. G. Derpanis, J. Kelly, M. A. Brubaker, I. Gilitschenski, and A. Levinshtein. SPIn-NeRF: Multiview segmentation and perceptual inpainting with neural radiance fields. In CVPR , 2023.

A. Mirzaei, R. De Lutio, S. W. Kim, D. Acuna, J. Kelly, S. Fidler, I. Gilitschenski, and Z. Gojcic. Reffusion: Reference adapted diffusion models for 3d scene inpainting. arXiv preprint arXiv:2404.10765 , 2024.

R. Mokady, A. Hertz, K. Aberman, Y. Pritch, and D. Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6038-6047, 2023.

C. Mou, X. Wang, J. Song, Y. Shan, and J. Zhang. Dragondiffusion: Enabling drag-style manipulation on diffusion models. arXiv preprint arXiv:2307.02421 , 2023.

C. Mou, X. Wang, J. Song, Y. Shan, and J. Zhang. Diffeditor: Boosting accuracy and flexibility on diffusion-based image editing. arXiv preprint arXiv:2402.02583 , 2024.

T. Müller, A. Evans, C. Schied, and A. Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM transactions on graphics (TOG) , 41(4):1-15, 2022.

A. Q. Nichol and P. Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning , pages 8162-8171. PMLR, 2021.

M. Niemeyer and A. Geiger. Giraffe: Representing scenes as compositional generative neural feature fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11453-11464, 2021.

X. Pan, A. Tewari, T. Leimkühler, L. Liu, A. Meka, and C. Theobalt. Drag your gan: Interactive point-based manipulation on the generative image manifold. In ACM SIGGRAPH 2023 Conference Proceedings , pages 1-11, 2023.

J. Park, G. Kwon, and J. C. Ye. Ed-nerf: Efficient text-guided editing of 3d scene using latent space nerf. arXiv preprint arXiv:2310.02712 , 2023.

K. Park, U. Sinha, J. T. Barron, S. Bouaziz, D. B. Goldman, S. M. Seitz, and R. Martin-Brualla. Nerfies: Deformable neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5865-5874, 2021.

B. Poole, A. Jain, J. T. Barron, and B. Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988 , 2022.

K. Prabhu, J. Wu, L. Tsai, P. Hedman, D. B. Goldman, B. Poole, and M. Broxton. Inpaint3d: 3d scene content generation using 2d inpainting diffusion. arXiv preprint arXiv:2312.03869 , 2023.

M. Qin, W. Li, J. Zhou, H. Wang, and H. Pfister. Langsplat: 3d language gaussian splatting. arXiv preprint arXiv:2312.16084 , 2023.

Y. Qu, S. Dai, X. Li, J. Lin, L. Cao, S. Zhang, and R. Ji. Goi: Find 3d gaussians of interest with an optimizable open-vocabulary semantic-space hyperplane. arXiv preprint arXiv:2405.17596 , 2024.

A. Raj, S. Kaza, B. Poole, M. Niemeyer, N. Ruiz, B. Mildenhall, S. Zada, K. Aberman, M. Rubinstein, J. Barron, et al. Dreambooth3d: Subject-driven text-to-3d generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 23492359, 2023.

A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 1(2):3, 2022.

J. Ren, L. Pan, J. Tang, C. Zhang, A. Cao, G. Zeng, and Z. Liu. Dreamgaussian4d: Generative 4d gaussian splatting. arXiv preprint arXiv:2312.17142 , 2023.

R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684-10695, 2022.

V. Rudnev, M. Elgharib, W. Smith, L. Liu, V. Golyanik, and C. Theobalt. Nerf for outdoor scene relighting. In European Conference on Computer Vision (ECCV) , 2022.

C. Saharia, W. Chan, H. Chang, C. Lee, J. Ho, T. Salimans, D. Fleet, and M. Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 conference proceedings , pages 1-10, 2022.

C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems , 35: 36479-36494, 2022.

A. Sargsyan, S. Navasardyan, X. Xu, and H. Shi. Mi-gan: A simple baseline for image inpainting on mobile devices. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7335-7345, 2023.

E. Sella, G. Fiebelman, P. Hedman, and H. Averbuch-Elor. Vox-e: Text-guided voxel editing of 3d objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 430-440, 2023.

P. Shamsolmoali, M. Zareapoor, and E. Granger. Transinpaint: Transformer-based image inpainting with context adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 849-858, 2023.

Y. Siddiqui, A. Alliegro, A. Artemov, T. Tommasi, D. Sirigatti, V. Rosov, A. Dai, and M. Nießner. Meshgpt: Generating triangle meshes with decoder-only transformers. arXiv preprint arXiv:2311.15475 , 2023.

J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning , pages 2256-2265. PMLR, 2015.

J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 , 2020.

C. Sun, M. Sun, and H.-T. Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5459-5469, 2022.

R. Suvorov, E. Logacheva, A. Mashikhin, A. Remizova, A. Ashukha, A. Silvestrov, N. Kong, H. Goka, K. Park, and V. Lempitsky. Resolution-robust large mask inpainting with fourier convolutions. In Proceedings of the IEEE/CVF winter conference on applications of computer vision , pages 2149-2159, 2022.

M. Tancik, E. Weber, E. Ng, R. Li, B. Yi, J. Kerr, T. Wang, A. Kristoffersen, J. Austin, K. Salahi, A. Ahuja, D. McAllister, and A. Kanazawa. Nerfstudio: A modular framework for neural radiance field development. In ACM SIGGRAPH 2023 Conference Proceedings , SIGGRAPH '23, 2023.

J. Tang, J. Ren, H. Zhou, Z. Liu, and G. Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309.16653 , 2023.

J. Tang, Z. Chen, X. Chen, T. Wang, G. Zeng, and Z. Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint arXiv:2402.05054 , 2024.

L. Tang, N. Ruiz, Q. Chu, Y. Li, A. Holynski, D. E. Jacobs, B. Hariharan, Y . Pritch, N. Wadhwa, K. Aberman, et al. Realfill: Reference-driven generation for authentic image completion. arXiv preprint arXiv:2309.16668 , 2023.

M. A. Uy, R. Martin-Brualla, L. Guibas, and K. Li. Scade: Nerfs from space carving with ambiguity-aware depth estimates. In Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

M. A. Uy, R. Martin-Brualla, L. Guibas, and K. Li. Scade: Nerfs from space carving with ambiguity-aware depth estimates. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16518-16527, 2023.

A. Vahdat, F. Williams, Z. Gojcic, O. Litany, S. Fidler, K. Kreis, et al. Lion: Latent point diffusion models for 3d shape generation. Advances in Neural Information Processing Systems , 35:10021-10039, 2022.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

D. Wang, T. Zhang, A. Abboud, and S. Süsstrunk. Inpaintnerf360: Text-guided 3d inpainting on unbounded neural radiance fields. arXiv preprint arXiv:2305.15094 , 2023.

Q. Wang, Z. Wang, K. Genova, P. P. Srinivasan, H. Zhou, J. T. Barron, R. Martin-Brualla, N. Snavely, and T. Funkhouser. Ibrnet: Learning multi-view image-based rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4690-4699, 2021.

E. Weber, A. Holynski, V. Jampani, S. Saxena, N. Snavely, A. Kar, and A. Kanazawa. Nerfiller: Completing scenes via generative 3d inpainting. In CVPR , 2024.

S. Weder, G. Garcia-Hernando, Á. Monszpart, M. Pollefeys, G. Brostow, M. Firman, and S. Vicente. Removing objects from neural radiance fields. In CVPR , 2023.

H. Weng, T. Yang, J. Wang, Y. Li, T. Zhang, C. Chen, and L. Zhang. Consistent123: Improve consistency for one image to 3d object synthesis. arXiv preprint arXiv:2310.08092 , 2023.

J. Wu, J.-W. Bian, X. Li, G. Wang, I. Reid, P. Torr, and V . A. Prisacariu. Gaussctrl: Multi-view consistent text-driven 3d gaussian splatting editing. arXiv preprint arXiv:2403.08733 , 2024.

R. Wu, B. Mildenhall, P. Henzler, K. Park, R. Gao, D. Watson, P. P. Srinivasan, D. Verbin, J. T. Barron, B. Poole, et al. Reconfusion: 3d reconstruction with diffusion priors. arXiv preprint arXiv:2312.02981 , 2023.

L. Yang, B. Kang, Z. Huang, X. Xu, J. Feng, and H. Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In CVPR , 2024.

J. Ye, P. Wang, K. Li, Y. Shi, and H. Wang. Consistent-1-to-3: Consistent image to 3d view synthesis via geometry-aware diffusion models. arXiv preprint arXiv:2310.03020 , 2023.

M. Ye, M. Danelljan, F. Yu, and L. Ke. Gaussian grouping: Segment and edit anything in 3d scenes. arXiv preprint arXiv:2312.00732 , 2023.

T. Yi, J. Fang, J. Wang, G. Wu, L. Xie, X. Zhang, W. Liu, Q. Tian, and X. Wang. Gaussiandreamer: Fast generation from text to 3d gaussians by bridging 2d and 3d diffusion models. In CVPR , 2024.

T. Yu, R. Feng, R. Feng, J. Liu, X. Jin, W. Zeng, and Z. Chen. Inpaint anything: Segment anything meets image inpainting. arXiv preprint arXiv:2304.06790 , 2023.

J. Zhang, Z. Tang, Y. Pang, X. Cheng, P. Jin, Y . Wei, W. Yu, M. Ning, and L. Yuan. Repaint123: Fast and high-quality one image to 3d generation with progressive controllable 2d repainting. arXiv preprint arXiv:2312.13271 , 2023.

L. Zhang, Y. Zhou, C. Barnes, S. Amirghodsi, Z. Lin, E. Shechtman, and J. Shi. Perceptual artifacts localization for inpainting. In European Conference on Computer Vision , pages 146-164. Springer, 2022.

L. Zhang, A. Rao, and M. Agrawala. Adding conditional control to text-to-image diffusion models, 2023.

R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR , 2018.

M. Zheng, Z. Haiyu, H. Yang, and D. Huang. Neuface: Realistic 3d neural face rendering from multi-view images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023.

Y. Zheng, X. Li, K. Nagano, S. Liu, O. Hilliges, and S. De Mello. A unified approach for text-and image-guided 4d scene generation. arXiv preprint arXiv:2311.16854 , 2023.

Q. W. K. L. C. J. X. K. Z. N. X. Y. L. Y. S. Y. C. Zhiheng Liu, Hao Ouyang. Infusion: Inpainting 3d gaussians via learning depth completion from diffusion prior. arXiv preprint arXiv:2404.11613 , 2024.

J. Zhuang, C. Wang, L. Lin, L. Liu, and G. Li. Dreameditor: Text-driven 3d scene editing with neural fields. In SIGGRAPH Asia 2023 Conference Papers , pages 1-10, 2023.

A Supplemental Material for 'In-N-Out: Lifting 2D Diffusion Prior for 3D Object Removal via Tuning-Free Latents Alignment'

A.1 Implementation Detail

For the inpainting network, we employ the stable-diffusion-2-inpainting model [71], which encodes a masked image into the same dimensional latent space and integrates conditions via concatenation. We set the denoising steps for inpainting at 20. To achieve better generalization, we propose sampling the base frame according to the geometrical centroid of the training camera poses, meaning the camera that sits most centrally among the training views. However, we found that Stable Diffusion occasionally inpaints strange artifacts in the masked region. To mitigate this, we propose sampling n candidate views around the geometrical centroid and selecting the one with the highest similarity votes. This approach automatically avoids such occasion artifacts without human intervention. In our implementation, we used five candidate views, and the similarity was calculated using perceptual hashing. In the reprojection procedure of ELA, we adjust the camera intrinsics to match the latent dimensionality. Furthermore, to refine the ILA mechanism, we incorporate Cross-View Attention (CVA) into every self-attention layer of the inpainting model. Each step in this modified approach is controlled with λ a set at 0.2.

For our 3D representation (NeRF) implementation, we utilize the "nerfacto" framework proposed by NerfStudio [83]. To ensure stable training, we deactivated the view-dependent effect. We pre-train the NeRF using 10000 iterations in stage 1 and jointly optimize it using 5000 iterations in stage 3. Our monocular depth estimation adopts DepthAnything [98], complemented by the depth loss outlined in DS-NeRF [17]. Moreover, we employ StyleGAN2 discriminator [35] to implement adversarial loss.

A.2 Sensitivity Analysis

We conducted several sensitivity analyses regarding the base view selection, λ a in ILA, and the subset selection. Due to the computational burden, we conduct the sensitivity analysis on six out of ten scenes with higher inpainting variability from the SPIn-NeRF dataset.

(a) Base View Selection:

To achieve better generalization, we propose sampling the base frame according to the geometrical centroid of the training camera poses, meaning the camera that sits most centrally among the training views. However, we found that Stable Diffusion occasionally inpaints strange artifacts in the masked region. To mitigate this, we propose sampling n candidate views around the geometrical centroid and selecting the one with the highest similarity votes. This approach automatically avoids such occasion artifacts without human intervention. In our implementation, we used five candidate views, and the similarity was calculated using perceptual hashing.

We tested our results under different settings (candidate numbers): 3, 5, 7, and 9. The base frame selection algorithm proved to be robust, with our algorithm typically yielding the same base frame. However, another factor influencing this step is the random seed. Setting different seeds causes the 2D inpainting model to produce different results, leading to different base frames being selected. We tested our methods under five different seeds, and the final scores are reported in Table 4. While different seeds cause the final NeRF to differ in the appearance of the masked region, the consistency of the multi-view inpainting results remains robust, resulting in minimal variance in the evaluation scores.

(b) λ a in ILA:

To effectively examine the effect of the hyper-parameter λ a in ILA, we evaluated our method's rendering quality with different λ a values of 0.2, 0.4, 0.6, and 0.8. The metrics are reported in Table 5. Quantitatively, the results are consistent across different λ a values, indicating that the effect of this hyper-parameter is relatively small. This conclusion is also supported by qualitative results. Larger λ a values tend to produce slight variations in some small regions, but the global structure and semantics are preserved. This stability is attributed to the significant role of the initial latent alignment in ELA, which effectively aligns the underlying inpainting structure, thereby maintaining low variability in appearance. Additionally, the self-attention layer, where cross-view attention is introduced, does not dominate the entire Stable Diffusion Unet. It is balanced by the presence of

other (residual and linear) layers, ensuring cross-view attention does not override the signal during the denoising process. Hence we simply set λ a as 0 . 2 in our implementation.

(c) Subset Selection:

We found that for reconstruction tasks, more views can enhance quality; however, for generation tasks, using the entire set of images can introduce unnecessary inconsistencies. Therefore, we propose selecting the subset according to the distribution of camera viewpoints.

We evenly split the viewpoints into 12 groups based on the base view's camera space (evenly 2 on the x and y axes and 3 on the z-axis) and select 50 percent within each group according to perceptual hashing similarity to the base view. This approach avoids redundant views introducing supervision conflicts while covering different viewpoints for effective supervision.

We also evaluated our method based on different percentages, as reported in Table 6. The quantitative scores are quite close, indicating that for most scenes, the difference isn't significant. For one complex scene with extremely high frequencies, setting the percentage too low (0.2) yields artifacts in the test view due to insufficient viewpoint coverage. Conversely, setting the percentage too high (0.8) introduces appearance conflicts due to the high variability of the inpainted results.

Overall, for most scenes, the subset selection algorithm is robust due to the consideration of viewpoints distribution. For extreme cases, careful selection of the percentage might be necessary. However, values between 0.5 and 0.7 remain a reliable choice.

(d) λ patch in patch loss:

To assess the sensitivity of the patch loss multiplier λ patch , we evaluated the method's performance using various values of λ patch : 0.001, 0.005, 0.01, 0.05, and 0.1. The results are reported in Table 7. Analysis of the table indicates that varying λ patch leads to similar performance across different

settings, with a low standard deviation of the metrics. However, there is an observable trend where setting λ patch too low or too high adversely affects performance. The multiplier λ patch is critical as it determines the extent of influence multi-view images have on the NeRF. Insufficient multi-view supervision can lead to inadequate training, whereas excessive supervision may result in conflicting inputs. Consequently, we have set λ patch at 0.01 in our implementation for optimal balance.

A.3 More Qualitative Results

This section presents extended qualitative results from our experiments on the SPIn-NeRF Dataset. Fig. 7 and Fig. 8 showcase a series of multi-view comparative inpaintings.

A.4 Details on User Study and Impact

To comprehensively evaluate our method using human subjects, we conducted a user study focusing on three aspects: (1) Background Coherence - assessing whether the inpainted area blends seamlessly with the remaining background, (2) Detail Preservation - determining if the inpainted area retains high-fidelity details, and (3) Overall Quality - gauging participants' preference rates for the inpainted results. For each method, we presented users with two multi-view test images from each scene and instructed them to choose the method that best met the criteria for each aspect. Clear instructions were provided to ensure participants understood the rating process. An example screenshot of the study interface is shown in Fig. 9.

The user study we conducted focused solely on collecting participants' preferences regarding different inpainting results, involving no sensitive or personal data collection beyond their aesthetic judgments. The study's design was inherently low-risk as it required participants to simply view and evaluate digital images based on their visual appeal and perceived quality. Furthermore, the participation was entirely voluntary, with clear instructions provided, allowing participants to withdraw at any time without any consequence. Given these factors, the potential for harm or discomfort to participants was negligible, ensuring the study maintained a minimal risk profile.

InFusion

NeRFiller

Ours

SPIn-NeRF

InFusion

NeRFiller

Ours

SPIn-NeRF

3DInpaintingUserStudy

This study is an evaluation of 3D Inpainting, as a task that virtually remove an unwanted object in

3D scene and inpaint the occluded area.

We will show some visual results of different 3D inpainting methods. Please select the best one

according to:

1. Background Coherence: if the inpainted area is coherent with the remained background.

seep Ajepyqu au smoys eaue pequjedu aun JI :Bujesed leseg 7

3. Overal Quality: a preference rate for the inpainted results.

*1. This is the original view of the scene:

In below there are two different views without the box, from top.to bottom are method

1,2,_3,4.

1:

2:

3:

4:

Please select the best according to:

1

Background Coherence

O

O

O

Detall Perserve

O

O

O

O

Overal Quality

O

O

O

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In the abstract and introduction, we clearly state that our method contributes to the field of object removal in neural radiance fields, offering a novel and effective solution to improve multi-view consistency.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: In the conclusion (Sec. 6) , we discuss the limitations of our approach, outlining the current bottlenecks.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Our paper does not present theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We detail our implementation in Sec. A.1. The evaluation settings are clearly described in Sec. 5.1, ensuring a fair comparison by standardizing the input across all baselines.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: Due to the extensive work required for this task, we will need to clean up the code before its release. However, to ensure reproducibility, we provide detailed instructions in Sec. 4 and Sec. A.1. The code will be made available upon completion of the cleanup process.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We clearly stated the evaluation setting and implementation details in Sec. 5.1 and Sec. A.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Previous 3D object removal methods did not report error bars, and the computational resources required for these experiments were relatively high, making it impractical to replicate the experiments multiple times.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provided details on the computational hardware used in experiment (Sec. 5.1 and Sec. A.1).

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the NeurIPS Code of Ethics and ensured that our research adheres to its guidelines.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: In the conclusion (Sec. 6), we discuss the positive societal impacts of our research and state the potential harms of technology misuse.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: No new dataset or model is proposed in this work.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All existing assets used in this work are properly credited and are the license and terms of use explicitly mentioned and properly respected.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [Yes]

Justification: We have provided clear instructions for human evaluation in our user study and included a sample screenshot in the supplementary materials, presented in Sec. 5 and Sec. A.4.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [Yes]

Justification: We have explicitly informed all participants in the user study that the collected data will be used solely for research purposes.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper introduces a novel approach termed "In-N-Out" for enhancing the performance of 3D object removal tasks by leveraging tuning-free latents alignment. The authors have addressed the challenges of geometric mismatches and color inconsistencies prevalent in existing methods.

STRENGTHS:
The authors have conducted an extensive review of related work and clearly delineated the distinctions and connections between their work and existing literature.The explicit and implicit latents alignment proposed by the authors is an intriguing concept that could inspire work in related domains.

WEAKNESSES:
The paper exhibits signs of being finished in a rush, which affects the overall quality of academic writing, and is hard to follow.
For example，
1. Line172，Omega(·) is not defined
2. Formula (5), D_hat is note defined
3. Line 208， by replace -> by replacing
The paper might has the potential to contribute to the field but requires significant revisions to improve the academic writing quality.

The method involves several hyperparamete. The paper does not discuss the sensitivity analysis or the robustness of the method against variations in hyperparameters.  Especially，the lambda_a, which plays an essential role in ILA.

QUESTIONS:
1. In ELA module, using latent z^T to replace color c for volume rendering. Will this compromise the view-dependent in NeRF? In NeRF，the color  is dependent on the position x and direction d, whereas in ELA, the z^T is merely reprojected to the image plane. 

2. How do you choose lambda_a in ILA.

LIMITATIONS:
The method involves several hyperparamete. The paper does not discuss the sensitivity analysis or the robustness of the method against variations in hyperparameters.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper deals with 3D inpainting problem in the NeRF setup with 2D diffusion models. To achieve multiview consistency, they took a \"inpaint-outstretch\" strategy. They first inpaint one key frame, and conditioned on which, generate a view-consistent inpainted image set. Finally, they use the multiview images to train the final NeRF model, together with the reconstruction losses. The method is compared to a few recent works (Spin-NeRF, NeRFiller, InFusion) and achieved better results both qualitatively and quantitatively.\n\nSTRENGTHS:\n- The method is interesting and seems effective. They improve the consistency of multiview generation in two ways. The first is explicit latent alignment (ELA) that initializes the latent from the inpainted base frame. The second is ILA that post-hoc modifies the attention layer of stable diffusion models by replacing the key and value with the ones from the inpainted base frame. ILA does not need finetuning.\n\n- The method is evaluated on the SPIn-NeRF dataset and the performance gain over existing methods is quite large, both quantitatively and via user study. \n\n- There is also an ablation study that shows the importance of ELA, ILA, and the patch-based losses applied on the generated image sets.\n\n- The method is well written and easy to follow.\n\nWEAKNESSES:\n- My biggest confusion is on the explanation of ILA. \n  - I don't fully understand why replacing the KV with the base frame KV would encourage consistency. I would expect the style to be consistent but not the local geometry, where the latter is more important for 3d reconstruction.\n  - What is the rationale of replacing KV with \"prior\" p, but not Q?\n- The algorithm relies on a pre-defined base frame p. How is the base frame chosen, and how sensitive is the algorithm to the chosen frame?\n\nQUESTIONS:\n- Is Eq (10) applied to Stage 3? If so, what is the purpose of the prior loss L_prior? Once the inpainted base image is used to generate a multiview training set, I would expect this information to be unnecessary? Can this term be merged into L_patch? \n- In L216, what is the subset? Why not inpaint the whole set of images and how much does it affect the results?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
This paper presents how to tackle the challenge of 3D object removal by involving 2D diffusion prior. The approach involves pretraining NeRF with inpainting prior and then jointly optimizing it with latent alignment to align feature priors.

STRENGTHS:
1. Introducing a 2D diffusion prior as a solution for the 3D inpainting task is an innovative and interesting approach.
2. The incorporation of latent alignment to align feature priors is a promising design.

WEAKNESSES:
1. Although the generated results outperform the listed baseline methods, the paper lacks a comparison with other baselines, such as "Reference-guided controllable inpainting of neural radiance fields" (ICCV 2023)", which achieves good results of novel view synthesis. 
2. The presentation of the paper could be improved, for example, it is uncertain how to get the patches in the patch-based loss (Equ. 9).

QUESTIONS:
1. In Equ. 5, does the first term indicate the use of ground truth images (with the object removed) for supervising the mask region? If so, are there any concerns regarding potential information leakage? 
2. Regarding efficiency, does the multi-stage training approach in this paper have a comparison with baseline methods in terms of computational efficiency?

LIMITATIONS:
The author provides a comprehensive discussion of the broader impacts in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to solve multi-view or 3D inpainting problem. Given multiple posed views and the masks in each view specifying an object to be removed. It first trains a NeRF on the unmasked regions. To in-paint the masked regions, a seed view is selected and inpainted with a pre-trained diffusion inpainting pipeline. Depth is also predicted for this seed view, together with the RGB value, fused into the NeRF representation. To propagate the features from this view to other views for NeRF supervision, it proposes explicit and implicit feature alignment methods. For the explicit one, it propagates the initial latent noise to another view via a depth/density-based aggregation method.  For the implicit one, it uses a method similar to reference-only control, i.e., extracting intermediate key/value pairs from the seed view and injecting them into other inpainting views via cross-attention.\n\nSTRENGTHS:\n(1) Experiments show that properly propagating anchor-view to other views with methods shows good performance for multi-view consistency. More notably, propagating the initial noise to other views is novel and interesting.\n\nWEAKNESSES:\n(1) The proposed method seems much simpler(or even over-simplified) than prior works such as NeRFiller and InFusion. The former uses a synchronized multi-view inpainting technique which is similar to the ILA of this paper. The latter one introduces new diffusion-based depth completion. Both prior methods have used a depth prediction model, and InFusion has shown that using a pre-trained depth model and aligning it with  NeRF/Gaussian depth is sub-optimal.  \nFor qualitative comparisons, there is little or only marginal difference between this paper and InFusion, while InFusion/NeRFillter has shown much more results than only the scenes included in this paper. \nI suspect that the result difference is due to the choice of base components, (i.e., inpainting pipeline, depth estimation model, etc.) which can not be considered as the novelty of this paper.\n\n(2) As I understand, the three stages run sequentially. i.e., Train unmasked NeRF(Stage-1) -> Update the NeRF with seed view and depth(Stage-1) -> Update the NeRF with novel inpainted views (guided by the seed view)(Stage2 then 3). Although Equation. (10) shows a total loss function. \n- This way, the proposed ELA approach seems over-complicated. After stage 1, there is no NeRF information in the masked region, and there is only one view with depth provided to supervise the masked area. There seems no need to use the NeRF density to propagate the feature from the reference inpainted view to other views. An alternative can be using the depth prior and the epipolar line to aggregate features. \n\n(3) Prior works show both object insertion and removal tasks, while the proposed method seems can only be applied to object removal tasks, as it heavily relies on the initial inpainting and depth estimation result. Moreover, the examples shown in this paper don't include complex structures and only show simple background, inconsistency caused by possible occlusion or partial observation has not been tested. Overall, this paper compares the results with the previous generation's scene completion method in a very limited scenario, which I believe is unfair. \n\n(4) The artifacts are obvious in the result videos, which can be caused by the noise/artifacts in the depth estimation model. More evidence is needed to make this paper stronger.\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nLimitations addressed, however, the author has not mentioned that the proposed method is only tested in a limited scene case.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}