In-N-Out: Lifting 2D Diffusion Prior for 3D Object Removal via Tuning-Free Latents Alignment

Authors: dongting  hu
huan  fu
jiaxian  guo
liuhua  peng
tingjin  chu
feng  liu
tongliang  liu
mingming  gong
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 17:03:30

Abstract: Neural representations for 3D scenes have made substantial advancements recently, yet object removal remains a challenging yet practical issue, due to the absence of multi-view supervision over occluded areas.  Diffusion Models (DMs), trained on extensive 2D images, show diverse and high-fidelity generative capabilities in the 2D domain. However, due to not being specifically trained on 3D data, their application to multi-view data often exacerbates inconsistency, hence impacting the overall quality of the 3D output. To address these issues, we introduce "In-N-Out", a novel approach that begins by inpainting a prior, i.e., the occluded area from a single view using DMs, followed by outstretching it to create multi-view inpaintings via latents alignments. Our analysis identifies that the variability in DMs' outputs mainly arises from initially sampled latents and intermediate latents predicted in the denoising process. We explicitly align of initial latents using a Neural Radiance Field (NeRF) to establish a consistent foundational structure in the inpainted area,  complemented by an implicit alignment of intermediate latents through cross-view attention during the denoising phases, enhancing appearance consistency across views. To further enhance rendering results, we apply a patch-based hybrid loss to optimize NeRF. We demonstrate that our techniques effectively mitigate the challenges posed by inconsistencies in DMs and substantially improve the fidelity and coherence of inpainted 3D representations.

Main paper: 
 introduction: 
 Neural Radiance Fields (NeRFs) [50, 2, 23, 58, 81, 37, 13, 10, 3, 92] have effectively revolutionized 3D scene reconstruction from multi-view images. These models offer high-fidelity novel-view synthesis, proving beneficial across a variety of domains [32, 87, 88, 43, 107, 6, 63, 72, 8, 60] . Despite the impressive ability to reconstruct highly detailed scenes, these learning-based methods depend on the availability of consistent multi-view training data. This reliance limits their generalizability, particularly in editing 3D representations for tasks like object removal and inpainting occluded areas.Recently, diffusion models (DMs) [30, 18, 71, 80] have gained significant attention in the field of generative modelling for 2D images. These models are well-known for their robustness as generative priors, capable of producing diverse and high-fidelity results in 2D inpainting tasks. However, adapting these 2D priors for 3D object removal is not straightforward. While the inherent diversity of DMs benefits the generation of varied outputs, it also poses a significant challenge: high variance in the inpainted results (Fig. 1 middle column). Consequently, these models frequently produce outputs that, while visually appealing in isolation, may appear misaligned when incorporated into 3D domain [53, 94, 93, 25, 19, 97, 84] . This misalignment often results in the loss of high-frequency details, crucial for realistic and coherent scene rendering. Previous studies addressing such 3D inconsistencies can be broadly categorized into two approaches: multi-view and single-view priors. The former tackled inconsistencies across multi-view inpainted images by optimizing NeRFs with modified objectives [53, 94, 93, 65, 91] . While these methods have shown promise in refining inconsistent inputs, they sometimes suffer from a loss of detail fidelity during the training process, as illustrated in Fig. 5 second column. Conversely, other studies have attempted to overcome the multi-view inconsistency bottleneck by anchoring the inpainting process to a single reference image that serves the entire scene [42, 51, 109] . This approach, however, places significant reliance on the selection of an appropriate reference image and the accuracy of depth estimates, which could lead to geometric artifacts during testing, as shown in Fig. 5 third column.To address these challenges, we aim to overcome 3D inconsistencies by guiding 2D DMs to achieve multi-view consistent inpainting results (Fig. 1 right column). Our analysis reveals that the variance in model outputs primarily comes from the random noise as the initial latent sample, and intermediate latents inferred by the denoising network. Each frame's initial latents are independently sampled, while intermediate latents are individually predicted, highlighting how view-dependent data impacts the generation process. Therefore, our approach focuses on aligning these two critical elements across multiple inputs. We introduce "In-N-Out", a conditional-sampling-like approach that inpaints a sampled view and outstretches it to multiple views. Our method contains three key components:1. Conditional Inpainting Pipeline: We propose a pipeline that first samples an inpainting outcome from a random view as an inpainting prior. This prior then serves as a condition to guide the inpainting process of multiple views, ensuring a consistent inpainting foundation.2. Explicit Latents Alignment: Leveraging the geometry derived from a pre-trained NeRF and the inpainting prior, we sample multi-view initial latents conditional on the geometry dictated by the inpainting prior. This ensures that the primary components within the inpainted areas are structurally consistent and align with the underlying 3D geometry.3. Implicit Latents Alignment: We employ a cross-view attention mechanism during the denoising steps to align predicted intermediate latents concerning the inpainting prior. This enhances the appearance consistency across the inpainted images.To further enhance our method's performance in the 3D domain, we have implemented a patch-based optimization strategy using a hybrid loss on our inpainted multi-view images. This strategy employs perceptual loss to rectify spatial mismatches, and adversarial loss to preserve high-frequency details.By addressing these key challenges, our framework effectively handles multi-view inconsistencies and enhances the fidelity and coherence of 3D representations. The effectiveness of our approach is demonstrated through both qualitative and quantitative evaluations of a challenging object removal dataset. Our results indicate comprehensive improvement compared to existing methods, highlighting our model's ability to achieve greater fidelity and consistency in inpainted scenes.2 Related Works 2D Editing with Diffusion Models Diffusion models [30, 33, 105, 59, 79, 80] , have revolutionized image generation with their capacity to create highly realistic images. These models facilitate customizable generation via textual prompts [18, 29, 69, 74] , predominantly using pre-trained Stable Diffusion [71] . Several editing methods [22, 57, 56, 61] allow users to adjust images by moving anchor points to new locations. Editing typically begins by inverting the latent representation of the image to be edited back to its initial noise [80] , with modifications made during the denoising phase. Prompt-to-Prompt (P2P) [26] edits images by adjusting the cross-attention between the image and text. Null-text inversion [55] addresses artifacts in DDIM inversion [80] when using classifier-free guidance [29] . Delta Denoising Score (DDS) [27] optimizes the latent image representation by aligning the predicted noises of the original and modified texts. Additionally, several studies [7, 9, 26] have identified a relationship between the appearance of images generated by diffusion models and the key-value pairs. While these advancements represent significant progress preserve some content from the original image in 2D image editing, they do not account for multi-view consistency, thus can not be lifted to 3D editing directly.Lifting 2D diffusion models for 3D editing Recent advancements in 3D editing and generation have effectively utilized 2D DMs to enhance these processes, as demonstrated in various studies [52, 76, 96, 62, 97, 99, 41, 95, 68, 49, 103] . Pioneering works have used images inferred by DMs for direct supervision. Instruct-NeRF2NeRF (IN2N) [25] approached the editing task by transforming 3D model editing into a 2D image editing task, utilizing Instruct Pix2Pix (IP2P) [5] to iteratively update 3D scenes. Similarly, ViCA-NeRF [19] addressed editing challenges by modifying reference images and integrating these changes into the scene. DreamEditor [110] opted for a different strategy by converting NeRF into a mesh for direct optimization. GaussianEditor [12] applies semantic tracing to identify and modify editing targets within 3D Gaussian Splatting (3DGS) [37] . Similarly, Gaussian Grouping [100] implements Identity Encoding for each Gaussian to create masks for editing. Conversely, Score Distillation Sampling (SDS) [64] provides an alternative way to guide 3D representations by backpropagating gradients from a diffusion model's denoiser [1, 16, 73, 71] into the underlying scene representation. This technique has been effectively applied to generate realistic 3D and 4D scenes using NeRFs [11, 40, 39, 108, 110] and 3DGS [70, 101, 84, 15] .2D and 3D Inpainting 2D inpainting methods reconstruct images by filling missing content in areas defined by a mask [20, 77, 104, 86, 46, 75, 102] . Early techniques, exemplified by [21] , relied on copying textures from known to unknown regions. LaMa [82] excels in restoring large missing areas using fast Fourier convolutions, extensive receptive fields, and large training masks. Although highly effective at generating plausible background textures within specified masks, LaMa limits fidelity of its outputs. In contrast, probabilistic diffusion models [30] have shown impressive results in image generation and offer a wide range of inpainted outputs. DMs can be adapted for inpainting without specific training, and modify known regions during each denoising step to fit the task [47] .Similarly, Stable Diffusion [71] excels at inpainting by operating within latent space, allowing for efficient and effective image generation. In this work, we adopt it as our 2D inpainter.3D scene inpainting aims to fill missing areas within a space, such as removing objects and generating coherent geometry and textures to complete the scene. Although 3D generative models have garnered large interest [4, 34, 38, 89, 78, 44, 31, 85, 45, 14, 12] , they are often limited by the scarcity of 3D training data, hence result in poor generalization, particularly in scene inpainting tasks. Therefore, most current 3D inpainting models [53, 51, 42, 65, 93, 94, 109, 91] enhance their effectiveness by adopting priors from 2D models. SPIn-NeRF [53] reduces multi-view inconsistencies by first inpainting views and then optimizing NeRF using perceptual loss. NeRFiller [93] tackles multiple frames simultaneously by tiling images for DMs. GaussianEditor [12] edit targets within 3DGS [37] , guided by inpainted multi-view images from DMs. While these methods show promise, they can sometimes compromise detail fidelity during training. Alternatively, some studies circumvent multi-view inconsistencies by using a single reference image for the entire scene [42, 51, 109] . Infusion [109] stands out in the inpainting of 3DGS, leveraging a pre-trained depth completion network to infer point clouds from a single inpainted view, though this method depends heavily on precise depth estimates. Concurrent works [65, 54] address these challenges using SDS objective [64] to better align 2D model priors with 3D scene consistency.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: In the abstract and introduction, we clearly state that our method contributes to the field of object removal in neural radiance fields, offering a novel and effective solution to improve multi-view consistency. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
limitations: 
 Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: In the conclusion (Sec. 6) , we discuss the limitations of our approach, outlining the current bottlenecks. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
neural radiance fields: 
 Neural Radiance Fields (NeRFs) [50] represents a breakthrough in 3D rendering by employing a multilayer perceptron (MLP), denoted as ϕ to represent a scene. This MLP serves as a continuous volumetric function to capture and reconstruct a scene in unprecedented detail. Specifically, NeRFs take as input the view direction d and a 3D coordinate r(τ ) sampled from a camera ray defined by r(τ ) = o + τ d. At each position along this ray r(τ ), the network predicts the volume density and view-dependent color, represented as (σ, c). To render a camera pixel, NeRFs perform an aggregation of the predicted densities and color emissions σ(τ i ), c(τ i ) along the camera ray. This process is mathematically formulated as an approximation of a volume rendering integral [48] , which is used to compute the final color of the pixel:EQUATIONwhere δ(τ i ) = τ i+1τ i is the distance between adjacent samples along the ray. During the training phase, rays are uniformly sampled from the training images, and the volumetric field is optimized using mean square error (MSE) to enhance the accuracy and realism of the rendered scenes.Diffusion models [30] consist of two processes: a forward process that gradually introduces noise to a data sample z 0 ∼ p data (z), and a learned reverse process that iteratively denoises a purely Gaussian noise sample z T ∼ N (0, 1) back into a clean image z 0 . The reverse process is parameterized by a conditional noise prediction network ϵ θ , trained to predict the noise using the simplified objective:EQUATIONwhere t is the time step in the diffusion process, z t is an intermediate noisy sample, and c represents a condition (e.g., images, masks, or text). Utilizing a deterministic sampler like DDIM [80] , the sample z t-1 can be obtained by z t-1 = z tϵ θ (z t , t, c); note that scaling is omitted for simplicity.In practice, as we use Stable Diffusion [71] , a latent diffusion as the inpainting backbone, z is latent and the generated image is obtained with a decoder Ω(z 0 ). Hence, the variability of the generated image z 0 depends solely on initial sampled latent z T and intermediate inferred latents z t-1 T t=1 .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Justification: Due to the extensive work required for this task, we will need to clean up the code before its release. However, to ensure reproducibility, we provide detailed instructions in Sec. 4 Justification: We provided details on the computational hardware used in experiment (Sec. 5.1 and Sec. A.1).Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
method: 
 Given a set of multi-view training images {I i } N i=1 from the scene with corresponding masks{M i } N i=1indicate the unwanted object in each frame, our approach seeks to generate consistently inpainted training set { Ĩi } N i=1 and use them to supervise NeRF. Our approach is structured into three key stages: An overview of our method is shown in Fig. 2 .• Stage 1: Pretrain a NeRF ϕ using {I i } N i=1 and {M i } N i=1 ,The initial stage involves training the NeRF on the unmasked region, we follow the original work [50] where simple MSE loss is applied: where R unmasked represent the unmasked pixels across all the training images. Then we sample a prior view I p with its mask M p and regularly inpaint it using Stable Diffusion θ. For illustration, we replace of condition in Eq. 2 with two components used by Stable Inpainting Diffusion [71] as e for the input prompt, and I ′ p for the masked image that fed into the diffusion models. Hence the inpainting process can be formulated as:EQUATIONF v R l K E n L U E z S h G B k r R e 5 + w J H p Y 8 T y 2 1 E k H w K p K C f l y K 1 4 V W 8 M O E / 8 K a m A K e q R + x V 0 U 5 x x I g x m S O u O 7 0 k T 5 k g Z i h k Z l Y N M E 4 n w A P V I x 1 K B O N F h P o 4 / g k d W 6 c I k V f Y J A 8 f q 7 4 0 c c a 2 H P L a T R V g 9 6 x X i f 1 4 n M 8 l l m F M h M 0 M E n h x K M g Z N C o s u Y J c q g g 0 b W o K w o j Y r x H 2 k E D a 2 s a I E f / b L 8 6 R 5 U v X P q 2 d 3 p 5 X a 1 b S O E j g A h + A Y + O A C 1 M A N q I M G w C A H z + A V vV D L S s N A F J 3 U V 6 2 v q O D G z W A R X J V E f C 2 L b n R X o S 9 o Q p h M J u 3 Q y S T M T I Q S s / B X 3 L h Q x K 2 / 4 c 6 / c d J m o a 0 H B g 7 n 3 M s 9 c / y E U a k s 6 9 u o L C 2 v r K 5 V 1 2 s b m 1 v b O + b u X l f G q c C k g 2 M W i 7 6 P J G G U k 4 6 i i p F + I g i K f E Z 6 / v i m 8 H s P R E g a 8 7 a a J M S N 0 J D T k G K k t O S Z B 0 6 b s o B k T o T U C C O W 3 e W 5 l 9 Q 8 s 2 4 1 r C n g I r F L U g c l W p 7 5 5 Q Q x T i P C F W Z I y o F t J c r N k F A U M 5 L X n F S S B O E x G p K B p h x F R L r Z N H 8 O j 7 U S w D A W + n E F p + r v j Q x F U k 4 i X 0 8 W M e W 8 V 4 j / e Y N U h V d u R n m S K s L x 7 F C Y M q h i W J Q B A y o I V m y i C c K C 6 q w Q j 5 B A W O n K i h L s + S 8 v k u 5 p w 7 5 o n N + f 1 Z v X Z R 1 V c A i O w A m w w S V o g l v Q A h 2 A w S N 4 B q / g z X g y X o x 3 4 2 M 2 W j H K n X 3 w B 8 b n D x U G l i c = < / l a t e x i t > Ĩp < l a t e x i t s h a 1 _ b a s e 6 4 = " 6 e 1 Q Q j F p p L G G 8 G a q R P u J f B h / K m Q = " > A A A B / 3 i c b V D L S s N A F J 3 U V 6 2 v q O D G z W A R X J V E f C 2 L b n R X o S 9 o Q p h M J u 3 Q y S T M T I Q S s / B X 3 L h Q x K 2 / 4 c 6 / c d J m o a 0 H B g 7 n 3 M s 9 c / y E U a k s 6 9 u o L C 2 v r K 5 V 1 2 s b m 1 v b O + b u X l f G q c C k g 2 M W i 7 6 P J G G U k 4 6 i i p F + I g i K f E Z 6 / v i m 8 H s P R E g a 8 7 a a J M S N 0 J D T k G K k t O S Z B 0 6 b s o B k T o T U C C O W 3 e W 5 R 2 u e W b c a 1 h R w k d g l q Y M S L c / 8 c o I Y p x H h C j M k 5 c C 2 E u V m S C i K G c l r T i p J g v A Y D c l A U 4 4 i I t 1 s m j + H x 1 o J Y B g L / b i C U / X 3 R o Y i K S e R r y e L m H L e K 8 T / v E G q w i s 3 o z x J F e F 4 d i h M G V Q x L M q A A R U E K z b R B G F B d V a I R 0 g g r H R l R Q n 2 / J c X S f e 0 Y VN m v B E t k O i g L M Y G p p p D u 1 U A h E h h 1 Y 4 u C 7 8 1 i N I x Z L 4 X g 9 T 8 A X p x S x i l G g j B f a + J 4 j u U 8 L z 2 1 H A H r x U M g H l w K 4 4 V W c M P E / c K a m g K e q B / e V 1 E 5 o J i D X l R K m O 6 6 T a z 4 n U j H I Y l b 1 M Q U r o g P S g Y 2 h M B C g / H 8 c f 4 S O j d H G U S P N i j c f q 7 4 2 c C K W G I j S T R V g 1 6 x X i f 1 4 n 0 9 G l n 7 M 4 z T T E d H I o y j j W C S 6 6 w F 0 m g W o + N I R Q y U x W T P t E E q p N Y 0 U J 7 u y X 5 0 n z p O q e V 8 / u T i u 1 q 2 k d J X S A D t E x c t E F q q E b V E c N R F G O n t EE g o S R J d S V v B e u Q = " > A A A B 7 X i c b V D L T s M w E N z w L O V V 4 M j F o k L i V C W I 1 7 G C C 8 c i 9 S W 1 o X J c p z V 1 7 M h 2 k E r U f + D C A Y S 4 8 j / c + B u c N g d o G W m l 0 c y u d n e C m D N t X P f b W V p e W V 1 b L 2 w U N 7 e 2 d 3 Z L e / t N L R N F a I N I L l U 7 w J p y J m j D M M N p O 1 Y U R w G n r W B 0 k / m t R 6 o 0 k 6 J u x j H 1 I z w Q L G Q E G y s 1 n 3 r x f b 3 Y K 5 X d i j s F W i R e T s q Q o 9 Y r f X X 7 k i Q R F Y Z w r H X H c 2 P j p 1 g Z R j i d F L u J p j E m I z y g H U s F j q j 2 0 + m 1 E 3 R s l T 4 K p b I l D J q q v y d S H G k 9 j g L b G W E z 1 P N e J v 7 n d R I T X v k p E 3 F i q C C z R W H C k Z E o e x 3 1 m a L E 8 L E l m C h m b 0 V k i B U m x g a U h e D N v 7 x I m q c V 7 6 J y f n d W r l 7 n c R T g E I 7 g B D y 4 h C r c Q g 0 a Q O A B n u E V 3 h z p v D j v z s e s d c n J Z w 7 g D 5 z P H w I v j s U = < / l a t e x i t > z T p < l a t e x i t s h a 1 _ b a s e 6 4 = " O d 1 I B R 1 m z H c y 4 g x R w f 7 G K r V J w g 8 = " > A A A B 7 X i c b V D L T s M w E N z w L O V V 4 M j F o k L i V C W I 1 7 G C C 8 c i 9 S W 1 o X J c p z V 1 7 M h 2 k E r U f + D C A Y S 4 8 j / c + B u c N g d o G W m l 0 c y u d n e C m D N t X P f b W V p e W V 1 b L 2 w U N 7 e 2 d 3 Z L e / t N L R N F a I N I L l U 7 w J p y J m j D M M N p O 1 Y U R w G n r W B 0 k / m t R 6 o 0 k 6 J u x j H 1 I z w Q L G Q E G y s 1 n 3 r s v l 7 s l c p u x Z 0 C L R I v J 2 X I U e u V v r p 9 S Z K I C k M 4 1 r r j u b H x U 6 w M I 5 x O i t 1 E 0 x i T E R 7 Q j q U C R 1 T 7 6 f T a C T q 2 S h + F U t k S B k 3 V 3 x M p j r Q e R 4 H t j L A Z 6 n k v E / / z O o k J r / y U i T g x V J D Z o j D h y E i U v Y 7 6 T F F i + N g S T B S z t y I y x A o T Y w P K Q v D m X 1 4 k z d O K d 1 E 5 v z s r V 6 / z O A p w C E d w A h 5 c Q h V u o Q Y N I P A A z / A K b 4 5 0 X p x 3 5 2 P W u u T k M w f w B 8 7 n D / d v j r 4 = < / l a t e x i t > z T i < l a t e x i t s h a 1 _ b a s e 6 4 = " r p q x L q o U 0 + c I Q w y A 2 n + D p / U 0 N s A = " > A A A C A X i c b V D L S s N A F J 3 U V 6 2 v q B v B T b A I r k o i v p Z F N y 5 c V L A P a E K Y T K f t 0 M k k z N y I J c S N v + L G h S J u / Q t 3 / o 2 T N g t t P T B w O O d e 5 p 4 T x J w p s O 1 v o 7 S w u L S 8 U l 6 t r K 1 v b G 6 Z 2 z s t F S W S 0 C a J e C Q 7 A V a U M 0 G b w I D T T i w p D g N O 2 8 H o K v f b 9 1 Q q F o k 7 G M f U C / F A s D 4 j G L T k m 3 t u i G F I M E 9 v M j 9 1 g T 5 A K i n J M t + s 2 j V 7 A m u e O A W p o g I N 3 / x y e x F J Q i q A c K x U 1 7 F j 8 F I s g R F O s 4 q b K B p j M s I D 2 t V U 4 J A q L 5 0 k y K x D r f S s f i T 1 E 2 B N 1 N 8 b K Q 6 V G o e B n s z v V b N e L v 7 n d R P o X 3 g p E 3 E C V J D p R / 2 E W x B Z e R 1 W j + m 0 w M e a Y C K Z v t U i Q y w x A V 1 a R Z f g z E a e J 6 3 j m n N W O 7 0 9 q d Y v i z r K a B 8 d o C P k o H N U R 9 e o g Z q I o E f 0 j F 7 R m / F k v B j v x s d 0 t G Q U O 7 v o D 4 z P H 8 / 2 l 8 k = < / l a t e x i t > L rec < l a t e x i t s h a 1 _ b a s e 6 4 = " W 4 G q c P O 2 G B u D 3 s 9 t Q Q 9 u M A A 7 u A Y = " > A A A C A 3 i c b V D L S s N A F J 3 U V 6 2 v q D v d B I v g q i T i a 1 l 0 4 8 J F B f u A J o T J d N o O n W T C z I 1 Y Q s C N v + L G h S J u / Q l 3 / o 2 T N g t t P X D h c M 6 9 3 H t P E H O m w L a / j d L C 4 t L y S n m 1 s r a + s b l l b u + 0 l E g k o U 0 i u J C d A C v K W U S b w I D T T i w p D g N O 2 8 H o K v f b 9 1 Q q J q I 7 G M f U C / E g Y n 1 G M G j J N / f c E M O Q Y J 7 e Z H 7 q A n 2 A N J Z M y C zV L j M N C d + c V q 1 s v F / 7 x u A v 0 L L 2 V R n A C N y H R R P + E W C C s P x O o x S Q n w s S a Y S K Z v t c g Q S 0 x A x 1 b R I T i z L 8 + T 1 n H N O a u d 3 p 5 U 6 5 d F H G W 0 j w 7 Q E X L Q O a q j a 9 R A T U T Q I 3 p G r + j N e D J e j H f j Y 9 p a M o q Z X f Q H x u c P n h K Y z w = = < / l a t e x i t > L prior < l a t e x i t s h a 1 _ b a s e 6 4 = " T q 9 r m L T l Q Q 6 T 8 0 G o W h I 4 3 a l n i 1 o = " > A A A C A 3 i c b V D L S s N A F J 3 U V 6 2 v q D v d B I v g q i T i a 1 l 0 4 8 J F B f u A J o T J d N o O n U z C z I 1 Y Q s C N v + L G h S J u / Q l 3 / o 2 T N g t t P X D h c M 6 9 3 H t P E H O m w L a / j d L C 4 t L y S n m 1 s r a + s b l l b u + 0 V J R I Q p s k 4 p H s B F h R z g R t A g N O O 7 G k O A w 4 b Q e j q 9 x v 3 1 O p W C T u Y B x T L 8 Q D w f q M Y N C S b + 6 5 I Y Y h w T y 9 y f z U B f o A a Y y B D L P M N 6 t 2 z Z 7 A m i d O Q a q o Q M M 3 v 9 x e R J K Q C i A c K 9 V 1 7 B i 8 F E t g h N O s 4 i a K x p i M 8 I B 2 N R U 4 p M p L J z 9 k 1 q F W e l Y / k r o E W B P 1 9 0 S K Q 6 X G Y a A 7 8 4 v V r J e L / 3 n d B P o X X s p E n A A V Z L q o n 3 A L I i s P x O o x S Q n w s S a Y S K Z v t c g Q S 0 x A x 1 b R I T i z L 8 + T 1 n H N O a u d 3 p 5 U 6 5 d F H G W 0 j w 7 Q E X L Q O a q j a 9 R A T U T Q I 3 p G r + j N e D J e j H f j Y 9 p a M o q Z X f Q H x u c P c 0 G Y s w = = < / l a t e x i t > L patch < l a tA A A C E X i c b Z D L S s N A F I Y n 9 V b r L S q u 3 A w W w V V J x N u y 6 M Z l B X u B J p S T 6 a Q d O p m E m Y l Q Q p 7 C J 3 C r T + B O 3 P o E P o D v 4 b T N w r b + c O D n P + d w D l + Q c K a 0 4 3 x b p Z X V t f W N 8 m Z l a 3 t n d 8 / e P 2 i p O J W E N k n M Y 9 k J Q F H O B G 1 q p j n t J J J C F H D a D k Z 3 k 3 7 7 i U r F Y v G o x w n 1 I x g I F j I C 2 k Q 9 + 8 g j I A j l m S c g 4 J B 7 i g 0 i 6 N l V p + Z M h Z e N W 5 g q K t T o 2 T 9 e P y Z p R I U m H J T q u k 6 i / Q y k Z o T T v O K l i i Z A R j C g X W M F R F T 5 2 f T 9 H J + a p I / D W J o S G k / T v x s Z R E q N o 8 B M R q C H a r E 3 C f / r d V M d 3 v g Z E 0 m q q S C z Q 2 H K s Y 7 x h A X u M 0 m J 5 m N j g E h m f s V k C B K I N s T m r s w g 5 Y a L u 0 h h 2 b T O a + 5 V 7 f L h o l q / L Q i V 0 T E 6 Q W f I R d e o j u 5 R A z U R Q R l 6 Q a / o z X q 2 3 q 0 P 6 3 M 2 W r K K n U M 0 J + vEQUATIONThe inpainted image then can be obtained by Ĩp = Ω(z 0 p ). We then use a monocular depth estimator on Ĩp to get a depth map Dp . We regress the scale and offset parameters to align Dp with the depth estimated from field ϕ on the unmasked pixels. Hence, we can introduce the geometry and appearance supervision of the inpainting prior Ĩp into the NeRF's optimization through:EQUATIONwhere R masked(p) denoted the masked (inpainted) pixels of Ĩp , and Dϕ is the depth estimated by NeRF. This stage is depicted in Fig. 2 (a).In this section, we introduce our key approach to condition the additional inpainted frames to have an inpainting feature based on the prior Ĩp . As discussed before, in deterministic sampling of the diffusion inpainting model θ, the generation structure and layout highly depend on (1) initial sampled noise z T i and (2) intermediate predicted latents {z t-1 i } T t=1 . Hence if we can align the latents from different views with the prior one, the model is likely to generate multi-view consistent latent z 0 i , hence image Ĩi . In this section, we discuss how to align two terms respectively. Explicit Initial Latent Alignment (ELA) Given the sampled latent of prior view z T p used in Stage 1 (Sec. 4.1), we could leverage geometric information to explicitly align the initial latent in 3D space. Given that estimated depth Dp , one possible solution is to warp the z T p into other views using camera matrices. However, Dp is not guaranteed to be accurate hence such hard projection could yield significant errors. Alternatively, we propose to leverage the pre-trained NeRF ϕ, as it's a naturally 3D-consistent representation. Specifically, to sample a resolution-grain initial latent z T (r), we utilize the original formulation of volume rendering (Eq.1) but with the substitution of color c by latent z T . We query the density σ from ϕ, and acquire z T by reprojecting the sampled point to the image plane of the prior latent view z T p :l i x k B J t c G i Q R G 1 Z r b t 2 d A 6 0 S r y A 1 K N A a V r 8 G o 5 i k g k p D O N a 6 7 7 m J 8 T O s D C O c z i q D V N M E k w k e 0 7 6 l E g u q / W x + 6 w y d W W W E w l j Z k g b N 1 d 8 T G R Z a T 0 V g O w U 2 k V 7 2 c v E / r 5 + a 8 M b P m E x S Q y V Z L A p T j k y M 8 s f R i C l K D J 9 a g o l i 9 l Z E I q w w M T a e i g 3 B W 3 5 5 l X Q u 6 t 5 V v f F w W W v e F n G U 4 Q R O 4 R= " > A A A B 7 X i c b V D J S g N B E K 2 J W 4 x b 1 K O X x i B 4 C j P i d g x 6 8 R j B L J A M o a f T k 7 T p Z e j u E c K Q f / D i Q R G v / o 8 3 / 8 Z O M g d N f F D w e K + K q n p R w p m x v v / t F V Z W 1 9 Y 3 i p u l r e 2 d 3 b 3 y / k H T q F Q T 2 i C K K 9 2 O s K G c S d q w z H L a T j T F I u K 0 F Y 1 u p 3 7 r i W r D l H y w 4 4 S G A g 8 k i x n B 1 k n N r m E D g X v l i l / 1 Z 0 D L J M h J B X L U e + W v b l + R V F B p C c f G d A I / s W G G t W W E 0 0 m p m x q a Y D L C A 9 p x V G J B T Z j N r p 2 g E 6 f 0 U a y 0 K 2 n R T P 0 9 k W F h z F h E r l N g O z S L 3 l T 8 z + u k N r 4 O M y a T 1 F J J 5 o v i l C O r 0 P R 1 1 G e a E s v H j m C i m b s V k S H W m F g X U M m F E C y + v E y a Z 9 X g s n p x f 1 6 p 3 e R x F O E I j u E U A r i C G t x B H R p A 4 B G e 4 Rq + 3 H r G u l N Q L P 0 8 T H H p N x + i M = " > A A A B 6 n i c b V D J S g N B E K 2 J W 4 x b 1 K O X x i B 4 C j P i d g x 6 8 R g x G y R j 6 O n U J E 1 6 e o b u H i E O + Q Q v H h T x 6 h d 5 8 2 / s L A e N P i h 4 v F d F V b 0 g E V w b 1 / 1 y c k v L K 6 t r + f X C x u b W 9 k 5 x d 6 + h 4 1 Q x r L N Y x K o V U I 2 C S 6 w b b g S 2 E o U 0 C g Q 2 g + H 1 x G 8 + o N I 8 l j U z S t C P a F / y k D N q r H T 3 e F / r F k t u 2 Z 2 C / C X e n J R g j m q 3 + N n p x S y N U B o m q N Z t z 0 2 M n 1 F l O B M 4 L n R S j Q l l Q 9 r H t q W S R q j 9 b H r q m B x Z p U f C W N m S h k z V n x M Z j b Q e R Y H t j K g Z 6 E V v I v 7 n t V M T X v o Z l 0 l q U L L Z o j A V x M R k 8 j f p c Y X M i J E l l C l u b y V s Q B V l x q Z T s C F 4 i y / / J Y 2 T s n d e P r s 9 L V W u 5 n H k 4 Q A O 4 R g 8 u I A K 3 E A Vj V H F o 8 l r H u B M y A F A p a K F B C J 9 H A o k B C O x j f 5 n 7 7 C b Q R s X r A S Q J + x I Z K h I I z t F K 7 h y N A V u l X a 2 7 d n Y E u E 6 8 g N V K g 2 a 9 + 9 Q Y x T y N Q y C U z p u u 5 C f o Z 0 y i 4 h G m l l x p I G B + z I X Q t V S w C 4 2 e z c 6 f 0 x C o D G s b a l k I 6 U 3 9 P Z C w y Z h I F t j N i O D K L X i 7 + 5 3 V T D K / 9 T K g k R V B 8 v i h M J c W Y 5F v R l K E n L U E z S h G B k r R e 5 + w J H p Y 8 T y 2 1 E k H w K p K C f l y K 1 4 V W 8 M O E / 8 K a m A K e q R + x V 0 U 5 x x I g x m S O u O 7 0 k T 5 k g Z i h k Z l Y N M E 4 n w A P V I x 1 K B O N F h P o 4 / g k d W 6 c I k V f Y J A 8 f q 7 4 0 c c a 2 H P L a T R V g 9 6 x X i f 1 4 n M 8 l l m F M h M 0 M E n h x K M g Z N C o s u Y J c q g g 0 b W o K w o j Y r x H 2 k E D a 2 s a I E f / b L 8 6 R 5 U v X P q 2 d 3 p 5 X a 1 b S O E j g A h + A Y + O A C 1 M A N q I M G w C A H z + A V vS R J d S V v B e u Q = " > A A A B 7 X i c b V D L T s M w E N z w L O V V 4 M j F o k L i V C W I 1 7 G C C 8 c i 9 S W 1 o X J c p z V 1 7 M h 2 k E r U f + D C A Y S 4 8 j / c + B u c N g d o G W m l 0 c y u d n e C m D N t X P f b W V p e W V 1 b L 2 w U N 7 e 2 d 3 Z L e / t N L R N F a I N I L l U 7 w J p y J m j D M M N p O 1 Y U R w G n r W B 0 k / m t R 6 o 0 k 6 J u x j H 1 I z w Q L G Q E G y s 1 n 3 r x f b 3 Y K 5 X d i j s F W i R e T s q Q o 9 Y r f X X 7 k i Q R F Y Z w r H X H c 2 P j p 1 g Z R j i d F L u J p j E m I z y g H U s F j q j 2 0 + m 1 E 3 R s l T 4 K p b I l D J q q v y d S H G k 9 j g L b G W E z 1 P N e J v 7 n d R I T X v k p E 3 F i q C C z R W H C k Z E o e x 3 1 m a L E 8 L E l m C h m b 0 V k i B Ux i T E R 7 Q j q U C R 1 T 7 6 f T a C T q 2 S h + F U t k S B k 3 V 3 x M p j r Q e R 4 H t j L A Z 6 n k v E / / z O o k J r / y U i T g x V J D Z o j D h y E i U v Y 7 6 T F F i + N g S T B S z t y I y x A o T Y w P K Q v D m X 1 4 k z d O K d 1 E 5 v z s r V 6 / z O A p w C E d w A h 5 c Q h V u o Q Y N I P A A z / AQ c K u + D o G v Q h e I p g H J M s y O 5 l N h s z O j j O z Q l j y E 1 4 8 K O L V 3 / H m 3 z h J 9 q C J B Q 1 F V T f d X a H k T B v X / X Y K S 8 s r q 2 v F 9 d L G 5 t b 2 T n l 3 r 6 m T V B H a I A l P V D v E m n I m a M M w w 2 l b K o r j k N N W O L y Z + K 0 n q j R L x I M Z S e r H u C 9 Y x A g 2 V m r f B f I E N Q M Z l C t u 1 Z 0 C L R I v J x X I U Q / K X 9 1 e Q t K Y C k M 4 1 r r j u d L 4 G V a G E U 7 H p W 6 q q c R k i P u 0 Y 6 n A M d V + N r 1 3 j I 6 s 0 k N R o m w J g 6 b q 7 4 k M x 1 q P 4 t B 2 x t g M 9 L w 3 E f / z O q m J r v y M C Z k a K s h s U Z R y Z B I 0 e R 7 1 m K L E 8 J E l m C h m b 0 V k g B U m x k Z U s i F 4 8 y 8 v k u Z p 1 b u o n t + f V W r X e R x F O I B D O A Y P L q E G t 1 C H B hEQUATIONwhere f p,i denote camera perspective projection according to p and i camera matrices. Such soft projection could avoid error accumulation in the inpainting process, and reduce the precision burden on the depth estimator. We illustrate this process in Fig. 3a . There are two key reasons why we propose fine-tuning the NeRF and using it as a geometric prior for ELA: (a) After finetuning the NeRF, the geometric is represented by NeRF as a sharp (low variance) unimodal distribution on the ray. Consequently, the aggregated feature remains sharp, preserving the variations in the initial latents. (b) We empirically found the depth prior inferred by the monocular depth estimator is not perfectly aligned with the NeRF. Fine-tuning the NeRF can also benefit this depth prior. Since NeRF learns relatively certain geometry in the known (unmasked) areas, this geometry constraint can improve the geometry of neighboring inpainted (masked) areas due to their geometric proximity. We compromise the view-dependent effect in NeRF within the ELA module. Due to the heuristic nature of diffusion models, incorporating such view-dependent effects into diffusion models' output remains elusive.Implicit Intermediate Latents Alignment (ILA) While the initial latent could be aligned using the explicit method, intermediate latents are predicted by denoising network ϵ θ which is hard to control. We address this issue by exploring the conditioning mechanism of the denoising network in Stable Diffusion [71] . Recall that in Eq. 4, denoising network ϵ θ relies on the input prompt e and masked image I ′ i to predict the noise occurrence in the current step. While we can use the unified prompt for all views to align the text condition in cross-attention of ϵ θ , the masked images{I ′ i } N i=1are inherently different due to multi-view nature. Note that I ′ i condition is introduced based on spatial self-attention (SA) [90] in the U-Net:EQUATIONwhere Q i obtained from each spatial resolution of the latent, K i , V i are derived from corresponding latent encoded from masked image I ′ i . We can impose the coherence of the denoising step by introducing cross-view attention (CVA) of the prior view (Fig. 3b ):EQUATIONwhere K p , V p are from masked base image I ′ p . We can then implicitly align the denoising step by replacing the original SA with a weighted sum from SA and CVA, i.e. λ a * SA(Qi , K i , V i ) + (1 - λ a ) * CVA(Q i , K p , V p ).Through this technique we ensure that the intermediate latents {z t-1 i } T t=1 are also conditioned on the prior {z t-1 p } T t=1 , while retain its distinctiveness due to the individual viewpoint. The rationale of replacing KV with "prior" p, but not Q is that the appearance information (V ) of the prior image should be considered when inpainting the other views, with the amount of information propagation is weighted by its attention key value (K). The attention query value comes from the current inpainting view i, Q i , representing the information the current inpainting for view i is searching for. Together with K p , it decides how much attention the view i inpainter should place on the prior view, and finally incorporates the information of the prior view V p into view i.As the original intention of this work, we seek to distill the inpainted views into NeRF ϕ in a way such that the high-fidelity is preserved as much as possible as the unmasked region. While some priors work in 3D editing [25, 96, 97, 93] propose to update the training set iteratively until converge, we empirically find it not suitable for inpainting task since the loss of fidelity is significant and could fall into local optima. Hence we propose to inpaint a subset of training images at once and regard them as supplementary guidance using a patch-based hybrid loss:EQUATIONwhere ρ is a patch sample from the masked area of subset views P sub , Îϕ (ρ) is NeRF predicted patch, and L lpips , L adv are perceptual distance LPIPS [106] and adversarial loss [24] . Here LPIPS is utilized to address geometry mismatches, while adversarial loss is employed to preserve high-frequency details. As shown in Fig. 2 , the final optimization objective is:EQUATIONThe patches are uniformly sampled within the bounding box of the mask, with a size of 256×256. Therefore, only the inpainted area is being optimized by the patch loss. 
evaluation setting: 
 Dataset: Aligning with methodologies employed in prior works [53, 51, 109] , our experiments utilize the SPIn-NeRF dataset [53] , selected for its comprehensive ground truth availability. This dataset is specifically designed for object removal evaluations and comprises 10 scenes. Each scene includes 60 images featuring an unwanted object (training views) and 40 images from which the object has been removed (test views). For both the training and test views, human-annotated masks indicating the object region are available. We further collected 9 forward-facing scenes with manually annotated masks to evaluate the effectiveness of our method. This dataset includes 4 indoor scenes and 5 outdoor scenes. In the training set, the masked region contains the unwanted object, while the test set contains the ground truth background in the masked region.Baselines: In our study, we benchmark our method against a variety of established 3D inpainting approaches to ascertain its relative performance. These include the perceptual-based SPIn-NeRF [53] , tiling-based NeRFiller [93] (both multi-view guidance) and InFusion [109] (single-view guidance).To ensure a fair comparison, we employ the same inpainting diffusion models across all methods and maintain consistency in the number of denoising steps and used prompts. We utilized the source code provided by the authors and ran all the methods using one NVIDIA A100 (80G) GPU.Metrics: To quantitatively evaluate the effectiveness of our approach, we employ two similarity metrics: LPIPS [106] and FID [28] . Additionally, we use MUSIQ [36] , a sharpness metric that quantifies the clarity and detail retention in the edited images. Following established protocols from previous studies [53] , all metrics are calculated specifically within the bounding boxes defined by the masks, focusing the evaluation precisely on the regions most affected by the object removal task. NeRFiller Ours InFusion Traning InputsWe first present the quantitative results in Tab. 1, where our method outperforms all baselines in terms of similarity metrics and sharpness. Our approach also excels in qualitative assessments, as demonstrated in Fig. 4 and Fig. 5 . It is important to note that in simpler scenes with low variability in inpainting results, where the inconsistency issue is less pronounced (Fig. 4 first row), most methods perform adequately. NeRFiller [93] , through its use of multiple joint denoising steps, ensures consistency but often produces overly smooth outputs that lack fine details. It is noteworthy that InFusion [109] performs well when geometry estimation is accurate. However, its performance deteriorates in scenarios where depth accuracy is compromised, leading to geometry artifacts (Fig. 5 ). This underscores the critical role of multi-view supervision in addressing such challenges. By incorporating consistent multi-view supervision, our method remains effective even when depth or geometry is inaccurate, achieving robust and promising results. This explains why our method shows little difference from InFusion when geometry is accurate, but excels when the depth is inaccurate. Additionally, the exclusive reliance on perceptual loss by SPIn-NeRF [53] fails to fully address the multi-view inconsistencies introduced by inpainting diffusion models, often resulting in a blurred effect, particularly visible in Fig. 4 third and fourth rows. To further validate our findings, we conducted a user study based on the SPIn-NeRF dataset, focusing on the coherence of the background within the inpainted area, the fidelity of detail preservation in the inpainted region, and overall preference. The results of this study are summarized in Tab. 2. This evaluation clearly demonstrates superior performance across all assessed criteria.Figure 5 : Qualitative results on the self-collected dataset. NeRFiller achieves consistent inpainting results by averaging noise predictions derived from multiple iterations, tending to produce smooth results. InFusion relies on a single reference view and its depth to represent the entire scene. This reliance exhibits artifacts when the assumption of accurate depth is violated. We initially demonstrate the efficacy of our latents alignment approach with an example in Fig. 6 . The first column displays the inpainting prior (sampled view), and the subsequent columns show the same training image being inpainted under different conditions. Notably, the variant without ELA (w/o ELA) retains colors similar to the prior but fails to preserve the texture structure. Conversely, the version without ILA (w/o ILA) maintains structural integrity but lacks appearance consistency with the prior. Our method effectively merges the strengths of both mechanisms, resulting in inpaintings that are highly consistent and cohesive across all evaluated aspects. This observation confirms our initial findings: the initial latents primarily influence the inpainting's structural pattern, whereas the intermediate denoising steps largely affect its appearance, including color nuances. Additionally, our patch-based loss plays a crucial role in the optimization process (w/o L patch ). Specifically, the L lpips loss helps to alleviate geometry mismatches (w/o L lpips ), and the L adv serves as a detail-preserving supervisor (w/o L adv ). These results highlight the effectiveness of our design choices in enhancing the overall quality and coherence of the inpainted outputs. T q 9 r m L T l Q Q 6 T 8 0 G o W h I 4 3 a l n i 1 o = " > A A A C A 3 i c b V D L S s N A F J 3 U V 6 2 v q D v d B I v g q i T i a 1 l 0 4 8 J F B f u A J o T J d N o O n U z C z I 1 Y Q s C N v + L G h S J u / Q l 3 / o 2 T N g t t P X D h c M 6 9 3 H t P E H O m w L a / j d L C 4 t L y S n m 1 s r a + s b l l b u + 0 V J R I Q p s k 4 p H s B F h R z g R t A g N O O 7 G k O A w 4 b Q e j q 9 x v 3 1 O p W C T u Y B x T L 8 Q D w f q M Y N C S b + 6 5 I Y Y h w T y 9 y f z U B f o A a Y y B D L P M N 6 t 2 z Z 7 A m i d O Q a q o Q M M 3 v 9 x e R J K Q C i A c K 9 V 1 7 B i 8 F E t g h N O s 4 i a K x p i M 8 I B 2 N R U 4 p M p L J z 9 k 1 q F W e l Y / k r o E W B P 1 9 0 S K Q 6 X G Y a A 7 8 4 v V r J e L / 3 n d B P o X X s p E n A A V Z L q o n 3 A L I i s P x O o x S Q n w s S a Y S K Z v t c g Q S 0 x A x 1 b R I T i z L 8 + T 1 n H N O a u d 3 p 5 U 6 5 d F H G W 0 j w 7 Q E X L Q O a q j a 9 R A T U T Q I 3 p G r + j N e D J e j H f j Y 9 p a M o q Z X f Q H x u c P c 0 G Y s w = = < / l a t e x i t > Lpatch < l a t e x i t s h a 1 _ b a s e 6 4 = " O 4 V o y X T R Z l H O J I s k e n Z K t w m B 1 o 0 = " > A A A C A 3 i c b V D L S s N A F J 3 U V 6 2 v q D v d B I v g q i T i a 1 l 0 4 8 J F B f u A J o T J d N o O n S T D z I 1 Y Q s C N v + L G h S J u / Q l 3 / o 2 T N g t t P X D h c M 6 9 3 H t P I D h T Y N v f R m l h c W l 5 p b x a W V v f 2 N w y t 3 d a K k 4 k o U 0 S 8 1 h 2 A q w o Z x F t A g N O O 0 J Sg t E V i H o u u h y X l L K I t Y M B p N x E U h x 6 n H W 9 4 X f i d E R W S x d E 9 j B P q h L g f s Y A R D E p y 9 X 0 7 x D A g m G e 3 u Z v Z Q B 8 g w / 4 o z 1 2 9 Z t b N C Y x 5 Y p W k h k o 0 X f 3 L 9 m O S h j Q C w r G U P c t M w M m w A E Y 4 z a t 2 K m m C y R D 3 a U / R C I d U O t n k g 9 w 4 U o p v B L F Q F Y E x U X 9 P Z D i U c h x 6 q r O 4 V 8 5 6 h f i f 1 0 s h u H Q y F i U p 0 I h M F w U p N y A 2 i j g M n w l K g I 8 V w U Q w d a t B B l h g A i q 0 q g r B m n 1 5 n r R P 6 t Z 5 / e z u t N a 4 K u O o o A N 0 i I 6 R h S 5 Q A 9 2 g J m o h g h 7 R M 3 p F b 9 q T 9 q K 9 a x / T 1 g W t n N l D f 6 B 9 / g D R W Z f K < / l a t e x i t > Ladv 
conclusion: 
 In this work, we demonstrate the significant improvement achieved through our novel latents alignment approach in 3D object removal. By integrating both explicit and implicit latent alignment mechanisms, we have successfully addressed key challenges associated with geometry mismatches and color inconsistencies that are prevalent in the baselines, enhancing the fidelity and detail of the inpainted 3D scenes. The improvements achieved through our work offer significant societal benefits, such as enhanced editability of radiance fields. However, it also poses risks, including the potential perpetuation of biases and discrimination. If the data used to train diffusion models is biased, our approach could inadvertently reinforce these biases.Despite these advancements, our method has certain limitations. It struggles to achieve full 3D consistency due to the intrinsic constraints of applying 2D diffusion models to multi-view data. To overcome this, future iterations could explore the integration of multi-view training directly within the 2D inpainting diffusion models (DMs) framework, or exact 3D generative models. Our approach currently requires predefined masks as a prerequisite for the task. Integrating advanced 3D perception methods [66, 67] could greatly improve the accuracy and flexibility of the approach. This integration would enable precise, language-driven interactions, enhancing automation and adaptability across various scenes and creating a more user-friendly, versatile framework for neural 3D scene editing. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: In the conclusion (Sec. 6), we discuss the positive societal impacts of our research and state the potential harms of technology misuse.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: No new dataset or model is proposed in this work.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All existing assets used in this work are properly credited and are the license and terms of use explicitly mentioned and properly respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [Yes]Justification: We have provided clear instructions for human evaluation in our user study and included a sample screenshot in the supplementary materials, presented in Sec. 5 and Sec. A.4.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [Yes] Justification: We have explicitly informed all participants in the user study that the collected data will be used solely for research purposes.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: For the inpainting network, we employ the stable-diffusion-2-inpainting model [71] , which encodes a masked image into the same dimensional latent space and integrates conditions via concatenation. We set the denoising steps for inpainting at 20. To achieve better generalization, we propose sampling the base frame according to the geometrical centroid of the training camera poses, meaning the camera that sits most centrally among the training views. However, we found that Stable Diffusion occasionally inpaints strange artifacts in the masked region. To mitigate this, we propose sampling n candidate views around the geometrical centroid and selecting the one with the highest similarity votes. This approach automatically avoids such occasion artifacts without human intervention. In our implementation, we used five candidate views, and the similarity was calculated using perceptual hashing. In the reprojection procedure of ELA, we adjust the camera intrinsics to match the latent dimensionality. Furthermore, to refine the ILA mechanism, we incorporate Cross-View Attention (CVA) into every self-attention layer of the inpainting model. Each step in this modified approach is controlled with λ a set at 0.2.For our 3D representation (NeRF) implementation, we utilize the "nerfacto" framework proposed by NerfStudio [83] . To ensure stable training, we deactivated the view-dependent effect. We pre-train the NeRF using 10000 iterations in stage 1 and jointly optimize it using 5000 iterations in stage 3. Our monocular depth estimation adopts DepthAnything [98] , complemented by the depth loss outlined in DS-NeRF [17] . Moreover, we employ StyleGAN2 discriminator [35] to implement adversarial loss.We conducted several sensitivity analyses regarding the base view selection, λ a in ILA, and the subset selection. Due to the computational burden, we conduct the sensitivity analysis on six out of ten scenes with higher inpainting variability from the SPIn-NeRF dataset.To achieve better generalization, we propose sampling the base frame according to the geometrical centroid of the training camera poses, meaning the camera that sits most centrally among the training views. However, we found that Stable Diffusion occasionally inpaints strange artifacts in the masked region. To mitigate this, we propose sampling n candidate views around the geometrical centroid and selecting the one with the highest similarity votes. This approach automatically avoids such occasion artifacts without human intervention. In our implementation, we used five candidate views, and the similarity was calculated using perceptual hashing.We tested our results under different settings (candidate numbers): 3, 5, 7, and 9. The base frame selection algorithm proved to be robust, with our algorithm typically yielding the same base frame. However, another factor influencing this step is the random seed. Setting different seeds causes the 2D inpainting model to produce different results, leading to different base frames being selected. We tested our methods under five different seeds, and the final scores are reported in Table 4 . While different seeds cause the final NeRF to differ in the appearance of the masked region, the consistency of the multi-view inpainting results remains robust, resulting in minimal variance in the evaluation scores. To effectively examine the effect of the hyper-parameter λ a in ILA, we evaluated our method's rendering quality with different λ a values of 0.2, 0.4, 0.6, and 0.8. The metrics are reported in Table 5 . Quantitatively, the results are consistent across different λ a values, indicating that the effect of this hyper-parameter is relatively small. This conclusion is also supported by qualitative results.Larger λ a values tend to produce slight variations in some small regions, but the global structure and semantics are preserved. This stability is attributed to the significant role of the initial latent alignment in ELA, which effectively aligns the underlying inpainting structure, thereby maintaining low variability in appearance. Additionally, the self-attention layer, where cross-view attention is introduced, does not dominate the entire Stable Diffusion Unet. It is balanced by the presence of other (residual and linear) layers, ensuring cross-view attention does not override the signal during the denoising process. Hence we simply set λ a as 0.2 in our implementation. We found that for reconstruction tasks, more views can enhance quality; however, for generation tasks, using the entire set of images can introduce unnecessary inconsistencies. Therefore, we propose selecting the subset according to the distribution of camera viewpoints.We evenly split the viewpoints into 12 groups based on the base view's camera space (evenly 2 on the x and y axes and 3 on the z-axis) and select 50 percent within each group according to perceptual hashing similarity to the base view. This approach avoids redundant views introducing supervision conflicts while covering different viewpoints for effective supervision.We also evaluated our method based on different percentages, as reported in Table 6 . The quantitative scores are quite close, indicating that for most scenes, the difference isn't significant. For one complex scene with extremely high frequencies, setting the percentage too low (0.2) yields artifacts in the test view due to insufficient viewpoint coverage. Conversely, setting the percentage too high (0.8) introduces appearance conflicts due to the high variability of the inpainted results.Overall, for most scenes, the subset selection algorithm is robust due to the consideration of viewpoints distribution. For extreme cases, careful selection of the percentage might be necessary. However, values between 0.5 and 0.7 remain a reliable choice. To assess the sensitivity of the patch loss multiplier λ patch , we evaluated the method's performance using various values of λ patch : 0.001, 0.005, 0.01, 0.05, and 0.1. The results are reported in Table 7 .Analysis of the table indicates that varying λ patch leads to similar performance across different settings, with a low standard deviation of the metrics. However, there is an observable trend where setting λ patch too low or too high adversely affects performance. The multiplier λ patch is critical as it determines the extent of influence multi-view images have on the NeRF. Insufficient multi-view supervision can lead to inadequate training, whereas excessive supervision may result in conflicting inputs. Consequently, we have set λ patch at 0.01 in our implementation for optimal balance. This section presents extended qualitative results from our experiments on the SPIn-NeRF Dataset. Fig. 8 and Fig. 9 showcase a series of multi-view comparative inpaintings.To comprehensively evaluate our method using human subjects, we conducted a user study focusing on three aspects: (1) Background Coherence -assessing whether the inpainted area blends seamlessly with the remaining background, (2) Detail Preservation -determining if the inpainted area retains high-fidelity details, and (3) Overall Quality -gauging participants' preference rates for the inpainted results. For each method, we presented users with two multi-view test images from each scene and instructed them to choose the method that best met the criteria for each aspect. Clear instructions were provided to ensure participants understood the rating process. An example screenshot of the study interface is shown in Fig. 10 .The user study we conducted focused solely on collecting participants' preferences regarding different inpainting results, involving no sensitive or personal data collection beyond their aesthetic judgments. The study's design was inherently low-risk as it required participants to simply view and evaluate digital images based on their visual appeal and perceived quality. Furthermore, the participation was entirely voluntary, with clear instructions provided, allowing participants to withdraw at any time without any consequence. Given these factors, the potential for harm or discomfort to participants was negligible, ensuring the study maintained a minimal risk profile. Justification: We have reviewed the NeurIPS Code of Ethics and ensured that our research adheres to its guidelines.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: This paper is about 3D object inpainting using 2D diffusion prior. The paper received mixed scores of two borderline reject, one weak accept, and one accept. The negative reviewers (S5oa, uTYn) have concerns about the details and motivation of the proposed ELA approach, missing comparisons with baselines, and the lack of computational efficiency comparisons. In the rebuttal, the authors provided more detailed descriptions of the proposed method, showed preliminary results on object completion, explained why reference-guided inpainting (ICCV 2023) is not compared (as it is not open-sourced), and provided computational efficiency results. Unfortunately, the reviewers have not responded to these new results. The other two reviewers appreciate the technical novelty of the method and believe that the evaluation is convincing. 

Based on the reviews and the author's rebuttal, the AC agrees with the reviewers that the authors' rebuttal largely addresses the initial concerns. The AC believes that this paper has sufficient merits and thus recommends its acceptance.

Reviews: review: paper_summary: paper_summary: This paper proposes to solve multi-view or 3D inpainting problem. Given multiple posed views and the masks in each view specifying an object to be removed. It first trains a NeRF on the unmasked regions. To in-paint the masked regions, a seed view is selected and inpainted with a pre-trained diffusion inpainting pipeline. Depth is also predicted for this seed view, together with the RGB value, fused into the NeRF representation. To propagate the features from this view to other views for NeRF supervision, it proposes explicit and implicit feature alignment methods. For the explicit one, it propagates the initial latent noise to another view via a depth/density-based aggregation method.  For the implicit one, it uses a method similar to reference-only control, i.e., extracting intermediate key/value pairs from the seed view and injecting them into other inpainting views via cross-attention.
strength_weakness: strength_weakness: (1) The proposed method seems much simpler(or even over-simplified) than prior works such as NeRFiller and InFusion. The former uses a synchronized multi-view inpainting technique which is similar to the ILA of this paper. The latter one introduces new diffusion-based depth completion. Both prior methods have used a depth prediction model, and InFusion has shown that using a pre-trained depth model and aligning it with  NeRF/Gaussian depth is sub-optimal.  
For qualitative comparisons, there is little or only marginal difference between this paper and InFusion, while InFusion/NeRFillter has shown much more results than only the scenes included in this paper. 
I suspect that the result difference is due to the choice of base components, (i.e., inpainting pipeline, depth estimation model, etc.) which can not be considered as the novelty of this paper.

(2) As I understand, the three stages run sequentially. i.e., Train unmasked NeRF(Stage-1) -> Update the NeRF with seed view and depth(Stage-1) -> Update the NeRF with novel inpainted views (guided by the seed view)(Stage2 then 3). Although Equation. (10) shows a total loss function. 
- This way, the proposed ELA approach seems over-complicated. After stage 1, there is no NeRF information in the masked region, and there is only one view with depth provided to supervise the masked area. There seems no need to use the NeRF density to propagate the feature from the reference inpainted view to other views. An alternative can be using the depth prior and the epipolar line to aggregate features. 

(3) Prior works show both object insertion and removal tasks, while the proposed method seems can only be applied to object removal tasks, as it heavily relies on the initial inpainting and depth estimation result. Moreover, the examples shown in this paper don't include complex structures and only show simple background, inconsistency caused by possible occlusion or partial observation has not been tested. Overall, this paper compares the results with the previous generation's scene completion method in a very limited scenario, which I believe is unfair. 

(4) The artifacts are obvious in the result videos, which can be caused by the noise/artifacts in the depth estimation model. More evidence is needed to make this paper stronger.
questions: questions: See weakness.
limitations: limitations: Limitations addressed, however, the author has not mentioned that the proposed method is only tested in a limited scene case.
paper_summary: paper_summary: This paper presents how to tackle the challenge of 3D object removal by involving 2D diffusion prior. The approach involves pretraining NeRF with inpainting prior and then jointly optimizing it with latent alignment to align feature priors.
strength_weakness: strength_weakness: 1. Although the generated results outperform the listed baseline methods, the paper lacks a comparison with other baselines, such as "Reference-guided controllable inpainting of neural radiance fields" (ICCV 2023)", which achieves good results of novel view synthesis. 
2. The presentation of the paper could be improved, for example, it is uncertain how to get the patches in the patch-based loss (Equ. 9).
questions: questions: 1. In Equ. 5, does the first term indicate the use of ground truth images (with the object removed) for supervising the mask region? If so, are there any concerns regarding potential information leakage? 
2. Regarding efficiency, does the multi-stage training approach in this paper have a comparison with baseline methods in terms of computational efficiency?
limitations: limitations: The author provides a comprehensive discussion of the broader impacts in the paper.
paper_summary: paper_summary: The paper deals with 3D inpainting problem in the NeRF setup with 2D diffusion models. To achieve multiview consistency, they took a "inpaint-outstretch" strategy. They first inpaint one key frame, and conditioned on which, generate a view-consistent inpainted image set. Finally, they use the multiview images to train the final NeRF model, together with the reconstruction losses. The method is compared to a few recent works (Spin-NeRF, NeRFiller, InFusion) and achieved better results both qualitatively and quantitatively.
strength_weakness: strength_weakness: - My biggest confusion is on the explanation of ILA. 
  - I don't fully understand why replacing the KV with the base frame KV would encourage consistency. I would expect the style to be consistent but not the local geometry, where the latter is more important for 3d reconstruction.
  - What is the rationale of replacing KV with "prior" p, but not Q?
- The algorithm relies on a pre-defined base frame p. How is the base frame chosen, and how sensitive is the algorithm to the chosen frame?
questions: questions: - Is Eq (10) applied to Stage 3? If so, what is the purpose of the prior loss L_prior? Once the inpainted base image is used to generate a multiview training set, I would expect this information to be unnecessary? Can this term be merged into L_patch? 
- In L216, what is the subset? Why not inpaint the whole set of images and how much does it affect the results?
limitations: limitations: Yes
paper_summary: paper_summary: The paper introduces a novel approach termed "In-N-Out" for enhancing the performance of 3D object removal tasks by leveraging tuning-free latents alignment. The authors have addressed the challenges of geometric mismatches and color inconsistencies prevalent in existing methods.
strength_weakness: strength_weakness: The paper exhibits signs of being finished in a rush, which affects the overall quality of academic writing, and is hard to follow.
For example，
1. Line172，Omega(·) is not defined
2. Formula (5), D_hat is note defined
3. Line 208， by replace -> by replacing
The paper might has the potential to contribute to the field but requires significant revisions to improve the academic writing quality.

The method involves several hyperparamete. The paper does not discuss the sensitivity analysis or the robustness of the method against variations in hyperparameters.  Especially，the lambda_a, which plays an essential role in ILA.
questions: questions: 1. In ELA module, using latent z^T to replace color c for volume rendering. Will this compromise the view-dependent in NeRF? In NeRF，the color  is dependent on the position x and direction d, whereas in ELA, the z^T is merely reprojected to the image plane. 

2. How do you choose lambda_a in ILA.
limitations: limitations: The method involves several hyperparamete. The paper does not discuss the sensitivity analysis or the robustness of the method against variations in hyperparameters.

Comments: title: Decision
Future work discussion.
Future work discussion.
comment: Thank you very much for your updated feedback and for considering an increase in your score. We greatly appreciate your support and recommendation for acceptance.
Thanks for the clarification. I'm willing to raise my score and recommend for acceptance.
Thanks you very much for your feedback.

Regarding L169, the two components described are the input prompt $e$ and the masked image $I_p^{\prime}$ , as detailed in L170.  Figure 2 illustrates the masked image $I_p^{\prime}$. The prompt $e$ is simply set as the description of the background we aim to inpaint, which is scene-dependent.

In L199, the prompt $e$ is defined same as In L169, we employ a unified prompt for each scene.

It's important to note that we did not involve any prompt engineering in our work, and we use the same prompt for each scene across all the methods in the experiments. We will provide further clarification on this point in the revised version.
Thanks for the rebuttal that clarifies my confusion. The additional ablations are helpful. I believe incorporating them will substantially strengthen the paper. 

Can you further clarify what are the two components in L169 and prompt e in L199?
Dear Reviewer uTYn,

I hope this message finds you well. We are grateful for the time and attention you've already dedicated to reviewing our work. We have submitted a rebuttal addressing the concerns raised, and we would greatly appreciate any additional feedback or comments you might have. As the discussion period is drawing to a close soon, we are eager to know if our rebuttal addresses your concerns or if there is any additional feedback you might have. Thank you very much for your continued support and guidance.

Best regards,

Authors of Submission 4061
Dear Reviewer S5oa,

I hope this message finds you well. We are grateful for the time and attention you've already dedicated to reviewing our work. We have submitted a rebuttal addressing the concerns raised, and we would greatly appreciate any additional feedback or comments you might have. As the discussion period is drawing to a close soon, we are eager to know if our rebuttal addresses your concerns or if there is any additional feedback you might have. Thank you very much for your continued support and guidance.

Best regards,

Authors of Submission 4061
Thank you very much for your feedback and for reconsidering your review score. We are glad to hear that our rebuttal addressed most of your concerns. We are committed to further refining our discussion on the relationship with 3D perception works in the revision, and we appreciate your support and constructive comments as they greatly enhance our work's development.
I appreciate the effectiveness and thoroughness of this work's experiments. After reading the author's rebuttal letter, which addressed most of my concerns and  promised to discuss the relationship with 3D perception works in the next revision, I have decided to raise my review score to 6—weak accept
Thank you for reviewer's valuable insights. We completely agree that 3D perception and segmentation methods are highly related to our task. Indeed, accurate 3D perception/segmentation methods are a prerequisite step, as the quality of multi-view masks directly impacts the performance of inpainting outcomes. In practice, we find these methods play a crucial role in machine vision automation and are applicable to a variety of downstream tasks.

Integrating 3D perception approaches such as LangSplat[1] and GOI[2] with object removal and 3D inpainting tasks can significantly enhance the effectiveness and flexibility of these applications. These advanced perception methods, which efficiently map language queries to 3D spatial contexts, crucial for specifying and identifying the exact areas for removal or inpainting in complex scenes. For instance, using language-driven approach, a system can precisely locate and manipulate objects as per user instructions, enabling more intuitive and interactive design adjustments. Moreover, the open-vocabulary capabilities of these methods ensure that the system can adapt to a wide range of objects and scenes, potentially improving the automation and generalization of 3D inpainting tasks. In the future work, we could combine our work with 3D perception methods to enable a unified framework for 3D scene editting. This will largely enhance flexibility and user-friendliness. More details will be discussed in the revised version.
The current model uses multi-view masks as input. In future work, it could be combined with 3D perception methods, such as LangSplat [1] and GOI [2], which are two SOTA open-vocabulary 3D  perception works. I hope you can include a discussion in the camera-ready version about the potential of integrating 3D perception approaches.

[1] Langsplat: 3d language gaussian splatting, CVPR 2024
[2] GOI: Find 3D Gaussians of Interest with an Optimizable Open-vocabulary Semantic-space Hyperplane, ACM MM2024
Thank you for reconsidering the revised experiments and for acknowledging the improvements in the results. We appreciate your decision to increase the score and are grateful for your thorough feedback.
The author has conducted additional experiments, and the results now appear satisfactory. Therefore, I have decided to increase my score by one point.