MVInpainter: Learning Multi-View Consistent Inpainting to Bridge 2D and 3D Editing
Authors: Chenjie Cao
Chaohui Yu
Fan Wang
Xiangyang Xue
Yanwei Fu
Venue: NeurIPS 2024 poster (2024)
Keywords: Multi-view synthesis
Image inpainting
3D editing
Primary area: generative_models
TL;DR: We propose MVInpainter, re-formulating the 3D editing as a consistent multi-view 2D inpainting task to enable generating vivid multi-view foreground objects seamlessly integrated with background surroundings.

Abstract: Novel View Synthesis (NVS) and 3D generation have recently achieved prominent improvements. However, these works mainly focus on confined categories or synthetic 3D assets, which are discouraged from generalizing to challenging in-the-wild scenes and fail to be employed with 2D synthesis directly. Moreover, these methods heavily depended on camera poses, limiting their real-world applications. 
To overcome these issues, we propose MVInpainter, re-formulating the 3D editing as a multi-view 2D inpainting task. Specifically, MVInpainter partially inpaints multi-view images with the reference guidance rather than intractably generating an entirely novel view from scratch, which largely simplifies the difficulty of in-the-wild NVS and leverages unmasked clues instead of explicit pose conditions. To ensure cross-view consistency, MVInpainter is enhanced by video priors from motion components and appearance guidance from concatenated reference key\&value attention. Furthermore, MVInpainter incorporates slot attention to aggregate high-level optical flow features from unmasked regions to control the camera movement with pose-free training and inference. Sufficient scene-level experiments on both object-centric and forward-facing datasets verify the effectiveness of MVInpainter, including diverse tasks, such as multi-view object removal, synthesis, insertion, and replacement. The project page is https://ewrfcas.github.io/MVInpainter/.

MVInpainter: Learning Multi-View Consistent Inpainting to Bridge 2D and 3D Editing
Chenjie 12...2,3, Chaohui 2. Fan Wang?.3, Xiangyang Xue', Yanwei Fu' FFudan University, "DAMO Academy, Alibaba Group, 'Hupan Lab {caochenjie.c ccj huakun. ych, fan. =)@alibaba- ieeaainn.com com {xyxue , yanweiful@fuuaa edu. cn
Abstract
Novel View Synthesis (NVS) and 3D generation have recently achieved promi- nent improvements. However, these works mainly focus on confined categories or synthetic 3D assets, which are discouraged from generalizing to challenging in-the-wild scenes and fail be employed with 2D synthesis directly. Moreover, these methods heavily depended on camera poses, limiting their real-world ap- plications. To overcome these issues, we propose MVInpainter, re-formulating the 3D editing as multi-view 2D inpainting task. Specifically, MVInpainter partially inpaints multi-view images with the reference guidance rather than in- tractably generating an entirely novel view from scratch, which largely simplifies the difficulty of in-the-wild NVS and leverages unmasked clues instead of explicit pose conditions. To ensure cross- view consistency, MVInpainter is enhanced by video priors from motion components and appearance guidance from concate- nated reference key&value attention. Furthermore, MVInpainter incorporates slot attention to aggregate high-level optical flow features from unmasked regions control the camera movement with pose-free training and inference. Suffi- cient scene-level experiments on both object-centric and forward farinn-aangg datasets verify the effectiveness of MVInpainter, including diverse tasks, such as multi- view object removal, synthesis, insertion, and replacement. The project page is httts://ewrffet wites..gittu.bb... io Io/MIInpaitter
Introduction
This paper studies editing 3D scenes by expanding one or few 2D manipulated references to other observed views. Particularly, with the development diffusion-basee text-to-image (T2I) mod-
els [59] 63 54], we have seen substantial success in novel view synthesis (NVS) [3,
30 97]. 3D
generation 155 40
79
96
44
67
43,
(66), and controllable generation [60] 98
TO [4] But most
existing synthesis methods 14
98 10 have only been proven useful in 2D scenarios. It is intuitive
to extend these pioneering methods to multi-view scenarios to bridge the gap between 2D and 3D editing. This raises the question: how to make unified framework to generate vivid multi-view foreground objects seamlessly integrated with their surroundings?
Despite the achievements in NVS and 3D generation, achieving multi-view consistent scene editing by inserting, removing, replacing objects like Fig. still present challenges. 1) object generation struggles to generalize to scene-level editing. Most object-centric 3D generation methods [44 67 31
31 with simplistic backgrounds, neglecting real-world factors like illumination and shadows, which are essential for natural editing. While some
are trained on large-scale synthetic datasets [18
17
methods integrate predefined 3D assets into NeRFs [65] and 3D Gaussian Splatting (3DGS) [15), they
still struggle with blending foreground and background elements seamlessly. 2) NVS methods have
"Corresponding Author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Objecs-centric novel syothesis
Original multivview images
Removing foregrounds
(b) Multaview consistent obycet renovvl
Novel view MD0G
insertion
(c) Mahi-siew object wwsetoooeeepteeeeeeen
(d) 3D seene editing
Figure I: MVInpainter addresses 2D/3D editing tasks: (a) novel view synthesis, (b) multi-view object removal, and (c) object insertion and replacement through nulti-view consistent inpainting ability. Given one inpainted or edited reference image, MVInpainter spreads it to other masked views without pose conditions. (d) MVInpainter could be applied to real-world 3D scene editing for dense point clouds by Dust3R (76) or Multi-View Stereo (MVS) [9] and 3DGS (35] with consistent generation.
difficulty generalizing across various categories. Even when enhanced by diffusion models, existing 73 29 1 only work in specific scenarios and often fail to generalize di- verse or unseen categories in scene data. 3) Instance-level 3D editing is time-consuming. Approaches and warp-and-inpanl
NVS-based methods [2
NVS [23 30 97 integrate priors from single-view T21 models [59] requiring costly dataset updates to address multi-view inconsistency. 4) Heavy reliance on explicit camera poses. All methods mentioned rely on accurate camera poses for both training and inference, limiting scalability with pose-free data and broader applicability, particularly in scenarios like short video editing where detailed poses may be unavailable.
like instance-level 3D editing 26
To this end, we present a novel perspective to enable 3D editing with a multi-view consistent impainting manner. By starting with an edited 2D reference image from any 2D generative models 98 14 and applying our method to masked sequences from other viewpoints, we achieve consistent multi-view inpainting results, effectively extending 2D generation into 3D scenarios. The key idea behind this inpainting approach is to focus on seamlessly synthesizing local regions rather than generating entirely new views. Many contextual cues like illumination, shadow, and camera motions are implicitly captured in unmasked regions, which can be activated by foundational T21 models. Thus, our method provides an end-to-end multi-view synthesis solution without requiring test-time optimization or explicit pose conditions, opening up possibilities for both 2D and 3D editing.
Formally, this paper introduces MVInpainter, multi-view consistent inpainting model built upon pre-trained StableDiffusion (SD) inpainting model [59] We incorporate domain adapters and motion modules (25 to MVInpainter as video priors for multi-view consistent structures. Moreover, to encourage appearance consistency, we propose the Reference Key& Value concatenation (Ref- KV), spatially concatenating the key and value features from the reference view to target ones in self-attention modules. Furthermore, MVInpainter operates without explicit poses, utilizing slot- attention mechanisms 45 87 of encoding and grouping motion priors from unmasked surroundings" optical flow for implicitly pose control. We train two MVInpainter sharing the same pre-trained SD backbone in object-centric (CO3D (57, MVImgNet 955 and forward-facing (Scannet++ [89], Reall0k 103), DL3DV (41) scenes, respectively. Experiments on unseen scenes and zero-shot datasets as D 6 and SPInNeRF [5T] show the efficacy of MVInpainter in various applications, such multi-view object removal, synthesis, insertion, and replacement.
In summary, we present MVInpainter as the first multi-view consistent inpainting model to bridge 2D and 3D scene editing. 2) MV MVInpainter is pose-free end-to-end approach with high-level flow- based motion control from unmasked regions. 3) MVInpainter largely simplifies the NVS difficulty, which can be generalized to all categories of in-the-wild CO3D, MVimgNet, and Omni3D datasets, as well as complicated forward-facing scenes of Scannet++, Real 10k, DL.3V, and SPInNeRF.
2 Related Work
Image Inpainting devoted to filling masked regions of the image with vivid textures and structures as long-standing challenge in computer vision, which has been widely investigated in both classical and

---

learning- -base methods 133 Compared to the low-level feature- based traditional manners [61, (6],
souring-basee ones gradually dominate this field and achieved substantial successes based on GANs
53
100
37
8
attention mechanism [93 10], adapted convolutions [42 94 70], and diffusion
Moreover, the image inpainting task can be further extended to reference-guice guided inpainting and video inpainting. Reference-gune guided inpainting completes the target image based on one or several reference images, which incorporates 3D information for accurate structures [I05] [OT] or T2I priors for examplar-based recovery and editing [86] 7 On the other hand, video inpainting methods [24] 39 [02] often include the optical flow to capture motions for superior spatial and temporal coherence. However, we should clarify that the aforementioned inpainting methods cannot achieve multi-view consistent inpainting for 3D scene editing. The reference- based manners lack multi-view consistency across all generated views, while video-based methods focus moving foregrounds rather than the synthesis with large viewpoint changes as verified in our experiments.
models 47
62
59
3D Generation. Given text descriptions or reference images, 3D generative models produce high- quality 3D assets, which have been widely investigated recently, benefiting from the rapid devel- opment of diffusion-basee 2D T2I models [59 63 54 21]. Some pioneering works with score distillation sampling (SDS) loss [75 55
leverage priors from diffusion-based 2D supervision for the 3D generation, which is further explored to better optimization objectives [06 79] and multi-stage learning [4 13 71. On the other hand, Zero 123 [44] fine-tuned the T2I model for object-level NVS, which is further investigated for consistent multi- villi--iewweee synthesis 667 66 43
46 Besides, enhanced
by the good 3D feature presentations. like tri- tri-plane [T], and scalable network backbones [74], foun- dational 3D generation models also achieved impressive results [09 85 BT.. training with extremely large 3D datasets from scratch. However, all these works are trained with large-scale synthetic 31) objects optimized through SDS without any interaction with complicated backgrounds, making it difficult to generalize to real-world editing scenarios with reasonable illustrations and shadows.
Novel View Synthesis (NVS), Before the diffusion models, most NVS works focused on learning promising feature encoder [69 [9]] with blur regression-based predictions. After the development of
3D-aware diffusion models 12 73 and fine-tuning from foundational T2I models (64, 29 (22.
NVS results are significantly improved. However, generating whole novel views is too challenging. Thus these methods still suffer from constrained generalization with seen categories expensive test-time optimization. Another way is to iteratively warp and inpaint the novel views through monocular depth estimation and 2D inpainting 192 23 30 977. Despite impressive scene-level synthesis results, these methods suffer from prohibitive time costs for the warp-and-inpain dataset update. Moreover, ambiguous depth estimations would degrade the structures of foreground objects.
NeRF and 3DGS Editing. With the development of NeRF [49] and 3DGS (35), many follow-ups
tried to integrate them into 2D generative models, including NeRF inpainting 151 81 50 textual-
guided semantic editing (26 [5] and local editing [108] 3 38) based on SDS loss. InseRF (65] unifies both NeRF editing and 3D generation by inserting an image-to-3l generation into multi-view images for the NeRF optimization. Unfortumately, local editing- based approaches suffer from unnatural and disharmonious results, especially for the illumination and shadow in boundary regions. Furthermore, all these works require scenes with accurate camera poses and costly test-time optimization to encourage multi-view consistency, limiting their real-world applications.
3 Approach
Overview. We show the overall pipeline and contributions in Fig. 2, which detailed in Fig. 3(a). The inputs of MVInpainter are sequential images 1o:N of the same scene and related masks M with N ++ 1 total views; 1 indicates the clean reference image manipulated by any 2D editing approach, while I I are other target views needed to be inpainted by MVInpainter with consistent results.
Multi- Inpainting Formulation. We focus on multi-view inpainting rather than naive NVS as it suits real-world editing better, because 1) most real-world edits do not need to synthesize complete novel views; 2) the inpainting formulation activates the inherent in-context priors from T2I models to ensure natural illumination and shadow, which are intractable to be addressed by 3D generation trained with synthetic data; 3) such a simplified formulation alleviates the training difficulty and the dependency on camera poses. Specifically, MVInpainter enjoyy consistent multi- view inpainting to bridge 2D and 3D editing, which is built upon a foundation 2D inpainting model, SDI.5-inpainting [59] with two data prerequisites. First, the input multi-view images 10:N should
3

---

Multi-view removal (optional)
Multi-view symthesis/insertion
3D reconstruction (optional)
2D-inpainting
SD2 SDXL impainting
MVInpainter-F
MVInpainter-O
3DGS-LPIPS
Main contributions
Figure 2: The overall pipeline and main contributions of MVInpainter. We primarily focus on multi-view inpainting, while the 3D reconstruction detailed in Appendix Sec.
Clbect-eenticc
Flow Grouping
Refirrnce
Selfattantin
On
Kg
V
Reference
Object-centri LoRA
Object
MY Inpailnten-O
Object-centric Motion
Target views EE K
Ref-KV of self-attention block in U-Net
Forward-lccin images
Grrouping
Grouping block x3
MANE
Refirenco
mi
Seene-level LoRA
Randon maskss
Secne-level Modion
Masked flow
queties
(a) Overview framework of MVIpaiitntrr Figure 3: (a) The overview of the proposed MVInpainter. MVInpainter-O is trained object-centric data, while MVInpainter-Fi is trained on forward-facing data with a shared SD-inpainting backbone of different LoRA/motion weights and masking strategies. The object-centrie MV MVInpainter focuses on the object-level NVS, while the forward- facing one devoted to object removal and scene-level inpainting. (b) The Ref-KV is used in spatial self-attention blocks of denoising U-Net. (c) The slot-attention based flow grouping module used to learn implicit pose features. Dashed boxes in (b) and (c) mean feature concatenation.
Architecturee Flow Grouping
be captured in an ordered camera trajectory to alleviate the pose requirement. Second, M is zero matrix to ensure the first view is clean without masking, while other masks M should cover all possible regions in respective views where the target object would be placed at. To elegantly meet this demand, we propose heuristic masking technique for the inference detailed in Sec. 3.4. Hence, the input of MVInpainter could be formulated as ordered video sequence with (N + frames as:
Tt = [= 0:N ; M) 0:N M 0:N (
where t indicates the timestep in the diffusion; 2 denote the 4-channel noised latent feature of I0:N after the VAE encoding [59]; M and 2(1 M)):N mean that the downsampled masks and noise-free latent features in unmasked regions are always concatenated as the input condition. We learn MVInpainter through the epsilon prediction eg 28 while the MSE loss be written as:
L 1(..). [[|c c0 (c). +)]
where c denotes the prompt text encoded by the textual CLIP T 56
3.1 MVInpainter Tasks
Here we tackle two related but distinct tasks: multi-view consistent object removal, and object insertion replacement. Due the differing challenges of each task, employ the same structure, called MVInpainter, but trained with different multi-view data, prompts, and masking strategies, leading to variants: MVInpainter-O and MVInpainter-FF
Object-Centrif and Forward-Facing Datasets. Particularly, we train these two MVInpainters to handle different distributions of multi-view data: object-centric and forward-facing data. Object-

---

centric datasets (CO3D 1571, MVImgNet 1951, Omni3D [6] feature single object at the center of all views, captured with the camera circling around it. On the other hand, serward-facing datasets [Real10K]] T0X3, Scannet++ [89] DI.3DV (41)) resemble static video data with camera movements but no specific foreground objects. Essentially, object-centric images heavily favor a single foreground object in all views, so an MVInpainter trained on them struggles to reliably remove objects without hallucinating any artifacts. Conversely, forward-facing images are not conducive to modeling multi- view object synthesis with significant viewpoint changes.
To address these challenges, we separately train MVInpainter- O and MVInpainter-F on object-centrie and forward- facing datasets, respectively. They involve different LoRAs, motion modules (Sec. and flow grouping modules (Sec. (.3). Note that we keep the SD-inpainting backbone frozen for both models. While fine-tuning the entire SD backbone for M MVInpainter- O might help convergence with mixed CO3D and MVImgNet datasets, we maintain consistent settings with MVInpainter-F for simplified discussion in this study. Please refer to Sec. B.5 of Appendix for more details.
Prompt. For MVInpainter-O, we empirically find that meaningful prompts benefit to preserve the identity and appearance of objects, as these prompts provide complementary information for the object synthesis with unseen viewpoints. Thus we utilize InternLM 19] to extract captions for training and inference. For MVInpainter-F which is mainly used for object removal and scene completion with minor viewpoint changing. we leverage the prompt tuning technique ? with 16 trainable tokens as the global prompts instead of specific descriptions for stable generation.
Masking Strategy. We adopt hybrid inpainting masks 770 & for MVInpainter, including random irregular, rectangular, and segmentation- based masks. In particular, we further employ the object- level tracking masks for MVInpainter-O training, which could be easily accomplished by SAM- tracking [88]. Moreover, to avoid the mask overfitting towards the object shape, we follow [7] to randomly disturb the object masks with rectangular and irregular masks. Note that MVInpainte--O preserves a little percentage (15%) to use random masks without the object ones to encourage the cross-view learning ability for the frame interpolation and sparse-view NVS (Appendix Sec. B.2).
3.2 Multi-View Consistent Inpainting Model
Motion Priors from Video Models. Since the input of MVInpainter could be regarded as the video sequence as in Eq. 1, it is intuitive to leverage motion priors from video models to improve performance. Thus we employ the domain adapted LoRA 32 and temporal transformers from AnimateDiff 25 pre-trained on video data as the initialized motion parts of MVInpainter. Although AnimateDiff is not trained for an inpainting model, we surprisingly find that could be well converged with only few hundred steps of fine-tuning, while both MVInpainter and AnimateDiff share the same SD1.5 backbone. Empowered by motion priors, MVInpainter achieves significantly superior structural consistency discussed in Sec. 4.4
Reference Key& Value Concatenation (Ref-KV). Inspired by (66 33 to further ensure appearance consistency, we adopted Ref-KV in the self- attention of denoising U-Net to activate the inherent capacity of T2I models. Ref-KV spatially concatenates reference features to target keys and values to inject appearance guidance during attention aggregation as in Fig. 3(b). We clarify the originality of Ref-KV as follows. Compared to aggregating all frames (29] Ref-KV only focuses on the first reference view, which reduces memory and computational costs, and also substantially enhances the appearance consistency as verified in Fig. 7 Different from the reference attention in [66] (3), MVInpainter is inpainting model, which always captures the unmasked reference latent without noise (Eq. 1), thus is unnecessary to re-scale the latent by multiplying a large scale (66) or use the noise-free latent from another U-Net [33
3.3 Pose-Free Flow Grouping
Benefiting from the inpainting formulation and the ordered input sequence of MVInpainter, our approach could be trained and tested without explicit camera poses. However, it is still non-trivial to inpaint the foreground object with correct poses while the masks are large the unmasked environment is ambiguous and textureless. To overcome this, we leverage the low-level optical flows
"This paper only focuses on data captured by object-centric camera trajectory rather than discovering decoupled observations through unsupervised feature learning (45 [04 34
5

---

Masche freer plaze
Perquctive warpety botte Taz2
Refranc
MVaquites oerrevall
Makaabpun
MVhpaite dessiiin
Neufirtof
FANSSC
Fill Creveiiiel
(a) Inforence pipeline
(b) Masking adagtion
(c) Perspective warping of the object plane
Figure 4: (a) The inference pipeline includes object removal, mask adaption, and object insertion. (b) The illustration of heuristic masking adaption, which is built from yellow points of the closed convex hull. (c) The perspective warping based on the basic plane and the bottom face. All matches are on the basic plane filtered by Grounded-SAM [88] with captions "table" and "tablecloth".
extracted by RAFT 72 to guide the MVInpainter generation. We regard the reverse N-frame optical flow as FO:N-1
E R (N IN-1))xX*xx? All flow inputs are masked to avoid leakage'.
In the pilot study, we first applied the 2D CNN and self-attention modules to encode the flow features and added them the U-Net input as additional conditions. But such simple incorporation failed to improve the MVInaainter flow injection leads to the overfitting pitfall, Le., MV Innpainter strongly controlled by the flow inputs and ignores other contextual clues learned from the foundational T2I model. We hope the flow feature should carry more high-level motion characters, such as the direction and speed the camera trajectory rather than the detailed correlation. Thus propose the flow grouping enhanced by the slot-attention [45 87 as shown in Fig. 3(c).
the "dense flow' setting in Tab. 3b Because such an explicit dense
Formally, slot-attention maintains K learnable query vectors as Q E IR xd where d denotes the channels; K in this paper. Key and value the same flow features as K V IR IRW' Then the slot-attention can be formulated as:
OKT J E Rxdd
Slot-Attn( Q, K. V) = softmar,( Vd
(3)
where Q, K, V indicate QW, KW. VW, encoded by linear weights Wg. Ws. W, E Rdxd Note that the slot-attention is similar to the cross-attention except that the former should be normalized in the query dimension, while the latter is normalized the key dimension. As the global queries, all slot features Q enjoy the high-level motion information aggregated by dense flow features. We take these slot features and subsequently use an FC layer to encode them into a single flow grouping embedding shown in Fig. 3(c). According the ablation study in Tab. 3b concatenate this embedding to the CLIP feature as an extra motion 'token' with comparable performance and less trainable weights. Moreover, we found that using temporal 3D attention for the flow grouping can further improve the results, while the slot features are shared across all views with more general information. Flow grouping achieves more robust guidance for MVInpainter than the dense flow features, even with some corrupted or inaccurate flow inputs.
3.4 Inference
We show the inference pipeline MVInpainter in Fig. 4(a). Specifically, given sequential images, MVInpainter could address the general multi-view editing through object removal and object insertion mentioned Sec. 1. The object removal stage can omitted if there are no foreground objects (inserting only) or the target object shares similar masks as the original one (replacing). For the object removal, we utilize the MVInpainter-FF based on the reference image inpainted by SD-inpainting with the caption "background", For the object insertion, the reference image can be achieved by any 2D
*e extract flows before masking, because foregrounds largely benefit the flow quality. We further dilate masks with 5 pixels for flow to avoid leakage. As low-level local features, no conflict is observed when using these masked flows for applications like removal and replacement.

---

generative models, such as T2I inpainting [59 77 and exemplar- -aapled--aaeed inpainting [8 14). Then MVInpainter-O is leveraged expand the single-view generation to the multi-view scenario. We propose heuristic masking adaption to tackle the most critical issue, i.e., building suitable masks without any pose conditions before the insertion.
Masking Adaption is based on points from the closed convex hull as shown in Fig. 4(b), which can be obtained from the 3D bounding box of the foreground object. We could use open-vocabulaad 3D grounding [6] or manually annotate four landmarks to achieve the bottom face and estimate the top face through the height of the 2D mask. To reasonably warp the 3D box, our masking adaption should meet an important and easily satisfied condition: the 3D box's 's bottom face and the basic plane on which the object is placed must be the same plane. Intuitively, the bottom face should be close to the ground. Therefore, the basic plane and the bottom face of the 3D box from different image pairs share the same perspective transformation matrix. Thanks to the dense matching (20] it is easy to obtain dense matching pairs on the basic plane through the Brounded-SAM (58 as in Fig. 4(c).
Subsequently, we achieve the transformation matrix M by RANSAC [2 with 100 sampled matching pairs and apply the perspective warping for the new bottom face. Note that this perspective warping cannot be used on the top face, because the top face is just parallel to the basic plane rather than close to it. Instead, we get the new top face following the bottom landmarks with a constant height from the 2D mask. Finally, we fill the convex hull and optionally mask it with irregular brushes as in 7 dilate the 3D box masks. The masking adaption is flexible strategy, which not only locates the position of the object across different views but also incorporates human priors by manually annotating for some objects with very special shapes, such as the baseball bat in Appendix Fig. 13
4 Experiments
Datasets. MVInpainter-O is trained on the object-centric data that includes full categories of CO3D [57] and MVImgNet (95). Moreover, we regard the Omni3D [6] as the zero-shot valida- tion. MV MVInpainter-F is trained on the forward-facing data with Real10K [I03], Scannet++ [89], and DL3DV [1]. including both indoor and outdoor scenes. We further employ comparison on verify the object removal ability. To mitigate the imbalanced category distribution, we sample an equivalent subset of scenes for each category in every epoch. All images are resized and cropped into 256 2266 for both inpainting and flow extraction. More details about the dataset discussed in Appendix Sec. B.I To the best of our knowledge, MVInpainter is the first scene-level generative model that could be generalized on all categories of both CO3D and MeImgNet.
SPInNeRF [51
Dynamic Frame Sampling. To alleviate the training costs from long sequences, we first train MVIn- painter with (N + 1) = 12 frames. Then, only few steps (1/10) of fine-tuning with dynamic frame numbers from (N 1) E [8, 24] are sufficient for good frame number adaption. For the inpainting of more frames, frame interpolation-basee inpainting introduced in Appendix Sec. B.2 should be considered. Besides, we also randomly sample the frame interval to encourage generalization.
Training Setup. All trainings are accomplished on 8 A800 GPUs. We train MVInpainter-O) and MVInpainter-F for 100k and 60k steps with batch size 64, frame number 12, learning rate le-4 for days and 2 days respectively. Then we fine-tune the model with dynamic frames for 10k steps.
Metries. In this paper, we evaluate our method with PSNR, LPIPS 99] FID 27 and KID [5). We also include CLIP score (56) for the NVS task to verify the identity maintenance capability. For the object removal, we further compare the similarity of DINOv2 features (52) extracted from masked regions to evaluate the inpainting consistency, denoting average DINOv2 similarity (DINO-A) and minimal DINOv2 similarity (DINO-M) among masked patches, respectively.
4.1 Object-Centrie Results
Results of object-centric data are compared in Tab. and Fig. We considered two types of aaased
approaches, including NVS manners: ZeroNVS (44; inpainting based methods: Nerfiller [80] and LeftRefill [7]. Note that ZeroNVS requires explicit camera poses. Besides, the inpainting-baat based methods are pose-free with enlarged bounding box masks to avoid masking shape leakage. Without the inpainting formulation, the scene-level ZeroNVS struggles to directly synthesize novel views without SDS. For inpainting-based approaches, I LeftRefill can capture the scene's 's main object but fails to maintain multi-view consistency. Nerfiller only retains the shape of several views, while
7

---

MVIngNaa
Figure 5: Object-centric results CO3D, MVImgNet, and Omni3D. The first denotes the reference (first column) and other masked inputs, while other results are sampled from Lefteefil ( Nerfiller 80), ZeroNVS (64. and our MVInpainter. Please zoom-in for details.
Table I: Quantitative results of object-centric NVS with enlarged bounding box masks compared on CO3D 57 and MVImgNet (95]. We also include Omni3D (6] as zero-shot test set without being trained by any competitor. All KID results are multiplied by 100.
CO3D+MVImgNet
Omni3D (zero-shot)
ZemNVS 64
12.44
0.606
41.90
0.981
0.6028
9.38
0.627
82.81
5.421
0.5451
Nerfiller 80
18.29
0.310
36.64
0.491
0.6603
16.10
0.272
37.04
1.056
0.6279
LeftRefill 7
17.74
0.283
38.06
0.826
0.6392
17.09
0.239
27.81
0.775
0.6484
Ours
20.25
0.185
17.56
0.154
0.8182
19.19
0.153
16.40
0.345
0.7667
other views suffer from significantly inferior structures and identities. Our model outperforms all competitors with prominent achievements on both the in-domain test set and the zero-shot Omni3D.
4.2 Forward-Facing Results
Object Removal. We quantitatively verify the object removal ability of MVInpainter-F on SPInNeRF test set (5I) in the left of Tab. [] while qualitative comparisons of the train set shown in Fig. of
Appendix. Other competitors include conventional single-view based inpainting manners: LaMa
70
MAT 337, and SDD-ippainiig [59]. We further compare the pose-free reference-inpainting Left- Refill and the video inpainting manner ProPainter [102]. Formally, MAT and SID-ippainting
suffer from inconsistent inpainting results with poor DINO-A and DINO-M. Though LaMa achieves the highest DINO-M, it generates consistent blur and artifacts as in Appendix Fig. B].). For the reference-based manners, LeftRefill only conditions on the first view without multi-view consistency, while ProPainter performs inferior in synthesis quality. Since the small test set, FID would be largely degraded if one result contains artifacts. Our method enjoys the best performance LPIPS, FID, and DINO-A with consistent generations without any unstable hallucination.
Scene-Level Inpainting. We provide quantitative results of multi-view right of Tab. [ corrupted with large irregular masks. Our method achieves the best results of all metrics, prominently outperforming video-based inpainting, reference-guidee guided inpainting, and other single-view inpainting approaches.
inpainting in the
4.3 WoaldWorld 3D Scene Editing
We verify the an-the-wild scene editing ability of MVInpainter following Sec. 3.4 in Fig. 6 where the background images are from the unseen MipNeRF360 (2]. MVVnpainter-F achieves stable and consistent object removal, and MVInpainter-O performs high-quality multi-view generation for various object shapes based on the flexible mask adaption. Benefiting from the consistent results, our method enjoys reliable reconstruction with stereo-based Dust3R (76] and MVS (9] Sec. Sec.[Cof of the Appendix discusses more details about the point cloud and 3DGS reconstruction.

---

Table 2:
Quantitative results of scene-level forward-facing NVS with masks. The clean SPIn- NeRF [5I] dataset with consistent object masks used to evaluate the object removal, while the unseen scenes from Scannet++ [89], Real10k [033. and DL3DV [4]] degraded by random masks are used to verify the basic inpainting ability. All KID results are multiplied by 100.
SPInNeRF (removal)
Scannet+ Real 10K (inpainting)
PSNRT
LPIPS!
FID
DINO-At
DINO-M
PSNR
LPIPS|
FID
KIDI
LaMa
70
28.62
0.054
15.26
0.8909
0.6019
17.61
0.337
38.47
0.981
MAT 37
27.05
0.067
28.81
0.8727
0.5760
15,47
0.377
37.38
0.899
SD
DD-inpaint
59
26.98
0.070
19.32
0.8556
0.4422
13.54
0.417
38.67
1.048
LeftRefil
7
30.29
0.102
18.02
0.8931
0.5652
15.14
0.380
38.06
1.334
ProPainter
102
31.72
0.047
12.25
0.8757
0.5534
20.42
0.306
61.76
2.642
Ours
28.87
0.036
7.66
0.8972
0,5937
20.91
0,173
15.58
0.252
Table Ablation studies on CO3D. 'w.o. inp' means the baseline without the inpainting formulation.
PSNR
LPIPS)
CLIP
PSNR1
I.PPS.J.
CLIPI
Baseline
17.16
0.305
0.750
No Flow
18.64
0.250
0.796
Baseline (w.o. inp)
14.35
0.443
0.648
Dense Flow
18.53
0.247
0.792
*AnimateDiff
17.31
0.308
0.756
+Ref-KV
17.90
0.283
0.773
Slot2D Flow (time-emb)
18.74
0.244
0.798
+Object mask
18.64
0.250
Slot2D Flow (cross-attn)
18.81
0.245
0.796
FFlow emb
18.93
0.240
0.798
Slot3D Flow (cross-attm)
18.93
0.240
0.798
(a) Ablation results different proposed components
(b) Ablation of various strategies to inject flow guidance
"Sockground"
(3) Masked images
Mulli-viow objoct removal
"Sandwwirr
aduption
object
Figure 6: Results of multi view scene editing, including (b) multi-view object removal, (c) mask adaption, (d) multi-vies object insertion. More results shown in Fig. T3 and Fig. of Appendix.
Reference and masked imputs
Reference and masked inputs
priors (AnimancD).t)
With viñao pricrs (AnimetdDil)
With Ref-XV
Figure 7: Qualitative ablation studies of AnimateDiff initialization and Ref-KV CO3D.

---

4.4
Ablation Study
To verify the effectiveness of each component from MVInpainter, we conduct ablation studies on CO3D as in Tab. [] From Tab. 3al the baseline is built upon SDI.S-inpainting with random-initializz LoRA and motion transformer blocks, and we also compare the alternative baseline without the inpainting formulation built upon naive SD1.5. The multi-view inpainting formulation largely improves the results, and more details about the inpainting comparison are discussed in Appendix Sec. B.4 Besides, show qualitative ablation results in Fig. 7 which indicate that video priors from Difm and Ref-KV could substantially facilitate the structure and appearance consistency respectively. We also realize that object-level tracking masks are critical for training M IVInpainner-OO Moreover, we analyze the effect of flow grouping in Tab. 3bb Without any flow guidance, our method fails in some ambiguous cases, such large pose changes of stop signs and laptops in Fig. [2 of Appendix, while dense flow slightly hinders the identity (PSNR and CLIP). The proposed slot-attention based flow grouping outperforms the vanilla flow injection, while incorporating flow embedding by cross-attention is more lightweight with comparable performance. Further, the flow grouping can be improved with 3D temporal attention learning.
5
Conclusion
This paper proposes MVInpainter, a multi-view consistent inpainting method to expand 2D genera- tions into 3D scenes by multi-view object removal, insertion, and replacement. MVInpainter enjoys a pose-free inpainting formulation built upon the SD-inpainting backbone with motion modules. Motion initialization based on video priors and Ref-KV are presented facilitate the structure and appearance consistency respectively. Furthermore, we propose to use flow grouping based on the slot-attention to encourage implicit motion control. For the inference, we present novel mask adaption strategy to warp object masks to novel views. Sufficient experiments on both object-centric and forward-facing datasets verified the effectiveness of MV MVInpainter.
Acknowledgements. We would like to thank Yanwei Fu. Dr. Fu is with School of Data Science in Fudan, Fudan SSTBI-ZINE Algorithm Centre for Brain-inspired Intelligence, Shanghai Key Lab of Intelligent Information Processing, and Technology Innovation Center of Calligraphy and Painting Digital Generation, Ministry of Culture and Tourism, China. The computations in this research were supported by the CFFF platform of Fudan University.
References
[1] Titas Anciukevicius, Fabian Manhardt, Federico Tombari, and Paul Henderson. Denoising diffusion via image-based rendering. In International Conference on Learning Representations, 2024.
[2] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5470-5479, 2022.
[3] Edward Bartrum, Thu Nguyen-Phuoc, Chris Xie, Zhengqin Li, Numair Khan, Armen Avetisyan, Douglas Lanman, and Lei Xiao. Replaceanything'd. Text-guided 3d scene editing with compositional neural radiance fields. arXiv preprint arXiv:2401.17895, 2024.
[4] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang. Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/daalle-3. pdf. 2(3):8, 2023.
[5] Mikolaj Binikowkii, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. In International Conference on Learning Representations, 2018
[6] Garrick Brazil, Abhinav Kumar, Julian Straub, Nikhila Ravi, Justin Johnson, and Georgia Gkioxari. Omni3D: A large benchmark and model for 3D object detection in the wild. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, Vancouver, Canada, June 2023. IEEE
10

---

[7] Chenjie Cao. Yunuo Cai, Qiaole Dong. Yikai Wang, and Yanwei Fu. Leftrefill: Filling right canvas based on left reference through generalized -ia--ee--mae diffusion model. In P Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
[8] Chenjie Cao, Qiaole Dong, and Yanwei Fu. Zits++: image inpainting by improving the incremental transformer structural priors. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[9] Chenjie Cao, Xinlin Ren, and Yanwei Fu. Mysformer++:: Revealing the devil in transformer's details for multi-view stereo. In International Conference on Learning Representations, 2024.
[10] Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22560- 22570, 2023.
[11] Eric R Chan, Connor Z Lin, Matthew Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry- aware 3d generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16123-16133, 2022.
[12] Eric R Chan, Koki Nagano, Matthew A Chan, Alexander W Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, and Gordon Wetzstein. Generative novel view synthesis with 3d-aware diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4217-4224 2023.
[13] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22246-22255 22256, 2023.
[14] Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao. Anydoor: Zero-shot object-level image customization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
[15] Yiwen Chen, Zilong Chen, Chi Zhang, Feng Wang. Xiaofeng Yang. Yikai Wang, Zhongang Cai, Lei Yang, Huaping Liu, and Guosheng Lin. Gaussianeditoo: Swift and controllable 3d editing with gaussian splatting. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
[16] Jang Hyun Cho, Boris Ivanovic, Yulong Cao, Edward Schmerling, Yue Wang. Xinshuo Weng, Boyi Li, Yurong You, Philipp Krähenbühl, Yan Wang. et al. Language-image models with 3d understanding. arXiv preprint wxXiv:2405.03683 2024.
[17] Matt Deitke, Ruoshi Liu, Matthew Wallingford. Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, et al. Objaverse-xl: A universe of 10m+ 3d objects. Advances in Neural Information Processing Systems, 2023.
[18] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13142-13153, 2023.
[19] Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, Xingcheng Zhang, Yu Qiao. Dahua Lin, and Jiaqi Wang. Internim-xcome xcomposer::: Mastering free-form text-image composition and comprehension vision- Languaee large model. arXiv preprint arXiv:2401.164200 2024.
[20] Johan Edstedt, Qiyu Sun, Georg Bökman, Márten Wadenbäck, and Michael Felsberg. RoMa: Robust Dense Feature Matching. 2024.

---

[21] Patrick Esser, Sumith Kulal, Andreas Blatimann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow trans- formers for high-resolution image synthesis. arXiv preprint arXiv:2403.03206, 2024.
[22] Martin A Fischler and Robert C Bolles. Random sample consensus: paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, (4(6):311-399) 1981.
[23] Rafail Fridman, Amit Abecasis, Yoni Kasten, and Tali Dekel. Scenescape: Text-driven consistent scene generation. Advances in Neural Information Processing Systems, 36, 2023.
[24] Chen Gao, Ayush Saraf, Jia-Bin Huang, and Johannes Kopf. Flow-edge guided video com- pletion. In Computer Vision-ECCC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XII 16, pages 713-729. Springer, 2020.
[25] Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. In International Conference on Learning Representations, 2024.
[26] Ayaan Haque, Matthew Tancik, Alexei A Efros, Aleksander Holynski, and Angjoo Kanazawa. Instrett-nerf2nerff: Editing 3d scenes with instructions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19740-19750, 2023.
[27] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner. Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[28] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840 33-88006651, 2020.
[29] Lukas Höllein, Aljaž Božič, Norman Müller, David Novotny, Hung- Yu Tseng, Christian Richardt, Michael Zollhöfer, and Matthias Nießner. Viewdiff: 3d-consistent image generation with text-to-image models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
[30] Lukas Höllein, Ang Cao, Andrew Owens, Justin Johnson, and Matthias NieBner. Text2room: Extracting textured 3d meshes from 2d text-to-image models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7909-7920, 2023.
[31] Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. In International Conference on Learning Representations, 2024.
[32] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu. Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[33] Li Hu. Animate anyone: Consistent and controllable image-to-vided synthesis for character animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages $153-8163, 2024.
[34] Jindong Jiang, Fei Deng, Gautam Singh, and Sungjin Ahn. Object-centric slot diffusion. Advances in Neural Information Processing Systems, 2023.
[35] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühhler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), July 2023.
[36] Haodong Li, Weiqi Luo, and Jiwu Huang. Localization of diffusion- based inpainting in digital images. IEEE transactions on information forensics and security, 12(12):3050-300 2017.
[37] Wenbo Li, Zhe Lin, Kun Zhou, Lu Qi, Yi Wang, and Jiaya Jia. Mat: Mask-aware transformer for large hole image inpainting. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, 2022.
12

---

[38] Yuhan Li, Yishun Dou, Yue Shi, Yu Lei, Xuanhong Chen, Yi Zhang, Peng Zhou, and Bingbing Ni. Focaldreamer: Text-driven 3d editing via focal-fusion assembly. Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 3279-3287, 2024.
[39] Zhen Li, Cheng Ze Lu, Jianhua Qin, Ch Guo, and Ming-Ming Cheng. Towards an end-to-end framework for flow-guided video inpainting. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 17562-17571, 2022.
[40] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng. Xun Huang, Karsten Kreis, Sanja Fidler. Ming- Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d content creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 300-309, 2023.
[41] Lu Ling, Yichen Sheng, Zhi Tu, Wentian Zhao, Cheng Xin, Kun Wan, Lantao Yu, Qianyu Guo, Zixun Yu, Yawen Lu, et al. DI3dv-10k: A large-scale scene dataset for deep learning- -ased 3d vision. 2024.
[42] Guilin Liu, Fitsum A Reda, Kevin J Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro. Image inpainting for irregular holes using partial convolutions. In Proceedings of the European Conference on Computer Vision (ECCV), pages 85-100, 2018.
[43] Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu, and Hao Su. One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. Advances in Neural Information Processing Systems, 36, 2023.
[44] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Aero-1-to-33 Zero-shot one image 3d object. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9298-9309, 2023.
[45] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances in neural information processing systems, 33:1 1525 11538, 2020.
[46] Xiaoxiao Long, Yuan- Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjic Liu, Yuexin Ma, Song-Hai Zhang, Mare Habermann, Christian Theobalt, et al. Wonder3d: Single image to 3d using cross-domain diffusion. arXiv preprint arXiv:2310. 15008, 2023.
[47] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11461- 11471, 2022.
[48] Luke Mclas- Kyriazi, Iro Laina, Christian Rupprecht, Natalia Neverova, Andrea Vedaldi, Oran Gafni, and Filippos Kokkinos. Im-3d: Iterative multiview diffusion and reconstruction for high-quality 3d generation. arXiv preprint
2024.
[49] Ben Mildenhall, Pratul P. Srinivasan, Mathew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision (ECCV), 2020.
[50] Ashkan Mirzaei, Tristan Gumentado-Armstroog Marcus Brubaker, Jonathan Kelly, Alex Levinshtein, Konstantinos G Derpanis, and Igor Gilitschenski. Reference- controllable inpainting of neural radiance fields. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 17815-17827 2023.
[51] Ashkan Mirzaei, Tristan Aumentado-Ammsroon Konstantinos G Derpanis, Jonathan Kelly, Marcus A Brubaker, Igor Gilitschenski, and Alex Levinshtein. Spin-nerf: Multiview segmen- tation and perceptual inpainting with neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20669-20679, 2023.
[52] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po- Yao Huang, Shang-Wen Li,
13

---

Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou. Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024.
[53] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 2536-2544, 2016.
[54] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In International Conference on Learning Representations, 2024.
[55] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In International Conference on Learning Representations, 2023.
[56] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark. et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PMLR, 2021.
[57] Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In International Conference Computer Vision, 2021.
[58] Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li. Qing Jiang, and Lei Zhang. Grounded sam: Assembling open-world models for diverse visual tasks, 2024.
[59] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 10684-10695, 2022.
[60] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aber- man. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22500-22510, 2023.
[61] Tijana Ruzic and Aleksandra Pizurica. Context-aware patch-based image inpainting using markov random field modeling. IEEE transactions on image processing, 24(1):4444556, 2015,
[62] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pages 1-10, 2022.
[63] Chitwan Sabaria, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, al. Photorealistic text-to- -o-tgee- diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479-3666 36494, 2022.
[64] Kyle Sargent, Zizhang Li, Tanmay Shah, Charles Herrmann, Mong-Xing, Yu, Yunzhi Zhang, Eric Ryan Chan, Dmitry Lagun, Li Fei-Fei, Deqing Sun, et al. Zeronvs: Zero-shot 360-degree view synthesis from a single real image. arXiv preprint arXiv:2310 10. 17994, 2023.
[65] Mohamad Shahbazi, Liesbeth Claessens, Michael Niemeyer, Edo Collins, Alessio Tonioni, Luc Van Gool, and Federico Tombari. Inserf: Text-driven generative object insertion in neural 3d scenes. arXiv preprint arXiv:2401.05335 2024.
[66] Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. base model, 2023.
a single image to consistent multi-view diffusion
[67] Yichun Shi, Peng Wang, Jianglong Ye, Long Mai, Kejie Li, and Xiao Yang. Mydream: Multi-view diffusion for 3d generation. 0rXIv:2308.16512. 2023.
14

---

[68] Ka Chun Shum, Jaeyeon Kim, Binh-Son Hua, Duc Thanh Nguyen, and Sii-Kit Yeung. Language-driven object fusion into neural radiance fields with pose-conditioned dataset updates. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, 2024.
[69] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3d- cd-sstucctur-aave awaree neural scene representations. Advances in Neural Information Processing Systems, 32, 2019.
[70] Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempit- sky. Resolution-robust large mask inpainting with fourier convolutions. Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2149-2155 2022.
[71] Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. Make-it-3d: High-fidelity 3d creation from a single image with diffusion prior. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22819 22229,, 2023.
[72] Zachary Teed and Jia Deng. Raft: Recurrent all all-pairs field transforms for optical flow. In Computer Asion-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pages 402-419. Springer, 2020.
[73] Ayush Tewari, Tianwei Yin, George Cazenavette, Semon Rezchikov, Josh Tenenbaum, Frédo Durand, Bill Freeman, and Vincent Sitzmann. Diffusion with forward models: Solving stochastic inverse problems without direct supervision. Advances in Neural Information Processing Systems, 36, 2023.
[74] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017
[75] Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 12619-12629, 2023.
[76] Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
[77] Su Wang. Chitwan Saharia, Ceslee Montgomery, Jordi Pont-Tuset, Shai Noy, Stefano Pellegrini, Yasumasa Onoe, Sarah Laszlo, David. Fleet, Radu Soricut, et al. Imagen editor and edithench: Advancing and evaluating text-guided image inpainting. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 18359-183699 2023.
[78] Yikai Wang, Chenjic Cao, and Yanwei Fu. Towards stable and faithful inpainting. arXiv preprint arXiv:2312.0483/ 2023.
[79] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Pro- lifiedreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems, 36, 2023.
[80] Ethan Weber, Aleksander Holyáski, Varun Jampani, Saurabh Saxena, Noah Snavely, Abhishek Kar, and Angjoo Kanazawa. Nerfiller: Completing scenes via generative 3d inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
[81] Silvan Weder, Guillermo tow, Michael Firman, and Sara Vicente. Removing objects from neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16528-16538. 2023.
Aron Monszpart, Marc Pollefeys, Gabriel Bros-
[82] Rundi Wu, Ben Mildenhall, Philipp Henzler, Keunhong Park, Ruiqi Gao, Daniel Watson, Pratul P Srinivasan, Dor Verbin, Jonathan Barron, Ben Poole, et al. Reconfusion: 3d reconstruction with diffusion priors. 2024.
15

---

[83] Hanyu Xiang, Qin Zou, Muhammad Ali Nawaz, Xianfeng Huang, Fan Zhang, and Hongkai Yu. Deep learning for image inpainting: A survey. Pattern Recognition, 134:109046, 2023.
[84] Shaoan Xie, Zhifei Zhang, Zhe Lin, Tobias Hinz, and Kun Zhang. Smartbrush: Text and shape guided object inpainting with diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 22428 22437,, 2023.
[85] Yinghao Xu, Hao Tan, Fujun Luan, Sai Bi, Peng Wang, Jiahao Li, Zifan Shi, Kalyan Sunkavalli, Gordon Wetzstein, Zexiang Xu, al. Dmv3d: Denoising multi-view diffusion using 3d large 2023.
reconstruction model. arXiv preprint
[86] Binxin Yang. Shuyang Gu, Bo Zhang, Ting Zhang. Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen. Paint by example: Exemplar-based image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18381-18391 2023.
[87] Charig Yang. Hala Lamdouar, Erika Lu, Andrew Zisserman, and Weidi Xie. Self-supervised video object segmentation by motion grouping. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7177-7188, 2021.
[88] Jinyu Yang, Mingqi Gao, Zhe Li, Shang Gao, Fangjing Wang, and Feng Zheng. Track anything: Segment anything meets videos, 2023.
[89] Chandan Yeshwanth, Yueh-Cheng Liu, Matthias Nießner, and Angela Dai. Scannet++: A high-fidelity dataset of 3d indoor scenes. In Proceedings of the International Conference on Computer Vision, 2023.
[90] Zili Yi, Qiang Tang, Shekoofeh Azizi, Daesik Jang, and Zhan Xu. Contextual residual aggregation for ultra high-resolution image inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7508-7517, 2020.
[91] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4578-4587, 2021.
[92] Hong-Xing Yu, Haoyi Duan, Junhwa Hur, Kyle Sargent, Michael Rubinstein, William T Freeman, Forrester Cole, Deqing Sun, Noah Snavely, Jiajun Wu, and Charles Herrmann. Wonderjourney: Going from anywhere to everywhere. arXiv preprint arXiv:2312.0388/ 2023.
[93] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with contextual attention. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5505-5511 2018.
[94] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas Huang. Free-form image inpainting with gated convolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1440. 2019.
[95] Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjic Ye, Yushuang Wu, Zizheng Yan, Tianyou Liang, Guanying Chen, Shuguang Cui, and Xiaoguang Han. Mvimgnet: A large-scale dataset of multi-view images. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, 2023.
[96] Xin Yu, QI. Text-to-3d with classifier score distillation. In International Conference on Learning Representations, 2024.
Guo, Yangguang Li, Ding Liang, Song-Hai Zhang, and XIAOJUAN
[97] Frank Zhang, Yibo Zhang, Quan Zheng, Rui Ma, Wei Hua, Hujun Bao, Weiwei Xu, and Changqing Zou. 3d-scenedreamer: Text-driven 3d-consistent scene generation. arXiv preprint arXiv:2403.094330 2024.
[98] Lvmin Zhang. Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. Proceedings of the IEEF/CVE International Conference on Computer Vision, pages 3836-3847, 2023.
16

---

[99] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 586-595, 2018.
[100] Shengyu Zhao, Jonathan Cui, Yilun Sheng, Yue Dong, Xiao Liang, Eric I Chang, and Yan Xu. Large scale image completion via co-modulated generative adversarial networks. In International Conference Learning Representationss 2021.
[101] Yunhan Zhao, Connelly Barnes, Yuqian Zhou, Eli Shechtman, Sohrab Amirghodsi, and Char- less Fowlkes. Geofill: Reference-based image inpainting with better geometric understanding. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pages 1776-1786, 2023.
[102] Shangchen Zhou, Chongyi Li, Kelvin CK Chan, and Chen Change Loy. Propainter: Im- proving propagation and transformer for video inpainting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10477-10486, 2023.
[103] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo mag- nification: Learning view synthesis using multiplane images. ACM Trans. Graph. (Proc. SIGGRAPII), 2018.
[104] Yi Zhou, Hui Zhang, Hana Lee, Shuyang Sun, Pingjun Li, Yangguang Zhu, ByungIn Yoo, Xiaojuan Qi, and Jae-Joon Han. Slot-vps: Object-centric representation learning for video panoptic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3093-3103. 2022.
[105] Yugian Zhou, Connelly Barnes, Eli Shechtman, and Sohrab Amirghodsi. Transfill: Reference- guided image inpainting by merging multiple color and spatial transformations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2266 2276, 2021.
[106] Junzhe Zhu, Peiye Zhuang, and Sanmi Koyejo. Hifa:
High-fidelity text-to-3d generation with advanced diffusion guidance. In International Conference on Learning Representations, 2023.
[107] Zixin Zhu, Xuelu Feng, Dongdong Chen, Jianmin Bao, Le Wang. Yinpeng Chen, Lu Yuan, and Gang Hua. Designing a better asymmetric vegan for stablediffusion. arXiv preprint arXiv:2306.0444 04632,, 2023.
[108) Jingyu Zhuang, Chen Wang, Liang Lin, Lingjie Liu, and Guanbin Li. Dreameditor: Text-driven 3d scene editing with neural fields. In SIGGRAPH Asia 2023 Conference Papers, pages 1-10, 2023.
[109] Zi-Xin Zou, Zhipeng Yu, Yuan-Chen Chen Guo, Yangguang Li, Ding Liang, Yan-Pei Cao, and Song-Hai Zhang. Triplane meets gaussian splatting: Fast and generalizable single-view 3d reconstruction with transformers. arXiv preprint arXiv:2312.09147 2023.
17

---

A Supplementary Results
We provide more qualitative results in this section, including object removal results Fig. and Fig. 9 multi-view scene editing of Fig. 13 and Fig. [4 the generalization of the proposed mask adaption of Fig. [3t results multi-view object-level NVS in Fig. TO object replacement by T2I inpainting model and the exemplar- based AnyDoor [44] in Fig. II We further provide some qualitative ablation
studies in Fig. 12 These visualizations verify the wide application of the proposed MVInpainter.
10 6150 Sp Lrei
10 Pahhhi
(00m
Figure Object removal visualization on SPInNeRF. The first row denotes the reference (first column) and other masked inputs, while other results are sampled from the inpainted sequence.
More Details and Comparison
B.1
Dataset Details
We list detailed data information in Tab. For the object-centric training data, CO3D [57] contains 24k sequences with 51 categories, and MVImgNet (95) contains about 210k sequences with 223 categories, while 15 categories of MVImgNet are seen as the zero-shot test set. Note that we discard some images with too large foreground masks. For the forward-facing data, we choose 190 unseen scenes from Real 10K [103], Scannet++ [8), and DL3DV [4] as the mixed scene-level validation. To quantitatively verify the object removal, 10 scenes are selected from SPInNeRF [51] test set with consistent object masks and ground truth images without foregrounds. For the quantitative comparison, we sample several test views for each scene at the maximum interval with officially provided consistent object masks. Besides, from the training set with objects. All quantitative validations are based on 24 views, while qualitative results are validated with 12 and 24 views separately. To simplify the visualization, paper just includes representative views rather than showing all views.
also provide qualitative results of these scenes sampled
Table 4: Details about the training and testing datasets of MV MVInpainter.
Train
Test
Datasets
Train Sequences
Test Sequences
Dutasets
Categories Sequences Categories Sequences
Rea
[03]
13k
50
CO3D S7
51
24k
51
102
DL3DV
41
5900
100
MVImgNet 995
223
210k
238
238
Scannet++
89
340
40
Omni3D
61
SPInNeRF
M1
10
(a) Object-centric data
(b) Forward-facing data
18

---

Figure 9: Object removal results from MVInpainter-F on CO3D (rows 1, 2) and MVImgNet (rows 3,
4. 5). The first view of each sequence is inpainted by SD-inpainting with "background" caption.
B.2 Frame Interpolation
To impaint extremely long sequences, MVInpainter could first inpaint some keyframes, then apply the frame interpolation-bant inpainting to extend these keyframes to more views. Formally, we first interactively mask and inpaint among keyframes like Fig. a)). Afterward, we uniformly sample long-range inpainted results fixed conditions at left (max to 12), and then interactively inpaint other views to preserve the identity and appearance as Fig. [6(b).
B.3 Asymmetric VAE Decoder
We found some color difference near the masking boundary in some inpainting results in Fig. 17 especially for indoor cases with textureless regions. Moreover, we further found that this issue also occurred with the original SD-inpainting. Hence, follow the asymmetric VAE decoder [107] which includes unmasked image pixels and masks as additional inputs, and apply the data augmentation in (78) fine-tune the VAE decoder. Though the quantitative results are almost unchanged, the augmented new decoder enjoys much more consistent visualizations shown in Fig. 17
B.4 Detailed Ablation for Inpainting Formulation
To verify the effect of the inpainting formulation, we further detail the comparison in Tab. 5 and Fig. 18 The 'Baseline" the same the one listed in Tab. 3a denoting the backbone of SD1.5- inpainting Both 'SD-blend' and 'SD-NVS' utilize vanilla SD1.5 as the backbone without any

---

Figure 10: Object-level NVS on CO3D (groups 3), MVImgNet (groups 4 to 6), and Omni3D
(groups 7 to 9). The first row of each group contains an additional reference view.
20

---

"seafood
"bread
"Donut"
"Sheep toy
AnyDoor
Figure 11: Object replacement results edited by T21 inpainting model and AnyDoor 14
Reforence and masked Inputs
STOP
With desse fow gakaaee
grouping
Figure 12: Qualitative ablation studies of flow guidance.
21

---

Backyrouua
"Brown Bay
"Chocolate Cake"
"Holdoy"
Brown cap
"Blue Bay
Apple
Baseball
Figure 13: Scene editing results and adaptively warped masks with different captions. Object
insertions are all based on the removal results with the caption: "hackground". Zoom-in for details.
22

---

Toddy Mear's Hlaad"
Chorolate Cake""
Matdlay with Tomato Plaste"
"Oringe Show
Book
Figure 14: Scene editing results and adaptively warped masks with different captions. Object
insertions are all based on the removal results with the caption: "background". Zoom in for details.
R
Figure 15: We show the robustness of the proposed mask adaption across various real-world scenarios, comprising the textureless table, the rough lawn, and the pool with sunlight reflection. SDXL- inpainting (54) used to generate the object of the first view.
23

---

Unmroker oredtioe
credition
Newly irappllied
Figure 16: The visualization of frame interpolated object removal and insertion. (a) shows expanding results with (x4) length from 6 inpainted views. (b) denotes the long-range interpolation with fixed conditions (first 7 views).
Masked
daseder
Masbal irput
Vanilk VAR cleeader
Figure 17:
Visualization of the color difference issue with different VAE decoders.
Table 5: Ablation study of the baseline method with inpainting formulation, and without inpainting formulation (SD-blend and SD-NVS).
PSNR
I.PPS.J.
CLIP
SD-blend
14.35
0.443
0.648
SD-NVS
11.61
0.663
0.677
Baseline
17.16
0.305
0.750
Inpast
Havoline SIXNVS SDXNend
Inpu:
Baseline SD-XVS
Figure 18: Results of bascline methods with (Baseline) and without inpainting formulation (SDbblnd, SD-NVS).
24

---

inpainting fine-tuning and masking. The denoising process of 'SD-blend' is blended with unmasked regions [47] while 'Sd-NVS' is fine-tuned with the whole view synthesis with noise-free first reference view. All models mentioned above are fine-tuned with random initialized LoRAs and motion transformer blocks without other improvements (Ref-KV, AnimateDiff, flow grouping) on CO3D. Both quantitative and qualitative results show the effectiveness of the inpainting-basee baseline, which enjoys properly good pose formulation even without any pose guidance.
B.5 Full Model Fine-tuning
Table 6: Comparison of MVInpainter-O based fine-tuning only LoRAs and motion transformers
with frozen backbone and the full-model fune-tuning. KID results are multiplied by 100.
COSD+MVImgNet
Omni3D (zero-shot)
PSNRT
LPIPS!
FIDU
KID
CLIPt
PSNRT
LPIPS]
FID
KIDI
CLIP1
LoRA+motion 20.25
0.185
17.56
0.154
0.8182
19.19
0.153
16.40
0.345
0.7667
Full fine-tuning 20.76
0.181
17.51
0.134
0.8210
19.56
0.147
16.11
0.335
0.7633
We found that fine-tuning the whole model with le-5 learning rate for SD backbone on mixed CO3D and MVImgNet could further slightly facilitate the performance, verified in Tab. 6 Thanks the effective inpainting formulation, the LoRA and motion transformer based fine-tuning is visually good enough to handle our tasks. Moreover, training MVInpainter-O and MVInpainter-FF with shared SD backbone enjoys efficient and flexible rooldworld usage. Because only a few parameters would be resumed for different applications. We regard training a foundationally more powerful MVInpainter as interesting future work.
B.6 Compared to NeRF Editing
We compare MVInpainter to the NeRF Editing method, SPIn-NeRF [1 in Fig. 19 and Tab. 7 Our contributions are orthogonal to NeRF editing-based manners [51] MVInpainter focuses tackling multi-view editing with a feed-forward model, while NeRF editing devoted to reconstructing instance-level scenes with test-time optimization. NeRF editing manners require exact camera poses and costly test-time optimization for each instance (SPIn-NeRF needs about hour for each scene). Moreover, NeRF editing manners fail to substitute for method: a) NeRF editing starts with inconsistent 2D-inpainting results, which leads to blurred results as shown in Fig. 19 while our method could refer to high-quality single-view view reference without confliets. b) Although both methods enjoy good consistency, rendering-bel bbsed inpainting suffers from color difference when blended with the original images (the last row of Fig. 19 As shown in Tab. 7 our method is comparable to SPIn-NeRF in consistency (DINO-S, DINO-L) with better image quality (PSNR, LPIPS, FID) and fidelity in their official object removal test set.
Table 7: Object removal compared to SPIn-NeRe 51
PSNRT
LPIPS
FIDI
DINO-ST
DINO-LT
Ours
28,87
0.036
7.66
0.8972
0.5937
SPIn-NeRF
25.82
0.084
38.13
0.8681
0.6350
B.7 Inference Time
Table Inference time cost tested on A800 NVIDIA GPU. are resized into x56x256
The view number is 24, while all inputs
Methods
Ours
AnimateDiff"
Nerfiller
LefiRefill
DDIM steps
50
50
20
50
Time
11.5s
10.1s
32.4s
33.0s
We validate the inference time of our MVInpainter, AnimateDiff 25),
Nerfiller [80) and Leftrefill 7 in Tab. 8 Note that all methods are based on 50 steps of DDIM except for Nerfiller, which uses 20
25

---

Masked inputs
Ours (blended with original inputs)
SPIn-NeRF (no blending)
SPIn-NeRF (blended with original inputs) Figure 19: Object removal compared SPIn-NeRF 51
steps as the official setting. Our method much more faster than other inpainting manners compared to Nerfiller and Leftrefill. Nerfiller requires costly iterative updating, which is slow even with much less de-noising steps. Leftrefill can only produce one view at once time rather than jointly produce all target views ours. Our method only costs little more inference time compared to the baseline AnimateDiff (+1,4s), which is mainly used by flow grouping and Ref-KV.
C
3D Scene Reconstruction
C.1 Dense Point Cloud
Since MVInpainter achieves consistent multi-view 2D generations, we simply employ some existing 3D reconstruction methods to further extend these results to 3D scenes without sophisticated pipelines.
Dust3R. First, we leverage the pose-free Dust3R 766 to achieve dense point clouds as the initialization of 3DGS. We empirically find that 12 views are sufficient to achieve high-quality point clouds as shown in Fig. (0(a) with only few seconds. To ensure consistent initialization for 3DGS training, we align all point clouds from Dust3R with the same camera pose system of Colmap. Although the combined point clouds in Fig. 20 a) suffer a little inconsistency and outliers caused by errors from Dust3R estimation and alignment, 3DGS could eliminate them.
Multi- View Stereo (MVS). For more difficult cases, such 'room' in MipNeRF360. Dust3R fails to achieve consistent point clouds across all image pairs. Instead, we choose the SOTA learning- based MVS method, MVSFormeree [9] to reconstruct the dense point cloud with 24 input views as shown in Fig. 21 MVS also takes few seconds for the depth estimation and re-projecting correction, which is as efficient Dust3R. The success of MVS verifies that MVInpainter can produce consistent multi-view inpainting, encouraging stereo-based reconstruction.
C.2 3DGS Reconstruction
To train 3DGS, we first interpolate inpainted results of 'kitchen' in MipNeRF330 to 48 views with frame interpolation (Sec. B.2 where 42 frames are the training set; and 6 frames are the test set, while the initialized point clouds are got from Dust3R. Besides, we use only 24 inpainted views of 'room' to verify the robustness of our method with sparser input views in complicated scenes, while
26

---

(c) Test view with LPIPS Figure 20: The visualization of 3D scene reconstruction. (a) denote aligned point clouds produced by Dust3R (76]. (b) and (c) are test views of 3DGS with and without local LPIPS loss respectively. Dust3R fails achieve consistent point clouds for the challenging baseball bat. But the local LPIPS loss can handle such a messily initialized 3DGS.
(a) Point clouds
(b) Test view w/o LPIPS
(a) MVS point clouds
(b) 3DGS test views
Figure 21: The visualization of 3D reconstruction based MVS. (a) denote point clouds produced by MVSFormer++ 9 (b) show rendered 3DGS test views.
MVSFormer++ is used to provide the initialized point clouds. Different from previous works that
27

---

the naive 3DGS could be simply converged with our raw multi-view results. Following 48 we find that LPIPS loss [9 could further
require heavy SDS loss [64] 3 and dataset updates 26
68 80
alleviate the influence of slightly inconsistent appearance. However, learning 3DGS with only LPIPS
and SSIM losses as [48] is very unstable for real-world scenes' reconstruction our pilot study.
Instead, we optimize LPIPS loss only for the foreground object as:
Ladge - XL1 + (1 - Aammm + m @ Atpipa
where m is the foreground mask from SAM-tracking 88]; ④ is element-wiss multiplication; À 0.2, Appips 0.1 respectively. Thanks the consistent multi-view results, empirically find that just 5k training steps with minutes could achieve good 3DGS results as Fig. 20(b)(c) Furthermore, the local-basee LPIPS loss further facilitates the performance with clear boundaries. Specifically, for the baseball bat in Fig. 20(a), a). the final dense point cloud struggles to perfectly unify point clouds from all views, because of the challenging long shape that magnified the Dust3R stereo and mask adaption errors. However, our local LPIPS- 3DGS can largely alleviate this issue with consistent outcomes. We recommend comparing the video results of inpainted multi-view images and 3DGS in our supplementary.
D Limitations
Figure 22: Limitations of MVInpainter-F, failing to tackle the complicated 360 scene inpainting, where the background is completely different from the reference one.
Although our work enjoys good consistency and outstanding generalization compared to previous manners, some dilemmas are still retained. As shown in Fig. 22 when meeting intractable 360 scene inpainting, our method achieves good results in the foreground regions which are consistently captured by the reference view. But our method fails to recover proper consistent structures for the completely unseen backgrounds, as magnified in red boxes columns 8 and 9 in Fig, 22 This is caused by the limited respective fields of RV and constrained training capacity (frozen SD backbone and limited training data of large viewpoint changes). However, we should clarify that the Ref-KV is sufficiently effective for most scenarios as verified in our ablation study, which is much more efficient than the full attention. Scaling up the multi-view inpainting model with more powerful attention mechanisms and more high-quality data is interesting future work.
E
Broader Impacts
This paper exploited multi-view consistent inpainting based on text-to-image models. Because of their powerful generative capacity, these models would produce misinformation fake images. So sincerely remind users to pay attention to it. Besides, privacy and consent are important considerations, as generative models are often trained on large-scale data. Furthermore, generative models may perpetuate some biases according to the training data, leading to unfair outcomes. Therefore, we recommend users be responsible and inclusive while using these text-to-image generative models. Note that our method only focuses on technical aspects. Both images and pre- me-ttriine models used in this paper are all open-released.
28

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: our abstract clearly clarifies our contributions: re-formulating 3D scene editing as a pose-free multi-view inpainting task.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: have included our limitation in Sec. D
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
This paper did not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: our experiments are reproducible, and all codes and models will be open- released.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
30

---

Answer: [Yes]
Justification:
All data used in our experiments are open-released, and our codes will also be
open-released.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
mublite/gaiies/odeeebbniissss/s ionlll for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurlPS code and data submission guidelines (https
//ips.ccc/pbllc/guuiii gwwgg//ddee//e/ Cooe/eueeeeeeeeeeeeee htt.asummissioooooliioooo ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: we include detailed implementation in Sec. The full details can be seen in our codes, which would be open-released.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: instead.
As a visual generative model, we provide sufficient visualization results
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
31

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: see in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: our work conforms to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: see Sec. B
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
32

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: we have cited related works and codes.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
33

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: the paper does not release assets for now. Codes and models will be open-released soon.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
34


Reviews:
[{"review_id": "b3JPhd535w", "reviewer": "Reviewer_jssh", "summary": "The paper studies the task of multi-view consistent real-world object removal and insertion, enabled by learning a model, MVInpainter, trained to perform multi-view 2D inpainting. The paper demonstrates its effectiveness on both object-centric and scene-level datasets with the task of object removal and object insertion.", "questions": "I think the paper is overall of good quality and tackles an important task. Some questions (as listed) could be explained for a better understanding of the method.", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The task that the paper addresses is important yet less explored. Learning strong priors for scene level multi-view consistent editing can serve as a basis model for various 3D applications.\n\nThe method introduces motion priors and optical flow to guide the generation, which seems to be interesting and effective with various quantitative and qualitative evaluations.\n\nThe presentation is good and the paper is overall easy to read.", "weaknesses": "1.Though the paper tackles the setting of video input (consecutive frames to obtain the motion prior), I am curious how the method performs with just multi-view inputs? Does the performance significantly drop? If the model can be applied to those as well, it could potentially lead to bigger impact and be applied to more real-world scenarios.\n\n2.How are the priors different from AnimateDiff and Flow Grouping? Why are they complementary?\n\n3.Could the authors please explain why the proposed method cannot be compared to SPIn-NeRF-type methods? Is it because of camera poses? If so, I am still curious to see how large the gap is? If the method is close to the methods with camera poses, it would make the paper stronger. \n\n4.One limitation is that the method still works on images with a relative simple background, as pictured in the supplementary materials Figure 19.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1. How does the method perform with unordered multi-view inputs?**\n\nThanks. Although our work mainly focuses on the sequential multi-view inputs to better leverage the video prior from video components, it can also achieve comparable performance with unordered inputs as exploratively verified in Table1 and Table2 (rebuttal pdf). \nUnordered inputs still enjoy competitive FID, DINO, and CLIP scores for object-centric novel view synthesis (NVS) and object removal, indicating good image quality and consistency. The proposed flow grouping and Ref-KV also strengthen the generated direction and appearance for unordered inputs, preserving stable PSNR. However, unordered NVS suffers from some structural distortions caused by irregular viewpoint changes as shown in Figure 1 of the rebuttal pdf. Thus, we would still clarify that using ordered images with video prior is a more efficient way to model the multi-view inpainting (Line 171-173). Maybe further fine-tuning our model with unordered images could alleviate this issue, which is beyond the scope of this paper.\n\n**Table1: Unordered generative results of object removal**\n\n|                       | PSNR ↑ | LPIPS ↓ | FID ↓ | DINO-S ↑ | DINO-L ↑ |\n|-----------------------|--------|---------|-------|----------|----------|\n| Ours (ordered)    | **28.87** | **0.036** | **7.66** | **0.8972** | **0.5937** |\n| Ours (unordered)      | 28.57  | 0.039   | 9.76  | 0.8943   | 0.5752   |\n\n**Table2: Unordered generative results of object-centric NVS**\n\n|                       | PSNR ↑ | LPIPS ↓ | FID ↓ | CLIP ↑   |\n|-----------------------|--------|---------|-------|----------|\n| Ours (ordered)    | **20.25** | **0.185** | 17.56 | **0.8182** |\n| Ours (unordered)      | 19.29  | 0.233   | **17.37** | 0.8173   |\n\n**W2. How are priors different from AnimateDiff and Flow Grouping? Why are they complementary?**\n\nThanks. The priors from AnimateDiff and flow grouping serve different but complementary purposes in our method. The video prior from AnimateDiff enhances structure consistency in the generated outputs (Line 307-308 and Figure 6 of the paper). This helps maintain a coherent structure across frames. Flow Grouping improves pose controllability (Line 191-193 and shown in Figure 11 of the Appendix). This component ensures that the synthesized poses are accurate and aligned with the intended viewpoint changes from unmasked regions. For the NVS inpainting, objects are consistently masked, leading to potential ambiguities in some scenes. These ambiguities can cause errors in pose generation, but video models alone struggle to address them (Figure 11). Quantitative results further verify this point (Table 3).\n\n**W3. Why not compare to SPIn-NeRF-type methods? Because of the requirement of camera?**\n\nThanks for this good point. Our contributions are orthogonal to NeRF editing-based manners (SPIn-NeRF[1]). \nMVInpainter focuses on tackling multi-view editing with a feed-forward model, while NeRF editing is devoted to reconstructing instance-level scenes with test-time optimization. Except for the exact camera poses, NeRF editing manners require costly test-time optimization (Line 109-112) for each instance (SPIn-NeRF needs about 1 hour for each scene).\nMoreover, NeRF editing manners fail to substitute for our method as evaluated in Figure 2 and Table 3 (rebuttal pdf).\na) NeRF editing starts with inconsistent 2D-inpainting results, which leads to blurred results as shown in Figure 2 (rebuttal pdf), while our method could refer to a high-quality single-view reference without conflicts.\nb) Although both methods enjoy good consistency, rendering-based inpainting suffers from color difference when blended with the original images (the last row of Figure 2 (rebuttal pdf)).\nAs shown in Table 3 (rebuttal pdf), our method is comparable to SPIn-NeRF in consistency (DINO-S, DINO-L) with better image quality (PSNR, LPIPS, FID) and fidelity in their official object removal test set.\nNote that our method is orthogonal to instance-level manners and can further boost their performance with much better inpainting initialization. \nFor example, experiments in Section C of the Appendix show that our method could be easily integrated with 3DGS to achieve consistent 3D outputs.\nWe would add these discussions to the paper, if accepted.\n\n**Table3: Object removal compared to SPIn-NeRF**\n\n|             | PSNR ↑    | LPIPS ↓  | FID ↓    | DINO-S ↑    | DINO-L ↑    |\n|-------------|-----------|----------|----------|-------------|-------------|\n| Ours    | **28.87** | **0.036**| **7.66** | **0.8972**  | 0.5937      |\n| SPInNeRF| 25.82     | 0.084    | 38.13    | 0.8681      | **0.6350**  |\n\n[1] Ashkan Mirzaei, et al. Spin-nerf: Multiview segmentation and perceptual inpainting with neural radiance fields, CVPR2023.\n\n**W4. Limitation of complex background.**\n\nThanks. It is important to note that all methods have certain constraints. Our approach is currently limited by the capabilities of StableDiffusion and AnimateDiff, making it challenging to inpaint 360° complex backgrounds that are out of reference view.\nHowever, our method is effective enough for most multi-view scene editing. We acknowledge that scaling up our model to a foundational video inpainting model would be interesting future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nMGnecS2w0", "reviewer": "Reviewer_VJEq", "summary": "This paper proposes a new 3D editing method by regarding it as a multi-view 2D inpainting task.  It ensures cross-view consistency through video priors and concatenated reference key/value attention and controls camera movement without explicit poses using slot attention. It shows effectiveness in object removal, insertion, and replacement.", "questions": "How is the generalizability? What is the performance in completely OOD cases?", "limitations": "the authors adequately addressed the limitations.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The slot-attention-based flow grouping has a novel and reasonable design. As demonstrated in various video/multi-view models, self-attention-based key and value concatenation is effective in ensuring view consistency. Overall, this paper proposes a well-structured recipe for multi-view editing.\n\n- The strategies used in the proposed method and their justifications are well explained in detail.\n\n- The video results and quantitative results demonstrate good performance.", "weaknesses": "- Some assumptions are heuristic, such as in L232. While these assumptions may be reasonable in most cases, they might not be robust across diverse user cases.\n- Overall, the figures and flow of the paper are complicated. It would be beneficial if the content of the figures were made more comprehensive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1. Heuristic assumption of mask adaption?**\n\nThanks for this point. We should clarify that the planar assumption of mask adaption in Line 232 is reasonable and straightforward to implement across diverse real-world cases.\nFirst, this strategy roughly decides the mask location rather than providing exact mask shapes, which would be further irregularly masked for better generalization (Line 241-242).\nSecond, the mask adaptation proves to be robust for the 'bottom face' of targets with various shapes, as verified in Figures 12 and 13 of the Appendix. Almost all objects have explicit or approximate 'bottom faces' which makes this assumption widely applicable.\nFor the `basic plane', the key ability is the dense matching method.\nWe find that the dense matching (RoMa [1]) used in our work can be generalized to various wild backgrounds (textureless table, rough lawn, and pool with sunlight reflection) as verified in Figure 4 (rebuttal pdf), denoting the robustness of mask adaption. \n\n[1] Johan Edstedt, et al. RoMa: Robust Dense Feature Matching. CVPR2024.\n\n**W2. Complicated figures and flow? Needing comprehensive content of figures.**\n\nThanks for this good advice! \nIndeed, finishing a 3D editing is an inherently non-trivial task, including sophisticated pipelines as seen in many pioneering works. \nIn this paper, we simplify the overall pipeline into the following steps: \n\n(1) 2D inpainting → (2) multi-view removal (optional) → (3) multi-view synthesis/insertion → (4) 3D reconstruction (optional).\n\nThese steps are illustrated in Figure 3 and detailed in the inference pipeline in Section 3.4.\nOur work focuses on steps (2) and (3), which are addressed by MVInpainter-F and MVInpainter-O respectively (Figure 2a).\nWe agree with the reviewer that providing a more comprehensive overview figure between Figures 2 and 3 could make readers understand our paper better. We design a new overview shown in Figure 5 (rebuttal pdf), which clearly indicates the steps that are the main focus of our paper and identify the models specifically designed for each step. \nWe will re-organize these points, if accepted.\n\n**Q1. Generalizability for OOD cases.**\n\nThanks. Benefiting from the T2I prior from StableDiffusion, our method enjoys the capacity to tackle OOD cases, which have been verified in our submission.\nFormally, 15 categories of MVImgNet are unseen test sets (Line 678). We further evaluate the OOD capability of MVInpainter in the zero-shot Omni3D (Table 1, Figures 4 and 9 of the paper).\nWe show some qualitative NVS results of OOD categories in Figure 3 (rebuttal pdf).\nMoreover, our method could also be used for unseen toys generated from exemplar-based synthesis[2] as verified in Figure 10.\nConsidering the difficulty of scene-level NVS, it is challenging to properly address some completely OOD cases, such as human bodies, which are limited by the capacity of SD and AnimateDiff, as well as the restricted in-the-wild multi-view datasets (CO3Dv2+MVImgNet). \nDespite these limitations, MVInpainter is, to the best of our knowledge, the most robust reference image-guided method that generalizes well to in-the-wild object NVS across various categories (Lines 253-254).\nWe would discuss this in the revision, and consider scaling up our model with more diverse training data to develop a foundational model as interesting future work.\n\n[2] Xi Chen, et al. Anydoor: Zero-shot object-level image customization. CVPR2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r93uxO0JLf", "reviewer": "Reviewer_n43z", "summary": "The paper formulates the 3D object editing task as a multi-view 2D in-painting task. Firstly, MVInpainter-F is employed to remove the object and obtain the background scene. Then, MVInpainter-O generates multi-view images based on the reference view.", "questions": "1. During training, the flow is obtained using RAFT and masked. In the inference stage of MVInpainter-F, is the flow only obtained from the masked image? Does the utilization of this low-quality flow negatively impact the results? \n2. How can we ensure that mask adaption remains robust when dealing with different masks and backgrounds?", "limitations": "The author provides a detailed discussion of limitations and broader impacts in the supplementary material.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Solving the 3D object editing task as multi-view generation is interesting and innovative.\n2. The thoughtful selection of different training datasets (indoor scene/object-centric) for training MVInpainter-F and MVInpainter-O effectively decomposes the object replacement task.\n3. The extensive experiments, including the generation of long sequences and the 3D scene reconstruction using the generated multi-view images, further demonstrate the 3D consistency of the resulting outputs.", "weaknesses": "It is uncertain whether the design of mask adaption is robust enough.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1, Q2. Whether mask adaption is enough robust? How to deal with different masks and backgrounds?**\n\nThanks for this good point. As long as the 'basic plane' where the object is placed can be approximated to a plane, the proposed mask adaption is robust enough with the theoretical basis of perspective transformation. We have shown the effectiveness of the mask adaption with various object shapes in Figures 12 and 13 of the Appendix. \nTo further clarify the robustness, we provide more examples with complicated backgrounds in the wild shown in Figure 4 (rebuttal pdf), including textureless table, rough lawn, and pool with sunlight reflection.\nWe find that the dense matching (RoMa[1]) used in our work enjoys good generalization on various backgrounds. Importantly, most backgrounds can be approximated to the 'basic plane' as claimed in Lines 231-233.\nWe would add these results to the paper.\n\n[1] Johan Edstedt, et al. RoMa: Robust Dense Feature Matching. CVPR2024.\n\n**Q1. Do low-quality flows obtained from masked images negatively impact the result?**\n\nThanks. We have discussed this point in the Line 194 (footnote of page 5). \nWe extract flows before masking because foregrounds largely benefit the flow quality. Note that we process flows with 5-pixel dilated masks to avoid any leakage. \nGiven that optical flows are typically low-level local features, we have not observed any conflicts when using masked flows as guidance. Furthermore, the quantitative results of dense flow presented in Table 3(b) verify that no leakage occurs (dense flows did not improve the NVS results), ensuring the integrity of quantitative results.\nWe would clarify about the detailed operation of flow masking (mask dilation) in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L9YJNbA4ez", "reviewer": "Reviewer_NjjA", "summary": "This paper proposes a new method for multi-view consistent inpainting. The input includes a video and a sequence of masks. The first image could be manipulated by any 2D editing approach, while the remaining images need to be inpainted using the proposed method. The method is built upon the SD1.5-inpainting model and fine-tuned with two LoRAs: one for object-centric and one for scene-level, using different datasets. It employs motion priors by fine-tuning the domain-adapted LoRA and temporal transformers from AnimateDiff, pre-trained on a video dataset. The technique of reference key & value concatenation from previous works is used to enhance consistency with the reference image. Flow features are extracted and added to the UNet as additional conditions. During inference, a heuristic strategy is proposed to warp and propagate the mask from the first frame to all subsequent frames.", "questions": "1. Have you considered directly fine-tuning a video inpainting model instead of using SD1.5-inpainting?\n\n2. Line 186: The sentence \"which always captures the unmasked reference latent without noise (Eq. 1), thus it is unnecessary to re-scale the latent before adding noise from another U-Net\" is unclear to me. Could you please clarify?\n\n3. Line 207: The sentence \"except that the former should be normalized in the query dimension, while the latter is normalized in the key dimension\" is unclear. Could you elaborate?\n\n4. Line 212: The phrase \"with comparable performance and fewer trainable weights\" needs clarification. Which two setups are being compared?\n\n5. Could you provide more details about the 3D attention mechanism?\n\n6. Line 309: In the context of not having object-level tracking masks, how do you obtain the mask?", "limitations": "Yes, discussed in supplementary.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The task of propagating a single 2D edited image to generate a consistent video is interesting.\n\n2. The overall pipeline is well-motivated, built upon the SD1.5-inpainting base model, fine-tuning two LoRAs on different datasets, and including temporal modules, flow features, and reference attention techniques.\n\n3. The proposed method doesn't require camera pose as input and has a short inference time.\n\n4. The paper evaluates the method on several datasets with various tasks and also provides some ablation studies.\n\n5. It's interesting to know that the dense feature is less effective than the slot embeddings.", "weaknesses": "1. While the title contains \"to Bridge 2D and 3D Editing,\" the proposed techniques are primarily designed for 2D images or videos, with no explicit 3D representation used or 3D output generated. It would be interesting to explore whether 3D output can be extracted from the resulting image sequence. Otherwise, it might be better to change the title \"3D editing\" (and other mentions in the text) to something less misleading.\n\n2. The utilization and fine-tuning of domain-adapted LoRA and temporal transformers from video models are intuitive and straightforward, but there do not appear to be any insightful design choices, limiting the novelty.\n\n3. The technique of \"Reference Key & Value Concatenation\" has been used in many previous works. We fail to see much novelty here.\n\n4. While the section introducing the techniques is generally clear, the concrete settings and details for training, inference, and evaluation are very hard to follow. I tried my best to understand the setup but still feel confused about many details, which is quite frustrating. The paper needs to be rewritten in a more well-organized way, introducing all necessary settings and details. Specifically, the following points need clarification:\n\n    a) During inference, you employ 2D models to generate reference images for different tasks (e.g., object removal/editing/replacement). However, what is the input-output pair during training? For example, do you have specific ad-hoc training pairs for the object removal task? \n\n    b) The experiment contains various setups, and the concrete settings are confusing. For each setup, it would be better to explicitly mention the task (e.g., inpainting with object/random masks, object generation, object removal, object replacement), dataset, the input, ground truth output, and how the **mask** are generated. Currently, it only says \"Object-Centric Results,\" \"Forward-Facing Results,\" and \"Real-World 3D Scene Editing.\" But the concrete tasks are unclear. For example, what is the difference between \"Object Removal\" and \"Scene-Level Inpainting\"? I assume \"object removal\" is also achieved by inpainting? If the setup for quantitative and qualitative experiments is different, please also mention it. For example, there are some qualitative results for \"object removal,\" but it should not be evaluated quantitatively (no ground truth)?\n\n5. In equation (1), you concatenated both the masked latents and unmasked noised latents. Some related questions:\n\n    a) The 9-channel input differs from the original input of SD1.5-inpainting. How do you achieve that?\n\n    b) The noise-free latent is currently masked after VAE encoding. Will this cause leakage? It might be better to mask before VAE encoding.\n\n6. What is the difference between the \"Masking strategy\" (line 163) and \"masking adaptation\" (line 228)? Is one for training and one for inference?\n\n7. The method extracts flows before masking. This may be acceptable for the tasks of removal and replacement, but will it cause leakage for the task of general NVS and inpainting, considering the flow model may have seen the ground truth image?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We rearrange and classify similar questions together. \n\n**W1. No 3D representation? Tuning down \"3D editing\" in title.**\n\nThanks. \nFirst, we would like to clarify that our paper has included a detailed discussion on the explicit 3D representation (3DGS) within our method in Section C of Appendix (Lines 727-762). Additionally, we have provided videos rendered using 3DGS in supplementary.\nSecond, the primary contribution of our work is to 'bridge' the gap between 2D and 3D editing, rather than focusing exclusively on either 2D or 3D synthesis. This bridging is fundamentally about achieving multi-view consistent inpainting (Lines 45-46), which MVInpainter effectively addresses.\nBenefiting from the consistent inpainting, our method enjoys reliable 3D representation (Lines 298-299).\nWhile the key contribution of our work lies in multi-view inpainting, discussion on 3D representation holds a lower priority.\nTherefore, we provided them in the Appendix.\nWe would further clarify this point in revision.\n\n**W2, W3. Limited novelty of LoRA, temporal transformers, and Ref-KV?**\n\nThanks. While we understand your perspective, we respectfully disagree. \nThe primary novelty of our paper lies not in the specific model design, but in the innovative task formulation that synergistically leverages video priors, optical flow, and the inpainting formulation to simplify the complex novel view synthesis (NVS) into a multi-view inpainting task (Lines 44-46). \nWe believe that the novel model design is not the only way to advance AIGC development. An effective new formulation can significantly reduce task complexity (Lines 122-123), thereby also providing significant novelty and insights to the community.\nBesides, except for video inpainting modules, we also propose other novel and effective components, including flow grouping and mask adaption.\n\n**Q2. Clarifying for Line 186: \"which always captures unmasked ... noise from another U-Net\".**\n\nThanks.  Line 186 details the differences between our usage of Ref-KV and previous works [Hu L et al,CVPR2024;Ruoxi Shi et al,2023]. We apologize for the incorrect citation here; AnyDoor[Chen, Xi, et al,CVPR2024] should be AnimateAnyone[Hu L et al,CVPR2024].\nFormally, AnimateAnyone used another diffusion U-Net to encode clear reference views (noise-free), while Zero123++ re-scaled the latent inputs by multiplying a constant to the reference latent (5x) to strengthen the effect of the reference view.\nIn our work, no additional modules or adjustments are necessary to encode the reference latent. This is because the first reference frame in our inpainting model is completely mask-free, providing straightforward guidance when concatenated as the unmasked input (Lines 185-187).\nWe will provide further clarification on these points and revise the citation.\n\n**W4. a) Setting details of tasks, datasets, and masks.**\nThanks. We have introduced the inference pipeline in Section 3.4 and Figure 3(a) of the main paper. The overall tasks of this paper are clear, i.e., multi-view object removal (MVInpainter-F) and object insertion (NVS by MVInpainter-O), while the replacement is just the combination of these two tasks as in Figure 3(a).\nWe introduced all input-output pairs for NVS and object removal in Lines 142-156, Figure 2(a) of the paper, which is further detailed in Table 4 of Appendix. NVS is trained on object-centric data (MVInpainter-O), while object removal is trained on face-forwarding data (MVInpainter-F). Masks are discussed in Lines 163-169.\n\n**Ad-hoc pairs for object removal?**\nNo ad-hoc training pairs are needed for object removal. \nBased on the common conclusion of previous inpainting works, training on the scene-level data with random masks is sufficient to learn the object removal ability. \nWe will further improve the task definition and data setting.\n\n**Q6. How to obtain training masks without object masks?**\nWithout the object-level tracking masks, we only use hybrid random inpainting masks (Lines 163-164) to train MVInpainter-O.\nNote that we have discussed that both MVInpainter-O and MVInpainter-O adopt random inpainting masks, while we additionally employed object masks to MVInpainter-O (Lines 164-165). \n\n**b) Various setups of experiments, Difference between Object Removal and Scene-Level Inpainting.**\n\nThanks. All sections are used to evaluate MVInpainter-O and MVInpainter-F separately.\nFormally, Section 4.1 (Object-Centric Results) is dedicated to evaluating the NVS performance of MVInpainter-O trained on object-centric data; Section 4.2 (Forward-Facing Results) is dedicated to evaluating the performance of MVInpainter-F trained on forward-facing data; Section 4.3 (Real-World 3D Scene Editing) shows the result of the combination of object removal (MVInpainter-F), NVS (MVInpainter-O) and mask adaption (Lines 296-298).\nWe further divide Section 4.2 into 'Object Removal' and 'Scene-Level Inpainting' to evaluate the abilities to remove objects with object masks (Line 279) and inpaint images with random masks (Line 291). So 'Object Removal' and 'Scene-Level Inpainting' are both tested with MVInpainter-F but with different mask types and test sets. \nWe also detailed the settings of datasets (Lines 246-254, Lines 675-687) and metrics (Lines 263-267) in the paper.\n\n**c) Setup for quantitative and qualitative experiments, especially for object removal.**\n\nThanks. All quantitative and qualitative experiments are conducted on the same dataset except for the object removal (Line 279-280), which were discussed in paper.\nQuantitative results of object removal are based on the **test set** of SPInNeRF without foregrounds which can be regarded as GT (Line 279), while qualitative results of object removal are evaluated on the **train set** of SPIn-NeRF with foregrounds (no GT, Line 280). More details are discussed in Lines 681-684 of Appendix.\n\n**Limited by the rebuttal context, other concerns from reviewer NjjA are answered in the global rebuttal (at the top of this page).**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate the valuable comments from all reviewers. We thank the positive comments of 'interesting and effective', 'novel and reasonable design' of slot-attention based flow grouping (jssh, VJEq); 'important yet less explored', 'interesting and innovative' of multi-view generation (n43z, NjjA). We address remained concerns for all reviewers. While our experiments are solid enough, we further provide some supplementary results in the rebuttal PDF to make our claims more convincing. Please refer to the rebuttal PDF for more details and results.

Here we further address remaining concerns from reviewer NjjA.

**W5. Questions about inpainting form (Eq1). 
a) 9-channel input differs from SD1.5-inpainting?**
Thanks. Our model is based on the SD1.5-inpainting (Lines 124-125), which originally contains 9-ch input, including noised latent (4-ch), mask (1-ch), and masked latent (4-ch).

**b) Why is noise-free latent masked after VAE encoding?**
Thanks for careful reading. Sorry for the confusing presentation of Eq(1). The masking is done before VAE encoding. Here we just want to denote the masked latent. We would revise Eq(1) in the revision.

**W6. Difference between 'masking strategy' (line 163) and 'masking adaption' (line 228)?**
Thanks. Yes, the masking strategy in Line 163 is introduced for training, while the masking adaptation in Line 228 is introduced for inference with good generalization as verified in Figures 12, 13, and Figure 4 (rebuttal pdf).

**W7. Leakage of flows in NVS?**

Thanks for this good point. 
We clarify that the flow used in our work does not leak information for NVS and inpainting:
1) As in the footnote of page 5, our method involves extracting flows from unmasked images and then applying masks to them. To further prevent leakage, we dilate flow masks by 5 pixels.
Given that optical flows are typically low-level local features, it is challenging for them to carry masked clues to unmasked regions. No conflicts are observed when we use masked flows as guidance for removal and insertion.
2) The proposed method leverages flow grouping with slot-attention to extract high-level motion from flow features (Line 200). These features primarily capture rough pose directions in unmasked regions rather than detailed information, making it difficult to leak masked info.
3) Importantly, quantitative results of dense flow injection in Table 3b indicate that simply adding flow features does not improve NVS quality. If there were any potential leakages in these masked flows, NVS results would be rapidly improved, which is not observed.

We will include detailed explanations of flow masking operations, including mask dilation in the revision.

**Q1. Fine-tuning a video inpainting model instead of SD1.5-inpainting?**

Thanks. We agree that fine-tuning a foundational video inpainting model is a promising direction to strengthen our work, which could be regarded as future work. 
Unfortunately, to the best of our knowledge, there are currently no open-released video inpainting models with sufficient capacity to address the NVS task mentioned in this paper. Existing video inpainting models fail to tackle our tasks (Lines 82-83).
On the other hand, fine-tuning a foundational video like SVD into a video inpainting model requires substantial computational resources and data, which is beyond the scope of this paper.
However, our approach, which unifies 2D-inpainting and AnimateDiff, proves both efficient and effective with good convergence (Lines 175-176).
Moreover, we believe that once a foundational video inpainting model becomes publicly available, MVInpainter can be seamlessly integrated into this new model, potentially yielding better performance.

**Q2 is answered in the previous rebuttal page.**

**Q3. Why slot-attention is normalized in query dimension?**

Thanks. Line 207 clarifies the difference between slot attention[1] and vanilla cross-attention.
Given $K$ query (slot) features $\mathbf{Q}\in\mathbb{R}^{K\times d}$ and key features $\mathbf{K}\in\mathbb{R}^{HW\times d}$ with different length $K$ and $HW$, we can achieve the attention matrix: $\mathbf{Q}\mathbf{K}^T\in\mathbb{R}^{K\times HW}$.
Following official implementation of slot attention[1], the softmax normalization is applied along the attention matrix over $K$ slot dimension. This ensures that attention coefficients sum to one for each slot query. This is why we mentioned that slot attention (called 'former' of Line 207) normalizes in the query dimension.
In contrast, the vanilla cross-attention (called 'latter' of Line 207) normalizes along the $HW$ key dimension.
According to [1], such normalization in the slot dimension improves the stability. We recommend referring to [1] for more details.
We will clarify this point in the revision.

[1] Object-centric learning with slot attention. NeurIPS2020.

**Q4. Line 212: Which two setups are compared?**

Thanks. We have compared different ways to inject flow grouping into our model in Table 3b. 
The way 'cross-attn' (Line 212) performs similar to 'time-emb' (injecting features through trainable AdaNorm before conv and attn), while the latter needs to train more weights for AdaNorm layers. We will revise it to 'ada-norm-emb' for a clearer presentation.

**Q5. More details about 3D attention mechanism.**

Thanks. The 3D attention mechanism refers to 'temporal 3D attention for the flow grouping' in Line 213.
In Figure 2(c), each flow grouping block contains a spatial-attention layer and a temporal-attention layer. This design is similar to the interleaved spatial-temporal attention layers used in video models. Since masked flow features contain rich temporal information, it is intuitive to extend slot-attention into 3D (spatial-temporal) receptive fields, making slot features carry more general information across all sequential views (Line 213-215). Quantitative results (Table 3b) also verified this. We will further clarify this.

**Q6 is answered in the previous rebuttal page.**

Meta-review: All reviewers acknowledged the novelty of this paper, as well as the solid and thorough experiments conducted. In the initial reviews, there were suggestions to further demonstrate the superiority of the proposed method from various perspectives. Especially, one reviewer raised several questions, including about explicit 3D representations based on our multi-view synthesis and technical details. During rebuttal, the authors tried to clarify the contribution of this paper and technical details. AC has also carefully reviewed the paper, the reviews, and the rebuttal, and concurs with most reviewers' opinions, such as the innovative conceptual contribution. Although some technical details could lead the misleading, AC thinks that these could be addressed in the final version. Therefore, AC recommends acceptance. It would be greatly beneficial to include the additional results from the rebuttal phase in the final version.
Decision: Accept (poster)