MVInpainter: Learning Multi-View Consistent Inpainting to Bridge 2D and 3D Editing

Authors: chenjie  cao
chaohui  yu
fan  wang
xiangyang  xue
yanwei  fu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-07 09:55:23

Abstract: Novel View Synthesis (NVS) and 3D generation have recently achieved prominent improvements. However, these works mainly focus on confined categories or synthetic 3D assets, which are discouraged from generalizing to challenging in-the-wild scenes and fail to be employed with 2D synthesis directly. Moreover, these methods heavily depended on camera poses, limiting their real-world applications. 
To overcome these issues, we propose MVInpainter, re-formulating the 3D editing as a multi-view 2D inpainting task. Specifically, MVInpainter partially inpaints multi-view images with the reference guidance rather than intractably generating an entirely novel view from scratch, which largely simplifies the difficulty of in-the-wild NVS and leverages unmasked clues instead of explicit pose conditions. To ensure cross-view consistency, MVInpainter is enhanced by video priors from motion components and appearance guidance from concatenated reference key\&value attention. Furthermore, MVInpainter incorporates slot attention to aggregate high-level optical flow features from unmasked regions to control the camera movement with pose-free training and inference. Sufficient scene-level experiments on both object-centric and forward-facing datasets verify the effectiveness of MVInpainter, including diverse tasks, such as multi-view object removal, synthesis, insertion, and replacement. The project page is https://ewrfcas.github.io/MVInpainter/.

Main paper: 
 introduction: 
 This paper studies editing 3D scenes by expanding one or few 2D manipulated references to other observed views. Particularly, with the development of diffusion-based text-to-image (T2I) models [59, 63, 4, 54] , we have seen substantial success in novel view synthesis (NVS) [23, 30, 97] , 3D generation [55, 40, 79, 96, 44, 67, 43, 66] , and controllable generation [60, 98, 10, 14] . But most existing synthesis methods [14, 98, 10] have only been proven useful in 2D scenarios. It is intuitive to extend these pioneering methods to multi-view scenarios to bridge the gap between 2D and 3D editing. This raises the question: how to make a unified framework to generate vivid multi-view foreground objects seamlessly integrated with their surroundings? Despite the achievements in NVS and 3D generation, achieving multi-view consistent scene editing by inserting, removing, replacing objects like Fig. 1 still present challenges. 1) 3D object generation struggles to generalize to scene-level editing. Most object-centric 3D generation methods [44, 67, 31] are trained on large-scale synthetic datasets [18, 17, 31] with simplistic backgrounds, neglecting real-world factors like illumination and shadows, which are essential for natural editing. While some methods integrate predefined 3D assets into NeRFs [65] and 3D Gaussian Splatting (3DGS) [15] , they still struggle with blending foreground and background elements seamlessly. 2) NVS methods have Given one inpainted or edited reference image, MVInpainter spreads it to other masked views without pose conditions. (d) MVInpainter could be applied to real-world 3D scene editing for dense point clouds by Dust3R [76] or Multi-View Stereo (MVS) [9] and 3DGS [35] with consistent generation. difficulty generalizing across various categories. Even when enhanced by diffusion models, existing NVS-based methods [12, 73, 29, 1] only work in specific scenarios and often fail to generalize to diverse or unseen categories in scene data. 3) Instance-level 3D editing is time-consuming. Approaches like instance-level 3D editing [26, 68, 3] and warp-and-inpaint NVS [23, 30, 97] integrate priors from single-view T2I models [59] , requiring costly dataset updates to address multi-view inconsistency. 4) Heavy reliance on explicit camera poses. All methods mentioned rely on accurate camera poses for both training and inference, limiting scalability with pose-free data and broader applicability, particularly in scenarios like short video editing where detailed poses may be unavailable.To this end, we present a novel perspective to enable 3D editing with a multi-view consistent inpainting manner. By starting with an edited 2D reference image from any 2D generative models [98, 14] and applying our method to masked sequences from other viewpoints, we achieve consistent multi-view inpainting results, effectively extending 2D generation into 3D scenarios. The key idea behind this inpainting approach is to focus on seamlessly synthesizing local regions rather than generating entirely new views. Many contextual cues like illumination, shadow, and camera motions are implicitly captured in unmasked regions, which can be activated by foundational T2I models. Thus, our method provides an end-to-end multi-view synthesis solution without requiring test-time optimization or explicit pose conditions, opening up possibilities for both 2D and 3D editing.Formally, this paper introduces MVInpainter, a multi-view consistent inpainting model built upon a pre-trained StableDiffusion (SD) inpainting model [59] . We incorporate domain adapters and motion modules [25] to MVInpainter as video priors for multi-view consistent structures. Moreover, to encourage appearance consistency, we propose the Reference Key&Value concatenation (Ref-KV), spatially concatenating the key and value features from the reference view to target ones in self-attention modules. Furthermore, MVInpainter operates without explicit poses, utilizing slotattention mechanisms [45, 87] of encoding and grouping motion priors from unmasked surroundings' optical flow for implicitly pose control. We train two MVInpainters sharing the same pre-trained SD backbone in object-centric (CO3D [57] , MVImgNet [95] ) and forward-facing (Scannet++ [89] , Real10k [103] , DL3DV [41] ) scenes, respectively. Experiments on unseen scenes and zero-shot datasets as Omni3D [6] and SPInNeRF [51] show the efficacy of MVInpainter in various applications, such as multi-view object removal, synthesis, insertion, and replacement.In summary, 1) we present MVInpainter as the first multi-view consistent inpainting model to bridge 2D and 3D scene editing. 2) MVInpainter is a pose-free end-to-end approach with high-level flowbased motion control from unmasked regions. 3) MVInpainter largely simplifies the NVS difficulty, which can be generalized to all categories of in-the-wild CO3D, MVImgNet, and Omni3D datasets, as well as complicated forward-facing scenes of Scannet++, Real10k, DL3DV, and SPInNeRF. 
related work: 
 Image Inpainting devoted to filling masked regions of the image with vivid textures and structures as a long-standing challenge in computer vision, which has been widely investigated in both classical and learning-based methods [83] . Compared to the low-level feature-based traditional manners [61, 36] , learning-based ones gradually dominate this field and achieved substantial successes based on GANs [53, 100, 37, 8] , attention mechanism [93, 90] , adapted convolutions [42, 94, 70] , and diffusion models [47, 62, 59] . Moreover, the image inpainting task can be further extended to reference-guided inpainting and video inpainting. Reference-guided inpainting completes the target image based on one or several reference images, which incorporates 3D information for accurate structures [105, 101] or T2I priors for examplar-based recovery and editing [86, 14, 7] . On the other hand, video inpainting methods [24, 39, 102] often include the optical flow to capture motions for superior spatial and temporal coherence. However, we should clarify that the aforementioned inpainting methods cannot achieve multi-view consistent inpainting for 3D scene editing. The reference-based manners lack multi-view consistency across all generated views, while video-based methods focus on moving foregrounds rather than the synthesis with large viewpoint changes as verified in our experiments.3D Generation. Given text descriptions or reference images, 3D generative models produce highquality 3D assets, which have been widely investigated recently, benefiting from the rapid development of diffusion-based 2D T2I models [59, 63, 4, 54, 21] . Some pioneering works with score distillation sampling (SDS) loss [75, 55] leverage priors from diffusion-based 2D supervision for the 3D generation, which is further explored to better optimization objectives [106, 79] and multi-stage learning [40, 13, 71] . On the other hand, Zero123 [44] fine-tuned the T2I model for object-level NVS, which is further investigated for consistent multi-view synthesis [67, 66, 43, 46] . Besides, enhanced by the good 3D feature presentations, like tri-plane [11] , and scalable network backbones [74] , foundational 3D generation models also achieved impressive results [109, 85, 31] , training with extremely large 3D datasets from scratch. However, all these works are trained with large-scale synthetic 3D objects or optimized through SDS without any interaction with complicated backgrounds, making it difficult to generalize to real-world editing scenarios with reasonable illustrations and shadows.Novel View Synthesis (NVS). Before the diffusion models, most NVS works focused on learning promising feature encoder [69, 91] with blur regression-based predictions. After the development of 3D-aware diffusion models [12, 73, 1] and fine-tuning from foundational T2I models [64, 29, 82] , NVS results are significantly improved. However, generating whole novel views is too challenging. Thus these methods still suffer from constrained generalization with seen categories or expensive test-time optimization. Another way is to iteratively warp and inpaint the novel views through monocular depth estimation and 2D inpainting [92, 23, 30, 97] . Despite impressive scene-level synthesis results, these methods suffer from prohibitive time costs for the warp-and-inpaint dataset update. Moreover, ambiguous depth estimations would degrade the structures of foreground objects.NeRF and 3DGS Editing. With the development of NeRF [49] and 3DGS [35] , many follow-ups tried to integrate them into 2D generative models, including NeRF inpainting [51, 81, 50] , textualguided semantic editing [26, 15] and local editing [108, 3, 38] based on SDS loss. InseRF [65] unifies both NeRF editing and 3D generation by inserting an image-to-3D generation into multi-view images for the NeRF optimization. Unfortunately, local editing-based approaches suffer from unnatural and disharmonious results, especially for the illumination and shadow in boundary regions. Furthermore, all these works require scenes with accurate camera poses and costly test-time optimization to encourage multi-view consistency, limiting their real-world applications. 
approach: 
 Overview. We show the overall pipeline and contributions in Fig. 2 , which is detailed in Fig. 3(a) . The inputs of MVInpainter are sequential images I 0:N of the same scene and related masks M 0:N with N + 1 total views; I 0 indicates the clean reference image manipulated by any 2D editing approach, while I 1:N are other target views needed to be inpainted by MVInpainter with consistent results.Multi-View Inpainting Formulation. We focus on multi-view inpainting rather than naive NVS as it suits real-world editing better, because 1) most real-world edits do not need to synthesize complete novel views; 2) the inpainting formulation activates the inherent in-context priors from T2I models to ensure natural illumination and shadow, which are intractable to be addressed by 3D generation trained with synthetic data; 3) such a simplified formulation alleviates the training difficulty and the dependency on camera poses. Specifically, MVInpainter enjoys consistent multiview inpainting to bridge 2D and 3D editing, which is built upon a foundation 2D inpainting model, SD1.5-inpainting [59] with two data prerequisites. First, the input multi-view images I 0:N should be captured in an ordered camera trajectory to alleviate the pose requirement. Second, M 0 is a zero matrix to ensure the first view is clean without masking, while other masks M 1:N should cover all possible regions in respective views where the target object would be placed at. To elegantly meet this demand, we propose a heuristic masking technique for the inference detailed in Sec. 3.4. Hence, the input of MVInpainter could be formulated as an ordered video sequence with (N + 1) frames as:EQUATION)where t indicates the timestep in the diffusion; z 0:N t denote the 4-channel noised latent feature of I 0:N after the VAE encoding [59] ; M0:N and z(1 -M) 0:N mean that the downsampled masks and noise-free latent features in unmasked regions are always concatenated as the input condition. We learn MVInpainter through the epsilon prediction θ [28] , while the MSE loss can be written as:EQUATION)where c denotes the prompt text encoded by the textual CLIP τ φ [56] .Here we tackle two related but distinct tasks: multi-view consistent object removal, and object insertion or replacement. Due to the differing challenges of each task, we employ the same structure, called MVInpainter, but trained with different multi-view data, prompts, and masking strategies, leading to two variants: MVInpainter-O and MVInpainter-F.Object-Centric and Forward-Facing Datasets. Particularly, we train these two MVInpainters to handle different distributions of multi-view data: object-centric and forward-facing data. Object-centric datasets (CO3D [57] , MVImgNet [95] , Omni3D [6] )2 feature a single object at the center of all views, captured with the camera circling around it. On the other hand, forward-facing datasets (Real10K [103] , Scannet++ [89] , DL3DV [41] ) resemble static video data with camera movements but no specific foreground objects. Essentially, object-centric images heavily favor a single foreground object in all views, so an MVInpainter trained on them struggles to reliably remove objects without hallucinating any artifacts. Conversely, forward-facing images are not conducive to modeling multiview object synthesis with significant viewpoint changes.To address these challenges, we separately train MVInpainter-O and MVInpainter-F on object-centric and forward-facing datasets, respectively. They involve different LoRAs, motion modules (Sec. 3.2), and flow grouping modules (Sec. 3.3) . Note that we keep the SD-inpainting backbone frozen for both models. While fine-tuning the entire SD backbone for MVInpainter-O might help convergence with mixed CO3D and MVImgNet datasets, we maintain consistent settings with MVInpainter-F for simplified discussion in this study. Please refer to Sec. B.5 of Appendix for more details.Prompt. For MVInpainter-O, we empirically find that meaningful prompts c benefit to preserve the identity and appearance of objects, as these prompts provide complementary information for the object synthesis with unseen viewpoints. Thus we utilize InternLM [19] to extract captions for training and inference. For MVInpainter-F which is mainly used for object removal and scene completion with minor viewpoint changing, we leverage the prompt tuning technique [7] with 16 trainable tokens as the global prompts instead of specific descriptions for stable generation.Masking Strategy. We adopt hybrid inpainting masks [70, 8] for MVInpainter, including random irregular, rectangular, and segmentation-based masks. In particular, we further employ the objectlevel tracking masks for MVInpainter-O training, which could be easily accomplished by SAMtracking [88] . Moreover, to avoid the mask overfitting towards the object shape, we follow [7] to randomly disturb the object masks with rectangular and irregular masks. Note that MVInpainter-O preserves a little percentage (15%) to use random masks without the object ones to encourage the cross-view learning ability for the frame interpolation and sparse-view NVS (Appendix Sec. B.2).Motion Priors from Video Models. Since the input of MVInpainter could be regarded as the video sequence as in Eq. 1, it is intuitive to leverage motion priors from video models to improve performance. Thus we employ the domain adapted LoRA [32] and temporal transformers from AnimateDiff [25] pre-trained on video data as the initialized motion parts of MVInpainter. Although AnimateDiff is not trained for an inpainting model, we surprisingly find that it could be well converged with only a few hundred steps of fine-tuning, while both MVInpainter and AnimateDiff share the same SD1.5 backbone. Empowered by motion priors, MVInpainter achieves significantly superior structural consistency as discussed in Sec. 4.4.Inspired by [66, 33] , to further ensure appearance consistency, we adopted Ref-KV in the self-attention of denoising U-Net to activate the inherent capacity of T2I models. Ref-KV spatially concatenates reference features to target keys and values to inject appearance guidance during attention aggregation as in Fig. 3(b) . We clarify the originality of Ref-KV as follows. Compared to aggregating all frames [29] , Ref-KV only focuses on the first reference view, which reduces memory and computational costs, and also substantially enhances the appearance consistency as verified in Fig. 7 . Different from the reference attention in [66, 33] ,MVInpainter is an inpainting model, which always captures the unmasked reference latent without noise (Eq. 1), thus it is unnecessary to re-scale the latent by multiplying a large scale [66] or use the noise-free latent from another U-Net [33] .Benefiting from the inpainting formulation and the ordered input sequence of MVInpainter, our approach could be trained and tested without explicit camera poses. However, it is still non-trivial to inpaint the foreground object with correct poses while the masks are large or the unmasked environment is ambiguous and textureless. To overcome this, we leverage the low-level optical flows The illustration of heuristic masking adaption, which is built from yellow points of the closed convex hull. (c) The perspective warping based on the basic plane and the bottom face. All matches are on the basic plane filtered by Grounded-SAM [58] with captions "table" and "tablecloth".extracted by RAFT [72] to guide the MVInpainter generation. We regard the reverse N -frame optical flow as F 0:N -1 ∈ R (N -1)×H×W ×2 . All flow inputs are masked to avoid leakage 3 .In the pilot study, we first applied the 2D CNN and self-attention modules to encode the flow features and added them to the U-Net input as additional conditions. But such simple incorporation failed to improve the MVInpainter as the 'dense flow' setting in Tab. 3b. Because such an explicit dense flow injection leads to the overfitting pitfall, i.e., MVInpainter is strongly controlled by the flow inputs and ignores other contextual clues learned from the foundational T2I model. We hope the flow feature should carry more high-level motion characters, such as the direction and speed of the camera trajectory rather than the detailed correlation. Thus we propose the flow grouping enhanced by the slot-attention [45, 87] as shown in Fig. 3(c ).Formally, slot-attention maintains K learnable query vectors as Q ∈ R K×d , where d denotes the channels; K = 4 in this paper. Key and value are the same flow features as K = V ∈ R HW ×d . Then the slot-attention can be formulated as:EQUATION)whereQ, K, Ṽ indicate QW q , KW k , VW v encoded by linear weights W q , W k , W v ∈ R d×d .Note that the slot-attention is similar to the cross-attention except that the former should be normalized in the query dimension, while the latter is normalized in the key dimension. As the global queries, all slot features Q enjoy the high-level motion information aggregated by dense flow features. We take these slot features and subsequently use an FC layer to encode them into a single flow grouping embedding as shown in Fig. 3(c ). According to the ablation study in Tab. 3b, we concatenate this embedding to the CLIP feature as an extra motion 'token' with comparable performance and less trainable weights. Moreover, we found that using temporal 3D attention for the flow grouping can further improve the results, while the slot features are shared across all views with more general information. Flow grouping achieves more robust guidance for MVInpainter than the dense flow features, even with some corrupted or inaccurate flow inputs.We show the inference pipeline of MVInpainter in Fig. 4 (a). Specifically, given sequential images, MVInpainter could address the general multi-view editing through object removal and object insertion mentioned in Sec. 3.1. The object removal stage can be omitted if there are no foreground objects (inserting only) or the target object shares similar masks as the original one (replacing). For the object removal, we utilize the MVInpainter-F based on the reference image inpainted by SD-inpainting with the caption "background". For the object insertion, the reference image can be achieved by any 2Dgenerative models, such as T2I inpainting [59, 77, 84] and exemplar-based inpainting [86, 14] . Then MVInpainter-O is leveraged to expand the single-view generation to the multi-view scenario. We propose a heuristic masking adaption to tackle the most critical issue, i.e., building suitable masks without any pose conditions before the insertion.Masking Adaption is based on points from the closed convex hull as shown in Fig. 4 (b), which can be obtained from the 3D bounding box of the foreground object. We could use open-vocabulary 3D grounding [16] or manually annotate four landmarks to achieve the bottom face and estimate the top face through the height of the 2D mask. To reasonably warp the 3D box, our masking adaption should meet an important and easily satisfied condition: the 3D box's bottom face and the basic plane on which the object is placed must be the same plane. Intuitively, the bottom face should be close to the ground. Therefore, the basic plane and the bottom face of the 3D box from different image pairs share the same perspective transformation matrix. Thanks to the dense matching [20] , it is easy to obtain dense matching pairs on the basic plane through the Grounded-SAM [58] as in Fig. 4(c ). Subsequently, we achieve the transformation matrix M by RANSAC [22] with 100 sampled matching pairs and apply the perspective warping for the new bottom face. Note that this perspective warping cannot be used on the top face, because the top face is just parallel to the basic plane rather than close to it. Instead, we get the new top face following the bottom landmarks with a constant height from the 2D mask. Finally, we fill the convex hull and optionally mask it with irregular brushes as in [7] or dilate the 3D box masks. The masking adaption is a flexible strategy, which not only locates the position of the object across different views but also incorporates human priors by manually annotating for some objects with very special shapes, such as the baseball bat in Appendix Fig. 13 . 
experiments: 
 Datasets. MVInpainter-O is trained on the object-centric data that includes full categories of CO3D [57] and MVImgNet [95] . Moreover, we regard the Omni3D [6] as the zero-shot validation. MVInpainter-F is trained on the forward-facing data with Real10K [103] , Scannet++ [89] , and DL3DV [41] , including both indoor and outdoor scenes. We further employ comparison on SPInNeRF [51] to verify the object removal ability. Metrics. In this paper, we evaluate our method with PSNR, LPIPS [99] , FID [27] , and KID [5] . We also include CLIP score [56] for the NVS task to verify the identity maintenance capability. For the object removal, we further compare the similarity of DINOv2 features [52] extracted from masked regions to evaluate the inpainting consistency, denoting average DINOv2 similarity (DINO-A) and minimal DINOv2 similarity (DINO-M) among masked patches, respectively.Results of object-centric data are compared in Tab. 1 and Fig. 5 . We considered two types of approaches, including NVS manners: ZeroNVS [64] ; inpainting-based methods: Nerfiller [80] and LeftRefill [7] . Note that ZeroNVS requires explicit camera poses. Besides, the inpainting-based methods are pose-free with enlarged bounding box masks to avoid masking shape leakage. Without the inpainting formulation, the scene-level ZeroNVS struggles to directly synthesize novel views without SDS. For inpainting-based approaches, LeftRefill can capture the scene's main object but fails to maintain multi-view consistency. Nerfiller only retains the shape of several views, while Nerfiller [80] , ZeroNVS [64] , and our MVInpainter. Please zoom-in for details.Table 1 : Quantitative results of object-centric NVS with enlarged bounding box masks compared on CO3D [57] and MVImgNet [95] . We also include Omni3D [6] other views suffer from significantly inferior structures and identities. Our model outperforms all competitors with prominent achievements on both the in-domain test set and the zero-shot Omni3D.Object Removal. We quantitatively verify the object removal ability of MVInpainter-F on SPInNeRF test set [51] in the left of Tab. 2, while qualitative comparisons of the train set are shown in Fig. 8 of Appendix. Other competitors include conventional single-view based inpainting manners: LaMa [70] , MAT [37] , and SD-inpainting [59] . We further compare the pose-free reference-inpainting Left-Refill [7] and the video inpainting manner ProPainter [102] . Formally, MAT and SD-inpainting suffer from inconsistent inpainting results with poor DINO-A and DINO-M. Though LaMa achieves the highest DINO-M, it generates consistent blur and artifacts as in Appendix Fig. 8 (a). For the reference-based manners, LeftRefill only conditions on the first view without multi-view consistency, while ProPainter performs inferior in synthesis quality. Since the small test set, FID would be largely degraded if one result contains artifacts. Our method enjoys the best performance in LPIPS, FID, and DINO-A with consistent generations without any unstable hallucination.Scene-Level Inpainting. We provide quantitative results of multi-view scene inpainting in the right of Tab. 2 corrupted with large irregular masks. Our method achieves the best results of all metrics, prominently outperforming video-based inpainting, reference-guided inpainting, and other single-view inpainting approaches.We verify the in-the-wild scene editing ability of MVInpainter following Sec. 3.4 in Fig. 6 , where the background images are from the unseen MipNeRF360 [2] . MVInpainter-F achieves stable and consistent object removal, and MVInpainter-O performs high-quality multi-view generation for various object shapes based on the flexible mask adaption. Benefiting from the consistent results, our method enjoys reliable reconstruction with stereo-based Dust3R [76] and MVS [9] . Sec. C of the Appendix discusses more details about the point cloud and 3DGS reconstruction. To verify the effectiveness of each component from MVInpainter, we conduct ablation studies on CO3D as in Tab. 3. From Tab. 3a, the baseline is built upon SD1.5-inpainting with random-initialized LoRA and motion transformer blocks, and we also compare the alternative baseline without the inpainting formulation built upon naive SD1.5. The multi-view inpainting formulation largely improves the results, and more details about the inpainting comparison are discussed in Appendix Sec. B.4. Besides, we show qualitative ablation results in Fig. 7 , which indicate that video priors from AnimateDiff and Ref-KV could substantially facilitate the structure and appearance consistency respectively. We also realize that object-level tracking masks are critical for training MVInpainter-O. Moreover, we analyze the effect of flow grouping in Tab. 3b. Without any flow guidance, our method fails in some ambiguous cases, such as large pose changes of stop signs and laptops in Fig. 12 of Appendix, while dense flow slightly hinders the identity (PSNR and CLIP). The proposed slot-attention based flow grouping outperforms the vanilla flow injection, while incorporating flow embedding by cross-attention is more lightweight with comparable performance. Further, the flow grouping can be improved with 3D temporal attention learning. 
conclusion: 
 This paper proposes MVInpainter, a multi-view consistent inpainting method to expand 2D generations into 3D scenes by multi-view object removal, insertion, and replacement. MVInpainter enjoys a pose-free inpainting formulation built upon the SD-inpainting backbone with motion modules. Motion initialization based on video priors and Ref-KV are presented to facilitate the structure and appearance consistency respectively. Furthermore, we propose to use flow grouping based on the slot-attention to encourage implicit motion control. For the inference, we present a novel mask adaption strategy to warp object masks to novel views. Sufficient experiments on both object-centric and forward-facing datasets verified the effectiveness of MVInpainter. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: see in Sec. E.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: our paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: we have cited related works and codes.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: We provide more qualitative results in this section, including object removal results of Fig. 8 and Fig. 9 ; multi-view scene editing of Fig. 13 and Fig. 14 ; the generalization of the proposed mask adaption of Fig. 15 ; results of multi-view object-level NVS in Fig. 10 ; object replacement by T2I inpainting model and the exemplar-based AnyDoor [14] in Fig. 11 . We further provide some qualitative ablation studies in Fig. 12 . These visualizations verify the wide application of the proposed MVInpainter. To inpaint extremely long sequences, MVInpainter could first inpaint some keyframes, then apply the frame interpolation-based inpainting to extend these keyframes to more views. Formally, we first interactively mask and inpaint among keyframes like Fig. 16(a) . Afterward, we uniformly sample long-range inpainted results as fixed conditions at left (max to 12), and then interactively inpaint other views to preserve the identity and appearance as Fig. 16(b) .We found some color difference near the masking boundary in inpainting results as in Fig. 17 , especially for indoor cases with textureless regions. Moreover, we further found that this issue also occurred with the original SD-inpainting. Hence, we follow the asymmetric VAE decoder [107] which includes unmasked image pixels and masks as additional inputs, and apply the data augmentation in [78] to fine-tune the VAE decoder. Though the quantitative results are almost unchanged, the augmented new decoder enjoys much more consistent visualizations as shown in Fig. 17 .To verify the effect of the inpainting formulation, we further detail the comparison in Tab. 5 and Fig. 18 . The 'Baseline' is the same as the one listed in Tab. 3a, denoting the backbone of SD1.5inpainting 4 . Both 'SD-blend' and 'SD-NVS' utilize vanilla SD1.5 5 as the backbone without any inpainting fine-tuning and masking. The denoising process of 'SD-blend' is blended with unmasked regions [47] , while 'Sd-NVS' is fine-tuned with the whole view synthesis with a noise-free first reference view. All models mentioned above are fine-tuned with random initialized LoRAs and motion transformer blocks without other improvements (Ref-KV, AnimateDiff, flow grouping) on CO3D. Both quantitative and qualitative results show the effectiveness of the inpainting-based baseline, which enjoys properly good pose formulation even without any pose guidance. We compare MVInpainter to the NeRF Editing method, SPIn-NeRF [51] in Fig. 19 and Tab. 7. Our contributions are orthogonal to NeRF editing-based manners [51] . MVInpainter focuses on tackling multi-view editing with a feed-forward model, while NeRF editing is devoted to reconstructing instance-level scenes with test-time optimization. NeRF editing manners require exact camera poses and costly test-time optimization for each instance (SPIn-NeRF needs about 1 hour for each scene). Moreover, NeRF editing manners fail to substitute for our method: a) NeRF editing starts with inconsistent 2D-inpainting results, which leads to blurred results as shown in Fig. 19 , while our method could refer to a high-quality single-view reference without conflicts. b) Although both methods enjoy good consistency, rendering-based inpainting suffers from color difference when blended with the original images (the last row of Fig. 19 ). As shown in Tab. 7, our method is comparable to SPIn-NeRF in consistency (DINO-S, DINO-L) with better image quality (PSNR, LPIPS, FID) and fidelity in their official object removal test set. We validate the inference time of our MVInpainter, AnimateDiff [25] , Nerfiller [80] , and Leftrefill [7] in Tab. 8. Note that all methods are based on 50 steps of DDIM except for Nerfiller, which uses 20Ours (blended with original inputs)SPIn-NeRF (no blending)SPIn-NeRF (blended with original inputs)Figure 19 : Object removal compared to SPIn-NeRF [51] .steps as the official setting. Our method is much more faster than other inpainting manners compared to Nerfiller and Leftrefill. Nerfiller requires costly iterative updating, which is slow even with much less de-noising steps. Leftrefill can only produce one view at once time rather than jointly produce all target views as ours. Our method only costs a little more inference time compared to the baseline AnimateDiff (+1.4s), which is mainly used by flow grouping and Ref-KV.Since MVInpainter achieves consistent multi-view 2D generations, we simply employ some existing 3D reconstruction methods to further extend these results to 3D scenes without sophisticated pipelines.Dust3R. First, we leverage the pose-free Dust3R [76] to achieve dense point clouds as the initialization of 3DGS. We empirically find that 12 views are sufficient to achieve high-quality point clouds as shown in Fig. 20 (a) with only a few seconds. To ensure consistent initialization for 3DGS training, we align all point clouds from Dust3R with the same camera pose system of Colmap. Although the combined point clouds in Fig. 20 (a) suffer a little inconsistency and outliers caused by errors from Dust3R estimation and alignment, 3DGS could eliminate them.For some more difficult cases, such as 'room' in MipNeRF360, Dust3R fails to achieve consistent point clouds across all image pairs. Instead, we choose the SOTA learningbased MVS method, MVSFormer++ [9] , to reconstruct the dense point cloud with 24 input views as shown in Fig. 21 . MVS also takes a few seconds for the depth estimation and re-projecting correction, which is as efficient as Dust3R. The success of MVS verifies that MVInpainter can produce consistent multi-view inpainting, encouraging stereo-based reconstruction.To train 3DGS, we first interpolate inpainted results of 'kitchen' in MipNeRF360 to 48 views with frame interpolation (Sec. B.2), where 42 frames are the training set; and 6 frames are the test set, while the initialized point clouds are got from Dust3R. Besides, we use only 24 inpainted views of 'room' to verify the robustness of our method with sparser input views in complicated scenes, while MVSFormer++ is used to provide the initialized point clouds. Different from previous works that require heavy SDS loss [64, 3] and dataset updates [26, 68, 80] , the naive 3DGS could be simply converged with our raw multi-view results. Following [48] , we find that LPIPS loss [99] could further alleviate the influence of slightly inconsistent appearance. However, learning 3DGS with only LPIPS and SSIM losses as [48] is very unstable for real-world scenes' reconstruction in our pilot study. Instead, we optimize LPIPS loss only for the foreground object as:EQUATION)where m is the foreground mask from SAM-tracking [88] ; ⊙ is element-wise multiplication; λ = 0.2, λ lpips = 0.1 respectively. Thanks to the consistent multi-view results, we empirically find that just 5k training steps with 3 minutes could achieve good 3DGS results as in Fig. 20 (b)(c). Furthermore, the local-based LPIPS loss further facilitates the performance with clear boundaries. Specifically, for the baseball bat in Fig. 20 (a), the final dense point cloud struggles to perfectly unify point clouds from all views, because of the challenging long shape that magnified the Dust3R stereo and mask adaption errors. However, our local LPIPS-based 3DGS can largely alleviate this issue with consistent outcomes. We recommend comparing the video results of inpainted multi-view images and 3DGS in our supplementary. This paper exploited multi-view consistent inpainting based on text-to-image models. Because of their powerful generative capacity, these models would produce misinformation or fake images. So we sincerely remind users to pay attention to it. Besides, privacy and consent are important considerations, as generative models are often trained on large-scale data. Furthermore, generative models may perpetuate some biases according to the training data, leading to unfair outcomes. Therefore, we recommend users be responsible and inclusive while using these text-to-image generative models. Note that our method only focuses on technical aspects. Both images and pre-trained models used in this paper are all open-released.NeurIPS Paper Checklist • The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).This paper only focuses on data captured by object-centric camera trajectory rather than discovering decoupled observations through unsupervised feature learning[45, 104,34].We extract flows before masking, because foregrounds largely benefit the flow quality. We further dilate masks with 5 pixels for flow to avoid leakage. As low-level local features, no conflict is observed when using these masked flows for applications like removal and replacement.https://huggingface.co/runwayml/stable-diffusion-inpaintinghttps://huggingface.co/runwayml/stable-diffusion-v1-5 


Decision: All reviewers acknowledged the novelty of this paper, as well as the solid and thorough experiments conducted. In the initial reviews, there were suggestions to further demonstrate the superiority of the proposed method from various perspectives. Especially, one reviewer raised several questions, including about explicit 3D representations based on our multi-view synthesis and technical details. During rebuttal, the authors tried to clarify the contribution of this paper and technical details. AC has also carefully reviewed the paper, the reviews, and the rebuttal, and concurs with most reviewers' opinions, such as the innovative conceptual contribution. Although some technical details could lead the misleading, AC thinks that these could be addressed in the final version. Therefore, AC recommends acceptance. It would be greatly beneficial to include the additional results from the rebuttal phase in the final version.

Reviews: review: paper_summary: paper_summary: The paper studies the task of multi-view consistent real-world object removal and insertion, enabled by learning a model, MVInpainter, trained to perform multi-view 2D inpainting. The paper demonstrates its effectiveness on both object-centric and scene-level datasets with the task of object removal and object insertion.
strength_weakness: strength_weakness: 1.Though the paper tackles the setting of video input (consecutive frames to obtain the motion prior), I am curious how the method performs with just multi-view inputs? Does the performance significantly drop? If the model can be applied to those as well, it could potentially lead to bigger impact and be applied to more real-world scenarios.

2.How are the priors different from AnimateDiff and Flow Grouping? Why are they complementary?

3.Could the authors please explain why the proposed method cannot be compared to SPIn-NeRF-type methods? Is it because of camera poses? If so, I am still curious to see how large the gap is? If the method is close to the methods with camera poses, it would make the paper stronger. 

4.One limitation is that the method still works on images with a relative simple background, as pictured in the supplementary materials Figure 19.
questions: questions: I think the paper is overall of good quality and tackles an important task. Some questions (as listed) could be explained for a better understanding of the method.
limitations: limitations: N/A
paper_summary: paper_summary: This paper proposes a new 3D editing method by regarding it as a multi-view 2D inpainting task.  It ensures cross-view consistency through video priors and concatenated reference key/value attention and controls camera movement without explicit poses using slot attention. It shows effectiveness in object removal, insertion, and replacement.
strength_weakness: strength_weakness: - Some assumptions are heuristic, such as in L232. While these assumptions may be reasonable in most cases, they might not be robust across diverse user cases.
- Overall, the figures and flow of the paper are complicated. It would be beneficial if the content of the figures were made more comprehensive.
questions: questions: How is the generalizability? What is the performance in completely OOD cases?
limitations: limitations: the authors adequately addressed the limitations.
paper_summary: paper_summary: The paper formulates the 3D object editing task as a multi-view 2D in-painting task. Firstly, MVInpainter-F is employed to remove the object and obtain the background scene. Then, MVInpainter-O generates multi-view images based on the reference view.
strength_weakness: strength_weakness: It is uncertain whether the design of mask adaption is robust enough.
questions: questions: 1. During training, the flow is obtained using RAFT and masked. In the inference stage of MVInpainter-F, is the flow only obtained from the masked image? Does the utilization of this low-quality flow negatively impact the results? 
2. How can we ensure that mask adaption remains robust when dealing with different masks and backgrounds?
limitations: limitations: The author provides a detailed discussion of limitations and broader impacts in the supplementary material.
paper_summary: paper_summary: This paper proposes a new method for multi-view consistent inpainting. The input includes a video and a sequence of masks. The first image could be manipulated by any 2D editing approach, while the remaining images need to be inpainted using the proposed method. The method is built upon the SD1.5-inpainting model and fine-tuned with two LoRAs: one for object-centric and one for scene-level, using different datasets. It employs motion priors by fine-tuning the domain-adapted LoRA and temporal transformers from AnimateDiff, pre-trained on a video dataset. The technique of reference key & value concatenation from previous works is used to enhance consistency with the reference image. Flow features are extracted and added to the UNet as additional conditions. During inference, a heuristic strategy is proposed to warp and propagate the mask from the first frame to all subsequent frames.
strength_weakness: strength_weakness: 1. While the title contains "to Bridge 2D and 3D Editing," the proposed techniques are primarily designed for 2D images or videos, with no explicit 3D representation used or 3D output generated. It would be interesting to explore whether 3D output can be extracted from the resulting image sequence. Otherwise, it might be better to change the title "3D editing" (and other mentions in the text) to something less misleading.

2. The utilization and fine-tuning of domain-adapted LoRA and temporal transformers from video models are intuitive and straightforward, but there do not appear to be any insightful design choices, limiting the novelty.

3. The technique of "Reference Key & Value Concatenation" has been used in many previous works. We fail to see much novelty here.

4. While the section introducing the techniques is generally clear, the concrete settings and details for training, inference, and evaluation are very hard to follow. I tried my best to understand the setup but still feel confused about many details, which is quite frustrating. The paper needs to be rewritten in a more well-organized way, introducing all necessary settings and details. Specifically, the following points need clarification:

    a) During inference, you employ 2D models to generate reference images for different tasks (e.g., object removal/editing/replacement). However, what is the input-output pair during training? For example, do you have specific ad-hoc training pairs for the object removal task? 

    b) The experiment contains various setups, and the concrete settings are confusing. For each setup, it would be better to explicitly mention the task (e.g., inpainting with object/random masks, object generation, object removal, object replacement), dataset, the input, ground truth output, and how the **mask** are generated. Currently, it only says "Object-Centric Results," "Forward-Facing Results," and "Real-World 3D Scene Editing." But the concrete tasks are unclear. For example, what is the difference between "Object Removal" and "Scene-Level Inpainting"? I assume "object removal" is also achieved by inpainting? If the setup for quantitative and qualitative experiments is different, please also mention it. For example, there are some qualitative results for "object removal," but it should not be evaluated quantitatively (no ground truth)?

5. In equation (1), you concatenated both the masked latents and unmasked noised latents. Some related questions:

    a) The 9-channel input differs from the original input of SD1.5-inpainting. How do you achieve that?

    b) The noise-free latent is currently masked after VAE encoding. Will this cause leakage? It might be better to mask before VAE encoding.

6. What is the difference between the "Masking strategy" (line 163) and "masking adaptation" (line 228)? Is one for training and one for inference?

7. The method extracts flows before masking. This may be acceptable for the tasks of removal and replacement, but will it cause leakage for the task of general NVS and inpainting, considering the flow model may have seen the ground truth image?
questions: questions: 1. Have you considered directly fine-tuning a video inpainting model instead of using SD1.5-inpainting?

2. Line 186: The sentence "which always captures the unmasked reference latent without noise (Eq. 1), thus it is unnecessary to re-scale the latent before adding noise from another U-Net" is unclear to me. Could you please clarify?

3. Line 207: The sentence "except that the former should be normalized in the query dimension, while the latter is normalized in the key dimension" is unclear. Could you elaborate?

4. Line 212: The phrase "with comparable performance and fewer trainable weights" needs clarification. Which two setups are being compared?

5. Could you provide more details about the 3D attention mechanism?

6. Line 309: In the context of not having object-level tracking masks, how do you obtain the mask?
limitations: limitations: Yes, discussed in supplementary.

Comments: title: Feel free to discuss any remaining questions or concerns
Feel free to discuss any remaining questions or concerns
comment: Thanks for the valuable feedback from all reviewers. We have carefully addressed all concerns in our rebuttal. Our models and codes will be open-released, including all details to reproduce our results. As the rebuttal deadline approaches, please feel free to discuss any remaining questions or concerns. We will try our best to answer your questions.
Thanks for your valuable feedback. We have carefully addressed all your concerns and provided the details in our rebuttal. Our models and codes will be open-released, including all details to reproduce our results. As the rebuttal deadline approaches, please feel free to discuss any remaining questions or concerns. We will try our best to answer your questions.