MVInpainter: Learning Multi-View Consistent Inpainting to Bridge 2D and
  3D Editing
Authors: name: Chenjie Cao; name: Chaohui Yu; name: Yanwei Fu; name: Fan Wang; name: Xiangyang Xue
Published: 2024-08-15 07:57:28

Abstract: Novel View Synthesis (NVS) and 3D generation have recently achieved prominent
improvements. However, these works mainly focus on confined categories or
synthetic 3D assets, which are discouraged from generalizing to challenging
in-the-wild scenes and fail to be employed with 2D synthesis directly.
Moreover, these methods heavily depended on camera poses, limiting their
real-world applications. To overcome these issues, we propose MVInpainter,
re-formulating the 3D editing as a multi-view 2D inpainting task. Specifically,
MVInpainter partially inpaints multi-view images with the reference guidance
rather than intractably generating an entirely novel view from scratch, which
largely simplifies the difficulty of in-the-wild NVS and leverages unmasked
clues instead of explicit pose conditions. To ensure cross-view consistency,
MVInpainter is enhanced by video priors from motion components and appearance
guidance from concatenated reference key&value attention. Furthermore,
MVInpainter incorporates slot attention to aggregate high-level optical flow
features from unmasked regions to control the camera movement with pose-free
training and inference. Sufficient scene-level experiments on both
object-centric and forward-facing datasets verify the effectiveness of
MVInpainter, including diverse tasks, such as multi-view object removal,
synthesis, insertion, and replacement. The project page is
https://ewrfcas.github.io/MVInpainter/.