MIVE: New Design and Benchmark for Multi-Instance Video Editing
Authors: name: Samuel Teodoro; name: Agus Gunawan; name: Soo Ye Kim; name: Jihyong Oh; name: Munchurl Kim
Published: 2024-12-17 13:00:04

Abstract: Recent AI-based video editing has enabled users to edit videos through simple
text prompts, significantly simplifying the editing process. However, recent
zero-shot video editing techniques primarily focus on global or single-object
edits, which can lead to unintended changes in other parts of the video. When
multiple objects require localized edits, existing methods face challenges,
such as unfaithful editing, editing leakage, and lack of suitable evaluation
datasets and metrics. To overcome these limitations, we propose a zero-shot
Multi-Instance Video Editing
framework, called MIVE. MIVE is a general-purpose mask-based framework, not
dedicated to specific objects (e.g., people). MIVE introduces two key modules:
(i) Disentangled Multi-instance Sampling (DMS) to prevent editing leakage and
(ii) Instance-centric Probability Redistribution (IPR) to ensure precise
localization and faithful editing. Additionally, we present our new MIVE
Dataset featuring diverse video scenarios and introduce the Cross-Instance
Accuracy (CIA) Score to evaluate editing leakage in multi-instance video
editing tasks. Our extensive qualitative, quantitative, and user study
evaluations demonstrate that MIVE significantly outperforms recent
state-of-the-art methods in terms of editing faithfulness, accuracy, and
leakage prevention, setting a new benchmark for multi-instance video editing.
The project page is available at https://kaist-viclab.github.io/mive-site/