Make-A-Protagonist: Generic Video Editing with An Ensemble of Experts
Authors: name: Yuyang Zhao; name: Enze Xie; name: Lanqing Hong; name: Zhenguo Li; name: Gim Hee Lee
Published: 2023-05-15 17:59:03

Abstract: The text-driven image and video diffusion models have achieved unprecedented
success in generating realistic and diverse content. Recently, the editing and
variation of existing images and videos in diffusion-based generative models
have garnered significant attention. However, previous works are limited to
editing content with text or providing coarse personalization using a single
visual clue, rendering them unsuitable for indescribable content that requires
fine-grained and detailed control. In this regard, we propose a generic video
editing framework called Make-A-Protagonist, which utilizes textual and visual
clues to edit videos with the goal of empowering individuals to become the
protagonists. Specifically, we leverage multiple experts to parse source video,
target visual and textual clues, and propose a visual-textual-based video
generation model that employs mask-guided denoising sampling to generate the
desired output. Extensive results demonstrate the versatile and remarkable
editing capabilities of Make-A-Protagonist.