MagicStick: Controllable Video Editing via Control Handle
  Transformations
Authors: name: Yue Ma; name: Xiaodong Cun; name: Yingqing He; name: Chenyang Qi; name: Xintao Wang; name: Ying Shan; name: Xiu Li; name: Qifeng Chen
Published: 2023-12-05 17:58:06

Abstract: Text-based video editing has recently attracted considerable interest in
changing the style or replacing the objects with a similar structure. Beyond
this, we demonstrate that properties such as shape, size, location, motion,
etc., can also be edited in videos. Our key insight is that the keyframe
transformations of the specific internal feature (e.g., edge maps of objects or
human pose), can easily propagate to other frames to provide generation
guidance. We thus propose MagicStick, a controllable video editing method that
edits the video properties by utilizing the transformation on the extracted
internal control signals. In detail, to keep the appearance, we inflate both
the pretrained image diffusion model and ControlNet to the temporal dimension
and train low-rank adaptions (LORA) layers to fit the specific scenes. Then, in
editing, we perform an inversion and editing framework. Differently, finetuned
ControlNet is introduced in both inversion and generation for attention
guidance with the proposed attention remix between the spatial attention maps
of inversion and editing. Yet succinct, our method is the first method to show
the ability of video property editing from the pre-trained text-to-image model.
We present experiments on numerous examples within our unified framework. We
also compare with shape-aware text-based editing and handcrafted motion video
generation, demonstrating our superior temporal consistency and editing
capability than previous works. The code and models will be made publicly
available.