MoViE: Mobile Diffusion for Video Editing
Authors: name: Adil Karjauv; name: Noor Fathima; name: Ioannis Lelekas; name: Fatih Porikli; name: Amir Ghodrati; name: Amirhossein Habibian
Published: 2024-12-09 15:30:09

Abstract: Recent progress in diffusion-based video editing has shown remarkable
potential for practical applications. However, these methods remain
prohibitively expensive and challenging to deploy on mobile devices. In this
study, we introduce a series of optimizations that render mobile video editing
feasible. Building upon the existing image editing model, we first optimize its
architecture and incorporate a lightweight autoencoder. Subsequently, we extend
classifier-free guidance distillation to multiple modalities, resulting in a
threefold on-device speedup. Finally, we reduce the number of sampling steps to
one by introducing a novel adversarial distillation scheme which preserves the
controllability of the editing process. Collectively, these optimizations
enable video editing at 12 frames per second on mobile devices, while
maintaining high quality. Our results are available at
https://qualcomm-ai-research.github.io/mobile-video-editing/