Animated Stickers: Bringing Stickers to Life with Video Diffusion
Authors: name: David Yan; name: Winnie Zhang; name: Luxin Zhang; name: Anmol Kalia; name: Dingkang Wang; name: Ankit Ramchandani; name: Miao Liu; name: Albert Pumarola; name: Edgar Schoenfeld; name: Elliot Blanchard; name: Krishna Narni; name: Yaqiao Luo; name: Lawrence Chen; name: Guan Pang; name: Ali Thabet; name: Peter Vajda; name: Amy Bearman; name: Licheng Yu
Published: 2024-02-08 22:49:32

Abstract: We introduce animated stickers, a video diffusion model which generates an
animation conditioned on a text prompt and static sticker image. Our model is
built on top of the state-of-the-art Emu text-to-image model, with the addition
of temporal layers to model motion. Due to the domain gap, i.e. differences in
visual and motion style, a model which performed well on generating natural
videos can no longer generate vivid videos when applied to stickers. To bridge
this gap, we employ a two-stage finetuning pipeline: first with weakly
in-domain data, followed by human-in-the-loop (HITL) strategy which we term
ensemble-of-teachers. It distills the best qualities of multiple teachers into
a smaller student model. We show that this strategy allows us to specifically
target improvements to motion quality while maintaining the style from the
static image. With inference optimizations, our model is able to generate an
eight-frame video with high-quality, interesting, and relevant motion in under
one second.