Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model
  Adaptation
Authors: name: Guy Yariv, name: Itai Gat, name: Sagie Benaim, name: Lior Wolf, name: Idan Schwartz, name: Yossi Adi

We consider the task of generating diverse and realistic videos guided by
natural audio samples from a wide variety of semantic classes. For this task,
the videos are required to be aligned both globally and temporally with the
input audio: globally, the input audio is semantically associated with the
entire output video, and temporally, each segment of the input audio is
associated with a corresponding segment of that video. We utilize an existing
text-conditioned video generation model and a pre-trained audio encoder model.
The proposed method is based on a lightweight adaptor network, which learns to
map the audio-based representation to the input representation expected by the
text-to-video generation model. As such, it also enables video generation
conditioned on text, audio, and, for the first time as far as we can ascertain,
on both text and audio. We validate our method extensively on three datasets
demonstrating significant semantic diversity of audio-video samples and further
propose a novel evaluation metric (AV-Align) to assess the alignment of
generated videos with input audio samples. AV-Align is based on the detection
and comparison of energy peaks in both modalities. In comparison to recent
state-of-the-art approaches, our method generates videos that are better
aligned with the input sound, both with respect to content and temporal axis.
We also show that videos produced by our method present higher visual quality
and are more diverse.