Seeing Voices: Generating A-Roll Video from Audio with Mirage
Authors: name: Aditi Sundararaman, name: Amogh Adishesha, name: Andrew Jaegle, name: Dan Bigioi, name: Hyoung-Kyu Song, name: Jon Kyl, name: Justin Mao, name: Kevin Lan, name: Mojtaba Komeili, name: ShahRukh Athar, name: Sheila Babayan, name: Stanislau Beliasau, name: William Buchwalter

From professional filmmaking to user-generated content, creators and
consumers have long recognized that the power of video depends on the
harmonious integration of what we hear (the video's audio track) with what we
see (the video's image sequence). Current approaches to video generation either
ignore sound to focus on general-purpose but silent image sequence generation
or address both visual and audio elements but focus on restricted application
domains such as re-dubbing. We introduce Mirage, an audio-to-video foundation
model that excels at generating realistic, expressive output imagery from
scratch given an audio input. When integrated with existing methods for speech
synthesis (text-to-speech, or TTS), Mirage results in compelling multimodal
video. When trained on audio-video footage of people talking (A-roll) and
conditioned on audio containing speech, Mirage generates video of people
delivering a believable interpretation of the performance implicit in input
audio. Our central technical contribution is a unified method for training
self-attention-based audio-to-video generation models, either from scratch or
given existing weights. This methodology allows Mirage to retain generality as
an approach to audio-to-video generation while producing outputs of superior
subjective quality to methods that incorporate audio-specific architectures or
loss components specific to people, speech, or details of how images or audio
are captured. We encourage readers to watch and listen to the results of Mirage
for themselves (see paper and comments for links).