FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized
  Sounds
Authors: name: Yiming Zhang; name: Yicheng Gu; name: Yanhong Zeng; name: Zhening Xing; name: Yuancheng Wang; name: Zhizheng Wu; name: Kai Chen
Published: 2024-07-01 17:35:56

Abstract: We study Neural Foley, the automatic generation of high-quality sound effects
synchronizing with videos, enabling an immersive audio-visual experience.
Despite its wide range of applications, existing approaches encounter
limitations when it comes to simultaneously synthesizing high-quality and
video-aligned (i.e.,, semantic relevant and temporal synchronized) sounds. To
overcome these limitations, we propose FoleyCrafter, a novel framework that
leverages a pre-trained text-to-audio model to ensure high-quality audio
generation. FoleyCrafter comprises two key components: the semantic adapter for
semantic alignment and the temporal controller for precise audio-video
synchronization. The semantic adapter utilizes parallel cross-attention layers
to condition audio generation on video features, producing realistic sound
effects that are semantically relevant to the visual content. Meanwhile, the
temporal controller incorporates an onset detector and a timestampbased adapter
to achieve precise audio-video alignment. One notable advantage of FoleyCrafter
is its compatibility with text prompts, enabling the use of text descriptions
to achieve controllable and diverse video-to-audio generation according to user
intents. We conduct extensive quantitative and qualitative experiments on
standard benchmarks to verify the effectiveness of FoleyCrafter. Models and
codes are available at https://github.com/open-mmlab/FoleyCrafter.