Video-Foley: Two-Stage Video-To-Sound Generation via Temporal Event
  Condition For Foley Sound
Authors: name: Junwon Lee; name: Jaekwon Im; name: Dabin Kim; name: Juhan Nam
Published: 2024-08-21 18:06:15

Abstract: Foley sound synthesis is crucial for multimedia production, enhancing user
experience by synchronizing audio and video both temporally and semantically.
Recent studies on automating this labor-intensive process through
video-to-sound generation face significant challenges. Systems lacking explicit
temporal features suffer from poor controllability and alignment, while
timestamp-based models require costly and subjective human annotation. We
propose Video-Foley, a video-to-sound system using Root Mean Square (RMS) as a
temporal event condition with semantic timbre prompts (audio or text). RMS, a
frame-level intensity envelope feature closely related to audio semantics,
ensures high controllability and synchronization. The annotation-free
self-supervised learning framework consists of two stages, Video2RMS and
RMS2Sound, incorporating novel ideas including RMS discretization and
RMS-ControlNet with a pretrained text-to-audio model. Our extensive evaluation
shows that Video-Foley achieves state-of-the-art performance in audio-visual
alignment and controllability for sound timing, intensity, timbre, and nuance.
Code, model weights, and demonstrations are available on the accompanying
website. (https://jnwnlee.github.io/video-foley-demo)