Video Object Segmentation-Aware Audio Generation
Authors: name: Ilpo Viertola, name: Vladimir Iashin, name: Esa Rahtu

Existing multimodal audio generation models often lack precise user control,
which limits their applicability in professional Foley workflows. In
particular, these models focus on the entire video and do not provide precise
methods for prioritizing a specific object within a scene, generating
unnecessary background sounds, or focusing on the wrong objects. To address
this gap, we introduce the novel task of video object segmentation-aware audio
generation, which explicitly conditions sound synthesis on object-level
segmentation maps. We present SAGANet, a new multimodal generative model that
enables controllable audio generation by leveraging visual segmentation masks
along with video and textual cues. Our model provides users with fine-grained
and visually localized control over audio generation. To support this task and
further research on segmentation-aware Foley, we propose Segmented Music Solos,
a benchmark dataset of musical instrument performance videos with segmentation
information. Our method demonstrates substantial improvements over current
state-of-the-art methods and sets a new standard for controllable,
high-fidelity Foley synthesis. Code, samples, and Segmented Music Solos are
available at https://saganet.notion.site