Sounding that Object: Interactive Object-Aware Image to Audio Generation
Authors: name: Tingle Li, name: Baihe Huang, name: Xiaobin Zhuang, name: Dongya Jia, name: Jiawei Chen, name: Yuping Wang, name: Zhuo Chen, name: Gopala Anumanchipalli, name: Yuxuan Wang

Generating accurate sounds for complex audio-visual scenes is challenging,
especially in the presence of multiple objects and sound sources. In this
paper, we propose an {\em interactive object-aware audio generation} model that
grounds sound generation in user-selected visual objects within images. Our
method integrates object-centric learning into a conditional latent diffusion
model, which learns to associate image regions with their corresponding sounds
through multi-modal attention. At test time, our model employs image
segmentation to allow users to interactively generate sounds at the {\em
object} level. We theoretically validate that our attention mechanism
functionally approximates test-time segmentation masks, ensuring the generated
audio aligns with selected objects. Quantitative and qualitative evaluations
show that our model outperforms baselines, achieving better alignment between
objects and their associated sounds. Project page:
https://tinglok.netlify.app/files/avobject/