Sounding that Object: Interactive Object-Aware Image to Audio Generation
Authors: Tingle Li
Baihe Huang
Xiaobin Zhuang
Dongya Jia
Jiawei Chen
Yuping Wang
Zhuo Chen
Gopala Anumanchipalli
Yuxuan Wang
Venue: ICML 2025 poster (2025)
Keywords: Sound Generation
Audio-Visual Learning
Multi-Modal Learning
Primary area: applications
TL;DR: We interactively generate sounds specific to user-selected objects within complex visual scenes.

Abstract: Generating accurate sounds for complex audio-visual scenes is challenging, especially in the presence of multiple objects and sound sources. In this paper, we propose an interactive object-aware audio generation model that grounds sound generation in user-selected visual objects within images. Our method integrates object-centric learning into a conditional latent diffusion model, which learns to associate image regions with their corresponding sounds through multi-modal attention. At test time, our model employs image segmentation to allow users to interactively generate sounds at the object level. We theoretically validate that our attention mechanism functionally approximates test-time segmentation masks, ensuring the generated audio aligns with selected objects. Quantitative and qualitative evaluations show that our model outperforms baselines, achieving better alignment between objects and their associated sounds.

Sounding that Object: Interactive Object-Aware Image to Audio Generation
Tingle Li Baihe Huang'
Xiaobin Zhuang?
Dongya Jia Jiawei Chen 2 Yuping Wang² Zhuo Chen Yuxuan Wang?
Gopala Anumanchipalii
Abstract
(e.g., crowd noise and wind) even when only one is intended, leading to inaccurate sound textures (McDermott & Simon- celli, 2011).
Generating accurate sounds for complex audio- visual scenes is challenging, especially in the pres- ence of multiple objects and sound sources. In this paper, we propose an interactive object-aware audio generation model that grounds sound gener- ation in user-selected visual objects within images. Our method integrates object-centric learning into conditional latent diffusion model, which learns to associate image regions with their correspond- ing sounds through multi- modal attention. At test time, our model employs image segmentation to allow users to interactively generate sounds at the object level. We theoretically validate that our attention mechanism functionally approxi- mates test-time segmentation masks, ensuring the generated audio aligns with selected objects. Quantitative and qualitative evaluations show that our model outperforms baselines, achieving bet- ter alignment between objects and their associ- ated sounds. Project site: https: / /igit net htt.:lyy..appppll///////abaaeeae
Recent progress in vision-based models (Sheffer & Adi, 2023) relies on analyzing the entire visual scene to pro- duce a single soundtrack, but this broad perspective may overlook subtle yet important sound sources. Text-based models (Liu et al., 2023), on the other hand, face diffi- culties when prompt represents multiple events, either omitting certain sounds or conflating them with others due to entangled feature correlations (Wu et al., 2023). While manually reweighting individual sound events in the dif- fusion latent (Xue et al., 2024) can mitigate these issues, it remains labor-intensive and impractical for large-scale applications. Fundamentally. these challenges arise because real-world sounds are often imbulanced and confourding in complex scenes, making it difficult to disentangle distinct sound sources.
To overcome these limitations, we propose an interactive object-aware audio generation model that grounds each generated sound in specific visual object. Inspired by how humans parse complex soundscapes (Gaver, 1993), model not only processes the overall scene context (e.g., a city street) but also decouples separate events (e.g., honks, footsteps). Drawing on object-centric learning (Greff et al., 2019), we build our model upon a conditional audio genera- tion framework (Liu et al., 2023) and introduce dot-product attention (Vaswani al., 2017) to learn sound- object associations through celf-supervision (Zhao al., 2018; 8; Afouras et al., 2020), which fundamentally overcomes the problem of forgetting or binding sound events.
1. Introduction
Humans naturally perceive the world as ensembles of dis- tinct objects and their associated sounds (Bregman, 1994). For example, in busy city street (Figure 1), we can identify sounds from multiple objects, such as honks, footsteps, and chatters. However, replicating such object-level specificity remains challenging for computational models. Despite notable advances in audio generation (Van Den Doel et al., 2001; Kong et al., 2019; Yang et al., 2023), existing methods often generate holistic soundscapes that fail to accurately reproduce the distinct sounds of specific objects (Pijanowski al., 2011). 1). In complex scenes, models may either forget subtle sounds (e.g., footsteps) bind co-occurring events
To provide finer control and interactivity, we leverage seg- mentation masks (Kirillov et al., 2023) to convert user queries into attention maps at test time, allowing users to select specific objects in image (e.g., car shapes) to gen- erate the corresponding sounds (e.g., engine sounds) with simple mouse clicks. Since these masks guide the model to focus on objects of interest, even subtle sound events can be captured more accurately than with scene-wide analysis alone. Moreover, because the entire image still informs the generation process, selecting multiple objects naturally blends their sounds into consistent environment, rather
University of California, Berkeley Byte Inc. Corre- spondence to: Tingle Li stingle@eces.. chinltt@@occ.bbekkele...dd
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Sounding that Object: Interactive Object-Aware Image to Audio Generation
Input Image
Audio generated to match user-selected objects
Figure 1: Interactive object-aware audio generation. We generate sound aligned with specific visual objects in complex scenes. Users can select one or more objects in the scene using segmentation masks, and our model generates audio corresponding to the selected objects. Here, we show busy street with multiple sound sources (left). After training, our model generates object-specific audio (right), such as crowd noise for people, engine sounds for cars, and blended audio for multiple objects.
than merely layering independent audio clips.
et al., 2024) introduces bounding boxes from external object detectors to generate audio from multiple sound sources. In contrast, our model interactively generates object-specific sound, without requiring explicit object segmentations and representations during training.
Through quantitative evaluations and human perceptual stud- ies, we show that our model generates more complete and contextually relevant soundscapes than existing baselines. Additionally, we provide qualitative results and theoretical analysis to demonstrate that our nism is functionally equivalent to segmentation masks. In summary, our contributions include:
mecha-
Predicting sound from images and text. Generating sounds from visual and textual inputs has gained notable attention recently. Image-hased methods focus on synthe- sizing sounds from visual cues such as physical interactions (Van Den Doel et al., 2001; Owens et al., 2016), human movements (Gan et al., 2020; Su et al., 2021; Ephrat Pe- leg, 2017; Prajwal al., 2020; Hu al., 2021), musical instrument performances (Koepke et al., 2020), and content from open-domain images and videos (Zhou et al., 2018; lashin & Rahtu, 2021; Sheffer & Adi, 2023; Luo et al., 2023; Tang et al., 2023; Wang et al., 2024; Tang et al., 2024; Xing et al., 2024; Zhang et al., 2024; Cheng et al., 2024a; Chen et al., 2024). These approaches typically generate audio that corresponds to the entire visual scene without isolating individual sound sources, resulting in holistic sound gen- eration. Text- base methods aim to produce sounds from textual descriptions using generative models like GANs and diffusion models (Yang et al., 2023; Kreuk et al., 2023; Liu et al., 2023; Huang et al., 2023b; Saito et al., 2025; Evans et al., 2025). However, when prompts contain multiple sound events, these methods often struggle to capture all the desired audio elements (Wu et al., 2023). Unlike these models, our method generates sounds for user-selected one or more objects within images. This offers enhanced control and precision in audio generation.
An interactive object-aware audio generation model that links sounds to user-selected visual objects via masks. mechanism that replaces attention with segmentation masks at test time, allowing fine-grained control over which objects, and thus which sounds, are present in the generated audio.
Empirical and theoretical validation demonstrating our model outperforms baselines in sound-object alignment and user controllability while maintaining audio quality.
2. Related Work
Object discovery. Object-centric learning aims to rep- resent visual scenes as compositions of discrete objects, enabling models to understand and manipulate individual entities within a scene. Unsupervised object discovery meth- ods have been developed to decompose visual scenes into object representations without explicit annotations (Greff al., 2019; Burgess al., 2019; Locatello et al., 2020). In the visial realm, prior studies have explored au- dio localization (A (Arandjelovic & Zisserman, 2018; Rou- ditchenko et al., 2019; Chen et al., 2021b; Mo & Morgado, 2022; Hamilton et al., 2024), separation (Zhao et al., 2018; Afouras et al., 2020), and spatialization (Li et al., 2024b) by using the correspondence between visual objects and their corresponding audio. In concurrent work, SSV2A (Guo
Audio-visual learning. Many works have focused on audio-visual associations due to their inherent correspon- dence in videos. A line of works explores the semantic
2

---

Sounding that Object: Interactive Object-Aware Image to Audio Generation
correspondence, identifying which sounds and visuals are commonly associated with one another (Arandjelovic & Zisserman, 2017). This includes representation learning (Morgado et al., 2021; Huang et al., 2023a), source local- ization (Chen et al., 2021b; Harwath et al., 2018; Chen et al., 2023b), audio stylization (Chen et al., 2022a; Li et al., 2024a), as well as scene classification (Chen et al., 2020; Gemmeke al., 2017; Du et al., 2023a) and generation (Li al., 2022b; Sung-Bin et al., 2023). Other studies leverage spatial correspondence between audio and visual streams (Owens& & Efros, 2018; Korbar et al., 2018; Patrick et al., 2021) to tackle tasks like source separation (Zhao et al., 2018; 2019; Ephrat et al., 2016; Gao et al., 2018; Li et al.,
Mel-spectrograms compression. We compress mel- spectrograms into lower-dimensional latent space using a variational autoencoder (VAE) (Kingma & Welling, 2013). The VAE encodes the mel-spectrogram E I TRTx into a latent representation = с IR. x xd where T and F' are reduced temporal and frequency dimensions, and d is the dimensionality of the latent embeddings.
Textual representation. We represent the textual input ty using a pre-trained text encoder from CLAP (Elizalde et al., 2023), which maps the text into an embedding space Ex(ty)) E RR, where L denotes the embedding dimension. These text embeddings capture semantic information about the desired sound and are used to condition the diffusion model through pross-attention mechanisms (Vaswani et al., 2017).
2020; Chen et al., 2023a; Dong et al., 2022; Cheng et al.,
2024b), Foley sound synthesis (Owens et al., 2016; Du et al., 2023b), and audio spatialization (Gao & Grauman, 2019; Morgado et al., 2018; Yang et al., 2020). Inspired by these works, we aim to generate sound from user-selected objects within images.
Classifier-free guidance. We employ classifier-free guid- ance (CFG) (Ho & Salimans, 2022) to encourage the model to learn both conditional and unconditional denoising, Dur- ing training, we randomly omit the conditioning input b, with a 10% probability. At test time, we use a guidance scale X 2 to interpolate between the conditional and unconditional predictions:
3. Interactive Object-Aware Audio Generation
Our goal to generate sound from user-selected objects within an image in interactive way. We cast this problem by learning the correlation between audio and its correspond- ing visual scene and then using this correlation to predict the sound from the activated region. To achieve this, we: (i) fine-tune an off-the-shelf conditional audio generation model for sound synthesis; (ii) train an audio-guided vi- sual object grounding model isolate the desired object: (iii) theoretically demonstrate the equivalence between the segmentation mask and our grounding model.
Eo En, n, tg) >-eeo(n, n, (1-A n, (2)
where 6z(2n, 0) is the unconditional prediction.
Waveform reconstruction. After generating the latent representation of the audio, we reconstruct the correspond- ing waveform. The decoder part of the VAE transforms the latent representation zu back into mel-spectrogram. Sub- sequently, a pre-trained HiFi-GAN neural vocoder (Kong et al., 2020a) is used to synthesize the time-domain audio waveform from the mel-spectrogram, producing the final audio output.
3.1. Conditional Audio Generation Model
Conditional latent diffusion model. We adopt pre- trained conditional latent diffusion model (Liu et al., 2023) to generate audio conditioned on textual inputs. Building upon latent diffusion models (Ho et al., 2020; Rombach et al., 2022), our model operates in the latent space to im- prove computational efficiency. Specifically, given text prompt t, describing the desired sound and a noise vec- tor e (O, I), the model iteratively denoises the latent variables over N steps to generate the corresponding audio.
3.2. Text-Guided Visual Object Grounding Model
Visual representation. To ground the visual objects corre- sponding to the desired sound, we extract features from the input image using a pre-trained visual encoder. Specifically, we utilize CLIP (Radford al., 2021) to encode the image into set of visual patches embeddings (1,),) PxL where &g is the input image, P the number of patches, and I. denotes the embedding dimension (matching that of the text embeddings). These embeddings capture both semantic and spatial information of the visual scene.
Our model is trained to predict the added noise at each denoising step conditioned on the textual input ty. The training objective minimizes the difference between the predicted noise and the true noise:
Lg =
co(2n.n,4 12.
Scaled dot-product attention. We employ scaled dot- product attention (Vaswani et al., 2017) to fuse the textual and visual inputs, allowing the model to focus on specific objects within the scene. Before computing the attention, the text embeddings (t,g)) and patch embeddings E, (1,))
where Zg is the latent representation of the ground truth audio, zn is the noisy latent at step n. and eg the denoising model parameterized by 0.
3

---

Sounding that Object: Interactive Object-Aware Image to Audio Generation
+
Ref Spectrogram
Output Spectrogram
2n
Zt
20
Latent Encoder
Latent Decoder
Latent Diffusion
Image Prompt
MLP
Image Encoder
HiFi-GAN Vocoder
Text Prompt
Attention
Stylized Audio
"A baby laughs and puppy barks."
Mask/ / Attention Map
Encoder
Model architecture. We encode the reference spectrogram via a pre-trained latent encoder. An image and text prompt are processed by separate encoders, and their embeddings are fused using an attention mechanism to highlight relevant objects. We then feed these conditioned features and noisy latent into a latent diffusion model to generate the object-specific audio. Finally, the latent decoder reconstructs the spectrogram, and a pre-trained HiFi-GAN vocoder generates the final audio waveform. At test time, we replace the attention with a user-provided segmentation mask, and the latent encoder for the reference spectrogram is not used.
Figure 2:
are linearly projected to obtain the query, key, and value matrices. Specifically, compute:
Segmentation mask at test time. After training, we sub- stitute the attention weights derived from the scaled dot- product attention with segmentation masks generated by the segment anything model (SAM) (Kirillov et al., 2023). We rescale the raw outputs of SAM into a normalized mask my E RR., matching the mean and variance of the attention weights. This allows us to generate the desired object's sound by focusing on the regions specified by the segmenta- tion mask. Since SAM's masks can be obtained using either text prompts or point clicks, our model supports interactive image to oudio generation, allowing users to intuitively se- lect objects of interest and generate their associated sounds.
Q = Et, K (1,) V V= E &i,
(3)
where WΩ, WK K and WV are learnable projectors.
We then compute the attention weights between the pro- jected text and each projected image patch, grounding the text in the visual domain:
QK
Attention( K,
K, V) = softmax
V.
(4)
Vd
where de is the dimensionality of the key embedding.
After obtaining the attention output, apply MLP layer (Murtagh, 1991) to further refine the fused representations, which enables the model to attend to image regions corre- sponding to the text input. In this way, we integrate the images ig with the diffusion process, allowing the model to learn to focus on the relevant regions in the image through self-supervision.
3.3. Theoretical Analysis
One may notice that our training pipeline uses both text and image encoders, but the test-time computation involves only the image encoder, where the softmax attention weights are replaced by the segmentation masks. This indicates an out-of out-of-distreee distributiio generalization ability (Lin et al., 2023), where our model trained the softmax attention weights computed by CLAP & CLIP embeddings (Equation 4) is able to generalize well on the segmentation masks computed by SAM. We hypothesize that this ability is rooted in the alignment of contrastive losses and the dot-producct attention mechanism. Recall that the InfoNCE loss (Gutmann & Hyvärinen, 2010; Oord et al., 2018) for the text encoder in
Learnable positional encoding. To enhance the model's 's ability to localize objects within the image, we incorporate learnable positional encodings (Devlin, 2018) into the at- tention mechanism. These encodings are added to the key and value embeddings, providing spatial information about the image patches. By learning positional information, the model can better distinguish between objects different locations, improving grounding precision.

---

Sounding that Object: Interactive Object-Aware Image to Audio Generation
contrastive learning is given by L1 (E), =:
where (,:) is the local inner product under E WK (W2)T/vdl (note that Coontrast is simply the differ- ence between the model's InfoNCE loss and the optimal InfoNCE loss, under the similarity metric (1,)))). Suppose || [V S BB, v is I.p-iipccitz, and f, S are Lipisscciizz, then we have
exp (,
EgT LiL -- log Σド= exp ((E, E()))))
(5)
where (x" 2ay) is the matching text-image pair, and zy. "2...., .fy are the negative image samples associated with x². Notice that if we substitute x² with the text input Eg.
erftest S S Ly (ev + B, . .(Esam + SEV/Econrast )
Ji.N with the image patches kg. and of with the matching image patch (with the text input), then the loss in Equation 5 becomes the Maximum Likelihood Estimation (MLE) loss of the softmax attention weights in Equation (under proper scaling in the exponents). Therefore, the encoders Eu, & are able to assign high attention weights to image patches that match with textual inputs, and low attention weights to irrelevant image patches, working effectively as the seg- mentation mask at test time. As such, the audio generation model is trained with the ability to focus only on the selected objects by segmentation masks.
+ Ly Eeam + Eg.
(6)
Due to space constraints, the proof is deferred to the Ap- pendix F. On the right hand side of Equation 6, the error terms <V: Csam; Ccontrast; Cf have been minimized by mas- sive training (Radford et al., 2021; Elizalde et al., 2023; Kirillov et al., 2023); furthermore, the regularity parameters Ly ,. BB are standard in learning theory literature (An- thony & Bartlett, 1999; Neyshabur et al., 2015; Bartlett et al., 2017) and can be bounded with guarantees (Tsuzuku et al., 2018; Combettes & Pesquet, 2020; Fazlyab et al., 2019). Consequently, Theorem 3.1 implies that the test-time eтror can be effectively upper bounded, hence supporting the substitution of attention weights derived from scaled dot-product attention with segmentation masks generated by the segmentation model during testing. Our theory is further corroborated by empirical findings in Section 4.3, where we observe that using dot- not-product attention weights achieves performance on par with using segmentation masks, while additive attention fails completely.
In the following theorem, we formalize the above argument into a test-time error guarantee. We let ʃ denote the com- position of the trained MLP layers and the trained audio generation model, such that f maps an attention output a to an audio output Sy (on query Q), and let a denote the value metric that maps a sound-image-nan to real number v(s, m) E R. Our goal is to bound the following test error
tuple (s, i, m)
crftest: == F. (p V*), dg Pa) - v (fm V). E m.))
iee the expected (over the randomness of test query q) gap between (i) the value v(f" (p, V"), &g. Pqy achieved by the optimal model (J*, V*) and ground-truth mask Pg, and (ii) the value u(f (m, V), ig. my) of the trained model (f, V) using SAM segmentation m, at test time. Here, f and V are the ground-truth counterpart of f and value ma- trix, Pq E A is the (normalized) ground-truth mask of ((g, for patch index k E {l....,P), ag represents the attention output com- puted by Equation 4. Note that f(m, V). the audio output of the trained model, depends on the segmentation mask m, instead of the ground-truth mask P. or text input ty.
4. Experiments
4.1. Experiment Setup
Dataset. We use AudioSet (Gemmeke et al., 2017) as our primary data source, which consists of 4,616 hours of video clips, each paired with corresponding labels and captions. To ensure audio-visual correspondence, we perform several preprocessing steps similar to Sound-VECaps (Yuan et al., 2024). This reduces the dataset 748 hours of video for training. We then evaluate models on the AudioCaps (also a subset of AudioSet) (Kim et al., 2019), a widely used bench- mark dataset for audio generation. Please see Appendix B for more details on the dataset.
query q such that Pq,s= = P(t,1,,))
Theorem 3.1. Let team == E,,mmm,- - polle,] denote the expected l error of the segmentation model. Let Ev denote the expected error of f and V under the pre-trained CLAP & CLIP embeddings respectively, and Ceontrast denote the expected contrustive loss of the encoders, more precisely,
Model architecture. We employ the pre-trained VAE and HiFi-GAN vocoder in AudioLDM (Liu et al., 2023). The VAE is configured with a latent dimensionality d of channels. For embedding extraction, we utilize the "ViT- B/32" CLAP audio encoder (Elizalde et al., 2023) and the CLIP image encoder (Radford et al., 2021). These em- beddings are then incorporated into the U-Net-based diffu- sion model through cross-attention. We implement a linear noise schedule consisting of N = 1000 diffusion steps, from 31 = 0.0015 to BN = 0.0195. The DDIM sampling
ff = =E (an); Pp)) - Elv(s(ag); Po),, ev= = V Voloo
log (xp (tg)) ((g.a)))
Ceontrust = By B.gddpp
Ee exp (xe (((()()))),(((())))))))
- -Eg -Egdd (-log
5

---

Sounding that Object: Interactive Object-Aware Image to Audio Generation
Method
(t)
()D
KL (1)
(7)
ATC
OVL
RET (t)
REI (1)
REO (t)
Groan Truth
0.962
4.12
0.06
4.02 0.05
4.06 +: 0.07
/
Retrieve Separate (Zhao et 2018)
0.276
4.051
1.572
1.550
0.764
2.73
0.02
2.54 0.05
2.76 1: 0.04
2.49 1: 0.04
Audiol.DM (1.iu et al., 2023)
0.336
3.576
1.537
1.545
0.724
2.83 0.07
3,09
1 0.03
2.92 1: 0.02
2.18 1: 0.04
AudioLDM 2 (Liu et al., 2024)
0,513
2.976
1.162
L.779
0.743
2.9%
0.04
3.19 3.19:: 1: 0.02
3.09
: 0.03
2.47 1 0.01
Captioning al., 2022a)
0.587
2.778
1.364
1.901
0.773
2.84 0.03
3.15 0.04
3.05
1:
Make-an-Audio (Huang et al, 2023b)
0,309
3,555
1.443
1.673
0.712
2.74 0.08
3,06 0:66:00 0.05
2.89 + 0.05
2,08 - 0.04
Im2 (Shetfer Adi, 2023)
0.499
3.602
1,526
1.872
0.798
2.88
0.05
3.02
3.01 = 0.05
2.48 - 0.06
SpecVQGAN (lastin & Raitu, 2021)
0.611
2.515
1,142
1.965
0.825
2.94 0.04
3.26 0.00
3.11 = 0.06
2.51 - 0.04
Foley (Lur al., 2023)
0.683
1.908
0.783
2,010
0.842
3.09
0.06
3.43 1.431.0.00
3.32 1 0.03
2.52 1 0.06
CoDi (Tang al., 2023)
0.672
1.954
0.856
1.936
0.833
3.00
0.04
0.00
3,31 + 0.05
2.34 - 0.02
Seeing & learingg (King et al., 2024)
0.668
1.923
0.794
1.954
0.722
3.08
0.05
3.38 1: 0.04
3.28 1 0.06
2.49 1 0.04
ColeyCrafter (Zhang al., 2024)
2,007
0.811
3.19
0.02
3,48 4:
+:
SSV2A (Guo et 2024)
0.806
1.265
0,525
2,100
0,893
3,22
0.02
3.50
0.03
3,35 = 0.02
3,48 - 0.06
Ours
0.859
1.271
0.517
2.102
0.891
3.31
0.04
3.62 1: 0.05
3.48 1 0.04
3.74 1: 0.07
Table I: Quantitative comparison of our method and baselines across different objective and subjective metrics. The subjective OVL, RET, REI, and REO scores are presented with 95% confidence intervals.
method (Song et al., 2020) is used with 200 steps to facili- tate efficient generation. At test time, we apply CFG with a guidance scale À set to 2.0.
samples. Each participant is provided with an input image, along with the corresponding generated audio, and is asked to rate each sample a scale from to 5 based on sev- eral criteria: (i) Overall Quality (OVL), which evaluates the general quality of the audio; (ii) Relevance to the Text Prompt (RET), which assesses how well the audio matches any associated text description; (iii) Relevance to the Input Image (REI), which judges the alignment between the audio and the visual content; (iv) Relevance to the Selected Ob- ject (REO), which focuses on how well the generated audio aligns with specific object in the visual scene.
Training configuration. To facilitate parallel training, each video's soundtrack is either truncated or zero-padded to achieve a fixed duration of 10 seconds and then converted to a 16 kHz sample rate. We apply a 512-point discrete Fourier transform with a frame length of 64 ms and a frame shift of 10 ms. For each video, single visual frame is randomly chosen to serve as the input image. The model is then trained using the AdamW optimizer (Loshchilov & Hutter, 2017) with a batch size of 64, learning rate of 10-1, 31 = 0.95, 822 = 0.999, c = 10-6, and a weight decay of 10-3 over 300 epochs.
We compare our method with several baseline models, each of which is adapted for our task: (i) Retrieve & Separate (Zhao et al., 2018), a two-stage object-aware model that first retrieves audio based on a text prompt (Elizalde et al., 2023), then separates the object-specific audio from the specified visual object (Zhao et al., 2018); (ii) Audi- oLDM 1 & 2 (Liu et al., 2023; 2024), which we fine-tune on our dataset for a fair comparison; (iii) Captioning (Li et al., 2022a), a cascade model that takes input image, generates captions and feeds them to a pre-trained AudioLDM 2; (iv) Make-an-Audio (Huang et al., 2023b), which supports ei- ther text or image prompts for audio generation. We extract its image branch and fine-tune it our dataset; (v) Im2Wav (Sheffer & Adi, 2023), an image-guided open-domain audio generation model that operates auto-regressively, Since the original model generates only 4 seconds of audio, we retrain it on our dataset to better suit our task; (vi) SSV2A and (vii) CoDi, which are sound-source-aware and any-to-any generative models respectively. We use their image-to-audio branch for comparison; (viii) Spec (lashin & Rahtu, 2021), (ix) Diff-Foley (Luo et al., 2023), (x) Seeing & Hear- ing (Xing et al., 2024), (xi) FoleyCrafter (Zhang et al., 2024), which are video-to-audie generative models. We modify them by using static images (randomly sampling a single frame from each video clip) as input and fine-tuning them on our dataset.
Baselines.
Evaluation metries. We use both objective and subjective metrics (see Appendix C for more evaluation details) to evaluate the performance of our model. For the objective evaluation, we employ several metrics, including Sound Event Accuracy (ACC), which leverages the PANNS model (Kong et al., 2020b) to predict and sample sound event logits based on the annotated labels and then compute the mean accuracy across the dataset. We also measure the semantic alignment between the output and target using four estab- lished metrics: (i) Fréchet Audio Distance (FAD) (Kilgour et al., 2019), which quantifies how close the generated audio is to the real audio in latent space; (ii) Kullback-Leiblee Di- vergence (KL), which assesses the alignment of distributions between the generated and target audio; (iii) Inception Score (IS) (Salimans et al., 2016), which evaluates the diversity of the generated audio; (iv) Audio-Visual Correspondence (AVC) (Arandjelovic & Zisserman, 2017), which measures how well the resulting audio match the visual context.
For subjective evaluation, we conduct a human study to assess the quality and relevance of the generated audio. We present both the holistic samples and the object-selected
6

---

Sounding that Object: Interactive Object-Aware Image to Audio Generation
Prompt Type
Generated Audio
Target Audio
Image
Text
Mask
AudioLDM
AudioLDM 2 Make-an-Audio
Im2Wav
SpecVQGAN
Dift-Folay
Qurs
"Dog growls goose honks"
"Distant chatter with passing"
"Train arriving with people talking"
"Jet roaring with crowd
chearing"
Figure 3: Qualitative model comparison. We show audio generation results for our method and the baselines, each of which is conditioned on an image, text, or segmentation mask.
4.2. Comparison Baselines
Method
Time
Attempts
Satisfaction (1)
Audiol al., 2023)
7.34
3.20
Quantitative results. Table compares our approach baselines the AudioCaps dataset (see Appendix E for evaluations on another dataset), showing that our model out- performs across metrics and generates high-quality audio. In particular, our method achieves the highest ACC scores, demonstrating its ability to generate sounds closely linked to visual objects in the scene. Among baselines, SSV2A performs competitively, likely due to object-level speci- ficity from the external object detector. Diff-Foley, Seeing & Hearing, and foleyCrafter perform competitively, likely due to their contrastive representations enhancing audio-visual consistency. Make-an ane--a-AAe Audio, Wav. and VQGAN achieve reasonable AVC scores but underperform on FAD and KL, suggesting limitations in audio quality. AudioLDM, Captioning, and CoDi show lower ACC and FAD metrics, likely reflecting that CLAP text embeddings fail represent complex audio events. Retrieve & Separate struggles with retrieval in multi-source scenes, limiting its performance in complex scenarios. These results demonstrate our model's s strength in leveraging object-level cues to generate contex- tually relevant sounds.
AudioLDM 2 (Liu et 2024)
5.10
2.40
2,80 = 1.04
FoleyCrafter (Zhang et al., 2024)
3.00
2,80
3,00 1: 1.96
SSV2A (Gno 2024)
2.95
1.80
3.40 + 1.42
Table 2: Interaction satisfaction evaluation user-driven audio generation methods. We report average time (min- utes), number of attempts, and satisfaction score (with 95% confidence intervals).
example, where dog and goose present, baselines generate only dog growls, missing the goose honks, while our method captures both sounds, demonstrating its object- aware capability. Similarly, in the second and third exam- ples, baselines produce only partial sound events, whereas the
our model generates the complete soundscape. nal example, featuring a small jet in the background with cheering crowd, vision-based models fail to detect the jet due to its small size, generating only crowd and wind noises, while text-based models struggle to combine mul- tiple sounds. Our approach captures all relevant sounds, highlighting its ability to generate accurate audio aligned with complex visual scenes. For more direct experience, please view the results video on the project webpage.
For subjective evaluation, randomly select 100 gener- ated samples from the test set, including 50 with manually created segmentation masks for specific objects (see Ap- pendix C). These samples are then rated by 50 participants. Our model achieves the highest average ratings across all measures, with significant lead in REO, indicating better alignment between generated sounds and objects in the im- age. Interestingly, baselines achieve similar REO scores, suggesting limited ability link audio object-level vi- sual cues. Moreover, model consistently outperforms in OVI, RET, and REI, further validating the objective metrics and demonstrating improved contextual alignment.
Interaction satisfaction. We conduct another human study focusing on user-driven audio generation, compar- ing our method to text-based baselines (we exclude those that do not allow user prompting). We ask 5 experienced participants generate "baby laughs and puppy barks" from single image (the one in Figure 2), and we measure the average time taken, the number of attempts required, and 5-point subjective satisfaction score. As shown in Table 2, text-based baselines often miss one of the sounds and require multiple prompt adjustments, leading to higher time and lower satisfaction. Our method, by contrast, consis- tently requires fewer attempts, takes less time, and achieves
Qualitative results. Figure 3 compares our method with generative baselines on the AudioCaps dataset. In the first

---

Sounding that Object: Interactive Object-Aware Image to Audio Generation
Method
ACC
FAD
KL
IS
AVC
Input image
Attention
Mask
Input image
Attention
Mask
Frowen Diffission
0.692
1.543
1.047
1.943
0.733
(0) Muiti-Head Attn.
0.415
2.238
1.903
2.115
0.887
(i) Additive Altn.
0.103
15.747
7,425
1.343
0.137
(1) Txt-Img Artn.
0,856
1.270
0.520
2.097
0.890
(V) Aud-Img Attn.
0.634
1.761
1.232
1.731
0.692
(vi) Mask Training
1.446
0.742
1.947
Figure 4: Visualization results. We visualize the difference between attention maps and segmentation masks using im- ages from Places (Zhou et al., 2017) and text prompts from BLIP (Li et al., 2022a).
Ours
0,859
1.271
0.517
2.102
0.891
Table 3: Quantitative ablation studies on the AudioCaps dataset.
could be because addition operations are not compatible with the contrastive losses used by CLAP & CLIP and seg- mentation masks generated by SAM, which disrupts our grounding model.
higher satisfaction, even for participants already familiar with prompting.
4.3. Ablation Study and Analysis
Table 3 summarizes the ablation experiments. We explore the following model variations: (i) freezing the latent dif- fusion weights rather than fine-tuning them; (ii) replacing single-head attention with multi-head attention; (iii) altering the attention mechanism from dot-product to additive atten- tion; (iv) using text-image attention instead of segmentation masks during inference; (v) substituting text-image atten- tion with mudio-image attention; (vi) using segmentation masks instead of attention during training. We also show additional results in Appendix D.
Choice of attention modality. We investigate the effec- tiveness of text-image attention compared audio-image attention model (Li et al., 2024b). Results show decline in performance, which could be attributed to the inherent limitations of the CLAP model in represent- ing overlapping audio. This limitation probably introduces noise, thereby weakening the model's ability to form audio- visual associations essential for audio generation.
adapted
Role of masking during training and inference. We compare the text-image attention to segmentation masks at test time. Results show that this attention achieves compa- rable performance to segmentation masks, suggesting both methods provide similar guidance (Section 3.3). Notably, using segmentation masks in both training and testing de- grades performance. We hypothesize that masking entire object regions imposes an overly rigid prior, as sound is typ- ically emitted from specific parts (e.g., a dog's head rather than its tail). From a probabilistic viewpoint, hard masks sampled from the ground-trun truth distribution exhibit high vari- ance, whereas soft attention, empowered by CLIP & CLAP, directly approximates the ground-truth distribution. This allows the model to focus on sound-relevant regions while maintaining audio accuracy at test time.
Effect of freezing diffusion weights. We test the impact of freezing the latent diffusion model weights instead of fine-tuning them during training. We observe that freezing the weights degrades the performance, which suggests that fine-tuning is required to achieve more coherent audio.
Impact of attention head. We compare our single- head attention mechanism with the multi-head counterpart (Vaswani et al., 2017). The multi-head approach enhances the alignment between textual inputs and the generated audio, leading stronger correspondence between text descriptions and sound outputs. However, this improve- ment reduces controllability when specifying specific audio characteristies based on the segmentation mask. We con- jecture that this limitation arises because each head in the meati-head attention focuses on different regions of the in- put (Voita et al., 2019; Hamilton et al., 2024). While this strategy increases text-audio alignment, the lack of clear definition for each head's specific scope reduces the inter- pretability of the final results. This likely contributes to the masking results deviating from expectations.
4.4. Cross-dataset Evaluation
Visualization between grounding and masking. In Fig- ure 4, we visualize the comparison between the attention maps generated by our model and the segmentation masks produced by SAM. For this, we use images from Places (Zhou al., 2017) and text prompts derived from BLIP (Li et al., 2022a). To visualize the attention maps, we apply bilinear interpolation match the resolution of the segmen- tation masks. Our results show a strong alignment between our model's 's attention maps and the segmentation masks, providing empirical evidence for the theory in Section 3.3 and the findings of the ablation study in Section 4.3. While the segmentation masks represent a form of hard attention,
Evaluation of attention scoring mechanism. We assess the role of the attention scoring function by replacing dot- product attention with the additive one (Bahdanau, 2014). The additive variant collapses significantly, indicating that segmentation masks are not a suitable replacement for this attention. As explained by the theory in Section 3.3, this
8

---

Sounding that Object: Interactive Object-Aware Image to Audio Generation
Input Image
Mask
Output Audio
Input Image
Mask
Output Audio
image segmentation masks on top. As illustrated in Figure 6, our model generates context-appropriate soundscapes. For instance, it generates rain sounds for dark skies, wind sounds for clear skies, water splashing for watery surfaces, and grass crunching for grassy areas. This demonstrates that our model successfully captures variations in visual textures to generate corresponding audio.
Balancing the volume of different objects. We find in Figure that specifying each object separately tends to assign similar volume to all sources. However, when mul- tiple objects are selected, our method dynamically accounts for context. For example, if large car dominates the scene, its siren may overwhelm subtle ambient sounds, creating a more realistic blend instead of flattening everything to equal volume. Moreover, we quantitatively confirm this context-driven behavior in Table I, 7, and 10, where our object-aware method better reflects how certain sources can overpower others or combine to create natural audio events.
Figure 5: Interactive audio generation. Our model gener- ates object-specific sounds in the city (left) and beach (right) scenes, and composes a complete soundscape when one or more objects are selected.
Input Image
Mask
Output Audio
Input Image
Mask
Output-Adiio
M
w Hoieee
Figure 6: Generating soundscapes from visual texture changes.. We generate different soundscapes by manipulat- ing the visual textures of the same scene, such as changing weather (left) or materials (right).
Interactions among multiple objects. We show in Figure 6 that our method captures interactions, like stick splash- ing water, instead of generating only generic water flowing sounds. These results indicate our model's ability to handle basic multi-object interactions from static images.
directly highlighting entire selected objects, our model gen- erates soft attention maps that probabilistically focus on the sound-relevant areas within each object. This similarity indicates that, through training, our model learns to capture object-specific regions similar to those identified by segmen- tation, achieving the desired grounding in a flexible manner. Furthermore, this observation suggests that attention maps can be replaced with segmentation masks at test time.
5. Conclusion
In this paper, we proposed an interactive object-aware audio generation model, focusing on aligning generated sounds with specific visual objects in complex scenes. To achieve this, we developed diffusion model grounded in object- centric representations, enhancing the association between objects and their corresponding sounds via multi-modal attention. Theoretical analysis demonstrates that our object- grounding mechanism is functionally equivalent to seg- mentation masks. Quantitative and qualitative evaluations show that our model surpasses baselines in sound-object alignment, enabling cross-dataset generalization and user- controllable synthesis. We hope our work not only advances controllable audio generation but also inspires further ex- ploration into the relationships between objects and sounds. We will release code and models upon acceptance.
Interactive audio generation. We ask whether our model will generate object-specific sounds by isolating individual objects within scene. As shown in Figure 5, we use the same image for each scene, separating different objects (cars, people, seagulls, etc.) to generate corresponding audio outputs. The results illustrate that our model successfully learns to generate distinct sounds for each object, such as car engines or footsteps, reflecting their unique sound textures. Furthermore, when multiple objects are selected together, our model is able generate the entire soundscape that represents the scene property. This capability highlights our model's strength in interactively synthesizing audio.
Limitations and broader impacts. Our model shows promising results in generating object-specific sounds from images but has certain limitations. First, relying on static images makes it challenging to produce non-stationary audio synchronized with dynamic events, such impact sounds (Figure 6). Second, it may lack precise control over the type of sound generated for similar objects, leading to ambiguity (e.g., a car might produce a siren or engine noise in Figure 3). Lastly, while useful for content creation like filmmaking, our model could be misused to generate misleading videos.
Sound adaptation to visual texture changes. We explore whether our method can generate soundscapes that adapt to changes in visual textures, inspired by audio-visual video editing (Lee et al., 2023). Starting with images from the Places (Zhou et al., 2017) and Greatest Hits (Owens et al., 2016) datasets, we apply an off-the-shelf image translation model (Park et al., 2020; Li et al., 2022b) to create paired scenes (e.g., sunny-rainy, water-grass), and then overlay full-

---

Sounding that Object: Interactive Object-Aware Image to Audio Generation
Acknowledgment
Conference on Computer Vision and Pattern Recognition, pp. 18858-18868, 2022a.
We thank Ziyang Chen, Hao-Wen Dong, Yisi Liu, and Zhikang Dong for their helpful discussions, and the anony- mous reviewers for their valuable feedback.
Chen, H., Xie, W., Vedaldi, A., and Zisserman, A. Vg- gsound: A large-scale audio-visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 721-725. IEEE, 2020.
Impact Statement
This paper introduces an interactive object-aware audio gen- eration model. trained on a publicly available dataset, i.e. AudioSet, which does not contain personally identifi- able information. We have taken steps ensure compliance with data usage policies, and our model does involve human subjects or raise privacy concerns. We believe our work poses minimal negative ethical impacts and societal implications, as it focuses on enhancing sound-object align- ment in a controlled research environment. However, we encourage responsible use our model, particularly when applied to real- wol-w scenarios.
Chen, H., Xie, W. Afouras, T., Nagrani, A., Vedaldi, A., and Zisserman, A. Audio-visuae synchronisation in the wild, arXiv preprint wrXiv:21/2.044:4 2021a.
Chen, H., Xie, W., Afouras, T., Nagrani, A., Vedaldi, A., and Zisserman, A. Localizing visual sounds the hard way. Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), 2021b.
Chen, J., Zhang, R., Lian, D., Yang. J., Zeng, Z., and Shi, J. iquery: Instruments as queries for audio-visual sound separation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14675- 14686, 2023a.
References
Afouras, T., Owens, A., Chung, S., and Zisserman, A. Self-supervised learning of audio-visual objects from video. In Computer Vision-ECCCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceed- ings, Part XVIII 16, pp. 208-224. Springer, 2020.
Chen, S., Wu, Y., Wang, C., Liu, S., Tompkins, D., Chen, Z., and Wei, F. Beats: Audio pre-training with acoustic tokenizers. arXiv preprint arXiv: 22X.v:2212.000555 2022b.
Chen, Z., Qian, S., and Owens, A. Sound localization from motion: Jointly learning sound direction and camera rotation. arXiv preprint arXiv:2303. 11399 2023b.
Anthony, M. and Bartlett, P. L. Neural Network Learning: Theoretical Foundations. Cambridge University Press, 1999.
Chen, Z.,, Seetharaman, P., Russell, B., Nieto, O., Bour- gin, D., Owens, A., and Salamon, Video-guided fo- ley sound generation with multimodal controls. arXiv preprint 2.11. 17698, 2024.
Arandjelovic, R. and Zisserman, A. Look, listen and learn. In Proceedings of the IEEE international conference on computer vision, pp. 609-617,2011 2017.
Arandjelovic, R. and Zisserman, A. Objects that sound. In Proceedings of the European conference on computer vision (ECCV), pp. 435 451, 2018
Cheng, H. K., Ishii, M., Hayakawa, A., Shibuya, T.. Schwing, A., and Mitsufuji, Y. Taming multimodal joint training for high-quality video-to-audio synthesis. arXiv preprint arXiv:2412.1532. 2024a.
Bahdanau, D. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv: 1et..0494.. 2014.
Cheng, X., Zheng. S., Wang, Z., Fang, M., Zhang, Z., Huang, R., Ma. Z., Ji, S., Zuo, J., Jin, T., et al. Om- nisep: Unifted omni-modality sound separation with query-mixup. arXiv preprint arXiv:2410.21669 2024b.
Bartlett, P. L., Foster, J., and Telgarsky, M. J. Spectrally- normalized margin bounds for neural networks. Advances in neural information processing systems, 30, 2017.
Combettes, P. L. and Pesquet, J.-C. Lipschitz certificates for layered network structures driven by averaged activation operators. SIAM Journal on Mathematics of Data Science, 22):529-5577 2020.
Bregman, A. S. Auditory scene analysis: The perceptual organization of sound. MIT press, 1994.
Burgess, C. P., Matthey, L., Watters, N., Kabra, R., Higgins, I., Bootviiick, M., and Lerchner, A. Monet: Unsupervised scene decomposition and representation. arXiv preprint arXiv: 399,, 2019.
Cramer, A. I.., Wu, H.-H., Salamon, J., and Bello, J. P. Look, listen, and learn more: Design choices for deep audio embeddings. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3852-3856. IEFF, 2019.
Chen, C., Gao, R., Calamia, P., and Grauman, K. Visual acoustic matching. In Proceedings of the IEEF/CVF
10

---

Sounding that Object: Interactive Object-Aware Image to Audio Generation
Devlin, J. Bert: Pre-training of deep bidirectional trans- formers for language understanding. arXiv preprint prXiv:1810.0440 2018.
Gao, R., Feris, R., and Grauman, K. Learning to separate ob- ject sounds by watching unlabeled video. In Proceedings of the European Conference Computer Vision (ECCV), pp. 35-53, 2018.
Dong, H.-W., Takahashi, N., Mitsufuji, Y., McAuley, J., and Berg- Berg-Kirkpaarick, T. Clipsep: Learning text-queried sound separation with noisy unlabeled videos. arXiv 2022.
Gaver, W. W. What in the world do we hear?: An ecolog- ical approach to auditory event pereeption. Ecological psychology. 5(1):1-29, 1993.
preprint arXiv:2212.07000
Du, C., Teng, J., Li, T., Liu, Y., Yuan, T., Wang, Y., Yuan, Y., and Zhao, H. On uni-modal feature learning in supervised multi-modal learning. In International Conference on Machine Learning, pp. 8632-8656. PMLR, 2023a.
Gemmeke, J. F., Ellis, D. P., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., and Ritter, M. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 776-780. IEEE, 2017.
Du, Y., Chen, Z., Salamon, J., Russell, B., and Owens, A. Conditional generation of audio from video via foley analogies. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 2426- 2436, 2023b.
Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K. V., Joulin, A., and Mísra, I. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15180-15190, 2023.
Elizalde, B., Deshmukh, S., Al Ismail, M., and Wang, H. Clap learning audio concepts from natural language su- pervision. In ICASSP 2023-2023 IEFF International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, IEEE, 2023.
Greff, K., Kaufman, R. L.., Kabra, R., Watters, N., Burgess, C., Zoran, D., Matthey, L., Botvinick, M., and Lerchner, A. Multi-object representation learning with iterative variational inference. In International conference on machine learning, pp. 2424-2433. PMLR, 2019.
Ephrat, A. and Peleg. S. Vid2speech: speech reconstruction from silent video. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5095-5099. IEEE, 2017.
Guo, W. Wang, H., Ma, J., and Cai, W. Gotta hear them all: Sound source aware vision audio generation. arXiv preprint arXiv:2411.15444 2024.
Ephrat, A., Mosseri, I., Lang, O., Dekel, T., Wilson, K., Has- sidim, A., Freeman, W. T., and Rubinstein, M. Looking to listen at the cocktail party: A ipeaker-indepennent audio- visual model for speech separation. ACM Transactions on Graphics (TOG), 37)4), 2016.
Gutmann, M. and Hyvärinen, A. Noise-contrastiv estima- tion: A new estimation principle for unnormalized statisti- cal models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 297-304. JMLR Workshop and Conference Proceedings. 2010.
Evans, Z., Parker, J. D., Carr, C., Zukowski, Z., Taylor, J., and Pons, Stable audio open. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2025.
Hamilton, M., Zisserman, A., Hershey, J. R., and Free- man, T. supervised visual grounding of sound and language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 131 13117-13127, 2024.
T. Separating the" chirp" from the" chat": Self-
Fazlyab, M., Robey, A., Hassani, H., Morari, M., and Pap- pas, G. Efficient and accurate estimation of lipschitz constants for deep neural networks. Advances in neural information processing systems, 32, 2019.
Harwath, D., Recasens, A., Surís, D., Chuang, G., Torralba, A., and Glass, J. Jointly discovering visual objects and spoken words from raw sensory input. In Proceedings of the European conference on computer vision (ECCV), pp. 665, 2018
Gan, C., Huang, D., Chen, P., Tenenbaum, J. B., and Tor- ralba, A. Foley music: Learning generate music from videos. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceed- ings, Part XI 16. pp. 758-775. Springer, 2020.
Ho, J. and Salimans, T. Classifier-free diffusion guidance. arXiv preprint arXiv:2207. 12598, 2022.
Gao, R. and Grauman, K. 2.5 visual sound. Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 324-333.3 2019.
Ho, J., Jain, A., and Abbeel, Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-6885 6851, 2020.

---

Sounding that Object: Interactive Object-Aware Image to Audio Generation
Hu, C., Tian, Q.. Li, T., Yuping, W., Wang, Y., and Zhao, H. Neural dubber: Dubbing for videos according to scripts. Advances in neural information processing systems, 34: 16582-16525 2021.
Kong, Q., Xu, Y., Iqbal, T., Cao. Y., Wang, W., and Plumb- ley, M. D. Acoustic scene generation with conditional samplernn. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 925-929. IEEE, 2019.
Huang, P.-Y., Sharma, V., Xu, H., Ryali, C., Fan, H., Li,
Y., Li, S.-W., Ghosh, G., Malik, J., and Feichtenhofer, C. Mavil: Masked audio-video learners, 2023a.
Kong, Q. Cao, Y., Iqbal, T., Wang, Y., Wang, W., and Plumbley, M. D. Panns: Large-scale pretrained audio neural networks for audio pattern recognition. IEEE/ACM Transactions on Audio, Speech, and Language Process- ing, 28:2880-2894 2020b.
Huang, R., Huang, J., Yang, D., Ren, Y., Liu, L., Li, M.,
Ye, Z., Liu, J., Yin, X., and Zhao, 7. Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. In International Conference on Machine Learn- ing (ICML), 2023b.
Korbar. B., Tran, D., and Torresani, L. Cooperative learn- ing of audio and video models from self-supervised syn- chronization. In Proceedings of the Advances in Neural Information Processing Systems, 2018.
Iashin, V. and Rahtu, E. Taming visually guided sound generation. In The British Machine Vision Conference (BMVC), 2021.
Kreuk, F., Synnaeve, G., Polyak, A., Singer, U., Défossez, A., Copet, J., Parikh, D., Taigman, Y., and Adi, Y. Audio- gen: Textually guided audio generation. International Conference on Learning Representations (ICLR), 2023.
Iashin, V., Xie, W., Rahtu, E.,, and Zisserman, A. Synch- former: Efficient synchronization from sparse cues. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. IEEE, 2024.
Lee, S. H., Kim, S., Yoo, I., Yang, E., Cho, D., Kim, Y., Chang, H., Kim, J., and Kim, S. Soundini: Sound- guided diffusion for natural video editing. arXiv preprint arXiv:23044 230400330.0 2023.
Kilgour, K., Zuluaga, M., Roblek, D., and Sharifi, M. Fréchet audio distance: A reference-free metric for evalu- ating music enhancement algorithms. In INTERSPEECH, pp. 2350-2354, 2019.
Li, J., Li, D., Xiong, C., and Hoi, S. Blip: Bootstrapping language-iman pre-training for unified vision-language understanding and generation. In International confer- ence on machine learning, pp. 11888-12900 PMLR, 2022a.
Kim, C. D., Kim, B., Lee, H., and Kim, G. Audiocaps: Gen- erating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume 1 (Long and Short Pupers), pp. 119-132, 2019.
Li, T., Lin, Q.. Bao, Y., and Li, M. Atss-net: Target speaker separation via attention- based neural network. In Inter- speech, pp. 1411-1415, 2020.
Kim, T. K. T test as a parametric statistic. Korean journal of annesthesoology, 68(6)5540-546, 2015.
Li, T., Liu, Y., Owens, A., and Zhao, H. Learning vi- sual styles from audio-visual associations. In European Conference on Computer Vision, pp. 235-252. Springer, 2022b.
Kingma, D. P. and Welling, M. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 2013.
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-. et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4025-4026, 2023.
Li, T. Wang, R., Huang, P.-Y., Owens, A., and Anu- manchipalli, G. Self-supervised visiol soundscape stylization. In Proceedings of the Europeun Conference on Computer Vision, 2024a.
Li, Z Zhao, B., and Yuan, Y. Cyclic learning for binaural audio generation and localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26669-26678, 2024b.
Koepke, A. S., Wiles, O., Moses, Y., and Zisserman, A. Sight sound: An end-to-end approach for visual piano transcription. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1838-1842. IEEE, 2020.
Lin, Y., Chen, M., Wang, W., Wu, B., Li, K., Lin, B., Liu, H., and He, X. Clip is also an efficient segmenter: A text-driven approach for weakly supervised semantic seg- mentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15305 15314, 2023.
Kong, J., Kim, J., and Bae, J. Hifi-gan: Generative ad- versarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 7322--2317, 2020a.
12

---

Sounding that Object: Interactive Object-Aware Image to Audio Generation
Liu, H., Chen, Z., Yuan, Y., Mei, X., Liu, X., Mandic, D., Wang, W., and Plumbley, M. D. Audioldm: Text-to-audio generation with latent diffusion models. In International Conference on Machine Learning (ICML), 2023.
Owens, A., Isola, P., McDermott, J., Torralba, A., Adelson, E. H., and Freeman, W. T. Visually indicated sounds. In Proceedings of the IEEE conference computer vision and pattern recognition, pp.
2016.
Liu, H., Yuan, Y., Liu, X., Mei, X., Kong, Q Tian, Q., Wang, Y., Wang, W., Wang, Y., and Plumbley, M. D. Audioldm 2: Learning holistic audio generation with self-supervised pretraining. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024.
Park, T., Efros, A. A., Zhang, R., and Zhu, J.-Y. Con- trastive learning for unpaired image-to-image translation. In Computer Vision-ECCK 2020: 16th European Con- ference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IX 16, pp. 319-345. Springer, 2020.
Locatello, F., Weissenborn, D., Unterthiner, T., Mahendran, A., Heigold, G., Uszkoreit, J., Dosovitskiy, A., and Kipf, T. Object-centric learning with slot attention. Advances in neural information processing systems, 33:11525-11536 2020.
Patrick, M., Huang, P.-Y., Misra, I., Metze, F., Vedaldi, A., Asano, Y. M., and Henriques, F. Space-time crop & attend: Improving cross-modal video representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11560-10572, 2021.
Loshchilov, I. and Hutter, F. Decoupled weight decay regu- larization. arXiv preprint arXiv: 1.1.:51111.05001, 2017.
Pijanowski, B. C., Villanueva- L. J., Dumyahn. S. L., Farina, A., Krause, B. L.., Napoletano, B. M., Gage, S. H., and Pieretti, N. Soundscape ecology: the science of sound in the landscape. BioScience, (33)2203-2 2011.
Luo, S., Yan. C., Hu, C., and Zhao, H. Diff-foley: Syn- chronized video-to-audio synthesis with latent diffusion models. arXiv preprint 1XXx::3306.120333 2023.
Prajwal, K., Mukhopadhyay, R., Namboodiri, V. P., and
McDermott, J. H. and Simoncelli, E. P. Sound texture per- ception via statistics of the auditory periphery: evidence from sound synthesis. Neuron, 71(5):926-94) 940, 2011.
Jawahar, C. Learning individual speaking styles for ac- curate lip to speech synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern 2020.
McHugh, M. L. Interrater reliability: the kappa statistic. Biochemia medica, 22(3):276-282, 2012.
Recognition, pp.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8766 PMLR, 2021.
Mo, S. and Morgado, P. Localizing visual sounds the easy way. In European Conference on Computer Vision, pp. 218-234. Springer, 2022.
Morgado. P., Vasconcelos, N., Langlois, T., and Wang, O. Self-supervised generation of spatial audio for 360 video. In Advances in Neural Information Processing Systems, 2018.
Ravi, N., Gabeur, V., Hu. Y.-T., Hu, R., Ryali, C., Ma, T., Khedr, H., Rädle, R., Rolland, C., Gustafionn, L., al. Sam 2: Segment anything in images and videos. arXiv preprint 240.:::400.001144 2024.
Morgado, P., Vasconcelos, N., and Misra, I. Audio-visual instance discrimination with mross-modal agreement. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 12475-12486, 2021.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Murtagh, F. Multilayer perceptrons for classification and regression. Neurocomputing, 2(5-6):183-197, 1991.
Neyshabur, B., Tomioka, R., and Srebro, N. Norm-based capacity control in neural networks. In Conference learning theory, pp. 1376-1401. PMLR, 2015.
Rouditchenko, A., Zhao, H., Gan, C., McDermott, J., and Torralba, A. Self-supervised audio-visual co- segmentation. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2357-2361. IEEE, 2019.
Oord, A. d., Li, Y., and Vinyals, O. Representation leamn ing with contrastive predictive coding. arXiv preprint arXiv:1807.03749 2018
Saito, K., Kim, D., Shibuya, T., Lai, C.-H., Zhong, Z., Takida, Y., and Mitsufuji, Y. Soundctm: Unifying score- based and consistency models for bull-band text-to-soum generation. In The Thirteenth International Conference on Learning Representations, 2025.
Owens, A. and Efros, A. A. Audio-visual scene analy- sis with sulf-super multisensory features. In Pro- ceedings of the European conference on computer vision (ECCV), pp. 631 631-648, 2018
13

---

Sounding that Object: Interactive Object-Aware Image to Audio Generation
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Voita, E., Talbot, D., Moiseev, F,, Sennrich, R., and Titov, I. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. arXiv preprint arXiv: 1905.09418 2019.
Sheffer, R. and Adi, Y. I hear your true colors: Image guided audio generation. In ICASSP 2023-2023 IEEE In- ternational Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Wang, H., Ma, J., Pascual, S., Cartwright, R., and Cai, W. V2a-mapper: A lightweight solution for vision-to-audio generation by connecting foundation models. In Proceed- ings of the AAAI Conference on Artificial Intelligence, pp. 15492- 15501, 2024.
Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020.
Wu, H.-H., Nieto, O., Bello, J. P., and Salamon, J. Audio- text models do not yet leverage natural language. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Su. J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing. 568:127063, 2024.
Su, K., Liu, X., and Shlizerman, How does it sound? Advances in Neural Information Processing Systems, 34: 29258-29273, 2021.
Xing. Y., He, Y., Tian, Z., Wang, X., and Chen, Q. Sceing and hearing: Open-domain visual-audio generation with diffusion latent aligners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7151-7161, 2024.
Sung-Bin, K., Senocak, A., Ha. H., Owens, A., and Oh, T.-H. Sound to visual scene generation by audio-to-visual latent alignment. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 6430- 6440, 2023.
Xue, J., Deng. Y., Gao, Y., and Li, Y. Auffusion: Leveraging the power of diffusion and large language models for text- lo-audio generation. arXiv preprint arXiv:2401.01044 2024.
Tang, Z., Yang, Z., Zhu, C., Zeng, M., and Bansal, M. Any- to-any generation via composable diffusion. Advances in Neural Information Processing Systems, 36:16083- 16099, 2023.
Yang, D., Yu, J., Wang, H., Wang, W., Weng, C., Zou, Y., and Yu, D. Diffsound: Discrete diffusion model for text- to-sound generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.
Tang, Z., Yang, Z., Khademi, M., Liu, Y., Zhu, C., and Bansal, M. Codi-2: In-context interleaved and interactive any-to-any generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 37425-27434 2024.
Yang, K., Russell, B., and Salamon,
J. Telling left from right: Learning spatial correspondence of sight and sound. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 9932-9941, 2020.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
Yuan, Y., Jia, D., Zhuang, X., Chen, Y., Liu, Z., Chen, Z., Wang. Y., Wang, Y., Liu, X., Plumbley, M. D., et al. Improving audio generation with visual enhanced caption. arXiv preprint arXiv:2407.0444
2024.
Tsuzuku, Y., Sato, I., and Sugiyama, M. Lipschitz-mant margin training: Scalable certification of perturbation invariance for deep neural networks, Advances in neural information processing systems, 31, 2018.
Zhang, Y., Gu, Y., Zeng, Y., Xing, Z. Wang, Y., Wu, Z., and Chen, K. Foleycrafter: Bring silent videos to life with lifelike and synchronized sounds. arXiv preprint 2024.
Van Den Doel, K., Kry, P. G., and Pai, D. K. Foleyautomatic: physically- based sound effects for interactive simulation and animation. In Proceedings of the 28th annual confer- ence on Computer graphics and interactive techniques, pp. 537-544, 2001.
Zhao, H., Gan, C., Rouditchenko, A., Vondrick, C., Me- Dermott, J., and Torralba, A. The sound of pixels. In Proceedings of the European conference on computer vision (ECCV), pp. 570-586, 2018
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, E., and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 30, 2017.
Zhao, H., Gan, C., Ma, W.- and Torralba, A. The sound of motions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1735-1744, 2019.
14

---

Sounding that Object: Interactive Object-Aware Image to Audio Generation
Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., and Tor- ralba, A. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6))1452- 14644 2017.
Zhou, Y., Wang, Z., Fang, C., Bui, T., and Berg, T. L. Visual to sound: Generating natural sound for videos in the wild. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2550-3558 2018.
15

---

Sounding that Object: Interactive Object-Aware Image to Audio Generation
A. Results Video
140000
120090
We provide a results video on the project webpage, which showcases our model's ability to generate sounds based on the masked object prompts. Specifically, this video demon- strates the following:
100000
Count
80000
Our model can interactively generate object-specific sounds within complex scenes.
40000
Despite being trained on the AudioSet (Gemmeke et al., 2017), our model can be successfully applied to out-of- domain visual scenes, including those from the Places dataset (Zhou al., 2017), the Greatest Hits dataset (Owens et al., 2016), and even random web images.
20000
0.0
0.2
0.4
0.6
6.8
1.0
Match Score
Figure 7: Distribution of matching scores. We present the scores for audio-visual pairs in the AudioSet.
Our model can capture variations in visual textures to generate corresponding audio.
Our model can capture diverse objects within an image and generate sounds more accurately than the baselines.
Input Video
ViTs
Adaptor
B. Dataset Preprocessing Details
Transformer
Classifier
Input Audio
B.I. Dataset Refinement
BEATs
Adaptor
We use the AudioSet (Gemmeke al., 2017) the primary source for this task. The original dataset comprises 4,616 hours of video clips, each paired with corresponding labels and captions. Inspired by Sound- VECaps (Yuan et al., 2024), we apply the following refinement steps to adapt the dataset for our use.
Figure Model architecture of audio-visual matching.
We train model to quantify the correspondence between video and its corresponding soundtrack.
Caption rephrasing. To ensure captions to focus exclu- sively on visible sounding objects, we utilize Llama (Tou- vron et al., 2023) with tailored prompt (Figure 9). Given the video and audio captions, our prompt instructs the model to generate single sentence highlighting the common fea- tures between the audio and visual content. The prompt emphasizes including only events present in both modali- ties, while excluding modality-speciffe details such as overly specific visual features. The model is guided to capture the order and parallel occurrence of events using temporal markers like "and then". "followed by", and "while". This process enhances the consistency between audio and visual descriptions.
Audio-visual matching. To ensure strong correspondence between audio and visual inputs, we train an visial matching model (Figure 8), which consists of a 6-layer non-causal transformer with rotary positional embedding mechanism (Su et al., 2024). Visual embeddings tracted using the ViT-B/16 Transformer module from CLIP (Radford et al., 2021), while audio embeddings are gen- erated using the BEAT's model (Chen et al., 2022b). Both embeddings are then passed through a 3-layer MLP to match 768-dimensional space. The model is trained in a self- supervised manner (Owens & Efros, 2018; Korbar al., 2018), treating audio-visual pairs from the same tempo- ral instance matches and those from different videos mismatches, which allows the model to learn audio-visual correspondences without human annotations.
Audio filtering. We filter out clips containing human vo- calizations (e.g., singing, talking), voiceovers, and music using a sound event detection model (Kong et al., 2020b) and the metadata from AudioSet. This step ensures that the remaining audio data largely consists of ambient and context-specife sounds that more likely to align with the visual content.
For training efficiency, the videos are standardized to frames per second, with each frame resized 224x224 pix- els. During the evaluation, our model achieves accuracy of 91% for matching scenarios and 85% for non-man matamhth scenarios set of 100 matched and 100 mismatched sam- ples, indicating its effectiveness in capturing audio-visual alignment. We use this model to score each clip in the Au- dioSet, with results shown in Figure 7. A threshold of 0.6 is then applied filter the dataset.
After applying these refinement steps, the dataset is reduced to 748 hours of video clips that most likely contain contin- uous sounds throughout each clip and exhibit high audio- visual correspondence.
16

---

Sounding that Object: Interactive Object-Aware Image to Audio Generation
Role-System:
helpful assistant for identifylng audio-visure events and generating sentences. Your task is to identify the overlapping You features between 1 10-second audio and the corresponding visual description, help the generate single sentence caption that represents this intersection.
The caption feature is a sentence generated by an audio-caption model: (enclap (enclap.caption).
The label feature is several audio events that happened in the audio: (audio [abel).
Lastly, the user is given several sentences which are the image description of the scene for each second, connected by "and then".
Please identify all the audio events and visual elements based all three features and conclude one single sentence describe this scene
shared audio-visual events or actions that present sound and sight together. Please emphasize time features to present the order of each event, such as "and then", "followed by", "after" order; "and", "while" etc., for parallel events.
Intersection Focus:
Based on the first caption feature, you might need to change or alter any wrong audio event, improve the sentence with more features,
such the weather, the emotion any people, the description the and
Keep only the features that are common between the audio and visual descriptions. If audio and the visual description, include the final caption.
element is mentioned both the
Omit any feature or detail that present in only one modality. This includes removing overly specific visual details, such as the color, shape, any text or label, name and what people are writing and so on, that do not align with the audio description and vice versa. Please ensure that the final caption accurately reflects the common elements of the audio-visual scene, maintaining the order of occurrence, and capturing the shared background, foreground, and context.
Role-User:
The descriptions of the frames are: (frame cattion)
Figure 9: Prompt for Llama. We extract common features between the audio and visual caption using Llama, ensuring the resulting caption focuses on events present both modalities while avoiding overly specific details.
Other
selecting to guide SAM in generating the mask.
15.5%
Vehiclo
C. Additional Evaluation Details
31.6%
2.4%
Engine
We use the PANNs model" (Kong et al., 2020b) to compute ACC for each audio clip, leveraging annotations from AudioSet. Each audio clip first processed through the pre-trained PANNs model to obtain logit values for all sound event classes, excluding classes like "Speech" and "Music." We then sample the logits for each clip based on its annotated labels in AudioSet. Since these logits are softmax outputs, they represent the model's 's confidence in each sound event, allowing us to interpret them as accuracy scores. Finally, we compute the mean of these sampled logits across all clips to determine the overall ACC score.
ACC.
3.9%
4 7%
Tool
Human
19.7%
15.8%
Animal
Environment
Figure 10: Categorical distribution of the filtered Au- dioSet. We show top& 8 categories derived from AudioSet annotations.
FAD, KL, and IS. We measure FAD, KL, and IS using the AudioLDM-Evaal toolbox'. The reference and gener- ated audio files are organized into separate folders, and the toolbox is run in paired mode.
B.2. Dataset Configuration
AVC. We measure AVC using a two-stream network (Arandjelovic & Zisserman, 2017). One stream extracts audio features, while the other extracts visual features. use OpenI.3* (Cramer al., 2019) to obtain these features and compute the cosine similarity for each image-audio pair.
Figure 10 shows the top-8 categorical distributions, derived from AudioSet (Gemmeke et al., 2017) annotations. We uniformly sample 48 hours across these categories for the test set, with the remaining used for training. Notably, there no overlap between training and testing videos. As most clips contain multiple sound sources, we randomly select 100 examples from the test set to assess our model's ability to generate
*ht h thu.. com / htmmm///iuuiingknggg audi oset_tagging_cmm hhttps://gg thub. cato.occoo/hhhhe oo eval
sounds through human evaluation. For 50 of these samples, we manually create object masks by splitting each caption into object snippets and then randomly
httpps hhttps://gt //gi www..thw..omcoc...ccooooooot.t com/.com/aa roppen1
17

---

Sounding that Object: Interactive Object-Aware Image to Audio Generation
Scale
ACC
FAD (\)
KL (U)
(t)
(t)
Threshold
ACC
FAD (4)
KL (L)
(t
AVC
0.413
0.914
1.336
0.674
0.4
0.521
1.874
0.888
1.432
0.696
À 1.5
0.657
1,558
0.762
1.617
0.751
0.5
0.743
1.536
0.691
1.625
0.774
2.0
0.859
1.271
0.517
2,102
0.891
0.6
0.859
1.271
0.517
2.102
0.891
2.5
0.807
1.440
0.589
2.012
0.853
0.7
0.845
1.387
0.612
1.967
0.882
À = 3.0
0.796
1.482
0.576
2.023
0.841
0.8
0.812
1,501
0.664
2.005
0.879
Table 4: Quantitative results under different CFG scales.
Table 5: Quantitative results under different audio-visual
matching scores.
Specifically, we employ the "env" content type model with a 512-dimensional linear spectrogram representation.
Method
ACC (1)
FAD (L)
KL (U)
18 (1)
AVC (1)
w/o PE w/ PE (Ours)
0.787
1.493
0.674
1.913
0.779
0,859
1.271
0,517
2.102
0.891
Human evaluation. We conducted a human evaluation to assess the quality and relevance of the generated audio using Amazon Mechanical Turk The interface for this study is shown in Figure 11. Each participant was presented with an imput image and the corresponding generated audio, then rated each sample on a scale from 1 to 5 based on the following criteria: (i) Overall Quality (OVL), assess- ing the general audio quality; (ii) Relevance to Input Text (RET), measuring the alignment of the audio with the asso- ciated text description; (iii) Relevance to Input Image (REI), evaluating how well the audio corresponds to the visual con- tent; and (iv) Relevance Selected Object (REO), focusing on the alignment of the audio with specific object in the image.
Table 6: Model performance comparison with and without positional encoding.
in a gradual decline across most metrics.
Different thresholds of audio-visual matching. We test our model's performance across different audio-visual matching thresholds, varying from 0.4 to 0.8 (Figure 7), The same held-out test set is used to assess the metrics, with results presented in Table 5. We empirically find that the model achieves optimal performance at a threshold of 0.6.
Effect of positional encoding. We assess the impact of positional encoding (PE) on our model's performance. As shown in Table 6, removing positional encoding leads to a significant degradation across all metries, highlighting its importance in the model's overall performance.
We randomly selected 100 samples for evaluation, each rated by 50 unique participants to ensure reliability. These sam- ples included both (50%) holistic and (50%) object-specific cases. To control for random responses, we incorporated set of noise-only samples. Consistently low scores for these control samples confirmed the reliability of participants. Ad- ditionally, we ensured that each participant spent at least 90 seconds evaluating each sample to guarantee thoughtful assessment.
Impact of overall scene context. We examine whether capturing the overall scene context benefits audio generation. To this end, we compare the Captioning & Mix baseline, where each detected object in the image is captioned sep- arately, passed to AudioLDM to generate individual audio clips, and subsequently mixed, against the Captioning base- line (as described in Section 4.1) that leverages the full scene. As shown in Table 7, although Captioning & Mix yields more accurate audio events (ACC), the perceptual metrics (FAD, KL, IS, and AVC) consistently favor the full- scene Captioning method. These results suggest that context awareness is crucial for generating high-quality audio.
To further validate our results, we computed the inter-rater reliability using Cohen's kappa (McHugh, 2012), which indicated a substantial agreement among raters (x 0.78). Furthermore, we conducted statistical significance test (paired t-test) (Kim, 2015) between our model and base- lines for each criterion, confirming that the improvements reported are statistically significant (p < 0.01). The final scores presented in the main paper are the mean ratings across all participants.
Choice of segmentation module. We replace SAM with SAM 2 (Ravi et al., 2024), more sophisticated segmen- tation method, and evaluate it the test set. We show in Table 8 that this substitution leads to further gains in genera- tion accuracy and quality, which confirms that more precise segmentation masks benefit our method and aligns well with Theorem 3.I.
D. A Additional Results
Different CFG scales. We evaluate our model's perfor- mance across CFG scales ranging from 1.0 to 3.0. As shown in Table 4, there a consistent improvement in metrics as increases from 1.0 2.0, reaching peak performance at 2.0. However, further increasing À beyond 2.0 results
Synehformer--aased metric. Inspired by Synchformer's contrastive pre-training (lashin al., 2024), we employ
Phttps: w/www//wwwww..//////////.w.. www.nturk.cco
18

---

Sounding that Object: Interactive Object-Aware Image to Audio Generation
Audio Evaluation Study
Sample 1 of 100
Input
leaght papay
Input Image:
Audio Sample:
Rate the Audio Sample:
Overal Quality
| (Pen
Rederaanc to irpul Tual.
Relevance to inpul Image:
[im eaart
Relevance Difect:
Figure 11: Human evaluation interface. We show the interface used for the subjective evaluation of generated audio samples. Participants are presented with input text, an image, and corresponding audio sample, and are instructed to rate the audio on four criteria. All ratings must be completed before advancing to the next sample.
Method
(1)
FAD (J)
KL (4)
(f)
(VC
Method
(?)
Captioning Mix
0.643
7.634
2511
1.443
0.645
Ground Truth
0.66
Captioning
Retrieve & Separate (Zhao al., 2018)
0.29
AudioLDM 1 (Liu al., 2023)
0.24
Table Model performance comparison with and without mixing strategy.
AndioLDM 2 (Liu al., 2024)
0.27
Captioning (1.i et al., 2022.)
0.31
Mako-an-Audio (Huang al., 2023b)
0.19
Im2Wav (Sheffer & Adi, 2023)
0.33
Spe:VQGAN
Diff-Foley (Iuto et al., 2023)
0.39
Method
(f)
(L)
(1)
(t)
AVC
Ours
0.45
w/ SAM
0.859
1.271
0.517
2.102
0.891
SAM2
1.53
0,472
2.295
Table 9: Comparison ImageBind (IB) scores across dif- ferent methods.
Table Evaluation comparison between SAM and SAM 2 modules.
video clips with better audio-visual synchronization. for test- ing. To obtain captions aligned with this dataset, we apply the same refinement procedure described in Appendix B.I. We assess our model and baselines (trained on AudioSet) using the VGG-Sound Syne dataset. As presented in Ta- ble our method outperforms all baselines across multiple metrics, particularly in ACC. These results indicate that model generates more accurate audio that captures the com- plexity of each scene, while preserving audio quality.
ImageBind (Girdhar et al., 2023) to measure audio-visual matching static images. By extracting features from both modalities and computing cosine similarity, we show in Table 9 that our method consistently outperforms baselines on this metric.
E. Additional Dataset Evaluations
VGG-Sound dataset. To further evaluate our method, we evaluate it on the VGG-Sound dataset (Chen et al., 2020),
ImageHear dataset. We also evaluate our method on the ImageHear dataset (Sheffer & Adi, 2023), an image-to- audio benchmark comprising 100 web-sourced images span- ning 30 visual categories (2-8 images per class). Although
which contains in-the-wild audio-visual data collected from YouTube. Following (Chen et al., 2021a), we use VGG- Sound Syne, 14-hour subset that contains about 5,000
19

---

Sounding that Object: Interactive Object-Aware Image to Audio Generation
Method
ACC
FAD (L)
(L)
18 (
AVC
Grand
Retrieve Sepurate (Zhao et 2018)
0,143
4.731
1.726
1.782
0.713
AudioL.DM (1.iu al. 2023)
0.256
3.876
1.634
1.901
0.634
AudioLDM 2 (Liu et al., 2024)
0.401
3.114
1.137
1915
0.687
Captioning al., 2022a)
0.491
2.378
1.089
2.001
Make-on-Audio (Huang al., 2023b)
0.395
3,436
1.571
1.876
0.721
Im2Wav (Shetfer Adi, 2023)
0,412
3,005
1.474
1.894
0.747
SpecVQGAN (lashin & Rahtu, 2021)
0.544
2.722
1.015
1.916
0.796
Dill. Foley (Luo el al., 2023)
0.607
1.834
0.941
2.161
0.851
Ouss
0.761
1.112
0.675
2.342
0.898
Table 10: Additional quantitative comparison of our method and baselines the VGG-Sound Sync dataset.
Method
CS (1)
(CC
ity and Cauchy-Schward inequality,
Make an Andio (Huang al., 2023b)
27.44
0.77
Tm2Wav (Sheffer & Adi, 2023)
9.53
0.49
Eng [DKL (Pq.allug.a))
Ceontrast
SpaeVQGAN (lashin & Rahtu, 2021)
18.98
0.49
Filt-Foley (Luo et al., 2023)
35.12
0.86
1
2 Eg [llaad ug.all2.)
Ours
47.37
0.88
1
2 (E, [llpadd Ug le.)2
Table 11: Additional comparison of our method and base- lines on the ImageHear dataset.
It follows that
Byll - Palleil S v Ccontrustt
each image contains only a single object, which does not align well with our object-aware setting, our method contin- ues to outperform all baselines in both clip-score (CS) and ACC, as reported in Table 11.
Returning to the proof of Theorem 3.1 in the main paper, let Sq := f(ag) = f(u, V) denote the audio output on query q by the trained model. We decompose erftest by
F. Proof of Theorem 3.1
Proof. For notation simplicity, let ug E A denote the softmax attention weight computed on query such that We first state the follow-
erftes
U Ei_, (sp((tt)))) ing lemma.
(Pg kg. Po)l - LE V"), Pa)l
+ By(v(S" (1lg V"), kg. Pa)) -R, E.gl(( (a))) 2q Pa)
("gV")
Lemma F.1. Under the same conditions in Theorem 3.1 of the main puper, we have
+ E,lo(S"(an) (u PT)]
E.g (())) Pa)]
Bulling Pylle, S v 2ccontrast
+ E By(v(f(ag) &g Pq)l Eyg [[ dg. m.g)
Proof. Notice that
+ By(v(s(an); my)1 - B. V), 2g.. m.))
Econtrast
log exp ((
By Lemma F. and 1o S By, we have
Eq
Ek=1 exp (t,) ((g.))))))
A < Ey lbo Lg B | pylley]
Ey,dre |- log Pq.al
S Lo Lg B, v..Vec /econtra
Eug.erp log Pq.d
Since V - Vll S cv and || 1, we have
Ug.d] EE, (DKL(P
B = Eylu(f (ug V.), - ( (ug V), kg.
where DKL denotes the KI. distance. By Pinsker's inequal-
S Lo Lg 1...50 .v..
20

---

Sounding that Object: Interactive Object-Aware Image to Audio Generation
By definition, C S es. Using the definition 6sam
Byllmm P we have
D Eg Lou l|mm Polle]]
Lo Geam
and using || I B, with Lemma F.1,
E s Ley Lig Br l|mm uglles)
S Log log By (esam + v 2econtraatt)
Combining, we have
erttest S Lo Lig (ev+ + By (feam + 2v contrast)
+ Lo Cram + CJ-
This completes the proof.
21


Reviews:
[{"review_id": "7L3ugMoEod", "reviewer": "Reviewer_MZUs", "summary": "This paper proposes an image-to-audio generation method with interactive object-aware design.\nIt mainly concentrates on decoupling separate events in visual scenes, while processing the overall scene context.\nTo train the visual object grounding model, the attention module is designed and substituted with a user-selected mask at inference.\nThe authors also provide a theoretical analysis for such a substitution.\nExperiments provide various ablation analyses and comparisons with baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and time.\n\n**Reliance on SAM.**\n\nFirst, our method does not fundamentally rely on SAM for its performance, but instead benefits from it functionally for enhancing user interactivity.  As shown in Table 2(iv), Theorem 3.1, and Figure 4 of our paper, comparable results can be obtained when SAM is replaced with text-image cross-modal attention at test time. Second, we respectfully disagree with the notion that leveraging strong pre-trained models weakens the contribution. Our use of SAM follows a common and productive paradigm in the community to build upon powerful existing components—examples include LLaVA (LLaMA + visual encoder), ControlNet (Stable Diffusion + extra conditioning), and Prompt-to-Prompt (Stable Diffusion + cross-attention control).\n\n**Limited novelty.**\n\nOur core novelty does not lie in proposing a new network architecture or method purely for state-of-the-art performance. Instead, our contributions are: (1) identifying and posing a surprising self-supervised learning signal—distinct objects and their associated sound—for audio generation, which is in the tradition of ICML papers like SimCLR [Chen et al., ICML 2020] that opened up promising research directions; (2) fundamentally overcoming the challenge of forgetting or binding sound events—by introducing multi-modal dot-product attention for audio generation, grounded in theory; (3) enabling a new interactive setup, where users can control which objects produce sound via simple mouse clicks.\n\n**Clarifying SSV2A.**\n\nSSV2A is an unpublished concurrent work and the code has not been fully released before the submission deadline, so we did not compare with it, per ICML policy. In fact, our method is quite different from SSV2A: (1) our training is self-supervised, while they require bounding boxes from external object detectors such as YOLOv8; (2) we allow fine-grained user control on objects, while their manifold pipeline lacks this capability. As suggested, we now evaluate SSV2A on our test set (the table below), showing that we generate more accurate and comparable high-quality audio.\n\n| Method | ACC (↑) | FAD (↓) | KL (↓) |IS (↑) |AVC (↑)|\n|-|-|-|-|-|-|\n|SSV2A|0.806|**1.265**|0.525|2.100|**0.893**|\n|Ours|**0.859**|1.271|**0.517**|**2.102**|0.891|\n\n**Comparison to additional methods.**\n\nAs suggested, we compare several video (or video + text) based methods (Seeing & Hearing, ReWaS, and FoleyCrafter), noting that ReWaS is concurrent, FoleyCrafter is unpublished, and SonicVisionLM has no public codebase so we do not compare to it. Since these works do not provide training codes, we instead run their public inference scripts on our test set. The table below shows that our method outperforms these baselines, largely because it introduces object-level specificity.\n\n| Method            | ACC (↑) | FAD (↓) | KL (↓)  | IS (↑) | AVC (↑) |\n|-|-|-|-|-|-|\n| Seeing & Hearing|0.668|1.923   | 0.794   | 1.954  | 0.722   |\n| ReWaS|0.694|1.552|1.134| 1.938  | 0.704|\n| FoleyCrafter      |0.732|1.760| 0.665   | 2.007  | 0.811|\n| Ours              |**0.859**|**1.271**|**0.517**|**2.102**|**0.891**|\n\n**Constrained dataset evaluation.**\n\nWe have provided quantitative evaluations on the commonly used AudioSet and VGGSound Sync (Tables 1 & 8 of our paper), which feature in-the-wild videos, and we include out-of-distribution examples (Section 4.4 of our paper) plus a demo video (Appendix A), using inputs from the internet or different datasets (Places & the Greatest Hits). These studies, conducted across diverse datasets at scale, demonstrate our model’s generalization capability.\n\n**Extending to video.**\n\nThanks for pointing out this important future direction and broader impact. Our paper’s main contribution is object-aware, user-driven audio generation based on static images. Images allow us to cleanly isolate object-to-audio relationships and provide more intuitive user control, while temporal dynamics (in videos) entail additional complexities such as motion tracking and scene changes, which lie beyond our paper’s current scope.\n\n**Comparison to frozen-diffusion.**\n\nNeither SSV2A nor ReWaS reports experiments contrasting frozen vs. fine-tuned diffusion, so they do not quantify how full adaptation might improve cross-modal alignment. The ablation in Table 2(i) of our paper reveals that freezing simplifies training but hinders fine-grained alignment (especially at the object level), leading to higher FAD. By fully fine-tuning, we reduced FAD and improved ACC because the model can capture richer object-specific cues.\n\n**Clarifying video-based baselines.**\n\nAs described on the right side of Line 288-289 of our paper, we randomly sampled a single frame from each video clip as input and fine-tuned them on our dataset.\n\n**Missing references.**\n\nThanks for pointing these out. We will add comparisons (see above) and discussions (the right side of Line 73–95 of our paper) on the suggested papers in the revised version.", "claims_and_evidence": "The problem claimed by the authors is valid, and they propose an interesting approach to incorporating user interaction into the visual-to-sound generation task.", "methods_and_evaluation_criteria": "One concern is that this cross-modal attention approach is no longer novel and relies on a well-trained, high-performance masked model during testing, which could lead to significant computational overhead. Another concern is that a similar method [1] already exists, making it necessary to clarify the methodological and experimental differences between this work and previous studies.\nThe evaluation metrics are appropriate. However, it is unclear how existing video-based approaches have been modified in this work.\n\n[1] W. Guo et al., \"Gotta Hear Them All: Sound Source-Aware Vision to Audio Generation\"", "theoretical_claims": "I don't have tackling points for theoretical analysis (sec.3.3).", "experimental_designs_or_analyses": "The ablation study and analysis provided by the authors, particularly Table 2, serve as strong evidence to support their claims.\nHowever, the primary baselines used for comparison are methods published up until 2023 (except for AudioLDM2, but it is a text-to-audio method), and there is a lack of performance comparison with more recent approaches. \nAdditionally, the evaluation is limited to a constrained set of datasets. \nThe authors follow multiple steps to refine the datasets, similar to Sound-VECaps, but evaluating the model on such a highly curated dataset may be insufficient to demonstrate its generalization capability.", "relation_to_broader_scientific_literature": "The paper addresses the training of an object-aware (grounding) module for interactive audio generation. However, the proposed method is limited to images rather than videos. To enhance its applicability, an extended solution for video processing is needed.", "essential_references_not_discussed": "Recent visual-to-sound generation methods have been missed in this paper.\nThe authors discussed previous approaches that were published until 2023.\nThis is my primary reason for rejection, as demonstrating superiority through performance comparisons with recent methods is crucial. \n\n[1] Y. Xing et al., \"Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners\".\n\n[2] Y. Jeong et al., \"Read, Watch and Scream! Sound Generation from Text and Video\".\n\n[3] Z. Xie et al., \"Sonicvisionlm: Playing sound with vision language models\".\n\n[4] Y. Zhang et al., \"Foleycrafter: Bring silent videos to life with lifelike and synchronized sounds\".", "other_strengths_and_weaknesses": "n/a", "other_comments_or_suggestions": "n/a", "questions_for_authors": "1. Recent approaches (see `Essential References Not Discussed') utilize not only video but also text as input. \nSince this method also processes textual information, a comparison with such approaches is necessary.\n\n2. Moreover, recent methods that utilize AudioLDM typically freeze the diffusion model during use. \nA comparison of such approaches such as [1] and [2] with its frozen diffusion (Table 2(i)) is also necessary.\nI am also curious about why the FAD score of the frozen model in Table 2 differs significantly from the original model.\n\n3. If I was not missing this part, I recommend explaining how the video-based model was processed. \n\n\n[1] W. Guo et al., \"Gotta Hear Them All: Sound Source-Aware Vision to Audio Generation\".\n[2] Y. Jeong et al., \"Read, Watch and Scream! Sound Generation from Text and Video\".", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mr1E5egf6h", "reviewer": "Reviewer_kpPn", "summary": "This paper introduces an object-aware image-to-audio generation framework built on top of pretrained AudioLDM. Given user-provided segmentation mask, the I2A generation method can generate the object-aligned sound. Experiments on AudioSet and VGGSound Sync datasets show the proposed method outperforms selected vision-to-audio generation algorithms in both objective and subjective evaluation. They also showcase the model's generation control by interactively editing the mask of visual objects in a soundscape scene.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and time.\n\n**Clarifying caption-based methods.**\n\nIn fact, we have evaluated two caption-based variants in Table 1 & 7 of our paper. In Captioning, we generated a single caption from the entire image and fed it to a pre-trained AudioLDM 2. In Captioning & Mix, we generated separate captions for each detected object, synthesized individual audio clips with AudioLDM 2, and mixed them. As expected, both methods perform worse than ours, but interestingly Captioning outperforms Captioning & Mix, suggesting that caption-based methods do not fully address context or proportional mixing—an observation also noted on the right side of Line 45-48 in our paper.\n\n**Region-based audio correlation.**\n\nWe demonstrate in Figure 5 and the demo video of our paper that naive region-based layering tends to assign similar loudness to all sources, leading to flat, unrealistic soundscapes. In contrast, our method dynamically accounts for context.  For example, if a large car dominates the scene, its siren may overwhelm subtle ambient sounds, creating a more realistic blend instead of flattening everything to equal volume. Likewise, Figure  6 of our paper shows that our method captures interactions—like a stick splashing water—instead of generating only generic water flowing sounds. Moreover, we quantitatively confirm this context-driven behavior on AudioSet and VGGSound Sync in Table 1 & 8 of our paper, where our object-aware method better reflects how certain sources can overpower others or combine to create natural audio events.\n\n**Clarifying SSV2A.**\n\nSSV2A is an unpublished concurrent work and the code has not been fully released before the submission deadline, so we did not compare with it, per ICML policy. In fact, our method is quite different from SSV2A: (1) our training is self-supervised, while they require bounding boxes from external object detectors such as YOLOv8; (2) we allow fine-grained user control on objects, while their manifold pipeline lacks this capability. As suggested, we now evaluate SSV2A on our test set, showing that we generate more accurate and comparable high-quality audio:\n\n| Method | ACC (↑)| FAD (↓) | KL (↓) | IS (↑) |AVC (↑)|\n|-|-|-|-|-|-|\n|SSV2A|0.806|**1.265**|0.525|2.100|**0.893**|\n|Ours|**0.859**|1.271|**0.517**|**2.102**|0.891|\n\n**Additional dataset evaluation.**\n\nThe ImageHear dataset contains only a single object per image, which does not align well with our object-aware setting so we did not evaluate on it. However, as suggested, we now compare our method to both image- and video-based baselines using metrics from ImageHear. As shown in the table below, our method continues to outperform them:\n\n| Method         | CS (↑) | ACC (↑)  |\n|----------------|--------|----------|\n| Make-an-Audio  | 27.44  | 77.31%   |\n| Im2Wav         | 9.53   | 49.14%   |\n| SpecVQGAN      | 18.98  | 48.76%   |\n| Diff-Foley     | 35.12  | 86.45%   |\n| Ours           | **47.37**  | **88.32%**   |\n\n**Missing references.**\n\nThanks for pointing these out. We will include these references in the revised version. We also clarify that, as noted on the right side of Line 73-95 of our paper, our method centers on user-selected object-aware audio generation in static images, allowing finer control than the concurrent work SSV2A’s bounding-box method (as discussed above) and the broader scene-level approaches like V2A-Mapper or Seeing & Hearing.", "claims_and_evidence": "The claim of object-aware image-to-audio generation is supported with experiments on AudioSet and VGGSound Sync. \n\nHowever, I'm curious about how naive method could solve the problem. For example, what if we use a captioning model to describe the user-selected region and then put the description to Text-to-Audio generation model such as AudioLDM? \n\nI understand there are subtle visual details that cannot be described by captioning model. Then a comparison to similar object-level image-to-audio generation approaches might be necessary. The compared methods are all scene-level V2A models which do not address object-level sound generation as intended in the proposed task. Although they are not intended to solve interactive object-level I2A, it seems SSV2A [1] also has the ability to generate object-level sound when specifying the visual region. \n\n[1] Guo et al., Gotta Hear Them All: Sound Source Aware Vision to Audio Generation, 2024", "methods_and_evaluation_criteria": "The evaluation metrics make sense in evaluating the alignment between the object and generated sound.\n\nHowever, since the authors propose to solve the image-to-audio generation task, experiments instead were performed on video datasets where image frame is selected. How about the performance on image dataset ImageHear[2]?\n\n[2] Sheffer and Adi, I Hear Your True Colors: Image Guided Audio Generation, ICASSP 2023", "theoretical_claims": "The theoretical analysis is neatly presented and looks good to me.", "experimental_designs_or_analyses": "I'm curious about how naive method could solve the problem. For example, what if we use a captioning model to describe the user-selected region and then put the description to Text-to-Audio generation model such as AudioLDM? \n\nI understand there are subtle visual details that cannot be described by captioning model. Then a comparison to similar object-level image-to-audio generation approaches might be necessary. The compared methods are all scene-level V2A models which do not address object-level sound generation as intended in the proposed task. Although they are not intended to solve interactive object-level I2A, it seems SSV2A [1] also has the ability to generate object-level sound when specifying the visual region. \n\n[1] Guo et al., Gotta Hear Them All: Sound Source Aware Vision to Audio Generation, 2024", "relation_to_broader_scientific_literature": "This paper proposes to solve object-aware image-to-audio generation where user can specify random regions by providing segmentation mask. I'm concerned about the necessity of the solution. Instead of introducing any training, what's the performance of just using existing captioning model to describe the masked region and then call a T2A model? According to the demo videos, there seems no correlation in identity between the sound generated by single object region and the larger region including the object. Then it seems text prompt could already suffice.", "essential_references_not_discussed": "The paper introduces object-aware image-to-audio generation but misses the mention of a highly related work SSV2A [1]. \n\n[1] Guo et al., Gotta Hear Them All: Sound Source Aware Vision to Audio Generation, 2024\n\nAlso citations to some seminal works in V2A are missing:\n\n[3] V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Models, AAAI 2024\n\n[4] Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners, CVPR 2024", "other_strengths_and_weaknesses": "The paper is well presented and easy to follow.\nI appreciate the demo videos which are very helpful in understanding the quality of the generated sound.\nThe proposed method seems simple and effective in grounding object representation in audio diffusion model.\n\nHowever my main concern is about the experimental design/analysis where a critical comparison to a naive method and a highly-related method is missing. The work could be more convincing if including this.", "other_comments_or_suggestions": "No.", "questions_for_authors": "I'm curious about how naive method could solve the problem. For example, what if we use a captioning model to describe the user-selected region and then put the description to Text-to-Audio generation model such as AudioLDM? \n\nI understand there are subtle visual details that cannot be described by captioning model. Then a comparison to similar object-level image-to-audio generation approaches might be necessary. The compared methods are all scene-level V2A models which do not address object-level sound generation as intended in the proposed task. Although they are not intended to solve interactive object-level I2A, it seems SSV2A [1] also has the ability to generate object-level sound when specifying the visual region. \n\n[1] Guo et al., Gotta Hear Them All: Sound Source Aware Vision to Audio Generation, 2024", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YSmG5LgMcD", "reviewer": "Reviewer_XU2w", "summary": "This paper proposed a novel object-aware audio generation model, that supports the interaction with users. This work achieves fine-grained control over which objects, and thus which sounds, are present in the generated audio. Empirical and theoretical validation demonstrating the strong performance of the model and user controllability while maintaining audio quality. The demo videos in the suppl. show some high-quality cases, which further demonstrate the high performance of the proposed framework.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and time.\n\n**Generating audio from multiple objects.**\n\nIn fact, we showed in Figure 5 and the demo video of our paper that our method accepts multi-object masks (including more than two objects) to generate an audio mixture that reflects each selected object. We also evaluated a multi-object subset in human perceptual study (the last four columns of Table 1 in our paper), further confirming that our model can handle multiple sources in a single scene.\n\n**Interactions among multiple objects.**\n\nWe illustrated in Figure 6 and the demo video of our paper that a single image showing a stick in water or grass can generate splashing or rustling by leveraging full-image segmentation masks. These results indicate our method’s ability to handle basic multi-object interactions from static images. However, we also note that more intricate physical interactions often rely on temporal cues, which are generally studied in video and lie beyond our current single-image scope.\n\n**Linear vs. MLP.**\n\nOur method used a lightweight MLP to capture nonlinear cross-modal interactions more effectively than a simpler linear layer. As shown in the first table below, retraining with a linear projection yields lower performance. Importantly, our MLP added roughly 1% more parameters to the AudioLDM backbone (see the second table below), resulting in a minimal additional computation cost.\n\n| Method     | ACC (↑) | FAD (↓) | KL (↓) | IS (↑) | AVC (↑) |\n|------------|---------|---------|--------|--------|---------|\n| Linear     | 0.811   | 1.432   | 0.605  | 1.974  | 0.876   |\n| MLP (Ours) | **0.859**   | **1.271**   | **0.517**  | **2.102**  | **0.891**   |\n\n| Method      | #Param | MACs  | FLOPs |\n|-------------|--------|-------|-------|\n| AudioLDM    | 317M   | 317M  | 587M  |\n| Linear      | 0.26M  | 0.26M | 0.5M  |\n| MLP (Ours)  | 3.15M  | 3.15M | 6.29M |\n\n**Comparison to CoDi.**\n\nAs suggested, we evaluate CoDi (we do not compare CoDi 2 since it is not open-source) on our test set in the table below. We find that our model still holds better performance across all metrics. Unlike CoDi’s global any-to-any generation, our model explicitly allows users to select specific objects within an image for fine-grained control in audio generation.\n\n| Method | ACC (↑) | FAD (↓) | KL (↓) | IS (↑) | AVC (↑) |\n|--------|---------|---------|--------|--------|---------|\n| CoDi   | 0.672   | 1.954   | 0.856  | 1.936  | 0.833   |\n| Ours   | **0.859**   | **1.271**   | **0.517**  | **2.102**  | **0.891**   |\n\n**Interaction satisfaction evaluation.**\n\nAs suggested, we conduct a human study focusing on user-driven audio generation, comparing our method to text-based baselines (we exclude image- and video-based baselines, as they do not allow user prompting). We ask 5 experienced participants to generate \"baby laughs and puppy barks\" from a single image (Figure 2 of our paper), and we measure the average time taken (in minutes), the number of attempts required, and a 5-point subjective satisfaction score (with 95% confidence intervals). As shown in the table below, text-based baselines often miss one of the sounds and require multiple prompt adjustments, leading to higher time and lower satisfaction. Our method, by contrast, consistently requires fewer attempts, takes less time, and achieves higher satisfaction—even for participants already familiar with prompting. We will include these findings in the revised version.\n\n| Method     | Time (↓) | Attempts (↓) | Satisfaction (↑) |\n|------------|----------|--------------|-------------------|\n| AudioLDM 1 | 7.34     | 3.20         | 2.00 ± 0.88       |\n| AudioLDM 2 | 5.10     | 2.40         | 2.80 ± 1.04       |\n| Ours       | **2.67**     | **1.60**         | **3.60 ± 0.68**       |\n\n**Missing references.**\n\nThanks for pointing these out. We will include CoDi and CoDi 2 in the revised version. We also clarify that, as noted on the right side of Line 73-95 of our paper, our method differs by enabling object-aware audio generation in static images rather than full video or text-centric approaches. We ground audio in user-selected objects, thus offering fine-grained controllability compared to CoDi’s global any-to-any generation.", "claims_and_evidence": "The main claims are two: 1) Interactive object-aware audio generation; and 2) fine-grained control over the objects.\n\nThe quantitive and qualitative results can support these claims.", "methods_and_evaluation_criteria": "**Method:**\n\n* The proposed framework demonstrates strong performance in generating audio based on a selected object. However, an important question arises: Can the model generate audio from more than two selected objects? Additionally, can the model simulate interactions among these selected objects? This capability needs further exploration and analysis.\n* Regarding the Multi-Layer Perceptron (MLP) designed within the framework, the author should provide a more comprehensive analysis of its effectiveness. Specifically, if the MLP is used merely to map the latent features of the Image Encoder to the latent space of the diffusion model, why not use a simpler linear layer instead? A comparative evaluation could clarify the necessity of using an MLP in this context.\n\n**Evaluation:**\n\n* The paper omits several important baseline models, such as CoDi [1] and CoDi-2 [2], which are relevant for a more robust comparison. Including these models would strengthen the evaluation and provide a clearer benchmark for performance.\n\n* For the human evaluation, the key purpose of such an evaluation is to validate user satisfaction with interacting with the model. However, the current experiment does not effectively fulfill this goal. To better validate the interaction capabilities, I recommend conducting more user studies in the rebuttal, particularly those aimed at evaluating interaction satisfaction.\n\n[1] Tang, Zineng, et al. \"Any-to-any generation via composable diffusion.\" Advances in Neural Information Processing Systems 36 (2023): 16083-16099.\n\n[2] Tang, Zineng, et al. \"Codi-2: In-context interleaved and interactive any-to-any generation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "theoretical_claims": "There is a clear proof of the proposed technology.", "experimental_designs_or_analyses": "Refer to the \"Methods And Evaluation Criteria\".", "relation_to_broader_scientific_literature": "None", "essential_references_not_discussed": "[1] Tang, Zineng, et al. \"Any-to-any generation via composable diffusion.\" Advances in Neural Information Processing Systems 36 (2023): 16083-16099.\n\n[2] Tang, Zineng, et al. \"Codi-2: In-context interleaved and interactive any-to-any generation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "other_strengths_and_weaknesses": "Please refer to \"Methods And Evaluation Criteria\".", "other_comments_or_suggestions": "If my problem is solved well, I will raise my score.", "questions_for_authors": "Please refer to \"Methods And Evaluation Criteria\".", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nX3r4RuxR9", "reviewer": "Reviewer_wC89", "summary": "This paper introduces a new object-level video-to-audio generation method that exploits SAM to specify which object in a video should have sound and the AudioLDM architecture to generate the sound. During training, scaled dot-product attention between text embedding and patch-wise image embedding is computed and fed into AudioLDM, while SAM's mask is used during inference", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and time.\n\n**Clarifying AVC.**\n\nIn fact, we employed AVC to measure the semantic (instead of temporal) correspondence between audio and image, since our primary goal is to generate audio aligned with objects from image (not video).\n\n**Why images only?**\n\nOur paper’s main contribution is object-aware, user-driven audio generation based on static images. Focusing on images allows us to cleanly isolate object-to-audio relationships and provide more intuitive user control. While incorporating temporal dynamics (in videos) is a natural extension, it entails additional complexities such as motion tracking and scene changes, which lie beyond our paper’s current scope. However, we agree that expanding to video is an important next step.\n\n**Additional Synchformer-based metric.**\n\nWe note that Synchformer specifically targets audio-visual synchronization for video-to-audio generation and does not directly apply to the image-to-audio task. However, inspired by Synchformer’s contrastive pre-training (like DeSync in MMAudio and AV-Sync in MultiFoley), we employ ImageBind [Girdhar et al., CVPR 2023] to measure audio-visual matching on static images. By extracting features from both modalities and computing cosine similarity, we show in the table below that our method consistently outperforms all baselines on this metric.\n\n| Method               | IB (↑) |\n|-----------------------|--------|\n| Ground Truth                    | 0.66   |\n| Retrieve & Separate   | 0.29   |\n| AudioLDM 1           | 0.24   |\n| AudioLDM 2           | 0.27   |\n| Captioning           | 0.31   |\n| Make-an-Audio        | 0.19   |\n| Im2Wav               | 0.33   |\n| SpecVQGAN            | 0.37   |\n| Diff-Foley           | 0.39   |\n| Ours                 | **0.45**   |\n\n**Comparison to SAM 2.**\n\nAs suggested, we replace SAM with SAM 2 and evaluate our method on the test set. We show in the table below that this substitution leads to further gains in generation accuracy and quality, which confirms that more precise segmentation masks benefit our method and aligns well with Theorem 3.1 of our paper.\n\n| Method    | ACC (↑) | FAD (↓) | KL (↓) | IS (↑) | AVC (↑) |\n|-----------|---------|---------|--------|--------|---------|\n| w/ SAM    | 0.859   | 1.271   | 0.517  | 2.102  | 0.891   |\n| w/ SAM 2  | **0.881**   | **1.153**   | **0.472**  | **2.295**  | **0.936**   |\n\n**Balancing the volume of different objects.**\n\nWe demonstrate in Figure 5 and the demo video of our paper that specifying each object separately tends to assign a similar volume to all sources. However, when multiple objects are selected, our method dynamically accounts for context. For example, if a large car dominates the scene, its siren may overwhelm subtle ambient sounds, creating a more realistic blend instead of flattening everything to equal volume. Moreover, we quantitatively confirm this context-driven behavior in Table 1, 7, and 8 of our paper, where our object-aware method better reflects how certain sources can overpower others or combine to create natural audio events.\n\n**Generating off-screen sound.**\n\nOur current method grounds audio into visible objects via segmentation masks. To incorporate off-screen events, we will add another textual cue (like background music) as extra conditioning for content that lacks a corresponding visual region. This requires no changes to segmentation since off-screen sources are not maskable. While off-screen sound lies outside our paper’s main scope, we agree it is a valuable direction for user-driven, interactive audio generation.\n\n**Missing references.**\n\nThanks for pointing these out. We will include the suggested references in the revised version. We also clarify that, as noted on the right side of Line 73-95 of our paper, our method differs by enabling object-aware audio generation from static images rather than full video or text-centric approaches. We ground audio in user-selected objects, thus offering fine-grained controllability compared to broader scene-level or textual conditioning alone.", "claims_and_evidence": "The author claims that exisiting video-to-audio models are designed to generate sound mixtures rather than object-level sounds, which limits creators from manipulating each sound separately. The proposed method performs better than some of the exisiting baselines in terms of audio quality and temporal alignment", "methods_and_evaluation_criteria": "During training, softmax attention weights between CLIP and CLAP embeddings are computed. In inference, SAM's masks are used instead to condition the audio generative model. Evaluation was done in both objective and subjective ways. AVC was employed to evaluate the temporal alignment between video and audio. In addition to standard metrics such as FAD, IS, a subjective test was conducted to support the evidence of the proposed claim", "theoretical_claims": "Theoritical analysis on the error bound is included in this paper. The derivation of eq. (6), written in Appendix, looks correct. The evaluation results also support this claim", "experimental_designs_or_analyses": "The paper mostly follows standard evaluation settings used in the past literature, such as FAD and KL. Audio-visual alignment is evaluated with AVC, which, to my knowledge, has not been used in prior works", "relation_to_broader_scientific_literature": "Most of the existing literature in this field focuses on mixture generation rather than object-level sound generation, which opens up a new door to the field", "essential_references_not_discussed": "Video-to-audio generation papers appeared 2024-present are completely missing in Related Works: \n- Video-Guided Foley Sound Generation with Multimodal Controls [CVPR'25]\n- Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis [CVPR'25]\n- Temporally Aligned Audio for Video with Autoregression [ICASSP'25]\n- STA-V2A: Video-to-Audio Generation with Semantic and Temporal Alignment [ICASSP'25]\n- Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow Matching [NeurIPS'24]\n- Tell What You Hear From What You See - Video to Audio Generation Through Text [NeurIPS'24]\n- FoleyGen: Visually-Guided Audio Generation [MLSP'24]\n- V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Models [AAAI'24]\n\nSome of the latest text-to-audio generation papers would be nice to include: \n- SoundCTM: Unifying Score-based and Consistency Models for Full-band Text-to-Sound Generation [ICLR'25]\n- Stable Audio Open [ICASSP'25]\n- SpecMaskGIT: Masked Generative Modeling of Audio Spectrograms for Efficient Audio Synthesis and Beyond [ISMIR'24]\n\nVideo and/or image-queried sound separation are also relevant: \n- OmniSep: Unified Omni-Modality Sound Separation with Query-Mixup [ICLR'25]\n- iQuery: Instruments as Queries for Audio-Visual Sound Separation [CVPR'23]\n- CLIPSep: Learning Text-queried Sound Separation with Noisy Unlabeled Videos [ICLR'23]", "other_strengths_and_weaknesses": "[Strength]\n- The proposed idea to use SAM at inference time is very new and would give a straightforward user interface for creators to select objects of interest\n- The theory behind the assumption is clear and correct\n\n[Weakness]\n- The proposed AVC metric is taken from a model proposed in 2017. I assume SyncFormer-based metrics are more reliable for evaluating audio-visual time alignment", "other_comments_or_suggestions": "- Using SAM2 and/or SAMRAI instead of SAM would be insightful to see if the results improve further by minimizing err_SAM\n- Recent relevant works use SyncFormer-based metrics to evaluate audio-visual alignment. I'd like to see the performance in terms of either of these: DeSync proposed in MMAudio, AV-Sync in MultiFoley, or Sync in V-AURA", "questions_for_authors": "Q1. How do you balance the volumes of generated sounds from different objects\nQ2. How do you generate off-screen sound in this framework?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: All reviewers recommend accepting the paper (1 accept and 3 weak accepts).

The rebuttal did a good job of addressing the concerns that the reviewers had. 

The AC thus follows this unanimous recommendation by the reviewers and recommends accepting the paper.
Decision: Accept (poster)