Both Ears Wide Open: Towards Language-Driven Spatial Audio Generation
Authors: Peiwen Sun
Sitong Cheng
Xiangtai Li
Zhen Ye
Huadai Liu
Honggang Zhang
Wei Xue
Yike Guo
Venue: ICLR 2025 Spotlight (2025)
Keywords: audio generation
multimodal learning
stereo audio
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: The multi-modal guided spatial audio generation dataset and method for immersive soundscapes

Abstract: Recently, diffusion models have achieved great success in mono-channel audio generation.
However, when it comes to stereo audio generation, the soundscapes often have a complex scene of multiple objects and directions.
Controlling stereo audio with spatial contexts remains challenging due to high data costs and unstable generative models. 
To the best of our knowledge, this work represents the first attempt to address these issues.
We first construct a large-scale, simulation-based, and GPT-assisted dataset, BEWO-1M, with abundant soundscapes and descriptions even including moving and multiple sources.
Beyond text modality, we have also acquired a set of images and rationally paired stereo audios through retrieval to advance multimodal generation. 
Existing audio generation models tend to generate rather random and indistinct spatial audio. 
To provide accurate guidance for Latent Diffusion Models, we introduce the SpatialSonic model utilizing spatial-aware encoders and azimuth state matrices to reveal reasonable spatial guidance. 
By leveraging spatial guidance, our model not only achieves the objective of generating immersive and controllable spatial audio from text but also extends to other modalities as the pioneer attempt.
Finally, under fair settings, we conduct subjective and objective evaluations on simulated and real-world data to compare our approach with prevailing methods. 
The results demonstrate the effectiveness of our method, highlighting its capability to generate spatial audio that adheres to physical rules.

Published as a conference paper ICLR 2025
Both Ears Wide Open: TOWARDS LANGUAGE-DRIVER SPATIAL AUDIO GENERATION
Peiwen Sun 27 Sitong Cheng" Xiangtai Li Zhen Ye Huadai Liu Honggang Zhang*
Wei X Yike Guo 90
V
Hong Kong University of Science and Technology.
Beijing University of Posts and Telecommunications,
Nanyang Technological University, Zhejiang University
Website:
https://peiwennsu220000 http:///wweenuuunoooigiiiubbbbbo
ABSTRACT
Recently, diffusion models have achieved great success in mono-channel audio generation. However, when it comes to stereo audio generation, the soundscapes often have a complex scene of multiple objects and directions. Controlling stereo audio with spatial contexts re- mains challenging due to high data costs and unstable generative models. To the best of our knowledge, this work represents the first attempt to address these issues. We first con- struct large-scale, simulation based,, and GPT-assisted dataset, BEWO-1M, with abun- dant soundscapes and descriptions even including moving and multiple sources. Beyond text modality, we have also acquired set of images and rationally paired stereo audios through retrieval to advance multimodal generation. Existing audio generation models tend to generate rather random and indistinct spatial audio. To provide accurate guidance for Latent Diffusion Models, we introduce the SpatialSonic model utilizing spatial-aware encoders and azimuth state matrices to reveal reasonable spatial guidance. By leveraging spatial guidance, our model not only achieves the objective of generating immersive and controllable spatial audio from text but also extends other modalities the pioneer attempt. Finally, under fair settings, we conduct subjective and objective evaluations on simulated and real-world data compare our approach with prevailing methods. The sults demonstrate the effectiveness of our method, highlighting its capability to generate spatial audio that adheres physical rules.
1 INTRODUCTION
The binaural hearing ability naturally enhances our perception of the world through the acoustic field, which became widely recognized in the 1980s with the rise of PCM (Lipshitz & Vanderkooy 2004) and MP4 (Sikora 1997) formats. the current era of audio generation, creating immersive experiences requires the production of stereo audio that adheres to specific location properties, which can be effectively achieved through end-to-end generative models. This generation task boosts applications in immersive VR/AR (Fi-)
tria 2023 Burdea & Coiffet 2024) and embodied AI (Liu et al. 2024d). Therefore, generating stereo audio that incorporates spatial multimodal context represents a valuable task within the community.
Significant progress has been made in monaural audio generation, with models such AudioLDM (Liu jt al. 2024a), Make-an-Audio 2 (Huang et al. 2023a) and Tango 2 (Majumder et al, 2024). These models leverage the diffusion architecture efficiently generate audio from textual prompts with a T5 model. For example, AudioL.MM 2 uses a latent diffusion model to generate a latent representation of mel-spectrograms, and Make-an-Audio 2 and Tango further explore the presence of events and their temporal ordering.
Equal contribution
Corresponding authors

---

Published as a conference paper at ICLR 2025
Previous Two-Stage G Generation
Description
Image
Bounding Box
Click
Izage
DifTusion
Lakest
1 cadlo
Condationl Filter
Spatad
ongine sound moves
Model
Simmulatoon
7t metoreycle find from vightto leftt
Previous One-Stage Gemeration
Our Spatial Generation
Test
Latent Diffesion
Skeve eato
BEWO-LK LM Dataset
Guided Genoration
Interactive Generation
(mt)
Model
SpecialA Control
Uncentrolibble
SpenallSniic
Tex
Image
Bug
Ctez
Engice
Human talking
(a) Comparison to papular gemeration model
(b) The objective of controllable spatial audio generation
Figure 1: Our SpatialSonic, as a one-stage model, alleviates the problem of error accumulation in a two- stage model and facilitates control with end-to-end finetuning in one-stage model. Moreover, our spatially enhanced system supports spatial audio generation from text and image, as well as interactive actions.
Compared to monaural audio generation, current research on spatial audid generation is still limited. The pioneering work (Dagli et al.) (2044) cascades the generation model with simulator to generate stereo audio from the image. Another direct thought is converting generated mono audio to stereo by integrating interaural time difference (ITD) and interaural level difference (ILD) (Desai & Mehendale (022). which are crucial for spatial perception in human auditory processing. Typically, to convert mono audio to stereo, Zhou et al. (2020); Parida et al. (2022) use visual and positional cues as conditions to produce spatial audio through a U-net style filter. Although it seems feasible to generate spatial audio by cascading effective audio generation models with simulations or filters, the two-stage approach in Fig. T a) incurs high computational costs and potential error accumulation.
We attribute the challenges of spatial audio generation to 3 aspects: 1) large data scale; 2) precise guidance construction; 3) proper evaluation metrics. This work presents the first exploration addressing these issues.
Although stereo audio is common in real life, captions of such audio with spatial descriptions still require massive human resources. For example, generating spatial audio that matches the textual description "a motorcycle engine sound moving gradually from front to left" requires extensive paired data of motorcycles in various directions of movement. Due to such high labeling costs, the lack of sufficient high-quality data becomes a barrier spatial audio generation models, compared to Mei et al. (2024); WWu al. (2023). To better facilitate the advancement of multimodal guided spatial audio generation models, we have developed a dual-channel audio dataset named Both Ears Wide Open 1M (BEWO- IM). contains up to 1 million audio samples through rigorous simulations and GPT-assisted caption transformation. BEWO- 1M contains an abundant soundscape, including moving-sourees multi-source and interleave-source scenarios with the spatial description or rational image. To ensure perceptual consistency with the real world, test sets from BEWO- 1M are checked by humans, and a real-world recorded subset is manually constructed and annotated.
Previous one-stage stereo audio generation model (Evans et al. 2024a b) training real-world data is able to generate stereo audio based on the caption but fails to generate rational spatial audio, as shown in Fig. Ta). After fine-tuning existing models with BEWO- IM, a basic capability can be obtained to understand the posi- tional description. Since the knowledge from the text fine-tuned model is trained on our enormous data, it is important for the image to utilize this knowledge through language-driver behavior. In our diffusion-basee SpatialSonic model, we first explore the multimodal spatial-aware guidance to encode images with regional perception way in Sec. 4.2 Then, we identify that due to the lack of explicit spatial guidance, simply fine- tuning the existing model with BEWO-IM still fails in precise T2A and 12A tasks. Therefore, we introduce azimuth guidance inducted by LLM through specific scheme clarify and integrate complex textual and visual contexts. Finally, along with proper classifier-free guidance training on the diffusion model, we obtain a controllable generation model. During inference, straightforward method can be easily applied to achieve user interaction. We propose using ITD-d objective metrics and opinion-based subjective evaluations
For clarity, define "spatial audio" as the "stereo audio" adheres spatial context.
2

---

Published as a conference paper at ICLR 2025
assess generated audio systematically. Our results show that SpatialSonic effectively generates realistic spatial audio, achieving 70% reduction in ITD error and higher opinion scores than popular models with minor adaption, including AM2ioLDM2. Overall, our contributions are:
Developing semi-automated pipeline create an open-source, large-scale, stereo audio dataset with spatial captions, BEWO-1M and supporting both large-scale training and precise evaluation.
Introducing one-stage, controllable, spatial audio generation framework, SpatialSonic, which is designed to generate dual-channel audio precisely adhering to multimodal spatial context.
Proposing series of subjective and objective metries based on ITD and opinion score to evalu- ate spatial audio generation models. Under fair experimental conditions, our framework produces audio with enhanced spatial information, obtaining more authentic soundscapes.
2 RELATED WORKS
Spatial Audio Understanding, Researchers including May et al. (2010) have been exploring the field of stereo audio through learnable methods since 2010. In recent years, the development of deep learning has led to more extensive exploration of stereo audio. The first area explored is binaural audio localization, where researchers are able localize the direction of sound sources by learning ITD and ILD in the single-
source scenarios (Krause et al. 2023 Yang & Zheng 2022 Cao et al. 2021 Shimada et al. 2022 Yasuda jet al.; 2020 Barrio-Barrios al. 2022) and the multi-source scenarios (Nguyen al. 2020). With the advancement of multimodal research, mono-to-stereo audio generation methods conditioned on visual (Gag
ct al. 202T Xu al. 202T; Liu et al. 2024c] Li et al., 2024 Zhou et al.] 2020), depth (Parida et al. 2022)
and location (Leng et al. (2222), have also been gradually developed. The input takes a mono audio and an image, and then spatial audio can be obtained through supervised learning. However, since a mono signal is still required, this task is not generation task from the current view. Additionally, researchers including Gebru et al. (202T): [Phokhinanan et al.] (2024); [Ben-Hur et al.] (202T); Geronazzo et al. (2020) hope to implicitly establish head-related transfer function (HRTF) through deep learning to form a reasonable signal mapping.
Text-to-audio (T2A) Generation. Audio generation from text is typically categorized into text-to-speech, lext-to-music, and text-to-audio. In this paper, we focus on the latter category. From a methodological
perspective, text-to-audio generation approaches can be broadly classified into autoregressive models and
diffusion models. The pioneering works from Kreuk et al. (002)); Yang et al. (2023); Lu et al. (2024); Liu
et al. (2024b) on autoregressive-basee audio generation into a sequence of tokens. Then, they apply an autoregressive network like GPT-2 (Radford et al.) 2019)
general audio tokenizer to convert waveforms
predict the next token in the sequence. However, autoregressive-based models often require substantial data and computational resources (Kreuk et al.] 20022. This has led to increased exploration of diffusion- based models for audio generation, with outstanding works including AudioLDM (Liu et al. 2023 2024a).
Audiobox (Vyas et al. 2023) and Stable Audio (Evans et al. 2024a b).
While the controllable problem in image generation (Cao al. 2024) persists in audio, the specific chal- lenges may differ. General controlling attempts are made by enhancing the text prompt (Copet et al. 2024 Huang et al. 2023b). To improve the precision of time and frequency control, [Huang et al. (2023a); Xie et al. (2024b) explore the temporal encoding along with the fusion mechanics. As for stereo audio, Mu- sicGen (Copet et al. 2024) and Stable Audio (Evans et al. 2024a b) can generate stereo audio but without spatial control. In this work, we reveal and deal with this spatial-controlling problem for the first time.
Other-to-audio Generation. With ImageHear (Sheffer & Adi 2023 pioneering the use of images as guid- ance, CLIP has been extensively employed for I2A generation tasks. Subsequently, Dong al. (2023) and Wang et al. (2024) have also leveraged CLIP to generate realistic spectra through diffusion guidance directly. However, the CLIP primarily focuses on aligning the global abstract semanties rather than the positional con- text. Therefore, spatial audio generation, as a position-aware task, requires further exploration of regional understanding of images.
3

---

Published as a conference paper ICLR 2025
Image
Original Caption
Step 2: Image Captions Transformation
aesennyellur rollng
Chols of Thought
Acoutio- Caption
Maxad Clockin
locates aidddee
PTA
Sesse and murcs rightat fast spesd.
9%
"The cungar Ausginng
dinattg
soone
separalds.
52%
FDI Sobrett
Room das
Preparation
Step 3: Audio Simvulation
Bod
Stegle Fowent Seledion
15:00
Andio Adtivity Tetwette
Soriag Dendes
10s sqpectss
CLAP Fllxtiaq
aose sackisa oific Coaniig
S sound peoolee
Caption Gettr
Somne
Step 2: A udio Captions Transt of maniioon
playing
Selection
Thoug::
Acoustic-rit Caption
28 Manual
mAnal.
Stats
Cyti
dop buciting
fruett let
right ohile
hea Figure 2: The pipeline of BEWO-IM data collection. The data machine is driven by LLM induction and rigorous simulation. particular, the data for testing are built with human checking. The diagram in Step 3 represents of the simulation scenarios. (a) illustrates the diversity of source and microphone positions. (b-f) show the abundant soundscapes in BEWO-IM.
Dysance
pubbars Sizttg
Won
DEWO IM.
3 MM DATASET
Due to high labeling costs, Tab. shows that stereo audio datasets usually have a short duration. The limited spatial audio datasets and the spatial constraints in existing models (Evans et al., (202bb) necessitate the creation of dataset with explicit spatial context, We propose BEWO- IM, a large-scale stereo audio dataset with spatial captions, as the first the best of our knowledge. BEWO-1M consists of audio-caption pairs and audio-image pairs. The construction pipeline in Fig. 2 follows the step-by-step procedure of preparing, transforming, and simulating to obtain a balanced dataset.
Data Preparation. Raw data collected online is noisy and requires pre-selection before simulation. Initially, samples with captions describing sin- gle sound sources to construct a single- source database. This makes each audio clip contain only single sound event, thereby ensuring realism and quality in simulation. We then apply sound tivity detection on each audio and ran- domly crop to 10s segments. Further, we remove segments with low CLAP (Wu) et al. 2023) similarity with their caption. See Appendix for more details.
Table Comparison existing audio-caption datasets.
Duration (hours)
Num. of
Paired
select
Task
Dataset
Audios
Type
AMOON-Auiia (Wu al. 2023
4.3k
633k
Text
WavCaps (Mei èi
7.5k
403k
Text
AudioCaps Kim et al
2019
110
46k
Text
SoundDeses Koopxe 4022
33k
Text
Event
Clutho (Drowsus el 21 2020
23
25L
Text
Audio Caption
m.
3.7k
Text
VGG-Sound Chen ai
11.00
550
200k
Video
AVE et 201
Video
PicoAAdio
2024b
15.6
5.6k
Text
Temporal
AudioTime Xie all
15.3
5.5k
Text
CompA-order (Goosh
2024
851
Text
SimBinaural Garg el 20245
116
22k
Video
FAIR-Play Gao Gramman
2019
1.9k
Video
Spatial
YT-ALL Morgado et 2018
113.1
1.1k
Video
MUSIC [2laao al. 2018
0.7k
Video
BEWO-IM (OUS))
2.8k
1,016k
Text
GPT-based Attributes Induction and Caption Transformation. With input as caption image-caption pair, we use GPT-4 and GPT-4o to induct sounding objects and their attributes for simulation, transforming raw captions into acoustic-rich cap- tions with spatial descriptions. The audio simulator normally requires certain essential attributes to simulate realistic audio, including scene size, sound source location, moving direction, and speed. We create an audio-object pool from sound objects inducted from audio-caption pairs. The sound objects inducted from images, and then the corresponding audio is retrieved from this pool. To obtain reasonable attributes and captions, we apply the Chain of Thought Prompting (CoT) (Wei al. 2023) to enhance the induction
BEWO-IM (Ours)
54
2.3k
Image
4

---

Published as a conference paper at ICLR 2025
ability of GPT-4 and GPT-4o. We predefined several patterns to describe each attribute, with each being an attribute element, such as "far" or "near" for the distance attribute. Specifically, we require GPT to select attribute labels that match the imput context and transform raw captions into acoustic-rich captions with positional and movement phrases. Fig. 2e) presents the statistics of caption length in BEWO-IM, and the transformed captions still remain concise with additional spatial and movement descriptions. See Appendix B.2 for more details.
Audio Simulation. We utilize the obtained attributes to simulate realistic and reasonable stereo audio. Following prior researches (Salvati et al., 2021; Chen et al. 2022b Dagli et al., 2024), we use Pyrooma- coustics (Scheibler et al., 2018) and gpuRIR and reflect real-world distribution, we introduce certain level of randomness into the inferred and selected attributes. To ensure scenery diversity, we also randomly set additional scene attributes like the microphone position and the room reverberation indicator, RT60. The simulation assumes a common ear distance of 16~18 cm. To make the dataset general, we do not consider the shadow effect of the head and leave the head adaptation achieved by future fine-tuning. The simulator then uses these attributes to generate the audio. Fig. Aaa shows the diversity of source positions in simulation. In the indoor scene, the simulator also generates audio with a static simulated room impulse response (RIR). For the moving source scenar- ios, we build trajectory to detail its positions. Our pipeline proficiently simulates audio across various environments, achieving both diversity and authenticity while meeting the criteria for ITD and ILD. See Appendix B.4 for details.
et al. 2021 for simulation. To enhance diversity
Post-Procsssing. To ensure data quality, we perform manual checking for part of the training set and entire test set. Tab. C14 shows that our automated pipeline can generate decent captions.
summary, we constructed 2.8k hours of training audio with more than IM audio-text pairs and ap- proximately 17 hours of valida- tion data with 6.2k pairs. As shown in Tab. []] our dataset, as the first audio-caption dataset with stereo audio and spatial de- scriptions, is comparable to other monaural audio-caption datasets. In addition to its large scale, the BEWO-IM dataset also notable for its high quality. As shown in Tab. the simulated audio receives high subjective ratings from human annotators. Fig. [[c, d) presents the diversity of attributes in BEWO- 1M, and Fig. f shows the sound events diversity. The details and statistics of subsets are provided in with BEWO- IM, and Fig. 3b,c) demonstrates our dataset significantly improves the spatial discrimination.
50
Lat: cottire elre: wigh Piez segle
Busilao BEMO
225
200
Ja Hurau Portptoo
18I Derouns palseowr BI BPMD-1M
10 grr BIMO-TM
Figure 3: In polar coordinates, the radius represents normalized audio energy, the angle denotes the perception angle (0 right, 180 for left), and five colors in the above legend signify the use of common directional terms to describe sound events. (a) is the human perception based on the questionnaire of volunteers. In (b), the baseline fails to generate the controllable audio. Obviously, (c) highlights the valuable knowledge from BEWO-IM. (d,e) highlights the superiority of our data and methods in controlling the generation 5f common directions and uniform fine-controlling matrices.
Appendix
C Finally, we fine-tune the baseline (Evans et al. 2024b)
4 METHOD
Overview. Our objective is to extract precise guidance from multimodal input and create stereo audio adher- ing spatial context. The overall pipeline of our SpatialSonic network consists of multimodal extraction, azimuth guidance, and latent code generation, as illustrated in Fig. 4 On top of Evans et al.) (2024b), we introduce multimodal encoder for the image's spatial perception in Sec. 4.2 to adapt the image to T2A
5

---

Published as conference paper ICLR 2025
Image
Visual Embeediing
Image
Azimuth Guidance
Model
</>
Fose
Profix
Adapter
Pronise A Atribcttes
Training
Rite
Coace
Poick
Encoder
Image
Frwet
Infereace
Cceos
Fite
VAP Encoder
Tahox
Fiuat
Inductive Attributes
Sana
Interactive
Text
Pretrain
shared
Input
payt moet)
[Tted dretion wil
Outgut
X:(Gee)
Diffusion
decoder
[Sooed: Tart)
Ragion
R/C
Filter
See Ga
Ax
*A metor is fast from to left.
Teat Zncoder
Sinng
Adaptar
Only peetrining
Only Inference
Multimodal imput
CODD
Fiot
Switcs
layers
Adapter
AFM
Generated 2-C audio
decoder
Figure 4: The overall pipeline of SpatialSonic. is one-stage controllable model that processes multimodal inputs to generate spatial audio, where GPT used to inject the specific azimuth state into the guidance.
model. Then, the azimuth fusion module in Sec. 4.3 provides extra clear conditions with the help of LLM and the azimuth scheme. Finally, the training and inference methodologies are presented in Sec. 4.4
4.1 PRELIMINARIES
Task Objective: Let R22L, represent a dual-channel audio signal, where Lg depends on the length of audio. As the autoencoder compresses the x into I, the audio generation process can be denoted as H. (C,e) + f z, where the multimodal condition, the Gaussian noise and H. the conditional generation process.
The pre-trained language model T5 encoder (Raffel et al. 2023) is used the text en- coder (Eeaa). It captures spatial context as text embedding Fieu E -x where L is a variable number depends on the text input. The ablation study in Tab. 7 reveals that incorporating CLAP speeds up conver- gence compared to T5; however, the spatial performance is inferior to T5.
Text Embedding:
Image Embedding:
Previous image encoder Eing used in I2A usually obtains Fing E Idd by CLIP. Considering the regional perception in Sec. 4.2 our image embedding follows Fing TR. Lxdus.
The attributes including number of sources K, start position Patans end position Mend- moving start time To and moving interval T is precisely known during simulation and can be used during training. During inference, inspired by Xie et al. (2024b) Qu et al. (202)) K, Pbegins Mends T can be inducted by GPT. Details are provided in Appendix
Azimuth Information:
4.2 IMAGE EMBEDDING WITH REGIONAL PERCEPTION
Popular 12A model (Sheffer & Adi 2023 using CLIP (Ramesh et al. 2022) focuses on aligning the global
abstract semantics rather than positional and relational context. Therefore, the objective of this module to gain regional image perception, while re- re-utilizing language-driven knowledge from massive text pairs.
To inject the spatial- semanties into the image encoder, we follow Cho al. (2021) to carry out a detection model as a regional perception to provide the rich positional context; this pre-training network is an decoder structure, firstly trained as vision- languale task. The network generates the acoustic description of the image, supervised by image-caption pairs in Sec. 3 The pre-training follows 3 steps below.
1) Use the detection model to obtain the N regions' embedding R R x Letp, corresponding coordinates C Nx4, and detected object class embedding I I. Nx where 0 N S Nmax- Obtain visual embedding Fieg E N Lis by linear projecting and element-wise adding R, R.C.I separately. 2) Concatenate the visual embedding specified text prefix embedding (i.e. "image acoustic captioning")
encoder input. 3) Update the multimodal encoder while keeping the weight of the decoder frozen. Overall, since the shared decoder is utilized during the pre-training of text and visual encoders, the latent features obtained from the
6

---

Published as a conference paper ICLR 2025
visual encoder Ling and text encoder Evext can be regarded in the same aligned space. The visual embedding output by Fing can be denoted Fing E R.xdu
4.3 CONTROLLING AZIMUTH FROM COARSE TO FINE
When text and image embedding are used directly as conditions, there is still large dispersity in Fig. B Therefore, it is crucial to design model that effectively supports precise generation using both text and images. On the one hand, the language phrase describe source direction is limited and rather subjective. By learning the underlying distribution of human perception in Fig. Ba), the generated audio becomes more realistic assessments. On the other hand, when generating stereo audio using continuous spatial information such as images, it is necessary to form more precise guidance. Inspired by layout-controllable image generation (Inoue et al. 2023] Qu et al. 2023), we introduce simple and clear guidance, azimuth state Nome which encodes azimuth at different times slots.
matrix for K sources S E I.
To fit the coarse guidance into a distribution of human perception, we developed a Gaussian-basee coarse- grained guidance for controllable generation. Given time and duration {t, T3 dame based on specified speeds, the center position j(tt at given moment adheres to the following physical principles:
t T (Hend Matart)
plt) Matart
(1)
where Mstarts Mend E Lazi- Here, for conciseness, we omit the expression before moving (L < To) and after moving (Tg + < t), which can be simply derived by Hand=Hatar- Further, the azimuth of k-th (0 S < K) object under the normal distributions (-) at different moments are illustrated by,
(1 V2TO² exp (-" m(())²
Senarse
[k] = -V u(().??)
22)
20
where the variance a is obtained from the statistics of real data. Each azimuth can be modeled as / = 1 for right and / = dime for left. Then St.r [k] is azimuth wise normalized before the next module.
For fine-grained purposes, the precise location is easily accessible to the nature of the simulation. Thus, we design the discrete state matrix of k-th object to represent the precise azimuth across time as
1
if = [u(t)]
Sfine
[
0
otherwise
This fine matrix can be regarded as the extreme situation of equation 2 without the uncertainty.
Then, we enhance the condition by fusing azimuth state and text embedding by azimuth fusion module (AFM). This AFM composes of the multi-layer cross-attention as CA(q, k. v) introduced as
S' = CA(S, F",, Fi)) + S for F', P'image
FV
if if I2A
(4)
= softmax(S F.)) + S,
where Flert> F F'maage are the embedding after transforming Frezts Fimage by adapter and S' is the attended state matrix. To enhance conciseness, the equation 4 for attention have excluded projection and scaling factors. This state matrix and the fusion module allow us to encode complex behaviors such as the speed and direction of sound source movement, and even support future custom matrices for any speed and azimuth.
4.4 TRAINING AND INFERENCE OF DIFFUSION MODEL
We utilize a diffusion model as to model å based on the azimuth state matrix S and modality embedding Fiett or Fimage- The forward process of the cosine form (Esser et al. 2024) is used to obtain the noised representation P, of each time step 7 by noise injection by
Pr ob 3c,
(5)
7

---

Published as a conference paper ICLR 2025
where follows isotropic Gaussian distribution, a cos (5T) and B = sin (डT). Then, the v-prediction
parameterization (Kingma & Gao 2024) is implemented for training for better sampling stability with few of the inference steps, so the overall training conditional objective is
L = 10 Wt fl ((Pr. cat((" Fm]] ) Vor 1or112
(6)
8Pr
Vo.
aE aBP
(7)
where the velocity v is calculated from noise schedule of E [0. 1]. and w, is the weight obtained from the signal-to-noise ratio of 'P,. cat(..-) means concatenation. Se is the estimation network to reconstruct and denoise from Py, to learn the conditional denoising as
- [ [Jo (CA (Pr," FM))) Be].
(8)
Initially, the T2A model is trained using the BEWO-IM M dataset. On top of this T2A model, it is fine-tuned using the spatial-aware image encoder to develop the 12A model. Drawing inspiration from current SAM- based interaction models (Ma et al. 2024 Kirillov et al. (023). we utilize clicks and bounding boxes (BBox) to select Region of Interest (Rol) from regions R, E R in Sec. 4.2) After generating the high-quality mask, it is then matched with region coordinates C' and regional feature R. We enable the selection of multiple RoIs as input of Emmage- The regions R,, coordinates C, and IDs I, are selected, where n E N. By using the selected R,,, Cn and In input, the image encoder Emng takes interactive embedding Fetck Fobox and finally generate the spatial audio of the target object.
5
EXPERIMENT
5.1 TRAINING AND EVALUATION
Dataset Our dataset is built on diverse combination of datasets detailed in Appendix B.1 We convert the sampling rate of audios to 16kHz and pad short clips to 10 seconds long after the data construction in Sec. As for images, we select the scenery with audible subjects from COC02017 (Lin et al. 2014).
Model Configurations We fine-tune the continuous autoencoder pre-trained by Stability A to compress the perceptual space with downsampling to the latent representation. For our main experiments, we train a text-conditional Transffrrere (DiT) (Levy al., 2023 Peebles & Xie 2023). which is optimized
using 8 NVIDIA RTX 4090 GPUs for 500K steps. The base learning rate is set to 26-5 with a batch size of 128 and audio length of 10s. Hyper-parameters are detailed in the Appendix
Evaluation Metries: 1) 1-C metrics: We adopt metrics from Huang et al. (203bb) and Liu et al. (2023)
calculate Fréchet Distance (FD), Inception Score (IS), Kullback-Leibler divergence (KL), Fréchet Au- dio Distance (FAD), CLAP score (CLAP), overall impression (OVL) and audio-text relation (REL) for T2A evaluation. Additionally, CLIP Score (Wu al. [022]] Sheffer & Adi (023) evaluates the relevance for 12A model. 2) 2-C objective metrie: To better examine the quality of the generated spatial audio, we propose novel evaluation methods based on TDOA Utilizing the non-silent segments with threshold of -16 dBFS in the audio, we compute TDOA distributions in intervals of 0.1 seconds using both the tra- ditional Generalized Cross-Correlation with Phase Transform (GCC-PHAT) (Knapp & Carter 1976) and the deep learning network StereoCRW (Chen et al.] (022b). Finally, the mean absolute error is com- puted based on the TDOA of the ground truth and generated audio as GCC MSE and CRW MSE, lower enror indicates better alignment with ground truth, but simple MSE fails in scenarios with multiple or moving sources. Expanding on FAD KKlloorr et al. 2018), a novel evaluation metric, Fréchet Stereo
http://giiuub..om/Sabiitty Allstoble-andio-tools
Thii quantity known as both difference of arrival (TDOA) and the interaural time difference (ITD).
8

---

Published as a conference paper ICLR 2025
Audio Distance (FSAD). is introduced. FSAD builds on FAD by leveraging the StereoCRW network in- stead of VGGish for FD computation. More methodologies and parameters are detailed in Appendix [.22 3) 2-C subjective metric: To assess the quality of the generated audio, we employ the Mean Opinion Score (MOS) through human evaluation to evaluate the quality of source direction (MOS-Direction) and event (MOS-Event) separately. We invite 15 experts to evaluate the quality on scale ranging from to 5, with 5 for the best quality. For further information on our MOS, please refer to Appendix F.4
5.2 RESULTS
Table 2: 1-C audio quality comparison of popular methods in T2A on Audiocaps test set.
Comparison on 1-C T2A Base- line: After averaging the generated stereo audio across channels, we compute metries for mono-channel (1-C) audio. The results are pre- sented in Tab. and highlight our model's strong performance across all metrics, particularly excelling in IS and CLAP, often matching or even surpassing best benchmarks. These results indicate that our model adeptly interprets textual cues to produce high-quality audio outputs.
Objective KI
Subjective
FDJ
1S-
FADL
CLAPP
OVL
REL.
AudioGen-L (Kreak ct
1.69
1.82
Uniaudio Yang ala 20125
2.60
3.12
3.05
3.19
Make An--AnAAnn [Taang
20236
18.32
7.29
1.61
2.66
0.539
3.53
3.59
Make-An- An-Andio (Huann at
2022211225 21230
11.75
11.16
1.32
1.80
0.645
3.72
3.57
AudieLDM (Liu et 20125
23.31
8.13
1.57
1.96
0.621
3.70
3.71
AudieLDM LTi cl 20z4a
19.93
9.39
1.64
1.86
0.652
3.78
3.76
Stablo-andio-ojaa (EVanl al
2024b
21.21
10.50
1,86
2.37
0,594
3.64
3.60
TANGO Ghosal ar 20255
26.13
8.23
1.37
1.83
0.650
3.65
3.66
TANGO [Majumaer
2024
18.85
10.09
1.12
1.90
0.675
3.73
3.69
SpatialSonic (Ours)
14.03
13.79
1.37
1.93
0.672
3.75
3.73
Table 3: 2-C audio quality comparison of popular methods in T2A on BEWO-IM test set. "f" means minor adjustment of the original structure with a conditional mono-to soteroo-tterro filter.
Objective CRW
Subjective
Task
Method
PSAD
MOS-Events
MOS-Direction
Simulation
Simulation
4.94
4.55
Andial DD22T
46.59
50.17
1.61
3.57
3.53
set)
MakkeAA-Audiiz2
0.97
Stable-audio-pen
38.73
34.36
0.63
3.73
3.76
SparialSoni(Ourr)
27.20
15.86
0.17
3,78
3.84
AM2ioLDM22
45.08
42.88
0.94
3.37
3.34
T2A (SD-sct)
Make-An-Audion
48.55
47.88
1.09
3.38
3.30
Stable audio
45,76
48.60
0.53
3.68
3.58
Suuts
44.36
31.91
0.26
3,86
3.71
DM2i
38.96
3.29
T2A (DS-set)
35.37
48.54
2.11
3.24
3.31
Stable-audio-ppen
32,63
36.30
0.87
3.60
3.61
SpatilSSniie
22,51
13.75
0.31
3,80
3.83
Audiol
3643
49.87
1.22
3.38
3.34
(M-set)
Make Audio2
36.01
1.32
3.29
auble-audio-open
34.20
48.06
0.53
3.54
3.58
SputaalSoni((uuss)
3.75
ADdi
46.98
44.66
1.54
3.28
3.35
T2A (RW-set)
Make-An-Audioz"
46.94
43.47
1.60
3.23
3,30
Stable-mddo-oppr
43.18
47.58
0.60
3.51
3.49
SpatialSonie(Ours)
30.27
23.19
0.28
3.79
3.76
Comparison on T2A Baseline: Based on the proposed evaluation method, we train all the models listed on the BEWO-IM train set and conduct series tests on spatial audio in Tab. 1) The authenticity of our ground truth simulation data is evaluated by humans, showing that our simulation received a high position and events score. 2) The evaluation metrics GCC MSE and CRW MSE, based on global averages, show limitations in representing complex subsets such as the SD-set and DS-set; thus, we primarily treat FSAD as our evaluation method. 3) Utilizing FSAD and MOS as the principal metric, our approach outperforms all baselines in objective performance and achieves higher recognition in subjective evaluations.
Comparison on 2-C 12A Baseline: As the objects in COC02017 often have common occlusion, our method still demonstrated advantages in objective and subjective evaluation metries, notably achieving 1.56 per- formance improvement in FSAD. Additionally, we extend our tests to earlier 12A datasets, which also show the superiority of SpatialSonic in the Appendix G.3
9

---

Published as a conference paper ICLR 2025
Table 4: 2-C audio quality comparison of popular methods in 12A generation. "S&H" means Seeing- -nd---nd--i- Hearing.
Table 5: 2-C audio quality comparison in an in-
teractive 12A generation.
Task
Method
CLIP
GCC
Objective
CRW
FSAD
MOS
Subjective
MOS
Prompt
Method
MOS
Subjective
MOS
Clarity
CRW
Score 1
MSR 5
MSE I
Events
Direction 1
Evenss
Direction
MA1
MA.
GT
Simelation
6.211
1.61
1.68
Sectanand
3.33
3.47
131
X.11
Seelsound
BBox
S&H1
V2A
S&H
S&H
4.591
4.737
96.11
62.55
2,08
3.53
3.39
347
SpatialSomic
Sec2oound
3.68
3.37
3.64
3.30
1002
17.32
7.11
3.79
Point
S&H
3.26
3.43
2.57
8.91
SpatialSonic
Comparison on 2-C Interactive 12A Baseline: Given the absence of ground truth audio for our interactive objective, we construct a small comprising around 150 images, 300 bounding boxes, and 300 click points derived from authentic user interactions (accessible in the Appendix C.3). We then evaluate the generation quality across two dimensions using subjective metrics. Additionally, the clarity of direction reflected by calculating the mean absolute ILD GCC MA and CRW MA.
Table 6: Ablation study of the coarse and fine strategy on differ- ent subsets.
Table 7: The ablation study on the textual guid- ance with 20% of training data BEWO-IM.
FSAD
FSAD
Training
w//o.
Inference
w/o. Matri
M-set J
0.53
RW-set
@12A
1.41
Text Encoder
Converge
leer.
Audiocaps
@Mant
w/o. Matix 8
0.56
0.67
0.84
my. CLAF
2335
0.56
0t.. Crarse & Fine
Fine S
0.32
0.48
0.52
0/. T5
TS+CLAP
1O5K
135K
2.31
2.35
0.24
0.40
w/. Coarse
0.16
0.28
0.96
Ablation on Each Component: 1) Text encoder: Just as stated Sec. 4.1 although T5 converges slowly,
has significant advantages in capturing spatial and temporal information, shown in Tab. [ 2) Coarse and fine azimuth matrix: The results Tab. 6 indicaae that coarse guidance is more suitable for the T2A task, while fine guidance is necessary for the T2A task. Quantitatively, using fine guidance in T2A causes too strict azimuth guidance, even for noise, whereas coarse guidance in I2A results in less unstable generation for complex scenarios.
More Experiments: Notably, extensive statistics, detailed experiments, and comprehensive user studies are presented in the appendix. More insights in Appendix Gare strongly recommended readers for thorough understanding of the challenges, including how azimuth guidance and caption length affect audio quality.
6 DISCUSSION AND CONCLUSION
Discussion: It is believed that BEWO-IM IM is able facilitate widespread application in various such as I) spatial cross-modal retrieval, 2) contrastive language-audio pre-training, 3) spatial audio captioning, 3) large-scale audio-language pertaining model. From methodology perspective, our SpatialSonic model rep- resents a pioneering effort to achieve controllable spatial audio generation. However, there is still potential for improvement. For example, the current image encoder's limited size restricts its ability to fully compre- hend the dynamics and behaviors across datasets with more diverse classes or in open open-world scenarios.
Conclusion: In this work, we introduce novel task to generate stereo audio from spatial context, which requires the machine to understand multimodal information and generate rational stereo audio. To advance this field, We develop the first apen-source, large-scale spatial audio dataset, BEWO- 1M, for training and evaluation. Our proposed SpatialSonic model further establishes robust baseline with enhanced spatial perception. During experiments, we compare our SpatialSonic with several existing models training on BEWO- 1M and demonstrate SpatialSonic's 's promising performance in generating high-quality stereo audio adhering to spatial locations. Our task and dataset have great potential in applications such as AR/VR and embodied AI to create immersive experiences. In the future, plan further expand the scale of the current dataset with 5. l-channel audios, a higher sampling rate, and more visual data, including images and videos, meet the growing data demands in the era of generation.
10

---

Published as a conference paper ICLR 2025
7 ACKNOWLEDGEMENTS
The research was supported by NSFC (No. 62206234), Early Career Scheme (ECS BBCS-HKUST2200122 Theme-based Research Scheme (T45-205/21-N) from Hong Kong RGC, and Generative Research and Development Centre from InnoHK
REFERENCES
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation, 2016.
abs 607. 08822
Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved corre- lation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005.
Zamir Ben-Hur, David Lou Alon, Ravish Mehra, and Boaz Rafaely. Binaural reproduction based on bilateral ambisonics and ear-alignn car-aligned hrtfs. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:901-913, 2021.
Grigore C Burdea and Philippe Coiffet. Virtual reality technology. John Wiley & Sons, 2024.
Pu Cao, Feng Zhou, Qing Song, and Lu Yang. Controllable generation with to--tige-mge diffusion models: A survey. arXiv preprint arXiv:24033 04279, 2024.
Yin Cao, Turab lqbal, Qiuqiang Kong, Fengyan An, Wenwu Wang, and Mark D Plumbley. An improved event-independent network for polyphonic sound event localization and detection. In ICASSP 2021- 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 885-889. IEEE, 2021.
Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visua dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pp. 721-725. IEEE, 2020.
Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma. Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Hts-at: hierarchical token-semanti audio transformer for sound classification and detection, 2022a. URL
https://arxiv: org/ abs/ 2202.2222222 2202.0077222
Ziyang Chen, David F Fouhey, and Andrew Owens. Sound localization by sulf-supervised time delay esti- mation. European Conference on Computer Vision, pp. 489-508. Springer, 2022b.
Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying vision-and-languagg tasks via text generation. In International Conference on Machine Learning, pp. 1931-1942. PMLR, 2021.
Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez, Simple and controllable music generation. Advances Neural Information Processing Systems, 36, 2024.
Kevin Crowston. Amazon mechanical turk: A research tool for organizations and information systems scholars. In Anol Bhattacherjee and Brian Fitzgerald (eds.), Shaping the Future of ICT Research. Methods and Approaches, pp. 210-221, Berlin, Heidelberg, 2012. Springer Berlin Heidelberg. ISBN 978-3-642- 35142-6.
Rishit Dagli, Shivesh Prakash, Robert Wu, and Houman Khosravani. See-2-sound: Zero-shot spatial
environment-to-ppttee sound. arXiv preprint orXiv:2406.06612 2024.
11

---

Published as conference paper ICLR 2025
Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. arXiv preprint arXiv:2210. 13438, 2022.
Dhwani Desai and Ninad Mehendale. A review sound source localization systems. Archives of Compu- tational Methods in Engineering, 29(7):4631 -4642, 2022.
Jacob Devlin, Ming-Way Wei Chang, Kenton Lee. and Kristina Toutanova. Bert: Pre-training of deep bidi- rectional transformers for language understanding, 2019. UK..ittpps///rraooorr/////s//
04805
David Diaz-Guerra, Antonio Miguel, and Jose R Beltran. gpurir: A python library for room impulse response simulation with gpu acceleration. Multimedia Tools and Applications, 80(4)565335671, 2021.
Hao-Wen Dong, Xiaoyu Liu, Jordi Pons, Gautam Bhattacharya, Santiago Pascual, Joan Serrà, Taylor Berg- Kirkpatrick, and Julian McAuley. Clipsonie: Text-to-audio synthesis with unlabeled videos and pretrained language-visien models. In 2023 IEEE Workshop Applications of Signal Processing Audio and Acoustics WASPAA), pp. 1-5, IEEE, 2023.
Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: An audio captioning dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 736-740. IEEE, 2020.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-forst International Conference on Machine Learning. 2024.
Zach Evans, Julian Parker, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons. Long-form music generation with latent diffusion. arXiv preprint arXiv:2404.1030/ 2024a.
Zach Evans, Julian D Parker, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons. Stable audio open. arXiv preprint arXhv:2407.14358 2024b.
Tira Nur Fitria. Augmented reality (ar) and virtual reality (vr) technology in education: Media of teaching and learning: review. International Journal of Computer and Information System (IJCIS), 4(1):14-25, 2023.
Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra. FSD50K: an open dataset of human-labeled sound events. /EEE/ACM Transactions on Audio, Speech, and Language Processing, 30: 829-852, 2022.
Ruohan Gao and Kristen Grauman. 2.5 visual sound. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 324-333, 2019.
Guillermo García- Barria- Daniel Aleksander Krause, Archontis Politis, Annamaria Mesaros, Juana M Gutiérrez-Ariola, and Rubén Fraile. Binaural source localization using deep learning and head rota- tion information. 2022 30th European Signal Processing Conference (EUSIPCO), pp. 36-40. IEEE, 2022.
Rishabh Garg, Ruohan Gao, and Kristen Grauman. Geometry-awand multi-task learning for binaural audio generation from video. arXiv preprint arXiv:2/11. 2021.
Rishabh Garg, Ruohan Gao, and Kristen Grauman. Visually audio spatialization in video with geometry-aware multi-task learning. International Journal of Computer Vision, 131(10):723-2733, 2023.
12

---

Published as a conference paper at ICLR 2025
Israel Gebru, Dejan Marković, Alexander Richard, Steven Krenn, Gladstone A Butler, Fernando De la Torre, and Yaser Sheikh. Implicit hrtf modeling using temporal convolutional networks. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1185-3389..0 IEEE, 2021.
Michele Geronazzo, Jason Yves Tissieres, and Stefania Serafin. A minimal personalization of dynamic bin- aural synthesis with mixed structural modeling and scattering delay networks. In ICASSP 2020-2020 JEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 411-415. IEEE, 2020.
Deepanway Ghosal, Navonil Majumder, Ambuj Mehrish, and Soujanya Poria. Text-to-audio generation using instruction guided latent diffusion model. In Proceedings of the 31 ACM International Conference on Multimedia, pp. 3590-3598, 2023.
Sreyan Ghosh, Ashish Seth, Sonal Kumar, Utkarsh Tyagi, Chandra Kiran Reddy Evuru, Ramaneswaran S. S Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. Compa: Addressing the gap in compositional reasoning in audio-language models. In The Twelfth International Conference Learning Representations, 2024. URL http:: htttt:::/opeereeeeees.s http://oppnrreeieeee net http//formmimdddee/
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Jürgen Herre, Johannes Hilpert, Achim Kuntz, and Jan Plogsties. Mpeg-h audio the new standard for universal spatial/3d audio coding. Journal of the Audio Engineering Society, 62(12):821-88) 2015a.
Jürgen Herre, Johannes Hilpert, Achim Kuntz, and Jan Plogsties. Mpeg-h 3d audio the new standard for coding of immersive spatial audio. IEEE Journal of selected topics in signal processing, 95):7770-77, 2015b.
Jiawei Huang, Yi Ren, Rongjie Huang, Dongchao Yang, Zhenhui Ye, Chen Zhang, Jinglin Liu, Xiang Yin, Zejun Ma, and Zhou Zhao. Make-an-audio 2: Temporal--enancee text-to-audio generation. arXiv preprint 2305v 18474, 2023a.
Rongjie Huang, Jiawei Huang, Dongchao Yang. Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. In International Conference on Machine Learning, pp. 13916-13932. PMLR, 2023b.
Naoto Inoue, Kotaro Kikuchi, Edgar Simo-Serra, Mayu Otani, and Kota Yamaguchi. Layoutdm: Discrete diffusion model for controllable layout generation. In Proceedings of the IEEF/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 10167-10176, 2023.
Insu Jang, Zhenning Yang, Zhen Zhang, Xin Jin, and Mosharaf Chowdhury. Oobleck: Resilient distributed training of large models using pipeline templates. In ACM SIGOPS 29th Symposium of Operating Systems and Principles (SOSP 223, 2023.
Gabriel Jekateryńczuk and Zbigniew Piotrowski. A survey of sound source localization and detection meth- ods and their applications. Sensors, 24(1), 2024. ISSN 1424-8220. doi: 10.3390/s24010068. URL ht https: www. mdpi. comm 1 1424 8220/24/1/69
Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi. Fry 'echet audio distance: A metric for evaluating music enhancement algorithms. arXiv preprint arXiv:1812.08460 2018
Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Generating captions for audios in the wild. In NAACL-HLT, 2019.
13

---

Published as a conference paper ICLR 2025
Diederik Kingma and Ruiqi Gao. Understanding diffusion objectives as the elbo with simple data augmen- tation. Advances in Neural Information Processing Systems, 36, 2024.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan- Yen Lo, al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4015-4026, 2023.
Charles Knapp and Glifford Carter. The generalized correlation method for estimation of time delay. IEEE Iransactions on acoustics, speech, and signal processing, 24(4):320-327, 1976.
Sophia Koepke, Andreea-Maria Oncescu, João F Henriques, Zeynep Akata, and Samuel Albanie. Audio retrieval with natural language queries: A benchmark study, IEEE Transactions Multimedia, 25:2675- 2685, 2022.
Qiugiang Kong. Yin Cao, Turab Iqbal. Yuxuan Wang, Wenwu Wang, and Mark D. Plumbley, Panns: Large- scale pretrained audio neural networks for audio pattern recognition, 2020. URL https://arxi:: orq hbs//ab//aa awtpaass.....ee...//. 1ttp:/991....2.1
Daniel Aleksander Krause, Guillermo García-Barrios, Archontis Politis, and Annamaria Mesaros. Binaural sound source distance estimation and localization for moving listener. 1EEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.
Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. Audiogen: Textually guided audio generation. arXiv preprint arXiv:2209. 15352, 2022.
Yichong Lnng, Zehua Chen, Junliang Guo, Haohe Liu, Jiawei Chen, Xu Tan, Danilo Mandic, Lei He, Xiangyang Li, Tao Qin, et al. Binauralgrad:: A two-stage conditional diffusion probabilistic model for binaural audio synthesis. Advances in Neural Information Processing Systems, 35:23689-23700, 2022.
Mark Levy, Bruno Di Giorgi, Floris Weers, Angelos Katharopoulos, and Tom Nickson. Controllable music production with diffusion models and guidance gradients. arXiv preprint arXiv:2311.0061 2023.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequenee pre-training for natural language generation, translation, and comprehension, 2019. URL /tt.//ttps///axx...oro////////// http://aa///l111 13461
Zhaojian Li, Bin Zhao, and Yuan Yuan. Cyclic learning for binaural audio generation and localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26669- 26678, 2024.
Chin-Yeu Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Stanley Lipshitz and John Vanderkooy. Pulse-code modulation an overview. Journal of the Audio Engi- neering Society, 52(3):200 2050000002220000 2004.
Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. arXiv preprint
arXiv:2301. 220.. 2023.
14

---

Published as a conference paper at ICLR 2025
Haohe Liu, Yi Yuan, Xubo Liu, Xinhao Mci, Qiugiang Kong, Qiao Tian, Yuping Wang, Wenwu Wang, Yux- uan Wang, and Mark D Plumbley. Audioldm 2: Learning holistic audio generation with self-supervised pretraining. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024a.
Huadai Liu, Rongje Huang, Yang Liu, Hengyuan Cao, Jialei Wang, Xize Cheng, Siqi Zheng, and
Zhou Zhao. Audiolem: Text-to-audio generation with latent consistency models. arXiv preprint
2rXiv:2406.003556 2024b.
Miao Liu, Jing Wang, Xinyuan Qian, and Xiang Xie. Visually guided binaural audio generation with cross- modal consistency. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 7980-7984, IEEE, 2024c.
Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama, and Kevin Murphy, Improved image captioning via policy gradient optimization of spider. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 873 881. IEEE, October 2017. doi: 10.1 00.1109/iee..211..000. URL CRLhttp::/// //xx .og. 10. 10..1109/I00 2A..10//ICV.....1..
Yang Liu, Weixing Chen, Yongjie Bai, Jingzhou Luo, Xinshuai Song, Kaixuan Jiang, Zhida Li, Ganlong Zhao, Junyi Lin, Guanbin Li, al. Aligning cyber space with physical world: A comprehensive survey on embodied ai. arXiv preprint arXiv:2407.06888 2024d.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01099 2022.
Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision language au- dio and action. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26439-26455, 2024.
Yue Ma, Yingqing He, Hongfa Wang, Andong Wang, Chenyang Qi, Chengfei Cai, Xiu Li, Zhifeng Li, Heung- Shum, Wei Liu, et al. Follow-your-cfiek: Open-domain regional image animation via short prompts. arXiv preprint arXiv:2403. 08268, 2024.
Navonil Majumder, Chia-Yu Hung, Deepanway Ghosal, Wei-Ning Hsu, Rada Mihalcea, and Soujanya Poria. Tango 2: Aligning diffusion- based text-to-audio generations through direct preference optimization. arXiv preprint arXiv:2404.09955 2024.
Tobias May, Steven Van De Par, and Armin Kohlrausch. A probabilistic model for robust localization based on a binaural auditory front-end. IEEE Transactions on audio, speech, and language processing, 19(1): 1-13, 2010.
Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D. Plumb- ley, Yuexian Zou, and Wenwu Wang. Wavcaps: A chatgpt-assisted weakly-lahelled audio captioning dataset for audio-language multimodal research. /EFF/ACM Transactions on Audio, Speech, and Lan- guage Processing, 32:3339-3354, 2024. ISSN 2329-9304. doi: 00.1109/aallp.224..4411 doi. ottp:///x..oi..rgg//iit. 1110119//SSAA 2024. 6449446
URL
Pedro Morgado, Nuno Vasconcelos, Timothy Langlois, and Oliver Wang. Self-supervised generation of 02587
spatial audio for 360 video, 2018.
Thi Ngoc Tho Nguyen, Douglas L Jones, and Woon-Seng Gan. A sequence matching network for poly- phonic sound event localization and detection. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 71-75. IEEE, 2020.
15

---

Published as a conference paper ICLR 2025
Jakeh Orr, William Ebel, and Yan Gai. Localizing concurrent sound sources with binaural microphones: simulation study, Hearing Research, 439:108888 2023. ISSN 0378-5955. doi: https///di..or//00.. 1016/ j.heares 2023. 108884. URL attps://www/ wttp://wwwwwn.eeeeee..c...eoeeee cct///ccieene///ttiic//////
108884
50378595523001
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Kranti Kumar Parida, Siddharth Srivastava, and Gaurav Sharma. Beyond mono to binaural: Generating binaural audio from mono audio with depth and cross modal attention. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3347-3356, 2022.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1195-4205, 2023.
Waradon Phokhinanan, Nicolas Obin, and Sylvain Argentieri. Auditory cortex-inspired spectral attention modulation for binaural sound localization in hrtf mismatch. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 8656-8660. IEEE, 2024.
Karol J. Piczak. ESC: Dataset for Environmental Sound Classification. In Proceedings of the 23rd An- nual ACM Conference Multimedia, pp. 1015-1018. ACM Press, 2015. ISBN 978-1-4503-335594 doi: 222114//733333288663333 URL 2806390
Leigang Qu, Shengqiong Wu, Hao Fei, Liqiang Nie, and Tat-Seng Chua, Layoutlim-12i: Eliciting layout guidance from I1m for text-to-image generation. In Proceedings of the ACM International Conference on Multimedia, pp. 643-654, 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David I Luan, Dario Amodei, Ilya Sutskever, et al. Language models unsupervised multitask learners. OpenA/ blog, 1(8):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer, 2023. URL https: http:///xxvvvvrggreee/ www.ooo///aaaaeaeoeoeeo.eeeeee/s 910. 10683
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen, Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204..0 06125, 1(2):3, 2022.
Daniele Salvati, Carlo Drioli, Gian Luca Foresti, et al. Time delay estimation for speaker localization using cnn-based parametrized gec-phat features. In Interspeech, pp. 1479-1483, 2021.
Robin Scheibler, Eric Bezzam, and Ivan Dokmanić. Pyroomacoustics: A python package for audio room simulation and array processing algorithms. In 2018 JEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 351-355115555 351-355. IEEE, 2018
Roy Sheffer and Yossi Adi. hear your true colors: Image guided audio generation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Kazuki Shimada, Yuichiro Koyama, Shusuke Takahashi, Naoya Takahashi, Emiru Tsunoo, and Yuki Mit- sufuji. Multi-accdoa: Localizing and detecting overlapping sounds from the same class with auxiliary duplicating permutation invariant training. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 316-320. IEEE, 2022.
16

---

Published as conference paper ICLR 2025
Kazuki Shimada, Archontis Politis, Parthasaarathy Sudarsanam, Daniel A Krause, Kengo Uchida, Sharath Adavanne, Aapo Hakala, Yuichiro Koyama, Naoya Takahashi, Shusuke Takahashi, et al. Starss23: An audio-visual dataset of spatial recordings of real scenes with spatiotemporal annotations of sound events. Advances in Neural Information Processing Systems, 36, 2024.
Thomas Sikora. The mpeg-4 video standard verification model. IEEE Transactions on circuits and systems for video technology, 7(1):19-31 111199-11111 1997.
Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio-visual event localization in unconstrained videos. In Proceedings of the European conference on computer vision (ECCV), pp. 247- 263, 2018.
Kiki van der Heijden and Siamak Mehrkanoon. Goal-driven, neurobiologicaalinspiin convolutional neu- ral network models of human spatial hearing. Neurocomputing, (00:432-442, 2022. ISSN 0925-2312. doi: cccane//art00e///////22222222210
2021 10..104. URL htt.p:: //wp:////www./. sc wwwp:wwwwww....eeeeo.co..o.e encedi rect. com/
Kiki van der Heijden, Josef P Rauschecker, Beatrice de Gelder, and Elia Formisano. Cortical mechanisms of spatial hearing. Nature Reviews Neuroscience, 20( (0))0)))99-,,, October 2019. ISSN 1471-003X. doi:
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4566- 4575, 2015.
Apoorv Vyas, Bowen Shi, Matthew Le, Andros Tjandra, Yi-Chiao Wu, Baishan Guo, Jiemin Zhang, Xinyue Zhang, Robert Adkins, William Ngan, al. Audiobox: Unified audio generation with natural language prompts. arXiv preprint 1582: 2023.
Heng Wang, Jianbo Ma, Santiago Pascual, Richard Cartwright, and Weidong Cai. V2a-mapper: lightweight solution for vision-to-audio generation by connecting foundation models. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou, Chain-of-thought prompting elicits reasoning large language models, 2023, URL https: http///axvvvvv. wttpa/iww...ggb////////aaaaaa//a
11903
Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar, and Juan Pablo Bello. Wav2clip: Learning robust audio representations from clip. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and IEEE, 2022.
Signal Processing (ICASSP), pp. 4563 4567.
Mengyue Wu, Heinrich Dinkel, and Kai Yu. Audio caption: Listen and tell. In ICASSP 2019-2019 IEEE International Conference Acousties, Speech and Signal Processing (ICASSP), pp. 2019.
IEEE,
Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor KirgpKirkattrick, and Shlomo Dubnov. Large- scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, IEEE, 2023.
Zeyu Xie, Xuenan Xu, Zhizheng Wu, and Mengyue Wu. Audiotime: A temporally-aligned audio-text benchmark dataset, 2024a. URL
17

---

Published as a conference paper ICLR 2025
Zeyu Xie, Xuenan Xu, Zhizheng Wu, and Mengyue Wu. Picoaudio: Enabling precise timestamp and frequency controllability of audio events in text-to-audio generation. arXiv preprint arXiv:2407.02869, 2024b.
Yazhou Xing, Yingqing He, Zeyue Tian, Xintao Wang, and Qifeng Chen. Sceing and hearing: Open-domain visual-audio generation with diffusion latent aligners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7151-7161, 2024.
Xudong Xu, Hang Zhou, Ziwei Liu, Bo Dai, Xiaogang Wang, and Dahua Lin. Visually informed binaural audio generation without binaural audios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15485-15494, 2021.
Xuenan Xu, Xiaohang Xu, Zeyu Xie, Pingyue Zhang. Mengyue Wu, and Kai Yu. A detailed audio-text data simulation pipeline using single-event sounds, 2024.
04594]
Dongchao Yang, Jinchuan Tian, Xu Tan, Rongjie Huang, Songxiang Liu, Xuankai Chang, Jiatong Shi, Sheng Zhao, Jiang Bian, Xixin Wu, et al. Uniaudio: An audio foundation model toward universal audio generation. arXiv preprint 1rXiv:2310.00700 2023.
Qiang Yang and Yuanqing Zheng. Deepear: Sound localization with binaural microphones. IEEE Transac- tions on Mobile Computing. 23(1):359-311 375, 2022.
Masahiro Yasuda, Yuma Koizumi, Shoichiro Saito, Hisashi Uematsu, and Keisuke Imoto. Sound event local- ization based on sound intensity vector refined by dnn-based denoising and source separation. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 651-655. IEEE, 2020.
Xueyao Zhang, Liumeng Xue, Yuancheng Wang, Yicheng Gu, Xi Chen, Zihao Fang, Haopeng Chen, Lexiao Zou, Chaoren Wang, Jun Han, al. Amphion: An open-source audio, music and speech generation toolkit. arXiv preprint arXiv:2312.09911 2023.
Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. 03160
The sound of pixels, 2018. URL
Hang Zhou, Xudong Xu, Dahua Lin, Xiaogang Wang, and Ziwei Liu. Sep-stereo: Visually guided stereo- phonic audio generation by associating source separation. In Computer Vision-ECCV 2020: 16th Eu- ropean Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XII 16, pp. 52-69. Springer, 2020.
18

---

Published as a conference paper ICLR 2025
A COMPARISON WITH TO PREVIOUS WORK
Compared to the generation tasks supported by other models, model not only supports T2A and 12A shown Tab. A8 but also enables controllable spatial audio generation. Overall, our model mainly supports various related generation tasks based spatial audio.
Table A8: Comparison audio generation methods. "T2A" means text-to-audio. "I2A" means image-to- audio. "V2A" means vidco-to-audio. "A2A" means audio-to-audio (audio impainting, style transfer..). / means "support". X means "not supported". means "partial support"; although the networks were not specifically designed for this purpose, their model demonstrates some capability meet the demand.
Supported Modality
Contrillable Features
T2A
12A
V2A
A2A
interactive2A
temporal
Starco
spatial
audiogen Kreuk al.
2023
X
X
Yung
Unified-IOr 10
Lar mi 2024
X
X
X
Make-an ando-mii
2023b
Make-an Bakke-an-adle andiaddoo
Hiing el
20250
X
X
Audial DM (Lau dra 2023
X
AudioLDM
TANGO Ghooli er ml. A024
X
X
X
TANGO
Majunidy
2024
PicaAudio Aic at 211410
Amphion Znaig et 2025
/
X
X
Sceing-and-
V2A-Mapper Wa 2021
andiohox Vyds mi 20243
X
stable suble-audio-open (EVan
SpatialSnnic(Onn)
B DATASET CONSTRUCTION PIPELINE
B.1 DATA PREPARATION
We utilize data from AudioCaps (Kim al. 2019). WavCaps (Mei et al. 2024). FSD50K (Fonseca al. 2022), ESC50 (Piczak) 2015) and VGG-Sound (Chen et al.) 2020) as raw data sources to construct our dataset. We first filter samples with captions describing multiple sound sources ensure each audio clip contains only single sound event. We then apply active detection on each audio and obtain 10-second active segments. If the active part shorter than second, it discarded, as short clips may lack sufficient information and be difficult to simulate as a moving source. If the audio duration is less than 10 seconds, we pad it to reach 10 seconds. Then, following Xu et al. (2024); ; Xie et al. (2024a): Huang al. (2023b). a CLAP model ( (Wu et al. 2023) evaluates the similarity between each audio clip and its caption, discarding clips with similarity scores below 0.3. When an audio-caption pair is used in the simulation, we randomly select one from all the corresponding 10-second segments. Tab. B9 provides statistics for the raw data collected from different sources before and after processing.
Table B9: Statistic of raw data and processed data. Since some datasets contain long-form audio with a single global caption that may not represent the local description, activity detection and CLAP filtering with cropping is needed. So, the necessary decrease of data is acceptable.
Before Processing
After Processing
Data
Num. of Audio
Total duration
Num. Audio
Total duration
AudioCaps
39597
110 Hours
12,400
34.4 Hours
WavCaps: FreeSound
262,300
6264 Hours
10,844
30.4 Hours
WayCaps: BBC Sound Effects
31,201
997 Hours
4.047
11.2 Hours
WayCaps: SoundBible
1,232
Hours
683
1.9 Hours
WayCaps: AudioSet SI.
108,317
300 Hours
8,294
23.0 Hours
FSD50K
51.197
108.6 Hours
16,981
47.2 Hours
VGGSound
199.467
550 Hours
157.230
436.8 8 Hours
ESC50
2,000
2.8 Hours
2,000
2.8 Hours
19

---

Published as a conference paper ICLR 2025
Table B10: The attributes frequently used to describe the simulated audio and corresponding options lists. Beyond these words, with the help of fine guidance and GPT, we still are able to generate description of "A dog is barking at 15 to the front left". The moving speed ratio is used as the denominator, so faster speeds have smaller values.
Attribute
Options List
Value
outdoors
100m
large
90m
Scene size
moderate
40m
small
~ 20m
left
N N(10",
front left
,
Source direction 0
frunt
12109
front right
12199
right
NI0",
far
010.6,
Source distance (ratio) d
moderate
(.0.3.0.6)
near
0(0.1.0.3)
still
Movement
moving
slow
(0.75, 0.85)
Moving speed (ratio)
moderate
(0.45, 0.55)
fast
(0.25,
instantly
As for the real-world subset (RW-world), we manually select around 200 samples from FairPlay (Gao &
Grauman 2019) for musical instruments and STARSS23 (Shimada al. (202)) and SimBinaural (Gar et al. 2023 for audio events. Then, invite experts to write the descriptions of spatial audio.
B.2 GPT-BASED ATTRIBUTES DERIVATION AND CAPTION TRANSFORMATION
Attributes. As mentioned previously, the audio simulator requires certain essential attributes to simulate realistic audio, including sound objects, scene size, sound source location, moving direction, and speed. The inferred attributes with corresponding audio clips used to construct audio-object pool and retrieve audio from images. Afterward, we convert the attributes selected by GPT-4 and GPT-4o into numerical values for audio simulation. All attributes and their mapping to values can be found in Tab. B10 The use of these attributes is detailed in Sec. B.4
To obtain reasonable attributes and captions, we apply the Chain of Thought Prompting (CoT) (Wei al., 2023) to enhance the common sense reasoning ability of GPT-4 and GPT-4o. We predefined several can- didates to describe each attribute, with each being an attribute element, such as "far", or "near" for the distance attribute. We require GPT to select attribute labels that match the input information. For direction, we require GPT to provide more detailed values, such as when specific angles appear in the caption when reasoning with images.
For attributes audio-image pairs, we provide image-caption pairs with object positions to GPT-4o, and it infers all attributes because the image and text caption carry enough information. Based on the provided positions, GPT-4o can infer more precise object directions instead of selecting from a list. This makes our generation more accurate. However, for audio-caption pairs, we provide the caption to GPT-4, and it only infers room size and moving speed based on the caption, as other attributes are not related to the text caption. The other attributes of audio-caption pairs are randomly chosen from the attribute lists.
Caption Transformation. As part of the prompts, we also utilize GPT-4 and GPT-4o to transform raw captions into merged captions with positional and movement phrases. We also back up the merged captions without spatial and movement information for further training. Noticing that merging multiple audio-caption pairs results in long and complex captions, we instruct GPT to shorten the final output.
20

---

Published as a conference paper ICLR 2025
Appendix presents all prompts used here with real examples.
B.3 AUDIO RETRIEVAL FROM IMAGE
We retrieve audio from the audio-object pool mentioned earlier to construct image-audio pairs. The current visual-caption pair (Wu et al. 2023) widely used in VLM excels at conveying visual information but lacks plausible acoustic descriptions. For instance, when describing photo of woman, typical VLM descriptions focus primarily on the visual semantics, including behavior, appearance, and the surrounding environment, form the description like "A woman with the white shirt is standing to use language-driven approaches to generate corresponding stereo sounds for images, we seek to capture acoustic-rich details about the direction and sounds from the image, e.g-. "A woman's 's laughter comes from left side". Therefore, we utilize LLM to obtain multiple alternative acoustic captions about objects in the image that may produce sound and finally obtain the description of the direction and possible moving of those sounds.
the left.". However, if we aim
Since we choose a text modality with abstract semantics a bridge to connect multi-channel audio and other modalities, we develop method to obtain tri-modal triplets based on three modalities in order to further promote the development of multi-mode modal guided audio generation.
We employ two methods to utilize images as cues for audio retrieval: 1) 2-C Audio Retrieval via Language Bridge: By using sentence embedding, we match visual and audio elements through text, considering sam- ples with a threshold exceeding 0.9 to be similar. 2) 1-C Audio Retrieval with Simulation: Building on previous research, we initially identified the sound subjects from the GPT-4o. Utilizing these subjects, we extract multiple samples from the audio-subject pool. Taking holistic view, these two methods are basically the same.
Finally, we allow each image to correspond to up to 10 possible audio clips, rather than just one. This approach of one-to-mulle correspondence enhances the diversity of our 12A generation.
B.4 AUDIO SIMULATION
In this section, introduce the details of audio simulation and how use the attributes. We use audio simulators like Pyroomacoustics and gpuRIR to simulate spatial audio.
For static audio. Pyroomacoustics uses inputs such as room size, microphone location, sound source loca- tions, and RT60. RT60 represents the time it takes for sound energy to diminish by 60 dB once the source stops. We randomly sampled RT60 values between 0.3 and 0.6 seconds to ensure data diversity. The simu- lated rooms are cubic, with sizes defined as
[Ro: R1, R2] = [r S S,1, + Er2),
where Ens ~ U.-O..OO,. and is the room size. The room size value can be obtained from its attribute label inferred from GPT-4 or GPT-4o. For the room size attribute, each label corresponds to a specific range. The value is randomly sampled from these ranges: ["small": U5, 20), "moderate": U(20, 40), "large": (40, 90), "outdoor": 100]. An anechoic room mode is used for outdoor scenes. Microphones are positioned centrally, using 2-microphone array with 0.16 0.1.0..m separation. The array's center is
[Mo Mr] = - Ro 2
+ Emo, R1 + Em1: 2
R2
+ Em2).
(10)
2
where Sms ~ U(-0.Ir,0.11) The two microphones are positioned at [Mo, M1 = M2]. where & U(0.08, 0.09). We sample the angle from normal distribution with standard deviation of 11s based on the source direction attribute label: ["left": N(1802, 1121)), "front left": N1135,, 121%). "directly front":
N (900, 121%), "front right": N(45P, 121%), "right": N(0", 121")]. The distance ratio aa that controls
21

---

Published as a conference paper ICLR 2025
the distance between the microphone array center and the source is sampled from ["near": (00.11, 0.3), "moderate": (0.3, 0.6). "large": (0.6, 0.9) The sound source is located at this angle, with distance from the microphone array center of
d ao min Ro Mo: Rı M1, Mo, M1).
(11)
Using variables in equation 10 and equation I1 the position of the source Ubegin can be caleulated as
Mhegin
[o + dsii 0, M1 + doss 0, M2]-
(12)
For dynamic scenes, gpuRIR simulates moving sources with specified trajectories. Room size, microphone location, and source position are determined as static scenes. In the outdoor scene, the absorption weights of six walls are set to 1 x 10 to simulate an anechoic chamber. Additional parameters include the speed and endpoint of the moving source Hend- The speed of the moving source
Ototal T
V
(13)
where Grotal presents the displacement from the start position to the end position, and T donates the moving interval. We can obtain T::
T a Trotal,
where a is sampled based on moving speed attributes: ["slow": (0..75, 0.85). "moderate": (0.455, 0.55), "fast": (0.25,, 0.35)] We also sample a Thegin = (0,00. x Teotal to indicate when the source starts to move. GpuRIR can simulate the moving source by the source trajectory (Po. Pı...., Pt], and each point on
the trajectory P; are calculated as
Pbegiin
for I. <. Thegin for Thegin S S Thegin + T,
P:
(14)
(begii + IV
where Abegin is the start position of source. We calculate the source position every 10ms. In instantly moving scenes, the source changes position to end position suddenly at random mooe U (0.2, 0.8) x Trotal.
the mixed subset, we use gpuRIR to simulate both stationary and moving sources in an outdoor scene. All other operations are the same as in the dynamic scene.
the dataset construction process, some attribute labels are randomly chosen from the list for audio-caption pairs, like source direction, distance, and moving speed. We record all attribute values (including RT60 and microphone location) for model training and statistics. During inference, all attribute labels can be inferred from input captions images with object positions.
C DATASET AND BENCHMARK STATISTICS
C.I AUDIO-CAPTION SUBSET
The dataset is divided into five subsets based on the number and motion state of the sound source:
Single Static Subset (SS-set): One stationary audio.
Double Static Set (DS-set): Two stationary sound sources.
Single Dynamic Set (SD-set): A single moving sound source.
Mixed Set (M-set): to 4 sound sources including both stationary and moving sources
Real-world Set (RW-set): naturally recorded audios with manually written descriptions for testing.
22

---

Published as a conference paper ICLR 2025
6.8
0.7
FSDSSK
DE9050
Narmlloee Raganry
Auudolaps
DVGGSond
02
6.1
0.1
0.2
0.3
0.6
0..5
0.6
0.7
0.8
0.9
Jaccard Similatty
Figure C5: Jaccard similarity between raw descriptions and our rewritten captions in the single static subset. We follow Mei et al. (2024) to conduct this analysis to show a generally low level lexical overlap across various sources.
For the single static subset, we utilize all single-source clips. To synthesize audio clip in the double static subset, we sample and mix two different audios from the single-event database. In the single dynamic subset, sound sources move in two modes: gradual, where the source moves from the start to the end position a constant speed; and instant, where the source changes position suddenly. For "instant", we perceive in the caption that two sound sources emit sound sequentially from different positions. For the mixed subset, we choose to 4 sources from the single-event database, with the possibility of moving. All captions are rewritten by LLM to include spatial and moving instruction. The are used to construct our test set, while all other data is used for the training set. The specifics of our dataset are detailed Tab. CI1 As mentioned in Appendix B.2, we also retrieve audio from images construct image-audio captions. Tab, C12 shows the detailed number of each modality pair in BEWO-IM.
and validation subsets from AudioCaps
Table C11: Audio-caption subsets statistics of BEWO- IM dataset. The four main subsets have rather uni- form distribution. Moreover, a real-world subset constructed manually to verify the perception consistency
between simulation and real-world.
Subset
Total Duration
Num. Audio
Avg, Caption Len
Max. Caption Len
Min. Caption Len
Single Static Subset
835.03 Hours
319,259
12.44 Words
Words
Words
Double Static Subset
573.95 Hours
205.880
19.54 Words
Words
4 Words
Single Dynamic Subset
572. Hours
205,
13.06 Words
77 Words
3 Words
Mixed Subset
791.74 Hours
285,00
24.20 Words
64 Words
6 Words
Real-World Subset
0.6 Hours
200
14.17 Words
34 Words
6 Words
Table C12: The details of modality pairs involved BEWO-IM.
BEWO-1M Text-Audio pairs
number
1.016k
Image-Text pairs Image-Audin triplets
3.2k
113K (20K unique audios)
presents the Jaccard similarity scores, it shows that our caption is quite different from the original caption. Coupled with the observed increase in length that includes spatial information, this suggests that transformed captions have significantly changed from the original descriptions and incorporate substantial additional information.
Fig. C5
23

---

Published as conference paper ICLR 2025
Table C13: Examples of captions our dataset. The final caption should not only involve the spatial context
but also maintain the concise of the sentence.
Task
Raw Caption
Final Cuption
Single still
A cell phone is vibrating.
A cell phone vibrating on the right side of the scene.
Double still
Prinier printing.
printer is printing on right of the scene, while the person is playing the didgeridoo directly in front.
Playing didgeridoo.
Single dynamic
Trumpet is being played.
right to front An engine slowly dying down is noticed on the left, children's laughter and whistling gemily move direcily front left.
Trumpei sound moves
muderrate speed.
A vehicle's & engine starts die down.
Mixed
Young children are whistling laughing.
Table C14: Correctness statistics of captions for each training subset without manual correction, A random samples from each subset was checked ensure captions match the simulated audio.
Subset
Correctness
Single Static Subset
97.87%
Double Static Subset
87.23%
Single Dynamic Subset
95.74%
Mixed Subset
91.48%
We show some caption examples in Tab. CT3 more examples are provided in our demo page hutps//immeersve-audee github.o We offer the raw caption and final caption the readers for reference. Tab. pre|pntssnts the correctness for different training subsets without manual check. We also test the over- all accuracy of attribute inference and caption transformation in the inference process, and overall, and it achieves an acceptable performance of 91.52%.
C.2 12A BENCHMARK SUBSET
As mentioned in Appendix B.3 we retrieve audio based on the image-caption pairs to construct our dataset. We use 2017-2017 (Lin et al. 2014) to obtain the train set and test set 12A. The LLM we use to obtain the acoustic semantic description is GPT-4o. During text retrieval, we extract the sentence embedding using SentenceTransformee Transformer" perform exact retrieval using Euclidean distance (L2). Our algorithm restricts each image to match with up to 10 audio files. For the approach of 1-C retrieval with simulation, the extra simulation based on inferred attributes similar to Appendix BB4 should be carried to form a dataset comprising 3.2k images and 20k audios. For the test set, experts are invited to evaluate the audio and image correspondence and drop all the low-quality samples manually.
constructing Image-Text-
triplets, we employ the FAISS library to
after retrieval. Ultimately, we generate 113k triplets
C.3 INTERACTIVE2A BENCHMARK SUBSET
We select 150 images from the COCO-2017 test and them allow real to choose the objects of interest. Each image is annotated by at least 4 participants. who identify the objects using boxes and points using makesense In total, there are about 300 boxes and 300 points used for testing.
C.4 DATASET LICENCE
We have taken investigation into each previous dataset involved in our BEWO-IM. For the purpose of open access, we follow each dataset involved in BEWO-IM and apply the license including the Creative Commons Attribution (CC BY) license to any Author Accepted Manuscript version arising.
'hups://sbert.net/
https://gituhbcoommaaccbwssesses hhtps//wwwwwmakeeeess..ii
24

---

Published as conference paper ICLR 2025
Figure C6: Interactive box labeling.
Figure C7: Interactive point labeling.
D BASELINE DETAILS
D.I TEXI-TO-SPATIAL-AUUIOO BASELINE
We select three popular T2A models to construct the baselines: AudioLDM 2, Make-an-audio 2, and Stable Audio Open. The model structures of them remain unchanged. They extract features from the text using TS and CLIP and use these features to guide the audio generation process. AudioLDM and Make-an- audio 2 can only generate mono audio, on U-Net, which has been proven effective. Same as the previous approach (Xu et al. 2021 Zhou et al. 2020). this filter also leverages position features as conditions to derive suitable mask based on ITD and ILD training in mono-to-bee mono-to-binarual reconstruction task. This mask, combined with basic operations on the generated mono audio, results in stereo audio. To facilitate more comprehensive comparison of different baselines, we fine-tune and optimize the previously advanced mono audio generation models while ensuring consistency all training datasets. Stable Audio Open can generate stereo audio directly, so we directly fine-tune it based on training dataset.
apply learnable conditional filter (Zhou et al. 2020) based
While the filter performance appeared promising according to Xu et al. (202T); Zhou et al. (2020), when confronted with large-scale dataset, low performance persisted despite our efforts to scale up the number of layers and parameters, as illustrated in Tab. 3 Moreover, on the demo page, we have presented qualitative comparative cases showing that techniques relying mono-channel generation models and filters exhibit fluctuating II.D and inadequately capture ITD. This underscores the strengths our end-to-end dual-channel audio generation approach.
D.2 -AAGE-TO-SPATIA-AAUU BASELINE
Similar to the construction method the Saat-to-Spatil-Auudie bascline, we utilize an Open-domain Image- to-Audio model (Xing al. 2024 generate monophonic audio. Additionally, sequentially append filter based on image conditions at the end of the model to facilitate the transition from mono-channel to binaural audio.
25

---

Published as a conference paper ICLR 2025
D.3 INTERACTIVE-TO-SPAIIAL--- BASELINE
Sce2Sound (Dagli et al. 2024 also leverages the Rol strategy to capture positional information of different objects. See2Sound begins by performing universal image segmentation to identify regions of interest for various objects in the image. Consequently, a small number of Rols in an image based on user- interretedcted bounding boxes and points, while keeping other components like the Depth-anything model unchanged. This zero-shot method is then used establish the baseline.
naturally employ filtering and selection approach to extract
Additionally, create other comparable baselines, enhance the Maage-to-ppatil--uudie model (Xing cet al.) (2024) by incorporating the SAM to set the relevant pixels transparent. By focusing on the region of interest that is not transparent, we then proceed with spatial audio generation based on the processed image as discussed in Sec. D.2
E MODEL. DETAILS
E.1 MODEL STRUCTURE
We use the oobleck (Jang et al. 2023) pre-trained by Stable" The configuration of our model includes an audio setup with 2 channels. The encoder features a hidden size of 128, while the decoder operates with an input channel size of 64 and an overall channel configuration of 128. The architecture employs channel multiples set at [1. 2, 4, 8, 16] and utilizes downsampling ratios of [2, 4, 4, 8, 8). effectively capturing the complexities of the audio data. This configuration is crucial for achieving the desired fidelity and efficiency in our audio processing tasks. It is trained by multi-resolution STFT loss with the left, right, summation, and subtraction of the dual channel. Moreover, discriminator like EnCodec (Défossez et al. 2022) can refine audio with multi- restuuiosoo to achieve finer quality.
Continuous Auto-encoder:
Diffusion Transformer: Our diffusion model is tailored to process 10-second audio samples with two chan- nels upon S It integrates a conditioning mechanism consisting of multiple configurations: a text-based prompt processed by '5 transformer model ("T5-base" with maximum length of 128), and azimuth state encoding with an output dimension of 768. The conditioning dimension is set at 768. The diffusion compo- nent utilizes a Diffusion transformer (DiT) with settings that include 64 input/output channels, an embedding dimension of 1536, 24 layers, 24 attention heads, and both local and global conditioning dimensions of 768 and 1536, respectively. Notably, the transformer operates with projecting condition tokens and adheres to a continuous transformer architecture. For training. an exponential moving average (EMA) is used alongside an Adam W optimizer with a learning rate of 2e 5, beta values of [0.9, 0.999], and weight decay of l 1e-3. 3, complemented by an R. scheduler that features an inv -a---aam of 1e6, power of 0.5, and a high warmup proportion of 0.99. This configuration underscores our commitment to refining audio quality and temporal alignment in generative tasks. During inference, we use the DPMSolver+ Lu et al. (2022) for 100 steps with classifier- free guidance (scale of 6.0). The adapter mentioned in this paper is a lay-r MLP.
Text Encoder: A pre-trained T5-base" encoder is utilized in the main experiment. The CLAP encoder we use in the ablation study is from Laior
Image Encoder: The classic Mask-RCNN (He et al. 2017) is used the detection model. Since the regional feature of Mask-RCNN is about the class itself rather than the behavior, the regional CLIP feature is then used understand both the behavior and class of each object. Our bidirectional multimodal encoder
26

---

Published as a conference paper ICLR 2025
comprises a stack of 12 transformer blocks, each featuring self-attention layer and fully connected layer, enhanced by residual connections. Similarly, our decoder consists of another 12 blocks stack mirroring the encoder's architecture but with each block augmented by an additional cross-attention layer. The text prefix is set to "image acoustic captioning:". The image network with regional perception is first trained as the image caption task and supervised by the acoustic description of the image. We train this bidirectional multimodal encoder for 100 epochs on 1 x NVIDIA RTX 4090. After training the T2A model, the pertained image encoder is built on top the T2A model. After another 100 epochs of iin-tunnig, all the I2A tests and Interactive2A tests carried out the same checkpoints.
Azimuth Fusion Module: This module consists of a 4-layer of cross-attention module with 4 heads. The mentioned in Fig. eequls to 4.
Interactive Matching: 1) A straightforward method for matching and filtering at the coordinate level. Initially, we SAM to generate high-quality mask, which is then matched with region coordinates C detected globally using the Intersection over Union (IoU). To enhance this baseline approach, we adopt the maximum continuation strategy, enabling the selection of multiple areas of interest for feature generation. 2) We further advance this by introducing feature-level matching technique, where we conduct fcature-level retrieval using CLIP between interactive areas and detected regions.
E.2 DIMENSION DETAILS
is variable depending on the complexity text and image. Lg is the sample size that relates to the sampling rate and audio interval. But the overall condition
to condition the whole diffusion is 768-channel, which means Leg = diext = dang = Lelip = dume = 768. The variance o = is obtained empirically from the real-world distribution. The maximum region number is Nmaz = 38 following Cho al. (2021) Luii is set to 64 and Lets is set to 80.
E.3 TRAINING MATRIX AND CAPTION COMPOSITION
During data construction, in addition to the spatial caption, we also transform and preserve the caption without spatial phrases for future training enhancement.
During T2A model training, three types of guidance compositions are used: coarse azimuth matrix with the transformed caption, fine azimuth matrix with the original caption, and fine azimuth matrix with the transformed caption. This joint training method requires the model to consider both the description and azimuth state matrix to generate the spatial audio.
However, for 12A training, only the fine azimuth matrix is used train the model.
METRICS DETAILS
F.1 1-C METRIC DETAILS
We use several common metrics in 1-C I-Cevaluation. 1) Fréchet Distance (FD) measures the similarity between two distributions. 2) Inception Score (IS) evaluates data generation by assessing diversity and resemblance to real data. 3) Kullback-Leibles divergence (KL) quantifies divergence between two probability distributions. 4) Fréchet Audio Distance (FAD) assesses audio quality by comparing generated audio real audio samples. 5) CLAP score evaluates the aligmment between audio and text for coherence in multimedia tasks. 6) overall impression (OVL) subjectively measures the general quality and appeal of generated content. Audio-text relation (REL) assesses coherence and relevance between audio content and accompanying text. 8) The
CLIP score measures the alignment between images and text using the CLIP model's shared space.
27

---

Published as a conference paper ICLR 2025
F.2 GCC-PHAT AND STEREOCRW DETAILS
The objective of the interaural time difference estimation problem is to ascertain the difference in arrival times of a sound at two microphones. In stereo recording scenario with 2 22 E R denoting waveforms, and function h Rn Raxd which calculates features for each temporal sample, frequently employed approach involves selecting time delay T that maximizes the generalized cross-correlation.
Rx x2 (T) = Eg [h1 h2(t - 7)]
(15)
where h; h(x;;) the features for Fy, and h; (t)) is the d-dimensional feature embedding for time The visualizations of TDOA over the 12A task can be found in Fig. F8
(a) Naturally recorded sample
Baseline sample
Ours sample
Figure F8: The visualization of DTTA across the time. (a) The real-world recorded sample is selected from FairPlay. (b) Baseline sample from BEWO-IM image test set. (c) Our sample BEWO-IM from our image set.
The popular Generalized Cross Correlation with Phase Transform (GCC-PHAT) tech- nique (Knapp & Carter 1976, (Salvati et al., (02T) whitens the audio signal by normalizing with the magnitude of the cross-power spectral density. This method offers the maximum likelihood solution in specific ideal, low-noise scenarios.
GCC-PHAT:
StereoCRW: As more advanced deep learning-bass based method for ITD estimation, StereoCRW (Chen et al., 2022b) aims to autonomously acquire these connections by utilizing advancements in tracking techniques. By modifying the contrastive random walk approach, a cycle-consistem representation is developed from unlabeled stereo sounds.
This metric computation process involves averaging the ITD results of the entire audio and then calculating the mean square error, treating the video as perceived difference in overall auditory channels. While this method exhibits high accuracy and computational efficiency for determining the direction of single sound source, its limitations are apparent in scenarios involving moving sources and multiple sources in different directions. Specifically, when calculating ITD, time unit of 0. seconds is employed. Subsequently, the overall video's ITD is augmented for significance by multiplying it with a coefficient of 100.
GCC MSE and CRW MSE:
FSAD: To address scenarios with moving and multiple sound sources, adapted the FAD methodology by leveraging the features from the final layer of StereoCRW, supplemented with adaptive pooling to derive a 2560-dimensional representation for the entire audio. Subsequently, FD is computed using this representa- tion. Through this approach, we have successfully developed a model for evaluating the generation task in complex sound source scenarios.
28

---

Published as conference paper ICLR 2025
All evaluation codes will be publicly accessible.
F.3 FSAD ANALYSIS
FSAD processes the ITD features from multiple time slots to obtain the distance of temporal perception. Given that the GCC MSE and CRW MSE represent only the temporal average ITD, in some moving scenar- ios, the effects from the left and right parts counteract each other. In the samples of Table [15 changes in the moving direction between No.1 and No.2 are not reflected by the MSEs, but perfectly reflected by the FSAD, Therefore, in moving scenarios, FSAD can be treated as better metric.
Table F15: Some cases to show the effectiveness of FSAD.
Type
No.
Source audio
Target audio
GecPHAT MSE
CRW MSE
FSAD
Single Dynamic
A car moving from left right.
car is moving from left to right.
0.08
Single Dynamic
2
A car moving from left to right.
car is moving from right to left.
0.58
0.45
4.43
Single Stationary
duck quacking on left.
duck quacking on the right.
132.13
107.19
4.90
MOS DETAILS
Table F16: Objectives and Interfaces of different tasks.
Prompt Type
Objective
Interface
(I-C)
Fidelity & Consistency (previously called OVL and REL)
Figure
P9
Image (1-C)
Fidelity & Consistency (previously called Fidelity and Relevance)
Figure
F10
Bounding Box
Event relevance & Direction relevance
Figure
F1
Point
Event relevance Direction relevance
Figure
Image
Event relevance & Direction relevance
Figure
Text
Event relevance & Direction relevance
Figure
F14
We conduct all subjective evaluations online with 15 participants using Amazon Mechanical Turk (AMT) (Crowston] 2012). Listeners evaluate the fidelity, consistency, and relevance of the events or di- rection of each sample on 5-point Likert scale while listening through headphones in a quiet setting. The evaluation consists of six tasks:
Participants are provided with an audio clip and an optional text caption. They evaluate the natu- ralness of the audio and its relevance to the caption. (See Fig. F9).
Participants receive an audio clip and an optional image. They rate the naturalness of the audio and its relevance the image. (See Fig. FTOP.
Participants are presented with an image containing a bounding box and corresponding audio. They score the agreement between the sound event and the object the bounding box, as well as the directional accuracy of the sound relative to the bounding box. (See Fig. FIT.
Participants receive an image with point and corresponding audio. They evaluate how well the sound event matches the pointed object and the directional accuracy of the sound relative the point. (See Fig. F12).
Participants are given an entire image and corresponding audio. They assess the alignment between the sound event and the image, and how accurately the sound direction matches the image. (See Fig.F13).
Participants receive caption and corresponding audio. They score the alignment between the sound event and the caption, and the directional accuracy of the sound concerning the caption. (See
hhaps///rqqueserrmuu.ccom
29

---

Published as a conference paper ICLR 2025
The evaluation objectives and interfaces are shown in Tab. F16 In the first four tasks, the scale options are: "1. Excellent Completely faithful events/direction". "2. Good Mostly faithful events/direction", "3. Fair Equally faithful and inconsistent events/direction". "4. Poor - mostly inconsistent events/direction" "5. Bad Completely inconsistent events/direction.. In the other two tasks, the scale options are: "1. Excellent Completely natural/faithful", "2. Good Mostly natural/faithful", "3. Fair Equally natu- ral/faithful and unnaturaliinconsiseen"" "4. Poor mostly unnatural/inconsitenn" "5. Bad Completely unnatural/incoonistemm"" Note that the scoring order in the Turk interface is reversed compared to that in our paper: in Turk, a score of represents the best, while in the paper, score of 5 represents the best.
G SUPPLEMENTARY EXPERIMENTS
G.1 COMPARISON OF 1-C METRICS ON CLOTHO
Although it is now more common to use real-data AudioCaps as the test set including Make-An-Audio 1&2, Au 1&2, Tango 1&2, we still manage conduct zero-shot experiments Clotho (Drossos al. 2020) to showcase the comparative performance. As shown in Table. G17, our SpatialSonic demonstrates comparative performance across almost several metrics, particularly in FD, FAD, and ISc. It slightly lags behind the popular methods on some metrics but overall, it appears to be the effective model among those listed for generating high-quality and relevant audio content.
Table G17: The 1-C zero-shot generation comparison on less usual dataset Clotho. All the methods listed below is tested on the evaluation set of Clotho.
Objective FADI
Subjective OVLi
Model
CLAPt
KL_
REL1
Make-an audio
0.331
27.32
6.10
6.94
3.15
3.27
3.32
lake-an-audio2
0.343
19.10
3.48
8.19
2.47
3,56
3,58
Audioldm2
0.340
25.39
3.49
7.93
2.62
3,47
Tango2
0,363
22.72
3.39
9.66
2.21
3.53
3.49
SpatialSonic (Ours)
0.361
18.81
3.37
10.31
2.36
3.61
3.63
G.2 COMPARISON OF 1-C METRICS ON DIFFERENT SUBSETS
Previous mono-channel metrics tested on dual-channel text-to-audio systems have revealed significant limi- tations. The CLAP score struggles to adapt to the challenges posed by dual-channel audio and longer texts. Since CLAP is trained on mono-channel data and text lacking directional information, its utility in eval- uating dual-channel audio is insufficient. Metrics such as FD, IS, KI, and FAD, though widely used, are originally designed for mono-channel applications and do not fully capture the quality of channel differences dual-channel generation. Therefore, the overall mono-channel evaluation remains somewhat inadequate for stereo audio.
For deeper insights, we analyze mono-channel evaluation metrics across various subsets in BEWO-1N. by channel-wise average in Tab. GI8 G18 Notably, CLAP's overall evaluation scores are relatively low due to its in- capacity to handle long-form captions and spatial phrases. Furthermore, from the mono-channel evaluation metrics, it is evident that audio generation for single sources is relatively straightforward. However, plexity increases with factors like long captions, multiple sources, and motions, the difficulty of generation also rises, leading corresponding variance in quality.
30

---

Published as conference paper ICLR 2025
Figure F9: MOS interface of audio-text pair.
Figure F10: MOS interface of audio-image pair.
Figure F11: MOS interface of bound box.
Figure F12: MOS interface of point.
Seltcc
Cued
Coca Mair
Sunggng
Figure F13: MOS interface of the entire image.
Figure F14: MOS interface of text caption.
31

---

Published as a conference paper ICLR 2025
Table G18: I-C audio objective comparison different subsets in our dataset. RW-set a completely unseen set, there still exists distribution gap between ground truth audio and generated audio. But it does not impediment the validation of using such a matrix on source azimuth evaluation.
BEWO-IM
Ohjective
Subset
FDL
1St
KLJ
FAD,
CLAPt
SS-set
24.62
10.33
1.66
2.30
0.4026
SD-set
17.33
13.09
1.60
2.14
0.3501
DS-set
25.20
2.09
2.16
0.3437
M-set
21.77
9.59
2.25
1.79
0.3824
RW-set
33.37
5.87
1.36
4.26
0.3873
G.3 COMPARISON ON PREVIOUS 12A BASELINE
ImageHear (hhfferr & Adi 2023) is an evaluation dataset containing images of specific categories used in previous channel audio. Given that model relies close-set detection model COCO-2017 for region-level perception, certain objects in ImageHear are undetectable. To address this limitation, we employ the GPT-4o and suitable prompt generate dual-channel output in language-driven approach, which is then averaged to produce mono-channel video. Our evaluation of performance metrics is outlined in Table. G19
Table G19: Image guided single channel audio generation test on Imagehear. V2A-mapper! is not open- source for now, in which we use the demo in their demo page to carry out the testing.
Objective CLIP-Score
Subjective
Method
Fidelity
Relevance
Im2Wav
9,843
3.43
3.13
CLIPSonic-IQ
11.392
3.70
3,03
V2A-mapper'
11.950
3.77
3.41
SpatialSonic(Ours)
11.976
3.75
3.42
Overall, our model demonstrates commendable generalization on the ImageHear dataset in comparison to mono-channel audio.
Furthermore, based on the previous work Dagli et al. (204)) we also attempt to compute previous eval- uation metrics. We utilize the audio outputs and images create modified scene-guided audio generated
using AVITAR. We present the average audio similarity scores among multiple such generated audios in Table. 620
Table G20: Image guided multi-channer audio generation on AVITAR.
Sec2Sound Eval
BAG Bench Eval
(+AViTAR)
MFCC-DTW]
7CRt
Chromat
Spect
MFCC-DTW
7CR
Chromat
Spect
CoDi
0,800
0.80
0.70
0.85
0.730 x10
0.53
0.37
SEE-2-SOUND
0.034 10
0.95
0.77
0.95
0.026 x
0.91
0.61
0.51
SpatialSonic(Ours)
0.027 10
0.97
0.77
0.97
0.021 3
0.93
0.53
G.4 x2 -TST OF USER PREFERENCE
In previous studies (Dagli et al. 2024). researchers initially assume a uniform distribution of generated audio quality and spatial information. Then they survey different samples and utilize chi-square (x2) ) tests to reject this hypothesis of uniform distribution. This process further substantiates strong correlation between the generated spatial audio and human perception elements like audio quality and spatial awareness. While this evaluation metric holds statistical validity, it proves insufficient for supervised generation assessment.
Nevertheless, we still conduct statistical analysis of the survey, yielding a series significant p-values in Tab. G21 We maintain the initial hypothesis that the responses adhere uniform distribution. Instead of
32

---

Published as a conference paper ICLR 2025
relying on continuous perception as in (Dagli al., 2024), we formulate multiple-choice questions for each target. In these questions, we designate 3, 4, and 5 as correct choices on 1-5 scale. Our user testing involves approximately 120 human evaluators, and we present the relevant metrics from our human evaluation. This p-value indicates that users express statistically significant preference for spatial audio quality, sound direction, and overall audio quality.
Table G21: x2-test test with the hypothesis of the responses following a uniform distribution. The significance of human preference is evaluated with xt-tests at p < 0.05.
Metric
v-vlle (1)
Spatial Audio Quality
Realism
3.99 x 10-1
Immersion
6.18 x 10-3
Accuracy
7.97 x 10
Clarity
9.08 x 10
Consistency
2.09 10-6
Audio Identification
Overall Localization
x 10-5
Audio Direction Identification
2.27 x 10-5
Distance Identification
1.93 x 10
Audio-Image Matching
Events Identification
2.09 x 10-6
Spatial Identification
1.81 x 10-7
Box Matching
Identification
x 10-6
Spatial Identification
3.18 x 10-5
Audie-point Matching
Events Identilication
9.11 x 10
Spatial Identification
1.82 x 1.5-
G.5 AUDIO-LANGUAGE RETRIEVAL
We follow the Audio-Language Retrieval experiments WavCaps (Mei et al. 2024). where the retrieval model learns Acoustic Semantic Embeddings (ASE) map audio clips closer to their paired captions in the embedding space.
Models: Our model architecture integrates audio encoder for audio representation and a language encoder for text representation. Specifically, we employ HTSAT (Chen et al. 2022a), a transformer network, the audio encoder and pre-trained BERT (Devlin et al. 2019) network as the text encoder. To project features into a shared embedding space, we implement a 2-layer MLP as the adapter for both encoders.
For single-channel experiments, first establish baseline models trained on Au- dioCaps and Clotho datasets. We then develop zero-shot models using WavCaps and BEWO-IM, utilizing 30% of the total training data from BEWO-IM. For stereo audio in BEWO-IM dataset, we perform chan- nel averaging to convert it to mono. To demonstrate BEWO-IM's pre-training capability, we subsequently fine-tune these zero-shot models using AudioCaps and Clotho datasets, respectively. The baseline and zero- shot models are trained for 15 epochs with a batch size of 128 and a learning rate of 5 x 10-5 with the Adam optimizer while fine-tuning conducted for 20 epochs on AudioCaps and Clotho. All audio inputs are standardized to 10-second segments through cropping or padding, and model checkpoints are selected based on validation performance. We evaluate the performance using Recall at rank & (R @k) on the test sets of AudioCaps and Clotho. R@k is 1 if the correct match appears in the top k retrieved items, and 0 other- wise, averaged across all queries. For dual-channel experiments, we maintain settings identical to those of the single-channel setup. We first establish baseline by evaluating the BEWO-1M 1M test set WavCaps- pretrained model using channel averaging. Another dual-channel retrieval model is trained on 30% data of the BEWO-1M training set.
b) Experimental Setup:
33

---

Published as a conference paper ICLR 2025
Table G22: The evaluation results of I-C audio-language retrieval on the test sets of AudioCaps and Clotho. A higher score means better performance. "ZS" refers to zero-shot, "FT" refers to fine- ine-tunee "A~*B" means the model is pre- trained on the dataset "A" and then tnnettuned on the dataset "B".
AndioCaps
Clotho
Model
Training Data
Text-tn-addi R@St
Andio to
Text-so audio
R611
R@
R611
R@51
R@100
RB 11
ReSt
R@101
R@
R85-
R@:101
HISAF-BER BERT (Baseline)
ACoCoooho
39.2
74.9
86.3
49.5
819
91.5
15.6
38.4
520
21.0
43.8
55.7
HTSAT BERT-Z5
Caps
28.6
61.1
75.8
40.2
69.4
80.3
16.5
35.8
50.9
20.0
43.3
56.6
HTSAT 2ERT-ZS
56.3
70.8
28.2
12.0
28.4
10.0
HISAT-BEKT-FT
WapCaps AACa Clotho
76.5
87.1
54.6
19.7
45.7
52.6
649
HESAT-BEREFT
41.6
77.3
87.7
53.8
83.9
929
18.5
43.0
56.3
21.0
43.7
57.4
Table G23: The evaluation results of 2-C audio-language retrieval on the test sets of BEWO- 1M.
Text-to-audio
Audio-to-text
Model
Training Data
R@1t
R@5t
R@101
R@11
R@51
R@101
TISAATBERT (Baseline) HTSAT-BERT
WayCaps BEWO-IM
10.9
39.3
54.1
11.6
38.6
53
14.5
46.8
61.2
16.0
46.2
61.5
<) Results and Analysis: The 1-C retrieval results presented in Tab. G22 demonstrate that our dataset achieves a comparative performance to the popular models across all metrics, showcasing pre-training ca- pabilities comparable to other large-scale datasets. In the dual-channel experiments, Tab. G23 shows that the 2-C retrieval model trained on BEWO-IM achieves superior retrieval performance compared to the one trained on WavCaps on the BEWO- IM test set. Notably, the BEWO- IM dataset and the retrieval model are not specifically designed for 2-C audio retrieval tasks; therefore, the experiments in this section provide sufficient proof for the extensive value and impact of our dataset.
G.6 AUDIO CAPTIONING
Audio captioning aims to represent audio content using natural language descriptions. In this part, we follow the Automated Audio Captioning experiments in WavCaps.
a) Models:
Audio captioning typically employs an mncoder-decoder architecture. The audio encoder extracts acoustic features, which then utilized by language decoder to generate natural language captions. In our implementation, we adopted CNN14 14 from PANNS (Kong et al.) 2020) as the audio encoder and a pre-trained BART (Lewis et al. 2019) model as the language decoder.
The setup of audio captioning is similar to our audio retrieval experiments. For single-channel data, we first train zero-shot models separately using WavCaps and BEWO-IM. Only 30% data of the BEWO-IM training set is used to train the model. Subsequently, we fine-tuned these pre-trained models on AudioCaps and Clotho datasets. The zero-shot models are trained for 15 epochs with learning rate of 5 x 10-5 Then, upon training on BEWO- IM, models are fine-tuned for 20 epochs with a learning rate of 10-6 The final checkpoints are selected based on validation performance, and evaluation performed on the test sets of AudioCaps and Clotho. We employ standard captioning metrics for evaluation, including BLEU (Papineni et al., 2002), ROUGE (Lin 2004), METEOR (Banerjee Lavie 2005). CIDEr (Vedantam et al. 2015). SPICE (Anderson et al. 2016), and SPIDEr (Liu et al.) (017). For dual-channel evaluation, models are evaluated on the BEWO- test set. When evaluating models pre- pre-ttrineed on WavCaps, the audio input is converted to mono through channel averaging. The dual-channel audio captioning models are trained on 30% data of the BEWO-IM M training set with minor adjustments on the baseline model.
b) Experimental Sctup:
Results and Analysis: As shown in Tab. G24 after fine-tuning, the 1-C model pre- treitra BEWO- 1M demonstrates comparable performance to that pre-trained on WavCaps, with only marginal differences. This is particularly noteworthy given that WavCaps is specifically designed for 1 1-C audio captioning tasks.
34

---

Published as a conference paper ICLR 2025
However, in 2-C evaluations, Tab. G25 shows the model trained on BEWO-IM significantly outperforms that pre-trained on WavCaps.
Table G24: The evaluation results of I-C audio captioning on the test sets of AudioCaps and Clotho. A higher score means better performance. "ZS" refers zero-shot, "FT" refers to fine-tune, "A B" means the model is pee-trraaee pre-trained on dataset "A" and then fine-tuned on dataset "B".
Model
Training
BLEV11
ROUGE/
METEOR1
AudioCape
CIDEit
SPIDErT
BLEU1
ROUGE,
METEORT
Chilhe
CIDEr
SPICET
SPIDErT
BART 25 BART 25 ENN-BART-FT
25.6
15.3
23.3
BEWO
14.5
135
588
7.6
War Capa. Challe
89.3
49.9
24.7
75.6
17.9
46.8
60.1
400
18.5
48.8
13.3
31.0
CNN-BART-FT
BEWO-IM BEWO-IM-ACHCRE ACC-CkIl
63.4
449
2007
57.2
158
364
545
364
16.
38.5
11.4
25.0
Table G25: The evaluation results of 2-C audio captioning on the test sets of BEWO- 1M.
Model
Training Data
BEWO-IM test sets
BLEU,
ROUGE
METEOR
CIDEI
SPICE
SPIDErT
CAN-BART
WavCaps
7,5
14.1
3,9
14.1
7.5
10.3
CNN-BART
BEWO-1M 1M
31,9
35.3
16.3
45.1
22.5
33.8
G.7 EVALUATION ON THE SCALE OF CLASSIFIER FREE GUIDANCE
20
0.68
18
0.67
16
0.66
0.65
0.64
0.62
6
4
0.61
AndioCaps@1St
AudioCapo@JDD
AudioCaps@CLAPP
2
0.6
CFG
0
0.59
10
Figure G15: The impact of CFG scale. Overall the scale of 6 is recommended the balance of the trade-off.
Due to the impact of different CFG scales on the generated audio, we conduct inference experiments using various CFG scales. As previously discussed, finding the most reasonable trade-off is crucial for achieving the best results. In Table. Table.[G26Jand atively optimal outcomes. Thus, unless otherwise specified, will set the CFG scale to 6 in all subsequent experiments.
our experiments demonstrate that a CFG scale of 6 provides rel-
G.8 EVALUATION ON PERTAINING OF IMAGE ENCODER.
Since not all images have acoustic semantic captions, we conduct tests on the image dataset of BEWO-IM The performance, pre-trained on our acoustic semantic captions, shown in Tab. G.8 reference. Since we only update the encoder for modality alignment, this performance might seem limited. However, we believe that the limited performance in visual captioning will not impede our model's ability develop regional perception of images. Additionally, the boost in performance with regional perception further demonstrates the effectiveness of the strategy.
35

---

Published as conference paper ICLR 2025
Table G26: The impact of CFG scale. Overall the setting of 6' recommended as the balance of the trade-off.
CFG scale m
==
1r == 5
w == 6
=
=
= 9
=
10
Audio Caps@CCCpCCC... CLPP
0.624
0.666
0.672
0.675
0.672
0.670
0.663
AudioCaps @IS
10.63
13.90
AudioCaps HDL
17.89
9.07
14.01
15.31
16.53
17.50
18.74
Mix-set FSAD]
0.147
0.153
0.160
0.167
0.168
0.191
0.243
Table G27: We use the metric of the caption generation to test the performance of the pre-trained image encoder and decoder.
Image Captioning on BEWO-IM
CIDEr
METEOR
SPICET
w/. regional perception
68.3
21.7
17.3
w/o. regional perception
62.1
20.4
16.6
G.9 SOURCE DIRECTION IMPACT ON THE AUDIO QUALITY
We maintain the overall caption and text length while only modifying the directional description in a few simple samples to assess the quality of the generated audio. Interestingly, we observe that larger disparity between the ears (as the sound source moves further left or right) leads to a reduction in audio quality, as shown in Tab, G28 However, it is not due to any bias in the dataset regarding sound source locations because we control its proportion in simulated audio. We attribute this to the fact that the entropy the latent code is limited, and the additional positional information competes with and displaces part of the original latent code, thereby slightly degrading the audio quality.
Table G28: A larger disparity between the ears (as the sound source moves further left or right) leads to a reduction in audio quality.
Direction
ISct
FD
FAD.
Left
10.72
18.70
5.08
Front left
11.01
18.34
5.15
Front
11.60
16.06
2.52
Front right
11.00
17.47
4.33
Right
10.86
18.80
4.75
Moving
10.48
18.51
4.89
G.10 CAPTION LENGTH IMPACT ON THE AUDIO QUALITY
We conduct extensive generation on 5000 audios to generate simple dog barks, using GPT-4 to extend the length of captions with less meaningful words. We roughly analyze the relationship between caption length and the generated audio CLAP score. It is found that increases in caption length led to a significant decrease in the CLAP score in Table. G16 On one hand, longer captions pose challenges for the generation. On the other hand, the CLAP score does not accurately reflect the quality of longer captions. To determine the specific reasons, further analysis and statistics are required.
G.11 POTENTIAL CLASS LEVEL BIAS
Inherent bias is significant concern for professionals. We conduct further observations to gain deeper insights. Before we begin the analysis, let's clarify this important question. For instance, when a piano recorded, the output from the LLM is typically static and indoors. In contrast, when it comes to living beings like humans or dogs, the LLM often assumes they are moving and outdoors. Therefore, this is a potential inherent bias in the data and the LLM itself.
36

---

Published as conference paper ICLR 2025
C1APa
0.74
072
1
066
Lengon
26 28 32
Figure G16: The chart of caption length causing a decrease and fluctuating on CLAP score.
Therefore, We evaluated five common classes individually. Only minor differences between them are ob- served, likely due complex reasons like data scale and individual differences. Generally, everything appears normal except for the generation of airplane echo sounds. It seems that LLM in data collection con- sistently assumes that airplanes cannot exist in a small Reverberating room, thereby introducing a specific type of bias.
is interesting and somehow controversial. this bias good or bad? We are trying to use "rationality" to avoid unrealistic scenarios like "an Airplane is passing by in the small reverberating room". In this case, this scenario is unrealistic and unnecessary at all. So, could be coin of two sides. But as a pioneer work, we still prove that most of the common objects do not suffer from such bias. The rest of this potential bias is more than welcome to explore in the future.
Table G29; To observe the potential class-wise bias, we select 5 common classes and evaluate each of them separately. Reverberation and distance means the special factor described in prompts and means the ability to control.
Target Classes
Stationary (FSAD)
Moving (FSAD)
Reverberation
Distance
Dog
0.141
0.137
Mun speaking
0.170
0.177
Airplane
0.179
0.164
X
Piano
0.243
0,215
Violin
0.267
0,251
G.12 VISUALIZATION OF COARSE AND FINE GUIDANCE
In addition using captions, also use attributes guide audio generation. Fig. G17 visualizes the guidance. By defining speed, azimuth, moving time, etc., we can fine-tune and infer guidance.
H PROMPTS USED IN DATA CONSTRUCTION AND INFERENCE
We design several prompts for GPT-4 and GPT-4o for captions transformation and attribute inference.
Tab H30 Dataset construction of audio-caption pair. We provide captions and pre-selected at- tributes (start/end positions and movements) to GPT-4. Then we require other attributes (scene size and speed) and the transformed caption.
37

---

Published as conference paper ICLR 2025
la) Course guidance of moving left right
(b) Course guidance scene
(e) Course guidance saif- defined matrix
the
Pine guidance of moving from left right
Fine guidance of sound on left front of the soone
(0) Fine guidance of self- defined aximuth matrix
Figure G17: The visualization of azimuth state matrix before normalization. Time for r-axis and angle for g-axis. Through this guidance, can define speed, azimuth, moving time, etc.
Hab
Dataset construction of image-caption pair. We provide caption, an image, and sition list of detected objects. Then we require sound objects with attributes (scene size, start/end position, and speed) and the transformed caption.
Tab|H32] Inference with caption input. We provide caption to GPT-4. Then we require sound objects with attributes (scene size, start/end positions, and speed).
Tab H33
Inference and interaction with image input. We provide image and a position list of detected objects. Then we require sound objects with attributes (scene size, start/end position, and speed) and the transformed caption.
38

---

caption should be like "a dog barks at left, then another dog barks Published as conference paper ICLR 2025
Data construction of audio-caption pair will provide you with captions and some attribute lists to describe an audio clip. The caption and elements in the lists are in the same order. You have to return new caption and other attributes
size and speed based my input. Please follow the procedures step-by-step strictly! Assess size the scene in which the audio occurs from: Scene size choices: 'louudoors'' '2:large', '3mooderaee, '4:small'.
2) Identify objects' sounds description and add their init directions from input Basic description: 'laugh- ing'. 'Speaking', "Meowing', 'Blowing', 'Pouring'
Read the movement of objects from imput. If not moving, skip step
4) the objects are moving, also read their end directions from input, and choose speeds for the moving object from: Speed choices: 'I=slow"; "2:moderate', '3:fast", '4:instantly'. 5) If the speed is '4:instantly'. you need to use special method rewrite the caption. It's not an object moving fast. you have to think of there
sound sources sounding sequentially from different positions. For example. if imput is "a dog barks"; movement:| mmovin"" "]: direction:["leti end direction: peattiooo""rihht"" ("right"], And you think its speed is "instantly". at right".
6) Ensure that the moving speed and scene size you choose correspond realistically with objects in the real world.
7) Brief me only overall sound description of possible audio with appropriate direction words. Try use less than 30 words for the caption. And ignore the other things that cannot produce sound.
8) All objects should be described with direction and movement. If it's still, you can drop the movement words. If an object has sound description, you have infer reasonable sound for it.
9) Provide your response in JSON format beginning with "(" like examples below. Example: with input as: caption: ["bus idles", "woman "]: direction: ['left", "right"]: movement:("mg moving", "moving"" i end direction:("leff" "'"ret"""dirretl front"]. You should respond: {"sizee: 1, ","objects": : (). "Woman" (""pped"": ...), "one sentence brief only audio caption": "The bus engine idles the left and woman walks from right to front slowly.")
("Bus":
Input
Caption: I"A dog is "arking".""uuitar"" init direction: ["front", "right"); movement: nnoo moving", "mov-
ing"): J: end direction:[" ("front""""""" "front left").
Output
[ ("size": 2, "objects": { "Dog": ). "Guitar": "speed": 2 J. Jone_sentence,_ coantence_brieffooom bwief_only cautio_c
"A dog barks in front while guitar strums from right to front left moderately."
Table H30: The prompt and an example in data construction based on audio-caption pair.
39

---

Published as conference paper ICLR 2025
Data construction image-caption pair
will provide you with an image with its caption, and a list of positions of objects.
The position list records the locations of some objects,
the top left corner (0, 0) and the bottom right (1,1).
You have return attributes and new caption based on my input.
Please follow the procedures step-by-step strictly!
1) Determine if the entire scene is likely to produce sound. Based on the picture and caption, identify the objects that may produce sound. If one object can make a sound, you can think it sound. If impossible, skip other steps. Sounding choices: '0: impossible', 'l: possible'
2) Assess the size of the scene in which the audio occurs from: Scene size choices: 'Looudoors', , '2:large'. "3:ooderate', '4:small'.
3) Identify the ohjects whose positions are in the position list, and think in the original order of the list. Discard objects that cannot produce sound and those in the position list. Then add the position with basic descriptions
each object based on the position list. descriptions direction and distance are based on the absolute position. not the positional relationship between objects. For example, (0.3, y) should be described as front left. You should consider both direction and distance: direction choices: '1:left', '2:front left', "3:directly front', '4:front right', '5:right'. Distance choices: "1:far', "2:moderate"; '3mear'.. Basic description: 'laughing', 'Spcaking". 'Meowing', 'Blowing', 'Pouring'
4) Identify if the object is moving or not. If not, skip step 5. Moving choices: '0: No moving', '1: Moving' S) some objects move, choose an end position that is different from the init direction, and also choose a speed for the object from: Speed choices: '1:slow', '2:moderate', '3:fast'.
For both init direction and end direction, note that you return decimal number make result precise based on input position. You can simply think the front 3.0, left 1.0 and right 5.0. Thus, with input position (0.3, y), its direction is 2.2.
7) Ensure that the positions, movements, and scene size you choose correspond realistically with objects in the real world.
Brief only overall sound description of the possible audio with appropriate direction and movement words. Try to use less than 30 words the caption. And ignore the other things that cannot produce sound. All the objects should consider the position list, raw caption, and image input.
9) All objects should be described with direction and movement. If it's still, can drop the movement words. You are only required to describe distance if it's
extreme distance.
10) Provide your response JSON format beginning with "(" the examples below.
Example: ("sound": 1, ","size": 1, 1,"objects"; {"Bus": ("init_diine direction":: 1, "init_dis": 1, "moving" :0, "Woman": ""init_direction" 5, "init-dis" "moving": "end direcciio::: 3, "end_dis":2, "speed"; cone_sentencc_brie ceee.ooolyaudo.cpptiiii "The bus engine idles the left and woman walks from right front slowly.",)
Input
Caption: "the large bird is flying threw the air.: Position: [(0.35, 0.25)]
Output
{ "sound": 1, "sizc": 1, "objects": { "Bird": "init_direction": 2.2, "init_dis": 2, "moving": "end. direction": 3.5, "end dis": 1, "speed": } }. seneencntence brief only audio caption": "The bird flaps its wings from front left to directly front moderate speed.")
Table H31: The prompt and an example in data construction based on image-caption pair.
40

---

Published as conference paper ICLR 2025
Inference with caption input
will provide you with caption. You have return attributes based my input.
Please follow the procedures step-by-step strictly!
1) Determine if the entire scene is likely to produce sound. Based on the input, identify the objects that may produce sound. If one object can make a sound, you can think it can sound. If impossible, skip other steps. Sounding choices: '0: impossible'. '1: possible"
Assess the size of the
in which the audio
from: Scene size choices: '1:outdoors' '2:large',
3:mooderate '4:small'
3) Identify objects' sound descriptions. Then determine the position and basic descriptions for each object based on the input. You should consider both direction and distance: direction choices: '1:left', '2:front left', '3:directly front', '4:front right', '5:right'. Distance choices: "1:far', '2moderate', '3:near'. Identify if the object moving
If not, skip step Moving choices: '0: No moving', '1: Moving' 5) some objects move, choose an end position that is different from the init direction, and also choose a speed for the object from: Speed choices: "I:slow", '2:moderate', '3:fast'.
6) For both init direction and end direction, note that you can return decimal number when input involves precise description of the angle. e.g. if the input "a dog barks front left 70 degree", you "init direction": 2.7, or "init. direction":
and you can infer what makes sense.
7) Ensure that the positions, movements, and scene size you choose correspond realistically with objects the real world and match the input.
8) Provide your response in JSON format beginning with "(" like the examples below.
Example: with input "The hus engine idles the left and woman walks from right front slowly.". should respond: ("sound": 1, "size": 1, "objects": ("Bus": {"init direction": 1, "init dis": "moving" :..). "Woman": ["imit direction": 5, "init dis":1, "moving": "end direction": 3, "end dis":2, "speed":1]]
Input
A man speaks in front while a dog barks from front right to left.
Output
"sound": 1, "size": 3, "objects": "Man": ("init_diree dineetdire::io"": : 3, "init_dis": 3, "moving": 0), "Dog": { "init direction": 4, "init dis": 3, "moving": 1, "end direction": 1, "end dis": "speed": 2
Table H32: The prompt and example in text inference.
41

---

Published as conference paper ICLR 2025
Inference and interaction with image input will provide you with image and list of objects' positions. position list records the locations some objects.
the top left corner (0, 0) and the bottom right (1,1).
You have to return attributes and caption based on my imput. Please follow the procedures strictly step-by-step!
Determine if the entire scene likely produce sound.
Based on the picture, identify the objects that may produce sound. If one object can make a sound. you can think it can sound. If impossible, skip other steps. Sounding choices: 'O: impossible', possible' 2) Assess the size of the scene in which the audio occurs from: Scene size choices: "loutdoors', '2:large', m3moodeate'', '4:small'
3) Identify objects whose positions are in the position list, think
original order Discard objects that cannot produce sound and those not in the position list. Then add the position with basic descriptions for each object based on the position list. The descriptions of direction and distance are based on the absolute position, not the positional relationship between objects. For example, (0.3, y) should be described as front left. You should consider both direction and distance: direction choices: '1:left', '2:front left', "3:directly front', '4:front right', '5:right'. Distance choices: '1:far', '2:modcrate', '3:ear''. Basic description: 'laughing'. 'Speaking" 'Meowing', 'Blowing'. 'Pouring'..
4) Identify if the object is moving not. If not, skip step 5. Moving choices: '0: No moving', 'I: Moving' S) some objects move, choose end position that different from the init direction and distance, and also choose a speed
object from: Speed choices: '1:slow' '2moddrraee "3:fast'.
6) For both init direction and end direction, note that you can return a decimal number to make result precise based on the input position. You can simply think the front 3.0, left is 1.0 and right is 5.0. Thus, with input position (0.3, y), its direction is 2.2.
7) Ensure that the positions, movements, and size you choose correspond realistically with objects the real world.
8) Brief me only 1 overall sound description of the possible audio with appropriate direction and movement words. Try to use less than 30 words for the caption. And ignore the other things that cannot produce sound. All the objects should consider the position list and image input.
it's still, you can drop the movement words. You are only required to describe distance it's at an extreme distance.
All objects should be described with direction and movement.
10) Provide your response in JSON format beginning with "{" like the examples below.
Example: ("sound": 1, 1,"size": 1," 1,"objects": ("Bus": ["init__ired directior""" 1, "init_dis"" 1, "moving":O, "Woman": ("init_direction": 5, "init_dis":1. "moving": 11 "end_ directioo::: 3, "end_dis":2 "speed": "one sentencn brief audio captionn:: "The bus engine idles on left and woman walks from right to front slowly.")
Input
Position:((O.6,, (0.4,.0.4,007))) 0.7))
Output
[ "sound": 1, "size": 3, "objects": { "Birdl": { "init direction": 4.2, "init dis": 2, "moving": 0 }}. "Bird2": "init direction": 2.8. "init dis": 2, "moving": }. "one sentence brief only audio caption": "Birds chirp softly from the left and front right."
Table H33: The prompt and example in image inference.
42

---

Published as conference paper ICLR 2025
I FREQUENT ASKED QUESTIONS
1.1 CLAP SCORE ON GT
Based on our experiments, the CLAP score for the ground truth audio in the AudioCaps test set is 0.67, which is the same as Huang et al. (2023)). The process of converting audio to description involves some information loss, which means that certain sounds in the test set may lack appropriate descriptions, resulting in ground truth CLAP score of less than 1. However, advanced models like TANGO 2, Audio-Boo Boxo, and SpatialSonic (ours) are currently able to achieve CLAP scores that approach or even exceed the ground truth. This is common result of these models accurately following text context. And we are willing and excited to open our checkpoints and audio samples for public verification.
1.2 DIFFERENCE BETWEEN STEREO AND SPATIAL AUDIO
Stereo audio primarily uses two channels to create a sense of width by directing different sounds each ear. However, it can be extended to include more channels in certain setups, enhancing the audio experience with additional depth and dimension. This allows listeners to perceive sounds from various directions, including above and below, often using formats like 5.1 or 7.1 surround sound (Herre et al. 2015a b) for a more immersive experience. Both stereo and spatial audio aim to create an immersive listening experience by simulating the natural way we perceive sound in our environment. In this work, we focus solely on azimuth as a metric for stereo and spatial audio. In this context, we equate stereo audio and spatial audio.
I.3 ELEVATION AND FRONT-BACK CONFUSION
audio localization research, sound source localization is divided into dimensions: azimuth and eleva- tion. Researchers use more microphones to balance these dimensions (Jekateryúczuuk & Piotrowski] 2024). our simulation, we use 2-channel omnidirectional microphone array with consistent sensitivity in all directions, which is designed for widely-used dual microphone devices. This microphone array in most of the simulators cannot simultaneously localize azimuth and elevation (Jekaterynczuk & Piotrowski 2024)
and experiences front-back confusion (van der Heijden & Mehrkanoon [022] van der Heijden et al. 2019
Orr et al. 2023). Therefore, when constructing the BEWO-TM dataset, we do not consider elevation or front-back, assuming the sound source to be within the front 180 range in azimuth. Adding more micro- phones, using microphones with artificial numan-shaper ears (Yang & Zheng 2022) and the (lead-Related Transfer Function (HRTF) algorithm
reduce these confusions (Orr al.) 2023) but bring challenges large-scale simulations. Our work proposes data synthesis and model pipeline, which can be adapted for more types of simulations and generations in the future, potentially requiring more detailed captions.
1.4 DISTANCE CONTROLLINGG AND EMBEDDING
The previous See2Sound used the Depth-Anything model for depth perception. However, our model does not specifically encode the distance attribute in azimuth state matrices. This because we require GPT-4 or GPT-4o to describe it in the text when distance is crucial. The distance information is directly presented in the modality embedding without requiring additional encoding in the azimuth matrix.
J REVISION LOG
Specifically, we have included the revision log here with the date format of ht/////yyyy We welcome any suggestions that may contribute the improvement of this paper.
43

---

Published as conference paper ICLR 2025
J.1 BEWO-1.1: 11/1//202
1. Better Presentation
(a) Revision of the summary of the dataset construction pipeline Sec. 3 and the model frame- work in Sec. 4
(b) Revision of image-related part in Sec. 4.2 and Sec. 4.4 for better integration.
(c) Replace MAE to MSE. Sorry about the mistake.
2. Additional Experiments
(a) Inclusion of supplementary experiments utilizing the less usual dataset Clotho, detailed in
Section C.1
(b) Conducting further experiments on text-to-audio retrieval and audio captioning, presented Sections and (isprespeciv These are included even though the primary focus of the paper remains on generation tasks.
3. Insights and Analysis
(a) Integration additional analysis regarding the cfg scale, elaborated in Section G.7
(b) Provision of further insights into the potential class-level bias, which is beneficial for the community, discussed in Section G.11
(c) Extension of analysis concerning the FSAD, documented in Section F33
4. Formatting Adjustments
(a) Minor modifications in Section to ensure the elimination of the widow words.
J.2 POTENTIAL REQUIREMENTS
This section is used to provide the community with maximum effort on spatial audio generation.
Please feel free contact any time if you have any requests from the list below. We will make the requested version of the data available open-source.
BEWO-IM with 44.1K Hz.
IM with 5. 5.1-channel microphone array.
44


Reviews:
[{"review_id": "Zk1nEqvXjF", "reviewer": "Reviewer_YTkB", "summary": "The paper presents SpatialSonic, an innovative framework for generating controllable stereo audio driven by text and images. The proposed model leverages a new large-scale dataset, BEWO-1M, which includes more than 1 million stereo audio-caption pairs. SpatialSonic's one-stage approach, which incorporates azimuth-aware guidance,  addresses the challenges of multi-stage models like computational inefficiencies and error propagation. The experimental results showcase that SpatialSonic consistently outperforms existing state-of-the-art models in terms of spatial quality and subjective human evaluations.", "questions": "The final rate can be changed depending on how the weaknesses are addressed.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Innovative Approach: The paper proposes a unique one-stage generation framework that integrates multimodal inputs (text and images) with azimuth-aware guidance, enabling controllable stereo audio synthesis. \n\n2. Significant Dataset Contribution: The BEWO-1M dataset is a notable contribution,  providing a large-scale foundation for training and evaluating spatial audio models. Its combination of audio, captions, and simulated spatial attributes is a strong asset for advancing future research. \n\n3. Comprehensive Methodology: The detailed explanation of the azimuth-based guidance and the fusion of multimodal embeddings make the technical approach clear and replicable.", "weaknesses": "1. Dependence on Simulated Data: The BEWO-1M dataset, while substantial, is built largely on simulated and GPT-transformed data. This could limit the model's ability to generalize to real-world audio data, which typically exhibits greater variability. Can you show the results on any related real-world datasets, e.g., dataset in [1]? \n \n2. Given the synthetic nature of BEWO-1M, how do you plan to adapt or extend the dataset to include more real-world audio data? Have you considered potential data augmentation techniques or collaborative efforts to collect diverse, annotated, real-world audio? \n \n3. Generalization Issues: The authors note the limitations of the image encoder in handling more diverse or open-world scenarios. This could impact the model's performance in applications that involve less structured inputs. What are the possible ways to eliminate it?  \n \n4. The training process requires substantial hardware resources. Do you foresee any potential strategies to reduce the computational load, such as model distillation or optimization techniques, to make the model more accessible?  \n \n5. You mentioned potential expansions involving 5.1-channel audio and higher sampling rates. What are the anticipated challenges with these enhancements, and how do you plan to overcome them? \n \n6. The code and algorithm are not available; it is challenging to know how the model was trained.\n\n[1]. Huang, R., Huang, J., Yang, D., Ren, Y., Liu, L., Li, M., ... & Zhao, Z. (2023, July). Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. In International Conference on Machine Learning (pp. 13916-13932). PMLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B4gP6dbt7t", "reviewer": "Reviewer_Ph1d", "summary": "This work explores spatial audio generation using latent diffusion. The authors propose to do this using a new 1M stereo audio dataset with paired captions which are LLM generated longform descriptions based on detected sound events where the LLM (GPT4*) includes explicit inferences of distance indicators like 'far/near' and room size; these are then used with a spatial audio simulator. For spatial audio generation, the authors then propose a conditional audio generation setup using text, image and positional (azimuth) conditions. The diffusion transformer model is essentially a fine-tuning on top of StabilityAI's audio work with defined conditions (768d) and using classifier-free guidance. Several ablations tracking both subjective (N=15) and classical audio generation metrics is then performed to assess quality; where the authors also introduce a twist on Frechet metrics to account for stereo-out.", "questions": "* How critical was classifier-free guidance? I wonder to what degree of spatial audio quality was affected by this.\n* Can the authors expand on the FSAD metric definition? How it's computed and why it is better suited to measure in the stereo-out setting? The section on this was so brief that I didn't completely understand it. It would help in the expansion to explicitly articulate why/how it's better than other metrics, and e.g. how meaningful percent improvements are.\n* It would help to break performance differences by certain sound categories, e.g. it's reasonable to expect an \"engine sound\" or \"dog bark\" moving left-to-right, however, it perhaps might be less likely to expect that with \"piano playing\". This relates also to the audio simulation setup (moving audio sources vs moving microphones) and priors that come from the LLM approach (e.g., will \"piano playing\" performance always be lower because its assumed always in the data to be indoors and therefore simulated w/ room reverb? Likewise, one would probably expect \"duck\" sounds to mostly be occurring outside).", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The problem itself is novel one as I don't think I've seen other works tackle the spatial audio generation problem in the same modern setup, e.g. via LDM.\n* I think the dataset portion was the most interesting and seemingly rigorous part of the work and the spatial audio generation was the least. The dataset collection pipeline in general seemed like a sensible approach, short of collecting it manually, whereas the spatial audio generation portion felt like a fairly straightforward application. Good to see the authors' intent to make the data available to the community.\n* I'm neutral on the claimed contribution of the metrics front since I didn't fully understand the benefits from the presentation (see question).", "weaknesses": "* I think the biggest challenge I had was that I was not qualitatively convinced of the quality from the demos. There were maybe only 2-3 samples in total where it felt like it was doing what was expected spatially, eg: \"A duck is quacking on the left.\" or \"A dog is barking on the right.\" don't actually sound like they're coming from their respective directions. That said, the framing of the paper's subjective results is that it is better _relative_ to competing methods, which is possible, but it feels like the differentiated delta is not perceptibly large. My guess is that this actually largely has to do with the way that the simulation works w/ room acoustics.\n* I think there are likely still issue with using GPT to infer audio attributes, but at the same time the authors attempted to address this using manual validation.\n* The whole image conditioning task honestly felt like a distraction to the rest of the paper. I would've just moved it into the appendix entirely and focused much more on the quality of spatial audio generation (text- / azimuth-conditioned) and additional ablation studies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zg3hSPDZEu", "reviewer": "Reviewer_pDcX", "summary": "This paper propose a new dataset and method for stereo audio generation. Proposed method can generate spatial audio from the instruction format of text, image or interactives. The model are proven effective from the sota comparison tabels.", "questions": "1. I would like to see if BEWO-1M could be a better dataset for audio-language pretraining than current datasets, such as WavCaps, on tasks like text-to-audio retrieval or audio captioning? The inferior conclusion  is acceptable due to that this dataset is not designed targeted for those tasks.\n2. In the stereo audio generation benchmark evaluation, arthors also compared their model with AudioLDM2, Make-An-Audio2, etc, the same as mono-channel benchmarks. Are there not any comparable method that also specialize in stereo audio generation?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The sterro audio generation is less studied than general mono channel audio generation but is meaningful for immersive experiences, and is a good research topic.\n2. Proposed dataset BEWO-1M is the first large-scale spatial audio dataset which is labeled through automatic pipelines, the construction pipeline sounds reasonable and I believe the dataset could be helpful for future research of this field.\n3. Proposed method are proven effective on both mono-channel benchmarks and proposed dual-channel benchmark,", "weaknesses": "1. The descriptions of dataset construction pipeline and model framework are not easy to follow enough.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3NlojpHadJ", "reviewer": "Reviewer_VEpT", "summary": "The paper titled \"Both Ears Wide Open: Towards Language-Driven Spatial Audio Generation\" presents a novel approach in the field of audio generation, focusing on the creation of stereo audio that adheres to spatial contexts described by text or images. The authors address the challenges of generating controllable spatial audio by introducing a large-scale dataset, BEWO-1M, and a model named SpatialSonic. This model leverages spatial-aware encoders and azimuth state matrices to generate immersive and directional audio in line with textual and visual prompts. The paper also discusses the subjective and objective evaluations conducted to validate the effectiveness of the proposed method against existing approaches. Overall, I think the task definition of this paper is good, the data collection process and model design are excellent, the paper writing and overall appearance are good, and it deserves a higher score.", "questions": "See [Weaknesses].", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The paper introduces a pioneering dataset, BEWO-1M, which contains a rich collection of audio samples with spatial descriptions, significantly advancing the field of spatial audio generation.\n2. The SpatialSonic model employs a sophisticated design that integrates multimodal inputs (text and images) to produce spatial audio, demonstrating a high degree of control over audio generation.\nWriting Ideas: The paper is well-structured, with a clear problem statement and a logical progression from dataset creation to model development and evaluation, making it easy to follow the authors' line of reasoning.\n3. The authors conduct both subjective and objective evaluations, providing a comprehensive assessment of the model's performance. In particular, the stereo effect in the demo provided in this article is amazing, which will have a wide range of applications in real life.", "weaknesses": "1. The authors \"apply sound activity detection on each audio and randomly crop to 10s segments.\", This method seems a bit random, will it affect the quality of the original audio data?\n2. According to the data collection pipeline in Figure 2, it seems easy to collect audio-text data paired with image data, but why is there so little image data in Table 1?\n3. Taking video as a data source and incorporating it into the process of dataset collection and model design will make this task more promising.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)