Learning Spatially-Aware Language and Audio Embeddings
Authors: Bhavika Suresh Devnani
Skyler Seto
Zakaria Aldeneh
Alessandro Toso
YELENA MENYAYLENKO
Barry-John Theobald
Jonathan Sheaffer
Miguel Sarabia
Venue: NeurIPS 2024 poster (2024)
Keywords: multimodal embeddings
spatial audio
contrastive learning
Primary area: speech_and_audio
TL;DR: We train a model that aligns 3D Spatial Audio with open vocabulary captions.

Abstract: Humans can picture a sound scene given an imprecise natural language description. For example, it is easy to imagine an acoustic environment given a phrase like "the lion roar came from right behind me!". For a machine to have the same degree of comprehension,  the machine must know what a lion is (semantic attribute), what the concept of "behind" is (spatial attribute) and how these pieces of linguistic information align with the semantic and spatial attributes of the sound (what a roar sounds like when its coming from behind). 
State-of-the-art audio foundation models, such as CLAP, which learn to map between audio scenes and natural textual descriptions, are trained on non-spatial audio and text pairs, and hence lack spatial awareness. In contrast, sound event localization and detection models are limited to recognizing sounds from a fixed number of classes, and they localize the source to absolute position (e.g., 0.2m) rather than a position described using natural language (e.g., "next to me"). To address these gaps, we present ELSA (Embeddings for Language and Spatial Audio), a spatially aware-audio and text embedding model trained using multimodal contrastive learning. ELSA supports non-spatial audio, spatial audio, and open vocabulary text captions describing both the spatial and semantic components of sound. To train ELSA: (a) we spatially augment  the audio and captions of three open-source audio datasets totaling 4,738 hours and 890,038 samples of audio comprised from 8,972 simulated spatial configurations, and (b) we design an encoder to capture the semantics of non-spatial audio, and the semantics and spatial attributes of spatial audio using contrastive learning. ELSA is a single model that is competitive with state-of-the-art for both semantic retrieval and 3D source localization.  In particular, ELSA achieves +2.8\% mean audio-to-text and text-to-audio R@1 above the LAION-CLAP baseline, and outperforms by -11.6° mean-absolute-error in 3D source localization over the SeldNET baseline on the TUT Sound Events 2018 benchmark. Moreover, we show that the representation-space of ELSA is structured, enabling swapping of direction of audio via vector arithmetic of two directional text embeddings.

Learning Spatially-Aware Language and Audio Embeddings
Bhavika Devnani !* Skyler Seto? Zakaria Aldeneh² Alessandro Toso
Elena Menyaylenko
Barry-John Theobald Jonathan Sheaffer Miguel Sarabia"
Georgia Institute of Technology Apple
bdevnani3dgatech edu, {sseto, zaldeneh, atoso)Gapple.com
{elenam, bjtheobald, sheaffer, miguelsdcc@apple.com
Abstract
Humans can picture a sound scene given an imprecise natural language description. For example, is easy imagine an acoustic environment given phrase like "the lion roar came from right behind me!". For a machine to have the same degree of comprehension, the machine must know what a lion (semantic attribute), what the concept of "behind" is (spatial attribute) and how these pieces of linguistic information align with the semantic and spatial attributes of the sound (what a roar sounds like when its coming from behind). -ftte-of-the-att audio foundation models, such as CLAP [7, 44], which learn to map between audio scenes and natural textual descriptions, are trained on non-spatial audio and text pairs, and hence lack spatial awareness. contrast, sound event localization and detection models are limited to recognizing sounds from fixed number of classes, and they localize the source to absolute position (e.g., 0.2m) rather than position described using natural language (e.g.. "next to me"). To address these gaps, we present ELSA (Embeddings for Language and Spatial Audio), a spatially aware- audio and text embedding model trained using multimodal contrastive learning. ELSA supports non-spatial audio, spatial audio, and open vocabulary text captions describing both the spatial and semantic components of sound. To train ELSA: (a) we spatially augment the audio and captions of three open-source audio datasets totaling 4.738 hours and 890,038 samples of audio comprised from 8,972 simulated spatial configurations, and (b) we design an encoder to capture the semantics non-spatial audio, and the semanties and spatial attributes of spatial audio using contrastive learning. ELSA is a single model that is competitive with state-of-the art for both semantic retrieval and 3D) source localization. In particular, ELSA achieves +2.8% mean audio-to-text and text-to-audio R@ above the LAION- CLAP [44] baseline, and outperforms by 11.6" mean-absolute--re in 3D source localization over the SeldNET [40] baseline on the TUT Sound Events 2018 benchmark [1]. Moreover, we show that the representation-spant of ELSA is structured, enabling swapping of direction of audio via vector arithmetic of two directional text embeddings.
Introduction
Humans use implicit context when communicating about and comprehending sounds their en- vironment. For instance, the instruction "Pull over if you hear siren from behind you" is easily understood by most humans. However, a machine would need to not only recognize the source the sound, i.e., the siren (a semantic cue), but also interpret the spatial reference implied by "behind"
Work done while Apple.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

relative to its own position (a spatial cue). The machine must then translate these linguistic cues into its understanding of spatial audio accurately identify, locate, and conditionally respond the sound. This degree of alignment between spatial audio and natural language is understudied in prior work.
Audio foundation models (AFMs), such as LAION-CLAP [44], have been used for multiple down- stream applications, such as language guided audio editing [42, 17J, language guided audio and music generation [16, 11, 45J, audio representations for image and text [14, 46], setting a precedent for the wide applicability of audio representations aligned with natural language. However, these models, and similar state-of- oftth-f-t AFMs, such as Pengi [5], LTU [12], and SALMONN [37], cannot wele--cnnnn/nnoo-psttteee Conversely. models such as SELDNet [1], and PILOT [33] are capable of precise spatial attribute classification and regression, but lack capability to generalize to natural language descriptions of spatial and semantic attributes.
capture the spatial artributes as the models are trained only on
audio.
To address these challenges, we introduce ELSA, multimodal foundation model that learns a joint representation space for the spatial attributes and semantics of audio aligned with natural language descriptions of the audio scene. Learning a joint contrastive representation model enables and improves several tasks including: retrieval, multimodal QA, captioning, and generation [44]. Prior work has shown the benefits of learning well-aligned encoders for multimodal tasks within the vision domain [23, 35]. In contrast, mapping between audio and natural language via a large language model, as in [48]. can significantly improve language reasoning tasks, especially for zero-shot generalization of pre-trained models, however yields worse performance in classification and QA tasks when fine- tuned in the language domain [41]. ELSA enables similar spatially spatiall--aware awaaee downstream applications, for instance, one can expand traditional roguage--giieed audio editing to manipulate spatial elements using natural language commands like: "Remove the sound of the plane flying above" or "move the sound of the dog barking from left to right". In this paper we focus, for the first time, on devising and analyzing multimodal task-agnostic representations that capture both the semantics and the spatial attributes of audio aligned with natural language.
Noting lack of paired spatial audio and language data that can enable training spatially aware audio-language models at scale, to train ELSA, we synthesize a spatial audio corpus consisting of 890,038 samples that span variety of acoustic room properties, such as size and reverberation, from audio clips of the AudioSet [9] and Freesound [8] corpora. We also synthesize natural language spatial audio captions to match the spatial audio using a large-language model (LLM) to rephrase the initial captions. We demonstrate that ELSA captures spatial attributes and semantics of audio by identifying a set tasks on which a standard AFM, such as LAION-CLAP, fails. Moreover, we show that ELSA achieves better zero-shot classification of spatial attributes than models trained only for that task. Finally, we show that ELSA maintains the ability to represent non-spatial audio by demonstrating performance competitive with existing ofte--ofttheart art for number of tasks.
Our key contributions are:
We present and release a new synthetic dataset of 4738.55 hours, with 890,038 samples and corresponding spatial captions across 8,972 simulated rooms with accurate parametric labels for the room properties and sound source locations. Additionally, we also record a small spatial real-world dataset verify transfer to the real-world (cf. Section 3).
We provide ELSA, a multimodal spatial audio-language model that jointly performs semantic classification (sound detection, retrieval), spatial localization, and direction of arrival. ELSA consists of an audio encoder paired with text encoder that jointly learns semantic and spatial attributes via contrastive learning (cf. Section 4).
We show that ELSA effectively captures spatial attributes and semantics competitive with baselines. ELSSA improves by 111.6 mean-absolute-erroo on 3D source localization, and by +2.9% on text-to-audio and audio-to-text mAP@ 10 scores. (cf. Section 5).
Further, we show that the representation-ssaaee of ELSA is structured, allowing for transpo- sition of spatial sound direction via addition or subtraction of two spatially descriptive text embeddings. (cf. Section 5.4).

---

Reverberston
Matorials
LIM Prompt
sound coming from e (D-pl
Rowne the following contence: when the
Attibutes
Socatell Air butes Regg regressioon
ELSA
Semmisteed Microphione
Spatial bulos
for
sound
barking
Semantics
Ei Brodee
LLM
terrTh
Spetnl
Ercodwt
Spatial
Spuiial Caption
cound of barking large the left
Bounds
(a) Our spatial audio pipeline uses simulated rooms with different di-
(b) We augment the original cap- tions by adding properties from
(c) We encode the spatially aug- mented captions and audio, and then align the representations using CLIP objective (see Fig. A.F. for full architecture).
mensions, materials, and reverber- ation, and with sources located different spatial locations.
the room simulations and prompt
L.MM to rewrite the sentence.
Figure I: Our pipeline for learning spatial-audio representations aligned with natural language.
2 Related Work
We provide an overview of the architecture choices and corresponding datasets for training and evaluation of models that can capture the semantics and spatial attributes of audio, including AFMs.
Audio-language Approaches Prior works (e.g., CLAP [7], LAION-CLAP [44], MULAN (15)) have extended the image-text contrastive pre-training approach introduced by CLIP [27] to link audio representations to textual descriptions. These models use two encoders, one for audio and another for text, to project the representations from the two modalities into a common embedding space. Once trained, the models enable zero-shot prediction and retrieval capabilities and textual descriptions. Despite their utility, the methods do not capture the spatial attributes of the modeled signals, rather they capture only their semantics. Another line of work (e.g.. Pengi [5], LTU [12], and SALMONN [37J) extends LLMs to enable audio understanding in open-vocabulary settings (c.g., audio captioning and audio question answering). Such models learn audio encoders to provide a prefix token to prompt a frozen pre-trained autoregressive L.M,, which is then used to generate unconstrained text. These prior methods do not explicitly model the spatial attributes of the audio. Zheng et al. [48] introduced BAT, an audio-based LLM that combines binaural spatial sound perception with natural language understanding with accompanying question-and and-answer dataset that enables model training. BAT focuses on enabling LLMs to reason about binaural spatial audio, which depends on the head-related transfer-function. In contrast our focus is on task-agnostic and device-agnostic representation of spatial audio aligned with text.
unseen sounds
Audio-language Datasets
Learning audio-languan languag models requires access datasets that link the two modalities. Clotho [6] and AudioCaps [18] are popular audio captioning datasets for which the textual descriptions were collected by annotating sound event datasets (e.g.. AudioSet [9]. or Freesound [8]) through crowd-sourcing platforms. LAIONNAAdio-65
[44] is large-scale audio- text dataset collected by downloading audio and relevant textual descriptions from publicly available websites. All three datasets focus on the semantic attributes of the audio signal and do not have labels for the spatial attributes. SPATTAL.SUNDOO, SOUNDQA [48] is dataset that consists of simulated binaural audio samples and question-ansee pairs, which used to train BAT [48]. The audio samples were sourced from AudioSet [9], and the question-ansee annseer pairs were paraphrased using GPT-4. In contrast with the our environmental description captions, the text in SPATIALSOUNDQA is geared towards peestion-anddanswee tasks. In addition, the dataset employs a binaural representation of spatial audio, rendering the data incompatible with ELSA. STARSS23 [36] is dataset of real-worle audio annotated with semantic labels for overlapping sound sources, and the equivalent annotations for the spatial attributes. However, a limitation is that the dataset lacks natural language descriptions of the sound scenes, which are required for aligning the spatial attributes with language descriptions.
3 Paired Spatial Audio and Text Datasets
Multimodal contrastive learning approaches, e-g.. CLIP [27] and CLAP [7], use large amounts of mul- timodal data pairs: 413M and 634k for the LAION versions of both models [32, 44]. Training model
3

---

capable of understanding spatial audio as natural language requires a spatial audio dataset annotated with natural language spatial descriptions (e.g.. "a dog barking in the far left corner of room"), To the best of our knowledge, no such dataset is available. Thus, we use spatial augmentation pipeline composed of two steps: simulating spatial audio in synthetie rooms (ef. Section 3.2 and Fig. la), and caption rephrasing using terms that refer to spatial audio attributes (cf. Section 3.2 and Fig. 1b). We use AudioCaps [18], Clotho [6], and Freesound [8] as base datasets for our augmentation pipeline.
The training set ensures at least two spatial augmentations per data point, allowing for the model to see the same audio with at least two different spatial augmentations per epoch. We generate two different sized versions of the evaluation and test sets. The larger version consists, once more, of at least two augmentations per audio sample, whilst the smaller version has no repeated samples and, consequently, is the same size as the original test set. The smaller dataset allows reporting retrieval results on the same sized dataset as the original, as size uniformity is key to consistency in retrieval metrics. The size of the respective datasets is reported in Appendix A.1. For all datasets, we use first-order ambisonics (FOA) as the encoding of spatial audio, which we describe next.
3.1 Spatial Audio Encoding: First Order Ambisonics
Monophonic, non-spatial audio, captures the spectral and temporal nature of sound, which carries a significant portion of context. Spatial audio provides additional context as it contains both spectral- temporal information and directional attributes, characterized by azimuth and elevation (0, Φ) E S?, and distance. Binaural audio, a common spatial audio distribution format, mimics the signal entering the ear canals. Whilst binaural audio may be a natural choice for playback over headphones, it presents challenges for encoding, storing, and processing spatial information due to the presence of head-related funsfer-functions in the signal [2]. To facilitate more processing flexibility, microphone array signals are often encoded as ambisonics [10]. This is accomplished by taking the spherical Fourier transform of the microphone signals and removing their radial component, which is equivalent to representing the spatial signal as a phase-coincident, infinite series in a spherical harmonic basis [30]. In practice, avoid spatial aliasing, this series is truncated at order proportional to the number of microphones in the array, with higher orders corresponding to a higher spatial resolution. Ambisonics are linearly mappable into a variety of audio playback formats, including binaural. First- order ambisonics (FOA) can be recorded using readily available four-channel microphone arrays, and have been shown to carry significant spatial information [50]. As such, we develop our models to ingest FOA signals. We leave generalization to higher orders for future work. It is worthwhile noting that once microphone array signals have been encoded into ambisonics, no a-priori knowledge on the structure of the capturing array is needed in order to perform any downstream spatial processing. Thus, ambisonics are agnostic to both recording and playback devices, making any embeddings derived from them equally generalizable.
3.2 Spatial Augmentation of the Audio and Captions
Like TUT Sounds Events 2018 [1] and BAT [48], we use simulator to spatially augment non- spatial audio. The augmentation pipeline mirrors that of Spatial LibriSpeech [31]. We specify room configurations parameterized by size, shape, and reverberation time, where reverberation time is function of the room structure and materials with characteristic absorption and scattering coefficients. The simulator further allows specification of the placement and direction of the receiver microphones relative to the source of the sound (see Fig. 1a). For each sample we remove leading and trailing silences, and repeat the audio signal to ensure that samples are at least four seconds long before simulation. A randomly chosen room, placement for the microphone, and placement for the static source is then selected. We ensure that the room augmentations do not overlap between the train, evaluation, and test datasets. The rooms vary in size between 13.3m²and and T30 reverberance ranges from 114.5ms to 2671 1.9ss. The full statistics of these synthetic rooms can be found in Appendix A.
their full-band
Our caption augmentation pipeline converts raw numerical values associated with the spatial audio attributes of the room simulator (e.g., distance to the microphone) into natural language descriptors (e.g., "near" or "far"). Our caption augmentation pipeline is shown in Fig. 1b. The full mapping from spatial audio attributes to natural language is given in Appendix A.2.

---

The original caption augmented with the spatial information makes up the input to LLaMA- 13B [38], which is prompted rephrase in the form of spatially augmented caption. The prompt is:
The sound: <original caption> is coming from the <distance> <elevation> <direction> of a <size> <reverb> room. Rephrase as a short English sentence describing the sound and all the details of its source.
This template prompt overcomes challenges like non-Engliih language in the original caption, missing spatial descriptors in the generated caption, and hallucinations that changed the meaning of the caption. We set the inference temperature of the LLM to 0.9 and the maximum tokens 1,024, Appendix A.3 contains examples of the obtained spatial captions. We note that the caption re- re-writes can lead to hallucinations, which is discussed further in Appendix A.4. We leave the quantification and mitigation of hallucinations for future work.
3.3 Spatial World Dataset
Our training data consists of synthetically auggentted audio and captions, so we also recorded a small dataset to verify generalization to real-world data (refer to Sections 5.2 and 5.3 for analysis). Our spatial real-world dataset recorded using Zylia 19 microphone spherical array at 48kHz with a bit-depth of 24-bits per sample. The dataset contains environmental sounds typically found in an apartment. In total, we recorded 70 samples of spatial audio in five rooms. Each spatial audio sample in the dataset was captioned with the semantic content (e.g.. "sound of a vacuum"), and the direction ("left", "right", "front", "back"), distance ("far", "near"), and elevation { ("up", "down", "level"). For privacy, no personally identifiable information was included in the dataset.
ELSA Pretraining for Spatial Audio and Language
Our architecture is derived from LAION-CLAP [44], which is composed of an audio encoder and a text encoder that aligns embeddings for similar samples across modalities whilst maintaining the original representational capabilities of the individual modalities.
4.1 Audio Input Features
The audio encoder must capture both the semantics of the audio (e.g.. "the sound of a fire alarm") and the spatial attributes (c.g.. "the upper right of a reverberant room"). Following LAION-CLAP [7] and BAT [48], we translate the raw audio into the frequency domain. Consider FOA signal represented by tensor, A CTXPX(N+11" where N is the spherical-haree harmonics order, T the number of time frames and F the number of frequency bins. More information on the derivation of A can be found log- mel spectrogram feature can be written as:
in Appendix A.5. The corresponding
MEL(t, v) log (IA(t, f)1 W mel (f.))..)
where Wmel is the corresponding filter, # is the filter index, is time, and f is frequency. As summarized in Table of SALSA [25], both mel-spectrograms and intensity vectors (IVs) effective spatial features for FOAs. We extract the IVs, [tt, f) as follows:
(t, Al,o(t,f) ] I reecttive (4,)) (L 3 Aoo(()))
(t,(t, f) flolt,f) AI,(L,
I Inctive () R (bo(((
A1,((1,))
where An,m are the th and th order and mode of the ambisonics signal corresponding to its omnidirectional (W) and three dipole (Z, X)) components, and (-)* denotes complex conjugation. Physical normalization constants are omitted here for brevity as IVs are scaled to unit-norm [25].
For ELSA to use semantic features from both non-spatial audio and FOAs, during training we use sample from both the spatially- -uggeeeed datasets and the original non-spatial dataset. Since first-order ambisonics has four channels, and non-spatial audio only one, we copy the single-channel non-spatial signal across all channels. Intensity vectors normalize the dipoles by the omni channel, and result in identical IVs for non-spatial audio. We let the model learn this condition. We ablate the effect of using both spatial audio and non-spatial audio Appendix A.6 and find that using both improves semantic retrieval.
5

---

4.2 Audio and Text Encoders
Our architecture is composed of an audio encoder and text encoder. The audio encoder consists of two branches: the semantic audio branch, and the spatial attributes branch. See Appendix A.7 for visualization of the full architecture.
For the semantic audio branch, we use HTSAT [3] since it was found to perform best in the LAION- CLAP evaluation [44]. HTSAT is nansformer-based audio encoder with self-attention blocks to achieve high performance in audio classification tasks. We initialize HTSAT with weights provided by LAION-CLAP", For spatial-audio input, we feed only the mel-spectrogram of the omni channel from the first-order ambisonics encoding. The omni channel does not contain spatial characteristies, so its role is equivalent to single channel, non- non-spatial audio. This branch has 30M parameters.
As far as we are aware, there is no existing established feature encoder for spatial audio. Thus, for our spatial attributes branch we propose two-branched CNN based on the architecture of [31] that was trained on a multi-task regression loss for azimuth, elevation, distance, and third-octave trrer-tt--eeverberant ratio. The branch trained for 100 epochs on Spatial LibriSpeech, which uses FOA spatial audio and has enough samples to train the spatial attributes branch. Further details, along with the full training active and reactive intensity vector features described in Eq. (2). This branch has 486k parameters.
are discussed in Appendix A.8. This branch fed the
The outputs of both the semantic (768-dimensional) and the spatial attributes (192-dimensional) branches are concatenated to form a 960-dimensional embedding. Using a two-layer multi-layer perceptron (MLP), they subsequently projected down to 512-dimensiin 12-dimensional embedding.
For the text branch, we follow the best performing model LAION-CLAP [44], and use RoBERTa- base [22]. RoBERTa is a general purpose bidirectional transformer [39], pretrained on dynamically masked token prediction task, which employs byte-pair encoding [34] for tokenization. We use the same pre-trained model [44] as the starting point'. The text encoder has 125M parameters, and the final embedding has a dimensionality of 712, which also is projected down to 512 by a two-layer MLP, matching the size of the audio encoder output.
4.3 Pretraining Objectives
We learn aligned representations using batched contrastive loss (popularized by CLIP [27]). The loss function rewards the alignment of representations from the same sample but different modalities, and penalizes the alignment of representations from different samples (see Fig. 1c). Our loss (in common with CLIP [27], CLAP [7], and LAION-CLAP [44]) is derived from the InfoNCE loss [26]. now describe. Given a set of embeddings of any modality X NxD where the 7th entry, 2; E R be matched with y E RR., the following InfoNCE sample loss maximizes the similarity between the pair x; and and minimizes the similarity between all other x and y pairs:
ssim (Iz, v) (X, v) = - -loog Σx, EX feim (I), v)
LintoNCE(X, Iz,
where (a, b) = exp(a b/T) is a similarity function with a learnable temperature parameter т. Taking the average across all audio-text pairs in the batch, where entries the 5th position match each other, we arrive at the CLIP loss:
N
N
1
LCLP -2
LInfANCR(2
Z" 22, +
LIatNNCR +(;,,(,=)))
24,=
N
N
2=0
1=0
(4)
faim (f)
faim (2/, 2.r)
Σ
log
+ log
2N
ES= falm (2g. 21{)
Σ)= === fsim (2) 29)
N
Since the rooms to spatially-augment the audio parametric, have accurate labels associated with spatial features of the audio source. We take advantage of these labels by adding three additional spatial regression objectives. We feed the generated 512 dimension audio embedding into
We use HTSAT-fullset- ietiiimaa--a--aaaaeee faap=0..67.cca5 ckpt from http://gtthahccom//AlNNNAAAC We use the weights for roberta-base from: http://llhhpiliiiiiiccooo/i//nne

---

Table Comparison of model capabilities and performance for retrieval of semantic captions from AudioCaps, and 3D sound localization for the REAL component TUT Sound Events 2018. ELSA is the only model that allows both open vocabulary language understanding and spatial localization, and performs comparably against the baselines for both tasks.
SEMANTIC
SPATIAL CAPARILITIES
AUDIOCAPS MAP 101
REAL 3D LOCAL.("M
CAPARILITIES
SeldNET [1]
X Limited vocab. Limited vocab.
X
26,6
PILOT [33]
X
4.2
Spatial Librispeech [3]]
X
X
12.4
LAION-CLAP [44]
Open vocab. Open vocub.
43.8
95.29
ELSA (ours)
44.2
14.97
three 2-layer MLPs of 33k parameters, which respectively regress the direction of arrival (azimuth and elevation) of sound in 3D space, distance the source to the receiver, and room floor area. These objectives, along with the CLIP loss in Eq. (4), define our final loss:
CELSA = CCLI + Ldir + Ldist + Larea:
(5)
where Ldir is the cosine similarity between the predicted and target angles, and Ldist and Carea is the mean-ssuaredd error between the predicted and target distances and room floor area respectively. We ablate the differences between LELSA and LCLIP in Appendix A.6 and find that at a negligible cost (0.4%) to semantic retrieval, we get 15.3% improvement to 3D localization capability and 12.3% improvement in distance estimation when using LELSA.
5 Experiments, Results, and Discussion
We demonstrate that ELSA jointly captures the semantics and spatial attributes of sound with either audio or text inputs by answering the following research questions:
RQ1 Does ELSA capture spatial attributes in spatial audio (Section 5.2)?
RQ2 Does ELSA capture semantic information both text and audio (Section 5.3)?
RQ3 Does ELSA transfer our real-world dataset? (Sections 5.2 and 5.3)?
RQ4 Does ELSA provide interpretable multimodal representations (Section 5.4)?
RQ5 Are ELSA embeddings capable of driving automatic captioning (Section 5.5)?
5.1 Training and Evaluation of ELSA
Table 2: Zero-shot classification accuracy using the cosine similarity between test set audio em- beddings and templated probe caption embed- dings. The template is sound coming from <spatial attribute>" and value for <spatial attribute> substituted into the template repre- senting the desired class (e.g., "near" or "far" for distance). A classification is correct the attribute in the closest test sample matches the attribute in the template. We cannot provide comparisons with baselines since this is task.
As indicated in Section 4.2, weights for the semantic audio encoder, the spa- tial attributes encoder, and the text encoder. All components of the model are fine-tuned, which corresponds 158M trainable parameters, increase of 0.86% over LAION-CLAPES.
pretrained
For our best model, we train for 40 epochs on 12 nodes, each with 8 NVIDIA 110 GPUs and 96 CPU cores with batch size of 2,304. Training converges within 17 hours. We use the Adam optimizer with learning rate of 5 x 10 and cosine scheduling. We select the checkpoint with the lowest mAP@ 10 retrieval the spa- tially augmented captions.
TASK
S-CLOTHO
S-AC
S-RWD
Distance (2-class)
96.0%
92.9%
67.1%
Direction (4-class)
92.0%
35.8%
Elevation (2-class)
100.0%
100.0%
72.1%
5.2 Spatial Attributes Evaluation
Room area (2-class)
76.6%
74.7%
N/A
Reverberation (2-class)
100.0%
83.3%
N/A
We show that ELSA captures the spatial at- tributes of sound (RQ1) by carrying out downstream regression and zero-shot spatial prompt classification. For regression to 3D sound localization, we train two-layer MLP with 32,768
7

---

Table 3: Semantic retrieval (R@1, I, R@5, and R@ 10) for CLAP and ELSA calculated over the original (non-spatial) versions of Clotho and AudioCaps. Although ELSA is trained using a mixture of non- spatial and spatial audio, it conserves the retrieval performance on non-spatial audio of LAION-AI CLAP, which was trained on only non-spatial data. For the training data, read C' as Clotho, AC as AudioCaps, LA as LAION- LAION-Audio-6300 ppatially-augmente equivalent dataset. We use Freesound, subset of LAION-Audio-ON its more permissive licensing. For a fair comparison, we train a version of CLAP locally with Clotho, Audiocaps and Freesound, which is not reported in the CLAP paper.
and FS as Freesound. A superscript $ denotes the
due to
AUDIOCAPS
CLOTHO TEXI-TO-AUD10 AUDIO-TO-TEXT
TEXI-TO-AUDI
AUDIO-TO- TIXT
MODEL
TRAIN DATA
R@1
R@5 R@ RØI RØ5 RØ10
R@I
R@5 R@10 R@1 R
R@10
CLAP (paper)
C, LA
34.7
70.5
83.2
45.3
79.5
89.2
16.4
39.0
51.0
21.8
44.6
60.1
CLAP (local)
C, FS
32.7
68.8
81.5
40.7
74.0
84.7
14.4
37.6
50.7
18.3
40.5
55.1
ELSA
C,AC, PS, C AAC FS
33.2
68.2
81.0
40.9
74.4
86.1
15.0
36.7
50.8
20.1
43,2
55.4
parameters using the ELSA audio embeddings generated from the training set. We then evaluate on the REAL component of the TUT Sound Events 2018 dataset [1]. Table confirms that CLAP cannot encode spatial attributes (95.29%), whereas ELSA achieves mean-aboaaa error (MAE) and maintains higher mAP@ 10 for semantic retrieval tasks than CLAP. Appendix A.9 shows that there error across various spatial attributes. However, we note
is little variability in the direction-of-aite the errors tend to be higher at the extrema of the dimensions. When compared to methods designed explicitly for 3D sound localization, ELSA performs better than SeldNET (the baseline included with TUT Sound Events 2018) (018)bB))) by +11.6%, and that achieves only -2.6 MAE compared to the model in Spatial LibriSpeech'. f ELSA does not reach the performance of PILOT (4.3)). but this model was specificalll- only for 3D sound localization on data derived from TUT Sound Events 2018 [33].
To verify that the spatial attributes are aligned with language, we create new captions using the tem- plate: "A sound coming from <spatial attribute>" >", where <spatial attribute> can be distance, direction, elevation, room size, and reverberation. For instance, a caption for distance might be "A sound coming from far away". The ELSA text embeddings for such captions are extracted from the pre-trained encoder and compared in a zero-shot fashion with ELSA audio embeddings for samples from the test set using cosine similarity. We classify the match correct if the spatial attribute in the closest audio sample matches the spatial attribute of the query caption, and we report accuracy in Table 2. ELSA achieves >90% correct retrieval for most spatial attributes. For room area, ELSA achieves 74.7% correct retrieval, which we hypothesize is due to the relatively small perceptual differences between small (<50m²) and large rooms (> 1>100m²). We observe a transfer gap on retrieval scores when evaluating on our spatial real-world dataset, with ELSA achieving 67.1% 1 (distance), 35.8% (direction) and 72.1 % (elevation) correct retrieval. Part of this performance difference is because the spatial attributes were only estimates by the annotators during data capture. On the other hand, performance using pre-trained MION-CLAD is close to random for all tasks (Appendix A.),), which is expected as CLAP was not trained with spatial audio or captions.
5.3 Semantics Evaluation
Following LLAION-CLAP [44], we calculate retrieval results when finding matches from audio-to-text and text-to-audio. To compute retrieval, we encode the test set for each modality, and for every sample we check whether the corresponding sample in the other modality has the closest cosine distance (R @1@ 1), is within the five closest samples (R @ 5), or within the ten closest samples (R @ 10). The results in Table show that in addition to learning representations of spatial captions and spatial audio, ELSA also performs on par with LAION-
on non-spatial tasks. Table A.T.8 in Appendix A. shows the retrieval results when using patially-augmentee versions of AudioCaps and Clotho. We remark that adding Freesound to the training set decreases the retrieval scores in Spatial AudioCaps, but improves retrieval scores in Clotho, due to Clotho being a infferently-capttinn of Freesound. We note that the spatial retrieval performance of ELSA is lower than the non- non-spatial retrieval performance (for instance, -9.4% and -13.3% on audio-to-text and text-to-audio R@ 10
subset
"The Spatial I LibriSpeech model can be considered supervised version of ELSAS contrastive learning, since the authors also train on synthetically augmented dataset.
8

---

AudioCaps). This reflects the fact that spatial captions are harder to match, since there is a larger number of attributes and since there are hard-negatives (same semantics, different spatial attributes). Still, ELSA achieves the highest retrieval scores on the spatial real-world dataset (Table A.T.9 in Appendix A. A.1 showcasing ability to transfer the real-world without fine-tuning. Note that we cannot provide comparisons with prior models since using these spatial augmentations is new task.
5.4 Interpreting the representation structure of ELSA
To confirm that directional characteristics in ELSA spatial caption embeddings are encoded in the same feature space as those in ELSA spatial audio embeddings, we train a direction re- gressor with two-layer MLP using the spatial audio embeddings in the training split. We subsequently regress the spatial text embeddings to azimuth values using our trained regres- sor and affix direction labels ("left", "right", "front", "back") to each sample based on the az- imuth values. We obtain 64.3% accuracy on the four-class problem, indicating alignment be- tween the encoding of the modalities. Similarly, we obtain an accuracy 76.5% for when classifying over distance labels ("far", "near") and 55.1% 1% over elevation labels ("up", "down"),
Besides using regression to confirm that ELSA embeddings capture spatial direction, we verify whether the ELSA embeddings can be clustered by spatial attributes. Fig. shows UMAP projec- tion of the ELSA embeddings from the test sets of Ceatial-AudioCapp and Spatial-Clond
Note that the UMAP projection was guided with the embed- dings and labels of the training sets of both datasets, The figure shows the embeddings cluster well with direction labels, though there some degree confusion between "back" and "front". This is roborated by the analysis in Appendix A.13, where compute Wassertein distances directly in the 512-dimensional space. We carried out a similar analysis for spatial distance and found the embed- dings cluster clearly between "near" and "far".
Left
gight
Front
Audio
Caption
Figure 2: UMAP projection of ELSA embed- dings the test splits of Spatial-Clotho and Spatial-AudiCCaps.
We validate that ELSA audio embeddings capture implicit spatial attributes that are latent the text- encoder by first training a classifier using the spatial audio in our training data, where the classes are broad directions, such above and below. We use LLaMA-13B [38] to generate descriptions of sounds that would typically come from each of these directions, e.g.. "the rhythmic drumming of raindrops skylight" (for above) and "the faint creaking of an old house settling" (for below). Ap- pendix A. lists all generated captions. Finally, we use the classifier trained on audio samples to classify ELSA embeddings for these generated captions with implicit directionality. We find that the classifier can correctly identify 68% of the sounds typically heard from above as being from above, showing that the latent space of the text encoder for ELSA is capturing directionality.
Filled markers are ob- tained from spatial audio, and hollow markers are obtained from spatial captions. The UMAP projection was fitted with the train splits of Spatial-Clotho and Spatial-Audio caps, and made use of supervised dimension reduction to highlight the direction differences rather than the semantic differences in the embeddings.
Lastly, show that with a simple text caption. We first obtain ELSA prototypes for four directions ("left", "right", "front", "back") with the template: "A sound coming from the direction". Next, train 4-class direction classifier with a two-layer MLP using the spatial audio in the training splits of our spatially-augmentte datasets. To swap the direction of the sound, subtract the text prototype of the original direction and add prototype for the new direction. For evaluation, we swap the spatial direction of every sample in our spatially-augmentee 4-class direction classifier (96.7% the audio embeddings). Our results show that 99.7% of the swapped samples are classified correctly with the new spatial direction, which highlights the strong alignment of spatial features across modalities, resulting in the ability to edit spatial attributes of
swap the spatial direction encoded by ELSA audio embedding
test that correctly classified by the

---

existing spatial audio using text in embedding space. Further details about this experiment are described in Appendix A.14. These results also point exciting avenues wherein text can condition the manipulation and generation of spatial characteristics of audio. We leave this application for future work.
5.5 Spatial Audio Caption Generation
Decoding multimodal embeddings into natural lan- guage can be achieved by prefixing an autoregressive causal language model [24, 13, 19, 4] where the prefix constructed from a projection of the multimodal em- beddings. To facilitate audio captioning using ELSA, we fine-tune a GPT-2 model [28] with 12 attention layers each having 12 heads (with 163M parameters). The ELSA embeddings are projected onto the prefix using single dense layer (393k parameters). With the ELSA encoder frozen, we train the GPT-2 model on 150k audioo embedding and caption pairs from Spatial-Clothh and Ceatial-Audioaappe We report caption generation metrics in Table 4 and show three generation samples in Appendix A.15. Overall, we find that automatic spatial audio captioning systems are viable though more work is needed to increase the vocabulary size of the generations.
Table 4: Evaluation of Spatial Audio Cap- tion Generation. Metrics were obtained from the Audio Captioning task of the DCASE Challenge" by comparing the gen- erated captions produced from spatial au- dio and the ground-truth captions from the test splits of Apatial-AudoCaaps (S-AC) and Spatial-Clothe
METRIC
RANGE
S-CLOTHO
S-AC
SPIDEr [21]
[0, 5.5]
0.19
0.34
FENSE [49]
[-1.0,+] +1.0]
0.59
0.68
# Unique words
10. o0)
1103
1258
Conclusions, Limitations, and Further Work
We have presented ELSA, an AFM that aligns representations of spatial audio and equivalent text descriptions. To train such representations we built a pipeline to spatially augmented the audio in existing non-spatial audio-text datasets, such as Clotho [6] and AudioCaps [18], and added spatial information to their respective captions. Our results show that ELSA embeddings capture both the semantic contents and the spatial attributes of the audio, with ELSA achieving +2.8% higher scores in audio-to-text and text-to-audio retrieval scores than the sfate-of-the-aat, and obtaining 22.6*MAE in direction of arrival error with respect to an equivalent baseline. Interestingly, by mixing spatial and non-spatial audio and caption pairs, EI .SA is able to represent non-spatial audio as well, Finally, we show that the representation space of ELSA is structured in that the directionality of spatial audio sample can be transposed by simple addition or subtraction of two text representations. Future work will explore acoustic scenarios with overlapping sound sources and sound sources that are moving in the scene. ELSA will also benefit from advances in spatial attributes encoders. In this work, we used the augmented spatial captions as is, but further work should ensure consistency with the semantics before and after augmentation, which will further improve the representational power of ELSA.
Perceiving spatial audio a fundamental aspect of human nature. As linking perception with language. Using spatial audio and a contrastive multimodal training approach, ELSA bridges the gap between feature rich spatial audio and language, paving the way for more intuitive and effective interactions by allowing for richer understanding of the users' environment and generation of immersive sound scenes from natural language.
Broader Impact Our research has the potential to be used in creation of immersive augmented or virtual reality environments. not controlled well, these immersive experiences have the potential to become addictive, and thus impact the mental health of individuals even society whole. Another danger is possibility of creating deepfakes of soundscapes, thus making it possible for generated 3D environments to sound very realistic. The prolifcration of deepfake soundscapes could lead to misinformation and manipulation, undermining trust in audio media.
Available at http://gjiiik.oommmaaaetaaacictt
10

---

Acknowledgements
The authors would like to thank Nicholas Apostoloff, Masha Fedzechkina, Rin Metcalf, Russ Webb, Megan Maher Welsh, and Luca Zappella for their insightful input and discussions on earlier versions of this paper. Moreover, we are thankful to Denise Hui and David Koski for technical support. Names are in alphabetical order by last name within group.
References
[1] Sharath Adavanne, Archontis Politis, Joonas Nikunen, and Tuomas Virtanen. "Sound Event Localization and Detection of Overlapping Sources Using Convolutional Recurrent Neural Networks". In: Journal of Selected Topics in Signal Processing 13.1 (2019).
[2] Jens Blauert. "The Technology of Binaural Listening" Springer, 2013.
[3] Ke Chen, Xingjian Du, Bilei Zhu, Zejum Ma, Taylor Berg- Kirkpatrick, and Shlomo Dubnov. "HTSSAT:: A Hierarchical Token-semantic Audio Transformer for Sound Classification and Detection". In: International Conference on Acoustics, Speech and Signal Processing. IEEE.
2022, pp. 646-650.
[4] Soham Deshmukh, Benjamin Elizalde, Dimitra Emmanouilidou, Bhiksha Raj. Rita Singh, and Huaming Wang. "Training Audio Captioning Models without Audio" In: International Conference on Acoustics, Speech and Signal Processing. IEEE. 2024, pp. 371-375.
[5] Soham Deshmukh, Benjamin Elizalde, Rita Singh, and Huaming Wang. "Pengi: An Audio Language Model for Audio Tasks". In: Advances in Neural Information Processing Systems. Vol. 36. 2023, pp. 18090-18118
[6] Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. "Clotho: An Audio Captioning Dataset". In: International Conference on Acoustics, Speech and Signal Processing. IEEE. 2020, pp. 736-740.
[7] Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. "CLAP: Learning Audio Concepts from Natural Language Supervision", In: International Conference on Acoustics, Speech and Signal Processing. IEEE. 2023, pp. 1-5.
[8] Frederic Font, Gerard Roma, and Xavier Serra. "Freesound Technical Demo". In: International Conference on Multimedia. ACM. 2013, pp. 411-412.
[9] Jort Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, Channing Moore, Manoj Plakal, and Marvin Ritter. "Audio Set: An Ontology and Human-labeled Dataset for Audio Events". " In: International Conference on Acoustics, Speech and Signal Processing. IEEE. 2017, pp. 776-780.
[10] Michael Gerzon. "Periphone (with Height Sound Reproduction)". In: Journal of the Audio Engineering Society MO7 (1972).
[11] Deepanway Ghosal, Navonil Majumder, Ambuj Mehrish, and Soujanya Poria. Text-to-Audio
Generation using Instruction- Tuned LLM and Latent Diffusion Model, arXiv: 2304. 13731 [eess.A) AS].
[12] Yuan Gong, Hongyin Luo, Alexander H Liu, Leonid Karlinsky, and James Glass. "Listen, Think, and Understand".. In: International Conference on Learning Representations. 2024.
[13] Sophia Gu, Christopher Clark, and Aniruddha Kembhavi. "I can't believe there's no images!: Learning Visual Tasks Using Only Language Supervision". In: International Conference on Computer Vision. IEEE. 2023, pp. 2672-2683.
[14] Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel. " "AudioCLIP: Extending CLIP to Image, Text and Audio". In: International Conference on Acoustics, Speech and Signal Processing. IEEE. 2022, pp. 976-980.
[15] Qingqing Huang, Aren Jansen, Joonseok Lee, Ravi Ganti, Judith Yue Li, and Daniel P W Ellis. "MuLan: A Joint Embedding of Music Audio and Natural Language". . In: International Society for Music Information Retrieval Conference, ISMIR. 2022, pp. 559-566.
[16] Rongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. "Make-An-Audio: Text-To-Audin Generation with Frompt-Enhance Diffusion Models".. In: International Conference on Machine Learning. PMLR, 2023, pp. 13916-13932.
11

---

[17] Xilin Jiang, Cong Han, Yinghao Aaron Li, and Nima Mesgarani. Listen. Chat, and Edit: Text-Guided Soundscape Modification for Enhanced Auditory Experience. 2024, arXiv: 2402.
03710 [eess. AS].
[18] Chris Dongjoo Kim, Byeongchang Kim, Hyummin Lee, and Gunhee Kim. "Audiocaps: Gener- ating Captions for Audios in the Wild". In: Conference of the North American Chapter of the Association for Computationnl Linguistics: Human Language Technologies. Association for Computational Linguisties. 2019, pp. 119-132.
[19] Minkyu Kim, Kim Sung-Bin and Tae-Hyun Oh. "Prefix Tuning for Automated Audio Caption- ing". In: International Conference on Acoustics, Speech and Signal Processing. IEEE. 2023, PP. 1-5.
[20] Rosanne Liu, Joel Lehman, Piero Molino, Felipe Petroski Such, Eric Frank, Alex Sergeev, and Jason Yosinski. " "An intriguing failing of convolutional neural networks and the coordcony solution". In: Advances in Neural information Processing Systems 31 (2018).
[21] Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama, and Kevin Murphy. "Improved Image Captioning via Policy Gradient optimization of SPIDEr". In: International Conference on Computer Vision. IEEE. 2017. pp. 873-881.
[22] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zetteeoyer, and Veselin Stoyanov. RoBERTa: A Robustly Optimized BERT Pretraining Approach. 2019. arxiv: 1907. 11692 [cs. .L.].
[23] Jack Merullo, Louis Castricato, Carsten Eickhoff, and Ellie Pavlick. "Linearly Mapping from Image to Text Space". In: International Conference on Learning Representations. 2022.
[24]
Ron Mokady, Amir Hertz, and Amit H. Bermano. ClipCap: CLIP Prefix Image Captioning. 2021. Xrv: 21 11. 09734 [ca.CC CV].
[25] Thi Ngoe Tho Nguyen, Karn N. Watcharasupat, Ngoc Khanh Nguyen, Douglas L. Jones, and Woon-Seng Gan. "SALSA: Spatial Cue- Cug-Aueeet Sog-Spectrogram Features for Polyphonic Sound Event Localization and Detection". In: Transactions Audio, Speech, and Language Processing 30 (2022), pp. 1749-1762.
[26] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Contrastive Predictive Coding. 2018 arXiv: 1807. 03748 [cs. LG].
[27] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. "Learning Transferable Visual Models from Natural Language Supervision". In: International Conference on Machine Learning. PMLR. 2021, pp.
[28] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Lan- guage Models are Unsupervised Multitask Learners. 2019.
[29] Boaz. Rafaely. "Analyssi and Design of Spherical Microphone Arrays". In: Transactions on Speech and Audio Processing 13.1 (2004), pp. 135-143.
[30]
Boaz Rafaely. "Fundamentals of Spherical Array Processing". Springer, 2019.
[31] Miguel Sarabia, Elena Menyaylenko, Alessandro Toso, Skyler Seto, Zakaria Aldench, Shadi Pirhosseinloo. Luca Zappella, Barry-John Theobald, Nicholas Apostoloff, and Jonathan Sheaf- fer. "Spatial LibriSpeech: An Augmented Dataset for Spatial Audio Learning". In: Interspeech. ISCA, 2023, pp. 3724-3722
[32] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. "LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs". In: Data Centric AI NeurIPS Workshop. 2021.
[33] Christopher Schymura, Benedikt Bönninghoff, Tsubasa Ochiai, Mare Delcroix, Keisuke Kinoshita, Tomohiro Nakatani, Shoko Araki, and Dorothea Kolossa. "PILOT: Introducing Transformers for Probabilistic Sound Event Localization". In: Interspeech. ISCA, 2021.
[34] Rico Sennrich, Barry Haddow, and Alexandra Birch. "Neural Machine Translation of Rare Words with Subword Units". In: Annual Meeting of the Association for Computational Lin- guistics. 2016, pp. 1715-1725.
[35] Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. "How Much Can CLIP Benefit Vision-and-Languaae Tasks?" In: International Conference on Learning Representations. 2022.
12

---

[36] Kazuki Shimada, Archontis Politis, Parthasaarathy Sudarsanam, Daniel A Krause, Kengo Uchida, Sharath Adavanne. Aapo Hakala, Yuichiro Koyama, Naoya Takahashi, Shusuke Takahashi, et al. "STARSS23: An Audio visual Dataset of Spatial Recordings of Real Scenes with Spatiotemporal Annotations of Sound Events". In: Advances in Neural Information Processing Systems Datasets and Benchmarks Track. Vol. 36. 2024, pp. 72931-7292
[37] Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. "SALMONN: Towards Generic Hearing Abilities for Large Language Models". In: International Conference on Learning Representations. 2024.
[38] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and Efficient Foundation Language Models. 2023, arXiv: 2302. 13971
[39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. "Attention Is All You Need". In: Advances in Neural Information Processing Systems. 2017, pp. 6000-6010.
[40] Qing Wang et al. The NERC-SLIP System for Sound Event Localization and Detection of DCASE2023 Challenge. Tech, rep. DCASE2023 Challenge, 2023.
[41] Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, and Colin Raffel. "What Language Model Architecture and Pretraining Objective Works Best for Zero-shot Generalization?" In: International Conference on Machine Learning. PMLR. 2022, pp.
[42] Yuancheng Wang, Zeqian Ju, Xu Tan, Lei He, Zhizheng Wu, Jiang Bian, and Sheng Zhao. "AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models", In: Advances in Neural Information Processing Systems 36 (2023), pp. 71340-71357.
[43] Earl G Williams. "Fourier Acoustics: Sound Radiation and Nearfield Acoustical Holography".
Academic Press, 1999.
[44] Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Bong-Kirkpatrick, and Shlomo Dub- nov. "Large-scale Contrastive Language-audio Pretraining with Feature Fusion and Keyword- to-Caption Augmentation". In: International Conference on Acoustics, Speech and Signal Processing. IEEF. 2023, pp. 1-5.
[45] Donghhao Yang. Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu. "Diffsound: Discrete Diffusion Model for Text-to-sound Generation". In: Transactions on Audio, Speech, and Language Processing 31 (2023), pp. 1720-1733.
[46] Guy Yariv, Itai Gat, Lior Wolf, Yossi Adi, and Idan Schwartz. AudioToken: Adaptation of Text- ConCiiondt Diffusion Models for Audio-to-Image Generation. 2023. arxiv: 2305. 13050 [CS.SD].
[47] Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsich. "Large Batch Optimization for Deep Learning: Training BERT in 76 minutes". In: International Conference on Learning Representations. 2020.
[48] Zhisheng Zheng, Puyuan Peng, Ziyang Ma, Xie Chen, Eunsol Choi, and David Harwath. "BAT: Learning to Reason about Spatial Sounds with Large Language Models". In: International Conference on Machine Learning. PMLR. 2024, pp. 61454-61469.
[49] Zelin Zhou, Zhiling Zhang, Xuenan Xu, Zeyu Xie, Mengyue Wu, and Kenny Q. Zhu. "Can Audio Captions Be Evaluated With Image Caption Metrics?" In: International Conference on Acoustics, Speech and Signal Processing. IEEE. 2022, pp. 981-985.
[50] Franz Zotter and Mathhis Frank. "Ambisonics: A Practical 3D Audio Theory for Recording, Studio Production, Sound Reinforcement, and Virtual Reality". Springer, 2019.
13

---

A Appendix
A.1
Dataset statisties
Table A.T.I presents summary of all the paired audio and text datasets we use for training and evaluation.
Table A.T.I: Audio-caption dataset descriptions. The first three rows correspond to the original publicly available datasets, and the subsequent rows correspond to our For each spatially augmented dataset, there are at least two spatial augmentations per original sample the train split.
variants.
DATASET
SPATIAL AUDIO
SPI ITS
NUM. SAMPLES
DURATION
CAPTION DESCRIPTION
Clotho
train, test
3,839
23.99
captions per audio 1-2 cuptions audio
AudioCaps
trinn val. lest
49,274
136.87
FreeSound
train, val. test
414127
2,528.
captions per aution, keyword tags spatially augmented captions per audio 1-2 spatially augimnuted captions per audio spatially augmented captions per audio
Spatial-Clatho
Synthetic Syuthetic
train, val, test
8,546
55.0
Syttial-AndiCapp
train, val. test
98,459
258.
Pputil-PreeSound
Synthetic
783,033
4.425.53
Sputial RWD
Recurded
lest
70
0.25
1-2 human annotated spatial captions per audio
For the augmenteed versions of Clotho, AudioCaps, and Freesound, we use 8,972 parametric rooms with the statistic described in Table A.T.2. Note that the parametric rooms in the test subset of the rooms the training set, however the sources locations on those rooms do not overlap.
Table A.T.2: Spatial attributes of room simulations used to spatially-augmenter audio and language datasets
TRAIN VALIDATION
TEST
Number of simulated moms
8,952
4,970
Source arimuth
180000 +180,0%]
[-180.0% +180.0
Source elevation
[47.5%, +4R.79)
[-29.8 +42,49]
Sturce distance
[0.5m, 4.0m]
[0.9m, 4.0m)
Room floor area
[13.3m 277 Am"
[14.3m?, 277.4m
Full hand T30
[144.5ms, 1,91]
[167.8ms, 1254.8ms]
A.2 Mapping of spatial attributes to natural language
As part of the spatial-augmennatiion pipeline (described in Section 3.2), the mappings in Table A.T.3 to convert spatial attributes to natural language.
Table A.T.3: Mapping between spatial features and natural language descriptors
SPATIAL FEATURE
RANOE
BOUNDS
LANGUADE DESCRIPTOR
Distance
[0m, 5m]
near
> m
far
I-55" -125P1
left
Direation
[-180"
[+55" +125")
[-35", +35"]
front
right
[-145%, +45")
back
Elevation
1.88.1 +48.7"
>-00.0
40.0%
up
down
1000ms
highly reverberant acousticully domponed
Reverberation
[144,5ms,, 2671.9mm)
200ms
Floor
[13.3m P, 277.4mm)
small 50m2
100m
large
14

---

A.3 Audio dataset captions
We report the mapping from raw spatial values spatial captions Appendix A.2. We select these bounds based on how an audio would be percieved by human ears, sound higher than 40 degrees elevation sounds like its coming from a height.
We also present a few random samples of regular to spatial text rewrites by the L.M::
Inputs to LLM
Original caption A bird is loudly making a lot of noises.
Distance far
Room Size medium
Rewritten caption
In medium-sized room, bird is emitting loud sounds from distant location.
Inputs to LLM
Original caption bunch people are talking in the background while man talks to another.
Room size medium
Rewritten caption
medium-sized room, the sound of multiple people conversing in the background can be heard, alongside a man speaking to another individual.
Inputs to LLM
Original caption motor vehicle is running with speed and stopped its engine.
Distance far
Reverb highly reverberant
Rewritten caption
The sound of motor vehicle running high speed and then abruptly stopping its engine is emanating from the far end of highly reverberant room.
Inputs to LLM
Original caption fire crackles in the background while faint knocking fades away over time.
Direction front Distance far
Room size small
Rewritten caption
The sound of fire crackling and a faint knocking can be heard emanating from the far end of small room.
Inputs to LLM
Original caption distant voice and chirping while the wind blows strongly.
Direction right
Distance near
Room size large
15

---

Rewritten caption
large room, sound distant voice and chirping is emanating from the right side, and it is accompanied by the strong wind blowing nearby.
A.4
Hallucinations in LLMs
We note that the caption re-writes can lead to hallucinations in captions. For instance, the phrase "the purr of carerra" describing the sound of the engine of Porsche Carerra rephrased as "the purring cat named Carerra". Another example, "tthe sounds of papers turning" with direction below basement", It is not clear what is the overall effect of the hallucinations: as in the first exam- ple changes the semantics of the audio, but in the second example adds plausible and welcome diversity to our caption set. As mentioned in the main section, we leave the quantification and mitigation of sub optimal hallucinations for future work.
rewritten "tte sound of someone shuffling cards in the
A.5 Further background on first-order ambisonics
Consider a continuum of plane-waves impinging on the surface of a sphere, p(kr; Q), where p is acoustic pressure, k = 2x fc 1 is the spatial frequency, is radial distance and Ω = (0, Φ) E g direction terms elevation and azimuth ø. The expansion of this function in a spherical harmonics basis, Prom (k) can be written as,
f a(k, 2)(,," (2)) (()))
Prom (k, r) (kr(k)
JnES?
(6)
bn (lmm) (A:):
Here, a(k, Q) denotes the plane-wer wave density function in the spatial domain, Y, (S2) are the spherical- harmonics basis function for order n and mode m, and b, (kr) is the radial function given for a rigid sphere by [43].
I' (kra) h, (kr()) hr, (Kro)
b (kr) = 4Ti² jn (nrr
where jn (kr) and h, (kr) are the spherical Bessel and Hankel function, respectively, and (.) denotes their first derivative with respect to the argument. In this work we employ real-valued spherical harmonics basis and radial functions corresponding to rigid sphere. We denote Anm (k) the spher- ical Fourier transform of the plane-wave density function, a(k, 22), and refer to it as an ambisonics signal. We further denote the inverse spherical Fourier transform of the ambisonics signal as,
7
al ②2 = Σ
Anm (K)Y," (S2).
(8)
n=0 10
Considering now microphone array with Q sensors, the integral in (6) becomes weighted finite summation. order to avoid spatial aliasing, we conform to Q = (N + 1)2 with optimal spatial sampling of Q [29] sensors, where N is the spherical harmonics order. Accordingly the outer in (8) is truncated at order N, and the transformation between the pressure p(kr, Q2) and ambisonics function Anm (k) is approximated by,
pkrr ②)
Σ (kr)
Σ
Anm (I)))"())))
rewriting (9) in matrix form and solving for Aums the linear encoding the microphone signals into ambisonies becomes,
anm = Y diag(bbb.. where Y is a Q (N 1)² matrix of spherical harmonics.
(10)
YO
y
Y (21)
Yo
Y (02)
Y (l2)
(11)
(2(22)
y ((2)
(22))
16

---

bn is the radial function vector, and dependency on the spatial frequency & is omitted for brevity. As in this paper we consider signals be outputs of short-time Fourier transform, further denote our ambisonics features as A E CT More specifically, for = 1 and given time frame t and frequency bin kc/ (2xt) features contained in the following ambisonics channels:
A0,0(1,
AI,
A(,
(12)
A1o(t,
[t,
these correspond to the W (ommidirectional) and Y, 2, X (dipole) components of the first order ambisonics approximation. is worthwhile noting that once microphone array signals have been encoded into ambisonics, no a-priori knowledge on the structure of the capturing array is needed in order to perform any downstream spatial processing. Thus, ambisonics are effectively agnostic to both recording and playback devices, making any embeddings derived from them equally generalizable.
A.6 Ablations across model architectures
We ablate over using static intensity vectors in place of the learned encoder, just the learned encoder without spatial regressors, and compare them with our current architecture. Results are shown in Table A.T.4.
Table A.T.4: Comparison of semantic and spatial retrieval performance across data input ablations. mAP@ 10 refers to the text-to-audio mean average precision @ 10 and audio-to-text mean average precision @ 10. is summary metric capturing the model's semantic retrieval capabilities.
SPATIAL (AUDIOCAPS CLOTHO)
MODEL
LOCAL.
(cm)
MAP@
Static Intensity Vectors
27.86
60.2
23.43
Pretnained Spatial Branch & CLIP Loss
27.4
54.31
24.93
Pretrained Spatial Branch & Spatial Losses
23.2
47.71
24.81
Table A. A.T:::
The retrieval metrics here demonstrate why mixing both mono and spatial audio is required to achieve the best possible retrieval performance. We that the model needs to both the regular and spatially augmented versions of the audio and captions to achieve best performance.
AUDIOCAPS
CLOTHO
TEXT-TO-AUDIO
AUDIO
TEXT-TO-AUDIO
MODEL
AUDIO
CAPTIONS
R@1
RØ5
RØ 10
R@1
RØ5
R6
RØ1
RØ5
R@10
RØ1
RØ5
R@10
CLAP
Non spatiaatial
Non Son-spatial
32.7
68.8
81.5
40.7
74.0
84.7
14.4
37.6
50.7
18.3
40.5
55.1
ELSA
Sparial
Non-spatial
27.1
76.1
32.6
44.4
28.1
ELSA
Spatial
Spatial
25.3
59.3
72.5
34.8
64,5
75.2
9,9
31.0
39.8
12.1
35.3
47.3
ELSA
Mixed
Mixed
33.2
68.2
81.0
40.9
74.4
86.1
15.0
36.7
50.8
20.1
43.2
55.4
A.7
ELSA architecture
The full architecture of the ELSA model shown in Fig. A.E.I.
A.8 Spatial attributes branch of audio encoder details
The full architecture for the spatial attributes branch the audio encoder shown below Fig. A.F.2.
The spatial attributes branch has 485,828 parameters, and was praitrained with learning rate of 10 on the LAMB optimizer [47] with weight decay factor of 0.01 and without scheduling the learning rate. The batch size was 1024 and the model was trained for 100 epochs on single node with 8 NVIDIA V100 GPUs and 80CPUs. Training took 12h 20min. The training set was composed of 134,712 10-second segments from the first-order ambisonics samples of the Spatial LibriSpeech train set. We used a multi-task regression loss, Lpres to jointly learn the azimuth, elevation, distance, room volume, and 20-third octave bins (between 100Hz and 8kHz) for direct-to-reverberant ratio and T30.
17

---

2-layer MLP
S de
FOA
Spatial
M032
Attributes
Ropoat Chamat
Encoder
Concatenate
2-layer MLP
2-layer MLP
Non Spatial Audio
Omni Chamme
Epe Mammoo
HTSAT
2- 2-layer MLP
L crea
Captions
Inso T7
2-layer MLP
RoBERTa
CLIP
ELSA
Figure A.F.1: Full architecture diagram for ELSA. Filled blocks include trainable parameters.
Convolutional Blocks
Bot-NormtD
NacPodiD
Canat
Batchoomdd
ELU
FOA
3-layer MLP
L
pre
Easoo No mb0
MesPestip
Bate Bate-hamaa ham-hammo
Comrob
Fallin
05 133200
EL
Convolutional Blocks
Figure A.F.2: Architecture diagram for Spatial Attributes Branch. Filled blocks include trainable parameters. The AddCoords2D block described in [20].
Cpre is the sum of the cosine loss for azimuth and elevation and the mean squared eтror for all other predictions.
A.9 Fine-grained of direction-of-arrrivll error analysis
We analyze the errors of a two-layer MLP trained to regress the direction-of-arriva. (same setting as the last column in Table 1). We observe how the errors vary along the following dimensions: source azimuth, source elevation, source distance, room floor area, room mean T30, and TUT Sound Events 2018 semantic classes. Results are rendered as boxplots in Fig. A.F.3 below.
Table A.T.6 further report mean, standard deviation, and number of samples per bin about the direction-of-arivv errors across the previously analysed dimensions.
A.10 Spatial attributes retrieval for LAION-CLAP
Section 5.2 showed that ELSA embeddings can be classified in a zero-shot fashion by using templated probe caption. Table A.T.7 shows the same experiment applied to LAION-CLAP (with ELSSA results retained for context) by feeding LAION-CLAP datasets. As expected the performance of LAION-CLAP in this setting is close to random, for instance LAION-CLAP achieves 48% accuracy on the two class distance classification task and 28.2% on the four-class direction classification task.
omni-channel of the spatial
18

---

3.09
3.09
abo
2.36
236
3.85
8
020
1.24
100 mp
(00-))
2.24
100
000
100000
1000
4000
0:31
0.31
-315
2.88
222
1000
000
022
126
1.59
3.4
48.5
.2.9
10.0
0.29
0.52
0.8
(a) Azimuth (rad)
(h) Elevation (rad)
3.09
2.78
pop
Absetute wwwettooooaaaatie.. emor
2.47
2.16
600
2.83
8
2.55
2.55
0000
0.91
4000
080
2.93
10000
800
200
0.2
3.00
14:00
40.00
M6000
92.95
11.50
142.8
169,00
19577
221.70
24745
233,6
122
3.22
(c) Distance (m)
(d) Floor area (m² ?
2.78
50 00
Abeshute wfd-drrraaaeer
3.76
2.36
2.55
2.10
0.03
2.65
0:31
5.00
Cherthoatt
Dewrsaa
Grawer
Koybwaar
Kayoddap
Laugeste
Preetam
100.44
458.
122.5
1007.0
1200.00
149
1937.64
207330
(e) Mean T30 (ms)
TUT Sound Events 2018 Semantic classes
Figure A.F3: Boxplots of absolute direction-of arrival errors predicted by 2-layer MLP. Figs. (a)-(e) show the Spatial Audiocaps and Spatial Clotho test sets errors by different categories. Fig. (f) shows the predictions of the test set of TUT Sounds 2018 by different semantic classes. For all figures, boxes represent the interquartile range, solid orange lines the median, and dashed green lines the mean.
19

---

Table A.T.6: Mean and standard deviation of absolute direction-of arrival errors (in radians) predicted by 2-layer MLP. Tables (a)-c) show the Spatial Audiocaps and Spatial Clotho test sets errors by different dimensions. Table shows the predictions of the of TUT Sounds 2018 by different semantic classes
(a) DOA error by azimuth
(b) error by elevation
AZIMUTH RANGES (rad)
ERR. MEAN
DOA ERR.
BIN SAMPLES
BLEVATION RANGES (rad)
DOA ERK. MEAN
DOA ERR.
STD.
STD. DEV
SAMPLES
-3113 2.51)
00.69, -0.55)
0.26
2.51, -1 -188)
0.29
0.46
251
-0.55, - 00.42)
0.15
0.11
20
-1.88. -1.25)
0.43
258
0..2.
0.25
-1.25,-0.60
0.24
0.37
245
-0.29.
0.24
0,28
312
-0.63, +0.00)
0.27
0.31
215
-0.15, -0.02)
0.26
0.35
660
+0.00, +0.63)
0,23
0.25
224
0.002 +0.11)
0.25
0.38
735
40.63, +1.20)
0.23
0.24
270
+0.11, +0.26)
0.24
0.33
456
+1.26, +1.88)
+0.25, +0.38)
+ 1.88, +2.51)
0.22
0.22
259
40.38, +0.51)
0.18
0.16
35
+2.51, +3.14)
0.24
0.29
244
(40.51, +0.05)
0.24
0.13
(c) DOA error by distance
(d) DOA error by room floor area
DISTANCE (m)
DOA ERR.
DOA ERR.
# BIN
FLOOR AREA RANGES (m²)
DOA ERR.
DOA ERR.
BIN
0.533, 0.88)
0.16
0.12
100
[14.10, 40.05)
0.27
0.36
820
0.88...223 1.23)
0.19
0.26
252
10.05, 66.00)
0.24
0.31
1..23,1.58 1.58)
0.21
0.26
340
66.00,, 91.95)
0.23
0.33
346
1.92)
91.95. 117.90)
[1.92, 2.27)
0.26
0.39
314
117.90, 143.85)
0.25
0.40
69
[2.27, 2.62)
0.28
0.38
287
143.55, 109.80)
0.38
0.65
2.62, 2.97)
0.26
0.29
268
169.80, 195.75)
0.28
0.32
2.97, 3.32)
0.27
0.33
208
195. 75, 221.70)
0.19
0.10
3.32,
203
247
(.666,.001
0.34
0.51
164
(247.65, 273.60)
0.15
0.05
(f) DOA by TUT Sound Events 2018 semantic
class
(e) DOA error by T30
DOA ERR. MEAN
DOA ERK.
# BIN SAMPLES
SEMANTIC CLASS
T30 RANGES (ms)
MEAN
STD. DEV
SAMPLES
Drawer
0.12
0.16
185.44, 404.49)
0.23
D.35
585
Laughter
0.16
0.22
95
404.49, 623.54)
0.25
0.34
1030
Cough
1061.64, 1280.69)
623.54, 842.59)
842.59, 1061 641)
0.29
0.27
0.22
0.35
0.35
0.15
$21
78
Clearthratt
Keyboard
Speech
0.14
0.16
0.17
0.17
0.15
0.29
105
115
1280.69, 1499.74
1499.74, 1718.79)
0.44
0.26
Phone
0.24
0.30
117
1718.79, 1937.84) 8-1)
0.30
0.19
Pageturn
0.12
0.15
115
2156..89, 2376.95)
1937.84, 2156.89)
0.20
0,18
0.15
0.18
Doorslam
Keysdrop
Knock
0.15
0.15
0.21
0.17
0.17
111
112
Table A.T.7: Complete version of Table with LAION-CLAP results. LAION-CLAY results are obtained by passing omni channel of spatial dataset through pre-trained model.
ELSA
LAION-CLAP
TASK
S-CLOTTO
S-AC
S-RWD
S-CLOTHO
S-AC
S-RWD
Distance (2-class)
96.0%
92.9%
67.1%
48.0%
54.3%
53.0%
Direction (4-class)
92.0%
92.8%
35.8%
28.2%
29.3%
27.3%
Elevation (2 (2-class)
100.0%
100.0%
72.1%
56.7%
51.3%
59.4%.
Room area (2-class)
76.6%
74.7%
N/A
46.3%
66.5%
N/A
Reverberation (2-class)
100.0%
83.3%
N/A
$7.3%
$2.5%
N/A
20

---

A.11 Text corpus used when testing ELSA's implicitly learned spatial attributes
As mentioned main text, we validate that ELSA's audio embeddings can capture implicit spatial attributes, which are latent in the text-encoder. We use LLaMA-13B [38] to generate descriptions of 50 sounds that typically come from each of above and below. The sentences do not necessarily include explicit spatial descriptions (e.g., "A fire alarm going off", a sound typically from above but not explicitly stated). The original prompt and generated sentences can be found below, We then train two-class classifier (top vs bottom) using the spatial audio in the train sets of our spatial-augmentt datasets, and classify each of the sentences. Results show an above-random classification accuracy of 68.75% for above sentences, 58.82% for below sentences. This experiment shows that, degree, the text-encoder (RoBERTa) able to leverage its semantic bias of placement of objects in the real world and encode it as spatial features that the spatial encoder understands. This task is made quite hard by the observation that our simulated dataset does not reflect the natural world (an acroplane sound could be simulated from below) and the fact that MoBERTa-base has significantly fewer parameters and a smaller training set more recent large- uage-language models such as LLaMA-133 13BB. This experiment is purely qualitative, and leave more in-depth exploration of knowledge sharing between the pretrained spatial and text encoders to future work.
Sentences for sounds from above
upper part sound aeroplane flying.
The sound of a bird chirping.
The sound of helicopter in the sky.
low rumble diatant thunder.
sharp
lightning striking.
The pitter-patter of rain on a roof.
The rhythmic shoosh of vind through trees.
distant siren wailing high the air.
nuffled
footstepa
upstairs.
child's laughter echoing from an upatairs roon.
playful neow or bark from a pet overhead.
The frantic buzzing of trapped fly.
The soft hum of ceiling fan rotating.
rbythmic beeping of
The distant hooting of an exl.
The rustling of leaves as squirrel scurries across roof,
The muffled boon of fireworka exploding high above.
melodic ringing church balls.
fighter jet screaning across heavens.
The rbythmic thump basketball bouncing overhead.
The scraping of furniture being moved on floor abore.
The high-pitched shine of nosquito circling your head.
skylight. loud thump followed by startled yelp (soneone tripped upataira).
rbythmic drumming raindrops
The rbythmic tap-tap-tap of a woodpecker on a tree trunk.
The frantic buzzing of swarn of bees overhead.
nournful ary of seagull circling above the beach.
bouncing The melodic einging of a bird outside your vindow.
rhythmic
reof.
The gentle whoosh of hot air balloon floating overbead.
The rhythmic Shuxp-thunp-e thunp of helicopter blades.
Someane valking
root.
soft cooing of pigeons perched on building ladge.
The gentle pitter-patter of rain on tent reof.
The high-pitched shrick of child on roller coaster.
The rhythmic chirping of crickets
field night.
muffled thump heavy object being dropped
The rhythmic whir of helicopter hovering nearby.
The faint nelody of lullaby drifting doen stsirvell.
sudden gust wind whietling through the trees.
rbythmic clicking of computer from upstairs.
ringing
svaying
21

---

The rhythmic pounding of rain metal roof.
rhythmic chirping
waking you up
The muffled conversation of people valking on floor above.
The nuffled anoring of soneoue aleeping upstairs.
The chirping and equasking of flock of birds taking flight.
rbythmic click-clack tap dancing
above.
rhythmic tapping woodpacker searching
Sentences for sounds from below
sound undarground aubsay train.
walking on pavement.
The muffled boon of distant explosion.
The rhythmic dripping of leaky pipe in the basement.
The gurgling of water in drainpipe.
nuffled thump something heavy being dropped dovnstairs.
The low hun refrigerator running.
The rbythmic squeak of floorboards underfoot.
The muffled clinking of glasses from beloz.
faint laughter echaing from dovnstairs.
rhythmic drumming
dryer
The rhythmic beeping of malfunctioning appliance in the bagement.
The scratching sound of a pet exploring under furniture.
The scurzying mice in the floorboarde.
you forgot nuffled roar furnace kicking on.
rhythmic dripping
off completely.
The rhythmic tick-tock of grandfather clock.
The faint vibration of subuoofer from domssaair stereo.
The muffled chatter of people
below.
kayboard The muffled thump door closing downstaire.
rhythmic click-clack
The rhythmic pinging of a piuball nachine in baeenent arcade.
The dripping sound melting ice from refrigerator.
rhythmic whoosh of basenent exhaust fan.
electrical wiring
The rhythmic druming of rain on a basement windor.
The low rumble of distant traffic filtering through the floorboards.
The faint creaking
old house settling.
rhythmic thump bouncing
The nuffled clinking of coina jar breaking on floor.
The rhythmic shoosh of a vacoun cleaner dounstaire.
The raint strains of music seeping from basement party.
The rhythmic beoping of smoke alarm the basement (hopefully false alarm).
children playing domesairr.
The rbythmic tapping of a sever pipe being repaired.
The rbythmic dripping of condensation on cold vater pipe.
The rhythmic click desdbolt lock being secured domnstairs.
muffled
rhythmic scratching pet trying dig
carpet.
The mutfled clinking of stlversare baing droppad in the sink.
The rhythmic thump of basketball bouncing the floor belav.
The rhythmic ping-pong
table tennis match in progress dounstairs.
vashing The muffled snoring of someone sleeping dounstairs.
rhythmic whirring
agitating
The rbythmic druming of rain on basenent windospane at night.
The faint, high-pitched whine of moaquito buzzing around your ankles.
nuffled clatter pote pane being around
rhythmic
sprinkler system vatering
22

---

A.12 Semantic retrieval for spatial data
We report ELSA's semantic retrieval on Spatial-Clothe and Spatial-Audiccas in Table A.T.8. Likewise, Table A.T.9 shows the retrieval scores of ELSA on our spatial real-world dataset.
Table A.T.: Semantic Retrieval Metries calculated over spatially augmented version of Clotho and AudioCaps eval sets identical in size as the non-spatial sets.
PPATAA--EDIOCAPS
SEATIAL-CLOTHO
TU-IIIOO
TEXT-O-AUDIS
MODEL
TRAIN DATA
R@1
RØ5 RØ10 Rel RØ5 Re10
RØ1
RØs R
RØ1
RØ5
R@10
ELSA
Sp(Clotho + AC)
25.3 58.9
73.5
32.6
61.5
73.0
9.4
25.9
38.0
10.5
27.8
40.2
Sp(Clotho FS)
59.42
30.72
32.0
Table A.T.9: Semantic Retrieval Metrics calculated over our spatial real-world dataset.
SPATIAL-RWD
TEXT AUDIOO AUDIO-TO TEXT
ELSA
Sp(Clotho AC)
18.6
54.3
75.7
25.7
55.7 71.4 67.14 80.0
ELSA
Sp(Clotho AC
41.4
68.6
88.6
35.7
A.13 Further analysis on embedding clusters
In Section 5.4 we analysed the UMAP projection of the ELSA embeddings the of Spatial AudioCaps and Spatial Clotho. Table A.T.10 (a) shows the Wasserstein distances computed directly in the 512-dimensional space, where see the data clusters by direction with lower Wasserstein distances between "front" and "back". Similarly, Fig. A.F.4 and Table A.1.10 (b) show the ELSA embeddings can be clustered according to spatial distance characteristics.
Table A.T.10: Wasserstein distances of 512-dimensional ELSA embeddings, clustered by either (a)
direction or (b) distance.
(a) Direction clustering distances
(b) Distance clustering distances
LEFT
RIGHT
FRONT
BACK
NEAR
FAR
LEFT
0.00
1.04
0.94
0.98
NEAR
0.00
1.10
RIGHT
1.04
0.00
0.92
0.97
FAR
1.10
0.00
FRONT
BACK
0.98
0.97
0.81
0.00
23

---

Far
Near
Audio
Caption
Figure A.F4: UMAP projection of ELSA embeddings of the test splits of Spatial-Clotho and Spatial- AudioCaps. Filled markers are obtained from spatial audio, and hollow markers are obtained from spatial captions. The UMAP projection was fitted with the train splits of Spatial-Clotho and Spatial- Audio caps, and we made use of supervised dimension reduction to highlight the distance differences rather than the semantic differences in the embeddings.
A.14 Swapping of Spatial Direction Experiments
As already introduced in Section 5.4. Our spatial direction swapping pipeline consists of the following steps:
1. We obtain the ELSA embeddings four directions ("left", "right", "front", "back") with the template: "A sound coming from the direction". These are our direction proto- types.
2. Train a 4-class direction classifier with 2-layer MLP (33k parameters) on the training set of Spatial-AudioCaps and Spatial-Clotho. We obtained a 96.7% classification accuracy of the test set of
and Spatial-Clotho.
3. For every correctly-clan claassiid sample the test sets, we obtain their ELSA embedding, subtract the prototype of the original direction, and add prototype for the new direction.
Additionally, we measure any changes in sound semantics by computing the difference in recall @ 10 between the ELSA audio embedding and the ELSA embedding of the sample's 's non-spatial description.
Detailed results are shown table Table A.T.1I. Overall, we find an average 99.7% of the samples correctly classified with the new direction after transposition, and average change of -0.2% in recall@ 10. These results show ELSA directional attributes can be linearly swapped without affecting the semantics of sound.
Table A.T.II: Direction swapping of ELSA embeddings. See Appendix A. 14 for detailed explana- tion of how swapped the embedding directions. & is the number of test samples misclassified by our direction classifier, and subsequently excluded. N is the number of samples that were used for direction transposition. @@ 10 is the recall@ 10 computed over the corresponding non-spatial captions. is the classification accuracy of the transposed sample. A A@i0 is the change in recall @ 10 after performing the change direction.
New DIRECTION
LEFT
FRONT
RIGHT
BACK
N
R@
10
A Aguio
0
AR100
Agsio
0
A gaio
ORIGINAL DIR.
LEFT
14
156
94.9%
100.0%
+1.3%
100.0%
+0.0%
100.0%
-1.3%
FRONT
13
486
81.1%
100.0%
+1.9%
100.0%
+0.6%
100.0%
.0.4%
RIGHT
12
196
92.3%
100.0%
-0.5%
100.0%
+0.5%
100.0%
-0.5%
BACK
-1.4%
100.0%

---

Furthermore, we wanted to verify what happened if we removed the original direction but did not add back new direction. Table A.T.12 shows this ablation. Interestingly, the classification does not result in random classification accuracy but rather 0% accuracy for all four original directions.
Table A. T.12: Direction removal of ELSA embeddings. See Appendix A. 14 for detailed explanation of how we swapped the embedding directions. & is the number of test samples misclassified by direction classifier, and subsequently excluded. N is the number of samples that were used for direction transposition. @@ 10 is the recall@ 10 computed over the corresponding non-spatial
captions. is the classification accuracy of the transposed sample. ARGI0 is the change recall @ 10 after performing the change of direction.
DIR. REMOVED
Anoi0
LEFT
14
156
94.9%
0.0%
40.6%
FRONT
13
486
81.1%
0.0%
-0.6%
5.11%
BACK 9
564
81.6%
0.0%
-2.3%
A.15 Further details on Spatial Audio Caption Generation
Section 5.5 introduced spatial audio caption generation system. In what follows, we illustrate some of the generations produced by the system as well the corresponding ground-truth annotation. Below that, in Fig. A.F.5, we include architecture diagram for the spatial audio caption generation system.
Generated caption
medium-sized room located the far back, an electric motor is emitting high-pitched whine, accompanied by a whirring noise. In the background, adult male voice can be heard speaking.
Ground-truth caption from test set
From deep within medium-sized room, the noise of robust industrial engine can be heard whirring loudly.
Generated caption
The sound of water flowing and splashing is emanating from the front of a room.
Ground-truth caption from test
The sound of gentle rowing and paddling in the water is emanating from the vicinity of medium- sized room.
Generated caption
The sound of cheering coming from crowd is heard the medium-sized room.
Ground-truth caption from fest set
25

---

The sound of applause, indicating that people are praising the musicians after their performance, is emanating from the medium-sized
FOA
ELSA
2-layer
GPT-2
Autoregrestrv cecocing
Audio Branch
DxS12
Figure A.F.5: Architecture diagram for spatial audio caption generation.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We propose a new dataset, discuss the dataset in Section 3, and demonstrate results of ELSA trained on this dataset in Section 5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of our work in terms of some pitfalls of using synthetically augmented captions in the conclusion Section of the main paper as well Section A.9 of the appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
27

---

Answer:
[NA]
Justification:
The paper does not contain any novel theoretical results.
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Dataset creation details are included in Section including details of prompts, and dataset creation. Hyperparameter details are included in Section 5.1 and the appendix. Implementation details of our architecture are included in Section 4. Furthermore, we will release our code and models aid reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
28

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification: Code, datasets, and models will be made publicly available httpaa//wwppppppmspapiaiiiai aadiio-eldi
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ //nips. wttpp.ccpppliiiiieepeppppppppppp mublic/guidess Suuddes/Codee (CodeSubmissionf
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We highlight training hyperparameters in Section 5.1. Details of the dataset creation are included Section 3.1, and details of our dataset evaluations are included in Sections 5.2-5.4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
The original datasets provide only a single partition for train- ing/testing/evaluutie splits. We base our experiments these splits that results are directly comparable with other state the art methods.
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
29

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
The training dataset took 2 weeks to generate, and utilized 96 CPUs and 16T disk space. Each full training run of the model takes roughly 1600 A100 GPUhs. This was tested multi-node GPU machines with up to 12 nodes. There were total of 40 estimated runs to completion. For the version of our model only trained on the smaller datasets Clotho and AudioCaps, convergence takes 4 hours otherwise takes 17 hours.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https http:::////eurr http////euurpppeeeee
Answer: [Yes]
Justification: For training we use publicly available datasets, for which we augment with only spatial information. The real-world dataset that we captured for our experiments was created without capturing personally identifiable information. Furthermore, the dataset contains sounds of everyday items, c.g., a coffee grinder or paper rustling, and we know of concerns regarding demographics of end-users.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
30

---

10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Positive impact is detailed in the introductionnSeccion 1) and conclu-
sion(Section 6). potential negative impacts are in the broader impacts paragraph (Section
6).
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our societal risks only apply for generation of soundscapes, which is out of the scope for the paper.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
31

---

Justification: All datasets and model architectures are appropriately and explicitly cited. Github links are also included for code used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. www.coo//aoooooo datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: New assets including models and datasets are included in Sections and and plan be released for reproducibility.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not conduct any experiments involving human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
32

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not conduct any experiments involving human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
33


Reviews:
[{"review_id": "Y83hvd8139", "reviewer": "Reviewer_JMmV", "summary": "The paper presents ELSA (EMbeddings for Language and Spatial Audio),  a spatially aware-audio and text embedding model. The training data is created by synthesizing spatial audio in ambisonic format and augmenting text captions with spatial information. A small real world data is also collected for evaluations. The model training itself largely follows standard CLIP/CLAP training by using contrastive losses. Additional losses for direction and distances are added for the spatial part. Evaluations are done on both semantic retrieval tasks and spatial tasks.", "questions": "Please address the weaknesses above", "limitations": "Please add some limitations.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "– The paper addresses a key part of multimodal contrastive embeddings. Sounds contain a significant amount of spatial information and humans naturally rely on directional information from sounds. Considering this it is expected that embeddings with spatial information are created. The paper is a good step in the right direction. \n\n– For the most part, the paper is well done. Spatial audio can present several challenges with respect to data (more so in multimodal settings, training approach). Considering the challenges around learning from spatial audio, the paper presents a good approach for learning spatially-aware language embeddings. The experiments are also reasonably good. \n\n– The paper is also well written and mostly clear.\n\n-------\nScore increased after rebuttal.", "weaknesses": "There are a few weaknesses which are worth addressing in the paper. \n\n– For table 2, I would be curious to see what CLAP on its own can achieve. It would be good to contrast this zero-shot classification on the spatial task. \n\n– How were the non-spatial audio-text pairs used in training (as shown in Table 3, last row) ?\n\n– Using non-spatial audio-text seems crucial for good semantic retrieval. This is evidenced by A.6 as well where the models training on just spatial audio-text pairs do not do well on semantic retrieval task. This is a bit surprising. The CLIP loss is still present in training, the semantics are also intact in spatial audio-text pairs. Why should there be a performance drop in that case ? it would be good to provide a  good discussion and justification\n\n– In Table A.7, the performance of the model trained on spatial Clotho and Audiocaps is better on RWD data than even on Clotho and Audiocaps itself. That is a bit surprising. We would expect that the model would be better in it’s own domain. The difference also is pretty big. \n\n– The discussion in Section 5.4 is a bit adhoc. I would suggest not referring to anecdotal observations. The experiments could be better designed. \n\n– Several of the classification experiments end up using 3-4 layers MLP. I think a more shallower model (maybe even just linear classifier) would provide a better confirmation of what information the embeddings store. Otherwise such deeper networks are able to push the numbers on their and it’s not clear how good the embeddings are. \n\n– Some form of clustering and distance visualization would be good. It has been incorporated in some form in Table 2, but it would be good to explicitly show how the distances between embedding represent the spatial information.  \n\n– All the spatial mapping in terms of the language is very discrete (A.2). The range for distance, direction etc. can appear a bit arbitrary and forced. While this is perhaps a good first attempt, a more continuous form of “spatial-language” is desirable. Another thing could be a perception driven approach can also be taken where the boundaries are decided by what people generally perceive as left or right w.r.t sound direction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your suggestions and comments, which we address here.  Please let us know if anything remains unclear.\n\n***For table 2, I would be curious to see what CLAP on its own can achieve.***\n\nPerformance using a pre-trained CLAP checkpoint is close to random for all tasks, which is expected as CLAP is not trained with spatial audio or captions that describe spatial features of audio.\n\n|                   |    S-Clotho     |  S-Audiocaps  |     S-RWD     |\n| ----------------- | --------------- | ------------- | ------------- |\n| Distance          |       48        |      54.3     |       53      |\n| Direction         |      28.2       |      29.3     |      27.3     |\n| Elevation         |      56.7       |      51.3     |      59.4     |\n| Room Area         |      46.3       |      66.5     |       N/A     |\n| Reverb            |      57.3       |      52.5     |       N/A     |\n\n\n***How were the non-spatial audio-text pairs used in training (as shown in Table 3, last row) ?***\n\nWe merge the spatial and non-spatial datasets, and use all samples from this mixed dataset in each epoch to learn ELSA. Samples are input to **both** the spatial attributes encoder and HTSAT (semantic encoder). For the HTSAT encoder, the non-spatial audio is used as is and we pass just the first channel for the spatial audio.  For the spatial attributes encoder, which expects four-channel FOA, we use the spatial audio as is and we repeat the single channel non-spatial audio four times.  The loss is weighted the same for both forms of input. We will revise Section 4.1 and Figure A.1 for clarity in the camera-ready version.  \n\n***Using non-spatial audio-text seems crucial for good semantic retrieval....***\n\nThis is a great question and something we investigated during the architecture design for ELSA.  Table A.6 evaluates retrieval on _spatial captions_ while Table 3 (main paper) and the table below evaluate retrieval on _semantic captions_. It is worth noting that the task in Table A.6 is harder than Table 3 as the model must retrieve a caption with **both** semantic and spatially correct elements. We will be clearer about this distinction in the camera-ready version of the paper.\n\n| Model |                     |                        | AC     |        |        |        |        |        | Clotho |        |        |        |        |        |\n|-------|---------------------|------------------------|--------|--------|--------|--------|--------|--------|--------|--------|--------|--------|--------|--------|\n|       | Training Data Audio | Training Data Captions |  T->A  |  T->A  |  T->A  |  A->T  |  A->T  |  A->T  |  T->A  |  T->A  |  T->A  |  A->T  |  A->T  |  A->T  |\n|       |                     |                        |   R@1  |   R@5  |  R@10  |   R@1  |  R@5   |  R@10  |   R@1  |   R@5  |  R@10  |   R@1  |   R@5  |  R@10  |\n| CLAP  |      Non-spatial    |       Non-spatial      |  32.7  |  68.8  |  81.5  |  40.7  |   74   |  84.7  |  14.4  |  37.6  |  50.7  |  18.3  |  40.5  |  55.1  |\n| ELSA  |        Spatial      |       Non-spatial      |  27.1  |  62.7  |  76.1  |  36.6  |  68.7  |  78.4  |  11.3  |  32.6  |  44.4  |  12.4  |  28.3  |   50   |\n| ELSA  |        Spatial      |         Spatial        |  25.3  |  59.3  |  72.5  |  34.8  |  64.5  |  75.2  |   9.9  |   31   |  39.8  |  12.1  |  35.3  |  47.3  |\n| ELSA  |         Mixed       |          Mixed         |  33.2  |  68.2  |   81   |  40.9  |  74.4  |  86.1  |   15   |  36.7  |  50.8  |  20.1  |  43.2  |  55.4  |\n\nWe remark that changing from non-spatial audio only to spatial audio only results in a drop  of -5.6% and -3.1% in T→A R@1 for Audiocaps and Clotho respectively. Similarly, training only with spatial captions results in further drops of -1.8% and -1.4% in T→A R@1 for Audiocaps and Clotho respectively. Mixing both captions and audio restores the performance to the original levels.\n\n***The discussion in Section 5.4 is a bit adhoc. I would suggest not referring to anecdotal observations.***\n\nApologies, we should not have used the word anecdotal. The experiments are indeed formal.  We will rephrase for the camera-ready version of the paper.\n\nWe have since run experiments for the remaining attributes to understand the alignment between captions and audio (extension of the experiment in paragraph 1, section 5.4) and obtained accuracies of 76.5% for distance and 55.1% for elevation. We have also performed clustering visualizations on the embeddings (Exp. A. in global response). We are happy to incorporate further changes or experiments that would strengthen this section.\n\n***Several of the classification experiments end up using 3-4 layers MLP...***\n\nWe will correct Section 4.3 — all MLPs consist of a two layer MLP with a hidden layer of 64 dimensions and an output layer.  We will add the parameter counts (33,345 parameters or only 0.02% of ELSA).\n\n***Some form of clustering and distance visualization would be good.***\n\nWe address this question in the global response (Exp.B). \n\n***All the spatial mapping in terms of the language is very discrete (A.2). The range for distance, direction etc. can appear a bit arbitrary and forced....***\n\nWe selected our mappings to correspond to general terms commonly used for broad spatial descriptions. We wholeheartedly agree that incorporating more natural spatial language is desirable. The idea of perceptually motivated boundaries is also excellent. Ideally both of these would be derived by running user-studies. Firstly to align with the perception of spatial attributes(eg, the azimuth would a human consider a sound to be coming from the left) , and secondly to understand the distribution of language people use to refer to spatial attributes. These user studies are not something that we would be able to setup and have approved in this rebuttal period, but is something we will strongly consider for future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X44w6mPuTB", "reviewer": "Reviewer_SQBX", "summary": "The paper presents ELSA (Embeddings for Language and Spatial Audio), a novel model designed to learn spatially-aware audio and text embeddings using multimodal contrastive learning. The primary aim is to address the limitations of existing audio foundation models, which lack spatial awareness, and sound event localization and detection models, which are constrained to a fixed number of classes and absolute positional descriptions. The authors spatially augment several classical open-source audio datasets in order to train ELSA.  Results show that ELSA is able to capture spatial attributes and semantic meaning of the audio.", "questions": "The experiments in Table 2 confuse me a lot. In Sec. 5.2, the authors mentioned that \"The ELSA text embeddings for such captions are extracted from the pre-trained encoder and compared in a zero-shot fashion with ELSA audio embeddings for samples from the test set using cosine similarity. We classify the match as correct if the spatial attribute in the closest audio sample matches the spatial attribute of the query caption\". \n  However, the number of classes of a spatial attribute is very limited (For instance, there are only two classes \"far\" and \"near\" for the \"distance\" attribute), which means there are only two captions that will be used for the \"distance\" attribute? Wouldn't there be very few captions being used for testing totally? \n  Hopefully, the authors can explain the experimental configuration a bit more.\n- To train ELSA on single-channel audio, the authors repeat the single channel 4 times to fake a 4-channel FOA audio and compute Intensity Vectors. However, the way IV is calculated possibly doesn't make sense for this kind of faked 4-channel audio. Why is it designed this way? Why not try to design a separate feature extractor for single-channel audio?\n- It is natural to understand computing bearing information from spatial audio, which is essentially a bit similar to calculating the \"Time Difference of Arrival\" based on different channels. But how to understand that the model can get distance information from spatial audio? In other words, where does the information about distance come from?", "limitations": "- As the authors mentioned, for creating a spatial audio caption dataset, using LLM to rewrite the caption might lead to hallucinations.\n- Model performance in real scenarios is yet to be verified.\n- ELSA looks very suitable to be used as a spatial audio encoder for a caption model to conduct spatial audio captioning, but unfortunately, the authors did not show this kind of capability in the paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The focus of this paper is on learning spatial audio embeddings associated with natural language description, which is a very interesting and rewarding problem for which there is a lack of models.\n- These authors synthesize large amounts of non-spatial audio data under various spatial configurations, which is a valuable contribution to the field of spatial audio understanding.", "weaknesses": "- For this paper, my biggest concern is the generalizability of the model to real scenarios. \n  While the synthetic dataset is extensive, there is a risk that the model might not generalize well to real-world scenarios due to potential biases in simulated environments. To show the performance of model generalization to real scenarios, the experiments only on a small real-world dataset appear too thin. Would it be possible to test ELSA in other real scenarios, for example, in some of the tasks in the latest DCase competition, e.g. Sound Event Localization?\n- For paper writing, too much important information is put in appendices, such as the structure figure of the whole model. Perhaps the layout of the writing could be adjusted to make it easier to read.\n- The citation format of the article is highly problematic and needs to be standardized.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to provide comments and suggestions.  We address the points raised below.\n\n***Would it be possible to test ELSA in other real scenarios, for example, in some of the tasks in the latest DCase competition, e.g. Sound Event Localization?***\n\n***Model performance in real scenarios is yet to be verified***\n\nTable 1 reports localization performance on the TUT sounds dataset, which is a dataset of real-world room impulse responses encoded as FOA and convolved with clean recordings.  We use this dataset specifically because it has properties similar to the synthetic samples we have used for model training.  In particular, each sample has a single and stationary point source, which allows us to focus specifically on performance differences due to the sim-to-real gap.  Other datasets, such as STARSS23 (latest dataset in the DCASE - SELD task), contain overlapping and moving sound sources (features we will support in future).\n\nTables 2 and A.6 present the spatial and semantic performance of our model on the spatial-RWD dataset (our real world dataset). We acknowledge that this dataset is small in size. However, it is important to note that its data distribution is significantly different from the training data used for our model. As such, the performance metrics on this dataset offer valuable insights into the model's generalization beyond simulated data.\n\n\n***ELSA looks very suitable to be used as a spatial audio encoder for a caption model to conduct spatial audio captioning .....***\n\nWe are thankful for this suggestion. As mentioned in the global response we trained a prefix encoder to perform spatial audio captioning. Please refer to the global response (Exp. C.) for implementation details and results.\n\n***For paper writing, too much important information is put in appendices, such as the structure figure of the whole model......***\n\nWe apologize for inconvenience due to the paper structure. We will move the model architecture to the main paper in the camera ready version. Please let us know if there are any other suggestions.\n\n***The citation format of the article is highly problematic and needs to be standardized.***\n\nApologies.  We have standardized the references and ensured everything is consistent for the camera-ready version of the paper. \n\n***The experiments in Table 2 confuse me a lot. In Sec. 5.2,  ... . However, the number of classes of a spatial attribute is very limited (For instance, there are only two classes \"far\" and \"near\" for the \"distance\" attribute), which means there are only two captions that will be used for the \"distance\" attribute? Wouldn't there be very few captions being used for testing totally? Hopefully, the authors can explain the experimental configuration a bit more.***\n\nThis task measures accuracy and alignment of spatial attributes encoded in the audio and text embeddings. For each spatial attribute, we have a finite number of values (e.g., `left`, `right`, `front`, and `back` for the attribute `direction`;  the full list of attribute/value pairs detailed in Table A.3 of the appendix). We then generate a text caption for each spatial attribute (e.g., for direction, it would be “Sound from the **left**”, “Sound from the **right**”,  and so on). Finally, the class assigned to a test sample is the class of the text embedding with highest cosine similarity to the audio embedding. This evaluation is conducted over the entire test set. You are correct that there are a limited number of classes per spatial attribute for this task. However, we also report open vocabulary spatial caption retrieval performance in Table A.6 of the appendix. This table reports retrieval accuracy over a much larger (~1000 sample) set of complex spatial captions (eg \"Sound of an alarm coming from the far left of a large room.\") for Spatial-Clotho and Spatial-AudioCaps.\n\n***To train ELSA on single-channel audio, the authors repeat the single channel 4 times to fake a 4-channel FOA audio and compute Intensity Vectors. However, the way IV is calculated possibly doesn't make sense for this kind of faked 4-channel audio. Why is it designed this way? Why not try to design a separate feature extractor for single-channel audio?***\n\nThis is a good question. The duplicated channels result in identical intensity vectors for the x, y, and z dimensions. The spatial attributes branch learns to associate this condition with lack of spatial bias. We tried alternative methods: using zeros for the other three channels, using random values for the other three channels, as well as selectively using the spatial attributes branch only for spatial audio samples. Empirically, our current design produced the best results. We will update the manuscript with this information.\n\n***It is natural to understand computing bearing information from spatial audio, which is essentially a bit similar to calculating the \"Time Difference of Arrival\" based on different channels. But how to understand that the model can get distance information from spatial audio?....***\n\nDistance is primarily encoded in the DRR (direct-to-reverberant ratio) of the soundfield, as well as the absolute level of the sound. Since our data synthesis is based off physical simulation of soundfields, these biases are inherently present in the resulting training examples, and may be learned by both branches of the audio encoder.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "elmNx1php2", "reviewer": "Reviewer_USXb", "summary": "This paper describes a method for learning to represent spatial audio (and text).  The proposed model is trained on synthetically spatialized audio data with corresponding text prompts.  The authors evaluate the system on audio captioning/retrieval and localization tasks, showing that the proposed model effectively represents both semantics and locality of audio.", "questions": "- How was the spatialization implemented?  I expect this was done via standard methods (ISM implemented by pyroomacoustics? something else?), but there is no mention of this in either the main text or the appendix.  Additionally, I think some of the details from the appendix (Table A.2) should be mentioned directly in the main text, such as the range of room sizes, angle distributions, etc.; these details are important, and do not take much space.  (If you do need to sacrifice anything, I don't think the definition of log-mel transformation is as critical to include since it is standard.)\n- Since TUT2018 is a finite vocabulary dataset, it would be informative (and entirely possible) to see a per-class and per-environment breakdown of the evaluations reported in table 1.  This would be informative because it's not necessarily a given that your spatialization is equally effective across categories (or rooms / room sizes).  If the model does turn out to perform consistently across categories - great!  If not, it may suggest a weakness in either the spatial rendering or prompt generation.  (If you do compute these results, it may or may not make sense to store in the appendix, depending on how interesting the results are.)", "limitations": "The limitations sections seems sufficient to me.\n\nOne potential caveat here is that the authors do not explicitly mention any limitations imposed by the accuracy of the spatialization process, e.g., whether it will only work well for simulated closed environments (shoebox model) or if it can accurately capture open environments.  This I think would be easy to resolve with a bit more information about the process and an extra line the limitations (if necessary).", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is well written, clearly organized, and easy to follow.  The proposed method makes intuitive sense and appears to be effective.  The empirical evaluation is reasonably thorough and the choice of tasks and baseline models seem appropriate.  Spatial audio is a growing area (in the context of machine learning / event detection / representation learning), and I think the work described here does fill an unaddressed area of the literature in an interesting way.  Overall I think they did a great job here.", "weaknesses": "I don't have much to fault here, but there are a few points that I think could be expanded to improve clarity and help readers understand the contribution here.  I'll elaborate on these points in the questions section, but the high-level gloss is:\n\n- While the spatial representation part of the work (ie FOA-derived input) is explained well, there is almost no explanation of how the spatialization was implemented.\n- There is little (if any) qualitative analysis of the results, only aggregate scores reported in tables.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time reviewing the paper and providing valuable feedback and suggestions.  We will do our best to clarify the points that you have raised below.  Please let us know if there are further questions.\n\n***While the spatial representation part of the work (ie FOA-derived input) is explained well, there is almost no explanation of how the spatialization was implemented. How was the spatialization implemented? I expect this was done via standard methods (ISM implemented by pyroomacoustics? something else?), but there is no mention of this in either the main text or the appendix. Additionally, I think some of the details from the appendix (Table A.2) should be mentioned directly in the main text, such as the range of room sizes, angle distributions, etc.; these details are important, and do not take much space. (If you do need to sacrifice anything, I don't think the definition of log-mel transformation is as critical to include since it is standard.)***\n\nThank you for the suggestion regarding Table A.2. We will move it over to the main paper in the camera ready version. As per your suggestion we will also include more details on the simulation algorithm. Please let us know if there are any other suggestions as well.\n\nThe augmentation pipeline mirrors that of Spatial LibriSpeech [1], employing a combination of geometrical- and boundary-element acoustical simulation. This allows us to physically model the way that sources radiate sound, as well as its propagation in both enclosed and open spaces.\n\n[1] Sarabia et al. (2023) Spatial LibriSpeech: An Augmented Dataset for Spatial Audio Learning. InterSpeech.\n\n***There is little (if any) qualitative analysis of the results, only aggregate scores reported in tables. + Since TUT2018 is a finite vocabulary dataset, it would be informative (and entirely possible) to see a per-class and per-environment breakdown of the evaluations reported in table 1. This would be informative because it's not necessarily a given that your spatialization is equally effective across categories (or rooms / room sizes). If the model does turn out to perform consistently across categories - great! If not, it may suggest a weakness in either the spatial rendering or prompt generation. (If you do compute these results, it may or may not make sense to store in the appendix, depending on how interesting the results are.)***\n\nThank you for this suggestion. Please refer to the general response and attached PDF for a detailed error analysis (Exp. B.). Briefly, we find little variation across different attributes, though the errors are higher at the extrema of the ranges. We will update the manuscript appendix with this analysis.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z4TqWr7oBA", "reviewer": "Reviewer_rKMo", "summary": "This paper presents an approach to learning spatially aware language representations. The authors propose a contrastive representation model that integrates spatial context into language representations, aiming to enhance the performance of tasks that require spatial reasoning. The model combines visual and textual data to create embeddings that are sensitive to spatial relationships. The contributions include an architecture of the proposed model, extensive experimental results demonstrating improved performance on spatial reasoning benchmarks, and an analysis of the model's ability to generalize across different spatial contexts.", "questions": "See weakness.", "limitations": "The authors have addressed the limitations of their work by discussing the datasets used and the model's computational requirements. Additional limitations can be found in the Weaknesses section.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper makes a strong contribution to spatial reasoning. The integration of spatial context into text-to-audio generation models is an important and underexplored area, and this work offers a novel and effective solution.\n2. The experimental setup is rigorous, with well-designed experiments that effectively validate the model's performance.\n3. The paper is well-written, with clear explanations of the methodology and results. \n4. The findings have significant implications for improving spatial language understanding in various applications.", "weaknesses": "1. The reliance on synthetic datasets may limit the generalizability of the findings. The authors could explore the way to train the model on in-the-wild data.\n2. The current interpretation experiments (Sec. 5.4) only study a four-class classification (\"left,\" \"right,\" \"up,\" \"down\"), which is insufficient for real-life scenarios. For instance, spatial audio applications often require more nuanced classifications, such as distance perception (e.g., strong/weak reverb in indoor/outdoor settings), which are critical for capturing and representing spatial information. The authors should consider extending the experiement to handle a wider range of spatial attributes to enhance its applicability in diverse settings. For example, the authors should consider using prompts like \"xxx is making a sound in the distance\" and \"xxx is making a sound nearby\" to figure out if the results are different.\n3. The paper could benefit from a more detailed error analysis, identifying common failure cases and understanding why the model fails in certain scenarios. This analysis would provide insights for further improvement and refinement of the model.\n4. While the model performs well on tasks like retrieval and source localization, its ability to generalize to spatial text-to-audio generation remains to be seen.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to provide comment and feedback on our submission.  We address the questions and concerns below.  Please let us know if further clarification is required.\n\n***The reliance on synthetic datasets may limit the generalizability of the findings. The authors could explore the way to train the model on in-the-wild data.***\n\nThe scarcity of publicly available, open vocabulary, labeled spatial audio datasets presents a challenge. While modalities like text, image, and video benefit from vast amounts of internet-sourced data, that is not the case for spatial audio. Collecting and annotating high-quality spatial audio data is considerably more complex, requiring specialized equipment (suitable microphones, headphones, and room measuring equipment) and human expertise for accurate spatial perception and labeling. We acknowledge the importance of exploring in-the-wild data and consider it a valuable future step in our research.  \n\n\n***The current interpretation experiments (Sec. 5.4) only study a four-class classification (\"left,\" \"right,\" \"up,\" \"down\"), which is insufficient for real-life scenarios. For instance, spatial audio applications often require more nuanced classifications, such as distance perception (e.g., strong/weak reverb in indoor/outdoor settings), which are critical for capturing and representing spatial information. The authors should consider extending the experiement to handle a wider range of spatial attributes to enhance its applicability in diverse settings. For example, the authors should consider using prompts like \"xxx is making a sound in the distance\" and \"xxx is making a sound nearby\" to figure out if the results are different.***\n\nExperiments in Section 5.4 were designed to better understand the embedding space of the audio and text. The experiment referred to in question (paragraph 1 of Section 5.4) does directional classification using a shallow (one hidden and one output layer) MLP on the audio embeddings, and uses the MLP to then classify corresponding text embeddings. For completeness, we have also run these experiments for **distance** and **elevation**, and obtained accuracies of 76.5% and 55.1% respectively. \n\nWe are a little unsure on how to best address the suggestion about natural prompts (`xxx is making a sound in the distance`....). Table 2 actually presents results for zero-shot classification accuracy using diverse prompts of the suggested form. For every spatial attribute, we take language descriptors (as detailed in Table A.3 of the appendix) and affix it in a prompt (`Sound coming from <>`). We then run classification using cosine similarity of the text and audio embeddings and report the accuracies.   Please let us know if we have misunderstood your question, we would be happy to clarify further or run any additional experiments.\n\n\n***The paper could benefit from a more detailed error analysis, identifying common failure cases and understanding why the model fails in certain scenarios. This analysis would provide insights for further improvement and refinement of the model.***\n\nThank you for raising this issue. As discussed in the global response (Exp. B.), we have carried out this analysis for the direction-of-arrival error according to azimuth, elevation, distance, floor area, mean T30, and TUT Sound Events 2018 semantic classes. Results show little variance across all those conditions, though errors are higher at the extrema of the conditions.\n\n***While the model performs well on tasks like retrieval and source localization, its ability to generalize to spatial text-to-audio generation remains to be seen.***\n\nThis study is an initial investigation into aligned representations of spatial-audio and spatial-captions.  To measure the success of ELSA, we chose perception-based tasks involving retrieval and source localization because these are well-understood and have clear ways of measuring performance objectively. We agree that spatial text-to-audio generation is an important task, however the success of the generation requires either user-studies or the development of new metrics to measure the quality of spatial audio generation.  We are not aware of any established benchmarks for reporting generation quality of _spatial_ audio.  For these reasons we opted for tasks for which we can directly measure performance, and leave generative tasks for future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are pleased to see such a strong positive sentiment from the reviewers about our work.  Reviewers highlighted how **interesting and rewarding** (SQBX), **strong and significant** (rKMo), and how **rigorous** (rKMo) our work is.  Reviewers also mentioned how **well-written** (JMmV, USXb) and **intuitive** (USXb) the proposed approach is. We agree that the research area is **under-addressed** (USXb) and **under-explored** (rKMo), and appreciate that reviewers feel ELSA addresses this gap in a **novel** and **effective** (rKMo) way. We want to thank all reviewers for their time and thoughtful feedback. 

In response to the reviewers' suggestions, we have conducted the following additional experiments that we believe will be of interest to all:

**Exp. A. Visualization of ELSA Embeddings**

Figure R.1 in the rebuttal document shows a UMAP projection of the ELSA embeddings from the test sets of Spatial-AudioCaps and Spatial-Clotho. Note that the UMAP projection was guided with the embeddings and labels of the training sets of both datasets. Additionally, we computed the Wasserstein distances between the 512-dimensional embeddings of both test sets:

|          |   left   |  right   |  front   |   back   |
| -------- | -------- | -------- | -------- | -------- |
| left     |   0.00   |   1.04   |   0.94   |   0.98   |
| right    |   1.04   |   0.00   |   0.92   |   0.97   |
| front    |   0.94   |   0.92   |   0.00   |   0.81   |
| back     |   0.98   |   0.97   |   0.81   |   0.00   |

Overall, both results show that the data clusters well with the direction labels, though there is some degree of confusion between back and front.  We carried out the same analysis for distance and floor area, and obtained similar results. We are happy to share with the reviewers if they are interested.   We will add these results to the appendix of our paper.

**Exp. B. Fine-grained of direction-of-arrival error analysis**

We analyzed the errors of a two-layer MLP trained to regress the direction-of-arrival (same setting as Table 1, last column in the paper).  We observe how the errors vary along the following dimensions: source azimuth, source elevation, source distance, room floor area, room mean T30, and TUT Sound Events 2018 semantic classes. Results are rendered as boxplots in figure R.2 in the attached PDF.

Overall, we find there is little variability in the direction-of-arrival error across the studied dimensions. However, we note the errors tend to be higher at the extrema of the dimensions. Regarding the TUT Sound Events 2018 semantic classes, we again find few differences between the different sound classes, with `drawer` having the lowest error and `phone` having the highest one.

**Exp. C. Spatial Audio-Caption Generation**

Inspired by reviewer SQBX’s suggestion, we trained a Spatial Audio caption generator.  Decoding multimodal embeddings into natural language can be achieved by prefixing an autoregressive causal language model [1-4], where the prefix is constructed from a projection of the multimodal embeddings. To facilitate audio captioning using ELSA, we fine-tune a GPT model with 12 attention layers each having 12 heads. The ELSA embeddings are projected onto the prefix using a single dense layer. With the ELSA encoder frozen, we train the GPT model on the audio embeddings of Spatial AudioCaps, and perform early-stopping to avoid overfitting. The following results are obtained with fine-tuning on only 150K embedding caption pairs from the train splits of Spatial Clotho and Spatial AudioCaps.

We evaluate results on the test splits of both Spatial AudioCaps and Spatial Clotho. We report metrics from Audio Captioning task of the DCASE Challenge [5] (the metric description are taken verbatim from their associated github page).  

| Metric | Short description | Range | Spatial Clotho | Spatial Audio Caps |
| ------ | ----------------- | ----- | -------------- | ------------------ |
| SPIDEr | Mean of CIDEr-D and SPICE | [0, 5.5] | 0.19 | 0.34 |
| FENSE  | Combines SBERT-sim (Cosine-similarity of Sentence-BERT embeddings) and Fluency Error rate (fluency errors in sentences with a pretrained model) | [-1, 1] | 0.59 | 0.68 |
| Vocab  | Number of unique words in candidates. | [0, $\inf$] | 1103 | 1258 |

Additionally, we also present examples of generated samples from both datasets:

(1) Generated caption
> In a medium-sized room located at the far back, an electric motor is emitting a high-pitched whine, accompanied by a whirring noise. In the background, adult male voice can be heard speaking.

(1) Ground truth caption from test set:
> From deep within a medium-sized room, the noise of a robust industrial engine can be heard whirring loudly.

(2) Generated caption:
> The sound of water flowing and splashing is emanating from the front of a room.

(2) Ground truth caption from test set:
> The sound of gentle rowing and paddling in the water is emanating from the vicinity of a medium-sized room.

(3) Generated caption:
> The sound of cheering coming from a crowd is heard near the medium-sized room.

(3) Ground truth caption from test set:
> The sound of applause, indicating that people are praising the musicians after their performance, is emanating from the medium-sized room.


[1] Mokady et al. (2021) ClipCap: CLIP Prefix for Image Captioning. ArXiV preprint.

[2] Gu et al. (2023) I Can't Believe There's No Images! Learning Visual Tasks Using only Language Supervision. CVPR.

[3] Kim et al. (2023) Prefix tuning for automated audio captioning. ICASSP.

[4] Deshmukh et al. (2024) Training audio captioning models without audio. ICASSP.

[5] https://dcase.community/challenge2024/task-automated-audio-captioning

Meta-review: The paper presents ELSA (Embeddings for Language and Spatial Audio), a novel model designed to learn spatially-aware audio and text embeddings using multimodal contrastive learning. The primary aim is to "fuse" existing audio foundation models, which lack spatial awareness, and sound event localization and detection models, which are constrained to a fixed number of classes and absolute positional descriptions. The authors synthesize training data by spatially augmenting several classical open-source audio datasets in order to train ELSA largely following a standard CLIP/CLAP approach with contrastive loss. Additional losses for direction and distances are added for the spatial part. Evaluations are done on both semantic retrieval tasks and spatial tasks. Results show that ELSA is able to capture spatial attributes and semantic meaning of the audio.

Reasons to accept:
– The paper creates for the first time spatially aware audio embeddings in an LLM framework (with natural language descriptions) -- good novelty. Sounds contain a significant amount of spatial information and humans naturally rely on directional information from sounds. The paper is a good step in adding this capability to machine hearing.
– For the most part, the paper is well written and the experiments are well executed. Considering the challenges around learning from spatial audio, the paper presents a good approach for learning spatially-aware language embeddings, even though simulated data may not be ideal.
– The integration of spatial context into text-to-audio generation models is an important and underexplored area, and this work offers a novel and effective approach. The findings have significant implications for improving spatial language understanding in various applications.
- Authors are committed to releasing the relevant model checkpoints, code, and data

Reasons to reject:
- The paper uses only synthetic data, and the generalizability/ performance on "real" data remains to be seen, even though early experiments show that the model has potential. There is also a performance drop in semantic similarity, spatiality on the language side, etc
- While the spatial representation part of the work (ie FOA-derived input) is explained well, there is almost no explanation of how the spatialization was implemented (can probably be addressed in the camera ready) and only four classes (left, right, up, down) are being implemented and tested. 
- A lot of relevant information is in appendices and some issues in the writing (these can be fixed in camera ready)
Decision: Accept (poster)