SING: Spatial Context in Large Language Model for Next-Gen Wearables
Authors: Ayushi Mishra
Yang Bai
Priyadarshan Narayanasamy
Nakul Garg
Nirupam Roy
Venue: ICML 2025 poster (2025)
Keywords: Spatial Speech ASR
Direction of Arrival
Large Language Models
Primary area: applications->language_speech_and_dialog

Abstract: Integrating spatial context into large language models (LLMs) has the potential to revolutionize human-computer interaction, particularly in wearable devices. In this work, we present a novel system architecture that incorporates spatial speech understanding into LLMs, enabling contextually aware and adaptive applications for wearable technologies. Our approach leverages microstructure-based spatial sensing to extract precise Direction of Arrival (DoA) information using a monaural microphone. To address the lack of existing dataset for microstructure-assisted speech recordings, we synthetically create a dataset by using the LibriSpeech dataset. This spatial information is fused with linguistic embeddings from OpenAI’s Whisper model, allowing each modality to learn complementary contextual representations. The fused embeddings are aligned with the input space of LLaMA-3.2 3B model and fine-tuned with lightweight adaptation technique LoRA to optimize for on-device processing. SING supports spatially-aware automatic speech recognition (ASR), achieving a mean error of 25.72°—a substantial improvement compared to the 88.52° median error in existing work—with a word error rate (WER) of 5.3. SING also supports soundscaping, for example, inference how many people were talking and their directions, with up to 5 people and a median DoA error of 16°. Our system demonstrates superior performance in spatial speech understanding while addressing the challenges of power efficiency, privacy, and hardware constraints, paving the way for advanced applications in augmented reality, accessibility, and immersive experiences.

SING: Spatial Context in Large Language Model for Next-Gen Wearables
Ayushi Mishra
Yang Bai
Priyadarshan Narayanasamy'
Nakul Garg'
Nirupam Roy'
Abstract
1. Introduction
Vision. Large language models (LLMs) have created new possibilitiess by enabling intuitive, contextual, and natural in- teractions with machine (Choi & Li,, 2024; Cui et al., 2024; Kumaran et al., 2023; Chen et al., 2024; Alayrac et al., 2022). Introduction of this capability to smart earbuds and other wearables is around the corner (FusionChat, 2025). Spatial understanding of speech in LLMS can open up a frontier for wearable applications, as it enables such ubiquitous devices to process directional cues, recognize speech from multiple sources, and associate spoken content its spatial context. This capability can be cornerstone in significantly enhane- ing user experiences by making devices more responsive, intuitive, and context-aware, particularly in applications like virtual assistants, augmented reality (AR), and acces- sibility tools. By enabling LLMs to reason about spatial acoustic cues, wearables can offer advanced features such as selective voice summarization in meetings (Ma et al., 2025; Kumar & Kabiri, 2022), sound-based navigation for visually impaired individuals (Chen et al., 2020; Gan al., 2020), and immersive XR experiences that react dynamically to the user's surroundings (Parkkola, 2023; Kailas & Tiwari, 2021).
Integrating spatial context into large language models (LLMs) has the potential to revolutionize human-computer interaction, particularly in wear- able devices. In this work, we present a novel sys- tem architecture that incorporates spatial speech understanding into LLMs, enabling contextually aware and adaptive applications for wearable tech- nologies. Our approach leverages microstructure- based spatial sensing to extract precise Direction of Arrival (DoA) information using a monau- ral microphone. To address the lack of existing assisted speech record- ings, we synthetically create dataset by using the LibriSpeech dataset. This spatial information is fused with linguistic embeddings from Ope- nAl's Whisper model, allowing each modality to learn complementary contextual representations. The fused embeddings are aligned with the input space of LLaMA-3:. -322 3B model and fine-tuned with lightweight adaptation technique LoRA to optimize for on-device processing. SING sup- ports spatially-aware automatic speech recogni- tion (ASR), achieving mean error of 25.72% substantial improvement compared to the 8 median error in existing work-with a word error rate (WER) of 5.3. SING also supports sound- scaping, for example, inference how many people were talking and their directions, with up 5 people and median DoA error of 16 Our sys- tem demonstrates superior performance in spatial speech understanding while addressing the chal- lenges of power efficiency, privacy, and hardware constraints, paving the way for advanced appli- cations augmented reality, accessibility, and immersive experiences.
dataset for microstructure-astii assssssst
Spatial context is yet to be explored its full potential with LLMs. BAT (Zheng al., 2024) is one of the first to support spatial understanding for LLM. While inspiring, BAT supports inference only non-verbal audio, such as dog barking or bird chirping, without the context of spoken language. In this paper, we aim to advance the idea of introducing spatial knowledge in LLMs for speech signals and at the same time make it more compatible for wearables and ubiquitous computing devices. This work strive to achieve high accuracy of directional audio sensing using novel monaural setup and extending its capability to spatial speech processing.
Challenges and Our Approach.
While promising, realization of this vision needs to over- come several unique challenges.
Egqual contribution 'Department of Computer Science, Univer- sity of Maryland, College Park, USA. Correspondence to: Nirupam Roy <niruroy @ udu.
A. Spatial cues in wearables. Sensing spatial features of sound, such direction-of-arrival (DoA) source lo- cation, requires sampling the wave in space using array of microphones. However, traditional methods are limited by space-time sampling constraints and requires large and
Proceedings of the 4dnd International Conference Machine Learning. Vancouver, Canada, PMLR 267, 2025. Copyright 2025 by the author(s).

---

SING: Spatial Context in Large Language Model for Next-Gen Wearables
power hungry microphone arrays (Yang et al., 2022; Wang et al., 2023) to achieve resolution of spatial cues essential for understanding acoustic physical contexts. These arrays require significant physical space, making them impractical for any miniature wearables and ubiquitous devices such as microscale Internet of Things (IoT) sensors or button-sized wearables. BAT (Zheng et al., 2024) achieves spatial under- standing using monaural or binaural microphones. While claimed monaural, the mean error rate (MAE) is 88.52", too high for real-world applications.
et al., 2024), employing simple linear layer as a projection matrix, denoted as W, to map the spatial features into the language embedding space of the LLM. Then, the fused embeddings are aligned with the input space of the LLM using lightweight adapter module and fine-tumed on task- specific prompts using low-rank adaptation (LoRA) (Hu et al., 2021). This approach allows the LLM to interpret spatially enriched embeddings and produce outputs that are both contextually and spatially aware, enabling advanced applications in spatial speech understanding and wearable interactions.
In a meeting scenario, an error of 88.52 would result in attributing spoken content to the wrong person sitting across the table, severely degrading the reliability of based summarization features in AR glasses or virtual assistants. In smart home systems, identifying which room or device a user is speaking from (c.g., turning toward a speaker to answer a question) would fail if the estimated direction is off by nearly 90", leading incorrect command execu- tion. Moreover, the physical configuration of the micro- phones for their binaural setup was not mentioned. As solution to minimize the size of the setup for wearables, we break away from the traditional spatio-temporal sampling model and build on pioncering idea (Garg et al., 2021) of
Our key contributions are summarized below:
We present a novel framework that integrates spatial au- dio cues with LLMs, enabling advanced spatial speech understanding for real-world wearable applications.
We design a novel spatial encoder tailored for microstructure- based spatial sensing, enabling precise DoA estimation with minimal hardware. This compact and efficient encoder is specifically optimized for small, wearable devices.
Our framework enables wearable devices to perform ad- vanced tasks such as spatially aware automatic speech recognition (ASR), meeting summarization with spatial context, and immersive audio experiences, showcasing the potential for applications in augmented reality, acces- sibility, and beyond.
micro-structure assisted miniature and low-power acoustic front-end. This setup requires only monaural microphone combined with a tiny microstructure to induce spatial diver- sity in the recorded signal, eliminating the need for multiple microphones while still enabling precise spatial encoding, making it an ultra-compact and low-power solution suitable for wearable applications.
2. Related Work
2.1. Spatial Audio Detection Spatial audio processing involves techniques from both tra- ditional signal processing and modern deep learning ap- proaches to extract spatial information from sound sources. Signal processing methods, such as beamforming (Xu et al., 2017a), generalized cross-correlation (GCC) (Knapp & Carter, 1976), and cigenvector-bart SIC (Schmidt, 1986) and ESPRIT (Roy & Kailath, 1989), utilize phase differences and time delays between micro- phone pairs to estimate the DoA. While effective, these methods often require large microphone arrays and can struggle in reverberant environments. Deep learning mod- els, including convolutional neural networks (CNNs) (Ada- vanne et al., 2018), recurrent neural networks (RNNs) (Xu et al., 2017b), and transformer-baset architectures (Park et al., 2021). have demonstrated success in learning spatial features from raw audio and spectrograms, often achieving superior results in complex scenarios, but microphone arrays are still required. Recent developments in microstructure- based sensing, where monaural microphone is embed- ded in a designed physical structure to create directionally- dependent frequency responses, offer a promising approach for minimalist spatial audio sensing (Garg et al., 2021).
B. Spatial speech context in LLM. To capture critical spatial audio cues such as directionality and reverberation, we leverage the microstructure- based spatial sensing (Garg et al., 2021), which enables compact setup for wearables and higher DoA estimation accuracy. These cues are processed into embeddings using lightweight architecture that only has 16.3 million parameters. The speech encoder utilized in this work is OpenAI's Whisper model (Radford et al., 2023). Its encoder generates features that are well-suited for modeling speech and effectively incorporate contextual information about background noises (Gong et al., 2023).
techniques like MU-
C. Aligning spatial speech context to LLM. To integrate spatial audio understanding into LLMs we propose a two- step alignment framework that bridges the spatial encoder, Whisper's 's speech-to- encoder, and the LLM. First, the spatial encoder processes acoustic signals to extract direc- tional and spatial cues, such as DoA, and generates embed- dings that encapsulate spatial information. Simultancously, Whisper encodes speech signals into rich linguistic repre- sentations, capturing both semantic content and background context. We adopt the approach outlined in L.Lava (Liu
2

---

SING: Spatial Context in Large Language Model for Next-Gen Wearables
This method, inspired by biological systems like owl hear- ing, introduce new opportunities for low-cost, efficient DoA estimation while balancing physical constraints with data- driven learning.
to speech understanding. Our work bridges this gap, empha- sizing low-power, privacy-preserving processing tailored for real-world wearable deployment.
3. Microstructure-Assittr Spatial Encoding
2.2. Multimodal I Large Language Models
Recent advancements in multimodal large language models (LiMs)(Yin et al., 2023; Song et al., 2023; Huang et al., 2024a; Hsieh et al., 2024) have significantly expanded their capabilities across various modalities, including audio, mu- sic, and visual data. AudioGPT (Huang et al., 2024b) inte- grates ChatGPT as a versatile interface for a wide array of audio and speech-related tasks, enabling complex reasoning and synthesis using natural language prompts. For music understanding, another work introduced framework com- bining the MERT music encoder (Li et al., 2023b) with an LLM, achieving state-of-the-art results in tasks such as mu- sic captioning and mood classification. While audio-based LLMs are gaining traction, multimodal LLMs have been more extensively explored in the visual domain (Li et al., 2024; 2023a; Liu et al., 2024; Sun et al., 2024). Several models focus on image understanding by combining LLMs with advanced vision encoders. BI.IP-2 (Li et al., 2023a) employs a pre-trained Vision Transformer (ViT) to extract visual embeddings, which are then aligned with an LLM for tasks such as image captioning, visual question answering (VQA), and reasoning about visual content. LLaVA (Liu et al., 2024) extends this framework with a lightweight vi- sual encoder and cross-moda training strategies, enabling improved performance in open-ended visual reasoning tasks. These advancements underscore the expanding landscape of multimodal LLMs, highlighting the importance of robust cross-modar architectures for diverse real-world applica- tions across audio, music, image, and video tasks.
3.1. Spatial Encoding Microstructure
Microstructure-assisted spatial encoding builds upon the innovative hardware design introduced in Owlet (Garg et al., 2021), which utilizes a compact two too-micoopoone array in- tegrated with a microstructure to achieve high spatial reso- lution. As shown in Figure 1, the microstructure modifies sound wavefronts as they propagate, introducing distinct acoustic patterns that encode angular information.
With minrotructurs Withhaut micrstrcctm
12 man
90
120
80
150*
n
30
Passive structure interacts with sound
Directional signature encoding
Low power miniaturized spatial sensing with acoustic micro-structures
Figure I. The vision and technical overview of Owlet (Garg et al.. 2021), low-power and miniaturized system for introducing spatial information into monaural recording of sound.
The Owlet system achieves spatial diversity in sound sensing with a monaural setup by leveraging the physical principles of diffraction, capillary effects, and structural resonance within a carefully designed microstructure. Diffraction oc- curs as sound waves bend and scatter when interacting with the edges and barriers inside the structure, creating phase shifts and amplitude variations that depend on the sound's 's DoA. Capillary effects arise from the narrow chan- nels within the microstructure, where confined spaces alter sound propagation by introducing pressure gradients and phase delays, further emphasizing directional differences in the received signal. Additionally, structural resonance occurs when specific sound frequencies align with the nat- ural resonance modes of the structure, amplifying certain frequency components while attenuating others based on the sound's 's angle of incidence. By combining these three mech- anisms, the Owlet design introduces directionally dependent frequency responses, enabling precise DoA estimation with minimal hardware complexity, avoiding the need for large microphone arrays while maintaining high spatial resolu- tion.
2.3. Spatial-Aware Large Language Models
LLMs have become increasingly popular and useful for var- ious Al-driven applications (OpenAI, 2023; Touvron et al., 2023; Team, 2023; Anthropic, 2023; MosaicML, 2023). However, the integration of LLMs with physical context, particularly in spatial audio perception, remains sparse. Re- cent papers, such as BAT (Zheng et al., 2024) and "Can Large Language Models Understand Spatial Audio?" (Tang et al., 2024), have explored integrating spatial audio with LLMs, focusing on non-speech sounds and requiring larger microphone arrays. contrast, our work targets spatial speech processing using monaural setup with microstruc- ture sensing, enabling superior angular resolution with a smaller hardware footprint suitable for wearables. The goal for using the spatial speech is to enhance speech clarity and intelligibility by allowing listeners to distinguish multiple speakers based on spatial positioning. SALMONN (Tang et al., 2023) and GAMA (Ghosh et al., 2024) address speech tasks but lack spatial awareness, limiting their applicahility
Monaural Recording Hardware: In its original design, Owlet used two microphones, with an additional one placed outside the microstructure cap. This extra microphone helped reduce the impact of room impulse response on di-
3

---

SING: Spatial Context in Large Language Model for Next-Gen Wearables
rectional cues. Despite utilizing multiple microphones, the system produces monaural recording, where the sound wave is essentially sampled at single spatial location. Un- like binaural recording hardware, which uses two separated microphones to achieve spatial diversity, Owlet's hardware captures multiple recordings using closely placed micro- phones. As elaborated in Section 4, we leveraged the sec- ond channel from the microphone outside the microstructure for speech encoding and the microstructuue-coveer micro- phone channel for DoA encoding.
did a one-time calibration of the impulse response. These responses represent how the Owlet microstructure- based array processes incoming speech signals from different di- rections. To convert the frequency-domain representation into the time domain, an Inverse Fast Fourier Transform (IFFT) (Heckbert, 1995) is applied:
ht(t)= = F (H(
(I)
where (t) is the time-domain impulse response for angle 0. This transformation is performed for all the desired an- gles @ E [1, 360), yielding angle-specific impulse responses. The 360 azimuth angles are defined in a canonical frame relative to the sensor hardware, assuming the device is in an upright orientation. In our current setup, we align the sen- sor's 's forward facing direction with 0%, and angles increase clockwise in the horizontal (azimuthal) plane. Each speech file Yooriginal (n), sampled at original rate foriginal is loaded and resampled to a uniform sampling frequency f. = 16 kHz. This ensures consistency across all signals for accurate con- volution wiht the impulse responses. The resampled signal v(n) is obtained as:
3.2. Spatial Speech Generation
Capturing and annotating spatial speech in real-world envi- ronments is often a labor-intensive process, further compli- cated by variations in acoustic properties and the constraints of recording equipment. To efficiently generate a diverse dataset that covers a wide range of sound sources while en- suring comprehensive ground-truth metadata, simulation- based approach has been adopted.
For this study, we utilized the Librispeech dataset (Panay- otov et al., 2015), a publicly available corpus of high-quality English speech, sampled at 16KHz. The dataset provides phonetically diverse speech recordings, large vocabulary, and clean and noisy subsets, making it ideal for speech recognition and spatial analysis under different acoustic con- ditions (Words, 2025). For the application of spatial ASR, we generated comprehensive 400-hour spatial speech dataset. We pick 500 original samples from the LibriSpeech dataset, and convoluted them with the impulse responses from 1 to 360" with 1 resolution.
v(n) = Resample (Joriginal (n), Ssx Suriginal
(2)
where the resampling operation adjusts the temporal resolu- tion of the signal while preserving it's content.
To simulate how speech signal is perceived at angle 0, the resampled speech signal y(n) is convolved with the corre- sponding impulse response (n)... The discrete-time convo- lution operation (Cheng & Desoer, 1982) is defined as:
For the application of soundscaping, leveraging LibriSpeech, we generated a comprehensive 2,000-hour spatial speech dataset that simulates scenarios involving 1 to 5 speakers speaking simultaneously. To ensure robust and representa- tive data, the number of speakers is evenly distributed across the dataset, with uniform coverage of DoA angles. This de- sign enhances the dataset's robustness to variations in DoA, speaker count, and speaker diversity. Multi-speaker sam- ples were created by randomly selecting speech segments and assigning distinct DoA values, ensuring realistic spa- tial distributions and promoting generalization to real-world scenarios. Our dataset symbolizes the 360 spatial diversity and multi-speaker dynamics it embodies, offering robust foundation for spatial speech processing.
DG
yboov e (n) = (y he)(n)
Σ
v(m) he m) (3)
where,
y(m) is amplitude of the speech signal at time index m.
(n-m) the impulse response of the system for angle
0, delayed by m samples.
(e the resulting signal after the speech is filtered by the spatial characteristies at angle 0.
Refer to the spectrogram of recordings from different direc- tions in Appendix K.
We created a syn- thetic spatial speech dataset by convolving the clean Lib- rispeech signals with Owlet-specific impulse response at discrete angles. The system utilizes a set of frequency- domain impulse response (w,, f), where w represents the angular frequency and 0 denotes the angle of arrival (in de- grees). Since the microstructure has a consistent shape, we
Spatial Speech Generation Process:
Our dataset provides high-fidelity synthetic spatial speech corpus tailored for evaluating monaural spatial sensing mod- els. By leveraging Owlet-specific impulse responses, our dataset accurately simulates how speech signals are per- ceived at different angles, ensuring realistic spatial encoding. The one-time calibrated impulse responses maintain consis- tency across experiments, eliminating variations caused by
4

---

SING: Spatial Context in Large Language Model for Next-Gen Wearables
environmental changes. Furthermore, the dataset is derived from Librispeech, widely used speech corpus, ensuring high-quality linguistic content. The rigorous preprocessing steps, including uniform resampling and frequency -o-tettttttte domain transformation via IFFT, guarantee temporal align- ment and signal integrity. By applying discrete-time con- volution with angle-specific impulse responses, our dataset faithfully replicates the directional filtering imposed by the Owlet microstructure, making it ideal benchmark for spatial speech recognition and DoA estimation tasks.
DoA Encoder. The DoA encoder is convolutional neural network, balancing computational efficiency with robust fea- ture extraction. The architecture consists of three sequential convolutional blocks, each comprising 2D convolutional layer, batch normalization for stabilizing training, a ReI.U activation function, and max-pooling for spatial dimension reduction. A flattening layer reshapes the output from the convolutional layers into a ID vector, which then passed through fully connected layer with 512 units, followed by a dropout layer to mitigate overfitting. The final linear layer maps the high-dimensional feature vector to DoA pre- diction output. Refer to the 3D UMAP visualization of the embeddings from different angles in Appendix G.
4. SING: Spatial Context to Wearable LLM
Our system is designed to integrate spatial speech under- standing into LLMs while addressing the unique constraints of wearable devices. It consists of three main components: spatial speech sensing, embedding alignment, and LLM fine- tuning. As shown in Figure 2, the spatial speech sensing module uses Owlet recording hardware to capture high- resolution spatial cues, such as directionality, without the need for a traditional large microphone array. The record- ings from the two channels of the Owlet are the input for the DoA encoder and the speech encoder. Spatial cues are pro- cessed into low-dimensional embeddings using lightweight models. Whisper encodes raw speech into dense speech em- beddings that capture linguistic features. These embeddings together with spatial embeddings are then aligned with the input space of LLMs. The LLM is then fine-tuned on our spatial speech dataset, enabling it to interpret speech and DoA.
4.2. Speech-to-text Encoder.
The speech encoder utilized in this work is OpenAI's Whis- per model (Radford et al., 2023). Its encoder generates features that are well-suited for modeling speech and ef- fectively incorporating contextual information about back- ground noises (Gong et al., 2023). To extract Whisper em- beddings, speech files are first processed to ensure con- sistency in sampling rate, mono channel format and fixed duration by padding to 30 seconds. The processed wave- forms are passed through the WhsiperProcessorr for feature extraction, which generates mel-spectroggaam features as input to the model. We use Siisserrlarge-3(Whiispp model for generating features. The encoder outputs hid- den states for each time frame, representing intermediate representations for each time frame that capture both seman- tic content and acoustic context of the speech signal. To reduce the dimensionality while preserving critical infor- mation, adaptive average pooling is applied to the hidden states along the temporal axis, yielding fixed-size embed- dings of shape [POOL_SIZF, hidden_din _dim] for each speech file, where POOL SIZE is 128 and hidden dim is 1024. The embeddings illustrate both temporal and semantic character- istics of the speech for downstream tasks such as transcrip- tion.
4.1. Spatial Speech Encoder
Speech signal preprocessing. Different from existing work that uses an array of microphones for spatial sensing, our system uses only monaural microphone inside the mi- crostructure. Given the different impulse responses from directions of speech, the amplitude of the signal changes overtime, and different frequencies have different patterns of amplitude change. Due to this nature, we convert the cap- tured signal from time domain z(n) to frequency domain X (t. f) using STFT
4.3. Alignment to LLM Space
Pretraining. The spatial encoder in our system is pre- trained on a spatial ASR task consisting of two subtasks: DoA prediction, and speech recognition. The DoA task estimates the directional angles of each speaker. The ASR task transcribes spoken speech into text. The two tasks are optimized using cross-entropy loss. The spatial embeddings derived from DoA and the Whisper encoder are not inher- ently aligned with the input space of LLMs. To address this, we adopted the approach outlined in LLavv (Liu et al., 2024), employing a simple linear layer as a projection matrix, de- noted as W, to map the spatial features into the language em- bedding space of the LLM. This linear projection layer was selected for its lightweight design, offering a more efficient
N-1 x(t,f)= = Σ I(n) win - t] e 2 n=
(4)
where w(n) represents a Window function of length N. Then we convert the output into mel scale,
Σ IX HH,(H) [IX +
(t) = log
(5)
where (k) is Mel filter for the j-th band, and is a small constant to avoid log of zero.
5

---

SING: Spatial Context in Large Language Model for Next-Gen Wearables
"The speaker degrees says; Good morning, ladles and gentlemen."
LoRA
Large Language Model
DoA Embeddings
Spatial Speech
Embeddings
Fully Connected Layer
Projection W
Text Embeddings
CNN 3
DoA Tokens
Speech Tokens
*
Speech
DoA Encoder
Encoder (Whisper)
"Can you summarize the speech? Who said it?"
Sel-Speetrogram
Text Instruction
Owlet Recording
Prompt
Hardware
Microstructure-assisee spatial encoding
Speech
Pre-trained DoA Encoder
Figure 2. Spatial-aware framework for direction and speech transcription.
and streamlined solution compared to the Q-Former connec- tion module used in BLIP2 (Li et al., 2023a), the perceiver resampler and cross-attention layers in Flamingo (Alayrac et al., 2022). This design choice prioritizes efficiency and simplicity, ensuring seamless integration without adding significant complexity to the model architecture.
number of speakers, call the corresponding DoA encoder, capturing the precise DoA angles. We train DoA encoders, ranging from 1 to speakers. The embeddings generated by the number-of-speaker encoder and the corresponding DoA encoder are concatenated to form a unified representation, which serves the output of our spatial speech encoder Z
Zspatial Concat =Coneat(aamsspppaeeer (pwasspaaeee(()) DoA(X)) (6)
Supervised Fine- TinniTnning. We fine-tuned our pre-trained model using an instruction fine-tuning approach to enhance the LLM's ability to follow human instructions and provide greater control over its output. Specifically, we adopted a su- pervised fine-tuning (SFT) strategy (SFT Trainer), enabling the model learn from spatial embeddings, prompts, and their corresponding responses under direct supervision. For this purpose, we use the LLaMA 3.2 3B model (Touvron et al., 2023) with LoRA (Hu et al., 2021), method that optimizes the fine-tuning process for spatial speech under- standing. (Refer to the detailed prompts Appendix I) Instead of modifying the entire model, LoRA introduces trainable low-rank matrices into the lower layers of the LLM, significantly reducing computational and memory overhead while accelerating training.
where X is 8-second speech input, Num-speaker() and DoA(-) are the number-of-speatr encoder and DoA enoder, Concat(-) is the concatenation operation along the feature dimension. The number-of-speaker encoder follows same structure as the DoA encoder introduced Section 4.1. The DoA encoders share the same structure involves adjusting the final fully connected layer to produce regression output representing multiple DoAs sorted in ascending order. This change allows the model to handle dynamic outputs while maintaining the original convolutional structure for feature extraction. The concatenated embedding also aligned to the LLM space following the same strategy as described in Section 4.3. We prepared question answer pair for finetuning the LLM model generating contextual responses for multiple DoA detection.
5. Soundscaping: Multi-DoA Encoder
6. Evaluation of Spatial-Aware ASR
Soundscaping for LLM will enhance human-Al interaction by creating more natural, immersive and context-aware au- ditory experiences. We present the first approach to incorpo- rate multi-direction of arrival (multi-DoA) information into an L.M. Our multi-DoA encoder consists of two key com- ponents: a number-of-speake encoder and a DoA encoder. The number-of-spea of-saee encoder predicts the count active speakers in the environment. (Refer to the performance of of-sekee encoder in Appendix J) Based on the
6.1. Dataset
To evaluate the proposed microstructure-based spatial speech encoder, there no publicly available real syn- thetic datasets that consist of general speech. Since DNN- based methods need sufficiently large datasets to train on, we first calibrate the impulse response of the microstructure from all angles, and then create a large synthetic dataset by convolving the calibrated impulse response with the speech
6

---

SING: Spatial Context in Large Language Model for Next-Gen Wearables
Table 1. Comparison of performance on spatially aware ASR with one sound source. BAT supports DoA estimation and audio source type recognition, while SALMONN focuses on speech ASR and other LLM functions but lacks spatial awareness. Our model uniquely integrates DoA and ASR for spatially aware ASR.
Model
Supported Task
Metric / Performance
BAT
DoA Estimation + Audio Source Recognition (No Speech)
MAE (*) 4: 88.52
SING [our work]
DoA + Speech ASR (Spatial Awareness)
MAE (*) 4: 25.72
SALMONN
Speech ASR and LLM Functions (No Spatial Awareness)
WER (%) J: 2.2
SING [our work]
Speech ASR (without DoA)
WER (%) L: 1.8
SING [our work]
DoA + Speech ASR (Spatial Awareness)
WER (%) U: 5.3
samples in the LibriSpeech dataset, as described in Section 3.2. After convolution with the impulse responses, we trim or pad these clips to 30 seconds. The resulting waveforms are monaural with only 1 channel at a 16kHz sampling rate. We use a window size of 400, hop size of 160, and 80 mel-bins to compute the Short-Time Fourier Transforms (STFTs) and mel-sppectrograms. As result, for 30-second recording, the Mel-spectrogram feature dimension is (I, 80, 3000).
noise and room reverberation, as shown in Appendix E, Moreover, we evaluated the performance for three other speech datasets and one audio dataset. Further details can be found in Appendix F.
In terms of ASR, while our model achieves word error rate (WER) of 5.3% for speech recognition in the spatial ASR configuration, which is higher than SALMONN's WER of 2.2%, it is important to highlight that our model incorporates spatial features, a capability absent in SALMONN. Integrat- ing spatial information, such as DoA, introduces additional complexity to the task by requiring the model to jointly process acoustic and spatial cues. This enables advanced spatial reasoning and makes our approach more versatile for applications involving spatially distributed sound sources. Furthermore, when spatial features are excluded, our model achieves WER of 1.8%, outperforming SALMONN and demonstrating its strong baseline performance for speech recognition. These results underscore that the inclusion of spatial features, while slightly increasing the WER, signifi- cantly broadens the utility of the model, offering a trade-off' between basic recognition accuracy and enhanced spatial awareness. Refer to the qualitative examples in Appendix C.
6.2. Training Details
DoA encoder. The encoder is trained on an A100 GPU, with each epoch taking approximately 20 minutes. The total number of epochs is 20. The encoder was first trained regression task where the output are the DoA values. We take the last fully-connected layer as the embedding.
LLM alignment and fine-tuning, This part is separated into two stages. We first freeze the LLM model to train parameters for the projection layer. After the loss has con- verged to stable, we fine-tune the LLM model using LoRA to enhance the LLM's ability to follow human instructions and provide greater control over its output. The training is completed on 3 H100 80 GB GPUs. Refer to the hyperpa- rameters for training in Appendix B.
7. Evaluation of Multi-DoA Encoder
6.3. Performance Spatially Aware ASR
There is currently no existing work that integrates both DoA estimation and ASR within a LLM framework. To provide a comprehensive evaluation, we separately compare our model's DoA performance with BAT and its ASR perfor- mance with SALMONN, as shown in Table 1.
7.1. Dataset
Five datasets are generated with no temporally overlap- ping sources, maximum two overlapping sources, maximum three overlapping sound sources, four overlapping sound sources, and five overlapping sound sources. We start syn- thesizing a recording by randomly choosing the number of speech samples. Then we randomly pick the number of DoAs that at least 10 degrees of separations are guaranteed between sound sources to avoid spatial overlapping. After convolving the speeches with the impulse response of its DoA, the convoluted speeches are summed together as the final recording. The final recordings are first loudness nor- malized by scaling them so that each clip has the same total sound energy. The final recording is trimmed or padded to 8 seconds. The resulting waveforms are monaural with
Since we only use the monaural microphone inside the mi- crostructure for DoA estimation, we compare our perfor- mance with BAT in monaural case. Our model achieves significantly lower Mean Absolute Error (MAE) of 25.72 compared to BAT's 88.52". This result underscores the strength of our microstructure-baar based spatial encoding, which enables precise localization of sound sources in a computationally efficient manner, outperforming BAT's ap- proach. Our system also shows robust performance under
7

---

SING: Spatial Context in Large Language Model for Next-Gen Wearables
Table Comparison of MAR DOA error and MEEM across models with known number of active sources. MEEM calculated
MSE x (Number of Microphones).
Model
Metric
Source
2 Sources
3 Sources
4 Sources
5 Sources
MAE (L)
90.03
x
X
X
SELDNet
MEEM (4)
360.12
x
X
X
Median Error (1)
90.14
x
X
X
X
MAE (L)
43.79
AudioMAE
MEEM (4)
43.79
X
X
Median Error (4)
27.79
X
X
X
X
MAE ())
25.72
24.16
28.11
23.31
17.08
SING (Ours)
MEEM (4)
25.72
24.16
28.11
23.31
17.08
Median Error (L)
13.00
13.00
20.00
18.00
13.00
only channel at a 16kHz sampling rate. We use a win- dow size 2048, a hop size of 512, and 128 mel-bins to compute the Short-Time Fourier Transforms (STFTs) and mel-spectrograms. As result, for 8-second recording, the Mel-spectrogram feature dimension is (1, 128, 251).
number of microphones, making it particularly useful for comparing models designed for minimalist setups like ours. By combining these two metrics, we not only assess the raw accuracy of the models but also highlight the efficiency of our approach in leveraging monaural microphone com- pared to multi-microphone arrays. We also show the median error of DoA as better understanding of the distribution of errors.
7.2. Baseline
Since our approach enables DoA estimation using monau- ral microphone, which is fundamentally not achievable with traditional signal processing methods like MUSIC (Kundu, 1996; Tang, 2014) that rely on multiple microphones, we instead compare our model against deep learning-based based al- gorithms, as they can be designed to handle both monaural and array-based spatial sensing setups. We compare our performance with array-based speech DoA detection SELD- Net (Adavanne al., 2018). Following the evaluation of BAT (Zheng et al., 2024), we train the monaural-microphoon based model AudioMAE (Huang et al., 2022) using our microstructtur--ccoodd monaural-micron miiropoooeeeeo dataset for DoA estimation.
7.4. Performance of DoA Estimation
Table presents the DoA estimation performance of various models under scenarios with known number of active sources. SELDNet (Adavanne et al., 201 8), utilizing four microphones, shows relatively high MAE error for single source (90.03"). Due to the lack of spatial information in AudioMAE, shows a MAE of 43.79% and a median error of 27.79%. Our proposed model, using only one microphone, achieves competitive performance despite its minimalist hardware setup. For source, it yields MAE of 25.72 and median error of 13.00%, which outperforms AudioMAE, showing the efficiency of microstructure.
Notably, for scenarios involving two or more sources, our model demonstrates robust performance with MAE val- ues of 24.16%. 28.11 23.31 and 17.08% for two, three, four, and five sources, respectively. Refer the CDF plots in Appendix H. These results highlight the ability of approach provide reasonable DoA estimation accuracy while minimizing hardware complexity, making it suitable for wearable and low-power applications. Additionally, the MEEM values underline the efficiency of our approach in leveraging minimal hardware without sig- nificant performance trade-offs. For qualitative results, refer to Appendix D.
7.3. Evaluation Metrics
To comprehensively evaluate the performance of our model and facilitate fair comparisons with prior works, we employ three metries: Mean Absolute Error (MAE), the Modified Error Efficiency Metric (MEEM), and median error. MAE is a widely used metric that quantifies the average angular difference, in degrees, between the estimated and ground- truth DoA. This metric directly evaluates the accuracy of the DoA predictions, offering an intuitive understanding of the model's spatial localization capabilities. In addi- tion MAE, we introduce MEEM, which is designed to normalize the performance by accounting for the number of microphones used in the system. MEEM is calculated of Microphones), where MSE
monaural--iccopphe
We also evaluated by comparing our model SING with CNN- based Owlet model. As shown in the CDF plots in Ap- pendix M, our model, SING, has comparable performance with the similar experiment for the DoA estimation accuracy reported the Owlet.
as MEEM=MSExNummee represents the mean squared error of the DoA predictions. Unlike traditional metrics, MEEM provides a balanced view of model efficiency by penalizing setups that require larger
8

---

SING: Spatial Context in Large Language Model for Next-Gen Wearables
8. Discussion and Future Work
dresses key challenges in spatial speech processing, such as computational efficiency, privacy, and hardware constraints. The proposed system paves the way for transformative ap- plications in augmented reality, accessibility, and interactive computing, enabling context-aware and intelligent interac- tions between humans and machines. Beyond its technical contributions, this work has the potential to improve ac- cessibility for individuals with disabilities, enhance safety in wearch-and-rescue operations, and redefine user experi- ences in immersive environments. While promising, the ethical implications of deploying such systems, including privacy and responsible use, must be carefully considered to maximize societal benefits.
Our work demonstrates the feasibility of integrating spatial speech understanding into LLMs through a novel combina- tion of spatial audio encoders, DoA estimation, and speech recognition. By leveraging compact microstructure- based sensing, we enable accurate speech transcription and spatial awareness using significantly compact microphones com- pared to traditional systems. Our method embeds spatial cues directly within a monaural recording, allowing precise DoA estimation in an extremely compact design, unlocking new possibilities for minimalist perception approaches and wearable sensing technologies. While our model achieves competitive performance in both numerical and qualitative evaluations, several limitations and opportunities for future exploration remain.
References
Future work. Future work will focus on extending the spa- tial encoder to support elevation angle estimation, enabling full 3D spatial audio processing. Furthermore, collecting real-world datasets with diverse acoustic conditions and speaker configurations will enhance the robustness and ap- plicability of the system. Another promising direction is the integration of multimodal inputs, such as combining visual information from cameras with audio, to improve spatial understanding and enable richer context-aware appli- cations. We also plan to optimize the system for low-power hardware, ensuring its viability for real-time processing on wearable devices.
Adavanne, S., Politis, A., Nikunen, J., and Virtanen, T. Sound event localization and detection of overlapping sources using convolutional recurrent neural networks. IEEE Journal of Selected Topics in Signal Processing, 13 (1):34 48, 2018.
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. Flamingo: a visual language model for few- shot learning. Advances in neural information processing systems, 35:23716-23736 2022.
proccessin
Anthropic. Claude language model by anthropic. https: / www..anthropi.. com/ www.cooomeeeeeeccocoooccocccooce inwex/ideee//ee caadex//cladde.../ 2023.
9. Conclusion
Ardila, R., Branson, M., Davis, K., Henretty, M., Kohler, M. Meyer, J., Morais, R., Saunders, L.,, Tyers, F. M., and Weber, G. Common voice: A massively-multiinuaal speech corpus. arXiv preprint
In this work, we introduced a framework for spatial speech understanding integrated with LLMs, enabling directional audio processing and speech recognition on compact wear- able devices. Through iicrosttrcturre-bssed spatial en- coder and alignment with Whisper embeddings, our system achieves DoA estimation and speech ASR performance with minimal hardware resources. This work paves the way for enhancing wearable technologies in applications such as augmented reality and accessibility, laying foundation for future advancements in spatially aware LLMs.
2019.
Chen, C., Jain, U., Schissler, C., Gari, S. V. A., Al- Halah, Z.,, Ithapu, V. K., Robinson, P., and Grauman, K. Soundspaces: Audio-visual navigation in 3d environ- ments. In Computer Vision-ECCY 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceed- ings, Part VI 16, pp. 17-36. Springer, 2020.
Chen, J. C.-Y,, Saha, S., Stengel-Eskin, E., and Bansal, M. Magdi: Structured distillation of multi-agent interaction graphs improves reasoning in smaller language models. arXiv preprint arXiv:2402.01620, 2024.
Acknowledgment
This work was partially supported by NSF CAREER Award 2238433. We also thank our industry partners for their continued support of the iCoSMoS Laboratory at UMD.
Cheng, V. and Desoer, C. Discrete time convolution control systems. International Journal of Control, 36(3):367- 407, 1982.
Impact Statement
This work introduces novel framework for integrating spa- tial audio context into LLMs, specifically designed for wear- able devices. By leveraging mierostructure-baat spatial sensing and efficient fusion techniques, this research ad-
Choi, H. K. and Y. Picle: eliciting diverse behaviors from large language models with persona in-context learning. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024.

---

SING: Spatial Context in Large Language Model for Next-Gen Wearables
Cui, C., Ma, Y., Cao, X., Ye, W., and Wang, Z. Drive as you speak: Enabling human-like interaction with large language models in autonomous vehicles. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 902-909, 2024.
Huang, D., Yan, C., Li, Q., and Peng, X. From large lan- guage models large multimodal models: A literature review. Applied Sciences, 14(12):5068, 2024a.
Huang, P.-., Xu, H., Li, Baevski, A., Auli, M., Galuba, W., Metze, F,, and Feichtenhofer, C. Masked autoen- coders that listen. Advances in Neural Information Pro- cessing Systems, 35:28708 28728008-28700, 2022.
CusionChat.
The future of listening: Google pixel buds to feature gemini IIm and ai assistant, 2025, URL https:
Huang, R., Li, M., Yang, D., Shi, J., Chang, X., Ye, Z., Wu, Y., Hong, Z.,, Huang, J., Liu, J., et al. Audiogpt: Understanding and generating speech, music, sound, and talking head. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 23802-23804 2024b.
-gemini-llm-ane aaw.----nnn-a-sssstaaaa Accessed: 2025-01-30.
Gan, C., Zhang, Y., Wu, J., Gong, B., and Tenenbaum, J. B. Look, listen, and act: Towards audio-visual embodied navigation. In 2020 IEEE International Conference Robotics and Automation (ICRA), pp. 9701-9707. IEEE, 2020.
Ito, K. The lj speech dataset. Online, 2017. URL http s : //ke/eei thi com Ltom/LJ--ppp L.S-Speech-Daaasee. Accessed: 2025-01-29.
Garg, N., Bai, Y., and Roy, N. Owlet: Enabling spatial in- formation in ubiquitous acoustic devices. In Proceedings of the 19th Annual International Conference on Mobile Systems, Applications, and Services, pp. 255-268, 2021.
Kailas, G. and Tiwari, N. Design for immersive experience:
Role of spatial audio in extended reality applications. In Design for Comorrow-Volund 2: Proceedings of ICoRD 2021, pp. 853-863. Springer, 2021.
Gerganov, G. whisper.cpp: High--peffomance inference of openai's whisper automatic speech recognition (asr) model in c/c++. https: / g huub. com/ ggml-o rr/whissper cpp, 2023. Accessed: 2025-05-09.
Knapp, C. and Carter, G. The generalized correlation
method for estimation of time delay. IEEE transactions on acoustics, speech, and signal processing, 24(4):320- 327, 1976.
Ghosh, S., Kumar, S., Seth, A., Evuru, C. K. R., Tyagi, U., Sakshi, S., Nieto, O., Duraiswami, R., and Manocha, D. Gama: A large audio-language language model with advanced au- dio understanding and complex reasoning abilities. arXiv preprint arXiv:2406. 1768, 2024.
Kumar, L. P. and Kabiri, A. Meeting summarization:
A survey of the state of the art. arXiv preprint arXiv:2212.08206, 2022.
Kumaran, V., Rowe, J., Mott, B., and Lester, J. Scenecraft: Automating interactive narrative scene generation in dig- ital games with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence and In- teractive Digital Entertainment, volume 19, pp. 86-96, 2023.
Gong, Y., Chung, Y.-A.. and Glass, J. Ast: Audio spec- trogram transformer. arXiv preprint arXiv:2104.01778, 2021.
Gong, Y., Khurana, S., Karlinsky, L., and Glass, J. Whisper- at: Noise-robust automatic speech recognizers are also strong general audio event taggers. arXiv preprint arXiv:2307.03188. 03,88 2023.
Kundu, D. Modified music algorithm for estimating doa of signals. Signal processing, 48(1):85-900 1996.
Heckbert, P. Fourier transforms and the fast fourier trans- form (fft) algorithm. Computer Graphics, 2(1995):15- 463, 1995.
Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023a.
Hsich, C.-Y., Zhang, J., Ma, Z., Kembhavi, A., and Kr- ishna, R. Sugarcrepe: Fixing hackable benchmarks for lanione compositionaliiy. Advances in neural information processing systems, 36. 2024.
Li, W., Fan, H., Wong, Y., Yang, Y., and Kankanhalli, M. Improving context understanding in multimodal large language models via multimodal composition learning, In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org. 2024.
Hu, E. J., Shen, Y., Wallis, P., AZu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106. 09685, 2021.
Li, Y., Yuan, R., Zhang, G., Ma, Y., Chen, X., Yin, H., Xiao, C., Lin, C., Ragni, A., Benetos, E., et al. Mert:
10

---

SING: Spatial Context in Large Language Model for Next-Gen Wearables
Acoustic music understanding model with large-scale self- supervised training. arXiv preprint arXiv:2306.00/17,, 2023b.
Roy, R. and Kailath, T. Esprit-estimation of signal parame- ters via rotational invariance techniques. IEEE Transac- tions acoustics, speech, and signal processing, 37(7): 984-995, 1989.
Liu, H., Li, C., Wu, Q., and Lee, J. Visual instruction tun- ing. Advances in neural information processing systems, 36, 2024.
Schmidt, R. Multiple emitter location and signal parameter estimation. IEEE transactions on antennas and propaga- tion, 34(3)277-280, 1986.
Ma, Z., Yang, G., Yang. Y., Gao, Z., Wang. J., Du, Z., Yu, F., Chen, Q.. Zheng, S., Zhang, S., al. Speech recognition meets large language model: Benchmarking, models, and exploration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 24840 -4848, 2025.
SFT Trainer. Hugging face sft trainer documentation. URL https: Phttps://uug/ //uggi ingiingg face. cotoccc/ccooo/o/cc/c////// ft_trainer.
Song, S., Li, X., Li, Zhao, S., Yu, J., Ma, J., Mao, X., and Zhang, W. How to bridge the gap between modalities: A comprehensive survey on multimodal large language model. arXiv preprint arXiv:2311.07594 2023.
McInnes, I.., Healy, J., and Melville, J. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv: 03426, 2018.
Sun, G., Yu, W., Tang, C., Chen, X., Tan, T., Li, W., Lu, L., Ma, Z., Wang, Y., and Zhang, C. video-salmonn: speech-enhanced audio-visual large language models. In Proceedings of the 41st International Conference Ma- chine Learning, ICML'24. JMLR.org, 2024.
MosaicML.
Introducing mpt-7b: A new standard for open- source, commercially usable language models. 2023. URL www.tttt///// wwt.:wwossal. mosai cml....... com/bloo///oooooo /mp t-7b.
Nagrani, A., Chung, J. S., and Zisserman, A. Voxceleb: a large-scale speaker identification dataset. arXiv preprint (rXiv:1706.08612,, 2017.
Tang, C., Yu, W., Sun, G., Chen, X., Tan, T., Li, W., Lu, L., Ma. Z., and Zhang, C. Salmonn: Towards generic hearing abilities for large language models. arXiv preprint arXiv:2310.14 13289, 2023.
OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023. URL htttt:///ax/ii.... g/abs/2303.0877
Tang, C., Yu, W., Sun, G., Chen, X., Tan, T, Li. W., Zhang. J., Lu, L., Ma. Z., Wang, Y., et al. Can large lan- guage models understand spatial audio? arXiv preprint Ttt::2406.07914. 2024.
Panayotov, V., Chen, G., Povey, D., and Khudanpur, S. Librispeech: an asr corpus based on public domain au- dio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 5206-5210. IEEE, 2015.
Tang, H. Doa estimation based on music algorithm, 2014.
Team, V. Vicuma: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality. LMSYS Org, 2023. URL https: http///msyss.e.// orw.orgg//oo.o.. wwwpooggggggoo//oe/eoo/// 03- - una/.
Park, S., Jeong, Y., and Lee, T. Many-to-many audio spec- trogram tansformer: Transformer for sound event local- ization and detection. In DCASE, pp. 105-109, 2021.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302.13977 2023.
Parkkola, K. Spatial audio techniques for inner voice repro- duction in virtual reality environments. 2023.
Pekmezci, M. and Genc, Y. Evaluation ssim loss function in rir generator gans. Digital Signal Processing, 154: 104685, 2024.
Wang, Y., Yang, B., and Li, X. Fn-ssl: Full-band and narrow- band fusion for sound source localization. arXiv preprint arXiv:2305.19610 2023.
Piczak, K. J. Esc: Dataset for environmental sound classi- fication. In Proceedings the 23rd ACM international conference on Multimedia, pp. 1015-1018, 2015.
Whisper. Whisper large v3 model. UtLhttts:://nn ingface. coogface.o//ooo ccoo/ppeaai.. wh sper-
Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. Robust speech recognition via large- scale weak supervision. In International conference on machine learning, pp. 28492-2855 28518. PMLR, 2023.
Words, W. W. Top sources for high-quality speech data, 2025. URL our
peech-data/ Accessed: 2025-01-15.

---

SING: Spatial Context in Large Language Model for Next-Gen Wearables
Xu, Q., Jiang, C.,, Han, Y., Wang. B., and Liu, K. R. Wave- forming: An overview with beamforming. IEEE Commu- nications Surveys & Tutorials, 20(1):132-149, 2017a.
network for robust sound source localization. CAAI Trans- actions Intelligence Technology, 7(3):446-458) 2022.
Yin, S., Fu, C., Zhao, S., Li, K., Sun, X., Xu, T., and Chen, A survey on multimodal large language models. arXiv preprint arXiv:2306. 13549, 2023.
Xu, Y., Kong, Q., Huang, Q. Wang. W., and Plumbley, M. D. Convolutional gated recurrent neural network incorporat- ing spatial features for audio tagging. In 2017 Interna- tional Joint Conference on Neural Networks (ICCNN), pp-
Zheng, Z., Peng, P., Ma, Z., Chen, X., Choi, E., and Harwath, D. Bat: Learning to reason about spatial sounds with large language models, arXiv preprint arXiv:2402.01591, 2024.
3461-3466 IEEE, 2017b.
Yang, B., Ding, R., Ban, Y., Li, X., and Liu, H. Enhancing direct-path relative transfer function using deep neural
12

---

SING: Spatial Context in Large Language Model for Next-Gen Wearables
A. Towards Spatially Aware Language Models
Large Language Models (LLMs) have proven invaluable for a wide range of applications by assisting with complex tasks such as legal text interpretation and other specialized domains including healthcare, finance and academic research. However, as human interactions and real-world scenarios increasingly demand context that goes beyond textual verbal cues, integrating spatial knowledge into LLMs has become essential. Humans not only understand semantic context but also perceive and interpret physical cues such sound and location, which inform more holistic decision-making in daily life. In this vein, the physics-aware LLMs can power applications like autonomous robotics that need to navigate and interact with three-dimensional spaces, or digital-twins that can simulate real-world environments for urban planning, energy management, and supply chain optimization. By grounding LLMs in the realm physical space, we ensure these models can provide more accurate, relevant and human like support, bridging the gap between abstract textual reasoning and tangible real-world contexts.
B. Hyperparameters
SING presents specific hyperparameters for DoA encoder, LLM pretraining and finetuning in Table 3.
Hyperparam
DoA Encoder
Num-of-speaker Encoder
LLM Pretraining
LLM Fine- Tuning
batch_size
32
32
8
num-epochs learning rate
20
20
5
10
0.001
0.001
1e-5
1e-5
patience
5
5
X
warmuppsse stepp
x
x
0
0
loss_function
MSELoss
Cross Entropy Loss
Causal LM loss
Causal LM loss
optimizer
Adam
Adam
AdamW
AdamW
scheduler
RodlaeLL
ReduceLROnPlatean
Linear
Linear
gradient.chkyp
x
X
Enabled
Enabled
max_se max_seq_length
X
512
512
LoRA _ana (r)
X
X
X
LoRA alpha
X
X
X
16
LoRA-dropout
X
X
X
0.1
device
CPU/GPU
CPU/GPU
Multi-GPU Dist.
Multi-GPU Dist. FP16 autocast
mixed pprcciion
X
X
FP16 autocast
Table Key hyperparameters for the DoA Encoder, Num-of-speaker Encoder, LLM Pretraining, and LLM Fine- Tuning. 'X' indicates that the hyperparameter does apply.
C. Responses Generated for Spatial-Aware ASR
We present several examples of the spatial transcriptions from SING shown in Table 4.
Spatial Transcriptions
The speaker is speaking approximately angle 61 degrees. The speech says: Lectures.
The speaker is speaking approximately angle 182 degrees. The speech says: The Emperor's Daughter. The speaker is speaking approximately angle 140 degrees. The speech says: The Wandering Singer. The speaker is speaking approximately angle 96 degrees. The speech says: Tom nodded worriedly.
Table Spatial-aware ASR.
D. Responses Generated for Multiple Direction of Arrival Detection
Table 5 presents several examples of number of sound source inference and the DoA detections from SING.
13

---

SING: Spatial Context in Large Language Model for Next-Gen Wearables
Num of Sources
Response Generated
There is one speech source. The speech source's 's degree of arrival 77.0 degrees.
There are 2 speech sources. Speech source I's degree of arrival is 39.0 degrees, and speech source 2's degree of arrival is 101.0 degrees.
There are speech sources. speech source I's degree of arrival is 18 degrees, speech source 2's degree of arrival is 39 degrees, and speech source 3's degree of arrival is 257 degrees.
There are 4 speech sources. Speech source I's degree of arrival is degrees, speech source 2's degree of arrival is 118 degrees, speech source 3's degree of arrival is 203 degrees, and speech source 4's degree of arrival is 242 degrees.
There are 5 speech sources. Speech source I's degree of arrival 34 degrees, speech source 2's degree of arrival is 161 degrees, speech source 3's degree of arrival is 195 degrees, speech source 4's degree of arrival is 234 degrees, and speech source 5's degree of arrival 317 degrees.
Table 5. Response generated for multiple DoA detection.
E. Performance under Noise and Room Reverberation
To evaluate the impact of noise and reverberation on our system's performance, we leverage the GTU-RIR dataset (Pekmezei & Gene, 2024). This dataset provides high-fidelity room impulse responses (RIRs) measured in diverse acoustic environments. including 11 rooms with types of classroom, conference hall, sports hall, office, hotel room and staircase, making ideal for simulating realistic reverberation conditions in evaluation. To systematically assess the robustness of our model, we conduct experiments under controlled conditions with varying levels of signal-to-noise ratio (SNR) and reverberation. Table 6 presents the results, showing the Mean Absolute Error (MAE) and Median Error in degrees across different test conditions. Our findings indicate that while reverberation and noise impact DoA accuracy, our system remains robust compared to existing work. Specifically, without noise or reverberation, we achieve a MAE of 25. 25.72%, which increases to 48.69 under reverberation, demonstrating the effect of multipath reflections. The addition of noise different SNR levels further degrades performance, with errors increasing as noise levels rise. These results underscore the importance of robust spatial encoding techniques in real-world applications, particularly in noisy and reverberant environments. Future work will focus on adaptive denoising strategies and spatial de-reverberation methods to enhance the resilience of our framework in challenging acoustic conditions.
Table Performance under noise and room reverberation: We employ our proposed architecture to infer under distinct scenarios, with and without noise (different SNR in dB levels) and and reverberation.
Noise
Reverberation
MAE (9)
Median Error (9) $
x
25.72
13.00
x
48.69
28.42
60dB
48.57
28.27
50dB
49.50
28.95
40dB
50.89
31.08
30dB
33.01
21.25
30dB
54.43
33.12
F. Performance on Different Datasets
To assess the generalizability of our proposed spatial-aware ASR system, we evaluate its performance across multiple datasets, including our synthetically generated dataset based on LibriSpeech, as well as other speech datasets Common Voice (Ardila et al., 2019), VoxCeleb (Nagrani et al., 2017), and LJ Speech (Ito, 2017). Our results in Table 7 indicate that the model achieves the lowest MAE and Median Error, benefiting from the model trained using the same dataset. However, when tested on other datasets, the MAE and Median Error slightly increases as expected. Despite this, our system still outperforms existing baselines in spatial- aware ASR tasks, demonstrating its robustness and adaptability different acoustic
14

---

SING: Spatial Context in Large Language Model for Next-Gen Wearables
conditions. Other than speech, also the performance audio dataset ESC-50 (Piczak, 2015), also shows robust performance speech.
Table Performance on Different Datasets: We evaluate our model across several datasets, reporting MAE and Median Error for DoA
estimation.
Dataset
MAE (
Median Error (9)
LibriSpeech (speech)
25.72
13.00
Common Voice (speech)
53.34
32.11
VoxCeleb (speech)
41.22
25.80
LJ Speech (speech)
80.21
34.73
ESC-50 (audio)
49.99
24.65
G. 3D UMAP Visualization of Spatial Embeddings
To evaluate the spatial embeddings derived from the DoA encoder, we utilized UMAP (Uniform Manifold Approximation and Projection/Melnnes et al., 2018) to visualize their structure in a seduced-dimensionn space. Figure showcases the UMAP visualizations, where the embeddings capture spatial cues from all angles, effectively encoding directional information. The clustering and angular organization of the points demonstrate the encoder's capacity to represent spatial diversity and preserve the underlying spatial structure of the speech signals. This provides valuable insights into the model's ability to generalize spatial features across multiple sound sources.
15
350
300
UMA Dimension
250
Angle (degrees)
0
200
150
110
100
15
50
UMAP Dimension 2
UMAP Dimension 1
5
10
15
-10
Figure 3. 3D UMAP visualization of spatial embeddings generated by the DoA encoder.
H. Cumulative Distributive Analysis for different speech sources for SING
We focus on analyzing the mean and median errors across varying conditions, illustrating our findings through comprehensive plots for clarity. Figure 10 shows the cumulative distribution function (CDF) plots for different speech sources.
15

---

SING: Spatial Context in Large Language Model for Next-Gen Wearables
CDF of Errors for spéech sources
speech
08
Cumulative Procaalitty
Prebabiity
0.5
04
0.
0.4
0.2
car Absotaee Drifernnes
Wean Emon 25 72"
Hean:
0.0
Mediaan Eron: 00
Median 13.00*
25
150
175
Absolute Erorr (degrees)
Absoluze idegrees)
Pigure CDF DoA
Figure 5. CDF for DoAs
CDF of Errors for speech sources 3
CDF of Errors for speech sources
1.
3.0
Pobablity
Proabilit
can
car Smors
28.11
19.00*
25
30
100
12b
150
175
20
40
50
100
229
240
Absolute Error (degrees)
Absoluue Error (degrees)
Figure 6. CDF for 3 DoAs
Figure 7, CDF for 4 DoAs
speech
Predicted Angles
[Multi-DeA speech so
Curuuaatvv Probabiity
Cumulativv Probabilty
0.4
0.4
0.2
0.7
CBr Absplote
CMF Fimms
Mediar From
inaa ./'0"
moon emonn 25.47" 50th Percontle: 34.00*
Median - 13.00*
idegrees)
(degrees)
Figure 8, CDF for 5 DoAs
Figure 9. CDF for all speech sources
Figure 10. CDF plots of absolute errors for different numbers of speech sources (DoAs). Mcan and median errors are highlighted for comparison.
16

---

SING: Spatial Context in Large Language Model for Next-Gen Wearables
I. Prompt Details
To evaluate the capabilities of our Spatial-Aware ASR system, we design various prompts corresponding to different speech processing tasks. These prompts facilitate direction estimation, speaker counting. and transcription summarization, enabling more comprchensive understanding of spatially distributed speech signals.
Type
System Prompts
Prompts
Response Templates
ASR (Whisper)
You are an assistant that pro- vides the summarization of the speech. Respond with the the correct summariza- tion.
What is the summarization of the speech?
Mr. Swift's eyes twinkled.
Spatial ASR (DoA Whis- per)
You are an assistant that pro- vides the direction of the speech in degrees and sum- marization of the speech. Respond with the exact an- gle degrees and the cor-
What is the direction of speech in degrees and sum- marization of the speech?
The speaker is speaking ap- proximately at angle 217 de- grees. The speech says: A low, deep moan broke from him.
rect summarization.
DoA Detection
You are assistant that identifies the direction of sound. Respond with the ex- act angle degrees.
What is the direction of the speech source?
The speech is coming from 45 degrees.
Number of Speakers
You are assistant that identifies how many people are speaking at the same time. Respond with the ex- act numbers.
How many people are speak- ing?
There are 3 speech sources.
Multi-DoA Detection
You are assistant that identifies how many people are speaking at the same time and the directions of speech. Respond with the exact angle in degrees.
How many people are speak- ing? What are the directions of the sound sources?
There are 2 speech sources. Speech source I's degree of arrival is 39.0 degrees, and speech source 2's degree of arrival is 101.0 degrees.
Table 8. Overview of different prompt types and their corresponding response templates in the Spatial-Awar ASR system. The system is capable detecting speech direction, counting number of speakers, and summarizing transcriptions.
J. Performance of Number of Speakers Estimation
We show the performance of number of speakers estimation from the num of speaker encoder. Figure 11 shows the normalized confusion matrix for the number of speakers' estimation task, represented as ratios. Fach cell indicates the proportion of predictions for given actual class, normalized by the total number of samples in that class. The diagonal values highlight the model's accuracy in correctly identifying the number speakers, with values close to indicating high precision. Off-diagonal values represent misclassifications, demonstrating the model's tendency to confuse certain classes. For example, the model exhibits strong performance for classes 1, 2, and 5, while class shows more confusion with class 5, reflecting the challenges in differentiating between these scenarios.
K. Spectrogram of Recordings from Directions
Figure 12 shows the difference between original speech and speech signals coming from angles, captured by the microphone built in microstructure. The differences the spectrograms are evident in the amplitude and frequency distributions, which
17

---

SING: Spatial Context in Large Language Model for Next-Gen Wearables
Confusion Matrix (Ratios)
1.00
0.00
0.00
0.00
0.00
0.8
0.00
0.07
0.00
0.00
0.6
Actual
0.00
0.01
0.91
0.08
0.00
0.4
0.00
0.00
0.02
0.54
0.44
0.02
0.0
Predicted
Figure 1. Confusion matrix for number of speakers estimation.
vary significantly depending on the Direction of Arrival (DoA). For example, angles 244 and 163", the spectrograms display distinct patterns in the intensity and distribution of energy across frequencies compared to the original speech, These variations are introduced by the microstructure 's unique modulation of sound waves, which embeds -ppccffc characteristics into the captured audio.
Original Speech
Convoluted Speech (Angle 244")
Convoluted Speech (Angle 163')
Mei-Spectrogram
Mei- Sal--peetrogrmm
Mei- Spectrorrar
Frequency (M4)
Frequiccy (H40
1400
60
15
60
00
Time 3g)
(0)
(9)
Figure 12. Comparison between original speech and speech captured from directions.
L. Ablation Study for DoA Encoder Design
To show an ablation study of our DoA encoder design, we compared the performance designed DoA encoder that has 3-layer CNN and transformer-bas based DNN encoder as shown in table This transformer-bast based DNN encoder was trained using the same architecture Audio Spectrogram Transformer (Gong al., 2021), which is popular transformer architecture for audio. The mean DoA errors of CNN-based DoA encoder and transformer-based encoder are 11.00 and 17.08 respectively, showing the effectiveness of CNN-based encoder.
Model SING
Mean DoA Error (*)
25.72
3-layer CNN
11.00
Audio Spectrogram Transformer
17.08
Table Mean DoA error for evaluated models.
18

---

SING: Spatial Context in Large Language Model for Next-Gen Wearables
CDF of Predicted True Angles
1.0
0.8
Empirical CDF
0.6
0.6
0
0.4
0.2
0.0
20
40
60
80
Error (Degrees)
(degrees)
Figure 13. CDF for SING
Figure 14. CDF for Owlet
M. CNN-based Owlet Model
Figure 13 and figure 14 shows the comparison of SING and Owlet model. Note that exact comparison infeasible as Owlet processes the audio signal differently and does not attempt to retain in speech information. SING not only supports DoA estimation, but also retains speech features in the audio.
N. Embedding Instead of Numerical Values for Spatial Information
While the proposed approach using explicit numerical values (e.g.. "120"") combined with transcribed speech and a prompt to generate structured outputs is viable for single-speaker scenarios, faces significant scalability and robustness limitations multi-speaker environments. In particular, multi-DoA scenarios involve multiple overlapping speakers and corresponding DoAs. Concatenating raw numbers crafting ad-hoc prompts for each speaker-direction pair becomes increasingly brittle and complex as the number of speakers increases. Embedding-basee based representations more scalable and generalizable, allowing us to encode both directional and contextual information in unified vector space that the LLM can attend to and reason about. This enables the system to handle variable-length inputs, concurrent speakers, and noisy observations more robustly.
O. Latency, Power, and Memory for Real-time On-device Usage
Our edge-cloud hybrid approach processes DoA and speech embeddings on-device, then transmits these small embeddings the cloud for LLM inference. For comparison, the Whisper-tiny model uses 273MB B of memory, 75M of disk usage, and < 500ms latency. This model has been deployed on mobile devices (Gerganov, 2023). Our DoA encoder is considerably lighter: it achieves an average latency of 62.93ms per speech file. Memory usage per speech file is 50M B, and drops to 16M when using the quantized model. During full inference, the overall memory footprint is 741.42M B. These results demonstrate that our method is well-suited for on-device processing in wearable or embedded systems, where both memory and power are constrained. Table 10 summarizes the latency, memory, and disk requirements for real-time on-device processing of speech embeddings, alongside the Whisper-tiny baseline.
System/ Model
Latency (ms/file)
Memory (MB)
Disk (MB)
Full Inf. Footprint (MB)
Notes
Whisper-tiny
<500
273
75
Deployed on mobile devices
DoA Encoder (full precision)
62.93
50
741.42
On-device embedding cloud LLM pipeline
DoA Encoder (quantized)
62.93
16
Quantized on-device model
Table 10. Latency, memory, and disk requirements for real-time on-device processing of speech embeddings (compared Whisper-tiny).
19


Reviews:
[{"review_id": "59nyuaDq7A", "reviewer": "Reviewer_o4uR", "summary": "This paper introduces SING, a system that integrates spatial speech understanding into LLMs for wearable applications. It leverages microstructure-based spatial sensing to extract Direction of Arrival (DoA) information using a single monaural microphone. Spatial cues are fused with Whisper embeddings and aligned to LLaMA-3.2 3B using a linear projection. The system achieves a DoA mean error of 25.72° (vs. 88.52° in prior work) and a WER of 5.3%. The work has potential applications in spatially-aware ASR, accessibility, and augmented reality.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1: Lacks real-world dataset validation. Results are based on synthetic data.**\n\nWe used real-world impulse responses measured from calibration of the acoustic frontend and followed standard practices from acoustic physics principles. This methodology enables creation of a diverse, controlled dataset while preserving real acoustic properties. Similar approaches are common in spatial audio research, such as the references below. We've verified key components of the data in real-world settings, showing strong correlation between synthetic and real-world performance. We will update the paper to better highlight our validation methodology and include additional real-world test cases from the AMI dataset to demonstrate generalizability.\n\nRishabh et al. \"Wave field synthesis: The future of spatial audio.\" IEEE Potentials 32, no. 2 (2013): 17-23.\n\nVille et al. \"Spatial sound scene synthesis and manipulation for virtual reality and audio effects.\" Parametric Time‐Frequency Domain Spatial Audio (2017): 347-361.\n\n**2: Linear projection may limit generalization across environments.**\n\nOur current evaluation contains results for the model's resilience to new environments (Appendix E). We selected a linear projection over attention-based fusion primarily for efficiency on wearable devices. Our tests across varied acoustic conditions demonstrate that this approach generalizes effectively despite its simplicity. We will better highlight these generalization results in the revised paper to address this concern.\n\n**3: No benchmarks on latency, power, or memory for real-time on-device use.**\n\nWe have conducted benchmark experiments and will add our results to the paper. Our edge-cloud hybrid approach processes DoA and speech embeddings on-device, then transmits these small embeddings to the cloud for LLM inference. \n\nFor comparison, Whisper-tiny model uses **273MB** memory, **75 MiB** disk usage, and **<500ms** latency. This model has been deployed on mobile devices. Please find the git repo below. Our DoA encoder is considerably lighter: it achieves an average latency of **62.93 ms** per speech file. Memory usage per speech file is **50 MB**, and drops to **16 MB** when using the quantized model. During full inference, the overall memory footprint is **741.42 MB**. These results demonstrate that our method is well-suited for on-device processing in wearable or embedded systems, where both memory and power are constrained. We will provide a summary table of latency and memory footprints in comparison to other works with similar scope as ours.\n\nhttps://github.com/ggerganov/whisper.cpp\n\n**4: No quantitative analysis of model robustness under speaker movement or occlusions.**\n\nApproaches for spatial analysis of sound in human-centric application consider low dynamicity of the source source. SING is designed for reasonably slow-moving environments where acoustic conditions don't change significantly within short timeframes. On the other hand, our current framework shows promising performance with static occlusions, with results shown in Table 6. For future work, we plan to handle fast moving speaker scenarios, dynamic tracking, and Doppler frequency shifts.\n\n**5: Omits comparisons with beamforming and HRTF-based methods.**\n\nPrimary focus of the current work has been enabling spatial awareness in ultra-compact form factors where traditional beamforming methods cannot operate due to physical constraints (requiring multiple spatially-separated microphones). While HRTF methods offer complementary benefits, they typically require binaural setups not suited for our target monaural applications. We cite relevant papers on these techniques and clarify these distinctions in the revised paper.\n\n**6: Deep-learning-based spatial ASR (e.g., SELDNet, SoundSpaces) is missing.**\n\nWe compared our method with SELDNet in Table 2, showing SING achieves a 25.72° MAE versus SELDNet's 90.03° MAE, demonstrating superior performance despite using fewer microphones. SoundSpaces is primarily an acoustic simulation platform rather than a competitive DoA system. We will revise the comparison section to better highlight these results.\n\n**7: Does not evaluate closely spaced speakers (<10° separation).**\n\nOur hardware demonstrates a 3° median error in controlled settings, indicating capability for fine-grained angular resolution. The 10° separation in our experiments was chosen for common human-centric scenarios. While the system can resolve speakers at closer angular separations, we focused on realistic use cases for initial evaluations.\n\n**8: Can SING be adapted to estimate elevation angles (3D spatial ASR)?**\n\nYes, SING can be extended to estimate elevation angles with minimal changes. While the current hardware is optimized for azimuth, the architecture supports 3D estimation if impulse responses are calibrated for elevation and the hardware is updated. We plan to explore this in future work.", "claims_and_evidence": "Claim: SING significantly improves DoA estimation using a monaural microphone.\n- Evidence: Achieves a mean error of 25.72° vs. 88.52° in BAT.\n- Limitation: Lacks real-world dataset validation; results are based on synthetic data.\n\nClaim: The system enables spatially-aware ASR.\n- Evidence: Demonstrates 5.3% WER with spatial embeddings.\n- Limitation: Linear projection may limit generalization across environments.\n\nClaim: Optimized for wearable devices.\n- Evidence: Uses LoRA for efficient fine-tuning.\n- Limitation: No benchmarks on latency, power, or memory for real-time on-device use.", "methods_and_evaluation_criteria": "Uses OmniTalk (synthetic dataset), derived from LibriSpeech, to simulate spatial speech.\nEvaluates DoA estimation and ASR performance but lacks real-world testing.\nComparison with BAT and SALMONN is valid but excludes state-of-the-art beamforming methods.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "DoA experiments are well-structured but rely on synthetic impulse responses.\nEvaluations on background noise and reverberation are limited to GTU-RIR dataset, lacking real-world generalization tests.\nNo quantitative analysis of model robustness under speaker movement or occlusions.", "relation_to_broader_scientific_literature": "Extends prior work in spatial ASR, multimodal LLMs, and monaural DoA estimation.\nCites BAT (spatial audio LLMs) and Whisper, but omits comparisons with beamforming and HRTF-based methods.", "essential_references_not_discussed": "Traditional microphone array DoA methods (e.g., MUSIC, GCC-PHAT) should be compared.\nDeep-learning-based spatial ASR (e.g., SELDNet, SoundSpaces) is missing.", "other_strengths_and_weaknesses": "Strengths\n\n1. Novel approach: First LLM-based spatial ASR using a monaural microphone.\n2. Efficient design: Uses LoRA for low-power adaptation.\n3. Strong performance: Outperforms BAT in DoA estimation.\n\nWeaknesses\n\n1. Linear projection limits generalization to unseen conditions.\n2. No real-world dataset evaluation; performance may degrade in natural environments.\n3. Wearable feasibility unverified; lacks latency and power benchmarks.\n4. .Does not evaluate closely spaced speakers (<10° separation).", "other_comments_or_suggestions": "NA", "questions_for_authors": "1. How does the model perform on real-world datasets, not synthetic data?\n2. What are the latency and memory requirements for on-device inference?\n3. How does SING handle occlusions or moving speakers?\n4. Why was a simple linear projection chosen over attention-based fusion?\n5. Can SING be adapted to estimate elevation angles (3D spatial ASR)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2zJ83pDtLV", "reviewer": "Reviewer_wmgF", "summary": "In this paper, the authors introduce SING, a system that integrates spatial speech understanding into LLMs to enhance context-aware applications for wearable devices. SING uses microstructure-based sensing with a monaural microphone to extract Direction of Arrival  information and combines it with linguistic embeddings from Whisper model. The fused embeddings are aligned with LLaMA-3.2 3B and fine-tuned using LoRA for efficient on-device processing. SING achieves improved ASR performance with a 25.72° mean DoA error and 5.3% WER. It also support multi-speaker detection with a median DoA error of 16°, demonstrating good performance under power, privacy, and hardware constraints for applications like AR and accessibility. Overall, there are both algorithm and system design in this paper.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comment 1: I suggest the authors to introduce or define what “spatial” or “spatial feature” is in the early of the Introduction section.**\n\nWe appreciate the reviewer’s insightful comment. We agree that the term “spatial” can be interpreted in multiple ways depending on the context. To clarify our intended meaning, we will revise the Introduction section to explicitly define “spatial features” in the context of our work as acoustic cues that convey directional or positional information of sound sources. We will also add a brief comparison to distinguish our usage from other interpretations of “spatial” such as geographical or kinematic contexts. \n\n**Comment 2: It comes to me that we can use a easier way to achieve the same output , e.g., \"who is talking about what at which direction\". For example, we design a simple method to detect the direction of the talking person and get x, which is a numerical value such as 120 degree. Then, combine x with the detected text language and give it to the LLM, with some kind of prompt design, the LLM will generate a similar output, \"who is talking about what at which direction. \" as shown in Fig.2.**\n\nWhile the proposed approach of using explicit numerical values (e.g., \"120°\") combined with transcribed speech and a prompt to generate structured outputs is viable for single-speaker scenarios, it faces significant scalability and robustness limitations in multi-speaker environments.\n\nIn particular, multi-DoA scenarios involve multiple overlapping speakers and corresponding DoAs. Concatenating raw numbers or crafting ad-hoc prompts for each speaker-direction pair becomes increasingly brittle and complex as the number of speakers increases. Embedding-based representations are more scalable and generalizable, allowing us to encode both directional and contextual information in a unified vector space that the LLM can attend to and reason about. This enables the system to handle variable-length inputs, concurrent speakers, and noisy observations more robustly.\n\nWe have added a discussion of this design in the camera-ready version.\n\n**Comment 3: In the Introduction, the authors mention “the mean error rate (MAE) is 88.52°, too high for real-world applicaitons”. Appropriate contexts could be given here to help the readers. Otherwise, regular readers have no idea what difference will it make if reduce from 88.52 to 8.52.**\n\nWe thank the reviewer for the thoughtful suggestion. We will revise the Introduction section to include specific real-world application contexts where accurate DoA estimation is critical, and where an MAE of 88.52° would be insufficient.\n\nFor example:\n\n- In a meeting scenario, an error of 88.52° would result in attributing spoken content to the wrong person sitting across the table, severely degrading the reliability of voice-based summarization features in AR glasses or virtual assistants.\n\n- In accessibility tools for the visually impaired, such as acoustic navigation or sound-based obstacle detection, such a large directional error could cause the user to orient toward the wrong direction, compromising safety and usability.\n\n- In smart home systems, identifying which room or device a user is speaking from (e.g., turning toward a speaker to answer a question) would fail if the estimated direction is off by nearly 90°, leading to incorrect command execution.\n\n- In immersive AR/VR applications, spatial misalignment of that magnitude would result in audio-visual desynchronization, breaking the immersion and confusing the user.\n\n**Comment 4: I acknowledge that comparing with baselines might be not easy given the speciality of this topic. The authors could provide more justifications about this, otherwise, it seems the baseline models are too limited.**\n\nWe appreciate the reviewer’s comment and agree that rigorous comparison with baseline methods is essential. To address this, we have carefully selected and implemented a widely used and fundamentally distinct traditional DoA estimation technique MUSIC algorithm. To ensure fairness, we applied two algorithms on same size of two-microphone array under matched conditions and identical SNR levels. SING significantly outperforms MUSIC algorithm in terms of the cumulative distribution of angular errors, with 3.6 degree median error for SING and 15.2 degree median error for MUSIC algorithm. This comparison provides a comprehensive and rigorous baseline evaluation. We will also add this result in the paper.\n\n**Comment 5: When introducing \"IoT sensor\" for the first time, it should be written as \"Internet of Things (IoT)\" in full.**\n\nThanks for pointing this out. We will use Internet of Things (IoT) at the first time we mention IoT.\n\n**Comment 6: In the last few sentences of the Vision paragraph, Introduction Section, some references could be added to support the potential applications.**\n\nThanks for the valuable suggestion. We will cite relevant references to support the potential applications.", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "not applicable", "experimental_designs_or_analyses": "designs are ok", "relation_to_broader_scientific_literature": "This method has potential applications in wearable computing, which is a hot topic.", "essential_references_not_discussed": "Some references can be added in the Vision section, in the introduction. I have mentioned the details below.", "other_strengths_and_weaknesses": "Overall, I enjoyed reading this paper because the research topic is interesting and has broader impact potential. \n\n\nHowever, I also have some concerns: \n\n[1] In the last few sentences of the Vision paragraph, Introduction Section, some references could be added to support the potential applications. \n\n[2] I suggest the authors to introduce or define what “spatial” or “spatial feature” is in the early of the Introduction section. For readers with other backgrounds, it might be confusing, e.g., spatial could mean geographical knowledge, or human movement in the 3-d environment. And I suppose those kinds of spatial context has been explored for a while by multiple studies. \n\n[3] When introducing \"IoT sensor\" for the first time, it should be written as \"Internet of Things (IoT)\" in full. \n\n[4] In the Introduction, the authors mention “the mean error rate (MAE) is 88.52°, too high for real-world applicaitons”. Appropriate contexts could be given here to help the readers. Otherwise, regular readers have no idea what difference will it make if reduce from 88.52 to 8.52. \n\n[5] I acknowledge that comparing with baselines might be not easy given the speciality of this topic. The authors could provide more justifications about this, otherwise, it seems the baseline models are too limited. \n\n[6] About the output of the model, I also have some concerns, correct me if I am wrong. It comes to me that we can use a easier way to achieve the same output , e.g., \"who is talking about what at which direction\". For example, we design a simple method to detect the direction of the talking person and get x, which is a numerical value such as 120 degree. Then, combine x with the detected text language and give it to the LLM, with some kind of prompt design, the LLM will generate a similar output, \"who is talking about what at which direction. \" as shown in Fig.2.", "other_comments_or_suggestions": "please see above", "questions_for_authors": "please see above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Rn26xM5rFo", "reviewer": "Reviewer_t6BY", "summary": "The paper leverages the Owlet monaural microphone, with superior direction-of-arrival sensing (DoA), to endow LLMs with spatial audio awareness towards more intelligent wearables and other usability scenarios.  To achieve this, the authors prepare synthetic variants of the LibriSpeech dataset with ground-truth source direction, train lightweight speech and DoA encoders, align the obtained speech embeddings into the LLMs input space, and apply supervised instruction fine-tuning with LoRA.  The approach is compared to related works including BAT, SALMONN, SELDNet, and AudioMAE, demonstrating significant improvements for the tasks of spatially-aware ASR and soundscaping.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comment 1: 0-speaker case should be considered.**\n\nThank you for pointing this out. In the current version of Table 2 and Figure 11, we focus on scenarios with one or more speakers. However, we agree that it would be valuable to explicitly consider the 0-speaker case. To address the 0-speaker case, we plan to incorporate a dedicated speech activity detection module, similar to those used in voice activity detection (VAD) systems such as the referenced papers below, which can detect whether any speakers are active before estimating their count or direction.\n\nRamırez, Javier, José C. Segura, Carmen Benıtez, Angel De La Torre, and Antonio Rubio. \"Efficient voice activity detection algorithms using long-term speech information.\" Speech communication 42, no. 3-4 (2004): 271-287.\n\nBall, Joshua. \"Voice Activity Detection (VAD) in Noisy Environments.\" arXiv preprint arXiv:2312.05815 (2023).\n\n**Comment 2: Add a tutorial for readers without signal processing background.**\n\nWe'll add a citation to a beginner-friendly tutorial and textbooks, such as Jurafsky & Martin or an online spectrogram/ASR tutorial, to help readers without a signal processing background, especially before Eq. 1 and in §3.2. Also including STFT and resampling in this tutorial.\n\n**Comment 3: Frame of reference the 360 degrees should be understood.**\n\nThe 360° azimuth angles are defined in a canonical frame relative to the sensor hardware, assuming the device is in an upright orientation. In our current setup, we align the sensor's forward-facing direction with 0°, and angles increase clockwise in the horizontal (azimuthal) plane. Traditionally werable systems rely on the embedded IMU to determine the sensor's orientation and, when needed, apply a standard coordinate transformation to convert angles from the sensor frame to the global frame. We will include a discussion on this topic in the revised paper.\n\nhttps://qsense-motion.com/quaternion-orientation-imu-sensor-fusion/\n\n**Comment 4: Other details revisions.**\n\nThank you for these detailed suggestions!\n\n- You’re right, “hidden states” in §4.2 refers to the intermediate latent representations output by the model layers. We’ll clarify the terminology to avoid confusion.\n\n- Thanks for the comment regarding §6.2. We’ll revise “20 minutes/epoch” to explicitly mention the total number of epochs in the main text for clarity, even though it's listed in Table 3 (Appendix B).\n\n- Yes, we will replace “lock the LLM” with “freeze” for correctness and consistency with standard terminology.\n\n- We will correct “trim and pad” to “trimmed and padded.” Thanks again for point this out.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes.  Given the lack of suitable datasets, synthetic data was generated following principled techniques from signal processing ($3.2).", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes.  Read through the experiments in S6 and S7.", "relation_to_broader_scientific_literature": "Good demonstration of the potential of integrating innovative hardware designs into advanced AI systems.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The paper is very well-written and easy to follow, making it a good example for subsequent works.  Good job!", "other_comments_or_suggestions": "- In both Table 2 and Figure 11, I wondered if the 0-speaker case should be considered?  Please clarify.\n- It would help to add a citation to a friendly text or tutorial for readers without the signal processing background.  This seemed needed before Eq. 1 and upon mentioning spectrograms, and even \"ASR\" (S3.2).\n  - Also, STFT wasn't defined before Eq. 4, only later in S6.1 and again in S7.1.\n- It would help to clarify how the resampling operation in Eq. 2 is implemented.\n- It would help to clarify in which frame of reference the 360 degrees should be understood.  Only later in future work there was a discussion of elevation angles.  I assume there's a canonical frame define w.r.t. the sensor hardware, so were datasets generated with the assumption the sensor is \"upright\"?\n\nMore nitpicking\n- (S4.2) Not sure I understand \"hidden states\".  Guessing it's simply meant as \"latent representations\"?\n- (S6.2) It mentions 20 minutes / epoch, so I was wondering how many epochs?  I see it's listed in Table 3 under Appendix B.\n- (S6.2) we first lock the LLM --> freeze?\n- (S7.1) final recording is trim and pad to 8 seconds -> trimmed and padded?", "questions_for_authors": "No further questions at this time.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pDXhEo3KNf", "reviewer": "Reviewer_mTjc", "summary": "This paper proposes a method for aligning spatial audio sensing to LLM embeddings. The approach includes CNN-based DoA estimation module and automatic speech recognition module from OpenAI. The paper introduces an OmniTalk dataset that is synthetically generated to train the module. The experiment evaluated the method with DoA and ASR.\n\n## update after rebuttal\n\nI appreciate the author's engagement during the rebuttal process. However, my concerns still remain, and I decide to keep my score.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1: Why we need a spatial positioning instead of speaker diarization.**\n\nThank you for the question. While speaker diarization answers \"who spoke when,\" it lacks spatial awareness—crucial for many applications. Our focus on Direction of Arrival (DoA) estimation adds spatial context to speech, enabling new capabilities in AR, robotics, and embodied AI.\n\nFor instance, AR glasses can answer location-based queries like “What did the person to my left say?” Robots can localize and respond to specific speakers, enhancing natural interaction. In meetings, spatial cues help disambiguate overlapping speech and attribute content by location.\n\nThough diarization is useful, it doesn’t ground speech in space. We see strong potential in combining it with DoA estimation to support both spatial and identity-aware speech understanding, which we aim to explore in future work.\n\n**2: The solution is just using Owlet framework, could not be claimed as own contributions.**\n\nWhile Owlet provides a powerful foundation, our work takes it one step further by integrating it with an LLM-based reasoning module, enabling new capabilities such as open-ended spatial understanding and dialogue-based interaction, which were not possible in the original framework.\n\n**3: The experiment should include the CNN-based model from Owlet as a baseline for DoA estimation.**\n\nThank you for the suggestion. While the CNN-based model from Owlet is effective for single-source DoA estimation, our work advances beyond that by estimating multiple simultaneous DoAs and, more importantly, enabling an LLM to understand and reason about these spatial cues in context, paving the way for open-ended spatial dialogue rather than just angle prediction. We agree it’s a valuable baseline and will include the Owlet CNN model in our experiments for comparison.\n\n**4: Unclear why the authors generated an additional dataset (OmniTalk) instead of utilizing Owlet for data collection.**\n\nOwlet's original dataset was designed for generic audio with only a single sound source, whereas our goal in OmniTalk is to model **speech-specific**, **multi-speaker scenarios** with up to 5 overlapping sources. While OmniTalk is simulated, it is built using impulse responses generated by the Owlet hardware, ensuring consistency with real-world spatial characteristics. We will highlight it in the paper.\n\n**5: DoA with speech ASR results in twice lower performance than the baseline (SALMONN). This questions the need for spatial awareness.**\n\nThe drop in performance is due to Whisper's sensitivity to spatial structure distortion in multi-source scenarios. To address this, we fine-tuned Whisper on spatialized speech mixtures, which led to a notable improvement in ASR accuracy—narrowing the gap with the SALMONN baseline and demonstrating that incorporating spatial awareness can indeed enhance robustness in challenging, multi-speaker environments. We will include the updated ASR results in the revised manuscript.\n\n**6: The supplementary material includes the analysis of the performance of estimating the number of speakers. However, the experiment details are not specified, and only the result of a single 5-speakers experiment is reported.**\n\nWe will include detailed experiment settings for speaker count estimation in the camera-ready version. The results for 1 to 5 speakers are already reported in Table 2 and Figure 10.\n\n**7: The paper just lists the methodologies from various related works to build SING. For example, it is not properly motivated why we need a mel-spectogram and 3x3 CNN structure.**\n\nWe use mel-spectrograms and 3×3 CNNs due to their proven effectiveness in speech tasks. Mel-spectrograms offer a perceptually meaningful time-frequency representation, widely used in ASR, localization, and classification. The 3×3 CNN, inspired by VGG, efficiently captures local spectrogram patterns. While SING builds on these established components, our key contribution is combining them with spatial supervision and LLM reasoning to jointly estimate and interpret spatial properties in natural language.\n\n**8: The current experiment does not properly reflect the superiority of the method for “aligning with LLMs” by evaluating only DoA/ASR with insufficient baselines.**\n\nOur evaluation focuses on aligning spatial audio with LLM reasoning, not just DoA or ASR. Tasks like spatial QA, location-aware summarization, and speaker counting test whether the LLM can interpret spatial cues in context. While traditional benchmarks ground the system, our goal is to assess how well spatial understanding supports reasoning and interaction. We agree that adding more baselines would strengthen the evaluation and plan to expand them in future work.\n\n**9: Figure 2 is slightly confusing - is the figure representing a supervised fine-tuning phase, considering DoA is already pre-trained?**\n\nYour understanding is correct. We will update the figure and writing to make it clearer.", "claims_and_evidence": "- Introduction-A: The paper claims the use of “micro-structure assisted miniature” as a challenge & approach for the proposed problem. However, the solution is just using Owlet (Anonymous, 2021) framework, thus could not be claimed as  own contributions.\n\n- Section 2.3: The paper necessitates spatial positioning to distinguish multiple speakers. Why do we need a spatial positioning instead of speaker diarization techniques [1, 2]?\n\n[1] Chang, Xuankai, et al. \"End-to-end monaural multi-speaker ASR system without pretraining.\" ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019.\n\n[2] Fujita, Yusuke, et al. \"End-to-end neural speaker diarization with self-attention.\" 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2019.", "methods_and_evaluation_criteria": "The experiment should include the CNN-based model from Owlet as a baseline for DoA estimation.\n\nIt is unclear why the authors generated an additional dataset (OmniTalk) instead of utilizing Owlet for data collection. How accurately does a simulated dataset reflect Owlet?\n\nIt is unclear if the experiment is properly designed to evaluate the “alignment to LLMs,” considering the main evaluation metrics are DoA and ASR.\n\nTable 2 does not report any multi-source comparison baselines. Would it be possible to compare with speaker diarization techniques?", "theoretical_claims": "There are no significant theoretical claims in the paper.", "experimental_designs_or_analyses": "The experiment shows that DoA with speech ASR results in twice lower performance than the baseline (SALMONN). This questions the need for spatial awareness.", "relation_to_broader_scientific_literature": "The paper is related to spatial speech understanding, speaker diarization, and aligning spatial speeches for language models. The paper improves the use of Owlet to enable monaural spatial speech sensing.", "essential_references_not_discussed": "The paper does not discuss any speaker diarization techniques [1, 2].\n\n[1] Chang, Xuankai, et al. \"End-to-end monaural multi-speaker ASR system without pretraining.\" ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019.\n\n[2] Fujita, Yusuke, et al. \"End-to-end neural speaker diarization with self-attention.\" 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2019.", "other_strengths_and_weaknesses": "Weaknesses:\n\n- Vague positioning: the paper necessitates spatial positioning to distinguish multiple speakers. Why do we need spatial positioning instead of speaker diarization?\n\n- Technical novelty: the paper just lists the methodologies from various related works to build SING. For example, it is not properly motivated why we need a mel-spectogram and 3x3 CNN structure.\n\n- Limited scalability: The multi-DoA encoder is hard-coded with 5 speakers and lacks discussion of its scalability. Also, the performance of DOA + speech ASR is lower than the baseline.\n\n- The experiment should be improved: The current experiment does not properly reflect the superiority of the method for “aligning with LLMs” by evaluating only DoA/ASR with insufficient baselines.", "other_comments_or_suggestions": "Please update the running title (+ current format is not the double-blind format).\n\nWhy is the Owlet (Anonymous, 2021) anonymized? They are already published in the conference proceedings.\n\nTypo: LLava → LLaVA (Page 2).\n\nFigure 2 is slightly confusing - is the figure representing a supervised fine-tuning phase, considering DoA is already pre-trained?", "questions_for_authors": "The paper necessitates spatial positioning to distinguish multiple speakers. Why do we need spatial positioning instead of speaker diarization? Also, Table 2 does not report any multi-source comparison baselines. Would it be possible to compare with speaker diarization techniques?\n\nWhy did the authors generate an additional dataset (OmniTalk) instead of utilizing Owlet for data collection? How accurately does a simulated dataset reflect Owlet?\n\nWhy did the experiment not include the CNN-based model from Owlet as a baseline for DoA estimation?\n\nWhy are the main evaluation metrics DoA and ASR if the main objective aligns with LLMs?\n\nThe experiment shows that DoA with speech ASR results in twice lower performance than the baseline (SALMONN). Then, what if we combine SALMONN with DoA estimation?\n\nHow is the method scalable with the number of speakers?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: After rebuttal, the paper received three week accepts and one weak reject. In general, the reviewers appreciated the novelty, solid integration of spatial sensing with LLMs, and promising results. The remaining concerns raised were about the limited ablation, reliance on synthetic data, and unclear need for LLM alignment over simpler modular baselines. The rebuttal addressed many of these issues, and in general, the merits of this paper can outweigh the weaknesses.
Decision: Accept (poster)