Teaching Physical Awareness to LLMs through Sounds
Authors: Weiguo Wang
Andy Nie
Wenrui Zhou
Yi Kai
Chengchen Hu
Venue: ICML 2025 poster (2025)
Keywords: LLM
Physical World Understanding
Audio
Spatial Audio
Acoustic Channel Simulation
Primary area: deep_learning->large_language_models
TL;DR: This paper explores and validates the feasibility of teaching LLMs to understand the physical world through sounds

Abstract: Large Language Models (LLMs) have shown remarkable capabilities in text and multimodal processing, yet they fundamentally lack physical awareness--understanding of real-world physical phenomena.
In this work, we present ACORN, a framework that teaches LLMs physical awareness through sound, focusing on fundamental physical phenomena like the Doppler effect, multipath effect, and spatial relationships. To overcome data scarcity, ACORN introduce a physics-based simulator combining real-world sound sources with controlled physical channels to generate diverse training data. Using this simulator, we build AQA-PHY, a comprehensive Audio Question-Answer dataset, and propose an audio encoder that processes both magnitude and phase information. By connecting our audio encoder to state-of-the-art LLMs, we demonstrate reasonable results in both simulated and real-world tasks, such as line-of-sight detection, Doppler effect estimation, and Direction-of-Arrival estimation, paving the way for enabling LLMs to understand physical world.

Teaching Physical Awareness to LLMs through Sounds
Weiguo Wang' Andy Nie
Wenrui Zhou'
Yi Kai Chengehen Hu'
Abstract
in an enclosed space or outdoors; and binaural hearing al- lows to pinpoint the sound direction. Apparently, these acoustic cues provide crucial information about physical dynamics and spatial relationships.
Large Language Models (LLMs) have shown re- markable capabilities in text and multimodal pro- cessing, yet they fundamentally lack physical awareness-understaddig of real-world physical phenomena. In this work, we present ACORN, a framework that teaches LLMs physical awareness through sound, focusing on fundamental physical phenomena like the Doppler effect, multipath ef- fect, and spatial relationships. To overcome data scarcity, ACORN introduce a physics- -bseed sim- ulator combining real-world sound sources with controlled physical channels to generate diverse training data. Using this simulator, we build AQA- PHY, comprehensive Audio Question-Answe Answer dataset, and propose audio encoder that pro- cesses both magnitude and phase information. By connecting our audio encoder to statc-of-the-ar LLMs, we demonstrate reasonable results in both simulated and real-world tasks, such as line-of- sight detection, Doppler effect estimation, and Direction-of-AAn Arriiaal estimation, paving the way for enabling LLMs to understand physical world.
Despite their ability to recognize and generate speech, cur- rent LLMs can not comprehend the physical properties of sound. This gap poses significant challenges in real- world applications. For example, voice-controlled vehicle might accept command such as Open the Window from someone outside the car, introducing security risk due to its inability to discern the physical origin of the voice. Also, in human-robot interaction, embodied Al systems without physical awareness fail to imitate human behaviors, such as turning towards the speaker when addressed.
In this paper, we take an initial step toward bridging this gap by proposing ACORN (Acoustic ing Network), framework that teaches LLMs to develop physical awareness through sound. ACORN equips LLMs with both passive and active acoustic sensing capabilities: Through passive sensing, L.LMs can detect line-of-sight (LOS) signal paths, interpret motion via Doppler effects, lo- calize sound sources, and analyze multipath effects. With ac- tive sensing, we extend their capabilities to include distance measurement by generating pulses and analyzing echoes, akin to sonar systems. We demonstrate that LLMs empow- ered by ACORN can not only understand the real world via sound but also interact with it.
Reason-
1. Introduction
Large Language Models (LLMs) have demonstrated remark- able capabilities in processing and generating text and have successfully expanded into multimodal understanding. How- ever, a fundamental limitation remains: LLMs lack physical awareness ability to understand and meaningfully inter- act with the physical world.
However, significant challenge arises in this endeavor: How collect and annotate large-scale dataset that cap- tures diverse physical phenomena? (1) Data collection re- quires extensive deployment of recording devices across var- ious environments and conditions, making it prohibitively expensive and impractical. (2) Data annotation is nearly infeasible because, unlike text or images where humans can directly annotate content, audio physical phenomena (like Doppler effects multipath reflections) can not be labeled by human alone and often requires sophisticated measurement equipment.
Physical awareness, particularly through sound, is comer- stone of how humans understand the environment, Sound provides rich, instinctive insights into our surroundings: The Doppler effect, with its frequency shifts, tells us whether a vehicle is approaching or receding; The multipath effect, where sound reflects off surfaces, reveals whether we are
'NIO Peking University. Correspondence to: Weiguo Wang <wangwg.wwg wwanggg.wwg@gmmii.ccm gmai..com>.
Our key insight to tackle this challenge is that: The sound that we hear or microphones capture can be decomposed into two independent components-Ile sound source and the physical channel through which it travels. This decom- position suggests elegant solution: instead of collecting
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Teaching Physical Awareness to LLMs from Sounds
real-world recordings, we can synthesize realistic audio data by combining sound sources from existing datasets with simulated physical channel.
2. Primer: Sound and Space
The received sound is determined by both its source and the environment through which it travels. When we hear a sound-whether through our ears or microphones-wwe are receiving a signal that has been transformed by its propagat- ing space. This transformation can be formally modeled as physical channel filter (Channel Impulse Response, CIR), h. Given a sound source s, the received signal y/ is given by
To this end, we devise a dedicate physical channel simula- tor based on advanced signal processing techniques. This simulator offers complete transparency over the physical channel parameters. It enables precise control over task- specific parameters. Mcanwhile, it allows for the systematic randomization of non-critical parameters, ensuring diverse range of physical channels. By convolving these simulated channels with existing sound sources, this approach facili- tates the building of large-scale, diverse sound datasets with annotated physical phenomena.
= h ( 8,
(1)
where @ denotes the convolution operation.
Leveraging the annotations of the simulated sounds, we cu- rate Question-Answer (QA) pairs that link language with the underlying physical phenomena. To ensure effective align- ment between language and audio, we propose workflow for generating both closed-form and open-form questions and answers that probe the model's understanding of these physical phenomena. In this way, we create AQA-PHY, an Audio QA dataset consisting of 1 million <Audio, Ques- tion, Answer> tuples, designed for supervised fine-tuning of LLMs to endow them with physical awareness.
The human auditory system possesses a remarkable capa- bility: from the received signal y, extracts rich spatial information embedded within the channel h. This channel contains valuable environmental data, capturing how sound waves interact with surfaces and objects in the environment. Through such acoustic information, humans naturally de- velop an understanding of their physical surroundings (Traer & McDermott, 2016).
Drawing inspiration from this innate human ability, we aim to endow LLMs with similar auditory capabilities, enabling them to understand physical world through sound.
To effectively capture physical information from sounds, we also introduce an effective audio encoder architecture. Unlike traditional audio encoders that primarily focus on content recognition based on magnitude information, our design explicitly incorporates both the magnitude and phase components of sounds. This distinction is crucial, physi- cal effects often manifest in subtle phase relationships that magnitude-only encoders may overlook (Bai et al., 2020).
3. Model Architecture
3.1. Overview
Figure I(a) illustrates the model architecture of ACORN. Following established practices the field, we adopt a common end-to-end architecture (Deshmukh et al., 2023; Gong al., 2023; Chu et al., 2023; 2024) to ensure the generality of our approach. The architecture consists of three main components:
We connect our audio encoder to two state-of-the-art L.MMs: ALama3.1-8B and Qwen2-7B. We train the model using AQA-PHY. The model achieves strong results across all tasks: 0.924 accuracy line-of-sight detection, 0.181 MAE in Doppler effect estimation, 0.907 MAE in direction of arrival estimation, 0.903 accuracy in multipath analysis, and 1.599 relative error percentage in range estimation. These demonstrate the fcasibility of teaching LLMs to understand physical phenomena through sound.
(1) an audio encoder, (2) a text encoder, and (3) an LLM. The audio and text encoders transform raw audio and text inputs into sequences of tokens, respectively. These tokens are then concatenated into a unified sequence. Following common practice, we wrap audio tokens with special tokens: <soa>> (start of audio) and <eoa>> (end of audio) establish clear boundaries for the audio content.
In summary, our contributions include:
physics-based channel simulator enabling generation of diverse audios with various physical phenomena.
The concatenated token sequence is then fed into the LLM, which generates a textual response based on both the text and the audio. As illustrated in Figure 1(a), consider a scenario where voice command, Open the Window, is issued from outside a locked vehicle. Through analysis of the raw audio, the model should be able to detect the absence of a LOS path, indicating that the command originates from outside the secured environment of the vehicle. In such cases, the model's physical awareness enables it to reject potentially unsafe commands.
A novel audio encoder capturing both magnitude and phase information for improved physical understanding.
The AQA-PHY dataset containing I million <Audio, Question, Answer> tuples for training LLMs.
Extensive evaluation demonstrating state-of the-art per- formance in both simulated and real-world environments.
2

---

Teaching Physical Awareness to LLMs from Sounds
Audio+ + Text Prompt
Pollow the waice cermanl From the Lapest avdlo
GELU
Audio Encoder
Maguiludde
Encoder
Concatenated
ulfviljum
GELU
Transfomes
T Tokens
canu>
A Audio Tokens
<002>
Text Tolerns
Audio Input
Phase
Position Embedding
Audio Tokenss
Large Language M Model
GELU
Text Response with Phystral A Awareness
(shyysccal- inarerisa>> Los detected, sound source is blocked. hejecttnng commare and as 201a putside the car. c/phusical. onarstios stiess) rejected. glve
Phase (cos)
Audio Spectrum
(a) Model Architecture
(b) Audio Encoder
Figure 1. Model architecture and audio encoder design. (a) Model architecture: Integrates audio and text encoders generate unified token sequences for LLMs, enabling physical reasoning (e.g.. detecting LOS). (b) Audio encoder: Processes magnitude and phase components audio to capture physical phenomena such as Doppler effects and multipath.
3.2. Audio Encoder
The features are then concatenated (3840 channels) and fused through two 3x3 convolutions, reducing the dimen- sionality 1280 channels. Sinusoidal positional embed- dings are added retain temporal context, the outputs processed by a 32-layer transformer, producing the final audio tokens.
To effectively understand complex physical phenom- ena ena-such as multipath and Doppler effects, or determining time differences between audio channels an audio encoder must capture fine-grained features beyond sound sources.
Figure 1(b) illustrates our proposed audio encoder. The key design feature is its ability to incorporate phase information through separate channels for the sine and cosine compo- nents, in addition to capturing the magnitude of the audio spectrum. By including phase information, our encoder can more effectively capture the characteristics of both the sound source and the acoustic channel, which are essential for understanding the physical world (Bai et al., 2020).
3.3. Language Model and Training
Here, the language model is responsible for understand- ing and reasoning about the audio with physical awareness, conditioned by the text prompt. We fuse our audio en- coder to two different L.Mss LLama3-8B and Qwen2-7B, respectively. The audio tokens are projected to the word embedding size of these models via a linear projection layer.
The process first transforms into its spectral components using Short-Time Fourier Transform (STFT). We extract three features from spectral components:
Both the audio encoder and the LLM trained jointly dataset consisting of <Audio, Question, Answer> tu- ples. The training objective is to maximize the likelihood of predicting the next token in the Answer, conditioned on both the Audio and Question inputs. Our audio encoder is initialized from Whisper-large-v2 (Radford et al., 2023) to leverage pretrained magnitude representations, while the phase-related subnetwork trained from scratch to capture fine- fineggained physical cues critical for physical awareness. L.LMs are fine-tuned using LoRA (Hu et al., 2022) to re- duce the training workload and to leverage its linguistic capabilities.
representing the intensity of various fre- quency components, the magnitude (f, t) is given by
Magnitude:
M (f, = (X)
(2)
where ((J,() is the STFT out.
Phase (sin) and Phase (cos): Instead of using the phase angle directly, we compute the sine and cosine of it. This approach is chosen to mitigate numerical instability caused by phase wraparound, occurring when the phase angle transitions from π to - and vice versa. That
4. Channel Simulator
sin (in(((fft)) (f,(LX(fft) t)) (o(.t))))) = (fx(()))
(3)
Training model requires large-scale dataset with anno- tations of physical phenomena. However, comprehensive data collection in the physical world presents fundamental scalability challenges, given the need to reproduce diverse acoustic phenomena under controlled conditions. To tackle this, we leverage a key insight: the received signal can be decomposed into the channel and the sound source s (see Eq. 1). This reveals that channel h independently captures
These help in understanding how the sound wave's phase changes, critical for analysis of acoustic channel.
Each of the three components is processed through a 3x3 ID convolution, expanding from 128 to 1280 channels, fol- lowed by a GELU activation (Hendrycks & Gimpel, 2016).
3

---

Teaching Physical Awareness to LLMs from Sounds
ECHO
Amplltade
LOS
Norm. Amplitule
0.5
Norm. Amplitude
0.5
LOS
Early
Reflections
Long tail
Source
ECHO 2
L.0
ECHO
0.0
0.0
Mic.
ECHO
-0.5
-0.5
15K 20K Time (in sample @48kHz)
5K
10K
0
500
1000 1500 2000
Tie Tachot
Toche
Time Delay
Time (in sample (48kHz)
(a) I.OS and Early Reflections
(b) Simplified CTR
(c) Real-world CTR
(d) Zoomed-in CTR
Figure 2. Illustration of Acoustic Channel.
the physical phenomena, isolating them from source 8.
envelopes to simulate decay. These subbands recom- bined create broadband impulse response. Detailed methodologies are provided in Appendix Section B.2.
Our channel simulator models five main components: (1) LOS paths and (2) early reflections, (3) reverberation. (4)
Doppler effects, and (5) microphone array reception. Each can be independently controlled to generate diverse physical phenomena. For more details, please refer to Appendix B.
By controlling the decay rate of R(T) and selectively in- cluding path components, we can simulate environments ranging from 'dry' spaces with minimal reverberation to those with severe echoes.
4.1. LOS Path and Early Reflections
4.3. Doppler Effect
We begin with LOS paths and early reflections as they form the basic building blocks of sound propagation.
The Doppler effect, a frequency shift resulting from relative motion between the transmitter and receiver, can be modeled by introducing time-varying delay in the CIR.
CIR models how sound source propagates and interacts with its environment before reaching the microphone. As depicted in Figure 2(a), these interactions include a direct path (LOS) and two reflected paths, ECHO and ECHO 2.
Mathematically, for a transmitter and receiver with relative velocity v, the distance d(t) between them varies as d(t) = (dg + v. t)), where do is the initial distance and e is the sound speed. The tare-ng CIR with this dynamic delay can be modeled as:
Figure 2(b) illustrates the CIR function, which can be ex- pressed as a series of delta functions for these paths:
((T)) A1 Tlss) A20(T (ecd(--Tcho))) ) Az0 Techoz): (4)
h(t, r) = 5(t 5(t-)(tt) -(t) 5(T do v.t
where Tlas Techol, and Techo2 denote the time delays of the LOS, ECHO I, and ECHO 2 paths, respectively, and A; is the corresponding path attenuation.
We further detail the simulation of the Doppler effect in Appendix Section B.3, along with efficient methods for processing this time-varying CIR.
4.2. Reverberation
Reverberation describes the continued interaction of sounds with the environment, creating a prolonged echo effect.
Source
Source
Time
Ad
Left
Figure 2(c) presents a real-world CIR, illustrating a sharp initial spike (representing the LOS) followed by smaller spikes (carly reflections) and long diffuse tail (reverbera- tion). Figure 2(d) provides closer view, offering more detailed depiction of these components. To effectively cap- ture reverberation, we extend the channel model as follows:
d
Source
Left
Right
Right
Time
TR
(a) Signal Reception
(b) Timeline
Figure 3. Microphone Array Model.
N ((T) = a. - T.) + R(T),
4.4. Microphone Array
(5)
Sound Direction, formally known as Direction of Arrival (DoA), can be estimated by measuring the Time Difference of Arrival (TDoA) between microphones. Due to their spa- tial separation, each microphone receives sound waves at slightly different times, enabling DoA estimation.
1=0
where the first represents the impulses of LOS and early reflections, and R(()) represents the reverberation tail.
We simulate reverberation R(T) by decomposing sound into frequency-speccffi subbands with unique decay character- istics. We generate Gaussian noise, apply bandpass filters for specific frequency ranges, and modulate with amplitude
Figure 3(a) illustrates a binaural recording setup with two microphones placed a distance d apart. When a far-field source (Wang et al., 2020; Benesty, 2008) emits sound from

---

Teaching Physical Awareness to LLMs from Sounds
Table Channel Configuration Settings for Different Tasks.
Iedio lubbet Sinee
+
Relapiate NIT)
LOS
Early Reffert.
Doppler Effeet
Mic. Array
Path
Clunal
Sterdaliio
Sapalaed
AarkiN
Cantac: PNY
1.0 Detection
T
Doppler Estimation
R
DoA Estimation
R
Mullipath Analysis R
R
T
Figure 4. Synthesis Workflow of <Audio, Question, Answer>.
Range Estimation
T
Note: V: enabled; x: disabled; R: candomly cuabledddisabldd; E: costrolled target parameser.
For Doppler Estimation, we vary source speeds generate different frequency shifts. For DoA Estimation, we adjust arrival directions and microphone spacing create TDoA variations. For Multipath Analysis, we modify decay rates to simulate various reverberation levels. For Range Esti- mation, we control reflection delays to represent different propagation distances.
direction 0, it creates a differential path length ad. The corresponding TDoA, To, is given by
Ad dooos())
70
c
Figure 3(b) visualizes the timeline of sound reception. For these two microphones, let Tı and In denote the delays to reach the left and right microphones. The CIRs, assuming no attenuation, can be simplified
Parameter Randomization. For each enabled or randomly- enabled component, we randomize internal parameters to enhance channel diversity while maintaining control over task-specific variables. Unlike existing methods that recon- struct CIRs by modeling specific geometric environments (Chen et al., 2022; Zheng et al., 2024). our approach simu- lates the entire life cycle of sound waves- from emission to reception-byy explicitly modeling its key components, including LOS propagation, early reflections, reverberation, and so on. This component-wise, lifecycle-based model- ing enables scalable and systematic generation of diverse physical channels, avoiding the complexity and limited gen- eralizability mnvironment-specinn reconstructions.
hz(T) 6(t -T)) 8(t TLOS)
hR(T)= = 6(l TT. TT) = 5(1 TLOS To). (8)
This general form allows deriving hT((T) ) by accounting for direction 0, facilitating extension from a single CIR to multiple acoustic channels.
4.5. Configuration
The above individual components- -LO path, early reflec- tions, reverberation, Doppler effects, and microphone array reception-combinee to create comprehensive acoustic sim- ulations. This modularity allows us to isolate and study specific phenomena while maintaining physical realism.
5. The AQA-PHY Dataset
To train model to understand physical world from sounds, we introduce AQA-PHY, a Audio Question-Answer dataset. Each datapoint is <Audio, Question, Answer>> tuple, where audio and question form the input, and answer serves as the target output. Figure illustrates the synthesis workflow of generating datapoint.
Table presents the task-specific configurations, with de- tailed task descriptions provided in Appendix A. For dif- ferent tasks, we configure simulation components as either enabled (V), randomly enabled/disabled (R), or disabled (X). When marked 'T', the component is systematically controlled for target parameter variation. To maximize chan- nel diversity, we aim to randomly toggle each component whenever possible. However, some tasks have specific re- quirements-for example, DoA estimation requires the LOS path to be enabled as it relies on direct sound propagation for DoA calculation (Benesty, 2008).
Sound Source. The synthesis workflow begins by sampling sound sources from existing dataset. For this, we utilize AudioSet (Gemmeke et al., 2017), which offers approxi- mately 2 million 10-second sound clips annotated with over 500 labels. This extensive collection provides a rich variety of sound sources, enhaneing the dataset's versatility.
Physical Channel. For each sound source, we generate physical channel with specific physical properties using configurations from Section 4.5. The Channel Simulation module transforms these specifications into CIR, which are then convolved with sound sources to calculate sample of received audio exhibiting desired physical effects.
Our simulation approach follows core principle: targeted physical parameter control with maximal diversity:
Task-specific Physical Parameterization. Generally, for each task, we control specific parameters to generate physi- cal channels. For LOS Detection, we control the presence or absence of LOOS path to create positive and negative samples.
Question-Answer Pair. Two types of QA pairs are de-
5

---

Teaching Physical Awareness to LLMs from Sounds
Table Overall Performance. Values are presented as (Merged Sole) where "Merged" indicates models trained on combined dataset and "Sole" indicates models trained separately for each task. By default, we focus Merged results, with Sole results provided for reference.
Model Architecture
Task Performances (Merged Sole)
Audio Encoder
LOS Detection BCA (t)
Doppler Estimation MAB, (1)
DoA Estimation MAE, (4)
Multipath Analysis TCA (1)
Range Estimation REP (L)
LLM
Llama3.1-8
0.867
0.906
1.213
3.147
5.585
5.601
0.845
0.889
12.572
17.182
Whisper
Qwen2-7B
0.881
0.910
1.042
0.575
2.716
| 6.873
0.848
0.897
10.609
12.901
Llama3.1-8 1-8B
0.920
0.965
0.791
0.557
1.423
1.349
0.890
0.945
1.764
1.446
ACORN
Qwen2-7B
0.924
0.962
0,181
0.263
0.907
1.167
0.903
0.944
1.599
1.751
Performance on Open QA (Our Encoder + Qwen2-7B) 0.898
0.953
0.487
0.398
2.314
2.043*
0.906
0.908
2.852
1.900"
Random Baseline"*
0.50
10.00
66.67
0.33
33.33
For both DoA Estimation and Range Estimation in Open QA, LLMs first extract basic measurements from audio signals and then apply step-by-step calculations to determine final results (see example QA in Table 10). For the final calculation results, the MAE of DoA Estimation 6.505 (Merged) and 5.685 (Sole), while the MAE of Range Estimation 0.472m (Merged) and 0.352m (Sole). ** Random Baseline values are computed based on task-specific assumptions: random guessing for LOS (BCA = 0.5) and Multipath (TCA= = 0.33), random frequency shifts (+15% at 50 m/s) for Doppler, random TDoA within #100 samples for DoA, and uniform relative errors (0-100%) for Range Estimation.
signed. Examples of QAs are provided in Table 10.
32 Transformer layers and 0.63 billion parameters), a rep- resentative magnitude-only approach widely used in audio processing, and our encoder proposed in ACORN with 0.65 billion parameters. To demonstrate the generalizability of our approach, we pair each encoder with two different large language models (LLMs): Llama3. 1-B -B-instruct (Dubey et al., 2024) and Qwen2-7H -a-n----iinttreet (Yang et al., 2024). This setting helps verify that our audio encoding improve- ments are model-agnostic and can benefit various LLM architectures.
Closed-form pairs focus on classification or quantitative assessments with well-defined answers. These include binary classifications (e.g.. "Does the audio contain LOS path?" with Yes/No answers), multi-class selections (e.g., Rich/MoodrraeeNeeggiiihee for multipath severity), and numerical estimations (e.g., time delays or Doppler shift percentages). These pairs are generated using fixed templates to ensure consistent evaluation.
Open-form pairs elicit detailed explanations of physi- cal phenomena. These questions are phrased in natu- ral language and require comprehensive responses that demonstrate understanding of the underlying physics. For example, in range estimation tasks, answers include step- by-step calculations and explanations of how distance is derived from time-of-flight measurements. We leverage ChatGPT-4 with specific prompts to generate these QA.
Audio Pre-processing, The audio rre-processing pipeline transforms raw waveforms (sampled at 16kHz) into spec- tral representations through Short-Time Fourier Transform (STFT) with window size of 254' and a hop length of 10 ms, chosen specifically align with the configuration of our baseline, Whisper. Unlike Whisper which converts STFT outputs to mel-spectrograms', we directly preserve the full spectral resolution. The process extracts three key components: the log-magnitude spectrum for energy distri- bution, and sine/cosine components of the phase for tem- poral information. These features are stacked to create a three-channel representation of the audio.
SFT Datapoint. Finally, the synthesized audio is paired with its corresponding question and answer to form a single datapoint: <Audio, Question, Answer>> This triplet is then formatted into prompt following the template detailed in Appendix G. For each task, generate 200,000 closed- form datapoints and 10,000 open-form datapoints.
Training. The models are trained using standard token prediction loss with the answer text in SFT dataset as labels. We train the models upon MS-SWIFT (Zhao et al., 2024) with modifications to accommodate our specific model ar-
6. Evaluation
Since the input audio is real-valued, the FFT output is sym- metric, resulting in (254/2 1) 128 unique frequency bins. 'While mel-spectrograms emphasize human speech frequen- cies, we preserve full spectral resolution for fine- ane-grained analysis of physical world sounds.
6.1. Methodology
Baseline, We compare two audio encoders: OpenAl's Whis- per (Radford et al., 2023) encoder (Whisper-large-v2 with
6

---

Teaching Physical Awareness to LLMs from Sounds
imental results demonstrate the fundamental feasibility of teaching LLMS physical awareness through sound. Across all tasks, our models achieve promising performance with BCA above 0.92 for LOS detection, MAE, below 0.8 for Doppler estimation, MAF, around 1.4 for DoA estimatton, TCA above 0.89 for multipath analysis, and REP below 1.8 for range estimation. These strong results across di- verse physical tasks demonstrate that LLMs can effectively develop physical awareness through acoustic signals.
10
Ouas- Train
10
Cuza Tran Oues- Vel
Ous--V
Loss
WhipperTna
L
WhippsIInin Vol
10"-
WhiggerVal Val
10'
10"
10
10000
20000 30000
10000
20000 50000
Step
Step
(a) Qwen2-7B
(b) Llama3.1-8B
Figure 5. Loss History.
Audio Encoder Superiority. The audio encoder proposed in ACORN shows clear superiority over the existing Whis- per baseline across all evaluation metrics. For LOS detec- tion, we achieve more than 5 percentage points of improve- ment in BCA (0.920 vs 0.867 with IJama). In Doppler Effect estimation, our MAE, shows significant reduction (0.181 vs 1.042 with Qwen2). For DoA Estimation, we reduce 74.5% MAE, (1.423 vs 5.585 with Llama). Perhaps most notably, in Range Estimation, our approach reduces the REP by 7x (1.764 vs 12.572 with Llama), demonstrating substantially more accurate distance estimation capabilities.
chitecture. The model trained on 4 NVIDIA A100 GPUs with batch size 32 and completes after 7 epochs. The total training time is about 61 hours. For response generation, the decoding parameters are set with temperature 1, top-p I, and top-k 50. Appendix C lists more train hyperparameters.
Evaluation Metrics. We employ task-specific metrics to evaluate the model across five tasks. (1) For LOS Detection, we use binary classification accuracy (BCA, f) to measure the model's Is ability to correctly identify the presence ab- sence of LOS path. (2) For Doppler estimation, we employ Mean Absolute Error (MAE) to quantify the model's preci- sion in estimating frequency shift percentage (MAE,). (3) DoA estimation accuracy is evaluated using MAE in time samples (MAEs), measuring the average TDoA deviation between predicted and actual sound source directions. 3 (4) Multipath Analysis performance is evaluated using Triple- class Classification Accuracy (TCA, 1) across three cate- gories (Rich, Moderate, and Negligible). (5) Range Estima- tion employs Relative Error Percentage (REP, 4) to provide a scale-invariam measure of distance estimation accuracy.
The convergence behavior further validates this superiority- as shown in Fig. 5, our approach achieves faster conver- gence and lower final loss values during training across both Llama and Qwen architectures. This consistent advantage in both convergence rate and final performance suggests that our audio encoder's s design- particularly its incorporation of both magnitude and phase information provides more effective foundation for learning physical awareness.
-adel-agnostic Property. The consistent performance improvements across different LLMs indicate that our au- dio encoding approach is robust and model-agnostic. Both Llama and Qwen show similar patterns of enhancement when paired with our encoder compared to the Whisper baseline, suggesting our method can generalize across dif- ferent LLM architectures.
6.2. Overall Performance
As shown in Table 2, our experimental results validate four main findings: (1) the fcasibility of teaching LLMs to under- stand physical phenomena through sound, as demonstrated by strong performance across all tasks; (2) the superiority of our audio encoder over existing approaches, showing consistent improvements in all evaluation metrics; (3) the model-agnostic nature of our approach, evidenced by similar performance of different LLM architectures; (4) the effec- tiveness of natural language communication, demonstrated by maintained reasonable performance on open-form QA.
Open QA Performance. As shown in Table 2, when testing oprn-form QA with our encoder and Qwen2-7B, the model demonstrates two key capabilities: natural language commu- nication of physical phenomena and multi-step calculations. For direct estimation tasks like LOS detection (0.898) and multipath analysis (0.906), it provides clear explanations in natural language. For tasks requiring multi-step calculations, such as DoA and range estimation, extracts measurements and applies physical formulas to derive results (DoA error: 6.505", Range error: 0.472m).
Note that, in Table the evaluation metrics show model per- formance in the format "(Merged Sole)", where "Merged" represents models trained on combined datasets for all tasks, "hile"Sole" indicates models trained separately for each individual task. Our evaluation primarily focuses on the Merged results as the default metric, while Sole results is provided for just reference and comparison.
6.3. Ablation Study of Acoustic Factors
We conduct ablation studies to understand how different acoustic factors affect the understanding of physical world. While these factors could potentially impact all tasks, we se- lect representative tasks for illustrative purposes to examine
Feasibility of Understanding Physical World. Our exper-
'The key parameter in estimating DoA is TDoA (see Fq. 6).
7

---

Teaching Physical Awareness to LLMs from Sounds
1.0
1.0
Liuea
Quen
1.0
Liurs
Quics
Llama
0.9
10 (yampl))
Qwen
Accuracy
0.8
0.8
0.6
0.7
0.4
Llama
0.6
0.6
0.2
W/ LOS
LOD
W Doppler
W/O Doppler
<10UB 10-20dB 20-40BB >400B
LOS Detection
L/R Detection
Figure 6. Impact of LOS.
Figure 7. Impact of Doppler.
Figure 8. Impact of SNR.
Figure Weal-World Results.
the effects of various acoustic phenomena. By default, we set the audio encoder to our proposed design.
and data collection setup are provided in Appendix F.
Results. Our real-world experiments focused on LOS de- tection and DoA estimation in vehicle environment. The model are trained solely on simulated data and evaluated on real-world data in a zero-shot manner. Figure 8 presents the experimental results. For LOS detection, distinguish- ing between interior and exterior sound sources, the Llama- based model achieved 0.845 accuracy while the Qwen-based model maintained 0.870 accuracy. In the DoA estimation task, implemented as left/right classification within the ve- hicle cabin, Llama achieved 0.8975 accuracy while Qwen maintained 0.925 accuracy. These real-world accuracies show expected degradation compared to our controlled ex- perimental results, reflecting typical challenges in transition- ing from simulated to real-world environments, Despite this performance drop, the results show the practical viability of our approach in the real world.
Impact of LOS. Figure demonstrates the impact of LOS paths on multipath analysis by comparing scenarios with and without LOS components. For Llama, TCA decreases from 0.901 to 0.880 when LOS paths are present, while for Qwen, it shows similar trend with smaller gap (0.912 to 0.895). This makes intuitive sense, as strong direct-path signals can overshadow the subtle echoes and reflections that characterize reverberation, making it challenging to assess multipath characteristics like reverberation time.
Impact of Doppler Effect. Similarly, Figure 7 illustrates how Doppler effects influence LOS detection. The introduc- tion of Doppler shifts slightly reduces the BCA from 0.930 to 0.910 for Llama and from 0.936 to 0.912 for Qwen. This slight decrease in performance is expected, as the Doppler effect can introduce signal distortion, adding complexity to the analysis of LOS components. For more comprehensive breakdown of the results, see Appendix Section E.
7. Related Work
Notably, our method demonstrates strong resilience to these acoustic variations, with slight performance degradations. This robustness makes our approach particularly suitable for real-world environments where multiple acoustic phe- nomena often occur simultaneously.
Audio LLM. Recent work has seen significant progress in extending large language models to handle audio modalities. AudioGPT (Huang al., 2024) and HuggingGPT (Shen et al., 2023) use LLMs as controllers to coordinate multiple specialized audio models for different tasks. In contrast, models like Pengi (Deshmukh et al., 2023), LTU (Gong et al., 2023), and Qwen-Audio (Chu et al., 2023; 2024) take a more unified approach by directly connecting audio en- coders with language models to enable end-to-end learning. SpeechGPT (Zhang et al., 2023) introduces novel direc- tion by discretizing speech into tokens that can be directly processed by LLMs, while BAT (Zheng et al., 2024) and the work (Tang et al., 2024) extend these capabilities to spatial audio understanding.
Impact of SNR. Figure 8 illustrates how signal-to-noise ratio (SNR) affects ranging estimation performance, As expected, estimation accuracy improves significantly with higher SNR levels. For Llama, the MAE decreases steadily from 5.64 samples at low SNR (<10dB) to 0.91 samples at high SNR (>40dB). Qwen shows a similar trend, with MAE reducing from 5,33 samples 0.80 samples as SNR increases. This trend aligns with the general rule of thumb in acoustic sensing or ranging systems.
However, these prior works primarily focus on semantic understanding of audio content or basic spatial properties, without explicitly modeling physical awareness. In contrast, ACORN aims to teach LLMs physical awareness through sound, covering dynamic effects like Doppler, multipath, and range estimation. Furthermore, our approach to audio encoding is distinct from BAT's method. While BAT com- putes Interaural Phase Difference (IPD) between each pair of microphone channels, our encoder extracts phase infor-
6.4. Real-World Experiments
Deployment. In line with standard practices in vehicle audio systems, four onni-directional microphones de- ployed throughout NIO ES6 vehicle cabin. Two micro- phones are positioned in the front seat area and two in the back, each with left-right configuration. The audio signals are synchronized and collected via a data acquisition board 16 kHz sampling rate. Detailed hardware specifications
8

---

Teaching Physical Awareness to LLMs from Sounds
mation directly from individual channels. This approach not only avoids the quadratic growth in pairwise computations as microphone count increases but also preserves complete phase information for each channel, enabling more general representation of acoustic signal.
9. Conclusion
We present ACORN, a framework that teaches LLMs physi- cal awareness through sound. ACORN addresses a funda- mental gap in LLMs' ability to understand and interact with the physical world. By introducing a thysics-based chan- nel simulator, novel audio encoder architecture, and the AQA-PHY dataset, we enable LLMs to understand various physical phenomena.
Physical Reasoning in LLM. Recent works have explored LLMs' capabiliitess in physical reasoning, NEWTON (Wang et al., 2023) introduces comprehensive benchmark eval- uating LLMs' physical reasoning through text and visual inputs. The work (Ghaffari & Krishnaswamy, 2024) ex- amines multimodal reasoning about physical dynamics in simulated environments. Works such as (Cherian et al., 2024; Memery et al., 2023) investigate whether LLMs can estimate parameters of physical systems. With the rise of generative LLMs, researchers also explore their applications in robotics, where physical understanding is vital. Works such as (Ahn et al., 2022; Liang et al., 2023; Driess et al., 2023) examine LLMs as back-end planners in situated envi- ronments. While these efforts primarily focus on physical reasoning through textual and visual modalities, our work explores how acoustic information can empower LLMs to develop physical awareness.
A Acknowledgments
We sincerely thank the anonymous reviewers for their in- sightful comments and constructive feedback, which helped improve the quality of this work.
Impact Statement
Our research on teaching physical awareness to LLMs through sound holds significant potential for applications across various fields:
Empowering Embodied AI Equipping LLMs with phys- ical awareness through sound significantly advances em- bodied AI. By imitating human auditory capabilities, such as turning toward sound sources and adapting to acoustic environments, these systems can engage in more natural and intuitive human-robot interactions. This development enhances usability and user engagement, fostering more effective collaborations between humans and AI.
Acoustic Sensing. Prior to the emergence of LLMs for au- dio understanding, acoustic sensing is dominated by hand- crafted signal processing techniques such as ToA, TDoA, FMCW, and CIR (Bai et al., 2020; Li et al., 2022). These methods extract physical properties for tasks like activity recognition (Li al., 2020), gesture tracking (Mao et al., 2016), communication (Wang et al., 2022a), and localiza- tion (Wang al., 2022b;c), but often rely on manual feature design, synchronization, or controlled settings. In contrast, our approach leverages pretrained audio models to learn physical awareness directly from data, potentially offering greater adaptability and scalability.
Ad Advancements in Safety Applications. This work ad- dresses a critical need for secure and reliable Al systems by enabling LLMS to reason about physical phenomena such as sound source localization and LOS detection. These ca- pabilities can prevent unauthorized access in safety-critical environments like autonomous vehicles and smart homes by rejecting unsafe voice commands originating from external or unauthorized sources. Furthermore, by integrating acous- tic sensing, our approach equips autonomous vehicles with the ability to detect and localize critical sound cues, such as emergency sirens or passenger screams. This advance- ment addresses potential limitation in current autonomous driving systems, often described as "deaf drivers".
8. Limitations and Future Work
Multi-turn Conversation. Our current model is limited to single-turn interactions. Future efforts will aim to incorpo- rate multi-turn conversations, enhancing the ability to guide users through more complex reasoning.
Internal Thought. Our approach currently evaluates di- rect outputs for physical world understanding, Inspired by OpenAl-ol (OpenAI, 2024) and Deepseek-R1 (Guo al., 2025), we may examine how LLMs internalize this under- standing to enhance response conditioning, thus improving the integration of physical awareness with LLM capabilities.
Innovations in Channel-Basee Simulation Our channel- based simulation framework can be directly applied to ex- isting audio LLMs, enabling the generation of large-scale datasets that capture diverse acoustic phenomena at signifi- cantly lower cost compared to traditional data collection.
Extensive Real-World Experiments. While our real-world experiments provide valuable insights, they are limited in scope and scale. To fully validate and refine our model's capabilities, more extensive real-workd testing is necessary.
References
Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Fu, C., Gopalakrishnan, K., Hausman, K., et al. Do as can, not as i say: Grounding language
9

---

Teaching Physical Awareness to LLMs from Sounds
in robotic affordances. arXiv preprint arXiv:2204.06691, 2022.
Gong, Y., Luo, H., Liu, A. H., Karlinsky, L., and Glass, J. Listen, think, and understand. arXiv preprint arXiv:2305. 2023.
Bai, Y. Lu, L.. Cheng, J., Liu, J., Chen, Y., and Yu, Acoustic-basee sensing and applications: survey. Com- puter Networks, 181:107447, 2020.
Guo, D., Yang, D., Zhang. H., Song, J., Zhang, R., Xu, R., Zhu, Q. Ma, S., Wang. P., Bi, X., et al. Deepseek-rl: In- centivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. /2948, 2025.
Benesty, J. Microphone array signal processing. Springer Verlag, 2008.
Chen, C., Schissler, C.,, Garg, S., Kobernik, P., Clegg, A., Calamia, P., Batra, D., Robinson, P. W., and Grauman, K. Soundspaces 2.0: A simulation platform for visual- acoustic learning. In Proceedings of the Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (NeurIPS), 2022.
Hendrycks, D. and Gimpel, K. Gaussian error linear units (gelus). arXiv preprint arXiv: 1rXiv:/606.08415, 2016.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y.. Wang, S., Wang, L.,, and Chen, W. LoRA: Low-rank adaptation of large language models. In Proceedings of the Tenth International Conference on Learning Representations (ICLR), 2022.
Cherian, A., Corcodel, R., Jain, S., and Romeres, D. LImphy: Complex physical reasoning using large lan- guage models and world models. arXiv preprint arXiv:241 1..00077 2024.
Huang, R., Li, M., Yang. D., Shi, J., Chang, X., Ye, Z., Wu, Y., Hong, 7.,, Huang, J., Liu, J., et al. Audiogpt: Understanding and generating speech, music, sound, and talking head. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pp. 23802-23880 2024.
Chu. Y., Xu, J.. Zhou, X., Yang, Q.. Zhang, S., Yan, Z., Zhou, C., and Zhou, J. Qwen-audio: Advancing univer- sal audio understanding via unified large-scale audio- language models. arXiv preprint arXiv:2311.07919, 2023.
Li, D., Liu, J., Lee, S. I,, and Xiong, J. Fm-track: push- ing the limits of contactless multi-target tracking using acoustic signals. In Proceedings of the 18th Conference on Embedded Networked Sensor Systems, pp. 150-163, 2020.
Chu, Y., Xu, J., Yang, Q. Wei, H., Wei. X., Guo, Z., Leng, Y., Lv, Y., He, J., Lin, J., et al. Qwen2-audio technical report. arXiv preprint arXiv:2407. 10759, 2024.
Deshmukh, S., Elizalde, B., Singh, R., and Wang, H. Pengi: An audio language model for audio tasks. In Proceedings of the Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS), volume 36, pp. 18090- 18108. 2023.
Li, D., Cao, S., Lee, S. I., and Xiong, J. Experience: practi- cal problems for acoustic sensing. In Proceedings of the 28th Annual International Conference on Mobile Com- puting And Networking, pp. 381-390. ACM, 2022.
Driess, D., Xia, F., Sajjadi, M. S., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., ct al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.033: 2023.
Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., and Zeng, A. Code as policies: Lan- guage model programs for embodied control. In IEEE International Conference on Robotics and Automation (ICRA), pp. 1493-9500 IEEE, 2023.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama herd models. arXiv preprint arXiv:2407.22 17733, 2024.
Mao, W., He, J., and Qiu, L. Cat: nigh-precision acous- tic motion tracking. In Proceedings of the 22nd Annual International Conference on Mobile Computing and Net- working, pp. 69-81. ACM, 2016.
Gemmeke, J. F., Ellis, D. P., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., and Ritter, M. Audio set: An ontology and human-labeled dataset for au- dio events. In IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2017.
Memery, S., Lapata, M., and Subr, K. Simlm: Can language models infer parameters of physical systems? arXiv preprint arXiv:2312.142155 2023.
Ghaffari, S. and Krishnaswamy, N. Exploring failure cases in multimodal reasoning about physical dynamics. In Proceedings of the AAAI Symposium Series, pp. 105-114, 2024.
OpenAL.
Introducing openai wttpiaae.oooooi///ce/aaaa
ol ol-preview.
2024.
Accessed: 2024-12-01.
10

---

Teaching Physical Awareness to LLMs from Sounds
Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. Robust speech recognition via large- scale weak supervision. In Proceedings of the 40th In- ternational Conference on Machine Learning (ICML), 2023.
models with intrinsic moos conversational abilities. arXiv preprint arXiv:2305.//000 2023.
Zhao, Y., Huang. J., Hu, J., Wang. X., Mao. Y., Zhang, D.,
Jiang, Z.., Wu, Z., Ai, B., Wang, et Swift: a scalable lightweight infrastructure for fine-tuning, arXiv preprint 24X8h:2408.05517. 2024.
Rao, S. Introduction to mmwave sensing: Fmew radars. Texas Instruments (II) mm Wave Training Series, pp. 1- 11, 2017.
Zheng, Z., Peng, P., Ma, Z., Chen, X., Choi, E., and Harwath, Bat: Learning to reason about spatial sounds with large language models. In Proceedings of the Forty-first International Conference on Machine Learning (ICML), 2024.
Shen, Y., Song, K., Tan, X., Li, D., Lu, W., and Zhuang, Y. HuggingGPT: Solving AI tasks with chatGPT and its friends in hugging face. In Proceedings of Thirth-seventh Conference on Neural Information Processing Systems (NeurIPS), 2023.
Tang, C., Yu, W., Sun, G., Chen, X., Tan, T., Li, W., Zhang, J.. Lu, L., Ma, Z., Wang, Y., et al. Can large language models understand spatial audio? In Proceedings of Interspeech, pp. 4159- 2024.
Traer, J. and McDermott, J. H. Statistics of natural rever- beration enable perceptual separation of sound and space. Proceedings of the National Acudemy of Sciences, 1 113 (48):E7856-E77)) E7865, 2016.
Wang, W., Li, J., He, Y.. and Liu, Symphony: localiz- ing multiple acoustic sources with single microphone array. In Proceedings of the 18th ACM Conference Embedded Networked Sensor Systems, pp. 82-94, 2020.
Wang, W., Li, J., He, Y., Guo, X., and Liu, Y. Motorbeat: Acoustic communication for home appliances via vari- able pulse width modulation. Proceedings of the ACM Interactive, Mobile, Wearable and Ubiquitous Technolo- gies, 6(1):1-24, 2022a.
Wang, W., Li, J., He, Y., and Liu, Y. Localizing multiple acoustic sources with single microphone array. IEEE Transactions Mobile Computing, 22(10):5933-5977 2022b.
Wang, W., Mottola, L., He. Y., Li, J., Sun, Y., Li, S., Jing, H., and Wang, Y. Mienest: Long-range instant acoustic localization of drones in precise landing, In Proceedings of the 20th ACM Conference on Embedded Networked Sensor Systems, pp. 504-517,2022
Wang, Y. R., Duan, J., Fox, D., and Srinivasa, S. Newton: Are large language models capable of physical reasoning? arXiv preprint arXiv:2310.070.. 2023.
Yang, A., Yang, B., Hui, B., Zheng. B., Yu, B., Zhou, C., Li. C., Li, C., Liu, D., Huang, F, et al. Qwen2 technical report. arXiv preprint axXiv:2407.1067. 2024.
Zhang, D., Li, S., Zhang, X., Zhan, J., Wang. P., Zhou, Y., and Qiu, X. Speechgpt: Empowering large language

---

Teaching Physical Awareness to LLMs from Sounds
A. Task Description
Table 3. Tasks for Physical Awareness Enhancement in LLMs
Task Name
Description
Example Applications
LOS Detection
Determine if the line-of-sight path blocked or not.
Reject voice commands from outside secured area like a car.
Doppler Estimation
Estimate the frequency shift in sounds due the rela- tive motion of the source.
Determine if the source is approaching or mov- ing away.
DoA Estimation
Estimate the direction from which a sound originates based on the TDoA.
Enable embodicd systems to rotate towards sources, imitating human head movements.
Multipath Analysis
Detect the severity of reverheration resulting from mul- tipath effects.
Assess sound quality by identifying distortions due to complex surrounding reflections.
Range Estimation
Measure the distance objects by employing sonar techniques.
Enable proximity sensing determine phone screen-lock behavior during calls.
The tasks listed in Table explore both passive and actíve approaches to understand physical world. The first four tasks line-of-sight (LOS) Detection, Doppler (Effeet) Estimation, Direction of Arrival (DoA) Estimation, and Multipath (Effect) Analysis demonstrate passive sensing capabilities, where LLMs analyze incoming sounds to understand physical phenomena. The fifth task, Range Estimation, explores the feasibility of active sensing, where the LLM actively probes the environment by playing, recording, and analyzing sound pulses, similar to how sonar systems operate.
A.1. Passive Sensing Tasks
Our passive sensing tasks evaluate the ability of LLMs to extract physical information from ambient or environmental sounds:
LOS Detection enables LLMs to determine whether there is a direct path exists between the sound source and receiver. When sound travels through a direct path, it arrives earlier than reflections and contains highest energy due minimal path attenuation. This capability crucial for security applications, such validating voice commands in vehicles or buildings. For example, a car's voice control system should reject voice commands from outside the vehicle to prevent unauthorized access. Similarly, smart home devices can use LOS detection to determine if voice commands originate from inside or outside the house.
Doppler Estimation focuses on measuring frequency shifts caused by relative motion between sound sources and receivers. When a sound source approaches, the perceived frequency increases due to wave compression; when it recedes, the frequency decreases due to wave expansion. This shift largely depends on the relative velocity. This physical awareness valuable for various applications, from estimating vehicle speeds enabling proximity-based responses in smart devices. For instance, device could adjust its behavior based whether the sound source is approaching moving away.
DoA Estimation teaches LLMs to determine where sound originates using multiple microphones. This task relies on analyzing time differences of arriving different microphones (i.e., Time Difference of Arrival, TDoA). This task is beneficial to natural human-robot interaction, allowing systems to turn toward heads just as humans do. might also enable spatial audio processing and directional noise suppression in applications like smart speakers and hearing aids.
Multipath Analysis involves understanding how sound reflects and reverberates in different environments. Sound waves bounce off surfaces, creating complex patterns of early reflections and late reverberation. This capability helps systems assess audio quality and adapt to various acoustic environments. For instance, if the LLM detects excessive reverberation degrading sound quality during voice communication, it could proactively suggest to the other party, 'It seems there's lot of background reverberations- -eteratiioosooiingggg. to another place might help.' This level of adaptive response enables the LLM to function similarly to human listener, enhancing user experience by facilitating clearer communication in less-than than-ideal acoustic conditions."
12

---

Teaching Physical Awareness to LLMs from Sounds
ToF
8
6
Freg, (kllz)
LOS
ECHO
2
Wall
LOS
ЕCHO
0
Time (scc.)
(a) Deployment
(b) Spectrogram
Figure 10. Ranging Estimation Illustration.
A.2. Active Sensing Tasks
Range Estimation explores active sensing capabilities, requiring LLMs to initiate measurements by transmitting sound pulses and analyzing their echoes. Unlike passive tasks that only observe existing sounds, this represents a significant advancement toward active environmental interaction. We envision that the LLMs can trigger pulse transmission through function calls, measure echo return times, and calculate distances based on the speed of sound. This active approach enables precise distance measurements independent of ambient sound conditions, demonstrating how LLMs can actively engage with their physical environment.
To better understand this, we give an example in Figure 10. As shown in Figure 10(a), the basic deployment involves system that actively transmits sound pulse. This pulse travels along two paths: a direct Line-of-Sige (LOS) path to the microphone, and reflected ECHO path that bounces off the wall. This configuration mirrors the principles used natural and technological sonar systems. The spectrogram in Figure 10(b) illustrates how these paths manifest in the received signal, demonstrating the feasibility of distance estimation. We can clearly observe two distinct bright patterns: the LOS signal arrives first, followed by its ECHO from the wall. The time difference between these signals, marked by the Time-of-Flight (ToF) dashed lines, makes distance calculation straightforward. Since know sound travels at approximately 343 meters per second in air, and the echo travels twice the distance to the wall (to the wall and back), we can determine the wall distance by taking half of the total distance that the echo traveled.
We want to mention that, range estimation is unique among our tasks not only in its active sensing approach but also in its sound source. While other tasks utilize real-world sound sources sampled from the AudioSet dataset, Range Estimation employs specially modulated signals for transmission. Specifically, uses Trequency-Modulatee Continuous Wave (FMCW) signals, also known as chirp, where the frequency changes linearly with time. These chirp signals, widely used Radar and Sonar systems (Rao, 2017), provide excellent range resolution properties. To ensure the diversity of source sources used for range estimation, we randomize several chirp generation parameters including the starting frequency, ending frequency, and chirp duration.
B. Implementation Detailed for Channel Simulator
B.1. LOS ad Early Reflections
In Eq. we use the Delta function (6) to explain CIR for better understanding. In practice, we employ damped sinusoids, which provide both precise parameter control and naturally model physical acoustic behaviors like attenuation and resonance. Mathematically, Each path in the CIR is modeled as damped sinusoid
=tt) cos(2.ff2)e
(9)
where fa is the sinusoid wave frequency, az is the decay rate, and is the time. Therefore, Eq. can be rewritten as
h(T) = A1a(F - Tia)) + A22(T - Techo,) ) + Ay2(T Techoz).
(10)
Table 4 lists the corresponding parameters used to simulate LOS and early reflections. Except amplitude, The parameters for LOS paths are largely fixed to model the direct path's 's consistent nature, as it experiences minimal environmental interactions. In contrast, the parameters for early reflections vary randomly to capture the diversity of channels.
13

---

Teaching Physical Awareness to LLMs from Sounds
Table 4. Parameter Specifications for Damped Sinusoid Path Generation
Value/Range
Sampling Method
Parameter
Description
Unit
LOS
Early Reflections
LOS
Early Reflections
Base Frequency (fe)
Sinusoid wave frequency
Hz
2,000
2,000
Fixed
Fixed
Frequency Modification Decay Rate (o
Factor for tuning base frequency
1.0
0.8 1.2
Fixed
Uniform
Base exponential decay
Hz
8,000
8,000
Fixed
Fixed
Decay Modification
Factor for tuning decay rate
1.0
1.0 1.5
Fixed
Uniform
Amplitude (4)
Path strength
5.0 20.0
0.5
Uniform
Uniform
Duration
Time duration path
second
0.01
0.01
Fixed
Fixed
B.2. Reverberation
During the simulation of reverberation, Gaussian noise is first generated as the base signal, characterized by normal distribution with a mean of zero and standard deviation of one, denoted as
n(t) (),
(11)
The noise generated with a sampling rate of 16 kHz, and to accommodate the requirements of our simulation, the duration of n(() is set to maximum of 2 seconds, translating to 32,000 samples. This establishes the temporal extent of the impulse response aim to simulate.
Subsequent processing involves bandpass filtering this noise to isolate specific frequency bands, crucial for accurately modeling the frequency-selective behavior observed in real-world reverberation, where different frequency bands decay at different rates. Each subband is obtained using Butterworth bandpass filter, selected for its flat frequency response within the passband and sharp cutoff characteristics. The frequency range for each subband i is specified by f!ow and phigi The filtered signal for the subband is given by
st bandpass(n(t).. flom, 1 fliggh..
(12)
Each filtered signal (1) is modulated by an amplitude envelope to simulate the natural decay characteristics of sound an environment. This decay typically exponential, mathematically represented (t) exp(-A,t). where A represents the decay rate for subband i.
Finally, the processed signals from each subband are summed to construct the broadband reverberation, defined as:
N Nhand
R(t) = E s et(),
(13)
where Noand is the number of subbands.
For a detailed breakdown of the parameters used in our reverberation simulation, including the specific frequency bands and decay rates employed, refer to Table 5.
B.3. Doppler Effect
This subsection provides an detailed explanation of the method for simulating the Doppler effect within a time-varying CIR and explains an efficient processing approach. Instead of using traditional convolution operation, we apply the Doppler-induced time-varying delay transformation directly to the transmitted signal, reducing computational complexity.
In stationary channel, the Channel Impulse Response (CIR), l((T), is typically constant over time, representing fixed delay set of delays for each multipath component. However, for moving transmitter or receiver, the CIR must vary with time to capture the Doppler effect, which introduces a continuous frequency shift due to the relative motion.
To model this, we introduce an additional time dependency into the CIR, such that h(t,, r) captures how the path delay ttu(t) changes with relative speed.
14

---

Teaching Physical Awareness to LLMs from Sounds
Table Parameter Specifications for Simulating Reverheration
Parameter
Description
Units
Value/Range
Sampling
Sampling Rate (fs)
Sampling rate of audio signal
Hz
16,000
Fixed
Duration (T)
Total length the Gaussian noise signal
second
2
Fixed
Number of Samples
Total number of audio samples
sample
32,000
Fixed
Number of Subbands (Niuna)
Number of frequency hands for filtering
6
Fixed
(50,
500)
(500, 1,000)
Subband Frequency (f (hiihh
Lower and upper frequency bounds of sub- Hz band i
Fixed
(1,000, 2,000)
(2,000, 4,000)
(4,000, 8,000)
Decay Rates (A:)
Amplitude decays for subband Order of the Butterworth bandpass filter
-05
Uniform
Butterworth Filter Order
4
Fixed
larger decay rate X; means the sound energy decays more quickly, resulting a shorter reverheration time (RT60). Specifically, a X; value of 1.5 yields an RT60 of approximately 2 seconds, while a A value of 60 shortens the RT60 to about 0.05 seconds.
Let's consider a scenario where the transmitter and receiver are separated by an initial distance dy and moving at relative speed As they move, the distance between them changes linearly with time: d(() do v Since the speed of sound is this changing distance translates to time-varying delay rt) in the signal:
d(t) do uu.t
r(t)
(14)
This delay means that the time it takes for each part of the transmitted signal reach the receiver continuously varies the distance changes. We can represent this time-varying delay in the CIR by using delta function that shifts in time. The CIR with Doppler effect then becomes:
h(t, = (T do + +
(15)
To obtain the received signal v(),, we convolve the transmitted signal zt1) with the time-varying CIR h(t, r):
v(t) =
a(T)h((tt r)dr
/ x(7)6 (t (t- du + v
dr
c
x do v.
(16)
This expression can be rewritten as:
2 -1 do
v(1)
(17)
As can see, the factor (1 2) compresses stretches the time variable, This aligns with the Doppler effect, where relative motion changes the observed frequency.
More importantly, This expression shows that the Doppler effect can be viewed as resampling the original source x() at a new effective sampling rate g. By interpreting the delay transformation as a resampling problem, we can achieve the Doppler simulation efficiently using 1D interpolation to calculate the received signal yt).
For connivent, Table summarizes the parameters relevant for simulating the Doppler effect.
B.4. Microphone Array
Similarly, Table summarizes the key parameters for simulating the reception of microphone array.
15

---

Teaching Physical Awareness to LLMs from Sounds
Table 6. Parameter Specifications for Simulating Doppler Effect
Parameter
Description
Units
Value/Range
Sampling
Sampling Rate (f.)
Rate at which the signal sampled Starting distance between transmitter and receiver
Hz
16000
Fixed Uniform
Initial Distance (do)
m
0.5 100
Relative Speed (v)
Relative velocity between transmitter and receiver
m/s
-50 -0
Uniform
Sound Speed (c)
Speed of sound
m/s
343
Fixed
Interpolation Method
Interpolation technique for resampling
Linear
Fixed
Table 7. Parameter Specifications for Simulating the Reception of Microphone Array
Parameter
Description
Units
Value/Range
Sampling
Inter-microphone Distance (dmic) Direction of Arrival (0)
Distance between left and right microphone
15
Uniform
The direction in which the sound source ar- degree rive at microphone array
0 180
Uniform
C. Training Hyperparameters
The main training hyperparameters configured are summarized Table
Table 8. Training Hyperparameters
Parameter
Value
GPUs
4 NVIDIA A100
Global Batch Size
32
Epochs
7
Optimizer
AdamW
Optimizer Parameters
31 0.9, 82 = 0.95, e = 1e-8
Learning Rate Schedule
WammupDecayl.R
Weight Decay
0.1
Warm-up Min Learning Rate
0
Warm-up Max Learning Rate
0.0001
Warm-up Ratio
0.05
LoRA Rank
8
LoRA Alpha
32
LoRA Dropout
0.05
D. Ablation Study of LoRA Rank
We keep the LoRA rank relatively low, 8, to preserve the LLM's language ability, as most representation learning occurs in audio encoder. This choice aligned with model-agnostic property of our model. Higher ranks showed minimal benefit but increased cost (see Table 9).
E. Detailed Analysis of Doppler Effect on LOS Detection Performance
Figure Il demonstrates how varying Doppler speeds affect LOS detection for the models Llama and Qwen. The analysis covers speed bins from -50 to +50 m/s. Both models show consistently stable performance across these speed ranges. Notably, performance variations are evident in conditions of higher speeds, underscoring the models' varied sensitivity to distortions caused by the Doppler effect.
16

---

Teaching Physical Awareness to LLMs from Sounds
Table 9. Ablation Study of LoRA Rank
Setting
Task Performances (Merged)
Model Architecture
LoRA Rank
LOS Detection
Doppler Estimation MAE, ())
DoA Estimation
Multipath Analysis TCA (t)
Range Estimation REP (L)
BCA (t)
MAE (4)
4
0.919
0.869
1.631
0.881
1.943
8
0.924
0.181
0.907
0.903
1.599
Our Encoder +
Qwen2-7B
16
0.908
0.193
0.840
0.881
1.561
32
0,910
0.187
0.731
0.890
1.724
0.94
Llama
Qwen
0.90
0.88
0.84
[-50, 335)
F-35, 2-5
[-25, (-25.-15)
[-15. -9)
(-
15,
[15, 25)
125, 35)
[35, 50)
Speed Bins (m/ss
Figure 11. Impact of Doppler Speed LOS detection.
F. Real-World Deployment
The data collection setup, as illustrated in Figure 12, consists of four omni-directional SPK0641HT4H digital microphones deployed throughout a vehicle cabin (specifically in a NIO ES6, a smart vehicle with Audio Assistant support). Two microphones (Mic. and Mic. 2) are positioned in the front seat area, with one on the left and one on the right, as shown in Figure 12(a). The remaining two microphones are positioned in the back seat area, following the same left-right configuration, depieted in Figure 12(b) and 9c)).
The microphones are driven and sampled by an XMOS XU216 data acquisition board (visible in Figure 12(a)), which ensures synchronization across all four channels. Operating at a sampling rate of 16 kHz, the board streams the synchronized audio signals to a laptop via USB UAC 2.0 protocol for data collection and storage.
For comprehensive evaluation, we place a speaker at various test locations. For exterior testing, we deploy the speaker outside at all four car doors (Figure 12(d) shows an example of one door placement). The speaker was also positioned at various interior locations including the front seat (Figure 12(e)) and back seat (Figure 12(()).
For data collection, we gather 200 samples for the LOS detection task, evenly split between interior (100 samples) and exterior (100 samples) positions. For the left/right detection task, we collected 400 samples in total, with 100 samples from each of the four seating positions fleft-front, right-front, left-back, and right-back seats).
G. Prompt Format
To ensure the generality and broad applicability, we adopt standardized prompt format commonly used in the field, similar to those employed by models such as Llama and Qwen. The basic format for single-audio tasks is structured as follows:
<| [lim_stare >>system( \n) You are helpful assistant.
< <lim_end|>>
Audio 1: < alaudio_boss | laudi_ (>n)) (Question)
17

---

Teaching Physical Awareness to LLMs from Sounds
Mic. 1 D0 Mic. 2
Mic. 3
Mic 4
Data Acq. Board
(a) Mic. and 2, and data acq. board
(b) Mic. 3
(c) Mic. 4
Speaker
Speaker
Speaker
(d) Speaker outside the
(e) Speaker front
(f) Speaker at back
Figure 12. Illustration of microphone and speaker deployment in the NIO ES6 vehicle. Microphones (Mic. 1-4) are positioned in the front and seat areas, while speaker placed various locations replaying audios.
< imm_edd | An)
<| im__tarrt bassistant )
(Answer)
<lim_end|>>
The format uses "You are a helpful assistant" as the default system prompt. The placeholders (Question) and [Answer] are replaced with the specific question and answer pairs from the dataset. For tasks requiring two audio inputs, we extend the format by adding a second audio segment:
<lim_start start >>system| (nn)
You are a helpful assistant.
<| <|m__edd/>(\n)
<lim_start /
Audio 1: <| bos >>< [AUDIO audio_eos |> (An)
Audio 2: < audio_bbos >< AUDICI >< audio_eos > (An)
(Question)
<lim_end|>{{
<| <|intstarti >asssistant \n)
(Answer)
im >
H. Comparison Between Real and Synthesized CIRs
To qualitatively validate the diversity of our synthesized CIRs, we retrieve the closest match for several real CIRs using similarity search. Representative examples are shown Figure 13.
Example QA Pairs
Table 10 presents the example QA pairs for five different tasks, including both close and open types.
18

---

Teaching Physical Awareness to LLMs from Sounds
Real CIR
Best-Matched Synthesized CIR
1.0
1.0
Amplitude
0.5
0.5
0.0
0.0
0.5
0.5
200
400
600
800
1000
200
400
600
800
1000
Sample Index
Sample Index
(a) LOS Example
Real CIR
Mest-MMacchd Synthesized CIR
1.0
1.0
Amplitude
0.5
0.5
0.0
0.5
0.5
-1.0
-1.0
200
400
600
800
1000
200
400
600
800
1000
Sample Index
Sample Index
(b) LOS Example
Real CIR
Best- Matched Synthesized CIR
1.0
1.0
Amplitude
0.5
0.0
0.0
00.
0..
1.0
1.0
200
400
600
860
1000
200
400
600
800
1000
Sample Index
Sample Index
(c) NLOS Example
Real CIR
Best-Matched Synthesized CIR
Amplitude
0.5
0.5
0.0
0.0
-0.5
0.5
-1.0
-1.0
200
400
600
800
1000
200
400
600
800
1000
Sample Index
Sample Index
(d) NLOS Example 2
Figure 13. Comparison between real and synthesized CIRs in both LOS and NLOS scenarios. For each example, we select real CIR (left) and retrieve the most similar synthesized CIR (right) using similarity search. Two LOS and two NLOS shown demonstrate the diversity and realism of our synthesized CIRs.
19

---

Teaching Physical Awareness to LLMs from Sounds
Table 10, Example QA Pairs for Tasks.
Task
Form
Example Question
Example Answer
Close
Audio I: <Audio> Does the audio contain line- of sight path?
Yes. No.}
LOS Detection
Open
Audio I: <Audio>> Is there clear LOS path in the audio, not?
No, the audio doesn't 't have a clear LOS path. I think the source reaches the receiver through indirect paths.
Close
<Audio>> Audio <Audio> You are pro- vided with a two-channel audio recording. The first channel is the origin the sound. The second channel the received audio. Please determine whether the sound source approaching receding.
Audio I:
{ Approaching.
Receding.)
Close
< Audio Audio 2: <Audio> You are provided with two-channel audio recording. The first channel the origin the sound. The second channel is received audio. Please estimate Doppler shift percentage.
Audio I:
{XX%.}
Doppler Effect
Open
Audio I: <Auiio>> Audio <Audio>> How percentage Doppler shift estimated from two-channel audio recording. where the first channel represents the original sound and the second channel captures the received audio?
The estimated Doppler shift is approximately 2.5%. The sound seems to approach you due to noticeable increase in frequency.
Close Audio <Auiio> Audio <Audio>> Given two-channel audio recording, where the first channel is from the left microphone and the second from the right, which microphone receives the sound first?
[Left.
Right. |ront.]
Close
Audio 1: <Audio> Audio 2: <Audio> Given
[XX samples.)
two-channel audio recording, where the first channel is from the left microphone and the second from the right. Please estimate the time difference of arrival.
DoA Estimation
Open
<Audio> Audio 2: <Audio>> In smart home system, two microphones are placed d = 10 cm apart the ceiling to localize sound source, such speaker, in the living room. Given the recorded assuming the speed of sound c = 343 m/s with sampling rate fs 16000 Hz, what is the Direction of Arrival (DoA) of the sound source?
Audio 1:
Let's solve this step by step: in From the audio signals
analysis, we estimate the time delay:),n n N samples n 2. Convert the time delay to time (r) in milliseconds:
T N fs -4 16000 -0.25 ms 3. Apply the DoA formula: n = arccos(( x cc d) = arccos((-0.25 1000 x 343) / 0. 0.1) 4 Simplify the calculation: in 0 Niccos(-0.857))
audio signals from both microphones
0.5404 radians' ) \ 5. Convert radians to degrees:)u \n Gang (0 T)) 180 30.96 degrees' \/n Final Answer: )a estimated Direction of Arrival (DoA) angle is approximately 30.96 degrees.
Close
<Audio>> How severe is the multipath effect in this audio?
Audio 1:
{Rich. Moderate. Negligible.}
Multipath
Analysis
Open
<Audio,> Could you evaluate the effect of multipath interactions within the given audio piece?
Audio I:
The multipath effect is very rich in this audio, with ex- tensive reverberation creating complex sound profile.
Close
are given audio data con- taining both the transmitted pulse and the echo. Please estimate the time of flight.
Audio I: <Auioo>
{XX ms.}
Range
Open
Audio 1: <Audio> You given acoustic data that includes transmilted sound pulse and its reflection. Please estimate the distance to the object.
The time of flight for the sound pulse is 59.0 ms. As- suming speed sound air 343 m/s, the round trip distance to the temperature inversion layer is calcu- lated as 343 multiplied by 59.0 / 1000, equaling 20.2 meters. The altitude of the layer is half of this value. which equals 10.1 meters.
Estimation
20


Reviews:
[{"review_id": "CzN4p8JMFo", "reviewer": "Reviewer_t8Co", "summary": "The paper presents a novel method to imbue large language models with physical awareness through sound by using a physics-based channel simulator that synthesizes realistic acoustic data, simulating phenomena such as the Doppler effect, multipath reflections, and LOS conditions. The authors design an audio encoder that processes both the magnitude and phase (via sine and cosine components) of the audio signal, enabling the model to capture subtle physical characteristics. This encoder is integrated with LLMs and trained on the AQA-PHY dataset—a large-scale collection of one million <Audio, Question, Answer> tuples—using supervised fine-tuning. Extensive evaluations across passive sensing tasks and active sensing tasks demonstrate the method’s effectiveness and model-agnostic improvements.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer t8Co\n\nWe sincerely thank the reviewer for the thoughtful and constructive feedback. We are encouraged by your recommendation! Our responses are as follows:\n\n ---\n**Regarding Innovation Compared to BAT**\n\n**Response:** We greatly appreciate BAT's pioneering work in bringing spatial audio understanding to LLMs, which partially inspires our work.  However, our work addresses fundamentally different problems and technical challenges, requiring distinct approaches:\n- **Different goals and focus:** BAT primarily addresses spatial audio perception for auditory scene understanding (e.g., \"Is sound A to the left of sound B?\"), while our work focuses on teaching LLMs to understand fundamental physical phenomena through sound. We enable LLMs to comprehend physics principles rather than just spatial relationships. \n- **Broader range of physical awareness tasks:** We explore physical tasks entirely absent from BAT, including Doppler effect estimation, multipath analysis, and active sensing capabilities (range estimation). Understanding dynamic phenomena like the Doppler effect requires fundamentally different modeling than BAT's spatial tasks. \n- **More scalable phase-aware audio encoding:** Unlike BAT, which uses Interaural Phase Difference (IPD) between microphone pairs, our encoder extracts phase information directly from individual channels. In this way, we preserve phase information **without introducing quadratic growth** in computations as microphone count increases. Notably, our encoder works effectively even with single-channel audio (e..g, LOS detection, Multipath Analysis), while BAT's approach reduces to a magnitude-only encoder in such cases. \n- **Open-format QA evaluation:** We introduce open-format QA to assess deeper physical understanding. While BAT primarily uses LLMs for classification or regression with fixed outputs, our approach leverages LLMs' reasoning capabilities through natural language explanations. This allows us to evaluate whether models truly understand physical phenomena rather than simply mapping inputs to pre-defined labels.\n- **Comprehensive physics-based channel simulator:** We've developed a dedicated simulator that models acoustic physical phenomena (LOS paths, early reflections, reverberation, Doppler effects) in controlled ways. This enables precise generation of diverse training data with accurate physical annotations, going beyond BAT's focus on spatial localization in static environments. \n\n---\n\n**Regarding Real-World Evaluation**\n\n**Response:** We agree that the scope of our real-world evaluation is currently limited, and we sincerely appreciate the reviewer’s suggestion. Our primary goal in including the real-world experiment was to establish a **proof of concept**—demonstrating that models trained purely on synthetic data (AQA-PHY) can generalize effectively to real acoustic environments, thereby validating the feasibility of sim2real transfer.\n\nWhile we acknowledge the importance of broader evaluations, collecting real-world data with reliable physical ground truth is non-trivial, often requiring specialized hardware and calibration. This challenge is especially pronounced for tasks involving **latent physical properties** (e.g., LOS, TDoA), which are not easily labeled.\n\nNevertheless, we fully agree that expanding the real-world evaluation would significantly strengthen the paper. We will revise the manuscript to clarify our current positioning and will make every effort to include additional real-world experiments.\n\n---\n\n**Regarding Random Performance Baseline**\n\n**Response:** Thank you for this good suggestion! We will include random performance baselines in Table 2 for the final version. For classification tasks (BCA and TCA), the random baselines are straightforward. For regression tasks, we calculate the expected error when randomly sampling outputs within the possible value range:\n- LOS Detection ($BCA$): 0.5 \n- Doppler Estimation ($MAE_f$): 10.0 (assuming random outputs within the possible frequency shift range of [-15%, 15%], with maximum speed of 50 m/s and sound speed of 343 m/s) \n- DoA Estimation ($MAE_t$): 66.7 (assuming random outputs within the possible TDoA range of [-100, 100]) \n- Multipath Analysis ($TCA$): 0.33 \n- Range Estimation ($REP$): 33.3 (assuming random outputs yielding relative errors between 0-100%)\n\nWe believe these baselines will highlight the improvements our approach achieves. Thank you again.\n\n---\n\n**Regarding Typo Correction**\n\n**Response:** Thanks. We will fix this in the revision.", "claims_and_evidence": "Yes. Evaluations across tasks like LOS detection, Doppler estimation, DoA estimation, multipath analysis, and range estimation demonstrate claims in the paper.", "methods_and_evaluation_criteria": "Yes. \n(1) LOS Detection: binary classification accuracy (BCA, ↑);\n(2) Doppler estimation, Mean Absolute Error (MAE);\n(3) DoA estimation: MAE;\n(4) Multipath Analysis: Triple-class Classification Accuracy (TCA, ↑);\n(5) Range Estimation: Relative Error Percentage (REP, ↓).\n\nAll evaluation criteria make sense.\nFor proposed method, which incorporates phase into its encoder to better capture physical phenomena.", "theoretical_claims": "The theoretical claims for LOS ad Early Reflections, Reverberation, Doppler Effect, Microphone Array are correct.", "experimental_designs_or_analyses": "All looks good.", "relation_to_broader_scientific_literature": "The paper extends established signal processing and acoustic simulation techniques—such as modeling the Doppler effect and multipath reflections—to enhance LLMs with physical awareness, building on prior work in audio understanding (e.g., BAT).", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n1. The paper is easy to follow.\n2. For the first time, this paper established a connection between physical acoustic simulation techniques and LLMs, which is quite novel.\n3. Sim2Real evaluation is valuable.\n\nWeaknesses:\n1. Compared to previous work—BAT, this paper is more likely to explain things from a physical perspective, but essentially it does not seem to offer sufficient innovation. Many aspects, such as the model architecture and encoder design, are fundamentally inherited from the settings in BAT.\n2. In sim2real case, the model performs well on LOS and Left/Right detection. However, since these tasks are basically binaural classification questions, which makes them quite easier to answer. I wonder if it's possible for authors to also evaluate more complicated tasks to show the model really generalize well on real situation.\n\n[1] BAT: Learning to reason about spatial sounds with large language models. Zheng, Z., Peng, P., Ma, Z., Chen, X., Choi, E., and Harwath, D.", "other_comments_or_suggestions": "Line 045 should have a period.", "questions_for_authors": "Add a random performance baseline in Tab 2 makes it easier to understand the overall performance relative to chance.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "EQrCiVV2Jy", "reviewer": "Reviewer_Gt9i", "summary": "The authors train an LLM to have knowledge of the physical world through acoustics. To do this, the authors create a large synthetic dataset of question-answer pairs that include audio from an acoustic simulator. The authors introduce an audio encoder that incorporates phase information, and they show that it improves performance on the acoustic tasks compared to an off-the-shelf encoder, that it works with multiple LLMs, and that it works with real world (non-simulated) data.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer Gt9i\nWe sincerely thank the reviewer for the thoughtful and constructive feedback. We are encouraged by your recommendation! Our responses are as follows:\n\n---\n**Regarding Physcial Awareness vs. Acoustic Calculator**\n\n**Response:** \nThank you for raising this insightful question!\n\nWe believe physical awareness emerges progressively as model scale, task diversity, and reasoning complexity increase—similar to trends in LLMs.\nThus, we (1) use a physics-based simulator to generate diverse, controllable acoustic scenes, and (2) design open-form QA tasks that encourage causal reasoning, not just output a number or a label.\n\nRather than treating the 'acoustic calculator' as an end goal, we view it as a **probing tool to assess whether LLMs can interpret raw waveforms**.  \n\nFor open-ended QAs, the goal is for the LLM to understand the underlying physical phenomenon—not merely to compute a value, but to reason about causes and express that understanding in natural language.\n\nWe will clarify this.\n\n---\n**Regarding Intermediate Step Accuracy**\n\n**Response:** We do report intermediate prediction errors in Table 2: DoA and range estimation results are presented in both time-domain estimation and final output (see Table footnote).\n\nFurther, we include two figures showing how TDoA error correlates with DoA error, and how ToF error maps to distance error. In the range plot, the discrete vertical steps are due to rounding—our dataset uses integer-valued distances (in m), so small variations in ToF can lead to quantization in distance error. These results show that the model extracts and reasons over physical intermediate variables. \n\n**Results:**\n- DoA: https://imgur.com/VkZlSvz\n- Range: https://imgur.com/0qBVk9K\n  \n---\n**Regarding Whisper underperforms on LOS Detection**\n\n**Response:**  Magnitude features capture signal content—e.g., what is being said.  One good example is AudioMAE, which can generate audio from magnitude only. In contrast, **phase encodes fine-grained temporal cues**, revealing how a signal propagates through space.\n\nLOS detection relies on phase cues:\n- A LOS signal shows a sharp onset and consistent phase.\n- Reflected paths exhibit delayed or disrupted phase due to multipath.\n\n**These distinctions are often invisible in magnitude but clear in phase**, making phase essential. We will explain this in the revision.\n\n---\n**Regarding Training Data**\n\n**Response:** The model is trained only on simulated data. Real-world results are zero-shot evaluations. \n\n---\n**Regarding Comparison with BAT**\n\n**Response:** While BAT is a pioneering work on spatial audio reasoning, our work targets physics-based understanding, with different goals, tasks, and evaluation formats (see response to reviewer t8Co), making direct comparison nontrivial.\n\nThe only overlapping task is left/right direction classification. We conducted a quick evaluation of BAT model on AQA-PHY and observed 84.8% accuracy, compared to 99.4% achieved by our encoder + Qwen2. We note BAT’s performance may be affected by setup differences (e.g., sampling rate, training dataset).\n\nWe will clarify these.\n \n---\n**Regarding Audio Encoder Architecture**\n\n**Response:**  Our encoder largely follows Whisper-large for a fair comparison, with the main difference being the input: we use [magnitude, phase_sin, phase_cos] and corresponding convolution layers.\n\n---\n**Regarding LoRA Rank**\n\n**Response:** We kept the LoRA rank relatively low (=8) to preserve the LLM’s language ability, as most representation learning occurs in the audio encoder. This choice is aligned with the model-agnostic property of our model. \nHigher ranks showed minimal benefit but increased cost (see attached table). Due to time limits, this is a preliminary study. We will conduct a more detailed one.\n\n**Initial Study of LoRa Rank:** https://imgur.com/p5HfJMQ\n\n---\n**Regarding Similar Prompts**\n\n**Response:** Thanks for pointing this out. While Table 9 examples seem similar, our open-form prompts are more diverse. Examples include:\n- \"Could you evaluate the effect of multipath interactions within the given audio piece?\"\n- \"What is the extent to which the multipath effect manifests in the sound?\"\n\nFor training, LLMs learns to distinguish based on prompt: closed-form prompts yield fixed answers, while open-form ones encourage open-form answers. \n\nWe will revise these.\n\n---\n**Regarding Function Call**\n\n**Response:** Sorry for the confusion. We do not train the LLM to perform function calls. Rather, we envision active sensing can be triggered via function calls, like activating a speaker to emit pulses. we focus on showing that the LLM can process the resulting echo to estimate range. We will clarify this.\n\n---\n**Regarding Final Answer Extraction**\n\n**Response:** We use GPT-4o to extract the final answer.\n\n---\n**Regarding Related Simulator and SpatialSoundQA**\n\n**Response:** Thanks. We will discuss them.\n\n---\n**Regarding Typos Correction**\n\n**Response**: Thanks. We will fix them.", "claims_and_evidence": "I believe that the primary claim, that training on this acoustic QA dataset teaches the model physical awareness, is not entirely supported. It is true that the model learns to compute various acoustic quantities from raw audio. But I feel that for the model to be \"aware\" and \"understand physical phenomena\" -- rather than simply be an acoustic calculator -- it should be able to use acoustics to solve other, useful tasks. The authors suggest several possible tasks in Table 3 (Appendix A).\n\n(I understand this usage of \"aware\" and \"understand\" may be contentious, but I believe this interpretation is within the scope of the paper.)", "methods_and_evaluation_criteria": "The setup of the acoustic simulation is simple and modular. One (relatively minor) missing detail is the setup of the 32-layer Transformer in the audio encoder. It is stated that it is a 32-layer Transformer, but does everything match the Whisper Large encoder?\n\nI find a few issues with the evaluation:\n1. The accuracy of the intermediate steps is never evaluated.\n2. The dataset essentially teaches the LLM to be a calculator for acoustics, rather than a tool that learns to apply acoustics to solve another task (see \"Claims\" above). I believe the real-world dataset may help here. e.g. does the LLM know that a sound is coming from \"inside the car\" instead of simply \"out of LOS\"? (This example is from Figure 1a.) If so, that could indicate the LLM is able to use its \"awareness\" to answer questions.\n3. In Section 6.4, it is not clear what data the model is trained on. (i.e., is it trained on any real data, or only simulated?)\n4. There is no quantitative comparison against the BAT model and its Spatial-AST encoder (Zheng et al., 2024) - this could done be on AQA-PHY or on SpatialsoundQA.", "theoretical_claims": "I checked the formulation of the channel impulse response (1) - (8).", "experimental_designs_or_analyses": "The experiments (sole/merged testing, ablations, and real world) seem sound.", "relation_to_broader_scientific_literature": "The authors show that training a phase-aware audio encoder can allow LLMs to reliably compute acoustic quantities through natural language. Unlike related spatial audio processing work (BAT, Zheng et al. (2024), the proposed method does not require preprocessing all audio inputs to calculate their relative phase.", "essential_references_not_discussed": "There should be some discussion about how this work relates to other acoustic/physical simulators like SoundSpaces (Chen at al. 2022), as well as similar datasets SpatialSoundQA (Zheng et al. 2024).\n\nChen et al., 2022: [SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning](https://arxiv.org/abs/2206.08312)", "other_strengths_and_weaknesses": "Other strengths:\n1. The simulated-to-real experiment shows that knowledge learned form the AQA-PHY dataset can transfer well to real data.", "other_comments_or_suggestions": "1. Two typos:\n\n    a. Line 290, \"finial\"\n\n    b. In Section 6.2 Line 352, \"5 percentages improvement\" should be more like \"5 percentage points of improvement\".", "questions_for_authors": "1. The primary difference between the proposed encoder and Whisper is the addition of phase information. Given that, why does Whisper perform so much worse on LOS detection, even when it is the sole task? (Table 2)\n\n2. How was the LoRA rank determined (Table 8)? With higher ranks, is there a trade-off in what the audio encoder learns versus the LLM?\n\n3. Examining the example QA pairs in Table 9, some of the Close & Open question forms seem very similar, like for LOS Detection and Multipath analysis. Does the model not become confused about which form to output? And if they are so similar, what benefit does training on the Closed form give here?\n\n4. How is the LM trained to do function calls for Range Estimation? In Table 9 I do not see an example of this (and more generally I don't see examples of the \"active sensing\" that this task is supposed to give).\n\n5. How is the \"final answer\" extracted from the LM response, so that it can be scored with BCA, MAE, etc.?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wi4JpZxULZ", "reviewer": "Reviewer_3nj6", "summary": "The paper introduces an approach to teach physical awareness to large language models (LLMs) through sound, using a physics-based audio simulator to create the AQA-PHY dataset. The dataset consists of 1 million audio-based question-answer pairs capturing phenomena such as Doppler effects, multipath, and spatial acoustics. It proposes a novel audio encoder that leverages both magnitude and phase information, outperforming baseline methods in tasks including LOS detection, Doppler estimation, and range estimation, with promising results in both simulated and real-world settings.\n\n## Update after rebuttal\n\nThe rebuttal has addressed my major concerns. My rating has been increased.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer 3nj6\n\n\nWe thank the reviewer for the thoughtful and constructive feedback. We recognize that certain aspects of **our presentation may have led to confusion**, and we appreciate the opportunity to clarify the following key points:\n- Our method **does support spatial audio** using multi-channel input, both in simulation and real-world evaluation.\n- The use of **Whisper as a baseline** is justified and configured for a fair network structure comparison.\n  \nBelow are our responses.\n\n---\n**Regarding Concern about Spatial Audio**\n\n**Response:** We respectfully clarify that **our method explicitly support spatial audio.**\n\n- Our system support both mono and multichannel audio input. For example, Direction-of-Arrival (DoA) estimation is one of our core tasks, which is inherently a fundamental spatial audio problem.\n- Tasks, such as DoA estimation, are trained and evaluated using two-channel input. For example, in Table 9, the questions for DoA Estimation explicitly include two `<Audio>` placeholders, corresponding to two audio channels from two microphones. \n- Our simulator (see Section 4.4) fully supports multi-microphone spatial audio simulation, including spatialized impulse responses and inter-channel phase differences.\n- Additionally, our real-world evaluation includes a left/right detection task using recordings from multiple microphones.\n  \nWe apologize for any lack of clarity in our presentation and will revise the manuscript to more explicitly highlight our support for multi-channel spatial audio. \n\n\n--- \n\n**Regarding Whisper as a Baseline**\n\n**Response:** We use Whisper as a structure-only baseline, not to compare pretrained capabilities. All encoders are fine-tuned on AQA-PHY to ensure a fair comparison focused purely on architectural differences. Importantly, Whisper is widely adopted as a standard audio encoder baseline.\n\nThis setup allows for a fair comparison of encoder structure and their ability to adapt to physical reasoning tasks.\n\nWe will clarify this setting in the revised manuscript to prevent misunderstanding.\n\n--- \n\n**Regarding Concerned about Real-World Evaluation**\n\n**Response:** We agree that the scope of our real-world evaluation is currently limited, and we sincerely appreciate the reviewer’s suggestion. Our primary goal in including the real-world experiment was to establish a **proof of concept**—demonstrating that models trained purely on synthetic data (AQA-PHY) can generalize effectively to real acoustic environments, thereby validating the feasibility of sim2real transfer.\n\nWhile we acknowledge the importance of broader evaluations, collecting real-world data with reliable physical ground truth is non-trivial, often requiring specialized hardware and calibration. This challenge is especially pronounced for tasks involving **latent physical properties** (e.g., LOS, TDoA), which are not easily labeled.\n\nNevertheless, we fully agree that expanding the real-world evaluation would significantly strengthen the paper. We will revise the manuscript to clarify our current positioning and will make every effort to include additional real-world experiments.\n\n\n---\n\nWe appreciate the opportunity to clarify these points and hope our responses address the reviewer’s concerns. We hope these clarifications contribute to a clearer understanding of our work.", "claims_and_evidence": "The authors claimed that \"extensive evaluation demonstrating state-of-the-art performance in both simulated and real-world environments.\" However, I have concerns about (1) very limited real-world data testing was conducted; (2) comparison to the Whisper encoder is not fair.", "methods_and_evaluation_criteria": "Regarding the method, it is unclear how spatial audio is modeled, as the current approach appears to use only mono audio signals. To effectively teach physical awareness, incorporating spatial audio would be crucial, as it allows models to learn spatial relationships from acoustic environments. However, this aspect is not clearly described in the paper.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The real-world experiments presented are interesting; however, they are quite limited due to the small dataset used and the evaluation conducted in only one specific vehicle audio environment.", "relation_to_broader_scientific_literature": "This work extends existing multimodal LLMs like Llama3.1 and Qwen2 with physical awareness through sound.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Overall, I think the authors explore an interesting research direction. However, I have several critical concerns. I would like summarize them here.\n\n- Regarding the method, it is unclear to me if spatial audio is used in the audio encoder, as the current approach appears to use only mono audio signals. To effectively teach physical awareness, incorporating spatial audio would be crucial, as it allows models to learn spatial relationships from acoustic environments. However, this aspect is not clearly described in the paper.\n\n- Whisper encoder is not a valid baseline. The authors build the dataset using sources from Audioset, which consists of ambient sounds, music, and speech. However, Whisper was trained only using speech data. I do not think it is a good baseline that can handle universal sound sources. \n\n- The real-world experiments presented are interesting; however, they are quite limited due to the small dataset used and the evaluation conducted in only one specific vehicle audio environment. These experiments cannot validate if the trained model using the collected synthetic data can be effectively transferred to handle real-world sounds. The authors are suggested to collect a real-world testing dataset with more diverse sources.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "The authors are encouraged to address the raised major concerns.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "Dok2vizeIB", "reviewer": "Reviewer_yVZE", "summary": "The paper proposes teaching Large Language Models (LLMs) to understand the physical world through sound. The authors created a physics-based simulator to generate a large audio dataset, AQA-PHY, annotated with physical phenomena like the Doppler effect and spatial relationships. They also developed a novel audio encoder that captures both the magnitude and phase of sound. By connecting this encoder to existing LLMs, they demonstrated the feasibility of enabling LLMs to achieve physical awareness in tasks such as detecting line-of-sight, estimating Doppler shifts, and localizing sound sources in both simulated and real-world environments.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer yVZE\n\nWe sincerely thank the reviewer for the thoughtful and constructive feedback. We are encouraged by your recommendation! Our responses are as follows:\n\n---\n**Regarding Semantics Understanding of Audio**\n\n**Response:** We agree that semantic understanding is an important dimension of audio perception. However, our current work focuses on teaching LLMs physical awareness—specifically, reasoning about phenomena such as Doppler shift, reverberation, line-of-sight, and time-of-flight. These capabilities are orthogonal and complementary to traditional audio tasks like classification or source separation, which have been extensively studied in existing models (e.g., Qwen-Audio, AudioGPT, Whisper, LTU, SpeechGPT).\n\nOur framework is fully compatible with semantic tasks. Since our simulator builds on AudioSet clips with known semantic labels, it can be naturally extended to support joint reasoning over both what a sound is (its semantic content) and how it propagates through space (the physical channel). We consider this a promising direction and will add it to the Limitations and Future Work section.\n\n\n---\n\n**Regarding Absence of Qualitative Dialogue Examples**\n\n**Response:** Thanks for you suggestion. We would like to clarify that the primary scope of this paper is to validate the feasibility of enabling LLMs to understand physical phenomena through sound. Accordingly, we focus on single-turn QA tasks that directly test the model’s ability to reason about physical dynamics.\n\nMeawhile, we do preserve the natural language generation capabilities of the LLM through open-form QA, as demonstrated in Table 9. These examples show that the model can produce interpretable, step-by-step explanations of physical effects such as range estimation or LOS detection.\n\nRegarding multi-turn conversation, we agree this is a promising direction for future work. Notably, as shown in Appendix F, our prompt format already adopts a conversation-like structure, which we currently use in a single-turn setting. This format could be naturally extended to construct multi-turn dialogue datasets that allow LLMs to reason about physical phenomena over a series of interactions. \n\n\n---\n\n**Regarding References Not Discussed**\n\n**Response:** Thank you for suggesting the reference \"Can Large Language Models Understand Spatial Audio?\" (Tang et al., 2024). We will include this reference and properly position our work relative to it.", "claims_and_evidence": "1. The paper shows that it is possible to teach LLMs to reason about the physical world through sounds using experiments such as line of sight detection and doppler effect estimation.\n2. Experiments are done on real-world scenarios to show the effectiveness of the proposed method in the real world.\n3. Physical awareness can have a broad definition. The paper does not show tasks such as source separation or audio classification. Can the LLM understand the semantics of the sound?\n4. There are no qualitative examples showing conversions between the LLM and end users.", "methods_and_evaluation_criteria": "1. The experiments done on tasks such as doppler effect estimation and range estimation are meaningful to test if LLMs can reason about the physical world using audio input.\n2. There are no experiments to show if LLM can understand the underlying semantics of the audio. Can it tell the difference between a car horn and a rooster crowing?", "theoretical_claims": "NA", "experimental_designs_or_analyses": "The experimental design is meaningful and captures real world scenarios.", "relation_to_broader_scientific_literature": "I am not very familiar with this domain to be able to connect with the existing broader scientific literature. However, the impact statement does provide an insight where this work can plug into existing works. For instance, in embodied AI applications allowing robots to interact with various environments through sound.", "essential_references_not_discussed": "Tang, Changli, et al. \"Can Large Language Models Understand Spatial Audio?.\" Proc. Interspeech 2024. 2024.", "other_strengths_and_weaknesses": "1. The paper proposed a novel audio encoder that processes magnitude and phase of audio signals outperforming Whisper.\n2. The authors created a novel audio question-answering dataset to fine-tune LLM to teach them physical awareness.", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4WGUG6UsDR", "reviewer": "Reviewer_9yAo", "summary": "This paper proposes a method to incorporate physical awareness into LLMs using audio signals, focusing on fundamental acoustic phenomena such as the Doppler effect, multipath reflections, and direction-of-arrival. The authors introduce a specialized simulator that generates large-scale training data for these phenomena by combining real-world audio clips with artificially controlled channel parameters. They then build an AQA-PHY dataset and design a phase-aware audio encoder to feed signals into two different LLM backbones. Through simulation-based experiments and a limited real-world test in a vehicle cabin, the paper demonstrates that the resulting system can detect line-of-sight, measure Doppler shifts, localize sound sources, analyze reverberation, and even perform active distance measurements.\n\n## Update after rebuttal\nAfter the discussion, I raised my rating since the authors addressed my concerns.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer 9yAo\n\nWe sincerely thank the reviewer for the thoughtful and constructive feedback. We are encouraged by your recommendation! Our responses are as follows:\n\n---\n\n**Regarding Simulator-to-Reality & Parameter Selection**\n\n**Response:** Our simulation approach follows the principle: **maximize parameter randomization while preserving physical realism**. We systematically randomize channel parameters like path number, strength, delay, and reverberation decay to ensure broad coverage of real-world conditions rather than fitting specific environments.\n\nTake in-cabin case as an example. \n- Vehicle interiors typically exhibit 3–5 strong reflections from surfaces like side windows, windshields, dashboards, and ceilings. Our simulator randomly set the number of reflection paths and randomizes their amplitudes and delays to replicate such conditions. \n- For reverberation, real-world RT60 values in vehicles range from 0.2–0.5 s. Our simulator spans a much broader RT60 range of 0.05–2.0s (Table 5), covering cases from 'dry' environments to highly reverberant spaces. \n  \nSimilarly, we randomize relative velocities from -50 to +50 m/s for Doppler simulation, and directional parameters (e.g., DoA angles from 0–180° and inter-microphone spacing from 8–15 cm) to encompass common real-world sensor setups.\n\nThis extensive randomization strategy intentionally produces training conditions more diverse than any individual environment, enhancing robustness and generalization. Our real-world experiments in a vehicle cabin (Section 6.4) confirm that models trained on these diverse simulations transfer effectively, even without environment-specific tuning. \n\nWe will clarify these in the revision.\n\n\n\n---\n\n**Regarding Real-World Evaluation Scope:**\n\n**Response:** We agree that the scope of our real-world evaluation is currently limited, and we sincerely appreciate the reviewer’s suggestion. Our primary goal in including the real-world experiment was to establish a **proof of concept**—demonstrating that models trained purely on synthetic data (AQA-PHY) can generalize effectively to real acoustic environments, thereby validating the feasibility of sim2real transfer.\n\nWhile we acknowledge the importance of broader evaluations, collecting real-world data with reliable physical ground truth is non-trivial, often requiring specialized hardware and calibration. This challenge is especially pronounced for tasks involving **latent physical properties** (e.g., LOS, TDoA), which are not easily labeled.\n\nNevertheless, we fully agree that expanding the real-world evaluation would significantly strengthen the paper. We will revise the manuscript to clarify our current positioning and will make every effort to include additional real-world experiments.\n\n\n\n---\n\n\n**Regarding Active Sensing in Complex Scenarios**\n\n**Response:** We appreciate the reviewer's question about active sensing in cluttered environments. Our approach maintains strong performance in such scenarios for several reasons:\n\nMultiple reflectors create temporally separable echoes in the received signal. For range estimation, we focus on the first-arriving echo (nearest object). Thanks to low sound speed, with a 8 kHz chirp bandwidth, the reflections can be separated with a resolution of ~2.1 cm ($\\frac{c}{2B}$​ [1]), allowing distinct identification of different reflecting objects.\n\nOur phase-aware encoder effectively captures temporal information, with phase information being particularly sensitive to small time shifts. This enables identification of the earliest echo even among multiple reflections from surfaces like windows, dashboards, and ceilings.\n\nRegarding Doppler effects, we observe minimal performance degradation under moderate motion. Chirp signals (i.e., FMCW) inherently resist Doppler shifts due to their linear time-frequency modulation. This is why it is used in radar systems for tracking fast-moving objects [1,2]. In our experiments, Doppler effects have negligible impact on range estimation accuracy.\n\nWe will include these into our revised manuscript.\n\n**References:**\n\n[1] S. Rao. Introduction to mmWave Sensing: FMCW Radars. TI mmWave Training Series, 2017.\n\n[2] W. Mao, J. He, and L. Qiu. CAT: High-Precision Acoustic Motion Tracking. Proc. ACM MobiCom, 2016.\n\n\n---\n\n**Regarding Impulse Response Comparison:**\n\n**Response:** Thanks. We will include visual comparisons between simulated and real CIR in the revision.\n\n**Preliminary examples of these comparisons** (LOS and NLOS) can be viewed at (We search for the most similar simulated CIR to the real case. Left: Real CIR, Right: Matched Simulated CIR):\n\n1. https://imgur.com/aRQoP6H\n2. https://imgur.com/xZ795Cb\n3. https://imgur.com/XIYw2WX\n4. https://imgur.com/5PfI8az\n\n\n---\n\n**Regarding Related Work on Acoustic Localization or Radar work**\n\n**Response:** Thanks. We will dicuss them in the revison.", "claims_and_evidence": "The main claim is that LLMs can be taught physical awareness by training on synthetic and limited real-world audio data. The authors support this claim with consistent accuracy and error metrics across multiple simulated tasks (e.g., line-of-sight detection reaching over 0.92 accuracy, Doppler estimation with low mean absolute error). They also conduct a small real-world test, showing the approach can generalize to actual environments, though on a limited scale. The evidence is mostly numerical, with relevant quantitative metrics across tasks. The real-world demonstration solidifies the claim, but more extensive in-the-wild evaluations might be needed to confirm broader robustness.", "methods_and_evaluation_criteria": "The authors use a physics-based simulator to synthesize channel impulse responses, which are convolved with audio signals to generate corresponding question-answer pairs. Each QA pair is either a closed-form question (e.g., classification or numerical estimation) or an open-ended reasoning question. This approach appears well-structured for the goal of training an LLM to interpret and reason about acoustic phenomena. Their choice of standard metrics aligns with typical acoustic analysis benchmarks.", "theoretical_claims": "The paper does not delve deeply into new theoretical proofs, but it provides a detailed derivation of how Doppler-shifted signals can be modeled by resampling. The channel modeling steps seem correct and are explained thoroughly. I did not find formal proofs that could be challenged.", "experimental_designs_or_analyses": "1. Experiments address five tasks, which I think are comprehensive simulations. However, the real-world experiment is small-scale -- only in a single vehicle scenario. It would be good to see expansions into other real-world environments (e.g., open areas, different types of indoor spaces).\n2. Section 6.4 (Real-World Experiments) would benefit from clearer explanations of how well the training data’s synthetic conditions match (or differ from) the actual in-cabin sound environment.\n3. The formulas in Appendix B (channel simulator design) are mostly coherent, but I wonder if there could be more detail on how the parameters (e.g., decay factors in reverberation, time-varying delays in Doppler) transfer to real-world conditions.", "relation_to_broader_scientific_literature": "Integrating audio-based physical reasoning into language models builds on lines of work in multimodal LLM development, such as prior methods that fuse audio encoders with text decoders (e.g., Qwen2-Audio-Instruct, Audio Flamingo, etc.). However, the unique twist here is on physical channel simulation rather than purely content-based audio tasks. This can be applied to some robotics/embodied-AI that gives LLMs actual or synthetic sensor data to reason about physical surroundings. The paper thus extends existing multimodal approaches with a new focus on physical channel cues.", "essential_references_not_discussed": "The current references are sufficient. However, the authors might consider adding more connections to the classic acoustic localization or radar-based distance-estimation work.", "other_strengths_and_weaknesses": "Addtional Weaknesses:\n1. The proposed model is tested on a relatively small set of tasks (five in total). Although these tasks are well-motivated, it would be nice to see an even broader range of physically grounded tasks (e.g., multiple source separation, room geometry estimation, etc.).\n2. While the paper claims LLMs can integrate active sensing, the experimental coverage is a bit brief. If distance measurement is a key highlight, it could be valuable to show more step-by-step error analysis in complex real scenarios (moving objects, multiple reflectors).", "other_comments_or_suggestions": "A table or figure that directly compares the simulator’s impulse response to measured impulse responses from real rooms or real vehicles would help illustrate simulation fidelity.", "questions_for_authors": "1. Could you provide more details on how the simulator’s parameters (e.g., decay factors for reverberation or velocity for Doppler) were selected or tuned to align with real acoustic measurements?\n2. For the active sensing (range estimation), how consistent were the echo results across different real objects and distances? Does clutter (many reflectors) degrade performance significantly?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel approach to teaching large language models (LLMs) to understand the physical world through sound. The authors introduce AQA-PHY, a large-scale audio dataset generated using a physics-based simulator and annotated with physical phenomena such as the Doppler effect and line-of-sight relationships. They also develop a new audio encoder that captures both magnitude and phase information, which is then connected to LLMs to enable physical reasoning tasks like Doppler shift estimation, sound source localization, and line-of-sight detection in both simulated and real-world settings. The authors addressed reviewers’ concerns thoroughly in their rebuttal, providing additional experiments and clarifications. All reviewers expressed satisfaction with the response and recommended acceptance.
Decision: Accept (poster)