Acoustic Volume Rendering for Neural Impulse Response Fields
Authors: Zitong Lan
Chenhao Zheng
Zhiwei Zheng
Mingmin Zhao
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Acoustic signals
Room impulse response
Neural radiance field
Wave propagation
Primary area: speech_and_audio
TL;DR: This paper introduces acoustic volume rendering for impulse response synthesis and inherently enforces multi-pose consistency.

Abstract: Realistic audio synthesis that captures accurate acoustic phenomena is essential for creating immersive experiences in virtual and augmented reality. Synthesizing the sound received at any position relies on the estimation of impulse response (IR), which characterizes how sound propagates in one scene along different paths before arriving at the listener position. In this paper, we present Acoustic Volume Rendering (AVR), a novel approach that adapts volume rendering techniques to model acoustic impulse responses. While volume rendering has been successful in modeling radiance fields for images and neural scene representations, IRs present unique challenges as time-series signals. To address these challenges, we introduce frequency-domain volume rendering and use spherical integration to fit the IR measurements. Our method constructs an impulse response field that inherently encodes wave propagation principles and achieves state of-the-art performance in synthesizing impulse responses for novel poses. Experiments show that AVR surpasses current leading methods by a substantial margin. Additionally, we develop an acoustic simulation platform, AcoustiX, which provides more accurate and realistic IR simulations than existing simulators. Code for AVR and AcoustiX are available at https://zitonglan.github.io/avr.

Acoustic Volume Rendering for Neural Impulse Response Fields
Zitong Lan'
Chenhao Zheng²
Zhiwei Zheng'
Mingmin Zhao'
University of Pennsylvania
University of Washington
Abstract
Realistic audio synthesis that captures accurate acoustic phenomena is essential for creating immersive experiences virtual and augmented reality. Synthesizing the sound received at any position relies on the estimation of impulse response (IR), which characterizes how sound propagates in one scene along different paths before arriving at the listener's position. In this paper, we present Acoustic Volume Rendering (AVR), a novel approach that adapts volume rendering techniques model acoustic impulse responses. While volume rendering has been successful in modeling radiance fields for images and neural scene representations, IRs present unique challenges as time-series signals. To address these challenges, we introduce frequency-donee volume rendering and use spherical integration to fit the IR measurements. Our method constructs an impulse response field that inherently encodes wave propagation principles and achieves state-of-the-an art performance in synthesizing impulse responses for novel poses. Experiments show that AVR surpasses current leading methods by a substantial margin. Additionally, we develop an acoustic simulation platform, AcoustiX, which provides more accurate and realistic IR simulations than existing simulators. Code for AVR and AcoustiX are available at httph: vtttps://zitonglaat www.ggttthbiooo
Introduction
Our acoustic environment shapes every sound we hear from the crisp echoes bouncing through hallways to the layered resonance of a symphony filling a concert hall. These spatial characteristics not only define our daily auditory experiences but also prove crucial for creating convincing virtual worlds [15] 60 At the core of these spatial characteristics is the impulse response (IR), which captures the complex relationship between emitted sound and what hear. Like unique acoustic fingerprint, the impulse response varies across different positions, encoding how sound waves interact with the environment through reflection, diffraction, and absorption [22] 41]. We can recreate the acoustic experience at any position by convolving the corresponding impulse response with any desired sound sources (e.g., music, speech). Given its foundational role in spatial audio synthesis, understanding and modeling the spatial variation of impulse responses in acoustic environments has emerged as a critical challenge and attracted increasing research 53].
attention 2 26 27
29
35 36
Current approaches construct neural impulse response field learned mapping that generates impulse responses given the emitter and listener poses. To model the high spatial variation of impulse responses, existing methods either fit neural network directly learn the field (29 4r rely on audio-visual correspondences to learn mappings from vision [26] (7]. While these methods can approximate the general energy trend, they struggle to capture the detailed characteristics of impulse responses, leading to incorrect spatial variation of impulse responses (Fig. ID.
We argue that key barrier to achieving better performance is the absence of physical constraints that inherently enforce consistency across multiple poses. Without such physical constraints, the network
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

INRAS NAF
AVR (Ours)
Ground truth
Phose
f
b)
Amplitude
Low
Figure 1: Left: From observations of the sound emitted by speaker, our model constructs an impulse response field that can synthesize observations novel listener positions. Right: Visualization of spatial variation of impulse responses on McshRIR (0). The synthesized impulse responses different locations are transformed into the frequency domain, where we visualize phase and amplitude distributions at specifie wavelength (Im).
tends overfit the training data and show poor generalizability. The received impulse response fundamentally arises from sound waves propagating through space, combining direct transmission with environmental reflections. This physical insight motivates us develop framework that inherently encodes wave propagation principles into the modeling of impulse response fields.
In this paper, we introduce Acoustic Volume Rendering (AVR) to model the field of acoustic impulse responses. Our approach draws inspiration from Neural Radiance Fields [33], which has demonstrated remarkable success in modeling 3D scenes by representing light transport through volume rendering. However, acoustic waves present several fundamental challenges that require adaptations to the volume rendering framework: First, acoustic impulse responses, unlike light transmission, inherently time-series signals, with acoustic waves from different locations reaching the listener varying delays. The issue is further compounded when dealing with discrete impulse responses sampled in the real world. Second, impulse responses exhibit high spatial variation, in contrast images where neighboring pixels show strong correlations. This characteristic makes network optimization particularly challenging [43] 45). Finally, unlike cameras that capture light with precise directional information (i.e.. pixels), microphones capture combined signals from all directions.
To address these challenges, we convert impulse responses from the time domain to the frequency domain with Fourier transforms and perform volume rendering in the frequency domain. We apply phase shifts to the frequency-doman domain impulse responses to account for time delays, bypassing the limits of finite time domain sampling. The frequency-domen lower spatial variation, facilitating network optimization. To account for signals from all possible rays uniformly across sphere and spherical integration synthesize the impulse response measurements. Additionally, this design enables personalized audio experience by integrating individual head-related transfer functions (HRTFs) [57] into spherical integration at inference time. Our evaluation results show that AVR outperforms existing methods by a large margin and can zero-shot render binaural audio (Sec. 4.3)
representation also exhibits
directions,
both simulated and real-world datasets [10 20
In parallel with AVR, we develop AcoustiX, acoustic simulation platform that generates more physically accurate impulse responses compared to existing simulators. While current simulators often introduce significant errors in signal phases and arrival times, AcoustiX
Time-of-flight )
SoundSpaces
Time- Time-of-fllge flightt (ms)
AcoustiX (Ours)
9.
6 6 e
Simulation
Simulation
Ground truth
Ground truth
produces impulse responses that better match the physical properties of real-worke acousties. Fig. dedmonstatss the inaccuracies in impulse responses generated by SoundSpaces 2.0 9 Some existing simulators assign random phases when generating impulse responses 9 [40]. which fails to reflect real-world acoustic be- havior [4]. Since current research in impulse response synthesis heavily relies on simulated datasets [IO 29) 44), these simulation inaceu- racies impede progress in the field. To
Distance speaker (m)
Distance to speaker (m)
Figure 2: AcoustiX for improved acoustic simulation. Time-of-flight indicates how long it takes for emitted sound to reach a listener. With sound traveling at a con- stant speed, the time-of-arrival should be proportional to the emitter-listener distance. While SoundSpace 2.0 simulations show significant time-of-flight errors, par- ticularly at short emitter-listnner distances, Acoustix produces more accurate arrival times. All simulations are performed in the Gibson Montreal room (56) with direct line-of-sight between emitter and listener.

---

relies audio address these limitations, we develop new simulation platform based on the Sionna ray tracing engine [16] and incorporate acoustic propagation equations to resolve the aforementioned issues. Similar to SoundSpaces 2.0, AcoustiX supports acoustic simulation in both user-provided 3D scenes and variety of existing 3D scene datasets 56].
In summary, this work makes the following contributions:
We introduce acoustic volume rendering (AVR) for the neural impulse response field inherently enforce acoustic multi-view consistency. We introduce a frequency-domain rendering method with spherical integration address the challenges associated with acoustic impulse response modeling.
We demonstrate that AVR outperforms existing methods by a large margin in both real and simulated datasets. AVR also supports zero-shot and personalized binaural audio synthesis.
We develop AcoustiX, an open-source and physics- impulse response simulator that provides accurate time delays and phase relationships through rigorous acoustic propagation modeling.
2
Related Work
Impulse Response Modeling. Traditional impulse response modeling 2 32 49
encoding, which encodes collected data and spatially interpolates the impulse response for unseen positions [2 32). However, this approach comes with large memory costs (29] and struggles to
generate impulse responses with high fidelity [10]. Machine learning techniques have been integrated in recent years to enhance the quality of synthesized impulse responses B5 36 37]. For instance,
generative adversarial networks (GANs) have been utilized for more realistic acoustic synthesis [36 37.. As implicit representations have become more popular, several works (26
in recent years proposed neural implicit acoustic fields and achieved state-of-the-are art performance. However, these learning-baid based methods still produce unsatisfying waveform shapes and show weakness in novel impulse response synthesis [10]. To this end, our tearning-based approach further integrates wave propagation principles 222, synthesizing high-fidelity impulse responses.
27
29 44
Neural Fields. Since the success NeRF [3], the concept of neural fields has been expanded comprehensively. Initially, incorporating depth supervision [12 38 51) into radiance fields proves helpful for novel view synthesis. Some following studies adopt occupancy [59] or signed distance
functions [I] [54 62], instead of density, represent scenes. Later, integrating volume rendering with measurements from sensors of other modalities, such as time-of-flight sensors [3] LiDAR 18 35)5
and structured light imaging [42), has achieved great performance. Extensions 28 61 have further
modified volume rendering to model andio-frequecy signals.
Acoustic Simulation. Acoustic simulation primarily relies on either wave-based or geometric approaches to approximate sound propagation in indoor environments. While wave-based meth- ods 77 14 46, 47 [52) are generally more precise for frefureyencyy sound, they require significant computation for high-frequ fre.ueneygneee signal simulation. For faster acoustic simulation, geometric ap- proaches have gained considerable attention. These methods, such as image source [1 5 39 and ray tracing S commonly-usee simulation platforms [ 9 suffer from inaccuracies in time-of-flight calculation and phase simulation for sound propagation. AcoustiX ensures physics-basee based accuracy, facilitating further research on cooustic-relate topies.
(6),, are often used practical applications like virtual reality. However, some
3 Method
Our objective is to learn an impulse response field for one scene to synthesize impulse responses for the unseen emitter and listener poses. The impulse response h(t) quantifies the received signal at specified listener location Pt oriented by wi. resulting from an impulse emitted from Pe with orientation w.. It encompasses how sound propagates within a specific scene. We begin by revisiting fundamental principles of acoustic wave propagation. We then introduce our impulse response field and our frequency-domain acoustic rendering. Lastly, we discuss the implementation specifics. In addition, summarize the key features of our simulator AcoustiX.
3

---

3.1 Acoustic Wave Propagation Primer
We first explain the simplest form of acoustic propagation highlight two basic properties of sound propagation: time delay and energy decay. Assuming an omnidirectional emitter at Pe transmits Dirac delta pulse 5(t at time t uniformly into open space, the resulting signal at listener pt is given by [22
8 (t - -),, where T Pt Pella
h(t)
(1)
1p Pe pell2
and v denotes the velocity of sound. The orientations of both the emitter we and the listener w/ are ignored under the omnidirectionality assumption. We note that the listener captures a dime-delayed emitted signal, with an amplitude decay inversely proportional to the distance traveled. Additionally, the phenomenon can be alternately represented in the frequency domain with the Fourier Transform:
H(f)= F
h(t)e 12 dt
j2m
(2)
1p: Pell
With this representation, the time delay observed in the time domain manifests as phase shift (e (e-J2aft) in the frequency domain.
3.2 Acoustic Volume Rendering
In real scenario, sound emitted by source undergoes complex interactions with the geometric structures of the environment. Each location the scene may absorb some energy from the incoming wave, resulting in signal absorption; it may also reflect and scatter the wave, leading to signal re-transmission. To model these complex effects, AVR represents scene as a field: given an emitter location Pa and its orientation we, the network F outputs two key acoustic properties for any point p in space given a direction wt
Fe: (e. + s(t).
where a represents acoustic volume density and the time-varying parameter s(t) models the acoustic signal transmitted out from the location p in direction -w, including both initial emission and subsequent re-transmission.
With this parameterization, we now render the signal (t) received at a listener position p from direction w, assuming the emitter is fixed and the impulse is emitted at time t=0. Similar to volume rendering for light, our process adopts volume rendering to accumulate the signals emitted from all locations along the ray p(u) = p + w, with predefined near and far bounds u, and uy. Differently, our approach also accounts for time delay and energy decay in acoustic signal propagation and performs alpha composition for time signals, resulting in acoustic volume rendering equation:
1 - to L(u)o(p(u))ss(t - - p()( w)du, where L(x) =exp (- [(p()))dz
he
(4)
Note that each emitted signal s(t ";p(u), w) associated with a time delay u. for the non-neglig sound propagation time, ensuring that the signal received by the listener at time originates from location p(u) at the earlier time n. To account for energy decay, We apply a factor 1 to all the signals along the ray, independent of their emission time. Since all signals originate from the impulse transmitted at time 0, the traveled distance of any signal received at time t is tu, whether it's from the original emission or a re- re-transmitted signal.
". This delay accounts
Listener receives signals from all directions, influenced by its gain pattern. The signal obtained from a single ray can not represent the whole received impulse response. To account for this, the final impulse response captured by the listener is a combination of signals from all directions:
h(()= G(w)
(5)
where G(-) represents the listener gain pattern that characterizes the directivity of the listener, and Ω denotes the complete sphere directions from which the listener receives signals. The whole acoustic rendering process is also shown in 3

---

Ray
sc(()
ss(t ds/1)
ds.
Delayed
de
sc(e)
sa(t da/w)
(u)) ydu
Fe
Delayed
at()
st dr/u)
h
d,
Delayed
h(t) G(w)hu(t)dw
Acoustic volume rendering
Spherical integration
Figure 3: Acoustic Rendering pipeline. We sample points along the ray that is shot from the microphone and query the network obtain signals s(t) and density a. Time delay (5) is applied account for the wave propagation. After that, we combine signals and densities to perform acoustic volume rendering each ray to get the directional signal (hos-(4). We integrate along the sphere to combine signals from all possible directions with gain pattern ((w) to obtain the final rendered impulse response (()).
3.3 Acoustic Volume Rendering in the Frequency Domain
While the above method renders continuous impulse response, the actual signal collected in the real world is discrete, sampled with a fixed interval I'. This discretization converts the target impulse response from h(t) hiln). where hi|n]= h(n2"). Accordingly, s[n] at these discrete timestamps. However, this discrete sampling presents a fundamental challenge in acoustic volume rendering. To capture the signal from one direction, h [n]. we need evaluate the time-delayed signal s(nT 5). The key issue arises when is not multiple of the sampling interval T, i.e. the required timestamps for delayed signals fall between our discrete samples, making accurate rendering difficult in the time domain. While one could add timestamp as an additional network input interpolate between samples, this would require multiple network queries for each point, severely impacting rendering efficiency.
train neural networks to output
We address this challenge by reformulating the problem in the frequency domain. A key insight is that time delays in the signal correspond to phase shifts in the frequency domain (Eq. 2 This allows us achieve arbitrary time delays E by transforming the predicted signal s(nl';p(u)) w/ into frequency domain f {s|nT'; pefnT"ppuu,,,.....) p(u), w]} and applying the corresponding phase shift, regardless of whether aligns with the sampling grid. Specifically, the delayed signal in the frequency domain can be obtained via:
4: F s(nnT -ooT-=;p(uu.e)
F (s|nT:p(u)) w| jBrfu//
(6)
The linearity of the Fourier Transform & allows to extend acoustic volume rendering to the frequency domain with the same alpha composition (i.e., integration) process:
H 3
cuy
L(u)o(p(u)) - - p(u),, du.
tu
Here, the multiplication with energy decay factor in Eq. becomes the convolution with & { the frequency domain. Eq.7ean can be regarded as Eqq. Fin the frequency domain by applying discrete Fourier Transform on both sides of the equation. Similarly, the final impulse response the frequency domain can be formulated analogously to 5
H[ / Huw)
(8)
Finally, time-domain impulse response h [m]] can be obtained through inverse Fourier Transform.
3.4 Sampling Rays and Points
To handle the integration in Eq. and Eq. g our sampling strategy includes both ray sampling over sphere and point sampling along ray. We perform ray sampling by selecting Ne azimuthal directions
5

---

and No elevational directions, resulting in No x N& distinct orientations that uniformly cover the sphere. For ray along one of the directions, point sampling conducted by evenly placing N, points between predefined near and far bounds, similar to [6]. Consequently, the network is queried Ng x N x N, points for the synthesis of an impulse response. We refer readers Appendix A for the details of our sampling strategy.
3.5 Optimization
We supervise the synthesized impulse responses Hff] and h[n] alongside the ground truth [f] and h" [n] in both frequency and time domains. We emphasize supervision in the frequency domain since the responses exhibit smaller local variability. In the frequency domain, Eypec measures spectral loss by comparing the real and imaginary components.
Spee = Re(H)) - Re(H)))lz + [[lm(H) - Im(
(9)
We also supervise the amplitude and phase of the synthesized signals with Lamp and Lphase
Camp
||| H - 111,,
(10)
Cphase =
||cos( (H) cos < || + ||sin( (H) sin(
(11)
For time domain signals, Lime employed to encourage amplitude consistency.
Lime
||A h" 11|
(12)
Our total loss is a linear combination of the above loss with different weights, including a multi- resolution STFT loss Lattt (58) and energy loss Lenergy similar in (30]:
Lintal
Lapee Aamphamp + Axphaee pphase + Atime Limme + Aan + Venergy fenergy:
(13)
3.6 Simulation platform
AcoustiX uses Sionna ray tracing engine (16). We modify the ray tracing in terms of ray interactions with the environment to support acoustic impulse response simulations. The simulator supports various ray interactions with the environment. Each material in the scene is assigned with frequency- dependent coefficients. This enables the tracing of cumulative frequency responses for each octave band to accurately simulate the impulse response. Room models can be created using Blender and exported as compatible XMI. files for our simulation setup. AcoustiX iX also supports the import of More details can be found in Appendix [
3D room models from the iGibson dataset 24 56)
Experiments
Implementation Details. The input to our model is emitter's pose (position Pe R3, direction We E ) and a 3D query point's pose (p E R3 w E RR)), The model outputs the corresponding density E IR and discrete time signal s[n] E RR at that query point. We first encode all input vectors into high-dimensional embeddings using hash grid [34]. The encoded input embeddings are then passed into 6-layer MLP. The first 3 layers of MLP take input locations (pe. p) and predict the density a and a 256-dimensional feature. The feature and encoded directions (We,w) are then concatenated and passed into the last layers, which outputs the signal sequence s(n)..
The sampling numbers used in the experiments are No = 80, No- = 40, and N, = 64. We set the weights of loss components to be Xamp Aphase = 0.5, Atme = 100, Aga = =1, Aeneryy = 5. We train our model for 200 epochs for each scene. We use Adam optimizer with a cosine learning rate scheduler that starts at learning rate 10 and decays to 10-4, The optimization process takes 24 hours single NVIDIA L40 GPU.
Evaluation Metric. We use comprehensive metrics to assess the quality of our method. Following 10 29), we measure the energy decay trend by Clarity (C50), Early Decay Time (EDT), and Reverberation Time (T60). For the measurement of the correctness of waveform shape, prior works only consider the amplitude in the frequency domain (e.g. STFT error) to assess the performance, which we argue only indicates part of the waveform information: the transformed frequency signal a complex number that jointly defined by amplitude and phase. We therefore also include the frequency- rrequency-domain domain phase error in measurement: the LI norm of the error in the cosine and sinc

---

McshRIR
RAF-Purnished
RAF-Emply
Method
Phase
Amp.
Env.
T60
C50
EDT
Phase
Amp.
Env.
T60
C50
EDT
Phase
Amp.
Env.
T60
C50
EDT
AAC-ncarest
1,47
0.91
1,40
8.6
2.20
58.8
1.60
1.09
4.83
13.0
3.41
73.5
1.60
1.09
4.83
13.0
3.41
73.3
AAC-lincar
1.44
1.42
2.29
58.9
1.60
0.99
3.81
12.4
3.65
90.2
1.10
5.22
13.1
3.25
71.5
Opus-nearee
1.45
0.72
1.37
5.2
1.26
35.7
1.60
1.19
5.35
14.4
3.78
80.3
1.16
4.58
13.3
4.25
100.6
Opus-linear
1.43
0.69
1.37
6.9
1.83
49.3
1.60
1.47
5.74
13.1
3.55
77.8
1.59
0.95
4.26
12.7
3.94
95.5
NAF
1.61
0.64
1.59
4.2
1.25
39.0
1.62
0.93
5.34
7.1
0.98
20.6
1.62
0.85
4.67
8.0
1.22
26.3
INRAS
1.61
0.77
1.85
3,4
1,47
40.7
1.62
0.96
6,43
6.9
1.08
21.4
1.62
0.88
4.72
7.6
1.21
25.8
AVR (Ours)
0.85
0.54
1.15
3.9
0.92
35.1
1.58
0.75
4.52
5.0
0,95
17.9
1.58
0.67
3.96
5.5
1.04
23.3
Table 1:
Quantitative results real datasets (0.1s IR duration). We report comprehensive metrics (lower
better) including phase error, amplitude error, envelop error( T60 reverberation time (%), clarity CS0 (dB), and Early Decay Time (millisecond). AVR outperforms existing methods by substantial margin. We note that the random phase error is 1.62, which means all carning-based methods except ours fail to learn valid phase information.
IR
Room 3D
Room 2D
Amplitude
Phase
Amplitude
Amplitude
INRAS
NAF
AVR (Ours)
Ground truth
Figure 4: Visualization of spatial signal distributions. We compare the spatial signal distributions between ground truth various methods on the MeshRIR dataset and two simulated environments. While NAF and INRAS fail to capture the signal distributions, our model can estimate amplitude and phase distributions accurately.
components of the phase. Besides the frequency domain, also measure the time-domain impulse response signal accuracy by calculating the LI error in the envelope, denoted as envelop error. Please refer Appendix B for the detailed definition of each metric.
Baselines, We compare with both learning- based methods using neural implicit representations and traditional methods. NAF(29) is the first method that uses neural implicit representation to model the impulse response field. It uses an MLP to predict the spectrogram of the impulse response signals. Another method INRAS 44 disentangles the sound transmission process into three different learnable modules. Different from NAF, INRAS models the impulse response in the raw time domain.
We also implement traditional audio encoding methods AAC [6 and Opus (50) and adopt the same setting as 29
4.1 Results on Real World Datasets
We evaluate model's performance the datasets collected from real scenes. We adopt two commonly used room impulse response datasets: MeshRIR (20] and Real Acoustic Field [IO]. MeshRIR collects monaural impulse response in cuboidal room. We use $1-M3969 dataset split featuring a fixed single speaker for evaluation and the impulse responses are resampled to 24 KHz sampling rate. Real Acoustic Field (RAF) recorded monaural impulse responses in real office space, with scenarios of the office being empty and the office being furnished. Different from MeshRIR, the speaker is directional and varies its position at different data points. The impulse responses in RAF are resampled to 16 KHz. All the impulse responses in these two datasets are cut to 0. We use 90% of the data to train and the rest 10% for testing (Refer Appendix C.2 for 0.32s on RAF dataset).
7

---

Me
Furnished
RAF Empty
Avonia
Room
INRAS
Pha ei:: 1 64 Emy. eri: 1:52
Pha err: 1 Eri er:: 9.89
Pho err: 1 1,57 Env. er:: 8.08
Pii eir: 1
Pho. em: 1665 Eniy. 4.30
EnV, er:: 6.02
NAF
1,63 Amp e 0.64
1. Amp, 0,1:
1.71
Amp. 0.
Amp. eir: 0.85
Amp e
ert:
erf; 5.57
AVR (Ours)
Pha, er:. 0,61 Env. er:: 0.99
Pha. err: 1.59 a Envy em 2.99
Pho, em 1 1.56 Eri. err: 2:45
Pho. er:: 0.79 Env. er:: 3.46
Pha am: 0.44 Env. em 1. 1.50
Ground truth
Figure 6: Examples of synthesized impulse response with different methods. We visualize the synthesized impulse responses as time signals (x-axis) on both real and simulated datasets. Orange lines represent model predictions and Blue lines represent ground truth. All plots share a common y-axis for easier comparison.
The results of all methods are shown in Tab, We find that our method significantly outperforms the existing state- of-the-art baselines in all the datasets. We note that no existing learning-bol based method but ours performs better than chance in the phase error metric. In Fig Fig4and and Fig. 5 show visualizations of the learned impulse response field for the entire scene. Our method captures much more spatial signal distribution than prior works. We also show an example of individual time-domain impulse response in Fig. 6 Although prior methods capture the general decaying trend of the impulse responses, the waveforms (e.g. the peaks) are misaligned with the ground truth. In contrast, our method captures the waveforms much better. We especially point the readers the time that the signal arrives for every impulse response, which indicates time- of-arrival. Our method has much smaller errors terms of time-of- of-arrival due to our physics-based rendering.
NAF
INRAS
Loud
AVR (Ours)
Ground truth
Quiet:
Figure S: Top-down view of loudness map on MeshRIR. AVR predicts an accurate loud- ness map, while NAF and INRAS have inac- curate patterns.
4.2 Results on Simulation Dataset
Due to the high cost of equipment required to collect large-scale impulse responses in real scene, MeshRIR and Real Acoustic Field are the only real-world impulse response datasets that are collected densely in an enclosed space. Researchers typically simulated impulse response data in virtual 3D environments to complement the real-world datasets (26]
We use our simulation platform to simulate monaural impulse responses in three rooms and evaluate all methods' performance (Tab. 2}. The simple 2D room 2D rectangular-shaper enclosed space with only one wall in the middle of the room, serving as toy example. We also include two complicated 3D rooms from iGibson dataset (24 [56). All rooms are equipped with a single omnidirectional speaker, with listeners placed randomly in the rooms. All the impulse responses are sampled at 16 KHz sampling rate with 0. 1s duration time. We follow the same split strategy used in real-world datasets.
8

---

2D Room
3D scene Avonia
3D scene Montreal
Method
Phase
Amp.
Env.
T60
C50
EDT
Phase
Amp.
Env.
T60
C50
EDT
Phase
Amp.
Env.
T60
C50
EDT
neC-nearest
1.40
0.88 2.88
11.0
2.77
32.6
1.62
0.95
6.52
10.4
2.01
27.3
1.62
0.95
5.57
10.4
2.01
27.9
AAC-linear
1.39
0.86
2.64
10.3
2.54
29.8
1.61
0.90
6.13
10.1
1.86
25.4
1.61
0,90
5.19
10.1
1.86
25.1
Opus- Opus-nearest
1.38
0.69
2.73
9.4
2.22
26.8
1.61
0.85
6.74
10,4
1.94
25.0
1.61
0.85
5.74
10.4
1.94
25.2
Opus-linear
1.34
0.67
2,53
9.1
2.21
25.6
1.70
0.71
6.31
9.8
1.87
24.0
1.60
0.71
5.32
9.8
1.87
24.3
NAF
1.62
0.69
6.72
7.6
2.02
23.5
1.62
0.99
9.12
10.2
2.12
25.4
1.62
0.81
5.53
7.9
1.38
19.1
INRAS
1.61
0.94
4.16
8,4
2.45
26.7
1.62
0.87
6.86
8.8
1.56
20.0
1.62
0.87
5.75
7.6
1.44
19.7
AVR (Ours)
1.04
0.65
2.35
7.5
1.62
20.5
1.52
0.63
5.93
8.9
1.44
19.1
1.53
0.65
5.16
7.6
1.23
17.1
Table 2: Quantitative results on simulation dataset. Our method significantly outperforms existing methods in both a 2D room with simple geometry and two 3D rooms with complex geometry.
Tab. shows that our method consistently outperforms existing methods in all datasets. From the spatial signal distributions in Fig. 4 we observe that even in the simple 2D room the prior methods fail to accurately capture the accurate field distribution, while the field distribution generated by our method consistently matches the ground truth in both 2D and 3D cases. Our estimated time-domain signals at unseen poses are also closely matched with the ground truth signals, shown in Fig. G
4.3 Zero-Shot Binaural Audio Rendering.
AVR can generate accurate binaural audio despite being trained only on monaural audio modeling (without any fine-tuning). The existing method for rendering binaural audio either requires training at binaural channel spatial audio data [13] or manually creating signal delays. We render impulse response of left and right ears separately (20cm apart) in the MeshRIR scene. We play a piece of music 3 meters away from a listener, who turns its head from left to right and back again, We conduct user study comparing the spatial perception of rendered binaural audio among NAF, INRAS, and our method. Seven users rated the similarity between expected head trajectories and their hearing experience on a 1-5 scale. Our method achieves the highest score of 4.71, compared to NAF's 1.42 and INRAS's s 1.86. Other methods fail to synthesize accurate binaural audio because they are trained solely on monaural audio. Audio examples are available on our project website.
AVR is able to achieve binaural audio rendering for multiple reasons. First, our model captures accurate phase information in the impulse response to the extent that simply rendering the impulse response at the positions of the left and right ears can provide accurate phase differences, i.e., time delay or interaural time differences (ITD). Second, our model can easily incorporate the head-related transfer function for modeling the shadowing and pinna effects. Specifically, these direction direetion-deeendeet filtering effects can be integrated into [ before summing responses from all directions. By replacing the direction-dependee can achieve a more accurate binaural sound effect and reduce directional ambiguity (e.g., front versus back). Furthermore, explicit incorporation of HRTF allows our method to work with customizable HRTF for different users, allowing for an accurate and personalized listening experience.
weight term G(w)) with a direction deeeeddn-eeendeent HRTF function, we
4.4 Computing Efficiency
A comparison of runtime efficiency between AVR and other methods are shown in Tab. 3 This includes the inference time for different methods when they are trained to output IR of 0. and 0.32s. Since AVR uses acoustic volume rendering over a sphere, is slower than the methods that directly output IR with a network. Encouragingly, various techniques have been proposed in recent years to significantly improve the efficiency of volume rendering and NeRF through efficient sampling strategies [17 25| 48
Method
0.1s IR
0.32s IR
NAF
3.2 ms
6.4 ms
INRAS
2.1 ms
3.2
AV-NeRF
4.6 ms
6.9 ms
AVR (Ours)
30.3 ms
90.7 ms
Table 3: Inference Time Comparison.
These approaches can be similarly
adapted for acoustic volume rendering. More analysis computing efficiency can be found in Appendix C..]
4.5 Ablation Study
We ablate different choices of the sampling parameters during volume rendering, rendering domain, and loss components (Tab. 4). All models are evaluated on the MeshRIR dataset.

---

Study Objectives
Variation
Phase.
Amp.
Env.
T60
C50
EDT
64 X 32 rays, 64 points
0.956
0.547
1.17
4.07
1.20
47.6
48 X 24 rays, 64 points
1.356
0.607
1.37
4.57
1.73
67.6
Sampling Parameters
X rays, 64 points
0.847
0.535
1.15
3.86
0.92
35.1
80 X 40 rays, 80 points
0.857
0.529
1.14
3.79
0.95
34,9
80 x 40 rays, 40 points
0.869
0.543
1.17
4.66
1.30
52.9
time-domain frequency-donme domain
1.181
0.642
1.43
4.28
1.23
39.6
Rendering Domain
0.847
0.535
1.15
3.86
0.92
35.1
w/o raw signal loss
0.722
0.558
1.16
3.89
1.74
46.4
Loss Component
w/o angle & spec loss
1.453
0.567
1.36
4.52
2.65
64.5
loss components
0.847
0.535
1.15
3.86
0.92
35.1
Table 4: Model ablations. Performance for the model variants on MeshRIR dataset.
Sampling Parameters. We study the sensitivity of our model to sampling parameters No. Ng. Ny. We find that both increasing the ray numbers and the sampling points will both enhance the performance, but come with the cost of low training speed and high memory consumption.
Rendering Domain. We train model using both time-domain volume rendering and frequency- domain volume rendering. Frequency-donee domain rendering effectively avoids issues associated with fractional time delays, aligning more accurately with the actual phenomenon of acoustic signal propagation. Consequently, this approach yields better results, confirming our argument Sec. 3.3
Loss Component. We also ablate loss components. We find that reducing any of the loss components results in decreased performance. However, is noteworthy that all model variants, except for the one trained without the angle and spectral loss, outperform the baselines discussed in Sec.
Discussion
Limitations and Future Work. Our rendering involves both spherically sampling rays and sampling points along each ray, which can lead to large memory consumption and longer inference time. Recently, many research works have been proposed to improve the efficiency of volume rendering and NeRF through efficient sampling strategies. We envision that similar methods could also be applied to acoustic volume rendering to speed up the rendering. Besides, AVR needs to train new model for novel scene, which requires effort to collect impulse response samples in the scene. Future work could explore generalization to novel scenes by incorporating multi-modes inputs, aiming synthesize impulse response field using only few visual acoustic samples.
Conclusion. This paper proposes acoustic volume rendering to reconstruct impulse response fields that inherently encode wave propagation principles. We introduce frequency-domale signal rendering and spherical signal integration to address the unique challenges in impulse response modeling. Ex- perimental results demonstrate that AVR significantly outperforms existing approaches. Additionally, we develop AcoustiX, an open-source simulation platform that provides accurate lime-of-arrive measurements. Our work advances immersive auditory experiences in AR/VR, spatial audio in gaming and virtual environments, teleconferencing, and acoustic modeling in architectural design. Our realistic auditory simulations also benefit autonomous navigation, acoustic monitoring, and assistive hearing technologies where accurate acoustic modeling is essential.
Acknowledgments
We thank the members of the WAVES Lab at the University of Pennsylvania for their valuable feedback. We are grateful to the anonymous reviewers for their insightful comments and suggestions.
References
[1] Jont Allen and David Berkley. Image method for efficiently simulating small-room acoustics. The Journal of the Acoustical Society of America, 65(4):943 1979.
10

---

[2] Niccolo Antonello, Enzo De Sena, Mare Moonen, Patrick A Naylor, and Toon Van Waterschoot. Room impulse response interpolation using sparse spatio-temporal representation of the sound field. IEEE/ACM Transactions Audio, Speech, and Language Processing, 25(10):1929-1441, 2017.
[3] Benjamin Attal, Eliot Laidlaw, Aaron Gokaslan, Changil Kim, Christian Richardt, James Tompkin, and Matthew O" Törf: Time-of-flight radiance fields for dynamic scene view synthesis. Advances neural information processing systems, 34:26289-26300 2021.
[4] Jens Blauert. Spatial hearing; the psychophysics of human sound localization. MIT press, 1997.
[5] Jeffrey Borish. Extension of the image model to arbitrary polyhedra. The Journal of the Acoustical Society of A'Erriaa 75(6):887--186, 1984.
[6] Marina Bosi, Karlheinz Brandenburg, Schuyler Quackenbush, Louis Fielder, Kenzo Akagiri, Hendrik Fuchs, and Martin Dietz. Iso/iec mpeg-2 advanced audio coding. Journal of the Audio engineering society, 45(10):789-81) 1997.
[7] Chakravarty Alla Chaitanya, Nikunj Raghuvanshi, Keith W Godin, Zechen Zhang, Derek Nowrouzezahrai, and John M Snyder. Directional sources and listeners in interactive sound prop- agation using reciprocal wave field coding. ACM Transactions on Graphics (TOG), 39(4):44-1, 2020.
[8] Changan Chen, Unnat Jain, Carl Schissler, Sebastia Vicene Amengual Gari, Ziad Al-Halah, Vamsi Krishna Ithapu, Philip Robinson, and Kristen Grauman. Soundspaces: Audio-visuaa navigation in 3d environments. In Computer Vision-ECCC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedinggs Part VI 16, pages 17-36. Springer, 2020.
[9] Changan Chen, Carl Schissler, Sanchit Garg, Philip Kobernik, Alexander Clegg, Paul Calamia, Dhruv Batra, Philip Robinson, and Kristen Grauman. Soundspaces 2.0: A simulation platform for visual-acoustic learning. Advances in Neural Information Processing Systems, 35:8896- 8911, 2022.
[10] Ziyang Chen, Israel D Gebru, Christian Richardt, Anurag Kumar, William Laney, Andrew Owens, and Alexander Richard. Real acoustic fields: An audio-visual room acoustics dataset and benchmark. arXiv preprint arXiv:2403. 18821, 2024.
[11] Junyuan Deng, Qi Wu, Xieyuanli Chen, Songpengcheng Xia, Zhen Sun, Guoqing Liu, Wenxian Yu, and Ling Pei. Nerf-loam: Neural implicit representation for large-scale incremental lidar odometry and mapping. In Proceedings of the IEEE/CVF International Conference on Computer 2023.
Vision, pages
[12] Kangle Deng, Andrew Liu, Jun-Ya Yan Zhu, and Deva Ramanan. Depth-supervised nerf: Fewer views and faster training for free. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12882-12891 2022.
[13] Ruohan Gao and Kristen Grauman. 2.5 d visual sound. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 324-333, 2019.
[14] Nail A Gumerov and Ramani Duraiswami. broadband fast multipole accelerated boundary element method for the three dimensional helmholtz equation. The Journal of the Acoustical Society of America, (25(1):191 190-225,, 2009.
[15] Dorte Hammershøi and Henrik Møller. Binaural technique basic methods for recording, synthesis, and reproduction. Communication acoustics, pages 223-254, 2005.
[16] Jakob Hoydis, Sebastian Cammerer, Fayçal Ait Aoudia, Avinash Vem, Nikolaus Binder, Guillermo Mareus, and Alexander Keller. Sionna: An open-source library for next-generation physical layer research. arXiv preprint arXiv:2203. 1154, 2022.
[17] Tao Hu, Shu Liu, Yilun Chen, Tiancheng Shen, and Jiaya Jia. Efficientnerf efficient neural radiance fields. Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 12902-12911, 2022.
11

---

[18] Shengyu Huang, Zan Gojcic, Zian Wang. Francis Williams, Yoni Kasten, Sanja Fidler, Konrad Schindler, and Or Litany. Neural lidar fields for novel view synthesis. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 18236-1824,, 2023.
[19] Christoph Kling. Absorption coefficient database. fachabteilungen.
hatpallunnggnaat//bbbeeeeee att/a/tt///b////eeee ag- http://////////poooooooooaaaaaaa htmll
2018. Accessed: 2024-09-08.
[20] Shoichi Koyama, Tomoya Nishida, Keisuke Kimura, Takumi Abe, Natsuki Ueno, and Jesper Brunnström. Meshrir: A dataset of room impulse responses meshed grid points for evaluating sound field analysis and synthesis methods. 202/ IEEE Workshop on Applications of Signal Processing Audio and Acoustics (WASPAA), pages 1-5, IEEE, 2021.
[21] Asbjørn Krokstad, Staffan Strom, and Svein Sørsdal. Calculating the acoustical room response by the use of a ray tracing technique. Journal of Sound and Vibration, 8(1):118-123 1968.
[22] Heinrich Kuttruff. Room acoustics. Cre Press, 2016.
[23] Eric A Lehmann and Anders M Johansson. Prediction of energy decay in room impulse responses simulated with an image-source model. The Journal of the Acoustical Society of America, 124(1):269-277, 2008
[24] Chengshu Li, Fei Xia, Roberto Martín-Martin Michael Lingelbach, Sanjana Srivastava, Bokui Shen, Kent Vainio, Cem Gokmen, Gokul Dharan, Tanish Jain, et al. igibson 2.0: Object-centric simulation for robot learning of everyday household tasks. arXiv preprint arXiv:2108.03277 2021.
[25] Ruilong Li, Matthew Tancik, and Angjoo Kanazawa. Nerfacc: general nerf acceleration toolbox. arXiv preprint arXiv:2210.04844 2022.
[26] Susan Liang, Chao Huang, Yapeng Tian, Anurag Kumar, and Chenliang Xu. Neural acoustic context field: Rendering realistic room impulse response with neural fields. arXiv preprint arXiv:2309.15977, 2023.
[27] Susan Liang, Chao Huang, Yapeng Tian, Anurag Kumar, and Chenliang Xu. Av-nerf: Learning neural fields for real-world audio-visual scene synthesis. Advances in Neural Information Processing Systems, 36, 2024.
[28] Haofan Lu, Christopher Vattheuer, Baharan Mirzasoleiman, and Omid Abari. Newrf: A deep learning framework for wireless radiation field reconstruction and channel prediction. In Forty-first International Conference on Machine Learning.
[29] Andrew Luo, Yilun Du, Michael Tarr, Josh Tenenbaum, Antonio Torralba, and Chuang Gan. Learning neural acoustic fields. Advances in Neural Information Processing Systems, 35:3165- 3177, 2022.
[30] Sagnik Majumder, Changan Chen, Ziad Al-Halah, and Kristen Grauman. Few-shot audio- visual learning of environment acoustics. Advances in Neural Information Processing Systems, 35:2522-255 2536, 2022.
[31] Anagh Malik, Parsa Mirdehghan, Sotiris Nousias, Kyros Kutulakos, and David Lindell. Tran- sient neural radiance fields for lidar view synthesis and 3d reconstruction. Advances in Neural Information Processing Systems, 36, 2024.
[32] Rémi Mignot, Gilles Chardon, and Laurent Daudet. Low frequency interpolation of room impulse responses using compressed sensing. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 22(1):205-216, 2013.
[33] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
[34] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM transactions on graphics (TOG),41(4):. 41())
15, 2022.
12

---

[35] Anton Ratnarajah, Sreyan Ghosh, Sonal Kumar, Purva Chiniya, and Dinesh Manocha. Av-rir: Audio-visual room impulse response estimation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 37164-27175, 2024.
[36] Anton Ratnarajah, Zhenyu Tang, Rohith Aralikatti, and Dinesh Manocha. Mesh2ir: Neural acoustic impulse response generator for complex 3d scenes. In Proceedings of the 30th ACM International Conference on Multimedia, pages 924 933, 2022.
[37] Anton Ratnarajah, Shi-Xiong Zhang, Meng Yu, Zhenyu Tang, Dinesh Manocha, and Dong Yu. Fast-rir: Fast neural diffuse room impulse response generator. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 571-575. IEEE, 2022.
(38) Barbara Roessle, Jonathan T Barron, Ben Mildenhall, Pratul P Srinivasan, and Matthias Nießner. Dense depth priors for neural radiance fields from sparse input views. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12892-12901, 2022.
[39] Robin Scheibler, Eric Bezzam, and Ivan Dokmanić. Pyroomacoustics: A python package for audio room simulation and array processing algorithms. In 20/8 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 351-355 IEEE, 2018
[40] Carl Schissler and Dinesh Manocha. Gsound: Interactive sound propagation for games. In Audio Engineering Society Conference: 41 International Conference: Audio for Games. Audio Engineering Society, 2011.
[41] Dirk Schröder. Physically based real-time auralization of interactive virtual environments, volume 11. Logos Verlag Berlin GmbH, 2011.
[42] Aarrushi Shandilya, Benjamin Attal, Christian Richardt, James Tompkin, and Matthew O' O'toole Neural fields for structured lighting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3512-3522, 2023.
[43] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in neural information processing systems, 33:7462-7443.. 2020.
[44] Kun Su, Mingfei Chen, and Eli Shlizerman. Inras: Implicit neural representation for audio scenes. Advances in Neural Information Processing Systems, 35:8 188881441188 2022.
[45] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keis Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions low dimensional domains. Advances in neural information processing systems, 33:7537-7547, 2020.
[46] Zhenyu Tang, Rohith Aralikatti, Anton Jeran Ratnarajah, and Dinesh Manocha. Gwa: A large high-quality acoustic dataset for audio processing. In ACM SIGGRAPH 2022 Conference Proceedings, pages 1-9, 2022.
[47] Lonny L Thompson. A review of finite-element methods for time-harmonic acoustics. The Journal of the Acoustical Society of America, 119(3):315--1330, 2006.
[48] Haithem Turki, Vasu Agrawal, Samuel Rota Buld, Lorenzo Porzi, Peter Kontschieder, Deva Ramanan, Michael Zollhöfer, and Christian Richardt. Hybridnerf: Efficient neural rendering via adaptive volumetric surfaces. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 19647-19656, 2024.
[49] Natsuki Ueno, Shoichi Koyama, and Hiroshi Saruwatari. Kernel ridge regression with constraint of helmholtz equation for sound field interpolation. In 2018 16th International Workshop on Acoustic Signal Enhancement (IWAENC), pages 1-440. IEEE, 2018.
[50] Jean-Mare Valin, Koen Vos, and Timothy Terriberry. Definition of the opus audio codec. Technical report, 2012.
13

---

[51] Guangcong Wang, Zhaoxi Chen, Chen Change Loy, and Ziwei Liu. Sparsenerf: Distilling depth ranking for few-shot novel view synthesis. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 9065-9076, 2023.
[52] Jui-Hsien Wang, Ante Qu, Timothy R Langlois, and Doug L James. Toward wave-based sound synthesis for computer animation. ACM Trans. Graph., 37( 109, 2018.
[53] Mason Long Wang, Ryosuke Sawata, Samuel Clarke, Ruohan Gao, Shangzhe Wu, and Jiajun Wu. Hearing anything anywhere. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 11790-11799 2024.
[54] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106. 10689, 2021.
[55] Hanfeng Wu, Xingxing Zuo, Stefan Leutenegger, Or Litany, Konrad Schindler, and Shengyu Huang. Dynamic lidar re-simulation using compositional neural fields. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[56] Fei Xia, Amir R Zamir, Zhiyang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: Real-world perception for embodied agents. In Proceedings of IEEE conference on computer vision and pattern recognition, pages
2018
[57] Bosun Xie. Head-related transfer function and virtual auditory display. J. Ross Publishing, 2013.
[58] Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim. Parallel wavegan: A fast waveform gen- eration model based on generative adversarial networks with multi-resolution spectrogram, In ICASSP 2020-2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 6199-6203. IEFF, 2020.
[59] Dongyu Yan, Xiaoyang Lyu, Jieqi Shi, and Yi Lin. Efficient implicit neural reconstruction using lidar. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 8407-848 8414. IEEE, 2023.
[60] Wen Zhang, Parasanga N Samarasinghe, Hanchi Chen, and Thushara Abhayapala. Surround by sound: A review of spatial audio recording and reproduction. Applied Sciences, 7(5):532, 2017.
[61] Xiaopeng Zhao, Zhenlin An, Qingrui Pan, and Lei Yang. Ner12: Neural radio-frequenc radiance fields. In Proceedings of the 29th Annual International Conference on Mobile Computing and Networking, pages 1-15, 2023.
[62] Xingguang Zhong, Yue Pan, Jens Behley, and Cyrill Stachniss. Shine-mapping: Large-scale 3d mapping using sparse hicrarchical implicit neural representations. In 2023 IEEE International Conference Robotics and Automation (ICRA), pages 8371-8377. IEEE, 2023.
14

---

A Sampling Rays and Points
The direction of ray can be represented by measures: azimuth 0 and elevation ø. We handle ray sampling by performing both azimuth and elevation sampling. For azimuth sampling, we apply stratified sampling between and 2T to obtain another Na rays, where is the index:
i No.
0, ~ u 2n
2T
(14)
No
/ For elevation sampling, we evenly distribute N₂ rays, with oj = arccos(2 N3 - 1), where j is the index. By combining all azimuth and elevation angles, we obtain No N₂ directions in 3D Cartesian coordinates, each represented follows:
Wij- = [cos 0g sin ØJ. sin 0g sin Øy, cos وg].
(15)
For each sampled ray, we uniformly sample N, points on it. Given ray p(u) + u starting from the point pr in direction w, the position of the mh point is given by:
p(um) Pt + ((u un) N, + u,
(16)
With this sampling, we approximate the integral in Eq. 7 using quadrature as follows:
Hnll = $
Tm exp(-om Au) { (
u ; p(uu))
to
"
(17)
m-1 and Au
Uf Un
where Tm = cxp
o2
ouAu
Nr
Combing our ray sampling strategy, we rewrite the Eq and as follows:
No
N
Hlf
Hwnr))Ha,ff) If].
(18)
B Evaluation Metric
Envelope Error. Given the time domain ground truth impulse response h² [n] and our prediction h(n]. we can compute the envelope error by first obtaining the envelope using the Hilbert transform to get the analytic signal and then applying the absolute value, follows:
Env* Hiilbert (h*)]
(19)
The normalized envelope error is defined as follows (we multiply it by 100 to avoid small numbers):
Env* Env
Envelope error 100 Mean
(20)
max
(Env*)
Phase and Amplitude Error. Given the frequency domain ground truth impulse response [f] and our prediction HJ), we use cosine and sine function encoded function to quantify the phase error:
Phase error Mean( cos cos(<H") cos( LH) (Hn) sin(2H))).
(21)
The amplitude error defined as follow:
Amplitude error Mcan (H*) - abs( H
(22)
abs(H1)
C More Evaluation Results
C.1 Computing Efficiency
We further examine the relationship between inference speed and the number of rays well as the number of points sampled along each ray. As illustrated in Fig 7 the inference speed scales approximately linearly with both the number of rays and the number of points along each ray.
15

---

Inference speed (ms)
Inference speed (ms)
150.2
140
150
120.5
125
121.3
120
105.6
100
90.7
100
90.7
81.4
72.3
80
77.9
75
49.
60
50
40
1152
1568
2048
2592
3200
48
56
64
72
80
Ray numbers
Point numbers
Figure 7: Impact of ray and point counts on inference speed.
C.2
Additional Results on RAF Dataset
We repeated experiments with 0.32s RIR duration. Tab55 the results the RAF-Fumished dataset. We also included AV-NeRF as a baseline and multi-resolltion STFT as an evaluation metric. With a 0.32s RIR duration, our method also outperforms these baselines. We provide loudness map visualization 8) for two different speaker positions at RAF-Furnished dataset with grid size of 0.1m. Our method can better capture sound level differences caused by geometry occlusion and has smoother spatial variation of loudness.
Method
Phase
Amp.
Env.
T60
C50
EDT
NAF
1.62
0.79
1.67
7.68
0.64
24.2
INRAS
1.62
0.89
1.34
5.41
0.57
22.8
AV-NeRF
1.62
0.93
1.59
6.54
0.61
25.9
AVR (Ours)
1.59
0.69
1.04
4.95
0.55
19.8
Table 5: Results on RAF dataset. Performance comparison between our method and others on the RAF dataset with a 0.32s RIR duration.
NAF
INRAS
AV-NeRF
Ours
Figure 8: Loudness map. We visualize the loudness map of various methods using the RAF- FAF-iuiis dataset,
which features the most complex structure among all the datasets utilized. Green dots and arrows represent the speaker positions and orientations from top view. Gray dots represent the room structures, outlining the geometry of walls, objects, other elements.
D
Acoustic Simulation Platform
D.1 Impulse Response Generation
AcoustiX is built based on Sionna [16] ray tracing engine that supports ray reflection, scattering, and diffraction. We modify the ray tracing engine in terms of ray interactions with the environment to support acoustic impulse response simulations. Each material the scene assigned reflection coefficient 3 and scattering coefficient a. For each reflection, the reflected wave's amplitude is
16

---

(1-1-aal) with E being the energy before the interaction. The scattered energy is E" aa3E. With these notations, the impulse response is formulated as follows:
N Σ dn
Kn
dn
((t)
ÓLP
t
(1 --
(23)
where d, is the accumulated total length of ntth path, v is the velocity of sound in the air, an,k and Bn, denote material properties of 4th reflection. We use the negative reflection coefficient definition discussed in [23]. the equation, we assume purely specular reflection for simplicity. If scattering or diffraction occurs along the path, replace the reflection term with the corresponding scattering or diffraction attenuation. ÓLP is a windowed sinc function defined similar to the one in [39].
In the implementation, we divide the whole frequency band into several octave bands and get their common acoustic properties. To assign eeeuencc-deeendent reflection and scattering coefficients to each material, we transfer Eq. 23 the frequency domain and assign coefficient to the amplitudes for each frequency band. The material coefficient is retrieved from [9
D.2 Acoustic Ray Tracing
Time
Figure 9: Example of simulated impulse response
In AcoustiX, users can determine the number of cast rays and maximum bouncing depth in the simulation. AcoustiX supports different ray-geometry interactions including reflection, scattering, and diffraction. By default, we enable depth to 30 and the number of cast rays to le6 to enable comprehensive path searching within the rooms. We provide flexible API usage in AcoustiX, allowing users to adjust the acoustic ray tracing configurations and balance between simulation quality and speed. Fig. 9 shows example of our simulated impulse responses.
the functions above and set the maximum bouncing
D.3
Room Model
AcoustiX supports customized room models. We create room structures with Blender, assign material names to objects, and export the scene in XML formats using Mitsuba blender Add-on") During simulation, each object is matched with its corresponding acoustic material properties by looking up a table mapping assigned names to properties. In addition to customizing room models, we also support importing 3D room models from the iGibson dataset [24 56 into our simulations, assigning acoustic propertics to each object.
E
Social Impact
As method synthesize high-quality impulse responses, work potentially enhance immersive VR/AR experiences and sound-dependent applications. AcoustiX fosters research and innovation acoustic topics. Potential negative social impacts include the creation of misleading audio content, which could be used to deceive or manipulate users. For instance, high-quality impulse response generation could be exploited fabricate realistic but fake acoustic environments or conversations, leading to misinformation or privacy violations.
17

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly state contributions in the abstract and introduction and conduct extensive experiments on different datasets to support our claim.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We point out the limitations in our Discussion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced
statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We clearly elaborate on our method step by step and include all parameters needed for network building. We also provide details on how we preprocess the data and the simulation platform details we adopt for reproducibility. The code and the simulation platform is open-sourced in the projeact website.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
19

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Our code, dataset, and simulation platform open-sourced in the project website.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cc)
aublic//uudees//ddeettbi.ss www.oeeeuuuuissssooooooooooo for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. wwtps/cccppplgggguiieeeeee
Ceaeee/CCoSSuSbisSS..S.. www.ssssiicgcyo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have all the training and testing details, including data splits, hyperparame- ters, ablation studies on hyperparameters and the description of the used optimizer.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We don't 't report error bars because this would be too computationally expen- sive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
20

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have all details the Experiment setup part.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
We adhere to the NeurIPS Code of Ethics and preserve anonymity.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We mention both potential societal impacts in the appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
21

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our model and data do not pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the corresponding paper when needed and follow the corresponding license when using public datasets and the licensee that the Sionna ray tracing engine uses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
22

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We talk about the details of our simulation platform when we introduce it.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
23

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "BeNh1u6VV0", "reviewer": "Reviewer_4897", "summary": "The authors reformulate neural fields to model the spatial interactions of sound, by learning to predict impulse response in a fixed transmitter / arbitrary receiver pose geometry. The method substantially outperforms contemporary work on simulated and real world datasets, and lays the groundwork for new immersive AR/VR applications.", "questions": "1. How does the computational complexity of your method compare to existing approaches, particularly for real-time applications?\n\n2. In lines 282-294 (discussion of zero-shot binaural audio rendering), are there any user studies / qualitative evaluations that could characterize the zero-shot performance vs. a network optimized for that particular task?\n\n3. In table 3, it seems counterintuitive that removing raw signal loss improves phase error.. do the authors have any insight into this result?\n\n4. In lines 302-304, its not clear to me why frequency-domain rendering gives significantly better results than time-domain rendering.. I understood the claim that the former is more compute-efficient, but don't see the claim that this should occur in Section 3.3 as the authors state -- could the authors clarify this?", "limitations": "The authors do acknowledge memory consumption (due to omnidirectional ray sampling) as a limitation. It would be good to expand on other potential limitations like generalization across environments, or applicability to time-critical applications.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The methodology is well-motivated and creative in its incorporation of acoustic / wave propagation principles in the learning algorithm\n\n2. Performance improvements over related work are nontrivial, in both simulated and real datasets\n\n3. The proposed method stands out in its ability to recover the phase of synthesized waves, which is crucial for many time-critical applications\n\n4. Frequency-domain rendering is a clever way to address the challenges in fractional time delay / discrete sampling in predicting IPR\n\n5. A new simulation platform (when released) would be a significant enabler to other researchers", "weaknesses": "1. I feel that the ablation study + associated commentary is limited (see my questions in the next section). This is exacerbated by the fact that performance metrics are reported without standard deviations. I saw that the authors' rationale in the checklist for omitting error bars was budget, but I still think that expanding this section to include experiments on different network sizes, loss coefficients, etc. would increase this work's utility to researchers with different computational budgets, and better elucidate the impact of each component of the method. It would be nice to expand the commentary in the discussion with the authors' hypothesized explanations for results as well. \n\n2. The paper lacks a crucial discussion on the computational complexity and runtime of the proposed method compared to existing approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for a thoughtful review and valuable feedback. We address the questions related to our work below.\n\n**Q1: Computing efficiency and runtime of framework.**\n\nPlease refer to **Computing efficiency** in the global rebuttal.\n\n**Q2: Include user study of qualitative evaluations about the zero-shot performances of binaural sound rendering.**\n\nPlease refer to **Zero-shot binaural audio rendering** in the global rebuttal.\n\n**Q3: Ablation study explanations.**\n\n1. Why does removing the time-domain loss improve the phase error?\n\nWe use both time-domain and frequency-domain losses to supervise our training. When the time-domain loss is removed, the frequency-domain loss becomes the primary focus, causing the network to prioritize matching the frequency-domain signal over capturing time-domain characteristics. This leads to improved phase accuracy, as the network is more focused on aligning frequency components. However, this also means the network may fail to capture certain time-domain signal variations, resulting in poor performance in metrics related to temporal accuracy, such as C50 and EDT.\n\n2. Why does frequency domain rendering give better result?\n\nAs the reviewer pointed out, frequency-domain rendering effectively avoids issues associated with fractional time delays. This is because time delay representation in the frequency domain is continuous, aligning more accurately with the actual phenomenon of acoustic signal propagation. Consequently, this approach yields better results.\n\n3. Ablation study on network size.\n\nWe change our network layer from 5 to 8 to expand the ablation study in Table 3 of the paper. We demonstrate that increasing the network beyond six layers does not result in significant performance variations. \n\n\n| Network layer | Phase.         | Amp.           | Env.          | T60           | C50           | EDT           |\n|:-------------:|:--------------:|:--------------:|:-------------:|:-------------:|:-------------:|:-------------:|\n| 5             | 0.901          | 0.575          | 1.17          | 4.12          | 0.98          | 36.8          |\n| 6             | **0.847** | 0.535          | **1.15** | 3.86          | **0.92** | **35.1** |\n| 7             | 0.855          | 0.548          | 1.16          | 3.97          | **0.92** | 35.7          |\n| 8             | 0.849          | **0.533** | 1.16          | **3.82** | 0.93          | 35.9          |\n\n\n**Other Potential Limitations.**\nThank you for this suggestion! We will expand the limitations to include efficiency, and generalization across environments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aJNMh3N3tD", "reviewer": "Reviewer_WhmA", "summary": "This work presents a method to render a neural impulse response field to generate room impulse responses at novel positions after being trained with a number of limited RIR samples. The method works similar to a Nerf, rays are cast from a sphere and sampled at discrete positions along the rays. Here the signal at each sample point is transferred into the frequency domain and a phase shift is applied to delay them in time. The goal is to render the acoustic signal transmitted and volume density for any observer-emitter pair with known orientations and locations. \n\nThe approach uses among other methods two datasets to compare itself against common baselines in the domain. The authors emphasize having the best phase reconstruction compared to other results.", "questions": "- Why is 0.1 s chosen as duration time? \n\n- What is the inference time of this method?\n\n- What is the expected computation time if a duration of 1 second would be used?\n\n- How is the inference speed dependent on the number of rays? Optimization seems to take 24 hours for 200 epochs which is not bad but if this can only be used for very short RIRs in very small rooms or with a reduced number of rays these are important limitations which should be discussed or measured and explained.\n\n- How do the authors choose the size of the sphere they sample on? For example is it room dependent? Do they need the measurements of the room? What about complex scenes like apartments with connected rooms that are not shoebox like?\n\n- Why do the authors not compare to AV-NeRF in table 1. The results from the RAF paper could have been taken to compare against a visual-audio method if split, sampling etc. would have been the same as in the RAF paper.\n\n- What is the intuition/reson to concatenate directions only into the last 3 layers of the MLP?\n\n- What is the bouncing depth of 30? Is that meters? Is that because 0.1 * speed of sound makes bouncing beyond 30 m irrelevant?\n\n- Why is the phase important for this use case? The authors argue with VR applications. Many audio and audio-visual tasks ignore phase. If the method, after correcting the issues mentioned with the evaluation, ends up being closer to the other methods or even worse, then what is the benefit of reconstructing phase? \n\n- To compare truly with the RAF paper, what is the STFT error? Having the waveform the authors could calculate the multi-resolution STFT loss to compare over the whole dataset instead of comparing waveforms on limited examples in figure 6.", "limitations": "The biggest limitation coming from using a duration of 0.1 seconds seems unexplained.\n\nInference time and impact of more complex rooms or longer durations seems important but is unexplained.", "rating": 7, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 3, "strengths": "The method has great potential to correctly learn and reconstruct the phase in synthesized RIRs. It uses physical properties of wave-propagation in a very explicit way, compared to black box approaches, which can also help interpretability of the results.\n\nThe work potentially provides an improved simulator and, if certain points are improved, a good alternative to current simulators. \n\nThe authors remark very interesting issues with SoundSpaces, such as the time-of-flight error. They also found a very intuitive illustration for the sound pressure and phase field in a room which can again help with interpretation of results and comparison with other approaches. \n\nThe figures are very clear and the method is clearly described apart from notable omissions (see weaknesses and questions).", "weaknesses": "There are several odd observations and design decisions in this paper which at least should be explained. Their impact on performance is potentially great as can be probably seen in their baseline comparison. They differ by a large amount from the existing literature. Also, runtime (inference time) and impact of design decisions on it are not discussed but probably a major hurdle for this method. Without further explanation about the comparison with the state of the art and the validity of the method this contribution can not be sufficiently judged:\n\nEspecially since one of the arguments is about issues with sampling, the relation to the sampling rate used should not go uncommented. The authors decide to resample impulse responses to 16 kHz yet the RAF dataset is sampled at 48 kHz. Does that mean in Table 1 all other methods are also compared at 16 kHz? In the RAF paper, Table 7 shows results for this 16 kHz case for NAF and INRAS. They combine the furnished and empty case but for example, Inras has a T60 error of 5.34. In Table 1 of this paper, Inras is described as having errors of 11.3 and 10.6 in the furnished and empty case. Why is the baseline roughly twice as bad as reported in that paper? The EDT values of INRAS seem to have an opposite relation with 0.019 sec (RAF paper) vs. 57.4 MICROseconds and 62.9 MICROseconds. Either the authors mean milliseconds, in that case the results are worse without explanation, or they actually mean microseconds, in that case the results of Inras are way better in the RAF paper also without explaining why. Most of the other results are also not consistent with the RAF paper. The author need to explain why their baseline results seem to be so much worse than in the dataset paper.\n\nIf the explanation is the different split the authors use, they should explain why they did not use the split of 80%, 5%, 15% for training validation and test as used in the RAF paper or why a difference of 5% in the split leads to these vastly different results.\n\nAnother reason could be cutting the samples to 0.1 s. RAF compares samples at 0.32 s. \n\nLarge churches may have over 10 seconds of RT60. A typical living room may have 0.5 s RT60. Limiting the method to 0.1 s seems arbitrary.\n\nThe supplementary material plays music which sounds as if it starts towards the front left and then moves behind the head. Figure 7 shows a different route of the music. I did on purpose listen first to the audio before I read the description because expectation can change this experience. It is possible to imagine the sound being in the front or the back and both being plausible. This is only my experience and a number of test-listeners would be necessary to verify my findings. However, for me the audio experience is different than real binaural recordings which make me wonder if using a HRTF can really be omitted. If the authors simply create a time-delayed signal but without shadowing effects of the head and other important factors of the HRTF this is not binaural sound, this is stereo and then the directional ambiguity in perception can be explained. The authors should go into detail why they consider this zero-shot binaural sound.\n\nThis is a minor point and I does not reflect at all on the quality of the manuscript or the research but I would like to suggest to the authors to rename N_azi and N_ele to N_az and N_el to be consistent with other notations such as N_pt and avoid the distraction which could stem from reading the variable name N_azi wrong.\n\nLittle mistakes:\n- In equation 12 the font of L_time is inconsistent with line 205\n- Line 503 has one additional \"frequency\" word\n- Line 509 starts with IIn", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you sincerely for the thoughtful review and valuable feedback. Below, we address the questions raised and outline the revisions that we will include in the camera-ready version of the manuscript.\n\n**W1: Binaural sound rendering.**\n\nPlease refer to **Zero-shot binaural audio rendering** in the global rebuttal. \n\n**Q1: Why is 0.1s chosen as RIR duration?**\n\nThank you for raising this question! We chose to use 0.1s since the vast majority of the reflected energy shows within the first 0.1s. Specifically, with the RAF dataset, the average energy density within the initial 0.1s is around 500 times larger than that between 0.1s and 0.32s. The difference between 0.1s used in our evaluation and the 0.32s used in RAF caused the difference in evaluation metrics. (The microsecond was indeed a typo, it should be **millisecond**).\n\nTo provide a better comparison with the results reported in RAF, we repeated our experiments with a 0.32s RIR duration. Table 1 shows the results on the RAF-Furnished dataset. We also included AV-NeRF as a baseline and STFT as an evaluation metric based on your suggestions. With a 0.32s RIR duration, the performance of the baseline methods aligns closely with the numbers reported in the RAF paper, and our method continues to outperform these baselines. We will include results for both 0.1s and 0.32s in the camera-ready version if accepted.\n\n\n\n| Method | STFT | Phase | Amp. | Env. | T60 | C50 | EDT | \n|:-------------:|:--------------:|:--------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|\n| NAF       | 1.54        | 1.62             | 0.79             | 1.67             | 7.68            | 0.64                 | 0.024                |\n| INRAS     | 1.41        | 1.62             | 0.89                  | 1.34                  | 5.41                 | 0.57                 | 0.022                | \n| AV-NeRF                          | 1.49                  | 1.62                   | 0.93                  | 1.59                  | 6.54                 | 0.61                 | 0.025                | \n| **Ours**                    | **1.28**         | **1.58**          | **0.69**         | **1.05**         | **4.85**        | **0.51**        | **0.019**       |\n||\n\n\n**Q2, Q3, Q4: What is the inference time of the method? What is the inference time if 1 second impulse response is to be rendered? How is the inference speed dependent on the number of rays?**\n\nPlease refer to **Computing efficiency** in the global rebuttal for a detailed discussion on the runtime performance.\n\n\n**Q5: How to choose the size of the sphere? Is it room dependent? Does it need the measurements of the room? What about complex scenes like apartments with connected rooms that are not shoebox like?**\n\nWe empirically choose the radius of the sphere (i.e., furthest point to sample on each ray) based on the rough size of the scene. This is the same as visual NeRF which needs a far point for sampling on each ray. Our radius parameter does not require precise measurements or detailed room models, unlike methods such as INRAS.\n\n**Q7: What is the intuition to concatenate directions only into last 3 layers of the MLP?**\n\nThe intuition is that density only depends on position, while signal depends on both position and direction. In our design, the density prediction only take position as input. The last 3 layers  predict directional-dependent signal, so they will additionally take directions as input. Many visual volume rendering works share a similar network design.\n\n\n**Q8: What is the bouncing depth of 30?**\n\nBouncing depth in ray tracing refers to the number of times a ray of light or sound can reflect off surfaces before being terminated.\nFor higher-order bouncing events, the acoustic ray's energy diminishes to a level that can typically be neglected. To balance the computation efficiency and precision, we choose 30 as our maximum bouncing depth in the simulation setup. \n\n**Q9: Why phase is important?**\n\nPhase in the frequency domain is crucial for modeling RIR since time delay is equivalent to a phase shift in the frequency domain. Not using or being able to benefit from phase information might be a missed opportunity in prior work and could partially explain their underperformance.\n\nFor binaural perception in VR applications, accurate sound localization is vital for creating immersive environments. Phase differences between the left and right ear are essential for the human auditory system to determine the direction of sound sources. Incorrect phase information can lead to errors in perceived direction and distance, breaking the illusion of a virtual space.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8RCpnRzazX", "reviewer": "Reviewer_DqiU", "summary": "This method proposes a physics-based approach for learning neural impulse response field through implicit wave propagation modeling. This is the first work that really incorporates acoustic wave propagation principles for constructing an impulse response field, leveraging volume rendering techinques in analagy to how it's applied in NeRF. There is also a new acoustic simulation platform introduced for simulating more realistic impulse respones. Experiments show that the proposed method outperforms several previous learning based methods.", "questions": "Apart from the questions in the Weakness section, another question I have is related to the zero-shot binaural audio experiment. The paper claims that the proposed method can even render binaural audio even without HRTFs, which is a bit confusing to me. Even if the simulator can render binarual audio by synthesizing the sound heard at the left and right ears, there are other ear/head-related effect are not factored in, so how can binarual audio be directly generated?", "limitations": "There are some limitations discussed, but it would be great to hear from the authors on more discussions on limitations especially on the compute efficiency (e.g., how much time it takes for the network to converge?) and memory usage compared to prior work, so that it can give peope the complete picture on the pros and cons of the proposed framework.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- There are many so-called audio-NeRF in prior work, but none of them actually are. This work is the most physics-based neural impulse response method I have seen, and the closest to what an audio-NeRF should look like.\n\n- The problem is well-formulated, and it's an elegant formulation to implicitly model wave propagation, and use volume rendering for synthesizing room impulse responses.\n\n- There are very informative visualizations and ablation studies that shows the advantage of the proposed method, especially on modeling the phase of the signal.\n\n- The new simulation platform introduced will also be very useful for the community, and the comparison with SoundSpaces suggests that the community should start to use more realistic simulation platforms.\n\n- Experiments on both simulated and real datasets have demonstrated the proposed method outperforms a series of prior baselines.\n\n- Generally, the paper is also very well written with sufficent details and illustrations.", "weaknesses": "- The paper introduces a new acoustic simulation platform, which is claimed as one major contribution of the paper. However, in the main text, there is no detailed description of the simulator, and only says it is based on Sionna [16]. I then found it in Supp, but it would be good to also briefly summarize the key characteristics of this new simulator in the main text.\n\n- Since it's the first method that implicity models wave propagation for neural impulse response rendering, it would be important to also compare with prior physics-based methods that is geometry-based to highlight the superity of modeling wave propogation, apart from those learning based methods. For example, how would the proposed method compare to this recent work https://arxiv.org/abs/2406.07532, that models the acoustic field using an image-source differentiably method.\n\n- It would also be good to have some separate comparisons for high-frequency components vs. low-frequency components, and far-field vs. near-field to better understand the limitations and strength of the proposed framework. For example, for equation 1, the 1 / d assumption might not hold well for near field.\n\n- The qualitative examples are somewhat lacking. It would be much more informative to include some qualitative comparisons to prior work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for a thoughtful review, valuable feedback and acknowledgement of our work. We provide point by point clarifications and answer questions below.\n\n**W1: Summarize the key characteristics of the simulator in the main text.**\n\nThank you for this suggestion, we will expand the description of simulation setup in the main text:\n\nOur acoustic simulator uses Sionna ray tracing engine. We modify the engine in terms of ray interactions with the environment to support acoustic impulse response simulations.\nThe simulator supports various ray interactions with environment. Each material in the scene is assigned with frequency-dependent coefficients. This enables the tracing of cumulative frequency responses for each octave band to accurately simulate the impulse response.\nRoom models can be created using Blender and exported as compatible XML files for our simulation setup. Our engine also support the import of 3D room models from the Gibson dataset.\n\n**W2: Comparison with physics-based methods.**\n\nThe method in the paper (https://arxiv.org/abs/2406.07532) needs a precise measurement of the simplified room geometries and approximate them as a series of planes. While these parameters are provided by the author in the original paper's dataset, we do not have these measurements in our datasets to guarantee reproducible results. We leave the comparison with image-based method for future work. \n\n\n**W3.1: Separate comparisons for high-frequency components and low-frequency components.**\n\nIn the RAF-Furnished dataset, we conducted additional experiments using the multi-resolution STFT loss to analyze both high-frequency (above 2 kHz) and low-frequency (below 2 kHz) components. The resulting errors are 0.91 and 1.07, respectively.\n\n**W3.2: Limitation in applying 1/d to near field.**\n\nIn the near field, the acoustic field is difficult to explicitly formulate. We use 1/d to approximate the free space loss in paper, and allow our model to learn complex interactions through each point's signal and density. \n\n\n**W4: Lack of qualitative examples with prior works.**\n\nIn the paper, we provide qualitative results on time domain impulse response and frequency domain signal distribution. Additionally, we provide loudness map visualization for two different speaker positions at RAF-Furnished dataset with grid size 0.1m (Please refer to supplementary PDF Figure 1). Our method can better capture sound level differences caused by geometry occlusion and has smoother spatial variation of loudness. We would also include qualitative results on our binaural sound rendering.\n\n**Q1: Confusion about binaural audio rendering**\n\nPlease refer to **Zero-shot binaural audio rendering** part in the global rebuttal.\n\n**Computing efficiency:**\n\nPlease refer to **Computing efficiency** in the global rebuttal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a7NZ8MNf9f", "reviewer": "Reviewer_RLmp", "summary": "This paper addresses the challenge of novel sound synthesis from arbitrary positions. To achieve this, acoustic fields are modeled using an implicit neural representation, integrating acoustic wave propagation rules. Subsequently, a specifically designed volume rendering technique is employed to ensure consistency across diverse positions. Results from both simulated and real datasets validate the effectiveness of our approach. Additionally, the authors have developed a new impulse response simulation platform.", "questions": "See Weakness.", "limitations": "See Weakness.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. It is novel to employ volume rendering to ensure consistency across different positions, offering an advantage in capturing detailed waveform characteristics of impulse responses.\n2. The proposed method demonstrates superior performance compared to state-of-the-art techniques.\n3. The development of a new impulse response simulation platform is a valuable contribution.", "weaknesses": "1. The network is not very big (6 layers), I do not understand why the training takes 24 hours. Actually, the cited paper [29] is very fast, could the training time be reduced?\n2. I enjoyed listening to the attached audio, which exhibits a strong sense of three-dimensionality. Including synthesized audios from other methods in the final Supplementary Material would enhance the comprehensiveness of the submission.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for valuable feedback and acknowledgement of our work. We answer questions point by point below.\n\n**W1: Network is not very big (6 layers). Why it takes 24 hours to train the model, while the Instant-NGP is fast? Could the training time be reduced?**\n\nMost of the runtime for our method is consumed by the acoustic volume rendering process, accounting for approximately 98% of the total, rather than by querying the network. This distinction from the typical visual NeRF setup results in longer training times compared to visual NeRF. Various techniques have been proposed in recent years to significantly improve the efficiency of volume rendering and NeRF through sampling strategies. We believe that these approaches can be similarly adopted and transferred for acoustic volume rendering, and we plan to explore this in our future work.\n\n\n**W2: Include synthesized audios from other methods in the final supplementary material.**\n\nThanks for your feedback.  We also render binaural audios for other methods (NAF and INRAS) and will include it in the camera-ready version through link.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to express our gratitude to all the reviewers for their insightful comments and feedback. Below, we address two common questions raised by multiple reviewers: computing efficiency and zero-shot binaural audio rendering. The rest of the questions and comments are addressed individually for each reviewer.

**Computing efficiency**

To provide a more comprehensive understanding of our model's performance, we have conducted an additional analysis of its runtime efficiency and compared it with other methods. The results and analysis provided below will be incorporated into the camera-ready version of the paper if accepted.

We first compare the inference time of our model and other methods in Table 1 below. This includes the inference time for different methods when they are trained to output RIR of 0.1s and 0.32s.
Not surprisingly, our proposed method that uses volume rendering based on physical properties of wave propagation is significantly slower than the methods that directly output RIR with a network. 
Encouragingly, various techniques have been proposed in recent years to significantly improve the efficiency of volume rendering and NeRF through efficient sampling strategies [1,2]. We believe that these approaches can be similarly adopted for acoustic volume rendering, and we plan to explore this in our future work.


|         | 0.1s RIR | 0.32s RIR |
|:-------:|:--------:|:---------:|
| NAF     | 3.2 ms   | 6.4 ms    |
| INRAS   | **2.1 ms**  | **3.2 ms**    |
| AV-NeRF | 4.6 ms   | 6.9 ms    |
| **Ours**    | 30.3 ms  | 90.7 ms   |
||


We also analyze how the number of rays and the number of points along each ray would impact the performance and inference time in Table 2.
Our method becomes more accurate with increased amount of sampling for rays and points.
At the same time, reducing the number of rays and points can proportionally reduce the inference time, without significant performance loss.
When it comes to real-time applications, besides all the potential approaches that can be employed to enhance the runtime efficiency of our method, predictive algorithms can be used to pre-calculate potential impulse responses based on likely user motion, which helps reduce perceived latency.

| Variation                | STFT | Phase | Amp. | Env. | T60  | C50  | EDT   | Runtime  |
|:---------------------------------:|:----:|:-----:|:----:|:----:|:----:|:----:|:-----:|:--------:|
| 80x40 rays x 64 pts | 1.17 | 1.58  | 0.65 | 1.03 | 4.79 | 0.48 | 0.018 | 150.2 ms |
| 72x36 rays x 64 pts | 1.21 | 1.58  | 0.68 | 1.03 | 4.83 | 0.50 | 0.019 | 121.3 ms |
| 64x32 rays x 64 pts | 1.28 | 1.58  | 0.69 | 1.05 | 4.85 | 0.51 | 0.019 | 90.7 ms  |
| 56x28 rays x 64 pts | 1.35 | 1.58  | 0.70 | 1.08 | 5.46 | 0.53 | 0.023 | 72.3 ms  |
| 48x24 rays x 64 pts | 1.45 | 1.59  | 0.69 | 1.06 | 6.34 | 0.61 | 0.026 | 49.1 ms  |
||
| 64x32 rays x 80 pts | 1.15 | 1.58  | 0.67 | 1.02 | 4.81 | 0.48 | 0.019 | 120.5 ms |
| 64x32 rays x 72 pts | 1.19 | 1.58  | 0.68 | 1.02 | 4.84 | 0.51 | 0.019 | 105.6 ms |
| 64x32 rays x 64 pts | 1.28 | 1.58  | 0.69 | 1.05 | 4.85 | 0.51 | 0.019 | 90.7 ms  |
| 64x32 rays x 56 pts | 1.33 | 1.59  | 0.69 | 1.07 | 4.93 | 0.50 | 0.020 | 81.4 ms  |
| 64x32 rays x 48 pts | 1.31 | 1.59  | 0.71 | 1.09 | 5.05 | 0.52 | 0.020 | 71.9 ms  |
||


[1] Turki, Haithem, Vasu Agrawal, Samuel Rota Bulò, Lorenzo Porzi, Peter Kontschieder, Deva Ramanan, Michael Zollhöfer, and Christian Richardt. "HybridNeRF: Efficient Neural Rendering via Adaptive Volumetric Surfaces." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19647-19656. 2024.

[2] Hu, Tao, Shu Liu, Yilun Chen, Tiancheng Shen, and Jiaya Jia. "Efficientnerf efficient neural radiance fields." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12902-12911. 2022.

**Zero-shot binaural audio rendering:**

Zero-shot in our case refers to the fact that our model can generate accurate binaural audio despite being trained only on monaural audio modeling (without any fine-tuning). Our method is able to achieve this for multiple reasons. First, our model captures accurate phase information in the RIR to the extent that simply rendering the impulse response at the positions of the left and right ears can provide accurate phase differences, i.e., time delay or interaural time differences (ITD).
Second, our model can easily incorporate the head-related transfer function (HRTF) for modeling the shadowing and pinna effects. Specifically, these direction-dependent filtering effects can be integrated into Equation 8 before summing responses from all directions. By replacing the direction-dependent weight term with a direction-dependent HRTF function, we can achieve a more accurate binaural sound effect and reduce directional ambiguity (e.g., front versus back).
Furthermore, explicit incorporation of HRTF allows our method to work with different HRTF from different users, allowing for accurate and personalized listening experience.

The rendered binaural audio provided in the supplementary material demonstrates accurate modeling of ITD by simply rendering at the left and right ears. We will include in the camera-ready version the binaural audio samples with the baseline methods for qualitative comparison. We note that **no** HRTF function was applied for our binaural rendering, and therefore there was directional ambiguity (e.g., front vs back). We will also include an audio sample with HRTF incorporated with our method.

Meta-review: The paper received uniformly positive reviews. Reviewers highlighted its potential impact on the physics-based neural impulse response field (RLmp, DqiU, WhmA, 4897), the quality of results (RLmp, DqiU, 4897), the novelty of the approach (RLmp, DqiU, 4897), and the clear presentation (WhmA). While there were initial concerns regarding computational complexity, missing comparisons, and a lack of important details, these issues were effectively addressed in the authors’ rebuttal.

After thoroughly reviewing the paper, the reviews, and the authors’ rebuttal, the AC concurs with the reviewers’ highly positive consensus and therefore recommends the paper for acceptance (spotlight).

For the camera-ready version, the authors should ensure all discussions from the rebuttal are incorporated into the main paper and supplementary materials. The specific changes that need to be implemented are:

**1. Improved presentation**: Summarize the key characteristics of the simulator in the main text (DqiU)

**2. Additional experiments**: Include an efficiency comparison (DqiU), results with a 0.32s RIR duration (WhmA), and user-study results for zero-shot binaural audio rendering.

**3. Audio results**: Include synthesized audio examples from other methods (RLmp).
Decision: Accept (spotlight)