Acoustic Volume Rendering for Neural Impulse Response Fields

Authors: zitong  lan
chenhao  zheng
zhiwei  zheng
mingmin  zhao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-26 22:38:26

Abstract: Realistic audio synthesis that captures accurate acoustic phenomena is essential for creating immersive experiences in virtual and augmented reality. Synthesizing the sound received at any position relies on the estimation of impulse response (IR), which characterizes how sound propagates in one scene along different paths before arriving at the listener position. In this paper, we present Acoustic Volume Rendering (AVR), a novel approach that adapts volume rendering techniques to model acoustic impulse responses. While volume rendering has been successful in modeling radiance fields for images and neural scene representations, IRs present unique challenges as time-series signals. To address these challenges, we introduce frequency-domain volume rendering and use spherical integration to fit the IR measurements. Our method constructs an impulse response field that inherently encodes wave propagation principles and achieves state of-the-art performance in synthesizing impulse responses for novel poses. Experiments show that AVR surpasses current leading methods by a substantial margin. Additionally, we develop an acoustic simulation platform, AcoustiX, which provides more accurate and realistic IR simulations than existing simulators. Code for AVR and AcoustiX are available at https://zitonglan.github.io/avr.

Main paper: 
 introduction: 
 Our acoustic environment shapes every sound we hear -from the crisp echoes bouncing through hallways to the layered resonance of a symphony filling a concert hall. These spatial characteristics not only define our daily auditory experiences but also prove crucial for creating convincing virtual worlds [15, 60] . At the core of these spatial characteristics is the impulse response (IR), which captures the complex relationship between an emitted sound and what we hear. Like a unique acoustic fingerprint, the impulse response varies across different positions, encoding how sound waves interact with the environment through reflection, diffraction, and absorption [22, 41] . We can recreate the acoustic experience at any position by convolving the corresponding impulse response with any desired sound sources (e.g., music, speech). Given its foundational role in spatial audio synthesis, understanding and modeling the spatial variation of impulse responses in acoustic environments has emerged as a critical challenge and attracted increasing research attention [2, 26, 27, 29, 35, 36, 44, 49, 53] .Current approaches construct a neural impulse response field -a learned mapping that generates impulse responses given the emitter and listener poses. To model the high spatial variation of impulse responses, existing methods either fit a neural network to directly learn the field [29, 44] or rely on audio-visual correspondences to learn mappings from vision [26, 27] . While these methods can approximate the general energy trend, they struggle to capture the detailed characteristics of impulse responses, leading to incorrect spatial variation of impulse responses (Fig. 1 ).We argue that a key barrier to achieving better performance is the absence of physical constraints that inherently enforce consistency across multiple poses. Without such physical constraints, the network ? Figure 1 : Left: From observations of the sound emitted by a speaker, our model constructs an impulse response field that can synthesize observations at novel listener positions. Right: Visualization of spatial variation of impulse responses on MeshRIR [20] . The synthesized impulse responses at different locations are transformed into the frequency domain, where we visualize phase and amplitude distributions at a specific wavelength (1m).tends to overfit the training data and show poor generalizability. The received impulse response fundamentally arises from sound waves propagating through space, combining direct transmission with environmental reflections. This physical insight motivates us to develop a framework that inherently encodes wave propagation principles into the modeling of impulse response fields.In this paper, we introduce Acoustic Volume Rendering (AVR) to model the field of acoustic impulse responses. Our approach draws inspiration from Neural Radiance Fields [33] , which has demonstrated remarkable success in modeling 3D scenes by representing light transport through volume rendering. However, acoustic waves present several fundamental challenges that require adaptations to the volume rendering framework: First, acoustic impulse responses, unlike light transmission, are inherently time-series signals, with acoustic waves from different locations reaching the listener at varying delays. The issue is further compounded when dealing with discrete impulse responses sampled in the real world. Second, impulse responses exhibit high spatial variation, in contrast to images where neighboring pixels show strong correlations. This characteristic makes network optimization particularly challenging [43, 45] . Finally, unlike cameras that capture light with precise directional information (i.e., pixels), microphones capture combined signals from all directions.To address these challenges, we convert impulse responses from the time domain to the frequency domain with Fourier transforms and perform volume rendering in the frequency domain. We apply phase shifts to the frequency-domain impulse responses to account for time delays, bypassing the limits of finite time domain sampling. The frequency-domain representation also exhibits lower spatial variation, facilitating network optimization. To account for signals from all possible directions, we cast rays uniformly across a sphere and use spherical integration to synthesize the impulse response measurements. Additionally, this design enables personalized audio experience by integrating individual head-related transfer functions (HRTFs) [57] into spherical integration at inference time. Our evaluation results show that AVR outperforms existing methods by a large margin in both simulated and real-world datasets [10, 20] and can zero-shot render binaural audio (Sec. 4.3). In parallel with AVR, we develop AcoustiX, an acoustic simulation platform that generates more physically accurate impulse responses compared to existing simulators. While current simulators often introduce significant errors in signal phases and arrival times, AcoustiX produces impulse responses that better match the physical properties of real-world acoustics. Fig. 2 demonstrates the inaccuracies in impulse responses generated by SoundSpaces 2.0 [9] . Some existing simulators assign random phases when generating impulse responses [9, 40] , which fails to reflect real-world acoustic behavior [4] . Since current research in impulse response synthesis heavily relies on simulated datasets [10, 29, 44] , these simulation inaccuracies can impede progress in the field. To address these limitations, we develop a new simulation platform based on the Sionna ray tracing engine [16] and incorporate acoustic propagation equations to resolve the aforementioned issues. Similar to SoundSpaces 2.0, AcoustiX supports acoustic simulation in both user-provided 3D scenes and a variety of existing 3D scene datasets [24, 56] .In summary, this work makes the following contributions:• We introduce acoustic volume rendering (AVR) for the neural impulse response field to inherently enforce acoustic multi-view consistency. We introduce a frequency-domain rendering method with spherical integration to address the challenges associated with acoustic impulse response modeling.• We demonstrate that AVR outperforms existing methods by a large margin in both real and simulated datasets. AVR also supports zero-shot and personalized binaural audio synthesis.• We develop AcoustiX, an open-source and physics-based impulse response simulator that provides accurate time delays and phase relationships through rigorous acoustic propagation modeling. 
related work: 
 Impulse Response Modeling. Traditional impulse response modeling [2, 32, 49] relies on audio encoding, which encodes collected data and spatially interpolates the impulse response for unseen positions [2, 32] . However, this approach comes with large memory costs [29] and struggles to generate impulse responses with high fidelity [10] . Machine learning techniques have been integrated in recent years to enhance the quality of synthesized impulse responses [35, 36, 37] . For instance, generative adversarial networks (GANs) have been utilized for more realistic acoustic synthesis [36, 37] . As implicit representations have become more popular, several works [26, 27, 29, 44] in recent years proposed neural implicit acoustic fields and achieved state-of-the-art performance. However, these learning-based methods still produce unsatisfying waveform shapes and show weakness in novel impulse response synthesis [10] . To this end, our learning-based approach further integrates wave propagation principles [22] , synthesizing high-fidelity impulse responses.Neural Fields. Since the success of NeRF [33] , the concept of neural fields has been expanded comprehensively. Initially, incorporating depth supervision [12, 38, 51] into radiance fields proves helpful for novel view synthesis. Some following studies adopt occupancy [59] or signed distance functions [11, 54, 62] , instead of density, to represent scenes. Later, integrating volume rendering with measurements from sensors of other modalities, such as time-of-flight sensors [3] , LiDAR [18, 31, 55] , and structured light imaging [42] , has achieved great performance. Extensions [28, 61] have further modified volume rendering to model radio-frequency signals.Acoustic Simulation. Acoustic simulation primarily relies on either wave-based or geometric approaches to approximate sound propagation in indoor environments. While wave-based methods [7, 14, 46, 47, 52] are generally more precise for low-frequency sound, they require significant computation for high-frequency signal simulation. For faster acoustic simulation, geometric approaches have gained considerable attention. These methods, such as image source [1, 5, 39] and ray tracing [8, 9, 21, 41, 46] , are often used in practical applications like virtual reality. However, some commonly-used simulation platforms [8, 9] suffer from inaccuracies in time-of-flight calculation and phase simulation for sound propagation. AcoustiX ensures physics-based accuracy, facilitating further research on acoustic-related topics.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We point out the limitations in our Discussion section. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 Our objective is to learn an impulse response field for one scene to synthesize impulse responses for the unseen emitter and listener poses. The impulse response h(t) quantifies the received signal at a specified listener location p l oriented by ω l , resulting from an impulse emitted from p e with orientation ω e . It encompasses how sound propagates within a specific scene. We begin by revisiting fundamental principles of acoustic wave propagation. We then introduce our impulse response field and our frequency-domain acoustic rendering. Lastly, we discuss the implementation specifics. In addition, we summarize the key features of our simulator AcoustiX.We first explain the simplest form of acoustic propagation to highlight two basic properties of sound propagation: time delay and energy decay. Assuming an omnidirectional emitter at p e transmits a Dirac delta pulse δ(t) at time t = 0 uniformly into open space, the resulting signal at listener p l is given by [22] :EQUATIONand v denotes the velocity of sound. The orientations of both the emitter ω e and the listener ω l are ignored under the omnidirectionality assumption. We note that the listener captures a time-delayed emitted signal, with an amplitude decay inversely proportional to the distance traveled. Additionally, the phenomenon can be alternately represented in the frequency domain with the Fourier Transform:EQUATIONWith this representation, the time delay observed in the time domain manifests as a phase shift (e -j2πf τ ) in the frequency domain.In a real scenario, sound emitted by a source undergoes complex interactions with the geometric structures of the environment. Each location in the scene may absorb some energy from the incoming wave, resulting in signal absorption; it may also reflect and scatter the wave, leading to signal re-transmission. To model these complex effects, AVR represents scene as a field: given an emitter location p e and its orientation ω e , the network F Θ outputs two key acoustic properties for any point p in space given a direction ω:EQUATIONwhere σ represents acoustic volume density and the time-varying parameter s(t) models the acoustic signal transmitted out from the location p in direction -ω, including both initial emission and subsequent re-transmission.With this parameterization, we now render the signal h ω (t) received at a listener position p from direction ω, assuming the emitter is fixed and the impulse is emitted at time t = 0. Similar to volume rendering for light, our process adopts volume rendering to accumulate the signals emitted from all locations along the ray p(u) = p + u • ω, with predefined near and far bounds u n and u f . Differently, our approach also accounts for time delay and energy decay in acoustic signal propagation and performs alpha composition for time signals, resulting in our acoustic volume rendering equation:h ω (t) = 1 tv u f un L(u)σ(p(u))s(t - u v ; p(u), ω)du, where L(x) = exp - x un σ(p(x))dx .(4) Note that each emitted signal s(t -u v ; p(u), ω) is associated with a time delay u v . This delay accounts for the non-negligible sound propagation time, ensuring that the signal received by the listener at time t originates from location p(u) at the earlier time t -u v . To account for energy decay, We apply a factor 1 tv to all the signals along the ray, independent of their emission time. Since all signals originate from the impulse transmitted at time 0, the traveled distance of any signal received at time t is tv, whether it's from the original emission or a re-transmitted signal.Listener receives signals from all directions, influenced by its gain pattern. The signal obtained from a single ray can not represent the whole received impulse response. To account for this, the final impulse response captured by the listener is a combination of signals from all directions:EQUATIONwhere G(•) represents the listener gain pattern that characterizes the directivity of the listener, and Ω denotes the complete sphere of directions from which the listener receives signals. The whole acoustic rendering process is also shown in 3. ) is applied to account for the wave propagation. After that, we combine signals and densities to perform acoustic volume rendering for each ray to get the directional signal (h dir (t)). We integrate along the sphere to combine signals from all possible directions with gain pattern G(ω) to obtain the final rendered impulse response h(t).While the above method renders a continuous impulse response, the actual signal collected in the real world is discrete, sampled with a fixed interval T . This discretization converts the target impulse response from h(t) to h[n], where h[n] = h(nT ). Accordingly, we train neural networks to output s[n] at these discrete timestamps. However, this discrete sampling presents a fundamental challenge in acoustic volume rendering. To capture the signal from one direction, h ω [n], we need to evaluate the time-delayed signal s(nT -u v ). The key issue arises when u v is not a multiple of the sampling interval T , i.e., the required timestamps for delayed signals fall between our discrete samples, making accurate rendering difficult in the time domain. While one could add timestamp as an additional network input to interpolate between samples, this would require multiple network queries for each point, severely impacting rendering efficiency.We address this challenge by reformulating the problem in the frequency domain. A key insight is that time delays in the signal correspond to phase shifts in the frequency domain (Eq. 2). This allows us to achieve arbitrary time delays u v by transforming the predicted signal s[nT ; p(u), ω] into frequency domain F {s[nT ; p(u), ω]} and applying the corresponding phase shift, regardless of whether u v aligns with the sampling grid. Specifically, the delayed signal in the frequency domain can be obtained via:EQUATIONThe linearity of the Fourier Transform F allows us to extend acoustic volume rendering to the frequency domain with the same alpha composition (i.e., integration) process:EQUATIONHere, the multiplication with energy decay factor 1 tv in Eq. 4 becomes the convolution with F 1 tv in the frequency domain. Eq. 7 can be regarded as Eq. 4 in the frequency domain by applying discrete Fourier Transform on both sides of the equation. Similarly, the final impulse response in the frequency domain can be formulated analogously to Eq. 5:EQUATIONFinally, time-domain impulse response h[n] can be obtained through inverse Fourier Transform.To handle the integration in Eq. 7 and Eq. 8, our sampling strategy includes both ray sampling over a sphere and point sampling along a ray. We perform ray sampling by selecting N θ azimuthal directions and N ϕ elevational directions, resulting in N θ × N ϕ distinct orientations that uniformly cover the sphere. For a ray along one of the directions, point sampling is conducted by evenly placing N r points between predefined near and far bounds, similar to [61] . Consequently, the network is queried at N θ × N ϕ × N r points for the synthesis of an impulse response. We refer readers to Appendix A for the details of our sampling strategy.We supervise the synthesized impulse responses H[f ] and h[n] alongside the ground truth H * [f ] and h * [n] in both frequency and time domains. We emphasize supervision in the frequency domain since the responses exhibit smaller local variability. In the frequency domain, L spec measures spectral loss by comparing the real and imaginary components.EQUATIONWe also supervise the amplitude and phase of the synthesized signals with L amp and L phase .EQUATIONEQUATION) For time domain signals, L time is employed to encourage amplitude consistency.EQUATIONOur total loss is a linear combination of the above loss with different weights, including a multiresolution STFT loss L stft [58] and an energy loss L energy similar in [30] :EQUATIONAcoustiX uses Sionna ray tracing engine [16] . We modify the ray tracing in terms of ray interactions with the environment to support acoustic impulse response simulations. The simulator supports various ray interactions with the environment. Each material in the scene is assigned with frequencydependent coefficients. This enables the tracing of cumulative frequency responses for each octave band to accurately simulate the impulse response. Room models can be created using Blender and exported as compatible XML files for our simulation setup. AcoustiX also supports the import of 3D room models from the iGibson dataset [24, 56] . More details can be found in Appendix D.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: Our code, dataset, and simulation platform are open-sourced in the project website.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiments: 
 Implementation Details. The input to our model is an emitter's pose (position p e ∈ R 3 , direction ω e ∈ R 3 ) and a 3D query point's pose (p ∈ R 3 , ω ∈ R 3 ), The model outputs the corresponding density σ ∈ R and discrete time signal s[n] ∈ R T at that query point. We first encode all input vectors into high-dimensional embeddings using hash grid [34] . The encoded input embeddings are then passed into a 6-layer MLP. The first 3 layers of MLP take as input locations (p e , p) and predict the density σ and a 256-dimensional feature. The feature and encoded directions (ω e , ω) are then concatenated and passed into the last 3 layers, which outputs the signal sequence s[n].The sampling numbers used in the experiments are N θ = 80, N ϕ = 40, and N r = 64. We set the weights of loss components to be λ amp = λ phase = 0.5, λ time = 100, λ stft = 1, λ energy = 5. We train our model for 200 epochs for each scene. We use Adam optimizer with a cosine learning rate scheduler that starts at a learning rate 10 -3 and decays to 10 -4 . The optimization process takes 24 hours on a single NVIDIA L40 GPU.Evaluation Metric. We use comprehensive metrics to assess the quality of our method. Following [10, 29] , we measure the energy decay trend by Clarity (C50), Early Decay Time (EDT), and Reverberation Time (T60). For the measurement of the correctness of waveform shape, prior works only consider the amplitude in the frequency domain (e.g. STFT error) to assess the performance, which we argue only indicates part of the waveform information: the transformed frequency signal is a complex number that is jointly defined by amplitude and phase. We therefore also include the frequency-domain phase error in our measurement: the L1 norm of the error in the cosine and sine components of the phase. Besides the frequency domain, we also measure the time-domain impulse response signal accuracy by calculating the L1 error in the envelope, denoted as envelop error. Please refer to Appendix B for the detailed definition of each metric.Baselines. We compare with both learning-based methods using neural implicit representations and traditional methods. NAF [29] is the first method that uses neural implicit representation to model the impulse response field. It uses an MLP to predict the spectrogram of the impulse response signals. Another method INRAS [44] disentangles the sound transmission process into three different learnable modules. Different from NAF, INRAS models the impulse response in the raw time domain. We also implement traditional audio encoding methods AAC [6] and Opus [50] and adopt the same setting as [29] .We evaluate our model's performance on the datasets collected from real scenes. We adopt two commonly used room impulse response datasets: MeshRIR [20] and Real Acoustic Field [10] . MeshRIR collects monaural impulse response in a cuboidal room. Ground truth The results of all methods are shown in Tab. 1. We find that our method significantly outperforms the existing stateof-the-art baselines in all the datasets. We note that no existing learning-based method but ours performs better than chance in the phase error metric. In Fig. 4 and Fig. 5 , we show visualizations of the learned impulse response field for the entire scene. Our method captures much more spatial signal distribution than prior works. We also show an example of individual time-domain impulse response in Fig. 6 . Although prior methods can capture the general decaying trend of the impulse responses, the waveforms (e.g. the peaks) are misaligned with the ground truth. In contrast, our method captures the waveforms much better. We especially point the readers to the time that the signal arrives for every impulse response, which indicates timeof-arrival. Our method has much smaller errors in terms of time-of-arrival due to our physics-based rendering.Due to the high cost of equipment required to collect large-scale impulse responses in a real scene, MeshRIR and Real Acoustic Field are the only real-world impulse response datasets that are collected densely in an enclosed space. Researchers typically use simulated impulse response data in virtual 3D environments to complement the real-world datasets [26, 29, 44] .We use our simulation platform to simulate monaural impulse responses in three rooms and evaluate all methods' performance (Tab. 2). The simple 2D room is a 2D rectangular-shaped enclosed space with only one wall in the middle of the room, serving as a toy example. We also include two complicated 3D rooms from iGibson dataset [24, 56] . All rooms are equipped with a single omnidirectional speaker, with listeners placed randomly in the rooms. All the impulse responses are sampled at a 16 KHz sampling rate with 0.1s duration time. We follow the same split strategy used in real-world datasets. Tab. 2 shows that our method consistently outperforms existing methods in all datasets. From the spatial signal distributions in Fig. 4 , we observe that even in the simple 2D room the prior methods fail to accurately capture the accurate field distribution, while the field distribution generated by our method consistently matches the ground truth in both 2D and 3D cases. Our estimated time-domain signals at unseen poses are also closely matched with the ground truth signals, shown in Fig. 6 .AVR can generate accurate binaural audio despite being trained only on monaural audio modeling (without any fine-tuning). The existing method for rendering binaural audio either requires training at binaural channel spatial audio data [13] or manually creating signal delays. We render impulse response of left and right ears separately (20cm apart) in the MeshRIR scene. We play a piece of music 3 meters away from a listener, who turns its head from left to right and back again. We conduct a user study comparing the spatial perception of rendered binaural audio among NAF, INRAS, and our method. Seven users rated the similarity between expected head trajectories and their hearing experience on a 1-5 scale. Our method achieves the highest score of 4.71, compared to NAF's 1.42 and INRAS's 1.86. Other methods fail to synthesize accurate binaural audio because they are trained solely on monaural audio. Audio examples are available on our project website.AVR is able to achieve binaural audio rendering for multiple reasons. First, our model captures accurate phase information in the impulse response to the extent that simply rendering the impulse response at the positions of the left and right ears can provide accurate phase differences, i.e., time delay or interaural time differences (ITD). Second, our model can easily incorporate the head-related transfer function for modeling the shadowing and pinna effects. Specifically, these direction-dependent filtering effects can be integrated into Eq.8 before summing responses from all directions. By replacing the direction-dependent weight term G(ω) with a direction-dependent HRTF function, we can achieve a more accurate binaural sound effect and reduce directional ambiguity (e.g., front versus back). Furthermore, explicit incorporation of HRTF allows our method to work with customizable HRTF for different users, allowing for an accurate and personalized listening experience. A comparison of runtime efficiency between AVR and other methods are shown in Tab. 3. This includes the inference time for different methods when they are trained to output IR of 0.1s and 0.32s. Since AVR uses acoustic volume rendering over a sphere, it is slower than the methods that directly output IR with a network. Encouragingly, various techniques have been proposed in recent years to significantly improve the efficiency of volume rendering and NeRF through efficient sampling strategies [17, 25, 48] . These approaches can be similarly adapted for acoustic volume rendering. More analysis on computing efficiency can be found in Appendix C.1.We ablate different choices of the sampling parameters during volume rendering, rendering domain, and loss components (Tab. 4). All models are evaluated on the MeshRIR dataset. Sampling Parameters. We study the sensitivity of our model to sampling parameters N θ , N ϕ , N r . We find that both increasing the ray numbers and the sampling points will both enhance the performance, but come with the cost of low training speed and high memory consumption.Rendering Domain. We train our model using both time-domain volume rendering and frequencydomain volume rendering. Frequency-domain rendering effectively avoids issues associated with fractional time delays, aligning more accurately with the actual phenomenon of acoustic signal propagation. Consequently, this approach yields better results, confirming our argument in Sec. 3.3.Loss Component. We also ablate loss components. We find that reducing any of the loss components results in decreased performance. However, it is noteworthy that all model variants, except for the one trained without the angle and spectral loss, outperform the baselines discussed in Sec. 4.1. 
discussion: 
 Limitations and Future Work. Our rendering involves both spherically sampling rays and sampling points along each ray, which can lead to large memory consumption and longer inference time.Recently, many research works have been proposed to improve the efficiency of volume rendering and NeRF through efficient sampling strategies. We envision that similar methods could also be applied to acoustic volume rendering to speed up the rendering. Besides, AVR needs to train a new model for a novel scene, which requires effort to collect impulse response samples in the new scene. Future work could explore generalization to novel scenes by incorporating multi-modal inputs, aiming to synthesize an impulse response field using only a few visual or acoustic samples.Conclusion. This paper proposes acoustic volume rendering to reconstruct impulse response fields that inherently encode wave propagation principles. We introduce frequency-domain signal rendering and spherical signal integration to address the unique challenges in impulse response modeling. Experimental results demonstrate that AVR significantly outperforms existing approaches. Additionally, we develop AcoustiX, an open-source simulation platform that provides accurate time-of-arrival measurements. Our work advances immersive auditory experiences in AR/VR, spatial audio in gaming and virtual environments, teleconferencing, and acoustic modeling in architectural design.Our realistic auditory simulations also benefit autonomous navigation, acoustic monitoring, and assistive hearing technologies where accurate acoustic modeling is essential.The direction of a ray can be represented by two measures: azimuth θ and elevation ϕ. We handle ray sampling by performing both azimuth and elevation sampling. For azimuth sampling, we apply stratified sampling between 0 and 2π to obtain another N θ rays, where i is the index:EQUATION)For elevation sampling, we evenly distribute N ϕ rays, with ϕ j = arccos(2 j N ϕ -1), where j is the index. By combining all azimuth and elevation angles, we obtain N θ × N ϕ directions in 3D Cartesian coordinates, each represented as follows:EQUATION)For each sampled ray, we uniformly sample N r points on it. Given a ray p(u) = p l + u • ω, starting from the point p l in direction ω, the position of the m th point is given by:EQUATION)With this sampling, we approximate the integral in Eq. 7 using quadrature as follows:H ω [f ] = F 1 tv * Nr m=1 T m (1 -exp(-σ m ∆u)F s(nT - u v ; p(u m ), ω) ,whereEQUATION)Combing our ray sampling strategy, we rewrite the Eq.8 and as follows:EQUATIONEnvelope Error. Given the time domain ground truth impulse response h * [n] and our prediction h[n], we can compute the envelope error by first obtaining the envelope using the Hilbert transform to get the analytic signal and then applying the absolute value, as follows:EQUATIONThe normalized envelope error is defined as follows (we multiply it by 100 to avoid small numbers):EQUATIONPhase and Amplitude Error. Given the frequency domain ground truth impulse response H * [f ] and our prediction H[f ], we use a cosine and sine function encoded function to quantify the phase error:EQUATION)The amplitude error is defined as follow:EQUATIONC More Evaluation ResultsWe further examine the relationship between inference speed and the number of rays as well as the number of points sampled along each ray. As illustrated in Fig 7 , the inference speed scales approximately linearly with both the number of rays and the number of points along each ray. We repeated our experiments with a 0.32s RIR duration. Tab 5 shows the results on the RAF-Furnished dataset. We also included AV-NeRF as a baseline and multi-resolution STFT as an evaluation metric. With a 0.32s RIR duration, our method also outperforms these baselines. We provide loudness map visualization (Fig 8 ) for two different speaker positions at RAF-Furnished dataset with a grid size of 0.1m. Our method can better capture sound level differences caused by geometry occlusion and has a smoother spatial variation of loudness. AcoustiX is built based on Sionna [16] ray tracing engine that supports ray reflection, scattering, and diffraction. We modify the ray tracing engine in terms of ray interactions with the environment to support acoustic impulse response simulations. Each material in the scene is assigned a reflection coefficient β and a scattering coefficient α. For each reflection, the reflected wave's amplitude isE ′ = (1 -α)βE with E being the energy before the interaction. The scattered energy is E ′′ = αβE. With these notations, the impulse response is formulated as follows:EQUATIONwhere d n is the accumulated total length of n th path, v is the velocity of sound in the air, α n,k and β n,k denote material properties of k th reflection. We use the negative reflection coefficient definition discussed in [23] . In the equation, we assume purely specular reflection for simplicity. If scattering or diffraction occurs along the path, we replace the reflection term with the corresponding scattering or diffraction attenuation. δ LP is a windowed sinc function defined similar to the one in [39] .In the implementation, we divide the whole frequency band into several octave bands and get their common acoustic properties. To assign frequency-dependent reflection and scattering coefficients to each material, we transfer Eq. 23 to the frequency domain and assign a coefficient to the amplitudes for each frequency band. The material coefficient is retrieved from [19] . In AcoustiX, users can determine the number of cast rays and maximum bouncing depth in the simulation. AcoustiX supports different ray-geometry interactions including reflection, scattering, and diffraction. By default, we enable all the functions above and set the maximum bouncing depth to 30 and the number of cast rays to 1e6 to enable comprehensive path searching within the rooms. We provide flexible API usage in AcoustiX, allowing users to adjust the acoustic ray tracing configurations and balance between simulation quality and speed. Fig. 9 shows an example of our simulated impulse responses.AcoustiX supports customized room models. We create room structures with Blender, assign material names to all objects, and export the scene in XML formats using Mitsuba blender Add-on 1 .During simulation, each object is matched with its corresponding acoustic material properties by looking up a table mapping assigned names to properties. In addition to customizing room models, we also support importing 3D room models from the iGibson dataset [24, 56] into our simulations, assigning acoustic properties to each object.As our method can synthesize high-quality impulse responses, our work can potentially enhance immersive VR/AR experiences and sound-dependent applications. AcoustiX fosters research and innovation in acoustic topics. Potential negative social impacts include the creation of misleading audio content, which could be used to deceive or manipulate users. For instance, high-quality impulse response generation could be exploited to fabricate realistic but fake acoustic environments or conversations, leading to misinformation or privacy violations.1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: We clearly state our contributions in the abstract and introduction and conduct extensive experiments on different datasets to support our claim. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We have all the training and testing details, including data splits, hyperparameters, ablation studies on hyperparameters and the description of the used optimizer.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: We don't report error bars because this would be too computationally expensive.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We mention both potential societal impacts in the appendix. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: Our model and data do not pose such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite the corresponding paper when needed and follow the corresponding license when using public datasets and the licensee that the Sionna ray tracing engine uses.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes]Justification: We talk about the details of our simulation platform when we introduce it.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: Our paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: Our paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.https://github.com/mitsuba-renderer/mitsuba-blender/tree/latest 
 
 Appendix:  


Decision: The paper received uniformly positive reviews. Reviewers highlighted its potential impact on the physics-based neural impulse response field (RLmp, DqiU, WhmA, 4897), the quality of results (RLmp, DqiU, 4897), the novelty of the approach (RLmp, DqiU, 4897), and the clear presentation (WhmA). While there were initial concerns regarding computational complexity, missing comparisons, and a lack of important details, these issues were effectively addressed in the authors’ rebuttal.

After thoroughly reviewing the paper, the reviews, and the authors’ rebuttal, the AC concurs with the reviewers’ highly positive consensus and therefore recommends the paper for acceptance (spotlight).

For the camera-ready version, the authors should ensure all discussions from the rebuttal are incorporated into the main paper and supplementary materials. The specific changes that need to be implemented are:

**1. Improved presentation**: Summarize the key characteristics of the simulator in the main text (DqiU)

**2. Additional experiments**: Include an efficiency comparison (DqiU), results with a 0.32s RIR duration (WhmA), and user-study results for zero-shot binaural audio rendering.

**3. Audio results**: Include synthesized audio examples from other methods (RLmp).

Reviews: review: paper_summary: paper_summary: The authors reformulate neural fields to model the spatial interactions of sound, by learning to predict impulse response in a fixed transmitter / arbitrary receiver pose geometry. The method substantially outperforms contemporary work on simulated and real world datasets, and lays the groundwork for new immersive AR/VR applications.
strength_weakness: strength_weakness: 1. I feel that the ablation study + associated commentary is limited (see my questions in the next section). This is exacerbated by the fact that performance metrics are reported without standard deviations. I saw that the authors' rationale in the checklist for omitting error bars was budget, but I still think that expanding this section to include experiments on different network sizes, loss coefficients, etc. would increase this work's utility to researchers with different computational budgets, and better elucidate the impact of each component of the method. It would be nice to expand the commentary in the discussion with the authors' hypothesized explanations for results as well. 

2. The paper lacks a crucial discussion on the computational complexity and runtime of the proposed method compared to existing approaches.
questions: questions: 1. How does the computational complexity of your method compare to existing approaches, particularly for real-time applications?

2. In lines 282-294 (discussion of zero-shot binaural audio rendering), are there any user studies / qualitative evaluations that could characterize the zero-shot performance vs. a network optimized for that particular task?

3. In table 3, it seems counterintuitive that removing raw signal loss improves phase error.. do the authors have any insight into this result?

4. In lines 302-304, its not clear to me why frequency-domain rendering gives significantly better results than time-domain rendering.. I understood the claim that the former is more compute-efficient, but don't see the claim that this should occur in Section 3.3 as the authors state -- could the authors clarify this?
limitations: limitations: The authors do acknowledge memory consumption (due to omnidirectional ray sampling) as a limitation. It would be good to expand on other potential limitations like generalization across environments, or applicability to time-critical applications.
paper_summary: paper_summary: This work presents a method to render a neural impulse response field to generate room impulse responses at novel positions after being trained with a number of limited RIR samples. The method works similar to a Nerf, rays are cast from a sphere and sampled at discrete positions along the rays. Here the signal at each sample point is transferred into the frequency domain and a phase shift is applied to delay them in time. The goal is to render the acoustic signal transmitted and volume density for any observer-emitter pair with known orientations and locations. 

The approach uses among other methods two datasets to compare itself against common baselines in the domain. The authors emphasize having the best phase reconstruction compared to other results.
strength_weakness: strength_weakness: There are several odd observations and design decisions in this paper which at least should be explained. Their impact on performance is potentially great as can be probably seen in their baseline comparison. They differ by a large amount from the existing literature. Also, runtime (inference time) and impact of design decisions on it are not discussed but probably a major hurdle for this method. Without further explanation about the comparison with the state of the art and the validity of the method this contribution can not be sufficiently judged:

Especially since one of the arguments is about issues with sampling, the relation to the sampling rate used should not go uncommented. The authors decide to resample impulse responses to 16 kHz yet the RAF dataset is sampled at 48 kHz. Does that mean in Table 1 all other methods are also compared at 16 kHz? In the RAF paper, Table 7 shows results for this 16 kHz case for NAF and INRAS. They combine the furnished and empty case but for example, Inras has a T60 error of 5.34. In Table 1 of this paper, Inras is described as having errors of 11.3 and 10.6 in the furnished and empty case. Why is the baseline roughly twice as bad as reported in that paper? The EDT values of INRAS seem to have an opposite relation with 0.019 sec (RAF paper) vs. 57.4 MICROseconds and 62.9 MICROseconds. Either the authors mean milliseconds, in that case the results are worse without explanation, or they actually mean microseconds, in that case the results of Inras are way better in the RAF paper also without explaining why. Most of the other results are also not consistent with the RAF paper. The author need to explain why their baseline results seem to be so much worse than in the dataset paper.

If the explanation is the different split the authors use, they should explain why they did not use the split of 80%, 5%, 15% for training validation and test as used in the RAF paper or why a difference of 5% in the split leads to these vastly different results.

Another reason could be cutting the samples to 0.1 s. RAF compares samples at 0.32 s. 

Large churches may have over 10 seconds of RT60. A typical living room may have 0.5 s RT60. Limiting the method to 0.1 s seems arbitrary.

The supplementary material plays music which sounds as if it starts towards the front left and then moves behind the head. Figure 7 shows a different route of the music. I did on purpose listen first to the audio before I read the description because expectation can change this experience. It is possible to imagine the sound being in the front or the back and both being plausible. This is only my experience and a number of test-listeners would be necessary to verify my findings. However, for me the audio experience is different than real binaural recordings which make me wonder if using a HRTF can really be omitted. If the authors simply create a time-delayed signal but without shadowing effects of the head and other important factors of the HRTF this is not binaural sound, this is stereo and then the directional ambiguity in perception can be explained. The authors should go into detail why they consider this zero-shot binaural sound.

This is a minor point and I does not reflect at all on the quality of the manuscript or the research but I would like to suggest to the authors to rename N_azi and N_ele to N_az and N_el to be consistent with other notations such as N_pt and avoid the distraction which could stem from reading the variable name N_azi wrong.

Little mistakes:
- In equation 12 the font of L_time is inconsistent with line 205
- Line 503 has one additional "frequency" word
- Line 509 starts with IIn
questions: questions: - Why is 0.1 s chosen as duration time? 

- What is the inference time of this method?

- What is the expected computation time if a duration of 1 second would be used?

- How is the inference speed dependent on the number of rays? Optimization seems to take 24 hours for 200 epochs which is not bad but if this can only be used for very short RIRs in very small rooms or with a reduced number of rays these are important limitations which should be discussed or measured and explained.

- How do the authors choose the size of the sphere they sample on? For example is it room dependent? Do they need the measurements of the room? What about complex scenes like apartments with connected rooms that are not shoebox like?

- Why do the authors not compare to AV-NeRF in table 1. The results from the RAF paper could have been taken to compare against a visual-audio method if split, sampling etc. would have been the same as in the RAF paper.

- What is the intuition/reson to concatenate directions only into the last 3 layers of the MLP?

- What is the bouncing depth of 30? Is that meters? Is that because 0.1 * speed of sound makes bouncing beyond 30 m irrelevant?

- Why is the phase important for this use case? The authors argue with VR applications. Many audio and audio-visual tasks ignore phase. If the method, after correcting the issues mentioned with the evaluation, ends up being closer to the other methods or even worse, then what is the benefit of reconstructing phase? 

- To compare truly with the RAF paper, what is the STFT error? Having the waveform the authors could calculate the multi-resolution STFT loss to compare over the whole dataset instead of comparing waveforms on limited examples in figure 6.
limitations: limitations: The biggest limitation coming from using a duration of 0.1 seconds seems unexplained.

Inference time and impact of more complex rooms or longer durations seems important but is unexplained.
paper_summary: paper_summary: This method proposes a physics-based approach for learning neural impulse response field through implicit wave propagation modeling. This is the first work that really incorporates acoustic wave propagation principles for constructing an impulse response field, leveraging volume rendering techinques in analagy to how it's applied in NeRF. There is also a new acoustic simulation platform introduced for simulating more realistic impulse respones. Experiments show that the proposed method outperforms several previous learning based methods.
strength_weakness: strength_weakness: - The paper introduces a new acoustic simulation platform, which is claimed as one major contribution of the paper. However, in the main text, there is no detailed description of the simulator, and only says it is based on Sionna [16]. I then found it in Supp, but it would be good to also briefly summarize the key characteristics of this new simulator in the main text.

- Since it's the first method that implicity models wave propagation for neural impulse response rendering, it would be important to also compare with prior physics-based methods that is geometry-based to highlight the superity of modeling wave propogation, apart from those learning based methods. For example, how would the proposed method compare to this recent work https://arxiv.org/abs/2406.07532, that models the acoustic field using an image-source differentiably method.

- It would also be good to have some separate comparisons for high-frequency components vs. low-frequency components, and far-field vs. near-field to better understand the limitations and strength of the proposed framework. For example, for equation 1, the 1 / d assumption might not hold well for near field.

- The qualitative examples are somewhat lacking. It would be much more informative to include some qualitative comparisons to prior work.
questions: questions: Apart from the questions in the Weakness section, another question I have is related to the zero-shot binaural audio experiment. The paper claims that the proposed method can even render binaural audio even without HRTFs, which is a bit confusing to me. Even if the simulator can render binarual audio by synthesizing the sound heard at the left and right ears, there are other ear/head-related effect are not factored in, so how can binarual audio be directly generated?
limitations: limitations: There are some limitations discussed, but it would be great to hear from the authors on more discussions on limitations especially on the compute efficiency (e.g., how much time it takes for the network to converge?) and memory usage compared to prior work, so that it can give peope the complete picture on the pros and cons of the proposed framework.
paper_summary: paper_summary: This paper addresses the challenge of novel sound synthesis from arbitrary positions. To achieve this, acoustic fields are modeled using an implicit neural representation, integrating acoustic wave propagation rules. Subsequently, a specifically designed volume rendering technique is employed to ensure consistency across diverse positions. Results from both simulated and real datasets validate the effectiveness of our approach. Additionally, the authors have developed a new impulse response simulation platform.
strength_weakness: strength_weakness: 1. The network is not very big (6 layers), I do not understand why the training takes 24 hours. Actually, the cited paper [29] is very fast, could the training time be reduced?
2. I enjoyed listening to the attached audio, which exhibits a strong sense of three-dimensionality. Including synthesized audios from other methods in the final Supplementary Material would enhance the comprehensiveness of the submission.
questions: questions: See Weakness.
limitations: limitations: See Weakness.

Comments: comment: We are pleased to hear that your concerns regarding comparisons with existing work, scalability, and computation details have been resolved satisfactorily. Thank you for your support and recognition of our work!
We appreciate your suggestion and will incorporate the efficiency comparison into the main paper. We are also excited about the opportunity to open-source our simulator and contribute to the community. Thank you for your enthusiasm!
We are grateful for your acknowledgement that all of your concerns have been addressed. Thank you for your time and effort in reviewing our work.
Thank you for your response. All of my concerns have been addressed.
Thank the authors for the additional comparisons and clarifications. The efficiency comparison is especially helpful for readers to fully understand the tradeoff of the proposed method, and I would encourage the authors to include a brief version of it in the main paper. This is a very interesting paper, and I would be very happy to see it appear in the conference and also see the open-sourcing of the new simulator for the community to use. Therefore, I am keeping my original rating and enthusiastically recommend the acceptance of the paper.
I would like to thank the authors for their detailed explanations, additional experiments and clarifications. My biggest concerns about comparison with existing work, scalability and being explicit about the computation, have been well addressed. 

Given the potential positive impact in the domain, providing phase, and due to the excellent work in providing further analysis I have changed my recommendation to accept.
Continue of **Zero-shot binaural audio rendering**:

We conducted an additional user study to compare the spatial perception of rendered binaural audio among NAF, INRAS, and our method. Seven users listened to each audio sample and evaluated the similarity between the expected head trajectories and their hearing experience. They then rated the similarity on a scale of 1 to 5, with 5 being the highest similarity and 1 being the lowest. Results show that our method achieved the highest similarity score of 4.71, while NAF and INRAS scored only 1.42 and 1.86, respectively. We will add this additional user study in the camera-ready version if accepted.