Learning Neural Acoustic Fields
Authors: Andrew Luo
Yilun Du
Michael J. Tarr
Joshua B. Tenenbaum
Antonio Torralba
Chuang Gan
Venue: NeurIPS 2022 Accept (2022)
Keywords: Implicit representations
neural fields
deep learning

Abstract: Our environment is filled with rich and dynamic acoustic information. When we walk into a cathedral, the reverberations as much as appearance inform us of the sanctuary's wide open space. Similarly, as an object moves around us, we expect the sound emitted to also exhibit this movement. While recent advances in learned implicit functions have led to increasingly higher quality representations of the visual world, there have not been commensurate advances in learning spatial auditory representations. To address this gap, we introduce Neural Acoustic Fields (NAFs), an implicit representation that captures how sounds propagate in a physical scene. By modeling acoustic propagation in a scene as a linear time-invariant system, NAFs learn to continuously map all emitter and listener location pairs to a neural impulse response function that can then be applied to arbitrary sounds. We demonstrate NAFs on both synthetic and real data, and show that the continuous nature of NAFs enables us to render spatial acoustics for a listener at arbitrary locations. We further show that the representation learned by NAFs can help improve visual learning with sparse views. Finally we show that a representation informative of scene structure emerges during the learning of NAFs.

Learning Neural Acoustic Fields
Andrew Luo Carnegie Mellon University
Yilun Du Massachusetts Institute of Technology
Michael J. Tarr
Joshua B. Tenenbaum
Carnegie Mellon University
MIT BCS, CBMM, CSAIL
Antonio Torralba Massachusetts Institute of Technology
Chuang Gan
UMass Amherst and MIT-IBM Watson Al Lab
Abstract
Our environment is filled with rich and dynamic acoustic information. When we walk into a cathedral, the reverberations as much as appearance inform us the sanctuary's 's wide open space. Similarly, as an object moves around us, we expect the sound emitted to also exhibit this movement, While recent advances in learned implicit functions have led increasingly higher quality representations of the visual world, there have not been commensurate advances in learning spatial auditory representations. To address this gap, we introduce Neural Acoustic Fields (NAFs), an implicit representation that captures how sounds propagate in physical scene. By modeling acoustic propagation in scene as a linear time-invariant system, NAFs learn to continuously map all emitter and listener location pairs to a neural impulse response function that can then be applied to arbitrary sounds. We demonstrate NAFs on both synthetic and real data, and show that the continuous nature of NAFs enables us to render spatial acoustics for a listener at arbitrary locations. We further show that the representation learned by NAFs can help improve visual learning with sparse views. Finally we show that a representation informative of scene structure emerges during the learning of NAFs. Project site: https://www.andrrew hww.:/wwwww......///////////t/oe
Acclsllcouussateee Fields
Introduction
The sound of the ball leaving the bat, much its visible trajectory, tells whether the hit is likely to be a home run or not. Our experience of the world around us is rich and multimodal, depending on integrated input from multiple sensory modalities. In particular, spatial acoustic cues provide with a sense of the direction and distance of a sound source without needing visual confirmation, allow estimate the properties of a surrounding environment, and critical subjective realism in gaming and virtual simulations.
Recent progress in implicit neural representations has enabled the construction of continuous, differ- entiable representations of the visual world directly from raw image observations [Sitzmann et al., 2019, Mildenhall al., 2020, Niemeyer et al., 2020, Yariv et al., 2020]. However, the physical world is informed not only by our visual observations, but also by the spatial acoustic cues present in the environment. As preliminary step in learning the acoustic properties of scenes, we explore an implicit model that represents the underlying impulse response of audio reverberations. As shown in Figure our model can model the spatial propagation of sound in physical scene.
perception of
Past work has explored capturing the underlying acoustics of scene [Raghuvanshi and Snyder, 201 4, 2018, Chaitanya et al., 2020]. These models, however, require handcrafted parameterizations which,
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

(c)
Figure I: Neural Acoustic Field (NAF) learns an implicit representation for acoustic propagation. (a) A 3D top-down view of the house with two rooms. (b)-(e) The loudness of acoustic field predicted by our NAF is visualized for an emitter located at the red dot. Notice how sound does not leak through walls, and the portaling effect open doorways can have. Louder regions are shown yellow.
critically, prevent such approaches from being applied to arbitrary scenes. In this work, we extend this approach by constructing implicit neural representation which captures, in generic manner, the underlying acoustics of a scene.
Learning representation of scene acoustics poses several challenges compared to the visual setting. First, how do we generate plausible audio impulse responses at each emitter-listener position? While we may represent the visual appearance of a scene with an underlying three-dimensions. vector, an acoustic reverberation (represented as an impulse response) can consist of over 10,000 time-domain values and, thus, is significantly harder to capture. Second, how do we learn acoustic neural representation that densely generalizes to novel emitter-listener locations? In the visual setting, ray-tracing can enforce view consistency across large portions of visual scene (modulo occlusions). While in principle, in a similar manner, we may reflect acoustic "rays" in a scene represented as an implicit function obtain impulse response. intractable amount of compute is necessary to obtain the desired representation [Srinivasan et al., 2021].
To address both challenges, we propose Neural Acoustic Fields (NAFs). To capture the complex signal representation of impulse responses in compact and spati ally continuous fashion, NAFs encode and represent an iepulse-response in the time- freeueeyy domain. Motivated by the strong influence of nearby geometry on anisotropic reflections [Raghuvanshi and Snyder, 2018], we propose to condition NAFs on local geometric information present at both the listener and emitter locations when decoding the impulse response. In our framework, local geometric information learned directly from impulse responses. Such a decomposition facilitates the transfer of local information captured from training emitter-listener pairs to novel combinations of emitters and listeners.
We show that NAFs are able to outperform baselines in modeling scene acoustics, and provide detailed analysis of the design choices in NAFs. We further illustrate how the structure learned by NAFs can improve mross-modal generation of novel visual views of scene. Finally, we illustrate how the learned representation of NAFs enable the downstream application of inferring scene structure.
2 Related Work
Audio Field Coding There is a rich history of sound field representation, encoding, and inter- polation methods for 3D spatial audio. Some approaches seek to directly approximate the sound field [Mignot al., 2013, Antonello et al., 2017, Ueno al., 2018) 18] while adding handcrafted priors. Others adopt parametric representation that seek to model only the perceptual cues [Raghuvanshi and Snyder, 2014, 2018, Chaitanya et al., 2020, Mehra et al., 2014, Ratnarajah et al., 2021]. Since the complete acoustic field of a scene is computationally prohibitive to simulate in real time, and expensive to store full fidelity, these methods have typically relied on a handerafted encoding of the acoustic field, prioritizing efficiency above reproduction fidelity. In recent years, there has been interest in using deep learning to directly Iearn a sound field from data, without making strong assumptions about the scene. However practice, these approaches either a stationary listener emitter [Richard et al., 2020, 2022]. In contrast, our work enables the querying of the sound field for arbitrary emitter and listener locations.
Implicit representations Our approach towards modeling the underlying acoustics a scene relies on the use of a neural implicit representations. Implicit representations have emerged as promising representation of 3D geometry [Niemeyer et al., 2019, Chen and Zhang, 2019, Park et al., 2019, Saito et al., 2019, Hong et al., 2022] and appearance [Sitzmann et al., 2019, Mildenhall et al., 2020, Niemeyer al., 2020, Yariv et al., 2020, Wang et al., 2021] I of scene. Compared to traditional discrete representations, implicit representations continuous mapping capable of capturing data

---

Local features
Emitter
Emitter feature
Implicit Decoder
Listener feature
Listener
Iggnnrrated ground truthll? Magnitude & Phase MSE Loss
Global features
Local feature grid Learned jointly with network
(x. Listener
(x'y)
0
k Left/Right
Emitter
Frequency
Time
Orientation
Overview of NAF architecture where listener and emitter share feature grid. Given listener position and an emitter location, we first query a grid local features which are learned together with network during training. We compute the sinusoidal embedding of the positions, frequency, and time, and query a discrete embedding matrix using the orientation and left/right ear. Our method predicts magnitude and phase.
Figure 2:
at an "infinite resolution". In [Jiang et al., 2020] proposed a grid based representation for implicit scenes, while more recently [DeVrree et al., 2021] has adopted spatial conditioning for 3D image synthesis, where in both settings, the grid enables a higher-fidelity encoding of the scene. Our work also leverages local grids model acoustics, but inductive bias and way generalize novel inputs.
Audio-Visual Learning Our work is also closely related to joint modeling of vision and audio. By leveraging the correspondence between vision and audio, work has been done to learn unsupervised video and audio representations [Aytar et al., 2016, Arandjelovic and Zisserman, 2017], localize objects that emit sound [Senocak et al., 2018, Zhao et al., 2018], and jointly use vision and audio for navigation [Chen al., 2020]. Recent work aims to propose plausible reverberations sounds from image input [Singh et al., 2021, Du et al., 2021], I, these approaches model the STFT using either convolution implicit functions, which also utilize. Different from them, work leverages the geometric features learned by modeling acoustic fields to improve the learning of 3D view generation.
3 Methods
We are interested in learning a generic acoustic representation an arbitrary scene, which can capture the underlying sound propagation of arbitrary sound sources across both seen and unseen locations in scene. We first review relevant background information towards modeling environment reverberations. We then describe Neural Acoustic Fields (NAFs), neural field which show can capture, in a generic manner, the acoustics of arbitrary scenes. We further discuss how we can parameterize NAF so that it capture acoustics property positions. Finally, we discuss the implementation details of our model illustrated in Figure 2.
unseen sound sources and listener
3.1 Background on the Propagation of Sound
The sound emitted by a sound source undergoes decay, occlusion, and scattering due both the geometric and material properties of a scene. For fixed location pair (q, q'), we define the impulse- response listener position g. as the sound pressure p(1; q, q') induced by an impulse q'.
Given an accurate model of the impulse-respone any sound waveform s(t) emitted at q', by computing the response r(t, q, q') at listener location by querying the continuous field and using temporal convolution:
p(t; q, q'), we may model audio reverberation of
r(t:q.g') s(t) + p(t:q.g')
(I)
3

---

Anechoic audio
Anechoic ① NAF impulse
(b)
(c)
(d)
(g) Anechoic GT impulse
Figure 3: Qualitative Visualization of Test Set Impulse Response Prediction. (a) Ground truth log- magnitude (b) Ground truth phase. (c) NAF predicted log-magnitude. (d) NAF predicted phase. (e) log-magnitude of an anechoic audio (without any reverberation). (d) The sound with reverberation impulse response from our NAF (e) The sound with the ground truth reverberation impulse response applied. Note phases are unwrapped for visualization purposes. Time & frequency are on the horizontal and vertical axes.
3.2 Neural Acoustic Fields
We are interested constructing a continuous representation of the underlying acoustics of scene, which may specify the reverberation patterns of an arbitrary sound source. The parameterization of an impulse-response introduced Section 3.1 provides us with a method to model audio propagation when given an omnidirectional listener and emitter. To construct model of a directional listener, we need to further model the 3D head orientation R2,, and ear k E {0, 1} (binary left or right) of a listener, in addition to the spatial position E RR of the listener and q' E IR3 of the emitter.
We may then model the time domain impulse response v using neural field which takes as input the listener and emitter parameters:
T R x 10, - TR. (q. O, k q¹) - $(g k 9') =
Directly outputting the impulse-responee waveform v IRT in the time domain with neural network is difficult due its high dimensional (over 10,000 elements) and chaotic nature. naive solution would be to further add / as additional argument to our neural field, but we found that such a solution worked poorly, due to the highly non-smooth representation of the waveform (see supplementary). We instead encode the impulse-response utilizing short-time Fourier transform (STFT) denoted USTFT, which we find to significantly more amenable to neural network prediction due to the smoother nature of the time- tmme-freueeccy space. In Figure 3 show magnitude spectrograms for ground truth impulse responses and those learned by our network. As USTFT is a complex value, we further factorize USTFT into log- rogmaaint and phase angle components. For phase angle, use the instantaneous frequency (IF) representation proposed in GANSynth [Engel et al., 2019) To compute the IF representation, the phase angle is unwrapped and has the finite difference taken across the time dimension for each frequency in the STFT. This transformation results in a phase representation that conducive learning due more regular structure.
Thus, our parameterization of NAF is neural field Ω that is trained estimate the impulse response function di and outputs [USTFT_mag: USTFT_IF) for given time and frequency coordinate:
Ω R x (0.1) -
(q,0, 0, k, q' f) llq. 0, ki q', (t, R (USIFT mag (1, f), USTFT (t. (t.
We train our model using MSE loss between the generated and ground truth spectrograms USIFT:
LNAP = [|2(q, 0, k, q' L, mm)a USTFT (STFTT_ngg( (1, 2)
a a| 0, k. q' t, f)r USIrr f)l
(4)
across spectrogram coordinates and f. Where a is scaling value used to balance the two losses.
3.3 Generalization through Local Geometric Conditioning
We are interested in parameterizing the underlying acoustic field, so that we may not only accurately represent impulse-response at emitter-listener pairs see during training, but also at novel combina-

---

(b)
Large
Large 2
Medium
Medium 2
Small
Small 2
Figure 4: Qualitative Visualization of Neural Acoustic Fields. (a) Top down view of the rooms. (b) Results inferred by our neural acooustic field. Loudness a sound given a emitter location indicated red, lighter color indicates louder sound. Note how openings and walls lead to portaling and occlusion the sound.
tions of emitter and listener seen at test time. Such generalization may be problematic when directly parameterizing NAFs utilizing a MLP with inputs specified in Equ (3), as the network may learn to directly overfit and entangle the relation between emitter and listener impulse-responses.
What generic information may we extract from a given impulse-repponss between an emitter and listener? In principle, extracting the full dense geometric information in a scene would enable us to robustly generalize to new emitter and listener locations. However, the amount of geometric information available in particular impulse-response, especially for positions far away from either current emitter and listener is limited, since these positions have little impact on the underlying impulse-resppe respoosse. In contrast, the local geometry near either emitter and listener positions will have a strong influence in the impulse-responee such geometry [Paasonen et al., 2017]. Inspired by this observation, aim to capture and utilize local geometric information, near either emitter or listener locations, as a means to predict impulse- responses across novel combinations.
as much of the anisotropic reflection comes from
To parameterize and represent these local geometric features, we learn a 2D grid of spatial latents which we illustrate in Figure 2. The spatial latents are randomly initialized and uniformly distributed the room. When predicting an impulse-response a given emitter and offset position, we query the learned grid features at both emitter and listener positions, and provide it as additional context into our NAF network Q. Such features provide rich information on the impulse-responss enabling NAF to generalize better to unseen combinations of both emitter and listener locations. In the rest of this work, we refer to the NAFs with local geometric features as S Qgrid. We learn grid latent features jointly with the underlying parameters of NAF. Additional details can be found in the supplementary.
Such design choice, however, still requires us consider how to further combine local geometric information captured separately from either listeners emitters. A naïve implementation would be to maintain separate feature grids for both listener and emitter positions. Such an approach fails to account for the fact that the local geometric information captured by emitter may also inform the local geometric information around listener. Examining Green's function, which is the solution to the wave equation, note that it is in fact symmetric with respect exchanging either listener emitter positions [Chaitanya et al., 2020], indicating that the impulse-responee does not change when omnidirectional emitters and listeners may in fact utilize the local geometric information captured near an emitter position interchangeably for either emitters and listeners. Thus, we propose local geometric information single latent grid, which we show to outperform the naïve dual grid implementation.
swapped (acoustic reciprocity). Such result means that
4 Experiments
In this section, we demonstrate that our model can faithfully represent the acoustic impulse response seen and unseen locations. Additional ablation studies verify the importance of utilizing local geometric features to enable test time generation fidelity. Next, we demonstrate that learning acoustic fields could facilitate improved visual representations when training images are sparse. Finally we show that the learned NAF can be used to infer scene structure.
5

---

GT-mxmest
AAC-linear
Opan-nearest
Open
Figure 5: Comparison of the acoustic fields. From left to right, visualize the loudness maps generated by the full ground truth, NAFs, and by AAC or Opus coding combined with lincar and nearest neighbor interpolation on the training set. Emitter location shown red. Our method can faithfully reproduce the loudness map present the ground truth.
4.1
Setup
For each scene, we holdout 10% of the RIRs randomly as a test set. Each scene is trained for 200 epochs, which takes around hours for the largest scenes on four Nvidia V100s. In each batch, we sample 20 impulse responses, and randomly select 2, 000 frequency & time pairs within each spectrogram. An initial learning rate of 5 10-4 is used for the network and the grid features. We add small amount of noise sampled from N (0, 0.1) to each coordinate during training to prevent degenerate solutions. For evaluating the learned acoustic fields, we use two different datasets:
Soundspaces.
Soundspaces [Chen et al., 2020, Straub et al., 2019] is synthetic dataset generated via ray-tracing, and is auralized with ambisonic head-related transfer function (HRTF). This dataset consists of R3 probe points for each scene, with each probe capable representing an emitter or listener location for up to R₂ emitter and listener pairs. The emitters are represented as omnidirectional, while the listener acts as a stereo receiver that can have one four different orientations. The listeners and emitters are at fixed height. Our NAFs are trained on 6 representative scenes, where consist of multi-room layouts; 2 consist single room with a non-rectangular walls; and 2 consist of single room with rectangular walls as in Figure 4.
The MeshRIR dataset [Koyama al., 2021] is recorded from real scene, and contains monaural data collected from a cuboidal room. The listener locations are at fixed height. The emitters surround the listeners both above and below the listener plane.
MeshRIR.
4.2 Architecture Details
The Soundspaces dataset lacks the full parameterization of acoustic field described in Equation so we train NAF with restricted parameterization that is available in the dataset. This allows for two degrees of freedom along the location of E R.2 The listener binaural with & E {0, 1), and can assume four possible orientations 0 E {0,90, 180, 270), while the emitter is omnidirectional. In particular, we utilize parameterization of Qegrid which maps an input tuple (ar y. x', y' J x (0, 90, 180, 270} x (0, 1} to two scalar values that represents the magnitude and phase for a given time and frequency in the STFT:
plane for the listener locations q E I and the emitter
Agrid (x, y, K, K'
[USTFT_mag(t. f), USTFT USTFT_IF(t, f)]
(5)
To encode the rotation f, as there are only 4 possible discrete rotations in the dataset, we directly query into learnable embedding matrix of shape 2x returning R vector. Similarily. to encode the left and right ear k we similarly query into learnable embedding matrix of shape R2xn returning 1 vector. The f, tuple representing the frequency and time respectively are scaled to 1- and processed with sinusoidal encoding using 10 frequencies of sin and cos. For MeshRIR, we set emitter g' E 3 to account for the emitters that can vary height, and do not utilize the orientation binaural embedding.
To obtain local geometric features for either an emitter or listener in scene, we assume that our scene is contained within set of pixels P = {PI..P% which form grid over the scene. For given position tuple (x. as query location, we then interpolate the local features. Where Z(:) the interpolation function. (pī ......p. are the set pixel that form the grid, and f(-) represents the features stored at a given pixel:

---

(x. = [(e, vi f(pi))
(6)
Σ wo fv((
(7)
Wy; is determined by a Nadaraya- Watson estimator with a Gaussian weighting kernel applied to the distance between query and grid coordinates:
k
= ((1x, y). (x., wcll// Σ
K ( v), (Ty, vs))
(8)
K(x, x²) = exp( 1ep(-11a 1| 2'11//20
Because this interpolation function is differentiable, we jointly learn the grid features during training. These queried features are combined with the coordinates processed with sinusoidal encoding using 10 frequencies of sin and cos functions. We process both the listener and emitter position tuples this way. We combine the grid based features with the sinusoidal embeddings and the discrete indexed embeddings as the input to our multilayer perceptron fa. Please refer to Figure 2 for visualization of our model, and supplementary for further details. We compare using shared local geometric feature with the emitter and listener, as well as having the emitter and listener query their own individual grids.
4.3 Evaluation of Neural Acoustic Fields
We first validate that we can capture environmental acoustics at unseen smitter-listenee positions.
Baselines. We compare our model against two widely used high
performance audio coding methods: Advanced Audio Coding (AAC) and Xiph Opus. We use low bitrates in order to attempt to approach the storage costs for our NAF. For each method, we apply both linear and nearest neighbor interpolation to the coded acoustic fields. Both linear and nearest neighbor approaches are widely used [Savioja et al., 1999, Raghuvanshi et al., 2010, Pörschmann al., 2020] in modeling of spatial audio. We further implement the binaural DSP baseline described by [Richard et al.. 2020], which uses the image source method and a KEMAR HRTF. We also compare the listener and emitter either sharing using individual local geometric features in our NAFs.
Method
Storage (MiB)
AAC
312.07
Opus
163.23
NAF (Share)))
8.41
Table 1: Average space consump-
tion across 6 Soundspaces scenes. Lower is better.
Each method is provided with the same train-test split. We visualize the acoustic fields produced by different methods in Figure Details of our baselines can be found in the supplementary section G.
Metrics. We evaluate the results of our synthesis by measuring the spectral loss [Défossez et al., 2018] between the generated and the ground truth log-specto,, as well as measuring the percentage error between the T60 reverberation time in the time domain. In this case, lower spectral loss and T60-error values indicates better result. Additional quantitative results can be found in the supplementary. We also perform a human evaluation where subjects are presented with a two-alternative forced-choice task. Each trial requires selecting if the NAFs the AAC-nearest auralized music samples best match with ground truth auralization.
Results. As shown in Table 2, our NAFs achieve significantly higher quality on the modeling of unseen impulse responses compared to strong interpolation baselines across all six scenes. Observing the qualitative results in Figure we observe that NAFs can predict smoothly varying acoustic fields that are affected by the physical surroundings. Extending our model to MeshRIR which is captured from real scene, we observe that our NAFs continue perform better on both spectral and T60 metrics. Comparing our results against baselines in Figure 5 and Table 1, our methods are able to better approximate the ground truth at a fraction of the storage cost, and does not exhibit the sound energy leakage present in linear interpolation. The size of a spatial acoustic field is important for real life applications. In our human evaluation with 21 subjects who were asked to judge 10 test-time RIRs, 82.38% of responses indicates that our NAFs were higher quality compared to AcC-Ncarest.
7

---

Large
Large 2
Medium
Medium 2
Small 1
Small 2
MeshRIR
Mean
Model
Spec.
T60
Spec.
T60,
Spec.
T60
Spec.
T60
Spec.
T60L
Spec.
T60L
Spec. T60l,
Spec.
To01
ACC-mearst
1.913
9.996 1,989
13.31 2,111
6.148
2.122
6.051 2.296
9.798 2.509
5.809 1.057
4.740
1.999
7.979
AAC-linear
1.904
8.847 1,964
11.63
2.105
4.585
2.116
4.422 2.299
8.253 2.521
6.021
1.081
6.697
1.998
7.208
Opus-nearest
1.740
12:20
1.817
15.15
1.887
7.875
1,898
7.897
2.058
10.68
2.238
7.564
1.711
5,068
1.907
9.493
Opus-linear
1.780
11.30
1.827
13.55
1.922
6,710
1.934
6,917 2.097
9.116 2.284
6,981
1.743 5,768
1.941
8.621
DSP
1.106
14.62
1.170
13.68
1.064
10.24
1.067
9.732
1.079
12.77
1.097
11.03 N/A
N/A
1.097
12.01
NAF (Dull)
0.413
6,288
0.421
7.111
0,386
3.173
0,387
3.169
0.365
3.497
0.361
2.210
0.403
4,201
0.388
4.241
0.396
4.166
0.413
6.075 0,384
3.110 0,384
3.072 0.356
3.378 0,344 2.098 0.403 4.191 0.380 3.650
Table 2:
Quantitative Results on Test Set Accuracy. We report the spectral loss hetween generated and ground truth log spectrograms across methods, as well as the percentage (%) difference for the T60 reverheration time. The best method for each room is bolded. For the nearest and linear haselines, we perform interpolation in the time domain using samples from the training set. The DSP is not implemented for MeshRIR due to the lack of absolute room coordinates.
Large Room
Large Room 2
PSNR 1
SSIM f
PSNR 1
SSIM
Training Images
75
100
150
75
100
150
75
100
150
75
100
150
NeRF NeRF NAF
25.41
27.36
29.85
0.872
0.892
0.926
25.70
27.74
29.34
0.821
0.853
0.879
26.19
27.59
29.90
0.895
0.911
0.927
26.24
28.22
29.45
0.837
0.866
0.879
Table 3:
Quantitative Results Cross-Modal Image Learning. Quantitative results joint training of NeRF and NAF jointly conditioned on single local grid. We very sparse training images in highly complex scenes. When evaluated on 50 test images, we observe that cross-modal learning helps improve PSNR and SSIM when the visual training data is more sparse.
comparison of using shared and dual local geometric features indicates that despite having fewer learnable parameters, we achieve better performance by sharing the local geometric features. Exam- ples of individual impulse responses generated by our model are shown Figure 3.
Generalization through Geometric Conditioning. We next assess the impact of utiliz- ing local geometric conditioning as means to generalize to novel combinations of emitter- listener positions. On the "Large l" room, in Figure 7 we evaluate test set spectral er- ror when NAF is trained with a limited percentage of the training data either with or without local geometric conditioning. We find that such geometric conditioning enables better test set reconstruction error, with the performance gap increasing with less data.
Lange
Lapp22
so0 Accanco
0,50
w/o geomatric conditioning w/ local genmetric conditioring
Spectral on tast
0.48
0.44
50% 70% 50%
Figure 6: Qualitative Visualization of Cross-Modal Image Learning. Qualitative comparison hetween NeRF+NAFs with RGB and acoustic supervision, and NeRF learned with only RGB supervision. (a)-(c) Three views from "Large 1". (d)-(f) Three views from "Large2""
Percentage fortraining
Figure 7: Local Geometric Conditioning. Comparison of NAF with and without local geometric conditioning trained with different amounts of data.
4.4 Cross-modal learning
In this experiment, we explore the effect of jointly learning acoustics and visual information when we are given sparse visual information. Recall that our NAF includes a local geometric feature grid P that covers the entire scene. For our cross-modal learning experiment, we jointly learn this feature grid with a NeRF network modified to accept local features sampled from this grid along with the traditional sinusoidal embedding. In the acoustics branch, we query the grid using emitter and listener positions. In the NeRF branch, we use point samples along the ray projected on the grid plane to query the features. In both cases, the process is fully differentiable. We use a standard implementation of NeRF with coarse and fine network.
8

---

La Largo
Largo 2
Structure
MFCC lincar
NAF latent
linear
Feature TSNE
Figure 8: Visualization of scene structure decoding with linear layer. Column 1: The ground
truth scene structure map, each position we visualize the distance the nearest wall. 2: Linear
decoding results using MFCC features. 3: Linear decoding results using NAF features. 4: TSNE applied to the NAF features.
In the NeRF only setting, minimize color C' reconstruction loss for a ray r batch of rays R: CRGB ErER llĆ(r) C(r)l In contrast, in the NAF NeRF experiment, jointly minimize LRGB + LNAF, where NAA defined in equation We utilize 64 coarse samples and 128 fine samples for each ray, and sample 1024 rays per batch.
Results. We train on the two large rooms our training set. For each room 75, 100, 150 images are used for training, while the same 50 images of novel views used for all configurations during testing. In Table 3 we observe that training with acoustic information helps improve the PSNR and SSIM of the visual output. This effect is more significant when the training images very sparse, the NAF network helps less when there is sufficient visual information. Qualitative results are shown in Figure 6, we there is reduction of floaters in free space.
4.5 Inferring scene structure
Given reverberant sound, humans able to build mental representation of the surrounding room and make a judgement about the distance of nearby obstacles Kolarik et al. [2016]. We investigate the intermediate representations constructed by our neural network in the process of learning acoustic field, and examine if these representations can be used to decode the scene structure.
Setup. The intermediate representation of the NAF depends on both listener locations q and emitter location q', the rotation angle 0, the car k, the time and frequency f. For consistency, at a given location (x*, y²) ) in the scene, extract the NAF latent by setting the emitter location of (x* vy") For the listener location, we iterate over five randomly selected points in the scene E [qa,..., [.]. which keep constant for all g.- The rotation angle is fixed O 0, and compute the representation average over all possible (k, t, f), and concatenate latents for the selected For our NAFs, latents extracted from the last layer prior the output which includes 512 neurons. As
Explained variance
Features
Large
Large
Medium
Medium
Small
Small 2 Mean
MFCC
0.501
0.458
0.614
0.642
0.820
0.723
0.626
NAF latents
0.908
0.891
0.900
0.923
0.936
0.916
0.913
Table 4: Quantitative Results scene structure decoding. We measure the explained variance scores of the predicted wall distance against the ground truth wall distance at test time locations after linear decoding. NAF latents consistently achieve higher explained variance than MFCC features.
comparison to our learned representation, we extract Mel-frequency cepstral coefficients (MFCCs) from the ground truth impulse response provided by a nearest neighbor interpolator. We use a similar setup as above, for given location we set this to be g/ and iterate over the same five listener locations

---

91...3. We average the MFCCs over the left and right ear, and concatenate for the selected After flattening, the MFCC features are approximately 500 dimensional for any given room.
We fit single linear layer to NAF and MFCC features respectively. For testing and visualization of the linear decoding results, we sample a regular grid of points with 0.1m distance between each point. For fitting the linear decoder, we randomly sample points within the scene such that the number of training points are 10% as many as the testing points. For each location in the scene, we extract the distance to the nearest wall as the decoding target.
Results. We visualize the results of our linear decoding in Figure 8. The intermediate representation of our NAFs reveals an underlying structure that is both smooth and semantically meaningful. In the multiroom scenes, the latent is well separated for each room. We are able to successfully decode the scene structure with a linear layer when using our NAFs, but decoding fails when using MFCC features. In Table 4, show the amount of explained variance of our decoding results the test set. Our learned features are able to consistently achieve much higher scores than those using MFCC features.
5 Discussion
Limitations and Future Work. Although our method achieves generalization and high quality representations of acoustic fields within single scene, NAFs do not currently generalize to multiple scenes. Future work may explore generalization novel scenes. One possible approaches may be to incorporate multi-modal inputs with the goal of synthesizing an acoustic field with few-shot visual or acoustic input.
Societal impact. Our work focuses learning high quality representation of acoustic fields. The primary use case for our work lies in virtual reality and gaming. As our work can lead to more believable and higher quality representations of spatial audio than alternative methods, it is possible that our work could increase the dependency and time spent on gaming. The more compact nature of our acoustic representations may allow for spatial audio to be deployed more systems, and enable more equitable access.
Conclusion. In summary, this paper introduces Neural Acoustic Fields (NAFs), a compact, continu- ous, and differentiable acoustic representation which can represent the underlying reverberation of different audio sources in scene. By conditioning NAFs locally on the underlying scene geometry, demonstrate that approach enables the prediction of plausible environmental reverberations even at unseen locations in the scene. Furthermore, we demonstrate that the acoustic representa- tions learned through NAFs are powerful, and may be utilized to facilitate audio-visus cross modal learning, as well as to infer the structure of scenes.
Acknowledgments and Disclosure of Funding
We would like to thank Leila Wehbe for providing feedback on our paper. This work was supported by the MIT-IBM Watson AI Lab, Amazon Research Award, ONR MURI, DARPA Machine Common Sense program, ONR Tianqiao and Chrissy Chen Institute.
NIDCD ROI DC015988, Mitsubishi Electric, and the
10

---

References
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Contin- uous 3d-structure-aware neural scene representations. In Proc. NeurIPS 2019, 2019.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Proc. ECCV, 2020.
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In Proc. CVPR, 2020.
Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. Proc. NeurlPS, 2020.
Nikunj Raghuvanshi and John Snyder. Parametric wave field coding for precomputed sound propaga- tion. ACM Transactions on Graphics (TOG), 33(4):1-11, 2014.
Nikunj Raghuvanshi and John Snyder. Parametric directional coding for precomputed sound propa- gation, ACM Transactions on Graphics (TOG), 37(4):1-14, 2018.
Chakravarty R Alla Chaitanya, Nikunj Raghuvanshi, Keith W Godin, Zechen Zhang, Derek Nowrouzezahrai, and John M Snyder. Directional sources and listeners in interactive sound propagation using reciprocal wave field coding. ACM Transactions on Graphics (TOG), 39(4):
44-1, 2020.
Pratul P. Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T. Barron. Nerv: Neural reflectance and visibility fields for relighting and view synthesis. In Proc. CVPR, 2021.
Rémi Mignot, Gilles Chardon, and Laurent Daudet. Low frequency interpolation of room impulse responses using compressed sensing. IEEE/ACM Transactions Audio, Speech, and Language Processing, 22(1):205-216 2013.
Niccolo Antonello, Enzo De Sena, Marc Moonen, Patrick A Naylor, and Toon Van Waterschoot. Room impulse response interpolation using sparse spatio-temporal representation of the sound field. JEEF/ACM Transactions on Audio, Speech, and Language Processing, 25(10):192911441, 2017.
Natsuki Ueno, Shoichi Koyama, and Hiroshi Saruwatari. Kernel ridge regression with constraint of helmholtz equation for sound field interpolation. In 2018 16th International Workshop on Acoustic Signal Enhancement (/WAENC). pages 1-440. IEEE, 2018
Ravish Mehra, Lakulish Antani, Sujeong Kim, and Dinesh Manocha. Source and listener directivity for interactive wave-based sound propagation. IEEE transactions on visualization and computer graphics, 20(4):495-500 2014.
Anton Ratnarajah, Shi-Xiong Zhang, Meng Yu. Zhenyu Tang, Dinesh Manocha, and Dong Yu. Fast-rir: Fast neural diffuse room impulse response generator. arXiv preprint arXiv:21 10.04057, 2021.
Alexander Richard, Dejan Markovic. Israel D Gebru, Steven Krenn, Gladstone Alexander Butler, Fernando Torre, and Yaser Sheikh. Neural synthesis of binaural speech from mono audio. In International Conference Learning Representations, 2020.
Alexander Richard, Peter Dodds, and Vamsi Krishna Ithapu. Deep impulse responses: Estimating and parameterizing filters with deep networks. In ICASSP 2022-2022 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3209-3213 IEEE, 2022.
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Occupancy flow: 4d reconstruction by learning particle dynamics. In Proc. ICCV, 2019.
Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. Proc. CVPR, pages 5939-5948, 2019.

---

Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proc. CVPR, 2019.
Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In Proc. ICCV, pages 2304-2314, 2019.
Yining Hong. Yilun Du, Chunru Lin, Joshua B Tenenbaum, and Chuang Gan. 3d concept grounding neural fields. arXiv preprint wrXiv:220.0064000 2022.
Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi-view image-based rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4690-4699, 2021.
Chiyu Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießner, and Thomas Funkhouser. Local implicit grid representations for 3d scenes, In Proc. CVPR, pages 6001-6010, 2020.
Terrance DeVries, Miguel Angel Bautista, Nitish Srivastava, Graham W Taylor. and Joshua M Susskind. Unconstrained scene generation with locally conditioned radiance fields. arXiv preprint arXiv:2/04.00670, 2021.
Yusuf Aytar, Carl Vondrick, and Antonio Torralba. Soundnet: Learning sound representations from unlabeled video. Advances neural information processing systems, 29:892-900, 2016.
Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In Proceedings of the IEEE International Conference Computer Vision, pages 609-617, 2017.
Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang, and In So Kweon. Learning localize sound source in visual scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4358 43664366, 2018.
Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. The sound of pixels. In Proceedings of the European conference on computer vision (ECCV), pages 570-586, 2018.
Changan Chen, Unnat Jain, Carl Schissler, Sebastia Gari, Ziad Al-Halah, Vamsi Krishna Ithapu, Philip Robinson, and Kristen Grauman. Soundspaces: audio-visual navigation in 3d environments. In ECCV, 2020.
Nikhil Singh, Jeff Mentch, Jerry Ng, Matthew Beveridge, and Iddo Drori. Image2reverb: Cross-model reverb impulse response synthesis. In IEEE/CVF International Conference on Computer Vision (ICCV), October 2021.
Yilun Du, M. Katherine Collins, B. Joshua Tenenbaum, and Vincent Sitzmann. Learning signal- agnostic manifolds of neural fields. In Advances in Neural Information Processing Systems, 2021.
Jesse Engel. Kumar Krishna Agrawal, Shuo Chen, Ishaan Gulrajani, Chris Donahue, and Adam Roberts. Gansynth: Adversarial neural audio synthesis. arXiv preprint 100.0:1902.087700 2019.
Juhani Paasonen, Aleksandr Karapetyan, Jan Plogsties, and Ville Pulkki. Proximity of sur- accustic and perceptual effects. Journal the Audio Engineering Society, 65(12):997-1004 2017.
Julian Straub, Thomas Whelan, Lingmi Ma. Yufan Chen, Erik Wijmans, Simon Green, Jakob J Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, et al. The replica dataset: A digital replica of indoor spaces. arXiv preprint arXiv: htt:::990..05999 2019.
Shoichi Koyama, Tomoya Nishida, Keisuke Kimura, Takumi Abe, Natsuki Ueno, and Jesper Brunnström. Meshrir: A dataset of room impulse responses on meshed grid points for eval- uating sound field analysis and synthesis methods. In 202/ IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pages 1-5. IEEE, 2021.
12

---

Lauri Savioja, Jyri Huopaniemi, Tapio Lokki, and Ritta Väänänen. Creating interactive virtual acoustic environments. Journal of the Audio Engineering Sociery, 47(9):675-705, 1999.
Nikunj Raghuvanshi, John Snyder, Ravish Mehra, Ming Lin, and Naga Govindaraju. Precomputed wave simulation for real-time sound propagation of dynamic sources in complex scenes. In ACM SIGGRAPH 2010 papers. pages 1-11. 2010.
Christoph Pörschmann, Johannes M Arend, David Bau, and Tim Lübeck. Comparison of spherical harmonics and nearest-neighbor based interpolation of head-related transfer functions. In Audio Engineering Society Conference: 2020 AES International Conference on Audio for Virtual and Augmented Reality. Audio Engineering Society, 2020.
Alexandre Défossez, Neil Zeghidour, Nicolas Usunier, Léon Bottou, and Francis Bach. Sing: Hembol-to-instruman neural generator. arXiv preprint arXiv:18100099755, 2018.
Andrew J Kolarik, Brian CJ Moore, Pavel Zahorik, Silvia Cirstea, and Shahina Pardhan. Auditory distance perception humans: a review of cues, development, neuronal bases, and effects of sensory loss. Attention, Perception, & Psychophysics, 78(2):373-395 2016.
13


Reviews:
[{"review_id": "n6GqNiQkgOO", "reviewer": "Reviewer_ossn", "summary": "This paper proposed a model for learning to interpolate room impulse responses given room geometry and emitter/listening position.\nExperiments on two datasets demonstrate that the proposed model can encode room geometry information, and that the learned representation can assist in cross-modal learning.", "questions": "Minor comment: Figure 3 shows spectrograms both upside-down (e-g) and sideways (a-d) and without axis labels.  Please, for the sake of consistency, stick to standard orientations for this kind of information.  If you're cramped for space, I suggest to simplify to a monophonic visualization as there's no information content in the left/right channels that's important for this particular figure.", "limitations": "There is no mention of societal impact.  The checklist response claims this will be included in a supplementary document.\n\nTechnical limitations are addressed.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper is clear and well-written.  The method appears to work well.  \n\n\nWeaknesses:\n\n\nThe quantitative evaluations (eg Table 1) look good, but there is no meaningful qualitative evaluation of the method's ability to interpolate RIRs.\nThe raw numbers are difficult to interpret, even for knowledgable practitioners, let alone the broader machine learning audience.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "eAfSj92IbMc", "reviewer": "Reviewer_u3Tz", "summary": "Disclosure: I reviewed a previous version of this paper submitted to another conference earlier this year. \n\nThis paper introduces the concept of Neural Acoustic Fields, inspired by the recent developments in Neural Radiance Fields (NeRF) in the context of visual scene generations from a few still images. The idea is, for a given complex acoustic environment, to learn using a neural network a function that maps a pair of 3D (emitter, receiver) locations, a receiver orientation, a receiver channel (left or right) and a time-frequency point (t,f) to the corresponding log-magnitude and phase spectrogram of the room impulse response (RIR) from that emitter to that receiver at that (t,f) point. The function is learned using a balanced least square loss in the log-magnitude and phase domains and positions are represented by a sinusoidal embedding as well as learned geometrical feature maps.\n\n", "questions": "I do not have specific questions except those arising from the lack of details provided in the paper (see below).", "limitations": "## Phase and binaural rendering\nThe main limitation of the paper is the evaluation. Indeed, the only two objective metrics used to evaluate generate RIRs are insufficient to validate the central claim made by the authors in the abstract, namely: \"continuous nature of NAFs enables us to *render spatial acoustics* for a listener at arbitrary locations\".\n\nThe first metric is the RT60 error, which is a single scalar parameters that globally captures the reverberation time but ignores fine properties of the reverberation which typically vary across frequency. The second metric used is the mean squared error on the log-magnitude spectrogram of the RIR. This metric completely ignores the phase reconstruction. Though, correctly reconstructing the phase is known to be of crucial importance for rendering reverberation, and even more so for rendering spatial effects, in binaural settings. Indeed, the primary cue for sound source localization are phase differences across frequencies, which need to be very accurately rendered. The authors claim that their model is able to render multiple orientations of a binaural listener. However, the main notable effect of rotating a listener on the spot for a fixed source in a fixed environment will be on the phase differences between the two channels. The authors to propose to predict the phase via an instantaneous frequency representation, but the reconstructed phases and phase differences are never evaluation.\n\nIn short, the evaluation is currently insufficient to support the implicit strong claim that the proposed method is able to \"render spatial acoustics\" in the binaural settings. They merely convince that the method is able to accurately render *monaural* reverberation, which is a significantly easier (though highly non-trivial) task. This strong limitation is not properly addressed anywhere in the manuscript.\n\n## Clarity and precision\nThe writing is unprecise and handwavy at a number of places. It gives the general impression that the main background of the authors is in NeRF rather than in acoustics/audio signal processing.\n- L44: \"an intractable number of rays are necessary\"  The term \"intractable\" is not precise enough here. Ray tracing is a widely used technique for acoustic simulation, for instance it was used to generate the SoundScape dataset [25] used by the authors themselves for training their method. In that paper, 200 rays are emitted from each source location. Is that considered an intractable number?\n- L47 \"NAFs encode and represent an impulse-response in the Fourier frequency domain.\" -> actually, in the time-frequency domain, which is quite different.\n- L60: \"The first approach encodes the sound field at a user-centric location by capturing the sound from spatially distributed sources [8,9,10,11]\". It is not clear what the authors mean here by \"capturing the sound from spatially distributed sources\": capturing in what sense? How do those methods specifically differ from the proposed approach? Curiously, the 4 references given seem completely unrelated to each other and span 50 years of literature. The most recent one [11], from 2021, actually seems highly related to what the authors propose, and some more in-depth comparison with this work should be made in the paper.\n- L113 \"Directly outputting the impulse-response v magnitude\" -> this seems incorrect, the term \"magnitude\" should probably be removed.\n- L119 \"due to the smooth nature of the frequency space\" -> the authors probably mean \"time-frequency space\".\n- L122 \"For phase, we use the instantaneous frequency (IF) representation proposed in [29].\" : more details on the phase representation used should be provided to make the paper self-contained / reproducible. In particular, one cannot directly reconstruct the phase from the instantaneous frequency, since it is by definition the time derivative of the phase. Hence, a global phase is missing at each frequency. It is not clear how this is handled by the authors.\n- L147-150 on local geometric features: It is hard to understand what the authors are talking about here without reading in details the supplementary material. Would be good to make this part more self-explanatory.\n- The authors compare an approach using a \"shared\" feature grid vs. a \"dual\" feature grid, the former outperforming the latter. However, in the supplementary material (fig. A4), only the network architecture for a dual grid is presented, which is confusing.\n- Section 4.1: when presenting the two datasets, the authors omit to discuss their limitations, although both are in fact quite limited and do not allow testing the proposed approach in its full generality. The first one only employs 2D grids while the second one is monaural.\n- L217: \"The bitrates are chosen to approach the storage costs for our NAF\": This is not precise enough to understand how the baselines were implemented exactly. The authors should report results as a function of bitrates for the different methods to get a clearer picture of the benefit of the approach compared to standard techniques. Moreover, Table 3 seems to contradict this statement: here we see that the space consumption of NAF is vastly inferior to that of AAC and Opus.\n\n## Typos\n- L44: \"an intractable number of rays are necessary to\" -> is necessary.\n- L179 \"on 6 representative scenes. Where\" -> remove full stop\n- L212 \"as well as using have\" -> incorrect phrasing.\n", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n- The proposed approach is sound both in terms of the underlying acoustical model and in terms of the neural network design and training process, and the general idea of NAFs is novel, interesting, and potentially impactful.\n- The ability of the method to correctly estimate log-magnitude time-frequency responses and reverberation times at any point in different complex environments across a synthetic and a real dataset is convincingly demonstrated.\n- A number of interesting experiments reveal that the proposed model learns geometrical structure that could be used for other purposes.\n\n## Weakness\n- An important weakness of the paper is the evaluation (See \"limitations\" for more details).\n- The paper lacks clarity as well as many important details to make it self-contained and reproducible (See \"limitations\" for more details).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "bI9r6wVyHA", "reviewer": "Reviewer_e4mi", "summary": "This paper describes a NeRF-like representation for binaural room impulse responses in a given acoustic space. It is an implicit representation of listener position, orientation, and ear, as well as source position, time, and frequency that outputs a scalar magnitude and phase. Experiments on the Soundspaces and MeshRIR datasets show that it is able to reproduce realistic effects in loudness maps like sonic shadows from doorways for irregularly-shaped rooms that would otherwise require expensive ray tracing for impulse response simulation. The model's predictions are compared to those of Opus and AAC and shown to be both smaller in space as well as more accurate in terms of T60 estimation and log spectrogram MSE. An ablation shows that a geometric encoding layer that is shared between listener and emitter works better than having separate or no encoding layer. Use of the proposed representation with NeRF for multi-modal representation improves the visual representation when training images are sparse. A further experiment shows that the distance to the nearest wall can be predicted more accurately from the learned representation than from MFCCs by a linear layer.", "questions": "What is the architecture of the actual model that is being fit? What are its hyperparameters?\n\nAre the AAC and Opus baselines encoding just the impulse responses? Since they are designed to encode natural sounds and not impulse responses, does encoding a sound convolved with the impulse responses give more accurate results?\n\nIn listening to the videos in the supplementary material, I didn't get a great sense of direction of arrival of the sound source. Why might this be? Can this be addressed?", "limitations": "The limitations section captures salient limitations of the approach and experiments.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* Novelty and timeliness: NeRF is very popular and while several papers have conditioned a visual NeRF model on audio of various kinds, this is the first, to my knowledge, that provides a meaningful application of the same idea of implicit scene encoding in an audio context.\n* High quality experiments: The experiments are well conducted and thorough. Useful ablations were conducted which provide non-trivial insights, i.e., that a shared geometric encoding works better than having separate encodings for source and listener.\n* Clarity: The paper is very well written and easy to follow\n* Supplementary material: The supplementary material is useful and informative without being overwhelming.\n\nWeaknesses:\n* A few missing details: the paper makes very efficient use of its space, but seems to leave out a few crucial details: the actual architecture of the model being trained, how exactly AAC and Opus were used to encode these libraries of impulse responses, and how the loudness maps are calculated.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8H_T55iVf8", "reviewer": "Reviewer_jrJA", "summary": "Modeling room and scene acoustics is vital for AR and VR problems. Realistic rendering of spatialized audio is important. This paper builds a representation learning model/schema for parametrizing such acoustic responses via a encoder-decoder style deep network interpolation machine. This is a neat idea. The paper is written well and the benefits of the proposal are explored. ", "questions": "Apart from addressing the aspects mentioned in Weaknesses, some other points to expand are: \n1. Talk a little more clearly about the decoder setup, and how does this architecture relate to recent advances in NNs for signal parameterization of impulse responses (like https://ieeexplore.ieee.org/abstract/document/9746135)\n2. The resolution level of T60 is rather coarse to clearly articulate the benefits of the parameterization in the paper. Can we look at the IRs themselves and capture differences in DRR or early reflections? DRR in particular should show clear differences between this proposal and signal processing baselines. ", "limitations": "Yes. This was addressed. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. Clearly motivated for NN modeling. Well written and presented. \n2. Reasonable to expect the benefits of NeRF to extend to acoustic fields. \n\nMain weakness: \nRoom and Scene acoustics is a well studied area in Audio. In particular, interpolation of room responses with and without geometry modeling including accounting for diffraction and dispersion patterns around object boundaries is being actively studied in signal processing and acoustics. Although the authors do build the related works, there are still relevant baselines that are missing (both from discussions and from evaluations perspective) like https://ieeexplore.ieee.org/document/7987742 and others. This is not to say that the evaluations are invalid, but more to understand what the network is really learning compared to classical approaches. \nAdditionally a bulk of the content in Section 3 introing the acoustic functions can be reduced and cited to appropriate references. Saving much real-estate. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept