Learning Neural Acoustic Fields
Authors: Andrew Luo
Yilun Du
Michael J. Tarr
Joshua B. Tenenbaum
Antonio Torralba
Chuang Gan
Venue: ICLR 2022 Submitted (2022)
Keywords: Audio-Visual Learning
Acoustic

Abstract: Our sensory perception of the world is rich and multimodal. When we walk into a cathedral, acoustics as much as appearance inform us of the sanctuary's wide open space. Similarly, when we drop a wineglass, the sound immediately informs us as to whether it has shattered or not. In this vein, while recent advances in learned implicit functions have led to increasingly higher quality representations of the visual world, there have not been commensurate advances in learning auditory representations. To address this gap, we introduce Neural Acoustic Fields (NAFs), an implicit representation that captures how sounds propagate in a physical scene. By modeling the acoustic properties of the scene as a linear time-invariant system, NAFs continuously map all emitter and listener location pairs to an impulse response function that can then be applied to new sounds. We demonstrate that NAFs capture environment reverberations of a scene with high fidelity and can predict sound propagation for novel locations. Leveraging the scene structure learned by NAFs, we also demonstrate improved cross-modal generation of novel views of the scene given sparse visual views. Finally, the continuous nature of NAFs enables potential downstream applications such as sound source localization.

Under review as a conference paper at ICLR 2022
LEARNING NEURAL ACOUSTIC FIELDS
Anonymous authors Paper under double-blind review
ABSTRACT
Our sensory perception of the world is rich and multimodal. When we walk into cathedral, acoustics much appearance inform of the sanctuary's is wide open space. Similarly, when we drop a wineglass, the sound immediately informs to whether it has shattered not. In this vein, while recent advances in learned implicit functions have led to increasingly higher quality representations of the visual world, there have not been commensurate advances in learning auditory representations. To address this gap, we introduce Neural Acoustic Fields (NAFs), an implicit representation that captures how sounds propagate in a physical scene. By modeling the acoustic properties of the scene as a linear time-invariant system, NAFs continuously map all emitter and listener location pairs to an impulse response function that can then be applied to new sounds. We demonstrate that NAFs capture environment reverberations of a scene with high fidelity and can predict sound propagation for novel locations. Leveraging the scene structure learned by NAFs, we also demonstrate improved cross-modal generation of novel views of the scene given sparse visual views. Finally, the continuous nature of NAFs enables potential downstream applications such as sound source localization. Qualitative results: sites. google le. com/v vi ew naafaaafss fs rell--rr2222/
INTRODUCTION
The sound of the ball leaving the bat, as much as its visible trajectory, tells us whether the hit is likely to be home run or not. Our experience of the world around is rich and multimodal, depending on integrated input from all of five sensory modalities vision, touch, smell, hearing, and taste. Understanding underlying scene structure relies on a continuous and diverse set of inputs: the visual appearance and geometry of the physical world, the way sounds bound off of are blocked by surfaces, the way odors diffuse across the local environment, and the way objects feel as we walk on them or touch them.
Recent progress in implicit neural representations has enabled the construction of continuous, differ- entiable representations of the visual world directly from raw image observations (Sitzmann et al., 2019; Mildenhall et al., 2020; Niemeyer et al., 2020; Yariv et al., 2020). These models typically utilize a neural renderer in combination with learned implicit representation to jointly capture and render images of a scene. By leveraging the multiview consistent nature of visual observations, these methods can infer images of the same scene from novel viewpoints once trained on sparse views.
However, our perception of the physical world is informed not only by our visual observations, but also by the acoustics of environmental sounds. When we in a large room, we expect hear highly reverberant echos (e.g., as in cathedral). Moreover, given only the acoustic properties of sounds in this space, we may infer variety of other properties of the environment, for example, the location of the emitted sound or the underlying room layout. As preliminary step in learning the acoustic properties of scenes, we explore modeling the underlying impulse response of audio reverberations.
Past work has explored capturing the underlying acoustics of scene (Raghuvanshi & Snyder, 2014; 4; 2018; Chaitanya et al., 2020). These models, however, require manually designing acoustic functions which, critically, prevent such approaches from being applied to arbitrary scenes. In this work, we extend this approach by constructing an implicit neural representation which captures, in a generic manner, the underlying acoustics of scene. In particular, following (Raghuvanshi & Snyder, 2014), we define the acoustic modeling problem as modeling the impulse-respone a listener receives given sound emitted at emitter location (as illustrated in Figure 1) and across all possible emitter-

---

Under review as a conference paper at ICLR 2022
STERE
(a) (b) [c) [d]
Fnittrr
(c)
[h]
Figure 1: Our neural acoustic fields learn an implicit representation acoustic propagation. (a) A top-down view of the room. (b) Walkable regions shown in grey. (c) and (d) NAF captures the directional nature sound. For a single emitter location, the left and right cars experience different loudness. (e)-(h) NAF constructs a continuous acoustic field is created as move the sound emitter in the scene. For (e) and (f) note that the sound does leak through the wall. For (g) and (h) note the portaling effect open doorways have. Lighter colors indicate a stronger response. Response strength is determined by log summed STFT magnitude components for a fixed emitter location.
listener pairs in scene. This learned representation inherently captures the pattern of all acoustic reverberations in scene,
Learning representation of scene acoustics poses several challenges compared to the visual setting. First, how do we represent, high fidelity, the underlying impulse response at each emitter-listener position? While we may represent the visual appearance of a scene with an underlying three- dimensional vector, an acoustic reverberation (represented as an impulse response) can consist of over 20000 values and, thus, is significantly harder to capture. Second, how do we learn acoustic neural representation that densely generalizes to novel lmitter-listener locations? In the visual setting, ray-tracing enforce view consistency across large portions of visual scene (modulo occlusions). While principal, in a similar manner, we may reflect acoustic "rays" in our scene to obtain an impulse response, a intractable number of rays necessary obtain the desired representation.
To address both challenges, we propose Neural Acoustic Fields (NAFs). To capture the complex signal representation of impulse responses, NAFs encode and represent impulse-response in the Fourier frequency domain. Motivated by the strong influence of nearby geometry on anisotropic reflections (Raghuvanshi & Snyder, 2018), we propose to condition NAFs on local geometric information present at both the listener and emitter locations when decoding the impulse response. In our framework, local geometric information learned directly from impulse responses. Such decomposition facilitates the transfer of local information captured from training emitter-listener pairs to novel combinations of emitters and listeners
By modeling the dense acoustic fields of an environment, NAFs are powerful method to extract structural information about the scene, which we demonstrate useful across variety of different tasks. In the cross-modal setting, we demonstrate how the learned acoustic structure can be utilized to aid learned visual representations, improving novel view synthesis. Further, by directly utilizing the acoustic information in NAFs, we demonstrate how one can localize different sound sources.
In summary, we present Neural Acoustic Fields (NAFs), a neural implicit representation which captures the underlying acoustics of scene in compact and spatially continuous fashion. We show that NAFs are able to outperform baselines in modeling scene acoustics, and provide detailed analysis of the design choices in NAFs. We further illustrate how the structure learned by NAFs, can improve cross-modal generation of novel visual views of a scene. Finally, we illustrate how the continuous nature of NAFS enable the downstream application of sound source localization.
2 RELATED WORK
Audio Field Coding There is rich history of encoding methods for 3D spatial audio. These approaches largely fall into two categories. The first approach encodes the sound field at a user- centric location by capturing the sound from spatially distributed sources (Gerzon, 1973; Breebaart
2

---

Under review as a conference paper at ICLR 2022
et al., 2005; Pulkki, 2007). While they may leverage perceptual cues that create the sense of spatial audio, they do not allow the listener to freely traverse the scene and experience sound from different locations. The second approach aims to model the sound heard as a listener moves in scene (Raghuvanshi & Snyder, 2014; Mehra et al., 2014; Raghuvanshi & Snyder, 2018; Chaitanya al., 2020). Since the complete acoustic field of scene is computationally prohibitive simulate in real time, and expensive to store in full fidelity, these methods have relied on handcrafted encoding of the acoustic field, prioritizing efficiency above reproduction fidelity. Our work allows listener to move and experience sounds that come from anywhere in scene, and can represent the acoustic field continuously high fidelity by directly learning from data.
Implicit representation Our approach towards modeling the underlying acoustics scene relies on the use of a neural implicit representations. Implicit representations have emerged as a promising representation of 3D geometry Niemeyer al. (2019); Chen & Zhang (2019); Park et al. (2019); Saito et al. (2019) and appearance Sitzmann et al. (2019); Mildenhall et al. (2020); Niemeyer et al. (2020); Yariv et al. (2020) of a scene. Compared to traditional discrete representations, implicit representations are a continuous mapping capable of capturing data at an "infinite resolution". Jiang et al. (2020) proposed a grid based representation for implicit scene reconstruction, while more recently Vrie et al. (2021) has adopted spatial conditioning for 3D image synthesis, where in both settings, the grid enables a higher-fidelle fidelity encoding of the scene. Our work also leverages local grids to model acoustics, but as an inductive bias and way to generalize to novel inputs.
Audio-Visioe Learning Our work is also closely related to joint modeling of vision and audio. By leveraging the correspondence between vision and audio, work has been done to learn unsupervised video and audio representations (Aytar et al., 2016; Arandjelovic & Zisserman, 2017), localize objects that emit sound (Senocak et al., 2018; 8; Zhao et al., 2018), and jointly use vision and audio for navigation (Chen et al., 2020). Recent work aims to propose plausible reverberations sounds from image input (Singh et al., 2021; Du et al., 2021), these approaches model the phase-free log- magnitude STFT using either convolution implicit functions, which we also utilize. Different from them, our work leverages the geometric features learned by modeling acoustic fields to improve the learning of 3D view generation.
METHODS
We are interested in learning a generic acoustic representation of an arbitrary scene, which can capture the underlying sound propagation of arbitrary sound sources across both seen and unseen locations in scene. We first review relevant background information towards modeling environment reverberations. We then describe Neural Acoustic Fields (NAFs), a neural field which we show can capture, in a generic manner, the acoustics of arbitrary scenes. We further discuss how we can parameterize NAF in a manner so that it can capture acoustics property even at unseen sound sources and listener positions. Finally, we discuss the underlying implementation details of our model.
3.1 BACKGROUND ON ENVIRONMENTAL REVERBERATION
The sound emitted by a sound source undergoes decay, occlusion, and scattering due to both the geometric and material properties of scene. For fixed location pair (q, q'), we define the impulse- response at a listener position q, as the sound pressure ptt; q, 4') induced by an impulse at q' Such behavior can be concisely and elegantly modeled utilizing the linear wave equation (Pierce, 2019):
0² c² 0t2
22
p(t, q, q') 5(t)o(q - -y),
where e is the speed of sound, p is the sound pressure, (g. q') being the listener and emitter location respectively, and 5 the Dirac delta representing the forcing function, where we refer to sound pressure p(1; q, q') as the impulse-response observed at listener position q.
Given an accurate model of the impulse-responss p(t: q: q') described in Eqn (1), we may model audio reverberation of any sound waveform s(t) emitted at q', by computing the response r(t, q, q') at listener location q by querying the continuous field and using temporal convolution:
rt: g. q'd = s(t) @ p(t: q. g')
(2)
3

---

Under review as a conference paper at ICLR 2022
Local features
Emitter
Emitter feature
Implicit Decoder
Listener feature
Listener
llgencrated ground truthi]
MSE loss
Global features
Local feature grid
(x, y)
(x'y')
Fenture per pixel corner
Listener
Emitter
Phase
Time
Orientation Left/Right
Figure 2: Overview of NAF. Given listener position and emitter location, first query grid for local features using bilinear interpolation. We compute the sinusoidal embedding the positions, phase, and time, and query a discrete embedding matrix using the orientation and left/right car. These features are fod to an implicit decoder. Our method trained with MSE loss with impulse responses.
3.2 NEURAL ACOUSTIC FIELDS
We interested in constructing continuous representation of the underlying acoustics of scene, which may specify the reverberation patterns of an arbitrary sound source. The parameterization of an impulse-response introduced Section 3.1 provides with method to model audio propagation when given an omnidirectional listener and emitter. To construct model of a directional listener, we further model the 3D head orientation 0 E F2,, and ear {0, 1} (binary left right) of a listener, in addition to the spatial position
R of the listener and q' E R of the emitter.
We may then model the time domain impulse response a using neural field which takes as input the listener and emitter parameters:
R (0. R" (q.8, )) D(q. 0, 9')
(3)
(a)
(b)
(c)
Original Impulse response
impulse response STFT
Figure 3: Qualitative Visualization of Test Set Impulse Response Prediction. Left: Example log STFT of impulse responses and predictions from NAF, Right: (a)-(c) Three examples of ground truth waveform of the impulse response, and corresponding NAF generated waveform reconstructed with Griffin-Lim.
Directly outputting the impulse-response E RT with neural network is difficult to its high dimensional (over 20000 elements) and chaotic nature. naïve solution would be further add as an additional argument our neural field, but we found that such solution worked poorly, due the highly non-smonee non-smooth representation of the waveform (see Table A2). We instead encode the impulse- response utilizing short-time Fourier transform (STFT) create log-magnitude spectrogram denoted USTFT, which we find to be significantly more amenable to neural network prediction, due the smooth nature of the frequency space. In Figure impulse responses and those learned by our network.
show spectrograms for ground truth
Thus, parameterization of NAF is neural field Ω that is trained estimate the impulse response function o, and outputs USTFT for given time and frequency coordinate:
Ω : TR 10 x {0, 1) + R. (g, k. q' f, + l2g. f, k q' (t, eSTr (t,
We train our model using MSE loss between the generated and ground truth -pespeecto USTFT!
CAA [52(q, 0, k, q' f) (t,
across spectrogram coordinates and
In our qualitative results we utilize Griffin-Linn reconstruction to recover from USTFT similar to prior work (Du al., 2021). While such an approximate reconstruction ignores the underlying phase
4

---

Under review as a conference paper at ICLR 2022
of the impulse response, phase free models are typically used in spatial acoustic modeling (Pulkki, 2007; Raghuvanshi & Snyder, 2018; Singh et al., 2021). 1). Such approximations are used because the phase locking cutoff in humans occurs at around 1~ 2kHz Jhhinn-Cunninghane et al., 2000; Joris & Verschooten, 2013), below which time delay provides important spatial cues.
We demonstrate in our qualitative results that NAFs with this approximation may still accurately model realistic spatial acoustic effects. Altemattvv methods for waveform recovery exist, including methods that can be learned from data (Oord al., 2016; Kalchbrenner al., 2018). These approaches are orthogonal to our work, and we leave this exploration to a future study.
3.3 GENERALIZATION THROUGH LOCAL GEOMETRIC CONDITIONING
We are interested in parameterizing the underlying acoustic field, so that we may not only accurately represent impulse-response at emitter-listener pairs we see during training, but also at novel combina- tions of emitter and listener seen at test time. Such generalization may be problematic when directly parameterizing NAFs utilizing a MLP with inputs specified in Eqn (4), as the network may learn to directly overfit and entangle the relation between emitter and listener impulse- repulse-respooses.
What generic information may we extract from a given impulse- listener? In principle, extracting the full dense geometric information in scene would enable us to robustly generalize to new emitter and listener locations. However, the amount of geometric information available in a particular impulse-response, especially for positions far away from either current emitter and listener is limited, since these positions has little impact on the underlying umpulse-response. In contrast, the local geometry near either emitter and listener positions will have a strong influence in the impulse-response, as much of the anisotropic reflection comes from such geometry (Paasonen et al., 2017). Inspired by this observation, we aim to capture and utilize local geometric information, near either emitter or listener locations, as a means to predict impulse- responses across novel combinations.
between an emitter and
To parameterize and represent these local geometric features, we learn 2D grid of of spatial latents which we illustrate in Figure 2. When predicting an impulse-response at a given emitter and offset position, we query the learned grid features at both emitter and listener positions using bilinear sampling, and provide it as additional context into our NAF network Q. Such features provide rich information the of both emitter and listener locations. In the rest of this work, we refer to the NAFs with local geometric features as Pgrid.Wi We learn grid latent features jointly with the underlying parameters of NAF, Additional details can be found in Appendix B.
enabling NAF to generalize better unseen combinations
Such design choice, however, still requires us consider how to further combine local geometric information captured separately from either listeners or emitters. A naïve implementation would be maintain separate feature grids for both listener and emitter positions. Such an approach fails to account for the fact that the local geometric information captured by emitter may also inform the local geometric information around listener. Examining Eqn (1), we note that it is in fact symmetric with respect to exchanging either listener or emitter positions (Chaitanya et al., 2020), indicating that the impulse-response does not change when listener and emitter are swapped. Such result means that we may in fact directly utilize the local geometric information captured near an emitter position interchangeably for either emitters and listeners. Thus, we propose our local geometric information as single latent grid, which we show in our ablations outperform the naïve dual grid implementation.
4 EXPERIMENTS
In this section, we demonstrate that model can faithfully represent the acoustic impulse response at seen and unseen locations. Additional ablation studies verify the importance of utilizing local geometric features to enable test time generation fidelity. Next, demonstrate that learning acoustic fields could facilitate improved visual representations when training images are sparse. Finally we show that the leamed NAF can be used for sound source localization.
5

---

Under review as a conference paper at ICLR 2022
4.1 SETUP
For evaluating the learned acoustic fields, we use the Soundspaces dataset (Chen et al., 2020). This dataset consists of R& probe points for each scene, with each probe capable of representing an emitter or listener location for up to O(R)) emitter and listener pairs. The emitters are represented as omnidirectional, while the listener acts as a stereo receiver that can have one of four different orientations. The listeners and emitters are at fixed height. For each scene, we holdout 10% of the RIRs randomly as a test set.
Our NAFs are trained on 6 representative scenes, selected such that 2 consist of multi-room layouts, 2 consist a single room with a non-rectangular walls, and 2 consist a single room with rectangular walls as in Figure 4. Each scene is trained for 400 epochs. which takes around 3.5 hours for the largest scenes on four Nvidia v100s, In each batch, we sample 20 impulse responses, and randomly select 2, 000 frequency & time pairs within each spectrogram. An initial learning rate of 5 x 10 is used for the network, while an initial learning rate of x 10-3 is used for the grid. We add a small amount of noise sampled from (O, 0.02) to each coordinate during training to prevent degenerate solutions.
4.2 ARCHITECTURE DETAILS
The Soundspaces dataset lacks the full parameterization of an acoustic field described in Equation 4, so we train NAF with restricted parameterization that is available in the dataset. This allows for two degrees of freedom along the location of E RR. The listener can assume four possible orientations E {0, 90, 180, 270), while the emitter is omnidirectional In particular, we utilize a parameterization of Dagria which maps an input tuple [z, y, xx, y', f. R x {0, 90, 180, 270} x {0, 1} to a single scalar value that represents the intensity for given time and phase in the STFT:
plane for the listener locations q E I2² and the emitter
agna(a, y, 0 k x' 3/ f) + (t,
(6)
To encode the rotation 0, as there are only possible discrete rotations in the dataset, we directly query into a learnable embedding matrix of shape RA4* returning a R. I vector. Similarily, to encode the left and right ear, we similarly query into a learnable embedding matrix shape Ik 2x returning IR 1xk vector. The f, / tuple representing the frequency and time respectively are scaled to 1-),1) and processed with sinusoidal encoding using 10 frequencies of sin and cos up to 200Hz.
To obtain local geometric features for either emitter or listener in scene, we assume that our scene contained with a set of bounding pixels P = {PI..P') which form a grid over the scene, For a given position tuple (I, y), we then query the local features:
(z,y) = C(x, y: f(pi), F(r)) F(S)) f(22))
Where C(-) is the bilinear interpolation function, (pi...i) are the four vertices that bound (z, y). and f(-) represents the features stored at given vertex. These queried features are combined with the coordinates processed with sinusoidal encoding using 10 frequencies of sin and cos functions up to 100Hz. We process both the listener and emitter position tuples this way. We combine the grid based features with the sinusoidal embeddings and the discrete indexed embeddings as the input to our multilayer perceptron fa- Please refer to Figure 2 for visualization of our model, and Appendix for further details. We compare using a shared local geometric feature with the emitter and listener, as well using have the emitter and listener query their own individual grids.
4.3 EVALUATION ON NEURAL ACOUSTIC FIELDS
We first validate that we can capture environmental acoustics with high fidelity, at unseen emitter- listener positions.
Baselines. We compare model against several strong interpolation baselines. We experiment with holding either the listener fixed and interpolating based on the emitter location, as well as holding the emitter fixed and interpolating based on the listener location. Both linear and nearest neighbor approaches are widely used (Savioja et al., 1999; Raghuvanshi et al., 2010; Pörschmann et al., 2020) in modeling of spatial audio. We also compare sharing and using individual local geometric features.

---

Under review as a conference paper at ICLR 2022
T Large
Large 2
Medium
Medium 2
Small
Small 2
Mean
Model
MSE)
1600
MSEL
T604
MSEI
T60,
MSE]
T60
MSE.
T60.
MSEJ
T60.
MSE]
T60.
Linear-Interp
3.349
7.357
3.831
8,975
3.108
7.515
3.425
7.958
3.601
9.741
6.487
11.62
3.967
8.862
Nearest-Interp
3.208
6.738
3.568
7.897
2.813
4.526
2.811
4.902
2.691
4.198
2.625
3.963
2.952
5.371
NAF (Dual)
1.399
4.311
1.423
4.880
1.376
5.004
1.383
4.200
1.398
6.687
1,533
3.801
1.419
4.814
NAF (Shared)
1.395
4.160
1.421
5.247
1.340
3.801
1.321
3.886
1.397
6.752
1.534
3.770
1.401
4.602
Table I: Quantitative Results on Test Set Accuracy. We report the MSE difference between generated and ground truth log spectrograms across methods, as well as the percentage (%) difference for the T60 reverberation time. The best method for each room is bolded. For the nearest and linear baselines, we perform interpolation in the time domain using samples from the training set.
(a)
(b)
(c)
Large 1
Large
Medium
Medium
Small
Small
Figure 4: Qualitative Visualization of Neural Acoustic Fields. (a) The 3D structure of the room, (b)
Walkable regions shown in grey. (c) Strength of predicted impulse response given a emitter location, lighter color indicates louder sound.
Results. We evaluate the results of our synthesis by measuring the mean squared error (MSE)
between the generated and the ground truth log-spectrograms, as well as measuring the percentage error between the T60 reverberation time in the time domain. In this case, lower MSE and T60-error values indicates a better result. As shown in Table 1, our NAFs achieve significantly higher quality on the modeling of unseen impulse responses compared to strong interpolation baselines. comparison of using shared and dual local geometric features indicates that despite having fewer learnable parameters, we achieve better performance by sharing the local geometric features. Examples of individual impulse responses generated by model are shown in Figure 3. Figures 4 shows the different scenes and the loudness change predicted by our NAFs, Our model is capable of predicting smoothly varying acoustic fields that affected by the physical surroundings.
Generalization through Geometric Conditioning. We next assess the impact of utilizing local geometric condi- tioning as a means to generalize to novel combinations of emitter-listener positions. On the "Large 1" room, in Fig- ure 5 we evaluate test set MSE error when NAF is trained with limited percentage of the training data either with or without local geometric conditioning. We find that such geometric conditioning enable substantially better test set reconstruction error, with the underlying performance gap increasing with less data.
1.75
weo local geometric cond lioning
3.70
local geamettic conditining
3.65
1.60
3.55
1.50
25%
30%
40%
50%
60%
70%
80%
Percentage data training
Figure 5: Local Geometric Conditioning. Comparison of NAF with and without local geometric conditioning trained with different amounts of data.
4.4 CROAL LEARNING
When NAF trained, as shown in Figure 6, a structured geometric representation emerges. This structured latent stands in sharp contrast with the noisy structure learned by NeRF models. In this experiment, we explore the effect of jointly learning acoustics and visual information when we are given sparse visual information. Recall that our NAF includes local geometric feature grid P that covers the entire scene. For our cross cross-modee modal learning experiment, we jointly leam this feature grid with NeRF network modified to accept both local features along with the traditional sinusoidal
7

---

Under review as a conference paper at ICLR 2022
Boon
Room Structure
Acoustia only
ROB Acoustia
Figure 6: Qualitative Grid Visualization. Visualization of the learned grid features with principal component analysis. When trained using acoustic information, the grid learns highly structured information. When using NeRF, the grid is highly noisy. Structure emerges when we train both frameworks together.
embedding. In the acoustics branch, we query the grid using emitter and listener positions. In the NeRF branch, we use point samples along the ray projected on the grid plane to query the features. In both cases, the process is fully differentiable. We use standard implementation of NeRF with a coarse and fine network. both the cross-modal and RGB only experiments we augment the fine network with learnable local feature grid. In the NeRF only setting, we minimize color C reconstruction loss for ray over a batch of rays R: LRGB = ExERRI((r)) - In contrast, in the NAF + NeRF experiment, we jointly minimize LRGB + CNAF. where CNAF is defined in equation 5. We utilize 64 coarse samples and 128 fine samples for each ray, and sample 1024 rays per batch.
C(r)l
iiss Accolts
Figure 7: Qualitative Visualization of Cross-Modal Image Generation. Qualitative comparison between NeRF learned jointly with a NAF with RGB and acoustic supervision, and NeRF learned with only RGB supervision. We observe fewer floating artifacts when jointly training with audio. (a)-(c) Three views from "Large 1". (d)-(f) Three views from "Large 2".
Large Room
Large Room 2
PSNR
MSE
PSNR
MSE |
Training Images
75
100
150
75
100
150
75
100
150
75
100
150
NeRF
25.41
27.36
29.85
6.618
3.506
1.740
25.70
27.74
29.34
6.921
3,905
2.185
NeRF NAF
26.19
27.59
29.90
5.209
2.983
1.625
26.24
28.22
29.45
5.641
3.075
2.034
Table 2:
Quantitative Results on Cross-Modal Image Learning. Quantitative results on joint training of NeRF and NAF jointly conditioned on single local grid. We use very sparse training images in highly complex scenes. When evaluated on 50 test images, observe that cross-modal learning helps improve PSNR when the visual training data more sparse. MSE results are multiplied by 10
Results. We train the two large rooms in our training set. For each room 75, 100, 150 images are used for training, while the same images of novel views are used for configurations during testing. In Table 2 observe that training with acoustic information helps improve the PSNR and MSE of the visual output. This effect is more significant when the training images are very sparse, the NAF network helps less when there is sufficient visual information. Qualitative results are shown in Figure 7, we see there is a reduction floaters in free space.
We visualize the local features learned in Figure 6. While the features learned using only RGB supervision lack coherent structure, the one we jointly learn with audio exhibits clear structure. To assess the structure in grid latents, fit linear layer regress the room position of each latent. For
8

---

Under review as a conference paper at ICLR 2022
room and respectively, achieve R of 0.109 and 0.085 when trained RGB only, compared to R2 of 0.385 and 0.201 when trained with RGB + Acoustics supervision. This demonstrates the spatially informative structure captured by jointly learning NAF and NeRF.
Enatee
(6)
16T
(c)
(d)
Figure 8: Qualitative Visualization of Sound Localization. Our NAFs can localize where sound is coming from given listener samples. (a)-(d) Given unperturbed audio, the NAF can model the sound by placing the source anywhere in the For each grid location emitter, evaluate how well the listener response matches observation. actual emitter location shown red. Lighter color indicates that if an emitter was placed here, there is lower error compared to observations, which matches well with the real emitter location.
Method
Chords
Bells
Piano
Sweep
ResNet-18
0.658
0.476
0.916
0.983
0.188
Max mag.
0.126
0,393
0.919
0.225
0.531
NAFs
0.004
0.025
0.038
0.004
0.015
Table 3: Quantitative Results Sound Localization. Quantitative results sound localization distance in normalized room coordinates. NAFs can accurately estimate location emitter.
4.5 SOUND SOURCE LOCALIZATION
Given a high fidelity representation of the acoustic field, it is possible to identify the location of sound source in a scene, regardless of where the listener is located? In this task, we demonstrate that a trained NAF can used for localizing an emitter based on sparse audio samples from the scene. We explore sounds in total, 4 of which are musical instruments and is a sine sweep (Sweep).
We assume that exists a set of audio recordings Vi E V from an object, recorded from 10 different locations and orientations that are known. We further assume that we are given the original unper- turbed waveform Vorig the object emits, with the underlying task being to locate the position of the emitter (a', y²) using these sound samples and the unperturbed sound.
We hold the listener position and orientations fixed and vary the emitter position to probe for the expected sound at each location. For each location, render log- seescttroogamm using trained NAF and compare this against the actual log-spectrogram recorded response Vi:
log (loggstt((Ver Vorig pe (t; x ))) log(s
We perform the search for the emitter location by discretizing the scene in 0.3m segments.
Baselines. We compare against two baselines, a ResNet-18 network that takes as input both the spectrogram of the transformed and original sounds, and is trained regress the object location relative to the listener frame. Given 10 individuals, we take an average of the predicted emitter location. We also compare against a modified maximum magnitude baseline, where given set of recording locations, we simply select the position where the sound is loudest.
Results. Our NAFs can achieve very accurate localization as seen in Table 3. We also visualize the етrоr map with the ground truth emtiter location labeled in Figure 8. It can be observed that the real location of the emitter matches well with the location of the lowest prediction error.
CONCLUSION
In summary, this paper introduces Neural Acoustic Fields (NAFs), continuous, differentiable acoustic representation which can faithfully represent the reverberation of different audio sources scene, as well as the scene's overall acoustic environment. By conditioning NAFs locally on the underlying scene geometry, demonstrate that approach enables the accurate prediction of environmental reverberations even at unseen locations the scene. Furthermore, we demonstrate that the acoustic representations learned through NAFs powerful, and may be utilized facilitate mudio-visual cross-modal learning, as well as to localize the underlying source of arbitrary sounds.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In Proceedings of the IEEE
International Conference on Computer Vision, pp. 609-617, 2017. 3
Yusuf Aytar, Carl Vondrick, and Antonio Torralba. Soundnet: Learning sound representations from unlabeled video. Advances neural information processing systems, 29:892-900, 2016. 3
Jeroen Breebaart, Jürgen Herre, Christof Faller, Jonas Rödén, Francois Myburg, Sascha Disch, Heiko Punhaagn, Gerard Hotho, Matthias Neusinger, C Kjörling, et al. Mpeg spatial audio coding/mpeg surround: Overview and current status. Preprint 1/9th Conv. Aud. Eng. Soc., (CONF), 2005. 2
Chakravarty R Alla Chaitanya, Nikunj Raghuvanshi, Keith W Godin, Zechen Zhang, Derek Nowrouzezahrai, and John M Snyder. Directional sources and listeners in interactive sound propagation using reciprocal wave field coding. ACM Transactions on Graphics (TOG), 39(4): 44-1, 2020. 3,5
Changan Chen, Unnat Jain, Carl Schissler, Sebastia Vicenc Amengual Gari, Ziad Al-Halah, Vamsi Kr- ishna Ithapu, Philip Robinson, and Kristen Grauman. Soundspaces: Audio-visun navigation in 3d environments. In ECCV, 2020. 3,
Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proc. CVPR, pp. 9939-5948, 2019. 3
Terrance DeVeiss, Miguel Angel Bautista, Nitish Srivastava, Graham W Taylor, and Joshua M Susskind. Unconstrained scene generation with locally conditioned radiance fields. arXiv preprint arXiv:2104.00670 2021. 3
Yilun Du, M. Katherine Collins, B. Joshua Tenenbaum, and Vincent Sitzmann. Learning signal- agnostic manifolds of neural fields. In Advances Neural Information Processing Systems, 2021. 3, 4
Michael A Gerzon. Periphony: With-height sound reproduction. Journal of the audio engineering society, 21(1):2-10, 1973.
Chiyu Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießner, and Thomas Funkhouser. Local implicit grid representations for 3d scenes. In Proc. CVPR, pp. 6001 2020. 3
Philip X Joris and Eric Verschooten. On the limit of neural phase locking fine structure in humans. Basic aspects of hearing, pp. 101-108, 2013. 5
Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aaron Oord, Sander Dieleman, and Koray Kavukcuoglu. Efficient neural audio synthesis. In International Conference on Machine Learning, pp. 2419-2419. PMLR, 2018. 5
Ravish Mchra, Lakulish Antani, Sujeong Kim, and Dinesh Manocha. Source and listener directivity for interactive bave-based sound propagation. IEEE transactions visualization and computer graphics, 20(4):495-503, 2014.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In Proc. ECCV, 2020. 3,
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Occupancy flow: 4d reconstruction by learning particle dynamics. In Proc. ICCV, 2019. 3
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In Proc. CVPR, 2020. 1, 3
Aaron den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv: /600:000999999 2016. 5
10

---

Under review as a conference paper at ICLR 2022
Juhani Paasonen, Aleksandr Karapetyan. Jan Plogsties, and Ville Pulkki. Proximity of sur- faces acoustic and perceptual effects. Journal of Audio Engineering Society, 65(12)9997-0004, 2017. 5
Jeong Joon Park. Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proc. CVPR, 2019. 3
Allan D Pierce. Acoustics: an introduction to its physical principles and applications. Springer, 2019. 3
Christoph Pörschmann, Johannes M Arend, David Bau, and Tim Lübeck. Comparison of spherical harmonics and nearest-neighbor based interpolation of head-related transfer functions. In Audio Engineering Society Conference: 2020 AES International Conference on Audio for Virtual and Augmented Reality. Audio Engineering Society, 2020. 6
Ville Pulkki. Spatial sound reproduction with directional audio coding. Journal of the Audio Engineering Society, 55(6):503-516, 2007. 3,5
Nikunj Raghuvanshi and John Snyder. Parametric wave field coding for precomputed sound propaga- tion, ACM Transactions on Gruphics (TOG), 33(4):1-11, 2014. 1,3
Nikunj Raghuvanshi and John Snyder. Parametric directional coding for precomputed sound propa- gation. ACM Transactions on Graphics (TOG), 37(4):1-14, 2018. 3,
Nikunj Raghuvanshi, John Snyder, Ravish Mehra, Ming Lin, and Naga Govindaraju. Precomputed wave simulation for real-time sound propagation of dynamic sources in complex scenes. In ACM SIGGRAPH 2010 papers. pp. 1-11. 2010.
Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In Proc. ICCV, pp. 2304-2314, 14, 2019. 3
Lauri Savioja, Jyri Huopaniemi, Tapio Lokki, and Ritta Väänänen. Creating interactive virtual acoustic environments. Journal of the Audio Engineering Society, 47(9):675-705, 1999. 6
Arda Senocak, Tac-Hyun Oh, Junsik Kim, Ming- Hsuan Yang, and In So Kweon. Learning localize sound source in visual scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4358 135664366,, 2018. 3
Barbara G Chinn-Cunningham, Scott Santarelli, and Norbert Kopco. Tori of confusion: Binaural localization cues for sources within reach of a listener. The Journal of the Acoustical Society of America, 107()):627-1636, 2000.
Nikhil Singh, Jeff Mentch, Jerry Ng, Matthew Beveridge, and Iddo Drori. Image2reverb: Cross-model reverb impulse response synthesis. In IEFF/CVF International Conference on Computer Vision (ICCV), October 2021. 3.5
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Contin- uous 3d-structure-awaan neural scene representations. In Proc. NeurlPS 2019, 2019. 1,3
Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. Proc. NeurIPS, 2020. 3,3
Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. The sound of pixels. In Proceedings of the European conference on computer vision (ECCV), pp. 570-586, 2018. 3
11

---

Under review as a conference paper at ICLR 2022
Appendix
A ADDITIONAL VISUALIZATION OF ROOMS
Figure A1: Additional Qualitative Predictions of NAF, Qualitative visualization of the loudness map as predicted by NAF across four different rooms.
We show additional NAF predictions of loudness as we move an emitter inside different rooms in Figure A1. For each room, note how the sound is affected by the geometry. In wide open spaces the sound is highly dispersed. While in thin structures the sound test to concentrate locally. As we move farther from the source, the loudness of the sound decreases.
12

---

Under review as a conference paper at ICLR 2022
B ARCHITECTURE AND TRAINING DETAILS
We visualize the two alternative models that we experiment with, in Figure A2 is network that uses different local feature grids for the emitter and receiver. The network uses the emitter and listener to positions to sample from the two different grids.
In Figure A3 show model that does not utilize any kind of local geometry conditioning. The listener, emitter, phase, and time input are transformed using sinusoidal embedding. while the orientation and left/right retrieved. All transformed inputs directly fed to the network.
Emitter
Local features
Emiteer Feature
Implicit Decoder
Listener feature
Listener
ligenerated ground truth
Local emitter feature grid Feature per pixel corner
MSE loss
Aom features
grid Feature per pixel corner
yl Listener
(s y")
Emitter
Phase
Orientation LiftRight
Figure A2: Architecture of the model that uses emitter and listener specific local geometry condition- ing.
Implicit Decoder
ligenerated ground truth]?
MSE loss
Global features
(x.y)
(x,,y)
Listener
Emitter
Phase
Time
Orlentation Left/Right
Figure A3: Architecture of the model that no local geometry conditioning.
Each network consists of 8 fully connected layers in feedforward fashion, as well a skip connection consisting of two fully connected layers. The skip connection takes the input and adds its output to that for the fourth intermediate layer. We utilize an intermediate feature size of 512, and softplus as the activation function. The grid is of size 322x32x04 where 32 is the size of the spatial dimensions, while 64 is the channel count. We initialize each element of the grid i.i.d. from N(0 ). For the network excluding the grid, we utilize an initial learning rate
4. For the grid, we utilize an initial learning rate of le - 3. The Adam optimizer is used when training our network. We utilize orientation embedding of shape R 7xx5112 where is the number of intermediate outputs, 4 is the number of orientations, and 512 is the feature dimension. For the left-right embedding. we use shape of R x2x2511 We perform additive conditioning by adding R512 vector to each intermediate output for both the orientation and the left/right.
For each scene, generate -pectrooroo for each impulse response, we compute the mean and standard deviation (rree: Oree for each frequency in the log-spectrogram, and normalize the data prior training:
Uneq
Ffreg
Ufoo
3.0 Ofrea
For the sinusoidal embedding, utilize both and sin with 10 frequencies each for encoding position, phase, and time. For encoding position we utilize a max frequency of 100Hz, while for encoding time and frequency utilize max frequency of 200Hz.
Since we do not know beforchand the time duration of an impulse response at an unseen location, compute the maximum impulse length for each scene and this length to zero pad the training
13

---

Under review as a conference paper at ICLR 2022
impulse responses. Because the padded regions do not contain useful information, want the network to focus modeling efforts on the early regions of the impulse response. We achieve this by stochastically padding the impulse response to maximum impulse length with 0.1 probability. Because the implicit function is trained on individual (t, f) coordinates within a given USTFT. training samples do not need to be of the same length. During test time, we perform inference up to the maximum duration of scene impulse response.
C DATASET VISUALIZATION
(a)
(b)
Figure A4: A room the emitter-listener probes. (a) The 3D structure of room. The prohes marking the location of emitters//isteners.
In Figure A4, visualize both the room and underlying set of probe positions in the training data. Due to occlusion and the geometry, even slightly moving the emitter or listener position can result in different results. As demonstrated in Table 7, both nearest neighbor and linear interpolation perform poorly compared to our learned solution. contrast, recovered acoustic fields from NAF trained these probe positions is substantially denser (Figure A1).
D STORAGE COMPARISON
Method
Storage (MB)
Linear-Interp
5970
Nearest-Inted
5970
NAFs (Dual Local Feat)
27.07
NAFs (Shared Local Feat)
25.37
Table Al: Approximate disk storage cost of different methods. We average the amount of data required for diffcrent methods of inference for the six scenes. Note that the linear and nearest interpolation methods require the entire training set, while the NAF based methods use constant storage.
We compare the averaged on disk storage cost of the different methods for inferring the spatial audio using a precomputed training in Table A.. Both linear and nearest interpolation methods require access to the entire training set, while our NAF based approaches compactly encode the acoustic scene into a fixed size.
E VISUALIZATION OF INTERPOLATED METHODS
In this section, we visualize the loudness when interpolation method is used. We compare nearest neighbor interpolation with linear interpolation applied to the training set, both in the time domain. The nearest neighbor method presents sudden step changes in loudness, while the linear interpolation fails when the queried location falls outside the convex hull of the training set. Also note the sound leakage in (c)-(d) in the bottom wall when linear interpolation is used.
14

---

Under review as a conference paper at ICLR 2022
(a)
Figure A5: Visualization using interpolated sound. We show the loudness at different locations
using (a)-(b) nearest-neighbor interpolation and (c)-(d) linear interpolation.
F ALTERNATIVE REPRESENTATIONS
Representation
MSF
T60
Magnitude only
1,395
4.160
Magnitude phase
1.439
5.254
Time domain
39.87
61.82
Table A2: Learning different representations We compare learning magnitude only. jointly learning magnitude and phase, as well as directly learning in the time domain.
Bround buth
Qualitative Visualization of impulse waveforms unseen emitter/listener locations. From left to right, we show the ground truth waveform, the Griffin Lifffin-Lim recovered waveform, and the waveform Icarned by network in the time domain for two locations (a)-(b) not seen in training.
Figure A6:
Our current method follows prior work in learning in the log- magnituge STFT domain. In this section, we investigate two possible alternatives: learning phase nog-maanitude, and directly learning the time domain. The MSE and T60 error percentage is presented in Table A2. We observe that jointly modeling phase log-magnitude degrades the performance slightly compared to modeling just the rog-manntudd, while modeling in the time domain performs poorly, We visualize the waveform at two test set listener/emitter locations Figure A6.
G Lg REGULARIZED GRID IN NERF
Large 1
Large 2
PSNR
MSE 1.
PSNR 1 MSE
NcRF + grid + La
22.69
6.956
24,86
7.128
NeRF + grid
25.41
6.618
25.70
6.921
Table A3: Regularizing the grid. In this experiment, we compare learning NeRF with grid without regularization, and with Lg regularization.
In Table A3 we compare NeRF that utilizes a grid and trained using image reconstruction loss, against variant where La penalty with weight le to ensure smooth latent space is added to the image reconstruction loss. There are 75 images used in the training set. We observe degraded performance when we apply this penalty.
15

---

Under review as a conference paper at ICLR 2022
H
INTERPOLATION OF LEFT/RIGHT LATENT
Left
Right
Left
Right
[b]
Figure A7: We interpolate linearly between left/right latents. Here show two examples (a)-(b) where we take linear steps between the latent representing the left ear, and the latent representing the right car. Pay attention to the onset (left) of each spectrogram.
As shown in Figure A7, are the output for two locations when take linear steps between the latent representing the left ear, and the right ear. We observe that the output smoothly changes as we move in the latent space.
16


Reviews:
[{"review_id": "yhS215HfqWi", "reviewer": "Reviewer_doFB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe proposed idea is novel and expected to promote other related researches in the future.\n\nWeaknesses:\nI think the details written in this paper is not enough to reproduce the results.\n1. How can one listen to the predicted impulse response when the estimate values are only the magnitude parts?\n2. How were the visualization done in Figure 1 and 4?\n3. What happens on the local grid parts where the impulse response was not collected enough if not at all? I assume those parts will not be trained and results in poor prediction. Is this correct?\n4. in section 4.2, the authors say something like \"processed with sinusoidal encoding\", but the details are not enough.", "summary_of_the_paper": "This paper proposes an neural acoustic field (NAF), which is an extended version of neural radience field applied on impulse responses defined between listener and emitter positions. It predicts an impulse response given the listener and emitter positions so that one can simulate the impulse response in arbitrary positions. One of the key idea to extend the model to unseen positions is to take local grid features as input. The authors show other applications with NAF such as source localization and multi-modal NERF. Experiment results show that the proposed method shows better results when comparing to baseline methods.", "summary_of_the_review": "The idea is seemingly novel but it is not enough for publication in this status.\nI recommend reject. But I'm willing to raise the score if the authors answer the questions with more details and write them in the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "lSeUEnzfXdr", "reviewer": "Reviewer_U1r6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The manuscript rightfully points to the importance of being able to roughly deduce the scene geometry from monaural recordings. To address this challenge, the authors propose a neural network architecture that not only takes into account the speaker and listener positions in a room but also some geometrical information as well. Overall, while I do think this is an important and challenging problem, I think the manuscript falls short of what it promises. I also found the manuscript unclear about important details at times. Overall, it should do better in terms of presenting convincing evidence to show that the neural network does more than learning to interpolate a smooth field (since log magnitudes are the only outputs of the neural network).\n\nHere are some more specific comments : \n\n1. Introduction, third paragraph : I agree that being able to infer scene properties given monaural recordings is something humans do, but I think the proposed method is a rather modest step in this direction. After all, for each individual room, you need to have many prerecorded room impulse responses to train the neural network. When you enter a totally new room (with no previous RIRs available), it's not clear to me how one could use the proposed neural network to infer the shape of the room. Could you elaborate?\n\n2. Page 2, \"... we propose to condition NAFs on the local geometric information present at both the listener and emitter locations...\" : I understand you provide a grid with the emitter, listener locations, as in Fig 2, but how do you form the grid exactly -- you write about this verbally, but can you be more specific about the details?\n\n3. eqn (3) : It's interesting that you encode binary information (which ear) as a real. I'm curious, what happens when you input $k = 0.5  (k_{\\text{left}} + k_{\\text{right}})$?\n\n4. In eqns (3), (4), please indicate the dimensions of the components (i.e., $x\\in\\mathbb{R}^3$, $\\theta \\in \\mathbb{R}^2$ etc.)\n\n5. It's a bit confusing to say that $\\Phi$ in eqn (4) is your NAF -- it's the version without geometric features, isn't it? I'd suggest declaring another function, and clearly stating the input dimensions of the various arguments, so as to separate it from $\\Phi$.\n\n6. For $\\Phi$ in eqn (4), you try to predict the logarithm of the magnitude -- how do you recover the phase? Isn't that an important part of the room impulse response?\n\n7. Section 4.3, \"...pad the training impulse response with 0.1 chance.\" : What does \"padding with a chance\" mean? I thought you'd zero pad short impulse responses so that the length of every impulse response is the same.\n\n8. Section 4.3 : How do you select the test set? Do you randomly sample the available RIRs? Or do you hold out a specific region entirely? The latter would be more interesting in terms of evaluating the generalization capability, touched upon in the third paragraph of the Introduction (see my comment-1).\n\n9. Eqn (8) : Isn't the idea to obtain $p_i$ from the NAF? How do you deal with the lost phase of the impulse response (see comment-6)?\n\n10. Section 4.5, baselines : I understand you minimize eqn (8) over $x$, right? If so, for the nearest neighbor method, are you doing a search over the available room impulse responses? I don't understand why you'd choose the loudest. Please clarify.", "summary_of_the_paper": "The manuscript considers the problem of encoding room impulse responses in a room. Specifically, the authors propose a network that takes in some geometrical cues from the room, along with speaker, listener positions, and train a neural network using pre-recorded room impulse responses. They argue that the neural network learns to make use of the geometrical cues, and can generalize beyond the dataset.", "summary_of_the_review": "Overall, the manuscript considers a very interesting problem, but I am not convinced that the proposed method would be sufficient to provide a solution as implied in the Introduction. Specifically, given a large database of room impulse responses, the neural network appears to learn how to store a smooth field based on those values (since log magnitude is the target), but it's not really clear how the geometrical side information could be useful for generalizing beyond a room. I also found the paper to be a light on some of the details, which I think are critical.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SEAXRaQgn1U", "reviewer": "Reviewer_V59y", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:  This paper shows a concrete methodology on how one can learn a NAF.  The use of a spectrotemporal representation, the particular way of representing positional information, the use of a local feature grid, and the architecture used, are interesting to know for anyone wanting to replicate that work.\n\nWeaknesses: There are a lot of questions that arise in this work, and I was disappointed to see space being used for unnecessary fluff (wave equation?!!?) as opposed to a deeper investigation.\n\n-- The choice to use a spectrotemporal representation is interesting, but it isn't clear why it works.  If the goal is to represent an RIR, a spectrogram would take at least as many values at the time-domain representation (actually more if you use windowing, which one should).  Why it is that it works better than using the time-domain data?  It isn't clear from the writing, but I assume that by \"log spectrogram\" you mean log( abs( STFT)) and not log( STFT).  If this is the case then you are not learning a room RIR, but something far simpler, which might explain why the problem becomes more tractable.  Is this the case?  If so, why would anyone care to estimate that quantity?  It's certainly of no value to any realistic audio problem, and doesn't resolve the issue of estimating the RIR which is what you start with.  Linked to this, comes the question of the loss function.  What is the MSE of the log spectrogram of an RIR, and why would anyone care to optimize it?  Clearly a lot of the most useful RIR information is in the phase component.\n\n-- The experiments are really weak.  I think the most solid experimental point is figure 5, which conclusively supports the use of geometry conditioning.  But the experiments in sections 4.4 and 4.5 seem very contrived.  Of course the NAFs will help the NeRFs since they are smoother functions (due to ignoring phase), and the localization task used is just unrealistic.  I don't see how these experiments strengthen any point in this paper, they are simple more elaborate variations of your MSE estimate.\n\nMinor things: \"in principal\" ⟶ \"in principle\"", "summary_of_the_paper": "This paper makes use of the NeRF idea on RIRs (coined here as NAFs).  The main idea is relatively straightforward, sample a room from multiple emitter/receiver locations and learn to predict the resulting RIR using a neural net that receives as input only the involved locations, and some positional information.  It is shown that this approach learns something, and it is used in a few demonstrative examples such as sound localization and enhancing NeRF estimation.", "summary_of_the_review": "Just as one can learn NeRFs, the authors show that it is possible to also learn (some version) of RIRs.  Representing RIRs is a big deal, and this paper can potentially be the first one published to use this approach.  However, had this been submitted to an audio conference I predict that it would be summarily rejected.  Unless I misunderstood your representation of the RIR, the problem solved is one that is made to be solvable, but not particularly useful for any real problem.  The log spectrogram of an RIR is not a quantity that one would employ in any audio processing, and the degree of approximation shown in figure 3 is way off from a high fidelity result (alas there were no qualitative listening experiments in this paper).  Although I appreciate the initiative, I would have expected a lot more depth in the first neural RIR paper, and this is coming short by a lot.  It's nice to see that a simpler version of this problem is solvable, but I do not see a useful contribution here.\n\nIf, on the other hand, I'm mistaken and you are actually estimating the phase as well, then I would like to see some results with more realistic audio experiments and potentially some quantitative listening numbers.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GM6iiUHEywK", "reviewer": "Reviewer_HhWA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- The idea of using a neural network to represent the acoustic fields of how sound propagates in space is new and interesting.\n\n- The paper is well motivated and it makes a lot of sense to represent neural acoustic fields or sound propagation as a neural network, which is differentiable.\n\n- Leveraging what is learned in the neural acoustic fields neural implicit representation, the paper demonstrates several tasks that are pretty interesting, including cross-modal generation and sound source localization.\n\nWeaknesses:\n\n- First of all, although this paper is well-motivated and set the stage to introduce Neural Acoustic Fields, which is an implicit neural representation that captures how sound propagates in a physical scene, actually the network only *remembers a dataset*. The background of the room impuse response and environmental reverberation is nice, but the dataset of room impulse responses is captured/introduced in another paper (Chen et al. 2020), which is not the contribution of this paper. The neural network that this paper introduces is basically a NeRF network and the paper trains such a network to remember the dataset. Fm the introduction of the paper, it sounds like the paper is going to introduce a differentiable neural network design that can model ray tracing and sound propagation in a fully differentiable way, but actually it's not.\n\n- There are multiple places the paper overclaims its contributions: For example, in the end of Audio Field Coding in related work, it is claimed that \"the work allows a listener to move and experience sounds that come from anywhere in a scene and can represent the acoustic field continuously at high fidelity by directly learning from data\". This is overclaimed, because this is not enabled by the design of the proposed neural network. It is enabled by the dataset, which uses simulation to model room impulse response. The paper only uses a network to remember this dataset.\n\n- Related to the discussions above, the paper didn't discuss at all why it is necessary to use such an implicit representation. Why not just directly use the dataset instead? \n\n- Providing grid features near both emitter and listener positions as additional context seems like a very strong assumption. It provides much extra info to the neural network.\n\n- For equation 6, it is mentioned v is a single scale value taht represents the intensity. But isn't *v* should be the room impulse response? This is very confusing.\n\n- For figure 4, it would be great to show sound ground truth from the dataset for comparison. I assume these maps can also be directly generated using the dataset.\n\n- The writing of the paper can also be further improved. There are multiple places in the paper that are not clear or need more details. Here are some detailed questions:\"\n\n1) More descriptions are need on the local feature grid part. Currently the descriptions are very vague. It is not clear what exactly is the local features and what is used to represent them.\n\n2) For Equation 3, it is unclear what theta and k is. Although it becomes clearer later when introducing the dataset, but the dataset is modeling a grid in 2d. So x and x' only have 2 dimensions instead. This should be clarified and discussed.\n\n3) The RGB only baseline in Figure 6 is unclear and there is no description of the baseline. \n\n4) How joint training is done for cross-modal image generation is also unclear. What the NeRF baseline is trying to encode?", "summary_of_the_paper": "This paper introduces the concept of Neural Acoustic Fields, which is an implicit representation to capture how sound propagates in a pysical scene. Using a NeRF-like network architecture, the model is able to predict room impulse response given the listener and source locations as input. The idea is new and interesing, but there are multiple overclaims in the paper. Some interesting tasks are demonstrated like sound source location, cross-modal generation, etc.", "summary_of_the_review": "In general, this paper studies an interesting problem and presents some preliminary attempts. But overall, the proposed method seems just another way to represent an existing dataset from another work. The paper overclaims its contribution and there are multiple places in the paper that needs more clarifications or analysis or writing improvement. Therefore, I recommend reject of the paper at this stage but happy to discuss more.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to use implicit neural representations to model how our surroundings affect the sounds reverberating within. Concretely, the proposed approach can produce impulse responses that capture environment reverberations between any two points in a scene.

Reviewers praised the novelty and originality of the idea (and I concur), but raised concerns about the clarity of the writing (especially w.r.t. modelling the phase component), lack of detail, insufficient or inadequate experiments and overclaiming of results. (There were also concerns about overclaiming of contributions, but I am inclined to agree with the authors that this isn't really the case.)

The authors have clearly taken the time to try to address these concerns, and I commend them on their willingness to engage with the reviewers' comments and suggestions. While one of the reviewers raised their score to "accept", I am inclined to agree with the other reviewers and recommend rejection. The required degree of revision is substantial, and therefore difficult to assess within a single review cycle. I believe this work must undergo another thorough assessment in its revised form, before it can be accepted for publication.
Decision: Reject