Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: Modeling room and scene acoustics is vital for AR and VR problems. Realistic rendering of spatialized audio is important. This paper builds a representation learning model/schema for parametrizing such acoustic responses via a encoder-decoder style deep network interpolation machine. This is a neat idea. The paper is written well and the benefits of the proposal are explored. 
strengths_and_weaknesses: Strengths: 
1. Clearly motivated for NN modeling. Well written and presented. 
2. Reasonable to expect the benefits of NeRF to extend to acoustic fields. 

Main weakness: 
Room and Scene acoustics is a well studied area in Audio. In particular, interpolation of room responses with and without geometry modeling including accounting for diffraction and dispersion patterns around object boundaries is being actively studied in signal processing and acoustics. Although the authors do build the related works, there are still relevant baselines that are missing (both from discussions and from evaluations perspective) like https://ieeexplore.ieee.org/document/7987742 and others. This is not to say that the evaluations are invalid, but more to understand what the network is really learning compared to classical approaches. 
Additionally a bulk of the content in Section 3 introing the acoustic functions can be reduced and cited to appropriate references. Saving much real-estate. 
questions: Apart from addressing the aspects mentioned in Weaknesses, some other points to expand are: 
1. Talk a little more clearly about the decoder setup, and how does this architecture relate to recent advances in NNs for signal parameterization of impulse responses (like https://ieeexplore.ieee.org/abstract/document/9746135)
2. The resolution level of T60 is rather coarse to clearly articulate the benefits of the parameterization in the paper. Can we look at the IRs themselves and capture differences in DRR or early reflections? DRR in particular should show clear differences between this proposal and signal processing baselines. 
limitations: Yes. This was addressed.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: Disclosure: I reviewed a previous version of this paper submitted to another conference earlier this year. 

This paper introduces the concept of Neural Acoustic Fields, inspired by the recent developments in Neural Radiance Fields (NeRF) in the context of visual scene generations from a few still images. The idea is, for a given complex acoustic environment, to learn using a neural network a function that maps a pair of 3D (emitter, receiver) locations, a receiver orientation, a receiver channel (left or right) and a time-frequency point (t,f) to the corresponding log-magnitude and phase spectrogram of the room impulse response (RIR) from that emitter to that receiver at that (t,f) point. The function is learned using a balanced least square loss in the log-magnitude and phase domains and positions are represented by a sinusoidal embedding as well as learned geometrical feature maps.


strengths_and_weaknesses: ## Strength
- The proposed approach is sound both in terms of the underlying acoustical model and in terms of the neural network design and training process, and the general idea of NAFs is novel, interesting, and potentially impactful.
- The ability of the method to correctly estimate log-magnitude time-frequency responses and reverberation times at any point in different complex environments across a synthetic and a real dataset is convincingly demonstrated.
- A number of interesting experiments reveal that the proposed model learns geometrical structure that could be used for other purposes.

## Weakness
- An important weakness of the paper is the evaluation (See "limitations" for more details).
- The paper lacks clarity as well as many important details to make it self-contained and reproducible (See "limitations" for more details).
questions: I do not have specific questions except those arising from the lack of details provided in the paper (see below).
limitations: ## Phase and binaural rendering
The main limitation of the paper is the evaluation. Indeed, the only two objective metrics used to evaluate generate RIRs are insufficient to validate the central claim made by the authors in the abstract, namely: "continuous nature of NAFs enables us to *render spatial acoustics* for a listener at arbitrary locations".

The first metric is the RT60 error, which is a single scalar parameters that globally captures the reverberation time but ignores fine properties of the reverberation which typically vary across frequency. The second metric used is the mean squared error on the log-magnitude spectrogram of the RIR. This metric completely ignores the phase reconstruction. Though, correctly reconstructing the phase is known to be of crucial importance for rendering reverberation, and even more so for rendering spatial effects, in binaural settings. Indeed, the primary cue for sound source localization are phase differences across frequencies, which need to be very accurately rendered. The authors claim that their model is able to render multiple orientations of a binaural listener. However, the main notable effect of rotating a listener on the spot for a fixed source in a fixed environment will be on the phase differences between the two channels. The authors to propose to predict the phase via an instantaneous frequency representation, but the reconstructed phases and phase differences are never evaluation.

In short, the evaluation is currently insufficient to support the implicit strong claim that the proposed method is able to "render spatial acoustics" in the binaural settings. They merely convince that the method is able to accurately render *monaural* reverberation, which is a significantly easier (though highly non-trivial) task. This strong limitation is not properly addressed anywhere in the manuscript.

## Clarity and precision
The writing is unprecise and handwavy at a number of places. It gives the general impression that the main background of the authors is in NeRF rather than in acoustics/audio signal processing.
- L44: "an intractable number of rays are necessary"  The term "intractable" is not precise enough here. Ray tracing is a widely used technique for acoustic simulation, for instance it was used to generate the SoundScape dataset [25] used by the authors themselves for training their method. In that paper, 200 rays are emitted from each source location. Is that considered an intractable number?
- L47 "NAFs encode and represent an impulse-response in the Fourier frequency domain." -> actually, in the time-frequency domain, which is quite different.
- L60: "The first approach encodes the sound field at a user-centric location by capturing the sound from spatially distributed sources [8,9,10,11]". It is not clear what the authors mean here by "capturing the sound from spatially distributed sources": capturing in what sense? How do those methods specifically differ from the proposed approach? Curiously, the 4 references given seem completely unrelated to each other and span 50 years of literature. The most recent one [11], from 2021, actually seems highly related to what the authors propose, and some more in-depth comparison with this work should be made in the paper.
- L113 "Directly outputting the impulse-response v magnitude" -> this seems incorrect, the term "magnitude" should probably be removed.
- L119 "due to the smooth nature of the frequency space" -> the authors probably mean "time-frequency space".
- L122 "For phase, we use the instantaneous frequency (IF) representation proposed in [29]." : more details on the phase representation used should be provided to make the paper self-contained / reproducible. In particular, one cannot directly reconstruct the phase from the instantaneous frequency, since it is by definition the time derivative of the phase. Hence, a global phase is missing at each frequency. It is not clear how this is handled by the authors.
- L147-150 on local geometric features: It is hard to understand what the authors are talking about here without reading in details the supplementary material. Would be good to make this part more self-explanatory.
- The authors compare an approach using a "shared" feature grid vs. a "dual" feature grid, the former outperforming the latter. However, in the supplementary material (fig. A4), only the network architecture for a dual grid is presented, which is confusing.
- Section 4.1: when presenting the two datasets, the authors omit to discuss their limitations, although both are in fact quite limited and do not allow testing the proposed approach in its full generality. The first one only employs 2D grids while the second one is monaural.
- L217: "The bitrates are chosen to approach the storage costs for our NAF": This is not precise enough to understand how the baselines were implemented exactly. The authors should report results as a function of bitrates for the different methods to get a clearer picture of the benefit of the approach compared to standard techniques. Moreover, Table 3 seems to contradict this statement: here we see that the space consumption of NAF is vastly inferior to that of AAC and Opus.

## Typos
- L44: "an intractable number of rays are necessary to" -> is necessary.
- L179 "on 6 representative scenes. Where" -> remove full stop
- L212 "as well as using have" -> incorrect phrasing.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper describes a NeRF-like representation for binaural room impulse responses in a given acoustic space. It is an implicit representation of listener position, orientation, and ear, as well as source position, time, and frequency that outputs a scalar magnitude and phase. Experiments on the Soundspaces and MeshRIR datasets show that it is able to reproduce realistic effects in loudness maps like sonic shadows from doorways for irregularly-shaped rooms that would otherwise require expensive ray tracing for impulse response simulation. The model's predictions are compared to those of Opus and AAC and shown to be both smaller in space as well as more accurate in terms of T60 estimation and log spectrogram MSE. An ablation shows that a geometric encoding layer that is shared between listener and emitter works better than having separate or no encoding layer. Use of the proposed representation with NeRF for multi-modal representation improves the visual representation when training images are sparse. A further experiment shows that the distance to the nearest wall can be predicted more accurately from the learned representation than from MFCCs by a linear layer.
strengths_and_weaknesses: Strengths:
* Novelty and timeliness: NeRF is very popular and while several papers have conditioned a visual NeRF model on audio of various kinds, this is the first, to my knowledge, that provides a meaningful application of the same idea of implicit scene encoding in an audio context.
* High quality experiments: The experiments are well conducted and thorough. Useful ablations were conducted which provide non-trivial insights, i.e., that a shared geometric encoding works better than having separate encodings for source and listener.
* Clarity: The paper is very well written and easy to follow
* Supplementary material: The supplementary material is useful and informative without being overwhelming.

Weaknesses:
* A few missing details: the paper makes very efficient use of its space, but seems to leave out a few crucial details: the actual architecture of the model being trained, how exactly AAC and Opus were used to encode these libraries of impulse responses, and how the loudness maps are calculated.
questions: What is the architecture of the actual model that is being fit? What are its hyperparameters?

Are the AAC and Opus baselines encoding just the impulse responses? Since they are designed to encode natural sounds and not impulse responses, does encoding a sound convolved with the impulse responses give more accurate results?

In listening to the videos in the supplementary material, I didn't get a great sense of direction of arrival of the sound source. Why might this be? Can this be addressed?
limitations: The limitations section captures salient limitations of the approach and experiments.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposed a model for learning to interpolate room impulse responses given room geometry and emitter/listening position.
Experiments on two datasets demonstrate that the proposed model can encode room geometry information, and that the learned representation can assist in cross-modal learning.
strengths_and_weaknesses: Strengths:

The paper is clear and well-written.  The method appears to work well.  


Weaknesses:


The quantitative evaluations (eg Table 1) look good, but there is no meaningful qualitative evaluation of the method's ability to interpolate RIRs.
The raw numbers are difficult to interpret, even for knowledgable practitioners, let alone the broader machine learning audience.
questions: Minor comment: Figure 3 shows spectrograms both upside-down (e-g) and sideways (a-d) and without axis labels.  Please, for the sake of consistency, stick to standard orientations for this kind of information.  If you're cramped for space, I suggest to simplify to a monophonic visualization as there's no information content in the left/right channels that's important for this particular figure.
limitations: There is no mention of societal impact.  The checklist response claims this will be included in a supplementary document.

Technical limitations are addressed.
review_title: null

Meta-review: metareview: The authors present Neural Acoustic Fields (NAF), which render sounds for arbitrary emitter and listener positions in a scene. Overall, the reviewers are very positive (8-8-8-5). The authors addressed many of the reviewers' questions about previous related work and rendering spatial binaural audio. 
confidence: Certain
Decision: Accept