Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an neural acoustic field (NAF), which is an extended version of neural radience field applied on impulse responses defined between listener and emitter positions. It predicts an impulse response given the listener and emitter positions so that one can simulate the impulse response in arbitrary positions. One of the key idea to extend the model to unseen positions is to take local grid features as input. The authors show other applications with NAF such as source localization and multi-modal NERF. Experiment results show that the proposed method shows better results when comparing to baseline methods.
main_review: Strengths:
The proposed idea is novel and expected to promote other related researches in the future.

Weaknesses:
I think the details written in this paper is not enough to reproduce the results.
1. How can one listen to the predicted impulse response when the estimate values are only the magnitude parts?
2. How were the visualization done in Figure 1 and 4?
3. What happens on the local grid parts where the impulse response was not collected enough if not at all? I assume those parts will not be trained and results in poor prediction. Is this correct?
4. in section 4.2, the authors say something like "processed with sinusoidal encoding", but the details are not enough.
summary_of_the_review: The idea is seemingly novel but it is not enough for publication in this status.
I recommend reject. But I'm willing to raise the score if the authors answer the questions with more details and write them in the paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper makes use of the NeRF idea on RIRs (coined here as NAFs).  The main idea is relatively straightforward, sample a room from multiple emitter/receiver locations and learn to predict the resulting RIR using a neural net that receives as input only the involved locations, and some positional information.  It is shown that this approach learns something, and it is used in a few demonstrative examples such as sound localization and enhancing NeRF estimation.
main_review: Strengths:  This paper shows a concrete methodology on how one can learn a NAF.  The use of a spectrotemporal representation, the particular way of representing positional information, the use of a local feature grid, and the architecture used, are interesting to know for anyone wanting to replicate that work.

Weaknesses: There are a lot of questions that arise in this work, and I was disappointed to see space being used for unnecessary fluff (wave equation?!!?) as opposed to a deeper investigation.

-- The choice to use a spectrotemporal representation is interesting, but it isn't clear why it works.  If the goal is to represent an RIR, a spectrogram would take at least as many values at the time-domain representation (actually more if you use windowing, which one should).  Why it is that it works better than using the time-domain data?  It isn't clear from the writing, but I assume that by "log spectrogram" you mean log( abs( STFT)) and not log( STFT).  If this is the case then you are not learning a room RIR, but something far simpler, which might explain why the problem becomes more tractable.  Is this the case?  If so, why would anyone care to estimate that quantity?  It's certainly of no value to any realistic audio problem, and doesn't resolve the issue of estimating the RIR which is what you start with.  Linked to this, comes the question of the loss function.  What is the MSE of the log spectrogram of an RIR, and why would anyone care to optimize it?  Clearly a lot of the most useful RIR information is in the phase component.

-- The experiments are really weak.  I think the most solid experimental point is figure 5, which conclusively supports the use of geometry conditioning.  But the experiments in sections 4.4 and 4.5 seem very contrived.  Of course the NAFs will help the NeRFs since they are smoother functions (due to ignoring phase), and the localization task used is just unrealistic.  I don't see how these experiments strengthen any point in this paper, they are simple more elaborate variations of your MSE estimate.

Minor things: "in principal" ⟶ "in principle"
summary_of_the_review: Just as one can learn NeRFs, the authors show that it is possible to also learn (some version) of RIRs.  Representing RIRs is a big deal, and this paper can potentially be the first one published to use this approach.  However, had this been submitted to an audio conference I predict that it would be summarily rejected.  Unless I misunderstood your representation of the RIR, the problem solved is one that is made to be solvable, but not particularly useful for any real problem.  The log spectrogram of an RIR is not a quantity that one would employ in any audio processing, and the degree of approximation shown in figure 3 is way off from a high fidelity result (alas there were no qualitative listening experiments in this paper).  Although I appreciate the initiative, I would have expected a lot more depth in the first neural RIR paper, and this is coming short by a lot.  It's nice to see that a simpler version of this problem is solvable, but I do not see a useful contribution here.

If, on the other hand, I'm mistaken and you are actually estimating the phase as well, then I would like to see some results with more realistic audio experiments and potentially some quantitative listening numbers.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces the concept of Neural Acoustic Fields, which is an implicit representation to capture how sound propagates in a pysical scene. Using a NeRF-like network architecture, the model is able to predict room impulse response given the listener and source locations as input. The idea is new and interesing, but there are multiple overclaims in the paper. Some interesting tasks are demonstrated like sound source location, cross-modal generation, etc.
main_review: Strengths:

- The idea of using a neural network to represent the acoustic fields of how sound propagates in space is new and interesting.

- The paper is well motivated and it makes a lot of sense to represent neural acoustic fields or sound propagation as a neural network, which is differentiable.

- Leveraging what is learned in the neural acoustic fields neural implicit representation, the paper demonstrates several tasks that are pretty interesting, including cross-modal generation and sound source localization.

Weaknesses:

- First of all, although this paper is well-motivated and set the stage to introduce Neural Acoustic Fields, which is an implicit neural representation that captures how sound propagates in a physical scene, actually the network only *remembers a dataset*. The background of the room impuse response and environmental reverberation is nice, but the dataset of room impulse responses is captured/introduced in another paper (Chen et al. 2020), which is not the contribution of this paper. The neural network that this paper introduces is basically a NeRF network and the paper trains such a network to remember the dataset. Fm the introduction of the paper, it sounds like the paper is going to introduce a differentiable neural network design that can model ray tracing and sound propagation in a fully differentiable way, but actually it's not.

- There are multiple places the paper overclaims its contributions: For example, in the end of Audio Field Coding in related work, it is claimed that "the work allows a listener to move and experience sounds that come from anywhere in a scene and can represent the acoustic field continuously at high fidelity by directly learning from data". This is overclaimed, because this is not enabled by the design of the proposed neural network. It is enabled by the dataset, which uses simulation to model room impulse response. The paper only uses a network to remember this dataset.

- Related to the discussions above, the paper didn't discuss at all why it is necessary to use such an implicit representation. Why not just directly use the dataset instead? 

- Providing grid features near both emitter and listener positions as additional context seems like a very strong assumption. It provides much extra info to the neural network.

- For equation 6, it is mentioned v is a single scale value taht represents the intensity. But isn't *v* should be the room impulse response? This is very confusing.

- For figure 4, it would be great to show sound ground truth from the dataset for comparison. I assume these maps can also be directly generated using the dataset.

- The writing of the paper can also be further improved. There are multiple places in the paper that are not clear or need more details. Here are some detailed questions:"

1) More descriptions are need on the local feature grid part. Currently the descriptions are very vague. It is not clear what exactly is the local features and what is used to represent them.

2) For Equation 3, it is unclear what theta and k is. Although it becomes clearer later when introducing the dataset, but the dataset is modeling a grid in 2d. So x and x' only have 2 dimensions instead. This should be clarified and discussed.

3) The RGB only baseline in Figure 6 is unclear and there is no description of the baseline. 

4) How joint training is done for cross-modal image generation is also unclear. What the NeRF baseline is trying to encode?
summary_of_the_review: In general, this paper studies an interesting problem and presents some preliminary attempts. But overall, the proposed method seems just another way to represent an existing dataset from another work. The paper overclaims its contribution and there are multiple places in the paper that needs more clarifications or analysis or writing improvement. Therefore, I recommend reject of the paper at this stage but happy to discuss more.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The manuscript considers the problem of encoding room impulse responses in a room. Specifically, the authors propose a network that takes in some geometrical cues from the room, along with speaker, listener positions, and train a neural network using pre-recorded room impulse responses. They argue that the neural network learns to make use of the geometrical cues, and can generalize beyond the dataset.
main_review: The manuscript rightfully points to the importance of being able to roughly deduce the scene geometry from monaural recordings. To address this challenge, the authors propose a neural network architecture that not only takes into account the speaker and listener positions in a room but also some geometrical information as well. Overall, while I do think this is an important and challenging problem, I think the manuscript falls short of what it promises. I also found the manuscript unclear about important details at times. Overall, it should do better in terms of presenting convincing evidence to show that the neural network does more than learning to interpolate a smooth field (since log magnitudes are the only outputs of the neural network).

Here are some more specific comments : 

1. Introduction, third paragraph : I agree that being able to infer scene properties given monaural recordings is something humans do, but I think the proposed method is a rather modest step in this direction. After all, for each individual room, you need to have many prerecorded room impulse responses to train the neural network. When you enter a totally new room (with no previous RIRs available), it's not clear to me how one could use the proposed neural network to infer the shape of the room. Could you elaborate?

2. Page 2, "... we propose to condition NAFs on the local geometric information present at both the listener and emitter locations..." : I understand you provide a grid with the emitter, listener locations, as in Fig 2, but how do you form the grid exactly -- you write about this verbally, but can you be more specific about the details?

3. eqn (3) : It's interesting that you encode binary information (which ear) as a real. I'm curious, what happens when you input $k = 0.5  (k_{\text{left}} + k_{\text{right}})$?

4. In eqns (3), (4), please indicate the dimensions of the components (i.e., $x\in\mathbb{R}^3$, $\theta \in \mathbb{R}^2$ etc.)

5. It's a bit confusing to say that $\Phi$ in eqn (4) is your NAF -- it's the version without geometric features, isn't it? I'd suggest declaring another function, and clearly stating the input dimensions of the various arguments, so as to separate it from $\Phi$.

6. For $\Phi$ in eqn (4), you try to predict the logarithm of the magnitude -- how do you recover the phase? Isn't that an important part of the room impulse response?

7. Section 4.3, "...pad the training impulse response with 0.1 chance." : What does "padding with a chance" mean? I thought you'd zero pad short impulse responses so that the length of every impulse response is the same.

8. Section 4.3 : How do you select the test set? Do you randomly sample the available RIRs? Or do you hold out a specific region entirely? The latter would be more interesting in terms of evaluating the generalization capability, touched upon in the third paragraph of the Introduction (see my comment-1).

9. Eqn (8) : Isn't the idea to obtain $p_i$ from the NAF? How do you deal with the lost phase of the impulse response (see comment-6)?

10. Section 4.5, baselines : I understand you minimize eqn (8) over $x$, right? If so, for the nearest neighbor method, are you doing a search over the available room impulse responses? I don't understand why you'd choose the loudest. Please clarify.
summary_of_the_review: Overall, the manuscript considers a very interesting problem, but I am not convinced that the proposed method would be sufficient to provide a solution as implied in the Introduction. Specifically, given a large database of room impulse responses, the neural network appears to learn how to store a smooth field based on those values (since log magnitude is the target), but it's not really clear how the geometrical side information could be useful for generalizing beyond a room. I also found the paper to be a light on some of the details, which I think are critical.
review_title: null

Meta-review: This paper proposes to use implicit neural representations to model how our surroundings affect the sounds reverberating within. Concretely, the proposed approach can produce impulse responses that capture environment reverberations between any two points in a scene.

Reviewers praised the novelty and originality of the idea (and I concur), but raised concerns about the clarity of the writing (especially w.r.t. modelling the phase component), lack of detail, insufficient or inadequate experiments and overclaiming of results. (There were also concerns about overclaiming of contributions, but I am inclined to agree with the authors that this isn't really the case.)

The authors have clearly taken the time to try to address these concerns, and I commend them on their willingness to engage with the reviewers' comments and suggestions. While one of the reviewers raised their score to "accept", I am inclined to agree with the other reviewers and recommend rejection. The required degree of revision is substantial, and therefore difficult to assess within a single review cycle. I believe this work must undergo another thorough assessment in its revised form, before it can be accepted for publication.
Decision: Reject