Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: Sound in a 3D scene depends on the emitter position and the listener position. The spatial acoustic information of the scene is important for having an immersive experience during scene modeling wherein how sounds from a particular location in a scene are perceived in another location in the scene. The authors designed INRAS: implicit neural representation for audio scenes as a way to model the spatial information. Given a geometry of the scene, emitter and listener positions, INRAS learns the Binaural impulse response which is combined with the source sound to render the spatial sound.  The INRAS system has two main components where the first component learns the audio scenes feature decomposition through the scatter, bounce, and gather module. Here the bounce point is any point in the surface of the geometry environment from which the sound can bounce from the emitter to the listener. The second component is the listener module that combines the decomposed feature representation from scenes by fusing the three components via concatenation of the features associated with each bounce point b to predict the Spatial Binaural impulse responses taking into account the head orientation and the location of the left/right ear for the listener.  

The disentangling of the feature representation in the first stage into scatter, bounce, and gather modules allows the model to generalize to multiple scenes with few trainable parameters. The proposed INRAS method outperformed the existing approaches on all the metrics of audio rendering and has greatly improved the inference, speed, and storage requirements since the model require less than 3 Mb of storage capacity with almost a 4-fold improvement in inference speed. Additionally, the authors provide evidence on why the three stages are necessary such as the removal of the bounce module eliminates the static scene feature impairing performance. 
strengths_and_weaknesses: The paper provides a good description of the related works and clarifies how the INRAS method is a light-weighted and efficient neural network module that can generate high fidelity spatial impulse response at the arbitrary emitter and listener positions. 

Strengths:
The model is a lightweight and efficient neural network that produces spatial impulse responses at arbitrary emitter-listener positions. INRAS is capable of modeling continuous implicit function which maps the corresponding time-domain directional and binaural impulse responses of the sound field. 

The training step leverages the fact that the geometry of the scene is static and thus it is possible to learn reusable scene-dependent feature representations which can be associated with the emitter and the listener. The design of the two-stage approach to first generate independent scene geometric features and then during the second stage fuse the features to generate a binaural impulse response is a novel and logical step in building such an acoustic representation.  

The approach generalizes to multiple scenes as the emitter and listener is made aware of scene geometry via the computation of the relative distance to bounce points in the scatter and gather modules. Here the bounce module will provide a static scene-dependent feature representation. The co-ordinate space of multiple scenes is normalized and the bounce points are collected from the different scenes to achieve generalization as the emitter and listener can realize the scene they are a part of. 

Evaluation metrics cover the three aspects such as impulse response quality, storage requirements, and inference speed and on all three metrics the approach outperforms the NAF method. The model has a significantly lower C50 error and has a third of the parameters with less than 3 Mb required to store the model and a 4 fold improvement in inference speed due to the smaller size. Additionally, the method outperforms all other relevant approaches when it comes to generalizing to multiple scenes leading to significant improvement in terms of SNR and PSNR for INRAS system. 

The paper is well written and organized with no/few grammatical errors and the supplementary section provides necessary information complementing the main paper. 

Weakness: 
The approach is dependent on the data and availability of the training data for any given number of scenes and thus limited in terms of scaling the system to any scenario. 

The choice of the bounce points is important as the authors show that when bounce points are chosen with missing some of the boundaries the performance reduces for the INRAS representation as the T60 error and the C50 error increase. This raises the question on what would happen to model performance in a situation where the complete boundaries of the 3D scene are not defined. 

questions: Authors have performed experimentation with INRAS where the optimal number of bounce points mentioned in the paper is 40 to 60 and it is unclear how the authors arrived at this conclusion. Why would the performance of the system drop with the addition of the bounce points? Wouldn’t a system with 100 bounce points extracted from the mesh boundary that covers the geometry of the environment be better? 

How would the modeling be changed/affected when the listener is placed in rooms of different sizes beyond the layouts that are covered in the Soundscapes dataset? How does the current approach take into account the number of obstacles in the room is unclear? 
limitations: The authors provide a limitation for their work where the boundary of the scene should be given as input to the INRAS system which I agree is a primary limitation. The above limitation is not a problem for scenes with 3D models where the boundary can be inferred if not given but the scenes without any geometry and recorded impulse response. The INRAS method would not work in these situations since the bounce points will be impossible to define in situations with unknown geometry.  The method is dependent on the availability of datasets such as SoundScapes which imply the need for INRAS to have a sufficient amount of training data to learn a reasonable acoustic representation of a given scene. This type of data collection is a problem in the case of real-world situations where a large number of impulse responses will be required for INRAS to work. 

With regard to the negative societal impact, the authors highlight a concern where rendered spatial audio can be used to manipulate the original nonspatial sound and create a non-authentic impression of the audio requiring the need for an additional authentication step on the output of INRAS preventing the unethical or illegal use. 

Another concern that I think authors need to consider is whether it is possible to deduce the environment in which the listener is seated where the malicious user may only require the need of spatial sound - the output of INRAS.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this paper the authors address the problem of generating high fidelity impulse responses for acoustic scene rendering for AR/VR and teleconferencing type of applications which rely on faithful recreation of acoustic scenes. The basic idea revolves around decomposing RIR generation into two sets of attributes: i) attributes pertaining to source and receiver locations and ii) attributes pertaining to the room geometry. Features are separately extracted for these attributes using a feed-forward net. Extracted features are then concatenated to estimate binaural impulse responses using another feed-forward net. All the FF nets are trained end-to-end using waveform reconstruction loss and a multi-resolution spectro-temporal reconstruction loss which accounts for reconstruction of magnitude and phase spectrograms separately. The proposed method is compared to audio coding standards and relatively new deep learning based approach for RIR generation (Neural Acoustic Fields - NAF). Experimental results show that the proposed decomposition of attributes and network training is effective in generating high quality impulse responses, reduce storage requirements, fast and can be used to generate IRs in multiple rooms of different shapes.  
strengths_and_weaknesses: Strengths:
1. The idea of decomposing RIR attributes separately for room and source/receiver locations is very interesting. The idea of using Bounce points and the three modules to capture interaction of emitter with the room (bounce points), propagation from bounce points, and receiver with bounce points is also very interesting. 

Weakness:
1. The paper is not very easy to read. I believe the presentation can be greatly simplified. 
2. Goal of the paper is not crisply stated and is scattered throughout the paper. 
3. Purpose of comparing with Audio coding standards is completely lost on the reader. 
4. Abstract is not very informative. 
5. Generalization experiments are a misnomer. Those experiments are mostly multi-condition training than generalization.
questions: 1. At a fundamental level, from the input-output specifications of the network it appears that the function that the network is trying to approximate is a one-to-many function as for a specified input - of emitter+receiver locations and bounce points, acoustic conditions can be changed by changing the reflection co-efficients of the walls. So are these networks trying to memorize a specific enclosure ? In this case it is not generalizable to other rooms with exact same geometry but with different reflection characteristics ?
2. Would it be possible to assess the generalization capabilities of the network in the true sense by generating RIRs of unseen environments ?
3. Was the point of comparing the proposed approach with AAC and Opus to say merely storing discrete IRs and interpolating will not help ? 
4. Generically stating 40-60 bounce points would suffice is not making sense to me. Is this not dependent on room geometry in general and size in particular ?
5. in line 226, shouldn't \alpha_k bea function of i ?
6. In Fig 3, d_{b_{i}}^s is this a vector of distances or difference in position vectors. Dimesnions of these inputs should be clarified. Similarly for the gather modules. 
limitations: A few limitations are raised in Questions section. Based on authors' response I will edit this section at a later stage.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes to use implicit neural representation to represent spatial audio in the scene. It renders high fidelity time-domain impulse responses at any arbitrary emitter-listener positions using neural network parameterization. Experimental results demonstrate that INRAS outperforms existing methods across a series of metrics.
strengths_and_weaknesses: Strengths:

- The paper is generally well-written and well-motivated. It nicely reviews prior methods of scene acoustic modeling: wave-based vs. geometry-based. The proposed method is motivated by prior methods on interactive acoustic radiance transfer, and proposes a neural network-based approach to mimic their settings.

- Dividing the whole pipeline into three sub-steps: scattering from the emitter, acoustic transfer, and gathering at listener makes sense. The design of INRAS well captures the physical process of how sound is propagated from the emitter to the listener.

- Compared to the baseline method NAF, the proposed method has large gains.

Weaknesses:

- The proposed method is motivated by prior methods on interactive acoustic radiance transfer, which is nice. And it is cool to represent spatial audio using a neural representation. However, what is the use of it and why a neural implicit representation is better is not discussed. The neural network basically tries to model how sound is propagated from the emitter to the listener. Why not just directly doing geometry-based audio simulation, for example bi-directional path tracing, to obtain the room impulse response. What is the motivation to use an implicit neural representation to replace the spatial audio simulation? Bi-directional path tracing can also be made real-time with some simplifications.


questions: 1. It is claimed "the proposed method renders high fidelity time-domain impulse responses at any arbitrary emitter-listener positions using neural network parameterization", but SoundSpaces only provides impulse responses at a discrete set of grid points in the spatial environment. How the evaluation is done at the points where there is no ground truth?

2. See weakness.

3. How sensitive is the performance to the number of bounce points used?
limitations: Yes, there are discussions of limitations and potential negative societal impact of the work in the supplementary materials.
review_title: null

Meta-review: metareview: This is a technically good paper, with some flaws. Parts of the paper are hard to read.   Several questions remain, e.g. how to determine the optimal number and location of bouncepoints, and how they depend on room layout and content.

The motivation behind some of the comparisons, e.g. to AACs is unclear.

Regardless, the overall paper presents a well-defined novel idea, and represent a significant contribution.  The authors have also addressed most of the reviewers' comments satisfactorily.

I am recommending that the paper be accepted. 


confidence: Certain
Decision: Accept