Novel View Acoustic Parameter Estimation

Authors: Ricardo Falcon Perez, Ruohan Gao, Gregor Mückl, Sebastia Vicenc Amengual Gari, Ishwarya Ananthabhotla
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: acoustic parameter estimation, room impulse response, reverberation, spatial audio, room acoustics, novel view acoustic synthesis

Abstract:
The task of Novel View Acoustic Synthesis (NVAS) -- generating Room Impulse Responses (RIRs) for unseen source and receiver positions in a scene -- has recently gained traction, especially given its relevance to Augmented Reality (AR) and Virtual Reality (VR) development.  However, many of these efforts suffer from similar limitations: they infer RIRs in the time domain, which prove challenging to optimize; they focus on scenes with simple, single-room geometries; they infer only single-channel, directionally-independent acoustic characteristics; and they require inputs, such as 3D geometry meshes with material properties, that may be impractical to obtain for on-device applications.  On the other hand, research suggests that sample-wise accuracy of RIRs is not required for perceptual plausibility in AR and VR.  Standard acoustic parameters like Clarity Index (C50) or Reverberation Time (T60) have been shown to capably describe pertinent characteristics of the RIRs, especially late reverberation.  To address these gaps, this paper introduces a new, intermediate task centered on estimating spatially distributed acoustic parameters, that can be then used to condition a simple reverberator to generate RIRs for arbitrary source and receiver positions. The approach is modeled as an image-to-image translation task, which translates 2D floormaps of a scene into 2D heatmaps of acoustic parameters. We introduce a new, large-scale dataset of 1000 scenes consisting of complex, multi-room apartment conditions, and show that our method outperforms statistical baselines significantly.  Moreover, we show that the method also works for directionally-dependent  (i.e. beamformed) parameter prediction.  Finally, the proposed method operates on very limited information, requiring only a broad outline of the scene and a single RIR at inference time.

Decision: reject
Reviewer scores: 6, 6, 3

Reviews and discussion:
<thread>
creation date:
1730496475389

thread_type:
official_review

<thread_object>
creator: reviewer_uqps

<content>
summary:
This paper proposed a novel novel view acoustic parameter estimation framework by learning a floor plan like acoustic heatmap. Instead of requiring to know full 3D geometry, this framework uses a 2D floormap and room impulse response (RIR) for inference, translating them into heatmaps representing acoustic parameters. Apart from this task, a new dataset with diverse apartment layouts was used to demonstrate that the proposed model outperforms baselines, offering efficient parameter predictions for varied source-receiver positions in complex, multi-room scenes​. The new dataset is built from SoundSpaces 1.0 simulator, requiring highly dense RIR sampling.

strengths:
1. This paper discusses a challenging yet important task -- estimating acoustic parameters.

2. The introduced large-scale new dataset for future exploration (although I have concerns regarding this dataset)

weaknesses:
1. **Motivation**: While it seems natural to use an image-to-image conversion to learn acoustic heatmap from 2D floorplan map, the motivation of using 2D floorplan as input to learn acoustic parameter isn't convencing. First, 2D floorplan input used in this paper is a degragated representation of the 3D room scene. Furthermore, this paper uses small-resolution (128x128) 2D floorplan as input without explicit room constructional detail. Essentially, the acoustic heatmap's goal is to predict RIR for arbitrary source and receiver position, I didn't the see the need to format it as floorplan like map. Second, obtaining 2D floorplan map is a challenging task in real-scenario, which makes the argument "learning from floorplan map" less practical in real world where the 3D mesh of the a room scene is usually missing. What is the advantage of using 2D floorplan to learn acoustic parameters?

2. **Acoustic Parameter Definition Unclear**: throughout this paper, it claims the task is to estimate the acoustic parameter for novel view, but in essense it estimates RIR. Equalizing RIR with acoustic parameter is confusing, as acoustic parameter usually indicates all relevant characteristics for a room scene that govern sound propagation behaviour, such as room layout, construction material and furniture. RIR is the overall resultant of all the acoustic parameters. To me, this work estimates the RIR, not directly the acoustic parameter. What is the concrete definition of acoustic parameter in the context of this paper? How is it related to RIR?

3. **Acoustic Heatmap**: Based on acoustic heatmap visualization in this paper, it looks like loudness map that depends on the source emitter position (Figure 1 and Figure 3 in this paper). How are all acoustic parameters encoded in the heatmap needs more clarification. The acoustic parameters should be independent on if there is any source eimitter. If so, how to understand the acoustic parameter via heatmap perspective? How does each channel of feature in the learned heatmap related to acoustic parameters?

questions:
1. Read the weakness section.

2. L220-222: These parameters are computed from RIRs captured at a discrete set of receiver locations. **What are these parameter?**

3. Equation 4, 5: I read this paper a couple of times but still confused, the input is the stack of 5 maps (Fig. 3), what is the corresponding ground truth heatmap? **How to obtain the acoustic heatmap during training?** This echoes my confusion of the definition of acoustic parameters heatmap.

4. **MRAS dataset**: L320-323, it writes: it includes a large collection of scene geometries. What collections? It is the same dataset of SoundSpace 1.0 introduced dense RIRs on Replica or Matterport 3D room scene?

5. **Experiment**: this paper writes in L62, previous work is limited by poor generalization to new scenes. But the experiment didn't explicitly discuss the generalization capability of their framework. Moreover, experiment on real-world data is missing, which makes the framework less convencing.

6. **Missing relevant work discussion**, e.g. [1] Y. He, et al., Deep Neural Room Acoustics Primitive, ICML24. [2] Das, O. et al., Room impulse response interpolation from a sparse set of measurements using a modal architecture. ICASSP 2021.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730534320038

thread_type:
official_review

<thread_object>
creator: reviewer_tnwt

<content>
summary:
The paper introduces a new task, Novel View Acoustic Parameter Estimation (NVAPE), which can be then used to condition a simple reverberator for arbitrary source and receiver positions. The motivation of bringing this task out is to address existed challenges such as poor generalization to new scenes, poor handling of complex geometries, ignoring of directional dependencies, and strict constraint on the input for the previous acoustic modeling methods. As solution, the paper proposed an image-to-image translation style framework to translate floormaps into heatmaps of acoustic parameters. They claim the proposed method can outperform statistical baselines and work for directionally-dependent parameter prediction, requiring only a broad outline of the scene and a single RIR at inference time. A new, large-scale dataset of 1000 scenes consisting of complex, multi-room apartment conditions is also proposed by this paper as the benchmark.

strengths:
+ This paper proposes an interesting task that estimating acoustic parameters given the layout and one reference impulse response. The paper proofs it's feasible to use an image-to-image way to correlate the floormap with the acoustic heatmap and is able to generalize over new rooms as well. It's insightful and potentially can help with other acoustic-visual tasks.

+ The paper introduces a challenging dataset of 1000 scenes consisting of complex, multi-room apartment conditions. The dataset can be good resource for the acoustic learning community.

+ Experiments and visualization analysis show the effectiveness of the proposed method. Details of dataset creating and method implementation are clearly described.

weaknesses:
- Problem Scope. Previous acoustic modeling methods usually output the impulse response wavform, so they can convolve with source sound with the impulse response to render virtual sound that matches the target environment and view pose. However, how to bridge the predicted acoustic parameter with the sound rendering is not explained in this paper. If not targeting on the sound rendering, regardless of presenting this as main application in the introduction, what can we use the parameters for? If sound rendering is indeed the final target, it will be better to include some results, conditioning on the acoustic parameter to generate IR and even target sound, comparing with baselines.

- Fair Comparison Concerns: In the Table 2, the paper compares the proposed method with baselines INRAS and NAF. Though it shows largely accuracy boost, the fairness seems to be an issue. It mentions the acoustic parameters are obtained from predicted IR of INRAS and NAF. However, for fair comparison, we can also compare with these methods by adapting their output from IR wavform to acoustic parameters as well (aligned with the proposed method), at least for INRAS and other baselines such as FewshotRIR [1] that support multi-scene learning. After supervised training and then compare will be more fair. Instead of using an image2image way conditioned on layout and reference IR, these methods rely on modeling the relationship between the target location and references (bounce points or reference locations). By comparing like this, we can better show the advantageous of the proposed method on learning the acoustic parameters. 

[1] Few-Shot Audio-Visual Learning of Environment Acoustics. Majumder, S et al.

questions:
Please check the weaknesses section for most of the concerns.
For the task, shall we call it "novel view" acoustic parameter estimation? Since it's only location based, and not condition on any target orientation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730573800669

thread_type:
official_review

<thread_object>
creator: reviewer_y9ru

<content>
summary:
The paper explores estimating room impulse response (RIR) given a spatially distributed pair of sources and receivers' positions.
In particular, the paper proposes a novel way of predicting acoustic maps in the formulation of image-to-image translation, given floor maps and acoustic parameters as input. The proposed method has been evaluated in two datasets, namely SoundSpace and MRAS, constructed to reflect realistic multi-room scenes of typical indoor apartments. Five existing methods have been compared to the proposed method.

strengths:
I find the formulation of RIR estimation into image-to-image translation interesting and novel. The paper is well structured and clearly addresses the problem and limitations of the existing approach. The evaluation result is convincing, and the proposed dataset also seems to reflect a more realistic environment of common apartments than the existing dataset, SoundSpace.

weaknesses:
The authors claim that the proposed method predicts acoustic parameters of 'unseen scenes'. Although I understand the context, it sounds like an overclaim. The proposed model takes a floor map as a geometric input, which requires a full 3D mesh of the scene. Given that the floor map, which requires significant work to build, is already given as input, I feel that it is not an entirely unseen scene. It would make the purpose of the paper clearer for readers by correctly addressing what the proposed aims to do in this regard. I would like to hear what the authors think about this.

questions:
(1) Regarding the weakness that I mentioned, how costly is it to extract the full mesh of the scene to generate the floor map? 

(2) The author mentions that the floor map is constructed by slicing at certain heights. How were the certain heights modelled? What is the impact of the different slicing heights?

(3) What are the parameters of low-pass filters?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper investigates the problem of novel view acoustic parameter estimation and proposes a method for novel view acoustic synthesis, that is, an intermediate representation from which Room Impulse Responses (RIRs) for unseen source and receiver positions in a scene can be generated. 2D spatially distributed acoustic parameters for an unseen scene are estimated casting the problem as an image-to-image translation task, just using in input a simple 2D floormap and a reference RIR. A new, large-scale dataset of 1000 scenes consisting of complex, multi-room apartment conditions (different source-receiver positions) is also proposed as benchmark.

This paper received contrasting reviews: a clear negative (3) and two marginally positive (6, 6) ratings are the scores originally assigned and also maintained after the authors' rebuttal. 

Positive points relate to the original formulation of the problem, good explanation of the methodology including its limitations, and convincing experimental evaluation, including the proposal of the new dataset.  

Negative points address the unclear actual scope of the problem, also related to the experimental analysis, namely, how to ultimately prove the goodness of the method. Other issues still regard the experimental analysis, specifically, about the unfairness of the comparative tests and the visualization of the acoustic heatmaps. In a similar vein, it was also argued about the acoustic parameter specification and estimation re RIR generation, that is, how much these parameters are suitable to lead to a correct estimation of the RIR.

The authors' rebuttal replied to all main comments, but did not move the original ratings: it addresses satisfactorily some points but leaves other unclear.

AC has read the reviews, the rebuttal and the discussion and, despite the majority of slightly positive ratings, he is more in line with the most critical reviewer.
In particular, the question about the relation between RIR and acoustic parameters has been tackled by inserting an experiment in Appendix F, but the final performance was just "informally" evaluated, and this is not sufficient. In general, the actual scope of the work in relation to resulting virtual sound rendering (as the result of the convolution of the impulse response and source sound) in novel views, does not seem well addressed by the authors' discussions. The raised issues regarding the use of solely a 2D map and a reference RIR (much less info than former methods), which is also quoted as one of the main points of the works, has been discussed and might also be reasonable as long as one can demonstrate that the final goal is reached (virtual audio rendering), but this does not seem to be sufficiently considered. Yet, it results unclear how that a 2D map (and a reference RIR) used for parameter estimation can consider the complexity of an environment, e.g., the room ceiling, which is actually an important part to consider in the acoustic propagation. 
Also the fact that some important acoustic parameters are not considered in the estimation is not helping the overall narrative at the basis of this work.

For these reasons, this paper cannot be recommended for acceptance at ICLR 2025.