Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a novel approach to predict Room Impulse Responses in a few-shot learning setting, where only very limited data is available. More specifically, only a small number of echoes obervation in the room along with RGB pictures of the room. The proposed model is a multi-modal transformer-based model, taking the RIR of a given location or an image as input, along with a pose encoding and a modality encoding. The model is trained with two losses, a fairly standard L1 loss and a novel energy decay loss which help the model focus on the most important part of the RIR. The model is trained and evaluated using synthetic data and compared with baseline and state-of-the-art models. The proposed model is shown to yield better perfomance on seen and unseen rooms. 
strengths_and_weaknesses: Strengths:
- The approach is entirely novel as far as I can tell.
- The experiments are sounds and clearly show the benefit on the proposed approach. The ablation study is a nice addition.
- The results are significant, as the proposed model yield better performance than state-of-the-art models, even one being published this year.
- The paper is clearly written and well structured.

Weaknesses:
- The only drawback is that the experiments are done on synthetic data, not real measurements. But as the authors discussed, there seems to be no available dataset for that. 

Overall the approach is very neat, very interesting, completely novel as far as I can tell and quite significant. Even if the field is very specific (RIR), I think the paper is worth publishing at NeurIPS as the approach is interesting and could be applied to other tasks.
questions: - The hyper-parameters of the transformer-based Encoder and Decoder are not documented anywhere (or I missed them), could the authors provide them?

limitations: The limitations are not discussed in the paper, the authors should discuss the issues of real life application, typically the data could be noisy, either the echo audio measurement (street noise) or the pictures (bad lightening, occlusion, etc).
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper introduces a transformer-based method that uses self-attention to build a rich acoustic context, then predicts RIRs of arbitrary query source-receiver locations through cross-attention. Additionally, a novel training objective is proposed to improve the match in the acoustic signature between the RIR predictions and the targets. Experiments show that the proposed method outperforms prior works. 
strengths_and_weaknesses: Strengths

- This paper is well written and easy to read
- The performance is good. 
- The training objective is novel

Weaknesses

- The transformer-based model is lack of novelty as it just contains normal architecture design with attention. 
- The claimed "few-shot learning" is confusing, as the method uses a lot of data for training. I understand that there are a few observations for each sample so it is called "few-shot" in the paper. But I am not sure whether it is suitable to call the task "few-shot". 
questions: See the weaknesses
limitations: The limitations and negative impact are discussed in the supplement.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: A typical way to render 3D audio simulating sounds emanating from specific directions is by means of convolving location-agnostic clean audios with binaural room impulse responses (BRIRs). This is useful for:
1. AR/VR applications 
2. Augmentations for training Neural networks which generalize well across different acoustic conditions. 
In this paper, the authors come up with a novel approach to incorporate the contextual information necessary for estimating RIRs in unseen environments for arbitrary placement of source and receiver. Essentially the idea is to use audio-visual information from an ego-centric  90 deg field-of-view (FOV) camera, RIR from a specified source location assuming receiver (microphone) is at the origin from N arbitrarily sampled spatial locations to create a contextual embedding. In the test phase while generating the required RIR for a specified source and receiver location, This contextual embedding is used with a query embedding (which incorporates the information about source and receiver locations) to generate an RIR-embedding which is dilated to generate Binaural RIR magnitude spectra. 
strengths_and_weaknesses: Strengths:
1. The idea is very interesting. 
2. the paper is well written easy to understand. 
3. The experiments are indeed insightful with relevant baselines. The authors explanation of results for each experiments is appropriate and concise. 

Weakness:
1. It is not clear how the authors convert the Binaural magnitude spectrogram of RIRs to waveform for construct signals for downstream source localization tasks. This conversion in itself could affect some RIR properties and it is not clear how this is addressed in this work. 
questions: 1. Why is the proposed approach expected to work in unseen environments ? More precisely say the network has always been trained on images from enclosures with smooth reflecting walls. How would this network learn the impact of rough walls where scattering is pre-dominant ?
2. What in the proposed approach helps the network learn difference in reflecting properties of different surfaces ?
limitations: 1. Visually similar looking surfaces with different reflection properties could be challenging for this proposed approach leading to unsuitable RIRs ? This is definitely not a societal impact but a technical limitation to consider and address.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a new method for RIR prediction given previously recorded RIRs and images for different locations in the scene. The authors propose a new architecture which fuses audio and visual encoders, followed by a transformer and a decoder. They use two loss functions, L1 and a novel Decay Loss which solves the non-differentiability of RT60.
strengths_and_weaknesses: Overall the paper is quite good. It is well written, and explains the motivation of different design choices and experiments. The included video was quite nice, especially the section on limitations. It showed the performance compared to ground truth, including a real-world environment which was not simulated. The experiments show comparisons with other baselines, such as nearest neighbors which is an important baseline. The novel loss for RT60 is a very welcome contribution, since L1 loss is suboptimal when trying to model reverb decay.

My main concern is that the problem formulation is not widely applicable and may not be too useful. In general, asking the user to capture ground truth RIR in multiple locations of an environment is difficult, and requiring images at those locations also make it hard. The authors claim that an advantage is not needing images at the new location for inferring RIR, but collecting an image at the new location is often easier than collecting ground truth RIR and images in multiple locations. Given that there are many papers which do image -> RIR, the contribution of this work is somewhat limited. The main application I could imagine is for robots navigating a space which have the ability to collect ground truth RIR.
questions: Can the authors explain the applicability of this precise method? Is there a specific setup where this way of solving the problem is optimal compared to image -> RIR, specifically where you might need to model source/emitter positions with such fine grained accuracy? A more detailed discussion of the advantages of this method would be useful in the paper. 
limitations: The authors briefly mention the collection of RIRs during inference by playing a short sweep. I think this process should be addressed in more detail, for example do you need a flat frequency response microphone (or does it generalize), how does the loudness of the emitter factor in, etc. It's quite likely that collecting these ground truth RIRs is difficult, especially if you don't have access to the environment at a perfectly quiet moment. This means that making this work in more realistic scenarios such as public places may not be reasonable.
review_title: null

Meta-review: metareview: Reviewers are in agreement that the paper should be accepted, and the authors were able to address concerns leading to an increase in score from one of the reviewers.
confidence: Certain
Decision: Accept