Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper presents a platform for audio-visual simulation that can be used for audio-visual rendering. The paper also presents a dataset called Sound-spaces-PanoIR for acoustic-visual learning tasks. The simulation platform has multiple configurable parameters.
strengths: 1. The platform can be very useful for the research community as it supports audio-visual rendering and can be used in any environment.
2. The platform supports acoustic and spatial continuity and has multiple configurable parameters like frequency bands, diffraction, e.t.c
3. The authors have shown results for multiple down-stram tasks such as far-field speech recognition.
weaknesses: 1. The baseline could have been improved.
2. The authors addressed improving the accuracy of the direct-to-reverent ratio (DRR) by fixing the bias of √4ℼ present in the original Soundspaces. However, no discussion is presented on this in the paper.
3. There is no discussion on the impact of reducing the number of rays in high-speed mode. How does the reduction in the number of rays affect the performance?
4. The sub-section 5.2 compares the simulation accuracy with Real IRs using the FRL apartment from the Replica dataset. The comparison should be made in more diverse settings (like crowdy apartments, environmental location).
correctness: Probably correct
clarity: Lack of clarity at multiple places. It could have been written better. There are a few typos, such as line no 49. 
relation to prior work: Yes
additional feedback: No
review_title: A platform for audio-visual simulation

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors present a second-generation of the soundscape tool, which is designed to simulate realistic acoustic reverberation from 3-dimensional room/scene models, such as are already used for rendering computer graphics. Such a tool facilitates large-scale rendering of audio for simulated scenes. The authors have already run this simulation at scale and release a data set of 10-million pairs of rendered images with corresponding acoustic impulse-responses (IRs). 

In addition to obvious uses in computer game audio rendering, such a tool may aid in the development of machine hearing algorithms, like speech recognition, which often show pathological behaviour in the presence of reverberation that humans would consider very moderate. 

Of course, the use of such a tool, and associated simulated dataset, is predicated on the simulation accuracy (which, itself, relies on the accuracy of properties of the user generated scene model). To demonstrate the accuracy of soundscape 2.0 IRs, the authors show two demonstrations that appear successful, but in this reviewer's view, are not as strong as they could, or should, be.  This does not mean the work isn't useful, but the paper should certainly acknowledge the shortcomings directly and clearly. 

Additionally, the authors present two additional benchmark tasks. In one, a simulated agent must localize a sound in a complex simulated environment. Here the others demonstrate that soundscape 2.0 improves upon the first generation soundscape in conveying (simulated) source location via (simulated) sound.  The other benchmark compares two soundscape 2.0 simulations of reverberation for one room, one made with "high-quality mode", and the other with "high-speed mode".  While these two benchmark tasks are not without merit, they do seem to belong in a different category than the tests that involve real-world audio.  I believe mixing all four tasks into one section on "Evaluation and benchmarks" obscures an important difference: the claims made with entirely simulated tasks are well justified, but those about accuracy are not.   

The authors deserve great credit for tackling a relevant problem, and providing tools that allow audio rendering to be run at scale with pre-existing scene simulation tools, as is already common with image rendering. The synthetic reverberation might even prove to be accurate, however, the task of validating this is more difficult than suggested by this paper, and that should be addressed. Moreover, there is no reason to think high-quality reverberation can be simulated with a flawed 3D-scene model, no matter how good the simulation algorithm, and the paper (and documentation) should warn users of pitfalls.
strengths: The codebase presented here provides a useful tool for synthesizing acoustic reverberation using mesh-grids formatted for common scene simulation tools. The codebase improves upon its first generation (soundscape) by allowing continuous sampling of locations for the sound source and receiver. This almost certainly results in more accurate changes of sound with listener location, which is plausibly why simulated agents can better localize sources in such an environment.  Spatial audio in VR environments may well also benefit from this update.

The authors have provided evidence of sim2real transfer for a speech recognition algorithm trained on synthetic reverberation. I think their test could have gone farther, as I will elaborate upon later, but nonetheless this first attempt, and the public release of the tools to build thereupon is laudable.   
weaknesses: The authors claim in the abstract that the simulated reverberation is "highly realistic". They might be correct, but I find the evidence for this disappointingly thin. None of the issues I bring up here are cause for immediate rejection.  Ray-tracing algorithms have a tremendous role to play in acoustics research and this tool will likely be useful in many research projects. Moreover, given the dearth of large-scale datasets of reverberant IRs, perhaps limited models are better than none. However, so that other researchers can make good use this toolset, these limitations should be clearly stated in the main text of the paper.

The major issues I find are as follows:

- The direct comparisons between simulated and recorded reverberation are limited to only one room.  While it is certainly true that real-world IR measurements are not easy, 3 measurements each in 2 rooms, would have been better than 7 measurements in one room.  Moreover, a great many past research papers on reverberation have measured 3--10 rooms, and some many more.  Time-consuming is not the same as unfeasible, and if the authors want the research community to train algorithms with 10 million examples of simulated IRs, a comparison set of at least 10 rooms doesn't seem unreasonable. Given this, the authors should moderate their claim of "highly realistic" and highlight more thorough tests with real-world IRs as future work.

- The comparison between the recorded reverberation and the simulated is rather coarse. Both the RT60 and DRR are frequency dependent in real rooms. And large rooms exhibit prominent resonent peaks in the late tail due to resonant modes.  Do the soundscape 2.0 IRs capture these structures? If the answer is yes, than so much the better for soundscape 2.0.  If no, the discrepancies will be informative and useful to know.

- The simulation requires specifying a set of parameters for any material in the simulation scene (absorption, scattering, transmission). It is good that the authors have provided default values for common materials, and it is good that users can specify such values vi an API.  But I still have many questions: how were the default values obtained? From measurements? Or physics models? Or are they all obtained to optimize the fit to one single room as implied in footnote 7?  Moreover, I am curious to know how dramatically the reverberation would change if these values were altered. If the authors re-simulated the 7 IRs they compare with real-world recordings with the acoustic randomization procedure they applied in the speech recognition task, by how much will the simulated RT60 and DRR vary? If the answer is "not much, the RT60 and DRR are robust", then a skeptical reader will be reassured that the close match shown in Fig 3 is not just due to over-fitting parameters to a single room. If otherwise, readers will be forewarned that their own simulations may require some care and thought in the initial setup to yield high-quality audio. In which case, how should a user know how to adjust these parameters?

- Whereas graphics rendering can ignore the thickness of walls and surfaces, acoustic rendering does not have this luxury.  The absorption/transmission coefficients of real rooms will certainly be different for thin-vs-thick surfaces.  Presumably, this can be fixed by manually altering the material parameters, but this may require manually adjusting these values for any given rendered room. Again, future users of thie simulation tool should be forewarned that they may have to think about these issues to get high-quality results. 

- In the supplemental material document the authors state, regarding Fig 4: "The main reason for the Matterport3D’s RT60 distribution skewing towards left is because there are lots of broken meshes in that dataset, which results in ray leaking from holes and smaller reverberation in general. On the contrary, Gibson and HM3D have higher quality mesh and have larger RT60s on average".  Of course the authors here cannot be blamed for problematic meshes.  A rendering algorithm can only be as good as the information fed into it.  But this does highlight an issue: standard 3D meshes are optimized for image rendering which faces different constraints than audio rendering (image rendering requires ray-tracers to track a much smaller number of reflections than audio renderers, so, holes in the mesh, which pose little problem for image rendering and might be common, can prove problematic for audio).  This implies that a fraction of the 10 million IRs in their public dataset underestimate the RT60 (how many?). The dataset is still of use, but this issue should be publicized.  Also, once more, those who wish to use the simulation tool should be forewarned they need to check their mesh quality. 

- In the speech recognition task, the authors demonstrate that a speech recognition trained on reverberant speech performs better when trained on soundscape 2.0 reverberation rather than pyroomacoustics.  I assume they ensured that the distribution of RT60s and DRRs for the two sets of simulated IRs were matched, but this is not stated. In addition, I would be curious to see the effect if the speech recognition algorithm was trained with recorded reverberation from another dataset (Presumably, soundscape 2.0 would yield better performance because of the size of the simulated dataset), or with perceptually inspired statistical reverberation synthesis models, such as used in hearing research, or in music digital effects (Presumably, here soundscape 2.0 would yield better performance because the reverberation is more realistic). If this were demonstrated, it would provide evidence that soundscape 2.0 really is a state-of-the-art audio renderer. Until then, I would suggest the authors moderate their claims of "highly realistic", and mention these other tests as useful future work.
correctness: As listed in weaknesses, I think some of the claims made by the authors are not well supported.  But I do not see anything that seems obviously incorrect.

clarity: The paper is well written, except some key information is buried in the supplemental document, as I have detailed above in "weaknesses".

relation to prior work: The references are good and thorough. Some mention that digital music tools create compelling renditions of reverberation via statistical techniques might be warranted.  It suggests that there may be other approaches to reverberation synthesis beyond high-resolution and CPU-intensive ray-tracing.

additional feedback: N/A.
review_title: Although the claims to realism are not as strong as I would like, this is nonetheless an useful tool

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: In this submission, the authors introduced SoundSpaces 2.0, a simulation platform that supports audio simulation with a number of improvements over existing platforms. In addition, the authors have validated the simulation accuracies with real-world recordings, introduced a dataset based on the environment, and conducted benchmarking on two tasks.
strengths: The contributions are significant. First, the simulation now supports real-time, continuous audio rendering; second, the rendering is also very realistic based on bidirectional path tracing; third, the simulation is now configurable and can be used as a general tool to simulate any 3D environment. All these features are much needed for embodied AI research.

The dataset and the benchmarking results are also good to have for the community.

weaknesses: I see two weaknesses regarding the submission.  First, the methods used in benchmarking are limited. The authors used one algorithm for each of the two benchmark tasks.  The one for navigation is also introduced in the previous version of the work.  It'd be good to see a more systematic evaluation with additional methods.

Second, the sim-to-real experiments can be strengthened.  The authors conducted some preliminary experiments in Benchmark 2.  It'd be great to see more of these experiments in more benchmarks and tasks to show that not only the simulation itself is realistic, but models trained in these environments do transfer to the real world.
correctness: Mostly yes. However, the methods considered in the evaluation are limited, and the simulation-to-real experiment is mostly a proof-of-concept and can be strengthened.
clarity: Yes, the writing is good.
relation to prior work: Yes, the related work section is complete. 
additional feedback: None.
review_title: Solid contribution to visual-acoustic simulation

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper proposed a new simulator for simulating acoustic effects. The major difference compared to the prior work is that this work can compute sound properties on the fly and thus avoid using precomputed data. Therefore, the new platform can support arbitrary geometry and continuous outputs. The authors also perform two experiments to provide evidence for the usefulness of this work

strengths: An upgraded version of the platform overcomes some major limitations of the prior work.
Some experiments demonstrate that continuous inputs do impact model performance. 
Provided datasets and platforms can be quite useful to the community.
weaknesses: Details of the sound sources are not clear. An object might drop and only produces one impact sound.  Not sure if the current platform supports this natural interaction. 

For experiment 2, the authors are using high-quality mode. It's interesting to see how the faster version compared to the slower version. So we can better understand the impact of this trade-off. 

Overall it’s a good paper. However, the authors should include more details to reach a broader audience and to help the community understand what can be done with the proposed platform. A more detailed limitation section or some comparison with real-world sound sources should help.

correctness: Yes
clarity: For the most part. 
relation to prior work: Very clear.
additional feedback: N/A
review_title: A new embodied AI platform simulating sound

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: Summary:
The authors introduce SoundSpaces 2.0, a platform for on-the-fly geometry-based audio 2 rendering for 3D environments. It allows continuous spatial sampling and demonstrates the effectiveness in two downstream tasks (embodied navigation and far-field automatic speech recognition).
strengths: 1. Novelty. The proposed platform for continuous spatial sampling is novel, and the ideas are original.
2. Detail pipeline. The authors provide the audio-visual rendering pipeline in detail and illustrate the heavy workload.
3. Experiments. The authors compare and benchmark two downstream tasks: continuous audio-visual navigation and far-field speech recognition, and further provide insights on designing these systems.
4. Good presentation. The paper is clearly constructed and presented.
weaknesses: 1. Though two downstream tasks have been benchmarked, it could be better to provide results on extended tasks, such as cross-modal learning and more audio-only research. 
2. It's better to provide statistics of the proposed dataset, including evaluation and visualization of the duration, number of speakers, and so on.
correctness: The experiment design is appropriate and performed correctly.
clarity: The paper is well and clearly written.
relation to prior work: SoundSpaces 1.0’s foremost limitation is its pre-computed, discretized nature. In contrast, SoundSpaces 2.0 performs on-the-fly geometry-based audio rendering for arbitrary environments. It allows continuous spatial sampling and generalization to novel environments
additional feedback: N/A
review_title: Review for SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper introduces SoundSpaces 2.0, the first geometry-based real-time acoustic simulation platform for 3D environments. The main difference from its previous version (SoundSpaces) is its continuous acoustic simulation (as opposed to discrete), which brings three benefits: 1) more accurate acoustic simulation, not restricted to predefined grid points, 2) generalization to new scenes (arbitrary input meshes), 3) better configurability (microphones, materials, etc). In terms of experimental evaluation, the authors compare the simulated sounds with real-world audio measurements. They also showcase two downstream tasks that leverage the simulator: audio-visual navigation and far-field automatic speech recognition. 
strengths: The paper proposes a solid, open-sourced implementation of an audio simulation environment that performs on-the-fly geometry-based audio rendering for arbitrary scenes. It’s first of its kind and can open up many research opportunities for (multi-modal) embodied AI. In terms of accessibility, everything seems to be open-sourced. The platform supports two modes - high-speed and high-quality, and also includes a large-scale offline dataset called SoundSpaces-PanoIR if the users are not interested in embodied AI tasks. 

weaknesses: The experimental evaluation section has a few noteworthy weaknesses. 

First, it’s a bit unclear why continuous audio simulation is important. Intuitively, it’s better because it’s closer to the real world. But the experimental results (Table 3 specifically) are not very convincing. Comparing row 2 and row 3, it’s expected that training in SoundSpaces and testing in SoundSpaces 2.0 will cause a large domain shift, which is known to be very detrimental for deep learning models. It’s still unclear whether having continuous audio simulation is important, if the ultimate goal is to deploy a home service robot in the real world (a motivation that is repeatedly mentioned in the paper). 

The results of Table 4 is more convincing: clearly, SoundSpaces 2.0 generates more realistic synthetic IRs than Pyroomaoustics for the purpose of ASR. However, table 4 should also include the result of “Finetuned on SoundSpaces”. In addition, figure 3 (b) is a bit suspicious. There seems to be a constant delta between the results of SoundSpaces and SoundSpaces 2.0. Where is that delta coming from? In other words, why is SoundSpace always producing DRR 10 dB higher?

Furthermore, in table 2, the authors should probably also compare with SoundSpaces (1.0) and ThreeDWorld. They have certain simplifications (discrete grid point for SoundSpaces, and “shoebox” simplification for TDW), and presumably, they can run faster. The users should be aware of the full spectrum of the simulation speed versus quality tradeoff when choosing which simulator to use. For example, if some user needs >100 FPS or >1000 FPS for their RL use case, maybe they will choose SoundSpaces over SoundSpaces 2.0.

Finally, there seems to be missing a clear comparison with ThreeDWorld in the experimental evaluation section. ThreeDWorld also supports real-time, continuous audio rendering and also impact sound simulation, which is missing from SoundSpaces 2.0. The main limitation seems to be its assumption of simplified scene geometry (“shoebox”). The authors should demonstrate how this limitation affects the learning of downstream tasks, including embodied ones (e.g. audio-visual navigation) or non-embodied ones (e.g. ASR).

correctness: The simulation environment and the dataset seem to be constructed in a very sound way. The paper proposes two example downstream tasks that can leverage the simulator: AV-Nav and ASR, although the simulator can probably unlock more research directions than what is demonstrated. 

clarity: The paper is clearly written. 

relation to prior work: The paper has a pretty comprehensive related work section that covers three aspects: acoustic simulation, audio-visual learning, and sim-to-real transfer. In the acoustic simulation section, it’s probably advisable to include ThreeDWorld (TDW is discussed in the introduction, but not in related work). It would also be better to explicitly acknowledge here that TDW simulates impact sounds of object collision, whereas SoundsSpaces 2.0 cannot. Finally, in the sim2real transfer section, the authors missed the citation of ObjectFolder 2.0 [1], which indeed conducted sim2real transfer of machine learning models (despite in slightly different scenarios, but still quite related) 

[1] Gao, Ruohan, et al. "ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfer." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

additional feedback: In the supplemental video, the section of comparison with real measurements: SoundSpaces 2.0 is definitely better than SoundsSpaces, but it’s still quite different from the real IR. What causes the gap and how you would propose to fill the gap in the future?

Minor typo in Fig 3 caption: It should be “(b) Comparing real measurements and simulation”, not “(c) Comparing…”
review_title: Significant infrastructure for visual-acoustic learning; room for improvement for experimental evaluation

Meta-review: metareview: This paper was reviewed by six experts and received all positive scores. AC feels this work is interesting and deserves to be published on NeurIPS 2022 dataset track. The reviewers did raise some valuable concerns that should be addressed in the final camera-ready version of the paper. The authors are encouraged to make the necessary changes in the final version.
confidence: 5: The area chair is absolutely certain
Decision: Accept