Title: AV-Cloud: Spatial Audio Rendering Through Audio-Visual Cloud Splatting
Venue: NeurIPS 2024
Accepted: yes
Keywords: audio-visual

audio scenes reconstruction

spatial audio

point-based scene rendering
Abstract: We propose a novel approach for rendering high-quality spatial audio for 3D scenes that is in synchrony with the visual stream but does not rely or explicitly conditioned on the visual rendering. We demonstrate that such an approach enables the experience of immersive virtual tourism - performing a real-time dynamic navigation within the scene, experiencing both audio and visual content. Current audio-visual rendering approaches typically rely on visual cues, such as images, and thus visual artifacts could cause inconsistency in the audio quality. Furthermore, when such approaches are incorporated with visual rendering, audio generation at each viewpoint occurs after the rendering of the image of the viewpoint and thus could lead to audio lag that affects the integration of audio and visual streams. Our proposed approach, AV-Cloud, overcomes these challenges by learning the representation of the audio-visual scene based on a set of sparse AV anchor points, that constitute the Audio-Visual Cloud, and are derived from the camera calibration. The Audio-Visual Cloud serves as an audio-visual representation from which the generation of spatial audio for arbitrary listener location can be generated. In particular, we propose a novel module Audio-Visual Cloud Splatting which decodes AV anchor points into a spatial audio transfer function for the arbitrary viewpoint of the target listener. This function, applied through the Spatial Audio Render Head module, transforms monaural input into viewpoint-specific spatial audio. As a result, AV-Cloud efficiently renders the spatial audio aligned with any visual viewpoint and eliminates the need for pre-rendered images. We show that AV-Cloud surpasses current state-of-the-art accuracy on audio reconstruction, perceptive quality, and acoustic effects on two real-world datasets. AV-Cloud also outperforms previous methods when tested on scenes "in the wild".


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes AV-Cloud, a framework for high-quality spatial audio rendering in 3D scenes without relying on visual cues. AV-Cloud addresses issues in current audio-visual rendering methods, such as audio lag and dependence on visual rendering quality, by introducing Audio-Visual Anchors and the Audio-Visual Cloud Splatting module. These components facilitate the generation of viewpoint-specific spatial audio synchronized with visual content. The method demonstrates superior performance on multiple benchmarks, outperforming existing baselines in audio reconstruction accuracy, perceptual quality, and acoustic effects.\n\nSTRENGTHS:\n1. The concept of using Audio-Visual Anchors and Cloud Splatting to decouple audio rendering from visual rendering is interesting.\n2. The paper demonstrates comprehensive experimentation and robust evaluation across multiple benchmarks.\n3. The paper is well-structured and the presentation of the framework is clear. The figures and supplement examples help the readers better understand.\n4. The proposed method addresses critical issues in real-time audio-visual rendering.\n\nWEAKNESSES:\n1. The mathematical formulation of the Audio-Visual Cloud Splatting module could be more detailed. For instance, Equation (2) introduces the softmax function applied to the relative vectors and visual features, but the reason behind this specific formulation and its implications are not sufficiently explained. Clarifying how the weights $a_{ki}$ are computed and how they influence the final output would enhance understanding.\n2. The technical derivation of the Spatial Audio Render Head (SARH) lacks depth. Specifically, the process described in Equations (4) and (5), where the mixture mask $m_m$ and the difference mask $m_d$ are used to compute the left and right channel outputs, is not fully elaborated. The significance of these masks and their impact on the final audio quality are not clearly discussed. Additionally, the role and impact of the convolution modules within the residual structure (Figure 3) are not sufficiently explained.\n3. While the method shows strong performance on benchmarks and some real-world examples, the provided examples are too idealized and lack challenging elements like interfering sound (e.g., crowd noise). I think the robustness of AV-Cloud in more complex and noisy real-world environments should also be validated.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nThe authors mention the limitations of their approach's challenges and potential drawbacks. The reliance on camera calibration and the potential issues with noise in real-world audio recordings are noted. Additional imitations can be found in the Weaknesses section", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

comments: SUMMARY:
A novel approach for rendering high-quality spatial audio in 3D scenes, called AV-Cloud, is proposed. This method synchronizes with the visual stream without relying on or being explicitly conditioned by visual rendering, enabling immersive virtual tourism through real-time dynamic navigation of both audio and visual content. Unlike current audio-visual rendering methods that depend on visual cues and may suffer from visual artifacts causing audio inconsistencies, AV-Cloud overcomes these issues. It uses a set of sparse AV anchor points, forming an Audio-Visual Cloud derived from camera calibration, to represent the audio-visual scene. The Audio-Visual Cloud allows for the generation of spatial audio for any listener location. A novel module, Audio-Visual Cloud Splatting, decodes these AV anchor points into a spatial audio transfer function for the listener’s viewpoint, which is then applied by the Spatial Audio Render Head module to transform monaural input into viewpoint-specific spatial audio. This approach eliminates the need for pre-rendered images and efficiently aligns spatial audio with any visual viewpoint. The results are satisfying.

STRENGTHS:
1. The AV anchors strategy seems to be interesting and effective for audio-visual scene representation. The Audio-Visual Cloud Splatting is novel for AV tasks but more likely to be a Q-former.
2. The experiment results are good and ablations are clear.

WEAKNESSES:
As I mentioned in the strengths, the Audio-Visual Cloud Splatting seems to be a Q-former like module.

QUESTIONS:
What is the difference between the AVCS and Q-former?

LIMITATIONS:
The authors adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-15

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores the problem of generating 3D audiovisual scenes – that is, generating 3D scenes with spatial audio. The proposed approach, AV Cloud, uses anchor points obtained from Structure-from-Motion (SfM) points. The anchors are then used with an AV Cloud splatting module which decodes the visuals and the audio. Experiments are done on RWAVS and Replay-NVAS with comparisons done with several prior works.\n\nSTRENGTHS:\n– 3d audiovisual scene generation is a really interesting problem to solve. WHile there is considerable literature on visual scene generation, generating 3d visual scene is an interesting problem with real-world applications. \n\n– The model claims to be able to generate the audio and the visuals in parallel. Essentially unlike prior work it decouples the generation of two modalities by not using the generated visuals for generating the audio. \n\n– On objective metrics, the paper claims to make good improvements\n\n---- \nincreased score after rebuttal\n\nWEAKNESSES:\n– The paper is a bit difficult to follow – especially the key part of AudioVisual anchor points. \n\n– First, a short primer on SfM is desirable, even if it is in Appendix. More importantly though, it is not clear why it makes sense to use SfM points and clustering on top of them to model AV anchor points and generation of spatial points. Why does it make sense to use SfM points or anchors derived from them as the starting point for AV generation ? What relation the anchors have with audio which motivates the fact that these anchors can be used for audio generation ? \n\n– Second, the details of AV anchor points are fuzzy. The visuals are used for SfM points which are then clustered to get the anchors. Where is the audio into picture here ? Are these anchors visual only ? If so, why are we calling it AV Anchors ? \n\n– In prior works, for example AV-Nerf, there is an an explicit AV-Mapper which learns the audio visual relations through which the spatial audio generatio happens. Here Visual2Audio splatting transformer is expected to model that ? \n\n– For the subjective tests, it would be good to actually get proper subjective ratings on the generated spatial audio. The current preference numbers are not very informative. Getting the spatial audio rated with respect to their quality and spatial characteristics would be much more meaningful. \n\n– Since NAF, INRAS and other works are considered here - I think it would be good to reference NACF ([R1]) below. NACF specifically focuses on using visuals and is ideal for comparison. \n\n[R1] Neural Acoustic Context Field: Rendering Realistic Room Impulse Response With Neural Fields\n\nQUESTIONS:\nPlease address the questions below.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-18"}