Title: AV-GS: Learning Material and Geometry Aware Priors for Novel View Acoustic Synthesis
Venue: NeurIPS 2024
Accepted: yes
Keywords: Spatial audio synthesis

Gaussian Splatting

Material characteristics

Geometry priors
Abstract: Novel view acoustic synthesis (NVAS) aims to render binaural audio at any target viewpoint, given a mono audio emitted by a sound source at a 3D scene. Existing methods have proposed NeRF-based implicit models to exploit visual cues as a condition for synthesizing binaural audio. However, in addition to low efficiency originating from heavy NeRF rendering, these methods all have a limited ability of characterizing the entire scene environment such as room geometry, material properties, and the spatial relation between the listener and sound source. To address these issues, we propose a novel Audio-Visual Gaussian Splatting (AV-GS) model. To obtain a material-aware and geometry-aware condition for audio synthesis, we learn an explicit point-based scene representation with audio-guidance parameters on locally initialized Gaussian points, taking into account the space relation from the listener and sound source. To make the visual scene model audio adaptive, we propose a point densification and pruning strategy to optimally distribute the Gaussian points, with the per-point contribution in sound propagation (e.g., more points needed for texture-less wall surfaces as they affect sound path diversion). Extensive experiments validate the superiority of our AV-GS over existing alternatives on the real-world RWAS and simulation-based SoundSpaces datasets. Project page: \url{https://surrey-uplab.github.io/research/avgs/}

AV-GS: Learning Material and Geometry Aware Priors for Novel View Acoustic Synthesis

Abstract

Novel view acoustic synthesis (NVAS) aims to render binaural audio at any target viewpoint, given a mono audio emitted by a sound source at a 3D scene. Existing methods have proposed NeRF-based implicit models to exploit visual cues as a condition for synthesizing binaural audio. However, in addition to low efficiency originating from heavy NeRF rendering, these methods all have a limited ability of characterizing the entire scene environment such as room geometry, material properties, and the spatial relation between the listener and sound source. To address these issues, we propose a novel Audio-Visual Gaussian Splatting (AVGS) model. To obtain an implicit material-aware and geometry-aware condition for audio synthesis, we learn an explicit point-based scene representation with an audio-guidance parameter on locally initialized Gaussian points, taking into account the space relation from the listener and sound source. To make the visual scene model audio adaptive, we propose a point densification and pruning strategy to optimally distribute the Gaussian points, with the per-point contribution in sound propagation (e.g., more points needed for texture-less wall surfaces as they affect sound path diversion). Extensive experiments validate the superiority of our AV-GS over existing alternatives on the real-world RWAS and simulationbased SoundSpaces datasets. Project page: https://surrey-uplab.github. io/research/avgs/

1 Introduction

Novel view synthesis [25, 1, 14] allows to generate images for any target viewpoints, which has been extensively studied and advanced. For real-world applications in augmented reality (AR) and virtual reality (VR), solely visual rendering of 3D scenes without spatial audio (i.e., deaf) fails to fully immerse users in the virtual environment. This thus inspires a recent surge of investigating novel view acoustic synthesis (NVAS) [4, 17, 5, 32]. By synthesizing binaural audio (two channels corresponding to the left and right ear) taking into account the factors like directionality, distance and relative elevation, this can create a spatial audio experience akin to real-life perception [23]. However, realistic binaural audio synthesis is challenging, since the wavelengths of sound waves are much longer, necessitating the modeling of wave diffraction and scattering. To illustrate, while blocking the sun with your thumb is easy, blocking thunder sounds with your thumb is difficult because sound waves wrap around obstacles. A sound wave propagating through a 3D space undergoes various sophisticated acoustic phenomena, like direct sound, early reflections, and late reverberations.

Aiming to render binaural audio from the mono audio for target poses, Neural Acoustic Field (NAF) [21] learns room acoustics in a synthetic environment with a 2D grid of implicit representations as a condition. However, deriving a 2D representation grid for real-view scenes is extremely challenging in the presence of unconstrained objects, materials, and occlusion in 3D scenes. Alternatively, A V-NeRF

Speaker

[17] leverages implicit representations of the visual cue by learning a vision NeRF model [25] that generates the listener's view. Nonetheless, this can only offer limited conditions, as the listener's view provides information from only the line-of-sight, whilst audio spreads around corners more widely (a listener cannot see behind but can hear sources that are behind). Further, NeRF's training and inference efficiency is typically low [37, 14].

To overcome these limitations, in this work, we present a novel Audio-Visual Gaussian Splatting (AV-GS) model, characterized by efficiently learning an explicit, holistic 3D scene condition with rich material and geometry information, as well as more comprehensive contextual knowledge beyond the listener's field of view for enhanced synthesis conditioning. Due to the intrinsic discrepancy between vision and audio as discussed earlier, extending existing 3D Gaussian splatting-based (3DGS) [14] from visual scenes to 3D audio ( i.e. , spatial audio) is non-trivial. Optimizing towards scene visual reconstruction, points tend to over-populate along object edges and under-populate in textureless regions such as walls, doors etc. But, such point distribution is rhetoric when learning sound propagation since, major changes in sound paths (absorption and diversion) primarily happen around those texture-less regions. The key challenge lies in jointly modeling 3D geometry and implicit material characteristics of the visual scene objects to instigate direction and distance awareness for realistic binaural audios. To that end, we decouple the physical geometric prior from the 3DGS representation to learn an acoustic field network by introducing audio-guidance parameters. These audio-guidance parameters combined with their relative distance and direction from the listener and sound source, are projected on-the-fly to derive holistic scene-aware and implicit material-aware conditions for synthesizing binaural audios (see Figure 1). We hypothesize that converging A V-GS on binaural audio reconstruction loss, requires location and density adjustment of the Gaussian points relative to the 'audio-guidance' provided by the individual point. Towards this direction, we design a Gaussian point densification and pruning strategy based on the individual per-point contribution in providing this 'guidance' for sound propagation, ultimately improving the overall binaural audio synthesis.

To summarize, we make the following contributions : (1) The first novel view acoustic synthesis work with conditions on holistic scene geometry and implicit material information; (2) A novel AV-GS model that learns a holistic geometry-aware material-aware scene representation in a tailored 3DGS pipeline; (3) Extensive evaluations validating the advantages of our method over prior art alternatives on both synthetic and real-world datasets.

2 Related Work

Audio binauralization converts monaural (single-channel) audio signals into binaural (two-channel) audio signals. It can enhance immersion by simulating sound directionality, distance, and spatial cues. Generating realistic binaural audio from mono audio is challenging [31, 30], prompting the exploration of various conditioning techniques in the literature.

Geometry and material conditioning Luo et al. [21] introduced Neural Acoustic Field (NAF) for room acoustics modeling using implicit neural representations of geometric features, capturing spatial information of speakers and receivers. Su et al. [34] also learned implicit neural representations for audio, focusing on interactive acoustic radiance transfer, relying on scene geometry as input. Anton and Dinesh [29] proposed a material-aware binaural sound propagation model, incorporating

material and topology data, necessitating an acoustic material database. In contrast, our approach learns explicit representation from 3D scenes and disentangles learning of physical and (implicit) material properties, eliminating reliance on scene geometry and material characteristics as inputs.

Visual cue conditioning Recent works leverage visual cues to generate spatial audio from mono audio, capturing complementary scene characteristics [9, 38, 27, 41, 18, 43, 19, 28]. Li et al. [15] proposed a multi-task approach optimizing binaural audio generation and flipped audio classification, sharing visual cues from corresponding video frames. Chen et al. [4] synthesized sound by analyzing input audio-visual cues, incorporating active speaker features, target pose, and encoded visual features in the acoustic synthesis pipeline. Alternatively, Liang et al. [17] encoded the listener's position and orientation, conditioned by the listener's view. It additionally provides local visual depth information dependent on the listener's position. However, focusing solely on the listener's view overlooks the broader 3D scene geometry's contribution, which is crucial for sound propagation. We thus propose to learn a holistic 3D scene representation, enhancing binauralization guidance with additional audio parameters.

3D Scene Representation Learning Point-based rendering techniques, initiated by [10], utilize point-based representation where each point affects a single pixel. Zwicker et al. [45] advanced this with ellipsoid-based rendering (splatting), allowing mutual overlap to fill image holes. At the absence of given geometry, Mildenhall et al. [25] explored neural implicit representation, NeRF, predicting view-dependent radiance via implicit density fields. NeRF requires combining colors of densely sampled points along the camera rays for high-quality rendering. 3D Gaussian splatting (3D-GS) [14], a novel-view synthesis method, employs explicit point-based representation, contrasting with NeRF's volumetric rendering. Since its real-time high-quality rendering capabilities that 3DGS has been applied to various domains, including simultaneous localization [13, 24], content generation [36], and 4D dynamic scenes [16, 39, 42], among others.

However, no works take the 3DGS advantages to improve the NVAS task. Hence, in this work, we for the first time exploit the 3D-GS for NV AS, to the best of our knowledge, including capturing the scene geometry and material-related information for audio signal processing purposes. Further, we adapt the point management mechanism to account for the characteristics of sound propagation. In a nutshell, our approach aims to improve binaural audio reconstruction loss in A V-GS by adjusting Gaussian point location and density based on their contribution to sound propagation guidance.

3 Method

Problem definition Deployed at a location X S in a 3D scene, a sound source S emits a mono audio a mono . A listener entity L moves around in this 3D scene, capturing multiple observations using a camera mounted with a binaural microphone. Each observation O p = ( V C , V A ) , constitutes a pair of a camera view V C and an auditory perspective V A , w.r.t a specific listener pose p = ( X L , d ) , where X L is the 3D position of the listener and d is the viewing direction corresponding to listener's head. A camera view V C defines that at the pose p , the listener would observe a RGB image, I (as their view). Similarly, an auditory perspective V A defines that at the pose p , the listener hears a binaural audio, a bi = ( a l , a r ) where l and r represent the left and right ear respectively. Given N observations from the listener, O = { O 1 , O 2 , ..., O N } , the task is to predict the binaural audio a bi ∗ for a novel observation O p ∗ having an arbitrary unseen pose p ∗ .

3.1 Audio-Visual Gaussian Splatting (AV-GS)

Our model is comprised of a 3D Gaussian Splatting model G , an acoustic field network F , and an audio binauralizer B . In order to model sound propagation in space and time, having a holistic scene prior as contextual guidance is essential. Crucially, this guidance must incorporate both geometric and material-related characteristics. For facilitating the learning of visual and auditory modalities with inherently distinct characteristics, we decouple the physical geometry and the acoustic field by introducing in-between an acoustic field network F that learns geometry-aware and implicit material-aware scene contexts.

In the following, we first, provide a brief regarding learning the scene geometry. Later, we explain how to construct an audio-focused representation G a . We further describe the process of decoding the parameters of G a on the fly using view-dependent information, followed by point densification and pruning tailored for sound propagation.

R

AdaptiveDensity

Control

3DGaussians

Projection

Differential

XSOSHR

Rasterizer

Sound

XL

Source

Q

?

B

abi

amono

Cs

R

Forward -G

Initialisation

Forward - Ga

Gradientflow

Audio-aware

3Dlocation

DensityAdaptation

3D Gaussian Splatting (3D-GS) [14] is a state-of-the-art novel-view synthesis method that learns an explicit point-based representation of the 3D scene. It is utilized here to capture the scene geometry. Specifically, each point in the explicit representation G is represented as a 3D Gaussian ellipsoid to which physical attributes like position X , quaternion R , scale S , opacity O and Spherical Harmonic coefficients ( SH ) representing view-dependent color are attached. For an arbitrary camera view V C ∗ with a pose p ∗ , we project/splat 3D Gaussians onto the 2D image plane to obtain the listener's view as an RGB image I ∗ . The projection of 3D Gaussian ellipsoids can be formulated as:

where Σ ′ and Σ = RSS T R T are the covariance matrices for 3D Gaussian ellipsoids and projected Gaussian ellipsoids on 2D image from a viewpoint with viewing transformation matrix W . J is the Jacobian matrix for the projective transformation. Please refer to [14] for more details on splatting.

3.1.1 Acoustic Field Network

As a part of the decoupling approach, we further introduce an audio-focused point-based representation G a . Concretely, every point in G a is parameterized using the location X , alongside a learnable audio-guidance parameter α to encapsulate implicit material-specific characteristics of the scene. X is initialized using the location of Gaussian points in G , in order to impose the scene geometry knowledge. To initialize α we concatenate the view-dependent color priors from G , particularly spherical harmonics SH and quaternion feature R . An intuition is to choose the parameters that provide information regarding the color and density characteristics, which are often correlated with material properties [8] (see ablation on the choice of different parameters from G for forming α in Section 4.4).

In order to drive the mono to binaural audio transformation, a pose-specific holistic scene context has to be derived from G a . To achieve this, we derive a position-guidance feature G w.r.t both, the listener L and the sound source S . Position-guidance G concatenated with the audio-guidance α is used as an input to the acoustic field network F to obtain a joint context for each point w.r.t to the listener and the sound source separately.

Concatenating (represented using ⊕ ) the listener-context and the speaker-context yields the overall pose-specific holistic scene context. We obtain the condition for binauralization by averaging the context across all points in G a , post dropping points outside the vicinity of the listener and sound source. The vicinity for the listener and sound source is defined by the nearest k L and k S points based on the Euclidean distance between X and X L and that between X and X S , respectively. We provide an ablation for selecting the size of vicinity points in Section 4.4.

3.1.2 Audio Binauralizer

An audio binauralization module B transforms the mono audio a mono emergent from the sound source S into binaural audio a bi = { a l , a r }, representing the left and right audio channels. The position and orientation of the listener (relative to the sound source) along with the learned holistic scene context C is used to condition this transformation.

where X L denotes the listener's position and orientation. We adopt a similar architecture for the binauralization as [17], wherein X L and C are fused to generate acoustics masks m m and m d representing the mixture and difference of the sound fields respectively. Specifically, an MLP combines X L and G a with a frequency query f ∈ [0 , F ] to produce a feature vector which is further projected using a linear projection to obtain a mixture mask m m for frequency f . This feature vector is concatenated with the transformed direction θ and passed to a second MLP to generate a difference mask m d . All frequencies within [0 , F ] are queried to generate complete masks for both the mixture and difference. The magnitude spectrogram s mono , computed using short-time Fourier transform (STFT) over a mono , is multiplied with m m and m d to obtain the mixture magnitude s m and the difference magnitude s d , respectively. Finally, an inverse STFT is operated on s m and s d to obtain the binaural audios for the left and right channels, a l and a r . The architecture of the binauralizer is discussed in our appendix A.2.

3.2 Model training

Due to our decoupling design, we adopt a dual-stage optimization, starting with an initial warmup stage that learns the physical properties of the Gaussian points using gradients obtained while reconstructing camera views (RGB images). The objective of this initial stage is to infer an explicit point-based representation G to capture the scene geometry. Optimization of G is adapted from the original 3D-GS [14] using the loss function,

In the second stage, we initialize the audio-guidance parameters of G a using physical parameters of the warmed-up 3D Gaussians. Subsequently, we learn the implicit material properties for the Gaussian points using gradients obtained in the binauralization task for every training auditory perspective. Optimization of G a is guided by a combination of the binaural audio reconstruction loss, L m and a volume regularization loss, L v .

where, L m is the summation of the L 2 loss calculated individually between the predicted magnitudes for s m , s l , s r (i.e., mixture, left and right audio channel, respectively) and their respective ground-truth magnitudes. L v = ∑ N a i =1 Prod ( α i ) , where Prod ( . ) is the product of the values of the audio-guidance parameter α of each Gaussian point in the auditory perspective, encouraging the audio-guidance parameters to be small, and non-overlapping.

The physical properties and material-related properties of the new Gaussians are decoded from the learned physical parameters and audio-guidance parameters respectively, in a view-dependent manner on-the-fly.

Audio-aware point management The location of points in G a are initialized from G , however since G is optimized using an image reconstruction loss, the initial placement of the points in G a is not necessarily contributive to an optimal audio guidance. Especially, it has been observed in recent works that 3D-GS is inadequate in densifying texture-less and less observed areas [20, 6, 2].

To address this problem, we propose an error-based point growing policy that populates new points in G a , wherever deemed significant. Specifically, the densification step is interleaved across multiple binauralization forward passes. During each densification step, we compute averaged gradients (across all steps up to the previous consecutive densification step) for each point in G a that lies in the vicinity, denoted as Θ g . Points with Θ g > τ g are deemed as significant, where τ g is a pre-defined threshold. Using the original 3D position as a probability distribution function (PDF), we sample new points. The audio-guidance parameters for these new points are obtained by random initialization. Additionally, we regularly employ a random elimination of points to avoid rapid expansion of new points (e.g., every 3000 iterations).

4 Experiments

4.1 Datasets

We evaluate both a real-world dataset-RWAVS and a synthetic dataset-SoundSpaces.

Real-World Audio-Visual Scene (RWAVS) [17] The RWAVS dataset offers realistic multi-modal training samples constituting camera poses, high-quality binaural audios, and images. The dataset consists of 11 indoor and 2 outdoor scenes. Randomly sampling the sound source and listener positions, the authors collected data ranging from 10 to 25 minutes for every scene. For every scene, a data sample includes a set of camera poses (sound source and listener), extracted RGB key frame, followed by one-second binaural audio (as received at the listener position) and one-second mono source audio (as emitted by the sound source). We follow an 80:20 train-validation split for every scene in the dataset.

SoundSpaces synthetic dataset [3] The Soundspaces dataset consists of 6 indoor scenes with varying degrees of complexity (2 scenes having a single room with rectangular walls; 2 scenes having a single room with a non-rectangular walls; 2 with multi-room layout). Since the dataset includes room impulse responses (RIR) recorded at receiver/listener positions, we replace the acoustic mask generation block with an RIR prediction block. The listener consists of a stereo listener, with four discrete head orientations among 0, 90, 180, and 270. We follow the same 80:20 train-validation split, as [17] for every scene in the dataset.

4.2 Implementation Details

Our complete implementation is based in PyTorch using a single Nvidia A550 GPU. The learning rate for all physical parameters in G is adopted from the original 3D-GS implementation [14]. For the acoustic field network F , we use MLP network with 64 nodes. Hyper-parameters for the binauralizer B , as well as the metrics are adopted from A V-NeRF [17]. For the audio-aware point management, we set τ g to 0.0004. Additionally, after every 3k iterations we randomly eliminate points using an outlier removal process (from Open3D [44]) that removes points that have less than 8 neighbors in a sphere less than radius of 0.1.

As a part of the proposed decoupling, we train the 3D-GS model G for 3k iterations, wherein each iteration involves randomly sampling a camera view V C and optimizing the parameters for G . Post this we initialize α with the physical parameters from G , and train for 40k iterations, wherein now each iteration involves randomly sampling an auditory perspective V A and optimizing the parameters for G a and B .

4.3 Results

Binaural audio synthesis - RWAVS dataset - Table 1 In order to have a fair comparison, we adopt the same metrics - magnitude distance [MAG] [41] (computed in time-frequency domain) and envelope distance [ENV] [26] (computed in time domain), as [17]. Particularly, (1) Mono-Mono duplicates the source audio to create fake binaural audio without modifications; (2) Mono-Energy scales the input audio's energy to match the target, generating stereo audio by duplicating the scaled audio; (3) Stereo-Energy separately scales the input audio's energy for each channel to match the target, then combines them for stereo audio; In addition to the codec baselines above, we also compare with NAF [21], INRAS [34], and AV-NeRF [17]. AV-GS significantly surpasses all prior arts across all scenes, by a significant margin. Particularly compared against A V-NeRF, which utilizes

Ground-truth

AV-NeRF

AV-GS

Ground-truth

AV-NeRF

AV-GS

M05- 011

R

R

(a)

(q)

audio-visual cues, AV-GS achieves a 5.7% and 3.4% relative improvement in terms of MAG and ENV respectively.

We highlight the qualitative improvement of AV-GS over AV-NeRF in Fig. 3 (and in Section A.1). AV-NeRF extracts visual cues from the listener's view using a 256x256 RGB and depth image (rendered by a pre-trained NeRF) and processed by a frozen ResNet18 [11] model that is trained on ImageNet [7]. When the listener's view includes multiple objects or complex geometries (Fig. 3(a)), or meaningless information (Fig. 3(b)), it leads to sub-optimal visual cue extraction and poor binauralization. In contrast, our A V-GS uses learned representations (combining α of points in G a w.r.t to the listener and speaker) to extract context at a more holistic level, remaining unaffected by these scenarios.

RIR generation - Soundspaces dataset - Table 2(a) Following [17] and [21], we compare AV-GS against classical high-performance audio coding methods: Advanced Audio Coding (AAC) [12] and Xiph Opus [40], applying both linear and nearest neighbor interpolation techniques to the coded acoustic fields. Additionally, we also compare with existing neural methods: NAF [21], INRAS [34], and AV-NeRF [17]. It can be observed that AV-GS outperforms all the prior arts by a significant margin.

(a) Comparison with state-of-the-art: Performance on SoundSpaces dataset using T60, C50, and EDT metrics. Lower score indicates a higher RIR generation quality. Opus is an open audio codec [40], and AAC is a multi-channel audio coding standard [12].

Improved efficiency Decoding α on-the-fly helps A V-GS retain the efficiency advantage of having an explicit point based representation over learning an implicit representation of the visual 3D scene. In contrast, A V-NeRF trains an additional NeRF model, whose outputs are used to render the listener's view which is the condition for their binauralizer module. In Table 3, we compare the inference time for a single scene (House scene 1) from the RWAVS dataset. Although the original implementation of AV-NeRF proposed a single view in the direction of listener's viewing direction (used as the listener's context), we modify their implementation in which the V-NeRF renders 2 and 4 perspective views with a field of view of 90 ◦ for each receiver's position. It can be observed, although using the additional views, helps provide additional local context around the listener (hence improving the binauralization performance), but only at the cost of increased inference time. AV-GS on the other hand requires least rendering time, while providing the best binauralization performance. It is important to note that, although we train the 3D-GS for only an initial warm-up stage of 3k iterations, we can still jointly train both G and G a on the corresponding camera view V C and the auditory perspective V A respectively. By doing so, A V-GS would be able to synthesize both binaural audio as well the visual view at the unseen target (novel) viewpoint, without any increase in the training time than that of as shown in Table 3 (last row). In this work, we instead focus on only the binaural audio synthesis and hence train only G a in the second stage.

Although we train the 3D-GS for just an initial 3k iterations, we can still train both G and G a together for the corresponding camera view V C and auditory perspective V A , such that the trained A V-GS is able to synthesize both the binaural audio and the visual view from a novel viewpoint , thereby making it a fair comparison with AV-NeRF [17]. However, in this work, we focus solely on binaural audio synthesis and only train G a in the second stage.

4.4 Ablation study

We use the binaural audio synthesis task for performing our ablation study and report average performance across all scenes in the RWAVS dataset.

Initialization of α . 3D-GS learns a point-based representation ( G ) of the 3D scene such that each point in the explicit representation is represented as a 3D Gaussian ellipsoid to which physical attributes like position X , quaternion R , scale S , opacity O and Spherical Harmonic coefficients ( SH ) representing view-dependent color are attached. As discussed above, we initialize the audioguidance parameter α using the learned physical parameters from G . Table 2(b) shows an in-depth study of the effect on the binauralization performance when choosing different parameters from G for the initialization. It is evident that SH and R , when considered individually as well as when combined, are crucial parameters that help provide an overall better binauralization performance.

Size of the vicinity The learned Gaussian points G representation normally consists of millions of points, and using all the points for computing the condition for the binauralization task will result in a huge computational overload. Moreover, intuitionally, points near the listener and speaker location in the 3D space ( i.e. vicinity) are more contributive to determining the local geometry and material-related characteristics, and greatly influence the sound field. Empirically in Table 4a, we find that using 15 percentile of the points in the vicinity (listener and speaker considered separately) yields the best performance. Using an extremely high percentile for capturing the vicinity generally helps, but only to a certain extent, since it increases the risk of adding unnecessary information. Moreover, it is important to note that in scenes with smaller room sizes or when the listener and speaker are

Percentile

5

10

15

20

25

.

Overall

MAG

↓

ENV

1.482

1.454

1.417

1.424

1.428

0.143

0.141

0.140

0.141

0.141

(a)

(b)

(c)

(d)

placed nearby, a larger vicinity capture might lead to overlapping points thus adding to redundant information (see Fig. 4 (d)).

Effect of audio-aware point management. AV-GS infers local geometry from an explicit pointbased prior and learns additional audio-specific parameters for every point. The local geometry can be inferred either directly from the sparse points produced during camera calibration using Structure-from-motion (SfM) [33] or warming up a 3D-GS model on the SfM points. From Table 4b it is evident that audio-aware point management helps improve the binauralization performance irrespective of the adopted point initialization approach. Texture-less regions such as walls, doors, etc are the regions with maximum sound absorption and diversion and hence having optimal point density promotes better modeling of sound propagation.

5 Conclusion

In this work, we present AV-GS, a holistic scene representation learning approach for conditioning novel view acoustic synthesis. The core of AV-GS lies in the proposed decoupled modeling of 3D scene geometry and material-related characteristics of scene objects for sound propagation, enabled by the introduction of additional audio-guidance parameters per point within an acoustic field network. We show that our approach leverages audio-aware Gaussian point positioning to improve the binauralization performance, on real world as well as synthetic 3D scenes, significantly in comparison to prior art alternatives.

Limitations Although AV-GS achieves a new state-of-the-art on the NVAS task by learning a holistic scene representation with geometry-aware and implicit material-aware characteristics, it faces some challenges. Like AV-Nerf, AV-GS currently learns a separate representation for each scene, posing a significant challenge for generalizability and transferability across multiple scenes, which is crucial for improving efficiency and reducing computational demands. Also, as observed in Table 1, the performance drops when the scene size increases (e.g., from Office to House), due to the difficulty in learning representations for larger, complex geometries with multiple rooms. An interesting research direction would be to model individual rooms using separate 3D-GS representations and then learn a transfer function across rooms.

References

Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In CVPR , 2022.

Samuel Rota Bulò, Lorenzo Porzi, and Peter Kontschieder. Revising densification in gaussian splatting. arXiv preprint arXiv:2404.06109 , 2024.

Changan Chen, Unnat Jain, Carl Schissler, Sebastia Vicenc Amengual Gari, Ziad Al-Halah, Vamsi Krishna Ithapu, Philip Robinson, and Kristen Grauman. Soundspaces: Audio-visual navigation in 3d environments. In ECCV . Springer, 2020.

Changan Chen, Alexander Richard, Roman Shapovalov, Vamsi Krishna Ithapu, Natalia Neverova, Kristen Grauman, and Andrea Vedaldi. Novel-view acoustic synthesis. In CVPR , 2023.

Ziyang Chen, Israel D Gebru, Christian Richardt, Anurag Kumar, William Laney, Andrew Owens, and Alexander Richard. Real acoustic fields: An audio-visual room acoustics dataset and benchmark. arXiv preprint arXiv:2403.18821 , 2024.

Kai Cheng, Xiaoxiao Long, Kaizhi Yang, Yao Yao, Wei Yin, Yuexin Ma, Wenping Wang, and Xuejin Chen. Gaussianpro: 3d gaussian splatting with progressive propagation. arXiv preprint arXiv:2402.14650 , 2024.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR , 2009.

Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In CVPR , 2022.

Ruohan Gao and Kristen Grauman. 2.5 d visual sound. In CVPR , 2019.

Jeffrey P Grossman and William J Dally. Point sample rendering. In Rendering Techniques' 98: Proceedings of the Eurographics Workshop . Springer.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , 2016.

International Organization for Standardization. Advanced audio coding (aac). ISO/IEC 138187:2006, 2006.

Nikhil Keetha, Jay Karhade, Krishna Murthy Jatavallabhula, Gengshan Yang, Sebastian Scherer, Deva Ramanan, and Jonathon Luiten. Splatam: Splat, track & map 3d gaussians for dense rgb-d slam. arXiv preprint arXiv:2312.02126 , 2023.

Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics , 2023.

Sijia Li, Shiguang Liu, and Dinesh Manocha. Binaural audio generation via multi-task learning. ACM Transactions on Graphics , 2021.

Zhan Li, Zhang Chen, Zhong Li, and Yi Xu. Spacetime gaussian feature splatting for real-time dynamic view synthesis. arXiv preprint arXiv:2312.16812 , 2023.

Susan Liang, Chao Huang, Yapeng Tian, Anurag Kumar, and Chenliang Xu. Av-nerf: Learning neural fields for real-world audio-visual scene synthesis. NeurIPS , 2023.

Yan-Bo Lin and Yu-Chiang Frank Wang. Exploiting audio-visual consistency with partial supervision for spatial audio generation. In AAAI , 2021.

Francesc Lluís, Vasileios Chatziioannou, and Alex Hofmann. Points2sound: from mono to binaural audio using 3d point cloud scenes. EURASIP Journal on Audio, Speech, and Music Processing , 2022(1):33, 2022.

Tao Lu, Mulin Yu, Linning Xu, Yuanbo Xiangli, Limin Wang, Dahua Lin, and Bo Dai. Scaffoldgs: Structured 3d gaussians for view-adaptive rendering. arXiv preprint arXiv:2312.00109 , 2023.

Andrew Luo, Yilun Du, Michael Tarr, Josh Tenenbaum, Antonio Torralba, and Chuang Gan. Learning neural acoustic fields. NeurIPS , 2022.

Andrew Luo, Yilun Du, Michael J Tarr, Joshua B Tenenbaum, Antonio Torralba, and Chuang Gan. Learning neural acoustic fields. 2022.

Sagnik Majumder, Changan Chen, Ziad Al-Halah, and Kristen Grauman. Few-shot audio-visual learning of environment acoustics. In NeurIPS , 2022.

Hidenobu Matsuki, Riku Murai, Paul HJ Kelly, and Andrew J Davison. Gaussian splatting slam. arXiv preprint arXiv:2312.06741 , 2023.

Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV , 2020.

Pedro Morgado, Nuno Nvasconcelos, Timothy Langlois, and Oliver Wang. Self-supervised generation of spatial audio for 360 video. NeurIPS , 2018.

Kranthi Kumar Rachavarapu, Vignesh Sundaresha, AN Rajagopalan, et al. Localize to binauralize: Audio spatialization from visual sound source localization. In CVPR , 2021.

Anton Ratnarajah, Sreyan Ghosh, Sonal Kumar, Purva Chiniya, and Dinesh Manocha. Av-rir: Audio-visual room impulse response estimation. arXiv preprint arXiv:2312.00834 , 2023.

Anton Ratnarajah and Dinesh Manocha. Listen2scene: Interactive material-aware binaural sound propagation for reconstructed 3d scenes. In IEEE VR , 2024.

Anton Ratnarajah, Zhenyu Tang, and Dinesh Manocha. Ir-gan: Room impulse response generator for far-field speech recognition. arXiv preprint arXiv:2010.13219 , 2020.

Anton Ratnarajah, Shi-Xiong Zhang, Meng Yu, Zhenyu Tang, Dinesh Manocha, and Dong Yu. Fast-rir: Fast neural diffuse room impulse response generator. In ICASSP , 2022.

Kazuki Shimada, Archontis Politis, Parthasaarathy Sudarsanam, Daniel A Krause, Kengo Uchida, Sharath Adavanne, Aapo Hakala, Yuichiro Koyama, Naoya Takahashi, Shusuke Takahashi, et al. Starss23: An audio-visual dataset of spatial recordings of real scenes with spatiotemporal annotations of sound events. NeurIPS , 2023.

Noah Snavely, Steven M Seitz, and Richard Szeliski. Photo tourism: exploring photo collections in 3d. In ACM siggraph 2006 papers , pages 835-846. 2006.

Kun Su, Mingfei Chen, and Eli Shlizerman. Inras: Implicit neural representation for audio scenes. Advances in Neural Information Processing Systems , 35:8144-8158, 2022.

Kun Su, Mingfei Chen, and Eli Shlizerman. Inras: Implicit neural representation for audio scenes. In NeurIPS , 2022.

Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309.16653 , 2023.

Ayush Tewari, Justus Thies, Ben Mildenhall, Pratul Srinivasan, Edgar Tretschk, Wang Yifan, Christoph Lassner, Vincent Sitzmann, Ricardo Martin-Brualla, Stephen Lombardi, et al. Advances in neural rendering. In Computer Graphics Forum , pages 703-735. Wiley Online Library, 2022.

Ruiqi Wang, Haonan Cheng, Long Ye, and Qin Zhang. Visual-guided scene-aware audio generation method based on hierarchical feature codec and rendering decision. Displays , 83:102708, 2024.

Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. arXiv preprint arXiv:2310.08528 , 2023.

Xiph.Org Foundation. Xiph opus. https://opus-codec.org/ , 2012.

Xudong Xu, Hang Zhou, Ziwei Liu, Bo Dai, Xiaogang Wang, and Dahua Lin. Visually informed binaural audio generation without binaural audios. In CVPR , 2021.

Zeyu Yang, Hongye Yang, Zijie Pan, Xiatian Zhu, and Li Zhang. Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting. In ICLR , 2024.

Hang Zhou, Xudong Xu, Dahua Lin, Xiaogang Wang, and Ziwei Liu. Sep-stereo: Visually guided stereophonic audio generation by associating source separation. In ECCV , 2020.

Qian-Yi Zhou, Jaesik Park, and Vladlen Koltun. Open3d: A modern library for 3d data processing. arXiv preprint arXiv:1801.09847 , 2018.

Matthias Zwicker, Hanspeter Pfister, Jeroen Van Baar, and Markus Gross. Surface splatting. In Annual conference on Computer graphics and interactive techniques , 2001.

A Appendix / supplemental material

Broader impact Potential applications of A V-GS lie in virtual reality, augmented reality, and immersive audio experiences, where accurate and efficient binaural audio rendering can significantly enhance user experience. However, enhanced acoustic synthesis technology could be used unethically in surveillance or privacy-invasive applications, where binaural audio could be synthesized to eavesdrop on conversations or create misleading audio experiences.

A.1 Direction and distance awareness showcased by AV-GS

Fig. 5 highlights the distance-awareness exhibited in the binaural audios synthesized by A V-GS. The top row shows the corresponding RGB frame or the listener's view, followed by AV-GS's learned holistic scene representation in the next row. The two rows show a plot of the rendered binaural audios (left and right respectively) with time along the x-axis and amplitude of the audio along the y-axis. (Please note that in the second row we slice the points in G a for along the vertical axis, essentially removing the points from the ceiling, in order to provide better visibility.) With the increase in the distance of the listener (blue sphere) from the sound source (yellow sphere), A V-GS is able to synthesize binaural audios with decreasing amplitude across both the left and right audio channels.

Increasing distancefrom thesoundsource

R

Decreasing amplitudein the binaural audio

Fig. 6 shows the direction-awareness showcased by AV-GS. In (a) the synthesized left channel has a higher magnitude compared to the right channel, owing to the proximity of the listener's left ear to

the sound source. Vice-versa, in (b) the right channel has a higher amplitude compared to the right channel owing to the proximity of the listener's right ear to the sound source.

R

R

R

(a)

(b)

A.2 Binauralizer

STFT

amono

query

ISTFT

1p

b

md

ISTFT

Embedding

Embedding

frequency f

XL

We adopt the architecture of binauralizer from [17] with some modifications to input the learned scene context from our acoustic field network F (see Fig. 7).

Embedding blocks take the normalized location position X L (only 2D x and y coordinates).

The main components include two Multilayer Perceptrons (MLPs), each comprising four linear layers with an additional residual connection. The width of each linear layer, is set to 128 for the RWA VS dataset and 256 for the SoundSpaces dataset. All linear layers are followed by ReLU activation layers, except for the last layer, where the ReLU activation is replaced with the Sigmoid function. The first MLP takes the listener's position (x, y) and the frequency f ∈ [0 , F ] as input, where F represents the number of frequency bins. It predicts a mixture mask m m for the given frequency f and generates a

feature vector with c channels. Prior to feeding them into the MLP, we apply positional encoding to the listener's position (x, y) and the frequency f . We set the maximum frequency used for positional encoding as 10. A coordinate transformation [17] to project the listener's direction θ into a high frequency space. We concatenate the transformed listener's direction and the feature vector, and feed it into the second MLP. The second MLP is appended with a Sigmoid layer and a scaling layer, ensuring that the difference mask m d estimated by the second MLP falls within the range of -1 , 1 . For each frequency query f , estimates two masks: m m and m d , both of which are scalars. We iterate over all frequencies f ∈ [0 , F ] to obtain the complete masks m m and m d .

For validating AV-GS on the room impulse response generation task using the Soundspaces dataset, following [17], we drop the mixture mask, and predict the impulse response directly for a corresponding time input, using the second MLP. We show these modifications in Fig. 8.

query

b

B

Impulse respose

→(IR)

2

Embedding

Embedding

Xs

Time t

XL

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The sections of Methods and Experiments clearly describe the claims we made.

Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations are discussed in Section 5

Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: the paper does not include theoretical results.

Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide all details for hyper-parameters as well as train and test settings wherever applicable. Additionally, we will publicly release our entire codebase along with pre-trained models upon acceptance.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We will publicly release our entire codebase along with pre-trained models on our project page (link provided in abstract). The datasets used are publicly available.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Discussed in Section 3 - Model Training

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Seeds are fixed to provide reproducibility of the results.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Provided in Section 4 - Implementation details, as well as, detailed discussion provided regarding inference and training times.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: All guidelines have been followed.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss broader impact as a part of our Conclusion.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper poses no such risks.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All existing data and code that has been used by us is properly cited and all of it is under the CC-BY 4.0 license.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: the paper does not release new assets.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: the paper does not involve crowdsourcing nor research with human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [Yes]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel view acoustic synthesis approach based on 3D Gaussian Splatting scene representation. The framework consists of a 3D GS model, acoustics field network, and audio binauralizer. It first trains a 3D GS model to capture scene geometry. Then they use attributes from each Gaussian to initialize a learnable acoustic point representation, and then average the feature obtained from nearby Gaussian points for acoustic modeling. \nThrough the experiments on the RWAVS and Soundspaces dataset, the authors demonstrate that the proposed method achieves state-of-the-art performance. They also perform the ablation study to ablate some components to demonstrate their technical advance.\n\nSTRENGTHS:\n- The paper brings up a clever way to leverage point-based scene representation from 3DGS for acoustic modeling and the results are promising. \n- The authors perform thorough experiments and ablations to demonstrate its technical advance. The proposed method outperforms baselines on the quantitative evaluation.\n\nWEAKNESSES:\n- The reviewer thinks there are several issues with the writing:\n  - First, **the method part isn't clearly stated**. It takes a long time for the reviewer to fully understand the method. The method is simple, using pretrained 3DGS to initialize a learnable acoustic point representation. While the method figure doesn't quite help with understanding. \n  - The reviewer strongly feels the paper is **overclaimed**. It claims it learns \"holistic geometry-aware material-aware scene representation\" while the experiments are only for acoustic modeling results. The author needs to use experiments to support this claim. One thought the reviewer has to visualize the PCA feature of learnable acoustic points to see if it has some cues. Otherwise, the reviewer is only convinced that this paper proposed a better acoustic modeling approach that leverages the GS features. \n  - The major experimental **results for the baselines are directly brought from AV-NeRF and INRAS papers** while the authors don't mention this in the paper. Also, the authors need to confirm if they have guaranteed the same training and test split and evaluation code.\n\n- The reviewer has a question on the Position-guidance G condition. The authors use unit vectors which get rid of distance information. The reviewer is confused as to why remove the distance. Would it be an important cue?\n- In the paper [5] Real acoustic field, the authors found out that energy decay loss shows a powerful improvement on those energy-based metrics. Did authors consider using this loss to improve?\n\nQUESTIONS:\nI have mentioned my concerns and questions in the weaknesses part and I hope the authors could address them during the rebuttal.\n\nA suggestion to the authors for paper writing: save the figures as PDF instead of screenshots or png.\n\nLIMITATIONS:\nYes, the authors adequately addressed the limitations", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new audio-visual Gaussian Splatting model for novel view acoustic synthesis. The proposed method explicitly models scene geometry and learns a point-based scene representation with an audio guidance parameter on locally initialized Gaussian points that takes the space relation from the listener and sound source into consideration. This work is the first that applies gaussian splatting to the novel view acoustic synthesis problem, and experiments on two datasets demonstrate the effectiness of the proposed method compared to prior work.\n\nSTRENGTHS:\n- The idea makes sense, and I was expecting gaussian splatting to be applied to this problem by somebody sooner or later, and this paper is indeed doing that.\n\n- Generally, the paper makes good attempts to apply Gaussian Splatting to this recently proposed task, a direct extension on how it is used for visual rendering. The method is clearly formulated and defined with informative notations.\n\n- Experiments are conducted on two datasets following prior work AV-NeRF [17], and it shows noticeable gains compared to prior baselines.\n\n- Many alation studies are performed including ablation on the size of vicinity w.r.t the listener and sound source position, ablation on the choice of physical parameters, which is extensive.\n\nWEAKNESSES:\n- There is a strong claim that the proposed model is learning material-aware priors, jointly modeling 3D geometry and material characteristics, holistic scene geometry and material information, etc. Then, I would actually expect materials are explicitly modeled, such as acoustic properties of surfaces in the form of some material parameters (e.g., absorption coefficients, Young’s modulus, etc.). However, all that is modeled is just a parameter alpha that claims to encapsulate material-specific characteristics of the scene. This is somewhat misleading as there is nothing constrains alpha to be physics-based or to be correlated with some material parameters. It's likely that it’s just learning some scene representation that may be more helpful for acoustic rendering.\n\n- Also the so-called acoustic field network is also nothing about audio by itself. By acoustic field network, I was thinking the method is explicitly modeling sound propagation by taking into account the material properties of surfaces and the geometry of the space. The audio binauralizer is basically following prior work AV-NeRF [17], and what changes is what the audio binauralizer conditions on. In AV-NeRF, it’s scene features from NeRF and in this work it’s the context scene features from gaussian splatting.\n\n- There were no qualitative examples given in supplementary materials, so it's really hard to appreciate the improvement that this method leads to.\n\n- There are also many typos or grammar mistakes here and there. For example:\n1. L45, 3D Gaussian splatting-based *methods*?\n2. Related work, mixed use of past tense and present tense. \n3. L125, a brief ??\n\nQUESTIONS:\n- See questions in the weakness section above.\n\n- Also, for the SoundSpaces synthetic dataset, why it only contains 6 indoor scenes. There are more scenes in the dataset, and it would be good to explain the reason why only these six are not used.\n\n- Is there any way to interpret the learned audio-guidance parameter to show that it’s material related?\n\nLIMITATIONS:\nYes some limitations are discussed in the end, though some might be missing as discussed above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes an efficient and performant method for novel view acoustic synthesis using Gaussian Splatting representations of 3D scenes. The proposed model integrates geometry and material conditioning, and offers a way to bridge the gap between visually-useful 3DGS representations and acoustically useful ones, using “audio-guidance” which considers the local environment in Experiments show this outperforms existing NeRF-based approaches while being more efficient.\n\nSTRENGTHS:\nOverall, I think this work makes a very useful contribution. Bridging Gaussian Splatting representations with audio rendering is likely to be of significant interest to the community, and the quantitative results and efficiency seem strong. The experiments and comparisons are helpful, and give me confidence that these results are quite robust.\n\nWEAKNESSES:\nThe value of $\\tau_g$ seems somewhat arbitrarily defined, and the range of tested vicinity sizes is somewhat narrow; given the efficiency of the method. It’s hard to extrapolate what the performance would be with more context (i.e. would it keep going down?). Most of my other suggestions are relatively minor, about the writing and presentation:\n- Abstract: “at a 3D scene” -> for clarity, should this be “within” a 3D scene?\n- Introduction: “visual rendering of 3D scenes without spatial audio (i.e., deaf)” -> this seems like an inaccurate characterization. Deafness is not about rendering, but about sensation. Spatiality is also not clearly related to this.\n- Introduction: “realistic binaural audio synthesis is challenging, since the wavelengths of sound waves are much longer” -> this is not the only reason that this is challenging. Even if you use a simplified (e.g. ray-tracing) model, which would be inaccurate at lower frequencies due to wave-like behavior etc., you would still need a good model of the environment for physics-based rendering, or a large number of spatial measurements for a data-driven approach.\n- Introduction: “…modeling 3D geometry and material characteristics of the visual scene objects to instigate direction and distance awareness…” -> while geometry has a direct impact on “direction and distance”, material impacts spatial perception indirectly (through absorption, reflection, and diffusion). Could this be rephrased more clearly?\n- Related work: “Anton and Dinesh” should maybe be “Ratnarajah and Manocha”, i.e. last names of authors?\n- Related work: Under “Geometry and material conditioning”, the final statement differentiates the current work from prior work by pointing out the non-reliance on scene and geometry inputs, but doesn’t clarify why this is useful. E.g. it could be pointed out that, in the real world, such inputs are often not available, so this work allows generalizing to such settings.\n- Figure 2: “Differential Rasterizer” -> “Differentiable Rasterizer”?\n- Figure 4: Could there be a subfigure showing 15%, since that seems to give the best overall performance? It would be helpful context, since otherwise the reader needs to interpolate between (c) and (d).\n- Section 4.4: “intuitionally” -> “intuitively”?\n\nQUESTIONS:\n- Is $S$ modeled as an omni-directional point source? I assume so because point source is the typical assumption, and an emission direction isn’t given (vs. listener direction d), but it’s worth clarifying.\n- I think the notation here could be clearer. For example, $V_C$ and $V_A$ don’t seem used, and it’s not clear to me what their role is? E.g. why is the observation $O_p$ = ($V_C$, $V_A$) instead of ($I$, $a_{bi}$)? If the pose $p$ captures the position and orientation, what do $V_C$ and $V_A$ add on top of this representation? Why is $I$ upper-case and $a$ lower-case here, when they both represent observations? Why is $a_{bi} = \\{a_l, a_r\\}$ notated as a set, but other sets are notated as tuples? Additionally, shouldn’t $k_l$ be $k_L$, similar for $S$?\n- Conceptually, there is a tension between considering the global scene context, as the paper argues for, and dropping points outside the local vicinity of the source and listener (since sound reflections can easily include points from other regions, in principle). Could the authors comment on this? It’s interesting that the optimal vicinity size is not the largest one tested. Is there some intuition for why this might be the case, given that it offers more information? Currently, the reasoning seems to be that this contains unnecessary information, but the useful information seems to be a subset of this?\n- The audio rendering task is cast as a binauralization task. I understand that this is practically necessary, i.e. the input is mono and output is stereo binaural. However, wouldn’t many of the problems in this paper still exist in the monaural listener context as well? E.g. compared with a binauralization task purely based on HRTFs or simplified room geometries (e.g. Richard et al., ICLR 2021), where the context $C$ is not as rich. Could this be clarified?\n- How was the $\\tau_g$ value decided?\n\nLIMITATIONS:\nI think the authors have fairly represented the limitations. The broader impact statement is a little bit less clear to me, i.e. I’m not sure I understand the surveillance applications. Could this be clarified? I would be interested for the authors to consider the potential impact of the limitations listed. For example, the limitation of rendering larger scenes; what practical negative impacts could this have in deployment scenarios?", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a 3D Gaussian splatting (3DGS) based method for improving the acoustic context modeling for novel view acoustic synthesis (NVAS). The proposed model outperforms multiple existing methods on both simulated and realworld data across different metrics. The method also trains and infers faster than the state-of-the-art AV-Nerf method. The paper also provides model ablations and analyses that help in better understanding the role of different model components.\n\nSTRENGTHS:\n1. The idea of using 3D Gaussian Splatting for better modeling of finegrained geometry and material aware acoustic context for improving the NVAS performance is interesting, novel and works well in practice.\n\n2. The paper reports strong results, where the proposed model outperforms existing methods on both simulated and realworld data\n\n3. The provided model ablations and analyses help in better understanding the contributions of different model components and design choices\n\nWEAKNESSES:\n1. Acoustic conditioning with G_a: the paper \"averages the context across all points in G_a\" (L156-7). Doesn't this lead to loss of finegrained point-specific information? A learned aggregation strategy (for example, by using the attention mechanism) would probably make more sense here?\n\n2. L192-4, \"L_v ... non-overlapping\": \n    i) the insight behind doing this is not clear from the text\n    ii) there are no references to support these steps\n    iii) there is no ablation for this loss, as well\n\n3. L208, \"The audio guidance ... random initialization\": won't random initialization during point expansion lead to local 'discontuity'? Did the authors try warm initialization to preserve local 'continuity' by doing nearest neighbor search or bilinear interporation on the existing alphas?\n\n4. Does the model generalize to more challenging scene datasets like Matterport3D [1]? \n\n5. Audio examples: the paper does not seem to provide audio examples and qualitiatively compare their quality with that of the baselines\n\n6. Minor:\n   i) L274-5, \"4 perspective views ... receiver's position\": do these 4 views cover the full 360 degree, otherwise it's not comparable with the field of view of the AVGS model?\n   ii) Para in L285-90 looks like repeated text\n\n\nReferences: \n[1] Matterport3D: Learning from RGB-D Data in Indoor Environments. Chang et al. 3DV 2017.\n\nQUESTIONS:\nCould the rebuttal comment (more) on the following?\n1. how the chosen strategy for aggregating the acoustic context in G_a affects model performance. See weakness 1 for details.\n\n2.  the rationale behind using L_m, and how the model performs without the loss. See weakness 2 for details. \n\n3. initialization of alphas during point densification in G_a. See weakness 3 for details.\n\n4. generalization of the model to more challenging datasets like Matterport3D. See weakness 4 for details.  \n\n5. my minor comments. See weakness 6 for details.\n\nAlso, could the authors anonymously provide a few audio examples from the model and baselines?\n\nLIMITATIONS:\nThe paper mentions limitations in Discussion but I could not find any discussion on societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}