AV-GS: Learning Material and Geometry Aware Priors for Novel View Acoustic Synthesis
Authors: Swapnil Bhosale
Haosen Yang
Diptesh Kanojia
Jiankang Deng
Xiatian Zhu
Venue: NeurIPS 2024 poster (2024)
Keywords: Spatial audio synthesis
Gaussian Splatting
Material characteristics
Geometry priors
Primary area: speech_and_audio
TL;DR: An audio-visual Gaussian Splatting approach to learn holistic scene priors for novel view acoustic synthesis.

Abstract: Novel view acoustic synthesis (NVAS) aims to render binaural audio at any target viewpoint, given a mono audio emitted by a sound source at a 3D scene. Existing methods have proposed NeRF-based implicit models to exploit visual cues as a condition for synthesizing binaural audio. However, in addition to low efficiency originating from heavy NeRF rendering, these methods all have a limited ability of characterizing the entire scene environment such as room geometry, material properties, and the spatial relation between the listener and sound source. To address these issues, we propose a novel Audio-Visual Gaussian Splatting (AV-GS) model. To obtain a material-aware and geometry-aware condition for audio synthesis, we learn an explicit point-based scene representation with audio-guidance parameters on locally initialized Gaussian points, taking into account the space relation from the listener and sound source. To make the visual scene model audio adaptive, we propose a point densification and pruning strategy to optimally distribute the Gaussian points, with the per-point contribution in sound propagation (e.g., more points needed for texture-less wall surfaces as they affect sound path diversion). Extensive experiments validate the superiority of our AV-GS over existing alternatives on the real-world RWAS and simulation-based SoundSpaces datasets. Project page: \url{https://surrey-uplab.github.io/research/avgs/}

AV-GS: Learning Material and Geometry Aware Priors for Novel View Acoustic Synthesis
Swapnil Bhosale University of Surrey, UK
Haosen Yang University of Surrey, UK
Diptesh Kanojia University of Surrey, UK
Jiankang Deng Imperial College London, UK
Xiatian Zhu University Surrey, UK
Abstract
Novel view acoustic synthesis (NVAS) aims to render binaural audio at any target viewpoint, given a mono audio emitted by a sound source at a 3D scene. Existing methods have proposed NeRF-based implicit models to exploit visual cues as a condition for synthesizing binaural audio. However, in addition to low efficiency originating from heavy NeRF rendering, these methods all have a limited ability of characterizing the entire scene environment such as room geometry, material properties, and the spatial relation between the listener and sound source. To address these issues, we propose a novel Audio- Visual Gaussian Splatting (AV- GS) model. To obtain an implicit material-aware and geometry- aware condition for audio synthesis, we learn an explicit point-based scene representation with an audio- muddii-g parameter on locally initialized Gaussian points, taking into account the space relation from the listener and sound source. To make the visual scene model audio adaptive, we propose a point densification and pruning strategy to optimally distribute the Gaussian points, with the per-point contribution in sound propagation (e.g., more points needed for texture-less wall surfaces as they affect sound path diversion). Extensive experiments validate the superiority of our AV-GS over existing alternatives on the real-world RWAS and simulation- based SoundSpaces datasets. Project page: 10/ ha/rseearch/avgg.
github
1 Introduction
Novel view synthesis 25 14 allows generate images for any target viewpoints, which has been extensively studied and advanced. For real-world applications in augmented reality (AR) and virtual reality (VR), solely visual rendering of 3D scenes without spatial audio (i.e., deaf) fails to fully immerse users in the virtual environment. This thus inspires a recent surge of investigating novel view acoustic synthesis (NVAS) 4 5 32). By synthesizing binaural audio (two channels corresponding to the left and right ear) taking into account the factors like directionality, distance and relative elevation, this can create a spatial audio experience akin to real-life perception 23 However, realistic binaural audio synthesis is challenging, since the wavelengths of sound waves are much longer, necessitating the modeling of wave diffraction and scattering. To illustrate, while blocking the sun with your thumb is easy, blocking thunder sounds with your thumb is difficult because sound waves wrap around obstacles. sound wave propagating through 3D space undergoes various sophisticated acoustic phenomena, like direct sound, early reflections, and late reverberations.
Aiming to render binaural audio from the mono audio for target poses, Neural Acoustic Field (NAF) [21 learns room acoustics in a synthetic environment with a 2D grid of implicit representations as condition. However, deriving 2D representation grid for real-view scenes extremely challenging in the presence of unconstrained objects, materials, and occlusion in 3D scenes. Alternatively, AV-NeRF
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Speaker
Sound propagation point patterns between listener (blue sphere) and emitter (yellow sphere) captured by our AV-GS. Notice the points outside the propagation path (points behind the speaker, points behind rigid walls). Please note slice the scene into half along the y-axis (omitting the points from the ceiling) in order to facilitate better visibility.
Figure 1:
[17] leverages implicit representations of the visual cue by learning vision NeRF model [25] that generates the listener's view. Nonetheless, this can only offer limited conditions, as the listener's view provides information from only the line-of-ssigh,, whilst audio spreads around corners more widely (a listener cannot see behind but can hear sources that are behind). Further, NeRF's s training and inference efficiency is typically low 37 14
To overcome these limitations, in this work, present novel Audio-Visual Gaussian Splatting (AV-GS) model, characterized by efficiently learning an explicit, holistic 3D scene condition with rich material and geometry information, as well as more comprehensive contextual knowledge beyond the listener's field of view for enhanced synthesis conditioning. Due to the intrinsic discrepancy between vision and audio discussed earlier, extending existing 3D Gaussian splatting-based (3DGS)
[4] from visual scenes to 3D audio (i.e., spatial audio) is non-trivial. Optimizing towards scene visual reconstruction, points tend to over-populate along object edges and under-populate in texture- less regions such as walls, doors etc. But, such point distribution is rhetoric when learning sound propagation since, major changes in sound paths (absorption and diversion) primarily happen around those texture-less regions. The key challenge lies in jointly modeling 3D geometry and implicit material characteristics of the visual scene objects instigate direction and distance awareness for realistic binaural audios. To that end, we decouple the physical geometric prior from the 3DGS representation to learn an acoustic field network by introducing audio-guidane parameters. These audio-guidance parameters combined with their relative distance and direction from the listener and sound source, are projected on-the-fly to derive holistic scene-aware and implicit matcrial- -waree conditions for synthesizing binaural audios (see Figure D). We hypothesize that converging AV-GS on binaural audio reconstruction loss, requires location and density adjustment of the Gaussian points relative to the muudio-guidance' provided by the individual point. Towards this direction, we design Gaussian point densification and pruning strategy based on the individual per per-point contribution in providing this "guidance" for sound propagation, ultimately improving the overall binaural audio synthesis.
To summarize, we make the following contributions: (1) The first novel view acoustic synthesis work with conditions on holistic scene geometry and implicit material information; (2) A novel AV-GS model that learns a holistic geometry-aware material-aware scene representation in tailored 3DGS pipeline; (3) Extensive evaluations validating the advantages of our method over prior art alternatives on both synthetic and real-world datasets.
2 Related Work
Audio binauralization converts monaural (single-channel) audio signals into binaural (two-channel) audio signals. It can enhance immersion by simulating sound directionality, distance, and spatial cues. Generating realistic binaural audio from mono audio is challenging [31, 30], prompting the exploration of various conditioning techniques in the literature.
Geometry and material conditioning Luo et al. 21 introduced Neural Acoustic Field (NAF) for room acoustics modeling using implicit neural representations of geometric features, capturing spatial information of speakers and receivers. Su et al. (34 also learned implicit neural representations for audio, focusing on interactive acoustic radiance transfer, relying on scene geometry as input. Anton and Dinesh [29] proposed a material-aware binaural sound propagation model, incorporating

---

material and topology data, necessitating an acoustic material database. In contrast, our approach learns explicit representation from 3D scenes and disentangles learning of physical and (implicit) material properties, eliminating reliance on scene geometry and material characteristics as inputs.
Visual cue conditioning Recent works leverage visual cues to generate spatial audio from mono audio, capturing complementary scene characteristics [2 38 27 41
43 19 (88). Li et al. [15 proposed multi-task approach optimizing binaural audio generation and flipped audio classification, sharing visual cues from corresponding video frames. Chen et al. 4 synthesized sound by analyzing input audio-visune cues, incorporating active speaker features, target pose, and encoded visual features in the acoustic synthesis pipeline. Alternatively, Liang et al. [17] encoded the listener's 's position and orientation, conditioned by the listener's 's view. It additionally provides local visual depth information dependent on the listener's position. However, focusing solely on the listener's 's view overlooks the broader 3D scene geometry's 's contribution, which crucial for sound propagation. We thus propose to learn holistic 3D scene representation, enhancing binauralization guidance with additional audio parameters.
18
3D Scene Representation Learning Point-based rendering techniques, initiated by 10 utilize point-based representation where each point affects a single pixel. Zwicker al. 45 advanced this with ellipsoid-based rendering (splatting), allowing mutual overlap to fill image holes. At the absence of given geometry, Mildenhall et al. [25] explored neural implicit representation, NeRF, predicting view-dependent radiance via implicit density fields. NeRF requires combining colors of densely sampled points along the camera rays for high-quality rendering. 3D Gaussian splatting (3D-GS) [14, a novel-view synthesis method, employs explicit point-based representation, contrasting with NeRF's volumetric rendering. Since its real-time high-quality rendering capabilities that 3DGS has
been applied to various domains, including simultaneous localization 13 24], content generation
36), and 4D dynamic scenes [16 39 42), among others.
However, no works take the 3DGS advantages to improve the NVAS task. Hence, in this work, we for the first time exploit the 3D-GS for NVAS, to the best of our knowledge, including capturing the scene geometry and material-related information for audio signal processing purposes. Further, we adapt the point management mechanism to account for the characteristies of sound propagation. In a nutshell, our approach aims to improve binaural audio reconstruction loss in AV-GS by adjusting Gaussian point location and density based on their contribution to sound propagation guidance.
Method
Problem definition Deployed at location Xs in 3D scene, sound source S emits mono audio amono: A listener entity L moves around in this 3D scene, capturing multiple observations using a camera mounted with binaural microphone. Each observation O₂- (Vc, VA), constitutes pair of a camera view Ve and an auditory perspective VA, w.r.t a specific listener pose p = (Xz, d). where X2 is the 3D position of the listener and d is the viewing direction corresponding to listener's head. A camera view Ve defines that at the pose p. the listener would observe a RGB image, / (as their view). Similarly,
auditory perspective VA defines that at the pose p. the listener hears binaural audio, 4sz = (at, a,) where and r represent the left and right ear respectively. Given N observations from the listener, O {O₁, O2, ..., ON J. the task is to predict the binaural audio axx for novel observation O,- having an arbitrary unseen pose p".
3.1 Audio-Visie Visual Gaussian Splatting (AV-GS)
Our model is comprised of a 3D Gaussian Splatting model G. an acoustic field network F, and an audio binauralizer& B. In order to model sound propagation in space and time, having a holistic scene prior as contextual guidance is essential. Crucially, this guidance must incorporate both geometric and material-related characteristics. For facilitating the learning of visual and auditory modalities with inherently distinct characteristics, we decouple the physical geometry and the acoustic field by introducing in-between an acoustic field network F that learns geometry-awan and implicit material-awe -wware scene contexts.
In the following, we first, provide a brief regarding learning the scene geometry. Later, we explain how to construct an audio-focused representation G., We further describe the process of decoding the parameters of G. on the fly using siewddependent information, followed by point densification and pruning tailored for sound propagation.
3

---

Adkptan Denetly
3D Gaussians
O SH
Projection
Eiterrnnaa Tes
dd
werie
Source
CESI
Onion
X
R
Intiallsation
Forward -G
LG.
Gradiemi flow
Audio Derstly Acoplation
Figure 2: Overview of our proposed AV-GS. Our model is comprised of a 3D Gaussian Splatting model G. acoustic field network F and an audio binauralizer B. We first train G to capture the scene geometry information. Next, we construct an audio-focused point representation Ga, with the location X and audio-guidance parameter initialized by the pre-trained G. Then the acoustic field network F is used to process the a parameters for all the Gaussian points in the vicinity of the listener and the sound source (in the 3D space). The output from F is finally used condition the audio binauralizer B3 which transforms the mono audio to binaural audio w.r. the listener and sound source location.
3D Gaussian Splatting (3D-GS) 14 is state-of-the-art art novel-view synthesis method that learns explicit point-based representation of the 3D scene. is utilized here to capture the scene geometry. Specifically, each point in the explicit representation G is represented which physical attributes like position X' quaternion R. scale S, opacity O and Spherical Harmonic coefficients (SH) representing view-dependent color attached. For arbitrary camera view Vo" with pose p". we project/splat 3D Gaussians onto the 2D image plane to obtain the listener's view as an RGB image I* The projection of 3D Gaussian ellipsoids can be formulated as:
3D Gaussian ellipsoid to
E' JW JT
(I)
where E' and Σ = RSST RT are the covariance matrices for 3D Gaussian ellipsoids and projected Gaussian ellipsoids on 2D image from viewpoint with viewing transformation matrix W. the Jacobian matrix for the projective transformation. Please refer to [14 for more details splatting.
3.1.1 Acoustic Field Network
As part of the decoupling approach, we further introduce audio-focused point-based representa- tion Gg. Concretely, every point in G2 is parameterized using the location X, alongside a learnable audio-guidance parameter to encapsulate implicit material-specific characteristies of the scene. X is initialized using the location of Gaussian points in G, order to impose the scene geometry knowledge. To initialize spherical harmonics SW. and quaternion feature R. An intuition to choose the parameters that provide information regarding the color and density characteristics, which often correlated with material properties
concatenate the view-dependent color priors from G, particularly
g (see ablation on the choice of different parameters from G for forming a in
Section 4.4
In order to drive the mono to binaural audio transformation, a pose-specific holistic scene context has to be derived from G.. To achieve this, derive posidion-guidance feature G w.r.t both, the listener L and the sound source Position-guidance concatenated with the audio-guidance a is used input the acoustie field network F to obtain joint context for each point w.r.t to the listener and the sound source separately.
C 3 C₁
X
X;
F(a,
i {S
(3)
|X X

---

Concatenating (represented using ()) the listener-context and the speaker- context yields the overall pose-specifie holistic scene context. We obtain the condition for binauralization by averaging the context across all points in Ga. post dropping points outside the vicinity of the listener and sound source. The vicinity for the listener and sound source defined by the nearest h; and ky points based on the Euclidean distance between X and X2 and that between X and Xs. respectively. We provide an ablation for selecting the size of vicinity points in Section 4.4
3.1.2 Audio Binauralizer
An audio binauralization module B transforms the mono audio amono emergent from the sound source S into binaural audio Gb; = {at, a, ), representing the left and right audio channels. The position and orientation of the listener (relative to the sound source) along with the learned holistic scene context is used to condition this transformation.
abi B( (omooo XL)
(4)
where Xr denotes the listener's position and orientation. We adopt a similar architecture for the binauralization [17], wherein X2 and C are fused to generate acoustics masks Mm. and ma representing the mixture and difference of the sound fields respectively. Specifically, an MLP combines X and G. with frequency query projected using a linear projection to obtain a mixture mask m. for frequency f. This feature vector is concatenated with the transformed direction 0 and passed to second MLP to generate difference mask myz All frequencies within [0, F] are queried to generate complete masks for both the mixture and difference. The magnitude spectrogram Smono: computed using short-time Fourier transform (STFT) over Omono. is multiplied with mm and ma to obtain the mixture magnitude 8m and the difference magnitude sd. respectively. Finally, an inverse STFT is operated 8m and s obtain the binaural audios for the left and right channels, aj and ar. The architecture of the binauralizer is discussed in our appendix A.2
[0, F] to produce feature vector which is further
3.2
Model training
Due to our decoupling design, we adopt dual-stage optimization, starting with an initial warm- up stage that learns the physical properties of the Gaussian points using gradients obtained while reconstructing camera views (RGB images). The objective of this initial stage is to infer an explicit point-based representation G to capture the scene geometry. Optimization of is adapted from the original 3D-GS [4] using the loss function,
La = - A) + ALSSIM
(5)
In the second stage, we initialize the audio-guidance parameters of G, using physical parameters of the warmed-up 3D Gaussians. Subsequently, we learn the implicit material properties for the Gaussian points using gradients obtained in the binauralization task for every training auditory perspective. Optimization of Go is guided by a combination of the binaural audio reconstruction loss, C. and volume regularization loss, L..
LG. = (1 Cm + Aa C.
(6)
Lm = (2(mm) + L2(s)) + (s,,
where, Lm is the summation of the Le loss calculated individually between the predicted magnitudes for 8ms S1, 5, (i.e., mixture, left and right audio channel, respectively) and their respective ground-truth magnitudes. L. = Σd', Prod(as), where Prod(.) is the product of the values the audio-guidance parameter a of each Gaussian point in the auditory perspective, encouraging the audio-guidance parameters to be small, and non-overlapping.
The physical properties and material-related properties of the new Gaussians are decoded from the learned physical parameters and audio aadio-guidance parameters respectively, in a view-dependent manner on-the-fly.
Audio-awer aware point management The location points in G, are initialized from G. however since G' is optimized using an image reconstruction loss, the initial placement of the points in G, is not necessarily contributive to an optimal audio guidance. Especially, has been observed in recent works that 3D-GS is inadequate in densifying texture-less and less observed areas 20 6 2
5

---

To address this problem, we propose an error-based point growing policy that populates new points in G,, wherever deemed significant. Specifically, the densification step interleaved across multiple binauralization forward passes. During each densification step, we compute averaged gradients (across all steps up to the previous consecutive densification step) for each point in G. that lies the vicinity, denoted as gg Points with 6 To are deemed as significant, where Ty is pre-defined threshold. Using the original 3D position as a probability distribution function (PDF), we sample new points. The audio-guidance parameters for these new points are obtained by random initialization. Additionally, we regularly employ a random elimination of points to avoid rapid expansion of new points (e.g., every 3000 iterations).
4 Experiments
4.1 Datasets
We evaluate both a real-world dataset-RWAVS and a synthetic Sataset-SoundSpaces..
Real-World Audio-Visual Scene (RWAVS) [17] The RWAVS dataset offers realistic multi-modal training samples constituting camera poses, high-quality binaural audios, and images. The dataset consists of 11 indoor and 2 outdoor scenes. Randomly sampling the sound source and listener positions, the authors collected data ranging from 10 to 25 minutes for every scene. For every scene, a data sample includes a set of camera poses (sound source and listener), extracted RGB key frame, followed by one-second binaural audio (as received at the listener position) and one-second mono source audio (as emitted by the sound source). We follow 80:20 train-validation split for every scene the dataset.
SoundSpace paccss synthetic dataset 3 The Soundspaces dataset consists of 6 indoor scenes with varying degrees of complexity (2 scenes having a single room with rectangular walls: scenes having a single room with a non-rectangular walls; 2 with multi-room layout). Since the dataset includes room impulse responses (RTR) recorded at receiver/listener positions, we replace the acoustic mask generation block with an RIR prediction block. The listener consists of a stereo listener, with four discrete head orientations among 0, 90, 180, and 270. We follow the same 80:20 valiiaaiittiion split, as 1
for every scene in the dataset.
4.2 Implementation Details
Our complete implementation is based in PyTorch using a single Nvidia A550 GPU. The learning rate for all physical parameters in G is adopted from the original 3D-GS implementation [14]. For the acoustic field network F, we use MLP network with 64 nodes. B, well the metrics are adopted from AV-NeRF [17]. For the audio-aware point management, we set To to 0,0004. Additionally, after every 3k iterations we randomly eliminate points using an outlier removal process (from Open3D (44) that removes points that have less than 8 neighbors in sphere less than radius of 0.1.
for the binauralizer
As part of the proposed decoupling, we train the 3D-GS model G for 3k iterations, wherein each iteration involves randomly sampling a camera view Vo and optimizing the parameters for G. Post this we initialize a with the physical parameters from G, and train for 40k iterations, wherein now each iteration involves randomly sampling an auditory perspective VA and optimizing the parameters for G2 and B.
4.3 Results
Binaural audio synthesis RWAV'S dataset Table I In order to have a fair comparison, we adopt the same metrics- magnitude distance [MAG] 41 (computed in time-frequency domain) and envelope distance [ENV] [26] (computed in time domain), as [17]. Particularly, (1) Mono-Mono duplicates the source audio to create fake binaural audio without modifications; (2) MnnoyEnnegy scales the input audio's 's energy to match the target, generating stereo audio by duplicating the scaled audio; (3) Stereo- Stereo-Energy separately scales the input audio's 's energy for each channel to match the target, then combines them for stereo audio; In addition to the codec baselines above, we also compare with NAF (1), INRAS R4 and AV-NeRF [17. AV-GS significantly surpasses all prior arts across all scenes, by significant margin. Particularly compared against AV-NeRF, which utilizes

---

Table I: Comparison with state-of-the of-the-art art methods RWAVS dataset.
Modality
Office
House L
Apartment !
Outdonrs
Overall
Methods
Audio
Visual
MAG
ENV
MAG
ENV
MAG
ENV
MAG
ENV
MAG
ENV
Mono-Mone
/
9.269
0.411
11,889
0.424
15.120
0.474
13.957
0.470
12.559
0.445
Mono-Energe
1,536
0.142
4,307
0.180
3
0.192
1.634
0.127
2.847
0.160
Stereo-Encrgy Energy
/
1,511
0.139
4,301
0.180
3.895
0.191
1.612
0.124
2,830
0.159
/
1.502
0.130
NAF
22
1.244
0.137
3.259
0.178
3.345
0.193
1.284
0.121
2.283
0.157
AV-NeRF
0.930
0.129
2.009
0.155
2.230
0.184
0.845
0.111
1.504
0.145
AV-GS (ours)
0.861
0.124
1.970
0.152
2.031
0.177
0.791
0.107
1.417
0.140
Figure 3: In the presence of (a) complex geometry, and (b) meaningless views, AV-NeRF, when compared our AV-GS makes errors binaural synthesis. For both scenarios showcase the corresponding listener view, used by AV-NeRF, as well as the learned holistic scene representation that is used by AV-GS, and hence unaffected by both scenarios.
audio-visual cues, AV-GS achieves 5.7% and 3.4% relative improvement in terms of MAG and ENV respectively.
We highlight the qualitative improvement of AV-GS over AV-NeRF in Fig. (and in Section A. AV-NeRF extracts visual cues from the listener's view using a 256x256 RGB and depth image (rendered by a pre-trained NeRF) and processed by a frozen ResNet18 [I]] model that is trained on ImageNet (7]. When the listener's 's view includes multiple objects or complex geometries (Fig. B(a)), meaningless information (Fig. bb)) it leads sub-optimal visual cue extraction and poor binauralization. In contrast, our AV-GS uses learned representations (combining a of points in G. w.r. to the listener and speaker) extract context at more holistic level, remaining unaffected by these scenarios.
RIR generation Soundspaces dataset Table 2[a) Following 17 and [21], compare AV-GS against classical high-performance audio coding methods: Advanced Audio Coding (AAC) 12] and Xiph Opus [40], applying both linear and nearest neighbor interpolation techniques the coded acoustic fields. Additionally, we also compare with existing neural methods: NAF [2I]. INRAS (34), and AV-NeRF [17]. can be observed that AV-GS outperforms all the prior arts by a significant margin.
Table 2: Comparison with the cate-offthe-ann and Ablation study on physical parameters.
(a) Comparison with state-of of-the-rr: Performance SoundSpaces dataset using T60, C50, and EDT metries. Lower score indicates higher RIR generation quality. Opus an open audio codec (40), and
(b) Ablation the choice of physical parameters from G, for initializing
AAC is multi-channel audio coding standard [22
Parameters
Dimension
Overall
ENV
Methods
Audio
Visual
T60 (%) 1
C50 (dB)
EDT (sec)
S
1,451
0.141
Opus-nese nearest
X
10.10
3.58
0.115
Opus-linear
0.097
SW
48
1.435
0.140
AAC-ncarest
/
x
9.35
1.67
0.059
8.0
1.481
0.141
AAC-linear
/
x
7,88
1.68
0.057
SH.0
49
1,466
0.141
INRAS 35
/
X
3.14
0.60
0.019
8,5H
51
1.549
0.143
/
x
1.06
0.031
SN.
52
0.140
AV-NeRF
/
2.47
0,016
S,SH O
52
1,500
0.141
AV-GS (ours)
/
/
2.23
0.53
0.014
R,
S.SH,
56
1.496
0.141

---

Table 3: Comparing for efficiency in terms of the inference time with the existing AV-NeRF using multiple perspective views. Please note the inference time includes the time for rendering the view from V-NeRF, extracting features through AV-Mapper (17) and then rendering the binaural audio from A-NeRF.
Time (L)
Overall (L)
Methods
Inference
MAG
ENV
AV-NeRF- view
1.4s
1.504
0.145
AV-NeRF- 2 views
2.9s
1.495
0.145
AV-NeRF- 4 views
7.6s
1.482
0.144
AV-GS (ours)
0.08s
1.417
0.140
Improved efficiency Decoding a on-the-fly helps AV-GS retain the efficiency advantage of having an explicit point based representation over learning an implicit representation of the visual 3D scene. In contrast, AV-NeRF trains an additional NeRF model, whose outputs are used to render the listener's s view which is the condition for their binauralizer module. In Table [3] we compare the inference time for a single scene (House scene I) from the RWAVS dataset. Although the original implementation of AV-NeRF proposed a single view in the direction of listener's viewing direction (used as the listener's ' context), we modify their implementation in which the V-NeRF renders 2 and 4 perspective views with field of view of for each receiver's position. It can be observed, although using the additional views, helps provide additional local context around the listener (hence improving the binauralization performance), but only the cost of increased inference time. AV-GS on the other hand requires least rendering time, while providing the best binauralization performance. It is important to note that, although we train the 3D-GS for only an initial warm-up stage of 3k iterations, we can still jointly train both G' and Go on the corresponding camera view Ve and the auditory perspective VA respectively. By doing so, AV-GS would be able to synthesize both binaural audio as well the visual view at the unseen target (novel) viewpoint, without any increase in the training time than that of as shown Table (last row). In this work, we instead focus on only the binaural audio synthesis and hence train only G. in the second stage.
Although we train the 3D-GS for just an initial 3k iterations, we can still train both G and G. together for the corresponding camera view Ve and auditory perspective VA. such that the trained AV-GS is able to synthesize both the binaural audio and the visual view from novel viewpoint, , thereby making it a fair comparison with AV-NeRF (17). However, in this work, we focus solely on binaural audio synthesis and only train G2 in the second stage.
4.4 Ablation study
We use the binaural audio synthesis task for performing our ablation study and report average performance across all scenes in the RWAVS dataset.
Initialization of a. 3D-GS learns a point-based representation (G) of the 3D scene such that each point in the explicit representation is represented as a 3D Gaussian ellipsoid to which physical attributes like position X, quaternion R. scale S, opacity O and Spherical Harmonic coefficients (SH) representing view-dependent color are attached. As discussed above, we initialize the audio- guidance parameter using the learned physical parameters from G. Table[[() shows an in-depth study of the effect on the binauralization performance when choosing different parameters from G for the initialization. It is evident that SH and R. when considered individually as well as when combined, are crucial parameters that help provide an overall better binauralization performance.
Size of the vicinity The learned Gaussian points G representation normally consists of millions of points, and using all the points for computing the condition for the binauralization task will result in a huge computational overload. Moreover, intuitionally, points near the listener and speaker location in the 3D space (i.e. vicinity) are more contributive to determining the local geometry and cntterial-reatter characteristics, and greatly influence the sound field. Empirically in Table 4a we find that using 15 percentile of the points in the vicinity (listener and speaker considered separately) yields the best performance. Using an extremely high percentile for capturing the vicinity generally helps, but only to a certain extent, since increases the risk of adding unnecessary information. Moreover, it is important to note that in scenes with smaller room sizes when the listener and speaker are
8

---

Table (a)Ablation on the size of vicinity w.r.t the listener and sound source position. (b) Effect of audio-award point management.
(a) Percentile-k denotes the top % points nearest to the listener and sound source.
(b) SIM: Structure-from
3
(i.e.camera calibration),
G: warmup using 3D-GS
14]
Overall J
Point management
Overall
Percentile
Initialization
MAG
ENV
MAG
ENV
1.482
0.143
X
1.668
0. 1.74
SSM
10
1.454
0.141
/
1.552
0. 0.143
15
1.417
0.140
X
1.481
0.141
20
1.424
0.141
G (3D-GS)
/
1.417
0.140
1.428
0.141
00
(
60)
Figure 4: Ablation on the size of vicinity w.r.t the listener and sound source position. Percentile-k denotes the top & % points nearest the listener and sound source are considered in computing the scene context. (a) RGB color (from G), (b) learned (from G.), (c) 5% percentile vicinity, (d) 25% % percentile vicinity.
placed nearby, a larger vicinity capture might lead overlapping points thus adding to redundant information (see Fig. (d)).
Effect of audio-aware point management. AV-GS infers local geometry from explicit point- based prior and learns additional audio-specific parameters for every point. The local geometry can be inferred either directly from the sparse points produced during camera calibration using Stracturefrrom-motiin (SIM) 33 or warming up a 3D-GS model on the SfM points. From Table 4bjit is evident that audio-aware point management helps improve the binauralization performance irrespective of the adopted point initialization approach. Texture-less regions such as walls, doors, are the regions with maximum sound absorption and diversion and hence having optimal point density promotes better modeling of sound propagation.
Conclusion
In this work, we present AV-GS, a holistic scene representation learning approach for conditioning novel view acoustic synthesis. The core of AV-GS lies in the proposed decoupled modeling of 3D scene geometry and material-related characteristics of scene objects for sound propagation, enabled by the introduction of additional audio-guidance parameters per point within an acoustic field network. We show that our approach leverages audio-aware Gaussian point positioning to improve the binauralization performance, on real world as well as synthetic 3D scenes, significantly in comparison to prior art alternatives.
Limitations Although AV-GS achieves new state-of the-rart on the NVAS task by learning a holistic scene representation with geoaetry-aware and implicit material-aware characteristics, faces some challenges. Like AV-Nerf, AV-GS currently learns a separate representation for each scene, posing significant challenge for generalizability and transferability across multiple scenes, which crucial for improving efficiency and reducing computational demands. Also, as observed in Table the performance drops when the scene size increases (e.g., from Office to House), due to the difficulty in learning representations for larger, complex geometries with multiple rooms. An interesting research direction would be to model individual rooms using separate 3D-GS representations and then learn transfer function across rooms.

---

References
[1] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In CVPR, 2022.
[2] Samuel Rota Bulò, Lorenzo Porzi, and Peter Kontschieder. Revising densification in gaussian splatting. arXiv preprint arXiv:2404.06100 06/09, 2024.
[3] Changan Chen, Unnat Jain, Carl Schissler, Sebastia Vicene Amengual Gari, Ziad Al-Halah, Vamsi Krishna Ithapu, Philip Robinson, and Kristen Grauman. Soundspaces: Audio-visual navigation in 3d environments. In ECCV. Springer, 2020.
[4] Changan Chen, Alexander Richard, Roman Shapovalov, Vamsi Krishna Ithapu, Natalia Neverova, Kristen Grauman, and Andrea Vedaldi. Novel-view acoustic synthesis. CVPR, 2023.
[5] Ziyang Chen, Israel D Gebru, Christian Richardt, Anurag Kumar, William Laney, Andrew Owens, and Alexander Richard. Real acoustic fields: An audio-visual room acoustics dataset and benchmark. arXiv preprint wxXiv:2403.1882) 2024.
[6] Kai Cheng, Xiaoxiao Long, Kaizhi Yang, Yao Yao, Wei Yin, Yuexin Ma, Wenping Wang, and Xuejin Chen. Gaussianpro: 3d gaussian splatting with progressive propagation. arXiv preprint arXiv:2402.14650, 2024.
[7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
[8] Sara Fridovich-Keil. Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa, Plenoxels: Radiance fields without neural networks. In CVPR, 2022.
[9] Ruohan Gao and Kristen Grauman. 2.5 visual sound. In CVPR, 2019.
[10] Jeffrey P Grossman and William J Dally. Point sample rendering. In Rendering Techniques' 98: Proceedings of the Eurographics Workshop. Springer.
[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[12] International Organization for Standardization. Advanced audio coding (aac). ISOЛEC 13818- 8- 7:2006, 2006.
[13] Nikhil Keetha, Jay Karhade, Krishna Murthy Jatavallabhula, Gengshan Yang. Sebastian Scherer, Deva Ramanan, and Jonathon Luiten. Splatam: Splat, track & map 3d gaussians for dense rgb-d slam. arXiv preprint arXiv:2312.02126, 2023.
[14] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 2023.
[15] Sijia Li, Shiguang Liu, and Dinesh Manocha. Binaural audio generation via multi-task learning. ACM Transactions on Graphics, 2021.
[16] Zhan Li, Zhang Chen, Zhong Li, and Yi Xu. Spacetime gaussian feature splatting for real-time dynamic view synthesis. arXiv preprint arXiv:2312.16812. 2023.
[17] Susan Liang, Chao Huang, Yapeng Tian, Anurag Kumar, and Chenliang Xu. Av-nerf: Learning neural fields for real- weal-wrrdd audio-visual scene synthesis. NeurlPS, 2023.
[18] Yan-Bo Lin and Yu-Chiang Frank Wang. Exploiting audio-visual consistency with partial supervision for spatial audio generation. In AAAI, 2021.
[19] Francesc Lluis, Vasileios Chatziioannou, and Alex Hofmann. Points2sound: from mono to binaural audio using 3d point cloud scenes. EURASIP Journal on Audio, Speech, and Music Processing, 2022(I):33, 2022.
10

---

[20] Tao Lu, Mulin Yu, Linning Xu, Yuanbo Xiangli, Limin Wang, Dahua Lin, and Bo Dai. Scaffold- gs: Structured 3d gaussians for view-adaptive rendering. arXiv preprint arXiv:2312.00109, 2023.
[21] Andrew Luo, Yilun Du, Michael Tarr, Josh Tenenbaum, Antonio Torralba, and Chuang Gan. Learning neural acoustic fields. NewrIPS, 2022.
[22] Andrew Luo, Yilun Du, Michael J Tarr, Joshua B Tenenbaum, Antonio Torralba, and Chuang Gan. Learning neural acoustic fields. 2022.
[23] Sagnik Majumder, Changan Chen, Ziad Al-Halah, and Kristen Grauman. Few-shot audio-visual learning of environment acoustics. In NeurIPS, 2022.
[24] Hidenobu Matsuki, Riku Murai, Paul HJ Kelly, and Andrew Davison. Gaussian splatting slam. arXiv preprint arXiv:23/2.06744, 2023.
[25] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. ECCV, 2020.
[26] Pedro Morgado, Nuno Nvasconcelos, Timothy Langlois, and Oliver Wang. Self-supervised generation of spatial audio for 360 video. NeurIPS, 2018.
[27] Kranthi Kumar Rachavarapu, Vignesh Sundaresha, AN Rajagopalan, al. Localize binaural- ize: Audio spatialization from visual sound source localization. In CVPR, 2021.
[28] Anton Ratnarajah, Sreyan Ghosh, Sonal Kumar, Purva Chiniya, and Dinesh Manocha. Av-rir: Audio-visual room impulse response estimation. arXiv preprint arXiv:2312.00834, 2023,
[29] Anton Ratnarajah and Dinesh Manocha. Listen2scene: Interactive material-aware binaural sound propagation for reconstructed 3d scenes. IEEE VR, 2024.
[30] Anton Ratnarajah, Zhenyu Tang, and Dinesh Manocha. Ir-gan: Room impulse response generator for far-field speech recognition. arXiv preprint arXiv:2010.13219, 2020.
[31] Anton Ratnarajah, Shi-Xiong Zhang, Meng Yu, Zhenyu Tang, Dinesh Manocha, and Dong Yu. Fast-rir: Fast neural diffuse room impulse response generator. In ICASSP, 2022.
[32] Kazuki Shimada, Archontis Politis, Parthasaarathy Sudarsanam, Daniel A Krause, Kengo Uchida, Sharath Adavanne, Aapo Hakala, Yuichiro Koyama, Naoya Takahashi, Shusuke Taka- hashi, et al. Starss23: An audio-visus dataset of spatial recordings of real scenes with spa- tiotemporal annotations of sound events. NeurlPS, 2023.
[33] Noah Snavely, Steven M Seitz, and Richard Szeliski. Photo tourism: exploring photo collections in 3d. In ACM siggraph 2006 papers, pages 835-846. 2006.
[34] Kun Su, Mingfei Chen, and Eli Shlizerman. Inras: Implicit neural representation for audio scenes. Advances in Neural Information Processing Systems, 35:8144-88 14448158,, 2022.
[35] Kun Su, Mingfei Chen, and Eli Shlizerman. Inras: Implicit neural representation for audio scenes. In NewrlPS, 2022.
[36] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309. /6653, 2023.
[37] Ayush Tewari, Justus Thies, Ben Mildenhall, Pratul Srinivasan, Edgar Tretschk, Wang Yifan, Christoph Lassner, Vincent Sitzmann, Ricardo Brutin, Stephen Lombardi, et al. Ad- vances in neural rendering. In Computer Graphics Forum, pages 703-735. Wiley Online Library, 2022.
[38] Ruiqi Wang, Haonan Cheng, Long Ye, and Qin Zhang. Visual-guided scene-aware audio generation method based on hierarchical feature codec and rendering decision. Displays, 83:102708, 2024.
11

---

[39] Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie. Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. arXiv preprini arXiv:2310008525 2023.
[40] Org Foundation. Xiph opus. https://opus-codess org/ 2012.
[41] Xudong Xu, Hang Zhou, Ziwei Liu, Bo Dai, Xiaogang Wang, and Dahua Lin. Visually informed binaural audio generation without binaural audios. In CVPR, 2021.
[42] Zeyu Yang. Hongye Yang, Zijie Pan, Xiatian Zhu, and Li Zhang. Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting. In ICLR, 2024.
[43] Hang Zhou, Xudong Xu, Dahua Lin, Xiaogang Wang, and Ziwei Liu. Sep-stereo: Visually guided stereophonic audio generation by associating source separation. In ECCV, 2020.
[44] Qian- Yi Zhou, Jaesik Park, and Vladlen Koltun. Open3d: A modern library for 3d data processing. arXiv preprint arXiv: (501::880.004.. 2018.
[45] Matthias Zwicker, Hanspeter Pfister, Jeroen Van Baar, and Markus Gross. Surface splatting. In Annual conference Computer graphics and interactive techniques, 2001.
12

---

A Appendix supplemental material
Broader impact Potential applications of AV-GS lie in virtual reality, augmented reality, and im- mersive audio experiences, where accurate and efficient binaural audio rendering can significantly enhance user experience. However, enhanced acoustic synthesis technology could be used unethi- cally in surveillance or privacy-invasive applications, where binaural audio could be synthesized to eavesdrop on conversations or create misleading audio experiences.
A.1 Direction and distance awareness showcased by AV-GS
Fig- highllights the distance-awarence exhibited in the binaural audios synthesized by AV-GS. The top row shows the corresponding RGB frame or the listener's view, followed by AV-GS's s learned holistic scene representation in the next row. The two (left and right respectively) with time along the x-axis and amplitude of the audio along the y-axis. (Please that in the second row slice the points in G, for along the vertical axis, essentially removing the points from the ceiling, in order to provide better visibility.) With the increase in the distance of the listener (blue sphere) from the sound source (yellow sphere), AV-GS able synthesize binaural audios with decreasing amplitude across both the left and right audio channels.
show plot of the rendered binaural audios
Increasing distance from the sound source
R
Decreasing amplitude in the binaural audio
Figure 5: Distance audio rendering. As the distance from the sound source increases the amplitude the synthesized binaural audio decreases. Yellow sphere location the sound source, blue sphere- location of the listener.
Fig. 6 shows the direetion-awareness showcased by AV-GS. In (a) the synthesized left channel has higher magnitude compared to the right channel, owing the proximity of the listener's left car to
13

---

the sound source. Vice-versa, in (b) the right channel has a higher amplitude compared to the right channel owing the proximity of the listener's right ear to the sound source.
R
Figure 6: Direction aware audio rendering. Relative to the viewing direction of the listener w.r.t to the sound source AV-GS synthesizes binaural audios with varying amplitude levels the left and right audio channel. The arrow on the listener (blue sphere) represents the viewing direction, and the yellow sphere depicts the sound source.
A.2 Binauralizer
8 8
STFT
amono
m7
query
ISTFT
ap
B
8 m
m7
8
ISTFT
Embedding
Embedding
frequency
X
Xs
Figure 7: Overview of the binauralizer module, B (with modifications from 17
We adopt the architecture of binauralizer from 17 with some modifications to input the learned scene context from our acoustic field network F (see Fig. 7
Embedding blocks take the normalized location position X, (only 2D and y coordinates).
The main components include two Multilayer Perceptrons (MLPs), each comprising four linear layers with an additional residual connection. The width of each linear layer, is set to 128 for the RWAVS dataset and 256 for the SoundSpaces dataset. All linear layers are followed by ReLU activation layers, except for the last layer, where the ReLU activation is replaced with the Sigmoid function. The first MI.P takes the listener's position (x, y) and the frequency f [0, F] as input, where F represents the number of frequency bins. It predicts mixture mask m.m for the given frequency of and generates
14

---

feature vector with channels. Prior to feeding them into the MLP, we apply positional encoding to the listener's s position (x, y) and the frequency f. We the maximum frequency used for positional encoding as 10. A coordinate transformation [17] to project the listener's direction & into a high frequency space. We concatenate the transformed listener's direction and the feature vector, and feed it into the second MLP. The second MLP is appended with Sigmoid layer and scaling layer, ensuring that the difference mask ma estimated by the second MLP falls within the range of -1, 1. For each frequency query f. estimates two masks: mm and m.s both of which are scalars. We iterate over all frequencies
[0, F] to obtain the complete masks Mm and mad
For validating AV-GS on the room impulse response generation task using the Soundspaces dataset, following [17]. we drop the mixture mask, and predict the impulse response directly for correspond- ing time input, using the second MLP. We show these modifications in Fig.
query
q
B
Impulse respose (IR)
Embedding
Embedding
t
Time
X
X
S
Figure 8: Changes to B for the RIR generation Soundspaces dataset (scores reported Table a)).
15

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the
relevant information is Not Available.
Please provide short (1-2 sentence) justificatinn right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors their evaluation.
While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] provvdded proper justification is given (c.g., "error bars are not reported because it would be too computationally expensive" or "we
unable to find the license for the dataset used"). In general, answering "[No] or "[NA]" is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the
paper's contributions and scope?
Answer: [Yes]
Justification;
The sections of Methods and Experiments clearly describe the claims made.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations are discussed in Section 5
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: the paper does not include theoretical results.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
16

---

Answer: [Yes]
Justification: We provide all details for hyper-parametee as well as train and test settings wherever applicable. Additionally, will publicly release our entire codebase along with pre-trained models upon acceptance.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We will publicly release our entire codebase along with pre-trained models on our project page (link provided in abstract). The datasets used are publicly available.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Discussed in Section 3 Model Training
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Seeds are fixed to provide reproducibility of the results.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Provided in Section Implementation details, well as, detailed discussion provided regarding inference and training times.
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: All guidelines have been followed.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss broader impact as a part of our Conclusion.
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: Our paper poses no such risks.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
17

---

Answer: [Yes]
Justification: All existing data and code that has been used by us is properly cited and all of it is under the CC-BY 4.0 license.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: the paper does not release new assets.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects. 15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
18


Reviews:
[{"review_id": "GXddyBSQeO", "reviewer": "Reviewer_m6xj", "summary": "The paper proposes a 3D Gaussian splatting (3DGS) based method for improving the acoustic context modeling for novel view acoustic synthesis (NVAS). The proposed model outperforms multiple existing methods on both simulated and realworld data across different metrics. The method also trains and infers faster than the state-of-the-art AV-Nerf method. The paper also provides model ablations and analyses that help in better understanding the role of different model components.", "questions": "Could the rebuttal comment (more) on the following?\n1. how the chosen strategy for aggregating the acoustic context in G_a affects model performance. See weakness 1 for details.\n\n2.  the rationale behind using L_m, and how the model performs without the loss. See weakness 2 for details. \n\n3. initialization of alphas during point densification in G_a. See weakness 3 for details.\n\n4. generalization of the model to more challenging datasets like Matterport3D. See weakness 4 for details.  \n\n5. my minor comments. See weakness 6 for details.\n\nAlso, could the authors anonymously provide a few audio examples from the model and baselines?", "limitations": "The paper mentions limitations in Discussion but I could not find any discussion on societal impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of using 3D Gaussian Splatting for better modeling of finegrained geometry and material aware acoustic context for improving the NVAS performance is interesting, novel and works well in practice.\n\n2. The paper reports strong results, where the proposed model outperforms existing methods on both simulated and realworld data\n\n3. The provided model ablations and analyses help in better understanding the contributions of different model components and design choices", "weaknesses": "1. Acoustic conditioning with G_a: the paper \"averages the context across all points in G_a\" (L156-7). Doesn't this lead to loss of finegrained point-specific information? A learned aggregation strategy (for example, by using the attention mechanism) would probably make more sense here?\n\n2. L192-4, \"L_v ... non-overlapping\": \n    i) the insight behind doing this is not clear from the text\n    ii) there are no references to support these steps\n    iii) there is no ablation for this loss, as well\n\n3. L208, \"The audio guidance ... random initialization\": won't random initialization during point expansion lead to local 'discontuity'? Did the authors try warm initialization to preserve local 'continuity' by doing nearest neighbor search or bilinear interporation on the existing alphas?\n\n4. Does the model generalize to more challenging scene datasets like Matterport3D [1]? \n\n5. Audio examples: the paper does not seem to provide audio examples and qualitiatively compare their quality with that of the baselines\n\n6. Minor:\n   i) L274-5, \"4 perspective views ... receiver's position\": do these 4 views cover the full 360 degree, otherwise it's not comparable with the field of view of the AVGS model?\n   ii) Para in L285-90 looks like repeated text\n\n\nReferences: \n[1] Matterport3D: Learning from RGB-D Data in Indoor Environments. Chang et al. 3DV 2017.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Questions (and weaknesses)\n\n**W1, Q1: how the chosen strategy for aggregating the acoustic context in G_a affects model performance.**\nL156-7: \"We obtain the condition for binauralization by averaging the context across all points in G_a, post dropping points outside the vicinity of the listener ..\"\n\nIn every iteration a random listener location is sampled. \nPlease note from Eq. (3) that the formed context is already conditioned on the direction (position-guidance), which is already an explicit attention.\nThe context averaging step happens only for the points within the vicinity (which is a smaller subset of all points that are representing the whole scene, ~400k). \n\nWe did try to learn an attention-based weighing among the vicinity points; however we often found that the weight predictor collapses terminating the optimization under a poor local minima (i.e. all points are assigned the same weight, 0.5 for a sigmoidal activated weight predictor) due to the sufficiently varied points provided in every iteration (the vicinity keeps changing drastically across every iteration depending on the sample listener location). One might think to assign weights to all points in the scene instead of  only the ones within vicinity, however this is very compute intensive for a single iteration.\n\n**W2, Q2: the rationale behind using L_v, and how the model performs without the loss.**\nApologies for not being clear. Our intuition behind introducing alpha is to help us determine each point's contribution to the context used by the binauralizer for generating binaural audio (for a particular listener location). The regularization term encourages lower alpha to prevent over-reliance on a few points, and make the contribution of points (for forming the context) well-distributed, promoting a diverse and representative context which is key for generalization to unseen listener locations.\n[10] proposed a similar regularization term however their purpose is significantly different from ours, where they use it for volume minimization to curb 'visual overlap' and enable faster raymarching.\nWe empirically show below that for AV-GS generally lower values of \\lambda_a (contribution of regularization loss, Eq. (6)) are preferred.\n\n| lambda_a | MAG (↓) | ENV (↓) |\n|----------|---------|---------|\n| 0.0      | 1.43    | 0.140   |\n| 0.01     | 1.417   | 0.140   |\n| 0.1      | 1.440   | 0.140   |\n\n**W3, Q3: initialization of alphas during point densification in G_a.**\nThat's a great question. We tried warm initialization using both repeat (similar to vanilla 3D-GS and Scaffold-GS) as well as the nearest neighbor (NN). In comparison to random initialization, they perform almost the same. We hypothesize this firstly because the number of points (within the vicinity) that satisfy the gradient condition are much smaller, secondly, the densifying step is carried after intervals of 100 iterations, which gives the optimization (which happens every iteration) sufficient space to curb discontinuity, if any. (Not to forget the outlier removal step, carried after an interval of 3k iterations).\n\n| Alpha initialization | MAG (↓) | ENV (↓) |\n|----------------------|---------|---------|\n| random               | 1.417   | 0.140   |\n| repeat               | 1.414   | 0.140   |\n| NN                   | 1.419   | 0.140   |\n\n**W4, Q4: generalization of the model to more challenging datasets like Matterport3D.**\nWe currently validate AV-GS on a real-world RWAVS and a synthetic dataset Soundspaces, which is in line with prior work in the field of novel view acoustic synthesis - NAF (NeurIPS'22), INRAS (NeurIPS'22), AV-NeRF (NeurIPS'23). As for challenges, RWAVS (real world dataset) is more challenging than Matterport3D (synthetic), especially the house and apartment scenes involving multiple real rooms in a single scene. Currently both RWAVS and soundspaces-synthetic are publicly available. We can incorporate this into our revised version with similar data (binaural audio and RGB pairs) for Matterport3D.\n\n**W5: Audio examples**\nBecause of rebuttal instructions regarding not sharing links, we have provided the AC with an anonymized link to the audio samples. Apologies for the inconvenience. We will also make our code repo and all rendered samples publicly available post the review period.\n\n**W6, Q5: Minor: i) L274-5, do these 4 views cover the full 360 degree? ii) Para in L285-90 - repeated text**\n(i) That is right, all 4 perspective views cover the 360 degrees of listener view. In Table 3, despite the 360 view AV-NeRF falls short of binauralization performance due to lack of a combined holistic scene condition, unlike our method. (ii) Got it, we will remove L285-90 it in the update.\n\n**Limitations:**\nSorry for the confusion, the impact was provided in the \"Appendix / supplemental material\". We will update it.\n\nImpact: Once an AV-GS has been trained to accurately model room geometry and material properties, synthesizing highly realistic binaural audios in real time should be possible. This could be used to create misleading audio experiences, such as simulating the presence of people or activities that are not actually occurring, which could be used to deceive or manipulate individuals.\n\nReferences:\n\n[10] Lombardi, Stephen, et al. \"Mixture of volumetric primitives for efficient neural rendering.\" ACM ToG", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kwMELMPFlr", "reviewer": "Reviewer_hPZr", "summary": "This work proposes an efficient and performant method for novel view acoustic synthesis using Gaussian Splatting representations of 3D scenes. The proposed model integrates geometry and material conditioning, and offers a way to bridge the gap between visually-useful 3DGS representations and acoustically useful ones, using “audio-guidance” which considers the local environment in Experiments show this outperforms existing NeRF-based approaches while being more efficient.", "questions": "- Is $S$ modeled as an omni-directional point source? I assume so because point source is the typical assumption, and an emission direction isn’t given (vs. listener direction d), but it’s worth clarifying.\n- I think the notation here could be clearer. For example, $V_C$ and $V_A$ don’t seem used, and it’s not clear to me what their role is? E.g. why is the observation $O_p$ = ($V_C$, $V_A$) instead of ($I$, $a_{bi}$)? If the pose $p$ captures the position and orientation, what do $V_C$ and $V_A$ add on top of this representation? Why is $I$ upper-case and $a$ lower-case here, when they both represent observations? Why is $a_{bi} = \\{a_l, a_r\\}$ notated as a set, but other sets are notated as tuples? Additionally, shouldn’t $k_l$ be $k_L$, similar for $S$?\n- Conceptually, there is a tension between considering the global scene context, as the paper argues for, and dropping points outside the local vicinity of the source and listener (since sound reflections can easily include points from other regions, in principle). Could the authors comment on this? It’s interesting that the optimal vicinity size is not the largest one tested. Is there some intuition for why this might be the case, given that it offers more information? Currently, the reasoning seems to be that this contains unnecessary information, but the useful information seems to be a subset of this?\n- The audio rendering task is cast as a binauralization task. I understand that this is practically necessary, i.e. the input is mono and output is stereo binaural. However, wouldn’t many of the problems in this paper still exist in the monaural listener context as well? E.g. compared with a binauralization task purely based on HRTFs or simplified room geometries (e.g. Richard et al., ICLR 2021), where the context $C$ is not as rich. Could this be clarified?\n- How was the $\\tau_g$ value decided?", "limitations": "I think the authors have fairly represented the limitations. The broader impact statement is a little bit less clear to me, i.e. I’m not sure I understand the surveillance applications. Could this be clarified? I would be interested for the authors to consider the potential impact of the limitations listed. For example, the limitation of rendering larger scenes; what practical negative impacts could this have in deployment scenarios?", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Overall, I think this work makes a very useful contribution. Bridging Gaussian Splatting representations with audio rendering is likely to be of significant interest to the community, and the quantitative results and efficiency seem strong. The experiments and comparisons are helpful, and give me confidence that these results are quite robust.", "weaknesses": "The value of $\\tau_g$ seems somewhat arbitrarily defined, and the range of tested vicinity sizes is somewhat narrow; given the efficiency of the method. It’s hard to extrapolate what the performance would be with more context (i.e. would it keep going down?). Most of my other suggestions are relatively minor, about the writing and presentation:\n- Abstract: “at a 3D scene” -> for clarity, should this be “within” a 3D scene?\n- Introduction: “visual rendering of 3D scenes without spatial audio (i.e., deaf)” -> this seems like an inaccurate characterization. Deafness is not about rendering, but about sensation. Spatiality is also not clearly related to this.\n- Introduction: “realistic binaural audio synthesis is challenging, since the wavelengths of sound waves are much longer” -> this is not the only reason that this is challenging. Even if you use a simplified (e.g. ray-tracing) model, which would be inaccurate at lower frequencies due to wave-like behavior etc., you would still need a good model of the environment for physics-based rendering, or a large number of spatial measurements for a data-driven approach.\n- Introduction: “…modeling 3D geometry and material characteristics of the visual scene objects to instigate direction and distance awareness…” -> while geometry has a direct impact on “direction and distance”, material impacts spatial perception indirectly (through absorption, reflection, and diffusion). Could this be rephrased more clearly?\n- Related work: “Anton and Dinesh” should maybe be “Ratnarajah and Manocha”, i.e. last names of authors?\n- Related work: Under “Geometry and material conditioning”, the final statement differentiates the current work from prior work by pointing out the non-reliance on scene and geometry inputs, but doesn’t clarify why this is useful. E.g. it could be pointed out that, in the real world, such inputs are often not available, so this work allows generalizing to such settings.\n- Figure 2: “Differential Rasterizer” -> “Differentiable Rasterizer”?\n- Figure 4: Could there be a subfigure showing 15%, since that seems to give the best overall performance? It would be helpful context, since otherwise the reader needs to interpolate between (c) and (d).\n- Section 4.4: “intuitionally” -> “intuitively”?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Weakness \n\n**W1: Value of T_g**\nT_g is inspired by vanilla 3D-GS paper (which empirically found a threshold of 0.0002) and consistently our threshold of 0.0004 was determined through empirical testing.\n\nIn 3D-GS optimization (trained for NVS), the point's gradient is compared with the T_g threshold, and is decided whether the point should be cloned/split or not. Particularly in this case the gradients correlate to ability of the points to contribute to the visual geometry of the scene when they are splatted in a particular camera-view direction. In our NVAS case, the gradient of the points correlate to the contribution of the point in providing an acoustic guidance when generating binaural audio for a given {X_L, X_S} pair. Please note NVAS is not limited to only the camera-view direction, but rather vicinity of the listener and sound source.\n\nOur empirical choice is in-line with 3D-GS, and also intuitively higher -because we adopt a dual stage training, wherein the rough geometry is already provided to 2nd stage (Structure-from-Motion points are provided to the first stage), and comparatively considered for a much different sample of points (volumetric camera view Vs listener and source vicinity). A higher value helps to save compute cost.\n\nGiven a reasonable memory budget, ideally one would want to empirically decide on the threshold for every different vicinity size.\n\n**Minor comments** We sincerely appreciate the discussion regarding general acoustics, we will incorporate this in our next version with the 15% vicinity plot. We will relate our Intro section to the wave-based (good for lower frequency bands --> resulting in diffusion, scattering .. i.e.  wave properties) and Geometric acoustic-based methods (good for higher frequencies --> behaves like rays --> ideally specular reflections .. i.e. ray properties).\n\n## Question \n\n**Q1: S omni-directional?**\nRight, the sound source is omni-directional, we adopt the same dataset as AV-NeRF [4]. We will mention this explicitly in our revised version.\nTo extend AV-GS for directional sources, directional components can be added (perhaps weighing the vicinity sphere in the direction).\n\n**Q2: Notations**\nSincere apologies for the confusion.\nV_C and V_A: We tried to use V to represent generic views; V_C - visual and V_A - audio modality.\n\nV_C = \\{I | p\\}; V_A = \\{a_{bi} | p, a_{mono}\\}\n\nFor the current scope of input used by AV-GS, you are right in saying, V_C = I and V_A = a_bi. However, in a more generalized NVAS setting where one may supplement the RGB with a depth view, in which case V_C = I+D (RGB + Depth). \n\nI and a: As mentioned these are both observations, however, 'I' is a 3D matrix (RGB) hence, uppercase. 'a' is the 1D audio (time series sequence) and hence lower case. We will update this.\n\nTuple or set: Apologies, all should be notated as tuples, and k_L and k_S.\n\n**Q3: Dropping points outside the vicinity …, optimal vicinity is not the largest one tested. Why?**\nAgain, Great question! Although it might intuitively feel enlarging the vicinity size would help, however please note in the context of the GS learned point based representation, we are typically looking at a scale like ~400k points for a single room scene. Increasing the vicinity would account for adding this huge number of parameters which would make it harder for the field network to be burdened - It is a trade-off. We did try voxelizing the representation, so that we can deal with lesser points (and in turn lesser alpha's) despite increasing the vicinity, but it worsened the performance -we hypothesize this is because the alpha parameter for the points post-voxelization cannot be computed with a simple average (or even a MLP, like anchor points - Scaffold GS [9]). A specialized attention based selection of points for the vicinity might be an interesting future direction to explore, but please note not all points closer to the source or listener are actually (more) helpful. \n\n**Q4: Wouldn't many of the problems in this paper still exist in the monaural listener context as well?**\nIt is correct in noting that many of the challenges in this paper are relevant to monaural contexts as well.\nThe primary contribution of AV-GS lies in capturing a holistic context of the entire scene and generating context within the 3D scene. Binauralizer arch. is adopted from AV-NeRF (who partly adopted from [1]), not claimed as a novel bit of this work. \nIn smaller shoebox rooms (like the one's experimented in Richard et. al ICLR'21, and Office scene in RWAVS), AV-GS's improvement is less pronounced due to limited room geometry impact.\nBut, AV-GS shows significant improvements in multi-room more complex scenes (e.g., \"house\" and \"apartment\" in RWAVS) with wall occlusions, where AV-NeRF's AV-Mapper is less efficient. (scores in Table 1 of paper)\nMulti-room scenes give rise to multiple wall occlusions - where AV-NeRF's AV-mapper approach is inefficient.\n\n**Q5: Value of T_g**\nPlease check W1 above.\n\n**Limitations: Broader impact unclear**\nThanks, we will clarify. Once an AV-GS has been trained to accurately model room geometry and material properties, synthesizing highly realistic binaural audios in real time should be possible. This could be used to create misleading audio experiences, such as simulating the presence of people or activities that are not actually occurring, which could be used to deceive or manipulate individuals.\nWith regards to rendering large scenes, we agree, that generalization to multiple room (4 or 5, or a multi-storey house) is indeed challenging. Additionally, the problem with adopting an explicit points based representation is that the number of points increases drastically for larger scenes, where although capturing fine-grained details is not necessary (as in the case of NVS), a rough geometry is still a crucial requirement for points to be able to guide NVAS.\n\n\nReferences:\n\n[9] Lu, Tao, et al. \"Scaffold-gs\" CVPR'24", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XDcTUhcgUE", "reviewer": "Reviewer_RHwj", "summary": "This paper proposes a new audio-visual Gaussian Splatting model for novel view acoustic synthesis. The proposed method explicitly models scene geometry and learns a point-based scene representation with an audio guidance parameter on locally initialized Gaussian points that takes the space relation from the listener and sound source into consideration. This work is the first that applies gaussian splatting to the novel view acoustic synthesis problem, and experiments on two datasets demonstrate the effectiness of the proposed method compared to prior work.", "questions": "- See questions in the weakness section above.\n\n- Also, for the SoundSpaces synthetic dataset, why it only contains 6 indoor scenes. There are more scenes in the dataset, and it would be good to explain the reason why only these six are not used.\n\n- Is there any way to interpret the learned audio-guidance parameter to show that it’s material related?", "limitations": "Yes some limitations are discussed in the end, though some might be missing as discussed above.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The idea makes sense, and I was expecting gaussian splatting to be applied to this problem by somebody sooner or later, and this paper is indeed doing that.\n\n- Generally, the paper makes good attempts to apply Gaussian Splatting to this recently proposed task, a direct extension on how it is used for visual rendering. The method is clearly formulated and defined with informative notations.\n\n- Experiments are conducted on two datasets following prior work AV-NeRF [17], and it shows noticeable gains compared to prior baselines.\n\n- Many alation studies are performed including ablation on the size of vicinity w.r.t the listener and sound source position, ablation on the choice of physical parameters, which is extensive.", "weaknesses": "- There is a strong claim that the proposed model is learning material-aware priors, jointly modeling 3D geometry and material characteristics, holistic scene geometry and material information, etc. Then, I would actually expect materials are explicitly modeled, such as acoustic properties of surfaces in the form of some material parameters (e.g., absorption coefficients, Young’s modulus, etc.). However, all that is modeled is just a parameter alpha that claims to encapsulate material-specific characteristics of the scene. This is somewhat misleading as there is nothing constrains alpha to be physics-based or to be correlated with some material parameters. It's likely that it’s just learning some scene representation that may be more helpful for acoustic rendering.\n\n- Also the so-called acoustic field network is also nothing about audio by itself. By acoustic field network, I was thinking the method is explicitly modeling sound propagation by taking into account the material properties of surfaces and the geometry of the space. The audio binauralizer is basically following prior work AV-NeRF [17], and what changes is what the audio binauralizer conditions on. In AV-NeRF, it’s scene features from NeRF and in this work it’s the context scene features from gaussian splatting.\n\n- There were no qualitative examples given in supplementary materials, so it's really hard to appreciate the improvement that this method leads to.\n\n- There are also many typos or grammar mistakes here and there. For example:\n1. L45, 3D Gaussian splatting-based *methods*?\n2. Related work, mixed use of past tense and present tense. \n3. L125, a brief ??", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Weakness\n\n**W1: Explicit modeling of alpha**\nWe appreciate the reviewer's insight regarding the explicit modeling of alpha. \n\n*\"there is nothing constraining alpha to be physics-based or correlated with material parameters\"*\n\nWhile we acknowledge the importance of deriving alpha from material properties input, we would like to clarify that AV-GS is motivated by the latest trend towards scalability and adaptability for real-world scenes like RWAVS[4] and Real acoustic field[7], where only RGB frames are available – a more practical and least constrained setting. In contrast, other works [5] [8] using absorption coefficients rely on synthetic scenes and pre-existing absorption databases, matching object labels with material descriptions which are available for synthetic scenes but typically unavailable in real world scenarios as considered in our work.\n\nPlease consider that currently, anyone with a phone and a binaural microphone can capture all input data that is required to train an AV-GS model (~4 min of recording for a single room scene) - something not possible if explicit material properties are required for model training.\n\nWe agree that alpha captures an implicit, rather than explicit, material representation derived from raw RGB images and will clarify this in the revised version. \n(Also, please see Q2 below, where we demonstrate that alpha, although implicitly learned, is related to material properties.)\n\n**W2: Explicit modeling of acoustic field network**\nExplicit sound propagation modeling is indeed valuable, but our acoustic field network design is intentionally non-trivial. We learn acoustic properties on 3D-GS points, accumulating points within an empirically validated vicinity of the listener and source. We fuse implicit material (alpha) with learned position guidance, using point view directions relative to listener/source. This is further facilitated by audio-aware adaptive density control to priortise point density across texture-less regions -something vanilla 3D-GS lacks, but is important for NVAS. We agree explicit beam-tracing within audio-guided 3D-GS is an interesting future research direction.\n\nRegarding Binauralizer, as mentioned in L166, it is not a part of our contribution. It was motivated in [1] and adopted by AV-NeRF[4]. We further improve it by deriving holistic conditions from 3D scenes, specifically- learning implicit material properties, fusing with position guidance, and adaptively densifying the point-based representation w.r.t audio reconstruction loss.\n\n**W3: Qualitative audio samples**\nBecause of rebuttal instructions regarding not sharing links, we have provided the AC with an anonymized link to the audio samples. Apologies for the inconvenience. We will also make our code repo and all rendered samples publicly available post the review period.\n\n**W4: Typos**\nApologies! we will address all the typos in our revised version.\n\n## Question\n\n**Q1: only 6 scenes from Soundspaces?**\nWe chose six indoor scenes from the SoundSpaces dataset to ensure consistency and valid fair comparisons with previous works, specifically NAF (NeurIPS'22), INRAS(NeurIPS'22), and AV-NeRF(NeurIPS'23), which established benchmarks using these six scenes. They represent a diverse range of environments: (1) Two single rooms with rectangular walls, (2) Two with non-rectangular walls, and (3) Two multi-room layouts.\nCurrently both RWAVS and soundspaces-synthetic are publicly available, and hence used for our fair and consistent validation. We can incorporate this into our revised version with similar data (binaural audio and RGB pairs) for other scenes.\n\n**Q2: Interpretability of audio-guidance parameter**\n(The PCA plot and the correlation matrix discussed below have been provided in the PDF attached above, in the common \"Author Rebuttal\")\nFor RWAVS scene 6, we analyze alpha by plotting its first two principal components and applying k-means (k=3) clustering, to project colors onto G_a. It can be observed that most of the objects are grouped by the same color (3 clusters - orange, green, blue). Further, we also manually match scene objects (e.g., fireplace, walls) with material labels (e.g., 'brickwork', 'solid wall' respectively) from an absorption database [6]. Using these absorption coefficients, we assign absorption vectors (Dim: [1x6]; coefficients for 6 frequencies from [6]) to every point that corresponds to the scene object matched with the material label from [6]. A correlation matrix of these absorption coefficients, across the points (grouped by clusters), shows high self-correlation within clusters and lower cross-correlation across clusters, i.e., the points within the same clusters have similar absorption values, and less correlated absorption across the clusters. While these clusters do not explicitly highlight fine-grained material properties such as Young's modulus, they provide implicit material cues for binauralization.\n\nAdditionally, we demonstrate the effect if alpha is frozen during our training (initialized using SH from vanilla 3D-GS). As shown in the table below, constraining alpha to SH results in a significant drop in binarualization. \n\n| Alpha- trainable | MAG (↓) | ENV (↓) |\n|------------------|---------|---------|\n| No               | 1.584   | 0.147   |\n| Yes              | 1.417   | 0.140   |\n\nCombined with the PCA/correlation matrix above, and this experiment, we hypothesize that the learned alpha is able to pick up implicit material properties on top of SH to aid binauralization.\n\nReferences:\n[1] Gao, Ruohan. \"2.5 d visual sound.\" CVPR’19\n\n[4] Liang, Susan, et al. \"AV-NeRF.\" NeurIPS’23\n\n[5] Ratnarajah, Anton \"Listen2Scene\" IEEE VR’24\n\n[6] C. Kling. Absorption coefficient database, Jul 2018.\n\n[7] Chen, Ziyang, et al. \"Real acoustic fields\" CVPR’24\n\n[8] Tang, Zhenyu, et al. \"GWA\" SIGGRAPH’22.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KBlsSgLOZb", "reviewer": "Reviewer_DaxK", "summary": "This paper proposes a novel view acoustic synthesis approach based on 3D Gaussian Splatting scene representation. The framework consists of a 3D GS model, acoustics field network, and audio binauralizer. It first trains a 3D GS model to capture scene geometry. Then they use attributes from each Gaussian to initialize a learnable acoustic point representation, and then average the feature obtained from nearby Gaussian points for acoustic modeling. \nThrough the experiments on the RWAVS and Soundspaces dataset, the authors demonstrate that the proposed method achieves state-of-the-art performance. They also perform the ablation study to ablate some components to demonstrate their technical advance.", "questions": "I have mentioned my concerns and questions in the weaknesses part and I hope the authors could address them during the rebuttal.\n\nA suggestion to the authors for paper writing: save the figures as PDF instead of screenshots or png.", "limitations": "Yes, the authors adequately addressed the limitations", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper brings up a clever way to leverage point-based scene representation from 3DGS for acoustic modeling and the results are promising. \n- The authors perform thorough experiments and ablations to demonstrate its technical advance. The proposed method outperforms baselines on the quantitative evaluation.", "weaknesses": "- The reviewer thinks there are several issues with the writing:\n  - First, **the method part isn't clearly stated**. It takes a long time for the reviewer to fully understand the method. The method is simple, using pretrained 3DGS to initialize a learnable acoustic point representation. While the method figure doesn't quite help with understanding. \n  - The reviewer strongly feels the paper is **overclaimed**. It claims it learns \"holistic geometry-aware material-aware scene representation\" while the experiments are only for acoustic modeling results. The author needs to use experiments to support this claim. One thought the reviewer has to visualize the PCA feature of learnable acoustic points to see if it has some cues. Otherwise, the reviewer is only convinced that this paper proposed a better acoustic modeling approach that leverages the GS features. \n  - The major experimental **results for the baselines are directly brought from AV-NeRF and INRAS papers** while the authors don't mention this in the paper. Also, the authors need to confirm if they have guaranteed the same training and test split and evaluation code.\n\n- The reviewer has a question on the Position-guidance G condition. The authors use unit vectors which get rid of distance information. The reviewer is confused as to why remove the distance. Would it be an important cue?\n- In the paper [5] Real acoustic field, the authors found out that energy decay loss shows a powerful improvement on those energy-based metrics. Did authors consider using this loss to improve?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Weakness \n\n**W1: Method part isn't clearly stated**\nWe will re-draw Figure 2 (Overview of our proposed AV-GS) to make it more clear and straightforward. And update all figures to PDF.\n\n**W2: PCA feature of learnable acoustic points**\n(The PCA plot and the correlation matrix discussed below have been provided in the PDF attached above, in the common \"Author Rebuttal\")\nFor RWAVS scene 6, we analyze alpha by plotting its first two principal components and applying k-means (k=3) clustering, to project colors onto G_a. It can be observed that most of the objects are grouped by the same color. Next, we manually match scene objects (e.g., fireplace, walls) with material labels (e.g., 'brickwork', 'solid wall' respectively) from an absorption database [6]. Using these absorption coefficients, we assign absorption vectors (Dim: [1x6]; coefficients for 6 frequencies from [6]) to every point that corresponds to the object (matched with the label). A correlation matrix of these absorption coefficients, across the points (grouped by clusters), shows high self-correlation within clusters and lower cross-correlation across clusters, i.e., the points within the same clusters have similar absorption values, and less correlated absorption across the clusters. While these clusters do not explicitly highlight fine-grained material properties such as Young's modulus, they provide implicit material cues for binauralization.\n\nAdditionally, we demonstrate the effect if alpha is frozen during our training (initialized using SH from vanilla 3D-GS). As shown in the table below, constraining alpha to SH results in a significant drop in binarualization. \n\n| Alpha- trainable | MAG (↓) | ENV (↓) |\n|------------------|---------|---------|\n| No               | 1.584   | 0.147   |\n| Yes              | 1.417   | 0.140   |\n\nCombined with the PCA/correlation matrix above, and this experiment, we hypothesize that the learned alpha is able to pick up implicit material properties on top of SH to aid binauralization. We agree that alpha captures an implicit, rather than explicit properties like Young's modulus, material representation derived from raw RGB images and will clarify this in the revised version.\n\n**W3: confirm if same train-test split and evaluation code as baseline**\nThank you for pointing this out. We adopt the same training and test split and the evaluation code provided by AV-NeRF (available on Github) and hence making the comparison fair. \nPlease note AV-NeRF provides train and test json files, and we adopt the same dataloader from their code, which makes the comparison fair.\n\n**W4: Position-guidance. Isn't distance an important cue?**\nGreat question! We found using only the view information for position guidance suffices for the model performance. We found that this normalization in Eq. (3) provides numerical stability during training.\nWith regards to the distance being an important que, please note that the position embedding of the listener location is already a part of the binauralizer, please check the Fig. 7 in A.2\nWe skip the arrow (showing X_L/X_S into the binauralizer) in the Fig 2, to maintain brevity and not deviate focus of the novel contributions. Instead we tried to highlight this in our Binauralizer Fig. 7 in A.2.  We will update this in the revised version.\n\n**W5: Does energy decay loss help?**\nThank you for pointing this out. We adopted the decay loss in addition to our proposed losses, however we did not find any improvement. (please note, for the RWAVS dataset, we compute the l1 loss between the decay curves for the GT and the predicted spectrograms of the binaural audio). \n\nFollowing RAF[7] we also ablated the weight of the decay loss in the overall loss computation. However, in contrast to RAF's findings where decay loss improves temporal domain metrics, but worsens temporal-frequency domain metrics (STFT error), we find that it worsens both temporal (Envelope distance (ENV)) and temporal-frequency domain metrics (spectrogram magnitude (MAG).\n\n| Lambda (decay weight) | MAG (↓) | ENV (↓) |\n|-----------------------|---------|---------|\n| 0.0                   | 1.417   | 0.140   |\n| 1.0                   | 1.481   | 0.142   |\n| 2.0                   | 1.510   | 0.142   |\n| 3.0                   | 1.486   | 0.141   |\n\n\nReferences:\n[1] Gao, Ruohan, and Kristen Grauman. \"2.5 d visual sound.\" CVPR'19 \n\n[2] Luo, Andrew, et al. \"Learning neural acoustic fields\" NeurIPS'22\n\n[3] Su, Kun, Mingfei Chen, and Eli Shlizerman. \"INRAS\" NeurIPS'22\n\n[4] Liang, Susan, et al. \"AV-NeRF.\" NeurIPS'23\n\n[6] C. Kling. Absorption coefficient database, Jul 2018.\n\n[7] Chen, Ziyang, et al. \"Real acoustic fields\" CVPR'24\n\n[8] Tang, Zhenyu, et al. \"GWA\" SIGGRAPH'22", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
This attached PDF contains the PCA plot and the correlation matrix which is referred in rebuttal response for RHwj - Q2 and DaxK - W2 below.

Meta-review: This is an important and timely contribution. Most technical solutions for novel view acoustic synthesis failed to reliably address both simulated and real datasets while also generating audio that is perceptually reasonable. This paper address all 3 aspects and improves state of the art. The authors back-up the claims with good set of evaluations/ablations. The approach opens up avenues for new material estimation and material swapping methods in audio-visual synthesis and other applications in augmented reality etc.

The key technical novelty comes from showing how to build splatters for audio data, and the overall complexity of the scenes is still limited. Hence, the paper is ready for publication but not at Oral threshold for the conference.  

Average rating - 6
Decision: Accept (poster)