AV-GS: Learning Material and Geometry Aware Priors for Novel View Acoustic Synthesis

Authors: swapnil  bhosale
haosen  yang
diptesh  kanojia
jiankang  deng
xiatian  zhu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 17:23:11

Abstract: Novel view acoustic synthesis (NVAS) aims to render binaural audio at any target viewpoint, given a mono audio emitted by a sound source at a 3D scene. Existing methods have proposed NeRF-based implicit models to exploit visual cues as a condition for synthesizing binaural audio. However, in addition to low efficiency originating from heavy NeRF rendering, these methods all have a limited ability of characterizing the entire scene environment such as room geometry, material properties, and the spatial relation between the listener and sound source. To address these issues, we propose a novel Audio-Visual Gaussian Splatting (AV-GS) model. To obtain a material-aware and geometry-aware condition for audio synthesis, we learn an explicit point-based scene representation with audio-guidance parameters on locally initialized Gaussian points, taking into account the space relation from the listener and sound source. To make the visual scene model audio adaptive, we propose a point densification and pruning strategy to optimally distribute the Gaussian points, with the per-point contribution in sound propagation (e.g., more points needed for texture-less wall surfaces as they affect sound path diversion). Extensive experiments validate the superiority of our AV-GS over existing alternatives on the real-world RWAS and simulation-based SoundSpaces datasets. Project page: \url{https://surrey-uplab.github.io/research/avgs/}

Main paper: 
 introduction: 
 Novel view synthesis [25, 1, 14] allows to generate images for any target viewpoints, which has been extensively studied and advanced. For real-world applications in augmented reality (AR) and virtual reality (VR), solely visual rendering of 3D scenes without spatial audio (i.e., deaf) fails to fully immerse users in the virtual environment. This thus inspires a recent surge of investigating novel view acoustic synthesis (NVAS) [4, 17, 5, 32] . By synthesizing binaural audio (two channels corresponding to the left and right ear) taking into account the factors like directionality, distance and relative elevation, this can create a spatial audio experience akin to real-life perception [23] . However, realistic binaural audio synthesis is challenging, since the wavelengths of sound waves are much longer, necessitating the modeling of wave diffraction and scattering. To illustrate, while blocking the sun with your thumb is easy, blocking thunder sounds with your thumb is difficult because sound waves wrap around obstacles. A sound wave propagating through a 3D space undergoes various sophisticated acoustic phenomena, like direct sound, early reflections, and late reverberations.Aiming to render binaural audio from the mono audio for target poses, Neural Acoustic Field (NAF) [21] learns room acoustics in a synthetic environment with a 2D grid of implicit representations as a condition. However, deriving a 2D representation grid for real-view scenes is extremely challenging in the presence of unconstrained objects, materials, and occlusion in 3D scenes. Alternatively, AV-NeRF Figure 1 : Sound propagation point patterns between a listener (blue sphere) and emitter (yellow sphere) captured by our AV-GS. Notice the points outside the propagation path (points behind the speaker, points behind rigid walls). Please note we slice the scene into half along the y-axis (omitting the points from the ceiling) in order to facilitate better visibility. [17] leverages implicit representations of the visual cue by learning a vision NeRF model [25] that generates the listener's view. Nonetheless, this can only offer limited conditions, as the listener's view provides information from only the line-of-sight, whilst audio spreads around corners more widely (a listener cannot see behind but can hear sources that are behind). Further, NeRF's training and inference efficiency is typically low [37, 14] .To overcome these limitations, in this work, we present a novel Audio-Visual Gaussian Splatting (AV-GS) model, characterized by efficiently learning an explicit, holistic 3D scene condition with rich material and geometry information, as well as more comprehensive contextual knowledge beyond the listener's field of view for enhanced synthesis conditioning. Due to the intrinsic discrepancy between vision and audio as discussed earlier, extending existing 3D Gaussian splatting-based (3DGS) [14] from visual scenes to 3D audio (i.e., spatial audio) is non-trivial. Optimizing towards scene visual reconstruction, points tend to over-populate along object edges and under-populate in textureless regions such as walls, doors etc. But, such point distribution is rhetoric when learning sound propagation since, major changes in sound paths (absorption and diversion) primarily happen around those texture-less regions. The key challenge lies in jointly modeling 3D geometry and implicit material characteristics of the visual scene objects to instigate direction and distance awareness for realistic binaural audios. To that end, we decouple the physical geometric prior from the 3DGS representation to learn an acoustic field network by introducing audio-guidance parameters. These audio-guidance parameters combined with their relative distance and direction from the listener and sound source, are projected on-the-fly to derive holistic scene-aware and implicit material-aware conditions for synthesizing binaural audios (see Figure 1 ). We hypothesize that converging AV-GS on binaural audio reconstruction loss, requires location and density adjustment of the Gaussian points relative to the "audio-guidance" provided by the individual point. Towards this direction, we design a Gaussian point densification and pruning strategy based on the individual per-point contribution in providing this "guidance" for sound propagation, ultimately improving the overall binaural audio synthesis.To summarize, we make the following contributions: (1) The first novel view acoustic synthesis work with conditions on holistic scene geometry and implicit material information; (2) A novel AV-GS model that learns a holistic geometry-aware material-aware scene representation in a tailored 3DGS pipeline; (3) Extensive evaluations validating the advantages of our method over prior art alternatives on both synthetic and real-world datasets. material and topology data, necessitating an acoustic material database. In contrast, our approach learns explicit representation from 3D scenes and disentangles learning of physical and (implicit) material properties, eliminating reliance on scene geometry and material characteristics as inputs.Visual cue conditioning Recent works leverage visual cues to generate spatial audio from mono audio, capturing complementary scene characteristics [9, 38, 27, 41, 18, 43, 19, 28] . Li et al. [15] proposed a multi-task approach optimizing binaural audio generation and flipped audio classification, sharing visual cues from corresponding video frames. Chen et al. [4] synthesized sound by analyzing input audio-visual cues, incorporating active speaker features, target pose, and encoded visual features in the acoustic synthesis pipeline. Alternatively, Liang et al. [17] encoded the listener's position and orientation, conditioned by the listener's view. It additionally provides local visual depth information dependent on the listener's position. However, focusing solely on the listener's view overlooks the broader 3D scene geometry's contribution, which is crucial for sound propagation. We thus propose to learn a holistic 3D scene representation, enhancing binauralization guidance with additional audio parameters.3D Scene Representation Learning Point-based rendering techniques, initiated by [10] , utilize point-based representation where each point affects a single pixel. Zwicker et al. [45] advanced this with ellipsoid-based rendering (splatting), allowing mutual overlap to fill image holes. At the absence of given geometry, Mildenhall et al. [25] explored neural implicit representation, NeRF, predicting view-dependent radiance via implicit density fields. NeRF requires combining colors of densely sampled points along the camera rays for high-quality rendering. 3D Gaussian splatting (3D-GS) [14] , a novel-view synthesis method, employs explicit point-based representation, contrasting with NeRF's volumetric rendering. Since its real-time high-quality rendering capabilities that 3DGS has been applied to various domains, including simultaneous localization [13, 24] , content generation [36] , and 4D dynamic scenes [16, 39, 42] , among others. However, no works take the 3DGS advantages to improve the NVAS task. Hence, in this work, we for the first time exploit the 3D-GS for NVAS, to the best of our knowledge, including capturing the scene geometry and material-related information for audio signal processing purposes. Further, we adapt the point management mechanism to account for the characteristics of sound propagation. In a nutshell, our approach aims to improve binaural audio reconstruction loss in AV-GS by adjusting Gaussian point location and density based on their contribution to sound propagation guidance. 
method: 
 Problem definition Deployed at a location X S in a 3D scene, a sound source S emits a mono audio a mono . A listener entity L moves around in this 3D scene, capturing multiple observations using a camera mounted with a binaural microphone. Each observation O p = (V C , V A ), constitutes a pair of a camera view V C and an auditory perspective V A , w.r.t a specific listener pose p = (X L , d), where X L is the 3D position of the listener and d is the viewing direction corresponding to listener's head. A camera view V C defines that at the pose p, the listener would observe a RGB image, I (as their view). Similarly, an auditory perspective V A defines that at the pose p, the listener hears a binaural audio, a bi = (a l , a r ) where l and r represent the left and right ear respectively. Given N observations from the listener, O = {O 1 , O 2 , ..., O N }, the task is to predict the binaural audio a bi * for a novel observation O p * having an arbitrary unseen pose p * .Our model is comprised of a 3D Gaussian Splatting model G, an acoustic field network F, and an audio binauralizer B. In order to model sound propagation in space and time, having a holistic scene prior as contextual guidance is essential. Crucially, this guidance must incorporate both geometric and material-related characteristics. For facilitating the learning of visual and auditory modalities with inherently distinct characteristics, we decouple the physical geometry and the acoustic field by introducing in-between an acoustic field network F that learns geometry-aware and implicit material-aware scene contexts.In the following, we first, provide a brief regarding learning the scene geometry. Later, we explain how to construct an audio-focused representation G a . We further describe the process of decoding the parameters of G a on the fly using view-dependent information, followed by point densification and pruning tailored for sound propagation. We first train G to capture the scene geometry information. Next, we construct an audio-focused point representation G a , with the location X and audio-guidance parameter α initialized by the pre-trained G. Then the acoustic field network F is used to process the α parameters for all the Gaussian points in the vicinity of the listener and the sound source (in the 3D space). The output from F is finally used to condition the audio binauralizer B, which transforms the mono audio to binaural audio w.r.t the listener and sound source location.3D Gaussian Splatting (3D-GS) [14] is a state-of-the-art novel-view synthesis method that learns an explicit point-based representation of the 3D scene. It is utilized here to capture the scene geometry. Specifically, each point in the explicit representation G is represented as a 3D Gaussian ellipsoid to which physical attributes like position X , quaternion R, scale S, opacity O and Spherical Harmonic coefficients (SH) representing view-dependent color are attached. For an arbitrary camera view V C * with a pose p * , we project/splat 3D Gaussians onto the 2D image plane to obtain the listener's view as an RGB image I * . The projection of 3D Gaussian ellipsoids can be formulated as:Σ ′ = JW ΣW T J T(1) where Σ ′ and Σ = RSS T R T are the covariance matrices for 3D Gaussian ellipsoids and projected Gaussian ellipsoids on 2D image from a viewpoint with viewing transformation matrix W . J is the Jacobian matrix for the projective transformation. Please refer to [14] for more details on splatting.As a part of the decoupling approach, we further introduce an audio-focused point-based representation G a . Concretely, every point in G a is parameterized using the location X, alongside a learnable audio-guidance parameter α to encapsulate implicit material-specific characteristics of the scene. X is initialized using the location of Gaussian points in G, in order to impose the scene geometry knowledge. To initialize α we concatenate the view-dependent color priors from G, particularly spherical harmonics SH and quaternion feature R. An intuition is to choose the parameters that provide information regarding the color and density characteristics, which are often correlated with material properties [8] (see ablation on the choice of different parameters from G for forming α in Section 4.4).In order to drive the mono to binaural audio transformation, a pose-specific holistic scene context has to be derived from G a . To achieve this, we derive a position-guidance feature G w.r.t both, the listener L and the sound source S. Position-guidance G concatenated with the audio-guidance α is used as an input to the acoustic field network F to obtain a joint context for each point w.r.t to the listener and the sound source separately.EQUATIONEQUATIONConcatenating (represented using ⊕) the listener-context and the speaker-context yields the overall pose-specific holistic scene context. We obtain the condition for binauralization by averaging the context across all points in G a , post dropping points outside the vicinity of the listener and sound source. The vicinity for the listener and sound source is defined by the nearest k L and k S points based on the Euclidean distance between X and X L and that between X and X S , respectively. We provide an ablation for selecting the size of vicinity points in Section 4.4.An audio binauralization module B transforms the mono audio a mono emergent from the sound source S into binaural audio a bi = {a l , a r }, representing the left and right audio channels. The position and orientation of the listener (relative to the sound source) along with the learned holistic scene context C is used to condition this transformation.EQUATIONwhere X L denotes the listener's position and orientation. We adopt a similar architecture for the binauralization as [17] , wherein X L and C are fused to generate acoustics masks m m and m d representing the mixture and difference of the sound fields respectively. Specifically, an MLP combines X L and G a with a frequency query f ∈ [0, F ] to produce a feature vector which is further projected using a linear projection to obtain a mixture mask m m for frequency f. This feature vector is concatenated with the transformed direction θ and passed to a second MLP to generate a difference mask m d . All frequencies within [0, F ] are queried to generate complete masks for both the mixture and difference. The magnitude spectrogram s mono , computed using short-time Fourier transform (STFT) over a mono , is multiplied with m m and m d to obtain the mixture magnitude s m and the difference magnitude s d , respectively. Finally, an inverse STFT is operated on s m and s d to obtain the binaural audios for the left and right channels, a l and a r . The architecture of the binauralizer is discussed in our appendix A.2.Due to our decoupling design, we adopt a dual-stage optimization, starting with an initial warmup stage that learns the physical properties of the Gaussian points using gradients obtained while reconstructing camera views (RGB images). The objective of this initial stage is to infer an explicit point-based representation G to capture the scene geometry. Optimization of G is adapted from the original 3D-GS [14] using the loss function,EQUATIONIn the second stage, we initialize the audio-guidance parameters of G a using physical parameters of the warmed-up 3D Gaussians. Subsequently, we learn the implicit material properties for the Gaussian points using gradients obtained in the binauralization task for every training auditory perspective. Optimization of G a is guided by a combination of the binaural audio reconstruction loss, L m and a volume regularization loss, L v .EQUATIONEQUATIONwhere, L m is the summation of the L 2 loss calculated individually between the predicted magnitudes for s m , s l , s r (i.e., mixture, left and right audio channel, respectively) and their respective ground-truth magnitudes. L v = Na i=1 P rod(α i ), where P rod(.) is the product of the values of the audio-guidance parameter α of each Gaussian point in the auditory perspective, encouraging the audio-guidance parameters to be small, and non-overlapping.The physical properties and material-related properties of the new Gaussians are decoded from the learned physical parameters and audio-guidance parameters respectively, in a view-dependent manner on-the-fly.Audio-aware point management The location of points in G a are initialized from G, however since G is optimized using an image reconstruction loss, the initial placement of the points in G a is not necessarily contributive to an optimal audio guidance. Especially, it has been observed in recent works that 3D-GS is inadequate in densifying texture-less and less observed areas [20, 6, 2] .To address this problem, we propose an error-based point growing policy that populates new points in G a , wherever deemed significant. Specifically, the densification step is interleaved across multiple binauralization forward passes. During each densification step, we compute averaged gradients (across all steps up to the previous consecutive densification step) for each point in G a that lies in the vicinity, denoted as Θ g . Points with Θ g > τ g are deemed as significant, where τ g is a pre-defined threshold. Using the original 3D position as a probability distribution function (PDF), we sample new points. The audio-guidance parameters for these new points are obtained by random initialization. Additionally, we regularly employ a random elimination of points to avoid rapid expansion of new points (e.g., every 3000 iterations). 
datasets: 
 We evaluate both a real-world dataset-RWAVS and a synthetic dataset-SoundSpaces.Real-World Audio-Visual Scene (RWAVS) [17] The RWAVS dataset offers realistic multi-modal training samples constituting camera poses, high-quality binaural audios, and images. The dataset consists of 11 indoor and 2 outdoor scenes. Randomly sampling the sound source and listener positions, the authors collected data ranging from 10 to 25 minutes for every scene. For every scene, a data sample includes a set of camera poses (sound source and listener), extracted RGB key frame, followed by one-second binaural audio (as received at the listener position) and one-second mono source audio (as emitted by the sound source). We follow an 80:20 train-validation split for every scene in the dataset. [3] The Soundspaces dataset consists of 6 indoor scenes with varying degrees of complexity (2 scenes having a single room with rectangular walls; 2 scenes having a single room with a non-rectangular walls; 2 with multi-room layout). Since the dataset includes room impulse responses (RIR) recorded at receiver/listener positions, we replace the acoustic mask generation block with an RIR prediction block. The listener consists of a stereo listener, with four discrete head orientations among 0, 90, 180, and 270. We follow the same 80:20 train-validation split, as [17] for every scene in the dataset.Our complete implementation is based in PyTorch using a single Nvidia A550 GPU. The learning rate for all physical parameters in G is adopted from the original 3D-GS implementation [14] . For the acoustic field network F, we use MLP network with 64 nodes. Hyper-parameters for the binauralizer B, as well as the metrics are adopted from AV-NeRF [17] . For the audio-aware point management, we set τ g to 0.0004. Additionally, after every 3k iterations we randomly eliminate points using an outlier removal process (from Open3D [44] ) that removes points that have less than 8 neighbors in a sphere less than radius of 0.1.As a part of the proposed decoupling, we train the 3D-GS model G for 3k iterations, wherein each iteration involves randomly sampling a camera view V C and optimizing the parameters for G. Post this we initialize α with the physical parameters from G, and train for 40k iterations, wherein now each iteration involves randomly sampling an auditory perspective V A and optimizing the parameters for G a and B.Binaural audio synthesis -RWAVS dataset -Table 1 In order to have a fair comparison, we adopt the same metrics -magnitude distance [MAG] [41] (computed in time-frequency domain) and envelope distance [ENV] [26] (computed in time domain), as [17] . Particularly, (1) Mono-Mono duplicates the source audio to create fake binaural audio without modifications; (2) Mono-Energy scales the input audio's energy to match the target, generating stereo audio by duplicating the scaled audio; (3) Stereo-Energy separately scales the input audio's energy for each channel to match the target, then combines them for stereo audio; In addition to the codec baselines above, we also compare with NAF [21] , INRAS [34] , and AV-NeRF [17] . AV-GS significantly surpasses all prior arts across all scenes, by a significant margin. Particularly compared against AV-NeRF, which utilizes audio-visual cues, AV-GS achieves a 5.7% and 3.4% relative improvement in terms of MAG and ENV respectively.We highlight the qualitative improvement of AV-GS over AV-NeRF in Fig. 3 (and in Section A.1). AV-NeRF extracts visual cues from the listener's view using a 256x256 RGB and depth image (rendered by a pre-trained NeRF) and processed by a frozen ResNet18 [11] model that is trained on ImageNet [7] . When the listener's view includes multiple objects or complex geometries (Fig. 3(a )), or meaningless information (Fig. 3(b )), it leads to sub-optimal visual cue extraction and poor binauralization. In contrast, our AV-GS uses learned representations (combining α of points in G a w.r.t to the listener and speaker) to extract context at a more holistic level, remaining unaffected by these scenarios.RIR generation -Soundspaces dataset -Table 2 (a) Following [17] and [21] , we compare AV-GS against classical high-performance audio coding methods: Advanced Audio Coding (AAC) [12] and Xiph Opus [40] , applying both linear and nearest neighbor interpolation techniques to the coded acoustic fields. Additionally, we also compare with existing neural methods: NAF [21] , INRAS [34] , and AV-NeRF [17] . It can be observed that AV-GS outperforms all the prior arts by a significant margin. Improved efficiency Decoding α on-the-fly helps AV-GS retain the efficiency advantage of having an explicit point based representation over learning an implicit representation of the visual 3D scene. In contrast, AV-NeRF trains an additional NeRF model, whose outputs are used to render the listener's view which is the condition for their binauralizer module. In Table 3 , we compare the inference time for a single scene (House scene 1) from the RWAVS dataset. Although the original implementation of AV-NeRF proposed a single view in the direction of listener's viewing direction (used as the listener's context), we modify their implementation in which the V-NeRF renders 2 and 4 perspective views with a field of view of 90 • for each receiver's position. It can be observed, although using the additional views, helps provide additional local context around the listener (hence improving the binauralization performance), but only at the cost of increased inference time. AV-GS on the other hand requires least rendering time, while providing the best binauralization performance. It is important to note that, although we train the 3D-GS for only an initial warm-up stage of 3k iterations, we can still jointly train both G and G a on the corresponding camera view V C and the auditory perspective V A respectively. By doing so, AV-GS would be able to synthesize both binaural audio as well the visual view at the unseen target (novel) viewpoint, without any increase in the training time than that of as shown in Table 3 (last row). In this work, we instead focus on only the binaural audio synthesis and hence train only G a in the second stage.Although we train the 3D-GS for just an initial 3k iterations, we can still train both G and G a together for the corresponding camera view V C and auditory perspective V A , such that the trained AV-GS is able to synthesize both the binaural audio and the visual view from a novel viewpoint , thereby making it a fair comparison with AV-NeRF [17] . However, in this work, we focus solely on binaural audio synthesis and only train G a in the second stage.We use the binaural audio synthesis task for performing our ablation study and report average performance across all scenes in the RWAVS dataset.Initialization of α. 3D-GS learns a point-based representation (G) of the 3D scene such that each point in the explicit representation is represented as a 3D Gaussian ellipsoid to which physical attributes like position X , quaternion R, scale S, opacity O and Spherical Harmonic coefficients (SH) representing view-dependent color are attached. As discussed above, we initialize the audioguidance parameter α using the learned physical parameters from G. Table 2 (b) shows an in-depth study of the effect on the binauralization performance when choosing different parameters from G for the initialization. It is evident that SH and R, when considered individually as well as when combined, are crucial parameters that help provide an overall better binauralization performance.The learned Gaussian points G representation normally consists of millions of points, and using all the points for computing the condition for the binauralization task will result in a huge computational overload. Moreover, intuitionally, points near the listener and speaker location in the 3D space (i.e.vicinity) are more contributive to determining the local geometry and material-related characteristics, and greatly influence the sound field. Empirically in Table 4a , we find that using 15 percentile of the points in the vicinity (listener and speaker considered separately) yields the best performance. Using an extremely high percentile for capturing the vicinity generally helps, but only to a certain extent, since it increases the risk of adding unnecessary information. Moreover, it is important to note that in scenes with smaller room sizes or when the listener and speaker are Effect of audio-aware point management. AV-GS infers local geometry from an explicit pointbased prior and learns additional audio-specific parameters for every point. The local geometry can be inferred either directly from the sparse points produced during camera calibration using Structure-from-motion (SfM) [33] or warming up a 3D-GS model on the SfM points. From Table 4b it is evident that audio-aware point management helps improve the binauralization performance irrespective of the adopted point initialization approach. Texture-less regions such as walls, doors, etc are the regions with maximum sound absorption and diversion and hence having optimal point density promotes better modeling of sound propagation. 
conclusion: 
 In this work, we present AV-GS, a holistic scene representation learning approach for conditioning novel view acoustic synthesis. The core of AV-GS lies in the proposed decoupled modeling of 3D scene geometry and material-related characteristics of scene objects for sound propagation, enabled by the introduction of additional audio-guidance parameters per point within an acoustic field network. We show that our approach leverages audio-aware Gaussian point positioning to improve the binauralization performance, on real world as well as synthetic 3D scenes, significantly in comparison to prior art alternatives.Limitations Although AV-GS achieves a new state-of-the-art on the NVAS task by learning a holistic scene representation with geometry-aware and implicit material-aware characteristics, it faces some challenges. Like AV-Nerf, AV-GS currently learns a separate representation for each scene, posing a significant challenge for generalizability and transferability across multiple scenes, which is crucial for improving efficiency and reducing computational demands. Also, as observed in Table 1 , the performance drops when the scene size increases (e.g., from Office to House), due to the difficulty in learning representations for larger, complex geometries with multiple rooms. An interesting research direction would be to model individual rooms using separate 3D-GS representations and then learn a transfer function across rooms.A Appendix / supplemental material Broader impact Potential applications of AV-GS lie in virtual reality, augmented reality, and immersive audio experiences, where accurate and efficient binaural audio rendering can significantly enhance user experience. However, enhanced acoustic synthesis technology could be used unethically in surveillance or privacy-invasive applications, where binaural audio could be synthesized to eavesdrop on conversations or create misleading audio experiences.A.1 Direction and distance awareness showcased by AV-GS (Please note that in the second row we slice the points in G a for along the vertical axis, essentially removing the points from the ceiling, in order to provide better visibility.) With the increase in the distance of the listener (blue sphere) from the sound source (yellow sphere), AV-GS is able to synthesize binaural audios with decreasing amplitude across both the left and right audio channels. We adopt the architecture of binauralizer from [17] with some modifications to input the learned scene context from our acoustic field network F (see Fig. 7 ).Embedding blocks take the normalized location position X L (only 2D x and y coordinates).The main components include two Multilayer Perceptrons (MLPs), each comprising four linear layers with an additional residual connection. The width of each linear layer, is set to 128 for the RWAVS dataset and 256 for the SoundSpaces dataset. All linear layers are followed by ReLU activation layers, except for the last layer, where the ReLU activation is replaced with the Sigmoid function. The first MLP takes the listener's position (x, y) and the frequency f ∈ [0, F ] as input, where F represents the number of frequency bins. It predicts a mixture mask m m for the given frequency f and generates a feature vector with c channels. Prior to feeding them into the MLP, we apply positional encoding to the listener's position (x, y) and the frequency f . We set the maximum frequency used for positional encoding as 10. A coordinate transformation [17] to project the listener's direction θ into a high frequency space. We concatenate the transformed listener's direction and the feature vector, and feed it into the second MLP. The second MLP is appended with a Sigmoid layer and a scaling layer, ensuring that the difference mask m d estimated by the second MLP falls within the range of -1, 1.For each frequency query f , estimates two masks: m m and m d , both of which are scalars. We iterate over all frequencies f ∈ [0, F ] to obtain the complete masks m m and m d .For validating AV-GS on the room impulse response generation task using the Soundspaces dataset, following [17] , we drop the mixture mask, and predict the impulse response directly for a corresponding time input, using the second MLP. We show these modifications in Fig. 8 .  
 
 Appendix: 38th Conference on Neural Information Processing Systems (NeurIPS 2024). 


Decision: This is an important and timely contribution. Most technical solutions for novel view acoustic synthesis failed to reliably address both simulated and real datasets while also generating audio that is perceptually reasonable. This paper address all 3 aspects and improves state of the art. The authors back-up the claims with good set of evaluations/ablations. The approach opens up avenues for new material estimation and material swapping methods in audio-visual synthesis and other applications in augmented reality etc.

The key technical novelty comes from showing how to build splatters for audio data, and the overall complexity of the scenes is still limited. Hence, the paper is ready for publication but not at Oral threshold for the conference.  

Average rating - 6

Reviews: review: paper_summary: paper_summary: The paper proposes a 3D Gaussian splatting (3DGS) based method for improving the acoustic context modeling for novel view acoustic synthesis (NVAS). The proposed model outperforms multiple existing methods on both simulated and realworld data across different metrics. The method also trains and infers faster than the state-of-the-art AV-Nerf method. The paper also provides model ablations and analyses that help in better understanding the role of different model components.
strength_weakness: strength_weakness: 1. Acoustic conditioning with G_a: the paper "averages the context across all points in G_a" (L156-7). Doesn't this lead to loss of finegrained point-specific information? A learned aggregation strategy (for example, by using the attention mechanism) would probably make more sense here?

2. L192-4, "L_v ... non-overlapping": 
    i) the insight behind doing this is not clear from the text
    ii) there are no references to support these steps
    iii) there is no ablation for this loss, as well

3. L208, "The audio guidance ... random initialization": won't random initialization during point expansion lead to local 'discontuity'? Did the authors try warm initialization to preserve local 'continuity' by doing nearest neighbor search or bilinear interporation on the existing alphas?

4. Does the model generalize to more challenging scene datasets like Matterport3D [1]? 

5. Audio examples: the paper does not seem to provide audio examples and qualitiatively compare their quality with that of the baselines

6. Minor:
   i) L274-5, "4 perspective views ... receiver's position": do these 4 views cover the full 360 degree, otherwise it's not comparable with the field of view of the AVGS model?
   ii) Para in L285-90 looks like repeated text


References: 
[1] Matterport3D: Learning from RGB-D Data in Indoor Environments. Chang et al. 3DV 2017.
questions: questions: Could the rebuttal comment (more) on the following?
1. how the chosen strategy for aggregating the acoustic context in G_a affects model performance. See weakness 1 for details.

2.  the rationale behind using L_m, and how the model performs without the loss. See weakness 2 for details. 

3. initialization of alphas during point densification in G_a. See weakness 3 for details.

4. generalization of the model to more challenging datasets like Matterport3D. See weakness 4 for details.  

5. my minor comments. See weakness 6 for details.

Also, could the authors anonymously provide a few audio examples from the model and baselines?
limitations: limitations: The paper mentions limitations in Discussion but I could not find any discussion on societal impact.
paper_summary: paper_summary: This work proposes an efficient and performant method for novel view acoustic synthesis using Gaussian Splatting representations of 3D scenes. The proposed model integrates geometry and material conditioning, and offers a way to bridge the gap between visually-useful 3DGS representations and acoustically useful ones, using “audio-guidance” which considers the local environment in Experiments show this outperforms existing NeRF-based approaches while being more efficient.
strength_weakness: strength_weakness: The value of $\tau_g$ seems somewhat arbitrarily defined, and the range of tested vicinity sizes is somewhat narrow; given the efficiency of the method. It’s hard to extrapolate what the performance would be with more context (i.e. would it keep going down?). Most of my other suggestions are relatively minor, about the writing and presentation:
- Abstract: “at a 3D scene” -> for clarity, should this be “within” a 3D scene?
- Introduction: “visual rendering of 3D scenes without spatial audio (i.e., deaf)” -> this seems like an inaccurate characterization. Deafness is not about rendering, but about sensation. Spatiality is also not clearly related to this.
- Introduction: “realistic binaural audio synthesis is challenging, since the wavelengths of sound waves are much longer” -> this is not the only reason that this is challenging. Even if you use a simplified (e.g. ray-tracing) model, which would be inaccurate at lower frequencies due to wave-like behavior etc., you would still need a good model of the environment for physics-based rendering, or a large number of spatial measurements for a data-driven approach.
- Introduction: “…modeling 3D geometry and material characteristics of the visual scene objects to instigate direction and distance awareness…” -> while geometry has a direct impact on “direction and distance”, material impacts spatial perception indirectly (through absorption, reflection, and diffusion). Could this be rephrased more clearly?
- Related work: “Anton and Dinesh” should maybe be “Ratnarajah and Manocha”, i.e. last names of authors?
- Related work: Under “Geometry and material conditioning”, the final statement differentiates the current work from prior work by pointing out the non-reliance on scene and geometry inputs, but doesn’t clarify why this is useful. E.g. it could be pointed out that, in the real world, such inputs are often not available, so this work allows generalizing to such settings.
- Figure 2: “Differential Rasterizer” -> “Differentiable Rasterizer”?
- Figure 4: Could there be a subfigure showing 15%, since that seems to give the best overall performance? It would be helpful context, since otherwise the reader needs to interpolate between (c) and (d).
- Section 4.4: “intuitionally” -> “intuitively”?
questions: questions: - Is $S$ modeled as an omni-directional point source? I assume so because point source is the typical assumption, and an emission direction isn’t given (vs. listener direction d), but it’s worth clarifying.
- I think the notation here could be clearer. For example, $V_C$ and $V_A$ don’t seem used, and it’s not clear to me what their role is? E.g. why is the observation $O_p$ = ($V_C$, $V_A$) instead of ($I$, $a_{bi}$)? If the pose $p$ captures the position and orientation, what do $V_C$ and $V_A$ add on top of this representation? Why is $I$ upper-case and $a$ lower-case here, when they both represent observations? Why is $a_{bi} = \{a_l, a_r\}$ notated as a set, but other sets are notated as tuples? Additionally, shouldn’t $k_l$ be $k_L$, similar for $S$?
- Conceptually, there is a tension between considering the global scene context, as the paper argues for, and dropping points outside the local vicinity of the source and listener (since sound reflections can easily include points from other regions, in principle). Could the authors comment on this? It’s interesting that the optimal vicinity size is not the largest one tested. Is there some intuition for why this might be the case, given that it offers more information? Currently, the reasoning seems to be that this contains unnecessary information, but the useful information seems to be a subset of this?
- The audio rendering task is cast as a binauralization task. I understand that this is practically necessary, i.e. the input is mono and output is stereo binaural. However, wouldn’t many of the problems in this paper still exist in the monaural listener context as well? E.g. compared with a binauralization task purely based on HRTFs or simplified room geometries (e.g. Richard et al., ICLR 2021), where the context $C$ is not as rich. Could this be clarified?
- How was the $\tau_g$ value decided?
limitations: limitations: I think the authors have fairly represented the limitations. The broader impact statement is a little bit less clear to me, i.e. I’m not sure I understand the surveillance applications. Could this be clarified? I would be interested for the authors to consider the potential impact of the limitations listed. For example, the limitation of rendering larger scenes; what practical negative impacts could this have in deployment scenarios?
paper_summary: paper_summary: This paper proposes a new audio-visual Gaussian Splatting model for novel view acoustic synthesis. The proposed method explicitly models scene geometry and learns a point-based scene representation with an audio guidance parameter on locally initialized Gaussian points that takes the space relation from the listener and sound source into consideration. This work is the first that applies gaussian splatting to the novel view acoustic synthesis problem, and experiments on two datasets demonstrate the effectiness of the proposed method compared to prior work.
strength_weakness: strength_weakness: - There is a strong claim that the proposed model is learning material-aware priors, jointly modeling 3D geometry and material characteristics, holistic scene geometry and material information, etc. Then, I would actually expect materials are explicitly modeled, such as acoustic properties of surfaces in the form of some material parameters (e.g., absorption coefficients, Young’s modulus, etc.). However, all that is modeled is just a parameter alpha that claims to encapsulate material-specific characteristics of the scene. This is somewhat misleading as there is nothing constrains alpha to be physics-based or to be correlated with some material parameters. It's likely that it’s just learning some scene representation that may be more helpful for acoustic rendering.

- Also the so-called acoustic field network is also nothing about audio by itself. By acoustic field network, I was thinking the method is explicitly modeling sound propagation by taking into account the material properties of surfaces and the geometry of the space. The audio binauralizer is basically following prior work AV-NeRF [17], and what changes is what the audio binauralizer conditions on. In AV-NeRF, it’s scene features from NeRF and in this work it’s the context scene features from gaussian splatting.

- There were no qualitative examples given in supplementary materials, so it's really hard to appreciate the improvement that this method leads to.

- There are also many typos or grammar mistakes here and there. For example:
1. L45, 3D Gaussian splatting-based *methods*?
2. Related work, mixed use of past tense and present tense. 
3. L125, a brief ??
questions: questions: - See questions in the weakness section above.

- Also, for the SoundSpaces synthetic dataset, why it only contains 6 indoor scenes. There are more scenes in the dataset, and it would be good to explain the reason why only these six are not used.

- Is there any way to interpret the learned audio-guidance parameter to show that it’s material related?
limitations: limitations: Yes some limitations are discussed in the end, though some might be missing as discussed above.
paper_summary: paper_summary: This paper proposes a novel view acoustic synthesis approach based on 3D Gaussian Splatting scene representation. The framework consists of a 3D GS model, acoustics field network, and audio binauralizer. It first trains a 3D GS model to capture scene geometry. Then they use attributes from each Gaussian to initialize a learnable acoustic point representation, and then average the feature obtained from nearby Gaussian points for acoustic modeling. 
Through the experiments on the RWAVS and Soundspaces dataset, the authors demonstrate that the proposed method achieves state-of-the-art performance. They also perform the ablation study to ablate some components to demonstrate their technical advance.
strength_weakness: strength_weakness: - The reviewer thinks there are several issues with the writing:
  - First, **the method part isn't clearly stated**. It takes a long time for the reviewer to fully understand the method. The method is simple, using pretrained 3DGS to initialize a learnable acoustic point representation. While the method figure doesn't quite help with understanding. 
  - The reviewer strongly feels the paper is **overclaimed**. It claims it learns "holistic geometry-aware material-aware scene representation" while the experiments are only for acoustic modeling results. The author needs to use experiments to support this claim. One thought the reviewer has to visualize the PCA feature of learnable acoustic points to see if it has some cues. Otherwise, the reviewer is only convinced that this paper proposed a better acoustic modeling approach that leverages the GS features. 
  - The major experimental **results for the baselines are directly brought from AV-NeRF and INRAS papers** while the authors don't mention this in the paper. Also, the authors need to confirm if they have guaranteed the same training and test split and evaluation code.

- The reviewer has a question on the Position-guidance G condition. The authors use unit vectors which get rid of distance information. The reviewer is confused as to why remove the distance. Would it be an important cue?
- In the paper [5] Real acoustic field, the authors found out that energy decay loss shows a powerful improvement on those energy-based metrics. Did authors consider using this loss to improve?
questions: questions: I have mentioned my concerns and questions in the weaknesses part and I hope the authors could address them during the rebuttal.

A suggestion to the authors for paper writing: save the figures as PDF instead of screenshots or png.
limitations: limitations: Yes, the authors adequately addressed the limitations

Comments: title: Response to rebuttal 2
Response to rebuttal
comment: Thanks. I don't have any further questions.
1. Our current implementation uses lambda_a as 0.01
2. Indeed. We will add all audio samples -multiple scenes and different mono audio source, on our project page.

We thank the reviewer for their time and recommending our paper for acceptance. We will incorporate all tables and analyses in the final version.
Thanks for the responses. Could you answer/comment on the following?

1. Lambda_a: what was the value for it in the paper?

2. Audio examples: thanks for the audio sample, but I think the paper needs a lot more qualitative examples, given that it is an audio spatialization paper. A few suggestions from my end: 1) same mono audio played at different locations in the same scene (2-3 scenes of different sizes and layouts should be evaluated) + qualitative comparison with baselines, 2) repeat 1 for different mono audio samples. 

I would urge the authors to add all the results tables and analyses provided in the rebuttal, in the next draft.
We thank the reviewer for the discussion and for increasing the score towards acceptance. We will update the final version with these details and mention the learned implicit material properties.
Thanks for the rebuttal and additional results, which helps address many of my concerns. The new PCA plot is especially helpful, and I would encourage the authors to include it in the main paper to demonstrate the correlation of alpha with real material properties. The authors are suggested to revisit this strong claim in the paper "the proposed model is learning material-aware priors", and make appropriate adjustments and provide supporting resutls where needed.

Regarding the anonymized link to the audio samples only shared with AC, I kindly request the AC to confirm and double check the audio results are meaningful. If that is confirmed, I am happy to raise my rating and fine with the paper getting accepted considering the interesting idea and task proposed in the paper.  The qualitative audio samples are also important to be included as part of the paper to help readers appreciate the task and the gains of the proposed method.
We thank the reviewer for recommending acceptance. We appreciate this discussion and will incorporate it in our final version.
Thank you for the response. The PCA results show that the model learns material-related properties, and the explanation of distance information clarifies my concerns. I’ll raise my score to ‘weak accept’ and hope these changes are reflected in the revised version.
We are grateful for the reviewer's encouraging comments. We will incorporate this discussion in the next version.
Thank you, I think these are helpful clarifications and I appreciate your detailed response. I continue to enthusiastically recommend acceptance.