NeRAF: 3D Scene Infused Neural Radiance and Acoustic Fields
Authors: Amandine Brunetto
Sascha Hornauer
Fabien Moutarde
Venue: ICLR 2025 Poster (2025)
Keywords: Audio-visual learning
Neural Fields
Implicit Representations
Novel-view Acoustic Synthesis
Multimodal learning
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: NeRAF is a method that jointly learns radiance and acoustic fields, enabling realistic audio-visual generation.

Abstract: Sound plays a major role in human perception. Along with vision, it provides essential information for understanding our surroundings. Despite advances in neural implicit representations, learning acoustics that align with visual scenes remains a challenge. We propose NeRAF, a method that jointly learns acoustic and radiance fields. NeRAF synthesizes both novel views and spatialized room impulse responses (RIR) at new positions by conditioning the acoustic field on 3D scene geometric and appearance priors from the radiance field. The generated RIR can be applied to auralize any audio signal. Each modality can be rendered independently and at spatially distinct positions, offering greater versatility. We demonstrate that NeRAF generates high-quality audio on SoundSpaces and RAF datasets, achieving significant performance improvements over prior methods while being more data-efficient. Additionally, NeRAF enhances novel view synthesis of complex scenes trained with sparse data through cross-modal learning.
NeRAF is designed as a Nerfstudio module, providing convenient access to realistic audio-visual generation. Project page: https://amandinebtto.github.io/NeRAF

Published as a conference paper at ICLR 2025
NERAF: 3D SCENE INFUSED NEURAL RADIANCE AND ACOUSTIC FIELDS
Amandine Brunetto, Sascha Hornauer, Fabien Moutarde
Center for Robotics, Mines Paris - PSI. University
Paris, France
{amandine brunetto, hornauur, fabien feteraneeeee.aieee..ooorrrrrreee .sllev
ABSTRACT
Sound plays a major role in human perception. Along with vision, provides essential information for understanding our surroundings. Despite advances in neural implicit representations, learning acousties that align with visual scenes remains challenge. We propose NeRAF, method that jointly learns radiance and acoustic fields. NeRAF synthesizes both novel views and spatialized room impulse responses (RIR) at new positions by conditioning the acoustic field on 3D scene geometric and appearance priors from the radiance field. The generated RIR can be applied to auralize any audio signal. Each modality can be rendered independently and at spatially distinct positions, offering greater versatility. We demonstrate that NeRAF generates high-quality audio SoundSpaces and RAF datasets, achieving significant performance improvements over prior methods while being more data- efficient. Additionally, NeRAF enhances novel view synthesis of complex trained with sparse data through cross-modal learning. NeRAF is designed as a Nerfstudio module, providing convenient access realistic audio-visual generation. Project page: https:/ amandinebte gi thub. i /NeRAF
INTRODUCTION
Sound is fundamental to human perception. Imagine standing in the middle of a bustling city. It's not only the visual cues but the sounds of car horns, footsteps, and chatter that guide your perception and decisions. Beyond conveying environmental details, sound provides crucial insights into context and atmosphere, enriching understanding of the world in ways that sight alone cannot. In gaming and AR/VR simulations, audio is key to immersion, enhancing the realism of virtual environments and making experiences feel authentic.
Advances in novel view synthesis have enabled the generation of high-quality, photorealistic images from any camera position using set of captured images (Mildenhall et al., 2020: Martin-Bruad et al., 2021; Zhang et al., 2020; Tancik et al., 2023), unlocking exciting applications for simulated environments. Yet, these methods lack acoustic synthesis, which is essential for fully immersive experiences, Learning scene acoustics presents additional challenges (Luo et al., 2022). As sound travels from its source the listener it influenced by the geometry of the space and the materials of objects and surfaces, Room Impulse Responses (RIRs) capture these complex interactions for specific lource-listener pairs, but collecting RIRs laborious process, requiring sounds to be played and recorded at multiple positions.
To address this, recent research has focused on estimating RIRs novel positions from sparse data (Majumder et al., 2022; Singh et al., 2021; Somayazulu et al., 2024). Inspired by NeRF's success image synthesis, emerging studies (Luo et al., 2022; Su al., 2022; Liang et al., 2023a;b) apply neural fields to learn the acoustic properties of an environment, allowing for audio synthesis from new source and microphone positions. However, these approaches often overlook the critical influence of 3D scene geometry acoustics (Luo al., 2022; Su et al., 2022; Liang et al., 2023a). or they rely on additional annotations that are unavailable in real-world scenarios (Liang et al., 2023b).
In response, introduce NeRAF, method that generates both novel views and RIRs new sensor positions by jointly learning radiance and acoustic fields (Figure 1). NeRAF queries the radiance field construct voxel representation of the scene that encodes radiance and density. This grid

---

Published as a conference paper at ICLR 2025
Known
Novel
Views
Views
NeRAF
Figure 1: NeRAF synthesizes audio-visual data novel sensor positions by learning radiance and acoustic fields from a collection of images and audio recordings. It enables audio auralization and spatialization, as well as improved image rendering, all which are crucial for creating a realistic perception of space. NeRAF leverages cross moosl learning without the need for co-located audio and visual sensors for training. Our method allows for the independent rendering of each modality.
conditions the acoustic field with appearance and geometric 3D priors, without the need for additional annotations. NeRAF's cross roos- approach benefits both modalities: it achieves state-of state-of-the-ar ahet RIR synthesis while being more data-efficient and enhances novel view generation in complex scenes. We validate our method on Sound Spaces (Chen et al., 2020a; 2022b), simulated dataset, and on RAF (Chen et al., 2024), real-world dataset. Nerfstudio (Tancik et al., 2023) is an easy-to-use, modular framework for NeRF, We design NeRAF as new module to offer convenient access to audio-visual generation necessary for immersive experiences. Although the approach cross-modal, each modality can be generated independently and the model can be trained without co-located audio and visual sensors, providing flexibility for various applications. We release NeRAF's code to the community.
2 RELATED WORKS
Neural Radiance Fields. NeRF (Mildenhall et al., 2020) synthesizes novel photorealistic views of scene by modeling it as a continuous function that maps 3D spatial coordinates and viewing directions radiance. Although it has demonstrated impressive results on objects and small bounded regions, struggles with complex scenes where the camera can be oriented in any directions and content may exist at any distance. Building upon NeRF, -Martin-Brualla Brualla et al., 2021) handles unconstrained photo collections, enabling the use of diverse image sets. Other works improved NeRF to learn more complex scenes (Barron et al., 2022: Zhang et al., 2020) with higher efficiency and quality (Müller et al., 2022; Xu et al., 2022). Recently, Nerfstudio (Tancik et al., 2023) provided a modular framework that allows for a simplified end-to-end process of creating, training, and testing NeRF. combines many existing NeRF improvements to create Nerfacto, model suited for real static scenes. NeRAF is designed as a Nerfstudio module, bringing novel audio synthesis to existing state-of-the-a NeRF methods. Its cross modal learning improves Nefacto performance on large complex scenes given a small image set.
Neural Acoustic Fields. Several works have extended the applicability of neural fields to the audio domain. Luo al. (2022) (NAF) and Su et al. (2022) (INRAS) model acoustic propagation in a scene by learning an implicit representation that maps emitter- listener positions to RIRs. NAF conditions the acoustic field by learning local geometric information, while INRAS performs audio scene feature decomposition based the acoustic radiance transfer model. NACF (Liang et al., 2023b) and AV-NeRF (Liang et al., 2023a) provide vision-base based information to the acoustic field. NACF assumes the pre-acquisition of RGB-D images along with acoustic coefficients of surfaces. However, the latter is unavailable in real-life scenarios. AV-NeRF relies on pre-trained Nerfacto model to render RGB-D information at the microphone's position. It then conditions the audio rendering
2

---

Published as a conference paper at ICLR 2025
NeRF
X, d
Density
Volume Rendering
Ly
Color
x
RGB
NAcF
d1
STFT
Features Extractor
dj
Binaural RIR
X d3
Grid Sampler
LA
X dm
Figure 2: NeRAF overview. NeRF maps 3D coordinates, X, and orientations, d, to density and color. The grid sampler fills 3D grid representing the scene by querying the radiance field with voxel center coordinates and multiple viewing directions. NAcF learns to map source-microphone poses and directions to STFT. It is conditioned by extracted scene features. Predicted RIRs can be convolved with audio to obtain auralized and spatialized audio matching the scene.
the same pose. contrast, NeRAF extends beyond the camera's field of view by using NeRF to capture 3D scene information without the need for more annotations providing richer insights for the acoustic field. Contrary to AV-NeRF, our method does not need co-located audio and visual sensors for training. Therefore, it handles unequal amounts of training data for each modality, which beneficial in real-world settings. Additionally, at inference, the vision modality is not necessary for audio rendering leading to greater versatility.
RIR Synthesis. RIRs have various applications, such as enabling immersive and spatialized audio for AR and VR, performing de-reverberation or providing insights about room acoustics. However, RIRs collection in real-world environment is time consuming and needs specialized hardware. Consequently, synthesizing them has been a longstanding research topic. Traditional methods rely simulated approaches such wave-based (Gumerov & Duraiswami, 2009; Thompson, 2006;
Raghuvanshi et al., 2009; Bilbao et al., 2019) or geometric methods (Savioja & Svensson, 2015; Schissler & Manocha, 2016; Krokstad al., 1968; Vorländer, 1989; Cao al., 2016; Sprunck et al., 2022). While these methods effectively simulate sound propagation, wave-based methods face computation complexity and geometric methods struggle to simulate low-frequency acoustic phenomena such as interference and diffraction. Recent works have proposed to leverage the modeling ability of neural networks learn room acousties and estimate RIRs (Majumder et al., 2022; Luo et al., 2022; Su et al., 2022; Rainarajah et al., 2021; 2022).
Audio-Visual Learning. Some works rely on both audio and visual information to perform acoustic related tasks such audio binauralization (Gao & Grauman, 2019; Garg et al., 2023), auralization (Chen et al., 2022a; Somayazulu et al., 2024), de-reverberation (Chen et al., 2023a; Chowdhury et al., 2023) or RIR prediction (Singh et al., 2021: Majumder et al., 2022). Audio-visual learning has also demonstrated promising results for other tasks such as improving depth prediction (Christensen et al., 2020; Brunetto al., 2023; Parida et al., 2021: Zhu al., 2024), performing floorplan reconstruction (Majumder et al., 2023; Purushwalkam et al., 2021) 1), navigating through a space (Younes et al., 2023; Gao et al., 2020; Chen et al., 2023c; 2020b; Gan et al., 2020) and estimating poses (Chen al., 2023c).
3 METHOD
NeRAF generates both RGB images and binaural RIRs for any camera and microphone-source position and orientation. It achieves this by jointly learning neural radiance and acoustic fields. Although trained together, the audio and visual components can be used independently. The NeRAF pipeline, shown in Figure 2, consists three modules: NeRF, the grid sampler, and NAcF. The NeRF learns the radiance field and generates RGB images. The grid sampler queries NeRF's radiance field
3

---

Published as a conference paper at ICLR 2025
with each voxel's center, filling the 3D grid with color and density. NAcF then renders binaural RIRs, conditioned by the implicit appearance and geometrie 3D priors derived from the features extracted from this 3D grid.
3.1 MODELING ACOUSTIC THROUGH RIRS
To model the acoustic of an environment, our method learns RIRs. They describe how sound propagates between one emitter and listener depending of the scene's geometry and material properties. For example, early reflections provide information about the distance to nearby surfaces, while late reverberation reflects the overall size and structure of the scene. Learning acoustics through RTRs offers several key benefits: (1) RIRs encapsulate the acoustic behavior between specific emitter and listener positions; (2) By convolving audio signals with RIRs, we can synthesize sound that mimics how it would be heard within the scene; (3) Binaural RIRs include head-related transfer functions (HRTFs), which are essential for accurately spatializing audio.
By learning to generate RIRs, our method captures essential sound propagation phenomena, enabling realistic audio synthesis with rich acoustic details.
3.2 NEURAL RADIANCE FIELD
Neural radiance field (NeRF) (Mildenhall al., 2020) renders photo-realistic images from new view points. It represents a static scene as a 5D vector-values continuous function whose input is a 3D location X (x, 2. and 2D viewing direction d (0, Φ). and output is color (r, b) and volume density a:
NeRF: (X + (c.
NeRF shoots rays r(t) = o id through image pixels from camera origin between the near fnear and far ftr boundaries of the scene. 3D points X,, + t are sampled along the ray and, along with d used as input of two MLPs. The first one maps X,, to the ied-indeeendent density on and a corresponding feature vector. The second MLP takes the feature vector and d produce view-dependent color c,, The final color C(r) of an image pixel are rendered using volume rendering equations:
flr T(In)one,
C(r)
where T(4x) is the transmittance expressed as exp(- - fehn okdk).
NeRAF builds on Nerfacto (Tancik et al., 2023) as its NeRF backbone. Nerfacto integrates advance- ments from several works following (Mildenhall al., 2020) including camera pose refinement (Wang et al., 2021; Lin et al., 2021), per image appearance conditioning (Martin-Brualla et al., 2021), proposal sampling (Müller et al., 2022), scene contraction and hash encoding. We selected Nerfacto for its robust performance in handling real-world static scenes. However, our method is aggn-agnostic it relies directly on the radiance field without altering the underlying NeRF model.
3.3 GRID SAMPLER
Sound propagation is omnidirectional and influenced by the 3D geometry of the environment. Therefore, we provide our neural acoustic field with 3D geometric and appearance priors about the scene.
To this end, we use our grid sampler, illustrated in Figure 3, to construct a 3D volume from the NeRF model. The scene is represented as 3D grid of 53 voxels with coordinates between 0 and I in accordance with NeRF's scene contraction. We shows in Section 4.3, that = 128 leads to good performances and use it for all experiments.
We populate the grid by sequentially selecting batches of voxel centers X as queries to the radiance field. By processing voxels in order, ensure that the grid is progressively updated reflect the
4

---

Published as a conference paper at ICLR 2025
Grid Sampler
Gpt
x = (Kiti Fpr)
d, (0)
C1-W
Volume Rendering
C2-N Cp
d
d, = ())
Figure 3: Grid sampler. We represent the scene as a grid of voxels. The grid is populated by querying the radiance field with the center coordinates of each voxel, X along with multiple viewing directions, d1-N. We average the color values obtained from these directions. It results in 7-channels 3D grid containing color C... density des and the 3D coordinates of the voxel centers.
improvements in the NeRF model. To avoid slowing down training, we do not update the entire grid each training iteration and process in batches. This sampling method guarantees complete grid population.
follow (Chen & Lee, 2023) and form N 18 viewing rays per X The goal is to obtain an average color of the voxel. The radiance field returns density value Øui and color value Cui, per j [1, N] viewing directions:
As NeRF color is view-dependent,
NeRF (X d(+x) + (
(3)
For color, apply the volume rendering equation to ray containing single point, X with each viewing direction and average the values across these directions to obtain Cos. For density, compute the alpha compositing value a = exp(-o where a chosen small value. We populate the grid with these values and the 3D coordinates of the voxels, resulting in 7-channel representation.
3.4 NEURAL ACOUSTIC FIELD
The goal of our neural acoustic field (NAcF), presented Figure 4, is learn continuous neural representation of the scene's acoustics. To this end, NAcF maps microphone coordinates, X,m = (I'm Vm, 2m), and orientations, d,, = (0m. (.n). along with source position, X, = (xr.. yss zs), and orientation, d, (0s, 2,), to the short-time Fourier transform (STFT) of room impulse response:
NAcF (Xm; dm X., ds,t) t) RIR(ftt) t).
(4)
The exact field parametrization adapted to the sensors' characteristics each dataset is presented in Appendix A.
We learn RIRs through their STFT representation, resulting in STFT e RC where C, F and T denote the number of channels, frequency bins, and time bins, respectively. This is motivated by the fact that the time-frequency representation smoother and more suitable to neural network prediction than the time domain (Luo et al., 2022). Thus, NAcF learns infer fog-magnitude STFTs. To better capture perceptual effects such as reverberation, we query NAcF with STFT time bins t. Consequently, NAcF outputs the F frequency bins corresponding the time query. We normalize t between [0, 1] and encode it with positional encoding. Similar to NeRF (Mildenhall et al., 2020) pose, microphone and source coordinates well direction are also embedded to a higher dimensional space using respectively positional encoding and spherical harmonic encoding.
Given the voxel grid, we concurrently train a ResNet3D (He al., 2016) to extract relevant features for sound propagation. They serve as implicit indicators of scene's 's 3D geometry, which is central for acoustic modeling. Moreover, as shown previous works (Chen et al., 2023b; Singh et al., 2021), information about materials can be inferred by a ResNet architecture from the scene's 's appearance and contribute to audio synthesis. We perform multimodal fusion by concatenating the features vector from the grid to the encoded poses, directions and the time query. Similar to NeRF, NAcF comprises two MLP blocks. The first block maps the input to a feature vector that embeds the input information and the acoustic scene. The second focuses learning microphone specifics. For
5

---

Published as a conference paper at ICLR 2025
NAcF
Audio STFT
ResNet 3D
Left
Multi-
modal
Poses (x))
Fusion
Right
Directions (d dz)
SHE
Time Query
Figure 4: Neural acoustic field. NAcF maps microphone-soune poses and directions to either binau- ral monaural RIRs, with the number of output heads adjusted accordingly. NAcF is conditioned on scene features extracted through 3D ResNet. Both poses and time queries are transformed into a higher-dimensional space using positional encoding, while directions use spherical harmonic encoding. The output of NAcF is vector containing F' frequency values for each time query, t.
binaural microphones, this block learns the HRTF necessary for audio spatialization. contains one MLP per microphone channel.
To render the complete STFT, we query all time bins E [0, Finally, we use Griffin-Lim (Griffin Lim, 1984; Perraudin al., 2013) algorithm obtain RIR waveforms from magnitude STFTs.
3.5 LEARNING OBJECTIVE
Cross-modal learning has proven beneficial in multiple works (Section Thus, NeRAF trains jointly NeRF and NAcF. NAcF training is slightly delayed as it begins when the grid has been updated a few times. The feature extractor is jointly trained with the radiance and acoustic fields.
We do not modify the NeRF loss function Lv. It usually consists in an MSE loss between ground- truth pixel color, C(r), and the rendered one, C(r), and complementary losses, Leomps specific to each NeRF
Cv C(r) C(r) Coomp.
(5)
Inspired by (Yamamoto et al., 2019; 2020), we use combination of spectral loss LSL (Défossez et al., 2018) and spectral convergence loss Lsc (Arik et al., 2018) as our audio loss C
LA = AscLsc + ASt. CSt.,
(6)
EsL = ||lo[([s] + - @| + +)l
(7)
|||s| |||$||| |
Lsc
(8)
where å is the predicted STFT, the ground-truth STFT, ||-I|F denotes the Frobenius norm, II-I1a the L2 norm and 10-3, 3 The values of our loss hyperparameters are found empirically and given in Appendix C. The spectral convergence loss emphasizes
spectral peaks helping especially in early phases of training while the spectral loss focus on small amplitude components which tends to be more important towards the later phases of training. Our loss differs from (Yamamoto al., 2019; 2020) as we found out that the MSE spectral loss leads to better trade-off between metrics than the L1 spectral loss (Su al., 2022) and the MSE loss on magnitude STFT (Luo et al., 2022; Liang et al., 2023a). NeRAF results with those three losses are presented in Section 4.3. The complete learning objective can resumed as:
L Lv AALA-

---

Published as a conference paper at ICLR 2025
Table 1: Comparison with state-of-the-art. Performance on the SoundSpaces dataset using T60, C50, and EDT metrics (Left) and on RAF (Right) with the same metrics and STFT error. Lower score indicates higher RIR quality.
Methods
(
(dB)
Methods
T60 (56)
C50 (dR)
EDT (sec) J
STFT emor (dB)
Opus-ncarest
10.10
3.58
0.115
Opus-linear
8.64
0.097
NAF
10.08
0.71
0,021
0,64
AAC-nearest
AAC-linear
7.88
9.35
1.68
1.67
0.059
0.057
INRAS
AV-NeRF
8.01
8.11
0.73
0.79
0.021
0.025
0.39
0.36
NAF
INRAS
3.14
3.18
0.60
1.06
0.019
0.031
NAT++ INRAS+
8.19
0.53
0.017
0.64
AV-NERI
2.47
0.57
0.016
NACF
6.62
0.59
0.017
0.39
NACF
2.36
0.50
0.014
NACF w/'
7.31
0.59
0.018
0.39
NeRAF
w/
2.14
2.17
0.38
0.010
NeRAF++
6.87
0.51
0.015
0.17
4 EXPERIMENTS
First, we benchmark NeRAF against related works (Luo et al., 2022; Su et al., 2022; Liang et al., 2023a;b) on the SoundSpaces and RAF datasets. Next, we demonstrate that cross-modal learning of the acoustic and radiance fields is also beneficial to vision, resulting in enhanced results for large complex scenes trained with sparse observations. Furthermore, we evaluate NeRAF's ability to generalize across multiple scenes. To assess the effectiveness of our method when trained with limited number audio recordings, we conduct few-shot experiment. Finally, we investigate the impact of the 3D grid and our loss combination NeRAF's performance. Video demonstrations showcasing NeRAF's andio-visual generation both real and synthetic environments are available on our project page.
4.1 DATASETS
SoundSpaces. SoundSpaces (Chen et al., 2020a) is audio simulator built upon Habitat Sim (Savva et al., 2019; Szot et al., 2021; Puig et al., 2024), 3D simulator for embodied AI research. provides binaural RIRs at discrete positions of 2D grid with spatial resolution of 0.5 with four head orientations accompanied by sound source position. This popular benchmark (Gao et al., 2020; Chen al., 2020b; Luo et al., 2022; Liang et al., 2023a; Su et al., 2022; Purushwalkam al., 2021; Parida et al., 2021; Chen et al., 2022a) enables us to evaluate our method on diverse scene types while being under same settings as previous works. Following (Luo al., 2022: Liang al., 2023a; Su et al., 2022; Liang et al., 2023b), we select six indoor scenes from Replica (Straub et al., 2019). Two are small rooms (office and room 2) with rectangular walls, two are medium room (frl apartment and 4) with more complex layout and objects and two are complete apartments containing multiple rooms (apartment and 2).
RAF Real Acoustic Fields (RAF) (Chen et al., 2024) is real-world dataset that includes high- quality, densely captured room impulse response data paired with multi-view images and precise 6DoF pose tracking data for sound emitters and listeners in rooms. The room impulse response are recorded using the "carful tower", which is equipped with omnidirectional microphones. This dataset was recently used to benchmark previous works on real-world scenes.
RWAVS. RWAVS (Liang al., 2023a) is real-world dataset used to benchmark AV-NeRF. It does not contain RIRs but audio recordings. Our results on the RWAVS dataset are presented in Appendix J.
Loud
Quiet
Empty Room
Furnished Room
Figure 5: Loudness maps visualization. We visualize the intensity of predicted RIRs at different microphone positions for a given loudspeaker position and orientation. Intensities are averaged over multiple heights.
7

---

Published as a conference paper at ICLR 2025
Table 2: Quantitative results on cross-modal learning. For highly complex scenes with very sparse training images, NeRAF audio-visual joint training improves vision performances, especially LPIPS. Evaluation on 50 test novel views. Delta are computed against Nerfacto (no audio).
PSNR
SSIM f
LPIPS
Images
75
100
150
75
100
150
75
100
150
Apurtment /
Nerfacto
22.48
24.77
27.00
0,838
0.882
0.902
0,246
0.161
0,118
NeRAF
23.33
25.07
27.70
0.858
0.884
0.915
0,216
0.140
0.105
ANeRAF
+3.78%
+1.23%
+2.60%
+2.39%
+0.28%
+1.37%
-12.20%
-13.22%
-11.25%
Apormweo 2
Nerfacto
19.10
22.67
25.62
0.777
0.836
0.884
0.381
0.232
0.160
NeRAF
20.53
23.46
26.06
0.802
0.856
0.895
0.322
0.206
0.150
ANeRAF
+3.49%
+1.69%
+3.26%
+2.31%
+1.20%
-15.27%
-11.14%
-6.03%
4.2 EVALUATION
Following (Su et al., 2022; Liang et al., 2023a), we assess the quality of the predicted impulse responses using Reverberation Time (T60), Clarity (C50) and Early Decay Time (EDT). T60 reflects the overall sound decay within a room by measuring the time an impulse response takes decay by 60 dB. C50 measures the energy ratio between the first ms of RIR and the remaining portion. It relates to speech intelligibility, and the clarity of acoustics. EDT focuses on early reflections. For RAF dataset, also evaluate the STFT error which is the absolute between the predicted and the ground-trutt log-magnitude STFTs (Défossez et al., 2018; Luo et al., 2022; Chen al., 2024). To assess the effect of cross-modal learning on novel view synthesis, employ commonly used metrics: PSNR, SSIM, and LPIPS.
Metrics.
Experimental Setup. Similar to (Su et al., 2022: Luo et al., 2022: Liang et al., 2023a), we use 90% of SoundSpaces audio data for training and 10% for testing. They are pre-ge gre-generated binaural RIRs with corresponding source and microphone positions. We resample the RIR to 22.05 kHz and compute STFT using NITT W 512 and H 128 where Nirr is the number of FFT bins, W the Hann window size and H the hop length. To train the vision part, we generate RGB images using Habitat Sim. Camera poses and parameters were selected to limit the amount of visual data. For small, medium and large rooms, we use respectively 45, 75 and 150 observations. Evaluation set comprises 50 images. More details on the motivations and the generation process are available in Appendix D.
For RAF, we follow previous works experimental setup: we keep 80% of data for training and 20% for evaluation. Audios cut to 0.32s, sampled to 48 kHz and processed into STFT using NEFT 1024, W = 512 and = 256. For the vision part, we randomly select one third of the data from cameras 20 to 25. Nerfstudio automatically keep 90% of them for training and 10% for evaluation.
Vaiuo Only
Apartment 1
Apartment 2
Figure 6: Visualization of cross-modal learning impact on vision performances. We compare Nerfacto (Vision Only) with NeRAF (Audio Vision) on apartment and Learning both radiance and acoustic fields improves the quality of novel views synthesis, reducing floating artifacts and enabling more detailed images.
8

---

Published as a conference paper at ICLR 2025
3.75
0.70
0.000
NettA Avilett
3.50
0.05
3.25
0.50
601 (603
3.00
0.55
0.50
0.45
0.012
0.40
0,010
Figure 7: Few-shot learning, With 25% less audio recordings, NeRAF outperform AV-NeRF trained on all the training data. Results averaged on office 4, fri apartment 2 and apartment 2.
To ensure statistically significant results, we conduct each experiment times and average results. More details on the implementation and architecture presented in Appendix B and Appendix C.
Results. We compare NeRAF with state-of sofftftthartts methods SoundSpaces in Table 1 (Left). Similar to previous works on neural acoustic field, we also compare our method to traditional audio encoding methods AAC and Opus (for Standardization, 2006; Foundation, 2012). AAC is multichannel audio coding standard and Opus is an open audio codec. NeRAF significantly outruns existing methods for C50 and EDT. It achieves 22.4% improvement compared to NACF with temporal alignment. Compared to AV-NeRF -wichh is the method most similar ours as it also employ Nerfacto and does not rely on large amount of information known a priori- we achieve respectively 13.4%. 33.3% and 37. 37.5% improvement for T60, C50 and EDT.
C50 and 28.6% on EDT
As SoundSpaces is simulated dataset that presents limitations in term of visual and acoustics realism, we also present results on RAF in Table 1 (Right). NeRAF outperforms previous works best results, achieving -6.7% 6.7% T60, -14.2% C50, -4.5% EDT and -52.5% STFT error. The second part of the table contains comparison of methods using an additional energy decay loss term (Majumder et al., 2022; Liang et al., 2023b). While NACF initially relies on this term, other methods, with this new term, are denoted by "++". NeRAF++ still achieves state-of-the-art results, in particular for C50, EDT and STFT error. We present details about this loss in Appendix C. We showcase qualitative results of audio synthesis in Figure 5 and in Appendix I.
Cross-Modal Learning. We next investigate the impact of jointly learning acoustic and radiance fields on vision in complex scenes with very sparse training views, challenging scenario where NeRF typically struggles. We train NeRAF and Nerfacto the two large rooms. 75, 100, 150 images are used for training and the same 50 novel views are used for evaluation. Table 2 we show that joint training of cross-moda learning improves novel view synthesis. It has a stronger impact on LPIPS which is related to the human-perceived similarity between two images. Overall, the performance improvement is more significant when the training images very sparse. We present qualitative results in Figure 6: NeRAF reduces floating artifacts and enables more detailed images. Results on real scenes and additional visualizations are presented respectively in Appendix J and Appendix I.
Generalization on Multiple Scenes. We eval- uate NeRAF's ability generalize across mul- tiple scenes. To this end, we train the acous- tic field across multiple scenes without increas- ing the model size. Since Nerfacto is scene- dependent, we first train one NeRF-model for each scene. We populate 128-grid per scene us- ing the grid sampler and trained NeRFs. These grids are fed into the ResNet3D, which is trained jointly with the acoustic field. Batches are drawn from all scenes during training to ensure global convergence. Table 3 compares single-scene to multi-scene training. The results show similar per- formance, demonstrating NeRAF's capability to effectively learn the acoustic field across multiple scenes with single model. In this setup, the visual modality supports the adaptation to different rooms. However, as NeRF and NAcF are not trained jointly, the visual component does not benefit from cross- modal learning. Detailed results for each scene provided in Appendix O.
Table 3: Multi-scene training. Comparison of NeRAF performance when trained per scene ver- sus on multiple scenes. Results are averaged across office 4, frl apartment 4 and apartment 2.
Training
T60 (%) 4
C50 (dB)
EDT (sec)
Single-scene
2.29
0.37
0.010
Multi-scene 2.24
0.37
0.010

---

Published as a conference paper at ICLR 2025
Table 4: Ablation studies. Left: Influence of the grid evaluated on RAF dataset. NG stands for no
grid, 128 w/ 0 for 128-grid filled with zeros, while 128 and 256 respectively correspond to grid of resolutions = 128 and S = 256. Bold and underlined respectively indicates best and second best.
Right: NeRAF's loss study. Performances averaged across SoundSpaces dataset.
Grid
T60 (%)
C50 (dB) J
EDT (sec).
Loss
T60 (%) J
C50 (dB) J
EDT (sec)
NG
8.15
0.674
0.212
MSE
1.89
0.53
0.015
128 w/ 0
7.75
0.626
0.0205
SC+SLLI
1.56
0.42
0.013
128
7.47
0.609
0.0201
SC+SLMSE
2.14
0.38
0.010
256
7.42
0.608
0.0204
Few-shot RIR Synthesis. Previous works evaluated on SoundSpaces rely on a significant amount of RIRs, which is often unavailable in real-life scenarios. Therefore, we assess the performance of our model when trained with fewer data. We conduct few-shot experiment on one small, one medium, and one large room and average the results. As shown Figure 7, NeRAF better than AV-NeRF on all metrics when trained with 75% of the data. It still outperforms AV-NeRF on EDT when trained with only 25% the data. On RAF dataset- which is real-world dataset that already contains a more realistic amount of training data NeRAF still outperforms the previous methods when trained with 25% less data. Results on RAF are presented in Appendix E.
4.3 ABLATION STUDY
Grid Impact. We assess the grid impact on performances in Table 4 (Left) by first removing the grid (NG) and then filling the 128-grid with zeros (128 w/ 0). In both cases, NAcF is no longer conditioned by the 3D priors from the scene. Removing the grid also means we reduce the number of trainable parameters as we no longer keep the features extractor. By setting the grid to zero we remove scene-related information and still keep the exact same architecture. This way, we show that the performance gap cannot be explained only by an increased number of parameters but rather to the information extracted from the 3D-grid. We also study the impact the resolution of the grid by comparing S 128 and S 256. Increasing the resolution to 256 improves T60 and slightly C50 but is accompanied by a minor EDT decline and increased training computation time.
Loss Impact. We examine the impact of our custom loss described in Section 3.5 compared to the MSE loss used in previous works (Liang et al., 2023a; Luo et al., 2022) and the combination with LI spectral loss proposed by (Yamamoto al., 2020) in Table 4 (Right). Our loss combination significantly improves C50 and EDT but decreases T60. Thus, this loss choice is a trade-off that can be adjusted the final application. Note that NeRAF also achieves state-of-the-an performances with e.,.Sinatttiiiotioo
5 DISCUSSION
I Limitation and Future Work. While the neural acoustic field can be trained on multiple scene, it still necessary to train NeRF separately for each scene, Future work should explore advancements in generalizable NeRF to develop unified, generalizable approach for synthesizing audio-viss vudiolvisual scenes. The current approach is also limited to static scenes. Addressing the challenge learning implicit representations for audio-visual scenes with dynamic sound sources would be significant advancement.
Conclusion. We introduce NeRAF, cross-modal method that Iearns both neural radiance and acoustic fields. By conditioning the acoustic field with scene features from 3D grid containing geometric and appearance information, NeRAF enables realistic audio auralization, spatialization and novel view synthesis. Our results demonstrate improvements over other methods, enhanced novel view synthesis in complex scenes and increased data efficiency.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by the French Agence Nationale de la Recherche (ANR), under grant ANR22-CE94-000 CE94-0003 and was granted access to the HPC resources of IDRIS under the allocation 2024-AD01 1015475 made by GENCI. We would like to thank Simon de Moreau, Raoul de Charette and the anonymous reviewers for their insightful comments and suggestions.
REFERENCES
Sercan Ö Anı,, Heewoo Jun, and Gregory Diamos. Fast spectrogram inversion using multi-her heati-head convolutional neural networks. Signal Processing Letters, 2018.
Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. CVPR, 2022.
Stefan Bilbao, Archontis Politis, and Brian Hamilton. Local time-domain spherical harmonic spatial encoding for wave-based acoustic simulation. Signal Processing Letters, 2019.
Amandine Brunetto, Sascha Homauer, X Yu Stella, and Fabien Moutarde. The audio-visur batvision dataset for research on sight and sound. In IROS, 2023.
Chunxiao Cao, Zhong Ren, Carl Schissler, Dinesh Manocha, and Kun Zhou. Interactive sound propagation with bidirectional path tracing. TOG, 2016.
Changan Chen, Unnat Jain, Carl Schissler, Sebastia Vicenc Amengual Gari, Ziad Al-Halah, Vamsi Kr- ishna Ithapu, Philip Robinson, and Kristen Grauman. Soundspaces: Audio-visual navigaton in 3d environments. In ECCV, 2020a.
Changan Chen, Sagnik Majumder, Ziad Al-Halah, Ruohan Gao, Santhosh Kumar Ramakrishnan, and Kristen Grauman. Audio-visual waypoints for navigation. CoRR, abs/2008.0966 2020b.
Changan Chen, Ruohan Gao, Paul Calamia, and Kristen Grauman. Visual acoustic matching. In CVPR, 2022a.
Changan Chen, Carl Schissler, Sanchit Garg, Philip Kobernik, Alexander Clegg, Paul Calamia, Dhruv Batra, Philip W Robinson, and Kristen Grauman. Soundspaces 2.0: A simulation platform for visual-acoustic learning. In NeurIPS, 2022b.
Changan Chen, Wei Sun, David Harwath, and Kristen Grauman. Learning audio-visue visual dereverbera- tion. In ICASSP, 2023a.
Changan Chen, Wei Sun, David Harwath, and Kristen Grauman. Learning audio-visur dereverbera- tion. In ICASSP, 2023b.
Yu Chen and Gim Hee Lee. Dreg-nerf: Deep registration for neural radiance fields. In ICCV, 2023.
Ziyang Chen, Shengyi Qian, and Andrew Owens. Sound localization from motion: Jointly learning sound direction and camera rotation. In ICCV, 2023c.
Ziyang Chen, Israel Gebru, Christian Richardt, Anurag Kumar, William Laney, Andrew Owens, and Alexander Richard. Real acoustic fields: An audio-visual room acoustics dataset and bench- mark. CVPR, 2024.
Sanjoy Chowdhury, Sreyan Ghosh, Subhrajyoti Dasgupta, Anton Ratnarajah, Utkarsh Tyagi, and Dinesh Manocha. Adverb: Visually guided audio dereverberation. In ICCV, 2023.
Jesper Haahr Christensen, Sascha Hornauer, and X Yu Stella. Batvision: Learning to see spatial layout with two ears. In ICRA, 2020.
Alexandre Défossez, Neil Zeghidour, Nicolas Usunier, Léon Bottou, and Francis Bach. Sing: neural generator. NewrIPS, 2018.
International Organization for Standardization. Advanced audio coding (aac), 2006.

---

Published as a conference paper at ICLR 2025
Xiph Org Foundation. Xiph opus. http::://pps--coce..oo
Chuang Gan, Yiwei Zhang, Jiajun Wu, Boqing Gong, and Joshua Tenenbaum. Look, listen, and act: Towards audio-visual embodied navigation. In ICRA, 2020.
Ruohan Gao and Kristen Grauman. 2.5 visual sound. In CVPR, 2019.
Ruohan Gao, Changan Chen, Ziad Al-Halah, Carl Schissler, and Kristen Grauman. Visualechoes:
Spatial image representation learning through echolocation, In ECCV, 2020.
Rishabh Garg, Ruohan Gao, and Kristen Grauman. Visually-guided audio spatialization in video with geometry-aware multi-task learning. IJCV, 2023.
Daniel Griffin and Jae Lim. Signal estimation from modified short-time fourier transform. Transac- tions on acoustics, speech, and signal processing, 1984.
Nail A Gumerov and Ramani Duraiswami. broadhand fast multipole accelerated boundary element method for the three dimensional helmholtz equation. ASA, 2009.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image
recognition. In CVPR, 2016.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:14/2.6988) 2014.
Asbjørn Krokstad, Staffan Strom, and Svein Sørsdal, Calculating the acoustical room response by the use of ray tracing technique. JSV, 1968.
Susan Liang, Chao Huang, Yapeng Tian, Anurag Kumar, and Chenliang Xu. Av-nerf: Learning neural fields for real-world audio-visual scene synthesis. In NeurlPS, 2023a.
Susan Liang, Chao Huang, Yapeng Tian, Anurag Kumar, and Chenliang Xu. Neural acoustic context field: Rendering realistic room impulse response with neural fields. AV4D workshop, ICCV, 2023b.
Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey. Barf: Bundle-adjusting neural radiance fields. In ICCV, 2021.
Andrew Luo, Yilum Du, Michael Tarr. Josh Tenenbaum, Antonio Torralba, and Chuang Gan. Learning neural acoustic fields. In NeurlPS, 2022.
Sagnik Majumder, Changan Chen, Ziad Al-Halah, and Kristen Grauman. Few-shot audio-visss visual learning of environment acoustics. NeurIPS, 2022.
Sagnik Majumder, Hao Jiang, Pierre Moulon, Ethan Henderson, Paul Calamia, Kristen Grauman, and Vamsi Krishna Ithapu. Chat2map: Efficient scene mapping from multi-ego conversations, In CVPR, 2023.
Ricardo -runila, Noha Radwan, Mehdi SM Sajjadi, Jonathan Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In CVPR, 2021.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Taneik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In 2020,
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. TOG, 2022.
Kranti Kumar Parida, Siddharth Srivastava, and Gaurav Sharma. Beyond image to depth: Improving depth prediction using echoes. In CVPR, 2021.
Adam Paszke, Sam Gross. Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. NeurlPS, 2019.
12

---

Published as a conference paper at ICLR 2025
Nathanaël Perraudin, Peter Balazs, and Peter L Søndergaard. A fast griffin-lim algorithm. In Workshop on applications of signal processing to audio and acoustics, 2013.
Xavier Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Tsung- Yen Yang, Ruslan Partsey, Ruta Desai, Alexander Clegg, Michal Hlavac, So Yeon Min, al. Habitat 3.0: A co-habitat for humans, avatars, and robots. In ICLR, 2024.
Senthil Purushwalkam, Sebastia Vicenc Amengual Gari, Vamsi Krishna Ithapu, Carl Schissler, Philip Robinson, Abhinav Gupta, and Kristen Grauman, Audio-visual floorplan reconstruction. ICCV, 2021.
Nikunj Raghuvanshi, Rahul Narain, and Ming C Lin. Efficient and accurate sound propagation using adaptive rectangular decomposition. TVCG, 2009.
Anton Ratnarajah, Zhenyu Tang, and Dinesh Manocha. IR-GAN: Room Impulse Response Generator for Far-Field Speech Recognition. In Interspeech, 2021.
Anton Ratnarajah, Shi-Xiong Zhang, Meng Yu, Zhenyu Tang, Dinesh Manocha, and Dong Yu. Fast-rir: Fast neural diffuse room impulse response generator. In ICASSP, 2022.
Lauri Savioja and U Peter Svensson. Overview of geometrical room acoustic modeling techniques. ASA, 2015.
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A Platform for Embodied AI Research. In /CCV, 2019.
Carl Schissler and Dinesh Manocha. Interactive sound propagation and rendering for large multi- source scenes. TOG, 2016.
Nikhil Singh, Jeff Mentch, Jerry Ng. Matthew Beveridge, and Iddo Drori. Image2reverb: Cross-modal reverb impulse response synthesis. In ICCV, 2021.
Arjun Somayazulu, Changan Chen, and Kristen Grauman. Self-supervised visual acoustic matching. NeurlPS, 2024.
Tom Sprunck, Antoine Deleforge, Yannick Privat, and Cédric Foy. Gridless 3d recovery of image sources from room impulse responses. Signal Processing Letters, 2022.
Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J. Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, Anton Clarkson, Mingfei Yan, Brian Budge, Yajie Yan, Xiaqing Pan, June Yon, Yuyang Zou, Kimberly Leon, Nigel Carter, Jesus Briales, Tyler Gillingham, Elias Mueggler, Luis Pesqueira, Manolis Savva, Dhruv Batra, Hauke M. Strasdat, Renzo De Nardi, Michael Goesele, Steven Lovegrove, and Richard Newcombe. The Replica dataset: A digital replica of indoor spaces. arXiv preprint arXiv:1906.0559)) 2019.
Kun Su, Mingfei Chen, and Eli Shlizerman. Inras: Implicit neural representation for audio scenes. In NeurIPS, 2022.
Andrew Szot, Alex Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimir Vondrus, Sameer Dharur, Franziska Meier, Wojciech Galuba, Angel Chang, Zsolt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, and Dhruv Batra. Habitat 2.0: Training home assistants rearrange their habitat. In NeurIPS, 2021.
Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi. Justin Kerr, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, David McAllister, and Angjoo Kanazawa. Nerfstudio: A modular framework for neural radiance field development. In SIG- GRAPH, 2023.
David Thery and Brian FG Katz. Anechoic audio and 3d-video content database of small ensemble performances for virtual concerts. In ICA, 2019.
Lonny I. Thompson. A review of finite-element methods for time-harmonic acousties. ASA, 2006.
13

---

Published as a conference paper at ICLR 2025
Michael Vorländer. Simulation of the transient and steady-state sound propagation in rooms using new combined ray-trracin//mmge-souu algorithm. ASA, 1989.
Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. Nerf-: Neural radiance fields without known camera parameters. arXiv preprint rrXiv:2102.07064 2021.
Linning Xu, Vasu Agrawal, William Laney, Tony Garcia, Aayush Bansal, Changil Kim, Samuel
Rota Bulò, Lorenzo Porzi, Peter Kontschieder, Aljaž Božič, Dahua Lin, Michael Zollhöfer, and Christian Richardt. VR-NeRF: High-fidelity virtualized walkable spaces. In SIGGRAPH, 2023.
Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and UIrich Neumann. Point-nerf: Point-based neural radiance fields. In CVPR, 2022.
Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim. Probability density distillation with generative adversarial networks for high-quality parallel waveform generation. Interspeech, 2019.
Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim. Parallel wavegan: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram. In ICASSP, 2020.
AbdeIrahman Younes, Daniel Honerkamp, Tim Welschehold, and Abhinav Valada. Catch me if you hear me: Audio-visual navigation in complex unmapped environments with moving sounds. RA-L, 2023.
Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07499 2020.
Lingyu Zhu, Esa Rahtu, and Hang Zhao. Beyond visual field of view: Perceiving 3d environment with echoes and vision. Workshop on sight and sound, CVPR, 2024.
14

---

Published as a conference paper at ICLR 2025
A FIELD PARAMETRIZATION
SoundSpaces. SoundSpaces dataset lacks the complete acoustic field parameterization described in Equation (4). Microphone rotates only around the up-axis, 6, and the source is omnidirectional. Thus, NAcF can be expressed as (Xm , dm 0, X., + (RIRchi, Rf,
RAF RAF dataset contains only monaural RIRs. Consequently, we replace the two heads presented in Figure by a single head. The microphone is omnidirectional but the source is directional. We can therefore express NAcF as (Xm. X., dg 0, f) RIR (f, ().
B ARCHITECTURES
NeRF NeRAF works with any NeRF method without modifications. It only requires a radiance field that the grid sampler can query using voxel center coordinates and viewing directions. Our method only relies on NeRF to fill a voxel-grid representation of the scene. In this paper, we choose Nerfacto from Nerfstudio (Tancik al., 2023). For more details its architecture invite readers to refer to Nerfstudio website and paper.
Grid Sampler. The grid sampler creates a 3D grid of voxels with a given resolution. Grid coordi- nates are comprised between [0, 1 fitting the scene contraction performed by NeRF, Consequently, knowing the exact dimensions of the room is not necessary. Voxels center coordinates and 18 viewing directions are queried to NeRF with no architecture modification. The grid is filled with color and density information along with voxels coordinates.
NAcF. We train a ResNet3D-50 to embed the grid into features that best condition the acoustic field. For smaller scenes (office 4, room 2, apartment 2 and RAF dataset), we rely on 1,024 features. For larger scenes (frl apartment 2, 4, apartment 2) we use 2,048 features. At inference, only these features, and not the complete grid, are required perform RIR synthesis. Note that we employ average pooling the end of the ResNet, adjusting according to the grid size consistently yield the desired number of features.
NAcF consists of 2 MLP blocks. The first MLP block takes as input the multimodal fusion of grid features, encoded positions, and directions obtained through vector concatenation. It comprises layers, each followed by Leaky ReLU activation with slope of 0.1, and outputs 512 intermediate representation of the acoustic field. The second MLP block takes this intermediate representation as input and predicts the F frequencies corresponding to the time query, aiming learn the microphone specifics (e.g., HRTF for binaural microphones in SoundSpaces). It consists of one separate MIPP head per microphone channel. The final activation layer is tanh function scaled between [-10, 10] to fit the STFT magnitude range.
C IMPLEMENTATION DETAILS
Hyperparameters. We implement our method using PyTorch framework (Paszke al., 2019). We optimize NAcF using Adam optimizer (Kingma & Ba, 2014) with 31 0.9 and (3 = 0.999 and 6 10-15, The initial learning rate is 10-4 It decreases exponentially to reach 10-5, 8, For NeRF, just as AV-NeRF we keep default Nerfacto parameters.
For the first 2k iterations, we only train the NeRF part. It allows the grid be filled and updated several times using batches of 4,096 voxel-centers. After, both NeRF and NAcF are train jointly. We use batch sizes of 4,096 for NeRF and 2,048 for NAcF. Note that NeRAF is trained by shuffling all STFT time slices in the train set. In NeRAF++, loss so we train NeRAF by querying all the time bins necessaries to form the STFT. that case we use batch size of 34 STFTs. NeRAF is trained for 500k iterations but most runs reach their peak performance before, depending of the room size. We train our method on a single RTX 4090 GPU.
require the full STFT apply the energy decay
Learning Objective Details. For the learning objective defined in Section 3.5, we empirically select X 10-3 Ase 10-1 and ÀsI. 1.
15

---

Published as a conference paper at ICLR 2025
To train NeRAF++, we add an additional energy decay loss term that reinforces the energy attenuation tendency of predicted RIRs. Note that unlike INRAS+ and NACF, we do not use multi-resolution STFT and only apply the loss the predicted STFT. The audio learning objective becomes:
LA = Asc + AsiCsL + AEDCED.
(10)
with Ase 3, Ası= = 1.5 and ÄED 5.
We follow Liang et al. (2023b) formula. Given a magnitude STFT, M, we first compute its energy in each time window by calculating the magnitude's 's square and aggregating it along the frequency dimension:
M Σ (MUN))
2 ISddD.
(1I)
where M' E RDx2 represents the energy in each time window. We then sum M' along the time dimension, aggregating the energy from the current step until the end for each time step d E [1, D] capture the overall energy decay trend:
M = MM IsdED.
(12)
i=d
where M" E xx Finally, we measure the LI distance between the ground-truth energy trend and the predicted one in the log space:
LED || 1og My 1og MI
(13)
SoundSpaces. To align with previous works on SoundSpaces, we resample all RIRs from 44, 14,100 Hz to 22,050 Hz. We compute STFT with 512 FFT bins, Hann window of size 512 and hop length of 128. We obtain log-magnitude STFT using log([STFT) + 10-3). ). Like Luo et al. (2022), we cut STFT to maximum length depending of the scene. If the STFT is shorter than the maximum length we pad it with its minimal value.
RAF According to RAF (Chen al., 2024), we cut audio to 0.32 and keep RIRs at 48 kHz. We compute STFT using 1,024 FFT bins, Hann window of size 512 and hop length of 256.
Queries Encoding. We give the 3D position of microphone and source to NAcF. They are normal- ized using minimum and maximum values of the poses in the training set. We add a m margin. Then, they undergo multi-scale positional encoding with N 10 frequencies and maximum frequency exponent of e = 8. We obtain direction vectors from the microphone rotation and apply spherical harmonic encoding with 4 levels. The time queries are normalized within the range (0, 1] and go through positional encoding with N = 10 and 8.
Evaluation Metrics. We inverse the transform of the predicted STFT using PyTorch Griffin-Lin algorithm running on GPU. We compute metrics against ground-trutt waveforms.
To evaluate the quality of synthesized RIR waveforms w, use T60, C50 and EDT errors obtained as follows:
T60(w, wGr) [60 w) T60(
(14)
T60(waT)
C50(w, wor) CC5(w) w) C50(war)l,
(15)
EDT(w, WGT) EED((w) - EDT(weT)).
(16)
For RAF dataset, we follow previous works and also assess the quality of the time-frequency representation using the STFT error. To ensure fair comparison with methods directly predicting RIRs in the time domain, we perform the metric on the STFT obtained from the RIR in time domain.
It means that we first go from the STFT predicted by NeRAF to a RIR waveform through Griffin-Lim algorithm and then compute its STFT before applying the metric. The STFT error is expressed the absolute error between the predicted and the ground- log-magnitude STFTs:
STFT error (M, Mcr) log log(Mrr)
(17)
To evaluate NeRF performances we choose the commonly used metrics such as PSNR, SSIM and L.IPS.
16

---

Published as a conference paper at ICLR 2025
Office 4
apartment
Apartment
Figure A1: Locations of visual observations for 3 SoundSpaces scenes. Blue dots corresponds to training views and green crosses to evaluation views. The exact poses coordinates are given in the supplementary material.
D SCENE VIEWS
SoundSpaces 1.0 provides code to generate RGB and depth observations at microphone positions and orientations. They corresponds to 128 x 128 resolution images with 60 field-of-view (FOV) sampled every 0.5 m with orientations 0 E [0 90" 180%, 270%). In NeRAF, we decided to not rely them for NeRF training and instead generate our own observations using Habitat Sim. This is motivated by real scenarios constraints. Views of the space may obtained via video recorded in the room. It is unlikely that someone will stop every 0.5 m and turn around to obtain images at each orientations. Moreover, those kind of views are not optimal for NeRF training leading, in the worst case, to the failure of less sophisticated methods, and, in the best case, the of more data. We also wanted to decorrelate microphone poses and camera poses for greater freedom in the usage of the method.
SoundSpaces.
With Habitat Sim we rendered higher resolutions images with size 512 x 512 and with a 90 FOV. The larger FOV ensures the room, orienting the camera to face its center with a random offset. For small room, medium and large room we use respectively 45, 75 and 150 training images. 50 test poses are randomly sampled in the room. Figure Al presents examples of the positions obtained.
overlap between views. We randomly select positions at the edge of
RAF, Real Acoustic Fields dataset captures high-quality and dense room impulse response data paired with multi-view images using VR-NeRF "Eyeful Tower" camera rig (Xu et al., 2023). contains 22 cameras at different heights and pointing different directions. We chose cameras 20 25 for training NeRAF's vision part. This choice ensures that we have sufficient overlapping views. The images have 0.7 megapixels with resolution (i.e., 684 1024). For more details on those data we invite readers to refer to VR-NeRF paper. To train NeRAF, as we want to limit the number of images necessaries, we randomly sub-select one third of those images. We let Nerfstudio select 90% of them for training and 10% for evaluation.
0.028
0.90
NeRAF
Avtiert
9.5
0.026
INRAS
T60
8.5
EDT
0.024
0.70
0.65
7.5
0.60
40
60
80
100
40
60
80
100
40
60
80
100
trainnnng
Figure A2: Evaluation of NeRAF on RAF with a decreasing number of training data.
17

---

Published as a conference paper at ICLR 2025
Table A1: Audio model number of parameters along with storage and speed requirements inference.
Grid Features
Parameters (Million)
Storage (MB)
Speed (ms)
1024
55,49
77.93
2.33
2048
71.89
97.84
2.64
E FEW-SHOT ON RAF.
RAF is real-world dataset that contains realistic number of training data compared SoundSpaces. We show in Figure A2 that NeRAF still outperforms INRAS, NAF and AV-NeRF with 25% less training data making it more data efficient.
F NUMBER OF PARAMETERS, STORAGE AND INFERENCE SPEED
We provide in Table A1 the numbers of parameters of the NeRAF audio model and evaluate its inference storage requirements
speed. We present results when the 1,024 and 2,048 features extracted from the grid. For RAF dataset, all results presented in the paper obtained using 1,024 features while for SoundSpaces half the scenes use 1,024 (office room 2, apartment 2) and the other half 2,048 (frl 2, frl 4, apartment 1). The speed was averaged over the RAF dataset a single RTX 4090. Note that at inference, the grid is not necessary and the extracted features are sufficient allowing the method be faster and more compact. As inference speed time is two orders of magnitude smaller than the length of the generated audio, NeRAF can be used for real-time applications.
G SOCIETAL IMPACT
By synthesizing realistic audio-visual scenes, NeRAF can enable immersive experiences in VR and gaming. Understanding acoustics is also useful for applications such as sound de- de-reverberation, source localization, and agent navigation. However, if misused, our method could contribute to the creation of misleading media, including the ability mask and lie about someone's location.
H GRID VISUALIZATION
We display examples of sectional views of grids obtained using the grid sampler in Figure A3. They are then encoded into features using ResNet, which condition our neural acoustic fields.
Office
Frl apartment 2
Apartment
Figure A3: Sectional views 1283 grids obtained using the grid sampler.
18

---

Published as a conference paper at ICLR 2025
R
R
Prediction
Inquency
6mz
Office
Fri apartment
Apartment
Figure A4: Examples of predicted log- megnituddd of binaural RIR STFTs compared to corresponding ground truth (GT). L and respectively stand for left and right.
ADDITIONAL RESULTS VISUALIZATION
Audio qualitative results. We provide examples of log- magnitude STFTs predicted with NeRAF in Figure A4.
Distance-aware Spatialized Audio. render binaural RIRs at various distances from the sound source and perform auralization by convolving anechoic sound with these RIRs. This demonstrates one of the primary applications of NeRAF. In Figure A5, we illustrate that NeRAF is distance-aware: the microphone approaches the sound source, the amplitude increases, and it decreases as the microphone moves away. Additionally, when the microphone is positioned to the right or left of the source, the audio channels' amplitudes reflect this spatial positioning.
We provide video examples of NeRAF generation in supplementary material. The chosen audios are open source and sourced from AVAD-VE (Thery Katz, 2019).
Video.
Cross-modas Learning. We provide additional qualitative results the effect of joint learning neural acoustic and radiance fields in Figure A6.
19

---

Published as a conference paper at ICLR 2025
Approaching Sound Source
Reen Soued
Boxe Soure ScRne
Nos bounce
Leaving Sound Source
Himoohes Fottonn
Morusaen Poction
Hicoptone Koetion Sours
Scund
Source on the left
Source on the right
Soaad huase
Soend
Figure A5: Examples of sounds generated with NeRAF on frl apartment 2. Our method successfully synthesizes spatialized and distance-aware sounds.
20

---

Published as a conference paper at ICLR 2025
Vision Only
Audio + Vision
Vision Only
Audio + Vision
Figure A6: Additional visualization of cross-modal learning benefits on the visual modality.
21

---

Published as a conference paper at ICLR 2025
Table A2: Comparison with state-of-the-ard methods RWAVS dataset. Bold is best, underlined is
second best.
Office
House
Apartment MAG
Outdoors
Overall
Methods
MAG
ENV
MAG
ENV
ENV
MAG
ENV
MAG
ENV
INRAS
1,405
0.141
3.511
0.182
3,421
0.201
1.502
0.130
2.460
0.164
NAF
1.244
0.137
3.259
0.178
3.345
0.193
1.284
0.121
2.283
0.157
ViGAS
1.049
0.132
2.502
0.161
2.600
0.187
1.169
0.121
1.830
0.150
AV-NeRF
0.930
0.129
2.009
0.155
2.230
0.184
0.845
0.111
1.504
0.145
NeRAF
0.876
0.125
2.312
0.156
2.366
0.184
0.829
0.110
1.596
0.144
Chen al., Novel-view Acoustic Synthesis, CVPR 2023
J RWAVS DATASET
Although we aware AV-NeRF releases and evaluates its method on the RWAVS dataset, we chose not present those results in the main paper. This is motivated by the difference between the tasks. NeRAF focus learning RIRs, allowing the auralization of any given audio at inference. However, the RWAVS dataset does not contain RIRs but recordings of audio signals (e.g., music) played in given environment. On this dataset, the network is therefore supposed to implicitly leam RIRs from them. To the best of our knowledge, it was never demonstrated that the representation learned can be used for the auralization of new audios.
Despite this, we present NeRAF performances on this dataset Table A2. Similar to AV-NeRF, adapt NeRAF query frequencies instead of time and learn STFT masks. The same inverse modifications were done by AV-NeRF to evaluate on Sound Spaces (please see their supplementary material for more details). For every indoor scenes we extract 4,096 features from the grid and 1,024 for outdoors. This difference can be explained by the complexity of indoor scenes (e.g.. many objects) compared outdoors in this dataset. Because our loss combination tailored for explicit RIR learning, we switch to MSE between magnitude STFTs.
Overall, NeRAF outperforms previous works tailored for RIR predictions and outperforms AV-NeRF in office, outdoors and slightly on apartment (ENV). We believe NeRAF is less performant on House because of the unusually large number of small objects in the scene. would need a very high-resolution grid to capture them.
RWAVS contains very complex environments on which we can show that NeRAF cross-modal training benefits vision. Results are presented in Table A3. This demonstrates that learning both acoustics and radiance fields is interesting on both real and simulated data.
Table A3: Quantitative results on RWAVS large complex scenes with sparse training data.
PSNR 1
SSIM
LPIPS 1
Training Images
1/6
1/4
1/6
1/4
1/6
1/4
House
Nerfacto
18.20
18.70
0.674
0.708
0.339
0.270
NeRAF
18.43
18.85
0.685
0.710
0.324
0.264
ANeRAF
+1.26%
+0.81%
+1.65%
+0.25%
-4.31
-2.14%
Apartment
Nerfacto
19.14
19.64
0.746
0.776
0.251
0.211
NeRAF
19.24
19.68
0.748
0.778
0.249
0.210
ANeRAF
+0.51%
+0.23%
+0.23%
+0.23%
-0.96%
-0.34%
22

---

Published as a conference paper at ICLR 2025
K RATIONALE FOR USING 3D PRIORS
Unlike light, sound propagates as a spherical wavefront in 3D from any point of emission. This means that sound waves spread outward omnidirectionally and interact with the entire 3D room's geometry, bouncing off surfaces, being absorbed, and reflecting throughout the space. Even with directional source, which emits sound more strongly in certain directions, the propagation remains omnidirectional in space, influenced by environmental interactions rather than constrained by the initial emission direction.
Previous works (Liang et al., 2023a;b) rely on images (2D) to provide scene information to the acoustic field but we argue that conditioning on 3D information better suits the properties sound propagation. For example, features like reverberation (linked to the room volume and surfaces) depend on complete 3D volume, not just what is visible front of the microphone or camera. By leveraging 3D information, acoustic fields can capture the acoustics of the scene more effectively.
In this paper, we propose to use NeRF as a convenient way for providing 3D scene priors - geometry and appearance directly to the acoustic field without needing explicit 3D annotations, such as meshes or point clouds, which are complex to obtain.
L IMPACT OF FURNITURES
We assess the impact of furnitures nearby a microphone on the predicted RIR. We conducted a first study the apartment and apartment scenes from SoundSpaces, which both include furnished and spacious areas. Given a source, comparing performance across these areas did not reveal a clear trend; differences were observed in both directions. To extend the analysis, we averaged performance across all microphones within each area, regardless of the source. We also conducted this study on a real-scene, Furnished Room, from the RAF dataset. Again, results varied by scene, with consistent pattern (see Table A4). Thus, we find no correlation between microphone's 's proximity to furnitures and prediction accuracy.
Table A4: Performance comparison for microphones located near furniture (Furnished) and in spacious areas (Spacious). No correlation is observed between a microphone's proximity to furniture and prediction accuracy.
Area
T60
C50
EDT
Apartment
Furnished
2.689
0.598
0.0165
Spacious
2.810
0.521
0.0116
Apartment
Furnished 2.382
0.493
0.0138
Spacious 3,470
0.566
0.0102
Furnished Room
Furnished
7.630
0.624
0.0171
Spacious
6.721
0.626
0.0156
M UNKNOWN SOURCES
To evaluate NeRAF's ability to generate RIRs at sources pose unseen during training, we create new split of the RAF dataset with 85% of sources for training and 15% for testing, maintaining similar data volume per split as in the original split. Indeed, the original RAF's split includes overlap in sources between training and testing, though not in mource-microphone pairs. This aligns with previous works focusing on RIR prediction at new source-microphone pairs instead of unknown sources (see Table 1). On the unseen source split, NeRAF achieves T60 8.04, C50 0.76, EDT 0.028 and STFT error 0.171, suggesting that NeRAF can generate RIR for unseen sources.
23

---

Published as a conference paper at ICLR 2025
N SEPARATE TRAINING OF EACH MODALITY
As shown Table 2 and Figure A6, joint training improves image generation in large, complex scenes with sparse visual observations. Here, we asses the impact of separate training on the audio modality. In this scenario, a NeRF (without NAcF) is fully trained and used to populate 3D grid. Subsequently, this grid is used train NAcF without the NeRF component. Audio generation performance remains comparable for both approaches, which is expected since the acoustic field in both setups leverages 3D grid representation of the scene. Regarding image generation performance, for smaller scenes with sufficient observations, such office 4, there is notable difference in visual performance between joint and separate training. Detailed results can be found in Table A5.
Table A5: Comparison of joint training vs. separate training for the audio and visual components. Joint training improves visual performance in complex scenes with limited training data, while audio generation remains comparable in both cases.
Training
T60 4
C50
EDT 4
PSNR
SSIM
LPIPS
Office 4
Joint
1.22
0.25
0.007
25.99
0.921
0.081
Separate
1.22
0.26
0.007
25.94
0.916
0.076
Apartment
Joint
2.73
0.57
0.015
27.70
0.915
0.105
Separate
2.71
0.57
0.015
27.00
0.902
0.118
O MULTI-SCENE TRAINING
To supplement the results presented Table we provide an evaluation of multi-scene model on each individual scene. The model was trained across three scenes: office 4, frl 4, and apartment 2, and evaluated on each of these scenes. We compare these results with those obtained from single-scene training in Table A6. Each grid is encoded into 1,024 features using ResNet3D.
Table A6: Comparison of NeRAF performances when trained multiple scenes against single scene.
Training
T600 +
C50 1
EDT L
Office
Single-scene
1.22
0.25
0.007
Multi-scene
1.37
0.28
0.007
Fr!
Single-scene
2.76
0.31
0.010
Multi-scene 2.74
0.31
0.010
Apartment 2.88
Single-scene
0.54
0.013
Multi-scene
2.62
0.53
0.013
Mean
Single-scene
2.29
0.37
0.010
Multi-scene
2.24
0.010
24


Reviews:
[{"review_id": "dgftISjPHH", "reviewer": "Reviewer_Y2XC", "summary": "The paper presents NeRAF, a novel framework for synthesizing both vision and spatial audio in 3D scenes (spatial audio is the main focus of this paper). NeRAF’s key innovation is its ability to jointly learn visual neural radiance fields and spatial acoustic fields. The approach leverages cross-modal learning, meaning it incorporates visual features to enhance audio field prediction, and vice versa. It improves efficiency and realism in generating room impulse responses (RIRs). Unlike previous methods, NeRAF does not require aligned audio-visual data, making it more versatile and data-efficient. The method achieves state-of-the-art results in accuracy and quality of audio rendering, particularly in benchmarks like SoundSpaces and the Real Acoustic Fields (RAF) dataset.", "questions": "1. See weakness section.\n\n2. Experiment: since the framework is scene-specific where the train and test are based on the same room, how do the authors guarantee the train and test data are different? Another important discussion, if we fix a source position, what the predicted RIR accuracy difference on microphone position close to furniture and position on spacious area?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. **Enhanced Realism**. NeRAF uniquely integrates visual and acoustic information, using visual scene features to inform the acoustic model and vice versa. This cross-modal approach improves the fidelity of both visual and acoustic outputs, leading to more realistic audio-visual experiences without the need for additional aligned datasets.\n\n2. **Data EVersatility**: Unlike other methods requiring dense, aligned audio-visual data, NeRAF operates effectively with sparse datasets and does not rely on co-located audio-visual sensors for training. This flexibility makes it suitable for various real-world applications, especially where dense data collection is impractical.\n\n3. **State-of-the-Art Performance**: This paper demonstrates measurable improvements over existing methods in key metrics like Clarity (C50), Reverberation Time (T60), and Early Decay Time (EDT), setting a new standard for realistic audio-visual scene synthesis.", "weaknesses": "1. **Problem Setting Correctness**: In section 3.4, the authors write both the audio source and two microphones' (two ears) have directions. If so, it contradicts with the claim that: Sound propagation is omnidirectional (L205), because sound propagation from directional source isn't uniformally directional and thus may not be measured by RIR (the situation becomes more complex if the microphones are also directional). Moreover, the SoundSpaces 1.0 data provided RIR are emitted by point source without specific direction. The authors should 1. provide more detail discussion and theoretical proof showing how RIR they learned fits for directional source and microphones setting. 2. More detail regarding how directional information are used in SoundSpace 1.0 data or their own Habitat-sim simulated data (like how to set the direction and how to ensure the simulated RIR is conditioned on the set direction).\n\n2. **Motivation Concern**: The task of this paper is to learn an acoustic field that can predict binaural RIR, but this paper proposes to learn auxiliary visual NeRF together with the acoustic field learning. One question naturally arises is that the authors should show why learning NeRF is essential for learning acoustic field, through either ablation study or theoretical analysis. If crossmodal visual learning is essential to improve acoustic field learning, is there any other more light weight visual learning mechanism other than NeRF can also improve the acoustic field learning. Given the title \"3D scene infused neural radiance and acoustic fields\" emphasizes the neural radiance field, more detailed discussion and investigation on the impact of visual radiance field on acoustic fields are required.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OSE7IJb2rR", "reviewer": "Reviewer_v8xr", "summary": "The paper introduces a method that jointly learns acoustic and radiance fields named as NeRAF, which can synthesize both novel views and spatialized room impulse responses at arbitrary target viewpoints. They claim by conditioning the acoustic field on 3D radiance field, NeRAF can benefit both visual and acoustic modalities. Experiments on both synthetic and real datasets show NeRAF generates high-quality audio and improves performance on both novel view synthesis and acoustic modeling tasks.", "questions": "Please see the weaknesses above.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ The paper aims for a very interesting topic: joint learning of visual and acoustic rendering for 3D scenes. The authors proposed an effective pipeline to solve this challenging task, and explore how the two modalities can combine together.\n\n+ Thorough experiments are conducted on both synthetic and real datasets, comparing with reliable baselines and plenty of ablations to show the effectiveness of the proposed modules. Especially for the grid sampling visualization, from which we can see what physical meaning the grids have learnt through the joint training.\n\n+ The paper is well organized and easy to follow with clear presentation.", "weaknesses": "- Weak cross-modality learning. The proposed cross-modality learning way for acoustic learning is to use ResNet3D to extract 3D geometric and appearance priors, and then inject it for acoustic rendering. However, I doubt it due to two aspects: \n    - The model is scene-dependent, which means fits one scene per model. In this case, it's hard to guide the ResNet3D to truly focus on properties that affect acoustic learning and physically correspond to geometry/materials etc., especially without any explicit supervision on it. The model can easily just take the grid sampling extractor as a large latent feature generator to help the model memorize the data distribution in the room.\n    - From visualization of grids in Figure A3, we can see the grids feature is more correlated to visual rendering task, which makes sense since visual rendering has strong supervision on the grids directly (density and color etc.). However, this can be tricky if we want to understand how the two modalities are actually integrated. The grids seem to fail to correlate with acoustics. According to the first aspect, strong evidence is needed here to support the two modalities are actually helping each other in a way that has claimed physical meanings (instead of just more latent features that benefits memorizing the scene)\n\n- One way that may help to really use the visual modality is to do multi-conditioning training, i.e. one model fits multiple scenes. In this case, the visual can be helpful to adapt to different rooms, and we may be able to observe how audio and visual modality integrate. With real visual representation, the model can do more than just fitting single room. \n\n- If the visual and audio modality are helping each other in a pure latent way without any physical meaning, the soundness of claimed motivations and contribution will be largely weaken.\n\n- The grids are in the fixed shape, which might be hard to fit large room, where the resolution can be super coarse and may degrade the performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SpXK0hiBN4", "reviewer": "Reviewer_h8tx", "summary": "This paper introduces NeRAF, a new method for simultaneously learning acoustic fields and visual radiance fields. The authors demonstrate that this joint training approach enhances the quality of both generated room impulse responses and RGB images. Additionally, the proposed method offers flexibility, allowing for independent generation of single modalities during inference. Experiments conducted on both simulated (SoundSpace) and real-world (RAF) datasets show that NeRAF outperforms state-of-the-art methods in both audio and visual generation tasks.", "questions": "- Why choosing magnitude spectrogram + griffin-lim instead of directly optimizing on waveform?\n- Acoustic data in SoundSpace is actually in 2D (RIRs are the same for different heights). How does the 3D grid features help? Are 2D features good enough?\n- In the demo video, are all videos shown on the right synthesized?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- In general, I like the proposed multi-modal joint training approach that is appealing for its concise design and its potential to enhance the generated quality of both visual and acoustic fields. The flexibility to leverage single modalities when needed is also an advantage.\n- The paper is generally straightforward to follow. Extensive studies have been conducted on both synthetic and real datasets, along with healthy supplementary materials, code, and a demo video.", "weaknesses": "- The task of rendering on both visual and acoustic signals has been previously explored (e.g., AV-Nerf), thus limiting the novelty of this work. From a technical perspective, the primary distinctions between NeRAF and AV-Nerf lie in 1) the use of 3D scene features for RIR generation and 2) the joint training pipeline. However, these technical contributions are not particularly significant to me.\n- More analyses on cross-modal learning are needed. Table 2, Figure 6, appendix I and J show that joint training of cross-modal learning improves novel view synthesis but it is unclear whether there is any improvement in acoustic field synthesis.\n- It is unclear why 3D ResNet is the best scene feature extractor. Some ablation studies would be helpful here.\n- The loudness maps in Figure 5 exhibit some non-smooth textures. Why does it happen? Can NeRAF effectively represent a continuous acoustic field?\n- NeRAF demonstrates better performance on RIR and image generation metrics, yet significant challenges persist. These include multi-scene representations, multiple or/and unknown sources, discrepancies between synthetic and real-world scenarios, and inverse problems. While this paper showcases improved training data efficiency, the impact is limited due to the density of synthetic datasets like SoundSpace. Even at 20%, the volume of training data remains substantial.\n- From the perspective of human perception, it is unclear whether users can perceive actual difference between NeRAF and AV-NERF.\n- NeRAF is obviously not a small model (Table A1). Especially it is still for single scene purpose.\n- Some minor comments:\n  - there was no context for Nerfstudio module until L90. Please clarify and add references in abstract and intro.\n  - typo: wrong subscript $\\sigma_{vi}$ in Eqn 4\n  - In demo video, it would be better to provide trajectories in real scenes as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5CQD0R6Rpp", "reviewer": "Reviewer_DBxo", "summary": "This paper introduces a novel method that simultaneously learns acoustic and radiance fields for a recently introduced problem, novel acoustic view synthesis. The core idea involves integrating 3D scene features when rendering room impulse responses for the acoustic field. Experiments conducted on both a synthetic dataset using SoundSpaces and the recently introduced real-world dataset, RAF, demonstrate the effectivenss of the proposed method over previous methods.", "questions": "Please see above.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper tackles an interesting and novel problem, and the concept of jointly learning acoustic and radiance fields is well-motivated, as visual and acoustic information can complement each other to learn spatial cues.\n\n- Although AV-NeRF also utilizes vision to aid in learning an acoustic field, the proposed approach uniquely learns a radiance field and introduces a grid-sampler to extract visual features, which the acoustic field is conditioned upon.\n\n- Table 1 demonstrates that the proposed method achieves notable improvements compared to AV-NeRF, INRAS, etc.\n\n- It's good to see that the proposed method is evalauted on a real-world dataset, apart from simulated data in SoundSpaces. The authors also faithfully evaluate on the RWAVS dataset and reported the results, despite the different settings of the tasks.", "weaknesses": "- In supplementary video, it only shows the comparisons of the proposed method againt audio-only baseline, which is a weak baseline? It would be compare the qualitative differences between the proposed method and the closest performing prior work (AV-NeRF or NACF), to demonstrate the quantitaive improvement is perceptually meaningful.\n\n- The main new component is the grid sampler component that extracts scene features to condition the RIR rendering. How beneficial is the joint training? What if separately training the visual component and a fixed grid sampler? How useful is the scene feature from grid sampler? What if a single image is provided as input and a ResNet 2D is used? How helpful is conditioning the task on 3D scene information?\n\n- Following on the above questions, some additional ablations could be informative, such as \n\n1. Comparing joint training vs. separate training of visual and acoustic components\n\n2. Evaluating the impact of using a fixed vs. trainable grid sampler\n\n3. Comparing 3D scene features from the grid sampler vs. 2D features from a single image\n\n4. Quantifying the benefit of 3D scene conditioning vs. no scene conditioning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)