NeRAF: 3D Scene Infused Neural Radiance and Acoustic Fields

## Abstract

Abstract Sound plays a major role in human perception. Along with vision, it provides essential information for understanding our surroundings. Despite advances in neural implicit representations, learning acoustics that align with visual scenes remains a challenge. We propose NeRAF, a method that jointly learns acoustic and radiance fields. NeRAF synthesizes both novel views and spatialized room impulse responses (RIR) at new positions by conditioning the acoustic field on 3D scene geometric and appearance priors from the radiance field. The generated RIR can be applied to auralize any audio signal. Each modality can be rendered independently and at spatially distinct positions, offering greater versatility. We demonstrate that NeRAF generates high-quality audio on SoundSpaces and RAF datasets, achieving significant performance improvements over prior methods while being more data-efficient. Additionally, NeRAF enhances novel view synthesis of complex scenes trained with sparse data through cross-modal learning.
NeRAF is designed as a Nerfstudio module, providing convenient access to realistic audio-visual generation.
Project page: https://amandinebtto.github.io/NeRAF

## 1 Introduction

Sound is fundamental to human perception. Imagine standing in the middle of a bustling city. It’s not only the visual cues but the sounds of car horns, footsteps, and chatter that guide your perception and decisions.
Beyond conveying environmental details, sound provides crucial insights into context and atmosphere, enriching our understanding of the world in ways that sight alone cannot. In gaming and AR/VR simulations, audio is key to immersion, enhancing the realism of virtual environments and making experiences feel authentic.

Advances in novel view synthesis have enabled the generation of high-quality, photorealistic images from any camera position using a set of captured images (Mildenhall et al., ; Martin-Brualla et al., ; Zhang et al., ; Tancik et al., ), unlocking exciting applications for simulated environments. Yet, these methods lack acoustic synthesis, which is essential for fully immersive experiences. Learning scene acoustics presents additional challenges (Luo et al., ). As sound travels from its source to the listener it is influenced by the geometry of the space and the materials of objects and surfaces. Room Impulse Responses (RIRs) capture these complex interactions for specific source-listener pairs, but collecting RIRs is a laborious process, requiring sounds to be played and recorded at multiple positions.

To address this, recent research has focused on estimating RIRs at novel positions from sparse data (Majumder et al., ; Singh et al., ; Somayazulu et al., ). Inspired by NeRF’s success in image synthesis, emerging studies (Luo et al., ; Su et al., ; Liang et al., ; ) apply neural fields to learn the acoustic properties of an environment, allowing for audio synthesis from new source and microphone positions. However, these approaches often overlook the critical influence of 3D scene geometry on acoustics (Luo et al., ; Su et al., ; Liang et al., ), or they rely on additional annotations that are unavailable in real-world scenarios (Liang et al., ).

In response, we introduce NeRAF, a method that generates both novel views and RIRs at new sensor positions by jointly learning radiance and acoustic fields (Figure 1). NeRAF queries the radiance field to construct a voxel representation of the scene that encodes radiance and density. This grid conditions the acoustic field with appearance and geometric 3D priors, without the need for additional annotations. NeRAF’s cross-modal approach benefits both modalities: it achieves state-of-the-art RIR synthesis while being more data-efficient and enhances novel view generation in complex scenes. We validate our method on Sound Spaces (Chen et al., ; ), a simulated dataset, and on RAF (Chen et al., ), a real-world dataset.
Designed as a Nerfstudio module, NeRAF offers convenient access to audio-visual generation necessary for immersive experiences. Although the approach is cross-modal, each modality can be generated independently and the model can be trained without co-located audio and visual sensors, providing flexibility for various applications. We release NeRAF’s code to the community.

Figure: Figure 1: NeRAF synthesizes audio-visual data at novel sensor positions by learning radiance and acoustic fields from a collection of images and audio recordings. It enables audio auralization and spatialization, as well as improved image rendering, all of which are crucial for creating a realistic perception of space. NeRAF leverages cross-modal learning without the need for co-located audio and visual sensors for training. Our method allows for the independent rendering of each modality.
Refer to caption: extracted/5907188/figures/concept_noSmiley.png

## 2 Related works

#### Neural Radiance Fields.

NeRF (Mildenhall et al., ) synthesizes novel photorealistic views of a scene by modeling it as a continuous function that maps 3D spatial coordinates and viewing directions to radiance. Although it has demonstrated impressive results on objects and small bounded regions, it struggles with complex scenes where the camera can be oriented in any directions and content may exist at any distance.
Building upon NeRF, (Martin-Brualla et al., ) handles unconstrained photo collections, enabling the use of diverse image sets. Other works improved NeRF to learn more complex scenes (Barron et al., ; Zhang et al., ) with higher efficiency and quality (Müller et al., ; Xu et al., ).
Recently, Nerfstudio (Tancik et al., ) provided a modular framework that allows for a simplified end-to-end process of creating, training, and testing NeRF. It combines many existing NeRF improvements to create Nerfacto, a model suited for real static scenes.
NeRAF is designed as a Nerfstudio module, bringing novel audio synthesis to existing state-of-the-art NeRF methods. Its cross-modal learning improves Nefacto performance on large complex scenes given a small image set.

#### Neural Acoustic Fields.

Several works have extended the applicability of neural fields to the audio domain.
Luo et al. () (NAF) and Su et al. () (INRAS) model acoustic propagation in a scene by learning an implicit representation that maps emitter-listener positions to RIRs. NAF conditions the acoustic field by learning local geometric information, while INRAS performs audio scene feature decomposition based on the acoustic radiance transfer model.
NACF (Liang et al., ) and AV-NeRF (Liang et al., ) provide vision-based information to the acoustic field. NACF assumes the pre-acquisition of RGB-D images along with acoustic coefficients of surfaces. However, the latter is unavailable in real-life scenarios. AV-NeRF relies on a pre-trained Nerfacto model to render RGB-D information at the microphone’s position. It then conditions the audio rendering at the same pose.
In contrast, NeRAF extends beyond the camera’s field of view by using NeRF to capture 3D scene information – without the need for more annotations – providing richer insights for the acoustic field. Contrary to AV-NeRF, our method does not need co-located audio and visual sensors for training. Therefore, it handles unequal amounts of training data for each modality, which is beneficial in real-world settings. Additionally, at inference, the vision modality is not necessary for audio rendering leading to greater versatility.

#### RIR synthesis.

RIR have various applications, such as enabling immersive and spatialized audio for AR and VR, performing de-reverberation or providing insights about room acoustics. However, RIR collection in real-world environment is time consuming and needs specialized hardware. Consequently, synthesizing them has been a longstanding research topic. Traditional methods rely on simulated approaches such as wave-based (Gumerov & Duraiswami, ; Thompson, ; Raghuvanshi et al., ; Bilbao et al., ) or geometric methods (Savioja & Svensson, ; Schissler & Manocha, ; Krokstad et al., ; Vorländer, ; Cao et al., ; Sprunck et al., ). While these methods effectively simulate sound propagation, wave-based methods face computation complexity and geometric methods struggle to simulate low-frequency acoustic phenomena such as interference and diffraction.
Recent works have proposed to leverage the modeling ability of neural networks to learn room acoustics and estimate RIRs (Majumder et al., ; Luo et al., ; Su et al., ; Ratnarajah et al., ; ).

#### Audio-Visual learning.

Some works rely on both audio and visual information to perform acoustic related tasks such as audio binauralization (Gao & Grauman, ; Garg et al., ), auralization (Chen et al., ; Somayazulu et al., ), de-reverberation (Chen et al., ; Chowdhury et al., ) or RIR prediction (Singh et al., ; Majumder et al., ).
Audio-visual learning has also demonstrated promising results for other tasks such as improving depth prediction (Christensen et al., ; Brunetto et al., ; Parida et al., ; Zhu et al., ), performing floorplan reconstruction (Majumder et al., ; Purushwalkam et al., ), navigating through a space (Younes et al., ; Gao et al., ; Chen et al., ; ; Gan et al., ) and estimating poses (Chen et al., ).

Figure: Figure 2: NeRAF overview. NeRF maps 3D coordinates, $\mathbf{X}$, and orientations, $\mathbf{d}$, to density and color. The grid sampler fills a 3D grid representing the scene by querying the radiance field with voxel center coordinates and multiple viewing directions. NAcF learns to map source-microphone poses and directions to STFT. It is conditioned by extracted scene features. Predicted RIRs can be convolved with audio to obtain auralized and spatialized audio matching the scene.
Refer to caption: extracted/5907188/figures/overview_iclr_flat.png

## 3 Method

NeRAF generates both RGB images and binaural RIRs for any camera and microphone-source position and orientation. It achieves this by jointly learning neural radiance and acoustic fields. Although trained together, the audio and visual components can be used independently.

The NeRAF pipeline, shown in Figure 2, consists of three modules: NeRF, the grid sampler, and NAcF. The NeRF learns the radiance field and generates RGB images. The grid sampler queries NeRF’s radiance field with each voxel’s center, filling the 3D grid with color and density. NAcF then renders binaural RIRs, conditioned by the implicit appearance and geometric 3D priors derived from the features extracted from this 3D grid.

### 3.1 Modeling Acoustic through RIRs

To model the acoustic of an environment, our method learns RIRs. They describe how sound propagates between one emitter and listener depending of the scene’s geometry and material properties. For example, early reflections provide information about the distance to nearby surfaces, while late reverberation reflects the overall size and structure of the scene.
Learning acoustics through RIRs offers several key benefits: (1) RIRs encapsulate the acoustic behavior between specific emitter and listener positions; (2) By convolving audio signals with RIRs, we can synthesize sound that mimics how it would be heard within the scene; (3) Binaural RIRs include head-related transfer functions (HRTFs), which are essential for accurately spatializing audio.

By learning to generate RIRs, our method captures essential sound propagation phenomena, enabling realistic audio synthesis with rich acoustic details.

### 3.2 Neural Radiance Field

Neural radiance field (NeRF) (Mildenhall et al., ) renders photo-realistic images from new view points. It represents a static scene as a 5D vector-valued continuous function whose input is a 3D location $\textbf{X}=(x,y,z)$ and 2D viewing direction $\textbf{d}=(\theta,\phi)$, and output is color $\textbf{c}=(r,g,b)$ and volume density $\sigma$:

$$ $\text{NeRF}:(\textbf{X},\textbf{d})\rightarrow(\textbf{c},\sigma).$ (1) $$

NeRF shoots rays $\textbf{r}(t)=\textbf{o}+t\textbf{d}$ through image pixels from camera origin o between the near $t_{near}$ and far $t_{far}$ boundaries of the scene. 3D points $\textbf{X}_{n}=\textbf{o}+t_{n}\textbf{d}$ are sampled along the ray and, along with d used as input of two MLPs.
The first one maps $\textbf{X}_{n}$ to the view-independent density $\sigma_{n}$ and a corresponding feature vector. The second MLP takes the feature vector and d to produce view-dependent color $\textbf{c}_{n}$.
The final color $C(\textbf{r})$ of an image pixel are rendered using volume rendering equations:

$$ $\displaystyle C(\textbf{r})$ $\displaystyle=\int_{t_{near}}^{t_{far}}\mathcal{T}(t_{n})\sigma_{n}\textbf{c}_ {n}dt_{n},$ (2) $$

where $\mathcal{T}(t_{n})$ is the transmittance expressed as
$\exp(-\int_{t_{near}}^{t_{n}}\sigma_{k}dk)$.

NeRAF builds on Nerfacto (Tancik et al., ) as its NeRF backbone. Nerfacto integrates advancements from several works following (Mildenhall et al., ) including camera pose refinement (Wang et al., ; Lin et al., ), per image appearance conditioning (Martin-Brualla et al., ), proposal sampling (Müller et al., ), scene contraction and hash encoding. We selected Nerfacto for its robust performance in handling real-world static scenes. However, our method is NeRF-agnostic as it relies directly on the radiance field without altering the underlying NeRF model.

### 3.3 Grid Sampler

Figure: Figure 3: Grid sampler. We represent the scene as a grid of voxels. The grid is populated by querying the radiance field with the center coordinates of each voxel, $\textbf{X}_{vi}$, along with multiple viewing directions, $\textbf{d}_{1\rightarrow N}$. We average the color values obtained from these directions. It results in a 7-channels 3D grid containing color $\mathbf{\widehat{C}}_{vi}$, density $\sigma_{vi}$ and the 3D coordinates of the voxel centers.
Refer to caption: extracted/5907188/figures/grid_sampler_cor.png

Sound propagation is omnidirectional and influenced by the 3D geometry of the environment. Therefore, we provide our neural acoustic field with 3D geometric and appearance priors about the scene.

To this end, we use our grid sampler, illustrated in Figure 3, to construct a 3D volume from the NeRF model.
The scene is represented as a 3D grid of $S^{3}$ voxels with coordinates between 0 and 1 in accordance with NeRF’s scene contraction.
We shows in Section 4.3, that $S=128$ leads to good performances and use it for all our experiments.

We populate the grid by sequentially selecting batches of voxel centers $\textbf{X}_{vi}$ as queries to the radiance field. By processing voxels in order, we ensure that the grid is progressively updated to reflect the improvements in the NeRF model. To avoid slowing down training, we do not update the entire grid at each training iteration and process in batches. This sampling method guarantees complete grid population.

As NeRF color is view-dependent, we follow (Chen & Lee, ) and form $N=18$ viewing rays per $\textbf{X}_{vi}$. The goal is to obtain an average color of the voxel.
The radiance field returns a density value $\sigma_{vi}$ and a color value $\textbf{c}_{vi_{j}}$ per $j\in[1,N]$ viewing directions:

$$ $\text{NeRF}:(\textbf{X}_{vi},\textbf{d}_{1\rightarrow N})\rightarrow(\textbf{c }_{vi_{1\rightarrow N},\sigma_{vi}}).$ (4) $$

For color, we apply the volume rendering equation to a ray containing a single point, $\textbf{X}_{vi}$, with each viewing direction and average the values across these directions to obtain $\textbf{^{C}}_{vi}$.
For density, we compute the alpha compositing value $\alpha=1-\exp(-\sigma_{vi}\delta)$ where $\delta$ is a chosen small value.
We populate the grid with these values and the 3D coordinates of the voxels, resulting in a 7-channel representation.

### 3.4 Neural Acoustic Field

The goal of our neural acoustic field (NAcF), presented in Figure 4, is to learn a continuous neural representation of the scene’s acoustics. To this end, NAcF maps microphone coordinates $\textbf{X}_{m}=(x_{m},y_{m},z_{m})$ and orientations $\textbf{d}_{m}=(\theta_{m},\phi_{m})$ along with source position $\textbf{X}_{s}=(x_{s},y_{s},z_{s})$ and orientation $\textbf{d}_{s}=(\theta_{s},\phi_{s})$ to the short-time Fourier transform (STFT) of a room impulse response:

$$ $\text{NAcF}:(\textbf{X}_{m},\textbf{d}_{m},\textbf{X}_{s},\textbf{d}_{s},t) \rightarrow\text{{RIR}}(f,t).$ (5) $$

We learn RIR through their STFT representation, resulting in $\text{STFT}\in\mathbb{R}^{C\times F\times T}$, where C, F and T denote the number of channels, frequency bins, and time bins, respectively. This is motivated by the fact that the time-frequency representation is smoother and more suitable to neural network prediction than the time domain (Luo et al., ). Thus, NAcF learns to infer log-magnitude STFTs.
To better capture perceptual effects such as reverberation, we query NAcF with STFT time bins $t$. Consequently, NAcF outputs the $F$ frequency bins corresponding to the time query.
We normalize $t$ between $[0,1]$ and encode it with positional encoding. Similar to NeRF (Mildenhall et al., ) pose, microphone and source coordinates as well as direction are also embedded to a higher dimensional space using respectively positional encoding and spherical harmonic encoding.

Given the voxel grid, we concurrently train a ResNet3D (He et al., ) to extract relevant features for sound propagation. They serve as implicit indicators of scene’s 3D geometry, which is central for acoustic modeling. Moreover, as shown in previous works (Chen et al., ; Singh et al., ), information about materials can be inferred by a ResNet architecture from the scene’s appearance and contribute to audio synthesis.
We perform multimodal fusion by concatenating the features vector from the grid to the encoded poses, directions and the time query.
Similar to NeRF, NAcF comprises two MLP blocks. The first block maps the input to a feature vector that embeds the input information and the acoustic scene. The second focuses on learning microphone specifics. For binaural microphones, this block learns the HRTF necessary for audio spatialization. It contains one MLP per microphone channel.

To render the complete STFT, we query all time bins $t\in[0,T]$.
Finally, we use Griffin-Lim (Griffin & Lim, ; Perraudin et al., ) algorithm to obtain RIR waveforms from magnitude STFTs.

Figure: Figure 4: Neural acoustic field. NAcF maps microphone-source poses and directions to either binaural or monaural RIRs, with the number of output heads adjusted accordingly. NAcF is conditioned on scene features extracted through a 3D ResNet. Both poses and time queries are transformed into a higher-dimensional space using positional encoding, while directions use spherical harmonic encoding. The output of NAcF is a vector containing $F$ frequency values for each time query, $t$.
Refer to caption: extracted/5907188/figures/NAcF_cor.png

### 3.5 Learning Objective

Cross-modal learning has proven beneficial in multiple works (Section 2). Thus, NeRAF trains jointly NeRF and NAcF. NAcF training is slightly delayed as it begins when the grid has been updated a few times. The feature extractor is jointly trained with the radiance and acoustic fields.

We do not modify the NeRF loss function $\mathcal{L}_{V}$. It usually consists in an MSE loss between ground-truth pixel color $C(\textbf{r})$ and the rendered one $\hat{C}(\textbf{r})$, and complementary losses, $\mathcal{L}_{\text{comp}}$, specific to each NeRF.

$$ $\mathcal{L}_{V}=\lVert\textbf{ ^{C} ({r}})-\textbf{C}(\textbf{r})\rVert_{2}^{ 2}+\mathcal{L}_{\text{comp}},$ (6) $$

Inspired by (Yamamoto et al., ; ), we use a combination of spectral loss $\mathcal{L}_{\text{SL}}$ (Défossez et al., ) and spectral convergence loss $\mathcal{L}_{\text{SC}}$ (Arık et al., ) as our audio loss $\mathcal{L}_{A}$:

$$ $\displaystyle\mathcal{L}_{A}=\lambda_{\text{SC}}\mathcal{L}_{\text{SC}}+ \lambda_{\text{SL}}\mathcal{L}_{\text{SL}},$ (7) $\displaystyle\mathcal{L}_{\text{SL}}=\lVert\log(\left|s\right|+\epsilon)-\log( \left|\hat{s}\right|+\epsilon)\rVert_{2}^{2},$ (8) $\displaystyle\mathcal{L}_{\text{SC}}=\frac{\lVert\left|s\right|-\left|\hat{s} \right|\rVert_{F}}{\lVert\left|s\right|\rVert_{F}},$ (9) $$

where $\hat{s}$ is the predicted STFT, $s$ the ground-truth STFT, $\lVert.\rVert_{F}$ denotes the Frobenius norm, $\lVert.\rVert_{2}$ the L2 norm and $\epsilon=10^{-3}$.
The values of our loss hyperparameters are found empirically and given in Appendix C.
The spectral convergence loss emphasizes on spectral peaks helping especially in early phases of training while the spectral loss focus on small amplitude components which tends to be more important towards the later phases of training.
Our loss differs from (Yamamoto et al., ; ) as we found out that the MSE spectral loss leads to a better trade-off between metrics than the L1 spectral loss (Su et al., ) and the MSE loss on magnitude STFT (Luo et al., ; Liang et al., ). NeRAF results with those three losses are presented in Section 4.3.
The complete learning objective can be resumed as:

$$ $\mathcal{L}=\mathcal{L}_{V}+\lambda_{A}\mathcal{L}_{A}.$ (10) $$

## 4 Experiments

Figure: Figure 5: Loudness maps visualization. We visualize the intensity of predicted RIRs at different microphone positions for a given loudspeaker position and orientation. Intensities are averaged over multiple heights.
Refer to caption: extracted/5907188/figures/loudness_RAF_flat.png

First, we evaluate NeRAF’s performance against related works (Luo et al., ; Su et al., ; Liang et al., ; ) on SoundSpaces and RAF datasets. Then, we demonstrate that cross-modal learning of the acoustic and radiance fields is also beneficial to vision, resulting in enhanced results for large complex scenes trained with very sparse observations. Next, we conduct a few-shot experiment to assess the effectiveness of our method when trained with a limited number of audio recordings. Finally, we investigate the impact of the 3D grid and our loss combination on NeRAF’s performance.
Video examples showing NeRAF audio-visual generation on real and synthetic scenes are presented in the supplementary material.

### 4.1 Datasets

#### SoundSpaces.

SoundSpaces (Chen et al., ) is an audio simulator built upon Habitat Sim (Savva et al., ; Szot et al., ; Puig et al., ), a 3D simulator for embodied AI research. It provides binaural RIRs at discrete positions of a 2D grid with a spatial resolution of $0.5$ m with four head orientations accompanied by sound source position. This popular benchmark (Gao et al., ; Chen et al., ; Luo et al., ; Liang et al., ; Su et al., ; Purushwalkam et al., ; Parida et al., ; Chen et al., ) enables us to evaluate our method on diverse scene types while being under same settings as previous works.
Following (Luo et al., ; Liang et al., ; Su et al., ; Liang et al., ), we select six indoor scenes from Replica (Straub et al., ).
Two are small rooms (office 4 and room 2) with rectangular walls, two are medium room (frl apartment 2 and 4) with more complex layout and objects and two are complete apartments containing multiple rooms (apartment 1 and 2).

#### RAF.

Real Acoustic Fields (RAF) (Chen et al., ) is a real-world dataset that includes high-quality, densely captured room impulse response data paired with multi-view images and precise 6DoF pose tracking data for sound emitters and listeners in two rooms. The room impulse response are recorded using the ”earful tower”, which is equipped with omnidirectional microphones. This dataset was recently used to benchmark previous works on real-world scenes.

#### RWAVS.

RWAVS (Liang et al., ) is a real-world dataset used to benchmark AV-NeRF. It does not contain RIRs but audio recordings. Our results on the RWAVS dataset are presented in Appendix J.

**Table 1: Comparison with state-of-the-art. Performance on the SoundSpaces dataset using T60, C50, and EDT metrics (Left) and on RAF (Right) with the same metrics and STFT error. Lower score indicates higher RIR quality.**
| Methods<br>T60 (%) $\downarrow$<br>C50 (dB) $\downarrow$<br>EDT (sec) $\downarrow$<br>Opus-nearest<br>10.10<br>3.58<br>0.115<br>Opus-linear<br>8.64<br>3.13<br>0.097<br>AAC-nearest<br>9.35<br>1.67<br>0.059<br>AAC-linear<br>7.88<br>1.68<br>0.057<br>NAF<br>3.18<br>1.06<br>0.031<br>INRAS<br>3.14<br>0.60<br>0.019<br>AV-NeRF<br>2.47<br>0.57<br>0.016<br>NACF<br>2.36<br>0.50<br>0.014<br>NACF w/ T<br>2.17<br>0.49<br>0.014<br>NeRAF<br>2.14<br>0.38<br>0.010 | Methods<br>T60 (%) $\downarrow$<br>C50 (dB) $\downarrow$<br>EDT (sec) $\downarrow$<br>STFT error (dB) $\downarrow$<br>NAF<br>10.08<br>0.71<br>0.021<br>0.64<br>INRAS<br>8.01<br>0.79<br>0.025<br>0.36<br>AV-NeRF<br>8.11<br>0.73<br>0.021<br>0.39<br>NeRAF<br>7.47<br>0.61<br>0.020<br>0.17<br>NAF++<br>8.19<br>0.53<br>0.017<br>0.64<br>INRAS++<br>6.17<br>0.57<br>0.017<br>0.39<br>NACF<br>6.62<br>0.59<br>0.017<br>0.39<br>NACF w/ T<br>7.31<br>0.59<br>0.018<br>0.39<br>NeRAF++<br>6.87<br>0.51<br>0.015<br>0.17 |
| --- | --- |

Figure: Figure 6: Visualization of cross-modal learning impact on vision performances. We compare Nerfacto (Vision Only) with NeRAF (Audio + Vision) on apartment 1 and 2. Learning both radiance and acoustic fields improves the quality of novel views synthesis, reducing floating artifacts and enabling more detailed images.
Refer to caption: extracted/5907188/figures/QualiVision.png

### 4.2 Evaluation

#### Metrics.

Following (Su et al., ; Liang et al., ), we assess the quality of the predicted impulse responses using Reverberation Time (T60), Clarity (C50) and Early Decay Time (EDT).
T60 reflects the overall sound decay within a room by measuring the time an impulse response takes to decay by $60\,\text{dB}$. C50 measures the energy ratio between the first $50\,\text{ms}$ of RIR and the remaining portion. It relates to speech intelligibility, and the clarity of acoustics. EDT focuses on early reflections.
For RAF dataset, we also evaluate the STFT error which is the absolute error between the predicted and the ground-truth log-magnitude STFTs (Défossez et al., ; Luo et al., ; Chen et al., ).
To assess the effect of cross-modal learning on novel view synthesis, we employ commonly used metrics: PSNR, SSIM, and LPIPS.

#### Experimental Setup.

Similar to (Su et al., ; Luo et al., ; Liang et al., ), we use $90\%$ of SoundSpaces audio data for training and $10\%$ for testing. They are pre-generated binaural RIRs with corresponding source and microphone positions. We resample the RIR to $22.05$ kHz and compute STFT using $N_{\text{FFT}}=W=512$ and $H=128$ where $N_{\text{FFT}}$ is the number of FFT bins, W the Hann window size and H the hop length. To train the vision part, we generate RGB images using Habitat Sim. Camera poses and parameters were selected to limit the amount of visual data. For small, medium and large rooms, we use respectively $45$, $75$ and $150$ observations. Evaluation set comprises 50 images. More details on the motivations and the generation process are available in Appendix D.

For RAF, we follow previous works experimental setup: we keep $80\%$ of data for training and $20\%$ for evaluation. Audios are cut to $0.32$ s, resampled to $48$ kHz and processed into STFT using $N_{\text{FFT}}=1024$, $W=512$
and $H=256$.
For the vision part, we randomly select one third of the data from cameras 20 to 25. Nerfstudio automatically keep 90% of them for training and 10% for evaluation.

To ensure statistically significant results, we conduct each experiment 6 times and average results. More details on the implementation and architecture are presented in Appendix C, Appendix B.

Figure: Figure 7: Few-shot Learning. With 25% less audio recordings, NeRAF outperforms AV-NeRF trained on all the training data. Results are averaged on office 4, frl apartment 2 and apartment 2.
Refer to caption: extracted/5907188/figures/few_shot_newNeRAF.png

#### Results.

We compare NeRAF with state-of-the-arts methods on SoundSpaces in Table 1 (Left). Similar to previous works on neural acoustic field, we also compare our method to traditional audio encoding methods AAC and Opus (for Standardization, ; Foundation, ). AAC is a multichannel audio coding standard and Opus is an open audio codec.
NeRAF significantly outruns existing methods for C50 and EDT. It achieves $22.4\%$ improvement on C50 and $28.6\%$ on EDT compared to NACF with temporal alignment. Compared to AV-NeRF – which is the method most similar to ours as it also employ Nerfacto and does not rely on a large amount of information known a priori – we achieve respectively $13.4\%$, $33.3\%$ and $37.5\%$ improvement for T60, C50 and EDT.

As SoundSpaces is a simulated dataset that presents limitations in term of visual and acoustics realism, we also present results on RAF in Table 1 (Right). NeRAF outperforms previous works best results, achieving $-6.7$% T60, $-14.2$% C50, $-4.5$% EDT and $-52.5$% STFT error. The second part of the table contains a comparison of methods using an additional energy decay loss term (Majumder et al., ; Liang et al., ). While NACF initially relies on this term, other methods, with this new term, are denoted by ”++”. NeRAF++ still achieves state-of-the-art results, in particular for C50, EDT and STFT error. We present details about this loss in Appendix C.
We showcase qualitative results of audio synthesis in Figure 5 and in Appendix I.

**Table 2: Quantitative results on cross-modal learning. For highly complex scenes with very sparse training images, NeRAF audio-visual joint training improves vision performances, especially LPIPS. Evaluation on 50 test novel views. Delta are computed against Nerfacto (no audio).**
|  | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| # Images | 75 | 100 | 150 | 75 | 100 | 150 | 75 | 100 | 150 |
| Apartment 1 |  |  |  |  |  |  |  |  |  |
| Nerfacto | 22.48 | 24.77 | 27.00 | 0.838 | 0.882 | 0.902 | 0.246 | 0.161 | 0.118 |
| NeRAF | 23.33 | 25.07 | 27.70 | 0.858 | 0.884 | 0.915 | 0.216 | 0.140 | 0.105 |
| $\Delta\text{NeRAF}$ | +3.78% | +1.23% | +2.60% | +2.39% | +0.28% | +1.37% | -12.20% | -13.22% | -11.25% |
| Apartment 2 |  |  |  |  |  |  |  |  |  |
| Nerfacto | 19.10 | 22.67 | 25.62 | 0.777 | 0.836 | 0.884 | 0.381 | 0.232 | 0.160 |
| NeRAF | 20.53 | 23.46 | 26.06 | 0.802 | 0.856 | 0.895 | 0.322 | 0.206 | 0.150 |
| $\Delta\text{NeRAF}$ | +7.48% | +3.49% | +1.69% | +3.26% | +2.31% | +1.20% | -15.27% | -11.14% | -6.03% |

#### Cross-Modal Learning.

We next investigate the impact of jointly learning acoustic and radiance fields on vision in complex scenes with very sparse training views, a challenging scenario where NeRF typically struggles.
We train NeRAF and Nerfacto on the two large rooms. 75, 100, 150 images are used for training and the same 50 novel views are used for evaluation. In Table 2 we show that joint training of cross-modal learning improves novel view synthesis. It has a stronger impact on LPIPS which is related to the human-perceived similarity between two images. Overall, the performance improvement is more significant when the training images are very sparse. We present qualitative results in Figure 6: NeRAF reduces floating artifacts and enables more detailed images. Results on real scenes and additional visualizations are presented respectively in Appendix J and Appendix I.

#### Few-shot RIR Synthesis.

Previous works evaluated on SoundSpaces rely on a significant amount of RIR data, which is often unavailable in real-life scenarios. Therefore, we assess the performance of our model when trained with fewer data. We conduct a few-shot experiment on one small, one medium, and one large room and average the results. As shown in Figure 7, NeRAF is better than AV-NeRF on all metrics when trained with 75% of the data. It still outperforms AV-NeRF on EDT when trained with only 25% of the data.
On RAF dataset – which is a real-world dataset that already contains a more realistic amount of training data – NeRAF still outperforms the previous methods when trained with 25% less data. Results on RAF are presented in Appendix E.

### 4.3 Ablation Study

#### Grid Impact.

We assess the grid impact on performances in Table 3 (Left) by first removing the grid (NG) and then filling the 128-grid with zeros (128 w/ 0). In both cases, NAcF is no longer conditioned by the 3D priors from the scene. Removing the grid also means we reduce the number of trainable parameters as we no longer keep the features extractor. By setting the grid to zero we remove scene-related information and still keep the exact same architecture. This way, we show that the performance gap cannot be explained only by an increased number of parameters but rather to the information extracted from the 3D-grid.
We also study the impact of the resolution of the grid by comparing $S=128$ and $S=256$. Increasing the resolution to 256 improves T60 and slightly C50 but is accompanied by a minor EDT decline and increased training computation time.

**Table 3: Ablation studies. Left: Influence of the grid evaluated on RAF dataset. NG stands for no grid, 128 w/ 0 for a 128-grid filled with zeros, while 128 and 256 respectively correspond to grid of resolutions $S=128$ and $S=256$. Bold and underlined respectively indicates best and second best. Right: NeRAF’s loss study. Performances averaged across SoundSpaces dataset.**
| Grid<br>T60 (%) $\downarrow$<br>C50 (dB) $\downarrow$<br>EDT (sec) $\downarrow$<br>NG<br>8.15<br>0.674<br>0.0212<br>128 w/ 0<br>7.75<br>0.626<br>0.0205<br>128<br>7.47<br>0.609<br>0.0201<br>256<br>7.42<br>0.608<br>0.0204 | Loss<br>T60 (%) $\downarrow$<br>C50 (dB) $\downarrow$<br>EDT (sec) $\downarrow$<br>MSE<br>1.89<br>0.53<br>0.015<br>SC+SL${}_{\text{L1}}$<br>1.56<br>0.42<br>0.013<br>SC+SL${}_{\text{MSE}}$<br>2.14<br>0.38<br>0.010 |
| --- | --- |

#### Loss Impact.

We examine the impact of our custom loss described in Section 3.5 compared to the MSE loss used in previous works (Liang et al., ; Luo et al., ) and the combination with L1 spectral loss proposed by (Yamamoto et al., ) in Table 3 (Right). Our loss combination significantly improves C50 and EDT but decreases T60. Thus, this loss choice is a trade-off that can be adjusted to the final application. Note that NeRAF also achieves state-of-the-art performances with SC+SL${}_{\text{L1}}$combination.

## 5 Discussion

#### Limitation and Future Work.

A primary limitation of our method is its scene dependency, requiring NeRAF to be trained separately for each scene. Future work should explore advancements in generalizable NeRF to develop a unified, generalizable approach for synthesizing audio-visual scenes.
The current approach is also limited to static scenes. Addressing the challenge of learning implicit representations for audio-visual scenes with dynamic sound sources would be a significant advancement.

#### Conclusion.

We introduce NeRAF, a cross-modal method that learns both neural radiance and acoustic fields. By conditioning the acoustic field with scene features from a 3D grid containing geometric and appearance information, NeRAF enables realistic audio auralization, spatialization and novel view synthesis. Our results demonstrate improvements over other methods, enhanced novel view synthesis in complex scenes and increased data efficiency.

## Acknowledgments and Disclosure of Funding

We would like to thank Simon de Moreau for his help and insights.
This work was supported by the French Agence Nationale de la Recherche (ANR), under grant ANR22-CE94-0003 and was performed using HPC resources from GENCI–IDRIS (Grant 2024-103789).

## Appendix A Field parametrization

#### SoundSpaces.

SoundSpaces dataset lacks the complete acoustic field parameterization described in Equation 5. Microphone rotates only around the up-axis, $\theta$, and the source is omni-directional. Thus, NAcF can be expressed as $(\textbf{X}_{m},\textbf{d}_{m}=\theta,\textbf{X}_{s},t)\rightarrow(\text{RIR}_
{\text{ch}1},\text{RIR}_{\text{ch}2})(f,t)$.

#### RAF.

RAF dataset contains only monaural RIRs. Consequently, we replace the two heads presented in Figure 4 by a single head. The microphone is omni-directional but the source is directional. We can therefore express NAcF as $(\textbf{X}_{m},\textbf{d}_{s}=\theta,\textbf{X}_{s},t)\rightarrow\text{RIR}_{
\text{mono}}(f,t)$.

## Appendix B Architectures

#### NeRF.

NeRAF works with any NeRF method without modifications. It only requires a radiance field that the grid sampler can query using voxel center coordinates and viewing directions. Our method only relies on NeRF to fill a voxel-grid representation of the scene. In this paper, we choose Nerfacto from Nerfstudio (Tancik et al., ). For more details on its architecture we invite readers to refer to Nerfstudio website and paper.

#### Grid Sampler.

The grid sampler creates a 3D grid of voxels with a given resolution. Grid coordinates are comprised between $[0,1]^{3}$ fitting the scene contraction performed by NeRF. Consequently, knowing the exact dimensions of the room is not necessary. Voxels center coordinates and 18 viewing directions are queried to NeRF with no architecture modification. The grid is filled with color and density information along with voxels coordinates.

#### NAcF.

We train a ResNet3D-50 to embed the grid into features that best condition the acoustic field. For smaller scenes (office 4, room 2, apartment 2 and RAF dataset), we rely on 1,024 features. For larger scenes (frl apartment 2, 4, apartment 2) we use 2,048 features.
At inference, only these features, and not the complete grid, are required to perform RIR synthesis. Note that we employ average pooling at the end of the ResNet, adjusting according to the grid size to consistently yield the desired number of features.

NAcF consists of 2 MLP blocks. The first MLP block takes as input the multimodal fusion of grid features, encoded positions, and directions obtained through vector concatenation. It comprises 5 layers, each followed by a Leaky ReLU activation with a slope of 0.1, and outputs a 512 intermediate representation of the acoustic field. The second MLP block takes this intermediate representation as input and predicts the $F$ frequencies corresponding to the time query, aiming to learn the microphone specifics (e.g., HRTF for binaural microphones in SoundSpaces). It consists of one separate MLP head per microphone channel. The final activation layer is a tanh function scaled between $[-10,10]$ to fit the STFT log-magnitude range.

## Appendix C Implementation Details

#### Hyperparameters.

We implement our method using PyTorch framework (Paszke et al., ).
We optimize NAcF using Adam optimizer (Kingma & Ba, ) with $\beta_{1}=0.9$ and $\beta_{2}=0.999$ and $\epsilon=10^{-15}$. The initial learning rate is $10^{-4}$. It decreases exponentially to reach $10^{-8}$.
For NeRF, just as AV-NeRF we keep default Nerfacto parameters.

For the first 2k iterations, we only train the NeRF part. It allows the grid to be filled and updated several times using batches of $4,096$ voxel-centers. After, both NeRF and NAcF are train jointly. We use batch sizes of $4,096$ for NeRF and $2,048$ for NAcF. Note that NeRAF is trained by shuffling all STFT time slices in the train set.
In NeRAF++, we require the full STFT to apply the energy decay loss so we train NeRAF by querying all the time bins necessaries to form the STFT. In that case we use a batch size of 34 STFTs.
NeRAF is trained for 500k iterations but most runs reach their peak performance before, depending of the room size.
We train our method on a single RTX 4090 GPU.

#### Learning objective details.

For the learning objective defined in Section 3.5, we empirically select $\lambda_{A}=10^{-3}$, $\lambda_{SC}=10^{-1}$ and $\lambda_{SL}=1$.

To train NeRAF++, we add an additional energy decay loss term that reinforces the energy attenuation tendency of predicted
RIRs. Note that unlike INRAS++ and NACF, we do not use multi-resolution STFT and only apply the loss to the predicted STFT.
The audio learning objective becomes:

$$ $\mathcal{L}_{A}=\lambda_{SC}\mathcal{L}_{SC}+\lambda_{SL}\mathcal{L}_{SL}+ \lambda_{ED}\mathcal{L}_{ED}.\\$ (11) $$

with $\lambda_{SC}=3$, $\lambda_{SL}=1.5$ and $\lambda_{ED}=5$.

We follow Liang et al. () formula. Given a magnitude STFT $\mathbf{M}$, we first compute its energy in each time window by calculating the magnitude’s square and aggregating it along the frequency dimension:

$$ $\mathbf{M}^{\prime(d)}=\sum_{f=1}^{F}\left(\mathbf{M}^{(f,d)}\right)^{2},\ 1 \leq d\leq D\enspace,$ (12) $$

where $\mathbf{M}^{\prime}\in\mathbb{R}_{+}^{D\times 2}$ represents the energy in each time window.
We then sum $\mathbf{M}^{\prime}$ along the time dimension, aggregating the energy from the current step $d$ until the end $D$ for each time step $d\in[1,D]$ to capture the overall energy decay trend:

$$ $\mathbf{M}^{\prime\prime(d)}=\sum_{i=d}^{D}\mathbf{M}^{\prime(i)},\ 1\leq d \leq D\enspace,$ (13) $$

where $\mathbf{M}^{\prime\prime}\in\mathbb{R}_{+}^{D\times 2}$. Finally, we measure the L1 distance between the ground-truth energy trend and the predicted one in the log space:

$$ $\mathcal{L}_{\mathrm{ED}}=||\log_{10}\mathbf{M}^{\prime\prime}_{g}-\log_{10} \mathbf{M}^{\prime\prime}_{p}||_{1}\enspace.$ (14) $$

#### SoundSpaces.

To align with previous works on SoundSpaces, we resample all RIRs from 44,100 Hz to 22,050 Hz. We compute STFT with 512 FFT bins, a Hann window of size 512 and hop length of 128. We obtain log-magnitude STFT using $\log(|\text{STFT}|+10^{-3})$. Like Luo et al. (), we cut STFT to a maximum length depending of the scene. If the STFT is shorter than the maximum length we pad it with its minimal value.

#### RAF.

According to RAF (Chen et al., ), we cut audio to $0.32$ s and keep RIRs at 48 kHz. We compute STFT using 1,024 FFT bins, a Hann window of size 512 and hop length of 256.

#### Queries Encoding.

We give the 3D position of microphone and source to NAcF. They are normalized using minimum and maximum values of the poses in the training set. We add a 1 m margin. Then, they undergo multi-scale positional encoding with $N=10$ frequencies and maximum frequency exponent of $e=8$. We obtain direction vectors from the microphone rotation and apply spherical harmonic encoding with 4 levels.
The time queries are normalized within the range $[0,1]$ and go through positional encoding with $N=10$ and $e=8$.

#### Evaluation Metrics.

We inverse the transform of the predicted STFT using PyTorch Griffin-Lim algorithm running on GPU. We compute metrics against ground-truth waveforms.

To evaluate the quality of synthesized RIR waveforms $w$, we use T60, C50 and EDT errors obtained as follows:

$$ $\displaystyle\text{T60}(w,w_{\text{GT}})=\frac{|\text{T60}(w)-\text{T60}(w_{ \text{GT}})|}{\text{T60}(w_{\text{\text{GT}}})},$ (15) $\displaystyle\text{C50}(w,w_{\text{GT}})=|\text{C50}(w)-\text{C50}(w_{\text{GT }})|,$ (16) $\displaystyle\text{EDT}(w,w_{\text{GT}})=|\text{EDT}(w)-\text{EDT}(w_{\text{GT }})|.$ (17) $$

For RAF dataset, we follow previous works and also assess the quality of the time-frequency representation using the STFT error. To ensure fair comparison with methods directly predicting RIRs in the time domain, we perform the metric on the STFT obtained from the RIR in time domain. It means that we first go from the STFT predicted by NeRAF to a RIR waveform through Griffin-Lim algorithm and then compute its STFT before applying the metric. The STFT error is expressed as the absolute error between the predicted and the ground-truth log-magnitude STFTs:

$$ $\text{STFT error }(\mathbf{M},\mathbf{M}_{GT})=|\log(\mathbf{M})-\log(\mathbf{ M}_{GT})|$ (18) $$

To evaluate NeRF performances we choose the commonly used metrics such as PSNR, SSIM and LPIPS.

## Appendix D Scene Views

#### SoundSpaces.

SoundSpaces 1.0 provides a code to generate RGB and depth observations at microphone positions and orientations. They corresponds to $128\times 128$ resolution images with a $60^{\circ}$ field-of-view (FOV) sampled every 0.5 m with orientations $\theta\in[0^{\circ},90^{\circ},180^{\circ},270^{\circ}]$.
In NeRAF, we decided to not rely on them for NeRF training and instead generate our own observations using Habitat Sim.
This is motivated by real scenarios constraints. Views of the space may be obtained via a video recorded in the room. It is unlikely that someone will stop every 0.5 m and turn around to obtain images at each orientations.
Moreover, those kind of views are not optimal for NeRF training leading, in the worst case, to the failure of less sophisticated methods, and, in the best case, to the use of more data.
We also wanted to decorrelate microphone poses and camera poses for greater freedom in the usage of the method.

With Habitat Sim we rendered higher resolutions images with size $512\times 512$ and with a $90^{\circ}$ FOV. The larger FOV ensures more overlap between views. We randomly select positions at the edge of the room, orienting the camera to face its center with a random offset.
For small room, medium and large room we use respectively 45, 75 and 150 training images. 50 test poses are randomly sampled in the room. Figure A1 presents examples of the positions obtained.

Figure: Figure A1: Locations of visual observations for 3 SoundSpaces scenes. Blue dots corresponds to training views and green crosses to evaluation views. The exact poses coordinates are given in the supplementary material.
Refer to caption: extracted/5907188/figures/VposesLoc.png

#### RAF.

Real Acoustic Fields dataset captures high-quality and dense room impulse response data paired with multi-view images using VR-NeRF ”Eyeful Tower” camera rig (Xu et al., ). It contains 22 cameras at different heights and pointing at different directions. We chose cameras 20 to 25 for training NeRAF’s vision part. This choice ensures that we have sufficient overlapping views. The images have a 0.7 megapixels with resolution (i.e., 684 × 1024). For more details on those data we invite readers to refer to VR-NeRF paper.
To train NeRAF, as we want to limit the number of images necessaries, we randomly sub-select one third of those images. We let Nerfstudio select 90% of them for training and 10% for evaluation.

Figure: Figure A2: Evaluation of NeRAF on RAF with a decreasing number of training data.
Refer to caption: extracted/5907188/figures/few_shot_RAF.png

## Appendix E Few-shot on RAF.

RAF is a real-world dataset that contains a more realistic number of training data compared to SoundSpaces. We show in Figure A2 that NeRAF still outperforms INRAS, NAF and AV-NeRF with 25% less training data making it more data efficient.

## Appendix F Number of Parameters, Storage and Inference Speed

We provide in Table A1 the numbers of parameters of the NeRAF audio model and evaluate its inference storage requirements and speed. We present results when the 1,024 and 2,048 features are extracted from the grid. For RAF dataset, all results presented in the paper are obtained using 1,024 features while for SoundSpaces half the scenes use 1,024 (office 4, room 2, apartment 2) and the other half 2,048 (frl 2, frl 4, apartment 1).
The speed was averaged over the RAF dataset on a single RTX 4090. Note that at inference, the grid is not necessary and the extracted features are sufficient allowing the method to be faster and more compact. As our inference speed time is two orders of magnitude smaller than the length of the generated audio, NeRAF can be used for real-time applications.

**Table A1: Audio model number of parameters along with storage and speed requirements at inference.**
| Grid Features | Parameters (Million) | Storage (MB) | Speed (ms) |
| --- | --- | --- | --- |
| 1024 | 55.49 | 77.93 | 2.33 |
| 2048 | 71.89 | 97.84 | 2.64 |

## Appendix G Societal Impact

By synthesizing realistic audio-visual scenes, NeRAF can enable immersive experiences in VR and gaming. Understanding acoustics is also useful for applications such as sound de-reverberation, source localization, and agent navigation.
However, if misused, our method could contribute to the creation of misleading media, including the ability to mask and lie about someone’s location.

## Appendix H Grid Visualization

We display examples of sectional views of grids obtained using the grid sampler in Figure A3. They are then encoded into features using ResNet, which condition our neural acoustic fields.

Figure: Figure A3: Sectional views of $128^{3}$ grids obtained using the grid sampler.
Refer to caption: extracted/5907188/figures/GridViews.png

Figure: Figure A4: Examples of predicted log-magnitudes of binaural RIR STFTs compared to corresponding ground truth (GT). L and R respectively stand for left and right.
Refer to caption: extracted/5907188/figures/predSTFT.png

## Appendix I Additional Results Visualization

#### Audio qualitative results.

We provide examples of log-magnitude STFTs predicted with NeRAF in Figure A4.

#### Distance-aware spatialized audio.

We render binaural RIRs at various distances from the sound source and perform auralization by convolving an anechoic sound with these RIRs. This demonstrates one of the primary applications of NeRAF.
In Figure A5, we illustrate that NeRAF is distance-aware: as the microphone approaches the sound source, the amplitude increases, and it decreases as the microphone moves away. Additionally, when the microphone is positioned to the right or left of the source, the audio channels’ amplitudes reflect this spatial positioning.

#### Video.

We provide video examples of NeRAF generation in supplementary material. The chosen audios are open source and sourced from AVAD-VR (Thery & Katz, ).

#### Cross-modal learning.

We provide additional qualitative results on the effect of joint learning neural acoustic and radiance fields in Figure A6.

Figure: Figure A5: Examples of sounds generated with NeRAF on frl apartment 2. Our method successfully synthesizes spatialized and distance-aware sounds.
Refer to caption: extracted/5907188/figures/Approaching.png

Figure: Figure A6: Additional visualization of cross-modal learning benefits on the visual modality.
Refer to caption: extracted/5907188/figures/MoreQualiVision.png

## Appendix J RWAVS Dataset

Although we are aware AV-NeRF releases and evaluates its method on the RWAVS dataset, we chose to not present those results in the main paper. This is motivated by the difference between the tasks. NeRAF focus on learning RIRs, allowing the auralization of any given audio at inference. However, the RWAVS dataset does not contain RIRs but recordings of audio signals (e.g., music) played in a given environment. On this dataset, the network is therefore supposed to implicitly learn RIRs from them. To the best of our knowledge, it was never demonstrated that the representation learned can be used for the auralization of new audios.

Despite this, we present NeRAF performances on this dataset in Table A2. Similar to AV-NeRF, we adapt NeRAF to query frequencies instead of time and learn STFT masks. The same inverse modifications were done by AV-NeRF to evaluate on Sound Spaces (please see their supplementary material for more details). For every indoor scenes we extract 4,096 features from the grid and 1,024 for outdoors. This difference can be explained by the complexity of indoor scenes (e.g., many objects) compared to outdoors in this dataset. Because our loss combination is tailored for explicit RIR learning, we switch to MSE between magnitude STFTs.

Overall, NeRAF outperforms previous works tailored for RIR predictions and outperforms AV-NeRF in office, outdoors and slightly on apartment (ENV). We believe
NeRAF is less performant on House because of the unusually large number of small objects in the scene. It would need a very high-resolution grid to capture them.

RWAVS contains very complex environments on which we can show that NeRAF cross-modal training benefits vision. Results are presented in Table A3. This demonstrates that learning both acoustics and radiance fields is interesting on both real and simulated data.

**Table A2: Comparison with state-of-the-art methods on RWAVS dataset. Bold is best, underlined is second best.**
| Methods | Office | House | Apartment | Outdoors | Overall |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MAG | ENV | MAG | ENV | MAG | ENV | MAG | ENV | MAG | ENV |  |
| INRAS | 1.405 | 0.141 | 3.511 | 0.182 | 3.421 | 0.201 | 1.502 | 0.130 | 2.460 | 0.164 |
| NAF | 1.244 | 0.137 | 3.259 | 0.178 | 3.345 | 0.193 | 1.284 | 0.121 | 2.283 | 0.157 |
| ViGAS ^1 | 1.049 | 0.132 | 2.502 | 0.161 | 2.600 | 0.187 | 1.169 | 0.121 | 1.830 | 0.150 |
| AV-NeRF | 0.930 | 0.129 | 2.009 | 0.155 | 2.230 | 0.184 | 0.845 | 0.111 | 1.504 | 0.145 |
| NeRAF | 0.876 | 0.125 | 2.312 | 0.156 | 2.366 | 0.184 | 0.829 | 0.110 | 1.596 | 0.144 |

**Table A3: Quantitative results on RWAVS large complex scenes with sparse training data.**
|  | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Training Images | 1/6 | 1/4 | 1/6 | 1/4 | 1/6 | 1/4 |
| House |  |  |  |  |  |  |
| Nerfacto | 18.20 | 18.70 | 0.674 | 0.708 | 0.339 | 0.270 |
| NeRAF | 18.43 | 18.85 | 0.685 | 0.710 | 0.324 | 0.264 |
| $\Delta\text{NeRAF}$ | +1.26% | +0.81% | +1.65% | +0.25% | -4.31% | -2.14% |
| Apartment |  |  |  |  |  |  |
| Nerfacto | 19.14 | 19.64 | 0.746 | 0.776 | 0.251 | 0.211 |
| NeRAF | 19.24 | 19.68 | 0.748 | 0.778 | 0.249 | 0.210 |
| $\Delta\text{NeRAF}$ | +0.51% | +0.23% | +0.23% | +0.23% | -0.96% | -0.34% |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. See weakness section.

2. Experiment: since the framework is scene-specific where the train and test are based on the same room, how do the authors guarantee the train and test data are different? Another important discussion, if we fix a source position, what the predicted RIR accuracy difference on microphone position close to furniture and position on spacious area?
soundness: 2
strengths: 1. **Enhanced Realism**. NeRAF uniquely integrates visual and acoustic information, using visual scene features to inform the acoustic model and vice versa. This cross-modal approach improves the fidelity of both visual and acoustic outputs, leading to more realistic audio-visual experiences without the need for additional aligned datasets.

2. **Data EVersatility**: Unlike other methods requiring dense, aligned audio-visual data, NeRAF operates effectively with sparse datasets and does not rely on co-located audio-visual sensors for training. This flexibility makes it suitable for various real-world applications, especially where dense data collection is impractical.

3. **State-of-the-Art Performance**: This paper demonstrates measurable improvements over existing methods in key metrics like Clarity (C50), Reverberation Time (T60), and Early Decay Time (EDT), setting a new standard for realistic audio-visual scene synthesis.
summary: The paper presents NeRAF, a novel framework for synthesizing both vision and spatial audio in 3D scenes (spatial audio is the main focus of this paper). NeRAF’s key innovation is its ability to jointly learn visual neural radiance fields and spatial acoustic fields. The approach leverages cross-modal learning, meaning it incorporates visual features to enhance audio field prediction, and vice versa. It improves efficiency and realism in generating room impulse responses (RIRs). Unlike previous methods, NeRAF does not require aligned audio-visual data, making it more versatile and data-efficient. The method achieves state-of-the-art results in accuracy and quality of audio rendering, particularly in benchmarks like SoundSpaces and the Real Acoustic Fields (RAF) dataset.
weaknesses: 1. **Problem Setting Correctness**: In section 3.4, the authors write both the audio source and two microphones' (two ears) have directions. If so, it contradicts with the claim that: Sound propagation is omnidirectional (L205), because sound propagation from directional source isn't uniformally directional and thus may not be measured by RIR (the situation becomes more complex if the microphones are also directional). Moreover, the SoundSpaces 1.0 data provided RIR are emitted by point source without specific direction. The authors should 1. provide more detail discussion and theoretical proof showing how RIR they learned fits for directional source and microphones setting. 2. More detail regarding how directional information are used in SoundSpace 1.0 data or their own Habitat-sim simulated data (like how to set the direction and how to ensure the simulated RIR is conditioned on the set direction).

2. **Motivation Concern**: The task of this paper is to learn an acoustic field that can predict binaural RIR, but this paper proposes to learn auxiliary visual NeRF together with the acoustic field learning. One question naturally arises is that the authors should show why learning NeRF is essential for learning acoustic field, through either ablation study or theoretical analysis. If crossmodal visual learning is essential to improve acoustic field learning, is there any other more light weight visual learning mechanism other than NeRF can also improve the acoustic field learning. Given the title "3D scene infused neural radiance and acoustic fields" emphasizes the neural radiance field, more detailed discussion and investigation on the impact of visual radiance field on acoustic fields are required.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: Please see the weaknesses above.
soundness: 2
strengths: + The paper aims for a very interesting topic: joint learning of visual and acoustic rendering for 3D scenes. The authors proposed an effective pipeline to solve this challenging task, and explore how the two modalities can combine together.

+ Thorough experiments are conducted on both synthetic and real datasets, comparing with reliable baselines and plenty of ablations to show the effectiveness of the proposed modules. Especially for the grid sampling visualization, from which we can see what physical meaning the grids have learnt through the joint training.

+ The paper is well organized and easy to follow with clear presentation.
summary: The paper introduces a method that jointly learns acoustic and radiance fields named as NeRAF, which can synthesize both novel views and spatialized room impulse responses at arbitrary target viewpoints. They claim by conditioning the acoustic field on 3D radiance field, NeRAF can benefit both visual and acoustic modalities. Experiments on both synthetic and real datasets show NeRAF generates high-quality audio and improves performance on both novel view synthesis and acoustic modeling tasks.
weaknesses: - Weak cross-modality learning. The proposed cross-modality learning way for acoustic learning is to use ResNet3D to extract 3D geometric and appearance priors, and then inject it for acoustic rendering. However, I doubt it due to two aspects: 
    - The model is scene-dependent, which means fits one scene per model. In this case, it's hard to guide the ResNet3D to truly focus on properties that affect acoustic learning and physically correspond to geometry/materials etc., especially without any explicit supervision on it. The model can easily just take the grid sampling extractor as a large latent feature generator to help the model memorize the data distribution in the room.
    - From visualization of grids in Figure A3, we can see the grids feature is more correlated to visual rendering task, which makes sense since visual rendering has strong supervision on the grids directly (density and color etc.). However, this can be tricky if we want to understand how the two modalities are actually integrated. The grids seem to fail to correlate with acoustics. According to the first aspect, strong evidence is needed here to support the two modalities are actually helping each other in a way that has claimed physical meanings (instead of just more latent features that benefits memorizing the scene)

- One way that may help to really use the visual modality is to do multi-conditioning training, i.e. one model fits multiple scenes. In this case, the visual can be helpful to adapt to different rooms, and we may be able to observe how audio and visual modality integrate. With real visual representation, the model can do more than just fitting single room. 

- If the visual and audio modality are helping each other in a pure latent way without any physical meaning, the soundness of claimed motivations and contribution will be largely weaken.

- The grids are in the fixed shape, which might be hard to fit large room, where the resolution can be super coarse and may degrade the performance.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: - Why choosing magnitude spectrogram + griffin-lim instead of directly optimizing on waveform?
- Acoustic data in SoundSpace is actually in 2D (RIRs are the same for different heights). How does the 3D grid features help? Are 2D features good enough?
- In the demo video, are all videos shown on the right synthesized?
soundness: 2
strengths: - In general, I like the proposed multi-modal joint training approach that is appealing for its concise design and its potential to enhance the generated quality of both visual and acoustic fields. The flexibility to leverage single modalities when needed is also an advantage.
- The paper is generally straightforward to follow. Extensive studies have been conducted on both synthetic and real datasets, along with healthy supplementary materials, code, and a demo video.
summary: This paper introduces NeRAF, a new method for simultaneously learning acoustic fields and visual radiance fields. The authors demonstrate that this joint training approach enhances the quality of both generated room impulse responses and RGB images. Additionally, the proposed method offers flexibility, allowing for independent generation of single modalities during inference. Experiments conducted on both simulated (SoundSpace) and real-world (RAF) datasets show that NeRAF outperforms state-of-the-art methods in both audio and visual generation tasks.
weaknesses: - The task of rendering on both visual and acoustic signals has been previously explored (e.g., AV-Nerf), thus limiting the novelty of this work. From a technical perspective, the primary distinctions between NeRAF and AV-Nerf lie in 1) the use of 3D scene features for RIR generation and 2) the joint training pipeline. However, these technical contributions are not particularly significant to me.
- More analyses on cross-modal learning are needed. Table 2, Figure 6, appendix I and J show that joint training of cross-modal learning improves novel view synthesis but it is unclear whether there is any improvement in acoustic field synthesis.
- It is unclear why 3D ResNet is the best scene feature extractor. Some ablation studies would be helpful here.
- The loudness maps in Figure 5 exhibit some non-smooth textures. Why does it happen? Can NeRAF effectively represent a continuous acoustic field?
- NeRAF demonstrates better performance on RIR and image generation metrics, yet significant challenges persist. These include multi-scene representations, multiple or/and unknown sources, discrepancies between synthetic and real-world scenarios, and inverse problems. While this paper showcases improved training data efficiency, the impact is limited due to the density of synthetic datasets like SoundSpace. Even at 20%, the volume of training data remains substantial.
- From the perspective of human perception, it is unclear whether users can perceive actual difference between NeRAF and AV-NERF.
- NeRAF is obviously not a small model (Table A1). Especially it is still for single scene purpose.
- Some minor comments:
  - there was no context for Nerfstudio module until L90. Please clarify and add references in abstract and intro.
  - typo: wrong subscript $\sigma_{vi}$ in Eqn 4
  - In demo video, it would be better to provide trajectories in real scenes as well.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 2
questions: Please see above.
soundness: 3
strengths: - The paper tackles an interesting and novel problem, and the concept of jointly learning acoustic and radiance fields is well-motivated, as visual and acoustic information can complement each other to learn spatial cues.

- Although AV-NeRF also utilizes vision to aid in learning an acoustic field, the proposed approach uniquely learns a radiance field and introduces a grid-sampler to extract visual features, which the acoustic field is conditioned upon.

- Table 1 demonstrates that the proposed method achieves notable improvements compared to AV-NeRF, INRAS, etc.

- It's good to see that the proposed method is evalauted on a real-world dataset, apart from simulated data in SoundSpaces. The authors also faithfully evaluate on the RWAVS dataset and reported the results, despite the different settings of the tasks.
summary: This paper introduces a novel method that simultaneously learns acoustic and radiance fields for a recently introduced problem, novel acoustic view synthesis. The core idea involves integrating 3D scene features when rendering room impulse responses for the acoustic field. Experiments conducted on both a synthetic dataset using SoundSpaces and the recently introduced real-world dataset, RAF, demonstrate the effectivenss of the proposed method over previous methods.
weaknesses: - In supplementary video, it only shows the comparisons of the proposed method againt audio-only baseline, which is a weak baseline? It would be compare the qualitative differences between the proposed method and the closest performing prior work (AV-NeRF or NACF), to demonstrate the quantitaive improvement is perceptually meaningful.

- The main new component is the grid sampler component that extracts scene features to condition the RIR rendering. How beneficial is the joint training? What if separately training the visual component and a fixed grid sampler? How useful is the scene feature from grid sampler? What if a single image is provided as input and a ResNet 2D is used? How helpful is conditioning the task on 3D scene information?

- Following on the above questions, some additional ablations could be informative, such as 

1. Comparing joint training vs. separate training of visual and acoustic components

2. Evaluating the impact of using a fixed vs. trainable grid sampler

3. Comparing 3D scene features from the grid sampler vs. 2D features from a single image

4. Quantifying the benefit of 3D scene conditioning vs. no scene conditioning

Meta-review: This paper proposes a method (called NeRAF) jointly learning spatial acoustic and radiance fields. NeRAF synthesizes both novel optical views and spatial audio (room impulse responses, RIR) at new positions by conditioning the acoustic field on 3D scene geometric and appearance priors from the radiance field. 

This work received originally slightly contrasting reviews (5, 5, 5, 6) , and after rebuttal and discussion they reached a contrasting balance (5, 5, 6, 6). The paper was discussed by reviewers and authors and between reviewers (and AC).

Main strong points regard enhanced realism, versatility (no need to audio-visual alignment), state-of-the-art performance, and convincing experimental analysis, including ablations.

The raised weak aspects were not a few, and regarded issues in the problem setting correctness and motivations to be strengthened, especially showing how acoustic field benefits from the visual radiance field and vice versa, better justifying the claimed physical meaning of such integration. Also novelty has been questioned by a reviewer, as well as the request of additional results and comparisons. Further remarks concern the experimental analysis. 

Most of these comments were addressed by the authors in an extensive way, but a couple of reviewers were still not fully convinced, even if one is possibilistic for acceptance despite the rating “marginally below threshold”. The AC read all the material and deems that authors satisfactorily replied to the most relevant comments in a reasonable way, e.g., AC thinks that novelty is not an issue in this work, there are sufficient differences wrt AV-NeRF.

In conclusion, the paper is accepted for publication to ICLR 2025.

Decision: Accept (Poster)