AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesis
Authors: Susan Liang
Chao Huang
Yapeng Tian
Anurag Kumar
Chenliang Xu
Venue: NeurIPS 2023 poster (2023)
Keywords: Scene synthesis
audio-visual
NeRF

Abstract: Can machines recording an audio-visual scene produce realistic, matching audio-visual experiences at novel positions and novel view directions? We answer it by studying a new task---real-world audio-visual scene synthesis---and a first-of-its-kind NeRF-based approach for multimodal learning. Concretely, given a video recording of an audio-visual scene, the task is to synthesize new videos with spatial audios along arbitrary novel camera trajectories in that scene. We propose an acoustic-aware audio generation module that integrates prior knowledge of audio propagation into NeRF, in which we implicitly associate audio generation with the 3D geometry and material properties of a visual environment. Furthermore, we present a coordinate transformation module that expresses a view direction relative to the sound source, enabling the model to learn sound source-centric acoustic fields. To facilitate the study of this new task, we collect a high-quality Real-World Audio-Visual Scene (RWAVS) dataset. We demonstrate the advantages of our method on this real-world dataset and the simulation-based SoundSpaces dataset. Notably, we refer readers to view our demo videos for convincing comparisons.

AV-NeRF: Learning Neural Fields for Real- Audio-Visse Scene Synthesis
Susan Liang' Chao Huang
Yapeng Tian
Anurag Kumar"
Chenliang Xu-
University of Rochester Meta Reality Labs Research
Abstract
Can machines recording an audio-visual scene produce realistic, matching audio- visual experiences at novel positions and novel view directions? We answer it by studying a new task-real-world audio-visual scene synthesis-and a first-of-its- kind NeRF-based approach for multimodal learning. Concretely, given video recording of an audio-visual scene, the task is to synthesize new videos with spatial audios along arbitrary novel camera trajectories in that scene. We propose acoustic-awas aware audio generation module that integrates prior knowledge audio propagation into NeRF, in which implicitly associate audio generation with the 3D geometry and material properties of a visual environment. Furthermore, we present coordinate transformation module that expresses view direction relative to the sound source, enabling the model to learn sound source-centric acoustic fields. To facilitate the study of this new task, we collect high-quality Real- World Audio- Visial Scene (RWAVS) dataset. We demonstrate the advantages of our method on this real-world dataset and the simulation-based SoundSpaces dataset. We recommend that readers visit our project page for convincing comparisons: lttps://liangsusan git..ithub..... wtw.:iittt....///jttt/////aaaaaa
Introduction
We study a new task, real-world audio-visual scene synthesis, to generate target videos and audios along novel camera trajectories from source audio-visual recordings of known trajectories. By learning from real-world source videos with binaural audio, aim generate target video frames and spatial audios that exhibit consistency with the given camera trajectory visually and acoustically. This consistency ensures perceptual realism and immersion, enriching the overall user experience.
As far as we know, attempts in the audio-visual learning literature [[HI]) have yet to succeed in solving this challenging task thus far. Although there are similar works 12 these methods have constraints that limit their ability to solve this new task. Luo et al. 122 propose neural acoustic fields to model sound propagation in room. Su et al. [13] introduce representing audio scenes by disentangling the scene's s geometry features. These methods are tailored for estimating room impulse response signals in simulation environment that difficult to obtain in real-world scene. Concurrent to our work, ViGAS proposed by Chen et al. [15] learns to synthesize new sounds by inferring the audio-visual cues. However, ViGAS is limited to few viewpoints for audio generation.
We introduce AV-NeRF, a novel Nbse- method of synthesizing real-world audio-visual scenes. AV-NeRF enables the generation of vidcos and spatial audios, following arbitrary camera trajectories. utilizes source videos and camera poses as references. AV-NeRF consists of two branches: A-NeRF, which learns the acoustic fields of environment, and V-NeRF, which models color and density fields. We represent a static audio field as a continuous function using A-NeRF, which takes the listener's 's position and head direction as input. A-NeRF effectively models the energy decay of sound as the sound travels from the source to the listener by correlating the listener's 's position with the
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

NoRF
Ours
Render Video wi Rinnuna Audio
Input Images
Render Images
Input Videos
Figure I: NeRF learns render visual scenes from novel viewpoints. Beyond visual rendering, we introduce AV-NeRF, method for synthesizing audio-visual scenes with video frames and binaural audios along new camera trajectories. Integrating coherent sight and sound creates an immersive and realistic perceptual experience for users.
energy reduction effect. Moreover, A-NeRF accounts for the impact of the listener's orientation on sound perception by establishing correlation between the audio channel difference and the listener's direction. Consequently, A-NeRF facilitates distance-sensitive and direction-aware audio synthesis, resulting in the generation of realistic binaural sound. We utilize the vanilla NeRF [16] as our V-NeRF for vision rendering since the rendering of the purely visual world has been extensively and is not the primary focus of our work.
researched 17 16 18 20
To further enhance synthesis quality, introduce acoustic-awarr audio generation method and coordinate transformation mechanism. Considering that the 3D geometry and material properties of an environment determine sound propagation (21 22 23], we propose acoustic-aware audio generation module called AV-Mapper. AV-Mapper extracts geometry and material information from V-NeRF and feed them to A-NeRF, enabling A-NeRF to generate audios with acoustic awareness. In visual space, NeRF [66 expresses the viewing direction (0, Φ) using an absolute coordinate system, where the directions of two parallel light rays represented identically. However, in auditory space, this expression method is unsuitable as it ignores the spatial relationship between the sound emitter and receiver. Human perception of sound source relies on the relative direction of the sound source to the head orientation. Given this observation, we propose coordinate transform mechanism that expresses the viewing direction relative to the sound source. This approach encourages our model to learn sound source-centric acoustic fields.
Our work represents the initial step towards addressing the audio-visusa scene synthesis task in real-world settings. As such, there is currently no publicly available dataset that fulfills our specific requirements. To facilitate our study, we curated high-quality audio-visual dataset called RWAVS (Real-Wort) Audio- Visual Synthesis), which encompasses multimodal data, including camera poses, video frames, and realistic binaural (stereo) audios. In order to enhance the diversity of our dataset, we collected data from various environments, ranging from offices and apartments to houses and even outdoor spaces. We will release this dataset to the research community. Experiments on the RWAVS and the synthetic SoundSpaces datasets can validate the effectiveness of our proposed approach. It demonstrates AV-NeRF's capability of synthesizing novel-view audio-visual scenes in a range of environments, including both world and synthetic, as well as indoor and outdoor settings.
In summary, our contributions include: (I) proposing a novel approach synthesize audio-visual scenes in real-world environments at novel camera poses; (2) introducing a new acoustic-aware audio generation method that incorporates our prior knowledge of sound propagation; (3) suggesting coordinate transformation mechanism to effectively express sound direction; (4) curating a high-
quality audio-visual dataset for benchmarking this task; and (5) demonstrating the advantages of our method through quantitative and qualitative evaluations.
2 Related Work
Our work is closely related to several areas, including novel-view synthesis, vision-guided audio spatialization, dimulation-based acoustic synthesis, and learning-base based audio generation. We elaborate on each of these topics in the following section.
Neural Fields. Our method is based on neural fields, particularly Neural Radiance Fields (NeRF) [66 NeRF employs MIPs to learn an implicit and continuous representation of visual scenes, enabling the synthesis of novel views. Building upon NeRF's foundation, several works have extended its applicability to broader domains, including in-the-wild images [4], (silent) video 254 audio

---

and audio-visual [14] content. Luo et al. 12 propose neural acoustic fields (NAF) to capture the sound propagation in an environment, and Su al. [ introduce disentangling a scene's geometry features for audio scene representation (INRAS). Despite the compelling results achieved by NAF and INRAS, their application in the real world is hindered by their reliance on ground-truth impulse response signals that are simulated in synthetic environments. Furthermore, the discretization of position and direction in these approaches restricts the range of camera poses they can handle, whereas our model can handle continuous position and direction queries. Du et al. [4) proposed manifold learning method that maps vectors from latent space to audio and image spaces. Although the learned manifold allows for audio-visue interpolation, the model lacks support for controllable audio- vistull generation. In contrast, our method focuses on learning audio-visual representations that explicitly conditioned on spatial coordinates, enabling controllable generation.
12 [31,
Visually Informed Spatial Audio Generation. Considering that images often reveal the position of a sound source and/or the structure of scene, many visually informed audio spatialization approaches have been proposed. Among them, Gao and Grauman 7 focus on normal field-of- view videos and binaural audios. Zhou et al. (28) propose a unified framework to solve the sound separation and stereo sound generation at the same time, Xu et al. [29] propose a Pseudo2Binaural pipeline that augments audio data using HRIR function [2]] Most recently, Chen et al. [5 proposed the Visually-Guien Acoustic Synthesis ( (ViGAS) method for generating novel-view view audios. However, AV-NeRF differs from this method in two ways: (1) ViGAS relies ground-trun -tuthttthh images for audio transformation, whereas AV-NeRF addresses the absence of ground-trute truth images by utilizing rendered images from learned vision fields; (2) ViGAS only supports limited number of viewpoints for audio synthesis, while AV-NeRF has the capability to render novel videos at arbitrary camera poses.
7
12
15
Geometry and Material Based Acoustic Simulation. Several works A 33
focus on sim- ulating acoustic environments by modeling their 3D geometry and material properties. SoundSpaces, proposed by Chen et al. [4], is an audio platform that calculates the room impulse response signals for discrete listener and sound source positions. Extending this work, Chen et al. [3 introduce SoundSpaces 2.0, which enables acoustic generation for arbitrary microphone locations. Tang et al. [23] propose calibrating geometric acoustic ray-tracing with a diffeeeeffeeence time-domain wave solver to compute high-quality impulse responses. Additionally, Li et al. (22) propose method that blends the carly reverberation portion, modeled using geometric acoustic simulation and fre- quency modulation, with late reverberation tail extracted from recorded impulse responses. In contrast to these simulation methods, our approach implicitly leverages learned geometry and material information for audio-visual scene synthesis.
34 23
Learning Based Audio Generation. searning-based methods exploit the powerful modeling ability of neural networks to synthesize audios. Tang et al. (34) propose the use of neural networks to estimate
reverberation time (Teo) and equalization (EQ). Ratnarajah et al. [35] 36 37 employ generative adversarial networks (GANs) to supervise realistic room impulse response generation. Richard et al.
[38) introduce a binauralization network that models the sound propagation linearly and nonlinearly. Recent research has also investigated learning from noisy data [39] and few-shot data [40].
3 Task Definition
The real-world audio-visual scene synthesis task is to generate visual frames and corresponding binaural audios for arbitrary camera trajectories. Given a static environment E and multiple observa- tions O- = {O1, O2, ... } of this environment, where N is the number of training samples and each observation O, includes the camera pose p = (z. y, 2, 0, σ). the mono source audio clip a,, the recorded binaural audio a,, and the image I, this task aims to synthesize new binaural audio of and a novel view I' based on a camera pose query p* and a source audio ag. where p" is distinct from the camera poses in all observations. This process can be formatted as:
(ot, f(p' .0,,,,))))
where ff is a mapping function. Combining predicted at and I" of all poses in the queried camera trajectory, we can synthesize a realistic video with spatial audio. This task poses several challenges: (1) p² is novel viewpoint distinct from all observed viewpoints, (2) the synthesized binaural audio aj is expected to exhibit rich spatial effects contributing to its realism, (3) during inference, no observation is accessible and the mapping function f relies solely on the learned fields for scene synthesis. It should be noted that the position of the sound source is known in the environment E.
3

---

V-NeRF
depc
rgb
Direction (0,4)
Density
Position (,
Color c
A-V Mapper
Encoder
Supervision
Synthesized
Direction Position (x,
Mix mm
Audio
Diff ma
A-NeRF
Input Audio
R
The pipeline of our method. Given the position (ar, y; and viewing direction (0, Φ) of listener, our method can render an image the listener would see and the corresponding binaural audio the listener would hear. Our model consists of V-NeRF, A-NeRF, and AV-Mapper. A-NeRF learns generate acoustic masks, V-NeRF learns to generate visual frames, and AV-Mapper is optimized to integrate geometry and material information extracted from V-NeRF into A-NeRF.
Figure 2:
Method
Our method aims to learn neural fields that can synthesize real-world audio-visual scenes from novel poses. The entire pipeline is illustrated in Figure Our model comprises three trainable modules: V-NeRF, A-NeRF, and AV-Mapper. A-NeRF is responsible for generating acoustic masks, V-NeRF focuses generating visual frames, and AV-Mapper is optimized extract geometry and material information from V-NeRF, integrating this information into A-NeRF.
4.1 V-NeRF
NeRF [66 uses Multi-Layer Perceptron (MLP) to implicitly and continuously represent visual scene. effectively leams a mapping from camera poses to colors and densities:
NeRF (a,, y. 2, 2, (c.
where X X= y. is the 3D position, d (0, Φ) is the direction, c = (r., g, b) is the color, and or is the density. In order to achieve view-dependent color rendering and ensure multi-view consistency, [6 introduces two MLP components within NeRF. The first MLP maps a 3D coordinate (x, y, 2) to density and corresponding feature vector. The second MLP then takes the feature vector and 2D direction (0, Φ) as inputs, producing the color c. This process is illustrated in Fig. 3 NeRF then uses the volume rendering method (41) generate the color of any ray r()) through the visual scene, bounded by near L, and far ly:
/d traverses
C(r) C(r)= 1"' (()))r(())(r(()ddd))
where T(t) = exp(- Sr, (r(s)))ss) The depth of any ray r can be calculated similarly by replacing c(r(l), with
((r) = 1.' (()a(r()))td
(4)
These continuous integrals are estimated by quadrature in practice depth images can be rendered for any given camera pose using V-NeRF (Fig. 3). We apply positional
42 Subsequently, RGB and
encoding to all input coordinates to preserve the fighffrequency information of the generated images.

---

V-NeRF
ddpph
Density
rgb
Position (x2
Color
Direction (0.4)
Figure 3: The pipeline of V-NeRF. Given the 3D position (x, y. z) and direction (0, Φ) of a camera pose, V-NeRF can render corresponding RGB and depth images. V-NeRF consists of two MLPs with the first MLP estimating density and the second one predicting color c.
4.2 A-NeRF
The goal of A-NeRF is to learn a neural acoustic representation capable of mapping 5D coordinates (z,, y, 2, 0, Φ) to corresponding acoustic masks mm: my E R², where mm quantifies the change in audio magnitude with respect to the position (z,y, y, while m. characterizes the impact of the direction (0, Φ) on the channel difference of binaural audios, and F is the number of frequency bins:
NeRF (x,, V. 2, 0,
(mm: ma)
After training A-NeRF, we can synthesize binaural sound that contains rich spatial and acoustic information by fusing the magnitude of any input audio with predicted mm and m
In practice, we simplify the problem of audio synthesis by discarding the & axis and the o direction, which represents coordinates related to height, and instead focus the 2D space. On RWAVS and SoundS paces datasets, we observe that the sound received by the listener exhibits more distinct variations along the horizontal plane, as opposed the vertical axis. Fig. 4 depicts the pipeline of audio synthesis. Similar to V-NeRF, A-NeRF consists of two MLP components that parameterize different aspects of acoustic fields. The first MLP takes input the 2D position (xr,) y) and frequency query f E [0, F). producing mixture mask mm for frequency f and a feature vector. The mask mm quantifies the change in audio magnitude based on the distance between the sound source and the sound receiver. The feature vector serves as an implicit embedding of the input information and the acoustic scene. Then we concatenate the feature vector with the transformed direction 6' (explained in Sec. (4.4) and pass them to the second MLP. The second MLP associates the direction 6' with the channel difference of binaural audios. Given a fixed sound emitter, the difference between two channels of the received audio changes as the direction /' of the sound receiver varies. For instance, when the receiver faces the emitter, the energy of the two channels will be approximately equal. Conversely, when the emitter is on the left side of the receiver, the left channel will exhibit higher energy compared to the right channel. The second MLP is optimized to generate a difference mask m, that characterizes such direction influence. We query A-NeRF with all frequencies f within [0, F] to generate the complete masks mm, and m,-
After obtaining the mixture mask mm and the difference mask my. we can synthesize binaural audios by composing these masks and input audios (Fig. 4.. For an input source audio ag. we initially employ the short-time Fourier transform (STFT) to calculate the magnitude
E R Fx XW where W represents the number of time frames and F denotes the number of frequency bins. We then multiply 8, with mask m., to obtain the mixed magnitude Sm- Additionally, we multiply 8m and mask m. to predict the difference magnitude sa. Then we add mixture magnitude 8m and difference magnitude sa to compute the magnitude of left channel 8t, and subtract from 8m to compute the magnitude of right channel 8,. We further refine 8 and 8, using 2D convolution. Finally, we apply Inverse STFT to the predicted 81 and 8,, respectively, to synthesize the binaural audio a; Because A-NeRF operates on the magnitude, we use the phase information of the source audio for the Inverse STFT process. Instead of estimating two channels of the target audio directly. predict the mixture and the difference between the two channels of the target audio. Gao and Grauman 77] suggest that direct two-channel predictions can lead to shortcut learning in the spatialization network.
4.3 AV-Mapper
Given the fact that 3D geometry and material property determine sound propagation in an environment (prior knowledge), we propose an acoustic-aware audio generation method that integrates color and depth information estimated by V-NeRF with A-NeRF, When V-NeRF learns to represent visual scenes, it can learn the color e and density function of the environment, thanks to the multi-view
5

---

Input/Outpun flow
Color Depth
Input Audio
-- Add/Subtract
Audio-visual flow
Magnitude
Audio flow
A-V Mapper
my Mask
Frequency f
Cony2D> ISTFT
L
mmmm
St
Position (x,y)
Synthesized Audio de
A-NeRF
Sm
Sy)
0'
Sound Source
ma
Sd
Sy
Conv2D ISTFT
Direction
(x,y)
Direction Transformation
Figure 4: The pipeline of A-NeRF. Given the 2D position (I,,)) y) and direction 0 of a camera pose, A-NeRF can render mixture mask mm and difference mask ma. which then used for audio synthesis. A-NeRF is also composed of two MLPs, with the first predicting mixture mask m. and the second estimating difference mask my. The direction transformed relatively to the sound source prior to inputting into A-NeRF.
we can synthesize the RGB and depth image for any given camera pose. The RGB image captures the semantics and category of each object, implicitly indicating their material properties. And the depth image depicts the geometric structure of the environment. By providing A-NeRF with combination of RGB and depth images, we can offer rich environmental knowledge to A-NeRF.
consistency constraint. Utilizing volume rendering [6
Specifically, when synthesizing the binaural audio for given camera pose, our process begins by rendering a pair of RGB and depth images with the same camera pose using Eq. B and Eq. 4 respectively. We exploit pre-trained encoder (c.g., ResNet-18 18 [43] pre-trained on ImageNet dataset 44 to extract color and depth features from the RGB and depth images. The extracted features can serve as important indicators of the environment. Next, we develop an AV-Mapper, implemented as an MLP network, to project the color and depth features into a latent space. Our goal for the A-V Mapper is two-fold: (1) to extract meaningful embeddings while discarding unnecessary features, and (2) to align the features in the original space of the pre-trained image encoder with the space relevant to our specific problem. The output embedding of the AV-Mapper, denoted representing the width of each linear layer in A-NeRF, is then added to the input of A-NeRF for controlling the audio synthesis. Ablation studies (Sec. 5.2) demonstrate the slight advantage of this fusion method over others.
E Rs, with
4.4 Coordinate Transformation
Viewing direction (0, o) in V-NeRF is expressed in absolute coordinate system. This is natural practice in visual space such that directions of two parallel light rays are expressed identically. However, this expression method in audio space is less suitable because the human perception of the sound direction is based on the relative direction to the sound source instead of the absolute direction. To address this limitation, we propose expressing the viewing direction of camera pose relative to the sound source. This coordinate transformation encourages A-NeRF learning a sound source-centric acoustic field, enhancing spatial audio generation.
Given the 2D position (I, v) and the direction 0 of the camera, as well as the position of the sound source (S., Sy). we obtain two direction vectors: V1 (S, S, y) and V, (cos(O), sin si)). V1 represents the direction from the camera to the sound source, and 2 represents the camera's direction in the absolute coordinate system. By calculating the angle between V1 and V2. denoted as the relative direction = <(V1, I, V2), we obtain the rotation angle relative to the sound source. This angle 0' allows different camera poses to share the same direction encoding if they face the sound source at the same angle.
After computing the relative angle 8', we choose learnable embeddings instead of positional encoding 16 45) to project 0' into high-frequency space. We use the embedding E RAxe to represent four discrete directions, namely, 0?, 90", 180", and 270", where 4 is the number of directions and is the width of A-NeRF. Given relative angle 0' [0a, 360%), we linearly interpolate the direction embedding @' E R according to the angle between 0' and four discrete directions. We add the

---

(a) Office
(b) House
(c) Apartment
(d) Outdoors
Figure 5: Example scenes of RWAVS dataset. RWAVS dataset consists of diverse audio-vissi scenes indoors and outdoors. Each environment possesses distinct acoustics and structures.
interpolated embedding 0' to the input of each linear layer in A-NeRF, thereby providing A-NeRF with direction information. Ablation studies (Sec. 5.2) show that this encoding method performs best.
4.5 Learning Objective
The loss function of V-NeRF is the same as 16
Ly [[C(r) č(r)
(6)
where C(r) is the ground-truth color along the ray and ((r) is the color rendered V-NeRF. After training V-NeRF, we optimize A-NeRF and AV-Mapper together with the L2 loss function:
L = 1l sm 112 + ||s - s1112 + ||s, - 8,112
(7)
where 8m.l.r are the predicted magnitudes, Sm...r are the ground-truth magnitudes, and subscript m,l, mixture, left, and right, respectively. The first term of L encourages A-NeRF predict masks that represent spatial effects caused by distance, while the second and third term encourages A- A-NeRF generate masks that capture the differences between two channels.
5 Experiments
5.1
Datasets
To the best of our knowledge, method is the first NeRF-based system capable of synthesizing real- world videos with perceptually realistic binaural audios at arbitrary poses. However, existing datasets do not meet the specific requirements of experiments, particularly in terms of simultaneously providing camera poses, high-quality binaural audios, and images. Therefore, we curated a high- quality mudio-visual scene dataset (real) address this gap and facilitate further research on this problem. Additionally, we utilize (synthetic) SoundSpaces dataset [4] to validate our method.
Scene (RWAVS) dataset to bench- mark our method. In order to increase the diversity of our dataset, we recorded data across different scenarios. Fig. shows the example scenarios used for data recording. including both indoor and outdoor environments, which we believe represent most daily settings. RWAVS dataset comprises multimodal data, including camera poses, high-quality binaural audios, and videos. Unlike Replay- NVAS dataset 15], where the environment and the recording viewpoint are constant, RWAVS dataset contains various viewpoints in diverse environments. During data recording, randomly moved around the environment while holding the device, capturing various acoustic and visual signals. RWAVS dataset encompasses all positions and directions (360²) within an environment.
(1) RWAVS Dataset. We collected the Real-World Audio-Visusa
In detail, we employed 3Dio Free Space XLR binaural microphone for capturing high-quality stereo audio, a TASCAM DR-60DMEK for recording and storing audio, and GoPro Max for capturing accompanying videos. Additionally, an LG XBOOM 360 omnidirectional speaker was used as the sound source. For each environment and sound source combination, collected data ranging from 10 to 25 minutes, resulting in a total collection of 232 minutes (3.8 hours) of data from diverse environments with varying source positions.
extract key frames fps from recorded videos and use COLMAP to estimate the corre- sponding camera pose. Each key frame is accompanied by one-second binaural audio and one-second
7

---

Table 1: Comparison with state-of-the-aae methods RWAVS dataset.
Office
House
Apartment MAG
Outdoors MAG
Overall
Methods
MAG
ENV
MAG
ENV
ENV
ENV
MAG
ENV
Mono-Mono
9.269
0,411
11.889
0.424
15.120
0,474
13,957
0,470
12.559
0.445
Mono-Energy
1,536
0.142
4.307
0.180
3,911
0.192
1.634
0.127
2.847
0.160
Sterco-Energy
1.511
0.139
4.301
0.180
3.895
0.191
1.612
0.124
2.830
0.159
INRAS 13
1.405
0.141
3,511
0.182
3.421
0.201
1.502
0.130
2.460
0.164
NAF 12
1.244
0.137
3.259
0.178
3,345
0.193
1.284
0.121
2.283
0.157
ViGAS 15
1.049
0.132
2.502
0.161
2.600
0.187
1.169
0.121
1.830
0.150
Ours
0.930
0.129
2.009
0.155
2.230
0.184
0.845
0.111
1.504
0.145
Table 2: Ablation studies. We break down AV-NeRF to analyze the contribution of each component. Left: the inclusion of AV-Mapper (AV) and Coordinate Transformation (CT), Middle: different multimodal fusion methods, and Right: different direction encoding methods.
Overall
Overall
Overall
Methods
Methods
Methods
MAG
ENV
MAG
ENV
MAG
ENV
Baseline
2.287
0.157
Concat Input
1.507
0.145
Absolute Direction
1.701
0.149
Ours w/o AV
1.791
0.150
Add Input
1.504
0.145
Relative Direction
1.508
0.145
Ours w/o CT
1.701
0.149
Add Layers
1,505
0.145
Relative Embedking
1.504
0.145
Ours
1.504
0.145
source audio, forming a complete data sample. For audio clips with noticeable background noise, we perform noise suppression using Adobe Audition 47. We split 80% data as training samples and the rest as validation samples. After pre-processing, validation, respectively. This dataset is challenging because of the diverse environments and various camera poses. We will release this dataset to the research community.
obtain 9850 and 2469 samples for training and
(2) SoundSpaces Dataset. While RWAVS offers realistic training samples, its realism restricts its scale because it is time-consuming to record high-quality multimodal data in the real world. Therefore, we use the synthetic SoundSpaces dataset augment our experiments. To evaluate our method on SoundSpaces dataset, we modify AV-NeRF to estimate impulse responses instead of the acoustic mask while keeping all other components intact. We follow NAF [2] selecting six representative indoor scenes, consisting of two single rooms with rectangular walls, two single rooms with non-rectangular walls, and two multi-room layouts. each scene, SoundSpaces dataset provides an extensive collection of impulse response signals for sound source and sound receiver pairs, which are densely sampled from 2D room grid. Each pair includes four discrete head orientations (09, 90% 180%, and 270%), and each orientation is associated with two-channel binaural RIRs We render RGB and depth images for each sound receiver pose using Habitat-Simn simulator [48 49]. We maintain the same training/test split NAF, allocating 90% data for training and 10% data for testing.
5.2 Results on RWAVS Dataset
Comparison with State-of-the-art. We compare AV-NeRF with the following baselines: (1) Mono- Mono duplicates the source audio a, twice to generate fake binaural audio without modifying the source audio; (2) Mono-Energy assumes that the average energy of the target audio is known, scales the energy of the input audio match the target, and duplicates the scaled audio generate a stereo audio: (3) Stereo-Energy assumes that the energy of the two channels of the target audio a; is known, separately scales the energy of the input audio match the target, and combines the two scaled channels to generate a stereo audio: (4) IRNAS [13 learns representing audio scenes by disentangling scene's geometry features with implicit neural fields, and we adapt INRAS predict wave masks on RWAVS dataset; (5) NAF [12] designs local feature grids and an implicit decoder to capture the sound propagation in a physical scene, and we modify NAF to predict magnitude masks on RWAVS dataset; (6) ViGAS [1]] achieves novel-view acoustic synthesis by analyzing audio-visuaa cues from source viewpoints. We select magnitude distance (MAG) 29 which measures the audio quality in the time-frequency domain, and envelope distance (ENV) 30 which measures the audio quality the time domain, to evaluate various methods. Please refer to the supplementary material for implementation details.
8

---

Rendered Image
Rendered Audio
GT Audio
Rendered Image
Rendered Audio
GT Audio
Figure 6: Visualization of synthesized audio-visual scene. We present the rendered image, synthesized binaural audio, and ground audio.
AV-NeRF outperforms all baselines across different environments, including office, house, apartment, and outdoors, by significant margin (Table AV-NeRF outruns INRAS with 0.956 on the overall MAG metric (39% relative improvement) and 0.019 on the average ENV metric (1 (11.6%). AV-NeRF surpasses NAF with 0.779 on MAG metric (34%) and 0.012 on ENV metric (8%). Our approach is better than ViGAS in terms of both MAG (1,504 compared to ViGAS's 1,830) and ENV (0.145 compared ViGAS's 0.150).
Ablation Studies. We conduct ablation studies on AV-NeRF to analyze the contribution of different components. We combine A-NeRF and V-NeRF
baseline, which does not contain AV-Mapper
(Sec. 4.3 or coordinate transformation (Sec. 4.4). The ablation results are as follows: (1) AV-Mapper (AV) and coordinate transformation (CT) play important roles in learning audio scenes (Table Sabble2left left). The exclusion of either component degrades the generation quality; ; (2) adding visual information to the input of A-NeRF is the most effective multimodal fusion method compared with concatenation and adding visual information to all layers of A-NeRF (Table middle); (3) using embeddings to represent relative angles outperforms applying positional encoding to either absolute or relative angles (Table right). "Absolute Direction" represents applying positional encoding to the absolute angle, "Relative Direction' means transforming the relative angle with the positional encoding, and "Relative Embedding" is the embedding method.
Visualization. We visualize the synthesized audio-visual scenes in Fig. 6 intuitively assess the generation quality of our model. AV-NeRF can synthesize realistic binaural audios that have the same signal envelope and channel difference the ground-truth audios.
5.3 Results on SoundSpaces Dataset
Table 3: Comparison with state-of-the-arr We report the performance on the SonudSpaces dataset using T60, C50, and EDT metrics. The lower score indicates a higher RIR generation quality. Opus is an open audio codec [50), and AAC is multi-
We compare AV-NeRF with traditional audio coding methods [50 51] and advanced learning- based neural field methods 22 [3] using T60, C50, and EDT metrics [13]. Please refer to our supplementary material for implementation de- tails. Table that AV-NeRF outruns both traditional and advanced methods, achieving 21% % relative improvement T60 metric pared with the previous state-of-the-art method INRAS, 5% C50, and 16% EDT.
channel audio coding standard [5].
Methods
T60 (%) 1
C50 (dB)
EDT (sec)
Opus-nearess
10.10
3,58
0.115
Opus-linear
8.64
3.13
0.097
AAC-ncarest
9.35
1.67
0.059
AAC-linear
7.88
1.68
0.057
NAF [12]
3.18
1.05
0.031
INRAS [13
0.60
0.019
Ours
2.47
0.57
0.016
Discussion
In this work, propose first-of-its-kin NeRF system capable of synthesizing real-world audio- visual scenes. Our model can generate audios with rich spatial information at novel camera poses. We demonstrate the efffectivnness of our method real RWAVS and synthetic SoundSpaces datasets.
Limitation. Firstly, we currently focus on static scenes with single fixed sound source in our study. However, it is worth exploring the challenge of learning implicit neural representations for audio-visual scenes with multiple dynamic sound sources. Second, while our model successfully generates audio-visual scenes, it does not account for reverberation effects present in the real world. Reverberation is essential for humans to perceive scene size and structure. Third, similar to the original NeRF, AV-NeRF must represent and render each scene separately. Developing neural field

---

that can either learn to represent all scenes or transfer knowledge to new environments is a crucial problem, particularly for industrial applications.
Broader Impact. Although AV-NeRF is designed to synthesize audio-visual scenes, it is important acknowledge its potential application in generating scenes involving artificial humans. The misuse of AV-NeRF could lead to the production of deceptive and misleading media.
References
[1] Andrew Owens and Alexei A Efros. Audio-visual scene analysis with self-supervised multisen- sory features. In ECCV, pages 631-648, 2018.
[2] Shentong Mo and Pedro Morgado. Localizing visual sounds the easy way. Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, ECCV, volume 13697, pages 21 218-234, 2022.
[3] Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio-visual event localization unconstrained videos. In ECCV, pages 247-263, 2018.
[4] Changan Chen, Unnat Jain, Carl Schissler, Sebastia Vicene Amengual Gari, Ziad Al-Halah, Vamsi Krishna Ithapu, Philip Robinson, and Kristen Grauman. Soundspaces: Audio-visual navigation in 3d environments. In ECCV, 2020.
[5] Changan Chen, Ziad Al-Halah, and Kristen Grauman. Semantic audio-visual navigation. In CVPR, pages 15516-15525, 2021.
[6] Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. The sound of pixels. In ECCV, pages 570-586, 2018.
[7] Ruohan Gao and Kristen Grauman. 2.5d visual sound. In CVPR, 2019.
[8] Hang Zhou, Xudong Xu, Dahua Lin, Xiaogang Wang, and Ziwei Liu. Sep-stereo: Visually guided stereophonic audio generation by associating source separation. In ECCV, pages 52-69, 69. Springer, 2020.
[9] Yudong Guo, Keyu Chen, Sen Liang, Yong-Jin Liu, Hujun Bao, and Juyong Zhang. Ad-nerf: Audio driven neural radiance fields for talking head synthesis. In ICCV, pages 5784-5794, 2021.
[10] Yuanhang Zhang, Susan Liang, Shuang Yang, Xiao Liu, Zhongqin Wu, Shiguang Shan, and Xilin Chen. Unicon: Unified context network for robust active speaker detection. In ACM MM, pages 1964-3972 2021.
[11) Jinxing Zhou, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, and Yiran Zhong. Audio-visual segmentation. In ECCV, 2022.
[12] Andrew Luo, Yilun Du, Michael J Tarr, Joshua B Tenenbaum, Antonio Torralba, and Chuang Gan. Learning neural acoustic fields. NeurIPS, 2022.
[13] Kun Su, Mingfei Chen, and Eli Shlizerman. Inras: Implicit neural representation for audio scenes. In NeurIPS, 2022.
[14] Yilun Du, M. Katherine Collins, B. Joshua Tenenbaum, and Vincent Sitzmann. Learning aignal-agnostic manifolds of neural fields. In NeurIPS, 2021.
[15] Changan Chen, Alexander Richard, Roman Shapovalov, Vamsi Krishna Ithapu, Natalia Neverova, Kristen Grauman, and Andrea Vedaldi. Novel-view acoustic synthesis. arXiv preprint w.xXiv:2301087..0 2023.
[16] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. ECCV, 2020.
10

---

[17] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3d-structure-awee neural scene representations. NeurIPS, 32, 2019.
[18] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In CVPR, pages 3504-35155 2020.
[19] Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. NeurlPS, 01-2492-2502, 2020.
[20] Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi, Justin Kerr, Terrance Wang. Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, David McAllister, and Angjoo Kanazawa. Nerfstudio: A modular framework for neural radiance field development. arXiv preprint arXiv: 2023.
[21] Carl Schissler, Christian Loftin, and Dinesh Manocha. Acoustic classification and optimization for multi-modal rendering of real-world scenes. IEEE TVCG, 24()):244-12555 2017.
[22] Dingzeyu Li, Timothy R. Langlois, and Changxi Zheng. Scene-aware audio for 360 videos. ACM TOG, 37(4), 2018.
[23] Zhenyu Tang, Rohith Aralikatti, Anton Jeran Ratnarajah, and Dinesh Manocha. Gwa: A large high-quality acoustic dataset for audio processing. In SIGGRAPH, pages 1-9, 2022.
[24] Ricardo PratinnBruualla Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Doso- vitskiy, and Daniel Duckworth. NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections. In CVPR, 2021.
[25] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In CVPR, 2021.
[26] Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang. Dynamic view synthesis from dynamic monocular video. In ICCV, 2021.
[27] Tianye Li, Míra Slavcheva, Michael Zollhöfer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard A. Newcombe, and Zhaoyang Lv. Neural 3d video synthesis from multi-view video. In CVPR, pages 5511-5521, 2022.
[28] Hang Zhou, Xudong Xu, Dahua Lin, Xiaogang Wang, and Ziwei Liu. Sep-stereo: Visually guided stereophonic audio generation by associating source separation. In ECCV, 2020.
[29] Xudong Xu, Hang Zhou, Ziwei Liu, Bo Dai, Xiaogang Wang, and Dahua Lin. Visually informed binaural audio generation without binaural audios. In CVPR, 2021.
[30] Pedro Morgado, Nuno Vasconcelos, Timothy R. Langlois, and Oliver Wang. Self-supervised generation of spatial audio for 360 video. NeurIPS, pages 360-370, 2018
[31] Changan Chen, Ruohan Gao, Paul Calamia, and Kristen Grauman. Visual acoustic matching. In CVPR, 2022.
[32] V.R. Algazi, R.O. Duda, D.M. Thompson, and C. Avendano. The cipic hrtf database. In Proceedings of the 2001 IEEE Workshop on the Applications of Signal Processing to Audio and Acoustics (Cat. No.01TH8575), pages 902 2001. doi: 10. 0009//ASAAA.201..999999
[33] Changan Chen, Carl Schissler, Sanchit Garg. Philip Kobernik, Alexander Clegg, Paul Calamia, Dhruv Batra, Philip W Robinson, and Kristen Grauman. Soundspaces 2.0: A simulation platform for visual-acoustic learning. arXiv, 2022.
[34] Zhenyu Tang, Nicholas J Bryan, Dingzeyu Li, Timothy R Langlois, and Dinesh Manocha. Scene-aware audio rendering via deep acoustic analysis. IEEE TVCG, 2020.
[35] Anton Ratnarajah, Shi-Xiong Zhang, Meng Yu, Zhenyu Tang, Dinesh Manocha, and Dong Yu. Fast-rir; Fast neural diffuse room impulse response generator. In ICASSP, pages 571-575, 2022.
11

---

[36] Anton Ratnarajah, Zhenyu Tang, and Dinesh Manocha. Ir-gan: Room impulse response generator for far-field speech recognition. INTERSPEECH, pages 286-290, 2021.
[37] Anton Ratnarajah, Zhenyu Tang, Rohith Aralikatti, and Dinesh Manocha. Mesh2ir: Neural acoustic impulse response generator for complex 3d scenes. In ACM MM, pages 924 934-933, 2022.
[38] Alexander Richard, Dejan Markovic, Israel D Gebru, Steven Krenn, Gladstone Butler, Fernando de la Torre, and Yaser Sheikh. Neural synthesis of binaural speech from mono audio. In ICLR,
2021.
[39] Alexander Richard, Peter Dodds, and Vamsi Krishna Ithapu. Deep impulse responses: Estimat- ing and parameterizing filters with deep networks. In ICASSP, pages 3209-3213, 2022.
[40] Sagnik Majumder, Changan Chen, Ziad Al-Halah, and Kristen Grauman. Few-shot audio-visual learning of environment acoustics. In NewrIPS, 2022.
[41] James T Kajiya and Brian P Von Herzen. Ray tracing volume densities. SIGGRAPH, 18(3): 165-174, 1984.
[42] Nelson Max. Optical models for direct volume rendering. IEEE TVCG, 1(2):99-108, 1995.
[43] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[44] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248-255, 2009,
[45] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, pages 5998-6008, 2017.
[46] Johannes L Schonberger and Jan- MichMel Frahm. Structue-frommmmotionn revisited. In CVPR, pages
2016.
[47] Adobe Inc. Adobe audition. Software, 2023. URL audition.html html
[48] Andrew Szot, Alex Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimir Vondrus, Sameer Dharur, Franziska Meier, Wojeiech Galuba, Angel Chang, Zsolt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, and Dhruv Batra. Habitat 2.0: Training home assistants rearrange their habitat. In NeurIPS, 2021.
[49] Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A Platform for Embodied AI Research. In ICCV, 2019.
[50] Xiph.Org Foundation. Xiph opus. https:://ppus-codes org./ 2012.
[51] International Organization for Standardization. Advanced audio coding (aac). ISO/IEC 13818- 7:2006, 2006.
[52] Manik Varma and Andrew Zisserman. Classifying images of materials: Achieving viewpoint and illumination independence. In ECCV, pages 255-271, 2002.
[53] Zhoutong Zhang, Jiajun Wu, Qiujia Li, Zhengjia Huang, James Traer, Josh McDermott, Joshua B Tenenbaum, and William T Freeman. Generative modeling of audible shapes for object perception. In ICCV, pages 1251-1260, 2017.
[54] Changan Chen, Wei Sun, David Harwath, and Kristen Grauman. Learning audio-visual derever- beration. In ICASSP, pages 1-5, 2023.
[55] Nikhil Singh, Jeff Mentch, Jerry Ng, Matthew Beveridge, and Iddo Drori. Image2reverb: Cross-modal reverb impulse response synthesis. In ICCV, pages 286-295, 2021.
12

---

[56] Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. NeRF Neural radiance fields without known camera parameters. arXiv preprint arXiv:2102.07064, 2021.
[57] Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey, Barf: Bundle-adjusting neural radiance fields. In ICCV, 2021.
[58] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. ACM TOG, 41(4):1-15, 2022.
[59] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, perforreforeeece deep learning library. NeurIPS, 32, 2019.
[60] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. Yoshua Bengio and Yann LeCun, editors, ICLR, 2015.
[61] Julius Orion Smith. Mathematics of the discrete Fourier transform (DFT): with audio applica- tions. 2008
13

---

A Additional Visualization Results
We present additional results in Figure that demonstrate AV-NeRF's capability of generating distance-aware auditory effects. For each scene, we select fixed source audio and synthesize new binaural audio at varying positions and distances from the sound source. The first row displays the rendered images, while the second row showcases the corresponding rendered audio. As illustrated in the figure, the amplitude of the rendered audio diminishes as the distance between the source and camera increases, and conversely, it increases as the camera approaches the source.
Approach Sound Source
Leave Sound Source
Bendesed Image
Rendered Audio
Sound
Rendeved
Rendered Audio
Leave Sound
Rendered Image
Bendered
Figure 7: Distance-awar audio rendering. We show some example scenes where AV-NeRF can successfully synthesize consistent binaural audio with the camera movements.
B
Failure Cases
We include several failure cases in Figure where we present the rendered images, synthesized binaural audio, and ground- truth audio. AV-NeRF makes wrong acoustie predictions when the audio-visual scene involves noticeable noise or the AV-Mapper can not extract reliable material and geometry information from the visual space.
Because we work for real-world audio-visual learning, there inevitably exist various types noise in scene, e.g., sounds from refrigerators, air conditioning, wind, or even workers make noise when collecting data. These ambient noises are recorded by the microphone and are present in the ground-truth binaural audio, which hinders AV-NeRF from accurately learning the acoustic field. In Figure (a), the ground-trum truth audio (used for training) marked with red boxes contains noticeable noise, resulting in inaccurate audio rendering.
AV-NeRF relies on the AV-Mapper to extract reliable material and geometry information from input images, enabling comprehensive understanding of the audio-visual environment. However, in cases where the AV-Mapper fails extract meaningful information from certain images (e.g.. images of plain walls or whiteboards), AV-NeRF predicts erroneous acoustic masks. In Figure (b), the rendered audio marked with black boxes displays inconsistent waveforms compared the ground-truth audio, indicating model failure due to meaningless material and geometry information.
14

---

Rendered Image
Rendered Audio
GT Andio
Rendered Image
Rendered Audio
Autio
(a) Noisy environment.
Rendered Image
Rendered Audio
GT Audio
Rendered Image
Rendered Audio
GT Audio
(b) Meaningless visual input.
Figure Failure cases. We present failure cases which AV-NeRF makes wrong acoustic predictions. For each case, we show the rendered image, the rendered audio, and the recorded ground-truth audio.
C Rationality of AV-Mapper
In our paper, we leverage RGB and depth images as implicit indicators of environmental material properties and geometry. While these images enable the model to perceive the environment implicitly, is important to note that achieving precise one-to-one mapping from images to material properties geometry is not guaranteed for AV-Mapper. Certain corner cases, such black desk and object occlusion, can lead to the failure AV-Mapper.
Nevertheless, existing studies have demonstrated the feasibility of inferring material and geometry information from images. For instance, back in 2002, Varma and Zisserman [52 proposed a filter- based approach for material classification in images. More recently, in 2017. Zhang et al. ( successfully recognized material and shape attributes using visual data. Moreover, there body of audio-visual research indicating that extracted geometry and material information via ResNet networks can contribute to audio synthesis (54, (55. Our paper further supports this concept. As illustrated in Table (left), the inclusion of AV-Mapper in AV-NeRF can improve the MAG score by 16% from 1.791 to I.504 and enhance the ENV score by 3% from 0.150 to 0.145.
D Necessity of Distance and Direction Coordinates
Table 4: Necessity of distance and direction information.
Methods
Overall
Position Direction
MAG
ENV
1.822
0.156
1.817
0.155
1.688
0.148
1.504
0.145
To validate the necessity of distance and direction coordinates when modeling audio fields, we conduct additional ablation studies. We intentionally exclude positional and directional coordinates when representing an auditory scene and evaluate the performance without these inputs. The results are presented Table 4 AV-NeRF achieves MAG metric score of 1,504 and an ENV metric score of 0.5 when both position and direction information are utilized. Removing either positional or directional input leads to a performance drop between to 21% In the absence of both positional and directional inputs, performance degrades further to 1.822 on the MAG metric and 0.156
15

---

the ENV metric. These outcomes distinctly affirm the indispensable role of distance and direction coordinates in acoustic modeling.
E
Multiple Sound Sources
Given that scenes with single sound source may not fully represent real-world complexities, extend both the RWAVS dataset and the AV-NeRF model to support multi-source scenes.
Specifically, collect two multi-source scenes, adhering to the recording settings outlined in Section 5.T. The only modification we make is placing two sound sources instead of a single one in the environment, thus creating multiple sound sources. We extend AV-NeRF by stacking multiple equal A-NeRF modules to support the parameterization of multi multi-source acoustic fields. AV-NeRF generates acoustic masks for each sound source separately. We conduct experiments in these molti-source scenes and present the performance of AV-NeRF and other baselines in Table As presented in the table, AV-NeRF outperforms other baselines in both MAG (0.282) and ENV (0.063) metrics. The experimental results clearly demonstrate the proposed method's capability to effectively handle multiple sound sources.
Table Comparison with state-of-the-art methods on multi-source scenes.
Multi- source
Methods
MAG
ENV
Mono-Mono
1.949
0.172
Mono-Energy
0.533
0.075
Stereo-Fnergy
0.527
0.073
INRAS
0,472
0,078
NAF 12
0.401
0.080
Ours
0.282
0.063
F Architectures
A-NeRF. A-NeRF consists of two Multilayer Perceptrons (MLPs), each comprising four linear layers with additional residual connection. The width of each linear layer, denoted is set to 128 for the RWAVS dataset and 256 for the SoundSpaces dataset. In A-NeRF, all linear layers are followed by ReLU activation layers, except for the last layer, where the ReLU activation is replaced with the Sigmoid function. The first MLP takes the listener's position (z, y) and the frequency I E [0, F) as input, where F represents the number of frequency bins. It predicts mixture mask mm E R for the given frequency ff and generates feature vector with e channels. Prior to feeding them into the MLP, we apply positional encoding the listener's 's position (x, y) and the frequency We set the maximum frequency used for positional encoding as 10.
Then, we adopt relative transformation (Sec. 4.4) to project the listener's direction O into high- frequency space. We concatenate the transformed listener's direction and the feature vector, and feed it into the second MLP. The second MLP is appended with Sigmoid layer and scaling layer, ensuring that the difference mask ma estimated by the second MLP falls within the range of [-1, 1]. scalars.
For each frequency query J, A-NeRF estimates two masks: m. and ma, both of which We iterate over all frequencies f E [0, F] to obtain the complete masks m,, and m After computing the masks, we synthesize the target audio a, according to the procedure discussed in Sec. 4.2.
V-NeRF. We utilize the nerfacto model provided by nerf-studio 200 as the V-NeRF. This model combines several well-established and successful methods, including camera pose refinement [56] 57]. image appearance conditioning (24,, hash encoding, and proposal sampling [58]. Due to its robust and effective performance real-world data, utilize the default settings of the nerfacto model without making any architectural modifications. For more detailed information regarding the architecture of V-NeRF, please refer the documentation provided by nerf-studio.
AV-Mapper. For each camera pose, we render both RGB and depth images using V-NeRF. We resize images to 256 x 256 and center-crop to 224 224 prior to feeding them into frozen ResNet-18 43
16

---

image encoder pre-traine pre-trained on ImageNet- K dataset 44 512-dimension feature vector. We concatenate the RGB and the depth feature vectors, and input them into the AV-Mapper to learn environmental knowledge of the sound acousties. AV-Mapper projects the input feature vectors latent embedding of e channels. We implement the AV-Mapper as a 3- 3-layer MLP, with each intermediate linear layer followed by ReLU activation function.
ResNet-18 18 embeds the input image as
G Implementation Details
RWAVS Dataset. We implement our method using the PyTorch framework [59]. We employ Adam optimizer [60] with 31 = 0.9 and B2 = 0.999 for model optimization. The initial learning rate is set to 4 and exponentially decreased 5e-6. We train the model for 100 epochs with a batch size 32.
Before feeding the camera position (z, v) to A-NeRF, we normalize it within the range [-1, 1] x [-1, 1] and apply positional encoding [16]. Additionally, we resample all audios to a frequency of 22050 Hz and utilize the Short-Time Fourier Transform (STFT) to convert waveform audios into the time-frequene domain. For this transformation, we set the number of ffts as 512, the window length as 512, and the hop length 128. A Hanning window is applied during the process. Finally, compute both the magnitude and the phase from the spectrogram.
We use magnitude distance (MAG) [29] and envelope distance (ENV) 30 as evaluation metrics for audio quality. The MAG metric quantifies audio quality in the time-frequency domain and is defined as follows:
MAG (mprd, Mgt) = ||[updd mg:l
(8)
where Mprd is the predicted magnitude, and mgt is the ground-truth magnitude. ENV metric that measures the audio quality in the time domain is formatted as:
ENV(app agt) |hibeer((prr)) - hilbert(age))1
(9)
where aprd is the predicted audio, agt is the ground-truth audio, and hilbert is the Hilbert transfor- mation function [61]
Our model is trained on the SoundSpaces dataset using the same training settings as RWAVS dataset. We resample the impulse responses to 22050 Hz following INRAS [3 The 2D position is normalized [-1, 1] [-1,1] 1] prior positional encoding.
SoundSpaces Dataset.
We tailor A-NeRF for impulse response prediction with some minor modifications: (1) The input frequency query f is replaced by time query t E [0, 7), where T represents the length of an impulse response signal: (2) the first MLP only generates a feature vector while discarding the mixture mask mmi (3) the second MLP predicts impulse response signals instead of difference mask m..
Since the generated impulse responses are in the time domain, we employ STFT to convert them into the time-frequency domain and calculate their magnitudes. We utilize an STFT configuration with 512 FFTs, a sliding window width of 512, a hop stride of 128, and Hanming window. We supervise the model training using the L2 distance between the ground-truth magnitudes and the predicted magnitudes.
For performance evaluation, we choose three metrics: T60, C50, and EDT 13 the reverberation effects in an audio signal by measuring the time takes for the audio's energy to attenuate by 60 dB. The T60 distance is calculated as follows:
T60 characterizes
T60(aped: agt) = (T60()))a) - (T60(aa)))
(10)
T60(ag;)
where aprd and agt are the predicted and ground truth impulse responses, respectively. C50 quantifies the energy ratio between early reflections and late reverberation, allowing it to represent the clarity and loudness of the audio. We format the C50 distance as:
C50(aprd: agt) |C50(apra) C50(ag))
(11)
The EDT metric shares similarities with T60 but places greater emphasis on capturing the early reflections of impulse responses. The EDT distance is defined as follows:
(D agt) = [EDT((ppa) EDT((a)))
(12)
With these three metrics, we can evaluate the generation quality of impulse responses from different aspects, including clarity, energy, and reverberation.
17

---

H
Setup of RWAVS Dataset
Recording Devices. We have assembled a recording system, as depicted in Fig. g capture high- quality audio-visual scenes in real-world environments. Our system comprises a 3Doo Free Space XLR binaural microphone for capturing stereo audio, a TASCAM DR-60DMKII for recording and storing audio, and a GoPro Max for capturing accompanying videos.
This system is portable, allowing tion it flexibly and capture scenes from different camera poses. In addition, we utilized LGG XBOOM 360 omnidirectional speaker to serve sound source, which plays music repeatedly, Figure 5 in the main paper illustrates the setup used to record data in four distinct environments: office, house, apartment, and outdoors. Within each environment, we positioned the speaker at multiple locations to capture diverse acoustic effects. Each combination of environment and sound source represents an audio-visual scene. We collected data ranging from 10 to 25 min- utes for each scene, resulting a total collection of 232 minutes (3.8 hours) of diverse data, en- compassing various environments and source positions.
posi-
Figure Recording system. comprises profes- sional binaural microphone, sports camera, and recorder. Example Scenes. In Fig 10, we present four
example scenes from RWAVS dataset along with the corresponding camera pose distributions. The first column showcases images of the example scenes. The second column displays the camera poses used for video recording: the black dot represents the sound source, and each blue triangle represents camera pose. We normalize all camera poses to the range of [, and visualize them in an x-y plane from a top-down view. The third column contains 2D density heatmaps, which illustrate the distribution of camera poses in each unit area: each pixel represents unit area and its color shows the number camera poses in this area. As shown the figure, RWAVS dataset encompasses densely covered camera poses for each environment. We also analyze the distribution of camera directions (shown in the last column of Fig 10). We present the direction distribution in a polar coordinate system with the angle representing the viewing direction and the radius meaning the number of camera poses in this region. RWAVS dataset consists of various viewpoints that approximately cover 360 range of viewing directions. In summary, the RWAVS dataset comprises diverse environments with a wide range camera poses.
18

---

Sourd Soure
Soand
Scarce
Example Sccnes
Camera Poses
Position Distribution
Direction Distribution
Figure 10: Example scenes. We present several example scenes along with their corresponding camera pose distributions. We display the position density heatmap and the direction distribution map. RWAVS dataset is composed of diverse environments with various camera poses.
19


Reviews:
[{"review_id": "Tu4Y3FvbRJ", "reviewer": "Reviewer_b4XE", "summary": "The paper proposes an audio-visual (AV-) NeRF model to synthesize binaural audio masks at novel poses by using audio-visual samples from a video walkthrough of a 3D scene. The synthesized audio masks can be convolved with any arbitrary anechoid audio signal to retrieve the corresponding binaural audio at the novel poses. Towards that goal, the paper proposes a model that uses the learned density information from a visual (V-) NeRF model to provide geometric and color information at a novel pose. Given this visual information, an audio (A-) NeRF model predicts masks that can be further used to generate the synthesize the binaural audio for a monaural sound source. Additionally, the paper proposes an alternate parameterization to the direction component of the camera pose parameters that accounts for the omnidirectionality of most sound sources. The paper also collects a realworld dataset for the task. The paper evaluates on both simulated and realworld data, compares against multiple baselines and reports impressive results.\n\n\nPost author-reviewer discussion: I have read the rebuttal. It addresses my concerns. I will keep my score and recommend accepting the paper.", "questions": "I would like to request the authors to address the concerns I mentioned in 'Weaknesses'.", "limitations": "The paper already discusses its limitations and societal impact.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1, The paper proposes a new AV-NeRF model for binaural audio synthesis that leverages learned color and geometry information for novel poses from a V-NeRF model.\n\n2. The alternate parameterization for the camera pose parameters is also a useful contribution that captures the physics of how sound travels from a omnidirectional sound source.\n\n3. The paper also introduces a new realworld dataset for the task, which could be a valuable contribution for the community.\n\n4. The paper evaluates on both simulated and realworld data, compares against multiple baselines and reports impressive results.", "weaknesses": "1. The paper doesn't compare with recently published ViGAS [14] model. The paper argues that ViGAS needs GT images and is limited to a few viewpoints. However, the images rendered by a standalone V-Nerf can be used as input to ViGAS (since it's not a contribution of this paper anyway), and ViGAS can be trivially adapted to render spatial sounds at arbitrary points. \n\nMinor:\n1. L131: [40] generalizes to new scenes, unlike the model proposed in this paper (also pointed out by the authors in limitations). Since there are no experiments on novel scenes, this statement looks unsubstantiated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your constructive comments and encouraging remarks. We address specific comments below.\n\n### Weakness\n**W1: Compare with ViGAS.** Because ViGAS is a concurrent work to our paper, we did not compare AV-NeRF with ViGAS in our submitted paper. However, in response to the reviewer's suggestion, we conduct an evaluation of ViGAS on the RWAVS dataset for performance comparison. We utilize the official open-source code available on GitHub and make minimal adjustments to the framework, enabling support for arbitrary camera poses. The obtained results of ViGAS's performance in each scene, as well as the overall performance, are presented below. \n\n| Methods | Office (MAG) | Office (ENV) | House (MAG) | House (ENV) | Apartment (MAG)|Apartment (ENV)|Outdoors (MAG)|Outdoors (ENV) |Overall (MAG)|Overall (ENV)|\n|:-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|INRAS | 1.405 | 0.141 |  3.511 |  0.182 |  3.421 | 0.201 |  1.502 |  0.130 |  2.460 |  0.164 |\n|NAF | 1.244 | 0.137 | 3.259 | 0.178 | 3.345 | 0.193 | 1.284 | 0.121 | 2.283 | 0.157 |\n|ViGAS | 1.049 | 0.132 | 2.502 | 0.161 | 2.600 | 0.187 | 1.169 | 0.121 | 1.830 | 0.150 |\n|Ours | **0.930** | **0.129** | **2.009** | **0.155** | **2.230** | **0.184** | **0.845** | **0.111** | **1.504** | **0.145** |\n\nWe can see that ViGAS achieves pretty good performance, and it outperforms both INRAS and NAF, serving as a strong baseline.\nOur approach is better than ViGAS in terms of both MAG (1.504 compared to ViGAS's 1.830) and ENV (0.145 compared to ViGAS's 0.150). These results further underscore the effectiveness of our method in accurately capturing the underlying acoustic field. \n\n**W2: Task definition.** We agree with the reviewer's point regarding the statement in our task definition. [1] can generalize to new scenes. In our revision, we will modify the task statement to address this concern.\n\n[1] Majumder, Sagnik, et al. \"Few-shot audio-visual learning of environment acoustics.\" NeurIPS. 2022."}, {"review_id": "p8pjZizF8f", "reviewer": "Reviewer_GFnA", "summary": "This paper describes an intriguing task that focuses on the synthesis of new perspectives of real-world audiovisual scenes. The task is to synthesize a new video with spatial audio along an arbitrary novel camera track in an audiovisual scene given a video recording of that audiovisual scene. A data acquisition system is built and a Real-World Audio-Visual Scene (RWAVS) dataset is collected in this paper.\nFirst, the paper proposes an acoustically aware audio generation module that integrates prior knowledge of audio propagation into NeRF, thus relating audio generation to the 3D geometry of the visual environment. In addition, a coordinate transformation module that represents the direction of observation with respect to the sound source is proposed. This directional transformation aids the model in learning the source-centered sound field. This paper's superiority is demonstrated by qualitative and quantitative results.", "questions": "As above.", "limitations": "As above.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The synthesis of new views on audiovisual scenes in the real world is an interesting task that contributes to a better understanding of the world and the dataset presented can contribute to the development of the field.\n\n2. The paper is well written and each module is clearly presented and easy to understand.\n\n3. The proposed AV-Mapper and the coordinate transformation mechanism can effectively fuse visual geometric information with audio information and effectively express sound direction.\n", "weaknesses": "1. The vanilla NeRF [15] structure is used in this paper's V-Nerf module, which is a relatively slow training and sampling model. Existing acceleration models for Nerf can already significantly improve training and sampling speed, and a better model may be able to achieve higher accuracy in this paper.\n\n2. How does the proposed model handle sounds that are not within the field of view?\n\n3. There are numerous audio-visual datasets available, such as the AVSBench (https://github.com/OpenNLPLab/AVSBench). What is the proposed method's performance on this dataset?\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your helpful suggestions and encouraging comments. We address specific comments below.\n\n### Weakness\n**W1: NeRF acceleration.** We concur with the reviewer's observation that one limitation of the vanilla NeRF is its slow rendering speed. In our study, we actually employ acceleration techniques for NeRF to enhance both the training and rendering processes. Sorry for the misunderstanding! We would like to further clarify the implementation details here. Specifically, we incorporate the `tiny-cuda-nn` library, which facilitates rapid NeRF training and querying [1, 2]. Furthermore, we utilize `nerf-studio` [3] to enhance rendering quality, as detailed in L23-L28 of the supplementary material. `nerf-studio` offers support for camera pose refinement, image appearance conditioning, hash encoding, and proposal sampling.\n\n**W2: Out-of-view sound.** Audio-NeRF parameterizes the acoustic scene by learning a mapping from 5D coordinates $(x,y,z,\\theta,\\phi)$ to corresponding acoustic masks $\\mathbf{m}_m$ and $\\mathbf{m}_d$ (see Equation 5). The prediction of acoustic masks is solely based on the query pose $(x,y,z,\\theta,\\phi)$. The RGB and depth images input into AV-Mapper serve as implicit indicators of the environment's material properties and geometry. Therefore, the absence of sound sources in the field of view has no impact on mask prediction.\n\n**W3: Results on AVSBench dataset.** We appreciate the reviewer for proposing the AVSBench dataset as a valuable resource. We acknowledge the significance of this dataset within the audio-visual community and will cite it in our revised manuscript. However, we would like to note that the AVSBench dataset may not align with our specific data requirements. Our proposed AV-NeRF model necessitates access to ground-truth audio data from both the sound source and the sound receiver. Regrettably, the videos provided by the AVSBench dataset lack the essential ground-truth sound information pertaining to the sound source. Furthermore, the videos encompass dynamic scenes, posing challenges for accurate camera pose estimation. Due to these limitations, conducting experiments using the AVSBench dataset becomes impractical for our study.\n\n[1] Müller, Thomas. Tiny CUDA Neural Network Framework.\n\n[2] Müller, Thomas, et al. \"Instant neural graphics primitives with a multiresolution hash encoding.\" ACM Transactions on Graphics (ToG) 41.4 (2022): 1-15.\n\n[3] Tancik, Matthew, et al. \"Nerfstudio: A modular framework for neural radiance field development.\" ACM SIGGRAPH 2023 Conference Proceedings. 2023."}, {"review_id": "DjrF5hUSB1", "reviewer": "Reviewer_Kx4V", "summary": "Inspired by the task of novel view synthesis, this paper additionally considers the audio modality and thus proposes an interesting new task: real-world audio-visual scene synthesis. Specifically, the task is to synthesize new videos with corresponding spatial audio along arbitrary novel camera trajectories by learning from a video recording of an audio-visual scene. To solve this problem, the authors resort to AV-NeRF, an A-NeRF for spatial audio synthesis, and a V-NeRF for video synthesis, respectively. For V-NeRF, they directly use the Vallina NeRF model. For A-NeRF, they explore extracting 3D geometry and material properties from the corresponding visual cues of V-NeRF. Moreover, a coordinate transformation module is proposed for more accurate spatial sound reconstruction. Better performance on a self-collected real-world dataset RWAVS and a simulation-based dataset SoundSpaces demonstrates the effectiveness of the proposed method AV-NeRF.", "questions": "1. How can the network obtain the distance between the camera and the sound source only from their positions? As we all know, distance calculation includes square and square roots, which are hard to approximate with simple neural networks.\n2. The coordinate system in each scene. How to determine the origin?", "limitations": "1. The biggest limitation is the inappropriate modeling of AV-Mapper. The authors claim this module can obtain the material and geometry information from V-NeRF. However, from the perspective of this reviewer, such important information can't be inferred from the current design. Please refer to the weaknesses for details.\n2. The current evaluation protocols can't validate the effectiveness of some modules employed in the paper.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposes an interesting task: real-world audio-visual scene synthesis, which extends the task of novel view synthesis from only visual modality to multiple modalities.\n2. The A-NeRF architecture introduced in this paper is well-designed. It considers sound propagation and view directions, which is quite reasonable. Besides, the coordinate transformation is also shown effective through comprehensive experiments.\n3. Exhaustive experiments show the superior performance of AV-NeRF over several strong baseline methods.\n4. The whole paper is well-written and presented in a coherent manner.", "weaknesses": "1. The design of the most important module AV-Mapper is not reasonable. First, the authors just use a pretrained ResNet-18 network to extract RGB and depth features from RGB and depth images. I'm curious how the ResNet-18 network could obtain material information from RGB images since the semantics can't imply the material properties at all. For a black desk, I can't tell if it's made of wood or steel just from the appearance. Actually, the authors can do some testing experiments, such as conducting color jittering and seeing what happens, or cross-scene testing to see whether the ResNet-18 network can extract materials and geometry information. Second, depth images can't inform all the geometry information. For example, the occlusion, which significantly influences sound propagation, can't be reflected on depth maps.\n2. For spatial audio modeling, a useful metric that can really reflect the necessity of distance and view directions is required. Can the distance help to determine the energy of sounds on the receiver side? It's impossible to tell only from MAG and ENV metrics. The quantitative result w.r.t. **a new useful metric** should be provided in the paper. Similarly, I can't tell whether the view directions are useful for modeling the relative energy of left and right channels. I also wonder which one is more important for spatial audio modeling.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your thoughtful review and valuable feedback. We address specific questions below.\n\n### Weakness\n**W1: Rationality of AV-Mapper.** Great question! We leverage RGB and depth images as **implicit** indicators of environmental material properties and geometry. While these images enable the model to perceive the environment implicitly, it is important to note that achieving a precise one-to-one mapping from images to material properties or geometry is not guaranteed for AV-Mapper. Certain corner cases, such as a black desk and object occlusion as mentioned by the reviewer, can lead to the failure of AV-Mapper. \n\nNevertheless, existing studies have demonstrated the feasibility of inferring material and geometry information from images. For instance, back in 2002, Varma and Zisserman [1] proposed a filter-based approach for material classification in images. More recently, in 2017, Zhang et al. [2] successfully recognized material and shape attributes using visual data. Moreover, there is a body of audio-visual research indicating that extracted geometry and material information via ResNet networks can contribute to audio synthesis [3, 4]. Our paper further supports this concept. As illustrated in Table 2 (left) of our paper, the inclusion of AV-Mapper in AV-NeRF can improve the MAG score by 16\\% from 1.791 to 1.504 and enhance the ENV score by 3\\% from 0.150 to 0.145.\n\n**W2: Necessity of distance and direction.** Thanks for the suggestion! To further validate the necessity of distance and direction, we conducted additional ablation studies. We intentionally exclude positional and directional coordinates when representing an auditory scene and evaluate the performance without these inputs. The results are presented below. AV-NeRF achieves a MAG metric score of 1.504 and an ENV metric score of 0.145 when both position and direction information are utilized. Removing either positional or directional input leads to a performance drop between 2\\% to 21\\%. In the absence of both positional and directional inputs, performance degrades further to 1.822 on the MAG metric and 0.156 on the ENV metric. These outcomes distinctly affirm the indispensable role of distance and direction coordinates in acoustic modeling.\n\nFor metrics, we follow existing audio-visual learning work [5, 6, 7], utilizing MAG and ENV metrics to evaluate the quality of generated binaural audio. MAG and ENV metrics reflect the generation quality in time-frequency and time domain, respectively. We appreciate that the reviewer suggested developing a novel metric to quantify the influence of distance and direction on acoustic modeling and to establish their significance. It would be a very interesting idea to explore in the future.\n\n|Methods (Position) |Methods (Direction) |Overall (MAG) | Overall (ENV) |\n|:-:|:-:|:-:|:-:|\n| | | 1.822 | 0.156 |\n| | &check;| 1.817 | 0.155 |\n| &check;| | 1.688 | 0.148 |\n| &check;| &check;| **1.504** | **0.145**|\n\n### Question\n**Q1: Distance between the camera and the sound source.** Audio-NeRF parameterizes the acoustic scene by learning a mapping from 5D coordinates $(x,y,z,\\theta,\\phi)$ to corresponding acoustic masks $\\mathbf{m}_m, \\mathbf{m}_d$ (Equation 5). The prediction of these acoustic masks solely relies on the query pose $(x,y,z,\\theta,\\phi)$. Audio-NeRF is designed to capture and learn acoustic effects caused by the distance, such as energy attenuation. However, it is not obligated to directly model and approximate the specific distance between the camera and the sound source. If we have misunderstood the reviewer's question, we kindly request further clarification, and we are more than willing to address any questions you might have.\n\n**Q2: Origin of the coordinate system.** We describe below the process of establishing the coordinate system for our camera poses $P=\\{p_1, p_2, \\dots, p_N\\}$, where each pose is denoted as $p_i = (x_i, y_i, z_i, \\theta_i, \\phi_i)$, and $N$ represents the total number of camera poses. To ensure uniformity, we normalize the 3D coordinates $(x_i, y_i, z_i)$ to be within the range of $[-1, 1]\\times[-1, 1]\\times[-1, 1]$. This normalization is achieved by dividing each coordinate by the maximum value of all corresponding coordinates along the same axis. For instance, we normalize $x_i$ as $x_i^* = \\frac{x_i}{\\max_{k=1}^{N}|x_k|}$. This coordinate normalization practice is commonly used in training NeRF. Once all camera poses are normalized, we designate $(0, 0, 0)$ as the origin of our coordinate system.\n\n### Limitation\n**L1: Please refer to W1.**\n\n**L2: Please refer to W2.**\n\n[1] Varma, Manik, and Zisserman, Andrew. \"Classifying images of materials: Achieving viewpoint and illumination independence.\" ECCV. 2002.\n\n[2] Zhang, Zhoutong, et al. \"Generative modeling of audible shapes for object perception.\" ICCV. 2017.\n\n[3] Chen, Changan, et al. \"Learning audio-visual dereverberation.\" ICASSP. 2023.\n\n[4] Singh, Nikhil, et al. \"Image2reverb: Cross-modal reverb impulse response synthesis.\" ICCV. 2021.\n\n[5] Gao, Ruohan, and Grauman, Kristen. \"2.5 d visual sound.\" CVPR. 2019.\n\n[6] Xu, Xudong, et al. \"Visually informed binaural audio generation without binaural audios.\" CVPR. 2021.\n\n[7] Chen, Changan, et al. \"Novel-view acoustic synthesis.\" CVPR. 2023.\n\n"}, {"review_id": "kSCYSIrx72", "reviewer": "Reviewer_iWmP", "summary": "The paper introduces a novel task called real-world audio-visual scene synthesis to generate novel views and spatial audio for any camera trajectories. The system contains a visual NeRF to synthesize novel view images, an audio NeRF to generate acoustic masks, and an audio-visual mapper to enable the visual information to enhance the A-NeRF. The authors also collect a new real-world dataset to demonstrate the task's importance and for benchmarking. Experiments on synthetic and real datasets show the effectiveness of the proposed method.", "questions": "1. If I understand correctly, the source mono audio is fixed for each scene in the RWAVS dataset. Have you tried to use another source audio during the inference time, and does it sound as good as before?\n\n2. Following the previous question, I guess one of the main differences between the real world and simulated data is that in the simulation platform, we could obtain dense impulse responses in arbitrary positions, but in the real world, it is obviously easier to play source audio and collect data along a trajectory continuously. However, how could we effectively validate the correctness at the inference stage if we are using new source audio without spatial audio ground truth?\n\n3. Since the goal is to generate video frames and spatial audio, the current A-V mapper aims to extract useful information from visuals to help A-NeRF. Is it also possible to use it to extract useful information from the audio side to enhance the synthesized visual frame?", "limitations": "N/A", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The task of audio-visual scene synthesis in the real world is essential. To my best knowledge, it is the first work that aims to generate both video and spatial audio in a real environment.\n\n2. Multimodal data collection for the real scene is challenging and requires lots of effort. I could see the new real data would be very useful for future research.\n\n3. Each component in the model pipeline is well-motivated and makes sense.\n\n4. The paper is generally well-written and easy to follow.\n\n5. Extensive experiments and ablation studies are done on real and synthetic data to show the effectiveness of the proposed method.", "weaknesses": "1. As the authors mention in the limitation, the current approach mainly works for static scenes with a single fixed sound source. However, as the initial exploration, the current setup makes sense.\n\n2. It is often challenging to interpret and understand the quantitative results of generated audio. While the proposed method outperforms baselines across all quantitative metrics. If possible, I would suggest authors add human subjective evaluations in the future to see the perceptual difference.\n\n3. As a new benchmark, it would be great to show some failure cases for the current method to indicate potential improvement in the future.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for your constructive comments and greatly appreciate your strong positive acknowledgment of our work. Below, we address each specific question.\n\n### Weakness\n**W1: Single sound source.** In our paper, we initially focused on a simplified scenario with a single sound source, acknowledging that it may not fully represent real-world complexities. We appreciate the reviewer's understanding that \"as the initial exploration, the current setting makes sense.\" To address this limitation, we extend both the RWAVS dataset and the AV-NeRF model to support multi-source scenes. Specifically, we collect two multi-source scenes, adhering to the recording settings outlined in Section 5.1. The only modification we make is placing two sound sources instead of a single one in the environment, thus creating multiple sound sources. We extend AV-NeRF by stacking multiple equal A-NeRF modules to support the parameterization of multi-source acoustic fields. AV-NeRF generates acoustic masks for each sound source separately. We conduct experiments in these multi-source scenes and present the performance of AV-NeRF and other baselines in the table below.\n\n|Methods|Multi-source (MAG) | Multi-source (ENV)|\n|:-:|:-:|:-:|\n|Mono-Mono | 1.949 | 0.172 |\nMono-Energy | 0.533 | 0.075 |\nStereo-Energy | 0.527 | 0.073 |\nINRAS | 0.472 | 0.078 |\nNAF | 0.401 | 0.080 |\nOurs | **0.282** | **0.063** |\n\nAs presented in the table, AV-NeRF outperforms other baselines in both MAG (0.282) and ENV (0.063) metrics. The experimental results clearly demonstrate the proposed method's capability to effectively handle multiple sound sources. In our revision, we will augment the RWAVS dataset with multi-source scenes and include the corresponding results in the main paper.\n\n**W2: Human subjective evaluation.** We agree with the reviewer's valid concern regarding the interpretability and comprehension of the quantitative results for the generated audio. To address this, we plan to conduct a human subjective evaluation in the future to perceptually assess the quality of the generated content.\n\n**W3: Failure cases.** As suggested by the reviewer, we include failure cases in Figure 1 (please refer to the attached PDF file in the global response). In the figure, we present the rendered images, synthesized binaural audio, and ground-truth audio. AV-NeRF makes wrong acoustic predictions when the audio-visual scene involves noticeable noise or the AV-Mapper can not extract reliable material and geometry information from the visual space. \n\n1. Because we work for **real-world** audio-visual learning, there inevitably exist various types of noise in a scene, e.g., sounds from refrigerators, air conditioning, wind, or even workers make noise when collecting data. These ambient noises are recorded by the microphone and are present in the ground-truth binaural audio, which hinders AV-NeRF from accurately learning the acoustic field. In the Figure 1 (a), the ground-truth audio (used for training) marked with red boxes contains noticeable noise, resulting in inaccurate audio rendering. \n\n2. AV-NeRF relies on the AV-Mapper to extract reliable material and geometry information from input images, enabling a comprehensive understanding of the audio-visual environment. However, in cases where the AV-Mapper fails to extract meaningful information from certain images (e.g., images of plain walls or whiteboards), AV-NeRF predicts erroneous acoustic masks. In Figure 1 (b), the rendered audio marked with black boxes displays inconsistent waveforms compared to the ground-truth audio.\n\n### Question\n**Q1: Replace source audio.** For quantitative evaluation, we use a fixed source audio input for AV-NeRF, which guarantees that the source audio and recorded ground-truth audio share the same auditory content. However, during the general inference process, AV-NeRF can accept any source audio of interest, allowing us to generate binaural audio with rich acoustic properties. For instance, we collect an outdoor scene in the RWAVS dataset using a music song as the source audio, and train AV-NeRF with both the source music and the recorded music. After completing the training, we feed AV-NeRF with arbitrary source audio to synthesize new binaural audio. In our demo videos, we replaced the music with human speech and successfully synthesized coherent binaural sounds. The generated speech audio demonstrated consistency with the camera movements acoustically.\n\n**Q2: Unavailable ground-truth audio.** We can synthesize pseudo ground-truth audio to sidestep the unavailability of ground-truth audio when we use new source audio to evaluate the correctness of an approach. Although ground-truth audio is not directly available for these new samples, we have access to the original source audio clip $a_s$ and the binaural audio clip $a_t$ from the dataset. Leveraging a deep neural network [1], we can estimate the corresponding impulse response from $a_s$ and $a_t$. By convolving the estimated impulse response with the new source audio, we synthesize the pseudo ground-truth audio. This approach effectively provides us with ground-truth binaural audio for performance evaluation. \n\n**Q3: Enhance visual synthesis.** We agree with the reviewer's suggestion that extracting valuable auditory information can aid in visual learning. As shown in [2], acoustic information can enhance the quality of visual reconstruction. Moreover, [3] demonstrates that acoustic features extracted from sound echoes can assist in spatial perception tasks, such as monocular depth estimation and surface normal estimation. It would be very interesting to explore audio-assisted vision learning in the future.\n\nPlease refer to the global rebuttal for reference."}, {"review_id": "Nx29Jv3O7J", "reviewer": "Reviewer_4psZ", "summary": "The paper introduces a new task called \"real-world audio-visual scene synthesis\", which involves synthesizing new videos with spatial audios along arbitrary novel camera trajectories in a scene. The proposed NeRF-based approach integrates a prior knowledge of audio into audio generation and associates it with the 3D visual environment. The proposed model is able to learn a sound source-centric acoustic field with the introduction of the coordinate transformation module that expresses a view direction relative to the sound source. Additionally, a high-quality Real-World Audio-Visual Scene (RWAVS) dataset is demonstrated along with the model. ", "questions": "1.\tWhile the method is spatially-aware, the method seems to be less effective when the distance of the sound source is changing from the demonstration of the video, could you provide more results?\n\n2.\tThe sound source demonstrated in the video is the same, could the sound source be changed, or the sound-source device has certain requirements? If so, the proposed method can be limited.\n\n3.\tAs mentioned in the weaknesses, please clarify the novelty of the proposed method again.\n", "limitations": "As mentioned in the paper, this work currently focuses on static scenes with a single fixed sound source. To better suit the word “real-world”, the method needs to be improved and adapted to multiple sound-source situations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper presents an interesting new task that associates the audio with 3D geometry and material properties of scenes and the video demo demonstrates the effectiveness of the proposed method.\n2.\tThe paper compares the proposed method with similar methods on both the RWAVS dataset and the SoundSpaces dataset and surpasses the previous state-of-the-art method. \n3.\tThe method proposes a novel dataset that can be further utilized in similar tasks.\n", "weaknesses": "1.\tThe novelty of the proposed network seems to be limited. To tackle the new task, AV-NeRF combines an audio-NeRF and visual-NeRF and introduces an AV-mapper and the novelty mostly lies in the AV-mapper based on my current understanding.\n\n2.\tThe paper introduces a new task called \"real-world audio-visual scene synthesis”.  However, as mentioned in the paper, this work currently focuses on static scenes with a single fixed sound source, which is a bit far from the word “real-world” since in real-world, there can be much more sound sources.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your insightful comments. Below are our responses to specific questions.\n\n### Weakness\n**W1: Novelty of AV-NeRF.** In addition to AV-Mapper, our paper introduces two essential components -- the innovative Audio-NeRF architecture and a novel coordinate transformation mechanism. \n\nWe would like to clarify that it is non-trivial to extend NeRF to the audio domain. \nOur A-NeRF pipeline is carefully designed to capture the influence of the receiver's position and direction on sound perception. It consists of two separate Multi-layer Perceptrons (MLP) to learn the acoustic field, with the first MLP parameterizing the energy attenuation with regard to the distance between the sound source and receiver and the second one modeling the channel difference of stereo sound caused by the receiver's viewing direction. By disentangling the acoustic field with these two modules, A-NeRF enables distance-sensitive and direction-aware audio synthesis. \n\nAnother contribution of our work is the proposed coordinate transformation. By studying the nature of human's perception of the sound source -- human perception of the sound direction is based on the relative direction to the sound source instead of the absolute direction -- we propose replacing the absolute coordinate system, which is commonly used in NeRF, with a relative coordinate system. By expressing the viewing direction of a camera pose relative to the sound source, we encourage A-NeRF to learn a sound source-centric acoustic field and establish a strong correspondence between the viewing direction and the spatial audio.\n\n**W2: Multiple sound sources.** We agree that there are multiple sound sources in real-world scenarios, and the initial setting of this work was simplified. To address this limitation, we extend both the RWAVS dataset and the AV-NeRF model to support multi-source scenes. Specifically, we collect two multi-source scenes, adhering to the recording settings outlined in Section 5.1. The only modification we make is placing two sound sources instead of a single one in the environment, thus creating multiple sound sources. We extend AV-NeRF by stacking multiple equal A-NeRF modules to support the parameterization of multi-source acoustic fields. AV-NeRF generates acoustic masks for each sound source separately. We conduct experiments in these multi-source scenes and present the performance of AV-NeRF and other baselines in the table below.\n\n|Methods|Multi-source (MAG) | Multi-source (ENV)|\n|:-:|:-:|:-:|\n|Mono-Mono | 1.949 | 0.172 |\nMono-Energy | 0.533 | 0.075 |\nStereo-Energy | 0.527 | 0.073 |\nINRAS | 0.472 | 0.078 |\nNAF | 0.401 | 0.080 |\nOurs | **0.282** | **0.063** |\n\nAs presented in the table, AV-NeRF outperforms other baselines in both MAG (0.282) and ENV (0.063) metrics. The experimental results clearly demonstrate the proposed method's capability to effectively handle multiple sound sources. In our revision, we will augment the RWAVS dataset with multi-source scenes and include the corresponding results in the main paper.\n\n### Question\n**Q1: Distance effects.** In response to the reviewer's question, we present additional results in Figure 2 (please refer to\nthe attached PDF file in the global response) that demonstrate AV-NeRF's capability to generate distance-aware auditory effects. For each scene, we select a fixed source audio and synthesize new binaural audio at varying positions and distances from the sound source. The first row displays the rendered images, while the second row showcases the corresponding rendered audio. As illustrated in the figure, the amplitude of the rendered audio diminishes as the distance between the source and the camera increases, and conversely, it increases as the camera approaches the source.\n\n**Q2: Sound source.** There are no specific requirements for the sound source device. It can be any object that emits sound, including phones, laptops, instruments, or even humans. In our case, we utilize a speaker as the sound source to simplify the data recording process.\n\n**Q3: Please refer to W1.**\n\n### Limitation\n**L1: Please refer to W2.**\n"}]

Author rebuttal:
We would like to express our gratitude to all the reviewers for their valuable comments and feedback. We address the specific questions of each reviewer individually. Additionally, we attach a PDF file containing some figures in response to certain reviewers.

**To reviewer 4psZ:** we show some additional results in Figure 2 of the attached one-page PDF file.

**To reviewer iWmP:** we show failure cases in Figure 1 of the attached one-page PDF file. Due to the word limit of rebuttal, we place the reference paper here:

[1] Richard, Alexander, Peter Dodds, and Vamsi Krishna Ithapu. "Deep impulse responses: Estimating and parameterizing filters with deep networks." ICASSP. 2022.

[2] Luo, Andrew, et al. "Learning neural acoustic fields." NeurIPS. 2022.

[3] Gao, Ruohan, et al. "Visualechoes: Spatial image representation learning through echolocation." ECCV. 2020.

Meta-review: All reviewers recommend acceptance. The AC sees no basis to overturn the reviews. Authors should attend to main points in the reviews, such as the concerns about evaluation (e.g. Reviewer iWmP), when preparing a final version.
Decision: Accept (poster)