SlowGuess/NeurIPS_2023_OCR huggingface dataset:
After computing the relative angle  $\theta^{\prime}$ , we choose learnable embeddings instead of positional encoding [16, 45] to project  $\theta^{\prime}$  into a high-frequency space. We use the embedding  $\Theta \in \mathcal{R}^{4\times c}$  to represent four discrete directions, namely,  $0^{\circ}$ ,  $90^{\circ}$ ,  $180^{\circ}$ , and  $270^{\circ}$ , where 4 is the number of directions and  $c$  is the width of A-NeRF. Given a relative angle  $\theta^{\prime} \in [0^{\circ}, 360^{\circ})$ , we linearly interpolate the direction embedding  $\Theta^{\prime} \in \mathcal{R}^{c}$  according to the angle between  $\theta^{\prime}$  and four discrete directions. We add the
(a) Office
(b) House
(c) Apartment
Figure 5: Example scenes of RWAVS dataset. RWAVS dataset consists of diverse audio-visual scenes indoors and outdoors. Each environment possesses distinct acoustics and structures.
(d) Outdoors
interpolated embedding  $\theta^{\prime}$  to the input of each linear layer in A-NeRF, thereby providing A-NeRF with direction information. Ablation studies (Sec. 5.2) show that this encoding method performs best.
4.5 Learning Objective
The loss function of V-NeRF is the same as [16]:
$$
\mathcal {L} _ {V} = \left\| C (\mathbf {r}) - \hat {C} (\mathbf {r}) \right\| ^ {2}, \tag {6}
$$
where  $C(\mathbf{r})$  is the ground-truth color along the ray  $\mathbf{r}$  and  $\hat{C} (\mathbf{r})$  is the color rendered by V-NeRF. After training V-NeRF, we optimize A-NeRF and AV-Mapper together with the L2 loss function:
$$
\mathcal {L} _ {A} = \left\| s _ {m} - \hat {s} _ {m} \right\| ^ {2} + \left\| s _ {l} - \hat {s} _ {l} \right\| ^ {2} + \left\| s _ {r} - \hat {s} _ {r} \right\| ^ {2}, \tag {7}
$$
where  $s_{m,l,r}$  are the predicted magnitudes,  $\hat{s}_{m,l,r}$  are the ground-truth magnitudes, and subscript  $m,l,r$  are mixture, left, and right, respectively. The first term of  $\mathcal{L}_A$  encourages A-NeRF to predict masks that represent spatial effects caused by distance, while the second and third term encourages A-NeRF to generate masks that capture the differences between two channels.
To the best of our knowledge, our method is the first NeRF-based system capable of synthesizing real-world videos with perceptually realistic binaural audios at arbitrary poses. However, existing datasets do not meet the specific requirements of our experiments, particularly in terms of simultaneously providing camera poses, high-quality binaural audios, and images. Therefore, we curated a high-quality audio-visual scene dataset (real) to address this gap and facilitate further research on this problem. Additionally, we utilize (synthetic) SoundSpaces dataset [4] to validate our method.
(1) RWAVS Dataset. We collected the Real-World Audio-Visual Scene (RWAVS) dataset to benchmark our method. In order to increase the diversity of our dataset, we recorded data across different scenarios. Fig. 5 shows the example scenarios we used for data recording, including both indoor and outdoor environments, which we believe represent most daily settings. RWAVS dataset comprises multimodal data, including camera poses, high-quality binaural audios, and videos. Unlike ReplayNVAS dataset [15], where the environment and the recording viewpoint are constant, RWAVS dataset contains various viewpoints in diverse environments. During data recording, we randomly moved around the environment while holding the device, capturing various acoustic and visual signals. RWAVS dataset encompasses all positions and directions  $(360^{\circ})$  within an environment.
In detail, we employed a 3Dio Free Space XLR binaural microphone for capturing high-quality stereo audio, a TASCAM DR-60DMKII for recording and storing audio, and a GoPro Max for capturing accompanying videos. Additionally, an LG XBOOM 360 omnidirectional speaker was used as the sound source. For each environment and sound source combination, we collected data ranging from 10 to 25 minutes, resulting in a total collection of 232 minutes (3.8 hours) of data from diverse environments with varying source positions.
We extract key frames at 1 fps from recorded videos and use COLMAP [46] to estimate the corresponding camera pose. Each key frame is accompanied by one-second binaural audio and one-second
Table 1: Comparison with state-of-the-art methods on RWAVS dataset.
Methods	Office	House	Apartment	Outdoors	Overall
MAG	ENV	MAG	ENV	MAG	ENV	MAG	ENV	MAG	ENV
Mono-Mono	9.269	0.411	11.889	0.424	15.120	0.474	13.957	0.470	12.559	0.445
Mono-Energy	1.536	0.142	4.307	0.180	3.911	0.192	1.634	0.127	2.847	0.160
Stereo-Energy	1.511	0.139	4.301	0.180	3.895	0.191	1.612	0.124	2.830	0.159
INRAS [13]	1.405	0.141	3.511	0.182	3.421	0.201	1.502	0.130	2.460	0.164
NAF [12]	1.244	0.137	3.259	0.178	3.345	0.193	1.284	0.121	2.283	0.157
ViGAS [15]	1.049	0.132	2.502	0.161	2.600	0.187	1.169	0.121	1.830	0.150
Ours	0.930	0.129	2.009	0.155	2.230	0.184	0.845	0.111	1.504	0.145
Table 2: Ablation studies. We break down AV-NeRF to analyze the contribution of each component. Left: the inclusion of AV-Mapper (AV) and Coordinate Transformation (CT), Middle: different multimodal fusion methods, and Right: different direction encoding methods.
Methods	Overall	Methods	Overall	Methods	Overall
MAG	ENV	MAG	ENV	MAG	ENV
Baseline	2.287	0.157	Concat Input	1.507	0.145	Absolute Direction	1.701	0.149
Ours w/o AV	1.791	0.150	Add Input	1.504	0.145	Relative Direction	1.508	0.145
Ours w/o CT	1.701	0.149	Add All Layers	1.505	0.145	Relative Embedding	1.504	0.145
Ours	1.504	0.145
source audio, forming a complete data sample. For audio clips with noticeable background noise, we perform noise suppression using Adobe Audition [47]. We split  $80\%$  data as training samples and the rest as validation samples. After pre-processing, we obtain 9850 and 2469 samples for training and validation, respectively. This dataset is challenging because of the diverse environments and various camera poses. We will release this dataset to the research community.
(2) SoundSpaces Dataset. While RWAVS offers realistic training samples, its realism restricts its scale because it is time-consuming to record high-quality multimodal data in the real world. Therefore, we use the synthetic SoundSpaces dataset to augment our experiments. To evaluate our method on SoundSpaces dataset, we modify AV-NeRF to estimate impulse responses instead of the acoustic mask while keeping all other components intact. We follow NAF [12] selecting six representative indoor scenes, consisting of two single rooms with rectangular walls, two single rooms with non-rectangular walls, and two multi-room layouts. In each scene, SoundSpaces dataset provides an extensive collection of impulse response signals for sound source and sound receiver pairs, which are densely sampled from a 2D room grid. Each pair includes four discrete head orientations  $(0^{\circ},90^{\circ}$ ,  $180^{\circ}$ , and  $270^{\circ}$ ), and each orientation is associated with two-channel binaural RIRs. We render RGB and depth images for each sound receiver pose using Habitat-Sim simulator [48, 49]. We maintain the same training/test split as NAF, allocating  $90\%$  data for training and  $10\%$  data for testing.
5.2 Results on RWAVS Dataset
Comparison with State-of-the-art. We compare AV-NeRF with the following baselines: (1) Mono-Mono duplicates the source audio  $a_{s}$  twice to generate a fake binaural audio without modifying the source audio; (2) Mono-Energy assumes that the average energy of the target audio  $a_{t}$  is known, scales the energy of the input audio to match the target, and duplicates the scaled audio to generate a stereo audio; (3) Stereo-Energy assumes that the energy of the two channels of the target audio  $a_{t}$  is known, separately scales the energy of the input audio to match the target, and combines the two scaled channels to generate a stereo audio; (4) IRNAS [13] learns representing audio scenes by disentangling scene's geometry features with implicit neural fields, and we adapt INRAS to predict wave masks on RWAVS dataset; (5) NAF [12] designs local feature grids and an implicit decoder to capture the sound propagation in a physical scene, and we modify NAF to predict magnitude masks on RWAVS dataset; (6) ViGAS [15] achieves novel-view acoustic synthesis by analyzing audio-visual cues from source viewpoints. We select magnitude distance (MAG) [29], which measures the audio quality in the time-frequency domain, and envelope distance (ENV) [30], which measures the audio quality in the time domain, to evaluate various methods. Please refer to the supplementary material for implementation details.
Rendered Image
Rendered Audio
GT Audio
Figure 6: Visualization of synthesized audio-visual scene. We present the rendered image, synthesized binaural audio, and ground-truth audio.
Rendered Image
Rendered Audio
GT Audio
AV-NeRF outperforms all baselines across different environments, including office, house, apartment, and outdoors, by a significant margin (Table 1). AV-NeRF outruns INRAS with  $0.956$  on the overall MAG metric ( $39\%$  relative improvement) and  $0.019$  on the average ENV metric ( $11.6\%$ ). AV-NeRF surpasses NAF with  $0.779$  on MAG metric ( $34\%$ ) and  $0.012$  on ENV metric ( $8\%$ ). Our approach is better than ViGAS in terms of both MAG (1.504 compared to ViGAS's 1.830) and ENV (0.145 compared to ViGAS's 0.150).
Ablation Studies. We conduct ablation studies on AV-NeRF to analyze the contribution of different components. We combine A-NeRF and V-NeRF as our baseline, which does not contain AV-Mapper (Sec. 4.3) or coordinate transformation (Sec. 4.4). The ablation results are as follows: (1) AV-Mapper (AV) and coordinate transformation (CT) play important roles in learning audio scenes (Table 2 left). The exclusion of either component degrades the generation quality; (2) adding visual information to the input of A-NeRF is the most effective multimodal fusion method compared with concatenation and adding visual information to all layers of A-NeRF (Table 2 middle); (3) using embeddings to represent relative angles outperforms applying positional encoding to either absolute or relative angles (Table 2 right). "Absolute Direction" represents applying positional encoding to the absolute angle, "Relative Direction" means transforming the relative angle with the positional encoding, and "Relative Embedding" is the embedding method.
Visualization. We visualize the synthesized audio-visual scenes in Fig. 6 to intuitively assess the generation quality of our model. AV-NeRF can synthesize realistic binaural audios that have the same signal envelope and channel difference as the ground-truth audios.
5.3 Results on SoundSpaces Dataset
We compare AV-NeRF with traditional audio coding methods [50, 51] and advanced learning-based neural field methods [12, 13] using T60, C50, and EDT metrics [13]. Please refer to our supplementary material for implementation details. Table 3 shows that AV-NeRF outruns both traditional and advanced methods, achieving  $21\%$  relative improvement on T60 metric compared with the previous state-of-the-art method INRAS,  $5\%$  on C50, and  $16\%$  on EDT.
Table 3: Comparison with state-of-the-art. We report the performance on the SonudSpaces dataset using T60, C50, and EDT metrics. The lower score indicates a higher RIR generation quality. Opus is an open audio codec [50], and AAC is a multi-channel audio coding standard [51].
Methods	T60 (%) ↓	C50 (dB) ↓	EDT (sec) ↓
Opus-nearest	10.10	3.58	0.115
Opus-linear	8.64	3.13	0.097
AAC-nearest	9.35	1.67	0.059
AAC-linear	7.88	1.68	0.057
NAF [12]	3.18	1.06	0.031
INRAS [13]	3.14	0.60	0.019
Ours	2.47	0.57	0.016
In this work, we propose a first-of-its-kind NeRF system capable of synthesizing real-world audiovisual scenes. Our model can generate audios with rich spatial information at novel camera poses. We demonstrate the effectiveness of our method on real RWAVS and synthetic SoundSpaces datasets.
Limitation. Firstly, we currently focus on static scenes with a single fixed sound source in our study. However, it is worth exploring the challenge of learning implicit neural representations for audio-visual scenes with multiple dynamic sound sources. Second, while our model successfully generates audio-visual scenes, it does not account for reverberation effects present in the real world. Reverberation is essential for humans to perceive scene size and structure. Third, similar to the original NeRF, AV-NeRF must represent and render each scene separately. Developing a neural field
that can either learn to represent all scenes or transfer knowledge to new environments is a crucial problem, particularly for industrial applications.
Broader Impact. Although AV-NeRF is designed to synthesize audio-visual scenes, it is important to acknowledge its potential application in generating scenes involving artificial humans. The misuse of AV-NeRF could lead to the production of deceptive and misleading media.
[1] Andrew Owens and Alexei A Efros. Audio-visual scene analysis with self-supervised multisensory features. In ECCV, pages 631–648, 2018.
[2] Shentong Mo and Pedro Morgado. Localizing visual sounds the easy way. In Shai Avidan, Gabriel J. Brostow, Moustapha Cisse, Giovanni Maria Farinella, and Tal Hassner, editors, ECCV, volume 13697, pages 218-234, 2022.
[3] Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio-visual event localization in unconstrained videos. In ECCV, pages 247–263, 2018.
[4] Changan Chen, Unnat Jain, Carl Schissler, Sebastia Vicenc Amengual Gari, Ziad Al-Halah, Vamsi Krishna Ithapu, Philip Robinson, and Kristen Grauman. Soundspaces: Audio-visual navigation in 3d environments. In ECCV, 2020.
[5] Changan Chen, Ziad Al-Halah, and Kristen Grauman. Semantic audio-visual navigation. In CVPR, pages 15516–15525, 2021.
[6] Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. The sound of pixels. In ECCV, pages 570-586, 2018.
[7] Ruohan Gao and Kristen Grauman. 2.5d visual sound. In CVPR, 2019.
[8] Hang Zhou, Xudong Xu, Dahua Lin, Xiaogang Wang, and Ziwei Liu. Sep-stereo: Visually guided stereophonic audio generation by associating source separation. In ECCV, pages 52-69. Springer, 2020.
[9] Yudong Guo, Keyu Chen, Sen Liang, Yong-Jin Liu, Hujun Bao, and Juyong Zhang. Ad-nerf: Audio driven neural radiance fields for talking head synthesis. In ICCV, pages 5784-5794, 2021.
[10] Yuanhang Zhang, Susan Liang, Shuang Yang, Xiao Liu, Zhongqin Wu, Shiguang Shan, and Xilin Chen. Unicon: Unified context network for robust active speaker detection. In ACM MM, pages 3964-3972, 2021.
[11] Jinxing Zhou, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, and Yiran Zhong. Audio-visual segmentation. In ECCV, 2022.
[12] Andrew Luo, Yilun Du, Michael J Tarr, Joshua B Tenenbaum, Antonio Torralba, and Chuang Gan. Learning neural acoustic fields. NeurIPS, 2022.
[13] Kun Su, Mingfei Chen, and Eli Shlizerman. Inras: Implicit neural representation for audio scenes. In NeurIPS, 2022.
[14] Yilun Du, M. Katherine Collins, B. Joshua Tenenbaum, and Vincent Sitzmann. Learning signal-agnostic manifolds of neural fields. In NeurIPS, 2021.
[15] Changan Chen, Alexander Richard, Roman Shapovalov, Vamsi Krishna Ithapu, Natalia Neverova, Kristen Grauman, and Andrea Vedaldi. Novel-view acoustic synthesis. arXiv preprint arXiv:2301.08730, 2023.
[16] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[56] Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. NeRF--: Neural radiance fields without known camera parameters. arXiv preprint arXiv:2102.07064, 2021.
[57] Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey. Barf: Bundle-adjusting neural radiance fields. In ICCV, 2021.
[58] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM TOG, 41(4):1-15, 2022.
[59] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. NeurIPS, 32, 2019.
[60] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, ICLR, 2015.
[61] Julius Orion Smith. Mathematics of the discrete Fourier transform (DFT): with audio applications. 2008.
A Additional Visualization Results
We present additional results in Figure 7 that demonstrate AV-NeRF's capability of generating distance-aware auditory effects. For each scene, we select a fixed source audio and synthesize new binaural audio at varying positions and distances from the sound source. The first row displays the rendered images, while the second row showcases the corresponding rendered audio. As illustrated in the figure, the amplitude of the rendered audio diminishes as the distance between the source and the camera increases, and conversely, it increases as the camera approaches the source.
Figure 7: Distance-aware audio rendering. We show some example scenes where AV-NeRF can successfully synthesize consistent binaural audio with the camera movements.
B Failure Cases
We include several failure cases in Figure 8, where we present the rendered images, synthesized binaural audio, and ground-truth audio. AV-NeRF makes wrong acoustic predictions when the audio-visual scene involves noticeable noise or the AV-Mapper can not extract reliable material and geometry information from the visual space.
Because we work for real-world audio-visual learning, there inevitably exist various types of noise in a scene, e.g., sounds from refrigerators, air conditioning, wind, or even workers make noise when collecting data. These ambient noises are recorded by the microphone and are present in the ground-truth binaural audio, which hinders AV-NeRF from accurately learning the acoustic field. In Figure 8 (a), the ground-truth audio (used for training) marked with red boxes contains noticeable noise, resulting in inaccurate audio rendering.
AV-NeRF relies on the AV-Mapper to extract reliable material and geometry information from input images, enabling a comprehensive understanding of the audio-visual environment. However, in cases where the AV-Mapper fails to extract meaningful information from certain images (e.g., images of plain walls or whiteboards), AV-NeRF predicts erroneous acoustic masks. In Figure 8 (b), the rendered audio marked with black boxes displays inconsistent waveforms compared to the ground-truth audio, indicating a model failure due to meaningless material and geometry information.
GT Audio
(a) Noisy environment.
Rendered Audio
GT Audio
(b) Meaningless visual input.
Rendered Audio
GT Audio
Figure 8: Failure cases. We present failure cases in which AV-NeRF makes wrong acoustic predictions. For each case, we show the rendered image, the rendered audio, and the recorded ground-truth audio.
C Rationality of AV-Mapper
In our paper, we leverage RGB and depth images as implicit indicators of environmental material properties and geometry. While these images enable the model to perceive the environment implicitly, it is important to note that achieving a precise one-to-one mapping from images to material properties or geometry is not guaranteed for AV-Mapper. Certain corner cases, such as a black desk and object occlusion, can lead to the failure of AV-Mapper.
Nevertheless, existing studies have demonstrated the feasibility of inferring material and geometry information from images. For instance, back in 2002, Varma and Zisserman [52] proposed a filter-based approach for material classification in images. More recently, in 2017, Zhang et al. [53] successfully recognized material and shape attributes using visual data. Moreover, there is a body of audio-visual research indicating that extracted geometry and material information via ResNet networks can contribute to audio synthesis [54, 55]. Our paper further supports this concept. As illustrated in Table 2 (left), the inclusion of AV-Mapper in AV-NeRF can improve the MAG score by  $16\%$  from 1.791 to 1.504 and enhance the ENV score by  $3\%$  from 0.150 to 0.145.
D Necessity of Distance and Direction Coordinates
Table 4: Necessity of distance and direction information.
Methods	Overall
Position	Direction	MAG	ENV
1.822	0.156
✓	1.817	0.155
✓	1.688	0.148
✓	✓	1.504	0.145
To validate the necessity of distance and direction coordinates when modeling audio fields, we conduct additional ablation studies. We intentionally exclude positional and directional coordinates when representing an auditory scene and evaluate the performance without these inputs. The results are presented in Table 4. AV-NeRF achieves a MAG metric score of 1.504 and an ENV metric score of 0.145 when both position and direction information are utilized. Removing either positional or directional input leads to a performance drop between  $2\%$  to  $21\%$ . In the absence of both positional and directional inputs, performance degrades further to 1.822 on the MAG metric and 0.156 on
the ENV metric. These outcomes distinctly affirm the indispensable role of distance and direction coordinates in acoustic modeling.
E Multiple Sound Sources
Given that scenes with a single sound source may not fully represent real-world complexities, we extend both the RWAVS dataset and the AV-NeRF model to support multi-source scenes.
Specifically, we collect two multi-source scenes, adhering to the recording settings outlined in Section 5.1. The only modification we make is placing two sound sources instead of a single one in the environment, thus creating multiple sound sources. We extend AV-NeRF by stacking multiple equal A-NeRF modules to support the parameterization of multi-source acoustic fields. AV-NeRF generates acoustic masks for each sound source separately. We conduct experiments in these multi-source scenes and present the performance of AV-NeRF and other baselines in Table 5. As presented in the table, AV-NeRF outperforms other baselines in both MAG (0.282) and ENV (0.063) metrics. The experimental results clearly demonstrate the proposed method's capability to effectively handle multiple sound sources.
Table 5: Comparison with state-of-the-art methods on multi-source scenes.
Methods	Multi-source
MAG	ENV
Mono-Mono	1.949	0.172
Mono-Energy	0.533	0.075
Stereo-Energy	0.527	0.073
INRAS [13]	0.472	0.078
NAF [12]	0.401	0.080
Ours	0.282	0.063
F Architectures
A-NeRF. A-NeRF consists of two Multilayer Perceptrons (MLPs), each comprising four linear layers with an additional residual connection. The width of each linear layer, denoted as  $c$ , is set to 128 for the RWAVS dataset and 256 for the SoundSpaces dataset. In A-NeRF, all linear layers are followed by ReLU activation layers, except for the last layer, where the ReLU activation is replaced with the Sigmoid function. The first MLP takes the listener's position  $(x,y)$  and the frequency  $f \in [0,F]$  as input, where  $F$  represents the number of frequency bins. It predicts a mixture mask  $m_{m} \in \mathcal{R}$  for the given frequency  $f$  and generates a feature vector with  $c$  channels. Prior to feeding them into the MLP, we apply positional encoding to the listener's position  $(x,y)$  and the frequency  $f$ . We set the maximum frequency used for positional encoding as 10.
Then, we adopt relative transformation (Sec. 4.4) to project the listener's direction  $\theta$  into a high-frequency space. We concatenate the transformed listener's direction and the feature vector, and feed it into the second MLP. The second MLP is appended with a Sigmoid layer and a scaling layer, ensuring that the difference mask  $m_d$  estimated by the second MLP falls within the range of  $[-1,1]$ . For each frequency query  $f$ , A-NeRF estimates two masks:  $m_m$  and  $m_d$ , both of which are scalars. We iterate over all frequencies  $f \in [0,F]$  to obtain the complete masks  $\mathbf{m}_m$  and  $\mathbf{m}_d$ . After computing the masks, we synthesize the target audio  $a_t$  according to the procedure discussed in Sec. 4.2.