SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Susan Liang $^{1}$  Chao Huang $^{1}$  Yapeng Tian $^{1}$  Anurag Kumar $^{2}$  Chenliang Xu $^{1}$ $^{1}$ University of Rochester  $^{2}$ Meta Reality Labs Research
Can machines recording an audio-visual scene produce realistic, matching audio-visual experiences at novel positions and novel view directions? We answer it by studying a new task—real-world audio-visual scene synthesis—and a first-of-its-kind NeRF-based approach for multimodal learning. Concretely, given a video recording of an audio-visual scene, the task is to synthesize new videos with spatial audios along arbitrary novel camera trajectories in that scene. We propose an acoustic-aware audio generation module that integrates prior knowledge of audio propagation into NeRF, in which we implicitly associate audio generation with the 3D geometry and material properties of a visual environment. Furthermore, we present a coordinate transformation module that expresses a view direction relative to the sound source, enabling the model to learn sound source-centric acoustic fields. To facilitate the study of this new task, we collect a high-quality Real-World Audio-Visual Scene (RWAVS) dataset. We demonstrate the advantages of our method on this real-world dataset and the simulation-based SoundSpaces dataset. We recommend that readers visit our project page for convincing comparisons: https://liangsusan-git.github.io/project/avnerf/.
We study a new task, real-world audio-visual scene synthesis, to generate target videos and audios along novel camera trajectories from source audio-visual recordings of known trajectories. By learning from real-world source videos with binaural audio, we aim to generate target video frames and spatial audios that exhibit consistency with the given camera trajectory visually and acoustically. This consistency ensures perceptual realism and immersion, enriching the overall user experience.
As far as we know, attempts in the audio-visual learning literature [1-11] have yet to succeed in solving this challenging task thus far. Although there are similar works [12-15], these methods have constraints that limit their ability to solve this new task. Luo et al. [12] propose neural acoustic fields to model sound propagation in a room. Su et al. [13] introduce representing audio scenes by disentangling the scene's geometry features. These methods are tailored for estimating room impulse response signals in a simulation environment that are difficult to obtain in a real-world scene. Concurrent to our work, ViGAS proposed by Chen et al. [15] learns to synthesize new sounds by inferring the audio-visual cues. However, ViGAS is limited to a few viewpoints for audio generation.
We introduce AV-NeRF, a novel NeRF-based method of synthesizing real-world audio-visual scenes. AV-NeRF enables the generation of videos and spatial audios, following arbitrary camera trajectories. It utilizes source videos and camera poses as references. AV-NeRF consists of two branches: A-NeRF, which learns the acoustic fields of an environment, and V-NeRF, which models color and density fields. We represent a static audio field as a continuous function using A-NeRF, which takes the listener's position and head direction as input. A-NeRF effectively models the energy decay of sound as the sound travels from the source to the listener by correlating the listener's position with the
Figure 1: NeRF learns to render visual scenes from novel viewpoints. Beyond visual rendering, we introduce AV-NeRF, a method for synthesizing audio-visual scenes with video frames and binaural audiors along new camera trajectories. Integrating coherent sight and sound creates an immersive and realistic perceptual experience for users.
energy reduction effect. Moreover, A-NeRF accounts for the impact of the listener's orientation on sound perception by establishing a correlation between the audio channel difference and the listener's direction. Consequently, A-NeRF facilitates distance-sensitive and direction-aware audio synthesis, resulting in the generation of realistic binaural sound. We utilize the vanilla NeRF [16] as our V-NeRF for vision rendering since the rendering of the purely visual world has been extensively researched [17, 16, 18-20] and is not the primary focus of our work.
To further enhance synthesis quality, we introduce an acoustic-aware audio generation method and a coordinate transformation mechanism. Considering that the 3D geometry and material properties of an environment determine sound propagation [21, 22, 4, 23], we propose an acoustic-aware audio generation module called AV-Mapper. AV-Mapper extracts geometry and material information from V-NeRF and feed them to A-NeRF, enabling A-NeRF to generate audios with acoustic awareness. In visual space, NeRF [16] expresses the viewing direction  $(\theta, \phi)$  using an absolute coordinate system, where the directions of two parallel light rays are represented identically. However, in auditory space, this expression method is unsuitable as it ignores the spatial relationship between the sound emitter and receiver. Human perception of sound source relies on the relative direction of the sound source to the head orientation. Given this observation, we propose a coordinate transform mechanism that expresses the viewing direction relative to the sound source. This approach encourages our model to learn sound source-centric acoustic fields.
Our work represents the initial step towards addressing the audio-visual scene synthesis task in real-world settings. As such, there is currently no publicly available dataset that fulfills our specific requirements. To facilitate our study, we curated a high-quality audio-visual dataset called RWAVS (Real-World Audio-Visual Synthesis), which encompasses multimodal data, including camera poses, video frames, and realistic binaural (stereo) audios. In order to enhance the diversity of our dataset, we collected data from various environments, ranging from offices and apartments to houses and even outdoor spaces. We will release this dataset to the research community. Experiments on the RWAVS and the synthetic SoundSpaces datasets can validate the effectiveness of our proposed approach. It demonstrates AV-NeRF's capability of synthesizing novel-view audio-visual scenes in a range of environments, including both real-world and synthetic, as well as indoor and outdoor settings.
In summary, our contributions include: (1) proposing a novel approach to synthesize audio-visual scenes in real-world environments at novel camera poses; (2) introducing a new acoustic-aware audio generation method that incorporates our prior knowledge of sound propagation; (3) suggesting a coordinate transformation mechanism to effectively express sound direction; (4) curating a high-quality audio-visual dataset for benchmarking this task; and (5) demonstrating the advantages of our method through quantitative and qualitative evaluations.
Our work is closely related to several areas, including novel-view synthesis, vision-guided audio spatialization, simulation-based acoustic synthesis, and learning-based audio generation. We elaborate on each of these topics in the following section.
Neural Fields. Our method is based on neural fields, particularly Neural Radiance Fields (NeRF) [16]. NeRF employs MLPs to learn an implicit and continuous representation of visual scenes, enabling the synthesis of novel views. Building upon NeRF's foundation, several works have extended its applicability to broader domains, including in-the-wild images [24], (silent) video [25-27], audio
[12, 13], and audio-visual [14] content. Luo et al. [12] propose neural acoustic fields (NAF) to capture the sound propagation in an environment, and Su et al. [13] introduce disentangling a scene's geometry features for audio scene representation (INRAS). Despite the compelling results achieved by NAF and INRAS, their application in the real world is hindered by their reliance on ground-truth impulse response signals that are simulated in synthetic environments. Furthermore, the discretization of position and direction in these approaches restricts the range of camera poses they can handle, whereas our model can handle continuous position and direction queries. Du et al. [14] proposed a manifold learning method that maps vectors from a latent space to audio and image spaces. Although the learned manifold allows for audio-visual interpolation, the model lacks support for controllable audio-visual generation. In contrast, our method focuses on learning audio-visual representations that are explicitly conditioned on spatial coordinates, enabling controllable generation.
Visually Informed Spatial Audio Generation. Considering that images often reveal the position of a sound source and/or the structure of a scene, many visually informed audio spatialization approaches [7, 28-31, 15] have been proposed. Among them, Gao and Grauman [7] focus on normal field-of-view videos and binaural audi. s. Zhou et al. [28] propose a unified framework to solve the sound separation and stereo sound generation at the same time. Xu et al. [29] propose a Pseudo2Binaural pipeline that augments audio data using HRIR function [32]. Most recently, Chen et al. [15] proposed the Visually-Guided Acoustic Synthesis (ViGAS) method for generating novel-view audios. However, AV-NeRF differs from this method in two ways: (1) ViGAS relies on ground-truth images for audio transformation, whereas AV-NeRF addresses the absence of ground-truth images by utilizing rendered images from learned vision fields; (2) ViGAS only supports a limited number of viewpoints for audio synthesis, while AV-NeRF has the capability to render novel videos at arbitrary camera poses.
Geometry and Material Based Acoustic Simulation. Several works [4, 33, 22, 34, 23] focus on simulating acoustic environments by modeling their 3D geometry and material properties. SoundSpaces, proposed by Chen et al. [4], is an audio platform that calculates the room impulse response signals for discrete listener and sound source positions. Extending this work, Chen et al. [33] introduce SoundSpaces 2.0, which enables acoustic generation for arbitrary microphone locations. Tang et al. [23] propose calibrating geometric acoustic ray-tracing with a finite-difference time-domain wave solver to compute high-quality impulse responses. Additionally, Li et al. [22] propose a method that blends the early reverberation portion, modeled using geometric acoustic simulation and frequency modulation, with a late reverberation tail extracted from recorded impulse responses. In contrast to these simulation methods, our approach implicitly leverages learned geometry and material information for audio-visual scene synthesis.
Learning Based Audio Generation. Learning-based methods exploit the powerful modeling ability of neural networks to synthesize audios. Tang et al. [34] propose the use of neural networks to estimate reverberation time ( $T_{60}$ ) and equalization (EQ). Ratnarajah et al. [35, 36, 37] employ generative adversarial networks (GANs) to supervise realistic room impulse response generation. Richard et al. [38] introduce a binauralization network that models the sound propagation linearly and nonlinearly. Recent research has also investigated learning from noisy data [39] and few-shot data [40].
3 Task Definition
The real-world audio-visual scene synthesis task is to generate visual frames and corresponding binaural audiors for arbitrary camera trajectories. Given a static environment  $E$  and multiple observations  $O = \{O_1, O_2, \ldots, O_N\}$  of this environment, where  $N$  is the number of training samples and each observation  $O_i$  includes the camera pose  $p = (x, y, z, \theta, \phi)$ , the mono source audio clip  $a_s$ , the recorded binaural audio  $a_t$ , and the image  $I$ , this task aims to synthesize a new binaural audio  $a_t^*$  and a novel view  $I^*$  based on a camera pose query  $p^*$  and a source audio  $a_s^*$ , where  $p^*$  is distinct from the camera poses in all observations. This process can be formatted as:
$$
\left(a _ {t} ^ {*}, I ^ {*}\right) = f \left(p ^ {*}, a _ {s} ^ {*} | O, E\right), \tag {1}
$$
where  $f$  is a mapping function. Combining predicted  $a_{t}^{*}$  and  $I^{*}$  of all poses in the queried camera trajectory, we can synthesize a realistic video with spatial audio. This task poses several challenges: (1)  $p^{*}$  is a novel viewpoint distinct from all observed viewpoints, (2) the synthesized binaural audio  $a_{t}^{*}$  is expected to exhibit rich spatial effects contributing to its realism, (3) during inference, no observation  $O$  is accessible and the mapping function  $f$  relies solely on the learned fields for scene synthesis. It should be noted that the position of the sound source is known in the environment  $E$ .
Figure 2: The pipeline of our method. Given the position  $(x, y, z)$  and viewing direction  $(\theta, \phi)$  of a listener, our method can render an image the listener would see and the corresponding binaural audio the listener would hear. Our model consists of V-NeRF, A-NeRF, and AV-Mapper. A-NeRF learns to generate acoustic masks, V-NeRF learns to generate visual frames, and AV-Mapper is optimized to integrate geometry and material information extracted from V-NeRF into A-NeRF.
Our method aims to learn neural fields that can synthesize real-world audio-visual scenes from novel poses. The entire pipeline is illustrated in Figure 2. Our model comprises three trainable modules: V-NeRF, A-NeRF, and AV-Mapper. A-NeRF is responsible for generating acoustic masks, V-NeRF focuses on generating visual frames, and AV-Mapper is optimized to extract geometry and material information from V-NeRF, integrating this information into A-NeRF.
4.1 V-NeRF
NeRF [16] uses a Multi-Layer Perceptron (MLP) to implicitly and continuously represent a visual scene. It effectively learns a mapping from camera poses to colors and densities:
$$
\operatorname {N e R F}: (x, y, z, \theta , \phi) \rightarrow (\mathbf {c}, \sigma), \tag {2}
$$
where  $\mathbf{X} = (x,y,z)$  is the 3D position,  $\mathbf{d} = (\theta ,\phi)$  is the direction,  $\mathbf{c} = (r,g,b)$  is the color, and  $\sigma$  is the density. In order to achieve view-dependent color rendering  $\mathbf{c}$  and ensure multi-view consistency, [16] introduces two MLP components within NeRF. The first MLP maps a 3D coordinate  $(x,y,z)$  to density  $\sigma$  and a corresponding feature vector. The second MLP then takes the feature vector and 2D direction  $(\theta ,\phi)$  as inputs, producing the color  $\mathbf{c}$ . This process is illustrated in Fig. 3. NeRF then uses the volume rendering method [41] to generate the color of any ray  $\mathbf{r}(t) = \mathbf{o} + t\mathbf{d}$  as it traverses through the visual scene, bounded by near  $t_n$  and far  $t_f$ :
$$
C (\mathbf {r}) = \int_ {t _ {n}} ^ {t _ {f}} T (t) \sigma (\mathbf {r} (t)) \mathbf {c} (\mathbf {r} (t), \mathbf {d}) d t, \tag {3}
$$
where  $T(t) = \exp(-\int_{t_n}^t \sigma(\mathbf{r}(s)) ds)$ . The depth of any ray  $\mathbf{r}$  can be calculated similarly by replacing  $\mathbf{c}(\mathbf{r}(t), \mathbf{d})$  with  $t$ :
$$
D (\mathbf {r}) = \int_ {t _ {n}} ^ {t _ {f}} T (t) \sigma (\mathbf {r} (t)) t d t. \tag {4}
$$
These continuous integrals are estimated by quadrature in practice [42]. Subsequently, RGB and depth images can be rendered for any given camera pose using V-NeRF (Fig. 3). We apply positional encoding to all input coordinates to preserve the high-frequency information of the generated images.
Figure 3: The pipeline of V-NeRF. Given the 3D position  $(x,y,z)$  and direction  $(\theta, \phi)$  of a camera pose, V-NeRF can render corresponding RGB and depth images. V-NeRF consists of two MLPs with the first MLP estimating density  $\sigma$  and the second one predicting color  $\mathbf{c}$ .
4.2 A-NeRF
The goal of A-NeRF is to learn a neural acoustic representation capable of mapping 5D coordinates  $(x,y,z,\theta ,\phi)$  to corresponding acoustic masks  $\mathbf{m}_m,\mathbf{m}_d\in \mathcal{R}^F$  , where  $\mathbf{m}_m$  quantifies the change in audio magnitude with respect to the position  $(x,y,z)$  while  $\mathbf{m}_d$  characterizes the impact of the direction  $(\theta ,\phi)$  on the channel difference of binaural audios, and  $F$  is the number of frequency bins:
$$
\operatorname {N e R F}: (x, y, z, \theta , \phi) \rightarrow (\mathbf {m} _ {m}, \mathbf {m} _ {d}). \tag {5}
$$
After training A-NeRF, we can synthesize binaural sound that contains rich spatial and acoustic information by fusing the magnitude of any input audio with predicted  $\mathbf{m}_m$  and  $\mathbf{m}_d$ .
In practice, we simplify the problem of audio synthesis by discarding the  $z$  axis and the  $\phi$  direction, which represents coordinates related to height, and instead focus on the 2D space. On RWAVS and SoundSpaces datasets, we observe that the sound received by the listener exhibits more distinct variations along the horizontal plane, as opposed to the vertical axis. Fig. 4 depicts the pipeline of audio synthesis. Similar to V-NeRF, A-NeRF consists of two MLP components that parameterize different aspects of acoustic fields. The first MLP takes as input the 2D position  $(x,y)$  and frequency query  $f\in [0,F]$ , producing a mixture mask  $\mathbf{m}_m$  for frequency  $f$  and a feature vector. The mask  $\mathbf{m}_m$  quantifies the change in audio magnitude based on the distance between the sound source and the sound receiver. The feature vector serves as an implicit embedding of the input information and the acoustic scene. Then we concatenate the feature vector with the transformed direction  $\theta^{\prime}$  (explained in Sec. 4.4) and pass them to the second MLP. The second MLP associates the direction  $\theta^{\prime}$  with the channel difference of binaural audios. Given a fixed sound emitter, the difference between two channels of the received audio changes as the direction  $\theta^{\prime}$  of the sound receiver varies. For instance, when the receiver faces the emitter, the energy of the two channels will be approximately equal. Conversely, when the emitter is on the left side of the receiver, the left channel will exhibit higher energy compared to the right channel. The second MLP is optimized to generate a difference mask  $\mathbf{m}_d$  that characterizes such direction influence. We query A-NeRF with all frequencies  $f$  within  $[0,F]$  to generate the complete masks  $\mathbf{m}_m$  and  $\mathbf{m}_d$ .
After obtaining the mixture mask  $\mathbf{m}_m$  and the difference mask  $\mathbf{m}_d$ , we can synthesize binaural audios by composing these masks and input audios (Fig. 4). For an input source audio  $a_s$ , we initially employ the short-time Fourier transform (STFT) to calculate the magnitude  $s_s \in \mathcal{R}^{F \times W}$ , where  $W$  represents the number of time frames and  $F$  denotes the number of frequency bins. We then multiply  $s_s$  with mask  $\mathbf{m}_m$  to obtain the mixed magnitude  $s_m$ . Additionally, we multiply  $s_m$  and mask  $\mathbf{m}_d$  to predict the difference magnitude  $s_d$ . Then we add mixture magnitude  $s_m$  and difference magnitude  $s_d$  to compute the magnitude of left channel  $s_l$ , and subtract  $s_d$  from  $s_m$  to compute the magnitude of right channel  $s_r$ . We further refine  $s_l$  and  $s_r$  using 2D convolution. Finally, we apply Inverse STFT to the predicted  $s_l$  and  $s_r$ , respectively, to synthesize the binaural audio  $a_t$ . Because A-NeRF operates on the magnitude, we use the phase information of the source audio for the Inverse STFT process. Instead of estimating two channels of the target audio directly, we predict the mixture and the difference between the two channels of the target audio. Gao and Grauman [7] suggest that direct two-channel predictions can lead to shortcut learning in the spatialization network.
4.3 AV-Mapper
Given the fact that 3D geometry and material property determine sound propagation in an environment (prior knowledge), we propose an acoustic-aware audio generation method that integrates color and depth information estimated by V-NeRF with A-NeRF. When V-NeRF learns to represent visual scenes, it can learn the color c and density  $\sigma$  function of the environment, thanks to the multi-view
Figure 4: The pipeline of A-NeRF. Given the 2D position  $(x, y)$  and direction  $\theta$  of a camera pose, A-NeRF can render mixture mask  $\mathbf{m}_m$ , and difference mask  $\mathbf{m}_d$ , which then be used for audio synthesis. A-NeRF is also composed of two MLPs, with the first predicting mixture mask  $\mathbf{m}_m$  and the second estimating difference mask  $\mathbf{m}_d$ . The direction  $\theta$  is transformed relatively to the sound source prior to inputting into A-NeRF.
consistency constraint. Utilizing volume rendering [16, 41], we can synthesize the RGB and depth image for any given camera pose. The RGB image captures the semantics and category of each object, implicitly indicating their material properties. And the depth image depicts the geometric structure of the environment. By providing A-NeRF with a combination of RGB and depth images, we can offer rich environmental knowledge to A-NeRF.
Specifically, when synthesizing the binaural audio for a given camera pose, our process begins by rendering a pair of RGB and depth images with the same camera pose using Eq. 3 and Eq. 4, respectively. We exploit a pre-trained encoder (e.g., ResNet-18 [43] pre-trained on ImageNet dataset [44]) to extract color and depth features from the RGB and depth images. The extracted features can serve as important indicators of the environment. Next, we develop an AV-Mapper, implemented as an MLP network, to project the color and depth features into a latent space. Our goal for the A-V Mapper is two-fold: (1) to extract meaningful embeddings while discarding unnecessary features, and (2) to align the features in the original space of the pre-trained image encoder with the space relevant to our specific problem. The output embedding of the AV-Mapper, denoted as  $\mathbf{e} \in \mathcal{R}^c$ , with  $c$  representing the width of each linear layer in A-NeRF, is then added to the input of A-NeRF for controlling the audio synthesis. Ablation studies (Sec. 5.2) demonstrate the slight advantage of this fusion method over others.
4.4 Coordinate Transformation
Viewing direction  $(\theta, \phi)$  in V-NeRF is expressed in an absolute coordinate system. This is a natural practice in visual space such that directions of two parallel light rays are expressed identically. However, this expression method in audio space is less suitable because the human perception of the sound direction is based on the relative direction to the sound source instead of the absolute direction. To address this limitation, we propose expressing the viewing direction of a camera pose relative to the sound source. This coordinate transformation encourages A-NeRF learning a sound source-centric acoustic field, enhancing spatial audio generation.
Given the 2D position  $(x,y)$  and the direction  $\theta$  of the camera, as well as the position of the sound source  $(S_x,S_y)$ , we obtain two direction vectors:  $\mathbf{V}_1 = (S_x - x,S_y - y)$  and  $\mathbf{V}_2 = (\cos (\theta),\sin (\theta))$ .  $\mathbf{V}_1$  represents the direction from the camera to the sound source, and  $\mathbf{V}_2$  represents the camera's direction in the absolute coordinate system. By calculating the angle between  $\mathbf{V}_1$  and  $\mathbf{V}_2$ , denoted as the relative direction  $\theta^{\prime} = \angle (\mathbf{V}_1,\mathbf{V}_2)$ , we obtain the rotation angle relative to the sound source. This angle  $\theta^{\prime}$  allows different camera poses to share the same direction encoding if they face the sound source at the same angle.