SlowGuess/NeurIPS_2023_OCR huggingface dataset:
V-NeRF. We utilize the nefacto model provided by nef-studio [20] as the V-NeRF. This model combines several well-established and successful methods, including camera pose refinement [56, 57], image appearance conditioning [24], hash encoding, and proposal sampling [58]. Due to its robust and effective performance on real-world data, we utilize the default settings of the nefacto model without making any architectural modifications. For more detailed information regarding the architecture of V-NeRF, please refer to the documentation provided by nef-studio.
AV-Mapper. For each camera pose, we render both RGB and depth images using V-NeRF. We resize images to  $256 \times 256$  and center-crop to  $224 \times 224$  prior to feeding them into a frozen ResNet-18 [43]
image encoder pre-trained on ImageNet-1K dataset [44]. ResNet-18 embeds the input image as a 512-dimension feature vector. We concatenate the RGB and the depth feature vectors, and input them into the AV-Mapper to learn environmental knowledge of the sound acoustics. AV-Mapper projects the input feature vectors to a latent embedding of  $c$  channels. We implement the AV-Mapper as a 3-layer MLP, with each intermediate linear layer followed by a ReLU activation function.
G Implementation Details
RWAVS Dataset. We implement our method using the PyTorch framework [59]. We employ Adam optimizer [60] with  $\beta_{1} = 0.9$  and  $\beta_{2} = 0.999$  for model optimization. The initial learning rate is set to  $5e - 4$  and exponentially decreased to  $5e - 6$ . We train the model for 100 epochs with a batch size of 32.
Before feeding the camera position  $(x, y)$  to A-NeRF, we normalize it within the range  $[-1, 1] \times [-1, 1]$  and apply positional encoding [16]. Additionally, we resample all audios to a frequency of  $22050\mathrm{Hz}$  and utilize the Short-Time Fourier Transform (STFT) to convert waveform audios into the time-frequency domain. For this transformation, we set the number of fts as 512, the window length as 512, and the hop length as 128. A Hanning window is applied during the process. Finally, we compute both the magnitude and the phase from the spectrogram.
We use magnitude distance (MAG) [29] and envelope distance (ENV) [30] as evaluation metrics for audio quality. The MAG metric quantifies audio quality in the time-frequency domain and is defined as follows:
$$
\operatorname {M A G} \left(\mathbf {m} _ {\mathrm {p r d}}, \mathbf {m} _ {\mathrm {g t}}\right) = \left| \left| \mathbf {m} _ {\mathrm {p r d}} - \mathbf {m} _ {\mathrm {g t}} \right| \right| ^ {2}, \tag {8}
$$
where  $\mathbf{m}_{\mathrm{prd}}$  is the predicted magnitude, and  $\mathbf{m}_{\mathrm{gt}}$  is the ground-truth magnitude. ENV metric that measures the audio quality in the time domain is formatted as:
$$
\operatorname {E N V} \left(a _ {\mathrm {p r d}}, a _ {\mathrm {g t}}\right) = \left| \left| \text {h i l b e r t} \left(a _ {\mathrm {p r d}}\right) - \text {h i l b e r t} \left(a _ {\mathrm {g t}}\right) \right| \right| ^ {2}, \tag {9}
$$
where  $a_{\mathrm{prd}}$  is the predicted audio,  $a_{\mathrm{gt}}$  is the ground-truth audio, and hilbert is the Hilbert transformation function [61].
SoundSpaces Dataset. Our model is trained on the SoundSpaces dataset using the same training settings as RWAVS dataset. We resample the impulse responses to  $22050\mathrm{Hz}$  following INRAS [13]. The 2D position is normalized to  $[-1,1]\times [-1,1]$  prior to positional encoding.
We tailor A-NeRF for impulse response prediction with some minor modifications: (1) The input frequency query  $f$  is replaced by a time query  $t \in [0, T]$ , where  $T$  represents the length of an impulse response signal; (2) the first MLP only generates a feature vector while discarding the mixture mask  $\mathbf{m}_m$ ; (3) the second MLP predicts impulse response signals instead of difference mask  $\mathbf{m}_d$ .
Since the generated impulse responses are in the time domain, we employ STFT to convert them into the time-frequency domain and calculate their magnitudes. We utilize an STFT configuration with 512 FFTs, a sliding window width of 512, a hop stride of 128, and a Hanning window. We supervise the model training using the L2 distance between the ground-truth magnitudes and the predicted magnitudes.
For performance evaluation, we choose three metrics: T60, C50, and EDT [13]. T60 characterizes the reverberation effects in an audio signal by measuring the time it takes for the audio's energy to attenuate by 60 dB. The T60 distance is calculated as follows:
$$
\mathrm {T} 6 0 \left(a _ {\mathrm {p r d}}, a _ {\mathrm {g t}}\right) = \frac {\left| \mathrm {T} 6 0 \left(a _ {\mathrm {p r d}}\right) - \mathrm {T} 6 0 \left(a _ {\mathrm {g t}}\right) \right|}{\mathrm {T} 6 0 \left(a _ {\mathrm {g t}}\right)} , \tag {10}
$$
where  $a_{\mathrm{prd}}$  and  $a_{\mathrm{gt}}$  are the predicted and ground-truth impulse responses, respectively. C50 quantifies the energy ratio between early reflections and late reverberation, allowing it to represent the clarity and loudness of the audio. We format the C50 distance as:
$$
\left. \mathrm {C} 5 0 \left(a _ {\mathrm {p r d}}, a _ {\mathrm {g t}}\right) = \left| \mathrm {C} 5 0 \left(a _ {\mathrm {p r d}}\right) - \mathrm {C} 5 0 \left(a _ {\mathrm {g t}}\right) \right| \right.. \tag {11}
$$
The EDT metric shares similarities with T60 but places greater emphasis on capturing the early reflections of impulse responses. The EDT distance is defined as follows:
$$
\mathrm {E D T} \left(a _ {\mathrm {p r d}}, a _ {\mathrm {g t}}\right) = \left| \mathrm {E D T} \left(a _ {\mathrm {p r d}}\right) - \mathrm {E D T} \left(a _ {\mathrm {g t}}\right) \right|. \tag {12}
$$
With these three metrics, we can evaluate the generation quality of impulse responses from different aspects, including clarity, energy, and reverberation.
H Setup of RWAVS Dataset
Recording Devices. We have assembled a recording system, as depicted in Fig. 9, to capture high-quality audio-visual scenes in real-world environments. Our system comprises a 3Dio Free Space XLR binaural microphone for capturing stereo audio, a TASCAM DR-60DMKII for recording and storing audio, and a GoPro Max for capturing accompanying videos.
This system is portable, allowing us to position it flexibly and capture scenes from different camera poses. In addition, we utilized an LG XBOOM 360 omnidirectional speaker to serve as a sound source, which plays music repeatedly. Figure 5 in the main paper illustrates the setup used to record data in four distinct environments: office, house, apartment, and outdoors. Within each environment, we positioned the speaker at multiple locations to capture diverse acoustic effects. Each combination of environment and sound source represents an audio-visual scene. We collected data ranging from 10 to 25 minutes for each scene, resulting in a total collection of 232 minutes (3.8 hours) of diverse data, encompassing various environments and source positions.
Figure 9: Recording system. It comprises a professional binaural microphone, a sports camera, and a recorder.
Example Scenes. In Fig.10, we present four
example scenes from RWAVS dataset along with the corresponding camera pose distributions. The first column showcases images of the example scenes. The second column displays the camera poses used for video recording: the black dot represents the sound source, and each blue triangle represents a camera pose. We normalize all camera poses to the range of  $[-1, 1]$  and visualize them in an x-y plane from a top-down view. The third column contains 2D density heatmaps, which illustrate the distribution of camera poses in each unit area: each pixel represents a unit area and its color shows the number of camera poses in this area. As shown in the figure, RWAVS dataset encompasses densely covered camera poses for each environment. We also analyze the distribution of camera directions (shown in the last column of Fig.10). We present the direction distribution in a polar coordinate system with the angle representing the viewing direction and the radius meaning the number of camera poses in this region. RWAVS dataset consists of various viewpoints that approximately cover a  $360^{\circ}$  range of viewing directions. In summary, the RWAVS dataset comprises diverse environments with a wide range of camera poses.
Example Scenes
Camera Poses
Position Distribution
Direction Distribution
Figure 10: Example scenes. We present several example scenes along with their corresponding camera pose distributions. We display the position density heatmap and the direction distribution map. RWAVS dataset is composed of diverse environments with various camera poses.