SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Training proceeds in three steps. (1) First we pretrain the de-biaser  $G$ . This entails pretraining a MetricGAN-U [14] with the Speech-to-Energy-Modulation-Ratio (SRMR) metric [11] on speech pre-processed with our off-the-shelf dereverberator (see Sec. 5). By refining the dereverberated output with the MetricGAN-U, we improve its quality and intelligibility without introducing additional supervision requirements. (2) Second, we pretrain the reverberators that perform (visual) acoustic matching. Specifically, we train  $R_v$  and  $R_b$  with  $\mathcal{L}_{\text{visual}}$  and  $\mathcal{L}_{\text{blind}}$ , respectively, using the dereverberated and SRMR-optimized outputs from the MetricGAN in step (1). (3) Finally, we jointly fine-tune both the de-biaser  $G$  and reverberators, using the acoustic residue metric (Eqn. 3) for the GAN metric  $\mathcal{M}$ , the generator and discriminator losses given in Eqns. 2, and 1 together with our alternating training scheme defined in Sec. 3.3. To improve stability in training, since the discriminator  $D$  starts with a good approximation of the SRMR metric, we continue training in step 3 using a weighted combination of SRMR and our residue metric:  $\alpha \text{SRMR}(A) + (1 - \alpha) \mathcal{M}(A, V)$ . Although the acoustic residue is approximately differentiable, the implementation of SRMR we use is not, motivating our use of MetricGAN over an end-to-end alternative. This approach also allows for extensibility to other non-differentiable speech quality scores such as PESQ.
At test time, we use LeMARA for visual-acoustic matching as follows: given a source audio  $A_{s}^{(q)}$  and target environment image  $V_{t}^{(q)}$ , we apply the trained de-biaser  $G$  followed by the visual reverberator  $R_{v}$ :
$$
f \left(A _ {s} ^ {(q)}, V _ {t} ^ {(q)}\right) = R _ {v} \left(G \left(A _ {s} ^ {(q)}\right), V _ {t} ^ {(q)}\right). \tag {6}
$$
Altogether, our approach adds the room acoustics depicted in  $V_{t}^{(q)}$  to the input audio. In the case that the source audio  $A_{s}^{(q)}$  is known to be anechoic (e.g., a user is using LeMARA to re-synthesize stock anechoic sounds for a new scene), then we simply bypass the de-biaser  $G$  and directly apply  $R_{v}$ .
Figure 4: Datasets. SoundSpaces-Speech (top row) renders panoramic views of people in indoor environments. AVSpeech-Rooms (bottom) contains a wide variety of naturalistic indoor environments with diverse acoustic properties.
We use two datasets: SoundSpaces-Speech [7] and AVSpeech [10]. See Figure 4. For all results, we test only on audio and environments that are not observed during training.
SoundSpaces-Speech SoundSpaces-Speech [7] is an audio-visual dataset created using the SoundSpaces audio simulation platform [4, 6] together with 82 3D home scans from Matterport3D [2] and anechoic speech samples from LibriSpeech [28]. SoundSpaces offers state-of-the-art generation of Room Impulse Responses (RIRs) computed at regular source-listener locations throughout the environment, accounting for all major real-world acoustic phenomena (reverberation, early specular/diffuse reflections, etc.) See [7] for details.
By convolving an anechoic sample with the appropriate RIR, it allows realistic simulation of audio reverberation at arbitrary locations in the pre-scanned real-world Matterport environments. A 3D humanoid is rendered at the source location in the Matterport home. SoundSpaces-Speech consists of anechoic speech samples from LibriSpeech paired with their acoustically-correct reverberated waveform (rendered using SoundSpaces) in any of 82 unique environments, together with an RGB-D image at the listener's position. To train on SoundSpaces-Speech in a self-supervised setting, we discard the source audio and only use the target audio (simulated reverberant audio). We use train/val/test splits of 28,853/280/1,489 samples.
AVSpeech-Rooms AVSpeech [10] is a large-scale video dataset consisting of 3-10 second clips from YouTube videos, most of which feature a single speaker and little background noise. Not all of the clips have naturalistic audio-visual correspondence, due to video editing tricks, microphone locations, virtual backgrounds, etc. Hence, we create a subset of AVSpeech, called AVSpeech-Rooms, that preserves samples with useful information about room geometry and material types (see Supp. for details). A randomly selected frame from the video clip is used as the target image. Our final set consists of 72,615/1,911/1,911 train/val/test samples. See Figure 4 (bottom).
Implementation Details We use the procedure outlined in Sec. 3.4 to train on SoundSpaces-Speech. For training on AVSpeech-Rooms, we proceed directly to step (2), pre-training  $R_{v}$  and  $R_{b}$  on audio that has been de-biased using the fine-tuned SoundSpaces-Speech de-biaser model (instead of an SRMR-optimized model trained on AVSpeech-Rooms). We then proceed to step (3) as in SoundSpaces-Speech. We refer to this setup as "shortcut training" to highlight our use of the SoundSpaces-Speech trained de-biaser to bypass step (1) when training on AVSpeech-Rooms. While AVSpeech can be trained with the full procedure outlined in Sec. 3.4 (see ablations in Supp.), shortcut training allows us the advantage of the strong prior for de-biasing learned by the fine-tuned SoundSpaces de-biaser.
We train LeMARA using the combined acoustic residue metric with  $\alpha = 0.7$ . We train a WaveNet-based dereverberator [36] ("off-the-shelf") on paired SoundSpaces-Speech audio which is "reversed" (reverberant input audio, anechoic target audio). We use this dereverberator for both LeMARA and the ViGAS [5] baseline. Prior VAM work [3] used an audio-visual dereverberator [7] trained on both
Table 1: VAM results on two datasets (unseen environments). RTE standard errors are less than 0.005, and STFT standard errors are 0.031 and 0.64 for the two datasets, respectively.
Train	SoundSpaces-Speech	AVSpeech-Rooms
Test	SoundSpaces-Speech	AVSpeech-Rooms	LibriSpeech
Model	RTE (s)	STFT	logSTFT	RTE (s)	STFT	logSTFT	RTE (s)
Input audio	0.320	1.427	1.274	0.310	1.327	2.107	0.358
AViTAR	0.080	2.471	1.629	0.136	2.894	1.290	0.239
ViGAS	0.108	4.373	1.232	0.109	7.007	0.662	0.254
LeMARA(no vis)	0.152	5.612	1.884	0.137	6.256	0.705	0.223
LeMARA (ours)	0.079	0.690	1.530	0.071	6.298	0.571	0.210
simulated and real-world data to pre-process reverberant audio. For fair evaluation, we train their model with their same original audio-visual dereverberator. $^3$
We adapt our code for the reverberator models and ViGAS from [36].<sup>4</sup>. We train ViGAS with the same hyperparameters and loss as our reverberators during pre-training. Our de-biaser is adapted from the speechbrain MetricGAN-U implementation [34]. Our RT60 estimator is trained on pairs of reverberant samples from a SoTA audio simulator [6] paired with the ground truth RT60 for the RIR used to produce the reverberant sample. We use a Resnet18 [18] model to encode our visual features on RGB images. The last feature map before the final pooling layer is flattened and used as the visual feature conditioner. See Supp. for training details and architecture for these models. We plan to release our code to facilitate further research.
Metrics We rely on three metrics to evaluate quality of VAM models. STFT Error and logSTFT Error compute the MSE loss and log MSE loss, respectively, between the magnitude spectrograms of predicted and target speech. Because we do not have access to the ground truth RIR, we also use an RIR reverberation metric that can be reliably estimated from audio, RT60 Error (RTE), which measures the MSE between RT60 estimates of predicted and target speech. The first two apply only when we have ground truth re-synthesized audio (in simulation), while the third is content-invariant and captures room acoustics signatures for any target.
Seen and unseen evaluation We report results on two different settings: unseen environments, where both the source audio  $A_{s}$  and target image  $V_{t}$  come from the test set; and seen environments, where an audio sample  $A_{s}$  from the test set is paired with a target image  $V_{t}$  from the train set (seen by the model). The unseen environment setting is important for evaluating generalization to novel scenes (e.g. matching an audio sample to an image from the Web), while the seen environment setting is useful for cases in which we already have video recordings, such as in film dubbing.
Baselines As baselines, we compare to state-of-the-art models for audio-visual re-targeting: (1) AViTAR [3], the only prior method that addresses the visual-acoustic matching task. It consists of a Transformer for audio-visual fusion, followed by a generator that synthesizes the reverberant waveform from the audio-visual latent feature. As discussed above, for self-supervised training, the authors use a pre-trained audio-visual dereverberation model [7] to create pseudo-source audio, which is passed as input to the model. (2) ViGAS [5], a model designed for novel-view acoustic synthesis, conditioned on a camera pose. We adopt its Acoustic Synthesis module, a WaveNet model based on [36], for our task. To apply it for VAM, we replace the camera pose with the flattened feature from the ResNet. (3) Non-visual LeMARA. We evaluate LeMARA with the blind reverberator  $R_b$  fine-tuned during training. (4) Input audio. We copy the dereverberated audio to the output. AViTAR and ViGAS are trained with the data augmentation strategy introduced in [3] (see Supp. for details).
Results on SoundSpaces-Speech Tables 1 and 2 report results on SoundSpaces-Speech (left three columns) for the unseen and seen settings, respectively. LeMARA outperforms the baselines on most metrics. Non-visual LeMARA performs significantly worse, indicating LeMARA effectively utilizes visual features during acoustic matching. This demonstrates the success of our acoustic de-biasing, which forces stronger learning from the visual stream. Notably, our model significantly outperforms
<sup>3</sup>https://github.com/facebookresearch/visual-acoustic-matching
<sup>4</sup>https://github.com/facebookresearch/BinauralSpeechSynthesis
Table 2: Results on seen environments. We evaluate VAM using audio samples from the test set paired with target images from the train set.
Model	SoundSpaces-Speech	AVSpeech-Rooms
RTE (s)	STFT	logSTFT	RTE (s)	STFT	logSTFT
AViTAR	0.062	3.186	1.506	0.131	2.852	1.273
ViGAS	0.076	6.386	0.962	0.111	6.946	0.657
LeMARA (ours)	0.060	0.667	1.417	0.067	6.198	0.570
Figure 5: Normalized RTE across reverberation levels (left). We evaluate RTE normalized by target RT60 on AVSpeech-Rooms samples with varying levels of reverberation. LeMARA consistently achieves lowest error at all reverberation levels. RT60 distribution (right). The distribution of RT60 values for audio generated by LeMARA best matches the target distribution.
ViGAS—which shares the same architecture as our reverberator—indicating that our performance improvement over AViTAR can be attributed to our novel training objective, and not simply due to a shift in architecture from Transformer to WaveNet.
Results on AVSpeech-Rooms Tables 1 and 2 show results on AVSpeech-Rooms (right four columns) for both unseen and seen settings, respectively. We further divide the unseen evaluation into two scenarios: (1) Where the source audio and target visual come from the same (unobserved) AVSpeech-Rooms sample and (2) where the source audio comes from LibriSpeech [29], a dataset of anechoic source samples of people reading passages in English (rightmost column). In both cases the target visual environment is a frame from an unseen AVSpeech video.
LeMARA outperforms the baselines in both settings on RTE. We outperform ViGAS in the LibriSpeech scenario despite sharing the same reverberator architecture, highlighting the efficacy of our novel training objective. Figure 5 (right) shows the distribution of RT60 values for audio reverberated by our model (blue), the baselines, and the target RT60 distribution (purple). Our model best matches the target RT60 distribution. LeMARA also outperforms baselines across samples with a diverse range of real-world reverberation intensities. We stratify the test set by ground truth RT60 and evaluate performance on samples within each reverberation bin (Figure 5, left). LeMARA outperforms the baselines across all reverberation levels, demonstrating its robustness to variation in real-world data.
Although our model performs poorly on STFT error, we significantly outperform existing approaches on logSTFT error, which better reflects perceptual quality given the logarithmic nature of human hearing. Furthermore, the naive baseline achieves the lowest STFT error, indicating that dereverberated audio itself strongly resembles reverberant audio even prior to acoustic matching. Models that use this dereverberator without further de-biasing will display artificially low STFT error when evaluated in-dataset (AVSpeech-Rooms  $\rightarrow$  AVSpeech-Rooms). Thus, it is important to balance the in-dataset evaluation with the LibriSpeech generalization case (far right in Table 1, where we soundly outperform Input audio) to gain a complete picture of model performance.
Furthermore, AVSpeech-Rooms contains a variety of non-speech sounds (e.g. air conditioning, clicking/tapping noises from object interactions) which make reverberation even more challenging. These signals may be perceptually weak, but they appear on spectrograms and contribute to the larger
Figure 6: Acoustically matched audio. LeMARA produces audio with more accurate acoustics than AViTAR [3] across diverse acoustic and visual scenes. Shown here for LibriSpeech setting.
STFT error we observe on AVSpeech-Rooms. In contrast, SoundSpaces-Speech contains no artifacts or background noise that could contribute to spectrogram errors.
An ablation study of our model analyzing the impact of varying the target metric during training shows the clear advantage of our residue metric compared to SRMR alone (see Supp.). In particular, training with an SRMR objective alone yields an RTE of 0.2308 on LibriSpeech, whereas training with a pure acoustic residue metric yields an RTE of 0.2156; our combined metric yields an RTE of 0.2123. This reinforces the efficacy of our metric as a training objective: de-biasing is distinct from generic enhancement.
Figure 6 shows examples of our model's generated sounds for different target images, compared to the output of AViTAR [3] (the best performing baseline). We display the RT60 of the source audio, the visual scene's ground truth RT60, and the RT60 of the two method's generated audio. In the majority of cases, our model produces audio that more closely matches the true acoustics of the visual scene. Our model learns to add little reverberation in enclosed environments with soft surfaces (top left), and to add more reverberation in open acoustic environments with hard surfaces (bottom right). We also highlight a failure mode in which our model does not inject proper acoustics (bottom left), likely due to the irregular shape of the room. See our project webpage to listen to examples.
Human perception study We augment these quantitative results with a human subject study, in order to gauge whether listeners perceive our results as successfully retargeting the audio to the pictured environment. We design a survey using 23 images  $V_{t}^{(q)}$  selected from AVSpeech-Rooms that show room geometry and materials clearly, and are representative of a diverse variety of acoustic environments. We couple those with 23 anechoic source samples  $A_{s}^{(q)}$  from LibriSpeech [29]. For each sample, we generate the acoustically matched audio with both LeMARA and AViTAR [3]—the best performing baseline. We anonymize and shuffle the generated audio, and ask 10 subjects with normal hearing to identify which room matches best with the audio among three given rooms, one of which is the true room  $V_{t}^{(q)}$ . Users correctly identified the target room with  $46.1\%$  accuracy on speech generated by LeMARA versus  $34.7\%$  accuracy with speech generated by AViTAR. This shows our model achieves higher quality acoustic matching according to human perception. That said, the subjects' accuracy rates are fairly low in the absolute sense, which suggests both the difficulty of the problem and subtlety of the perception task. Our results have pushed the state of the art, both objectively and subjectively, but there remain challenges to fully mature visual acoustic matching.
We introduced a self-supervised approach to visual acoustic matching. Built on a novel idea for disentangling room acoustics from audio with a GAN-debiaser, our model improves the state of the art on two datasets. Our acoustic residue metric and adversarial training has potential to generalize to other multi-modal learning settings where there is risk of unintentionally silencing a paired modality during training. For example, our framework could be explored for binauralization of mono sounds using video or audio-visual source separation. In future work, we plan to explore generalizations to spatial dynamics that would account for movement of a speaker throughout 3D space.
Acknowledgments: Thanks to Ami Baid for help in data collection. UT Austin is supported in part by the IFML NSF AI Institute. K.G. is paid as a research scientist at Meta.
[35] Chandan K A Reddy, Vishak Gopal, and Ross Cutler. Dnsmos: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors. In *arxiv*, 2021.
[36] Alexander Richard, Dejan Markovic, Israel D. Gebru, Steven Krenn, Gladstone Alexander Butler, Fernando Torre, and Yaser Sheikh. Neural synthesis of binaural speech from mono audio. In International Conference on Learning Representations, 2021.
[37] Julius Richter, Simon Welker, Jean-Marie Lemercier, Bunlong Lay, and Timo Gerkmann. Speech enhancement and dereverberation with diffusion-based generative models. In *arxiv*, 2022.
[38] Antony W. Rix, John G. Beerends, Mike Hollier, and Andries P. Hekstra. Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codescs. 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.01CH37221), 2:749-752 vol.2, 2001.
[39] Manfred R. Schroeder. New method of measuring reverberation time. In The Journal of the Acoustical Society of America 37, 409, 1965.
[40] Nikhil Singh, Jeff Mentch, Jerry Ng, Matthew Beveridge, and Iddo Drori. Image2reverb: Cross-modal reverb impulse response synthesis. In ICCV, 2021.
[41] Christian J. Steinmetz, Vamsi Krishna Ithapu, and Paul Calamia. Filtered noise shaping for time domain room impulse response estimation from reverberant speech. In 2021 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pages 221-225, 2021.
[42] Jiaqi Su, Zeyu Jin, and Adam Finkelstein. Acoustic matching by embedding impulse responses. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 426-430. IEEE, 2020.
[43] Daxin Tan, Guangyan Zhang, and Tan Lee. Environment aware text-to-speech synthesis. In InterSpeech, 2022.
[44] Ke Tan, Yong Xu, Shi-Xiong Zhang, Meng Yu, and Dong Yu. Audio-visual speech separation and dereverberation with a two-stage multimodal network. IEEE Journal of Selected Topics in Signal Processing, 14(3):542-553, mar 2020.
[45] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. In arXiv, 2016.
[46] Zhichao Wang, Qicong Xie, Tao Li, Hongqiang Du, Lei Xie, Pengcheng Zhu, and Mengxiao Bi. One-shot voice conversion for style transfer based on speaker adaptation. In ICASSP, 2022.
[47] Feifei Xiong, Stefan Goetze, Birger Kollmeier, and Bernd T Meyer. Joint estimation of reverberation time and early-to-late reverberation ratio from single-channel speech signals. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(2):255-267, 2018.
[48] Cheng Yu, Ryandhimas E. Zezario, Syu-Siang Wang, Jonathan Sherman, Yi-Yen Hsieh, Xugang Lu, Hsin-Min Wang, and Yu Tsao. Speech enhancement based on denoising autoencoder with multi-branched encoders. In arxiv, 2020.
7 Supplementary
In this supplementary material we provide the following:
1. A video for qualitative evaluation of our model's performance (7.1)
2. Details regarding AVSpeech-Rooms curation (7.2) (referenced in Sec. 4 of main paper)
3. Details on our ablation study with different metric training objectives (7.3) (referenced in Sec. 5 — "Results on AVSpeech-Rooms" of main paper)
4. A comparison of audio quality and reverberation metrics between reverberant, de-reverberated, and de-biased audio (7.4)
5. A sample survey slide from our human perception study (Figure 7)
6. Model/training details for our RT60 estimator, de-biaser, discriminator, and reverberator (7.5) (referenced in Sec. 5 — "Implementation Details" of main paper)
7. Details on our data augmentation strategy (7.6) (referenced in Sec. 5 — "Baselines" of main paper)
8. A brief discussion of our work's limitations and broader impact (7.7 7.8)
9. A tabular description of our three-stage training strategy (Table 8)
10. Pseudocode for a discriminator training epoch detailing our reverberator update mechanism (Algorithm 1)
11. A visualization illustrating the effect of speaker distance on the Direct-to-Reverberant Energy Ratio (DRR) in SoundSpaces-Speech audio samples (Figure 9)
12. A visualization of the effect of speaker distance on STFT Error in SoundSpaces-Speech audio samples (Figure 10)
13. A visualization comparing SRMR scores of de-biased and dereverberated data across various reverberation levels for both AVSpeech-Rooms and SoundSpaces-Speech (Figure 11)
7.1 Supplementary video
Our video contains several illustrative examples generated by LeMARA on both SoundSpaces-Speech and AVSpeech-Rooms. We provide audio generated by the current state-of-the-art (AViTAR) for reference on each example. We recommend wearing headphones for a better listening experience.
7.2 AVSpeech-Rooms
Acoustic AVSpeech consists of audio clips from YouTube videos along with an RGB image frame selected randomly from the corresponding video clip. To create AVSpeech-Rooms, we design a set of criteria which we use to filter out samples in which the image contains uninformative, non-natural, or misleading acoustic information about the space. We focus on cases in which the room is not visible, a microphone is being used, or a virtual background/screen is present — any of which will disturb the natural room acoustics for the speaker's voice. We query each sample with our criteria using a Visual Question Answering (VQA) model [20], which we found more reliable than manual annotations we originally obtained on MTurk. 3 contains information about our criteria.
Table 3: Filtering criteria and % of Acoustic AVSpeech samples removed.
Question	Answer	dataset %
Is a microphone or headset visible in the image?	yes	7.2
Is there a whiteboard/blackboard in the background?	yes	3.4
Is the entire background one solid color and material?	yes	23.4
Is there a large projector screen covering most of the background?	yes	2.2
Is part or all of the background virtual?	yes	1.3
Are there multiple screens in the image?	yes	3.5
Is the wider room clearly visible?	no	3.0
For each slide, listen to the audio clip and choose the image which best corresponds to the acoustic qualities of the audio. The goal is to identify which space the audio was recorded in. Make sure to disregard the speaker(s) in the image and only use the room when making a selection.
Figure 7: Human perception study. The instructions given to the user at the start of the survey (left), and a sample slide from the survey (right). The user is asked to listen to the audio clip, and identify which room image most closely matches its acoustics.
Table 4: Ablation study using different metric training objectives. AR denotes the proposed acoustic residue metric.
Metric	LibriSpeech RTE
SRMR [11]	0.2308
AR	0.2156
AR (combined)	0.2123
AR (combined) w/ shortcut	0.2100
7.3 Ablations