SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Arjun Somayazulu<sup>1</sup> Changan Chen<sup>1</sup> Kristen Grauman<sup>1,2</sup>
$^{1}$ UT Austin  $^{2}$ FAIR, Meta
Acoustic matching aims to re-synthesize an audio clip to sound as if it were recorded in a target acoustic environment. Existing methods assume access to paired training data, where the audio is observed in both the source and target environments, but this limits the diversity of training data or requires the use of simulated data or heuristics to create paired samples. We propose a self-supervised approach to visual acoustic matching where training samples include only the target scene image and audio—without acoustically mismatched source audio for reference. Our approach jointly learns to disentangle room acoustics and resynthesize audio into the target environment, via a conditional GAN framework and a novel metric that quantifies the level of residual acoustic information in the de-biased audio. Training with either in-the-wild web data or simulated data, we demonstrate it outperforms the state-of-the-art on multiple challenging datasets and a wide variety of real-world audio and environments. Project page: https://vision.cs.utexas.edu/projects/ss_vam
The acoustic properties of the audio we hear are strongly influenced by the geometry of the room and the materials that make up its surfaces—large, empty rooms with hard surfaces like concrete and glass lead to longer reverberation times, while smaller, more cluttered rooms with soft materials like curtains and carpets will absorb sound waves quickly and produce audio that sounds dull and anechoic. Human perception exploits this acoustic-visual correspondence, and we rely on perceiving natural-sounding audio that is consistent with our environment as we navigate daily life.
Likewise, this phenomenon is important in virtual environments, such as in AR/VR applications. When one hears audio that is acoustically consistent with the virtual environment they are seeing, their brain can better integrate audio and visual information, leading to a more immersive experience. Conversely, when one hears audio that does not match the expected acoustics of the virtual environment, the perceptual mismatch can be jarring. The problem of audio-visual acoustic correspondence extends well beyond AR/VR applications. Film and media production involve recording audio in diverse spaces, which can be expensive and logistically challenging. Similarly, interior designers and architects face the problem of previewing how a space will sound before it is built.
Today's approaches for modeling acoustic-visual coherence typically assume physical access to the target space [40, 25, 5, 43], which can be impractical or impossible in some cases. In particular, in acoustic matching, the audio captured in one environment is re-synthesized to sound as if it were recorded in another target environment, by matching the statistics (e.g. reverberation time) of audio samples recorded in that target environment [9, 15, 21, 24, 26, 47]. Visual acoustic matching (VAM) instead takes an image of the target space as input, learning to transform the audio to match the likely acoustics in the depicted visual scene [3] (see Figure 1(a)). In both cases, however, learned models ideally have access to paired training data, where each training audio clip is recorded in two different environments. This permits a straightforward supervised learning strategy, since a model can learn to transform one clip (source) to the other (target). See Figure 1(b), left. Unfortunately, this approach
Figure 1: Self-supervised visual acoustic matching. (a) Given source audio and target image $^{1}$ , the goal is to re-synthesize the audio to reflect the acoustics of the target environment. b) Two possible data settings: the paired audio setting (left) observes both the source audio and the audio in the target environment, allowing for supervised training, while the unpaired audio setting (right), observes only the audio in the target environment. Our self-supervised strategy handles the unpaired setting.
puts heavy demands on data collection: large-scale collection of paired data from a variety of diverse environments is typically impractical.
In-the-wild web videos provide us with a large, readily available corpus of diverse acoustic environments and human speakers. However, this data is unpaired—we only observe the sound recorded in the target space, without a second recording in a different environment for reference. Prior work attempts to turn unpaired data into paired data by using an off-the-shelf dereverberator model [7] to produce pseudo-anechoic source recordings from in-the-wild reverberant audio, which are then passed to a visual acoustic matching (VAM) model with the true reverberant audio as the target [3]. While that approach is inspiring, it has a fundamental limitation. The automatic dereverberation process is necessarily imperfect, which means that residual acoustic information indicative of the target environment's acoustics can remain in the (pseudo) source example. In turn, the acoustic matching model trained to produce the target audio can learn to use those residual acoustic cues in the audio—instead of the target space's visual features. When evaluated at test-time on arbitrary source audio and unseen images, the residual acoustic clues exploited during training are no longer available, leading to poor acoustically matched audio outputs.
We propose a self-supervised visual acoustic matching approach that accommodates training with unpaired data from in-the-wild videos (See Figure 1(b), right). Our key insight is a training objective that explicitly removes residual acoustics in the audio, forcing reliance on the visual target. In particular, our approach jointly trains an audio-visual debiaser—which is trained to adversarially minimize residual acoustic information in the dereverberated audio—alongside a reverberator that performs visual acoustic matching. To this end, we develop an acoustic residue metric that quantifies the level of residual acoustic information in a waveform, based on the difference in performance between a) an acoustic matching model that conditions on the target image and b) a model that does not condition on any image. Intuitively, training on audio with low residual acoustic information frees the model from relying on that (unrealistic) information during training, allowing it to instead leverage the necessary visual cues.
We use a time-domain conditional GAN as our debiaser, and continually update the integrated reverberator as the distribution of generated audio shifts during training. Unlike prior work, our approach allows training directly on unpaired videos and speech.2
Our proposed LeMARA model outperforms existing approaches [3, 7, 14] on challenging in-the-wild audio and environments from multiple datasets. Further, to benchmark this task, we introduce a high audio-visual correspondence subset of the AVSpeech [10] video dataset. Though we focus on the task of visual-acoustic matching, our insight potentially has broader implications for other self-supervised multi-modal learning tasks in which one wants to control the impact of a dominating modality.
Room acoustics and spatial sound Audio-visual acoustic matching has limited prior work, though there is growing interest from vision and audio communities. Image2Reverb [40] learns to map an input image to its corresponding Room Impulse Response (RIR)—the transfer function that
<https://rulonco.com/features-modern-auditorium-design/
2We focus on clean human speech in indoor settings given its relevance to many of the applications cited above, and due to the fact that human listeners have strong prior knowledge about how room acoustics should affect speech. However, our model design is not specific to speech and could be applied to other audio sources.
characterizes acoustics at a particular listener/speaker location—which can then be convolved with an arbitrary source waveform. In audio-only approaches, [41, 32, 33] generate RIRs from reverberant audio clips, and [42] uses a WaveNet model to warp source audio into a target space conditioned on a learned embedding of the target audio. Generalizing RIR inference to full 3D environments, Few-Shot RIR [25] and Neural Acoustic Fields [23] sample impulse responses at multiple places in an environment in order to synthesize the RIR at novel locations with a transformer. In related tasks, Novel View Acoustic Synthesis [5] directly synthesizes a source audio at a new camera pose in the room, while other methods binauralize monaural source audio [16, 36]. Unlike our model, which can train from arbitrary images, existing methods require knowledge of the ground truth RIRs [41, 32, 33, 40, 23, 25] or paired source-target audio [5, 16, 36]. Most relevant to our approach, AViTAR [3] uses a cross-modal transformer to re-synthesize the audio; it relies on an off-the-shelf audio-visual dereverberator trained in simulation [7] to produce a pseudo-clean source signal, and suffers from the acoustic residue issue discussed above. Our results illustrate how our model overcomes this shortcoming.
Speech synthesis and enhancement Recent work for speech re-synthesis treats acoustic properties as a "style" which can be disentangled from underlying speech and used to perform acoustic-style matching of audio [27, 31, 30, 46]. However, they require either paired data or speaker embeddings to be learned. Web videos (our target domain) can consist of entirely unique speakers, making it difficult to learn a robust speaker embedding. While environment-aware text-to-speech [43] is applicable even in settings where each speaker in the dataset is unique, the model requires access to paired speech for supervised training. Supervised methods for speech enhancement and dereverberation assume access to paired training data, i.e., anechoic "clean" reference waveforms alongside the altered waveforms [44, 7, 48, 37, 1, 12, 13]. Unsupervised speech enhancement approaches [22, 19, 14] such as MetricGAN-U [14] optimize generic perceptual speech quality scores, such as PESQ [38] (which requires paired samples) and speech-to-reverberation modulation energy (SRMR) [11] (which does not). While we share the objective of relaxing supervision requirements, our overall goal is distinct: rather than optimize a generic quality metric, we aim to retarget the input sound to match a specific (pictured) environment. To achieve that goal, we introduce a novel debiasing objective applicable in a conditional GAN framework.
We present Learning to Match Acoustics by Removing Acoustics, LeMARA, to achieve self-supervised visual acoustic matching (VAM). Let  $A \in \mathcal{A}$  denote an audio waveform and let  $V \in \mathcal{V}$  denote an image frame. During training, we are given  $N$  unlabeled examples of target audio and scenes  $\{(A_t, V_t)\}_{t=1}^N$ , taken from frames and accompanying audio in YouTube videos (cf. Sec. 4 for dataset details). While the data is multi-modal, it is unpaired: it has both audio and visual features, but it lacks a paired sample of the audio in some other source domain (see Fig. 1). With this data, we wish to learn a function  $f(A_s, V_t): \mathcal{A}, \mathcal{V} \to \mathcal{A}$  that takes source audio  $A_s$  (which may or may not be reverberant) and a target image  $V_t$ , and produces the audio re-synthesized to sound like it was recorded in the target scene.
To self-supervise  $f$  using unpaired training data, we would like a model that (1) dereverberates  $A_{t}$  to strip it of its room acoustics, yielding pseudo-source audio  $\hat{A}_{t}^{(s)}$  and then (2) reverberates  $\hat{A}_{t}^{(s)}$  by "adding" in the room acoustics of image  $V_{t}$  to regenerate the target audio:  $f(\hat{A}_{t}^{(s)}, V_{t}) \approx A_{t}$ . A naive joint training of two such modules—a dereverberator and reverberator—would collapse to a trivial solution of  $f$  doing nothing. A better solution would pre-train the dereverberator with a well-trained audio-only model, yet as we show in experiments, this leaves residuals of the target environment in the audio  $\hat{A}_{t}^{(s)}$  that handicap the reverberator at inference time, when such signals will not exist.
The key insight of our approach is to make the dereverberation stage a de-biasing stage that is explicitly trained to remove room acoustics information not available in the target image  $V_{t}$ . This forces reliance on the visual target and helps our model  $f$  generalize to unseen audio and scenes.
Our model has two main components that are trained jointly: (1) a de-biaser model  $G$  responsible for removing room acoustics from the target audio and (2) a visually-guided reverberator  $R$  that injects acoustics from the target environment into the output waveform. We first introduce the de-biaser architecture (Sec. 3.1), followed by our novel acoustic residue metric that is optimized during de-biaser training (Sec. 3.2), and our training strategy that allows joint fine-tuning of the
reverberator and de-biaser (Sec. 3.3). Finally, we present our approach for training self-supervised VAM (Sec. 3.4). Figure 3 overviews our model.
3.1 De-biasing Conditional Generative Adversarial Network
The role of our de-biaser is to dereverberate audio samples in a way that minimizes any residual room acoustics information. To that end, we devise an adversarial de-biaser based on MetricGANs [12, 14]. MetricGANs are similar to conventional generative adversarial networks (GAN) [17]—with a generator that aims to enhance a speech waveform—except the discriminator's job is to mimic the behavior of some target quality function. Our de-biaser module extends the basic MetricGAN, augmenting it with novel acoustic residue metric (Sec. 3.2) and training procedure (Sec. 3.3) that accounts for the evolving distribution of the de-biased audio.
Our GAN consists of a generator  $G$ , a discriminator  $D$ , and an audio quality metric  $\mathcal{M}$ .  $D$  is trained to approximate the quality metric  $\mathcal{M}$ , and  $G$  is trained to maximize the metric score on generated data, using  $D$  as a learned surrogate of the metric function  $\mathcal{M}$ .  $G$  is a conditional generator: given an input waveform, it produces a modified waveform which optimizes the quality metric score.
Let  $\{A_t, V_t\}$  be a dataset of target audio-image pairs, and let  $\mathcal{M}(A_t, V_t) \in [0,1]$  be our quality metric  $\mathcal{M}$  (defined below) that produces a scalar measure of the residual room acoustic information in audio. As in the conventional GAN framework, we alternate between discriminator and generator updates. During an epoch of discriminator training,  $D$  trains to approximate the metric function  $\mathcal{M}$ 's scores on both raw audio  $A_t$  and generated audio  $G(A_t)$ . The discriminator loss function is:
$$
\mathcal {L} _ {D} = \| D (A _ {t}) - \mathcal {M} (A _ {t}, V _ {t}) \| _ {2} + \| D (G (A _ {t})) - \mathcal {M} (G (A _ {t}), V _ {t}) \| _ {2} + \| D (A _ {\text {h i s t}}) - s _ {\text {h i s t}} \| _ {2}, \tag {1}
$$
where the first and second terms incentivize  $D$  to produce score estimates that approximate the metric function when evaluated on raw input audio  $(A_{t})$  and generated audio  $G(A_{t})$ , respectively. Following [12], the third term trains the discriminator on samples from a historical replay buffer  $\{(A_{hist}, s_{hist})\}$ , where  $A_{hist} = G_{prev}(A_{t})$  is a generated sample from a previous epoch, and  $s_{hist} = \mathcal{M}_{prev}(G_{prev}(A_{t}), V_{t})$  is its associated metric score. Training on these historical samples helps improve stability and mitigates issues with catastrophic forgetting in the discriminator.
The generator is trained with an adversarial loss, using the discriminator  $D$  learned from the previous epoch of discriminator training (which depends only on  $A_{t}$ ) as a surrogate of the true metric  $\mathcal{M}$  (which depends on both  $A_{t}$  and  $V_{t}$ ). The generator loss is
$$
\mathcal {L} _ {G} = \| D \left(G \left(A _ {t}\right)\right) - 1 \| _ {2}. \tag {2}
$$
Our metric is normalized to produce scores between 0 and 1 (1 being optimal), so this loss forces  $G$  to generate audio that maximizes the estimated metric score. Next, we introduce our metric  $\mathcal{M}$ .
3.2 Acoustic Residue Metric
Rather than train the de-biaser GAN to optimize a generic speech quality metric [12, 14], we wish to quantify the amount of residual room acoustics information in an audio sample. Hence, we define a metric  $\mathcal{M}$  that allows the downstream reverberator model  $R$  itself to quantify the level of residual acoustic information in the waveform. Specifically, the metric consists of two models trained to perform reverberation on dereverberated speech. Importantly, one model  $R_v$  has been trained to perform VAM (using the target image as conditioner), while the other,  $R_b$ , has been trained to perform blind acoustic matching (without the target image as conditioner). We next define the reverberator modules, and then return to their role in  $\mathcal{M}$ .
Inspired by recent work in time-domain signal modeling [45, 36, 5], we use a WaveNet-like architecture for the reverberators consisting of multiple stacked blocks of 1D convolutional layers, with an optional gated fusion network to inject visual information for  $R_{v}$ . Similar to [5], the reverberators use a sinusoidal activation function followed by two separate 1D conv layers that produce residual and skip connections, the latter being mean pooled and fed to a decoder to produce reverberated audio. We choose this model because it is parameter-efficient, consisting entirely of 1D convolutions, and because it allows time-domain conditional waveform generation. See Sec. 5 for training details.
We use these models to compute the acoustic residue metric. Given input audio  $A$  and image  $V$ , our metric is defined as:
$$
\mathcal {M} (A, V) = \sigma \left(\frac {\left| \mathcal {R T} \left(R _ {b} (A)\right) - \mathcal {R T} \left(A _ {t}\right) \right| - \left| \mathcal {R T} \left(R _ {v} (A , V)\right) - \mathcal {R T} \left(A _ {t}\right) \right|}{\max  \left(0 . 1 , \mathcal {R T} \left(A _ {t}\right)\right)}\right), \tag {3}
$$
where  $A_{t}$  denotes the known target audio and  $\mathcal{RT}$  is a scalar-output function characterizing the general reverberant properties of its input audio, which we define below.
Eqn. 3 quantifies the level of acoustic residue—that is, how much greater the blind reverberator's error is compared to the visually-guided reverberator's error. If de-biasing of  $A$  has gone well, this value will be high. When evaluated on audio that contains a high level of residual acoustic information, the visual features will not provide additional useful information, resulting in similar performance by both visual and blind reverberation models. In other words, if the two errors are similar, visual is not adding much, and there is room acoustic information lingering in the audio  $A$ . This pushes the  $\mathcal{M}$  score to be smaller
(poor quality under the metric). On the other hand, when a waveform contains very little residual acoustic information, the visual features will help the visual model  $R_{v}$  produce a more accurate acoustically matched waveform model with lower error compared to the blind model  $R_{b}$ . This will result in a higher metric score.
Figure 2 demonstrates the effect of de-biasing. Reverberant audio (left) has been imperfectly dereverberated (middle), leaving residual reverberation trails which contain information about the original acoustic environment. De-biased audio (right) removes these residual artifacts. The waveform plots (right) show de-biased audio (green) significantly attenuates the long sound decay present in both reverberant (blue) and dereverberated (orange) audio. This forces the reverberator to learn acoustics from the target image.
For the function  $\mathcal{RT}$  in Eqn. 3, we leverage a classic content-invariant metric for characterizing room acoustics: the Reverberation Time at 60 or "RT60". RT60 is the amount of time after a sound source ceases that it takes for the sound pressure level to reduce by 60 dB—which depends directly on the geometry and materials of the space. For example, no matter the initial direct sound, a big cathedral will yield a high RT60, and a cozy living room will yield a low RT60. While RT60 can be quantified by sensing when one has access to the physical environment, we use a learned estimator for RT60 to allow generalization (see Sec. 5 for details). In Eqn. 3, the normalization by the RT60 of the target audio improves stability of discriminator training. We use the clipping function  $max(0.1,\cdot)$  here to prevent samples with extremely low RT60 from destabilizing training.
Training with this acoustic residue metric allows the downstream reverberator models  $R_v, R_b$  themselves to improve the performance of the de-biaser model  $G$ . Unlike SRMR, DNSMOS [35], or any existing off-the-shelf metric that quantifies general speech quality, our metric directly addresses the problem of residual acoustic information in audio. Although  $G$  may learn a function similar to that of dereverberation, we use the term de-biaser to describe the generator to highlight the novel training objective it is trained against, which distinguishes it from a conventional dereverberation model.
3.3 Joint Training of the De-biaser and Reverberators
At initialization,  $R_v$  and  $R_b$  are trained on a certain distribution of speech. When training the de-biasing GAN with the acoustic residue metric, generated speech can eventually fall out of the distribution on which these reverberators were trained, causing  $\mathcal{M}$  to produce unreliable metric scores that destabilize training. To address this, we introduce a strategy to update  $R_v$  and  $R_b$  during training of the de-biasing GAN. Updating these models ensures that  $\mathcal{M}$  consistently produces reliable acoustic residue scores as the distribution of generated speech shifts over the course of GAN training.
At the start of GAN training, we initialize the target networks  $R_v^t$  and  $R_b^t$  as copies of  $R_v$  and  $R_b$  respectively. During discriminator training, each batch of  $\{(G(A_i), V_i)\}$  samples are passed to  $R_v$  and  $R_b$  to compute metric scores under their current frozen state. This batch is also passed to the
Figure 2: De-biased audio. The de-biaser removes residual acoustic traces in audio, and attenuates long sound decay faster than in dereverberated speech.
Figure 3: LeMARA overview. a) Training procedure. Reverberant audio is first processed with an off-the-shelf dereverberator. It is then input to a de-biaser model which strips acoustics from the audio. The clean audio is passed to the reverberator along with the target image for acoustic matching. b) De-biaser architecture.  $G$  is trained to adversarially maximize the score of the discriminator  $D$ , which learns a surrogate of the acoustic residue metric  $\mathcal{M}$  (Sec. 3.1 and 3.2). c) Acoustic residue metric. Both  $R_v$  and  $R_b$  are continually trained on generated data to ensure that they provide accurate metric scores as the distribution of generated data evolves during training (Sec. 3.3). At test time, we use the trained de-biaser  $G$  and the visual reverberator  $R_v$  to perform VAM.
target networks, which compute the losses
$$
\mathcal {L} _ {\text {v i s u a l}} = \left| \left| R _ {v} ^ {t} (G (A), V) - A \right| \right| _ {2} \tag {4}
$$
$$
\mathcal {L} _ {\text {b l i n d}} = \left| \left| R _ {b} ^ {t} (G (A)) - A \right| \right| _ {2} \tag {5}
$$
and perform an update step. Every  $E$  epochs, the target networks' weights are copied over into the metric networks. This update strategy allows  $R_{v}$  and  $R_{b}$  to be jointly fine-tuned with the de-biaser model  $G$ . Figure 3 overviews the model components and data flow.