Title: Self-Supervised Visual Acoustic Matching
Venue: NeurIPS 2023
Accepted: yes
Keywords: Audio-Visual learning

Visual Acoustic Matching
Abstract: Acoustic matching aims to re-synthesize an audio clip to sound as if it were recorded in a target acoustic environment. Existing methods assume access to paired training data, where the audio is observed in both source and target environments, but this limits the diversity of training data or requires the use of simulated data or heuristics to create paired samples. We propose a self-supervised approach to visual acoustic matching where training samples include only the target scene image and audio---without acoustically mismatched source audio for reference. Our approach jointly learns to disentangle room acoustics and re-synthesize audio into the target environment, via a conditional GAN framework and a novel metric that quantifies the level of residual acoustic information in the de-biased audio. Training with either in-the-wild web data or simulated data, we demonstrate it outperforms the state-of-the-art on multiple challenging datasets and a wide variety of real-world audio and environments.

Self-Supervised Visual Acoustic Matching

Abstract

Acoustic matching aims to re-synthesize an audio clip to sound as if it were recorded in a target acoustic environment. Existing methods assume access to paired training data, where the audio is observed in both the source and target environments, but this limits the diversity of training data or requires the use of simulated data or heuristics to create paired samples. We propose a self-supervised approach to visual acoustic matching where training samples include only the target scene image and audio-without acoustically mismatched source audio for reference. Our approach jointly learns to disentangle room acoustics and resynthesize audio into the target environment, via a conditional GAN framework and a novel metric that quantifies the level of residual acoustic information in the de-biased audio. Training with either in-the-wild web data or simulated data, we demonstrate it outperforms the state-of-the-art on multiple challenging datasets and a wide variety of real-world audio and environments. Project page: https: //vision.cs.utexas.edu/projects/ss_vam

1 Introduction

The acoustic properties of the audio we hear are strongly influenced by the geometry of the room and the materials that make up its surfaces-large, empty rooms with hard surfaces like concrete and glass lead to longer reverberation times, while smaller, more cluttered rooms with soft materials like curtains and carpets will absorb sound waves quickly and produce audio that sounds dull and anechoic. Human perception exploits this acoustic-visual correspondence, and we rely on perceiving natural-sounding audio that is consistent with our environment as we navigate daily life.

Likewise, this phenomenon is important in virtual environments, such as in AR/VR applications. When one hears audio that is acoustically consistent with the virtual environment they are seeing, their brain can better integrate audio and visual information, leading to a more immersive experience. Conversely, when one hears audio that does not match the expected acoustics of the virtual environment, the perceptual mismatch can be jarring. The problem of audio-visual acoustic correspondence extends well beyond AR/VR applications. Film and media production involve recording audio in diverse spaces, which can be expensive and logistically challenging. Similarly, interior designers and architects face the problem of previewing how a space will sound before it is built.

Today's approaches for modeling acoustic-visual coherence typically assume physical access to the target space [40, 25, 5, 43], which can be impractical or impossible in some cases. In particular, in acoustic matching , the audio captured in one environment is re-synthesized to sound as if it were recorded in another target environment, by matching the statistics (e.g. reverberation time) of audio samples recorded in that target environment [9, 15, 21, 24, 26, 47]. Visual acoustic matching (VAM) instead takes an image of the target space as input, learning to transform the audio to match the likely acoustics in the depicted visual scene [3] (see Figure 1(a)). In both cases, however, learned models ideally have access to paired training data, where each training audio clip is recorded in two different environments. This permits a straightforward supervised learning strategy, since a model can learn to transform one clip (source) to the other (target). See Figure 1(b), left. Unfortunately, this approach

a)

b)

Paired

Unpaired

Source

Target

Sourc

Target

"

?

HF

puts heavy demands on data collection: large-scale collection of paired data from a variety of diverse environments is typically impractical.

In-the-wild web videos provide us with a large, readily available corpus of diverse acoustic environments and human speakers. However, this data is unpaired -we only observe the sound recorded in the target space, without a second recording in a different environment for reference. Prior work attempts to turn unpaired data into paired data by using an off-the-shelf dereverberator model [7] to produce pseudo-anechoic source recordings from in-the-wild reverberant audio, which are then passed to a visual acoustic matching (V AM) model with the true reverberant audio as the target [3]. While that approach is inspiring, it has a fundamental limitation. The automatic dereverberation process is necessarily imperfect, which means that residual acoustic information indicative of the target environment's acoustics can remain in the (pseudo) source example . In turn, the acoustic matching model trained to produce the target audio can learn to use those residual acoustic cues in the audio-instead of the target space's visual features. When evaluated at test-time on arbitrary source audio and unseen images, the residual acoustic clues exploited during training are no longer available, leading to poor acoustically matched audio outputs.

We propose a self-supervised visual acoustic matching approach that accommodates training with unpaired data from in-the-wild videos (See Figure 1(b), right). Our key insight is a training objective that explicitly removes residual acoustics in the audio, forcing reliance on the visual target. In particular, our approach jointly trains an audio-visual debiaser -which is trained to adversarially minimize residual acoustic information in the dereverberated audio-alongside a reverberator that performs visual acoustic matching. To this end, we develop an acoustic residue metric that quantifies the level of residual acoustic information in a waveform, based on the difference in performance between a) an acoustic matching model that conditions on the target image and b) a model that does not condition on any image. Intuitively, training on audio with low residual acoustic information frees the model from relying on that (unrealistic) information during training, allowing it to instead leverage the necessary visual cues.

We use a time-domain conditional GAN as our debiaser, and continually update the integrated reverberator as the distribution of generated audio shifts during training. Unlike prior work, our approach allows training directly on unpaired videos and speech. 2

Our proposed LeMARA model outperforms existing approaches [3, 7, 14] on challenging in-the-wild audio and environments from multiple datasets. Further, to benchmark this task, we introduce a high audio-visual correspondence subset of the A VSpeech [10] video dataset. Though we focus on the task of visual-acoustic matching, our insight potentially has broader implications for other self-supervised multi-modal learning tasks in which one wants to control the impact of a dominating modality.

2 Related work

Room acoustics and spatial sound Audio-visual acoustic matching has limited prior work, though there is growing interest from vision and audio communities. Image2Reverb [40] learns to map an input image to its corresponding Room Impulse Response (RIR)-the transfer function that

characterizes acoustics at a particular listener/speaker location-which can then be convolved with an arbitrary source waveform. In audio-only approaches, [41, 32, 33] generate RIRs from reverberant audio clips, and [42] uses a WaveNet model to warp source audio into a target space conditioned on a learned embedding of the target audio. Generalizing RIR inference to full 3D environments, Few-Shot RIR [25] and Neural Acoustic Fields [23] sample impulse responses at multiple places in an environment in order to synthesize the RIR at novel locations with a transformer. In related tasks, Novel View Acoustic Synthesis [5] directly synthesizes a source audio at a new camera pose in the room, while other methods binauralize monaural source audio [16, 36]. Unlike our model, which can train from arbitrary images, existing methods require knowledge of the ground truth RIRs [41, 32, 33, 40, 23, 25] or paired source-target audio [5, 16, 36]. Most relevant to our approach, AViTAR [3] uses a cross-modal transformer to re-synthesize the audio; it relies on an off-the-shelf audio-visual dereverberator trained in simulation [7] to produce a pseudo-clean source signal, and suffers from the acoustic residue issue discussed above. Our results illustrate how our model overcomes this shortcoming.

Speech synthesis and enhancement Recent work for speech re-synthesis treats acoustic properties as a "style" which can be disentangled from underlying speech and used to perform acoustic-style matching of audio [27, 31, 30, 46]. However, they require either paired data or speaker embeddings to be learned. Web videos (our target domain) can consist of entirely unique speakers, making it difficult to learn a robust speaker embedding. While environment-aware text-to-speech [43] is applicable even in settings where each speaker in the dataset is unique, the model requires access to paired speech for supervised training. Supervised methods for speech enhancement and dereverberation assume access to paired training data, i.e., anechoic "clean" reference waveforms alongside the altered waveforms [44, 7, 48, 37, 1, 12, 13]. Unsupervised speech enhancement approaches [22, 19, 14] such as MetricGAN-U [14] optimize generic perceptual speech quality scores, such as PESQ [38] (which requires paired samples) and speech-to-reverberation modulation energy (SRMR) [11] (which does not). While we share the objective of relaxing supervision requirements, our overall goal is distinct: rather than optimize a generic quality metric, we aim to retarget the input sound to match a specific (pictured) environment. To achieve that goal, we introduce a novel debiasing objective applicable in a conditional GAN framework.

3 Approach

We present Le arning to M atch A coustics by R emoving A coustics, LeMARA , to achieve selfsupervised visual acoustic matching (VAM). Let A ∈ A denote an audio waveform and let V ∈ V denote an image frame. During training, we are given N unlabeled examples of target audio and scenes { ( A t , V t ) } N t =1 , taken from frames and accompanying audio in YouTube videos (cf. Sec. 4 for dataset details). While the data is multi-modal, it is unpaired: it has both audio and visual features, but it lacks a paired sample of the audio in some other source domain (see Fig. 1). With this data, we wish to learn a function f ( A s , V t ) : A , V → A that takes source audio A s (which may or may not be reverberant) and a target image V t , and produces the audio re-synthesized to sound like it was recorded in the target scene.

To self-supervise f using unpaired training data, we would like a model that (1) dereverberates A t to strip it of its room acoustics, yielding pseudo-source audio ˆ A ( s ) t and then (2) reverberates ˆ A ( s ) t by "adding" in the room acoustics of image V t to regenerate the target audio: f ( ˆ A ( s ) t , V t ) ≈ A t . A naive joint training of two such modules-a dereverberator and reverberator-would collapse to a trivial solution of f doing nothing. A better solution would pre-train the dereverberator with a well-trained audio-only model, yet as we show in experiments, this leaves residuals of the target environment in the audio ˆ A ( s ) t that handicap the reverberator at inference time, when such signals will not exist.

The key insight of our approach is to make the dereverberation stage a de-biasing stage that is explicitly trained to remove room acoustics information not available in the target image V t . This forces reliance on the visual target and helps our model f generalize to unseen audio and scenes.

Our model has two main components that are trained jointly: (1) a de-biaser model G responsible for removing room acoustics from the target audio and (2) a visually-guided reverberator R that injects acoustics from the target environment into the output waveform. We first introduce the de-biaser architecture (Sec. 3.1), followed by our novel acoustic residue metric that is optimized during de-biaser training (Sec. 3.2), and our training strategy that allows joint fine-tuning of the

reverberator and de-biaser (Sec. 3.3). Finally, we present our approach for training self-supervised VAM (Sec. 3.4). Figure 3 overviews our model.

3.1 De-biasing Conditional Generative Adversarial Network

The role of our de-biaser is to dereverberate audio samples in a way that minimizes any residual room acoustics information. To that end, we devise an adversarial de-biaser based on MetricGANs [12, 14]. MetricGANs are similar to conventional generative adversarial networks (GAN) [17]-with a generator that aims to enhance a speech waveform-except the discriminator's job is to mimic the behavior of some target quality function . Our de-biaser module extends the basic MetricGAN, augmenting it with novel acoustic residue metric (Sec. 3.2) and training procedure (Sec. 3.3) that accounts for the evolving distribution of the de-biased audio.

Our GAN consists of a generator G , a discriminator D , and an audio quality metric M . D is trained to approximate the quality metric M , and G is trained to maximize the metric score on generated data, using D as a learned surrogate of the metric function M . G is a conditional generator: given an input waveform, it produces a modified waveform which optimizes the quality metric score.

Let { A t , V t } be a dataset of target audio-image pairs, and let M ( A t , V t ) ∈ [0 , 1] be our quality metric M (defined below) that produces a scalar measure of the residual room acoustic information in audio. As in the conventional GAN framework, we alternate between discriminator and generator updates. During an epoch of discriminator training, D trains to approximate the metric function M 's scores on both raw audio A t and generated audio G ( A t ) . The discriminator loss function is:

where the first and second terms incentivize D to produce score estimates that approximate the metric function when evaluated on raw input audio ( A t ) and generated audio G ( A t ) , respectively. Following [12], the third term trains the discriminator on samples from a historical replay buffer { ( A hist , s hist ) } , where A hist = G prev ( A t ) is a generated sample from a previous epoch, and s hist = M prev ( G prev ( A t ) , V t ) is its associated metric score. Training on these historical samples helps improve stability and mitigates issues with catastrophic forgetting in the discriminator.

The generator is trained with an adversarial loss, using the discriminator D learned from the previous epoch of discriminator training (which depends only on A t ) as a surrogate of the true metric M (which depends on both A t and V t ). The generator loss is

Our metric is normalized to produce scores between 0 and 1 (1 being optimal), so this loss forces G to generate audio that maximizes the estimated metric score. Next, we introduce our metric M .

3.2 Acoustic Residue Metric

Rather than train the de-biaser GAN to optimize a generic speech quality metric [12, 14], we wish to quantify the amount of residual room acoustics information in an audio sample. Hence, we define a metric M that allows the downstream reverberator model R itself to quantify the level of residual acoustic information in the waveform. Specifically, the metric consists of two models trained to perform reverberation on dereverberated speech. Importantly, one model R v has been trained to perform VAM (using the target image as conditioner), while the other, R b , has been trained to perform blind acoustic matching (without the target image as conditioner). We next define the reverberator modules, and then return to their role in M .

Inspired by recent work in time-domain signal modeling [45, 36, 5], we use a WaveNet-like architecture for the reverberators consisting of multiple stacked blocks of 1D convolutional layers, with an optional gated fusion network to inject visual information for R v . Similar to [5], the reverberators use a sinusoidal activation function followed by two separate 1D conv layers that produce residual and skip connections, the latter being mean pooled and fed to a decoder to produce reverberated audio. We choose this model because it is parameter-efficient, consisting entirely of 1D convolutions, and because it allows time-domain conditional waveform generation. See Sec. 5 for training details.

We use these models to compute the acoustic residue metric. Given input audio A and image V , our metric is defined as:

where A t denotes the known target audio and RT is a scalar-output function characterizing the general reverberant properties of its input audio, which we define below.

Eqn. 3 quantifies the level of acoustic residue-that is, how much greater the blind reverberator's error is compared to the visually-guided reverberator's error. If de-biasing of A has gone well, this value will be high. When evaluated on audio that contains a high level of residual acoustic information, the visual features will not provide additional useful information, resulting in similar performance by both visual and blind reverberation models. In other words, if the two errors are similar, visual is not adding much, and there is room acoustic information lingering in the audio A . This pushes the M score to be smaller

Reverberant (  )

Dereverberated (  )

De-biased (  )

Waveform

(poor quality under the metric). On the other hand, when a waveform contains very little residual acoustic information, the visual features will help the visual model R v produce a more accurate acoustically matched waveform model with lower error compared to the blind model R b . This will result in a higher metric score.

Figure 2 demonstrates the effect of de-biasing. Reverberant audio (left) has been imperfectly dereverberated (middle), leaving residual reverberation trails which contain information about the original acoustic environment. De-biased audio (right) removes these residual artifacts. The waveform plots (right) show de-biased audio (green) significantly attenuates the long sound decay present in both reverberant (blue) and dereverberated (orange) audio. This forces the reverberator to learn acoustics from the target image.

For the function RT in Eqn. 3, we leverage a classic content-invariant metric for characterizing room acoustics: the Reverberation Time at 60 or 'RT60". RT60 is the amount of time after a sound source ceases that it takes for the sound pressure level to reduce by 60 dB-which depends directly on the geometry and materials of the space. For example, no matter the initial direct sound, a big cathedral will yield a high RT60, and a cozy living room will yield a low RT60. While RT60 can be quantified by sensing when one has access to the physical environment, we use a learned estimator for RT60 to allow generalization (see Sec. 5 for details). In Eqn. 3, the normalization by the RT60 of the target audio improves stability of discriminator training. We use the clipping function max (0 . 1 , · ) here to prevent samples with extremely low RT60 from destabilizing training.

Training with this acoustic residue metric allows the downstream reverberator models R v , R b themselves to improve the performance of the de-biaser model G . Unlike SRMR, DNSMOS [35], or any existing off-the-shelf metric that quantifies general speech quality, our metric directly addresses the problem of residual acoustic information in audio. Although G may learn a function similar to that of dereverberation, we use the term de-biaser to describe the generator to highlight the novel training objective it is trained against, which distinguishes it from a conventional dereverberation model.

3.3 Joint Training of the De-biaser and Reverberators

At initialization, R v and R b are trained on a certain distribution of speech. When training the de-biasing GAN with the acoustic residue metric, generated speech can eventually fall out of the distribution on which these reverberators were trained, causing M to produce unreliable metric scores that destabilize training. To address this, we introduce a strategy to update R v and R b during training of the de-biasing GAN. Updating these models ensures that M consistently produces reliable acoustic residue scores as the distribution of generated speech shifts over the course of GAN training.

At the start of GAN training , we initialize the target networks R t v and R t b as copies of R v and R b respectively. During discriminator training, each batch of { ( G ( A i ) , V i ) } samples are passed to R v and R b to compute metric scores under their current frozen state. This batch is also passed to the

Reverberant

Audio

At

Dereverberator

De-biaser

Reverberator

(G)

(Rv)

Metric

V

M(A)

(M)

中

三

MSE Loss

RGB Image

Reverberant or

Discriminator

+D(A)

Alternately i cleaned audio

(D)

Vt

updated-

(A)

Alternately

updated

Rv

Av

Generator

Discriminator

Loss

(G)

Acoustically-

(D)

A

M(A)

Dereverberated

cleaned audio

Audio

D(G(At)）<

Target

Rb

Ab

<>At

MSE

Score

Loss

Loss

target networks, which compute the losses

and perform an update step. Every E epochs, the target networks' weights are copied over into the metric networks. This update strategy allows R v and R b to be jointly fine-tuned with the de-biaser model G . Figure 3 overviews the model components and data flow.

3.4 Training and Inference

Training proceeds in three steps. (1) First we pretrain the de-biaser G . This entails pretraining a MetricGAN-U [14] with the Speech-to-Energy-Modulation-Ratio (SRMR) metric [11] on speech pre-processed with our off-the-shelf dereverberator (see Sec. 5). By refining the dereverberated output with the MetricGAN-U, we improve its quality and intelligibility without introducing additional supervision requirements. (2) Second, we pretrain the reverberators that perform (visual) acoustic matching. Specifically, we train R v and R b with L visual and L blind , respectively, using the dereverberated and SRMR-optimized outputs from the MetricGAN in step (1). (3) Finally, we jointly fine-tune both the de-biaser G and reverberators, using the acoustic residue metric (Eqn. 3) for the GAN metric M , the generator and discriminator losses given in Eqns. 2, and 1 together with our alternating training scheme defined in Sec. 3.3. To improve stability in training, since the discriminator D starts with a good approximation of the SRMR metric, we continue training in step 3 using a weighted combination of SRMR and our residue metric: α SRMR ( A ) + (1 -α ) M ( A,V ) . Although the acoustic residue is approximately differentiable, the implementation of SRMR we use is not, motivating our use of MetricGAN over an end-to-end alternative. This approach also allows for extensibility to other non-differentiable speech quality scores such as PESQ.

At test time, we use LeMARA for visual-acoustic matching as follows: given a source audio A ( q ) s and target environment image V ( q ) t , we apply the trained de-biaser G followed by the visual reverberator R v :

Altogether, our approach adds the room acoustics depicted in V ( q ) t to the input audio. In the case that the source audio A ( q ) s is known to be anechoic (e.g., a user is using LeMARA to re-synthesize stock anechoic sounds for a new scene), then we simply bypass the de-biaser G and directly apply R v .

Lowreverb

High reverb

4 Datasets

We use two datasets: SoundSpaces-Speech [7] and AVSpeech [10]. See Figure 4. For all results, we test only on audio and environments that are not observed during training.

SoundSpaces-Speech SoundSpaces-Speech [7] is an audio-visual dataset created using the SoundSpaces audio simulation platform [4, 6] together with 82 3D home scans from Matterport3D [2] and anechoic speech samples from LibriSpeech [28]. SoundSpaces offers state-of-the-art generation of Room Impulse Responses (RIRs) computed at regular source-listener locations throughout the environment, accounting for all major real-world acoustic phenomena (reverberation, early specular/diffuse reflections, etc.) See [7] for details.

By convolving an anechoic sample with the appropriate RIR, it allows realistic simulation of audio reverberation at arbitrary locations in the pre-scanned real-world Matterport environments. A 3D humanoid is rendered at the source location in the Matterport home. SoundSpaces-Speech consists of anechoic speech samples from LibriSpeech paired with their acoustically-correct reverberated waveform (rendered using SoundSpaces) in any of 82 unique environments, together with an RGBD image at the listener's position. To train on SoundSpaces-Speech in a self-supervised setting, we discard the source audio and only use the target audio (simulated reverberant audio). We use train/val/test splits of 28,853/280/1,489 samples.

AVSpeech-Rooms AVSpeech [10] is a large-scale video dataset consisting of 3-10 second clips from YouTube videos, most of which feature a single speaker and little background noise. Not all of the clips have naturalistic audio-visual correpondence, due to video editing tricks, microphone locations, virtual backgrounds, etc. Hence, we create a subset of A VSpeech, called A VSpeech-Rooms, that preserves samples with useful information about room geometry and material types (see Supp. for details). A randomly selected frame from the video clip is used as the target image. Our final set consists of 72,615/1,911/1,911 train/val/test samples. See Figure 4 (bottom).

5 Experiments

Implementation Details We use the procedure outlined in Sec. 3.4 to train on SoundSpacesSpeech. For training on AVSpeech-Rooms, we proceed directly to step (2), pre-training R v and R b on audio that has been de-biased using the fine-tuned SoundSpaces-Speech de-biaser model (instead of an SRMR-optimized model trained on AVSpeech-Rooms). We then proceed to step (3) as in SoundSpaces-Speech. We refer to this setup as "shortcut training" to highlight our use of the SoundSpaces-Speech trained de-biaser to bypass step (1) when training on AVSpeech-Rooms. While AVSpeech can be trained with the full procedure outlined in Sec. 3.4 (see ablations in Supp.), shortcut training allows us the advantage of the strong prior for de-biasing learned by the fine-tuned SoundSpaces de-biaser.

We train LeMARA using the combined acoustic residue metric with α = 0 . 7 . We train a WaveNetbased dereverberator [36] ("off-the-shelf") on paired SoundSpaces-Speech audio which is "reversed" (reverberant input audio, anechoic target audio). We use this dereverberator for both LeMARA and the ViGAS [5] baseline. Prior VAM work [3] used an audio-visual dereverberator [7] trained on both

simulated and real-world data to pre-process reverberant audio. For fair evaluation, we train their model with their same original audio-visual dereverberator. 3

We adapt our code for the reverberator models and ViGAS from [36]. 4 . We train ViGAS with the same hyperparameters and loss as our reverberators during pre-training. Our de-biaser is adapted from the speechbrain MetricGAN-U implementation [34]. Our RT60 estimator is trained on pairs of reverberant samples from a SoTA audio simulator [6] paired with the ground truth RT60 for the RIR used to produce the reverberant sample. We use a Resnet18 [18] model to encode our visual features on RGB images. The last feature map before the final pooling layer is flattened and used as the visual feature conditioner. See Supp. for training details and architecture for these models. We plan to release our code to facilitate further research.

Metrics We rely on three metrics to evaluate quality of V AM models. STFT Error and logSTFT Error compute the MSE loss and log MSE loss, respectively, between the magnitude spectrograms of predicted and target speech. Because we do not have access to the ground truth RIR, we also use an RIR reverberation metric that can be reliably estimated from audio, RT60 Error (RTE) , which measures the MSE between RT60 estimates of predicted and target speech. The first two apply only when we have ground truth re-synthesized audio (in simulation), while the third is content-invariant and captures room acoustics signatures for any target.

Seen and unseen evaluation We report results on two different settings: unseen environments , where both the source audio A s and target image V t come from the test set; and seen environments , where an audio sample A s from the test set is paired with a target image V t from the train set (seen by the model). The unseen environment setting is important for evaluating generalization to novel scenes (e.g. matching an audio sample to an image from the Web), while the seen environment setting is useful for cases in which we already have video recordings, such as in film dubbing.

Baselines As baselines, we compare to state-of-the-art models for audio-visual re-targeting: (1) AViTAR [3], the only prior method that addresses the visual-acoustic matching task. It consists of a Transformer for audio-visual fusion, followed by a generator that synthesizes the reverberant waveform from the audio-visual latent feature. As discussed above, for self-supervised training, the authors use a pre-trained audio-visual dereverberation model [7] to create pseudo-source audio, which is passed as input to the model. (2) ViGAS [5], a model designed for novel-view acoustic synthesis, conditioned on a camera pose. We adopt its Acoustic Synthesis module, a WaveNet model based on [36], for our task. To apply it for V AM, we replace the camera pose with the flattened feature from the ResNet. (3) Non-visual LeMARA . We evaluate LeMARA with the blind reverberator R b fine-tuned during training. (4) Input audio . We copy the dereverberated audio to the output. A ViTAR and ViGAS are trained with the data augmentation strategy introduced in [3] (see Supp. for details).

Results on SoundSpaces-Speech Tables 1 and 2 report results on SoundSpaces-Speech (left three columns) for the unseen and seen settings, respectively. LeMARA outperforms the baselines on most metrics. Non-visual LeMARA performs significantly worse, indicating LeMARA effectively utilizes visual features during acoustic matching. This demonstrates the success of our acoustic de-biasing, which forces stronger learning from the visual stream. Notably, our model significantly outperforms

0.8

0.4

0.6

RT60

1.0

1.2

0.2

0.0

RT60 Distribution

Normalized RTE vs Target RT60

AViTAR

ViGAS

1.0

LeMARA

Normalized RTE

0.8

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.8

1.0

TargetRT60

Source (LibriSpeech)

AViTAR

ViGAS

LeMARA

Target

ViGAS-which shares the same architecture as our reverberator -indicating that our performance improvement over AViTAR can be attributed to our novel training objective, and not simply due to a shift in architecture from Transformer to WaveNet.

Results on AVSpeech-Rooms Tables 1 and 2 show results on AVSpeech-Rooms (right four columns) for both unseen and seen settings, respectively. We further divide the unseen evaluation into two scenarios: (1) Where the source audio and target visual come from the same (unobserved) AVSpeech-Rooms sample and (2) where the source audio comes from LibriSpeech [29], a dataset of anechoic source samples of people reading passages in English (rightmost column). In both cases the target visual environment is a frame from an unseen AVSpeech video.

LeMARA outperforms the baselines in both settings on RTE. We outperform ViGAS in the LibriSpeech scenario despite sharing the same reverberator architecture, highlighting the efficacy of our novel training objective. Figure 5 (right) shows the distribution of RT60 values for audio reverberated by our model (blue), the baselines, and the target RT60 distribution (purple). Our model best matches the target RT60 distribution. LeMARA also outperforms baselines across samples with a diverse range of real-world reverberation intensities. We stratify the test set by ground truth RT60 and evaluate performance on samples within each reverberation bin (Figure 5, left). LeMARA outperforms the baselines across all reverberation levels, demonstrating its robustness to variation in real-world data.

Although our model performs poorly on STFT error, we significantly outperform existing approaches on logSTFT error, which better reflects perceptual quality given the logarithmic nature of human hearing. Furthermore, the naive baseline achieves the lowest STFT error, indicating that dereverberated audio itself strongly resembles reverberant audio even prior to acoustic matching. Models that use this dereverberator without further de-biasing will display artificially low STFT error when evaluated in-dataset (A VSpeech-Rooms → AVSpeech-Rooms). Thus, it is important to balance the in-dataset evaluation with the LibriSpeech generalization case (far right in Table 1, where we soundly outperform Input audio) to gain a complete picture of model performance.

Furthermore, AVSpeech-Rooms contains a variety of non-speech sounds (e.g. air conditioning, clicking/tapping noises from object interactions) which make reverberation even more challenging. These signals may be perceptually weak, but they appear on spectrograms and contribute to the larger

Source

Target Image

AViTAR

LeMARA

Source

Target Image

AViTAR

LeMARA

RT60=0.0178

RT60 =0.0932

RT60 =0.1619

RT60=0.0236

RT60 =0.3096

RT60=0.1054

RT60=0.2748

RT60=0.0081

RT60=0.3698

RT60 =0.1272

RT60=0.0808

RT60 = 0.5129

RT60 = 0.1199

RT60=0.2947

STFT error we observe on AVSpeech-Rooms. In contrast, SoundSpaces-Speech contains no artifacts or background noise that could contribute to spectrogram errors.

An ablation study of our model analyzing the impact of varying the target metric during training shows the clear advantage of our residue metric compared to SRMR alone (see Supp.). In particular, training with an SRMR objective alone yields an RTE of 0.2308 on LibriSpeech, whereas training with a pure acoustic residue metric yields an RTE of 0.2156; our combined metric yields an RTE of 0.2123. This reinforces the efficacy of our metric as a training objective: de-biasing is distinct from generic enhancement.

Figure 6 shows examples of our model's generated sounds for different target images, compared to the output of AViTAR [3] (the best performing baseline). We display the RT60 of the source audio, the visual scene's ground truth RT60, and the RT60 of the two method's generated audio. In the majority of cases, our model produces audio that more closely matches the true acoustics of the visual scene. Our model learns to add little reverberation in enclosed environments with soft surfaces (top left), and to add more reverberation in open acoustic environments with hard surfaces (bottom right). We also highlight a failure mode in which our model does not inject proper acoustics (bottom left), likely due to the irregular shape of the room. See our project webpage to listen to examples.

Human perception study We augment these quantitative results with a human subject study, in order to gauge whether listeners perceive our results as successfully retargeting the audio to the pictured environment. We design a survey using 23 images V ( q ) t selected from AVSpeech-Rooms that show room geometry and materials clearly, and are representative of a diverse variety of acoustic environments. We couple those with 23 anechoic source samples A ( q ) s from LibriSpeech [29]. For each sample, we generate the acoustically matched audio with both LeMARA and AViTAR [3]-the best performing baseline. We anonymize and shuffle the generated audio, and ask 10 subjects with normal hearing to identify which room matches best with the audio among three given rooms, one of which is the true room V ( q ) t . Users correctly identified the target room with 46.1% accuracy on speech generated by LeMARA versus 34.7% accuracy with speech generated by AViTAR. This shows our model achieves higher quality acoustic matching according to human perception. That said, the subjects' accuracy rates are fairly low in the absolute sense, which suggests both the difficulty of the problem and subtlety of the perception task. Our results have pushed the state of the art, both objectively and subjectively, but there remain challenges to fully mature visual acoustic matching.

6 Conclusion

We introduced a self-supervised approach to visual acoustic matching. Built on a novel idea for disentangling room acoustics from audio with a GAN-debiaser, our model improves the state of the art on two datasets. Our acoustic residue metric and adversarial training has potential to generalize to other multi-modal learning settings where there is risk of unintentionally silencing a paired modality during training. For example, our framework could be explored for binauralization of mono sounds using video or audio-visual source separation. In future work, we plan to explore generalizations to spatial dynamics that would account for movement of a speaker throughout 3D space.

Acknowledgments: Thanks to Ami Baid for help in data collection. UT Austin is supported in part by the IFML NSF AI Institute. K.G. is paid as a research scientist at Meta.

References

Deepak Baby and Sarah Verhulst. Sergan: Speech enhancement using relativistic generative adversarial networks with gradient penalty. In ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 106-110, 2019.

Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Nießner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. In 3DV , 2017.

Changan Chen, Ruohan Gao, Paul Calamia, and Kristen Grauman. Visual acoustic matching. In CVPR , 2022.

Changan Chen, Unnat Jain, Carl Schissler, Sebastia Vicenc Amengual Gari, Ziad Al-Halah, Vamsi Krishna Ithapu, Philip Robinson, and Kristen Grauman. SoundSpaces: Audio-visual navigation in 3d environments. In ECCV , 2020.

Changan Chen, Alexander Richard, Roman Shapovalov, Vamsi Krishna Ithapu, Natalia Neverova, Kristen Grauman, and Andrea Vedaldi. Novel-view acoustic synthesis. In CVPR , 2023.

Changan Chen, Carl Schissler, Sanchit Garg, Philip Kobernik, Alexander Clegg, Paul Calamia, Dhruv Batra, Philip Robinson, and Kristen Grauman. Soundspaces 2.0: A simulation platform for visual-acoustic learning. In NeurIPS , 2022.

Changan Chen, Wei Sun, David Harwath, and Kristen Grauman. Learning audio-visual dereverberation. In ICASSP , 2023.

Sasha Devore and Bertrand Delgutte. Effects of Reverberation on the Directional Sensitivity of Auditory Neurons across the Tonotopic Axis: Influences of Interaural Time and Level Differences. Journal of Neuroscience , 30(23):7826-7837, 2010. Publisher: Society for Neuroscience _eprint: https://www.jneurosci.org/content/30/23/7826.full.pdf.

James Eaton, Nikolay Gaubitch, Allistair Moore, and Patrick Naylor. Estimation of room acoustic parameters: The ACE challenge. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 24(10), 2016.

Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T. Freeman, and Michael Rubinstein. Looking to listen at the cocktail party. ACM Transactions on Graphics , 37(4):1-11, jul 2018.

Tiago H. Falk, Chenxi Zheng, and Wai-Yip Chan. A non-intrusive quality and intelligibility measure of reverberant and dereverberated speech. IEEE Transactions on Audio, Speech, and Language Processing , 18(7):1766-1774, 2010.

Szu-Wei Fu, Chien-Feng Liao, Yu Tsao, and Shou-De Lin. MetricGAN: Generative adversarial networks based black-box metric scores optimization for speech enhancement. In ICML , 2019.

Szu-Wei Fu, Cheng Yu, Tsun-An Hsieh, Peter Plantinga, Mirco Ravanelli, Xugang Lu, and Yu Tsao. Metricgan+: An improved version of metricgan for speech enhancement. In InterSpeech , 2021.

Szu-Wei Fu, Cheng Yu, Kuo-Hsuan Hung, Mirco Ravanelli, and Yu Tsao. Metricgan-u: Unsupervised speech enhancement/ dereverberation based only on noisy/ reverberated speech. In arxiv , 2021.

Hannes Gamper and Ivan J Tashev. Blind reverberation time estimation using a convolutional neural network. In 2018 16th International Workshop on Acoustic Signal Enhancement (IWAENC) , pages 136-140, 2018.

Ruohan Gao and Kristen Grauman. 2.5d visual sound. In CVPR , 2019.

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. In NeurIPS , 2014.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , pages 770-778, 2016.

Bryce Irvin, Marko Stamenovic, Mikolaj Kegler, and Li-Chia Yang. Self-supervised learning for speech enhancement through synthesis. In arxiv , 2022.

Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without convolution or region supervision, 2021.

Florian Klein, Annika Neidhardt, and Marius Seipel. Real-time estimation of reverberation time for selection of suitable binaural room impulse responses. In Audio for Virtual, Augmented and Mixed Realities: Proceedings of 5th International Conference on Spatial Audio (ICSA 2019) , pages 145-150, 2019.

Hsin-Yi Lin, Huan-Hsin Tseng, Xugang Lu, and Yu Tsao. Unsupervised noise adaptive speech enhancement by discriminator-constrained optimal transport. In NeurIPS , 2021.

Andrew Luo, Yilun Du, Michael J. Tarr, Joshua B. Tenenbaum, Antonio Torralba, and Chuang Gan. Learning neural acoustic fields. In NeurIPS , 2022.

Wolfgang Mack, Shuwen Deng, and Emanuël AP Habets. Single-channel blind direct-toreverberation ratio estimation using masking. In INTERSPEECH , pages 5066-5070, 2020.

Sagnik Majumder, Changan Chen, Ziad Al-Halah, and Kristen Grauman. Few-shot audio-visual learning of environment acoustics. In NeurIPS , 2022.

Prateek Murgai, Mark Rau, and Jean-Marc Jot. Blind estimation of the reverberation fingerprint of unknown acoustic environments. In Audio Engineering Society Convention 143 . Audio Engineering Society, 2017.

Ahmed Omran, Neil Zeghidour, Zalán Borsos, Félix de Chaumont Quitry, Malcolm Slaney, and Marco Tagliasacchi. Disentangling speech from surroundings in a neural audio codec. In arxiv , 2022.

Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An asr corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 5206-5210, 2015.

Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP) , pages 5206-5210. IEEE, 2015.

Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Abdelrahman Mohamed, and Emmanuel Dupoux. Speech resynthesis from discrete disentangled self-supervised representations. In InterSpeech , 2021.

Kaizhi Qian, Yang Zhang, Shiyu Chang, David Cox, and Mark Hasegawa-Johnson. Unsupervised speech decomposition via triple information bottleneck. In arxiv , 2021.

Anton Ratnarajah, Zhenyu Tang, and Dinesh Manocha. IR-GAN: Room Impulse Response Generator for Far-Field Speech Recognition. In Proc. Interspeech 2021 , pages 286-290, 2021.

Anton Ratnarajah, Zhenyu Tang, and Dinesh Manocha. Ts-rir: Translated synthetic room impulse responses for speech augmentation. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) , pages 259-266, 2021.

Mirco Ravanelli, Titouan Parcollet, Aku Rouhe, Peter Plantinga, Elena Rastorgueva, Loren Lugosch, Nauman Dawalatabad, Chou Ju-Chieh, Abdel Heba, Francois Grondin, William Aris, Chien-Feng Liao, Samuele Cornell, Sung-Lin Yeh, Hwidong Na, Yan Gao, Szu-Wei Fu, Cem Subakan, Renato De Mori, and Yoshua Bengio. Speechbrain. https://github.com/ speechbrain/speechbrain , 2021.

Chandan K A Reddy, Vishak Gopal, and Ross Cutler. Dnsmos: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors. In arxiv , 2021.

Alexander Richard, Dejan Markovic, Israel D. Gebru, Steven Krenn, Gladstone Alexander Butler, Fernando Torre, and Yaser Sheikh. Neural synthesis of binaural speech from mono audio. In International Conference on Learning Representations , 2021.

Julius Richter, Simon Welker, Jean-Marie Lemercier, Bunlong Lay, and Timo Gerkmann. Speech enhancement and dereverberation with diffusion-based generative models. In arxiv , 2022.

Antony W. Rix, John G. Beerends, Mike Hollier, and Andries P. Hekstra. Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs. 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.01CH37221) , 2:749-752 vol.2, 2001.

Manfred R. Schroeder. New method of measuring reverberation time. In The Journal of the Acoustical Society of America 37, 409 , 1965.

Nikhil Singh, Jeff Mentch, Jerry Ng, Matthew Beveridge, and Iddo Drori. Image2reverb: Cross-modal reverb impulse response synthesis. In ICCV , 2021.

Christian J. Steinmetz, Vamsi Krishna Ithapu, and Paul Calamia. Filtered noise shaping for time domain room impulse response estimation from reverberant speech. In 2021 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) , pages 221-225, 2021.

Jiaqi Su, Zeyu Jin, and Adam Finkelstein. Acoustic matching by embedding impulse responses. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 426-430. IEEE, 2020.

Daxin Tan, Guangyan Zhang, and Tan Lee. Environment aware text-to-speech synthesis. In InterSpeech , 2022.

Ke Tan, Yong Xu, Shi-Xiong Zhang, Meng Yu, and Dong Yu. Audio-visual speech separation and dereverberation with a two-stage multimodal network. IEEE Journal of Selected Topics in Signal Processing , 14(3):542-553, mar 2020.

Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. In arXiv , 2016.

Zhichao Wang, Qicong Xie, Tao Li, Hongqiang Du, Lei Xie, Pengcheng Zhu, and Mengxiao Bi. One-shot voice conversion for style transfer based on speaker adaptation. In ICASSP , 2022.

Feifei Xiong, Stefan Goetze, Birger Kollmeier, and Bernd T Meyer. Joint estimation of reverberation time and early-to-late reverberation ratio from single-channel speech signals. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 27(2):255-267, 2018.

Cheng Yu, Ryandhimas E. Zezario, Syu-Siang Wang, Jonathan Sherman, Yi-Yen Hsieh, Xugang Lu, Hsin-Min Wang, and Yu Tsao. Speech enhancement based on denoising autoencoder with multi-branched encoders. In arxiv , 2020.

7 Supplementary

In this supplementary material we provide the following:

A video for qualitative evaluation of our model's performance (7.1)

Details regarding AVSpeech-Rooms curation (7.2) (referenced in Sec. 4 of main paper)

Details on our ablation study with different metric training objectives (7.3) (referenced in Sec. 5 - "Results on AVSpeech-Rooms" of main paper)

A comparison of audio quality and reverberation metrics between reverberant, dereverberated, and de-biased audio (7.4)

A sample survey slide from our human perception study (Figure 7)

Model/training details for our RT60 estimator, de-biaser, discriminator, and reverberator (7.5) (referenced in Sec. 5 - "Implementation Details" of main paper)

Details on our data augmentation strategy (7.6) (referenced in Sec. 5 - "Baselines" of main paper)

A brief discussion of our work's limitations and broader impact (7.7 7.8)

A tabular description of our three-stage training strategy (Table 8)

Pseudocode for a discriminator training epoch detailing our reverberator update mechanism (Algorithm 1)

Avisualization illustrating the effect of speaker distance on the Direct-to-Reverberant Energy Ratio (DRR) in SoundSpaces-Speech audio samples (Figure 9)

A visualization of the effect of speaker distance on STFT Error in SoundSpaces-Speech audio samples (Figure 10)

Avisualization comparing SRMR scores of de-biased and dereverberated data across various reverberation levels for both A VSpeech-Rooms and SoundSpaces-Speech (Figure 11)

7.1 Supplementary video

Our video contains several illustrative examples generated by LeMARA on both SoundSpaces-Speech and AVSpeech-Rooms. We provide audio generated by the current state-of-the-art (AViTAR) for reference on each example. We recommend wearing headphones for a better listening experience.

7.2 AVSpeech-Rooms

Acoustic AVSpeech consists of audio clips from YouTube videos along with an RGB image frame selected randomly from the corresponding video clip. To create A VSpeech-Rooms, we design a set of criteria which we use to filter out samples in which the image contains uninformative, non-natural, or misleading acoustic information about the space. We focus on cases in which the room is not visible, a microphone is being used, or a virtual background/screen is present - any of which will disturb the natural room acoustics for the speaker's voice. We query each sample with our criteria using a Visual Question Answering (VQA) model [20], which we found more reliable than manual annotations we originally obtained on MTurk. 3 contains information about our criteria.

D

Foreachslide,listentothe audioclipand choosetheimage whichbestcorrespondstotheacousticqualitiesofthe audio. The goal is to identifywhichspace the audiowas recorded in. Makesuretodisregard thespeaker(s)intheimageandonlyuse theroomwhen making a selection.

2

3

GreatGrappling.com

49

7.3 Ablations

Table 4 displays our experiments with different self-supervised training objectives. We report performance on the LibriSpeech evaluation setting. The first three rows correspond to experiments in which we do not utilize the shorcut training strategy (referenced in Sec. 3 - "Training" of main paper). Using SRMR alone (row 1) produces the largest (worst) RTE. Training with the acoustic residue metric instead (row 2) leads to a large improvement in RTE, providing empirical support for our metric as an effective training objective. Using our combined metric and the shortcut training strategy (both described in Sec. 3 - "Training" of our main paper) further improves the performance by a small margin.

7.4 Evaluation of de-biaser performance

Table 5 evaluates the quality of de-biased audio compared to both reverberant and dereverberated audio. De-biased audio is less reverberant than dereverberated audio as measured by both absolute RT60 metric and RTE (computed against the ground truth anechoic waveform for SoundSpacesSpeech). De-biased audio also achieves a higher SRMR score than dereverberated audio, indicating superior quality and intelligibility.

7.5 Model/training details

RT60 estimator We adopt the RT60 estimator from [3]. The estimator takes a spectrogram as input, encodes it with a ResNet18 [18], and outputs a scalar RT60 estimate. The model is trained on 2.56s clips of reverberant speech simulated on the SoundSpaces platform [4] paired with the ground truth RT60 computed from the RIR used to generate the reverberant speech. The model trains using MSE loss between predicted and ground truth RT60 values. Ground truth RT60 is computed using the Schroeder method [39].

De-biaser architecture The de-biaser G takes a magnitude spectogram as input. This is passed to a bi-directional LSTM with input size 257 and two hidden layers each of size 200, which produces an output with the same temporal length as the input spectrogram. This is passed through a linear layer of size 300 and a leakyReLU activation, followed by another linear layer of size 257 and a Sigmoid activation. The final mask is multiplied with the input magnitude spectrogram to create the generated

magnitude spectrogram. A resynthesis module computes phase information from the input audio waveform, combines this with the generated magnitude spectrogram, and performs an inverse STFT to produce the generated waveform. The discriminator D consists of 4 2D Convolutional layers with kernel size (5,5) and 15 output channels, followed by a channel averaging operation and two linear layers of sizes 50 and 10. A LeakyReLU activation with negative slope = 0.3 is used after each intermediate layer. The final layer outputs a scalar-valued metric score estimate.

De-biaser training In stage (1) (see Sec. 3 - "Training" of our main paper), we train with batch size 32. During stage (3) fine-tuning, we use a batch size of 2. G and D are trained with learning rates of 2e-6 and 5e-4 respectively in both stages. In each epoch, We train on 10k samples randomly selected from the train set without replacement. The reverberator models R v and R b are updated with the target networks at a frequency of E = 8 epochs. For all models, we clip each audio sample to 2.56s during training and evaluation.

Reverberator training We train the reverberators with batch size 4 and a learning rate of 1e-2 in stage (2). During stage (3) fine=tuning, we use batch size 2 and a learning rate of 1e-6. Both reverberator models and the ViGAS baseline are trained with MSE loss between the log magnitude spectrogram of predicted and ground truth audio.

Baseline training details We use a learning rate of 1e-2 and a batch size of 4 to train ViGAS. We train A ViTAR with batch size 4 - all other hyperparameters are set as described in [3].

Compute All models are trained on 8 NVIDIA Quadro RTX 6000 GPUs.

7.6 Augmentation strategy

We follow a data augmentation strategy similar to that proposed in [3] for training the baseline models, which was shown to produce better generalization performance on the LibriSpeech setting than when trained without this augmentation strategy. In particular, to each batch of dereverberated audio we add colored noise, perform a polarity inversion on the waveform with p = 0 . 5 , and convolve the waveform with a randomly selected Room Impulse Response (RIR) from a different acoustic environment with p = 0 . 9 . At test time, we evaluate without these audio augmentations. This strategy is designed to mask over residual acoustic information in dereverberated audio during training. We do not use this augmentation strategy in our approach as our model directly learns to remove residual acoustic information, obviating the need for a heuristic strategy to mask it out.

7.7 Limitations

Our approach focuses on visual acoustic matching on mono-channel audio exclusively. However, binaural cues in audio play a fundamental role in our perception of reverberation and room acoustics [8]. We leave it to future work to extend our approach to binaural audio.

7.8 Broader impact

While training on in-the-wild web videos allows wider access to a diverse variety of speakers and environments, it also introduces uncontrolled biases, speaker privacy concerns, and potentially harmful content into the model.

7.9 Data examples

Refer to video to view samples from both SoundSpaces-Speech and AVSpeech-Rooms.

7.10 LeMARA training procedure

7.11 Discriminator epoch - Reverberator update procedure

Algorithm 1 Discriminator training epoch on generated data.

R t v ← R v n ← current epoch #

end for

end if

▷ Initialize target networks with current reverberator network weights.

▷ every E epochs, copy target network weights into current reverberators.

Direct-to-Reverberant Energy Ratio (DRR) vs. Speaker Distance

40

20

R

D

-20

-40

09-

0

5

10

15

20

25

30

35

40

Speaker Distance (m)

STFT Error vs.Speaker Distance (SoundSpaces-Speech)

7.0

-AViTAR

↑-

ViGAS

6.0

-LeMARA

5.0

STFTError

4.0

3.0

2.0

1.0

0.0

2

4

6

8

10

12

14

16

SpeakerDistance(m)

MeanSRMR atVarious Reverberation Levels(SoundSpaces-Speech)

MeanSRMR atVariousReverberation Levels(AVSpeech-Rooms)

de-bias

14

T

dereverb

20.0

17.5

12

15.0

10

MeanS

12.5

de-bias

dereverb

8

10.0

6

7.5

5.0

4

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.2

0.4

0.6

0.8

1.0

Reverberation Level (RT60)

Reverberation Level (RT60)

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors proposed a self-supervised approach to match acoustic conditions via visual information without the need of paired audio-visual data. With this approach, the in-the-wild web data or simulated data can be utilized. They show that this approach can outperforms the state-of-the-art on multiple datasets.\n\nSTRENGTHS:\n- The proposed approach is novel in several aspects, 1. does not require paired audio-visual data, 2. Able to leverage more data from the web or from simulation.\n- The authors provide human perception study, which is important for their claims in perception accuracies.\n- This technique has potential applications in various audio domains such as domain adaptation, help improving in-the-wild acoustic event classification, etc.\n\nWEAKNESSES:\n- The writing of this paper is not easy to follow and missing content, for example in line 200 it refers to Section 4 for details in \"RT60 to allow generalization\", which does not existing in Section 4, mostly focusing on datasets. In line 226, the off-the-self dereverberator is referred to Section 5, which is difficult to locate for the readers.\n- In Section 5, for all the implementation details, suggest the authors to organize and highlight with a table, what data is used to train in which step. Currently it is not easy to follow in the scripts and have clear understanding of the experimental designs.\n- The paragraph of line 188 seems to appear in the wrong place, referred Figure 4 is in the results section, while the narrative appears in the approach section.\n\nQUESTIONS:\n- The use of RT60 needs more explanation, is the intuition to model just the acoustic environment and disentangled with the content involved?\n- Why is Table 1 on the far right, train with AVSpeech-Rooms and test with LibriSpeech does not include comparisons of STFT?\n\nLIMITATIONS:\n- No potential social or ethical implications.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a self-supervised method for visual acoustic matching (VAM), where the training samples consist of only the target scene image and audio. The paper is well-written and easy to follow. The experimental results have validated the effectiveness of the proposed approach. We have the following comments: \n(1) The proposed system is somewhat simple as it incorporates existing methods. Therefore, the scientific depth presented in the paper may not be up to the standards presented at the prestigious conference NeurIPS.\n(2) It would be beneficial to include an analysis of the dereverberator. The authors can evaluate the dereverberator's performance using SRMR with different levels of reverberation. It would also be interesting to see the results with and without the De-biaser component based on SRMR. \n(3) Please provide results for environments with mild, moderate, and severe reverberant conditions. This will help demonstrate the effectiveness of the LeMARA approach more clearly. \n(4) It is crucial to showcase the impact of the De-biaser component. In addition to the spectrograms presented in Figure 4, I recommend that the authors perform additional experiments using supplementary metrics like SRMR or a readily available ASR model. This will provide further evidence and insight into the effectiveness of the De-biaser approach.\n(5) It would be informative to report the results of the LeMARA approach separately for seen and unseen images. \n(6) Figure 4 shows that LeMARA exhibits greater variation in RT60 compared to AViTAR. Please provide an explanation for this observation. \n(7) We suggest preparing an anonymous website where several sets of sound samples of the source, target, AViTAR, and LeMARA can be presented. \n(8) Figure 5 indicates that AViTAR outperforms LeMARA in the left-down example. Please provide an explanation for this result.\n\nSTRENGTHS:\nThe paper introduces a self-supervised method for visual acoustic matching (VAM), where the training samples consist of only the target scene image and audio. The paper is well-written and easy to follow. The relevance of the research task holds significant value in AR/VR systems and can serve as a crucial element. The experimental results have validated the effectiveness of the proposed approach.\n\nWEAKNESSES:\nSome analyses, which can further demonstrate the effectiveness of the proposed approach, are missing, for example:\n(1) The authors can evaluate the dereverberator's performance using SRMR with different levels of reverberation. It would also be interesting to see the results with and without the De-biaser component based on SRMR. \n(2) Please provide results for environments with mild, moderate, and severe reverberant conditions. This will help demonstrate the effectiveness of the LeMARA approach more clearly. \n(3) It is crucial to showcase the impact of the De-biaser component. In addition to the spectrograms presented in Figure 4, I recommend that the authors perform additional experiments using supplementary metrics like SRMR or a readily available ASR model. This will provide further evidence and insight into the effectiveness of the De-biaser approach.\n(4) It would be informative to report the results of the LeMARA approach separately for seen and unseen images. \n(5) In Line 30, there is one reference missing.\n(6) In Fig. 2(b), D(G(A_T)) should be D(G(A_t)).\n\nQUESTIONS:\n(1) A critical point is to showcase the effectiveness of the De-biaser component. Please provide additional experiments to demonstrate its performance.  \n(2) It s interesting to see the achievable performance of LeMARA approach under mild, moderate, and severe reverberant conditions. \n(3) It is crucial to report the results of the LeMARA approach separately for seen and unseen images.\n\nLIMITATIONS:\n(1) The proposed system is somewhat simple as it incorporates existing methods. Therefore, the scientific depth presented in the paper may not be up to the standards presented at the prestigious conference NeurIPS.\n(2) Figure 4 shows that LeMARA exhibits greater variation in RT60 compared to AViTAR. Please provide an explanation for this observation. \n(3) Figure 5 indicates that AViTAR outperforms LeMARA in the left-down example. Please provide an explanation for this result.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the task of \"visual acoustic matching\" (VAM): taking a source audio clip and target visual environment (i.e. an image), and modifying the source audio clip such that it sounds like the clip was recorded in the target environment. The paper proposes a self-supervised approach for training neural networks to solve this task, which can train on examples that only contain the target audio and target visual environment. The idea is to disentangle the room acoustics from the audio content, such that the source room acoustics can be removed, then an audio-visual model can transform this audio to match the target visual environment. The method uses a conditional GAN framework, where an \"acoustic residue\" metric is defined that measures the discrepancy between audio-only and audio-visual reverberation. The acoustic residue metric is used to train a discriminator that can be used to train a \"de-biaser\" generator model that strips additional reverberation from an initially dereverberated signal. The output of the de-biaser can then be fed to the audio-visual reverberator.\n\nAlthough the method can theoretically be applied to other types of signals, the paper focuses on clean speech as the audio signals. Datasets used are SoundSpaces-Speech (reverb impulse responses (RIRs) simulated from Matterport3D scans of homes with 3D-rendered human at source location and audio from LibriSpeech) and AVSpeech-Rooms (YouTube videos that mostly feature a single speaker with little background noise). Objective metrics are used to measure performance: MSE between magnitude spectrograms of predicted and ground-truth speech, and MSE between RT60 estimates of predicted and ground-truth speech. The proposed method achieves better objective scores compared to comparable methods. A perceptual study is also done, which shows users could identify the room from the proposed method 46.1% of the time, versus 34.7% for a baseline method.\n\nSTRENGTHS:\n**S1)** The method provides a means of training self-supervised models for VAM.\n\n**S2)** Evaluation includes a human perceptual study.\n\n**S3)** Demo video is clear and helps with understanding the method. Also, thanks for providing audio demos, they are very useful for evaluating the method and how it compares to the AViTAR baseline.\n\nWEAKNESSES:\n**W1)** The evaluation of the method is weak. Watching the demo video, it seems like this is certainly a difficult task to evaluate, since the effect can be rather subtle. Nevertheless, I think the evaluation metrics could be improved.\n\nFirst, there are better options than just measuring MSE on magnitude spectrograms. Human hearing is logarithmic, so MSE on linear magnitude is a poor match to human perception. An easy alternative is to measure MSE between log spectrograms, although this encounters an issue with where to set the floor on the log to avoid -infinity. A solution to this is to use magnitude raised to a power (e.g. 0.3), which approximates the log, but also goes to 0 where the magnitude is 0. This may help this this issue: \"Models that use this dereverberator without further de-biasing will display artificially low STFT error when evaluated in-dataset\"\n\nSecond, there are other important properties of reverb impulse responses besides RT60, such as direct-to-reverberant ratio (DRR). DRR measures the ratio of energy of the direct path to the energy of the reverberant part, and can be an important cue for distance of the source. Also, distance of the source is not really discussed, and DRR is a crucial property to help measure this (see W3).\n\nThird, the comparison of the RT60 distributions in Figure 4 are not very convincing, because they are just box plots, and it's not clear how the proposed approach \"more closely matches the ground truth target distribution.\": the median is closer to ground-truth than the baseline, but it seems like a different type of plot would allow more detailed comparison of the distributions, like histograms or violin plots.\n\n**W2)** The paper restricts its focus to a clean single speaker, which suggests the method may be limited on real-world data, which can contain a great variety of non-speech sounds, and also multiple speakers. I don't think the proposed method could be applied directly to these scenarios, because all sounds would need to first be separated, then each processed by the pipeline (i.e. each sound dereverberated and re-reverberated)\n\n**W3)** The paper does not discuss the effect of distance of an object, nor the effect of distance on the reverberation. In particular, the DRR is an important property of reverberation that is totally ignored in this paper. At the very least, assumptions about how the method handles distance and location of sources needs to be clearly described and discussed, and also I think evaluation metrics should take distance into account (see W1).\n\n**Minor comments and typos**\n\na) \"data, we\" -> \"data, and we\"\n\nb) \"target space [?,..\": update missing ref\n\nc) \"In-the-wild Web\" -> \"In-the-wild web\"\n\nd) \"We focus on human speech in indoor settings\": clearly specify that it's clean speech, without background noise\n\ne) \"this leaves signals of the target environment\": I think \"residuals\" would be better than \"signals\"\n\nf) \"Unlike SRMR, DNSMOS [31], or any existing off-the-shelf metric that quantifies dereverberation,\": I don't think DNS-MOS is trained to quantify reverberation specifically, it's overall quality, which can be affected by other properties such as background noise and/or artifacts. May be good to adjust this description\n\nQUESTIONS:\nQ1) Looking at the spectrograms of de-biased audio in Figure 4 and listening in the demo video, de-biased audio seems like it is applying fairly aggressive suppression, perhaps removing some energy of the anechoic audio. This seems like a weakness of the method. Are there mechanisms to prevent over-suppression of the anechoic audio? Does this suggest a trade-off for the de-biaser between dereverberating and suppressing signal? Some discussion of this would be helpful.\n\nQ2) In Figure 2, where does \"A\" come from (lower right panel)? I guess this is the de-biased audio? Would be good to make this more clear\n\nLIMITATIONS:\nLimitations described adequately.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a method for visual acoustic matching, the task of processing an audio signal so the room acoustics are perceived as originating from within a certain room based on an image. While paired data is generally required for this task, the proposed approach is self-supervised and trained using “in-the-wild” videos of speakers in various rooms. To achieve this, a dereverberation system along with a de-biaser, a visually-conditioned reverberator, and a blind reverberator are used. The de-biaser is trained by optimizing a novel acoustic residue metric that measures the relative difference between the amount of acoustic information in two recordings. While similar to dereverberation, this de-biasing process can be seen as post-processing step that aims to further reduce information in the dereverberated output so the visually conditioned reverberator model must use cues from the supplied image to match the target room acoustics. To use this acoustic residue metric as an objective function, the de-biaser is trained in a MetricGAN fashion where the discriminator is trained to regress the true metric value. Two evaluation datasets are used and results indicate that the proposed method outperforms existing baseline systems. In addition, a human perception study also validates these results.\n\nSTRENGTHS:\n1. The proposed method is original and focuses on combining multiple existing approaches along with a novel acoustic residue metric in order to achieve a system that can be trained without paired data. These approaches include the use of various pretrained submodules, including a dereverberation model and a RT60 estimation model, as well as the MetricGAN approach that enables the use of the acoustic residue metric during optimization.\n\n2. As outlined by the authors, existing approaches are often greatly limited by the availability of paired data. Not only is the proposed approach capable of being trained without paired data, the evaluation indicates superior performance to existing methods.\n\nWEAKNESSES:\n1. Some relevant references for work in audio-only acoustic matching are omitted. Works such as [1] are similar in nature but rely on conditioning from an audio signal instead of an image. Major components of the reverberator architecture are similar as well, such as the use of a WaveNet architecture for audio processing. This would be highly relevant in the context of the blind reverberator in this work. Also, other audio only approaches such as [2] would be relevant to include in the context of related work. It would also be relevant to address other GAN-based reverberation models such as [3, 4].\n\n2. While overall the manuscript is well organized and clear, further refinement with regards to the explanation of the process for training the complete system may help reduce the burden for readers. For example, perhaps an algorithm listing each step of the process and referencing the variables used in the text would be helpful. \n\n3. While the evaluation considers two datasets and appears relatively sound, it is still somewhat difficult to understand the relative improvement afforded by the proposed method in comparison to the baselines. \n\n4. While the authors are commended for including a human perception study, the design of the study may be somewhat flawed, making it difficult to make any conclusion from the results. \n\n\n[1]\tSu, Jiaqi, Zeyu Jin, and Adam Finkelstein. \"Acoustic matching by embedding impulse responses.\" ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020.\n\n[2]\tSteinmetz, Christian J., Vamsi Krishna Ithapu, and Paul Calamia. \"Filtered noise shaping for time domain room impulse response estimation from reverberant speech.\" 2021 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA). IEEE, 2021.\n\n[3]\tRatnarajah, Anton, Zhenyu Tang, and Dinesh Manocha. \"IR-GAN: room impulse response generator for speech augmentation.\" INTERSPEECH (2021).\t\n\n[4]\tRatnarajah, Anton, Zhenyu Tang, and Dinesh Manocha. \"Ts-rir: Translated synthetic room impulse responses for speech augmentation.\" 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2021.\n\nQUESTIONS:\nIs it fair to call the proposed approach “self-supervised”? While the approach does not require paired data directly, it appears that paired data is indirectly required since pretrained models, such as the RT60 estimator and dereverberation models. In other words, it appears that it would not be possible to train the proposed system in a fully self-supervised manner from scratch, without these pretrained modules, which in fact use labeled/paired data. Instead, it may be more accurate to label the proposed method as one capable of using an “unpaired” dataset instead of self-supervised. Perhaps terms such as “semi-supervised” or “weakly-supervised” are more appropriate? \n\nWhat is the motivation for using the MetricGAN approach to construct a differentiable proxy of the acoustic residue metric $\\mathcal{M}$? Based on the text, it appears that the RT60 estimation model $\\mathcal{RT}$ is implemented with a neural network and hence differentiable. While (3) involves non-differentiable functions such as the absolute value and $\\max$, these are both approximately differentiably, indicating that it may be possible to directly backprop through the metric $\\mathcal{M}$. Is this something the authors considered? If so, was the failure of this approach the motivation to use the MetricGAN approach? Some discussion on the motivation for the MetricGAN approach would be beneficial. \n\n\nLine 30: A reference has not been rendered properly “?”\n\nLIMITATIONS:\nIt could be beneficial to include not only the mean RT60 error (RTE (s) as reported in Table 1, but also the bias and correlation coefficient, which is common practice in RT60 estimation evaluation. This can help to establish how the model is failing especially if the test dataset has a non uniform distribution of RT60 targets. \n\nOne limitation of the use of the RT60 estimation in the proposed acoustic residue metric is that this measurement appears to consider only the broadband RT60. It is generally accepted that in most rooms the RT60 can vary quite significantly across frequency, for example, with the higher frequencies decaying at a significantly faster rate than the low frequencies (generally due to absorption from the material properties). As a result, this variation of RT60 across frequency often provides significant insight into the character of the room, which is often detectable by a human listener. Therefore, using only the broadband RT60 in the acoustic residue metric may lead to some cues about the bandwise RT60 remaining in the de-biased recording as long as the broadband RT60 is perturbed. Future work could consider a band-wise RT60 estimation.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This work aims to perform acoustic matching with unpaired training data, i.e. observing the audio only in the target environment without samples of the same audio in the source environment. The basic process involves the common process of dereverberator and reverberator. But one key point in this work is to introduce an acoustic residue metric to measure the residual information in a waveform to ensure the quality of dereverberated audio. Finally, experiments on SoundSpaces-Speech and AV Speech show its effectiveness.

STRENGTHS:
The presentation is very clear. The motivation to tackle the task is also intuitive and meaningful. The method is reasonable. I like this smart and simple manner proposed in this paper to tackle the key limitation in visual acoustic matching.

WEAKNESSES:
(1) Is it limited to evaluate with only two metrics, STFT and RTE? (2) Although the number of methods using unpaired data may be few, the number of methods using paired data should be large. There could be some comparison with them to show the performance level of the proposed method. For example, comparing on other paired datasets with existing methods which use paired data could show the difference of the methods use and not use paired data.

QUESTIONS:
(1) The error of STFT changes obviously in AVSpeech-Rooms compared with other settings, i.e. about 6~7 vs 1~2, what’s the possible reason? I t would be nice to give the clarification. (2) Two questions about the evaluation metrics and the comparison are as listed in weakness.

LIMITATIONS:
yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-10